vol 2: Synopsis
part IV: Divine dynamics
page 28 Claude Shannon
Site map
Directory
Search this site
Home
1: About
2: Synopsis
3: Development
Next: page 29: Quantum information theory
Previous: page 27: Alan Turing
This site is part of the natural religion project The natural religion project A new theology A commentary on the Summa The theology company
Claude Shannon
(1916-2001)
Shannon founded the mathematical theory which underlies the current revolution in communication. He first defined information, and then showed how to transmit information faultlessly over a noisy channel by suitable encoding. The requirements for a good code have guided the search for codes ever since, and we can see that communication in the natural world is governed by the same model. From Shannon's time on, people have begun to look at the universe more as an information (entropy) than an energy processor.
Information is defined as 'that which removes uncertainty'. The measure of an item of information is the amount of uncertainty it removes. Shannon chose entropy as his measure of uncertainty. Entropy had entered physics through the thermodynamics of heat engines. In thermodynamics it serves as a measure of disorder, complexity, or the quality of energy. The second law of thermodynamics tells us that entropy does not decrease.
The mathematical expression of entropy goes beyond its physical origins. We now see it as a measure of the size of a space. The information necessary to specify a point in any space is equal to the entropy of the space. So, if we think of the English language as a space of, say, one million words, we may say that the entropy of English is one million per word, and the transmission of one English word conveys one million units of information.
The mathematical definition of entropy uses a logarithmic scale, and takes into account the relative frequency of different points in the space to be measured, but entropy remains fundamentally a count of points in space. The essence of error free communication is for the sender to transmit designations of points in the communication space to the receiver without error. This is achieved by coding messages so that points of interest are as far apart as possible, and so unlikely to be confused.
Shannon modelled a communication system as a source of messages, and a channel through which the messages are transmitted. The source is understood to emit a sequence of symbols drawn from a certain alphabet. By considering the number of these symbols and the probability of each, we calculate the source entropy. A similar calculation yields the entropy of the channel. Shannon found that a channel, no matter how prone to error, can transmit the output of a given source provided that the channel entropy is sufficiently greater than the source entropy.
The trick lies in encoding. The entropy of any source with a fixed alphabet is maximized when all the symbols are equiprobable. In most realistic cases, this is not so. We see this in our own language. Some words, like a, and, the etc are very common, whereas others, like quixotic or entrepreneurial are very rare. The entropy of the average source of English words is much less than its theoretical maximum. This sub-optimal coding is called redundancy, and can be exploited as a barrier against error.
Coding takes advantage of redundancy by transforming the original output of the source into a new string of symbols, arranging things so that these symbols are equiprobable. Such coding exploits the full entropy of the channel. Coding works by collecting sections of the source output and transmitting them as large blocks which have a negligible change of being confused with one another. The receiver then decodes these blocks to get the original message.
We see this mechanism operating in nature. Although physicists talk of 'wave-particle duality', all observable communications in the universe in fact comprise strings of particles, each one separate and easily distinguished from its fellows.
Books
| Brillouin, Leon, Science and Information Theory, Academic 1962 Introduction: 'A new territory was conquered for the sciences when the theory of information was recently developed. ... Physics enters the picture when we discover a remarkable likeness between information and entropy. ... The efficiency of an experiment can be defined as the ratio of information obtained to the associated increase in entropy. This efficiency is always smaller than unity, according to the generalised Carnot principle. ... ' Amazon back |
| Campbell, Jeremy, Grammatical Man: Information, Entropy, Language and Life, Allen Lane 1982 Amazon back |
| Chaitin, Gregory J, Information, Randomness & Incompleteness: Papers on Algorithmic Information Theory, World Scientific 1987 Jacket: 'Algorithmic information theory is a branch of computational complexity theory concerned with the size of computer programs rather than with their running time. ... The theory combines features of probability theory, information theory, statistical mechanics and thermodynamics, and recursive function or computability theory. ... [A] major application of algorithmic information theory has been the dramatic new light it throws on Goedel's famous incompleteness theorem and on the limitations of the axiomatic method. ...' Amazon back |
| Gatlin, Lila L, Information Theory and the Living System, Columbia University Press 1972 Amazon back |
| Khinchin, A I, Mathematical Foundations of Information Theory (translated by P A Silvermann and M D Friedman), Dover 1957 Jacket: 'The first comprehensive introduction to information theory, this book places the work begun by Shannon and continued by McMillan, Feinstein and Khinchin on a rigorous mathematical basis. For the first time, mathematicians, statisticians, physicists, cyberneticists and communications engineers are offered a lucid, comprehensive introduction to this rapidly growing field.' Amazon back |
| Pierce, John Robinson, An Introduction to Information Theory: Symbols Signals and Noise, Dover 1980 Jacket: 'Behind the familiar surfaces of the telephone, radio and television lies a sophisticated and intriguing body of knowledge known as information theory. This is the theory that has permitted the rapid development of all forms of communication ... Even more revolutionary progress is expected in the future.' Amazon back |
| Shannon, Claude, and Warren Weaver, The Mathematical Theory of Communication, University of Illinois Press 1949 'Before this there was no universal way of measuring the complexities of messages or the capabilities of circuits to transmit them. Shannon gave us a mathematical way...invaluable...to scientists and engineers the world over." Scientific American http://www.amazon.com/exec/obidos/ASIN/0252725484/tnrp">Amazon back |
Click on an "Amazon" link in the booklist at the foot of the page to buy the book, see more details or search for similar items
Related sites:
Concordat Watch
Revealing Vatican attempts to propagate its religion by international treaty
Copyright: You may copy this material freely provided only that you quote fairly and provide a link (or reference) to your source.
