Suppose that the scientist in figure
1.1 cannot see the results of the coin toss because a screen is placed between him and the
coins (figure 1.2).
Figure 1.2: Trapped Scientist Who Cannot See the Results
In figure 1.2, the camera monitors the results of
the coin toss and sends the results to the computer. Depending on the results of the coin
toss, the computer is programed to do four things: open the door, close the door, beep
once, and beep twice.
Coin 1 Coin 2 Coin 3
H H
H --------->opens the door
H H
T --------->opens the door
H T
H --------->closes the door
H T
T --------->closes the door
T H
H --------->computer beeps twice
T H
T --------->computer beeps twice
T T
H --------->computer beeps once
T T
T --------->computer beeps once
The scientist cannot observe the results of the coins. He can only observe
what the door and computer do after he tosses all three coins. The door opens for 2 of the
8 possible results. If the door is already open, and the camera observes H-H-H or H-H-T
then the door will stay open. Two results will close the door if it is open, and have no
effect if the door is already shut. Two results will cause the computer to beep once, and
two results will cause the computer to beep twice. Does the scientist still acquire 3 bits
of information when he tosses three coins?
Case 1: the door opens or stays open.
2(information) = (8 possible outcomes/2 outcomes that cause this result) =4.
Since 22 =4, this result contains 2 bits of information.
Case 2: the door closes or stays closed.
2(information) = (8 possible outcomes/2 outcomes that cause this result) =4. So
this result also contains 2 bits of information.
Case 3: the computer beeps once, 2 bits of
information are acquired.
Case 4: the computer beeps twice, 2 bits of
information are acquired.
The average amount of information acquired each time the scientist
tosses all 3 coins is now 2 bits. He is using 3 coins or 3 bits to transmit 2 bits of
information. He must do this because the code that translates the result of the coin toss
into what the door and computer do is not the optimal code. The optimal code should only
require 2 coins to transmit 2 bits. One possible optimal code is as follows:
Coin 1 Coin 2
H H --------->opens the
door
H T --------->closes the
door
T H --------->computer
beeps once
T T --------->computer
beeps twice
The average uncertainty per symbol (or coin in this example) is called the
Shannon entropy. Shannon entropy* measures on average how much each observed symbol or
coin decreases uncertainty. Because information corresponds to a reduction in uncertainty,
Shannon entropy is also a measure of information. When 3 coins are used to transmit 2 bits
(non-optimal code), the Shannon entropy is 2/3 of a bit per coin. With the optimal code,
the Shannon entropy becomes 1 bit per coin. The total information transmitted in both
cases is the same because 3 coins x 2/3 bit per coin = 2 coins x 1 bit per coin = 2 bits.
*Shannon entropy should not be confused with the term entropy as it is used in chemistry
and physics. Shannon entropy does not depend on temperature. Therefore, it is not the same
as thermodynamic entropy.1 Shannon entropy is a more general term that can be used to
reflect the uncertainty of any system. Thermodynamic entropy is confined to physical
systems.
References:
1) Brillouin, Science and Information Theory, 1956.
2) Reza, An Introduction to Information, 1961.
3) Pierce, An Introduction to Information Theory, Symbols, Signals and Noise, 1961.
Previous: Definition of Information
Home: Theory of Evolution
|
