The following equation defines
information:
Example 1: a scientist is told to toss a coin 3 times and remember the
results. How much information does he acquire when he observes the result? Answer: The
total number of possible outcomes is 2x2x2 = 8, and only 1 outcome will be observed. So 2 (information)
= 8/1. Because 23 = 2x2x2 =8, the scientist acquires 3 bits of information.
Example 2: Suppose the result in example 1 is H-H-T, but the scientist is unsure about the
outcome because he cannot remember whether the first coin landed head or tails. How much
information has he acquired? Answer: there are still 8 possible outcomes. The scientist
observed either H-H-T or T-H-T, but he is not sure which. So both must be counted as
observed outcomes. So 2 (information) = 8/2 =4. Because 22= 2x2 =4,
the scientist acquires 2 bits of information.
Note that the odds of an event like winning the lottery are often expressed
as one in some number, like a million. The one is the observed outcome, and the million
represents the total number of possible outcomes. So the information acquired by knowing
who wins this lottery is easy to calculate: 2(information) = (1 million
outcomes / 1outcome). Because 220= 1,048,576, approximately 20 bits of
information are acquired when the outcome of this lottery is observed.
The equation for information can be solved explicitly for information with
the use of logarithms. Since many calculators have a log function, the next equation is
often easier to use, but less intuitive than the first. See appendix 4 for more
information.
Next: Information Theory
Previous: Information and Probability |
