Ever wonder how humanity went from waving giant wooden arms on towers in 1790s France to doom-scrolling cat memes in seconds? This wild ride through the chaotic history of information theory has it all: drunk inventors, spy telegraphs, and one quiet genius named Claude Shannon who finally tamed the chaos with something called a “bit.” If you like your tech history equal parts hilarious and mind-blowing, buckle up — you’ll never look at your phone the same way again.
Time and space precluded without forgetting or planning either way our life’s sphere of influence, when we mapped thought to math it was over for divinity as it were. Perspective of that fact is not necessary for its fundament or elemental function.
But, for our localized perception lets skip to…1939 Europe has descended into total war Across the Atlantic the strategy in the United States is to wait and see But behind the scenes American scientists and engineers were working on ideas that would not only help win the war but they would change the course of history At Bell Labs a young engineer fresh out of MIT Claude Shannon spent his days finding ways to encrypt speech to keep it hidden from the enemy But at night he was working on his own idea one that would go far beyond the war effort It was an idea that would revolutionize the very way we think about information But Shannon’s theory had its foundations in the work of earlier pioneers including his colleagues at Bell Labs Nyquist and Hartley and even further in the intellectual revolutions of the Renaissance and industrial age…
It is the summer of 1794 Two mighty armies are about to clash near the town of Larus On one side the Austro Dutch coalition stand for monarchy and the preservation of the aristocracy On the other side the revolutionary French army see themselves as liberators spreading democracy and liberty The technology of warfare saw two major innovations that day The first the invading French army used a balloon for aerial reconnaissance a first in battle And as the battle raged the French deployed the second It would normally take about 24 hours for a message to be sent from this distance back to Paris but the French had a trick up their sleeves This was the first military application of a new type of longrange communication system called the optical semaphore The invention was a shape-shifting structure which could signal information almost instantaneously across vast distances As the musket smoke cleared Europe had entered a new era of warfare one where rapid communication shaped the outcome of battle The key innovation was invented by Claude Chapper His semaphore could be positioned into 92 distinct positions Each a separate configuration of the beam and sidearms These would be called symbols
And into each symbol could be encoded one of 92 possible messages words sentences anything the coder desired But Chapa’s system went one step further Instead of coding into the 92 primary symbols he coded his message into a set of two consecutive symbols This now gave 92* 92 or 8,464 unique combinations This larger set would be known as secondary symbols The system used a code book The first primary symbol told the receiver the page and the second the index on the page To the French military important commands like attack or retreat would be coded to a particular secondary symbol as would every other word or sentence in the code book Chapa’s system was a revelation and soon his semaphore chains had spread across France Chaper had started something that would not be fully resolved for 150 years
Electromagnetism and the Morse Telegraph (04:47 – 11:25)
In 1820 a discovery was made that would fundamentally transform communication In his laboratory in Copenhagen Hans Christian observed that an electric current could deflect a magnetic needle revealing the connection between electricity and magnetism Remarkably Unsteady found this effect was repeatable He used a primitive battery a conductor and a small compass When he closed the circuit the needle deflected from its normal north south position When he turned the current off the magnetic needle returned to its usual position Earth’s findings would send shock waves around Europe And this breakthrough laid the groundwork for a revolution Unlike Claude Chaper’s Semaphore a slow clunky system of visual symbols that could be easily intercepted electricity offered a new possibility sending information as pulses through hidden conductors traveling far beyond the reach of the human eye Urst discovery ushered in a new age of communication
Samuel Morse had been in Europe since 1829 Now in 1832 it was time to go home On the package ship Sully Morse entered a discussion with Charles Thomas Jackson a physician and scientist well-versed in the emerging field of electromagnetism Jackson noted to Morse that electricity travel through wires at near instantaneous speeds Morse an artist by trade immediately recognized the possibilities Jackson also remarked that a new device capable of detecting electric currents had been invented in Europe The idea was planted in Morse’s mind and when he returned to the United States he immediately got to work
January the 11th 1838 a bitterly cold chill filled the air in Morristown’s Speedwell Iron Works It would be here that the first public demonstration of Morse’s new telegraph system would be held By now Morse had help in the form of business partner Alfred Vale Whilst Morse was an artist and a visionary Vale was an engineer a machinist a practical problem solver Between them they invented a machine which could impress pulses onto electrical conductors These pulses would travel for tens or hundreds of miles Vale assigned a sequence of dots single pulses and dashes double pulses to encode each letter and number But Morse and Vale were not the only show in town They had competition In Great Britain Cook and Wheatstone had their own invention A telegraph system of five magnetic needles and six wires Pulses were sent in parallel The operator at one end would press the keys and within moments the needles at the other end would swing into position spelling out the message letter by letter But in its simplest form Morse’s system only needed a single wire It was cheaper and easier to manufacture but it could only send one pulse at a time But out of this limitation Vale hit on something that would later be fundamental to information theory Vale studied the statistics of the English language In particular the relative frequency of each letter He noted that letters like E were far more frequent than others For example the letter zed From this he assigned shorter codes to more frequent letters and longer codes to less frequent letters Overall this meant fewer pulses on average were required to send messages Morrison Vale had no mathematical basis for doing this It was pure intuition but it worked On May 24th 1844 Morrison Vale conducted another public demonstration except this time it would rock the foundations of America Professor Morse seated amidst a hushed gathering of distinguished national leaders in the chambers of the United States Supreme Court in Washington tapped out a message on a device of cogs and coiled wires 40 mi away in Baltimore Alfred Vale received the electric signals and sent the message back For thousands of years messages have been limited by the speed at which messengers could travel and the distance at which eyes could see signals Neither Alexander the Great nor Benjamin Franklin knew anything faster than a galloping horse Now instant longrange communication had become a reality
By 1876 Morse’s system had dominated world telegraphy But it was a manual laborintensive enterprise Thousands of skilled operators were necessary to process the sequence of clicks into words and numbers Emile Bordeaux was a French engineer who saw an opportunity to automate telegraphy He invented an encoding scheme called Bordeaux code Rather than using a variable length code he assigned each letter or number to a fixed 5-bit code capable of sending one of 32 possibilities Looking back through the lens of information theory we realize that Bordeaux’s code is less efficient because it fails to consider the redundancy in the information source No matter how infrequent every letter or number is given the same five-bit code But what Bordeaux gave us is a way to automate His telepinter seamlessly impressed the necessary five bits onto the line just with the click of a keyboard The receiver also automatically read the signal back This was the seeds of the modern digital age
Chapter 3: Hartley, Nyquist, and Pre-Shannon Quantifications (11:26 – 18:15)
1927 The globe is now covered in a patchwork of lines and cables Morse and Bodau had provided near instantaneous global connectivity But there was something missing There was as yet no theory to quantify information still no way to put bounds on the limits of information capacity and still no grasp of how noise or bandwidth shaped what a wire or wave could carry The analytical framework of information theory was ready to be built and two men in the late 1920s would lay the foundations
In September 1927 Ralph Hartley an employee of Bell Labs made his way to the International Congress of Telegraphy and Telefan in Bonito Mussolini’s Italy On the shores of Lake Ko Hartley presented a paper which claimed to quantify information It would go down as one of the seinal works of information theory Hartley claimed to have an objective measure of information removed from what he called the elastic manner in which information had previously been defined He defined the communication problem a group of physical symbols which convey a certain meaning as assigned by the designer of the system This might be the visual semaphors of chaper or the dots and dashes of Morse code But now Hartley considered what happens if we send symbols consecutively The total number of symbol sequences Hartley said becomes greater the more symbols we send Every time we select an additional symbol we select one message from an ever growing array of possible messages For example apples are red These three words represent one precise specific message from an exponentially growing set of possible messages In the same way if we send consecutive symbols each new selection grows the total number of possible messages exponentially Hartley’s idea was simple but profound The quantity of information is bound to the total possible messages we select from And this said Hartley can be quantified with mathematical precision Let’s say we have a symbol alphabet of three and two consecutive selections There are 3 to the power of two or nine possible messages But with three selections there are now 27 Four selections would give 81 possible ways to encode a message Consider Chapper’s Semaphore He had 92 primary symbols to select from But recall that he used two consecutive symbols This gives 92* 92 possible ways to encode a message Each new consecutive selection in Chapa’s system gave a factor of 92 increase In Bordeaux’s teleprinter there are 32 ways to encode a message But if n selections are made the number of possibilities increases by 32 to the power of n Each new selection broadens the total number of possible sequences we can use to assign messages to And when we select one message from a larger set the information content said Hartley is greater But Hartley immediately hit a problem Imagine Chapper decided to use a code book with 10 consecutive messages instead of two Now there would be 92 to the power of 10 possible ways to encode a message This is greater than the number of stars in the observable universe If this is our measure of information it scales up very very very quickly In his attempt to quantify information Hartley had observed a monster that grows exponentially out of control For this reason Hartley made a suggestion Rather than use the raw number of possible sequences it is far more sensible to quantify information as its logarithm This means as the number of selections goes up information increases linearly not exponentially
Hartley talked about sending messages comprised of sequences of symbols But to send a symbol we need a signal Something which represents the symbol we’re trying to send Nyquist was the first to make the connection between discrete symbols and analog signals He looked at the pulses of a telegraph system sequences of ones and zeros But he went further He decomposed these pulses into their sinosoidal components And he realized something fundamental As a signal travels through a channel higher frequencies get stripped away The extent to which this happens depends on the bandwidth of a channel The problem is that your signal becomes distorted Nyquist realized that there is a connection between how many signaling events or symbols can be transmitted per second and the bandwidth of the channel And this places a fundamental limit on how many signals can be sent per second
In 1937 a 21-year-old from a small town in Michigan submitted his master’s dissertation at MIT Many now regard this thesis as the birth of digital electronics and the greatest master’s thesis in all of engineering history Claude Shannon had showed how the mathematical algebra of bool the logic of ones and zeros true and false could be applied to electrical circuits and it was a blueprint for digital logic the foundation of computers and all modern digital systems Shannon went on to receive his PhD degree in theoretical genetics and later took up a position at Bell Labs It was here that he would spend his days contributing to wartime cryptography but at night he was working on a side project
Chapter 4: Shannon’s Framework and the Introduction of Entropy (18:16 – 27:02)
In 1948 Claude Shannon proposed a set of ideas that would revolutionize the way we think about information But his starting point was indeed the observation of Hartley A measure of information is the number of messages in the set we select from all choices being equally likely And Shannon also agreed with Hartley that the logarithm was the most natural way to measure information He proposed that the base 2 is the most natural measurement of information This is what we know as a bit Then Shannon defined the problem A communication system is made up of an information source which produces a sequence of messages A transmitter which operates on the messages to produce the signal like the pulses of a telegraph The medium or the channel which is used to transmit the signal A receiver which performs the inverse operation of the transmitter to reconstruct the original sequence of messages And finally the destination the intended recipient of the message Hartley had provided the first real quantitative measure of information He said that the amount of information transfer is the log of the total number of available sequences from which that message is selected using the base 2 This means chaper semaphore has a rate of 6 1/2 bits per symbol and Bordo’s teleprinter can send five bits per symbol But Shannon realized that the channel by itself does not convey information A position of chapa semaphore or a particular sequence of ones and zeros in Bordeaux’s teleprinter does not in itself convey a message It simply provides a way to transport information from the information source This might seem counterintuitive Look at how Chapper and Bordeaux allocated their messages to channel signals Every message was allocated to every channel signal But Shannon showed something remarkable The information source itself has a rate something innate an essence of the source rooted in its own behavior By separating the information source and the channel Shannon had made the first step towards a number of groundbreaking discoveries
As a preamble Shannon showed how the capacity of an arbitrary channel can be calculated Unlike the information source the channel capacity is simply a function of how many sequences can be constructed from the available symbols And we’ve already seen that the capacity of the Bordeaux teleprinter is 5 bits per symbol So over time the capacity is 5 n bits per second where n is the transmitted symbols per second But Shannon also showed how the capacity of Morse’s telegraph system can be calculated This one is different because we’re dealing with symbols of different lengths and restrictions on the kind of sequences we can have Nonetheless Shannon showed that the capacity is the limit as t tends to infinity of the log of the allowable sequences divided by time But the impact of Shannon’s next section would be monumental
He turned his attention to the information source and he asked a question nobody had asked before At what rate is the source generating information this is separate to asking what capacity a channel has What Shannon saw that Hartley did not is an information source that is not a random generator of symbols but a statistical process Each new symbol generated is not independent of the previous symbols When we count sequences some are more likely than others Some sequences are so unlikely that they might as well never occur at all This is most easily observed in human language There are of course a huge number of ways we can arrange words but only a small subset of these possible sequences are commonly used Hartley’s approach was to count all possible sequences But Shannon introduced a new perspective one that considered the likelihood of different sequences occurring He then developed a model a mathematical machine capable of producing symbols and generating information He modeled an information source as a markoff process where the source is represented as a complex web of interconnected states that transition between each other randomly but according to set probabilistic rules If this was a language these rules would be dictated by the normal structure of that language But any information source has this kind of statistical behavior It contains patterns repetition and redundancy Out of Shannon’s machine came symbols But how can we measure how much information is being generated shannon produced an idea that would become a fundamental pillar of not just information theory but data compression cryptography machine learning artificial intelligence economics and neuroscience His idea was simple Information is best measured as the uncertainty resolved not just the number of possible choices
Imagine you flip a fair coin Hartley had said that the information is the log of the possible choices which here is one bit of information per flip You can keep flipping the coin and each new event will give one extra bit of information Now consider if we make the coin biased so there’s an 80% chance of heads According to Hartley there are still two possible outcomes and the information is still one bit per flip of the coin But is this correct shannon realized that the measure of choice gets smaller as the probabilities of the outcomes become more skewed In reality information is the log of a smaller number of effective choices Shannon called this new measure of information entropy
To calculate entropy Shannon calculated the surprise of each new selection from an information source This quantifies how surprised we are that this symbol has been chosen or in other words how much uncertainty that selection has resolved For example if we have four options and if we select the symbol with a 50% probability this is one bit of surprise Even though this is within a symbol space of four selecting the 50% option provides the same amount of information as flipping a coin But the 12 1/2% symbol provides three bits of surprise because it’s more unlikely So more uncertainty is being resolved For the flip of a fair coin the surprise is always one bit But what happens if the coin is biased of course the more likely option is chosen most often but the surprise is relatively low But every so often the unlikely event happens The surprise is greater But what Shannon showed is entropy is the average value of this surprise and it maximizes when the probabilities are equal Entropy is the weighted sum of the surprise values of each possible outcome weighted by how likely each outcome is Consider again an event with two possible outcomes with probability P and Q If this was an information source its entropy would be related to how balanced these probabilities are If one outcome is certain entropy is zero No information is gained if we learn the outcome But if the probabilities are balanced this conveys the maximum amount of information and the entropy is at its highest However if the probabilities are skewed the entropy inevitably drops But why does this matter we still need to encode every possible symbol into the channel right is entropy just a philosophical and academic measure of what we perceive as information what Shannon did next is prove that entropy places a fundamental lower bound on how efficiently we can represent information and therefore make best use of the channel
Chapter 5: Source Compression and Typical Sets (27:03 – 33:22)
Suppose I have an information source which is represented here as four colors So there’s a 50% chance I’ll select red 25% chance I’ll select blue and a 12% chance I’ll select white or black I’m going to put these inside a bag and I’ll select randomly So we need to assign a code to each one of these symbols to convey this over a channel So the question is which code should we assign to each of the colors to get the most efficient coding scheme and it’s going to be a random selection When we think about coding these symbols we might follow the advice of Bordeaux and assign a fixed code In this case we require two bits to describe all four possible outcomes Of course the average number of bits per symbol will always be two bits But can we do better let’s try a variable length code And the trick is to assign a code length which matches the surprise of each symbol So when red is selected there’s a 50% chance So there’s one bit of surprise With blue a 25% chance two bits of surprise Matching code length to surprise gives us the optimum coding scheme This is the direction of course that Morse and Veil were heading all those years ago The key is assigning shorter code words to more frequent symbols In this experiment we achieved an average of 1.78 bits per symbol which is getting close to the entropy of the source 1.75 bits But if we carried on the experiment we would actually converge on this value The job of the coder is to reshape the output of the source into a bitstream whose structure is statistically close to the most efficient possible use of the channel which is when symbols are encoded near their surprise level Shannon said that doing this will allow you to maximize the number of symbols which can be conveyed and make full use of the channel capacity In the previous example if our channel capacity was let’s say 14 bits per second we could send seven symbols per second with the bad encoding scheme But if we use the good encoding scheme we could send eight symbols per second In general Shannon said we can compress an information source down to its entropy But if we go lower we are almost guaranteed to lose information But we need to be careful because not every information source is so easy to encode The example we just looked at is a very special case It works out neatly because the symbols are independent and follow a simple probability distribution But what about something more complex like the English language in this case we can get close to the true entropy if we encode over long sequences of text That’s because the statistical structure of the language like grammar word choice and context only really show up when we look at larger chunks Letters aren’t independent and some combinations are far more likely than others So how did Shannon prove his theory was correct in all cases that you can compress information down to its entropy in the general case how did Shannon prove that any source can be compressed down to its entropy to demonstrate I’ll start with the simple information source we used previously Remember there’s a 50% chance we’ll pick a red 25% chance we’ll pick a blue and 12% chance we’ll pick a black or a white I’m going to pick at random again and try to generate long sequences But because I don’t have the patience to do it for real I’ve written a piece of code to simulate this information source And I’m going to simulate lots of these information sources simultaneously So each will generate its own long sequence Shannon noticed that something interesting happens if we make the number of selections very large Can you see that the percentage of red blue black and white begins to converge to its expected value 50% red 25% blue and so on This happens for all of the information sources we simulated Even though the sequences produced by our sources are all different they belong to the same typical set And as the number of selections approaches infinity this probable set is all we are left with A small set from an astronomically large possible number of sequences And the number of bits required to specify this probable set It is the entropy per symbol of the information source Hartley had taught us that information is the log of the number of choices Shannon had taken the sequence to infinity and showed the true number of choices is specified by the entropy And Shannon said the same result applies for any source So even if we simulate a complex source like a Markov chain it will also eventually boil down to its own set of typical sequences and it too can be specified by its entropy In the limit as we approach infinity Shannon had proved that the information source boils down to its entropy the essential essence of the source any fewer than h bit bits per symbol and we cannot guarantee that every probable sequence can be encoded Now Shannon’s proof relies on taking a number of sequences which approaches infinity but it shows the limit of compression You cannot compress any better than the entropy Shannon had introduced a new method of quantifying information entropy and he showed that any information source can be compressed down to its entropy This allows communication with the minimum number of bits per symbol and the most efficient use of the channel capacity But there’s a problem Every practical channel is noisy and it means that there can never be a guarantee that the bits you receive are the bits that were sent What Shannon did next would be the most audacious part of his paper and its implications would reverberate decades into the future
Chapter 6: Noisy Channels, Conditional Entropy, and Capacity limits (33:23 – 41:15)
Imagine I flip a fair coin heads or tails When I tell you the result you gain one bit of information But now let’s say I lie 10% of the time So if I flip a heads I’ll tell you its tails And if I flip a tails I’ll tell you it’s a heads but only 10% of the time How much information am I now conveying you might be tempted to say well if 90% of the time you’re telling the truth maybe I’m still getting.9 bits of information 9 bits out of 10 But that’s not right because you don’t know when I’m lying That uncertainty is actually baked into every message How did Shannon account for this and adjust the rate accordingly consider a communication system with an observer capable of seeing both what is sent and what is received The observer notes the errors and then communicates them over its own channel a correction channel The easiest way to do this is to send a stream of bits with one indicating the position of an error and zero otherwise But these bits could occur anywhere All we know is that the frequency of ones for a long message will be 10% or whatever the bit error rate happens to be Shannon modeled this as a communication channel in its own right And he realized that the capacity of the correction channel coded down to it entropy is the missing information at the receiver due to the additional uncertainty of the noise He called this conditional entropy It is the average ambiguity of the received signal In our example of the coin the entropy of the information source starts as one bit per flip but the uncertainty due to the noise is47 bits per flip Therefore the rate of information is actually 0.53 bits per flip Shannon’s hypothetical correction channel shows how the conditional entropy represents the missing information due to the noise As an example imagine being at a loud party and your friend says a word and let’s say that word is selected from 2 to the^ of 10 possible words So 10 bits of information is contained within that message But because of the noise you can only narrow it down to one of eight possible words That means there’s still three bits of uncertainty But the information you gained is the narrowing of 2 to the 10 possibilities down to 2 to the 3 possibilities So seven bits of information has been transmitted
When noise is present there is an inherent uncertainty at the receiver and this means there could be multiple possible messages for each received sequence This is not practical for communication over noisy channels because the receiver can never be sure about what the transmitted message actually was If we think back to Chapa Semaphore imagine the problems they might face if the weather is foggy it becomes difficult to see what symbol is actually being sent But you might be able to reasonably narrow it down to a limited set of possibilities In such a case you might be tempted to limit the rate of transfer by restricting the number of symbols that can be sent effectively separating the symbols according to the ambiguity One might consider the capacity of this channel to be the maximum possible rate given the uncertainty of the fog But the system needs to be designed properly to reach this capacity while still avoiding errors Shannon was now ready to make his most audacious claim yet He said it was possible with the right coding to transmit at the capacity of the channel with close to zero errors Imagine a source X producing information at the rate of entropy H of X Let’s assume this source has reached the channel capacity Over time this source produces around 2 ^ of h * t probable sequences As Shannon showed if a source keeps generating sequences eventually you end up with just a set of typical sequences and the number of them grows exponentially at a rate defined by the entropy If we now allocate these sequences to the channel the number of possible channel output sequences also grows roughly at the exponential rate The trick as we’ve seen is to assign messages to channel inputs using a code book mapping each message to a channel sequence But here’s the problem In a noisy channel we’re guaranteed to have more than one plausible input that could produce a given output that is more than one reasonable cause for what the receiver sees So Shannon tried something different He reduced the rate of the source to R where R is now less than the channel capacity Now because the source is generating information more slowly it produces fewer sequences compared to the number of available channel output sequences It’s interesting to see what happens when we simulate the rate of growth of sequences of R and the channel sequences Even with a small reduction given enough time the ratio becomes vanishingly small which means there are now a huge number of channel sequences available to encode messages And this means the channel has more than enough room to spread the messages out to keep them far apart Shannon then assigned messages at random to channel sequences and he realized even though the channel is noisy the chance that a received sequence could be reasonably interpreted as more than one message drops to close to zero given a long enough block length
Shannon’s final trick was to extend the concept of entropy to continuous signals and in doing so he provided a way to calculate the capacity of continuous channels This culminated in the Hartley Shannon theorem a unifying expression that brought together the insights of Nyquist Hartley and Shannon himself The result was a single equation that defines the capacity of a channel constrained by power and corrupted by noise The ultimate speed limit for reliable communication
This single formula changed everything, turning information from a vague concept into a hard, measurable reality. It didn’t just explain how to send messages; it laid the foundation for the entire digital world we live in today. From the code running on our computers to the data streaming across the internet, it all traces back to that quiet revolution that began at Bell Labs.
The integration of the three(trinity) distinct eras through the Shannon-Hartley theorem:
$$C = B \log_2 \left(1 + \frac{S}{N}\right)$$
Where:
- $C$ is the ultimate channel capacity in bits per second (Shannon’s metric).
- $B$ is the channel bandwidth in Hertz (Nyquist’s metric).
- $S/N$ is the signal-to-noise power ratio over the transmission line (Hartley’s geometric bounds modified for stochastic noise).
Leave a Reply