A Gentle Introduction to Information Theory

Information theory is the science of measuring, storing, and reliably transmitting information — and it began with a single 1948 paper by Claude Shannon. Its ideas quietly power everything from your phone signal to the error correction inside a QR code.

Before 1948, "information" was a fuzzy, everyday word. Then a Bell Labs engineer named Claude Shannon published a paper that gave it a precise mathematical meaning and, almost overnight, founded a whole field. This gentle tour covers his big ideas — the bit, entropy, channel capacity, noise, and redundancy — and shows how they lead straight to why a damaged QR code still scans. 🐾

Who founded information theory?

Claude Shannon founded information theory with his 1948 paper "A Mathematical Theory of Communication." In it he did something remarkable: he separated the meaning of a message from the quantity of information it carries, and showed that the quantity could be measured, predicted, and engineered.

That separation was the key that unlocked the digital age. By treating information as a measurable resource — like energy or mass — Shannon made it possible to reason precisely about how much can be squeezed down a wire, stored on a disc, or printed into a little square of black and white.

What is a bit?

Shannon gave us the bit as the fundamental unit of information. One bit is the amount of information in a single choice between two equally likely outcomes — one flip of a fair coin, one yes-or-no answer resolved.

This is a subtle, powerful idea: information is measured by how much uncertainty it removes. Learning the answer to a 50/50 question gives you exactly one bit. The bit is now the universal currency of the digital world, and it's the same bit that appears as a dark or light square in a QR code — see binary, bits and bytes for how bits build up into everything else.

What is entropy?

Entropy, in Shannon's sense, is a measure of uncertainty — how unpredictable a message is on average. The more uncertain or surprising the outcomes, the higher the entropy, and the more information each message carries.

An intuitive example: if a friend always texts "on my way," those texts carry almost no information because you can predict them. But if their texts are unpredictable, each one tells you something genuinely new — higher entropy. Entropy sets a hard floor on how much you can compress data: you can't squeeze a message below its entropy without losing information.

Entropy measures surprise. A message you could have guessed carries little information; a message you couldn't carries a lot. That's the whole reason predictable text compresses so well.

What is a channel and what is noise?

Shannon modeled communication as sending a message through a channel — any medium that carries information from sender to receiver. A phone line, a radio link, a fiber cable, and even a printed QR code being read by a camera are all channels.

The trouble is that real channels are noisy. Noise is anything that corrupts the signal along the way: static on a line, interference on the air, a smudge on a printed label. Noise is why messages arrive garbled, and dealing with it is one of the central problems Shannon set out to solve.

What is channel capacity?

One of Shannon's deepest results is channel capacity: every channel has a maximum rate at which information can be sent through it reliably, no matter how clever your coding. Push data faster than capacity and errors become unavoidable; stay under it and — astonishingly — you can drive the error rate as low as you like.

This was a stunning claim at the time. It said that noise doesn't force you to accept mistakes; it only limits your speed. Below capacity, near-perfect communication over a noisy channel is possible. The catch — and the beautiful part — is how you achieve it: with redundancy.

How does redundancy enable reliability?

Here is Shannon's most practical gift. He showed that by adding carefully structured redundancy — extra information beyond the bare message — you can protect data against noise and recover it even after corruption. This is the theoretical bedrock beneath all error-correcting codes.

Redundancy sounds wasteful, but it's the price of reliability, and Shannon proved the price is worth paying. A little extra, chosen well, lets a receiver detect and even repair errors without ever asking for a resend. The practical schemes that deliver on this promise — from Hamming codes to Reed–Solomon codes — are engineering answers to Shannon's theoretical question.

How does information theory relate to QR codes?

A QR code is a tiny, printed embodiment of Shannon's ideas. The information is measured in bits — visible as dark and light modules. The "channel" is the code sitting in the world and a camera reading it, and the noise is everything that can go wrong: scratches, stains, poor lighting, a logo dropped in the middle.

To survive that noise, QR codes carry redundancy in exactly the way Shannon prescribed. Extra error-correction data is woven into the grid so the original message can be rebuilt even when part of the code is lost. The result is that a QR code can lose a chunk of itself and still decode — a direct, everyday demonstration of reliable communication over a noisy channel. The mechanics are laid out in QR code error correction.

Shannon's concept	In a QR code
Bit	A dark or light module
Channel	The printed code and the scanning camera
Noise	Scratches, smudges, glare, a logo
Redundancy	Error-correction codewords in the grid

Why does information theory still matter?

Shannon's framework didn't just explain the technology of his day — it set the ground rules for all of it since. Every time you stream a video that buffers instead of freezing, place a call that stays clear as bars fade, or scan a code in dim light, you're benefiting from engineers working within the limits he defined. The bit, entropy, and channel capacity are not abstractions gathering dust; they're the measuring sticks by which modems, storage, and codes are still designed and judged. Understanding them gives you a lens on the whole digital world: information is a measurable resource, noise is the enemy, and clever redundancy is the reliable way to win.

Information theory in one sentence

Shannon's information theory measures information in bits, quantifies uncertainty as entropy, sets a reliable speed limit called channel capacity, and shows that redundancy tames noise — the exact recipe that lets a scuffed QR code still deliver its message.

Want to hold a piece of information theory in your hand? You can create a free QR code with QR Puppy and see bits, channels, and redundancy at work.

Frequently asked questions

Who invented information theory and when?

Claude Shannon founded it in his 1948 paper "A Mathematical Theory of Communication." That single work introduced the bit, entropy, and channel capacity, and effectively created the field in one stroke.

What is entropy in simple terms?

It's a measure of uncertainty or surprise in a message. Highly predictable messages have low entropy and carry little information, while unpredictable ones have high entropy and carry more — which is also why predictable data compresses well.

What is channel capacity?

The maximum rate at which information can be sent through a channel reliably. Stay below it and you can make errors as rare as you like with good coding; exceed it and errors become unavoidable no matter how clever you are.

How does redundancy help against noise?

Adding structured extra information lets a receiver detect and repair corruption without a resend. Shannon proved this is what makes reliable communication over noisy channels possible, and it's the basis of every error-correcting code.

What does information theory have to do with QR codes?

A QR code is Shannon's model made physical: bits are modules, the channel is the code plus a camera, noise is damage, and redundancy is the error-correction data. That's why a partly damaged code still scans.