Imagine trying to fit the entire Library of Congress
into a single standard size shoebox. Right. I
mean, you literally just can't do it. You can't.
You have to start ripping out pages or like shredding
the covers, printing the text microscopically.
And eventually the structural integrity of the
box just explodes. Exactly. Just gives up. Because
you don't magically change the physics of the
cardboard, right? You're forced to build an entirely
different kind of container. And back in the
1960s, software engineers essentially tried to
do exactly this with human language. Yeah, they
really did. And the architectural fallout from
that decision is something we are literally still
dealing with today. It's basically the ultimate
story of architectural growing pains. You know,
when you build a foundation for a small house
and then suddenly decide that that house actually
needs to be a 100 -story skyscraper. The workarounds
get messy. Incredibly complex, yeah. So, welcome
to today's Deep Dive. Our mission for you today
is pretty simple. We are going to explore the
hidden, often messy architectural choices that
allow your computer to speak every language on
Earth. And it's quite the journey. Oh, it is.
We're going from the cramped 8 -bit data limits
of the 1960s to the vast, flexible systems running
today's global software. And if we connect this
to the bigger picture, the stakes for you are
remarkably high. Every single time you read an
international news site or, you know, text a
friend in another language. Or even just run
a piece of code. Right. You are relying entirely
on the evolution of what we call the wide character.
Because without it, global digital communication
literally breaks down into unreadable gibberish.
Which nobody wants. So let's set a baseline here.
A wide character is at its core, A computer character
data type that has a size greater than the traditional
8 -bit character. It's physically larger in memory.
Exactly. It was engineered specifically to accommodate
larger coded character sets. But to really grasp
why that leap to wider characters was so revolutionary,
we have to... Look at the tiny boxes computers
originally forced language into. We have to go
back to the 1960s. Yes. OK, let's unpack this.
Take us back. So in the 60s, mainframe and many
computer manufacturers were all just trying to
establish some common ground because before this
point, different computers used completely different
sizes for their basic data chunks. Like it was
the Wild West. It really was. Some used six bits,
some used nine. But the industry, heavily influenced
by systems like the IBM System 360, began to
standardize around the 8 -bit byte. And that
became the standard smallest fundamental data
type, the building block. Exactly. That was the
foundational block for storing information. But
they weren't using all eight of those bits for
the actual letters. Right. Right. Like this goes
back to the 7 -bit ASCII standard. Right. The
7 -bit ASCII standard. Which was the industry
standard for encoding alphanumeric characters,
mostly for those old teletype machines and early
super clunky computer terminals. Right. And seven
bits gave them exactly 128 possible combinations.
Which is not a lot. It's really not. It was just
enough to encode the uppercase and lowercase
English alphabet, numbers 0 through 9, basic
punctuation, and a few control characters. Like
what, carriage return? Yeah, carriage return
to tell the printer to physically go to the next
line. Okay, so seven bits for the text, which
leaves exactly one bit left over in our 8 -bit
box. That eighth bit. A famous eighth bit. Historically,
this was used as a parody bit. which I like to
think of it as the digital equivalent of packing
peanuts. Packing peanuts, okay, I like that.
Yeah, because you're shipping a delicate package.
which is your data, and you use that extra space
in the box entirely for safety, just to make
sure the package arrives intact. That's a great
way to look at it. To take that analogy a step
further into the actual mechanism, here is how
those packing peanuts actually worked. In the
early days of computing, hardware was incredibly
noisy and unreliable. Electromagnetic interference
could literally flip a zero to a one while data
was moving through a wire. Oh, wow. Just randomly
in transit. Static on the line, yeah. So the
parity bit was a clever math trick to catch that.
If you had seven bits of data, the computer would
count how many ones were in that data. Right.
If the number of ones was an even number, it
would set the eighth bit to a zero. If it was
odd, it set the eighth bit So it's forcing the
math. The total number of 1s in the entire 8
-bit byte was always an even number. Precisely.
It always added up to even. So when the receiving
computer got the data, it counted the 1s. If
it counted an odd number, it immediately knew,
hey, a bit got flipped in transit. This data
is corrupted. Say it again. Exactly. It was a
brilliantly simple error detection mechanism.
But as computing technology improved, the hardware
got much more reliable, right? The cables got
better, the weird interference dropped. Yeah,
considerably, yeah. And simultaneously, computers
started to go global. Manufacturers looked at
their 8 -bit box and realized, well, the English
alphabet isn't going to cut it in Europe or Asia.
No, they needed much more room. So they dumped
the packing peanuts. They ditched the parity
bit entirely. And by freeing up that eighth bit,
they doubled the capacity of the byte from 128
combinations to 256. Just by removing the safety
material. Exactly. This gave us the famous 8
-bit extensions that became commonplace in the
70s and 80s. Things like IBM code page 37 or
PET -C for com - machines and the ISO 80 -89
standards. Suddenly those early terminals had
support for Greek, Cyrillic, Hebrew, and a bunch
of other regional alphabets. Okay, so you've
doubled the capacity of your box, but here's
the driving question for this whole deep dive.
If we freed up the eighth bit and we got all
these new alphabets, why wasn't that enough?
Well, because the math still maxes out at 256
combinations. Right. The fatal flaw of 8 -bit
extensions is that they were entirely region
-specific. So you had one lookup table for Greek,
a completely different lookup table for Cyrillic,
and another one for Arabic. But they all shared
the exact same limited numeric space. Exactly.
The number 200 might mean a Greek omega on one
computer, but on a Russian computer, that exact
same number 200 meant a Cyrillic letter. Which
leads to what we call destructive translation,
or what's affectionately known in the industry
as Mojibake. MojiBake, yes, it's a great word
for a terrible problem. MojiBake is that phenomenon
where you open a text file or an email and instead
of readable text, it's a completely random string
of wingdings, question marks, and accented gibberish.
Because the underlying binary data hasn't changed,
like the sending computer sent the number 200,
but the receiving computer is using the wrong
regional lookup table. Right, it's blindly translating
that number into whatever symbol happens to sit
at slot 200 in its local directory. And if you
try to actively convert data into a target set
that didn't even have a slot for the character
you were using. The system would just replace
it with a generic question mark. And the original
data was mathematically destroyed. You could
never get it back. It was a wildly unsustainable
way to build a global communication network.
I mean, you couldn't even have a single document
with both Greek and Russian characters without
writing incredibly complex, fragile, special
conversion routines. Just telling the computer
to constantly swap its lookup tables mid -sentence.
Which is a nightmare. Which brings us to 1989.
The International Organization for Standardization,
or ISO, begins work on the Universal Character
Set. or UCS. The dream of one massive multilingual
character set to rule them all. No more swapping
tables, right? No more swapping. Every single
character from every single language gets its
own permanent unique mathematical number. But
to achieve that... UCS required encoding values
using either 16 -bit, which is 2 bytes, or 32
-bit, which is 4 bytes. Right. And the 8 -bit
box finally shatters. It just physically cannot
hold values that large. So to store these new
massive character values in active memory, you
needed a data type fundamentally larger than
8 bits. And thus, the term wide character was
born, specifically to differentiate these new
expansive data types from the traditional 8 -bit
ones. OK. So we made the data type bigger. to
fit the world's alphabets. We have a 16 -bit
or 32 -bit wide character problem solved, right?
Well, not exactly, because here's where it gets
really interesting. Solving the storage problem
accidentally highlighted a massive transmission
problem. Yeah, it really did. To understand this,
we need to clarify a crucial architectural distinction.
When we say wide character, we are strictly talking
about the size of the data type. in the computer's
memory, the RAM. A Y character does not state
how each value in a character set is actually
defined. Right, the definition of the values
is the job of the character sets themselves,
like UCS or Unicode. The wide character is just
the blank physical container sitting in memory,
waiting to hold those massive definitions. Exactly.
So your wide character is sitting comfortably
in your computer's active memory, taking up 16
or 32 bits of space, representing a complex kanji
character or a modern emoji. Wait, I'm going
to push back here. Yeah. Because if a 32 -bit
wide character solves all the space issues and
prevents destructive translation, why didn't
the entire computing industry just agree to upgrade
the pipes? What do you mean? Like, why not just
make the cables, the routers, and the internet
protocols 32 -bit across the board so we can
just send these wide characters natively? Uh,
because you are talking about replacing... trillions
of dollars of global infrastructure. By the time
wide characters were invented, the world had
already spent decades laying undersea cables,
launching satellites, and writing network protocols
that were fundamentally physically hardwired
to process data in 8 -bit chunks. You can't just
flip a software switch to fix that. No, you can't
make an 8 -bit physical router suddenly swallow
a 32 -bit object. If you try to shove a 32 -bit
chunk of data down a pipe strictly designed for
8 -bit symbols, the hardware gets confused, it
misreads the boundaries of the data, and the
transmission completely fails. So we have massive
characters in memory, but tiny pipes for the
internet. How do we get the data from point A
to point B? The engineering workaround for this
transmission bottleneck is the multi -byte character
encoding. And the most famous example of this
today is UTF -8. Ah, multi -byte. Because we
lack those wide data paths, multibyte encoding
systems use multiple 8 -bit bytes in a row to
encode a value that is simply too large for a
single 8 -bit symbol. Right. They break it down
for transit. And the mechanism behind it is brilliant.
Right. Like in a multi -byte system like UTF
-8, the very first bits of the byte act as a
signal to the receiving computer, right? Yes.
If the byte starts with a zero, the computer
knows, OK, this is a standard old school 8 -bit
character. I can read it immediately. But if
the byte starts with a specific sequence, like
110, the computer knows, wait, this is incomplete.
This is part of a larger character. I need to
grab the next byte. Exactly. I need to grab the
next byte and read them together to figure out
the math. You know, the C programming standard
actually officially splits these two concepts
up, which I find incredibly helpful for visualizing
this. Oh, the distinction between multibyte and
Y. Yeah. According to the C standard, multibyte
encodings. those variable length chains of 8
-bit chunks, are primarily used in source code,
external files, and network transmission. Whereas
wide characters, the fixed massive 16 or 32 -bit
containers, are the runtime representations of
characters sitting in single objects. They exist
solely in the active volatile memory of the system.
Right. It's basically like buying IKEA furniture.
IKEA? Yeah, hear me out. MultiByte is the flat
pack you shove in your car for transit. Okay.
And the wide character is the fully assembled
bookshelf seating in your living room's active
memory. OK, that analogy actually holds up perfectly.
The assembly process is the runtime conversion
your processor performs. Exactly. But here is
where the history of tech giants makes things
really complicated. Because different operating
systems decided to build their living rooms and
assemble those booktiles at very different times
in computing history. Right. And it created completely
different architectural philosophies that still
clash today. The great schism. The great schism
indeed. Let's look at the early adopters. Systems
like Microsoft Windows, the .NET framework, and
the Java programming language. The heavy hitters.
Right. In the early 1990s, they were eager to
solve the internationalization problem. They
wanted to be global immediately. So they jumped
on the very early version of the universal character
set, specifically a standard called UCS2, which
essentially became Unicode 1 .0. And UCS2 was
a strict 16 -bit system. Right. It offered 65
,536 possible characters. And at the time, engineers
genuinely believed that was more than enough
space to encode every single living human language
with room to spare. Oh, the hubris. I know. So
Windows and Java lock their foundational architecture
into a 16 -bit wide character. In their systems,
the default wide character type -like raw chart
in C++ on Windows, or char in Java, was hard
-coded to be exactly 16 -bit. Which seemed like
plenty of space, but then we get the historical
plot twist. Unicode didn't just stay with living
languages. No, it grew immensely. Right, we get
the 1996 update, Unicode 2 .0, and suddenly they
are adding dead historic scripts, complex mathematical
symbols, and eventually the thousands of emojis
we use today. The full range of human expression
blew way past the 65 ,000 character limit. It
expanded to require 21 bits of space. So what
happens to Windows and Java? They are already
locked into an architecture that physically only
has 16 bits of space per character. They get
stuck in what we can call the 16 -bit trap because
they can't easily tear out the foundational architecture
of their entire operating system without breaking
millions of legacy programs. It would be catastrophic.
It would. So these systems now have to rely on
a complex workaround called surrogate pairs.
Okay, break down how a surrogate pair actually
functions under the hood because it sounds messy.
Oh, it is. Remember how we had 65 ,000 possible
combinations in the 16 -bit space? Yeah. Engineers
went in and permanently reserved a specific block
of those numbers. They explicitly said, these
numbers no longer represent actual printable
characters. Instead, they act as warning flags.
When the Windows operating system reads a 16
-bit wide character and sees that it falls into
this reserved high surrogate block, the hardware
knows it cannot print a letter yet. It has to
wait. Yes. It must hold that data in suspension,
grab the next 16 -bit wide character, combine
the mathematical values of both, and then look
up the resulting massive character. So they are
essentially duct -taping two 16 -bit boxes together
just to store one single modern Unicode character.
Like a smiling emoji. Basically, yeah. It's a
massive workaround. It gets very messy because
it breaks a fundamental programming assumption.
Their wide character data types no longer map
one -to -one with actual printed characters.
No, not at all. One printed letter might be one
wide character in memory, or it might be two.
Which forces programmers to write highly defensive
complex code just to count how many letters are
in a word. Exactly. Meanwhile, you have the Unix
-like systems Linux, Mac OS, sitting across the
aisle. What was their strategy? Well, they took
a wait -and -see approach. They did. Unix -like
systems generally waited until the dust settled
on the Unicode expansion. When they finally standardized,
they adopted a massive 32 -bit write chart as
prescribed by the C90 standard. Why? By going
straight to 32 bits, they gave themselves enough
room to comfortably fit the entire modern 21
-bit Unicode code point into a single solitary
wide character container. So no surrogate pair
is required. Every wide character is exactly
one printed symbol. Precisely. So if we look
at the modern landscape today, for you the listener
navigating these systems, you're dealing with
two very different preferences born entirely
from this history. Operating systems heavily
influenced by Unicode 1 .0, like Windows, tend
to prefer using wide strings made up of these
16 -bit character units. Yes, whereas Unix -like
systems, despite having that massive 32 -bit
wide character available to them, actually tend
to retain the old 8 -bit narrow string convention
for handling text. Wait, really? Even in memory?
Yeah. Because UTF -8 multi -byte encoding became
so efficient and universally adopted on the internet,
Unix systems prefer to just keep the text in
that variable multi -byte format, even in memory.
Going back to the analogy, they prefer to keep
the furniture flat packed for as long as possible,
only assembling it into a 32 -bit -wide character
at the exact millisecond they need to manipulate
it. That's a great way to put it. The historical
circumstances of when an operating system was
built absolutely dictate what types of encoding
they prefer to process today. Which raises a
really important question. How do modern programmers
actually deal with this historical baggage? If
the size of a wide character changes from 16
bits on Windows to 32 bits on Mac, how do you
write a piece of software that works everywhere?
This is where we look at the language lottery.
How different programming languages have actively
tried to clean up this mess over the decades.
And it really starts with the absolute wild west
of C and C++ prep. Oh man. Listen to how the
original C90 standard defined the wide character
data type. Rotschart, I have it right here. Go
for it. It called it an integral type whose range
of values can represent distinct codes for all
members of the largest extended character set
specified among the supported locales. Yeah,
that is essentially a legal loophole masquerading
as computer science. It really is. They basically
defined it as make it as big as it needs to be
for whatever system you happen to be running
on. It was entirely implementation defined. Which
creates a total nightmare for portability. If
you write a program on a Mac expecting your wide
character to hold 32 bits of data, and someone
compiles that exact same code on an old Windows
machine, where the compiler defines it as 16
bits... Your program crashes. Right. It literally
attempts to stuff 32 bits of math into a 16 -bit
physical box, causing memory overflows. The ambiguity
was so dangerous that the ISO -IE Unicode standard
itself had to issue a staggering warning. What
did it say? It explicitly stated... The width
of rawchart is compiler specific and can be as
small as eight bits. Programs that need to be
portable across any C or C++ compiler should
not use rawchart for storing Unicode text. Wow.
The global standard itself is telling programmers,
do not use the wide character type to store global
text if you want your code to survive. Yeah,
it was a massive red flag. So how did C and C++
finally fix it? Well, it took until their 2011
revisions. Both C and C++ finally introduced
fixed -size character types to the language.
They explicitly created char16 for guaranteed
16 -bit storage and char32 for guaranteed 32
-bit storage. Which finally provided unambiguous
representations. They left the old write chart
in the language as a legacy artifact so old programs
wouldn't break, but they gave modern programmers
new, precise physical dimensions to work with.
OK, so that's C and C++. plus trying to patch
the leaks. Let's look at Python's evolution,
because I think it perfectly illustrates a massive
realization in the tech industry regarding the
wide character. Python's journey is fascinating
here. It is. If you go back to Python 2 .7, the
language relied heavily on whatever the operating
system dictated. Its character type was tied
to the underlying C compiler's rod chart. Still
shackled to that unpredictable underlying system.
Right. But then by PyCon 3 .3, they realized
this was inefficient. They introduced a flexibly
sized storage system for strings. The language
would dynamically look at a string of text, figure
out the largest character in it, and then allocate
memory based on that. Smart. But here's the real
aha moment. As of Python 3 .92, they dropped
the use of write chart for Python strings entirely.
It's a complete paradigm shift. It really is.
They realize that forcing text into these massive
32 bit wide character arrays and memory is actually
a massive waste of RAM. Oh, totally. Most text
on the Internet is still in the standard ASCII
range. English letters, basic numbers, which
only requires one byte of storage. Right. So
if you force an entire English paragraph into
a 32 -bit wide character array, you are inflating
your memory usage by 400 % with empty zero -filled
bits. Just waste space. Exactly. So Python decided
to lean entirely into multi -byte flexibility.
Now, they just store the text as UTF -8 multiply
strings and cache it. Modern CPUs are so incredibly
fast at decoding UTF -8 on the fly that keeping
a massive white character in memory is just...
obsolete. They move from store it wide so we
can access it fast to store it narrow because
our processors are finally fast enough to decode
it instantly. Yeah. And to add to that on the
complete opposite end of the spectrum you have
modern language like Rust. Ooh, Rust took an
intentional uncompromising rebellion against
the decades of pain caused by C plus ambiguity.
Absolutely. In Rust, a char data type is exactly
32 bits, and it represents a valid Unicode scalar
value? Period. No guessing. No compiler -specific
baggage. No surrogate pairs. None. The designers
of Rust looked at the historical mess of 16 -bit
traps, variable -size row chart types, and the
constant fear of memory overflows, and they established
an absolute physical dimension. 32 bits. It solves
the portability issue immediately by refusing
to compromise. Right. It's like they just built
a living room so massive that you never have
to flat pack the furniture ever again. Which
is wildly inefficient for memory space, but it
is incredibly safe for the programmer. Which
showcases the eternal trade off in computer science.
You know, you can optimize for memory space like
Python or you can optimize for unshakable stability
like Rust. So what does this all mean? When we
step back and look at the whole picture, a wide
character isn't just a dry technical specification.
No, it's really not. It is a living artifact.
It is the fossil record of computers painfully
learning to accommodate the sheer, messy volume
of human language. We started in these cramped,
8 -bit cardboard boxes, tried to squeeze extra
alphabets into the packing peanuts, broke the
box entirely with destructive translation— Shamoji
bake everywhere. Exactly. And eventually, we
had to engineer these complex, 32 -bit global
memory systems just to say hello in every language.
The real takeaway for you listening to this is
to appreciate the staggering, invisible labor
your devices perform every single day. It's mind
blowing. It is. Whenever you type a character
that isn't standard English, there is a massive
mathematical translation effort happening under
the hood. Your system is constantly switching
between massive memory representations and narrow,
multi -byte transmission streams. Juggling surrogate
pairs and historical architectures. Counting
ones and zeros just to render that single text
message accurately. on your screen. It is a minor
miracle of engineering every single time you
send an emoji, which leaves me with one final
thought to ponder today. If the entire history
of the wide character is essentially a story
of brilliant engineers chronically underestimating
exactly how much digital space human expression
requires, what future forms of human communication
are we currently underestimating the storage
for right now? That is a profound question. I
mean, spatial computing, neural interfaces, the
data requirements will be unimaginable. We'll
leave you to think on that. We've gone from the
8 -bit box to a 32 -bit world. But human expression
never really stops expanding, does it? Keep diving
deep.