So welcome to the deep dive. We're going to try
a little thought experiment to start us off today.
I want you to imagine it's the early 1990s. A
very different computing landscape back then.
Oh, completely. You've got your computer, and
the hard drive is, I mean, it's a mere 20 to
80 megabytes, which is nothing today. And expanding
it costs a staggering $10 per megabyte. Yeah,
which was a huge barrier for, well, Pretty much
everyone. Exactly. So what do you do? Do you
empty your wallet for a new drive or do you turn
to this almost magical software wizardry from
the 90s known as disk compression? Because that's
our mission today. We are exploring this technology
that seemingly just double a computer's physical
storage out of thin air. It really did feel like
magic to users back then. I mean, you have to
remember software was rapidly getting bigger
and physical storage was incredibly expensive.
Right. OK, let's unpack this. Because the idea
of just downloading space sounds impossible.
I always picture it like a magical suitcase.
You have the suitcase, and the exact second you
pack your clothes into it, it automatically shrinks
them down to half their size. That's a great
way to visualize it. Right. And then the second
you pull a shirt out, it instantly expands back
to its normal size. And the wild part is you
never even have to press a compress button. It
just happens. And that automatic nature is what
grounds your analogy. in the actual technical
reality of our sources, because we really need
to establish the difference between standard
file compression and true disk compression. Which
are two very different beasts. Right, completely.
With file compression, like zipping a file, it's
interactive. You manually tell the system to
pack it up, and you manually tell to extract.
But disk compression was totally non -interactive.
It operated entirely in the background. It was
on the fly, completely transparent. and happened
in real time. So the user, and I guess the software
they were running, had no idea this was even
happening. Exactly. Your word processor, your
games, they didn't know they were asking for
compressed data. They just asked the system for
a file. And the disk compression utility would
grab it, decompress it instantly, and hand it
over. OK, so how did it actually pull off? this
illusion because you're basically creating physical
space using only software. Well, it did this
by essentially hijacking the file access layer
of the operating system. When you installed one
of these utilities, instead of saving thousands
of individual files scattered all over the physical
magnetic platters of your hard drive. Which is
how it normally works. Right. Instead of that,
the software created a single massive hidden
file on the host drive. They usually called it
a virtual drive. So every single thing you saved
was secretly being compressed and stuffed directly
into this one giant monolithic file by a special
device driver. Wait, if everything is living
inside one giant hidden file, doesn't that mean
the computer is basically lying to the user about
what a drive even is? What's fascinating here
is that yes, the computer was entirely abstracting
reality. It was a complete deception. Yeah, the
setup process was wild. The utility would first
create this empty virtual drive, which again,
is just one massive hidden file. And then it
would systematically transfer almost all of your
files from the physical host drive into that
compressed file. Wait, you said almost all. Did
it leave some stuff behind? It had to. You couldn't
compress vital system files. specifically the
swap file. Because if you compress the swap file,
which manages virtual memory, the computer would
basically lock up trying to decompress its own
working memory to function. So it leaves that
alone, moves everything else into the hidden
file, and then it simply inflates the reported
size of that virtual drive. So it just tells
the computer, hey, look at all this free space
I suddenly have. Exactly. It reports the expanded
theoretical size, leaving the operating system
totally oblivious to the real physical limits.
But, I mean, if the entire C drive is secretly
just a hidden file on the physical drive, how
does the computer even wake up? Like, when you
turn a PC on, it needs to read the C drive to
boot, but if the drive is fake and the driver
needed to read it is locked inside the fake drive,
that's... Well, it's a total chicken and egg
problem. It is a massive paradox. And solving
it required some incredible sleight of hand during
the boot sequence. Let's look at the historical
DOS boot process from the sources to see how
they trick the system. OK, lay it on me. So you
press the power button. The system BIOS wakes
up and it looks at the very first sector of the
hard drive, the master boot record, or MBR. Right,
the absolute baseline hardware level. The MBR
then loads the volume boot record. which in turn
loads the DOS BIOS. These are hidden system files,
usually iOSYS or IBMBIO .com. At this stage,
DOS is loading into basic memory, but, and this
is the crucial part, it hasn't loaded the config
-dos -sys file yet. Meaning no third -party drivers
are loaded. The computer still thinks the physical
hardware is the real C -DRAF. Exactly. And if
it kept booting normally, it would just crash,
because all the user files and the rest of the
OS are trapped inside that unreadable compressed
monolith. So how did they bypass the crash? In
MS -DOS 6 .0, Microsoft introduced an undocumented
challenge response interface known as the Preload
API. Here's where it gets really interesting,
because I picture this like intercepting the
computer's morning coffee. The machine is just
waking up going through its groggy morning routine,
and this Preload API just steps in front of it.
That's a perfect way to put it. Right after the
DOS BIOS loads, but literally milliseconds before
it reads the user's config files, the system
pauses. It scans the root directory for specific
preload drivers, files like dblspace .bin or
stacker .bin. So compression drivers? Right.
And if it finds one, it initiates this secret
handshake. If the driver gives the correct mathematical
response, the system loads it into memory immediately,
completely bypassing normal configuration. And
what does the driver do once it's in? It immediately
executes a radical maneuver. It literally swaps
the drive letters in the system's RAM. It swaps
them. So it puts on a mask and says, hey, look
at me. I am the C drive now. Yes. The physical
drive is shoved aside, usually reassigned to
a random letter like H, and the fake compressed
file is mounted and declared the C drive. And
from that millisecond on, the computer finishes
its morning coffee, completely oblivious that
its entire reality was just remapped. That is
just wild. But I mean, tricking the computer's
brain and stuffing literally all your data into
a single hidden file sounds incredibly risky.
Like, what were the unintended consequences of
doing this? Because this has to have a catch.
Oh, there were several massive catches. But before
we even get to the data disasters, we have to
talk about the performance paradox. OK. You'd
naturally assume that forcing a computer to do
complex math to compress and decompress every
single file in real time would make the whole
system sluggish. Right, yeah. I mean, you're
giving the CPU a massive extra chore for every
single task. The sources show it actually depended
on where your bottleneck was. If your system
was CPU -bound, meaning the processor was already
struggling to keep up with basic tasks, then
yes, disk compression slowed everything to a
crawl. The math was just too heavy. So it was
just choking the processor? Right. But if your
system was IO -bound, meaning you had a fast
processor, but a really slow physical hard drive.
Which was pretty common back then, waiting for
the physical platters to spin. Exactly. For those
IO -bound machines, disk compression actually
increased overall performance. Wait, really?
It made the computer faster? How? Because there
was physically less data to read and write from
the slow mechanical drive, the fast processor
could read a tiny compressed file from the disk
and decompress it in RAM significantly faster
than the slow hard drive needle could read a
massive uncompressed file. Oh, wow. So the math
was actually faster than the physics of the spinning
disk. That's brilliant. But obviously, performance
aside, the risks to your actual data had to be
astronomical. They were. The nightmare scenarios
usually began with a very simple user action,
trying to uninstall the software. Because it's
a trap. A total trap. If your expanded uncompressed
data exceeded the physical size of the actual
hard drive, uninstallation was mathematically
impossible unless you deleted enough files to
fit back into reality. You literally couldn't
escape the virtual space you built. Exactly.
And it gets worse. A massive danger lurk during
the initial setup phase. The compression utility
relied completely on the integrity of your drive's
FAT file system to move your data safely. The
file allocation table, basically the index card
catalog for where data lives on the disk. Right.
So if your drive had pre -existing errors, particularly
something called cross -link files, where two
files accidentally claim the same physical cluster
on the disk, the results were catastrophic. Uh
-oh. Because the setup program is sweeping through,
grabbing clusters, compressing them, and then
what? Deleting the originals to save space? Yes.
It would compress the cluster for the first file,
delete the original, and move on. So when it
eventually got to the second file that claimed
that exact same physical space, the data was
already gone. Oh, man. Massive data loss during
the initial transaction. Massive. It would corrupt
files and could even destroy the virtual translation
table itself, ruining the entire drive before
the installation even finished. Which leads to
my biggest question. What happened if a user
was just, you know, poking around their hard
drive, making hidden files visible, and they
saw the suspiciously massive file eating up basically
100 % of their space, what if they just deleted
it? I mean, that is the most human error of all.
Yeah. Because the file was hidden, but curious
users often found it. And because they didn't
grasp the abstraction, they didn't realize that
one file was their entire operating system, and
all their personal data, they'd hit Delete to
free up space. Oh. Total annihilation. Deleting
that one file meant losing every single thing
stored on the compressed drive instantly. Just
boom, your whole digital life is gone. But despite
that risk, people still used it because, well,
it was either this or shell out hundreds of dollars
for a new drive. Exactly. The high cost of hard
drives made this software an absolute necessity,
which sparked a massive multi -million dollar
industry arms race. So let's talk about that
arms race because the tech really had to evolve
to keep up, right? It did. In the early days,
CPUs were just too weak to handle the math. So
the first solutions were actually hardware -assisted.
Oh, like physical add -on boards? Yeah. Companies
like Stack Electronics with their stacker product,
or InfoChip Systems with Expanse, they sold physical
coprocessor boards that plugged into the motherboard.
These had custom chips designed specifically
to do the compression math, taking the load off
the main CPU. But eventually processors got faster,
like the 486 and the Pentium? Right. And once
the CPUs gained enough power, the hardware boards
became obsolete. Software -only solutions completely
took over the market. We saw programs like Squish,
Superstore, and DoubleDisk dominating. And computer
resellers loved this, right? I mean, they were
bundling this software onto PCs just to claim
their computers had more storage without actually
buying bigger hard drives. Oh, it was a massive
marketing gimmick. And the operating system developers
caught on quickly. Digital research bundled Superstore
into DockerDOS 6 .0, and Microsoft bundled DoubleSpace,
which licensed DoubleDisk Tech into MS -DOS 6
.0. Okay, I wanted to ask you about this because
there's a very curious detail in the sources.
I see that MS -DOS 6 .22 bundled a program called
DriveSpace, but the version right before it,
MS -DOS 6 .21, shipped it with no compression
software at all. And the sources mention legal
reasons. What does that tell us about the climate
back then? Impartially speaking, without getting
into the weeds of the specific patent lawsuits,
that gap in MS -DOS versions really highlights
how intensely competitive and legally fraught
this industry had become. I mean, it was virtual
real estate. Exactly. Companies were aggressively
patenting their algorithms and their integration
techniques. It was a highly lucrative market.
And the fact that a flagship OS like MS -DOS
had to temporarily strip out its biggest feature
shows you just how fierce the legal battles over
this intellectual property were. Right. Everybody
wanted to own the rights to thin air. And the
technology was getting incredibly advanced before
the whole industry eventually collapsed. For
instance, the sources detail how Novell DOS 7
bundled a DPMS -enabled version of Stacker. What
does that mean for the user? It meant that if
you had Stacker on your local machine and Stacker
on a remote network server, they could exchange
data packets across the network while keeping
them compressed. Wait, so they didn't have to
uncompress the file, send it, and recompress
it? They just sent the shrunken version. Exactly.
Which effectively doubled the network bandwidth.
It was a brilliant use of the technology to speed
up entire local area networks. That's actually
amazing. So what does this all mean for the listener
today? I mean, where did all this fiercely contested
tech actually go? Because we don't buy box software
to double our drives anymore. It essentially
faded into disuse by the late 1990s. Hard drive
manufacturing advanced so rapidly and the cost
per megabyte dropped so low that physical storage
became incredibly cheap. So it just didn't make
sense to risk your data with a virtual drive
anymore. Right. But the DNA of the technology
absolutely survives today. Windows still supports
NTFS compression, which blends traditional file
compression with those transparent on the fly
concepts we just talked about. So it's still
running in the background. Yeah. And the open
source community kept the legacy alive too. LENX
developers created drivers like DMSDS specifically
to read those old 90s stacker and double space
volumes preserving that history. Which is pretty
cool. It just quietly became part of the background
architecture of how we compute. Which actually
leaves us with a really fascinating final thought
to mull over today. Yeah, because for an entire
generation of computer users, disk compression
was their very first introduction to the concept
of virtualization. It was the first time we learned
that the hardware you think you're interacting
with, the space, the folders, the drive, can
actually just be a software illusion. That's
a really good point. It completely decoupled
the physical reality from the digital experience.
Exactly. So, as you go about your day today,
you look at your cloud storage or use virtual
machines or log into simulated environments,
ask yourself, how much of your current digital
world is actually real physical hardware? And
how much is just a modern version of a giant
hidden file pretending to be a hard drive? Something
to think about. Definitely. We'll leave you with
that. Thanks for joining us on this deep dive.