Welcome to the deep dive. So we often talk about
compute as being, you know, the engine of the
cloud. But if compute's the engine, then storage.
Well, storage is everything else. It's the fuel
tank, it's the cargo hold. Your data is absolutely
vital. And knowing where to put it, how to keep
it safe, and maybe most importantly, how it affects
your bill. That's crucial. Get it wrong, and
you're really building on shaky ground. Absolutely
foundational. And look, for anyone who remembers
what it was like before the cloud, storage was
a huge headache. Oh, yeah. You're talking big,
expensive boxes, SANs, storage area networks,
NAS devices. Right. You had to spend a ton of
money upfront guessing. literally guessing how
much space you'd need years down the line. Which
never worked out right. Either you bought way
too much and half of it sat there gathering dust
costing a fortune or inevitably you ran out.
Usually at the worst possible time like 2 a .m.
on a Monday then it's panic stations trying to
get new discs shipped overnight. Exactly and
that whole nightmare scenario. That's what AWS
storage just completely changed. The core idea
is elasticity, global reach and crucially pay
as you go. No more crystal ball gazing for capacity.
Okay, so that's our goal today then we want to
give you the blueprint essentially We're gonna
decode the whole landscape of AWS storage s3
EBS EFS all those different tools The aim is
to give you that clear practical knowledge. You
can use right away Let's uh, let's start unpacking
this blueprint. Sounds good But before we even
name a single AWS service, like S3 or anything
else, we really need to get our heads around
the three basic ways data gets stored. Everything
else fits into one of these buckets, so to speak.
OK, three fundamental types. Let's walk through
them. First up. First is object storage. Now,
this one might feel a bit different from your
laptop's hard drive. How so? Imagine a massive
valley parking service, but for your data. You
give them your file. It could be a photo, a video,
a backup file, whatever. And they give you back
a unique ID, a ticket. You don't worry about
folders inside folders. It's a flat structure.
Your object is the file plus some metadata, and
you use that ID, that ticket, to get it back.
And because it's flat, that makes it Super scalable,
I guess. Good for just huge amounts of unstructured
data, like all those photos and videos. Exactly.
Massive scale. Yeah. Perfect for unstructured
stuff. So that's object storage. But OK, what
if you need storage that an operating system
can actually use, like to boot from or run a
database directly on? Right. Not just storing
files to download later. That sounds different.
That's where block storage comes in. That's block
storage. Totally different concept. Think of
it like a huge grid of equally sized boxes. blocks,
each with a number. When you attach this kind
of storage to a server, the operating system
sees it like a raw, unformatted hard drive, just
a bunch of blocks it can use. Ah, so the OS manages
the file system on top of those blocks itself.
Precisely. And that direct block access gives
you the really high speed and low latency you
need for things like databases or the operating
system drive itself. Got it. OK, so object for
massive scale, web access. Block for all performance
OS level access. What's the third flavor? The
third one is probably the most familiar file
storage This is your classic folders and subfolders
hierarchy like the shared drive on a network
exactly like a shared network drive The key thing
here the big difference from block storage Especially
is that multiple servers multiple computers can
connect to the same file storage and see the
same files at the same time It's designed for
sharing. Okay, the shared filing cabinet analogy
makes sense object block file The basics. Right.
Those are the building blocks. No pun intended.
Huh. OK. So let's dive into the AWS services
themselves. And we have to start with the biggest
one, right? The Titan. You have to start with
S3, Amazon Simple Storage Service. It's just
fundamental to so much on AWS. It's their massive,
super durable, super scalable object storage
service. The magic, infinitely deep closet you
called it earlier. That's a good way to think
about it, yeah. Whether you've got tiny files
or petabytes of data, S3 just handles it. Application
code, documents, backups, website assets, the
foundation for huge data lakes. It often ends
up in S3. And a couple of key terms people need
to know. Your data goes into buckets. And these
bucket names... They have to be unique across
the entire world of AWS, right? Globally unique,
yeah, that's important. It really underscores
that S3 is a global service, not tied to one
region in the same way some other services are.
And inside the buckets you have your objects,
which are the actual files plus their metadata.
But the headline feature, the one everyone talks
about, is the durability. Oh yeah, the durability.
AWS guarantees 99 .99999999 % durability. That's
11 nines. 11 nines. It sounds almost unbelievable.
What does that actually mean in practice? It
means, statistically speaking, if you stored,
say, 10 million files in S3, you could expect
to maybe lose one single file over the course
of 10 ,000 years. Wow. OK. That's pretty durable.
How do they even achieve that? By automatically
making copies of your data and storing them across
multiple separate physical data centers, what
AWS calls availability zones within a region.
It's redundancy built in at a massive scale.
Which explains why it's the default choice for
so many critical things. You mentioned backups,
disaster recovery, data archiving, even hosting
static websites directly from S3. Absolutely.
It's incredibly versatile because of that scale
and durability. OK, but within S3, it's not just
one size fits all, is it? You mentioned cost
being a factor. How do we manage that? Right,
so because S3 holds everything from data you
need instantly multiple times a second to archives
you might look at once a decade, they created
S3 storage classes. This is all about balancing
cost versus access time and frequency. And this
is a really key area for anyone designing systems
or even taking an AWS exam, I imagine, getting
the storage class right. Definitely. It can have
a huge impact on your bill. So let's start with
the default, the kind of main one. That's S3
standard. This is for frequently accessed data,
stuff you need fast. It gives you low latency,
high throughput. It has the highest storage cost
per gigabyte. But accessing the data is cheap
and instant. Right, website assets, active application
data. Right, the prime shelf for your most active
data. But what if my access patterns are, well,
unpredictable? Or they change over time? I don't
want to be constantly moving files between tiers
myself. Ah, for that, AWS is something quite
clever. S3 Intelligent Tiering. It's kind of
the set it and forget it. How does that work?
It uses machine learning, basically algorithms,
to watch how you access each object. If you stop
accessing something frequently, it automatically
moves it to a cheaper infrequent access tier
behind the scenes. If you access it again, it
moves it back. You get automatic cost savings
without lifting a finger. That sounds pretty
useful. OK, so that handles automatic optimization.
What about the tiers specifically designed for
data you don't access often, the infrequent access
tiers? Right, so the main one there is S3 standard
and frequent access, or S3 standard IA. The storage
cost per gigabyte per month is lower than S3
standard. Cheaper to store. Okay, what's the
catch? The catch is the retrieval fee. You pay
a per gigabyte fee every time you access or retrieve
data. from Standard IA. Ah, okay, so if I store
archives there cheaply but then suddenly need
to download terabytes of it, that could get expensive
fast. It could. You need to be mindful. Standard
IAs is great for things like older backups or
data you need long -term attention for, but don't
expect to access regularly. But crucially, when
you do need it, access is still fast. Just like
S3 standard, you just pay for that retrieval.
Got it. And there's an even cheaper IA option,
isn't there? S3 one zone IA. What's the trade
off there? It sounds less resilient. It is less
resilient. That's the key trade off. One zone
means exactly that. Your data is stored in only
a single availability zone within a region, unlike
the other S3 classes that spread it across multiple
AZs. So if that specific AZ has an issue, power
outage, flood, whatever that data is unavailable,
and potentially lost if the AZ is destroyed.
Unavailable during the outage, yes. And potentially
lost in a disaster scenario affecting that single
AZ, although that's rare. So it's much cheaper
storage, the cheapest non -archived tier, but
you accept lower durability. It's really only
suitable for data you can easily recreate if
lost, like maybe thumbnail images generated from
original stored elsewhere, or logs you also have
copies of. Not for primary data. OK, that's a
critical distinction. So standard for frequent,
intelligent tiering for unknown, standard IA
for infrequent but fast retrieval needed, one
zone IA for recreatable infrequent data. What
about really cold storage? The deep freeze. Now
we're talking about the archive tiers, which
used to be mainly under the Glacier brand name.
These are for maximum cost savings on data you
rarely access. Glacier. There's a glacier instant
retrieval. This one's interesting. It offers
the lowest cost storage for data that's rarely
accessed. But when you do need it, you need it
back in milliseconds, just like S3 standard or
IA. So archive economics, but immediate access.
Right, bridging that gap. Then the more traditional
glacier. Yeah, Glacier Flexible Retrieval. This
is the classic archive option. Very cheap to
store, but retrieval isn't instant. It can take
anywhere from minutes to several hours to get
your data back, depending on what retrieval option
you pick. Minutes to hours, okay. Definitely
for archives. And the absolute coldest, cheapest
option. That's Glacier Deep Archive. Rock bottom
storage costs. We're talking fractions of a cent
per gigabyte per month, but the retrieval time
reflects that. You're looking at 12 to 48 hours
to get your data back. 12 to 48 hours, okay.
This is for data you might literally access once
every few years. Think regulatory archives that
you have to keep for seven years but probably
never look at, or maybe the final backup before
decommissioning a system entirely. Long -term,
deep -free storage. Wow. Okay, that's a whole
spectrum just within S3. Standard, intelligent
tiering, standard IA, one -zone IA, Glacier Instant,
Glacier Flexible, Glacier Deep Archive. Lots
to choose from based on access patterns and cost.
Exactly. Matching the class to the need is key.
All right. We've spent a lot of time on S3, which
makes sense given its importance, but that's
mostly storage accessed over the internet, right?
Object storage. Let's shift gears. What about
storage attached directly to our virtual servers,
our EC2 instances, inside the virtual data center?
Right. Now we're moving away from object storage
and primarily into block and file storage territory,
directly connected to compute. And the main player
for block storage attached to EC2 is? That would
be Amazon EBS, the elastic block store. Remember
our definition, high -performance block storage.
EBS provides persistent block -level storage
volumes for use with EC2 instances. Think of
it as the virtual hard drive for your virtual
server. virtual hard drive. But there's a really
important limitation or characteristic people
need to grasp about EBS. Isn't there something
about location? Absolutely critical. An EBS volume
exists in and is bound to a single availability
zone, AZ. Just one AZ? Just one. You can only
attach an EBS volume to an EC2 instance that
is running that exact same availability zone.
It cannot cross AZ boundaries natively. That's
huge for designing resilient applications, right?
If that whole AZ has a problem, your EC2 instance
goes down and the EBS volume attached to it is
also unavailable. Precisely. High availability
for applications using EBS often involves strategies
like replicating data across instances in different
AZs or using those EBS snapshots. Ah, yes, snapshots.
How do they work? Are they full backups every
time? They're point -in -time backups of your
EBS volume. And the clever part is they are incremental.
After the first full snapshot, subsequent snapshots
only save the blocks that have changed since
the last one, which saves storage space and time.
And where do these snapshots get stored for durability?
They get stored under the hood in S3. Ah. So
AWS uses the incredible durability of S3, those
11 nines, as the backend to make EBS snapshots
reliable. That's smart. It's a very smart architecture,
yeah. So typical use cases for EBS are pretty
clear. The boot volume for your EC2 instance
where the operating system lives, or as the data
drive for databases like SQL Server, MySQL, PostgreSQL,
or NoSQL databases running directly on a single
EC2 instance needing that low latency block access.
OK, so EBS high performance block storage for
a single EC2 instance in a single AZ. What if
I need that shared network drive experience we
talked about earlier? Multiple EC2 instances
needing to access the same file simultaneously.
Now you're talking about Amazon EFS, the Elastic
File System. This is AWS's fully managed, scalable
file storage service. And the key differentiator
again? The key is that multiple EC2 instances
can connect to the same EFS file system concurrently.
That's the fundamental difference from EBS. So
perfect for things like a fleet of web servers
that all need access to the same set of website
files or shared application assets. Exactly.
Content management systems like WordPress or
Drupal running across multiple web servers, share
code repositories, home directories for users
logging into multiple machines. EFS fits that
perfectly. And how does it handle things like
scaling and availability? Well, it's elastic,
so it grows and shrinks automatically as you
add or remove files. You know, pre -provision
storage. And importantly, a regional service.
It automatically stores your data across multiple
availability zones within a region for high durability
and availability. Okay, so unlike EBS, which
is AZ specific, EFS spans multiple AZs within
a region. That's a big plus for availability.
Any specific protocols or OS limitations? It
primarily uses the NFSv4 protocol, network file
system version 4. which means it's primarily
designed for and works best with Linux based
EC2 instances. While there are ways to connect
from Windows, it's not the primary use case.
Got it. So EFS shared elastic regional file storage,
mainly for Linux workloads via NFS. You got it.
Okay, we've covered the big three. S3 for object,
EBS for block, EFS for shared file. Are there
other more specialized storage services we should
touch on? Maybe for specific workloads or hybrid
setups? Yeah, there are a couple more worth knowing
about, especially for bridging on -premises worlds
or handling specific application needs. Let's
hear about the bridge first. Connecting existing
data centers to AWS storage. That would be the
AWS storage gateway. Think of it as a hybrid
cloud storage appliance, like a magic portal.
A magic portal? Okay, intriguing. You typically
deploy a virtual machine, or sometimes a hardware
appliance, in your own data center. This gateway
then connects securely back to AWS and makes
cloud storage like S3 or even EBS volumes appear
as if it's local storage within your data center.
Ah, so my old backup software running on my local
server could think it's writing to a local tape
library or a network share. Exactly. But it's
actually sending the data through the gateway
up to S3 for durable, scalable cloud storage.
It lets you integrate cloud storage without having
to rewrite your existing on -premises applications
or workflows. It smooths that transition. That
sounds incredibly useful for migration and backup
scenarios. OK, what about specialized file systems?
That's where the Amazon FSX family comes in.
FSX stands for File System X, basically. It provides
fully managed third -party file systems optimized
for specific needs. Like what kind of specific
needs? So the two big ones are FSX for Windows
File Server and FSX for Lustre. OK, FSX for Windows
sounds pretty self -explanatory. It is. If you're
migrating Windows applications to AWS, and those
applications rely on a shared Windows file server
using the standard SMB protocol and Windows NTFS
permissions, FSx for Windows provides exactly
that, fully managed. It makes lifting and shifting
those apps much easier. Gotcha. So if my app
needs file super share, FSx for Windows can provide
that file super share in the cloud. Precisely.
And then there's FSx for Lustre. Lustre. Never
heard of it. Lustre is a different beast altogether.
It's an open source parallel file system designed
for extreme speed and scale. Think massive throughput.
Parallel file system. What kind of workloads
need that? We're talking high -performance computing,
HPC, scientific simulations, genomic analysis,
financial modeling, also heavy -duty machine
learning training, or big media workloads like
rendering 4K or 8K video. When you need to process
huge data sets across many compute nodes really,
really fast, FSX or Lustre is often the answer.
It's built for speed. OK, so FSx for Windows
handles the specific needs of Windows shared
file systems, and FSx for Lustre handles the
extreme performance needs of HPC and similar
workloads. That's the gist of it. Specialized
tools for specialized jobs. Wow. OK, that was
a whirlwind tour through the AWS storage universe.
S3, EBS, EFS, Storage Gateway, FSx. Quite a lot
to take in. It is a broad portfolio for sure.
But hopefully we've managed to demystify it a
bit. Let's try and boil it down to that quick
blueprint for success. When you're faced with
a storage requirement, how do you quickly decide?
OK, the quick decision tree. First, think about
the data type and access pattern. Is it unstructured
data photos, videos, backups, logs that needs
to scale massively and be accessible potentially
from anywhere, maybe over the web? If yes. That
screams S3. Start there. Think about the right
S3 storage class for cost and access speed. Okay.
What if it's not unstructured files, but you
need a high -performance virtual hard drive for
just one specific EC2 instance, like for its
operating system or a database running on that
instance? That's EBS. Remember, it's tied to
a single instance in a single AZ. Right. And
if you need that shared drive experience... multiple
servers, probably Linux, needing to access and
modify the same set of files concurrently. That's
your queue for EFS, Shared Elastic Regional File
Storage. And then the special cases. Migrating
a Windows app that needs a traditional Windows
file share. Look at FSx for Windows file server.
And if the requirement is just absolutely blistering
speed for something like HPC or massive video
rendering, then FSX for Lustre comes into play.
S3, EBS, EFS. Those are the main three pillars
to get right. Absolutely. And maybe if we tie
this back to where we started with durability.
Remember, S3 is 11 nines. That comes from AWS
managing the complexity of replicating data across
multiple physical locations, multiple availability
zones. They abstract away the hardware failure
for you at that level. Which is a huge shift
from the old days of managing your own ASAN,
right? Where one device failure could be catastrophic.
A massive shift. So maybe here's a final thought
for you, the listener, to ponder based on that.
If AWS provides that incredible 11 nines of physical
data durability with a service like S3, how does
that change how you, as an application developer
or architect, should think about data safety?
Do you still need to build all the same complex
application level checks and redundancy for data
loss that you might have done 10 15 years ago,
or has the cloud platform itself fundamentally
shifted some of that responsibility? Where does
the line sit now? Something to think about for
your next project. Definitely something to consider.
The landscape has changed. Indeed it has. Thanks
for sharing all that insight today. My pleasure.
Until next time. Keep diving deep.