Imagine walking into a hospital in the early
1980s. You have, you know, perfect grades, a
stellar academic record, glowing recommendations.
You submit your medical school application, but
a computer automatically just, it throws it in
the trash. Right. Without a second thought. Exactly.
Not because you aren't highly qualified, but
simply because your name, quote, sounds foreign.
You'd probably think there was a glitch, right?
Because math and computer code are supposed to
be objective. You type an equation into a calculator,
the screen shows a number, and that is the absolute
truth. It's a very comforting illusion, honestly.
We inherently want to believe that machines operate
in this clean binary realm of true or false,
completely removed from the messy subjectivity
of human emotion. Welcome to today's Deep Dive.
If you're a learner who loves pulling back the
curtain on how the world actually works, you
are in the exact right place. Today, we are taking
a really comprehensive look at the research and
history behind algorithmic bias. And our mission
for this deep dive is to completely dismantle
that illusion of the neutral machine. Is a big
mission. It is. We're going to uncover how deeply
human flaws get baked directly into computer
code, why this unseen code dictates so much of
your daily life, and how society is attempting
to fight back. And this is such a vital mission,
because we are all currently falling victim to
a psychological phenomenon known as automation
bias. Automation bias. OK, what is that exactly?
Well, as algorithms increasingly organize our
financial systems, our legal institutions, our
daily behaviors, our brains naturally trick us
into trusting a machine's output more than we
trust human experts. Oh, wow. Yeah. We just assume
the machine has crunched the numbers perfectly,
which causes us to switch off our critical thinking.
We just accept the output. OK, so let's unpack
this, because we need to look at the reality
of that output. The stakes here are so much higher
than like why your streaming service keeps recommending
mediocre movies. Far higher, yeah. We're talking
about invisible code, deciding who gets hired
for a job, who receives life saving health care,
and who gets flagged for a prison sentence. To
understand how algorithms came to rule our present
with such authority, we actually have to look
backward. Right. This problem isn't some modern
phenomenon caused by social media or big tech.
It began the exact moment humanity started teaching
computers how to, quote, think. So how far back
are we talking? If we go all the way back to
1976, an early artificial intelligence pioneer
named Joseph Weizenbaum was already sounding
the alarm. Wait, 1976? We were, like, barely
out of the punch card era of computing. Exactly.
Computers took up entire rooms and had a fraction
of the power of a modern smart toaster. Yet,
Weizenbaum understood something fundamental about
the architecture of computing. He argued that
computer programs, by their very nature, embody
law. Embody law? What does he mean by that? Well...
When a programmer writes code to solve a problem,
they're enforcing a very specific, rigid way
to reach a solution. And because a human being
has to write those rules, the code naturally
and inevitably incorporates that specific programmer's
imagination, their expectations, and their implicit
biases regarding how the world is supposed to
work. That makes sense. The source material actually
has a brilliant analogy from Weissenbaum that
really crystallizes this. Oh, the tourist one.
Yeah. He says, trusting a computer program you
don't fully understand is like a tourist trying
to find their hotel in a strange city by flipping
a coin at every single intersection. Right. Sure,
they might eventually arrive at a bed, but just
because there's an end result doesn't mean the
process was reliable or accurate or safe. Exactly.
But, I mean, you would think early computer scientists
realized this, right? If they knew they were
building their own biases into the code, wouldn't
they actively test for that? You would certainly
hope so, but, um... That brings us right back
to that 1980s hospital scenario you mentioned
at the start. Wait, that wasn't a hypothetical.
No, that wasn't hypothetical at all. It's one
of the first major documented algorithmic scandals.
It took place between 1982 and 1986 at St. George's
Hospital Medical School in the UK. Oh, wow. OK.
The administration wanted to make their admissions
process more efficient, so they implemented a
new computer guidance system to screen applicants.
And the easiest way to teach a computer who to
admit is to show it who the school admitted in
the past. Precisely. The algorithm was trained
on the school's historical admissions data. It
just looked for mathematical patterns and the
types of candidates the human administrators
had previously accepted. And humans are flawed.
Right. Because the human administrators possessed
inherent biases, the algorithm learned those
biases as mathematical rules for success. So
it just automated the discrimination. Yeah. As
a result, the system systematically denied entry
to around 60 women and people with non -traditional
foreign -sounding names every single year it
was in operation. That is staggering. And, I
mean, the most terrifying part is that it wasn't
a glitch. Not at all. The algorithm was working
perfectly according to its design. It was just
accurately automating society's existing prejudices
at lightning speed. That historical bias really
forms the foundation of the problem. But to understand
our modern landscape, we have to look at how
bias physically enters the algorithms we interact
with today. Right, because it's not just one
thing. Exactly. It doesn't happen through a single
faulty line of code. It infiltrates the system
at multiple distinct stages of an algorithm's
lifecycle. Okay, so let's trace that lifecycle.
Before a single line of code is even doing its
job in the real world, the training data itself
can be corrupted, right? Yes. The source material
calls this pre -existing bias, and specifically
highlights something called label choice bias.
What is that? Label choice bias happens when
programmers need an algorithm to measure a highly
complex, nuanced concept, but computers can only
understand hard data. Okay, so they have to simplify
it. Right. So the programmers choose a measurable
proxy to represent that complex concept. The
source text details a widely used healthcare
algorithm designed to identify patients with
highly complex health needs. So the hospital
could allocate extra resources to them? Exactly.
But the algorithm couldn't inherently measure
sickness. Sickness is too abstract. Instead,
it used healthcare costs as a proxy for healthcare
needs. Which, I mean, on a purely surface level
seems logical. Usually the sicker a person is,
the more money their medical care costs. Sickness
equals expense. It seems perfectly logical until
you apply historical context. Historically, due
to deep systemic inequalities, lack of geographic
access to specialists, and historical distrust
of the medical system, black patients incurred
much lower health care costs than white patients.
Oh, I see where this is going. Yeah. When the
algorithm processed this data, it didn't understand
the socioeconomic reasons behind the lower costs.
It simply looked at the numbers and mathematically
predicted that the black patients were significantly
healthier than equally sick white patients. So
an algorithm designed to help the sickest patients
ended up denying care to the people who needed
it most simply because it confused how much money
you spend with how sick you actually are. That
is the insidious nature of label choice bias.
But the problem goes far beyond just as choosing
the wrong numerical proxy. The bias is fundamentally
woven into the actual words we use to communicate
with these machines. Here's where it gets really
interesting, because this is a massive issue
right now with the explosion of large language
models or LLMs. Oh, absolutely. But if an algorithm
is just reading text, how can it be biased against
a dialect or a specific way of speaking? Well,
we have to look at how these massive AI models
are trained. They are fed almost incomprehensible
amounts of data scraped from the internet just
billions of pages Right and because the early
internet and the bulk of digitized academic and
corporate text is primarily in English These
models tend to present a very specific Anglo
-american worldview as the absolute universal
truth But the source text reveals that it goes
much deeper into active dialect prejudice A 2024
study showed that these models exhibit covert
racism against speakers of African -American
English. Wait, covert racism from an AI? Yeah.
Because of the mathematical weights assigned
to certain words and phrases in their training
data, the models actually harbor negative stereotypes
about the dialect that are sometimes even more
extreme than recorded human biases. That's insane.
And this language bias affects software we normally
think of as purely technical and objective, too.
Like what? Take Turnitin, for example, the plagiarism
detection software used by high schools and universities
all over the world. The source notes that Turnitin's
detection algorithms actively penalize non -native
English speakers far more often than native speakers.
It is crucial to understand why that happens.
It's not because non -native speakers are plagiarizing
more. Right. It comes down to the mechanics of
the software's string matching code. Right, because
native English speakers possess a natural intuitive
vocabulary. If they're summarizing a text, they
inherently know how to swap out words and use
synonyms, which breaks up the string of text.
Exactly. The software looks for a matching string
of 10 words, but the native speaker changed the
fifth word, so the software sees it as original
thought. But for a non -native speaker... A non
-native speaker might rely more heavily on the
original phrasing to ensure their grammar is
correct, resulting in an exact string match that
triggers a severe plagiarism penalty. That is
a textbook example of technical bias, where the
constraints of the software itself inadvertently
favor one demographic over another. So let's
say a team of engineers manages the impossible.
They perfectly clean their training data, they
avoid all the technical traps with language,
and they build a pristine, mathematically sound
algorithm. They're safe, right? Actually, no.
Because the moment that pristine code hits the
messy, unpredictable real world, it breaks in
completely new ways. Oh, great. Sociologists
call this emergent bias. A classic example from
the source material happened in 1990 with the
National Residency Match program. That's the
program that uses an algorithm to place graduating
medical students into hospitals across the country,
right? Yes. For years, the algorithm worked perfectly.
But as social dynamics shifted and significantly
more women entered medical school, the program
suddenly encountered married couples applying
together. Ah, and they naturally wanted to be
placed in the same city. Exactly. The human element
introduced a completely new variable, compromise.
But the algorithm was coded for rigid optimization,
not human compromise. Right. When it encountered
a couple, it didn't know how to find a middle
ground geographic location. It simply looked
at the mathematical scores, weighed the location
choices of the higher rated partner first, and
placed them. Completely ignoring the other person.
It effectively penalized the lower rated partner's
preferences entirely, often sending couples to
opposite ends of the country. And when emergent
bias involves complex societal systems rather
than just job placements, it creates something
incredibly dangerous, which is feedback loops.
Oh, these are terrifying. Yeah. I want to look
closely at the PredPol software example from
the source text. This was a predictive policing
algorithm used in Oakland, California. The goal
was to predict where future crimes would happen
so the city could proactively deploy police resources.
And to make those predictions, the software was
fed historical public crime reports. Based on
that past data, the algorithm determined that
certain black neighborhoods were high risk. Right.
And it automatically dispatched more police cruisers.
to patrol those specific streets. But think about
how human behavior operates. If a neighborhood
is suddenly saturated with police cruisers driving
up and down the street, the officers are naturally
going to observe and record more minor infractions
simply because they are physically present to
see them. And what does the algorithm do with
those new police reports? Oh wow. It creates
a self -fulfilling prophecy. The algorithm ingests
those new reports, pats itself on the back for
making a correct prediction, and mathematically
decides that the neighborhood is even more dangerous
than it originally thought, so it sends more
police. The algorithm is entirely blind to the
fact that its own initial deployment caused the
spike in recorded data. It just locks the city
into an endless racist loop of over -policing,
all while hiding behind the defense of objective
math. these mechanisms, how bias infiltrates
at the design stage through technical constraints
and through emergent real world feedback loops,
you start to see the vast real world impacts.
It's everywhere. These invisible systems are
making highly consequential decisions about your
daily life, your finances and your identity.
And we should be clear that these impacts aren't
always accidental. Sometimes the bias is deliberately
engineered for commercial manipulation. That's
very true. As early as the 1980s, American Airlines
intentionally built a flight finding algorithm
that artificially boosted its own flights to
the top of the search results, regardless of
whether a competitor offered a cheaper or more
convenient route. Right. Today, that concept
has evolved into what scholars call digital gerrymandering.
Research shows that search engine algorithms
can be manipulated to subtly shift the voting
outcomes of undecided voters by up to 20 percent.
just by altering the order in which political
information is presented. Exactly. That invisible
manipulation is incredibly powerful because the
user believes they are conducting an independent
search. But the most devastating impacts outlined
in the source material occur when algorithms
interact with human identity. specifically gender,
race, and LGBTQ plus status. Right. Let's look
at the mechanics of gender bias. Amazon spent
years developing an automated AI recruiting tool
to sort through job applications. They eventually
had to completely scrap the project. Because
it was biased. Because the algorithm actively
penalized resumes that contained the word women's.
So if an applicant listed that they were like
the captain of the women's chess club, Their
resume was mathematically downgraded. Yes. And
again, this goes back to the training data. Amazon
trained that AI on 10 years of their own historical
resumes. Because the tech industry has historically
been male dominated, the vast majority of successful
resumes in that training data came from men.
Exactly. The AI essentially learned that the
mathematical pattern for a successful employee
was being male. So it ruthlessly filtered out
any indicators of female identity. We see similarly
invasive algorithmic behavior in retail, too.
Target developed an algorithm capable of predicting
when female customers were pregnant before they'd
even announced it to their own families. Wait,
how is that even possible? Purely by detecting
subtle mathematical shifts in their purchase
histories, like buying unscented lotion instead
of scented. That feels like a huge invasion of
privacy. It is. Meanwhile, LinkedIn used to have
a feature that would correct female names in
search queries. If a user searched for Andrea,
the algorithm would prompt, did you mean Andrew?
But it never did the reverse. It never did the
reverse because the algorithm weighed the male
name as the higher probability. default reality.
The impacts become even more alarming when we
move from retail into surveillance and law enforcement.
The source text highlights a landmark 2018 study
on commercial facial recognition systems. This
is a huge one. The researchers found that the
algorithms had error rates of up to 35 percent
when trying to identify darker skinned women.
But when tasked with identifying lighter skinned
men, the error rate plummeted to less than one
percent. This failure is a combination of bad
training data and physical technical bias. The
databases used to train these facial recognition
models were overwhelmingly filled with images
of white faces. Leaving the algorithm mathematically
blind to the diverse facial features of darker
skinned individuals? Right, and the consequences
of this are not abstract. This technological
failure has directly led to the wrongful arrests
and incarcerations of black men in the United
States. And globally, these biased surveillance
algorithms are being actively weaponized. In
China, algorithms are used to heavily restrict
the Uyghur Muslim minority. The source notes
that members of this demographic aren't even
allowed to purchase a simple kitchen knife unless
they pass algorithm security protocols and have
a QR code containing their personal data physically
edged into the blade of the knife. It's dystopian.
Here in the U .S., the justice system relies
on an algorithm called COMPASS. Judges use it
to predict the likelihood of a defendant reoffending.
And studies show that COMPAS erroneously labeled
black defendants as high risk twice as often
as white defendants. Baking historical arrest
disparities directly into future sentencing guidelines.
The LGBTQ plus and disabled communities face
severe algorithmic exclusion as well. Grindr's
recommendation algorithm was once inexplicably
linked by the Android Store to apps designed
to track sex offenders. Creating a highly dangerous
association. Exactly. Uber implemented a facial
recognition security check for his drivers that
completely failed to recognize transgender drivers
who were actively transitioning. because their
faces were changing? Right, the algorithm flagged
their changing appearance as fraud and automatically
suspended their accounts, cutting off their livelihoods.
And if you are a person with a speech impairment,
you are often entirely excluded from the modern
digital world. Voice assistants like Alexa or
Siri simply aren't trained on diverse voice data
rendering the technology useless to millions
of people. Now, I want to pause here for a second
because it's important to note that this isn't
just about corporate or social bias. It impacts
global politics, too. And we want to be completely
impartial here and just, you know. share what
the data in our source material shows without
taking any sides at all. Right. We're just looking
at the findings. Exactly. Looking purely at the
data in the source material, we see that AI doesn't
pick sides based on objective truth. It just
mirrors its training data. The source mentions
a 2025 study by the Anti -Defamation League,
which found that several major LLMs, including
ChatGPT and Gemini, showed anti -Israel bias.
Which highlights the core reality of machine
learning. These systems are just reflections.
They do not possess objective reasoning. They
just spit back what they read. Whatever political,
cultural, or social biases exist in the millions
of documents they digest will be regurgitated
as authoritative fact, regardless of where that
data falls on the political spectrum. You might
be listening to all of this and thinking, OK,
if we know these algorithms are sexist, racist,
and commercially manipulative, why don't we just
open up a laptop? find the bad lines of code,
and hit delete. If only it were that simple.
Right. Like, why is it so hard to just fix the
math? It comes down to a concept defined by sociologist
Bruno Latour called blackboxing. Blackboxing.
Latour observed that when a piece of technology
becomes highly successful and highly efficient,
it becomes invisible and opaque. We only look
at the inputs, what we ask the computer to do,
and the outputs what the computer gives us. Right.
We completely ignore the internal complexity.
We used to think of computer code like a recipe.
If the cake tastes bad, you look at the recipe,
see that you added too much salt, and you change
it. But modern algorithms aren't recipes anymore,
are they? Not at all. It is much more accurate
to think of them like a biological brain, a massive
interconnected neural network. If I ask you why
your favorite color is blue, you can't point
to the exact physical neuron in your brain that
made that decision. I have no idea, yeah. Modern
algorithms operate the same way. The search engine
Bing, for example, runs up to 10 million subtle
A -B test variations of its service every single
day. 10 million a day. Every single day. They're
constantly tweaking weights, pathways, and connections.
The tech companies have built digital brains
so massive and fluid that even the original engineers
cannot trace exactly which line of code made
a biased decision. That is wow. So it's like
trying to fix a faulty recipe when you aren't
allowed in the kitchen. You can't see the ingredients
and the chef is changing the recipe 10 million
times a day. Exactly. And on top of that, sheer
technical complexity. These algorithms are fiercely
protected as corporate trade secrets. Of course.
The companies refuse to let independent auditors
examine the digital brain because they don't
want competitors copying their product or users
learning how to game the system. But there is
also a massive philosophical roadblock here.
Even if we could open the black box and see the
code, we still have to define what fairness actually
looks like. And the source points out that this
is mathematically impossible to agree on. It
really is. Because if you program an algorithm
for equality of outcome, meaning every demographic
gets the exact same statistical result that inherently
contradicts equality of treatment, where you
treat individuals differently based on their
specific nuanced context. So if the math is hidden
and we can't even agree on the definition of
fairness, are we just doomed to live under the
authority of alien algorithms? We certainly aren't
doomed. But fighting back against algorithmic
bias requires a massive multi -pronged approach.
It combines technical innovation, a radical shift
in human diversity, and strict government regulation.
OK, so what's happening on the technical side?
Well, organizations like the IE, The world's
largest technical professional organization have
published new standards, like the 7003 -2024
standard, which provides creators with specific,
rigorous methodologies to identify and eliminate
bias during the design phase. That's a good step.
There is also a massive industry push for explainable
AI, which essentially forces the neural network
to show its work and justify how it reached a
decision. The source also mentions tools like
data sheets for datasets and model cards. The
best way to think about these is like a mandatory
nutrition label on the back of a cereal box,
but for AI. I like that analogy. Yeah, instead
of listing calories and sugar, these model cards
would clearly list exactly what demographic data
the AI was trained on, what data it is entirely
missing, and what real -world tasks it absolutely
should not be used for. Those technical tools
are essential, but they are entirely insufficient
without human solutions. The source notes a severe
diversity crisis in the field. Currently, only
12 % of machine learning engineers are women.
Just 12%. That's wild. That's a huge problem.
Initiatives like Stanford's Institute for Human
-Centered AI and groups like Black in AI are
actively fighting to fix the overarching whiteness
and male -dominated culture of AI development.
Because if the people building the tools don't
reflect the diversity of the real world, the
tools never will. And because we can't just rely
on tech companies to self -regulate, governments
are finally stepping in to pry open that black
box. In Europe, the General Data Protection Regulation,
or GDPR, has a specific clause Article 22, which
explicitly prohibits solely automated decisions
that have legal or significant effects on a person's
life. You inherently have the right to demand
a human in the loop. Here in the U .S., local
governments are taking action, too. In 2023,
New York City implemented a groundbreaking law
that actually requires employers to conduct independent
third -party bias audits for any automated hiring
tools they use. And they have to publicly publish
the results, right? Yes, they do. We are also
seeing significant federal movement. President
Biden signed executive orders mandating best
practices for AI safety, specifically targeting
fraud and algorithmic discrimination. And globally.
Globally, India's proposed personal data bill
takes a similar stance, explicitly targeting
discriminatory treatment as a legal harm that
companies can be prosecuted for. That's fascinating.
But if we connect this to the bigger picture,
the most important shift here is the change in
perspective. The ultimate solution isn't just
writing better, cleaner math. It is adopting
frameworks like the Toronto Declaration, which
treats algorithmic harm not as a computer glitch,
but as a fundamental human rights violation.
Oh, wow. It demands legal accountability and
liability from the private corporate actors who
deploy these untested systems into our public
spaces. Let's bring all of this together. What
we've uncovered today is that algorithms are
not neutral calculators. They are not objective
judges immune to human flaws. They are highly
polished mirrors reflecting humanity's best intentions
right alongside our worst, most deeply ingrained
historical prejudices. Very well said. Before
we wrap up, I want you to just take a second
and consider how many invisible algorithms you
had to interact with today just to listen to
this deep dive. Your phone's facial recognition
to unlock the screen, the podcast platform's
recommendation engine that surfaced the episode,
the network routing the audio to your headphones.
It's everywhere. The unseen code is constantly
making decisions about what you see and what
you can do. And as we integrate these complex
systems even deeper into the fabric of our society,
the responsibility I want to leave you with a
final lingering thought to mull over. Let's say
we eventually manage the impossible. Let's say
that one day we build an AI that is perfectly
objectively fair, completely stripped of all
human bias, emotion, and historical baggage.
OK. Whose specific version of fairness will it
use? And more importantly, would we humans, with
all of our inherent flaws, our contradictions,
and our messy emotions, even be willing to accept
the rigid decisions of a machine that no longer
thinks anything like us? That is the real question.
Thank you for joining us on this deep dive. Keep
questioning the invisible code running your world.