You know, how comforting the math on a school
test is. Like, 90 % is an A, 80 % is a B. Right.
It's just straightforward. Exactly. You tally
up the correct answers, and the final result
just sits right there on the page. I mean, the
math does not care about your background, your
demographics, or your zip code. It's completely
objective. Or at least, that's what we like to
think. Right. And that brings us to today's topic.
Welcome to the deep dive. Today, we are looking
at a stack of sources, primarily focusing on
this really dense, but incredibly relevant Wikipedia
article on fairness in machine learning. Mm -hmm.
It's a huge topic right now. It really is. Our
mission today is to figure out what happens when
that comforting, seemingly objective math is
handed the power to decide, well, to decide major
things for you, like if you get a mortgage or
land a job. or even go to jail. Yeah, the stakes
are incredibly high. They are. Because what if
the grading rubric itself is secretly rigged
from the start? We are exploring how these cold
calculating algorithms are basically inheriting
our worst human biases. And, you know, to really
understand this, we have to look back. Because
while artificial intelligence feels like this
brand new futuristic frontier, the scientific
community has actually been wrestling with this
specific problem for a surprisingly long time.
Wait, really? Like, how long are we talking?
We're talking mid -1960s and 1970s. Oh, wow.
I wouldn't have guessed that. Yeah, it was following
the American Civil Rights Movement. You had the
passage of the U .S. Civil Rights Act of 1964,
and suddenly researchers were desperately trying
to figure out quantitative statistical ways to
test for unjust discrimination. Oh, right. Like,
in testing and hiring practices. Exactly. But
here's the wild part. By the end of the 1970s,
that debate had largely just... Well, it vanished.
Why? Did they solve it? No, not at all. The different
competing mathematical notions of what fairness
even meant left so little room for clarity that
the discussion just stalled out. People couldn't
agree. So let me get this straight. Decades later,
we have basically resurrected that exact same
unresolved philosophical debate. But now we've
just handed it over to machines. That is exactly
what happened. We gave an unresolved human problem.
to computers. OK, well, before we get into the
headache of how engineers are, you know, trying
to actually fix AI, we have to look at how these
systems are failing people in the real world
right now because the stakes here, they aren't
theoretical. No, they are very real. And the
legal system. is perhaps the most heavily scrutinized
area for this. Right. The sources mention that
big report. Yeah, the 2016 Po Publica report.
They did a major investigation into a risk assessment
software called Compass. And Compass is used
in the U .S. judicial system, right? Mm -hmm.
It's a tool widely used to predict a defendant's
likelihood of recidivism. Which is a... the probability
they will reoffend. Exactly. Now, ProPublica
claimed this algorithm was racially biased. Their
data showed that black defendants were almost
twice as likely to be incorrectly labeled as
a higher risk than white defendants. Almost twice
as likely, just incorrectly labeled. Yes. And
meanwhile, the software made the inverse error
with white defendants. It frequently labeled
them as lower risk when they actually went on
to reoffend. Wow. But the creator of the software
pushed back on that, didn't they? They did. The
creator Northpoint Inc. heavily disputed the
report. They argued their tool was completely
fair because from their mathematical perspective,
a high risk score meant the exact same likelihood
of reoffending regardless of the defendant's
race. OK, so North Point is saying, look, our
math is sound, but ProPublica is saying, look
at the disproportionate harm these errors are
causing. Right. ProPublica refuted North Point's
defense, arguing that the differing error rates
were what mattered. It became this massive clash
of what the data was actually saying. And what
metric of fairness mattered more, really? And
we see similar clashes in corporate America,
too. Oh, absolutely. Like Amazon built an AI
software to review job resumes, and it turned
out to be systematically penalizing women. Yeah,
that was a famous case. The algorithm had been
trained on a decade of past resumes, which, you
know, came predominantly from men. So the AI
learned to literally penalize applications that
included the word women. It's wild. Like, if
an applicant listed they were captain of the
women's chess club, they're score was docked.
Just for having the word in there. And then there
was the apple cart issue. in 2019. Right. The
algorithm determining credit limits came under
heavy fire. It was giving significantly higher
credit limits to men than to women, even for
married couples. The sources pointed out this
was happening even in cases where couples shared
all of their finances. They had merged assets
and possessed the exact same credit history.
It's a perfect example of how the math isn't
just objective and beyond financial unemployment
algorithms, visual and language biases are deeply
pervasive to. Oh, yeah. The image recognition
failures are just startling. Back in 2015, Google
Photos mistakenly labeled images of a black couple
as gorillas. Which is horrific. And then in 2020,
users discovered that an automatic image cropping
tool used by Twitter was actively prioritizing
lighter skinned faces. Like when it generated
thumbnail previews for timelines, it just defaulted
to the lighter faces. And those visual biases,
they extend directly into the large language
models everyone is interacting with today, like
chat GPT. Right. Everybody is using those now.
And current models suffer heavily from statistical
sampling bias, specifically what we call language
bias, because the vast majority of their training
data is in English. These models tend to present
Anglo -American political and cultural views
as, well, universal truths. The source material
highlighted a fascinating example of this. If
you ask a language model what is liberalism,
it defaults to the Anglo -American perspective.
Emphasizing human rights and equality. But it
completely ignores the dominant Vietnamese perspective,
which views liberalism as opposing state intervention
and personal life. Exactly. And it also ignores
the prevalent Chinese perspective, which focuses
on the limitation of government power. The chatbot
presents itself as this worldly, omniscient oracle,
but it is effectively blind to non -English perspectives.
Simply because of the language distribution in
its training data. You know, listening to all
these examples, it really shatters the illusion
of the objective machine. I mean, AI is not some
super smart alien entity arriving with perfect
logic. Not at all. It operates more like a parrot,
just eagerly repeating the Internet's worst habits
and historical inequalities right back to us.
That's a great analogy. And for you, the listener,
the crucial takeaway here is how deeply This
invisible math is currently judging you. Judging
us every day. Every day. Whether you are applying
for a mortgage, where algorithms are currently
more likely to reject non -white applicants,
or you're simply scrolling through a personalized
news feed on social media. You're constantly
being evaluated by systems built on data that
reflects a very flawed world. Exactly. Which
naturally leads to the big question. If we know
the machines are biased, why don't the engineers
just, you know, open up the code and program
them, to be fair? Just type in a math - mathematical
rule that says treat everyone equally? Well,
here is where we hit a massive mind -bending
contradiction. Defining fair mathematically actually
breaks the system. Wait, really? How does fairness
break the system? In machine learning, you have
a target variable, the thing you are trying to
predict, like who will repay a loan. And you
have sensitive characteristics, like gender,
ethnicity, or age. Researchers have established
three distinct mathematical definitions for group
fairness. Okay, leave them on me. The first is
called independence. It's sometimes referred
to as demographic parity or statistical parity.
So independence is about equal outcomes, right?
Like if a company is hiring and the local applicant
pool is 50 % men and 50 % women, the final group
of people who actually get hired must reflect
that exact same 50 -50 ratio. Exactly. The outcome
is totally independent of the sensitive trait.
Makes sense. What's the second one? The second
definition is called separation or equalized
odds. This metric demands that the true positive
rate and the false positive rate must be identical.
across all groups. So it's looking at the errors.
Right. This means the algorithm's accuracy and
its mistakes must be evenly distributed, it asks.
Out of all the people who truly deserve the job,
did equal percentages of qualified men and qualified
women get hired? And out of the people who were
unqualified, were they rejected at equal rates?
Precisely. Okay, so independence is kind of a
blunt instrument guaranteeing an equal overall
outcome ratio, but separation is more like a
microscope looking at equal accuracy among the
people who actually meet the criteria. That's
a very good way to put it. Now, the third definition
is sufficiency. This dictates that if an AI predicts
two people belong to a positive group, their
actual real -world probability of belonging to
it must be equal. Wait, how is it different from
the second one? It's about predictive reliability.
If the algorithm labels a man and woman as high
risk for defaulting on a loan, the actual chance
that they default needs to be identical. The
prediction's reliability cannot change depending
on the demographic group it is analyzing. Okay,
I think I'm wrapping my head around this. We
have independence for equal outcomes, separation
for equal error rates, and sufficiency for equal
predictive reliability. Three solid ways to measure
fairness. Yes, three very logical definitions.
So why not just program the AI to satisfy all
three simultaneously? Problem solved. Because
mathematically, achieving what researchers call
total fairness is impossible. It is a proven
mathematical paradox. Impossible, like literally
impossible. Literally, unless the baseline data
has absolutely zero correlation between the sensitive
trait and the outcome, which is virtually never
the case in the real world, you are forced to
choose. I'm going to need an everyday example
to understand this. Let's say I am hiring software
developers. OK, imagine you are hiring from two
different demographic groups, group A and group
B. Due to historical systemic wealth disparities,
Group A has had widespread access to expensive
computer science boot camps for the last 20 years.
And Group B has not. Right. Therefore, in your
baseline data, a higher percentage of Group A
currently possesses the specific coding skills
you need. If you enforce the first rule, independence,
you must hire exactly 50 % from Group A and 50
% from Group B. But because of that historical
disparity, the only way I can hit that 50 -50
quota is by lowering my hiring threshold for
Group B and raising it for Group A. Precisely
the issue. By lowering the threshold for Group
B, you are mathematically forced to hire some
candidates who might not actually have the required
skills. Which increases their false positive
rate. Exactly. And by raising the threshold for
Group A, you are forced to reject highly qualified
coders, increasing their false negative rate.
Oh, I see it now. And doing that completely dishores
the second rule, separation, because equalized
odds demands that those false positive and false
negative rates be identical across both groups.
It is a zero -sum game. Wow. So if I guarantee
equal outcomes, I am mathematically forced to
accept unequal error rates. You cannot mathematically
correct a historical imbalance without throwing
another fairness metric entirely out of whack.
You've nailed it. And because group fairness
is a mathematical paradox, many researchers argue
we should stop trying to fix the group entirely
and instead focus purely on the individual. OK,
so zooming in on the person, not the demographic.
Yes. This shift brings us to the concept of individual
fairness. It started with a very intuitive approach
called fairness through unawareness, or FTU.
You essentially institute algorithmic blindness.
Well, that makes total sense on the surface.
Just delete the sensitive data. Do not tell the
AI the applicant's gender, race, or age. Right.
The logic is, if the machine cannot see the demographic
data, it cannot discriminate against it. It sounds
ideal. Does it work? It completely falls apart
due to what we call proxy variables. An AI is,
at its core, a pattern recognition engine. Even
if you scrub the data set of race or gender,
the AI can effortlessly deduce those sensitive
traits through non -sensitive attributes that
are highly correlated. Oh, like a zip code. Exactly.
A zip code can act as a massive proxy for race
due to decades of historical housing segregation.
Or the specific extracurricular clubs listed
on a resume can instantly reveal gender. Right,
like the women's chess club example from earlier.
Yes. So an algorithm can perfectly comply with
fairness through unawareness while still engaging
in rampant discrimination. It is like putting
a blindfold on a referee to judge a sprint, but
completely ignoring the fact that someone already
placed heavy hurdles in half of the lanes. That's
a perfect analogy. The referee does not see color
or gender, but they also don't see the systemic
obstacles affecting the runner's times. And because
the blindfold approach fails so spectacularly,
researchers like Cynthia Dwork introduced a counter
concept in 2012 called Fairness Through Awareness,
or FTA. Through awareness. So we intentionally
tell the AI, the race and gender. Yes. This argues
that instead of blinding the AI, we must explicitly
use the demographic data to ensure we are mapping
similar individuals similarly. This led to a
really fascinating branch of study known as counterfactual
fairness. Counterfactual. Okay, imagine an AI
is scanning your loan application right now.
For that decision to be counterfactually fair,
how does the AI process your information? The
AI uses causal models to basically play out an
alternate reality. It isolates your sensitive
attribute, let's say your race. It then asks
if Every single detail about this individual's
life remained exactly the same, but they were
a different race. Would my final decision change?
Whoa. So it looks at the causal chain of events.
Exactly. If changing only that demographic node
flips the outcome, the model is definitively
biased. We are asking machines to simulate alternate
dimensions to audit their own biases. That is
brilliant. But how are the engineers actually
implementing this in the code? There have to
be practical strategies happening under the hood.
There are. Engineers generally attack the problem.
at three distinct phases of the machine learning
pipeline. Pre -processing, in -processing, and
post -processing. OK, let's break this down.
Pre -processing, that tackles the problem before
the AI even gets a chance to look at the data,
right? Yes. The assumption here is that the algorithm
is fine, but the historical data set it is about
to learn from is contaminated with bias. So how
do they clean it? The preprocessing stage attempts
to alter that historical data set to remove discriminatory
information while preserving the useful data.
One method is called reweighing. Reweighing,
like changing the value of the data points. Exactly.
The software assigns a mathematical weight to
each data point. If the historical data reflects
a bias against a certain demographic, the algorithm
assigns a heavier positive weight to successful
outcomes for that disadvantaged group. Oh, I
see. And then it does the opposite for the privileged
group. Right. It assigns a lighter weight to
successful outcomes for the historically privileged
group. It is like adjusting golf handicaps before
a tournament begins. You know, one player is
using like vintage wooden clubs while the other
has state of the art titanium gear. So you adjust
the baseline scoring logic in the data set. That
way, when the AI finally looks at the data, it
learns from a simulated, equitable world rather
than our flawed reality. OK, that's pre -processing.
What about the second phase? Moving to the second
phase, in -processing adds constraints to the
algorithm while it is actively learning. One
of the most compelling techniques here is adversarial
debiasing. Adversarial debiasing? That sounds
intense. It is. Instead of training a single
AI, engineers train two distinct neural networks
and pit them against each other using gradient
-based methods. Wait, gradient -based methods?
Are we just talking about how the AI mathematically
scores its own performance and, you know, updates
its brain? Exactly. The AI adjusts its internal
logic based on a mathematical gradient of how
wrong its previous guess was. In adversarial
debiasing, you set up a competition. A competition
between the two AIs. Yes. The first neural network
is the predictor. Its only job is to solve the
problem, like predicting who will repay a loan.
The second network is the adversary. And what
does the adversary do? Its sole purpose is to
analyze the predictor's output and attempt to
guess the sensitive variable, like the applicant's
gender, based purely on that decision. Oh, wow.
So the engineers force the AI to play an intense
game of hide and seek with its own biases. Exactly.
The predictor wants to correctly guess the loan,
but it also desperately wants the adversary to
fail at guessing the gender. Right. The predictor
actively modifies its internal weights, mathematically
scrubbing the bias out of its own logic. It learns
to make accurate loan predictions using only
data patterns that give absolutely no clues about
gender to the adversary. That is incredibly clever.
Okay, so if preprocessing fixes the past data
and in -processing fixes the active learning,
post -processing must be the final safety net.
What happens here? Do we just put a human at
the end of the line to double check the AI's
math? Close, but we put another algorithm at
the end of the line. Post -processing adjusts
the results after the AI has already made its
initial prediction. A common strategy here is
called reject option -based classification. Reject
option? How does that work? When an AI makes
a prediction, it generates a probability score.
If it is 90 % certain an applicant will repay
a loan, it approves it. If it is 10 % certain,
it rejects it. But what about the people hovering
right around 50 %? Ah, the borderline cases where
the AI is essentially flipping a coin. For those
highly uncertain predictions, this algorithm
intervenes. If the borderline applicant belongs
to a historically deprived group, the algorithm
automatically gives them the benefit of the doubt
and overrides the AI to label them positive.
And if they belong to the privileged group? It
labels them negative. It specifically optimizes
the threshold to give a boost where the AI is
unsure without disrupting the highly certain
predictions. It is the algorithmic equivalent
of saying the tie goes to the runner. But specifically
ensuring the tie goes to the runner, who we know
has historically faced severe headwinds. So we
can mathematically re -weigh data sets. We can
set up adversarial neural networks to battle
each other inside a black box. We can adjust
post -decision thresholds. But despite all of
this brilliant engineering, there is an ultimate
limitation that brings this entire conversation
crashing back down to reality. Yes. A human element.
It always comes back to us. It does. Recent research
highlighted in the sources points out a tragic
irony in this field. You can perfectly execute
all three phases of mitigation. You can achieve
an incredibly fair, mathematically sound algorithmic
output. Fuck. But in the vast majority of real
world applications, that AI does not make the
final call. It makes a recommendation to a human
operator. And humans are deeply susceptible to
confirmation bias. Extremely. Studies show a
phenomenon where human decision makers exhibit
automation bias, trusting the machine, but they
apply it selectively. What do you mean selectively?
They tend to accept AI recommendations only when
those recommendations align with their own pre
-existing prejudices. If a perfectly fair AI
recommends an outcome that contradicts the human
operator's internal bias, the human simply overrides
the machine. Oh man, so that completely neutralizes...
the millions of dollars and countless hours spent
engineering the fairness protocols. In many cases,
yes. Which is exactly why this topic matters
so much for you, the listener. As someone who
seeks out knowledge to understand the systems
running our world, your critical thinking is
the ultimate safeguard. It really is. We cannot
blindly trust the output of a machine because
a human might be using that seemingly objective
math to launder their own subjective bias. We
like to imagine the machine takes the human error
out of the equation, but often the human just
uses the machine to justify the error. Which
leaves us with a profound philosophical puzzle
to mull over. If an artificial intelligence is
trained on completely accurate historical data
and it flawlessly learns to predict the biased
unequal world we currently live in, is the AI
broken or is the world broken? That is the ultimate
question. Should we be designing algorithms to
accurately reflect our current reality, or should
we be using them to engineer a fairer one? Something
to think about next time a machine makes a decision
for you. Until next time.