All right, everyone, welcome to another deep dive.
Today, we're going to be looking at AI governance.
Ooh, exciting.
Yeah, it is, right?
I feel like it's something we hear all the time,
you know, AI governance this, AI governance that.
But I think it can be kind of hard to wrap your head around.
Absolutely.
So you sent me some really cool research and articles
about it, and I think we can unpack some of that today
and help our listeners kind of understand,
like, what are the key things to think about when
we talk about AI governance?
Like, what even is this wild west of AI
that we hear about all the time?
And like, why should I care about data governance
if I'm not a data scientist or some kind of tech person?
So maybe we can start with one of the first articles
you sent me, which was from Gender,
and it was about data governance, which is a term I feel
like I hear all the time, but maybe you don't fully understand.
Can you kind of break it down for us?
Yeah, so what Gender really focuses on in this article
is that data governance is really the foundation
of AI governance.
So AI governance begins with solid data governance,
and that's because AI, its intelligence,
relies on the data that fuels it.
So if you have bad data or sensitive data
that you're not supposed to be using,
then it's going to lead to what we hear about AI hallucinations,
which is where the AI just kind of makes things up
or reveals information that could lead to a data breach
or reputation damage to your company.
So it's all about the quality of the data.
It really is all about the quality of the data.
That you're putting into these systems.
Yeah, think of it like baking a cake.
OK.
You put in rotten eggs.
Yeah.
Not going to be a good cake.
Makes sense.
So it's really like garbage and garbage out.
Yeah.
You have to start with good data to get good results.
So what are some real world consequences
of using bad data in an AI?
OK, so let's say you're a financial institution
and you're using an AI system to assess loan applications.
OK.
But the data that's used to train your AI system
has historical biases against certain demographics.
Oh, wow.
So that the AI could unknowingly perpetuate these biases
and lead to unfair lending practices.
Yeah, that's a huge deal.
Yeah, that's a big one.
And that's why data governance is so important in this context.
Yeah, it's not just like a technical thing.
It actually has real world consequences.
Yeah, and it goes even beyond those biases,
like you were saying.
Inaccurate data can lead to these AI hallucinations
where the AI generates outputs that are just
like completely nonsensical.
Let's say a health care provider is using an AI system
to analyze medical images.
OK.
But the data that they use to train this AI
is incomplete or corrupted.
Right.
The AI might misinterpret those images
and lead to misdiagnoses.
Oh, why not?
Or even incorrect treatment recommendations,
which is a scary thought.
Yeah, that's not good.
That's a bad situation.
So data governance, we've established that it's important.
But once we have this good data, what else
do we need to think about when we're talking
about governing these systems?
Yeah, so then we need to think about model discovery.
And this is once you have your data sorted.
OK.
Model discovery is essentially being
aware of what specific AI models you're using
and for what purpose.
So we know we have good data, but we also
need to know what specific tool we're using.
Yes, exactly.
OK.
Think of it like you're doing a home repair.
OK.
You need to choose the right tool.
Right.
You wouldn't use a hammer to screw in a light bulb.
Definitely not.
Yeah.
So it's all about understanding what is being used.
That's right.
And for what reason.
OK, so why is that so important?
Like why do I need to know what model is being used?
Like what's the big deal?
So as AI becomes more and more popular and integrated
into our lives, stakeholders from customers to regulators
are going to want to know, like what's under the hood,
how are these decisions being made?
Right.
And this is especially important as regulations
like the EU AI Act emerge.
Right.
One of the articles you sent me mentions that.
Yes.
That act specifically, yeah.
Yeah.
The EU AI Act, it categorizes AI systems
based on their risk levels.
OK.
And the higher risk your application,
the more regulations you're going to have.
Right.
The stricter requirements you're going
to have for transparency and all of that.
So it's not even just like, oh, nice to have.
It's like, you need to do this.
That's going to be a legal requirement in some cases.
OK.
So this is getting serious.
Yes, it is getting serious.
OK.
So we have this good data.
We know what model we're using.
But then to get into like the actual meat of what
good AI governance looks like, in the research
that you sent me, gender had these five pillars
of AI governance.
Yes.
Can we kind of break down what each of those are?
Yeah.
So the first one is transparency, which we already
kind of talked about.
So that's knowing which models you're using,
understanding how they work, how they reach their conclusions,
and what data was used.
Yeah.
So it's not just like a black box.
Yeah, OK.
And this can be achieved using what's
called explainable AI, which focuses on AI systems that
provide clear justifications for their decisions.
Oh, that's cool.
Yeah.
So it's more understandable to humans.
I like that.
Yeah.
So explainable AI, pillar number one, what about number two?
Pillar number two is ethical use.
OK.
And that focuses on ensuring that AI systems are used
in a way that aligns with ethical principles.
OK.
So tackling things like bias.
Right.
And making sure that it doesn't discriminate.
OK.
For example, if a company is using an AI-powered recruitment
tool, they need to make sure that this AI is not unfairly
discriminating against certain candidates.
Right.
So making sure that AI is being used for good, not for evil.
Exactly.
And then the third pillar is accountability.
And this is as AI takes on more and more responsibility.
Yeah.
It's important to establish clear lines of accountability.
Right.
So like if something goes wrong, we know who's responsible.
Exactly.
So who is liable if an AI system makes a mistake?
OK.
For example, let's say a self-driving car
gets into an accident.
Right.
Who is accountable?
Right.
Is it the developer, the manufacturer, the passenger?
That's a good question.
Yeah.
These are all important questions to consider.
OK.
So that's accountability.
What about number four?
Number four is compliance.
OK.
And this means making sure that the AI systems follow
the rules and regulations.
So not just doing the right thing,
but making sure you're following all the rules too.
Exactly.
Like data privacy regulations, for example.
OK.
So compliance, and then the last one, pillar number five.
Pillar number five is risk management.
OK.
And that focuses on identifying and mitigating
any potential risks.
So being prepared?
Exactly.
Like thinking ahead?
Yes.
Thinking ahead, taking a proactive approach
to AI safety and security.
So considering data breaches, algorithmic bias,
and any unintended consequences.
Right.
Because sometimes you don't know what those are.
Right.
Exactly.
And so risk management is just having a plan in place
to address these issues.
So it's really about thinking about all these different facets
of what could go wrong.
Yeah.
OK.
So those five pillars, I really like that framework.
I think it's helpful to kind of break down
what responsible AI development looks like.
Exactly.
It's a really good framework.
So we've talked about data governance.
But in one of the other pieces you sent me,
Matter talks about this idea of the wild west of AI.
Yes.
That's a good one.
What does that even mean?
Like, help me understand that analogy.
Yeah.
So what Matter's really talking about
is that in a lot of companies, there's
a lack of control and visibility when it comes to AI.
So people are experimenting with AI in different departments
using different tools, different data sets.
And oftentimes there aren't clear guidelines in place.
So it's like everyone's just kind of doing their own thing.
Yeah.
Exactly.
And there's no one to kind of.
Yeah, there's no sheriff in town.
Wrangle it all in.
Exactly.
OK.
And this can be really risky.
What kind of risks are we talking about here?
So if you don't know where AI is being used in your company,
you can't effectively govern it.
OK.
You might have different teams using different models.
Yes.
That have conflicting assumptions.
Right.
Or even contradictory goals.
Yeah.
So it's like having a bunch of chefs in a kitchen
all making different dishes.
Yes.
And you don't know what you're going to get.
Yeah.
It could be a mess.
And you don't know what you're going to get.
It could be a disaster.
So then how do we address that?
How do we wrangle this Wild West?
Yeah.
So what you can do is establish a centralized AI governance
function.
OK.
So it could be a dedicated team or individual
that's responsible for implementing these policies
across the board.
So like a sheriff?
Yeah, exactly.
The sheriff comes in.
To kind of keep everyone in line.
Yeah.
But it sounds like that could be a big lift for smaller
organizations that maybe don't have a ton of resources.
Right.
And the good news is that AI governance doesn't
have to be an all or nothing proposition.
OK.
You can implement it in stages.
OK.
Starting with the most critical areas.
Right.
And then gradually expanding it.
So it just starts somewhere.
Start somewhere.
Take those first steps.
OK.
So we've talked about good data.
Right.
And the importance of like model discovery.
But in another one of the articles you sent,
Roy had a really interesting point.
Yes.
About the need for a strong engineering perspective
when we're thinking about AI governance.
Yeah.
He really emphasized the importance
of defining the problem you're trying to solve with AI.
OK.
Before you even think about data or algorithms.
So before you get all technical.
Yeah.
Before you get into all that.
You need to think about like the big picture of what
we're even trying to do.
Exactly.
And like why are we doing this?
Yeah.
What problem are we trying to solve?
Right.
Because you can end up creating a whole new set of problems.
You can create more problems.
You're not careful.
And it's not just identifying the problem.
It's also understanding the context in which the AI will
be operating.
What do you mean by context?
So let's say a company is developing an AI system
to personalize product recommendations.
OK.
The context matters.
Right.
So a recommendation engine for like a clothing store
is going to be very different than one
for a health care website.
Exactly.
You have to think about things like user privacy, data
security, and potential ethical implications.
Right.
Because like what might be an OK recommendation in one
context could be really bad in another.
Exactly.
So you need to think about all that stuff up front
before you even get into the technical bits.
Before you write any code.
OK.
And I think Roy's point about diverse perspectives
is really key here.
It's not just the engineers who should be in the room
when we're making these decisions, right?
Absolutely not.
Like who else should we be talking to?
We should be talking to data scientists, product managers,
legal experts, even end users.
Yeah.
That makes sense.
It has to be a collaborative effort.
Because if you just have a bunch of engineers,
they might build something super cool that
is totally unusable.
Exactly.
Or like, you know.
Unethical.
Yeah.
So it's important to get everyone's perspective.
OK.
So we've talked about a lot of internal things
that companies need to do.
Yeah.
But one of the articles you sent me also mentioned third party
risks, which I thought was really interesting.
Yeah.
That's a big one.
So can you talk a little bit about that?
Like what is that?
So third party risks are essentially
even if your organization has strong AI governance
practices, you could still be exposed to risks
if the vendors that you're working with
don't have those same standards.
So it's like you're only as strong as your weakest link.
You're only as strong as your weakest link, exactly.
OK.
So let's say you're a bank and you're
using a third party vendor for customer service chatbots.
OK.
And that vendor might be using an AI model that's
trained on data that has bias.
Right.
Even if your AI system internally is perfectly fine,
you're still going to be at risk.
Right.
Because you're using this other system.
Right.
Yeah, that's tricky.
It's very tricky, yeah.
So how do you even know to ask these questions of your vendors?
Yeah, well, I think including AI governance requirements
in your contracts with your vendors
is a good place to start.
So making it really clear upfront what your expectations are.
Yeah, like what AI models are you using?
What data are you using to train them?
Right.
How are you mitigating bias?
So it's really about having those conversations early on.
Early on.
And then, of course, ongoing monitoring.
OK.
You have to make sure that your vendors are actually
adhering to those standards.
So trusting but verify.
Exactly.
Trusting but verifying.
Got it.
And this can involve things like document review, audits,
or even running tests on the AI system itself.
OK, so this is getting very complicated.
It can be very complicated.
It's a lot to think about.
It is a lot to think about.
And it sounds like these third-party risks
are something we really need to be paying attention to.
You really do, because AI is constantly evolving.
Yeah.
And new risks emerge all the time, so you have to stay informed.
Wow, OK.
So it's like a constant process of learning.
Yes.
OK, so we've covered so much today.
I feel like I have a much better understanding
of what AI governance is.
Yeah.
Maybe we can just take a second to kind of summarize
what we've talked about.
Yeah, good idea.
So number one, AI governance is a lot more than just data
governance.
Right.
It's really about all these different things,
like transparency, ethics.
Accountability, compliance, risk management.
Yeah, it's a lot.
It is.
And then, you know, we talked about how data governance is
so important, because the quality of the data
that you're putting in really affects the results.
The results.
And then we also talked about model discovery
and why it matters to know what model you're using.
Exactly for what purpose.
Yeah.
Transparency is key.
And then we got into those five pillars,
which I thought was really helpful.
Really good frame rate.
And kind of break down all the things
that we need to be thinking about.
Yes.
And then, of course, this idea of the Wild West.
The Wild West, yes.
Which is a little bit scary.
It is a little scary.
But it sounds like there are things that organizations can do
to kind of wrangle that.
And then, of course, these third party risks, which I think
is something that we often forget about.
We do, yes.
So it's important.
Yeah, it's a lot.
But I feel like we have a really good foundation.
I think so, too.
For understanding AI governance.
Yes.
Now.
Good place to start.
Yeah.
It really is a lot to consider, isn't it?
Yeah.
You know, one thing that I think is really interesting
that Matter brings up is this idea of really understanding
where the AI is being used within a company.
Right.
It's one thing to have these high level policies in place.
But you need to actually know the specifics of,
is it being used in marketing?
Is it being used in finance?
So it's not just a company having an AI ethics statement up
on their website.
It's actually knowing where is this stuff happening.
Yeah.
You have to know which teams are using, which tools, what
data are they using?
Like what are they actually using it for?
Yeah.
Because if you don't have that, it's kind of like you're
just trying to steer a ship, but you don't know where you are
on the map.
You don't know what direction you're going in.
So it's like, even if you have this great AI governance team
and you have these wonderful policies,
if you don't know where the rubber meets the road,
then it's not going to be very effective.
That's not going to work.
Yeah.
OK, can you give me a specific example
of what could go wrong if you don't have this granular level
of understanding?
Yeah.
Let's say your marketing team is using an AI tool
to personalize email campaigns.
OK.
But they don't realize that the data that is feeding into that
tool contains sensitive customer information that should not
be used for marketing purposes.
No.
Right.
So then you have a potential privates for violation.
That's a problem.
Yeah.
And reputational damage.
Yeah, that's a big one.
And if you don't know where the AI is being used
and what data it's accessing, it's
really easy for those oversights to occur.
So it's really all about having the systems and processes
in place to keep track of this stuff.
Yeah, it's not just about the intentions.
It's like, how are you actually going to implement this?
Yeah.
How are you going to monitor this stuff?
So this kind of connects back to the model discovery
that we were talking about.
It's like, you not only need to know what model you have,
but you need to know where it's being used.
Yeah, where it's being used and for what purpose.
Yeah.
It's all connected.
And I imagine this becomes even more important
when you start talking about third party risks and vendors.
Absolutely.
Because it's one thing if it's your own company,
but if you're sharing data with other people
and you don't even know what they're doing with it,
that can be really scary.
Right, because even if you have great data security practices
in place, if you're sharing that data with a company that
doesn't, you're evolving.
It's vulnerable.
It's like having a really secure house,
but then you just give your key to a random person.
Right, exactly.
Like, that's not very helpful.
It defeats the purpose.
OK, so we're working with all these vendors.
We're sharing all this data.
How do we even know what questions to ask them?
Right, so you want to ask them things like,
what AI models are you using?
Where did you get the data to train them?
What are you doing to make sure that it's ethical?
How are you mitigating bias?
Yeah, so it's really like having those open and honest
conversations.
Exactly.
With those third parties to make sure
that everyone's on the same page.
Yeah, it's not just about checking a box.
It's like actually understanding what's going on.
Right, because they might have a great AI ethics statement too.
But if they're not actually doing the work,
then it doesn't really matter.
Exactly.
OK, so this is really about a deeper level of.
Yeah, it's about the trust.
Trust and understanding.
And communication.
Yeah.
OK, and this kind of brings us back to Roy's point
about diverse perspectives.
It's not just the engineers who need
to be in the room.
We need the ethicists, the legal experts, the data
scientists, all these different people.
And the end users.
Right, the end users.
Because if you just have a bunch of engineers,
you might end up with something that's really
technically brilliant, but completely misses the mark
in terms of how it's actually going to be used.
It's like designing a car that's super fast and efficient,
but it has no brakes.
Yeah, exactly.
What's the point?
It's cool, but it's not going to work.
So you need all these different people to come together.
Absolutely.
To kind of check each other's blind spots.
Yeah.
I like that it's like a team effort.
It is a team effort, for sure.
And this isn't just about building AI responsibly
within a company.
It's also about building trust with the public.
Absolutely.
Right, because we hear all these stories about AI gone wrong.
And people are understandably, I think, a little bit
hesitant about what this all means.
It's scary for people.
So if companies are being really transparent and showing
that they're thinking about these things,
maybe we can start to feel a little bit more comfortable.
Exactly, and it's not just transparency.
It's accountability, too.
You have to be willing to take responsibility
if things go wrong.
Right, because they probably will at some point.
Oh, they will.
It's inevitable.
It's technology.
Things break.
So it's more about how do we deal with that?
When it happens.
Yeah, how do we respond?
And I think as we're talking about this,
it strikes me that it's not just about big companies, right?
This is stuff that we as individuals
need to be thinking about, too.
Absolutely.
We're all impacted by AI.
Yeah, think about the algorithms that
are shaping our social media feeds or recommending
products to us or even facial recognition technology.
Like this stuff is everywhere.
And I think it's easy to just kind of let it wash over us.
Yeah, just accept it.
But we should be thinking critically about,
how is this affecting me?
Right, how is it using my data?
Yeah.
What data is it collecting?
And one of the articles that you sent me
mentioned data literacy.
Oh, yeah.
And I thought that was really interesting.
Can you talk a little bit more about that?
What does that even mean?
So data literacy is basically being
able to understand and work with data.
OK.
So it's being able to read it, interpret it, analyze it,
and communicate about data.
So kind of like being able to speak the language of data.
Exactly, yeah.
So if you understand how data works, how it's collected,
analyzed, and used or misused, then you
can understand how AI is making decisions.
OK, so this is about us being more informed citizens
in a world that's increasingly shaped by data.
Absolutely, yeah.
OK, can you give me a specific example
of how data literacy might be helpful?
Yeah, let's say you're using a health app that's powered by AI.
OK.
Data literacy would help you interpret that information
that that app is giving you and make better decisions
about your health.
That makes sense.
Or if you're reading a news article
that's citing a study that was done with AI.
Right.
Data literacy would help you understand,
is that a valid study?
OK, so it's about not just taking things at face value,
but really understanding where did this come from?
Yeah, what are they basing these conclusions on?
Yeah, OK, so how do we become more data literate?
Is this something that we all need to go back to school for?
Not necessarily.
OK.
There's a lot of resources available online, courses,
tutorials, books, articles.
Start by being curious about the data you encounter every day.
What do you mean by that?
So next time you see a chart or a graph in a news article,
don't just scroll past it.
Yeah.
Take a minute to look at it.
What is it showing you?
OK.
What are the units of measurement?
Are there any trends or patterns you can see?
So it's like being more mindful and engaged
with the data that we're seeing all the time.
Yeah, because we see data everywhere.
Yeah, that's true.
And I think, like you said, we just let it wash over us.
Yeah, it's like background noise almost.
Right.
But it's not.
It's important.
Yeah.
It's very important.
And I think, as we're talking about all this, it's like AI
governance.
It's not just this abstract concept.
It's something that affects all of us
in very real ways.
It's not just for organization.
It's for individuals, too.
OK, well, this has been a really great conversation.
I feel like we've covered so much.
We have.
Maybe we can just take a second to recap
the key takeaways from this part.
Yeah, good idea.
So we started by talking about understanding
where AI is being used within organizations.
Yeah, and having a map.
Yeah, having that map.
Of the AI landscape, which sounds really cool, actually.
It is cool.
And then we talked about how this understanding is
so important for like.
For data privacy.
Data privacy, security.
Yeah, especially when you're working with third parties.
Right, because you don't have control over that.
Yeah, you don't know what they're doing.
And then we talked about diverse perspectives and how.
Yeah, we need everyone in the room.
It's not just a one-person job.
It's a team effort.
Yeah.
And then we talked about this idea of data literacy,
which I think is so important.
Absolutely.
All of us as individuals, to start understanding.
Yeah, the more we understand data,
the more we can understand AI.
Yeah.
And how it's impacting our lives.
This is really about empowering individuals
to be more informed and take control of their data
and their interactions with AI.
We need to be active participants.
Yeah, not just passive consumers.
Not passive, yes.
This has been a really great conversation.
It has.
But we do have more ground to cover.
Yes, we do.
Including some specific steps that individuals and organizations
can take.
Yes.
So stay tuned for part three.
I think it's easy to get caught up in all the scary stuff
about AI.
Yeah.
But it's important to remember, it can also be used for good.
Oh, absolutely.
AI has the potential to revolutionize
so many aspects of our lives.
Right, like health care, education.
Sustainability, social justice, you name it.
Yeah, it could really help us solve
some of these big problems that we're facing.
Huge problems.
Like, can you imagine AI helping us develop
new medical treatments?
Yeah, like, that would be amazing.
Or like, personalized learning for every student.
I would be so cool.
Yeah.
So it's really about how do we harness that power?
Yeah, harness that power for good
and make sure that we're not letting it run away from us.
Exactly.
We need to create a framework that
allows us to unlock that potential, but also
safeguard against the harms.
Right, so it's like innovation, yes,
but not at the expense of our humanity.
Exactly, it's all about balance.
Yeah.
And it's a collective effort.
Like, it's not just on governments or tech companies.
Right.
It's on all of us.
Yeah, we all have a role to play.
Absolutely.
In making sure that AI is developed ethically.
Yes, we need to be asking questions.
Yeah.
Demanding transparency.
Yeah, holding these companies accountable.
Yes.
So it's not about being afraid of AI.
It's about shaping it to be what we want it to be.
OK, so as we're wrapping up this deep dive into AI governance,
I want to leave our listeners with a question.
What role do you want to play in shaping the future of AI?
It's an important question.
Yeah, like what questions are you
going to ask in your workplace?
Right.
How are you going to engage with this topic
in your communities?
Exactly, because this isn't just some far off thing.
This is happening now.
AI is already here.
It's affecting our lives.
In so many ways.
And we need to make sure that we're
being thoughtful about it.
Absolutely.
We have to be intentional.
Yeah, and that we're using it for good.
And remember, we talked about data literacy.
Yes, data literacy is key.
Like the more you understand data,
the more you're going to understand AI.
Exactly.
They're interconnected.
So keep learning.
Yes.
Keep asking those questions.
Keep pushing for a better future.
Yeah, a future where AI is a force for good.
Absolutely.
Thanks for joining us for this deep dive.
Yes, thanks for listening.
And we'll see you next time.