Hey everyone, welcome back to another episode
of Data and AI with Mukundan, the show where
we dive deep into tools, stories and breakthroughs
in the world of data science and artificial intelligence.
So whether you're on your morning walk, prepping
for your interview in your car or building something
amazing from scratch, I hope this podcast helps
you think a little sharper and ask better questions.
And build smarter. Now, let me ask you something.
Have you ever opened a data set and thought to
yourself, I have no idea where to begin. You're
basically at work. And you're asked to look at
some data. And someone tells you like, hey, this
is, I mean, you started a new job here. They're
asking you to look at the data set that you're
going to be working with all the time. But all
they say is get familiar with this data. But
you don't know how to really get familiar because
I guess you don't know what questions to ask.
Where to even begin. And you're not alone in
this because I'm sure every person who is starting
a new role goes through this. in fact this exact
moment actually inspired me to write a blog post
first on this which gained a lot of traction
and a lot of people i felt could relate uh because
i mean in in the seven days or something that
i've that i've written the blog post it gained
like about i think 500 plus you know read time
on it and a read time is 500 plus reads so that
would mean that roughly people are spending 30
plus seconds per read on average right so it
was getting a lot of traction and it has been
my most successful blog happy to say that so
that's why I wanted to create this episode which
I felt like this is something people face a lot
at you know whatever part of the journey they
are especially if they're in the data field right
your data scientist data analyst data engineer
whatever you just uh you're faced with this challenge
that you're asked to look at data but really
don't know what questions to ask um so i i created
a tool that i uh you know i felt like could be
a real game changer especially for me and i think
it could be for you as well So buckle up because
today I'm going to be walking you through how
I built an AI tool that thinks like a data analyst
or maybe a data scientist. And not only that,
but I also want to unpack the problem that it
solves, show you how the app works behind the
scenes and share how you can use it right now
to level up your own analysis. So what's the
real struggle behind an exploratory data analysis?
So let's be real, right? Exploratory data analysis
or EDA isn't as straightforward as textbooks
make it sound. Textbooks may say that, you know,
load the data, check for nulls, data types, you
know, check for the data types, like what the
different data types present. and it could tell
you like to do some histograms to see what to
the count frequency and everything and by textbook
i mean like let's say you did actually do maybe
a course in data analysis data scientist or or
you went to school for it for your undergrad
or your master's or your phd program or whatever
right i'm sure not at the phd level but definitely
at the master's level you are doing more eda
then you would be doing at the PhD level for
sure. The drill is basically that you load the
data, check for missing values or outliers really,
data types and the EDA, right? The actual data
visualization where you look for the counts.
But real -life data work, when you're in the
actual workplace, it is way messier. So you're
given a CSV with no documentation, no context,
and vague instructions. Could be like, find something
interesting in this data set, right? And this
could be something like in interviews, you're
doing a take -home exam, for example. You're
working with a new client and maybe you're in
a consulting company and you're working some
new client project. and they'd ask you to find
something interesting and maybe that's a very
quick turnaround time as well so how do you how
do you be super fast in this right um or maybe
just uh your even in your internal stakeholder
meetings they'd be asking you to do this and
and the other thing which i spoke about was also
onboarding right when you're starting a new company
role they might ask you to look at data set not
everybody has too much time to help you understand
the data set uh maybe you're doing in a startup
role they they still just give you the data and
to uh look at it right and that uh and the turnaround
time that you would have to you know produce
results from it would be very quick so so you
may be just struggling with the orientation aspect
and it's not that you're struggling with the
analysis it's just the way you've been told to
do it um but here's the thing you don't you don't
need to clean the data every time you just need
to ask better questions and so which brings me
to the next section here the spark what what
really inspired me right um so i was doing a
take -home assignment For a company that I really
admired. And they gave me a data set of car listings.
And they asked me to explore it. So there was
like no specific KPI, no business goal. Just
tell us something useful from the data set. And
even though I did all the standard stuff, right?
Looking at some price distributions, missing
values, correlation plots, nothing popped up.
Then it hit me. Maybe it's not an analysis problem.
It's just a question problem. And let me elaborate
what that means. Like, I needed prompts that
could nudge me. Like, you know, asking the right
questions from the data, right? That's what I
meant as a question problem. So in this case,
it was like, do accident -prone cars sell for
less? Does city influence price? Are new cars
listed for shorter durations? So what I wanted
was something that could think like an analyst
with me. And that's when I started building this
tool. yeah obviously it could have helped me
during the interview but I think the main goal
of this was to maybe set me up for success for
future interviews and if it's a great tool share
it with others to help them along with their
process as well right so this this is what the
app does really like you upload a CSV that's
step one step two you mean the the AI tool summarizes
each column with the data type unique values
that are present in the data set and ranges examples
and it builds a custom prompt for GPT -4 and
finally GPT -4 then returns ten thoughtful exploratory
questions so actually the summarization of the
columns that is a code logic that's being written
but gpt4 where the ai comes in is it returns
these exploratory questions that you can ask
from the dsa so they are not shallow summaries
they are very nuanced very context aware and
they're very useful in real workflows so like
i said right before with these examples so imagine
you just joined a team and you get handed raw
data to look at so instead of freezing up you
get like a head start here because this ai tool
is doing that analysis for you i mean it's asking
those questions like what questions you should
be asking so that you are then um maybe using
a sql code a python code or whatever other right
uh way to explore the data set But now you know
what questions to ask because this tool is already
giving you that head start. It's giving you a
relevant way to frame the questions. And it's
giving you an instant analytical momentum. And
that's where the magic happens. So let's look
at behind the code, which I'll be sharing in
the blog post which accompanies. you know, this
podcast. So it'll be in the show notes for anybody
wondering. So for my fellow builders and, you
know, code tinkerers, I want to call it, here's
a quick walkthrough of the backend code. So the
app itself, it's built with Streamlit. So it's
a Python -based library and it's for the UI.
It's a very easy library to use. OpenAI is GPT
-4 API and Pandas for data profiling. So once
a user uploads a CSV, the app parses the columns.
It categorizes them by data type and generates
human readable summaries. So it's easy to understand
those summaries right then it feeds those summaries
into a gpt4 prompt like this so the prompt goes
like this right you are a data analyst here's
a data set summary what are 10 smart questions
we should ask to understand this better and gpt4
then does its magic we show the output in the
app you copy paste the questions or you can use
them to guide the dashboards that you build from
it the analysis that you want to do from this
in your take -home exams in your site projects
client side projects your interviews anything
really right like you're onboarding with a new
company so a lot of use cases but yeah you know
what questions to ask now because well uh the
tool is doing that for you so you're just focusing
on the analysis right so this is your guide now
um who is this for so this is for you if you're
a new hire who's ramping up on mexi data sets
um like i mentioned so even if you're even if
you're a student who's practicing eda for your
school projects This could be something that
guides you there. And let's say you're a data
analyst who's building portfolio projects for
your interviews so that you can get interviews
in the future, right? So this is very handy.
Or now, let's say you got the interview and it's
like a take -home exam and they ask you to still
explore the data set. This is where you can use
it. you're a freelancer scoping up raw data for
clients. So let's say you're just on a client
contract and they ask you to look at raw data
to generate meaningful insights. This is where
the tool comes in. So even senior analysts can
use it to spark inspiration first. Even if you're
like maybe an established person in a company,
I think even if you're looking at something new,
for the first time and again don't know what
questions to ask this can give you that jump
start right it's not a replacement for deep thinking
it's a partner for better thinking so you still
want to be able to think deep but also have a
better thinking partner right So one of my favorite
use cases for this so far has been one newsletter
subscriber reached out to me last week and they
were prepping for an interview and they were
asked to paste in a dataset. I mean basically
they were asked to look at dataset and generate
questions. So what they did was they used this
tool which I built in. they were able to generate
questions because the app generated for it for
them and that this isn't something that they
even considered so they picked like three or
four of those 10 questions which were generated
they ran an analysis in the ada basically and
built a slide deck which said that helped really
gain her gain their confidence back right like
i mean i i felt like they were a bit stuck here
and this this kind of tool really helped them
get unstuck so my purpose of sharing this is
just showing what's possible when we go from
you know getting blank screen in our minds to
more structured curiosity. So I wasn't sharing
this to hype the tool, but just to show what's
possible to do, right? And that's what this tool
really helps you to do. Here's where I'm taking
it next. So as I mentioned, I will be sharing
the show notes, the blog post that accompanies
this podcast. And it will be in the show notes,
which will have the full code of how to do it.
And I mean, I explained the process. if you want
to be able to do that yourself you can do that
based on the process as well and what's next
is like i will be doing a csv upload with column
profiling and using a custom tone a stakeholder
aware gen question generation as well custom
tones as in like the way you want the data set
to sound So what that means is like, if you want
to ask more emotional questions from the data,
so something like that would be an ideal next
step for this. So like make it, make it smarter
essentially. Right. And I want to do it like
a Slack bot or a chat GPT plugin. So I think
Slack allows you to, or maybe at least chat GPT
allows you to do it. directly but the idea is
to do it like not just as a data uh not just
as a data upload you know in a in chat jpt this
allows you to use an app and i think apps are
cool so um that's why i wanted to do this and
i want to add some visualization support to this
some more data visualization support more charts
to look at smarter charts to look at and so that's
like some of the you know use cases I'm thinking
this is just the beginning if this resonated
with you try it out paste in a data set that
you have struggled with and let the AI give you
that head start. And if it gives you even one
useful question, let me know. Tag me, message
me, share the output. Because better questions
build better insights. And better insights build
better careers. So thanks for listening to another
episode of Data and AI with Mukundan. And until
next time, keep thinking, keep questioning, and
keep building.