Ever felt like you're just sort of guessing with
your AI, you know, tinkering, adjusting things,
holding your breath, just hoping it's actually
getting better, hoping it matches what you envisioned?
What if we could move past that intuition, past
I think this is better? What if you could say,
I know this is better, and here's the data, the
actual proof to back it up? Yeah, ditching the
gut feelings for cold, hard facts. That's really
the superpower we're diving into today. It's
a fundamental shift, actually. Welcome to the
Deep Dive. Today we're unpacking a really crucial
topic, how to evaluate your AI workflows, specifically
within environments like NAK and moving away
from those subjective feelings towards objective
data -driven decisions. This isn't just about
small tweaks, you know. It's about building systems
with real precision, a much more reliable approach.
We're going to explore why traditional testing
methods often fall short when you're dealing
with AI's unique complexities. We'll definitely
get into the critical importance of what we call
a gold standard data set and walk through some
really illuminating real -world examples so you
can see this in action. And then, yeah, give
you a step -by -step guide to setting up your
very own evaluation system. Think of this deep
dive as your personal guide. Your path to becoming
a workflow wizard. Someone who crafts solutions
with proven precision. Not just someone who,
well, tinkers and hopes for the best. It's a
game changer for how you approach automation.
And what you can achieve, really. So let's start
with maybe a stark image, one we're calling the
medieval doctor problem. It's about why... For
too long, many of us have been kind of flying
blind when it comes to optimizing AI. You know
the drill. You build an AI workflow. It's not
quite right. You think, OK, if I just change
this one thing, maybe we'll get her. So you change
it, you run it, and then you just sort of feel
if it's better or worse. You repeat this loop
until you're either totally frustrated or you
just settle for good enough. It really is like
that medieval doctor prescribing leeches. Lots
of confidence, maybe, but absolutely zero data
backing up the decisions. Exactly. And that's
such a perfect analogy because, you know, in
the probabilistic world of AI, your feelings
can genuinely deceive you. And the problem with
that guess and check approach just leads to wasted
time and honestly unreliable AI outcomes. Decisions
aren't based on facts. They're based on subjective
feelings, which, well, we know are notoriously
unreliable with AI output. Workflow evaluation,
though, that offers. objective proof. It tells
you exactly what works and what doesn't. It turns
those educated guesses into properly informed
decisions. So why does this guess and check approach
really hinder progress in AI? Well, simply put,
it leads to wasted time and unreliable AI outcomes
because decisions are based on subjective feelings.
Okay, let's unpack this a bit more. The difference
between testing traditional code and AI is absolutely
fundamental. Think of it like a glass box versus
a black box. Traditional code is the glass box.
It's deterministic. You put 100 identical inputs
in, you'll get 100 identical outputs every time.
You can see everything, know exactly how it works.
It's transparent, predictable. Right, but AI
models... They're the black box. They're probabilistic.
Give them 100 identical inputs and you might
get 100 slightly different nuance variations.
It's kind of like asking 100 different human
experts the same question, you know. This complexity
comes from evolving models, that inherent probabilistic
nature, and the fact that you're often optimizing
for multiple goals at once, like accuracy, cost,
speed. Because of all this, you really need a
full dashboard of key dials. Think of it as performance
for accuracy, reliability for consistency, efficiency
for speed and cost and then quality for that
sort of subjective goodness. So if AI is this
opaque black box, why is isolating variables
so crucial for testing? How do we even figure
out what's working? The golden rule. It's truly
the core of scientific testing here. Isolate
your variables. Think about it. If you tweak
the prompt and change the model and adjust the
temperature all at once, then maybe your accuracy
goes up great, right? But you have no idea what
actually caused that improvement. Was it the
prompt, the model, some weird combination? Right,
you're lost. Exactly. Isolating variables precisely
identifies which specific changes improve or
harm your workflow's performance. Otherwise,
you're just guessing again. What's really fascinating
here, and maybe the absolute foundation of this
whole process, is your gold standard data set.
This is your source of truth. And honestly, your
evaluation is only as good as the data you feed
it. A brilliant test system with, well, garbage
data. It just produces garbage results. Simple
as that. I remember struggling with that, trying
to convince stakeholders why building out a really
robust test data set was worth the upfront effort.
It felt like extra work then. But it truly is
the bedrock, isn't it? Your evaluation data is
like your perfect measuring stick. So what makes
a good gold standard data set? Well, it has to
be accurate, obviously, with undeniably correct
answers. Consistent, so no contradictions in
there. Comprehensive, covering every scenario
you expect. Representative of real -world usage,
that's key. And crucially, full of... Edge cases,
those weird, tricky examples most likely to break
your system. Only edge cases are critical. I've
seen workflows handle 99 % of inputs perfectly,
only to completely fall apart on one unusual
phrasing, those edge cases. They're gold for
finding the real weak spots. Exactly. And where
do you find this data goldmine? Often it's right
there in your own company's history. Think about
high -quality support tickets with perfect resolutions
or expert responses to common questions. Even
top -performing marketing content could be useful.
But honestly, the absolute best source for validating
both your test data and the AI's output quality,
your subject matter experts. SMEs, the people
who did the job manually before the AI, they
know the nuances. That SME connection is truly
invaluable. But then the question always comes
up, OK, how much data is enough? For early testing,
maybe 50 to 100 examples are a decent start.
Get your feet wet. For production readiness,
though, you'll probably want 250 to 750 examples
for statistically significant results. And for
those really mission critical systems where accuracy
is just non -negotiable, you're looking at a
thousand examples, maybe more. Big numbers sometimes.
Pro tip here. Start collecting this data months
before you actually think you'll need it. You
will thank yourself later. Definitely. So what's
the biggest risk if your gold standard data?
isn't truly gold well flawed data doesn't just
give you bad results it gives you confidently
wrong results you end up building your whole
strategy on quicksand thinking you're making
progress when you might be optimizing for the
wrong thing or maybe making things worse it leads
to misleading evaluation results and ultimately
just bad decisions all right enough theory for
a minute let's get our hands dirty the best way
to understand this power is really to see it
in action so let's walk through a couple of real
world any and evaluation scenarios First up,
imagine an email tagging agent. Goal is simple.
Read incoming emails, tag them with a specific
category and a priority level. Okay, so the setup
for this experiment involves a test dataset,
let's say just six emails, each with a known
correct category and priority. Then we use NANN's
evaluation nodes, which are like these pre -built
components designed to compare AI output against
a correct answer, to run the test and measure
the results against those known answers. Pretty
neat. And the first run reality check was, well,
a bit of a disaster, honestly. Priority accuracy
was mediocre, maybe 57%. But category accuracy,
a big fat 0%. Zero. Wow. Now, the diagnosis was
immediately clear. The AI had no system prompt
to guide it at all. It was just making up its
own category names, like billing issue, instead
of the required category billing. So every single
category test failed because of this, well. Fundamental
oversight. Right. Makes sense. So a simple fix
then probably made a huge difference. Monumental.
A clear system prompt was added to the AI node,
giving it a constrained list of the exact categories
it was allowed to choose from. The final results.
Category accuracy jumped from 0 % to a perfect
100%. That's amazing. 0 to 100. It really highlights
how the simplest, most systematic fixes discovered
through evaluation can have the biggest impact.
It's often not about super complex engineering,
it's just about clarity and constraints. Totally.
So, okay, second example, a bit more complex,
the FAQ response agent. The goal here is read
a customer email, find the relevant info in an
FAQ database, and then craft a helpful natural
language response. The big challenge here is
that the output is subjective, right? You can't
just check if two long paragraphs of text match
exactly. How do you measure things like helpfulness
or tone? This is where it gets really interesting.
How does using a second AI as a judge help with
those really subjective tasks? Well, the solution
is using a second AI. AI to act as an impartial
judge. You feed this evaluator AI the original
email, the known gold standard answer from your
test set, and your agent's generated response.
The evaluator's only job is to provide an objective
quantitative score, say, on a one to five scale
for the quality of the agent's response. It provides
an objective quantitative score for quality where
exact text matches just aren't possible. It's
a super useful technique. And this led to a model
showdown. with some surprising results, didn't
it? The initial hypothesis was something like
Google's Flash model will be faster, but the
more expensive GPT -40 Mini will be more accurate.
Makes sense on the surface. Yeah, that was the
guess. But the data delivered a twist. GPT -40
Mini scored a pretty mediocre 3 .5 out of 5.
And yeah, it was slower and more expensive. Google
Flash, on the other hand, delivered a much stronger
4 .3 out of 5. It's like a 23 % improvement in
quality. Plus, it was twice as fast and significantly
cheaper. So the verdict was clear then. Crystal
clear. The cold, hard data proved that the cheaper
and faster alternative was actually the superior
choice in terms of quality. Without the systematic
data -driven evaluation, most people would have
probably just stuck with the more famous, more
expensive model, assuming it was good enough.
This shows a real, tangible, competitive adage
you can get from evaluation. Mid -roll sponsor
read, provided separately. So you're probably
sitting there wondering, okay, how do I actually
set this up for myself? We've designed what we
call a final exam system. It's a practical four
-step guide to building your own AI evaluation
setup. Step one, design your exam paper. This
is your test data set. You'll create a simple
Google sheet, maybe, with clear columns for your
input data, like the email body you want to test,
and the expected correct answer, say, the specific
category you want the AI to output. Keep it simple.
Right. Then step two is building your testing
room using NEN's evaluation nodes. The workflow
is pretty straightforward. An evaluation trigger
node loads all those test cases from your Google
Sheet. That data then flows through your AI workflow
to get processed. Finally, a set of evaluation
nodes records the AI's actual answer and directly
compares it to the expected correct answer from
your sheet. It's basically setting up the proctors
for your AI student's exam. And for step three,
you run the exam and analyze the grades. execute
this evaluation workflow, NEN provides a really
detailed report card. This includes the overall
accuracy percentage, a breakdown of performance
on each individual test case, and other key metrics
like execution time and API costs. It's all right
there, laid out for you. Actionable information.
And this brings us to maybe the most critical
step for long -term improvement. Step four, keeping
a lab notebook. You know, a good scientist always
keeps a detailed log. You should create your
own testing log. Maybe a Google Sheet or Notion
page works great. For every single test run,
document what you changed. The prompt, the model,
the workflow step, whatever. Why you changed
it, what was your hypothesis, and then the final
results. The new accuracy, speed, cost. Honestly,
this documentation is pure gold. I have to admit,
I still wrestle with prompt drift myself sometimes.
Or I accidentally change some tiny variable I
didn't mean to, and then my results are just...
Chaos. But the lab notebook always brings me
back. Ah, prompt drift. That's when even a really
subtle change in your instructions can completely
alter the AI's behavior in unexpected ways. It
really highlights just how sensitive these systems
can be, doesn't it? So beyond just getting the
results, why is documenting every single change
so vital? Because it builds institutional knowledge,
it prevents you from repeating the same errors,
and it ensures systematic long -term improvement.
Simple as that. Okay, now that we know how to
set up the system, let's talk about the different
measurement tools in your toolkit. Think of this
like a doctor's diagnostic kit again. You wouldn't
use a stethoscope to analyze a blood sample,
right? You need the right metric for the right
job, a specific tool for a specific problem.
That's an excellent analogy. First up, we have
categorization metrics. Think of this as your
stethoscope. This is perfect for tasks that involve
putting items into predefined buckets like email
tagging, content classification, maybe sentiment
analysis. It works as a simple exact match comparison.
Did the AI pick the right category? Yes or no?
Pretty straightforward. Then there are correctness
metrics. These are maybe more like a blood test.
These are perfect for subjective generative tasks,
like evaluating the quality or factual accuracy
of a written response. This is where you use
that AI evaluating AI technique we just talked
about. The evaluator AI provides an objective
one to five score, perhaps, for the response's
correctness and helpfulness. It adds that layer
of objective judgment to inherently fuzzy tasks.
Next up. Similarity metrics. Let's call this
your MRI scan. This provides a deep, nuanced
comparison. It's perfect for tasks where the
goal is to match a specific style or tone or
format. This metric measures how close the AI's
output is to a known gold standard example from
your data set. It's all about stylistic adherence.
And finally, there are custom metrics. The specialist
test may be, these are for any unique business
-specific requirements that the standard metrics
just don't quite cover. You define the criteria
and the scoring system yourself, often by building
out... specific logic using something like an
N8n code node, which is basically a small block
of custom JavaScript right inside your workflow.
So why do we need so many different metrics,
like a stethoscope versus an MRI scan? It's simply
because different AI tasks demand different diagnostic
tools. Categorization is just a binary check,
yes or no. Correctness is about subjective quality,
but scored objectively. Similarity looks at stylistic
nuance. Each one helps you understand a different
dimension of your AI's performance. It allows
you to accurately measure and diagnose its strengths
and weaknesses. Okay, cool. So you now have the
core framework. Let's move into the professional
playbook. Some advanced tips. troubleshooting,
and a clear action plan that will really elevate
your evaluation game. First, the pro level playbook
has three golden rules. The consistency principle
is totally non -negotiable. Keep your evaluation
model consistent across all tests. If you have
an AI judging another AI, don't change the judge
AI midway through your testing. If you do, all
your comparisons become invalid. Absolutely crucial.
Then there's the documentation imperative we
talked about that lab notebook. N8n shows you
the results, sure, but not what you change to
get them. A simple Google sheet tracking your
changes, hypotheses, and results is key to systematic
improvement. It becomes your institutional knowledge
base. And finally, the iteration strategy. Start
small. Don't try to boil the ocean. Maybe 10
to 20 examples just to validate your setup is
working. Then scale up to 5 ,100 for more serious
testing and maybe 250 or more for production
-ready systems. It's a gradual, deliberate climb.
Right. And of course, out in the field. you'll
run into issues it happens so here's a quick
field guide for troubleshooting some common things
if the built -in set metrics node is giving you
errors or not doing what you need try creating
your own custom evaluation agent instead you
have that flexibility if your evaluation results
seem inconsistent run to run you're almost certainly
changing multiple variables at once remember
the rule focus on one change at a time And if
your test data doesn't seem to reflect real -world
results, your data set probably isn't representative
enough. You need to collect data over a longer
period, maybe include more of those tricky edge
cases. You know, this isn't just about tweaking
your workflows a little bit better. This is fundamentally
about transforming your entire approach to AI
automation. The before state is guesswork, frustration,
settling for good enough. The after state, it's
data -driven decisions, continuous improvement,
and a real durable competitive advantage, faster
optimization. better cost control, proven quality
assurance, you actually know it works. Whoa.
Imagine scaling this precise evaluation method
to hundreds of complex workflows across an organization,
ensuring every single interaction is top tier,
perfectly aligned with business goals. That's
a massive competitive advantage right there.
So let's give you your mission briefing, a seven
-step action plan to get you started today. One,
choose your first evaluation target. Pick a workflow
that's maybe almost good enough, but could be
better. 2. Create a small test data set. 20 to
50 examples is fine to start. 3. Set up the basic
NADN evaluation nodes to get your baseline measurement.
4. Run your first test and document that initial
performance in your lab notebook. 5. Make one
single change. Just one. 6. Run the test again
and compare the results. See what happened. 7.
Iterate and improve based on that data. And remember,
you can often find complete workflow templates
and even test datasets in online automation communities
to help jumpstart your journey. Don't reinvent
the wheel. So thinking about all that, what's
the single most impactful takeaway for someone
looking to get started right now? Start small
with one workflow, build a small dataset, and
really commit to documenting every single change
you make. That's the foundation. Today, we've
walked through how to transform your AI workflow
optimization process, moving it from an art of
guesswork, really, to a science of data -driven
decisions. It's a profound shift in mindset.
Yeah, from recognizing that medieval doctor problem
to understanding the black box challenge. crafting
those gold standard data sets and applying precise
evaluation metrics, you now truly have the tools
you need. It's all about moving from, I think
this works better, to I know this works better,
and here's the data to prove it. That evidence
-based approach, that conviction based on facts,
that's the real key. So don't just hope your
AI works. Start knowing it works. Pick a workflow,
gather some data, and begin your first evaluation.
The power is really in your hands now. The difference
between an amateur and a professional AI automator.
It often lies squarely in this shift. This is
how you build AI systems that truly deliver on
their promises. Systems you can justify. Systems
that give you a real tangible edge. Yeah, it's
time to stop reading about it and actually start
evaluating. Make that leap. We really hope this
deep dive has given you clarity and conviction.
Until next time, keep digging, keep learning,
and keep building smarter. Otiero Music.