Have you ever spent weeks engineering an AI agent
only to have it confidently deliver an answer
that was just demonstrably catastrophically wrong?
It's the paradox of this current AI generation,
isn't it? The delivery is perfect, but the information
is just terrible. You ask for Q4 revenue, you
get a number off by a factor of 10. Or you ask
for a contract summary, and it misses the one
single liability clause that matters. Exactly.
It's a hallucination, but it's delivered with
this conviction of a seasoned expert. Welcome
back to the Deep Dive. Today, we are tackling
that exact engineering frustration. For anyone
out there building agents, the secret we found
is that the flaw, well, it's rarely the language
model itself. No. The problem is almost always
rooted in how we feed the data to our systems.
Which is what we're calling context engineering.
Precisely. It's the whole process of ensuring
your retrieval augmented generation, your RAG
system, has the exact information in the right
format at the precise time it needs it. So RAG
is basically giving your LLM some external knowledge
to keep it grounded. Yes, and our mission today
is to help you transform your agents from these
confident guessers into truly verifiable analysts.
OK, let's unpack this. We need to move beyond
those intro tutorials that always seem to lead
to these frustrating failures. For sure. We're
going to cover four powerful, specific solutions.
We'll start with a deep dive into why the basic
default methods fail so badly. Then we'll get
into the reliable methods, starting with structured
data simple filters and SQL queries. And finally,
we'll define when you should actually use full
context retrieval or when that traditional vector
search is still the right tool for the job. Let's
do it. All right. So when you first start building
these RIG systems, the default method everyone
learns is chunk -based retrieval. It's usually
powered by vector search. Great. You take a massive
document, you cut it up into these little pieces,
these chunks. And then the AI only gets to see
a handful of them. maybe the three or four things
are most relevant. It's like stacking scattered
Lego blocks of data and just hoping they form
a perfect blueprint. That standard, that fragmented
approach, it basically guarantees three critical
issues. The first one is the most common, missing
context. The little fragments, they might contain
keywords, sure, but they lack all the surrounding
narrative. The setup, it's like, if you were
summarizing a 100 -page book and I only gave
you pages 12, 45, and 88. You'd have no idea
what's going on. You couldn't possibly grasp
the main character's motivation. You'd miss the
crucial setup from chapter one. The LLM, when
it's faced with these disconnected pieces, it's
just forced to guess. Yeah, and I can offer a
vulnerable admission here just based on my own
early failures. The second issue is what I call
the math wall. Oh, yeah. Language models are,
I mean, they're incredible text processors, but
they are fundamentally terrible at complex, verifiable
math. They're calculators that are bad at math.
Exactly. I once asked my internal agent for Q3
sales figures from this sprawling 2000 line report.
It pulled four records that happened to contain
the word total. And it just added those four
up. It summed those four lines and presented
that tiny result. as our company's entire quarterly
revenue. And that is the core absurdity, right?
The agent calculated just a tiny fraction of
the necessary data, presented the result with
99 % confidence. And you have to go use external
tools just to realize it's completely broken.
It happens because that chunking process, it
prioritizes what sounds similar over actual data
integrity. Right. And that leads to the third
flaw, which is the problem of bad summaries.
Yeah. So you need a comprehensive chronological
summary of a 50 page technical manual. If your
system just randomly hands the AI four paragraphs
from all over the document. The summary is going
to be useless. It's like trying to write a detailed
review of a three hour documentary after only
watching like 30 seconds of the trailer. You
lose the plot. the sequence, everything. So if
we accept that these math errors and logic gaps
are so easily created by fragmentation, how should
a developer even approach verifiable accuracy
at this point? Well, math errors are just inherently
masked by the LLM's confidence. You have to force
the calculations onto external specialized computation
engines. OK. So let's pivot. Let's talk about
the first real solution, moving away from all
this guesswork. Right. And this one is specifically
for structured data. So data that lives in neat
rows and columns, think an internal dashboard
or even just a Google sheet. Instead of asking
the AI to search and guess, we teach it to operate
more like a database admin. We teach it to filter.
And this approach relies on what's called function
calling, or defining these very specific tools
the AI can use. We don't ask it to read the data.
We ask it to construct a precise query. So for
a sales database, for example, we define functions
like product name query or date query. And the
technical depth here, the really important part,
is defining that tool's specification. You have
to dictate the exact inputs. So for instance,
requiring the date to be strictly in that yyymmdd
format. That input validation is so crucial.
It is. And we had to give the agent a core system
prompt, an instruction that says, always use
the most specific tool available for the user's
intent. And that instruction, it turns the agent
into this highly disciplined engineer, not some
creative writer trying to guess what you mean.
Exactly. So consider a real -world query. How
many wireless headphones were sold in the Northeast
region on 2025 -0915? The agent doesn't search
a vector database. It sees the query. It identifies
the need for filters. And it just chains the
tools together. So it uses product name query,
then region query, then the date query to narrow
everything down. And then finally, it hands that
small, perfect data set to a dedicated calculator
or computation engine to just sum the results.
The outcome is 100 % accurate. Because the heavy
lifting was done by precise data filtering, not
by linguistic guesswork. Right. But I should
say, for listeners who might not have a deep
programming background. Yeah. The simplicity
of that outcome, it really masks the complexity
of the setup. I mean, defining those tools and
making sure they don't suffer from prompt drift,
where the agent just starts ignoring your instructions
over time. I still wrestle with that myself.
It's a constant battle. It absolutely is. So
what makes implementing these specific filters
superior to just using generalized chunking,
even for really large structured data sets? Filters
provide targeted data access, which eliminates
the risk of data fragmentation and guarantees
100 % verifiable calculation inputs. OK, so filters
are great for selection. But what if you need
more complex calculations, things like generating
averages or standard deviations or sorting complex
reports. Yeah, the AI is going to fail those
manual calculations every single time. And this
is where we bring in the language of databases.
SQL. So think of this as giving your ARAGAY LLM
a superpowered calculator that's optimized for
massive data operations. Exactly. The process
is pretty elegant. The LLM writes the SQL code,
maybe a complex group PO by Y function for a
Postgres database, sends that query to the actual
database engine. And the engine just returns
the single precise result. But the critical engineering
detail. The part you cannot get wrong is in the
system prompt. You have to include the exact
table schema. So you have to explicitly tell
the AI, hey, there's a table named sales data,
and then list every single column. Customered,
INT, total price, float, order date, date. Because
if you omit the exact column names or the data
types, the AI just guesses. And that leads to
immediate execution failure. Right. And that's
why this is a step up in complexity from simple
filtering. You're actually relying on the LLM
to write code. And you have to instruct it with
absolute clarity. Never attempt math manually.
You force it to delegate the calculation using
functions like sum, abg, and group pby. You're
pushing all that mathematical load onto the database.
That raises a critical point, though. If we're
having an LLM write code, doesn't that open us
up to huge security concerns like SQL injection
vulnerabilities? Isn't it much slower than just
fetching the data? That is a crucial technical
challenge. Yes, the developer has to implement
really strict input sanitization. And you typically
have to limit the types of queries the LLM can
even construct to prevent malicious injection.
But you use SQL for speed on complex aggregation,
not simple filtering. When a user asks, who are
our top three customers by total spending this
year? The SQL script processes millions of rows
instantly. Ah, OK. So you need that power when
a simple filter just won't cut it for the complex
sorting and counting. Exactly. We let the database
do the heavy lifting for absolute reliability.
So why does providing the specific column names
and data types prevent the most common SQL failures
in these agents? Without the exact schema, the
AI guarantees query failure by producing type
mismatch errors or ambiguous column references.
Makes sense. Now let's move back to unstructured
data. We're talking about those long legal contracts,
technical manuals, or sprawling video transcripts.
Sometimes you really do need the AI to understand
the entirety of a source, not just little fragmented
parts. modern large context windows, like 128k
tokens, this idea of full context retrieval is
not only viable, but it's often the most accurate
option. Right, you're giving the model a huge
memory, the full text. But there are two different
ways to do this and understanding the cost differences,
well, it's key. Method A is the simple prompt
method. You just copy and paste the entire document
into the AI agent's static system prompt. But
the cost drawback of that simple method is immediate.
And it's heavy. Yeah. Since that long text is
just hard -coded into the prompt, you pay for
every single one of those tokens, the entire
document, every single time the agent runs, even
if the user just asks, hello, how are you? You're
just burning tokens for no reason. Exactly. And
that's why we really recommend method B, the
tool -based method. We wrap the entire document
inside a tool definition. It's the difference
between forcing an employee to carry every file
in their arms all day. versus just giving them
a key to the filing cabinet. Maximum savings,
on -demand access. The agent only fetches and
reads that full, expensive document if the user's
query specifically triggers that tool. So if
the user asks about the document, the agent gets
the full content instantly. But if they ask about
the weather, it saves the cost. And the performance
increase is just stunning. We saw a test where
standard vector search created this messy chronological
nightmare from a video transcript. But full context.
Because it had the entire script, it produced
a perfect sequential and comprehensive summary.
So if accuracy on a single document is paramount
and it fits within that context window, this
is the choice. Whoa. Imagine scaling this precise
chronological retrieval to a billion queries
across complex legal documents without losing
a single key detail. The analysis potential is
just immense. It guarantees context continuity.
So given the cost and the speed trade -offs,
why should we always avoid that simpler prompt
method for long documents? Avoid the prompt method
because you will incur unnecessary token cost
when the user's query does not require accessing
the document. Okay, so... We've been a little
tough on chunk -based retrieval, but let's be
absolutely clear. Vector search is not a bad
technology. Not at all. It's just deployed inappropriately,
like 90 % of the time. Vector search or chunk
-based retrieval, it remains the undisputed king
for one specific thing. Massive scale. It's the
library card catalog analogy, right? You use
vector search when you have 10 ,000 documents,
a massive library of employee handbooks, old
support tickets, internal memos. You simply cannot
afford to feed the AI all of them using full
context, and they're way too unstructured for
filters or SQL. And the beauty of it is the indexing.
It embeds all that text into these numerical
representations, stores them in a vector store.
And can rapidly retrieve the pages that are numerically
closest to the user's question. It helps you
find the right book or the right page within
the right book really quickly. So the distinction
is crucial. You use vector search when the answer
could be anywhere in this sprawling library.
It's amazing for the find the needle in the haystack
kind of query. But it's fundamentally terrible
for analytical tasks. Things like summarize everything
or count the total revenue. So of all these powerful
context methods, what unique capability does
vector search offer that the others just can't
replicate? VectorSearch excels at rapidly indexing
and retrieving relevant information from massive
sprawling libraries of unstructured documents.
Okay, so that brings it all together. How do
we decide which of these four methods to use
in a real -world scenario? Well, the decision
framework is actually incredibly simple. We should
always start with this question. How would a
human knowledge worker answer this exact question?
OK. So if a human would naturally use an Excel
filter, a query like, show me all orders from
yesterday where the product was blue, then you
have to use filters. Right. And if a human would
need a pivot table or a complex calculator, a
question like, what's the average lifetime spending
of our top 10 % of customers, then you must use
SQL queries. And if a human would need to read
the whole file sequentially, like summarize this
40 page contract, noting all termination clauses,
they use full context retrieval. And finally,
if a human would need to search through a massive
filing cabinet, if the answer to, how do I reset
my password, is buried among thousands of old
user guides, then you use vector search. Okay,
but what about the layered complexity? What if
you get a query like, summarize the Q1 marketing
strategy document for new customers in Texas
who spent over $500? That is where context engineering
really earns its name. the agent has to chain
these things together. First, it uses the filter
tool to identify the relevant customers. Texas
greater than $500. Then it uses the SQL tool
to aggregate and sort their spending trends.
And then finally. Finally, it uses the full context
retrieval tool to fetch and summarize that single
marketing strategy document that was mentioned
in the query. It's like a chain of custody for
information. That layered approach ensures you
get maximum accuracy at every single step. So
finally, maybe a few essential technical warnings.
Yes. Don't be lazy with your system prompts.
Give your tools descriptive, specific names that
tell the agent exactly what they do and what
data format they require. And never ever combine
structured data like numbers, dates, or precise
inventory counts with unstructured text. blindly
in a vector database. Keep that structured data
separate. If you mix your math with your text,
you are guaranteeing the mathematical analysis
will fail. That feels like the core lesson here.
I think so. The main takeaway we've distilled
today is that building reliable RRAG LLMs, it
isn't about finding the perfect model. It's all
about context engineering. It's about ensuring
the AI has the exact, verifiable information
in the right format at the precise time. We move
past all the guesswork and into reliable, accountable
analysis. And remember, the best agents are polyglots.
They use a mix of these methods. You might filter
to narrow down a custom race, then use full context
to read their contract, then use SQL to report
on their spending. That layered accuracy really
is the new standard. So here's your final thought
for reflection. Look at your current AI workflows.
Ask yourself if you're using that generalized
vector search where a simple precise filter or
a structured SQL query would give you 100 % more
accuracy and auditability. Try changing just
one tool in your pipeline this week. Go build
something amazing with this knowledge and we
look forward to the next deep dive with you.