Have you ever poured hours into building an automation
workflow? You know, seen it work just perfectly
in your test environment. Yeah, flawless. Only
for it to just fall apart the moment it hits
the real world. Happens all the time, like that
sandcastle analogy you used. Looks great until
the first wave. Exactly. So today we're diving
into how to build systems that don't just work,
but well, systems that survive. Yeah. We're talking
about taking your automations from looks good
on my laptop to bulletproof in the wild. We'll
unpack five core techniques. Five techniques.
Yeah. To make your workflows resilient, visible,
and, you know, truly professional. It's kind
of like giving your automation a superhero cape.
Or maybe a suit of armor. A suit of armor. I
like that. We'll cover everything from centralizing
error reports to smart retries, even backup AI
models. Think of it as a playbook for peace of
mind, transforming those workflows from ticking
time bombs into robust, production -ready powerhouses.
That's the goal. No more midnight alerts, hopefully.
So the core challenge, I feel almost philosophical,
right? Yeah. Building something in a sterile
test lab is, well, Easy compared to the real
world. It is. Yeah. Because the real world is
messy. Users do weird things. APIs go down. Servers
just have bad days. So what's the biggest hurdle
when moving from that lab to production? It's
realizing that production ready doesn't just
mean it worked when I tested it. Not even close.
Right. It means building an anti -fragile system.
Anti -fragile. Okay. What does that mean exactly?
Different from just robust. Yeah. Subtly different,
but important. Robust resists stress, stays the
same. Think a strong wall. Anti -fragile actually
gets better from stress, from errors, like our
immune system. It learns, adapts. Okay. So it
involves what? Handling failures gracefully.
Gracefully, yeah. Without total collapse, you
need instant notification when something important
breaks. Okay. You need intelligent logging, so
debugging isn't guesswork. And crucially, a built
-in plan B, retry logic, fallback logic. Plan
B. Always need a plan B. Absolutely. And it needs
to fail safely. No bad email sent, no critical
data deleted by accident because, look, failures
are inevitable. Right. You can't stop every single
one. Exactly. The job isn't to prevent all failures.
It's to build systems that fail intelligently.
That's the shift. So the core shift in thinking
when aiming for production ready. Yeah. What
is it? It's about expecting failures and building
systems that adapt rather than break. Expecting
failures, building systems that adapt. Got it.
And to do that, we're using this onion or suit
of armor idea, these five techniques, as layers.
Yep, layers of protection. We've got error workflows,
retry on failure, the fallback LLM, continue
on error, and pulling. Okay, let's peel back
that first layer then, or maybe buckle on the
first piece of armor. Error workflows. You said
this is fundamental, non -negotiable. Absolutely
foundational because the big problem here is
what we call the silent killer. The silent killer
sounds ominous. It is. A standard workflow often
fails quietly. Imagine an automation processing
new leads every night. If an API changes or a
credential expires, poof. It could silently drop
leads for days, weeks even. You don't know until
sales complains. Yeah. That's bad. Very bad.
So the solution? A centralized mission control
for errors. Think of it like a single security
desk for your whole NAN operation or whatever
tool you use. A central hub. Exactly. All error
signals pipe back to this one place. How do you
build that mission control? Is it complicated?
Surprisingly simple, really. Two steps. Step
one, create the emergency response team workflow.
Just use as an error trigger node. Its only job
is to listen for errors. Okay. Listening for
trouble. Step two, connect the red phone. In
every single one of your active workflows, you
go into settings and point its error output to
that new error workflow. Like installing an emergency
line everywhere. Precisely. Step three, design
the alert and log protocol. The error workflow
grabs crucial data workflow name, error message,
which step failed, maybe even the input data.
All the content. Right. Logs it somewhere central,
Google Sheet, Airtable, a database, and sends
smart notifications. Slack, email, whatever works
for your team. Okay, I can see that. We had an
issue once. An Airtable credential failed silently.
Took ages to notice from weird reports. How would
this have helped? Instantly. Instead of silent
failure, you'd get a Slack message like, Sat
anan workflow error. Sat workflow. Telegram AI
assistant. Failing node. AI agent. Error. Node
operation error. With a link straight to the
failed run. Wow. Okay. Actionable. Immediate.
Totally changes the game from detective work
weeks later to a fix in minutes. And you mentioned
a pro upgrade, tiered alerting. Yeah, because
not all errors are DEF CON 1, right? True. A
payment failure. Big deal. I have channel alert
now. A minor summary task failing. Maybe just
log it to the sheet. You use a switch node in
the error workflow to route critical errors to
high priority alerts and non -critical ones to
just logging. Keeps the noise down. Smart. So
what's the biggest danger if you skip this whole
error workflow step? Losing data silently and
not knowing your automations are fundamentally
broken. Losing data silently. Yeah, you definitely
want to avoid that. Okay, foundational layer
sorted. Now, technique number two. This one sounds
almost too simple. The turn it off and on again
button. Retry on failure. Yeah, it sounds basic,
but honestly, a huge percentage of failures are
just temporary. Clips, network glitches, server
overload for a second. The hiccups. Exactly.
Amateur workflows just give up. Professional
ones, they retry. It's often the first and, frankly,
most effective line of defense against those
transient things. And setting it up is easy.
Super easy. Most automation tools, like in AN,
have it built into almost every node. Just find
the settings for that node, toggle retry on fail
to on end. Okay. Then you set max tries, maybe
three to five is a good starting point, and...
a wait time between tries, like five seconds.
That's usually it. Two clicks, two numbers. Handles
most of those temporary issues. The vast majority,
yeah. But there's a bit of an art to it, depending
on the task. Ah, strategy. Okay, tell me more.
Well, for calling external APIs, maybe three
to five retries with a five -second delay is
good. Gives their server time to recover. Makes
sense. For AI models, maybe... Two, three retries,
five second delay. If it fails three times, it's
probably a bigger issue than just a blip. Right.
File operations. Those often fail because the
file is temporarily locked. So maybe five plus
retries, but with a very short delay, like one
or two seconds. Okay. Tailored to the type of
potential failure. You mentioned an OpenAI hiccup
example. How does retry actually play out there?
Right. So imagine OpenAI's server gets slammed
for like 30 seconds. Your workflow makes a call,
gets an error. And the amateur workflow just
stops. Dead. Yep. But yours? With retry on fail
set to three tries, five second delay, it fails,
waits five seconds, tries again. Maybe fails
again, waits five seconds, tries a third time.
By now, the spike is over, the call goes through.
And the workflow just continues like nothing
happened. Exactly. The end user or the overall
process is completely unaware there was ever
a problem. It just smoothed itself out. That's
pretty powerful for such a simple setting. And
there's an even more advanced version. Exponential
backoff. Yeah, this is what the big players like
Google and Amazon use, instead of waiting the
same 5 seconds each time. You wait longer. Exponentially
longer. First retry waits 5 seconds, second waits
10, third waits 20. Ah, giving the server more
and more breathing room. Precisely. Especially
crucial for mission -critical APIs that might
be under heavy, sustained load? Whoa. I mean,
imagine the resilience that gives you when you're
handling millions of requests. It's smart scaling.
So for most external services, how many retries
are usually enough? Three to five attempts with
a short delay often solves most transient issues.
Three to five, short delay, good rule of thumb.
Yeah. So retries handle the blips. But what if
it's not a blip? What if, say, OpenAI is just...
Like really down for an hour. Retries won't help
that. Correct. That's when retries run out and
you need the next layer. You need a plan B, a
real backup. The fallback LLM. Exactly. The AI's
backup singer analogy. Your main AI, maybe GPT
-40 mini, is your star. Yeah. But if they suddenly
lose their voice. The show must go on. The show
must go on. So you have another capable AI, maybe
Cloud 4 or Google Gemini, waiting in the wings,
ready to take over automatically. Okay. And setting
this up, is it in the node settings too? Often,
yes. In tools like NNN's AI Agent node, you'd
first make sure retry and fail is on. Then there's
usually a checkbox like add fallback model. Right.
You check that and it lets you connect a second
different AI model. Different is key here. Absolutely
critical. The golden rule is... diversify providers.
Why? Because if open AI is having a major outage,
switching to another open AI model probably won't
help. They might share the same underlying problem.
Ah, same infrastructure. Exactly. It's like having
a backup generator that runs on the same potentially
disrupted power grid. Useless. You want your
backup on a completely different fuel source.
So primary open AI, fallback, anthropic, or Google?
Makes sense. Yeah. Or if you use an aggregator.
Like OpenRouter. Yeah. Maybe your primary is
via OpenRouter and your fallback is a direct
connection to Google Gemini. Bypasses the middleman
entirely. You mentioned testing this by deliberately
breaking the primary key. We did. Gave the primary
AI a bad API key. It failed. Retry kicked in.
Failed again. Expected. Then, automatically,
the system switched to the configured fallback
model Google Gemini in our test. And Gemini processed
the request successfully. The workflow completed.
The end user just saw a slightly longer pause.
No error message. Seamless failover. That's impressive.
Yeah. But you also mentioned a challenge. Prompt
drift. Ah, yeah. It's something I still wrestle
with, honestly, even with fallbacks. When you
switch from, say, GPT to Claude, even with the
exact same prompt. They might interpret it slightly
differently. Exactly. Different architectures,
different training data. You can get subtle shifts
in tone, style, maybe even how it emphasizes
certain points. So the failover might be seamless
technically, but you still need to. test and
maybe tweak the prompts for your fallback model
to ensure the output quality and brand voice
stay consistent. It's an ongoing tuning process.
Good point. Consistency matters. So why is it
so important to use a different AI provider for
the fallback? To ensure your backup isn't reliant
on the same potentially failing infrastructure
as the primary. Different provider, different
infrastructure. Got it. Okay, next layer. Technique
number four. Continue on error. You said this
is a personal favorite for pros? Oh, yeah. Especially
for batch processing. It's a lifesaver. Think
of it as another vital layer of armor. Okay,
so what's the problem it solves? You called it
the assembly line shutdown. Right. Imagine that
content factory again, pulling 1 ,000 leads,
researching each, adding to CRM. What if lead
number three has some weird character in its
data that breaks the research step in a normal
workflow? The whole thing stops dead at number
three. Exactly. One bad apple spoils the whole
batch. 997 perfectly good leads never get processed
because of one tiny error. Huge waste. Okay,
that's inefficient. So continue on error prevents
that. It builds a smart assembly line. Instead
of shutting down, it intelligently pulls that
one defective item off the line for inspection,
while the other 999 keep moving smoothly. How
does that work in the tool? Are there different
modes? Typically, yes. In most node settings,
you'll see error handling options. One might
be just... Continue, which basically ignores
the error and moves on. For low stakes stuff,
maybe. But not ideal. No. The professional option
is usually called something like continue using
error output or similar. This is the game changer.
Oh, so. It doesn't just ignore the error. It
creates two separate paths out of that node.
A success path for items that worked. The green
lane. And an error path for the specific item
that failed. The red lane. Exactly. It isolates
the problem child without stopping everything
else. You tested this with Google, Meta, and
a deliberately broken NVIDIA entry. Yep. Put
quotes around NVIDIA to make it invalid JSON.
Without continue on error, it would process Google
Meta, then crash on NVIDIA. Stop. Right. With
continue using error output enabled. Google and
Meta processed fine, went down the success path.
NVIDIA hit the error. And instead of stopping.
It got routed cleanly down the error path. The
workflow itself kept running for any subsequent
items. 99 .9 % success, 0 .1 % oscillated for
review. That's incredibly useful for large data
sets. And the pro upgrade here. Self -correction.
Yeah, this is where it gets really cool. That
error path doesn't just have to go to a log or
a notification. It can trigger more automation.
Exactly. It can lead to its own mini workflow.
Maybe it tries sending the failed NVIDIA item
to a different AI model with a simpler prompt
or uses a different lookup tool. Trying to fix
the problem automatically. Right. And if that
fix works, the result can then be merged back
into the main success path downstream. That's
peak self -healing. Wow. Okay. So how does...
Continue on error help most with large data sets.
It processes good data efficiently while isolating
problematic items for separate handling. Isolates
the problems. Very smart. Okay. Okay, final technique.
Layer five. Polling. Sounds patient. Oh, it is.
This one's key for asynchronous tasks. Stuff
where you ask for something and the answer is
an instant, like generating a complex report
or a big AI image. Right, things that take time.
What's the problem without polling? The agony
of guess and wait. You kick off a job then? What
dough? Add a wait node for five minutes, ten
minutes. You're just guessing how long it'll
take. Exactly. Guess too short. Your workflow
tries to grab the result before it's ready. Failure.
Guess too long. You're just sitting there wasting
time and resources. It's fragile. So polling
is the fix. The pizza tracker analogy. Perfect
analogy. You order a pizza. You don't just stare
at the door guessing when it arrives. You check
the tracker app. Making. baking out for delivery.
Polling is that tracker for your automation.
It asks the service, are you done yet? Are you
done yet? And only moves on when the answer is
yes. So how does that look in practice, say for
AI image generation? Typically a few steps. Step
one, initial request. You send the POST request
to start the image job. The service replies with
like a task and in status. Queued. Okay. Order
placed. Step two. Initial wait. Don't pull immediately.
Give it a reasonable time to start. Maybe a wait
node for 40 seconds. Let the chefs start working.
Step three. The status check loop. This is the
core. It's usually a loop containing an HTTP
request node to get the status using the task
id, an IF node to check if the status is still
processing or if it's completed, and another
wait node, maybe 20 seconds before checking again.
So it keeps checking every 20 seconds. Yep. Get
status. Is it completed? No. Wait, 20 is get
to status again. It repeats until the IF node
sees completed, then the loop breaks, and the
workflow continues with the finished image data.
Clever. Are there best practices for this? Golden
rules. Four main ones. One, set a reasonable
initial wait. Don't hammer the API immediately.
Okay. Two, use sensible check intervals. 15,
30 seconds is often good. Don't check every second
unless the API docs say two. Be polite to the
server. Exactly. Three, always, always have a
maximum retry limit on your loop. An escape hatch.
What if the service breaks and never reports
completed? You need the loop to stop eventually,
maybe after 10 tries, and go down an error path.
Prevents infinite loops. Crucial escape hatch.
Got it. and four understand the api's status
vocabulary read the docs does it say processing
running pending does it say completed succeeded
done you need to know the exact words to check
for read the manual okay and there's an alternative
to polling web hooks yeah the more modern often
more efficient way web hook callbacks how's that
different with polling your workflow keeps asking
are you done with a web hook callback When you
make the initial request, you give the service
a unique URL, your NAN webhook URL. You basically
say, call me back at this address when you're
finished. So the service calls you. Yep. Your
workflow then just sits at a wait for webhook
node doing nothing until the external service
sends the completed result back to that URL.
No loop, no constant checking, much cleaner if
the service supports it. More efficient. So what's
the main benefit of polling over just... guessing
wait times it ensures you proceed only when data
is truly ready avoiding premature failures ready
and waiting makes sense mineral mentioned refer
to separate script okay so we've covered these
five specific techniques these layers of armor
but you mentioned there's a broader approach
to the guardrail mindset Yeah, it's kind of the
philosophy that ties it all together. Because
fundamentally, you don't know what you don't
know. Meaning? Production environments are chaotic.
You'll encounter weird data, unexpected API responses,
edge cases you never dreamed of during testing.
You can't predict everything. So the mindset
is about? Being proactive and learning from the
chaos. It's a three -step process, really. First,
log everything. Use that error workflow. Maybe
add more logging. Capture every error, every
weird pattern. Second, identify patterns. Don't
just let logs pile up. Review them regularly,
maybe weekly. Look for common issues. Is a certain
type of input always causing trouble? Is one
specific third party API flaky? Find the recurring
problems. And third, build targeted guardrails
based on those patterns, create specific fixes.
Like we saw lots of failures because an AI was
outputting slightly malformed JSON. So we built
a guardrail. a little code node right after the
AI call that specifically sanitizes the JSON
output, fixing common issues before it gets sent
to the next API. Ah, a custom fix for a known
problem pattern. Exactly. That one little guardrail
turned a workflow that failed maybe 10 % of the
time into one that succeeded 100%. That's the
guardrail mindset. Learn from failures, build
smarter defenses. So let's put it all together.
An example, a content research and generation
system. How would all five layers work here?
Okay, imagine that system. Layer 1, error workflows.
All errors go to a central log. Critical failures
ping Slack immediately. Visibility, check. Layer
2, retry on failure. Every external API called
research tools, AI models, is set to retry three
times with a 15 -second delay. Handles blips,
check. Layer 3, fallback LLM. The primary AI,
say GPT -4A -mini, has Google Gemini Pro configured
as its automatic fallback if it fails consistently.
Plan B for AI, check. Layer four, continue on
air. If the research step fails for one specific
topic out of 20. It doesn't stop the other 19.
Right. That one failed topic gets routed to a
separate manual review list. The rest continue.
Isolates problems. Check. Layer five, polling.
After the AI generates the content for each successful
topic, which might take a minute, a polling loop
patiently waits for the completed status before
saving the content and moving to the next topic.
Waits intelligently. Check. All five layers working
together. Yeah. The result isn't just a script
that runs. It's an anti -fragile system. It handles
partial failures, gives you visibility, maximizes
the work that actually gets done successfully.
That makes a lot of sense. So wrapping up, the
core big idea really seems to be failures aren't
just possible. They're going to happen. They
are inevitable. Yeah. And the professional edge
isn't building systems that never fail because
that's impossible. It's building workflows that
fail intelligently. That's it exactly. Achieving
resilience, visibility, and what we call graceful
degradation. Failing partially, but effectively.
Amateur scripts might look good in tests, but
break easily under pressure. While professional
workflows assume failure and handle it gracefully.
That's the difference. And that leads to actual
peace of mind. You trust your systems. And the
implementation. People don't have to do all five
layers at once, right? No, definitely not. Start
with the error workflow. That visibility is key.
Then add retry on failure to most nodes. That's
usually easy. Okay. Then maybe the fallback LLM
for your critical AI steps. Then look at continue
on error and polling where they make sense for
your specific flows. And always keep that guardrail
mindset review logs, build targeted fixes. Start
simple, layer it up, analyze and adapt. We really
hope this deep dive gave everyone a clear roadmap
for making their automations truly robust. What
stands out to you most about building these resilient
systems? For me, it's the shift from hoping things
don't break to knowing you have systems in place
to handle it when they inevitably do. We encourage
everyone listening, just pick one technique,
start there, add that first layer of armor. Even
that small change can make a huge difference.
Thank you for joining us on the deep dive. Yeah,
thanks for listening, and remember, the real
Pro Edge, it's deploying automations knowing
they can handle the messy real world, letting
you kind of forget about them, and most importantly,
sleep soundly at night. OTRO music.