I assume you have a mini ready to go. I don't
have one ready to go because you didn't tell
me how you were feeling today. To be fair, you
didn't ask. How are you feeling? Okay, thank
you. Yeah, you're welcome. I am feeling like
an AI update. Again? The last one we got wasn't
really AI, it was robots. It was too, it was
too. Don't even call into question what it was.
Okay, I only have one more in here. So let's
see if it's filed properly. And I feel like it's
bordering on news of another category. So this
one is from zdnet .com. Never heard of it. Written
by Tiernan Ray, who is a senior contributing
writer, April 18, 2025. AI has grown beyond -
The world of artificial intelligence has recently
been preoccupied with advancing generative AI
beyond simple tests that AI models easily pass.
The fame Turing test has been beaten in some
sense, and controversy rages over whether the
newest models are being built to gain the benchmark
tests that measure performance. The problem,
say scholars at Google Deep Mind Unit, is not
the tests themselves, but the limited way AI
models are developed. The data used to train
AI is too restricted and static, and will never
propel AI to new and better abilities. In a paper
posted by DeepMind last week, part of a forthcoming
book by MIT Press, researchers proposed that
AI must be allowed to have experiences of a sort
interacting with the world to formulate goals
based on signals from the environment. Quote,
incredible new capabilities will arise once the
full potential of experiential learning is harnessed."
Write DeepMind scholars David Silver and Richard
Sutton in the paper. Welcome to the era of experience.
The two scholars are legends in the field. Silver
most famously led the research that resulted
in AlphaZero, DeepMind's AI model that beat humans
in games of chess and Go. What's go? It's a Chinese
game that has black and white pieces. It's widely
known that it has like basically infinite different
setups of how it can play out. So it's supposed
to be one of the hardest games to actually map
out for a computer. More than chess? Yeah, infinitely
more. Oh, weird. It sounds not like something
I would want to play. Sounds difficult. I get
that. Even though it was fairly vague. Sutton
is one of the two Turing Award winner developers
of an AI approach called reinforcement learning
that Silver and his team used to create AlphaZero.
The approach the two scholars advocate builds
upon reinforcement learning and the lessons of
AlphaZero. It's called STREAMS and is meant to
remedy the shortcomings of today's large language
models which are developed solely to answer individual
human questions. Silver and Sutton suggest that
shortly after AlphaZero and its predecessor AlphaGo,
first on the scene, generative AI tools such
as ChatGDP took the stage and discarded reinforcement
learning. That move had benefits and drawbacks.
Gen .AI was an important advance because AlphaZero's
use of reinforcement learning was restricted
to limited applications. The technology couldn't
go beyond, quote, full information. end quote
games, such as chess, where all the rules are
known. Gen. AI models, on the other hand, can
handle spontaneous input from humans never before
encountered, without explicit rules about how
things are supposed to turn out. However, discarding
reinforcement learning meant something was lost
in this transition, an agent's ability to self
-discover in its own knowledge, they write. Instead,
they observe that LLMs, which is, let me just
remind you. Yeah, good memory. Rely on human
prejudgment or what the human wants at the prompt
stage. That approach is too limited. They suggest
that human judge imposes an impenetrable ceiling
on the agent's performance. The agent cannot
discover better strategies underappreciated by
the human radar. Not only is human judgment an
impediment, but the short, clipped nature of
prompt interactions never allows the AI model
to advance beyond question and answer. I always
feel like AI should be in a question and answer
phase, shouldn't it? Like it needs a prompt from
somebody to move on, lest it be able to take
its own steps outside of that question answer
dialogue. Yeah, that's exactly what I was just
thinking. You would think it would need a prompt
in a form where it's on a computer, but what
What if you have AI running a robot that's in
a race? Where they weren't in what we saw on
the mini. However, what if it was AI? What if
it could run on its own? Right? Wouldn't it run
without a prompt then? I would hope it would
still need a prompt. Yeah. Go run this race.
And I guess then it would run the race to completion.
Yeah. Until it needed, yeah. But no, I just I
like the idea of having a prompt for it to respond
to as opposed to doing its own thing Yeah, it
sounds like we're getting a little too advanced
in my mind. Yeah, and that's what the article
is about. I'm pretty sure. In the era of human
data, language -based AI has largely focused
on short interaction episodes. For example, a
user asks a question, perhaps after a few thinking
steps or tool use interactions, and the agent
responds, the researcher writes. The agent aims
exclusively for outcomes within the current episode,
such as directly answering a user's questions.
I am just using quotations willy -nilly in this.
I'm very sorry. There should be a lot of quotations
in here that I'm just not using, so I hope it's
making its way across. I just don't want to say
quote. It feels like too much today. I had a
bee attack recently. We basically had the deep
state trying to stop us. basically. All in all,
I'll continue now. There's no memory. There's
no continuity between snippets of interaction
and prompting. Quote, typically, literate know
information carries over from one episode to
the next, precluding any adaptation over time,
end quote, writes Silver and Sutton. However,
in their proposed age of experience, quote, agents
will inhabit streams of experience rather than
short snippets of interaction, end quote. Silver
and Sutton draw an analogy between streams and
human Silver and Sutton argue that, quote, today's
technology, and quote, it really adds to it when
I put the quotes in, is enough to start building
streams. In fact, the initial steps along the
way can be seen in developments such as web browsing,
AI agents, including OpenAI's deep research.
quote recently a new wave of prototype agents
have started to interact with computers in an
even more general matter by using the same interface
that humans use to operate a computer and quote
they write the browser's agent marks a transition
from exclusively human privileged communication
to much more autonomous interactions where the
agent is able to act independently in the world
as AI agents move beyond just web browsing they
need a way to interact and learn from the world,
Silver and Sutton suggests. Are they suggesting
that it evolves? Is that what I'm getting from
this? I don't know if... Essentially... They
mean evolve. I think they're just meaning that
they need to be able to do this. The robots.
No, like the companies who own the robot. They
propose that the AI agents in streams will learn
via the same reinforcement learning principle
as AlphaZero. The machine is given a model in
the world in which it interacts, akin to a chessboard
and a set of rules. As the AI agent explores
and takes actions, it receives feedback as rewards.
These rewards train the AI model on what is more
or less valuable among possible actions in a
given circumstance. The world is full of various
signals providing those rewards. If the agent
is allowed to look for them, Silver and Sutton
suggest. Quote, where do rewards come from? If
not from human data, once agents become connected
to the world through rich action and observation
spaces, there will be no shortage of grounded
signals to provide a basis for reward. In fact,
the world abounds with quantities such as cost,
error rates, hunger, productivity, health metrics,
climate metrics, profit, sales, exam results,
successes. visits, yields, stock, likes, income,
pleasure, pain, economic indicators, accuracy,
power, distance, speed, efficiency or energy
consumption. In addition, there are innumerable
additional signals arising from the occurrence
of specific events or from features derived from
raw sequences of observations and actions." To
start the AI agent from a foundation, AI developers
might use a world model simulation. The world
model lets an AI model make predictions test
those predictions in the real world, and then
use the reward signals to make the model more
realistic. Quote, as the agent continues to interact
with the world through its stream of experience,
its dynamics model is continually updated to
correct any errors in its predictions. End quote,
they write. Silver and Sutton still expect humans
to have a role in defining goals for which the
signals and rewards serve to steer the agent.
For example, a user might specify a broad goal
such as improve my fitness, and the reward function
might return a function of the user's heart rate,
sleep duration, and steps taken, or the user
might specify a goal of help me learn Spanish,
and the reward function could return the user's
Spanish exam results. The human feedback becomes
the top level goal that all us serves. That's
what I was just wondering. That sounds incredibly
intrusive. What if it's not getting those rewards?
Is that, I don't know anything about how AI is
set up. I don't know. I mean, my AI - It's just
us living around in the hardware, or sorry, in
a warehouse somewhere. As much as I shouldn't
be giving away our secrets, my AI would be rewarded
by our high level profits on each episode. Which
is why we're multi -million dollar enterprise.
Yes. Please still give us money. We need it so
bad. That is why we do all the episodes such
as we do about corporations because we are one.
There was a red herring out there so that you
guys are off the trail. We really fooled you.
We really fooled you. We're trying to throw those
fucking orcas off our yachts. Yeah, I don't know.
I don't know how AI works. I'm certainly not
giving my AI treats. A big thing that I'm worried
about with how they were describing this is they're
talking about like a lone AI doing this, but
that wouldn't be the case. It would be multiple
different AIs potentially doing this at the same
time, which likely would then mean they're interacting
out there without human interaction with each
other, which is literally just the dead internet
theory. that the internet's just going to be
robots at the end. I've never heard this theory.
We can do an episode on the dead internet theory
at a different time, which is really coming to
life. I would really like to do that. This does
keep going on a lot. So I hope you get the point.
I don't know that I fully wrap my head around
this. You guys got 48 hours and then you can
just move on with it and just learn about the
stuff that we're teaching you on Friday and just
forget that this other one exists. I mean, I
probably will. Yeah.