This is the Convergent Science Network podcast. Leading researchers in the domain
of neuroscience, brain theory and technology are interviewed by Paul Verschoor and Tony Prescott.
This is Paul Verschoor with the Convergent Science Network podcast.
And today I'm with Mark Toussaint, who's a speaker in our summer school.
And Mark, as a physicist, has been presenting his important work in the domain of machine learning.
So Mark, you started your presentation with like a little test for your audience,
where you showed them a few equations and you want to check out who in the audience
would recognize these equations.
So why were these equations so important to you? Well, first just to check what
kind of background people have.
I'm not so sure if they're so important, but I think they are important to eventually
understand what the point is of these formulations that I presented and to sort
of understand that the third equation on that slide is something that we can
get out of these new formulations. Okay.
And actually, I think you looked a bit disappointed because it was not a large
group of people who immediately jumped up like yes i know this one so yeah there is something um,
i mean reinforcement learning is a very very basic model of
how behavior could be organized and how behavior could be
organized in a way that is goal directed and um
although you know there is many many questions whether this helps you
understanding the brain of course there's many questions but still
this should not um i don't know free you of knowing these
things i think it would be good to teach them yeah absolutely um
now what you initially emphasized were
your your methods for let's say to understand planning as a form of inference
yeah so why do you think that that's a useful way to to think about planning
first it's an it's an alternative way to think about planning,
which is a good thing in itself.
I think, you know, turning over problems and looking at the same problems from
different perspectives is important.
I think the perspective of looking at planning from the perspective of planning.
You know, offers different, you know, options of how to think about a computation
that actually realizes planning.
So more concretely about why planning is inference, I think it does allow you
to be creative to use other kinds
of computational algorithms that exploit different types of structures.
To be more concrete, if you think about how to do planning on factored representations,
one way of thinking, the more classical one, would be to have structured representations
of the value function, but it will always be the value function, right?
Whereas if you roll out that process and sort of formalize it as the whole model
as a graphical model and show that inference in a graphical model can solve
planning problems, it really leads to different types of algorithms and different types of,
approximations that you could use to solve planning.
But would that mean you really have removed the value function from the story,
or you have redefined it?
That's also very interesting. That's a good question.
So I have removed it in the computational algorithm itself so that the algorithm
does not have to have explicitly a representation of the value function in its
memory or the computer doesn't have to.
But in some cases, in particular on normal Markov decision processes,
the algorithm will compute things which are one-to-one to the value function.
For instance, on normal Markov decision processes, the backward messages are
equivalent to the value function.
But again, all of this is only true on a normal Markov decision process.
If you go to the actually interesting problems where you cannot do exact inference
or exact value iteration, like in POMDPs or other kind of factored things.
The messages do not anymore correspond to value functions. They really correspond
to probabilistic posteriors.
And it would be unclear how to do the same things that it could do with inference,
how you could do the same things with value functions.
Right. So in that sense, also if you go to the Bellman equation view,
which would be more dominated by a value function, then for you this is also,
let's say, a contrast, right? is okay.
That's one way to look at planning. And you want to look at alternative models to solve it.
That might be more simple or are they just different formulations of the Bellman view?
They are different. I think they're different to the Bellman view.
So the Bellman view is really a backward.
The Bellman equation is a backward thinking.
So if you have already solved a small problem, you can sort of backward iterate
and sort of move your horizon backward and solve the larger problem and so forth.
So in the Bellman view, it's not very symmetric. It doesn't think about,
for instance, forward messages or forward information.
The inference view does. It computes both, forward and backward messages.
And the result is then a product of both of these being the posterior.
So I think it is a different view than the Bellman view. But I could argue that
I might, let's say, describe your inference-based approach in terms of a Bellman equation.
I can show an equivalence or is there really something missing?
I don't think that the computational things that are happening during message
passing, you could describe as being a form of Bellman equation.
So computationally, the things that are being computed and the kind of messages
that are being computed are different.
As a result, they will also solve somewhat different problems?
Well, they lend to different approximations. For instance, as I said, in a POMD piece.
Well, in POMD piece, sometimes you can do exact inference as well.
But if you do message passing, you can cope differently with large problems
or with factor problems as you
could do with trying to approximate value functions and iterate over them.
Right. So now you mentioned that that's sort of introductory remarks,
but you mentioned that on the one end, you found problem-solving from the perspective
of inference theoretically interesting,
but also you said, look, it can give us a handle on a more large-scale integration
and possibly on some form of multidisciplinary interaction.
Right. So what's the perspective there exactly? So the,
I think generally, actually, probabilistic modeling and also graphical models,
there's two things about them.
Some people say they're great tools because they work well, but another thing
to say about them would be they're nice because they offer one nice language
that can be applied on many different disciplines.
So graphical models can be used in linguistics, in robotics,
for inference, for whatever.
It's a bit of a similar thing that I wanted to state with planning as inference.
The first point to make is that it's one language that allows you to talk coherently
about both decision-making, doing perceptual inference, sensor processing, while also planning.
So not only immediate decision-making, but also planning in one coherent set of concepts.
And it's actually a rather small set of concepts, just inference,
right? I think this is good for communication between disciplines.
That's really one point. And as I said, one example of that is that having these
concepts that it can show, that they explain planning and other things,
allows to build bridges between disciplines.
As I said, there is neuroscientists, and I don't have a known opinion,
but there is neuroscientists who like to argue that neurocomputation can be
abstracted as functionally doing something as Bayesian inference.
Or to say it even weaker, that neurosystems could at least also realize something as Bayesian inference.
But that is an interesting statement because if now in other fields we can show
that Bayesian inference can solve planning problems, we immediately have a hypothesis
of how neurosystems could solve planning problems.
But this is also a goal in your own work? I wouldn't really say so.
So particularly in the last years, I became more and more of a roboticist, really. And, um.
So in a positive sense, yes, I am inspired. And I'm inspired by the questions
of how actual systems like the brain can solve problems of goal-directed behavior,
manipulation, all of these things.
But it would be too much to say that I see myself as a researcher,
like a brain researcher.
Future right okay so then um let's
say the the approach that you have a great confidence in at this point in time
is what you call stochastic optimal control which is a problem definition not
an approach yeah so but why why
do you think this is a an important set of problems then to to deal with.
That's again a good question. So let me say what I guess many people would,
other people would say, right?
Stochastic optimal control is a problem definition similar to Markov decision
processes with reward and it's a very general one.
People like when things are being formulated general and problems being formulated
general and allows to sort of you know, describe all kinds of planning problems in there.
And it's well, of course, accepted within the control theory itself.
And it's quite a lot of work on within machine learning in terms of Markov decision processes.
Let me put it the other way. If I would not have shown and embedded my theory
to be important or useful for solving MDPs and optimal control,
I could have not convinced the whole control theory community or machine learning
community that it's worth anything. Okay.
So how did you then solve this problem? Solve which problem?
Well, stochastic optimal control.
You're saying, look, you try to show that your framework is applicable in that domain.
So in that sense, I guess you want to have an impact that sort of also delineates
your proposal from others.
So what are the unique features of what you propose in that context?
Why did it work better or why was it more appreciated than alternatives? Right.
Um, I mean, one of the, I mean, achievements, I think, is a theoretical achievement
in the sense that we could show that many, many other algorithms,
which themselves have, you know, proven also,
um, I mean, empirically that they work well, that there are special cases of
the formulation that we have.
So this is a purely theoretical statement, right?
Algorithms 1 to 10 are special cases of the firmware. Right.
And algorithms 1 to 10 had done some effort, fortunately for us,
already to show that you're computationally okay.
So that's one thing. The other thing is that we could also derive one algorithm,
which is the model-free reinforcement learning algorithm, that I find conceptually
quite interesting and that we could also directly compare it to other algorithms.
If we now first look at the first part of this argument where you say,
well, we had the most fundamental formulation of a solution to this set of problems.
What are the key ingredients of that solution?
I'd say that the key ingredients have already previously been formulated by
people like Kappen and Opper and so on.
Um the key ingredients is uh to think about optimum control as you know formally
it's a minimization problem um but matching two different distributions the
one being the control one and the
other one being the uncontrolled but conditioned one so these key ideas um,
Yeah, have previously actually already been formulated. And I must say it's
also not, I mean, you could even go back and talk about Kalman duality in general.
Even Kalman already said that the problems of control and the problems of filtering,
like state estimation or Bayesian filters, are very similar.
And what Bert Kappen did before us, and we now also do, just explicates exactly
that relation. and makes it explicit.
Now, the difference in the formulation is really that we're talking about processes
over state control and not really making any assumptions about the dynamics
or control costs or noise in the process,
whereas the previous formulations, they discussed processes on the state and
for that reason had to do some special assumptions to actually get the theory properly.
So in some sense, you had a more reduced formulation of the problem.
You just left out a certain number of aspects that you saw as being irrelevant
really to the solution. You could put it like this.
So how should I imagine the solution that you came up with for these kinds of problems?
How does this problem solver really operate?
I think most concretely, you could imagine it in terms of that policy that I
described, which is actually described by Boltzmann energy.
So you actually have a policy which is represented by a Boltzmann energy,
which is not really any assumption about the policy, because any conditional
distribution can be represented as a Boltzmann distribution.
Well, and then the iterative solutions that we proposed translate to iterative
updates of that Boltzmann energy.
So concretely, I should say that these iterative equations are actually equations
which perform that Karpig-Leibner minimization that originally the theory proposes should be done.
So these updates of the Boltzmann energy are concretely the argument that has
to be implemented eventually.
Right, but now basically it means I have a set of policies over which I want to optimize, right?
They all have their own level, their Boltzmann energy attached to it,
right? And now you're going to update these iteratively.
So that means that you are sampling from some set of states that might describe
the task that this policy has to be applied to.
So to get this iterative function to work reliably, what should be the properties
of the states I'm sampling over?
Can I just follow any distribution or should it be very regular?
So first I want to say that we have a set of policies, which is one,
but being described by a distribution.
In the exact update where we assume we have a model, we update that Boltzmann
energy over the whole space of points.
So the whole function we update for all x, for all states.
That is the model-based case or the stochastic optimum control case where we assume to have a model.
I think what you refer to is when you ask, but for which states do I update
it or which states do I have to sample?
This is the model-free case, right? Where the system really has to interact with the environment.
Well, in that case, the system would have to unroll episodes of experience with the environment.
And these episodes of experience give you samples from the process.
And you can use these samples of the process as a standard, also with TD and
Q-learning and so forth, to do the necessary update of the Boltzmann energy in a stochastic way.
So with a learning rate alpha instead of doing the exact update of the energy. Okay.
So the point is, what I'm looking for, what are the boundaries on that solution?
Not only to formulate the solution, but then to prove that it actually will
work and converge, you do have to make assumptions about the space in which
this algorithm operates.
So what would be these limiting factors?
The first one is we only considered observable environments.
There is no partial observability in what we discussed.
I wouldn't really know yet how it would transfer to partially observable environments.
The second thing I think in terms of the,
When we can do these updates exactly, I think we prove convergence without any
further assumptions, right?
So these iterates, they converge.
We prove convergence without further
assumptions. But this is true if the updates are exact of the energy.
So under what circumstances can it be exact? It can be exact if the state and
control space is discrete, because then we can represent the energy just as
tables over discrete states and actions.
In that case, it will converge, no assumptions.
If the state space is continuous or otherwise has to be encoded more compactly,
the convergence is more difficult.
If in the continuous case actually the dynamics is
so simple for instance just linear and credit costs um then you can just make
a quadratic assumption about that energy and it's almost like the recut equations
and so forth you can do the updates exactly and again um it will work um if
the dynamics is non-linear well you would have to use a function approximation
to represent extended energy.
And at that point, exactly, we lose that strict convergence proof and can only
hope really, as so often with reinforcement learning and the function approximation.
Right. So then the other thing, and sorry if you want, a trick that you applied in this approach,
that you in some sense, let's say, completely fragmented your value function,
function right because you now added a variable to your system that was locally linked to every state,
that was it's not a reward function that relates to that state and if i understood
it correctly these reward functions do have to satisfy certain regularities
for for this system to operate correctly or or not what what what's the price
you pay by distributing your value function in this way hmm.
I'm not sure. I mean, these reward functions, we didn't really put assumptions
on them, except perhaps for a convergence that they are bounded, as also for Q-learning.
So to prove convergence of Q-learning, you would have assumed that they're bounded,
but otherwise there's not really constraints.
But the point was that in some sense, also contrasting again with this Bellman
perspective, where you would have an explicitly defined value function.
In this case, you have a more implicitly defined, it seems, value function.
Or not. I wouldn't call it value function. We have the rewards and we have the
Boltzmann distribution.
Nothing more. But you could argue that in iterating these distributed value
states, you are approximating something like a value function.
Function. In iterating the update of the Bellman, not the Bellman,
sorry, the Boltzmann energy.
So what I'm after is just to say that you seem to draw a distinction between
a value-based approach and your approach where you say it's not value-based in some sense.
What I'm saying, well, but maybe implicitly you are value-based,
but now you just have sort of hidden it more by putting a distributor in the
system linked to every single state. Yeah.
So first, yes, the relations in the end become very close. I'll elaborate in a second.
But the distinction is not so much between, or I didn't mean to initially distinguish
between a value-based approach and our approach, but more like an approach which
computes value functions by iterating back the Bellman equation versus by computing other things.
In terms of probabilistic inference, right?
So the Boltzmann energies are eventually computed minimizing couple of club
letter divergences. Fine. Okay.
In the Model 3 case, I derived that one equation like that, the Model 3 reinforcement learning equation.
And you can analyze the fixed point properties of that equation.
I should emphasize it's the fixed point properties, right? It's not the transient
of the learning process itself.
In that fixed point, this Boltzmann energy has very interesting properties.
It's set for non-optimal actions. It goes to minus infinity,
making these actions very improbable to be chosen in terms of action selection.
For the other ones, it turns out in the fixed point, for optimal actions,
it corresponds to the optimal value function, the Boltzmann entity,
which is surprising, but it's actually a simple outcome of analyzing the fixed
point equation of the update.
But now the other thing that was interesting in the approach you described is
that you very strongly relied on the inference component also with respect to
sort of looking at the consequences of future states that the system might anticipate onto its priors.
Right that you sort of as you described yourself as if you just imagine a future
goal and then suddenly you just pretend actually you've achieved it you look
at the consequence that has on the system um.
So what's that dynamic exactly and why did you approach it in these terms?
I mean, I don't quite understand what you mean by dynamic.
I mean, it's a bit like… Well, it's dynamic because I have to now imagine a future state.
I may now include it in my priors and now I can make decisions on that basis
so I can propagate myself now
to future new events, right? And I can go through that same loop again.
Right. Right, so you imagine future events as happening, right?
You condition on them and compute a posterior.
And in the last framework, yeah, you do iterate by using that posterior again
as a prior and condition that again on the same future event and again compute a posterior.
Right, I mean... But do you consider that problematic or not? I don't...
Algorithmic-wise, I don't see why it's problematic. Do you mean problematic
in terms of interpreting how, I don't know, living systems are doing it?
Well, to me it seems that this works because you have conditioned it on a very
specific definition of the problem domain.
So for instance, that I can only have single goals existing at any one point in time, as an example.
Oh, no. Why not? No, no. You can arbitrarily condition your future.
The future can be represented in a factored way, there can be goals in different
representations at different points in time.
But wait, if you now propagate that back into your priors, you might have,
let's say, unexpected conflicts between these goals that you do have to resolve now in some way.
So inferencing graphical models, if there is evidence in a graphical model which
are inconsistent, the likelihood is just zero, of that thing,
and the messages diverge.
And that might, in fact, happen.
And that happens in graphical models if there is almost deterministic dependencies,
and you get these really inconsistencies.
So that's the case when you condition on variables which are observations which
are totally inconsistent.
If there is a chance, a slight chance of consistency, inference will exactly,
generate a compromise. That's the point of it, right? Yes.
So, yeah, if you have deterministic models of our future and we condition too
many things, it will diverge in a sense.
Otherwise, it just behaves as inference in graphical models.
Right. But then, so in some sense, that means I just get points in this graph
structure that have lost their validity.
I do not consider them anymore and I rely on others to make my decisions.
Don't know so I mean inferencing graphical models wouldn't really do that right
it would find a compromise in a probabilistic sense,
another constraint I was worried about is just time like for instance you also
mentioned that you have an interest in mapping this to robotics and if you deal
with robots the key thing is real world real time so for instance,
goal setting might also evolve over different time windows, right?
Or I might get, let's say, goal interrupts because the world interferes with
my own ideal planning world.
So I was wondering how that, how your solution would take these kinds of,
let's call them exceptions, into account, or how robust it would be in the face of that.
So we don't, I don't know if you have so much experience.
Certainly we used that planning as inference method for models that are related
to relational reinforcement learning. I didn't talk about this yet.
And these are relational.
Models on a symbolic level and can describe problems like should I grasp this
object or manipulate this object to achieve a goal or to build a tower or things like that.
Do I first have to open a door before I get an object and all these kind of things.
So we use these models and planning as inference in these kind of models which is really fast.
Really, because it's on a symbolic level. Very fast compared to all the low-level motion stuff.
So in those cases, what you call interrupts or unexpected events or things like
that, would just require to sort of redo the whole inference,
which, because it's on a symbolic level, is really very quick.
So, I don't know, currently in practice in robotics, I'd say the inference is
so fast that you just update online all the time.
Okay, so you're saying this will just be equalized out by compute speed.
You could put it like that, yeah. By just having the system to just update itself
and recompute the posterior whenever some new evidence comes in.
Right. So then also at some point in your presentation, you gave a little scheme
where you distinguished different variations of problem-solving approaches,
like model-based versus model-free.
Oh, that one. So what's that structure that you had exactly in mind there?
How does it organize the different approaches that are around?
Um i didn't really i didn't really think that there is like one structure which
combines all these approaches uh i mean that that diagram was just to to sort
of give an overview of what,
what kind of approaches people follow in reinforcement learning, right?
So model-based, the path going over first learning transition models and model-free,
just learning value functions.
I mean, you know Dyna, which combines the two in one framework.
I don't know. So our own work is mostly, actually, the work that we use in robotics,
which I didn't talk much about today, is really fully model-based.
So we always actually follow model-based approaches.
And it's only in what I talked about today, in that more recent work,
that we came up with a model-free reinforcement learning algorithm,
which I find interesting for other reasons.
But in robotics, I must say, I'm more a model-based proponent.
Why is that? Why do you find that more interesting or relevant?
In particular, for the types of problems that we did in relational reinforcement
learning, where the state space is inherently exponential in the number of objects.
So the state is described by all the relations between objects.
So if you have 10 objects, there is on the order of 10 squared binary relations between them.
All of them can have a value, so your state space is, say, 2, 2, 2, you know.
So the state space is exponential in those cases and how do you do learning
in these kinds of spaces? Um, and, and.
There is model-free approaches as well, who sort of represent also the policy
directly on some relational features, so first-order logic features,
and then use policy gradients to actually optimize them.
But we think that the generalization is actually much stronger if you try to
learn models from the experiences.
And the models in those cases are represented by stochastic relational rules.
Almost a bit like STRIP, but there is stochastic in their first order.
And it's not our own work. We use that work to learn these rules.
And we find it fascinating how strongly they generalize. So from only a couple
of experiences that something rolls when you push it, the robot generalizes
quite, I don't know, rationally one could say, I don't know,
quite interestingly to other objects and so forth.
So it is what I find interesting about model-based approaches
is the ability to generalize experience and but
the problem of model-based of course is well now you have that
nice model but how do you translate it to actions and this
is where then the planning is inference right exactly but to
what extent are these models uh actively acquired uh nice work so the the last
two years or so we've been um actually investigating a lot in in active exploration
or active learning so where a system should you know decide on actions that
maximize information gain.
And there's the fear of active learning in machine learning, right?
So one of the questions we had is how can we sort of transfer these methods,
the existing methods, on the relational for relational reinforcement learning.
And that requires that machine.
Estimating information gain or estimating your uncertainty of predictions,
as it is done also implicitly in R-Max or the Bayesian Exploration Bonus,
how to do the same thing with stochastic rules.
We did that. We call this relational exploration. That means that if the robot
has observed a ball rolling,
a green ball rolling and then a blue ball rolling, then it It maybe would not
be interested anymore in another yellow ball, but instead try to roll something
which looks different to a ball.
So absolutely. So these are exploration strategies that we investigated and
which are really important in these exponential state spaces to lead to efficient learning.
But for you, the main difference is that you say, look, if this state space
gets too large, you just need a better strategy to map it.
And that's an active learning. learning but or an active exploration component
while um intrinsically intrinsically you don't need to change your whole approach
it's just the way how you explore that state space i i agree so,
intrinsically uh it's a funny choice of word because of intrinsic motivation
but um it's the same principle so it's still the same principle of maximizing
information gain uh which is approximated as it is in Rmax and our vision exploration bonus.
What is necessary is to transfer that same intrinsic principle to other types
of representations, namely those relational representations or these stochastic rules we had.
Would you think this game would change a lot when I would impose a memory capacity constraint?
Um don't know um you mean
the exploration strategy would change
yeah or or the model the whole model construction
phase well in the model construction there is a regularization penalizing size
of the model so putting a hard limit on the model sometimes can be even shown
to be sort of dual to actually regularization but um yeah i'm not sure if it
would change So why this is interesting to me is, look, I try to understand how the brain works.
And the brain just doesn't have these luxuries of, you know,
infinite memory or infinitely fast processing speeds and so on.
And so this, of course, raises this issue that maybe the brain is operating
in a part of problem-solving state space where we just are not really looking
with our algorithms because we're not looking at the same constraints.
So it's for that I'm asking if I take a constraint not as a memory capacity,
would that change the game from your perspective.
I'd say I don't I don't know if I can,
say anything to that. I have the impression that the kind of models that we
learn are so small compared to the things that humans really learn.
Humans would have much more capacity than what our models that use these stochastic
relational rules, for instance.
Actually, I find it sometimes quite surprising how much humans can memorize,
children especially, without actually abstracting just as if they would just
throw it away before they actually later on. Without even being programmed by you. It's amazing.
Yes. So the other thing here is also, for instance, a lot of these methods that
are very advanced and you guys have a deep understanding of them,
but they are very often predicated on fairly strong assumptions.
For instance, one thing that I find often worrisome is that,
for instance, people sort of
very loosely say, well, okay, let's assume I have defined my state space.
And now on the basis of that I'm going to show you all sorts of properties of
my algorithm or I'm going to learn a model I'm going to do problems with and so on.
Isn't that assumption actually too strong? Isn't it?
Shouldn't we also think more about policy learning together with really the
state space acquisition from experience?
Yeah. Totally agree.
Yes, I totally agree. I mean, the question of where the representations comes
from is absolutely fundamental.
Maybe there are two ways to go about this.
The one is really having the ambition that the system should really invent its
own notions of state, its own representations and everything from scratch,
which has been the ambition,
for some while by researchers, and ideally even under partial observability and so forth.
Which, even myself, I was thinking about that. But more and more,
I think this is very tough.
Still, we should still continue trying, right? The more I actually work with robotics,
the more I have the impression that it might be worth before trying to invent
algorithms that can invent representations for all possible worlds,
to actually understand that actually our world, our 3D world, is quite special.
And it's special in the sense that it's 3d there is
physics a lot of things in our world are actually about rigid bodies and it's
becoming now very robotics talk right there's a lot of things about really kinematics
so there is degrees of freedom in our environment that you can manipulate and
so forth so our actually true world is very very structured,
So in that sense, why not as a simple step, because we're limited as researchers,
as humans, first try to understand those particular structures that are inherent
in our world and think about what would be representations to actually deal with those.
So that's also a more robotics-like, pragmatic approach.
But it's also funny that you're sort of following a little bit your physics
training or intuitions again by saying, oh no, let's make the spherical cow
and then we start from there. Frankly, I'm not sure.
I think it's the opposite in physics and training. I think the typical physicist
would go the first approach because he wants to be always general and work in
every possible world and derive optimal solutions in every possible world.
And in a sense, also machine learning and optimal control and MDPs, they do that, right?
They define problems which are seemingly total generally because you can describe
everything as an MDP or a control problem, right?
And you can embed everything in a vector space, which is sort of true.
But it might neglect the fact.
The actual problems we are faced with are very structured.
And acknowledging that specific structure is maybe less a theoretical physicist
thing, but more really in engineering and robotics.
But do you feel that there are already solutions of that kind on the table? No.
Looking more and more into robotics, I think that there is, of course, there is engineering,
which means that the people who program the roboticists, they have these concepts
and these specific representations and understanding of kinematics of worlds and so forth.
And from their understanding, then program the robot to deal with that.
But I do not have the impression that the machines themselves,
so let's say the inference mechanism in the models that I talked about today,
that they would actually do inferences about true physical situations.
Right. Or that we would have inference machines which could do inferences about
what is a stable physical situation, like probabilistic inference over that,
or where might be a degree of freedom in the world.
So doing inferences in these spaces, they're so structured that I think we do
not yet know how we could do probabilistic inference in those spaces.
Right. So what would be a good benchmark for these kinds of models?
Hmm...
Okay, so there is the old Köhler, Wolfgang Köhler, right?
He was a psychologist in the beginning of the 20th century. He did these experiments with monkeys.
Japanese, yeah. Exactly. I loved them on which island? I forgot.
So he has this book, which is called Intelligence Tests of Apes, something like that.
In these books he describes actually very nice behaviors which I would call
really goal directed behaviors and one of these behaviors is where there is
a banana at the ceiling and the robot is trying to get it but it's too high
it's a chimpanzee who tries to get it not a robot oh sorry,
you see where I'm getting at right so yeah the ape wants to get it it's too
high he jumps and for five minutes doesn't achieve it,
gets sort of depressed at least that's the storytelling that Köhler actually
does in the book sits in the corner.
Uh sort of depressed for a while and then suddenly uh
points his eyes towards the banana and then points his
eyes at the box in a corner and then again at
the banana in the box and then jumps up grabs the box goes on top of the box
and gets it um and i i wish you know robots could do that and i i think if i
would be psychologist so i would be very interested to to actually model what's
going on in that head of the monkey.
Because I think he did inference in a very structured way, inference about our physical world.
He did inference about, let me pull that box here and get on top of that.
And it's exploiting physics very, very much.
Yeah. We had a beautiful lecture on this last week by Alex Kacelnik,
exactly on this famous experiment by Koehler with also many other experiments
in that domain. So you would follow, you would stick to such a benchmark,
you'd be happy with that.
Absolutely. If the robots would do that, I'd be… Very good.
But now, I don't know whether you guys ever apply your own methods in an autobiographical way.
Because you could also say, look, every machine learning person is trying to
optimize their own policy in this complex world.
And in some sense, we also would like to make statements about biological systems,
about physical, psychological systems, and so on using these methods.
But there might be a possibility that, of course, you guys have optimized yourself
in some sort of local minima in this larger space of all possible policies and
problem-solving solutions,
that actually the links with natural systems has gotten lost.
Okay. So where do you stand on that?
To what extent also the models we now very superficially discussed now in this
interview, I mean, where is really the leverage, right?
Because, for instance, we all know the big hype that has been going around and
the big enthusiasm and hope for the last 30 years on these kinds of methods.
And sometimes when we look back, we can also say, well, yeah,
okay, it kept a lot of people busy. That's all very positive.
But did we really sort of make a huge step forward in understanding biological
systems or psychological systems?
And there, it's not so clear. Yeah, I agree that there is not necessarily always
links to the biological system.
Of the methods. And I would even claim that's very often not the goal to keep
these links to the biological paradigm.
Why not? It can be one goal for some people, but it's also an engineering goal
really to just design systems which do things good in a good way and optimize something or whatever.
But let me put it the other way.
I think in order to understand living systems, it is also a good idea to,
to understand our world, our environment, and to understand what it means to
behave, to organize behavior in that environment.
Totally, just on that level, I tend to say functional also, without caring for the substrate,
without caring for, I don't know, the constraints, the concrete biological constraints
that there are, but just to understand what is actually the structure of problems
that things in the world are being faced with, no matter if these are robots or not or other systems.
And I think that the engineering methods or robotics can contribute on that side.
And I also think that, well, it's important to understand actually the problems
when you then would go back and ask how could biological systems actually face these problems.
So it's a bit more a normative perspective then.
Like this is what the system should do.
Should do is even too much. what the system is confronted with,
understand the structure of what it is confronted with.
But now, if we look at nature or psychology, we might see that actually notions
of optimality might not hold so well, right?
Like human decision making is in many occasions suboptimal.
I'm here wasting your time and you're still being polite so that you could also
consider that suboptimal, right?
So in the face of let's say psychological reality and behavioral reality,
how do these notions of optimality actually hold up to that?
Optimality, so I have a very pragmatic actually relation to optimality I think,
many empirical things can be described as if they would adhere optimality principles,
it's a misunderstanding to actually say that this would have any meaning like
if it wants to be optimal or whatever let me take an example,
I mean the trajectory of a particle in physics, right? You can say it minimizes action.
Or some kind of field in physics behaves as if it would minimize an energy.
That energy or that action is only a scientific concept that we scientists have
devised to describe the actual system there.
Cost functions or optimality objectives like these things.
These are just designed by us scientists to describe systems in the world.
It's just a means to describe systems in the world, to say the system behaves as if it is optimal.
So I think it's a misunderstanding to really think that we believe machines must be optimal.
It's just a scientific tool to describe them as being optimal with respect to
some arbitrary cost function.
But for instance, humans have many biases in their decision-making,
right? And there's some beautiful books written about that.
One might be, for instance, many people in gambling believe they have much more
control over the outcome of a bet than they really have.
So, you see, in psychological reality, you might not be able to really optimize
so cleanly on some goal function, even though that is really at the core of
the methods that you're developing.
No, I think, and this is almost a trivial statement, that any behavior can be
described as being optimal. It's just a question of optimal with respect to what.
And I mean this, and it's no more statement when I'm saying I'm interested in
describing algorithms that can do optimal things.
It's only that I think that this is an abstraction of how to describe behavior.
I think even what people call sub-rational behavior or limited behavior,
of course you can describe it as being optimal with respect to something, some other objective.
Okay, but then, of course, there's another risk that's looming,
another challenge that I could say, yeah, but then, in that case,
your method is super powerful because it can never fail.
So then what do we learn? Well, the thing is that you now sort of shifted your
level of scientific description.
You know, you do not describe….
Behavior phenomena phenomena directly anymore in
a direct representation in terms of describing the policy but
you described you shifted your level of scientific description to the
level of describing what they optimize and at
first sight it's nothing more than just a shift you didn't gain anything by
that but potentially what you could gain is that this other description is more
compact and that's really the only scientific gain of that that you can describe
things more compactly. And this is how it was in physics always.
The reason why you would want to describe particles as minimizing action,
so the trajectory of particles, is because it's a very concise description of what's happening.
I mean, alternatively, you could just write down for every possible thing what
happened, but it's more concise to describe it like that.
Same with behavior like human motion.
Saying that human motion, as in
Walpert's work, behaves as if it would be stochastically optimal control.
Don't overstate this. It's not meaning that they want to optimize it or so.
It's just a statement that you can shift your scientific description of human
behavior onto an abstraction where you say the behavior, the motion is as if
it would minimize that objective function.
Right. Just a shift of description, nothing more. And a description of it, right?
Right. The hope would be that it's so concise that, for instance,
with robots, it's easier to specify or be creative in saying what they might
want to optimize and then derive behavior from that rather than programming behavior directly.
It's a shift of programming languages almost, if you want to say.
Now we're programming objective functions instead of policies directly. Exactly.
So Mark, so you're deeply involved in developing these sort of also new perspectives
on problem solving and machine learning, which also have quite some impact.
But so given this experience, what would you see as Mark's law for us to understand reality? Yeah.
Ah, I have no idea.
Mark's law. Acknowledge the structure. I don't know.
That's an important thing, which sometimes I don't feel myself that my own research
makes so much progress with that. But actually, I think this would be the most important thing.
Acknowledge really the structure of the problems.
Because, yeah, that's important. Right. And last point.
So five years from now, I'm going to come visit you wherever you are.
Maybe Stuttgart and I'm going to confront you with a prediction you're going
to make today but I'm going to say look Mark you predicted to me today or five
years back that the following would happen now did it really come out like that
was your prediction really supported,
what's this prediction you would like to make today um that you can test me in five years right,
um um well so our goal actually is and I usually say to my students in five
years as a point of motivation, that we would have that robot that autonomously
explores its environment,
and discovers all degrees of freedom.
That sounds like a simple statement initially, but if you look around the room
right now and think about all degrees of freedom in that room,
kinematic ones, there's a lot.
And in order to discover them, the robot would have to start pushing,
kicking, letting fall, letting drop, I don't know, everything grasping all of
these things and shake it and see if something moves. And that's something I would want to...
Okay, I'll come and see if the robot tore down your lab five years from now.
Yeah. So, Marc Doussaint, thank you very much for this conversation.
You're welcome. Thank you.
Music.
The CSN Podcast was produced by the Convergent Science Network of Biometrics
and Biohybrid Systems, a project funded by the European 7th Research Framework Programme.
For more interviews, recorded lectures or upcoming conferences in the field
of biometrics and biohybrid systems, go to csnnetwork.com.
Music.