This is the Convergent Science Network podcast. Leading researchers in the domain
of neuroscience, brain theory and technology are interviewed by Paul Verschure and Tony Prescott.
This is Paul Verschure with the Convergent Science Network podcast.
And in this episode, I'm speaking with Giovanni Pizzulo, who is a speaker at
our summer school, Barcelona Cognition Brain and Technology summer school.
And Giovanni, you started out with investigating this whole issue of the predictive brain.
Yes. So what does it mean for your predictive brain exactly?
Well, in a sense, the predictive brain is just a big idea.
Well, the idea is that the brain is not a passive organ.
So rather than just sitting down and waiting for the next stimulus,
in a sense, it really always tries to anticipate what comes next,
to set up some internal goals, and so to have some big internal processing.
So rather than being stimulus-based, it's really trying to anticipate the necessities
or maybe the opportunities for action.
So that's really the big principle that drives all the brain processing because
now, because the brain has some predictions, some goals, it can really organize
its sensory processing, its attention processes,
and its motor preparation processes for quickly responding and doing some adaptive actions.
So to do that, it has to be predictive and proactive. Mm-hmm.
But then where does that really start? What are the good examples of this kind of prediction?
Well, in the literature, there are many, many partially disconnected literature on prediction.
So predictions in sensory processing, predictions in the motor domain,
predictions also in higher cognition, in that, for instance,
I can try to anticipate what your next question will be.
So all these literatures can now proceed a little bit disconnected.
But for instance, in the sensory domain, you see a lot of predictive stuff going
on, such as the anticipation of the next stimulus.
In the motor control domain, you really have to get rid of what are the consequences
of your action for many reasons.
One big reason in the motor domain is that sensory perceptions are very ambiguous,
so by also predicting what comes next, you better estimate the state of the world.
Another big reason is for decision-making. So if you can really anticipate what
the effects of your action are,
then you can also select them on different courses of actions beforehand.
Beforehand, such as, for instance, if I anticipate that by going left,
I will get some big reward, and by going right, I will get some big punishment,
then that's the decision.
But now, in some sense, let's say also the physiological study of learning that
started with Pavlov, one of
the first observations of Pavlov was that the brain is predicting, right?
So that's also a key feature of classical conditioning. So in some sense,
the concept of prediction is with us for a long time.
And as some speakers have said, like sometimes certain things get so old that
they sound new again, right?
So what is actually really new in this current movement of the brain as a predictor?
Well, actually, I completely agree that also in classical conditioning,
you have this predicting dynamics going on.
But I think that we should try to really distinguish at least two kinds of predictions.
It's not a sharp distinction, but it's useful to do probably.
So one is an implicit kind of prediction in that, for instance,
in the conditioning studies.
So a stimulus, which is typically highly predictive of another stimulus,
which is in turn highly predictive of reward.
So the first stimulus, which is predictive of another stimulus,
it becomes itself good to achieve.
So in that case the brain becomes implicitly
predictive in that it stores the relationships between
the first and the second stimulus but to do
that you don't really need to to maintain into
your memory or to an internal representation of the
predictive relation between the two you just attach some good label to the to
the first stimulus whereas there is a second kind of prediction which is a more
explicit prediction in which you keep into your brain some model of the environment.
So the point is that in the first case you really have some implicit mechanisms
that link good or bad to sequences of things, whereas in the second mechanism
you also maintain a model of these sequences of things.
For instance in the motor domain you can, for instance, let's say you have to grasp a moving ball.
There are two different ways to do that. So one way is to look at the moving
ball and then jump to some future state.
To some future state that the ball will reach at some point.
This can be done in two ways. The first way is just learning the contingencies automatically.
So I know that if I see the ball moving, then I have to go to some position,
which is not the current position, but the future position.
Without even anticipating that position explicitly, you can do that automatically.
The second way to do that is really doing what is called a forward model,
so a predictive model that really tells you where the ball will be in the future.
Use this forward model online, so this predictive mechanism online,
and then move the hand to the predicted position.
So to recapitulate this idea, there are implicit and explicit mechanisms,
and the novelty of or the predictive brain hypothesis, if you want,
is the hypothesis that the brain systematically incorporates,
regularities of the external environment in a structured way so as to form strong
models, statistical models, for instance, of the regularities of the external environment.
And then it systematically uses these models to plan what to do next and also
to drive perception, to drive attention, and so on.
So, whereas in the first formulation of prediction,
it was more an implicit prediction, in the predictive brain hypothesis,
at least in some of its formulation, it's much more about building models for
the world, on top of which you can run explicit predictions. Right, so then….
If you would look at, let's say, the theoretical literature on the predictive
brain, what in your mind are right now the outstanding examples of this approach?
Well, there are many of them.
So probably in a very famous paper by Rao and Ballard, it was already in the
99, if I remember well, it was about this predictive coding idea,
predictive coding idea in perception.
So the idea is that the brain continuously generates prediction about the stimulus,
and it uses these predictions in a top-down manner, whereas it uses prediction errors,
so the difference between the prediction and the sensory evidence,
as a revision mechanism.
So that was a milestone in a sense.
But, well, there are many other papers now. There is a very big framework put
forward by Carl Freestone, which is called the Free Energy Framework or the
Active Inference Framework.
In that sense, Freestone really tries to look at the whole brain,
not just the perceptual system, as a big prediction machine.
There are many other frameworks. One is put forward by Moshibar,
again, in the perceptual domain,
whereas in the motor domain, also the leading view, one of the most authoritative
view put forward by people such as Danny Wolpert or by Shadmer or many other
people, is that in reality,
what you really predict is the sensory consequences of your actions.
So that framework is more tied to motor predictions.
Other people, such as Mark Sherrow or others, they have tried to expand this
framework from the prediction of the sensory consequences of actions to more
complicated forms of cognition,
such as reusing this prediction for understanding of the action of others,
such as reusing this prediction for imagery,
and so thinking at more abstract situations.
So, I would say that there are many fields in which prediction have been studied,
and also some converging frameworks.
Right. So now, if we talk about your own experiments in the role of prediction in problem solving,
you showed some examples of what you called embodied problem solving. Yes.
Right? So how does that, which aspects of the predictive brain are these experiments exercising?
Well, it's not so simple to explain the videos that I've shown,
but just to recap, it's just a video of a climber.
And maybe you know, but climbers prior to a competition, they have some time
to look at the climbing wall that they will then climb during the competition.
The nice point is that they see this climbing wall for the first time and there
are many in these climbing walls,
there are many climbing holes arranged all over or through the wall and the
climber has one minute or a few minutes for figuring out how to better climb
this wall and if you look at the video or if you look at the climbing competition you see people,
People, climbers really moving the arms such as to really anticipate the next moves.
They see moving the arms in one direction, then the other direction.
That's really a form of problem solving because they see the climbing wall for the first time.
They have to figure out a good plan for action.
I call it problem solving, not only planning, because it's really complex.
There are many constraints. It depends on where you go.
Then you can reach or cannot reach the other hold. So you can also find in your
mind many, many solutions, compare them.
And in this specific kind of setups, the hypothesis is that it is really the
motor system that is governing this process.
So expert climbers are really able to anticipate the force they have to put into this motor act.
They can really anticipate a lot of proprioceptive information.
They can really figure out if a hold is in their reach or out of reach,
if this hold is too little to be really grasped, if it is too far away for them,
then they have to take another route.
So that's, I call it embodied problem solving for many reasons.
So one reason is that you really use knowledge of your body and of your motor
system to figure out how to better solve this problem.
Problem and another reason is that it's also
overtly embodied in that you really use your
body your body movements as a scaffold as a help a helping tool for solving
the problem that's a bit tricky to explain without the video but in a sense
by simply moving your yourself you don't need to to keep everything into your
memory you use your body as part of the problem solving process.
Okay, but now you could also see it as a form of, let's say, motor programming.
That you say, look, I have to execute a movement sequence very rapidly. Okay.
Um, and I'm just going to now rehearse this movement sequence that I can sort
of automatically trigger one set of behavioral motion motor patterns after the
other, and it can climb up the wall more rapidly.
Yeah, that's, that's part of the, that's part of the story. So the point is
forming this motor program.
Uh, the, the reason why I call it an embodied problem solving is that the solution is not so trivial.
So the point is that you have to try to form this motor program by assembling
partial skills that you used in the past in novel ways.
So there is a lot of flexibility in this new assemblage of motor or small motor programs.
Yeah, but the problem solving would suggest that there is also,
let's say, a goal state in the world that you want to achieve in order that
you also have to manipulate aspects of that world.
Yeah. Well, in this case, it's essentially making sure your body goes through
a certain sequence of motions.
So how would your view on embodied problem solving then generalize to problem solving in general?
Because I would assume that you want to identify some more, let's say,
generic aspects of problem solving as opposed to specialized aspects of problem solving.
Yeah, okay. So there are two parts of this story. So one part is that even in
that case, even in this climbing example, you have a goal. So the goal is reaching the top.
And then it is not simply running through a trajectory, but also finding out
which is the good trajectory.
So it's not so different from the Tower of London or other typical problem-solving setup.
So you have a goal state, and you have many, many sequences of moves that you
can do, and you explore this space of possibilities.
But the trick is that you explore it in intelligent ways, so you don't simply try out all of them.
You really use your expertise to find out how to better explore this.
The second point is that you really use knowledge incorporated into your body,
your motor programs, to figure out what are the constraints of the problem.
The climbing problem has many constraints because, as I said before,
if you are too much to the left of the climbing wall, you cannot reach any more the holds on the right.
Or vice versa, if you jump too high, you cannot probably make your body stiff
enough to really hold the climbing hold.
There are many constraints that you really discover while solving the problem
by reenacting this motor knowledge. That's my point.
And the second part of the answer was about how much this generalizes to more
complex and abstract problem solving.
So, well, first of all, this is just a nice video that I showed just for illustrating
the possibility of solving problems by reusing motor knowledge or sensory motor knowledge.
It could also be affective knowledge. So knowledge incorporated into the same
models that you use for acting in the external world.
So I think that as a general strategy, that's the way we should look at higher cognition.
The challenge is looking at how you solve higher cognitive skills,
such as problem solving in abstract domains, by reusing these strategies that
you acquired so as to efficiently deal with the external environment.
So when you have these sensory-motor strategies, then probably our earlier ancient
evolutionary ancestors, they
had only these simple strategies to deal with their current situations.
Whereas higher cognitive skills is mostly about very complex abstract situations,
non-perceptually available events, distal goals.
But the challenge is seeing how simple strategies that we used in the past to
survive in the environment can be sophisticated and reused in these more complex cognitive domains.
So that's a challenge. My example was just to illustrate the possibility that
you can really solve problems by reusing systematically knowledge incorporated
into your internal models in an intelligent way.
And that allows also for a lot of flexibility. possibility is not for planning
the next action, but also for solving very complex problems.
That's a bit the wish, right? That's the wish. Yes. Then the question is,
how far are you in realizing that wish?
So first in the climber case, do you see a big difference between expert and novice climbers?
Yeah, absolutely. Yes. So part of the story, which is also one of the reasons
why I'm interested in it, is that But in a sense, it is the expertise that not
only modulates the climbing ability,
but also the ability to think about problems.
That's supporting for the embodied cognition view, for the view that it's really
knowledge incorporated into your motor system that helps you not only in executing the actions,
but also for instance in imagining the actions, in reusing the repertoire for solving problems.
At the same time, we have evidence that knowledge incorporated in your motor
system also helps you understand what other people are doing.
Right. So that's the idea in a sense.
Yeah, that's in the step into the social cognition component.
But in some sense what you're saying, look, the standard view would be,
let's say we perceive the world, we have some sort of high fidelity in the end
interpretation of the world on the basis we should make decisions and perform action.
But now the consequence of what you're saying is say, look, I'm
actually acting in this world and the way I act in the
world is now modulating or directly filtering the
way I'm going to perceive this world absolutely so the processing
goes backwards in that sense so the expert
climbing is looking at this world very differently as a novice climber yes and
you have evidence for that uh well we yes we perform them uh well of course
we have a tiny evidence up to now but i think the this evidence points in the
right direction so one memory study that we performed,
comparing novice and expert climbers.
So we simply asked these novice and expert climbers to look at three climbing routes,
one simple, that both of them were able to execute, one difficult,
that only the expert group was able to do, and one impossible,
which was not climbable, actually.
It was not really a climbing route. It was just a random displacement of climbing holds.
And so the task was simply remembering the climbing holds in the right sequence.
What we found is that in the easy condition that both novice and expert were
able to climb, both were also able to remember quite well.
So there was no evidence for a better remembering for experts.
While for the difficult route, only the experts were really able to remember it well.
And this, in our opinion, this points to the fact that really the way experts
structure the perceptually also the situation can also help in this memory task.
And so we had this control condition in which for the impossible route,
in that route, we did not find any advantage for the experts because this ability,
this increased memory ability is really tied to the climbability of the wall.
It's not a generic ability to remember climbing holds without context.
But to respond completely to your question, I would say that this sensory motor approach,
so this sensory motor influence of thinking on cognition, to me,
at least, it acts on two timescales, at least two timescales.
One timescale is learning and development.
So the point is that because I have some sensory motor skills,
then I change the way I perceive the world.
So I structure my memory and my perceptual abilities in such a way that then
supports better cognitive abilities.
So that acts on a longer time scale of learning and development.
The second way the sensory motor system supports cognition is more in the online cognition.
Cognition, because I can reenact my motor problems right now,
my motor programs right now, so I can use them so as to support,
for instance, imagery or action perception.
So there are two sides of the same coin. So in a sense, this framework predicts
that if you increase your sensory motor abilities in one domain,
then you really shape your perception, your memory, and all your cognitive processing.
And then you are also better able to run imaginary experiments based on this
motor expertise that you can now reenact.
Right. But then would the expert climbers recognize the unclimbable wall more
rapidly than the novices, novice climbers?
Well, probably, well, if the task is exactly recognized whether or not it is
climbable or unclimbable, that's the task.
If this is the task, well, of course, this is a bit of a tricky task because
unclimbable could be, is not such a clear concept.
But anyway, yes, I would say that in that case, what will happen is just that
the climbers try to climb it mentally.
So they run this imaginary climbing simulation, and then because the expert
will anticipate some failure in climbing, because they have high confidence
in their internal models, they will say, no, no, no, this cannot be climbed.
Whereas for the novices, maybe they will try out to anticipate.
They will also fail, but without noticing the, well, without trusting too much their simulations.
So that will be my... Right.
But now, still a problem for this point of view is that you could say,
look, the climber is a highly specialized human being.
So it's not very surprising that they have certain cognitive capabilities because
they have been overtrained in dealing with certain kinds of environments like climbing walls.
So how would you see this sort of.
But you want to look at this as an example where
you say well it's actually very basic sensory motor patterns or it's really
action itself and the way we generate action from which the rest of cognition
is sort of bootstrapped or is depends upon it's constrained by so how is it
going to work exactly how do you see that work,
Well, I would say that now we don't have any perfect story yet,
but the point is really that action and the goals implied in the action and
the predictions implied in the action, they are really key to cognition and
to develop increasingly more complex cognitive skills.
So at the beginning, so that could be a simple story.
So at the beginning, the brain is simply organized around these goal selection
and specification and selection tasks.
So they have to integrate information from the external world,
from the memory, from affective states, to very, very quickly jump into a good action selection.
And to do that, as I told before, you're mostly relying on your internally generated
goal state, affective processes, attentional processes.
The stimulus helps you, but then it's the brain, which is autonomous, is doing the job.
But then as the sensory motor abilities of this primitive architecture develop
over time and increase your abilities, they increase their ability to control
the external world and to predict it.
So they start incorporating more and more knowledge, more and more also structure
from the external environment into their predictions, into their motor control actions.
So how are you going to validate this prediction?
Is it a pure experimental exercise or are there other ways to get this validated?
The framework, you mean? Well, the framework, that's much more a lifelong research
program, if you want, because we are trying to do many, many validations.
A few empirical data, of course, already exist.
If you think of the literature, how similar the brain networks for imagery and
for execution or for motor preparation, they are.
So you already see that some mental operations are really supported by the sensory
motor system in the brain. So that's some evidence.
There is evidence, well, there are also patient studies that say that in some
conditions, the imaginary action cannot be really stopped from being overtly executed.
So that's another part of the story because
in this story it is it is the internalization of
the predictive mechanism that really supports cognition so you
first predict predict things in the external world then you internalize this
capability of predicting and this eventually lets you to rehearsing entire sequences
of action without executing them but then a prediction of this framework is
that if you really do this covert mental imagery if you want or discover covered predictive magnets,
and you cannot really separate these internal processes from the overt execution,
then this implies that what you imagine, you immediately do.
And there is evidence that this unfortunately happens in patients with bilateral parietal lesions.
And other evidence of this close connection between what you think and what you do is,
also exist for utilization behavior for instance utilization behavior
patients they are not really able to inhibit their
motor programs for grasping objects and uh just
reacting in a very uh quick way to what they say so this uh this tells a story
on on how uh this more um abstract thinking part is tied to the sensory motor
programs okay but now this these this this in how do you what's What's the role
of the internal simulation in that case?
Because on the one hand, you're saying, well, we run these internal simulations.
Apparently, they would run in parallel.
But on the other hand, if you look at the deficits that you mentioned in these
patients, it's not that they are executing a whole sequence of actions in parallel, right?
In utilization behavior, you would grasp a single object and the whole organism
would be focused on that.
So therefore, is this kind of evidence coming more from the clinic,
from clinical studies, really supporting this hypothesis of internal assimilation
and the role of action in the structure of cognition?
Well, for the first evidence I mentioned of the patient who was unable to inhibit
the imagined action, that's quite clear, because then what you think is what you do.
So if you imagine, let's say, pointing to one direction, then you point in that
direction, you cannot really stop from doing that.
So in this framework that I'm supporting, the idea is that you run this imaginary
simulation of pointing.
This is typically done by also inhibiting the overt motor execution.
But if you have a deficit, you cannot inhibit that output.
It rapidly turns out into an overt action.
In the utilization behavior, the link is tiny in a sense.
The idea is that one idea in the literature of affordances, also the canonical
neurons literature, is that when you look at an object,
then you automatically, let's say, pre-potenciate some motor programs that are
good for interacting with that object.
Jet okay but typically this does not result into an overt execution so in a
sense this is just a pre potentiation which can be also interpreted in terms
of simulating grasping this cap in front of me,
So in that case, I'm not reasoning about simulating the action.
This is just a more automatic process of mentally rehearsing of the good action that they can do.
Again, if you don't have this, let's say, this mechanism working well,
you cannot really inhibit also the overt execution.
I would say that while in my first example of the imaginary situation,
the link is clearer, in this second example, it's less clear.
But to me, it's another supporting example that your internal mental process
is always oriented towards anticipating possibilities for action.
In some cases, this is just a simple automatic process that tells you what is possible to do.
In other cases, this is more intentional imagery.
So you really run long-term simulation imaging sequences of actions.
But I see a continuity in this.
So the good thing about this framework is that you do not have to postulate
any different set of cognitive representation for different tasks,
any module for thinking that is completely segregated from the action control and specification,
but you have a continual reuse of the same abilities in more complex ways.
Okay, but then would you see this as being, let's say, layered in some way,
that you have, let's say, sensory motor capabilities of, let's say,
varying levels of complexity with some sort of discrete steps between them,
or is it really a continuum?
Well, that's a hard question. one important
thing in this framework that probably answers partially to
your question is that we have
to probably focus we have to talk about
actions as the unity and also we
know that action can be specified at different levels of detail
so there are actions that are specified
at the level of movement of a single finger whereas there
are other actions in which the action but also
the goal of the action is specified at a more abstract
level that is grasping this uh this object
this cup whereas there is still another level of
action and intention associated which is uh let's say drinking from this cup
so the actions uh all these all these uh levels are are in a sense they they
are they support one another in a sense so of Of course,
the more abstract actions have to be finally specified in more fine-grained
terms of the movements of the fingers.
But this is a structure, a cognitive structure that has different layers probably.
Or we don't know exactly, but at least you have a big structure in which you
can really specify actions at different levels of complexity,
a different level of abstraction with more complex intentions.
And the arguments that we do is not that we always use the lowest level,
the lowest possible levels.
At some point, you can really think and do some imagery by using actions and
their associated intentions at intermediate or even at abstract levels.
So that's probably answers. Right. So now one way to test some of these ideas,
you were performing and presenting experiments on joint action.
Yes. So how does joint action now help you to understand this model of cognition and action in the end?
Well, the reason why I went into joint action in my presentation is that, as I told before,
I would like to come out with a convincing story or more complex cognitive abilities
developed based on simpler cognitive abilities.
So here the steps that I presented are, okay, I start doing my action,
but then I am in a social domain, and so maybe I need to do some action together with you.
So in this case, I can reuse a lot of what I know about my action system to
anticipate you and to get into your intentions.
So to get rid of your motor system or your motor action of your intentions.
In this case, this is just the beginning of a story in which I reuse my sensory
motor knowledge or my action execution and prediction abilities to get into
more and more complex situations.
In this case, it is a social situation, but it could be even a non-social situation.
Still another step in this passage is that, okay, now I'm able to use my predictive
abilities to predict you.
I use my body and my action system as a model to understand your body and your action system.
And then I maybe use these abilities to plan long-term joint action.
To do that then i have planned some more coordinating actions
so i have to plan how to really achieve them in practice i
have maybe to to invent some way to better
coordinate with you and maybe at some
point you invent more and more complex things you extend the
boundaries of your control to the control of my body or
to the control of our body or our combined actions
and maybe also to the control of your internal
states so your beliefs your intentions so
i do some action to change your intention so not only
i control my body i now control me and you and i
control you and i know that uh even after
this interaction you will have some new beliefs i extend my control abilities
control capabilities from myself to you so that that's uh this the very beginning
of a story in which we begin to to go into more and more complex cognitive abilities
is starting from simpler ones. Right.
So for the joint action experiment, you were looking at hand movements essentially. Right.
You're tracking hand mostly yes and then you
also build a model of that yes okay so so
which aspect now of the joint action control of
hands do you capture in this in this model well if we think the model specifically
then uh well in a sense the key idea of the model is that it's not completely is not new.
There are many similar models, but the idea in a sense is that I really use
my own action abilities as a model to understand your action right now.
So the model can predict for instance the hand trajectory, but at the same time
it can also give some hint into your intentions and goals.
Because I know what are the goals that are linked to my model actions.
How does the model know that? Well, let's imagine I see you moving your arm towards a cup, okay?
So now the hypothesis is that I run an internal simulation of the possible actions
that I could be that are quite compatible with the actions that I'm seeing.
I start predicting better and better your trajectory, and then let's see that
this model fits quite well the data, so your hand trajectory.
Because I know what is the goal
of my hand action, I also can have some hypothesis of what is the goal.
So because I know that my trajectory eventually reaches the cap and then grasps
it, then I can also infer or at least hypothesize that you will do the same.
So I also know what is your goal.
That's the key idea. But then the intentional labeling of the sequence is then
with reference to your own action or that is in reference in some way to observing the other?
So the idea of these kind of models is that you use what you know about the
link between the trajectory of your arms and your goal. You know them because you execute them.
You use exactly the same link to get some understanding of what the other is doing.
So that's the idea. So you have first some sort of self-monitoring system that
develops a self-model that you then generalize to the other.
This would be the step. Exactly. So I will not necessarily call it a self-monitoring
in that you simply need a forward model and an inverse model.
So you need some control capabilities for yourself.
And then you transfer, you use this exactly as a model to understand the other.
So that's more or less the key idea. And this idea is pursued by many people
actually also. Right. Yeah. Okay.
So what's, do you feel confident then that this original idea you had about
this more, let's say, embodied problem solving, action as grounding, cognition,
what tells you in this joint action task where you now are generalizing this
model to social interaction, that this is working?
Because in some sense, I could also argue, well, it works because you have been
imposing, you have really made the task pretty abstract, right?
Because I'm just observing these hands moving in one plane. plane,
I don't have to do anything about, let's say, invariant recognition of posture, of limbs and so on.
So to really now generalize this to, let's say, a realistic scenario where you
have to observe another human moving about in space, is that just a small matter
of programming or are there some fundamental steps still missing?
Well, to answer this question, probably we have to go into a bigger picture.
The bigger picture to me is that now, although I have emphasized these motor
predictions, that's not the whole story.
In a sense, my opinion is that the brain is a smart guy.
The brain always uses all the knowledge that it has, at least in principle, to do the job.
The point here is that in more realistic joint action or action observation
scenarios, there is a lot of information
available. There is perceptual information, which is available.
There is some context information, such as, for instance, I see what are the
objects within your pre-personal space. So that's my prior.
I can have a lot of prior information on you. For instance, I know that you like very much the beer.
So if there are in front of you, I see some beer and I see some Coke,
I would maybe anticipate that you will grasp the beer.
And then I also use this motor simulation that I told, because it is also useful.
So I'm not assuming that the brain only uses this motor simulation part.
It uses the motor simulation to support this action understanding,
because in this context, it is very salient and very useful.
Why is it very useful? Because, well, I have a very good model of my body.
So it's a very good model, very good predictive model for your movements.
Of course, I have also some purely perceptual predictive mechanisms that can help me predicting you.
But I would say that in the case of human understanding, the model of myself
is a very, very good model of you in most cases.
So this is what the brain will use the most.
But if you go into the biggest picture, you can think of it as a big Bayesian
process happening. So in a Bayesian process, you basically use all the information
you have, information which comes first, then can be also used as a prior to
run the rest of the process.
And then all the new information that is collected is fused in an intelligent way.
Intelligent here means that it is weighted depending on its uncertainty.
So if the motor system is a very good model, the information it provides will be used very much.
Much if this information is not good or it is failing then it will not be used
very much yeah but you seem to now to drift away a little bit from your original.
Proposal because i thought you started out by saying look cognition is really
predicated on action yeah right but now you seem to say something like well
motor the motor system is one of many sources of information you could consider
because the bayesian system basically use all the knowledge it has,
and it has knowledge about the motor system, but also about,
let's say, perceptual systems, or from memory.
So aren't you a bit drifting now in that argument?
Well, I don't think so, because as I told before, the way you collect this information
in the first place is through learning.
And through learning, the motor system has really influenced you a lot.
So also through learning, this perceptual information that I have,
I have acquired it by also interacting with the world.
So the motor system really is used very much not only online,
as I told now, but also during learning.
Of course, we don't have to be radical on that. So we don't think that only
the motor system does something in the brain.
There are many parts of the brain that are used.
And probably the point is slightly different.
The point is not all about the motor system. So the point is that the ability
that you have to interact with the external world,
which can be supported by many systems in the brain,
prominently by the motor system, but well many system
you reuse them consistently to achieve new
cognitive abilities so to develop and and
then to achieve new new goals specified at different levels okay but um so then
you're saying well action let's say impregnates all other aspects yes of of
cognition and perception through the cause of learning okay and this so it has
a direct and otherwise an indirect impact yeah yeah Yeah, there are both.
I would say that the first one is through learning. The second one is during
overt interaction with the external world.
And then through the internalization of this action and prediction process,
you can imagine new scenarios.
But this doesn't mean that when you can have some motor simulation,
you close your eyes, you don't use percentile information.
That would be foolish from the point of view of the brain. and and
there is an important uh point of this view
is that uh as you know in action understanding
there are many views one view is that you
i call this the mirrorless view is that
the first thing that is active is the recognition of the goal then the recognition
of the goal in the mirror system of course it eventually helps also predicting
the action there is a second view which is related but in some sense also very
different because this view is that what you do first is prediction.
So it is prediction which comes first and prediction helps then recognizing the goal.
But still another view is that all these motor simulation things,
they are not so important because we really use much more abstract knowledge
of what the other person should do, given the context.
So that's more theological knowledge, if you want.
Well, in the view I'm supporting, it is not so useful to say what is more used and what is less used.
So what comes first can be used as a prior for the rest of the things.
And what is more reliable is used more than the other things.
It happens, at least to me, that
this motor simulation in that specific context, they are very reliable.
This is why I think they play a prominent role, not because of some categorical distinctions.
And also relating to this issue of what comes first, goal recognition or prediction,
that's really depending on the task.
So if I have a big prior knowledge of what the most likely goals will be,
then it is probable that I will force some hypothesis or some probability distribution
over those goals much before starting predicting you.
But if I cannot see what are the goals, let's say the objects you can grasp,
I will probably start by predicting and by prediction I will get into some hypothesis of what is your goal.
So everything can be very flexible.
Okay, but you are saying ideally you take a model-based approach towards yes
in this case also social perception you want to say yes you want to get to the
goal of the intentional state of the other agent as quickly as possible and
from there you just reconstruct the rest of the action,
yeah yes in some sense so yeah.
Typically, I want to go into the goal when this is useful for my task.
So if my task is understanding which cup will you grasp, that's the goal information that I need.
Whereas if I don't want to hurt you, it's more the trajectory that I want to know.
So for example, when we are driving, I don't really need to know where are you going.
I don't care. I don't want to cross you or to hurt you. So it's not so much
the goal of what you're doing that is important, but it is the trajectory that you are doing right now.
Of course, also the goal is interesting because the goal tells me a lot about your trajectory.
So I would say that what you ask, what you want to infer is very much task dependent.
And as a side effect, you will also infer many other things that give you useful
priors or useful hints. experiments. So that's my view.
You don't only run experiments, right? To test these ideas, you also use robots.
Yes. So how have robots helped you to make progress on these issues?
Well, yes, we are running many computational and also robotic experiments.
So So one thing that we have recently seen is that probably this is very interesting
again for the argument that more and more complex cognitive abilities can be
developed on top of this sensory-motor interaction.
One thing that we are investigating with computational models and robots is
the ability to signal in social context.
So just to explain this very quickly, the point is that if we are supposed to
do one joint action okay let's say building some tower together but of red and
blue blocks okay one red one blue one red one blue okay now we are interacting
we are supposed to do that,
and we are able to coordinate to do that but now let's imagine only i know.
What is the goal? What is the tower to be built? You don't know.
So one thing that we are investigating with robots is how we come out with some
good coordination with only one person knowing the task.
In that case, without using overt or linguistic communication.
So what we have seen also in robot, that's a very hard problem.
Because when you run computational robotic models, then you put your ideas into practice practice,
and you see that by simply predicting the action of the others doesn't work
too much because then I start predicting you,
but so the guy who doesn't know the job has to predict the actions of the other,
but it is always one step later, it's too late.
So it doesn't work very well. So we come out with the idea that the guy who
knows the job can also in some sense support the predictive processes of the
other or help the other in some sense.
So how would I help my other robot? Well, one simple example is that I can make
my behavior more predictable, or I can make my behavior more informative for you.
Such as I can use my choice of action to letting you understand very well what are my intentions.
So, you typically try to understand my intentions, but if the uncertainty is
too high, I can in some sense help you with the wise choice of actions.
That can be very intuitive.
But it could also be a way to test hypotheses about the other.
It's not just making your behavior more predictable. It must be a way to test
whether the other has the right model of you.
Absolutely. Yes, that's the idea. So, we design… But it's not the same thing, right? No, no, no.
Well, they are related. So, in one sense,
Also, in my example, let's call the guy who has the knowledge the leader and the other the follower.
So in a sense, the leader can have some uncertainty on what are the models of the follower.
So, of course, he can use some actions and monitor the reactions of the other
person just to know what are the good models, what are the models of the other
agents, what is he doing.
Right. And when the leader infers that the follower does not have good models, then he will fail.
Then he can, in a sense, try to help the follower.
So the two processes are interconnected.
So by monitoring the process and by always trying to keep track of the uncertainty
of the other person, you really can plan helping actions. Right.
But now the thing was interesting that, so we go from action now to interaction.
Yes. And in your research plan, that's really a very crucial step, right?
Because you really seem to think that or propose that it's by getting a handle
on interaction that you really can get to, let's say, a more generalized understanding
of social behavior and also cultures.
Yeah. So how should I see that generalization exactly?
The point is that I think that for many disciplines, such as the study of language
and communication, the natural starting point would be naturalistic joint action.
That will be a nice starting point because now the hypothesis is that the mechanism
that we use for interacting with the others in sensory motor domains,
then they provide a scaffold.
So they provide some help for developing, let's say, for instance,
linguistic communication.
The hypothesis, which has been called by Levinson this more interaction engine hypothesis,
is that by simply interacting in the external world, we really have some strong
pragmatic abilities to infer the action of the other, to do some truth-taking,
to do some joint attention,
to anticipate the action and the intentions of the others.
And this is a strong universal basis also for language communication.
So, he hypothesizes, for instance, the children use this interaction engine
to develop, so to understand and to learn their natural language.
And that's exactly a point that we want to make.
So we started studying this kind of joint actions. Now we come out with a formulation
of the problem in which the two persons happen to solve a joint action optimization process.
This joint action optimization process means that it is not only so I don't
have to care only about my action, but also the joint outcome of the action.
Because I have to care about the joint action and the joint outcome of our action,
which is a joint goal, then I also have to care about you.
This is why in some cases, as I told before, I help you, not because I'm altruistic.
I'm helping you because by helping you, I help the joint goal.
All. So I'm sorry but I'm not so much altruistic, that's all individualistic
but at the same time I think that this helping action,
is is really the first primitive form of a
communicative action because as i told before i
can make my action more observable by
you more predictable by you more understandable or more
diagnostic for you diagnostic here simple
means simply means that if you have two hypotheses on what i'm doing i make
my action uh let's say for instance i exaggerate the movement such as to make
you understand very quickly what i what is my right intentions that's diagnostic
but that That exaggeration that I do, it's really a form of communication.
That's really the bootstrap also of all other forms of communication,
including linguistic communication.
So the general hypothesis here, this is why I'm interested in this passage from
joint action to more complex form of cognition, is that by interacting,
because there are some constraints,
some joint constraints that we have to fulfill, feel
then in this joint optimization
framework it's quite obvious that i have
to do something also for you for your sake right and
in our formulation that that that means that in addition to having some motor
intention we also have some communicative intention the communicative intention
is has a cost because for instance for exaggerating the movement i pay a cost
in terms of biomechanical a cost of something.
So such as, for instance, when I am in a noisy pub, I have to over-articulate.
Or if you think the mother is, for example, when the child directed speech,
so you exaggerate, that's done for the sake of the other person.
Ultimately, we say that's done for the sake of achieving a good joint goal,
which is understanding one another.
But that's really the first form of motor of communicative intention that I have.
So it's for you, it's really the signaling mechanisms that would allow the bootstrapping
of communication and language, right?
But that's still a project that will be realized in the future.
Yes, but I would say that the important component is always the missing one
from the current studies, which is the more pragmatic component of language.
So the pragmatics of language as the early pragmatists, which study language know very, very well.
So the pragmatic component is prominent because then the ground ego symbols,
the grammar, of course, they are hard problems, but the pragmatics,
they are the most important part, such as for instance, knowing Knowing what requesting means,
knowing what telling means or promising means or all these joint action and joint attention,
tour-taking mechanism, all of them provide a big scaffolding for the emergence
of linguistic communication. Right.
So I got that. But now, so to finish up, two questions.
So you have this really big ambitious program, and in the end,
you'll generate a whole of culture from action, which is going to be still a
long trajectory, but you're well on your way.
But in doing this, what would be Giovanni's Law?
Giovanni's law? Okay. Well, my law would be maybe starting understanding cognitions
from the primitive equipment of our ancestors.
So that would be more an evolutionary route towards the more complex cognitive skills.
Well, I can probably give three
examples that are very simple in the three domains that we target now.
One is the social domain that we have discussed.
So you start from controlling your body, your actions, to controlling our actions,
to controlling then your mental states.
That's communication in a sense. Communication is really putting something into your brain.
And then you also develop culture and pedagogy, for instance,
for controlling that at a very, very longer timescale.
So by developing some culture, the human beings can really control the behavior
of many people over millennia probably, and also by pedagogy.
So that's in the social domain.
You can see this also in the control of the external world, on this environmental
scaffolding or the environmental shaping.
So we start from the control of our body to the control maybe of the reality.
Our peripersonal space with some objects and then
you start controlling tools for instance
for doing more complex actions and but then
tools you have also to build tools maybe at some
point you build the city of barcelona so then in a sense
you extend the boundaries of your control from the control your body to the
control of the whole environment we humans really modify our environment very
much such that then they support much better our cognition so that's another
story not only in the social domain also in the environmental domain the third
the third and last one is in the more,
thought domain or the cognitive control domain so you just start from controlling
your action and then you end up in some way by controlling your thought processes
and by controlling your cognition over long long long time scales for instance
one example that i do well,
let's imagine i want to become a prominent scientist
which apparently doesn't doesn't happen right
now so i have to control my behavior for 20 years
maybe so the way i do that is by well by cognitive control in a sense by setting
some goal states into my mind then then control my behavior on longer longer
longer time scales then again you have this control of the body the or the control
of my short-term actions that become the control of my entire life.
I have given three examples of how this research program, I would say,
would start from very simple sensory motor abilities to very,
very complex abilities that cross the boundaries of many disciplines.
Sociology on the one hand, and let's say architecture on the other hand,
or maybe the study of consciousness on the other hand.
And you have even integrated it in your own career planning.
Ah, of course. so the last thing the very last question then is prediction so
if we come back then five years from now as a famous scientist to the summer school again.
So what's the one prediction we can test on you by then?
So what's the one prediction you would like to make today that you feel very strongly about?
And we can ask you about five years from now.
Okay, so about the future of this field or about the… Your research program. Okay.
So about my research program. Well, I think I elucidated the principle,
my key goal, which is understanding higher cognition, how our cognition is based
on sensory motor and predictive abilities.
So the prediction is that at least we will have some nice demonstration in these
three fields that I've mentioned before, much stronger demonstration than the ones that we have now.
And from that, we can really try to build an understanding of how the brain
implements higher cognitive skills.
So at least one demonstration for each of the three fields. That's very good.
So Giovanna Pizzuto, thank you very much for this conversation. Thank you, Paul.
The csn podcast was produced by the convergent science network of biometrics
and biohybrid systems a project funded by the european sevens research framework program.
Music.