All right. So this is Tony Prescott and Paul Verschure talking with Michael
Arbib after his presentation at the Barcelona Brain, Cognition and Technology Summer School.
And we want to revisit some of the main themes that Michael has been talking
about to us. So, Michael, would you like to give a short summary in a few words?
So what few of the key messages were in your two lectures?
Well, for me, the grounding interest has been how vision is related to action.
And for that, I've been looking at two different approaches and trying to integrate them.
One is what I call the schema-based approach, is to try and take an overall
behavior and think about what processes must interact in parallel and distributed
way with each other to explain that behavior.
And then that's balanced by how could those schemas, as those units are called,
play out over particular neural networks of the brain.
And of course, sometimes the original schema model dies because it's not consistent
with the available neurophysiology, but there's a loop then of explanation.
And the other part is that at any time, we, I claim, cannot model every detail of the brain.
So we're always making selections as to which brain regions we will implicate in our models.
At what level of detail will we look at those particular brain regions?
And then as time goes by, we learn which details have to be added, which can be ignored.
So I looked first at the control of rapid eye movements, saccadic eye movements,
and stressed that we have below the sort of standard brain, the cortical structures,
there is the brain stem, the superior colliculus, which can take visual input
and control these movements.
But once we get into interesting things like, don't look now,
but you can look later, or you just heard two noises, look towards the first,
then towards the second, where you have to bring in memory and sequencing of
actions, then you have to bring in cortical structures.
And then you get this balance between the back of the brain,
the parietal system that seems to be saying, what do I need to pay attention
to that's relevant to my action?
And the front of the brain that's saying, well, what actions should I do?
And then we bring in another part of the brain called the basal ganglia that
do the scheduling or even scheduling of these actions. So that was the framework there.
And then I moved on to another system where, again, we have this interaction
of prefrontal and parietal systems, namely the visual control of hand movements.
Then I reported that my colleague Giacomo Ruzzolati and his group at Palmer
had made a discovery that within the premotor area involved in hand movements,
there was a subset called mirror neurons,
which had this amazing property that they were active not only during the animal's
execution of particular hand movements, but also when he recognized other hand movements.
And then that suddenly got interesting when we turned to human brain imaging
and said, well, we can't monitor individual mirror neurons in the human the way we can in the monkey,
but at least we can look for a brain region that lights up in a way that indicates
it might contain the mirror system.
This plus anatomical data converged to say that the area of mirror neurons seemed
to exist in the human brain in what had been thought of as a speech area.
What speech got to do with recognition of hand movements? Well,
we know there is sign language.
Language can exist in the manual domain as well.
This gets us into what is called the gestural origins of Jim's theory of language
that maybe, although for most of us speech is predominant, we all use co-speech
gestures to embellish our speech.
And so, I outlined a fairly elaborate,
I would say, network of models rather than a single model for what might have
been the 11 evolutionary changes from our common ancestor with the monkey 20 million years ago,
our common ancestor with the chimpanzee of 7 million years ago,
to build a system where the mirror neurons were still a core system,
but we'd also gone beyond the mirror.
How do we get from just recognizing actions to imitating novel actions?
How can our use of actions to practical effect on objects.
Provide the basis for pantomime and beginning to use hand movements for communication.
What social interactions yield a system of conventionalized gestures rather than ad hoc pantomime?
How does speech come into the picture so that we can move from purely gestural
proto-sign to a proto-language that is in the spoken domain?
And so there we got fairly elaborately into the back and forth between what
happens in in terms of biological evolution,
opening up new possibilities for brain activity and the cultural historical
development of the human species which finds new ways of exploiting the brain
that were not exploited before.
One of the concluding suggestions was to emphasize the notion that it's probably
not the case that our brain evolved to give us language in the sense of a big
lexicon, lots of grammatical rules and so on,
but it rather evolved to allow us over many tens of millennia to discover more
and more aspects of what now constitute what we take for granted as part of human language.
That's a great summary, Michael. It's a good thing I was listening.
But what was interesting is, what's now the role of schema theory in this second part?
Should we consider these as two separate proposals, or did the schema theory,
the idea of schemas that you have been pushing for quite some time,
give you leverage to look at this mirroring system and to think about language?
So what's the relationship between these two?
Okay, well, two parts. When we were looking at the control of hand movements,
then in fact, the idea that we made a preliminary analysis, what do you have
to notice about an object to be able to interact with it?
So there are perceptual schemas for just the shape of the object.
Because the details of that are going to be very important to the shaping of
the hand, recognizing the location of the object, very important to how we move
the arm to get the hand into place.
So we had, therefore, the shape of the model was in terms of what are the perceptual
schemas to know how to interact with the object, but also what are the perceptual
schemas to know about the object.
We recognize something as a coffee mug, then we can call on knowledge about
the use of the handle to lift it.
Whereas if we had a nonsense object, we wouldn't be able to call on those more
meaningful schemas, and there are some correlates of where in the brain the processes might occur.
So our big analysis of visual control of grasping is essentially a schema level,
and then we said we can now go in, because we have recording data from the monkey,
where we can begin to say, where are those schemas computed?
How do we have to modify our understanding of those schemas to see how different
brain regions must interact to support it?
So in the case of visual control of grasping and then bringing in the mirror
neurons, well, that's where we had a very solid neurophysiological data to then
reflect back into the schema-level theory.
That's part one. But part two is when we turn to language, we essentially have
zero in the way of cellular data.
We are at the level of analogies from the monkey brain to the human brain and
homologies from the monkey brain to the human brain.
But our other data are just people have a brain lesion, the system doesn't work
so well, or we do some brain imaging and part of the brain lights up.
But the trouble with part of the brain lights up is that it's just giving you
information that one part of the brain is perhaps more active in task A than
in task B, but it doesn't rule it out as being vital for task B.
So we now develop, as it were, our informed data analysis is at the level of
schemas. How are the different processes?
How do I know what a word is? That's a schema-level description.
It's not yet a neuron-level description. But what we're hoping to do in future
is to say that because of what we've learned from the monkey brain,
we can make informed hypotheses about the circuitry in the human brain for which
we don't have detailed neurophysiological recordings to come up with better
and better neural models.
So in the end, we can render a consistent understanding at the schema level
and the neural level that embeds our understanding of hand movements for practical
ends where we can share a lot with other creatures with this refined use of
language, which is particularly human.
But would you equate a schema level with a computational level or an algorithmic
level, or how should I relate these levels of description, these constructs? Okay.
So if we look at computers, there is a machine language, which is the basic
language of zeros and ones.
And then above that, there will be something like an assembly language,
which allows you to say, well, how can I think of these patterns of zeros and
ones as recognizing letters or symbols or patches of pixels on a graphic screen?
But the language that people who program in is a level up from that,
something like Java or C++, which is using relatively high-level constructs,
and they don't know actually how that plays out over the hardware.
And then for most of us, we're at an even higher level where we just have an
app which somebody else has programmed at that level.
So schemas are probably describing computation at that level from the high-level
programming language up to the app hierarchical levels there,
and then the neurons are the computations.
They correspond to the machine code in the computer.
So it's an intermediate to high-level description of how computations occur in the brain. Right.
But, Tony, so you said that you don't want to model the brain in all this detail.
Oh, I do, but I know I won't. Okay. So what are your criteria for deciding which
phenomena in the brain are important for informing the models?
How do you apply those criteria in the process of modeling?
Is it that you look for some key aspects of data that you try and switch at
your model and then extend what you're looking at to try and bring in more phenomena?
Right. So we now know a lot of details about the synaptic structure of the brain.
We know a lot of details about how different molecules provide the ability of
a synapse to take signals from one neuron to another and apply learning rules and so on.
So one could make a model which basically loses itself in just the details of one synapse.
Or one could be a little simpler about the synapse and blow a whole supercomputer
on just a few interacting neurons.
And for some people, that's the career path. For me, it really is starting from this cognitive level,
visual control of hand movements, visual perception control
of action language and so
there i'm taking a sort of
survey approach where i say what is known at the neurophysiological level of
correlates what is known at the psychological level how can i make a preliminary
model perhaps just using pure schemas to make sense of the psychological data
now how can i constrain that to meet the neurophysiological Now,
how can I refine those schemas so that they not only yield the behavior and
how the behavior is damaged by lesions, but also can give me explanations for
how individual cells are farmed?
And then the literature just keeps pouring in.
Filtering in a perhaps not very intelligent way of saying oh
here's a new paper that looks really important i have to be
able to either show my model can explain it
or expand my model to be able to address those data
here's something else to somebody to another
person that might appear very important but i'm finite
so i'll hope they'll address it and i'll have
to leave that out so it's a it's opportunistic once
the first set of big models is in
place i think so you're interested fit in these sort of cognitive behavioral
phenomena and your goal is a decomposition of that
task that you observe the personal monkey doing into computation
elements that you call schema and you're not necessarily at that first stage
particularly concerned about mapping the schema onto the brain is that no i
would say that no i i'm very much engaged in mapping it onto the brain but what
i'm suggesting is if you just look at the neurons and try to make sense of them.
You may not succeed so by starting with a hypothesis about what the schemas
are you're not saying how is this complex thing being able to speak english,
mapped to neurons you're saying here is
how do we recognize a particular auditory profile
as a word for example then that's a tractable problem
and recognizing the words of a vocabulary would be would be schemas within a
language understanding system for example so the notion is that you go top-down
from the psychology and the behavior to negotiate what seemed to be the necessary,
intermediate-level processes.
Then you use whatever data are available to say, but I'm not happy as a neurocomputational
type to just say I've got schemas as abstract computational processes.
That might be enough if I'm building a robot to say, okay, that's a good architecture for the robot.
But if I really want to understand the human brain, as indeed I do,
then I don't rest with the schema analysis if there are data available which
will let me be more explicit about how plausible neural networks in the brain
will actually implement those schemas and that means the original schema level
model may get restructured to accommodate more neural level data.
In a comment on another talk you mentioned that you weren't entirely happy with
the split proposed by David Mark which is algorithms and the implementations.
Your suggestion was that if you want to understand the brain algorithms,
then you might want to come through the implementation.
So that's what we're talking about. It's looking at what we can see about the
implementation in the decomposition of the brain and then say your schema-level
system sounds like the algorithm, but it is informed by... Right.
So I think the problem was that a lot of David Maher's writing and a lot of
the way people quote his statement is the idea I can specify the problem,
then I can come up with the algorithm, and then I can implement it.
The point is that if we want to understand the brain, you've really got to look
at a dynamic loop where, yes, you already understand the behavior and have the top level fixed,
but the algorithm is going to depend so crucially on whether you're using neural
nets or serial computers that that's an ongoing loop.
I will have a schema model as an initial algorithm.
Then I will see how well I can implement it. The feedback from that may change
my schema level model so I have a loop of understanding.
But in that approach, you also were using this concept of causal completeness
to guide your choices with respect to the constraints you want to consider.
So, but how complete can you actually be in reality and in defining these kinds of models?
So, is causal completeness a hope or a reality of building? Oh, no, it's a reality.
I mean, the point of a computational model is that it's causally complete in
the sense that when you provide the appropriate input and you build on the appropriate
memories, you get the observed behavior. behavior.
But if I'm causally complete with respect to, let's say, an analysis of saccadic
eye movements, that same model is not going to be causally complete with respect
to arm movements, let alone language.
So it's not going to be causally complete for every possible behavior.
It's causally complete with respect to that behavior. Again,
the level of description that you start with will determine if I'm looking at
subtle learning effects,
then I may have to go back and iterate the model to include data about synaptic
plasticity to understand the timing of learning so that the causal completeness
is not saying I have covered everything in the universe.
The causal completeness is saying that where the experimentalist could just
go in and say say, I'm monitoring activity in a part of the brain,
and it correlates with some particular behavior.
He doesn't have to say how the sensory stimuli got to the point where they could cause that.
He doesn't have to specify how that activity could get to the muscles to yield the overt behavior.
He's just saying, here's a fascinating correlate. I have to say.
The model is causally complete in the sense that if I stimulate my model with
a representation of the sensory stimuli,
then my representation of those cells will fire in that way,
and I have a network which will show how that emerges in the behavior.
So it's causally complete with respect to the level of description.
Right, and also then given the assumptions I have made about the primitive elements
that are playing the key role in my model, that you say, okay,
below that level, I don't need to go.
And again, the point I made yesterday was that in some learning models,
we have the idea that we have an input signal and a training signal,
which tells you to remember that input or to change your response to that input.
And a lot of those models are at the event level.
So you say, here is the input at this event, here is the training signal at
this event, what's the output?
And they were saying in some cases in the real brain, though.
The output might be planning a movement, and the actual work by the muscles
follows later, and then the observable effect of that result.
So the training signal might well be 200 milliseconds later than the brain activity.
And then that forces me to say,
what is going on in the brain that bridges across that fifth of a second?
So there I was forced to look at details of synaptic function that were not
engaged in the initial model, which was just event by event,
rather than looking at the actual time course of action.
Coming on to your later work stop eating or lie like a shell so just can you
just clarify why you moved into this area of the evolution of language from
trying to understand how the brain implements some of these very important,
actions and cognitions and perceptions that you've now it's actually a different
it's not that I moved into it But firstly,
as a schoolboy, I was very much intrigued by the history of the English language.
So that has always been an interest. Then in my mid-career at the University
of Massachusetts in the 80s, I helped found the Cognitive Science Program,
where I worked with the linguists.
So therefore, the forging of connections between my work on visual control of
action and their work on language became a very important topic,
and we had about four PhD theses on that.
Then I moved to the University of Southern California in 1986.
I got busy with other things. And then when Ritz-Zolartes' group discovered
the mirror neurons in the early 90s, I was already working with that group on
visual control of hand movements.
And my group at USC did brain imaging, which established that the activity that
looked like mirror neurons in the human brain was a broker's area or a speech
area, which we now understand more as a language area.
That provided the path after a 10-year break back into the study of language
because now there was a really strong connection that exploited what I had said before.
In fact, from 1979 to 1985, I was publishing in the area of models of language.
The work you're doing now is always anticipating the models.
The modeling is is pulling behind in some sense, or has it caught up with where
you are theoretically in terms of your ideas about language evolution?
Or is that the way it's always been, that you've always had the theories first,
and then the model followed on?
No, I think, again, it's a loop that at times you raise questions and form hypotheses,
and then look for data to test the resulting model.
At other times, you're confronted with a body of data and you're trying to make sense of it.
In the case of language, I think the state of neurolinguistics is very fragmentary
from a computational point of view.
So here we are establishing a range of models that we hope will begin to fill in the landscape.
But meanwhile, the overarching model I have is a conceptual model,
not an implemented model.
But it is already engaged in many conversations, talking to primatologists.
What do we know about communication in monkeys and apes?
How does that make one set of hypotheses more plausible than another at the conceptual level?
Looking at people working with sign languages,
How do we change our view of what language is because we realize it's not speech,
it's a more general capability?
And then just getting into debates about what is the nature of language.
We have Noam Chomsky on the one hand looking at syntax as an abstract structure.
There are other people who are looking at language as a flexible means of communication.
So I find myself moving over there to saying, how do I capture certain aspects
of that more action-oriented approach to language that I can now bring back
to the brain in a way that I think is consistent with my other work on the brain?
So there's a great deal of work going on to create this framework in which some
models already exist, but in which we're also defining spaces for future modeling.
But if I look at this from the outside,
that does give the impression of a discontinuity from a very much action-oriented
view with the schemas to this view of the mirror system and language that actually
seems to start with the primitive element of gestures or any communicative actions.
So it seems to me that there's a discontinuity between actions as in relation
to objects in the world and now communicative gestures as the starting point
of developing language. So how should I relate to these two?
Well, my joke is that my work on language evolution is to replace one big miracle
by a series of small miracles.
And so what I'm trying to say is that getting all the way from a monkey-like
brain, that's a simplification, but let's say from whatever our common ancestor
had 20 million years ago to today in one leak is too much.
But if I can break it into, okay, getting from recognizing other actions.
As one similar to those they already have, to understanding other actions as
a means to imitating them, to being able to understand complex actions in terms
of the structure of goals and movements,
then these are reasonable miracles in which to address specific modeling, as we are now doing.
And then again, the transition from the use of these actions for praxis to others
imitating them to get praxis, to being able to build pantomime is another small miracle.
And then once I get to pantomime, then the social ritualization of those into
symbolic gestures is, again, a meaningful step.
And then once I've got that use of arbitrary gestures for communication,
bringing the vocal apparatus back into play is a reasonable thing.
So in some sense, what I'm doing is, as I say, I'm breaking it into miracles
that are small enough that, yes, they're discontinuities. Evolution is a discontinuity.
We have forelimbs. Birds have wings. We have a common ancestor.
So there's always going to be divergence points.
But the issue is, how can we define that in a way that it becomes plausible
that a relatively small suite of genetic changes could support that divergence?
So, at the moment, I'm trying to consolidate data from many different disciplines
to come up with what I think is a plausible set of bifurcations in our evolutionary history,
and then to build before and after models.
And say, this is what the brain was like before, this is what the brain was like after.
And then hopefully, in the end.
Evolving work in genetics and molecular biology will catch up and say,
we can begin to understand the genetic correlates of the changes in neural architecture,
brain architecture, that you've posited to provide that set of stepping stones
from the common ancestor of 20 million years ago to the language-ready brain of the human.
So to get to a conclusion of our short interview, I have two generic questions.
So you are in this field now for a really long time.
You trained with some of the heroes of the cybernetic age, Wiener and McCulloch.
So in your long experience in this field, what's the law of RBEAP that we,
sort of younger, representatives of the younger generation should take on board
to actually help us understand the brain and cognition? Yeah.
Well, I think schema theory has been a very minor theme in the field,
and I have a certain pride of ownership, and I think it should be a bigger theme.
The idea that there is a functional decomposition to be placed in conversation
with a neural decomposition.
Often we'll get something like, well, I want to look at vision,
and oh, I want to look at stereo, and then I jump immediately to the neural networks.
And the idea of thinking about stereo in terms of how it contributes to an overall
set of interacting schemas for vision in the service of planning behavior gets lost.
So I think my big lesson is I want people to think more about how to develop
schema theory as a high-level, if you will, brain programming language,
which can then be either played out on circuitry for the design of robots or
put in conversation with lesion data.
Imaging data, neurophysiological data to try and move into very complex systems
where approaching it from the level of,
here's a lot of synapses or here's a lot of neurons is doomed to
failure and i think as as we go into
this world of really large integrated
systems with complex suites of behavior this
will become more and more necessary than it has
been in the past okay and then the concluding question
for me would be um if we're going to meet up again five years from now you know
in science it's all about prediction so what's what's the prediction i can hold
you to five years from now whether we can find out whether it was true or false
what is one prediction you want to stick your neck out for today.
I don't really have, in the sense of a limited prediction.
It's rather what I've laid out, especially in my second talk here,
has been a framework for the study of how the brain supports language that is integrated with,
despite evolutionary divergences, from the way in which sensory data are processed
to perceive our relation with the world and move us on.
It also has emphasized the way that neuroscience has to go from a focus on the
isolated individual responding to sensory data with a course of action to more
and more thought about social interactions, a trend that's already started.
So my prediction is a very wishy-washy prediction.
It is that five years from now, the framework that I have presented to this
point will still be seen as correct in its overall structure,
but there will be a lot of very specific models,
embedded within that structure, which will change some of the details,
but not change the overall conceptual framework. Very good.
Michael Arbique, thank you very much for joining us, and we hope to see you
back very soon. I look forward to it.