This is the Convergent Science Network podcast. Leading researchers in the domain
of neuroscience, brain theory and technology are interviewed by Paul Verschoor and Tony Prescott.
Paul Verschoor with the Convergent Science Network podcast here at our Barcelona
Cognition Brain Technology Summer School 2018, together with my colleague Tony Crescott.
And we have Lars Mugli here. And Lars, welcome to the podcast.
So Lars, you spoke about internal models and counterfactual cognition predicting
our environment. And as the title might suggest, prediction was very much at
the center of your presentation.
So why do you believe that prediction gives us sort of this strategic lever
to understand how the brain works?
I think a few years back, I kind of tried to conceptualize everything I know
so far from visual processing, let's say 10 years ago.
And I thought that at some point we have a generally accepted narrative of a
visual hierarchy that goes of increasing features represented,
for example, in the visual system.
And in this story, I felt that there is a particular capability missing.
Missing um if let's say
we are driving around in a car and um
we this would be
for our brains in this conventional narrative
kind of an overdrive there's lots of
varying stimuli there's lots of um
signal that needs to be attended to that
are competing for attention it is integrated
bind bound together these complex features and yet it is possible to effortless
navigate around and have thoughts about your holiday about your next research
grant about complicated um.
Conceptual theories and thoughts
and so i thought if we use
the kind of narrative that is coming out of describing early visual cortex and
then higher stages of visual cortex as the template of how the entire brain
works we have a very busy and buzzing neuronal system which is combining features.
Um and we run out of brain space before
we have an explanation of those really exciting
things that we are doing so i thought something isn't quite right and um i think
predictive processing framework brings to the table a narrative that is refreshingly
different by By saying, well.
It makes sense if you do something like driving in the car,
where lots of those features, even though they are quite complex,
and even though they have spatial temporal dynamics that from a bottom-up system is complicated,
complicated um you can see that a hierarchical system
is well equipped to explain that away to to
say to focus on only the surprising light
that comes up and signals that you're out of
petrol and ignore the commercials
that are running by and these kind of things because lots of
that can be explained away by different um
precision explanations you you're
expecting something to fly by you're expecting commercials
to try to attract your attention and
you ignore it so so i think there's a lot that
can be brought to the table by this top by this global narrative so i mean that
feels also a bit an idea of how do you let's say optimize information processing
by actually just looking at your errors so that means stuff you didn't expect
Yeah, I suppose the stuff you expect. Yeah.
But was there also something in the data that you were working with at the time
that was suggestive of moving in that direction?
Was there sort of a parsimony in the data that you felt needed to be explained in those terms? Yeah.
So 10 years ago, we have worked in principle with very simple stimuli that looked at a pair in motion,
which is from one perspective, maybe the most simple visual illusion you can
think of because it only requires two dots that blink in a certain spatial temporal
pattern so that your visual system combines them to the illusion of a continuous illusion.
Movement from this one location to the other location and if it's long distance,
it may appear as one jump but it's still one object and.
Commercials work with this, you see this all around, it's everywhere.
But it's a very simple illusion that we looked at and that we used to show how
the visual system integrates this information and feeds back to early visual cortex.
So that was an early finding. finding and
let's say I started with a surprising finding of some feedback in the intermediate
space between these dots which have which induce their prayer motion but which is then filled by,
some activity and we try to make sense of this activity so it was actually driven
by an experimental result from fmi that this intermediate space was filled up
with activity that we then thought of how to explain that which then brought
us to several competing hypotheses one was.
Related to consciousness and representing
of an item and the other was more
related to predictive coding framework which we
then followed up with several experiments which we
then came up with a sampling strategy to test
predictive processing which led us to the conclusion that there is a predictive
model that is projected down to early visual cortex and and that's how we started
and the the story i told you about you know 10 or um.
12 years ago or so, was when I then left Frankfurt and went to Glasgow.
And I was thinking of new grand ideas that I put in grand applications.
And coming from this very simple stimulus, I wanted to make it more realistic,
using more realistic complex stimuli and using also more interactive subjects.
So we created one extrapolation was including eye movements.
Another one was using complex stimuli and looking for predictive processes with
these more complex scenes.
Yeah, so I'm interested in how
and why you arrived at the idea of using apparent motion as the paradigm.
As you say, you're motivated by this predictive framework and you began by describing
some ideas from people like David Mumford and his active blackboard idea that
there is a sort of representation of what you expect to see in the world in higher brain areas.
And you also described a study from Hanson et al., which talked about,
if we're watching snippets from a Hollywood movie, we build up this picture
in our heads in the higher areas of the visual pathway of the meaning of the story and so on.
And this feeds back and influences processing in the lower visual areas.
But the task you choose to focus on,
apparent motion is of course very well known in psychology from centuries back
the Gestalt psychologists found this and focused on it and really talked about
it as a way of thinking about.
Without perhaps too much theoretical insight into it, but seeing it as such
a dynamical process that's happening in the brain.
Not so much necessarily a hierarchical process, though.
So it is a bit of a leap to go from a gestalt process like apparent motion to
a hypothesis about hierarchy. Okay.
Interestingly, I think when we started brain imaging, and that goes back to
1996 when I joined Rainer Goebbels'
lab, But apparent motion and other Gestalt laws were kind of the first start
of how we wanted to look into the human brain to see where these Gestalt laws are represented.
And motion was something that we started with.
Actually, the second experiment was imagery of motion, and we're surprised to
see that that worked and activated V5.
And while we did these apparent motion experiments,
one was about bistable apparent motion and the switches between perceiving the
motion content and not perceiving the motion content was something that you
can follow in the activity of V5.
But then we found something in the data that actually V1 knows something about,
gets informed about some of those integration processes.
And it was with Nico Kriegesquarte at that time that we were late at night discussing
what could be the kind of feedback along the trays in the apparel motion,
how that could be informative.
And we came then to David Mumford's kind of seminal papers of the 90s in which
he motivated this active blackboard theory, not totally.
And being explicit whether this active blackboard is a subcortical thalamic
structure or primary visual cortex,
but the idea that these expert areas kind of converge to their best guesses
in a scene and scribble this onto a blackboard to kind of negotiate the scene
and the surprises within the scene and so on was quite attractive.
And we thought okay how can we test this and um
and i think that started the the following experiments and the paramotion is
good in the sense that you can have the inducing stimuli far away from a region
in the middle where you have the illusion and since the the the early part of my brain imaging,
I always use retrotopic mapping and.
The trick that the spatial separation of components can be done very cleanly in fMRI,
maybe better than in some other methods so in EEG you always have everything
together, you have a better temporal resolution but the signal can be can be separated so well.
And in fMRI, one of the advantages, you can do individual maps and then get
the single signals out of those retrotopic components.
And I think it's this tool that I wanted to explore and use more and more over
the years up to our most recent finding where we can see that in more complex scenes,
this occluded part or a rhizotopic space can show that you have something like
a mental map or mental hypothesis drawn out.
I think one of the interesting issues here is,
and that came out in the talk, was how much of the kinds of constraints that
the brain is using to do something like a parent motion tax are going to be
implemented within a level of the hierarchy and how many of them are going to
be higher levels of the hierarchy.
So I think most of your talk was about how the upper levels might influence
the process in the lower levels.
But you presumably would also recognize that there are some processes happening
within V1 itself which are going to do things like gestalt properties or really
encourage things like completion.
Yeah, I mean, this is a very interesting feature of using a parallel motion.
Motion, because the motion that we trigger is a very fast one and a long distance one.
So it's a 12 degrees visual angle, it goes to 60 degrees per second.
Those are features that we, one, cannot process and high visual areas like V5,
if it's considered or to be higher,
but specialized visual access trial areas are specialized to process tuning
along these motion energies, high speed,
long distances.
And so it's a particular good case in which we can look into V1's typical feed-forward features
and see the addition
of complex features that aren't
typical to v1 if you want to say in
a different way v1 has a certain language to
speak and um we we kind of test the multi-linguality of of v1 because other
other areas speak in a different language and the question is Because some of
the interesting and still not totally resolved questions is,
are these areas translating 2v1?
So that is v5 translating its prediction
and explaining 2v1 in a spatial localized way at a very high speed?
You expect a dot to see here, here, here, here, and here, and this and this orientation?
Or is it more of a kind of envelope translation to say there's high energy motion
in this part of the visual field,
but your knowledge about the concrete incidence, orientation,
and so on will combine to a better understanding of the situation.
And that's, I think, the kind of model that I have in my mind.
There isn't a full translation. I touched upon this with mapping the precision
of the filling in in complex scenes.
There's a line drawings, but the precision is around four degrees.
V1 has a precision of one degree. So it's more like saying there's a line somewhere
around here, or there's a motion of that kind of speed somewhere around here.
And this is what top-down can give as a prediction and combined with the lateral information saying,
oh, there was a dot that just disappeared here together with the energy prediction up there,
I can now make a very precise prediction.
By combining those two constraints.
But before we get there, because that's sort of after a number of experiments, right?
Right. And I think that the first question is whether, given the data you have
on this apparent motion paradigm,
whether this predictive mind hypothesis or the active inference hypothesis,
whatever you want to call it,
is the most parsimonious explanation of the data that you have.
Because in what you showed in your first experiment, it had apparent motion.
So we have the two fields are being stimulated in a certain order.
So you have this idea that something moving.
And now you're going to flash a stimulus in the center area either synchronized
with disappearing motion right or out of out of sync out of phase with this apparent motion,
uh where you you look at the time difference of one frame right so this would
be about 15 milliseconds or something like this i don't know so it's a very
slight deviation right of the predicted path.
And then what you do observe in your fMRI is that eight seconds after,
stimulation, either congruent or incongruent,
you see a slight deflection or deviation of the BOLD response in the area that
you stimulated with this predictor-unpredictor stimulus,
which is slightly higher the bolt response is slightly higher for the deviant
stimulus as compared to the congruent stimulus this is the main effect and the deviation,
the bolt signal is of the order of about what 8 to 10 percent,
something like this right and it's also slightly enhanced in both cases relative
to the baseline response,
so now Now, why do you believe that this predictive coding into the power hypothesis
interpretation is the most parsimonious explanation of the data?
Andy Clark has a wonderful cover picture in his book on predictive processing,
I think, or it's called differently.
It's Surfing Uncertainty.
And if you take the picture, we're in Barcelona. I've seen the beach this morning.
The model is a little bit like this. You have this wave of motion energy, of prediction.
It is translated to something that needs energy in V1, like a prediction.
And then you place on this wave a surfer that serves the waves and then is efficient
and comes through and is detected. That's the dot that's in time.
And the surfer that has a little bit of a timing problem won't make this and
will fall off the wave. And that's the prediction error.
And that makes a small difference. And the surprising result is that this one
frame difference is detectable in bold signal.
And my most parsimonious explanation the way I think is,
yeah, you see at this moment the mismatch between the prediction and the surfer,
the dot that doesn't quite make it to be within this envelope.
Now, as I said earlier, this envelope moves very fast, 60 degrees visual angle.
That's faster than any motion detector on B1.
B1 has lots of lateral connections, but I doubt that they have the speed to
process this. But I'm not opposed to this explanation.
I've had several discussions. People are on both sides saying,
well, there could be a contribution.
And in the model that I just pictured before, I do think there is a contribution
of lateral interaction given the feedback signal of a certain motion envelope.
And giving the neighborhood relation of the just disappearing dot.
Gives the high precision of of this well i
would say the long range long range lateral interactions in v1 are iso orientation
right so neurons that like similar features linked together over long distances
and that would of course be a perfect substrate to exploit to get a fair motion because,
the activity will be very actively guided along neurons with a similar response
tuning to move in a certain direction, right?
So this would give you essentially a kind of entrainment response,
right, to explain the parent motion, which might be also more in line with actually
the Gestalt ideas, right?
So can you… For this model, that's
why I like to use the image of the
paper of mock
I'm blanking on her name but and who did their promotion with the gratings and
the filling in is inducing a new feature which is a new orientation which is
not just neighborhood relation it's a smooth transition now.
Maybe the case can be made that the lateral interaction smoothly goes from one
orientation over space to bleeds into other orientation.
But it's not such that the feature that's picked up on the paramotor trace is
just replication of the neighboring feature in orientation.
So to me, it looks more as a constructed feature.
But that would just be a matter, as long as I take enough freedom to fiddle
with the topography of the lateral interactions, I could also get this apparent feature, right?
But another answer, of course, could be you could still say,
look, you see, we agree, because the predictive model still holds,
it's just that the substrate is also included already in V1 wiring.
But for some reason, in your view, you want to see it like a hierarchical system, right?
And without ascribing a big functional role to the local structure of V1 in this case.
Well, you also know that 97, 98% of the synapses in the V1 volume originate inside V1.
Only a tiny fraction comes from outside V1, right?
So, do you believe that this tiny fraction of long-range projections coming
out of V2 or out of your thalamus or other cortical areas are sufficient to
carry your predictive model?
So, there are two lines of evidence where we really looked at this interaction.
One that I presented, one that I didn't present.
And maybe to add the one that I didn't present, we used an EEG experiment.
Again, with a prior motion design.
And we found two components that have energy components.
Motion energy components as compared to flicker to
flicker and those components were around 100
milliseconds and around 140 milliseconds after
stimulus onset and we then had the question which one is from retinotopic regions
and which one is from v5 and what we found is we applied the a paramotion in
the upper visual field and in the lower visual field,
which induces an EEG component in the dorsal and in the ventral stream,
so they have different orientation.
And we subtracted one where we had a paramotion in the upper visual field from
in the lower visual field and found that the early component was subtracted away,
meaning that the motion sensitive area V5, which in both cases has the same localization,
is the early one, around 100 milliseconds, and the later one,
the 140, is the upper and the lower of the retrotopic.
So the retrotopic component comes 40 milliseconds later.
So the other experiment I showed was the TMS experiment at which we stimulate
V5 50 milliseconds, 40 milliseconds before the onset of the flicker,
which is on the power motion trace.
And this TMS takes away the predictability effect on the power motion trace.
So both of those indicators, I think, speak more to a communication between V5,
which is tuned to process this high velocity motion across huge space.
And the more localized features in v1.
And so therefore I think it's a good paradigm because.
Different features are processed in different different regions and and they
are naturally optimized for this and you can see this kind of interaction um
but now it's an alternative,
for for your two experiments i could also argue look see your first experiment
you show v5 leading v1 in this response right and um then i could argue about
maybe you need a minimum volume of response actually be detectable with your methods of fMRI.
This kind of, given the heavy convergence to V1, you will only get that initial
critical response that is detectable at the V5 level and after the V1.
So it's more an artifact of your measurement technique than really reflecting
the underlying dynamics. Can you exclude that?
So it's true that we usually have these block designs, but in the EEG experiment, we measure these.
Short, I mean there's 100 milliseconds after onset of the apparent motion energy
component as compared to blinking for example and also in our experiment where
we combine apparent motion with saccades it's such that it even works after
the saccade in the new hemisphere.
Which Which, if I understand you correctly, you're saying you need repetition
over time to build up the V5 signal.
That's convergence. That's plain convergence.
That you just have a sufficient drive onto that volume of cells to become really
a clean signal that you can extract with your method.
As opposed to a more distributed, diffused signal in V1 that is more difficult to detect.
Well yeah i mean if you like in if if you take v1 after the saccade a blinking dot,
there's the inducer the last stimulus of their paramotion
and a blinking dot next to it the target those two are always perfect a paramotion
if that would be the only event if v1 in the new hemisphere would process that
on its own they are as much related to one another the in time is as much related as the out-of-time.
The only thing that makes these two stimulation conditions different is the
history that was processed in V5 and in V1 in the other hemisphere.
So it's bringing this history to this new situation that creates a difference.
It might be true that it needs to build up over time, this hypothesis in V5,
because a prior motion is a peculiar situation.
If you have just two dots and you perceive them as a prior motion,
there's no other way describing that as postdiction.
You need to, after the second event only, you can understand that it was a motion
and every point in between would be integrated at a later time point.
So it's like you're reversing the time frame.
In our experiments, we have always had eight iterations of a pair motion.
So that takes care, the visual system kind of has time to catch up with the
delay in simulation to then run in time.
Time if the famous experiment is if you're doing a pair of motion you take away one stimulus,
so it's repetitive and then you take a
stimulus away you will still continue to see a pair of motion because your visual
system is now so predictive that it fills in this that it can catch up it needs
to do post-diction of the missing stimulus to to realize so your fair motion
is there but the question is do we really see a strong contribution to the phenomenon
at the signal level of something we might want to call a predictive model, right?
So then also for your second example, you could also see where you talk about your TMS experiment.
So look, if I zap V5, I can sort of disrupt the error detection signal.
This is how you would interpret it, right? And then the error detection signal,
the deviant trigger signal signal ends up roughly the same magnitude as the congruent stimulus.
But that's, of course, I could still
argue, but that's at best a necessary condition of a predictive model.
And it doesn't prove in any way that what comes from V5 is a predictive model
that is processed into an error at the V1 level. Can you exclude that?
I'm struggling to find a way in which we could prove that. your distinction how,
so which aspect are you,
questioning the content of the model so to say V5 speaks to V1 and okay,
and the content of
the message has something to do with the power motion and And are you now saying
there could also be a contribution of lateral interaction in V1 that contributes
to creation of this complex prediction that we test?
Right. So this is an alternative explanation that so far you couldn't exclude yet.
No, I'm not. Except that you're saying, well, but I have a TMS experiment.
And then I'm saying, yeah, but wait, I could still have all this process playing
out at the V1 level. We know anatomically V5 projects back to V1.
So if I zap V5, it's not so strange. Something happens to the V1 level,
but it doesn't tell us anything about whether this is a prediction being fed
back or just there's an anatomical connection between the two areas and signal exchange.
So I think Jeff Hawkins uses this kind of explanation of this hierarchical predictive
memory prediction framework, which is as you know, you...
At every level, the neuronal microunit
is trying to explain its own activity given the surrounding stimulus history
in the context of a message from a higher level top-down prediction.
So and the example
he's using is like if you have a melody then
the higher up areas in
auditory cortex would tell you what is expected to continue in this melody and
giving the local information of transients at this lower level and giving the
envelope context from higher you create these neurons kind of create what's
most likely to happen next.
And so it's within this kind of model that I would also, and there's actually
a good model that does that for a pair of motion or for motion behind occluders,
where it does exactly this, that the top-down is the kind of envelope and the
lateral is the additional information that then converges to a precise prediction.
So, the example here being you have a motion of a certain energy disappearing behind a.
Occluder, and the way it's modeled is there's a motion energy expected in this area, and.
But you can't be very precise where it is.
The disappearance of the dot gives an additional constraint.
And these two information together make a very precise prediction of the trajectory
of the dot. And that's the way I think about it.
So I'm not excluding the lateral interaction. So you're saying then that the
idea of a hierarchical predictive model is a functional concept that's not mimicked
or in sort of an isotropic way mapped to the anatomical hierarchy,
because it might be implemented by some classic confluence of recurrent projections
and local interactions in neural circuits. Exactly. This will be your point. Yes.
All clear. because also in that sense you
made the point and also resonates with
this the traditional view and that's also where
Hawkins I think sits would be rather dogmatic in that sense right there's a
top down prediction and this comes together at a lower level in the hierarchy
where it leads to an error there's a real comparison taking place between the
state that this lower level area believes is correct
it gets a reference as you want from a top-down area and now a comparison happens
and I have an error right and your data would not really reflect that in such a literal sense right,
because you in in essentially in your what you'll see a ball signal for the
matching the congruent stimulus also leads to a deflection of the signal as
does the incongruent stimulus although the deflection is somewhat different
right so that means it's not about the error processing So what do you then
think is being processed,
is being generated in that local V1 circuit if it's not an error as the traditional
model suite would predict?
Well, I think this is, of course, a very good point, but it seems like a point
about the labels we add to those coding differences, right?
So if a prediction is violated and the violation of the prediction gives a different signal,
or whether it's a confirmation of a prediction that gives another signal,
It is the combination of input signal and expectations that are combined.
And they can be subtracted, which is the prediction of the predictive coding proper way,
or it could be multiplied and resonate.
And it's very difficult to define the level at which we can resolve that debate
because I'm currently not sure how we would resolve this because at the neuronal
level, so what is the right level of description?
We have the microcircuitry within V1 that is a combination of excitatory and
inhibitory neurons neurons,
the error units that can be also a combination of excited or inhibitory neurons,
so it will be very difficult to resolve that.
But the narrative of the predictive processing framework is very clear about
those units in principle. So prediction error matters.
Reduction of prediction error or...
Is a currency that might be very useful.
And also in the predictive coding framework, you have the resonating,
the confirmation signal for a predicted signal that keeps an internal model running.
So you have all those components, And I think it's, you know.
It's going to be, do you have a suggestion how to solve that?
I mean, do you have, is there some variance that is not explained or that would
be more possible to explain by an alternative explanation?
I'm not so sure. Yes. Okay.
Temporal populating codes. But before we get there,
so I feel one way, if I would be a real cynic and observing your results,
I could say, look, you have done a fabulous job dismantling the predictive modeling framework.
Work, because the second thing is, also if you go back to the traditional idea
of, you mentioned Mumford,
and then we have a number of other people who have variations on that,
and before that also Barlow was talking about it, you would expect that in the
cortical circuit, and it was very literally mapped to the anatomy,
that in a cortical, people would put several layers in the cortex that would
then deal with the prediction and with the current state and how the error would be computed.
So very precisely to cortical circles, right?
But even if you do your layer-specific analysis, you don't necessarily see such
an asymmetry among the layers in their response.
There seems to be a very gradual distribution of layers.
Of the signal that you get in your task. In that case, it was more like an occlusion
task, but I think it makes the same point.
So in that sense, aren't you showing it actually goes on in these circuits,
might reflect aspects of prediction, but it's not necessarily playing out in
the way as anticipated in these more traditional models.
Would you agree with that? I'm totally experimental and not dogmatic, right?
So these are interesting questions.
I want to see how does traditional neuroscience looks at the stimulus response
to an unpredicted random stimulus.
And of course, in those experiments, you can never see the capability of predicted sequences and so on.
So you need to do the experiments in which the stimulus history is predictive
for a certain variation and so on.
And I think that's what we have done and others and whether this creates a,
I mean, the one point in which I deviate with the Rauer and Ballard model, if you like,
is that what we see is that the top-down prediction is creating a signal in
where there is nothing, right?
So predictive coding is a story in which you explain away as much data as possible,
but now we show that in the non-stimulated region, where there's nothing to
explain away, there's something created.
There's a model, there's a prediction, a created and used energy to put their
hypothesis to then be tested.
It um so in in this respect
it deviates and um and i mentioned flores
de lange showing similar results for illusory contours they
are created they are put into into a map it's an active blackboard framework
in which you know the chalk is picked up and to to draw something on the active
blackboard and say i you know it would make sense if there's something moving
and it would make sense if there's some information missing at these and these points.
In a global sense, that might be very useful and minimize energy in this sense
that it prepares the organism to respond to more or less surprising stimuli.
But it's not all about a quick explaining way of all energy in hierarchy as
it was shown by Rau and Ballard,
which is just a very simplified model to illustrate some principles of predictive processing.
Um, yeah, so combining with that, also what we, you also mentioned much more
detailed physiology that you did with Larkham and other people inside the O-Brain project.
And also there, you went over those results rather quickly,
but still at best they showed there is indeed interaction between higher and lower areas,
but it was not necessarily a clean signature of anything that might only interpret
like a strong prediction or or an error, or anything along these lines.
It seems a more non-specific sensory processing that depends on the active dendrite.
Yeah, so, I mean, this is ongoing data that is recorded and still analyzed.
Like, well, I think in the classical V1 IceCube model and following some claims that, you know.
You know, Jack Gallant would say, you know, it's almost explained,
you know, 80% of the variance is explained in V1.
We have a very good idea of our models about V1 and that is true actually for
in certain restrictions.
So it is in those models in which you have a strain of surprising stimuli,
it doesn't explain ongoing activity,
baseline activity, and these kinds of things, which are a huge contributor to
the energy level that is boiled off in B1 volume.
So it's maybe only the waves on top of that that are then explained.
Now, in a classical receptive field V1 explanation model, you don't have the
description of these feedback signals that I described.
Another one that I haven't described today, but we have done research is in
blindfolded subjects when we play auditory scenes, you have some activity in
V1 that is related to these auditory scenes.
Um, so the, the, the, the properties and the mechanisms in the envy one.
Are still having room for negotiations
of expectations and predictions in whatever kind of mechanisms.
And I think Christian Liefeld is doing a wonderful job in decoding also the
level of interneuronal activity that contributes to the detection and the explaining away.
And I think he has found a subtype of
inhibitor neurons that is very strong in exactly being silent during visual
stimulation and being active in between stages.
So it could be a role of working hard to explain things away that we still will find out.
You know, um, so there are many sub units that can still be,
um, contributing to this, to this model and to this compartment of visual cortex
that we think we, we understand best.
Maybe we want this thought to be one of those regions that are best studied and best understood.
And yet there's, there are like languages of feedback signals that we don't fully conceptualize.
You mentioned one of the results that you found was a difference between paranoid
schizophrenic patients and controls on this apparent motion task.
So what do you read from that in terms of support for your hypothesis about top-down predictions?
And also, what does it tell us about schizophrenia?
So we actually didn't find a difference between schizophrenic subjects and controls.
Not in the dimension that we expected it, in the predictability effect,
because we could confirm the predictability effect.
We saw overall a difference in the amount of a paramotion perception.
If you like, the responses to detection of a paramotion was different in controls in schizophrenia.
I mean, there's a general result, isn't there, that schizophrenics are less
susceptible to some visual illusions?
Exactly, yeah, like the hollow face mask illusion, and the same is true for paramosion.
But what we didn't find is that the effect that we see,
the advantage of a flickering stimulus that is consistent within a paramotor
context was more or less in schizophrenic subjects than in control subjects.
So the original idea was that it could be that the signatures of creating a
prediction error and the processing of a prediction error are altered in schizophrenic subjects.
Subjects, they are less tuned to process the prediction error,
or higher tuned. Actually they are both hypotheses.
But we didn't find that, so we basically just replicated this.
Now we are testing the same thing in autistic subjects, we're just starting
to do that, same experiment.
Some have suggested that it should precisely be the difference between autistic
subjects and schizophrenic subjects, in which you have this low-level difference
in visual predictability,
that matters in autistic subjects but not in schizophrenic.
So we will see that. But the prediction here would be that autistic pupils would
be less tuned in explaining away the predicted stimulus.
It would be as unexplained, as surprising as the unpredicted one.
And, well, it shows a tuning for predictability that is important and essential.
And we could think of it in the sense that if you experience lucid dreams,
it's maybe a situation you're aware, you're conscious, but you don't respond to prediction errors.
Uh you the the story you don't wake up and realize you are in your bed but uh
you continue your dream and knowing being in your bed and knowing to dream does not um.
Bring the dream to an end and and so that's that's maybe a situation that that
comes very closely to to a auditory hallucination in in schizophrenic subject
it is it is causing some prediction error They're surprised,
they're worried, but it's not resolved.
It doesn't make the internal representation
go away or being replaced and overwritten by an alternative.
SL. But now in terms of the actual data, in a schizophrenics issue,
we do the parent motion task.
How big is the deflection you would see in their birth response as compared to a healthy control?
What is the exact difference? Uh, so, um, sorry for having been so brief,
but in the, in the talk, I only presented behavioral results and we haven't
done functional brain imaging with him.
It's just a behavioral observation that they are not detecting the.
In-time stimulus as good as the... What would be your prediction in much respect
to what you would see if you would do fMRI on them?
Yeah, that's a good one, right? In the autistic subjects, if we find behavioral.
No difference between the predicted and the non-predicted flash stimulus,
we would like to do the fMRI experiment using layer-specific fMRI to see whether
then we have a continuous low-precision prediction in the superficial layer
that doesn't discriminate and doesn't trigger an error signal could be one hypothesis.
So another one would be that a prior motion is never creating any activity along
the trace, so that the envelope is maybe processed feed-forwardly,
but there's no feedback message.
So everything that comes in hits the clear slate.
So now, since I'm on this crusade to sort of demolish the predictive model,
another piece of the data that feeds that assault is the data that you presented
on the CIRBAN, which is really interesting, right?
Because when you opened up your analysis in your whole brain,
you also look at how other structures would be involved in this.
Now, soon the cerebellum starts to also show activity in this task, right?
So now I could say, aha, you see,
apparently this sort of clean isomorphic mapping of prediction hierarchies to
cortex is insufficient to explain the paradigm, to explain the behavioral effect, right?
So how does the results you get from the cerebellum not question this predictive
hierarchy model that we started out with?
So I had been a cortical chauvinist by accident, not by conviction.
So we just started out scanning cortex because cerebellum didn't make it into
our slab. We have changed that now.
And I'm still on my learning curve to learn more about the cerebellum.
How much I understand of it, or some of the hypotheses are that it is a machinery
that has an architecture to create a temporal forward model that is used in,
for example, in tracing behavior.
So seeing, you know, tracing a curve.
Monkeys can follow this curve by smoothly,
which is a visual motor integration task,
that by taking away a cerebellum, this movement becomes very jittered and incoherent and so on.
And there's a long line of different research, but so the cerebellum in a way
is not directly connected to the outside world.
It receives cortical input, processes this, and is the archetypical region for
an afferent copy processing, which is basically a predictive machinery.
And so, as I've heard in a recent talk, if evolution came out with a perfect
prediction machinery and is connected potentially to the entire brain.
What would you use that machinery for?
And so I think the conclusion is that lots of the forward modeling.
That is necessary for survival is done,
is offloaded, or is done under the participation with the cerebellum.
So it is, if you like, our predictive machinery that communicates then with
the different sensory and motor output areas.
Now, this was a funny one.
The Cervelo projections are mainly going to frontal parts of the cortex, almost densely.
So then to get that signal back into V1, we can take some cascade of recurrent projections.
I thought so. that there are some I think there are some direct connections
to visuals so there's some visual.
Cerebellum connections I'm not entirely sure if they go to V1 or to exostride
areas and so on but I think there must be something like a visual sub-part of
cerebellum Sure Okay so the last part of your talk talk,
went a little bit more in the direction of a more computational understanding
of what this whole system might be doing, right?
So the bottom line would be Cortex builds hierarchies of internal models that
are coupled through prediction errors.
Um and so that's
the starting that was a starting point and that's been a great heuristic for
you to to perform some some really fantastic experiments that
also led to to new insights but new insights would
also allow us to to reformulate the model right so so if you would have to sort
of crystallize that that current view what would be your your current summary
of of the model or the theory of of neocortex how should you think about neocortex
given the data that you have in your hands?
I think that, of course, one of the big struggles is to find the right level
of abstraction in answering your question in the description,
right? How many units do we need?
What kind of abstraction is useful for what kind of understanding?
Understanding we're still on we neuroscience is a data rich um science with
relatively little theories um at least that's what some of the criticism um
says about neuroscience so um,
and we haven't this is a question we haven't fully resolved which level of description
do Do we need spiking neurons explanation? Do we need principles?
And I think this predictive processing has some hypothesis,
some units, which could be in single neurons or microcircuits,
which is the description of prediction errors, for example, and important currency, I think.
And also the kind of um.
Population code of internal models so
some for more complex uh human
tasks it is a it is extremely important to be
able to simulate an counterfactual situation
to plan behavior to plan a interaction
or to plan a career we can think about
i mean there are many very fascinating explain you
know how how can how can you explain that a
biological system is capable of committing suicide how how did evolution make
it possible that you can you know you create an internal model that is somehow
rewarding that you simulate and say that makes sense and then commit suicide which
in biology is extremely unbiological,
right?
So there is only a few components that you need to be able to explain how internal
models can create a good description about external facts, right?
So deep encoding works well in the visual system, for example,
to now label visual scenes and learn the different objects by coming up with
a condensed description,
but just by learning an incredible amount of visual stimuli and trying to extract
the most essential features.
So I think there are some of those components,
that are sufficient to come up with a hierarchy of extracted features by using
prediction error minimization.
But I think what's unresolved so far, what I want to look into future,
if that was your question, is the kind of, how do you lift up this internal
models that are totally, that are then totally connected and well describing
the environment to something that is alternative,
that you can think, plan alternative to the currently existing outside world.
I was looking for an alternative to the classical model.
I don't really hear it yet, but Tony. Well, you mentioned the deep convolution
neural nets a couple of times in your talk and just now.
And, you know, sort of Paul and I are both interested in putting models of the
brain into robots and getting them to do tasks in real time.
So, and I take from some of the things you're saying that maybe there's some
useful mileage to be had in using these convolution neural networks works as
an approximation to what the visual pathways might be doing.
But, I mean, say we get that working in some sort of first pass,
what would be the things that we would be missing out on and what would we want
to add to make this a more realistic model?
So I think the flexibility, the
adaptation to new environments is something that is a potential of expanding
deep encoding neural networks that are feed-forward networks by expanding to recurrent looping.
Might be able to make them even more clever.
To, let's say, the context-dependent amplification and de-amplification is something
that could potentially,
once we find the right architecture, make deep encoding networks more flexible
to adapt to new environments.
So this is, I mean, you're more
the experts in this, and I'm sure this is something that's widely debated.
From our research, we just see the great difference between.
The big success story of deep encoding networks, convolutional networks,
that are very similar to the hierarchy in the visual system.
However, they're both feed-forward.
So they're feed-forward processing architecture.
And the only thing that's fed back is the error signal.
But if we take our research serious and add a feedback cascading network to the feedforward,
then what we get, for example, is a network that can fill in occluded images,
which is one of the examples that we have used in our lab, in which we see that
humans have line drawings fitted in by their visual cortex and our deep encoding networks,
connected to an autoencoder, so a U-shaped network, is able to then make predictions
about occluded objects.
Images that are currently not in sight.
And so that could be an architecture that is also.
If you like, an active blackboard of predictions, what is happening when something
disappears over time and makes the world enriches, so to say,
the environment in which the network is working.
But wouldn't that imply that we would need much more brain volume with our skull
and the size of a Skippy ball because for those models, if you want to bring
in any kind of invariance, scale, rotation, position,
you have to duplicate your wires, right?
So you would run out of wires very quickly. So would it scale?
Would that approach really scale, you think? I mean what we have done for the
U-shape network is just doubling the network right it's a,
so after the conversion it diverges and has lateral connections and to reconstruct
the images and that makes it to if you like an active blackboard architecture.
Nico has done it in a slightly
different way used object recognition task
in a network that is either feed
forward or has also lateral and
feedback connections this blt network and could
show that this kind of network architecture um
is better in recognizing overlaid numbers cluttered scenes and so on and that
architecture came from kind of joint discussions we had about you know the question
how can we how would you add what would feedback you know,
add to your feedforward networks? How could it improve that?
How could we design experiments that challenge the current recognition system?
And so cluttered scenes and overlaps was one of those examples that he tried.
And I think, you know, the Alex Neck kind of architecture had something like
88% correct or more, or 95% correct classification.
And then his BLT, bottom-up, lateral, and top-down network,
was a few percentage better or part of percentage or something significantly.
So it's small, but you need to find new tasks and new challenges that those
networks can play out and maybe their potential because they're already in object recognition.
That seems to be a field where they are ceiling now.
So coming back to the point, it might be that they are now more flexible,
maybe there are advantages in training rates or something. I'm not an expert.
So we need to add in these sort of top-down cascades and we can get some improvement
and presumably some sharpening of the representations in B1 I think one of the
times you said in your talk,
you know, you usually think of V1 as the bottom of the hierarchy,
but in a sense, it's much higher up because the information ascends and comes back down.
So you're reconstructing this high resolution version of the scene. But I mean, the...
You also talked about how we can apply machine learning algorithms or support
vector machines to read out from the brain to what the brain is actually seeing.
So it seems to me that in some ways we're getting back to what really was an
old idea about the visual system,
that the brain sort of, or the visual brain represents its best
guess at what's in the world in some almost literal sense.
The whole idea of the movie screen in the head, which was dismissed by everybody as naive.
But with these kind of approaches, with this notion that you're getting this
high-resolution map which is tuned by your predictions,
is there in some sense something like a representation in V1 of your best guess
of the high-resolution scene as we know it?
Um it's an interesting question right it's it's
something it it does make sense it's intuitive and in another sense it it seems
a bit um homunculus um it seems like the screen that yeah as you said you you
you can laugh about but you open the door but I told you I did,
I did, but this is totally improved.
I know. So I think, yes, we've won now looks to us as if you, um,
um, you have kind of mental line drawings, um, uh, put together and we want,
and maybe that's, that is, is, is a bit the language of V1.
And I mean, it's an incredible story. I think if you look back, Peter, um.
Patrick Kavanagh talks about this 10,000 years of neuroscience by describing
the, if you look at the line drawings in caves,
that is a form of communication that works.
And that's surprising because line drawings work even though line drawings don't exist in the world.
What we see or what's out there are textures and texture borders,
but from a texture border to come to a line drawing is an abstraction that seems
to work incredibly good for our visual system.
And since we were totally surprised that if we asked the subjects to fill in
the missing information, they came up with the same line drawing.
And maybe there's a clue because um we find
a communication that works it's so proved it's so
error prone uh proven
so so um you know if you
you try to do an error a line drawing of an
animal and um you're not happy with it
you optimize it until you're happy and you think
this is a good illustration of a of an
um deer or
so and then you present that to someone else and
the value the quality and the the goodness of fit to their internal model is
so immediate because we have the same kind of visual system and that opens up
a communication if we would open up this kind of communication with monkeys
you know they could start drawing out their internal models and you You know,
that's interesting, right?
This is a speculation we could have had in the 60s by saying,
oh, the language of V1 are contrast boundary contours, right?
So in some sense, what's interesting about this, as opposed to having sort of
complex predictive hierarchy, maybe you need the majority of machinery of vision just sits in V1.
And what these hierarchies are doing is basically, if you want,
just in a very coarse way, modulating this process in V1.
So the whole hierarchy is now collapsing into V1, essentially.
And all the rest does is say, well, let's ignore these bits, right?
Or, okay, let's attend more to that part. Without telling it precisely what
it should be seeing, maybe that's sort of a mistake people make.
What about the predictive model is therefore maybe not telling you exactly,
you should see this texture in this position.
It may be the prediction just tells you, look, there's something interesting
there. Check it out, right? Yeah. Can you buy that? note?
Um, yeah, I mean, there's, like...
What are we also experts in is face processing, right?
So if you can… Face with pH or… Oh, face with… No, the emotion.
We can read out the emotion and the gender out of faces and we can see kinship
relations and all these kind of incredible tasks in faces.
Now, if you imagine the face of persons that you know very well,
um your line drawings aren't
very precise and and and it's it needs a
lot of training to be good at that to to do the
um happy phase of um your wife or um you know so from this but we see we have
done experiment in which we see that the um encoding of emotions and gender
discrimination makes differences in V1.
So whether you have the task to recognize gender or to recognize the emotional expression.
We mapped out retrotopic spaces around mouth and eyes, but you can see a feedback
signal in the other parts that are still having a signature,
whether it is happy or fearful and so on.
So this, I think, doesn't speak to a feedback signal that is very precise,
let's say, like a line drawing of a precise emotion of a face.
But it's like a marker, a token that says, here's a face, should be happy,
you know very well your daughter, and that is a prediction.
Now, that is a good template. that if then the stimulus comes up and there's
a change in the facial expression you have FFA and other the face network to
work out the differences between your prediction and the actual input but the.
Token, the marking of where something is expected, is negotiated with higher
and earlier visual areas.
So I think the idea that I triggered of a cave drawing, in a way,
B1 is something like active blackboard, which works like line drawings,
and tokens, maybe tokens adjusted to that, where you say, oh,
bad animal, and here is a phase of something.
By this token, this is a particular B1. No, no, not the content.
More like, you know, we open up a channel that we speak to each other.
So you need to be a channel of high spatial frequencies because you're now interested
in the emotion around the mouth.
And that is interesting in this part of V1.
V1 doesn't know anything about those phases.
But the token is in the communication and higher.
I thought you put a token also inside V1. So that sounded confusing to me.
Like this, I get it. Absolutely. Yeah.
That makes sense, because then we have very parsimonious linking,
but then it's not necessarily defined along this notion of hierarchies of foreign models.
It's a rather different kind of framework we're then in.
Well we're talking about the generative capacity of the brain
now and how that can uh you know
be a tool for thought you know whether you're imagination and
dreaming all of these processes where there's no visual input uh you can reconstruct
activity throughout the visual hierarchy and including in v1 if you want to
think about the details of say a line drawing you can imagine and that might
require or involve activity in V1,
which then sets off cascades of activity elsewhere in the brain.
So we don't have to invoke a homunculus to see why it's worth reconstructing
the visual scene at the V1 level. Look, I appreciate that.
Speaking up for Lars now. No, I'm not. This is very much my own idea about what
we might, why V1 might operate in this way.
Look, I agree with this. I don't have a problem with that.
Okay. My remark with the criticism was a bit different.
Like, we started with a rather explicitly defined notion of prediction hierarchies,
which are all sort of convergent to something like a Kalman filter.
It's very explicitly defined and it really dictates to you what should happen
in these cascades of interactions.
And what I think we've seen at the end of the discussion, that there are cascades
of interaction, but they might
not follow this very restricted view on hierarchies of predictive filters.
We might have to open up that perspective more. Like how Lars now describes
the idea of having a higher level area that's a very abstract token type representation
of something and it seeks information in preceding areas is not necessarily
following this idea of prediction hierarchies.
Maybe it just looks for confirmation, doesn't give a damn about errors, right?
And that's how you can hallucinate things because you don't care about errors.
You just care about confirmation about whatever stuff you believe in higher areas.
So that was one of my challenges. And maybe the data and discussion is leading
us to the point that we should open up the perspective.
These are not just strictly defined prediction hierarchies. There's not enough data to support that.
And maybe this model of token representations are maybe an alternative that
is richer and maybe also closer to data.
That was basically what I was saying.
So are you happy with that, Tony? Or do you think I'm sort of misconstruing the discourse now?
I think you're pushing it. Which I happily do most of the time.
You're very much pushing it in a certain direction which I don't think Lars
was necessarily agreeing with. But he was not mishearing.
Maybe he's just tired. I think that's probably the case and he has a plane to
catch. You've drowned him down.
Good, you see? And Gern wins today again.
So, Mars, now, before you can escape, there's two last hurdles you have to take.
The first one is, though, this is not easy territory.
It's really involved experimental methods.
It's linked to theory. It's very precise. It's hard work, right?
This stuff doesn't come for free.
So, I'm sure you have plenty of scars, being in this domain.
So if we would like to follow your route to try to understand the brain, what is Lars' law?
What is Lars' law that we should follow to understand the brain?
Um okay so
of course there is
a kind of learning of
the tools that we try
to sharpen and to get better to explore
so what we bring to the table and what
every neuroscience labs brings to the a table or a
different you know background is um expertise
with the tools and um making
them sharper being able to to explore more and more
um cutting edge for example we do the ultra high resolution to layer specific
fmi um and using this uh techniques of brain reading for non-stimulated areas
this is something that we learned over time that is possible and it's there's
a lot of methodological kind of question.
Now, once you have this kind of toolset that you think is very good to explore,
with vector topic mapping and individual subject analysis and all this, then it's important to,
find.
Relevant questions and explain them
in such terminology that your corner field becomes bigger and wider and you
find more and more overlap with other labs and…,
This is a law. We have to be able to put it on.
I don't know what you mean by law. I kind of understood what's my method.
The law of effect, right?
Or the law of how to do science. I'm talking more about how to do science.
I think there's a lot that we start to learn by doing multidisciplinary.
Multiscale, multi-method science, which
is a very difficult thing to do because everyone is leaving a little bit of
their comfort zone and a little bit of terminology that is very well and precisely
defined to open up to other fields in which different terminology is applied
and different problems.
But by leaving a little bit of this comfort zone, the more we do that,
the more we find overlap, the bigger questions we can tackle.
It's something that certainly we have all experienced in the Human Brain Project
where we come together and start discussing our different approaches at which
you see incredible work,
but it is when the incredible works of different labs come together and find
common language that you trigger an enormous amount of synergy.
And that's something that's very hard work and it's very difficult and not so
comfortable and it's something very, very important in science.
I mean, I'm sometimes surprised when you, for whatever reason,
step outside of your field and experience some other science,
be it during a review process, during a conference.
Usually a conference you wouldn't usually go to, and you realize how fragmented science is.
And so it is important to converge, to find bigger stories, to not fight only
the corner to sharpen your methods,
but also to step outside and get the bigger picture, how it converges and how it fits together.
And there's a lot of synergy for many parts of science where I find it extremely exciting. Mm-hmm.
There's a bit in the trend of Ortega Gazette, right? Who spoke of counteracting
the barbarism of specialization.
So something he was saying, don't get sucked into the specialization,
keep an open mind, right? Look at the other surrounding domains.
Yes, that's part of it. I'm also a fan of using your methods very precisely.
I think brain imaging has been a bit scarred by rapid exploitation of stories.
And sometimes it has a bad image.
People have seen and believed images because they were on the brain and they
looked so scientifically and the stories connected to those were not always
linked to very hard and precise signs.
And so I think it's very important to gain back some trust there.
And, you know, I think we push the limits in this a lot in our lab. It's a lot of work.
And I think others are doing this in their field as well.
But then there's a lot of reward by opening up a little bit and seeing these
discussions over dinner and lunchtime when everyone is suddenly starting to
discuss other things outside of their scientific expertise.
There's a lot of, as I said, synergy and new ideas and converging ideas, I think.
Especially now that we're able to have a meringue that might reflect activity
of estrocytes instead of nerve.
But then the last question is uh you know
the last time that tony bought me a beer or paid for
a beer it's a long time ago and and
this has to do with the fact that he lives actually in for
a long time so you have me virtually neighbors right you're in glasgow not sure
you're over that but for that reason tony will come visit you in four years
in glasgow assuming We are still there and I'm sure you will be doing great
work and it's going to just come in a notebook to check whether you have confirmed or falsified,
a specific hypothesis that you're going to share with us today.
So what's the hypothesis, the most critical hypothesis in your research program
that you want to see tested in this four-year timeframe?
From now on in four years, what I want to see tested… So you're going to have
a shifting time window from now four years.
I'm not going to give away my best ideas.
You just have to make a prediction.
What will happen in my field or in other fields I think we'll see a lot of,
he's coming to your lab to check a specific so you're going to say and you will
come as well and you will see that you'll get more than a beer we are very generous
in Glasgow, we don't like Edinburgh,
we don't have the same No,
no So you're up for a prediction error there. Exactly.
No, I don't have a good prediction in this.
I mean, what I'm most intrigued, I think maybe four years is not going to be enough,
but I want to see the double coding of internal models that are about current situations.
They are predictive about the scene and
the mental models about something else being simultaneously processed in our
brains which we know and we know in which areas more or less but we don't know
how these codes coexist without interfering with one another now having layer um um.
Resolution fmi i believe we might
be able to find different codes simultaneously in
these different areas in the different layers and i
have to disagree with the remark that you made earlier that the differences
in the layers that they there aren't differences in the layers i think they
are you know i said that it's more gradual there's more gradual yeah if it's
not as discrete as these models yeah so we have some new data in which we we
have um visual illusions that are.
Explained away like motion-induced blindness which you
see a different code is still there but it's
not conscious so so there's more to come from this
but the the double coding of
mental imagery and visual predictions
i think is something that i'm i'm interested it must come at some level we we
and we might find some ways in which we can um see these double codes some Some
for the internal models that are currently happening and then other ones that are counterfactual.
And it's something that maybe human fMRI or human research is particularly well-tuned
to because these kind of experiments are relatively difficult to do in animals.
You would need to instruct them to do a visual process and simultaneously do a mental imagery task.
It's not totally impossible, but that's something I think I want to have more
knowledge about. How do you have these two codes simultaneously?
That will be a long discussion over a lot of beers. Lars Möckli,
thank you very much for this conversation.
Thank you. Thanks for the invitation. It was a pleasure.
The CSN podcast was produced by the Convergent Science Network of Biometrics
and Biohybrid Systems, a project funded by the European 7th Research Framework Program.
For more interviews, recorded lectures, or upcoming conferences in the field
of biometrics and biohybrid systems, go to csnnetwork.eu.
Music.
And thank you for listening.