WEBVTT

00:00:03.597 --> 00:00:06.517
This is the Convergent Science Network podcast.

00:00:08.517 --> 00:00:13.037
Leading researchers in the domain of neuroscience, brain theory and technology

00:00:13.037 --> 00:00:16.797
are interviewed by Paul Verschoor and Tony Prescott.

00:00:19.577 --> 00:00:23.517
So this is Paul Verschoor with the Convergent Science Network podcast.

00:00:24.997 --> 00:00:29.657
And I'm speaking now with Friedemann Pulvermuller, who is also a speaker at

00:00:29.657 --> 00:00:31.337
our summer school here in Barcelona.

00:00:32.457 --> 00:00:38.157
And Friedemann, you started with a tribute to Valentino Brattenberg in your talk. Why that?

00:00:39.057 --> 00:00:43.517
Well, because he's my teacher and mentor, and I learned a lot from him.

00:00:44.437 --> 00:00:47.317
Especially the question how to

00:00:47.317 --> 00:00:50.737
address cognitive processes

00:00:50.737 --> 00:00:53.737
this is from a neuroscience perspective

00:00:53.737 --> 00:00:57.437
looking for mechanistic answers

00:00:57.437 --> 00:01:00.597
to cognitive questions this uh

00:01:00.597 --> 00:01:04.277
general approach i would say i inherited from

00:01:04.277 --> 00:01:10.057
him but now um brettenberg

00:01:10.057 --> 00:01:13.357
is an anatomist to a large extent yes okay there was

00:01:13.357 --> 00:01:16.977
also this incredible project on vehicles and psychology which

00:01:16.977 --> 00:01:19.777
is really an amazing issue on detour in his

00:01:19.777 --> 00:01:22.617
in his career with quite an impact right but at

00:01:22.617 --> 00:01:25.617
heart he was an anatomist but in your own work

00:01:25.617 --> 00:01:29.317
you're not really doing anatomy anymore well not

00:01:29.317 --> 00:01:32.357
at the moment i would say now nowadays we have dti

00:01:32.357 --> 00:01:35.057
and then methods to look at

00:01:35.057 --> 00:01:38.877
cortical connectivity uh which i'm

00:01:38.877 --> 00:01:43.817
interested in we haven't published about that but but in principle uh that would

00:01:43.817 --> 00:01:51.757
be very much in the in the in the range of my inner interests and And there

00:01:51.757 --> 00:01:58.497
are projects such as we are looking at activation spreading over the cortex.

00:01:58.717 --> 00:02:03.477
And of course, this reflects to a degree the spreading through long-distance

00:02:03.477 --> 00:02:06.177
cortical connectivity.

00:02:06.637 --> 00:02:14.877
And of course, the models we are using to explain these activation spreadings,

00:02:14.877 --> 00:02:19.017
They are grounded in neuroanatomy,

00:02:19.057 --> 00:02:26.297
so we read a lot of neuroanatomical work and try to make realistic computer

00:02:26.297 --> 00:02:30.597
models of the brain and its substructures,

00:02:30.617 --> 00:02:37.137
like the different areas of motor system and the auditory system with a belt

00:02:37.137 --> 00:02:38.897
and parabelt and whatever it is.

00:02:38.897 --> 00:02:44.637
And then also putting in those connections that have been documented neuroanatomically.

00:02:44.857 --> 00:02:49.657
So neuroanatomy plays a big role, but I'm not a neuroanatomist.

00:02:49.857 --> 00:02:55.497
Right, okay. But then the other thing that you mentioned in relation to Breitenberg was that.

00:02:56.078 --> 00:03:00.098
He had this working hypothesis of the brain as an information mixing system.

00:03:00.298 --> 00:03:02.478
Yes. Okay, so what does it actually really mean?

00:03:02.838 --> 00:03:09.318
Well, it means that it's a system in which indeed there's specialization in

00:03:09.318 --> 00:03:12.558
the sense that there's an area where visual information comes in,

00:03:12.598 --> 00:03:15.178
from which motor activation goes out.

00:03:15.178 --> 00:03:22.298
But that the actual purpose of the cortex is to link as many as possible areas

00:03:22.298 --> 00:03:29.458
to each other and to provide information mixing in the sense that the motor neuron, finally,

00:03:29.658 --> 00:03:37.298
isn't a motor neuron only anymore after the linkage with the visual neuron.

00:03:37.298 --> 00:03:41.378
But the motor neuron becomes a little bit visual as well as,

00:03:41.398 --> 00:03:45.818
and in the very same way, by mutual linkage with the motor neuron,

00:03:45.938 --> 00:03:49.198
the visual neuron may become a little bit motor.

00:03:49.438 --> 00:03:55.058
So there are distributed cortical circuits that now carry multimodal,

00:03:55.058 --> 00:04:00.638
cross-modal properties, or if you wish, they become quite abstract. All right.

00:04:00.658 --> 00:04:05.418
So, if I understand you correctly, this information mixing capability is something

00:04:05.418 --> 00:04:08.938
you ascribe more to the neocortex, not to the brain overall.

00:04:09.358 --> 00:04:14.938
Absolutely. The picture I showed was actually a picture of the cortex.

00:04:15.198 --> 00:04:17.758
The cortex would be the information mixer.

00:04:18.358 --> 00:04:27.218
The cerebellum would be more a hetero-associative network that just links one

00:04:27.218 --> 00:04:29.118
pattern to a subsequent pattern.

00:04:29.738 --> 00:04:32.898
So, the nature of that is very different. Okay.

00:04:32.898 --> 00:04:37.758
And indeed, also in your own work, you focus very much on these mixing capabilities

00:04:37.758 --> 00:04:43.258
of the neocortex, in particular with respect to meaning and semantics with respect

00:04:43.258 --> 00:04:44.738
to language, right? So, word meaning.

00:04:45.738 --> 00:04:48.058
So, what's the problem really around meaning?

00:04:49.218 --> 00:04:53.018
Why are you worried about meaning? I'm not so much worried about meaning.

00:04:53.138 --> 00:04:58.058
Okay. I think meaning is a very straightforward example case,

00:04:58.258 --> 00:05:04.218
and we can understand aspects of it very easily.

00:05:05.978 --> 00:05:10.198
But I'm surprised that you say you're not worried about it, because in some

00:05:10.198 --> 00:05:15.478
sense, also your presentation, right, you try to show how actually in the past,

00:05:15.518 --> 00:05:19.518
or sort of standard models of meaning are actually insufficient, right?

00:05:19.518 --> 00:05:22.738
The standard models of meaning are often said, look, if somewhere in the brain

00:05:22.738 --> 00:05:27.898
there's a meaning module with all sorts of magical capabilities, that's insufficient.

00:05:28.778 --> 00:05:31.898
So this is really the challenge you try to answer now.

00:05:32.078 --> 00:05:37.318
So what is wrong with these sort of textbook models of meaning and meaning modules? Yes.

00:05:38.227 --> 00:05:44.647
Well, one thing that is wrong or insufficient is if I have a meaning module

00:05:44.647 --> 00:05:50.947
in which the meaning of each word or each concept is defined by relationship

00:05:50.947 --> 00:05:53.067
to other words and other concepts,

00:05:53.287 --> 00:06:01.067
then the so-called symbol grounding problem remains unanswered.

00:06:01.067 --> 00:06:04.887
So if I can use the word

00:06:04.887 --> 00:06:17.247
red very well in context such as a strawberry is red and lips are red and red

00:06:17.247 --> 00:06:21.287
symbolizes love or whatever,

00:06:21.467 --> 00:06:25.787
it's not sufficient.

00:06:25.787 --> 00:06:32.727
In order to document that I'm able to use the word red, I would need to point

00:06:32.727 --> 00:06:34.947
to the right color in a certain context.

00:06:35.047 --> 00:06:39.727
In order to know what a strawberry is, I would also need to find out which of

00:06:39.727 --> 00:06:41.267
the berries is actually the strawberry.

00:06:41.867 --> 00:06:49.007
If I don't have this capability, one of the criteria of semantic knowledge is not satisfied.

00:06:49.007 --> 00:06:57.847
And if I define word meaning just by semantic relationships,

00:06:57.987 --> 00:07:05.927
relationships between symbols, I do not have this grounding in knowledge of the world.

00:07:06.667 --> 00:07:12.427
This point has been made by Harnatt, Searle, many scientists. Right, exactly. Yeah.

00:07:13.027 --> 00:07:15.147
But what does this mean in practical terms? Does it mean that?

00:07:15.793 --> 00:07:20.953
If we want a theory of meaning, must map it back to interaction with the real world?

00:07:21.053 --> 00:07:24.473
Or should just map it back to the real world with respect to, let's say, reference?

00:07:25.173 --> 00:07:28.533
Let's say a strawberry just means that I can point to an object in the world

00:07:28.533 --> 00:07:33.653
and say, okay, this impression, the sensor states that are triggered by this

00:07:33.653 --> 00:07:37.033
object, as long as you can bring these together, this is meaning.

00:07:37.573 --> 00:07:42.193
Or is it really about me physically interacting and acting up on,

00:07:42.293 --> 00:07:45.333
that's to picking it up and smelling it and tasting it and so on?

00:07:45.333 --> 00:07:50.953
Is that action component critical or just one among many components that provide

00:07:50.953 --> 00:07:53.493
statistics on which meaning flows?

00:07:53.833 --> 00:08:01.333
Well, object reference, of course, meaning that I know to which object my given

00:08:01.333 --> 00:08:05.193
word like strawberry applies is a necessary component.

00:08:05.573 --> 00:08:11.293
So for words like strawberry, I want to show the point of the right thing.

00:08:11.293 --> 00:08:13.733
I want to know more, of course.

00:08:13.773 --> 00:08:16.533
I want to know how it tastes, how it smells as well.

00:08:16.993 --> 00:08:24.293
If I'm slightly deprived, if I'm blind, if I cannot taste so well,

00:08:24.373 --> 00:08:27.533
or if there's a sensory deprivation of different time, of course,

00:08:27.573 --> 00:08:29.753
I may be limited on one of these dimensions,

00:08:29.973 --> 00:08:35.973
but there are still the other dimensions, sensory knowledge,

00:08:36.093 --> 00:08:39.333
so to speak, that is of semantic relevance.

00:08:41.293 --> 00:08:47.933
There are other kinds of words, such as words with which we speak about actions.

00:08:48.153 --> 00:08:53.833
For example, a word like grasp, here there's no object to point to.

00:08:53.913 --> 00:08:56.893
Here there's no object reference in the same sense.

00:08:57.053 --> 00:09:02.133
We use it to speak about actions, but we also need to have this knowledge in

00:09:02.133 --> 00:09:05.993
which action contexts the word can be properly applied.

00:09:05.993 --> 00:09:13.973
And in which contexts it fails to be appropriately used.

00:09:14.173 --> 00:09:22.453
So it's not just visual perception to which the words relate objects.

00:09:23.293 --> 00:09:28.833
It's also actions. And for abstract concepts, it's a very similar issue,

00:09:28.933 --> 00:09:38.053
only possibly slightly more complex. If I want to teach my child what freedom means, what would I do?

00:09:39.379 --> 00:09:47.039
I would explain him or her that somebody who is living in jail and who is allowed

00:09:47.039 --> 00:09:50.199
out after several years is now free.

00:09:50.599 --> 00:09:57.859
And I would say, well, and this guy who is in handcuffs and somebody unlocks

00:09:57.859 --> 00:10:02.339
them and lets him go, freeze this guy.

00:10:02.339 --> 00:10:07.859
And I could also say that a judge sitting there and just doing nothing except

00:10:07.859 --> 00:10:15.819
for saying a sentence, well, you are not guilty and you are free.

00:10:16.239 --> 00:10:19.719
You do not have to stay in our prison here.

00:10:19.859 --> 00:10:25.599
So these are different instantiations of the freedom concept, so to speak.

00:10:25.599 --> 00:10:32.479
And in order to be able to use this abstract term, I would need to know how

00:10:32.479 --> 00:10:35.859
to apply it in such prototypical circumstances.

00:10:36.139 --> 00:10:41.259
Of course, now everybody has his or her own experiences with strawberries,

00:10:41.539 --> 00:10:43.239
with freedom, and so on and so forth.

00:10:43.359 --> 00:10:47.579
But there is a common knowledge about prototypical situations,

00:10:47.939 --> 00:10:51.519
both for the abstract and for the concrete items. Okay.

00:10:51.659 --> 00:10:56.579
So then you're saying meaning as such comes in degrees, let's say.

00:10:56.659 --> 00:10:59.939
You can tie in more or less modalities and submodalities.

00:11:00.259 --> 00:11:04.159
Absolutely. Right. And are there some boundaries to that?

00:11:04.359 --> 00:11:07.679
Is there some boundary? Let's say, does it need to be symbolic in some sense

00:11:07.679 --> 00:11:09.219
or can also be just purely analog?

00:11:10.759 --> 00:11:14.579
Does it sort of means how literal should be the connection to,

00:11:14.659 --> 00:11:19.259
let's say, the neuronal activity that an object might trigger?

00:11:19.739 --> 00:11:23.699
So are the boundaries on this notion of meaning if it comes in gradations?

00:11:24.839 --> 00:11:26.719
And now the term symbolic.

00:11:28.136 --> 00:11:31.136
Is well should probably be

00:11:31.136 --> 00:11:34.456
explained so may i ask what you how you

00:11:34.456 --> 00:11:37.496
use this well it's in artificial intelligence

00:11:37.496 --> 00:11:42.836
it's sometimes a symbolic approach is sometimes that approach that doesn't have

00:11:42.836 --> 00:11:48.276
the neuronal basis right now of course we are talking neurons and and therefore

00:11:48.276 --> 00:11:52.916
it doesn't it's not just symbolic it's always with a neural basis so the main

00:11:52.916 --> 00:11:56.356
question breitenbergian question if you wish,

00:11:56.476 --> 00:12:03.716
is how can we find a neurobiological underpinning for things such as symbols?

00:12:03.936 --> 00:12:09.536
So in that sense, it doesn't stay symbolic, but it becomes symbolic with a brain grounding.

00:12:09.816 --> 00:12:12.836
Right, with sort of an implementation of the physical component.

00:12:12.996 --> 00:12:14.456
Yes, with a mechanism behind it. Exactly.

00:12:14.556 --> 00:12:18.776
So with symbolic in the way I intend this also, that's why I mentioned analog,

00:12:19.176 --> 00:12:22.476
I think this is an important position.

00:12:22.476 --> 00:12:27.916
Because in the analog case, the object will drive a sensor sheet,

00:12:28.136 --> 00:12:32.816
sensor sheet will drive neural activity, and it is sort of a direct mapping, right?

00:12:32.876 --> 00:12:37.456
So the activity is not transformed in some way, but it's really this vector

00:12:37.456 --> 00:12:42.836
of responses, the state of responses induced by the way the object in the outside

00:12:42.836 --> 00:12:46.196
world tickles the sensor sheet that gives you now an analog representation.

00:12:46.836 --> 00:12:50.436
A symbolic representation doesn't have that feature. It's really disconnected

00:12:50.436 --> 00:12:53.576
from the way in which the outside world is tickling the sensor sheets.

00:12:54.756 --> 00:12:58.716
So it's a decoupling between an internal representation and how this can be

00:12:58.716 --> 00:13:00.096
activated by the outside world.

00:13:01.669 --> 00:13:07.049
Well, I'm not so sure about the latter, as I try to explain in this case of

00:13:07.049 --> 00:13:10.829
the abstract meaning of freedom or free, of the word free.

00:13:12.569 --> 00:13:18.309
If we learn the meaning or if we teach, well, the practical example is always

00:13:18.309 --> 00:13:25.109
related to practical activities like teaching a child what the meaning is. And what would you do?

00:13:25.189 --> 00:13:33.209
You would present concrete examples, a range of communication contexts where

00:13:33.209 --> 00:13:37.229
the word is appropriately applied, interaction contexts also.

00:13:37.229 --> 00:13:46.449
But so there isn't a complete detachment between the meaning and these typical instantiation.

00:13:46.669 --> 00:13:52.489
One could even claim that the knowledge about the typical instantiation and

00:13:52.489 --> 00:14:00.369
instantiations is necessary for the knowledge about the symbolic meaning.

00:14:00.989 --> 00:14:08.309
And I wouldn't go so far that I would say it exhausts the meaning, but it's necessary.

00:14:08.689 --> 00:14:17.969
And what is different from a case of knowing the meaning of the word strawberry,

00:14:18.209 --> 00:14:21.709
where there is probably one could argue there's one prototype.

00:14:22.349 --> 00:14:27.949
A main difference is that in In this freedom case, there are many different prototypes.

00:14:28.309 --> 00:14:32.489
And therefore, one couldn't say, well, and it's actually this hand movement

00:14:32.489 --> 00:14:38.809
of unlocking the handcuffs, which relates to the freedom concept.

00:14:38.869 --> 00:14:46.609
It's actually a vast variety of different actions that could go through as good

00:14:46.609 --> 00:14:51.709
examples of freeing somebody, a verbal action, a hand action, and so on and so forth.

00:14:51.709 --> 00:14:55.869
But in the case of a strawberry it's.

00:14:56.583 --> 00:15:04.383
There could be one prototype only. So we need more computational power for the abstract concept.

00:15:04.463 --> 00:15:10.443
We need to calculate something like an either-or function over different instantiations.

00:15:10.443 --> 00:15:13.883
And therefore, as you correctly mentioned,

00:15:14.283 --> 00:15:24.123
there's a degree of detachment between the instantiation, the concrete situational

00:15:24.123 --> 00:15:28.943
context, prototypical situational context, because there are so many of them.

00:15:28.943 --> 00:15:35.183
By the way, the correlation rule captures that very well. If I correlate two

00:15:35.183 --> 00:15:39.303
things with each other, then there may be a link.

00:15:40.243 --> 00:15:50.523
But if I try to store the knowledge that one thing can have 10 different corresponding instantiations,

00:15:51.223 --> 00:16:00.703
then the correlation rule would actually produce a little bit of a link for each.

00:16:00.703 --> 00:16:07.223
But also, if I learn that A and B belong together, there's strengthening of the connection.

00:16:07.463 --> 00:16:10.623
But if I learn that there's A and C that also belong together,

00:16:10.883 --> 00:16:13.943
there's a little bit of weakening. Yeah, but now we're jumping forward, right?

00:16:13.983 --> 00:16:17.423
This is also very much related to the model that you built of meaning.

00:16:17.703 --> 00:16:22.663
And before we get to the model, I would like to first look at sort of the data

00:16:22.663 --> 00:16:24.943
you have collected on meaning in the brain.

00:16:25.823 --> 00:16:30.463
And before getting to the data, another definitional issue that also came up

00:16:30.463 --> 00:16:33.703
this morning and would be useful to, I think, get it out there,

00:16:34.483 --> 00:16:39.683
was this whole issue about, again, but what then is the difference between meaning

00:16:39.683 --> 00:16:41.663
and semantics and the notion of a concept?

00:16:42.463 --> 00:16:46.483
But why not just speak of concepts? Why not say, look, if it's just about,

00:16:46.603 --> 00:16:50.023
let's say, the kind of information in the outside world.

00:16:51.344 --> 00:17:00.084
That co-occurs, right, given a certain label, a word, why not call that just a concept, right?

00:17:00.124 --> 00:17:02.524
That's just a cloud of points in some high-dimensional space,

00:17:02.784 --> 00:17:05.604
so they belong together because they co-occur, so I call it a concept.

00:17:05.704 --> 00:17:11.244
But when do we call that a semantic meaning as opposed to just a concept?

00:17:12.624 --> 00:17:15.804
Well, in semantic theory, it's

00:17:15.804 --> 00:17:22.344
just conventional to distinguish between concepts and semantics of words.

00:17:22.604 --> 00:17:29.264
And the very simple difference is that one could say that semantics.

00:17:30.364 --> 00:17:33.504
The meaning of a word, semantics is the same thing as meaning,

00:17:34.184 --> 00:17:41.144
and the meaning of a word is the concept with a regular relationship to a word form or a symbol.

00:17:41.144 --> 00:17:46.704
So it's just, if you wish, a subset of the space of concepts,

00:17:46.964 --> 00:17:54.804
but those that have a clear relationship to words established in the language community.

00:17:54.804 --> 00:18:02.224
So if I personally have a bad experience with fish and therefore would have

00:18:02.224 --> 00:18:07.524
a, well, always a startle response when seeing or tasting one.

00:18:08.284 --> 00:18:16.624
The word fish wouldn't have the meaning of being something bad because it's just my own experience.

00:18:16.624 --> 00:18:19.824
So that the the word semantics would

00:18:19.824 --> 00:18:23.084
always have this language community aspect so

00:18:23.084 --> 00:18:28.104
it should be shared by many right otherwise in order to allow that what i say

00:18:28.104 --> 00:18:33.624
is understood by others in the same way so if my personal experience is slightly

00:18:33.624 --> 00:18:38.484
different from the from the from the from what is established in the in the

00:18:38.484 --> 00:18:40.124
language community then one would,

00:18:40.812 --> 00:18:47.532
It would need to be discussed whether this aspect should belong to a semantic space, so to speak.

00:18:47.712 --> 00:18:55.292
So the concepts basically define, let's say, the co-occurrences of certain statistical

00:18:55.292 --> 00:18:56.752
states out there in the world.

00:18:56.912 --> 00:19:02.092
But as soon as these start to co-occur together with words, you call it semantics. Yes.

00:19:02.932 --> 00:19:07.992
But it also would mean that your theory in principle would generalize to the notion of concepts.

00:19:08.512 --> 00:19:12.852
Absolutely. And you use the semantic linguistic case more as your test case

00:19:12.852 --> 00:19:16.372
in some sense to get access to this conceptual space.

00:19:16.792 --> 00:19:22.332
Also, for the case of language, I think for me this is much simpler because

00:19:22.332 --> 00:19:28.152
asking what is the meaning of the word can be very closely related to the question,

00:19:28.312 --> 00:19:32.592
how do I teach the meaning of this word, for example, to a language learner, a child.

00:19:32.852 --> 00:19:37.752
However, the question about the nature of concepts is much more nebulous.

00:19:37.752 --> 00:19:43.152
So, I can say, well, and it's very easy to say, well, concepts are inborn.

00:19:43.252 --> 00:19:49.892
They are just in our heads by nature, by natural laws and genetics.

00:19:50.132 --> 00:19:52.772
And I don't want to know about it.

00:19:53.832 --> 00:19:58.872
I just postulate it. However, if I do this for semantics, then I run into a problem.

00:19:58.872 --> 00:20:09.872
Because I assume I have an inborn knowledge of stinginess.

00:20:10.612 --> 00:20:18.272
Now, how would I become able to link the word form stinginess,

00:20:18.372 --> 00:20:22.572
the word stinginess, to this particular inborn concept?

00:20:22.572 --> 00:20:28.072
So the semantic learning would require that stinginess would become manifest

00:20:28.072 --> 00:20:30.892
somehow, otherwise I couldn't teach it.

00:20:31.012 --> 00:20:37.792
So therefore, the semantic case is much simpler and much easier to address than

00:20:37.792 --> 00:20:44.192
the conceptual case. If I come up with a conceptual theory, I can stay in nebulous space.

00:20:44.632 --> 00:20:47.952
But apparently you did figure out I'm Dutch.

00:20:48.152 --> 00:20:52.832
That's why I took stinginess as your example. No, no, I'm Swabian,

00:20:52.832 --> 00:20:59.192
and people in Swabia, southwest Germany, they are said to be very stingy.

00:20:59.192 --> 00:21:01.032
Really? Oh, the Dutch is the same. Okay.

00:21:01.152 --> 00:21:04.372
So we are innately stingy. Yes. Okay.

00:21:05.352 --> 00:21:09.092
So there must be a genetic relationship between Swabians, Dutch,

00:21:09.312 --> 00:21:12.412
and Scotsman. That's going to be our next collaborative project.

00:21:13.932 --> 00:21:18.092
Okay, so now we did the preliminaries, right? So the definitions are out of the way.

00:21:18.252 --> 00:21:23.392
Yeah. We look now at, now we can move on to this notion of the mechanisms of

00:21:23.392 --> 00:21:26.152
meaning, because that will bring you to the brain, right?

00:21:26.192 --> 00:21:31.032
And there you distinguish that's referential semantics, abstract semantics,

00:21:31.232 --> 00:21:34.612
emotional semantics, and combinatorial semantics.

00:21:34.832 --> 00:21:37.992
Yes. Why these four? Why these four?

00:21:41.199 --> 00:21:47.639
Well, the case of referential semantics we have discussed, or if there's a link

00:21:47.639 --> 00:21:54.639
between words and the world, or even construction sentences and the world.

00:21:54.779 --> 00:22:01.679
We speak about things, about actions, about interactions, and this knowledge

00:22:01.679 --> 00:22:06.959
of this relationship has to do with the meaning of the terms. of the terms.

00:22:07.559 --> 00:22:12.319
Then there's this claim that there's a relationship among the symbols.

00:22:13.439 --> 00:22:17.439
And this is to a degree relevant for the meaning.

00:22:17.679 --> 00:22:20.899
And there's no doubt that this idea is correct.

00:22:21.119 --> 00:22:26.199
For example, a blind person can learn that strawberries are red,

00:22:26.319 --> 00:22:30.939
not because this person has ever seen a red strawberry,

00:22:31.079 --> 00:22:36.259
but simply because the word red and the word strawberry frequently co-occur in sentences.

00:22:36.719 --> 00:22:41.159
And it's not sentences that also have a not in them.

00:22:41.599 --> 00:22:50.599
So there's some combinatorial data around that also allows us to draw conclusions

00:22:50.599 --> 00:22:53.179
that are semantic in nature,

00:22:53.259 --> 00:22:57.639
in the sense that we can deduce semantic statements,

00:22:58.719 --> 00:23:05.299
of the thought strawberry has to do something Something to do with red or is

00:23:05.299 --> 00:23:10.199
red, if I know that red is a feature of objects.

00:23:12.219 --> 00:23:18.479
Now, this is, and of course we know that words occur together,

00:23:18.659 --> 00:23:21.679
other words occur together not so frequently.

00:23:22.359 --> 00:23:28.739
There's a correlation not only between words and the world. So I use the word

00:23:28.739 --> 00:23:34.399
to speak about this type of object, and therefore the object may co-occur with

00:23:34.399 --> 00:23:37.099
the occurrence of the word form, or I would...

00:23:38.283 --> 00:23:42.863
It doesn't require that the strawberry is indeed present in the environment

00:23:42.863 --> 00:23:45.623
while I speak. I could use the word strawberry.

00:23:45.763 --> 00:23:51.243
I could also imagine, think of the object and at the same time use the word,

00:23:51.243 --> 00:23:56.523
so simulate the object or the scene.

00:23:57.123 --> 00:24:02.763
And this could lead to a semantic linkage of a referential type.

00:24:02.763 --> 00:24:08.983
But then the co-occurrence of the words in strings would also trigger similar processes.

00:24:09.123 --> 00:24:15.943
We know that nerve cells in the cortex strengthen their links when they are active together.

00:24:16.123 --> 00:24:22.083
And assume that there's a neural population that corresponds to my word number

00:24:22.083 --> 00:24:24.963
one, red, and my word number two, strawberry.

00:24:25.743 --> 00:24:30.123
If they occur together very frequently, then the connections between them,

00:24:30.183 --> 00:24:31.863
if any, would strengthen.

00:24:31.863 --> 00:24:39.523
And if they, however, if they appear in different contexts, each on its own,

00:24:39.623 --> 00:24:42.723
then this connection might weaken again.

00:24:42.723 --> 00:24:48.963
So there is a degree of mapping also of word co-occurrences,

00:24:49.043 --> 00:24:58.403
and this leads to the postulate that we have this storage of combinatorial semantics,

00:24:58.623 --> 00:25:05.543
which complements the semantic knowledge that relates to the world.

00:25:05.543 --> 00:25:12.043
So, there's this word-world relationship, but also this word-word relationship

00:25:12.043 --> 00:25:19.803
that is easily mapped by a correlation storage device, such as our cortex.

00:25:19.803 --> 00:25:27.923
So, over the years, you have actually accumulated a lot of evidence for this hypothesis, right?

00:25:27.983 --> 00:25:36.263
So what do you see as the outstanding pieces of data that support this view on semantics?

00:25:38.265 --> 00:25:46.425
If you allow me, your previous question was actually aiming at two more facets.

00:25:46.705 --> 00:25:51.585
Yes. One about abstract semantics, one about emotional semantics. Yes, exactly.

00:25:51.905 --> 00:25:55.845
And maybe I could very briefly address those two too.

00:25:56.125 --> 00:26:04.045
Okay. So the abstract semantics is special in the sense of the freedom example

00:26:04.045 --> 00:26:10.385
I started our discussion with. I presented it at the start of our discussion.

00:26:10.745 --> 00:26:14.925
So the strawberry example is very simple.

00:26:15.785 --> 00:26:23.065
One object scheme, one word, linked together. Freedom, much more difficult semantically.

00:26:23.785 --> 00:26:29.345
Very different prototypical instantiations of the meaning of that concept,

00:26:29.645 --> 00:26:37.845
of that many meaning facets, if you wish. and there needs to be more computational power.

00:26:38.105 --> 00:26:41.485
There needs to be a neural element

00:26:41.485 --> 00:26:46.165
that says it can be this or it can be this or that or that or that.

00:26:46.365 --> 00:26:53.365
So a list of prototypes so to speak and an either or or just an or connection between them.

00:26:53.765 --> 00:27:00.825
And this is the special thing that in this case such or or either or connections

00:27:00.825 --> 00:27:05.045
links would need to come in So a little bit more, if you wish,

00:27:05.165 --> 00:27:07.845
symbolic neuronal mechanisms.

00:27:09.341 --> 00:27:14.081
But we have no problems with these mechanisms because this is actually what

00:27:14.081 --> 00:27:17.401
nerve cells are made for, to do such computations.

00:27:17.401 --> 00:27:23.081
One reason why I skipped over it to fill that in further and to go to the neural

00:27:23.081 --> 00:27:30.601
mechanism was because I wanted to challenge you later on these distinctions

00:27:30.601 --> 00:27:32.181
you're making. Maybe we can do that now.

00:27:32.601 --> 00:27:37.841
Because in some sense, I could argue, look, let's take referential semantics.

00:27:37.841 --> 00:27:39.341
Okay, so we take headphone.

00:27:39.581 --> 00:27:43.381
Okay, so I see a headphone out there in the world, but the headphone in itself

00:27:43.381 --> 00:27:45.801
is comprised of many components.

00:27:45.981 --> 00:27:49.621
And certainly if I look at the headphone as just a visual stimulus.

00:27:50.321 --> 00:27:52.801
It is completely fragmented in my visual cortex.

00:27:53.041 --> 00:27:59.341
When it enters my V1, it's just a massive puzzle with millions of pieces that

00:27:59.341 --> 00:28:00.901
I have to sort of actively reassemble.

00:28:01.301 --> 00:28:07.041
So in some sense, at that level of now visual meaning, meaning I'm already solving

00:28:07.041 --> 00:28:09.041
a combinatorial problem, right?

00:28:09.101 --> 00:28:12.981
So in some sense, this raises this issue about the distinctions you're making

00:28:12.981 --> 00:28:18.341
here and whether they're really a minimal interpretation, right?

00:28:18.381 --> 00:28:23.481
I could say, well, maybe combinatorial is subsumed also in referential.

00:28:23.601 --> 00:28:25.181
So this is not really a clear distinction.

00:28:25.581 --> 00:28:30.781
And I could possibly make the same argument with abstraction because to go from

00:28:30.781 --> 00:28:37.321
the headphone input states of my primary visual cortex text to my concept headphone,

00:28:37.581 --> 00:28:39.901
so before it reaches a semantic stage.

00:28:40.001 --> 00:28:44.001
Now, just the integration of all this information that floats around in my brain

00:28:44.001 --> 00:28:47.241
around headphone, also that is a form of abstraction.

00:28:47.461 --> 00:28:50.821
It must be abstracted. I cannot rely on what's out there in the world.

00:28:50.921 --> 00:28:56.241
So in some sense, this distinction you make might be a little bit arbitrary.

00:28:56.501 --> 00:28:58.561
There might not be hard borders between them.

00:28:59.808 --> 00:29:04.888
Indeed. I think this criticism is, to a degree, appropriate.

00:29:05.288 --> 00:29:09.468
So there are different types of chairs, different types of headphones,

00:29:09.828 --> 00:29:16.068
and if we speak about headphones, we might want to include the more or less prototypical ones.

00:29:16.568 --> 00:29:18.388
And, of course, if we speak about

00:29:18.388 --> 00:29:28.488
animals, there's a long list of possible referent objects and objects.

00:29:29.808 --> 00:29:35.108
I agree that they are not just the extremes of the spectrum.

00:29:35.328 --> 00:29:43.788
So an item such as a fruit, which comes with very little variation,

00:29:45.748 --> 00:29:50.528
even though if you think of strawberries, they can be very small,

00:29:50.608 --> 00:29:53.728
very large, so a degree of abstraction is necessary there too.

00:29:54.788 --> 00:30:00.068
And on the other end of the scale, there's an abstract concept where there's

00:30:00.068 --> 00:30:08.468
really such a wide range of different instantiations that there's no way to

00:30:08.468 --> 00:30:11.908
achieve much without such logical operations.

00:30:12.028 --> 00:30:18.708
In between, there are cases of larger category terms or then of terms,

00:30:18.868 --> 00:30:20.468
as you said, like headphones,

00:30:20.668 --> 00:30:26.828
which can come in very different shapes where a degree of either-or computation

00:30:26.828 --> 00:30:28.748
might still be necessary.

00:30:29.468 --> 00:30:35.648
At a at a basic level category right exactly no yeah but i i agree so there's

00:30:35.648 --> 00:30:42.908
no clear boundary here this is entire there they are the extreme cases but there's

00:30:42.908 --> 00:30:47.028
a lot right lots of material in between so you could say ontologically uh in

00:30:47.028 --> 00:30:49.048
terms of what's really going on in the brain.

00:30:50.446 --> 00:30:55.406
It might be more a diffuse and a continuous process, but as a research heuristic,

00:30:55.666 --> 00:30:58.746
it could still help you to actually get access to this, right?

00:30:58.806 --> 00:31:00.986
So this is maybe how we should look at that.

00:31:01.066 --> 00:31:09.366
And what we do is we then ask our experimental subjects and ask them 150 questions

00:31:09.366 --> 00:31:12.066
about the aspects of the meaning of words.

00:31:12.066 --> 00:31:17.926
And we pick out some of those words with very simple semantics where they say,

00:31:18.046 --> 00:31:22.826
for example, this word relates to actions I typically perform with my hand, for example.

00:31:23.706 --> 00:31:29.486
And of course, the action verb to free is not among those.

00:31:31.086 --> 00:31:35.266
And same with object words as well.

00:31:35.266 --> 00:31:44.066
And if they are then words with very variable semantics, then those would fall

00:31:44.066 --> 00:31:47.466
out of these experiments. Right, exactly.

00:31:48.026 --> 00:31:50.946
Okay, so, but I think we got agreement here. This is very good.

00:31:51.286 --> 00:31:55.206
Because now when you can go back to the follow-up question, which was,

00:31:55.266 --> 00:31:56.906
okay, but where's the data now, right?

00:31:56.966 --> 00:32:03.146
Which is, where is the data that would support this interpretation of semantics?

00:32:03.426 --> 00:32:08.786
Yes. So where should we start there? What was the first observation,

00:32:08.846 --> 00:32:13.426
let's say, that gave you hope that this more, let's say, statistical interpretation

00:32:13.426 --> 00:32:18.806
of word meaning would actually pan out? That's really what the brain is relying on.

00:32:19.451 --> 00:32:24.111
Sorry, again, this is now about statistical. No, no, wait, wait. I just want to go back.

00:32:24.431 --> 00:32:27.671
So let's go back again now to the referential semantics.

00:32:27.811 --> 00:32:33.251
Yes. So we just go to the case where we have word meaning with respect to objects

00:32:33.251 --> 00:32:35.031
out there in the world, right? The direct reference.

00:32:35.471 --> 00:32:40.811
And this direct reference, the word meaning is now, there is occurring because

00:32:40.811 --> 00:32:45.971
aspects of this object co-occur systematically.

00:32:46.411 --> 00:32:50.171
Right? So statistically, there's a pattern. The brain can pick up this pattern.

00:32:50.371 --> 00:32:53.871
And now, you know, Friedemann Pulvermuller, seeing this from the outside,

00:32:54.011 --> 00:32:56.711
says, aha, here is word meaning. Right?

00:32:57.071 --> 00:33:01.711
So what were these pieces of data? What were the observations that allowed you to say that?

00:33:02.011 --> 00:33:07.471
Yeah. Well, if you ask me about the history of it or what gave me hope for the

00:33:07.471 --> 00:33:15.111
first time, then I should say, well, reading a paper by Helen Neville from 1992.

00:33:15.111 --> 00:33:22.671
1992, and this was about words related to grammar, function,

00:33:22.911 --> 00:33:27.371
grammatical function words, and content words, nouns and verbs mainly.

00:33:27.651 --> 00:33:29.291
And they were presented in an

00:33:29.291 --> 00:33:34.611
EEG experiment, and there was a degree of activation of both hemispheres,

00:33:34.851 --> 00:33:41.671
and it was EEG recordings, and there was a lot of activation on both sides of

00:33:41.671 --> 00:33:44.151
the brain, while for these grammatical

00:33:44.151 --> 00:33:48.611
words, there was just a localised activation in the left hemisphere.

00:33:51.191 --> 00:33:55.111
And this is of course compatible with the view that these content words,

00:33:55.231 --> 00:34:00.891
these action verbs and object nouns and maybe some other items as well,

00:34:01.031 --> 00:34:03.131
they activate a lot of the.

00:34:04.057 --> 00:34:10.057
A lot of semantic links in various cortical areas, visual cortex, maybe motor cortex.

00:34:10.177 --> 00:34:13.697
This is more speculation than data. The data would actually tell you,

00:34:13.737 --> 00:34:16.937
but it fitted nicely into the model.

00:34:17.397 --> 00:34:25.217
And we then went on and learned to use EEG by then and replicated this experiment

00:34:25.217 --> 00:34:30.077
now with better matching of the stimuli in order to exclude a range of confounds.

00:34:30.077 --> 00:34:35.597
Looked that these words were exactly matched for a range of psycholinguistic

00:34:35.597 --> 00:34:40.457
features which we now know influence the brain response very much.

00:34:40.597 --> 00:34:45.817
And we could confirm that there's a strong laterality for the grammatical items

00:34:45.817 --> 00:34:51.937
and the whole brain, or there's a brain activation pattern compatible with whole

00:34:51.937 --> 00:34:55.397
cortex activation for these more meaningful items.

00:34:55.497 --> 00:34:59.637
Now, the whole series of experiments is, of course, entirely insufficient to

00:34:59.637 --> 00:35:03.817
address the question, but you asked what gave me hope. Yeah, exactly.

00:35:04.477 --> 00:35:11.097
Because this kind of experiment was heavily confounded. Function words are grammatical items.

00:35:11.337 --> 00:35:14.077
There's a lot of syntactic knowledge attached to them.

00:35:14.157 --> 00:35:20.517
Now the content words, they have meaning, but they are combinatorially very different.

00:35:20.677 --> 00:35:29.017
They are imageable. They give rise to secondary processes, cognitive processes

00:35:29.017 --> 00:35:32.037
like imagination, function, words, tone, and so on and so forth.

00:35:33.387 --> 00:35:37.167
And we could match some basic psycholinguistic variables,

00:35:37.387 --> 00:35:45.927
as I said, and the differences persisted, but still it's not the best way of

00:35:45.927 --> 00:35:48.547
addressing the semantic questions.

00:35:48.547 --> 00:35:57.507
And the next thing that gave me hope was work on word category processing or

00:35:57.507 --> 00:35:59.287
category-specific processes.

00:35:59.447 --> 00:36:04.967
For example, Alex Martin at NIH did experiments on naming,

00:36:05.347 --> 00:36:08.867
tool naming and animal naming, and found differential activation

00:36:08.867 --> 00:36:15.647
at that point with PET and a little bit of motor cortex activation already then

00:36:15.647 --> 00:36:26.547
for the tool words and a lot of visual and temporal activation for the animal word naming.

00:36:26.947 --> 00:36:31.367
Now, the problem here is, of course, that you start with pictures and you don't

00:36:31.367 --> 00:36:37.747
know to what degree the physical features of the pictures influence this and

00:36:37.747 --> 00:36:42.887
to what degree the picture processing invokes some differential activation or

00:36:42.887 --> 00:36:44.827
whether it's language and concept related.

00:36:46.447 --> 00:36:50.247
And we went on to look at nouns and verbs.

00:36:50.887 --> 00:36:57.547
Nouns and verbs activated. If we take concrete action verbs and object nouns,

00:36:57.807 --> 00:36:59.227
there's a difference of activation.

00:36:59.407 --> 00:37:04.087
But of course, also here, linguistic confounds come in. So all of this work

00:37:04.087 --> 00:37:08.727
gave us a lot of hope, but it wasn't the final answer to the question.

00:37:08.907 --> 00:37:13.247
I think the first time we were really convinced that now we have something in

00:37:13.247 --> 00:37:19.407
hand to postulate that it's semantics and only semantics that is reflected in local brain response.

00:37:19.407 --> 00:37:25.487
And that this local brain response could also be related to activations that

00:37:25.487 --> 00:37:32.487
have to do with action, with perception, was when we looked at different action patterns.

00:37:33.387 --> 00:37:37.727
Verbs, actually, action verbs that relate to different parts of the body.

00:37:37.867 --> 00:37:42.967
And then we found, for example, that words such as grasp, actions you perform

00:37:42.967 --> 00:37:47.907
with the hand, would activate the hand motor representation,

00:37:48.307 --> 00:37:54.027
actually overlapping with those areas in those experimental subjects that were

00:37:54.027 --> 00:37:59.327
also activated when they actually moved their finger or some upper extremity.

00:37:59.327 --> 00:38:05.887
And the same thing we found for leg-related words like kick or walk,

00:38:05.987 --> 00:38:12.847
and they activated regions overlapping now with those regions that were also

00:38:12.847 --> 00:38:17.587
activated when people were tapping their foot or something, moving the foot.

00:38:18.007 --> 00:38:24.047
And that gave us, because here the linguistic confounds had been ruled out,

00:38:24.107 --> 00:38:28.787
so the words were similar, also according to grammatical and whatever other possibilities.

00:38:29.147 --> 00:38:33.627
They were equally imagable, action-related, and so on and so forth,

00:38:33.727 --> 00:38:36.287
but there was a different body-part relationship.

00:38:36.587 --> 00:38:41.727
Now, we wouldn't claim, and we haven't actually, that the motor system here

00:38:41.727 --> 00:38:44.147
houses semantics and only the motor system.

00:38:44.447 --> 00:38:50.367
But we would say that by activation in the motor system, some aspects of the

00:38:50.367 --> 00:38:53.287
meaning of these words are reflected.

00:38:53.287 --> 00:39:02.327
And I think this has been replicated in many studies since, even if there's

00:39:02.327 --> 00:39:06.727
a degree of variability of these motor activations as a function of context,

00:39:06.947 --> 00:39:09.347
as a function of task, and interestingly.

00:39:10.107 --> 00:39:16.547
As a function of the communicative function the words have in a particular situation.

00:39:16.547 --> 00:39:26.687
Situation, but I think it's generally it could be confirmed and there is evidence that words that.

00:39:27.446 --> 00:39:32.606
That semantically relate to actions, involve to a degree the motor system.

00:39:32.906 --> 00:39:38.686
Right. But now, would you see something similar for words that invoke visual perception?

00:39:38.986 --> 00:39:44.846
Absolutely, yeah. We did very similar experiments now for color and shape words,

00:39:45.046 --> 00:39:49.386
where there was a similar double dissociation in the visual system.

00:39:49.646 --> 00:39:55.106
Right. So it's a very general feature that you observe. And there are other

00:39:55.106 --> 00:39:59.706
groups who have done similar work.

00:39:59.786 --> 00:40:09.246
For example, Bartholow and Simmons, they looked at color words and those areas

00:40:09.246 --> 00:40:12.966
that are especially important for color processing.

00:40:13.106 --> 00:40:19.246
And there was an overlap of activations. And Markus Kiefer in Ulm, Germany,

00:40:19.366 --> 00:40:31.666
he looked at sound-related words, and a range of people here in the vicinity of Barcelona,

00:40:31.966 --> 00:40:42.126
they looked very closely at other modality words, such as odor and taste words.

00:40:42.666 --> 00:40:52.626
So, with respect to this referential semantics, the first observation was,

00:40:52.706 --> 00:40:55.826
look, action allows you to interpret the response, right?

00:40:55.866 --> 00:41:00.706
Because you could see how these responses were invading motor execution systems. Yes.

00:41:01.426 --> 00:41:04.166
But now this raises a number of questions, right? But one is,

00:41:04.266 --> 00:41:12.326
okay, but then now the brain is mapping all these inputs into a conceptual space.

00:41:12.626 --> 00:41:15.806
And this conceptual space has some dimensionality because now you're saying,

00:41:15.906 --> 00:41:18.866
well, it might go towards motor execution systems.

00:41:18.886 --> 00:41:23.246
It could go towards vision, audition, affection, taste.

00:41:23.626 --> 00:41:28.666
But is that the intrinsic dimensionality of this semantic space in your mind?

00:41:28.766 --> 00:41:32.166
Or is that not how it's organized?

00:41:32.166 --> 00:41:41.046
Well, I would say these modality dimensions are certainly part,

00:41:41.166 --> 00:41:45.066
as I would construe it, part of the semantic space.

00:41:45.346 --> 00:41:47.926
They wouldn't exhaust the semantic space.

00:41:48.406 --> 00:41:53.226
I wouldn't… So what other dimensions are there?

00:41:54.597 --> 00:41:58.897
If it doesn't exhaust the space, then there must be other aspects to its organization

00:41:58.897 --> 00:41:59.977
that it doesn't capture.

00:42:01.277 --> 00:42:05.857
One aspect we skipped entirely, this was emotion relationship.

00:42:06.617 --> 00:42:12.637
Of course, as we can speak about actions we do with our body,

00:42:12.737 --> 00:42:15.257
we can speak about objects in the world.

00:42:15.257 --> 00:42:21.837
In a very similar sense, we can speak about entities that are,

00:42:22.037 --> 00:42:29.597
in some views, enclosed in our body, internal states, as they have been called.

00:42:30.017 --> 00:42:36.377
Like if I speak about happiness or joy, this is nothing that would be an object

00:42:36.377 --> 00:42:39.317
in the world or nothing, and not an action either.

00:42:39.497 --> 00:42:45.657
So how would that work? And of course, we link the meaning of these items to

00:42:45.657 --> 00:42:48.117
activation in the limbic system.

00:42:48.337 --> 00:42:56.117
So it's probably limbic circuits and basal ganglia activation,

00:42:56.637 --> 00:43:05.537
amygdala and cingulum activation, anterior cingulum, and some parts of the insula play a role.

00:43:05.537 --> 00:43:19.237
So, we have this corticolimbic linkage here for internal state emotion words especially.

00:43:20.077 --> 00:43:24.957
Now, the problem here is how to teach a child.

00:43:27.417 --> 00:43:35.177
Which internal concept, internal state a word is used to speak about.

00:43:35.437 --> 00:43:43.117
Because you cannot see the internal state and you cannot point to an object and say, and this is joy.

00:43:43.637 --> 00:43:47.017
So you can and there's a problem.

00:43:48.097 --> 00:43:53.857
And the answer, which actually goes back to the language philosopher Wittgenstein,

00:43:53.997 --> 00:44:02.277
he says He says, well, I can teach the meaning of these words because the child

00:44:02.277 --> 00:44:08.717
has a natural tendency of expressing these internal states in behavior, in actions.

00:44:08.997 --> 00:44:18.217
So if I want to teach a child what joy means, there's a very simple pathway.

00:44:18.217 --> 00:44:22.197
Way i wait until the the child shows

00:44:22.197 --> 00:44:28.677
joy behavior expresses joy in in its behavior and then i say well you're you're

00:44:28.677 --> 00:44:33.857
joyful today all right yeah and and this way they i can establish a correlation

00:44:33.857 --> 00:44:41.917
uh between between the word use and the and and the limbic activation which by assumption would.

00:44:43.077 --> 00:44:44.817
Produce this behavior.

00:44:45.037 --> 00:44:50.177
However, the link only works because there's an expression in behavior.

00:44:50.417 --> 00:44:56.177
There's a manifestation of the internal state in behavior, in actions,

00:44:56.957 --> 00:44:58.497
and therefore the motor system.

00:44:59.332 --> 00:45:02.172
Comes in here and we actually showed actually

00:45:02.172 --> 00:45:05.432
my doctorate student rachel mostly in cambridge she

00:45:05.432 --> 00:45:08.652
she showed that for abstract emotion words the

00:45:08.652 --> 00:45:12.092
most highly abstract emotion word there is motor systems

00:45:12.092 --> 00:45:15.192
activation and interestingly it's those parts of the

00:45:15.192 --> 00:45:18.112
motor system that are usually used to

00:45:18.112 --> 00:45:21.712
express emotions namely the face and the hands so

00:45:21.712 --> 00:45:24.472
it's the upper part of the body so this is

00:45:24.472 --> 00:45:27.632
really amazing right because you're you're basically saying of course there's

00:45:27.632 --> 00:45:30.692
a little little caveat because limbic system might also

00:45:30.692 --> 00:45:33.832
involve then subcortical structures no that's no

00:45:33.832 --> 00:45:36.852
caveat at all okay that's part of the game no but

00:45:36.852 --> 00:45:41.352
earlier you wanted to stick to the neocortex as your mixer your mixing system

00:45:41.352 --> 00:45:45.832
yes so now now you start to bring in so maybe it's a bit broader than only absolutely

00:45:45.832 --> 00:45:51.932
absolutely so so this this semantic model is not restricted to the cortex It

00:45:51.932 --> 00:45:55.172
has these limbic tails, if you like.

00:45:55.812 --> 00:46:01.632
It's good that we agree on that. So now, what you say is very profound,

00:46:01.792 --> 00:46:06.172
but on the other hand, it might also appear very sort of, in some sense, obvious.

00:46:06.172 --> 00:46:10.092
Because you could say what your observation is,

00:46:10.412 --> 00:46:15.792
so over a large number of experiments using different techniques in humans that,

00:46:15.892 --> 00:46:21.452
okay, meaning in the end is expressed in a broad response in the brain that

00:46:21.452 --> 00:46:28.712
is expressing, let's say, the experience of the subject with whatever the evoking event is.

00:46:28.952 --> 00:46:33.012
And if it's an emotional component, emotion areas of the brain will come in.

00:46:33.072 --> 00:46:35.872
If it's an action component, action areas will come in, et cetera.

00:46:36.172 --> 00:46:39.892
But then you could say okay but.

00:46:40.672 --> 00:46:45.332
Isn't that in some sense obvious because I'm just sort of mirroring the statistics

00:46:45.332 --> 00:46:52.132
of the world in which I exist and this is then again reflected and then it seems so unspecific,

00:46:53.012 --> 00:47:00.172
so that means it's not so the problem there is maybe also that you rely for

00:47:00.172 --> 00:47:03.732
instance a lot on fMRI, fMRI is a fairly slow signal.

00:47:04.815 --> 00:47:09.475
So, this might make it difficult to actually understand what really the core

00:47:09.475 --> 00:47:10.835
is of that meaning network.

00:47:11.855 --> 00:47:14.755
There might be a beginning and an end to this meaning network.

00:47:15.035 --> 00:47:19.875
Maybe what you're looking at are like really the last few ripples in the semantic

00:47:19.875 --> 00:47:26.195
system that indeed have sort of pervaded the system to really its periphery

00:47:26.195 --> 00:47:28.175
in meaning space in this case.

00:47:28.255 --> 00:47:31.935
And you have not really looked at the core of that meaning system.

00:47:32.875 --> 00:47:38.975
Yeah. What would be the criterion for core and distant relationship?

00:47:39.215 --> 00:47:44.295
Well, let me try to rephrase your question. Of course, if I hear a word,

00:47:44.435 --> 00:47:47.455
I would first understand its meaning, and then I would think,

00:47:47.535 --> 00:47:51.095
and then the word may remind me of something else.

00:47:51.095 --> 00:47:55.675
I may have a second-order process of being reminded of an event.

00:47:55.775 --> 00:48:04.635
I hear about freeing, and then I'm reminded of the news last night about some

00:48:04.635 --> 00:48:09.295
report from a certain country,

00:48:09.395 --> 00:48:14.555
and then I think about that and reprocess that and think about the freeing action

00:48:14.555 --> 00:48:19.495
some rebels have performed there.

00:48:19.495 --> 00:48:24.855
And of course this takes time after a while I may think about very distantly

00:48:24.855 --> 00:48:29.975
related issues is this what you mean for instance yeah and so here I,

00:48:30.760 --> 00:48:36.260
We have, of course, a handle to address this experimentally.

00:48:36.460 --> 00:48:40.980
FMRI, as you said, isn't probably the best tool here because it's sluggish,

00:48:40.980 --> 00:48:42.640
it's slow, it's like a snail.

00:48:44.500 --> 00:48:48.660
And we cannot distinguish whether this activation, strictly speaking,

00:48:48.920 --> 00:48:54.860
of the hand motor cortex when hearing the word grasp or to write,

00:48:55.800 --> 00:49:00.840
whether this activation of the hand region in this case is actually the result

00:49:00.840 --> 00:49:05.480
of understanding the word or relates to the understanding of the word or thinking

00:49:05.480 --> 00:49:11.080
twice about it or imagining something distantly related to the word. We cannot.

00:49:11.260 --> 00:49:17.460
No way. Because hopping from one association to the next in this experiment

00:49:17.460 --> 00:49:24.340
may take only 200-400 milliseconds and this is not the temporal resolution of MRI for that.

00:49:24.340 --> 00:49:31.760
We need imaging with exact temporal resolution, with millisecond precision.

00:49:32.840 --> 00:49:37.080
EEG is good for that. MEG, magnetoencephalography, is ideal.

00:49:37.340 --> 00:49:41.380
And we have done such experiments. And in a nutshell,

00:49:41.540 --> 00:49:47.520
we see in MEG and EEG experiments that these motor systems activation,

00:49:47.680 --> 00:49:57.820
for example, also the differential activation of visual areas in color and form-related

00:49:57.820 --> 00:50:03.420
word processing come up very quickly within 200 milliseconds.

00:50:03.820 --> 00:50:07.920
And this is as early as the earliest signs of semantic processes.

00:50:08.220 --> 00:50:11.980
But can you say something about the sequence in which this is unfolding?

00:50:12.500 --> 00:50:16.460
So I give you a word. It says, let's say, grasp.

00:50:16.700 --> 00:50:21.380
Yes. So what will be now the sequence of events in the brain.

00:50:22.334 --> 00:50:26.894
That show you that this semantic network is unfolding.

00:50:27.754 --> 00:50:31.854
So we started with, let's say, you're reading it. So first we have some response

00:50:31.854 --> 00:50:33.634
in the early visual system.

00:50:33.854 --> 00:50:37.034
But what's next? Is it really unfolding in a very sequential way,

00:50:37.154 --> 00:50:39.194
like we're slowly crawling up this hierarchy?

00:50:39.634 --> 00:50:46.034
Or do you jump, let's say, to a more forward frontal area that could be,

00:50:46.034 --> 00:50:49.714
let's say, more abstract, and from there you go back into these modality-specific

00:50:49.714 --> 00:50:53.134
representations? What's the order in which this really occurs?

00:51:20.754 --> 00:51:28.874
And from there, activation spreads to other regions, the posterior superior

00:51:28.874 --> 00:51:34.954
temporal region and inferior frontal cortex, and as well to the motor system.

00:51:34.954 --> 00:51:41.234
So, for example, with spoken words, we have measured with MEG a delay of peak

00:51:41.234 --> 00:51:48.294
activations in superior temporal cortex of about 120 milliseconds after the

00:51:48.294 --> 00:51:54.774
point of word recognition when the subject could first identify the upcoming word.

00:51:54.774 --> 00:52:00.374
Of course, if you have a word like crocodile, you cannot be sure whether it

00:52:00.374 --> 00:52:01.634
will be crocodile or crocus.

00:52:01.754 --> 00:52:06.594
But at one stage, you are sure it will be crocodile and not something else.

00:52:06.734 --> 00:52:13.894
And at that point, it takes another 120 or so milliseconds until we see this

00:52:13.894 --> 00:52:16.914
activation peak already in superior temporal cortex.

00:52:17.054 --> 00:52:22.294
And it's another 20 milliseconds more until the inferior frontal cortex.

00:52:23.612 --> 00:52:27.112
Cortex and the lower part of the motor strip comes in.

00:52:27.332 --> 00:52:32.432
And for leg words, the delays are a little bit larger.

00:52:32.832 --> 00:52:36.232
Activation has to travel to the top of the motor strip.

00:52:36.952 --> 00:52:43.672
So there we measured 170 milliseconds. So there's a slight difference in the time delays.

00:52:44.012 --> 00:52:49.912
It's rather quick. 130 milliseconds in the back, 150 in the front,

00:52:50.012 --> 00:52:52.652
and 170 at the top of the motor strip.

00:52:52.652 --> 00:53:00.392
And interestingly, already these inferior frontal and superior central activations

00:53:00.392 --> 00:53:03.952
at 140, 50, 60, 70 milliseconds,

00:53:04.272 --> 00:53:12.332
they show a modulation which can be related, correlated with meaning aspects of the words. Okay.

00:53:13.012 --> 00:53:18.332
But now, do you see this unfolding process as being tightly regulated by some,

00:53:18.392 --> 00:53:22.092
let's say, a hub somewhere that's connected to all these structures that is

00:53:22.092 --> 00:53:23.992
regulating how they unfold?

00:53:23.992 --> 00:53:28.892
This has been a proposal, an important proposal by Karamazov, for example.

00:53:29.032 --> 00:53:34.752
So the idea is, well, let me just rephrase this colloquially,

00:53:34.772 --> 00:53:37.412
that this motor system activation is more or less for fun,

00:53:37.472 --> 00:53:45.572
but that there's somewhere a semantic hub or a symbolic semantic system which pulls all strings.

00:53:45.572 --> 00:53:50.192
Strings, and at one stage activation may overflow to the motor system,

00:53:50.292 --> 00:53:54.532
there may be a little bit of a contribution here and there, but the semantic

00:53:54.532 --> 00:53:57.792
system, the real semantic processing should be done elsewhere.

00:53:59.469 --> 00:54:04.629
Now, we don't see this. We don't see this generally, that activation always

00:54:04.629 --> 00:54:12.129
goes to temporal pole, for example, or to a different region before it goes to motor systems.

00:54:12.429 --> 00:54:19.989
And we see for abstract sentence and abstract concept processing,

00:54:20.189 --> 00:54:25.189
we see some activation which is outside the motor system, outside sensory systems,

00:54:25.209 --> 00:54:29.049
for example, in dorsolateral prefrontal cortex and anterior.

00:54:29.469 --> 00:54:32.089
Parts of the inferior frontal cortex.

00:54:33.169 --> 00:54:40.589
But these activations, they usually come up at latencies that are very similar

00:54:40.589 --> 00:54:50.209
to those where also modification of the motor strip activation reflecting semantic aspects,

00:54:50.449 --> 00:54:54.029
referential semantic aspects, have been observed.

00:54:54.029 --> 00:54:57.909
A recent experiment by my colleague Veronique Boulanger,

00:54:58.169 --> 00:55:01.189
who did a postdoc in Cambridge a few years ago,

00:55:01.429 --> 00:55:08.949
clarified this issue very nicely by looking at idiomatic expressions such as

00:55:08.949 --> 00:55:18.889
grasp the idea or kick the habit. bit.

00:55:20.509 --> 00:55:26.309
So these are kick and grasp sentences, hand sentences and leg sentences and

00:55:26.309 --> 00:55:30.729
they activated to a degree differentially the motor strip and,

00:55:31.210 --> 00:55:37.690
And this differential motor strip activation happened around 150 to 200 milliseconds,

00:55:37.890 --> 00:55:41.330
the earliest significant differences were there.

00:55:41.650 --> 00:55:49.090
Now, we also had literal sentences in this experiment and compared those with idiomatic ones.

00:55:49.330 --> 00:55:53.590
And this general idiomaticity difference, that was.

00:55:54.410 --> 00:55:58.130
Especially nicely reflected in dorsolateral prefrontal cortex,

00:55:58.250 --> 00:56:01.990
not in the motor system, interestingly and also

00:56:01.990 --> 00:56:05.510
in anterior temporal cortex so and and

00:56:05.510 --> 00:56:08.210
and now one could of course have a race when is

00:56:08.210 --> 00:56:11.670
this general symbolic difference abstract versus

00:56:11.670 --> 00:56:19.030
versus versus concrete meaning idiomatic versus versus literal when is this

00:56:19.030 --> 00:56:24.830
when does this first come up which would allow for a conclusion on any general

00:56:24.830 --> 00:56:30.170
symbolic system activation if if one wanted to phrase it this way.

00:56:30.570 --> 00:56:34.690
And when would this so-called embodied activation of the motor system,

00:56:34.830 --> 00:56:42.470
differential between arm and leg sentences, even though they are abstract, come up?

00:56:42.510 --> 00:56:47.670
And we use this trigger point here, not of course the action-related words,

00:56:47.830 --> 00:56:56.010
but those words that disambiguated the meaning between idiomatic or concrete

00:56:56.010 --> 00:56:58.010
literal sentence meaning,

00:56:58.170 --> 00:57:06.470
so the point of when they catch the ball or catch the apple,

00:57:06.650 --> 00:57:13.330
this noun at the end would then distinguish between the literal and the idiomatic meaning,

00:57:13.490 --> 00:57:16.310
or grasp the idea deer

00:57:16.310 --> 00:57:19.130
or grass to

00:57:19.130 --> 00:57:22.710
cup yeah exactly and when when

00:57:22.710 --> 00:57:25.390
those come up and out and and it

00:57:25.390 --> 00:57:30.830
was just about 150 milliseconds after this word was flashed on screen that we

00:57:30.830 --> 00:57:37.910
saw both the this prefrontal anterior temporal idiomaticity effect and also

00:57:37.910 --> 00:57:43.010
the the hand leg dissociation in the motor strip So the embodied,

00:57:43.170 --> 00:57:54.790
there's parity between the embodied and this potentially more general idiomaticity effects.

00:57:55.930 --> 00:58:00.350
It's not so that one can easily be said to drive the object.

00:58:00.370 --> 00:58:02.790
Exactly. So this is an important consequence of this, right?

00:58:02.790 --> 00:58:07.870
So in your mind, this is not really orchestrated by some hidden module somewhere.

00:58:08.210 --> 00:58:10.950
No, there isn't a semantic homunculus. Right, exactly. Exactly.

00:58:10.970 --> 00:58:16.590
This is really just evolving out of the network dynamics without a central orchestration.

00:58:16.630 --> 00:58:23.110
Yes. The orchestration is actually, as we would say, is done by the neuronal

00:58:23.110 --> 00:58:25.670
units that can be widely distributed.

00:58:26.755 --> 00:58:31.075
Exactly. But there's not a central conductor who says now you and now you. Yes, exactly.

00:58:31.295 --> 00:58:35.895
So this is good, right? So here we have this dynamical response,

00:58:36.175 --> 00:58:39.495
which will vary dependent on the references to the outside world.

00:58:40.695 --> 00:58:44.935
But on the other hand, there's this issue of specificity because your test case

00:58:44.935 --> 00:58:50.355
is very much action words, if you want, where you can show specificity to some extent.

00:58:50.355 --> 00:58:53.955
If it's, let's say, a hand word, you would see more, let's say,

00:58:53.975 --> 00:58:58.795
hand-related responses in the sematosensory system.

00:58:59.155 --> 00:59:03.695
While it's kicking a foot action, you would see more foot-related responses.

00:59:03.695 --> 00:59:09.875
But you were also fair enough this morning to show results from some of your

00:59:09.875 --> 00:59:13.675
colleagues or competitors, if you want, who showed that actually these responses

00:59:13.675 --> 00:59:14.835
might not be that specific,

00:59:15.015 --> 00:59:19.795
that you also might see responses in this motor area to completely nonsensical

00:59:19.795 --> 00:59:22.155
words or non-action words.

00:59:23.935 --> 00:59:27.415
So this might be a threat to your theory because they say, look,

00:59:27.475 --> 00:59:29.835
these things might be responding all the time.

00:59:30.755 --> 00:59:35.855
Not quite. Okay. Because motor semantics or action semantics is not the only

00:59:35.855 --> 00:59:38.255
thing that drives the motor system.

00:59:38.415 --> 00:59:44.915
So we have shown in a little series of experiments that also phonological features

00:59:44.915 --> 00:59:51.695
of words and nonsense sounds drive the motor system.

00:59:52.595 --> 00:59:55.735
And we even hear it to natural sound.

00:59:55.915 --> 00:59:59.835
We even see this motor system's activation to natural sounds.

00:59:59.835 --> 01:00:01.475
For example, take this out.

01:00:02.735 --> 01:00:13.175
My colleague, Olive Hoke, in Cambridge, he did an experiment where he used some neutral sounds,

01:00:13.375 --> 01:00:17.875
like a metronome, and then occasionally there was an unexpected sound.

01:00:26.143 --> 01:00:32.183
These were clicks, either just clicks that have no body relationship.

01:00:32.563 --> 01:00:37.683
Or it was sounds produced by the tongue, sounds produced by the finger.

01:00:37.923 --> 01:00:44.463
And interestingly, he found for these entirely non-linguistic sounds,

01:00:44.543 --> 01:00:47.963
activation of different parts of the motor strip.

01:00:48.043 --> 01:00:53.583
Very early, very similar to our word, evoked activation, but it's not specific

01:00:53.583 --> 01:00:55.803
to the meaning of words in no way.

01:00:56.143 --> 01:01:06.343
And when looking now at brain activation elicited by language sounds that are

01:01:06.343 --> 01:01:10.363
produced with a tongue or with the lips, like p,

01:01:10.583 --> 01:01:12.663
t, t, p,

01:01:12.983 --> 01:01:21.083
p, k, they are either sounds produced by the tongue, sounds produced by the lips.

01:01:21.083 --> 01:01:24.023
When people hear this,

01:01:24.163 --> 01:01:29.783
we see a trace of activation of the motor strip that reflects the body part

01:01:29.783 --> 01:01:35.283
relationship of these sounds, namely the articulator that plays the biggest

01:01:35.283 --> 01:01:36.583
role in the articulation,

01:01:36.703 --> 01:01:39.763
either the tongue or the lips.

01:01:39.763 --> 01:01:45.063
So there, even for nonsense words, for example, those starting with a P or a

01:01:45.063 --> 01:01:52.583
T, there's good reason to see activation in the, at least in the inferior part of the motor strip.

01:01:52.703 --> 01:01:57.623
So this is not damaging at all to the semantics proposal.

01:01:57.763 --> 01:02:05.043
Of course, it calls for very careful experimental manipulation, because if I compare….

01:02:06.149 --> 01:02:14.089
Words that start with t with hand semantics to words that start with with p and uh with with,

01:02:14.889 --> 01:02:20.769
face relationship semantic face relationship then there may be a severe confound

01:02:20.769 --> 01:02:26.589
and experiment results maybe may look different much different from if if this

01:02:26.589 --> 01:02:31.289
phonological features were were matched exactly so so this is but another interesting

01:02:31.289 --> 01:02:33.889
consequence i think of your work is that you can say,

01:02:33.949 --> 01:02:36.409
look, maybe in terms of our linguistic conventions.

01:02:37.069 --> 01:02:42.229
The word might not have such a motor component, but maybe for your brain it does.

01:02:42.509 --> 01:02:46.909
And we just don't really understand yet how that's wired into a broader semantic system.

01:02:47.189 --> 01:02:51.389
So in that sense, maybe it's not so much a challenge to your theory.

01:02:51.529 --> 01:02:58.249
It also opens a new window on how we can rethink semantics of maybe even non-action words, right?

01:02:58.289 --> 01:03:03.049
There will be a whole set of non-action words that is able to trigger activation in the motor system.

01:03:03.429 --> 01:03:07.329
So possibly in the semantic networks the brain is building up,

01:03:07.489 --> 01:03:11.329
there are actually action components that we have overlooked in our linguistic analysis.

01:03:11.789 --> 01:03:17.889
Yes. Yeah. And the most obvious example here are these abstract emotion words,

01:03:18.209 --> 01:03:22.969
where nobody would have imagined an activation of the motor system.

01:03:23.269 --> 01:03:29.669
There's semantically, Basically, if one asks about action features or object

01:03:29.669 --> 01:03:33.389
relationship referential features, there's no nothing there.

01:03:33.649 --> 01:03:38.829
But one wouldn't say this is what one uses these words to speak about actions.

01:03:38.829 --> 01:03:43.089
One would say it would speak about internal states, but nevertheless,

01:03:43.569 --> 01:03:51.269
semantic theory of a different kind would stipulate, postulate that the meaning

01:03:51.269 --> 01:03:57.569
of these words can only be learned by way of having actions.

01:03:59.560 --> 01:04:04.760
That are manifestations, criteria for the presence of these internal states.

01:04:05.180 --> 01:04:09.220
Right. But then… So this is something that's a new contribution.

01:04:09.480 --> 01:04:13.600
Exactly. And a contribution that actually speaks to an issue which is of great

01:04:13.600 --> 01:04:19.500
relevance for the theory of language and semantics and philosophy of language.

01:04:19.680 --> 01:04:24.140
Absolutely. No, I completely get the point. And I don't think anyone has really

01:04:24.140 --> 01:04:28.960
followed it up yet within the linguistic community, or am I wrong?

01:04:29.440 --> 01:04:33.900
Have the linguists responded yet to this? Well, it would be a bit quick because

01:04:33.900 --> 01:04:36.340
the paper has just been published a few months ago.

01:04:36.340 --> 01:04:39.540
Yeah, but in some sense it's already a consequence of earlier work that,

01:04:39.580 --> 01:04:45.220
for instance, that you see responses in motor areas to non-action words is already

01:04:45.220 --> 01:04:47.360
raising that question. Yes. Right?

01:04:47.880 --> 01:04:52.040
So that question is now on the table for some time. Yes.

01:04:52.300 --> 01:04:57.220
So that should be followed up. But now, what you have done, your theory,

01:04:57.940 --> 01:05:02.000
which in some sense goes back to Donald Happ and it might even go back to Thorndike

01:05:02.000 --> 01:05:06.320
who also would say the brain is, you know, a connection machine. Yes.

01:05:07.020 --> 01:05:10.120
And in some sense, you postulate a very simple rule. You say,

01:05:10.160 --> 01:05:12.560
look, you know, the brain just follows the statistics of its inputs.

01:05:13.580 --> 01:05:16.700
And if these statistics linguistics coincide with

01:05:16.700 --> 01:05:20.340
a word yeah then we have semantics yes

01:05:20.340 --> 01:05:23.400
indeed and but of course when when saying this

01:05:23.400 --> 01:05:30.720
in the in a cognitive theory context uh well there are some colleagues who look

01:05:30.720 --> 01:05:37.560
at you as if the devil was confronted with holy water so to speak that's right

01:05:37.560 --> 01:05:42.680
you are the devil in that analogy or the holy water i was about to phrase it the other way around.

01:05:44.320 --> 01:05:51.220
Well, you can see it from whatever perspective you approach it.

01:05:51.280 --> 01:05:58.640
So it was a sacrilegious, it was an entire no-go to speak about correlations

01:05:58.640 --> 01:06:01.220
in such abstract domains as semantics.

01:06:01.440 --> 01:06:07.120
And some people are really very much opposed to that.

01:06:09.417 --> 01:06:15.717
And this has a certain relationship to statements that had been made also in

01:06:15.717 --> 01:06:17.897
the behaviorist tradition.

01:06:18.597 --> 01:06:26.237
Of course, relating the meaning of a word to actions and stimuli and responses

01:06:26.237 --> 01:06:29.557
that had been, such ideas had been.

01:06:29.657 --> 01:06:32.157
That sounds like Skinner's theory of language. Exactly, yeah.

01:06:32.277 --> 01:06:39.137
There is a relationship. Now, if you read Skinner, this is much too primitive to work at any level.

01:06:39.417 --> 01:06:45.017
And, of course, it was exactly the behaviorists who denied any internal states.

01:06:45.437 --> 01:06:50.917
So, theorizing about cell assemblies in the cortex that represent cognitive

01:06:50.917 --> 01:06:54.077
representations would be an entire no-go.

01:06:54.917 --> 01:06:59.057
Well, except when you would talk with more cognitive behaviorists like Tolman and Hull.

01:07:00.077 --> 01:07:04.697
So, I think Skinner and Watson were a bit more at one extreme end of the scale.

01:07:04.977 --> 01:07:08.897
But, yes, you seem to be alluding to these kinds of concepts.

01:07:08.897 --> 01:07:12.557
And of course, that's asking for trouble when you are confronted with domains

01:07:12.557 --> 01:07:15.777
that are still more dominated by this cognitivist perspective.

01:07:15.777 --> 01:07:23.157
Yeah, but I think leaving aside all these to a degree religious ideas,

01:07:23.357 --> 01:07:30.557
we should be clear that we want to understand cognitive mechanisms and we want

01:07:30.557 --> 01:07:34.877
to spell out their basis in terms of neural circuits.

01:07:34.877 --> 01:07:41.897
We are not satisfied and many, many, I think the majority I can say is not satisfied

01:07:41.897 --> 01:07:44.157
any longer with box and arrow diagrams.

01:07:44.377 --> 01:07:53.037
We want to become more concrete and using certain standard perceptron architectures

01:07:53.037 --> 01:08:02.277
is still a little bit distant from cortical anatomy and function as well.

01:08:02.801 --> 01:08:08.041
So we are trying to use models that are more inspired by cortical anatomy.

01:08:08.481 --> 01:08:12.621
What would be the minimal ingredients of the model?

01:08:12.741 --> 01:08:17.241
What are the minimum components the model should have to build up these kinds

01:08:17.241 --> 01:08:18.701
of semantic representations?

01:08:19.401 --> 01:08:24.041
I guess we need sort of neural units. They need to have a firing state.

01:08:24.561 --> 01:08:30.281
They need to have connections. These connections must be able to change dependent on activity levels.

01:08:30.281 --> 01:08:35.201
But what are then the minimum agreed ingredients of this model to replicate

01:08:35.201 --> 01:08:39.601
your results? Well, there would need to be a relationship between the model

01:08:39.601 --> 01:08:41.901
and its parts and the brain.

01:08:42.281 --> 01:08:48.181
And that's not so easy because there are many areas that are of relevance.

01:08:48.481 --> 01:08:53.741
And there's the back part of the language cortex, the pericelial regions,

01:08:54.121 --> 01:09:01.141
the front part, there's motor-premotor, prefrontal cortex, there's auditory

01:09:01.141 --> 01:09:04.001
cortex, auditory belt, auditory parabelt.

01:09:04.001 --> 01:09:08.941
Well, then visual cortices come in.

01:09:09.021 --> 01:09:15.281
Other motor fields for controlling hand and leg actions are important.

01:09:15.541 --> 01:09:19.521
So it's quite a range of different areas and their connectivity.

01:09:19.861 --> 01:09:22.801
So it's quite a little bit of work. But you could argue, well,

01:09:22.881 --> 01:09:26.661
I don't know, maybe that's easier than it sounds because certainly if we want

01:09:26.661 --> 01:09:31.281
to restrict ourselves to the neocortex, you do know that the different areas

01:09:31.281 --> 01:09:36.221
you point to in terms of the local circuits are relatively similar.

01:09:36.461 --> 01:09:42.801
Yes. So in terms of the computational unit of your model, it might actually

01:09:42.801 --> 01:09:43.701
be fairly straightforward.

01:09:44.501 --> 01:09:51.621
Or not? No. Okay, why not? Because the whole architecture influences the nature

01:09:51.621 --> 01:09:53.301
of the computational unit.

01:09:53.301 --> 01:10:00.061
It just happens that if we have correlated activation in such a network on the

01:10:00.061 --> 01:10:03.841
motor end and on the auditory end of the network,

01:10:04.081 --> 01:10:11.061
that then waves of activation spread through the various areas of the,

01:10:11.081 --> 01:10:15.221
let's call them areas instead of layers, areas of the network.

01:10:15.221 --> 01:10:22.581
And therefore, populations of neurons strengthen their connections all over the network.

01:10:22.581 --> 01:10:26.261
And what we finally end up is our cell assemblies,

01:10:26.361 --> 01:10:32.041
neuronal assemblies, neuronal representations that link between motor and auditory

01:10:32.041 --> 01:10:41.741
visual input-output patterns and have a center actually in those areas that

01:10:41.741 --> 01:10:45.401
link motor systems to auditory and visual systems.

01:10:45.401 --> 01:10:54.601
Like, in fact, the prefrontal cortex and some higher association temporal parietal region.

01:10:55.627 --> 01:11:02.907
And those are particularly important for holding together these distributed units of processing.

01:11:03.167 --> 01:11:09.227
So, we get what Hebb postulated, namely these cell assemblies.

01:11:09.307 --> 01:11:15.287
And in Breitenberg's hands, actually, the proposal was made for the first time

01:11:15.287 --> 01:11:19.387
that these networks could actually bridge between cortical areas.

01:11:19.907 --> 01:11:23.347
And this is also what we find in the simulations.

01:11:23.347 --> 01:11:28.707
And what we think is the best explanation for this extremely rapid spreading,

01:11:28.987 --> 01:11:34.967
which relates to aspects of the meaning of words and construction.

01:11:34.967 --> 01:11:38.387
Right, so in some sense that would mean, okay, there's a, let's say there's

01:11:38.387 --> 01:11:40.587
an input to this network, to the brain.

01:11:41.547 --> 01:11:46.307
The input triggers different areas. They start to propagate their activities

01:11:46.307 --> 01:11:47.387
through the rest of the network.

01:11:47.647 --> 01:11:51.027
These activation patterns might coincide in different places.

01:11:51.347 --> 01:11:55.367
I pick that up in my connections, and now I have, let's say,

01:11:55.407 --> 01:11:59.807
additional memory-related responses to these semantic events.

01:12:00.187 --> 01:12:01.987
This would be roughly the right interpretation.

01:12:02.427 --> 01:12:05.247
Yes, yes. But wouldn't that imply then you would have hubs, no?

01:12:05.607 --> 01:12:10.847
Then you are actually involving hubs in your network. Well, what we end up with

01:12:10.847 --> 01:12:15.387
after this learning exercise are functional units.

01:12:16.327 --> 01:12:22.907
It's actually quite in their behavior, not analog, but discrete processing units.

01:12:22.907 --> 01:12:25.667
These cell assemblies, they are strongly connected networks,

01:12:25.907 --> 01:12:32.607
and partial activation of a significant part of the network would lead to an

01:12:32.607 --> 01:12:33.927
explosion-like process,

01:12:34.167 --> 01:12:38.107
activation spreading through the network.

01:12:38.367 --> 01:12:40.887
And so the cell assembly becomes active.

01:12:41.687 --> 01:12:46.907
And there's actually a danger of having activation spreading everywhere.

01:12:46.907 --> 01:12:52.207
And we need inhibition, we need to have regulation, regulation mechanisms in

01:12:52.207 --> 01:12:56.687
order to keep down the general level of activation and make sure that with one

01:12:56.687 --> 01:13:00.787
activation of a word or conceptual or semantic representation,

01:13:01.107 --> 01:13:04.587
we do not activate 10 more within an instance.

01:13:04.587 --> 01:13:15.067
So we have strong inhibition, and this inhibition could be titrated so that

01:13:15.067 --> 01:13:18.187
only one cell assembly becomes active at a time.

01:13:18.307 --> 01:13:24.647
Of course, if it's not properly adjusted, you get… Yeah, it will explode, basically.

01:13:24.727 --> 01:13:29.687
Explode schizophrenia or epileptic networks.

01:13:30.727 --> 01:13:32.087
But no, there's another…,

01:13:33.085 --> 01:13:36.285
prediction here i'm not sure if that's the one you would like to see but,

01:13:37.305 --> 01:13:40.745
um if signals travel in the brain you will

01:13:40.745 --> 01:13:44.885
have a transduction latency of the signal with the distance it has to travel

01:13:44.885 --> 01:13:50.705
yes and this the largest the semantic net the potential semantic network is

01:13:50.705 --> 01:13:54.645
rather let's say roughly could be anything in that in the neocortex so we forget

01:13:54.645 --> 01:13:59.685
limbic system but that means i'm going to link together areas at varying distances,

01:14:00.405 --> 01:14:06.245
So, these intersection points, right, where the activity will overlap will vary

01:14:06.245 --> 01:14:08.845
with the distances between these areas, right?

01:14:08.845 --> 01:14:11.645
Because I have rhythmic activity, there's a transduction latency.

01:14:11.785 --> 01:14:15.985
So, the points where these activation waves will overlap, so learning can occur

01:14:15.985 --> 01:14:21.425
following your rule, will vary with, let's say, the initial points of activity.

01:14:21.425 --> 01:14:27.485
So you get regular tessellation of these hubs or however, these intermediate

01:14:27.485 --> 01:14:32.145
points dependent on the initial spatial configuration that you trigger.

01:14:32.605 --> 01:14:35.365
Is it also something you see in your fMRI data?

01:14:39.845 --> 01:14:45.085
That with different word types we see, well.

01:14:46.145 --> 01:14:50.925
Maybe it's not clear. No, I wasn't fully with you.

01:14:50.925 --> 01:15:01.465
Are you aiming at a potential imaging difference between one of these areas

01:15:01.465 --> 01:15:05.885
that hold together the cell assembly and the peripheral?

01:15:06.485 --> 01:15:11.865
Let's do auditory action words that are spoken, and I'm reading action words.

01:15:12.165 --> 01:15:18.705
Yeah. So now the distance from temporal cortex or my auditory responses and

01:15:18.705 --> 01:15:24.325
the motor cortex, that distance is different from the distance between my occipital

01:15:24.325 --> 01:15:26.565
cortex, my visual response, and the motor cortex.

01:15:26.765 --> 01:15:31.305
So now both these areas will become activated in some way, right?

01:15:31.305 --> 01:15:35.145
So now their activity starts to propagate through the broader network,

01:15:35.345 --> 01:15:39.805
but the intersection points of this activation will be at different distances

01:15:39.805 --> 01:15:42.785
in between these areas because their distance is different.

01:15:43.245 --> 01:15:47.685
And if your network model is correct and would predict,

01:15:47.825 --> 01:15:53.905
I see activity islands building up somewhere between temporal and motor when

01:15:53.905 --> 01:16:01.765
it's auditory and more, let's say, more towards occipital when it's visual motor. This is correct.

01:16:02.005 --> 01:16:06.705
And we see, for example, in the latency that with visual word presentation,

01:16:07.025 --> 01:16:13.385
the delay of this motor systems activation is a little bit delay, a little bit larger.

01:16:13.625 --> 01:16:16.785
So it's 200 to 220, 40 milliseconds.

01:16:17.125 --> 01:16:21.945
Now with auditory, as I said before, it's 150 to 160, 70 milliseconds.

01:16:21.945 --> 01:16:30.165
Second so it's earlier with the auditory with the earlier presentation as as far as we can draw um uh.

01:16:31.150 --> 01:16:34.190
This this comparison because of course there are

01:16:34.190 --> 01:16:37.890
basic differences between auditory presentation which

01:16:37.890 --> 01:16:40.790
is piecemeal crop or dial i i

01:16:40.790 --> 01:16:46.570
get a phoneme by phoneme uh now with with crocodile being presented flashed

01:16:46.570 --> 01:16:50.570
on the screen i have all the information in one shot right it could also be

01:16:50.570 --> 01:16:56.730
that this this heavy shot of information uh requires more time and therefore

01:16:56.730 --> 01:16:58.210
the delay the delay is larger.

01:16:58.350 --> 01:17:04.570
So we cannot uniquely attribute these delay differences to cortical distance.

01:17:05.230 --> 01:17:10.130
There's possibly an influence of the nature of the experiment.

01:17:10.450 --> 01:17:19.130
But as far as this differential activation dynamics are concerned,

01:17:19.390 --> 01:17:25.430
there are indeed differences in the delays and also in the pathway of the cortical spreading. Okay.

01:17:25.610 --> 01:17:32.190
But is it also expressed in, let's say, the activation of areas in between these…

01:17:32.190 --> 01:17:38.510
Yes, especially inferior frontal cortex, Broca's region, seems to play a role. Okay.

01:17:38.770 --> 01:17:43.190
But another consequence of the model that I was wondering about is that your

01:17:43.190 --> 01:17:45.650
model is highly sensitive to statistics, right?

01:17:45.650 --> 01:17:54.110
So the more components my semantics has, so the more combinatorial it gets,

01:17:54.250 --> 01:17:56.630
the more input states I have to sample.

01:17:57.710 --> 01:18:02.410
So the prediction of that would be that in order to build up a response to a

01:18:02.410 --> 01:18:06.130
word that has more components, I need more exposure.

01:18:06.250 --> 01:18:10.070
I need to see it more often because I'm frequency dependent.

01:18:10.710 --> 01:18:14.190
So learning will depend on the frequency with which things occur.

01:18:14.190 --> 01:18:16.970
Occur so do you do you observe that

01:18:16.970 --> 01:18:20.170
as well in your fmri data let's say let's say words action

01:18:20.170 --> 01:18:22.970
words that are less frequent will trigger different responses weaker

01:18:22.970 --> 01:18:26.850
responses than action words that are more frequent well

01:18:26.850 --> 01:18:31.950
we have a we have a degree of these are not very very strong effects but there

01:18:31.950 --> 01:18:36.990
there's a there's a degree of uh of of variability of this for example these

01:18:36.990 --> 01:18:43.950
motor systems and and and middle temporal cortex activations varying with word frequencies as Okay,

01:18:44.310 --> 01:18:48.510
but would you take that as a prediction of your model?

01:18:50.110 --> 01:18:56.130
Well, as you said, or at least implied, not only the frequency as such should

01:18:56.130 --> 01:19:02.270
be relevant, but also the complexity of the semantic relationship.

01:19:02.270 --> 01:19:13.770
So, if very variable semantic relationships exist of one given word, then of course the,

01:19:14.330 --> 01:19:18.450
additional learning steps may just lead to a weakening of the connection.

01:19:18.950 --> 01:19:25.110
So if I have, for example, a word that is used to speak about,

01:19:26.235 --> 01:19:38.815
about a wide range of objects and then water animals and then a word with the

01:19:38.815 --> 01:19:41.995
same frequency that is specifically used,

01:19:42.935 --> 01:19:46.975
to speak about one particular type of animal like frog.

01:19:47.175 --> 01:19:51.855
And then, of course, the semantic links may be very strong in this.

01:19:53.975 --> 01:19:58.835
Or unicorn. Unicorn would be not a difficult case, right? Yeah, indeed.

01:19:59.155 --> 01:20:02.415
Yeah. Yeah, unicorn would be especially difficult. Exactly.

01:20:02.795 --> 01:20:08.755
One would need to build it from different parts. Right. Horns and horses.

01:20:10.455 --> 01:20:15.675
But another amazing implication of your work is that you have pursuits.

01:20:15.675 --> 01:20:19.755
Like we talked earlier about this controversy, if you want, with more cognitivist approaches.

01:20:20.055 --> 01:20:24.395
Yes. Get out of here. This is too easy. It will never work like this.

01:20:24.395 --> 01:20:25.755
This is not possible. And so on.

01:20:28.355 --> 01:20:32.175
But you have taken actually, I think, a very courageous step by saying,

01:20:32.255 --> 01:20:36.815
okay, look, if this is all correct, I must be able to actually treat aphasia with this effectively.

01:20:37.195 --> 01:20:41.755
Yeah. So what was the concept there? How did you make that switch? What's the idea?

01:20:43.775 --> 01:20:49.275
Well, if you want the true and honest answer, it was the other way around.

01:20:49.535 --> 01:20:55.275
Okay. I started, it was aphasia therapy that started it all.

01:20:55.535 --> 01:21:03.775
Okay, interesting. I did my PhD in linguistics then on aphasia therapy,

01:21:03.895 --> 01:21:10.935
being convinced that if a theory has something to say, it must have a practical implication.

01:21:10.955 --> 01:21:16.535
This wasn't very popular in the last century, I should say.

01:21:16.755 --> 01:21:21.835
It still isn't that much, right? And I did this in a linguistics department,

01:21:21.935 --> 01:21:26.115
and the linguist said, well, is this really linguistics what this guy is doing?

01:21:27.735 --> 01:21:35.295
But I was kind of imprinted a little bit by my grandfather who suffered from

01:21:35.295 --> 01:21:42.455
a severe aphasia, and this always was a problem, and nobody could help him essentially.

01:21:42.455 --> 01:21:50.595
And so after studying linguistics, after learning about language therapy and

01:21:50.595 --> 01:21:52.795
also studying biology and learning

01:21:52.795 --> 01:21:56.675
about brain mechanisms, I thought that specializing in this direction,

01:21:56.975 --> 01:22:01.455
learning about brain and language relationships in order to produce something

01:22:01.455 --> 01:22:08.095
sensible for helping these patients would be an important thing to do.

01:22:08.095 --> 01:22:16.795
And then as a PhD student, I made a few proposals based on semantic theory,

01:22:17.035 --> 01:22:21.595
language philosophy, and some...

01:22:22.661 --> 01:22:25.641
Knowledge about neurobiological mechanisms.

01:22:26.901 --> 01:22:34.081
And this resulted then after, this was then not very successful.

01:22:34.121 --> 01:22:41.881
At the start, 1991, the first paper was published and was not received at all.

01:22:42.081 --> 01:22:47.361
And about 10 years later, I met with a famous U.S.

01:22:48.421 --> 01:22:54.721
Neuroscientist, Edward Taub, who had developed constrained-use movement therapy over dinner.

01:22:54.921 --> 01:22:59.301
He said, well, wouldn't you have an idea how we could project this into the

01:22:59.301 --> 01:23:05.201
language domain and link up with some behaviorally relevant language training

01:23:05.201 --> 01:23:08.701
similar to the motor therapy?

01:23:08.901 --> 01:23:12.601
And then I reactivated my previous.

01:23:14.641 --> 01:23:18.501
Interest and experience also.

01:23:18.621 --> 01:23:24.621
The idea of the therapy was to embed language in action context,

01:23:24.821 --> 01:23:32.781
actually in interactions, communicative interactions and this was very handy

01:23:32.781 --> 01:23:42.221
also for use in a in a mass practice context and we then ran a a randomized controlled trial,

01:23:42.421 --> 01:23:44.301
and it was extremely successful.

01:23:44.501 --> 01:23:51.781
That paper on aphasia therapy was really very well received,

01:23:52.381 --> 01:23:55.201
and we are continuing with this.

01:23:55.301 --> 01:24:00.681
And I hope that we will, at one stage, be successful to secure some money to do some important work.

01:24:03.919 --> 01:24:14.879
Work to optimize it further, to link it to computational work and to have computer

01:24:14.879 --> 01:24:19.639
versions of the therapy so that the intensity couldn't go up even further. Exactly.

01:24:19.919 --> 01:24:26.219
But in your case, you could interpret your approach also as if you want an inverted

01:24:26.219 --> 01:24:27.679
mirror approach, right?

01:24:27.679 --> 01:24:31.599
Because in mirroring, the mirror mechanisms, it's very often interpreted like,

01:24:31.679 --> 01:24:38.239
okay, it's the sensory state perception, visual perception of action that drives

01:24:38.239 --> 01:24:40.319
the motor execution pathway.

01:24:40.319 --> 01:24:44.179
So we go from the sensory state, the perceptual state, to the motor state.

01:24:44.559 --> 01:24:47.879
But in your case, you turn it around and say, no, no, wait, if I can tickle

01:24:47.879 --> 01:24:51.339
the motor state by having the subject gesture,

01:24:51.779 --> 01:24:58.019
I can actually now exactly follow that pathway backwards and tickle all the

01:24:58.019 --> 01:25:01.099
areas that are sort of upstream from it,

01:25:01.159 --> 01:25:04.859
that were under normal conditions talking to this motor system.

01:25:05.039 --> 01:25:06.839
Yes. Is this a correct interpretation?

01:25:07.439 --> 01:25:13.879
Yes. Okay. Okay, so then how effective is this as a therapeutic tool?

01:25:14.139 --> 01:25:19.139
The idea is that the language system and the motor system or the action system

01:25:19.139 --> 01:25:24.959
or the general action system of the brain, they are heavily linked to each other, interwoven. Yeah.

01:25:25.830 --> 01:25:32.350
And that if one fails, if one suffers from a lesion due to the strong connection

01:25:32.350 --> 01:25:37.930
to the other, this other system, this partially still intact system,

01:25:38.050 --> 01:25:41.970
could help restoring neural patterns of activation.

01:25:42.470 --> 01:25:49.330
And so that's the theory behind it. So lesion network could get help from a strong associate.

01:25:49.650 --> 01:25:54.690
The language system could get help from the action system. The theory and the

01:25:54.690 --> 01:25:57.110
data, they are pretty favorable.

01:25:57.290 --> 01:26:01.650
So we did randomize controlled trials. And it seemed in one study that this

01:26:01.650 --> 01:26:08.090
approach is more efficient than the therapy patients get.

01:26:08.310 --> 01:26:13.850
That's in chronic patients or acute patients? Chronic. We have been restricting

01:26:13.850 --> 01:26:20.670
ourselves to chronic patients for methodological reasons because in acute patients,

01:26:20.830 --> 01:26:29.290
there's such an unpredictable amount of spontaneous recovery that any conclusions

01:26:29.290 --> 01:26:38.790
of a randomized controlled trial is always in danger of receiving criticism.

01:26:38.790 --> 01:26:46.010
Because there could be just randomly, you could just by chance choose good recoverers

01:26:46.010 --> 01:26:50.270
and have bad ones in the other group.

01:26:50.390 --> 01:26:55.810
So you're never safe. But if you go to stages where spontaneous recovery is

01:26:55.810 --> 01:27:02.890
not so likely and minimal normally, then this possibility can be excluded.

01:27:02.970 --> 01:27:07.830
We chose patients who were several years after the stroke in the average.

01:27:07.830 --> 01:27:09.390
Which I think it was six or so.

01:27:09.510 --> 01:27:11.890
Okay. Yes, and so quite a long time.

01:27:12.150 --> 01:27:19.670
But in some sense, it might have started with a discussion on constraint-induced therapy. Yes.

01:27:19.990 --> 01:27:23.250
But what are the constraints you are imposing?

01:27:23.310 --> 01:27:27.050
Because essentially you are mobilizing the system, asking people to do more

01:27:27.050 --> 01:27:28.410
than they would usually do.

01:27:28.630 --> 01:27:34.250
Well, the term constraint-induced has produced a lot of misunderstanding,

01:27:34.250 --> 01:27:39.230
understanding and there's this association of the, of the, uh,

01:27:40.127 --> 01:27:46.307
of torture, doing something bad to already poor patients.

01:27:46.627 --> 01:27:54.007
So we try to promote the term intensive language action therapy,

01:27:54.327 --> 01:27:57.187
which I think might be more appropriate.

01:27:57.847 --> 01:28:02.747
Constraints are nevertheless important. We wouldn't speak about constraints

01:28:02.747 --> 01:28:09.807
because of the negative connotation, but we would, to formulate it more positively,

01:28:10.127 --> 01:28:15.267
I would say we focus the patients on their communicative needs.

01:28:16.407 --> 01:28:24.227
In these patients, as in Ed Taub's motor therapy patients, there's a natural

01:28:24.227 --> 01:28:29.507
tendency of avoiding whatever is complicated, whatever is usually not successful.

01:28:29.867 --> 01:28:38.047
So as a patient, if I realize that certain complex words do not work normally, I just stop trying.

01:28:38.227 --> 01:28:47.427
Yes. And if I cannot do complex sentences, I try to avoid that and just use the words I can do.

01:28:47.487 --> 01:28:50.327
I may end up with telegraphic style.

01:28:50.687 --> 01:28:54.027
And this is what we want to avoid.

01:28:54.207 --> 01:28:57.447
We want to really force them to try their best.

01:28:57.447 --> 01:29:01.667
And in order to that, we constrain or we focus the therapy,

01:29:01.807 --> 01:29:12.207
and we introduce rules in the interaction games so that the communication task

01:29:12.207 --> 01:29:15.587
is difficult enough to challenge them.

01:29:15.667 --> 01:29:18.367
Right, exactly. To challenge, if possible, each patient.

01:29:18.587 --> 01:29:25.627
Right. So we titrate the requirements for each patient in order to have optimal practice.

01:29:25.887 --> 01:29:28.987
Right. And I think that's very important, even though constraint,

01:29:30.116 --> 01:29:36.936
The word constraint has produced many misunderstandings, to say the least. Yeah, exactly.

01:29:38.156 --> 01:29:43.536
But I think it's really marvelous that you have made that step towards the clinic,

01:29:43.716 --> 01:29:46.396
which is, as you know, extremely challenging.

01:29:47.296 --> 01:29:51.516
Certainly, if you talk about rehabilitation of stroke, a lot of people stay

01:29:51.516 --> 01:29:54.756
away from these kinds of, because usually things fail in that domain.

01:29:54.856 --> 01:29:58.596
It's just really hard. And it's really amazing that you pushed that so far and

01:29:58.596 --> 01:30:02.436
also had such an impact. And that, of course, gives you, I think… It was because

01:30:02.436 --> 01:30:08.776
I started this stupid enterprise before I knew how complicated it is. Right, exactly.

01:30:09.076 --> 01:30:14.176
And therefore, I was in that business already, and I was actually positively

01:30:14.176 --> 01:30:16.656
surprised. Right, exactly. That's really impressive.

01:30:16.916 --> 01:30:21.676
But also, this gives you ammunition, I think, in this debate with the more,

01:30:21.736 --> 01:30:26.716
let's say, theory-inclined cockativists, because at least you can show you have real impact.

01:30:26.716 --> 01:30:34.976
So now, to finish up my two questions to which you already have been exposed, so you're prepared.

01:30:37.336 --> 01:30:41.456
So working on this issue of language, semantics, and the brain,

01:30:41.636 --> 01:30:47.436
and also aphasia treatment, so what's the law of Friedemann that we should follow

01:30:47.436 --> 01:30:48.756
in trying to understand the brain?

01:30:48.756 --> 01:30:59.316
Well, I wouldn't search for any Freedmanian law, but if anything,

01:30:59.596 --> 01:31:05.376
I would speak of a Breitenbergian approach to mechanizing cognition.

01:31:05.376 --> 01:31:13.176
To look out for, if possible,

01:31:13.396 --> 01:31:26.856
the simplest possible neural instantiations, neural mechanistic circuits that could underlie...

01:31:28.010 --> 01:31:33.310
Complex cognitive abilities, such as meaning processing. And we actually want

01:31:33.310 --> 01:31:35.750
to draw circuits that do the job,

01:31:35.870 --> 01:31:43.690
and we want to simulate the buildup and the activation of these circuits in.

01:31:44.570 --> 01:31:47.310
Realistic computer models that mimic the brain.

01:31:47.450 --> 01:31:56.170
We want to use these neurocomputational models to predict cortical activation

01:31:56.170 --> 01:32:00.610
patterns, predict and then explain cortical activation patterns.

01:32:00.950 --> 01:32:06.770
And if it doesn't come out in the experiments like this, we want to adjust these

01:32:06.770 --> 01:32:13.810
neural models in order to improve them and to get a better idea about the mechanisms.

01:32:13.930 --> 01:32:20.550
I think that's the pathway to make progress to better understand cognition.

01:32:20.550 --> 01:32:27.030
I think we have seen these box and arrow diagrams now for 30, 40,

01:32:27.210 --> 01:32:36.810
50 years, and I think it's really time now to go in a mechanistic direction

01:32:36.810 --> 01:32:41.010
where we theorize not in terms of abstract mechanisms,

01:32:41.170 --> 01:32:48.470
but in terms of concrete neuronal mechanisms that solve abstract problems. Exactly.

01:32:48.850 --> 01:32:56.050
Very good. No, I like that. So then five years from now, I'm going to go visit you there in Cambridge.

01:32:56.550 --> 01:33:01.610
Not in Cambridge, in Berlin now. Oh, you're in Berlin now. Sorry, yes. My mistake. Yeah.

01:33:02.070 --> 01:33:06.730
And I'm going to confront you with a hypothesis you're going to generate today.

01:33:06.930 --> 01:33:12.570
So what's the one hypothesis you're most passionate about today that you believe

01:33:12.570 --> 01:33:14.930
will be confirmed five years from now?

01:33:20.691 --> 01:33:25.091
Um the hypothesis well they

01:33:25.091 --> 01:33:28.791
are hypotheses related to the language therapy enterprise and

01:33:28.791 --> 01:33:33.271
of course optimizing this further and showing that that that the motor system

01:33:33.271 --> 01:33:40.071
indeed activates more after this therapy and reflects actually the motor system

01:33:40.071 --> 01:33:44.671
activations or strength stronger links between motor and language systems actually

01:33:44.671 --> 01:33:48.391
are the result of successful language action therapy.

01:33:48.691 --> 01:33:53.871
That would be a big hope to make progress in this direction.

01:33:55.131 --> 01:34:01.471
Improving the therapy further has very high priority for me.

01:34:01.951 --> 01:34:06.431
Now, if your question aims more at the theoretical level,

01:34:07.131 --> 01:34:10.611
there is a very important question which I would like to learn something about,

01:34:10.871 --> 01:34:18.931
which links to your question about the difference between conceptual and semantic system.

01:34:19.731 --> 01:34:32.391
And I said that for the time being, we could consider semantics a subset of

01:34:32.391 --> 01:34:37.491
concepts or conceptual space.

01:34:37.771 --> 01:34:40.111
Now, this is not entirely correct.

01:34:40.451 --> 01:34:43.071
Because there's an idea that.

01:34:43.881 --> 01:34:47.701
Linking concepts to language also helps

01:34:47.701 --> 01:34:50.601
conceptual processing and the

01:34:50.601 --> 01:34:53.841
and the the idea is that the different

01:34:53.841 --> 01:34:59.521
differentiation of concepts discrimination of very similar concepts is only

01:34:59.521 --> 01:35:04.941
possible if we link it to a to a symbol system such as language the reason being

01:35:04.941 --> 01:35:11.861
that That if you do neurocomputational models and try to build representations

01:35:11.861 --> 01:35:13.641
for very similar things,

01:35:13.741 --> 01:35:16.201
for a crocodile and for an alligator.

01:35:16.561 --> 01:35:23.881
We may end up with cell assemblies, with neural representations that overlap very much.

01:35:24.221 --> 01:35:28.721
Well, of course, we can use neurocomputational tricks such as Kohonen networks

01:35:28.721 --> 01:35:33.581
where we force separate non-overlapping representations.

01:35:33.581 --> 01:35:40.821
If you don't do this, and we just map the cell assemblies and the overlap of

01:35:40.821 --> 01:35:45.541
these networks mirror the semantic similarity, then we are in a hopeless situation.

01:35:45.801 --> 01:35:50.501
We have 95% overlap of semantic features, and then the two glue together.

01:35:50.641 --> 01:35:52.541
We cannot separate them. Hopeless.

01:35:52.781 --> 01:35:57.481
What we can do is use tricks in

01:35:57.481 --> 01:36:02.721
order to separate the two, to make these representations more dissimilar.

01:36:02.721 --> 01:36:08.781
If we manage to link alligator with one word, namely the word alligator,

01:36:09.061 --> 01:36:15.241
and the other concept, the crocodile concept, with an entirely different word, namely crocodile,

01:36:15.561 --> 01:36:21.321
then this union of the conceptual and the word representation,

01:36:21.321 --> 01:36:26.281
So, the entire semantic network, they would now,

01:36:26.481 --> 01:36:35.781
if the conceptual overlap was 95%, then the overlap of the linguistic conceptual

01:36:35.781 --> 01:36:40.461
semantic network would only be below 50%,

01:36:40.461 --> 01:36:42.481
47.4%.

01:36:42.481 --> 01:36:46.101
So, it would give you some form of, let's say, a top-down decorrelation. Right.

01:36:46.906 --> 01:36:52.206
Indeed, of course there's a problem how to link them up with each other if they

01:36:52.206 --> 01:36:54.806
are already overlapping so much in the first place,

01:36:54.966 --> 01:37:03.006
but assuming that this can be solved with special tricks and inhibitory mechanisms.

01:37:03.006 --> 01:37:11.346
So the specific prediction would be that semantics as language-related representations

01:37:11.346 --> 01:37:15.146
is not only built on top of a conceptual space,

01:37:15.366 --> 01:37:19.866
it in turn recurrently helps you to organize and boost the conceptual space.

01:37:19.946 --> 01:37:24.266
Exactly. So the idea is language helps conceptual discrimination.

01:37:24.666 --> 01:37:28.426
Okay. And in five years, 10 years, 15 years… No, it was five.

01:37:28.646 --> 01:37:33.966
I would like to have clear evidence for that. Great. Well, Friedman Pulvermoor,

01:37:34.066 --> 01:37:35.406
thank you very much for this conversation.

01:37:35.706 --> 01:37:40.046
Thanks so much. This was fantastic. And I hope we can do something in the future

01:37:40.046 --> 01:37:43.626
on language action therapy.

01:37:43.846 --> 01:37:46.806
That would be fantastic. That would be great, yes. Wonderful. Absolutely.

01:37:47.346 --> 01:37:50.066
Thanks so much for the interview. Thank you. It was a great pleasure.

01:37:51.926 --> 01:37:57.406
The CSN Podcast was produced by the Convergent Science Network of Biometrics

01:37:57.406 --> 01:38:04.206
and Biohybrid Systems, A project funded by the European Sevens Research Framework Programme.

01:38:06.320 --> 01:38:32.163
Music.