This is the Convergent Science Network podcast. Leading researchers in the domain
of neuroscience, brain theory and technology are interviewed by Paul Verschure and Tony Prescott.
So this is Paul Verschure with the Convergent Science Network podcast.
Here we are at BCBT 2018 together with my colleague Tony Prescott.
Hi, Tony. Hello. Good to see you.
And we're here with Luis Fuentemila, who is a great researcher of memory. Right.
So what's your definition of memory?
I think memory, because it's kind of a complex question.
But definitely the way I would perceive memory is the only one of the only functions
that allows us to bring some persistence over time.
So basically it links moment to moment into a continuity. That's the way I see it these days.
Okay. But now the title of your talk was To Shape the Unfalling Experience into a Memory Code.
So the idea of the title was a bit to kind of like point out.
That at the end of the world, or at the end of the day, The way we create a
world is based on the way we make representations of our inputs.
And that transformation, which is like in essence one of the biggest problems
nowadays to face, somehow should be shaped by a representational system and
some mechanisms that should be framed into the memory, into the memory function, let's say.
So if you put this argument into the extreme, I would say that somehow,
right, the way you perceive or the way you see the world or interact with the
world, it should be definitely shaped by a memory system.
So it's kind of like a loop thing.
So now, to start your talk, you also emphasize this whole issue that also a
memory system is also actually about forgetting, right? It's about selectivity.
So how do you see the selectivity of a memory system? What defines its selectivity?
Well, I think this is one of the important puzzles that we are facing in the
memory, because assuming that you're forgetting, it's quite logic,
because we experience that, I think everybody would accept this.
But on the other hand, we are failing to this fallacy of accepting that something
doesn't exist because we cannot observe it.
So at the end, like from a clinical point of view, it makes sense to assume
that anytime you cannot recall things, it's because they are forgotten and that might have a function.
But on the other hand, there's this sort of like a fallacy of assuming that
this doesn't exist. And I think the more you, so I'm not going into the response
directly to what you answered, to what your question was.
But I think this is one of the fundamental problems is this,
like, what do we really forget?
And if we forget anything, because at the end, you can assume that at the end
you create mental schemas, or you can transform into mental configurations that
might also impact into the future. So that's on the one hand,
the problem of forgetting.
But on the other hand, I think that from like a mechanistic point of view,
it makes total sense that somehow most of our information should kind of like
disappear from our, to be able to come directly available.
And then forgetting should be useful for this. So it has to be some sort of
like a pruning process, at least from the availability point of view.
And it kind of raises the question, what is memory for? I mean, why do we have memory?
What's, I mean, what is the practical use for an animal to have memory or for a human?
Well, I guess the one, maybe the way I see it now, at least with the research
I'm currently doing is because it helps you learning more.
It helps you interact like effectively and, uh, and, um, I wouldn't say take
decisions because that enters into the decision making, the decision making world.
Um, and I'm not like spotting into this, but, uh, but somehow the idea that
memory, uh, basically shapes your experience because you assume that that should
be useful in the future for something.
So you make, right? It's kind of like, I guess it's like in the middle of a
desperdictive coding scheme as well, somehow.
I think there's a sort of contrast with our everyday idea of what memory is
for, which is for retrieving things that have happened in the past and reliving them and so on.
Whereas, in fact, maybe why the systems have evolved as they are is to help
us in the here and now to make better choices.
Yeah, exactly. Well, I guess that, yeah, that's kind of like an important question.
So the extent to which memory just can be simplified, if you wish, or kind of like a,
chunk into a kind of a function to take decisions or not, or whether this brings
you out something else in the higher level functions like the self or,
you know, like this sort of like a perseverance of your identity, if you wish.
I think there's a mystery. three, I'm as a psychologist, I am not that into
the idea of like, uh, assuming that memory is just simple function or it can
be simplified as a function to take decisions for the future.
I would like to think that there's something, it brings up something,
something else as well, like the cell for instance.
Which I'm not that sure, maybe we can discuss this possibly,
to what extent the self or the agency can be also simplified.
It exists because it helps us to take decisions for the future, optimal decisions.
How about mental time travel?
Well, I mean, that's kind of like the Endel-Tulbing suggestion from the very
beginning, isn't it? Like the essence of episodic memory and what basically
distinguishes us from animals.
I sympathize with the idea that memory, it's the only way that we can just loop
around time in a virtual world, which is extremely relevant for many of our
interactions with that.
I guess the crucial point is whether we need to be conscious about this constantly,
the autonomic function of memory, or let's say property.
In my view, actually, I think that actually this mental time travel like property
is essential, and I fully believe that it's like a critical point for, at least in humans.
Yeah. So in your own research that you discussed, the starting point were these models of memory,
that had a dual process perspective, where you would say, look,
we have acquisition of memory, two phases, and there's retention and expression of memory.
Acquisition is the more hippocampal process, and long-term retention and also
expression is a more cortical process.
So why does dual process seem most relevant as a starting point for your research?
Well, so basically this kind of like dual system model, it's like an ancient
thing and it's a kind of traditional view from the 90s and I still think it persists heavily.
And I would say that actually that's something that people like is taking it
more and more and more serious or basically because they're proving why it seems
to be a valuable point of view to address many of the problems that are needed to understand for,
basically to understand how the memory is supported by the brain.
So I like the idea of this dualism because memory should be a function that
should deal with the present.
Um you cannot just think about like i don't really like the idea of like thinking
about memory as a an encoding process and a retrieval process as if you were
these two were completely separate um because it's completely pointless in their
life activity we're not just you know like.
Basically canceling out everything from the surrounding and just retrieving properly
or just incorporating information at least
when we are grown because right the memory is always there
and representations are there so at the moment that you are trying to
simulate this is reality then you need a
system that deals with this and uh this dualism
seems to be at least from kind of
like a right the very the principle or the kind of like the essence of the mechanism
seems to be quite strict to understand it that's why i feel comfortable with
that and i think that's why people now that basically are trying to bring uh
this um conceptualization of memory as something that might deal with the present
is comfortable with as well so the moment you bring like computational models,
like you start talking about states,
how states are basically relevant for decision-making, all these sort of questions
that people are bringing up now.
I think these models, that's why these models are so influential again and again.
But now the experimental paradigm that you pursued focuses very much on this
notion of reactivation, right?
So in some sense, We have a simplified model so far in our minds,
which is we have acquisition, retention, expression.
But now you bring in additional components in that process, which is reactivation.
So why do you think reactivation gives you the lever in understanding how memory functions?
I think reactivation is based on the frame of this event these days.
I'd say that it's a direct parallel of this sort of like a metaphor of,
let's say, of a living machine.
So it's one of the instances from a memory point of view and from this idea
that there's these two systems that basically interact heavily at the moment.
On the one hand, there should be an interaction, and this interaction should
be based on a transformation of the inputs.
Should be kind of like, right? I mean, it should be something,
it should be, the two of them should talk into a language that should be memory-based, let's say.
And reactivation helps us, or it helped me at least, to basically formulate
a mechanistic hypothesis that fit quite well with this possibility.
It's one of these mechanisms that really allowed me to think about how we instantly
transform, rapidly transform any experience into a kind of like, let's say, memory.
Memory buffer, very initially, like walking memory, if you wish,
but also at long term, right, after sleep consolidation.
That's why I kind of like to frame this mechanism as a kind of a critical ingredient
for initial transformation of not just perceptual system itself,
but for what we call, let's say, experience memory transformation,
if you wish, something slightly more complex.
And this replay can incorporate anything. I mean, at the moment that you create
a parallel world, even if it's instant, rapid, and interacts with the rest of
the brain, let's say, Then at that point, you can incorporate as many things as you wish, like goals,
right, value, whatever you wish in there. And I think it's relevant because it's quite effective.
I mean, would you say that replay was the mechanism for consolidation or are there other ways?
Well, I guess it's not the only one. I wouldn't believe that that's the only one.
It's one that fits well with the systems-level consolidation model.
But for instance, there's another one, which is quite intriguing,
in my view, from Giulio Tononi, like the synaptic myostasis hypothesis,
which basically states that the brain has this sort of like a,
or has this sort of like a threshold state that basically is lowered during
the night, during REM session, that's what they claim.
And at that time, anything that has not like the strength in connection that
basically surpass this threshold dies and forgetting is based on that.
So at the end, I think it's a combination of many mechanisms,
but replay seems to be critical at least, you know, like to fit into this sort
of like holistic model of how this hippocampus with the brain interacts.
I think in sort of associative models of memory, you need something like replay
because one-shot learning is not practical in most of those models,
you know, neural network models, for example.
Don't learn well off one example they have to repeat the
example and they generally have to interleave it with lots of other examples to
maintain balance so uh and
this brings you to this uh opinion that you that you said that you have to have
this fast learning one-shot learning system and then that trains the slow learning
systems so you see a big difference there between the the way that the fast
learning system will operate uh and how consolidation is happening elsewhere in the brain.
Exactly. I mean, it's like simulating the rehearsal phenomena.
So you train the other system to acquire this knowledge.
I mean, still we assume that replay is a kind of like direct correspondence
with input somehow. Yeah. Something that I'm not that sure yet.
For instance, like we use this sort of like methodology with the classification approach, right?
With the pattern classifiers, multivariate decoding algorithms,
which is basically essentially you assume, right, that input or the elements
or basically neural responses,
neural patterns elicited during experience should be similar to those that basically
are linked to the replay activity or to that.
And that comes because we know the animal model with the face perception of the sequence.
But I mean, right, the animal studies are slightly limited compared to human experience.
And I think we are really like far away still to kind of like make this kind
of bridge this gap, right?
We take this assumption that sequenced pattern completion, all these sort of
elements that involve replay that has been shown in animals,
should correspond to our human experience, or we'll say our learning experience.
And I'm not that sure yet about that. I'm not sure if anybody knows it yet.
But we take this for granted, but I'm not sure.
I guess that for the living machines, these sort of questions are quite relevant,
isn't it? because for some of the projects you guys are working on these days.
Well, I think the… This is the right point.
Are we talking about… Absolutely. Consolidating, interaction, unidirection. Okay.
All right. I mean, I think the process of consolidation can be thought of as
multistage because your consolidation, you're still consolidating episodic memory.
But at some point, where there are commonalities across episodic memories,
you consolidate further into declarative memory
and you lose the sort of maybe or
you it's less important to have those memories
specifically tagged to particular events that have happened so you learn something
that's common across events so um mindy is that distinction useful to you or
do you think everything you're studying is episodic no absolutely not actually
that's one of the parts of for instance like we now we're running one of the experiment with uh.
Uh, in a, let's say in a real world, which we ask people to,
uh, kind of like these, uh, we use these wearable devices and we ask people
to wear this camera for a long time.
Because one of the puzzles is that when you bring people and you test,
let's say episodic, uh, um, episodic, you ask them to recall episodic sequences of their own life.
Um, most of the things are basically forgotten. They are, if you are into a
routine they love, you go to the office, right? and you go home,
you basically, things are so highly overlapping that you may say that most of
your episodic memories are forgotten.
So at the end of the day, especially with, so at the end of the day,
when you test for the classic episodic memory recollection, let's say,
this bivviness completely disappears. People are messing up.
And we think that actually- Or they're using a schema to sort of fill out- And
we need some, I think that actually.
In the system, the way we replay information, the way we consolidate it,
We need to put some more weight into this sort of like mental models that are
slightly more relevant.
Of course, episodic memory, I think in essence, it's extremely important and
defines many aspects and it has a direct link with animals.
But still in our day life, it might not have the impact that we may show.
I'm not sure what this argument is, but it's a fact, right?
I mean, we've been doing that. We ask people to wear these cameras.
We do have like instances.
We can even like record like short clips. We bring them back to home,
to the lab. We ask for their memories.
And they realize that it's their own life, but they cannot recollect the full experience.
They cannot recollect the event. It's like, come on, guys, it's your own life.
I mean, isn't it supposed to be?
Well, it's another skill, isn't it, to actually to be able to convert memories into narrative.
And I think the development, yeah, and the developmental literature shows that
children right to school age are terrible at sort of telling stories about their
lives, but they learn that. As adults, we get better at it.
Some people are better at it. Well, it may illustrate your earlier point about
memories also forgetting.
Exactly. That's our story. Exactly.
So you guys, for me, are moving really quick because you have an experimental
paradigm where you test memory using reactivation and you do it using a conditioning paradigm,
which actually is, again, bringing another possible confounding that we should look at.
Because basically what you do is you have pairs of stimuli or sets of stimuli,
one of which you condition your subject with a sound.
And then you use the sound during sleep to, as you believe, reactivate that memory.
And hopefully, through that, trigger whatever chain of items or memories it
is related to. This is the core paradigm.
And what you see with your healthy subjects is that you get about a 10% improvement
in recollection of associated items. Mm-hmm.
From that stimulus that you trigger with the sound as compared to stimulus that
you don't stimulate, you don't reactivate.
So now we know we could say, well, so you've shown that reactivation during
sleep leads to a slightly improved recollection, 10%. Yeah, yeah.
Imagine how it would be like to have more than 10%, right? We would have like the key of a...
No, look. It's quite subtle there.
Then, okay, so here we go. But now, curiously, does it matter when you play
the sound, how long and what intensity, how often? What's the trick there?
So one of the... No, that's a really important aspect, actually,
of the design that I haven't mentioned in the presentation. I realized that.
I was just halfway through of my life.
I know what happened. No, seriously. I mean, I really wanted to present the last time.
I don't know. I normally am not that bad in time. I mean, it's kind of like
time-scaling thing. Somebody was interrupting. Oh, no, it was funny, absolutely.
Ooh. Really? But, no. Right.
So we are not sure about this kind of like intensity of the sounds. I'm not sure.
I'm not aware of any kind of like study saying that the intensity has to be
in a certain level so that, you know, like the trigger things during sleep. That's all.
We definitely have to repeat the presentation every year, more than once. Repeat a few times.
At least we repeat it like seven times each of the sounds per condition or per association.
But we know from others that sometimes they have to repeat it even more.
So that's an important aspect, you're right. But the critical point is something
we control because we know it from many studies,
is that at least it seems that to induce some sort of like beneficial effects
of this sort of like artificial reactivation, let's say, design.
Design, these sort of memories have to be specifically reactivated during slow-wave
sleep, which is like one of the deep sleep, what they call,
or one of these stages, like it's basically stage three and four,
so it's basically taking place normally in the first half of the sleep.
So sleep has this small architecture, kind of like with five stages,
but there's two big sleep stages that are basically defined by brain activity,
REM sleep, slow-wave sleep.
Slow-wave sleep is the critical one because they claim there are few neurophysiological
signals and responses and mechanisms that seem to be specially dedicated to
promote this sort of interaction between hippocampus and cortical regions.
Isn't that counterintuitive? Wouldn't you have expected that playing it during
REM sleep would work better?
Well, the point is that during REM sleep, it is believed that the cortical activity
is highly active and is completely uncoupled to hippocampus.
So then the model wouldn't work in that way because the model assumes that,
right, this dual model system learning thing basically assumes that memories
are somehow like initially stored in the hippocampus,
probably together with our neocortex.
And during this sort of like consolidation window, this optimal state,
these two regions, or these two kind of like networks.
Let's say, interact heavily, and that needs to be in a moment in which they do it.
And probably, also, it's known that this is a moment where you are fully disconnected,
from the environment, which is basically, you might focus a bit more,
kind of like a tie-up, a tie-up, a bit, you know, like the mess you have in your brain, if you wish.
So it's, uh, yeah, I mean, you say, because the REM sleep involves like a higher,
let's say, more, um, cortical- So that would already point to the fact that as you like a system,
memory in this case is a system property that exists in interaction with hippocampus and in cortex.
Yes, exactly. Well, that's the idea of the model and that, I mean,
there are many papers showing that that makes sense in animal studies with entorhinal
cortex and with, uh, recently with, uh, with auditory areas using similar,
similar sort of like designs.
And in humans, with fMRI, there are a few papers as well. Also with kids,
it was just recently published. Right.
So some of the data you're pointing to are showing replay in hippocampus and
replay subsequently in cortex.
There was that in the sensory cortical area, the visual cortex, was it?
So you interpret the replay in cortex as being about memory consolidation as
well as the replay in hippocampus because the replaying cortex could just be
the playing out of the memory.
So the consolidation could happen in one or both places or in some other place
where you're not measuring.
No, you're right. I mean, this model, the model that basically these two systems
are fully separated and basically start interacting in specific moments with
geoconsolidated functions and then completely uncouple again.
I think it's a bit extremistic, I would say, if you wish. I mean,
it's not totally valid in the context of our interaction with the world.
One of the things we know is that whenever you recall information,
at least from my studies and some animal studies, whenever you're recalling vividly information,
somehow like sensory related areas, maybe not like the primary areas,
but sensory related areas linked to these memories tend to be active.
Active um i mean in essence
it would be like so the full idea is that uh right this kind
of like vividness of your memories should be supported by some
sort of like uh represent representative formats yeah so in that sense assuming
that replay uh involves um representational stages of this information to help
out transforming it it's not uh incongruent i mean when we're thinking Thinking
about what's been consolidated,
presumably we're imagining a very different signal in the hippocampus from the
one that's in the cortex.
The hippocampal one is going to be a very much compressed version of the memory
and then it's going to be reconstructed in the,
particularly the sensory cortex, it's going to be reconstructing something like
the original stimulus or the effect that stimulus would have in cortex. things.
So, I mean, for consolidation to be effective,
I guess it does have to happen in multiple areas of the brain,
or it does have to be the case that this system can reliably reconstruct based on a compressed trace.
Do you think of it in those terms? Yeah, I think it's a really wonderful idea.
You know, kind of like there's somebody mentioning that in this kind of like
event, this idea that, um, uh, right.
Like, um, uh, memory or memory representation, or let's say like a beautiful
representation of the world. It's timeless.
Right. And we give, we give some sort of like a structure, we give some sort
of like a real sense of like a reality whenever you bring it to a singular sensory input,
something, somebody just mentioned
this possibility and I think I kind of like, uh, I think it was Josh,
maybe something, maybe it was, you know.
So how are these sort of like compression phenomena, I think it makes sense
in this framework, so that the memory doesn't really care about time itself,
for instance, it's just, you know, like a compact version.
And then maybe at the moment that you interact with the system that is fully
dedicated to interact with the world, then you have to give this other code.
Is it what you are like? Well, the thing is, when you think it through,
it gets rather complicated because you have of perceptual
learning systems which are encoding signals
sending them to the basis to the history the campus
which is storing them in some way and then
that's being retrieved and replayed and has to be decoded that decoding so that
that encoding is being learned and can change over time so anything you've stored
in the campus could become outdated and couldn't may not no longer match the
decoding apparatus. So everything has to be kept in balance.
And so all this sort of, in computer terms, offline learning that's going on
during sleep may be about rebalancing all these different systems so that they're all in sync.
Yeah. Yeah. Well, that's good to know. Louis, your second experiment you discussed
also speaks to that, right?
Because there you looked at the competition between memories where you had two
kinds of associations, A and B, and B and C.
You looked at how then triggering B would affect the reactivation-dependent recollection of A and C.
But then what you looked at also is the time delay in that process.
If you saw A, B three hours ago, and now I'm going to do a reactivation experiment.
Does this time delay influence the memory? So is there a competition between memories?
So I hope I summarized the experiment correctly because the point there was
that you did see a competition between memories, as you call it. Right? That's correct.
Correct. But do you see this as an active competition or passive?
That means, let's say, since I saw something a while albacta might be as a decay
of a trace, a non-specific decay. It's not regulated by anything.
Alternatively, it might be an actively regulated competitive process.
It really compares the activity levels and when it takes all, whatever.
So how do you see this competitive process play out, given the data that you have?
I think it's a really important point and that we couldn't disentangle.
But we did observe something quite a realm that might speak about the possibility
that there's something active.
Although it contradicts a bit that the active phenomenon or active process could
happen while you sleep, right?
So in one of the points of the experiment, it was just a very fast slide at the very end.
So we also studied how are the neural signals or neural oscillatory activity
elicited by the sounds of the two conditions while people were sleeping.
So people were completely unaware of the environment. They were fully,
we could control for this, like slow-wave sleep were on track.
And then we studied which were the neural responses when you gave the sound,
when you presented the sound,
and the condition which associated memories were boosted afterwards,
and under the condition in which associated memories were forgotten,
or kind of like forgetting was promoted.
So it was exactly the same circumstance, exactly the same one.
People were basically sleeping the same sort of sounds because they're counterbalanced,
but the neural signals were completely different.
So it seemed that actually the brain treated this input completely different
as a function of what it has to be done.
That was the interpretation we did because we had the Big Hero Effect.
And that, of course, we could not attribute that as if it was an active or a
passive. But it seems to me that.
Somehow the brain entailed the memory reactivation phenomena,
but when it detected that it competed somehow because of where we just claimed,
we just hypothesized that it was because of the strength of this association
could be potentially complicated to deal with in the future,
then this activates also this other mechanism,
this extra mechanism that we couldn't see in one condition.
So like to, to, to, to kind of like summarize this point, we observed that under the same conditions,
um, experimental conditions with the same sort of like stimuli and during sleep
in one of the, the, the, the neural responses were slightly different and,
uh, one component just appeared in one of the conditions, not the other one.
So I think it seems to me that actually, um, these, uh, um, organizational principles
are active, um, I'd say. And you would place them in hippocampus,
in cortex, or in the redirection?
And what's the mechanism behind that? How do you envision that?
Well, I guess from a psychology point of view, the easiest way to see this is
under the framework of this sort of induced forgetting phenomena, for instance.
So there are these studies from Michael Anderson and others who basically state
that whenever you inhibit yourself to retrieve information,
right, actively, right, you stop your thinking about this, the consequence of
this at the long term is that suddenly or eventually you might not recall this information anymore.
This is like an effect that has been like framing to specific experimental design,
and we know it depends on this, but somehow the possibility that this may happen
also speaks of this idea that this active forgetting may take place in the brain
and uh it helped us out understand uh this phenomenon at least in the context of this study,
right um of course i mean i guess that like you know like um making the parallel
of our experience where people were sleeping with.
With induced forgetting, where people are putting an effort to forget this,
it's a completely different story. But still, maybe the mechanism is similar.
But that then also links to this idea you discussed in your third experiment
on how to avoid memory clutter.
What memory clutter would actually mean, that you actively also forget or suppress.
Sure. I think actually that the way we understand or the way we see forgetting
is extremely essential.
To um so then the operation of forgetting allows you to understand,
right that the memories are organized some way
otherwise I mean thinking about like you know like the things that are associated
considering that most of our experiences especially when we are like experts
in learning and we acquire so much knowledge the links that we can make with
things are so high and so at any point that we need some sort of like principles that govern this this,
problem and forgetting it's extremely useful isn't it One of the principles
that you pointed to was what you called the non-monotonic plasticity hypothesis.
And that suggested that the sort of generalization that could happen occurred
to similar stimuli, but then there was a dip.
And so what occurred to me thinking about that was one of the challenges you
have in episodic memory is to distinguish memories that are similar in some
way, but that you need to keep separate.
Sort of what we would call pattern separation in neural networks.
And that this non-monotonic plasticity is maybe about keeping these memories
separate that you need to remember what was different about coming to the lab
today compared to coming to the lab yesterday,
even though much of the context is the same.
Is that a useful concept for you, thinking about pattern separation as a key
thing here, and where also you might want to have active mechanisms for pushing patterns apart.
Yes, I think this is like this kind of duality, kind of like the pattern separation,
pattern completion phenomenon.
It's an essential mechanism to make us understand many aspects,
like how things that happen in the same context can still be represented separately.
I mean, it's difficult because, as you say, it's very context-dependent. Yes, absolutely.
So some things you want to see them as the same for uh depending on
the question you might want to see it as the same effect there's no
monotonic uh plasticity hypothesis that you mentioned
from newman and norman to me seems to say more about active forgetting because
you're speaking of a threshold plasticity if you fall below the threshold then
then you you get the question right you really you dissolve those connections
so this would not necessarily get you better separation,
more pattern sharpening, removing the.
The low frequency components of it, that you sharpen and you want it.
The initial idea of this model, when they basically proposed it,
was to try to explain the phenomena of active forgetting in the sense of their experiments.
I think actually the pattern separation and pattern completion phenomena,
which is a complex thing, and especially to be tested in humans,
is still like many people that might complain about the possibility to quantify
this in humans, even though there are wonderful experiments around the world
on this topic with humans.
But even though there's kind of like a concern, there's also something that
should be incorporated in these models, which is like, because the essence here
is what is the elements of experience,
what is the contextual elements of experience, because the two are basically
playing an essential role in pattern separation, pattern completion,
the way we conceptualize this.
But the way we understand context many, many times involves like our physical surroundings, right?
Or you can even like jump it up to a mental schema, if you wish.
Like, I don't know how the hierarchy can go up.
But there's this sort of like idea that there's this kind of like temporal context as well, right?
And in fact, it exists, this model I call like temporal context modeling from Aikohana.
And I think this temporal context idea is also fundamental and has not been
treated well it's been treated like kind of fairly but has not been at least
to my understanding has not been like incorporated into pattern completion pattern
separation phenomena the sense that,
we can tease apart phenomena, things that happen in the same context in different days.
That's one of the puzzles. But maybe the temporal context might explain quite
easily because the time domain of these experiences are that simple.
And the way we code for time, which is another important aspect,
might help us understand it.
Wait, just from my understanding, earlier you were saying the memory system
you study is agnostic about time.
It just time warps everything to make it really very compact.
Well, that was one of the ideas that one of the speakers suggested.
And I thought that was wonderful, but I already came up with some of the findings
that people are finding these days.
The existence of time cells, what they call them. So basically,
cells in hippocampus, which basically seem to be coving like temporal gaps between intervals.
I mean, I kind of like the idea that time should be represented differently
in the brain and the memory system kind of like does not rely on the way we understand time.
But still, right? It seems... If you want to do mental time jowl,
you better do it faster than real time.
It won't help you very much. This whole separation between the time warped and
compressed to represent the more event and order.
In which it is separately again, let's say, inflated time-wise to bring it back
to behavioral time, okay, that would make sense.
But I think as a memory system, you want to compress. You don't want to forget.
You want to get real information.
So maybe you don't want to have, like say, the real-time interval mixed in with the event sequence.
Yeah. In fact, when you explore autobiographical memory, abilities of people
in these experiments I've been in, and discussions with other experts or experts
on autobiographical memory field,
you just realize that the first thing that people forget is this kind of like
temporal context information, right?
One of the things, one of the properties of experience that seems to be disappearing
quite rapidly compared to others, like salience, novelty, spatial context seems
to be quite, whenever you can recall information, seems to be quite preserved.
But the temporal information, which experience was before, when was exactly, was like,
I mean, you have the sort of sense sense that it was not yesterday and it
was whether it was yesterday it was one week ago but you
are right you're missing uh it seems that it's something that
disappears quite rapidly but that's uh right that's uh that's i like that and
i agree with you without so too um so before we really move to the mag uh work
that you did on on memory um so we started out with a dual process model, right?
It's like an acquisition and retention model, the hippocampus cortex,
right? It's not necessarily saying they work together.
It's saying, first hippocampus, then the cortex.
So, given the experiments we've looked at so far, what should be our conclusion
to how cortex and hippocampus work together to realize memory?
Let me try to get the point.
So maybe one way to think about these two systems in the context of Visiband
is that the hippocampus is essential, is the very first one to generate a virtual world.
And without that, it would be completely impossible to kind of like,
no, that's what I would think, actually.
Where is the item information now?
Which item? Sorry. Well, also your task. I have items.
Oh, okay, okay. Yes. So we have an episode, and it's actually a number of items
tied together, right? So you see that also as a hippocampal process?
I think it's definitely in both, as if it was like trying to resolve the binding problem.
Whether it's stored there or it requires the neocortex for that.
Okay. So would you say they worked, cortex and hippocampus worked together from
beginning till the end of the existence of this membrane?
Yeah, yeah. Okay. So this whole idea of sort of traditional models,
like hippocampus acquires and cortex retains, that you don't support now, given the data you have?
I wouldn't say so, right? That's fine.
I wouldn't say so at all, actually. I think it's a slightly more,
the interaction between the two are more relevant that we can think about.
And then you would put, what would be the relative contributions of the two?
If you would have to give them a functional label, what would the hippocampus
contribute, what does the cortex contribute?
I think actually that representation itself, like the way we understand,
similar to what Tony was mentioning.
So the idea that the representation of like something that resembles truly related
to the inputs we had must be stored somewhere in your codex.
And the hippocampus should be free to do other sort of like things.
Could work for instance, like as an index, as a pointer, something that allows
to kind of like bind, distribute the information we need, because we need to
recall for anything, we need to give agency, we need to give value,
all these sort of things.
It also may, we would, But the hippocampus might also be an optimal place to do comparisons,
rapid comparison of what you expect priors with current inputs,
and basically to drive signals to update models, for instance.
So this sort of like computational issue should be driven by the hippocampus.
That's what I think. While the representation itself, the way we understand
representation in the way might might be better stored in the cortex.
That's the way I understand that. Okay. So that's why at the end,
we always see the two of them connected because they always work together.
Okay. So now we have an idea of the system, right? And so now we can complexify,
right? Because then you start to look at MEG.
And then what you observed is that if you just look at the frequency distributions
or the power spectrum of your MEG data.
That better performance in these recall tests tests, was correlated with a much
more pronounced and coordinated response in theta, let's say somewhere around
8 Hz or so, as compared to an impoverished recall,
where you saw a much more distributed distribution across the power spectrum
of energy in the different frequencies.
So is that a significant option? This is informative for us now,
if we start to think about this hippocampal particle system.
So now suddenly, Tata pops out in a very discreet time window,
well, two seconds, I think, after someone's presentation.
So what's that telling us now about this memory system? How is that working?
So I think actually the relevance of this finding is that it basically expresses
these, or it puts some evidence in humans, which was the point at that time.
It put this evidence in humans by showing that there's some sort of like principle
of organizational information.
If you understand information representation as if it could be captured by a
pattern, because if I, which is something that could, right, be endless discussed.
But assuming that this is like something that we can argue, you.
It basically provides a mechanistic model that basically confirms the idea that
we need some sort of organization during maintenance.
Otherwise, information wouldn't be preserved in a bound manner, if you wish.
Okay, but does that bring you to this notion of an enlistment of a Tertigamma code from memory?
That was the point from the very beginning. that was the point so the idea is
like conceptually speaking we didn't advance so there's no conceptual advancement
compared we just so that it was
a much more methodological tool the force let's say because at that time,
this model of this sort of like representation or this sort of like mechanism
that are tightly related to representations
were not couldn't be tested in humans at least as far as we knew,
and I think the data was was quite nice but completely in line with the model, at least at that time.
I think it's relevant because it also, like, breaches the gap with,
like, mechanistic models that were kind of, like, shown in animals in many aspects.
And it confers some relevance to the idea that this kind of,
like, a face-coding mechanism that seems to be quite that relevant in many memory functions. Right.
So now, for instance, like, so that paper came out, like, 2011,
I think. It's kind of like an old one now these days.
And I know from some of the work with it, afterwards, because at that time we
were one of the first doing these sort of things, but now more people are working
on this. We've discussed that.
Maybe not linking it with theta, but somehow it seems that this sort of dual
approach between oscillations and this sort of possibility to search for representation,
which is another question, right?
So this kind of tool, or this sort of combination of tools.
So the universe looks like very consistent, right?
So we have an idea about reconsolidation or reactivation, how we can use that for consolidation.
We have an idea how hippocampus and cortex work together. We also see signatures
of dynamics that look comfortably close to data gamma coding.
So things look consistent, right, and manageable.
But then you start doing these decoding experiments. with your MEG data,
where you looked at the reoccurrence of frequency patterns in your MEG signal, right?
So you were measuring your MEG over many leads, I don't know how many hundreds
you had. No doubt. Plenty.
Then from there you extracted, if you will, templates of frequency responses,
and then you just used the classifier to say, well, if I'm acquiring the memory.
Can I see any kind of echoing of that frequency pattern in a memory interval
as I'm waiting to perform a recall test? This was the experiment you performed.
And then what you showed to us was surprisingly complicated.
And also surprisingly complicated because this is MEG, so you sit at the outside of the skull.
So now we measure a very average signal across all brain structures,
not only hippocampus anymore, right? The whole brain is important.
So, on the one hand, it's amazing because you could really recover a pattern.
The patterns were replayed in the waiting period, right?
Before the recall test was applied. So yes, you saw the dynamic pattern that
the brain formed to store the memory is retained in the waiting period. Okay, great.
But on the On the other hand, we would also say, yeah, and that pattern is occurring everywhere.
It's across the whole of Clifton because, as far as the data,
it was not very restricted to, let's say, the temporal lobes, correct me if I'm wrong.
So is that not now really some rocking the boat in a rather dramatic way?
We were doing so well. We looked so neatly organized, and now we have this MEG
classification result, which It seems to throw everything again in disarray.
So what do we do with that?
That's actually, at that time, that was one of the things that we discussed heavily in the group.
So we assume that there's some representation. We assume that there should be
some coding mechanisms for the representation.
But other than that, we don't know anything. Or at least not that we can make
any strong hypothesis about.
One would say, for instance, It's like, well, because it's an offline representation,
that should not involve sensory systems, for instance.
But that's not the case. I mean, people have shown that in animals and humans.
Right, okay, let's forget all that. So then let's think, actually,
this information should be kind of represented in the higher visual areas,
for instance, which it's true, but it might still affect other areas,
associative areas, or frontal regions, for instance.
So at that time, to be honest, we didn't know how to deal with this,
and we still see it complicated.
I wouldn't say we didn't resolve this.
Actually, we kind of discussed this with one of the reviewers and we said,
yeah, I mean, that's the way it is. That's the way we saw it.
And we think that it's extremely complex.
I mean, at least in that paper, we used at least more than 50 frequencies.
We included more than 50 frequencies. From delta to gamma at all?
No, we didn't include theta, up and that uh and values our coupling measure
um and so we increased from like i think it was 14 hertz up to like 80 or something
like these other like you know like the highest ones never um,
But at the end of the day, unless you use some sort of clustering algorithm
that allows you to slightly summarize into MEG in a statistical way,
you always see so much noise in the data.
We didn't implement any sort of clustering approach because we didn't know what to observe.
And at that time, we didn't know how to deal with the decoding algorithm that well.
I mean, we just jumped into using this sort of deep net thing or hidden layers,
which was super complex.
That was another problem of the project.
Because we used the hidden layer algorithm, we couldn't be fully certain about
which were the patterns that were basically accounting for the reactivation.
So at the end, we kind of like inferred that because we show some of these patterns
in the supernatant material.
But we never wanted to make any argument because methodological reasons didn't
allow us to do it. And on the other hand, we didn't have any sort of hypothesis.
So yeah, I mean, I guess it speaks again about the representational problem,
isn't it? But in your mind, was it the end of Tathagamma code or...
No, no, because we never, we never, we couldn't ever like talk about sequential patterns.
Okay. And I think that's a critical aspect to talk about, right?
Yeah, but I, you know, a device such as the brain, like a neural network,
doesn't necessarily need to treat space and time differently.
You know, you can convert one into the other and vice versa.
So, you know, that's, and people do, if you're compressing a movie,
time's just another dimension along which you're compressing the data alongside
space, you know, so you're looking for patterns that are temporal and spatial
within the image and temporal across frames.
And so you'd think the hippocampus is going to do something similarly clever.
It's just going to look for ways of compressing information across all these dimensions.
I don't think time is going to be particularly privileged there.
And so you could imagine a spatial code for time, which is what the time cells in CA3 may indicate.
Or you could imagine these phase precession patterns. patterns,
but I think this is something a little bit naïve to assume that because there's
a temporal sequence coming in, it's got to have a temporal storage.
Well, there's order information you might want to conserve. Yeah,
but you can do that spatially.
You can map it into a spatial pattern, which has order.
But then how do you read it out then?
It has to be in the way it's encoded. There has to be some
knowledge about how to read it out you know you so you
have a a cell which has a timestamp but
it's a you know you or you have you know depending
on where you are in the in that bit of of of ca3 you're encoding something about
time in the sequence you know these these these things are totally possible
you know the the the space encodes time look at the physiology uh off of sweeps
and and short wave with ripples and phase precession.
But there you go, that's a way of decoding space, is that you have a ripple,
and so the ripple gives you your temporal sequence.
So the encoding, I mean, yeah, I mean, I think we're not disagreeing fundamentally.
You have both space and time as mechanisms for storing and retrieving information.
But there isn't any necessary given that temporal information has to be stored
in a way that's more time.
Uncomplete, we completely win you on that. I only threw time away,
halfway in this discussion.
So you're not necessarily combining time with your item and context information.
There's no need to do that. Your time worked the whole thing.
That was my point also earlier.
You don't want to mentally time travel in real time. It doesn't buy you anything. Yeah, exactly.
So that's why you also want to segregate from your time.
Yeah, yeah. To manipulate it, if you will. And that means you don't have to
privilege time with having a special storage mechanism.
But that's something else as saying the temporal dynamics of this circuit is
encoding relationships within items.
Yeah, yeah. You have to unpack it, and that has to involve time.
One reason why you want to kick out time.
Yeah, yeah. So you can exploit the temporal code. Yeah, yeah. That's ambiguous.
Yeah, yeah, yeah. So, look, Luis, now we dragged you along with this speculation,
and we see that you still want to believe in the Tathagatma code which is good,
this is a great idea but so you're as a psychologist you're in this,
neurophysiology and also if you want the neuropathology of memory which is a
word that I've left to patients right so if people would like to follow in your
footsteps in this this complex domain what would be louis law work a lot.
Work a lot work a lot yes no yeah work around yes and uh be surrounded by uh
great people i think this is like the element actually um so i did my phd ambassador
but then i moved to london,
And, um, of course I learned like, uh, this is like, again, like,
uh, different ways of learning, right?
So the two of them gave me completely different stories. But,
uh, so when I was in London, um, I had this opportunity to be surrounded by
like people who are amazing and not just because of the, you know, like the sort of papers.
So in Barcelona they were not so amazing. No, no, no, no, no.
It was a completely different.
So in Barcelona what I learned because I had this sort of opportunity to do the PhD with myself.
I'm sorry. We're kind of, kind of right. And then you learn that walking is
an important aspect. Well, we know how science goes, right? That's one of these.
So you have to dedicate a lot of time. But you're right. I mean,
the academic culture in London, yeah, fantastic.
And then I just realized... And Sheffield. And Sheffield, I know. UK generally.
UK generally. Well, it didn't last long, did it? I'm not saying London itself.
I'm just saying like this sort
of like a drive that gives you being surrounded by stimulating people.
And I think actually this is like a critical aspect. When I talk to,
now I have my students quite, I mean, I came back a few years ago,
and I really have to express these ideas to them when they don't want to go
out and do a postdoc for instance.
I say, look, it's not just, you know, like living your place is that it's an
essential ingredient of science.
You need to not just, you know, like survive in another world.
You have to realize that this is something real or partially real.
And, and, and I would, so basically, yeah, work a lot and basically.
Make friends. Put yourself out there.
But the second thing is, there's something about the relationship between Tony
and myself that you're not aware of yet.
Tony likes traveling, right? And I have to sponsor that.
So always after a podcast, four years later, he visits the lab of the person we interviewed.
Oh, great. To check when their prediction they made during the podcast is confirmed or not.
And since in this case, it's just a tram ticket, we can also include your lab
now. Well, you can come too.
I might leave the lab.
So what's the prediction that you really would like to stick your neck out for today?
For my lab? Or in general? For my work. Yeah, because it turns out to your lab
to check whether you go fund it or not.
I kind of have this sort of feeling that the sort of line of research and the
research that I'm currently doing, which is it's going to incorporate more and
more, I'd say on the one hand, risky projects,
because that's something that I actually, I believe that this system in Spain
allows you to do it, and on the one hand, on the other hand, bring technology,
more technology, other than, um, you know, like classic imaging studies,
uh, something slightly more creative.
I mean, that's why we started using like wearable systems, blah,
blah, blah, all these sort of things. We are not the only ones, of course.
And something I'm not sure about, but I think that actually the system is pushing us to do it.
It's something more applicable, something more into, I wouldn't say industry,
maybe like for patients, but something like to fully dedicate it to kind of like to solve problems.
So all these sort of ideas are already in the, the seeds are there because all
the projects have this sort of elements now these days.
But I think that these three will be, that would be my wish.
Yeah, but no, that's not the prediction I was expecting. Okay.
If I'm going to be alive. I made a different prediction on your prediction.
I'm going to say, I'm going to prove that all recall depends on gamma range
responses coupled. Theoretical ones.
I see. But only above 80 hertz.
That's it. Science is hypothesis driven. That's what I tell us all the time.
So,
in four years, How many years? Except for... Qualified. I want two, that's fine.
So I think, actually, I need to think a bit more.
It's the kind of thing you have printed on a T-shirt, right?
So actually.
So what I think is actually we will be able to explain many fundamental mechanisms,
many of them would be applied to real-life situations.
And that's something Ampetish will do it in two or four years,
like pattern completion, pattern separation.
But not just for lab-based experiments, something completely naturalistic.
Ampetish will be able to do it. Will we be able to read out and replay people's
episodic memories through fMRI? For instance.
No, my hippocampus always lies. But now, for instance, what we can do It's one
of the experiences we wanted to have in our company.
Now, it's already like the data that we have in collaboration with people that
have already worked with FMRI, which is fantastic, but at the same time,
the EEG gives you the possibility to do it in a, let's say, the fine-grained thing.
We can, if you watch a movie, like standard movie, I don't know if you know the papers.
Yeah, I know these. It's a bit grainy.
We did it with the EEG, and it worked fantastically well. I truly believe in this.
And it's amazing. I mean, you can't simply like disentangle and then look for
units. What's resolution?
You're gonna say like, oh, no, no. Okay, the subject is looking at a human being,
or you're gonna say the subject is looking at,
a spanish no no no sorry so i was not saying like
that you're like decoding this sort of information i'm saying
like when you watch a movie i'm saying that from the
signal right you're watching a movie yeah watching like our
interview here and uh right right by didn't ask anything and just by your brain
signal we can now know which sort of chunks of information are going to be forgotten
in the later recall okay okay okay right okay so you can tell me afterwards,
which information the movie didn't care about?
Maybe. That's maybe not the kind of application I would spend money on.
But, no, seriously, because I'm...
Advertising. But that's one small step.
A small step for Louise, a big step for neuromarketing. That's a real ambition for four years.
Well, you see, Tony, you better be prepared. Luis Fontanilla,
thank you very much for this conversation. Thank you very much for the interview. Fantastic.
The CSN podcast was produced by the Convergent Science Network of Biometrics
and Biohybrid Systems, a project funded by the European Sevens Research Framework Program.
For more interviews, recorded lectures, or upcoming conferences in the field
of biometrics and biohybrid systems, go to csnnetwork.eu.
Music.
And thank you for listening.