This is the Convergent Science Network podcast. Leading researchers in the domain
of neuroscience, brain theory and technology are interviewed by Paul Verschoor and Tony Prescott.
This is Paul Verschoor with the Convergent Science Network podcast.
And today I have a conversation with Dana Ballard, who is also a speaker in
our summer school here in Barcelona.
And Dana, in your presentation, which focused very much on vision,
you started out by making a pretty strong point about why, let's say,
the standard notion of feed-forward saliency maps actually don't really do the job in vision.
So what's the problem with sort of the standard notion of a saliency map and
why is it not sufficient?
Wow, Paul. Well, first, let me thank you for inviting me here.
It's a lot of fun, and the establishment is very new and beautiful.
But as for your question, the problem,
people who aren't vision scientists take vision for granted,
and every person has the sensation of vision of being a person on the street
and being in the middle of a very beautiful three-dimensional movie.
So the act of seeing is completely taken for granted.
Whereas, as a scientist, if you start to examine the brain and how vision works,
how the images are gathered and sent to the brain, then you get a completely different picture.
And it's almost an untenable picture from the standpoint of everyday experience.
And so the hardest thing to come to grips with, as I mentioned in my talk,
is the binocular visual system has only good resolution in a tiny,
tiny place in the middle of the gaze vector. So it's one degree.
So if a person holds his arm at arm's length, the width of the thumb is the
place where you have very, very good resolution you can read.
Outside of that, the resolution becomes so poor that you can't even read out that.
And as I mentioned in my lecture, that people found this out by changing the
letters outside of the gaze point.
So people started to think, hmm, if this is true, where should the guys go?
If there's a tiny little sort of pistol of good resolution that you're pointing
around the visual world,
and I should also mention that the visual system is discrete so that we're not
aware of it but every third of a second we change our eyes to a different point
and so if this is true, where should we look?
And so a substantial group of scientists thought that where you would look is saliency.
So the pieces of the image that are most visually complicated,
like if you're wearing a wristwatch or if you're holding some bracelet or just
the eyes of another person, those are places where the image is busy.
And the thought is that, well, those would be candidate places.
And so the original thinking was, well, those are candidate places,
and somehow the brain knows how to pick one of those from moment to moment.
But I would say another camp, a camp of which I'm a member, is the idea that,
well, that might not be true because it might be more agenda-driven.
So this is very, very hard to get used to,
but it could be that vision is really a succession of information gathering
experiences and where you point your eyes to get some quanta of information
and then somehow the brain knows
how to integrate these quanta into some sensation, which you call seeing.
And no one knows exactly how this works.
If anybody figures it out, it's a Nobel Prize for sure.
But at the moment, we're still sort of trying to guess some of the constraints
that would lead us on the right path towards understanding this.
And that's the saliency thing.
So the saliency is really an initial try.
And now the agenda faction thinks they are the second try. Okay,
but now would you believe in some sort of compromise between these two schools of thought?
Is that the solution or you really think it's going to be either one or the other?
Okay, I know I'm in Europe and it's appropriate to compromise.
So one issue where you might be able to compromise is the following,
is that if vision is entirely agenda-driven, then the issue comes up of,
well, when would you ever change your agenda?
So if you're driving along and you're paying attention to the car in front of
you and what would make you pay attention to, say, a small child runs in front
of the car, or what would make you pay attention.
So a compromise would be to take busy places in the image, but then you get
to pick and choose them according to your agenda.
So in front of the car, you would set it up so that you're sensitive to motion.
Any irregular movement, so in the car when you're moving along,
there's lots of motion, but it's very expected motion.
But if there's some unexpected motion And you can quantify this on a computer.
If there's some unexpected, let's call it salient motion, then you would allow
yourself to be interrupted and deal with that.
So the compromise would be that you have a gender-driven saliency where you
can modify what's interesting in the image depending on your…,
So one experiment that you described was, let's say, your litter-gathering experiment, right?
Where you actually tried to also dissect now what this more agenda-driven form
of attention might look like, what its components could be and how they could
possibly work together.
So what was the key idea behind that experiment?
Well, the key idea is that everyone would like to, every scientist or such a
huge group of scientists would like to solve the last big mystery, how does the brain work?
And so how does thinking work? And there's a very sharp fork in the road as
to whether you can think the system can be composed of primitives.
So is there some way to break intelligence down so that you have these little
quantum pieces and then so you can get very complicated behaviors by picking
and choosing pieces like a puzzle?
And one thought is that, well, from what we are learning about the brain.
The amount of pieces in your puzzle has to be small, say less than 10.
And so this is another idea that's very hard to get used to,
but people are starting to think along this line.
You have a lot of puzzle pieces, but for the puzzle you want to make,
you have to pick a small amount in the time.
And so, of course, in the laboratory, we tend to become more and more modest
in the kinds of problems we tackle because of technical difficulties.
And so in the problem you mentioned, we actually have human subjects in a virtual environment.
They're walking down a sidewalk in the city, and there is litter there to pick up.
They have that job, and then there's obstacles to go around,
and then they have to stay on the sidewalk. box. That's three things to do at a time.
And so we ask the question, well, can we keep track of the agenda during these
three simultaneous tasks?
Can we watch it and tell which one the brain's working on? And so we use the eye movements.
We analyze the eye movement traces and report on what task is immediately being worked on there.
And the hope is, of course, that if we can understand how to do three tasks,
then lots and lots of tests are just around the corner, as long as we can pick
them in successions of three or five or ten.
Right. But in this case, the task you try to decompose is always going back
to an eye movement, right?
It's either an eye movement in the service of picking up the litter or an eye
movement in the service of avoiding an obstacle.
Or is that a too limited interpretation? No, not at all.
I mean, you have your five senses, and I always say vision is the most important.
Other people say, no, no, taste is the most important, because if you lose that,
you'll eat something, you'll eat poison and die.
But if you want to get information from a considerable distance and you want
to get very elaborate, rich sources of what's out there in the world, you can't beat vision.
And so the number of people studying vision probably dwarfs the number of people
studying the rest of the senses.
And so, vision is so important that we feel if we understand that one,
the other senses will fall.
The other people are working on the other way. They think, let's go simple first
and try to understand one of the simpler senses, and then we can build up.
But vision has certainly attracted the attention of many, many researchers.
Right. But now in your task, in this decomposition of a task,
so we have the litter gathering, avoiding the obstacles, stay on the path,
right? then you map that into, let's say, three behavioral modules that would
drive the eye movements or the gaze.
But now in some sense, that's like a one-to-one mapping of, let's say,
properties of the virtual world because either it's litter, that's one module,
or an obstacle, another module, or it's the path, it's the other module.
So if you have a mapping, let's say, one-to-one between behavioral modules and
the relevant objects in the world, would you get an explosion of behavioral modules? Yes.
Well, you would, of course, if you did it incorrectly.
And so it's always embarrassing in science how you sweep certain problems under the rug.
And so, as you point out, for our three tasks, there's a unique feature identifying the task.
But in general, you want to categorize the world into objects or,
as an American psychologist said, affordances.
And into things not based on
their sort of native features per se but rather
in are they going to be useful and so um if you're in a dark alley and and there's
people coming towards you and you need some kind of weapon to protect yourself
it might be a piece of wood might be a pipe might be a stone there's there's
but some what are those things that have in common they have some feature that
would help you from this test.
And so there are people in vision looking at ways to convert.
These elementary geometric and mass properties description of objects into some
sort of tool-like feature.
So that problem is being worked on, but not by me.
Well, actually, also in your experiment, you showed that the eye movement trajectory
varied with whether the object had to be avoided or not. Yes.
Right? Because then I think you also showed that in case it was,
let's say, a to-be-avoided object in the task, the eye movements were more at
the edges, so you wouldn't bump into it, while if it was more a viewing-related.
Task, the eye movements were more going towards the center of the object.
So in that sense, I think this also shows how the affordance of that object
varies depending on the task that in turn translates in the
in the eye movement behavior. Is that correct? That's true.
And that sort of gets us into the sort of research that would unify what you
and I are sometimes doing is this idea of embodied cognition.
So it's actually not only the properties of the object, but it's the interaction
of the user of the object and the object itself.
It's sort of how the human is coupled to the world, which would lead us to perhaps
the most shocking topic of all all, is in the eye movement,
the original thinking was that when the eyes looked out in the world,
it captured an image, that somehow that image was copied in the brain in some way.
And so people quickly debunked that because then there would have to be somebody
in the brain looking at that image, and then that never stops.
So the other end of the spectrum that is getting a lot of weight is that when
you look at a certain place,
you actually are not painting an image at all, but you're after this property that we talked about.
After some feature of the spot, you're looking at this, some information you
want, and like you mentioned going around an object, you look at the edge because
you want to rotate around the object.
If you're picking up, you look for the center because you want to go right towards it.
And so this is an idea that really takes a lot of time to get used to the idea
that every third of a second, that when you move your eyes to a point,
you're doing that for some visual test.
If it's reading, you're trying to decode the word you're looking at in the text.
But in the real world, if you're picking up objects or you have some tasks or
you're cooking, driving a car, the tests get very interesting and different in each case.
And one of the tasks is to see if somehow we can categorize all the different
tests that you do. that you do. But now, so in...
This would illustrate this notion of your agenda, if you want,
dictating how you deal with objects in the world visually, how you actually
extract information from these objects, right?
But now, this could be interpreted in two ways. Either you could say like,
well, the object affordance relationship, like the object, what you can do with
it, gets defined in a rather different way. That's almost a different category.
It's not, in some sense, internally an object.
Or you could argue, well, actually, you really detect the same object.
It's still the same sort of obstacle, but the same shape, and in both cases,
you extract that, but it's more like a biasing of how you process that object.
So, which of these two interpretations would you favor? Hmm.
I mean, I think that's a tricky one.
I mean, it really gets, I think we have to go back to agenda-driven, you know.
So when you pick a particular agenda item to work on, when you pick the task,
that task has the properties of the object you're interacting with written in some internal form.
And so you just have to query the world to see if the object you're looking
at, in fact, satisfies those properties. And so I'm trying to think.
I know this is going to come down one side or the other of your question,
but I'm going to let you pick one.
Okay. Well, so my bet would be that you build up a scene.
Which will contain this object, but I think the way you bias it with respect
to your action will vary.
So it's not, on the other hand, you could argue, yes, okay, but if you would
look at examples of, let's say, inattentional blindness, then the objects are
actually out there, but you're not seeing them because it's not related to your agenda.
So in that sense, it looks like it's a difficult problem to solve at this stage,
but you were the one giving the talk and me, so you're the one who has to solve it now for me.
I see. But I think you opened the door to attentional blindness,
and perhaps we should just revisit the idea that in the rather astonishing variety
of experiments done by Dan Simons and others,
that people don't know huge changes in their visual world.
And one of the famous examples was some variant of a person approaching a counter,
like you would if you're checking into a hotel and you're dealing with the clerk.
But, of course, it's an experiment. So the clerk ducks under the table to get
something, supposedly, and there's a person hiding behind there,
and another person pops up.
And if the person who replaces the original person is only vaguely similar,
the person who came to the table would never, never notice the difference.
And so the thinking, of course, from what we call our Cartesian view of the
world, where everything's a picture and there are objects in it that we can sort of label,
like an inverse paint-by-numbers world, world, that just defies explanation
because we should notice the changes.
But in an agenda-driven view of the world, well, we don't because we can be
nice to the clerk and polite, but we don't expect the relationship to go on forever.
And so we don't quote a lot of details about what the person looks like or et cetera, et cetera.
So the change blindness example is actually...
Quintessential example of
a gender-driven vision at work. You just need the features to do the job.
And so I would say that the fact that in our manufactured world,
we give lots of things helpful hints, like putting them in cylinders and writing Coke on them.
But for the most part, we come prepared to just exploit the functional features
of something and getting to the tool use stage.
So then, from that perspective, so in this agenda-driven view of perception,
then your gaze behavior becomes one of, if you want, information-seeking or
uncertainty reduction, the notion you used for that.
So how does it relate exactly to the agenda-driven view?
Well, so let's see, we'd have to try to summarize this question.
Succinctly. And so one central issue in the brain and how the brain uses vision
is, of course, that you have these agenda items, which we think of as programs,
internal programs that neurons are in charge of.
And the question is, how does that all happen?
And why we don't know even the beginnings of it. We have some clues.
And one is that what the brain has to do is be its own programmer.
So if you're working for a company, you can be a programmer and write programs for it.
But if you, the person, have internal programs, how do you code them?
And so the prevailing view is that somehow the neural system has a way of suggesting
programs, and then you would score them as to how effective they are.
And this internal scoring is believed to be this chemical molecule dopamine.
So dopamine is like an internal currency.
I call it in the classroom, I call it the neuro in honor of the euro.
But it's just an internal pay scale for rating different programs.
And the idea of you as a surviving person who wants to spread your genes around,
you would you try to earn the most, you're designed to earn the most neurons.
And the neurons in your head, they can't see out.
They just have to deal with what their internal structures are.
And so they do things like these visual tests and try the visual test.
Everyone comes with its own rate of return and the internal gender-driven programs
that you're running, they are all worth something,
and some part of you is trying to pick the best ones.
Now, what makes me earn neuros exactly?
Oh, you come out of the box. You come out of the box as a neuro… Accumulator.
Neuro accumulator.
It's rather interesting. thing the the are you saying I'm out there really trying
to collect drops of juice and sugar and sex drugs and rock and roll,
Or am I also earning neuros for doing other things?
The answer is both. So your brain evolved in stages. And the part we associate
with being some kind of a computer is the forebrain.
And so that's the last to go. And it's very complicated. It has lots of parts
that we could talk about, but maybe we shouldn't.
But it sits on top of an earlier system that contains the chemical rewards.
And the neurons that are communicating in this computer part that give reward,
they're dopaminergic neurons.
And so you have a vast system of wires that goes through all your modern forebrain.
But the part down in the brainstem that's handing this out is right next to
the part that has your basic rewards, your drives. the four Fs are called for
fight, flee, feed, and reproduction.
And so basically the way your brain works, the forebrain works,
even though you do these elaborate programs like algebra and physics and trading
on the stock market, they all have to communicate somehow with the basic drives.
So the basic drives, somehow your forebrain comes up with these elaborate,
It has to do the elaborate translation of why you're doing what you're doing is worth this reward.
Right, but now if we map that back to gaze behavior, right?
So you mentioned that gaze is driven by, let's say, the need to reduce uncertainty about the world.
So now what makes me earn euros? when I identify the spots that give me the
maximum uncertainty reduction?
Or do I earn neuros for jumping to the spot where I get the juice?
Reward? Well, I mean, it depends how that can fall out in different ways in different contexts.
But the fact that you've come this far means you've, from our perspective that
we share, I think, is that we've totally bought into this program.
So some evidence like Kenji Doya and others are suggesting another one of these
molecules, serotonin, is responsible for risk.
So when you, So if we think back to choosing one of these agenda-driven behaviors,
and we don't really want to know what it is, we just want to know how to characterize
it in the most basic sense. It's like a gamble.
So how much reward and how much risk is it?
And so the brain doesn't want to do these apples and oranges comparisons.
It wants to reduce everything to a common denominator.
So how many neurons? What's the risk? So your internal programs,
the way they run, it's like making a bet.
And so from that perspective, the eye movement system can help by reducing risk.
So if you can reduce your risk, then your bet will become more of a sure bet,
and you're going to get more reward.
And if the internal thing is matched to the external world, then it'll be an
accurate rendition of the value to you.
It's also yesterday you showed a more theoretical experiment where you also
then try to, let's say, extract the neurons that a viewer would be accumulating
with a certain scan path to the world, giving a task.
So does that really make sense?
So what the regularities so if you map an eye movement pattern back to an inference
on what the value would be or could be how consistent is that what you get out of that well,
I mean it.
In the lab, what can we do? So if we think we got it, if we think we got it.
So practically, a couple of things first.
Practically, people have shown that in the brain, the neurons are sensitive to dopamine reward.
So there's miles and miles of evidence where people have recorded from the cells
passing out reward and show they behave in a very consistent fashion.
So the monkeys doing tasks are doing tasks under different circumstances,
and under these circumstances, they should get more or less reward.
And their actual neural recordings are very consistent, showing that they get
more, the cells are firing more furiously, passing out more reward than when
they should. So that says, okay, well, maybe this is on the right track.
But in terms of what we're working on, you know, that's one of the issues is
that, you know, if you eat an apple, your body structure can convert that into calories for you.
So your brain can be told what's the calories to neurons conversion. version.
But if you're writing a scientific paper and you get it accepted in a journal.
What is, how many neurons is that worth?
And that's a delicate problem, is that you handing out the reward and you earning
reward are the same person.
And so the question you raise of how to keep that in calibration is a very delicate and important one.
And in the lab, we're not working with monkeys, we're working with human subjects
in virtual environments.
So we can change the environments in a way that would suggest the reward should
behave in a certain way or the uncertainty should interact with the reward in a certain way.
And then if those results come back consistently, then we think,
oh, we're on the right track.
Right. But now, if you look at this one interpretation of a dopamine system,
going back to Wolfram Schultz and others, would be that dopamine responds to unpredicted reward.
But in some sense, if you take this sort of uncertainty reduction interpretation
of gaze behavior, then I'm gazing to positions where I'm expecting something, right?
So that means it's not an unexpected reward.
If you hit the right spot and it helps to reduce uncertainty,
it's an expected reward. So then dopamine should not fire.
So what did I miss? You didn't miss anything,
but you might have skidded over what we would think of as a fairly minor technical
point, is that how exactly is the information coded in the brain?
And since basically the brain's programs are run over and over and over again,
gain, that, for example, eye movements,
these fast eye movements we talked about, are made at the rate of 150,000 eye gaze points per day.
And so for all the tasks like cooking, making coffee and stuff,
you've done it many, many times.
And so before you run the program, you have a very good estimate of what you should get.
And so it's cheaper for the brain. This is Wolfram Schultz's work,
predominantly, that it's cheaper for the brain to record the difference between
what you thought you'd get and what you'd get.
And there's a lot of evidence for this, that this is the way the brain chooses,
because it's just cheaper to just keep track of when something isn't what you expected.
Either way, you got more than expected or less than expected.
And technically, some of the algorithms that we use work on that principle.
They work on the difference coding principle.
But it's really, I would say, a bit secondary because it's sort of an economical
coding principle rather than the main lesson,
which is the brain has to keep the secondary reward model of things. Right.
But then the other issue, if you take this sort of the law of effect approach
of Thorndike, which it goes back to, which basically means you optimize your rewards.
So you reinforce the things that gives you reward and you try to stamp out,
as he called it, the things that do not give you reward.
Then in your case, you might end up following gaze patterns that are tuned to
a certain task in a certain environment because you optimize reward.
And it might be maladapted to, let's say, changes in the task or to other tasks.
And then you have to first stamp out that whole gaze pattern you have acquired
and reacquire another one from scratch.
So it might lead to inefficiencies if you talk about switching between different
tasks. So how would your model deal with that?
So let's say we go from picking up litter to catching birds or something like this.
Well, listen, we have to tie this back to some things we talked about at the beginning, really.
I actually, this is, Gay's work, I should mention, is the work of Nathan Sprague.
He's a former PhD student, and I actually told him, when he started to think
about this problem, I gave him the Thorndike line that I thought,
just looking at the right place, should you get you the most reward?
And Nathan came back and explained to me no it's the if you want to account for the data then,
reward the reducing the reward weighted uncertainty is the right way to go so
it's not reward per se but it's the reduction in the looking increases the.
Odds of winning your bet in our gender-driven world.
And so he showed, basically, that the original Thorndike idea was unstable when you apply it to gays.
But if you did this product of reward and uncertainty, it was stable and actually
could outperform the alternatives.
Okay. So that would lead to a somewhat different interpretation of,
let's say, the cues that drive reward. It would not be just,
let's say, a flat-out drop of juice or something like this or some image that looks very pleasing.
It is really first the detection of an uncertainty reduction that in itself
will be driving a reward signal.
That would be the interpretation of this, right? Right. Okay.
There's a very famous, perhaps we can add, there's a very famous tape from John
Senders, who was really a pioneer in thinking about information properties of gaze.
And he asked the question, what if you can't use your eyes?
And he built a very special device that would, while driving a car,
it would eliminate his gaze for different amounts of time.
This huge clamshell came down and blocked his vision.
And it was driven by a motor, so he could have it secured for one second,
two seconds, three seconds, four seconds.
And about four or five seconds while driving a car on the highway,
you get into terrible trouble, and you really have a very visceral sense that
it's the uncertainty on where you are that's the driving force.
And I think this was Nathan's kind of insight, too, is that when you have reward
and uncertainty, in the case of gays, it's the uncertainty that's dominating
and that needs to be paid attention to. Right.
So, but then if we would take this literal interpretation of the dopamine signal
as unexpected reward, would you say, look.
This idea of uncertainty reduction as a reward-driving signal,
is it important for you that this gets mapped to dopamine, or you also would
be happy if it would be some other neuromodulatory system that is conveying
that signal to the rest of the brain,
or really specific for you to the dopamine system?
No, I mean, as a modeler, of course, it's a scalar something.
And so if it was something else, that would be fine. But of course,
like, you know, the Nobel Prize has already been given out for the discovery
of dopamine's role in this context.
So you'd be swimming. We really don't know what the alternatives,
what a good alternative is in this place.
No, but that's good. So you really, you would fight and die more for the general
idea of the scalar value. Absolutely.
It doesn't have to be that. But there's a lot of evidence for it.
And all the addictions like nicotine and cocaine are linked to breaking into
the dopamine storehouse.
That's right. But how would, let's say, these addictions be informative on,
let's say, uncertainty reduction?
Right? I mean, sure, that relates to reward. That's clear.
But your specific angle on it is the uncertainty reduction. Well, okay.
But we have to keep in mind where we started here is that we started thinking
about vision and we've started by this very esoteric and high performance gaze
system that can really move the eyes to different parts in the visual world
at speeds up to 700 degrees per second.
So the eye movement system is really remarkable and very different.
And so that at that end of the spectrum with that part of the human machine,
then uncertainty comes into play.
But when you're breaking into a house and taking the high-definition television
so that you can sell it and have more cocaine and more dopamine,
then you're dealing with the… There are uncertainties, but you're really after the reward.
The reward is driving you, and overwhelmingly so.
Yeah, okay. That's clear. So then you also showed experiments where you were
generalizing this way of thinking to driving cars in virtual reality, right?
Yes. So how were these experiments with following another car and trying to
see – so people have a task to follow another car at a certain speed.
This car moves along some highway.
And what you're looking at is then the switching between the car you're following
and the speedometer because you have to stick to a certain speed.
So how has this been informative on this notion of uncertainty reduction? Yeah.
Well, of course, like we talked about John Sender's experiment,
driving the car, and if you don't pay, of course, being a researcher,
the style is to defend certain hypotheses that you're trying to run down.
And so their thought was, well, driving is perhaps a little like walking down
a sidewalk in the following senses.
You have a limited agenda of things that you can do in this multitasking sense.
And what could they be? And in the lab we picked, in the demonstration you're
talking about, we picked something simple or relatively simple,
following a car at a certain speed. So dual task.
And then in our subjects, in a virtual car, so we have a car simulator.
Simulator, the gaze pattern goes back and forth from the car they're following to the speedometer.
But of course, we know in real life that multitasking in driving is critical.
And in the US, there's a terrible problem with teenagers and texting and talking
on the phone during driving.
And so that's a very demanding task to be doing and competes often,
sometimes fatally with a normal driving test.
Even in simple situations like driving on a freeway and things like that,
people will wander into the wrong lane.
And finally, the television is trying to alert people, particularly young people,
that they shouldn't do this.
Because in the U.S., it's very hard to pass a law against anything.
It's a free country, but this is definitely a place where we shouldn't be a free country.
We shouldn't be texting and talking on the phone while driving the car.
But what it points out, though, I think, rather cruelly, is this idea of uncertainty,
because here's where you start multitasking on a phone, or typing something
on an Android keyboard, sorry for Android, any old phone.
Then that just steals your cycles, if you wish, computer cycles, and it steals your gaze.
So your gaze is now staring at this keyboard when it should be looking out at driving.
And so it's multitasking. So you're sneaking some gazes on the road,
hopefully, but in the accidents, you're not doing it enough.
And so you're just spending too much time on the keyboard, not enough time.
But if you think about it abstractly, it's just one of these task things.
So you're texting or you're looking at the road.
Your gaze is going back and forth and you're trying to make a decision about uncertainty.
But since you're a teenager, you don't quite have the numbers right.
And so you're not giving enough weight to looking and looking at what you're doing.
But how rapidly in this driving task, what can you say about,
let's say, the speed or the rate at which this uncertainty increases over time?
Time is that let's say linear with time
or is there some other relationship oh we want to
know but that's a laboratory that's a research question
i mean we're we're trying to think it's exactly
what kinds of manipulations we would do to pin that down if we we would be very
very very satisfied if we could come up with some sort of model for that but
you have it no because it's the probability to see an eye movement from the
car you're tracking to the speedometer or vice versa.
It's like when a person looks at the speedometer, so now I'm not tracking the
car, now uncertainty about the position of the car is building up.
So, and that should, I guess, correlate with now the probability to see a saccade back to the car.
No, no, no, you got it, you got it, Paul. I think that's a good suggestion.
Certainly, we want to do something like that.
The only thing that makes it tricky is we don't know from first principles how
much reward we should give to the individual tasks.
And so there's a confound. We can't quite, if we have a product of reward and uncertainty.
And so we don't, the tricky part is to thinking of experiments that would decompose
those factors. so we could pin down how much of the factor is the reward and
how much is the uncertainty.
So that's the only thing that makes it ticklish, but that's what we wanted to do.
But then you just can make it a game where the instruction varies from just
worry about speed to just worry about the car and all combinations in between with some gradation,
and then you want to plot the probability to see these eye movements from one
to the other, and that would tell you then the uncertainty buildup, I guess.
At this point, I'm thinking we should bring you to Texas.
Where i'm from because clearly clearly you
can solve this problem for us okay good
you got a deal but um so
so but this is great right because now we saw you so
you we're looking at perception as an agenda-driven uh active process um with
respect to and the measures in eye movement but then i could say yeah that's
all really nice dana but i think you're a bit too vision-centric because Because
I'm not flapping my ears around in a similar way, right?
My ears have sort of almost omnidirectional access to the auditory world,
to the sound pressure waves.
So I don't have that problem. So I don't have to have the same kind of selectivity
in now sucking in the information because I don't have something like an auditory phobia, right?
But I have to say….
So, your listeners may not know that you're a relatively young person.
And so, if you were an old person, you would answer that question completely
differently because… So, I lost my ticket to Texas now.
No, no, no. We have old people. Old people can come there too.
Okay, good. I mean young people.
But what I think you remembered is in the auditory system, one nice illustration
is the cocktail party effect.
Whereas if you're in a crowded cocktail party and someone calls out your name,
you immediately recognize it.
And so what you can do with the auditory system, you really have something like the visual problem.
Is that in a situation with lots and lots of background noise that you can tune
in to one particular speaker and one particular conversation.
And so it's not exactly the same as vision, but it shares a lot of the same
abstract features in that in the auditory picture, in the visual world,
you want to pick out a place where the object of interest is.
In the auditory world, you want to pick a part of the auditory spectrum where
the actual voice signal is being generated.
And so at an abstract level, some of the problems are quite similar.
And also, you have some ability, since you have two ears and the auditory signal
is different depending on the location and space it's coming from,
you have some ability to filter the signal from where it's coming from in space.
And so, there are similarities.
And so, in an agenda where there's multiple speakers, that would be like multitasking.
Right. Okay, so that means, so you would say, look, maybe the implementation
might be somewhat different because we're not flapping our ears around.
But in both cases, you have an information bottleneck.
It's like the central processing of information about the world is restricted, right?
So there's a limited bandwidth and you have to pre-filter in some sense what you allow to enter.
And that is done on the basis of this agenda that you're driving.
And you would say this would generalize to also olfaction or somatosensory information? Right.
Yes. I mean, I think I'm not an expert in any of the other senses,
but I think the other ones, certainly the haptic sense of, like, for example, your skin.
Everybody in Texas has had the sensation of ants crawling on them.
And we do have fire ants where they're trained to bite simultaneously.
And a lot of people are allergic and they need to carry an EpiPen with them.
And so when they get bitten, they have to inject themselves right away.
But there you have the same kind of issue. You can think, let's leave a fire
ant out of it and just take a regular ant.
That a regular ant can crawl all over your skin unnoticed.
Or you may be able to sense where it
is in in some part of your skin and so there you have the
same issue of you have space your your your the skin covering your body and
then there's a location on it where something of interest is happening and you
know it's it it becomes some of these things become rather similar olfaction
and taste um maybe we have to give them special um special
exemptions right exactly no but this
is this is very good right because in this would imply
that from this perspective of uncertainty reduction right you can really have
a modality independent view of this process right this is the power of what
you're doing so this is very good absolutely so so we're not stuck with vision
alone no no no no very good so now um so now that you solved the vision problem.
Um then you thought okay now i solve vision uh
let's do motor control right so you jumped into motor control where did why
did that happen well i moved to texas and and um when we moved the lab to texas
um for a lab that we had to recreate it it's quite expensive so the the university of Texas at Austin,
gave us a very nice package that allowed us to recreate the lab in a new location.
But in that move, I made a personal decision that we should start taking more
risks rather than continue exactly on the same line that we were doing.
And so we thought that we would take a look at moving the body,
not just the eyes, but But what if you move the rest of the body?
What would that look like computationally? So what we do is this embodied cognition
field, which is rather strange because you build these computer models of the human.
And you try to respect a lot of the knowledge coming out of neuroscience and psychology.
But within those boundaries, you try to...
Find what the principles are, like you just did this wonderful summary of what
we're about in terms of getting abstract models.
And so we thought we'd try to do that for motor control. And we've started.
It's just all new and different.
We have what's called MOCAP, motion capture system, that allows us to capture
the movement of an entire body and making arbitrarily poses and motions.
And then we can convert that to a skeletal representation.
So we have this elaborate computer software package developed at Stanford, OpenSim.
And that allows you to have what we can loosely characterize as half a human.
So a human has 600 muscles and what they call 300 degrees of freedom. Let's call them joints.
And the software package has just half of that. So 300 possible muscles and
150 possible joints or degrees of freedom, as we call them.
And so we've been working with that to try to come up with abstract characterizations of movements.
And we're making some headway, we think. Okay.
But now, in looking at this motor control issue, you drew a parallel between,
let's say, your standard robot, or humanoid robot, and a cat in this case.
Right. So, what are, let's say, the obvious differences between these two?
Like, what makes a robot so different from a cat?
Well, it's a delicate thing to explain, but it centers around the notion of abstraction.
And the easiest way to start is with Mozart.
So if you take sheet music, everything Mozart ever wrote fits on one CD.
But, of course, he was very prolific and he wrote hundreds of things.
And so if you have the CD for all the music that's not coded in sheet music,
it'll take many, many, many CDs.
And so the thought is, so what's the difference? It's because,
let's think in terms of the piano, the code is the piano has keys,
but then each key plays a complicated note.
And so playing the note per se gets all these high-frequency vibrations going.
And that's expensive to reproduce. produce, whereas the hecto note,
it's very easy to write down the symbol that will do that.
So in the same way, the cat, you mentioned the cat, and also humans,
for that matter, have the spinal cord.
And so when it comes before the brain, inside the spinal cord is what's called
pattern generators that are very loosely analysis. Yes.
Loosely could be put in correspondence with a piano. So basically your spinal
cord is playing the role of the piano and then your brain then just has to have
the sheet music and play the spinal cord.
And so that's the insight where the particular path we're going down and looking into.
Whereas if we take a conventional robot, most conventional robots do not have this concept.
And so basically there's no concept of sheet music. you are actually creating
the notes with very, very high performance silicon computation.
But then, so if you look at this encapsulation, because it's basically what you're saying, right?
So you have a basic functionality encapsulated at your spinal cord level.
Let's say I might have something like these force fields I'm controlling on
my spinal cord, so I can move the effectors in some space.
And then how many piano players are you considering above that?
Is that one? one but for instance
the brainstem might be pushing these keys and then the frontal
areas might be pushing the keys of the brainstem etc right so
how many layers of piano players would you consider well roughly
i would say we'll make one more distinction
before we get there and so if you want a movement that's repetitive that's a
part of all your movements so so one One thing in the spinal cord I should have
one little elaboration that will make things simpler is that rather than have a different,
what's the size of the piano we could be arguing?
And mathematically, there'd be a way of basically playing chords so that you
can have some basic chords.
Keys, frequencies, and then combine them in many, many different ways.
And so you can get a huge spectrum of movements rather cheaply in that regard.
But another problem you face is that when you put on your backpack to go home
as a student, then you've changed where your center of gravity is.
So all the movements that are carefully designed to keep you from falling over.
Won't work unless they get some help.
And so you have a part of your brain that's much older than in the forebrain,
is called the cerebellum.
And so inside the cerebellum, it's been shown that the job of the cerebellum
is just to put these temporary adaptations in there.
And so one big thing you have to solve before we get to the question you raised is load balancing.
So there's something about your body that's changed. In some way,
you're swimming in the ocean, or you've got a backpack on, or something strange
has happened. It has to be adjusted.
That's the cerebellum. So once the cerebellum is putting that capability.
Then now we come to the task that you put on us, and that is that somehow if
you have a movement, like getting the kettle off the stove or whipping up an
omelet, those movements,
some coordinates in the world have to be translated into coordinates in the body.
And so the forebrain is really in charge of that.
And so we don't know how this is done.
Nobody knows how this is done. But the idea, but what might be,
what the job might be is believed to be somehow doing this translation.
So somehow the newer forebrain has to take the task requirements and put them
into some sheet music-like code.
Right. And we know this isn't easy because the forebrain itself is organized
into layers. So you start out with basic ideas and then you put more and more
abstract layers on top of it.
So the code for these movements is not going to be easy to find out.
But at least we have some ideas on how it might be organized. Right.
So the model would be a little bit like I have my piano keys and then my spinal cord.
Then I have, let's say, some sort of background modulator, which is the cerebellum,
that keeps this all a little bit calibrated.
And then you have your piano player in a four-brain structure pushing specific
keys so you can actually walk the stairs, something like this.
That would be the model. Yes.
Yes. But then it's almost like you're an expert musician. So you can somehow
chunk huge pieces of music into one common code that you remember.
So you can kind of economize it.
It's almost like any other kind of athlete where you find huge sections of body
movements you can repeat from
memory without thinking about it because they're sort of coded by rote.
And so the forebrain is sort of the generator of these kinds of more and more advanced codes.
Yeah, but so now we have this multi-stage system.
But if you now go back to the robotics example, there actually there's a very
standard procedure of how to control movement.
Because you would say, okay, look, I want to move the endpoint to some XYZ position in space.
Space and now i just need some
sort of inverse um a model that
now maps that to the forces have to apply to my to my joints
and then i move my endpoint in space that just
however turns out to be a tricky problem right but
so do you see then this division you see between spinal
cord cerebellum forebrain as we just sketched it
as mapping onto that division that engineers
are using in the the control of robots or that's
that's sort of an alter another kind of solution it's another
kind of solution i think it's a huge a huge fork
in the road and i don't know if this would help
but we could think of trigonometry so if we take the function the sine of an
angle we have an angle we want its sine function and so there's two ways in
computers there's two ways to do that one is you could take small divisions
of angle and pre-compute the sine for each one of those so So you can have a table,
an elaborate table. You supply the angle.
I get that angle, and I go to my table, and I look up, what's the sign?
So in that table, it's rather spacious, but it's very fast.
And the other way to do it is I could use a series expansion.
So for sine, we could, what's the sine of theta?
Well, it's theta minus theta squared over factorial two plus theta cubed over
factorial three. I think I'm remembering the formula correctly.
But you can write out a formula for it. And then every time you want to know
the value of an angle, you could just run it through this summation,
which you can roll out to whatever accuracy you want.
So the two, let's summate the two approaches.
One is you have your table that has the fast lookup. And then the other one
is you compute it as needed.
But the computation is much more extensive. One table lookup,
many, many terms to be computed and added up together.
And so you could think of the robotics approach,
one interpretation of the robotics approach is it's very much the series expansion
mode, the computationally intensive mode, which we can go down that road because
the computers are so fast.
We have a lot of cycles, so it's tempting to use them.
Whereas if you can think of the body and the human body as not having the cycles
because the neural hardware is over a million times slower than silicon,
so that the human has to use these stable lookup approaches.
And furthermore, the table lookups are sort of burned in over evolution and also development.
Each human starts out not knowing how to make movements, and then you sort of
rather painstakingly fall on the floor, learn to crawl, learn to raise your
head, et cetera, et cetera.
And so you burn them in over, really over years, you're willing to take the
time to develop the movements you'll need as an adult. So it's like you're trading
off memory versus processing. Exactly.
Exactly. It's exactly right. So in your...
You would think that in this division that you just early made of the brain.
It actually relies on memory to do this, perform this task efficiently,
as opposed to computing this all the time, right?
And these ideas have been around a long time. They're not new.
And Chris Atkinson had them early on with MIT and then Georgia Tech,
but amongst others. but a lot of math has been developed to come up with more compact codes.
So the codes for these tables are becoming cheaper and cheaper so that they're
becoming more possible as being conceptualized as the way the humans might be doing it.
So what's the benchmark you would like to put out there also for the roboticists
to which you would like to also compare your own solutions in this kind of motor control task?
What would be a benchmark that you think is plausible and convincing?
Well, I have a graduate student, Joseph Cooper, and his benchmark is he's making a racquetball player.
And so he has to get his PhD, he'll have a racquetball player that he can play against.
In the physical world? In the virtual world.
In the virtual world. So his virtual, an avatar of him, will play against another avatar.
Right. And may the best avatar win.
Okay. I agree with that.
So Dana, I mean, you're around in this business for a while,
made incredible progress on this understanding of perception, active vision.
What would you see as Dana's law? What's Dana's law that we should adhere to
in understanding the brain?
Well, I think that as an academic, it's hard to have a law because no one obeys you.
That's the nature of your calling. It's an ideal world.
Everybody will obey Dana's law. That's right. Well, I think one thing I do think
that is sorely needed in understanding the brain, we'd all like to understand
the brain, is the idea that an idea that is,
totally essential to silicon computation, that's abstraction.
So we can't even think about computation without thinking of different layers of abstraction.
So you have the operating system that runs your program. You have your program
that's written in a high-level language.
That program gets translated into machine language.
That program gets translated into a microcode that the particular machine architecture
that can understand, that code runs on gates, right?
The gates are composed of layers of silicon, and it keeps going and going.
And so if we didn't have this essential concept of layers of abstraction, we'd be stuck.
And Alan Newell, really, he articulated this most elegantly in his book,
Unified Theories of Cognition.
But I think that's missing. And I think what we really need,
and I think what the math developments,
all the math modeling and machine learning that it's helping us with,
is we really need those kinds of concepts in thinking about how the brain works, because it's obvious,
or at least we think it's very, very necessary for the brain to succeed.
It has to be somehow organized into layers of abstraction.
And in this course thing, so spinal cord, et cetera, et cetera,
et cetera, we can come up with good guesses, but when we get to the forebrain, we're not done.
And I think that that's really the work site, is to go into the forebrain and
try to figure out what are the useful layers of abstraction that the brain is using.
So Dana's law is abstraction is good for you.
That's right. Twice a day. Very good. So then to finish up, five years from
now, probably earlier, but let's say five years from now, I'm going to come
down to Texas and I'm going to remind you of a hypothesis you want to generate today,
which is what hypothesis do you feel most passionate about today that I can
ask you about five years from now and then it will turn out to be verified? Right.
Well, I'll pick one. One of many hypotheses. Right. And will this be like a beer bet?
It's more serious than that. More serious? Like a dinner out?
What are we talking here? That's nothing.
Come on. We're talking about serious bets here. The housing market?
Maybe it's recovered by now.
Now you're talking. Yeah, okay.
Here's one rather specialized hypothesis that's very important to me.
And so we talked about the forebrain. And inside the main memory system of the
forebrain is the cortex.
And the neurons in the cortex, they communicate by sending spikes.
And those spikes are sent at very low rates.
So somewhere 10 spikes per second, 50 spikes per second.
Those are the communication channels of nerve cells in the brain's main memory
system. And one of the most popular current hypotheses is rate coding.
So neurons are trying to send a number to the other neurons they communicate with.
And one of the astonishing things that the brain's memory system has,
each cell can talk faithfully to 10,000 other cells.
And this is a feat that silicon can't come close to. do.
And so the thought is that these other cells, what they want to know is they
want to know what this number is and how they estimate is by counting spikes.
And so I just think that for a variety of reasons, this is untenable because
you just can't communicate fast enough to get to do programs.
And that the what's called rate coding hypothesis is actually a correlate of
different kinds of codes that the neurons actually use.
And so five years from now, this will be generally believed to be true.
And so as a wonderful substitute is that the brain uses some kind of latency code.
So it has each little agenda task has a clock and what's called the gamma frequency
that's somewhere between 30 and 90 Hertz.
And each agenda task gets a frequency, and then the spikes can send a little
analog number by delaying their spike with respect to this clock pulse.
So if you're right on the clock pulse, you're a big number, if you're.
If you're delayed, you're a smaller number. And so the actual spikes are actually numbers.
And if you can do that, then all of a sudden the doors are open for a lot of fast communication.
So if you want a prediction, that's my prediction, that five years from now, a spike will be a number.
Okay, very good. Thank you. So, Dana Ballard, thank you very much for this conversation.
Oh, I was delighted to be here, Paul. You and I have been friends for a long
time, and it was fun to do this, too.
The CSN podcast was produced by the Convergent Science Network of Biometrics
and Biohybrid Systems, a project funded by the European 7th Research Framework Programme.
Music.