This is the Convergent Science Network podcast. Leading researchers in the domain
of neuroscience, brain theory and technology are interviewed by Paul Vershoor and Tony Prescott.
Okay, so this is Paul Vershoor with the BCBT Summer School and the CSN podcast
here with Joe Ayers, one of our speakers.
And Joe, you started out telling us about a recent report of DARPA that was
telling us that algorithmic control actually doesn't really work very well.
So that sounds a bit surprising because most of the world is running on different
algorithms that are implemented in various ways on digital hardware. So what's the problem?
Well, the fundamental issue is that if you try to control a robot algorithmically,
you have to basically anticipate every possible situation it's going to be in
and program an explicit escape strategy for every situation.
If you can imagine what a lobster does in the bottom of the ocean,
I'd be impressed, or an octopus, etc.
And as a result, that's kind of a futile exercise.
And the robots end up
getting tested on very constrained environments and if
you remember the darpa challenge in the first year the
robots got a couple of hundred yards and that was it and then the second year
they made it to the finish well they changed the rules and so it was possible
to do that you know right and uh so but animals are not in a position to anticipate
what the world is going to be like in general, unless they're very territorial.
So any animal that migrates or goes into new places is going to have to have
an adaptive strategy that enables them to do that.
Now, if you watch most animals, when they get stuck, what they do is they wiggle and squirm.
And what they're doing is they're clearly exploring their full parameter space.
And we think that what they're doing is they're increasing the level of chaos
in the networks that would generate things like navigation and locomotion.
And the chaotic variations on the locomotion and navigation are the wiggling and squirming.
Okay, but now we've made quite some steps forward. This is really a fast forward, right?
Yeah, yeah. So now we have to sort of fast rewind and then revisit some of these
issues because in some sense you're saying, well, these algorithmic approaches
might be nice in a controllable world, but as soon as you start to think about
real world behavior as an animal behavior,
we face problems. Unpredictable environments.
Right. So this is the sticking point there, right? So then if we talk about
unpredictability in the world and the adaptation that sort of evolution has
generated to this kind of unpredictability,
what are the key tricks there?
So you say chaos, but what do you really mean? Well, I think another approach
is critter camps, where you put a camera on the back of the animal and see what
the world looks like for them.
Okay, what do you see? And that's going to give you a little more insight.
We do that on lobsters all the time. And what we're trying to establish is what
sense organs they're contacting the environment with and the patterns of contact
that exist so we can basically tune up our sensors,
which are things like antennae and bump sensors on claws and things like that.
So what kind of sensors do we have on a lobster? Because this is one of your
favorite preparations.
Yeah, yeah. Yeah. Well, most people have focused on joint receptors and joint
receptors tell the animal something about the angle of the joints and the action of the muscles.
But when you really see what happens, for example, on a claw of a lobster,
when it bumps into something, it moves all the joints at once.
So it's getting some signal from all the things, which I think is just a bump It's a lump response.
But if it comes from the side as opposed from the front, it's going to have
a slight variation, which is going to give the animal some subtle information
about where this insult is coming from. But for that to work, we need some…,
corollary discharge for an efference copy, because the animal must be knowing what it wants to do.
Then it detects some perturbation on its effectors that make these joints sort
of bent in different ways than it expects.
Yeah, yeah, exactly. And that is in the collision detection.
Would you agree with that?
Well, as to whether they're using a comparator with efference copy with a sensory input, I don't know.
I don't know of any real examples. But if they're walking, imagine here we have
a lobster, right? You're walking, you're a lobster. Yeah, yeah, yeah.
Okay, so I'm moving my legs, and now one of my legs hits a rock while I'm walking.
So part of the changes in my joint angles are due to my own walking movements I'm initiating.
Yeah, yeah. But now on a few of my
joints, this joint angle will come out differently because of an obstacle.
Mm-hmm. So how do we know this is an obstacle and not me walking?
Well, we don't deal with that. Yeah. I mean, I went through originally a real
plan to have joint receptors and having them feedback.
And I had a whole set of reflex pathways.
But before we put those on the robot, we actually tried the robot without them
over some very irregular substrates, cobble fields and stuff like that. And they do just fine.
And one of the things about a
lobster weighs about one-eighth its weight underwater as it would in air.
So a seven-pound lobster weighs just a few ounces underwater.
And the actual forces against gravity are very, very small, especially when
compared with a lateral hydrodynamic resistance to flow.
So when they get in surge, they're getting a lot more action from the surge than they are from.
Perturbations of their orientation relative to gravity and the
muscles tend to be pretty compliant so you
know right now on the newest generation
of lobster robot that we're building i have no plans to put limb proprioceptors
at all okay all right so this is interesting right so so what you've done why
don't you be studying a lobster in great detail yeah and then in order to let's
say understand that loves you've been building robot lobsters Yeah.
So how many generations of robot lobsters have you built? We're now on the fourth generation.
So when did you build the first one? How long ago was that? The first one was started in 1998.
Okay. So how did this system now progress? What's the difference between version
one versus version four?
Okay. Version one was the infinite power, infinite bandwidth variety.
Okay. And it was basically a set of legs on a hull.
It had a claw and tail on it. and then it was controlled by an external computer
through a serial interface.
And what we would do is send byte commands to some control boards and the byte
camp commands would say, turn on this muscle at a particular frequency or turn it off.
And we would be sending these byte commands over a serial line.
And that robot performed really quite well and that we got the basic patterns
of coordination working and stuff like that.
The second robot had an onboard computer, but it also had the ability to be
controlled through a serial line, so we could go either way.
And then it had onboard power.
So it was carrying the full mass.
And what we do is just compensate for mass with buoyancy.
So we would put syntactic foam on it so it had about the same mass as a normal
seven-pound lobster underwater.
And now version four? So version four is totally different.
It has basically the same mechanical system, except the basilar joint,
which used to be vertical, is now candid at a 45-degree angle.
And that gives the limb tip a rolling action like a wheel as opposed to a more rectilinear motion.
And that's also consistent with the morphology of the real lobster?
It's exactly like a real lobster. That's exactly the angle that a real lobster's
leg is at. And what's the advantage of that? The advantage of that is it lets the animal rear back.
So if it's trying to walk up a slope or down a slope, it has a more natural
angle of attack of the leg.
Okay. So now you emphasize very much just the legs.
Um what kind of sensors are you considering well we
have again from one to four yeah yeah right uh well
the basic sensor suite we have on is a compass we
have a pitch and roll inclinometer that is
also an accelerometer and that gives
us basically three axes of pitch roll
and yaw as well well as um the rate
of change of pitch roll in the off and then
we have bump sensors on the
claws and they basically have an accelerometer they full wave rectify the signal
low pass filter it and then create a square wave that indicates a bump and that
has a duration associated with it so it has a little bit of memory of having had bumped.
And then we have antennae, and the antennae have bin sensors in them.
And the antennae can be deployed at different angles, so they can be held straight
out in front of the animal or off to the sides laterally.
And their bin gives us a very quantifiable measure of the flow rate of the ocean around it.
And we're also putting optical flow sensors on the new robot.
So it can use optical flow information and compare that with a hydrodynamic
flow information and get a clearer picture of things that might be ambiguous with one sensor.
Okay, so if we talk about sensor processing now, are you saying that also lobsters use optic flow?
Oh, no question they do. Really? Yeah, I published a paper on that in Science in 1972.
I was barely born.
So they they often live in
rather murky and dark environments yeah so how much
help but their light their eyes are very are very sensitive in low light okay
so they can and viresma showed years ago that they have unidirectional optical
flow sensors and we work with jeff barrows from sentai on the RoboBee program,
and we're going to adapt the sensors from that program on the robot lobster.
Right. And on the RoboLamprey, for that matter. Right. But then we're skipping ahead, right? Yeah.
So now we have our lobster. We have the sensors. We have some of the sensors. Now we have the walking.
Now how about the control? Okay, there's where the big difference is.
So the new lobster, rather than being controlled by what was effectively a state
machine that mimicked the operation of central pattern generators,
it's now controlled by true central pattern generators that are formed from
what are called discrete-time map-based neurons.
And discrete-time map-based neurons were developed by Nikolai Rukov,
my colleague from the Institute for Nonlinear Science at UC San Diego.
And these are neurons that are a one-dimensional map that are a,
oh, what's the word for it? I'm blanking on this.
A phonological model of neurons. Right.
Okay, so they try to capture the dynamics of neurons, and they have two control
parameters that let us change the neurons from being either truly spiking neurons
that fire in tonic spiking patterns or bursting patterns,
or we can configure the two control parameters so they become chaotic.
Okay, but now can you control the spike frequency quite easily,
and the burst, and the bursting? Oh yeah, really. Yeah, there's another parameter,
which is the synaptic current.
So we can modify the synaptic current.
And we can modify that parametrically as would occur during neuromodulation,
or we can apply regular synaptic pulses from other neurons.
Okay, so how do you build a central pattern generator with that?
Well, the neurons are basically a...
Two equations, and the equations have some fuzzy logic over different ranges of membrane voltage.
And those equations represent the voltage in cycle n plus one as a function of voltage in cycle n.
And you loop through cycle by cycle and keep calculating the voltage in cycle
n plus one, and that turns out to be the voltage of the neuron,
the transmembrane voltage.
So by changing the rate, you can speed them up. So we need to speed them up,
for example, in RoboBee, but we can run them quite slowly in RoboLobster.
So these elements are intrinsically oscillating. Yeah. Okay,
so that's what you're exploiting then.
Well, by varying these two control parameters, alpha and sigma,
we can put them into a bursting regime.
Or for postural behavioral practice, we can put them in a spiking regime.
But now to, for instance, get coordinated movement of the six legs,
you have to also coordinate among these oscillators.
Yeah, and we have coordinating neurons that do that.
So the coordinating neurons pass information from a governing oscillator to
a governed oscillator so that the governed oscillator maintains the proper phase
with the governing oscillator to maintain a gate,
which is an attempt to create a pattern of footfall support to keep the thing
stable in the pitch and roll plane. Okay.
And how well is then such a control model validated in biological terms?
Do you have, let's say, analogs in the lobster nervous system somewhere that
would match, for instance, to this master oscillator that controls these sub-oscillators?
Do you have examples of that? Yeah, so the model is based initially off dynamical
analysis that I did from electromyograms in behaving lobsters in the 70s.
And then one of Al's, I mean, a Franco-Clarac student named Abraham Shrashri
did paired recording of many neurons in the thoracic ganglia,
worked out the synaptic network,
and we tried to capture that synaptic network in the network model that we built with the neurons.
So the neurons are connected by synapses, and the synapses are chemical synapses
that take into account the presynaptic voltage,
the postsynaptic voltage, and inject current appropriate to the difference between
pre- and postsynaptic voltages. Okay.
But then in some sense, I could argue, look, that's nice and well,
but it might just be a very phenomenological model that is sort of at the functional
end gives you something that is lobster-like walking.
But how do I kind of know more specifically that this is actually informing us about the lobster?
How does it help us understand what lobsters really do and how their brains work?
Well, for all intents and purposes, the neurons operate in the same patterns
that you would see from electromyograms.
So the timing, the patterns of output that we see are quite indistinguishable
from the patterns that you would see in behaving animals under the same circumstances.
They don't capture the details of the conductance mechanisms, okay?
But if we tried to use parallel conductance theory,
we'd have to use differential equations given the
fact that some of these neurons probably have five or six currents there
would probably be 10 or 12 differential equations for each
neuron and we wouldn't be able to model very many on a real-time processor right
we're using a model that's based on difference equations and as a result we
can do hundreds of neurons and synapses on a single digital signal processing
chip right but we don't program them as.
Whereas algorithmically, we program them by wiring up networks,
and we establish the dynamics of the neurons by putting these two control parameters,
alpha and sigma, in the appropriate range.
So now, have you found a correlate in the lobster nervous system of these two
control parameters? What would they be?
Again, these are phenomenological models, and there's no one-for-one mapping
of these phenomenological control parameters on any ionic conductance parameter.
And that's something that at UCSD they spent several years, and as Henry or
Bob and I would put it, there were a lot of bodies out in the hall trying to solve this problem.
And the bodies were decomposing.
Exactly.
Let this be a warning. So now we are at level thoracic ganglia, right?
Yeah. So we're controlling these legs, they're moving, we have a neural model
to do that. But actually, most of the bulk of the nervous system of this animal
sits above these thoracic ganglia.
Yeah, yeah. So what are they doing in your robot?
Okay, so we have a lot of exteroceptive reflexes that we've layered in what
would be the brain of this robot.
And, you know, there's been a lot of talk in this meeting about Breitenberg machines.
Well, I don't know of any alternative to a Breitenberg machine, you know.
Neurons receives input on one side or the other.
They project to one side or the other, and they can either project to the same
side or they can decussate to the opposite side.
There aren't a whole lot of alternatives in a bilaterally symmetrical nervous system.
So we have layered reflexes for optical flow.
We have layered reflexes for hydrodynamic flow, for bump, for deviations in
the pitch and roll plane and we have...
As we layer on more and more sensors, and very soon we'll have some very good
chemical sensors, we'll be able to begin to layer those sorts of reflexes on
top of these more fundamental reflexes. But now, which structures of that brain are you modeling?
Do you, for instance, approach the optical lobes of the lobster brain?
Are these also modeled at the neural level? We're not modeling the… So,
this is very interesting.
We operate at the level of what we call releasing mechanisms.
So our sensors are typical analog electrical sensors.
And then what we do with them is we create a spiking discharge,
typically range fractionated, where we take an input variable that might be
the amount of bending of the right antennae.
And then we create from that a set of interneurons that are recruited in order
of size that represent the magnitude of that input variable.
So if it were, say, if the antennae were held out to directly at right angles
to the long body axis, as flow came from the front, at low flow rates,
the antennae would be bent a little bit.
As the flow rates increased, they'd be bent more and more.
Sure. And we usually quantize these with about three levels of range fractionation,
which is fairly typical for the receptors we see in these animals.
Okay, but then we're really at the periphery of the animal, right? Yeah, yeah, yeah.
So what's the difference between the structure of, let's say,
a lobster brain and a drosophila brain?
If you look at the key structures, are these roughly similar?
Similar no no there are no mushroom bodies in the
lobster okay okay so so lobsters
have a series of four um um
there's a lamina ganglionaris a medulla externa medulla interna and a medulla
terminalis which are the four integrative layers going from the omentidia into
the optic nerve projecting in the central nervous system these are where virsma
used to be doing pin recordings in crayfish fish,
and crabs to establish all the six different types of interneurons that come in from the eye.
The thing that really distinguishes crustaceans from, say, humans is that we
just have one kind of, or two kinds of ganglion cells.
We have the on-center, off-surround, and the off-surround, on-center.
And these animals have, for example, fibers that Wiersma used to call,
oh, I'm blanking on these names now. So what did he call them?
Sustaining fibers which would respond to an
increase in illumination in one area there were
dimming fibers that would respond to a decrease in in
illumination in an area there were what he called unidirectional motion fibers
they're what he called space constant fibers and then he had a kind of fiber
he called seeing fiber in which these things would respond to things like movement
of the contralateral legs,
images in the contralateral eye.
He described their behavior as complex as the behavior of the whole animal.
And these would be like whole field. Yeah. And he also had another class of
fiber called jittery movement fibers, which we would call in a frog bug detectors.
Right. Okay. And so the thing about lobsters is that they have 30 or 40% of
their central neurons in their brain are out in their eye stalks doing optical processing.
Okay. And the number of central cell bodies within the brain itself are confined
largely to the motor neurons that go to the head appendages and then some releasing
mechanisms that respond to input from the statuses,
from the antennules, and from the antennae.
So that would mean that relatively little hardware is dedicated to olfaction?
In the lobster, you know, that's Barry Aki's world.
And chuck derby's world they focus on that and um you know we really haven't
gotten into that yet and as we get more of a capability of building sensors
using um synthetic biological approaches,
then we're going to start working in that area but that's that's right now something
i've just been funded to do and we really haven't started yet okay but now so
um so you started this work Or actually, you're a physiologist, right?
Yeah, I'm a systems neurophysiologist. Right. So at some point,
you decided to put this robot lobster together.
And I guess it was with the ambition to actually understand the real lobster.
Well, no. The way this all started is very interesting.
So I had spent a lot of time working on sea lamprey.
And we were interested in how they recover from spinal cord injuries.
And um we were successful at identifying how they recover the ability to turn on swimming.
And then the next grant review i put out um i got a review that said uh this
is very fundable work if you do it in a in a mammal and i don't do that you
know i'm you're not going to see me working on mice and rats, I guarantee you.
Uh, and so I decided to go back to the lobster and,
And this was at a time when people were really beginning to establish a really
incredible library of neuromodulatory substances in the stomatogastric ganglion.
And I think at that point, there were about 35 known substances that would alter
the motor output patterns generated by the stomatogastric ganglion. them.
So, um, working with Al Silverston and George Heinzel, um,
we decided to try to get a handle on which of those were really operating in
lobsters and which were artifactual because for them to be 35 different modulatory
substances was a little bit over the top.
So I got together with a company called Massa Products,
and we started developing a sonar biotelemetry system, which was a physiological
telemetry system, where we would record from the muscles that are controlled
by the stomatogastric ganglion.
And many of these muscles have only one motor neuron.
Or I think max, there's four motor neurons in a muscle.
And if you are studying feeding in
a lobster the lobster feeds when you feed it
what you feed it it doesn't have a chance to
select the chinese food or the italian food and it
doesn't have a chance to pick when it's going to eat so if you really want to
see what the normal patterns of operation of the stomatogastric ganglion are
you have to do it in freely behaving animals so the goal of this project was
to record for the muscles controlled by this ganglion in animals that were free
to run around in the world.
And the idea is that we took electromyographic recordings from these muscles,
did some signal processing, and then would transmit using sonar the on times
and off times of these different muscles.
And I created a webpage about this and got some funding from the Office of Naval Research.
And I got a phone call from a program officer at DARPA,
and he had seen this website and wanted to know if we could use this to take
control of a giant lobster to use the lobster for remote sensing purposes.
And he and I conversed for quite a while, and I told him that it was really
my belief that if you try to control an animal, it's going to do what it damn
well pleases. You're not going to have much luck doing this with a lobster.
And I personally thought it was easier to build a robotic lobster.
And I had been listening to Randy Beer talk about how they were building robotic insects.
And this looked like a pretty interesting endeavor to me.
Well, he took me very seriously and asked me how I would do this.
And I developed some ideas.
And then he asked me to write a proposal. And he gave me a considerable sum
of money to build a robotic lobster.
So I went out and hired a bunch of engineers.
And I had a very interesting experience at this point because I found that the
engineers that I was working with could be divided into two categories.
One were guys that knew how to make stuff work. And the other were what I called
experts on what's impossible.
And they turned out to be kind of lethal because no matter what you wanted to
build, they would find some reason why in principle it couldn't work.
And I found myself in situations where we would have something working and some
of the participating engineers would tell me that that couldn't possibly be happening.
And so I found it quite necessary to weed out this crew in order to be successful.
And that was the genesis of the lobster robot. Okay.
So here we have your neurophysiology, your system's neurophysiology.
Now we have sort of the robot lobster.
But now in retrospect, because you're doing this now for quite some time,
has it helped you in any way to understand the biological system?
Oh, very much so. Or if that was just a game, would it have been better to stick
to the neurophysiology? Yeah, yeah, yeah, yeah.
No, I think what really happens when you try to build a complete system is you
very quickly identify the lacunae in your knowledge.
And that really gives you some ideas on what you should really be looking for.
So for example i have right now a student
in my laboratory that's doing recordings from the
brain connectives going from the brain down to the lower
ganglia to look at the patterns of discharge of some of the the systems that
we can use to inform the way we drive the systems in our electronic nervous
system okay so that's really a clear finding from doing that but,
The other thing that building the robot lobster informed me on was this idea of bump sensing.
You know, if you look at the literature on crustaceans, everybody gets so worried
about what inner neurons come from what joint.
And I think a lot of these inner neurons are just responding to gross disturbances
of the sort that occur when the claw hits a rock, for example.
But how would that be achieved? I mean, is that a direct linking to some mechanoreceptor,
or is it more complicated?
We're not in a position to create cortitonal organs or bipolar neurons.
So we do this by using cheap analog sensors, and then we take either a small
PIC microcontroller or something like that, and create what we call a releasing mechanism.
And the output of that releasing mechanism are patterns of neuronal activity,
which form the input of our sensors.
Right. Okay. So then in your field tests, these robots have shown to be pretty
robust, which is actually remarkable because you could say, look,
this is in some of the minimal amount of control you could give them.
And then you give them this sort of chaos-based way to escape from local minima
when they're stuck in whatever, between obstacles or whatever.
And it seems to be sufficient, at least that's what it looks like,
to give you fairly robust behavior.
But where does it actually, what are the weaknesses? Are there weaknesses in
this approach? Does it really get stuck somewhere?
Are there problems it cannot solve? Yeah, yeah. Well, deciding how you're stuck
is one of the problems that we're working on right now.
And we do this with accelerometry. a tree. So what we do is we take a behaving animal,
And we look at the output of an accelerometer in real time, and we ask ourself,
what is the pattern of acceleration you would see when a lobster normally starts walking forwards?
Now, if you tell your robot lobster to walk forwards, and you see a different
pattern of acceleration, that's a pretty good indication you're stuck.
If you back up and you see the normal pattern of acceleration,
that's a pretty good indication that you aren't impeded in this direction.
So by comparing the patterns of movement in response to a command-initiated
behavior with the actual movement, we can determine whether we're stuck or not.
So that's in fact one of the areas of research that we're really going to get
into big time with the fourth generation robot.
Okay but but then then we are back at something like
a corollary discharge to make that happen there there's yeah
yeah yeah and in fact i think the the
you hit the nail right on the head that the in fact what we would see in response
to normal acceleration would be the corollary discharge exactly yeah yeah all
right but i think that would more reflect what the command neuron would be doing
rather than the output of the CPG.
But still, it would need some internal model of how its own body is acting.
I think, well, the way we actually plan to do this, and this is something that
we are fooling around with, is to have a leaky integrator that receives input
from the command neuron.
And by adjusting the rise and fall time of the leaky integrator so it mimics
what the normal pattern of acceleration would be.
And then we can compare that with the actual pattern of acceleration but this
leaky integrator you now propose you sort of.
Do not care whether at this stage you know whether the real lobster has that
same leaky integrator or not.
Well, again here, the robot's going to inform what to look for in the real animal. Exactly.
And then that's the kind of experiment we will continue to do in the real animals.
Right. So I'm always going to have with every model I work with somebody doing
biology and somebody doing robotics.
And they're going to inform each other. Yeah, well, it's interesting because
in analyzing all the history of the project, it's more that the robot is based
on, let's say, some informed imagination that you test in the biology.
Yeah. But not that the biology has made concrete suggestions of what to do on
the robot, except to give it six legs and a claw.
Yeah, well, I mean, I don't think we, before we started building robots,
that we really knew what to look for as well as we do after having built robots. Okay.
Okay, so it really helps you, right? Yeah, yeah. But now, in some sense,
this approach has been extremely successful because you have been now expanding
into many other projects, right?
And one of the issues, there's a great interest in these kinds of machines and
robots also because practical applications of, let's say, autonomous technology
are actually still fairly limited, right?
So you pointed out the reality of what it means to keep one of these drones up in the air, right?
So what's the problem there exactly? Exactly.
What's the problem with keeping drones going, and will we actually ever make them autonomous?
Well, most of the military robots right now are teleoperated.
And they're teleoperated primarily from the perspective that...
What's the best way to say this? This is complicated. First of all,
most of them are very large, expensive pieces of equipment. So,
a Predator, for example, is a big piece of equipment that you don't want falling
in somebody's backyard.
Nope. As a result, nobody attempts even to try to fly these autonomously.
And I don't really know what the details of this are.
I imagine we're hearing on the radio now that the pilot of a typical passenger
airplane is really only flying at about three minutes.
And the rest of the time it's on autopilot. So I guess the autopilot is autonomous
behavior that's very closely watched by a human operator.
But the risk management issues,
given the cost of these vehicles and the potential danger of them having a mishap,
has led the military to use teleoperation for almost every vehicle.
Now, once they get used to teleoperation, you know, they're not going to be.
So willing to play with autonomous behavior because they might lose a very expensive piece of equipment.
Now, the way we build these robots, probably the most expensive part outside
of the intellectual capital is the batteries.
So we can build vehicles where if you lose a few, it's all right.
And if we base the fabrication and our fourth generation robot,
is really designed for manufacture so it'll
be very cheap to produce these in quantity and and
we're already building four of them right now now when we used to our original
robots we built them one at a time right so we're just starting off with a mess
of them from the beginning so we won't feel so bad if one of them rides off
over the horizon yeah sure yeah so it's So the point is that,
so there's this great interest to have more,
let's say, a biomimetic approach to building these kinds of machines we can
use to identify or deal with agents of harm like landmines or missing nukes or what have you.
And a number of programs are underway in the US now to also realize these systems,
right? Like the MESS program from DARPA and so on.
So in that context you have now been expanding from let's say the lobster also into the insect right,
that was an artificial bee project so to what extent will the artificial bee
be a flying version of your lobster what's different to it.
I mean, I'm a comparative physiologist, and I don't think that we're ever going
to completely understand any organism.
But I think we're going to find general principles that are shared by broad
numbers of organisms that by using comparative physiology to figure out how
these things work in the most technically accessible species,
we can assemble a pretty complete library of control principles that that apply to all arthropods.
So, I mean, that's the sort of approach I take. You know, I don't think we're
gonna be able to record from very many neurons in behaving flies ever,
or behaving bumblebees, or behaving bees.
Their nervous system is quite small, the neurons are small, the electrodes we
use are still pretty big, you know.
There may be some advances using optogenetics where we are able to look at some
of these neurons using optical recording in the future that will give us better access.
But I think we're going to be forced to stick with this idea of find the general
principles by looking at animals where you get the best technical access and
try to build a more general model.
So in that regard, I think a lot of the control principles that apply to the
lobsters, certainly optical flow control.
Pitch and roll control, accelerometry, tree that sort of
thing will apply equally well to the bee okay so
so what are the principles that that stand out for you now for let's say the
control of the behavior of of these animals well certainly use of decussation
and and ipsilaterally projecting uh,
things give us both positive and negative feedback control that's proportional.
So, for example, we can use differences in the magnitude of an input stimulus
on the two sides to give us proportional control to maintain course,
say, in the yaw plane, for example.
So I think that's a principle that applies to all of our robots.
I think using range fractionation to fractionate sensory inputs into different
levels that we can use to apply different logical control projections over different
ranges of stimulus magnitude is another principle.
So say, for example, if you've got optical flow information going from front
to rear at low velocities, you might want that to project to the opposite side.
But as you approach a wall or an object, you might want it to project to the
same side to cause it to steer away, for example.
Right. But then you could argue if we go from lobster to the bee,
in some sense, the medium is changing. We go from water to air.
The scale is changing. We go from a pretty big animal to a pretty small animal.
So this has quite an impact on the dynamics of the behavior. Yeah.
And you could argue maybe these principles of the lobster will just fall down
or crumble in the face of the scale reduction and also this change in the relationship to the medium.
So the bee is following different principles, right? So what makes you hopeful
that this generalization will work?
Well, we're testing it right now. Okay.
So right now we're building helicopter-based robots to mimic the bee,
trying to work out these principles where we can use larger sensors and find
ways to get the coding right and then find ways to miniaturize that.
Right. And at that point, we'll sort of test the idea of whether scaling works or not.
And I don't think we can really do those tests until we have the hardware.
Work but then do that does it also mean that
you see the b brain as a miniaturized version of the lobster brain
i mean if to some extent yeah yeah this
should be true right yeah yeah yeah i think for certainly if we were going to
put chemoreception on the b and flow sensing etc i think those all are pretty
generally organized among the animal among the arthropods okay but but bees
have mushroom bodies and lobsters
do not yeah yeah yeah so where does that difference and come from Well,
I mean, we got to ask ourselves what the mushroom bodies are doing, you know.
Good question. Yeah, yeah. That's a very good question.
And I think my approach is to see what we're missing with the implementation
of the reflex pathways that we know about.
And that will certainly give us some indication of what we might want to look
to in the mushroom bodies. Okay, well, mushroom bodies in the insect literature
would be seen as a classification, a memory system, right? That's certainly
my impression. Of multimodal input.
So if you look at the chemical world, and I also would assume that for a lobster,
most of its interactions with the world are chemical.
Certainly when we talk about distal interactions with the world.
So is the complexity of the kind of compounds it's exposed to lower than what
you would expect from a bee?
Might that be a difference that explains the absence of a mushroom body?
Yeah. I mean, the lobsters really have two sets of, they have a smell receptor
in the antennules and a taste receptor on the walking legs.
And that's just one type of receptor. So there's no variable set of receptors.
Well, Chuck Gerby's done a lot of work on this and Beriaki, and they know exactly
what amino acids are responding to and in what proportion of the hair cells respond to that.
Again, in that we're just beginning to deal with that, I haven't paid the level
of attention that I should be paying and will be paying.
You'll be back. Yeah, exactly. No, that's very good. No, but this is interesting,
right? Because when you say, and this is completely plausible, right?
That these, let's say functional principles to generalize, then in some sense
we're committing ourselves to the implication which is, well,
these brains should also generalize, right?
We should find similarities between the structure because that,
in the end, gives you that function, one way or the other, right?
Yeah, yeah. But now one other principle that you...
Mentioned is the one of reflex chaining to to
let's say deal with with a bit
more complex behavior so what do you mean with reflex chaining
exactly well i mean i think the best example of reflex chaining is feeding in
the leech the work that was originally done by mike dickinson and charlotte
chuck lamb in which they found that at each stage uh of feeding that the animals
would encounter reflex feedback,
which would trigger the next phase of the behavior.
And that went from initially perceiving the water waves to swimming,
to contact with the leech, where they would then start crawling,
to contact with a warm spot, where they would then start biting,
to the flow of blood, which would then start them to do this suction paralysis, I mean, peristalsis.
So that was one of the best examples of reflex training. What we're looking
at with the RoboBee is when it leaves the hive, it's going to be given a search
vector, which is a compass heading and some odometry information.
We're going to have it fly out for a distance specified by the odometer,
at which point it will switch to looking ahead with UV ommatidia in a 3x3 array.
And in the horizontal plane, those ommatidia are going to cause yaw changes,
and in the pitch plane, they're going to cause pitch changes,
so that the robot will home in on a UV source, which would be a flower. Okay.
Once it, it will then use optical flow information to slow itself down as it
approaches the flower and then sort of bump around and get some pollen.
And at that point it will reverse heading and fly back to the hive using the
inverse of the odometry information that used to fly out.
And then we'll have a UV LED on the hive, which you can use to home in to a
docking station to recharge. Okay.
So that could work in a vacuum, right?
But if we now do it in an airflow, it will be drift.
So how is your autometer going to be corrected for drift?
Well, we're going to use optical flow information to respond to deviations of
rotation that are superimposed on the normal translation to get it to come back on heading.
Okay. And we'll probably periodically, if it exhibits a big deviation of that,
we'll probably get it to listen to its compass a bit and get back on heading.
Have you considered using a solar compass?
We're toying with that idea. So we have some people on the team that are involved
in the whole visual sense of the bee and the idea of a sun compass that uses
polarized light is something we're exploring.
All right. That would be a powerful solution. So now here we have the robot bee.
It's like a scaled down version of the lobster. Sure.
Okay, this is my claim. This is not your claim. Okay, but I'm sort of summarizing this.
So we have these sort of more uniform principles that we try to identify.
But the other thing, you also really want to deploy this.
So that means there's a lot of engineering involved to make such a system actually really work.
I mean, you can have these ideas based on neurothology, how we can control it.
But to get it done is actually not the story. We have to think about power,
sensing, integration, computation.
So how are you going to get that done? Well, there's a big team.
We have eight other investigators that are charged with addressing all those parts.
And my part is really to focus on the innate control of flight behavior.
There's another crew that's doing colony interactions. There are people that
are building the airframes.
There are people that are doing the wings, the airfoils. There are people doing the power supply.
There are people doing the and there are people doing the power electronics
to provide power to the actuators.
So this is not a one-man operation by any stretch.
Right. And the PI, Rob Wood, has really put together an extraordinarily elegant
plan for coordinating this.
And we're in our second year right now and and boy it's
looking good yeah okay yeah i mean i'm uh we just
did our our second year progress report to nsf we
had a site visit and and they were quite
happy with where we're at at this point uh-huh right oh that's
impressive so when are you going to see the first one fly well they
there are flying now but they're flying in sort
of open loop and they do crash and burn and we're we're we're flying the helicopters
that's where we're really try to do the ground truth experiments on all these
sensor systems so when will we see the first one fly autonomously for longer
than 10 minutes the b yeah.
The current expectation with the available power source rehab right now is on
the order of three minutes right okay um,
the whole issue of power for these things is an area of intense exploration.
We're looking at varieties of chemical batteries.
We're looking at solid oxide fuel cells, and we're also looking at supercapacitors.
So there's a broad variety of power supplies we might employ,
and this is why this is not a one-year project.
Right, exactly. you know but it's sort of humbling right
because we're sitting here having all these concerns about the brain but we
can give these things a super brain and still they won't fly because we just
have don't have the right power sources yeah so there are some other problems
and and how does actuation look how reliable is actuation really the wing the
wing control extraordinary and um.
Where my colleagues are in the process of writing up the fabrication process,
which is truly impressive.
And out of deference to their intellectual property rights, I think it's best
to not talk about that. Right. But we're just going to say it's fantastic.
Yeah, it's fantastic. I am a true believer right now.
There was a period of time where I was skeptical, but I really think this is going to happen.
And I think you hit the nail right on the head that getting the power to get
the appropriate duration of flight is one of the bigger challenges we face. Right, absolutely.
So here we have to be.
But now another challenge that you highlighted was this whole issue of chemical sensing, right?
So chemical sensing is a really interesting problem. I think it's also very
much an underestimated problem.
And you could also argue that this might be the oldest sense that we have from
an evolutionary perspective.
This is how cells will start to deal with the world, it's through chemical sensing.
So what you were sketching is that in terms of our technology for chemical sense,
if you want to understand this biological phenomenon and again build the technology,
right now this technology is not good enough in terms of sensitivity and robustness and so on.
And you are seeking to overcome this, taking a sort of a synthetic biology approach, right?
So what does it actually really mean with respect to chemical sensing?
Okay. With chemical sensing, there's basically three processes that we have to deal with.
One is the actual sensor itself, the receptor that binds an odorant molecule
that is going to cause some change in the cell, which might be an inward current.
It might be an enzymatic cascade that leads to a great amplification of the
response so that you might get a big cellular response to a single input molecule.
And then some sort of reporter that's going to report to us that,
in fact, this cell has contacted the odorant.
And then finally, some transduction mechanism by which we take the reporter
and activate of eight is sensory neuron and electronic nervous system.
So those are the three levels that we're working at.
Now, clearly for the actual receptor, we're going to use some sort of G protein coupled receptor.
And in some of these, they can be coupled with a calcium channel through a G
protein, such that when it binds an odorant, there'll be an inward current of calcium.
We can also have a G-protein coupled receptor that's coupled to nitric oxide
synthase so that the cell will generate nitric oxide, and then we can use a
nitric oxide electrode,
which is basically a Nafon membrane over a pair of silver and carbon electrodes
that would report with nitric oxide.
Okay. Okay, so then those electrodes,
which might be a photodiode that responds... Well, wait a minute.
When the calcium in the first case enters the cell, we would put an aqueorin
and silenzorime, which give off light.
Or we might put a luciferase, which gives off light.
And then our actual transduction mechanism would be a photodiode.
Right. Mm-hmm. But so, what will be the spectrum of compounds such a sensor can be sensitive to?
Well, there are a broad variety of things.
Pretty much anything you can smell has a G-protein coupled receptor.
And if you go to the library of biological parts at MIT, you can find a library of receptors.
And you can use those genes.
There are some two-component receptors, for example, that respond to RDX,
which is the explosive in C4.
And Jude Mulford has done some work with this where they, if you've seen these
experiments where they put a explosive receptor into grasses and the grasses
change color if they're planted over a mine.
So there's some really interesting tricks that can be put there.
There but then if you want to have multiple compounds you'll have
multiple diodes yeah right so you
might have a problem at your readout level then yeah yeah
yeah so you can have this reporting with light of different
wavelengths or with nitric oxide okay and and we can there's a uh a charge coupled
kind of photodiode uh that we can put filters over and then we can get different
filters to respond to different but then you would have a very so then in some of the very local cool.
But if you look at bees, it's true for your lobster, they have many chemoreceptors
distributed over their antenna, over parts of the body.
And this information arguably is also used for detection and localization.
So how do you scale up from let's say this very localized detection and readout
to this more global, spatially organized way of detection and reporting to some
controller what's going on out there. Yeah, yeah, yeah.
Well, I think we've got to create sense organs, and the sense organs might have a variety of sensors.
And one of our ways of increasing the scale of the kinds of features that we
can use is to use a new technology called EJET printing.
And EJET printing is a new technology that was developed by John Rogers and
Andrew Elwine at University of Illinois at Urbana-Champaign,
in which they use a sputter-coated microelectrode that's sputter-coated with gold.
And they put an ink in that and then
put that over a nanometer stage that can
move in x and y planes in nanometer increments
and then use very high voltages to put droplets to print very very small droplets
and they've been able to print protein features that are five microns in diameter
so then we can use this to create very very fine detail on an electrode system
that could be combined several photoreceptors on the same sense organ.
And the sense organ might be constructed on a glass cover slip, for example,
where we glue the bacterial cells that have the receptors we want on one side,
and then have the photodetectors on the opposite side of a piece of glass,
and we could be sniffing for several different things at once there.
Right. So that's one way we could approach that, yeah. Okay,
that's pretty impressive.
So in chemical sensing, another issue is that, for instance,
if you look at the moth, the male moth, the sensitivity at the periphery is
an order of magnitude or so lower as that at the neural end.
That's when you can look at changes in heart rate due to a pheromone,
and you can see that you can already detect changes in heart rate at concentration
levels that single receptors cannot detect reliably.
So there's a boost somewhere occurring in the neural processing of these signals.
So are you already looking into that issue or that's the future?
That's the future, definitely. Okay.
So now, so okay, we have the lobster, we have the bee project,
and this one was in the robotics domain, right?
So what's cooking in parallel to this in the neurophysiology?
At this point, we have proposals under review that I don't think I should be talking about.
But they're fantastic. They are fantastic. Exactly.
And in that they're under review
at this point, I just don't think it's appropriate to bring that up.
No, don't worry about it. We have plenty of ideas, believe me.
Oh, yeah, no, I'm sure about that.
So in a broader sense, right, outside of the specifics of your proposals,
where do you see this field go?
Go do you see that the field of biomedics as you
envision it is actually having an impact is it changing the way we do science
certainly so yeah yeah and i think really what's going on here is that we've
switched from analytical neuroscience to synthetic neuroscience and it's time
we start building things using principles that we established through analytical neuroscience.
Yeah but i could also argue that this might be true for you and very few of
your colleagues but that the majority of neuroscience has ended up in more sort
of a massive data-gathering exercise.
So will this have an impact to the majority of neuroscientists and how they
think about their discipline?
Well, I think it will depend on how many of them read our papers and take advantage
of the knowledge we've been able to create.
But I understand you're still optimistic. I'm very optimistic.
I think this is really where it's got to go.
I mean, this is the wide open frontier of neuroscience.
There is so much opportunity here. And I think that my colleagues that lag in
getting involved in this are going to miss out.
But so then the other issue is how do you see the scale up? Because I could
argue like, well, that's nice and well for the bee.
But in some sense, who cares about the bee? That's not really advanced organism.
Organism why don't you show me something around the
macaque or why don't you go up to humans or
even a rodent you know that would be more impressive so
how do you see the scaling up challenge i'm quite satisfied eating my experiments
and the lobsters are an extraordinary model for that and if i just uh create
a truly biomimetic lobster that does all the things that lobsters do I think
that will be quite an accomplishment,
you know, I, I.
I'm not particularly interested in working on mammals. I do enjoy some of the
technical advantages of working on lower, simple lower vertebrates like sea lamprey.
But in terms of the warm-blooded animals and consciousness and all that,
I'll leave that to others. Okay.
But you see this more as a personal idiosyncrasy or that's really also a very
clear scientific strategy?
In other words, you say, look, if I can crack this nut of the lobster or of
the bee, I can capture these general design principles, then I'm really close
to understanding any other brain that evolution has generated.
Yeah, but I think we need to figure out the lobster first, and that may be my
own personal idiosyncrasy.
Okay, right. So then to finish up, two questions.
So you came a long way in some sense.
I'm quite sure when you were postdoc, you weren't really thinking about ending
up talking about robots.
I'm not sure, but this is what I imagine. In fact, the whole robot business came up 10 years ago.
And if you had asked me 11 years ago if I was going to be working on robots,
I would have given you an odd look.
Exactly. But you tend to do that anyway, I realized.
But so on the basis of all this experience in terms of trying to understand
the brain, what's this one law of John Ayer you would like to give us?
The John Ayer law. The Ayer's principle. Ayer's principle, right?
The Ayer's principle is choose a model organism you can eat.
You can eat or you're allowed to eat. There's no problem eating lobsters. Okay.
All right. And I do have a recipe in my cookbook for honey smoked lobsters.
We're beginning to integrate all these lines of work.
So the last question is, five years from now, I'm going to trace you down and
I'm going to confront you with the prediction you're going to give me today.
So what's this one strong prediction you would commit yourself to today in this
domain of building lobsters, understanding lobsters, building bees, understanding bees?
What's the one prediction that you care about the most? Well,
I believe that we're going to be able to produce vehicles that will...
To realize what i call reactive autonomy uh at the scale certainly of a of a lobster,
uh within five years i have no doubt of that
and that that looks very very clear at this point all right and so where i think
i should define by what i mean by supervised autonomy well we actually we call
it supervised reactive autonomy right and the idea is that say if you take your dog for a walk.
Your dog is autonomously following you around.
If you chose to throw a stick out in the water and the dog swims out and retrieves
the stick, it's reactively autonomous under your supervision.
And it's doing that portion of the mission on its own.
And certainly my sense is that the people that would like to be operating robots
in the field would like to have that level of control over them.
I don't, I think the idea of these things just going off and dreaming up what
they want to do on their own is not the best idea.
And we wouldn't even want our dogs doing that, you know?
Right. So I think maintaining some modicum of supervision, uh,
is, is the best idea, but I think we can truly expect these robots to be able to perform,
small aspects of missions completely autonomously.
But okay, now that you went to qualify autonomous.
There's this issue that controls often also illusion, right?
We often believe we control our technology, and that we can have controlled
autonomy, and then in practice turns out actually our control was an illusion.
There was no real, also partially your dog might be conditioning you,
and just thinks, well, look, he likes to throw sticks, so I bring him another one.
So, to what extent can you actually
really quantitatively constrain and define this notion of control?
That's an interesting issue. Well, we're certainly going to have control over
the amount of power and the mission duration they're going to have.
So I think that's going to give us ultimately a degree of control that we can be confident in.
I think the general impression of the state of the art of robotics is that the
state of the art of autonomy is for a teleoperated vehicle to be able to recover
its teleoperative link if it loses it.
And I think there's this really interesting example that occurred last summer
where the Navy was doing an exercise in Chesapeake Bay with 13 Remuses,
and they lost four of them.
What are Remuses? Remuses are a torpedo-like robot.
It's the only autonomous vehicle in the Navy inventory right now.
And they were operating three of
them as a group and four of
them lost their teleoperative link and got lost
and they ended up recovering one of
the four that were lost was a marine mammal which is what they were intended
to replace in the in the beginning right so i think that will give you this
the good quantitative measure of the state of the art right exactly it's it's
sobering so joe ayers thank you very much for this conversation it's a pleasure
Pleasure. That's great.
The CSN podcast was produced by the Convergent Science Network of Biometrics
and Biohybrid Systems, a project funded by the European 7th Research Framework Program.
For more interviews, recorded lectures, or upcoming conferences in the field
of biometrics and biohybrid systems, go to csnnetwork.eu.
And thank you for listening.
Music.