This is the Convergent Science Network podcast. Leading researchers in the domain
of neuroscience, brain theory and technology are interviewed by Paul Verscher and Tony Prescott.
This is Tony Prescott for the Convergent Science Network podcast from the Barcelona
Summer School on Cognition, Brain and Technology 2011.
I'm here with Maria Chiara Carroza from PISA, who's the director of the Scuola
Superiore Santana and the leader of the Neuro-Robotics Research Group.
So, Chiara, in your talk today, you used the phrase human-robot symbiosis.
Can you explain what you mean by that?
Symbiosis for us, we are developing robots
and means that the robot and the
user of the robot should share the same objectives and are interdependent in
going to task and in making actions and in doing some activities of daily living.
So the robot and the human being are in contact and share the same goals.
Okay, so how is it that the robot can understand the goals of a person?
So this is the crucial problem to be solved,
how to exchange information between the robot and human subject and in order
to have the subject being in control of the robot with what we call natural control.
And the title of your group is the Neuro-Robotics Research Group.
So that implies that you're in some way trying to read the neural activity of
the person. Is that right?
The neuro-robotics means that we would like to join the knowledge in the field
of robotics and in the field of neuroscience in order to develop better robots
robot able to read the activity of the brain.
It's not important to read the activity of neurons, but of the brain.
So we would like to exploit the signals that are available in order to understand
what the subject really wants to do and try to send this command to the robot
in order to have the robot execute the action that the human being wants to perform.
So for the robot to To understand the human, there are various signal channels that you could access.
And so which are the ones that you think are the most promising?
It's very difficult to... We are still investigating, so it's difficult to say
which are more promising.
We are for the moment using non-invasive signals, which means that we are not
implanting interfaces in the brain and in the cortical areas of the brain.
We are trying to exploit the signals that are available without providing a
surgical implant to the subject.
That means that we are using electromyographic signals from the surface electrodes on the skin.
We are also using the movements of the limbs in order to detect the intention.
We are using the movement of the eyes and all physiological signals that are
available in order to understand what the subject wants to do,
and also the conditions of the subject, which are important to provide a good
interaction between the robot and the subject.
So it sounds like this is some kind of wearable technology that people would have. Yes, yes.
We are investigating wearable robots and in particular we are investigating wearable interfaces.
So interfaces that are in contact with the human skin and the human body,
able to detect the movements and also able to stimulate the human skin in order
to provide some sensory feedback to the user.
Because percepting the environment is important for the user.
And so we have to provide some sensory feedback in order to achieve,
to reach the perception in the human brain, which is a mix of sensory feedback,
experience and knowledge and motivation.
So we need several agents in order to provide perception.
And perception is fundamental in order to provide natural control.
So the types of robots that you're mainly interested in are ones that are interacting
very closely with people.
And one type of robot, I guess, would actually be a prosthesis,
something like a replacement limb, which would be attached to a person and read
these kinds of signals from their skin, for instance.
Yes, we are investigating prosthetic hands and in particular how to control
the hand with the brain, which means to achieve this body ownership with an artificial hand.
That means that the artificial hand is in contact with the human skin because
it's implanted on the stump of the subject, and we want to develop.
A bridge between the hand and the brain in order to provide this exchange of
signals between the hand and the brain. So this is the main point.
So a lot of people are building robot hands, but what is special or different
about a robot hand that could be worn by a person as a prosthesis?
First of all, you have to develop a hand which is cosmetic, which means that
the external appearance must be good.
You cannot think that the user were sort of non-static hand with iron or similar,
reminding us to Terminator or to fiction robots.
So the user wants a hand which is similar to the natural hands and wants to
have social interaction and satisfaction in contacting people and in being in the external world.
So first of all, the hand should be with a cosmetic glove, then must be lightweight
because it is implanted on the stump, so the mechanical interaction between
the hand and the stump must be good.
And also there is the problem of power consumption. If the hand is active,
the power internal to the hand is coming from the battery, and And in order
to provide eight hours or eight operating hours,
the motors and the activities of the hand must be provided by a single battery.
So you cannot think to plug the hand during normal activities.
For example, during working activities or entertainment activities or similar,
you must provide eight hours or normal operation.
That means that the system must monitor the energy consumption and try to distribute
the energy in an appropriate way without losing energy.
And also that there is another problem.
It's related to sensory feedback.
So the hand is in contact with the environment.
So you should provide the sensors for monitoring the interaction between the
hand and the environment and
also sensors for providing proprioception and body ownership to the user.
So you mentioned body ownership and you described a really interesting experiment
in your talk where people, subjects in the experiment somehow get a sense of
body ownership for a hand that's made out of rubber.
Now, how can that come about?
Yes, this is an interesting example, well known in neurophysiology,
and we are trying to export this typical experiment of neurophysiology into robotics.
That is an experiment where a subject is watching a cosmetic hand,
a rubber hand, which is close to the natural hand.
And the subject, by doing some interaction, by brushing on the hand,
on the rubber hand, on the natural hand, the subject is looking only at the artificial hand.
If the hand is really similar to the natural hand, the subject starts to develop
a sort of body ownership for the rubber hand, which starts to belong to the subject.
So when you stab Stab the rubber hand the subject feels that you are stabbing the real hand.
So it's something about the correlation of the sensory signals that I get from
touch on my real hand and seeing somebody stroking the rubber hand that gives
me this illusion that the rubber hand is part of my body.
Yes. And so it's a mixture of the visual signal and the tactile signal.
Okay, the task for neurophysiologists, neuroscientists is to understand if it
is only related to vision and involving only the areas that are related to vision in the brain.
And there are some papers demonstrating that also the sensory motor areas of
the brain are involved in the task.
That means that there is this involvement of perception for us.
We are roboticists, we are much more interested in the implications for that.
That means that we are in principle able to provide a sort of rubber hand, which really.
Can be introduced in the loop with the brain and also the sensory motor areas
of the brain can be involved in perceiving the hand as a natural hand.
So, what we would like to develop is a sort of cosmetic active hand,
which is replacing the hand which is missing.
So it's a step forward because in that case the subject is without hand and
must own the new hand, which is artificial.
So, existing prostheses are presumably made out of plastic or wood or whatever.
But if I was wearing such a hand, presumably I still have some feel of ownership
over the hand because when I touch something with a prosthesis,
I would still get feedback through my arm from that contact,
even though it's mediated by a large piece of plastic.
But you plan to go beyond that and give some direct feedback using sensors on the hand? Yes, yes.
The stamp is providing some direct feedback about the force and the movement
of the hand because the proprioception on the stump of the subject is still there.
And so the subject can feel something in the stump.
But we would like to do more because what we would like to do is that in the
socket, which is the interface between the hand and the stump, The socket is the place,
the mechanical interface between the subject body and the hand.
In the socket we would like to put some new motors and these motors are intended
to provide some stimulation on the skin.
So we made a lot of experiments in understanding how far we can go in delivering
different stimuli by varying the frequency and also the amplitude of the stimuli.
And in order to assess the discriminational ability of the subject,
what we would like to do is to make a sort of mapping between the activity of
the hand and the stimuli that we provide internal to the socket, to the skin.
So the subject can learn about and can learn to recognize the activity of the
hand thanks to this mapping.
For example, we touch the finger and we provide some stimuli and the subject
feels on the stamp a stimuli according to the intensity of the contact force or similar.
Or the force, the intensity of the force can be modulated by providing some
frequency signals to the stamp of the subject.
So this activity is in fundamental in order to provide this sensory feedback
and to develop this body ownership for the subject.
So you say that they feel this on the skin of the arm, but the sense is on the
tip of the robot finger. Now, is it possible?
That, after wearing this device for some time, even though the stimulus is on
the arm, the stimulus is actually coming from the finger.
So is it possible that the experience of the subject migrates so that they,
like in the rubber hand illusion, they have the experience that the stimulus
is on the tip of the finger?
Yes, this is exactly the kind of experiment that we are doing now.
We did some experiments with the subject with
an implanted interface in the
peripheral nervous system by stimulating the fibers of the nervous system and
providing a sort of stimulation we can elicit some sense of the fingers that
the subject had before having the amputation.
So in principle, we think that it's possible to provide this kind of mapping
and try to investigate on the reaction of the subject.
But in order to do that, we must provide a sort of flexible interface with different
vibrators and stimulators internal to the socket.
And we would like to make experiments with amputees
in order to understand how far we can go
because in principle we have some models from neurophysiology
we know from different experiments but
the data with human beings are very few and
thanks to the experiment which is not non-invasive it's not requiring surgical
intervention we think that we would find some amputees, volunteers.
In order to make experiments on that.
And one of the things that we're taking advantage of here is that the brain
maps for the body are very adaptive.
And I guess they can change and learn to incorporate something which is actually
an artificial limb as a replacement for a real limb.
Yes, we rely on that. Neuroscientists are telling us that the ability of the
brain to adapt is enormous.
We know that the motivation for the amputee, for the subject to be in the loop
and to achieve this kind of body ownership is strong.
We know that subjects using prosthetic hands are waiting for a new development,
and feeling the external environment is one of the major points when they raise
issues about future development of the hands.
They want to feel the external environment. environment so
we think that because we know that
the motivation from from the subject is fundamental in
this kind of research so we we know
that we can we can do something for them
but we need some experiments so sometimes
people that have lost a limb report a
phantom limb experience where they feel they still
have the limb and that it's twisted or in some painful position
so is it possible that a prosthetic would
be able to help them get over that phantom limb experience
yes we know that we already developed a
sort of virtual reality system based on our hand in order to investigate on
people suffering from phantom limb pain and we hopefully we will be able When
we will know more about this artificial body ownership.
We will be able also to face this challenge, which is still a challenge for some individuals.
They are suffering this kind of pain, which cannot be treated with normal therapy.
So we know that this can be also a side effect, a positive side effect of our research.
So another kind of robot technology that you're developing is not so much a
prosthesis, but an add-on to the body that you call an exoskeleton.
Could you tell me what that is? The exoskeleton is a sort of external structure
which is coupled to the joint.
So in the case of upper limb is a sort of external suit with articulated system
which is providing assistance to the limb.
So it's providing, for example, additional torque in order to accomplish some task in the upper limb.
For example, in reaching an object, when the subject is weak after suffering
a stroke or similar pathology and has the limb which is weak,
this kind of external structure is able to provide some force to perform the task.
So that can be done for assistance, which means a sort of tool to provide some
additional force for the subject, or can be seen as a therapy.
What is called the neurorehabilitation therapy, which means that this,
by doing some exercises in combination with the exoskeleton.
Like exercise in reaching an object in the space, so flexo-extension of the elbow or the wrist,
the subject is recovering the synergies and recovering the ability to move again the limb.
And these are additional to the normal, ordinary physical therapy and must be
delivered by a therapist in combination with normal therapy.
And we know that there are some data on behavioral experiments and also clinical
data telling us that this kind of exercise size in combination with a robot
are good for restoring movements.
So I'm imagining some device that's strapped to my forearm, and then there's
a hinge at the elbow, and then there's a piece on the upper arm as well.
Yes. And what kind of patients would benefit from using this?
This kind of therapy in general is aimed at restoring motion in subjects with post-stroke.
Stroke so with weak limb after stroke and they in general they have problems in moving again their.
Limb and they are doing some therapy for learning again to control the movement
for the first step is for for reaching an object in the space so the first step
is to reach the object the second is to open the hand and use the hand.
So these kind of exercises are repetitive, which means that they have to do
the exercise several times, and it's an exercise following a trajectory.
For example, by holding a sort of handle in the space,
they have to hold the handle of the joystick and move the joystick in the space
according to a trajectory by following a prescribed trajectory,
trajectory and doing this kind of reaching and moving in the space several times
is useful in order to obtain motor recovery.
So doing that with a therapist is useful but doing that with a robot assisting
the subject which is able to provide the same force, the same pattern and in a very.
Control the environment can provide
a better recovery so these are people
that have got completely normal limb function and
the problem that they have is in the brain yeah where the stroke has damaged
the part of the brain which would normally control the arm yes so we're counting
on brain plasticity to find another way of controlling the arm so if we can
in a repeatable way, move the arm using the robot,
eventually the brain will take over. Is that how it works?
Yes, it's like that. It's more complex because also the arm is weak because it's not used anymore.
And the final objective is to recover also the correct synergies in the muscular
structure of the subject.
So it's not only a problem of brain, but it's also a problem of muscles and
structures and to the correct movements in the space by moving the muscles, but it is exactly that.
And from our point of view, in robotics, we have to develop such kind of systems
which are acting in parallel to the natural limb.
So it's completely different from prosthetics, where you replace a limb which
is missing, so it's not a parallel.
A system is substituting the hand.
In that case of exoskeleton, it's much more complicated because you are introducing
a system which is acting in parallel.
So you have two manipulators in parallel. One is the natural limb and one is
the artificial limb and they are moving together and this is the real symbiosis, which means that the.
Natural and the artificial manipulators must share the same goal and do the same actions.
If this symbiosis is not working, the system is failing.
So this is a really big research problem, that if I have an artificial hand
and I'm trying to control it using my brain and the hand isn't understanding
my intentions, it could do all sorts of things that maybe I don't want it to do.
So how are we going to avoid that? It's very difficult.
This is our problem in developing shared control, which means that you have
the subject, which is who is supervising the hand, and then there is an autonomous
control system internal to the hand.
And so for robotics, the most important thing is to have the system working
together with appropriate harmony.
And so when the system for us is failing,
when it is not executing exactly what the subject is willing to do,
and in that case, the subject wants to open the hand and the hand is not opening.
And that is frustrating for the subject and
in that case the subject refuses to
use the hand because the hand is not good
in performing the task and also for social interaction for the subject is not
good if the sub if the hand is opening or closing or doing things when the the
subject doesn't want in causing some embarrassing.
Situation for the subject.
So we know and we in our experiments we measure the,
success percentage of tasks. So which means that if the system is good,
you can go to 90% of success rate.
If you go to 60%, the system is not good because that means that you have 40%
of cases when the hand is not executing the intended task and the system is not good.
And Because one of the goal is that they use the burden for the user to learn
and then to use the system must be low.
So in one of the Star Wars films, I'm sure you know, Luke Skywalker loses a
hand in a fight with Darth Vader.
And he goes on a hospital ship and they replace it with an artificial hand.
Is that science fiction or is it something that we might achieve one day?
I think we will achieve that.
I'm sure. I'm pretty sure we will achieve. I don't know when.
I don't know if I will be alive, but I think we will achieve that.
And this is exactly our objective.
So you have a wider approach.
So you described the exoskeleton as a way of doing some kind of therapy,
but you're interested more generally in how we could use robots in therapy.
Are there some other ideas related to that that you're developing?
We are also developing artificial skin and we are interested in developing the
skin because the skin is fundamental in order to provide a safe interaction.
It is not the skin of the robot, it is also the skin between the robot and the subject,
which is much more important, because the interface between the robot and the
human body must be monitored in order to avoid situations where you can provide
too much force or too much stress to the skin of the subject.
So we are interested in developing the skin of the robot for what concerns the
interaction with the environment, which means the properties of the object that
is manipulated or similar.
And also we are interested in developing the skin between the robot and the human body.
And in this kind of skin, we are also including the stimulation system that
we were illustrating before.
The skin between the robot and the human body, so this is something that goes
inside the sleeve that fits over the arm?
Yes, inside the sleeve, yes. Right, okay.
And that will be sort of a tight fit but not invasive? Tight fit but not invasive. Yeah.
And this fit is providing you the distribution of forces and stress on the human skin.
So it's fundamental because one of the reasons why the subjects are refusing
system is that they are providing arm to the skin and pain and even too much
force in some particular positions.
And this kind of force is dangerous to provide pain for the subject.
So we know that the good fitting is fundamental for wearing the prosthesis and also the exoskeleton.
So the big research challenge is right now, because it's still in the laboratory,
this technology, isn't it?
You're not yet at the stage of being able to... We are developing experiments
in the laboratory and also we are doing experiments in clinical facilities with
amputees in order to make trials for our interfaces.
So it's not commercial at all.
What do the amputees report about?
It depends on the experiments. So their reports is useful because it's a feedback for our design.
So our design is cyclic, which means that we make the prototype,
we make the experiments, and then we have the feedback on the design,
and that is fundamental.
Some ideas are already transferred to the commercial areas, but it's not directly
that we develop a new hand.
We develop some ideas, some details, also some research lines and guidelines
for designs and also mechanism and patterns.
And these are pieces that are useful for commercial hands and for new products.
It's interesting that you say it's an iterative process
so you're relying on user feedback and that's
one of the things that guides you and where you go with the design process
yes all the development in medical devices
is done like that by making
experiments with subjects and users and going
back in changing the
design according to the user assessment okay well this is all sounds really
exciting I wonder if you have any general predictions for how this technology
is going to change the way our society operates in the 10-15 years from now
I think that the Star Wars prediction is enough for the moment.
That's a pretty ambitious one The hand plug and play Thank you very much and
thank you for coming to BCBT.
The CSN podcast was produced by the Convergent Science Network of Biometrics
and Biohybrid Systems, a project funded by the European 7th Research Framework Programme.
For more interviews, recorded lectures or upcoming conferences in the field
of biometrics and biohybrid systems, go to csnnetwork.eu.
Music.