This is the Convergent Science Network podcast. Leading researchers in the domain
of neuroscience, brain theory and technology are interviewed by Paul Verschoor and Tony Prescott.
This is Paul Verschoor with the Convergent Science Network podcast.
And this episode that we also recorded as part of our
CSN Barcelona Cognition Brain Technology Summer School
I'm talking with Gunter Knoppleg and Gunter
um your the topic
of your presentation is joint action those are actually the
main topic of your of your research right and you decomposed
um joint action uh in a
number of let's say constituent components that you sort of
try to then again investigate in themselves
so why don't you can you explain to me how you see
joint action and how do you see this decomposed in its subcomponents right okay
so um i mean i'm i'm most interested in joint action where people actually perform
some sort of skilled action together like playing football or you know trapeze
artists and um uh one big question in this field is
how do they achieve temporal coordination?
How do they not miss each other? How do they pass the ball in time to score a goal and so on?
So there are different mechanisms to actually achieve this, psychologically speaking.
And it starts, and I was actually describing five different ones,
ranging from entrainment to providing signals for each other that it's of the coordination.
Okay, so you had entrainment, speeding, simulation, monitoring,
and then signaling. Exactly.
So it would be... But now let's backtrack a bit to the joint action issue itself.
So what makes joint action special as compared to,
Other forms of action. Right. So in individual action, the only thing you need
to do is get your own action right somehow.
Whereas in joint action, what you need to do is you need to coordinate your
action with somebody else in space and time.
And it's actually even in the most simple action, like handing a glass to somebody
else, it's actually from the point of view of the motor system.
That's a very tricky task because, for instance, you need to take the other's
actions into account in order to pass the glass on.
And the question is, how can this be achieved? And there are many,
many sort of real-time constraints where the motor system has to solve very
difficult problems in a very short sort of time window.
But then I could argue, okay, but the motor system is already tuned to deal
with a dynamic world. It might not be an intentional world, but a dynamic world.
So what then would make joint action different from, let's say, catching a ball?
Right. I mean, it's not necessarily qualitatively different all the time,
but it's certainly true that biological movement patterns are most of the time
different from simple physical movements.
I mean, they can also be described, of course, as physical movement,
but I think the intentionality behind it makes it more complex and also,
in a sense, there's more variety of movements, so to say.
And it follows different laws than simply sort of gravity.
Because the kinematics of human movement, for instance, they reflect actually
the working of a very complex apparatus that functions,
that produces, you know, kinematic patterns that are quite different than the
pattern that a ball has, you know, in a simple trajectory,
let's say. Okay, so can you give an example of that difference?
Yeah, I mean, think about two players in football, where actually,
you know, one is asked to anticipate where the other is going,
in order to place the ball in the right spot.
This could involve like, quite an intricate prediction,
you know, that requires knowing the player, which way will he turn,
there will be more sort of cognitive level, but it could It could also imply
something like how fast can he move and things like that.
And all of these things implicitly or explicitly could go into the kicking of the ball.
Right. Okay. So now let's turn to entrainment. So how should I think about entrainment
in this context of joint action?
Right. So entrainment is sort of perhaps the most simple way of getting people synchronized.
And that's actually a process that is not necessarily only restricted to humans
because it actually also works for clocks.
So if your entrainment is a principle that actually comes out of physics and
that just states that if you have an oscillation and if you have oscillating
devices like a metronome and they have a physical coupling,
they actually tend to go into phase with one another.
And dynamical systems psychologists have actually made the point that the same
sort of principles could hold for informational couplings, perceptual couplings between people,
so that in a sense there's a general pull towards synchrony
built into the systems so that people
have a tendency if they
perform sort of rhythmic movements to to be in
synchrony with one another so for instance you can observe that when people
many people cross a bridge or so that or walk close to each other they will
tend to walk in the same speed and assimilate at speed so that and this is a
very automatic process that people aren't aware of.
They're just pulled towards this synchronization. Right.
So do we know anything about the genesis of this? What's the generator of that synchronization?
The generator? You mean like an internal clock?
I don't know, yes. Well, look, earlier you said yourself, right?
Oscillators would do this, right? So where's this oscillator?
I think in a sense where this comes from is not… I mean,
there's not really an explicit theory about you know oscillations in the brain
that drive this sort of process,
it's more a particular view in psychology that wants to see sort of social interactions
between people partly driven,
by these sort of coupled oscillations so that in a sense the whole human who
is involved in this thing becomes the oscillator and the coupling is actually
between people So it's not a very cognitivist position,
it's more a sort of applying physical principles to the psychological domain.
Right, but at what point in human behavior does this kind of coupling break down?
Now, we are talking to each other, and at some level of description,
I could say, well, in some state space, I could say we are coupled because I
pose questions and you generate an answer and I try not to interrupt you and so on.
Well, let's say at the level of our physical actions, there might be not a single
signature of such coupling.
So how far can this really go? Where does it break down, this idea of coupling?
I think that this is sort of a very powerful way to look at interpersonal processing,
but it is only one possibility.
And I think that there are many types of social interactions and also joint
actions that cannot be explained by simple entrainment.
But I would still hold that even, so, you know, most of the time it's really
used to explain unintentional cases of synchrony, so that people synchronize
even though they don't want to,
like simulating velocities and rocking chairs, things like that.
Right. Or, you know, nodding their head like we do now in the same rhythm.
In the same sort of temporal frequency.
Mm-hmm.
But it can, of course, also be used, let's say, in dance, as something that
becomes part of an intentional joint action.
But in these cases, of course, the pure synchronization is not enough.
But it can actually help you to achieve what you want to achieve.
But it's only sort of part of the processing. and
in other cases I would say entrainment actually works against the coordination
that is required because we don't always need to be in the same rhythm or in
the same pattern exactly but the interesting thing is in some sense if it would
be entrainment in the way you now just define it it would be.
Actually invalidating the notion of joint action because action
would imply that there's an intentional component to
it while the entrainment would be unintentional right that's perhaps
a little bit of a problem with our our definition of
joint action which we want to be very broad so that
uh when i say action it doesn't necessarily imply
a uh an intention let's
say so it can include just let's say
behavioral patterns yes any kind
yes okay so i wouldn't i wouldn't want to make
a distinction between intention non-intentional action right away because there's
so many cases where it's very difficult to to really draw the boundary between
intentional unintentional action that um i think for our purpose which is sort
of trying to find out the nuts and bolts of joint action.
We rather work with a very broad definition of joint action,
which basically says that we want to talk of a joint action for anything that
involves coordination between people in space and time. So it's super broad.
It ignores perhaps also a little bit philosophical sort of basic distinctions,
but it's been so far very useful actually in trying going to sort of relate
processes that people would normally think of as very different.
Right, but it might also then lead to misclassifications, right?
Imagine we get thrown out of a plane and we're falling down,
free fall, flailing our arms and legs,
then a psychologist using your definition could say, ah, there are entrained,
but basically we're just both falling, racing towards earth and just in panic or flailing around.
Right. I think, but I mean, this example, my analysis would be that the joint
action stops in the moment when you've pushed me, for instance.
And then, you know, what I'm trying to do is to save my life.
And from then on, I would call it separate action.
But in a sense uh i would
perhaps even accept like you know an action like pushing somebody out as a plane
out of the plane as something that could be conceived of as a joint action perhaps
not with the same intended outcome but with different intended outcomes i know
it gets tricky in a way but i don't want to have you know sort of um,
Also, competition versus cooperativeness.
I also want to stay away from that distinction because for these sort of rather
low-level nuts and bolts coordination processes,
it might not matter so much whether you're in a competitive game or whether
you're trying to play cooperatively.
For instance, in table tennis, you can do both. And the coordination processes
might not be so different.
Exactly. This is very good, right? Because by using a broad definition of law,
you should look at, let's say, also the interaction of multiple factors.
And otherwise, you will be really pushed, let's say, drawing boundaries maybe
between them, which would not help you in understanding joint action.
So this is to understand.
But then, so we have entrainment. So now a lot of we were nodding and sort of
we're smiling at regular intervals. So we're really untrained.
But now we're speeding, right? So I'm speaking faster or slower.
Or apparently contributing to our joint action. So how should I think about that?
Yeah, I mean, entrainment is a process, you know, where we don't need to have
any internal representations about one another.
It just happens between us that we are pulled into something.
Speeding is a process where people actually try to sort of modify their own
task in order to make coordination possible.
And that is something that could come in handy when you don't know your partner's task very well.
And I think we have discovered this by accident because we were actually looking
for a different coordination process.
And we found in several different experiments that in situations where the task
was somehow ill-posed, people tended to just speed up.
And this could have been an instance of a very old sort of psychological phenomenon
that's called social facilitation.
You just get faster if you're doing stuff together with other people,
even if you do it not together, but also if you're just in the same room, let's say.
But then we notice that basically speeding
up also implies something statistic you know which is
just a statistic effect because it reduces variability if
you get faster you go to your limits you know
and you you can't become faster and that automatically means that your actions
timing becomes less variable so what you require for coordination is actually
in for instance in order to be synchronous or to engage in a certain rhythm.
Basically less variability helps you to be more synchronous, something like that.
So in some ways you're saying by finding maybe the right speed,
because it's not that you operate at the maximum speed, you just speed up as
compared when you act on your own.
Yes, but actually in our task most of the time that means going to your limits actually.
Okay like what's a task um uh in our
in our case it was a simple reaction time task but
it could also be you know any motor task i'm pretty sure
but but the point you to make
is okay by speeding up you become more predictable and this
would facilitate the joint action i mean it would
help the other to deal with you yeah it's
it's not even that i mean you're not trying to make
yourself predictable for the other you're just speeding up and you
know that in itself basically um
reduces your variability and so
basically it it increases the chances that you will be more synchronous with
the other person so for instance if you if you uh if you go faster and faster
and faster um basically you know if you if you have to clap a very slow rhythm together for instance.
And you need to stay one second apart or something like that.
It's pretty likely that if you try to do that now with clapping,
we would produce large asynchronies and our clapping wouldn't fall together.
If we clapped very fast, we'd have a pretty good chance actually that actually
these variabilities would be much lower.
And in the same sense basically, speeding up can can help you to be in time with one another.
So the speeding up effect is more closely related still to the entrainment we talked about before.
Or not? Because you're saying by speeding up, you increase the probability to
synchronize, so that would be in the form of entrainment or not.
I think we have to make a distinction between synchronicity and entrainment.
I think entrainment is one process that can produce synchronicity,
but there are other processes that can produce synchronicity too.
I think what I'm saying is that speeding can also produce use synchronicity
all right understand so for your synchronicity is in some sense then,
synonymous to a joint action in a broad sense like joint
action by definition must show some synchronicity
many joint actions need to show that
i mean some other joint actions would actually require turn taking okay fair
enough yes okay very good yeah you're right so but for these for this subclass
of joint actions because you know i mean that's sort of there was also one one
main message of the talk to say that, in a sense,
in order to solve different coordination problems that you have in order to perform joint actions,
you can have different processes for solving these coordination problems.
And it's not a single one, so to say. And speeding is one that can actually
help you to be in synchrony with another person if you don't know a lot of other
stuff about that person.
And it is a strategy because we were only observing it in the experiments when
we asked people to be synchronous with one another and not when they sort of
performed the task independently.
So in a sense, speeding can be an intentional strategy that can actually lead
to achieving synchronicity.
But now tell me, if in the speeding, in this component of speeding,
it seems to imply that the behavior itself is again oscillatory in some way.
It has some frequency component and I'm sort of speeding or slowing that frequency element.
This now helps me to catch up with this other oscillator, which is another human doing something.
And then once I caught up, I can now set my frequency equal to the other.
Right. Okay, so maybe speeding is not the best way to name this.
Okay. Because we don't mean sort of speeding up in a task.
We mean sort of speeding up compared to individual performance. Mm-hmm.
So that basically what we had was actually individual reactions to a stimulus.
So it's not a rhythmic action like, you know, in a rocking chair.
Right. But it's sort of discrete responses to stimuli.
Sure. So if you need sort of this, for these discrete responses to stimuli,
speeding up would be a strategy to be synchronous.
But then if I understand it well, then it's like.
If you compare the joint case versus individual case, you see that in processing
discrete stimuli, like my reaction time to the stimulus, there I'm faster,
right? So this is the data.
But then in your interpretation of something like, oh, but by speeding up,
I make it more likely to synchronize with the other.
That's something that is supposedly going to happen after you saw the stimulus
because we only know now we're speeding up to see the stimulus.
No, I think it's a general strategy, really. I mean, it's sort of a different
mindset of approaching the task because what we saw in our experiments is that
people actually didn't speed up during the experiment.
So there's a constant difference between the individual and joint condition
right from the start. Okay.
So that it just means when you are asked to do it synchronously with me, you're going faster.
And you do that right away, so to say. And that modulation is an order of magnitude, I guess not.
It is pretty much. So it's a task where people would normally,
in the individual condition, would normally require about 400 milliseconds to respond.
The joint conditions, they are 50 milliseconds faster.
So it's quite substantial. I mean, it's only 50 milliseconds.
Seconds but uh in experimental psychology in that sort of time window it means
a lot exactly sure so i mean it's a huge it's a huge difference so um.
I mean, I think our most convincing data is in analyzing the correlations between
reaction times, variability, and asynchrony. Okay, because what people need
to achieve is asynchrony.
And what you can show is that if you ask people to synchronize intentionally,
basically, people, you have a big relationship between reaction times and variability.
And again, between variability and asynchrony, and no relationship between reaction
time, direct relationship between reaction times and asynchrony.
Whereas in an unintentional condition, the main relationship is actually a direct
relation between the reaction times and the asynchrony.
And variability doesn't play a role. Which really means that the relationship,
that people are really sort of speeding up in order to reduce variability,
which in turn actually reduces the asynchrony.
That at least is our interpretation. Right, exactly. Because now you see there's a causality here.
Right. And it's only correlations. I have to admit that.
Okay. Okay, so I want to ask whether you had any way to really now dissect this
assumed causal structure. Have you made progress on that?
Well, I think our argument would be that you have to do basically exactly the
same, at least primary task, in an unintentional condition and in the coordination condition.
And just the requirement for coordination completely changes the pattern of
correlations between the three parameters.
But again, it's only correlations, and I think we have to find manipulations that allow us to add,
for instance, variability in order to actually
provide further evidence right i was wondering about
how this would generalize right because imagine it's not
let's say an image uh detection task
or something like that but it is a task where you
use different modalities or maybe it's even more an active task which you also
believe or do you have information that the same speeding occurs um i think
it's a very general strategy so if you um because it It just means that as soon as you don't know,
when you know you need to be synchronous,
and if you and another person have roughly the same task, basically when you
speed up, it's likely to work better.
But I could also argue, well, maybe it's a form of competition.
You want to just be better than the other. That's why you speed up.
Is that an alternative explanation? It is an alternative explanation.
And it would be, I think, you know, many people would be sort of more interested.
Also in social facilitation, you know, heightened arousal is the explanation.
But I think our only point is, what we want to point out is,
here's this very simple strategy.
How, you know, changing the performance on your own task could actually help
you to achieve synchronization with somebody else.
Where you know most of the time people would think
well in order to to achieve that you need to decode the
other's intention or you know simulate the other's actions
right and we want to propose this you know
as a computationally or so cheap uh way of actually
getting um things synchronized uh
between people so for you this is like a joint action
reflex or joint action heuristic or
something like that but completely it's a it's innate in the
system it will always do this you don't need to learn this no okay
people just do it right i mean that's what we saw
in our experiments they didn't have to acquire this they just do
it did have you looked at how how this appears in development
no uh but this was i think it would be really interesting if if kids showed
uh the same tendency right exactly yeah all right so so now we have let's say
entrainment the most simple form of sort of synchronization now we have speeding where we Indeed,
I'll start looking at, let's say, behavioral strategies to facilitate synchronization.
And the next one is now really like a big jump up in, let's say,
the cognitive ladder, because now we're going to talk about simulation. Right. Right. So-
So how does simulation help me now in joint action? What am I simulating?
What you're simulating is the other's task.
The task or the intentions? Actually, not the intentions. It's aspects of the other's performance.
And, of course, you need to sort of represent somehow the other's task,
at least to some detail, and to which detail is actually unknown.
And that can also vary a lot, but that will be a different talk.
But you have to represent that the other person has a task and you have to represent
some details of this task.
And the idea then is that your own action knowledge allows you to actually predict
how difficult it will be for the other person to perform this task or how long
it will take for the other person to perform this task.
Am I modeling the user, the other, and the other's performance?
Or am I really modeling the task? If I model the task... I think you model the
other's performance. Okay, okay.
Because then the question is, is now my modeling of the other's performance,
some sort of modulation of my own model, of my own performance,
or is it something that is really separated from that?
Right. So actually, I mean, the idea is that you use your own motor system,
and it sounds perhaps a bit funny,
that you use your own motor system to plan your own actions,
and at the same time you use the same motor system to actually also predict
what the other person will do so the idea is really of.
In a sense simulating what the other does with your own action knowledge actually biases,
how you perform your own action so in
a sense and this would also involve a predictive of the
prediction of the consequences of the action so in a
sense it's an idea that there could be something
like a parallel prediction for your own action and the
other section so that means if i'm clumsy if i'm
a clumsy person then i will have a tendency to also model you as
a clumsy person but we have to perform a task together yes so
in a sense it would also depend on your experience for
instance so for instance if if you're a very good
dancer and you you perform dancing you will
have more motor activation in your system because you you can sort of simulate
better what you're observing um than when you're not an experienced dancer because
you will simulate a dance movement more like a walking movement let's say or
something that's familiar to you you can still sort of simulate aspects of it,
but not in a very fine-grained detail.
So that means you're sort of in this Merleau-Ponty school there of a perception
that my model of you is grounded in my model of myself.
Yeah, but not in a phenomenological sense, actually.
It's more in a functional sense, sense really um because
um uh i the assumption would
be that the the simulation itself um actually
is sort of completely unconscious it's not you know it's it's not a particular
experience connected to it it's more like um um a parallel prediction of what
could happen in the outside world so um it's i think it's more closely related
to you know the uh mirror system sort of theories or
common coding theories that sort of postulate a close link between perception and action. Mm-hmm.
And I think it's sort of, you know, you can link both action planning perception to external events.
And then basically, once you make that move, the motor system can actually contribute
to action understanding, to action simulation for other people too.
Okay. But then your simulation model is a bit more like a forward model in some
sense, right? Yes, that's true. I'm just predicting states of the world.
And the other is just another aspect of that world who then also has an impact
on that world. So changing its property.
So I can just make now forward predictions about what this impact will be.
That is true. But, you know, in some situations, I mean, one task that we use
in an EEG study was, you know, I'm handing a bottle to you. That was basically the task.
And then we were looking at preparation components for motor actions, for motor initiation.
And the interesting finding there was that the person who was actually sort
of receiving the bottle was showing a sort of motor peak that you normally see
for processes of motor simulation at the point where the person who was giving
initiated their action.
Action, so much before they actually initiated their own receiving action,
which was sort of very clear evidence that the motor system was actually making a contribution,
was sort of simulating the other's actions in addition to one's own action. Right.
So now, in simulation, it's very operational.
You define it in very operational terms, right? So you're not including anything
about making inferences about the goals of the other.
So you think that is not really a required component to account for joint action?
Oh, it's absolutely a required component.
That's sort of the aspect of joint action I didn't talk so much about,
which is sort of you have to have a goal representation that allows you to define
different roles in the joint action and also the joint outcome that you want to achieve.
And the idea is that this sort of goal representation actually drives the simulation
and reaches into the motor system.
And so that basically, you know, this sort of top-down modular,
this sort of high-level planning structure, let's say, actually sort of ties,
gets tied into the motor system.
Them and then does the appropriate simulation also takes care of the integration
of the information that comes out of the motor system.
Okay. But then the specific task that you described here was this musical task,
right? We have two people playing on a piano.
Right. It was under monitoring, though. Oh, sorry. Okay. I'm sorry.
Yeah. This is one step further. But it was interesting to see that also in that
task, actually, it had an impact on, let's say, with respect to predictions
pertaining to the other or the self.
Right. But let's discuss that experiment when we hit the monitoring.
Maybe I'm moving too fast here.
But now for the simulation, to what extent?
So you could imagine that I have a generic simulation system in my brain,
like a mirror mechanism and with all sorts of bells and whistles attached to it.
Would a generic modeling system or simulation system be sufficient to support
joint action or should it have some special ingredients that make joint action
then standing out from just predicting other aspects of the world?
I think it would capture different regularities. So I don't think that it would
necessarily need to be completely different in that it would also,
of course, capture regularities.
But the regularity that it captures would be very different.
And especially in those cases where people perform highly coordinated actions.
So for instance you know if you
take two dancers uh by coordinated tango dances
and so on um so that in
a sense uh you know the specifics of the other's body and movements would become
part of your own prediction models so that and so you know one way of thinking
about this is would be saying it's not different because it also captures regularities
just like any other forward model Yeah.
The other way of thinking about it would be that here we have a system that
captures a different type of regularity that doesn't only depend on my body,
but also on another body.
And could that perhaps capture regularities that are then interesting to use
for other functions in the system so that being in close interactions with others
could perhaps even have, let's say,
something like implications for your bimanual coordination, something like that.
And the other way around would be that perhaps in order for joint action,
you could, and that would be the similarity again,
could I use by manual models where I have to have some similar coordination
problems as when coordinating with other people?
So if you're shaking hands, I can do that with my two hands too.
Can I use the bimanual models as a proxy to bootstrap my joint action model from? Things like that.
Okay. But you do see a continuity there. There's not… Yeah. Okay.
It's not fundamentally different, but I could say, imagine, for instance,
if you wanted to talk about modules or separation of function for a bit.
Because of the different type of regularity that these forward models capture,
because there's perhaps more dynamics or more complex dynamics,
or because the biological movement has invariant characteristics somehow,
it could still be a specified way of modeling.
But the capturing of regularities would be the same. Right, exactly.
But then how quickly in development do you see this?
This ability to capture this and to make predictions. Right.
I don't think we know anything about this actually empirically,
but my hunch would be that in a sense these sort of low-level components are
in place before the sort of planning structures.
So that you have sort of forward models which are able to actually deal with,
let's say, combined human motion in place.
Before children start to perform intentional joint actions, where they sort
of take the initiative in the joint action.
Because, of course, in the broad definition of joint action.
Infants are involved in it right from the first day of birth because they're
being carried around and so on.
They make posture adjustments adjustments that
make it easy for their parents to to carry them
around and so on so i would expect that you
know a lot of this sort of motor stuff is in place and
that sort of the developmental boundary they
have to cross really is more in the development of um
of uh task representations that allows
them to keep you know self and other together assign different
roles to different actors things like that right okay so
now now I can simulate and so
that means I have more power in let's say
predicting and adjusting but now
now I must be able to let's say then monitor the quality of
my performance and the quality of the performance of the other okay
so so how does
this exactly work right i mean i
think we know that um people are very good at
monitoring the outcomes of their own actions and there's
also some previous research that shows that surprisingly uh when
we observe others performing actions you know we also monitor whether they make
an arrow or not and that uh the brain seems to do pretty much the same thing
in both cases so that you can can actually observe the same ERP-EG patterns.
ERP components in both cases.
Now, one question is whether.
In monitoring, you know, you have a separate monitoring mechanism for the joint
outcome of the actions, because in a piano duet,
basically what counts is the coordinated actions of both of us and not the individual playing.
So, you know, we can do a perfect performance of our part, of our individual
parts, and it can sound very terrible.
Right, exactly. And so the question is, you know, how do we represent this destroyed
outcome and are we able to monitor it?
So in this experiment, there's two piano players.
They play a fixed score, two notes at a fixed rhythmicity.
And then you started to sort of perturb the sound that was produced. Right.
So basically what we do is they have, these are expert pianists and they know
how to play pieces very well and basically plant errors in their performances while they're doing it.
So they play the right note, but they hear the wrong tone.
And uh we could so we were measuring eg from from two people at the same time
while they were playing a duet and uh we planted mistakes for the person themselves
for the other person and then we
basically uh planted two types of mistakes one mistake would basically affect
the joint performance a lot and the other mistake would affect it less and so
it would sort of either change or not change,
jointly produce harmonies.
And what we found is that early sort of monitoring components and error components
reacted in exactly the same way to self errors, other errors,
and it also didn't make a difference of whether to which extent a joint result was affected.
Whereas later components that are interpreted as signals of conscious error processing.
Were actually scaled with self-relevance.
So these components, so there was sort of more of an error signal when the person
made a mistake themselves than when the other person made the mistake.
But the second factor, and that's more interesting, is that these signals were
larger for self and other when the joint outcome was more affected.
So which really indicates that, you know, people have to do this monitoring of the joint outcome.
But how was the modulation factor there?
Because in that sense, now you had really parametric control over the error
that you could induce, or not?
No, I mean, it's not a parametric simulation because it's not on a sort of continuous pitch dimension,
because the music harmonies sort of constrain a lot what you can do there, so to say.
So it's not really true that we could have had parametric violation.
So we could only sort of come up with, and that was tough enough actually,
experimental manipulations where we had, we actually changed the joint harmony or not.
Okay. So, so either there was an error or there was no error. This is, this is.
I mean, there wasn't, okay, so in all four cases, we planted errors.
But these errors could be sort
of individual in the sense that they didn't affect the overall harmony.
They still fit into the jointly produced harmony.
So it was just, you know, we replaced one tone that belongs to a particular
chord, let's say, with another tone that also could belong to that chord.
So the chord remains unchanged.
Whereas in the other case, we actually change the pitch so that the chord actually flips.
And that changes the joint result more. Right.
But now, if you look at this P300 response, so these late components for error monitoring,
monitoring um how does it what does
it look like if we compare let's say own errors
versus the errors of the other right so
for the for the for the later component uh um basically uh it um it matters
more you know it's the component is much higher when you make a mistake yourself
okay but But there's still an error signal when the other person makes the mistake,
but only when it has consequences for the joint result.
In other words, sorry, sorry to interrupt, but just to clarify,
it has no impact on the joint outcome if the harmony is not affected,
is that what you're saying?
Okay so the error signal so if
i'm if if uh if you if i
planted an error in your playing you you
would always find the error component yeah exactly and
if i'm your partner in the in the piano duet um
we would only find uh an
error component if my arrow
affects our harmony but not if
it doesn't affect our harmony okay all right and your
arrow component for your own error is is
also higher when the individual when
the sorry when the joint harmony is affected so even
for your own mistake you know the job uh it's
it's sort of worse it's there's a bigger error signal
when it affects the joint reside more so
the error you know that uh in this component so to say the error there's less
of a of an error signal when you make an individual mistake that doesn't affect
our joint harmony so that means um errors detected in my own play yeah are.
Sort of lead to more vigorous responses yes
than those in the play of others right of
the other and um how is this
expressed really in the at the erp level is it like really
an amplitude difference or is it so okay so it's
a it's a difference in the p300 and the
p p300 is really scaled in our four
conditions so it's highest if if it's
your own error and it affects the joint actions it's second highest and significantly
different if it's your own error and if it uh if it doesn't affect the joint
outcome it's third highest if it's my error the other's error and it affects the joint outcome,
and there's no error component whatsoever if I make the error and it doesn't
affect the joint outcome. Okay.
But then in case of monitoring, so here we see that monitoring affects both
the joint action and the own action.
But now, isn't that a bit in a contradiction with the idea of the simulation
where you would say, They say, well, the simulation runs on a more generic infrastructure.
So then you would expect that self and other monitoring would lead to roughly similar responses.
I would guess, okay? I mean, you know better than me. But isn't it surprising
to see that there's such a big difference between these two? Yeah.
Yeah, I think it depends a lot on the domain again, really.
And I think for the type of continuous action that's required for piano playing,
where you also have sort of, you know, normatively, actually,
you know, in the notation, to play two different pieces,
I'm not exactly sure how much simulation could do.
But I also have to say, it has to do something,
thing because we actually know that you know piano players who
actually hear piano playing do have a lot
of the motor activations so um
perhaps it you know depends a
little bit on which aspect so that you know in the piano playing when
it when it comes to pitches and harmony you actually
have to keep it separate for safe and other and you know the joint harmony has
to govern stuff but if you looked at the expressive dynamics perhaps you know
perhaps you would still apply aspects of your own actions to these expressive timing aspects.
There's no reason to believe that only one or the other process can be active
at a time in a joint action. Right.
So now we have a monitoring system. And then our other element is then signaling.
So how does signaling now play a role in this?
Because so I can simulate, I can monitor, I can use this to sort of synchronize my actions.
So why do
i need signaling now to keep on going right i
mean i think that's i think uh
you know the uh the the simulating business
uh uh and perhaps also the monitoring business if you don't have a lot of feedback
about one another can be difficult so for instance the simulation mechanism
i talked about is uh you can actually you actually have yourself to use as a
proxy for or the other person and go from there.
And sometimes that might not be the best thing to do.
Or perhaps you don't have the skill that you would need to be simulating if
you want to perform a particular joint action with another person.
So that signaling is another way of actually achieving coordination.
For instance, when you don't share a skill, or it might just be the more effective
method in a particular situation.
So, for instance, when you can give each other signals, when you can sort of, how should I say,
when a discrete way of sort of affecting the action planning is actually more important than...
So let me give you an example. I think that would make it better to understand.
So if you're carrying a box together, I could probably simulate your behavior,
but you know, we have different perspectives, you walk in a different way than me, and so on.
So if I'm sort of shoving the box in a particular direction.
This communicative signal might actually be much more effective in giving a, let's say,
unambiguous coordination signal than all of the simulations and the monitoring that I'm doing.
So that this is sort of our attempt to now link up to a more sort of symbolic
way of achieving coordination but also retaining the link to the motor system
because in joint action,
you say when we carry the box, we need to coordinate our action in space and
time and we want to provide this signaling
as a let's say um interface perhaps
uh for you know what would could also be
a language input you know you could also bias things in
a language way but people actually um often also do it in a more continuous
manner with these little sort of shoving signals because you know sometimes
it's hard to talk about exactly the actions that you need to do and you know
a haptic push or sort of a proprioceptive signal for the other person might
be more effective than talking a lot.
Right, exactly. But now the task you looked at was this pendulum task.
Actually, it's a box, there's a pendulum, then you have to pull a string left
or right which you can sort of move the pendulum up and down,
and then the instruction is to keep the pendulum within a certain range,
and what you looked at was, you know, when a single person performs this task,
pulling the two strings,
or a dyad, so two people will perform this test, each pulling one string. Yes.
So what's the difference between these two conditions? What do you observe?
Well, I mean, the interesting difference between these conditions is that people
do it in a completely different way.
So you can actually, what individuals do when they're confronted with balancing
this pendulum is that they apply forces in turn to the right and left side to keep it stable,
which means that the forces don't overlap very often on both sides,
which sort of saves them energy.
And it also means that the one hand doesn't get a lot of proprioceptive signals
about what the other hand is doing. Okay.
In a sense, if you perform this individually, you don't need these signals,
because you know what your other hand is doing, you know, within the system.
Whereas in the case where we are doing this together, and, you know,
we have to really control this pendulum movement in real time,
I don't know exactly, you know, how your action will unfold in real time.
So it's actually a good idea to pull at the same time so that we have a lot
of force overlap because then we can feel each other on both sides.
And that is how people achieve coordination.
And we can observe this in two different strategies to move the pendulum because
people sort of in the joint action condition pull at the same time and then
they lose at the same time and they start pulling at the same time again and so on.
Yeah, but as was pointed out by
Andreas Engel in your talk, people have different forms of feedback here.
It's not only the proprioceptive feedback or the force they might feel on the
string like the haptic feedback, but they also see what the other is doing.
So how would that translate in the specific synchronization effect that you observe?
It seems very odd, right? Because what you observe is that the diator,
two people basically start to pull simultaneously, which your device allows,
while the single user is sort of pushing or pulling alternately left or right. Right, yes.
And in both conditions, does the pendulum end up swinging at the same frequency?
Well, we instructed them to achieve a certain frequency and a certain amplitude.
So, and we observed the same difference across, you know, all sorts of different task conditions.
So, you know, whether you had to move the pendulum in the frequent,
let's say three times per second from one side to the other,
or let's say two times, didn't matter.
And also not how far you had to move it, whether you had to let it fall down
almost completely or whether you had to turn it sort of only in a small amplitude.
So across all conditions that we observed, it was pretty much all that the device
allowed, we actually observed the same difference.
And your point about vision, of course, vision was involved all of the time.
But you had equal visual information in the visual case and in the joint case.
So that in both cases, in a sense, you need visual control of the pendulum in
order to do the task at all.
But I think the signaling went through the haptic channel.
Because it would have been hard to observe, to create a false overlap in vision.
I think that's also really an interesting aspect of proprioception,
because it's sort of less this in a sense.
Perhaps it has different advantages compared to the visual system as an information channel.
Well, but in some sense it was also biased by the task. I mean,
people couldn't do anything else than just pull the string.
Yes, absolutely. But that's what we have to do as experimental psychologists.
Of course. No, this is clear.
But now, the alternative could be that, let's say, people just prefer to move
with a certain movement frequency.
Yes. Right. So, that means the individual user alternating effectively has the
same, let's say, effort movement frequency as the dyad that's sort of pulling simultaneously.
Right. So, with one swing of the pendulum, you still move an equal amount of
pull, an equal amount of times.
Right. Can you exclude that, that sort of movement frequency?
We can exclude that with the tempo manipulation that we did.
Because in a sense we were going from, I think it was like one turn per second,
or actually two turns per second, to up to five turns per second.
And that is a wide range of tempo, basically, and we observe the same differences
in all of these conditions,
whether you were going first, you know, whether individuals and groups were
going fast or small, the difference was always there.
But now the other thing about signaling is that in some sense you could argue
this haptic feedback is more like an implicit signal because if I'm part of the diet,
I'm not really sort of deciding to, okay, let's inform the other about what
I'm doing. You get it for free because I'm pulling the string.
I think that's a really interesting aspect of it.
And I'm not sure whether I would actually want to talk a full-blown communication here.
But I find it even more interesting than if it was full-blown communication,
because here seems to be a case where perhaps you have something in between
intending to communicate, and actually something that the task requires.
And my hunch is that if you talk about evolution development of these things,
this could be sort of an interesting in-between case.
And I think in general, haptic coupling in these sort of practical joint actions
is one way of trying to bootstrap communication, intentional communication,
from something simpler.
Right. It's more like implicit communication in this case, yeah? Yeah. Okay.
That's very good. Good. Okay, so now we have this list going from entrainment to signaling.
So how many steps are still missing in the list?
I don't know. I mean, I think we will find out in the future.
I'm pretty sure that there are still steps missing.
And I think it will also be interesting to see how the different processes can work together there.
And then also, you know, which kind of task representations and separations
between task representations. you need and how specifically your representations
of the other staffs need to be for these different processes to be useful.
So for entrainment, for instance, you don't need any representation of the other
person, whereas for simulation you absolutely need it.
So then, if I now want to build a robot that can pull the string of the pendulum
together with you, where would I start?
Would I really immediately have to include all of these five?
Or would you have one or two preferred elements where you say,
no, it should really start here, it's really bootstrapped from these two or three?
I think i you should implement
all of them okay it's sort
of a and i think you know some of i mean it's a cookbook um
i i really i mean at least that was the idea sort of and i think in terms of
of uh implementing them uh there's huge differences in how difficult it would
be because i think the the entrainment one and the speeding for instance could
be very easy um there's uh you You know,
everything that has to do with simulation will probably be very costly.
And I'm not exactly sure about the signaling because it would require something
like a very close crosstalk between modules that do perception action links
and communication modules or task representation modules.
And I think, to my knowledge, they are mostly kept separate in current architectures in robotics.
But that's also sort of the challenge we want to pose in robotics,
to think about the closer links between the different… But if I would go to
the other extreme, and I would look at other kinds of, let's say,
social animals, if I go to ants, for instance.
Would you qualify ants having joint actions?
It depends on your definition of joint action, again.
And I think, I mean, perhaps with our current definition, they would even count.
You know, in our current definition, it all depends on what the social interaction is, really.
And they can do coordination space and time. So I would think that,
you know, certainly training would exist on a level between ends.
And then I think what they do, and we haven't explored that at all,
is also use the environment a lot to actually achieve coordination.
So yes, I think I wouldn't be completely opposed to call some of what ends through joint actions,
but I would also try to be clear about exactly what coordination mechanisms
that involves and what it doesn't involve.
And they would certainly not have very elaborate task representation
right but you would see a continuum there
in some sense absolutely okay absolutely so I wouldn't
I mean that's sort of our general research strategy
not to make any sort of top you know tough
delineation on the way and just
see you know whether that can improve our
understanding okay very good so then to finish up two questions so you are studying
this whole domain of let's say social interaction or joint action intensely
for many years so in our in our study of this phenomena what would be gunter's law gunter's law yeah,
oh that's that's a tricky one.
Gunter's law,
um would be not to postulate too much internal representations and joint intentionality,
for social interaction and to consider what the sensory motor system can actually
do for social cognition quite in general.
Okay. And then last question. So five years from now, I'm going to visit you
there in Budapest and remind you of the hypothesis you're going to generate for me now,
which is, so what's the hypothesis or the prediction that you're that you're
most committed to today that you would like to remind you of five years from
now to see if it really came came true or not a prediction yeah,
about joint action about joint action a specific prediction on on the research you do.
Okay, maybe I should rephrase that as a hope, but I will predict that we will
have increasing impact on sciences that are interested in culture,
and at least we will try to make some links that allow anthropologists to understand
how sensory-motor processing coordination can actually contribute to the development of rituals.
And more sort of, how can I say, more sort of encultured actions, high-level actions,
and how there's a continuity between joint action, ritualized joint action,
and then sort of full-blown communication.
And that's sort of a path that we want to explore, actually together with developmentalists.
Also, we want to understand much more about the development of this phylogenetically
and autogenetically. That's beautiful.
Well, Gunther Knopple, thank you very much for this interview. Okay, thank you.
The CSN Podcast was produced by the Convergent Science Network of Biometrics
and Bio-Hybrid Systems, a project funded by the European 7th Research Framework Programme.
Music.