Good morning, good afternoon, good evening. This
is John Marchando and welcome to another edition
of the Two Guys in a PLC podcast. I'm joined
by my colleague, Dave Gutshall. Good morning,
good afternoon, good evening all. Just a reminder.
For those of you who have not listened to us
before, everything that we say today is for entertainment
purposes only. And all of our opinions are our
own, not reflective of the day job, spouses,
first girlfriend, boyfriend, partners, old flames,
plants, alligators. Did I miss anybody, John?
I never know what you're going to say here, so
I'm always waiting for the, what can I jump in
on? Because that last part always changes. I
love it. So, man, we have got an awesome guest
today, Dave. I think we feel kind of like we're
not worthy of having this gentleman on. So, I
mean, crazy, right? I'm telling you. I
was surprised that, maybe not surprised, the
wrong word. I was just maybe shocked that he
agreed to be on today. So, hopefully we can.
Yeah, hopefully the discussion is as, you know,
entertaining and as useful. And, man, I hope
I learned something today, right? Yeah, really.
I'm telling you. Yeah. So, yeah, let's turn it
over. So, Mr. Gary Workman, how are you, sir?
Oh, doing just fine. Outstanding. Great here
in Michigan. Yeah, you're closer to John than
I am close. Pretty warm on the west side. Yeah,
exactly. Yeah, just a little bit. I was starting
to think that I changed positions with someplace
on the equator. Holy heck, that was ridiculous.
And if this doesn't get published until November,
it's going to sound a little strange. Happy Thanksgiving,
right? Happy Thanksgiving, that's right. Yeah,
we have a couple of those from before. So, Gary,
why don't you spend 30 to 60 seconds and just
tell everybody who you are, a little bit of background.
Wow. Okay, I'm an old retiree. What do they call
those? One of the gray beards from, I think,
some of your previous podcasts. Yeah, I resemble
some of that, so yeah. Well, I spent 48 years
with General Motors, fresh out of high school,
went to General Motors Institute. One of the
unusual things is I graduated with a master's
degree from Purdue before I got my bachelor's
degree from General Motors Institute. Oh, wow.
I was on a bachelor master program there. And
so, yes, that was a little bit unusual. A few
years in an energy conservation group, then I
got an opportunity that was just wonderful to
attend Harvard Business School on a GM fellowship.
Came back and started my networking career. I
was the... I worked my way up to becoming the
chief map architect in the 1980s. Learned an
awful lot about open system networking. Wow.
I haven't talked about map in a long time.
I haven't talked about map in an awful long
time. And then in the late 90s, I was given the
opportunity to investigate the use of Ethernet
for real -time control. And I was given that
opportunity because in the... 80s with a map
program, I would travel around the world saying,
whatever the hell you do, don't use Ethernet
for your control system networks. Ethernet is
a probabilistic network. So by the late 90s,
switched Ethernet had come around. And Ethernet
had changed a lot. And so I changed my tune to,
hey, it's not your father's Ethernet. Drawing
upon the Oldsmobile commercial of the time. And
so I became like a reformed smoker when it came
to. Ethernet and was going around to, hey, if
you haven't looked at Ethernet for your control
system network, you're really missing out on
something here. And it was just amazing because
within the company, I'd keep running into people
who knew me from before and would sit there and
tell me exactly. They would repeat the presentations
I made 10 years ago. And I got to eat over my
dead body. And my reaction was, geez, I'm sorry
to hear you're going to pass away so soon. So,
Gary, what year was that approximately? That
was in the late 90s. By 1997. I had done some
studies that showed that Ethernet outperformed
all of the control system networks that we were
using at the time. Now that was an isolated Ethernet,
but by 2001 we were running our first production
implementations of an Ethernet IP network. So
in the late 90s we were comparing the speeds
of different possible options. We made the determination
that PROFINET and Ethernet IP were virtually
identical in their performance output, despite
what the PROFINET people claim. And we went with
Ethernet IP primarily on my recommendation because
it had the ability to route control system traffic
in a unicast fashion. And for then the next 17...
years, I was implementing Ethernet IP throughout
the corporation. It became the standard first
in North America and then globally. We installed
it particularly with some industrial Ethernet
switches that was interesting to run. And then
I retired in 2017. When COVID came about, I got
the opportunity to co -write a book. about Ethernet
IP networks and talking about how to properly
architect them. So I captured most of what I
learned in that 15 -year career of installing
networks in a book available from Amazon. I don't
know if I'm allowed to promote it like that.
Go for it. Go for it. All good. The Everyman's
Guide to Properly Architecting Ethernet IP Networks,
available from Amazon .com. You can... You can
avoid paying the price because they let you preview
the first chapter. And if you read the first
chapter, you'll get 90 % of the value of the
book just by reading the table of contents because
it tells you what the key recommendations are.
But if you want to hear about some of the internal
discussions and the ITOT culture gap, at least
in one major corporation, you might want to pick
up the book. and read the Gary's View sections
where I pass along some - Totally love it, Gary.
My goodness. Some stories about fighting with
IT. Yeah. And so you wrote this with John Rinaldi,
right? I did write, it is co -written with John
Rinaldi. Okay. And - Interesting story there
that when I met John Rinaldi, I was at an OPC
seminar and - I saw someone walking around with
a book called The Everyman's Guide to Ethernet
IP. I'm sitting there, oh, I know a little bit
about that. Oh, I want to read that, see if it's
worth letting people know about. Went up to the
gentleman who was carrying this book, and he
said, I don't know, I haven't read it yet, but
go out in the lobby, and there at the real -time
automation booth, you can meet a guy who's handing
these books out. And I went out there, and I
didn't see the book on the table. And so I asked
the man, and he happened to be John Rinaldi.
And he said, who are you? And I told him, I work
for General Motors. Oh, I've heard of you. I've
heard of you. I only have a couple copies left,
but if you promise that you will write a review
on Amazon, I will give you one of those copies.
for you to read and hey for a free book i'm a
cheapskate uh picked up the book read it that
night i mean took it home read it cover to cover
like 2 a .m i'm appalled i could not believe
it this book did not have the first thing i learned
about ethernet networks for controls In retrospect,
it wasn't a bad book about the Ethernet IP spec,
but it was written from a supplier's point of
view. Here's how you create an Ethernet IP node.
What I was with the users with General Motors
is I was installing Ethernet IP networks. And
so it's how you put together all these nodes
from all the different companies to interop and
perform a useful function. And they didn't talk
at all. It called itself the everyman's guide.
And it didn't have the first thing about it.
Whatever you do, don't have any collisions on
your Ethernet IP network. It never even mentioned
that. And so I wrote a scathing review. I mean,
it was, oh, it was. But I emailed it to John.
I said, you made me promise I would write a review.
This is the review. I will post it on Amazon
if you insist that I do so. But please, I don't
think. I don't think you'd appreciate it. And
I got a phone call like the very next day. Oh,
you can hold off on your review. Let's talk about
this. And he came to town and we talked about
it. And he said, hmm, okay, I see you have a
different perspective. Yeah, why don't you write
a chapter and I'll write an amendment. And so
I spent a month trying to write a chapter to
modify that book. And it just, no, it took more
than a chapter to write. Also, the whole premise
was, you know, it wasn't building an individual
node. It was getting an interoperable network.
I just came back to John and said, John, I'm
sorry. I'm not so negative on your book. It's
not bad describing the specification per se.
It doesn't help the users. It's not an everyman's
guide. And he goes, well, then we're just going
to have to write another book. And so we did.
And it was a wonderful experience. And it's one
of the things I'm most proud of now, having captured
probably 15 years of learning on the evolution
of Ethernet and Ethernet for control system networks
in particular. Yeah, it's amazing. I mean, so
I got to be honest with you, Gary, as you're
running through kind of all of this history,
I'm coming up with more questions than we possibly
have time for to get to on this podcast. Which
means time for a second podcast. That's exactly
right. We'll have to schedule another one at
some point. But I want to go back to something
you brought up kind of earlier on, especially
because you've seen this evolution for, you know,
since its early days. Everybody calls it something
different. It's mentioned in some type of context.
I feel like somebody the other day said it's
old and tired. Everybody knows it. And that's
this concept of converging the organizations
or converging the technology parts of IT and
OT. So when I say IT and OT convergence to you,
I'm not calling out people or process. Gary,
what? What does that, like, what did you do?
What made it work? Where did you run into problems?
Just start downloading on that a little bit.
Or did it ever work, right? Or did it ever work?
Let's start with the idea of ITOT convergence.
I mean, I first heard that term probably about
15 years ago. And it certainly seemed to come
out from, I believe it was Cisco that was promoting
the idea. It was IT -OT technology convergence.
And I remember it so well because I'd never heard
the acronym OT before. OT in my industry was
overtime. OT convergence doesn't make any sense
at all. And then it was, oh, okay, I'm a little
skeptical. I'm a little skeptical because there's
only one IT department. And there's an executive
vice president in charge of IT. Try and find
an executive vice president or a vice president
of OT. You won't find it. There's just so many
different OTs in the automotive industry. I mean,
there was facilities. There was the press group.
There was the paint group. There was the robotic
automation group. I mean, there's all these different.
There was the material handling and shipping
and logistics and whatnot. So there's a variety
of OT departments. Yeah, which part of production
are you talking about? And which one are you
talking about? Because they have different control
system networks. Primarily, I was working much
more with our powertrain division, and our powertrain
division also had an industrial Ethernet strategy,
much as we did. I was involved in some of that.
But they were going about it differently than
we were. And we were looking at technology, and
we had an executive director that was coming
up with a... control system philosophy a control
system strategy where we were trying to get away
from the usual way of procuring things he was
saying technology is revolving so rapidly let's
try and freeze technology so that we can make
a a Plants identical. Four or five plants identical.
Because the technology curve is so rapid, you'd
put in a plant, then the next plant you'd put
in, you'd use a different technology. Then the
next plant you'd put in is a different technology.
And if you recall back in the 80s and 90s, GM
had multiple different vehicle divisions. And
each division had their own control system philosophy.
And so what we were doing was trying to come
up with a control system strategy that was useful
and appropriate across all the various vehicle
divisions and whatnot. And so you're talking
about like CPC, BOC, those kinds of, okay. Exactly,
exactly. Small car, medium car, large car, truck,
Saturn, each had their own control system philosophy.
So our executive director, he put together a,
he got the control, the primary control system
engineers from each division, got us all in a
room, says, I'm going to make you an offer you
can't refuse. You have five different ways of
manufacturing a vehicle, and they all work. I
want you engineers to agree on the way. We are
going to manufacture our vehicles, the control
system architecture for all our vehicles, or
I'm going to choose one of the five that I know
that works, and I'm going to impose it on everyone
else. So have at it, gentlemen, and gave us like
three months to come up with it. And I was the
person coming from the staffs that was there
in terms of network support. So I was coming
up with the networking architecture strategy.
And we came up with, oh, okay, you have, device
-level networks. You have control -level networks.
You have the information technology networks.
And there's a different technology for all of
those. Well, the thing we were dealing with was
10 different device -level networks. People were
using Profibus. People were using Interbus S.
People were using DeviceNet. People were, I mean,
it was just, it was amazing. And so we... stopped
and had a got everyone in the room all discussed
all debate it took probably six months that we
came up with the thing that made most sense for
general motors as well as general motor suppliers
because we interviewed all our suppliers as well
was that we support device net and so then we
started every division was going to start using
device net Well, that decision alone took six
months. This is too damn long. We want a whole
control system architecture. And so they merged
the control system groups together, and we came
up with a global – well, it was North American
initially, common controls architecture. Then
we compared and contrasted and went over and
discussed and debated it with – our Opel division
in Europe and finally came up with a global strategy
agreeing on span of... It was the functional
needs that we were going to implement and whatnot
and the functionality. When it came down to technologies,
we chose control net for the control level networks,
device net for the device level networks, Ethernet
for the IT networks. All well and good. I mean,
Ethernet was everywhere. And the question came
about... Do we really need three different technologies?
Can we push DeviceNet to be a control -level
network? Can we bring Ethernet down to be a control
-level network? Could we bring ControlNet down
to be a device -level network? And I was put
in the effort with that, even whenever we agreed
that ControlNet would be our control -level network.
It's done based on you will do a study comparing
and contrasting and see if we really need to
support three different technologies. And that's
where I did my Ethernet studies and demonstrated
that Ethernet and isolated Ethernet could outperform
any one of the control system network technologies
we were using at the time. And so that became
the effort is how do we eliminate the need for
different network technologies and collapse it
just to device level networks? and the information
slash control level networks, and where is the
distinction there? And that then became an effort
as, how do we do real -time control on Ethernet,
and how do we integrate a real -time control
Ethernet with the IT Ethernet networks? How much
gnashing of teeth and consternation was there
between the groups that were involved in that,
Gary? I don't think there was an... I mean, it
really was. If you thought your division would
be the one the executive director would pick,
you'd have fought it all the way. It really was.
Everyone, I think, came together and seriously
looked at, I mean, we were looking at 110 volts
versus the 110 AC versus the 24 volt DC for all
our sensors and actuators. Because different
divisions chose different. different standards
there so we agreed internationally it sure looked
like the 24 volt dc standard was the way to go
so everyone agreed to that so that got all our
sensors in line uh we did that study we came
up with and device net so everyone's going to
do device net uh we were primarily Rockwell -oriented
in terms of our PLCs. We're primarily PLC -oriented.
There was an investigation of PC -based controls,
but the early ones, I do believe the biggest
issue we ran into was the obsolescence, the rapid
pace and speed of the... personal computer industry
they just changed operating systems they changed
cpus they changed i mean it changed so rapidly
you could not you know you in the amount of time
it takes to install build design install a automotive
factory yeah they'd be on to a new two or three
levels of technology change and you could not
get the spare replacement parts to maintain a
system that you've got a five plus year You know,
Delta on here's we want to do this. And, you
know, by the time it gets implemented in a plant,
it's obsolete. Yeah. You look back at that time
from roughly, let's say, the early 90s to that
10 year span from the early 90s to the mid aughts.
How fast those operating systems hardware standards
changed. It's just ridiculous. I can't I can't
imagine even to your point. I think you said
five plus years. I mean. most of this stuff you
were you know you were designing for 10 20 minimum
you see these things coming out every you know
18 months 24 months my god how do you do anything
with that the industrial market versus the commercial
market yeah it's well understood well pc -based
controls was too much dependent upon the commercial
market hardware and that evolved so fast that
uh uh uh and So one of the things I was investigating
was Ethernet IP versus PROFINET. And we were
involved with a study at the University of Michigan
on some prototype products and the performance
and whatnot. And the difference between them
was a fraction, a tiny fraction of a millisecond
in terms of performance output. And so basically,
PROFINET, you can't route the real -time signal.
Ethernet IP, it was routable because they used
the UDP protocol to convey the control system
traffic. So it was routable, and that's what
led us to primarily pick Ethernet IP. It was
almost a coin flip, but... That seemed to be
significant enough, and I believe it's proven
out to be that that's the case. You bring back
a lot of memories around some of my early days
stuff, which was later years, at least on the
control side. out of the box in a lot of cases
even on the ethernet ip side generally those
ttls are set to a default of one so like out
of the box in a lot of cases that traffic isn't
routable unless you change it but man that that
bit us in some of the early designs where you
know we're expecting to be able to talk to rockwell
controller and ent uh enbt card not on the same
you know l3 segment and it's like why in the
heck is the freaking traffic not getting over
there you you know pull an ether real trace which
is what it was at the time up and suddenly it's
like wait a minute ttl is coming out of one it's
been generated as a one what the you know what
the devil's the deal with that um but to your
point it's actually changeable where whereas
you know it wasn't Now, we were talking about
– well, okay, I just wanted to build up. We were
talking about convergence. Yeah, right. Let's
get back to that. Yeah, and it's – okay, so the
original – whenever – it wasn't over time. I
got to explain that it was operational technology.
And there's an element of truth to the idea of
– IT -OT technology convergence that was going
on 10, 15 years ago. And that element of truth,
because the reason the idea was proposed was
to explain the culture gap between the IT organization
and the OT organizations. And what's happening
is the OT organizations are concerned with production
and they're concerned with real -time control.
And the IT organizations are concerned with...
information technology transfer. And what happens
is, yes, both organizations were looking at using
switched Ethernet, but they're using it in completely
and very vastly different ways. The OT organizations
aren't concerned about anything other than control
traffic and the other traffic, but it's primarily
the control system traffic for production purposes.
And the IT organization was interested in globally
getting information technology, not only throughout
the corporation, but between corporations and
your customers, between corporations and your
suppliers. I mean, they are interested in routing
globally. Whereas a control system network, geez,
the idea of the network is to replace a physical
wire that used to have those bundles of wires.
connecting all the different pieces of equipment.
And now you wanted to replace that with a network.
And you were dealing with the idea of a local
area network and a true local area network. Whereas
with IT, they're using the local area network
to access the internet and access the wide area
network. And so if you take a look at it, I mean,
one of the huge differences is 90 plus percent
of the traffic on an OT control network stays
within the network. Whereas with an IT network,
90 % of the local traffic is going to servers
and the Internet or another country, e -mails.
I mean, off -net. And so it is really an access
to the outside world is the way they were using
switched Ethernet. And so. It really wasn't.
They were using the same technology, but they're
using in vastly different ways. And it helped
explain why there was that divergence and whatnot.
Well, all of a sudden it became so common that
I noticed about five years ago, people started
dropping technology and they stopped using it
as an explanation for why there's a culture gap.
And it became a proposed solution for resolving
the culture gap. And now. I think that's where
we've gone too far, and it's time to stand up
and say it was a slightly flawed narrative. When
you try to use it as a solution to a legitimate
issue, it becomes even more of a flawed narrative.
And what you're trying to do is get IT and OT
collaboration. Not convergence. Yeah, that's
a great point because I think we often misuse
that collaboration versus convergence word. I
think collaboration is better because OT is going
to still have to do what OT does because you're
making your money in OT. You're not making it
in IT. You're producing product in the OT environment.
And downtime, quite frankly, downtime is a killer.
When people are talking about convergence as
a solution, what they start looking for is what
is that converged solution? And so they're sitting
here trying to eliminate the differences between
them because I'm going to a solution for both.
You're absolutely right. joint needs and you're
not trying to sit here and, you know. Who owns
the data? Who owns the equipment? I mean, you
got that back and forth finger pointing contest.
I mean, that is so crazy. Well, and it just,
it killed me. One of my experiences is, I mean,
we had an IT department that called themselves
the real time control group. And what they ended
up being was the scheduling group. They were
responsible for putting out a production schedule.
And we would just laugh at them. They seemed
to think that providing a schedule gave them
a degree of control over what was actually happening
on the plant floor. Now, you try to meet the
schedule, but reality... Reality exists. And
I'm sorry, you telling me to produce that whenever
the machine is down, I ain't producing it. You
don't control a damn thing. Right. Your definitions
around the difference between convergence and
collaboration, man, I feel like we could have
a whole separate podcast just on the cultural.
You touched on a number of things that I think
are so critical. And that is this is always a
we us. That's the reason you're having the discussions.
It's not a one group or one process. It's not
an I or me thing. It's a we thing. That's right. Exactly.
Yeah, it's a we -us thing. It's a we -us. Yes,
the competitor was Toyota, not my IT department.
Exactly, exactly. Well said, well said. Yes,
spot on, Gary. Yeah, and you know, John, I mean,
just going back to like, you know, what we do
on a daily basis, I literally want to almost,
you know, maybe massage our talk track a little
bit because I don't think we do. double click
enough to use one of our favorite favorite colleagues
terms i don't think we double click enough on
making sure that we definition set what that
you know what because i think sometimes we started
using convergence i think now we're using partnership
which is better um but they're still okay great
you got you know both teams need to be talking
but double clicking a little bit more on exactly
when we we're doing this, we're talking about
doing something on the technology side. What
does that really look like? It's not about their
standards or their standards or what they want
or what they want. It's about figuring out the
right thing for the company that is what you
both want and figuring out. Here are my needs.
Here are your needs. How do we find a way to
make those things work together? And if we can't,
then let's agree that we can't. Right. I don't
think we can't. I really, I mean, I then extend
and extrapolate this into the security arena
where IT is responsible for securing the integrity
and the data and the trade secrets and whatnot.
But if you take a look at the data that... OT
in a control system network is dealing with,
they're dealing with control signal traffic.
I mean, in the Ethernet IP specification, it's
called implicit message traffic. It's an implicit
message. Take a look at the definition. It's,
I understand it, you understand it already. It's
not... context independent and it doesn't have
to be you know it's it's not shared throughout
the world and i think where we run into the biggest
issue with security is now what we're trying
to do is get get contextualized data out of the
control system networks now that has to be secured
But it cannot be done so. It can't be acquired
or it can't be acquired intelligently at the
impact of the control signal traffic that has
to go on independent of it. Yeah, I mean, this
is this is very much under, you know, and I think
some manufacturers and some technology vendors
do it different. But I'll pick on the, you know,
the idea because this is the first thing that
popped in my mind. There is concepts of firewalls
failing open or failing closed. And meaning that
when, to your point, it. impacts the controls
traffic. Which one's worse? Right, exactly. Which
one's worse? And in this case, in a lot of cases,
it depends. This is the reason why security,
you know, security in depth and the Swiss cheese
model is so important because having something
down there that you can leverage, but putting
it in a fail open scenario at the edge so that,
you know, the number one care about is still
keep the process running, keep the line running,
keep the plant running, because that's still
that number one care about. Well, I got into
several arguments with our security experts where
they were saying, tell me what your problem is
and we'll tell you the solution. And it's just
kind of like, that's not good enough. I have
to be guaranteed that your solution isn't worse
than my problem. And frankly, at the time, my
number one problem was you, IT, sending virus
protection software. To the limited number of
PCs on the playing floor. So how do you stop
yourselves, please? Yep. I have a story. They
did not appreciate being told that. I'm sure
not. Back in about 20, this would have probably
been, I have to get the dates right, sometime
between probably 2010 and 2011, we're doing some.
line runoff validation testing. And we're running,
not maybe at rate, but we're getting in a groove.
It's all looking good. This was essentially during
my old stomping crowns back in my Harley days
when we essentially deployed Ethernet to the
factory floor 10 years after you guys did it,
Gary. I kid you not, I could see this rolling,
I'm going to call it a rolling blackout. It started
over at online and began moving across the factory.
All these heartbeats were being lost. Heartbeat
lost in that corner. Then it would move over
here. Then it would move over here. And, of course,
everybody's like, what's going on? You know,
the network. It's the network. So I'll cut the
story short and say it was a security team doing
port scanning. The controllers simply couldn't
handle it. It was overloading the comms side
of the controllers and tipping the controllers
over. To your point, it's like, what do you remember
afterwards? You don't remember all the times
they saved you. You remember the one time they
took the plant down. They insisted that we be
able to route traffic up to where... where they
could analyze it, as opposed to coming out to
the plant floor and plugging into the equipment
and analyzing it locally. And the idea, I found
out about this feature that they were wanting
to insist that the OT people implement by a six
-hour outage at the factory where they forgot
to reset the port mirroring. that was going on
to troubleshoot. And that caused a traffic overload
in a segment and caused equipment to stop functioning.
And it was just kind of like, and you expect
me to put something in that has proven to halt
production? Yeah, that's, you'll never live that
down if you do. Yes, there really is a culture
gap. There's not an appreciation of the legitimate
needs of each. And just because it's called Ethernet,
there really is the ability to get a lot more
data off the plant floor. But you have to do
it in an intelligent and proper fashion. And
you've got to prioritize the data. The control
traffic must get through. I mean, that's the
reason the network is out there when it's a control
system network. And getting data off of that
network is secondary and of distant secondary
importance. So I got a question for you. Since
we're talking about the data on the factory floor,
what's your perspective on AI and what do you
think that's going to do for industry and just
your thoughts there? I think AI requires contextualized
data. That's the starting point. And so you're
talking about non -real -time data. And so as
long as it is properly handled as non -real -time
data, I don't have a problem with trying to get
the data from the floor. out to support the AI
type. But right now, I think we're still a little
too... We're not properly treating the... We're
just adding data thinking that bandwidth alone
solves the problem. Oh, I'm going to go up to
gigabit networks. No, you got to sit here and
properly prioritize the data. You got to have
the proper understanding of... what data is needed
and when it's needed, when it impacts the control
system networks. And that was always my perspective,
was the control system. I have some real -time,
hard real -time or soft real -time operation
going on. Well, the contextualized data that
you try to acquire from a system, put in a second
network. if you aren't sure that it is not going
to impact the control system traffic and behaviors.
And that was the one thing, I mean, Ethernet,
it sits there, it works until it doesn't. I mean,
it can have a lot of... flaws and still function
it doesn't have to be healthy to function it
it tries to uh operate even whenever there's
uh you know so you what but with control data
you have to be sure that under the worst possible
circumstances so so monitor your control system
network make it healthy and properly prioritize
contextualized data but it is okay if you get
a busy signal don't sit there and now try and
Push through. Find out why you got a busy signal.
Don't brute force it. Right. Right. Don't brute
force it. So, Gary, you know, in your mind, then
you wouldn't even start looking at. leveraging
AI to do any kind of real time tweaking within
the process. You essentially are like, look,
let's leverage these models for looking at historical.
Let's look at to try and uncover insights that
we didn't have before based on training it to
better understand. you know, pre -failure maintenance
is needed scenarios and future preventive maintenance
scenarios, but kind of keep it out of the control
loop is that use the controllers the way they
have been. Don't try and think that, you know,
you can tweak any of this stuff live and that
anybody can do it better than kind of we originally
set it up for. And it sure seems like most of
this, you're trying to get the data. What you're
trying to do is improve the next product, not
the current one. It's the control is working.
The control system knows about the one I just
produced, the one I'm in the process of producing,
and the very next one to be produced. And that
is the historical nature of a control system
network. And if you're trying to, you know, if
what you're trying to do is... solve future production,
improve future production. You're not dealing
with real -time control in the control system
scheme of things. Yeah, got it. Go ahead, John.
Yeah, no worries. I was just going to say, what
Dave and I often talk about with our teams, because
everybody's talking about AI right now, us, customers,
and so we try to talk about make sure you understand
what the business needs are, the outcome you're
trying to have before you say, let's put an AI
strategy together, because you may not need one.
Things may be fine today as things stand. And
so we've got to be really careful. That's a really
slippery slope, in my personal opinion, on going
too far down that path without really understanding
what's going on. Yeah, don't do it just because
it's the buzzword, right? Don't do it just because
everybody else is saying we need to do it because
it's 12 of the 16 squares on the bingo card.
Again, I start with it requires contextualized
data. And contextualized data is not control
signal data. So properly intermix control signal
data with contextualized data and don't try to
push it beyond that. If you start impacting your
control signal traffic, you're... You're going
to regret it. Yeah. Yeah, I hadn't really thought
about it in that context. Here we're talking
about contextualized data, and I'm going to say
the word context, but I hadn't really thought
of it that way, Gary, and I think your definition
is brilliant. I love it. Yeah, I'm picturing,
to your point, John, I think of everything inside
the control loop right now as control data, and
that we might be under certain circumstances.
maybe live tweaking that with some model that
we deploy. Yeah. Right. Yeah. What Gary's essentially
saying is no, that's the control side of the
controller, not the data that I'm sharing or
that I'm leveraging to make a control system
decision. It's still separate. Yeah. I'm with
you, John, kind of maybe mind blown a little
bit there is that I don't, I generally don't
think of it that way and we should. I think we
should. Absolutely. I love it, Gary. Thank you
for that perspective. I really, really think
that kind of sets my head in the right direction.
I want to go back to I love history lessons,
Gary. I love why certain decisions were made,
why you did it one way versus another way, all
those kinds of things. So I want to go back to
the kind of your anti -Ethernet days for a little
bit and kind of sort of the flip. Was there a
specific thing, some, you know, the suddenly
100 megabit interfaces, just the fact that suddenly
we didn't have shared mediums anymore. So the
collision sense multiple access part kind of
went away once the, you know, the switch came
about. What was the thing that pushed it from,
there is no freaking way it belongs in my control
systems on the factory floor. And wow, it's cheap,
fast, and pretty good. It is a collision -free
Ethernet environment. That was the thing. But
Ethernet was defined by its collision behavior.
Collisions were an inherent fundamental part
of the early Ethernet. That's right. And the
behavior defined the performance. And the performance
just, as traffic increased, the performance peaked,
then dropped off. And it was, I mean, right whenever
you need to network the most, everyone's screaming
for alarms and no traffic gets through. So it
was establishing a collision -free environment
because that was the behavior that was so objectionable.
And that required switched full duplexed Ethernet.
Yeah, so that was the big thing. Gary, do you
remember some of your... like the old test days.
I'd love to hear some history of the testing,
what you brought in, what worked, what didn't,
what blew your mind, any kind of war stories
from kind of that. And I got a question around
that. I mean, was this done in the lean lab at
GM, or did you do real -world plant stuff or
a little bit of both? Oh, it certainly was a
lot of both. We would test it in the lab. Then
we would pilot test it. And then we would test
again for endurance and durability. So stress
test it. And then we would pilot it. And after
a successful six -month pilot, it was deemed
ready for production purposes. So an awful lot
of that was done. I did my earliest testing.
on using shared Ethernet because the switches
were so expensive back then. But it was done
in the realization that switches were coming
and the cost curve was going to make switches
be cost comparable to the shared media hubs.
So I... emulated the performance of a switch
network on a shared media isolated network, just
assured that there was no background traffic
and didn't tie it into the outside world. It
was exclusively the control system traffic on
there. Then we went and we added all sorts of
traffic and found that under our operating conditions,
we got better performance on a... isolated Ethernet
network than we did under the, it was RIO at
the time, the token passing network that virtually
90 % of the vehicle production we were doing
was utilizing. And so demonstrating that under
the worst possible conditions we could conceive
of traffic being on that particular segment,
Ethernet always outperformed. And then we proceeded
to go about and do the same thing with Ethernet
versus DeviceNet. Then we did Ethernet versus
ControlNet. And with the Ethernet IP implementations
and the performance tests that we were doing,
demonstrated that now you had a real issue back
then that people had to properly implement the
TCP IP protocols. So some of the testing we did
with the PC -based controllers, you would find
the PC -based controllers where the control algorithm
was executing under the real -time kernel of
the real -time operating system on the PC, but
the Ethernet communication was running under
a shell of the DOS or... commercial operating
system so the real -time operating system was
time slicing and the ethernet was on a low priority
and so you'd be trying to get control traffic
out of a controller and it would get time sliced
out and parsed and so you'd get unacceptable
behavior So you needed to make sure your TCP
IP protocol stack was properly implemented. So
we came up with some performance tests with that,
and we actually shared our performance test methodology
with the ODVA organization, the owners of the
Ethernet IP spec, to make sure that they would
perform those type of tests whenever they were
validating Ethernet IP implementations. Transport
and network layers of the protocol were controlled
by a real -time operating system kernel that
was given appropriate priority. So was that back
in the FlowPro days, Gary, or was that after
that? That was slightly after the FlowPro days.
Okay. Yeah, I'm familiar with that from my time
at GE Fanuc and our cooperation with them at
GM Powertrain. I mean, the flip to switching
in hardware from just software -based switching
did so much for that because suddenly you could
essentially guarantee that there was space in
that frame buffer, in that packet buffer. Even
if the switch was busy doing something else with
its CPU cycles, you were guaranteed that it would
switch first and then do everything else later.
And man, that helped it. You know, talk about
the importance of determinism. It certainly was.
It certainly was. I mean, way back then, I mean,
we were doing some and some of the some of the
switches weren't weren't non -blocking switches.
I mean, you had the switch itself could get overloaded
in the early days of switching. Absolutely. You
had to make sure that the hardware you were purchasing
for an industrial grade switch was of line speed,
non -blocking. Remember that. Technology was
being used. Yep. Well, if you can believe this,
Gary, we are heading to the end of our hour.
I don't have any idea how we're already here.
I know, right? I'm crazy, right, John? Would
love to wrap. So I just got kind of a question,
Gary, and that is, if you had... I don't know,
the top three things you think about your, would
you say, 48 year career in the business? You
learn so much. You put a bunch of it in a book.
You brought a bunch of it up earlier on the on
the podcast here. What are like the three things
people like if you listen to me and you learn
these three things, your life in this space will
be easier. What are those three things? I am
a reformed. And if you're not looking at Ethernet
for your control system networks, you're missing
the boat. You got to properly segment your Ethernet
networks. Security is paramount today. I mean,
that's that's. Yeah. Look at Ethernet. Make sure
it's secure. And OK, be don't don't be quite
as hard on your IT people as I was. I love it.
Get your ally. Not sure. Yeah, thank you so much,
sir. It was just great. Loved the time. It just
disappeared in a flash, sir. All right. Well,
I'd be interested to hear how it comes across.
Yeah, same. Thanks, everybody, for listening
in today. Had a great exchange with Gary. So
this is John Marchiando signing off, saying goodbye.
See you on the next one.