WEBVTT 00:00:12.460 --> 00:00:14.919 Good morning, good afternoon, good evening. This 00:00:14.919 --> 00:00:18.739 is John Marchando and welcome to another edition 00:00:18.739 --> 00:00:22.379 of the Two Guys in a PLC podcast. I'm joined 00:00:22.379 --> 00:00:26.079 by my colleague, Dave Gutshall. Good morning, 00:00:26.140 --> 00:00:28.800 good afternoon, good evening all. Just a reminder. 00:00:29.500 --> 00:00:31.519 For those of you who have not listened to us 00:00:31.519 --> 00:00:35.060 before, everything that we say today is for entertainment 00:00:35.060 --> 00:00:39.600 purposes only. And all of our opinions are our 00:00:39.600 --> 00:00:43.679 own, not reflective of the day job, spouses, 00:00:44.219 --> 00:00:47.679 first girlfriend, boyfriend, partners, old flames, 00:00:48.000 --> 00:00:50.679 plants, alligators. Did I miss anybody, John? 00:00:51.320 --> 00:00:53.100 I never know what you're going to say here, so 00:00:53.100 --> 00:00:55.100 I'm always waiting for the, what can I jump in 00:00:55.100 --> 00:00:58.039 on? Because that last part always changes. I 00:00:58.039 --> 00:01:00.439 love it. So, man, we have got an awesome guest 00:01:00.439 --> 00:01:05.560 today, Dave. I think we feel kind of like we're 00:01:05.560 --> 00:01:09.379 not worthy of having this gentleman on. So, I 00:01:09.379 --> 00:01:11.079 mean, crazy, right? I'm telling you. I 00:01:11.079 --> 00:01:14.099 was surprised that, maybe not surprised, the 00:01:14.099 --> 00:01:16.959 wrong word. I was just maybe shocked that he 00:01:16.959 --> 00:01:19.519 agreed to be on today. So, hopefully we can. 00:01:20.329 --> 00:01:23.629 Yeah, hopefully the discussion is as, you know, 00:01:23.629 --> 00:01:26.150 entertaining and as useful. And, man, I hope 00:01:26.150 --> 00:01:27.930 I learned something today, right? Yeah, really. 00:01:28.310 --> 00:01:31.569 I'm telling you. Yeah. So, yeah, let's turn it 00:01:31.569 --> 00:01:35.390 over. So, Mr. Gary Workman, how are you, sir? 00:01:36.290 --> 00:01:39.109 Oh, doing just fine. Outstanding. Great here 00:01:39.109 --> 00:01:42.469 in Michigan. Yeah, you're closer to John than 00:01:42.469 --> 00:01:44.069 I am close. Pretty warm on the west side. Yeah, 00:01:44.150 --> 00:01:47.010 exactly. Yeah, just a little bit. I was starting 00:01:47.010 --> 00:01:49.769 to think that I changed positions with someplace 00:01:49.769 --> 00:01:52.310 on the equator. Holy heck, that was ridiculous. 00:01:52.489 --> 00:01:55.209 And if this doesn't get published until November, 00:01:55.390 --> 00:01:57.769 it's going to sound a little strange. Happy Thanksgiving, 00:01:57.790 --> 00:02:00.290 right? Happy Thanksgiving, that's right. Yeah, 00:02:00.329 --> 00:02:03.329 we have a couple of those from before. So, Gary, 00:02:03.530 --> 00:02:06.230 why don't you spend 30 to 60 seconds and just 00:02:06.230 --> 00:02:08.270 tell everybody who you are, a little bit of background. 00:02:09.120 --> 00:02:13.180 Wow. Okay, I'm an old retiree. What do they call 00:02:13.180 --> 00:02:15.620 those? One of the gray beards from, I think, 00:02:15.620 --> 00:02:17.400 some of your previous podcasts. Yeah, I resemble 00:02:17.400 --> 00:02:21.740 some of that, so yeah. Well, I spent 48 years 00:02:21.740 --> 00:02:24.460 with General Motors, fresh out of high school, 00:02:24.580 --> 00:02:27.960 went to General Motors Institute. One of the 00:02:27.960 --> 00:02:31.199 unusual things is I graduated with a master's 00:02:31.199 --> 00:02:34.479 degree from Purdue before I got my bachelor's 00:02:34.479 --> 00:02:37.099 degree from General Motors Institute. Oh, wow. 00:02:38.539 --> 00:02:41.340 I was on a bachelor master program there. And 00:02:41.340 --> 00:02:46.580 so, yes, that was a little bit unusual. A few 00:02:46.580 --> 00:02:49.340 years in an energy conservation group, then I 00:02:49.340 --> 00:02:52.379 got an opportunity that was just wonderful to 00:02:52.379 --> 00:02:55.379 attend Harvard Business School on a GM fellowship. 00:02:56.460 --> 00:03:00.120 Came back and started my networking career. I 00:03:00.120 --> 00:03:03.409 was the... I worked my way up to becoming the 00:03:03.409 --> 00:03:06.689 chief map architect in the 1980s. Learned an 00:03:06.689 --> 00:03:09.990 awful lot about open system networking. Wow. 00:03:10.330 --> 00:03:11.729 I haven't talked about map in a long time. 00:03:11.750 --> 00:03:14.150 I haven't talked about map in an awful long 00:03:14.150 --> 00:03:20.650 time. And then in the late 90s, I was given the 00:03:20.650 --> 00:03:23.770 opportunity to investigate the use of Ethernet 00:03:23.770 --> 00:03:26.689 for real -time control. And I was given that 00:03:26.689 --> 00:03:29.599 opportunity because in the... 80s with a map 00:03:29.599 --> 00:03:31.800 program, I would travel around the world saying, 00:03:31.979 --> 00:03:35.300 whatever the hell you do, don't use Ethernet 00:03:35.300 --> 00:03:39.020 for your control system networks. Ethernet is 00:03:39.020 --> 00:03:43.639 a probabilistic network. So by the late 90s, 00:03:43.639 --> 00:03:47.080 switched Ethernet had come around. And Ethernet 00:03:47.080 --> 00:03:50.379 had changed a lot. And so I changed my tune to, 00:03:50.560 --> 00:03:53.439 hey, it's not your father's Ethernet. Drawing 00:03:53.439 --> 00:03:58.539 upon the Oldsmobile commercial of the time. And 00:03:58.539 --> 00:04:02.620 so I became like a reformed smoker when it came 00:04:02.620 --> 00:04:07.169 to. Ethernet and was going around to, hey, if 00:04:07.169 --> 00:04:09.569 you haven't looked at Ethernet for your control 00:04:09.569 --> 00:04:11.610 system network, you're really missing out on 00:04:11.610 --> 00:04:14.770 something here. And it was just amazing because 00:04:14.770 --> 00:04:18.250 within the company, I'd keep running into people 00:04:18.250 --> 00:04:21.550 who knew me from before and would sit there and 00:04:21.550 --> 00:04:25.529 tell me exactly. They would repeat the presentations 00:04:25.529 --> 00:04:29.870 I made 10 years ago. And I got to eat over my 00:04:29.870 --> 00:04:34.350 dead body. And my reaction was, geez, I'm sorry 00:04:34.350 --> 00:04:38.689 to hear you're going to pass away so soon. So, 00:04:38.769 --> 00:04:40.750 Gary, what year was that approximately? That 00:04:40.750 --> 00:04:47.740 was in the late 90s. By 1997. I had done some 00:04:47.740 --> 00:04:51.459 studies that showed that Ethernet outperformed 00:04:51.459 --> 00:04:54.339 all of the control system networks that we were 00:04:54.339 --> 00:04:57.519 using at the time. Now that was an isolated Ethernet, 00:04:57.660 --> 00:05:01.399 but by 2001 we were running our first production 00:05:01.399 --> 00:05:05.500 implementations of an Ethernet IP network. So 00:05:05.500 --> 00:05:07.720 in the late 90s we were comparing the speeds 00:05:07.720 --> 00:05:11.829 of different possible options. We made the determination 00:05:11.829 --> 00:05:15.290 that PROFINET and Ethernet IP were virtually 00:05:15.290 --> 00:05:18.509 identical in their performance output, despite 00:05:18.509 --> 00:05:23.550 what the PROFINET people claim. And we went with 00:05:23.550 --> 00:05:26.569 Ethernet IP primarily on my recommendation because 00:05:26.569 --> 00:05:30.050 it had the ability to route control system traffic 00:05:30.050 --> 00:05:36.949 in a unicast fashion. And for then the next 17... 00:05:37.310 --> 00:05:40.569 years, I was implementing Ethernet IP throughout 00:05:40.569 --> 00:05:44.209 the corporation. It became the standard first 00:05:44.209 --> 00:05:49.730 in North America and then globally. We installed 00:05:49.730 --> 00:05:53.329 it particularly with some industrial Ethernet 00:05:53.329 --> 00:05:57.209 switches that was interesting to run. And then 00:05:57.209 --> 00:06:03.389 I retired in 2017. When COVID came about, I got 00:06:03.389 --> 00:06:07.220 the opportunity to co -write a book. about Ethernet 00:06:07.220 --> 00:06:10.199 IP networks and talking about how to properly 00:06:10.199 --> 00:06:13.339 architect them. So I captured most of what I 00:06:13.339 --> 00:06:16.019 learned in that 15 -year career of installing 00:06:16.019 --> 00:06:20.980 networks in a book available from Amazon. I don't 00:06:20.980 --> 00:06:22.399 know if I'm allowed to promote it like that. 00:06:22.500 --> 00:06:25.939 Go for it. Go for it. All good. The Everyman's 00:06:25.939 --> 00:06:29.379 Guide to Properly Architecting Ethernet IP Networks, 00:06:29.379 --> 00:06:33.389 available from Amazon .com. You can... You can 00:06:33.389 --> 00:06:35.910 avoid paying the price because they let you preview 00:06:35.910 --> 00:06:38.949 the first chapter. And if you read the first 00:06:38.949 --> 00:06:41.750 chapter, you'll get 90 % of the value of the 00:06:41.750 --> 00:06:45.009 book just by reading the table of contents because 00:06:45.009 --> 00:06:48.930 it tells you what the key recommendations are. 00:06:49.230 --> 00:06:53.149 But if you want to hear about some of the internal 00:06:53.149 --> 00:06:57.470 discussions and the ITOT culture gap, at least 00:06:57.470 --> 00:07:00.310 in one major corporation, you might want to pick 00:07:00.310 --> 00:07:03.040 up the book. and read the Gary's View sections 00:07:03.040 --> 00:07:06.720 where I pass along some - Totally love it, Gary. 00:07:06.860 --> 00:07:08.959 My goodness. Some stories about fighting with 00:07:08.959 --> 00:07:11.540 IT. Yeah. And so you wrote this with John Rinaldi, 00:07:11.600 --> 00:07:13.779 right? I did write, it is co -written with John 00:07:13.779 --> 00:07:19.160 Rinaldi. Okay. And - Interesting story there 00:07:19.160 --> 00:07:26.120 that when I met John Rinaldi, I was at an OPC 00:07:26.120 --> 00:07:30.519 seminar and - I saw someone walking around with 00:07:30.519 --> 00:07:34.019 a book called The Everyman's Guide to Ethernet 00:07:34.019 --> 00:07:36.800 IP. I'm sitting there, oh, I know a little bit 00:07:36.800 --> 00:07:39.139 about that. Oh, I want to read that, see if it's 00:07:39.139 --> 00:07:42.540 worth letting people know about. Went up to the 00:07:42.540 --> 00:07:44.560 gentleman who was carrying this book, and he 00:07:44.560 --> 00:07:46.699 said, I don't know, I haven't read it yet, but 00:07:46.699 --> 00:07:49.959 go out in the lobby, and there at the real -time 00:07:49.959 --> 00:07:52.920 automation booth, you can meet a guy who's handing 00:07:52.920 --> 00:07:56.589 these books out. And I went out there, and I 00:07:56.589 --> 00:07:59.110 didn't see the book on the table. And so I asked 00:07:59.110 --> 00:08:02.170 the man, and he happened to be John Rinaldi. 00:08:02.430 --> 00:08:07.569 And he said, who are you? And I told him, I work 00:08:07.569 --> 00:08:09.949 for General Motors. Oh, I've heard of you. I've 00:08:09.949 --> 00:08:14.389 heard of you. I only have a couple copies left, 00:08:14.529 --> 00:08:17.329 but if you promise that you will write a review 00:08:17.329 --> 00:08:20.550 on Amazon, I will give you one of those copies. 00:08:21.290 --> 00:08:24.110 for you to read and hey for a free book i'm a 00:08:24.110 --> 00:08:28.329 cheapskate uh picked up the book read it that 00:08:28.329 --> 00:08:31.170 night i mean took it home read it cover to cover 00:08:31.170 --> 00:08:35.370 like 2 a .m i'm appalled i could not believe 00:08:35.370 --> 00:08:40.129 it this book did not have the first thing i learned 00:08:40.129 --> 00:08:43.769 about ethernet networks for controls In retrospect, 00:08:43.950 --> 00:08:47.049 it wasn't a bad book about the Ethernet IP spec, 00:08:47.210 --> 00:08:49.450 but it was written from a supplier's point of 00:08:49.450 --> 00:08:52.070 view. Here's how you create an Ethernet IP node. 00:08:52.889 --> 00:08:55.490 What I was with the users with General Motors 00:08:55.490 --> 00:08:58.649 is I was installing Ethernet IP networks. And 00:08:58.649 --> 00:09:00.809 so it's how you put together all these nodes 00:09:00.809 --> 00:09:03.509 from all the different companies to interop and 00:09:03.509 --> 00:09:06.590 perform a useful function. And they didn't talk 00:09:06.590 --> 00:09:09.649 at all. It called itself the everyman's guide. 00:09:10.309 --> 00:09:12.149 And it didn't have the first thing about it. 00:09:12.279 --> 00:09:16.559 Whatever you do, don't have any collisions on 00:09:16.559 --> 00:09:20.100 your Ethernet IP network. It never even mentioned 00:09:20.100 --> 00:09:23.159 that. And so I wrote a scathing review. I mean, 00:09:23.179 --> 00:09:27.019 it was, oh, it was. But I emailed it to John. 00:09:27.100 --> 00:09:31.120 I said, you made me promise I would write a review. 00:09:31.240 --> 00:09:33.700 This is the review. I will post it on Amazon 00:09:33.700 --> 00:09:38.960 if you insist that I do so. But please, I don't 00:09:38.960 --> 00:09:42.809 think. I don't think you'd appreciate it. And 00:09:42.809 --> 00:09:45.669 I got a phone call like the very next day. Oh, 00:09:45.750 --> 00:09:48.870 you can hold off on your review. Let's talk about 00:09:48.870 --> 00:09:51.830 this. And he came to town and we talked about 00:09:51.830 --> 00:09:55.509 it. And he said, hmm, okay, I see you have a 00:09:55.509 --> 00:09:58.149 different perspective. Yeah, why don't you write 00:09:58.149 --> 00:10:01.409 a chapter and I'll write an amendment. And so 00:10:01.409 --> 00:10:03.769 I spent a month trying to write a chapter to 00:10:03.769 --> 00:10:07.129 modify that book. And it just, no, it took more 00:10:07.129 --> 00:10:12.720 than a chapter to write. Also, the whole premise 00:10:12.720 --> 00:10:15.279 was, you know, it wasn't building an individual 00:10:15.279 --> 00:10:17.539 node. It was getting an interoperable network. 00:10:17.779 --> 00:10:20.980 I just came back to John and said, John, I'm 00:10:20.980 --> 00:10:25.659 sorry. I'm not so negative on your book. It's 00:10:25.659 --> 00:10:28.720 not bad describing the specification per se. 00:10:29.120 --> 00:10:31.799 It doesn't help the users. It's not an everyman's 00:10:31.799 --> 00:10:34.379 guide. And he goes, well, then we're just going 00:10:34.379 --> 00:10:37.419 to have to write another book. And so we did. 00:10:37.639 --> 00:10:42.279 And it was a wonderful experience. And it's one 00:10:42.279 --> 00:10:44.759 of the things I'm most proud of now, having captured 00:10:44.759 --> 00:10:48.580 probably 15 years of learning on the evolution 00:10:48.580 --> 00:10:51.639 of Ethernet and Ethernet for control system networks 00:10:51.639 --> 00:10:55.519 in particular. Yeah, it's amazing. I mean, so 00:10:55.519 --> 00:10:57.480 I got to be honest with you, Gary, as you're 00:10:57.480 --> 00:10:59.379 running through kind of all of this history, 00:10:59.500 --> 00:11:02.340 I'm coming up with more questions than we possibly 00:11:02.340 --> 00:11:05.559 have time for to get to on this podcast. Which 00:11:05.559 --> 00:11:07.559 means time for a second podcast. That's exactly 00:11:07.559 --> 00:11:10.259 right. We'll have to schedule another one at 00:11:10.259 --> 00:11:11.919 some point. But I want to go back to something 00:11:11.919 --> 00:11:14.620 you brought up kind of earlier on, especially 00:11:14.620 --> 00:11:18.639 because you've seen this evolution for, you know, 00:11:18.639 --> 00:11:23.340 since its early days. Everybody calls it something 00:11:23.340 --> 00:11:27.419 different. It's mentioned in some type of context. 00:11:27.759 --> 00:11:29.799 I feel like somebody the other day said it's 00:11:29.799 --> 00:11:32.120 old and tired. Everybody knows it. And that's 00:11:32.120 --> 00:11:35.799 this concept of converging the organizations 00:11:35.799 --> 00:11:38.960 or converging the technology parts of IT and 00:11:38.960 --> 00:11:44.059 OT. So when I say IT and OT convergence to you, 00:11:44.120 --> 00:11:47.440 I'm not calling out people or process. Gary, 00:11:47.500 --> 00:11:50.250 what? What does that, like, what did you do? 00:11:50.370 --> 00:11:52.870 What made it work? Where did you run into problems? 00:11:53.450 --> 00:11:56.750 Just start downloading on that a little bit. 00:11:57.029 --> 00:11:58.950 Or did it ever work, right? Or did it ever work? 00:11:59.570 --> 00:12:02.950 Let's start with the idea of ITOT convergence. 00:12:03.450 --> 00:12:05.830 I mean, I first heard that term probably about 00:12:05.830 --> 00:12:09.129 15 years ago. And it certainly seemed to come 00:12:09.129 --> 00:12:12.509 out from, I believe it was Cisco that was promoting 00:12:12.509 --> 00:12:18.759 the idea. It was IT -OT technology convergence. 00:12:19.240 --> 00:12:22.539 And I remember it so well because I'd never heard 00:12:22.539 --> 00:12:26.779 the acronym OT before. OT in my industry was 00:12:26.779 --> 00:12:32.120 overtime. OT convergence doesn't make any sense 00:12:32.120 --> 00:12:37.820 at all. And then it was, oh, okay, I'm a little 00:12:37.820 --> 00:12:41.440 skeptical. I'm a little skeptical because there's 00:12:41.440 --> 00:12:46.159 only one IT department. And there's an executive 00:12:46.159 --> 00:12:50.580 vice president in charge of IT. Try and find 00:12:50.580 --> 00:12:52.659 an executive vice president or a vice president 00:12:52.659 --> 00:12:55.220 of OT. You won't find it. There's just so many 00:12:55.220 --> 00:12:59.159 different OTs in the automotive industry. I mean, 00:12:59.179 --> 00:13:03.899 there was facilities. There was the press group. 00:13:03.980 --> 00:13:06.840 There was the paint group. There was the robotic 00:13:06.840 --> 00:13:09.720 automation group. I mean, there's all these different. 00:13:09.820 --> 00:13:12.340 There was the material handling and shipping 00:13:12.340 --> 00:13:15.220 and logistics and whatnot. So there's a variety 00:13:15.220 --> 00:13:18.100 of OT departments. Yeah, which part of production 00:13:18.100 --> 00:13:20.279 are you talking about? And which one are you 00:13:20.279 --> 00:13:22.519 talking about? Because they have different control 00:13:22.519 --> 00:13:27.149 system networks. Primarily, I was working much 00:13:27.149 --> 00:13:30.269 more with our powertrain division, and our powertrain 00:13:30.269 --> 00:13:33.070 division also had an industrial Ethernet strategy, 00:13:33.509 --> 00:13:37.029 much as we did. I was involved in some of that. 00:13:37.490 --> 00:13:41.570 But they were going about it differently than 00:13:41.570 --> 00:13:45.889 we were. And we were looking at technology, and 00:13:45.889 --> 00:13:48.769 we had an executive director that was coming 00:13:48.769 --> 00:13:54.620 up with a... control system philosophy a control 00:13:54.620 --> 00:13:57.940 system strategy where we were trying to get away 00:13:57.940 --> 00:14:00.460 from the usual way of procuring things he was 00:14:00.460 --> 00:14:04.379 saying technology is revolving so rapidly let's 00:14:04.379 --> 00:14:07.779 try and freeze technology so that we can make 00:14:07.779 --> 00:14:12.649 a a Plants identical. Four or five plants identical. 00:14:13.129 --> 00:14:16.110 Because the technology curve is so rapid, you'd 00:14:16.110 --> 00:14:18.269 put in a plant, then the next plant you'd put 00:14:18.269 --> 00:14:20.070 in, you'd use a different technology. Then the 00:14:20.070 --> 00:14:21.990 next plant you'd put in is a different technology. 00:14:22.509 --> 00:14:27.669 And if you recall back in the 80s and 90s, GM 00:14:27.669 --> 00:14:31.799 had multiple different vehicle divisions. And 00:14:31.799 --> 00:14:34.440 each division had their own control system philosophy. 00:14:34.899 --> 00:14:37.340 And so what we were doing was trying to come 00:14:37.340 --> 00:14:41.440 up with a control system strategy that was useful 00:14:41.440 --> 00:14:44.940 and appropriate across all the various vehicle 00:14:44.940 --> 00:14:47.379 divisions and whatnot. And so you're talking 00:14:47.379 --> 00:14:51.539 about like CPC, BOC, those kinds of, okay. Exactly, 00:14:51.740 --> 00:14:55.200 exactly. Small car, medium car, large car, truck, 00:14:55.480 --> 00:14:59.679 Saturn, each had their own control system philosophy. 00:15:00.879 --> 00:15:04.779 So our executive director, he put together a, 00:15:04.960 --> 00:15:08.840 he got the control, the primary control system 00:15:08.840 --> 00:15:12.100 engineers from each division, got us all in a 00:15:12.100 --> 00:15:14.179 room, says, I'm going to make you an offer you 00:15:14.179 --> 00:15:18.980 can't refuse. You have five different ways of 00:15:18.980 --> 00:15:23.519 manufacturing a vehicle, and they all work. I 00:15:23.519 --> 00:15:27.769 want you engineers to agree on the way. We are 00:15:27.769 --> 00:15:30.990 going to manufacture our vehicles, the control 00:15:30.990 --> 00:15:34.169 system architecture for all our vehicles, or 00:15:34.169 --> 00:15:35.990 I'm going to choose one of the five that I know 00:15:35.990 --> 00:15:37.809 that works, and I'm going to impose it on everyone 00:15:37.809 --> 00:15:42.309 else. So have at it, gentlemen, and gave us like 00:15:42.309 --> 00:15:45.269 three months to come up with it. And I was the 00:15:45.269 --> 00:15:47.950 person coming from the staffs that was there 00:15:47.950 --> 00:15:50.889 in terms of network support. So I was coming 00:15:50.889 --> 00:15:53.789 up with the networking architecture strategy. 00:15:53.990 --> 00:15:57.669 And we came up with, oh, okay, you have, device 00:15:57.669 --> 00:16:01.350 -level networks. You have control -level networks. 00:16:01.629 --> 00:16:04.509 You have the information technology networks. 00:16:04.690 --> 00:16:06.970 And there's a different technology for all of 00:16:06.970 --> 00:16:10.149 those. Well, the thing we were dealing with was 00:16:10.149 --> 00:16:13.350 10 different device -level networks. People were 00:16:13.350 --> 00:16:15.830 using Profibus. People were using Interbus S. 00:16:15.970 --> 00:16:18.850 People were using DeviceNet. People were, I mean, 00:16:18.870 --> 00:16:22.440 it was just, it was amazing. And so we... stopped 00:16:22.440 --> 00:16:26.379 and had a got everyone in the room all discussed 00:16:26.379 --> 00:16:29.779 all debate it took probably six months that we 00:16:29.779 --> 00:16:32.720 came up with the thing that made most sense for 00:16:32.720 --> 00:16:35.419 general motors as well as general motor suppliers 00:16:35.419 --> 00:16:37.940 because we interviewed all our suppliers as well 00:16:37.940 --> 00:16:42.460 was that we support device net and so then we 00:16:42.460 --> 00:16:44.879 started every division was going to start using 00:16:44.879 --> 00:16:48.120 device net Well, that decision alone took six 00:16:48.120 --> 00:16:52.039 months. This is too damn long. We want a whole 00:16:52.039 --> 00:16:56.179 control system architecture. And so they merged 00:16:56.179 --> 00:17:02.379 the control system groups together, and we came 00:17:02.379 --> 00:17:06.000 up with a global – well, it was North American 00:17:06.000 --> 00:17:09.059 initially, common controls architecture. Then 00:17:09.059 --> 00:17:11.799 we compared and contrasted and went over and 00:17:11.799 --> 00:17:14.759 discussed and debated it with – our Opel division 00:17:14.759 --> 00:17:17.759 in Europe and finally came up with a global strategy 00:17:17.759 --> 00:17:22.819 agreeing on span of... It was the functional 00:17:22.819 --> 00:17:25.579 needs that we were going to implement and whatnot 00:17:25.579 --> 00:17:29.079 and the functionality. When it came down to technologies, 00:17:29.440 --> 00:17:33.500 we chose control net for the control level networks, 00:17:33.940 --> 00:17:37.059 device net for the device level networks, Ethernet 00:17:37.059 --> 00:17:40.099 for the IT networks. All well and good. I mean, 00:17:40.099 --> 00:17:42.480 Ethernet was everywhere. And the question came 00:17:42.480 --> 00:17:45.470 about... Do we really need three different technologies? 00:17:45.910 --> 00:17:49.109 Can we push DeviceNet to be a control -level 00:17:49.109 --> 00:17:52.289 network? Can we bring Ethernet down to be a control 00:17:52.289 --> 00:17:54.809 -level network? Could we bring ControlNet down 00:17:54.809 --> 00:17:58.430 to be a device -level network? And I was put 00:17:58.430 --> 00:18:01.950 in the effort with that, even whenever we agreed 00:18:01.950 --> 00:18:04.390 that ControlNet would be our control -level network. 00:18:04.890 --> 00:18:07.809 It's done based on you will do a study comparing 00:18:07.809 --> 00:18:09.890 and contrasting and see if we really need to 00:18:09.890 --> 00:18:12.390 support three different technologies. And that's 00:18:12.390 --> 00:18:15.650 where I did my Ethernet studies and demonstrated 00:18:15.650 --> 00:18:19.970 that Ethernet and isolated Ethernet could outperform 00:18:19.970 --> 00:18:22.650 any one of the control system network technologies 00:18:22.650 --> 00:18:25.650 we were using at the time. And so that became 00:18:25.650 --> 00:18:29.809 the effort is how do we eliminate the need for 00:18:29.809 --> 00:18:32.250 different network technologies and collapse it 00:18:32.250 --> 00:18:36.250 just to device level networks? and the information 00:18:36.250 --> 00:18:39.289 slash control level networks, and where is the 00:18:39.289 --> 00:18:42.609 distinction there? And that then became an effort 00:18:42.609 --> 00:18:46.069 as, how do we do real -time control on Ethernet, 00:18:46.089 --> 00:18:49.069 and how do we integrate a real -time control 00:18:49.069 --> 00:18:53.490 Ethernet with the IT Ethernet networks? How much 00:18:53.490 --> 00:18:56.990 gnashing of teeth and consternation was there 00:18:56.990 --> 00:18:58.529 between the groups that were involved in that, 00:18:58.609 --> 00:19:03.500 Gary? I don't think there was an... I mean, it 00:19:03.500 --> 00:19:07.619 really was. If you thought your division would 00:19:07.619 --> 00:19:09.660 be the one the executive director would pick, 00:19:09.859 --> 00:19:12.660 you'd have fought it all the way. It really was. 00:19:13.220 --> 00:19:16.500 Everyone, I think, came together and seriously 00:19:16.500 --> 00:19:19.180 looked at, I mean, we were looking at 110 volts 00:19:19.180 --> 00:19:24.720 versus the 110 AC versus the 24 volt DC for all 00:19:24.720 --> 00:19:27.180 our sensors and actuators. Because different 00:19:27.180 --> 00:19:29.369 divisions chose different. different standards 00:19:29.369 --> 00:19:32.109 there so we agreed internationally it sure looked 00:19:32.109 --> 00:19:34.829 like the 24 volt dc standard was the way to go 00:19:34.829 --> 00:19:37.190 so everyone agreed to that so that got all our 00:19:37.190 --> 00:19:40.250 sensors in line uh we did that study we came 00:19:40.250 --> 00:19:42.769 up with and device net so everyone's going to 00:19:42.769 --> 00:19:48.460 do device net uh we were primarily Rockwell -oriented 00:19:48.460 --> 00:19:51.859 in terms of our PLCs. We're primarily PLC -oriented. 00:19:52.119 --> 00:19:55.339 There was an investigation of PC -based controls, 00:19:55.480 --> 00:19:59.240 but the early ones, I do believe the biggest 00:19:59.240 --> 00:20:04.019 issue we ran into was the obsolescence, the rapid 00:20:04.019 --> 00:20:08.240 pace and speed of the... personal computer industry 00:20:08.240 --> 00:20:11.319 they just changed operating systems they changed 00:20:11.319 --> 00:20:15.339 cpus they changed i mean it changed so rapidly 00:20:15.339 --> 00:20:20.119 you could not you know you in the amount of time 00:20:20.119 --> 00:20:25.319 it takes to install build design install a automotive 00:20:25.319 --> 00:20:30.309 factory yeah they'd be on to a new two or three 00:20:30.309 --> 00:20:33.329 levels of technology change and you could not 00:20:33.329 --> 00:20:36.630 get the spare replacement parts to maintain a 00:20:36.630 --> 00:20:40.299 system that you've got a five plus year You know, 00:20:40.299 --> 00:20:42.900 Delta on here's we want to do this. And, you 00:20:42.900 --> 00:20:44.740 know, by the time it gets implemented in a plant, 00:20:44.940 --> 00:20:48.359 it's obsolete. Yeah. You look back at that time 00:20:48.359 --> 00:20:52.880 from roughly, let's say, the early 90s to that 00:20:52.880 --> 00:20:56.480 10 year span from the early 90s to the mid aughts. 00:20:56.619 --> 00:21:00.920 How fast those operating systems hardware standards 00:21:00.920 --> 00:21:05.380 changed. It's just ridiculous. I can't I can't 00:21:05.380 --> 00:21:07.839 imagine even to your point. I think you said 00:21:07.839 --> 00:21:10.279 five plus years. I mean. most of this stuff you 00:21:10.279 --> 00:21:12.759 were you know you were designing for 10 20 minimum 00:21:12.759 --> 00:21:15.779 you see these things coming out every you know 00:21:15.779 --> 00:21:18.640 18 months 24 months my god how do you do anything 00:21:18.640 --> 00:21:21.539 with that the industrial market versus the commercial 00:21:21.539 --> 00:21:25.680 market yeah it's well understood well pc -based 00:21:25.680 --> 00:21:29.019 controls was too much dependent upon the commercial 00:21:29.019 --> 00:21:33.279 market hardware and that evolved so fast that 00:21:33.279 --> 00:21:39.200 uh uh uh and So one of the things I was investigating 00:21:39.200 --> 00:21:44.559 was Ethernet IP versus PROFINET. And we were 00:21:44.559 --> 00:21:46.819 involved with a study at the University of Michigan 00:21:46.819 --> 00:21:49.019 on some prototype products and the performance 00:21:49.019 --> 00:21:51.400 and whatnot. And the difference between them 00:21:51.400 --> 00:21:55.599 was a fraction, a tiny fraction of a millisecond 00:21:55.599 --> 00:21:59.920 in terms of performance output. And so basically, 00:22:00.839 --> 00:22:06.079 PROFINET, you can't route the real -time signal. 00:22:06.440 --> 00:22:10.319 Ethernet IP, it was routable because they used 00:22:10.319 --> 00:22:16.019 the UDP protocol to convey the control system 00:22:16.019 --> 00:22:20.019 traffic. So it was routable, and that's what 00:22:20.019 --> 00:22:24.619 led us to primarily pick Ethernet IP. It was 00:22:24.619 --> 00:22:30.359 almost a coin flip, but... That seemed to be 00:22:30.359 --> 00:22:32.480 significant enough, and I believe it's proven 00:22:32.480 --> 00:22:37.279 out to be that that's the case. You bring back 00:22:37.279 --> 00:22:40.319 a lot of memories around some of my early days 00:22:40.319 --> 00:22:42.660 stuff, which was later years, at least on the 00:22:42.660 --> 00:22:48.099 control side. out of the box in a lot of cases 00:22:48.099 --> 00:22:52.079 even on the ethernet ip side generally those 00:22:52.079 --> 00:22:55.660 ttls are set to a default of one so like out 00:22:55.660 --> 00:22:58.579 of the box in a lot of cases that traffic isn't 00:22:58.579 --> 00:23:01.599 routable unless you change it but man that that 00:23:01.599 --> 00:23:04.940 bit us in some of the early designs where you 00:23:04.940 --> 00:23:08.259 know we're expecting to be able to talk to rockwell 00:23:08.259 --> 00:23:12.180 controller and ent uh enbt card not on the same 00:23:12.750 --> 00:23:15.430 you know l3 segment and it's like why in the 00:23:15.430 --> 00:23:17.190 heck is the freaking traffic not getting over 00:23:17.190 --> 00:23:19.529 there you you know pull an ether real trace which 00:23:19.529 --> 00:23:23.109 is what it was at the time up and suddenly it's 00:23:23.109 --> 00:23:25.329 like wait a minute ttl is coming out of one it's 00:23:25.329 --> 00:23:27.609 been generated as a one what the you know what 00:23:27.609 --> 00:23:30.309 the devil's the deal with that um but to your 00:23:30.309 --> 00:23:32.690 point it's actually changeable where whereas 00:23:32.690 --> 00:23:36.599 you know it wasn't Now, we were talking about 00:23:36.599 --> 00:23:39.140 – well, okay, I just wanted to build up. We were 00:23:39.140 --> 00:23:41.859 talking about convergence. Yeah, right. Let's 00:23:41.859 --> 00:23:45.299 get back to that. Yeah, and it's – okay, so the 00:23:45.299 --> 00:23:49.000 original – whenever – it wasn't over time. I 00:23:49.000 --> 00:23:51.240 got to explain that it was operational technology. 00:23:51.980 --> 00:23:57.559 And there's an element of truth to the idea of 00:23:57.559 --> 00:24:01.450 – IT -OT technology convergence that was going 00:24:01.450 --> 00:24:04.970 on 10, 15 years ago. And that element of truth, 00:24:05.069 --> 00:24:08.910 because the reason the idea was proposed was 00:24:08.910 --> 00:24:14.190 to explain the culture gap between the IT organization 00:24:14.190 --> 00:24:18.890 and the OT organizations. And what's happening 00:24:18.890 --> 00:24:21.690 is the OT organizations are concerned with production 00:24:21.690 --> 00:24:23.829 and they're concerned with real -time control. 00:24:23.990 --> 00:24:26.589 And the IT organizations are concerned with... 00:24:26.809 --> 00:24:31.069 information technology transfer. And what happens 00:24:31.069 --> 00:24:34.589 is, yes, both organizations were looking at using 00:24:34.589 --> 00:24:37.569 switched Ethernet, but they're using it in completely 00:24:37.569 --> 00:24:44.170 and very vastly different ways. The OT organizations 00:24:44.170 --> 00:24:49.470 aren't concerned about anything other than control 00:24:49.470 --> 00:24:53.589 traffic and the other traffic, but it's primarily 00:24:53.589 --> 00:24:56.150 the control system traffic for production purposes. 00:24:56.890 --> 00:25:02.130 And the IT organization was interested in globally 00:25:02.130 --> 00:25:05.309 getting information technology, not only throughout 00:25:05.309 --> 00:25:08.430 the corporation, but between corporations and 00:25:08.430 --> 00:25:10.309 your customers, between corporations and your 00:25:10.309 --> 00:25:14.089 suppliers. I mean, they are interested in routing 00:25:14.089 --> 00:25:18.289 globally. Whereas a control system network, geez, 00:25:18.509 --> 00:25:23.029 the idea of the network is to replace a physical 00:25:23.029 --> 00:25:27.049 wire that used to have those bundles of wires. 00:25:27.819 --> 00:25:29.819 connecting all the different pieces of equipment. 00:25:30.059 --> 00:25:32.240 And now you wanted to replace that with a network. 00:25:32.420 --> 00:25:35.039 And you were dealing with the idea of a local 00:25:35.039 --> 00:25:37.799 area network and a true local area network. Whereas 00:25:37.799 --> 00:25:39.940 with IT, they're using the local area network 00:25:39.940 --> 00:25:42.839 to access the internet and access the wide area 00:25:42.839 --> 00:25:45.980 network. And so if you take a look at it, I mean, 00:25:45.980 --> 00:25:51.460 one of the huge differences is 90 plus percent 00:25:51.460 --> 00:25:55.500 of the traffic on an OT control network stays 00:25:55.500 --> 00:25:59.160 within the network. Whereas with an IT network, 00:25:59.420 --> 00:26:03.380 90 % of the local traffic is going to servers 00:26:03.380 --> 00:26:08.759 and the Internet or another country, e -mails. 00:26:08.759 --> 00:26:12.819 I mean, off -net. And so it is really an access 00:26:12.819 --> 00:26:16.099 to the outside world is the way they were using 00:26:16.099 --> 00:26:22.509 switched Ethernet. And so. It really wasn't. 00:26:22.509 --> 00:26:24.650 They were using the same technology, but they're 00:26:24.650 --> 00:26:28.130 using in vastly different ways. And it helped 00:26:28.130 --> 00:26:30.890 explain why there was that divergence and whatnot. 00:26:31.190 --> 00:26:34.549 Well, all of a sudden it became so common that 00:26:34.549 --> 00:26:36.970 I noticed about five years ago, people started 00:26:36.970 --> 00:26:40.170 dropping technology and they stopped using it 00:26:40.170 --> 00:26:42.549 as an explanation for why there's a culture gap. 00:26:42.710 --> 00:26:46.410 And it became a proposed solution for resolving 00:26:46.410 --> 00:26:50.289 the culture gap. And now. I think that's where 00:26:50.289 --> 00:26:54.549 we've gone too far, and it's time to stand up 00:26:54.549 --> 00:26:57.950 and say it was a slightly flawed narrative. When 00:26:57.950 --> 00:27:02.230 you try to use it as a solution to a legitimate 00:27:02.230 --> 00:27:07.329 issue, it becomes even more of a flawed narrative. 00:27:07.549 --> 00:27:11.289 And what you're trying to do is get IT and OT 00:27:11.289 --> 00:27:16.160 collaboration. Not convergence. Yeah, that's 00:27:16.160 --> 00:27:19.460 a great point because I think we often misuse 00:27:19.460 --> 00:27:23.380 that collaboration versus convergence word. I 00:27:23.380 --> 00:27:26.359 think collaboration is better because OT is going 00:27:26.359 --> 00:27:28.279 to still have to do what OT does because you're 00:27:28.279 --> 00:27:31.200 making your money in OT. You're not making it 00:27:31.200 --> 00:27:34.480 in IT. You're producing product in the OT environment. 00:27:34.740 --> 00:27:38.059 And downtime, quite frankly, downtime is a killer. 00:27:39.259 --> 00:27:42.740 When people are talking about convergence as 00:27:42.740 --> 00:27:46.099 a solution, what they start looking for is what 00:27:46.099 --> 00:27:49.880 is that converged solution? And so they're sitting 00:27:49.880 --> 00:27:53.680 here trying to eliminate the differences between 00:27:53.680 --> 00:27:57.099 them because I'm going to a solution for both. 00:27:57.279 --> 00:28:10.839 You're absolutely right. joint needs and you're 00:28:10.839 --> 00:28:15.220 not trying to sit here and, you know. Who owns 00:28:15.220 --> 00:28:17.359 the data? Who owns the equipment? I mean, you 00:28:17.359 --> 00:28:19.640 got that back and forth finger pointing contest. 00:28:19.819 --> 00:28:23.299 I mean, that is so crazy. Well, and it just, 00:28:23.380 --> 00:28:28.500 it killed me. One of my experiences is, I mean, 00:28:28.500 --> 00:28:31.480 we had an IT department that called themselves 00:28:31.480 --> 00:28:35.880 the real time control group. And what they ended 00:28:35.880 --> 00:28:38.839 up being was the scheduling group. They were 00:28:38.839 --> 00:28:41.059 responsible for putting out a production schedule. 00:28:43.140 --> 00:28:46.079 And we would just laugh at them. They seemed 00:28:46.079 --> 00:28:49.119 to think that providing a schedule gave them 00:28:49.119 --> 00:28:52.039 a degree of control over what was actually happening 00:28:52.039 --> 00:28:55.160 on the plant floor. Now, you try to meet the 00:28:55.160 --> 00:28:59.859 schedule, but reality... Reality exists. And 00:28:59.859 --> 00:29:02.720 I'm sorry, you telling me to produce that whenever 00:29:02.720 --> 00:29:05.440 the machine is down, I ain't producing it. You 00:29:05.440 --> 00:29:09.420 don't control a damn thing. Right. Your definitions 00:29:09.420 --> 00:29:12.559 around the difference between convergence and 00:29:12.559 --> 00:29:14.920 collaboration, man, I feel like we could have 00:29:14.920 --> 00:29:17.920 a whole separate podcast just on the cultural. 00:29:18.200 --> 00:29:19.960 You touched on a number of things that I think 00:29:19.960 --> 00:29:23.299 are so critical. And that is this is always a 00:29:23.299 --> 00:29:26.400 we us. That's the reason you're having the discussions. 00:29:27.119 --> 00:29:31.980 It's not a one group or one process. It's not 00:29:31.980 --> 00:29:35.000 an I or me thing. It's a we thing. That's right. Exactly. 00:29:35.299 --> 00:29:38.519 Yeah, it's a we -us thing. It's a we -us. Yes, 00:29:38.559 --> 00:29:43.859 the competitor was Toyota, not my IT department. 00:29:44.039 --> 00:29:47.819 Exactly, exactly. Well said, well said. Yes, 00:29:47.920 --> 00:29:51.859 spot on, Gary. Yeah, and you know, John, I mean, 00:29:51.859 --> 00:29:53.640 just going back to like, you know, what we do 00:29:53.640 --> 00:29:56.500 on a daily basis, I literally want to almost, 00:29:56.700 --> 00:29:59.259 you know, maybe massage our talk track a little 00:29:59.259 --> 00:30:03.289 bit because I don't think we do. double click 00:30:03.289 --> 00:30:05.750 enough to use one of our favorite favorite colleagues 00:30:05.750 --> 00:30:08.569 terms i don't think we double click enough on 00:30:08.569 --> 00:30:12.690 making sure that we definition set what that 00:30:12.690 --> 00:30:14.789 you know what because i think sometimes we started 00:30:14.789 --> 00:30:17.009 using convergence i think now we're using partnership 00:30:17.009 --> 00:30:21.829 which is better um but they're still okay great 00:30:21.829 --> 00:30:24.230 you got you know both teams need to be talking 00:30:24.230 --> 00:30:27.309 but double clicking a little bit more on exactly 00:30:27.309 --> 00:30:29.730 when we we're doing this, we're talking about 00:30:29.730 --> 00:30:32.410 doing something on the technology side. What 00:30:32.410 --> 00:30:35.250 does that really look like? It's not about their 00:30:35.250 --> 00:30:38.130 standards or their standards or what they want 00:30:38.130 --> 00:30:41.029 or what they want. It's about figuring out the 00:30:41.029 --> 00:30:43.650 right thing for the company that is what you 00:30:43.650 --> 00:30:46.450 both want and figuring out. Here are my needs. 00:30:46.589 --> 00:30:49.589 Here are your needs. How do we find a way to 00:30:49.589 --> 00:30:51.750 make those things work together? And if we can't, 00:30:51.750 --> 00:30:55.490 then let's agree that we can't. Right. I don't 00:30:55.490 --> 00:31:00.609 think we can't. I really, I mean, I then extend 00:31:00.609 --> 00:31:04.029 and extrapolate this into the security arena 00:31:04.029 --> 00:31:10.769 where IT is responsible for securing the integrity 00:31:10.769 --> 00:31:14.549 and the data and the trade secrets and whatnot. 00:31:14.750 --> 00:31:18.430 But if you take a look at the data that... OT 00:31:18.430 --> 00:31:21.349 in a control system network is dealing with, 00:31:21.430 --> 00:31:23.569 they're dealing with control signal traffic. 00:31:24.049 --> 00:31:28.569 I mean, in the Ethernet IP specification, it's 00:31:28.569 --> 00:31:31.549 called implicit message traffic. It's an implicit 00:31:31.549 --> 00:31:35.289 message. Take a look at the definition. It's, 00:31:35.289 --> 00:31:39.490 I understand it, you understand it already. It's 00:31:39.490 --> 00:31:42.519 not... context independent and it doesn't have 00:31:42.519 --> 00:31:45.240 to be you know it's it's not shared throughout 00:31:45.240 --> 00:31:50.400 the world and i think where we run into the biggest 00:31:50.400 --> 00:31:54.140 issue with security is now what we're trying 00:31:54.140 --> 00:31:58.240 to do is get get contextualized data out of the 00:31:58.240 --> 00:32:02.160 control system networks now that has to be secured 00:32:03.130 --> 00:32:08.930 But it cannot be done so. It can't be acquired 00:32:08.930 --> 00:32:11.769 or it can't be acquired intelligently at the 00:32:11.769 --> 00:32:15.349 impact of the control signal traffic that has 00:32:15.349 --> 00:32:18.369 to go on independent of it. Yeah, I mean, this 00:32:18.369 --> 00:32:21.329 is this is very much under, you know, and I think 00:32:21.329 --> 00:32:24.210 some manufacturers and some technology vendors 00:32:24.210 --> 00:32:27.869 do it different. But I'll pick on the, you know, 00:32:27.869 --> 00:32:29.609 the idea because this is the first thing that 00:32:29.609 --> 00:32:33.509 popped in my mind. There is concepts of firewalls 00:32:33.509 --> 00:32:37.690 failing open or failing closed. And meaning that 00:32:37.690 --> 00:32:41.160 when, to your point, it. impacts the controls 00:32:41.160 --> 00:32:43.759 traffic. Which one's worse? Right, exactly. Which 00:32:43.759 --> 00:32:45.920 one's worse? And in this case, in a lot of cases, 00:32:46.000 --> 00:32:47.559 it depends. This is the reason why security, 00:32:47.660 --> 00:32:50.200 you know, security in depth and the Swiss cheese 00:32:50.200 --> 00:32:53.140 model is so important because having something 00:32:53.140 --> 00:32:57.019 down there that you can leverage, but putting 00:32:57.019 --> 00:33:00.680 it in a fail open scenario at the edge so that, 00:33:00.680 --> 00:33:03.019 you know, the number one care about is still 00:33:03.019 --> 00:33:05.119 keep the process running, keep the line running, 00:33:05.180 --> 00:33:07.079 keep the plant running, because that's still 00:33:07.079 --> 00:33:10.529 that number one care about. Well, I got into 00:33:10.529 --> 00:33:15.470 several arguments with our security experts where 00:33:15.470 --> 00:33:19.230 they were saying, tell me what your problem is 00:33:19.230 --> 00:33:22.549 and we'll tell you the solution. And it's just 00:33:22.549 --> 00:33:25.410 kind of like, that's not good enough. I have 00:33:25.410 --> 00:33:28.230 to be guaranteed that your solution isn't worse 00:33:28.230 --> 00:33:31.769 than my problem. And frankly, at the time, my 00:33:31.769 --> 00:33:36.450 number one problem was you, IT, sending virus 00:33:36.450 --> 00:33:39.450 protection software. To the limited number of 00:33:39.450 --> 00:33:42.309 PCs on the playing floor. So how do you stop 00:33:42.309 --> 00:33:46.549 yourselves, please? Yep. I have a story. They 00:33:46.549 --> 00:33:49.690 did not appreciate being told that. I'm sure 00:33:49.690 --> 00:33:52.230 not. Back in about 20, this would have probably 00:33:52.230 --> 00:33:54.609 been, I have to get the dates right, sometime 00:33:54.609 --> 00:33:58.529 between probably 2010 and 2011, we're doing some. 00:33:58.970 --> 00:34:03.849 line runoff validation testing. And we're running, 00:34:03.930 --> 00:34:06.690 not maybe at rate, but we're getting in a groove. 00:34:06.809 --> 00:34:09.230 It's all looking good. This was essentially during 00:34:09.230 --> 00:34:13.050 my old stomping crowns back in my Harley days 00:34:13.050 --> 00:34:15.929 when we essentially deployed Ethernet to the 00:34:15.929 --> 00:34:18.969 factory floor 10 years after you guys did it, 00:34:18.989 --> 00:34:23.610 Gary. I kid you not, I could see this rolling, 00:34:23.829 --> 00:34:25.710 I'm going to call it a rolling blackout. It started 00:34:25.710 --> 00:34:29.630 over at online and began moving across the factory. 00:34:29.809 --> 00:34:32.449 All these heartbeats were being lost. Heartbeat 00:34:32.449 --> 00:34:34.269 lost in that corner. Then it would move over 00:34:34.269 --> 00:34:36.730 here. Then it would move over here. And, of course, 00:34:36.730 --> 00:34:38.070 everybody's like, what's going on? You know, 00:34:38.110 --> 00:34:40.949 the network. It's the network. So I'll cut the 00:34:40.949 --> 00:34:46.309 story short and say it was a security team doing 00:34:46.309 --> 00:34:51.500 port scanning. The controllers simply couldn't 00:34:51.500 --> 00:34:54.519 handle it. It was overloading the comms side 00:34:54.519 --> 00:34:56.579 of the controllers and tipping the controllers 00:34:56.579 --> 00:35:00.760 over. To your point, it's like, what do you remember 00:35:00.760 --> 00:35:03.139 afterwards? You don't remember all the times 00:35:03.139 --> 00:35:05.380 they saved you. You remember the one time they 00:35:05.380 --> 00:35:10.940 took the plant down. They insisted that we be 00:35:10.940 --> 00:35:18.019 able to route traffic up to where... where they 00:35:18.019 --> 00:35:21.920 could analyze it, as opposed to coming out to 00:35:21.920 --> 00:35:24.480 the plant floor and plugging into the equipment 00:35:24.480 --> 00:35:28.599 and analyzing it locally. And the idea, I found 00:35:28.599 --> 00:35:32.539 out about this feature that they were wanting 00:35:32.539 --> 00:35:38.440 to insist that the OT people implement by a six 00:35:38.440 --> 00:35:40.960 -hour outage at the factory where they forgot 00:35:40.960 --> 00:35:47.599 to reset the port mirroring. that was going on 00:35:47.599 --> 00:35:51.039 to troubleshoot. And that caused a traffic overload 00:35:51.039 --> 00:35:54.500 in a segment and caused equipment to stop functioning. 00:35:54.780 --> 00:35:57.099 And it was just kind of like, and you expect 00:35:57.099 --> 00:36:01.619 me to put something in that has proven to halt 00:36:01.619 --> 00:36:06.239 production? Yeah, that's, you'll never live that 00:36:06.239 --> 00:36:12.239 down if you do. Yes, there really is a culture 00:36:12.239 --> 00:36:14.880 gap. There's not an appreciation of the legitimate 00:36:14.880 --> 00:36:18.960 needs of each. And just because it's called Ethernet, 00:36:19.019 --> 00:36:22.940 there really is the ability to get a lot more 00:36:22.940 --> 00:36:25.440 data off the plant floor. But you have to do 00:36:25.440 --> 00:36:28.059 it in an intelligent and proper fashion. And 00:36:28.059 --> 00:36:30.539 you've got to prioritize the data. The control 00:36:30.539 --> 00:36:34.070 traffic must get through. I mean, that's the 00:36:34.070 --> 00:36:37.409 reason the network is out there when it's a control 00:36:37.409 --> 00:36:40.429 system network. And getting data off of that 00:36:40.429 --> 00:36:44.010 network is secondary and of distant secondary 00:36:44.010 --> 00:36:47.829 importance. So I got a question for you. Since 00:36:47.829 --> 00:36:51.550 we're talking about the data on the factory floor, 00:36:51.789 --> 00:36:56.309 what's your perspective on AI and what do you 00:36:56.309 --> 00:36:59.289 think that's going to do for industry and just 00:36:59.289 --> 00:37:06.199 your thoughts there? I think AI requires contextualized 00:37:06.199 --> 00:37:10.320 data. That's the starting point. And so you're 00:37:10.320 --> 00:37:14.159 talking about non -real -time data. And so as 00:37:14.159 --> 00:37:16.860 long as it is properly handled as non -real -time 00:37:16.860 --> 00:37:22.000 data, I don't have a problem with trying to get 00:37:22.000 --> 00:37:25.550 the data from the floor. out to support the AI 00:37:25.550 --> 00:37:30.070 type. But right now, I think we're still a little 00:37:30.070 --> 00:37:37.530 too... We're not properly treating the... We're 00:37:37.530 --> 00:37:40.769 just adding data thinking that bandwidth alone 00:37:40.769 --> 00:37:43.070 solves the problem. Oh, I'm going to go up to 00:37:43.070 --> 00:37:45.849 gigabit networks. No, you got to sit here and 00:37:45.849 --> 00:37:47.929 properly prioritize the data. You got to have 00:37:47.929 --> 00:37:51.989 the proper understanding of... what data is needed 00:37:51.989 --> 00:37:55.889 and when it's needed, when it impacts the control 00:37:55.889 --> 00:37:59.869 system networks. And that was always my perspective, 00:37:59.869 --> 00:38:04.690 was the control system. I have some real -time, 00:38:04.750 --> 00:38:07.949 hard real -time or soft real -time operation 00:38:07.949 --> 00:38:12.130 going on. Well, the contextualized data that 00:38:12.130 --> 00:38:17.570 you try to acquire from a system, put in a second 00:38:17.570 --> 00:38:21.480 network. if you aren't sure that it is not going 00:38:21.480 --> 00:38:24.139 to impact the control system traffic and behaviors. 00:38:26.260 --> 00:38:28.599 And that was the one thing, I mean, Ethernet, 00:38:28.599 --> 00:38:32.340 it sits there, it works until it doesn't. I mean, 00:38:32.380 --> 00:38:36.380 it can have a lot of... flaws and still function 00:38:36.380 --> 00:38:38.840 it doesn't have to be healthy to function it 00:38:38.840 --> 00:38:42.320 it tries to uh operate even whenever there's 00:38:42.320 --> 00:38:46.099 uh you know so you what but with control data 00:38:46.099 --> 00:38:48.800 you have to be sure that under the worst possible 00:38:48.800 --> 00:38:52.739 circumstances so so monitor your control system 00:38:52.739 --> 00:38:56.639 network make it healthy and properly prioritize 00:38:56.639 --> 00:39:01.280 contextualized data but it is okay if you get 00:39:01.280 --> 00:39:04.059 a busy signal don't sit there and now try and 00:39:04.510 --> 00:39:06.929 Push through. Find out why you got a busy signal. 00:39:08.150 --> 00:39:10.550 Don't brute force it. Right. Right. Don't brute 00:39:10.550 --> 00:39:13.730 force it. So, Gary, you know, in your mind, then 00:39:13.730 --> 00:39:18.309 you wouldn't even start looking at. leveraging 00:39:18.309 --> 00:39:22.190 AI to do any kind of real time tweaking within 00:39:22.190 --> 00:39:25.510 the process. You essentially are like, look, 00:39:25.610 --> 00:39:28.949 let's leverage these models for looking at historical. 00:39:29.610 --> 00:39:32.369 Let's look at to try and uncover insights that 00:39:32.369 --> 00:39:35.210 we didn't have before based on training it to 00:39:35.210 --> 00:39:38.909 better understand. you know, pre -failure maintenance 00:39:38.909 --> 00:39:42.010 is needed scenarios and future preventive maintenance 00:39:42.010 --> 00:39:45.849 scenarios, but kind of keep it out of the control 00:39:45.849 --> 00:39:49.550 loop is that use the controllers the way they 00:39:49.550 --> 00:39:52.829 have been. Don't try and think that, you know, 00:39:52.829 --> 00:39:55.309 you can tweak any of this stuff live and that 00:39:55.309 --> 00:39:57.690 anybody can do it better than kind of we originally 00:39:57.690 --> 00:40:02.460 set it up for. And it sure seems like most of 00:40:02.460 --> 00:40:04.179 this, you're trying to get the data. What you're 00:40:04.179 --> 00:40:07.159 trying to do is improve the next product, not 00:40:07.159 --> 00:40:09.820 the current one. It's the control is working. 00:40:09.960 --> 00:40:13.000 The control system knows about the one I just 00:40:13.000 --> 00:40:14.980 produced, the one I'm in the process of producing, 00:40:15.099 --> 00:40:18.860 and the very next one to be produced. And that 00:40:18.860 --> 00:40:21.840 is the historical nature of a control system 00:40:21.840 --> 00:40:26.840 network. And if you're trying to, you know, if 00:40:26.840 --> 00:40:30.150 what you're trying to do is... solve future production, 00:40:30.269 --> 00:40:32.530 improve future production. You're not dealing 00:40:32.530 --> 00:40:35.869 with real -time control in the control system 00:40:35.869 --> 00:40:40.909 scheme of things. Yeah, got it. Go ahead, John. 00:40:41.710 --> 00:40:44.010 Yeah, no worries. I was just going to say, what 00:40:44.010 --> 00:40:47.469 Dave and I often talk about with our teams, because 00:40:47.469 --> 00:40:49.889 everybody's talking about AI right now, us, customers, 00:40:50.150 --> 00:40:53.909 and so we try to talk about make sure you understand 00:40:53.909 --> 00:40:57.280 what the business needs are, the outcome you're 00:40:57.280 --> 00:41:01.559 trying to have before you say, let's put an AI 00:41:01.559 --> 00:41:03.559 strategy together, because you may not need one. 00:41:03.639 --> 00:41:05.940 Things may be fine today as things stand. And 00:41:05.940 --> 00:41:08.300 so we've got to be really careful. That's a really 00:41:08.300 --> 00:41:11.659 slippery slope, in my personal opinion, on going 00:41:11.659 --> 00:41:13.360 too far down that path without really understanding 00:41:13.360 --> 00:41:14.280 what's going on. Yeah, don't do it just because 00:41:14.280 --> 00:41:16.699 it's the buzzword, right? Don't do it just because 00:41:16.699 --> 00:41:18.719 everybody else is saying we need to do it because 00:41:18.719 --> 00:41:21.099 it's 12 of the 16 squares on the bingo card. 00:41:21.610 --> 00:41:26.210 Again, I start with it requires contextualized 00:41:26.210 --> 00:41:32.150 data. And contextualized data is not control 00:41:32.150 --> 00:41:37.650 signal data. So properly intermix control signal 00:41:37.650 --> 00:41:41.150 data with contextualized data and don't try to 00:41:41.150 --> 00:41:44.969 push it beyond that. If you start impacting your 00:41:44.969 --> 00:41:49.199 control signal traffic, you're... You're going 00:41:49.199 --> 00:41:53.420 to regret it. Yeah. Yeah, I hadn't really thought 00:41:53.420 --> 00:41:55.380 about it in that context. Here we're talking 00:41:55.380 --> 00:41:57.659 about contextualized data, and I'm going to say 00:41:57.659 --> 00:41:59.679 the word context, but I hadn't really thought 00:41:59.679 --> 00:42:02.719 of it that way, Gary, and I think your definition 00:42:02.719 --> 00:42:06.940 is brilliant. I love it. Yeah, I'm picturing, 00:42:07.019 --> 00:42:09.860 to your point, John, I think of everything inside 00:42:09.860 --> 00:42:13.820 the control loop right now as control data, and 00:42:13.820 --> 00:42:16.840 that we might be under certain circumstances. 00:42:17.929 --> 00:42:21.170 maybe live tweaking that with some model that 00:42:21.170 --> 00:42:24.429 we deploy. Yeah. Right. Yeah. What Gary's essentially 00:42:24.429 --> 00:42:28.769 saying is no, that's the control side of the 00:42:28.769 --> 00:42:32.869 controller, not the data that I'm sharing or 00:42:32.869 --> 00:42:35.489 that I'm leveraging to make a control system 00:42:35.489 --> 00:42:40.250 decision. It's still separate. Yeah. I'm with 00:42:40.250 --> 00:42:43.190 you, John, kind of maybe mind blown a little 00:42:43.190 --> 00:42:45.250 bit there is that I don't, I generally don't 00:42:45.250 --> 00:42:48.250 think of it that way and we should. I think we 00:42:48.250 --> 00:42:50.389 should. Absolutely. I love it, Gary. Thank you 00:42:50.389 --> 00:42:52.710 for that perspective. I really, really think 00:42:52.710 --> 00:42:54.909 that kind of sets my head in the right direction. 00:42:55.269 --> 00:42:59.110 I want to go back to I love history lessons, 00:42:59.449 --> 00:43:02.550 Gary. I love why certain decisions were made, 00:43:02.650 --> 00:43:05.670 why you did it one way versus another way, all 00:43:05.670 --> 00:43:09.550 those kinds of things. So I want to go back to 00:43:09.550 --> 00:43:12.849 the kind of your anti -Ethernet days for a little 00:43:12.849 --> 00:43:16.599 bit and kind of sort of the flip. Was there a 00:43:16.599 --> 00:43:21.360 specific thing, some, you know, the suddenly 00:43:21.360 --> 00:43:24.579 100 megabit interfaces, just the fact that suddenly 00:43:24.579 --> 00:43:26.880 we didn't have shared mediums anymore. So the 00:43:26.880 --> 00:43:29.619 collision sense multiple access part kind of 00:43:29.619 --> 00:43:31.800 went away once the, you know, the switch came 00:43:31.800 --> 00:43:35.380 about. What was the thing that pushed it from, 00:43:35.460 --> 00:43:39.179 there is no freaking way it belongs in my control 00:43:39.179 --> 00:43:42.599 systems on the factory floor. And wow, it's cheap, 00:43:42.639 --> 00:43:47.139 fast, and pretty good. It is a collision -free 00:43:47.139 --> 00:43:51.300 Ethernet environment. That was the thing. But 00:43:51.300 --> 00:43:56.380 Ethernet was defined by its collision behavior. 00:43:57.199 --> 00:43:59.940 Collisions were an inherent fundamental part 00:43:59.940 --> 00:44:02.260 of the early Ethernet. That's right. And the 00:44:02.260 --> 00:44:05.840 behavior defined the performance. And the performance 00:44:05.840 --> 00:44:09.840 just, as traffic increased, the performance peaked, 00:44:09.840 --> 00:44:13.760 then dropped off. And it was, I mean, right whenever 00:44:13.760 --> 00:44:16.539 you need to network the most, everyone's screaming 00:44:16.539 --> 00:44:22.539 for alarms and no traffic gets through. So it 00:44:22.539 --> 00:44:25.880 was establishing a collision -free environment 00:44:25.880 --> 00:44:29.300 because that was the behavior that was so objectionable. 00:44:29.820 --> 00:44:33.719 And that required switched full duplexed Ethernet. 00:44:34.579 --> 00:44:38.179 Yeah, so that was the big thing. Gary, do you 00:44:38.179 --> 00:44:42.639 remember some of your... like the old test days. 00:44:42.659 --> 00:44:46.000 I'd love to hear some history of the testing, 00:44:46.340 --> 00:44:48.900 what you brought in, what worked, what didn't, 00:44:48.900 --> 00:44:52.300 what blew your mind, any kind of war stories 00:44:52.300 --> 00:44:55.239 from kind of that. And I got a question around 00:44:55.239 --> 00:44:57.340 that. I mean, was this done in the lean lab at 00:44:57.340 --> 00:45:00.199 GM, or did you do real -world plant stuff or 00:45:00.199 --> 00:45:03.699 a little bit of both? Oh, it certainly was a 00:45:03.699 --> 00:45:08.760 lot of both. We would test it in the lab. Then 00:45:08.760 --> 00:45:14.019 we would pilot test it. And then we would test 00:45:14.019 --> 00:45:18.159 again for endurance and durability. So stress 00:45:18.159 --> 00:45:21.719 test it. And then we would pilot it. And after 00:45:21.719 --> 00:45:25.480 a successful six -month pilot, it was deemed 00:45:25.480 --> 00:45:30.519 ready for production purposes. So an awful lot 00:45:30.519 --> 00:45:34.039 of that was done. I did my earliest testing. 00:45:34.679 --> 00:45:40.539 on using shared Ethernet because the switches 00:45:40.539 --> 00:45:45.619 were so expensive back then. But it was done 00:45:45.619 --> 00:45:48.280 in the realization that switches were coming 00:45:48.280 --> 00:45:52.800 and the cost curve was going to make switches 00:45:52.800 --> 00:45:55.760 be cost comparable to the shared media hubs. 00:45:55.960 --> 00:45:59.340 So I... emulated the performance of a switch 00:45:59.340 --> 00:46:02.679 network on a shared media isolated network, just 00:46:02.679 --> 00:46:04.679 assured that there was no background traffic 00:46:04.679 --> 00:46:07.780 and didn't tie it into the outside world. It 00:46:07.780 --> 00:46:10.739 was exclusively the control system traffic on 00:46:10.739 --> 00:46:14.280 there. Then we went and we added all sorts of 00:46:14.280 --> 00:46:19.000 traffic and found that under our operating conditions, 00:46:19.260 --> 00:46:27.719 we got better performance on a... isolated Ethernet 00:46:27.719 --> 00:46:32.199 network than we did under the, it was RIO at 00:46:32.199 --> 00:46:34.760 the time, the token passing network that virtually 00:46:34.760 --> 00:46:39.199 90 % of the vehicle production we were doing 00:46:39.199 --> 00:46:43.920 was utilizing. And so demonstrating that under 00:46:43.920 --> 00:46:46.239 the worst possible conditions we could conceive 00:46:46.239 --> 00:46:49.440 of traffic being on that particular segment, 00:46:49.820 --> 00:46:55.340 Ethernet always outperformed. And then we proceeded 00:46:55.340 --> 00:46:59.559 to go about and do the same thing with Ethernet 00:46:59.559 --> 00:47:01.760 versus DeviceNet. Then we did Ethernet versus 00:47:01.760 --> 00:47:07.719 ControlNet. And with the Ethernet IP implementations 00:47:07.719 --> 00:47:11.139 and the performance tests that we were doing, 00:47:11.219 --> 00:47:15.239 demonstrated that now you had a real issue back 00:47:15.239 --> 00:47:17.780 then that people had to properly implement the 00:47:17.780 --> 00:47:21.849 TCP IP protocols. So some of the testing we did 00:47:21.849 --> 00:47:25.469 with the PC -based controllers, you would find 00:47:25.469 --> 00:47:29.010 the PC -based controllers where the control algorithm 00:47:29.010 --> 00:47:34.389 was executing under the real -time kernel of 00:47:34.389 --> 00:47:38.289 the real -time operating system on the PC, but 00:47:38.289 --> 00:47:42.150 the Ethernet communication was running under 00:47:42.150 --> 00:47:50.659 a shell of the DOS or... commercial operating 00:47:50.659 --> 00:47:52.980 system so the real -time operating system was 00:47:52.980 --> 00:47:56.440 time slicing and the ethernet was on a low priority 00:47:56.440 --> 00:48:00.260 and so you'd be trying to get control traffic 00:48:00.260 --> 00:48:05.559 out of a controller and it would get time sliced 00:48:05.559 --> 00:48:09.360 out and parsed and so you'd get unacceptable 00:48:09.360 --> 00:48:15.030 behavior So you needed to make sure your TCP 00:48:15.030 --> 00:48:18.809 IP protocol stack was properly implemented. So 00:48:18.809 --> 00:48:20.829 we came up with some performance tests with that, 00:48:20.929 --> 00:48:24.889 and we actually shared our performance test methodology 00:48:24.889 --> 00:48:29.070 with the ODVA organization, the owners of the 00:48:29.070 --> 00:48:33.530 Ethernet IP spec, to make sure that they would 00:48:33.530 --> 00:48:36.369 perform those type of tests whenever they were 00:48:36.369 --> 00:48:42.780 validating Ethernet IP implementations. Transport 00:48:42.780 --> 00:48:46.139 and network layers of the protocol were controlled 00:48:46.139 --> 00:48:50.960 by a real -time operating system kernel that 00:48:50.960 --> 00:48:54.360 was given appropriate priority. So was that back 00:48:54.360 --> 00:48:56.940 in the FlowPro days, Gary, or was that after 00:48:56.940 --> 00:48:59.400 that? That was slightly after the FlowPro days. 00:49:00.219 --> 00:49:02.260 Okay. Yeah, I'm familiar with that from my time 00:49:02.260 --> 00:49:04.940 at GE Fanuc and our cooperation with them at 00:49:04.940 --> 00:49:09.659 GM Powertrain. I mean, the flip to switching 00:49:09.659 --> 00:49:13.019 in hardware from just software -based switching 00:49:13.019 --> 00:49:16.219 did so much for that because suddenly you could 00:49:16.219 --> 00:49:20.500 essentially guarantee that there was space in 00:49:20.500 --> 00:49:25.900 that frame buffer, in that packet buffer. Even 00:49:25.900 --> 00:49:28.599 if the switch was busy doing something else with 00:49:28.599 --> 00:49:32.019 its CPU cycles, you were guaranteed that it would 00:49:32.019 --> 00:49:35.000 switch first and then do everything else later. 00:49:35.469 --> 00:49:37.210 And man, that helped it. You know, talk about 00:49:37.210 --> 00:49:39.210 the importance of determinism. It certainly was. 00:49:39.349 --> 00:49:41.429 It certainly was. I mean, way back then, I mean, 00:49:41.449 --> 00:49:44.110 we were doing some and some of the some of the 00:49:44.110 --> 00:49:46.329 switches weren't weren't non -blocking switches. 00:49:46.989 --> 00:49:52.210 I mean, you had the switch itself could get overloaded 00:49:52.210 --> 00:49:54.869 in the early days of switching. Absolutely. You 00:49:54.869 --> 00:49:58.170 had to make sure that the hardware you were purchasing 00:49:58.170 --> 00:50:02.650 for an industrial grade switch was of line speed, 00:50:02.730 --> 00:50:05.880 non -blocking. Remember that. Technology was 00:50:05.880 --> 00:50:09.460 being used. Yep. Well, if you can believe this, 00:50:09.539 --> 00:50:13.119 Gary, we are heading to the end of our hour. 00:50:13.159 --> 00:50:15.320 I don't have any idea how we're already here. 00:50:15.880 --> 00:50:19.380 I know, right? I'm crazy, right, John? Would 00:50:19.380 --> 00:50:22.820 love to wrap. So I just got kind of a question, 00:50:23.039 --> 00:50:26.800 Gary, and that is, if you had... I don't know, 00:50:26.840 --> 00:50:29.679 the top three things you think about your, would 00:50:29.679 --> 00:50:33.219 you say, 48 year career in the business? You 00:50:33.219 --> 00:50:35.679 learn so much. You put a bunch of it in a book. 00:50:35.840 --> 00:50:38.320 You brought a bunch of it up earlier on the on 00:50:38.320 --> 00:50:41.460 the podcast here. What are like the three things 00:50:41.460 --> 00:50:44.559 people like if you listen to me and you learn 00:50:44.559 --> 00:50:47.800 these three things, your life in this space will 00:50:47.800 --> 00:50:51.139 be easier. What are those three things? I am 00:50:51.139 --> 00:50:54.380 a reformed. And if you're not looking at Ethernet 00:50:54.380 --> 00:50:57.019 for your control system networks, you're missing 00:50:57.019 --> 00:50:59.800 the boat. You got to properly segment your Ethernet 00:50:59.800 --> 00:51:03.340 networks. Security is paramount today. I mean, 00:51:03.360 --> 00:51:06.500 that's that's. Yeah. Look at Ethernet. Make sure 00:51:06.500 --> 00:51:12.239 it's secure. And OK, be don't don't be quite 00:51:12.239 --> 00:51:17.019 as hard on your IT people as I was. I love it. 00:51:17.119 --> 00:51:20.920 Get your ally. Not sure. Yeah, thank you so much, 00:51:20.960 --> 00:51:25.079 sir. It was just great. Loved the time. It just 00:51:25.079 --> 00:51:28.179 disappeared in a flash, sir. All right. Well, 00:51:28.340 --> 00:51:32.159 I'd be interested to hear how it comes across. 00:51:33.199 --> 00:51:35.639 Yeah, same. Thanks, everybody, for listening 00:51:35.639 --> 00:51:39.139 in today. Had a great exchange with Gary. So 00:51:39.139 --> 00:51:41.440 this is John Marchiando signing off, saying goodbye. 00:51:42.280 --> 00:51:43.539 See you on the next one.