📍 Have you ever had an idea for a physical product that you just knew had the potential to change the world? Maybe it was something you dreamt up in your garage or a solution to a problem you encountered in your daily life. Whatever it was, you knew it was a winner. If only you can turn that idea into something... well You are not 📍 alone! Countless entrepreneurs and innovators have stood exactly where you stand filled with passion and drive, but unsure of where to begin, and that's where "The Builder Circle" comes in. My name is Sera Evcimen and I'm a mechanical engineer by trade. I spend my time helping hardware, founders, and teams through my startup consulting company, Prateek. I've had the privilege of working with numerous hardware companies that are passionate about solving some of the biggest challenges in the world. And I will be your host as we explore the exciting and complex world of hardware development.  hello listeners. Welcome back to season two of the builder circle. I am so excited for the incredible lineup that's to come , this season. We're going to start off with Elecia white in this episode to talk about firmware, which is something that we didn't cover in season one. We talk about the general. Product life cycle of firmware, how to make trade-offs. And what common pitfalls to avoid and the general. Development cycle that firmware engineers go through and how they interact with electrical and mechanical engineers and as always towards the end of the episode. You will find the TLD L section where I go through all of the key takeaways and learnings for the episode, so that if you don't have time, you can just grab them and go. And when you have more time, you can come back and listen to the rest of the episode. So without further ado i leave you with Elessia Welcome to the Builder Circle. Today I have the wonderful Elecia White. She is the host of the Embedded Podcast which really dives into a lot of firmware related Content, but also just really focuses on people engineering and technology so Alicia, thank you so much for being on the podcast. I'm so excited to talk about firmware with you. I am happy to be here. I love to talk about firmware. Um, how, so I, I really would love for the listeners to get to know who you are and also embedded as a podcast because there's so much wonderful information there as well. I know you've been doing it since podcasts were basically a thing. So I'd love to have you introduce yourself and give a little bit of background. So, uh, As you said, my name is Elysia White. I have been an embedded software consultant for over a decade and I have been in the industry for more than two, a lot more than two. And I have done everything from working at HP before they split into HP and Agilent. And HP labs, I have done FAA certified software. So things that go on planes, I've done FDA certified things that go into ICUs and test equipment. I've done consumer products that were sold in the millions. And I have done many scientific equipment projects, which is where I am right now. I enjoy the science along the way. I have been an engineer. That's how I started. I've been a manager and I've been a director and I love building things. Mentoring people is a lot of fun, but the politics are for me. So it's all about actually building things. And that's what I do now. I help small companies. In one medium sized company that does science, try to get their products to market. And sometimes that's a matter of debugging their impossible problems, and sometimes it's just a matter of giving them an architecture where they can buy things off the shelf. And they don't need me. Along the way, I wrote a book it's called Making Embedded Systems. O'Reilly published it for me, which was great. And we started a podcast nominally to advertise the book, although they're pretty far apart in time and the podcast we only meant to do six episodes, but it turns out I really like asking people impertinent questions about their products and background. And I've met a lot of people along the way. It's been amazing. We're up to 450 episodes in over 10 years. And I feel like I know so many more people and so much more information. And now people come to me and tell me what I need to know instead of me having to go out and find things to look at. That's awesome. And it's so funny that you say that you first set out to do six and now you're at 450. That's quite a scope creep there. It really is. That's awesome. Yeah, that all of the work that you've done is incredibly impressive and firmware is something that right before we started recording, I told you it's definitely outside of my comfort zone. I've never worked on firmware personally. I've worked with teams that I was adjacent to teams that were working on. And I was like, that looks like magic. Awesome job guys. But I've always wanted to drill down and personally learn more. But also. Because my podcast is really geared towards hardware products, although it varies. It could be, like, a physical product where it doesn't have a embedded system associated with it. But this is more so geared towards products that do have embedded systems associated with it and firmware associated with it. I really wanted to have someone that's a subject matter expert in this so that I can ask sometimes Maybe too high level, but questions and get just a general framework for hardware entrepreneurs that really don't have the background, but do have to hire either firmware engineers internally or have to outsource it so that they have a kind of a concept of what's to come, what to expect, how it's going to go, what the risks are, what are the pitfalls and all that stuff. So I'm so absolutely grateful for you to be here. Going to be like six hours long, right? yeah, exactly. I, this is going to be a specialty episode and it will last an entire entire day. Sounds about right. excellent. Perfect. I'm glad you have the time. Okay from, starting from very much the beginning, for those who are listening that are mechanical engineers or founders with no firmware background, could you talk about what firmware is and why it's important for hardware to have a very robust firmware system architecture? You've said two different words. You've said firmware and embedded systems. And We usually call them interchangeable. Although there is someone screaming out there saying, no, firmware is microcode and that's even lower level but when you're hiring someone, you're probably hiring a firmware and embedded software engineer. And they're equivalent enough. And the, they. The thing that you want to know about firmware as opposed to software is that we're used to working in very constrained environments. Software, you want to sort a thousand data points into biggest to smallest. They don't care. The web, it's huge. It just does what you want it to do. But when you talk about firmware, you're talking about little tiny processors, things that really were like your computer 10 years ago. Or 15. Or the littlest ones, about 25. There are some very small processors out there. And That means that there's a separate level of puzzle to the problem. And because it's affecting hardware, we're not just shipping electrons, it's not just software. Which your hardware engineers and your mechanical people know about that. But the combination of having software that should be tested and should be updated and should be developed under good conditions. And the hardware part of, I'm affecting the world, I can create fires I can actually harm people. That's where Embedded Systems lies. is that between place, which is why, it's hardware and software, and then there's firmware in between. That makes a lot of sense. I always the very high level vision that I had around firmware is that kind of bridge between having a hardware do what a software is asking it to do, but like something needs to communicate that essentially. The hard part is that you go to school for hardware or you go to school for software. Almost never does anybody go to school for embedded systems or firmware. There are some degrees now, but even so there's computer engineering, but it's not always what you do in the industry. And so oftentimes you have For a hardware, a double E doing firmware, or you have a software person doing firmware, which means they're missing half the job, Yeah. or they learn it on the job and they don't know what they don't know. So it's a weird place. And people who say, oh, we have a software engineer doing web programming, let's just have them do the embedded firmware. That's not going to work. The folks who have the EE and say can you do firmware? That's going to work to get the system up, but it's not necessarily going to work to get the system robust. And all of the things that the software engineers know about testing and keeping things solid. I know, we all talk about software bugs. I do too. But there are ways to create impossible, unsolvable software bugs if you don't really understand what the software is doing. It's a good point. I personally as a mechanical engineer didn't even touch that in my education. I didn't come out being like I, I know exactly what an entire system architecture it's just like the education system, I feel is. It's siloed in a way that doesn't translate well to the real world and also this becomes a big issue when startups are trying to hire because they want someone jack of all trades so that they can do all of these system integrations. But it's really hard to find those people and it's not really the skill sets are often not transferable, but learned, as you said in the job. I have seen some job ads that are like, We want someone who can do board schematics and embedded software and web software, the full stack engineer. And there are probably people who can do that and they're 55, 60, because they're all careers. And if you want to get good at your career, you have to dig in. yeah. So talking through that and maybe this is going to be a difficult question, but I keeping it at a relatively a high level could you potentially walk us through a typical firmware build out and development process and potentially comment on like how long it takes, how many iterations, I know you're going to say it depends But I would love to hear that like broad stroke of how the development goes. Would it be too simple to talk about a computer mouse, maybe? Okay. Computer mouse. So what you're doing is you're taking a couple of sensors. and you are putting them out over some form of communication. USB, if we're talking about connected or Bluetooth or one of these little dongles I have. And so that seems simple, you sensors in data out, and there's not a lot of processing. Maybe the sensors need to be massaged so that they are giving you where you are data instead of just what have I passed, but there's a small algorithm in there. This is a pretty simple system. I'm not going to try to say this is very complicated and you only need experts to do this because is this actually, this is the sort of system that I would encourage people to. Okay. I don't usually go with the Arduino route, but this is one where you might want to just start with, Because the idea of software is often, let's say, object oriented, because that's a good way to design software, is to make everything separate so that you can split it apart and try it out differently. And so with a computer mouse, you have your sensors, maybe two or three different kinds, so that you have a good, solid design, and maybe some buttons. Each of those can be objects because you can test them individually. And then you have your communication method. That you can test individually by faking a mouse move, you just have a list of commands that you would send, and then you can verify your communication path back to the computer works. And this is how I do the architecture. You break it apart as much as you can, and then you put it back together, trying not to let things talk to each other. Of course the sensors have to eventually talk to the output. But there's little tiny pipes instead of all clumped together, then it's going to be a much better design. So the process, but you asked about the process, and I'm all about the design. To the process, you start out, I don't know whether to do the ideal process or what actually happens. So I guess I'll do the ideal first and I will... Sort of subtext what actually Perfect. Ideally you get everybody around the table and you say, okay, we're building a mouse This is what we want to build. Does anybody know about new sensors or new stuff happening? Blah blah blah meeting and then the Sensors are identified and the EE and ideally the embedded engineer go off and read the data sheets And it's funny Nobody says reading datasheets is a teachable skill, but it totally is. It's not, you don't start on the front page if you're the, this is the first datasheet you've ever read. You go off, you read the datasheets, you try to figure out how you're going to use it, what's going to work, what isn't going to work, what do you need to make things better if stuff fails, and then you have identified the sensors. Okay, things are good. The EE goes, they make a schematic, the software, the firmware person goes and starts out making a test set so that when the board comes in, they're all ready to go. Then the board comes in and maybe it has an issue, tiny issue. But at that point. How do we know there's an issue? If the embedded person has done their job, then they can test each of these modules individually. And they can find it. Cool. Golden. Everything goes great. Do another board spin. Now we're ready to ship it. Maybe when we want to update this mouse? I don't know why, but let's just go with that. And we want to update this mouse in the field with new firmware. We want to do that firmware update. We've used all the space. We've used all the space doing the mouse things and how do we update the firmware? is where things go badly because we designed the system to be small and efficient and cheap, mostly cheap, and then at the end we've added this giant feature. And over the air firmware is a giant feature because what you're doing is you're replacing its brains then. We have to decide, are we taking out features to make space? Are we doing that thing where you optimize until your eyes cross and fitting it in barely, but not very well, so that every once in a while when you update, it doesn't work and you end up with a brick instead of a mouse? After we've sorted this out, we need to go to manufacturing. Manufacturing, every time one of these mice comes off the factory line, we need to know if it works. It's not mechanical. We need to verify that this thing turns on. And so there has to be manufacturing tests. I strongly encourage those board bring up tests turn into manufacturing tests because there's a lot of similarity between them. But it's still a matter of You don't want them to type at it like you would a EE who's testing their hardware. Want it to be a red light or a green light. And so there's this testing aspect. So you start off with the design, and I encourage having embedded people there with the design. The boards, the testing, the back and forth. You do need to put it in front of users. I forgot about them because who cares? But yeah, you need to put it in front of users because You don't want the first time something goes out to be the first time it's used. You need to have the debugging and the testing and not just the desktop debugging. Because users do entirely random things, and they will crash your system just because they don't know that you always push this button before that one. Of course you do, that button says on, but they won't do that. Firmware release, and then you release the system, and then you've Send it out and then you do firmware updates. know a lot of times you open the package, the first thing you do is the firmware update because the firmware often needs to be released six months before it actually gets manufactured, put on a boat, shipped to where it's going, and sold. And in those six months you can add a lot of features. But it means that the user has to update the firmware the very first thing, which is very irritating as a user. So that's the path. There's, there's a lot of alpha, beta, when do you update the hardware? When do you do a schematic review? And how do you do the mechanical bits? Because sometimes, the double E usually is between the firmware person and the mechanical person. But if you're doing interesting buttons or interesting things that need description I have some. robotic motors on my desk, and they're servos, and they're not hard to control from an embedded perspective. But my question, I keep asking them, what happens when they jam? They're servo motors, so they don't light on fire, which is great, but what happens to the system if they jam? And the double D's yeah, I don't really care. And I'm just like, no, that's not right. No, we really need to know sometime it's going to jam. So that's part of the design process. That was an excellent overview that I think clarifies it a lot for me as well. And also when I look back at some of the the products that I had to work with, some of the firmware engineers would talk in meetings and I would go into la land because I was like, I have no idea what they're talking about. And certain things you said, I was like, Oh that's what they meant in that meeting I was in three years ago. And I didn't understand what they were saying. But. So when starting the, form, with the, sensor decisions or whatever subcomponents that you're putting into the board, then that then Gets firmware put on it. Does it start with a very similar kind of design process of like requirement generation? What is this supposed to do? And like function diagram or something along those sorts? Or should it? I just, I'm just curious. That is actually the first step, or do you go by, because you're trying to go off the shelf and you're looking at modules and components, do you first look at what's out there and that generates more requirements? Like how does that interaction go? It depends a lot on your team. People come to me and say what are the latest sensors that can do this? And I know, because I've been looking. But a lot of times it's a, these are the sensors we're going to work with. And this is the product we want to make here from where have a magic box, make those two things happen. It would be lovely to start with the requirements and hardware as an end thing so that we could actually develop, but that's never really set in stone, things change. And it's hard to be flexible when you know you have a limited system. It's hard to say, oh yeah, we can save up this much space, this much RAM, this much cycle, this many processor cycles, that much code space to do this feature we don't know anything about. . But. We're trying to make it cheap. And so we keep buying the processor that does the minimum it needs to do. right. And so that's the big disconnect. It's a necessary disconnect. You're fighting price and features, which, that all we need to do there is time to market. And we have the triangle. And when it comes to you mentioned your example of servo motors, that made me think of just regular failure mode analysis that usually is, because I always think in like mechanical senses, I figure, I always think about, okay, the physical world, what could go wrong? Like, how could it hurt a person? And then also how could the system hurt itself? Is there one, I'm sure that there are... Failure modes in the physical world that cause failure modes in the firmware world, which you just mentioned, and then are there failure mode analysis in firmware land, or is it something like a fault tree, is it something different, or is that the right way to go about it? It's seldom that formal Okay. When we're bringing things up, we have a whole bunch of bugs. The sensor doesn't work yet. The sensor works 97 percent of the time. The sensor works, but doesn't get the precision it needs. But these are all normal states. These are bring up states. We're just working. We're not dealing with failures at this point. And the failure analysis because they affect. Humans or they can be dangerous Those things are handled before the firmware. It's usually a matter of making sure the device, even if the firmware has gone off into la land, can still be safe. So we don't usually have to do that as much. We probably should, but no, we don't. Instead, we have test procedures where we list a bunch of things that you, that the users should, that the Quality engineer or the embedded software engineer should go through and verify work each time because we're Mechanical and hardware once you get it, right you keep building the same thing with firmware You can have versions and you need to know that you didn't break anything when you updated the last one I see. And you asked about the paper the papers the requirement documents this actually I want to go back to that because I forgot agile so In software, in web software, they decided, okay, users are first. We're just going to keep updating until the users are happy. We're going to do this fast, Except how do you do that in mechanical? How do you do that in hardware? It takes a while for things to ship. If we decide to use a different processor, we've spent money and that's not as true in web software, but like I said. Embedded software tends to be between them, and so we often are supposed to work as an agile team, even though we've got a fixed platform we're working with. So that can be a problem where we don't get a set of functionality, we don't get a set of goals, and instead you add firmware update at the very end and everybody cries. Yeah. It's a good point. Yeah, when thinking about Agile software development, like the shipment, like the lead time associated with shipping something and trying it out, getting feedback, and then working it back into the design process could be minutes, hours. Depending on what it is in hardware just, I'm thinking, not hardware, sorry I'll use the word more mechanical structures. That process is very extensively long, because if you design something, now 3D printing, additive manufacturing, quick turn prototypes, like that, that has made life easier and shortened that lead time, still long, and in firmware it's so annoying, you're like, I could make the software, I could put it on, but... This hardware doesn't exist yet. So it's yeah, I have it. I have it ready to ship, but I'm not going to get results back. So like one, one part of the swirl of agile development is quick. And then the other part where it actually goes out and you are waiting for feedback, that's slow. So that's so interesting to think about of just the in between. And when you're working with embedded software engineer, there are plenty of times where Agile totally works. That, that bring up and getting it working the first time. And then you say, Oh, but this doesn't work well enough. And you're like, yeah, but for that, we need another sensor. Another sensor is a six week lead time. And then we need a board just goes on and on. So there's this weird interstitial space. Not to say that I don't find that entirely amusing, I really enjoy trying to make it work. And there's, there's that puzzle part of it and the algorithms and hardware and trying to get more out of your sensors, more out of your system, more out of your motors without adding cost or without adding a lot of lead time. That seems like a very fun problem to solve in deep constraints. They have a lot of the games now that they make you do embedded programming. And it's because it's a game, because it's so difficult to try to, put the square pegs into the round holes. And I completely enjoy the fact that my job sometimes is a game and they make them program in assembly too. That's awesome. Okay, so there, there seems to be obviously a lot of steps and phases of development when it comes to firmware. And depending on what kind of product entrepreneurs are building, the way that they potentially want the firmware to be a core competency versus just saying, you know what, we really need to just get this done. So they potentially choose to outsource. I would love to get your just general thoughts and experience with how hardware entrepreneurs should go about navigating if they should insource firmware development versus outsourcing. Okay. This is, this is a very hard question Yeah. depends is definitely the answer, Yeah. but I went to a small startup that wanted to do something similar to dog locators with GPS and radio. And this was before that was going to happen. It was when GPS was only barely starting. So that gives you a time range and they were making their own GPS modules. Even though there were ones you could buy and they were expensive, they had decided in order to make costs, you had to make your own GPS modules. Company did not succeed because making your own GPS modules is an entirely separate company. I went to another company. I interviewed for a job at another company and they were telling me about their distributed network system and their radios and how expensive the radios were. And my first comment was, you're not building your own radios. Are you? That's stupid. Because that's not what the product is. If there's a company out there making a submodule, then you have to assume it's hard. Yeah. it's not like you should build your own thing. If you go, you shouldn't start from first principles if you're trying to ship a product. And that's true for embedded software too. You shouldn't try to do everything yourself if there is someone out there who can do it all for you. And I'm not volunteering. I do run a consulting company, but no, I'm not volunteering. I have a lot of fun. I stay busy and Don't call me if you need help. No, that's not true. Call me if you need interesting help. Anyway, self advertising aside, so there, if you have secret sauce, you have algorithms that are important I worked with a company that had a couple of inertial measurement units, and they're putting them around joints. And so you could see how joints were moving for physical therapy, which was really cool. And all of the Bluetooth communication between the phone and between the two units, that all is not what they care about. What they care about was the, how to figure out how these sensors track to joints. And it's a complicated problem with a lot of math and a lot of nice physics. And they had two choices. They could either document that and have somebody implement it. Or they could do it in house. And both processes work. Because that algorithm is interesting. But the Bluetooth part, it's not interesting. It's a commodity. Yeah, And so you don't, you shouldn't develop commodities. You should let other people do that because it doesn't matter. Even better, you should just buy the commodity and not have to develop for it. But if you can't do that, then insource the good stuff. Insource the tricky bits but let somebody else do the boring bits. I really like that. Don't develop commodities. I think that could be a t shirt that's, it's true. I feel like this is done. I it's, it, obviously it's all engineering. So there's always parallels. And I feel like this happens even in mechanical engineering where it's like, why it's almost like designing a screw. Like, why would you just get an M4? Get it and pop it on there. Don't try to build a screw. And if you, if it turns out you need your own variety, or you need to cost reduce so that you can use plastic ones, do that in the second gen. You're not going to ship if you keep fiddling with the first version. And you have to ship in order to figure Yeah, if there's a market. and also figure out what you need to optimize too. It's all going off of assumptions that you need to optimize certain things. It's okay, maybe it's a little bit more expensive, but the amount of work that's gonna be put in, it's gonna be more that's such a good point. It's just get it out there, get the feedback. Feedback is the most important to get as soon as possible. And you don't want to optimize the 10 cent screws when you have hundred dollar motor components. You need this, you need to look at it from a system perspective. Right on. I really appreciate that perspective. Beautifully said. Okay I guess when people are planning for firmware, when there's an entire program dedicated to a hardware development and there's a firmware component to it, what do you feel like are the common misconceptions or I guess miscalculations when it comes to firmware whether it's timeline, scope and yeah, what do people get wrong when planning for firmware development? Firmware engineers get a reputation for saying no to everything. Which I do sometimes share, yes. Because some things are hard some things are easy and sometimes it's hard to determine which is which? I just have to flip a bit. That's easy. You want me to round corners? Oh my god, you're kidding. That's so hard. You don't want me to round corners on the screen. it can be surprising what people balk at and which things are easy and which things are hard. And I have totally forgotten the question. No, it's just, just like common misconceptions around firmware. So if a hard the perspective I'm trying to get is basically say a hardware entrepreneur is a founder, doesn't have firmware background and doesn't have a, like a technical program manager that. Also understands firmware really well When looking at a firmware project what is oftentimes either underestimated, overestimated, misunderstood before going into it? So many things. Okay. A lot depends on your firmware engineer. Because it is an odd job that goes between disciplines. What can be easy and what can be hard for them depends very much on their background. And I hate saying that because I do want to be able to say, oh, just go find someone to do that. Anybody can do it. It's almost never true. And so you do need to The better your design, the better your initial requirements, the better laid out it is, the more often an embedded person can say, yes, I can do that or no, I can't. But I think that's true all the time. If your requirements are really good, your project's likely to be a lot easier than if you just keep throwing in requirements. And hoping that at the end we can fit the 10 mil part into the 1 mil slot. So that's the common issues is that the common misconceptions tend to be that firmware can do anything or that firmware can't do anything. It seems to be one or the other, either they think firmware can't do anything and everything has to go to the software, it's just a pipeline, or the firmware has magical pixie dust and can automatically do AI inside a processor that's 8 bits and has 32 bytes of RAM. And the answer to that is no. But there are processors that can! So there's a lot of flexibility, and the flexibility depends on both the engineer and the whole situation. Thank you. Gotcha. Maybe shifting the question a little bit and asking the question of, when designing a product, I feel like the crossroads that oftentimes you enter into is it solved by, is it solved by physical structure? Is it solved by software? Is it solved by firmware? And that, that question of which one to pick, I'm just curious what what your opinions are when it comes to that crossroad of making that decision. Okay, let's talk about microphones. So you have a microphone, you could solve some problems mechanically, pop filters, even some frequency problems. You could change frequencies and shift and smooth things in the double E area, in the electrical, because that's, we've been doing analog filters forever. Yeah? You can have digital filters inside the firmware, and you could process it later in software. All of those are perfectly valid. There's no real right answer. Doing it mechanically may cost more. Although it's at the front end, so you may get a better sound. Doing it electrically will cost less than mechanically, but more than firmware. Because firmware, once you discount the non recurrent engineering and the human part, it's free. And you can do it in software because then you have so much more flexibility. How do you decide? Cost is part of it. Definitely if you're looking for a cheaper solution, you're looking more at software and firmware. If you're looking for time to market, I would do the electrical engineering, but there's firmware. Yeah, there, I want to say you do this, and this, and there isn't a good solution because there are perfectly valid reasons to do it in any of the parts. Yeah, I think the metrics you talked about are what I was trying to like, understand and also provide to the listeners of just what are your priorities? Is it to get the cost down? Is it to get time to market? Is it the like perceived quality? Or, and it's always a balance between those three pillars anyway, like any design decision is, but I think yeah. Because of this siloed approach to product development that exists in the world due to the, miseducation of mechanical engineers, electrical engineers, and software engineers I think it's important to know that certain solutions could be solved with one another and thinking through that, even as a mechanical engineer, being like, is this really a mechanical problem? Being able to have a team where that question gets asked I think is, Really important. So I just wanted to from a firmware perspective, get your thoughts. And I think I got exactly what I was asking for. Podcast Break And it was a 📍 really great answer. Podcast Break This podcast is presented to you by Pratik, a startup advising and coaching company that is geared to help hardware entrepreneurs get their ideas from a napkin sketch into a lab and out into the world. And now we'll go into a little bit of a podcast break where I like to call Hardware Horror Stories where I ask our guests to focus on a hardware startup or a product that just potentially got something wrong that caused issues and how it could have been avoided. So I'm sure that you've I've seen so many but if you could maybe pick one or two that's exciting and crazy of a horror story, I'd love to hear it. I've already mentioned having over the air firmware added at the very last minute. That can work, but doesn't, usually. And I mentioned making GPS and radios when that's not what your product is. Sadly, that one I could add a long list to of things you shouldn't have thought about making, you should have just bought. It's funny, none of my hardware stories, hardware horror stories, involve products. People get really cagey about their ideas. Like I'm making something super secret and I know what happens. I worked at leapfrog, which is a children's educational toy. And definitely in the factory, there were replicas and the replicas went to the Chinese market. And so leapfrog missed out on that, that was after they were big, after they were shipping. After all of that I don't see that happening so much with small companies with good ideas. If your idea breaks down to I turn something on via the internet, yes that is an idea that will be stolen because it's already been done. No, you wanted me to talk about hardware horror stories. Let's see. There's always not listening. How am I going to describe all of the times I haven't been listening, listened to, and the gods punished the people and every, all hell broke loose. So that, that's Cassandra version 10. Okay, so a good hardware horror story. I feel like we need a campfire. It's marshmallows. I know. I mentioned the company that did the joints, and that I actually was really happy that they outsourced the BLE (Bluetooth Low Energy) parts. But they didn't really talk to their users. And I feel like that's, that happens so often. Yeah. And it goes back to, you need to ship it as fast as possible because you need to get that input to figure out if this is a product or not. And, I use them as an example, but there are plenty of it. Oh yeah. As soon as someone used it, they realized what went wrong. At LeapFrog, where it seemed half the products had all of the alphabet on them, there were this, there were times when you needed to actually try it, or they would never have had the kitchen related one that the end was nibbling on nuts, because Anybody who's 13 or older finds that hilarious. You have to try it. You have to have somebody look at it who isn't the designer. I'm not a fan of design by committee. I think everybody should have an input, but one vision makes it a little better because you aren't unfocused. You're focused towards a direction, and if your direction is a good one, fantastic. And it keeps your team together, it keeps them going in one direction, and not just trying to come up with all the things. You need the input to say, okay, there's this new sensor that can sense force, and if we do that, we don't need these other three sensors. But you also want someone who has It's a good plan and a good direction and can take feedback, but sticks to it. And yeah, I just, you need a plan and you need to stick to it as best you can. And then you need to have other people look at it. Yeah. What people looking at who What happened when it went into users? Did they just not interact with it the way that it was expected? Or did they just not like it and not use it at all? pays for it, that was what was the problem. Do people pay for it themselves or do clinicians and doctors and insurance? Oh, that's a big one. Yeah. And that, it was a great idea. It was a really great idea, but it wasn't cheap enough to make it a consumer product. And it didn't show specific clinical benefit to make it a medical product. Oh. didn't want to get medical certified, which is, expensive and difficult. Yeah, So there wasn't anything wrong with the hardware or the software. I wasn't thrilled with the app, but as time went by, it went pretty well. It just didn't have a market. And as an engineer, I don't think in those terms. Yeah, it's very true. It happens a lot. Yeah, I, my first company was HP, which. In those days, they used to describe HP's marketing team as saying that sushi was cold, dead fish because they were so literal. And I think in a tools and test company, that's cool, but it doesn't work for a lot of other people. It doesn't work for people who aren't engineers. They don't find cold, dead fish and attractive lunch. Yeah, and I feel even in technical projects the marketing aspect is really important because it does, it should dictate what your product requirements are to some capacity, who you are trying to go for, and if there are Either paying customers or a paying system that's associated. I went to a demo day recently and there were a lot of products where they talked about the technology. And there was a lot of excitement around what they had created. But it wasn't really clear. What part of the even with software products, it's you really need to have a really good understanding of who your ideal user would be and generate, have, I don't know, call them Jimmy and talk to Jimmy in your mind so that it, it generates this product requirements and then validate I feel like people get really sucked into technological development and the fun, exciting bits of it and forget that in order for it to exist, it's so much more than that. Products are a team sport. And I am definitely one of the people who thinks, oh, you should just do it my way. You should just make this product because the technology is cool and I will use it. So therefore other people use it. And the truth is, that's not true. The things I use are not. I'm an early adopter, I admit it. I like shiny new things, not everybody does. And that's okay. It takes all sorts. But forgetting that definitely leads to products you can't sell because you've marked them poorly, you haven't marketed them, you haven't realized there's nobody who can pay for it. Your user Jimmy, okay, you have this great idea for Jimmy, but if he can't pay for it himself, then what good does it do? If it's not in his price range, Absolutely. hurts me to say that marketing is one of the most important parts because I didn't want that to be true. I know, I feel the same way, but it, oh man, even with this podcast, I think that's what I'm learning too. It's, it doesn't it definitely matters that content and the in, in a product sense the quality of a product is really high. But if you don't get it in front of the right people and the right people don't have a means to get to it, say if I had this podcast on a very expensive premium platform and people didn't have maybe had exposure to it, but didn't want to pay for it, then no one will listen to it. Like similar to that. It's just it's really interesting to start to think about that kind of multifaceted approach to product development because it's never one thing. It's, you need to have all the pillars in place for it to rise up and see the light of day. And if you don't, then all of that effort, all of that excitement around the technology suffers. And it's unfortunate, but it's true. But, excellent call out. I think that is... Actually, that's a horror story that's come up in the past, too, with some of the episodes that we've had. I really appreciate having a catalogue of these. It's just please think of all of the components. Of what it takes to produce and launch product. But okay, shifting gears back into the conversations that we were having. And we left it off at this kind of intersection between mechanical engineers, firmware engineers, electrical engineers, software engineers do you feel like there is a best way of how those teams could integrate with each other? And there might not be a prescribed path that exists, but in your experience maybe, what would you have wanted those cross functional teams to interact differently and do differently to be better being a team? People are the key component here, Not their jobs. A lot of times with firmware and embedded systems, some folks get into long term feuds with their electrical engineers. Because it's really easy for firmware to say it's hardware's fault, and it's really easy for hardware to say it's firmware's fault. It doesn't matter whose fault it is. It needs to be fixed for the team. And so that's the way to interact. As a team that is trying to do something, trying to ship something, trying to make something. And leaving out... Parts can make the whole thing go a lot faster, but then you may not get to what you want. I do, I would not want to be part of all of the mechanical engineering meetings. Please. No, no more meetings. But there are times where you either need the ME to come into one of the bigger meetings or to have a presentation where you say, okay, this is our plan. Tell us now. If you have concerns or you want to talk about it further, the best way to interface is to communicate. It's a tautology. I know that really is not the most brilliant thing, but. Oftentimes, we are siloed. We are in our own little world. I'm making I'm between the embedded person, or I'm between the electrical person and the software person, and I'm cranking away on my stuff. And it never occurs to me to go to the mechanical engineer and say, This button bounces a lot. How about a different spring? Because I can solve it. I can totally solve it. The EE could solve it if I complained enough. The software person could not, but we don't, we let ourselves be siloed Sometimes we even enforce it ourselves. So I think, I don't think you should have forced group lunches, but I do think maybe occasionally taking out the other engineers, a little bit of one on one time with the other engineers really is very helpful. It, it opens the communication lines. It gives you a back channel to ask the dumb questions because they're not dumb. Yeah. It's just a really good way to make sure that everybody's. Interfacing, communicating, actually being part of a team and not doing their job and hoping that's enough. That's not enough I think... That makes a lot of sense. And if I were to summarize all of the things that we've talked about and apply it to this question from what you've said and also my own My own, encouragement for it is to be a little curious. I think that is sometimes people like to be domain experts and so they're very much solely focused on what they're doing because it's easy for them. It comes easily they're trained in it. And then there is potentially a less curiosity around other components. I think second is system level thinking. Once you put like a system architecture together, you understand the plugins and plug outs of the inputs and outputs of all of the systems. And there you see, Oh, this is where the firmware engineer lights up. This is where the electrical engineer lights up. And all of that is encased in what the mechanical engineers is working on and so on and so forth. And I think. Systems level thinking you've mentioned it a lot. I mentioned it a lot in this podcast is just really the key to getting that embedded team culture that has those communication channels. So I really appreciate you saying communications because it is complex. Like it communications between these cross functional teams is really difficult, but I think it, it stems from system level thinking and curiosity. curiosity, though it is we're engineers we're naturally curious yeah. but oftentimes we need to feel like system or subject matter experts, because while we're curious, we hate to be seen as not knowing things. We, engineers are know it alls. Yeah. One of the reasons I say take out your ME or your EE for lunch occasionally, because you want to be able to say, I'm not an expert in this, in a safe way. Yeah. not in a meeting where all your bosses are looking on. we're curious, but we're also often very afraid of looking dumb. And it's sad because those two things together really will make you look dumb. Yeah. Yeah it's the thing we fear, but it, when you act in that way, you end up kind of pigeonholing yourself to end up being that way. self fulfilling prophecy, yes. Correct. Oh, darn. Yeah, that's. I think that's a that's a general problem in engineering projects and just, we could, this is like a whole can of worms, just like an inclusive culture, a culture where people have the ability to say I don't know and not get scolded Imposter syndrome should be on that list. syndrome always And it's not just a girl problem. No it's everyone, Yeah. It's absolutely everyone I completely agree yeah. Okay. This took a turn that I really appreciate. I think it's never as prescriptive or analytical to have to build teams and have them function well in a product development environment. It's actually a little bit more on the metacognition of what causes those silos and so on and so forth. So that's that makes a lot of sense. Thanks. Going back into a little bit more, of the analytical side of things, you talked about when talking through the typical process of firmware build out and development in, in your mind, what are some critical tests to do to ensure firmware risk, risks are mitigated, and I know that depending on the vast scale of either sensors or subcomponents that you choose, those risks will be different but are there just kind of buckets of tests you absolutely need to do these? Okay, so the first bucket of tests is the prototype, and this is not your whole prototype. This is the, you buy as many pieces off the shelf as you can, and you stick them together using sticky tape, hot glue, and a little bit of firmware logic. You don't necessarily do a great job, but you have something you can type at and test each of the sensors to make sure they'll do what you need to do. That is a really important part of the system, and it's one that's become much easier over my career. There was times when we didn't have dev boards. That wasn't a thing. And now you can buy all the sensors off of Adafruit, and it's fantastic. First critical test point is that prototype. Because that's going to tell you so much about what the hardware you have and what you want. The second one is board bring up. This is often a spot where people get into a lot of trouble because is it hardware? Is it software? Is part of it. And the thing you need there is have you tested all of the parts? Have you tested all of the parts well enough you could manufacture the system? The third test that I think is critical is the firmware software. Simulation, or mock, or emulator, so that the software person doesn't need to talk to the whole system every time. A way to fake it. And whether this has actual hardware in the loop and has a little bit of functionality or whether it's pure software, I don't care. But they need a way of doing their job without having a hundred units on their desk. It doesn't help them. So those are the three things that I think really help mitigate the risks. Making sure that your system can do what it's supposed to do with the prototype, making your system does what it's supposed to do with board bring up and manufacturing tests, and making sure other parts of the system can function without going through production. You don't want the software folks to wait until you've produced 100, 000 of these to realize what they really needed was one extra piece of information. That is excellent. Thank you so much for going through all of that. And in order to wrap up the episode do you have any, general recommendations, advice for hardware entrepreneurs that potentially don't have a background in firmware, but are thinking of having a firmware component to their product? Any either encouragement or caution that you would like to provide any message to the listeners? Okay, so we'll do a little bit of self promotion. I have a book. Chapter 2 is all about system architecture. Have fun. It's not hard to find. There will be a new version coming out, but I'm not sure when, so go ahead and get this one. But things that I really think are important people remember, we've covered a lot of them. Engineers are human communication is important, make sure you have markets, hire the people you need, don't make commodities. I think, if you're going to have a hardware startup, figure out the legal parts of it, which is not something that I would have thought was important, but What happens when you and your best friend co founder need to do other things? And it doesn't have to be, an acrimonious breakup, it could just be they need to go do some other thing for their family and now you are alone. That's the sort of advice you should be getting from any small business. Sort of interaction. And there are small business companies, or not small business companies, there are small business resources that help you through setting all that up. One question for you. What else about firmware? Do you want to know what else is intimidating? I think up until this point what has been intimidating for me is because I never really understood where in the system architecture firmware landed. I didn't know as a mechanical engineer how to interact with firmware engineers. Whether it was. Asking for features, I wouldn't know to go to a firmware engineer to ask for a feature and I didn't really I also, whether it's that or while designing something as a mechanical engineer, so a structural box of sorts I wouldn't know what to ask a firmware engineer and what constraints I would be putting on them when doing a mechanical design. Like I said, the double E is usually between us. And the constraints usually involve LED light pipes or buttons. But when you get to more interesting products like robotics or consumer devices. Consumer devices is actually one where Emmys and Firmware get to talk to each other more because the Emmy is trying to shave pennies and push that into Firmware. Yeah. At LeapFrog we made some really interesting button decisions. And it was because if you stamp it out of plastic, it's really cheap. And so that's all we're going to do. You should figure out how to read those buttons, even though there's no electrical part of them. There, there isn't a good way. You can walk by and say, if only I had this feature, it would all be better and hope that somebody notices you. Yeah. Or you could, sit down and say, what are the hard parts? Is this a possible feature? Yeah. Yeah, I here, have a cup of coffee. Yeah, I think starting with the cup of coffee is the best way to any engineer's heart. But I think I do I, I really appreciate your, patience and collaboration on this episode because I really I, I say hardware all around my my podcast and what is hardware without firmware? Most of the time that is the case. And I, I did, I, this was a really important episode in terms of the content I want to be able to provide the listeners. So I greatly appreciate. You going with it and answering my potentially super high level, but it was really potentially super high level questions, but overall it was very enriching to me. Enjoyed this. If you want to do it again, I'm up for it. Oh, thank you so much, Alicia. It was such a pleasure and thank you for being on the show. Thank you, Sarah. Before we head into TLDR, which is coming up, I'd love to introduce this new segment, which I like to call hardware resource spotlight. Where I will highlight an interesting resource or opportunity that hardware founders can benefit from. Today's spotlight is not sponsored, but a very interesting opportunity for climate tech entrepreneurs. If you were a climate tech innovator, looking to commercialize your hardware product. You should consider applying for the scale for climate tech? cohort. Scale for climate tech offers, manufacturing education, access to supplier network tools and templates and technical support as well as facilitated partnerships with the climate tech ecosystem. It is a N Y S E R D a supported program. And it's administered by second muse and next corpse criteria for the application., In order to apply, this is the criteria. You need to be a climate tech innovation that supports decarbonisation. You have a product that works like and looks like a climate tech prototype. You have an operational plan with demonstrated benefits for the New York state. Financial resources above. A hundred thousand dollars and accessible funds for product manufacturing and a technological readiness level of four and above. You can learn more at www.fourclimatetech.org/ . Innovator dash services. 📍 The application deadline is December 4th. So if this sounds like a good opportunity for you, make sure to take advantage of it. If you know of or are a resource for hardware startups messaged us on linkedin for a chance to be featured on this spotlight for future episodes If you've made it this far, or you just scrolled here. Welcome to the too long. Didn't listen , or as I like to call it the TLD L section. Where I. Go through the episode, I listened to it myself and I write down notes and bullet points of key takeaways. So that if either you're an entrepreneur or your product team, and you really just need to get those key takeaways. You can come here and learn about what Lisa talked about. And then another time when you have time, you can just go back and listen to her so let's get right into it. we started by talking about how firmware compared to software. And one of the things that Alicia said was that firmware compared to software works in a much more constrained environment due to the processors being very small. And that causes a trade-off mentality when deciding what to deploy onto them. It takes both the landscape of software and hardware. So it's the in-between. We talked about the, for more development process and she was describing the ideal firmer process, which. She also admittedly said that usually doesn't happen. But the ideal firmer process is different from what actually happens. And this is how she described it. It starts with asking the question, what do we want to build? Are there any new sensors or technologies out there? What sensors do we want to use? So he embedded software engineers, read data sheets to identify the right sensors. IE. Starts making a schematic. For more starts planning the tests for when the board comes in. So after that, the board. Comes in, ideally it works great, but in reality, there's usually an issue. And then you ask the question of how do you find the issue? You test the modules individually to decouple the results. Do do we want to update the product in the field? Ask this, you asked the question of, do we want to update the product in the field? System is designed to be small, efficient, and cheap, but what can happen? Is that you may not be able to, you might not have space left, so a B, because you filled it with functional bed. If you want to update it in the field, now you need to figure out what you're going to do with that space. So you look at different types of features that you have. One example she gave was over the air from where. There's a giant future. Do you take out features to make space? Can you optimize so that it fits? And now there are manufacturing tests to verify that the product is working off of the manufacturing line. She also mentioned that you need to incorporate user debugging into the process the entire time to ensure that the feedback gets incorporated otherwise. You'd only be doing desktop debugging, which is only half the problem. She says the users will interact with a product in the most unprecedented ways. And you need to account for that randomness. Then we got into kind of agile and she said that agile is a development method that originated from software world, that prioritizes user feedback and continuous improvement. And then checks the product until the product is loved by the users. The method is great. However, it becomes difficult when hardware gets into the loop and causes lead time issues with embedded systems is particularly hard because they're expected to operate as an agile team because they work very closely with software. But. They deploy everything on hardware. So they're constrained by the hardware world while living in a software world, essentially. She also said to make sure you understand that not all subsystems need to be built in house. And a good rule of thumb is if a system has a company. That's building only that system, it's a commodity and that you shouldn't spread your team too thin and try to make a commodity don't. Design. Or build commodities, just buy them instead. Is what she said a few times, and I totally agree. You need to optimize your route to get to market the fastest and collect feedback. Not perfect. Every single component. Common misconception with firmware is that firmware can do anything or firmware can't do anything. And product development is a team sport. It's important to think through all the parts, even marketing. And we finished off with the general test, every firmware embedded product we'll go through. We'll involve prototype, build board, bring up and mock. And with that, we ended the episode and it was a really concise and great kind of intro for some. And then Probably a very obvious episode for others, but hopefully. While you're building your hardware product, if you have a firmware component, Hopefully this clarifies the scope of the work that your firmware team will be doing on a high level. And then some of the details, hopefully that she mentioned will be helpful for the firmware team members. I hope that you enjoyed the episode and that all of the content in the episode helps you build your product and your startup. The opinions and information shared on this podcast are for informational purposes only. We always recommend that you seek professional advice before taking any action related to your business or personal ventures. Thank you for listening, and I hope that you enjoyed the episode