The Journey from Uncertainty to Certainty and Back!

Every engineering decision is a bet about the future, but when should you actually try to optimize a design, and when should you just build for robustness instead? In this episode of The Engineering Passion Express, we trace the answer through 2,000 years of engineering history, from Roman aqueduct builders who worked by trial and repetition, to Newton's calculus turning motion and force into something predictable, to steam engines and aircraft becoming genuine optimization problems once the physics were well understood.

Then we push into today's frontier: modern semiconductors, where individual transistors are some of the most precisely optimized structures ever built, but the interaction of billions of them together creates a level of complexity we can't fully predict. That's where the real question becomes "How do you design for what you don't yet understand?"

Along the way we borrow an idea from an unlikely source, Jeff Bezos's "regret minimization framework," and apply it to engineering: instead of chasing peak performance, minimizing the cost of being wrong.

Join me as we explore:

1. Why optimization only works once assumptions, constraints, and variables are well understood
2. How steam engines and early aircraft mark two different eras of engineering certainty
3. Why modern semiconductor design has to move from optimization to robust design
4. How "regret minimization" gives you a different way to design under uncertainty
5. Why every leap in certainty eventually creates a new frontier of complexity we don't understand yet

Transcript

Welcome to the Engineering Passion Express podcast. On today's episode, we're going to talk about the endless cycle of engineers dealing with certainty and uncertainty. And in your career, you'll no doubt have to face some level of doubt about whether what you're designing can actually be computed performance-wise or whether you have to just accept certain trade-offs and accept that there's a lack of understanding.

We'll get into it right after a short introduction. One of the fundamental questions of engineering is when should we optimize? From this standpoint, every engineering decision is a bet about the future. An optimization problem assumes there's a correct objective function, that the constraints are known, that the variables are measurable, and that the relationships between variables are well understood.

Mathematically speaking, find the values of x that maximize or minimize y subject to certain constraints. But in real engineering, it's often how wrong can these assumptions be before the solution fails? And the difference between these two questions defines whether you can optimize or whether you're just dealing with trying to find a robust design. Let's go back in history real quick and think about how we came to this era.

For millennia, there was engineers that would design things like aqueducts, like buildings, and these engineers, basically they were just builders. They built the things. If they stood or lasted, they sort of documented what they did, spans that they could have beams span certain lengths, and they would, out of certain materials, they would note that.

Maybe even areas were noted for their stone quality or things like that, but they weren't measurable. They weren't certain about the properties. Instead, they were trying to build things that looked like other things that had stood the test of time, and they repaired things where it was necessary.

There wasn't much certainty in the way things behaved, but humans, that's one of the things that we're great at trying to come up with models so that we can have a sense of certainty. And I'm not sure there's many other animals on the planet that do that at the same level. So one of the things that we came up with, Isaac Newton came up with, was calculus.

And that really changed engineering forever. Engineers could describe motion, acceleration, forces, fluid behavior, heat transfer, and they had the mathematical framework of figuring out maximas and minimas. So once we could come up with more complex physics equations, we could use calculus to come up with optimal designs, and we could use calculus to predict the behavior of those designs.

Suddenly, instead of building and testing every single design, we can predict the behavior before we build it. And this ushered in the first major optimization era. So if we look at steam engines that were the driver of the industrial revolution, early steam engines were built through experimentation.

This experimentation led to understanding of pressure limits, material failures, sort of trade-off in efficiencies. But once thermodynamics matured, optimization became possible. We were able to use higher pressures, better expansions.

We improved the efficiency. And the question moved from, will this work? To how close can we get to the theoretical efficiency limit? And that was an early complex optimization problem. There weren't software tools around back then.

All of that work had to be done by hand. The amount of people that understood that level of complex calculus was very small, but it no doubt grew the era and let some of the knowledge diffuse to become more popular than it was. And while there's probably other items that happened in between there, the next big optimization system that came into place was aircraft design.

When you think of early aviation, you probably think of somebody like the Wright brothers. And the Wright brothers weren't really solving an optimization problem. They were trying to come up with stability, control, lift generation, structural integrity, but they weren't trying to maximize all those.

They were just trying to get to something usable and they achieved that. And if you watch or read anything about the Wright brothers, it's mentioned again and again that how they just didn't give up. They just iterated.

And that is certainly not what you do when you have deep understanding and can predict behavior in advance. Now, as aerodynamic theory matured, engineers started optimizing aircraft, wing shapes, drag efficiency, weight, fuel efficiency. The modern aircraft is one of the most extraordinary optimization problems out there.

We optimize airflow, structural weight, fuel burn, and because the models are highly reliable, we have a lot of confidence in those. The interesting thing is that there's still some uncertainty in these designs. We don't perfectly account for turbulence.

We don't perfectly account for whether there's uncertainty in the fatigue of materials, and then there's a huge host of human factors as well. So while we've done really well to take this super complex system and capture some sort of optimized model, we still have to come up with a design that is robust and can handle some uncertainty. And I think if you move today, one of the most complex problems that is being worked on at the moment is modern semiconductors.

I'm working to understand this space more and more, and the level of complexity involved is unreal. I mean, you have thermal challenges and structural challenges and electromagnetic challenges. You have aging problems, and that doesn't even get into the fact that there's a huge amount of variability in how these things are fabricated.

And since you're dealing with such small scale items, these even small atomistic changes can cause behavior problems. But the bigger interesting item here is that a semiconductor is essentially a bunch of small transistors layered together in the billions, and a single transistor itself is maybe one of the most aggressively optimized engineered structures ever created. Engineers have understood electron mobility and threshold voltage and capacitance and leakage mechanisms and switching behaviors.

And so at the device physics level, optimization is incredibly powerful. We can optimize transistor dimensions, doping profiles, gate structures, materials. The reason this all works is because the physics are well characterized.

But there are times where we deal with systems that have become too complex or are currently complex beyond our understanding in some ways. If you think about when Isaac Newton first came up with the theory of gravity, increased our understanding of forces. It increased our understanding of momentum.

It increased our understanding of attraction. And when you move on to Einstein and the theory of relativity, it increased our understanding, but in many ways it created such complexity that in all cases, we weren't ready to yet utilize it and get all the maximum value and take away out of it that we could. There just wasn't enough people who understood it really well and the applications of it for that to happen.

Every time we push the envelope of understanding and making something certain, you can almost guarantee that we're going to start building systems beyond that scale that are so complex. That we can't yet reliably understand them. And if we go back to semiconductors here, the problem is not that individual components became unpredictable.

The problem is the interaction of tens of billions of components and electrons and heating and all of that together. The interactions make it not very predictable. So when you get into these stages where things aren't predictable, you have to move from an optimization mindset to a robust design mindset.

In early integrated chip design, there was fewer transistors, larger geometries, a lot more design margin. Optimization focused on area speed power and the models were relatively simple. In modern semiconductor design, engineers optimized performance for frequency, latency, throughput, power, dynamic power and leakage power, area, transistor density, routing efficiency and yield, which is the manufacturing success rate, which is probabilistic at times these days.

But all of these objectives can conflict. Improving one is often a trade off with another. And if you wanted to get even more complex, there's uncertainty as these chips age.

These things are operating at extremely high frequencies, billions of cycles in their lifetime, or maybe way beyond that. They have electromigration, which is where your current density causes the actual metal atoms to move over time. So even if you optimize wire widths and current capacities and densities, you still have the uncertainty of how will the widths of these change in these tiny structures over these billions of cycles.

And you have to then optimize not only for the initial performance, but what will the performance be in the future? And high energy carriers can damage transition structures over time. And then even at the beginning of that time, the manufacturing reality is that if we draw this, we don't necessarily build it the same way. The fabrication itself introduces uncertainty due to lithography variation, which is as you're laying the material down, it doesn't lay down in a nice even thickness all the time.

There can be doping variations because atom placement is statistical and a small number of atoms can change the transistor behavior. So the question changes from what is the transistor to what distribution of transistor behaviors will we get? And then you have process variations, again, even vibrations are going to change things at the atomistic level. And so you try to account for things with statistical modeling and Monte Carlo simulations and design margins and all of this with the goal to give you a more robust design.

I think one interesting thing here, and ironically, I first heard about this in the business space from Jeff Bezos. And Jeff Bezos said one way he built Amazon was using a regret minimization framework. Essentially, what would you regret not doing? On its surface, I don't think it's clear how that applies to engineering.

But when you think about it, it is an approach to robust design. So if you were trying to design something really complex and it had tons of competing multi-objective functions and a huge, you know, let's say thousands of constraints. And so a perfect optimum of some metric isn't really possible.

What is another way you can approach that design? Well, you could think about all the regrets that you could have for that design. You could have regret that one faulty version of this might kill somebody. You can have a regret that this product doesn't last five years.

You can have a regret that the price is too high and that the market for it won't exist as a result. And so it's a wasted design. And you can use these regrets to formulate a different set of objective functions and generate not something that's peak performance, but something that minimizes all of these downsides.

So you move from what gives me the best expected outcome to what choices minimize my regret

if my assumptions are wrong. A simple example, you design a processor. Option A, maximum clock speed, minimum margins, highest benchmark score.

Option B, slightly slower, wider operating range, better reliability. If your model is perfect, A wins. If your assumptions are wrong and A fails dramatically, say six months into its operating cycle, then it's not really optimized because in fact, you can't rely on it.

Neither can the customer that bought it. So robust design minimizes the cost of being wrong. Now I've been in this space of simulations for 14 years now.

And in that time, I've been in conversations about optimizing. I've been in conversations about arguments about what we know for sure and what can't be known and what we can bound up with some level of certainty and what we can't. And even over time, those conversations have shifted.

Some items have become more certain. Some have become less. We're moving into an area of AI and simulation, and there's a lot of attempts to expand that boundary of certainty.

And it will no doubt happen. We'll get to a point where we can simulate aging in semiconductors really well. We'll allow engineers to optimize problems that today seem like they only have to minimize the regrets about.

But once we reach there, we will no doubt develop the next level of systems that is too complex to have certainty about again. And this cycle will continue of measurement, understanding, modeling, optimization, complexity, unknown interactions, robustness, better models, optimization, on and on and on. Because engineering progress is not the elimination of uncertainty.

It is the continuous expansion of the territory we can confidently optimize. And when we can't, building systems resilient enough for everything we don't understand. If you're out there and you're solving a problem that you're really good at optimizing, perhaps the next thing you should be doing is looking to design a system that's just a bit more complex than you can optimize.

That might be the next big opportunity that you have. And if you're on the other end of the spectrum and you're designing a system that's really complex and you're white knuckling what's going to happen next, maybe the next step for you is to chase some certainty. Figure out how to model some things that you can't model today and get a little bit closer to optimization.

It's been great having you on the Engineering Passion Express today. And I look forward to seeing you on the next episode. I really appreciate it when you stick around like this.

As I mentioned on the last episodes, I'm still working on an episode about Boeing. I think it's really interesting. It's coming together a little bit better than it was in the last episode.

And I hope you're tuning in and listening until that episode is ready. I think it will be good. I've enjoyed having you as a listener.

If you haven't, please subscribe now. It would be very appreciated.