Profound
Ramblings about W. Edwards Deming in the digital transformation era. The general idea of the podcast is derived from Dr. Demming's seminal work described in his New Economics book - System of Profound Knowledge ( SoPK ). We'll try and get a mix of interviews from IT, Healthcare, and Manufacturing with the goal of aligning these ideas with Digital Transformation possibilities. Everything related to Dr. Deming's ideas is on the table (e.g., Goldratt, C.I. Lewis, Ohno, Shingo, Lean, Agile, and DevOps).
Profound
Profound - Dr Deming - E2 S5- Gene Kim- Socio-Technical Maestros
In this episode, Gene Kim and I talk about all things DevOps. We discuss some of his podcasts on the Idealcast along with his current work with Dr. Spear. Almost all of Gene's work can be found at Itrevolution.com.
Hey it's John Willis again, is the Deming profound podcast. I've got a good friend, old friend now we've known Mr. Jean Kim. Hey Jean, you want to say hi to
Gene Kim:John? Good? Good seeing you again. And happy 2022 to you. 12 years.
John Willis:Crazy. Yeah, I was I was like, think about like, great. We were at that DevOps days at LinkedIn, it was the first DevOps days in the US. And we were both on a panel together. I think I got stuffed in there at the last minute. I think Patrick bar was the moderator. I like, this is my sort of version. I think it's accurate. And I remember you giving me some kind of compliment, and, and then when I got off the stage, I was, you know, Damon Edwards comes right up for you that was on the panel, like, you know, and I do remember sort of having that dialogue with you and emails, but but back then Damon was running through all the legwork in
Gene Kim:the brains of the operation
John Willis:guided you and, and, you know, and then he says that was, you know, Jean Kim. I'm like, Oh, the visible ops guy. And I think I went looking for you. And I was like, Oh, he's gone. And then we met at South by Southwest. And, you know, we started this 12 year sort of journey of like, lots of fun, so
Gene Kim:that you're one of the early reviewers of the Phoenix Project. We were co authors together on the DevOps handbook. And we did the be on the Phoenix Project together. You were one of the original members of the program committee for the DevOps enterprise Summit. I mean, like, how many adventures that we've been on? Yeah, no, it's over yet.
John Willis:It's been Yeah, it's definitely not fun, good stuff. And, you know,
Gene Kim:the safety panel with oh, your cook, Dr. Spear? And? Yeah, yeah. No,
John Willis:I mean, jacker. You know, I that whole idea of like getting, you know, because remember, that was something I was just driving me. It was just gnawing at me that these two bodies of work who I so respected, I mean, like, you know, you turned me on to Mike Rother. You turned me on to CEO, you literally would say, Hey, John, read this book, and I'd read it read this book. And, you know, I read Rother and I read hivelocity edge. And I'm like, like, Oh, my God, it's like, same thing, as you know, I was thank you for, you know, I don't know if you remember this, before you give me an early early copy of it wasn't even called the Phoenix Project. You suggested, I read this book by this guy called Le vo rat. And that was such a gift. Because I think, you know, if I were to read him in the different order, I wouldn't really got it, you know, so I, I read the goal. And then I read critical chain, I like, I read a whole bunch of his books, because I was like, Oh, my God, this guy, like, he gets me. But I felt
Gene Kim:I felt so sorry, that I actually, that was sort of the condition. But I felt like, you know, the goal was such an important book that, you know, I felt like you should read that first. Right, so that you could make an honest and objective measurement, like, does this book kind of, you know, adequately, is it an adequate homage to the book, right? Or is like, you know, genius crap.
John Willis:That was That, to me, that was such a gift, because I just wouldn't have understood it the way. You know, I did like the sort of impact of what you try. And then what we did beyond the goal. And then, you know, that was part of it. Like, there was a deeper explanation of like, what you tried to do there in the Phoenix Project from the goal,
Gene Kim:right, along with fellow co authors, George Spafford, Kevin Barry.
John Willis:And, but but I do, um, you know, as we sort of had those early conversations, and I remember, you're talking about the DevOps movement, and then you know, and then you told me about the Phoenix Project, we had discussions about that. And then, then you were, I think you had already started talking to Patrick, about sort of doing this sort of cookbook idea, because the thing that, like, stood out is like, people are gonna read the finished project. And they're like, Okay, what do I do next? So we started, you know, multi, multi year discovery, and I remember, like, you know, some of the mind maps that we were building back in the day, right, like, just, we spent many years just like really walking through, you know, sort of an education for all of us.
Gene Kim:I think it was four, maybe even five years before the DevOps handbook came out.
John Willis:Yeah. No. And we have Mike Orson member Mike Carson was involved. That's right. Yeah. So, um, I think, you know, the, the, a couple more sort of history, you know, the history nostalgia, which was, you know, the, the talk about starting the DevOps enterprise Summit, right. And one of the things I loved about the original, I think, there was a small group of us who understood that these ideas of DevOps were universal. I don't think I understood the, the roots of it going for Lean and Operations Management at that point. But, but we did understand that this is like, these are universal ideas. And, but there was this larger group of people that was sort of saying, you know, wow, you know, that's kid stuff, there's, you're not gonna be able to do it in the enterprise. And I remember I interviewed Matt Damon, we're getting into his, like, week long Twitter battles with, with people from the big four big five. Right. And, and to me, that's sort of the the moment I knew that like, like, the ideals that we were talking about, we're going to work without having to sort of change them or dilute them was when we saw those first CFP responses. You know, we sort of target we sort of, you know,
Gene Kim:Apple, one Nationwide Insurance, Blackboard, right. I mean, all these organizations have been around for decades and centuries old as citizen, the Mark Schwartz from US Citizenship and Immigration Services inside of DHS. So hey, you know, that first year, I think it was just an incredible proof point, right, that these are relevant to every industry vertical. Here, we got a little how old the organization is, in play this little kind of funny point, right. I mean, I think the oldest organization that presented was a UK HMRC Her Majesty's Revenue collections, which is their version of the IRS, which was created in the year 1200. Right. So I think,
John Willis:yeah, now, I think it was the second year, like you ping me and said, because I think that was the thing. It was like, to me, it was Pader. Like, we're right, like, and they their their proposal for presentations. And then when we ran the conference, it was not uncompromising. They weren't doing what so some of the big consultants are saying that you're never going to be able to do this and a large bank, they were saying, No, we're, you know, it's hard. And we're doing it, but it's hard, you know, two steps forward, one step back. But the next year, I remember seeing a proposal from like Western Union, and whatever that sort of the sort of the paint company that you find Sherwin Williams Oh, yeah. Okay, now, this is for real. Right. Like, so.
Gene Kim:Yeah. It's to, to get to, I think, the core thing of whether it's, you know, Deming, Goldratt. You know, whatever comes next is that there are certain principles at play, right, that are universal. And, you know, I think the objections what they sound like, in the beginning was, oh, you had to be a web scale company, the hyperscale. Companies, which, on the surface, just, I think, to us just seemed absurd, right? Like the technology value stream actually exists, regardless of whether you're a hyper scalar or not some. So I think it was really cool, because there's no better way to prove it, then existence proofs.
John Willis:Have these companies are now over the years? I mean, what what is the catalog now of how many companies enterprises have presented? I
Gene Kim:think at last count, it was about 1000 plus videos. So I would just say about 800 a day.
John Willis:It's incredible. You know, even so I was telling you, I wouldn't know, you know, I wanted to catch up on some of your podcasts. And I, I've been fascinated by this argument about just in time. This is if people are pinging me and see John, now, what do you think about just as if, like, we've proven it doesn't work? Right. And then, you know, we're both huge fans of Dr. Sphere, right. And I just love the podcast, where, you know, he, you know, he explained it to us, all right. And, you know, and I just, you know, and your references. Like, I think there's just so much like you'll say, though, you know, there was this presentation by Walmart back in, you know, and be able to sort of use all that which we did in the DevOps handbook as well. Right. We will use all that material from all those sort of presentations. But the idea that there are just so many proof points. Now, when you want to tell a story.
Gene Kim:Yeah, I mean, I think the the point that, that podcast episode with the spear was about was to rebut the narrative that companies are in trouble because of the lean supply chains. And I think the argument that Dr. Spear was very forcefully arguing was that no, no, it's actually we are in as good shape. We are in, you know, unprecedented times because of well managed supply chains. And it was just one of the things that, you know, to substantiate this claim. I went to the Fred sites us the Federal Reserve, economic data site, run by the St. Louis branch, and it's just pretty incredible. So I calculated the inventory sales ratios going back almost 100 years and I mean, it's just so you can imagine a line. That's like basically kind of going up, and then it dips, like, you know, so visibly Yeah. And you know, around 1982, which is, so that was kind of so I divided the data points into two populations, one for 92. And y 92 is because typically economists will divide up things at the end of a recession. So 1982 is when they marked the end of that particular recession, and then afterwards, and the inventory to sales ratio, post 1982 is 1/3 of that from before. And so arguably, right, that just means that, you know, the businesses and anyone who's carrying inventory has freed up two thirds of capital to deploy it to more useful things. Right. So so and as opposed to it being tied up in inventory. So I think that's just incredible evidence that suggests, you know, what the huge value and the ways a site is changed just by better management supply chains?
John Willis:Yeah, no question. I mean, again, you guys talk about, like, I love this sort of students here says, you know, when you next time you pick up tomato, like, try to understand, like, how that got that beautiful tomato that you're cutting up for, like a sandwich or your meal, Howard got to you, you know, through the incredible complexity of other things, I you know, the one thing I thought about when I was listening to that, like, you know, I've got Deming on the brain at all times, and but, you know, like, 1982 is, you know, like, 1980 was the beginning of quote, unquote, Demming mania in United States, there was the NBC documentary, and 82 out of a crisis was published, it just seems like, more than a coincidence, but possibly a coincidence, that a lot of the ideas behind you know, sort of Deming starts about suppliers, you know, toward a supply chain concepts or, you know, and just, you know, that just variation, we talked a little about variation. In one of the last podcasts we did on deal cast. I have some more thoughts there. But, um, but like, it just seems interesting coincidental that, like, we have this amazing growth in just very soda, sophisticated plot supply chain, and the economics related to these models of delivery.
Gene Kim:Yeah, so we're talking beforehand about like, Why does 92 come up? So often? And I think maybe part of it might be coincidental, right. I mean, the publication, but I think the reason that we're talking about is that, yeah, that period was the darkest period of the economy before the recovery. And so when do people need help the most? Who needs help? When things are going well, nobody who needs help, you know, when it basically every, when you're sailing against the wind, and every factor in the economy, and and so I think that is probably what contributes to 92 Showing up so often, you know, in terms of a as a division between before and after,
John Willis:yeah, and I guess the other thing, you know, a lot of times when we think about software companies, we always look for these inflection points, right? Like, which is, what is the point at which somebody might be, you know, like, going and telling somebody to change when everything's normal is like a very hard sell, when they're all sort of pulling their hair out and saying, you know, like, you know, everything's broken, you kind of swoop in with a new idea. And so, you know, that part of that coincidence might be some of those new ideas. You're at this amazing inflection point of like, what we've been doing, but it's terrible. To do something new. Um, what are your thoughts? I asked most guests that so like, what would you say, you know, why is Dr. Deming important in our conversation here in 2022? Yeah, in fact,
Gene Kim:I was trying to find my book on it. One of my favorite books that I read in it as we were researching the Phoenix Project. Oh, my gosh, I just pulled up the Amazon page, last purchase on August 4 2000. For what called Deming and Goldratt. Oh, yeah, yes, yeah. Okay. All right. And this, uh, basically, it was I just loved the book, because it was so understandable. But essentially, it was the author dominico Lapore
John Willis:before Yes. I've got him queued. And we go, no kidding. Yeah, it's
Gene Kim:a great book. And it just did a I just thought phenomenal job and saying, you know, those core concepts and principles about what you need to manage well, and I think there's kind of classically three people who are credited for the manufacturing revolution. One is certainly Dr. Deming. In fact, Dr. Goldratt wrote endlessly about how he was standing on the shoulders. I'm not one to, you know? Yeah, it's the fact to me it was such a
John Willis:yeah, go rat would not like it has to be like really, really true.
Gene Kim:Exactly. He didn't give those out that kind of credit. Exactly right. And so Dr. Goldratt, I would certainly consider another one of those. And then the Lean movement. And I think between those three, I mean, I think six sigma goes in there somewhere, right? Between those four, certainly, you know, that was the majority of the market share of people who said, There's got to be a better way to run things. And so I think if you were to paint the genealogies You said so wonderfully throughout your podcast episodes, which I so much enjoyed. The one on that autonomous driving one was just mind blowing. To me, that was amazing. But that, you know, I think all of them would credit Dr. Deming as being the person who really planted the seeds, the core ideas that everyone tried to improve upon.
John Willis:Yeah, yeah. No, I think, you know, the conclusion of all the research I've done on Deming now is, um, you know, like, one of the things you hear often he's the guy that changed Japan, he's the one that created the miracle in Japan. Japan changed Japan, Japan, Korea, Japan. Yeah, he was a conduit, he basically helped a lot of sort of influence in that economy that needed help change the way they thought, but, but you know, I think, yeah, I think early on, I would say things like, you know, so Deming did this and we wouldn't happen like kiss Deming. Deming, like anybody else was just incredible influencer of like, really grounded ideas. You know, like Dr. Schulz works, this is the press control. You know, all that. Yeah. In fact,
Gene Kim:one of my favorite books is called The Structure of Scientific Revolution by Dr. Thomas Kuhn. And, basically, the punch line is, whenever there's a true revolution, whether it's Copernican to Newtonian Newtonian to Einsteinian in each one of those cases, you know, there are many, many people working on that problem, sometimes in cooperation, sometimes in competition. But then, you had love this image that he paints in the book is that and then something happens is he called it like, almost like sublimation, when gas turns into a solid instant, and then one person gets the credit. Yeah. And I think Deming, is that person, right? Ideas crystallized around him.
John Willis:Yeah, no, I think you, you know, you. Yeah. So you know, you get if you think about, you know, the, the sort of aircraft is, Taran, there's all these like, incredibly evil, and there's ones I found the listing, but he does seem to get the credit, but then, you know, the, like, some people are an inflection point. I mean, even like Amazon's cloud, like, you know, like, I, you know, me being in this industry for so many years, you know, and here's the old timers, and we were doing a club back in the 90s. And, you know, what's new, you know, like, it's new buddy, like, I was doing your version of a cloud in the 90s. But like, in 2006, when Amazon created this API infrastructure, that way, you can start off 100 compute instances, by hitting a button, you know, hitting Command line, that was a little different than, you know, credit, right, like,
Gene Kim:Absolutely, and use the word inflection point. And that term, often gets credit to Dr. Andy Grove, former CEO of Intel, but that term actually came from Dr. Thomas Kuhn. And so good cheer example, right? The inflection point of AWS easy to write everyone got it? Like, oh, not everybody got it, but as the people who mattered got it, right. And that was the first sort of like, there was pre EC two. And there was probably others like AECT. But EC two is one that can help kick off the cloud movement. Yeah, be I think one narrative.
John Willis:So So what can you tell us about this book? Your doctor like, I'm sure everybody's like, Oh, man, what's this gonna be all about? Oh, yeah. Crazy fascination with spear like, I remember what writing the handbook. I felt like we had to pull the end on cord on you period. On Shawn Spears stories like Jean, you've had, like, you've met. You met your quota here, like no more spirit stories, like just one more but yeah,
Gene Kim:right. In fact, I mean, just to fully explain a story. So I took a workshop from him. It was a two day workshop that he was giving an MIT Sloan. And I was speaking about before and after, right? In my mind, it was like before the spear workshop, and then after, and I remember coming back, and yeah, and that's when, like a whole bunch of things suddenly became apparent to me. And so one of the key teachings that he had was like what he called at the time, let's call the three S's or four S's, right? See, solve, form, and share. And I, I remember, I think it was in one of our calls with you, it's like, oh, share, like, we're totally missing, share, share, platform, share codebases, shared, sharing, you know, dojos, and all of those things are a way to spread greatness. And I think that's what resulted in basically all the changes started making Darkspear this Darkspear that then a certain point in time, I think he got really frustrated because Darkspear the word starter spirit can't be half the book. So yeah, we did have to dial it back. But yeah, I think ever since then. So that was eight years ago. Um, uh, certainly after the sometime, in the last three years, I've become endlessly fascinated, and maybe obsessed with Trojan sand, why, and how Steven spear students sees the world. And I think to cast, in fact, the whole idea of cast is in service to try to understand that goal, which is why, wow, how and why do organizations work the way they do, both in the ideal and not ideal, and to be able to explain it and ideally predicted, you know, using the most parsimonious principles as possible. So, and that notion of the principles parsimony, I just love because, say the goal of science explained the most amount of observable phenomenon, with the fewest number of principles, confirmed, deeply held intuitions and reveal surprising insights. And I think the, I think what we're working towards is, you know, to be able to say, hey, how do I organization work, it can actually be very simple. It's a function of primarily structure. So that could be like the org chart. So the way we structure our teams, and then in this in certainly the software space, it also includes the architecture we work within. So Conway's Law, right, saying the structure should be isomorphic, to the architecture, right. And also, those bad things happen if they're not congruent. And then you know, the dynamics and the outcomes are almost entirely a function of that. And so even culture, we say, is a is a dynamic. So we can create a culture where weak signals are amplified, you know, reinforced by the top. So that's weak signals, a failure can be averted. Or we can create a system where, you know, everyone's afraid to tell bad news. So weak signals of failure are suppressed, or maybe even extinguished entirely. And so think about maybe most recently, the 737 max issue of Boeing, right? What is it that happened, where so many people inside of Boeing said, you know, we could see this coming? Right. And yet, it resulted in, you know, failures that killed hundreds of people. So and so most recently, we've been working on what are the specific characteristics of structure that must exist to be high performing, you know, it has to be simple meaning modular. which ties into his earliest work, duck Spears work in 1989, the most widely downloaded Harvard Business Review article of all time, decoded DNA production. So that's all about simplicity and architecture. The second is standardization. So this means that there has to be a strong declaration of how things should work. So, you know, we can think about standardized work, we could think about Netflix Chaos Monkey, right, then the Simian Army, all of them declare how things should be working, which gets to the third characteristic, which is standardization, simplicity, simple, standard, synchronized, have stabilized, right? So stabilize means that when something goes wrong, it creates it can be fixed locally, without blowing up to become a global issue. So think and on cord, and then fifth is synchronized, meaning, you know, things are the thinking is happening at the edges, not at the center, right, which simply cannot keep up. And so, you know, we've come up with, we've kind of projected those four characteristics into, you know, for example, Chaos Monkey versus a tightly coupled, monolithic architecture, right. Why you know, why? The first has all four characteristics while the second has none. Whether it's CSDs, ci CD pipeline versus you know, infrequent deployments, right? Showing one has all one has none. Toyota versus General Motors, right? Why Toyota has all four General Motors has none or back in the 1980s TEMA Teams in the beginning why it had none right. But by the end it had all of them. So it's been incredibly satisfying, you know, to be able to I feel like what we're doing is creating something has a lot of explanatory power. It can no Blaine. Why?
John Willis:Why No, I, you know, again, like an AIPAC issue. You covered a lot of that. And then, you know, if we go back to like that we were talking about that panel we did with Dr. Speier, Richard Cook, and Sidney Decker, right. Two and a half hours if you're willing to door geek ism, but that was that my frustration is that I kept hearing these voices, I really saw Rathore and the biggest tragedies, we couldn't get Rafa to be there, but but, you know, should have been Rother and sphere on one side of the table and Dr. Cook and, and Sidney Decker on the other and, you know, I come to you and I said, you know, like, it's, it really is, you know, frustrating to me that, I can see that they're both telling the same story. But they can't see that they're telling you. You know, and I think that, you know, when we, when we, you know, if you ever get if people get a chance to watch that the, you know, you mean you someday we'll do the Mystery Science Theater 3000. But like, in the beginning, they're, they're sort of like, you know, sort of an animals in the safari in Africa, you know, sort of, you know, the horns are touching, but they're not, you know, in in somewhere, they start, like having these sort of glorious agreements. And but then, you know, I get the sense that, like, that was just a point in time. And now, I don't see them collaborating. But if you read hivelocity edge, and then you sort of you, you sort of read something by Decker you like, like, if you're not deeply in one of those tribes, like, you see the commonality of what they're saying,
Gene Kim:Yeah, I think that is what's so exciting to me. And I, and I really want to thank you, for you pointed this out. So early on, and here's how I would describe it now is that, you know, whether you're talking about system dynamics, or learning organizations, or you know, psychological safety, or lean, or, you know, architecture, modularity, or culture, I think all of them are incomplete expressions of a greater whole. I would say, This sounds maybe a little bit overly grand, but I think I can view this work with a spear is trying to come up with, towards a theory of organizations. And, yeah, without acknowledging that all, you know, so all of those are deep centers of expertise and knowledge and a corpus in and of itself. Alright, but that's got to be wouldn't be great. Right to be able to serve, as you've noticed, and your as what you suspected before we put together that panel, right, is that they, they are they're not totally disparate? I do with the Venn diagram that they, they have a lot in common. And I
John Willis:mean, like, like, they're like lockstep on a lot of things, you know, when you talk about the sort of the resilience and safety, you know, how they think about, like, in your case, you know, the, the edge, you know, you know, making sure that Lisa, you know, the sort of dominant, like, control structures, you know, kill people, you know, what I mean, you know, where, whereas, you know, so that, you know, that whole idea of, you know, the sort of the gamba idea, or the the sort of the, the learning and Walker talked
Gene Kim:about, right.
John Willis:You know, I mean, that's sort of deeply entwined in sort of Decker woods and cooks writing. Yeah, in
Gene Kim:fact, to your point, I mean, it's, it's a bit of an aha moment, right? I mean, the whole notion of a gamba, right, is that you don't learn from my desk, you know, through status reports, you have to go to the shop floor, dude, go to the, you know, IDs and so forth, right to actually see how we're working to be performed. And so that means, in kind of the language here is that, you know, this is, this is the learning happens at the edge. Right, and this is not gonna happen in the center, right, the center has to go to the edge. And you know, for me, just maybe one of the big aha moments for me is that, I think the classic example of low performance is that the edge is working in slow mode, and the center is working in fast mode. So fast mode is expediting your handling, you know, escalations and so forth. Right? Because the edge can't do it nice. So it is stuck in slow mode, where we really want is the edge to be working in fast mode, right? In routines in state of flow. You know, they get to help when they need help they get the help they need. And that allows leadership to be truly in slow mode and more slow, a slower, more deliberative, contemplative style, like General Stanley McChrystal. You know, when he inherited the story in the team teams, right. When he asked, are we achieving the goals of dismantling the enemy terrorist networks in Iraq? And the answer was no So you reconfigured the organization, so that you know, the end state, they could go from, you know, and me citing by 22 year old drone operator to capture 45 minutes later, right? That could not happen absolutely not happen if you have to escalate up eight levels. And then down eight, right in order to go from citing capital, there will be no capture.
John Willis:But you'd mentioned like the Boeing 737 Max, right. And like, did anybody right, you know, so Steve's here has like these brilliant stories about what is it the the Challenger, or the Columbia? Like, I think it was Columbia, right. Which the thermal panel and, and how, like, you know, this sort of, like, No, it was a sort of anti gamba story, like, you know, like, we've seen that before. He's got a couple of great stories. I just did a whole a deeper analysis on night capital, you know, and I, and I know that I'm going to get pounded, but my good friend, John and his, they're gonna throw a counterfactual at me. And I tried to say like, this is a thought exercise. But, you know, the question. So not only was we know, the story of like, the sysadmin, deployed to seven servers in an eight server cluster, there was some old code on a close, sort of closer, that old code was actually test code that did that literally did high frequency, buy high sell low to validate the algorithms, right? But it got turned on in production at scale, right, like, you know, so we won't so like they lost while they were like 6 billion in lost like liquidity in like, 40 minutes, they lost 500 million, they were sort of out of business. But I asked the question of all the things that sort of led up to that right, you know, and in like, the question I ask is, did anybody did anybody see this? train wreck coming? Right? Did you know I got into a long conversation with Dr. Kirk one time about, like, you know, I said, like, did anybody even ask if they were using chef puppet or Ansible? And not like, and he was like, counterfactual the counterfactual about a penalty, right, a 15 yard penalty. But but the point is, like, you know, I think in a lot of organizations, you know, we set this trap to your point about bowling is that we don't enable a social structure to allow people feel a psychological safety, to feel comfortable, because when you look in these, in a postmodern perspective, I believe it's unfair to sort of judge them. And that's the that's sort of the resilience point is the counterfactual. You weren't there. You don't understand decisions they had to make. But I do think it's fair to ask the question, you know, were there people involved in this? Yeah, really. So. So you know, like, hey, you know, maybe what do you order stop this? Right. And by the
Gene Kim:way, I do acknowledge the danger of counterfactuals. In fact, I wish I could remember this quote about like, why, you know, it's, it's so dangerous, but I think what's interesting to me, I believe, here's how I would reward that. So Dr. Ron Westrom introduced me to the term, the technical Maestro. And I'm just going to change the language slightly and call it a socio technical Maestro. And as he said, it has basically five characteristics, high energy, high standards, great in the large, also great in the small, so that they know how to ask good questions, and they love walking the floor. And when I heard that, I mean, it was jarring. I was like, can you say that? Again? It just for me, it has such incredible explanatory power. For me. It simultaneously explains, you know, those situations, you know, in our lives when things are going amazing, fully supported by who's at the top. And it also explains when things are going terribly, fully enabled by who's at the top. And so there's a corollary to this. He said, it was raffinose Law number 23. If you have a dope at the top, you will have or soon will have dopes all the way down. So, so, so I think what's amazing, is in these high performing organizations of which, for this book, you know, that will hopefully come out in 2023, maybe 2024, right? We're saying, you know, in 150 years, whether it's agile DevOps, you know, uh, you know, let's take a look at in 150 years of great organizations, decimating horrible organization. So, what are those Team of Teams after versus before the US Navy in World War Two versus a Japanese Navy, the US Navy Nuclear Reactor program versus the Soviet Navy's the Polish special program versus the Apollo space program versus the US Space Shuttle Program? Toyota versus gentlemen. DevOps versus waterfall? Yeah, Apple iPhone versus Nokia. Yeah. As I think, you know, in general, right, in fact, I believe, in each case, you, there has to been that sociotechnical Maestro and and the, a great example that I just did a recent aha moment is, you know, the you and I both read a lot of books about Admiral Hyman Rickover, the father of the US nuclear Navy, I just read a book called from the other side of the periscope. And it was written by a US Navy submarine captain and the Soviet Captain counterpart, and it was the longest trail, I think that span 30 days of a US sub trailing a Soviet Echo to class up. Anyway, there was one line that blew me away, it was apparently, in the US Navy, the NR program, the Nuclear Reactor program has a chain of command as outside of the standard chain of command. So in order for a ship to sail, they have to get clearance from usnr, which is led by a four star Admiral, and so. So here's the associate sociotechnical mechanism to make sure that, you know, you're uncompromising in terms of safety, as I think the these maestro's understand not just the technology side, but they understand the social structures, the social circuits that need to be there, you know, for us to get the right things, right, which I would hypothesize, right, it was at, you know, those mechanisms were absent, at going right, so that these signals of these early warnings, right, could not get transmitted to the right place to meet to let the right things happen, right, probably nightcap night capital pile. And I can think of a
John Willis:classic example, which was, you know, and so, again, like I worry about sort of getting pounded by the whole counterfactual thing, but the, you know, they're about to launch the first sort of NYSC, black box trading or, you know, basically RLP system, like the industry has just announced this ability to sort of run your code in one of the NYS C's, it was a big deal. So Knight capital is basically. And it's one of those things where they had 30 days to implement. But here's the thing. And, in fact, I'd like to get your opinion on this, because somebody said, I'll tell you, sorry, then I'll tell you what somebody said to me. But the day of that launch, the CEO had overlapping knee surgery and was out of the office. Alright, and on top of that, so why does the CEO need to be involved, let's let's table that. The, for whatever reason, the NYSC only had his contact information. So the NYSC is watching a level of trading. So there's already sacred circuit breakers in place because of price swings. But there are no sick ranges for insane trading, I feel that the number increase was within the first hour on that day. And so their first response is to go to try to contact the CEO. And so there's a whole delay between that and getting to the CIO, and, and, you know, and some of those, and why does the CEO need to know about this? I'm like, Well, I mean, it's the one of the biggest sort of trading changes that they've made the onus or LP, and but the I think the point is, there was like, people couldn't make decision, there was all this, like, if you go in and sort of read the details, there are a bunch of people, even sis admins that were monitoring the situation at night capital might have been able to, you know, dealer, just really pull the NR cord and say, hey, you know, you know, and I've even talked to other high frequency trading guys, and they would say, yeah, no, no, we would really kill the routers, you know, we like, we've seen this in about five, you know, so again, counterfactual, but they said that happened. Like, there's already a protocol to shut off the border routers, right. And, and, but it was interesting. Yeah.
Gene Kim:And let me just rephrase that. I don't claim any expertise or knowledge about any of this. So sounds like what you just said is that in more typical high frequency trading firm, everyone understands is a risk of a runaway program, right? And so, if anyone sees that, anyone, you know, if someone if they decide that something really terrible is happening, they will disconnect the router, right, and at least isolate.
John Willis:You have to wait to get called a guy who was like sitting on his bed, you know, medicated because,
Gene Kim:right. And I guess the My question would be what, aside from that, were there other weeks failure signals that that could have hinted at, you know, the, the fact that there was a vulnerability in the system?
John Willis:Yeah. And I think I mean, again, this is sort of a, a counterfactual musical, if you will, but I But yeah, I mean, all the things that again, I go back to like, I think, you know, there had to be some people watching what was happening. That was just, you know, me and you know, we talked to a lot of corporations when you talk to people like, yeah, no, I knew that wasn't gonna work or Yeah, I could I tried to challenge him. Right. And you know, and for some reason, you're right, these were weak signals that just didn't. You know, like, the question is like, what is your sort of? What is your social technical construct? Yep. Yep. It is. Where? You know, I mean, you've talked about the poem, Neil. Story, right? Which is a fantastic story. Right. You know, Paul knew when he went into our core, and you know, and that whole sort of, you know, that sort of reverse, you know, like, he wanted to know, every injury within 24 hours, and it created the sort of reverse it create a communication model where people sort of learned,
Gene Kim:yeah, I love this. And yeah, the essentially the way, this showing up in the spirit, Kim book, which is unnamed yet is that, you know, the more consequential and the more irreversible the an unforgiving the production environment is, the more you have to focus on planning and preparation. So this comes from Admiral John Richardson plan, perform, I'm sorry, plan, prepare, perform, so the three Ps. And so to the night capital issue, and again, I don't claim any knowledge or expertise on this is that but I mean, I think in any unforgiving production environment, whether it's a hostage rescue, high frequency trading operation,
Unknown:hospital patient care, hospital patient care,
Gene Kim:right? If you, then the more unforgiving that is, the more you have to prepare, and before that you have to plan. And so this is where you do the red teaming. This is where you do the drills. This is where you develop shorthands, you know, and routines between the teams. And so I loved reading the Gene Kranz book failure is not an option.
John Willis:Yes, you turn to that too. Yeah. Oh, so fun.
Gene Kim:I mean, yeah. So apparently, like in Apollo 11. In preparation for that, I think there in the simulations, the simulations were all about going through the checklist, right? And seeing, you know, can we land on the moon, and I think they crashed, then Neil Armstrong and team crashed 10 times in a row. And they realized they weren't giving enough discretion to the pilots. Right. And so this, he changed the procedures so that they had more discretion to choose landing site or something. And now they did the first landing. But at that, I think that's an example that says that, you know, for the Apollo program, there was this incredible emphasis on drills on understanding every anomaly. So a potential failure signal, there was something called there was this ritual that gene Krantz put in place that, at the end of each shift, they had to resolve every funny, so funny was defined as any anomaly or something that didn't that deviated from the expectation. And by doing that, right, I mean, they're chasing down every week failure signal. And understanding that either modifying the procedures or adding So knowledge versus what happened, arguably, in both cases of the spatial program, the Colombian challenger where essentially a doctor Western said that essentially the same mistake that happened twice, there was something happens. It they investigate and say it happened before, so therefore, it must be okay. Which leads to you know, a theory disaster.
John Willis:Okay, that's the those are the sort of worst sins are another ones where we've seen that before, don't worry about it. I mean, that's why I take the whole, you know, the end on cord, mythologies, not to sort of denigrate it, what it is, which was what Rathore said, which is, the first thing somebody would say to you, when you pull the end on, first off, they wouldn't call you up and say, I'm acted, you go ahead, they wouldn't like be four offices away, or four floors away and say, why'd you poliana cord, you know, it, like, they came, they went there, like, they went to the, and then the first thing they said was, thank you, right, because you helped us sort of better understand, you know, whatever it is that you thought was, you know, and I think you know, in, in gene Krantz, you know, the thing I love is really sort of the movie version, which is it's in his book, Vol. 13, where literally, they don't know what to do. And so the the process at that point is, you know, when Ed Harris says, not on my dime, and then the next day, he's playing gene Krantz. Right and, and like, what did he do? They don't set up a committee to figure out like, what's the best place to buy tin foil, what's the best place to buy your Oh, they had the prototype, the solution of those sort of what was like, you know, taking co2 out of the air or whatever, right? And like, this all broke off and deal with it in such a different way. He's our enterprises work, right? Where enterprises would be. Yeah, we need to solve this, like, massive problem. Okay, let's form a committee to figure out, you know, what, let's ship, put an RFP out for 10 suppliers for 10. For, you know, let's put out like, do a study on whether we should use coral or not right. Whereas in the gene Krantz sort of model, which was put a bunch of smart people together, and like,
Gene Kim:we're not I love that scene in the movie, figure out how to replace this using only these parts. Yeah, I just just came across Twitter. Literally. Two, three days ago, the Krantz dictum that came out of after the Apollo one tragedy, he wrote this kind of famous memo said spaceflight will never tolerate carelessness, incapacity, or neglect. somewhere, somehow we screwed up. It could have been in design, build or test, whatever, whatever it was, we should have caught it goes on for about five more paragraphs, but essentially, oh, it says From this day, forward, Flight Control will be known by two words, tough and competent, tough means we are forever accountable for what we do, or what we fail to do, we will never again compromise the responsibilities. Every time we walk into Mission Control, we will know what we stand for competent means we will never take anything for granted. We will never be found short in our knowledge and their skills. Mission Control will be perfect. Oh, geez. Chill. So
John Willis:you know, it's funny, because, you know, there's a there's an interesting, and I'll come back to a sort of a damning thing that, like, you know, a lot of times people will say, well, Deming said this, and then, you know, and and so I think in failure is not an option, but Gene Kranz. Right? That's really, I think of it as a play on words, right? Like, he's not saying that, like there is this deterministic button you can push that will guarantee that a failure will ever happen. What he's saying is you need to understand failure. Yeah, be able to beat head on. Because that's the it's not an option. Like, are you not going to my grace or true north? Right? Yeah, I'm gonna get there. And, you know, um, there's a thing that I found like, Deming was once asked about Japan, and they, you know, he responded to, to, somebody was interviewed, and he said, there was only one man with profound knowledge in Japan. And so you know, the system and found knowledge, but, and, you know, and if you look at it, like, Wow, that's pretty cocky man like you like, but but then he was very sort of he, he chose his words really, like, even to the point where like, some people don't understand them, but he's like, you either understand me or don't I don't? Like I don't, you know, I don't think he worried so much about the people. Like he didn't suffer fools or, and and as I dug into it, what he was saying was there was there's this way you have to understand things. And it's he he called the system profound knowledge. And what he was saying profound knowledge was understanding system thinking, understanding psychology, understanding variation, and understanding knowledge, epistemology. And so I guess I'm making my point, like, if you sort of look at Dean crisis book in one way, you could say failure is not an option, like good luck, bad. Or you could say, things are awesome, because like, we know failure will happen. And we are going to basically do everything we can. And when it when Deming is saying there was only one man with profound knowledge in Japan, he's not saying hey, I was the only guy that could figure this out. He was saying more, like, sort of a verb, if you will, instead of a noun, right? Like, like, like, you had to understand. No sisters thinking you had to understand, you know, theory of variation, which is just the price control. You had to understand this sort of psychology. And then you know, you like, you know, I think that, you know, go back to Steven spears. You know, HBr and I guess that was his sort of thesis. One of my favorite quotes from that is, huh, put it was a community of scientists continue to experiment. I mean, that's theory of knowledge, which is epistemology, which is PDSA I mean, that's it.
Gene Kim:Yeah, I love that. And I think that's the I think this that's a such a great example of even there's a there's a greater whole that I'm hoping To simplify things, yeah, sorry, and show the provide a causal theory, it just just to go back to your knight capital, I'm having trouble dropping that in my head. So I without playing the game of counterfactuals, I would just love to know, Are there examples of profit proper? So I'm just going to this Krantz quote. Were there examples of issues that couldn't get escalated to or resolved fast enough, right, that are not trying to play the would have could have should have been, but are essentially the socio technical circuitry showing the way, you know, kind of showing the weaknesses, right. I just wouldn't be amazing to find those stories.
John Willis:I totally, you know, I mean, John also has a really good article, basically, calling, you know, bullcrap on the SEC, via, you know, their their findings was all counterfactual, you know, like, counterfactuals, or mute, hidden called Musical, what do you some sort of fun terms of it? I totally agree with him. And part of his point was, like, it would be better to know, instead of just saying they didn't have a procedure for x, it would have been really interesting. No, sort of what were the dynamics of why, and to your point? I think they would do that. But you know, that what I understand is, it's been very tight lipped. Nobody, you know, very know, you can't really find people that talk about it. And even if they do know, a little bit of it, like they're sort of under sort of tight restrictions about what they can say. But yeah, I think there's like that, to me is, um, it's such a concise, you know, the bowing thing, like what I don't know, you know, like, I just know, sort of, I know less than you. And I just know what I sort of read it service level. But that's over years of sort of mistakes, that sort of, and even the Challenger and those things. But think capital was literally something that happened 30 days, literally, the SEC said, like, hey, in fact, the CEO was sort of not really, he hadn't decided whether he wanted to use this type of, you know, latency arbitrage, or sort of high frequency trading that lives in somebody else's sort of exchange or service. And then literally, he flipped flopped. About the time the NYC said, Hey, we're gonna have one, everybody got 30 days to do it. So I mean, this is such a, like, this is like, there's a great like, you know, sort of what's called autopsy that, like, could be done here, because you get to see the complexity of it sort of a critical system. Yeah, in a window that really happened in 30 days.
Gene Kim:Here's why I think it's so promising. So I was talking to our good mutual friend, Jeffrey Frederick, and who you interviewed as well. And I was, I was describing the MIT beer game to him, I asked him, What do you think about the MIT beer game goes? And, to my surprise, I mean, he's so I mean, he's so much more well read than me. And yet he hadn't heard of it. And so I couldn't describe, you know, what it was no simulation for players, retailer, distributor, wholesaler and factory feedback cycle of, you know, four turns. And he immediately said, Oh, that will never work. Right, you're gonna get some sort of bullwhip effect. That was like, how, how in the world? Did you surmise that? And he said, I was obvious. One way communication? And what, and the only time you get reciprocal feedback is for translator, is it doomed to fail? So I feel like what he demonstrated was that what sociotechnical maestro's are able to do is see the structure and characters in their head simulate, right? Is this a one that's going to be very responsive? Or is this one that is hopeless? And I would claim that by looking at other incidents, right, you know, with in a way that doesn't rely on counterfactuals one can make a pretty good prediction of you know, what, truly what the outcomes of a truly bad incident would be.
John Willis:Yeah, I don't think you can I think you could sort of understand the pathology of an incident in a way without like, just do it as a thought exercise right? I think you know, a definitely think you can do that. Well, I mean, it's so the you know, the I think the I just I wanted to explain in my damning book the Diablo game, right, which is you're sort of similar but like there's, there's a more I found a better version of the one guy doing the envelope you know, the envelope game is you can either use a single piece flow or you can sort of a mass production right and the single piece flow is is going to be in like, it doesn't seem like the original guy did it like sit did a follow on he said like, he didn't say you got death threats. But he like people were really upset by by the outcome of single piece flow not making sense of why it was so much more efficient. There's, there's a better version of it where they take two teams at three. And then somebody tells them, hey, they tell team like to, I want you to create 10 envelopes. And they've got to do it in a mass production way like so the first guy has to fold 10 envelopes, then passes it on to the second person who stuffs them in the envelope. And then the third person seals it. And the timing is when they get to 10. The other group is that single piece flow. And the end result is the sooner we sold those, like twice as many as the other team, by the time they get to 10. But what happens are all these other things, which are he throws in some sort of sort of some turns and twists and turns where he says, okay, oh, by the way, the paper we're using is not you know, we got to switch paper, and the bat, the Batch Group, like have to start all over? That's right. Yeah. And and, you know, when you see those, and you start, you know, you start understanding, you know, the how this flow sort of works. So there's beer games, the envelope games, I think those are. So I know what I was trying to get to, which is, I think one of the things you've done really well is trying to create sort of lighthouse for us, right? In that like the DevOps enterprise Summit, obviously, Phoenix Project was was, was something that allowed us all to sort of circle around to see enterprise DevOps to that lens, your current project, certainly, I think, at the end of the day, you have to simplify, you know, I mean, like to your first principle, like, we have to think maybe that's what Deming was great at, to a certain extent, which was he simplified shoe? Which ideas? You know, like, like, why does he get credit? Well, like, he kind of explained it in a way. Yeah, did that guy who wrote that book with all that math, I didn't get it, you know. And I think it is the simplification of any interview, you know, I can't think of two better people sort of equipped to sort of drive this, if you and Dr. Spear can sort of create us, like, the patterns in a more simplified way, you know, one of the things I love that you've done in your books, too, is you, you know, like, I didn't get it till I saw the unicorn project, where you had your sort of your that, what do you call it? Not the ways, but you had the, you know, the prints or the five ideals? Yeah, that's right. And then you did the same thing. And if Yeah, and that's a brilliant sort of thing. Like, I can tell you a story. Or I can tell you a story with like, here's what you should have gotten out of it. And, and so I think if you guys could do that, I guess the other thing I know, I'm rambling and all like sort of, but I'm just like Amazon, you know, I think you guys I know us, you're probably going to cover a lot about Walmart, just based on what I've seen, from I think your Amazon's gotta be the new Toyota. Like, there'll be a point of which, you know, all the sort of discussions that we've had over the last 20 3040 years related to sort of agile, Lean software development DevOps, like, like, we like the sort of route is Toyota Toyota purchase system. So most of it, I gotta believe there'll be a point in time in the future, where Amazon will be that you know, so like, you almost like we don't talk much about what Ford did. Although, like him, taking, you know, division labor, and like, the things he did were like, like, oh, no, could have never done what he did without him. But we talked about Toyota way more than we talked about Ford. And I'm wondering if there's a time in the future, and it will be not wonder I know, for a fact that we time a future where Amazon will be the Toyota? Yeah, it's
Gene Kim:funny, you mentioned that. So this is actually an area of deep study for us right now. And this has actually been a theme of recurrent conversation with Dr. McPherson. And, you know, because, okay, why is this such a helpful for me to try to prove to myself that this is important? Yeah. So what we're saying is that structure is one of the top is one of the few leading indicators performance that we have. So just like Geoffrey Frederick was able to assimilate in his head, right, right. Yeah, this is gonna work or not, and you know, OKRs right. And BeOS right. Now, essentially, those are trailing business measures, right to the trailing indicators. So, if we if structure is the, you know, one of the most important tools and particular the performance that we have, then it says that the job of the leader, you know, where destruction comes from the leader is responsible for creating that and so it becomes One of the most important things is like what is the quality or the skills that leader to be able to create good structure. And to your point, one of the best examples of that is Amazon. And so the whole notion of two pizza teams, single threaded owners, right, I think those are important, right? And, you know, to some degree, very new, and we're trying to better understand to what, in fact, I mean, one interpretation of the book working backwards is in the idea, we would always have a single threaded owner who owns all of the different functional specialties in order to, you know, create value, but you know, that a certain point, you run out of those, and then you have to have, you know, the three in the box, right, where you have, you know, product, Dev and Ops, you know, all living together, right. And so maybe one interpretation of that is, if you don't if you can't find these sociotechnical maestro's who can be singly responsible for, you know, all aspects of creating a product, then you have to split it between different people. So, yes, still, I think that's a important aspect. That's definitely real
John Willis:tone spear I you know, but so one of the things I've been threading on heavily, like one of the things I've been trying to figure out what was Demings influenced Amazon in my linchpin is Jeff Wilkie. So I don't know, if you've looked into Jeff Wilkie. He was basically considered the number two manager at Amazon. He was brought in in 1999. And, and I don't know, to the extent of what basis familiarity was with lean and Lean principles. I suspect some maybe a lot. You know, I know at some point, the goal was one of his, you know, favorite books. But Wilkie came in from MIT, the RSC stuff, Operations Management, statistical process control, Theory of Constraints. I mean, he came in and landed hard. And, you know, and his first tackle was the, you know, sort of rethinking distribution centers to fulfillment centers. And I would make the argument that a lot of Amazon success, you know, being the sort of Deming on the brain all the time, you know, sort of came was sort of institutionalized by their adherence to understanding lean through an evolutionary scale model unprecedented. And like that, just becoming. And, you know, I'd like to think that, like, there was, I can't find any direct, like, sort of like Wilkie or anybody or basil sort of quoting damning, but I'd like to think they were aware of sort of Demings ideas, and, you know, certainly statistical process control and operations research. Anyway. So I, I think that that Wilkie has an interesting story there, which is basically sort of, if it is and I believe it is, then it confirms that you know, that you get you sort of get that you only get that kind of scale with this kind of thinking, which serve it all the way back. Just time is not dead, folks.
Gene Kim:And I would say, I don't think you can pull off what Amazon achieved without these sociotechnical maestro's. I mean, I remember reading the book called The Everything Store, describing kind of what was required to actually run these fulfillment centers in the holidays in the early 2000s. And how they would do things like for these very, these runaway hit toys, you know, they had to find, they couldn't, they had no tolerance for not being able to find these inventory items right into, you know, when they're convinced that they didn't have any in the film center, they would go out and buy them in the retailer, right? It was just this incredible, orchestrated effort to achieve a goal with you know, incredible attention to detail, you know, grand understanding the system. And I don't think they tolerated fools.
John Willis:Yeah, no, no, no, no, no, you know, things are working backwards. I did an interview with Doris queen who spent the last two years with Dr. Deming. Yeah, and one of the things she said which is every time she'd show up meet him in the airport. She the first thing they'd say on the plane is to me and say, Okay, what questions do you have for me, Doris, and if she didn't have anything like he he'd be mad. Right and and working backwards. Stay with describe that's exactly how basil sauce was like, if you flew it, basil, you better come gun with a bunch of sort of ideas and questions.
Gene Kim:And by the way, in the book, The Everything Store Wilkie his name is mentioned 25 times yeah. Oh, yeah.
John Willis:They don't mention that much and working backwards. But yeah, I mean, if you you know, I've done the research and like there's there's unquestioned he was considered the number two guy for a long period of time. I mean, he was deeply involved in like, if you could make the argument Amazon Prime probably couldn't happen okay. Without his without the infrastructure that was evolved there. You'll get he came in from a world they didn't know like he had supply chain, he knew operations research. Anyway. So
Gene Kim:I think that's an important point. This is actually for my benefit. socio technical Maestro can be a maestro, even though they may not be the best expert on that domain. So yeah, so Admiral Hyman Rickover was not the best expert in nuclear reactors. However, he was able to be the maestro for the socio technical system.
John Willis:Well, that's the other guy the right, they might get so turned on, right like this a great example of he didn't like the only way he was going to pass an inspection. He was not going to understand that new sub, he was going to have to sort of do an intent model. So yes, right.
Gene Kim:That's great. That's great.
John Willis:No, it's good stuff. Jean, you know, like, we could do this for hours and hours. Of course, we can't, but that's what it is. But I really appreciate you spend some time with me on this. And, you know, so hopefully get back to live stuff so we can hang out. And I saw an old picture of me and you walking in the street at one time in Portland. Some guy came up to us with a copy of the Phoenix Project. And he was so weird. It was so it was it? Was it early, early days. Oh, yeah. And we took a picture of the two of us. And you sign in a book. Yeah. It's like this is like, really early days. I just saw that on, like, something popped up. So that was kind of cool. All right, my friend.
Gene Kim:Well, this has been great. And by the way, thank you for the incredibly thought provoking conversation. And I'm hoping you and I will have more discussions with Nick capital. Right?
John Willis:Yeah, again, if you guys want to sort of, you know, a fourth, I guess, a fourth wheel on any of those discussions to do right. You know, I, you know, like I said, I think the WorkKeys stuff is can be explored. I think the night capital could be an interesting, you know, sort of way to understand that compact view of how everything go wrong in such a short period. So anyway,
Gene Kim:all right. Amen. Yeah, in fact, looking forward to seeing you soon in person. And so there'll be a big, high five and hug, buddy.
John Willis:No kidding. All right, buddy. Take care. Thank you.
Unknown:Bye bye.