[Note: please be advised that this transcript contains strong language]
Transcript
Moderator: Yesterday there was a track called Software Supply Chain, and it was a whole day, it was a whole bunch. What I heard a couple of times is there was a lot of questions, or people tried to define what it meant. I don't really care what the canonical definition of it is, but what does supply chain mean to you all, and then, I guess, this software supply chain thing?
Edwards: I think what's interesting about it is in the past we could keep it all in front of us. I think the world was small enough where we could keep it in our heads these are the 10 components of this particular service. It was easy to track, and we could do more ad hoc and semi-manual efforts to track what software is out there, what versions are going out there, what are the components of that version? I think now when you see the explosion of microservices and all these cloud-native technologies – has anybody seen those Death Star diagrams? Have you seen these cool visualizations where they put all your microservices in a circle and then draw the interconnectivity? It makes this crazy, quite pretty fractal-like diagram in the middle.
If you think about that, you can't keep that in your head. We've totally lost the ability to keep track of these things. We've lost the ability to manage in any one person's head, let alone keep something consistent across the whole team of folks' head. I think now we're looking to new techniques to treat it more like a supply chain. If you're FedEx you can't track every single package. Or if you're Amazon, or if you're Toyota, you need to use other techniques and tools to manage your supply chain. I think that's what's driving this impetus to start to think about the software supply chain, which is, when it ships when it leaves the developer's hands, so to speak, through the rest of the supply chain, how do we keep track of those things?
Swetel: I have a question, if I could ask a follow up. Should we hold everything in our head? Should that be an expectation?
Edwards: No, but I'm saying in the past we could. It was less necessary. We could keep things in front of us and think about it, and now I think we have to have new techniques because we can't. The world has turned complex. Or maybe it's always been complex.
Rockwood: The reality is we're trying to build software faster and faster. We're trying to build more complex systems and we're building those on top of primitives – an increasingly large number of primitives. I think the thing that's a part of the problem of almost everything, but certainly we see it's a problem here in software supply chain, is trust. It's like we go and we grab libraries, and we implement them because that's what everyone uses. Why is it a good library? Why is it a safe library? Because most people use it. I saw it on Hacker News and it's got more stars than everything else so I'm going to use that.
How do you know that you can trust that piece of software? You generally don't. You're taking people's words for it. We see this in the real world, particularly the health market. We see this in the food industry all the time. One day this is healthy, the next day it's not. Why? The people that we trust told us that it was good and we trust them, and then all of a sudden they change their mind and it's not so good. In software we depend on an incredible number of things. We know that's particularly important for security. If I can hack your app, ok, but what if I can hack OpenSSL? Everyone's built on that. That's a much better vector to go for.
Now as we take it upstack into cloud services, we're all depending on cloud services like RDS, and ELB, and whatnot. If I can attack those vectors you're probably not actually paying much attention to those pieces of the supply chain. One of the things that's interesting to us at Packet, we're bare metal cloud. One of the things that we think about is not just software supply chain but actually the firmware supply chain. These machines come out of Asia. They come to us, we make sure that they have the latest firmware. These machines today have so many pieces of firmware and so many layers of firmware that it's really impossible to know that anything is secure or it's working as well as it can. There's a huge amount of trust that has to be applied, and it's difficult, and dangerous, and there are a lot of places where people can attack that supply chain, and also disrupt service.
Moderator: I'm going to bounce around, because I fed them some of the ideas that we're going to talk about, but I see a lot of you falling asleep so I want to wake you up a little bit. I'm going to go to you, Cat [Swetel], for this one. In general, one of the things that we have gotten from lean is tools. Your presentation is about sociotechnical, and there's a set of tools that have been used over time. I know you're all familiar with some of these lean value-stream mapping, all these things that's done. You've taken it now to this thing called Wardley mapping. Give me a sense of your evolution of what you use as tools, and how did you get to Wardley mapping, and why is that important? I don't want to steal too much thunder from your presentation.
Wardley Mapping – Evolution and Tools
Swetel: How did I get there? I guess because I didn't come from anything cool. I don't have a CS degree, I didn't go to a fancy school. I grew up poor, I just had to use whatever tools I could to level the playing field, and so I guess that's what drew me to things like Wardley mapping. That's not something that's people in fancy blue suits bestow upon you. There's no magic behind it. It just is what it is, and it allows you to figure out what you know and not question it. I don't know whatever the fancy people – I won't say company names – say, so that's what I liked.
Then it does really irritate me when people who work in the technology space say that we should be thinking all about psychology, and just the technology is irrelevant, and it's just doing whatever it does. That is an abdication of your free will and I find it offensive. I like that Wardley maps don't try to ignore that.
Edwards: Give us that 30-second definition of what a Wardley map is.
Swetel: It follows, basically, the steps that were outlined in the original socio-technical theory from Trist, and Emery, and all those dudes. First, you take what are the components that are needed in order to deliver some unit of value, and then you look at what are the relationships that are needed in order to deliver that unit of value. Then, you look at it all together as a system, and so that's what Wardley mapping helps you do. It provides a discipline for doing that, and you're looking basically at how we as a society treat those components and how firms treat those components. You can learn a lot of interesting things.
If you, a firm, are treating a component in a different way than the rest of the industry or society as a whole, then that can be an advantage, or a disadvantage, or whatever the case might be. Also, it just helps me know what I know. A lot of times you have a feeling or something like that. You can't really say why, and going through the process of mapping allows me to articulate why I have that feeling, or potentially why I shouldn't have that feeling anymore. More not so much trusting other people or trusting components, but trusting myself.
Rockwood: Isn't one of the cool things about Wardley mapping that as opposed to being a static map of the system it actually has a time component?
Swetel: Yes, it has a time component. The x-axis is evolution which happens over time, but maps are maps. As soon as you write it down it's already outdated. It's just an artifact.
Moderator: I remember Simon Wardley is the gentleman who created maps, and one of the things when he first started introducing the idea, he talked about crossing the chasm with single dimensional. There was these evolution-y stages, but that's it. He was saying that when you do strategy and all, you have to think of it as there's lakes and there's hills. The mapping is based fundamentally on idea, but then there's a multi-dimensionality onto how you really think.
Another thing I've heard is somebody called it the anti-PowerPoint. We go in and we do strategy and design by everybody looking at one slide. This is something that you can draw on a whiteboard. In fact, I used to joke sometimes. I'd go in a company and start doing a map, and they'd be, "You've been here five minutes. How could you know anything?" I'm like, "Here's the marker. Take over, do it." What about some of the other folks? You've spent a lot of time value-stream mapping, and that's been pretty effective.
Edwards: Yes, that's a different thing. Going back to the lean thing we were talking about earlier, they have this notion of go into the Gemba, which is don't be a manager who sits. If you imagine an old factory, the manager sits on that second floor looking down. Don't be that manager that sits up there and just goes, "Look at all my people working," I'm looking for the reports at the end of the month and wondering why the people aren't doing what they thought they wanted to do.
They have this idea of go to the Gemba. To be a manager you have to go down and actually watch the workflow. See how things are going, see where the wastes are, see where the problems are, hear what people are telling you. In this technology world that we live in there is no Gemba. We don't walk to people's desks. What does that tell you? The Gemba is inside people's heads. John Allspaw has this great idea, it's not only his, but from a system safety world they talk about above the line, below the line. Above the line is all these representations of things that we work on, and below the line is the actual system. We can't actually see the system working. We can't see the zeros and ones flying by. All we see is our interactions with these interfaces, these tools, these config files, these ticket systems.
The idea of what the Gemba is in my head is probably different than what's in John [Willis]'s head, in Cat [Swetel]'s head, in Ben [Rockwood]'s head. We can't actually look at the factory. We all have a different view of the factory in our head. I know John [Willis]'s done this a lot. I did this back in my consulting days. Basically get on the board, and you use value-stream mapping notation, but it's not really about creating a value-stream map like in the traditional manufacturing sense. It's really about graphical storytelling to say, "Let's just start from the beginning, and let's talk specifics." Not, "Let's talk what generally happens. What happened on the Tuesday release?"
Get the people in the room who are from where the requirements came from all the way through to what happened, and just walk through and do a black pen pass to say, "This is what happened." It's incredible how everybody's, "We already know this," and then 10 minutes in people are, "I had no idea." Then you can look for the problems, and the flows. The whole real trick of it is that you're creating that Gemba on the board. Why that's important is now we have a visual representation we can all start talking about so we aren't just talking past each other.
It's incredible how much gets done and how much improvement falls from when people just all share that same mental model, even if the mental model isn't totally accurate. If everyone is on the same page it's incredible how people go, "If we fix this and you did that, wouldn't this flow a whole lot better?" "Gee, it sure would," You see things start to move, when before you have a lot of arguing and nothing ever changes.
Value-Stream Mapping
Moderator: Something I always find amazing when I've done value-stream mapping is you get a bunch of people of all different experts to one service, and you start doing these boxes. What's the first thing that happens? You're supposed to go right to left but that makes everybody's head burst.
Swetel: It's supposed to make everyone's head burst.
Moderator: I just find I waste a lot of time. I get it, that is the canonical lean value-stream mapping tool, but I just find I lose more people.
Edwards: I think just start like training wheels. Just get people used to the idea of oh, there's value in seeing this on the board. Left to right works fine, but the nice thing of right to left is eventually you're going to get people to say, "Let's start with the end in mind and then let's go backwards of all the things that we had to do." It uncovers all kinds of problems of "Why did we even do all that?" Yes, you have to start somewhere.
Moderator: The canonical version is right because that's what the customer gets. I was just going to say that what's interesting is when you start, whatever direction you're going, you start seeing these boxes. Somebody's like, after about 30, 40 minutes they hadn't said a word, and they're, "Hold on, there's five boxes between those two boxes." It's this magical thing you start seeing. Then somebody is "I always wondered why it took so long to get from there to there," and it turns out somebody sends an email to somebody.
Edwards: One more thing I want to add about that that's funny is, when you ask people going into these sessions, "So what's wrong around here?" it's like, "We don't use Chef properly." It's always some thing that everyone's holding in their mind that's, "This is the boogeyman that's slowing us down." Then, when you actually lay everything out and they list out all the problems, whatever that was is invariably it's either not on the board or it's tenth on the list. There's all these things out there that no one even thought about, and they seem so basic.
They're like, "That can't actually help us." You actually see it all put together, you're, "If we just do this before this, or if we just separate out those config files into two files," and so the things that actually change are only three or four items. We would stop having all these fat finger errors. I mean, it's these little things jump to the top of the list that have a huge impact. The folklore of, "We don't use X tool properly," or, "We don't have have Y,".I've never seen it once be the top of the list.
Rockwood: The thing that we certainly learned out of what happened in the '30s, '40s, '50s, '60s as things got more and more complex is that we have small brains that are really analytical. We can drill into components really well, but when we have to step back and conceive the whole system we really struggle. The tool that we gravitated towards was just visualization in general. There are a lot of different ways to visualize different things, and it's important to not get caught up in any particular tool. Kanban is just a way of taking work that you're doing and making it visual in a way that becomes a lot easier to process. I think we've all seen tremendous gains from that.
User story mapping is a great way to take your product development and the work that you're doing, break it into components, and then be able to work in two dimensions rather than one. We don't tend to work well in two dimensions inside of our heads, especially not in a collaborative way. Value-stream mapping started as a specific tool for doing the map of, or seeing a system and its different components in the industry. Certainly we can draw parallels to that, but what you see today is, typically in our CI/CD build systems, now we have complex build systems that process things from a code repository all the way to running in production.
If we can take that, break it into phases, and then visualize those phases, we can get a lot of different economies of scale, and improve each of those components of the system. When you commit something you don't actually need to see a visual representation of your pipeline building, but we all really like it. When we can take those things and improve each of those components over time we get a better system and we can work in a much more collaborative way.
What DevOps Mean to Developers
Participant 1: I'm a manager. You're not speaking development. I know everything you're saying as a manager. I don't know how to actually apply it to the developers. What does everything that you mean, mean for developers? Can you actually not talk about the theory but about people at their desks? What does lean and DevOps mean to them? What does the tools that you're describing mean to them?
Edwards: If you want to just speak to them personally you can talk about the benefit to the business. I think the reality is there's a frustration that a lot of people feel that when they're working against a system. When things aren't flowing, when they aren't getting that fast feedback you lose touch with the work; it's a miserable place to be. I think on a very human level there's just this ability of removing friction. They're, "Why is development and operations always fighting," It's not because they're cats and dogs, or there's not some innate law where these people fight. It's because the friction between their work and the competing goals, and they both see the system from different perspectives.
Trying to apply these technologies, or these tools, and these processes, you can talk about how it's going to help the business. It's going to help things flow. I think fundamentally being able to tell developers, tell people in operations that, "Look, this is going to remove that friction. It's going to remove that frustration." By getting people together and looking at how the system flows together, they're like, "You mean that thing that's been bugging me this whole time? Huh, we can fix this if we just do that over there," but it's usually some place, somebody else's responsibility. They go, "I could totally do it that way. That wouldn't be hard at all. Thanks."
Little connections like that, because they see the end-to-end system and they understand "The thing I'm doing pushes all this pain on that group over there. This thing that group over there is oblivious to or think they have to do is pushing all this pain onto me." If you get them together to talk about these things they'll start to resolve that friction because people don't want to have friction in their life. When you resolve that friction work will flow through the system better, and as a business you'll perform better as well. I always just take it down to friction. What gets in the way here? What gets in the way of you doing your job? Let's get people together and go fix that.
All these tools or these techniques we're talking about are just to visualize and surface that friction so people can solve it themselves. Often, as the manager, you can't see it. It's buried in people's work, that friction. How do you surface this so they can solve their own problems and do it in a cohesive way so the work flows," which is good for business?
Moderator: Cat [Swetel], you were a quick developer a fair amount.
Swetel: Yes, I'm an engineering manager so I believe there are developers in the room that I work with each day. For me it's about giving people the resources to articulate where there is a problem and why it's a problem worth solving. I think there are things that we can't talk about in organizations in general, and in technology organizations specifically. I think a lot of what the individual benefit of lean, or DevOps, or any of the techniques they're in, it's about giving developers the resources to say, "This is a problem and I believe we should solve it."
Moderator: I'm going to jump in. David J. Anderson, in general there's this debate, but he's the person who defined Kanban for software. If you read one of his earlier, why he came up with that, he was looking at lean, and Toyota, and flow, and people like Eliyahu Goldratt. That whole idea of visualizing flow in a way with limits, work-in-progress limits, that all comes from lean. That's just one example. I was co-author on a book called "DevOps Handbook," and we tried to do as much as we could to attribute. There is a lot of similarities between mechanical flow and work assembly lines and knowledge work. I think that's the answer is the more you can look at 50 or 60 years of somebody like Toyota just doing incredible things in the market, a lot of the DevOps lean conversation is around, what are those things that actually map really well to knowledge work? Kanban is an example.
I mean, some of this stuff Ben [Rockwood] will talk about later works really well. I think the answer is the more on the things that we can treat as an assembly line, not everything. Ideation is a hard problem, but commit to production, which your developers are reasonably involved with. I know we're sounding very meta, but that was our starting point.
Cargo Culting
Rockwood: Something I'm going to get into later but we'll bring it up now is, how is that cargo culting working for us? Not so good. How many companies have started SRE programs? They went into their engineering team, they went into their operations team. They went, "You guys sit with them. You sit with them." They were remote. You have a bunch of meetings now and we're going to do SRE. I'll write a book about it, it's great. Just do that. How did that work out?
Swetel: Arguably pretty well. I mean, all these companies that came up with that fake SRE group are still making bazillions of dollars. Is anyone here working for a company that's just going down the shitter right now?
Rockwood: Did anything change? You took a team that used to be system administrators and then you called them operations. Then you changed them from operations to DevOps.
Swetel: I'm not saying anything changed.
Rockwood: Then you changed them from DevOps to SREs. Did anything actually change inside the organization?
Swetel: No.
Participant 2: Our SRE team is made up of some really very experienced, highly qualified troubleshooters that are always present. I mean, I think they're great. They do a really good job and they bring a certain amount of cohesion. They bring people in. They know who to call. I think it's going well for us anyway.
Rockwood: Sure. I'm not saying that those groups don't have value. They've always had values, and that's the tribe I come from. The question is, are they something other than just your sys admins, and how so?
Participant 2: I work for a software company, and three of the members of the team are highly certified in our product, and come from a background where they worked with our product on-prem. Nobody else has that experience. They bring a ton of experience with what goes wrong with our application in production environments.
Rockwood: Are they writing code?
Participant 2: Yes, they are. Yes, one of them is born in the code coder.
Rockwood: Nice, and how big is your engineering organization?
Participant 2: We have 1,000 employees, probably 600 developers.
Rockwood: Ok, nice. How many SREs do you have?
Participant 2: I think there are five or six at this point. I think the team is still growing.
Rockwood: Five for a 1,000-person engineering team?
Participant 2: For our cloud offering. Our RSD group that does the cloud offering for our product is 150 people.
Edwards: This reminds me of should there be a team called the DevOps team. I think sometime giving a name to something and putting attention to it can be a good thing. Eventually could it impede further progress, or maybe it is just re-branding something, but the fact you re-brand it gives people a new invigoration to go.
Swetel: Even just getting the [inaudible 00:25:53] resources to make sense of your world. Yes, if you give something a name and you say, "These are the things that we would associate with that," then even if those folks, those five or six awesome folks, they probably existed as awesome folks before that. Now that we have this SRE language, a distinct lexicon, we can give that to people to make sense of the work that they do, and work that was probably before undervalued.
Rockwood: Yes, and I think the key that we're going for is adaptation. This is working for you today. The challenge that we really see is when people want to go from a handful of really smart subject matter experts inside of an organization and turn it into when you want 200 SREs, how do you replicate those individuals who are star performers? A lot of organizations tend to have those star performers and they're great, but then when we try to scale them up it's really hard because those individuals have become who they are because of their own tenacity. They rolled up their sleeves, they worked really hard.
Participant 2: If you need 200 SREs, you got a bigger problem.
Rockwood: Remember, one of the things that spawned this, let's remember what got us to where we are. One, the world is based on technology. We're writing software in an increasingly large rate. The other thing is that we're not just writing software, putting it into a box and then selling it. That was a model that worked in the '90s. We knew that model pretty well. This is what got us to separation of developers and operators. Now that we're in a world where we're actually operating the software that we build and increasingly not putting it in a box and shipping it to people, we need to have a very tight relationship between the people who operate the software, the systems that operate the software, and the engineers that are actually building that software. This is why we can move to an entirely agile perspective in terms of shipping code five minutes after we've written it and putting it in production. This isn't something that was possible in the past. There's a certain amount of adaptation in trying to take our organizational structures and the individuals in our staffs and aligning them closely to that new model of software delivery.
Swetel: I think that word is the magic word. Relationships. We're coming from this highly transactional model and now we've gone to where the software that we develop and deliver is a relationship, but then our organizations didn't form those relationships to support that model. The market is demanding that we provide a relationship, but then within a firm we haven't developed the maturity to actually deliver that. That's what we're trying to do.
Moderator: I think the other thing is, what Ben [Rockwood]'s point was in that, we're not saying SRE stinks or it doesn't work. I think everybody on this panel believes that model is the right way to do it. The problem that I see in large banks is they're all trying to take old legacy into the teams and renaming to SRE teams without making any changes. Then, they're fighting the prescriptive model of errors budgets and SLOs. What they don't realize, that's almost impossible in a bank where your system of record is sitting on a Db2 database that is front end by things like Webster and Mq that go through MuleSoft APIs, that then come back to modern jobs, that then talk to front ends that get to the phone. It's really hard.
The thing I tell people, most banks, you're not Google. Google has one abstraction. It was called the Borg, it was called Omega, we see it as Kubernetes. It's really easy to make an SLO contract between developers and infrastructure where developers don't have to carry any amount of infrastructure. In a bank with 50 years' worth of vertical technologies and hundreds of dependents, it's very hard to do error budgeting as well. I think there's a cargo culting nature of larger companies saying, "I want to be like Google. We're going to create an SRE team." It's really just the same, that they're doing iTool. They're doing old stuff and they just call themselves SRE.
Biggest Challenges
Participant 3: What I want to know, because you have sessions, and actually I think in all of them put as my favorites, because the topics that you discuss are actually very important for the industry, as you said. They're getting more complex, they're pushing for speed, and it's hard to do. Everything that lean and DevOps are doing is focused on that. In your experience in the field, what are the biggest challenges for you when you actually are trying to implement these practices?
Swetel: Golden handcuffs. I come from highly regulated industries, healthcare, and financial services, and all that jazz. Those companies, I don't know if you've heard, but they make a lot of money. When you go there and say, "I think I see these problems," people will tell you, "That's not a problem. Do you know how much money we make? It's not a problem," For me, the biggest issue that I run into is, "We've been wildly successful for," in some cases, "Hundreds of years, so why should we change anything when we have all of this historical data that says that we're awesome?"
Participant 3: Can you also add how you approach [inaudible 00:32:16]?
Swetel: I think it's really important to have respect for history. I do just think it's shitty when people speak really negatively about mainframe, or whatever the case may be. I think it's important to show respect for history and say, "Yes, you have a long history of success. If we want to continue to serve that legacy" and I mean legacy in the cool way, not the vintage way. If we want to continue to treat those components with the respect that they deserve and to treat whatever this history with the respect that it deserves, then we need to think about how we are delivering the software or whatever.
Moderator: I was the ninth person at Chef and I was the old guy. We'd go into these clients, and really people I love still to this, would go in, five minutes in they'd say, "You're doing it wrong." I'd have to pull them out of the room "You're telling people who've been there 25, 30 years that have, in some cases, helped build the company, that they were doing it wrong. You can't do it." To your point, there's a history of how that company makes money, and I think that's a general message. We can't have that hubris. Our passion can't be seen as hubris. In other words, we're passionate "We got to fix this." You got to understand, they might hear that as, "They're stupid and we're smart." I think I'd double down on what you just said.
Edwards: Yes, and I think the one message you could always sell is just take it down to what gets in the way. Everybody will have an answer. You walk into anyone's office today, "What gets in the way of you doing your job?" Everyone will just start talking. They'll have enough things that annoy them that get in the way that stop them from doing what they need to do, and they'd be more effective in what they're doing. I think if you break it down to that fundamental of a message and ignore all the other labels, then, say, the passion that comes with it, it goes a lot better.
Personal story, this is back in our consulting days. We were called in by a CIO of a large insurance company, and the operations team was really not on board with the whole DevOps idea at all. We were doing these workshops just to try to get people to learn how to do that graphical representation. We had been to dinner later that night with their exec team to talk DevOps, and the head of operations is, "I like you guys, you're ok." This whole drawing on the wall, see what gets in the way, we'll fix that, and it's fine. This DevOps thing, this is BS. We're going to get rid of it. We got rid of those agile folks. We're going to get rid of these DevOps folks, because if those product folks can't make up their mind once a quarter on what they want to do, don't tell me I'm bad at my job. They're bad at their job," You're just, "Wow, this is a major..."
Swetel: Obstacle?
Edwards: Yes, exactly. This is a household, very successful organization, but that was the mindset, because it was always approached as this dogmatic thing, and they got it all wrong. What appealed to them, and it really broke down that barrier of entry was just, "I like this way that we go, and find, and fix what gets in people's way," That's the most human thing you can get down to, and it speaks to the business level as well in any company. You could always make more money, you could always be more effective. We're just going to focus on, how do we find and fix what gets in the way? If you get the organization to catch that fever and do it systemically, you've seen that IBM quote, "Make elephants dance."
Rockwood: Yes, I think in my day to day the biggest challenge is time. Being reflective takes time. Stepping back and looking at the bigger picture is extremely time consuming. With your engineers, for managers, if you have engineers, how often do you talk to your engineers and say, "Where do you really want to be in the next 5 years, 10 years? What does the future look like for you?" A lot of times they're not thinking like that. Most of them, a lot of them aren't. Some of them are, but not a lot of them.
The same is true of us in our organizations. Where are we today? Where are we going to be in a year? Where are we going to be in two years? Not the product, not the company revenues. We spend lots of time thinking about that, but what is your organization going to look like? What are your systems going to look like? Do you even know how your systems work today? In a lot of cases, particularly smaller and younger companies, we don't even have that mapped out. Things like mapping tools, some of the tools that we've developed, and more importantly that we adapt to our organizations, can really be valuable sense making tools, so that as we invest in different parts of our business, we're doing it in such a way that it strengthens the entire system and makes us stronger as individuals, stronger as an organization, and builds the systems together.
Otherwise we end up cargo culting these things out, and I also call this techno fetishism. How many organizations are going out and implementing a tool simply because, "It's the tool we're supposed to implement now"? We did the same thing with organizational practices frequently. If we don't have a strong conviction about how our system works and what its strengths and weaknesses are, we're prone to just take whatever the new thing is and apply it indiscriminately, which just really kicks the can down the road.
Moderator: It's not just revenue. I tried to sell services to one of the largest soda companies in the world, and one of the things that when you go in and you talk to people that are actually a small group that's trying to do transformation – some of those might be in the room, that's why I'm not saying. The end of the day, this company, all they do is sell syrup. They just sell syrup to franchises, that's it. By the way for a penny they can get millions, maybe billions. Every penny they spend on marketing, because the brand is so big, it just dwarfs anything you try to do to say, "Let's try this technology."
The thing is is that having happy people and not burned out people, and a healthy social work force, there's value in that. The reason Toyota was incredibly successful was everybody had a mentor-mentee relationship. Read Mike Rother's "Toyota Kata" book. It's an amazing book, and he'll tell you the real deep side of what happened there. These people, they were prideful. They understood every role they had in the pull system. They literally had a healthy mentality at their work job.
A lot of these companies could make money really quicker by just turning some – in the financials it's a quant who can do one little bit twiddle and make more than we do with transformation. It's a bigger social picture of how people behave, how people get along. What's your career in life, and burn out, too?
Edwards: I want to add something to what Ben [Rockwood] said that's very important. The going slow to go fast idea. Let's stop and have a retrospective. Let's stop and actually figure out how things work. That is against every impulse in large organizations. I think from the management perspective, anything you can do to try to create that capacity, even I've seen companies where they just basically bake a 20% extra time into every estimate and just don't tell anybody about it. That's just their budget that they're going to use to do all the analysis and the improvement work to make things better.
There are hacks and things at a management level. I think that's one of the best things you can do is just try to get in the way of that impulse to say every hour must be filled. We must go faster and faster. The reality is, if you want to go faster as an organization, a lot of times you have to go a little bit slower. It's counterintuitive. Anything you can do to create that space by hook or by crook, I think you're going to do a lot.
DevOps Mindset
Participant 4: There's no way you guys are going to answer this one in the time allotted. Here's the situation I'm dealing with right now. I'm working at a company where the executive team is bought into a DevOps mindset, and they've all read "Accelerate," and they've been preaching the book, and they have all the high-level buy-in. At the lower level they're getting frustrated because they're wondering, "Why isn't stuff moving faster? These teams are fully empowered." If you ask the teams, they're going to say, "All you do is tell us to do more stuff. We don't have the system permissions we need. We don't have any ability to go procure what we need. We have not enough people and we keep piling it on." How do you close this chasm?
Rockwood: Go to the Gemba. That's the idea of going to the Gemba, where the work is done, or where the value is created. Did any of those people read the book actually go and sit with the engineers for a couple hours? Not in a conference room, but actually at their desk while they work?
Participant 4: In one word, no.
Rockwood: No, of course they didn't. We've seen this. Motivation is a whole series of discussions. There are lots of books that have been written about it, and to some degree when you start getting to, how do I connect lean ideas or DevOps ideas, where all the stuff gets really happy, touchy, feely, we want to share our feelings, we want to contribute and collaborate, it generally comes down to trust and it comes down to motivation. What motivates them to actually do that? A lot of that is rooted in trust, and that comes down to a shared sense of purpose, a shared vision, and a shared sense of the challenges that we face.
If you're just going to say, "This is our new initiative, go faster," that would be just as effective as giving everyone a $20,000 raise and saying, "Now you'll go faster." It might for a second, but it's not going to be sustainable. There are systems in place that have to be looked at, understood, and broken down, and the first thing is to get engineers and everyone in the organization to feel that their voice really does matter, which means their voice actually needs to be heard and responded to, and there needs to be empathy engaged.
It's really hard to do that in a fast-moving business. It's very easy to talk about, it's very hard to do. This is one of the things that's really interesting for us as a DevOps movement to look back at lean. If you read "Gemba Walks," and a couple other books out of that space you'll find that a lot of companies did exactly what we did in DevOps in the early days of DevOps. They adopted the tools, they liked the ideas, but then they were just "Go therefore and sin no more," and that was it. Nothing really changed. Ultimately, to just cut it short, the way that things really change, crisis. This is one thing we know from literature and whatnot. Things don't really change until life is threatened, then things change.
See more presentations with transcripts
