Delivering Great Software and Building Great Teams at CircleCI

Transcript

Shane Hastie: Hey, folks, QCon London is just around the corner. We'll be back in person in London from March 27 to 29. Join senior software leaders at early adopter companies as they share how they've implemented emerging trends and best practices. You'll learn from their experiences, practical techniques, and pitfalls to avoid, so you get assurance you're adopting the right patterns and practices. Learn more at qconlondon.com. We hope to see you there.

Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today, I'm sitting down with Rob Zuber. Rob is the CTO of CircleCI, and has been there for about eight years, I believe, Rob.

Rob Zuber: Yeah, that's right. A little over.

Shane Hastie: Well, first of all, thank you very much for taking the time to talk to us today.

Rob Zuber: Thanks for having me. I'm excited.

Shane Hastie: And let's start with who's Rob.

Introductions [00:57]

Rob Zuber: So as you said, CTO of CircleCI. I've been with the company quite a while now, joined through an acquisition, but it was a very different company at the time, small company acquiring small company sort of thing. And so prior to that was also doing some CI and CD related activities. Specifically around mobile, was the CTO of three of us. We didn't even really have titles. It's kind of silly at that point, but many startups before that, so have been, I guess both a practitioner from a software development perspective, a leader. And then now as you know, we're specifically in the space of CI/CD, trying to help organizations be great at software delivery.

I have this really fun position of thinking about how we do it, but also thinking about how everyone else does it and looking at, oh, well that's interesting. I'd be interested to learn more from this customer. Or going and meeting customers and sharing with them and understanding their problems and talking to them about how we do things. So for me, it's having worked in many different spaces, from telco to consumer marketplaces to embedded devices. Selling to me, I guess is the best way to think about building a product for me as the customer, both as a leader and an engineer, it's just been a really, really fun ride. I've always enjoyed software and it's so fun to just be at the core of it.

Shane Hastie: So what does it mean to be great at delivering software?

What does it mean to be great at delivering software? [02:09]

Rob Zuber: Oh, wow. At the highest level, I think being great at delivering software comes down to a couple things. And this is not necessarily what we would put in our stats or the DORA metrics or whatever, but feeling like you have confidence that you can put something out in front of your customers and you can learn from it successfully. And I don't mean feeling like everything you build is going to be perfect and it's going to be what customers want. Because I think that's never true. What's true is that you'll get to what they want if you feel really comfortable moving quickly, trying things and putting things in front of your customers in a way that doesn't throw off their day, but gives you real feedback about what it is they're trying to do. Whether the thing that you've done most recently is helping them and really kind of steer towards and adapt towards the real customer problem.

I'll say customers, but I'm a customer of many different platforms that this is so that I'm including myself in this. We're sort of notoriously bad at expressing what it is that we actually, the problem we want to solve. We sort of say, "Oh, it would be really cool if you had this, or it'd be really cool if you had this," but we don't understand what it is we're talking about 50% of the time.

I think designers are on the receiving end of this more than anybody. Everybody's a designer, but the best they can come up with is, "I'll know it when I see it." And so feeling like you can really move quickly so you have all the confidence in terms of the quality of your software and knowing what you're building and that you're building it correctly, but you being able to use that to really adapt and learn and solve customer problems. If you're not solving customer problems that are real and are important and are a pain point, and ideally doing that faster than your competition, then there's areas where you could be better, I guess is the best way I would say.

Shane Hastie: Within CircleCI, how do you enable your teams to do this?

Building a culture of safety and learning [03:50]

Rob Zuber: One of the things that I called out in there is having the confidence that you could do that. And a lot of that stems from tooling. Unsurprisingly, we use our own tooling, we think about the customer problem, try to orient ourselves around the customer problem. That's all important. But that confidence piece is also in how you behave. Building that model that says, that mental model that says we are trying to learn. What we're trying to do here is we're trying to learn, we want to put things out. Those things are going to be small. There's not an expectation that we're going to be perfect and know everything in advance, but rather we are learning by testing. We're starting with a hypothesis, we're validating that hypothesis and we're building on top of it. And so I have my own podcast and on that, we spent the last season of it talking about learning from failure.

And I think that's a really, really important part of this is, A, how you frame failure in the first place. There's a big difference between, we thought we were going to be perfect and it didn't work out versus, we're trying an experiment, we have a hypothesis, and we know going in that we're trying to learn. It creates a safe space to learn and move quickly. And then how you react when you actually did expect something to go well. And that might be maybe it was an operational issue or something else like that. And creating the space where the response to something going poorly, let's say, is great. What can we learn from this? How can we be better next time? Versus how could an individual have done this thing? Sadly, this comes up probably in all engineering organizations, but you'll see the difference. The issues come up everywhere.

The difference between you should have known better and how on Earth does the system allow you to be in a place where this could happen? Those are very, very different questions. And my belief at this point is, if you allow humans to do things in systems manually as just as an example, you've set yourself up to have a problem. And it's certainly not the person who did the manual thing and finally made the mistake that everyone knew was coming sort of thing. So A, it's that overall cultural picture. And then as you start to identify those things, you say, "Okay, cool. How do we fix the system? How do we create a system where it's safe for us to experiment and to try?" The culture has to be safe and the systems have to be safe. You need to have risk mitigations basically to allow you to try things, whether it's the product or shipping new capabilities.

Shane Hastie: How do you make sure that you do that when things go wrong?

Leaders need to model safety and curiosity, especially when things are going wrong [06:11]

Rob Zuber: It's an excellent question. I think if you're asking me personally, and there's two levels to this, how do you do that personally and then how do you build that in an organization? I think the best thing that I can do as a leader is signal. My behaviors, unfortunately for me, carry more weight than other behaviors in the organization. If an individual engineer says, "Why'd you do that?" They're peer individual engineers, like, "You don't understand how hard this is. Whatever, I'm going to get on with my day." When a CTO shows up and says, "Why would you do that?" That carries a lot more weight. So really, it's hard, particularly if we're talking about, we'll call them the catastrophic failures. Hey, we were experimenting, we knew this. That's easy. It's easy to set up, it's easy to structure. We knew that in advance. The catastrophic failures actually, we allowed this project to run for nine months and it turns out it was a complete waste of money or something happened and we were operating with the best information that we had at the time, but that information wasn't very good.

And now we've got a big problem on our hands. Those are usually crisis moments. And I would say people's guards are down. You're not carefully crafting every word. And so it's practice. That has to be the habit. And I will say it's been a long time and nobody's perfect. I would say I'm not perfect, but acknowledging and recognizing the impact. And so still speaking to the leader point of literally everything you say and do and taking that moment to stop and think, "Okay, what is the opportunity here? What is the learning opportunity? How do I help people get through?" Often it's the crisis in the first place. How do I get through this situation? And then, how do we learn from it? And also, I mean if we're talking about specifically those crisis moments, usually there's crisis remediation and then sort of learning and acknowledging and separating those things, I think it's easy.

I've seen this behavior before to show up in... We're still solving the crisis and start asking questions, how could this ever have happened? And that is just not important right now. We need to get through that moment and then we need to make sure we have a long look to say, "How did we get ourselves into this situation? What is it about?" I mean, the reality in the situation... I've used examples, just this required human intervention and humans ultimately are going to make a mistake. It just happens, right? I don't care how well you document something, how careful you ask someone to be. It doesn't help. You need a system in place. So there's that. But if you look at other more complex scenarios, what you'll find is they're complex. Someone else has been asking for something or creating a sense of urgency around something that maybe wasn't as urgent or something was misinterpreted.

Backing your way through all of that is challenging. And I think again, for me, it's about curiosity. If you're truly curious, if you're genuinely curious about what is it in this organization that's leading to these types of scenarios, well, I guess that's number one and then if you act. Right? If I say, "I'm open to your feedback, what's going on?" And then you give me some feedback, I'm like, "No, no, that's not it." Are you going to give me more feedback? I doubt it. Right? That was just a waste of your time. And it was. You're like, "No, I'm literally seeing this problem." If I say, "Oh, tell me more about that, let's sit down and think through it. What could we do differently? How would this turn out differently next time?" And then go, I mean you're not always going to succeed, but go try to implement some change around that so it's safe and the action gets taken, then that's where you start to have people offer up ideas. Right?

And come out and say, "Actually, I was in this scenario where I made this mistake or whatever. I didn't understand the..." I mean I'm using sort of specific examples, but it could be anything from I decided to launch this feature that it turns out wasn't helpful and we invested money in it. We had a production issue where we, whatever, pick any example, it maps anywhere. We honestly, openly, curiously explore the issue, right? Sure, we can say this person took this action, but when the reason we're asking, understanding that is, what did that person know? What could they have known? Why were they prevented from knowing that thing, et cetera. Then that's an open, honest, just curious exploration versus I need names, whatever. And so all of that, I guess I'll just stop at signaling. At a high level, if your leaders don't behave in that way, the rest is a waste of time to talk about, honestly.

Shane Hastie: Looking back over your career, and we can start the progression from developer through to senior exec today, what would you share? What advice would you give others who are looking at the same pathway when they're along that journey?

Advice for aspiring leaders [10:26]

Rob Zuber: The first piece of information I usually give is don't take my advice. Mostly because my progression is not at all a progression. So maybe there's something in there. I started out working in an electronics factory actually doing process engineering because that was related to what I had studied in school, which was not software. Built some software to kind of understand our systems. Enjoyed that. Went to work at a startup, late '90s, it was the time and the rest is history. And then did a bunch of small startups. So I didn't really have a clear progression, but it's actually one of the things I talk about with a lot of people getting into the field or even considering what field they want to get into, which is that you don't have to have a straight progression. You can bring value to yourself in terms of this expression from the book Range, “match quality”, the thing that you end up ultimately doing, being a really good match for the strengths that you bring to the table, the most likely way that's going to happen is not by filling out some form before you apply to college.

It's actually going and doing a bunch of things and saying, "Wow, I'm really good at this and I'm really bad at this." And just being able to say, "I'm actually really not that good at this," whatever. I mean, I was a head of business development in an organization for a brief stint. I have a ton of empathy for people who do business development and sales. I think that makes me a better executive, but I was absolutely garbage at that job. I was like, "Oh, we're doing business development. I should probably write some code to get this done." The idea of cold calling, trying to build relationships with business, it's not my thing. So that's fine, good to have done it because it helps me understand where my strengths are and it helps me understand what other people might do. And so I think a lot of people get on this, I need to be a developer and then a senior developer and then a staff and then this, and then that.

And then if I'm not moving forward, I'm moving backwards, but part of the non-linearity of my career is I've been a CTO for over half of it because I tried a bunch of things and then I went and started some companies and I was like, "Well, there's no one else here to be the CTO, so I guess I'll just do that." And it wasn't the same CTO job that I have now, but in particular, if you want the full spectrum and I guess a crucible of learning, starting a business and being responsible for other people and trying to build a product and trying to find product market fit, all of those things is a great way to get that learning. But it's not for everybody. So there's other ways you could get that learning, but really for me, I kind of ignored the path, not because I was brilliant, but because I didn't really have a plan and I believe it paid off for me.

Shane Hastie: Circling around, why is the culture at CircleCI good?

What makes the Circle CI culture good? [12:54]

Rob Zuber: Well, I think there's a few things that play into it. I mean, one, company was founded by people. So I'm not a founder of CircleCI. When I say we, there was three of us that were acquired in and one is now the CEO. So we stepped into an environment where we were fortunate to inherit some things. And the company was founded over 10 years ago, but culture lasts. And so we stepped into a group of people who were genuinely curious about a problem, cared deeply about their customers, and were building a product that people were excited about. The people that worked there were excited about, the customers were excited about. So there was a lot of connection between customer and the engineers building every day. Of course, in a 14-person company, which is how big it was, there's always going to be that kind of connection.

Might be possible to not have it, but I mean, we did our own support calls and everything else, you're much closer. But being able to tap into that and build on it and then make tweaks as you go, I think is a great thing to start from. We weren't coming in and saying, "Oh, my gosh, we got to turn 180 degrees. And I think one of the things that I reflect on sometimes is, there were great goals culturally that we had to reimplement as we grew, but we appreciated the goals. So one of those is transparency. I think this was common at the time, I don't know if this still happens, but it was an organization where every piece of information was shared with everybody. Everybody was on every email distribution list. Everything was just wide open. And we've never been concerned with that level of transparency other than the overhead.

If I give you A, the reams of data, you will selectively choose whether it's emails or anything else, you'll selectively choose which parts to look at because they're interesting to you. And if you and I look at it, even exactly the same data, we'll draw different conclusions. So it's hard to keep alignment with raw data. It's important for leaders, others in the organization to say, "Hey, I looked at this and this is the conclusion I drew." Here's the data if you're interested in challenging my conclusions, but let's come to agreement on the conclusions because that's what's going to drive how we behave now. So I think that openness and with that, the willingness to say, "You might have something really interesting to add here." I'm not saying I have all the answers, please just take my answers. I want to build that context and share the context.

But again, if I just give you the raw data, you're going to go do something totally different because we have different interpretations of what really matters. So let's get together and align on that, but not because I'm just telling you, "Here's the answer," because I actually really value people around me challenging my interpretation. I don't have all the answers. I'm open to asking the questions, and if no one has any suggestions, I'll seed the conversation or try to ask better questions to get people to think a little bit more deeply. There's always someone who knows more than me. And I mean I'm sort of making this about me, but I think the leadership team that we have built is reflective of... So Jim is our CEO, he and I worked together before CircleCI, but our perspective of how we want to work, and we're always looking for people that will challenge us comfortably.

Like, "Oh, that's interesting, but maybe you didn't consider this. Because when I looked at it, I saw that this other way." "Oh, okay, tell me about that." There's always someone who's closer to the problem. There's always someone who knows more or there's someone coming in from the outside with a different set of experiences, maybe more experience in a particular area or just a, "Hey, we tried this at my other organization, have you considered that?" Right?

So I think that willingness and openness to that and then to go experiment, right? Say, "Cool, we have a lot of ideas, let's try something. Now let's not sit here and talk about all our ideas, but let's try something and let's know that we're trying something." Right? When everyone frames it that way. And again, we're not perfect, but I think that this is a thing that we really strive for is let's try and we might be wrong, we might be right, but when we try, we'll learn something. We'll have new information and then we can try something that we know to be closer to the ultimate goal or the ultimate answer. Whether that's product design, whether it's how we run our organization, could be anything. But I'll happily sit down and have an open conversation with you about what I'm seeing. I'll share the context from where I'm seeing it, and then let's kind of see how your information and knowledge can combine with mine to come up with a great answer. That would be my favorite part. Call it that.

Shane Hastie: So when you're bringing new people into the organization, what do you look for?

Bringing new people into the organization [17:06]

Rob Zuber: There's a couple big things. What I've just described in terms of open, comfortable, bringing different perspectives to extend our view of a particular problem set or in a particular area, whatever that might be. This is me personally, but also belief that we would have in the organization. Not a huge fan of you've done this thing, this exact thing before, because the length of time for which that thing is going to be our problem is probably measured in hours or days. And we don't measure hires in hours or days. If I need to understand how to do a very, very specific thing, I can hire a consultant for a week who's going to tell me how to do it. And I got the knowledge that I need. I want to know how did you learn to do that thing? What were the circumstances in which you were trying to do something and you realized this was a great solution to that problem, and how did you quickly learn how to take advantage of this thing and make it work?

On the flip side, I guess this is the balance. If there is a known solution, I don't want you to quickly learn a new thing just because that would be fun for you. I want you to be focused on the customer problem and I want to scope down what we do to solving the problems of our customers. If there's a thing we're doing that has been solved a million times before, let's take one of those million and focus on the thing that we do that's unique and valuable in what our customers pay us for.

But in solving that problem where we do have unique, interesting challenges, I want to know that you're comfortable not just doing the thing that you did before, but seeing the customer problem, identifying with it, and then finding a path to learn as quickly as possible, get there, solve it, we'll implement something new if we have to, and being able to see the difference. So I think that balance is really important. And then perspective, if everybody thinks the same way, if everybody comes from the same back, I mean, known issue but not solved, then you end up in this group think and you get the solution you're going to get instead of actually seeing it from a bunch of different angles and being able to get usually a really novel and exciting new perspective.

Shane Hastie: So how do you deliberately get different perspectives, the diversity of viewpoints?

Deliberately bring in diverse perspectives and backgrounds [19:06]

Rob Zuber: I don't have a great answer to this, but I'll tell you what I've been thinking about lately. Because if I'm not going to be honest, then who's going to be honest with me? So one of the things that I've been thinking about a lot lately after wrapping up a season on failure, now we're talking about teams. We're building up a season, is building teams. In software, and this is true in other departments. Obviously, I'm from an engineering background and in a company delivering products to engineers, but this is true in other departments for sure. But we deliver software as a team, right? We work as a team every day. Or I say, right, if you're a solo open source developer, not true. But for most organizations you're building teams and then teams of teams. And then we often hire, and by we, I mean in the industry, but we're imperfect as well, as if every person is being measured individually, as if every person needs to come with a set of skills and capabilities and whatever that would cover the whole thing.

And so that makes it not only not okay to be an imbalanced... And I'm saying I'm totally imbalanced. I've already said I'm really good at some things and I'm absolutely garbage at other things. And then there's spots in between. And if you know that and can say, "Hey, I'm really good at this. I have gaps over here. It would be great to have someone else on the team who's strong over here." And this isn't, not getting to all of the ways that we could talk about diversity, but just, I am actually trying to build the strength of the team. And I have junior people on the team. I probably need a senior person on the team or two or whatever, however you think about the makeup of your team. But it's great to have energy and enthusiasm and be super smart and whatever. But there's pathways you don't have to go down that someone can say, "Oh, yeah, I did that before. Here's some additional context to consider. Would you still go that way or would you try something else?"

You could save a lot of time when someone has experienced something, balanced against we never try anything new because we just do whatever they say. And so I think that team unit is the thing that's really, I'm trying to pick apart and think better about how to build for, because again, that's how we do all of our work. And not everyone is into sports analogies, but I think it's really easy to make them in this regard. You build a team of players with different strength. And I think about any team that I'm structuring in that way, meaning not everyone has to be able to do everything. The team needs to be able to perform with all of their responsibilities.

And that might be like, I understand databases and you understand front end or whatever the full scope of our responsibility is. And it might be I'm super extroverted and I love presenting our work to the rest of the organization. You're really quiet, but you spot the avenues that we were about to go down that are just going to ruin us. We have very, very different perspectives, but as a team, we achieve everything that we... We save ourselves some really big problems and we make sure everyone knows what we're working on. If we just hired extroverts, that would be a really weird challenge in software engineering, but we would have a whole bunch of people who were clamouring and then would be fighting over who got to do the presentation. It's usually not the case, but I need someone or someone for whom that's a goal, and that might be the way that I fill it in.

It's like, oh, we don't necessarily have all the skills here, but the manager is really good at that and we're hiring someone who wants to grow into that and we see the potential in them. So let's try to summarize this. The way we prevent ourselves from diversifying and getting a lot of different perspectives and different backgrounds and all of the other forms is we fashion one archetype and say all of our team members need to be this, but it's solving the wrong problem. It's the team that I want to be great and so I need some coverage in this area and some coverage in this area and some coverage in this area. And I can have people who are great at each of those things if I allow for them to be not as great in other areas because there are people who are great in those other areas. And the team as a unit is great.

Shane Hastie: Thank you very much. Some really interesting thoughts there. I'd like to end with, what's the biggest challenge facing our industry today and what do we do about it?

Complexity is a big problem for software development [23:04]

Rob Zuber: I will say a big, because I don't have the confidence to say that I know exactly which is the problem that everybody needs to solve, but the problem that I think about on a regular basis is complexity. I will define complexity, but why do I think about complexity? We're on a natural evolution. When I started in software, we wrote almost every line of code that we are going to put in production. And now we're delighted that we can move really quickly because we're standing on the shoulders of giants. People have built entire cloud. Just the cloud, that didn't exist when I started building out SaaS platforms or whatever we called them back then, and open source libraries. Open source was just kind of, okay, yeah, you could use that, but that seems like a sketchy idea, right? Now, this is what everybody does. So I've got underlying infrastructure that I don't understand.

I've got a whole bunch of libraries that I don't understand. And that's great because I'm writing this tiny little piece of software on top of that that allows me to do amazing things. But all of that stuff needs to be understood as a system. And all of it is constantly changing. People upgrading versions of my libraries and then there's a security patch to apply here. And then AWS or GCP or whoever is upgrading their underlying structure, and they're swapping out disc drives that I don't know about. All that stuff is constantly changing and we're sometimes fooling ourselves thinking, oh, we only need to understand this little piece, but all of it can have impact on what we're building and operating. And so I think, A, getting your head around that complexity is a hard problem. And it comes into how we design and think about systems.

And some of it is just acknowledging that we are actually building on top of all that complexity versus pretending we're not, because it's easy to put something out, and then doing that in a way that allows us to actually keep executing, right? Say, okay, we get it. We have control over it, unsurprisingly, this is what we think about and build for at CircleCI, but get my head around it, feel like it's easy to become then the deer in headlights. You know what I mean? This is terrifying. I don't even want to write code. I don't even know how to exist in this environment. Getting that ability to deliver on top of that, because again, we did all these things. We created the cloud and libraries and frameworks and all this stuff so that we could deliver faster, so that we could focus our energy on unique customer problems.

We're well past the days of manually optimizing table space layout on disk arrays. Thank goodness, that was not a highlight of my career, but someone's doing it. And so being able to reason about, Oh, this is still a thing that could impact me if it wasn't done well," right? I've never met the person that did it and I don't know how they did it, but feeling comfortable with that, getting a grasp on it so that we can reap the reward of focusing on the customer value and delivering against that customer value. And so I think we're getting there. Certainly we're trying to be a part of the solution to that, but we're getting there. But it's a hard problem and it's not slowing down, right?

We are accelerating ourselves because we keep building more layers to build on top of, allowing us to do bigger things with less work. And then if you want to take the whole thing off the rails, add in machines building things, right? That's next. So the complexity is going to get deeper and deeper, bigger and bigger, whatever the measure of complexity is. And so staying on top of that, feeling good about it, delivering effectively so that we as the humans who are bringing the brain power that ultimately ties all this stuff together can continue to be effective and not mired down in all that complexity.

Shane Hastie: Well, thank you very much indeed. If people want to continue the conversation, where do they find you?

Rob Zuber: LinkedIn, on Twitter at Z00B, Z-0-0-B. Terrible decision, but I'm living with it. And again, I have my own podcast The Confident Commit where I spend lots of time talking about and thinking about with people who are way smarter and have done way more interesting things than me, the overall delivery of software and how to be good at it. And a lot of the topics that we covered here that clearly I'm obsessed with.

Shane Hastie: Thank you so much.

Rob Zuber: Thank you. It was awesome to chat.

Mentioned

• Circle CI
• Book: Range
• Rob on LinkedIn and Twitter
• The Confident Commit podcast