Johnny Boursiquot on Serverless Go and Site Reliability Engineering at Heroku

01:17 Introductions

01:17 Bryant: So, could you briefly introduce yourself please and share a bit about your background or recent career highlight please?

01:23 Boursiquot: Yeah. Sure thing. Boursiquot is the last name. Folks usually wonder how do you spell the last names so best get that out of the way right now. So, currently, I am an SRE for Heroku which is owned by Salesforce. That acquisition happened a long time ago. It's interesting to still find out that people don't know that Heroku is a Salesforce organization. So, that has presented its own sort of set of challenges as well when you're working within a team within a broader organization. But as an SRE, I found that role has really sort of taken advantage of my ability to be a diplomat in some ways something that I think has always been an interesting aspect of sort of how I deal with systems and people in engineering. The people aspect of things has always been something that's interesting to me.

02:03 Boursiquot: I've sort of developed that over the years but my career spans everything from CTO to project manager to obviously the ops work I'm doing now. Now, the SRE stuff. I've been a front-end developer. I've even done design in the very beginning. I've had almost 23 years now in the industry to sort of try a bunch of different things and because of that I think that's really where the people aspect of things comes from regardless of the role, you end to do with people and I've taken that to heart and sort of a really tried to pay attention to that aspect of my career development.

02:33 Bryant: So, I say the technology is the easy part, right?

02:35 Boursiquot: Exactly.

02:35 Bryant People are the hard part.

02:36 Boursiquot: Yeah. It really is.

02:37 What are the primary use cases for building and deploying Go applications on a function-as-a-service platform?

02:37 Bryant: I actually bumped into your profile I think from Brian Liles tweeting. I was looking for someone's chat about Go, about serverless, about SRE and you tick all the boxes there, which is perfect. So, diving in sort of one of the topics there, I saw you publish some stuff with O'Reilly around Go and serverless. Go as in the language and serverless. Could you give us a bit of background on what kind of used cases you were looking at solving with those technologies?

03:01 Boursiquot: So, it's interesting so the two things serverless and Go. It was one of those things because I was sort of interested in both of them and passionate about both of them. I sort of put them together and sort of upgraded courses around them and with regards to O'Reilly, I'll be teaching a serverless Go and also Go itself with O'Reilly in the coming weeks. The love affair really started would Go a few years ago.

03:23 Boursiquot: I attended the very first GopherCon back in 2014. My interest basically started developing right around late 2012, 2013 and sort of keeping an eye on the language. As a technologist, you're always keeping your ears to the ground as to, "Okay, what's the next thing that I should be paying attention to?" Because nobody wants to go up obsolete, right? But in 2014, I went to the conference, a friend of mine, we flew from Boston, we went to Denver. I think that was the location.

03:47 Boursiquot: How sitting in the audience and watching the speakers and the topics and the performance characteristics of Go and the language, the syntax, and everything just started to click for me. I was like, "Okay. This might be it." I've never felt this sort of enthusiastic about a language about technology before and I just pursued that and really invested some time into trying to learn it and made some mistakes early on and trying to overuse some of the features and stuff like that. So, it's been a really fun journey and years later, I'm still very much involved in Go community and that takes shape in the form of a teaching. It takes shape in the form of running user groups.

04:21 Boursiquot: I currently co-run the Baltimore Go user group and I've held workshops for Go bridge, which is an organization that cares about sort of diversity inclusion within tech specifically within the Go community. I just enjoy teaching. I do corporate trainings every now and then and I've done video courses and I'm doing the live variety training coming up. So, I enjoy being around the people in this community. I enjoy teaching the technology. I enjoy building things with it.

04:48 Boursiquot: I don’t know. It's been sort of a bet that I placed on the technology that has really been worthwhile considerably. The serverless stuff comes in really a few years ago as serverless started becoming the thing. That seems to be the next big thing. In some ways, it's a part of modern infrastructure, modern architecture for systems. There is the tendency for I should do over hype on certain things like we even have memes about it. There's this hype cycle, all that stuff. I think the serverless has its place definitely within an infrastructure, and as your business grows, as your sort of team grows, as your needs grow there's room for that technology as well so a lot of my work in my career has been building sort of long lived services and that kind of thing.

05:30 Boursiquot: So, when macro-services was all the rage, in some cases, in some corners still is, and then serverless came in, I'm like, "Okay. Are we doing nano services? What is this thing like? Where do I fit in into this whole thing?" In the end, you figured out, "Okay. Well, no business has been around long enough is going to be all in on one thing. It's always a mix unless you have a startup that started yesterday, you don't have a completely homogenous sort of a stack. It's not all serverless."

05:55 Boursiquot: I mean some companies do strive for that and companies are doing that and perhaps quite well. But, again, over time, especially as new technologies come and go, there's no one tech to rule them all. It's always going to be constraint based the decisions that you're making. Where does this fit in with this technology or this paradigm apply better here than this other one. I'm kind of married the two things, my love for Go and my seeking to understand serverless and where it fits into these trainings and this examination really of how well can I apply sort of the lingo language? How well does it work within this world of serverless?

06:28 What makes Go a good language choice for serverless apps?

06:28 Bryant: Perfect. I'd love to come back to some of the teaching stuff there because as you and I have saying off mic at the beginning, that is super important and it's super hard to get right sometimes. But I want to dive into something you just said at the end there. What makes Go good for serverless apps?

06:40 Boursiquot: So, if you Google around, you'll find all kinds of different answers. Everybody has opinions and us, techies, we have opinions for days on these things. Personally, I found that the speedy start up for Go was a key factor. Being able to have a statically binary where I have no dependencies around anything else other than just say, "Hey, run this thing when this container fires up." It was really perfect.

07:05 Boursiquot: It was like, "Okay. Well, I can build my Go program the same way I traditionally would even if I had a sort of a long-lived service." I can build that, package it up and have a cloud platform, sort of a vendor run it for me. For me, I didn't have to change the way I was doing Go programming. So, when I write might Go code, there are certain idioms, there's certain practices that you follow and those are sort of commonly known as you'll hear people talk like either you might Go within the Go community that carries a certain weight and that means certain things.

07:35 Boursiquot: I follow some of those practices for the most part as well but the biggest thing for me is that, "Okay. I didn't have to change the way I was doing Go, some of the same practices that I had learned over the years, they still applied." It was like, okay, what's changed is how long the applications going to be running for. That's really what's change and, of course, that's going to affect your design, it's going to affect what you rely on within the environment, right? You have to expect that, okay, you're going to be launched, you're going to do some work, and then you can't assume you can hang on to stake, you can't assume you can hang on to certain things, right?

08:03 Boursiquot: So, network connections. You have to build a more resilient application. Your code has to be a lot more resilient. Those constraints you do factor them in and to some degree they matter as well for longer services, but that was really the big draw for me, I was like, "I don't have to change the way I do Go code to take advantage of the serverless aspects of how it's deployed and how it's run?"

08:24 What was your number one pain point or learning when you started deploying onto serverless platforms?

08:24 Bryant I definitely learned some lessons the hard way when I started going Java and serverless. What was your number one pain point or learning thing when you moved into serverless?

08:34 Boursiquot: Go is well-known for its concurrency primitives and one of the hard lessons that I learned was that if all you do is consume the marketing from all the different cloud providers that are saying, "Hey, you're serverless, right? You can do all these things, whatever it is, and the code samples it looks so simple and they're so neat." They're well-prepared, well-presented. They make it seem like it's the easiest thing to just do. Just drop your code and zip it up and put it in a bucket. We'll take care of all the things.

09:02 Boursiquot: It's like, "We'll take up the concurrency as many of these things that you want. We'll auto scale them up. We'll scale them down. Don't worry at all about the scaling aspects of your code." See, I'd be writing my code, not really taking advantage of the concurrency primitives within the features within the Go language. Basically saying, "Ah, yeah. Every time request comes in, for example, I have this thing here that's going to invoke my function and it's going to do that work, and then when it's done, it's going to shut down and that's it."

09:29 Boursiquot: If you have say a million of these requests come in and the infrastructure providers, I'm going to spin these things up, a million of them and do the work. Then, you don't realize that, okay, it doesn't have to be a one-to-one sort of thing. One invocation of your requests that spins up your application and does work. Depending on the kind of problem you're solving, you can still take advantage of the concurrency primitives of your application.

09:51 Boursiquot: So, typically, you have, in the case of AWS Lambda, you have about 15 minutes to get some piece of work done. So, if I can, if the workload is such that I can spin up in the case of Go, bunch of Go routines to do work and currently, ideally in parallel, I can take advantage of everything that the language has to offer within that one invocation of the program. I don't need to rely on the fact that, okay, every time a request comes in, I'm going to spin up this thing, wait for the boot time to do the work, and then expect that, okay, I can't process anything else until the next invocation comes in.

10:23 Boursiquot: We can do a lot of work within that 15 minutes, again, taking advantage of the features of the language itself. I've had some interesting problems that I've had to sort of a re-approach and redesign in that way and basically saying, "Hey, you know what? Take advantage of the features of the cloud provider writer in terms of the auto-scaling and the concurrency that they promise but don't forget that the language you're using, the technology you're using also has capabilities." Those don't go away just because there's a new deployment mechanism or a new runtime. Take advantage of the language of the tool that you have to do the work that you need to do.

10:57 What was your experience of the toolchain like for building and deploying serverless Go applications?

10:57 Bryant: What's it like in regard to the tool chain? Are you using the same ID? Are you building things the same way? Are you using pipelines or how do that differ?

11:05 Boursiquot: I like to think of this whole serverless ecosystem as going through several phases. In the beginning, it was incredibly hard to be confident that the stuff you were writing locally would work well in the cloud environment without actually putting it in the cloud environment. That story has gotten better over time as the providers have created sort of local environments that you can test things in. Again, to use AWS. You can have a local DynamoDB if you want. You can have a local Lambda runtime if you want. You can have a mock API gateway if you want.

11:38 Boursiquot: So, all of the things that are typically part of the demos and the pieces of technology, the platform that they like to sort of put together, to present sort of a unified solution for serverless technologies. So, they've created locally runnable versions of these things that simulate what the cloud environment is going to look like. But at the end of the day, you still need an integration testing, at the end of the day, you still need to sort of ensure that things are going to work together.

12:01 Boursiquot: Sometimes you can't mock a service you don't own. You can only go so far with that tool set but the story for serverless development without having to put things in a cloud to test them and to make sure they work, that story has gotten better and I think it will continue to get better over time as the developer experience becomes a priority on the future roadmap for these providers that's going to get better. Obviously, the community as well is sort of pitching in with various projects to sort of help mitigate the lack of good tools in that area. But, again, because developers, the mind share, that all the cloud providers want a good experience for developers, and at the end of the day, whoever's got the best experience for developer mindshare, chances are, they're going to also get more market share.

12:43 Boursiquot: So, that whole experience is getting better but, again, you still need, in my opinion, study a staging environment if you can and we have the discipline and you have the right infrastructure for it, you should be testing in production as well. So, the tool chain itself, the stuff you're doing locally, yes, you're going to write your unit tests, yes, you're going to do a lot of things that you need to do right regardless of sort of deployment mechanism but that's getting better. But that doesn't exclude the need for actually doing that integration testing with all the other bits on the actual provider.

13:12 What's your experience of canary releasing or dark launching with serverless applications?

13:12 Bryant: That makes total sense. It's something I bump into quite a lot at the moment as in that need, that local dev experience, I do a lot of stuff in Kubernetes. We talk about this quite a lot, and the testing and production like the joke is everyone testing production, whether you know it or not, right? But then it's really key, sitting with things like functions as a service of functions, you can spin them up and then write traffic accordingly. So, what's your experience been like around the story of say canary releasing or dark launching, all these kind of cool things you do where you're controlling at the edge what service, what function you may hit? Is that currently a good story or not on your experience?

13:44 Boursiquot: It's absolutely possible. It's doable but it is not simple. It's complicated. So, when I think back of the promise of serverless, the marketing. I was sort of give the marketing a hard time because they make these things seem so simple. In reality, it's far from. You need operations. You need somebody who's on your team who's operationally minded. You need infrastructure configuration. You need somebody to worry about the operability of whatever it is that you're going to put out there and you need somebody who understands what it means to have a canary release, what it means to do some traffic shifting. You need somebody with that mindset.

14:22 Boursiquot: This whole serverless marketing makes it seem like, okay, all you need to your developer building features is to just don't worry about the orchestration bits, don't worry about the operational bits. Just write your features, put it out there and let the cloud provider just figure that out for you. It's zero touch ops. It's no ops. I hate it. I'm like, "That is a pile of…" Yeah. Let's keep it clean here but it's the fallacy to think that just because you're drinking the Kool-Aid from these providers that somehow you don't have to have that conversation around, okay, I built this thing, it's an nano-service architecture, micro-service, Lambda serverless, whatever, it doesn't matter.

15:00 Boursiquot: Having that conversation around operations, if you're waiting till the very end to figure out, okay, well, we built it, how do we now run it and operate it? If you wait till the last minute to actually have that conversation, you're doing yourself a disservice.

15:15 How do engineers debug serverless applications when something is going wrong?

15:15 Bryant: Yeah. I was going to draw on your SRE experience here a little bit. What's the story like in the moment around observability or monitoring-logging? Because, again, you've alluded several times showing that we're building distributed systems definitely with functions. So, when a request comes in it can often pinball all around the place. How do we as engineers debug when something's going wrong?

15:32 Boursiquot: So, whether it be serverless or micro-service, in this world, once you start playing in this sandbox, tracing, I'd argue, is way more important than logging. That's been my experience because if you think about it, I missed the monolith days because you could have a request coming, you can see where the load balancer hands the request off to my application and from top to bottom, in one log file, you can kind of trace everything that the system did, right? Everything's being drained at the same log system and you can kind of go into one dashboard and kind of see and maybe, hopefully, identify the particular request and kind of see everything nicely sort of linear within the application, right?

16:10 Boursiquot: In this world where you have bits and pieces are distributed and they all talking to each other behind the scenes, if you don't have tracing, you're in deep doo-doo. You need to somehow be able to sort of see where the handoffs are happening. Who's receiving what request and from other part of the system and how long they're spinning, what other systems are they interacting with? You need tracing. I have this joke with some of my peers on my… Okay, I'm not a logger. I just traced a lot.

16:36 Boursiquot: Then, for those who know the hip-hop song that they'll sort of kind of pick up on the reference there. So, it's way easier for me to find out what's going on when a problem happens to basically pull up a tracing interface and see which part of the system dropped the ball or took the longest or whatever it is. The logging comes when you really know where to look. When you know where to look logging helps.

17:02 Boursiquot: When you don't know where to look, you're trying to figure out, okay, what was the last thing to touch this request? Without tracing, it's very, very hard to do, otherwise, you're graphing your logs and that is not a good look.

17:15 What does the role of site reliability engineer (SRE) look like at Heroku?

17:15 Boursiquot: I never thought I'd get Big Pun reference to the podcast -- love that. Well played, well played Shifting gears a little bit now. I want to dive into some of the SRE world because definitely some folks around the world, SRE is still a bit of a buzzword. They have this Ops, this DevOps as SRE now, site reliability engineering. What does it look like at Heroku? What does SRE look like?

17:35 Boursiquot: So, let's set the scene a little bit here. Heroku was been around and doing this thing for a long while, for a long time. You have developers that rave about sort of the ease of building the applications, all they have to do is do a git push Heroku Master and their application is running. They don't have to worry about operations. They don't have to worry about a lot of these things. They went into this scale. They just increased their nano-counts. The add-on system is fantastic. I'm not saying that because I worked there, I've been using Heroku way before I even worked there. I've always sort of really liked what the platform has to offer.

18:04 Boursiquot: That means that because Heroku has been around a long time and it certainly predates the term SRE, certainly predates that terminology and those titles and everything else. That means that this team and obviously has changed and morphed over time has had to sort of know and understand what they were doing, right? Which is what obviously made Heroku popular and still is. They kind of had to sort of develop practices and understands or really how to put systems, distribute systems together that can run at the scale that Heroku runs at.

18:34 Boursiquot: So, in a sense, some of the values, some of the principles of SRE, they've already been practicing all these things. They just didn't have the title for these things. Certainly, SRE brings certain disciplines, really relying on SLIs and SLOs in order to set your SLAs with customers and everything else. So, there's processes and procedures and depending on what company you work at, you'll see various amounts of these practices within the organization. Oh, and by the way as a side note, SRE looks different on every company.

19:02 Boursiquot: I've never seen a job change so drastically from company to company like any other job. I mean we'll touch on that later but at Heroku, it has become, the SRE team that I'm part of really we are almost like a team of diplomats, a team of production engineering sort of best practices, operational excellence kind of team, whereby we identify components within the system that need a little more TLC, that need a little more attention from an operation standpoint and we work with those teams to basically, "Hey, let's go to the process. Let's go through our production readiness process and figure out, okay, what do you have right now? What do you not currently have? Where are your pain points?"

19:39 Boursiquot: It's more of a conversation. It is how can we serve you, this team, the owner of this component or this set of components, how can we make your life easier? How can we reduce toil? How can you make it so that when we have an incident, we can solve the problem a lot more quickly? How can we reduce the MTTR here? It's really that kind of conversation where we're really there to serve. We can't go in with a tome, the guidelines, the mandate, come into a team and slap it on some sort of metaphorical desk and says, "Hey, you need to go through these checkpoints."

20:11 Boursiquot: Nobody likes that. Even if you could do that, nobody likes to come in and being told what to do, especially with an organization like Heroku where people have been operating really large-scale systems for so long. You can't come in and say, "Hey, we just picked up a book from the big G and now we know we've got processes. We want to change everything and we want to sort of…" No. We can't apply the blue book. We can't just apply it to just one-to-one. It just doesn't work like that. Like I said before, SRE looks different at every company. You have to pick the nuggets, the insights and what works best for this company.

20:43 Boursiquot: We're constantly studying and learning and saying, "Hey, this is working for this company or this team, how can we incorporate that into what we need to do, make that part of our sort of repertoire, our tool belt to be able to say, "Hey, you know what? For you, we noticed that you're having problems with this particular aspect of this component, this is something that's working for this team. How about we work together to figure out whether that can work for you too?" It's a way more diplomatic, way more sort of in service of than coming in and say, "Hey, this is how we do SRE, you're going to do it this way."

21:10 What's your experience with SRE error budgets?

21:10 Bryant: What's your experience with error budgets? Is it something you did at Heroku or not?

21:14 Boursiquot: Let me set the stage here for how we treat our budget. So, when we were trying to introduce more SRE principles at Heroku, we had this ask that team start to measure SLOs and sort of figure out sort of identify really what their SLO should look like and what are the SLIs that they need to sort of capture and what information they need to capture to be able to really have a meaningful and intelligent conversation around the systems?

21:38 Boursiquot: Again, know that because Heroku has been Heroku and operating at the scale that has successfully for many, many years. Now, basically, it was a matter of, okay, can you prove that you're operating? We know you're operating, you have operational excellence, we know that you've been doing well for many, many years, otherwise, you wouldn't still be in business. So, we know you've been doing the right thing. For, at least, for the majority of your existence. We know you've been doing the right thing.

22:00 Boursiquot: Now, how can we identify the SLOs that we already meeting to serve? So, basically say, "Hey, this is the baseline we know we're at right now." It's more like identifying the SLOs and saying, "This is what we're identifying" It's like measuring current state. Now, where the error batch has come, obviously, you want to make sure that as teams push out new software, there's system changes. You want to make sure that, okay, when something happens, you're chipping away at your budget. That means you need to go back and this, you look at the last release that caused an incident perhaps and say, "Okay. Well, what do we need to do here?"

22:33 Boursiquot: It's more at this stage in our implementation of SRE, this is more of, it's not the thing that prevents teams from releasing new software. We're not coming at you with the error budget hammer and says, "Ah, you exhausted your budget for this month or for this period, you can't really send any more new software." Again, this is another example where as prescribed by the book, we can't just take that and apply it like as a hard-and-fast rule and says, "Hey, no more releases this month." We can't do that. It serves as informational. It helps us with sort of knowing where to look. What do we pay attention to right now because this system has caused a couple of outages this month? Let's pay some more attention to that.

23:11 Boursiquot: We look at the error rates. We look at all the dash… I mean we have dashboards galore. We have logs galore. I mean we ingest a ton of logs, right? All these things help to inform where to pay attention to but we can't pick any one thing from any one feature from the books and say, "Hey, this is now a mandate. This is how we do business."

23:31 How can SREs play the role of the diplomat?

23:31 Bryant: That seems like a very sensible approach. Something you mentioned is the sort of role of the diplomat and I like that, not heard it said that way before but I like that a lot and how does that work? Is it's sort of a push or a pull? Is it a case of you go out to the teams and say, "Hey, here's how I can help you" or do you advertise what you're doing and wait for them to come to you? I'm guessing it's probably a hybrid between the two but how does that work?

23:52 Boursiquot: It is a hybrid. It's one of those things where depending on a team. So, some teams within Heroku and I've seen this in other companies and other organizations as well. Some teams because of the nature of the kind of services and kind of components they're dealing with, maybe they're dealing with the cloud infrastructure directly so they're the abstractions between the developer who wants to just do a git push Heroku Master and the interaction with the cloud providers and the different instances and the compute engines and database engines all that stuff.

24:21 Boursiquot: So, you have these teams that require a tremendous amount of discipline to basically to provide that abstraction between developers, the developer experience and all the bits and pieces. The sausage. It seems like how it's made basically, they have to have sort of that discipline. So, for those teams, they might have specific pain points that they want you to address but they're not going to say, "Hey, SRE, why don't you come in here and do a complete overhaul of everything we've been doing?" That's not realistic.

24:50 Boursiquot: So, they make come to us and say, "Hey, we're experiencing pager burden, the pager burden this month has been killer." We have these components here, they're the chronic cause of certain incidents and certain outages, we need a way, we're busy putting out fires over there. We need somebody to be worrying about the process improvements so that the things we need to do to make sure that those hot spots we can address them.

25:15 Boursiquot: So, they'll bring us in or they say, "Hey, we need some help here." Because we just don't have the time, it's not our forte to be thinking about process and how to make this aspect of things easier to deal with because we're over there putting out some fires. This is where we're sort of called in to sort of help provide sort of that guidance. Again, it's diplomacy because, again, you don't want to go in with a rule book and saying this is how you must do things. They've been operating at an excellent level for years, right? Every team has pain points, every team has a service that's problematic.

25:45 Boursiquot: The one service nobody wants to touch. Maybe it's written at a different language that whoever wrote it and has long left. I mean every company has that one service. The capability you provide is an outsider's view and says, "Hey, here are all the facts that you've presented, giving what would be able to find out, we even have our team members on board unto their different teams, be part of the development and release workflow." That way we get to absorb all of the pain.

26:09 Boursiquot: Then, and only then, we sit down, we have a conversation, and say, "Hey, this is what you could do here. This is what you need to hook into that we already have that's going to alleviate some pain for you and this is where you can improve. This is what we've noticed." Again, it's a relationship not some sort of going to come in and we're going to provide a mandate. Really, it is a relationship, and as we all know, hopefully, we all know by now. A successful relationship is based on good communication and if you don't have that it all falls apart.

26:34 What do you think about the current state of cloud platforms today?

26:34 Bryant: That's been a great story of SRE. Johnny, I really appreciate your insight there. These are unique things which I'll definitely ponder on a bit more. I want to take a step back now. So, I think you're in a very unique place with your role at Heroku and your past experience, your love of serverless. What do you think about the current state of cloud platforms today? I'm a big Kubernetes person. So, I do a lot of Kubernetes stuff but I'm totally conscious. It's kind of the foundation, like the amount of times I've gone to company and I just want a Heroku like experience there.

26:59 They want to get pushed to Kubernetes. That doesn't exist unless you buy something else or plug in something else. What's your opinion of the state of cloud platforms and where we're going?

27:09 Boursiquot: So, it's interesting because I've been around long enough to have seen various waves of the next big thing. I'm doing air quotes on audio. Everybody thinks they need Kubernetes and when I hear that. I'm like, "Really? Really? What makes you think…" As you say they want that past experience. They want that, "Yeah. I just want my developers to just easily just worry about the code, just push the bits somewhere, and then everything magically happens and we think Kubernetes is the path to that promise land."

27:39 Boursiquot: I'm like, "Okay. First of all, unless you are in the business of providing pass, unless that is your business, you want to think carefully around taking on the operational burden of something like Kubernetes and bring it into your team because guess what? If you didn't have a dedicated head count for this or at least a couple depending on size of your team, depending on size of your needs, you can download the bits of a github and whatnot but you now need head count you need people right to understand this thing operate this thing.

28:08 Boursiquot: Kubernetes is evolving every day and vendors are now producing things to that plug in and work with Kubernetes. I mean, now you have people don't understand, "Okay, what's a service mesh? How do I get my systems to talk to each other? I need to do sidecars now I'm doing this. I'm doing that. I need to understand Kubernetes API. You are introducing a whole host of new challenges into your world and that's even before you can get something standing for your developers to actually push code and actually, the value actually trying to derive out of getting to such a sort of orchestration system.

28:40 Boursiquot: It's going to take you a while to get there. Again, everybody is going to be doing it a little differently because that's the way business is and no two businesses are exactly the same, at least, not in my experience. So, every business is going to have different sort of constraints that require you to have a certain implementation, certain do Kubernetes in particular way that suits the business. That's the thing folks don’t understand. It's like, "Look, unless you are in the business of say a Heroku, right? Unless you are in the business of providing where this is all you do or you are large enough company to take on this burden."

29:13 Boursiquot: I mentioned before that Heroku is owned by Salesforce. We have teams at Salesforce whose entire job is to worry about Kubernetes and naturally as you might expect. We have the person power. We have the budget. We have the people and the teams dedicated towards creating that world for developers who don't need to worry about operations for them to just do their job and put their stuff out there.

29:36 Boursiquot: The hype, again, it gets people no matter how many times that we experience a hype cycle with every new piece of technology that's supposed to be the new silver bullet that solves everything. We go through this process where we do some thinking. We think back like, "Hmm, maybe we were too hasty about certain things." Again, I'm not throwing shit at Kubernetes at all. I'm just saying that if you are looking at Kubernetes as a solution to your sort of operations problem, unless you spend some time really digging into what kind of problem you need to solve, first of all, understanding what your problem actually is.

30:10 Boursiquot: If you can get away with running your stuff on a Heroku or writing directly with AWS or GCP or Azure, whatever, even before Kubernetes was around that these vendors are already providing best practices and this is how your architect systems, anybody who's been in the AWS world for a while they'll be very familiar with the well architected framework. There's all these best practices out there for how to run and they have products that abstract away the instance management.

30:34 Boursiquot: I mean the whole serverless thing with Lambdas and EventBridges and how all these things work together. Those vendors are already providing abstractions on top of the raw compute stuff already. It might be that you just need an application that is built on those raw primitives, following the best practices of those vendors. Then, the other thing is, "Hey, Kubernetes because I wanted to be able to run Azure, AWS, and GCP." I'm like, "Okay. If that is indeed part of your business model, if that is indeed what you want to be able to do so that you're not locked in into any one vendor, if that is indeed a business priority for you, great."

31:11 Boursiquot: For a lot of folks it's just, okay, like the cost of having an abstraction that can run on top of all the different vendors and every vendor has something a little different, you can't just drop it, and then it all works right. If you're going to be running Kubernetes within the US, you still have some concerns and constraints that you're not going to have GCP, same thing for Azure. So, everything requires a bit of tweaking and management again somebody somewhere on your team and your organization needs to worry about these bits and pieces, this configuration.

31:37 Boursiquot: So, again, it's all about what is the business trying to get done? We hired to solve business problems not to play around with tech, not to play around with YAML files all day long. That's not why we're hired. That's not why we're there. We're there to provide value. So, understand what kind of business problem you're trying to solve.

31:54 What are your thoughts on build vs buy when creating a platform to deploy applications onto?

31:54 Bryant: To have a platform, we need a bunch of tooling and one antipattern I see a lot with engineers and I'm sure I've done it myself to be completely honest is discounting our time. Yeah. We forget. It's the classic kind of hack and use. I could build that in a weekend. I'm like, "Maybe you could. Can you scale it? Can you run it? Can you look after it?" Have you got any advice for junior engineers or just even me, the old school folks on how to overcome that bias to discount our time and think about, we could just pay some money and get tools to do this, for example?

32:24 Boursiquot: The quickest thing sort of junior engineers realized in my experience is that, okay, so you want to build something on a weekend. Let's take Facebook or Twitter. Yeah. That's just like a stream, as you scroll down you just do an Ajax call. I might be dating myself. I'm not sure we call it Ajax anymore, but you do some call in the background, and then you update the DOM and you just keep the same coming in.

32:45 Boursiquot: Okay. Yeah. It might make for a nice toy, like a nice learning experience and we haven't even started talking about what it means to actually have a product. That's a whole different conversation. We tend to think that it's just about technology. I'm just going to figure out what is the right amount of JavaScript and react or view whatever it is just sprinkle on this thing to have like a functional sort of MVP as it were.

33:08 Boursiquot: We realize, okay, just for technology alone, building the features, honestly, that is the easy part. That is the stuff where you don't have to worry about scale for things. It's just you and your localhost just hacking away at something, you're clicking and everything is working just fine. Your database requests are quick because you've got local MySQL or Postgres or whatever it is running. Everything is just smooth like butter.

33:30 Boursiquot: Then, say you go to the next stage. Some you want to deploy, you want to put it on a cloud vendor be it Heroku or AWS or whatever, you want to put it out there and say you start getting a little bit of success and maybe you find your way on the front page of Hacker News and now you start getting traffic, and then you realized, "Oh, crap." The database queries are not as quick anymore. Meaning the more requests you start receiving like, "Okay. I need to do some scaling here. Do I do like horizontal scaling or vertical scaling? What is even the horizontal or vertical scaling?"

34:03 Boursiquot: Now, you have to worry about these things that perhaps as a developer, especially as a junior and I've been bidding there. I definitely thought, "Hey, yeah, I can be done in the weekend." You realize, okay, there is so much more to actually having something that can work for people, not just you that can work for people. The more popular whatever your thing is the more you have to spend on the operability of whatever that thing is. That's just the technology part.

34:28 Boursiquot: Then, now you say, "Okay. Well, yeah, I can build a Twitter in a weekend." "Okay. What about the actual products out of things? Do you know how to market? Do you know how to advertise? Are you tracking your UX? There are all these other things. This is selling. I don't know about you but I'm perhaps horrible at sales. I'm a technologist at heart. I get on a sales call and I'm biting my tongue not to start jumping in with all kinds of technical jargon.

34:50 Bryant Yeah. Solutions.

34:51 Boursiquot: Yeah. I'm an engineer. I'm here to solve problems. You all like go over this stuff you're talking about here like marketing and sales and all these things. I had to learn over the years. Okay, you have to learn to communicate at the right level. You have to educate yourself around all the different bits and pieces that go into really sort of taking something from just an idea that I'm hacking around on my laptop to an actual business, an actual product.

35:14 Boursiquot: So, again, I have the benefit of time so I have seen these things. I've been part of startups several now where I've seen this whole thing sort of play out. I have an appreciation for it but if you're a junior engineer and you don't have the benefit of that experience, read, consume, hit up people and say, "Hey, I noticed you've been at this for a while, I'd like some advice." Most of the time people will respond in kind. "Yeah. Sure. I don't mind having 10 minutes Zoom with you and talking about some stuff."

35:40 Boursiquot: I do that now. A lot of people reach out to me and I usually make myself available to do that as a way of giving back in some way. It's not just about technology. That's something that I had to learn many, many years ago. The technology is the easy part. It's the people. It's all the things, all the bits and pieces that come after.

35:55 How can engineers learn to become more effective teachers?

35:55 Bryant: Superb, Johnny. Now, that nicely fits into my final question here as we wrap up. I'm definitely a big fan of learning, love what you're doing on the O'Reilly platform. If folks are looking to follow in your footsteps, could you give a few words of wisdom as to how to get started with learning how to teach even?

36:10 Boursiquot: I have the saying that has helped me ground myself in a lot of different contexts and that saying is, "Ego is the enemy." I found out that my ego, where I use it, I found out that my ego prevented me from doing a lot of things. For example, say you're doing pairing with somebody. If you are in your head worrying about, "Okay. What is this person going to think about my code, my ability to code? How I reason about something."

36:35 Boursiquot: I'm not that smart. They're like god-like. Look how fast they're typing. Look at how well they're able to reason about something. Especially if you're pairing with a senior engineer or an architect or something like that and they seem like magical beings that have all this knowledge as esoteric, sort of understanding of things and you start going in your own head and thinking, "Oh, I can't hold a candle to this person."

36:57 Boursiquot: You start getting in your head and that detaches you from the experiences you'll be having right now, which is I'm pairing with somebody, they might be typing or I might be driving. We're trying to solve a problem. You don't realize that, okay, be in the moment, be in the experience that you're having, don't be in your own head, you trying to learn, in some cases, you're trying to teach. So remove yourself from the equation.

37:20 Boursiquot: The moment you remove yourself from the equation and you allow the experience to be what you're focused on, the ego takes a backseat. Now, you're all about sort of learning from the experience about the domain you're in and also about the actual experience of engaging in a pairing session or code review session or an architecture meeting or whatever the case may be. So, for teaching, in particular, I had to remove myself from the equation there and making that okay, "I'm here standing in front of these people, be it physically or virtually, they are here to get something from me, try to be able to do their jobs well or better."

37:57 Boursiquot: A lot of times I'm introducing people to programming for the very first time, which is an incredible privilege, by the way, to teach people who've never programmed before how to do that. I mean that is the honor of my life. I've had the privilege of doing that a number of times now and really that these are the things I'm going to remember most when I go to my deathbed. I'm going to be like, "Okay. I introduce people to program for the first time and perhaps change their lives and the lives of their families forever."

38:18 Boursiquot: To me, that there's no greater honor. Removing myself from, "Okay. I'm not here to show you how much I know. I'm not here to show off. I'm not here to razzle and dazzle you with terminology. The matter of fact, when I teach I try to leave technical jargon even if applicable, I try to leave as much as out of the way I teach as possible, especially to a beginner crowd. Because you can easily lose them. Again, the ego. You go in your own head start thinking, "Okay. How can I impress? How can I sound intelligent and smart and all these things?"

38:49 Boursiquot: Again, you're making it about you and it's never about… A safe bet really and most things in life, it's never about you. Okay. You remove yourself in the equation. I keep that in mind whether I'm teaching, whether I'm pairing, whether somebody's asking me for advice and even if you are in a position where you ought to feel like proud of yourself and whatnot, resist the temptation to make it about you because the moment you do that, it's no longer about them and what they're trying to get out of the conversation.

39:15 Boursiquot: So, the secret that I've learned over the years is that the more you give to people the more you're going to get back. Stopped worrying about yourself so darn much. Give. Give of your time. Right now my time is the most precious thing I have but I still choose to give some of it to a certain degree. I'm realistic. There's only so much I can do. So, I'll give some of my time to say a student to or somebody who's about to go into the tech career and they don't know where to start or somebody who's been in business for a while and they know I have a particular domain expertise in an area, they'll come and ask me for some insight.

39:47 Boursiquot: I'm willing to share that knowledge and sometimes when I have questions, I don't know it all. It's foolish to think that I know it all, right? So, I'll go to the people I know could provide some insight and answer some questions. So, when you give it's easy for you to receive as well. I mean I know it's cliché to say that but really, I've gotten so much in return from my giving to my community and to the people in my life like I almost don't worry about what I'm going to get out of things anymore. If it's something that is worthwhile to do, which teaching is always worthwhile to do, mentoring is worthwhile to do, I freely give that because in the back of my mind I'm like, "Hey, I'm giving back. I'm doing a good thing and who knows, maybe I'll get something in return, maybe not." That's not the point.

40:28 Boursiquot: Giving is the best way of getting back and these lessons I've learned over the years and they've served me well. I don't know a lot. I'm not a guru. I'm not a philosopher but these things I know to be true.

40:38 Bryant: That's a perfect way to end the podcast I think. Lots of great wisdom there. Yes. It's been a great chat. Really enjoy chatting. Thank you very much.

40:44 Boursiquot: Yeah. Thank you for having me. It's been a great fun.