Scaling Organizations with Platform Engineering

Transcript

Cordero: Welcome to my talk on scaling organizations with platform engineering. I'm currently a tech lead for The New York Times observability team, which sits under our wider platform engineering organization. As someone who is responsible for the direction of my team, and as an engineering leader across all of The Times, I spend my time thinking about the function of platform engineering, and reliability engineering, and how it can facilitate success across all of The Times. Before The Times, I was also a platform engineer. This talk is a compilation of my consistent approach that I bring to each of my roles. Ultimately, what that looks like is coming from the perspective of enabling our department to work in a way that ensures long-term organizational sustainability. We'll start by talking about what we even mean by when we say organizational sustainability.

Organizational Sustainability with Platform Engineering

More specifically, I define sustainability as the continuous practice of operating in a way that enables short-term growth opportunities, while ensuring long-term success. There's a lot to unpack there, so let's break it down. First, sustainability is a continuous practice. Even if we spend a lot of upfront time thinking about how to ensure long-term sustainability, circumstances change, often quickly, and we need continuous avenues to ensure long-term success. Secondly, enabling short-term growth opportunities. Sometimes those risky short-term growth opportunities are what lead to our long-term success. The emergence of bundles of tools has worked really well for some companies, including The Times. We don't want to give those up, but putting on my reliability management hat, we also need to prepare for the risk of those opportunities. Which leads us to this component of enabling long-term success. We frequently see companies take their core business for granted in the name of growth. For every successful growth opportunity, most opportunities do fail. Companies with a strong core product mission, neglect its maintenance and modernization to focus on new opportunities that are potentially a large source of risk. Preparation for this type of risk is essential.

Now that we've defined the goal, organizational sustainability, let's define the strategy, platform engineering. Using my definition of platform engineering, platform engineering drives organizational sustainability by practicing sociotechnical principles that provide a community driven support system for application developers using our standardized shared platform architecture. These three highlighted components are what form the basis of what it means to provide a platform. I said that this talk is called scaling organizations with platform engineering, which we've defined here and how it's a framework for driving organizational sustainability. What I haven't answered yet, is at what point is this framework even necessary? We frequently talk about scaling software, but what does it actually mean to scale organizations? The answer is ultimately that our ability to scale our organization is directly tied to our ability to scale our software. When we think about scaling our software, we have to be intentional about addressing the inevitable complexity that comes with that growth. To address this complexity, we have to bring this intention into how architecture can enable those needs. Because complexity makes development so much harder, that we as a collective industry have evolved the way that we even build applications. For example, the modular monolith has become an increasingly popular architecture style, especially as an intermediate step towards adopting distributed architecture patterns that eventually enable us to work and grow our applications. Just like we've evolved the way that we build applications to embrace these new architectural patterns, we must evolve the delivery strategies we use to build with these new architectural patterns. If these architectural patterns are a solution to the technical complexity of scaling our applications, platform engineering is a sociotechnical solution to the organizational complexity of scaling our applications. To summarize concisely, platform engineering is a sociotechnical solution to organizational complexity of scaling our applications.

Platform Principles (Definition)

We'll spend the rest of this talk decomposing each of these components, the principles that guide us, the community driven support that enables application developers, and the architecture that we use along the way. First, we have the principles that guide the rest of our platform. Having focused on reliability management, the principles overview are heavily influenced by DevOps, particularly because DevOps principles take a strong consideration for both the technical and social components of what it means to develop and operate software. Let's get into those principles. The CALMS framework was coined by Jez Humble, a co-author of "Accelerate," which is basically a framework of principles that should be the core of DevOps organizations. While all five could easily be applicable here, we're going to just focus on three, specifically culture, automation, and measurement. Starting off with culture, the CALMS framework tells us that it drives a culture of continuous improvement and reduces silos by intentionally sharing knowledge and feedback. You can note that while I've removed the S for sharing in CALMS, its presence still shines through this principle anyway. The same is true here for platform engineering. I'll talk about it more directly by putting it in the context of community. We often talk about breaking down silos in the DevOps space, and that problem is ultimately where DevOps emerge from. The way we bridge that is by sharing knowledge. Shared knowledge means to connect. Connection and communication are key for preventing the silos that would hinder our ability to make continuous progress. When we're talking about organizations, especially as an organization grows, the most effective way to not only manifest this culture of sharing, but also to improve employee engagement and retention, which are other ways of ensuring that our business's sustainability is sustainable, because we all know how much turnover is disruptive, is to think about how we can cultivate a strong community that fosters this culture at scale. Because, ultimately, the opposite of isolation is to be in community with other people.

Next, we have automation, which improves our software delivery process by reducing human error, improving our efficiency, and enabling faster delivery. This means thinking critically about the type of work that doesn't require business specific knowledge, and figuring out whether their work can be consolidated into software that's managed centrally by platform teams. In this we can reduce the cognitive load product engineers often have to indulge in by managing all aspects of their software. The type of work that's important, but can be consolidated in an automated or centralized way is work that's repeatable and manual, which is what SRE often refers to as toil, or what product engineers might refer to as boilerplate software. Another aspect of platform engineering is how we should be explicit about improving efficiency by leaning into solutions built by third parties, whether through vendor solutions or open source solutions. The reason for this is because we need to reduce our own cognitive load, just as much as product engineers. A lot of our work can require a lot of subject matter expertise, whether that's to manage infrastructure, build CI/CD pipelines or build runtime frameworks. Reinventing the wheel is not supposed to be the inherent function of platform engineering, no matter how tempting the technical challenges might be. Large companies often set a precedent to build everything in-house. Sometimes that is genuinely necessary. For those of us who are in earlier stages of our engineering stories, that shouldn't be our first instinct. The reason for this is directly tied to sustainability. Hiring for very specific skill sets is hard, and people often leave. The question we should be asking ourselves every time we develop a new tool from scratch is what would happen if these engineers who built it left. What if my business was at risk and I had to heavily prioritize our work towards growth opportunities? How much would that maintenance be considered a burden versus a genuine value add?

I would be naïve to not mention the issues of vendor lock-in that often emerge from relying on vendors, which is why I think paying close attention to open standards and open source is a very underrated skill for platform engineers. Using my domain as an example, in observability we've had a huge issue with vendor lock-in often because there's so much drift in the space. We still see lots of observability platforms popping up. With the emerging OpenTelemetry standard, which targets this exact problem, we're able to move away from being so coupley tied to our observability vendors. I mention this not only because we should keep that in mind for our own work, but also because it's a good tool evaluation criteria. Asking and answering questions like how a given vendor supports the standards in the domain can be very telling for the type of user experience that they're trying to cultivate. Again, to use observability as an example, some of the indicators I've seen on how supportive a vendor may or may not be, how much vendor specific code would I have to break? Are they contributors to the open standard? Do their cost models encourage or discourage using their standard? For example, some vendors treat all metrics submitted by OpenTelemetry as custom metrics with their own specific cost model. That obviously leads to a substantial amount of cash spend. To elaborate on my point about open source, it also presents a good opportunity to practice industry citizenship by being contributors to open standards. By contribution, it takes many forms. It's just not code contributed, also means providing feedback on use cases, reporting bugs, providing documentation, evangelizing their work in the way that I'm perhaps doing now.

Lastly, we have measurement. First, let's talk about the function of measurement, which is ultimately that we have feedback loops for whether our work is actually having its intended impact. These feedback loops consist of both quantitative and qualitative feedback loops for continuous improvement. The way that this principle connects to sustainability is again by eliminating the time spent on work that doesn't ultimately lead to business goals. For example, if a tool that we've spent weeks on isn't actually serving our product engineering users, which leads to a lack of adoption, we've now wasted time that we could have spent serving our paying users. We end up missing the ultimate goal of building applications that age well with evolving business needs. In other words, these feedback loops keep us on the right path towards a continuous improvement that enables us to build new features while maintaining our existing software.

Platform Support

Now that I've defined the principles, we'll shift over to talking about the support strategies for application developers that are integrating with our platform. We'll speak to both scalable and flexible strategies and put it in context with the ways that we measure success, consolidate efforts, and cultivate a culture that values sustainability. The first is the most scalable strategy of all, which is to make building sustainable software easy. Sounds pretty basic. We'll dive into this more in the next section when we talk about our platform architecture. To set the stage for that, let's remember from earlier that sustainable organizations are directly tied to their ability to build sustainable software. What we mean by that, again, is software that ages well with evolving business needs. I'm sure we've all worked on software that clearly wasn't built with much foresight, we've probably also built that software. This is usually what we refer to as tech debt. Tech debt is inevitable, but we can mitigate its risk by being intentional and strategic about when and where we inject tech debt. Tech debt should have a function, usually for the purposes of growth opportunities. Tools that aren't easy to integrate with are a type of technical debt that typically doesn't pay off in the long term. It's especially our duty as platform engineers to build easy to use tools to minimize the time we'd spend needing to support application developers. That's not to say we shouldn't be willing to support developers. In fact, the opposite is true. We just need to be intentional about when the right time to support is and how we can leverage a culture of communal learning and support so that we're not singular points of failure. Not only does this alleviate some tension on us but it enables us to be community driven in the way that we support product engineers. We talked about measurement as a principle earlier. An important measurement of our effectiveness is the adoption of our developer platform. What intention looks like here is ultimately defining the ways that we approach support, and clearly communicating it to avoid mismatched expectations that will degrade trust between platform and product engineers.

I'll only provide one example of this, which is to provide an easy decision framework driven by the status of your tool. A common pitfall for platform engineers is to take on too much integration work. Like the last slide showed, there is room for that in the early experimental mode, but that's not scalable nor are we often even the right people to be integrating our tools into applications that we have no context with. A relatable example here is testing. Platform engineers can certainly create or recommend testing tools or methods, but we can't be the ones ultimately writing those tests. That requires domain expertise. Luckily, if we're doing our jobs right, the time that we save, we save them by providing support, frees up product engineering time to take on that application specific work. Lastly, driving a sustainable culture. Sustainability doesn't just apply to software, it applies to people. The two are very closely tied. Referencing "Accelerate," yet again, it talks about self-sustainability, and the different types of burnout to look out for, whether that's physical, emotional. I've been on product engineering teams before and the reality is that their work is often higher stress. While we have the very real challenges of getting buy-in, that our work is important, which is where this talk came from to begin with, our pressure is often indirect. I found it a lot easier to lean into my leadership and serve as an example of what it looks like to operate in a way that achieves excellence without sacrificing people along the way. Our work is naturally collaborative, and can touch so many parts of our organization, so drive this culture with everything that you can do.

Platform Architecture

Lastly, we have our platform architecture, which is the architecture that platform engineers are building to support application development. Again, the function of this software ties directly to our first support strategy that we talked about, which is to make building sustainable software easy by consolidating efforts, and cognitive load centrally. We've already talked about our high-level principles. Next, we'll review a couple architecture design principles. The first is to embrace design driven architecture as a core principle. To my final point before, we often have more of a privilege to move with intentional design. This intention is what enables us to minimize the technical debt and risk that leads to unsustainable orgs. This principle can be broken down into many pieces. For now, we'll omit it because we'll talk about that and how to alleviate them. Secondly, our architecture should be complementary to those of our users. This is really the core of what we mean when we say when is the right moment to embrace platform engineering. That doesn't mean that we shouldn't try to anticipate developer needs. There's a value in thinking about where our platform architecture might be heading, and where our overall architecture might be heading. To the last principle, we need to design with the future in mind not built for immediately. Much in the same way that we might design a monolith application in a way that will enable us to decompose it into a distributed architecture in the future. When we prioritize our work, we should be driven by the needs of application developers in our organization, whose architecture should be a reflection of our pain and user needs. This is why platform engineering is a steady evolution instead of a fast one. The way that we organize ourselves and prioritize work should be grounded in the reality of the types of problems that product engineers are facing. That's why companies in the earlier stages don't really need a platform engineering from the beginning. It's often too soon to be able to make accurate and thorough decisions on what problems we need to solve for. That often leads to potential consequences that we talked about before, like lack of adoption. Again, as you grow, themes will naturally emerge. Those are what will make a genuine difference. Those are the things that we ultimately need to be paying attention to. That might lead to prioritizing certain domains, whether that's CI/CD, or observability, or runtime language support. Within those domains are problems you're trying to solve. You'll also need to build in a way that's responsive to evolving architecture and developer needs. For example, if we want to improve our runtime experiences of developers, we should prioritize the languages that are actually used by them, not the ones that we want to support, no matter how much we love Go. Ultimately, we're not here to tell developers what to do, we're here to support them in what they need to do. Another reason this is important is because our platform architecture can easily lead to what I would call platform chaos. I know, that's pain.

An overwhelming number of tools, especially custom ones is not the goal. Tools are enablers of our goal to build successful engineering organizations. Instead, we need to be managers of chaos, which leads me to the design tensions that I mentioned earlier. First, we have to think about what I think is the hardest tension to balance, which is standardization versus flexibility. The shared nature of a developer platform is an awesome opportunity to reduce the risk of drift. We have to hold that intention with the flexibility that developers might need, especially as the organization grows, and the number of technical needs grows with it. For example, my organization right now is facing the consequences of building tens of services with an opinionated framework that has since really not aged well. Not only do we have to revisit how we approach runtime support, but we also have to reconcile the tech debt that manifested from this decision years ago. That's reflecting on where we went wrong, but that we're doing now is to engage with our users more. Pretty simple. I started off this talk by saying the opposite of isolation is community. Now we're approaching it from the standpoint of dragging standard definitions with actual teams and our learning communities of practice. Next, we have the tension of simplicity and complexity. As you respond to the evolving needs of your users, complexity becomes harder to manage, because the architecture that supports them is likely subject to change, whether that's speaking using event-driven communication styles or embracing client-side rendered frontends. This becomes another area that we need to be intentional about. Like tech debt, complexity is inevitable, but we can compartmentalize it somewhat by making sure that developer facing interfaces are simple.

Which leads me to the most common source of complexity in software engineering, which is integrations. We know the common design principle of reducing coupling between services, and the same applies to our work. Integrations are high risk because avoiding coupling is incredibly difficult. It's why a huge selling point for some vendors is their integrations. These integrations are also what lead us to lock-in and whatnot. Speaking of vendors, remember our automation principle from earlier. Even though I just spent some time talking through design principles for building platforms, I'm also here to say, give yourself permission to not build at all. The decision to build versus buy versus reuse should be our bread and butter. Deciding that you don't want to take on the work of building and maintaining a tool is a very valid one, because every line of code you write is a liability. Code isn't necessarily the bread and butter of platform engineering, research, design, and technical decisions are. Like I mentioned in the principle's section, we do have to worry about vendor lock-in now. I'll emphasize the point that we should all be paying attention to how industries are defining open standards like OpenTelemetry, open SLOs, open features, which all need contributors, so I'm just going to throw that out there.

Holding these tensions in mind, let's talk through what I would call the platform recipe. This recipe is a reusable framework with four pillars. These allow us to enable, compartmentalize, or communicate the tradeoffs of the tensions that we just talked about. Together, they make up the ultimate developer platform architecture. Let's briefly review each of these pillars, and then provide some examples. First, we have the parts of our architecture that provide standardization. Two examples of how we might do that is through templates or opinionated frameworks. Second, we have the parts of our architecture that provide more flexibility. For example, we might modify our framework to become more modular so that developers can pick and choose based off of their context. We might also extend the way that we do templating by providing a CLI that will make them more dynamic. Thirdly, we have the parts of our architecture that compartmentalize complexity through integrations since the first two pillars are simple interfaces I mentioned earlier. These may be APIs that tie the first two pillars together or plugins to frontend interfaces that developers use to provide visibility that they even exist. These tools are often not directly used by developers in their development, or product developers in their development, and are instead tools that we tend to use to support developers in an automated way. For example, the tools that pull our technologies together to support on-demand environments, engineers will certainly use those environments. The technical details of how we implement those should be abstracted away so that they can focus on product development. Lastly, we have the parts of our architecture that communicate our design and decisions along the way, whether that's API docs, decision records for when we need to build versus buy versus reuse. It might feel a little bit strange to have this considered part of our architecture. As I've emphasized throughout this talk, touch points between platform engineers and product engineers is essential. Especially in a world that's embracing and should embrace remote work, software is going to be able to facilitate that. Documentation is the first obvious one with support channels as a complementary. There's also a potential need for automating certain aspects of our support as we scale up the number of engineers that we're supporting. For example, if our primary communication tool is Slack, integrations that consolidate questions and feedback essentially can enable us to have more feedback loops, that can be really valuable in avoiding adoption issues.

In the integrations pillar, I mentioned plugins, as an example, to provide visibility of what tools exist. That's an essential question to answer because visibility is needed to drive adoption. Let's answer that question of how to provide visibility into our architecture. Backstage is an open source project created by Spotify. Because of its emergence, developer portals have become increasingly popular as a way to provide centralized visibility into the developer platform ecosystem. That's because the concept of developer platform portals is a really compelling one. It tells the story of what it means to develop software at your company. By providing a unified view, not only does our platform become visible, it also facilitates visibility into any service that's using that portal. Integrations are a crucial part of developer portals, but because it's a centralized entry point, we do have to consider the risk of singular points of failure. This is why I embrace this idea of plugins, which should be designed to be easily added or removed from the host application without total coupling. In other words, it provides similar benefits that we see with microservices by having a concrete separation of concerns. I mentioned how developer portals tell a story and what it means to develop software at your company. I want to give more insight on what that looks like by providing some concrete examples that tie these concepts together so far.

Feature Development Cycle (Development)

All you really need to know here is that this is ultimately the story that we're telling, the one that's around our feature development cycle. The ones that are most relevant to us are going to be development, production readiness, and release stages. Those are the ones that we'll focus on. Note that the examples that we're going to see are not the solution, they're a solution. The examples I provide are actually on the naïve side, but the concepts are the same. Starting with development, which supports the end-to-end development cycle, particularly runtime and architectural needs. We embrace standardization here, the first part of our recipe through runtime templates, which minimize repeatable code by centralizing it into a template. What this looks like practically might be a server file like this. This template file not only provides a reusable file, but it also acts as a standardization technique by including our organizational framework of choice, Express. From this, developers would only need to modify the parts that need service context, which I highlighted here. Over time, we could extend this template to also include standards for dependencies like Express that you see here, linters, testing frameworks, which might be reflected in JSON config files. When beginning to build out these templates, we should start simple and build our way out. Rushing into the development too quickly can lead us towards a path of building a developer story, and golden paths that don't actually serve the needs of our users. Alternatively, you might decide to build a runtime framework that codifies standardization techniques, or organizational standards. You can imagine that as your organization grows its standards, that this template file might quickly become overwhelming. For example, we might extend our functionality to include middleware to standardize how we write APIs. As we add more of this code to a template file, it might become impossible for developers to figure out what's being provided for them. Sure, comments can help, but there's a reason that we use abstractions in computer science. In an effort to keep that template simple, we can abstract the details away and replace the Express code with our own framework. Here's what that package code might look like, as well as a simple server file that we'd now be providing. There are of course, many downsides to this. This is a fairly large dependency injection. Now we have to also maintain this code for probably years. It's a commitment that shouldn't be made lightly. It requires a lot of intentional design. It does address drift a lot more effectively, because templates are easier to modify, since the internals of the framework isn't code that they own. The classic tradeoffs of software engineering.

If you find frameworks to be too inflexible for your organization's needs, and want to provide more flexible options, runtime packages are a great tool of our recipe. Here, we still have a lot of decisions to make, including where on the spectrum of flexible we want to be, since packages still can be an avenue where we inject standardization. To round this in an example, we might have this function in a package. This header might indicate something related to authorization that we want to have a standard for. This is a naïve example, so the standard here is just to have the header value be all uppercase. While this function is opinionated, unlike a template or a runtime framework, this functionality is opt in, meaning that while we might want to incentivize developers to use it, we can't really require them to use it. Again, the goal here is to save developers time that they would have spent writing this code themselves. Saving this time for a relatively few amount of developers might not seem that impactful, but as your organization grows, that time building libraries will be paid back in the long term. An additional benefit of packages is that should we ever want to revisit the possibility of building a centralized framework, we now already have some building blocks that we can use to assemble our ultimate framework. Building modules or packages is often a first great step when adopting platform engineering because, again, it allows us to incrementally build for the needs of our developers without investing a huge amount of effort that a full-fledged framework might require.

Now let's move on to the next pillar of our recipe, which are integrations and plugins. Extending on our example, as our organization grows, and our architecture evolves, so might the number of our packages. This was the second architecture principle from earlier. Developers still will need visibility into these different packages. Naturally, we'll probably want to integrate with a package manager, which should be managed centrally, so that it has clear ownership and so that it doesn't become too bloated with packages or artifacts. Conveniently, there might even be a plugin available for our developer platform so that's even more visible. Along the way, we should be documenting our work so that users know how to use our tool smoothly. Combining this with the second principle from earlier, finding ways to automate this will make our lives a lot easier and set our docs up to be less stale. For Node.js, which are my code examples here, this might mean integrating with TypeDoc to automatically generate them, which might then be used by a plugin in our developer portal to render the docs there.

Delivery

Next, we have delivery, which supports deployment by focusing on infrastructure management. Similar to our last example, we can embrace standardization through configuration templates. These can be completely static, or use templating syntax if you decided to use Helm or some equivalent. For our example, we decided to use Helm to minimize the surface area. If you're earlier in your platform engineering story, you probably won't start there. That's the example we're going to use for now. This helps minimize the surface area that developers need to interact with, keeping it simpler and subject to less human error. What this ends up looking like is something like this. Here, we've provided two YAML file templates, one that represents a Kubernetes deployment and the other one that represents the values used for that deployment file. Now developers have less config to manage their software or their infrastructure. The main thing that they have to navigate here is that values YAML file. Next, we embrace flexibility through a similar technique from before, which is modularization. We see, again, similar benefits as we did in development cycle, new flexibility that enables engineers to pick and choose which modules make sense for their needs. It also makes it more maintainable, especially if we extend our platform to embrace a chart generation integration, which we can do in an automated way by setting up a build or deployment pipeline that performs this process. Here, you can see that we need a new YAML file chart.yaml, to set up chart generation. During this process, the values file gets copied from the service repository into the chart generation repository, which replaces that placeholder file. Using a Helm specific CI/CD tool, our pipeline can then generate the Helm chart based on those files. This makes change management easier, especially if you build visibility into the process with observability. Lastly, even though infrastructure detail should be abstracted away from product engineers as much as possible, there are still decisions here that we'll need to make with context that's specific to their services. We saw a values template file a few slides back. While we can provide strong defaults, these defaults likely won't work for any given service using those defaults. Not only do we need to provide context on how we chose those defaults, we need to provide documentation on when those values need to change and how to go about making those decisions.

Production Readiness

Lastly, we have production readiness, which supports the post-delivery needs of software through production readiness techniques. This work often builds on top of the examples we just saw. For example, we might extend the config templates to include features that enable reliability. Again, since observability is a focus area for me, I've gone through it with an example within that space. Let's say that we use OpenTelemetry standard, after considering the tradeoffs of defining our own standard using often open source or using a vendor-specific one. Given this decision, we choose template standardization further by adding a new config in the values YAML to support the standard. Now our developers don't have to worry about managing the config needed to use the OpenTelemetry Collector. Instead, they can focus on instrumenting their applications for telemetry that requires service context. Switching gears into runtime packages, we would extend that development experience to include modules that provide features relevant to reliability management. Again, so using my observability example, that might mean providing an optional middleware function to enable telemetry collection. In this very naïve example, we see a similar attention as earlier where we standardize the key on an attribute, but provide flexibility by managing the value through environment variables, which is not the best thing, but it was the easiest one.

I mentioned that our work needs to align with the needs of our product engineering peers. Let's say that our product has become really successful, we're solidly in the distributed architecture space. Now some of our product teams have new technical requirements that require them to be very cautious about any performance implication, and they also need strong concurrency support, so our organization decides to introduce a new standard language into our organization, Go. This does make our lives as platform engineers more difficult. We now have to support another runtime language, but we shouldn't meet that with resistance. We can provide feedback as peers, perhaps this decision does need some pushback. Being dogmatic about our golden path too isn't much of a recipe for success either. That said, let's say we do have to start thinking about Go development, using the observability example, that means we need to start thinking about what telemetry collection looks like for Go. We already use OpenTelemetry, so we will use that decision for the sake of consistency, except now we come across this. Currently a work in progress. What do we do? We know that telemetry collection is important. How else are we going to debug our production systems? Do we rely on older methods of collection, manual instrumentation, vendor-specific agents? Those are all super valid conclusions, and probably the most practical. I'm still going to take the opportunity to give a nudge to open source.

Our second principle, we've talked about the role of automation in our work, and how one form of that is to reduce the duplicate work that often happens in engineering. More specifically, after the solution of deciding when to build, buy, or reuse existing solutions. Part of the reason for that is not only because we end up spending unnecessary time reinventing wheels, but also because we have to admit to ourselves that a lot of times other people are going to do it better than us. Admitting that doesn't make us any less of an engineer, in fact, it shows humility and appreciation for the work of our peers. It provides an opportunity to learn from one another. We cannot be experts at everything. Engineering is an underrated way to build community, and it doesn't have to be isolated to just your company either. Bringing me back to my observability example, knowing that it's a really exciting time for the observability space, and always many more, we know that there are really smart people already thinking about this. Clearly, there's still so much to learn and work on. Community driven support was a central theme of our previous section. Don't try to learn and build in silos. Learn and contribute to the communities that are already doing this work. Because as much as the sustainability of our organization is about our ability to build sustainable software, the sustainability of our organization is about our industry's collective ability to learn together and make progress together.

Moving on to the third part of our platform recipe. One thing I didn't mention with OpenTelemetry config example is that it uses the decision to integrate with Helm earlier, which has the added benefit of making the interface for developers simple by abstracting away those details. On top of this, however, we can use the embedded telemetry, more specifically metrics to enable automated rollbacks in our CI/CD pipeline for when we release a bad change. Lastly, all these different components and decisions can quickly get overwhelming, even when designed with a lot of intent. Not only do those decisions need to be reflected somewhere, we need to understand that infrastructure incidents are ours too. That the process of incident management should be one that fosters collaboration, and knowledge sharing, which is the value that's reflected in our principles. What this looks like can be a little tricky, depending on whether you have an SRE org, and their specific operating model. Regardless of organizational structure, driving clarity over what incident support looks like, is crucial to avoiding broken stress when our platforms inevitably fail.

Putting it All Together

Now that we've walked through the three components of our platform, let's put this all together by framing it through this lens of the feature development cycle from earlier. More specifically, we'll approach this from the lens of the lifecycle of a service, focusing on the flow that makes up a golden path. Starting from the left here, let's say you're building a new backend service in an organization with a fairly mature platform. To get started, you went to the developer portal, integrated with an artifact manager, select a bundled standardized service template, and enter details like service name. Then an API integration that platform engineering owns, generates that service using an integration with GitHub. In the beginning stages of our platform, we might have only one golden path, that's where I would recommend starting is like focusing on one golden path and then developing from there. Over time, we might have multiple decision points for developers. Perhaps the decision to use a framework-based template over a module-based template, or maybe even to not use one altogether. As a support system, platform engineering should be available to consult and guide them through the process. Now we're ready to develop our service, and depending on our development needs, we might utilize more modules or use some framework features. From there, we'll continue to use GitHub either directly, but now we can also do it through the developer portal since there's an integration, until our PR approval process is done. Insert caveat that that stage isn't quite so isolated from the rest of the cycle here, but that was simpler to make a graph for.

From there, we enter the delivery focused part of our platform. That bundled service template that we use to generate service also includes a config template that the CI stage can use to build tests to perform static code analysis. Now, the developers only have to worry about choosing good values and modifying the values YAML file. Since our portal includes a Drone integration, developers can also easily get visibility into each of these steps, which is especially useful for when they fail. As an additional best practice here, platform engineers should be very intentional about making it clear when CI/CD pipelines are failing because of our tools, versus when they're failing because of an application bug. If the CI stage passes, a CI integration then pushes the build artifacts that we generated, providing access to the rest of the stages of our pipeline. Again, the developer portal provides visibility through its artifact repo manager and Drone integrations. Those artifacts would be used during continuous deployment to perform things like infrastructure management, pre-production testing. Using GitHub or its portal integrations, we move on to the tasks related to release management like approvals, release tagging, release documentation. Then after successful testing and approvals, the service is deployed to production using similar mechanisms as the CD stage, but with additional safeguards like the automated rollback features that I mentioned in the production readiness example. Finally, we've concluded the delivery process and head into maintenance mode, which is supported by the production readiness tools that we provide to observe and monitor our systems.

Recap

I've made a lot of parallels between DevOps and platform engineering throughout this talk. More than anything, I hope it's obvious how important DevOps has been to this discipline, much in the same way that it has been for SRE. Our systems are complicated because people are complicated. Much like architecture design, this type of culture doesn't just happen, it requires intent. The most compelling part of DevOps is ultimately the way that it transforms our culture to be one that fosters exactly that, intent. I started off this talk by saying platform engineering is a solution to building sustainable organizations, but perhaps the solution that we all really need is actually just intent.

Questions and Answers

Participant 1: My primary job is helping companies with outsourcing. Not many people like me. I'm educated and we have a strong pool. What is your opinion on the extension of your presentation, what it means for asking for more help offshore?

Cordero: I think that's definitely a valid thing to do. Arguably, one form of outsourcing is like paying for a vendor. It's a way of outsourcing your platform needs. I think, at the same time you do need to have a strong product partnership. I don't think you can outsource all aspects of it, you still do need that product lens that has context for your organization. While some parts I think are definitely possible to outsource, or solve away with a vendor or whatnot, you still need organizational context. Because, ultimately, like I mentioned a few times, your platform should be reflective of your overall architecture. Unless you have that context, it's going to be hard to do that effectively.

Participant 2: You gave the example of going from like Node, JavaScript, and then you just can do a language like Go, and then you said some of those processes of using telemetry and OpenTelemetry carried over. What are some of the considerations like when in Go, and there's like a whole dictum? The introduction of something new, like a language introduces other new things, like what are some of the considerations?

Cordero: I think I mentioned towards the end that you should start off with one golden path, especially if you're starting on earlier, which is great. Ultimately, you're probably going to have multiple golden paths, actually. Or you'll have one golden path with a lot of backdoors outside of it. That's usually what ends up happening. To my point, I think sometimes you have to push back on decisions to drive basically a new standard. Ultimately, if it's legitimately good for the business, then it's not our job to say, "Sorry, we can't support it," which I think I've definitely seen. We have platform engineering, as much as I obviously love the space, can have a huge elitism problem. That was part of why I said we are not here to tell them what to do, we're here to support them on what they need to do.

Participant 3: You've leaned into OpenTelemetry a lot. Mostly just want to know like, could you talk a little bit more about OpenTelemetry tracing. You mentioned vendor lock-in, I'm wondering how you take in that instrumentation and telemetry and derive value from it and what kind of storage [inaudible 00:47:00] on top of that raw data that I'll start using and getting queried.

Cordero: For The Times, OpenTelemetry is a core part of our observability strategy. We are in Datadog, so our biggest tensions are building our libraries in a way that isn't super specific to Datadog. We've had to make compromises, we still have vendor-specific code. Ideally, the way we're building it is in a way that should we ever need to migrate it's possible. When you're embracing a vendor solution, for example, try to lean into the open standards as much as possible in case one day you need to swap that out. Similar if you try to build something from scratch, try to build with that foresight, because sometimes it will pay off.

The business value that's going to be easiest to anchor on is incidents, because no one likes to lose money when there are incidents, if your ads go down. It's a great business case for prioritizing observability or reliability needs. I think in terms of the buy-in, that's the high-level angle I go with. It's not BS. That's true, like observability will enable you to debug your applications and solve things quicker. That's the concrete business value is ultimately preparing for those possibilities. Because if you don't have the data and also the skill set to handle those well, which also leads to my community idea where you collectively work on your skill sets for debugging applications, you're going to lose business value because of incidents and whatnot. When I talked about, we take for granted maintaining and modernizing our architectures, that's one of the ways that it manifests often is like, we're boggled down by on-call because we've basically been neglecting the architecture that is our core business.

See more presentations with transcripts