Transcript
Lane: I'd like to share with you a high level view of why API specifications matter. How OpenAPI, AsyncAPI, JSON Schema and others are helping us standardize and deliver API lifecycle in a way that the machines understand. Also, allows us humans to better communicate, collaborate, and move forward across the API lifecycle.
My name is Kin Lane. I'm the Chief Evangelist with Postman. I lead our Open Technologies team, which includes our DevRel, our API lifecycle group, but also investment in API specifications like OpenAPI, AsyncAPI, and JSON Schema, as well as the tooling that goes around them. API specifications are very important to me, and at Postman. I wanted to share a little bit about how we see API specifications impacting how we produce and consume APIs.
API Specifications Are Not New
API specifications are nothing new. WSDL has been defining web services for the last 20 years. As soon as the more web based approach to APIs emerged, we also saw WADL evolve to help us describe the surface area of our APIs. Really, it wasn't until Swagger came out in 2010, and '11, helping us make our documentation much more interactive, more hands-on. Helping developers understand what an API does and how they can put it to use, that we really started seeing the benefits of API specifications. By 2015, Swagger had grown. It was at version two. It was put into the Linux Foundation, creating the OpenAPI Initiative to move forward the spec, and rebranding it as OpenAPI. Helping us describe our HTTP APIs and webhooks for the last five, six years. Providing a cornerstone of how we define, design, deliver, deploy, and deprecate our APIs. Along the way, AsyncAPI emerged as a sister specification, allowing us to use the same vocabulary but for describing event and message driven APIs, providing a format that we can use across multiple protocols.
OpenAPI
Let's look at just OpenAPI. OpenAPI allows us to describe the surface area of our synchronous APIs, describing the information, what an API does, where you can find it, the servers. More importantly, the individual paths, parameters, and detail of each response, including the schemas that are parsed as request bodies, or returned as part of each of the responses. OpenAPI provides a machine readable JSON or YAML way for us to describe what our APIs are, HTTP/1.1 APIs do, and allows developers to then use that specification at different points of the API lifecycle.
AsyncAPI
AsyncAPI allows us to describe the surface area of our asynchronous APIs, building on what OpenAPI gave us, but allowing us to describe the channels, the messages, as well as the bindings. Doing a publish and subscribe API across TCP, MQTT, AMQP. Allowing for us to define how these APIs work across multiple protocols, and define what schemas are used as part of the payload for publishing and subscribing what messages get sent back and forth to these asynchronous APIs. Both OpenAPI and AsyncAPI allows us to describe most of the APIs that are in our toolbox today.
JSON Schema
JSON Schema allows us to describe and validate the payloads of these APIs. Giving us a machine readable way to describe the objects, their properties, what type of data they contain, and then what's actually required so that we can validate those objects. JSON Schema is used across multiple OpenAPI, as well as AsyncAPI. They have their own vocabularies of JSON Schema that allow them to describe and validate the request and response payload in the case of OpenAPI. The publish and subscribe payloads in the context of AsyncAPI. Providing a machine readable way that allows us to describe what our APIs are doing. What resources are being parsed back and forth, and then what capabilities are encapsulated in these very abstract APIs that can oftentimes be hard to see and validate.
API Specifications Power the API Lifecycle
API specifications are used to power multiple stops along the API lifecycle. Most notably, it's used for documentation. A lot of people who use Swagger believe Swagger is documentation. In fact, Swagger is a specification, and you can use it to generate documentation. Swagger, OpenAPI, and AsyncAPI are used to describe the technical details of what each API does. Then allows us to publish documentation. Allowing us to always keep our documentation up to date, deliver consistent documentation across teams by using that API specification to generate docs using open source or commercial options. The API specifications are also used to generate mock servers. These mock servers can be part of a design-first approach to an API. Designing it and mocking it so that you can make sure it does what it needs to do. It can be used as part of QA and testing. Then, some groups are using it for sandboxes in production environments, so developers don't actually have to be playing with production data when they're learning what an API does. API specifications are also being used as a scaffolding for testing. Allowing API producers to generate contract, integration, performance, and other types of tests that help them make sure an API is doing what it should do, running those tests as part of pipelines or scheduling them using a monitor.
The same approach can be used for security. You can use your OpenAPI specifications or async to scan the surface area of your APIs looking for vulnerabilities, audit the authentication. Check for other common vulnerabilities using the OWASP Top 10, trying to understand whether your APIs are as secure as they should be consistently across teams. Then another common way that API specifications are powering the lifecycle is through code generation. Many companies are using OpenAPI and AsyncAPI to generate client or server side code, so you can actually use the specification to generate code that allows you to deploy the API, deploy it to a gateway, or generate SDKs in a variety of programming languages. Allowing companies to produce SDKs, libraries and code snippets in a variety of programming languages without having to have those programming language skills on their team. This is really an important look at why specifications matter. It's not just for documentation, it's for the whole lifecycle. One spec for an API can be used to generate docs, mocks, tests, secure that API, and then generate the code you're going to use to actually bring that API to life or put it to use in an integration or an application.
Many Ways to Bring your API Specification to Life
There's many ways you can bring your API specification to life. One of the most notable and more recent approaches is through a design-first approach. You begin with an OpenAPI or AsyncAPI, generating a template, and then hand designing it, making sure it does exactly what you need it to do. Then generating a mock server and publishing docs so that other stakeholders, other team members can use that and give feedback on that API, allowing you to design an API, mock it, and document it before you actually write the code. Then once you've iterated upon it enough, you can publish it as an API.
Second common way for producing an API specification is code-first. Teams begin writing code, doing what they do best. Then, while you're writing the code, you either annotate it, or you use a gateway or a proxy. You can generate the OpenAPI and AsyncAPI from the code. You code first. You generate the OpenAPI or AsyncAPI, which then can be used to generate documentation tests and help secure your API. Another common way is to reverse engineer web or mobile applications and generate the OpenAPI or AsyncAPI for APIs that are already in production, already being used in applications. Then, just need to have documentation published and kept up to date. Generate tests and secure that API, allowing existing APIs to be brought into a more standardized API lifecycle, and along with the other APIs that are either design or are code-first.
Industry Standards Use Specifications
API specifications are playing a key role in industry standards, most notably PSD2. You can find OpenAPIs for the PSD2 standard out of the European Union, which is being used to govern and regulate how financial and payment APIs are being deployed and evolved. You can see OpenAPIs being used for the fire specification, defining how APIs are delivered in the healthcare space. Providing all the technical details of each request and response, helping make healthcare more interoperable. Open Referral is another API standard that uses OpenAPI to provide definitions for municipal, city level, county level health and human services, making it so that cities can share data more easily across states and across a country. Another is OpenTravel, shaping how we deliver APIs within the travel industry, working on the airlines, hospitality activities and other things. Providing a common set of standards that can be used to deploy travel APIs, making the industry more interoperable. Another example is the Geospatial Consortium. They're doing geo API standards, and you can find OpenAPI specifications for their APIs. This is just a snapshot. There's plenty of other standards, including the automobile industry, and insurance. Most industries where APIs are making an impact, you can find OpenAPI, and increasingly AsyncAPI in use to define that landscape.
Specifications Provide a Framework for Governance
When in place, OpenAPI, and AsyncAPI, and JSON Schema provide us with a framework for realizing governance across large enterprise organizations. OpenAPI and AsyncAPI provide us with the ability to set design guidelines. How should our paths be described? What parameters? Should they be camel case or snake case? What should schema look like? How do schemas evolve to minimize breaking changes? All of these design guidelines can be codified and applied in a machine readable way to OpenAPI and AsyncAPI. Linting rules. It's very common to lint the OpenAPI or AsyncAPI describing an API, so you can validate that it has a certain structure and form. That's what linting provides us. We can define rules that make up our design guidelines and our overall enterprise governance. We can apply those rules against these API contracts that we've defined. Then we can apply that using contract testing, so we can validate that those rules are being applied at design time, at develop, as well as build time.
These specifications allow us to get a handle on change management. These objects defined by schemas. These APIs defined by OpenAPI and AsyncAPI providing access to those objects are going to change over time. OpenAPI, and AsyncAPI, and JSON Schema allow us to quantify that change. Apply semantic date based or other types of versioning to this, and get a handle on how these resources and capabilities are evolving. Then we can automate all of this. We can automate it so it's in the design tools and you get real-time feedback, when it comes to editing OpenAPI, Async, or JSON Schema. You can apply at pipeline or at develop time in the IDE or the command line, and allow governance to be realized throughout the API lifecycle, and across development teams, helping them automate how governance is applied. Then learn along the way, good governance always has the necessary resources baked into it, so that each governance opportunity becomes an opportunity for learning about why that governance matters and how it works.
API specifications are how we work together as a team, so you see that evidence in documentation. Why Swagger UI and Redoc and specification driven documentation has become so critical. It's how we API producers communicate with API consumers. These specifications as they're growing and being applied across the API lifecycle, is opening up opportunities for collaboration. We can now share artifacts. I can share what I mean when I say API pagination. This is the specification for it. I can share what I mean when I say this is an image object. I can share the JSON Schema for that. That sharing and collaboration around these artifacts, allows us to work closer together and have the same meaning codified in these structures. Then because we can bake this in with our existing source control, we have these artifacts. They can live alongside the code that we use to deploy, automate, and consume, and work with these APIs.
Our OpenAPIs and AsyncAPIs live in the workspaces where they're being designed. They live in the GitHub, GitLab, and Bitbucket repositories where the code is being committed, and being used across infrastructure to deploy and automate with those APIs. All of this allows us to standardize across teams, so we're communicating better, we're collaborating, we're sharing, and we have these artifacts that help us understand what it means when we talk about these very abstract API concepts. It helps us get on the same page, standardize how we do things, and then automate that through source control. This can be done at the team level, the organizational level, but also using standards at the industry level, allowing APIs to be more consistent and iterative throughout their lifecycle.
Specifications are the Language of Our Business Operations
These OpenAPIs, when they mature, provide us with the ability to define the different domains of our operations. This is how we separate the different lines of business, the different domains that we use to operate. It can very much describe and define how our teams are made up. How we organize our teams, and how they work within these different lines of businesses and groups. OpenAPI and AsyncAPI and JSON Schema, with tagging across all of them, allow us to carve up all of our resources and digital capabilities within these different domains, describing the different resources that we have. The different images, videos, people, and addresses, all the different digital resources we have, as well as the capabilities, the algorithms and the actions that are taken across those resources. Describing all of this in OpenAPI, tagging things, helps us establish and formalize our vocabulary for how we describe this. We're using these schemas that we're publishing to APIs as part of requests, getting back as responses, publishing, subscribing in an event driven API world. We're standardizing how we describe these objects, how we talk about them. We're reducing redundancy across these resources and capabilities, creating much more logical domains for how we operate. Make sure that is efficient and well defined as possible. Then the domain experts can really help guide and manage this part of our business operations.
Specifications Are For Humans and For Machines
Specifications are for humans and for machines. Even though they're written in JSON or YAML, and very much machine readable and structured so that they can be imported into different services and tools. They can be used as part of source control. The API lifecycle can be automated in that way. They are very much for humans as well. It's very important that we use them. Iterate upon them. Document them, and make them accessible to humans for use across their work. Specifications are often seen as a developer tool. Increasingly, they're being used by project and engineering managers, and API designers and architects to shape the domains across their enterprise organizations, and used as a way of doing design reviews, security reviews, and other ways. Using it as an artifact to guide conversations with developers who are actually building the APIs and the applications on top of them. It's important for specifications to be there for them to be up to date, because the humans depend on this. The specifications are helping us address a lot of the human challenges and friction that we encounter across our organizations. It's important to elevate it, zoom out, and see specifications as not just being documentation or not just being code gen, or just a JSON artifact that's on a server somewhere. This is very much a contract that humans are using to move forward our operations and find success.
A Standardized, Specification-Driven API Lifecycle
A standardized API lifecycle means that we have all of these resources and capabilities defined as machine readable artifacts. We're not just focusing on HTTP/1.1, REST, or the existing ways of deploying APIs. We have a diverse toolbox of artifacts that allow us to describe event driven and message driven, GraphQL, gRPC, Kafka, NATS. We're able to use a full suite of API solutions that help us describe, and define, and deliver, and operate these APIs in a way that benefits both the producer and the consumer. This helps us standardize how we're defining and designing APIs. How we're delivering. How we're doing it at scale across teams, across organizations. When you go to a large enterprise group, and you go to put an API to work, it has the same characteristics. It has the same properties, parameters. It has consistent documentation. It has the code libraries I need, and they use conventions that match the programming language I'm using.
All of this is driven from these API specifications. OpenAPI and AsyncAPI are very much being used as the guardrails for this. They're being published to a central catalog, either internally within an organization or externally, with partners and third party consumers. These OpenAPIs, and AsyncAPI, and JSON Schema can be used in any services or tools. The API producer has a much smoother, high velocity, efficient, agile, and nimble way to delivering APIs, being able to respond to changes and get their APIs out there. The consumers of those APIs have access to those same contracts, so they know when things are changing. They're able to automate, and build, and integrate around those APIs without having to wade through the documentation. They can just import an OpenAPI. They can import an AsyncAPI. They can use the JSON Schema to validate and test. Then they can grab all of the tools that an API producer has built around these specifications, and put them to work.
A standardized specification-driven API lifecycle is what you're seeing emerge within most enterprise organizations who have spent the last decade doing public APIs, or the last five or six years heavily investing in microservices. You see them really working to get a handle on using OpenAPI across their enterprise organization. Adopting and learning how to use AsyncAPI. Then using JSON Schema to get a handle on their internal data, schema, artifacts, that's being parsed around. This is the number one priority for organizations, for architects, for leadership in some of these more progressive forward leaning companies to adopt these API specifications. Standardize around them. Then, apply it across a well-known API lifecycle.
Summary
I know a lot of folks are down in the weeds when it comes to APIs and just focus on docs or code gen, or design and the architecture around these OpenAPIs. Hopefully, this zooms out and shows how OpenAPI, AsyncAPI, and JSON Schema are working together. More importantly, how they're applied across a consistent API lifecycle. Because once you can standardize and stabilize it across the lifecycle in this way, this is where you're going to start seeing the benefits when it comes to operational velocity, performance, quality, and overall governance of what's going on. I recommend heading over to OpenAPI, or AsyncAPI, or JSON Schema, all three open source solutions that let you describe how you're going to be doing your APIs. Each of those communities have a lot you can learn from and put to work as part of your operations.
Questions and Answers
Betts: Specifications are for humans and machines, seems to be a recurring theme. The keynote was on the programmer's brain and what it takes to read and understand code. There was another talk later that hit on the readability factor, that code needs to be readable. Can you go a little bit deeper into how it is readable? Because I see YAML, and I don't think that's human readable. It might be programmer readable. What makes it human readable?
Lane: YAML is a start. I've seen business stakeholders get more involved when there's a switch from JSON to YAML, just amazing what brackets and commas and quotes going away will do. I agree. It's still meant for programmers. How do you make it more readable is you make them smaller in scope. Rather than creating big monolithic APIs, you create smaller microservices and smaller product based APIs that are simpler to understand what they do in scope. Then you follow a domain driven design, and you do your event storming. You really find the vocabulary in the language that is as meaningful and purposeful as possible. The design of your API is intuitive. It speaks to the domain. It speaks to the folks who are stakeholders in the business conversation. All of those things are going to contribute to it. It's less Latin or Greek, trying to read it when it's in YAML, and it's a familiar vocabulary that we're used to. It's simple in scope, and it's easier to digest for folks. All of that design effort really contributes to that. I agree, we still got a long ways to go, and an OpenAPI or a JSON Schema is still a little more technical than it should be.
Betts: Is that somewhere that the tooling comes into place that you can then generate documentation that is more human readable? That it's just, use the YAML, use the JSON Schema for the organization of the common words, and then someone's going to read your Swagger page, whatever it is.
Lane: I'm the first to admit, really OpenAPIs, AsyncAPI, they're just config files for whatever this other outcome that we're looking for. Really, us humans shouldn't be that bothered with it. We should be able to hit view-source. We should be able to get to the config file. Really, that end result is what we should be getting in. It's up to the tooling and service providers to get us there. Until then, we need to elevate these specifications up a little bit and get more eyeballs and get more people using them, so that the tooling can get us there.
Betts: Since you mentioned DDD, I was going to ask about that because DDD has the idea of ubiquitous language. It's the ubiquitous language within a bounded context. Here, you're talking about your API, which is the external interface. It's still a ubiquitous language, but it's at a different layer. Are there different people involved? Is it the business stakeholders, as opposed to maybe the development team working with their technical product manager?
Lane: Yes, very much, depending on the scope of your APIs and services you're building. OpenAPI, whether it's more product, external facing API, whether it's a microservice. Whether you got event-driven microservices, and you're connecting them together with a more async Kafka, that type of approach, an event driven approach. Your domain conversations are going to shape based upon what you're trying to accomplish. Is it just internal? Is it just private? Is it with partners? Is it industry focus? Your domains are going to be very scoped by those goals and those objectives. It varies across the landscape. Really, that's the domain driven design exercise you should do, is flesh a lot of that out. What is our domain? What are the bounded context? What should our interfaces look like? What should all the objects that we're parsing around? They work in concert. Just based upon whether it's internal, partner, private, who your audience is, is really going to define a lot of that.
Betts: These specifications talk about the request-response, the input, and then the output that comes right back, but it doesn't go as deep as providing what happens later. Are there examples for that? Are there specifications of what are the events that get fired when I send this request to my API? Because then you can document through all the specifications.
Lane: Yes. AsyncAPI has the concepts of event driven publish and subscribe. There's a Venn diagram overlap between the two. I wouldn't say it's as coherent and clear, as it should be. Say, I'm working with a regular API making requests and responses, how do I learn about the webhooks so I can then get a push? After I make a request, I get a response back that I got my response, but there's some other thing that's going to happen after it. In OpenAPI, there's link objects that you can say, this is what's next. This is what comes next. I believe you can even go so far as to say, if you want to subscribe to this, here's a WebSockets header that would then point you over to a WebSockets to actually subscribe to this. I've seen a couple Bitcoin APIs that did that hand-off between HTTP and TCP, which is WebSockets. As far as how you describe that in OpenAPI and AsyncAPI, that relationship between request and response and event driven worlds, it's not as smooth as it should be. I see a lot of folks working on trying to make it more meaningful.
Betts: One of the other questions was about sub-specifications for common parts like paging, expansion, internationalization? Can you link between two files so that everyone doesn't have to repeat the same definition of signs that is common across all of your services?
Lane: Yes, reuse. In OpenAPI, JSON Schema, and AsyncAPI, you can reference, so there's a dollar sign ref. OpenAPI 3 was very much about reuse. It has a components library where you can put parameters and put things, and then just reference them within the doc. You can also do that outside of an OpenAPI. You can have an OpenAPI that's just like all of your common parts, your pagination, your request bodies, all the things that you reuse. Then you can reference those with a full URL in your OpenAPI, and reuse those. That concept exists across all three of the specs. The problem is tooling support for that, as tooling support is not as robust and consistent as it should be. That's something that we're really working on because there's a lot of great patterns out there, a lot of great reusable patterns that we can put to work. The tooling to help you do it, you got to know how to do it and be a hardcore editing your OpenAPI by hand rather than relying on the tool to do it in these days. We'll get to fixing that in the next year or two.
Betts: On that same trajectory, you mentioned that Swagger was around 10 years ago, but then became OpenAPI, became part of the Linux Foundation, 2015. AsyncAPI is a few years delayed. It just got into Linux Foundation this year. It just hit version two. Is it the same thing, just step back a few years? Can we expect a similar, it takes a few years to get there, or is it going to ramp up faster?
Lane: I think it's going to ramp up faster. I feel like we're very much 2012 when it comes to Swagger, like the amount of tooling, the amount of energy in the space around it. We're at that same point with AsyncAPI, but we're going to get there faster because of the core tooling. OpenAPI Initiative does not manage any tools. It only manages the specs as part of the Linux Foundation. Back when Tony Tam created Swagger and ran Swagger, he had Swagger UI. He had Swagger code gen. He had core tooling. There's that symbiotic relationship that allowed Swagger to grow rapidly, in the 2.0 version. We've had trouble with that in 3.0. The core tooling, and the relationship between the parser, the generator, the docs, the code gen, and the specs is critical. That's why AsyncAPI is going to explode. It's already exploding, and it's just going to, exponentially, going further.
Betts: It's similar to your API starts getting used, you have to worry about the backwards compatibility. You have to make more people happy and make more decisions, and maybe go a little slower as you evolve, so you can get to the next great thing.
Lane: You got to be more thoughtful.
Betts: All these resources can create this very diverse toolbox. You mentioned REST, gRPC, and GraphQL. Those tools exist for all the tech stacks, but are they mature for all of them, or is the REST, the JSON much further than gRPC and GraphQL?
Lane: We're reaching maturity point. GraphQL and gRPC, both are very mature in the enterprise toolbox, seeing wide adoption. The community scopes aren't quite as big, and some of the confidence isn't fully there, from a storytelling, what you hear in the space. We're there. There, definitely, I'm seeing a lot of enterprise adoption that involves both of those. I'm seeing across the spectrum. I'm seeing WebSockets with GraphQL as publish and subscribe, which is an interesting hybrid. Then with gRPC, I'm seeing a lot of binary investment in serialization of JSON Schema and other things. They're reaching a point of maturity, and they're starting to influence the rest of the spectrum, but REST has a massive head start.
See more presentations with transcripts
