Fran Méndez on AsyncAPI

Transcript

Introduction [00:21]

Thomas Betts: Hello, and thank you for joining us for another episode of the InfoQ podcast. I'm Thomas Betts, co-host of the podcast, lead editor for Architecture and Design at InfoQ and a senior principal software engineer at Blackbaud. Today I'm speaking with Fran Mendez about AsyncAPI. Fran is the founder and project director for AsyncAPI and the director of engineering at Postman. Fran, welcome to the InfoQ podcast.

Fran Mendez: Thank you, Thomas. Happy to be here. Thanks for inviting me.

What is AsyncAPI? [00:46]

Thomas Betts: So, let's just start with what is AsyncAPI?

Fran Mendez: Boom, right. The big question. AsyncAPI is an initiative, and I would like to clarify this. It's an initiative composed of a specification and multiple tools to ease the development of event-driven architectures. So far, we have seen similar initiatives in the REST API space, like OpenAPI, like RAML, and many others. And we were missing something for event-driven architectures. And that's why I created the AsyncAPI specification first, which was like taking a fork, if you want, of OpenAPI, but modifying the essential things to define messaging systems instead of REST APIs. And from there, it grew into a bunch of things, not just the spec, but a bunch of tooling, a bunch of really basic tools. But when I say basic, it's not as in the complexity they have, but as in the leave that people may have with these tools, right? Like a parser, code generators, documentation generators, and react component, a playground and many other little tools that are composing this whole bundle of different tools.

Thomas Betts: So, you say a fork from OpenAPI, I think. It's OpenAPI, but a lot of people still call it Swagger, which is what I still refer to it as. And so you started with the spec, saying here's how we're going to define an asynchronous contract. Very similar to how you'd use OpenAPI to define a RESTful synchronous API contract?

Fran Mendez: Correct. So, the idea behind the spec is to define contracts, to define the endpoints, which are usually called topics or channels or event names. There are many names for this, right? So, if you use Kafka, these are topics. If you use MQTT, these are topics as well. If you use RabbitMQ or AMQP protocol, these are routing keys. So, there are many names for this thing. And aside from the definition of the endpoints, you want to define if your application is publishing or is subscribing to a specific endpoint, which kind of message can you expect there? What's the shape of the message which is described using JSON schema? And from there, we managed to be able to generate documentation, generate code, and do a lot of other stuff. I said that you can define the message using JSON schema, but the truth is that we decoupled this a year ago from JSON schema.

Fran Mendez: And now you can also specify other languages, like Avro, for instance. So, you can use AsyncAPI with Avro and natively. So, it's natively supported on AsyncAPI as well. So, you can just say, I want to define my messages with Avro and you can specify a specific filter to say that. And if you already have Avro schemas, you can just reference them from a AsyncAPI file, or you can reference your Avro schema in a schema registry. So, you don't have to really find the whole thing again, using this schema.

Shared channel contracts planned for version 3 [03:51]

Thomas Betts: Let me parse back a couple of things you said. One, you mentioned that the definition will say whether you are a publisher or a subscriber, and then the contract seems to be the shared piece. So, does it make sense to have that contract called out separately and then the publisher and the subscriber each individually reference it and say, I want to use that, and you're going to use that as well?

Fran Mendez: Actually, this is one of the things that we want to introduce on version three, because we've seen many people struggling with this, that they had a contract saying, I'm a publisher on this topic and I'm a subscriber and you have another one saying I'm a subscriber on the same topic. And as you can expect the shape of the message is the same, because we're talking about the same message flowing through a pipe if you went through the topic channel. And people were struggling to reuse this definition. So, one thing that we want to do for the next major release is to decouple this operation, which is publish subscribe from the channel definition and the message definition. So, different services can make a reference to a contract. Let's say, channel contract, if you want.

AsyncAPI is protocol independent [04:56]

Thomas Betts:  And then the other part was you mentioned Kafka and RabbitMQ. So, this is really language and platform and tooling independent, it is just the specification. And so what you choose to implement it in is still up to the individual developers.

Fran Mendez: Correct. So, we don't stick to any specific protocol. Actually the protocol field on the AsyncAPI document, it's a free form string. So, you can put whatever you want there. And the reason is on version one, there was a release of supportive protocols, but we realized there was a problem and it's that many companies have, especially the big ones have custom protocols like internal protocols. And they weren't able to use AsyncAPI because the list was a fixed list of supported protocols. They wanted to use their protocol, but of course they also don't want to publish their protocol to be able to use AsyncAPI. So, what we did instead is, okay, it makes sense to open this protocol field to any string. It's just the responsibility of the tooling to understand it. And then you might find tooling that understands a specific protocol or not.

Fran Mendez: And it's up to the developer to specify which protocols are supported. Our tools, I will say that we tend to support all the protocols that we have. Not all the protocols, but most of the most used protocols, if you want. It's impossible to support all the protocols. And I'm saying it's impossible because there are many protocols that we might not even know about. And that happens very frequently that we just hear about a new protocol and it's like, oh wait, really? What is this new protocol? So, it's impossible for us to keep up with all the protocols, but at least main ones like AMQP, MQTT, Kafka WebSockets, Storm, NATS, for instance, as well. All these protocols are for sure supported by all our tools. So, we make sure that our tools support it. But again, our tools support it.

Fran Mendez: The spec is agnostic in regards to protocol, and in regards to the system that we have behind. The only thing that you do with the spec is define how do you connect. How do you send or receive messages from an endpoint. It's like we don't care what's behind that endpoint. If it's Kafka or if it's, I don't know, MQTT broker mosquito or something, that's up to you. That's not our problem. This is the definition of your API. It's a way to interact with the system behind. We treat the system behind as a black box. We don't want to know information about how your system is done behind the scenes, mainly because you probably want to share this AsyncAPI document with the public, with your audience. And then you don't want to define how your system is done behind the scenes there.

Thomas Betts: Also gives you the ability to change things, how you're implementing it, but you keep the contract the same.

Fran Mendez: Exactly. So it's... I can't remember the principle, the computer science principle, but it's... I think it's information hiding. If I'm not mistaken. So, it's separation of concerns here.

Tooling for generating a spec from your code [07:59]

Thomas Betts: Good obstruction. So, anyone who's had to hand roll and write their own Swagger docs and the OpenAPI documents, either in YAML or JSON knows that you quickly want to have the tooling to help you out and generate. So, I've read through the documentation online and it says, here's what all the fields represent. And here's how you define it. But you've talked a little bit about the tools. So, what do you have in terms of... It seems like there's two sides to this. Generate the AsyncAPI spec, but also to take the AsyncAPI specification and generate a client that will agree to that contract.

Fran Mendez: So, on the second side of things, taking an AsyncAPI file and generating clients or servers or different things, that's covered. So, we've got this covered using our generator tool, it's actually called Generator. We're not very creative here, with the naming. On the first part, how do you craft an AsyncAPI document? Right now we have nothing yet. It has to be done by hand. It's true. We've seen this for years on the OpenAPI community, on the Swagger community as well, that people struggled with this. And there were some tools popping up to help you with that task. So, our hope here is that we'll see tools appearing or the system we want giving support to AsyncAPI. It's not just hope. So, it's not that I'm sitting back here in my chair and wishing that someone does it.

I'm always having conversations with different parties and telling them how difficult will it be to implement support for AsyncAPI. One of them, I will say, is obvious from my position. So, Postman is going to implement AsyncAPI support. So, from their, UI you will be able to generate your AsyncAPI document. And not now, obviously it's not yet implemented, but then along this year, I will say, we'll have this stuff. And also you will be able to test and mock asynchronous APIs with Postman, which is really happening. We only introduce WebSocket support. But I don't want to end the conversation around Postman. So, things like, for instance, the other products like Stoplight, which came to my mind in terms of tools to create OpenAPI documents with a nice UI. I think they are in the best position to give support to AsyncAPI.

And actually we are the most requested feature on Stoplight to give support for AsyncAPI. So, that's funny. I don't know if they're working on it, but I'm sure that they will work on it. And I think that because I mean, they're great people, they've a great product, and I think that at some point this is going to happen. But yeah, we're missing that part. We're also going to work on a tool which is called AsyncAPI Studio. And it's still in the very, very early days. And our idea is to create a UI that you can install or deploy yourself. We're not going to offer this as a service. And it's a tool that allows you to do precisely that. That allows you to create AsyncAPI files, not just one, but maybe multiple and draw your architecture. And from that drawing, an interactive chart, let's say, and you can generate AsyncAPI files. So, that's something that is on the roadmap and it's being worked on right now, but it's in the very early days.

Tooling for generating clients and servers from a spec, mocking and testing [11:14]

Thomas Betts: So, what's on the tooling on the other side? Then, I have my specification, what can I do with it? What tooling can I use to read that specification and then pull that into my code?

Fran Mendez: Most of the tooling is actually in that side. So, if not all of the tooling, I would say. So, you have a generator, our tool, which is called Generator. And the generator is capable of generating markdown documentation for your service or for your client. It's capable of generating HTML documentation. There's a react component as well, for the documentation. You can generate nodeJS Cloud, Java code with Spring Cloud Stream framework. There's a Python generator as well. There's a go code generator as well.

So, there are a few of them and more are appearing. And there's community contributed tools like microbes, for instance, and Microcks allows you to use OpenAPIs and KPI, and many other formats to mock and test asynchronous APIs. So, I totally recommend people to have a look at Microcks and they really did a great job. And they're contributing to the spec, based on the feedback from the users. And it's a really good tool. And this is all coming from the community. This is not something that we maintain. So, with an AsyncAPI file, you can do things like this. You can mock and test like Kafka broker or Kafta endpoint. Kafka topic, which is super cool. And they're going to add support for WebSockets now. So, it will be even cooler.

Thomas Betts: Yeah, that's good to hear, because I know this is one of those problems that people have with asynchronous and event-driven architectures, is the disconnected nature of testing it becomes that extra challenge. How do you set it all up? How do you test it without sending messages around? And if they don't come back, what do you do? So, just being able to mock that out reliably and know that you're following your contract is very good to hear.

Developing a proxy, similar to an API gateway, but for event-driven protocols [12:58]

Fran Mendez: On that side as well. We're building now an event gateway. So, that's for production use. Like this is for runtime use, I would say. And the idea behind that is like an API gateway, but for event-driven protocols, like Kafka, like MQTT, AMQP. So, we call it gateway because people are more familiar with gateways, but it really is a proxy, if you think about it. It's a TCP proxy that is just reading your messages, whatever your protocol is. It's a transparent thing. So, people don't have to notice that there is a proxy there and it's validating, for instance. Imagine that you put it between your Kafka client and your Kafka broker and you put it in the middle. You put the gateway in the middle. And what it's doing is analyzing your messages and comparing them with your AsyncAPI definitions.

And if they're not valued, then you can do many things. One of them is blocking the message at all. You can let it pass. You can let it pass and trigger an alarm, whatever. There are many, many things. And this is something that we're developing as part of the initiative. We want to make it like a white label product, so that other people can reuse and probably resell and offer as a service. Or just simply just install it themselves in their infrastructure.

Current state of the open source initiative [14:14]

Thomas Betts: You talked about how the specification itself is being modified. And so this is all an open source project, an open source initiative, started with the specification. And so the specification itself is still open source as it evolves. What version are you on right now?

Fran Mendez: On version 2.1. as of today. We just released that at the date of this recording, we've released 2.1 today. And yeah, pretty excited because actually, I will say, before we started recording the call right... So, I want to share this with the audience. I was just saying that I'm super excited because it's the first time that I'm not involved in any of the features of the spec. So, I was super happy, like, oh, I'm glad that this is alive. It's not just my stuff. Just Fran doing some crazy stuff on his spare time.

Thomas Betts: Right. That's a good indication of a strong, open source project. Good community support.

Fran Mendez: Exactly. And that's what I've been trying to advocate for doing this all these years. And I wanted to make it clear. It's not that I don't want to work on the project that I created. It's totally the opposite. I want it to be a community project. I don't want it to be a project of one company or one guy. I want it to be community-driven project. Truly community-driven project. Happy that it's happening.

All tooling and the specification are on GitHub [15:26]

Thomas Betts: Yeah. That's very exciting news. The tooling and all those other things. Is this all just in one shared GitHub repository or is there multiple for all the different projects in different languages?

Fran Mendez: So, everything that we maintain as an initiative, let's say... I don't want to call it official tooling, but let's say that initiative tooling it's on the AsyncAPI organization on GitHub. Everything hangs in there, like it's hangs from there. So, all the reports are stored there, if you want. And everything is open source. We do nothing that is not what I have to be cautious with double negations in English, but we didn't do anything that is not open source. So, everything we do is open source. For us, exactly it goes against our ethos to create something that is not open source. So, we want to think about this.

Thomas Betts: If someone says, I have a need for... And maybe it exists and you just didn't list off, but .NET Core support. And they say, I want to have a generator that writes to .NET. Someone can just go and create a new generator and add it up there and-

Fran Mendez: Please do it. Unless you don't want to share it. But our intentions here are that we create a strong community, like I said. And if you contribute something, then you become the owner. If you want, of course, you become the owner of that piece and you become part of the TSC of the initiative. So, you will become a TSC member just because you contributed the tool. But let's say your super powers as a TSC member is restricted to your project. So, you manage your project, but you can also take decisions and votes. You can vote on decisions that affect the whole initiative. You can participate in the voting, which is funny. We've made it that way so that everybody can contribute to the initiative and everybody can vote. But the fun thing here is that all the discussions are happening on specific repos or specific tools.

And we always come up with an agreement on that place and we didn't even have to vote yet for anything. We always come up with a consensus on the specific place, which is super cool. That's what we were trying to do. We have voting system on the TSC, in case we want to vote about something that you're not able to come up with a decision, but that never happens. Usually people discuss in a healthy way and that's something I also want to make it clear and to put it like in a big headline.

The AsyncAPI community is very welcoming. People are encouraged to join [17:45]

Fran Mendez: AsyncAPI is a super welcoming community and a super friendly community. And everybody can see just by joining our Slack and workspace that... We have people there from all the, let's say, different levels. If you want of experience, different ages of experience. And we're always helping each other as much as we can. Sometimes not even related to is in KPI, but to programming in general or other stuff. And yeah. So, I try to always explain and tell people that really, that we want to be. We are a really welcoming community and we want to continue being like that. I encourage people to join.

Thomas Betts: Yeah, that sounds really good. So, when you talk about the TSC, I assume technical steering committee, just to unpack the acronym.

Fran Mendez: Yeah. Technical steering committee. Sorry for using this. Yeah. I should have clarified.

Thomas Betts: Just over a radio it's easier to spell things out sometimes. So, is everything that the TSC does over Slack or do you have virtual meetings? Because everyone's scattered around the globe.

Fran Mendez: Like I said, we never needed to do a voting yet, so we didn't have to do anything or put it anywhere because it wasn't needed. Maybe it happens on Slack. The discussion happens on Slack, but the result of the decisions are always put on specific GitHub issues or pull requests. So, all the things are there.

Fran Mendez: We try not to have meetings as much as possible. And so far it's working. At least not for taking decisions. So, we never take decisions in meetings. And that's because we're in a huge world here with many time zones. And people have lives. So, you might not be able to join a meeting today, but you want to be part of it. You want to be considered as part of the decision. So, we always do it in [inaudible 00:19:23]. We have meetings every two weeks for the people to join and ask whatever they want. That's something that we do. One week is Europe and Americas friendly. And two weeks later is more like ACI and Europe friendly. So, we cover all the time zones as much as we can. But again, we try not to make any decisions on meetings because we want to be async.

History of AsyncAPI versions [19:46]

Thomas Betts: I was just going to make that reference, that you can't have synchronous demands. Everyone has to get into a meeting together. And then your asynchronous model falls apart. So, you're on to version 2.1 as of today, again, the day we're recording. So, this will be coming out in a few weeks. Obviously 2.0 must have come out a little while ago. How long has AsyncAPI been in the works? How long have you been working on it?

Fran Mendez: That's a good thing to highlight. So, I release version 2.0 in September, 2019. So, it's taken almost two years to release this minor version. But don't be scared. That was done on purpose. So, at that point we realized that the spec was good enough, wasn't perfect, but it was good enough to provide value. So, then we decided to focus our efforts on tooling. Our limited efforts at the time. All efforts are limited, but at that time they were even more limited than today. So, Lukasz and I decided that we wanted to focus on building tooling, on providing value out of the spec. So people can actually get the value and experiment and give feedback to the spec as well, at the same time. Well, we've done that for a year and something. And now we've decided a few months ago, I think it was March, we asked the community, we have an issue open for several weeks and we've been asking practically many people like, Hey, what will be the new release cadence for AsyncAPI?

Future releases of the spec will be four times per year [21:18]

Fran Mendez: And we came up with an agreement between everyone in the community that wanted to participate. And we're going to be releasing four times a year from now on. So, we're releasing always in the same months. We're releasing in June, in September, in January. And if I'm not mistaken, I think it's April. Yes, April. We're always going to be releasing in these month. No matter the year. No matter the content of the release. We don't know if it's going to be major or minor. If it's minor, we always try to grab together all the breaking changes together. And if we do a major release, we put all the breaking changes there.

Fran Mendez: We don't want to be pushing breaking changes every three months. So, for instance, we want to publish version three, which is a new major version. And most probably this will be done in January, not in September, because again, we want to group all the breaking changes as much as possible. And so this is going to be happening on average every three months. We're going to be releasing spec versions every three months. That's not to mention that, for instance, if there is something that we need to fix. Any trail fixes, typos, things like this. These are released immediately. So, you probably see in the upcoming weeks, you probably see a 2.1.1, 2.1.2. But if you read the spec, it says that the last number, the patch number of the version is non-normative. So, will not affect anything in the spec. Just fixes, typos and things like this.

Long-term support of spec versions [22:46]

Fran Mendez: So, these are completely automatic and they're released immediately. And as for how long are we maintaining the spec versions? Because that's the next question. Because if we're going to be delivering so fast, then what happens with support? So, we don't have an end date of version two yet. And I don't think it's going to happen anytime soon. We probably will be maintaining version two for, I don't know, say five or six years probably, because that needs to be done this way. But that shouldn't stop us from delivering new versions. If people want to upgrade, they should be able to upgrade. If they want to get stuck on version two, that's fine as well. Just be aware that at some point this is going to be discontinued. That support is going to be discontinued. But yeah, like I said, in a very long time. This is something that we actually have to decide, but we don't want to decide ourselves alone or a small group of people. We always try to involve the community to make these kind of decisions. Feel free to join.

Partnership with Postman [23:45]

Thomas Betts: I know you briefly mentioned Postman, but I wanted to circle back to what the partnership was with Postman. And also you had announcement last year about working with the Linux Foundation for hosting. So, you can talk about those two things?

Fran Mendez: First place, we were like in summer last year, it's actually a year now. We were... Lukasz, which is my friend working on the spec and the initiative, and I we were discussing how to make the project sustainable. And that wasn't easy. It's impossible to maintain such a big project and grow in project with donations only. So, we were just trying to come up with ideas. One of the ideas was to create our own company and sell a product or a service and try to sell services around AsyncAPI. It can be a startup. It can be a consulting company. Or maybe both in the beginning, who knows. I was doing consulting at that time. So, that's how I was able to sustain myself and Lukasz at the time for, let's say for, for a year or so. but that wasn't going to take long.

Joining the Linux Foundation[24:48]

Fran Mendez: And more if we want to grow. So, we decided we have to find something else. Creating a company was an option, but we weren't happy with the option of finding venture capital to work on a standard. We wanted to become a standard, an open source standard and usually standards and money interests, I would say, are not very well aligned. So, we want to keep it like separate and just focus on the standard part. Even if we have less money, if that's okay, then we'll find a different solution. Then luckily I got my friend, [inaudible 00:25:25], come to me and say, hey, let me speak to Abhinav. Abhinav is the CEO of Postman and let's have a chat. And so I went to tell the whole story, but after a few months we announced that we are joining Postman. And the agreement here, the partnership here is that we joined Postman.

So, we are Postman employees. We need to clarify that. We're Postman employees, but we're working full-time on the spec, on the AsyncAPI initiative. So, on spec and tooling. We don't work on the Postman product. Only myself, I am the only one who has a double role, if you want. I'm helping Postman implement support for other protocols and for AsyncAPI. I am the only person with this double role and the rest of the team is purely working on the AsyncAPI initiative full-time. So, we wanted to make it clear. But saying something is one thing and doing something is another. So, we wanted to make it clear for the people that Postman wasn't buying AsyncAPI. That was the main concern that people had at the time. And so this only made it faster to come up with this next step, which is joining a neutral ground, like Linux Foundation.

And that's why we joined Linux Foundation. That was the main reason. We wanted to be part of a neutral ground. So, it's not perceived as Postman is controlling the project or AsyncAPI is a Postman project now. We want it to be neutral. Postman is obviously investing a lot of money there because it's hiding [inaudible 00:27:01]. So, this is all the money they're investing, but they're not the only ones there. Solace, MuleSoft, Salesforce, Slack, and lately IBM as well. They're contributing engineering time to the spec and to the tools. So, that's something to take into account. And that's something that we want to make it grow. We want to have more companies to commit. Money is okay. But engineering time, I will say, is even more valuable.

Thomas Betts: That's good to hear. I said earlier, having the community support that you didn't have to write all the code for the version 2.1, it sounds like it's a really healthy project. I'm really happy to hear how it's all moved along. I personally want to go find some time to try and use it out myself and see if it would help my teams. Because I know this is one of the things that we struggle with, is how do you document the contracts? We've got strong support using OpenAPI. And I think our async communications, it's a little bit behind the curve and I think that's common for a lot of companies these days.

Fran Mendez: Well actually, one of the items that we have on our roadmap, which isn't our website by the way, is that we want to make the learning curve of event-driven architectures a little bit flatter. So, it's easier to learn event-driven architectures and not just to learn, but implement. And implement something valuable in a few minutes. And I don't want to make an announcement yet, because I want to make it in September, but in September there's going to be a huge move around this. So yeah, stay tuned for that.

Thomas Betts: We'll definitely stay tuned. Well, I want to thank you again, Fran, for being on the podcast today. We'll make sure to have links in the show notes to the AsyncAPI website, the Slack channel, and everywhere else. The GitHub repos, that people can find ways to participate. And if people want to follow you particularly, where can they most easily find you?

Fran Mendez: I'm usually on Twitter and LinkedIn as well, but mostly on Twitter. My handle is fmvilas. That's my second surname. In Spain we have two surnames. So, my handle is fmvilas. You can find it there on Twitter as well.

Thomas Betts: Well, thanks again for joining me today, Fran Mendez. And thank you to all of our listeners for joining us on the InfoQ podcast.

Fran Mendez: Thank you.

Mentioned

• AsyncAPI Initiative 
• AsyncAPI Initiative GitHub 
• AsyncAPI Slack workspace 
• AsyncAPI Studio
• Stoplight 
• Microcks
• InfoQ news:
  • AsyncAPI and Postman Partner to Bring New Tooling to Asynchronous APIs
  • The Linux Foundation Announces Hosting of AsyncAPI
• OpenAPI Initiative
• Postman
• Lukasz Gornicki (twitter @derberq)