QCon London: Efficient Serverless Development

At QCon London, Yan Cui, a serverless advocate at Lumigo, shared patterns for effective local development with AWS serverless technologies. The focus areas were testing approaches, deployment practices, and application environments.

Cui began by discussing the need for fast feedback loops in the development process and the common myth that achieving this with serverless technologies is difficult. He went on to outline that the pain points articulated in the myth can be resolved if appropriate patterns are adopted for testing, deploying, and provisioning serverless application environments.

Starting with a focus on testing serverless applications, Cui shared five possible approaches. The first, Hexagonal Architecture, focused on structuring applications with ports and adapters to abstract away the details of external clients or downstream integrations. While this allows for universal applicability or portability to any runtime environment, Cui made clear the drawbacks of this approach, which included the upfront work to create the abstractions and the absence of direct tests against integrations.

For the second testing approach, "local simulation", Cui shared the applicability of the LocalStack platform to emulate serverless deployment environments on a local desktop. While this could enable fast feedback on a range of serverless technologies outside of AWS Lambda functions, it always suffers from not being a perfect representation of the cloud services.

The third testing paradigm, Lambdalith, involved deploying whole applications to AWS Lambda without decomposing business logic across multiple functions. While this could allow for portability from AWS Lambda to other compute environments and, therefore, ease testing, it would more likely suffer cold-start penalties with larger frameworks and be less optimized for granular access controls across the application’s components.

A fourth testing approach, "deployment frameworks", involved utilizing deployment harnesses like AWS SAM or SST to create representative test environments for local development. While this option has the least overhead, it would not be well-placed for automated regression testing as part of a Continuous Integration process.

The fifth option, "remocal", coined by Cui, a portmanteau of the terms "remote" and "local", referred to the approach of having locally running AWS Lambda function code, communicating with actual remote AWS Services. He argued that this approach best combined the representativeness requirements of testing with the fast feedback of local development.

Cui outlined the main drawbacks of a remocal paradigm with regard to the requirement to provision additional AWS environments and their inability to capture upstream integrations. To mitigate the latter risk, Cui recommended a multi-level strategy, with end-to-end tests covering gaps left by a remocal paradigm. This was illustrated with an AWS API Gateway to AWS Lambda integration.

Following an outline of testing approaches, Cui moved on to elucidate the pitfalls to avoid in the deployment phase. He advocated reducing complexity in AWS Lambda deployment practices and sparing the use of features like AWS Lambda Layers or Container Runtimes because they make a remocal testing approach more difficult to achieve.

Finally, Cui discussed using ephemeral environments as a mechanism to enable remocal testing. With this, the goal would be to ensure that any engineer can have a self-contained grouping of AWS dependencies required to test their branch of code. After successfully testing the developed feature, the environment could be deprovisioned to minimize costs. Cui described this in the context of the Serverless Framework but made clear that it was an approach that could be adapted to any other resources, including server-ful resources like AWS RDS databases, by using ephemeral tables.

Access recorded QCon London talks with a Video-Only Pass.