Using Golang to Build Microservices at The Economist: A Retrospective

I joined The Economist engineering team with the job title of Drupal Developer. However, my real task was to engage in a project that would fundamentally reshape the technology delivering Economist content. My first few months were spent learning Go, working with an external consultant  for several months on building an MVP, and then rejoining the team to guide them on their journey to Go. 

This shift in technology was driven by The Economist’s mission to reach a broader digital audience as news consumption moved away from print. The Economist needed more flexibility to deliver content to increasingly diverse digital channels. To achieve this goal of flexibility and maintain a high level of performance and reliability, the platform transitioned from a monolith to microservice architecture. Services written in Go was a key component of the new system that would enable The Economist to deliver scalable, high performing services and quickly iterate new products.

Implementing Go at The Economist:

• Allowed engineers to quickly iterate and develop new features
• Enforced best practices on fast-failing services with smart error handling
• Provided robust support for concurrency and networking in a distributed system
• Lacked some maturity and support in some areas required for content and media
• Facilitated a platform that could perform at scale for digital publishing

Why did The Economist choose Go? 

To answer this question, it’s helpful to highlight the overall architecture for the new platform. The platform, called the Content Platform, is an event based system. It responds to events from different content authoring platforms and triggers a stream of processes run in discrete worker microservices. These services perform functions such as data standardization, semantic tagging analysis, indexing in ElasticSearch, and pushing content to external platforms like Apple News or Facebook. The platform also has a RESTful API, which combined with GraphQL, is the main entryway for front end clients and products.

While designing the overall architecture, the team investigated what languages would fit the platform needs. Go was compared against Python, Ruby, Node, PHP, and Java. While every language had its strengths, Go best aligned with the platform’s architecture. Go's baked in concurrency and API support along with its design as a static, compiled language would enable a distributed eventing systems that could perform at scale. Additionally, the relatively simple syntax of Go made it easy to pick up and start writing working code, which was a quick win for a team going through so much technology transition. Overall, it was determined that Go was the language best designed for usability and efficiency in a distributed, cloud-based system.

Three years later, did Go meet these ambitious goals? 

Several elements of the platform design were well aligned with the Go language. Failing Fast was a critical part of the system since is was composed of distributed, independent services. Aligning with the Twelve Factor App principles, applications needed to start and fail quickly. Go's design as a static, compiled language enables fast start up times and the performance of the compiler has continually improved and never been an issue for engineering or deployments. Additionally, the Go error handling design allowed applications to not only fail faster, but fail smarter.

Error Handling

A difference engineers quickly notice in Go is that is does not have exceptions, rather it has an Error type. In Go, all errors are values. The Error type is predeclared and is an interface. An interface in Go is essentially a named collection of methods, and any other custom type can satisfy the interface if it has those same methods. The Error type is an interface that can describe itself with a string.

type error interface {
    Error() string
}

This provides engineers with greater control and functionality around error handling. By adding an Error method that returns a string in any custom module, you can create custom errors and generate them like with the New function below, which comes from the Errors package.

type errorString struct {
    s string
}

func (e *errorString) Error() string {
    return e.s
}

What does this mean in practice? In Go, functions allow multiple return values, so if your function can fail, it will likely return an error value. The language encourages you to explicitly check for errors where they occur (as opposed to throwing and catching an exception), so you code will commonly have an "if err != nil" check. This frequent error handling can seem repetitive at first. However, Error as a value enables you to use the error to simplify your error handling. In a distributed system for example, one can easily enable retries by wrapping errors. 

Network issues are always going to be encountered in a system, whether  sending data to other internal services or pushing to third party tools. This example from the Net package highlights how you can take advantage of error as a type to distinguish temporary network errors from permanent ones. The Economist team used similar error wrapping to build in incremental retries when pushing content to external APIs.

package net

type Error interface {
    error
    Timeout() bool   // Is the error a timeout?
    Temporary() bool // Is the error temporary?
}

if nerr, ok := err.(net.Error); ok && nerr.Temporary() {
    time.Sleep(1e9)
    continue
}

if err != nil {
    log.Fatal(err)
}

The Go authors believe not all exceptions are exceptional. Engineers are encouraged to sensibly recover from errors rather than let the application fail. Additionally, the Go error handling allows you to have more control over errors, which can improve things like your debugging or the usability of errors. Within the Content Platform, this design feature of Go enabled developers to make thoughtful decisions around errors, which resulted in stronger reliability of the system as a whole.

Consistency

Consistency is a critical factor in the Content Platform. At The Economist content is the core of business and the Content Platform’s goal is to ensure content can be published once and read everywhere. Therefore it’s essential that every product and consumer has consistency from the Content Platform API. Products primarily use GraphQL to query the API, which requires a static schema that serves as a contract between consumers and the Platform. Content processed by the Platform needed to consistently align with this schema. A static language helped enforce this and enabled an easy win in ensuring data consistency.

Testing with Go

Another feature that improved consistency was Go's testing package. Go's fast compile times combined with testing as a first class feature enabled the team to embed strong testing practices into engineering workflows and quick failures in build pipelines. The Go tooling for tests makes them easy setup and run. Running "go test" will run all tests in a current directory and the test command has several helpful feature flags. The "cover" flag  provides a detailed report on code coverage. The "bench" test runs benchmark tests, which are denoted by starting the test function name with the word “Bench” rather than “Test”. The TestMain function provides methods for extra test setup, such as a mock authentication server. 

Additionally, Go has the ability to create table test with anonymous structs and mocks with interfaces, improving test coverage. While testing is nothing new in terms of a language feature, Go makes it easy to write robust tests and embed them seamlessly into workflows. From the start, The Economist engineers were able to run tests as part of build pipelines with no special customization and even added Git Hooks to runs tests before pushing code to Github. 

However, the project wasn’t without struggles in achieving consistency. The first major challenge for the platform was managing dynamic content from unpredictable backends. The platform consumes content from source CMS systems primarily via JSON endpoints where the data structure and types were not guaranteed. This meant the platform couldn’t use Go's standard encoding/json package, which supports unmarshalling JSON into structs, but panics if the struct field and incoming data field types do not match. 

To overcome this challenge, a custom method for mapping backends to a standard format was needed. After a few iterations on the approach, the team implemented a custom unmarshalling process. While this approach felt a bit like rebuilding a standard lib package, it gave engineers fine grained control of how to handle source data. 

Networking Support

Scalability was a focus for the new platform and this was supported with Go’s standard libraries for networking and APIs. In Go, you can quickly implement scalable HTTP endpoints with no frameworks needed. In the below example, the standard library net/http package is used to setup a handler that takes a request and response writer. When the Content Platform API was first implemented, it used an API framework. This was eventually replaced with the standard library as the team recognized it meet all of their networking needs without additional bloat. Golang HTTP handlers scale because each request on a handler is concurrently run in a Goroutine, a lightweight thread, with no customization needed. 

package main

import (
    "fmt"
    "log"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello World!")
}

func main() {
    http.HandleFunc("/", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Concurrency Model

Go’s concurrency model provided several performance gains across the platform. Working with distributed data means wrestling with the guarantees promised to consumers. As per the CAP theorem, it is impossible to simultaneously provide more than two out of the following three guarantees: Consistency. Availability. Partition tolerance. In The Economist platform, Eventual Consistency was accepted, meaning reads from data sources will eventually be consistent, and moderate delays in all data sources reaching a consistent state is tolerated. One of the ways this gap was minimized is by taking advantage of Goroutines. 

Goroutines are lightweight threads managed by the Go runtime to prevent thread exhaustion. Goroutines enabled optimizing asynchronous tasks across the platform. For example, one of the Platform’s data stores is Elasticsearch. When content is updated in the system, content referencing that item in Elasticsearch is updated and reindexed. By implementing Goroutines, reprocessing time was reduced, ensuring items are consistent faster. This example illustrates how items eligible for reprocessing are each reprocessed in a Goroutine.

func reprocess(searchResult *http.Response) (int, error) {
	responses := make([]response, len(searchResult.Hits))	
	var wg sync.WaitGroup
	wg.Add(len(responses))
	
	for i, hit := range searchResult.Hits {
		wg.Add(1)
		go func(i int, item elastic.SearchHit) {
			defer wg.Done()
			code, err := reprocessItem(item)
			responses[i].code = code
			responses[i].err = err
		}(i, *hit)
	}
	wg.Wait

	return http.StatusOK, nil
}

Designing systems is more than simply programming, engineers have to understand what tools work where and when. While Go was a strong tool for most of The Economist’s Content Platform needs, certain limitations required other solutions. 

Dependency Management

When Go was released it had no dependency management system. Within the community several tools were developed to meet this need. The Economist used Git Submodules, which made sense at the time as the community was actively pushing for a standard dependency management tool. As of today, while the community is closer to an aligned approach for dependency management, it's not there yet. At The Economist, the submodules approach didn’t pose significant challenges, but it has been challenging for other Go developers and is something to consider when transitioning to Go. 

There were also requirements for the Platform that Go’s features or design was not best suited for. As the Platform added support for audio processing, the Go tooling for metadata extraction at the time was limited, and so the team chose Python's Exiftool instead. Platform services run in docker containers, which enabled installing Exiftool and running it from the Go application.

func runExif(args []string) ([]byte, error) {
	cmdOut, err := exec.Command("exiftool", args...).Output()
	if err != nil {
		return nil, err
	}
	return cmdOut, nil
}

Another common scenario for the Platform is ingesting broken HTML from source CMS systems, parsing the HTML to be valid, and sanitizing the HTML. Go was initially used for this process, but because the Go standard HTML library expect a valid HTML input, a large amount of custom code was required to parse the HTML input before sanitation. This code quickly became brittle and missed edge cases, and so a new solution was implemented in Javascript. Javascript provided more flexibility and adaptability for managing the HTML validation and sanitation process.

Javascript was also a common choice for the event filtering and routing in the Platform. Events are filtered with AWS Lambdas, which are light weight functions spun up only when called. One use case is to filter events into different lanes, such as fast and slow lanes. This filtering is done based on a single metadata field in an event wrapper JSON object. The filtering implementation took advantage of the Javascript JSON pointer package to grab an element in a JSON object. This approach was far more efficient compared to the full JSON unmarshalling that would be required with Go. While this type of functionality could have been achieved with Go, using Javascript was easier for engineers and provided simpler Lambdas.

Go Retrospective

After implementing the Contact Platform and supporting it in production, if I was to hold a Go and the Content Platform retro, my feedback would be the following:

What Went Well?

• Key language design elements for distributed systems
• Concurrency model that’s relatively easy to implement
• Enjoyable to write and fun community

What Could be Improved?

• Further progress on versioning and vendoring standards
• Lacks maturity in some areas
• Verbose for certain use cases

Overall, it's been a positive experience and Go is one of the critical elements that has allowed the Content Platform to scale. Go will not always be the right tool, and that's fine. The Economist has a polyglot platform and uses different languages where it makes sense. Go is likely never going to be a top choice when messing around with text blobs and dynamic content, so Javascript is in the toolset. However, Go's strengths are the backbone that allows the system to scale and evolve. 

When considering if Go would be the right fit for your, review the key questions for system design:

• What are your system goals?
• What guarantees are you providing your consumers?
• What architecture and patterns are right for your system?
• How will your system need to scale?

If you’re designing a system that aims to tackle the challenges of distributed data, asynchronous workflows, and high performance and scaling, I encourage you to consider Go and how it can accelerate your system goals.

About the Author

Kathryn Jonas currently works as a Software Engineer with Teachers Pay Teachers and previously worked at The Economist as Tech Lead for the Content Platform. Jonas has lead projects for organizations in Beijing, London, and New York, applying technology to diverse challenges such as mission impact evaluation, editorial transparency and trust, and online learning and collaboration. She thrives on engaging software architecture debates and working with animated, empowered teams.