Debugging Go Code: Using pprof and trace to Diagnose and Fix Performance Issues

Software development relies heavily on debugging techniques, which become crucial for handling matters of efficiency efficiently. The frustration that users feel upon experiencing slow program execution emphasizes the importance of identifying and solving underlying issues effectively through debugging tools.

However, debugging performance issues in software can be hard due to the vast codebases or intricate systems involved in its creation and implementation processes.

In Go, developers have access to powerful built-in tools that can help diagnose and fix performance problems. Two of these tools are the pprof and trace packages.

The pprof package allows you to profile and analyze the execution of Go programs, whereas the trace package allows you to trace and visualize events and goroutine activity. These tools, when used together, can provide useful insights into the inner workings of a Go application and assist developers in identifying and resolving performance bottlenecks.

In this article, we will look at how to identify and fix performance issues in Go programs using the pprof and trace packages.

We will begin by covering the fundamentals of profiling and tracing before delving into practical examples of how to use these tools in real-world circumstances. By the end of this article, you will have a solid understanding of how to use these powerful tools to improve the performance of your Go applications.

Understanding Performance Issues

Performance issues in Go programs, or any software application, can have a substantial effect on user experience, resulting in lower efficiency, application usage, and profit for businesses. Performance issues in Go programs can occur for a variety of reasons. In this section, we will look at some of the most common reasons for performance concerns and how they affect a system.

• Inefficient Algorithms: Inefficient algorithms can have a major impact on performance, especially when working with huge datasets. These algorithms take additional CPU cycles and memory resources, which can slow down the overall application. Examples include brute force search methods and ineffective sorting algorithms. In this type of issue, the CPU resources are not distributed evenly throughout the board, resulting in some functions having limited access to resources.
• Blocking Operations: An application may occasionally wait for an I/O activity to complete, such as reading or writing data to disk or connecting to a network. During this process, blocking operations can occur and cause delays in execution, resulting in decreased performance. When an application is blocked, it is unable to do other useful tasks, resulting in a loss in overall performance.
• Excessive Memory Usage: Go apps that utilize a lot of memory might cause performance issues, especially on low-resource systems. If an application consumes more memory than the system has available, the system may begin swapping to disk, drastically slowing the application's performance. This can also occur if the application does not manage memory effectively, resulting in memory leaks and other problems.

Slow applications result in poor user experience, dissatisfied consumers, and lost income. Therefore, in order to achieve the best possible experience, it is crucial to optimize your Go applications.

Profiling with pprof

pprof is an built-in package in Go that provides a profiling tool for developers to measure how their Go programs use CPU and memory. The data from this measurement is then collected and analyzed to discover patterns that provide evidence of performance issues. With the pprof package, developers can easily measure and identify functions that consume more CPU memory than normal, as well as parts of the program that allocate the most memory.

Let's assume a Cash App makes use of Go, and that it has a feature that allows users to send friends money using QR codes. After updating the feature, its developers noticed that the app was running significantly slower than usual, with 40% of users complaining about delays of up to 15 seconds when scanning QR codes and sometimes payment failing. To properly analyze the issue, the development team can use pprof to generate CPU profiles while users are scanning QR codes. By analyzing the profiles, they might discover what functions are taking up too much CPU memory or what algorithms are inefficient. After discovering the issues and fixing them, they can once again test and use pprof to make sure the performance has improved and the experience is faster and seamless.

Types of profiles

There are different types of profiles in Go and they can be generated using the pprof package. In this section we will take a look at these different profiles and their usefulness:

1. CPU profile: measures how the functions consume different amounts of CPU time, making it easier to identify which functions consume more of it and could be potential bottlenecks.
2. Memory profile: measures how an application makes use of memory and which parts of the application allocate more memory.
3. Block profile: this shows where the program is blocking (e.g.  I/O or synchronization primitives), making it easier to identify areas where concurrency may need to be improved.
4. Goroutine profile: makes it easy to detect areas with too much parallelism or where concurrency might need to be improved, by returning statuses of running, blocked and waiting.
5. Trace profile: this captures a detailed log of events that occur during the execution of a program, e.g. goroutine creation and destruction, scheduling, network activity, and blocking operations. It is very useful in detailed analysis of an application’s performance.

Analyzing profiles

To better understand the concept of profiling and how it helps the debugging process, let’s take a look at the example below. In a new file called main.go, add the code below:

package main
import (
    "fmt"
    "math/rand"
    "os"
    "runtime/pprof"
)
func main() {
    // Create a CPU profile file
    f, err := os.Create("profile.prof")
    if err != nil {
        panic(err)
    }
    defer f.Close()

    // Start CPU profiling
    if err := pprof.StartCPUProfile(f); err != nil {
        panic(err)
    }
    defer pprof.StopCPUProfile()

    // Generate some random numbers and calculate their squares
    for i := 0; i < 1000000; i++ {
        n := rand.Intn(100)
        s := square(n)
        fmt.Printf("%d^2 = %d\n", n, s)
    }
}

func square(n int) int {
    return n * n
}

In the preceding code:

• The handleRequest function generates a random number between 1 and 1000 and then computes its square root. The result is finally written to the response body.
• The pprof.StartCPUProfile function initiates CPU profiling, which results in the creation of a profile file that can be analyzed later.
• The defer pprof.StopCPUProfile() statement makes sure that CPU profiling is stopped at the end of the program, whether it finishes normally or due to an error.
• The http.ListenAndServe function starts and manages the HTTP server on port `8080`.
• In the main loop, we create some CPU-bound workload by calling rand.Intn(100) 100 million times. This creates random integers between 0 and 99 and is a CPU-intensive operation that we can profile.

To test, run the command below, which will display the squares of a few numbers:

go run main.go

This will run the program and create a profile.prof file in the same directory as the main.go file. To analyze the profile, run the following command, which will give you an interactive shell to explore the profile:

go tool pprof profile.prof

To show the top functions by cpu usage, type the command top

To get the memory profile, modify the code to use the pprof.WriteHeapProfile() function to write the heap profile to a file. You need to add code to write a memory profile to a file (mem.prof) after generating the random numbers and calculating their squares. You will also add a time.Sleep(5 * time.Second) call to allow time for the memory profile to be written to the file. Find the updated version of the code below:

package main

import (
    "fmt"
    "math/rand"
    "os"
    "runtime/pprof"
    "time"
)

func main() {
    // Create a CPU profile file
    cpuProfileFile, err := os.Create("cpu.prof")
    if err != nil {
        panic(err)
    }
    defer cpuProfileFile.Close()

    // Start CPU profiling
    if err := pprof.StartCPUProfile(cpuProfileFile); err != nil {
        panic(err)
    }
    defer pprof.StopCPUProfile()

    // Generate some random numbers and calculate their squares
    for i := 0; i < 10; i++ {
        n := rand.Intn(100)
        s := square(n)
        fmt.Printf("%d^2 = %d\n", n, s)
    }

    // Create a memory profile file
    memProfileFile, err := os.Create("mem.prof")
    if err != nil {
        panic(err)
    }
    defer memProfileFile.Close()

    // Write memory profile to file
    if err := pprof.WriteHeapProfile(memProfileFile); err != nil {
        panic(err)
    }
    fmt.Println("Memory profile written to mem.prof")

    time.Sleep(5 * time.Second)
}

func square(n int) int {
    return n * n
}

After running go run main.go, a mem.prof file will be generated. In the interactive shell, type top to analyze the memory usage of our program. To bring out this interactive shell, run the command:

go tool pprof mem.prof 

To show the top functions by CPU usage, type the command top

As we can see from the examples we just reviewed, the pprof tool provided valuable insights into the performance of the programs, allowing us to easily identify any irregularities and potential areas for optimization. We can see that by incorporating profiling into the development process, it becomes easy to proactively identify and address performance issues, resulting in faster and more efficient applications.

In the first example, we saw how to create a CPU profile and analyze it using the pprof tool. The output showed the number of times each function was called, and the total time spent executing each function. This allowed us to identify functions that were consuming the most CPU time and potentially optimize them. In the second example, the output showed the memory usage of each function, including the number of allocations and bytes allocated. This allowed us to identify functions that were using excessive amounts of memory and potentially optimize them to reduce memory usage.

By incorporating profiling into our development process, we can effectively identify and address performance issues, resulting in faster and more efficient applications.

Tracing with trace

Sometimes, we need more detailed information about how our program runs. In cases like this the trace package is a very powerful and useful tool. In this section we will go over it.

trace is a tool that allows you to collect detailed information about how your program is running. It is very useful in understanding things like how goroutines are created and scheduled, how channels are being used, and how network requests are being handled. It provides a timeline view of your program's execution, which can be useful in identifying performance issues over time and other types of bugs.

trace can collect data on a wide range of events that occur while your program is running. Those events include the following: Goroutine creation, destruction, blocking, unblocking, network activity, and garbage collection. Each trace event is assigned a timestamp and a goroutine ID, allowing you to see the sequence of events and how they relate to one another.

Analyzing Trace Data

To begin, we will make a new go file, name it trace.go. To generate trace data, import the runtime/trace package and call trace.Start at the start of your program. To stop the trace collection, call trace.Stop at the end of your program. Here's what it would look like:

package main

import (
    "fmt"
    "math/rand"
    "os"
    "runtime/pprof"
    "runtime/trace"
)

func main() {
    // Create a CPU profile file
    f, err := os.Create("profile.prof")
    if err != nil {
        panic(err)
    }
    defer f.Close()

    // Start CPU profiling
    if err := pprof.StartCPUProfile(f); err != nil {
        panic(err)
    }
    defer pprof.StopCPUProfile()

    // Start tracing
    traceFile, err := os.Create("trace.out")
    if err != nil {
        panic(err)
    }
    defer traceFile.Close()

    if err := trace.Start(traceFile); err != nil {
        panic(err)
    }
    defer trace.Stop()

    // Generate some random numbers and calculate their squares
    for i := 0; i < 10; i++ {
        n := rand.Intn(100)
        s := square(n)
        fmt.Printf("%d^2 = %d\n", n, s)
    }
}

func square(n int) int {
    return n * n
}

Run the command below to kick-start your program:

go run main.go

To analyze the trace data, you can use the go tool trace command followed by the name of the trace file:

go tool trace trace.out 

This will launch a web-based visualization of the trace data, which you can use to find out how your program is running and identify performance issues.

You can also see details about the various goroutines and how the various processes ran! trace is an excellent tool for understanding various flow events, goroutine analysis, minimum mutator utilization, and much more!

Analyzing and Fixing Performance Issues

After collecting performance data with pprof and trace, the next step is to analyze the data and identify possible performance issues.

To interpret the output of pprof, it is necessary to first understand the various types of profiling data available. The most common types of profiles are CPU and memory profiles, just like in the examples cited earlier. It is possible to identify functions that consume a lot of resources and may be potential bottlenecks by analyzing these profiles. pprof can also generate other types of profiles, such as mutex contention and blocking profiles, which can assist in determining synchronization and blocking issues. A high mutex contention rate, for example, may indicate that multiple goroutines are competing for the same lock, which can result in blocking and poor performance.

As mentioned, trace data contains more comprehensive data about an application's behavior, such as goroutines, blocking operations, and network traffic. Trace data analysis can be used to detect sources of latency and other performance issues, such as excessive network latency or inefficient algorithmic choices.

There are several methods for optimizing performance once performance issues have been identified. A common strategy is to reduce memory allocations by reusing objects while reducing the use of large data structures. It is possible to reduce CPU usage and improve overall program performance by reducing the amount of memory that can be allocated and garbage collected.

Another method is to use asynchronous I/O or non-blocking operations to reduce blocking operations such as file I/O or network communication. This can help to reduce the amount of time the program spends waiting for I/O operations to complete, as well as improve overall program throughput.

Additionally, optimizing algorithms and data structures can significantly improve performance. The amount of CPU time needed to complete operations can be decreased and overall program performance can be enhanced by choosing more effective algorithms and data structures.

Conclusion

It is important to optimize performance in Go applications to ensure that they run efficiently and effectively. We can improve user experience, lower the costs of running applications, and improve the overall quality of our code by doing so. We can use tools like pprof and trace to analyze CPU and memory usage and identify bottlenecks and other issues in Go applications. We can then make targeted improvements to our code based on the output of these tools, such as reducing memory allocations, minimizing blocking operations, and optimizing algorithms. Profiling tools such as pprof and trace are vital for identifying and addressing performance issues.