Dynamic Value Stream Mapping to Help Increase Developer Productivity

the system that people work in and the interaction with people may account for 90 or 95 percent of performance.
W. Edward Deming

Technology and software have come a long way in improving engineering teams' and organizations' productivity, however, the effect is limited. Over the last 3 years, we see large organizations start investing in a more systematic approach to engineering productivity: Google acquired DORA, DevOps industry research group, Microsoft has their developer velocity lab that created SPACE framework. 

We can see thousands of tools, dozens of methods, and services available designed to minimize the effects of efficiency hindering issues: partially work done, unneeded features, handoffs, wait time, context switching, extra processing, and simply defects. Netherreless, the software industry still experiences relatively poor engineering productivity because of the absence of a systematic and holistic approach, avoiding local optimization or narrowly focused solutions. Exploring why this is the case, we can point to the invisible nature of engineering work, where there is no physical conveyer belt, there are no raw materials you can see, and due to a number of specialized systems (Task Tracking, Code Repository, CI/CD, etc.), it is hard to observe the end-to-end process of value creation, from idea to feature consumption. 

Recently Gartner recognized the Value Stream Management market, the Value Stream Management consortium uniting key industry players was founded early in 2021 to promote holistic value stream management techniques optimizing the whole chain of  software delivery capabilities that rely on lean concepts.    

In other industries such as manufacturing, supply chain, or distribution we can find a wide adoption of process analysis and lean practices for efficiency gain. 

Let’s consider Value Stream Mapping, the key practice facilitating process analysis and further improvement. A Value Stream Map depicts every step in the process and categorizes each step in terms of the value it adds and what value is wasted. Despite the fact that the software industry has been adopting lean principles [Lean Software Development Principles (Poppendieck and Poppendieck, 2003) and The Principles of Product Development Flow (Reinertsen, 2009)]  the technique of value stream mapping has not gained a lot of traction in the software industry. 

 So what prevents software engineering organizations from adopting value stream mapping as a foundation for software delivery capability optimization? Even though the practice has existed for a long time and is well known among agile and lean coaches, knowledge and adoption among enterprises are falling behind. It might be the absence of a specific role dedicated to the value stream mapping. 

Dr. Mik Kersten one of the leaders of the value stream management consortium suggested the introduction of a new role, value stream architects who will own the software value streams and focus on their flow acceleration. Even without the formal designation of  such a role, the responsibility for software delivery capability optimization resides within organizations, and it is spread across managerial roles and is often done poorly. 

It might be a lack of industry knowledge and therefore education. Most engineering managers come with a computer science background; their project and program management book of knowledge mentioned lean methodologies only in passing, if at all. 

Let’s take a look at how value stream mapping can be used. The creation of a value stream map is just the beginning of the journey. Once we recognize the flow and reflect on it, we need to analyze and identify where the waste comes from to find the bottleneck and plan what to change and what to change to. 

Here is a list of the steps required to utilize value stream practice.

Figure 1. Value Stream Usage Process

Below we provide a simplified example of the end-to-end software development process. Where you can find the value stream map with key steps of business analysis, development, quality assurance, and deployment. 

Figure 2. Example of the Simple Value Stream Map 

Our experience working with multiple clients across various industries shows that there are a number of challenges in the way of end-to-end software process change implementation, the most common challenge is not a technology one rather it is organizational. Most organizational structures are functionally-oriented, therefore their objectives are aligned with their functions, not with the flow of value. So it is not a surprise to have unbalanced flows characterized by long lead times, unreliable quality, misalignment between business and engineering teams. Decoupling of the value delivery team composition from the formal organizational structure and realignment of the incentives model to the value stream is a prerequisite for success. 

The second common challenge of value stream mapping is data collection. It takes a lot of effort and time to overcome contemporary organizational bureaucracy and valid data security concerns. Getting access to the systems isn’t enough  data engineering and analytics are required before we can plug data into the map. 

To understand how value stream mapping is used among lean practitioners working in non-software industries and in software,  we  ran a survey. We worked with two groups, with these profiles:

• The Disciplined Agile group is comprised of 12,000+ members most of them are associated with Agile practice; 72 people participated in the survey;
• The Value Stream Mapping group is comprised of 40,000+ members from lean, manufacturing, supply chain, and other production-related industries; 296 people participated in the survey.

We asked people two questions “How long does it take to collect the required information to depict a value stream map?” and “How much time do you need to validate Value Stream change Impact?” 

We found that to build a value stream map agile coaches need more time, as well as to plan change and evaluate its impact. These findings might be related to the fact that value stream mapping is a rather emerging tool among agile coaches, or maybe data collection is more complicated, or maybe there is a lack of simulation tools that can shorten the time for making a decision, or the complexity of the processes that lean practitioners work on is too high to build models. It is hard to conclude what is the cause of such a deviation and we encourage continued research in the area.  

One of the limitations of the Value stream mapping is that it gives you a snapshot in time, even though the processes are dynamic. And it is hard to do a process improvement without observing the  system in motion, therefore dynamic value stream simulating tools become an essential decision-making tool. Dynamic value stream simulation helps not only to understand current fluctuation, but also to find answers to several future-state design questions that are hard to answer in another way:

• What is the takt time of the process
• How we can balance the process to avoid inventory/queue growth
• How variability will affect the process performance 
• What is the resilience of the process
• And many others  

 Stephen L. Woehrle, Ph.D., and Louay Abou-Shady, MS., summarize key benefits from the dynamic value streams in the following way:

• Simulation helps in exploring several alternatives to the future state that can be generated by having different responses to the design questions. 
• Simulation can help management get a better comparison between the performance of the existing system and that of the suggested lean system, thus supporting the decision to adopt a lean approach.
• VSM Simulation can help organizations quantify the expected benefits of lean manufacturing at the planning and evaluation stage.
• The information generated from the simulation can be used as a benchmark to measure the effectiveness of implementing the lean changes.

You can explore the dynamic characteristics of the simple software development example provided above  

It shows us what would have happened with the lead time, work in progress, throughput, and activity ratio on the time scale given the specified parameters: new requests distribution, batch process (scrum) or continuous flow (kanban), about of rework at each step, number of people required for a specific step execution, etc. There are many options that we can tweak the dynamic value stream model to explore various scenarios.    

An option worth exploring is if we create a cross-functional team working on the implementation and testing of a requirement simultaneously.

First of all, we can see 20% improvement in the speed of delivery 5 days and 6 hours (126 hours) vs 6 days and 17 hours (161 hours); therefore, the number of items delivered for the same period of time went up to 226 versus 168. As a consequence, the efficiency of the process (Activity ratio of processing time to lead time) was doubled from 3% to 6%, still it is quite low. Such a low percentage might suggest that we don’t have enough people to process all requests coming into the process also we have sprints with different cycles durations that is not synchronized. What would be if we switch to the continuous flow (no cycle time) and add more people to our bottleneck (development and testing)? We will give a reader opportunity to explore and experimenting with these and many other options to to get to the optimal state.

Given all benefits and all successful cases among multiple industries, value stream mapping is still an emerging practice for software development. However, we have learned several lessons through adopting this practice in large enterprises. Summarizing our experience, we can say that the successful adoption of value stream mapping requires several factors:

Leadership support driven by business needs. The most successful adoption happens when a company pivots its business model toward tech-enabled businesses where software development,  information, and digital technologies become a competitive advantage. A typical driver for such a  case is time-to-market acceleration. Therefore, the efficiency of software delivery capabilities becomes a strategic imperative for business.

Start small and then scale. The Value Stream mapping practice is a holistic method that requires the involvement of all process participants to create a map and focus on the end-to-end optimization, so it is wise to start small, build credibility, and only after that adopt at scale. And to start there is no need to look for a tool and run through the long procurement process, installation, and pilot to get to the current state only. It is much more important to find the right process, designate and empower the team that will run value stream optimization. 

Data democratization and data literacy. Most of the processes are rather complicated and span across a number of tools. It is always a challenge to get underlying data since a company needs to have people such as data engineers and/or data analytics to get the right number. If a company lacks data literacy and expertise it might take time. To circumvent such a situation and get results faster, we recommend relying on qualitative data and confirming it through simple data export capabilities from tools.