Getting Rid of Wastes and Impediments in Software Development Using Data Science

Impediments are usually common in software development teams. But impediments can act as a disguise for wastes, and they are hard to identify and tackle. I’ve found that data science can be an interesting tool to help teams fight against impediments and wastes.

During my master's degree, I carried out research on how to discover wastes and impediments in a value flow using the lean waste concept and data science. This article presents the way I use data science as a way to detect wastes and impediments, and some concepts and related information that could help teams to figure out the root cause of impediments they often struggle to get rid of. The knowledge discovered during the research included an expanded waste classification, and the use of trends to uncover undesired situations like hidden delayed backlog items and defects trends.

Understanding waste within software development industry

During my career, I've observed things like teams trying to gather all requirements for a few days and having many requirements cancelled during the development of the product. These teams were worried about gathering as many requirements as possible without considering what value meant to the customer. After a couple of weeks, the customer usually cancelled some requirements. Considering the main concept of waste, which considers activities that don't value the customer, the set of cancelled requirements became waste. The questions were: what was the main reason behind the existence of cancelled requirements? Did those teams know how to define value within their contexts?

In this case, the teams dealt with cancelled requirements as something normal (we're agile, and requirement changes are normal). On the other hand, I witnessed teams understanding what value meant and delivering value continuously to the customer. The discovery process was also continuous. So, I understood that waste was not something that doesn't bring value to the customer, but also affects the value flow and could even terminate the relationship between companies.

The concept of waste was coined by Taiichi Ohno when he was working on the Toyota Production System. It is regarding activities that don't bring value to the customer. During the research, I discovered some criticism of this concept when applied to the software development industry, as it came from the manufacturing industry, and this industry has its own knowledge domain which is different from that of the software development domain. 

I found a concept of waste that better fits our industry: waste as something that creates some obstacle towards value creation. This concept came from Power and Conboy (2014), and was defined in their research on waste in Impediments to Flow: Rethinking the Lean Concept of ‘Waste’ in Modern Software Development. Their research took place within the software development industry.

Impediments and wastes have the following relationship: a waste could be an impediment, or could result in an impediment. An outdated manual, for instance, is not an impediment per se, but it could lead to a rework caused by some misdirection. On the other hand, waiting for some management approval is an impediment and waste at the same time. 

During the bibliographic review, I've carried out an analysis on 40 scientific papers, which had investigated wastes, impediments, obstacles and challenges faced by companies and software developers. That analysis allowed the identification of a set of wastes that happened within each company (or set of companies), and resulted in the creation of a map that presented how likely two wastes can appear simultaneously.

The result was a waste model that contained 22 wastes grouped into six categories and a map that demonstrated the relationship among wastes. The categories were: product, process, information and communication, product, process, and organization. The subsequent deliverables were a method and software that helped with the detection of wastes. Based on the information provided by the software, it is possible to understand what is going on, which wastes were found, and investigate the root cause of wastes.

Removing impediments

The next phase was the creation of a method that included an algorithm and software that implemented the algorithm. The software reads issues and detects rising trends regarding defects, stuck issues (delay waste), deliveries, and summarizes the whole information regarding the flow situation in an indicator which is presented through the gauge below. This indicator answered the following question: are we going towards a good or bad situation? The set of trends granted the further investigation of wastes and impediments based on the detected wastes and the waste relationship map created before.

This investigation starts by understanding what waste trend is rising (delay, defects, or both). Based on that, it is possible to look for related wastes and the cause and effect relationship among the found wastes to define what the root cause is.

I reached out to some companies to run the experiments, but only one agreed to do so. The way I found to assess whether the entire method was working or not was by calculating the variability of deliveries and the ratio between defects and other deliveries in a period of four months. This ratio assessed whether defects were increasing or decreasing. The time period was chosen because it was intended to assess the behavior of defects before and after the start of the experiment. The experiments carried out in the company that joined the research led to a reduction in both metrics. I gave some demonstrations to other companies, but there was no interest on their part in continuing with the experiment.

Each experiment was done using information gathered from the company's Jira. The software I created performed the analysis on the information gathered and it was possible to run an investigation about which wastes were found and what the root cause of wastes was. The person who I was in touch with in the company did the necessary job to eliminate the root cause, and after the elimination of the root cause I ran a new analysis of the flow to verify whether the solution worked. 

Some root causes of detected wastes included: a vague definition of ready, and a lack of sufficient expertise about tests among developers. The software detected a rising trend in delay waste. Based on the correlation map among wastes, it was possible to detect that tests were taking longer than expected, and the root cause of that was the lack of sufficient expertise in testing among developers. Upon reaching this conclusion, the action taken was a mentoring session about automated tests with all developers in the team.

Look for trends

I realized that trend analysis is a tool that impacts people without judgement. Trend analysis, for instance, can tell the team that there is a rising trend of some waste rather than some issue being stuck for 149 days, for instance. When we make this kind of information explicit, it seems like a pointless embarrassment. The slope of the charts usually demonstrates how the deliveries and wastes will behave in the next 10 days. Those trends were useful in triggering the curiosity to further investigate what was going on in a team.

I stayed in touch with other companies in the hopes they would join my research, but none agreed to do so. I gave a demonstration of the method and the software to them. The first company's CTO was terrified after I showed him the trend of defects, but the delay trend was worse. When I pointed out the delay trend, he became interested in further investigation of that trend. During the investigation, we found out the root cause of the delay; it was a regular change of priorities made by C-level. Unfortunately, I don’t know whether the root cause was dealt with by the company.

The other company that I stayed in touch with presented a similar situation. I received some data from their Jira, ran the software and had a meeting to talk about the results. The data came with the following label: this is the worst team's data. The data had this label as the Agile coaches' perception was that that was the worst team in terms of performance. The software detected a problem related to the delay trend. The root cause I discovered was that the C-level had defined a zero-defect goal to the whole company. This resulted in people working overtime and leaving the company. After that, the company hired new employees, and the training took time. Therefore, items in the backlog took too much time to get done, and this consequence brought the perception of "worst team" for one of the teams. When the Agile coaches from the company became aware of this reality, they stopped the demonstration.

Focus on what matters most

During the experiments made with the company, dealing with wastes based on the concept designed for software development helped to understand what went wrong in that environment, along with the wastes model defined during the research phase and the software developed. For example, wastes like outdated information, scattered information, cognitive load, and a lack of shared understanding were useful to understand what went wrong within that company. These wastes go beyond the concept of activities that don't bring value to the customer. 

In addition, dealing with facts rather than opinions on trends led to reducing the deliveries variability along with the experiment. For example, investigating the delay waste led to the awareness of developers' lack of knowledge on the automated test. It was a huge opportunity to increase the developers' expertise and team spirit by promoting mentoring sessions among them. 

Finally, the team's analysis based on trends allowed for the investigation without focusing on raw numbers and judgement. It brought the necessary psychological safety to process improvement. The set of trends brought a big picture about what is going on, and presenting some metrics like throughput helped the understanding of the big picture.