BUILDING BRIDGES – Struggling to scale a new product past its MVP phase, Theodo tried a number of things before realizing that the only way to retain quality and speed is kaizen.

Words: Fabrice Bernhard, Co-Founder and Chief Technology Officer, Theodo

Admiral is a car insurance specialist based in Cardiff, Wales. Anticipating future disruptions, in 2016 the company started to invest in new digital products to hopefully reach new markets while growing their digital capability. One of their new products is Veygo, a digital car sharing insurance, launched by Jean-Baptiste Limare with the help of Theodo.

The first step for Admiral was to learn to build its new digital product fast. In the digital world, rather than go big, one should start small and release as early as possible to learn quickly about the market. Indeed, having direct access to your customers enables you to learn in real-time how they react to your products.

In the lean startup world, the first release of a digital product is referred to as a minimum viable product (MVP), which Werner Vogels, CTO of Amazon, defines as a “heavily-simplified but rock-solid version of your product that should be built and released in less than two months”.

Theodo’s use of agile methodology and Admiral’s idea that “perfect is the enemy of good” proved to be an excellent match for the MVP phase. Together, we released Veygo in a mere seven weeks!

Not only was the MVP a great success in terms of delivery; it quickly became a business success, too. As we learned and iterated, the customer base for Veygo grew and, because the MVP proved viable, the company made an investment to continue to develop the product for another 10 months.

As the product scaled, however, the code and infrastructure became exponentially more complex… and bugs started to appear.

WHAT TO DO?

The first reaction to the scale-up problem was to bring in a full-time tester. He helped the team to spot some obvious issues, but unfortunately it turned out that most of the bugs resulted from rare situations, that no one – not even the dedicated tester – could foresee 100% of the time before the product was released to thousands of users. (For example, customers who had just passed their driving test but had not updated their Veygo profile accordingly.)

The bug problem wasn’t solved and, even worse, as the product continued to grow, it became clear that having a human tester was actually slowing the team down.

At that point, people came up with what they thought would be a solution: writing automated functional tests. (Automated tests function as sensors that are run automatically at frequent intervals to check that functions or features continue working as expected, even after some adjacent functions or features have been modified.) The team had been writing automated tests at the “unit” level since the beginning, but these unit tests did not address the issues arising from the integration of different features.

Functional tests automated the journey of a typical user to identify unexpected problems before the team pushed the changes to production. The same the human tester did, but in a more scalable way.

To ensure that errors got detected as early as possible, automated tests were run every time a developer pushed new code – a step known as the “build”. Adding more and more automated tests, however, considerably increased the build time, which went up to 20 minutes. What’s worse is that the builds were failing half of the time for reasons unrelated to the code produced. In a team of 12 developers, this meant hours wasted every day, and of course a slower lead-time.

At one point it looked as if losing velocity was simply inevitable when trying to scale a digital product.

KAIZEN TO THE RESCUE

As we were working with Admiral, a team from Theodo went to Japan for a study tour. While there, we visited a Toyota factory that seemed to be running at full speed while staying extremely flexible and maintaining incredible levels of quality. This could be no coincidence! As we looked around trying to understand what allowed the plant to run as efficiently and effectively as it did, we realized that the common denominator appeared to be teams doing kaizen. It was used to address anything from the smallest problem to the most strategic of business challenges.

This was clearly Toyota’s scalable solution to maintain quality and flexibility while growing to become the biggest car manufacturer in the world. Could it help Theodo and Admiral, too?

As a company, we were already aware of kaizen. The concept resonated with the stories we had heard at Velocity, a leading conference for web companies. It appeared that GAFAs (Google, Apple, Facebook and Amazon), too, rely on kaizen and lean to scale – even though they might not mention them explicitly. (An exception being Jeff Bezos, who talked about “kaizen” in his 2008 and 2014 letters to Amazon shareholders.)

The engineering teams from Google, AirBnb and Netflix, for instance, talk about devops and its “CALMS” framework as a necessity to scale fast. CALMS stands for “Culture, Automation, Lean, Measurement and Sharing”, all ingredients reminiscent of the kaizen efforts we had seen at Toyota.

So, like Toyota and Amazon, we decided to use kaizen on our products to address our scaling problems.

To do this, we developed the following framework:

• Set a clear target to address the problem – “Halve the build time by October 31, 2017”.

• Structure the work by creating a team and assigning clear roles to a team leader, a coach and a sponsor.

• Implement a white board displaying: a clear indicator that tracks the evolution of the situation, to be updated every day/week; the last identified problems; ideas for experiments; expected results and checks; and resulting improvements based on current standards.

• Ensure time is spent by the team on kaizen.

At Admiral, kaizen encouraged people to try a few things out. The team started tracking the percentage of failed builds, as well as the total build time, spending a few hours every week experimenting with ideas that could improve the situation.

Some experiments were rather complex, like refactoring the testing code to identify which tests were potentially impacted by the last modifications and only run those. Others were much simpler, but still brought us huge gains: for instance, we modified the testing platform to stop at the first failed test (just like Sakichi Toyoda’s 1896 automated loom).

The results were impressive: the build failure rate was reduced by almost 70% and the build time halved over the following seven months. In the meantime, the development team and the product continued to grow fast.

THE SECRET OF A SUCCESSFUL SCALE-UP

Kaizen is now the approach we use on all scaling projects, to maintain velocity and quality past the MVP phase. We observed that the teams who engage in good kaizen have had impressive results, finding new smart ideas to tackle bugs, improve performance and boost security.

It’s also proved as a great tool to teach leadership skills: the time dedicated to continuous improvement reminds everyone that they can react to problems. That everybody has the power (and shares the responsibility) to tackle issues as they appear. Once they have identified a smart solution, we coach people to ensure their knowledge is shared with the rest of the organization. Learning that you can and should react when you see a problem and then learning to empower others to bring lasting change in the company is a great opportunity to develop leadership skills Not only does it help the product to scale, it also contributes to improving the organization as a whole.

_{THE AUTHOR}