“We live in a world of uncertainty…projecting data from the past to assess the success of a product or service in the future is increasingly risky,” says Jeanne Liedtka. “Even efficiently testing new ideas with existing data is problematic. Asking people what they think, using focus groups and surveys, often produces skewed results, most worryingly, in the direction of false positives…As a species, we find it difficult to accurately predict even how we ourselves will react.”
Conventional routes are often unhelpful, “improving accuracy by mimicking all aspects of the future… translating a new idea into a fully formed and functional product or service, even just at minimum viable product (MVP) level, is expensive and time consuming.” A low level ‘soft’ market launch might be cheaper and quick but it can suggest a lack of faith. Equally, elaborate, formal experiments, like randomised control trials, can take years, which is hardly ideal in a rapidly changing marketplace.
In their new publication, The Experimentation Field Book, Liedtka and her fellow authors argue, a much more effective way to innovate:
“Early stage experiments that treat an idea as a hypothesis, surface the critical make-or-break assumptions” and “using low fidelity prototypes give us more accurate data but with less cost and risk,” she says, insisting: “To my mind, they are one of the most valuable and teachable, yet enormously underused, business tools available today.”
Some already realise the value of experiments. Organisations, including e-commerce giants such as Amazon, eBay and TikTok and innovative consumer brands such as Nike have embedded experimentation into their culture. A test by a young creative team at supermarket chain Tesco proved that they could generate profits by promoting organic food to shoppers who bought seed for wild birds.
Problem solving and experimenting are part of human DNA, so why don’t we experiment more often? And, just as importantly, why don’t our experiments succeed more often?
“Experiments can be challenging because they require fundamental shifts in mindset and behaviour. Learning in action sounds simple but it’s often hard to do well,” says Liedtka.
“We need new ways of thinking and doing. The very human desire to avoid error, to think of disproved hypotheses as ‘mistakes’, rather than evidence of learning, is hard-wired in most of us. In addition, while many organisations say innovation is a priority, they have a low tolerance for what they perceive as ‘failure’.”
The irony is that we can’t innovate without learning and we can’t learn unless we experiment. The thesis of the The Experimentation Field Book, evolved through conversations with DK&A founder David Kester, Natalie Foley and Elizabeth Chen. It features DK&A’s work with British rail company, South Western Railway as one of its four experimentation case studies.
The right experiment can speak truth unto power and help resolve internal deadlock. At Monash University Medical Centre in Melbourne, Australia, innovators wanted to pilot a scheme where, carefully selected and trained lay people, supported high risk elderly patients through maintaining telephone contact with them. Hospital medics were unconvinced initially but, agreed to co-design the experiment. It went ahead and, 300 patients later, was shown to dramatically lower emergency admissions and hospitalisation rates.
One of many myths surrounding innovation, is that it is something that only geniuses like Steve Jobs can do. Everyone else has to wait around until someone of that calibre turns up. None of the book’s authors buy this myth. David Kester recently debunked the idea in a recent DK&A insight post. In reality, we can all innovate. Experiments provide the right environment for innovation to flourish by eliminating many of the usual barriers such as:
The authors’ extensive research and experience in the field has led them to conclude that successful experiments are built on a five-step process:
Experiments must start with complete clarity about the hypothesis the organisation is trying to explore:
“Are our initial hypotheses actually testable? There’s no point in testing a product without specifying who it is for and what value it is meant to produce. We want to test a value proposition – and a new product is only one part of that.”
Organisations need to think hard about the evidence they require.
Don’t be tempted to use data you already have, acquired through normal business processes, seek the right data to genuinely test your hypothesis.
An experiment at Whiteriver Hospital, on the Fort Apache Indian Reservation in Arizona, offers striking proof of the difference the right data can make. A team of innovators focused on the financial implications of 18% of patients leaving the hospital’s emergency room without being seen. The problem was in part due to a failure to separate patients with basic medical concerns from those who needed emergency treatment. A four-day trial of a redesigned approach to triage reduced that percentage from 18% to 2%.
Looking at the average daily income Whiteriver generated from each patient, and factoring in other matters such as revenue from drug prescriptions and federal grants for more efficient use of beds, the team suggested that a $150,000 redesign of the emergency room would save the hospital $6m. Even factoring in an unusually wide margin of error, the case was compelling enough to persuade administrators.
Knowing what data you need will help you to design an appropriate test. “People usually equate experimentation with the kind of sophisticated A/B testing pioneered in the tech sector,” says Liedtka. “But there are much quicker, easier ways to test nascent ideas. You can walk people through a journey map of the new experience, using a storyboard or create a realistic front-end for the user with an early stage simulation” You don’t necessarily need the back-end functionality that costs so much to build.
Trying too hard too soon can be counterproductive. “There’s no point in wasting money and time on prototypes that are higher fidelity than they need to be,” MVPs don’t need to happen until much later,
The final challenge is to analyse what you have learned, reiterate the idea if the data suggests this and execute it successfully. Liedtka advocates, is organisations should try to “document your experiments in ways that make your results compelling. Don’t forget you have an audience to convince – and that ‘proof’ is in the eye of the beholder.” This was an essential strategy for DK&A working on the SWR experiments. The business case came from the data and evidence. Videography then humanised the story.
How do you know if an experiment was successful? “Success is not how effectively or economically the experiment was run, but whether the organisation actually learned from it – and how valuable the insights proved to be. As obvious as that sounds, many people find it difficult to translate what they find into a refined product or service, or to pivot –to a decision to walk away.”
A simple experiment could be to set up a ‘lemonade stand’. This is the name of a basic experiment “where we set up a booth somewhere our target market gathers (it could be a convention or an annual meeting), and offer them an array of options captured in simple forms like posters and storyboards, let them sign up for the new service they like best and ask them to discuss why they made that selection,” Liedtka says.
Another favorite of Liedtka’s from the book is the ‘Wizard of OZ’ that delivers the illusion of a seemingly real and complete service. Nike used this approach to ascertain whether people would return old trainers, an integral part of a new sustainable concept they were developing. “Purchasers got a postage-paid bag with their new shoes and were asked to return their old pair. And rather than being sent to a new shoe recycling facility, which Nike had designed but not built, the shoes were mailed to a back office, to be tracked and logged.
The experiment relied on a simplified operational front-end but didn’t require the company to invest in an expensive functional backend. This was hardly an MVP but it did successfully test a make-or-break aspect of Nike’s new idea for minimal time and investment.”
Although a certain kind of command-and-control leader can find the whole process of experimentation distinctly troubling, if well designed, they can be a cost-efficient, timely way of discovering whether your revolutionary new idea is the equivalent of the Apple Newton or the Apple iPad.
Being first mover in such an uncertain marketplace only gives you advantage if your first move is to launch a product or a service people actually want to buy.
The Experimentation Field Book is now available to purchase. Published by Colombia University Press. It‘s a hands-on manual for crafting and conducting useful experiments in real-life settings. It guides readers from any background or discipline through the fundamentals of identifying testable ideas, selecting an evidence base, prototyping, and testing, building users’ skill sets and channelling their creativity through an interactive, exercise-oriented format.
The book details a step-by-step framework, with user-friendly instructions and a case study illustrating the process at work at each step, as well as templates for readers to customise in their own projects. It draws on design thinking as well as other practical business approaches.
Find out more and purchase here with our exclusive discount code CUP20 for 20% off.
"*" indicates required fields
