Explore, learn and decide through Adaptive Experimentation
Observing natural behaviour will give us some insights into complex and emergent situations and how we may be able to influence them to achieve our objectives (create organizational change, launch a successful new product, improve the performance of a team, etc), but often we also need to perturb or probe the situation to test our ideas and insights and to generate further insights. This is Adaptive Experimentation.
Imagine that you are required to develop a presentation to report to the Executive Team on the progress of a complex project. You have a range of options about how to position and structure the presentation. You could structure it as a timeline of activities and progress, or could focus only on the project achievements and progress against budget, or on the current and projected benefits for the organisation. You could primarily use bullet points, or images and video, or no visual aids at all. You could include contributions from the members of the project team or from stakeholders in the organisation. The possibilities are numerous. How do you decide between these possibilities? You could just modify a presentation from a previous project or you could design a presentation that fits the context of this project and audience and that clearly emphasizes the key messages you want get across. Clearly, the customized design-based approach will produce the better result, but it takes time and effort. Adopting an Adaptive Experimentation approach will develop an effective presentation with minimum effort.
 |
The first phase of the Adaptive Experimentation ‘funnel’ is to explore the scope of possibilities. In the presentation example introduced above, we saw that there are various possibilities about how to position and structure the presentation. How do we evaluate these possibilities and choose the best one? We can do this by ‘experimenting’. For example, we can jot down a number of rough outlines of the presentation and test how they flow and the extent to which are engaging and convey the general tone and messages that we want to get across. In a sense, each of these rough outlines is a mini experiment. We may review the outlines ourselves, or may get feedback from someone who is detached form the project but knows the Executive Team. We may select one or two of the outlines and start to rough out the support material – charts, images, videos, examples, etc – and where they will fit. We may also meet with one or two people on the Executive Team determine their expectations of the presentation and to test our ideas about how to structure it.
At the conclusion of this ‘explore the possibilities’ phase we will have a selected an overall approach for the presentation and have an outline of the content.During the ‘explore the possibilities’ phase the focus is on rapid, low cost and low risk experiments. In general, these include:
The objective of these initial experiments is to quickly canvas a broad range of design options and to learn more about the design context. The focus is on establishing and testing the broad design concept and structure. The objective is to develop one or more outline designs that are likely to satisfy the design objectives, boundaries and constraints. This also helps test early in the design process whether we need to question and review those objectives, boundaries and constraints. It is much more efficient to learn at this early stage that the stated design objective unnecessarily constrains the design. A key aspect of these early low risk experiments is to make the design ideas as tangible as possible – to give a clear voice to the creative and conceptual thoughts of those involved in the design process. Forcing the early design ideas to become tangible has two key benefits. First, as noted above, it tests the whether the high level design criteria are appropriate and whether those criteria are clearly and commonly understood by all those involved. Second, it creates a common and tangible focus for evaluation and improvement of the design. It is much easier to identify and explain possible improvements if you are working with a representation of the design that is tangible and contains a practical level of detail than if you are working with a broad description. (Although thought experiments are not tangible, they should include sufficient descriptive detail to enable those involved to become immersed in the imagined situation. They must also be structured as a forward looking exploratory ‘experiment’, not as a rationalization or explanation of a desired outcome.) |
 |
The second phase of Adaptive Experimentation is to assess the detailed choices. Here, the experiments use more detailed and realistic representations of the design. The design is increasingly tested in the context in which it will used.
In the presentation example, the ‘assess the detail’ phase will involve getting comments and feedback on the detailed text, images and/or charts, and on the sequence and timing of the presentation. Comments and critiques are made on drafts and versions that are increasingly close to the final version. Practice runs may be held. During the ‘assess the detail’ phase, the experimental options include:
These increasingly realistic experiments need to be structured to evaluate how the design performs in context. They need to reveal any unintended consequences, any unanticipated constraints, any emerging systemic behaviour (especially non-linear behaviour) and any lessons for wider implementation. At this stage it is also important to be open to any surprises – positive or negative. In summary, these experiments need to be sufficiently realistic and broad to assess what works and what does not. They also provide another opportunity to test the foundational design choices (objective, boundaries/ constraints, architecture and metaphor). The results of these more realistic experiments may indicate that some or all of the foundational design choices need to be fine-tuned, or even substantially revised. |
 |
The final form of ‘experiment’ is the real implementation of the design. Although, it is not structured as an experiment, it is, by definition, the most realistic opportunity to observe what works and what does not. This is especially important for the situations that are complex and emergent. In these situations, the introduction of the design is likely to have a ripple effect on the context (resistance to organizational change might increase – or decrease, competitors may bring forward the timing of a new product, reinforcing complementary initiatives may be stimulated in the industry, etc). Consequently, it is important to continue the ‘evolving implementation’ phase of Adaptive Experimentation throughout the life of the design, but especially during the early stages of introduction.
In the presentation example, the ‘evolving implementation’ phase involves responding to signals from the audience during the presentation. The body language of the Executive might indicate a particular interest in one aspect of the presentation so the presenter could slow down and give more emphasis to this aspect. Comments or questions during the presentation could indicate that some members of the Executive Team have had some recent negative feedback about the project. The presenter could change the timing and emphasis of the rest of the presentation to better address the feedback. |
Follow Us!