Evaluating Complexity
There are several things to consider when evaluating complexity.
Last updated
There are several things to consider when evaluating complexity.
Last updated
When evaluating complexity you should:
Consider all aspects of the system (including interdependencies and relationships) – we need to consider the values or relationships between nodes, not just study the nodes themselves. In fact, studying the relationships between parts of a systems is more important than studying the parts in and of themselves.
Search for effective principles – we need to acknowledge the underlying principles and values of what is going on, to understand what is driving change. Michael Quinn Patton, a world leader in the field of evaluation, published a book on this called Principles-Focused Evaluation: The Guide (2017).
Watch for emerging patterns – patterns will tell us a lot about what is going on (this is not the same as correlation, but the patterns may help us to understand what is going on)
Challenge the existing mental models about the strategies to effect change – is more valuable than solely focusing on the end product/ result. We need to move away from a linear model of thinking about change, to one that is dynamic and changing/ adaptive
Iterative experiments (learning at all levels through feedback loops) – perhaps most importantly, we need to incorporate learning throughout our work – collecting data, redesigning/ scrapping or scaling a prototype and testing it out again, while tracking our assumptions and uncertainties along the way).
1) Brainstorm metrics Make sure you have your: Theory for change, Stakeholder maps, and Systems map (if you have one) in front of you. Individually, list as many metrics as you can about your challenge (including impact indicators, outcome indicators and output indicators). Write one per sticky note and share them with your team.
Now, organize them on a flipchart or whiteboard. Identify whether they are impact, outcome or output indicators. Draw arrows between them to identify the flow:
Which impact metrics are influenced by which outcomes?
Which outcomes are influenced by which outputs?
2) Identify metrics fro the exercise As a team, identify one “branch” of metrics (from outputs to outcomes to impact). Use that to focus on for your worksheets.
3) Identify key measures for impact As a team, write down your impact indicator at the top of the worksheet. Under Key Measures, write down one impact indicator.
Tip: Make sure the indicator is specific enough to both identify what is being measured, and how data is collected.
4) Identify standard of evidence for your impact metric Refer to the Standards of Evidence table. From levels 1-5, what is the level of evidence that you believe will have a positive impact (tip: you likely won’t have a level higher than 1 or maybe 2).
5) Identify drivers for your impact metric Refer back to your theory for change (hypothesis).
What do you believe is within your control that will change these measures?
What do you believe is out of your control that will change these measures?
6) Identify communication Work with your team to identify:
Who you need to communicate these measures to?
Why do you need to communicate with them?
For example, this can include communication for shared learning, or communication for co-design, or learning about your complex problem itself.
7) Identify learning Identify how these measures will help you to generate, improve or scale your solutions.
8) Repeat for outcome and output measures Repeat this exercise from Step 3 onwards for: Outcome Measures Template and Output Measures Template.
