Evaluating Complexity
There are several things to consider when evaluating complexity.
When evaluating complexity you should:
- Consider all aspects of the system (including interdependencies and relationships) – we need to consider the values or relationships between nodes, not just study the nodes themselves. In fact, studying the relationships between parts of a systems is more important than studying the parts in and of themselves.
- Search for effective principles – we need to acknowledge the underlying principles and values of what is going on, to understand what is driving change. Michael Quinn Patton, a world leader in the field of evaluation, published a book on this called Principles-Focused Evaluation: The Guide (2017).
- Watch for emerging patterns – patterns will tell us a lot about what is going on (this is not the same as correlation, but the patterns may help us to understand what is going on)
- Challenge the existing mental models about the strategies to effect change – is more valuable than solely focusing on the end product/ result. We need to move away from a linear model of thinking about change, to one that is dynamic and changing/ adaptive
- Iterative experiments (learning at all levels through feedback loops) – perhaps most importantly, we need to incorporate learning throughout our work – collecting data, redesigning/ scrapping or scaling a prototype and testing it out again, while tracking our assumptions and uncertainties along the way).
Designing
Prototyping
1) Brainstorm metrics
Make sure you have your: Theory for change, Stakeholder maps, and Systems map (if you have one) in front of you. Individually, list as many metrics as you can about your challenge (including impact indicators, outcome indicators and output indicators). Write one per sticky note and share them with your team.
Now, organize them on a flipchart or whiteboard. Identify whether they are impact, outcome or output indicators. Draw arrows between them to identify the flow:
- Which impact metrics are influenced by which outcomes?
- Which outcomes are influenced by which outputs?
2) Identify metrics fro the exercise
As a team, identify one “branch” of metrics (from outputs to outcomes to impact). Use that to focus on for your worksheets.
3) Identify key measures for impact
As a team, write down your impact indicator at the top of the worksheet. Under Key Measures, write down one impact indicator.
Tip: Make sure the indicator is specific enough to both identify what is being measured, and how data is collected.
4) Identify standard of evidence for your impact metric
Refer to the Standards of Evidence table. From levels 1-5, what is the level of evidence that you believe will have a positive impact (tip: you likely won’t have a level higher than 1 or maybe 2).
5) Identify drivers for your impact metric
Refer back to your theory for change (hypothesis).
- What do you believe is within your control that will change these measures?
- What do you believe is out of your control that will change these measures?
6) Identify communication
Work with your team to identify:
- Who you need to communicate these measures to?
- Why do you need to communicate with them?
For example, this can include communication for shared learning, or communication for co-design, or learning about your complex problem itself.
7) Identify learning
Identify how these measures will help you to generate, improve or scale your solutions.
8) Repeat for outcome and output measures
Repeat this exercise from Step 3 onwards for: Outcome Measures Template and Output Measures Template.
Identifying Impact Measures.pdf
50KB
PDF
Template - Identifying Impact Measures
Identifying Outcome Measures.pdf
49KB
PDF
Template - Identifying Outcome Measures
Identifying Output Measures.pdf
49KB
PDF
Template - Identifying Output Measures
1) Identifying assumptions/uncertainties
Individually, list as many assumptions and uncertainties as you can about the measure you selected. Share them with your team, and list them in your worksheet.
2) Identify learning goals
Before you go ahead with your evaluation, what are some important things that you feel you need to probe, gain more knowledge about or validate about your evaluation? That is, what are your learning goals?
3) Identify your research or prototype plan
If these are your learning goals, propose either a research approach or a prototype that can help to reveal data about your learning goals (eg. testing your assumptions). What will help you to satisfy your learning goal?
Write out your ideas (either a research idea or a prototype idea) on the worksheet, for each of your learning goals you identified.
4) Reflection
Reflect as a group, what did you learn about the importance of identifying metrics for the change you want to measure, and how it will influence your process.
Research and Prototype Plan Part 2.pdf
46KB
PDF
Template - Research and Prototype Plan
Last modified 4yr ago