Evaluating Complexity

There are several things to consider when evaluating complexity.

When evaluating complexity you should:

• Consider all aspects of the system (including interdependencies and relationships) – we need to consider the values or relationships between nodes, not just study the nodes themselves. In fact, studying the relationships between parts of a systems is more important than studying the parts in and of themselves.

• Search for effective principles – we need to acknowledge the underlying principles and values of what is going on, to understand what is driving change. Michael Quinn Patton, a world leader in the field of evaluation, published a book on this called Principles-Focused Evaluation: The Guide (2017).

• Watch for emerging patterns – patterns will tell us a lot about what is going on (this is not the same as correlation, but the patterns may help us to understand what is going on)

• Challenge the existing mental models about the strategies to effect change – is more valuable than solely focusing on the end product/ result. We need to move away from a linear model of thinking about change, to one that is dynamic and changing/ adaptive

• Iterative experiments (learning at all levels through feedback loops) – perhaps most importantly, we need to incorporate learning throughout our work – collecting data, redesigning/ scrapping or scaling a prototype and testing it out again, while tracking our assumptions and uncertainties along the way).

Designing

Instructions: Designing an Evaluation Strategy for Complex Problems

1) Brainstorm metrics Make sure you have your: Theory for change, Stakeholder maps, and Systems map (if you have one) in front of you. Individually, list as many metrics as you can about your challenge (including impact indicators, outcome indicators and output indicators). Write one per sticky note and share them with your team.

Now, organize them on a flipchart or whiteboard. Identify whether they are impact, outcome or output indicators. Draw arrows between them to identify the flow:

• Which impact metrics are influenced by which outcomes?

• Which outcomes are influenced by which outputs?

2) Identify metrics fro the exercise As a team, identify one “branch” of metrics (from outputs to outcomes to impact). Use that to focus on for your worksheets.

3) Identify key measures for impact As a team, write down your impact indicator at the top of the worksheet. Under Key Measures, write down one impact indicator.

Tip: Make sure the indicator is specific enough to both identify what is being measured, and how data is collected.

4) Identify standard of evidence for your impact metric Refer to the Standards of Evidence table. From levels 1-5, what is the level of evidence that you believe will have a positive impact (tip: you likely won’t have a level higher than 1 or maybe 2).

5) Identify drivers for your impact metric Refer back to your theory for change (hypothesis).

• What do you believe is within your control that will change these measures?

• What do you believe is out of your control that will change these measures?

6) Identify communication Work with your team to identify:

• Who you need to communicate these measures to?

• Why do you need to communicate with them?

For example, this can include communication for shared learning, or communication for co-design, or learning about your complex problem itself.

7) Identify learning Identify how these measures will help you to generate, improve or scale your solutions.

8) Repeat for outcome and output measures Repeat this exercise from Step 3 onwards for: Outcome Measures Template and Output Measures Template.

50KB

Identifying Impact Measures.pdf

pdf

49KB

Identifying Outcome Measures.pdf

pdf

49KB

Identifying Output Measures.pdf

pdf

Prototyping

Instructions: Prototyping your Evaluation Strategy

1) Identifying assumptions/uncertainties Individually, list as many assumptions and uncertainties as you can about the measure you selected. Share them with your team, and list them in your worksheet.

2) Identify learning goals Before you go ahead with your evaluation, what are some important things that you feel you need to probe, gain more knowledge about or validate about your evaluation? That is, what are your learning goals?

3) Identify your research or prototype plan If these are your learning goals, propose either a research approach or a prototype that can help to reveal data about your learning goals (eg. testing your assumptions). What will help you to satisfy your learning goal?

Write out your ideas (either a research idea or a prototype idea) on the worksheet, for each of your learning goals you identified.

4) Reflection Reflect as a group, what did you learn about the importance of identifying metrics for the change you want to measure, and how it will influence your process.

46KB

Research and Prototype Plan Part 2.pdf

pdf