written by Salimah Samji
Many development projects fail because of poor design. They have no clear roadmap of how they will get from A to B and therefore no way of knowing whether they are on the right track. However, design alone is not enough for success. In fact, many development projects that are well designed, fail because of last mile implementation problems. These include (but are not limited to) existing capacity constraints, difficulties with finding skilled staff, lack of funds, politics and misaligned incentives.
Spending time, effort and resources to rigorously evaluate projects like these can be a wasteful exercise. It would make more sense to begin by testing the logic of your intervention by using low-cost tools, like process audits or collecting and analyzing meaningful data on outputs and intermediate outcomes, that feed back into the design and implementation thus increasing the possibility of success. Does the project make sense in your context?
Several years ago when I worked for Google.org, I had suggested that a grantee of ours, one who conducted large-scale assessments, consider a process audit to identify gaps and to improve implementation. They were very open to the idea. I wanted to hire an external consultant to ensure impartiality, but also ensure that the consultant was seen as a team player who was hired to help them. The grantee was thrilled with the findings and told me that this simple process audit had been much more useful and cost-effective than the other Randomized Control Trials (RCTs) they had been part of. They invited the consultant to share the findings with their entire team and together brainstormed ways to improve implementation. It marked the beginning of a long collaboration between the two – but that is another story …
Recently, I was pleasantly surprised to read a paper by Diana Epstein and Jacob Alex Klerman entitled, When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model. They argue that a program that cannot achieve the intermediate goals specified by its own logic model, will not have the desired impacts and should therefore not proceed to Rigorous Impact Evaluation (RIE). And the determination of whether a project can achieve its own intermediate goals can be done using conventional process evaluation methods, that is, careful observation of program operation, at low-cost.
They highlight five common forms of failure of programs to satisfy their own logic models. These include:
- Failure to secure required inputs: ability to establish inter-organization partnerships and to recruit and retain certain types of staff.
- Low program enrollment: it does not attract the target number of clients/participants because there is no demand.
- Low program completion rates: initially enroll in the program but do not complete the expected treatment.
- Low fidelity: the program as implemented falls short of what was envisioned in the logic model.
- Lack of pre/post improvement: sometimes clients show minimal or no progress on pre/post measures of the intermediate outcome which could be due to history and maturation.
Then, for each form of failure, they discuss how it could have been detected by a process evaluation – through the observation of the operation of a program, an experienced outsider can suggest direct and immediate ways to improve the program’s operation.
They also note, “it seems plausible that lessons learned in the first year or two of operation might lead to program refinements, improved program implementation, and improved client outcomes.” This fits nicely with our paper, It’s All About MeE: Using Structured Experiential Learning (‘e’) to Crawl the Design Space which helps develop a learning strategy that achieves more than monitoring, is cheaper than impact evaluations, has timely dynamic feedback loops built into the project and extends the insights gained into the design and management of development projects.