Informing Budget Reform in Mozambique: The Problem Driven Iterative Adaptation Approach

Katya Gonzalez-Willette Hello and welcome to the Building State Capability Program, Harvard University’s podcast series. In today’s BSC podcast, director Salimah Samji interviews Professor Matt Andrews, senior lecturer in public policy at the Harvard Kennedy School and Faculty Director of the BSC Program on the first report of the PDIA and Practice series. This series will cover a few of the research engagements done by the BSC program in the past eight years and details what results emerged, what we learned and what were the next steps for each of these engagements. The first report covers the team’s experience working with officials in Mozambique’s public financial management sector between September and December 2009.

Salimah Samji The Building State Capability Program at CID builds the capability of public organizations to implement policies and programs, and they use the approach that’s developed by our faculty called Problem Driven Iterative Adaptation or PDIA. We’ve just launched a new series called The PDIA in Practice series, which are short a series of short notes, which kind of tell the story of where PDIA came from, where these ideas were generated, and how we learned about them. Today with us, we have Matt Andrews, who is really the brain behind the PDIA in practice and has written the first of the several series that we are launching. Welcome Matt. We’re really excited to have you on the podcast. I want to start with saying, you know, the Building State Capability Program is now six years old. And at this point, why is it that we are now writing about where PDIA came from? How did we learn about this? Where did these ideas come from? How have we essentially updated PDIA through the past years?

Matt Andrews Well, thank you. Thanks for asking a really good question. I think the basic argument or the basic rationale behind doing this was that, you know, ten years ago now, people like myself, Lant Pritchett and Michael Woolcock looked at the world and said, some things work and something starts, and why don’t the things work when other things do? Why are we stuck, right? And we came up with what we thought were very good intellectual arguments, supported with data about what was going wrong in the world. And then we came up with some ideas about how you might counter the things that are going wrong to do things better. And at that point in time, I think we were all sitting with these ideas and saying, you know, here is this this this approach that people can use in management called PDIA, which at the time was really a set of principles, right? Instead of jumping to solutions, start with your problems. Instead of planning everything out in one go at the beginning, before you do anything in tough contexts, iterate and see what the context is like and make changes as you move along. Instead of just working with narrow group of experts, engage lots of people so that you can build the ownership and so that you can get their ideas, etc., etc. These were great ideas, but you know, when we got back they were really just ideas. And I think our perspective at the time was we can’t just leave these as ideas. We need to try real things in the real world to see how they work. We also at the time thought, you know, this is not a standard academic exercise. We can’t do randomized controlled trials on this because one of the things we were observing was that these are real things and that things go wrong in Nonrandomized non-controlled settings where real people exist. So we started trying out the ideas, and when we think back to when we started trying them out, it was 2009 and we’ve been on a journey since 2009, trying the ideas, learning what works, learning what doesn’t work. Iterating ourselves and coming to a point where we feel that the model has stabilized and we’re PDIA now is kind of like a thing. It’s a process that we have done now in a fairly stable way for probably the last three years. So it made sense now to say, Well, where did it come from? We want to explain to people as transparently as possible, what we’ve done, what we’ve learned, why we do things certain ways and not other ways. And given that we kind of feel that we are at the time and like a stable place, I’m sure we will change in the future. It seemed like the right time to tell the story.

Salimah Samji Great. Thank you. The first PDIA in practice series, which we’re going to call PIPS because everything in development needs an acronym. So in PIPS 1, you tell the story about Mozambique in September to December 2009, and I was wondering if you could tell us about the workshops that you held and just. Walk us through. Put us in the room. What was it like to actually do that and have that experience?

Matt Andrews Yes. You know, we date this back to Mozambique 2009 as the first time where we started saying, let’s try something that’s like PDIA. And the working in Mozambique emerged from two directions. One of the directions was we had some Mozambicans in an executive class of mine that worked in line ministries and were senior in line ministries. And in attending a class on some of the original thoughts that we had on capability traps, where we were sharing the idea that governments were doing things for year after year after year that looked like great ideas but weren’t getting you anywhere. One of the ministers came to me afterwards and said, Mozambique is is that place right? And we need your help because we have some things that look really, really good on paper and people are very impressed with, but I don’t think they work very well. At the same time, I had some former colleagues from the World Bank who were working in Mozambique and they were saying, especially in the public financial management arena, which is where I had worked when I was at the bank, that the government had developed kind of best in class reforms, that everyone in Africa was kind of looking at and kind of signing off on, but that some observers, including my colleagues in the bank, were saying have problems with that. Some of the reforms were not working as well as they had been kind of presented as. And so they said to me, come in and have a look at this. And it created a great opportunity to do something that we were thinking about in PDIA at the time, which is let’s get people to think about the problems with the reforms, not just the solution itself. Most works that I’d done beforehand, when people came and said, Come and look at the reform, what I was really being asked to do was to give them a better version of what they were already doing. But so, you know, you’re doing this, it’s fantastic, but the Germans are doing something better, so do the German thing. And and this gave me an opportunity to come in and say, look, we’re thinking about this new thing called PDIA. And in PDIA, a our approaches. Why don’t you focus on your problems rather than your solutions? Because if you solve your problems, you might actually find there’s a greater level of domesticated creativity in that process and a greater degree of kind of emergence and empowerment that comes through that. And you might find yourself in a better place. So that’s kind of where the story came from.

Salimah Samji Great. And so you went and you did a workshop. Can you tell us how that worked out?

Matt Andrews The government was at an inflection point where they had been doing reforms for about 12, 13 years. As I said, the reforms were actually considered really successful. One of the interesting things in development is that it’s very, very unusual to have points in the reform process where people sit back and say, okay, we’re doing a lot of things well, but all we really getting everything we need out of it. That’s an interesting observation because if you’re in a private firm, most private firms, when they do these kinds of interventions, whether it’s introducing a new IT system or a new human resource management system, they build in these kind of points whereby they are reflecting on does it do what we want it to do? Yeah. And one of the reasons why there’s a literature on failure of change processes in the private sector is because in the private sector they ask those questions really quickly because they really are looking for the functional improvements after the change. You don’t really have those in in the development community that much. This created an interesting opportunity because the government had been doing things for about 12, 13 years. Everyone had been saying, well done. But now they had come through their initial 12, 13 year plan and they wanted to do a new vision. So the World Bank said, come in and advise on the vision document that has already been written up. Tell them what you think about this. And I said to them, you know, I don’t want to do that, because when I look at the vision document, the vision document essentially says, let’s continue with where we’ve been. And what I would rather do is try and see if we could broker a conversation about where are the gaps in what we’ve achieved. Is it really getting us where we want to? Where are the kind of weaknesses? And the World Bank has said, that sounds fantastic. So the first thing to say here is, you know, people say, well, you can’t do it in development enough. Actually, you can. And actually, the World Bank has said that sound makes a lot of sense. And the first workshop was a workshop with the authorizers, as we now call people in in the PDIA work, which were the kind of the leaders, writers, the Minister of Finance, the head of the Treasury, the head of the Budget Department, the head of the entities that were running the reforms at the time. And the goal of the workshop was for us as experts to do an analysis of what was working and what wasn’t working this system after 12 years of reform. And so we took the data that was exist that that existed in the development community at the time. It was called the PEFA data. And you still have it now. It’s actually much more now where PEFA stands for public expenditure, financial accountability. And it’s an analysis that donors do of the government system and that in using kind of about 60 to 70 indicators, it ranks the system according to what they call good international standards. And using that, we could look across their public financial management system from budgeting to external audit to even accountability in the parliament. And we could say, Where are you doing better than in we’re reading worse. And it was a way in which we could almost kind of help them get a window that they could look into this system and say, okay, here are the areas that we kind of like have achieved and here are the areas we should still be concerned about. And maybe at that time, then they could adapt the way they were thinking about their reforms to say, is there something systemic about kind of how we are achieving and how are we’re not achieving? And so the workshop was focused on that. It was an external analysis that wasn’t what should you do next, but rather how can we use data to give you a window on kind of where things aren’t working?

Salimah Samji And how did that go?

Matt Andrews It went well, actually, you know, and and it was it was fairly easy to do as outsiders because we had the data, we had the attention of the government counterparts and the serious attention of them. And, you know, the one thing that donor agencies can do in development is get the attention of government counterparts.

Salimah Samji Definitely, they can do that.

Matt Andrews And I think actually it was a a refreshing session for the people in the government because they were being spoken to about their reforms and they were being presented information that I think made sense to them, but no one had ever kind of presented to them before. You know, there were a lot of people in Mozambique at the time who were saying, gee, the reforms are good, but and the big but was there gaps in compliance? Not everyone uses the new systems like they should. Not everyone complies with the laws like they should. And the second big part was that the compliance issues are worse when you move away from the capital. So the further you move away from the poor to and you move to the districts and you move to the the local governments and you move up north into the northern provinces and the more you move into the line ministries. So you move into, let’s say, a northern province, a district government, a school in the education sector. The more the reforms were breaking down, people were talking about this, but there was no data. And in our analysis, we got to show them evidence that those gaps were real, evidence that the reforms seemed to be much stronger when you were dealing with the Ministry of Finance, only Ministry of finance, and particularly when you were dealing with things that the Ministry of Finance could control. What they couldn’t do is ensure that people in those other places were complying with the law. What they couldn’t do is ensure that what was going into the system was not garbage. And the data basically took things that people spoke about, they talked about it, put it into numbers and said the reality is you’ve done well in many, many things, but here are the gaps. And the gaps actually could compromise the functionality of your system. And so we found actually that a lot of people in the meeting were they wanted to know more. And the most positive thing that came out of the meeting was that the Minister of Finance and most of the people in the Treasury, etc., said, we do want to know more, but our idea is that we want to now put your expert outside ideas in front of our people and we want our people to give their views on what they think. So let’s hold another workshop where you present your data. You get their views and you compare them. It was very interesting. We didn’t think about that. But it’s almost like if you think about a therapist and a therapist, a therapist coming to a couple with her own professional opinion, and that professional opinion must spark the couple’s interest. But what the therapist really tries to do is get the couples to give their views on their own situation. So it’s almost like they were asking us to do that. Now, just there was a second response in the meeting, and the second response was less accommodating of the expert views. The people who are running the reforms. There were two or three entities that essentially were what we would maybe call like a PR project implementation unit, and they actually weren’t that. But that’s kind of what they were. They were more defensive. They said, We see your data, but your data is from outside. It’s not from us. And we don’t think that those gaps are there or they’re not as bad as you’re saying. So they said what we are going to do is we’re not going to be part of this workshop. We’re instead going to get the audit unit to do an internal audit of our system to prove that you’re wrong. Great. So it was two very interesting responses. The one was saying, We think that you’re on to something, but we want our people to kind of give their views. The other one is we don’t think you’re onto something, but we do realize that we need to prove ourselves, so therefore we’re going to do something. And at the time, we really interesting the one response and meant that there was a second stage that included us, which was we’re going to have a workshop with all these other groups. The other one didn’t include us, which was we’re going to go away and we’re going to do an analysis and then we’re going to tell you how wrong you are.

Salimah Samji That’s great. And then so you did the second workshop. How did that go?

Matt Andrews So the second workshop was really fun because in the second workshop we were really trying something we hadn’t done before, which was we are now the external people, but no one wants to see our analysis, right?

Salimah Samji You’re not the expert anymore.

Matt Andrews We’re not the expert anymore. Now, and this was the first time I think that we in PDA became facilities and they said what we want to do is we want you to find out what our people think about this. Right. So it was a really funny meeting. It was it was in a hole in Maputo. And we had a table, three tables, about 30 people. So, you know, we’re not getting the whole government there. You can’t get everyone there. But it’s it was at one table, all the people who had been involved in kind of designing the reforms from the Ministry of Finance. And then at another table were Ministry of finance officials, but who worked in the provinces. So they brought them back. So they may have worked in the Ministry of Finance, but they actually worked in the provinces. And so they were what we would call more D concentrated. And then at the other table, we had people from the line ministries and people from districts. So these were people who were, let’s say, maybe the consumers or the users of the reforms that were generated by people at the first table. And so what we did is we said, look, there’s power dynamics in the room and we don’t want the power dynamics to get in the way. So we don’t want to kind of just have a conversation about the reforms. What we want to do is get people’s views and get them in ways that we can kind of project them back at them and then facilitate a conversation. So we had everybody a fill in a questionnaire of how effective they thought the laws were at different points in the PFM system, how effective they thought implementation of the laws at the central government level were at different levels of the system, how effective they thought implementation was at the provincial level and how effective they thought the implementation was at the provincial level. And we said to them, No one has to put their name on it, but what we’re going to do is we’re going to calculate the average for each table so that we can see what the reform leaders think, what the provincial people think, and then what the kind of the users think. And so they they gave us the number, the data, and it was all it was all written down. We didn’t have to in 2009, which would have been really cool. And then we quickly summarized them and we had data for all the tables and we and we just put the data up on the screen in front of them. And you could see that the the reform table thought that the laws were fantastic. And when you went down to the provincial guys, they thought the laws weren’t quite as good. And when you went down to the people in the districts and the ministries, they actually thought the laws were quite bad. And these laws were like, we’re like the centerpiece of the reforms. And then when you looked at implementation, everyone, everyone believed that implementation got worse the further you got from the center, including the people who were far from the center. But everyone that was not at the reform table was actually more negative even about their own performance. Even about the performance, they were like, we do worse than you guys think. And so we put this data up there and it just generated this fantastic conversation. Again, in every country, people complain they mode, they talk and you can hear these things, but there’s never a systematic communication of these things. So the meeting was fantastic. It even went into it was two days. And at the end of the first day, the Ministry of Finance guys came in and they said, one of the things we’re realizing is that our communication is not good. Is there any exercise you can do that actually measures that for us because we are seeing that this is useful. So the next day we had a a tool that we basically just asked everyone in the room, how many conversations do you have and who do you talk to about reform? And we came up with this thing basically saying all the conversations, all with the Ministry of Finance and the donors. That’s what it came out with. And all of these guys who actually had to use the reforms and had to kind of spend money and comply with systems, they were like, no one ever talks to us. And these guys were saying, look, you guys develop a system and the system is good and we appreciate it for what it can do. But you also need to realize that we live in different political arenas to you. You give us a system that requires certainty about transactions. We don’t have certainty about transactions. And you send us one training thing and then you think we can do it and we can’t and that’s it. The other thing is that you assume that we can use an EITI system. An EITI system requires electricity. We only have electricity a couple of times a month, so we don’t use it. And it was these really practical, real conversations. And then they said, okay, how can we move ahead? And they said, well, look, this new vision we developing, why don’t we identify all of the gap areas we’ve indentified? Why don’t we put together teams that include people who are uses, etc., etc., and we’ll kind of have those teams develop their own strategies for what they can do, and that can be the basis of the PFM vision. So it was a really, really great conference. It was all workshop. So we all left and we were very excited about it. And from my perspective, it was one of the first times where we had not been the ones leading the conversation. We had been facilitating a conversation about problems.

Salimah Samji Yeah, that’s fantastic. So after this great workshop, you know, one would think, Wow, reform happens, the world changes. But they actually ended up choosing their vision statement as is and making no changes and moving ahead. And and that’s kind of a disappointment in terms of, you know, you have all of this change. You bring the stakeholders together for the first time. You have them discuss the real problem that they’re trying to solve as opposed to retrofit solutions. And yet then nothing happens and you do a really nice job in the in the note explaining why you think this happened and what we learned from this actual experience.

Matt Andrews Yeah. Yeah. So, I mean, at the time I was, I was kind of working on the usual development schedule of when you go to countries and when you don’t, which means you tend to visit maybe three times a year and that’s a lot, right? So we, we had the workshop and then we left. And then three months later, you know, you come back and say, okay, so did you create the teams that you and I say, No, we’re going with the PFM vision that was written before you came. Right. So are we going to be doing more EITI, more laws, more formal reforms? It taught us a lot also about, you know, how much of a status quo bias there is in a lot of reforms. It’s not it’s not that there’s that is just in governments is that the way in which reform is done is it’s done by experts. It’s done by people who sit in certain offices. They don’t really want to engage these people. It’s difficult to do. So it’s unclear exactly why they continued. But I think, you know, there was a vision that had been written, donors had contributed to it, they’d paid for it. There were people who had interests in it. And the next steps of kind of moving into this more problem driven version were not clarified enough for them. And so, you know, it was they had time pressure, they had political pressure. They needed a PR firm vision. So they went ahead with that. It wasn’t very exciting at the time. And and, you know, it seemed like things really didn’t work. And I think that, you know, over the years when we’ve done PFM, when we’ve tried to get people to think about using PDIA, sometimes people will have like a problem workshop, They’ll come out of the problem workshop and and then they’ll say, well, nothing happened afterwards. And, you know, just to say we feel that we felt that right. And at the time we didn’t really have anything as a next step. And so it can be it it can be quite upsetting.

Salimah Samji Yeah, definitely. So then what tweaks or changes did you make based on this experience in terms of how PDIA, what do you follow a problem construction with and and how do you think about this?

Matt Andrews What we’ve learned after this is that you need to be able to say to people, how do you take the energy from that kind of workshop into action? People want to know what the action is.

Salimah Samji And the how to, right? I think that’s something that people really struggle with. It’s the yeah, I know we need to do something different, but how do I do that? And yet I have something that’s here, a ready made solution. I don’t I know it’s probably not the best and it probably won’t work, but I know I can do this as opposed to, what else can I do?

Matt Andrews Yeah. And I think I think even more than that, when you start to get people speak about problems, if you really start to get people seriously talking about the things that are going wrong, you cannot leave people hanging.

Salimah Samji That’s a very good point.

Matt Andrews Because there is a courage that that is required to talk about the things that are going wrong. What we do now is a few things, and we don’t really discuss it in this night, but we do. In the second one that’s coming up is we have thought very differently about time. One of the things that we don’t believe in the PDIA, is that you can work in months and years. You can’t come back after three months. We believe that you need to have people think in terms of days and weeks. And so once you have people construct their problem and realize what it is, you need to take people very quickly, very urgently, into a conversation about what are the causes of their problem. Because when you get into the conversation about the causes, you unpack the problem into things that you can deal with. You make something that is big and scary and a negative story into something that is much more manageable and that you can actually make positive. Then you want to take them into a conversation about where can you start and where can you start immediately, and then you want to take them into the start. So it’s not just kind of, okay, you can start. You want to take them into the iterations where they start and in the work where we have now moved into and where I say it’s kind of stabilized, we take people into start the next week and we bring them into kind of week by week iterations where they are actually doing something and then the teams are created, the teams are moving, the teams are building the momentum. They are seeing that they can do something about their problems. If you call and help people get into that stage, I think they will fall back to the previous place and it could even be a place that they can’t get back from.

Salimah Samji Yeah, you have coined this new phrase adaptation window, which is a really nice word. Do you want to explain what what an adaptation window is?

Matt Andrews Yeah. So, you know, when we did this work, the idea is that you trying to create an opportunity for government to reflect on their reforms and what isn’t going right in their reform and what is going right so that you can create an opportunity for them to adapt like a window that they can look through and see some evidence and then reflect on things where the idea of adaptation window comes from is from John Kennedy’s work page on Kingdom Works. He writes about policy windows where he says new policy ideas come onto the agenda when a window is created, where someone identifies a problem, when someone identifies an alternative, and where there is political will and those things come together. And that’s what he speaks about. So it’s you know, we are kind of saying we’re not talking about bringing new policy onto the agenda. We are talking about reflection points in policy or reform processes where you create the window, very similar idea, where people can look into what’s going on and they can they can create some shifts. So that’s kind of what the idea is about. It’s almost like an intervention of sorts that you can provide that is not an intervention where you’re coming up with new ideas. It’s not an intervention where you’re telling people what to do. It’s an intervention where you are helping people in governments reflect on their reform journey and to make necessary adaptations to this. When we think now about our work, we facilitate these kinds of adaptation windows all the time, and they are very easy to do, as I’ve described it. You can just use whatever that is out there. There’s a role for external people to play. We found that the more that data involved in that is internal and even if it’s kind of opinion based, then it’s not kind of from a big survey. The better it is. People have their own beliefs and all you want to do is you want to help frame those beliefs. You want to help let people see how their beliefs and their context relate to other people’s beliefs in their context and to look at it. And then you want to say to them, So now what? So you have this issue. So now what? And you know, as you’ve explained in Mozambique, after the second workshop, we thought that maybe there’d be a really big “so now what?” And it kind of wasn’t immediately but as in Mozambique, as in other places, we found that down the line there were some really interesting surprises because these kinds of interventions allow for reactions that people don’t have if they don’t have the space to look at their reality in any way.

Salimah Samji Great. Thank you very much. You can read the PDIA in practice series one note and you can read it. There’s an accompanying paper which tells the longer story. And stay tuned for PDIA in practice 2, which is again, the rest of the story on Mozambique, which tells a story of two other things. So while they went with their old vision, they did do two other things that led to ridiculous amounts of success. So stay tuned for a podcast number two. Thanks.

Matt Andrews Thank you.

Katya Gonzalez-Willette If you want to learn more about the Building State Capability program, please visit bsc.cid.harvard.edu. Thank you for listening.