Public Policy Failure

Katya Gonzalez-Willette Hello and welcome to Building State Capability at Harvard University’s podcast series. In this book podcast Salimah Samji, Director of Building State Capability, has a conversation with Professor Matt Andrews, Faculty Director of Building State Capability. The two discuss public policy implementation failures.

Salimah Samji Welcome Matt. It’s always a pleasure to be able to do a podcast together with you. Today I wanted to talk about implementation failure. You know, we hear that there are so many amazing policies that are out there, but there’s a large percentage of them that just fail. Why? Why do we have so much failure in public policy implementation?

Matt Andrews You know, it’s kind of an interesting question because on the one hand, if you come to a school like our school, people are going to talk a lot about the successes and we teach case studies about everything that goes right. If you’re out on the street, people are often talking about everything that’s going wrong. So, you know, it’s almost these two narratives. On the one hand here we celebrate government and we celebrate public policy. And out there, people complain about it a lot and you just have to read the newspapers and you can see that second narrative, that tech narrative about things don’t work very well. And I think when you put the two together, you realize we have a lot of very good ideas that we want to be successful policies, but somehow they don’t get there often. And the question is how often? Right. And we’ve been thinking about this very seriously over the last couple of years, as you know, partly inspired by people like Frank Fukuyama and Stanford and others who’ve been identifying that, you know, we don’t necessarily teach well about taking the ideas and executing the ideas. And I think that this is an important gap thinking about failure. So, you know, the first thing that I’ve been asking myself as well, how often do we not succeed? And that’s that’s an interesting question. And when you start to ask the question, you say, well, how often do we succeed? Which side is Russia? You know? And the first thing that to observe is it’s actually very hard to see that because there are very few governments that actually evaluate, do we do well, do we succeed? There are very few agencies or entities across the world that actually look at this question.

Salimah Samji Why do you think that is?

Matt Andrews Well, you know, I think that for one thing, that’s very hard to do. And the first thing to think about public policy is if you want to do public policy, it’s the hardest thing in the world to do that. I have a lot of friends who are entrepreneurs and they work in the private sector and they would be shocked that I’ve said that. But it’s hard, right? You know, I came to this stuff through economics and economics. You learn why do you need government? Well, you need government because of market failure, right? So the very idea that you need government is, well, there’s some things that the private sector can’t do. There’s some things that the private sector fails at doing. It under provides, right? Now you’re saying, well, okay, the government is the right place to do it. But what you’re doing is you’re saying let’s take the things that the market can’t do and it’s given to the government. And then we’re surprised that the government can’t do it either. So I think the first thing is these things that make it hard problems. Oftentimes they involve doing things that we don’t really know how to do, bringing people together in ways that we don’t know how to bring them together, and engaging in political spaces that are fraught with complexity, that are often very adversarial and developing mechanisms that allow us to do things that we don’t know how to do. And that may be impossible. So in one sense, I think that’s one of the reasons why we don’t always assess success, because we are not really sure how to do what we’re doing. And worse than that, we’re not really sure how to evaluate if we are being successful in what we’re doing either.

Salimah Samji All right, so you have this paper that you’ve written on implementation failure, and you’ve kind of looked at the World Bank and some of its projects. Can you share with us some of the findings that you had from looking at at least an organization that’s looking at failure and success and evaluation?

Matt Andrews So, you know, the World Bank is interesting. I used to work in the World Bank and we’re interested in development, but it is one of those organizations to give a lot of credit that actually evaluates every project that they do. And, you know, people would say, well, this isn’t a government. Yes. But it’s a public policy organization that works with governments all over the world. And if you look at the composition of the World Bank, it kind of looks like a government. Theres the people working on health, the people work an education. There’s almost every department you could imagine. The government, the World Bank does that. They support these projects and they fund them like you would through a government budget, and then they evaluate them. And the first thing to say is, you know, some people read the article and think I’m being nasty to the World Bank. Unfortunately, I’m looking at them and I’m asking the question, how often do you fail? It’s not being asked of you at all. I can’t find any other organization that evaluates every project that it does, and that actually puts that information out so we can look at that. So from a researcher’s perspective, it’s just like a wonderful thing. It’s a gold mine. The interesting thing is that when they do the evaluation, the evaluations department actually talks a whole lot of interesting questions, and they have been filled with people who are very serious in thinking about this, what this failure look like. They ask questions about are we satisfied with how the project is going. Is it going on time, is it on budget? Is it addressing the objectives that we had at the beginning? They also ask questions like is it meeting the broader development, impact objective we were thinking of. For instance, you might say, We are doing a project to build schools, right? And the immediate objective is to build 5000 schools in five years. And so they would ask, did we satisfactorily do that within the budget that we have, etc., which are questions which we would call the project management questions or product questions, Did we produce the product right? But the reason why you’re building the schools is so that people can learn. Right. And that’s the development impact. Right. And every government has these kind of things where in one hand we are allocating money and we’re doing it in a period of time and we are producing a specific product and we want to know, do we do those things? But we’re doing that for the reason. And that’s the second question. And the world then asks those questions that at least the evaluators do. They also ask, Is it sustainable? Great question. They ask a bunch of other things. And on the evaluation side, they really have thought about this stuff, you know, sustainable. They ask questions like, you know, is this politically sustainable? Important question. When the World Bank as an organization, not the evaluators, report on these findings, they only report on the satisfaction, but only reports on did we do it on budget, that we do it on time and that we reach the objectives. And on that for every project they assess on different criteria. And this one, they they report on the percentage of projects that are considered moderately satisfactory or better. And on that day, 75% of ticked off. We’re successful with large percent. That’s a large percentage. Yeah, it’s good in any organization that’s good. And basically what they saying is we committed to do this project and what we couldn’t control is how the money was spent. And if it was spent as we promised, that it would be spent as in any good project management organization. And they saying 75% of the time we do that. The second question you won’t find the managing director reporting on every year of do we think that what we produced is going to improve the development impact?

Salimah Samji Is it going to solve the problem? 

Matt Andrews Yeah. Now that the venue editors are evaluating that, they are asking the question, well, turns out 75% moderately satisfactory above only 49% of the time when we think that we are actually going to impact things. They are asking the question, but they are not speaking about it now. It’s a really interesting thing because you could say, well, it’s kind of obvious the 75% sounds better, The 49% doesn’t sound it doesn’t sound that good. I think there’s something else going on. Our organizations have become project management organizations. And if you’re a project management organization, you’re focused on the project and the product success. Did we do it on budget day to do it on time and to reproduce what we said we produce? Because that’s what we were accountable for and that’s what we can plan for and that’s what we can control for. The thing of them saying, is it going to be sustainable? Is it going to impact things? There’s a lot of uncertainty there. There’s a lot of stuff that we as the organization in this case, the bank or the government, whichever entity doesn’t control, we can’t control those things because there’s politics that comes in. Maybe there’s surprises. Maybe there are shocks. Right. And those are things that we say, well, we can’t really be held accountable for those things in the scope of our project because that’s unfair to us. Now, the difficulty with public policy is that that’s kind of where the rubber meets the road. Because it’s not about building the schools, it’s about having the kids learn. And the difficult thing that we have is when we evaluate like this, I think we end up with a lot of schools, but with a big question what about learning. And this is just one example, right? We could give a lot of examples. We could say, well, in Flint, Michigan, we end up with water treatment plants, but we still have dirty water. We could talk about youth unemployment in certain parts of the United Kingdom and say we end up with a lot of programs and projects where we spend the money as we promised to, but the thing that we did didn’t actually solve the problem. And because of the project that we had, we didn’t have the scope to adapt and change and do something different. So it’s almost kind of an interesting thing that the way that we’re evaluating, trying to be accountable to the taxpayers, trying to be accountable to our bosses may be the thing that actually precludes us from implementing the thing that actually solves the problem. And that’s kind of one of the messages of the paper. And then the papers kind of saying, we need to be evaluating these things, but we need to be incorporating this other question of do we actually think we’re going to solve the problem more as well? And that’s kind of a big question mark as to how you do that. 

Salimah Samji So do you want to share some thoughts you may have on how does one get out of this? Like what can be done differently? Because the examples that you give are exactly what we see. And we work in so many countries and we teach here at the Kennedy School and there are so many students here. And so this is a recurring story that you hear probably every country, every city has examples where they’re doing lots. So you ask them, are you doing something about this problem? The answer is almost always yes. Is it working? Probably not or not as well as it could. How do you get out of this trap?

Matt Andrews It actually where we started off and said, we live in this world where half of the people are saying we’re doing well and then the citizens are saying you’re not. I think this explains it because what we’re doing is we’re saying, but of course, we building the schools and they saying, yes, but we’re not learning. Right. Well, but what we have this kind of youth unemployment program and we’ve been moving the money and we’ve hired people and we’re doing a good job and we’re excited. And people are saying the youth are still unemployed. Right. And so we’re evaluating ourselves in a way that says legitimately, we’re doing a good job, but we’re not solving the problem because of the gap. Now, it is an interesting thing. How do you get out of this? Because this leads to lack of confidence in the state, even while the state thinks it’s doing a good job. 

Salimah Samji It’s like a spiral down. It’s self fulfilling prophecy that just continues over and over again.

Matt Andrews You know, the easy way out of a spiral is to spiral the other way, right?

Matt Andrews It’s kind of say, you know, if you want to change a negative feedback loop, throw some positive feedback into it. The thing that’s really tricky, if we look at these projects and you say, Well, just do better on the project, we already being 75% successful in this project in the way we’re being successful. Now when you look at these documents, these assessment documents, and they say, well, how do you do better? They would say plan better. They’d say, execute on the plan better, move the money better. Now what they’re doing is they telling you how to be more successful in the 75% measure. And that’s what they’re doing, right? They’re not telling you how to do better on the 49%. So it’s saying, yes, we didn’t build some of the schools, so let’s make sure we finish all of those next time round. Let’s do the things we can control. And usually that’s where the advice on doing better comes from. And, you know, we teach that. We say to the students here, be even smarter with your plan. That’s what we say to them. Control even better. What we need to be recognizing is that the issue with failure is on the things that we can’t plan for.

Salimah Samji Which is pretty much if you’re working with complex problems, which are almost all of these things, you have no control.

Matt Andrews That’s exactly what it is. So we’re delivering on the complicated stuff that you can plan and control, and we’re struggling on the stuff that requires you to be dealing with complexity where you’re adapting and learning. And actually the way in which we are advising people to do better on the project makes it harder for them to learn. Because an example would be in many organizations if I look at the World Bank assessments, one of the things that you’ll see very commonly is they’ll say if things didn’t go well, it’s often there was a political surprise, something happened, and then they’ll say, How do you deal with that next time round? Because the evaluators and again, the evaluators do a very good job, is they’ll say you need to do a better political economy analysis at the beginning. And so I’m like, okay, what you’re telling us is that we need to have done a better evaluation of what was going to happen now four years ago. But we don’t know what’s going to happen now. And when you do that evaluation, you do the better political economy analysis that gives you more confidence in your plan, which means that when the surprise happens, you actually going to make a bigger mistake because you call it that. The issue is not do better plan, do better control. The issue is we need to build in more scope for adaptation. We need to have an approach where we recognize the things that are complex, that are not easily planned for, and we need to have a better way of doing those things.

Salimah Samji So more flexibility.

Matt Andrews More flexibility. So our approach in that sense needs to say, okay, what is it that we don’t know what parts of the project do we need to learn about? And how do we develop a project management methodology that allows us to learn as we go and to adapt? It doesn’t have to be your whole project, but it needs to be part of your project. When you can’t plan, don’t plan. Now, it doesn’t mean, you know, you just do nothing. And this is where we come into our work on problems of an iterative adaptation. We say develop a methodology that allows the facilitated emergence of the solution in a structured, responsible, accountable way, but that actually builds in the mechanism for learning and adapting.

Salimah Samji Because the truth is you don’t know what’s going to emerge, you don’t know what the solution is, and you want to allow the space for this to just emerge, as we’ve seen, they do.

Matt Andrews And that’s exactly what is is saying. This gap, this gap between 49 and 75% is all about stuff that surprised us. Yes. And to say, well, plan better for the stuff that surprises you, it’s like, well, that’s silly. You know, it’s I mean, it’s not totally silly because there are some cases with people that bad plans. But again, to give credit to the World Bank, most of the projects are well planned. Yeah. I mean, I mean, they have very good experts. They’re planning things well, they’re doing it with the government. This is not a city organization. And I would say this, most governments I work with, they have good planners. They have people who do this. The thing where things fall apart is all these surprises. And one of the things that we teach is when you are thinking about your project, you need to ask how much uncertainty is there? And if you find that there’s a lot of uncertainty, you’re saying there’s complexity here that we can’t plan for because if it’s uncertain, how do you plan for it? You know, it will say to people, how much is it, you know, that you know, how much is it that you know that you don’t know? And then how much is it that you don’t know that you don’t know? I know reality is that many, many unknown unknowns.

Salimah Samji Right.

Matt Andrews And so if you have unknown unknowns, you can’t plan for those things. So even asking those questions at the beginning allows you to work out is my project one where I should really be more adaptive as I move on, where I should really think carefully. Now, the beautiful thing about management is that management gives us many ways to deal with many different problems. We have a plan in control methodology that we can use when we know a lot of things, right? And even if we know what we don’t know, we can use planning control because there’s risk mitigation that we can bring into it. That’s what a good project does. The second thing we could do is we could use agile or adaptive methods where we know what we want to do, but there’s unknowns about how to fit it to the context. There’s things that we know that we want to use a new IT system in the health sector, but we’re not exactly sure what the uses in the health sector can or can’t do. So we can use an Agile method to kind of build that solution as we go.

Salimah Samji Right.

Matt Andrews Then there’s other methods that are actually really good if we don’t know anything and if we don’t really understand the problem well, we don’t understand what we want to do. We don’t understand the context. Well, this is where facilitated emergence comes in, where we can start with the problem and we can build our way towards a solution as we move along. All of these are management mechanisms. The problem that we see in many governments is that you have a lot of problems with lots of unknowns that require facilitated emergence or with significant unknowns that require agile and adaptive. But everyone is using plan and control.

Salimah Samji Why do you think that is? Because I’m sure some of them know, right? And we run into people like this all the time that they know that their plan and control method is probably not going to work as well, but they do it anyways. Why?

Matt Andrews I think a few things. I think one thing is that public organizations have to be accountable, and so we think in order to be accountable, we need to be telling people everything we’re doing and we need to spend two years planning it out in advance and we’re not going to get it done. And I think there’s some truth to that. I think another thing is that, you know, we have false confidence in our abilities. I mean, I’ve worked in government, I worked in the World Bank, and in all those times, sometimes I think I’d say my solution is so good that I’m pretty sure everyone’s just going to get on board, right? And it didn’t happen that way. But I really kind of believe that well-meaning, you know, as it may be. I think the kinds of people we train and the way we train, I don’t think we train people to learn that there are other options out there. So people look at something and they say, really, there’s a lot of uncertainty. I don’t think the plan is really going to work, but they haven’t been taught any other way. So if you go to the World Bank and you say to people, how do you do a project, they’re going to all sit down and they’re going to say, you need to have a pre ID phase, an ID phase, an appraisal phase. It’s the same for everything. That’s how the organization works. That’s how people are trained. Many of the people who work in public policy, whether they are in the World Bank, whether they need government, it doesn’t matter. They haven’t necessarily been trained in alternative ways of implementation. They’ve been trained in doing policy analysis. Right. And then they think that’s going to be the answer for everything, even if they’ve seen that it’s not the case. I think people sometimes just don’t know. It’s one of the reasons why we’re trying to say to people, the first thing you need to do is identify what your problem is. And often when we teach people about that, they say, well, there’s a method for doing that. Yes, there really is. And then we say to them, The beautiful thing about management is that there’s different options to match the different problems. And I say, Wow, is that really the case? It really is. So sometimes I think it’s that simple. It’s just giving people new options, saying to them, there’s things that you can try. And we have found that that really helps people in addressing these things in better ways.

Salimah Samji And you could even think about it as one problem, having different management methods you use for different parts, right? It’s not that one or the other. You can have a project with some plan and control, some facilitated emergence, or some adaptive management. It’s a combination of it’s not thinking about, Oh, my project has to fit into one of the buckets.

Matt Andrews Exactly. If you think about an education project, you could have an education project where you building schools and then in each school you want to have some kind of I.T. based system where the teachers use another teacher based system to engage with the students. And then you want to introduce some method to ensure that the students are learning. You could say the building of the schools is a construction project. We all know how to do that. We need experts to do that. We need a very good plan. We need a way in which we can bring those experts in in a very methodical way. And and that’s plan and control. Absolutely. The IT system is going to be a little bit, it looks like plan and control, but with IT systems, there’s always the question about who’s going to use them. And because there’s a question about who’s going to use them, we know the general solution we need. We need an IT system they’ll will use in the school, but we don’t really know exactly what those teachers are going to be able to do with it. So let’s use an Agile method. And that Agile method is going to allow us to say we need an IT system, but let’s build it as we move along. But then the issue of, well, how are the kids going to learn? Well, that one, we just the more unknowns about that. So why don’t we come in at the beginning and mobilize the teachers together and say to them, What do you understand as the problems of learning and then put them to work in trying to solve the problem and trying to find their own solutions? So that would be more like facilitated emotions and you could do all three in the same project. And that, I think is the key is ensuring that using the right method for the challenge that you’re facing. And then at the end of the day, evaluating it in the right way, too.  First one, we could ask the question, did you build the schools? And the second one, we can ask the question, is the system useful? And the third question, we can say, are we learning how to help kids learn different questions for evaluation? And if we ask those, I think it becomes less threatening to. You don’t necessarily end up with this gap of 50 to 75% because you get information in that gap to say, here’s how we progressing in that direction.

Salimah Samji Absolutely. And for accountability too, you can imagine, like the building of the schools is the largest cost, right? And you have all the metrics and the plan and control requires to be able to say, I did not pocket that money. It actually went there were schools that got built. But there is also that area where you really don’t know when you need to learn, and that doesn’t often necessarily take money. It’s a lot more other things, right?

Matt Andrews Absolutely. And at the end of the day will say, you know what, we are finding is getting back to the evaluations, we were looking at the World Bank evaluation Department. They ask, is it sustainable? And they ask, will it have a development impact? The interesting thing there is that those things happen after the project. They happen after the building teams have gone, after the contractors have left, and that’s about the people who are there. And really the question you’re asking is, do they have the capability to take it the next month? And that’s where we never ask that question. Now, if you’re doing facilitated emergence because you’re saying we don’t really know how to do that, we don’t know how to facilitate sustainability, how do you get teachers to inhabit a school building in a way that makes it a school, a community of learning? Well, we want to build those capabilities. Now. You can in a very accountable way, say in the period of time do we see the teachers more involved? Are they engaged? Are they learning? Are they spending more time with the kids? Now, those are things that happen when those capabilities are being built. And that’s where we would have a different set of questions about what success looks like, where we don’t see those being asked in pretty much anywhere at the moment.

Salimah Samji Great. That’s really, really helpful. My last question is, you mentioned we don’t teach this. What should we be teaching in schools? Because I do agree that there is a failure on all of our parts, right. To kind of teach people that, oh yeah, this is easy. All you need is a good technocratic solution and bingo, you’re going to make change happen. And we all know that’s not true. That’s not how change happens.

Matt Andrews Well, you know, I mean, the first thing is that even teaching people how to do good plan and control implementation would be a good start. You know, we have too many policy students who frankly leave our school without knowing how to run a budget. And, you know, that’s a scary thing. So even even doing a good plan, we have a really great idea. We put a plan. I say to my students, if you really believe your plan is that good, then you need to invest in the time to understand how to get it implemented. And then you need to understand how do I mobilize finances? How do I mobilize people, how to organize people? How do I manage politics? Right. Yeah. You can’t complain. At the end of the day, I had a great idea, but the politicians let me down.

Salimah Samji No, you should expect it.

Matt Andrews You should expect it, and you should have a strategy.

Salimah Samji Absolutely.

Matt Andrews And we should be helping people to learn how to do that. There are ways of doing that. Right. And then we need to teach people that there are different ways of doing this. I think we should teach people standard project management. We should teach people how to use Agile tools. We should teach people how to do facilitated emergence. And in our experience, we found that when students learn these things, they are much more adept to engage with the real world. These are practical, practical tools. They are tools that they can take with them when they finish. They tools we can teach people in degree programs. We can teach people in executive programs. We can teach people in the field. Now, when we don’t teach this, what we are hoping for or assuming is that people are just so smart that they even know it.

Salimah Samji Which we know is not true.

Matt Andrews Or that these people, when they get into their jobs, are given the space on the job to learn it. And I think we also know that that doesn’t happen. That doesn’t happen because one of the things that is peculiar about public policy organizations is that they are quite stressful environments, and the scope for learning on the job can be less than we think because failure is not always rewarded. So I think that we need to take upon ourselves the opportunity to bring people out of that environment and teach them some of the tools that can help them to be more successful in doing this. When they get into their work environments in the public policy space.

Salimah Samji I would totally agree because even in my own experience, a lot of things that I have learned have been high stakes. Right. When you’re already on a project, it’s $500 million. Yeah, that’s $500 million that’s already been signed. And there’s not much leeway to be able to change things. And it would be really nice to be able to have those tools before. So you don’t get to figuring out at $500 million that this isn’t working.

Matt Andrews Working, even if it’s even if it’s not $500 million, but it’s $50,000. If you are doing something as a public servant, you’re doing them for people who need you to do it for them. And in that situation, it always high stakes. It’s if you care for your job and if you came to this work of public policy because you actually care for the mission, it’s always high stakes because you’re working on behalf of people to solve problems that they need to have solved. I think even personally, things are always high stakes when you’re working on behalf of others. When you’re working on behalf of myself, I’m happy to take a risk on because I can manage that risk. But when you’re working on behalf of others, it gets a little bit more scary. Yeah. And so I think people get a lot more risk averse. And we have a long literature, a very big literature that tells us that people who work in this area are more risk averse than entrepreneurs. So I think we need to equip them so that they can engage in risky areas with more confidence.

Salimah Samji That’s right. And compartmentalize the risky right. Do what you know how to do in areas what we know how to do, and then the areas where we don’t know. You use these tools that will really help you learn and adapt.

Matt Andrews And I’ll say the other thing that we hoping to do in some of the work, especially in executive programs put in place, is we’re trying to build communities of practice as well. Because I think you can teach people, but if you could also have people who are trying to implement things better, work with each other, talk to each other across the world, across boundaries, learn from each other. I think that it will help people to manage this risk as well. I’m part of a community. I don’t need to know all the answers. I can learn as I go. I can click on to my community of practice and send a message and say, you know what? I’m on a project where we building school buildings, but I really want the kids to learn. Does anyone have any experience? And people will come back and they’ll say, this is what we’re doing. One of our goals is to build over time and we’re right at the beginning, so if anyone sends a message and says, how are you doing that? I’ll say, We don’t know. We’re going to facilitate its emergence. We want to build the biggest community of implementation practitioners in the world so that we can create this environment for learning. Ask us in five years how it’s going or better than that, join the community in five years. Hopefully you would be communicating with community members all over the world about implementation in ways that you can’t do right now.

Salimah Samji Great. Thank you so much Matt. It’s always a pleasure having a podcast with you.

Matt Andrews Thank you.

To learn more about the Building State Capability program, visit bsc.cid.harvard.edu.