written by Mark Heyward
During the first half of 2018, a group of 21 development practitioners from the Innovation for Indonesia’s School Children (INOVASI) program and partners, formed cross-program groups and completed the intensive, 15-week online course conducted by the Building State Capability program at Harvard’s Centre for International Development, called ‘Practice of PDIA; Building Capability by Delivering Results’. In addition to INOVASI personnel, participants came from the Indonesian Ministry of Education and Culture (MOEC), the Australian Department of Foreign Affairs and Trade (DFAT), the DFAT-funded Technical Assistance for System Strengthening (TASS) and Research for Improving Systems of Education (RISE) programs, the Social Monitoring and Early Response Unit (SMERU) Research Institute, and the independent Kuark organization. They worked together in four small groups to address real-world problems related to INOVASI’s aims:
- How to improve learning outcomes in Indonesian primary classrooms?
- How to systematically improve learning outcomes in Indonesian districts and schools?
- How to build an evidence base on what works to improve learning outcomes – to inform policy and practice?
A smaller group of INOVASI personnel had previously taken part in the online course in early 2017. This work thus built on an earlier experience with problem driven iterative adaptation (PDIA). Consultations with Professor Lant Pritchett, one of the authors of PDIA, also served to sharpen INOVASI’s approach and clarify strategic directions to pilot design and working with government partners.
As a result of the online course in 2018, the approach to PDIA in INOVASI has been redefined and strengthened. While the earlier focus was primarily on practitioners exploring class-based problems, and defining and trialling solutions, the interpretation of PDIA is now broader, more systemic in focus, adopting a politically smart approach to identifying problems at national and district levels, as well as school and classroom levels, and placing greater emphasis on the solutions (finding out what works) than on the process. This article describes that journey and how participation in the PDIA course resulted in new directions for INOVASI.
The ‘wicked hard’ challenge of education development in Indonesia
Indonesia has made considerable advances in basic education. Government spending has doubled over the past 15 years, and enrolment in primary education is almost 100%. But despite more children having access to learning opportunities, this has not yet resulted in better learning outcomes. As Lant Pritchett put it in the title of his 2013 book: ‘Schooling Ain’t Learning’.
This is what is called a ‘wicked hard’ problem (Andrews, Pritchett & Woolcock 2017). International donors have been working with the Indonesian Government since the 1970s to improve the country’s basic education system, to introduce school-based management and an active learning approach in classrooms. At least USD 5 billion has been granted or loaned to Indonesia for education development in this time (Cannon 2017). But the problem remains.
While these funds have been spent on a range of interventions, the most common approach consists of cascade training. Projects have typically been designed, delivered and evaluated in a top-down way. Training modules are produced by experts to support the implementation of international ‘best practice’, particularly relating to active learning and school-based management. Since the 1980s, successive national curriculum frameworks have mandated an active learning pedagogy. The training is ‘delivered’ to groups of teachers in selected school clusters by local facilitators, and some level of support is provided to district government to encourage them to ‘scale up’, ‘disseminate’ or ‘replicate’ the training to teachers in other schools. The focus is on ensuring that the centrally-designed training is faithfully applied as it is ‘rolled out’ to more schools.
The question that should be asked is, if this top-down, pre-design approach is effective, why are foreign experts and donor-funded projects still required to reform teaching and teacher management? Why is Indonesia’s performance on international benchmarking tests so disappointing? Why have Indonesian government policies on active learning and teacher management not been widely implemented? While there are many contributing factors, such as unattainable curriculum guidelines combined with poorly-trained teachers, local political imperatives, poorly-trained facilitators cascading mandated curricula, and the absence of basic facilities required for active learning in the poor regions, the failure of reforms to sustain and go to scale is a serious problem for the system and a challenge for reformers.
Photo: Indonesia’s international benchmarking test results continue to disappoint
The INOVASI program aims to answer these questions and address these concerns. The program, which commenced in 2016, is a partnership between DFAT and MOEC. It is being implemented by Palladium. It is part of a global movement that includes programs such as RISE and the Gates Foundation’s new Global Education Strategy and seeks to improve children’s learning outcomes.
To achieve this, INOVASI investigates what works (and, conversely, what doesn’t work) to improve literacy and numeracy at the classroom, school and district levels. INOVASI is currently working in 17 districts across four provinces to pilot a range of approaches to improve teaching and learning in the classroom, to improve the policy framework and organizational capabilities so teachers can teach more effectively in the classroom, and to enable all children in the classroom to reach their learning potential.
How to improve learning outcomes in Indonesian primary classrooms?
The key question facing Indonesian education reformers is how to improve learning outcomes. If the top-down technical training provided by earlier projects has not worked, what will? INOVASI’s design is underpinned by the belief that many previous reform initiatives which aimed to improve student learning outcomes in Indonesia were designed around externally-identified problems with pre-determined ‘one size fits all’ solutions. These borrowed policies and ‘best practice’ solutions are not always relevant for Indonesia and its many diverse cultural contexts – and this is why the success of previous efforts has been limited.
To find out what works in context, INOVASI has adopted PDIA as its main program approach. An important principle of this approach is that working with local partners to define problems and find solutions ensures the ‘best fit’, with potential for institutional reform. Solutions that are consistent with local understandings, values, capacities and resources are more likely to transform ongoing behaviour and practices than solutions imported from elsewhere (Andrews et al. 2012; Guthrie 2018, Sopantini 2014). It was with these assumptions in mind, that the four groups undertook the ‘Practice of PDIA; Building Capability by Delivering Results’ course in early 2018.
Photo: an INOVASI District Facilitator from West Nusa Tenggara explores different aspects of the PDIA approach.
Groups One and Two took slightly different paths to address the question of how to improve learning outcomes in Indonesian primary classrooms. Group One included two of INOVASI’s education program development team, a program officer from DFAT, a representative of the local non-government organization, Kuark, which has been developing science materials for primary school students, and a project operations officer from Palladium. Group Two consisted of two INOVASI personnel from the remote island of Sumba, in East Nusa Tenggara, two members of the program’s national education development team, and another Kuark representative.
Group One defined their key problem as follows: teachers don’t know how to support students’ learning based on their needs. Group Two came up with the following key problem: early graders are able to decode but they have low reading comprehension. Both developed complex fish-bone analyses to explore the problem. Sub-causes identified, included the following:
- Schooling is driven by textbooks and a standardized curriculum that does not allow teachers to differentiate the learning according to children’s needs
- The curriculum does not systematically address reading skills
- Teachers are too busy with compliance and administration tasks (lesson plans, reports)
- Principals and supervisors focus on the grades instead of meaningful learning in the classroom
- Teachers do not see the connection between children’s assessment results and the design of their teaching practices
- Pre-service education does not equip teachers with strategies for teaching reading
- School culture does not promote discussion of learning difficulties or solutions
- Books are generally expensive, unattractive, and inappropriate to the children’s context and reading levels
- District funding for books is inadequate
Group One sought to address their problem by seeking positive deviance among the schools on Sumba Island. They learned that a systematic approach is needed to identify positive deviance. Group Two felt they needed more information to address their problem. They undertook a systematic review of grade one, two and three textbooks, in order to find out the kind of reading assessments included. They discovered that text books and teacher guides assume that all students are able to read fluently. There is no curriculum for the teaching of reading. The curriculum is thematic in early grades. Assessment tasks are provided for each theme (animal and plant reproduction, energy, friendship, etc.) and consist mostly of factual recall questions. The teacher books explain how to score the assessments. However, there is no explanation about how to use the results to improve teaching and learning.
The groups also visited schools and used the program’s network of local facilitators to find out how national policies are being implemented in Sumba. This focused on the practice of a daily 15-minutes of silent reading and the creation of reading corners for children. They found that implementation is inconsistent, dependent on the leadership of principals and local school supervisors, and that building a ‘reading culture’ through daily reading and book corners is disconnected from the mainstream curriculum. As such it is seen as an ‘add on’ and of less importance than following the text books. Work to address these issues is ongoing.
The team is now ‘crawling the design space’, working with district partners to restructure and revitalize the school cluster system, to co-implement cluster-based training for teachers, and to develop policies and programs to support the development of a reading culture in schools and communities. Teachers, principals and supervisors need to be given explicit license from senior officials at district, province and national level to introduce literacy activities within normal lessons as part of the school day, rather than as an ‘extra’. Discussions are underway to achieve this objective.
How to systematically improve learning outcomes in Indonesian districts and schools?
Group Three consisted of a technical officer from the SMERU research institute working on the RISE program, an education program officer from DFAT, and a senior manager and advisor from INOVASI. The group set out to address the challenge of improving learning outcomes through Indonesia’s decentralized education systems. The problem statement they came up with was: Why is improvement of early learning outcomes not systematically supported in schools in some districts in Indonesia?
Sub-causes identified, included the following:
- There are no credible measures of reading ability in early grades
- The system (districts, schools, teachers, parents) focuses heavily on learning in upper grades and assessment in grade 6
- Demand is low for improvement of learning outcomes – there is a lack of a sense of problem or crisis
- The tradition of top-down, compliance-focussed government stifles local innovation
The team set out to address the challenge of how to get early grade literacy outcomes to matter. The main approach was (1) to explore existing successful practices (positive deviance), including those developed by non-government agencies such as Save the Children and UNICEF, and (2) to identify or build a body of evidence to highlight the problem of poor literacy in early grades and use this evidence to advocate for reform.
The group found some great examples of positive deviance, of local programs really making a difference, and, in some cases, producing compelling evidence of what works to improve learning outcomes. Similarly, the evidence to show poor reading levels in the general school population is there, but it is patchy and needs collating and presenting in compelling ways to decision makers, community, mass media and teacher groups. Indonesia’s new national student assessment known as Asesment Kompetensi Siswa Indonesia (AKSI) holds promise. Routine assessing and reporting of reading ability to parents of early grade children was also suggested as a solution. Discussions with the Pratham Institute took place to learn about their approach to community-based assessment and reporting on early grade reading in India.
As a result of all this, INOVASI is now working with national and local partners to develop cluster-based training in literacy, numeracy, and use of mother tongue to transition to Indonesia’s national language in early grades: local teams are piloting short courses in teacher working groups (school clusters) and are working with local officials to ensure that the districts ‘own’ the pilots and give explicit authorization for teachers, school heads and supervisors to try different approaches in their classrooms. The short courses are being accredited by local government so that teachers gain credits and are incentivized to participate.
The course materials are well-informed from a technical perspective, but are delivered and adapted at the local level, through an iterative ‘learning by doing’ approach, so that both delivery and content are aligned to local context – and to ensure that they are politically informed. This involves district planning meetings and visits for district officials to schools which demonstrate good practices in other locations. At these meetings the teams present compelling data analysis from AKSI and other studies to highlight the problem of low levels of literacy and numeracy in Year 4, and the need to adopt different approaches to teaching reading and number in early grades to address the problem. Simple classroom-based assessments on early grades reading and numeracy are being introduced as part of the short courses.
Photo: a teacher from East Sumba district trials a possible solution to mother tongue language transition, as part of an INOVASI pilot, during the PDIA pre planning process.
How to build an evidence base on what works to improve learning outcomes – to inform policy and practice?
Group Four consisted of an official from MOEC’s Policy and Evaluation Unit, an education program officer from DFAT, the Monitoring, Evaluation, Research and Learning Manager, and the Research Advisor, from INOVASI, a technical manager from TASS, a doctoral research candidate from the Australian National University, and a representative of the SMERU research institute.
In their problem analysis, the group found that policy is not generally linked to evidence:
- policy-makers ignore or cherry-pick the evidence to suit their political agendas,
- top-down policy making ignores local context, local evidence,
- policy implementation is more concerned with budget absorption and short-term political gain than long-term strategy
- knowledge management is weak – evidence is often not accessible and not shared between government agencies; the evidence is often highly technical and is not communicated in a digestible form to decision-makers
While the group explored many policy issues and their links to evidence, at the end of the course they narrowed their efforts to teacher management as a policy focus. At this time, INOVASI and TASS were supporting MOEC to analyse teacher adequacy. The team’s approach was to adapt the information available to conform to the political agenda and mindset of the policy makers at the time; playing a role as knowledge broker and leveraging short-term political gain.
Around 2.3 million teachers are currently employed in state schools. Over half of these (65%, or around 1.5 million) are civil servants, paid from general fund transfers from central to district governments. The remaining 800,000 are contract teachers, funded by school operation grants and district budgets. This suggests that Indonesia needs almost one million additional civil servant teachers to ensure that all state schools are properly staffed. In other words, Indonesia needs to nearly double the number of teachers, and it’s spending in education. This is clearly not possible. Efficiencies must be found in teacher deployment. As a result of this work, building on that of previous and concurrent programs (including the World Bank, ACDP and USAID PRIORITAS), the Minister has now agreed on the following policy options:
- Teacher utilisation will be improved by increasing compulsory face-to-face teaching hours from 24 to 30 lessons per week
- Multi-grade teaching will be introduced in primary schools, and multi-subject teaching in secondary schools.
Following on from the PDIA course work, the TASS team assisted MOEC to calculate teacher needs – and how the new policies can reduce those needs by 25%, which enabled the Minister to demonstrate that he is considering efficiency in approaching the issue. The team also supported MOEC’s Head of Planning – and the Minister – to present the policy options to the Vice President, and is now working with the Vice President’s Secretariat to reduce the amount of information to be presented, making the presentation of evidence more compelling and digestible for busy policy makers, including the Minister and Vice President. Meanwhile, INOVASI is co-designing and preparing to pilot a program to introduce and support multi-grade teaching in a district in East Java.
Reflecting on the PDIA experience
Taken together, the outcomes from the four groups point the way forward for INOVASI and partners in tackling the ‘wicked hard’ problems identified – and ultimately in improving learning outcomes for Indonesia’s 27 million primary school children.
All 21 participants found the course demanding. Working in mixed teams, generally not co-located, keeping up with the reading and assignments (in a second language for most) and meeting the intellectual demands were substantial challenges. But all stuck it out to the end, and the teams developed a strong identity and sense of shared purpose along the way. The course was a major investment of time and effort. The key question now is, what have we learned?
The process of identifying and analyzing the problems was challenging, but resulted in deeper understandings of the challenges and pointed to potential solutions, which the groups began to explore in the iterative process at the end of the course, as described. It also resulted in a much stronger and politically savvy approach to addressing the problems.
In the early phase of INOVASI, PDIA was interpreted as a way of improving the quality of teaching by getting teachers to identify their problems and figure out ‘what works’ to address them. Strong emphasis was put on getting these stakeholders – mainly teachers – to master the ‘five whys’ and fishbone techniques; and on the process of pre-piloting proposed solutions. Because of this emphasis on individual teacher performance, finding instances of positive deviance dominated as one of the main purposes of applying PDIA: local solutions to local problems.
As the groups started following the PDIA course in 2018, it soon became evident that all the provided assignment readings and examples dealt with problems at the organizational and policy level of a system. An important reflection was that these kind of problems are different from the matters of technical know-how and lack of relevant discipline knowledge that limit the teaching competence of many in the Indonesian context. The PDIA process of building and convening consensus groups to address organizational dysfunction seems much better adapted to management issues than to technical knowledge issues. The INOVASI team and participating teachers seemed intuitively to realize this, and had assimilated the PDIA process of problem investigation and solution finding to the well-established methodology of classroom action research.
Another realization was that, even if cases of positive deviance were discovered, they could not be adopted or scaled out without working at the organizational and policy level. Most mass employment systems depend on institutional incentives for changing practice at scale rather than the contagion of good peer practice.
Within the course, one of the concepts that proved most influential was the idea that the key to triggering change is getting the problem to ‘matter’; to making a systemic failure that has previously been accepted and lived with, to be no longer acceptable. Most citizens – and thus most politicians and bureaucrats – in Indonesia are satisfied with the standard of education (Blane & Daan 2009). Yet, international benchmark tests and recent in-country studies show that Indonesian children lag far behind their peers in other countries, and struggle with the basics of literacy and numeracy; ‘schooling ‘aint learning’. The PDIA course has provided strategies for getting this problem to ‘matter’; these strategies can be further developed, but the team now has a way of thinking about this, a conceptual framework to work within.
Another very useful concept for working politically with stakeholders is the mapping of an authorizing environment, identifying authority as something that exists not just at the summit of a government or organization but where the chain of command influences responsiveness — and particularly the recognition that authority lines can overlap and contradict one another. This idea is linked into the wider network of ideas associated with mapping the change space — mapping as a strategic approach to action; and particularly, the three concepts of authority, acceptance and ability. This was useful because it showed the importance of working simultaneously to develop and strengthen all these different dimensions of bringing about change in organizational approaches to improvement: the political, the cultural and the technical.
In summary, and comparing the experience of PDIA in INOVASI before and after the course, it is important for teams to understand the deeper logic behind the brand devices such as the ‘five whys’ before they take on implementing PDIA, because this logic gives front-line implementers an independence of judgment about particular practices that may or may not aid success, while at the same time a good understanding of how to steer the big strategies behind the approach.
To learn more, visit our website or download the PDIAtoolkit (available in English and Spanish).