Evaluation of 2003-2006 COL strategic plan

Consultant's report

Evaluation of the 2003-2006
Commonwealth of Learning Strategic Plan

Prepared by:
Patrick Spaven
Spaven Research & Evaluation
Brighton & Hove, United Kingdom

March 2006

 

Introduction

1.  The Commonwealth of Learning (COL) is an intergovernmental organization created by Commonwealth Heads of Government to encourage the development and sharing of open learning/distance education knowledge, resources and technologies. It has been operational since 1989 from its headquarters in Vancouver, Canada. COL's core operations are financed by voluntary pledges from Commonwealth governments. Its six major voluntary funding contributors - currently Canada, India, New Zealand, Nigeria, South Africa and the United Kingdom - each have a seat on COL's Board of Governors. Its average annual core budget over the last three years has been around $7m. It has also periodically undertaken fee-for-service consulting for international agencies and national governments, amounting to around $2m annually.

2.  In its current planning period - 2003-06 - it has implemented programmes in the areas of

• ODL Policies - fostering the adoption and implementation of open and distance learning policies within the broader educational and human resource development strategies and policies of member nations.

• ODL Systems Development - assisting in the development of open and distance learning systems that build on existing capacity or assist in creating new capacity appropriate for the contexts of member states.

• ODL Applications - demonstrating how open and distance learning applications can benefit individual learners, institutions and member states by accelerating human resource development.

3.  Since 2000, it has planned on a three year cycle to coincide with meetings of Commonwealth Education Ministers who are the most important decision-makers for COL's resourcing. COL has commissioned evaluations of its last two plans, this one being the second.

4.  The full terms of reference (ToRs) for this evaluation are included at Annex C. The key points in the ToRs are:

• The overall purpose of the evaluation is to ascertain what difference COL has made to date (March 2006) during the 2003-06 plan period.

• The main client for the evaluation is COL and its Board of Governors. Interested stakeholders are the 53 Commonwealth countries and in particular the partner institutions with whom COL works.

• The 3 key questions that the evaluation should answer are:

1. To what extent has COL made a difference in the life of its current 3 year plan?

2. What lessons can COL, its Board of Governors and partners, take from the operation of the plan into future planning periods?

3. How appropriate are the current sub-programmes in realizing the aims of the three strategic programmes, ODL Policy, ODL Systems and ODL Applications.

5.  The following elements of the ToRs are also important coordinates for the evaluation:

• The evaluation should be guided by the OECD Development Assistance Committee's evaluation criteria of relevance, effectiveness, efficiency, impact and sustainability.

• The evaluation should consider both the overall strategy, and the programmes at the initiative level.

• Longer term impacts may be more difficult to ascertain for initiatives of less than five years duration, but this should be done where possible.

 Executive Summary

6.  The Commonwealth of Learning (COL) is an intergovernmental organization created by Commonwealth Heads of Government to encourage the development and sharing of open learning/distance education (ODL) knowledge, resources and technologies. It is headquartered in Vancouver, Canada.

7.  It has an annual budget of around Cdn$9m, $7m of which is provided by voluntary contributions from a number of Commonwealth member governments. Among its staff are 10 education specialists who lead the implementation of its programmes in support of ODL policies, systems and applications.

8.  COL has commissioned a summative evaluation of its performance from July 2003 to March 2006, coinciding with most of its 2003-06 plan. This evaluation is largely qualitative, based on key informant consultations and case studies of 10 of COL's initiatives, at least one from each of its current 8 sub-programmes.

9.  COL is a small organization with a small budget. Its sub-programmes - such as its support for ODL in teacher education - cost around Cdn$500,000 annually including direct staffing.

10. COL aims to be relevant to development priorities in the countries that it serves. It is achieving this aim with its increasing focus on the MDGs which is real and not rhetorical.

11. COL thinks strategically but has not always translated that into robust, practical planning. Its monitoring and evaluation have been weak. COL is conscious of this and has been discussing significant improvements for its 2006-09 plan.

12. COL has a mandate to apply technology to ODL where appropriate and in general it has not allowed this to lead it into inappropriate initiatives. It should however take steps to integrate its technology work more with other programmes.

13. COL's specialist staff are a very valuable resource. They are knowledgeable and committed. When they are new to the organization they bring valuable experience, expertise and networks from one or more field of ODL operation. COL staff need to work in a more integrated way, within a more pro-active performance management framework.

14. COL has had difficulties explaining to external stakeholders and even to itself, the logic of its programmes: what value it creates and how. COL needs to develop a more accessible and balanced narrative of its worth. This implies, amongst other things, better use of the knowledge management tools it has created.

15. Most of COL's work has been in small interventions rather than large projects, However, there has been a trend during the course of the 2003-06 period to concentrate interventions around themes or institutions. It should continue this trend while retaining a degree of flexibility and agility which are among COL's assets.

16. COL is a consummate networking agency. It has constructed an extensive informal network of networks in ODL spanning most developing Commonwealth countries with appropriate links to expertise in the developed world.

17. COL is sensitive to the environments in which it works. It draws its staff from many of those environments and they travel to them in listening mode. This is partly through corporate ethos and personal inclination, and partly out of necessity. COL does not pay for its place at the table, it has to earn it through offering appropriate solutions that are best constructed through a process of iterative dialogue.

18. COL engages with ODL at all levels from national policy down to applications. There is a consensus that it should move steadily upstream, engaging more with governments and major agencies over ODL in policies and development strategies.  But this is a matter of balance and COL should continue to work on capacity building for ODL systems and the application of ODL in capacity building. In the last area - applications - COL must be very clear, through appraisal of its proposed initiatives, that they are either likely to lead directly to good outcomes with wide reach, or to have powerful multiplier or self-replicating effects.

19. COL contributes to a wide range of outcomes, although most of them can be classed as increased or enhanced capacity in or through ODL. Among the initiatives studied for this evaluation, the following stand out:

• Continued improvement and extension of the capacity of two very important ODL institutions: the National Teachers Institute (NTI) Nigeria, and the National Institute for Open Schooling (NIOS) India; and the mobilisation of NIOS as a resource for the development of open schooling systems in other countries.

• The introduction of ODL capacity in institutions previously delivering training and education through conventional modes. Two of these institutions - the Centre for Environment Education (CEE) India, and the International Crops Research Institute for the Semi-Arid Tropics (ICRISAT) - offer extensive reach or multiplier potential.

• ODL policy development and implementation in Gambia and Sierra Leone.

• A model with potential for adaptive replication, through which sustainable social and economic development appears to be taking place in rural communities in South India by means of a virtuous cycle of learning and productivity.

20. Some interventions have not led to positive outcomes or have not produced sufficient value for the time and other resources invested. Although this is inevitable in development, these examples should not be written off, as there is a big opportunity cost in nugatory work. Lessons should be learnt. The biggest disappointments in this period have been in COL's work in client-funded projects suggesting that this is an area that should be approached with caution.

21. Top-of-mind views of COL among informants were over 90% positive. Probing and challenging produced qualifications to the positives and some negatives, but with most people these were minor, relative to the positives. Its partners expressed a very high level of satisfaction. They regard COL as an organization of integrity.

22. The main groups that COL has to work harder to fully convince are some of its own Board members and representatives of bi-lateral and multi-lateral donor agencies.