MEASURING COL’S IMPACT: STAKEHOLDER SURVEY 2010

COL commissioned a Stakeholder Survey in 2010, using the occasion of the Sixth Pan-Commonwealth Forum on Open Learning in Kochi, India as an opportunity to gather input. A results-based management and evaluation consultant, Dr. Patrick Spaven, conducted the research, which provided valuable insight into the views and experiences of COL among a range of stakeholders. By investigating how COL behaves, this survey makes direct links to accountability – and opportunities for improvement. This is a summary of Dr. Spaven’s findings.

SOLICITING STAKEHOLDER INPUT

COL’s four-day Pan-Commonwealth Forum on Open Learning in November 2010 attracted a representative range of COL’s partners, clients, governors and resource people. Thus, it provided an appropriate opportunity to conduct a stakeholder survey. Forty-three people engaged with the survey, which included facilitated group discussions, one-on-one interviews and short self-complete questionnaires. One of the facilitated group discussions involved most of COL’s Education Specialists and a range of other staff. Participants were selected to represent a range of COL’s initiatives, different regions of the Commonwealth and a gender balance.

Conducting the survey at COL’s conference could bring bias to the research, Dr. Spaven explains in his report. Since most of the participants in the survey are actively engaged with COL or have reason to expect to be in the future, it is likely that the results are more appreciative than for a survey conducted in a strictly randomised and anonymous manner. Despite this possible limitation, he still contends that the Stakeholder Survey is a valid way of assessing views and experiences of COL.

STAKEHOLDER SURVEY FINDINGS

The facilitated discussion groups, individual interviews and questionnaires provided insight into three general areas: what COL does, how COL adds value and where COL could do better.

What COL Does

The majority of partners value COL as a helping and guiding hand, often through a combination of contributions such as advice, capacity building and advocacy. Partners usually perceive COL’s help as capacity building. Increased capacity in the partner organisation is the legacy, even if other outputs and outcomes fail to live up to their promise.

The Stakeholder survey suggests that COL has successfully mainstreamed quality in its work with partners. Given historic concerns about quality in open and distance learning (ODL), this would appear to be a major achievement.

Many participants found it difficult to pinpoint precisely the difference that COL is making. There was more of a feeling that COL was a long-term partner that guides and assists them in a general sense. This is a positive finding for partner relationships, but COL needs to record and report in precise terms the results it helps its partners achieve.

COL has helped us tackle the quality enigma in teacher training. It is having a huge impact on the quality of learning.  –Stakeholder survey respondent

How COL Adds Value

The survey finds that COL is valued for the following assets and behaviours:

Assets

• Long, deep and broad experience of ODL – and the confidence to apply it
• Effective staff
• A direct line to – and credibility with – governments

I can’t believe how small a group can have such an amount of reach. –Stakeholder survey respondent

Behaviours

• A disposition to listen and build interventions around partner needs, rather than come with its own agenda and impose solutions
• An insistence that its partners have ownership of the programmes on which they collaborate
• The willingness and ability to be a long-term partner
• Accessibility and flexibility – a lack of bureaucracy in its dealings with partners
• Excellent communication
• A business-like, results-oriented approach that rubs off on partners
• An even-handedness towards member states of the Commonwealth

Women involved in COL’s Lifelong Learning for Farmers programme in Kenya celebrate opening personal bank accounts

COL is a more effective organisation than it was three years ago, the survey finds. Its current staff are praised for their experience and energy. COL is also seen to be more focused and results-oriented.

At the end of five years, they [other organisations] washed their hands off, but at COL, they care for you throughout. –Stakeholder survey respondent

Where COL Could Do Better

A substantial minority of the survey participants believe that COL still tries to do too much, given the limits of its resources, and spreads itself too thin. A similar small number of respondents felt that COL needs to work upstream in all of its initiatives to achieve wider impact.

A frequent comment was that COL does not always leverage the potential of its direct line to governments. This was mentioned in three contexts:

• Where countries have extensive ODL infrastructure but it is seen intrinsically to be a low quality option,
• In sections (such as farming) or sub-sectors (such as technical and vocational education and training – TVET) where ODL has less of a track record than in higher education, and
• In countries with undeveloped ODL infrastructure and policy in general – mainly small states and particularly in the Pacific.

People who had a view on COL’s approach to gender thought it was somewhat superficial. They urged more pro-activity.

Respondents readily identified COL with the application of technology in ODL. There was broad consensus that COL promoted technology appropriately, and survey participants were grateful to COL for showing them what to aspire to in terms of technology.

The Pan-Commonwealth Forum on Open Learning is valued without exception but is thought to need a thorough conceptual overhaul. In particular, it is seen to be formulaic with little opportunity for deep engagement with new ideas and research findings.

Be more aggressive with governments – not just Ministries of Education. –Stakeholder survey respondent

MOVING FORWARD: MORE CONFIDENT ODL ADVOCACY

The strongest message from the Stakeholder Survey is that COL in general is doing the right things and doing them right. It should not change for change’s sake.

Both focus group and interview participants were asked to give COL one piece of advice for its next Three-Year Plan. A plea for more advocacy with governments was the most pervasive advice, particularly from participants from Africa and small states. There was a clear assumption that COL’s voice would be listened to in these environments. Advocacy for ODL in technical training and community-level livelihoods was the most urgent context for these efforts. The other area that should receive more resources than now is TVET.

“It is important for us to understand what our stakeholders think about us, both our behaviour as an organisation and the effectiveness of our work,” said COL Vice President Professor Asha Kanwar. “This mid-term evaluation has been extremely useful in identifying the areas that we need to address further, particularly policy and gender.”

The development of COL’s next Three-Year Plan, 2012-2015 will also benefit from regional meetings of Focal Points and online discussions.

The full Stakeholder Survey report is available online at www.col.org/3YP, where readers will also find the current Three-year Plan.