Training Impact Study

EXECUTIVE SUMMARY

Overview
The Commonwealth of Learning (COL) has always held training to be one of its main functions. This was confirmed in the 1988 Memorandum of Understanding and COL's Three-year Plan, currently in preparation, which continues to envisage the organisation as a capacity builder and Commonwealth resource for training. Furthermore, 'Priorities for the Future' on the COL web site declares that the organisation intends to become more prominent as a training services provider.

Over the period 1990-2000, COL has expended considerable resources on various capacity-building initiatives including 114 training events and the development of six Training Toolkits. The purpose of this Training Impact Study was to assess the contribution that the COL training program has made to the achievement of the overall mandate of the organisation and specifically, according to the Terms of Reference for the Study:

the effectiveness of the training methods and approach;

the extent to which the objectives of the trainers and the participants were met;

the extent to which the training was reflected in improved practice;

the evidence of impact at the organisational level; and

the extent to which the training was achieved at acceptable cost.

The Study was required to take into account both the intended beneficiaries (the participants) and the secondary beneficiaries (peers, supervisors, policy-makers, etc). To this end, the lead researchers and six Research Associates in various regions of the Commonwealth surveyed open and distance education practitioners in six representative Case Study Sites, previous participants in the training events, the Workshop Co-ordinators and specially-selected Key Informants. A process of triangulation was used to establish patterns and correlations in the many hundreds of responses received.

The researchers established that overall, there is widespread appreciation of, and satisfaction with, the COL-provided training events, the funding and professional support provided by COL and the more recently developed Training Toolkits. There are clear indications of these events improving the understanding, skills and performance of individuals and indications they are having a major impact upon at least some of the distance teaching institutions. The participants observe that the events serve to strengthen regional, national and international understandings, foster the development of groups with shared interests and encourage rapport and collaboration between the participants, all of which they value highly. The Workshop Co-ordinators feel that the training events have met most of the intended goals and comment favourably on the dedication and participation of the participants. It is evident that training was seen as an important part of COL's mission by both Key Informants and other respondents. They were of the opinion that it must continue to receive a high profile, high priority and as high a level of support as COL can muster from its own and its partners' resources.

The training events have taken many different forms and have been provided in many very different contexts, often with extremely heterogeneous groups. Respondents note various strengths and weaknesses in current training provision and after ten years of operation, it is understandable that they might suggest some changes and new initiatives. There follows a summary of the main issues revealed in these responses and the lead researchers' recommendations to address these.

Main observations
There was found to be great diversity in the standard of the evidence and analysis in the Workshop Co-ordinators' reports. In terms of performance indicators, the researchers found weaknesses and omissions in the design of, and feedback from, the Standard Training Evaluation Form. The researchers find also that the training events could gain from a greater use of formative evaluation.

While there was overall satisfaction with venues, logistics and administrative arrangements, there were a number shortcomings, most of which could be anticipated and planned for. Some training events held at, or close to, the host institutions suffered from constant interruptions with participants being called away to attend to normal duties. Difficulties arose at some training events through inadequate lead-times for informing and funding the participants and sending them workshop material in advance.

There was extensive evidence of an enormous need for training ranging from basic orientation programs in distance and open education to advanced programs in new information technologies and institutional management. The scale of need and the various local factors likely to facilitate or constrain successful implementation suggest the need for more focus on 'training the trainers'.

Not only was there shown to be great variation in the participants' backgrounds, experience, knowledge and skill, but some indication that the most appropriate persons are not always selected to attend the training events due to nepotism and other factors. There is evidence of great variation in the participant's motivations for attending the training events and expectations of the outcomes. It is important that the Co-ordinators and presenters are aware of, and can measure, the success of the events against these. It is also important that they fully appreciate the barriers to change and innovation which may confront the participants on returning to their institutions.

The vast majority of training events have been at the operational level, concerned with course or material design and educational technology. Very few courses have been targeted at managers and yet many respondents express concern over the continuing lack of awareness and commitment among politicians, senior government officers and institutional managers. Many respondents decry lack of follow-up to the training events. The lack of subsequent evaluation of the outcomes and impact come across as another major weakness in the current training strategy. The impression is therefore gained of a series of 'one-off' events and piecemeal and ad hoc provision, which accords with the concern by McWilliams and Mugridge (1998) that COL has not yet been able to respond to needs in a consistent, comprehensive or cohesive fashion.

The researchers believe that a more systematic and innovative approach is called for, one which accords with COL's revised mission, the extent of the need, the call for follow-up, the great potential for collaboration, uses of technology and mixed-mode delivery. The researchers conclude that while it is important for COL to continue to be responsive to institutional and individual needs, to achieve maximum impact and returns on outlay, it is also important that the training events should match COL's strategic goals. Training events need to target strategically/politically important topics, focus on specific target groups, and be designed with specific outcomes in mind. These events might be provided centrally by COL or by COL sponsoring, or co-funding, programs proposed by regional or local providers in response to the stated strategic goals. The latter approach may create a greater sense of ownership and help institutions to also think holistically and strategically. It is also suggested that COL should do more to target managers, train trainers, provide training in advanced forms, using the widest possible range of delivery strategies, encourage and support more certified training provision and work toward the establishment of a collaborative 'virtual training institution'. Such is the scale and strategic thrust needed to accelerate change and enhance professionalism in distance education that COL also needs to take every advantage of strategic alliances and collaborative initiatives in sharing resources, networks and expertise.

Many respondents to this survey have suggested using a mix of mediated and face-to-face training. It is entirely appropriate the COL should itself make greater use of distance education strategies and technologies to support change and enhance performance and there is great potential in assembling or creating material that:

builds upon the excellent beginnings represented by the Training Toolkits;

encourages and supports innovation, change and performance improvement;

embodies the best principles of instructional design for self-instruction by individuals and groups;

empowers and skills the trainers or practitioners so that they can then implement the new ideas with the help of local support groups and mentoring, face-to-face on online.

Much of this material already exists around the world and it may well be a question of negotiating terms and adapting existing guides, reports, etc. rather than COL starting from scratch and bearing the full cost. Delivery may need to be in print and electronic. The forms of delivery are not the major issue; what will finally count is the quality of the content and self-instructional strategies.

RECOMMENDATIONS
Workshop Co-ordinators' reports and the COL Standard Training Evaluation Form

Recommendation 1:

COL should review its expectations of the Workshop Co-ordinators' reports and require all Co-ordinators to submit their reports to a common format and with the level of detail and analysis necessary to monitor trends, issues and future training needs.

Recommendation 2:

COL should review and re-design its Standard Training Activity Evaluation instrument to ensure that it yields the necessary data and analysis and should require all Co-ordinators to submit completed forms, a summary and their findings and conclusions, to a standard format.

Recommendation 3:

COL should strongly recommend that Co-ordinators apply day-by-day formative evaluation to help ensure that the training events are constantly in accord with the participants' needs and wishes.

Venues, logistics and administrative arrangements

Recommendation 4:

Workshop Co-ordinators or their representatives should be enabled to make preliminary visits to the workshop venues to check the accommodation, technical and logistical support and to subsequently arrive at these venues at least one day before the actual events to ensure that all preparations are in place.

Recommendation 5:

COL should always ensure that training events are off-campus and residential, away from the resource persons' and participants' places of work and outside teaching and examination periods to avoid any possible distractions for the participants and to ensure optimum time-on-task and continuity.

Recommendation 6:

COL should institute a standard checklist to ensure that all arrangements under its control are planned, confirmed and known to all parties, and that all foreseeable contingencies have been anticipated and planned for.

Recommendation 7:

COL should ensure that there is sufficient lead-time to facilitate participants' travel and other arrangements and for them to study the pre-workshop material, and even undertake pre-workshop assignments, so that they are fully prepared for, and able to take full benefit from, the training events.

Training the trainers

Recommendation 8:

COL should ensure that every training event provides 'train the trainers' opportunities, with locally-recruited resource persons and other potential trainers 'paired with' and 'shadowing' the Co-ordinators or other presenters throughout the planning, workshop presentation and evaluation.

Recommendation 9:

COL should ensure maximum opportunities for 'train the trainers' workshops, using the Training Toolkits as appropriate, to expose potential workshop presenters to, and rehearse them in and advise them on, the various training strategies and techniques open to them.

Workshop participants

Recommendation 10:

COL should take a more active role in setting the criteria for selecting the participants to ensure that they can not only benefit from the training but have the capacity, status and resources to implement the new ideas and practices. There should be early and careful selection of the participants, and the Co-ordinators should be provided with participant profiles including their motivations and expectations (see also Recommendation 11).

Recommendation 11:

In revising the Standard Training Evaluation form, care should be taken to so frame and link the question(s) eliciting the participants' specific motivations for attending the training events, specific expectations of the outcomes and their satisfaction levels in these regards as to yield useful information about objectives met and unmet. There should also be an item asking participants to identify the institutional barriers to implementing the ideas and practices promoted in the training events.

Follow-up

Recommendation 12:

COL should ensure that the training provided is more than a series of one-off events. The Co-ordinators should be contracted to provide or organise follow-up or reinforcement activities and self-help groups or networks to encourage and support implementation of the ideas and practices promulgated in the workshops.

Recommendation 13:

Impact studies should be conducted by COL and/or the Co-ordinators six months after the training events to gauge the extent of individual and institutional change and the barriers to change.

Developing understanding and changing attitudes in those holding influence

Recommendation 14:

COL should initiate a series of regional think tanks or seminars at which senior persons can consider the role, costs and benefits of open and distance education, the global trends and local needs, how they can support the identified needs for change, collaboration, and address funding and other resource constraints.

Future training strategies

Recommendation 15:

While it is important that COL remains responsive to its customers' needs, it is also important that the training provided should accord with COL's strategic goals, achieve maximum impact, returns on investment and be available on the widest possible scale. COL should therefore explore new topics and forms of delivery, including the internet/web and a collaborative 'virtual training institution' to accelerate change and enhance performance (see also Recommendation 20).

Recommendation 16:

COL should explore where there is still need and potential for further formal programs in open and distance education and articulation, collaboration and innovation in delivery in such programs throughout the Commonwealth.

Recommendation 17

COL should explore mixed-mode training delivery, combining face-to-face presentations and demonstrations with on- and off-campus applications of the Training Toolkits in print and electronic form, follow-up assignments, projects, impact studies and mentoring to facilitate implementation.

Recommendation 18

COL should further develop and extend the Training Toolkits, dealing with more advanced topics and in particular, by commissioning a Training Toolkit on staff development techniques in support of a 'train the trainers' strategy.

Recommendation 19

COL should organise a Commonwealth-wide (possibly in conjunction with the next Pan-Commonwealth Forum) and/or a series of regional forum(s) on training needs, strategies and networks in order to develop mutual support systems.

Recommendation 20

COL should seek more strategic alliances with other organisations such as the IEC and UNESCO to achieve the scale and strategic thrust needed to enhance professionalism in distance education, maximise available resources, networks and expertise and work towards a collaborative 'virtual training institution'.