Research & Evaluation

This e-conference was moderated by Charlotte Creed and Bernadette Robinson who have been working on the development of the COL PREST materials[1] for the last eighteen months. In the course of this work, several things became apparent:

• the need for more and better research in ODL and the existing gaps in research on ODL;
• the interest of practitioners in learning more about doing research and evaluation;
• the limited opportunities available to practitioners and researchers to discuss matters of research;
• the need, ideally, for a community of practitioners who could identify and demonstrate fundamental principles of good research in ODL, to guide ODL researchers, evaluators and publishers
• the need for a stronger knowledge base to assist planners and decision-makers:

Politicians, businesspeople and increasingly the general public, have the idea that distance learning can change radically the face of education.... We have a duty to let them know whether or not these expectations are grounded in reality. 

Daniel, J. 2002:2

 

This experience and the interest of COL in assisting the development of researchers led to this e-conference on research and evaluation in ODL.

 

Who participated?

 

The e-conference attracted 231 international participants, of which 69 were active contributors to the threaded discussions and made in total 269 postings. The participants were located in  14 countries[2], with the largest groups in Canada, South Africa, India, Australia, and the United Kingdom . They came from a wide variety of backgrounds and institutional settings and included higher education students (graduate, MA and PhD levels) and academics, ODL journal editors, educational consultants, professional researchers working in higher education research units, agencies and ngos, ODL programme managers, Ministry of Education officials (including a Permanent Secretary and a Senior programme officer) and knowledge resource management experts. The majority were working in higher education although there were others working in  the areas of teacher education, secondary schooling, agricultural extension and vocational programmes.

 

The participants were therefore a diverse group, ranging from novices thinking of doing some research to 'professional' and highly experienced researchers. Inevitably, their interests varied accordingly and we felt that the e-conference was able to meet the needs of some participants more than others. On the basis of the feedback we received, it seemed to us that those who were relatively new or inexperienced in research, or to the field of ODL, got the most out of the experience. One of the most difficult choices we had to make, as moderators, was whether to keep the discussion open and broad, to meet the wide range of interests and expertise, or to focus it down to more specialist discussion. We chose to do the former in order to involve new practitioners. In the course of the discussion distinctions emerged: between researchers based in industrialised and developing countries, and between 'academic' researchers pursuing higher degrees and practitioners wanting to pursue issues arising from their work. These last two issues were neatly captured by Judith Kamau, Botswana , when she wrote:

Many of the contributions by participants from the developed world talk of on-going research projects for PhD and other qualifications ...... While these issues are valid, no contribution so far has addressed the wider scope of training in ODL to capacitate people to even carry out a simple needs assessment appraisal for implementation of ODL programmes. I would have been delighted to see comments that  strive to bring ODL practitioners on an equal footing in terms of knowledge about research methodology. So when colleagues say this is not one of the issues I get rather worried because this is a demonstration of the fact that we are at different levels of development in research particularly comparing the developed and the developing world. Email contribution  in the pre-PCF3 e-conference on Research and Evaluation in ODL, May 18th-25th, 2004.

 

Starting questions:

1. Does researching or evaluating open and distance learning (ODL) present any distinctive methodological challenges? If so, what are they? How can the challenges be met?
2. What constitutes good quality research and evaluation in ODL? What criteria should we use in judging research and evaluation quality?
3. Do research and evaluation practitioners in ODL need any particular sets of skills? How can practitioners improve their skills? What are the best ways of doing this?

 

Themes and issues arising

 

Out of the resulting discussion, several themes and issues emerged:

• the need for 'rigour' in research done, whether in ODL or other fields of education;
• the lack of funding and political will in supporting research in ODL;
• the need for research directed towards improving practice in specific contexts as well as towards generating new 'truths', conceptual frameworks and generalisations that transcend specific situations;
• access to the research literature on ODL and the impact on current research choices and conduct;
• the quality of research done (and the varying views on this, with some thinking it was good and others that it was not:;
• the need to expand formal and informal opportunities for research skills development in developing and developed countries and appropriate to researchers working in a range of contexts (e.g. ministries, education institutions, international agencies, etc);
• the particular challenges of research capacity building and of doing research in developing countries;
• the difficulty of making sense of the ODL research landscape and of understanding its topography;
• the importance of dedicated databases systematically collecting, filtering and disseminating information and literature on ODL as essential underpinning for research in ODL and the problems resulting from the demise of some databases and information services (such as ICDL)
• why the research done is not made use of sufficiently. One participant (Barbara Fillip) suggested three possible reasons and one possible solution:
  • Research isn't addressing the immediate knowledge needs of practitioners?
  • Research findings are not presented in user-friendly formats?
  • Research findings are not easily accessible?
  • Can action research address some of the current concerns by forging more ongoing linkages between researchers and practitioners by making the educators, administrators and students essential members of the research team?

It was clear from the discussion that some of the issues are complex, that participants had a wide range of needs and interests and that the three starting questions still need more debate. We hope that the PCF3 conference will provide some opportunity for this. There's clearly no shortage of constructive ideas!

 

One of the most productive outcomes of the e-conference was resource identification: participants have shared information, references and websites. Nick Gao at COL 's Information Resource Centre provided excellent backup in providing comprehensive reference for requested areas such as the research agenda in e-learning, the research agenda in teacher Education, and references on the use of Distance education in agriculture and Extension education. Two sets of references about research in ODL appear at the end of this summary.

 

This exchange was a good example of how a community of practitioners can help to build research capacity and initiative. We hope that this kind of sharing can continue in some ways and we include this in our list of possible next steps.

 

Recommendations

 

Based on the exchanges in the discussion, there seems to be a clear demand for more formal and informal forums for exchange of information, expertise and references in the area of research and evaluation in ODL. COL and other lead organisations might like to consider some of the following practical ideas:

• Short, expert-led seminars and study circles (virtual or otherwise) in particular areas of research in ODL or research skills.
• Promoting research and evaluation skills development in a wide range of ways, e.g. as part of a development project, encouraging research partnerships, mentoring schemes, study leave, donor/funding pressure, in journals and newsletters.
• Investing in a dedicated one-stop website for ODL researchers with links to other centres of excellence, pools of information and training resources.
• New strategies to help a maintain critical overview of information and literature in ODL (in an age of information overload): 
  • regular commissioning of critical state-of-the-art research overviews in particular areas of ODL, e.g. in terms of the subject areas (teacher education, agricultural extension, health, literacy), areas of investigation in ODL (student support, management of DE delivery, course design) types of research (impact studies, cost studies, formative evaluations) educational levels (higher education, basic education, vocational) or reconfigured in new ways (e.g. the contribution of ODL to the Millennium Development Goals (MDGs) and Education for All (EFA)
  • dedicated databases/websites for particular areas in ODL (e.g. teacher education, basic education, agricultural extension) which, guided by board of experts in the field, collect and categorise key references, attempt to provide an overview and research agenda for the area, provide good practice guidelines and well-documented case-studies. These could act as one-stop shop for policy makers and researchers in the field.
  • Abstracting
  • Selected and annotated bibliographies in particular areas of ODL.
  • Investment in a high quality, internationally-orientated dedicated database which can systematically collect and categorise literature in such a way that researchers can conduct finely tuned searches and are kept up to date with latest ODL news and information (rescuing ICDL from oblivion?).

 

Moderators:

 

Dr. Charlotte Creed, International Research Foundation for Open Learning (IRFOL), The Michael Young Centre, Cambridge CB2 2PG, UK
ccreed@irfol.ac.uk,   www.col.org/irfol

 

Professor Bernadette Robinson, Centre for Comparative Education Research, School of Education, University of Nottingham, UK .
b.robinson@berobinson.net

 

 

References

 

Anderson, T. 2004 'Practice guided by research in providing effective student support services'. Paper presented at the Third EDEN Research Workshop Supporting the learner in distance education and e-learning, March 4-6, 2004 Carl Von Ossietzsky Universitat, Oldenburg, Germany .

 

Berge, Z. & Mrozowski, S. 2001 'Review of research in distance education 1990 to 1999', American Journal of Distance Education, The American Journal of Distance Education, 15/3, pp 5-19

 

Cookson, P. 2002 'Access and equity in distance education: research and development and quality concerns', Keynote address at the Annual Conference of the Asian Association of Open Universities 22-26 February 2002, New Delhi: IGNOU, available on http://www.athabascau.ca/html/staff/admin/cookson/AAOU_keynote.doc

 

Daniel, J 2002 'Why research distance learning? In proceedings for CRIDALA conference, Hong-Kong': Open University of Hong Kong, June 5-7, 2002. Available on: http://www.ouhk.edu.hk/cridal/cridala2002/discus/messages/4/daniel.pdf

Jegede, O. 2000 Contextualising research methodologies in open and distance learning, paper from Centre for Research in Distance and Adult Learning, the Open University of Hong Kong.

 

Jegede, O. 1999 'Priorities in open and distance education research: opinions of experts and practitioners within the Commonwealth', paper in the Pan-Commonwealth forum on open learning: empowerment through knowledge and technology, 1-5 March 1999, Brunei Darussalam, Vancouver: Commonwealth of Learning

 

Mishra, S. 1998 Distance education research: a review of its structure, methodological issues and priority areas, Indian Journal of Open Learning 7(3), pp 267-282

 

Naidu, S. forthcoming 'Research, scholarship and publishing in distance education: weaknesses, opportunities and challenges', paper to be presented at Pan Commonwealth Forum, Dunedin, New Zealand, 4-8th July 2004.

 

Panda, S. 2000 'Mentoring, Incentives and rewards in research as professional development', keynote paper for the Conference on Research in Distance and Adult Learning in Asia (CRIDALA), Open University of Hong Kong, June 21-24, 2000. Hong Kong: Open University of Hong Kong

 

Perraton, H. 2000 Rethinking the research agenda, International Review of Research in Open and Distance Learning, 1/1, 11-11.

 

Phipps, R., & Merisotis, J. 1999 What the difference? A review of contemporary research on the effectiveness of distance learning in higher education, available on http://www.ihep.com/Pubs/PDF/Difference.pdf

 

Ramanujam, P. 2000 Quality and research in distance open learning, Indian Journal of Open Learning, 9/1, 63-72.

 

 

Further references

 

1. Ten Efficient Research Strategies for Distance Learning
http://www.westga.edu/~distance/ojdla/spring71/wright71.html

 

2.  Forty years of distance education research: results and future development
http://www.eden-online.org/papers/EDEN-Prague.pdf

 

3. Status and trends of distance education research 
http://www.eden-online.org/papers/EDEN-Prague.pdf

 

4. Carving a New Path for Distance Education Research
http://home.earthlink.net/~davidpdiaz/LTS/html_docs/newpath.htm

 

5. Issues in Distance Learning This review of literature and current information related to distance learning deals with several primary research issues. These include redefining the roles of partners in distance education teams, technology selection and ado http://carbon.cudenver.edu/%7Elsherry/pubs/issues.html

 

6. Thach, E., and K. Murphy, "Competencies for Distance Education Professionals," Educational Technology Research and Development 43, No.1, (1995): 57-79. Summary at
https://courses.worldcampus.psu.edu/welcome/facdev101/course/instintro/instintr2.html

 

7. Reflections on the State of the Art of Research in Open and Distance Education in Asia http://www.asianjde.org/

 

8. Conceptualising educational research in distance education
Quarterly Review of Distance Eduation, 4(3), pp. 331-343

 

9. The future of distance education; research, conceptual development and practice
http://www.uwex.edu/disted/conference/Resource_library/proceedings/03_29.pdf

 

10. Distance Education And Career And Technical Education: A Review Of The Research Literature http://scholar.lib.vt.edu/ejournals/JVER/v28n2/zirkle.html

 

11. Directions for Future Research in On-line Distance Education:
http://tojde.anadolu.edu.tr/tojde12/articles/sadik.htm

 

12. Research, Scholarship and Publishing in Distance Education: Opportunities and Challenges Som Naidu http://www.cemca.org/newsletter/jun2003/jun2003.pdf

 

13. The Implications of Brain Research for Distance Education
http://www.uwex.edu/disted/conference/proceedings/DL2002_35.pdf

 

14. Pedagogical frameworks and action research in open and distance learning
http://domino.lancs.ac.uk/edres/csaltdocs.nsf

 

15. A Cross Sectional Review of Theory and Research in Distance Education
http://www.westga.edu/%7Edistance/ojdla/summer62/simeroth62.html

 

16. Approaches to Studying Research and Its Implications for the Quality of Learning From Distance Education
http://cade.athabascau.ca/vol2.2/8_Kember_and_Harper.html

 

17. Research in distance education - past, present and future (1994)
http://www.nettskolen.com/pub/artikkel.xsql?artid=139

 

18. Synergy between Flexible and Distance Learning and Research and Technological Development 
http://info.tuwien.ac.at/hsk/sefi/papers/ramalhot.htm

 

19. APPRAISAL OF RESEARCH AND EVALUATION IN DISTANCE EDUCATION; THE TANZANIAN EXPERIENCE By Dr. Eustella Bhalalusesa
http://www.col.org/pcf2/papers/bhalalusesa.pdf

 

20. The instruction of research methodologies in graduate-level distance education degree programs http://www.cade-aced.ca/icdepapers/clevelandjones.htm

 

21. Research in distance education / edited by Terry Evans : revised papers from the 5th Research in Distance Education Conference, Deakin University , 2000.
http://www.deakin.edu.au/education/RIPVET/RIDE_Papers/RIDE_Book.htm

 

22. The research-teaching nexus in tertiary distance education
 http://www.deakin.edu.au/education/RIPVET/conferences/2000/RIDE/Ch_4_White.pdf 

 

23. Researching distance education: Penn State 's Online
Adult Education MEd Degree on the world campus
http://www.avetra.org.au/abstracts_and_papers_2001/Askov-Simpson_full.pdf

 

24. Research and Evaluation Priorities for Distance Education
WebSite:www.unl.edu/nn21/krockwell/related.html

 

25. Review of Research in Distance Education, 1990 to 1999 by Zane L Berge and Susan Mrozowski.  In: American Journal of Distance Education vol.15, no.3, 2001 pp5 - 19.

 

26. Internet-Based Distance Education: Barriers, Models, and New Research
Word Short Paper for WebNet 2000
http://itc.ollusa.edu/secs/education/presentations/barriers.PDF

 

27. Research and Evaluation Needs for Distance Education: A Delphi Study
http://www.westga.edu/~distance/ojdla/fall33/rockwell33.html

 

28. Old Concerns with New Distance Education Research 
http://www.educause.edu/ir/library/pdf/eqm0126.pdf

 

29. A Tangled Web Indeed: The Difficulty of Developing a Research Agenda for Distance Education DEOSNEWS Vol. 11 No. 10, ISSN 1062-9416.

 

30. A review of distance education research: An annotated bibliography approach
http://www.ed.psu.edu/acsde/deos/deosnews/deosnews8_3.asp

 

31. Research in distance education: A status report.
International Review of Research in Open and Distance Learning 1 (1): 1-9. Available at http://www.irrodl.org/content/v1.1/farhad.pdf

 

32. Carving a New Path for Distance Education Research
 http://home.earthlink.net/~davidpdiaz/LTS/html_docs/newpath.htm

 

33. A profile of researchers in open, distance and adult education in Asia and their research foci
 http://www.ouhk.edu.hk/~openlink/current/9910/e-resd3.htm

 

34. Research in Distance Education: A Status Report by Farhad Saba 
International Review of Research in Open and Distance Learning
ISSN: 1492-3831 Vol. 1, No. 1 (June 2000)

 

35. Distance Education Research Guide
<http://distancelearn.about.com/gi/dynamic/offsite.htm?site=http://www.uidaho.edu/evo/dist10.html>

 

___________________
[1] Practitioner Research and Evaluation Skills Training (PREST) resources to help ODL practitioners develop and extend basic research and evaluation skills - info@col.org.

 

[2] Australia, Bangladesh, Botswana, Canada,  Germany, Ghana, Guyana, Hong Kong, India,  Nigeria, Mauritius, South Africa, United Kingdom, United States of America.

 

 

--Charlotte Creed and Bernadette Robinson

"Politicians, businesspeople and increasingly the general public, have the idea that distance learning can change radically the face of education.... We have a duty to let them know whether or not these expectations are grounded in reality."
--Daniel, J. 2002:2

The quality and quantity of research and evaluation in open and distance education (ODL) has been criticised by several writers and the need for improvement is widely acknowledged. One consequence is that research and evaluation is making a limited contribution to the understanding and improvement of educational practice on ODL. Given the international expansion in the use of distance education and the great expectations it has inspired, this is a serious situation which needs careful addressing if it is to change.

In this virtual conference, we invite you to discuss the issues behind the problem and to suggest how best we, as a community, can improve the quality of research and evaluation in ODL. Questions we could consider might include:

• How best can we raise the skill levels of practitioner researchers? 
• What are the distinctive methodological challenges of research and evaluation in open and distance learning? 
• What are the distinctive challenges of research and evaluation in developing countries in contrast to those of richer, industrialised countries? 
• What criteria should we use in judging research quality? 
• Can we, as a community, construct a framework reflecting fundamental principles of good research in ODL, to guide ODL researchers, evaluators and publishers?

Professor Bernadette Robinson is Professor of Continuing Education at the University of Nottingham, UK. She works as an education consultant for COL, the World Bank, UNESCO, UNICEF, the EU and the International Research Foundation for Open Learning. She has wide research and evaluation experience. Professor Robinson has been the Academic Editor for the Practitioner Research and Evaluation Skills Training Programme (PREST), a set of training resources to help ODL practitioners develop and extend their research and evaluation skills. www.col.org/irfol

Dr. Charlotte Creed is a Research Fellow at the Cambridge-based International Research Foundation for Open Learning, a specialist research agency affiliated to the Commonwealth of Learning as COL's research arm and whose function is to carry out research that will guide policy on ODL. As part of IRFOL's remit to COL to provide support and training for ODL researchers -- particularly those working in institutions with limited resources -- she is currently co-ordinating the PREST Programme. In her work for agencies such as the World Bank, UNESCO, and the British Department for International Development, Dr. Creed has conducted international metasurveys comparing ODL practices across institutions. Her research interests include teacher and basic education by distance in developing-country contexts, the ethical dimensions of ODL and the relationship between research and policy development.  www.col.org/irfol