Promoting Quality in Distance, Flexible and ICT-based Education: Reflections on the meaning of quality in open and distance learning

Promoting Quality in Distance, Flexible and ICT-based Education

Rio De Janeiro, Brazil 3rd - 6th September 2006.

Reflections on the meaning of quality in open and distance learning.

Helen Lentell
Commonwealth of Learning

Abstract

This paper, whilst recognising the significance of the quality movement for ODL, makes the case that rather than quality in ODL programmes being addressed in terms of inputs - best practices and procedures - quality should be determined by outcomes - e.g. did this ODL programme achieve its objectives? The paper argues that if we did this we might have a more relevant approach and one which enabled us to explore different models of implementing sustainable ODL.

Setting the scene

When I accepted this kind invitation to speak to you all - and September back in February seemed a long way away, and Brazil so warm and attractive to windy, wet Vancouver - I have to admit I did not think too hard about what you were asking me to do. I accepted with alacrity. But then the weeks went by and my conscience began to prick and the emails from ABED came ever thicker and faster - and I realized that I must put my mind to the task. If I had thought about the task I must admit that I would have been more circumspect in my acceptance. The reason is this - the word "quality" and "promoting quality", assuring "quality" or whatever we say we are doing in the context of education with respect to "quality", is exasperating me. And in the context of development (the context within which I work at COL) it is not an unambiguously helpful concept.

Let me get this of my chest before I say more. Who is not in favour of quality? It is like "mother hood and apple pie". I have a growing feeling of unease when I hear politicians, bureaucrats and civil servants talking about "quality" in education, let alone quality in distance education. Do we all share the same understanding? I don't think so. We often discuss quality as though it were a homogeneous attribute of educational processes and products. When I was very new to this business of ours, when I was a youngster and believed the world was simple, I thought this was true. We were in the ODL business to do "good". We were attracted to its philosophy of access and equity. The learners were all that mattered. But now I am older and wiser and I know more. I know there are many stakeholder groups who have to be satisfied. What is quality for one group of managers, one group of employers, one group of civil servants, may not have much impact on the quality of learners' experiences and learning outcomes. Indeed they may well have deleterious effects on learners. I have in mind here for example the disastrous consequences to education in my country (in my view) of introducing numerous national tests at all levels of schooling in order to ensure quality. Thereby subverting the education of young people to a series of tests and examinations that had to be prepared for and taken. And in so doing undermining the enlightenment philosophy of education that empowers individuals with broad knowledge and transferable skills, and a stronger sense of values, ethics, and civic engagement.

However I am not politically naïve, I recognize that there are many stakeholders and they have a legitimate right to some element of quality assurance in relation to our ODL programmes. After all we know the history of distance education has not been a totally virtuous one. We have all heard of those dreadful correspondence courses that took huge fees from their students, and made even bigger promises regarding students' futures, but left learners struggling on their own without support or meaningful feedback. Actually I say "we all know" but perhaps our collective memory has lapsed. COL has been running a series of virtual conferences in preparation for its conference in Jamaica later this year. One of the postings drew my attention because it pointed to the same naïve faith in e learning that led many companies to spend thousands of dollars on developing e training which "despite enlisting the skills of their best instructors, who consistently used best practices - e-mail and phone messages to encourage participants", led to "only 30 percent of students...(managing)...to complete the pilot course, resulting in a whopping 70 percent dropout rate." The posting also cited a "recent poll conducted by the Cambridge, Mass.-based Forrester Research, a market research firm focusing on the Internet and technology, in which (..they report..) more than half the people enrolled in e-learning programs in general failed to complete them because of lack of interactivity. Dropout rates intensify on par with the duration of the course. A follow-up study ... reported that on a multipart, 25-hour-e-track programme, 72 percent completed the first three-hour course, but only 52 percent completed two courses, and just 19 percent (not even a fifth) completed the full 25 hours. Participants cited "distractions, interruptions and conflicting priorities" as main reasons for the poor results. ¹

My point is that if we know this - and we do ² - why are we still doing it? If our quality standards are not about inputs (best instructors), or processes (encouraging participants) but about outcomes - we would ask different sets of questions: did this achieve what it was intended to? And if it did not we might not do it anymore. We might do something else. We might use our money more appropriately, more effectively and more efficiently.

But I am getting ahead of myself. What I have been saying is that quality is an ambiguous and slippery concept. In education it means different things to different people and promoting "quality" does not necessarily lead to quality educational outcomes for learners. But before I am branded as being anti-quality, let me say that I am in favour of quality - because to be against it would be like saying I am against quality and for whatever the opposite is - which I guess is rubbish. And I think it would be unthinkable for an educationalist to say they were in favour of rubbish education! But I must admit to a prejudice here - I am anti the subversion of education to technical, bureaucratic processes - the application of which has little relevance to the real world of education and training and where teachers are on the whole trying to do their best under very difficult circumstances with minimal resources.

However I do want to make the case that the quality movement has been very significant for ODL - first because it has enabled ODL to demonstrate its value on the same playing field as traditional face to face institutions and secondly because it drew attention to the critical under pinning systems requirements of ODL - systems that might otherwise go un-noticed to policy makers. But the stakeholders of these early processes were not the learners - rather they were governments addressing national political agendas concerned with institutional accountability. If our concern on the other hand is the learning experience of our students, and the way we as professionals concern ourselves about the impact of what we do, then we have to look at other approaches and processes - or, to steal a phrase from Bruce King, we need to undertake "extreme evaluation" ³ Extreme evaluation is all about how we gather data about our teaching, how we analyze and reflect on this data, and the way we develop and improve our practice as a consequence. I would maintain that this reflective approach to practice and our commitment to quality improvement are the hallmarks of professionalism and it is the responsibility of educational managers to foster a climate - a culture if you will - in which questioning, critiquing, arguing, analyzing, is encouraged and rewarded. And the results of this process are listened to and acted upon. Without this no learning takes place and no improvements occur. Indeed if we do not have this approach the adoption of ODL by developing countries will always focus on the inputs rather than the educational and training problems that ODL methodologies are being harnessed to address. And before I finish I want to shamefully promote some materials COL has produced: PREST - The Practitioner Research and Evaluation Skills Training - these training resources provide practitioners working in open and distance learning (ODL) with access to basic knowledge on how to conduct research and evaluation in their everyday work. That is the tools for promoting quality in distance, flexible and ICT-based education.

The Pros and Cons of the Inputs approach to quality in ODL.

In the 1990s, with the rapid development of state quality assurance systems in e.g. the UK and Australia distance learning and face-to-face universities found themselves subject to similar external quality assessment systems. Daniel ⁴ (argues that for distance education it was a good development because it emphasized that quality assurance for distance learning should be part of the overall quality assurance scheme for tertiary education and not some special add-on. In the UK the quality assurance system focused on six generic aspects of tertiary education. These were:

  Curriculum Design, Content and Organisation
  Teaching, Learning and Assessment
  Student Progression and Achievement
  Student Support and Guidance
  Learning Resources
  Quality Management and Enhancement

Daniel argues that one could divide up the tertiary education process differently. "However, the key point ... is that whatever the list, it is much easier to conduct quality assurance and assessment for distance learning than for face-to-face teaching. This is because distance learning uses the well-tried industrial principles of division of labour and specialisation, operating more systematically and self-consciously than a campus operation." He says "that when the quality assessors came to the Open University it was very easy to show them the systems and the materials for each of these processes." Actually it was not at first so easy - I was involved in the first audit and the assessors had to be educated about teaching and learning in ODL. They had no understanding that our face to face tutorial sessions were only a small part of this - and not the whole thing, so used were they to the idea of lectures! But as we know advocacy is what we in ODL have to do all the time! But my personal suspicion is that we only really impacted on the quality assessors when our tutorial staff, themselves became quality assessors. They understood ODL and ODL methodology. And remember the OU at any one time has some many thousands of part time lecturers, most of who hold the OU in the highest of esteem, and are an army of advocates. But Daniel's point is well taken and I have often argued that the Open University in the UK led the way in raising the quality of teaching and learning across the whole HE sector in the UK by its very openness - everything was in the public domain for public scrutiny, unlike the private world of the face to face classroom.

So distance education has an advantage in the processes of quality audit. And distance education practitioners, associations and leaders have painstakingly been documenting the criteria for quality in our modality. E.g. the ADEA (Association for the Development of Education in Africa) working group on distance education published in 2004, "Distance Education and Open Learning in Sub-Saharan Africa: Criteria and Conditions for Quality and Critical Success Factors". There are a number of criteria. Here for example are the criteria for quality in course materials. These are materials in which

"The content, assessment and teaching and learning approaches in the course materials support the aims and learning outcomes; the materials are accessibly presented; they teach in a coherent way that engages the learners; there is an identified process of development and evaluation of course materials" ⁶.

This statement is then broken down into eleven component parts - which I summarise as the following:

Clearly laid out aims and learning outcomes, including notional hours of study.
The content and teaching approach supports learner's in achieving learning outcomes
Learner friendly
Content accurate and up to date, and relevant
Language levels appropriate
Context relevant
Supports active learning
Content is an unfolding argument rather than discreet bits of information that are unconnected
The various elements of the course and the media are clearly integrated
Course materials are accessible
Technical quality is high - e.g. presentation, lay-out etc.

I am in complete agreement with these criteria for learning materials. As indeed I am with all their other quality criteria, although I secretly wonder if the working group found any ODL programme that come anywhere near satisfying all of them! Are they setting a "gold standard" few can achieve? But the ADEA working group did a very good job in establishing the ideal. Especially as the standards makes clear that careful attention has to be made to developing ODL systems. This requires thorough planning and implementation. In my context at COL this is so rarely comprehended by politicians and funders - who often behave as though they do not understand the need for under pinning ODL systems, or think that these systems can be put in place overnight! However having a system that is open, broken down into component parts and with widely accepted industry standards does not mean that ipso facto ODL institutions are involved in quality provision, even when they say they are, and claim to be the arbiter of their country's quality in ODL standards. E.g. over the last few years COL has done an extensive review of the courseware of masters programmes in ODL offered by some of the premier Open Universities in the world. I mention these courses because if any courses ought to be exemplars in our field these are they. This is an extract from the executive summary of one of the reviews

"The principal finding of the review is that although the two programmes are wide-ranging in their coverage of distance education, and their content reflects a considerable degree of experience and knowledge, their effectiveness is significantly affected by a number of weaknesses. These are principally the lack of currency of course materials, the non-utilisation of information and communications technology, and the poor level of editing and proof-reading. In each of these areas the (name of institution) programmes compare unfavourably with programmes offered by other institutions." ⁷

This is not good news. But many students continue to study on this programme - and I can tell you on all other areas too the quality report on this programme was below par. But this programme continues to be offered and many students are enrolled on it world-wide. When the market is so buoyant and the demand so great - what is the pressure on the institution to do anything about its practice?

I mention this here - not because it is an argument against quality criteria in ODL but rather because it raises questions of compliance. Who is responsible for this? Many countries have appointed semi-autonomous bodies to undertake this to protect their citizens as consumers. This is an important development especially as education is a traded commodity and where there is a demand for education, as there is in the developing world; it is open for exploitation by unscrupulous providers ⁸. These national bodies have been involved as we have seen in defining standards. But it does raise further questions about how independent the QA body truly is. If we take the case of India, the Distance Education Council (DEC) is the regulatory body for ODL set up by the Indian government. Its objective is "to ensure the quality of education and to assure the standard of degree/diploma earned through the distance mode of education." ⁹ However it was placed under the chairmanship of the VC of IGNOU, which inevitably raises anomalies and questions of credibility. Reddi noted this:

"It is the funding and quality assurance organization for ODL; but has the structural anomaly of being set up in an organization for which it is both the donor, quality assurance agency, and yet is subordinate to IGNOU." ¹⁰

This Indian example may be an extreme case - but most quality assurance bodies are made up of the professionals from the business. So e.g. in the USA quality assurance and accreditation is done by regional associations of universities. What these bodies are doing, as I have indicated earlier, is laying down best practice derived from the practice of "successful" organisations in the business. That is what those working in the business collectively agree best practice is. So for example the UK OU is seen as a successful organisation. It has existed for getting on for forty years and has good teaching and research ratings. Therefore its practices are frequently quoted as best practice/quality processes for the business. But are they the best practises for all distance education? Are their practices appropriate in different contexts?

Let us take learner support. Throughout my professional life in ODL I have tried to champion learner support as an essential and integral part of successful ODL. I know from my years at the UK OU that without it ODL would fail the vast majority of learners. An exclusive or primary concern for educational content which seems to be dominating much effort in the developing world's move to ODL is destined to failure if equal concern is not given to learner support. This has been expressed so eloquently by Professor Kader Asmal the former Minister of Education in South Africa. He said of learner support in ODL that he highlighted learner support because:

"... My own experience of correspondence study those many years ago was one of deep loneliness. I refer to it as "the loneliness of the long distance learner". It was not that I did not understand the text or grasp the concepts at a general level, though many students desperately needed support in these aspects. It was rather that I had no opportunity for interaction, both with peers and with those who might have more knowledge than I. I had no opportunity to test my ideas to extend my conceptualisation, or to make the knowledge my own so that I could apply it meaningfully. I believe it is essential to provide proper opportunities for proper interaction. Indeed, for me, one of the greatest challenges of distance education is to find creative ways to establish accessible leaning structures which enable this interaction.

Providing meaningful, high quality learner support would be one of my key requirements for the merger of South Africa's dedicated distance education providers. It would also be a major requirement of any provider, public or private or international that offers distance education in South Africa." ¹¹

If we take the UK OU as the "gold standard" for quality in ODL which many practitioners in our field do, see for example the referencing of the UK OU in the case studies in "Perspectives on Distance Education: Towards a Culture of Quality" ¹² published earlier this year, we might be looking for tutors who have high expertise in their subject/discipline and can therefore offer support in that subject/discipline to learners. This was the kind of core competency we were looking for when it was part of my responsibility to employ and train tutors at the UK OU. Moreover we were appointing tutors on a ratio of one tutor to 25 students, because we considered that numbers larger than this would impede the provision of individualised and meaningful feedback. How realistic is this in a developing world context where the very reason for adopting ODL methodology is to overcome the gross shortage in human resources? Where would these tutors come from? Nowhere have I found this issue addressed in the literature although I have searched high and wide for it. Indeed I suspect it is our failure to find a convincing argument to this dilemma that has impeded the successful implementation of ODL methodology in many developing countries - after all it is not very convincing to develop materials for distance education study but not to have any idea about the marking of assignments and the general support of learners. And this must be part of the suspicions of ODL held, for example, by the Ministry of Education in Mozambique. For ODL to receive the recognition it deserves

"to become part of "the" solution for the Mozambican educational challenges, and to also receive the institutional commitment it needs to succeed, it cannot be seen as adding to the enormous pressures on an already weak system. It must be seen as some form of educational reform. ODL programmes must be designed to resort to alternative and creative partnerships to implement successful learner support strategies." ¹³

I would argue what we need to be doing to take forward quality in ODL is finding many models of learner support, many different, more appropriate practices that will support our learners, rather than simply judging the system on established normative standards. And I would posit that this will look very different from what we see now as best practice. Moreover the criteria by which we will judge these models will be based on whether or not they work. That is whether or not they help to bring about the educational and training goals that have been set. This is a very different model of quality than what we see now. For it will be based on outputs not inputs. In other words: to what extent did the ODL programme or ODL methodology increase access? What did learners' learn? How did this compare with conventional provision? To what extent did our processes, i.e. what we did, lead to these results? I believe until we do this ODL will not fulfil its potential

This point is made by Tony Bates in his forthcoming review on the COL book "Perspectives on Distance Education: Towards a Culture of Quality" ¹⁴

"We need to look much more at outputs if we seek to inculcate a culture of quality, and this inevitably means more and better record keeping and analysis, and more and better research and evaluation. Since there are trade-offs always in terms of resources, it may mean switching away from some of the more bureaucratic approaches to describing and measuring inputs and putting more emphasis on output measurement and relating these directly to the models of course design, production and learner support and interaction being used by an institution."

Thus far I have argued that the growing concern with quality in education in general and ODL in particular is both a good and bad thing. It has enabled ODL professionals to demonstrate the efficacy of ODL and the way the component parts of the ODL system are carefully planned, developed and implemented in an integrated whole. And it has allowed the best of ODL to be seen among the best of face to face. This is true of both single and dual mode institutions. But still lingering are issues of quality assurance in ODL - e.g. there remain questions about what quality is - i.e. who defines it and what are their interests in this process? Who is responsible for assuring quality - it is really not good enough to say, as Tony Bates notes in the review quoted earlier, "we used best practice; therefore it must be a quality programme." ¹⁵ A more rigorous approach to quality, I would argue, would be one that shifts the focus of attention from inputs to outputs. This offers, in the context of the developing world, an approach which might enable the building of more appropriate models. Models that would be based on a rigorous analysis of our practice in terms of the objectives our programmes were designed to address and not in terms of ODL self-referencing its own practice.

Measuring quality in terms of outputs some preconditions.

In closing I want to make a few comments about enhancing the quality of our learning as professionals that is predicated on the professional obligation to understand what we do and to use that intelligence for improvement. For I continue to be amazed that so much money can be ploughed into developing courseware, as quoted from the posting at the beginning of my talk, despite the piles of research that continually tells us that this is not sufficient. That learner support is vital. In order for us to achieve quality ODL outcomes we need to ensure an ODL community that is actively learning itself. To achieve this we need, among other things:

1. The management of ODL that encourages learning at the heart of the ODL operation. Strange to say, and this may be a bit of a provocative statement - educational bodies are not very good at learning. Educational leaders need to create a non blame environment where it is possible to ask difficult, analytical questions of both educational successes and failures.

2. ODL practitioners need the tools to research and evaluate what they do, so that improvements can continually be made, and knowledge shared. I am proud to say that The Commonwealth of Learning has developed just such materials, and they are freely available from our web site. The Practitioner Research and Evaluation Skills Training (PREST) ¹⁶ are a series of print-based materials (PDF files) designed to help Open and Distance Learning practitioners plan and conduct "principled and systematic" research and evaluation. The series design is based on the development (by the reader) of a locally relevant research project. Each of the core modules (Planning, gathering data, analyzing, reporting etc) is presented in a way that assists the reader to apply the new information to their own context. These materials are made available to our ODL community and we share them with ABED and hope that they will somehow be able to translate them into Portuguese and host them on the ABED web site as you have with some of our other materials.

Thank you.

References

Posting from COL virtual conference PCF 4 Learning referencing: Training Newsletter January 18, 2006 Brought to you by the editors of Training magazine.
See Zemsky, R and Massey W "Thwarted Innovation: What happened to learning and why? 2004 http://www.irhe.upenn.edu/WeatherStation.html
King, B "Refocusing quality: moving towards extreme evaluation." Paper presented at the 2005 Cambridge conference
Daniel J Higher Education, Teacher Education, Distance Education: What is Quality and Who says so? Lecture at Workshop on Materials Development for Quality Assurance in Higher Education Bangalore, India February 2006. http://www.col.org/colweb/site/pid/3565
SAIDE (South African Institute of Distance Education) Distance Education and Open Learning in Sub-Saharan Africa: Criteria and Conditions for Quality and Critical Success Factor 2004 (Commonwealth of Learning).Page 41.
Ibid page 47.
Harry K "Review of course materials from XXXXX Postgraduate Diploma in Distance Education (PGDDE) and Master of Arts in Distance Education (MADE) programmes" Commonwealth of Learning 2005.
Quality and Recognition in Higher Education: The Cross-border Challenge 2004 OECD Publishing.
The Distance Education Council web site: http://www.dec.ac.in/index.htm accessed 6th August 6th 2006.
Reddi U (6th August 2006) private correspondence with author.
Kader, A "Open Learning: Transforming Education for Development"
2002 Address at the Pan Commonwealth Forum on Open Learning, Durban South Africa.
Koul N and Kanwar A "Perspectives on Distance Education: Towards a Culture of Quality" (2006) Commonwealth of Learning
Passos R, e mail correspondence to author on how to implement sustainable ODL in Mozambique August 2006
See Bates T, review of Koul N and Kanwar A (2006) forth coming edition of Open Learning 2007 Routledge
Ibid
http://www.col.org/prest