Commonwealth of Learning - Quality Assurance in ODL

Introduction

Assuring the quality of education provision is a fundamental aspect of gaining and maintaining credibility for programmes, institutions and national systems of higher education worldwide.

Despite a long and generally successful track record, open and distance learning (ODL) is still required to prove that the quality of student learning is at least equivalent to face-to-face teaching. A comprehensive quality assurance (QA) system can help accomplish this.

QA is designed to prove and improve the quality of an institution's methods, and educational products and outcomes. In ODL this includes developing and producing learning materials, academic programmes, services and support, as well as standards of student learning.

A systematic and consistent QA system helps to establish an institution's good reputation and image. It includes defined standards of achievement, documented procedures for all identified processes, established ways of responding to issues and clear accountability for outcomes. The result is greater public confidence, more satisfied students, efficient processes and staff who are confident in their jobs. Students are more likely to experience better quality instruction, learning materials and interactions with the institution and its staff, leading to enhanced learning outcomes. Satisfied students are more likely to choose that institution again or to recommend it to others.

QA approaches

Common QA procedures for internal institutional audit include:

Self study or self evaluation.
Peer review by an expert panel.
Use of relevant statistical information and performance indicators, such as completion rates, overall grades and profitability.
Surveys of key stakeholders such as students, graduates and employers.

These are generally used in combination to provide a comprehensive set of data about what is occurring, and how successfully. Sound QA practices also combine internal self audit with external assessments. Professional QA bodies can monitor an institution's systems through external review. There are also independent or integrated education organisations that promote and support quality improvement, that may produce best practice guidelines or provide support networks for practitioners. Associations of education providers, such as the British Association of Open Learning, the Canadian Association of Distance Education and the Norwegian Association of Distance Education, establish voluntary standards and accept a code of practice.

The International Council on Open and Distance Education Standards Agency (ISA) conducts institutional audits leading to accreditation and provides advice on planning, costing and establishing new approaches and student support services.

International standards such as ISO 9000 may assist in assuring quality in distance education (DE). External controllers examine and interpret the quality of an institution's DE system, using documentation and guidelines. There are obvious benefits for an institution to having an external assessor judge the quality of what is done and in being able to claim external validation of quality. Other unanticipated benefits include greater staff awareness of quality, and enhanced cross-functional communication and collaboration.

Groups such as the Distance Education and Training Council (DETC) can contribute substantially to QA. DETC endorsement is viewed by the public as a recognition of quality performance standards. The DETC describe twelve standards covering all aspects of DE operations and policy; feedback about institutional performance against these standards can assist an institution in improving the quality of what it does.

From the early 1990's many governments adopted an explicit role in QA, outlining policies and practices and establishing national or regional agencies. In the United Kingdom and Argentina, government funding for education is also linked to quality. The focus of these national QA systems varies. In England and Scotland it is on teaching effectiveness, while in Hong Kong it is on management processes. In the United States, QA relies heavily on accrediting agencies such as the U.S. Department of Education and the Council for Higher Education Accreditation (CHEA), while Mexico emphasises research productivity.

The ODL context

Should QA in ODL environments be different from QA in conventional education? The DE guidelines of the Quality Assurance Agency for Higher Education (QAA), U.K., argue that DE quality needs to be assured in the same way as any other form of higher education. The experience of ODL organisations can guide conventional education providers about implementing QA in distributed learning environments.

To establish credibility and equivalence, QA in ODL must address the following organisational differences:

Larger number of stakeholders, specialists or sites involved in development and delivery.
Separation of learner and institution.
More separate activities, roles and participants.
More detailed planning of production and budgets in advance of programme presentation.
More administrative tasks distributed between different sites and partners.
More diverse target audiences.
Greater use of the Web and other information and communications technologies (ICTs).
More delegation of aspects of learning and assessment to local or workplace partners.
Distributed organisational structure for materials creation, teaching or learner support.
Different interpretation of what constitutes teaching, for example, a learner-based instead of a lecture-based approach.

Quality in ODL

Quality in ODL is frequently judged in terms of learning materials because student learning is at the centre of the ODL experience. Success depends on how effectively course production, delivery and student support sub-systems operate, underpinned by academic standards and management processes. However, a course is more than just the materials. It is the total experience of the learner, and quality systems must acknowledge this.

Not all ODL occurs in the same way, and new technologies have created new learning environments. The quality of the student experience and of student learning changes inherently with the learning context and a range of other variables. The kinds of QA systems that are eventually established may also depend on the size and nature of an organisation. Regardless, the underlying principles are similar: organising and managing quality in ODL is essentially about managing operational and academic standards, which means establishing those standards and ensuring their regular review.

A framework for managing ODL quality should address:

General philosophy: Policy and mission statements, ethos and culture of the organisation, mottoes, attitudes of staff and levels of staff commitment.
Products: Learning materials, courses, resources, media, outputs (progression and retention rates, number of graduates), assessment outcomes ( pass rates, standards of performance).
Services: Registration and advisory services, tutoring, counseling, feedback and guidance on learning, support for learner progress, provision and management of study centres and resources, customer service, ICT help desks, responsiveness to issues.
Support processes: Delivery systems, record keeping, scheduling, electronic backup, warehousing and stock control, QA procedures.

Getting started

A quality QA system should:

Be systematic.
Provide fundamentals of practice.
Be manageable so that it people will use it.
Be integrated.
Allow scope for individual initiative and professional judgment.

QA is an approach to organising work that ensures:

The institution's mission and aims are understood by everyone.
Systems are well designed, foolproof and communicated to everyone.
Everyone's responsibilities are clear and understood.
The institution's view of quality is defined and documented.
Systems are in place to monitor performance.
There are established methods for correcting problems.

QA in a new institution

It is essential that the institution fully endorse QA and that senior management show leadership by example, through commitment to quality. To address institutional quality, many institutions appoint a QA leader or "champion," form a quality group or instruct managers of functional work areas to work together.
Start with reviewing the institution's mission statement. It fundamentally describes the educational experience the institution aims to provide, the students it serves, its values and how it intends to achieve the standards it has set for itself.
Develop a university policy on quality that defines the institution's goals and standards and that reflects the values and principles of its mission statement.
Ensure that this policy is documented, endorsed by the most senior management and communicated to all staff.
Develop an operational plan that describes the processes that will monitor quality and performance indicators, and that assigns responsibility for implementation.

The following activities will contribute to the operational plan:

Identify, analyse and map critical functions. Describe the steps in all procedures, workflow between units or areas and standards of performance-this applies to production processes as well as to academic programme development.
Document and communicate processes and procedures to everyone concerned.
Set performance indicators or standards of performance, such as turnaround time for student assignments and technical standards for materials production.
Identify the time frame for performance review and reporting.

In developing a QA system for the first time it is useful to look at what others have done, to provide a starting point for researching best practice examples and benchmarks of achievement. These external reference points can help establish well-documented operational and academic standards, within a framework shaped by the university's mission. A new QA system should also have built-in monitoring and revision plans.

QA in an established institution

Established QA systems may experience slippage over time. It is also important to ensure that systems and processes keep up with new practices and technologies.

Review ODL or DE operational plans with the following questions in mind:

What do we say we do (what products or services are offered)?
How well do we do it (what are the standards of performance)?
How do we know (what monitoring techniques are used)?
What do we do to improve (what is the response to problems or shortcomings)?

Keep records of the answers to these questions, to form the basis for institutional self-review:

Establish the reasons for taking a particular quality approach before developing procedures. This may be done through meetings, working parties or as directed by management. The success of QA will depend on how well staff understand the rationale for it.
Map and review existing QA practices. This is frequently done based on outlining functional areas of responsibility, as described above.
Identify, map and analyse processes and operations within each functional area. Mapping makes processes explicit and clear to everyone, and identifies gaps, overlaps and cross-functional issues that need to be resolved.
Document processes or revise existing documentation.
Evaluate performance against existing standards and establish new standards if necessary.
Develop a plan for managing processes and set dates for future reviews.
Form cross-functional goal groups to identify persistent problems and to develop solutions.

QA considerations

Curriculum and instruction

Specify academic standards and quality. What procedures will ensure these are met? Specify the educational processes, learning experiences and resources that will be used; describe the student achievements and capabilities the standards are meant to reflect. How will student learning be assessed? Refer to external reference points such as qualifications frameworks, benchmarks and codes of practice. Use internal and external subject and course reviews that draw on a range of assessment criteria.
Establish standards for ensuring quality in programme design, approval and review procedures. This may include internal and external peer review of learning materials, moderation of assessment and course evaluation.
Establish a cycle of regular component and overall programme review and monitoring, using a variety of data.
Establish and schedule mechanisms and processes for evaluating the effectiveness of teaching, learning and assessment strategies. Reviews are frequently scheduled at three to five year intervals and staggered across programmes to distribute workload.
Specify how programme delivery will be managed, including who is accountable for doing what.
Develop strategies for monitoring student progress, and establish processes that ensure the student has a reasonable chance of achieving academic success.
Establish quality controls that include regular monitoring and review of the performance of local agents, tutors or assessors, and online tutors or markers.

Staff support

Train all ODL staff-such as teachers and course designers, local or online tutors, advisors and administrators-in QA procedures.
Provide teaching staff with training in educational design and pedagogy, ICT use, and policy and procedures. For example, it is important that staff are aware of institutional policies about assessment of student learning, plagiarism and return of student assignments, as well as how to optimise use of email and online learning management systems. Teachers and tutors should also be aware of the support services available to students, and of how students can access them.
Train all staff in the relevant aspects of student management systems, and in relevant procedures and policy. This can be done face-to-face, online or through self-paced modules. Ensure consistency by using standard training methods procedures manuals.
Specify the qualifications and experience required of local administrative and teaching staff, and establish reliable systems for maintaining communication.

Student support

Establish standards of performance, and document procedures for student support. For example, how are student enquiries managed? Specify response times, responsibilities and associated record keeping needs.
Document the nature and content of student and staff records, and assign responsibility for tracking them.
Establish procedures for monitoring and tracking student registration and for materials dispatch and delivery. Specify standards of performance that account for different conditions; for example, expected delivery times for different locations may vary.
Establish policy, procedures and timelines for student
assignment submission and return. Include tracking,
monitoring and reporting processes.
Establish timelines and procedures for developing learning materials, including printing, production and delivery. Develop timelines for multimedia and e-learning resource development.
Establish, communicate and monitor standards for student access to tutors.
Specify student support services, including locations, scope, standards of service and accessibility.
Establish procedures for the scheduling, notification, administration and conditions of examinations, along with processes that ensure the integrity of examinations and marking. For example, will an external invigilator be used? Who would be acceptable for this? How will exams be kept secure? Will all ODL students be required to sit for exams?
Schedule regular monitoring and review against specified performance indicators.

Student Outcomes

Establish acceptable rates of student progression and retention.
Develop mechanisms for monitoring student progress and strategies for identifying and responding to problems.
Ensure that students can progress, by examining scheduling, calendars, mode of offering and pre-requisites.
Develop strategies to monitor standards of performance within subjects and across courses and programmes, identifying external benchmarks or standards.
Develop, distribute and monitor policies on assignment submission, grading practices and grade distribution.

Implementing QA

Whether QA is being established in a new institution, or existing practices are being refreshed, there are a number of pre-requisites:

An institutional quality policy and plan.
A QA "champion" who will lead the QA effort and ensure it is embedded in the institutional culture and practice.
Opportunities for all staff to contribute to the design of a QA system. This increases the sense of ownership and ensures that strategies are meaningful, realistic and more likely to be adopted.
An effective communications strategy that ensures everyone knows their own and other's responsibilities, understands why a QA system is being implemented and what should occur. It also allows stakeholders to identify aspects of QA that may have been missed or that are not working.
Mechanisms that encourage good practice will maintain staff interest and involvement in improving quality. For example, exposure to examples of good practice from other sub-systems or work units through meetings or newsletters can stimulate thinking about new quality practices, and encourage collaboration and integration.
A range of staff development strategies designed to improve quality, where possible linked to staff performance review. Reward structures can also encourage staff commitment to QA.

Case Studies

QA for new activities: The University of New England , U.S. (UNE, www.une.edu )

UNE has a long and successful DE history. Staff are experienced in teaching students at a distance. It has well-established procedures for developing, producing and dispatching DE learning materials. However, the growth of e-learning created a need for new procedures and associated QA systems.

The university adopted an institutional approach to e-learning QA, featuring:

A university policy on online teaching that specifies an online learning management system.
Classification and description of Web use levels for different courses.
A high level of technical, administrative and pedagogical
support for staff.
Basic training for all online teaching staff.
A team approach to e-learning materials development, comprising an educational designer, teacher, academic developer, university copyright officer, programmer and graphic designer.
Procedures for e-course development that follow university approval guidelines, including submission to the Academic Board.
Clear designation of online courses in the university handbook and all other public documents, to ensure students are aware of technology requirements before enrolling.
Complete design and technical verification of e-learning materials before teaching begins, including editorial checks for technical problems such as broken links.
Course materials approval and "sign off" by the academic supervisor before going live.
Extended help desk and phone support hours for staff and students.
Service agreements with the university's Division of Information Technology designate e-learning a mission critical activity, ensuring problems are resolved immediately.
Annual benchmarking of e-learning procedures against similar universities, and nationally through the Australasian Council on Open, Distance and E-learning (ACODE).
Online course review on a three-year cycle, using criteria equivalent to print course reviews.
Regular external peer review of selected online courses.

QA in a new university: Open University Malaysia (OUM, www.oum.edu.my )

As the first private university in Malaysia, and the first to offer open and distance learning there, OUM needed to show the public that it provided educational products, experiences and outcomes of ensured quality that were at least equal to that delivered by public on-campus universities.

While OUM's quality management system addresses key performance areas of corporate culture, improving and monitoring quality in curriculum and courseware development are at the heart of its QA effort.

OUM adopted a strategy of putting the customer first and providing quality services and products, featuring:

Identified best practice standards in ODL management through benchmarking visits to renowned international open universities.
Academic standards for programmes set by external regulators, ensuring quality and equivalence.
Well-developed procedures for courseware development. To ensure viability and that courses will meet client needs, market survey is conducted before any academic programme is developed.
First drafts of any curriculum formulated by a team comprising deans of faculties, lecturers and experts from professional bodies.
Feedback on curriculum drafts by a Board of Studies, comprising experts from academia, government and industry. Drafts are then scrutinised by the university's Academic Board, comprising senior faculty members and academic stakeholders.
Curriculum approval from the Private Education Department and National Accreditation Board at the Ministry of Education.
Development of learning materials by a team of subject matter experts, academic moderators, module writers, tutors and consultants.

References

Butterfield, S., Chambers, M., Moseley, B., Prebble, T., Uys, P. and Woodhouse, D. 1999. "External quality assurance for the virtual institution." AAU Series on Quality 4. New Zealand Universities Academic Audit Unit (AAU).
www.aau.ac.nz

CHEA Institute for Research and Study of Accreditation and Quality Assurance. 2002. "Accreditation and Assuring Quality in Distance Learning." CHEA Monograph Series 1.
www.chea.org/pdf/mono_1_accred_distance_02.pdf?pubID=246

Dhanarajan, G. 1997. Globalisation, Competitiveness and Open and Distance Education: Reflections on Quality Assurance. Speech presented at the Eleventh Annual Conference of the Asian Association of Open Universities, Kuala Lumpur, Malaysia.
www.col.org/colweb/site/pid/3704

Harun, H. 2004. The Challenge of Managing Quality at the Open University of Malaysia. Ninth International Conference on ISO 9000 and Total Quality Management, Bangkok, Thailand.

Institute for Higher Education Policy. 2000. Quality on the Line: Benchmarks for Successful Internet Based Distance Education.
www.ihep.com/Pubs/PDF/Quality.pdf

Middlehurst, R. 2001. "Quality Assurance Implications of New Forms of Higher Education, Part 1: A Typology." ENQA Occasional papers 3. Helsinki, Finland: European Network for Quality Assurance in Higher Education. www.enqa.eu/files/newforms.pdf

Middlehurst, R. and Campbell, C. 2003. "Quality Assurance and Borderless Education: finding pathways through the maze." Observatory Report 17. Observatory on Borderless Higher Education.
www.obhe.ac.uk/products/reports

Monash University Secretariat. 1997. Policy and Procedures for Quality Assurance in Off Campus Learning and Open Learning Programs.
www.adm.monash.edu.au/unisec/pol/acad15.html

Observatory on Borderless Higher Education. 2003. "Quality Assurance in Borderless Higher Education: six initiatives." Observatory Briefings 11.
www.obhe.ac.uk/products/briefings.html

Perraton, H. and Hulsmann, T. 1998. Planning and Evaluating Systems of Open and Distance Learning. Report to the Department for Education and Employment by the International Research Foundation for Open Learning. www.col.org/irfol

Quality Assurance Agency for Higher Education. 1999. Guidelines on the quality assurance of distance learning.
www.qaa.ac.uk

Reid, C. N. and Robertshaw, M. 1991. "The Quest for Quality, East and West." In Atkinson, R., McBeath, C. and Meacham, D., eds. Quality in Distance Education: ASPESA Forum 91. Paper presented at the Tenth Biennial Forum of the Australian and South Pacific External Studies Association.
www1.worldbank.org/disted/Management/Governance/q-03.html

Robinson, B. 2003. "Governance, accreditation and quality assurance in open and distance education." In Perraton, H. and Lentell, H., eds. Policy for Open and Distance Learning: World review of distance education and open learning 4. Vancouver: Commonwealth of Learning/RoutledgeFalmer. www.col.org/worldreview

Robinson, B. 1995. "The management of quality in open and distance learning." In Indira Gandhi National Open University, Structure and Management of Open Learning Systems. Proceedings of the Eighth Annual Conference of the Asian Association of Open Universities, New Delhi, India.
www1.worldbank.org/disted/Management/Governance/q-01.html

WEBSITES

Australian Universities Quality Agency (AUQA). www.auqa.edu.au

Australasian Council on Open, Distance and E-learning (ACODE).
www.acode.edu.au

Canadian Association for Distance Education (CADE). www.cade-aced.ca

Council for Higher Education Accreditation (CHEA). www.chea.org

Distance Education and Training Council (DETC). www.detc.org

European Network for Quality Assurance in Higher Education (ENQA).
www.enqa.eu

Hong Kong Council for Academic Accreditation (HKCAA). www.hkcaa.edu.hk

International Council on Open and Distance Education (ICDE): ICDE Standards Agency (ISA). www.icde.org

International Network for Quality Assurance Agencies in Higher Education (INQAAHE). www.inqaahe.org

International Organization for Standardization (ISO). www.iso.org

New Zealand Universities Academic Audit Unit (NZUAAU). www.aau.ac.nz

Norwegian Association of Distance Education (NFF/NADE).
www.nade-nff.no

Praxiom Research Group Ltd.: ISO 9000. www.praxiom.com

Quality Assurance Agency for Higher Education (QAA). www.qaa.ac.uk

Spanish National Agency for Quality Assessment and Accreditation (ANECA). www.aneca.es

U.S. Department of Education: Accreditation. www.ed.gov/admins/fi

Quality Assurance in Open and Distance Learning

Written and researched by Denise Kirkpatrick, Director, Learning and Teaching, Monash University, Australia