Second International Conference on
Quality Assurance in Higher Education in Africa
Dar es Salaam, Tanzania
17-19 September 2007
Can Invitational Institutional Audits make a
Cost-Effective Contribution to Quality Enhancement?
Willie Clarke-Okah & John Daniel
Commonwealth of Learning
Abstract
Most national quality assurance agencies conduct quality audits of higher education institutions in their jurisdictions. This paper draws lessons from institutional audits that were carried out on the initiative of universities themselves, either in anticipation of a future audit by the national QA agency (UNISA, South Africa) or as a catalyst for reform (University of Ghana). These audits provided useful guidance but their direct and indirect costs were considerable (at least $100,000 in out-of-pocket expenses and many hours of institutional staff time) because they involved large international teams of experts. Can such audits be made more cost-effective whilst retaining their credibility? The paper suggests four measures for doing this.
First, most QA agencies require self-assessments to be conducted prior to audit visits using set procedures. These should also be carried out before invitational audits under procedures agreed by the institution and its audit team and should include any student or graduate survey data. Where self-assessment is a novel experience, this process has the additional benefit of providing a basis for ongoing QA structures and processes.
Second, the institutional authorities (Vice-Chancellor/President; Governing Board Chair) must be committed to the audit and the audit team must understand the context for reform and any major national or institutional constraints. The institution should make experienced staff available as a secretariat for the audit team.
Third, an invitational audit will likely highlight significant issues of governance. Audit teams must be equipped to analyze them and institutional leadership be ready to address them.
Fourth, in inviting members to join the audit team, the institution must choose both local and international figures likely to be perceived within the institution as both credible and objective. The team should include (or be able to call on) expertise in academic programmes, governance & management, and infrastructure & resources.
If this model is followed it should be possible to provide the institution with a cost-effective audit on the basis of a visit of a week or less by a team of not more than six members.
Introduction
We are delighted to be here and would like to thank the organizers of this conference for giving us the opportunity to share with you some of the lessons that we have learnt in our recent involvement in two invitational institutional audits, one a trial institutional quality audit to prepare Unisa for the real thing to come in 2008, which will be conducted by South Africa's Higher Education Quality Committee (HEQC). The other, at the University of Ghana, was an exercise undertaken partly to help the institution recover from problems stemming from an excessive increase in student numbers and a major breakdown in examination security but also, more generally, to help it chart a path for the future.
But, first, a quick backstory about the place of higher education in priority sub-sector financing by governments in recent times: at the behest of the donor agencies in the 1990's, higher education in many parts of the developing world took a beating as resources were heavily deployed in favour of basic education, particularly at the primary school level. Higher education institutions (HEIs) in countries that were under military rule, as Ghana was for a number of years, were already starved (of resources) to near extinction as many experienced faculty voted with their feet and left their institutions for greener pastures elsewhere, facilities and labs deteriorated, supplies were in short supply, library stocks were badly outdated, and very importantly, the quality of provision at the HE level suffered badly, so much so that institutions that were once highly respected internationally for their rigour and quality were now prime suspects in, and perceived to be, churning out barely literate graduates. What flowed from the donor-inspired neglect of the sub-sector were badly wounded and weakened institutions, putting human capital development, vital for national development, at serious risk. For the few international intergovernmental organizations like COL that stayed the course in supporting higher education, it was a lonely battle.
However, the good news is that there is now a renewed focus by governments and HEIs themselves on re-energizing and revitalizing higher education to meet the growing need of mass higher education and, importantly, national development goals and objectives. There is a growing appreciation of the fact that higher education is the strategic heart of education, not only because it trains the teachers, but because it generates the ideas that illuminate curricula and pedagogy. Increasingly, HEIs are recognizing the imperative of ensuring quality while making access a priority. It is likely, therefore, that in future we will be seeing more invitational institutional quality audits.
In this paper, we look at the three phases of the institutional audit process - the pre-audit phase, the audit phase and the post-audit phase - and outline the lessons that we have drawn from them.
Pre-Audit Phase
Before the actual audit takes place, several steps are put in place to ensure that the resulting audit is successful. As a first step, the reasons for the audit and who should conduct it must be clear and the project must be fully embraced by the head of the institution and its senior management with their support communicated to the rank and file members of the institution.
The second step is to designate a unit (a Quality Assurance unit if one exists or a strategic planning unit) to coordinate and oversee the audit process.
Thirdly, using set procedures, various operational and academic units need to conduct self-assessments that are not a happiness dance but an open and frank detailing of a unit's mandate, goals and objectives, what it actually does and has accomplished against stated objectives, resources available to it, its plans and aspirations for the future, and what its major challenges are. Such assessments can throw up some surprising lacunae. At the University of Ghana, for example, some departments reported that they had "no idea" how many students they had. These self-assessments are then synthesized and produced in a binder or book form, ready for the auditors.
In self-assessing, the more reflective, analytical, evaluative and self-critical an institution is, as opposed to being defensive and merely descriptive in a self-serving manner, the more productive the audit is likely to be. It also could save time and money as the auditors do not spend more time than is necessary chasing after one detail or the other, not to mention the possibility that good self-assessments could forestall any tendency toward excessive state intervention in the affairs of the university. In this respect, the Unisa self-assessments were painfully frank and, needless to say, very helpful.
For institutions with a limited experience in self-assessments, the burden of proof is usually on the institution and it is costly as the institutions need to produce volumes and volumes of documentation. Over time, however, with experience in the use of existing internal processes, the scope of critical self-analysis can be reduced by sheer simplification - thereby simultaneously reducing costs and that burden of proof.
Selection of the audit team is very important. It could be done by the host institution or by a trusted outside body. The composition of the audit team depends on the range of activities that it is being called upon to examine and interrogate and the knowledge, skills and experiences required to cover the audit scope. Getting highly experienced people to conduct the audit does help to cut down the costs that are associated with training. It helps if some of the external members already have some familiarity with the country and the institution. Members who have experience of higher education in a range of universities and even in a number of countries are particularly valuable in countering the tendency to propose solutions from a single foreign institution. Finally, the team must be able to function and work as a team, led by a chairperson who is capable of maintaining focus and discipline at all times.
Next come the briefing sessions. In the Unisa case, the audit team met and were briefed over a day and a half by the host institution at a central location - in Cambridge, England - with team members coming from Canada, France, Hong Kong, India, Nigeria, South Africa and the UK. Prior to assembling in Cambridge, the team members had been provided with a formidable array of documentation - via e-mail and in hard copies in two hefty binder volumes - on the University's strategic plan; policies covering tuition, assessment, research, e-communications, HIV/AIDS, language, research grants, recognition of prior learning and so forth; student satisfaction surveys; operational plans and audit guidelines established by national bodies.
The pre-audit briefing was useful in two respects: it helped to build a team spirit among the audit team members, some of whom were meeting the others for the first time, and it afforded them the opportunity to ask questions and receive answers in real time. However, in retrospect, we feel that savings could have been had if we had dispensed with the briefing meeting in Cambridge and conducted what needed to be communicated by e-mail and by faxes and teleconferencing when/if necessary. The team could also have met at the field site a day or two before the audit began to clarify any matters that needed to be clarified. If a field site briefing is not possible, the briefing should be conducted as suggested no more than 3-to-4 weeks before the audit begins; the time lag between when the briefing in the Unisa case was held (March) and when the audit was conducted (June) was too long.
The University of Ghana team had no prior briefing meeting, but had the advantage of making two week-long visits to Ghana at an interval of four months. Much of the first week was spent on visiting parts of the institution, meeting staff and launching the self-assessment process. The second visit focused on developing analyses and recommendations and testing them on key stakeholders such as the deans and the University Council.
The audit teams for the two institutional audits were large: 11 members for the University of Ghana audit and 9 for Unisa's. In hindsight, and in the perspective of using this mechanism more widely, these numbers are too large to be really cost-effective. Some four or five people, carefully selected, with a well-defined audit scope, would normally be sufficient to accomplish the task and enjoy considerable savings in cost.
The Audit Phase
The trial audit was held over five working days at the main campus of Unisa. A building complex was dedicated to the audit and Unisa's strategic planning office set up a temporary office at the complex, overseen throughout the process by its executive director. Meals and refreshments were provided on site - in the complex. And a war room was set aside for the audit team to deliberate in confidence. This room and one other had a wall-to-wall shelf of binders containing descriptions of course offerings at the University, available for the team to examine at their discretion. Several other rooms in the complex were reserved, which allowed multiple and back-to-back interview sessions to take place simultaneously. Scribes were available to assist with note-taking and reporting.
The audit team had a heap of material to go through and did put in quite easily 15-hour days. Even then, there was not enough time to sift through, in a substantive way, all the documentation and supporting material in order to conduct proper audit trails. Their work would have been eased considerably if they had given thought to, and prepared in advance, questions to ask - at least to cover the first day of the audit interviews.
Then, there is the issue of sheer numbers. The team interviewed over 400 unduplicated interviewees, ranging from Alumni to Council members. While this may seem overwhelming, it is essential preparation for Unisa. However, the feedback process was not as effective as it should have been. Staff members were debriefed via web link. Since the Unisa secretariat, the audit team's war room and the interview sessions were under one roof, as it were, a group just interviewed could have been brought immediately after an interview into a room by the Unisa secretariat staff to find out what transpired, what went wrong and what went well. This could then be fed into groups yet to see the audit team. It would also have been a useful feedback to the audit team to measure how well they were doing.
One way to avoid large numbers of interviewees is to rely more on selected audit trails than on documented portfolios of evidence or self-assessments. If this is properly done, the audit team may not need to see a particular group of interviewees. Experience has shown that this approach could lead to a reduction in costs by as much as 50%.
The team visiting the University of Ghana divided itself into three sub-groups, reflecting the three headings of the mandate from the University, which was: to review Governance, Management and Administration; Academic Operations; and Infrastructure and Resources. This worked very well both in allowing the team to multiply its efforts and in giving members of the team specific tasks to get their teeth into. However, the full group met every day, at least over lunch and usually in plenary sessions to compare notes.
The Post-Audit Phase
Immediately following the audit, it is useful to hold a debriefing meeting with the Vice-Chancellor and his team of senior executives to provide them with a a high-level summary of the audit team's findings. This gives the University's senior management an opportunity to share with the team their preliminary response to the findings before the team disperses.
For the institution just audited to maintain momentum in the post-audit period, it is important that the audit report, written according to an agreed or prescribed format, be issued within 9-to-10 weeks of the completion of the audit. This has the merit of allowing all those directly engaged in the audit process to remain focused and to follow up on the areas recommended for improvements. In an environment in which senior management is highly committed to change or reform, as was our experience with Unisa, action can be taken to effect improvements even before the final draft of the report is issued.
Rather than having every team member write bits and pieces of the report, it is advisable that one person be designated to do the writing, particularly in producing the first draft. For the Unisa audit report, this function was performed by the chairperson of the audit team who maintained an overall perspective on the audit process. The same approach was used by the Ghana team.
There may be issues that need follow-up and it is wise for an institution, as a contingency measure, to make financial provision for special additional visits. In our Ghana experience, this approach was instrumental in getting external specialist help in sorting out the finances of the University.
Conclusion
Invitational institutional audits can indeed make a cost-effective contribution to quality enhancement at the levels of governance, management and program delivery. Our experience has led us to conclude that the following major considerations are helpful in making cost-effective decisions:
- The larger an audit team is, the higher the cost of the audit is likely to be; a team size of about 5 people, composed of experienced local and international experts, is sufficient to conduct a successful audit. Although the teams for both the Unisa and University of Ghana audits were quite large, they came with very impressive credentials;
- Pre-audit briefings of an audit team are less expensive when conducted electronically or undertaken just prior to the start of the audit;
- If and wherever possible, in preparing and providing documentation for an audit, good practice suggests the use of existing documentation. Audit teams demanding non-readily available documentation could be costly for the institution;
- Reflective, analytical and self-critical assessments not only lead to more productive audits, they save money as well;
- Self-assessments can be used for both internal quality assurance and external accreditation processes, resulting in considerable savings;
- Housing the audit operation in a dedicated office complex for the entire duration of the audit is a very cost-effective measure;
- Institutional audits can be helpful in uncovering significant issues of governance and management, which, if left unattended, could become even more costly for the institution. Committed leadership is required here - to implement needed changes;
- For institutions just starting out on their QA journey, audits of the kind under review do stimulate and enhance the basis for on-going QA structures and processes.
On that happy note...ladies and gentlemen, thank you!
