Introduction
Curriculum evaluation has become of increasing interest to educators and curriculum planners since the 1960s (Richards 200 I), and since then it has embraced various aspects of language programs. Evaluation has not only been defined in terms of the purposes it serves, but it has also been given a central role by professionals and development organizations. This noticeable interest in evaluation is evidently due to the many benefits it offers like grater accountability, better decisions, and better sharing of lessons.
For the purpose of discussion in this paper, a definition of evaluation by Brown (1989) is adopted. He defines evaluation as: ‘the systematic collection and analysis of all relevant information necessary to promote the improvement of curriculum, and assess its effectiveness and efficiency, as well as participants’ attitudes within a context of particular institutions involved’ (cited in Weir and Roberts, 1994, p.4). Based on this particular definition, evaluation is going to be discussed in this paper as serving two broad purposes; ‘accountability’ and ‘development’. The next part of this paper gives an account of each of these two purposes of evaluation separately and then it looks at a broader approach to evaluation where it serves both purposes simultaneously. After that, the role of baseline studies is discussed with relation to each function of the evaluation process. Finally, some focus on the specific context of the writer is given.
1. Accountability-oriented evaluation:
This type of evaluation is primarily concerned with making decisions and/or determining the effectiveness and efficiency of programs and projects or any element of curriculum in general. It, then, informs decisions on whether to continue something or stop it. Therefore, the purpose here is not for improving the functioning of curriculum or classroom practice, but rather for assessing their worth or merits. Weir and Roberts (1994, p.4) state that ‘accountability refers to the answerability of staff to others for the quality of their work’. This type of evaluation is thus not normally motivated by teachers and staff implementing the project, but rather by some external authorities which have some kind of relation to the context being evaluated. Some of those will be policy makers and sponsors. Rea-Dickins and Gennaine (1992) see a link between power and evaluation for accountability which again confirms the above claim by Weir and Roberts. This link between evaluation for accountability seems to be really existing, in that the results of this evaluation help stakeholders take further decisions on programs and projects by means of comparing plans with actual events as well as by assessing achievement of objectives.
The audience for this type of evaluation is usually of a wider scope responsibilities and, therefore, would not normally be interested in details of the actual processes that takes place within the implementation stag. Rather, they wou1d be interested in some sort of summative account of the success or failure of the project or program that has been implemented. For this reason, in accountability-oriented evaluation the focus is usually on the overall outcomes and hence it is usually summative in nature, it usually examines the effects of a program or project at significant points or at its completion date. (Weir and Roberts 1994, p.5).
Data for accountability purposes may be collected on routine basis by project management or any other senior staff involved in the project. They can also be collected by an external evaluator on basis of periodic checks. (Weir and Roberts 1994, p.5).
Some limitations on accountability-oriented evaluation are worth mentioning here. First, this type of evaluation is summative in nature and therefore the actual processes that have taken place during the implementation stage may be ignored. Data collected from the implementation stage are very important in that they give explanations for the success or failure of the project or program and, hence, excluding these data in summative evaluations for accountability purposes makes it impossible to give rational explanations and justifications of their findings. The second limitation is related to the extrinsic orientation of these evaluations and that they follow an inspection approach. Weir and Roberts (1994) point out that in this type of evaluation only observed or measured phenomena (e.g. events and expenditure) may be applied as criteria of success whereas less readily measured phenomena (e.g. staff morale and attitudes) may be ignored or regarded as insufficiently reliable evidence with which to justify spending decisions.
2. Development-oriented evaluation
Unlike accountability-oriented evaluation, evaluation for the purpose of development aims at influencing improvements and changes to the curriculum. It is normally formative in nature, in that data are collected from different people( e.g. teachers, students ,and management) over a period of time.( Rea-Dickins and Germaine 1992, p,25).
Cronbah 1982 points out that this type of evaluation: ‘regards programme as fluid and seeks ways to better it. (cited in Weir and Roberts, 1994, p. 5). The formative nature of this evaluation allows it to discover the weak points as well as the strong ones, which all contribute to the improvement of the program or project being evaluated. ‘Formative evaluators …aim …to ensure that the program be implemented as effectively as possible. The formative evaluator watches over the program, alert both for problems and good ideas that can be shared’ (Morris and Fitz-Gibbon 1987, cited in Weir and Roberts 1994, P.5).
This type of valuation also does not usually depend much on tests, measurement and statistical data, but rather is more qualitative in nature. It tends to describe the actual situation being implemented and this entails various methods like meetings, self monitoring by students and teachers, classroom observations, informed conversations etc. (Sharp, 1990, p33 and Rea-Dickins and Germaine, 1994, p. 25).
A final issue to be discussed here about development-oriented evaluation is the issue of ‘who motivate this kind of evaluation?’. It is normally the intrinsic concerns of insiders that guide this evaluation unlike the case with the accountability-oriented evaluation. This is because it is the insiders themselves who usually feel the need to develop their ways of teaching, teaching materials and the curriculum in general. Therefore they seek ways to identify strengths which they can build upon as well as obstacles to progress; to work out and to realize more effective means to achieve desired objectives. Putting this evaluation into practice can, however, be either by internal or external evaluators, but is generally held to be more effective when carried out by a combination of both. (Weir and Roberts 1994, p.7).
3. A more comprehensive approach to evaluation
In the last two sections we have seen how evaluation could be carried out for the purpose of accountability and for the purpose of development. Throughout the discussion we have come across some limitations that could affect the objectivity of the evaluation when carried out for one distinctive function with the other function being excluded from the aim of such evaluation. In this section I will discuss critically how a broader approach for evaluation aims at serving both accountability and developmental functions.
This broader approach to evaluation would not only seek measurement of educational products, but also would explain the process and identify the reasons that have led the products turning the way they are. Weir and Roberts (1994) claim that: ‘it would satisfy contractual or bureaucratic (governmental) demands and produce information for programme or project improvement as well’ (p.8) In this way the two purposes of evaluation are not seen in conflict, rather they are seen as serving each other, which results in a more productive evaluation.
Weir and Roberts (1994), reporting on an evaluation project for a pre-sessional language program, show that formative evaluation was conducted during the course to facilitate the improvement of that course and also a summative test and questionnaire data were generated. The data were collected summatively to examine the value of the course from a bureaucratic perspective and were also used by the program director and teachers for taking educational decisions which relate formatively to subsequent programs. This example shows us how successfully the two functions of evaluation could be incorporated simultaneously.
The two constraints related to this approach are that it overlooks the actual process taking place in the implementation stage, and that the approach ignores the less readily measured phenomena.
In discussing the summative focus of accountability-oriented evaluation, I mentioned two constraints related to this approach; the first one was that this approach overlooks the actual process taking place in the implementation stage, and the second one stated that the approach ignores the less readily measured phenomena (e.g. Staff morale). It seems however that when summative evaluation is combined with a formative one, these two constraints will be resolved. The formative data will help understand how the results collected summatively came about. Additionally, the less readily measured phenomena could be examined through the many methods used in formative evaluation like observations, interviews, feedback from staff and students etc. This combination of formative and summative methods is however what the comprehensive approach to evaluation advocates.
As a result of the combination of both dimensions to evaluation, the roles of internal and external evaluators become of a cooperative nature. The external evaluators could offer help with regard to the formulation of evaluation designs for they have wider experience than insiders. They could also be involved in collecting formative information, other than that required for extrinsic accountability purposes. On the other hand, insiders can collect information to meet external accountability demands, other than collecting information for the more usual development purpose. (Weir and Roberts 1994, p.9).
The cooperative environment encouraged by this approach can forge stronger and more trusting relations between the bureaucracy, staff, and external evaluators, (Weir and Roberts 1994). The kind of evaluation in this approach is encouraged both intrinsically and extrinsically, which makes the insiders more likely to address the changes recommended by external evaluators because they have been part of the whole process of evaluation.
Furthermore, this approach has the potentiality of improving the professional competence and capabilities of local staff by engaging them in active work with external evaluators who have much experience and can share it with the less experienced insiders.
Although this approach is the most favourable one, it is still not free of constraints. I would like to divide these constraints into two major types; the degree of cooperation that external evaluators are ready to be involved in with insiders, and the degree of objectivity that insiders are ready to show in the formative evaluation. As far as the first one is concerned, there are situations (e.g. when the final product is considered valid basis for decisions) where the role of the external evaluator would be compromised by any prior involvement in formative studies (Weir and Roberts 1994), therefore one of the main roles of cooperation between insider and outsider evaluators (namely the help that external evaluators offer to insiders) is missing. There are also some situations where the information resulting from accountability-oriented evaluations is not easily made available to insider staff for formative purposes due to reasons related to financing and power considerations (Weir and Roberts 1994, p.9).
On the other hand, the formative evaluation by insiders is questioned for objectivity when it comes to the fact that the information obtained by them will be used for accountability purposes. They might cover some drawbacks of the program in order not to get into troubles in their careers.
The role of baseline studies
Information collected by baseline studies is considered valuable to the success of evaluation projects. Alderson (1992) points out that:
If the evaluation process is intended to estimate the impact of a project, then it is desirable to establish what things were like before the project begins. This necessitates the gathering of baseline data. Such data will need to relate to the predicted outcomes of the project. (cited in Tribble, 2000, p. 20).
Similarly Weir and Roberts (1994) emphasize the importance of baseline studies and regard it as a sign of a more systematic approach to evaluation. Baseline studies are important in both developmental and accountability focused evaluations and therefore offer great benefits to both the implementers of projects as well as to other stakeholders like funding or sponsoring bodies.
There are now several models of baseline studies and these are not restricted to the beginning of a project implementation. Murphy et al (1995) show that they can be undertaken at: ‘ the pre-planning stage, [...] at the planning stage, [...] and at some point in a project’s implementation. (cited in Tribble, 2000, p. 321) This particular classification will be adopted and discussed with relation to the evaluation process for both functions.
1. baseline studies at the pre-planning stage
Before planning the project itself, there seems to be a need to determine the feasibility of this project. Information on the feasibility of a project would be of great importance for the funding bodies of the project. They need to know the appropriateness of the project before they offer to sponsor it. This information would also be of vital importance to the implementers themselves. Bray and Luxon (1999) point out that in this stage the intention is to establish (1) whether or not the project is worth doing, (2) whether certain basic conditions for its feasibility exists, (3) whether there is an adequate framework (p.32). These studies then establish general view about the situation and conditions before the implementation stage.
2. baseline studies at the planning stage
While the previous type of baseline studies aims to assess the need for the project as a whole, baseline studies at the planning stage aim to establish a basis for future evaluation (Tribble, 2000, p. 319). They help narrow down the categories of information in which data will be collected in the future. ‘If evaluators are to determine what effect a programme has had, they need to establish what conditions were like immediately before it.’(Weir and Roberts, 1994, p.51). This type of baseline study seems to be the most popular one. Tribble (2000) sees it as the default form of baseline evaluations. A baseline study at this stage has both accountability and developmental functions. Bray and Luxon (2000) point out to the main functions of these studies as follows:
- To analyze the needs of a particular situation and a particular population.
- To collect information that can later be used to evaluate the impact of project activity and outputs at the project purpose level.
- To collect information that can be used to identify zones of innovation for project activity.
- To establish credibility for the project by demonstrating that it is based on sound knowledge on the context.
While the first and third functions concern development-oriented evaluation, the second and fourth ones concern accountability-oriented evaluation.
3. Baseline studies at the implementation stage
This third type occurs at some significant points in the project. They are also called ‘milestone’ baseline studies (Tribble 2000). They are usually conducted in longer- term projects that consist of several stages. Therefore, data is needed at the end of each significant stage which could be used for both accountability and developmental functions.
From the above account of the role of baseline studies, it has been clear that they offer great benefits to the process of evaluation. As far as accountability is concerned, they describe the project environment as it exists and help the implementers look at various dimensions of the project. This gives convincing qualitative and quantitative evidence of change and explains the success or lack of success of the project. On the other hand, baseline studies help find ways of implementing innovation in projects and specify the exact needs of certain contexts. Furthermore, they help implementers increase their ability of managing projects by starting with small-scale studies that give them both experience and courage to carry on larger projects.
Focus on the situation of my context
I work in The Language Center at Sultan Qaboos University, Oman. The Center is considered a very important and supporting body in the university and therefore its staff are of a high academic and professional expertise. The Language Centre is a partnership of students, instructors, coordinators and administrators dedicated to the development of student confidence and competence in English. Most staff are holders of MA and PHDdegrees plus three years of subsequent teaching experience which makes the center capable of carrying out local evaluation projects both for accountability and developmental functions. The current faculty number is over 170 , 40 Of which are Omanis. Professional development, creativity, and commitment in the teaching staff are all positively encouraged.
The Language Centre provides English instruction for students of all seven Colleges of the University. The goal is to prepare students for English medium courses in their subject areas and to prepare them to use English in their professions after graduation. The centre also uses a well-rounded curriculum of the four skill areas, the latest textbooks and teacher-prepared materials which are constantly reviewed and up-dated to direct students towards the acquisition of English.
Moreover, the center is dedicated to offer excellent English language programs to the staffs as well as to the students of the University. The Center’s curriculum unit is responsible for offering guidance and assistance on planning courses and choosing teaching materials for the various programs offered by the Center. The curriculum is comprehensively designed to develop the reading, writing, speaking and listening skills at an appropriate level of the student. Instruction is totally in English and high standards of academic achievements are promoted by attention to the individual needs of students. There are over 18.000 students at the university with over 5.000 registered in Language Centre courses. Teaching materials used in the Center include both published and in-house produced materials and both of these are frequently reviewed, developed, and updated.
The Language Centre has continuous assessment of student progress which includes internally developed testing, class assignments and observation, as well as a UK standardized test for final exit from the English intensive language programme into credit courses. All exams are made by the Language Centre and are a combination of proficiency and achievement.
Accountability-oriented evaluation seems to be the only type of evaluation considered in the Language Centre. Programme coordinators regularly observe teachers in each course and evaluate their performance. They are also responsible for handing a report at the end of each course in which they evaluate the efficiency of teaching and teaching materials. There is also another accountability evaluation that is conducted at the end of each semester by a special unit belongs to the administration of the University. In this type of evaluation, students are asked to evaluate both the teachers and the curriculum or the content of the course.
Personally, I think that it would be better to concentrate on both the reasons for not using developmental evaluation in the Language Centre and the factors that would help in the future to start using all types of evaluation. These are discussed as follows:
Difficulties of applying developmental evaluation
There are some reasons that make it extremely difficult to carry out developmental evaluation in the Language Centre. Perhaps the most obvious one that should be taken into account at this particular institution is the teaching and administrative load of the teachers. Although staff working on projects are released of some teaching hours, this could not be granted for all teachers participating in a way or another in these projects especially those who are involved in secondary tasks. Many teachers, seemingly, will not accept additional responsibility and complain that they are already involved in too many things to do. There is also the constraint of staff instability because most staff are expatriate and their job contracts are not always possible to renew. This might cause breakdowns in the evaluation process especially in long-term evaluations which might sometimes extend into years. Moreover, not all teachers in the Language Centre are familiar with this kind of evaluation and not all of them have the experience to carry it out.
Factors for using all sorts of Evaluation
As I mentioned above, the institution possesses many supporting points that enable it to conduct evaluation work for both accountability and developmental functions. The administration is very encouraging, supportive and well structured. There are several English language programs in The Center each is responsible for a particular college of the university’s seven colleges. Teachers of each programme are required to engage in preparing the syllabus and teaching materials for their respective programme. Their work normally requires them to do some kind of evaluation for the materials they prepare or even evaluation of course books suggested for the courses in the program. Each program, however, gets support from The Curriculum Development Unit, which is mainly responsible for improving curriculum in The Center and usually conduct evaluations in cooperation with different language programs. Furthermore, staffs who are usually involved with main roles in projects are usually released from some teaching hours so as to allow them some time to focus on these projects. These points are very helpful, in that they help in facilitating a suitable environment for carrying out evaluation projects in the institution.
Generally speaking, the work on evaluation projects is not very widely conducted in the center and it really needs to be encouraged more. At the meantime, evaluation is mostly restricted for the purpose of teaching material development and the accountability-oriented evaluation is the most common one. The other functions of evaluation like the developmental evaluation are not very popular if anyway exist. As an important institution in the university, the Language Center, however, needs to think of other policies with regard to evaluation so as to include its comprehensive functions more widely. Developmental evaluation should be introduced as soon as possible to the teachers of the Centre so that they would become more familiar with it and they would appreciate the results of such evaluation. As a starting point, teachers my be introduced to the idea of gradually evaluating their own performance, conducting some action research related to self- evaluation and trying out several new teaching methods in order to keep themselves up to date with all what is being innovated in the field of English language teaching and learning. It would be a great success if the teachers of the Language Centre adopt and acquire developmental evaluation as a habit.
Conclusion
The chief concern of this paper has been discussing critically the role of evaluation studies when conducted in a more comprehensive approach, that is when both functions; accountability and development are combined together in one evaluation process. We have seen how both formative and summative evaluations are present in this broader approach and how they serve each other. The issue of insider and outsider evaluators has also been raised and it has been made clear that the role of each of those is seen as supporting the other one and that combined work of the two results in a more productive evaluation. The role of baseline studies has also been discussed in terms of its relation to the evaluation process. The final part discussed some constraints as well as some positive factors concerning the situation in The Language Center at Sultan Qaboos University in The Sultanate of Oman.
References:
Bray, T. and T. Luxon. The role o/baseline studies in ELTprojects. In Ketmedy, C. 1999. p.32-39.
Kennedy, C. 1999. Innovation and Best Practice. Longman.
Rea-Dickins, P. and K. Germaine. 1992. Evaluation. Oxford, Oxford University Press
Richards, J. C. 2001. Curriculum Development in Language Teaching. Cambridge. Cambridge University Press.
Sharp, A. 1990. Staff/student participation in cou/:~e evaluation: a procedure/or improving course design. ELT Journal. 44/2: 132-37.
Tribble, C. 2000. Designing evaluation into educational change process. ELT
Journal. 54/4: 319-27
Weir, C and J. Roberts. 1994. Evaluation in ELT. Oxford. Blakwell Publishers
About the Author:
Rashid Al Maamari has a BA in English for English Specialists from Sultan Qaboos University (2001) and an MA in ESP from the University of Warwick (2003). He has been teaching English Language in the Language Centre at Sultan Qaboos University since 2001
Recent Comments