**3. Changes in vision of the project team: Responses to formative evaluation of the programme during the initial funding phase**

#### **3.1 Evaluation design**

54 International Perspectives of Distance Learning in Higher Education

However, a number of shifts took place from this initial conception of a programme originally designed to implement an approach which had been tried and tested elsewhere in the world. Shifts initially took place from a purely measurement-based summative evaluation design, towards a more contextual, holistic, formative and multimethod approach to programme evaluation (Potter, 1993a; 1993b). The programme's behaviourist assumptions were also challenged (Leigh, 1995; Perraton, 2000), being replaced with a more responsive and process approach to curriculum development and evaluation (OLSET, 1995;

This section provides an overview of the implementation of the South African Radio Learning programme as outlined in its funding contract with USAID. The programme's brief was to offer direct support to the South African Department of Education in improving the quality of teaching in primary schools. This would be achieved by targeting the development of English language competencies in the junior primary phase, through the

After an initial pilot (Leigh, 1992), 118 radio lessons were scripted in 1993 to support the teaching of English at Grade One level. These were implemented with 14 500 learners in five regions of South Africa in 1993 (Potter & Leigh, 1995; Leigh, 1995). This was followed in 1994 by the development of 130 radio lessons at Grade Two level. The lessons were of half

As radio broadcasting in South Africa at the time was controlled by the national broadcaster (the SABC), initial implementation of the programme took place through the medium of audio-cassettes. These were packaged in cardboard boxes and delivered to the participating schools in each region by a regional coordinator, together with a radio tape-recorder and other classroom materials. These included a teacher's manual and printed classroom posters and alphabet friezes to support the radio lessons. As the tapes and materials were developed in stages over the year, the regional coordinator was able to establish ongoing

The regional coordinator was also able during her visits to the schools, to act as adviser on matters relating to programme implementation. Her roles included observing lessons, suggesting ways of supporting the programme at classroom level, as well as ways of linking the programme and other facets of the curriculum. Participating teachers were encouraged in the process to make contact with each other, and to form teacher support groups. Workshops on use of the programme and on primary school teaching were also organized regionally, and then implemented through the school visiting and teacher support group

By the end of 1994 when the initial funding grant provided to the programme by USAID came to an end, a network of teacher and school support had been set up across five provinces of South Africa, centred around materials distribution, teacher support groups and in-service training workshops to support the teaching of English at lower primary school level had been established. At the Grade One level, 118 radio lessons and supporting

personal contact with the teachers and principals of the participating schools.

Potter, Dube, Kenyon et al., 1995).

medium of interactive radio.

structures.

These changes are traced in the initial sections which follow.

**2. Initial implementation of the programme** 

an hour duration, and were recorded on audio-tapes.

Like its predecessors using interactive radio in other parts of the world, the South African Radio Learning Programme was subject to evaluation for developmental and accountability purposes. The initial evaluation design was conceptualised as based on measurement involving development of a test of English language vocabulary which would be applied in pre and post-testing of learners in a number of schools involved in the programme. The performance of these learners would then be compared with pre- and post-test performance of learners in a similar number of contrast schools not involved in the programme. If learning gains could be demonstrated using audio-cassettes, this evidence would then be used to make the case to the national broadcaster to provide a national radio footprint to support national implementation of the programme.

At the time of the time of the programme's inception in 1992, a number of other projects in South Africa were also working to develop procedures for teaching English at the lower primary level (such as the Molteno project; MAPEP) and at the upper primary level (such as SELP; TELIP; TOPS; the Molteno Project; READ). In a context of rapid change in South African in its transition to democracy, the evaluation design was criticized as insufficiently responsive to context of South African schools, as well as the changes occurring in South Africa on both a socio-political and educational level. A broader contextually based evaluation design was then developed (Potter, 1993a), in which pre-and post-testing would form one element or strand. The design was evolved consultatively and responsively (Stake, 1983), and addressed six questions through use of multiple data sources and multiple methods (Potter, 1993a; 1993b), as follows:

Is the "English in Action" programme effective in teaching primary English?

Are teachers who use the programme empowered, supported in their jobs and assisted in professionalisation?

 Is there acceptance of the programme by the community, inclusive of teachers, parents, principals and other stakeholders?

Teacher Development Through Distance Education:

developed by the programme had promise.

evaluation (OLSET, 1995; Potter, Dube, Kenyon et al., 1995). .

Grade Two children (Potter, Arnott, Hingle et al., 1995).

innovation would take place.

(OLSET, 1995).

Contrasting Visions of Radio Learning in South African Primary Schools 57

was necessary. At the same time, however, the evaluators commented that the infrastructure

Specific criticisms were that, while individual members of the project team were highly skilled in the technical side of materials writing and production, these activities were taking place in the absence a coherent policy framework as regards language teaching. There were similar gaps with respect to the teacher development side of the project's work. In both areas, there was a need for the development of a guiding programme theory, which would provide the necessary framework within which the work of the staff, and the overall

The project team's response to an unfavourable initial evaluation was to focus on both teaching and learning issues, and to change its view of curriculum from a focus on the skills development of learners, to include focus on the professional development of their teachers in the context of school and classroom. This implied a focus on issues relating to classroom teaching and learning relative to teacher development (OLSET, 1995). As these involved major shifts within the programme's vision and focus, recommendations were made for ongoing internal, formative and process evaluation, in addition to external and summative

In their second interim report in 1994, the evaluators were thus able to report that the programme had moved to address the conceptual and curricular issues raised in the initial 1993 report. This was evident both in the setting up of mechanisms in the programme for staff to meet and discuss the form of the materials and their usage, as well as in the appointment of a number of persons with qualifications and experience in second language teaching to assist with the development of the classroom side of the programme's work.

The teacher development side of the programme was also being addressed through the appointment of a teacher development coordinator to provide conceptual direction to the school and teacher support provided by the programme. The programme was thus developing an increasingly coherent curricular framework, and had also produced evidence of learning gains at school and classroom levels, both with respect to Grade One as well as

For the reasons that these changes still needed to be implemented and their viability tested, the evaluators cautioned in their final evaluation reports (Potter, 1994a; Potter, Arnott, Hingle et al., 1995) that, despite evidence of significant learning gains favouring project learners at both Grade One and Grade Two levels (Arnott, Mansfield, & Mentis, (1993b; Arnott, Mentis, & Mansfield, 1994), both the radio learning project's classroom and inservice training curricula were not ready for wider implementation. It was suggested that what was required at the next stage should involve revision of the materials base, and implementation of the existing materials via various delivery media (e.g. via radio; via cassettes) in line with the open learning principles recommended in the project's blueprint

This would need to be accompanied by evaluation of the support processes and transactions necessary to successful implementation with teachers and in schools (Potter, Dube, Kenyon et al., 1995). It was thus envisaged that the development of both the classroom-based and teacher support curricula would need to take place simultaneously with revision of the

project's existing materials base and ongoing materials evaluation (OLSET, 1995).


Table 1. Focusing Questions used in the 1993-1994 Formative Evaluation of the South African Radio Programme

To answer the evaluation questions, evidence from a number of sources was utilised. Qualitative data were gathered from school visits, classroom visits and observation of lessons, the reports of teachers, principals and parents concerning pupil progress, observation of teacher support groups, interviews with project staff and with teachers, and narrative case studies of the programme in the schools (Potter, Arnott, Mansfield, et al., 1993). Focus groups were also conducted in all regions in which the programme operated, involving a range of community stakeholders (Nene, 1993).

Quantitative data were also used to answer the evaluation questions, based on pre- and post-testing of learners (Arnott, Mansfield, & Mentis, 1993a; 1993b) conducted in 71 schools (36 project and 35 matched comparison schools), and yielding evidence of significantly better performance on English language tests favouring the project schools (F = 118,31 (70); p < ,0001; R = 0,39). In addition, an economic analysis was conducted, projecting the costs of radio learning in relation to different scales of implementation in terms of learner numbers, and comparing these data with alternative forms of instruction (Cobbe, 1994).

#### **3.2 Issues raised by the evaluators**

In their initial review of the programme in 1993, the evaluators focused on both product and process issues (Hamilton, 1976; Stenhouse, 1975; 1980; 1981, Walker, 1969; 1971). They reported that after a shaky start with respect to credibility, the programme had been successful in establishing an infrastructure and a body of materials which could form the basis for future work. The intervention was also regarded positively by both teachers and parents. However, it lacked a coherent conceptual and curricular framework in terms of both the classroom and teacher support sides of its work. Many of the staff also lacked qualifications and experience in second language teaching, which reflected in the approach to and type of materials being developed (Potter, Arnott, Mansfield et al., 1993).

Despite quantitative evidence of advantage among learners in the project schools in 1993, it was evident that there were major gaps both in the way in which the project's curriculum was being conceptualized, as well as a lack of relationship of many of the activities included in the radio lessons with the South African lower primary school curriculum. Recommendations were made that the team involved in the radio learning project should place emphasis on the development of a coherent theoretical, policy and curricular framework, which could guide its future work.

The initial evaluation report (Potter, Arnott, Mansfield et al., 1993) was critical of the programme's assumptions, its conception of curriculum, its behaviourist view of learning, and the technicist vision of educational change implicit in an attempt to improve education through increasing the English language skills of learners in the absence of focus on the context of school and classroom. A broader vision based on both product and process issues

Are radio and cassette efficacious as a delivery medium?

involving a range of community stakeholders (Nene, 1993).

implementation?

African Radio Programme

**3.2 Issues raised by the evaluators** 

framework, which could guide its future work.

Is "English in Action" having an effect on the school environment?

Is the programme cost-effective, and are there economies of scale for national

Table 1. Focusing Questions used in the 1993-1994 Formative Evaluation of the South

To answer the evaluation questions, evidence from a number of sources was utilised. Qualitative data were gathered from school visits, classroom visits and observation of lessons, the reports of teachers, principals and parents concerning pupil progress, observation of teacher support groups, interviews with project staff and with teachers, and narrative case studies of the programme in the schools (Potter, Arnott, Mansfield, et al., 1993). Focus groups were also conducted in all regions in which the programme operated,

Quantitative data were also used to answer the evaluation questions, based on pre- and post-testing of learners (Arnott, Mansfield, & Mentis, 1993a; 1993b) conducted in 71 schools (36 project and 35 matched comparison schools), and yielding evidence of significantly better performance on English language tests favouring the project schools (F = 118,31 (70); p < ,0001; R = 0,39). In addition, an economic analysis was conducted, projecting the costs of radio learning in relation to different scales of implementation in terms of learner numbers,

In their initial review of the programme in 1993, the evaluators focused on both product and process issues (Hamilton, 1976; Stenhouse, 1975; 1980; 1981, Walker, 1969; 1971). They reported that after a shaky start with respect to credibility, the programme had been successful in establishing an infrastructure and a body of materials which could form the basis for future work. The intervention was also regarded positively by both teachers and parents. However, it lacked a coherent conceptual and curricular framework in terms of both the classroom and teacher support sides of its work. Many of the staff also lacked qualifications and experience in second language teaching, which reflected in the approach

Despite quantitative evidence of advantage among learners in the project schools in 1993, it was evident that there were major gaps both in the way in which the project's curriculum was being conceptualized, as well as a lack of relationship of many of the activities included in the radio lessons with the South African lower primary school curriculum. Recommendations were made that the team involved in the radio learning project should place emphasis on the development of a coherent theoretical, policy and curricular

The initial evaluation report (Potter, Arnott, Mansfield et al., 1993) was critical of the programme's assumptions, its conception of curriculum, its behaviourist view of learning, and the technicist vision of educational change implicit in an attempt to improve education through increasing the English language skills of learners in the absence of focus on the context of school and classroom. A broader vision based on both product and process issues

and comparing these data with alternative forms of instruction (Cobbe, 1994).

to and type of materials being developed (Potter, Arnott, Mansfield et al., 1993).

was necessary. At the same time, however, the evaluators commented that the infrastructure developed by the programme had promise.

Specific criticisms were that, while individual members of the project team were highly skilled in the technical side of materials writing and production, these activities were taking place in the absence a coherent policy framework as regards language teaching. There were similar gaps with respect to the teacher development side of the project's work. In both areas, there was a need for the development of a guiding programme theory, which would provide the necessary framework within which the work of the staff, and the overall innovation would take place.

The project team's response to an unfavourable initial evaluation was to focus on both teaching and learning issues, and to change its view of curriculum from a focus on the skills development of learners, to include focus on the professional development of their teachers in the context of school and classroom. This implied a focus on issues relating to classroom teaching and learning relative to teacher development (OLSET, 1995). As these involved major shifts within the programme's vision and focus, recommendations were made for ongoing internal, formative and process evaluation, in addition to external and summative evaluation (OLSET, 1995; Potter, Dube, Kenyon et al., 1995). .

In their second interim report in 1994, the evaluators were thus able to report that the programme had moved to address the conceptual and curricular issues raised in the initial 1993 report. This was evident both in the setting up of mechanisms in the programme for staff to meet and discuss the form of the materials and their usage, as well as in the appointment of a number of persons with qualifications and experience in second language teaching to assist with the development of the classroom side of the programme's work.

The teacher development side of the programme was also being addressed through the appointment of a teacher development coordinator to provide conceptual direction to the school and teacher support provided by the programme. The programme was thus developing an increasingly coherent curricular framework, and had also produced evidence of learning gains at school and classroom levels, both with respect to Grade One as well as Grade Two children (Potter, Arnott, Hingle et al., 1995).

For the reasons that these changes still needed to be implemented and their viability tested, the evaluators cautioned in their final evaluation reports (Potter, 1994a; Potter, Arnott, Hingle et al., 1995) that, despite evidence of significant learning gains favouring project learners at both Grade One and Grade Two levels (Arnott, Mansfield, & Mentis, (1993b; Arnott, Mentis, & Mansfield, 1994), both the radio learning project's classroom and inservice training curricula were not ready for wider implementation. It was suggested that what was required at the next stage should involve revision of the materials base, and implementation of the existing materials via various delivery media (e.g. via radio; via cassettes) in line with the open learning principles recommended in the project's blueprint (OLSET, 1995).

This would need to be accompanied by evaluation of the support processes and transactions necessary to successful implementation with teachers and in schools (Potter, Dube, Kenyon et al., 1995). It was thus envisaged that the development of both the classroom-based and teacher support curricula would need to take place simultaneously with revision of the project's existing materials base and ongoing materials evaluation (OLSET, 1995).

Teacher Development Through Distance Education:

acquisition of such skills" (Buckland 1984, 383).

the apartheid era as:

and post-testing.

Arnott, Mansfield et al., 1993).

Contrasting Visions of Radio Learning in South African Primary Schools 59

Commission in the early 1980's (Buckland, 1984). Buckland draws on the work of Apple (1979) and Giroux (1981) to characterize technicism as applied in South African education in

"The treatment of social issues as technical problems open to technical solutions, the componential thinking implicit in the division of education into eighteen separate areas, the ahistorical approach and the orientation to the future all serve to rule out consideration of how schooling has been used to further political and economic ends. Knowledge is seen in terms of skills, with an 'appropriate value system' serving to legitimate the differentiated

Buckland's analysis is pertinent to the behaviourist conception of the South African Radio Learning Programme's curriculum between 1992 and 1994, as well as the shifts in conception of curriculum which took place in the programme at the end of its first funding phase (OLSET, 1995). The assumptions of the project planners in 1992 can be characterized as technicist in the sense that the South African Radio Learning Project was initially planned with a focus solely on the form and content of the radio lessons, and their role in developing the skills of learners. It was assumed that English would be improved using principles of distributed learning in scripts applying principles of educational technology (Rowntree, 1974; 1982) with judgement of programme success being based on evaluation through pre

The problem with this vision was its simplistic and mechanistic logic (Stenhouse, 1975; 1980). Essentially, despite evidence in 1993 and 1994 that learners in project schools made greater learning gains than learners in non-project schools, the programme was heavily criticised by the evaluators for its behaviourist and technicist assumptions, the application of similar content to the Kenyan scripts by the scriptwriters without reference to the South African primary school curriculum or the needs of teachers who had borne the brunt of the apartheid struggle, and the project team's acontextual way of working as evidenced by lack of contact

Given the conflict which had taken place in schools and in South African education over an extended period prior to the project's inception in 1992, the need was for a coherent intervention strategy based on the needs of teachers and schools over South Africa's transition to democracy. The programme's intervention model, however, did not focus on issues relating to the teaching of English as a second language. Nor was there mention of the skills of teachers, and the need for in-service training and support of teachers (Potter,

In response to these criticisms, curriculum development took place in the project, and the programme was remodeled. The project team developed a curriculum blueprint at the end of 1994 (OLSET, 1995), reflecting very different assumptions. The project team was guided through the process by an experienced facilitator with considerable experience in teaching English as a second language in South African schools, who was versed in the literature on second language teaching and the bilingual classroom (e.g. Langhan, 1992; Willig, 1985), and who was able to juxtapose the needs for support and empowerment of teachers against a

The curriculum document developed through this process (OLSET, 1995) reflected a vision of educational change as not only skills-based but person-centred. To undertake teacher

view of education focused on skills development in learners (MacDonald, 1990).

with others working in South African education (Potter, Arnott, Mansfield et al, 1993).

Even at this early stage in the project's development (end 1994), the evaluators also noted that there was overwhelming evidence that teachers were supportive of "English in Action", and had used the materials to the benefit of their teaching, and their learners. With respect to in-service training of the teachers, there was value in the teacher support group structure developed by the project. With respect to the evaluation, teacher support groups had provided observational, focus group and case study data, suggesting a potential for interactive radio as a framework for teacher, school and community development, as well as wider possibilities for educational technology in curriculum development in the primary school. The teacher support groups also provided the project with a broad-based and country-wide basis for developmental work directed at both networking and empowerment (Potter, Dube, Kenyon et al., 1995).
