English Language Honours AE, NIE, 2003
The process involved in the rating of performance tests of oral proficiency is not well understood, despite a growing body of qualitative research into raters' decision making of second language speaking proficiency over the past decade (Douglas, 1994; Brown, 1995; Lazaraton, 1996; Brown & Hill 1998, Pollitt & Murray, 1996; Meiron & Schick, 2000; Brown, 2000; Orr, 2002).
The purpose of this study is to investigate the decision-making process of raters involved in the rating of the Picture Conversation task used in Singapore primary schools. Both quantitative and qualitative analyses were used in this study. The quantitative analysis involved examining the scores awarded to the candidates by three trained raters. These raters then retrospected and thought aloud on how they reached a final decision in their rating of the three candidates' oral performances. Retrospective Verbal Protocol Analysis (RVPA) was used to elicit data from the three raters. This data was then coded and analysed both quantitatively and qualitatively.
Results indicate that the same candidate's performance elicited different judgements from the trained raters, despite the use of uniform band descriptors. It was also found that raters heeded a wide range of non-criterion information as compared to criteria found within the band descriptors. In addition, raters tended to draw on their own frames of reference and personal criteria during their decision-making process.
These findings could offer important implications for future rater training, and for revising or developing the wording of the band descriptors.