Abstract

English L2 speech evaluation test results can have important impacts on future education, employment, and immigration opportunities (Munro, 2008). However, the use of human raters in speech evaluation introduces the possibility of rater bias stemming from their backgrounds (e.g., Carey et al., 2011; Isaacs & Thomson, 2013; Winke et al., 2013). This study involved 203 listeners with different background factors rating speech samples from 44 English L2 speakers (31 Japanese L1, eight Thai L1, and five Bengali L1). The listener background factors examined included the listener's L1 (English, Japanese, and Other L1s), previous teaching experience, location (Canada or Japan), and sex. The speech samples were rated on a nine-point scale for seven aspects of speech (overall comprehensibility, fluency and coherence, lexical resources, grammatical range and accuracy, segmental accuracy, chunking, and intonation). Many facet Rasch measurement (MRFM) analyses found significant bias/interaction effects between the speaker's L1 and the four listener background factors examined for all aspects of speech except for lexical resources. In addition, listeners were asked to identify which aspect of speech was the easiest and the most difficult to rate and give a reason for their choice. The qualitative responses were coded, and response frequencies were compared by listener background factors. The results showed that listeners with teaching experience were significantly more likely than those without experience to state that lexis was easy to rate. This presentation will discuss the reasons given by raters regarding what makes lexis either easy or difficult to rate in a speaking test and the implications for English L2 speech evaluation by human raters.