Title for TESOL 2022: Evaluating Lexical Resource of Spoken English With Vocabulary Profiling
Daniil M. Ozernyi, Soe Young Lee
English Vocabulary Profile (EVP) as a part of English Profile Project by Cambridge Assessment English (Harrison and Barker, 2015; Capel, 2012; Kurtes and Saville 2008) is a follow-up to T-series to draw corpus-based lines between lexicons of the CEFR levels. Beyond its informativity per se, EVP has been shown to correlate with results of subjective writing assessment (Leńko-Szymańska, 2015). It is only reasonable to assume that assessing speaking performance should follow the same pattern. The objective of this study is to establish whether vocabulary profiling can be used as a reliable measure of lexical resource of a student’s spoken performance. The topic is highly relevant and applicable for assessments at all and any scales.
The continuing experiment includes 30+ Ukrainian-Russian simultaneous bilinguals acquiring English as a third language with age range {14; 20}. The participants took a 10-minute speaking test, and were subsequently evaluated overall and specifically on Vocabulary (using a specifically developed rubric). The transcript of the speech produced by the students was profiled using EVP, and the results were analyzed for possible correlations with the two raters’ scores.
The results revealed that vocabulary profiling in its pure form is highly inapt to assess a student’s lexical resource. We report on a wide range of correlations and rankings (e.g. B2-level,%_types Kendall’s tau=.1682 with p=.2954 and B1-level,%_types Kendall’s tau=.2242 with p=.1604). We identify and analyze two invalidating factors: a) the speaker’s lexical mistakes taken to be correct speech by the profiler, and b) polysemy of lexical units of English which are not identified by the profiler correctly. The results of this study questions those of Leńko-Szymańska (2015) on writing, both skills (speaking/writing) being productive. Concluding, we speculate on how such factors as corpus size and other profiling measurements, including frequency-based ones (e.g. BNC/AWL/COCA, etc.) might have affected the results.
REFERENCES
- Capel, A. (2012). Completing the English vocabulary profile: C1 and C2 vocabulary. English Profile Journal, 3.
- Harrison, J., & Barker, F. (2015). English profile in practice (Vol. 5). Cambridge University Press.
- Kurtes, S., & Saville, N. (2008). The English Profile Programme – an overview. Research Notes, 33, 2–4.
- Leńko-Szymańska, A. (2015). The English Vocabulary Profile as a benchmark for assigning levels to learner corpus data. Learner Corpora in Language Testing and Assessment, 115–140.
Abstract (50w): This presentation reports on initial findings from an investigation into reliability of vocabulary profiling (VP) as a tool of evaluation of lexical resources of students’ spoken English. Findings revealed that because of learner’s lexical mistakes and polysemy, VP is highly unreliable. Implications for both large-scale and small-scale assessments are discussed.