Using Automatic Speech Recognition Technology with Elicited Oral Response Testing
DOI:
https://doi.org/10.11139/cj.29.4.601-618Keywords:
Speaking Assessment, ASR, Elicited Imitation, Sentence RepetitionAbstract
This study examined the use of automatic speech recognition (ASR) scored elicited oral response (EOR) tests designed to predict speaking ability, as well as the relationship between ASR-scored EOR and other language proficiency measures and the ability of the ASR to rate speakers without bias to gender and native language. To that end, 179 subjects were given an ASR-Scored EOR Test with 60 items followed by an oral interview and a battery of other language tests. Findings suggest that ASR-scored EOR results could be used alone to predict speaking ability in specific situations and for limited purposes such as initial placement of students in language training situations. However, if more certainty should be required, adding a listening component would improve the assessment. Analysis of the study results also suggests that differences in gender and native language had no significant effect on ASR-scoring of EOR. While EOR is not an authentic performance assessment of the specific construct in questions, it does have high reliability, good content validity and seems to be a practical estimate of speaking proficiency that could be used for initial placement of students in situations where speaking is not currently being assessed.
References
Bachman, L. F., & Palmer, A. S. (1997). Language testing in practice. New York, NY: Oxford University Press.
Benzeghiba, M., De Mori, R., Deroo, O., Dupont, S., Erbes, T., Jouvet, D., …Wellekens, C. (2007). Automatic speech recognition and speech variability: A review. Speech Communication, 49(10-11), 763-786.
Bernstein, J., Van Moere, A., & Cheng, J. (2010). Validating automated speaking tests. Language Testing, 27(3), 355-377.
Berry, P. B. (1976). Elicited imitation of language: Some ESNS population characteristics. Language and Speech, 19(4), 350-63.
Bley-Vroman, R., & Chaudron, C. (1994). Elicited imitation as a measure of second-language competence. In M. Tarone, S. Gass, & A. Cohen (Eds.), Research methodology in secondlanguage acquisition (pp. 245-261). New York, NY: Routledge
Bond, T. G., & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
Brown, D. H. (2004). Language assessment: Principles and classroom practices. White Plains, NY: Pearson/Longman.
Buck, K., Byrnes, H., & Thompson, I. (1989). The ACTFL oral proficiency interview tester training manual. Yonkers, NY: ACTFL.
Chapelle, C. A., & Chung, Y. R. (2010). The promise of NLP and speech processing technologies in language assessment. Language Testing, 27(3), 301-315.
Chapelle, C. A., & Douglas, D. (2006). Assessing language through computer technology. Cambridge, UK: Cambridge University Press.
Chiu, T. L., Liou, H. C., & Yeh, Y. (2007). A study of web-based oral activities enhanced by automatic speech recognition for EFL college learning. Computer Assisted Language Learning, 20(3), 209-233.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum.
Coombe, C. A., Folse, K. S., & Hubley, N. J. (2007). A practical guide to assessing English language learners. Ann Arbor, MI: University of Michigan.
Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. The Behavioral and Brain Sciences, 24(1), 87-114; discussion 114-85.
Cucchiarini, C., Neri, A., & Strik, H. (2009). Oral proficiency training in Dutch L2: The contribution of ASR-based corrective feedback. Speech Communication, 51(10), 853-863.
Derwing, T. (2008). Curriculum issues in teaching pronunciation. Phonology and second language acquisition. Amsterdam, The Netherlands: John Benjamins Publishing Company.
Ellis, R., & Barkhuizen, G. (2005). Analysing learner language. New York, NY: Oxford University Press.
Erlam, R. (2006). Elicited imitation as a measure of L2 implicit knowledge: An empirical validation study. Applied Linguistics, 27(3), 464-491.
Eskenazi, M. (1999). Using automatic speech processing for foreign language pronunciation tutoring: Some issues and a prototype. Language Learning & Technology, 2(2), 62-76.
Flynn, S. (1986). Production vs. comprehension: Differences in underlying competencies. Studies in Second Language Acquisition, 8(2), 135-164.
Fulcher, G. (2003). Testing second language speaking. New York, NY: Pearson Education.
Gallimore, R., & Tharp, R. G. (1981). The interpretation of elicited sentence imitation in a standardized context. Language Learning, 31(2), 369-392.
Graham, C. R., Lonsdale, D., Kennington, C., Johnson, A., & McGhee, J. (2008). Elicited imitation as an oral proficiency measure with ASR scoring. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC 2008) (pp. 1604-1610). European Language Resources Association.
Graham, C. R., McGhee, J., Sanchez-Tenney, M., & LeGare, M. (2011 June). Examining the validity of an elicited imitation instrument to test oral language in Spanish. Presentation at 33rd Annual Convention of the Language Testing Research Colloquium (LTRC 2011). Ann Arbor, MI, USA.
Hamayan, E., Saegert, J., & Larudee, P. (1977). Elicited imitation in second language learners. Language and Speech, 20(1), 86-97.
Hatch, E., & Lazaraton, A. (1991). The research manual: Design and statistics for applied linguistics. New York, NY: Heinle & Heinle.
Henning, G. (1983). Oral proficiency testing: Comparative validities of interview, imitation, and completion methods. Language Learning, 33(3), 315-332.
Hudson, T., & Clark, M. (2008). Designing sorting hats: Foreign language placement processes. Case Studies in Foreign Language Placement: Practices and Possibilities, 1, 1-6.
Knapp, T. R. (1990). Treating ordinal scales as interval scales: An attempt to resolve the controversy. Nursing Research, 39(2), 121-3.
Lamere, P., Kwok, P., Gouvêa, E., Raj, B., Singh, R., Walker, W., Warmuth, M., & Wolf, P. (2003). The CMU SPHINX-4 speech recognition system. In IEEE international conference on acoustics, speech and signal processing. Hong Kong.
Luoma, S. (2004). Assessing speaking. Cambridge, UK: Cambridge University Press.
Markman, B. R., Spilka, I. V., & Tucker, G. R. (1975). The use of elicited imitation in search of an interim French grammar. Language Learning, 25(1), 31-41.
Matsushita, H., Lonsdale, D., & Dewey, D. (2010). Japanese elicited imitation: ASR-based oral proficiency test and optimal item creation. In G. R. S. Weir & S. Ishikawa (Eds.), Corpus, ICT, and language education (pp. 161-172). University of Strathclyde Publishing.
Millard, B., & Lonsdale, D. (2011). French oral proficiency assessment: Elicited imitation with speech recognition; Selected proceedings from LSRL 2011. Manuscript submitted for publication.
Miller, G. A. (1956). The magical number seven plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81-97.
Naiman, N. (1974). The use of elicited imitation in second language acquisition research. Working Papers on Bilingualism, 2, 1-37, 0319-5171.
Norman, G. (2010). Likert scales, levels of measurement and the "laws" of statistics. Advances in Health Sciences Education: Theory and Practice, 15(5), 625-32.
O’Shaughnessy, D. (2008). Automatic speech recognition: History, methods and challenges [invited paper]. Pattern Recognition, 41(10), 2965-2979.
Okura, E., & Lonsdale, D. (2012). Working memory’s meager involvement in sentence repetition tests. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society (pp. 2132-2137). Austin, TX: Cognitive Science Society.
OPI Tester Certification. (2012). Retreived March 14, 2012, from http://www.actfl.org/i4a/pages/index.cfm?pageid=3350
Pickett, J. M., & Morris, S. R. (2000). The acoustics of speech communication: Fundamentals, speech perception theory, and technology. The Journal of the Acoustical Society of America, 108, 1373.
Rypa, M. E., & Price, P. (1999). VILTS: A tale of two technologies. Calico Journal, 16(3), 385-404.
Scott, M. L. (1994). Auditory memory and perception in younger and older adult second language learners. Studies in Second Language Acquisition, 16(3), 263-281.
Selinker, L. (1972). Interlanguage. International Review of Applied Linguistics in Language Teaching, 10(1-4), 209-232.
Slobin, D., & Welsh, C. (1968). Elicited imitation as a research tool inn developmental psycholinguistics. Working Paper 10. Reprinted in C.A. Ferguson & D.I. Slobin (1973). Studies of child language development (pp. 485-497). New York: Holt, Rinehart, and Winston Inc.
Tomita, Y., Suzuki, W., & Jessop, L. (2009). Elicited imitation: Toward valid procedures to measure implicit second language grammatical knowledge. TESOL Quarterly, 43(2), 345-350.
van den Noort, M. W. M. L., Bosch, P., & Hugdahl, K. (2006). Foreign language proficiency and working memory capacity. European Psychologist, 11(4), 289-296.
Vinther, T. (2002). Elicited imitation: A brief overview. International Journal of Applied Linguistics, 12(1), 54-73.
Wachowicz, K. A., & Scott, B. (1999). Software that listens: It’s not a question of whether, it’s a question of how. CALICO Journal, 16, 253-276.
Xi, X. (2010). Automated scoring and feedback systems: Where are we and where are we heading? Language Testing, 27(3), 291-300.
Zechner, K., Higgins, D., Xi, X., & Williamson, D. M. (2009). Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10), 883-895.
