Quantitative, Notional, and Comprehensive Evaluations of Spontaneous Engaged Speech

Authors

  • Garry Molholt
  • María Jose Cabrera West Chester University of Pennsylvania
  • V. K. Kumar West Chester University of Pennsylvania
  • Philip Thompsen West Chester University of Pennsylvania

DOI:

https://doi.org/10.11139/cj.29.1.1-23

Keywords:

Spoken Proficiency, Fluency, Quantitative Analysis, Notional Analysis, Comprehensive Analysis

Abstract

This study provides specific evidence regarding the extent to which quantitative measures, common sense notional measures, and comprehensive measures adequately characterize spontaneous, although engaged, speech. As such, the study contributes to the growing body of literature describing the current limits of automatic systems for evaluating spoken proficiency, provides examples of the essential nature of various notional and comprehensive variables, supports continued development of hybrid systems, and includes suggestions for the possible utilization of additional variables for automatic analyses. Data for this study were gathered and analyzed as follows. After 4 weeks of activities related to career development, 20 native English speaking college freshmen made recordings in English explaining their career preferences. Three experiments were then conducted. Experiment 1 analyzed the recordings according to current quantitative analyses used in fully automatic evaluations of fluency. Experiment 2 examined the recordings through a perception study according to common everyday notions of fluency. Experiment 3 analyzed the recordings according to an adaptation of the comprehensive rubrics used by the Educational Testing Service (ETS) for evaluating oral proficiency. The comprehensive evaluation (Experiment 3) provided the most insight, and temporal quantitative measures (Experiment 1) provided the least insight concerning the proficiency of the 20 speakers.

References

Amar, R. (2005). HUDAP manual. Jerusalem: The Hebrew University of Jerusalem.

Audhkhasi, K. (2009). Automatic evaluation of fluency in spoken language. IETE Technical Review, 26, 108-114.

Aylett, M. (2003). Disfluency and speech recognition profile factors. In R. Elkund (Ed.), Proceedings of the Disfluency in Spontaneous Speech Workshop (pp. 49-52). Goteborg, Sweden: Gothenburg Papers in Theoretical Linguistics.

Balogh, J. (2008). A case for automation in aviation English language assessment. Upper Saddle River, NJ: Pearson Education, Inc. Retrieved from http://www.pearsonassessments.com/NR/rdonlyres/121BCB12-7231-41C6-B051-D3BF98624D7A/0/WhitePaper_CaseForAutomation.pdf

Blake, R., Wilson, N., Cetto, M., & Pardo-Ballester, C. (2008). Measuring oral proficiency in distance, face-to-face, and blended classrooms. Language Learning & Technology, 12(3), 114-127. Retrieved from http://llt.msu.edu/vol12num3/blakeetal.pdf

Branigan, H., Lickley, R., & McKelvie, D. (1999). Non-linguistic influences on rates of disfluency in spontaneous speech. In C. Fossier-Lussier, S. Greenberg, & M. Slaney (Eds.), Proceedings of the Fourteenth International Congress of Phonetic Sciences San Francisco, California: ICPhS 1999 (pp. 387-390). Retrieved from http://www.ltg.ed.ac.uk/papers/99disfluency2.pdf

Brown, J. D. (2003). Promoting fluency in the EFL classrooms. In S. Yamashita, A. Howard, & C. Rinnert (Eds.), Proceedings of the 2nd Annual JALT Pan-SIG Conference (pp. 1-12). Kyoto, Japan: Kyoto institute of Technology.

Chalboub-Deville, M., & Wigglesworth, G. (2005). Rater judgment and English language speaking proficiency. World Englishes, 24, 383-391.

Cucchiarini, C., & Strik, H. (1999). Automatic assessment of second language learners’ fluency. In C. Fossier-Lussier, S. Greenberg, & M. Slaney (Eds.), Proceedings of the Fourteenth International Congress of Phonetic Sciences San Francisco, California: ICPhS 1999 (pp. 759-762).

Cucchiarini, C., Strik, H., & Boves, L. (2000). Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology. Journal of the Acoustical Society of America, 111, 2862-2873.

Cucchiarini, C., Strik, H., & Boves, L. (2002). Quantitative assessment of second language learners’ fluency: Comparisons between read and spontaneous speech. Journal of the Acoustical Society of America 117, 989-999.

Deshmukh, O., Kandhway, K., & Verma, A. (2009). Automatic evaluation of spoken English fluency. In Y. Ma (Ed.), Proceedings of the International Conference on Acoustics, Speech and Signal Processing (pp. 1-4). Taipei, Taiwan: ICASSP09.

Donald, L., & Cantor, D. (1990). Temporal and trait facets of personnel assessment. Applied Psychology: An International Review, 39, 413-429.

Elizur, D., & Sagie, A. (1999). Facets of personal values: A structural analysis of life and work values. Applied Psychology: An International Review, 48, 73-87.

Franco, H., Neumeyer, L., Digalakis, V., & Ronen, O. (1998). Combination of machine scores for automatic grading of pronunciation quality. Speech Communication, 30, 121-130.

Guttman, L. (1982). Facet theory, smallest space analysis, and factor analysis. Perceptual Motor Skills, 54, 491-493.

Hilton, H. (2008). The link between vocabulary knowledge and spoken L2 fluency. Language Learning Journal, 36, 153-166.

Holland, M., & Fisher, F. (Eds.). (2008). The path of speech technologies in computer assisted language learning: From research toward practice. London: Routledge.

KayPentax, (2009). MultiSpeech Software. Lincoln Park, NJ: Author.

Kondo, Y ., Tsutsui, E., & Nakano, M. (2008). Fundamental research on automatic speech evaluation of L2 English. In T. Koyama (Ed.), Proceedings of the 13th Conference of the Pan-Pacific Association of Applied Linguistics (pp. 85-86). Tokyo, Japan: Waseda University.

Molholt, G. (1988). Computer assisted instruction in pronunciation for Chinese speakers of American English. TESOL Quarterly, 22, 91-111.

Molholt, G. (1998). Accent reduction via acoustic analysis. Lincoln park, NJ: Kay Elemetrics Corporation.

Molholt, G., Cabrera, M., Kumar, K., & Thompsen, P. (2009, March). Correlating quantitative measures of speech with perceptions of fluency. Paper presented at the annual conference of the Computer Assisted Language Instruction Consortium (CALICO), Arizona State University, Tempe, AZ.

Molholt, G., & Hwu, F. (2008). Visualization of speech patterns for language learning. In M. Holland & F. Fisher (Eds.), The path of speech technologies in computer assisted language learning: From research toward practice (pp. 91-122). London: Routledge.

Molholt, G., Morgan, J., & Park, J. (2007, May). Implementation of automatic fluency checking using freely available toolkits. Paper presented at the annual conference of the Computer Assisted Language Instruction Consortium (CALICO), Texas State University, San Marcos, TX.

Molholt, G., & Pressler, A. (1986). Correlation between human and machine ratings of test of spoken English reading passages. In C. Stansfield (Ed.), Technology and language testing (a collection of papers from the Seventh Annual Language Testing Research Colloquium, held at ETS, Princeton, NJ, April 6-9, 1985) (pp. 111-127). Washington, DC: TESOL.

Neri, A., Cucchiarini, C., & Strik, H. (2002). Feedback in computer assisted pronunciation training: Technology push or demand pull. In E. Voorhees (Ed.), Proceedings of the International Conference on Spoken Language Processing (pp. 1209-1212). Denver, Colorado: Interspeech 2002.

Neri, A., Cucchiarini, C., Strik, H., & Boves, L. (2002). The pedagogy-technology interface in computer assisted pronunciation training. Computer Assisted Language Learning, 15, 441-467.

Richards, J. C., Platt, J., & Weber, H. (1985). Longman dictionary of applied linguistics. London: Longman.

Schlesinger, J., & Guttman, L. (1969). Smallest space analysis of intelligence and achievement tests. Psychological Bulletin, 71, 95-100.

Stansfield, C. (Ed.). (1986). Technology and language testing (a collection of papers from the Seventh Annual Language Testing Research Colloquium, held at ETS, Princeton, NJ, April 6-9, 1985). Washington, DC: TESOL.

US Patent No. 5634086. (1997). Method and apparatus for voice-interactive language instruction. Retrieved from http://www.patentstorm.us/patents/5634086/fulltext.html

VERSANT. (2006). Fully automated spoken English test. Retrieved from http://www.versant.jp/e_seminar.htm

VERSANT. (2008a). Delivery. Retrieved from http://www.ordinate.com/technology/delivery.jsp

VERSANT. (2008b). Scoring. Retrieved from http://www.ordinate.com/technology/scoring.jsp

VERSANT. (2008c). Validation. Retrieved from http://www.ordinate.com/technology/validation.jsp

Xi, X., Zechner, K., & Bejar, I. (2006). Extracting meaningful speech features to support diagnostic feedback: An ECD approach to automated scoring. M. Kolen (Ed.), Proceedings of the NCME Annual Meeting. San Francisco, CA: NCME. Retrieved from http://www.ets.org/Media/Research/pdf/speechrater_5.pdf

Xi, X. (2008). Investigating the criterion-related validity of the TOEFL speaking scores for ITA screening and setting standards for ITAs. (TOEFL iBT Research Report No. TOEFLiBT-03). Princeton, NJ: Educational Testing Service.

Zechner, K., Higgins, D., & Xi, X. (2007, May). SpeechRater: A construct-driven approach to scoring spontaneous non-native speech. Paper presented at the Annual Meeting of the International Speech Communication Association. Retrieved from http://www.ets.org/Media/Research/pdf/speechrater_2.pdf

Downloads

Published

2013-01-14

Issue

Section

Articles

How to Cite

Molholt, G., Cabrera, M. J., Kumar, V. K., & Thompsen, P. (2013). Quantitative, Notional, and Comprehensive Evaluations of Spontaneous Engaged Speech. CALICO Journal, 29(1), 1-23. https://doi.org/10.11139/cj.29.1.1-23

Most read articles by the same author(s)

1 2 3 4 5 6 7 8 9 10 > >>