Test Takers' Experiences with Computer- administered Listening Comprehension Tests
Interviewing for Qualitative Explorations of Test Validity
DOI:
https://doi.org/10.1558/cj.v21i2.339-371Keywords:
Computerized Language Testing, Test Validation, Measurement Error, Qualitative Research MethodsAbstract
In this study, retrospective interviews were used to investigate reliability (and thus validity) threats to a computerized ESL listening comprehension test administered at a university in the US. The participants in the investigation, six international graduate students, were asked to respond to semi- and open-ended questions during individual interviews. Their responses were used to confirm a model of potential sources of systematic and random error variance and to generate ideas about unanticipated sources of systematic and random error variance for further investigation. The results suggested that the specific retrospective interview methodology proposed in the report was useful, dependable, and credible in generating broad based, yet focused information specific to the use of a particular test. Based on the results, further areas of research into the inescapable and complex issues of test-taker attributes as they interact with test method and conditions were suggested.
References
Alderson, J. C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press.
American Council on Education (1995). Guidelines for computerized-adaptive test development and use in education. Washington, DC: Author.
American Educational Research Association (1999). Standards for Educational and Psychological Testing. Washington, DC: Author.
Bachman, L. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
Bachman, L., & Palmer, A. (1996). Language testing in practice. Oxford: Oxford University Press.
Bahr, M. W., & Bahr, C. M. (1997). Educational assessment in the next millennium: Contributions of technology. Preventing School Failure, 41, 90-94.
Banerjee, J., & Luoma, S. (1997). Qualitative approaches to test validation. In C. Clapham & D. Corson (Eds.), Encyclopedia of language and education Volume 7: Language testing and assessment (pp. 275-287). Dordrecht, The Netherlands: Kluwer Academic Publishers.
Bowers, D. A., & Bowers, V. M. (1996). Assessing and coping with computer anxiety in the social science classroom. Social Science Computer Review, 14, 439-443.
Bradshaw, J. (1990). Test-takers’ reactions to a placement test. Language Testing, 7 (1), 13-30.
Brown, A. (1993). The role of test-taker feedback in the test development process: Testtakers’ reactions to a tape-mediated test of proficiency in spoken Japanese. Language Testing, 10 (3), 277-303.
Brown, J. D. (1996). Testing in language programs. Upper Saddle River, NJ: Prentice Hall Regents.
Brown, J. D. (1997). Computers in language testing: Present research and some future directions. Language Learning & Technology, 1 (1), 44-59. Retrieved from http://llt.msu.edu/vol11num1/BROWN/default.html
Chou, C. (2000). Constructing a computer-assisted testing and evaluation system on the World Wide Web—The CATES experience. IEEE Transactions on Education, 43 (3), 266-271.
Cohen, A. (1994a). English for academic purposes in Brazil: The use of summary tasks. In C. Hill & K. Parry (Eds.), From testing to assessment: English as an international language (pp. 174-204). London: Longman.
Cohen, A. (1994b). Assessing language ability in the classroom. Boston: Heinle & Heinle.
Cohen, A. (1997). Towards enhancing verbal reports as a source of insights on test-taking strategies. In A. Huhta, V. Kohonen, L. Kurki-Suonio, & S. Luoma (Eds.), Current developments and alternatives in language assessment: Proceedings of the LTRC 96 (pp. 339-365). Jyvaskyla, Finland: University of Jyvaskyla.
Committee on Professional Standards and Committee on Psychological Tests and Assessment (1986). Guidelines for computer-based tests and interpretations. Washington, DC: American Psychological Association.
Corbel, C. (1993). Computer-enhanced language assessment. Sydney, Australia: The National Centre for English Language Teaching and Research.
Culligan, B., & Gorsuch, G. (1999). Using a commercially produced proficiency test in a one-year core EFL curriculum in Japan for placement purposes. The JALT Journal, 21 (1), 7-28.
Davies, A. (1984, June). Computer-assisted language testing. CALICO Journal, 41-42, 48.
Davies, A., Brown, A., Elder, C., Hill, K., Lumley, T., & McNamara, T. (1999). Studies in language testing 7: Dictionary of language testing. Cambridge: Cambridge University Press.
Dunkel, P. (1999). Considerations in developing or using second/foreign language proficiency computer-adaptive tests. Language Learning & Technology, 2 (2), 7793. Retrieved from http://llt.msu.edu/vol2num2/article4/index.html
English Language Institute (1986). English Language Institute listening comprehension test manual. Ann Arbor, MI: Author.
Flinders, D. J. (1997). [Review of the book InterViews: An introduction to qualitative research interviewing]. Evaluation and Program Planning, 20 (3), 287-288.
Fulcher, G. (1999). Computerizing an English language placement test. ELT Journal, 53 (4), pp. 289-299.
Gorsuch, G., & Austin, K. (In press). From paper and pencil to the web: A testing and technology partnership. Assessment Practices: TESOL Case Studies. Alexandria, VA: TESOL.
Green, A. (1998). Verbal protocol analysis in language testing research: A handbook. Cambridge: Cambridge University Press.
Green, B. (1988). Construct validity and computer-based tests. In H. Wainer & H. Braun (Eds.), Test validity (pp. 77-86). Hillsdale, NJ: Lawrence Erlbaum Associates.
Green, B. (2000). System design and operation. In H. Wainer, N. Dorans, D. Eignor, R. Flaugher, B. Green, R. Mislevy, L. Steinberg, & D. Thissen (Eds.), Computerized adaptive testing: A primer (2nd ed.) (pp. 23-35). Mahwah, NJ: Lawrence Erlbaum Associates.
Guba, E., & Lincoln, Y. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.
Hamilton, L., Nussbaum, E., & Snow, R. (1997). Interview process for validating science assessments. Applied Measurement in Education, 10 (2), 181-200.
Heinssen, R. K., Glass, C. R., & Knight, L. A. (1987). Assessing computer anxiety: Development and validation of the Computer Anxiety Rating Scale. Computers in Human Behavior, 3, 49-59.
Henning, G. (1991). Validating an item bank in a computer-assisted or computer-adaptive test. In P. Dunkel (Ed.), Computer-assisted language learning and testing: Research issues and practice (pp. 209-222). New York: Newbury House.
Hill, R. A. (1995). ToPE: Test of proficiency in English: The development of an adaptive test. In J. C. Alderson & B. North (Eds.), Language testing in the 1990s: The communicative legacy (pp. 237-246). Hertfordshire, UK: Phoenix ELT.
Kirk, J., & Miller, M. (1986). Reliability and validity in qualitative research. Newbury Park, CA: Sage.
Kenyon, D., & Malabonga, V. (2001). Comparing examinee attitudes toward computerassisted and other oral proficiency assessments. Language Learning & Technology, 5 (2), 60-83. Retrieved from http://llt.msu.edu/vol5num2/kenyon/ default.html
Kirk, J., & Miller, M. L. (1986). Reliability and validity in qualitative research. Newbury Park, CA: Sage.
Kunnan, A. J. (1995). Studies in language testing 2: Test taker characteristics and test performance. Cambridge: Cambridge University Press.
Kupermintz, H., Le, V., & Snow, R. (1999). Construct validation for mathematics achievement: Evidence from interview procedures (ERIC Document Reproduction Service No. ED428 125).
Kvale, S. (1996). Interviews: An introduction to qualitative research interviewing. Thousand Oaks, CA: Sage.
Larson, J. (1989). S-CAPE: A Spanish computerized adaptive placement exam. In W. F. Smith (Ed.), Modern technology in foreign language education: Applications and projects (pp. 277-289). Lincolnwood, IL: National Textbook Company.
LeCompte, M. D. & Goetz, J. P. (1982). Problems of reliability and validity in ethnographic research. Review of Educational Research, 52 (1), 31-60.
Luoma, S. (1993). Validating the Certificate of Foreign Language Proficiency: The usefulness of qualitative validation techniques (ERIC Document Reproduction Service No. ED362 027).
Madsen, H. (1991). Computer-adaptive testing of listening and reading comprehension. In P. Dunkel (Ed.), Computer-assisted language learning and testing: Research issues and practice (pp. 237-257). New York: Newbury House.
McNamara, T. (2000). Language testing. Oxford: Oxford University Press.
Miles, M., & Huberman, A. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage.
Niederhauser, D. S., Reynolds, R. E., Salmen, D. J., & Skolmoski, P. (2000). The influence of cognitive load on learning from hypertext. Journal of Educational Computing Research, 23 (3), 237-255.
Ozok, A. A., & Salvendy, G. (2000). Measuring consistency of web page design and its effects on performance and satisfaction. Ergonomics, 43 (4), 443-460.
Porter, D. (1995). Affective factors in language testing. In J. C. Alderson & B. North (Eds.), Language testing in the 1990s: The communicative legacy (pp. 32-40). Hertfordshire, UK: Phoenix ELT.
Purpura, J. (1999). Learner strategy use and performance on language tests: A structural equation modeling approach. Cambridge: Cambridge University Press.
Roever, C. (2001). Web-based language testing. Language Learning & Technology, 5 (2), 84-94. Retrieved from http://llt.msu.edu/vol5num2/roever/default.html
Russell, M., & Haney, W. (1997). Testing writing on computers: An experiment comparing student performance on tests conducted via computer and via paper-andpencil. Education Policy Analysis Archives, 5 (3), 1-21. retrieved from http:// olam.ed.asu.edu/epaa/v5n3.html
Sawaki, Y. (2001). Comparability of conventional and computerized tests of reading in a second language. Language Learning & Technology, 5 (2), 38-59. Retrieved from http://llt.msu.edu/vol5num2/sawaki/default.html
Schmitt, N. (1999). The relationship between TOEFL vocabulary items and meaning, association, collocation and word-class knowledge. Language Testing, 16 (2), 189-216.
Taylor, C., Kirsch, I., Eignor, D., & Jamieson, J. (1999). Examining the relationship between computer familiarity and performance on computer-based language tasks. Language Learning, 49 (2), 219-274.
Testing and Certification Division. (1972). English Language Institute listening comprehension test. Ann Arbor, MI: Author.
Wainer, H. (2000). Introduction and history. In H. Wainer, N. Dorans, D. Eignor, R. Flaugher, B. Green, R. Mislevy, L. Steinberg, & D. Thissen (Eds.), Computerized adaptive testing: A primer (2nd ed.) (pp. 1-21). Mahwah, NJ: Lawrence Erlbaum Associates.
Worthington, V. L., & Zhao, Y. (1999). Existential computer anxiety and changes in computer technology: What past research on computer anxiety has missed. Journal of Educational Computing Research, 20 (4), 299-315.
Wright, P. C., Fields, R. E., & Harrison, M. D. (2000). Analyzing human-computer interaction as distributed cognition: The resources model. Human-Computer Interaction, 15, 1-41.
Zandvliet, D., & Farragher, P. (1997). A comparison of computer-administered and written tests. Journal of Research on Computing in Education, 29 (4), 423-438.
