Practical Design and Implementation Considerations of a Computer Adaptive Foreign Language Test

The Monash/Melbourne French CAT

Authors

  • Jack Burston

DOI:

https://doi.org/10.1558/cj.v13i1.26-43

Keywords:

Computer adaptive testing, item response theory, Rasch model, foreign language proficiency testing, French placement testing

Abstract

While the psychometric and statistical models underlying the design of computer adaptive tests (CAT) are well understood, relatively few working models exist for the purpose of foreign language assessment. Likewise, little has been published concerning the practical considerations affecting the implementation of such tests. in the process of constructing the Monash/Melbourne French CAT, we discovered much about putting testing theory into practice. The present paper reports this experience in three parts. In a preliminary section, we describe the academic context in which the French CAT was created and trialed. This is followed by a detailed consideration of the test presentation platform and operating algorithms. Lastly, we give an evaluation of the first administration of the French CAT, accompanied by a discussion of the test's reliability and validity as a placement instrument for first year Australian university students.

References

Bachman, L.F. (1990). Fundamental Considerations in Language Testing. Oxford: Oxford University Press.

Bock, R., and R. Mislevy (1982). “Adaptive EAP Estimation of Ability in a Microcomputer Environment.” Applied Psychological Measurement 6, 4, 431-444.

Brown, J.D., and K.M. Bailey (1984). “A Categorical Instrument for Scoring Second Language Writing Skills.” Language Learning 34, 4, 21-42.

Burston, J. (1993). “The Validity of French VCE Results in Tertiary Level Language Placement.” Paper presented at the Modern Language Teachers’ Association of Victoria Annual Congress, Melbourne, July 5-6.

______. (1995). “The French CAT: An Assessment of its Empirical Validity.” Australian Review of Applied Linguistics 18, 1, 52-68.

______., M. Monville-Burston, and J. Harfouch (1996). “Comparison of Evaluation Measures in First Year Students’ Placement.” Issues and Innovations in the Teaching of French, Special Issue of the Australian Review of Applied Linguistics, edited by J. Burston, et al. (forthcoming).

Canale, M. (1986). “The Promise and Threat of Computerized Adaptive Assessment of Reading Comprehension.” Technology and Language Testing, edited by C.W. Stansfield, 29-45.

Carlson, S.B., and R. Camp (1985). “Relationships Between Direct and Indirect Measures of Writing Ability.” Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL, March 31-April 4. (ERIC document ED255543).

Davies, A. (1990). Principles of Language Testing. London: Blackwell.

______. (1992). “Is Proficiency Always Achievement?” Melbourne Papers in Language Testing 1, 1, 1-16.

Dunkel, P., ed. (1991). Computer Assisted Language Learning and Testing, New York, NY: Newbury House.

Farhady, H. (1983). “The Disjunctive Fallacy Between Discrete-Point Testing and Integrative Tests.” Issues in Language Testing Research, edited by J.W. Oller, 311-322. Rowley, MA: Newbury House.

Hambleton, R.K, and L.L. Cook (1977). “Latent Trait Models and Their Use in the Analysis of Educational Test Data.” Journal of Educational Measurement 14, 2, 75-96.

Hamp-Lyons, L. (1989). “Applying the Partial Credit Mod3el of Rash Analysis: Language Testing and Accountability.” Language Testing 6, 1, 109-118.

Henning, G. (1991). “Validating an Item Bank in a Computer Assisted or Computer Adaptive Test.” Computer Assisted Language Learning and Testing, edited by P. Dunkel, 209-222.

______. (1987). A Guide to Language Testing. New York: Newbury House.

______. (1984). “Advantages of Latent Trait Measurement in Language Testing.” Language Testing 1, 1, 123-133.

______., T. Hudson, and J. Turner (1985). “Item Response Theory and the Assumption of Unidimensionality for Language Tests.” Language Testing 2, 2, 141-154.

Kaya-Carton, E., A.S. Carton, and P. Dandonoli (1991). “Developing a Computer Adaptive Test of French Reading Proficiency.” Computer Assisted Language Learning and Testing, edited by P. Dunkel, 259-84.

Lange, D.L. (1990). “Priority Issues in the Assessment of Communicative Language Abilities.” Foreign Language Annals 23, 403-407.

Larson, J.W. (1991). “S-CAPE: A Spanish Computerized Adaptive Placement Exam.” Modern Technology in Foreign Language Education: Applications and Project, edited by W.F. Smith, 277-89. Lincolnwood, IL: National Textbook Company.

______., and H. Madsen (1985). “Computerized-Adaptive Testing: Moving Beyond Computer Assisted Testing.” CALICO Journal 3, 2, 32-36.

Lord, F.M. (1980). Applications of Item Response Theory to Practical Testing Problems.. Hillsdale, NJ: Erlbaum.

Lunz, M.E., J.A. Stahl, and B.A. Bergstrom (1993). “Targeting, Test Length, Test Precision, and Decision Accuracy for Computerized Adaptive Tests.” Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, April.

Lunz, M.E., and B.A. Bergstrom (1991). “Comparability of Decisions for Computer Adaptive and Written Examinations.” Journal of Allied Health Winter 1991, 15-23.

McNamara, T.F. (1991). “The Role of Item Response Theory in Language Test Validation.” Current Developments in Language Testing, edited by A. Anivan, 165-184. Singapore: SEAMEO Regional Language Centre.

______. (1990). “Item Response Theory and the Validation of an ESP Test for Health Professionals.” Language Testing 7, 1, 52-75.

Madsen, H.S. (1991). “Computer Adaptive Testing of Listening and Reading Comprehension: The Brigham Young University Approach.” Computer Assisted Language Learning and Testing, edited by P. Dunkel, 237-257.

Meunier, L. (1994). “Computer Adaptive Language Tests (CALT) Offer a Great Potential for Functional Testing. Yet Why Don’t They?” CALICO Journal 11, 4, 23-39.

Oller, J.W. (1976). “Evidence for a General Language Proficiency Factor: An Expectancy Grammar.” Die Neueren Sprachen 75, 165-174.

Rasch, G. (1960). Probabilistic Model for Some Intellig4ence and Attainment Tests. Chicago, IL: University of Chicago Press.

Spolsky, B. (1995). Measured Words: The Development of Objective Language Testing. Oxford: Oxford University Press.

Stansfield, C.W. ed. (1986). Technology and Language Testing. Washington, D.C.: TESOL.

Stevenson, J., and S. Gross (1991). “Use of a Computerized Adaptive Testing Model for ESOL/Bilingual Entry/Exit Decision Making.” Computer Assisted Language Learning and Testing, edited by P. Dunkel, 223-235.

Traub, R.E., and R.G. Wolfe (1981). “Latent Trait Theories and the Assessment of Educational Achievement.” 1981 Review of Research in Education, edited by D.C. Berliner. Washington, DC: Educational Research Association.

Tung, P. (1986). “Computerized Adaptive Testing: Implications for Language Test Developers.” Technology and Language Testing, edited by C.W. Stansfield, 12-28.

Wainer, H. (1990). Computerized Adaptive Testing: A Primer. Hillsdale, NJ: Educational Testing Service.

______. (1983). “On Item Response Theory and Computerized Adaptive Testing.” Journal of College Admissions 28, 4, 9-16.

Warren, J. (1996). “How Students Pick the Right Answer: A Protocol Study of the French CAT (Computer Adaptive Test).” Issues and Innovations in the Teaching of French, Special Issue of the Australian Review of Applied Linguistics, edited by J. Burston, et al. (forthcoming).

Weiss, D.J. (1983). “Introduction.” New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing, edited by D.J. Weiss, 1-8. New York: Academic Press.

______. (1982). “Improving Measurement Quality and Efficiency with Adaptive Testing.” Applied Psychological Measurement 6, 4, 473-492.

______., and G.G. Kingsbury (1984). “Application of Computerized Adaptive Testing to Educational Problems.” Journal of Educational Measurement 21, 4, 361-375.

Wright, B.D., and G. Masters (1982). Rating Scale Analysis. Chicago, IL: MESA.

______., and M.H. Stone (1979). Best Test Design. Chicago, IL: MESA.

Downloads

Published

2013-01-14

Issue

Section

Articles

How to Cite

Burston, J. (2013). Practical Design and Implementation Considerations of a Computer Adaptive Foreign Language Test: The Monash/Melbourne French CAT. CALICO Journal, 13(1), 26-43. https://doi.org/10.1558/cj.v13i1.26-43