Journal of Research Design and Statistics in Linguistics and Communication Science, Vol 5, No 1-2 (2018)

Analyzing Processing Time Data in Applied Linguistics and Second Language Research: A Multivariate Mixed-effects Approach

Bronson Hui
Issued Date: 19 Dec 2019


In this paper, I introduce the use of an extension of the popular linear mixed-effects models: multivariate mixed models which can handle multiple outcome variables. This technique is especially useful for applied linguists and second language researchers who use processing time data (e.g., accuracy and reaction times) obtained from timed decision tasks, text-based eye tracking, and self-paced reading. It can address the long-standing issue of multiple comparisons as a result of having multiple outcomes (e.g., first fixation durations and total reading times in text-based eye tracking, and time spent in different regions in self-paced reading). This technique also provides exciting opportunities for researchers to ask new questions that could not be addressed in a straightforward manner with traditional statistics. With this technique, researchers are able to investigate differential effects of a predictor on different outcomes. Through a demonstration in R using published, open eye-tracking data, I contextualize my discussion of the technique, offering also practical, step-by-step, and annotated guidelines for interested researchers.

Download Media

PDF Subscribers Only

DOI: 10.1558/jrds.39117


Baayen, R. H. (2008). Analyzing Linguistic Data: A Practical Introduction to Statistics using R. Cambridge: Cambridge University Press.


Baayen, R. H., Davidson, D. J., and Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59 (4): 390-412.


Baldwin, S. A., Imel, Z. E., Braithwaite, S. R., and Atkins, D. C. (2014). Analyzing multiple outcomes in clinical research using multivariate multilevel models. Journal of Consulting and Clinical Psychology 82 (5), 920-930.


Barr, D. J., Levy, R., Scheepers, C., and Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68 (3): 255-278.


Cunnings, I. (2012). An overview of mixed-effects statistical models for second language researchers. Second Language Research Analyzing multiple outcomes in clinical research using multivariate multilevel models 28 (3): 369-382.


Cunnings, I. and Finlayson, I. (2015). Mixed effects modeling and longitudinal data analysis. In L. Plonsky (Ed.), Advancing Quantitative Methods in Second Language Research, 159-181. New York: Routledge.


Draheim, C., Hicks, K. L., and Engle, R. W. (2016). Combining reaction time and accuracy: The relationship between working memory capacity and task switching as a case example. Perspectives on Psychological Science 11: 133-155.


Draheim, C., Mashburn, C. A., Martin, J. D., and Engle, R. W. (2019). Reaction time in differential and developmental research: A review and commentary on the problems and alternatives. Psychological Bulletin 145 (5): 508-535.


Elgort, I., Brysbaert, M., Stevens, M., and Van Assche, E. (2018). Contextual word learning during reading in a second language: An eye-movement study. Studies in Second Language Acquisition 40 (2): 341-366.


Godfroid, A. (2019). Eye Tracking in Second Language Acquisition and Bilingualism: A Research Synthesis and Methodological Guide. New York: Routledge.


Godfroid, A. and Winke, P. (2015). Investigating implicit and explicit processing using


L2 learners' eye-movement data. In P. Rebuschat (Ed.) Implicit and Explicit


Learning of Languages, 325-348. Amsterdam: John Benjamins.


Gries, S. Th. (2015). The most under-used statistical method in corpus linguistics: Multi-level (and mixed-effects) models. Corpora 10 (1): 95-125.


Hadfield, J. D. (2010). MCMC methods for multi-response generalized linear mixed models: The MCMCglmm R package. Journal of Statistical Software 33 (2): 1-22.


Hamaker, E. L. and Klugkist, I. (2011). Bayesian estimation of multilevel models. In J. J. Hox and J. K. Roberts (Eds), Handbook of Advanced Multilevel Analysis, 137-161. New York: Routledge.


Hox, J. J., Moerbeek, M., and Van de Schoot, R. (2018). Multilevel Analysis: Techniques and Applications. New York: Routledge.


Indrarathne, B., Ratajczak, M., and Kormos, J. (2018). Modelling changes in the cognitive processing of grammar in implicit and explicit learning conditions: Insights from an eye‐tracking study. Language Learning 68 (3): 669-708.


Jeon, E. H. (2015). Multiple regression. In L. Plonsky (Ed.), Advancing Quantitative Methods in Second Language Research, 131-158. New York: Routledge.


Jiang, N. (2013). Conducting Reaction Time Research in Second Language Studies. New York: Routledge.


Keating, G. and Jegerski, J. (2015). Experimental designs in sentence processing research. Studies in Second Language Acquisition 37: 1-32.


Lagrou, E., Hartsuiker, R. J., and Duyck, W. (2011). Knowledge of a second language influences auditory word recognition in the native language. Journal of Experimental Psychology. Learning, Memory, and Cognition 37 (4): 952-965.


Larson-Hall, J. and Herrington, R. (2010). Improving data analysis in second language acquisition by utilizing modern developments in applied statistics. Applied Linguistics 31 (3): 368-390.


Linck, J. A. and Cunnings, I. (2015). The utility and application of mixed-effects models in second language research. Language Learning 65 (S1), 185-207.


Marsden, E., Morgan-Short, K., Thompson, S., and Abugaber, D. (2018). Replication in second language research: Narrative and systematic reviews and recommendations for the field. Language Learning 68: 321-391.


Marsden, E., Thompson, S., and Plonsky, L. (2018). A methodological synthesis of self-paced reading in second language research. Applied Psycholinguistics 39 (5): 861-904.


Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., and Bates, D. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language 94: 305-315.


Norouzian, R., de Miranda, M., and Plonsky, L. (2018). The Bayesian revolution in second language research: An applied approach. Language Learning 68 (4): 1032-1075.


Plonsky, L. (2014). Study quality in quantitative L2 research (1990-2010): A methodological synthesis and call for reform. The Modern Language Journal 98 (1): 450-470.


Plonsky, L. and Oswald, F. L. (2017). Multiple regression as a flexible alternative to ANOVA in L2 research. Studies in Second Language Acquisition 39 (3): 579-592.


Polio, C. and Gass, S. M. (1997). Replication and reporting: A commentary. Studies in Second Language Acquisition 19: 499-508.


Porte, G. (2012). Replication Research in Applied Linguistics. Cambridge: Cambridge University Press.


Raudenbush, S. W. and Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed). London: Sage.


Rayner, K. (2009). Eye movements in reading: Models and data. Journal of Eye Movement Research 2: 1-10.


Rayner, K., Reichle, E. D., and Pollatsek, A. (2005). Eye movement control in reading and the E-Z Reader model. In G. Underwood (Ed.), Cognitive Processes in Eye Guidance, 131-162. Oxford: Oxford University Press.


Regev, S. and Meiran, N. (2014). Post-error slowing is influenced by cognitive control demand. Acta Psychologica 152: 10-18.


Ross, S. J. and Mackey, B. (2015). Bayesian approaches to imputation, hypothesis testing, and parameter estimation. Language Learning 65 (S1), 208-227.


Tabachnick, B. G. and Fidell, L. S. (2013). Using Multivariate Statistics (6th ed.) Boston, MA: Pearson.


Van de Schoot, R., Winter, S., Ryan, O., Zondervan-Zwijnenburg, M., and Depaoli, S. (2017). A systematic review of Bayesian papers in psychology: The last 25 years. Psychological Methods 22: 217-239.


Von der Malsburg, T. and Angele, B. (2017). False positives and other statistical errors in standard analyses of eye movements in reading. Journal of Memory and Language 94: 119-133.


Whitford, V. and Titone, D. (2019). Lexical entrenchment and cross-language activation: Two sides of the same coin for bilingual reading across the adult lifespan. Bilingualism: Language and Cognition 22 (1): 58-77.


Winke, P. M., Godfroid, A., and Gass, S. M. (2013). Introduction to the special issue: Eye-movement recordings in second language research. Studies in Second Language Acquisition 35 (2): 205-212.




  • There are currently no refbacks.

Equinox Publishing Ltd - 415 The Workstation 15 Paternoster Row, Sheffield, S1 2BX United Kingdom
Telephone: +44 (0)114 221-0285 - Email:

Privacy Policy