Quantitative considerations for improving replicability in CALL and applied linguistics
DOI:
https://doi.org/10.1558/cj.v32i2.26857Keywords:
computer-assisted language instruction, SLA, replication, quantitative research methods,Abstract
There are a number of methodological practices commonly employed by CALL researchers that limit progress in the field. Some of these practices are particular to replication research, but most are more general and are found throughout the field. I describe in this paper two studies that are fabricated but that resemble much of what is found in published second language research. Each study corresponds to and contains a set of methodological issues. Following each study, I address the issues they illustrate, providing comments and suggestions for how the analyses could be modified to produce greater replicability and/or replicational value. I conclude with a summary of suggestions for quantitative reforms related to improving replication research and quantitative practices more generally in CALL and applied linguistics.References
Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.
Gass, S., & Valmori, L. (in press). Replication in interaction and working memory research: Révész (2012) and Goo (2012). Language Teaching. http://dx.doi.org/10.1017/S0261444815000038
Godfroid, A., & Spino, L. (in press). Reconceptualizing reactivity research: Absence of evidence is not evidence of absence. Language Learning.
Jeon, E. H., & Yamashita, J. (2014). L2 reading Comprehension and its correlates: A meta-analysis. Language Learning, 64 (1), 160–212. http://dx.doi.org/10.1111/lang.12034
Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.
Larson-Hall, J., & Plonsky, L. (in press). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. In J. M. Norris, S. Ross, & R. Schoonen (Eds), Improving and extending quantitative reasoning in second language research. Malden, MA: Wiley.
Norris, J. M. (in press). Statistical significance testing in second language research: Basic problems and suggestions for reform. In J. M. Norris, S. Ross, & R. Schoonen (Eds.), Improving and extending quantitative reasoning in second language research. Malden, MA: Wiley.
Norris, J. M., & Ortega, L. (2006). The value and practice of research synthesis for language learning and teaching. In J. M. Norris & L. Ortega (Eds), Synthesizing research on language learning and teaching, 3–50. Philadelphia, PA: John Benjamins.
Norris, J. M., Plonsky, L., Ross, S. J., & Schoonen, R. (in press). Guidelines for reporting quantitative methods and results in primary research. Language Learning.
Oswald, F. L., & Plonsky, L. (2010). Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30, 85–110. http://dx.doi.org/10.1017/S0267190510000115
Plonsky, L. (2012). Replication, meta-analysis, and generalizability. In G. Porte (Ed.), Replication research in applied linguistics, 116–132. New York: Cambridge University Press.
Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655–687. http://dx.doi.org/10.1017/S0272263113000399
Plonsky, L. (in press). Statistical power, p values, descriptive statistics, and effect sizes: A ‘back-to-basics’ approach to advancing quantitative methods in L2 research. In L. Plonsky (Ed.), Advancing quantitative methods in second language research. New York: Routledge.
Plonsky, L., & Oswald, F. L. (2014). How big is ‘big’? Interpreting effect sizes in L2 research. Language Learning, 64 (4), 878–912. http://dx.doi.org/10.1111/lang.12079
Porte, G. (2013). Who needs replication research? CALICO Journal, 30, 10–15. http://dx.doi.org/10.11139/cj.30.1.10-15
Smith, B., & Schulze, M. (2013). Thirty years of the CALICO Journal – replicate, replicate, replicate. CALICO Journal, 30 (1), i–iv. http://dx.doi.org/10.11139/cj.30.1.i-iv
