Replicability of (Socio)Linguistics studies
DOI:
https://doi.org/10.1558/jrds/6783228282Keywords:
language variation, methodology, replicability, research design, sample sizeAbstract
In this paper I address four methodological issues of relevance to the design of (Socio)Linguistics studies. Issues addressed in this paper are: the importance of a proper power analysis during the design stages, or the probability of correctly rejecting the null hypothesis when it is false in the population, the analysis of the structure of the data to select the proper statistical analysis to be used in these studies, polychotomization, or categorization, of continuous variables and its effect on the power of the study, and the determination of sample size to achieve an adequate level of power. After a careful analysis of the review of literature on copula + adjective and the lack of reports on power and sample size calculations to support the generalizations derived from each study, I discuss the design and replicability implications of studies where these design issues have not been addressed.
References
Aguilar-Sánchez, J. and Kelly, K. (forthcoming). Sample Size Planning in Sociolinguistic Research: A methodological contribution to the study of variation.
Aguilar-Sánchez, J. (2007). The use of Spanish Ser and Estar + adjectives: A sociolinguistic pilot study on the oral Spanish of Costa Rica. Paper presented at the 5th Hawaii Annual International Conference on the Arts and the Humanities, Hawaii.
Aguilar-Sánchez, J. (2009). Syntactic Variation: The case of copula choice in the Spanish of Limón, Costa Rica. (Doctor), Indiana University, Bloomington, IN.
Aguilar-Sánchez, J. (2012). Formal Instruction and Language Contact in Language Variation: the Case of Ser and Estar + Adjective in the Spanishes of Limón, Costa Rica. In K. Geeslin and M. Díaz-Campos (eds), Selected Proceedings of the 14th Hispanic Linguistic Symposium (pp. 9-25). Sumerville, MA: Cascadilla Proceedings Project.
Aguinis, H. (1995) Statistical power problems with moderated multiple regression in management research. Journal of Management 21 (6): 1141–1158.
Algina, J. and Olejnik, S. (2000) Determining sample size for accurate estimation of the squared multiple correlation coefficient. Multivariate Behavioral Research 35 (1): 119–137. http://dx.doi.org/10.1207/S15327906MBR3501_5
Beck, C. T. (1994) Achieving statistical power through research design sensitivity. Journal of Advanced Nursing 20 (5): 912–916. http://dx.doi.org/10.1046/j.1365-2648.1994.20050912.x
Bentivoglio, P. and Sedano, M. (1993) Investigación sociolingüística: Sus métodos aplicados a una experiencia venezolana. Boletín de Lingüística 8: 3–35.
Cohen, J. (1969) Statistical Power Analysis for the Behavioral Sciences. New York: Academic Press.
Cohen, J. (1988) Statistical Power Analysis for the Behavioral Sciences (2nd edn). Hillsdale, NJ: L. Erlbaum Associates.
Cohen, J. (1992a) A power primer. Psychological Bulletin 112 (1): 155–159. http://dx.doi.org/10.1037/0033-2909.112.1.155
Cohen, J. (1992b) Statistical power analysis. Current Directions in Psychological Science, 1 (3): 98–101. http://dx.doi.org/10.1111/1467-8721.ep10768783
Díaz-Campos, M. and Geeslin, K. L. (2011) Copula use in the Spanish of Venezuela: Social and linguistic sources of variation in Spanish. Spanish in Context 8 (1): 73–94. http://dx.doi.org/10.1075/sic.8.1.04dia
Fishman, J. A. (1972) The Sociology of Language; An Interdisciplinary Social Science Approach to Language in Society. Rowley, MA: Newbury House Publishers.
Gutiérrez, M. (1994) Ser y estar en el habla de Michoacán, México. Universidad Nacional Autónoma de México, México.
Hedges, L. V. (1987) How hard is hard science, how soft is soft science. American Psychologist, 42 (5): 443–455. http://dx.doi.org/10.1037/0003-066X.42.5.443
Holden, J. E., Kelley, K. and Agarwal, R. (2008) Analyzing change: A primer on multilevel models with applications to nephrology. American Journal of Nephrology, 28 (5): 792–801. http://dx.doi.org/10.1159/000131102
Hox, J. J. (2002) Multilevel Analysis: Techniques and Applications. Mahwah, NJ: Lawrence Erlbaum Publishers.
Johnson, D. E. (2009). Getting off the GoldVarb Standard: Introducing Rbrul for Mixed-Effects Variable Rule Analysis. Language and Linguistics Compass 3 (1): 350–383. http://dx.doi.org/10.1111/j.1749-818X.2008.00108.x
Johnson, D. E. (forthcoming) Progress in regression: Why sociolinguistic data calls for mixed-effects models.
Kelley, K. Maxwell, S. E. and Rausch, J. R. (2003) Obtaining power or obtaining precision. Evaluation & the Health Professions, 26 (3): 258–287. http://dx.doi.org/10.1177/0163278703255242
Kraemer, H. C. and Thiemann, S. (1987) How many Subjects? : Statistical Power Analysis in Research. Newbury Park, CA: Sage Publications.
Labov, W. (1967) The Social Stratification of English in New York City, Dissertation Abstracts: Section A. Humanities and Social Science (Vol. 28, pp. 655A–655A). Ann Arbor, MI.
Labov, W. (1972). Sociolinguistic Patterns. Philadelphia, PA: University of Pennsylvania Press.
Lavandera, B. R. (1975) Linguistic Structure and Sociolinguistic Conditioning in the Use of Verbal Endings in ‘si’-clauses (Buenos Aires Spanish). Philadelphia, PA: University of Pennsylvania.
Leon, A. C. (2004) Sample-size requirements for comparisons of two groups on repeated observations of a binary outcome. Eval Health Prof 27 (1): 34–44. http://dx.doi.org/10.1177/0163278703261198
Lindsay, R. M. and Ehrenberg, A. S. (1993) The design of replicated studies. The American Statistician 47 (3): 217–228.
Lipsey, M. W. (1990). Design Sensitivity: Statistical Power for Experimental Research. Newbury Park, CA: Sage Publications.
Maxwell, S. E. and Delaney, H. D. (1993) Bivariate median splits and spurious statistical significance. Psychological Bulletin 113 (1): 181–190. http://dx.doi.org/10.1037/0033-2909.113.1.181
Murphy, K. R. and Myors, B. (1998) Statistical Power Analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests. Mahwah, NJ: L. Erlbaum Associates.
Onwuegbuzie, A. J. and Leech, N. L. (2004) Post hoc power: A concept whose time has come. Understanding Statistics 3 (4): 201–230. http://dx.doi.org/10.1207/s15328031us0304_1
Ottenbacher, K. J. (1996) The power of replications and replications of power. The American Statistician 50 (3): 271–275.
Rossi, J. S. (1990) Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting & Clinical Psychology 58 (5): 646–656. http://dx.doi.org/10.1037/0022-006X.58.5.646
Schmitz, S., Cherny, S. S. and Fulker, D. W. (1998) Increase in power through multivariate analyses. Behavior Genetics 28 (5): 357–363. http://dx.doi.org/10.1023/A:1021669602220
Sedlmeier, P. and Gigerenzer, G. (1989) Do studies of statistical power have an effect on the power of studies? Psychological Bulletin 105 (2): 309–316. http://dx.doi.org/10.1037/0033-2909.105.2.309
Silva-Corvalán, C. (1994) Language Contact and Change : Spanish in Los Angeles. Oxford: Clarendon.
Silva-Corvalán, C. (2001) Sociolingüística y pragmática del español. Washington, DC: Georgetown University Press.
Tagliamonte, S. A. (2006) Analyzing Sociolinguistic Variation. Cambridge: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511801624
Tversky, A. and Kahneman, D. (1971) Belief in the law of small numbers. Psychological Bulletin 76 (2): 105–110. http://dx.doi.org/10.1037/h0031322
Wilkinson, L. and The American Psychological Association Task Force (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist 54 (8): 594–604. http://dx.doi.org/10.1037/0003-066X.54.8.594
Ziman, J. (1967) Public Knowledge. Cambridge: Cambridge University Press.
Zodpey, S. P. (2004). Sample size and power analysis in medical research. Indian Journal of Dermatology, Venereology & Leprology 70 (2): 123–128.
Aguilar-Sánchez, J. (2007). The use of Spanish Ser and Estar + adjectives: A sociolinguistic pilot study on the oral Spanish of Costa Rica. Paper presented at the 5th Hawaii Annual International Conference on the Arts and the Humanities, Hawaii.
Aguilar-Sánchez, J. (2009). Syntactic Variation: The case of copula choice in the Spanish of Limón, Costa Rica. (Doctor), Indiana University, Bloomington, IN.
Aguilar-Sánchez, J. (2012). Formal Instruction and Language Contact in Language Variation: the Case of Ser and Estar + Adjective in the Spanishes of Limón, Costa Rica. In K. Geeslin and M. Díaz-Campos (eds), Selected Proceedings of the 14th Hispanic Linguistic Symposium (pp. 9-25). Sumerville, MA: Cascadilla Proceedings Project.
Aguinis, H. (1995) Statistical power problems with moderated multiple regression in management research. Journal of Management 21 (6): 1141–1158.
Algina, J. and Olejnik, S. (2000) Determining sample size for accurate estimation of the squared multiple correlation coefficient. Multivariate Behavioral Research 35 (1): 119–137. http://dx.doi.org/10.1207/S15327906MBR3501_5
Beck, C. T. (1994) Achieving statistical power through research design sensitivity. Journal of Advanced Nursing 20 (5): 912–916. http://dx.doi.org/10.1046/j.1365-2648.1994.20050912.x
Bentivoglio, P. and Sedano, M. (1993) Investigación sociolingüística: Sus métodos aplicados a una experiencia venezolana. Boletín de Lingüística 8: 3–35.
Cohen, J. (1969) Statistical Power Analysis for the Behavioral Sciences. New York: Academic Press.
Cohen, J. (1988) Statistical Power Analysis for the Behavioral Sciences (2nd edn). Hillsdale, NJ: L. Erlbaum Associates.
Cohen, J. (1992a) A power primer. Psychological Bulletin 112 (1): 155–159. http://dx.doi.org/10.1037/0033-2909.112.1.155
Cohen, J. (1992b) Statistical power analysis. Current Directions in Psychological Science, 1 (3): 98–101. http://dx.doi.org/10.1111/1467-8721.ep10768783
Díaz-Campos, M. and Geeslin, K. L. (2011) Copula use in the Spanish of Venezuela: Social and linguistic sources of variation in Spanish. Spanish in Context 8 (1): 73–94. http://dx.doi.org/10.1075/sic.8.1.04dia
Fishman, J. A. (1972) The Sociology of Language; An Interdisciplinary Social Science Approach to Language in Society. Rowley, MA: Newbury House Publishers.
Gutiérrez, M. (1994) Ser y estar en el habla de Michoacán, México. Universidad Nacional Autónoma de México, México.
Hedges, L. V. (1987) How hard is hard science, how soft is soft science. American Psychologist, 42 (5): 443–455. http://dx.doi.org/10.1037/0003-066X.42.5.443
Holden, J. E., Kelley, K. and Agarwal, R. (2008) Analyzing change: A primer on multilevel models with applications to nephrology. American Journal of Nephrology, 28 (5): 792–801. http://dx.doi.org/10.1159/000131102
Hox, J. J. (2002) Multilevel Analysis: Techniques and Applications. Mahwah, NJ: Lawrence Erlbaum Publishers.
Johnson, D. E. (2009). Getting off the GoldVarb Standard: Introducing Rbrul for Mixed-Effects Variable Rule Analysis. Language and Linguistics Compass 3 (1): 350–383. http://dx.doi.org/10.1111/j.1749-818X.2008.00108.x
Johnson, D. E. (forthcoming) Progress in regression: Why sociolinguistic data calls for mixed-effects models.
Kelley, K. Maxwell, S. E. and Rausch, J. R. (2003) Obtaining power or obtaining precision. Evaluation & the Health Professions, 26 (3): 258–287. http://dx.doi.org/10.1177/0163278703255242
Kraemer, H. C. and Thiemann, S. (1987) How many Subjects? : Statistical Power Analysis in Research. Newbury Park, CA: Sage Publications.
Labov, W. (1967) The Social Stratification of English in New York City, Dissertation Abstracts: Section A. Humanities and Social Science (Vol. 28, pp. 655A–655A). Ann Arbor, MI.
Labov, W. (1972). Sociolinguistic Patterns. Philadelphia, PA: University of Pennsylvania Press.
Lavandera, B. R. (1975) Linguistic Structure and Sociolinguistic Conditioning in the Use of Verbal Endings in ‘si’-clauses (Buenos Aires Spanish). Philadelphia, PA: University of Pennsylvania.
Leon, A. C. (2004) Sample-size requirements for comparisons of two groups on repeated observations of a binary outcome. Eval Health Prof 27 (1): 34–44. http://dx.doi.org/10.1177/0163278703261198
Lindsay, R. M. and Ehrenberg, A. S. (1993) The design of replicated studies. The American Statistician 47 (3): 217–228.
Lipsey, M. W. (1990). Design Sensitivity: Statistical Power for Experimental Research. Newbury Park, CA: Sage Publications.
Maxwell, S. E. and Delaney, H. D. (1993) Bivariate median splits and spurious statistical significance. Psychological Bulletin 113 (1): 181–190. http://dx.doi.org/10.1037/0033-2909.113.1.181
Murphy, K. R. and Myors, B. (1998) Statistical Power Analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests. Mahwah, NJ: L. Erlbaum Associates.
Onwuegbuzie, A. J. and Leech, N. L. (2004) Post hoc power: A concept whose time has come. Understanding Statistics 3 (4): 201–230. http://dx.doi.org/10.1207/s15328031us0304_1
Ottenbacher, K. J. (1996) The power of replications and replications of power. The American Statistician 50 (3): 271–275.
Rossi, J. S. (1990) Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting & Clinical Psychology 58 (5): 646–656. http://dx.doi.org/10.1037/0022-006X.58.5.646
Schmitz, S., Cherny, S. S. and Fulker, D. W. (1998) Increase in power through multivariate analyses. Behavior Genetics 28 (5): 357–363. http://dx.doi.org/10.1023/A:1021669602220
Sedlmeier, P. and Gigerenzer, G. (1989) Do studies of statistical power have an effect on the power of studies? Psychological Bulletin 105 (2): 309–316. http://dx.doi.org/10.1037/0033-2909.105.2.309
Silva-Corvalán, C. (1994) Language Contact and Change : Spanish in Los Angeles. Oxford: Clarendon.
Silva-Corvalán, C. (2001) Sociolingüística y pragmática del español. Washington, DC: Georgetown University Press.
Tagliamonte, S. A. (2006) Analyzing Sociolinguistic Variation. Cambridge: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511801624
Tversky, A. and Kahneman, D. (1971) Belief in the law of small numbers. Psychological Bulletin 76 (2): 105–110. http://dx.doi.org/10.1037/h0031322
Wilkinson, L. and The American Psychological Association Task Force (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist 54 (8): 594–604. http://dx.doi.org/10.1037/0003-066X.54.8.594
Ziman, J. (1967) Public Knowledge. Cambridge: Cambridge University Press.
Zodpey, S. P. (2004). Sample size and power analysis in medical research. Indian Journal of Dermatology, Venereology & Leprology 70 (2): 123–128.
Published
2013-08-23
Issue
Section
Articles
How to Cite
Aguilar-Sánchez, J. (2013). Replicability of (Socio)Linguistics studies. Journal of Research Design and Statistics in Linguistics and Communication Science, 1(1), 5-25. https://doi.org/10.1558/jrds/6783228282
