Connecting Criterion scores and Classroom Grading Contexts: A Systemic Functional Linguistic Model for Teaching and Assessing Causal Language

Authors

  • Hong Ma Iowa State University
  • Tammy Slater Iowa State University

DOI:

https://doi.org/10.1558/cj.v33i1.26562

Keywords:

AWE, Criterion score, systemic functional linguistics

Abstract

This study utilized theory proposed by Mohan, Slater, Luo, and Jaipal (2002) regarding what they refer to as the Developmental Path of Cause to examine issues of AWE score use in classroom contexts. Utilization of this model enabled this study to investigate the accuracy of the AWE scores by comparing them to ratings based on teachers’ intuition as well as to scores generated based on existing rubrics. The qualitative data collected from focus group interviews of three experienced teachers’ justifications for their intuitive evaluations of essays suggested that the Developmental Path of Cause helped teachers articulate their intuitions, identifying the core features of the model. The quantitative results showed that the grades provided by raters trained to use the Developmental Path of Cause tended to support Criterion scores more strongly than did instructor grades. The findings from this study suggest that AWE scores from Criterion not only closely correlated with teachers’ intuitions and with raters trained to use the Developmental Path of Cause, but that the use of the Developmental Path of Cause for teaching may support the use of AWE systems in the classroom context, and would help students focus on the core of a cause-effect essay: appropriateness and sophistication of causal language.

Author Biographies

  • Hong Ma, Iowa State University
    Hong Ma is a PhD candidate in Applied Linguistics and Technology at Iowa State University. Her primary research interests lay in computer-assisted language learning and language testing. She is currently leading multiple research projects, which intend to develop and evaluate a vocabulary-learning tool and extract a more pedagogy-informed vocabulary list using programming language.
  • Tammy Slater, Iowa State University
    Tammy Slater is an associate professor in Applied Linguistics and Technology at Iowa State University. Her research draws upon Systemic Functional Linguistics to understand the development of academic language through content-based and project-based teaching and learning, particularly as it informs English language education.

References

American Council on the Teaching of Foreign Languages. (2002). Program standards for the preparation of foreign language teachers (Initial level- undergraduate & graduate) (For K-12 and secondary certification programs). Retrieved from http://www.actfl.org/sites/default/files/pdfs/public/ACTFLNCATEStandardsRevised713.pdf.

Attali, Y., & Burstein, J. (2005). Automated essay scoring with e-rater version 2.0 (ETS RR-04-45). Princeton, NJ: Educational Testing Service.

Attali, Y., Bridgeman, B., & Trapani, C. (2010). Performance of a generic approach in automated essay scoring. The Journal of Technology, Learning and Assessment, 10 (3), 1–17.

Bachman, L. (2004). Statistical Analysis for Language Assessment. New York: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511667350

Burstein, J., Chodorow, M., & Leacock, C. (2003a). Criterion online essay evaluation: An application for automated evaluation of student essays. AI Magazine, 25 (3), 27–35. http://dx.doi.org/10.1609/aimag.v25i3.1774

Burstein, J., Marcu, D., & Knight, K. (2003b). Finding the WRITE stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems: Special Issue on Natural Language Processing, 18 (1), 32–39. http://dx.doi.org/10.1109/MIS.2003.1179191

Chen, C.-F.E., & Cheng, W.-Y.E. (2008). Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12 (2), 94–112.

Cheville, J. (2004). Automated Scoring Technologies and the Rising Influence of Error. English Journal, 93 (4), 47–52. http://dx.doi.org/10.2307/4128980

Conference on College Composition and Communication (2006). Writing assessment: A position statement. Retrieved July 20, 2007, from http://www.ncte.org/cccc/resources/positions/123784.htm.

Deane, P. (2013). On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing, 18, 7–24. http://dx.doi.org/10.1016/j.asw.2012.10.002

Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5 (1). Retrieved from http://www.jtla.org

Ebyary, K., & Windeatt, S. (2010). The impact of computer-based feedback on students’ written work, International Journal of English Studies, 10 (2), 121–142.

Ericsson, P. F. (2006). The meaning of meaning: Is a paragraph more than an equation? In P. F. Ericsson & R. H. Haswell (Eds.), Machine scoring of student essays: Truth and consequences, 28–37. Logan: Utah State University Press.

Green, A. (2007). IELTS washback in context: Preparation for academic writing in higher education. Cambridge, UK: Cambridge University Press.

Grimes, D., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8 (6), 4–44.

Halliday, M. A. K. (1998). Things and relations: Regrammaticising experience as technical knowledge. In J.R. Martin & R. Veel (Eds), Reading science: Critical and functional perspectives on discourses of science, 185–235. New York: Routledge.

Halliday, M. A. K., & Martin, J. R. (1993). Writing Science: Literacy and Discursive Power. Washington DC: The Falmer Press.

Li, Z., Link, S., Ma, H., Yang, H., & Hegelheimer, V. (2014). The Role of Automated Writing Evaluation Holistic Scores in the ESL Classroom. SYSTEM Journal, 44, 66–78. http://dx.doi.org/10.1016/j.system.2014.02.007

Klenbaum, D. G., Kupper, L. L., & Muller, K. E. (1988). Applied regression analysis and other multivariable methods. Boston: PWS-KENT Publishing Company.

Low, M. (2010). Teachers and texts: Judging what English language learners know from what they say. In A. Paran & L. Sercu (Eds), Testing the untestable in language education (pp. 241–255). Bristol, UK: Multilingual Matters.

Messick, S. (1996). Validity and washback in language testing. Princeton, NJ: Education Testing Services.

Mohan, B., & Slater, T. (2004). The evaluation of causal discourse and language as a resource for meaning. In J. A. Foley. (Ed.), Language, education & discourse: Functional approaches, 255–269. London: Continuum.

Mohan, B., Slater, T., Luo, L., & Jaipal, K. (2002). Developmental lexicogrammar of causal explanations in science. Paper presented at the International Systemic Functional Linguistics Congress (ISFC29), Liverpool, UK.

Saldana, J. (2009). The coding manual for qualitative researchers. Washington, DC: SAGE.

Slater, T. (1998). Evaluating causal discourse in academic writing. MA thesis. University of British Columbia.

Slater, T. (2004). The discourse of causal explanations in school science. PhD thesis, University of British Columbia.

Slater, T., & Mohan, B. (2010). Towards systematic and sustained formative assessment of causal explanations in oral interactions. In A. Paran & L. Sercu (Eds), Testing the untestable in language education, 256–269. Bristol, UK: Multilingual Matters.

Wang, J., & Brown, M. S. (2007). Automated essay scoring versus human scoring: a comparative study. Journal of Technology, Learning, and Assessment, 6 (2).

Ziegler, W.W. (2007). Computerized Writing Assessment: Community College Faculty Find Reasons to say ‘Not Yet’. In P. F. Ericsson & R. Haswell (Eds.), Machine Scoring of Human Essays: Truth and Consequences, 138–153. Logan, Utah: Utah State University Press.

Downloads

Published

2016-01-30

Issue

Section

Articles

How to Cite

Ma, H., & Slater, T. (2016). Connecting Criterion scores and Classroom Grading Contexts: A Systemic Functional Linguistic Model for Teaching and Assessing Causal Language. CALICO Journal, 33(1), 1-18. https://doi.org/10.1558/cj.v33i1.26562

Most read articles by the same author(s)

1 2 3 4 5 6 7 8 9 10 > >>