Towards Better ESL Practices for Implementing Automated Writing Evaluation
DOI:
https://doi.org/10.11139/cj.31.3.323-344Keywords:
automated writing evaluation, CALL, technology, L2 writing instructionAbstract
In the past decade, a new wave of writing assessment has evolved as a result of advances in computer technology, highlighting the potential of automated writing evaluation (AWE) in the L2 writing curriculum (Warschauer & Ware, 2006). Today’s innovative AWE programs offer ESL instructors promising solutions for providing immediate feedback and meeting the demands for better practices in the digital age. However, the outcomes of using AWE programs are not free of challenges and necessitate comprehensive insights about effectively integrating AWE and improving students’ writing (Chen & Cheng, 2008). This longitudinal qualitative study investigates the practices and perspectives of five university-level writing instructors as they meet challenges and possibilities of integrating AWE into seven semester-long university ESL writing courses. Data collection included observations, individual interviews, and delayed focus group interviews. In this paper, we highlight ESL instructors’ teaching strategies, perception of the effectiveness, satisfaction, and concerns with the software during and after one semester of teaching. Results conclude with final suggestions from current instructors. Our research intends to expand the current knowledge of AWE programs by defining appropriate and effective implementation of AWE technologies.
References
Attali, Y. (2004). Exploring the feedback and revision features of Criterion. Archives of the Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/erater_NCME_2004_Attali_B.pdf
Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3), 1-21.
Burstein, J. (2003). The e-rater scoring engine: Automated essay scoring with natural language processing. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A crossdisciplinary perspective (pp. 113-131). Mahwah, NJ: Lawrence Erlbaum Associates.
Burstein, J., Kukich, K., Wolff, S., Lu, C., Chodorow, M., Braden-Harder, L., & Harris M. D. (1998). Automated scoring using a hybrid feature identification technique. In Proceedings of the Thirty-Sixth Annual Meeting of the Association for Computational Linguistics (pp. 206-210). East Stroudsburg, PA: Association for Computational Linguistics.
Chen, C.-F. E., & Cheng, W.-Y. E. (2008). Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12(2), 94-112. Retrieved from http://llt.msu.edu/vol12num2/chencheng.pdf
Chung, G. K. W. K., & Baker, E. L. (2003). Issues in the reliability and validity of automated scoring of constructed responses. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 23-40). Mahwah, NJ: Lawrence Erlbaum Associates.
Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five approaches. Thousand Oaks, CA: Sage.
Edge, J., & Richards, K. (1998). Why best practice is not good enough. TESOL Quarterly, 32(3), 569-576. doi: 10.2307/3588127
Educational Testing Service. (2011). About the e-rater® scoring engine. Retrieved from http://www.ets.org/erater/about/
Elliott, S. (2003). IntellimetricTM: From here to validity. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 71-86). Mahwah, NJ: Lawrence Erlbaum Associates.
Esterberg, K. G. (2002). Qualitative methods in social research. New York, NY: McGraw-Hill.
Grimes, D., & Waschauer, M. (2006). Learning with laptops: A multi-method case study. Journal of Educational Computing Research (JERIC), 38(3), 305-332. doi: 10.2190/EC.38.3.d
Grimes, D., & Warschauer, M. (2010). Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. Journal of Technology, Learning, and Assessment, 8(6), 4-43. Retrieved from http://files.eric.ed.gov/fulltext/EJ882522.pdf
Hubbard, P. (2004). Learner training for effective use of CALL. In S. Fotos & C. Browne (Eds.), New perspectives on CALL for second language classrooms (pp. 45-69). Mahwah, NJ: Lawrence Erlbaum Associates.
Hubbard, P., & Levy, M. (2006) Teacher education in CALL. Philadelphia, PA: John Benjamins.
Lantolf, J. (2000). Sociocultural theory and second language learning. Oxford: Oxford University Press.
Lantolf, J., & Thorne, S. L. (2006). Sociocultural theory and the genesis of second language development. Oxford: Oxford University Press.
Li, Z., & Hegelheimer, V. (2013). Mobile-assisted grammar exercises: Effects on self-editing in L2 writing. Language Learning and Technology, 17(3), 135-156. Retrieved from http://llt.msu.edu/issues/october2013/lihegelheimer.pdf
Li, Z., Link, S., Ma, H., Yang, H. & Hegelheimer, V. (2013). The role of automated writing evaluation holistic scores in the ESL classroom. Manuscript submitted for publication.
Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
Marshall, C., & Rossman, G. B. (2006). Designing qualitative research (4th ed.). Thousand Oaks, CA: Sage.
McKeon, D. (1998). Best practice: Hype or hope?. TESOL Quarterly, 32(3), 493-501. doi: 10.2307/3588119
Merriam, S. B. (2002). Qualitative research in practice: Examples for discussion and analysis. San Francisco, CA: Jossey-Bass.
Ortega, L. (2007). Second language acquisition explained? SLA across nine contemporary theories. In B. VanPatten & J. Williams (Eds.), Theories in second language acquisition (pp. 225-250). Mahwah, NJ: Lawrence Erlbaum Associates.
Page, E. B. (2003). Project Essay Grade: PEG. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 43-54). Mahwah, NJ: Lwarence Erlbaum Associates.
Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage.
Rogers, M. E. (2003). Diffusion of innovations (5th ed.). New York City, NY: Free Press.
Shermis, M. D., & Burstein, J. (Eds.). (2003). Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Lawrence Erlbaum Associates.
Shermis, M. D., & Hamner, B. (2013). Contrasting state-of-the-art automated scoring of essays. In M. D. Shermis & J. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 313–346). New York, NY: Routledge.
Shermis, M. D., Raymat, M. V., & Barrera, F. (2003, April). Assessing writing through the curriculum with automated essay scoring. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago: IL.
Vantage Learning. (2011). IntelliMetric®. Retrieved from http://www.vantagelearning.com/products/intellimetric/
Wang, Y., Shang, H., & Briody, P. (2012). Exploring the impact of using automated writing evaluation in English as a foreign language university students' writing. Computer Assisted Language Learning 26(3), 234-257. doi: 10.1080/09588221.2012.655300
Ware, P. (2011). Computer-generated feedback on student writing. TESOL Quarterly, 45(4), 769-774. doi: 10.5054/tq.2011.272525
Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal, 3, 22-36. doi: 10.1080/15544800701771580
Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 1-24. doi:10.1191/1362168806lr190oa
Weigle, S. C. (2013). English language learners and automated scoring of essays: Critical considerations. Assessing Writing, 18(1), 85-99.
Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2-13. doi: 10.1111/j.1745-3992.2011.00223.x
Zemelman, S., Daniels, H., & Hyde, A. (2005). Best practices: Today’s standards for teaching and learning in America’s schools. Portsmouth, NH: Heinemann.
