Automated Error Detection for Developing Grammar Proficiency of ESL Learners
DOI:
https://doi.org/10.1558/cj.v33i1.26507Keywords:
automated writing evaluation, formative feedback, grammatical errors, error correction, evaluation of AWE systems, design based researchAbstract
Thanks to natural language processing technologies, computer programs are actively being used not only for holistic scoring, but also for formative evaluation of writing. CyWrite is one such program that is under development. The program is built upon Second Language Acquisition theories and aims to assist ESL learners in higher education by providing them with effective formative feedback to facilitate autonomous learning and improvement of their writing skills. In this study, we focus on CyWrite’s capacity to detect grammatical errors in student writing. We specifically report on (1) computational and pedagogical approaches to the development of the tool in terms of students’ grammatical accuracy, and (2) the performance of our grammatical analyzer. We evaluated the performance of CyWrite on a corpus of essays written by ESL undergraduate students with regards to four types of grammatical errors: quantifiers, subject-verb agreement, articles, and run-on sentences. We compared CyWrite’s performance at detecting these errors to the performance of a well-known commercially available AWE tool, Criterion. Our findings demonstrated better performance metrics of our tool as compared to Criterion, and a deeper analysis of false positives and false negatives shed light on how CyWrite’s performance can be improved.References
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research?. Educational Researcher, 41 (1), 16–25. http://dx.doi.org/10.3102/0013189X11428813
Attali, Y. (2004). Exploring the feedback and revision features of Criterion. Paper presented at the National Council on Measurement in Education Annual Meeting, San Diego, CA.
Bender, E. M., Flickinger, D., Oepen, S., Walsh, A. & Baldwin, T. (2004). ARBORETUM: Using a Precision Grammar for Grammar Checking in CALL. In Proc. InSTIL/ICALL Symposium on Computer Assisted Learning, Venice, Italy.
Bitchener, J., & Ferris, D. (2012). Written corrective feedback in second language acquisition and writing. New York: Routledge.
Bramer, M. (2013). Logic programming with prolog. London: Springer. http://dx.doi.org/10.1007/978-1-4471-5487-7
Burstein, J. (2003). The e-rater scoring engine: Automated Essay Scoring with natural language processing. In M. D. Shermis and J. C. Burstein (Eds), Automated Essay Scoring: A cross disciplinary approach, 113–121. Mahwah, NJ: Lawrence Erlbaum Associates.
Burstein, J., Chodorow, M., & Leacock, C. (2004). CriterionSM online essay evaluation: An application for automated evaluation of student essays. In Proceedings of the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence, Acapulco, Mexico. Retrieved from http://www.ets.org/research/policy_research_reports/publications/chapter/2004/cwjd
Celce-Murcia, M., & Larsen-Freeman, D. (1999). The grammar book: An ESL/EFL teacher's course. Boston, MA: Heinle & Heinle.
Chapelle, C. A., Cotos, E., & Lee, J. (2015). Validity arguments for diagnostic assessment using automated writing evaluation. Language Testing, 32 (2), 385–405.
Chukharev-Hudilainen, E., & Saricaoglu, A. (2014). Causal discourse analyzer: Improving automated feedback on academic ESL writing. Computer Assisted Language Learning. http://dx.doi.org/10.1080/09588221.2014.991795
Connors, R. J., & Lunsford, A. A. (1988). Frequency of formal errors in current college writing, or Ma and Pa Kettle do research. College Composition and Communication, 395–409. http://dx.doi.org/10.2307/357695
Cowan, R. (2008). The teacher's grammar of English: A course book and reference guide. Cambridge: Cambridge University Press.
Craig, J. L. (2013). Integrating writing strategies in EFL/ESL university contexts: A writing-across-the-curriculum approach. New York: Routledge.
Davies, M. (2010). The corpus of contemporary American English as the first reliable monitor corpus of English. Language and Literary Computing, 25 (4), 447–464. http://dx.doi.org/10.1093/llc/fqq018
De Feliece, R. (2008). Automatic error detection in non-native English (Unpublished doctoral dissertation).University of Oxford, England.
De Marneffe, M.C., MacCartney, B., & Manning, C.D. (2006). Generating typed dependency parses from phrase structure parses. In Proceedings of LREC (Vol. 6), 449–454. Genoa: ELRA.
DeCapua, A. (2008). Grammar for teachers: A guide to American English for native and non-native speakers. Boston, MA: Springer. http://dx.doi.org/10.1007/978-0-387-76332-3
Echevarria, J., Short, D., & Powers, K. (2006). School reform and standards based education: A model for English-language learners. The Journal of Educational Research, 99, 195–210. http://dx.doi.org/10.3200/JOER.99.4.195-211
Ferris, D. R. (1999). The case for grammar correction in L2 writing classes. A response to Truscott (1996). Journal of Second Language Writing, 8, 1–10. http://dx.doi.org/10.1016/S1060-3743(99)80110-6
Ferris, D. R. (2006). Does error feedback help student writers? New evidence on the short- and long-term effects of written error correction. In K. Hyland & F. Hyland (Eds), Feedback in second language writing: Contexts and issues, 81–104. Cambridge: Cambridge University Press. http://dx.doi.org/10.1017/CBO9781139524742.007
Ferris, D. R. (2010). Second language writing research and written corrective feedback in SLA: Intersections and practical applications. Studies in Second Language Acquisition, 32, 181–201. http://dx.doi.org/10.1017/S0272263109990490
Ferris, D., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does it need to be? Journal of Second Language Writing, 10, 161–184.
Fitzpatrick, M. (2011). Engaging writing 2: Essential skills for academic writing. White Plains, NY: Pearson Education.
Guide to Grammar and Writing. (n.d.). Retrieved from https://www.guidetogrammar.org/grammar/
Hagerman, C. (2011). An evaluation of Automated Writing Assessment. JALT CALL Journal, 7 (3), 271–292.
Hayes, A.F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1 (1), 77–89. http://dx.doi.org/10.1080/19312450709336664
Heift, T. (2004). Corrective feedback and learner uptake in CALL. ReCALL, 16 (2), 416–431. http://dx.doi.org/10.1017/S0958344004001120
Hinkel, E. (2011). What research on second language writing tells us and what it doesn't. In E. Hinkel (Ed.), Handbook of Research in Second Language Teaching and Learning, Volume 2, 523–538. New York: Routledge.
Hung, H-T. (2011). Design-based research: Designing a multimedia environment to support language learning. Innovations in Education and Teaching International, 48, 159–169. http://dx.doi.org/10.1080/14703297.2011.564011
Kelly, A. E., Lesh, R. A., & Baek, J. Y. (2008). Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. New York: Routledge.
Kennedy-Clark, S. (2013). Research by design: Design-based research and the higher degree research student. Journal of Learning Design, 6 (2), 26–32. http://dx.doi.org/10.5204/jld.v6i2.128
Klein, D., & Manning, C.D. (2003). Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics – Volume 1, 423–430. Association for Computational Linguistics. Morristown, NJ: ACL.
Krippendorff, K. (2011). Computing Krippendorff’s alpha reliability. Departmental Papers (ASC), 43, 1–10.
Leacock, C., Chodorow, M., Gamon, M., & Tetreault, J. (2010). Automated grammatical error detection for language learners. Synthesis Lectures on Human Language Technologies, 3 (1), 1–134. http://dx.doi.org/10.2200/S00275ED1V01Y201006HLT009
Li, J., Link, S, & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation in ESL writing instruction. Journal of Second Language Writing, 27, 1–18. http://dx.doi.org/10.1016/j.jslw.2014.10.004
Li, Z., Feng, H.-H., & Saricaoglu, A. (in press). The short-term and long-term effects of AWE feedback on ESL learners’ development of grammatical accuracy. CALICO Journal.
Li, Z., Link, S., Ma, H., Yang, H., & Hegelheimer, V. (2014). The role of automated writing evaluation holistic scores in the ESL classroom. System, 44, 66–78. http://dx.doi.org/10.1016/j.system.2014.02.007
Link, S., Dursun, A., Karakaya, K., & Hegelheimer, V. (2014). Towards better ESL practices for implementing automated writing evaluation. CALICO Journal, 31 (3). http://dx.doi.org/10.11139/cj.31.3.323-344
Lund, A. (2005). Collective epistemologies in an upper secondary school. A preliminary analysis. Paper presented at the EARLI conference, Nicosia, CY.
Lund, A. (2008). Wikis: A collective approach to language production. ReCALL, 20 (1), 35–54. http://dx.doi.org/10.1017/S0958344008000414
Lund, A. & Smordal, O. (2006) Is there a space for the teacher in a Wiki? In: Proceedings of the 2006 International Symposium on Wikis (WikiSym '06), 37–46. Odense, Denmark: ACM Press.
Lunsford, A. A. (2012). The everyday writer (5th ed.). Boston, MA: Bedford/St. Martins.
Lunsford, A. A., & Lunsford, K. J. (2008). ‘Mistakes are a fact of life’: A national comparative study. College Composition and Communication, 59 (4), 781–806.
Meurers, D. (2012). Natural language processing and language learning. In C. A. Chapelle, (Ed.), Encyclopedia of Applied Linguistics. Oxford: Wiley-Blackwell.
Nassaji, H., & Fotos, S. (2011). Teaching grammar in second language classrooms. Integrating form-focused instruction in communicative context. London: Routledge.
O’Donnell, M. (2008). The UAM CorpusTool: Software for corpus annotation and exploration. In Proceedings of the XXVI Congreso de AESLA, Almeria, Spain.
Pardo-Ballester, C., & Rodríguez, J. C. (2009). Using design-based research to guide the development of online instructional materials. In C. A. Chapelle, H. G. Jun, & I. Katz (Eds), Developing and evaluating language learning materials, 86–102. Ames, IA: Iowa State University.
Pardo-Ballester, C., & Rodríguez, J. C. (2010). Developing Spanish online readings using design-based research. CALICO Journal, 27, 540–553. http://dx.doi.org/10.11139/cj.27.3.540-553
Pardo-Ballester, C., & Rodríguez, J. C. (2013). Design principles for language learning activities in synthetic environments. In J. Rodri?guez & M. Pardo-Ballester (Eds), Design-Based Research in CALL, 183–209. San Marcos, TX: CALICO.
Plomp, T. (2009). Educational design research: An introduction. In T. Plomp & N. Nieveen (Eds), An introduction to educational design research, 9–35. Enschede: The Netherlands: SLO Netherlands Institute for Curriculum Development.
Ranalli, J., Link, S., & Chukharev-Hudilainen, E. (2014). AWE for formative assessment: Investigating accuracy and efficiency as part of argument-based validation. Paper presented at The 3rd Teachers College, Columbia University Roundtable in Second Language Studies, New York.
Richards, J. C., & Rodgers, T. S. (2001). Approaches and methods in language teaching (2nd ed.). Cambridge: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511667305
Rodríguez, J. C. & Pardo-Ballester, C., (2013) (Eds). Design-based Research in CALL. CALICO Monograph Series, Volume 8. San Marcos, TX: CALICO.
Run-on sentence. (n.d.). In Merriam-Webster Online Dictionary. Retrieved from http://www.merriam-webster.com/dictionary/run-on+sentence
Schneider, D., & McCoy, K. F. (1998). Recognizing syntactic errors in the writing of second language learners. In Proceedings of the 17th international conference on Computational linguistics-Volume 2, 1198–1204. Association for Computational Linguistics. http://dx.doi.org/10.3115/980432.980765
Shutler, R. (2012). A study of student and teacher perceptions of criterion, an online writing program (Unpublished master's thesis). Carleton University: Ottawa, CA.
Strijbos, J.-W., & Stahl, G. (2007). Methodological issues in developing a multi-dimensional codingprocedure for small-group chat communication. Learning and Instruction, 17 (4), 394–404. http://dx.doi.org/10.1016/j.learninstruc.2007.03.005
Sun, X. (2014). Analysis on negative transfer of native language syntax structure in English compositions of Chinese college students. In 3rd International Conference on Science and Social Research (ICSSR 2014). http://dx.doi.org/10.2991/icssr-14.2014.229
The Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32 (1), 5–8. http://dx.doi.org/10.3102/0013189X032001005
Wang, F., & Hannafin, M. J. (2005). Design-based research and technology enhanced learning environments. Educational Technology Research and Development, 53 (4), 5–23. http://dx.doi.org/10.1007/BF02504682
Wu, H. P., & Garza, E. V. (2014). Types and attributes of English writing errors in the EFL context – A study of error analysis. Journal of Language Teaching and Research, 5 (6), 1256–1262. http://dx.doi.org/10.4304/jltr.5.6.1256-1262
Yuan, Z., & Felice, M. (2013). Constrained grammatical error correction using statistical machine translation. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning (CoNLL 2013): Shared Task, 52–61. Madison, WI: Omnipress.
Yutdhana, S. (2005a). Design-based research in CALL. In J. L. Egbert & G. Mikel Petrie (Eds), CALL research perspectives, 169–178. Mahwah, NJ: Lawrence Erlbaum Associates.
Yutdhana, S. (2005b). The development of a teacher-training for model in using the Internet for teaching English as a foreign language. (Unpublished doctoral dissertation). Suranaree University of Technology, Thailand.
