CALICO Journal, Vol 33, No 1 (2016)

Automated Error Detection for Developing Grammar Proficiency of ESL Learners

Hui-Hsien Feng, Aysel Saricaoglu, Evgeny Chukharev-Hudilainen
Issued Date: 30 Jan 2016


Thanks to natural language processing technologies, computer programs are actively being used not only for holistic scoring, but also for formative evaluation of writing. CyWrite is one such program that is under development. The program is built upon Second Language Acquisition theories and aims to assist ESL learners in higher education by providing them with effective formative feedback to facilitate autonomous learning and improvement of their writing skills. In this study, we focus on CyWrite’s capacity to detect grammatical errors in student writing. We specifically report on (1) computational and pedagogical approaches to the development of the tool in terms of students’ grammatical accuracy, and (2) the performance of our grammatical analyzer. We evaluated the performance of CyWrite on a corpus of essays written by ESL undergraduate students with regards to four types of grammatical errors: quantifiers, subject-verb agreement, articles, and run-on sentences. We compared CyWrite’s performance at detecting these errors to the performance of a well-known commercially available AWE tool, Criterion. Our findings demonstrated better performance metrics of our tool as compared to Criterion, and a deeper analysis of false positives and false negatives shed light on how CyWrite’s performance can be improved.

Download Media

PDF Subscribers Only

DOI: 10.1558/cj.v33i1.26507


Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research?. Educational Researcher, 41 (1), 16–25.

Attali, Y. (2004). Exploring the feedback and revision features of Criterion. Paper presented at the National Council on Measurement in Education Annual Meeting, San Diego, CA.

Bender, E. M., Flickinger, D., Oepen, S., Walsh, A. & Baldwin, T. (2004). ARBORETUM: Using a Precision Grammar for Grammar Checking in CALL. In Proc. InSTIL/ICALL Symposium on Computer Assisted Learning, Venice, Italy.

Bitchener, J., & Ferris, D. (2012). Written corrective feedback in second language acquisition and writing. New York: Routledge.

Bramer, M. (2013). Logic programming with prolog. London: Springer.

Burstein, J. (2003). The e-rater scoring engine: Automated Essay Scoring with natural language processing. In M. D. Shermis and J. C. Burstein (Eds), Automated Essay Scoring: A cross disciplinary approach, 113–121. Mahwah, NJ: Lawrence Erlbaum Associates.

Burstein, J., Chodorow, M., & Leacock, C. (2004). CriterionSM online essay evaluation: An application for automated evaluation of student essays. In Proceedings of the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence, Acapulco, Mexico. Retrieved from

Celce-Murcia, M., & Larsen-Freeman, D. (1999). The grammar book: An ESL/EFL teacher's course. Boston, MA: Heinle & Heinle.

Chapelle, C. A., Cotos, E., & Lee, J. (2015). Validity arguments for diagnostic assessment using automated writing evaluation. Language Testing, 32 (2), 385–405.

Chukharev-Hudilainen, E., & Saricaoglu, A. (2014). Causal discourse analyzer: Improving automated feedback on academic ESL writing. Computer Assisted Language Learning.

Connors, R. J., & Lunsford, A. A. (1988). Frequency of formal errors in current college writing, or Ma and Pa Kettle do research. College Composition and Communication, 395–409.

Cowan, R. (2008). The teacher's grammar of English: A course book and reference guide. Cambridge: Cambridge University Press.

Craig, J. L. (2013). Integrating writing strategies in EFL/ESL university contexts: A writing-across-the-curriculum approach. New York: Routledge.

Davies, M. (2010). The corpus of contemporary American English as the first reliable monitor corpus of English. Language and Literary Computing, 25 (4), 447–464.

De Feliece, R. (2008). Automatic error detection in non-native English (Unpublished doctoral dissertation).University of Oxford, England.

De Marneffe, M.C., MacCartney, B., & Manning, C.D. (2006). Generating typed dependency parses from phrase structure parses. In Proceedings of LREC (Vol. 6), 449–454. Genoa: ELRA.

DeCapua, A. (2008). Grammar for teachers: A guide to American English for native and non-native speakers. Boston, MA: Springer.

Echevarria, J., Short, D., & Powers, K. (2006). School reform and standards based education: A model for English-language learners. The Journal of Educational Research, 99, 195–210.

Ferris, D. R. (1999). The case for grammar correction in L2 writing classes. A response to Truscott (1996). Journal of Second Language Writing, 8, 1–10.

Ferris, D. R. (2006). Does error feedback help student writers? New evidence on the short- and long-term effects of written error correction. In K. Hyland & F. Hyland (Eds), Feedback in second language writing: Contexts and issues, 81–104. Cambridge: Cambridge University Press.

Ferris, D. R. (2010). Second language writing research and written corrective feedback in SLA: Intersections and practical applications. Studies in Second Language Acquisition, 32, 181–201.

Ferris, D., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does it need to be? Journal of Second Language Writing, 10, 161–184.

Fitzpatrick, M. (2011). Engaging writing 2: Essential skills for academic writing. White Plains, NY: Pearson Education.

Guide to Grammar and Writing. (n.d.). Retrieved from

Hagerman, C. (2011). An evaluation of Automated Writing Assessment. JALT CALL Journal, 7 (3), 271–292.

Hayes, A.F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1 (1), 77–89.

Heift, T. (2004). Corrective feedback and learner uptake in CALL. ReCALL, 16 (2), 416–431.

Hinkel, E. (2011). What research on second language writing tells us and what it doesn't. In E. Hinkel (Ed.), Handbook of Research in Second Language Teaching and Learning, Volume 2, 523–538. New York: Routledge.

Hung, H-T. (2011). Design-based research: Designing a multimedia environment to support language learning. Innovations in Education and Teaching International, 48, 159–169.

Kelly, A. E., Lesh, R. A., & Baek, J. Y. (2008). Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. New York: Routledge.

Kennedy-Clark, S. (2013). Research by design: Design-based research and the higher degree research student. Journal of Learning Design, 6 (2), 26–32.

Klein, D., & Manning, C.D. (2003). Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics – Volume 1, 423–430. Association for Computational Linguistics. Morristown, NJ: ACL.

Krippendorff, K. (2011). Computing Krippendorff’s alpha reliability. Departmental Papers (ASC), 43, 1–10.

Leacock, C., Chodorow, M., Gamon, M., & Tetreault, J. (2010). Automated grammatical error detection for language learners. Synthesis Lectures on Human Language Technologies, 3 (1), 1–134.

Li, J., Link, S, & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation in ESL writing instruction. Journal of Second Language Writing, 27, 1–18.

Li, Z., Feng, H.-H., & Saricaoglu, A. (in press). The short-term and long-term effects of AWE feedback on ESL learners’ development of grammatical accuracy. CALICO Journal.

Li, Z., Link, S., Ma, H., Yang, H., & Hegelheimer, V. (2014). The role of automated writing evaluation holistic scores in the ESL classroom. System, 44, 66–78.

Link, S., Dursun, A., Karakaya, K., & Hegelheimer, V. (2014). Towards better ESL practices for implementing automated writing evaluation. CALICO Journal, 31 (3).

Lund, A. (2005). Collective epistemologies in an upper secondary school. A preliminary analysis. Paper presented at the EARLI conference, Nicosia, CY.

Lund, A. (2008). Wikis: A collective approach to language production. ReCALL, 20 (1), 35–54.

Lund, A. & Smordal, O. (2006) Is there a space for the teacher in a Wiki? In: Proceedings of the 2006 International Symposium on Wikis (WikiSym '06), 37–46. Odense, Denmark: ACM Press.

Lunsford, A. A. (2012). The everyday writer (5th ed.). Boston, MA: Bedford/St. Martins.

Lunsford, A. A., & Lunsford, K. J. (2008). ‘Mistakes are a fact of life’: A national comparative study. College Composition and Communication, 59 (4), 781–806.

Meurers, D. (2012). Natural language processing and language learning. In C. A. Chapelle, (Ed.), Encyclopedia of Applied Linguistics. Oxford: Wiley-Blackwell.

Nassaji, H., & Fotos, S. (2011). Teaching grammar in second language classrooms. Integrating form-focused instruction in communicative context. London: Routledge.

O’Donnell, M. (2008). The UAM CorpusTool: Software for corpus annotation and exploration. In Proceedings of the XXVI Congreso de AESLA, Almeria, Spain.

Pardo-Ballester, C., & Rodríguez, J. C. (2009). Using design-based research to guide the development of online instructional materials. In C. A. Chapelle, H. G. Jun, & I. Katz (Eds), Developing and evaluating language learning materials, 86–102. Ames, IA: Iowa State University.

Pardo-Ballester, C., & Rodríguez, J. C. (2010). Developing Spanish online readings using design-based research. CALICO Journal, 27, 540–553.

Pardo-Ballester, C., & Rodríguez, J. C. (2013). Design principles for language learning activities in synthetic environments. In J. Rodríguez & M. Pardo-Ballester (Eds), Design-Based Research in CALL, 183–209. San Marcos, TX: CALICO.

Plomp, T. (2009). Educational design research: An introduction. In T. Plomp & N. Nieveen (Eds), An introduction to educational design research, 9–35. Enschede: The Netherlands: SLO Netherlands Institute for Curriculum Development.

Ranalli, J., Link, S., & Chukharev-Hudilainen, E. (2014). AWE for formative assessment: Investigating accuracy and efficiency as part of argument-based validation. Paper presented at The 3rd Teachers College, Columbia University Roundtable in Second Language Studies, New York.

Richards, J. C., & Rodgers, T. S. (2001). Approaches and methods in language teaching (2nd ed.). Cambridge: Cambridge University Press.

Rodríguez, J. C. & Pardo-Ballester, C., (2013) (Eds). Design-based Research in CALL. CALICO Monograph Series, Volume 8. San Marcos, TX: CALICO.

Run-on sentence. (n.d.). In Merriam-Webster Online Dictionary. Retrieved from

Schneider, D., & McCoy, K. F. (1998). Recognizing syntactic errors in the writing of second language learners. In Proceedings of the 17th international conference on Computational linguistics-Volume 2, 1198–1204. Association for Computational Linguistics.

Shutler, R. (2012). A study of student and teacher perceptions of criterion, an online writing program (Unpublished master's thesis). Carleton University: Ottawa, CA.

Strijbos, J.-W., & Stahl, G. (2007). Methodological issues in developing a multi-dimensional codingprocedure for small-group chat communication. Learning and Instruction, 17 (4), 394–404.

Sun, X. (2014). Analysis on negative transfer of native language syntax structure in English compositions of Chinese college students. In 3rd International Conference on Science and Social Research (ICSSR 2014).

The Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32 (1), 5–8.

Wang, F., & Hannafin, M. J. (2005). Design-based research and technology enhanced learning environments. Educational Technology Research and Development, 53 (4), 5–23.

Wu, H. P., & Garza, E. V. (2014). Types and attributes of English writing errors in the EFL context – A study of error analysis. Journal of Language Teaching and Research, 5 (6), 1256–1262.

Yuan, Z., & Felice, M. (2013). Constrained grammatical error correction using statistical machine translation. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning (CoNLL 2013): Shared Task, 52–61. Madison, WI: Omnipress.

Yutdhana, S. (2005a). Design-based research in CALL. In J. L. Egbert & G. Mikel Petrie (Eds), CALL research perspectives, 169–178. Mahwah, NJ: Lawrence Erlbaum Associates.

Yutdhana, S. (2005b). The development of a teacher-training for model in using the Internet for teaching English as a foreign language. (Unpublished doctoral dissertation). Suranaree University of Technology, Thailand.


  • There are currently no refbacks.

Equinox Publishing Ltd - 415 The Workstation 15 Paternoster Row, Sheffield, S1 2BX United Kingdom
Telephone: +44 (0)114 221-0285 - Email:

Privacy Policy