Informing Automated Writing Evaluation Using the Lens of Genre: Two Studies

Authors

  • Jill Burstein ETS
  • Norbert Elliot New Jersey Institute of Technology
  • Hillary Molloy ETS

DOI:

https://doi.org/10.1558/cj.v33i1.26374

Keywords:

AWE, automated writing evaluation, natural language processing, writing research

Abstract

Genre serves as a useful lens to investigate the range of evidence derived from automated writing evaluation (AWE). To support construct-relevant systems used for writing instruction and assessment, two investigations were conducted that focused on post-secondary writing requirements and faculty perceptions of student writing proficiency. Survey research is described from a national study and a second site study at American University, a 4-year private university in Washington, DC, to illustrate writing requirements and perceptions of writing proficiency in school and workplace settings. A mixed-methods analysis of faculty focus groups in the site study afforded more detailed discussions that were used to understand student writing support needs. Through the lens of genre, study results illustrated how the role of AWE might be expanded and aligned with instruction in four-year post-secondary institutions.

Author Biographies

  • Jill Burstein, ETS
    Jill Burstein, Ph.D., is the Research Director for the Natural Language Processing Group in ETS’ Research & Development division. Her background and expertise is in natural language processing (NLP) with a focus on building NLP systems in the service of learning. Her inventions include e-rater®, an automated essay scoring system; an essay-based discourse analysis system; and Language Muse Activity Palette™, an NLP-driven instructional authoring tool to support teachers of English learners. Her research interests include NLP systems that evaluate discourse coherence, argumentation, and use of sources in writing. Dr Burstein holds a number of patents for NLP-based, educational technology inventions. Dr Burstein received her BA in Linguistics and Spanish from New York University and her MA and Ph.D in Linguistics from the Graduate Center, the City University of New York.
  • Norbert Elliot, New Jersey Institute of Technology
    Norbert Elliot, Ph.D, is professor emeritus of English at New Jersey Institute of Technology. His research focuses on the history, theory, and measurement of writing ability. He currently serves on the editorial boards of Assessing Writing, IEEE Transaction on Professional and Technical Communication, Research in the Teaching of English, and WPA: Writing Program Administration. With Edward M. White and Irvin Peckham, he is co-author, most recently, of Very Like a Whale: The Assessment of Writing Programs (Utah State University Press, 2015).
  • Hillary Molloy, ETS
    Hillary Molloy is a research assistant at Educational Testing Service in Princeton, NJ. She primarily supports the Natural Language Processing and Speech group, but has also worked extensively with the Foundational and Validity Research group and the Center for Academic and Workforce Readiness and Success. Since starting at ETS, she has been involved with the research and development of different features for e-rater® (an automated essay scoring system), projects examining the validity and fairness of global assessments, and the development of a workforce curriculum funded by the Trade Adjustment Assistance Community College and Career Training grant. She graduated summa cum laude from Dickinson College in Carlisle, PA, with a major in Anthropology and a minor in French and Francophone studies.

References

ACT. (2005). Crisis at the core: Preparing all students for college and work. Iowa City, IA: ACT. Retrieved from http://www.act.org/research/policymakers/pdf/crisis_report.pdf

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Applebee, A. N., & Langer, J. A. (1984). Contexts for learning to write: Studies of secondary school instruction. Norwood, NJ: Ablex.

Applebee, A.N., & Langer, J. A. (2009). What is happening in the teaching of writing? English Journal, 95, 19–29.

Applebee, A.N., Lehr, F., & Austen, A. (1981). Learning to write in the secondary school: How and where. English Journal, 70, 78–82. http://dx.doi.org/10.2307/817387

Appleman, D., & Green, D. E. (1993). Mapping the elusive boundary between high school and college writing. College Composition and Communication, 44, 191–199. http://dx.doi.org/10.2307/358838

Association for Colleges and Research Libraries. (2014). Framework for information literacy for higher education. Chicago, IL: American Library Association. Retrieved from http://acrl.ala.org/ilstandards/wp-content/uploads/2014/11/Framework-for-IL-for-HE-draft-3.pdf

Banks, A. (2015). Ain’t No Walls behind the sky, baby! Funk, flight, freedom. College Composition and Communication, 67, 267–279.

Bazerman, C. (2010). A rhetoric of literate action: Literate action (vol. 1). Fort Collins, CO: WAC Clearinghouse and Anderson: Parlor Press.

Bazerman, C., & Paradis, A. (Eds.) (1991). Textual dynamics of the professions: Historical and contemporary studies of writing in professional communities. Madison, WI: University of Wisconsin Press.

Beck, S.W., & Jeffery, J.V. (2009). Genre and thinking in academic writing tasks. Journal of Literacy Research, 41, 228–272. http://dx.doi.org/10.1080/10862960902908483

Beigman-Klebanov, B. & Flor, M. (2013a). Argumentation-relevant metaphors in test-taker essays. In Proceedings of the First Workshop on Metaphor in NLP, Annual Meeting of the Association of Computational Linguistics, 11–20. Atlanta, GA: Association of Computational Linguistics.

Beigman Klebanov, B., & Flor, M. (2013b). Word association profiles and their use for automated scoring of essays. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 1148–1158. Sofia, BG: Association for Computational Linguistics.

Beigman Klebanov, B., Madnani, N., Burstein, J., and Somasundaran, S. (2014). Content importance models for scoring writing from sources. In Proceedings of the Annual Meeting of the Association of Computational Linguistics, 247–252. Baltimore, MD: Association of Computational Linguistics. http://dx.doi.org/10.3115/v1/p14-2041

Biber, D., Nekrasova, T., & Horn, B. (2011). The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis (RR-11-05, TOEFLiBT-14). Princeton, NJ: Educational Testing Service. Retrieved https://www.ets.org/Media/Research/pdf/RR-11-05.pdf

Bridgeman, B., & Carlson, S. (1984). Survey of academic writing tasks. Written Communication, 1, 247–280. http://dx.doi.org/10.1177/0741088384001002004

Britton, J. (1992). Language and learning: The importance of speech in children’s development. London: Penguin.

Britton, J., Burgess, T., Martin, N., McLeod, A., & Rosen, H. (1975). The development of writing abilities, 11–18. Urbana, IL: National Council of Teachers of English.

Burstein, J., Elliot, N., Lentini, J, Molloy, H., Shore, J., Steinberg, J. & Vezzu, M. (2014). Investigating genre for writing measurement and automated writing evaluation, Coordinated Session at the Annual Meeting of the National Council on Educational Measurement, April 2014. Retrieved from http://jillburstein.com/uploads/Genre-AES_NCME2014.Final.pdf.

Burstein, J., Tetreault, J., & Chodorow, M. (2013). Holistic annotation of discourse coherence quality in noisy essay writing. [Special issue]. Dialogue and Discourse, 4. Retrieved from http://elanguage.net/journals/dad/article/view/2825/3568

Casner-Lotto, J., & Barrington, L. (2006). Are they really ready to work? Employers’ perspectives on the basic knowledge and applied skills of new entrants to the 21st century US workforce. New York: The Conference Board. Retrieved from http://www.conference-board.org/pdf_free/BED-06-Workforce.pdf

Coppola, N. (2010). The technical communication Body of Knowledge initiative: An academic-practitioner partnership. Technical Communication, 57, 11–25.

Cotos, E. (2011). Potential of automated writing evaluation feedback. CALICO Journal, 28, 420–459. Retrieved from https://www.calico.org http://dx.doi.org/10.11139/cj.28.2.420-459

Council of Writing Program Administrators. (2014). WPA outcomes statement for first-year composition (Revisions adopted 17 July 2014). WPA: Journal of Writing Program Administration, 38, 142–146.

Crusius, T. W. (1989). Discourse: A critique and synthesis of major theories. New York: Modern Language Association.

Eblen, C. (1983). Writing across-the-curriculum: A survey of a university faculty’s views and classroom practices. Research in the Teaching of English, 17, 343–348.

Elliot, N., & Kilduff, M. (1991). Technical writing in a technological university: Attitudes of department chairs. Journal of Technical Writing and Communication, 21, 411–424.

Faigley, L., & Romano, S. (1995). Going electronic: Creating multiple sites for innovation in a writing program. In J. Janangelo & K. Hansen (Eds), Resituating writing: Constructing and administering writing programs, 46–58. Portsmouth, NH: Boynton/Cook, 1995.

Ferris, D.R. (2014). Responding to student writing: Teachers’ philosophies and practices. Assessing Writing, 19, 6–23. http://dx.doi.org/10.1016/j.asw.2013.09.004

Gardner, S. & Nesi, H. (2013). A classification of genre families in university student writing. Applied Linguistics, 34, 25–52. http://dx.doi.org/10.1093/applin/ams024

Gere, A. R., Aull, L., Escudero, Lancaster, & Lei (2013). Local assessment: Using genre analysis to validate directed self-Placement. College Composition and Communication, 64, 605–633.

Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99, 445–476. http://dx.doi.org/10.1037/0022-0663.99.3.445

Harris, J., & Hult, C. (1985). Using a survey of writing assignments to make informed curricular decisions. WPA: Journal of Writing Program Administration, 8, 7–14.

Heath, S. B. (1993). Rethinking the sense of the past: The essay as legacy of the epigram. In L. Odell (Ed.), Theory and practice in the teaching of writing: Rethinking the discipline, 105–131. Carbondale, IL: Southern Illinois University Press.

Herrington, A. (1985). Writing in academic settings: A study of the contexts for writing in two college chemical engineering courses. Research the Teaching of English, 19, 331–359.

Horowitz, D. (1986). What professors actually require: Academic tasks for the ESL classroom. TESOL Quarterly, 20, 445–462. http://dx.doi.org/10.2307/3586294

Jenkins, S., Jordan, M. K., & Weiland, P. O. (1993). The role of writing in graduate engineering education: A survey of faculty beliefs and practices. English for Specific Purposes, 12, 51–67. http://dx.doi.org/10.1016/0889-4906(93)90027-L

Kane, M. T. (2013). Validating the interpretation and uses of test scores. Journal of Educational Measurement, 50, 1–73. http://dx.doi.org/10.1111/jedm.12000

Kiuhara, S., Graham, S., & Hawken, L. (2009). Teaching writing to high school students: A national survey. Journal of Educational Psychology, 101, 136–160.

Krueger, R. A. & Casey, M. S. (2000). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage. http://dx.doi.org/10.1037/10518-189

Melzer, D. (2009). Writing assignments across the curriculum: A national study of college writing. College Composition and Communication, 61, W240–W261. Retrieved from http://www.ncte.org/cccc/ccc/issues/v61-2

Melzer, D. (2014). Assignments across the curriculum: A national study of college writing. Logan, UT: Utah State University Press.

Miller, C. (1984). Genre as social action. Quarterly Journal of Speech, 70, 151–167. http://dx.doi.org/10.1080/00335638409383686

Miller, C. (1994). In A. Freedman & P. Medway (Eds), Rhetorical community: The cultural basis of genre, 67–97. London: Taylor & Francis.

National Center for Education Statistics (2012). The Nation’s Report Card: Writing 2011 (NCES 2012–470). Washington, DC: Institute of Education Sciences, U.S. Department of Education, Washington, D.C. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012470.pdf

National Governors Association. (2014). Common core state standards initiative. Washington, DC: National Governors Association. Retrieved from http://www.corestandards.org

National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Committee on Defining Deeper Learning and 21st Century Skills, J.W. Pellegrino & M.L. Hilton (Eds). Board on Testing and Assessment and Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Odell, L., Goswami, D., & Herrington, A. (1983). The discourse-based interview: A procedure for exploring tacit knowledge of writers in non-academic settings. In P. Mosenthal, L. Tamor, & S. Walmsley (Eds), Research on writing: Principles and methods, 220–236. New York: Longman.

Pimentel, S. (2013). College and career readiness standards for adult education. Washington, DC: US Department of Education, Office of Vocational and Adult Education. Retrieved from https://lincs.ed.gov/publications/pdf/CCRStandardsAdultEd.pdf

Relles, S. R., & Tierney, W.G. (2014). Understanding the writing habits of tomorrow’s students: Technology and college readiness. Journal of Higher Education, 84, 477–505. http://dx.doi.org/10.1353/jhe.2013.0025

Rose, M. (1983). Remedial writing courses: A critique and a proposal. College English, 45, 109–128. http://dx.doi.org/10.2307/377219

Shermis, M., Burstein, J., Elliot, N., Miel, S., & Foltz, P. (2016). Automated writing evaluation: A growing body of knowledge. In C. MacArthur, S. Graham, & J. Fitzgerald (Eds), Handbook of writing research, 395–409. New York, NY: Guilford Press.

Somasundaran, S., Burstein, J., & Chodorow, M. (2014, August). Lexical chaining for measuring discourse coherence quality in test-taker essays. Paper presented at The 25th International Conference on Computational Linguistics (COLING), Dublin, IE.

Song, Y., Heilman, M., Beigman Klebanov, B., & Deane, P. (2014). Applying argumentation schemes for essay scoring. In Proceedings of the 1st workshop on Argumentation Mining, 69–78. Baltimore, MD: Association for Computational Linguistics. http://dx.doi.org/10.3115/v1/W14-2110

Sparks, J. R., Song, Y., Brantley, W., & Liu, O. L. (2014). Assessing written communication in higher education: Review and recommendations for next-generation assessment (ETS Research Report No. RR-14-37). Princeton, NJ: Educational Testing Service.

Thrift, N. (2005). Knowing capitalism. London: Sage.

Wardle, E. (2009). ‘Mutt genres’ and the goal of FYC: Can we help students write the genres of the university? College Composition and Communication, 60, 765–789.

Downloads

Published

2016-01-30

Issue

Section

Articles

How to Cite

Burstein, J., Elliot, N., & Molloy, H. (2016). Informing Automated Writing Evaluation Using the Lens of Genre: Two Studies. CALICO Journal, 33(1), 117-141. https://doi.org/10.1558/cj.v33i1.26374

Most read articles by the same author(s)

1 2 3 4 5 6 7 8 9 10 > >>