Creating and validating assessment instruments for a discipline-specific writing course
DOI:
https://doi.org/10.1558/japl.v2.i1.75Keywords:
Writing Assessment, Disciplinary Writing, Holistic Assessments, Analytic Scales, Interdisciplinary Research, Mediating Role of Applied LinguisticsAbstract
This paper reports on a sustained interdisciplinary effort between applied linguists and chemistry faculty to create and validate writing assessment instruments for an advanced-level Write Like a Chemist course, one component of a much larger interdisciplinary project. The article describes a multiple-year effort to form valid analytic and holistic assessment instruments to be used by chemistry faculty to assess the writing performance of chemistry majors. Emphasis is placed on the joint contributions of applied linguists and chemists in (a) the identification of meaningful writing criteria, (b) the development of assessment scales with distinct score points and descriptors, (c) socialization sessions that prepared chemists to help build the assessment instruments, and (d) the validation of assessment instruments with other chemists. Outcomes exemplify the mediating role that applied linguistics can play in the design of a discipline-specific course, instructional materials, and assessment instruments that support the development of disciplinary expertise. The results also demonstrate the positive consequences of crossing disciplinary boundaries for both subject-area faculty and applied linguists
References
Bachman, L. and Palmer, A. (1996) Language Testing in Practice. New York: Oxford University Press.
Beall, H. and Trimbur, J. (2001) A Short Guide to Writing about Chemistry. (Second edition) New York: Longman.
Bressette, A. R. and Breton, G. W. (2001) Using writing to enhance the undergraduate research experience. Journal of Chemical Education 78: 1626–7.
Brown, A. (1995) The effect of rater variables in the development of an occupationspecific language performance test. Language Testing 12: 1–15.
Candlin, C. N. and Candlin, S. (2003) Health care communication: a problematic site for applied linguistics research. In M. McGroarty (ed.) Annual Review of Applied Linguistics 134–54. New York: Cambridge University Press.
Candlin, C. N. and Maley, Y. (1997) Intertextuality and interdiscursivity in the discourse of alternative dispute resolution. In B.-L.
Gunnarson, P. Linell and B. Nordberg (eds) The Construction of Professional Discourse 201–22. London: Longman.
Candlin, C. N. and Sarangi, S. (2004) Making applied linguistics matter. Journal of Applied Linguistics 1(1): 1–8.
Coppola, B. P. and Daniels, D. S. (1996) The role of written and verbal expression in improving communication skills for students in an undergraduate chemistry program. Language and Learning Across the Disciplines 1: 67–86.
Dodd, J. S. (ed.) (1997) The ACS Style Guide. (Second edition) Washington, DC: American Chemical Society.
Douglas, D. (2000) Assessing Language for Specific Purposes. New York: Cambridge University Press.
Douglas, D. (2001) Language for specific purposes assessment criteria: where do they come from? Language Testing 18: 171–85.
Dudley-Evans, T. and St. John, M. J. (1998) Developments in English for Specific Purposes: a multi-disciplinary approach. New York: Cambridge University Press.
Ebel, H. F., Bliefert, C. and Russey, W. E. (2001) The Art of Scientific Writing: from student reports to professional publications in chemistry and related fields. (Second edition) New York: John Wiley.
Ferris, D. R. and Hedgcock, J. S. (2005) Teaching ESL Composition: purpose, process, and practice. (Second edition) Mahwah, NJ: Lawrence Erlbaum.
Flowerdew, J. (1993) Content-based language instruction in a tertiary setting. English for Specific Purposes 12: 121–38.
Flowerdew, J. and Peacock, M. (eds) (2001a) Research Perspectives on English for Academic Purposes. New York: Cambridge University Press.
Flowerdew, J. and Peacock, M. (2001b) The EAP curriculum: issues, methods, and challenges. In J. Flowerdew and M. Peacock (eds) Research Perspectives on English for Academic Purposes 177–94. New York: Cambridge University Press.
Gordon, N. R., Newton, T. A., Rhodes, G., Ricci, J. S., Stebbins, R. G. and Tracy, H. J. (2001) Writing and computing across the USM chemistry curriculum. Journal of Chemical Education 78: 53–5.
Hamp-Lyons, L. (2003) Writing teachers as assessors of writing. In B. Kroll (ed.) Second Language Writing: research insights for the classroom. New York: Cambridge University Press.
Henning, G. and Davidson, F. (1987) Scalar analysis of composition ratings. In K. M. Bailey, T. L. Dale, and R. T. Clifford (eds) Language Testing Research: selected papers from the 1986 colloquium. Monterey, CA: Defense Language Institute.
Hudson, T. (2005) Trends in assessment scales and criterion-referenced language assessment. In M. McGroarty (ed.) Annual Review of Applied Linguistics 205–27. New York: Cambridge University Press.
Hyland, K. (2002) Teaching and Researching Writing. London: Longman.
Hyland, K. (2003) Second Language Writing. New York: Cambridge University Press.
Hyland, K. (2004) Disciplinary Discourses: social interactions in academic writing. Ann Arbor: University of Michigan Press.
Jacoby, S. and McNamara, T. (1999) Locating competence. English for Specific Purposes 18: 213–41.
Johns, A. M. (1997) Text, Role, and Context: developing academic literacies. New York: Cambridge University Press.
Klein, B. and Aller, B. M. (1998) Writing across the curriculum in college chemistry: a practical bibliography. Language and Learning Across the Disciplines 2: 25–35.
Kovac, J. and Sherwood, D. W. (2001) Writing Across the Chemistry Curriculum: an instructor’s guide. New York: Prentice Hall College Division.
Kuldell, N. (2003) Read like a scientist to write like a scientist: using authentic literature in the classroom. Journal of College Science Teaching XXXIII(2): 32–5.
Lumley, T. (1998) Perceptions of language-trained raters and occupational experts in a test of occupational English language proficiency. English for Specific Purposes 17: 347–67.
Lumley, T. (2002) Assessment criteria in a large-scale writing test: what do they really mean to the raters? Language Testing 19: 246–76.
McNamara, T. (1996) Measuring Second Language Performance. New York: Longman.
Nature (2001) Learning to speak and write. Nature 411: 1.
North, B. and Schneider, G. (1998) Scaling descriptors for language proficiency scales. Language Testing 15: 217–63.
Oliver-Hoyo, M. T. (2003) Designing a written assignment to promote the use of critical thinking skills in an introductory chemistry course. Journal of Chemical Education 80: 899–903.
Paulson, D. R. (2001) Writing for chemists: satisfying the CSU upper-division writing requirement. Journal of Chemical Education 78: 1047–9.
Sarangi S. and Candlin, C. N. (2003) Trading between reflexivity and relevance: new challenges for applied linguistics. Applied Linguistics 24: 271–85.
Sarangi, S. and Roberts, C. (eds) (1999) Talk, Work, and Institutional Order: discourse in medical, mediation, and management settings. Berlin: Mouton de Gruyter.
Shibley, I. A., Milakofsky, L. M. and Nicotera, C. L. (2001) Incorporating a substantial writing assignment into organic chemistry: library research, peer review, and assessment. Journal of Chemical Education 78: 50–3.
Smith, S. (2003a) The role of technical expertise in engineering and writing teachers’ evaluations of students’ writing. Written Communication 20: 37–80.
Smith, S. (2003b) What is ‘good’ technical communication? A comparison of the standards of writing and engineering instructors. Technical Communication Quarterly 12: 7–24.
Stoller, F. L., Jones, J. K., Costanza-Robinson, M. S. and Robinson, M. S. (2005) Demystifying disciplinary writing: a case study in the writing of chemistry. Across the Disciplines: interdisciplinary perspectives on language, learning, and academic writing. Retrieved 30 May 2005. http://wac.colostate.edu/atd/lds/stoller.cfm
Swales, J. M. (1990) Genre Analysis: English in academic and research settings. Cambridge: Cambridge University Press.
Swales, J. M. (2004) Research Genres: exploration and applications. Cambridge: Cambridge University Press.
Wardle, E. A. (2004) Can cross-disciplinary links help us teach ‘academic discourse’ in FYC? Across the Disciplines: interdisciplinary perspectives on language, learning, and academic writing, 1. Retrieved 10 January 2004. http://wac.colostate.edu/atd/articles/ wardle2004/
Weigle, S. C. (2002) Assessing Writing. New York: Cambridge University Press.
Whelan, R. J. and Zare, R. N. (2003) Teaching effective communication in a writing-intensive analytical chemistry course. Journal of Chemical Education 80: 904–06.
Beall, H. and Trimbur, J. (2001) A Short Guide to Writing about Chemistry. (Second edition) New York: Longman.
Bressette, A. R. and Breton, G. W. (2001) Using writing to enhance the undergraduate research experience. Journal of Chemical Education 78: 1626–7.
Brown, A. (1995) The effect of rater variables in the development of an occupationspecific language performance test. Language Testing 12: 1–15.
Candlin, C. N. and Candlin, S. (2003) Health care communication: a problematic site for applied linguistics research. In M. McGroarty (ed.) Annual Review of Applied Linguistics 134–54. New York: Cambridge University Press.
Candlin, C. N. and Maley, Y. (1997) Intertextuality and interdiscursivity in the discourse of alternative dispute resolution. In B.-L.
Gunnarson, P. Linell and B. Nordberg (eds) The Construction of Professional Discourse 201–22. London: Longman.
Candlin, C. N. and Sarangi, S. (2004) Making applied linguistics matter. Journal of Applied Linguistics 1(1): 1–8.
Coppola, B. P. and Daniels, D. S. (1996) The role of written and verbal expression in improving communication skills for students in an undergraduate chemistry program. Language and Learning Across the Disciplines 1: 67–86.
Dodd, J. S. (ed.) (1997) The ACS Style Guide. (Second edition) Washington, DC: American Chemical Society.
Douglas, D. (2000) Assessing Language for Specific Purposes. New York: Cambridge University Press.
Douglas, D. (2001) Language for specific purposes assessment criteria: where do they come from? Language Testing 18: 171–85.
Dudley-Evans, T. and St. John, M. J. (1998) Developments in English for Specific Purposes: a multi-disciplinary approach. New York: Cambridge University Press.
Ebel, H. F., Bliefert, C. and Russey, W. E. (2001) The Art of Scientific Writing: from student reports to professional publications in chemistry and related fields. (Second edition) New York: John Wiley.
Ferris, D. R. and Hedgcock, J. S. (2005) Teaching ESL Composition: purpose, process, and practice. (Second edition) Mahwah, NJ: Lawrence Erlbaum.
Flowerdew, J. (1993) Content-based language instruction in a tertiary setting. English for Specific Purposes 12: 121–38.
Flowerdew, J. and Peacock, M. (eds) (2001a) Research Perspectives on English for Academic Purposes. New York: Cambridge University Press.
Flowerdew, J. and Peacock, M. (2001b) The EAP curriculum: issues, methods, and challenges. In J. Flowerdew and M. Peacock (eds) Research Perspectives on English for Academic Purposes 177–94. New York: Cambridge University Press.
Gordon, N. R., Newton, T. A., Rhodes, G., Ricci, J. S., Stebbins, R. G. and Tracy, H. J. (2001) Writing and computing across the USM chemistry curriculum. Journal of Chemical Education 78: 53–5.
Hamp-Lyons, L. (2003) Writing teachers as assessors of writing. In B. Kroll (ed.) Second Language Writing: research insights for the classroom. New York: Cambridge University Press.
Henning, G. and Davidson, F. (1987) Scalar analysis of composition ratings. In K. M. Bailey, T. L. Dale, and R. T. Clifford (eds) Language Testing Research: selected papers from the 1986 colloquium. Monterey, CA: Defense Language Institute.
Hudson, T. (2005) Trends in assessment scales and criterion-referenced language assessment. In M. McGroarty (ed.) Annual Review of Applied Linguistics 205–27. New York: Cambridge University Press.
Hyland, K. (2002) Teaching and Researching Writing. London: Longman.
Hyland, K. (2003) Second Language Writing. New York: Cambridge University Press.
Hyland, K. (2004) Disciplinary Discourses: social interactions in academic writing. Ann Arbor: University of Michigan Press.
Jacoby, S. and McNamara, T. (1999) Locating competence. English for Specific Purposes 18: 213–41.
Johns, A. M. (1997) Text, Role, and Context: developing academic literacies. New York: Cambridge University Press.
Klein, B. and Aller, B. M. (1998) Writing across the curriculum in college chemistry: a practical bibliography. Language and Learning Across the Disciplines 2: 25–35.
Kovac, J. and Sherwood, D. W. (2001) Writing Across the Chemistry Curriculum: an instructor’s guide. New York: Prentice Hall College Division.
Kuldell, N. (2003) Read like a scientist to write like a scientist: using authentic literature in the classroom. Journal of College Science Teaching XXXIII(2): 32–5.
Lumley, T. (1998) Perceptions of language-trained raters and occupational experts in a test of occupational English language proficiency. English for Specific Purposes 17: 347–67.
Lumley, T. (2002) Assessment criteria in a large-scale writing test: what do they really mean to the raters? Language Testing 19: 246–76.
McNamara, T. (1996) Measuring Second Language Performance. New York: Longman.
Nature (2001) Learning to speak and write. Nature 411: 1.
North, B. and Schneider, G. (1998) Scaling descriptors for language proficiency scales. Language Testing 15: 217–63.
Oliver-Hoyo, M. T. (2003) Designing a written assignment to promote the use of critical thinking skills in an introductory chemistry course. Journal of Chemical Education 80: 899–903.
Paulson, D. R. (2001) Writing for chemists: satisfying the CSU upper-division writing requirement. Journal of Chemical Education 78: 1047–9.
Sarangi S. and Candlin, C. N. (2003) Trading between reflexivity and relevance: new challenges for applied linguistics. Applied Linguistics 24: 271–85.
Sarangi, S. and Roberts, C. (eds) (1999) Talk, Work, and Institutional Order: discourse in medical, mediation, and management settings. Berlin: Mouton de Gruyter.
Shibley, I. A., Milakofsky, L. M. and Nicotera, C. L. (2001) Incorporating a substantial writing assignment into organic chemistry: library research, peer review, and assessment. Journal of Chemical Education 78: 50–3.
Smith, S. (2003a) The role of technical expertise in engineering and writing teachers’ evaluations of students’ writing. Written Communication 20: 37–80.
Smith, S. (2003b) What is ‘good’ technical communication? A comparison of the standards of writing and engineering instructors. Technical Communication Quarterly 12: 7–24.
Stoller, F. L., Jones, J. K., Costanza-Robinson, M. S. and Robinson, M. S. (2005) Demystifying disciplinary writing: a case study in the writing of chemistry. Across the Disciplines: interdisciplinary perspectives on language, learning, and academic writing. Retrieved 30 May 2005. http://wac.colostate.edu/atd/lds/stoller.cfm
Swales, J. M. (1990) Genre Analysis: English in academic and research settings. Cambridge: Cambridge University Press.
Swales, J. M. (2004) Research Genres: exploration and applications. Cambridge: Cambridge University Press.
Wardle, E. A. (2004) Can cross-disciplinary links help us teach ‘academic discourse’ in FYC? Across the Disciplines: interdisciplinary perspectives on language, learning, and academic writing, 1. Retrieved 10 January 2004. http://wac.colostate.edu/atd/articles/ wardle2004/
Weigle, S. C. (2002) Assessing Writing. New York: Cambridge University Press.
Whelan, R. J. and Zare, R. N. (2003) Teaching effective communication in a writing-intensive analytical chemistry course. Journal of Chemical Education 80: 904–06.
Published
2007-02-17
Issue
Section
Articles
License
Equinox Publishing Ltd.
How to Cite
Stoller, F. L., Horn, B., Grabe, W., & Robinson, M. S. (2007). Creating and validating assessment instruments for a discipline-specific writing course. Journal of Applied Linguistics and Professional Practice, 2(1), 75-104. https://doi.org/10.1558/japl.v2.i1.75
