Aspects of rating communicative interaction
Effects on reliability and agreement
DOI:
https://doi.org/10.1558/jircd.v5i2.245Keywords:
Communication assessment, conversational interaction, reliability and agreement, communication partner training, aphasiaAbstract
Communication partner training can be effective in improving communication in aphasia. However, further research is needed on how to measure the outcome of such interventions. In this paper we discuss the phenomenon of reliability in assessments in relation to the results of analyses using a rating scale designed to measure the ability to support a person with aphasia in natural conversational interaction. The scale was used by four assessors to rate 45 video recordings. Calculations of reliability and agreement produced varying results but were mostly satisfactory. However, the results highlight how interaction between factors such as complexity of assessments; design of the rating scale; factors inherent in the individual assessor; and the statistical measures used to analyse the outcome may result in a conflict between aspects of validity and reliability. Interpretations of outcome obtained with rating scales thus need to be based on knowledge about factors influencing the results.
References
Allwood, J. (2000). An activity based approach to pragmatics. In H. Bunt and W. J. Black (eds) Abduction, Belief and Context in Dialogue: Studies in Computational Pragmatics, 47–80. Amsterdam: John Benjamins. http://dx.doi.org/10.1075/nlp.1.02all
Behn, N., Togher, L., Power, E. and Heard, R. (2012). Evaluating communication training for paid carers of people with traumatic brain injury. Brain Injury 26 (13–14): 702–1715. http://dx.doi.org/10.3109/02699052.2012.722258
Bland, J. M. and Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 327 (8476): 307–310. http://dx.doi.org/10.1016/S0140-6736(86)90837-8
Bland, J. M. and Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research 8: 135–160.
Blom Johansson, M., Carlsson, M., Östberg, P. and Sonnander, K. (2013). A multiple-case study of a family-oriented intervention practice in the early rehabilitation phase of persons with aphasia. Aphasiology 27 (2): 201–226. http://dx.doi.org/10.1080/02687038.2012.744808
Bunton, K., Kent, R. D., Duffy, J. R., Rosenbek, J. C. and Kent, J. F. (2007). Listener agreement for auditory-perceptual ratings of dysarthria. Journal of Speech and Hearing Research 50 (6): 1481–1495. http://dx.doi.org/10.1044/1092-4388(2007/102)
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment 6 (4): 284–290. http://dx.doi.org/10.1037/1040-3590.6.4.284
Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G. and Pankratz, V. S. (2009). Effects of rater training on reliability and accuracy of Mini-CEX Scores: A randomized, controlled trial. Journal of General Internal Medicine 24 (1): 74–79. http://dx.doi.org/10.1007/s11606-008-0842-3
Cordes, A. K. (1994). The reliability of observational data: I. Theories and methods for speech-language pathology. Journal of Speech, Language and Hearing Research 37 (2): 264–278.
Enderby, P., John, A. and Petheram, B. (2006). Therapy Outcome Measures for Rehabilitation Professionals. (2nd edn). Chichester: Wiley.
Feldman, M., Lazzara, E. H., Vanderbilt, A. A. and DiazGranados, D. (2012). Rater training to support high-stakes simulation-based assessments. Journal of Continuing Education in the Health Professions 32 (4): 279–286.
Fox, S., Armstrong, E. and Boles, L. (2009). Conversational treatment in mild aphasia: A case study. Aphasiology 23 (7–8): 951–964. http://dx.doi.org/10.1080/02687030802669526
Grice, H.P. (1975). Logic and conversation. In P. Cole and J. Morgan (eds) Syntax and Semantics 3: Speech Acts 41–58. New York, NY: Academic Press.
Jakobsson, R. (2010). Bedömning av konversationsförmågan vid afasi (Assessment of conversation ability in aphasia). (Unpublished master’s thesis). Uppsala University, Uppsala.
Kagan, A. (1999). Supported conversation for adults with aphasia™: Methods and evaluation. (Unpublished doctoral dissertation). Institute of Medical Science, University of Toronto, Toronto.
Kagan, A., Black, S. E., Duchan, J. F., Simmons-Mackie N. and Square, P. (2001). Training volunteers as conversation partners using ‘Supported Conversation for Adults with Aphasia’ (SCA): A controlled trial. Journal of Speech, Language and Hearing Research 44 (3): 624–638. http://dx.doi.org/10.1044/1092-4388(2001/051)
Kagan, A., Winckel, J., Black, S., Duchan, J. F., Simmons-Mackie, N. and Square, P. (2004). A set of observational measures for rating support and participation in conversation between adults with aphasia and their conversation partners. Topics in Stroke Rehabilitation 11 (1): 67–83. http://dx.doi.org/10.1310/CL3V-A94A-DE5C-CVBE
Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajewski, B. J., Hróbjartsson, A. and Roberts, C. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology 64 (1): 96–106. http://dx.doi.org/10.1016/j.jclinepi.2010.03.002
Kreiman, J. and Gerratt, B. R. (1998). Validity of rating scale measures of voice quality. The Journal of the Acoustical Society of America 104 (3): 1598–1608. http://dx.doi.org/10.1121/1.424372
Kreiman, J., Gerratt, B. R., Kempster, G. B., Erman, A. and Berke, G. S. (1993). Perceptual evaluation of voice quality: Review, tutorial, and a framework for future research. Journal of Speech, Language and Hearing Research 36 (1): 21–40.
Lawlis, G. F. and Elba, L. (1972). Judgment of counseling process: reliability, agreement, and error. Psychological Bulletin 78 (1): 17–20. http://dx.doi.org/10.1037/h0032935
Off, C., Rogers, M. and Alarcon, N. (2006). Three methods of quantifying the quality of communication in aphasia (conference paper). Retrieved on 10 September 2013 from http://aphasiology.pitt.edu/archive/00001747/
Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., van den Elsen, L., Grol, R. P. T. M. and van der Vleuten, C. P. M. (2011). In-training assessment using direct observation of single-patient encounters: A literature review. Advances in Health Sciences Education 16 (1): 131–142. http://dx.doi.org/10.1007/s10459-010-9235-6
Perkins, L., Whitworth, A. and Lesser, A. (1997). Conversation Analysis Profile for People with Cognitive Impairment (CAPPCI). London: Whurr Publishers.
Purdy, M. and Hindenlang, J. (2005). Educating and training caregivers of persons with aphasia. Aphasiology 19 (3–5): 377–388. http://dx.doi.org/10.1080/02687030444000822
Rayner, H. and Marshall, J. (2003). Training volunteers as conversation partners for people with aphasia. International Journal of Language and Communication Disorders 38 (2): 149–164. http://dx.doi.org/10.1080/1368282021000060308
Roch, S. G., Woehr, D. J., Mishra, V. and Kieszczynska, U. (2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology 85 (2): 370–395. http://dx.doi.org/10.1111/j.2044-8325.2011.02045.x
Saldert, C., Backman, E. and Hartelius, L. (2013). Conversation partner training with spouses of persons with aphasia: A pilot study using a protocol to trace relevant characteristic. Aphasiology 27 (3): 271–292. http://dx.doi.org/10.1080/02687038.2012.710317
Saldert, C., Bergman, M., Holstensson, J., Jönsson, S., Nygren, K., Vennman, F. and Ferm, U. (2012). Combining methods in the assessment and analysis of communication in aphasia: Benefits and shortcomings of different approaches. Journal of Interactional Research in Communication Disorders 3 (2): 141–169. http://dx.doi.org/10.1558/jircd.v3i2.141
Schegloff, E. A. (1982). Discourse as an interactional achievement: Some uses of ‘uh huh’ and other things that come between sentences. In D Tannen (ed.) Analyzing Discourse: Text and Talk, 71–73). Washington, DC: Georgetown University Press.
Simmons-Mackie, N., Raymer, A., Armstrong, E., Holland, A. and Cherney, L. R. (2010). Communication partner training in aphasia: A systematic review. Archives of Physical Medicine and Rehabilitation 91 (12): 1814–1837.
Sorin-Peters, R. (2004). The evaluation of a learner-centred training programme for spouses of adults with chronic aphasia using qualitative case study methodology. Aphasiology 18 (10): 951–975. http://dx.doi.org/10.1080/02687030444000453
Streiner, D. L. and Norman, G. R. (2008). Health Measurement Scales: A Practical Guide to their Development and Use. Oxford: Oxford University Press. http://dx.doi.org/10.1093/acprof:oso/9780199231881.001.0001
Tinsley, H. E. A. and Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgements. Journal of Counseling Psychology 22 (4): 358–376. http://dx.doi.org/10.1037/h0076640
Togher, L., Power, E., Tate, R., McDonald, S. and Rietdijk, R. (2010). Measuring the social interactions of people with traumatic brain injury and their communication partners: The adapted Kagan scales. Aphasiology 24 (6): 914–927. http://dx.doi.org/10.1080/02687030903422478
Turner, S. and Whitworth, A. (2006). Conversational partner training program in aphasia: A review of key themes and participants’ roles. Aphasiology 20 (6): 483–510. http://dx.doi.org/10.1080/02687030600589991
Yeates, P., O’Neill, P., Mann, K. and Eva, K. W. (2013a). ‘You’re certainly relatively competent’: Assessor bias due to recent experiences. Medical Education 47 (9): 910–922. http://dx.doi.org/10.1111/medu.12254
Yeates, P., O’Neill, P., Mann, K. and Eva, K. W. (2013b). Seeing the same thing differently. Mechanisms that contribute to assessor differences in directly-observed performance assessments. Advances in Health Science Education 18 (3): 325–341. http://dx.doi.org/10.1007/s10459-012-9372-1
Zaki, R., Bulgiba, A., Ismail, R. and Ismail, N. A. (2012). Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: A systematic review. PLoS ONE 7 (5): e37908. http://dx.doi.org/10.1371/journal.pone.0037908
Behn, N., Togher, L., Power, E. and Heard, R. (2012). Evaluating communication training for paid carers of people with traumatic brain injury. Brain Injury 26 (13–14): 702–1715. http://dx.doi.org/10.3109/02699052.2012.722258
Bland, J. M. and Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 327 (8476): 307–310. http://dx.doi.org/10.1016/S0140-6736(86)90837-8
Bland, J. M. and Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research 8: 135–160.
Blom Johansson, M., Carlsson, M., Östberg, P. and Sonnander, K. (2013). A multiple-case study of a family-oriented intervention practice in the early rehabilitation phase of persons with aphasia. Aphasiology 27 (2): 201–226. http://dx.doi.org/10.1080/02687038.2012.744808
Bunton, K., Kent, R. D., Duffy, J. R., Rosenbek, J. C. and Kent, J. F. (2007). Listener agreement for auditory-perceptual ratings of dysarthria. Journal of Speech and Hearing Research 50 (6): 1481–1495. http://dx.doi.org/10.1044/1092-4388(2007/102)
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment 6 (4): 284–290. http://dx.doi.org/10.1037/1040-3590.6.4.284
Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G. and Pankratz, V. S. (2009). Effects of rater training on reliability and accuracy of Mini-CEX Scores: A randomized, controlled trial. Journal of General Internal Medicine 24 (1): 74–79. http://dx.doi.org/10.1007/s11606-008-0842-3
Cordes, A. K. (1994). The reliability of observational data: I. Theories and methods for speech-language pathology. Journal of Speech, Language and Hearing Research 37 (2): 264–278.
Enderby, P., John, A. and Petheram, B. (2006). Therapy Outcome Measures for Rehabilitation Professionals. (2nd edn). Chichester: Wiley.
Feldman, M., Lazzara, E. H., Vanderbilt, A. A. and DiazGranados, D. (2012). Rater training to support high-stakes simulation-based assessments. Journal of Continuing Education in the Health Professions 32 (4): 279–286.
Fox, S., Armstrong, E. and Boles, L. (2009). Conversational treatment in mild aphasia: A case study. Aphasiology 23 (7–8): 951–964. http://dx.doi.org/10.1080/02687030802669526
Grice, H.P. (1975). Logic and conversation. In P. Cole and J. Morgan (eds) Syntax and Semantics 3: Speech Acts 41–58. New York, NY: Academic Press.
Jakobsson, R. (2010). Bedömning av konversationsförmågan vid afasi (Assessment of conversation ability in aphasia). (Unpublished master’s thesis). Uppsala University, Uppsala.
Kagan, A. (1999). Supported conversation for adults with aphasia™: Methods and evaluation. (Unpublished doctoral dissertation). Institute of Medical Science, University of Toronto, Toronto.
Kagan, A., Black, S. E., Duchan, J. F., Simmons-Mackie N. and Square, P. (2001). Training volunteers as conversation partners using ‘Supported Conversation for Adults with Aphasia’ (SCA): A controlled trial. Journal of Speech, Language and Hearing Research 44 (3): 624–638. http://dx.doi.org/10.1044/1092-4388(2001/051)
Kagan, A., Winckel, J., Black, S., Duchan, J. F., Simmons-Mackie, N. and Square, P. (2004). A set of observational measures for rating support and participation in conversation between adults with aphasia and their conversation partners. Topics in Stroke Rehabilitation 11 (1): 67–83. http://dx.doi.org/10.1310/CL3V-A94A-DE5C-CVBE
Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajewski, B. J., Hróbjartsson, A. and Roberts, C. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology 64 (1): 96–106. http://dx.doi.org/10.1016/j.jclinepi.2010.03.002
Kreiman, J. and Gerratt, B. R. (1998). Validity of rating scale measures of voice quality. The Journal of the Acoustical Society of America 104 (3): 1598–1608. http://dx.doi.org/10.1121/1.424372
Kreiman, J., Gerratt, B. R., Kempster, G. B., Erman, A. and Berke, G. S. (1993). Perceptual evaluation of voice quality: Review, tutorial, and a framework for future research. Journal of Speech, Language and Hearing Research 36 (1): 21–40.
Lawlis, G. F. and Elba, L. (1972). Judgment of counseling process: reliability, agreement, and error. Psychological Bulletin 78 (1): 17–20. http://dx.doi.org/10.1037/h0032935
Off, C., Rogers, M. and Alarcon, N. (2006). Three methods of quantifying the quality of communication in aphasia (conference paper). Retrieved on 10 September 2013 from http://aphasiology.pitt.edu/archive/00001747/
Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., van den Elsen, L., Grol, R. P. T. M. and van der Vleuten, C. P. M. (2011). In-training assessment using direct observation of single-patient encounters: A literature review. Advances in Health Sciences Education 16 (1): 131–142. http://dx.doi.org/10.1007/s10459-010-9235-6
Perkins, L., Whitworth, A. and Lesser, A. (1997). Conversation Analysis Profile for People with Cognitive Impairment (CAPPCI). London: Whurr Publishers.
Purdy, M. and Hindenlang, J. (2005). Educating and training caregivers of persons with aphasia. Aphasiology 19 (3–5): 377–388. http://dx.doi.org/10.1080/02687030444000822
Rayner, H. and Marshall, J. (2003). Training volunteers as conversation partners for people with aphasia. International Journal of Language and Communication Disorders 38 (2): 149–164. http://dx.doi.org/10.1080/1368282021000060308
Roch, S. G., Woehr, D. J., Mishra, V. and Kieszczynska, U. (2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology 85 (2): 370–395. http://dx.doi.org/10.1111/j.2044-8325.2011.02045.x
Saldert, C., Backman, E. and Hartelius, L. (2013). Conversation partner training with spouses of persons with aphasia: A pilot study using a protocol to trace relevant characteristic. Aphasiology 27 (3): 271–292. http://dx.doi.org/10.1080/02687038.2012.710317
Saldert, C., Bergman, M., Holstensson, J., Jönsson, S., Nygren, K., Vennman, F. and Ferm, U. (2012). Combining methods in the assessment and analysis of communication in aphasia: Benefits and shortcomings of different approaches. Journal of Interactional Research in Communication Disorders 3 (2): 141–169. http://dx.doi.org/10.1558/jircd.v3i2.141
Schegloff, E. A. (1982). Discourse as an interactional achievement: Some uses of ‘uh huh’ and other things that come between sentences. In D Tannen (ed.) Analyzing Discourse: Text and Talk, 71–73). Washington, DC: Georgetown University Press.
Simmons-Mackie, N., Raymer, A., Armstrong, E., Holland, A. and Cherney, L. R. (2010). Communication partner training in aphasia: A systematic review. Archives of Physical Medicine and Rehabilitation 91 (12): 1814–1837.
Sorin-Peters, R. (2004). The evaluation of a learner-centred training programme for spouses of adults with chronic aphasia using qualitative case study methodology. Aphasiology 18 (10): 951–975. http://dx.doi.org/10.1080/02687030444000453
Streiner, D. L. and Norman, G. R. (2008). Health Measurement Scales: A Practical Guide to their Development and Use. Oxford: Oxford University Press. http://dx.doi.org/10.1093/acprof:oso/9780199231881.001.0001
Tinsley, H. E. A. and Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgements. Journal of Counseling Psychology 22 (4): 358–376. http://dx.doi.org/10.1037/h0076640
Togher, L., Power, E., Tate, R., McDonald, S. and Rietdijk, R. (2010). Measuring the social interactions of people with traumatic brain injury and their communication partners: The adapted Kagan scales. Aphasiology 24 (6): 914–927. http://dx.doi.org/10.1080/02687030903422478
Turner, S. and Whitworth, A. (2006). Conversational partner training program in aphasia: A review of key themes and participants’ roles. Aphasiology 20 (6): 483–510. http://dx.doi.org/10.1080/02687030600589991
Yeates, P., O’Neill, P., Mann, K. and Eva, K. W. (2013a). ‘You’re certainly relatively competent’: Assessor bias due to recent experiences. Medical Education 47 (9): 910–922. http://dx.doi.org/10.1111/medu.12254
Yeates, P., O’Neill, P., Mann, K. and Eva, K. W. (2013b). Seeing the same thing differently. Mechanisms that contribute to assessor differences in directly-observed performance assessments. Advances in Health Science Education 18 (3): 325–341. http://dx.doi.org/10.1007/s10459-012-9372-1
Zaki, R., Bulgiba, A., Ismail, R. and Ismail, N. A. (2012). Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: A systematic review. PLoS ONE 7 (5): e37908. http://dx.doi.org/10.1371/journal.pone.0037908
Published
2014-07-31
Issue
Section
Articles
How to Cite
Eriksson, K., Bergström, S., Carlsson, E., Hartelius, L., Johansson, C., Schwarz, A., & Saldert, C. (2014). Aspects of rating communicative interaction: Effects on reliability and agreement. Journal of Interactional Research in Communication Disorders, 5(2), 245-267. https://doi.org/10.1558/jircd.v5i2.245
