A 'functional' projection from visual contrast to mode
Creating building blocks for reading multimodal visual narrative
DOI:
https://doi.org/10.1558/lhs.36177Keywords:
Mode, Multimodal Discourse, Function, SemioticsAbstract
In multimodal visual narratives such as Japanese manga, meaning is created through components of different modes - from verbals to onomatopoeias, emblems, and graphics. With the aim of integrating different modes (from text to image) so as to suggest an integrated solution that collectively defines the composing elements of comics, this paper proposes a Function of Mode, which applies semiotic and mathematical principles to expound on the subconscious process of recognizing visual contrasts that are drawn, printed or digitally displayed into various modes that form the building blocks of multimodal discourse. This function sits at the base of multiliteracy for understanding multimodal visual narrative and enables the reader to distinguish different modes by accentuating relevant semiotic channels of meaningmaking - iconicity for imagistic description, symbolization for linguistic features, entitativity for metaphorizing and visualizing non-graphic images, and diegesis for demarcating which elements are meaningful, ideationally or interpersonally.
References
Atkin, A. (2006/2013). Peirce’s Theory of Signs, The Stanford Encyclopedia of Philosophy (Summer 2013 Edition), Edward N. Zalta (Ed.), URL = <http://plato.stanford.edu/archives/sum2013/entries/peirce-semiotics/>. Retrieved 10-06-2016.
Atkin, A. (2008). Peirce’s final account of signs and the philosophy of language. Transactions of the Charles S. Peirce Society: A Quarterly Journal in American Philosophy, 44 (1): 63–85.
Bateman, J. (2014). Text and Image: A Critical Introduction to the Visual/verbal Divide. Abingdon: Routledge. https://doi.org/10.4324/9781315773971
Bateman, J. A. and Veloso, F. O. (2013). The semiotic resources of comics in movie adaptation: Ang Lee’s Hulk (2003) as a case study. Studies in Comics, 4 (1): 135–157. https://doi.org/10.1386/stic.4.1.135_1
Bordwell, D. and Thompson, K. (2008). Film Art: An Introduction (8th Edition). New York: McGraw-Hill.
Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., and Volpe, G. (2004). Multimodal analysis of expressive gesture in music and dance performances. In Gesture-based Communication in Human-computer Interaction, 20–39. Berlin and Heidelberg: Springer. https://doi.org/10.1007/978-3-540-24598-8_3
Feng, D. and O’Halloran, K. L. (2012). Representing emotive meaning in visual images: A social semiotic approach. Journal of Pragmatics, 44 (14): 2067–2084. https://doi.org/10.1016/j.pragma.2012.10.003
Forceville, C. (2005). Visual representations of the idealized cognitive model of anger in the Asterix album La Zizanie. Journal of Pragmatics, 37 (1): 69–88. https://doi.org/10.1016/j.pragma.2003.10.002
Fuse, H. (2004). Manga wo Kaibousuru [Anatomizing manga]. Tokyo: Chikumashobo.
Halliday, M. A. K. and Hasan, R. (1985/1976). Cohesion in English. London and New York: Longman.
Halliday, M. A. K. (1994). An Introduction to Functional Grammar (2nd edition). London: Edward Arnold.
Halliday, M. A. K. (2002a). The Collected Works of M. A. K. Halliday. Vol. 1. (J. J. Webster, Ed.). London: Continuum.
Halliday, M. A. K. (2002b). The Collected Works of M. A. K. Halliday.Vol. 2. (J. J. Webster, Ed.). London: Continuum.
Halliday, M. A. K. and Webster, J. J. (2014). Text Linguistics: The How and Why of Meaning. Sheffield: Equinox Publishing.
Hatfield, C. (2009). An art of tensions. In J. Heer and K. Worcester (Eds), A Comic Studies Reader. 132–148. Jackson, MS: University Press of Mississippi.
Jacobs, D. (2013). Graphic Encounters: Comics and the Sponsorship of Multimodal Literacy. New York: Bloomsbury.
Jewitt, C. (2009). The Routledge Handbook of Multimodal Analysis. London: Routledge
Kaitani, S. (2005/2010). Liar Game-1. Tokyo: Shueisha.
Kirby, J. T. (1996). Classical Greek Origins of Western Aesthetic Theory. In B. Allert (Ed.) Languages of Visuality, 29–45. Detroit, MI: Wayne State University Press.
Kishimoto, M. (2000/2013). Naruto-1. Tokyo: Shueisha.
Kress, G. and Van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.
Kress, G. and Van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design (2nd Ed.). London: Routledge. https://doi.org/10.4324/9780203619728
Kress, G., and Van Leeuwen, T. (2002). Colour as a semiotic mode: Notes for a grammar of colour. Visual Communication, 1 (3): 343–368. https://doi.org/10.1177/147035720200100306
Martin, J. R. and Rose, D. (2008). Genre Relations: Mapping Culture. London and Oakville, CN.: Equinox.
Matthiessen, C., Teruya, K., and Lam, M. (2010). Key Terms in Systemic Functional Linguistics. London: A&C Black.
McCloud, S. (1994). Understanding Comics: The Invisible Art. New York: HarperPerennial.
Mitchell, W. J. T. (1984). What is an Image?. New Literary History, 15 (3): 503–537. https://doi.org/10.2307/468718
Mitchell, W. J. T. (2009). Beyond comparison. In J. Heer, and K. Worcester (Eds), A Comic Studies Reader, 116–123. Jackson, MI: University Press of Mississippi.
MODE. (2012). Glossary of multimodal terms. URL=<https://multimodalityglossary.wordpress.com/>. Retrieved 25-10-2016.
O’Halloran, K. L. (Ed.). (2004). Multimodal Discourse Analysis: Systemic Functional Perspectives. London and New York: Continuum.
Oda, E. (1997/2013). One Piece-1. Tokyo: Shueisha.
Peirce, C. S. (1932). Collected Papers of Charles Sanders Peirce. Volume II: Elements of Logic. Cambridge: Harvard University Press. https://doi.org/10.2307/2270913
Peirce, C. S. and Houser, N. (1998). The Essential Peirce: Selected Philosophical Writings (Vol. 2). Bloomington, IN: Indiana University Press.
Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X. F., Kirbas, C., … and Ansari, R. (2002). Multimodal human discourse: gesture and speech. ACM Transactions on Computer-Human Interaction (TOCHI), 9 (3): 171–193. https://doi.org/10.1145/568513.568514
Rosen, K. H. (2007). Discrete Mathematics and Its Application (6th Ed, International Edition). New York: McGraw-Hill.
Short, T. L. (2007). Peirce’s Theory of Signs. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511498350
Stewart, J. (2009). Multivariable Calculus (6th Ed., Metric International Version). Thomson Brooks/Cole.
Stöckl, H. (2004). In between modes: Language and image in printed media. In E. Ventola, C. Charles, and M. Kaltenbacher (Eds). Perspectives on Multimodality, 9–30. Amsterdam and Philadelphia, PA: John Benjamins.
Togashi, Y. (1998/2012). Hunter×Hunter-1. Tokyo: Shueisha.
Van Leeuwen, T. (2006). Towards a semiotics of typography. Information Design Journal, 14 (2): 139–155. https://doi.org/10.1075/idj.14.2.06lee
Veloso, F. and Bateman, J. (2013). The multimodal construction of acceptability: Marvel’s Civil War comic books and the PATRIOT Act. Critical Discourse Studies, 10 (4): 427–443. https://doi.org/10.1080/17405904.2013.813776
Yang, X. and Webster, J. (2015). To be continued: Meaning-making in serialized manga as functional-multimodal narrative. Semiotica, 2015 (207): 583–606. https://doi.org/10.1515/sem-2015-0066
Yang, X. and Webster, J. (2017). Meaning-making in multimodal visual narrative: patterns of visual weaving. In J. Webster and X. Peng (Eds). Applying Systemic Functional Linguistics: the State of the Art in China Today. London and New York: Bloomsbury Publishing.