Multimodality, ethnography and education in south america

References

Bezemer, J., & Kress, G. (2016). Multimodality, Learning and Communication: a social semiotic frame. Oxon Routledge.

Jewitt, C., Bezemer, J., & O'Halloran, K. L. (2016). Introducing multimodality. London: Routledge.

Maagerø, E., & Østbye, G. L. (2016). What a Girl! Fighting Gentleness in the Picture Book World: An Analysis of the Norwegian Picture Book What a Girl! by Gro Dahle and Svein Nyhus. Children's Literature in Education, 1-22. doi:10.1007/s10583-016-9276-4

Mangen, A., & Hoel, T. (2017). Samtalebasert lesing med bok eller nettbrett: Gjør mediet en forskjell? Norsk Pedagogisk Tidsskrift, 101(04), 339-351. doi:10.18261/issn.1504-2987-2017-04-06 ER

Nikolajeva, M., & Scott, C. (2006). How Picturebooks Work. New York: Routledge.

Multimodal social semiotic and other semiotic approaches to designed material artefacts – similarities, differences and possibilities

Hans-Christian Jensen, Associate Professor

Department of Design and Communication, University of Southern Denmark
[e-mail]

Toke Riis Ebbesen, Assistant Professor

Department of Design and Communication, University of Southern Denmark
[e-mail]

Since the turn of the millennia the intricate question of the semiotics of designed material artefacts has been dealt with within a number of different academic disciplines like Design Studies (Matozzi, Proni), Material Culture Studies (Dant, Kean, Knappett), studies in marketing and branding (Batey, Danesi, Heilbrunn) and latest but not least also within multimodal social semiotics (Björkvall, Björkvall & Karlsson, Wagner).

Although these different approaches all pursue the same objective, a more specific and precise theoretical understanding of the semiotics of things – the mutual awareness and exchange is remarkedly low. In this paper the above mentioned different approaches will be brought together and mapped in relation to each other. This will be done with special regard to propound a placement and characterization of multimodal social semiotics of artefacts within an overall semiotics of things.

A key finding of the comparison of multimodal social semiotics in relation to the other semiotics treatments of artefactual meaning making is that the former maintain a very open approach to the semiotic resources which could be prompted by things. However other disciplines tend to assert more or less restricted or all-embracing typologies to chart and arrange commonplace semiotic resources usually occurring in symbolic and material interaction with things.

The case will be made that multimodal social semiotics of artefacts could benefit theoretically by including such typologies in analysis without losing the flexibility and sensitiveness towards actual meaning making processes.

References

Batey, M. (2008). Brand Meaning. Abingdon: Routledge.

Björkvall, A. (2009). “Practical function and meaning: A case study of IKEA tables”. In The Routledge Handbook of Multimodal Analysis (2nd ed.), 342–353.
Björkvall, A., & Karlsson, A.-M. (2011). “The materiality of discourses and the semiotics of materials: A social perspective on the meaning potentials of written texts and furniture”. Semiotica, 187, 141-165.
Danesi, M. (2006). Brands. Abingdon: Routledge.
Danesi, M. (2013). “Semiotizing a product into a brand”. Social Semiotics, 23(4), 464–476.
Dant, T. (2008). ”The ’Pragmatics’ of Material Interaction”, Journal of Consumer Culture, 8(1), 11-33.
Heilbrunn, B. (2015). Market Mediations. London: Palgrave Macmillan UK.
Keane, W. (2003). “Semiotics and the social analysis of material things”. Language & Communication, 23(3–4), 409–425.
Knappett, C. (2005). Thinking through material culture: an interdisciplinary perspective. Philadelphia: University of Pennsylvania Press.
Mattozzi, A. (2009). “A model for the Semiotic Analysis of Objects”. S. Vihma & T.-M. Karjalainen (Eds.), Design Semiotics in Use. 41-64. Helsinki: University of Art and Design Press.
Proni, G. (2002). “Outlines for a Semiotic Analysis of Objects”. Versus–Quaderni Di Studi Semiotici, 91(92), 37–59.
Wagner, K. (2015) “Reading packages: social semiotics on the shelf”, Visual Communication, 14(2), 193-220.
Multimodal depth in film: A proposal for a multimodal and sound-oriented approach to intersemiotic analysis

Signe Kjær Jensen, PhD student in Comparative Literature

Linnaeus University Centre for Intermedial and Multimodal Studies, Department of Film and Literature, Linnaeus University, 351 95 Växjö, Sweden
[e-mail]

Film is an inherently multimodal medium. Despite this, film studies and film musicology alike have a tradition of approaching the discussion of formal interaction in film as a question of how well the image track and the music relate to each other, rather than as a question of complex multimodal intersemiosis (for example in epitomic works such as: Carroll (1996); Eisenstein and Leyda (1969); Gorbman (1987)). Even though this reductive theorisation has been questioned and challenged by media and (film) musicology scholars (e.g. Chion, 1994; Cook, 1998; Langkjær, 2008) and from within the field of multimodality (e.g. Bateman and Schmidt (2013); Tseng (2013)), the notion that film = audiovisual = moving image + music, is still largely dominant.

Following Walter Murch and Iben Have’s ideas about an audiovisual dimension (Chion, 1994; Have, 2008), I want to propose that the formal interrelations in film can be analysed according to a multimodal dimension and that the depth of this dimension is decided by the level of dialogue in the multimodal complex. Rather than seeing film as an addition of images and music, I propose that film consists of a number of visual modes (e.g. lighting, viewing perspective, facial gestures) and a number of auditory modes (e.g. dialogue, instrumentation, and musical harmonics), which are orchestrated as different ‘voices’ (drawing on Bakhtin’s ideas of dialogue and polyphony (Bakhtin, 1981)). The intersemiosis is a result of a dialogic interrelationship of all of these voices, or modes. It is my working hypothesis that a high level of dialogue equals a more complex meaning potential and thus a deeper multimodal dimension, whereas a low level of dialogue equals a more straight-forward meaning potential and a flatter multimodal dimension.

Keywords: Multimodality, film analysis, semiotics, intersemiosis

References

Bakhtin, M. M. (1981). Dialogic Imagination : Four Essays. Austin, TX, USA: University of Texas Press.

Bateman, J., & Schmidt, K. H. (2013). Multimodal Film Analysis: How Films Mean: Taylor & Francis.

Carroll, N. E. (1996). Theorizing the moving image: New York: Cambridge Univ. Press, 1996.

Chion, M. (1994). Audio-vision: sound on screen. New York: Columbia University Press.

Cook, N. (1998). Analysing musical multimedia. Oxford: Oxford University Press.

Eisenstein, S., & Leyda, J. (1969). Film Form: Essays in Film Theory: Harcourt, Brace.

Gorbman, C. (1987). Unheard melodies: Narrative film music: Indiana University Press.

Have, I. (2008). Lyt til tv: Underlægningsmusik i danske tv-dokumentarer. Aarhus: Aarhus Universitetsforlag/Aarhus University Press.

Langkjær, B. (2008). Mediernes lyd. En multimodal analysemetode. MedieKultur. Journal of media and communication research, 22(40).

Tseng, C. (2013). Cohesion in Film: Tracking Film Elements: Palgrave Macmillan UK.

A multimodal study of the textbooks for the doctoral course of English for Academic Communication in mainland China

Ting Jiang

School of Foreign Languages and Cultures, Chongqing University, Chongqing, China
[e-mail]

Jianwei Kang

School of Foreign Languages and Cultures, Chongqing University, Chongqing, China
[e-mail]

The study investigates the multimodal resources and the relations between images and texts in the two textbooks for the doctoral course of English for Academic Communication under the framework of multimodal content system and Martinec’s system of logico-semantics by conducting a detailed coding process of content analysis. This study reveals that there is a shortage of multimodal and design features in the two textbooks considering the results of the analysis of visual, text block and layout. Besides, logico-semantics for images and texts are mostly realized by exemplification, but some images and texts form no relation. These findings have important implications for EAP textbook design and EAP teaching in China.

Keywords: Multimodality, logico-semantics, EAP textbooks, China

A discourse-based approach to analyze image-text relations in children’s picture books

Wang Jing

Colleges of Foreign languages and cultures, NanJing University of Aeronautics and Astronautics, China
[e-mail]

In children’s picture books, words and images interact to make meanings in new ways. Based on Martinec＆Salway’s (2005) network of logical semantic relations between texts and images: elaboration, enhancement and extension, this paper adopts a discourse-based approach to analyze image-text relations complementary to existing grammar-based approaches. Intersemiotic consequential and temporal relations are also found in contemporary picture books, which can hopefully shed light on decoding multimodal children’s picture books.

Gesture and the Acquisition of L2 English Tense and Aspect

Qu Congyi Joy

The Chinese University of Hong Kong
[e-mail]

As an important category of nonverbal behaviour, gesture is defined as the spontaneous, synchronized, and meaningful hand and arm movements produced by people when they speak (McNeill, 2005). “Gestures are spatio-visual phenomena influenced by contextual and socio-psychological factors, and also closely tied to sophisticated speaker-internal, linguistic processes” (Gullberg, 2008: 149). Gestures are closely linked to speech and mind, and they can offer valuable insights into the processes of L2 acquisition, like L1 transfer, processing difficulties, and interlanguage.

Conceptualization of time in mind can be expressed by verbal language through multiple means including morphological means (i.e. tense and aspect markings on verbs) (Bardovi-Harlig, 2000). Conceptualization of time in mind can also be expressed by nonverbal gesture. People first conceptualize abstract time as space in mind, and then map the concrete spatial representations onto gestures. These gestures, expressing temporality and occurring simultaneously with oral temporal devices such as tense and aspect marking, are called “Temporal Gestures” (Cooperrider et al, 2014: 1781), the intersection and linkage between gesture and tense-aspect system.

Previous research into L2 tense and aspect acquisition has mainly focused on verbal modality and neglected the nonverbal gesture modality. To study through gesture can add a more vivid and comprehensive dimension of teaching, learning, acquiring, and processing L2 tense and aspect. The study looks into the acquisition of L2 English tense and aspect through the lens of gesture. Combining both qualitative and quantitative, offline and online research methods and techniques from sociolinguistics, psycholinguistics, and language education, it discovers: 1) the roles of pedagogical gesture in teaching and learning L2 English tense and aspect; 2) the online processing of L2 English tense and aspect as reflected in sensitivity to and inhibitory control of temporal gesture-speech incongruencies; 3) the acquisitional trajectory of temporal gesturing as a developing system in its own right.

References

Bardovi-Harlig, K. (2000). Tense and Aspect in Second Language Acquisition: Form, Meaning and Use. Oxford: Blackwell.

Cooperrider, K., Núñez, R., & Sweetser, E. (2014). The conceptualization of time in gesture. In Müller, C., Cienki, A., Fricke, E., Ladewig, S. H., McNeill, D., & Bressem, J. (Eds.), Body-Language-Communication. Volume 1: an International Handbook on Multimodality in Human Interaction. Berlin; Boston: De Gruyter Mouton.

Gullberg, M. (2008). Gestures and some key issues in the study of language development. Gesture, 8 (2), 149–179.

McNeill, D. (2005). Gesture & Thought. Chicago: University of Chicago Press.
Teaching EFL Secondary School Students to Critically Read Literary Classics: a Discourse Analysis of “The Merchant of Venice”

Qu Congyi Joy

The Chinese University of Hong Kong
[e-mail]

This study conducts a discourse analysis of excerpts of “The Merchant of Venice” in secondary school English textbooks used in an EFL context, shedding light on how to teach EFL secondary school students to critically read literary classics. The general theoretical framework consists of genre and register analyses in the light of systemic-functional linguistics. In the process of genre analysis, some specific concepts and tools used include communicative purposes, generic structures, episode analysis, critical literacy, character perspective chart, etc. In the process of register analysis, some specific concepts used include field, tenor and mode. To illustrate some data, examining comments and narration in the excerpts reveals the predominant traditional bias against Shylock, a protagonist of the play, and it leads to the advocacy of critical literacy which takes multiple perspectives into consideration and one of whose tools is character perspective chart. The demonstration of the use of character perspective chart exposes the absence of some critical information such as the historical background of prevalent discrimination against Jews which can to some extent justify Shylock’s behavior, and this brings the traditional bias towards Shylock into question. Moreover, the mode of the texts is written discourse in itself, and since “The Merchant of Venice” is a play, it is of multimodality: the written discourse can be adapted to physical performance and spoken discourse, and different modes can complement different perspectives to facilitate more comprehensive and thorough interpretations. To be concrete, pedagogical implications include: to lead students to watch movie versions, to ask students to change their roles from receptive readers to productive writers by rewriting the story from the perspective of Shylock, and to organize role-play activities such as drama and “readers’ theatre” so that students can become productive performers and put themselves into different characters’ shoes.

Krampus, Hollywood and the semiotics of fear

Martin Kaltenbacher

University of Salzburg; Department of English and American Studies
[e-mail]

Fear is one of the major emotions that humans can feel. In addition to being a psychological state, "fear has a physiology" (Burke 2005): it causes particular muscles in the human body to contract, makes the hair bristle and accelerates the heartbeat. Apart from the physical appearance of fear in a frightened person, humans have always desired to visualize their fear, giving it a concrete shape. This has led to the evolution of many monsters, like ghosts, vampires or Krampus.

Krampus is an ambiguous figure with unclear, probably heathen origins. It has many regionally variant manifestations. One of its most eminent forms is that of a hybrid creature with a furry humanoid body with goat hooves and Capricorn horns. Originally it was the devilish companion to Saint Nicholas and visits families, farms and households between the fourth and the sixth of December to punish lazy, disobedient children and servants. In Austria, Bavaria and Switzerland, hordes of up to several hundred disguised Krampusses parade through towns and villages to celebrate the cultural tradition of the Krampus in professionally organized community events.

In the course of time, Krampus has undergone various semiotic transformations. Most recently, its figure has been transformed by film monsters like orcs or skeleton pirates. As a result, traditional mask carvers have begun to model their wooden Krampus masks on such films. Traditional regional companions to Krampus, e.g. Perchts (two-faced witch-like creatures, often with long, carved beaks) have likewise been replaced by more globally known creatures like vampires and black angels. That this is not such a modern phenomenon as often claimed was shown by Koenig (1983), who photographed wooden Krampus masks called "concentration camp inmate" and "the Hunchback of Notre Dame", dating back to the 1940s.

In December 2015, Universal Studios released a horror movie called "Krampus", which introduced the figure of Krampus to an American audience not familiar with Saint Nicholas' companion. Therefore, the tradition of Krampus had to be re-semioticized to make it easier to understand for an American audience. Consequently, Krampus became "Santa Claus' Shadow", and his mythological companions transformed into more familiar Hollywood monsters like child-devouring horror clowns or monstrous ginger-bread men.

In this paper, I will give an overview of the semiotic rise of Krampus as a mythological horror figure and analyse the recent processes of re-semiotization that this ancient figure has undergone. I will show which mythologically unrelated figures are exploited in the staging of Krampus as a modern horror monster and which other semiotic modes (sound, smell, fireworks) are employed in the mise-en-scène of a 21^st century Krampus parade.

References

Burke, Joanna (2005) Fear. A Cultural Hisrtory. London: Virago.

Koenig, Otto (1983) Klaubauf – Krampus – Nikolaus: Maskenbrauch in Tirol und Salzburg. Wien: Edition Tusch.


All-Encompassing Theory and “MM+EM” Interpreting Teaching

Zhifeng Kang

College of Foreign Languages and Literatures, Fudan University, P.R. China

Dan Jiao
Foreign Languages College, Henan University of Technology, P.R. China

Based upon the All-Encompassing theory and the methods of modern information technologies, this study explores the application modalities of “MM+EM” interpreting teaching. This holistic modality is based on the “1+4” multimodalities (MM) of interpreting teaching with all-encompassing network and the extended modality (EM) constructed, i.e. the application modalities of “MM+EM” interpreting teaching. The application modalities of “MM+EM” are ones with students-centered, teacher-guiding and the use of modern information technologies to create situational training environments. EM is constructed by the application of modern SP and ISM etc. in interpreting teaching after the “1+4” MM as the originality of experiencing worlds. The practices of interpreting teaching in the classrooms and out of them, the tests to the student interpreters and the calculations through SPSS, show that the application modalities of “MM+EM” have achieved good results with its new methods, which go beyond time and space. With the more perfect and popular development of SP and ISM etc., the application modalities of “MM+EM” are hoped to spread to the whole country.

Keywords: The all-encompassing theory, “MM+EM”, interpreting teaching, multimodality, ISM

Types of agency and language learners’ screen-based semiotic work

Janine Knight
Universitat Internacional de Catalunya, Barcelona and Universitat Oberta de Catalunya, Spain
[e-mail]

Whilst many researchers agree that the role of learner agency is central in a learner’s semiotic work (e.g. Bezemer and Kress, 2016) including the work of language learners (Lantolf, 2000; van lier, 2008), much less research has focused on the relationship between screen-based semiotic resources and agency in computer-mediated communication (CMC) language learning scenarios. Given that communication through a screen plays a greater role in modern society than ever before (Liou, 2011), research in this area is important.

Following this, we present the notion of ‘types of agency’ that emerged from case study research on the agency of language learners (Knight, Barberà and Appel, 2017) whilst carrying out spoken interaction tasks using a CMC tool. The types identified highlighted the relationship between what learners could do/chose to do and could not do/chose not to do with the screen-based (semiotic) resources in relation to their motor, sensory (visual) and language systems. This led to types of agency emerging, namely directional agency, representational agency, organisational and strategic agency. Whereas all types involved speech they also involved the use of a screen-based image and/or text, and/or navigational buttons. Directional agency involved the somatic mode of physically clicking; representational agency involved learners’ relationship with the topic and organisational and strategic agency involved the use of spoken language for task management and completion.

Results confirm that learner agency is not monolithic (Mercer, 2012) but rather different types of agency can relate to screen-based resource-use for specific purposes. In addition to the screen being considered ‘a layer’ (Goodwin, 2013) in the action of task completion, results suggest that some screen-based resources can also be orientated to by learners as potential ‘others’ (Raudaskoski, 1999). Specifically with navigational resources, learner-learner CMC can momentarily resemble triadic participation, whereby resources act as social agents (Knight, Barberà, Dooly, forthcoming) in the discourse.

Keywords: Learner agency, screen-based resources, language learning, tasks, computer-mediated communication (CMC)

Dialogic Reading as a Tool for Fostering Language – An Example of Applied Multimodality

Arne Krause, M.A., Postdoctoral Researcher

University of Hamburg
[e-mail]

Picture books are multimodal texts in which the interplay of pictures and written language is of crucial importance (i.e. Moya Guijarro 2014; Painter, Martin & Unsworth 2014). The written language is often read aloud by adults to children, making joint picture book reading a complex multimodal interaction with several modes to be considered. Dialogic Reading, a well-known tool for fostering language development, is a form of joint picture book reading that aims at stimulating oral language production by children through various evocative techniques (cf. Whitehurst et al. 1994; Mol et al. 2009). Recently, Dialogic Reading has been reevaluated in a series of research projects by developmental psychologists (i.e. Ennemoser, Kuhl & Pepouna 2013) in which it could be shown that Dialogic Reading improves language and literacy development very effectively when implemented systematically. To make the experimental results replicable in kindergarten routine, a manual is in development in which the developmental psychologist's findings are paired with linguistic knowledge on language and literacy acquisition as well as on multimodality and multilingualism.

In this talk, the question will be approached of how the ineluctable multimodality of Dialogic Reading has to be considered for the development of said manual. This provides the impulse for the corollary question whether a manual like this is an example of Applied Multimodality – and what can be understood as Applied Multimodality in the first place.

References

Ennemoser, M., Kuhl, J. & Pepouna, S. (2013). Evaluation des Dialogischen Lesens zur Sprachförderung bei Kindern mit Migrationshintergrund. Zeitschrift für Pädagogische Psychologie 27 (4), 229-239.

Mol, S. E., Bus, A. G., de Jong, M.T. & Smeets, D. J. H. (2008). Added Value of Dialogic Parent-Child Book Readings: A Meta-Analysis, Early Education & Development 19, 7-26.

Moya Guijarro, A. J. (2014). A Multimodal Analysis of Picture Books for Children. A Systemic Functional Approach. Sheffield/Bristol: equinox.

Painter, C., Martin, J. R. & Unsworth, L. (2014). Reading Visual Narratives. Image Analysis of Children's Picture Books. Sheffield/Bristol: equinox.

Whitehurst, G. J., Arnold, D. S., Epstein, J. N., Angell, A. L., Smith, M. & Fischel, J. (1994). A picture book reading intervention in day care and home for children from low-income families. Developmental Psychology 30, 679-689.