Multimodality, ethnography and education in south america

References

Maton, K. (2014) Knowledge and Knowers: Towards a realist sociology of education.London: Routledge.

Maton, K. and Doran, Y. J. (2017a) ‘Semantic density: A translation device for revealing complexity of knowledge practices in discourse, part 1 – wording. Onomázein, Special Issue on Systemic Functional Linguistics and Legitimation Code Theory. 46-76.

Maton, K. and Doran, Y. J. (2017b) ‘Condensation: A translation device for revealing complexity of knowledge practices in discourse, part 2 – clausing and sequencing. Onomázein, Special Issue on Systemic Functional Linguistics and Legitimation Code Theory. 77-110.

MULTIMODALITY IN THE BRAZILIAN CONTEXT: THE STATE OF THE ART

Záira Bomfante dos Santos

Federal University of Espírito Santo (UFES)
[e-mail]

Clarice Lage Gualberto

Federal University of Minas Gerais (UFMG)
[e-mail]

In this paper, we present a panoramic view of contributions to and the unfolding of the Multimodality Approach in the Brazilian context. Since our study is under a Social Semiotic perspective, we begin with the postulations of Kress (1997; 2010); Kress & van Leeuwen (2001); and Bezemer & Kress (2008; 2016), seeking to interact with the sociosemiotic perspectives of the communications in order to understand the semiotic landscape, the possibilities and the availability of the semiotic modes. In the second moment, we describe how this perspective has been placed in the Brazilian scenario, answering the following questions: when were the first studies on Multimodality published in Brazil? What kind of publications can we find (papers, thesis, etc.)? What are the most common themes found in such studies? Where in Brazil are the researchers located? What kind of material has been analyzed (video, print images, websites…)? What kind of methodology has been used? The quantitative part of this paper used two main search engines: Google Scholar and the online database of CAPES – The Coordination for the Improvement of Higher Education Personnel, an institution funded by the Brazilian Government, responsible for both evaluating and regulating universities and faculties. After organizing and categorizing the results, we noticed a growing number of publications as well as an engagement in understanding the complexity of communication in the use of the various semiotic modes in the production and orchestration of meanings.

Keywords: Multimodal approach, social semiotics, Brazilian context

References

Bezemer, J., & Kress, G. (2008) Writing multimodal texts: A social Semiotic Account of Designs for learning. Written Communication, 25(2), pp.166-195. doi: 10.1177/0741088307313177

Bezemer, J., & Kress, G. (2016). Multimodality, Learning and Communication: A Social Semiotic Frame. London: Routledge.

Kress, G. (1997). Before Writing: Rethinking the Paths to Literacy. London: Routledge.

Kress, G. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge

Kress, G., & van Leeuwen, T. (2001). Multimodal Discourse: The modes and media of contemporary communication. London, United Kingdom: Oxford University Press.

Multimodality in L1 steering documents and classroom practice: Nordic status and research implications

Nikolaj Elf, University of Southern Denmark

[e-mail]

Øystein Gilje, University of Oslo

[e-mail]

Christina Olin-Scheller, University of Karlstad

[e-mail]

Anna Slotte, University of Helsinki

[e-mail]

This paper presents findings from project Multimodal literacy practices in L1 (abbr. MultiL1), which aims at establishing a coherent research agenda for exploring literacy practices in L1/Language arts in Nordic countries. MultiL1 claims that the understanding of language and literacy has been extended due to the development of communication technologies and that this holds important implications for teaching and learning literacy within school and in L1 in particular (Green, 2018; Kress, 2004; Ongstad et al., 2007).Theoretically, MultiL1 draws on a social semiotic understanding of literacy and learning (Bezemer & Kress, 2016) as well as didactic/curriculum theory conceptualizing subjects on programmatic and enacted levels (Deng & Luke, 2008).The project asks the following research questions: What characterizes multimodal literacy practices in Nordic L1 subjects grade 7-9 (age group around 12-16)? Methodologically, MultiL1 conducted two qualitative studies: a comparative analysis of current national steering documents in Denmark, Norway, Sweden and Finland, and a conceptual review of empirical classroom research in the four countries comprising 46 included studies.

Analyses suggest that there is a discrepancy between the programmatic and realized curriculum (Elf, Gilje, Olin-Scheller, & Slotte, in press). The curriculum analysis finds that multimodal composing now ‘counts as knowledge’ in all Nordic L1 subjects on secondary level. Moreover, students’ development of multimodal literacy practices in L1 is part and parcel of literacy reforms in the four Nordic countries. However, the review study reveals that multimodal teaching represents 1) a major cultural change for L1 teachers; 2) is dominated by receptive, not productive practices; and 3) that the evaluation of multimodal production is experienced as particularly challenging for L1 teachers. Considering implications, teachers call for the development of common criteria for multimodal evaluation. We argue that research could focus more on the intrinsic link between multimodal production, microdiscursive evaluation practices and knowledge production.

References

Bezemer, J., & Kress, G. (2016). Multimodality, learning and communication: A social semiotic frame. London: Routledge.

Deng, Z., & Luke, A. (2008). Subject Matter: Defining and Theorizing School Subjects. In M. F. Connelly, M. Fang, & J. Phillion (Eds.), The SAGE Handbook of Curriculum and Instruction (pp. 66-88). Thousand Oaks: SAGE Publications.
Elf, N., Gilje, Ø., Olin-Scheller, C., & Slotte, A. (in press). Nordisk status og forskningsperspektiver: Multimodalitet i styredokumenter og klasserumsrumspraksis. [Nordic status and research implications: Multimodality in steering documents and classroom practice]. In M. Rogne & L. R. Waage (Eds.), Multimodalitet i skole- og fritidstekster. [Multimodality in school and leisure time texts]. Fakbokforlaget.
Green, B. (2018). Engaging Curriculum: Bridging the Curriculum Theory and English Education Divide. New York: Routledge.
Kress, G. (2004). Gains and losses: New forms of texts, knowledge, and learning. Computers and Composition, 22(Elsevier Inc.), 5-22.
Ongstad, S., van de Ven, P.-H., & Herrlitz, W. (Eds.). (2007). Research on Mother Tongue Education in a Comparative International Perspective: Theoretical and Methodological Issues. Utrecht: Rodopi.

Interacting with data visualizations in the news media

Martin Engebretsen

University of Agder

Data visualizations (DVs), in the forms of charts, diagrams, maps etc., represent a visual form growing in prevalence and impact in many contemporary genres (Kennedy et al. 2016). Particularly interesting is the development of (often innovative forms of) DV in digital, public media – like online news sites (Cairo 2017). A news story on an epidemic outbreak will today be expected to be accompanied by a diagrammatic and cartographic representation of statistical, historical and geographical information. Sometimes these visual representations work in a stand-alone format, connected to news stories only by links. Other times, they are integrated in news packages together with verbal and photographic elements. But they always involve specific orchestrations of visual, verbal and numeric resources, interplaying in complex semiotic structures (Engebretsen & Weber 2017). This paper asks: How can data visualizations be described as multimodal text forms, and what are their effects on the readers of online newspapers?

Theoretically, the study leans on investigations from several corners of the field in question. Starting from a basic description of DV as a semiotic and discursive resource (Bertin 2011, Kress and van Leeuwen 2006) it will discuss the interplay between meanings and feelings in the context of DV interaction (Kennedy et al. 2016; Lemke 2015). However, the main focus is on the reception of DV in the context of news media. A user study with mixed data from 10 respondents informs an investigation of skills, habits and attitudes regarding young people’s interactions with data visualizations.

References

Bertin, Jacques (2011 [1967]) Semiology of Graphics: Diagrams, Networks, Maps. ESRI Press.

Cairo, A. (2016) The truthful art. Data, charts, and maps for communication. CA: New Riders.

Engebretsen, M. & Weber, W. (2017). “Graphic Modes”. In C. Cotter & D. Perrin (eds.) Handbook of language and media. London: Routledge.

Kennedy H., Hill, R.L., Aiello, G., & Allen, W. (2016). “The work that visualisation conventions do”. In Information, Communication and Society 19(6): 715-735.

Kress, Gunther and Theo van Leeuwen (2006). Reading Images. The Grammar of Visual Design London: Routledge.

Lemke, Jay (2015) “Feeling and Meaning: A Unitary Bio-Semiotic Account”, in Trifonas, P. (ed.) International Handbook of Semiotics, Springer. DOI 10.1007/978-94-017-9404-6_27


Multimodal situated configurations in a physics interactive
learning environment dealing with circular motion

Moa Eriksson

Department of Physics and Astronomy, Uppsala University, Sweden
Department of Physics, Lund University, Sweden
[e-mail]

Urban Eriksson

Department of Physics, Lund University, Sweden
[e-mail],

Cedric Linder

Department of Physics and Astronomy, Uppsala University, Sweden
[e-mail]

The aim of this presentation is to contribute to the theorizing of disciplinary learning from a social semiotic perspective. The particular exploratory focus being reported on being the physics of circular motion in an introductory, university level interactive classroom. Our starting point for this work is that all disciplinary learning has critical features that need to be discerned in a meaningful, reflective way (Fredlund et al. 2015a; Eriksson 2014). A circular-motion learning situation is used to explore how such reflective discernment (Eriksson et al. 2014) is brought about in response to the semiotic landscape (Jewitt 2008) of the learning experience as a function of both experienced variation (Marton & Booth, 1997; Marton, 2015) and constituted translation (Bezemer & Kress 2008; Kress 2010). Against this backdrop, analysis of preliminary data that consists of audio and video recordings of students engaging with the object of learning in a classroom interactive environment vis-à-vis the forms of representation that make up the teaching and learning environment will be presented. This data analysis characterizes the arising multimodal situated configurations (Jewitt 2008), which will be discussed in terms of the theorizing presented by Fredlund et al. (2015b) for enhancing the possibilities for learning physics.

References

Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: a social semiotic account of designs for learning. Written Communication, 25(2), 166-195.

Eriksson, U., Linder, C., Airey, J. & Redfors, A. (2014). Introducing the anatomy of disciplinary discernment: An example from astronomy. European Journal of Science and Mathermatics Education, 2(3), 167-182.

Eriksson, U., Linder, C., Airey, J., & Redfors, A. (2014). Who needs 3D when the Universe is flat? Science Education, 98(3), 412-442.

Fredlund, T., Linder, C. & Airey, J. (2015a). A social semiotic approach to identifying critical aspects. International Journal for Lesson & Learning Studies, 4(3), 302-316.

Fredlund, T., Airey, J., & Linder, C. (2015b). Enhancing the possibilities for learning: Variation of disciplinary-relevant aspects in physics representations. European Journal of Physics. 36 (5), 1-11.

Jewitt, C. (2008). Multimodality and Literacy in School Classrooms. Review of Research in Education 32; 241, DOI: 10.3102/0091732X07310586

Kress, G. (2010). Multimodality. A Social Semiotic Approach to Contemporary Communication. London: Routledge.

Marton, F. (2015), Necessary Conditions of Learning, Routledge, New York, NY.

Marton, F., & Booth, S. (1997). Learning and Awareness. Mahwah, New Jersey: Lawrence Erlbaum Associates.

Spontaneous use of dance in an astronomy activity

Elias Euler, Elmer Rådahl, Bor Gregorcic Elias Euler

Physics Education Research, Department of Physics and Astronomy, Uppsala University

In science education, there has been recent interest in the ways that the body plays a part in the process of thinking and, concurrently, how learning activities can be designed to explicitly incorporate more embodied engagement. In this paper, we present a case study of an instance where two students spontaneously recruited their bodies as they came to understand the celestial mechanics in an astronomy activity. Working on an open-ended task in a technologically-rich environment, the students instinctively held hands and leaned outward to metaphorically account for the centrality and reciprocity of gravitational forces. Their embodied engagement included a multimodally-rich coordination of talk, gaze, posture, body placement, and haptically-responsive touch in a manner similar to a partnered dance. With an interpretive lens which combines social semiotics, embodied cognition, and conceptual metaphor, we examine the ways in which dance functioned as part of the students’ reasoning about astronomy, as well as reflect on the how the environment afforded the opportunity for such a metaphorical, embodied representation to be recruited in the process of learning.

Unconscious Representations in Students’ Drawings: How Prevailing Discourses Go Unchallenged

Margarita Felipe Fajardo, EdD

Ateneo de Naga University, Naga City, Camarines Sur, Philippines
[e-mail]

Multimodal representations of meaning are beginning to be the norm in today’s classrooms. The Internet era, with its use of varied modes to communicate information (visual, linguistic, audio, gestural, spatial or the interplay between or among these modes), has paved the way for educational institutions to consider students’ multimodal designs as evidence of learning. In tertiary classes, for example, multimodal compositions such as PowerPoint presentations, digital stories, short videos and web designs are increasingly becoming common end-of-term projects, instead of only term papers or essays, which mainly use words to convey meaning. On the other hand, drawing pictures by hand inside the classroom as a form of composition is usually thought of as belonging in the realm of children’s literacy practices. This study argues that illustration can be a legitimate form of multimodal composition in the tertiary classroom, especially where access to technology may be limited and when students’ ideological positions on issues are to be explored. Using critical multimodal discourse analysis, this study explored how four groups of tertiary English language learners in one class in the Philippines interpreted Edwin Arlington Robinson’s poem, Richard Cory, through hand-drawn illustrations. Critical analysis of the illustrations using Kress and van Leeuwen’s (2006) theory of author representation and positioning in images revealed that students unquestioningly reproduced the same ideological stance as the author. The findings of the study suggest the importance of introducing students to theories of how ideological positions are revealed in images. Without knowledge of such theories, students’ compositions may let prevailing discourses go unchallenged. Implications of the affordances and limitations of illustration as an alternative form of composition in an English-language learning class will also be discussed.

Developing Multimodal Literacy in Students – A Pedagogical Approach

Victor Lim Fei

Ministry of Education, Singapore

In this presentation, I will introduce the basic tenets of multimodal literacy and argue that the increasing multimodal nature of our communicative landscape presents implications for literacy education. I will describe a project to develop instructional strategies for teaching multimodal texts in the Singapore Secondary School context that is informed by research in the Systemic Functional Multimodal Discourse Analysis (SFMDA) approach to multimodal texts (O’Halloran & Lim, 2014) and situated within the Learning by Design Framework (Cope & Kalantzis, 2015).The frameworks and approaches developed for critical viewing are based on the pioneering work in multimodality by O’Toole (2010) and Kress & van Leeuwen (2006). This is later extended by Tan, Marissa and O’Halloran (2010), Lim, O’Halloran, Tan, & E. (2012), and Lim & Tan (2017). The frameworks are structured along the metafunctional organisation of meaning and adopt a genre-based approach to multimodal texts. They provide scaffolds for students to access the meanings made in multimodal texts by introducing the features and typical functions of the text, as well as highlighting the common strategies used in these multimodal texts to make meaning. The approaches also leverage educational technology for collaborative annotation, discussion, and production. The goal is, as expressed in the Singapore Ministry of Education English Language Syllabus 2010, for students to actively construct meaning from multimodal texts, comprehend closely and critically a variety of different types of texts, and think critically to become critical viewers.

References

Cope, B. & Kalantzis, M. (2015). An Introduction to the Pedagogy of Multiliteracies. In

Cope, B. & Kalantzis, M. (eds) A Pedagogy of Multiliteracies. Hampshire: Palgrave Macmillan. 1-36.

Kress, G., van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design (2^nd edition). London & New York: Routledge (1^st edition 1996).

Lim, F.V., O'Halloran, K. L., Tan, S., & Marissa, K. L.E. (2015). Teaching visual texts with multimodal analysis software. Educational Technology Research and Development, 63(6), 915–935.

Lim, F.V. & Tan, K.Y.S. (2017). Multimodal Translational Research: Teaching Visual Texts. In Seizov, O. & Wildfeuer, J. (eds) New Studies in Multimodality: Conceptual and Methodological Elaborations. London/New York: Bloomsbury. 175-200.

O’Halloran, K.L., & Lim. F.V. (2014). Systemic Functional Multimodal Discourse Analysis. In Norris, S., & Maier, C. (eds) Texts, Images and Interactions: A Reader in Multimodality.

Berlin: De Gruyter. 135-154.

O’Toole, M. (2010). The Language of Displayed Art (2^nd edition). London & New York: Routledge (1^st edition 1994).

Tan, S., E, Marissa K. L., & O’Halloran, K. L. (2012). Multimodal Analysis Image (Teacher edition and student edition). Singapore: Multimodal Analysis Company.


The influence of multimodal ensembles on EFL students' audio-visual comprehension

Natalia Norte Fernandez-Pacheco

University of Alicante, Spain
[e-mail]

During the last decade innovations in Information and Communication Technologies (ICT) have tremendously influenced how people interact with each other. A proliferation of new forms of discourses have been constantly emerging and contributing not only in the development of communication, but also in how languages are taught and learnt. Printed elements have been replaced by digital resources (Jewitt, Bezemer, Jones, & Kress, 2009), which bring together diverse modes of communication. As a consequence, the idea of interaction just through written or oral words has been transformed into a more multimodal one in which other modes such as gestures, music and images are also taken into consideration in the process of meaning-making. In educational contexts, digital language materials (e.g., Ipack, blogs, social networks and vodcasts) are gaining importance inasmuch as they are seen as potential tools that promote an innovative, attractive and motivating multimodal approach to language learning. Since multimodality focuses on multimodal ensembles (Jewitt, 2013; Jewitt and Kress, 2003; Kress, 2010), I would like to focus on how they may enhance language students’ audio-visual comprehension. This presentation will be based on the description of some extracts from a multimodal analysis carried out on two language learning vodcasts from the British Council. Making use of ELAN as the main multimodal annotation tool, I will report on the different orchestrations of modes contained in both vodcasts and how I was able to elaborate comprehension tests according to the ensembles that appeared in both vodcasts. These tests were necessary to check the impact of each ensemble on students’ comprehension. The results corroborated the main hypothesis of this study: when there was a greater number of orchestrated modes, EFL students’ audio-visual comprehension improved. The conclusions refer to the importance of multimodality and the implementation of audio-visual materials to improve students’ listening skills.  

Keywords: Multimodal ensembles, language teaching materials, digital tools, audio-visual comprehension

References

Jewitt, C. (2013). “Multimodal methods for researching digital technologies”. In S. Price, C. Jewitt, & B. Brown (Eds.). The SAGE handbook of digital technology research. London: Sage, 250-265.

Jewitt, C., Bezemer, J., Jones, K. and Kress, G.  (2009). “Changing English? The impact of technology and policy on a school subject in the 21st century”. English Teaching: Practice and Critique, 8 (3), 8-20. Retrieved from http://edlinked.soe.waikato.ac.nz/research/files/etpc/files/2009v8n3art1.pdf

Jewitt, C., & Kress, G. (Eds.). (2003). Multimodal literacy. New York: Peter Lang.

Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. London: Routledge.


Images of climate change in science textbooks - Negotiating the social and physical complexities of anthropogenic climate change within the school science discourse

Eirik Granly Foss, Ph.D.-candidate

University College of South-east Norway
[e-mail]

Marthe Øidvin Burgess, Assistant Professor

University College of South-east Norway
[e-mail]

The disciplinary discourses of the natural sciences traditionally represent natural phenomena as something happening in neutral chains of causality without connection to human culture (Halliday & Martin. 1993, Veel. 1998). However, anthropogenic climate change is not wholly a natural phenomenon in the traditional sense, but rather something that arises where human culture intersects with nature (Ekström & Svensen, 2014, p. 18). This duality makes it a challenging topic for school science discourse and education, yet also a possible source for renewal (Knain, 2015, p.111).

In this paper, we aim to explore the choice of modes in representing these dual aspects of anthropogenic climate change, and whether these choices can be related to modal affordances. We investigate the epistemological commitment (Bezemer and Kress, 2008) involved in language-image interactions that are not necessarily relationally expositional (Martinec and Salway, 2005).

Our starting hypothesis is that the need for technical precision in representing the physical aspects of climate change tend to overshadow the social aspects in written modes as well as multimodal diagrams, and that the textbook images frequently counterbalance this by focusing on the interpersonal aspects of the social issues involved, such as victims and consequences. The textbook images may therefore function to anchor the text as a whole in an environmentalist discourse.