Multimodality, ethnography and education in south america

References

The New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60-93.

The New London Group. (2000). A pedagogy of Multiliteracies designing social futures. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the design of social future (pp. 9-35). New York: Routledge.

A Multimodal Discourse Analysis of Unreliable Narrative in Detective Films

Kong Youqi, PHD candidate

Department of Linguistics and Translation, City University of Hong Kong
[e-mail]

Booth (1961) first introduces the notion of the unreliable narrator and regards it as a storytelling device. As we enter into an age of digitalization, unreliable narrative is most likely to be developed with the utilization of diversified modes. Therefore, the current research aims to explore how unreliable narrative is instantiated verbally and visually in filmic discourse, especially detective films. As we know, one noticeable feature of the detective film is that the narrative provided by a character is not likely to be completely sincere and some vital information may be misrepresented to viewers in a tricky way to distance them from the truth. In the current study, data comes from Agatha Christie movies and Multimodal Analysis Video software is utilized to annotate the video clips and visualize the patterns of system choice by different narrators. Under the theoretical framework of Metafunction of Systemic Functional theory (Halliday, 1978), the current study mainly discusses the interpersonal meaning realized by different systems such as SPEECH FUNCTION, MODALITY and POLARITY in language and visual elements such as GAZE, CAMERA DISTANCE and VIEWING PERSPECTIVE. It is discovered that distinct patterns of choices are made by the unreliable narrator in terms of the above-mentioned SYSTEM, or combination of different SYSTEMs. The system choices made in their conversations and expressions reveal to a large extent what the author/director wants to convey to the reader/listener about who is trustworthy. Furthermore, deeper meaning and effects arising from unreliable narrative is further pursued in the discussion. Along with examples, the findings are elucidated clearly to unveil the complex meaning making process in unreliable narrative in detective films. Hence, this study is expected to facilitate our understanding concerning multimodality in storytelling.

References

Booth, W. C. (1961). The Rhetoric of Fiction. Chicago: University of Chicago Press.

Halliday, M. A. (1978). Language as social semiotic. London: Edward Arnold.

Aligning Human and Machine Perspectives on Multimodality: Epistemological Questions in an Age of Artificial Intelligence

Christiane Zehrer

Formerly at University of Hildesheim (now employed outside academia)
[e-mail]

The starting point of social semiotic multimodality was the moment researchers discovered that the signs used by humans were complex. Complexity here means that different aspects - often called “semiotic resources” - work together and interact with context resulting in regular, yet fuzzy patterns (cf. Halliday 1985, Kress & van Leeuwen 1990, Kress 1993).

A very conclusive account of how humans use semiotic resources comes from Clark (1996), who develops a taxonomy of conventional and non-conventional signs turned “common ground” - the basis and starting point of interaction. Clark reveals that all signs are context first, and that they are moulded into signs either by the very interaction they are used in or by the total of human interactions produced in similar situations. The regularity induced by the potential and limitations of human interaction is called “perceptual salience” by Clark. Similar mechanisms are described from a cognitive linguistics perspective by Langacker (2008).

While all of the abovementioned work displays very sound reasoning and an unquestionable theoretical foundation, data-driven empirical research into human signing is still a rare occurrence. This author did one such study in the 2010-2013 time span (Zehrer 2014). I used iterative pattern search to detect patterning in successive interactions. Now I revisit my material using the more sophisticated machine-learning tools on the market today (see Krüger 2017), and hope other researchers do, too.

Two of the key questions that arise with such new tools are: What differences are there between the patterns machines detect and the ones that humans actually conventionalise and use as signs? And what do the possible differences tell us about our model of human interaction - and about meaning-making? Finally, we will also have to re-consider the character of multimodality research as a human vs. a technical discipline.

References

Clark, Herbert H. (1996): Using language. Cambridge: Cambridge Univ. Press.

Halliday, M.A.K., (1985): An Introduction to Functional Grammar. London: Edward Arnold.

Kress, G. (1993): Against arbitrariness: the social production of the sign as a foundational issue in critical discourse analysis. In: Discourse and Society 4(2), pp. 169-191.

Kress, G., van Leeuwen, T. (1990): Reading Images. Geelong, Viktoria: Deakin University Press.

Krüger, Ralph (2017): Von Netzen und Vektoren. Neuronale maschinelle Übersetzung. In: MDÜ 63(1), pp. 33-39.

Langacker, Ronald (2008): Cognitive Grammar: A Basic Introduction. New York: Oxford University Press.

Zehrer, Christiane (2014): Wissenskommunikation in der technischen Redaktion. Die adäquate Gestaltung situierter Kommunikation. Berlin: Frank & Timme.

Towards a social semiotic framework for researching literacy practices: Integrating multimodal analysis and Jacobson’s functional linguistics

Kunkun Zhang, Assistant Professor

College of Foreign Languages and Cultures, Xiamen University, China
[e-mail]

Linguistic/cultural diversity and the development of new technologies have been shaping children’s literacy development and challenged traditional language-based approaches to literacy research (New London Group, 1996; Kress, 2003). To respond to the challenges that face literacy studies, this paper proposes a social semiotic framework for researching literacy practices, drawing upon social semiotic multimodal theory (e.g. Kress & Van Leeuwen, 2001; Van Leeuwen, 2008) and Jacobson’s (1960) model of linguistic communication.

The proposal of this framework is based on the assumption that children learn literacy through interacting with other human beings. Therefore, the framework takes Jakobson’s model of verbal communication as its grounds, and integrates the many elements of literacy practices into the model. The framework can be represented as:

Context

Social actor ……………………….Text……………………………Social actor

Medium

Mode

As is illustrated in the figure, the framework takes into account the social actors who are learning literacy and those with whom they interact, the socio-cultural and situational contexts of the interaction, the modes of meaning making used in a literacy practice, the texts used for literacy teaching and learning, and the media platforms through which multimodal texts are presented.

I will discuss this framework using data collected for my PhD project (Zhang, 2017) as well as analyses and discussions of these data in previous publications (Zhang, Djonov, & Torr, 2016a, 2016b). The data include the shared readings of a picture book by preschool-aged children and their mothers as well as the practice of reading the same book in children’s television programmes.

This paper provides literacy researchers with a cogent framework for analysing literacy practices, which meets the challenges facing literacy research at the present time when children engage with multimedia discourses and their path to literacy involves using more than one mode of meaning making. I will also discuss the applicability of social semiotics in the analysis of literacy practices.

References

Jakobson, R. (1960). Closing statement: Linguistics and poetics. In T. A. Sebeok (Ed.), Style in language (pp. 350-377). Cambridge, MA: MIT Press.

Kress, G. (2003). Literacy in the new media age. London: Routledge.
Kress, G., & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Hodder Arnold.
New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60-92.
Van Leeuwen, T. (2008). Discourse and practice: New tools for critical discourse analysis. Oxford and New York: Oxford University Press.
Zhang, K. (2017). Representation of picture books and book reading on children’s television: A social semiotic study with implications for early literacy learning. PhD Thesis. Macquarie University, Sydney, Australia.
Zhang, K., Djonov, E., & Torr, J. (2016a). Reading and reinterpreting picture books on children’s television: Implications for young children’s narrative literacy. Children’s Literature in Education, 47(2), 129–147.
Zhang, K., Djonov, E., & Torr, J. (2016b). Reading aloud as performance and its representation on television programmes for children. Social Semiotics, 26(4), 424–444.

A cultural taxonomy of online video-sharing: A contrastive analysis of YouTube and Chinese-language platforms

Leticia Tian Zhang

University of Pompeu Fabra
[e-mail]

Sumin Zhao

University of Southern Denmark
[e-mail]

While multimedia platforms such as YouTube have been widely studied in various fields, native video portals in Asia remain largely unknown. This project aims to bridge this gap by introducing an increasingly popular viewing system in Japan and China, known as danmu (or danmaku). This knowledge is important as it sheds light on emerging technologies in social media and how it shapes social and semiotic practices.

Danmu is a collaborative video annotation system that overlays users’ comments on the video, in a horizontal scroll moving from right to left (Figure 1). Comments are sent asynchronously by anonymous users, but they are embedded in the video and appear at the points of insertion, performing a direct response to the plot, character, music, etc. This content and visual interrelation simulates a synchronous co-viewing experience for the audience.

Figure 1. A danmu video

This paper considers danmu as a semiotic technology, i.e. technologies for meaning-making (Van Leeuwen & Djonov, 2013). Following previous studies on semiotic software like PowerPoint (Zhao, Djonov, & van Leeuwen, 2014), we analyze danmu as a semiotic practice comprising three dimensions - the video player’s design, the insertion of a comment, and the projection - and two semiotic artefacts, the composition of the multimedia player and the annotated video produced by users. We illustrate how the multimodal affordances (i.e. visual, audio and textual resources) enable ordinary users to maximize their online participation, recreating the original video with supplementary and entertaining information.

This study derives from the first author’s work on the role of danmu in language and intercultural learning. By adopting a social semiotic approach, this study innovates the research in applied linguistics and discourse analysis, and contributes to the most recent development of the multimodal paradigm.

References

Van Leeuwen, T., & Djonov, E. (2013). Multimodality and software. In C. Chapelle (Ed.), Encyclopedia of applied linguistics. Oxford: Wiley-Blackwell (Area Editor: Sigrid Norris).

Zhao, S., Djonov, E., & Van Leeuwen, T. (2014). Semiotic technology and practice: A multimodal social semiotic approach to PowerPoint. Text & Talk, 34(3), 349–375

Scifotainment: Evolving multimodal engagement in online science news

Yiqiong Zhang

Guangdong University of Foreign Studies, People's Republic of China
[e-mail]

This article adopts a multi-analytical perspective to examine engagement in an evolving science communication website, drawing upon the theoretical construct of genre (Martin 1992) and tools from multimodal discourse analysis informed by Systemic Functional Linguistics (Halliday and Matthiessen 2004). Engagement strategies for academic discourse (Hyland 2005) are also drawn upon for my analysis. The website has evolved to engage readers from a “verbal nucleus” inherited from traditional news to a “hypermodal nucleus” that caters to the information consumption style of social media, and further to a “scifopost” that integrates social media post into the design. The evolution is driven by the shift from a “deficit” model to a “democracy” model in science communication and a larger cultural shift in the nature of knowledge creation and transmission. Furthermore, the more malleable multimodal resources as compared with verbal resources spearhead the shift from channels for science information to entertainment and engagement, which in turn triggers a similar shift of functions in verbal resources.

Keywords: Multimodal engagement, online science news, social media, deficit model, democracy model, webpage design

References

Halliday, Michael A.K., and Christian M.I.M. Matthiessen. 2004. An Introduction to Functional Grammar. 3rd ed. London, UK: Edward Arnold.

Hyland, Ken. 2005. “Stance and Engagement: A Model of Interaction in Academic Discourse.” Discourse Studies 7 (2): 173-192.

Martin, James R. 1992. English Text: System and Structure. Amsterdam, NL: John Benjamins Publishing Company.

The Social Semiotic Recontextualization of One Scene of The Danish Girl ― from Novel to Script to Film

Zhen Zhang

University of Technology Sydney
[e-mail]

This presentation focuses on the method of doing recontextualization for different genres from the social semiotic perspective. A scene of The Danish Girl which represents the resolution of the male protagonist's gender confusion is selected for detailed study. More precisely, the presentation will discuss the relation of adaptation (Desmond & Hawkes 2006), recontextualization (Van Leeuwen 2008) and resemiotization (Iedema 2003) first. After that, how the scene in David Ebershoff's novel (2000) is recontextualized into the film script by the screenwriter (Coxon 2015) and how the latter is recontextualized into the film by the film director Tom Hooper (2015) with the help of social semiotic resources will be explored.

This presentation aims to investigate factors that influence the selection of semiotic modes and cause the change of genres in the process of recontextualization through comparing the similarities and differences in the method of presenting the selected scene in the three types of texts. The presentation also attempts to propose criteria for future adaptation work.

Keywords: Genre, social semiotics, adaptation, resemiotization, recontextualization, The Danish Girl

References

Desmond, J.M. & Hawkes, P.J. 2006, Adaptation: Studying Film and Literature, McGraw Hill New York.

Iedema, R.A. 2001, 'Resemiotization', Semiotica: journal of the international association for semiotic studies, vol. 137, no. 1/4, pp. 23-9.

van Leeuwen, T. 2008, 'Discourse as the Recontextulization of Social Practice', Discourse and practice: New tools for critical discourse analysis, Oxford University Press, Oxford and New York, pp. 3-22.

Interplay between the verbal and the visual realization of stance markers in the discourse of broadcast news reports

Hongqiang Zhu, associate professor

College of Foreign Studies, Jinan University, Guangzhou, China
[e-mail]

Although aspiring to the claim of “objectivity”, it is impossible for news discourse to achieve an absolute objectivity in news practice. Thus, the expression of stance, closely related to news discourse studies, has become a focus of attention from scholarship. The discourse of broadcast news reports is featured with its televisual attributes and thus practiced through interaction between the visual and the verbal messages. The expressions of stance marked in the discourse of broadcast news reports are usually realized through the interplay between the visuals and the verbal messages. By drawing upon CCTV-News reports as data, the paper applies the taxonomy of word-image relation proposed by Meinhof to explore how stance markers are realized through the interaction of visual-verbal. This paper identifies three types of stance markers in the discourse of broadcast news reports, which includes “co-reference between visual and verbal”, “verbal-oriented authorial voice” and “image-oriented voice attributing to the participants”.

Keywords: expression of stance, the discourse of broadcast news reports, word-image relations, journalistic objectivity


ON MECHANISMS OF SENSE FORMATION AND TEXTUALIZATION IN SOCIAL AND CULTURAL COMMUNICATION

Suren Zolyan, Dr. (Philological Sciences), professor

Institute for Humanities, Kant Baltic Federal University, Alexander Nevsky 12, Kaliningrad, Russia. Institute of Philosophy, Sociology and Law, National Academy of Sciences, 44, Arami, Yerevan, 0010, Armenia.
+ (37410)530571
[e-mail]

Human activity is based on such values as sense and meaning. All the forms of representation of this activity (culture, history, literature, art, politics, law, etc.) can be considered as semantic ensembles. At the same time, the problem - what unites these so different "senses of meaning" or "meanings of meaning" – still remains insufficiently clarified. The interdisciplinary and transdisciplinary description of this diversity can explicate general patterns of the various manifestations of semantics and semantification, and provide an opportunity for mutual understanding also between different humanities.

Meaning production is understood as a result of the conjunction of linguistic and extra-linguistic systems in the process of social interaction and communication. This can be described as a mechanism for correlating multimodal text and set of actual and latent contexts with sets of its possible values (interpretations, worlds, texts) and discourses (language games). With this approach it becomes possible to relate logical and semantic characteristics with cognitive ones. Semantics of the text appears as a dynamic context-dependent value, function of its linguistic, cognitive and referential characteristics. The dynamic approach allows us to reflect on the capacity of texts to generate new values within processes of its generation and functioning. Verbalization and construction of mental representations are considered as mutually influencing multi-stage processes. Understanding is not limited to operations on signs and texts, but by patterns of behavior and interpersonal interaction.

A multimodal Text, not a sign, is the basic unit of communication and the main object and instrument of a consistent process of creation-transfer-preservation-transformation of information. This theory can become the basis for social semiotics dealing with not only signs and texts, but also with semantic aspects of social behavior. This new version is to be a synthesis of social semiotics with interpretative sociology, assigning meaning to complexes of verbal and nonverbal actions.

Wall-texts and signs of learning

Bjørnar Ødegårdstuen

Uit- campus Alta
[e-mail]

What kind of texts does one find on the walls in Norwegian classrooms in the fifth to seventh grades? This is the question that forms the basis of my project. Wall-texts are something one probably finds in most classrooms, but, at least in the Norwegian context, no systematic surveys have been made of these texts.

These wall-texts are part of a school context and an interesting approach is to try to understand what they say about learning. Here it is interesting to draw on Kress and Bezemer's definition of learning as transformative engagement. They describe learning as "the inevitable outcome of any and every engagement with the (socially made) world” (2016, p. 37). In line with their broad view of learning, they also have an inclusive view of how learning is made evident. An important part of their study is to describe different signs of learning.

Bezemer and Kress do not primarily write about a school context, but how they describe learning implies a challenge to the school. (s. 61) In an almost programmatic formulation they write: “In a social semiotic approach, the aim is to document, analyze and evaluate what is learned, not what is not learned. It is to notice and render visible learning that often goes unnoticed, and that, in being taken for granted, has been too often still remains invisible” (s. 61)

As this abstract is written, I am collecting my material and do not know exactly what I will find. However, it seems that Bezemer and Kress`s open attitude towards what signs of learning can be and how knowledge “looks”, will offer an interesting way to look at my data. Traditionally school knowledge has been closely connected to verbal (and especially written) language. Hopefully, my project can help to discover and reflect on how other modalities also can be signs of learning.

References

Bezemer, J. & Kress, G. (2016) Multimodality, Learning and Communication: A social semiotic frame. Routledge

Academic argument within multimodal representations – A study of students’ communication within a blended learning environment in higher education

Author Eva Svärdemo Åberg

Stockholm University, Sweden
[e-mail]

Academic literacy practices are changing, due to an overall digitized development in higher education (Lea 2004; Beaufort 2007). During the last decades, several research studies have claimed the importance of providing new digital learning resources to students in higher education. Many research studies emphasize the role ICT plays in students´ learning process and the opportunities it provides to students´ multimodal communication in higher educational practices (Browns et.al, 2008; Taffs & Holt 2013). In an on-going research study, the purpose is to explore how multimodal academic arguments are represented by students. Drawing on both social semiotics (Kress 2003; Archer & Huang 2017) and socio cultural theory (Bakhtin 1991; Matusov 2011), the study will contribute to knowledge about how students’ academic arguments operate and are made salient within and across modes in multimodal compositions. The purpose of the study is embodied in the following overarching research questions: How can arguments and opposing arguments in the students’ multimodal representations be interpreted? And in what ways do the use of different modes afford meaning in the students’ arguments? The research strategy is influenced by net ethnography (Kozinets 2010) and the data consist of writing essays, podcast talks and Power-point presentations which were documented by digital resources. The data were collected within a graduate course in the context of higher education and the students were engaged in different learning assignments by using digital learning resources. An analytical description of the students’ academic argumentation within multimodal representations will be emerged by using the above theoretical frameworks. The analysis will focus on how the mix of different modes within the students´ audio-visual representations will influence academic arguments and the potentials of meaning making. In this paper, I will present a tentative conceptual framework for combining theoretical concepts from different origin and discuss challenges in the data analysis.