Multimodality, ethnography and education in south america

References

Knain, E. (2015). Scientific literacy for participation : A systemic functional approach to analysis of school science discourses. Rotterdam: Sense Publishers.

Education, 88(6), 901-914.
Kineticons: looking at text-based mobile communication multimodally

Dr Agnieszka Lyons

Language Centre/Department of Linguistics, SLLF, Queen Mary University of London

This paper is concerned with the appropriation of the affordances of text-based communication in digital media to evoke associations with multimodal communication, specifically visual, auditory and haptic experiences accompanying observed nonverbal phenomena and actions.

Through the analysis presented in the paper, I identify a user-initiated language development serving to express multimodal meanings within a written medium often simplistically treated as mono-modal. Multimodal meaning-making is achieved in text-only digital discourse in the form of discursive enactment expressed by kineticons defined as typographically marked graphical representations of actions and observable phenomena in digital communication, and characterised by a set of formal and functional features which distinguish them from mere description of actions and phenomena. Rather than describing, kineticons trigger associations with nonverbal communication from prior experience. Syntagmatically, kineticons form part of the organisational structure of messages in which they occur based on their functional relationship as attitudinal indicators whose scope extends over the preceding stretch of text. In terms of intentionality, kineticons demonstrate a complex relationship to the discursively constructed alterae personae of communicators.

I conclude by recommending that instances of discursive enactment expressed through kineticons should be analysed in the context of their multimodal meaning-making potential and seen as linked to the emergence of the phenomenon of discursively evoked multimodality.

Who am I? A multimodal self-presentation by newly arrived young immigrants

Eva Maagerø

University College of Southeast Norway
[e-mail]

Henriette Hogga Siljan

University College of Southeast Norway
[e-mail]

Aslaug Veum

University College of Southeast Norway
[e-mail]

Many of the young immigrants who recently have arrived in Europe have neither competence in English nor in the majority language of the country that they have come to. When they start school, they, as all young people do, want to communicate and present themselves to the classmates and the teachers. Oral language is the natural mode of meaning when people meet in most new settings. In this case, the lack of a common language makes communication difficult. Other modes of meaning need to be explored (Mills, 2016; Mills, Davis-Warra, Sewell, & Anderson, 2016).

In this project, we investigate how newly arrived immigrants, 15 – 17 years old, use multimodal texts to make presentations of themselves. Their texts consist of images made by their mobile devices and written language (short verbal texts written in the majority language, in this case Norwegian, and in the students’ mother tongue). The photos are of important things or activities in the students’ daily life. The multimodal texts are presented in the classroom, and also in an exhibition organized in school where students from other classes, teachers and members of the community board are the audience. The exhibition consists of one multimodal text from each student.

In this paper, we present our analysis of the students’ multimodal texts and pay special attention to the interplay between image and language (Blunden, 2016; Painter, Martin, & Unsworth, 2013). In addition, we discuss how multimodal texts like these can serve as a self-presentation when language cannot be the prominent mode. We also want to comment on the use of exhibition to give these students a voice in their new community (Walker Rettberg, 2014).

References

Blunden, J. (2016). The language of displayed art(efacts). Linguistic and sociological perspectives on meaning, accessibility and knowledge-building in museum exhibitions. Doctor of Philosophy. University of Technology Sydney, Sydney.

Mills, K. A. (2016). Literacy Theories for the Digital Age. Social, Critical, Multimodal, Spatial, Material and Sensory Lenses Bristol: Multilingual Matters.

Mills, K. A., Davis-Warra, J., Sewell, M., & Anderson, M. (2016). Indigenous ways with literacies: transgenerational, multimodal, placed, and collective. Language and Education, 30(1), pp. 1-21.

Painter, C., Martin, J. R., & Unsworth, L. (2013). Reading visual narratives: image analysis of children's picture books Sheffield:Equinox.

Walker Rettberg, J. (2014). Seeing ourselves through technology: how we use selfies, blogs and wearable devices to see and shape ourselves. Basingstoke: Palgrave Macmillan

Multimodal and Hypermodal Knowledge Mangement Across Audiences: An empirical investigation of Harvard Business Review’s Communication and Dissemination Strategies

Carmen Daniela Maier, Jan Engberg

Aarhus University, Denmark

As part of an on-going research work related both to mapping and reflecting upon the latest multimodal strategies of communicating and disseminating academic knowledge (Maier and Engberg, 2013 and 2014; Engberg and Maier, 2015, 2016 and forthcoming), this paper explores the adjusting strategies employed in the transition from research-based articles to other generic forms when remediating knowledge for different audiences in different contexts.The digital context of the Harvard Business Review represents the source of the present empirical evidence which includes articles and related digital materials (e.g. videos, webinars, “Explainers”, animation films, etc.) targeted at academics and/or practitioners.

The research work is founded on a multimodal approach in order to deal with and reflect upon how knowledge is managed through specific knowledge building processes when addressing audiences motivated by different needs and expectations. We focus on the semiotic modes’ interaction in the hypermodal generic networks in order to identify the impact of the semiotic interplay upon the adjusting strategies employed. Furthermore, we demonstrate how the level of complexity of the knowledge communicated in the articles and their related digital materials may be investigated. Thus, we combine the multimodal perspective with a perspective upon knowledge management concerned with the complexity and explanatory quality of the knowledge communicated and disseminated in the digital context of Harvard Business Review.

The communication and dissemination of academic knowledge will continue to evolve in ways that can be unforeseen at the present moment due to rapid technological developments, and therefore we consider that keeping pace with these developments might be one of researchers’ biggest challenges in years to come.

The findings of this research work are primarily meant to contribute to a better understanding of the multimodal adjusting strategies required today and in the future when academics have to address various audiences and have to manage knowledge accordingly. 

References

Maier, C. D. & Engberg, J. (2013). Tendencies in the Multimodal Evolution of Narrator’s Types and Roles in Research Genres. In Gotti, M. & Sancho G. C. (eds.) Narratives in Academic and Professional Genres. Bern: Peter Lang.

Maier, C. D. & Engberg, J. (2014). Tendencies of Multimodal Gradations in Academic Genres Network. In Engberg, J., Maier, C. D. & Togeby, O. (eds) Encounters between Literature, Developing Communicative Norms and Knowledge, Tübingen: Gunter Narr Verlag.

Engberg, J. & Maier, C. D. (2015). Exploring the hypermodal communication of academic knowledge beyond generic structures. In Bondi, M.& Mazei, D. (eds.) Discourse in and through the media. Cambridge Scholars Publ.

Engberg, J. & Maier, C. D. (2016). Challenges in the new multimodal environment of research genres: What future do "Articles of the Future" promise us? In N. Artemeva & Freedman, A. (eds.) Genre Studies around the Globe: Beyond the Three Traditions, Inkshed Publications.

Engberg, J. & Maier, C. D. (forthcoming). Researchers’ move from page to screen: Addressing the effects of the video article format upon academic user engagement and knowledge building processes. In C.S. Guinda (ed.). Engagement in Professional Genres. John Benjamins.

Identifying Features of Multimodal Structures Used in Modern Mass Media Discourse
Larysa Makaruk

Lesya Ukrainka Eastern European National University, Ukraine

When we read modern English-language newspapers and magazines, we periodically come across structures which would be unrecognizable for readers a century ago. It may involve a portion of a word-combination, a sentence or a text that contains within it certain graphic characters or symbols having various structural, semantic, or pragmatic values.

Conventional alphabetical symbols are customarily dominant in those words, but the insertion of even one symbol which differs from this pattern instantly gives the word a totally different, multimodal character; this alters the way readers are likely to react to the text containing this structure. In tandem with this, special stylistic effects can be achieved using this technique, but a certain level of background knowledge and specialized literacy is needed to decipher these structures and to comprehend to writer’s intention.

In the majority of cases a pictogram, graphic symbol or a modified letter which is substituted for a single letter is complete in itself, but it becomes an integral part of a word, or even a complete text. The most common patterns of using alphabetical characters in combination with other entities to create a multimodal ‘word’ involve the insertion of pictograms, punctuation marks, figures, mathematical symbols, or pictures, or else a modified, amalgamated or unexpectedly supplemented form of some letter, to achieve a specific effect.

These processes affect the classical paradigms and theories relating to signs and sign systems, and were the catalyst of our desire to make a personal contribution to the development of a theoretical and methodological foundation for multimodality studies in the twenty-first century.

Types and Characteristics of English Non-verbal Graphic Means
Larysa Makaruk

Lesya Ukrainka Eastern European National University, Ukraine

Nataliya Kashchyshyn

Ivan Franko National University of Lviv, Ukraine

The diverse range of semiotic resources which are employed in modern written communication involves a number of processes for which an in-depth theoretical and methodological basis has not yet been devised in modern linguistics.

This research shows the necessity of creating a classification of the graphic elements used in modern English communication. They have been placed in several distinct groups in terms of their common features, in order to study the structure, form and meaning with the prospect of identifying their true value and the possibilities they open up in the field of communication.

On the basis of available mass media resources we identified non-verbal devices and divided them into several groups on the basis of common features. These groupings are the following: segmentation; supplementary graphic effects; font and colour; non-pictorial and non-photographic graphic elements; iconic elements; and infographics.

Each item has distinctive features, but when several items or patterns are combined they produce special effects, changing the meaning of the meaning which is implied by the text or text fragment. The aspects which deserve further consideration include the development of possible text models based on the affordance of the modes, and the choice of an appropriate methodology for studying them.

In our research, on the basis of the dominant component, we single out two types of models—monomodal models and multimodal models. The monomodal category is further divided into graphic and visual forms; the multimodal ones are subdivided into graphic and graphic-visual categories. All of these subtypes can be further subdivided in terms of the dominant multimodal component, such as iconic, ideographic, diacritical, symbolic, and others. Each has its own structure and peculiarities. All these modes, as well as the textual models which they produce, will be presented in detail.


Colors in Dan Mu: Emerging Meaning-making Norms and Community Online

Teng Man
University of Macau, Department of English

[e-mail]

Recently there is an emerging comment tool provided by most video-sharing websites in China. Comments sent by using this tool appear, instead of under the videos, on the video screen (Figure 1), which enables video audiences to send their comments and read other users’ comments without pulling down the screen. Such comments are called “Dan Mu” which is the pinyin of Chinese characters “弹幕”. This paper intends to investigate how Internet users apply various semiotic resources provided by the tool to accommodate online communicative needs, and at the same time construct online identity and community. By adopting ethnographic observation and concepts taken from multimodality, it is found that despite various modes involved when using the tool (color, written texts, sound and moving image, etc), the meaning making process of Dan MU is basically a process of keeping cohesion between one multimodal ensemble (audience-generated comments) and another one (either user-generated or institution-generated videos). However, taking color as an example, the paper tries to illustrate that it is by no means an easy task to achieve this cohesion. There are various norms that need to be followed in order to send a Dan Mu in the “right” color. Furthermore, the paper suggests that these norms are multilayered, interwoven with each other: norms for Dan Mu forms and contents, for combination of multiple modes, for social-cultural considerations.

Keywords: Mode, multimodality, online comments, norms, colors

Figure 1 Excerpts with Dan Mu (http://www.bilibili.com/video/av4000223/ )

If it looks like a duck. Names as shared signifiers for discussing “cuteness” in healthcare robotics

Emanuela Marchetti, Assistant Professor

Department for the Study of Culture, Media Studies, University of Southern Denmark, Odense

“Cuteness” emerged as a key value within the Smooth Project [7], which targets the design of a robot to facilitate daily tasks in retirement homes. The Smooth project is planned from 2017 to 2020 and it follows a user centred design approach [4]. During the first stage, observations and interviews were conducted at the home and three scenarios were established for the robot. The first two scenarios are of a practical nature such as collecting laundry and garbage from the residents’ rooms, the third scenario is social, as the robot should accompany the residents to the dining room at the time of their meals. During the second (current) stage, different designs were explored through 3D and 2D digital visualisations. These visualisations were discussed in two co-design workshops and email exchanges within the research team, which includes: academics, stakeholders (local companies and institutions), and end users from the home. Data for this study were gathered through field notes during the workshops and discourse analysis of email exchanges [2]. A testing workshop with an initial prototype is planned at the home for spring 2018.

This study builds on multimodality [3] and digital ecologies ([5] and [1]), to explore the symbolic function of verbal language in shared meaning making, regarding the aesthetic and interactivity of the robot. At the same time, the robot is understood as an embodied system contextualised within a digital ecology that encompasses social activities, individuals and physical artefacts with different roles [6]. In this study animal names, such as: penguin, bear, giraffe, and swan, acted as shared signifiers, enabling the research team to decode the different visualisations of the robot and explore its aesthetics, functionalities, and possible interactions.

References

De Greef, J. and Belpaeme, T. (2014) “Why Robots Should Be Social: Enhancing Machine Learning through Social Human-Robot Interaction.” PLoS ONE 10(9).

Drotner, K. and Mosberg Iversen, S. (2017) “Digitale Metodologier. At skabe, analysere og dele data.” Samfundslitteratur.

Kress, G. (2010) “Multimodality. A Social Semiotic Approach to Contemporary Communication.” Taylor and Francis.

Preece, J., Rogers, Y., and Sharp, E. (2015) “Interaction Design. Beyond Human Computer Interaction.” Wiley and sons.

Raptis, D., Kjeldskov, J., Skov, M. B. and Paay, J. (2014) “What is a Digital Ecology? Theoretical Foundations and a Unified Definition.” Australian Journal of Intelligent Information Processing Systems. 13 (4), pp. 1-5.

Riek, L. (2017) “Healthcare Robotics.” Communications of the ACM, 60 (11), pp. 68-78.

Smooth Project: http://smooth-robot.dk/en/home/

Multimodal evangelism, or, is everything really multimodal? Reflections on multimodality and gender

Kate Maxwell

University of Tromsø
[e-mail]

Lilli Mittner

University of Tromsø
[e-mail]

Multimodality has proved to be a fruitful methodology in many areas, from plant science (e.g. Cross and Decambiaire 2017) to the performing arts (e.g. Atã and Queiroz 2017). Recently, it has been shown that the intersection of multimodality and gender is also a productive research area (Machin, Caldas-Coulthard, and Milani 2016; Maxwell and Mittner 2018). One result of the various wanderings of multimodality is the potential for the new disciplines to feed back into multimodality itself. Gender studies offers critical reflections on power relations, embodied knowledge, the ramifications of white supremacy, and ecological challenges as matters of political and social transformation. But is multimodality as a research area ready for – or even in need of – such reflection?

A multimodal analysis of the work of gender reveals that gendered structures, gender bias, and gendered meaning are all carried out across a variety of modes. Moreover, a social-semiotic approach reveals how deeply these are rooted in society and in what ways not only bodies, language, and representation but also – as for example in the field of music – sound, voices, instruments, and stylistic choices are greatly influenced by gender. Nevertheless, in this period of reflection which we are currently witnessing within multimodality, the questions ‘is everything really multimodal?’ and ‘is multimodality really everything?’ need to be seriously considered. Gender studies (particularly Black and postcolonial gender studies) tells us that for an equal and non-discriminatory meeting among different parties, the interchange has to be two-way (hooks 1992, Ahmed 2017). Such a notion is also essential to multidisciplinary collaboration, which is becoming increasingly encouraged and rewarded by national and international policymakers (Maxwell and Benneworth 2018).

Therefore, as it seeks to move forward in today’s research environment and society, can multimodality learn from gender studies that essentialist binaries such as ‘male’ and ‘female’– or ‘us’ and ‘them’ – are exclusionary and unhelpful? Or is it time to close ranks and establish the identity of multimodality more firmly in order to better understand its place in the academy and beyond?

References

Sara Ahmed 2017. Living a Feminist Life. Durham NC: Duke University Press.Pedro Atã and João Queiroz 2017. ‘Multimodality in Distributed Cognition: An analysis of Challenges of Improvised Oral Poetry’. Paper presented at the conference Multimodality: Toward a New Discipline, Bremen, 20-22 September

Judie Cross and Elisabeth Decambiaire 2017. ‘The Rainbow Effect’. Paper presented at the conference Multimodality: Toward a New Discipline, Bremen, 20-22 September

bell hooks 1992. ‘Eating the Other: Desire and Resistance’. In Black Looks: Race and Representation. Boston: South End Press, pp. 21–39

David Machin, Carmen Rosa Caldas-Coulthard, and Tommaso M. Milani 2016. ‘Doing Critical Multimodality in Research on Gender, Language and Discourse’. Gender and Language 10:3, Special issue: Gender and Multimodality, 301–308.

Kate Maxwell and Lilli Mittner 2018. ‘Multimodal Aesthetics and Gender in Beck's Song Reader’. In Elise Seip Tønnessen and Frida Forsgren, eds, Multimodality and Aesthetics. London: Palgrave Macmillan (forthcoming)

Kate Maxwell and Paul Benneworth 2018. ‘The Construction of New Scientific Norms for Solving the Grand Challenges: Reflections from the Norwegian Idélab Programme’. Palgrave Communications (in press).
Embodied collaboration in a film production practice with iPads

Thilde Emilie Møller, Assistant Professor, Ph.D.

University College UCC
[e-mail].

The aim of the present study was to examine students’ interactions in a film production practice with iPads in the upper level of the Danish primary school (grades 4-6) through Sigrid Norris’ multimodal (inter)action analysis (Norris, 2004, 2011a, 2012a). This analytical method is applicable for analyzing how social actors construct meaning systems in specific social actions (Norris, 2004). There is a great need for applying this analytical perspective in education studies where digital technology is a learning resource because the technologies enable new ways of making meaning. Many schools in Denmark have invested in iPads and the Danish school curriculum encourages teachers to include digital technology in their teaching (Undervisningsministeriet, 2014). Through video observations, I have looked at the situated interplay between the students’ use of modes in a collaborative film production practice. Through Norris’ concept of system of representation, I analyze the students’ choice of modes in an interaction, and I draw on Norris’ concept of multimodal configuration to analyze which modes may take on special importance in the specific context (Norris, 2004, 2011a, 2012a, 2014). My findings indicate that the students’ meaning systems were constructed through embodied interaction, and especially the students’ ability to collaborate through the gaze and the hands was of great importance (Møller, 2017). My analysis shows that the touchscreen and the screen size of the iPad bring potentials and challenges into film education, which we have to consider. My study gives rise to discussion of a renewed awareness of media technologies in film production practices, and on this basis to discuss film didactic. My contribution to this conference especially will highlight how my study contributes to developing Norris’ multimodal (inter)action analysis in connection with collaborative and digital literacy practices.

References

Burnett, Cathy, Julia Davies, Guy Merchant & Jennifer Rowsell (Eds.) (2014). New Literacies around the Globe: Policy and Pedagogy. New York: Routledge.

Jewitt, Carey (2008). Multimodality and Literacy in School Classrooms. Review of Research in Education 32: 241–67.

Jewitt, Carey (2009). Technology, Literacy, Learning, a Multimodal Approach. London: Routledge. Jewitt, Carey (Ed.) (2014). The Routledge Handbook of Multimodal Analysis. London: Routledge.

Kress, Gunther R. (2003). Literacy in the New Media Age. Literacies. London: Routledge.

Kress, Gunther (2009). Multimodality: A Social Semiotic Approach to Contemporary Communication. UK: Taylor & Francis.

Kress, Gunther (2010). Multimodality, a Social Semiotic Approach to Contemporary Communication. London: Routledge.

Kress, Gunther, Carey Jewitt, & Jon Ogborn (2001). Advances in Applied Linguistics: Multimodal Teaching and Learning: The Rhetorics of the Science Classroom (1). London: Continuum.

Kress, Gunther & Theo van Leeuwen (2001). Multimodal Discourse, the Modes and Media of Contemporary Communication. London: Arnold, Hodder Education.

Kress, Gunther, Carey Jewitt, Jill Bourne, Frank Anton, Hardcastle, & Ken Jones (Eds.) (2004). English in Urban Classrooms: A Multimodal Perspective on Teaching and Learning. London; New York, NY: Routledge.

Merchant, Guy (2014). Young Children and Interactive Story-Apps. In C. Burnett, J. Davies, & J. Rowsell (Eds.), New Literacies around the Globe: Policy and Pedagogy, 148–66. New York: Routledge.

Merchant, Guy (2015). Keep taking the tablets: iPads, story apps and early literacy. Australian Journal of Language & Literacy 38 (1): 3–11.

Norris, Sigrid (2004). Analyzing Multimodal Interaction, a Methodological Framework. London: Routledge.

Norris, Sigrid (2011a). Identity in (Inter)action, Introducing Multimodal (Inter)action Analysis. Berlin New York: Walter de Gruyter Mouton.

Norris, Sigrid (2011b). Three Hierarchical Positions of Deictic Gesture in Relation to Spoken Language: A Multimodal Interaction Analysis. Visual Communication 10 (2): 129–147.

Norris, Sigrid (2012a). Multimodality in Practice, Investigating Theory-in-Practice-through-Methodology. New York: Routledge.

Norris, Sigrid (2012b). Practice-based research: multimodal explorations through poetry and painting. Journal Multimodal Communication 1 (1).

Norris, Sigrid (2014). Modal Density and Modal Configuration. In C. Jewitt (Ed.), The Routledge Handbook of Multimodal Analysis (86-99). London: Routledge.

Norris, Sigrid, & Rodney H. Jones (2005). Discourse in Action, Introducing Mediated Discourse Analysis. London: Routledge.

Sakr, Mona, Carey Jewitt, & Sara Price (2014). The Semiotic Work of the Hands in Scientific Enquiry. Classroom Discourse 5 (1): 51–70.

Undervisnings Ministeriet. 2014d. “Vejledning for faget dansk”. Set 6. september 2015. Http://www.emu.dk/modul/vejledning-faget-dansk. Vejledning for faget dansk. http://www.emu.dk/modul/vejledning-faget-dansk.

Walsh, Maureen & Alyson Simpson (2013). Touching, tapping ... thinking? Examining the dynamic materiality of touch pad devices for literacy learning. Australian Journal of Language & Literacy 36 (3): 148–57.

Walsh, Maureen & Alyson Simpson (2014a). Exploring literacies through touch pad technologies: the dynamic materiality of modal interactions. Australian Journal of Language and Literacy 37 (2): 96–106.

Walsh, Maureen & Alyson Simpson (2014b). Pedagogic conceptualisations for touch pad technologies. Australian Journal of Language & Literacy 37 (2):128-138.

Wertsch, James V. (1998). Mind as Action. New York Oxford: Oxford University Press.

Wohlwend, Karen E. (2013). Play, Literacy, and the Converging Cultures of Childhood. In J. Larson & J. Marsh (Eds.), The sage handbook of early childhood literacy (80-95). London: SAGE.

Wohlwend, Karen E. (2015). One Screen, Many Fingers: Young Children’s Collaborative Literacy Play With Digital Puppetry Apps and Touchscreen Technologies. Theory Into Practice 54 (2): 154–62.

Wohlwend, Karen & Buchholz (2014). Paper pterodactyls and popsicle sticks: Expanding school literacy through filmmaking and toymaking. In C. Burnett, G. Merchant, J. Rowsell & J. Davis (Eds.), Conversations about literacy in education: New meaning making practices (33-49).

Cultivating a multimodal gaze: A classroom research study in a higher education English Studies program

Nora Nagy
Department of English Applied Linguistics, University of Pécs, Hungary
School of English and American Studies, Eötvös Lorand University, Budapest, Hungary
[e-mail]

To explore the learning potential of a course in multimodal discourse analysis, I designed a course for BA and MA level university students in English Studies at a Hungarian university. My main objective was the development of their multimodal literacy and the cultivation of a multimodal gaze in the participants of the course. As an English language teacher, researcher and editor I noticed the need for building knowledge about multimodality. These students are mostly unaware of the multimodal aspect of the world that surrounds them and lack the language and thus the knowledge to talk and write about their multimodal experiences. Although they are expected to interpret and produce multimodal texts, their multimodal literacy is rarely developed explicitly. During the semester I collected data with the help of writing tasks and questionnaires which I then analyzed qualitatively. Based on these findings I present the most beneficial tasks and course materials. My findings indicate the relevance of such a course in teacher education and the importance of developing skills in social semiotic multimodal analysis in language learners and teacher trainees.

‘How do you illustrate the atmosphere of a story?’A multimodal social semiotic approach to making illustrated books for language learners

Nora Nagy
Department of English Applied Linguistics, University of Pécs, Hungary, School of English and American Studies, Eötvös Lorand University, Budapest, Hungary
[e-mail]

Making illustrated readers for foreign language (FL) learners is a complex intersemiotic process which is negotiated among editors, illustrators, authors and the directors of publishing houses. Both the ideational and textual metafunctions (Halliday, 1978; Kress & van Leeuwen, 1996) of such texts are affected by the participants in this creative process. Assumptions made by the editors about the needs and literacy development of the language learner and their knowledge of the affordances of illustrations and image-text relations influence the creative process of the illustrator, whose understanding of the purpose of the materials shapes their visual work. The editor is often more interested in the verbal aspect of the narratives while the illustrator is more concerned with the visual elements. The communication between the two, that is their ability to express their opinions, expectations and approaches, often leads to misunderstanding and confusion. This paper will illustrate how the editors’ and illustrators’ communication, knowledge and beliefs shape the content of these books. A series of interviews and questionnaires were conducted with editors and illustrators of such materials. Based on the findings of the study, I propose a set of multimodal strategies that bookmakers can apply in their creative process to foster better understanding of the choices they make. A multimodal social semiotic approach to creating multimodal texts aimed at FL learners can contribute to the production of books with rich meaning-making potential and pedagogical affordances.

References

Halliday, M.A.K. (1978). Language as Social Semiotic. London: Edward Arnold.

Kress, G., & van Leeuwen, T. (1996). Reading Images: The Grammar of Visual Design. London: Routledge.


Simplified visual language: Graphic abstraction in Finnish crossword puzzles

Silja Nikula, Doctor of Arts

Faculty of Art and Design, University of Lapland, Finland
[e-mail]

Crossword puzzles including the small drawings inside them are a Finnish speciality. The puzzles are like games for entertainment and problem-solving. The pictures in them are simplified, most often black-and-white, and solving their meanings is the idea of the puzzle. In my presentation, I ask: What are the typical features in the puzzle imagery; what is the structure of the imagery and in which way do the pictures refer to the content they represent?

I first classify the pictures according to Peircean sign classes: icon, index and symbol. This classification is fulfilled using metonymy and metaphor. After that, methods for graphic abstraction are paid attention to: how pictures are simplified and how they represent items and ideas. There are certain principles found in using lines, shapes and perspectives. Colours do not make the pictures clearer but are used for interest and entertainment. Similarities to other visual imagery, such as cartoons can be found. Humorous style is also used.

The graphic abstraction is based on mental images and organized in categories. In pictures, differences between categories are emphasized. Prototypes and stereotypes are common. The pictures have to be very ordinary although cryptic pictures might be more interesting for problem-solvers. Connotations of words become visible through simplified images.

I collected the material between 2013–2016 and continue to solve new puzzles. The study pays attention to the imagery that is conventional and renews itself very slowly. Crossword puzzles serve as a cultural container.

Children making meaning of mathematics textbooks - a multimodal study

Malin Norberg, PhD in Education

Mid Sweden University
[e-mail]

This ongoing study focuses on the meeting between child and textbook. Mathematics textbooks are a common teaching resource. Over 75 percent of primary school children, worldwide, are taught mathematics from textbooks, and in Sweden more than 90 percent use textbooks in their mathematics education (Mullis, Martin, Foy & Arora, 2012). In order to analyse the textbook as a resource for meaning making and children’s meaning making, the concepts of theoretical semiotic potential and actual semiotic potential (van Leuween, 2005) will be used. The former concept is understood as the meaning the mathematics textbook has been designed to offer, and the dichotomous concept, actual semiotic potential, as the meaning the individual discovers in her encounter with the textbook. A delimitation to Swedish primary school year 1 (children 7-8 years) will be done, and both digital and printed textbooks will be included. The data consists of video material, and documents in the form of children’s representations. The video material is comprised of 18 children working with printed textbooks, and approximately the same amount working with digital textbooks. A multimodal analysis will be done. This study aims to explore the relationship between the designed and the discovered. The expected outcome of the study is that the mathematical content the textbook was designed to offer is not always what the children in fact discover. This is of great concern for teachers as well as textbooks authors and illustrators. Knowledge of how the relationship between theoretical semiotic potential and actual semiotic potential appears is of interest, and can in the long run support teachers’ planning and implementation of their mathematics teaching.

References

Mullis, I. V., Martin, M. O., Foy, P., & Arora, A. (2012). TIMSS 2011 international results in mathematics. International Association for the Evaluation of Educational Achievement. Herengracht 487, Amsterdam, 1017 BT, The Netherlands.

van Leeuwen, T. (2005). Introducing social semiotics. London: Routledge.

Theorising “New” Semiotic Resources: Paper

Nina Nørgaard, Associate Professor, Dr

Department of Language and Communication, University of Southern Denmark
[e-mail]

Within the social semiotic multimodal paradigm, handling modes such as wording, images, sound and colour is relatively straight-forward since well-developed grammars of those modes already exist (e.g. Halliday 1994; Kress and van Leeuwen 1996; van Leeuwen 1999, 2011). However, when trying to develop a comprehensive framework for multimodal analysis of the novel (Nørgaard in press), I faced the challenge of handling the materiality of my object of analysis, including paper. While some might think of paper in the novel as a semiotically insignificant canvas for the literary narrative, my collection of literary data proved this perception wrong. In my presentation, I will explore the meaning-potential of paper in the novel and consider how this semiotic resource may be systematised within a multimodal framework. My talk will point to challenges involved in handling “new” semiotic resources – new in the sense that they have previously not typically been perceived as semiotic and new in terms of being handled within a social semiotic multimodal framework. Amongst the topics to be discussed are the development of system networks for “new” semiotic resources and the phenomenon of symptoms that turn into signs.

References

Halliday, M. A. K. 1994. An introduction to Functional Grammar (2nd edn.). London: Arnold.

Kress, Gunther & Theo Van Leeuwen. 1996. Reading Images: The Grammar of Visual Design (1st edn.). London and New York: Routledge.

Mourier, Mette and Eric Mourier. 2013. Bogdesign: Tilrettelægning af Illustrerede Bøger (2nd edn.) Copenhagen: Forlaget Grafisk Litteratur.

Nørgaard, Nina (in press). Multimodal Stylistics of the Novel. More than Words. London and New York: Routledge.

van Leeuwen, Theo. 1999. Speech, Music, Sound. London: Palgrave MacMillan.

van Leeuwen, Theo. 2011. The Language of Colour: An Introduction. London and New York: Routledge.

Williamson, Hugh. 1983. Methods of book design: The Practice of an Industrial Craft (3rd edn.). New Haven and London: Yale University Press.

On the Relations between Modes: How Writing Systems can Affect Visual Semiosis in Multimodal Texts

Rumiko Oyama

Meiji University, Tokyo, Japan
[e-mail]

The aim of the paper is to demonstrate that visual semiosis is culturally specific while there are some aspects that are universal. Visual images that are seemingly transparent and easy to decode actually require a framework that is specific to the particular context in which each multimodal text comes into being. The paper explores what determines visual semiotics in relation to linguistic factors of the culture in question. Specifically, the impact of the writing system of Japanese on its counterpart of visual semiosis will be focused on. In Japanese multimodal texts, there still exists a strong sense of directionality of left to right although most textual formats take the horizontal, left to right directionality as in English counterparts.

The paper will therefore draw attention to some Japanese examples which can signify this conventionally prominent directionality of writing and explore how that tendency is manifested in how visual semiosis appears to be conditioned in multimodal texts. For example, the distribution of meaning in the left/right domain of multimodal texts will be discussed in relation to the possibly underlying conditioning of writing systems of right to left. I would also like to address the significance of (visual and writing) directionality as a crucial dynamic that determines the overall meaning making of multimodal texts. Questions to be addressed are how the meaning distribution of left and right manifests certain negative-ness and positive-ness and authority in multimodal texts.

The paper points to that current social semiotic multimodal theory, which is primarily based on Anglophone semiosis in its selection of textual data, could and should now be expanded and developed in relation to the language mode with an emphasis on the impact of one mode on another.

Multimodal communication through puppets in a project with refugee children

Maria Papadopoulou, professor

University of Thessaly, Greece
[e-mail]

Magda Vitsou, tutor

University of Thessaly, Greece
[e-mail]

Eleni Gana, assistant professor

University of Thessaly, Greece
[e-mail]

Over the last years Europe has experienced the greatest mass movement since the Second World War. Nearly a million displaced people fleeing home due to conflicts, economic devastation, human rights violation and persecution, crossed Greece seeking a better future in the Western and Northern European countries. In most cases, societies mobilized and responded to the crisis by supporting refugees in many aspects, such as children’s education. In Greece since October 2016, many refugee children attended the Greek educational system. Needless to say that education for refugee children who have missed years of schooling or had no previous access to it proved to be a rather complicated issue, which was aggravated by the transitory stage of their living. Organizing catch-up programs to meet the needs of the children in education, while at the same time empowering them to overcome trauma and stress was imperative. Therefore, prior to and in parallel with the official courses in schools, informal education activities took place in many refugee camps implemented by Universities, NGOs and volunteers. In this context, we designed a six months’ project called “Literacy through Drama” taking into account that a collaborative environment that is devoid of fear of failure could result in stronger linguistic outcomes (Krashen, 1988˙ Payne, 2011). The project, which was implemented in a reception class in a public school of Volos (Greece), aimed at facilitating children’s literacy by creating an opportunity for authentic situations in a more playful and less schoolish manner (Almond, 2005˙ Calvert, & Sheen, 2015). A large part of the project was carried out through puppets. Their use, in a setting of learning interaction, served as a mediating tool to engage children, explain abstract ideas, and demonstrate processes and concepts, facilitating thus the learning process. The puppets, as a multimodal medium, encouraged children to co-operate, to express themselves in a creative way and to communicate in many modes overcoming language borders. During the making of the puppets and their animation, the factor of 'stress' was significantly decreased and therefore children improved their communication skills and their creativity. The puppets stimulated children through play to think up unexpected linguistic constructions or harmonies, longer monologues and dialogues, to play with different voices, to use all available resources for communication.

The paper discusses multimodal communication practices adopted by the participants during the project, especially focusing on interactions between children and between children and tutors while using puppetry (puppets, persona dolls and others). Video recordings of the sessions were analyzed by means of a multimodal framework to provide an insight into the ways refugee children conveyed meanings through the ‘voice’ of the puppets. Children used a variety of representational modes to communicate, such as named languages (children’s native language, English and Greek), facial and body expressions, gestures and gaze while animating puppets. Through puppets’ animation, children were empowered to express themselves in many modes and to produce stronger linguistic outcomes.

When words and graphs move the eyes: How do experimental eye-tracking data complement with an opinion-based survey?

Giovanni Parodi
Pontificia Universidad Católica de Valparaíso, Chile
[e-mail]

Cristóbal Julio
Pontificia Universidad Católica de Valparaíso, Chile
[e-mail]

Multisemiotic texts composed of different semiotic systems are widely employed in academic and professional environments. Graph/word texts have been identified as fundamental written discourse tools in the construction and transmission of specialized knowledge. Nevertheless, research on multisemiotic texts constituted by words and graphs has been scarce with only a few exceptions. In light of this background, the objective of this presentation is twofold. On the one hand, eye movement behavior was studied in seventy-seven high school Chilean students, in order to determine the effects on reading of causally related texts in Spanish, composed of words and graph. On the other, a twenty-four-question survey on reading habits was administered to 989 university students of five professional careers focusing on reading practices on multisemiotic digital/paper texts. Our general aim was to put together in a broad complementary panorama declarative knowledge (survey) and procedural knowledge (eye tracking measures). The general results partially confirmed our hypothesis: the findings show a concentration on reading words more than graphs, and point to a main speed-up effect when the causal connective is explicit, but the graph is omitted. Complementarily, the major findings of the survey point to a preference for paper format in academic environments, rather than focus on digital texts. Moreover, words more than graph artefacts are selected, when reading for study purposes. These declarative tendencies tend to occur irrespective of the discipline. Interestingly, both approaches complement in a preference for reading words, more than focusing on graphical information.


Multimodal writing tutoring

Akisha Pearman

The University of Cape Town
[e-mail]

Global citizens have diverse resources available to draw on to make meaning. However, having the ability to use their resources in particular contexts is often where power and inequality surface. “Multimodal communicative competence involves the knowledge and use of language concerning the visual, gestural, audio and spatial dimensions of communication, including computer-mediated-communication” (Heberle, 2010: 102). ‘Multimodal competence’ could be considered essential, given today’s diverse communities and the varied mediums in which they communicate. Tutors located within writing centers in higher education have a significant role to play within this larger global context. They not only help students to negotiate meaning within modes such as writing, image, and speech, but also should be experienced in using and negotiating meaning in these modes themselves. In South Africa, where diversity of cultures, people and languages is rich, a key to understanding how to develop ‘multimodal competence’ could be explored by examining multimodal semiotic resources. To respond to this context, the author has developed a doctoral research project, which examines how a group of writing tutors develops themselves as writers and facilitators. They do this through an exploration of musical, visual and written resources in the construction of a personal writing journey video narrative. The data collected will include video and audio recordings, and tutor produced and selected images and written texts. The data will be analyzed using a framework of semiotic metafunctions developed from Halliday (1978) and Kress and Van Leeuwen (2006). The preliminary data will be used to explore how engagement with multimodal resources as a training method stimulates thinking about academic writing, and could provide a window into how this thinking shapes person-to-person interaction and meaning making.

Multimodality and Creativity through Transmedia Storytelling in the Malaysian Higher Education Context

Melissa Shamini Perry

National University of Malaysia
[e-mail]

The technological advancements of the 21^st century have elevated the art of storytelling and allowed for stories to be told and disseminated through multiple modes of meaning. The demands of a generation of audiences that are multiliterate digital natives also require that storytelling is multimodal, engaging, interactive, immersive and personalised. Transmedia storytelling is the creation of story worlds through the use of multiple media platforms that provide participatory, immersive, interactive and multifaceted storytelling experiences for audiences. Transmedia storytelling has been used in the entertainment and media industry as an effective tool for advertising, marketing and branding. Increasingly, transmedia storytelling is being applied in the education sector as a pedagogical tool. This paper explores the pedagogical applications of transmedia storytelling in developing multimodality and creativity among tertiary students in Malaysia. The paper details the development of multimodal literacies through the conceptualisation and presentations of transmedia storytelling campaigns by tertiary students for a literature and media course. Through the conceptualisation of transmedia storytelling campaigns, tertiary students were found to have engaged with visual, spatial, linguistic, literary, audio, gestural and technical modes of meaning and have developed creative thinking skills. As research on pedagogical applications of transmedia storytelling is in its infancy in Malaysia, this paper paves the way towards utilizing transmedia storytelling as a pedagogical tool to develop multimodal literacies and increase knowledge and scholarship of transmedia storytelling in the Malaysian tertiary environment.

Keywords: Multimodality, transmedia storytelling, literature and media, higher education, Malaysia
Motivation in ESP Multimodal Practice

Arancha García Pinar

Politechnic University of Cartagena

[e-mail]

Multimodality has made educational institutions reappraise traditional aspects of the education environment and institutions that teach English for Specific Purposes (ESP) seem to increasingly include multimodal perspectives in their curricula. Multimodality research questions now are positioned alongside more traditional ESP research questions; “questions of language forms in monomodal frames’” (Prior 2013:520): How do students react when facing the whole interaction of modes? Does meaning conveyed through space, visuals and language simultaneously make understanding easier? It seems therefore pressing to research if a multimodal approach to ESP learning motivates students more than traditional approaches. This paper details the study that is being conducted in the Polytechnic University of Cartagena (UPCT), Spain and that explores L2 motivation of undergraduates that study Technical English following a multimodal pedagogy that mainly draws on the use of TED Talks. These crafted talks offered by leading experts on a wide range of fields, have unveiled as a set of guidelines for compelling communication. They incorporate a set of modes (language, gesture, body language, and visuals) that work together ‘in order to produce a greater meaning than either mode could on its own’ (Rowsell 2013:147). It is by addressing the different verbal and nonverbal modes that teachers can unveil how speakers at TED achieve compelling communication whilst being effective onstage. Students, in turn, if able to realize how different modes are orchestrated in these talks, might be encouraged to voice their ‘Ideas worth spreading’ in a Ted-style. The angle from which this study will research motivation of L2 students is Dörnyei’s L2 Motivational Self System (2009) which relies on the theory of possible selves (Markus and Nurius, 1986). This paradigm encourages a viewpoint of L2 motivation from the lens of learners as future L2 users.

Keywords: Multimodality, motivation, possible selves, verbal and non-verbal modes

REFERENCES

Dörnyei, Z. 2009. “The L2 Motivational Self System.” In Dörnyei, Z. and Ushioda, E. (Eds.), Language Identity and the L2 Self (pp. 9-42). Bristol: Multilingual Matters.

Markus, H. & Nurius, P. (1986). Possible Selves. American Psychologist, 41(9), 954-969

Prior, P. (2013). Multimodality and ESP Research. In B. Paltridge & S. Starfield (Eds.), The handbook of English for specific purposes (pp. 519-534). Malden, MA: Wiley-Blackwell.

Rowsell, J. (2013). Working with Multimodality. London: Routledge.
Employing affect as a semiotic resource in multimodal theorizing

Tiina Pitkäjärvi, PhD candidate

Uppsala University
[e-mail]

All meaning making is embodied (Kress, 2010: 77). The ‘affective turn’ (Koivunen, 2000) in the social sciences has for nearly two decades engaged in challenging the binary models of thinking that detach interpretation from sensation and mind from body. Despite multimodal endeavours within this area, ‘sensoriness’ is still “not a strong focus of most multimodal approaches” (MODE 2012) and affect often remains a missing piece in the puzzle in multimodal theorizing.

Some affect theorists (e.g. Massumi 2002) claim that the logic of affect does not translate into semantic models of explanation, while other scholars locate affect within discourse: Koivunen argues that analysing affective encounters means ”to examine media forms, representations and narratives, cultural framings and meaning-making processes” (Koivunen et al. 2000:7). Also Wetherell and Ehlers & Krupar demonstrate that affect indeed is operational, and not only constructed through representations, practices, and interpersonal relations, but also “socially and materially arranged; and, […] conditioned through sets of conventions” (Ehlers & Krupar 2014: 388).

In the presentation, I empirically draw on the Swedish breast cancer awareness and fundraising campaigns by examining the role of affect and how it can be incorporated into multimodal analysis. Informed by a social semiotic approach as well as different ‘turns to affect’, I will try to illustrate how affect can be studied as part of discourse, relying in particular on the concept of affective practice by Wetherell (2012).

Keywords: Affect, multimodality, social causes

References

Ehlers & Krupar 2014. Hope Logics: Biomedicine, Affective Conventions of Cancer, and the Governing of Biocitizenry, Configurations, Volume 22, Number 3, Fall 2014, pp. 385-413

Koivunen & Paasonen (eds), 2000. Conference proceedings for affective encounters: rethinking embodiment in feminist media studies, University of Turku.

Kress, 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. Routledge, London.

Massumi, 2002. Parables for the Virtual: Movement, Affect, Sensation.

MODE, 2012. Glossary of multimodal terms. 

Wetherell, 2012. Affect and Emotion: A New Social Science Understanding, SAGE Publications Ltd.

Multimodal teaching and learning in Greek History class as an alternative to traditional educational approaches

Dr. Georgios H. Potamias, Phd in the field of Pedagogics in Primary Education
University of Athens, Greece, Teacher
[e-mail]

Multimodal teaching and learning introduces the logical sequence and harmonious semiotic cooperation of multiple modes so that a comprehensive semiotic message is successfully given (Kress, 2003, 2010; Jewitt & Kress, 2003; Walsh, 2009). The Greek Educational system in general is considered to be strictly textbook oriented. This research focuses on the analysis of teaching History through the principles of multimodality in theory and practice in the fourth grade of the Greek Primary School as an alternative to traditional textbook oriented teaching approaches. The first part of the research analyzes the basic principles of multimodal literacy in History that are directly linked to education and can be applied in the classroom (Walsh 2009, 2011).

The first part also includes:

• Content analysis of 4 chapters of the official Greek History textbook of the fourth grade.

• Semiotic analysis of the same text.

• Multimodal analysis of the same content.

• Analysis of the embededness of the given educational content in the final layout of the History textbook.

The second part of the research includes:

• Observation and analysis of the educational process of 10 teachers attempting to apply the basic principles of multimodality in History class, using previous semiotically analyzed chapters of the History textbook.

• Descriptive and inductive analysis of the given data.

•Correlation of results.

From the analysis of the observations we conclude that the use of the textbook is predominant in the teaching of History in the fourth grade. We conclude that both male and female teachers find it difficult to smoothly integrate basic principles of multimodal teaching and learning in the educational process and to move away from traditional textbook oriented approaches. In the final part of the paper suggestions are given for the successful introduction and implementation of multimodal teaching in History class in praxis.

Keywords: Multimodality, history, education pedagogy, visual literacy, multimodal literacy, communication, teaching history, multimodal analysis, content analysis, multilevel analysis, embededness, effective communication, effective teaching

References

Jewitt, C. & Kress, G. (Eds.). (2003). Multimodal Literacy. New York: Peter Lang

Kress, G. (2003). Literacy in the New Media Age. London & New York: Routledge.
Kress, G. (2010). Multimodality. A Social Semiotic Approach to Comtemporary Communication. London & New York: Routledge.
Walsh, M. (2009). Pedagogic Potentials of Multimodal Literacy. In Tan Wee Hin, L., and Subramanian, R. (Eds.) (Eds.), Handbook of Research on New Media Literacy at the K~12 Level: Issues and Challenges (Vol. I and II, pp. 32–47), U.S.: IGI Global.
Walsh, M. (2011). Exploring classroom discourse: language in action. London & New York: Routledge.


Digital resemiotization of financial counselling as a social practice

Søren Vigild Poulsen

Department of Language and Communication, University of Southern Denmark
[e-mail]

Jette Ernst

Department of Language and Communication, University of Southern Denmark

Torben Worm

The Maersk Mc-Kinney Moller Institute, University of Southern Denmark

Since the financial crisis in 2008, the social practice of financial counselling has changed considerably. The very nature of counselling has been drastically transformed and the working conditions of banking professionals have become increasingly individualized. A consequence of these changes is a move from traditional pay structures to a system of individualized performance-based pay arrangements. One factor, which has both initiated and continues to push this process, is technological innovation. In addition, this factor threatens to deskill workers (Ellis and Taylor, 2010), for instance, with the increasing use of artificial intelligence (‘bots’) in the financial counselling profession.

The aim of this paper is to critically examine how financial counselling is resemiotized through new digital technologies. The digital solutions are used in various situations, such as person-to-person meetings or online usage. The study is part of an ongoing project that examines how customized IT-systems are built, designed and used and furthermore, how they shape social practices of the private financial sector.

The paper applies a new methodology that combines semiotic technology (Djonov & van Leeuwen, 2017) and software engineering (Sommerville 2016) to explore the technical and social-semiotic aspects of digital counselling services. Data are drawn from a pilot study of a major Danish bank, and the study combines data type analysis, semiotic analysis and ethnographic field studies.

The paper presents preliminary findings from this study, and then proceeds to discuss its implications. Specifically, the mapping out of how new features of counselling systems are designed, i.e. data mining techniques that generate a vast amount of information about customers. Subsequently, the paper also aims to investigate how these new digital features are perceived and used by managers and professionals as well as how physical and digital realities are integrated in the counselling situation.

The paper contributes with new knowledge and innovative methodology by presenting how the professional practices and identities of banking professionals, as well as the social relations to colleagues and customers, are affected by new digital tools.

References

Djonov, E. & Van Leeuwen, T. (2017). The power of semiotic software: A critical multimodal perspective. In J. Flowerdew, J. & J. Richardson (Eds.), The Routledge handbook of critical discourse analysis (pp. 566-581). London/New York: Routledge.

Sommerville, Ian (2016). Software engineering. 10 ed. Harlow Singapore: Pearson

Ellis, V. & Taylor, M. (2011). Banks, bailouts and bonuses: a personal account of working in Halifax Bank of Scotland during the financial crisis. Work, Employment and Society, Vol 24, Issue 4, pp. 803- 812.

Using student-selected precedent and guiding themes in landscape architectural meaning-making trajectories

Christine Price, PhD student

University of Cape Town, Cape Peninsula University of Technology

In social semiotics any meaning-making process, the selection of appropriate resources for a particular situation, by definition is considered ‘design’ (Kress, 2010). Moore’s explanation that landscape architectural design “is a snapshot of what we believe and value at a particular time in response to a particular problem and context” (Moore, 2010, p. 132), is not unlike meaning-making as described by social semiotics, with the clarification that landscape architectural design is an iterative and ongoing process. During the design process, the use of ‘guiding themes’ can assist a designer to make appropriate choices for each particular project. While guiding themes can be used to assist designers to select appropriate semiotic resources or to provide coherence to existing landscape spaces, the range of these semiotic resources can be described as ‘precedent’. Precedent is described as the repertoire or library of design ideas that landscape designers make use of when designing. Expert designers typically have a large range of precedent resources from which to draw on when designing (Björklund, 2013). While (particularly first year) students may not have a large range of landscape architectural precedents available, their experiences, skills, ideas, procedures, norms and materials are all semiotic resources that can be drawn on during the landscape design process. The larger study investigates the semiotic resources of diverse students learning to produce texts in, and move between, spatial, visual and verbal modes while learning to design a spatial model, in the context of the Diploma in Landscape Architecture at a South African Higher Education institution. This specific paper seeks to focus on how precedent and guiding themes are used by students to transform semiotic resources in an ongoing trajectory of a landscape architectural design project.

References

Björklund, T. A. (2013). Initial mental representations of design problems: Differences between experts and novices. Design Studies, 34(2), 135–160.

Kress, G. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Routledge.

Moore, K. (2010). Overlooking the visual: Demystifying the art of design. Abingdon: Routledge.

Assessment of multimodal academic numeracy practices

Robert Prince

University of Cape Town, South Africa
[e-mail]

The importance of the role of academic numeracy practices in higher education is increasingly being recognized. Many academic disciplines make significant demands on students’ academic numeracy practices, even in disciplines such as in Law and Humanities, which may not appear to be quantitative in nature. Quantitative disciplines, such as Engineering and Sciences, also make complex demands, for which traditional mathematics courses do not always prepare students adequately. Texts in quantitative disciplines are constructed using written language, images such as information graphics (charts, maps or diagrams), and mathematical notation. The primary function of information graphics is ‘to consolidate and display information graphically in an organized way so a viewer can readily retrieve the information and make specific and / or overall observations from it’ (Harris 1999:198).

This paper focuses on meaning-making across writing and information graphics in a multimodal task in a university entrance assessment in South Africa. This assessment attempts to develop an understanding of the extent to which prospective higher education students are prepared for the academic numeracy practices they face. This is important if teaching and learning is to be optimal. Prospective student responses to the task will be analysed. This analysis will examine the academic numeracy practices of the incoming students. The practices include drawing inferences, perceiving logical relationships, evaluating cause and effect, ordering and sequencing, and constructing hypotheses. This analysis will enable a better understanding of the extent to which these students’ academic numeracy practices are aligned with the academic numeracy practices expected in higher education. The findings will inform the way in which future university entrance assessment tasks could be developed, which is important in a developing country like South Africa where there is a large need to increase diversity of access into Higher Education in a context where there has been inequitable access in the past.

References

Harris, RL (1999) Information Graphics: A comprehensive illustrated reference, Oxford University Press, New York.

ENGLISH LANGUAGE TEACHING MEDIATED BY THE TEXTBOOK FROM A MULTIMODALITY APPROACH IN THE BRAZILIAN CONTEXT

Andressa Biancardi Puttin, Masters student

UFES, Brazil
[e-mail]

Záira Bomfante dos Santos, Doctor in Linguistics Studies (UFMG) and Professor UFES, Brazil

[e-mail]

This paper aims to discuss English language teaching in basic education in the Brazilian context by means of multimodality and the contribution of the textbook in this mediation process. Considering that multimodality is present in all the texts that circulate socially, from the written to the gestural language, it becomes elementary to observe how the potentials of significance can be explored and worked in the teaching and learning process. Therefore, we resort to the contributions of Kress (2010); Kress and Bezemer (2008) and Cope & Kalantzis (accessed on December 24,2017) to observe the semiotic landscape of the texts, more precisely the materialized texts in the textbooks, considering that the students, especially the ones from the public school, count with the textbooks that bring social practices of the language from a gamut of textual genres: comic strips, lyrics, online profiles, web pages, cartoons, letters, e-mails and others in which multimodality is present, allowing us to explore it by the written, sound, gestural and visual way. It is important to emphasize that the students are exposed to a series of literacy events outside of the school environment, varying their attitudes and actions that most of the time are daily to all of them, like: playing video game, reading e-mails, participating in chats in social networking, listening to music, checking online news, researching, etc. contributing, in this way, to the learning of the English language. In this context, the semiotic landscape of the textbook contributes to the agency of the students, involving them in the different textual genres and different styles of communication, so that they are able to use the English language in the different contexts of their daily life.

References

BEZEMER, Jeff; KRESS, Gunther. Writing in Multimodal Texts. (2008). A Social Semiotic Account of Designs for Learning. Written Communication, 25(2), 166-195. Avaliable in