Multimodality

## Definition

A multimodal text conveys meaning th​​rough a combination of two or more modes, for example, a poster conveys meaning through a combination of written language, still image, and spatial design. Each mode has its own specific task and function in the meaning making process, and usually carries only a part of the message in a multimodal text. In a picture book, the print and the image both contribute to the overall telling of the story but do so in different ways.

Images may simply illustrate or expand on the written story, or can be used to tell different aspects of the story, even contradicting the written words.

Effective multimodal authors creatively integrate modes in various configurations to coherently convey the meaning required, ‘moving the emphasis backwards and forwards between the various modes' throughout the text.

The complexity of the relationships between the various meanings or semiotic systems in a text increases proportionately with the number of modes involved. For example, a film text is a more complex multimodal text ​​than a poster as it dynamically combines the semiotic systems of moving image, audio, spoken language, written language, space, and gesture (acting) to convey meaning.

## Modes

The foll​​owing overview of how meaning can be composed through different semiotic resources for each mode (spoken language, written language, visual, audio, gestural, and spatial) is informed by The New London Group (2000), Cope and Kalantzis, (2009), and Kalantzis, Cope, Chan, and Dalley-Trim (2016). EAL/D learners engage with all of these meaning making practices through multicultural and/or multilingual lens.

### Written meaning

Conveyed through written language via handwriting, the printed page, and the screen. Choices of words, phrases, and sentences are organised through linguistic grammar convention​​​​s, register (where language is varied according to context), and genre (knowledge of how a text type is organised and staged to meet a specific purpose).

### Spoken meaning

Conveyed through spoken language via live or recorded speech and can be monologic or ​​dialogic. Choice of words, phrases, and sentences are organised through linguistic grammar conventions, register, and genre. Composing oral meaning includes choices around mood, emotion, emphasis, fluency, speed, volume, tempo, pitch, rhythm, pronunciation, intonation, and dialect.

### Visual meaning

Conveyed through choices of visual resources and includes both still image and moving images. Images may include diverse cultural connotations, symbolism and portray different people, cultures and practices. Visual resources include: framing, vectors, symbols, perspective, gaze, point of view, colour, texture, line, shape, casting, saliency, distance, angles, form, power, involvement/detachment, contrast, lighting, naturalistic/non-naturalistic, camera movement, and subject movement.

### Audio meaning

Conveyed through sound, including choices of music representing different cultures, ambient sounds, noises, alerts, sil​​ence, natural/unnatural sounds, and use of volume, beat, tempo, pitch, and rhythm. Lyrics in a song may also include multiple languages.

### Spacial meaning

Conveyed through design of spaces, using choices of spatial resources including: scale, proximity, boundaries, direction, layout, and organisation of objects in the space. Space ​extends from design of the page in a book, a page in a graphic novel or comic, a webpage on the screen, framing of shots in moving image, to the design of a room, architecture, streetscapes, and landscapes.

### Gestural meaning

Conveyed through choices of body movement; facial expression, eye movements​ and gaze, demeanour, gait, dance, acting, action sequences. It also includes use of rhythm, speed, stillness and angles, including ‘timing, frequency, ceremony and ritual.’ Gestures and body language may have diverse cultural connotations.

## Types of multimodal texts

Multimodality do​​es not necessarily mean use of technology, and multimodal texts can be paper-based, live, or digital.

* **Paper-​​based** multimodal texts include picture books, text books, graphic novels, comics, and posters.
* **Live** ​​multimodal texts, for example, dance, performance, and oral storytelling, convey meaning through combinations of various modes such as gestural, spatial, audio, and oral language.
* **Digital** multimodal texts include film, animation, slide shows, e-posters, digital stories, podcasts, and we​​b pages that may include hyperlinks to external pronunciation guides or translations.

## Source

https://www.education.vic.gov.au/school/teachers/teachingresources/discipline/english/literacy/multimodal/Pages/multimodaloverview.aspx