Multimodality

MULTIMODALITY is the human ability to combine several modes of world cognition and communication – verbal, visual, kinetic (with the help of gestures) and some others; a theory which looks at how people communicate and interact with each other, not just through writing, but also through speaking, gesture, gaze, and visual modes. Two types of multimodality – cognitive and verbal – are interconnected.

The notion of multimodality in linguistics originated from M. Halliday’s ideas, who highlighted the importance to overcome the “monomodality” of linguistic studies, which traditionally analysed only the verbal aspects of communication. The multimodal approach does not only consider the language as a set of grammatical rules, but focuses on the interaction of different sign systems.

The aim of this approach is to analyze and describe the whole range of means (e. g. visual, written mode, gestures, etc.) used by humans in different spheres, and to understand how these units are organized to create meanings. The central notion of this theory is mode. It does not have a universally accepted definition yet, but generally it is understood as a channel of representation or communication. A combination of a visual image with a verbal text, a phrase with a gesture, posture and mimics with a speech act, a video clip with a commentary may serve as examples of mode combination.

Another important notion in the theory of multimodality is modal affordance, related to practical and cultural aspects. In other words, modal affordance contributes to the decision about which mode to choose for this or that kind of representation.

Multimodal research of different types of discourse allows us to look at such traditionally linguistic notions as metaphor and discourse from a broader perspective.

According to the semiotic theory worked out by G. Kress and T. van Leeuwen, any kind of communication is multimodal [Kress, van Leeuwen 2001].

Further reading

Jewitt, C.(ed.) The Routledge Handbook of Multimodal Analysis. — London: Routledge, 2009.

Kress G., van Leeuwen T. Multimodal Discourse: The Modes and Media of Contemporary Communication. — L.: Arnold, 2001.

Kress G.Multimodality: a Social Semiotic Approach to Contemporary Communication. — London: Routledge, 2010.

Olga Iriskhanova

Translated by Ivanenko Ksenia