In some respects this is a rhetorical question because, as we shall find out, in the approach taken here, all communication is realised through texts and all texts are multimodal. To begin, given that one of the key aims of this thesis is to contribute to debates on the nature of multimodality in relation to media texts it is, at this point, necessary to clarify exactly what is meant by such terms as “multimodal” as well as its application to a range of so called “media texts” (Graddol and Boyd-Barrett, 1994; Burn and Parker, 2003). Although, both these terms and the accompanying property “multimodality” are discussed in detail and applied to some specific cases in the following chapter. A definition of multimodal texts and its application to forms of media text will be provided here, first, by way of making distinctions between the use of “text”, in general usage, and the way that “media texts” as a category relates to specific types of text and, one that has been adopted within the theoretical literature on communication and mass media to account for a wide range of “communicative artefacts” (Graddol, 1993, p. 41). Suffice it to say that multimodal media texts are specific forms of media text. Text, it shall be seen, will be defined beyond its most common usage, that of carrier or container for language , or in its material form. Text, in communications research has a much more nuanced meaning and relates to communication as a process and that way that communication events are realised:
Communication - whatever the mode - always happens as text. The 'stuff' of our communication needs to be fixed [...] in a mode: knowledge or information has no outward existence other than in such modal fixing. The fixing provides the material resource through which or in which it is to be materialised (Kress 2003, p. 47).
Text, in this context, is the vehicle or the physical and material substance through which communication happens and the important thing here is that communication, in the sense that Kress articulates it, would not happen without it.
This approach to texts has been developed from the work of Michael Halliday (1978) and his work on language as a “social semiotic” and the study of multimodal communication stems from this very specific brand of semiotics and in relation to the notion of "language as a social fact" (1978, p. 1). Since then, these ideas have been developed to account for a range of different modes and media. Hence, the fact that we now speak in terms of multimodal communication, texts, multimodality etc.
On the other hand, the term “visuality” stems from the notion of “vision as a social fact” (Foster, 1988, p. ix) will be used later in this chapter to indicate both the general shift in modes of communication; it is presented here by way of a specific critique of the way that visual literacy has been adopted to account for some of the consequences of these shifts. Visuality is also, perhaps more importantly, adopted within theoretical studies of the visual aspects of communication. But whilst it is used in the current chapter to assist in a critique of Visual Literacy, it is multimodality that is preferred throughout the thesis as a whole. The argument being developed in this chapter will justify this approach.
Multimodal communication refers to the fact that all communication by its very nature in multimodal in the sense that there is never only just one mode or system of representation at play in any communication. In conversation, for example, there is not just the sound of the spoken utterances that form the content of the entire message that is conveyed. Added to this are such things as gesture, intonation, eye contact and so on, which all provide important cues within the conversation. The same is true of forms of media. Something relative simple as a printed document still relies on the integration of material from many different modes, type, layout and composition etc. all of which are distinctive modes.
All media texts are by their very nature multimodal (Kress and Van Leeuwen, 1998, p. 186). They all rely on a variety of representational system for their realization. On one level this is quite an obvious thing to say about all communication, or all texts, Kaltenbacher ed. (2004, p. 190) yet this style of research is still relatively new. An important question would be along the lines of why so much research in the past has been ‘monomodal’, so to speak. Why has the study of language as a singular mode been for so long the main object of study? This we shall see will be an important debate in itself in the sections that follow In addition, Ventola et al (2004) provide, in their various papers, an overview of current trends within the growing area of ‘multimodal studies’. In his introduction to an edited collection of articles on approaches to multimodal research he begins with the basic premise that "the analysis of language alone is not enough" (Veltola et al, eds, 2004, p. 1) and that once the object of analysis is moved away from language a wide variety of objects of study become possible. Articles in Ventola’s book develop arguments about a range of artifacts from printed documents to film, video and other forms of electronic media. Striking, if only in virtue of the wide variety of approaches and in the array of possible objects of study, O’Halloran (2004), also provides a series of papers that analyse a wide range of texts types including: architecture, film, exhibition design, print advertisements, to name only a few; all analysed from the point of view of how meanings are realised through the combination of modalities. In the first instance, therefore, one of the most striking things about research of this kind is the wide variety of texts that can be analysed using the multimodal approach. Or, to put it another way, it is telling the breadth and variety of contexts which this notion of a text is able to perform once it becomes associated with the construction of meaning independent of modality, in fact text here is the means through which different modalities can combine to create meaning.
So, the first and most basic premise in a multimodal approach is that meanings are created through the combination of different codes, modes of representation, or meaning making systems, and as such that are brought together and combine to form integrated and coherent messages. A film, for example, brings together different ways of expressing and articulating messages – including, moving image, spoken word, soundtracks, music, voice-over etc. – and these occur across different sensory channels – vision, voice, listening and so forth. In this sense a communicative artefact such as a film is multimodal (Allen and Goodall, 2007, Rheindorf, 2004) and it is so because of the different modes that it employs in order to make meaning. A film is also a form of text in the sense that it has a material form, it is the surface or physical form in which communication, meanings and messages, are made concrete and therefore provide access to the meanings it conveys.
So, multimodality has become an important strand within recent semiotic analysis of media texts. In particular, the work of Kress and van Leeuwen’s work on visual design (1996, 1998), Michael O’Toole (1994) on painting, sculpture and architecture are both major influences on the case studies that follows in the next chapter and beyond. In addition, much of the recent research undertaken by Kress and van Leeuwen constitute attempts to explain, as well as to provide some empirical evidence for the shifts in communication towards the visual through their analysis of the visual design of texts and more generally how meaning is created through the visual channel. They have argued convincingly that “All texts are multimodal” (1996, p. 186). However, it will be seen that, even in spite of the innovative nature of their approach, as stated earlier, it is not without its problems and its contradictions and these are issues that are addressed directly in the section “Critique of Visual Literacy” that follows.
Having said this, their position is a major departure from language as being either the dominant or privileged mode in human communication and is, in the first instance, through their adoption of the concept of multimodality, an acknowledgement of the complexity and variety of human communication, both through interaction, say in conversation, and in the meaning of texts. The key point here being that there will never be just one single system of representation in operation at any one time. Kress and van Leeuwen (1998) continue by stating that:
Language always has to be realized through and comes in the company of other semiotic modes. When we speak, we articulate our message not just in words, but through a complex interplay of speech and sound, of rhythm, of intonation; accompanied by facial expression, gesture and posture. When we write, our message is expressed not only linguistically, but also through the visual arrangement of marks on the page. Any form of text analysis which ignores this will not be able to account for all the meaning expressed in texts (p. 186).
Thus, any communication relies on there being a variety of modalities through which messages are articulated and subsequently realized and this will be the case with all texts.
In terms of a methodology then, focusing on the multimodal properties of media texts, will afford certain lines of enquiry. In the first instance, one of the consequences of taking this kind of approach is that a wide range of media forms and artefacts can be analysed from a similar perspective. In some sense multimodal texts are just media texts seen from this more complex form of semiotics that see all forms of communication as multimodal.
In more general commonsense terms “text” is usually associated with written or printed communication, often to the exclusion of other forms of communication, or indeed, other modalities. Thus, when we talk about the text of an email message, the text of an Act of Parliament or the text of a newspaper story, there is a tacit assumption that the “text” is the material and visible form in which language or narrative is being expressed Gradoll, 1994, p. 41 – 42). More often than not, text is most often associated with the printed form of language, for example, the set text of a course of study ordinarily refers to a book and to the printed copy that it contains.
In multimodal approaches, therefore, the term “text” has a much more specific meaning. To illustrate this, let us take an example of a story in a newspaper. As a text, it can be analysed from the point of view that the messages being communicated are being conveyed through more than just one channel or mode. If the text is formed from the interplay of modalities, of systems of representation, or codes, as suggested earlier, then all of the elements that combine to make up the news story, including the headline, the body copy of the story itself, photographs, as well as any other graphical elements, will all combine to produce a complete, integrated and coherent messages. Even the style of presentation of the news story contributes to the composition and construction of the meanings communicated by the text. For instance, even the size, shape and orientation of the newspaper as a whole will make it distinct from others and define its market and style. The distinction between quality press and tabloids is first and for most articulated through the size and shape of its pages. Moreover, it is argued throughout the thesis, that space, in the form of compositional devices such as page layout and the spatial organisation of all elements within the text, contributes significantly to the way that messages and meaning are constructed and most importantly to the manner in which they are received and interpreted, in short, its framing.
In this context it can easily be argued that even a simple graphical element such as type, one of the most basic features that make up a newspaper story and, however simple it might appear as a graphical element in the design of a newspaper is not confined to a single modality. It is very much a part of the multimodal characteristics of the text. In the first instance, any form of type, as a graphical element, relies on both visual and verbal signs for its interpretation by readers. Type is a way of presenting a verbal message (language) but it does so in visual form (image) and its realization is entirely dependent the visual modality. In addition, type is also a specific form of layout element and embodies distinctively spatial attributes, meaning that it can also be analysed as a specific feature of the visual design of the page. Type occupies space and these spatial characteristics also add to its meaning. In this manner, therefore, type is used as a “semiotic resource” by designers (Kress (Fei, 2004, p. 51 – 56). This is also the case with the use of space which itself can be characterised as another distinctive modality. All typographical elements occupy a certain amount of space on the page and this forms a component within the composition of the page as a whole. The spatial dimension thus functions as a specific modality which is communicated visually and it forms another key aspect in the visual design of multimodal texts.
