Knowing with New Media

Gunther Cress (2010) points out that the understanding of multimodality as a semiotic resource lies in 'an attempt to bring all means of making meaning together under one theoretical roof'
(p. 5). Social semiotic systems such as script, images, gestures, music, animation, videos and so on are assembled together to compose a coded narrative. The need for this arises from ongoing changes in the web of social, economic and cultural communication catalysed by sweeping technological advancements.

Cinematic writing is a multimodal semiotic resource. It takes its roots from Michael Halliday's (1985 [2014]) theory of systemic functional linguistics (SFL), which views communication, and language in particular, as a resource for making meaning. Because language evolves to satisfy a society's need for communication, its functionality is not arbitrary but reflects society's underlying structures. In this regard, cinematic writing responds to the contemporary cultural need for multimodal communication by building on the most popular semiotic register — linguistic mode, written and spoken. Taken as the most established semiotic system, the linguistic mode becomes a foundational fabric into which non-linguistic modalities such as images, sounds and movements are interwoven.

Borrowing from SFL theory, cinematic writing acknowledges its two structural dimensions: syntagmatic order — systemic organisation of operational elements; and paradigmatic order — categories of elements that can be selected for composing meaning.

Translated into the language of new media, according to Lev Manovich's (2002) proposition, these two dimensions are database— existing, as well as generated, collections of multimodal elements and pre-designed operations offered by software; and narrative — approaches used in telling a story (making meaning) with database elements. Database narrative progression is formed with the human computer logic circularity – that is, an emerging process of representing meaning-making evolved from an interplay between psychological demands and capacities of the knower and the affordances of digital media.

Paradigmatic dimension – database: words, photos, videos, drawings, graphics, diagrams, audio elements, movements as well as pre-programmed software options for applying various effects.

Syntagmatic dimension – narrative: the way in which the database elements are organised into one meaningful composition.

Essential Ripples

Database Narrative

Human Computer logic

Representing Meaning-making

Tacit Explicit

'Knowledge is made and given shape in representation …'

Kress, 2010

From cave drawings to emojis: Communication comes full circle.

Danesi, Marcel (October, 2016). TEDxToronto [Youtube video].

Retrieved July, 2018, from: https://www.youtube.com/watch?v=0_QylCztffk&t=58sHayden, Scott (November, 2017). Semiotics 101: An Introduction to Semiotics [Prezi presentation]. Retrieved: 15 July, 2018, from: https://prezi.com/dhmmxrypzorq/semiotics-101/.

Semiotic systems serve different domains of knowledge by providing systems of signs through which meaning can be coded and conveyed. For example, in linguistics, the primary sign is an alphabetic letter. The letters are coded into words and the words into sentences, becoming an embodiment of meaning.

Michael Halliday

(1985)

Gunther Kress

(2010)

Lev Manovich

(2002)

Thomas A. Sebeok

(2001)

Cinematic Writing as a Semiotic Resource