Knowing with New Media



Gunther Cress (2010) points out that the understanding of multimodality as a semiotic resource lies in 'an attempt to bring all means of making meaning together under one theoretical roof'
(p. 5). Social semiotic systems such as script, images, gestures, music, animation, videos and so on are assembled together to compose a coded narrative. The need for this arises from ongoing changes in the web of social, economic and cultural communication catalysed by sweeping technological advancements.

Cinematic writing is a multimodal semiotic resource. It takes its roots from Michael Halliday's (1985 [2014]) theory of systemic functional linguistics (SFL), which views communication, and language in particular, as a resource for making meaning. Because language evolves to satisfy a society's need for communication, its functionality is not arbitrary but reflects society's underlying structures. In this regard, cinematic writing responds to the contemporary cultural need for multimodal communication by building on the most popular semiotic register — linguistic mode, written and spoken. Taken as the most established semiotic system, the linguistic mode becomes a foundational fabric into which non-linguistic modalities such as images, sounds and movements are interwoven.

Borrowing from SFL theory, cinematic writing acknowledges its two structural dimensions: syntagmatic order — systemic organisation of operational elements; and paradigmatic order — categories of elements that can be selected for composing meaning.

Translated into the language of new media, according to Lev Manovich's (2002) proposition, these two dimensions are database— existing, as well as generated, collections of multimodal elements and pre-designed operations offered by software; and narrative — approaches used in telling a story (making meaning) with database elements. Database narrative progression is formed with the human computer logic circularity – that is, an emerging process of representing meaning-making evolved from an interplay between psychological demands and capacities of the knower and the affordances of digital media.

Paradigmatic dimension – database: words, photos, videos, drawings, graphics, diagrams, audio elements, movements as well as pre-programmed software options for applying various effects.

Syntagmatic dimension – narrative: the way in which the database elements are organised into one meaningful composition.

Essential Ripples

Database Narrative

Human Computer logic

Representing Meaning-making

Tacit Explicit


'Knowledge is made and given shape in representation …'

Kress, 2010

From cave drawings to emojis: Communication comes full circle.

Danesi, Marcel (October, 2016). TEDxToronto [Youtube video].

Retrieved July, 2018, from:, Scott (November, 2017). Semiotics 101: An Introduction to Semiotics [Prezi presentation]. Retrieved: 15 July, 2018, from:


Semiotic systems serve different domains of knowledge by providing systems of signs through which meaning can be coded and conveyed. For example, in linguistics, the primary sign is an alphabetic letter. The letters are coded into words and the words into sentences, becoming an embodiment of meaning.

Examples of linguistic, mathematical and chemical semiotic systems

A coded combination of linguistic signs that signifies a specific object



An object signified by the coded combination of linguistic signs



A combination of chemical signs that encodes the property of matter

A mathematical operation (addition) signified by a combination of mathematical signs

A mathematical code that signifies
a mathematical operation (addition)



Matter signified by a combination of chemical signs

Michael Halliday


Gunther Kress


Lev Manovich


Thomas A. Sebeok


Cinematic Writing as a Semiotic Resource

In cinematic writing, non-linguistic modes of expression – images, videos, sounds and movements – are integrated into the fabric of alphabetic writing to create 'a conversation' with each other.

Collected database elements are drawn into the dialogue for reasons that need to be discovered and articulated through the progression of 'this conversation'. The knower constructs 'the conversation' in his/her own unique way involving the circularity of tacit explicit knowledge.

The task of the knower turns into decoding what each element tries to say to the knower and the world around him/her. 'Rippling' within database narrative and tacit explicit loops, the knower makes connections, and maps the most optimal positioning of his/her personal world within the surrounding medium of existence.



All content and artwork (with the exception of the embedded video From cave drawings to emojis: Communication comes full circle) on this site is Copyright © 2018 Ripples Pedagogy PTY. LTD.