The Language of New Media

1. New Media Emerges from the Convergence of Computing and Media.

The synthesis of these two histories? The translation of all existing media into numerical data accessible for computers.

A historical merger. New media is not a sudden invention but the result of two distinct historical trajectories converging: the development of computing machines (starting with Babbage's Analytical Engine in the 1830s) and the development of modern media technologies (like Daguerre's daguerreotype from the same era). Both were essential for modern mass societies, enabling data processing and mass communication, respectively.

From calculation to culture. Initially separate, these paths merged when media was translated into numerical data, making it computable. The computer, originally a calculator, became a universal media machine capable of processing, storing, distributing, and accessing text, images, sounds, and spatial constructions. This transformation is arguably more profound than the printing press or photography revolutions, affecting all stages and types of media communication.

Jacquard's loom reborn. The connection between computing and media was present from the start, as Babbage was inspired by Jacquard's loom, a programmed machine for weaving images. This early link foreshadowed the computer's return to its origins as a media synthesizer and manipulator, no longer just an analytical engine for numbers but a powerful tool for cultural production and access.

2. Five Core Principles Define the Language of New Media.

This list reduces all principles of new media to five: numerical representation, modularity, automation, variability and cultural transcoding.

The foundation of computability. Numerical representation means all new media is digital code, making it mathematically describable and subject to algorithmic manipulation. This turns media into programmable data, allowing for automated processes like noise reduction or image enhancement.

Building blocks and flexibility. Modularity, or the "fractal structure," means new media objects are composed of discrete samples (pixels, polygons) that retain their identity when assembled into larger objects. This allows for easy modification, substitution, and non-linear organization, unlike the fixed nature of traditional media copies.

Efficiency and transformation. Automation leverages numerical coding and modularity to remove human intentionality from parts of the creative process, from simple filters to complex AI-driven content generation. Variability allows new media objects to exist in potentially infinite versions, customized automatically based on user information or other parameters, reflecting the post-industrial logic of individual customization. Finally, transcoding describes the most significant consequence: the influence of the computer's organizational logic (data structures, algorithms, interfaces) on the cultural layer of media, blending human and computer meanings.

3. New Media Interfaces Blend Traditions of Print, Cinema, and HCI.

In short, we are no longer interfacing to a computer but to culture encoded in digital form.

Culture as interface. As computers become universal media machines, we increasingly interact with cultural data (texts, images, films) through interfaces. These "cultural interfaces" are not neutral but shape how we perceive and interact with digital culture, imposing their own logic and models of the world.

A hybrid language. Cultural interfaces draw heavily on familiar cultural forms:

• Printed Word: Conventions like the rectangular page, sequential organization, table of contents, and index influence layouts and navigation (e.g., Web pages, CD-ROMs). Hyperlinking, while seemingly new, can be seen as a radical departure from traditional text organization, favoring metonymy and non-hierarchy.
• Cinema: Cinematic ways of seeing, structuring time, and narrating stories are extended to computer users. The mobile camera becomes an interface convention for navigating 3D data, rectangular framing persists, and cinematic techniques are encoded in software and hardware.
• Human-Computer Interface (HCI): Principles like direct manipulation, windows, menus, and icons provide the basic grammar of interaction, blending with conventions from older media.

Navigating competing logics. Cultural interfaces often negotiate between the richness of control in general HCI and the immersive experience of traditional media. This creates tensions, such as hiding hyperlinks within continuous images (illusion vs. control) or balancing standardization (HCI) with originality (traditional art). The result is a hybrid language still in its early stages, reflecting the ongoing process of computerizing culture.

4. The Computer Screen Evolves from Static Window to Interactive Control Panel.

This is the third, after classic and dynamic, type of a screen — the screen of real time.

A long history of framing. The screen, a flat rectangular surface framing a virtual world, has a history stretching back to Renaissance painting ("classical screen"). Cinema introduced the "dynamic screen," displaying images changing over time, fostering viewer identification and immersion.

Real-time and interactive. The computer screen represents a new stage: the "real-time screen," capable of continuous updates reflecting changing data. Crucially, it also becomes interactive, allowing users to give commands and directly affect reality through the display. This shift originated in military surveillance (radar, SAGE system) where the screen evolved from a passive display to an active control panel.

Beyond the frame. Modern computer interfaces challenge the traditional screen in two ways:

• Window Interface: Displays multiple coexisting windows, breaking the dominance of a single image and aligning more with graphic design layouts than cinematic screens.
• Virtual Reality (VR): Eliminates the screen altogether, completely immersing the user in the virtual space and requiring physical movement to navigate, though often tethering the body to a machine.

From passive viewing to active control. The computer screen embodies a fundamental shift from the viewing regime of the dynamic screen era (identification, immersion) to one of interaction and control. It functions simultaneously as a window into an illusionistic space and a virtual instrument panel, reflecting its origins in military command and control systems.

5. New Media Operations Reflect and Reshape Cultural Practices.

They are not only ways of working with computer data but also general ways of working, ways of thinking, and ways of existing in a computer age.

Software as cultural filter. Beyond the interface, application software provides a layer of operations that shape how we create and interact with new media. These operations, like copy, cut, paste, search, filter, and composite, are not media-specific but apply across different data types due to media's status as computer data.

Operations as cultural logic. These operations are embedded in software but extend into the social world. They become cognitive strategies and ways of existing in a computer age. The communication is two-way: software design reflects social logic, and using software shapes our understanding of ourselves and the world.

Examples of key operations:

• Selection: Choosing from pre-defined menus or libraries (3D models, filters, templates). This reflects a cultural shift towards assembling from ready-made parts, seen in modern society's consumption patterns and post-modern art's reliance on quotation and pastiche.
• Compositing: Assembling elements from different sources into a seamless whole. This is central to digital image and video creation and reflects a desire for continuity, contrasting with modernist montage.
• Teleaction: Acting over distance in real-time via representations (e.g., telepresence, remote control). This builds on older image-instruments (maps, x-rays) but adds instantaneous feedback and real-time manipulation, enabled by electronic telecommunication.

These operations, encoded in algorithms and materialized in software, exemplify the transcoding principle, where computer logic influences cultural practices and vice versa.

6. Digital Compositing Transforms Moving Images from Montage to Seamlessness.

Most often the moving image constructed through compositing simulates a traditional film shot.

Assembling virtual reality. Digital compositing, combining multiple image sequences and stills into a single sequence, is a core operation in new media production, particularly for moving images. It allows for the construction of seamless 3D virtual spaces from diverse elements, simulating camera moves and media artifacts.

Beyond traditional montage. While montage (temporal juxtaposition) was cinema's key technique for creating fake realities, digital compositing (spatial blending) offers an alternative aesthetics of continuity. Elements are blended, boundaries erased, creating a single gestalt, unlike the deliberate dissonance of montage.

New dimensions of control. Compositing, especially in software interfaces, highlights new spatial dimensions of the moving image alongside time:

• Spatial order of layers (2.5D space).
• Virtual space constructed (3D space).
• Movement of layers within the frame (2D space).
• Relationship between image and linked information (2D space).

This shifts the focus from temporal editing to spatial organization and blending. While often used to achieve photorealistic simulation, compositing can also create new forms of "spatial montage," "ontological montage" (co-existence of incompatible elements), or "stylistic montage" (juxtaposition of different media styles), as seen in works by Rybczynski, Zeman, and Tobreluts.

7. Synthetic Realism Simulates Photographic Vision, Not Human Perception.

Synthetic computer-generated image is not an inferior representation of our reality, but a realistic representation of a different reality.

Faking the photograph. The primary goal of 3D computer graphics research has been photorealism – creating images indistinguishable from photographs. This is not the same as simulating human perception or reality itself, but rather simulating the specific artifacts and limitations of the photographic lens and film (depth of field, grain, tonal range).

Too perfect, too real. Paradoxically, synthetic images are often "too perfect" before being degraded to match film's imperfections. They are free from lens artifacts and noise, possessing unlimited resolution and detail. This hyperreal quality suggests they are not inferior representations of our reality but realistic representations of a different reality – the vision of a computer, a cyborg, a digital grid.

Messengers from the future. Unlike traditional photographs that point to the past, synthetic images point to the future. They depict a reality yet to come, one potentially reduced to geometry and efficient representation. Films like Jurassic Park and Terminator 2 exemplify this, with dinosaurs masking the future vision or cyborgs embodying it directly, often degrading the perfect computer image to blend with familiar film or using its starkness to signify the alien future.

This pursuit of photorealism, driven by military and entertainment needs, results in an uneven realism, excelling in simulating specific phenomena (landscapes, figures) while struggling with others, reflecting the priorities of its sponsors and the cultural icons of mimesis.

8. Database and Narrative Compete as Dominant Forms in Computer Culture.

Therefore, database and narrative are natural enemies.

The rise of the collection. In the computer age, the database emerges as a key cultural form, challenging narrative's dominance. Many new media objects are collections of items without a fixed beginning, end, or inherent sequence, reflecting the world as an unstructured collection of data records.

Narrative as a database trajectory. While some objects are explicitly databases (encyclopedias, Web sites), most are implicitly so. Creating new media often involves constructing an interface to a multimedia database. From this perspective, a narrative can be seen as just one possible trajectory or path through a database, a specific ordering of elements from a larger collection.

Privileging the paradigm. New media reverses the traditional relationship between narrative (syntagm) and the underlying set of possibilities (paradigm). The database (paradigm) is given material existence and becomes the center of the creative process, while the narrative (syntagm) is de-materialized into a set of links or a virtual trajectory. Interactive interfaces further foreground the paradigm by presenting users with explicit menus of choices.

Despite the inherent database logic of the medium, the desire for narrative persists, leading to attempts to merge database and narrative into new forms, such as interactive narratives or database-driven films, exploring how narrative can be constructed from and interact with the underlying database structure.

9. Navigable Space Becomes a New Media Type and Cultural Form.

For the first time, space becomes a media type.

Traversing the virtual. Navigable space, where users move through simulated 3D environments, is another key form in new media, prominent in games, virtual worlds, simulators, and data visualization. It spatializes data and experience, equating narrative and time with movement through rooms, levels, or abstract landscapes.

Beyond static representation. Unlike traditional representations of space (maps, paintings, architecture), computer space is navigable. It can be instantly transmitted, stored, retrieved, and manipulated like other media types. This makes it a fundamentally new kind of spatial representation.

Aggregate, not systematic. Despite the underlying Cartesian coordinate system, computer spaces are often aggregate collections of objects rather than coherent, continuous environments. Techniques like polygonal modeling and superimposing sprites over backgrounds create worlds of separate entities, lacking the "space-medium" quality found in some modern painting or traditional animation. This reflects the modularity of computer data and potentially cultural tendencies towards fragmentation.

From flâneur to data cowboy. The experience of navigating virtual space connects to historical figures like Baudelaire's flâneur and the American explorer. The virtual flâneur traverses data landscapes, while the virtual explorer navigates game worlds, building character through spatial movement and exploration. This form, rooted in military simulators and extending into civilian applications, reflects a cultural shift towards spatialized data access and interaction.

10. Digital Cinema Redefines Film as a Subset of Animation.

Digital cinema is a particular case of animation which uses live action footage as one of its many elements.

Crisis of indexicality. Traditional cinema's identity was rooted in its ability to record physical reality using lens-based photography – it was an art of the index. Digital technology challenges this by allowing photorealistic scenes to be generated entirely in a computer or extensively manipulated after digitization.

Manual construction returns. Digital filmmaking relies on techniques previously marginalized and delegated to animation:

• Generating scenes directly in a computer (3D animation).
• Treating live action footage as raw material for manipulation (painting, processing, compositing).
• Constructing final images manually from diverse elements.
• Collapsing distinctions between editing and special effects.

From kino-eye to kino-brush. The filmmaking process shifts from automatically recording reality to manually constructing images, frame by frame if necessary. Live action footage loses its privileged status and becomes just another element alongside computer-generated graphics. This makes digital cinema fundamentally a form of animation, a "painting in time," where the artist works with a "kino-brush" rather than a "kino-eye."

This transformation brings techniques previously used by the avant-garde (collage, painting on film) into the mainstream, embedding them in software design and challenging the classical realist style of cinema.

11. New Media Language Finds Inspiration in Pre-Cinematic and Avant-Garde Forms.

And this is why Vertov’s film has a particular relevance to new media.

Revisiting abandoned paths. To develop a new aesthetics for new media, designers can draw inspiration from cultural history, particularly strategies and techniques abandoned or marginalized by traditional cinema. This includes pre-cinematic practices and avant-garde experiments that worked against the dominant norms of film.

The power of the loop. Nineteenth-century moving image devices were based on loops, a form later relegated to animation and low-art. New media, facing similar storage and bandwidth constraints, often uses loops (QuickTime, games, virtual worlds). Works like Akvaario and Flora petrinsularis demonstrate how the loop can function as a new narrative engine, bridging linear progression and interactive control, and offering a new temporal aesthetics for computer-based cinema.

Spatial montage reborn. Traditional cinema privileged temporal montage (sequential shots), marginalizing spatial montage (simultaneous images). Computer displays with multiple windows and the modular nature of digital images make spatial montage feasible again. Works like My boyfriend came back from war! explore this, creating narratives through the juxtaposition and interaction of multiple images on screen, reviving a form prominent in pre-cinematic visual culture and explored by avant-garde filmmakers.

By looking back at these earlier forms and techniques, new media artists can find alternatives to simply replicating traditional cinematic language and explore the unique aesthetic possibilities offered by the computer medium.

Last updated: May 1, 2025