Background: Interdisciplinary Research

My background is as a media artist working with film, video, photography and sound. Much of this work questioned linear structure as an organising principle. It sought ways by which the viewer of television, the audience for cinema, the visitor to an exhibition could participate, be made more responsible for, the making of meaning within the process of representation.

A film, ‘Red Plus Blue Plus Green’ (Mike Leggett 1975) was made with a 16mm industrial step-printer using a series of carefully constructed graphical images bi-packed with motion picture images from nature: red berries, blue sky and green grass such that over a period of 12 minutes, the primary colours were mixed onto unexposed duplicating film with “..variable loop length providing relative determinacy of the generative system….” (Stoneman 1979/80, p. 42). Whilst such a work could be enjoyed for the ‘abstract’ dynamic interplay of movement and colour, for the reflective viewer with some basic knowledge of the filmic process, the keys to the generative system were available through the feedback process of perceptual analysis.

Image Con Text (M. Leggett 1978) was a research project which began as a lecture performance utilising audio-visual material in a variety of formats, (16mm, 8mm, video, slide, audio cassette), before becoming distributable on U-matic video tape.(M. Leggett 1984, 1985b)

“The presentation is a site for the intersection of discourses which differ in origin, form, organisation and function. In their variety and totality they do not constitute an exemplary text or a composite work but rather a truncated description of a contestation, a confrontation indicating a series of power relations that take place in and through discourse.” (Stoneman 1979/80, p. 44)

Discourse was of course expandable when Image Con Text was presented to a live audience, the feedback to its form and its content palpable. Meaning was not given but developed out of the needs and priorities of the audience who had first engaged with the relations proposed: between on the one hand the means of presentation (signifier) and on the other the issues and polemics, (signified).

‘The Body on Three Floors’ (M. Leggett 1985a) was commissioned for television and working with an inter-disciplinary team, (an ethologist, a dancer, a clown, a playwright and an art historian), devised a narrative form in series of blocks that could be rearranged according to context. Clearly not an option at the time of broadcast but in the context of the general usage of video tape recorders, a tool which enabled the program to become distributed and thus used later in whatever way the owner of the tape determined.

Form and context, process and practice of this earlier work sought a dynamic means, both physically and polemically, by which systems of representation could be articulated.

By the mid-90s I had completed curating an exhibition, Burning the Interface<International Artists’ CD-ROM>, for the Museum of Contemporary Art in Sydney (M. Leggett 1996). This was based on some research I had started into the various ways visual artists around the world had begun to explore the creative possibilities of interactive multimedia, as it was then called. Based on this experience, the conclusions I had reached, the conversations with colleagues that followed, a production project commenced with the working title of ‘Strangers on the Land’ (SonTeL). With seed funding from the Australian Film Commission, this was the first prototype of an approach to what I now describe as mnemonic indexing. The successful outcomes of the research for both these projects completed the requirements for the award of Master of Fine Art at UNSW. (M. Leggett 2000)

Background: Literature

Through the 90s, I was able to access the means for being able to develop fresh approaches to the representation of what Norbert Wiener called ‘contingencies of relationality’ (Wiener 1997) and this was through the potential apparent via the microprocessor and the personal computer.

A deeper engagement with interactive media research commenced which eventually lead to a year long literature review for the PhD program in the Creativity and Cognition Studios in the Faculty of Information Technology at the University of Technology Sydney.

The literature review has been wide but has concentrated on interdisciplinary research into the closely related areas of mind and memory, (Heidegger c1977; Sutton 2004; Yates 1966), perception and cognition, (Clark 1997; Gibson 1979), presence and embodiment, (Dourish 2001; Mantovani 1999) ‘personalised representation’ (D Ballard 1991), creativity and ‘meta-design’ (Fischer 2003). This review will provide a context for the proposed research that will be evaluating an HCI-related approach to investigating machine memory and audio-visual digital media, employing practice-based research (Candy & Edmonds 2002; Scrivener & Chapman 2005 (press)) and I will begin by addressing some of these interdisciplinary topics.

“Memory is a label for a diverse set of cognitive capacities by which humans and perhaps other animals retain information and reconstruct past experiences, usually for present purposes.” (Sutton 2004)

Technologies, from notebooks to computers to language itself are good examples of what Andy Clark has described as:

“…the pervasive tendency of human agents to actively structure their environments in ways that will reduce subsequent computational loads.” (Clark 1997, p. 150)

Interacting with external memory machines such as collections and libraries of knowledge located on computer servers around the globe are central to academic pursuit and increasingly, the education and edutainment of the population. The index has been central to retrieval of text-based data. Complex indexes have become subject to “…classifying or arranging in classes, according to common characteristics or affinities;“ (OED)

My background reading this year has included acquiring some knowledge of existing machine memory systems that take a taxonomical approach to indexing based on text. Examples using this approach and related to visual appearance have been developed, for instance, by copyright protection agencies under a series of international agreements. International Classification of Products and Services for the Purposes of the Registration of [Trade]Marks, known as the Nice Agreement was one. Another, the Vienna Agreement Establishing an International Classification of the Figurative Elements of Marks facilitates the searching of design marks by permitting the classification of the figurative elements of the design. (Vienna).

Schneiderman notes in the late 90s that the machine memory industries specialising in servicing this demand by storing data and knowing how to retrieve it again, were moving away from notions of information retrieval and database management towards information gathering, seeking, filtering and visualisation. (Schneiderman 1998, pp. 510-511)

Other papers concerned with methods of improving storage and retrieval of audio-visual digital media have been concerned with machine vision systems and/or thesauri / text-based indexing. The AT&T Laboratories in Cambridge have used image segmentation and neural net classifiers (Town & Sinclair c.1998) to describe frame content (“classes of stuff”!) searchable by text or visual query. French Telcom commissioned work on Visual Information Retrieval (VIR) that used a similar semantic analysis based on the measurement of colour, texture and shape (Obeid, Jedynak & Daoudi 2001).

My current research is also informed by approaches pursued by ICT manufacturers who develop text / keyword-based multimedia file management applications (such as Extensis, Canto etc) for aided-retrieval. Hewlett Packard Labs in Palo Alto developed a prototype application for non-expert users – Fotofile – that “..blends human and automatic annotation methods.” The approach assumed that the ‘intuitive interface’ would be a text-based annotation system at worst and a thumbnail browsing system at best. In the final outcome it combined the two together with the addition of some ‘automatic’ (machine) features. A crude face recognition feature which offered users matched faces to confirm and name, used a ‘hyperbolic tree’ diagram visual device to link each face with its occurrence in other images. The paper provided no quantitative assessment though some compelling qualitative comments:

‘Photography and home movies are activities that address deep human needs; the need for creative expression; the need to preserve memories, the need to build personal relationships with others. Digital photography and digital video can provide powerful and novel, ways for people to express, preserve and connect. However, the new technologies often raise new problems; the problems of multimedia organisation and retrieval…’ (Kuchinsky et al. 1999)

These problems multiply as the uptake of domestic digital technologies continues to accelerate. Other ICT researchers, McDonald and Tait developed visual query tools. They noted that:

‘Search success was best when the task required the user to retrieve and image they had previously seen…’ and that ‘The good performance of subjects on this particular task suggests that colour-based visual searches might be a useful addition to personal image archive software…’ The searcher with ill-defined goals ‘.…can also be symptomatic of users not being able to formulate queries and therefore [having a] reliance upon browsing….’ (McDonald & Tait 2003)

Browsing, as the means by which users match an image to memory or a perceived need, has itself been aided by the work of Lim, Smith and Lu from Monash University, who “…designed i-Map, an interactive system for visualising and navigating a large scale image database…” that by clustering images onscreen (Content Based Image Retrieval), enabled the user to “…explore areas which look more promising…” before selecting an initial image which the system would then seek matches for before re-clustering. (Lim, Smith & Lu 2004)

Relational models of this kind were described by Ballard and Brown in the early 80s as turning away from representing models, to matching models from within a knowledge base. Thus proposition and inference became important aspects of interaction with the database. These approaches have become central to scientific, medical and surveillance sorting, storage and retrieval systems. (D. Ballard & Brown 1982, p. 9)

A decade later Ballard used the term “personalised representations” (D Ballard 1991) to describe the means we use to facilitate everyday behaviour. Correctly identifying our toothbrush in a bathroom shared by the household is an example I suggest: some residents may use colour differentiation whilst others, distrustful of their colour memory, prefer placing their toothbrush in a part of the bathroom different to the others. Clark has described this as action-orientated representations “…that simultaneously describe aspects of the world and prescribe possible actions, and are poised between pure control structures and passive representations of external reality.” (Clark 1997, p. 49).

The relational terms “more”, “same”, “less” are of interest in this context, (and were explored in the screen layout for i-Map). These same words were used by child development researchers (Griffiths, Shantz & Sigel c.1968) continuing to develop Jean Piaget’s Conservation Task, first defined in the 1940s. The OED defines ‘conservation’ in this context as a branch of psychology:

…faculty of conservation: memory proper, or the power of retaining knowledge, as distinguished from reproduction or reminiscence, the power of recalling it. (Author’s emphasis)

Derivation of the term can be traced back to Aristotle. He distinguished between Memory, as the faculty of Conservation, from Reminiscence, the faculty of Reproduction. Within the terms of the current investigation, this classical distinction will be retained, especially in relation to loci.

The ancient Greek system of Ars Memoria was described by Frances Yates in the 60s, “…a series of loci or places. The commonest, though no the only type of mnemonic place system was the architectural type….” (Yates 1966). Much of the work made using multimedia tools in the 90s developed this approach. Some researchers, whilst nominally connecting ‘place and knowledge’, take approaches that use the computer to link image symbols with specific narrative structures, replacing a book’s table of contents with the desktop e-book.(Kiriyama & Chen 2000). Others move the user into a virtual reality (VR) space with similar intent, where,

“Place motifs were embedded in the virtual environments as Placemarks – fragments of narrative … We wanted to populate the environments with archetypal Critters with which human participants could merge. The narrative goal here was to give people character materials to play with.” (Laurel & Strickland 1994, p. 124)

Brenda Laurel went on to conclude that:

‘Working on this piece has demonstrated to me that the art of designing in VR is really the art of creating spaces with qualities that call forth active imagination. The VR artist does not bathe the participant in content; she invites the participant to produce content by constructing meanings, to experience the pleasure of embodied imagination.’ (Laurel & Strickland 1994, p. 127).

These principles are good but qualities of presence and embodiment are not restricted to the virtual realm. The researcher seeking to establish a relationship between temporal media, external reality and the computational mapping of ‘aspects of the world’, its objects, its associations, can use far less complex approaches. These can offer a means of increasing our engagement with the world as embodied agents through, in this discussion, the extension of memory function and the actions that flow from it such as knowledge creation.

Background : Conclusion

Industry tools to aid the professional and the amateur multimedia producer exampled here have concentrated on management issues prior and during production. The model followed derives from the convergence of ICT with film and television production methods. Annotations are collected in the form of hand-written logs at the time of the origination of sound and picture material or during its first viewing. This becomes attached to the script of the production, whether a television show, an interactive disc or a website. The annotations are then often keyed into the file structure as meta-data and captured to the file management application.

Such methods are often cost-effective for the a team making linear product. But even for those productions which require some physical interaction, the audience is assumed to be more or less passive receptors of the ‘content’ of the production. Interactive options can follow part-linear structures (‘branching’) that complete an overall plan.

We are familiar with the ordering of images for cinema or television, the linear delivery of narrative which we are able to decode and interpret and thus acquire knowledge or edutainment. By contrast, structural acquisition happens, on the one hand as a highly focussed heuristic activity; encountering “..the anonymity of a definable structuring concept…” in an abstract artwork, (Le Grice 2001, p. 66) or comprehending the progress and outcomes of an orderly research enterprise. Or on the other hand a series of less focussed, parallel activities; reading the newspaper, whilst listening to the radio news and holding a conversation. The relational models I have referred to as part of this review, may be a useful way of creating knowledge by procedure, through a reconsideration of the ordering of visual media within a spatial mode of representation.