Media streams:
an iconic visual language for video annotation

By Marc Davis

Abstract

Our prototype system, Media Streams, enables users to create multi-layered, iconic annotations of video content. Within Media Streams, the organisation and categories of the Director's Workshop allow users to browse and compound over 2500 iconic primitives by means of a cascading hierarchical structure which supports compounding icons across branches of the hierarchy. Icon Palettes enable users to group related sets of iconic descriptors, use these descriptors to annotate video content, and reuse descriptive effort. Media Time Lines enable users to visualise and browse the structure of video content and its annotations. The problems of creating a representation of action for video are given special attention, as well as describing transitions in video.

1 Introduction: the need for video annotation

	1.1 Video annotation today

	1.2 Video annotation tomorrow

2 Design criteria for video annotation languages

3 Representing video

	3.1 Streams vs. clips

	3.2 Categories for media annotation

	3.3 Video syntax and semantics

4 Media Streams:  an overview

5 Why icons?

6 Director's Workshop

	6.1 A language for action

	6.2 Character actions and object actions

	6.3 Characters and objects

	6.4 Relative positions

	6.5 Mise-en-scene: time, space, and weather

	6.6 Cinematography

	6.7 Recording medium

	6.8 Screen positions

	6.9 Thoughts

	6.10 Transitions

	6.11 Extensibility of the icon language

7 Media Time Lines

8 Conclusions and future work

Acknowledgements

References

1 Introduction: the need for video annotation

Given the current state of the art in machine vision and image processing, we cannot now, and probably will not be able to for a long time, have machines 'watch' and understand the content of digital video archives for us. Unlike text, for which we have developed sophisticated parsing technologies, and which is accessible to processing in various structured forms (ASCII, RTF, PostScript), video is still largely opaque. We are currently able to automatically analyse scene breaks, pauses in the audio, and camera pans and zooms (41, 21, 31, 33, 34, 38, 39), yet this information alone does not enable the creation of a sufficiently detailed representation of video content to support content-based retrieval and repurposing.

In the near term, it is computer-supported human annotation that will enable video to become a rich, structured data type. At this juncture, the key challenge is to develop solutions for people who already devote time and money to annotating video, because they will help create the necessary infrastructure (both economically and in terms of the content itself) to support the ubiquitous use and reuse of video information. Today, simple queries often take tens of hours and cost thousands of dollars. If recorded reusable video is going to become a ubiquitous medium of daily communication, we will need to develop technologies which will change the current economics of annotation and retrieval.

1.1 Video annotation today

At least, Pat should be able to use Pat's annotations.

Slightly better, Chris should be able to use Pat's annotations.

Even better, Chris' computer should be able to use Pat's annotations.

At best, Chris' computer and Chris should be able to use Pat's and Pat's computer's annotations.

In the main, video has been archived and retrieved as if it were a non-temporal data type which could be adequately represented by keywords. A good example of this approach can be seen in Apple Computer's Visual almanac which describes and accesses the contents of its video and image archive by use of 'keywords' and 'image keys' (4). This technique is successful in retrieving matches in a fairly underspecified search but lacks the level of granularity and descriptive richness necessary for computer-assisted and automatic video retrieval and repurposing. The keyword approach is inadequate for representing video content for the following reasons:

• Keywords do not describe the complex temporal structure of video and audio information.

• Keywords are not a semantic representation. They do not support inheritance, similarity, or inference between descriptors. Looking for shots of 'dogs' will not retrieve shots indexed as 'German shepherds' and vice versa.

• Keywords do not describe relations between descriptions. A search using the keywords 'man,' 'dog,' and 'bite' may retrieve 'dog bites man' videos as well as 'man bites dog' videos - the relations between the descriptions highly determine salience and are not represented by keyword descriptions alone.

• Keywords do not scale. As the number of keywords grows, the possibility of matching the query to the annotation diminishes. As the size of the keyword vocabulary increases, the precision and recall of searches decrease.

Today, in organisations and companies around the world whose business it is to annotate, archive, and retrieve video information, by and large, the structure of the data is mostly represented in the memories of the human beings whose job it is to handle it. Even in situations in which keyword-based computer annotation systems are 'used,' short-term memory and long-term memory are the real repositories of information about the content of video data. 'Joe and Jane in the basement' are the real indexing and retrieval mechanisms in almost all video archives. Human memory is very good at retrieving video due to its associative and analogical capabilities; it has memory structures which any computerised retrieval system would want to emulate. Nevertheless, there are significant problems in sharing the contents of one human memory with others and of transferring the contents of one human memory to another. There are also severe limitations in terms of storage capacity and speed for human memory that aren't acceptable if we are going to scale up to a global media archive in which video is accessed and manipulated by millions of people everyday.

We need to create a language for the representation of video content which enables us to combine automatic, semi-automatic, and human annotation so as to be able to make use of today's annotation effort long into the future.

1.2 Video annotation tomorrow

By having a structured representation of video content - meaningful bits about the bits - future annotation and retrieval technology will enable users to mix video streams according to their contents and to manipulate video at various levels of granularity. With this kind of representation, annotation, and retrieval technology we will create tools which enable users to operate on higher level content structures of video data instead of being stuck with just bits, pixels, frames, or clips.

2 Design criteria for video annotation languages

A video annotation language needs to create representations that are durable and sharable. The knowledge encoded in the annotation language needs to extend in time longer than one person's memory or even a collective memory, and needs to extend in space across continents and cultures. Today, and increasingly, content providers have global reach. German news teams may shoot footage in Brazil for South Korean television which is then accessed by American documentary film makers, perhaps ten years later. We need a global media archiving system that can be added to and accessed by people who do not share a common language, and the knowledge of whose contents is not only housed in the memories of a few people working in the basements of news reporting and film production facilities. Visual languages may enable the design of an annotation language with which we can create a truly global media resource. Unlike other visual languages that are used internationally (e.g., for traffic signage, operating instructions on machines, etc. (18)) a visual language for video annotation can take advantage of the affordances of the computer medium. We can develop visual languages for video that utilise colour, animation, variable resolution, and sound in order to create durable and sharable representations of video content.

3 Representing video

3.1 Streams vs. clips

Stream of 100 frames of video

Traditionally, one or more parts of the stream of frames would be segmented into clips which would then be annotated by attaching descriptors. The clip is a fixed segmentation of the video stream that separates the video from its context of origin and encodes a particular chunking of the original data.

A 'clip' from frame 47 to frame 68 with descriptors

In our representation, the stream of frames is left intact and is annotated by multi-layered annotations with precise time indexes (beginning and ending points in the video stream). Annotations could be made within any of the various categories for media annotation discussed below (e.g., characters, spatial location, camera motion, dialogue, etc.) or contain any data the user may wish. The result is that this representation makes annotation pay off - the richer the annotation, the more numerous the possible segmentations of the video stream. Clips change from being fixed segmentations of the video stream, to being the results of retrieval queries based on annotations of the video stream. In short, in addressing the challenges of representing video for large archives what we need are representations which make clips, not representations of clips.

The stream of 100 frames of video with 6 annotations resulting in 66 possible segmentations of the stream (i.e., 'clips')

3.2 Categories for media annotation

These categories - actions, characters, objects, locations, times, and weather - would be nearly sufficient for talking about actions in the world, but video is a recording of actions in the world by a camera, and any representation of video content must address further specific properties. First, we need ways of talking about cinematographic properties, the movement and framing of the camera recording events in the world. We also need to describe the properties of the recording medium itself (film or video, colour or black & white, graininess, etc.). Furthermore, in video, viewers see events depicted on screens, and therefore, in addition to relative positions in space, screen objects have a position in the two-dimensional grid of the frame and in the various layered vertical planes of the screen depth. Finally, video recordings of events can be manipulated as objects and rearranged. We create transitions in video in ways not possible in the real world. Therefore, cinematic transitions must also be represented in an annotation language for video content.

These categories need not be sufficient for media annotation (the range of potential things one can say is unbounded), but we believe they are necessary categories for media annotation in order to support retrieval and reuse of particular segments of video data from an annotated stream.

These minimal annotation categories attempt to represent information about media content that can function as a substrate:

• on top of which other annotations may be layered

• out of which new annotations may be inferred

• within which the differences between consensual and idiosyncratic annotations may be articulated.

3.3 Video syntax and semantics

First of all, it is important to realise that video images have very little intrinsic semantics. Syntax is highly determinative of their semantics, as evidenced by the Kuleshov Effect (30). The Kuleshov Effect is named after Lev Kuleshov, a Soviet cinematographer whose work at the beginning of the century deeply influenced the Soviet montage school and all later Soviet cinema (19, 20). Kuleshov was himself an engineer who, after only having worked on one film, ended up heading the Soviet film school after the October Revolution. Kuleshov was fascinated by the ability of cinema to create artificial spaces and objects through montage (editing) by virtue of the associations people create when viewing sequences of shots, which if the shots were taken out of sequence would not be created. In the classic Kuleshov example, Kuleshov showed the following sequence to an audience:

the passive face of an actor - a bowl of soup - go to black

the same face of the actor - a coffin - go to black

the same face of the actor - a field of flowers - go to black.

The syntax of video sequences determines the semantics of video data to such a degree that any attempts to create context-free semantic annotations for video must be carefully scrutinised so as to determine which components are context-dependent and which preserve their basic semantics through recombination and repurposing. Any indexing or representational scheme for the content of video information needs to be able to facilitate our understanding of how the semantics of video changes when it is resequenced into new syntactic structures. Therefore, the challenge is twofold: to develop a representation of those salient features of video which, when combined syntactically, create new meanings; and to represent those features which do not radically change when recontextualised.

4 Media Streams: an overview

The system has three main interface components: the Director's Workshop (see figure 1); icon palettes (see figure 2); and media time lines (see figure 3). The process of annotating video in Media Streams using these components involves a few simple steps:

1) In the Director's Workshop, the user creates iconic descriptors by cascading down hierarchies of icons in order to select or compound iconic primitives.

2) As the user creates iconic descriptors, they accumulate on one or more icon palettes. This process effectively groups related iconic descriptors. The user builds up icon palettes for various types of default scenes in which iconic descriptors are likely to co-occur, for example, an icon palette for 'treaty signings' would contain icons for certain dignitaries, a treaty, journalists, the action of writing, a stateroom, etc.

3) By dragging iconic descriptors from icon palettes and dropping them onto a media time line, the user annotates the temporal media represented in the media time line. Once dropped onto a media time line, an iconic description extends from its insertion point in the video stream to either a scene break or the end of the video stream. In addition to dropping individual icons onto the media time line, the user can construct compound icon sentences by dropping certain 'glommable' icons onto the Media Time Line, which, when completed, are then added to the relevant Icon Palette and may themselves be used as primitives. For example, the user initially builds up the compound icon sentence for 'Arnold, an adult male, wears a jacket' by successively dropping the icons , and onto the media time line. The user then has the compund icon on an icon palette to use in later annotation. By annotating various aspects of the video stream (time, space, characters, character actions, camera motions, etc.), the user constructs a multi-layered, temporally indexed representation of video content.

5 Why icons?

• quick recognition and browsing of content annotations

• visualisation of the dense, multi-layered temporal structure of video content

• an accurate and readable time-indexed representation of simultaneous, sequential, overlapping and contained actions (natural languages are not very good at this task)

• articulation of the boundaries between consensual and idiosyncratic annotations (icons can have attached textual annotations and can thus function as the explicit consensual tokens of various idiosyncratic textual descriptions)

• global, international use of annotations

• visual similarities between instances or subclasses of a class (visual resonances in the iconic language).

The iconic language gains expressive power and range from the compounding of primitives and has set grammars of combination for various categories of icons. In Korfhage's sense Media Streams is an iconic language as opposed to being merely an iconography (28). Similar to other syntaxes for iconic sentences (13, 35) icon sentences for actions have the form of subject-action, subject-action-object, or subject-action-direction, while those for relative positions have the form subject-relative position-object. Icon sentences for screen positions are of the form subject-screen position, while cinematographic properties are of the form camera-movement-object (analogous to subject-action-object), as in 'the camera-is tracking-Steve' or 'the camera-zooms in on-Sally.'

6 Director's Workshop

• Cascading hierarchy with increasing specificity of primitives on subordinate levels

Cascading icons are organised in hierarchies from levels of generality to increasing levels of specificity. Similar to cascading menus on the Macintosh, when a user cascades down an icon hierarchy by clicking on a cascading icon, its subordinate icons are displayed to the right of the cascading icon. These subordinate icons are arranged horizontally and represent an increased level of specificity. Some of the icon hierarchies cascade to as many as 7 or 8 levels deep, yet, similarly to the semantic hierarchies of the CYC Project (29), the design of the categories themselves and their first three or four levels is the hardest and most important representational task.

• Compounding of hierarchically organised primitives across multiple axes of description

In many icon hierarchies on the Director's Workshop, there exists an additional form of organisation. When subordinate icons are arranged vertically, they represent independent axes of description whose icon hierarchies can be cascaded through separately and whose respective subordinate icons can be compounded together across these axes to form compound iconic descriptors. This form of organisation enables a relatively small set of primitives to be compounded into a very large and rich set of descriptors.

It is also important to note that the icon hierarchy of the Director's Workshop is structured not as a tree, but as a graph. The same iconic primitives can often be reached by multiple paths. The system encodes the paths users take to get to these primitives; this enriches the representation of the compounds which are constructed out of these primitives. This is especially useful in the organisation of object icons, in which, for example, the icon for 'blow-dryer' may be reached under 'hand-held device,' 'heat-producing device,' or 'personal device.' These paths are also very important in retrieval, because they can guide generalisation and specialisation of search criteria by functioning as a semantic net of hierarchically organised classes, subclasses, and instances.

6.1 A language for action

A representation of cinematic action for video retrieval and repurposing needs to focus on the granularity, reusability, and semantics of its units. In representing the action of bodies in space, the representation needs to support the hierarchical decomposition of its units both spatially and temporally. Spatial decomposition is supported by a representation that hierarchically orders the bodies and their parts which participate in an action. For example, in a complex action like driving an automobile, the arms, head, eyes, and legs all function independently. Temporal decomposition is enabled by a hierarchical organisation of units, such that longer sequences of action can be broken down into their temporal subabstractions all the way down to their atomic units. In (29), Lenat points out the need for more than a purely temporal representation of events that would include semantically relevant atomic units organised into various temporal patterns (repeated cycles, scripts, etc.). For example, the atomic unit of 'walking' would be 'taking a step' which repeats cyclically. An atomic unit of 'opening a jar' would be 'turning the lid' (which itself could theoretically be broken down into smaller units - but much of the challenge of representing action is knowing what levels of granularity are useful).

Our approach tries to address these issues in multiple ways with special attention paid to the problems of representing human action as it appears in video. It is important to note in this regard - and this holds true for all aspects of representing the content of video - that unlike the project of traditional knowledge representation which seeks to represent the world, our project is to represent a representation of the world. This distinction has significant consequences for the representation of human action in video. In video, actions and their units do not have a fixed semantics, because their meaning can shift as the video is recut and inserted into new sequences (30, 27). For example, a shot of two people shaking hands, if positioned at the beginning of a sequence depicting a business meeting, could represent 'greeting,' if positioned at the end, the same shot could represent 'agreeing.' Video brings to our attention the effects of context and order on the meaning of represented action. In addition, the prospect of annotating video for a global media archive brings forward an issue which traditional knowledge representation has largely ignored: cultural variance. The shot of two people shaking hands may signify greeting or agreeing in some cultures, but in others it does not. How are we to annotate shots of people bowing, shaking hands, waving hello and good-bye? The list goes on. In order to address the representational challenges of action in video we do not explicitly annotate actions according to their particular semantics in a given video stream (a shot of two people shaking hands is not annotated as 'greeting' or alternately as 'agreeing'), but rather according to the motion of objects and people in space. We annotate using physically-based description in order to support the reuse of annotated video in different contexts - be they cinematic or cultural ones. We create analogy mappings between these physically-based annotations in their concrete contexts in order to represent their contextual synonymy or lack thereof.

6.2 Character actions and object actions

• conventionalised physical motions

• abstract physical motions.

Object actions are subdivided horizontally into actions involving a single object, two objects, or groups of objects (see figure 5). Each of these is divided according to object motions and object state changes. For example, the action of a ball rolling is an object motion; the action of a ball burning is an object state change.

We represent actions for characters and objects separately in the Director's Workshop because of the unique actions afforded by the human form. Our icons for action are animated which takes advantage of the affordances of iconography in the computer medium as opposed to those of traditional graphic arts.

6.3 Characters and objects

Objects are subdivided vertically into various types of objects and number of objects.

6.4 Relative positions

6.5 Mise-en-scene: time, space, and weather

Space is subdivided vertically into geographical space (land, sea, air, and outer space), functional space (buildings, public outdoor spaces, wilderness, and vehicles), and topological space (inside, outside, above, behind, underneath, etc.)

Weather is subdivided vertically into moisture (clear, partly sunny, partly cloudy, overcast, rainy, and snowy) and wind (no wind, slight wind, moderate wind, and heavy wind) (see figure 9). Temperature is not something that can be directly seen. A video of a cold clear day may look exactly like a video of a hot clear day. It is the presence of snow or ice that indirectly indicates the temperature.

We use these icons to represent two very different types of space, time, and weather on a Media Time Line: the actual time, space, and weather of the recorded video and the visually inferable time, space, and weather of the video. The difference can be made clear in the following example. Imagine a shot of a dark alley in Paris that looks like a generic dark alley of any industrialised city in Europe (it has no distinguishing signs in the video image which would identify it as a Parisian dark alley). The actual recorded time, space, and weather for this shot differ from its visually inferable time, space, and weather. This distinction is vital to any representation for reusable archives of video data, because it captures both the scope within which a piece of video can be reused and the representality of a piece of video, i.e., some shots are more representative of their actual recorded time, space, and weather than others.

6.6 Cinematography

6.7 Recording medium

6.8 Screen positions

6.9 Thoughts

6.10 Transitions

Shot 1 Person enters elevator, elevator doors close

Shot 2 Elevator doors open, person exits elevator

The viewer infers that a certain amount of time has passed and that a certain type of spatial translation has occurred. Noel Burch has developed a systematic categorisation of spatio-temporal transitions between shots in cinema (11). He divides temporal transitions into continuous, forward ellipses in time of a determinate length, forward ellipses of an indeterminate length, and the corresponding transitions in which there is a temporal reversal. Spatial transitions are divided into continuous, transitions in which spatial proximity is determinate, and transitions in which spatial proximity is indeterminate. Burch's categorisation scheme was used by Gilles Bloch in his groundbreaking work in the automatic construction of cinematic narratives (8). We adopt and extend Burch's categorisation of shot transitions by adding 'temporal overlaps' as a type of temporal transition and the category of 'visual transitions' for describing transition effects which unlike traditional cuts, can themselves have a duration (icons for transition effects which have durations are animated icons). In the Director's Workshop, we horizontally subdivide transitions between shots according to temporal transitions, spatial transitions, and visual transitions (cuts, wipes, fades, etc.) (see figure 12).

When a transition icon is dropped on the Media Time Line, Media Streams creates a compound icon in which the first icon is an icon-sized (32 x 32 pixels, 24 bits deep) QuickTime Movie containing the first shot, the second icon is the transition icon, and the third icon is an icon-sized QuickTime Movie containing the shot after the transition. So returning to our example of the two-shot elevator sequence, the compound icons would be as follows:

Temporal transition
(forward temporal ellipsis of a determinate length)

Spatial transition
(spatial translation of a determinate proximity)

Visual transition
(simple cut with no duration)

We intend to use transition icons to improve Media Streams' knowledge about the world and to facilitate new forms of analogical retrieval. A search using the icons above would enable the user to find a 'matching' shot in the following way. The user could begin with a shot of a person getting into an automobile and use one or more of the transition icons as analogical search guides in order to retrieve a shot of the person exiting the automobile in a nearby location. The query would have expressed the idea of 'find me a Shot B which has a similar relation to Shot A as Shot D has to Shot C.'

6.11 Extensibility of the icon language

Users can also create new icons for character and object actions by means of the Animated Icon Editor (see figure 14). This editor allows users to define new icons as subsets or mixtures of existing animated icons. This is very useful in conjunction with our complete body model, because a very wide range of possible human motions can be described as subsets or mixtures of existing icons.

Applying the results of work on automatic icon incorporation would be a fruitful path of exploration (22). Already in our icon language, there are many iconic descriptors which we designed using the principle of incorporation (by which individual iconic elements are combined to form new icons). Creating tools to allow users to automatically extend the language in this way is a logical extension of our work in this area.

7 Media Time Lines

Since video is a temporal medium, the first challenge for representing and annotating its content is to visualise its content and structure. In the Media Time Line we represent video at multiple time scales simultaneously by trading off temporal and spatial resolution in order to visualise both the content and the dynamics of the video data. We create a sequence of thumbnails of the video stream by subsampling the video stream one frame every second. For longer movies, we sample a frame every minute as well. The spatial resolution of each thumbnail enables the user to visually inspect its contents. However, the temporal resolution is not as informative in that the sequence is being subsampled at one frame per second.

In order to overcome the lack of temporal resolution, we extend a technique pioneered by Ron MacNeil of the Visible Language Workshop of the MIT Media Laboratory (31) and used in the work of Mills and Cohen at Apple Computer's Advanced Technology Group (33). We create a videogram. A videogram is made by grabbing a centre strip from every video frame and concatenating them together. Underneath the subsampled thumbnail frames of the video is the videogram in which the concatenated strip provides fine temporal resolution of the dynamics of the content while sacrificing spatial resolution. Because camera operators often strive to leave significant information within the centre of the frame, a salient trace of spatial resolution is preserved.

In a videogram, a still image has an unusual salience: if a camera pans across a scene and then a centre strip is taken from each video frame, a still will be recreated which is coherently deformed by the pace and direction of the camera motion and/or the pace and direction of any moving objects within the frame. Our contribution is that by simultaneously presenting two different, but co-ordinated views of video data - the thumbnails, with good spatial resolution and poor temporal resolution, and the videogram, with poor spatial resolution but good temporal resolution - the system enables the viewer to use both representations simultaneously in order to visualise the structure of the video information (see figure 15). This idea of playing spatial and temporal resolutions off one another is also utilised in Laura Teodosio's work on 'salient stills' (36) and holds promise as a general guideline for creating new visualisations of video data. An example of this spatial/temporal trade-off can be seen in the figure below in which the movement of Arnold through the frame is visible in the right hand side of the videogram and the fact that swath of extended face corresponds to the central figure can be seen from the thumbnail above.

With little practice, users can learn to read this representation to quickly scan the dynamics of video content from this spatial representation. Scene breaks are clearly visible as are camera pans, zooms, tracking, and the difference between handheld and tripod recorded video footage. The deformation of the still image in the videogram provides a signature of camera and/or object motion as in the example above.

Audio data in the media timeline is represented by wave form depicting amplitude as well as pause bars depicting significant breaks in the audio. Currently our algorithm uses a set threshold which works fairly well for many videos but a more robust algorithm is needed. Significant work has been done by Barry Arons on pause detection and audio and speech parsing in general (6); we hope to incorporate these results into our system. Arons' work uses dynamic thresholding and windowing techniques to facilitate better detection of pauses in speech and the separation of speech from background noise in unstructured audio recordings. Similarly, work by Michael Hawley in developing specialised audio parsers for musical events in the audio track could be applied to automatically parsing the structure and enriching the representation of audio data (26).

In annotating the presence or absence of audio events within the data stream, our representation makes use of the fact that in thinking about audio, one thinks about the source that produced the audio. Icons for different objects and characters are compounded with the icon for the action of producing the heard sound in order to annotate audio events. This concept correlates to Christian Metz's notion of 'aural objects' (32).

Annotation of video content in a Media Time Line is a simple process of dropping down iconic descriptors from the Icon Space onto the Media Time Line. Frame regions are then created which may extend to the end of the current scene or to the end of the entire movie. The select bar specifies the current position in a movie and displays the icons that are valid at that point in time. Icons are 'good-till-cancelled' when they are dropped onto the Media Time Line. The user can specify the end points of frame regions by dragging off an icon and can adjust the starting and ending points of frame regions by means of dragging the cursor. A description is built up by dropping down icons for the various categories of video annotation. The granularity and specificity of the annotation are user determined.

8 Conclusions and future work

Acknowledgements

References