[Next]
RTMW'96
24/10/96
Issues in Temporal Representation of Multimedia Documents : The Madeus
Experience
Nabil Layaïda
Opéra Project : INRIA
[Next] [Previous] [First]
Design Goals
The goal of the Madeus application is to meet the following
requirements:
- Structured Modelling and the definition of an Open Representation of
Multimedia Information (Open Format)
- Merging the various dimensions of Multimedia Documents
- Incremental Manipulation of the Document through Interactive Interfaces
- Maintaining the Consistency of Multimedia Documents
- Automatic Production of Spatial and Temporal Layout
- Runtime support and remote access to multimedia streams
[Next] [Previous] [First]
Multimedia Composition Model
Multimedia documents are manipulated through four dimensions:
- Logical (structuring)
- Spatial (positioning)
- Hypermedia (linking)
- Temporal (synchronization)
[Next] [Previous] [First]
Temporal Composition Requirements
- Complete language (relational grammars)
- Incremental authoring methods (consistency, temporal layout)
- Flexible specifications (adjustable delays, ranges of durations,
indeterministic behaviour)
[Next] [Previous] [First]
Temporal Abstractions
Abstract Representation of a multimedia object => Temporal
Interval.
Multimedia objects can be classified into:
- Discrete
e.g.
text, still pictures, graphics.
- Continuous e.g. video, audio,
virtual images.
- Indeterministic e.g. queries, user interaction, computations.
Types of intervals:
- Free : [l, u]f value can be assigned freely in this domain
- Contingent : [l, u]c value known at end of its presentation
[Next] [Previous] [First]
Temporal Interval
Four important instants of the interval are considered :
- Mapping (allocation of media channels)
- Start (start of play time)
- Stop (end of play time)
- UnMapping (deallocation of media channels)
[Next] [Previous] [First]
Temporal Synchronization
Different levels of specification :
- Structure level (1 composite => 1 primitive)
X= { A, B, C, D, ... } ; List_Before{X};
- Document level (n objects => m relations)
X= {A, B, C, ..} ; A equals B; B before C; ....
- Object level (1 object => several sub-intervals or time slices)
A where A1=A [0,5], A2=[5,end]
- System level (n objects => lip synchronization)
X=Lip_Sync{A, B, Max_Skew} ;
- Protocol level (1 object => 1 protocol)
Object Locator (X) = protocol://server/Object_Id
[Next] [Previous] [First]
A document Source Example
<doc> background = image1.jpg, synchronization=List_before
dynamic_style : { ( [0:3] sec , volume = [0, 50] ) }
<composite> name=scene1
<obj> name=A, url=http://opera.inrialpes.fr/toto.mpeg, loop =2
dynamic_style : { ( [0:10] sec, xpos = [100 : 300 pt ] );
( [end-5:end] sec, box_effect = explode ) }
</obj> ......
<relations>
A equals B; B before C; .....
</relations>
</composite>
.......
<composite> name=scene2
........
</composite>
.....
</doc>
PS : There's more syntax for spatio-temporal behaviour
[Next] [Previous] [First]
Temporal Constraints Management
Author:
- Add new media objects (Basic, Composite)
- Specify temporal constraints between media objects
- Define containers (for groups of related objects)
For each new specification:
- Represent the new specification in the constraint
network
(Introducing/modifying chains)
- Test the consistency of the introduced/modified chains
- In case of Inconsistency, restore previous consistent state
[Next] [Previous] [First]
An Example of Document Construction
Nodes hold synchronization information + Spatio-temporal actions
[Next] [Previous] [First]
Types of Inconsistencies
- (a) : detecting cycles in the temporal graph
- (b) : searching for a solution for the constraint network described by the
graph
- (c) : using flexibility of objects and delays to adjust dynamically the
temporal scenario
( for more details see : http://opera.inrialpes.fr )
[Next] [Previous] [First]
Architecture of Madeus
A layered architecture of the Madeus application:
- An authoring tool allowing the manipulation of the document structure
- Modification of the objects attributes
- Insertion of new objects and relations
- A set of functions maintaining the internal representation of the document
- A Runtime Layer : Scheduler + TAC management + Navigation
- Object Layer :
- Audio/video : Mpeg audio, AU format, Mpeg video, MJpeg,
- Images : Png, Gif, Jpeg, ..
- Formatted text (colors, fonts, ...)
- Buttons, Links (internal and external)
- Stream Layer :
- HTTP (Progressive handling of media streams)
- Buffering (Prefetching by looking ahead in the temporal graph)
[Previous] [First]
Conclusion
In our prototype we emphasize on:
- An open document format ( simple tagging and Objects + Attributes +
Relations)
- Integrating heterogeneous media objects in a seamless way (text, images,
audio, video, programs, user interactions ...)
- Taking full advantage of temporal flexibility:
- Free delays of temporal operators
- Using the temporal properties of objects (Shrink/Stretch)
- Handling of unpredictable behaviour (user interaction, )
- Powerful representation based on temporal graphs
- Maintaining the consistency of the document
- Temporal Formatting (static and dynamic)
- Scheduling the presentation
- A full featured test bed prototype Madeus.
Several documents have been written with Madeus