W3C/SMPTE Joint Workshop on Professional Media Production on the Web

Something unimaginable just 5 years ago has happened: movie and TV production assets are processed and stored in the cloud. In parallel, the web has evolved from text pages with opaque plug-ins to a rich audio-visual medium. The time feels right for professional media production and the web platform to come together.

Modern productions are a collection of millions of files created by 100s of users around the world to create a single piece of content surrounded by an array of marketing, licensing and distribution needs. In this talk i'll focus on how information moves throughout the creator/production/studio ecosystem. The high velocity of files sizes, types, qualities all with Tier 1 security requirements and real time review expectations create an interesting mix of challenges.

There are many interoperable solutions for video and audio pipelines that start or terminate in a browser and incorporate ingest, playback, editing and other workflows. When it comes to production metadata associated with video and audio, it's much more of a mess.

Metadata might be standardised like a GPS coordinate or time of day in ISO 8601 format. It might be proprietary like a tripod's gimbal position or characteristics of on-set lighting. Metadata might be bursty, slow and not synchronised to the video like a log of who's in shot or it might be high speed, binary and related to free-running clock like lens position or zoom setting. Metadata might be generated by a device like the velocity of a car, buggy, dolly or crane. Metadata might be derived from on-set stimuli such as the LED wall virtual set velocity.

The only common factor to all of this metadata is that none of it makes it to the VFX / Editor / Mastering suite unless you do a bunch of customs stuff in your workflow. I think we can do better. What do you think?

Consumer and professional media web applications used for creating and managing different media content require integrated collaboration applications to aggregate and generate new media content.

Due to the rapid growth of structured/unstructured data and user-generated media content the volume of data is becoming too big or it moves too fast.

For an evolving environment it is necessary to understand what knowledge is required for a task domain with an adaptive ontology matching.

Content supply chain for media production includes nowadays several use cases, including localization, validations and in an open-collaborative environment using multiple systems: transcoder, editing tools, AutoQC and even AI tools. Using a Media Integration platform will leverage the implementation of these use cases as it will be possible to define these workflows using a LowCode/NoCode paradigm and most importantly future proof. In this talk we will present the challenges and what barriers will break focus on the need of standardization on the APIs and Datamodels.

Ed Gray talks about the business case for excellence in accessibility, development directions driving accessibility, some key lessons learnt and lastly shows how a user of native creative tools with accessibility needs can benefit from thoughtful design.

This talk offers a real time coding demo to familiarize viewers with the core interfaces of WebCodecs. The demo will demux, decode, and render video from an mp4.

In this talk I'll present the current state of VideoEncoderConfig dictionary. The idea is to show you what we have so you can tell me what we're missing.

This talk discusses memory access patterns when WebCodecs is used in conjunction with other APIs, what WebCodecs does today to minimize memory access overhead, and envisioned solutions for further improvement in the near- and longer-term future so that WebCodecs' performance profile can match that in a native application.

"What are the things that you need to think about if you were to build an audio app on the web today?" In this presentation, we discuss how to take advantage of Web Audio API by understanding its architecture and the performance charactereistics. Also it is followed by the introduction of a set of tools named "Web Audio perf toolkit", which allows you to debug, profile, and measure your application from several different angles. We also briefly touches critical problem areas such as privacy and input/output latency.

For many in broadcast and streaming, WebRTC is not “complete”, as it lacks a standard signaling protocol to make it work like RTMP or RTSP.

WHIP, the WebRTC HTTP Ingest Protocol, was developed to solve the biggest pain point with adopting WebRTC as a serious, professional, robust contribution protocol: Media Ingest. WHIP enables WebRTC to retain its technical advantages over older protocols like RTMP when it comes to resiliency over bad network conditions, adaptability, end-to-end encryption, and new codec support (hello AV1 SVC). It also removes the barrier WebRTC had with a lack of standard signaling protocol that has made it hard to support as a software solution, and difficult for hardware encoders to implement WebRTC.

Developers love WebRTC because it is an IETF & W3C standard that makes it easy to write client applications with native broadcast and playback support on billions of devices worldwide. And the WISH working group at the IETF is currently reviewing WHIP with a milestone to publish it as a standard by December 2021.

Implementing the open source WHIP library in your software or hardware encoder is all you need to support the entire WebRTC stack on the sender side.

It’s time to WHIP WebRTC into shape and take advantage of WebRTC end-to-end, as it was meant to be, natively on every device.

Introduce some possible ways for web developers to do video-transcoding based on WebAssembly and WebCodecs, and the chance if we could make video-transcoding easier to achieve by WebCodecs APIs.

On the web today, creative applications can't implement tools that allows users to select colors from any pixels on their screens. Microsoft is proposing (and has implemented in Chromium) a new API that enables authors to use a browser-supplied eyedropper. This talk describes the problem that this API solves, shows a little demo of the feature in action, and describes the API.

Creative tools have long been able to push the boundaries of performance on desktop PCs by taking full advantage of hardware capabilities, often by interfacing directly with hardware. This talk describes many of the challenges that exist with building tools based on the web platform that are just as capable as existing desktop tools. We will also cover some of the current and proposed APIs that will help overcome these challenges.

Initially using WebAssembly to run cross compiled codecs, we have just ported to using WebCodecs instead. The talk discusses the impact of WebCodecs for Browser Hosted Video Editing and how WebCodecs interacts with the wider browser development environment.

WebAssembly makes it possible for us to migrate our C++ video editing engine to the web, but the development experience is not perfect. I want to talk about some possible improvements.

This focuses on accuracy when working with media on the web. Accuracy is crucial In professional media workflows and Web media APIs can be lacking in this area.

This talk focuses on the use of WebRTC in professional live broadcast workflows. In a live production environment, users need to rely on their tools being accurate and performant. Furthermore, greater flexibility in the WebRTC API would allow novel workflows and optimisation.

Clipchamp's mission is to enable everyone to tell stories worth sharing through the power of video. To live up to that goal, we have built a video editor that runs fully in-browser. This also means running the full video processing pipeline on the client and engineering around the challenges that the browser still poses for resource-intensive workloads. Offering access to hardware-accelerated video decoding and encoding, the Clipchamp team has started integrating WebCodecs into our video processing stack early on. In the production version, we wrap the WebCodecs API into custom codec plugins. These form a part of the FFmpeg WebAssembly port that is the beating heart of Clipchamp. This talk covers some of the challenges we encountered along the way and discusses the engineering decisions that we made in response to that. We also make suggestions for some future iterations of the standard, hoping to see it further improve over time.

This talk will reflect on Olympics 2021 experiences in media assets productions and distribution from China Mobile, as the solo live broadcaster for new media in China, and explore the combination of 5G-based remote production, lightweight web system as well as HDR features.

Classic video conferencing and assets sharing tools are not adapted for group work when it comes to high fidelity content such as art, animations, video. We created a novel approach where virtually any website or web-enabled tool can gain additional functionality for live co-experience in viewing (common operating picture), commenting, and editing creative content. The desired web resource is loaded and operated in the cloud, while exact incremental changes are broadcasting to all connected clients.

This talk presents a wishlist of common needs when developing media editing applications, along with relevant API features that could be used if they were supported across web browsers.

WebAssembly is great for audio plugins, AudioWorklet deliver real time experience for music production. Still latency, stability and audioworklet resource priority is essential to give a usable experience. I'll demonstrate how it should work on my M1 macbook Air (in chrome) with demos from the WebAssembly music project.

Soundtrap implements a collaborative Digital Audio Workstation (DAW) in the browser and makes heavy use of WebAudio and other web APIs. We highlight two needs for audio/midi recording use case:

• Low latency for 'monitoring'. Ex, feeding the input through effects or playing a software instrument.
• Accurate latency information to perform 'latency compensation' when recording multiple 'tracks'.

In the DisneyResarch|Studios (DRS) machine learning pipeline DRS frequently works with production assets in the OpenEXR format. Monitoring the training, inspecting results, and eventually judging the quality of these new approaches require tracking and comparing those assets. DRS decided for an approach where a server provides high dynamic range results as EXR files and compiled OpenEXR using emscripten to allow in-browser decoding and viewing. Using WebGL, simple adjustments and a visualization of per pixel error metrics can be performed by the client on the fly. In this talk, DRS will present the problem, the chosen solution, challenges that DRS has encountered, and demonstrate JERI.io, the web-based EXR viewer, which DRS has released as open source.