W3C/SMPTE Joint Workshop on Professional Media Production on the Web

Preserving metadata in production workflows

Related GitHub issue: Issue #27.

Chris Needham: We would like to spend the next 20 minutes talking about metadata. This was a topic that Bruce introduced in his presentation. Bruce, if you would like to say a few words, you introduced an interesting diagram showing the different levels of information, is it textual, binary information, is it asynchronous, is it metadata. Can we dig a bit on the detail around that and to think about what, in terms of a browser API context, what we should be looking at that's interesting.
... We talked previously about the metadata that's carried in the bitstream so I don't know if there is anything in there we want to expand on that. Interested to hear your thoughts.

Bruce Devlin: Let's give a minute outlining the problem as I see it, as we start to bring more and more devices into this Internet context, more and more of them emit metadata, in particular in the media production space. Metadata could be anything. It could be the focal length of a lens, that in itself is a difficult thing to define and I'm not getting into that level of detail today. It may be the GPS location of a cameraman running along with a camera. It may also be the pan and tilt, the velocity of the gimbal upon which the camera is sitting. The person holding the steadycam may be on the back of the truck with a GPS but that truck may be a trailer being towed and you may have that information that's towing the truck, the trailer, and the cameraman pointing it to the most important point on set, the actor. There is all of this stuff going on.
... What's interesting, when you shoot that scene, you know everything about it. When I look at metadata papers, I see people showing how clever it is to look at the video and figure out who is in front of the camera in that scene. We knew who was in front of the scene on set! We threw that metadata away. By the time you need it, going into the VFX pipeline or the distribution pipeline, you have to go and reinvent the metadata from scratch.
... That's the general context. There's a number of things that you may like to do with the metadata on set or if you're backhauling the metadata to do stuff with it. Firstly, you may want to visualize it in a more friendly fashion. The metadata coming out of the devices, it typically is one of four categories as you pointed out, Chris. It may be in text form, in binary form so that's the first context. That's important, it tells you about how you'll process it, what sort of processing engineering you need.
... The next axis that's equally important, the time axis. Some metadata, is it asynchronous? The clock is ticking somewhere, it may be the audio sample or video sample clock, it may be a crystal oscillator while on the gimbal on the truck and it is a crash detector and syndicator at the same time. So there's synchronous metadata versus non-synchronous and contextual versus non-contextual. When you know what quadrant the metadata is in, you have a good grasp of what data you have to put metadata around to put it in transport: WebTransport, WebRTC, SMPTE ST 2110 or some other random transport system to get it from A to B.
... The problem I'm trying to figure out, it is do we have to just consider the transport getting the metadata from where it was originated to where it needs to be visualized. Is there something that we should reconsider such as the format of the metadata? If you try to transcode the metadata, generally you destroy it.
... The first problem that I wanted to throw on the table here for the combined brain trust of W3C and SMPTE: is there something we could do, a bit of extra work to make something generally useful to get the data from where it is made to where it needs to be used? At that point, I'll stop talking.

Chris Needham: Is there anybody in the call who would like to respond or share your own point of view? In your experience, do you have an idea of what the web platform could do to actually solve this?

Bruce Devlin: I think part of the thing that has to be done is visualization of the metadata while it is happening and there may be many layers of that data needed to make it useful. I have spent a lot of my career working with people who are doing simple things like shot logging to figure out who kicked the football, who caught the ball, who is doing what at the same time. Those things are all really custom bits of software glued onto great huge monolithic pieces of software around the backend and they tend to be database driven.
... Looking at the wonderful stuff that W3C is doing with the transport, the logging, the insertion and extraction that's more generic, the generic web browser and generic tools that with the metadata visualization plug in with an API, you could make the generic logger something that's not just human assisted but possibly AI assisted, so don't know if it is a human assisting this or an AI or just... I feel there is something there but I'm not quite expert enough to figure out the shape of the thing that I think needs to be built.
... Maybe starting from a pure technology layer, WebRTC we have heard is one way to do live transmission of audiovisual data. Does that support channels other than audio and audio and video? Can you transmit the text point or the arbitrary data or is it limited to only the essence that's supported by codecs on the web platform?

Chris Needham: There is the data channel. You can imagine, on a conference call like this, we can share audio and video but we can also exchange in chat, so this supports that kind of information as well. As far as I'm aware, it does not provide any synchronization guarantees, it is a side channel to the audio and the video.

Bruce Devlin: In this problem space that's one of the important things that hasn't been done before. This is why I'm clutching a bit at the solutions. I think there are certain classes of metadata that have to be absolutely hard locked to the originating audio and video. They're a first class citizen in terms of the synchronization. There are other bits of asynchronous metadata for which it is a bit more loose and floppy with less guarantee. It certainly needs to be resynchronized at a later date, not at the point of emission but definitely re-synchronized after some sort of a delay for subsequent downstream processing.

Chris Needham: This makes me think of a couple of things. With WebCodecs, we have the ability to encode and decode the audio and video. If you're doing an encode, you also need to insert asynchronous metadata, if the metadata needs to go into the bitstream alongside the audio and video, then perhaps WebCodecs does not currently give you that capability.
... If the metadata goes inside of your media container format with the bitstream, then perhaps that's something that a library would do rather than the WebCodecs API and it raises the question of how do you implement that. Part of what we do in building the web APIs is to think about what are the capabilities that really only the browser can provide and web developers can build on top of that.

François Daoust: The document that Dominique shared on the chat is the use cases and requirement exercise that the WebRTC working group where through to discuss the next version of WebRTC.
... It includes some use cases about the synchronization issues that we mentioned actually in the past couple of live sessions as well. There are situations where you have data on the side and you need to synchronize it with some media stream. That includes captions, which we mentioned already, but may include other types of data, for instance for virtual reality environments.

Dominique Hazaël-Massieux: Just to add to this, this need of synchronizing data and media has been brought up a few times in WebRTC, I guess a big part of making it happen would be to see the strong demand. Use cases in the document that's being shown are is mostly linked to real VR/AR at this point but if there are specific use cases for the media production workflows that could emerge, I think it helps with getting it higher up the list.

Bruce Devlin: I think I can probably provide some use cases from my random post-it notes scattered around the office, if only there was a solution we could do this, trying to get a structured document, I can certainly provide things that people have contacted me about.
... You mentioned a few use cases that would be worthwhile digging into a bit. Analyzing what's in the media to extract the metadata from it is interesting. Some things we know upfront, who will be in the scenes, so you don't need to analyze.
... I guess there are cases where the media gets run through machine learning algorithms to detect and extract extra information and so one of the things that perhaps the web platform could provide is other hooks to enable you to insert your own analysis, for example into a media processing pipeline.
... Just mentioned one really important one, which actually the BBC is one of the leaders with along with the New York Times and Washington Post and various other news sources, is a thing called the C2PA, an anti fake news identifier and it is really important that it is synchronous with the content. If it is not synchronous with the content you can't tell the provenance of the news being shown. That's one that's really, really important that the synchronicity is kept pretty much to a frame level so that you can prove the source of not just the camera that generated the content but also what editing stages took place between then and now so that you know that if the stuff was fake, at least you know who faked it.

Chris Needham: We have had actually in W3C an initial introduction from the C2PA group. We haven't yet really dug into what the requirements are from a browser integration perspective but that's definitely an area that we're aware of.

Bruce Devlin: I would be certainly interested to follow up. I will give a use case that's very browser centric. I was talking to a designer of the C2PA and it came down to: what if somebody presses pause in the browser. You hit the pause on a keyframe, you have the keyframe, that's what codecs tend to do. If that's the only frames manipulated so that they were faked actually you have increased the statistical likelihood of somebody sharing a still from your video that was faked rather than the original, hence the synchronousity of the information, it has to be very, very good.

Chris Needham: A thing that interests me, you talked about this difference between stuff that happens in sort of monolithic databases and stuff that happens at a high level. What I'm interested in, what use cases we have for mutability of data which changes on a per-frame basis as opposed to data which has to be accurate, which can be considered generally unlikely to have the interval of change of frame by frame every so many milliseconds.
... Stuff that lives in databases and can be consumed by traditional APIs could still be frame accurate and doesn't scale well to being mutable on a per frame basis. A lot of the stuff that you said as being per frame accurate, is that a capture and is it considered immutable from point of capture onwards? It seems to me like there is still merit in considering these things as relatively distinct entities unless there is a lot of overlap where there is stuff that changes every single frame and is curated after initial creation, if that makes sense.

Bruce Devlin: It does. I'll give you one specific use case that actually I'm doing work on today, which is lens focal length, lenses are complicated. With physical lenses, if you change the focal length actually everything about the lens breathes, but the focal length reported by most lens metadata is wrong, because the secondary effects of the breathing of the internals of the lens moves the reference plain slightly different than through the lens thinks it moves from and it is a per lens characteristic thing that needs calibrating.
... If you're doing the effects stuff, you're pulling focus in the effects of other things around it, you want to actually match that second order effect, not the first order effect because it is visual otherwise. There is a group that I'm working with, trying to figure out, first, how do you capture it accurately enough because the sampling is not done in the framework, done faster than that. What the crew would like to do, it is to get that 100 kilohertz lens data and get the trajectories of the lens movement absolutely right, including the second order effects, specially for big lenses, and being able to put that in a VFX pipeline and resynthesize what the image would have been after this is applied. That's the pipeline I'm working on at the moment. That's a specialist use case which the lenses tell you about today.

Chris Needham: It would be interesting to have an offline discussion.
... I would like to come to Kaz and James to wrap up this part of the agenda today.

Kazuyuki Ashimura: The expected metadata for media production in general shall include various data like hardware, location, the location not only of the hardware itself, but the cameraman and so on, so the users and also the data about the codec and so on. Those various kinds of data should be clarified via collection of main use cases from studios.

Bruce Devlin: Absolutely. There are a couple of studios that have internal projects where they have lists of stuff and there is not that much collaboration going on that I'm aware of. I'm trying to get the collaboration going so that we can have one very big list that we can then start choosing metadata items from. I would be mostly concerned about the transport and the visualization of this data and the people onset are concerned with capturing it without hurting anybody.

Kazuyuki Ashimura: I'm working on the Media and Entertainment Interest Group and I work with other groups. We have other things related to IoT and that group is also working on metadata for IoT devices and broadcasting devices, production metadata may be also directed to IoT.

Bruce Devlin: I think you're right. Good observation.

James Pearce: The metadata that's significant will vary wildly, the shape of that and what metadata we're capturing would vary wildly depending on use cases. Is the idea to provide assistance with validating that data, making sure it is in the right shape or are we suggesting just that we'll store opaque data and we'll let the application deal with making sure that it conforms to some schema?

Bruce Devlin: I think it is a bit of both. At a high level, you know the children's toy with a square and a circle and a star and a rectangle, I have visualized that most metadata I have found that people want to use fits in to one of these four holes that I talked about. Those holes, they're text, binary, synchronous and asynchronous, but then within each one of the holes the metadata itself would probably follow a schema depending on the sort of metadata it is.
... I have this idea that if we were able, for each one of the four holes, to provide a framework, a methodology, an API to give you all the bare bones of being able to shunt stuff from A to B quickly, you may take more than 50% of the work out of the overall system and you are just left with the JSON, an XML or a validation exercise at either end. It is a gross over implication, I feel there is something there.

Processing workflow APIs

Related GitHub issues: issue #28, issue #29.

Chris Needham: Julian, I'm wondering if you have your own perspective that you would like to share on this.

Julian Fernandez-Campon: I think it is complementary to what Bruce is saying. You manage and process the data, whether it is binary or text metadata. What I see is that we need to standardize this metadata at some point to be able to abstract different systems that are processing that metadata coming from the different applications, as was mentioned about the tools and content.
... One possibility for implementation, it is using this for all of the editorial metadata that can be IPTC. If you're dealing with other metadata, the thing is to standardize the metadata that you're using for the editorial and for all which is related with production and things.
... I think that also the ACES, the SMPTE ST 2065, it is a good option.
... What I mean, it is something like this
... [presenting slides]
... Imagine that you have a workflow and this workflow is doing some things and getting some media or something. We have one that's the analysis and in this case, we're using the analysis tool which is getting information about the codec's frame rate and many other things and it can be GPS information of things which are coming. For some reason, I want to change this one and instead of using this, I use this AI tool, called in this case Julio, and the use case that we're proposing here, it is that the workflow doesn't change. The information which is going to the next stage, which is the transcoder in this case, it is taking technical information, other information which is part of this analysis, or if we want to be using something else instead of this, we want to use for example the media info or for example we know we'll be receiving a comment card, it doesn't change.
... If we have standardized the metadata coming in the editorial metadata or the metadata that's technical, related to other things, we can do things like this, constructing the workflows with the transformations and the specific data model which is generating the information. That's what I wanted to show you. You have the profile, and now I'm able to be analyzing this.

Chris Needham: It is interesting. There are different groups that are interested in different aspects of this. I think that, in W3C, we tend to focus on the browser capabilities and so some of what you have spoken about in your talk was about having consistent APIs across different processing components.
... There are other places where this standardization happens and, like what Bruce said, how do we bring the collective expertise together so that we're not overlapping in terms of what each standardization group is doing but doing things that are complementary? There is a question in my mind about the role for W3C in terms of metadata standardization.

Julian Fernandez-Campon: This is an open question. As you say, there are many standards. W3C could for example standardize what I have been showing here. It will require, of course, more discussion. Again, this is one level up of the equal word limit, you have the database, how it is represented, it is more on that.
... If there is a way of standardizing this metadata. The other thing I want to show you, the other thing that was referred to, is the mapping between the different standards so for example here, you have the one that we have adopted.
... If you see, in the IPTC, the correspondence, again, this is one of the questions, if at some point the W3C could be proposing or suggesting a schema that could be IPTC or could be extending what is needed, depending on the type of content to this. Then we know all of the applications like this one for example that I was showing, it is what we have, didn't have any implementation from another vendor, you know that when you're going to be getting metadata from somebody in the browser, content, they know that they'll always have these feelings. The only problem would be translating these fields into the internal data model that they have or as I was showing, translating into the metadata which is going to be supported by the other system.
... This is what I wanted to throw in here to frame the presentation, to see what can be done for the standardization with the metadata at the higher level if it is possible to have more work on this. I think it is a big question. We may not be able to answer this here and now. It is interesting to follow this up with those relevant groups.
... Thank you for the opportunity, also happy to share thoughts. If someone thinks that it doesn't really work, we have to think in another way so we're totally open, of course.

Chris Needham: This has been a great discussion. We are a little bit over time and perhaps we should have allowed more time for each of the topics.