<tidoust> scribe: tidoust

Chris: Today's meeting is largely focused on topics linked one way or the other to the CTA WAVE project. 3 topics: CTA WAVE update, testing effort, and Media Integration Guidelines. We may also talk about the byte stream format for CMAF that CTA WAVE has been working on.

Web Media API

JohnRiv: I'll briefly touch on what CTA WAVE is.
... I talked about WAVE last year.
... CTA WAVE is from CTA, WAVE for Web Application Video Ecosystem. Looking to improve interoperability for media on the Web.
... Not looking to create any new protocol, but rather to reference existing ones such as MSE/EME, etc.
... Within WAVE, there are a number of task forces
... Content Specification TF is led by John Simmons. Based on CMAF. Goal is to produce common content for DASH/HLS.
... HTML5 API TF, which I'm leading, develops the Web Media API snapshot, published every year.
... Functional guidelines for playback interoperability on devices. Specs and features that are supported across browsers.
... CMAF Byte Stream Format group is working on the byte stream format that Chris mentioned
... DASH-HLS Interoperability Group led by Zach from Hulu.
... Last, Common Media Client Data Group to develop a common media client data spec, led by Will Law.
... Lots of things going on in WAVE.
... The way we're working is by referencing existing specifications from other organizations.
... Digging into the Web Media API Snapshot.
... Referencing a bunch of specs from W3C, WHATWG, ECMA, and Khronos Group
... We identify and document a minimum set of web standards for playback of audio/video content in HTML5. The main focus is "if you want to playback media content, what technologies do you need?".
... For device manufacturers, content providers, app developers.
... Focused on technologies supported across 4 main browser engines.
... If a specification is not supported across them, we will very likely not include it.
... The Snapshot work happens both in CTA WAVE and in the Web Media API CG
... Target is anywhere where you're going to embed a browser.
... The spec is updated every year.
... Both CTA and W3C co-publish the Web Media API Snapshot, as a CG report from a W3C perspective.
... As a CTA WAVE specification.
... Even more importantly, we have an automated test suite to run against devices so that we can ensure that devices support the APIs.
... In 2020, what are the updates?
... Not ideal to reference living standards when device manufacturing is concerned. We reference Review drafts as a result, and update every year.
... Moved to ECMAScript 2020.
... A handful of new CSS specifications were added (e.g. Scroll Snap).
... More security specifications.
... We added a section for Web performance specifications.
... We also now do call out limitations around available hardware decoders, e.g. if you want to playback multiple streams whereas there is only on hardware decoder available.
... Progress happens on GitHub.

Chris: How do you determine the interoperability between the different browser engines? Using Can I Use? MDN? Your own tests?

JohnRiv: That's a combination of all of this.
... These sources are not always accurate.
... We look at changes, and submit PR as we see fit to have the sources updated.

Jeff: Over the last year or so, we had a bunch of conversations about having a workshop about how web developers use media APIs.
... Wonder about how CTA WAVE interacts with developers.

JohnRiv: A lot of the WAVE members are W3C members. We look at where developers have issues. But we want to do more on that, and want to collaborate more with W3C on that.

Chris: Other question on the security stuff. I saw that there was a TAG issue around passing limited information in the user-agent string. One of the things we do as content providers is to look at this string to assess capabilities.
... I wonder if that has come up in CTA WAVE discussions. I believe the idea is to move to UA Client Hints, which could protect privacy a bit more.

JohnRiv: We're looking at the Media Capabilities API.
... What does the device support? Can I send this content?
... We have some feedback for the Media WG on Media Capabilities.

Takio: From Yahoo! Japan. Interested in CMAF and support for MPEG2-TS.

Will-Law: The interop looked at both containers and decided that CMAF was more forward-looking and better. It is correct that the document does not address MPEG2-TS.

Device Platform Tests

Louay: From Fraunhofer FOKUS.
... Web Platform Tests provides cross-browser test suites for Web platform specs.
... WPT provides both tests and a test runner.
... But there is a limitation for Web Media APIs, which is limited support for embedded devices.
... This is the reason why WAVE started the Web Media API Snapshot test suite project
... The goal was to add support for embedded devices, subset the tests accordingly to the specification, and integrate the updates back into WPT.
... Limited interaction capabilities on embedded devices (e.g. if you have to use a remote control). We added a new way to run tests "remotely" and control progress;
... We have a concept, called subsetting, where we check which tests from WPT tests are relevant for each snapshot.
... You can select tests that only pass across main browsers, or in one specific browser.
... There are also external test suites, e.g. the ECMAScript and WebGL test suites, which we had to integrate as well, so that you can run them along with other test suites and get only one report for the whole thing.
... Project started in 2018 with the publication of the test suite. Same thing in 2019. In 2020, we successfully ported the test suite to Python and upstreamed it in the WPT project.
... I'll show you a demo. Two devices: the TV under test, and a companion device such as a smartphone to run the test.
... The companion device starts by scanning a QR code displayed on the TV. Then you can select which tests to run. Minimal interaction with the TV set, everything happens through the companion device.
... You can also pause and resume sessions.
... And export the results.
... Through the companion page, you can connect to a test session, directly from your browser.
... Once tests are done, you can generate the report.
... Report is compatible with the WPT project. You can download it.
... These are the main functions of the test suite.
... [going through history of changes in test suites over the years]
... We managed to contribute the changes back into WPT, major result from the project.
... Last part where we're currently working on is integration of the test runner which we extended to add features such as observation, e.g. to add recordings of the device from a camera and check the results of the test later on.
... That's it from my side.

Chris: Do you have test cases that are covering the CMAF byte stream format for CMAF?
... Are these tests contributed back to WPT?

Louay: We are working on this. I may add a link to the GitHub repository on this.
... We are working in a separate repository for the time being, but there is no blocker to push it to WPT repo.

Igarashi: Question about embedded devices. You say that you support a subset of tests? Does this mean that CTA WAVE will provide a profile of tests?

Louay: The subsetting is basically: a subsetting of the APIs themselves, filtering out those that are not listed in the Web Media API Snapshot. But also, when you have the test suites, you may still select the APIs you want to run, e.g. to debug.
... If you use Web Media APIs 2017, you will only consider what is included in this version of this snapshot.
... Most of the APIs are already available in the WPT project. Some other APIs (ECMAScript, WebGL) are elsewhere, and we added support for them.
... We have an importer.

Igarashi: I'm very curious about the test results about the embedded devices. Do you have information about how many devices pass the test suites you created?
... Percentage of devices?

Louay: I cannot say device names, or company names. We had tests on 3 embedded devices, game console, set top box, TV for the first version.
... If you restrict tests to those that pass across the 4 main browser codebases, then you have higher chance of passing the tests.
... Of course, some tests can crash the device, but we can resume the tests afterwards, and add the tests to a blocklist afterwards.

Kaz: WoT WG is working on PoC for testing. This framework seems very useful to WoT as well. Having a joint discussion about that would be good.

Louay: Right, the core is moving the heavy stuff to the server, as embedded devices are thin clients.
... Maybe you can use adapters.

Chris: What's the best way to follow-up with questions?

Louay: I'll add links to GitHub projects to the slides, best way to raise questions.

Media Integration Guidelines

JohnRiv: We talked about this in a previous IG meeting this year.
... There are issues that cause interoperability pain with web media APIs. e.g. with hardware video & audio decoders.
... Initially, we started to compiling issues in CTA WAVE in porting repo.
... We migrated the discussion and issues to the IG
... The goal of this effort is to create a W3C Note around Web Media API Integration Guidelines, focused on hardware media decoders.
... We'd like involvement from browser vendors.
... If there are discrepancies, we'd like to liaise with the right groups, but the goal is to collect and document practical guidelines.
... At this point, we've mostly been working on migrating the issues.
... The Chrome team has done a great job at responding to some of these issues. We're looking for feedback from other browser vendors.
... If you can comment on existing issues, and provide issues & examples, that would be excellent!

<kaz> Media Guidelines Issues

JohnRiv: Thanks to Dan Senders, Chris Cunningham and Chris Poole for providing feedback already.
... Any question or comment?

Chris: The scope is the main browser engines that you describe in the Web Media API. Embedded devices may add further restrictions. Are you looking to integrate them in the guidelines?

JohnRiv: Yes, if we can gather the appropriate info, we'd welcome that.

Chris: To what extent can we help with gathering input from other browser vendors? Absent direct feedback, is there another way to gather that information?

JohnRiv: We could look at the source code, also run the test suite. The insights we got from the Chrome team gives us a lot of useful info about why things work a certain way.

CMAF byte stream format

JohnSimmons: The discussion has pivoted around the fact that CMAF is an ISOBMFF file format. Same MIME type. If you added an annex to the existing byte stream format with additional constraints, there needs to be a way for an app to identify that it is in this case. Usually done by looking at the MIME type.
... That is the stumbling block so far. Discussion between Media WG people and CTA WAVE people.
... Does CMAF require updates to the ISOBMFF byte stream format spec?
... CTA WAVE put some work into the spec, currently designed as a separate spec.

Chris: Should the IG do something about this?

JohnSimmons: The right way for the media industry to provide feedback is, I believe, for the IG to take the lead. I would personally prefer this come through the IG and be a dialog within W3C.

Chris: OK, we'll follow this up.

<JohnRiv> +1 to what JohnSimmons just said :-)

JohnSimmons: There should be a conversation between the IG, WAVE and the Media Working Group. Also on how to make other issues identified in WAVE move forward. Process is not well-defined for now.

Chris: Let's get back to that in our second meeting.

[adjourned]