Meeting minutes
Introduction
Chris: Welcome. Collaboration with CTA WAVE and W3C, closely related groups. Hope to organise future meetings to follow up on this and other topics.
CTA WAVE Streaming Media Test Suite - Devices
Louay: Jon Piesing and
I will present. Jon with the first part, then I'll go into
technical details
… We have a demo I'll show
Jon: CTA is the
Consumer Technology Association. ANSI approved standards and trade
association in the US market
… WAVE is Web Application Video Ecosystem, video on
CE devices
Jon: Here's a list of
specs that WAVE has delivered. Point specifications that solve
real-world pain points
… Device Playback Capabilities spec, CMSD, CMCD,
Web Media API Snaphot, and the Content spec
… We have test suites and tools, developed for CTA
by Fraunhofer
… WAVE has been a contributor of updates to the
DASH validator, with DASH-IF, DVB, HbbTV
Jon: The Device
Playback Capabilites test suite exercises the Device Playback
Capabilites spec
… How you'd deploy MSE and EME in a real world
situation
… The majority is in sections 8 and 9, you can see
the points that are covered
… It's a slightly old version of the document, some
things added
… Playback of fragements, switching sets, and more.
It's all CMAF
… We have audio and video media
profiles
… We went back and forth on subtitles. But the
reality is they're handled in JS, not browsers. We focused on
things like timing, interested if we missed anything
there
Louay: Many partners
have contributed to the project. This slides shows the high level
architecture of the test suite
… We'll share slides, with links to the GitHub
repos
… And give some hints on how to get
started
… Start with mezzanine content. Annotated audio and
video content. CMAF content based on the mezzanine
content
… We have a test runner based on WPT test runner.
HTML and JS templates
… And the observation framework
Louay: Mezzanine
content. It's annotated video content in many resolutions
… It gives you the opportunity to do automatic
observation. So no need for human to do that
… Annotated in the video is a QR code, some
information about the current frame, resolution, frame rate,
timestamp. Human readable text for debugging
… A flashing square for A/V sync. The red triangle
is for checking all the content is visible on screen
… Content based on pseudo-random noise
Louay: The CMAF test
content is validated from the mezzanine content. Metadata in the
form of a DASH MPD
… Groups of CMAF streams, content option
combinations, options for debugging, resolutions for testing CMAF
switching sests. Also tests for splicing and
encryption
… We have sparse matrix of test content and
options
… We have validation of the content as valid CMAF
usng the DASH-IF validator
<kaz> cta-wave/dpctf-tests
Louay: After creating
the content, we need to create test instructions. This is done in
HTML and JS
… Sequential track playing, random access to
times
… There's a template, using MSE and EME. Implements
instructions for the DPC spec
… We're not using open source players like DASH.js,
just a basic MSE player. Don't want additional features not
relevant for doing the tests
… So we can see the test really implements what's
in the DPC spec
… For sequential track playback, make sure the
content plays from beginning to end
… How to do that? In a JS implementation, can rely
on events triggered by the video element
… This isn't enough. If there are issues with
integration of the browser in a device, the media element tells you
the video is playing, but you see skipped frames or just a black
screen
… Things yuo can't discover from a JS API, so need
information observation externally
… An important aspect, while we're playing the
content we show information on top of the content, showing the
mezzanine annotations, and QR codes from the JS app
… So when recording, and in the observation
framework, you can see differenes between what's shown on the
screen and what's reported in JS
… Check whether it follows the DPC or not, and if
the test assertion passes or fails
<kaz> cta-wave/dpctf-test-runner
Louay: We have a test
runner, software that can do test automation
… We use web platform tests, not only in the DPC
but the Web Media API Snapshot. We extended WPT to be able to run
on TV and STB devices
… We've contributed changes back to WPT. The
difference is, the test logic it outsorced to a test
server
… WPT can be run on a device under test, but can't
do that on a TV device
… So we extended the runner, we have a server that
manages the tests and results
… There's an option to configure the tests.
Difficult to customise just using a TV remote
… This is why we have a companion app, can a QR
code, configure the tests from the companion page
… We have a REST API for test automation. This is
used in HbbTV test tools
… Also, when a test session completes, you can
export results to HTML and JSON
… These are compatible with the WPT test
runner
Louay: The observation
framework can run on a PC or tablet, any browser platform that
supports MSE
… Record all the test runs. We use a camera, e.g.,
a smartphone, to record the test session
… The OF analyses the recording automatically,
extract QR code from each frame, process the data and compare to
DPC spec
… Can depoy the OF to the same device or test
runner. Or to a more powerful machine, because you're doing video
decoding, media processing, so needs a device with good
performance
… You can run it in a Docker container, but make
sure the performance fits your requirements, e.g., to produce fast
results
Louay: There's a
landing pages for the test suite. This should be your starting
point
… Go to the webpage, there's an explainer, link to
the GitHub repository from where you can deploy the test
runner
<kaz> cta-wave/dpctf-deploy
Louay: I'll show a demo
video
… You see the device under test in our lab. It's in
HbbTV mode
… There's a smartphone camera that records the test
run, and a cable to the smartphone, it's the audio output from the
TV
… If you just use the smartphone mic, it doesn't
work, needs cleaner audio
… And you have the audio and video in the same
file
… Scan the QR on the TV, then you can open a
companion page where you can configure and select the tests to run
on the TV
… We have long time playback tests, content more
than 2 hours. It gives you a fast way to select the relevant
tests
… Start the test run, and between each test you'll
get a screen showing what the test is
… Tests are 30 or 60 seconds, or the longer 2 hour
tests
… You can see annotations from the mezzanine
content on the screen. A QR generated in JS
… Encodes hat is the last action, timestamp. Used
in the OF
… The OF detects that the first frame is displayed,
to check the content is played from the beginning to the
last
… The test runner then starts the next
test
… From human observation, we can see it's fine. But
some things are harder to detect, so why we need the
OF
… On the last page, it shows session completed. The
companion screen shows all the info about the test run
… Number of passes and fails, buttons to download
the results as JSON or an HTML report
… Now copy the recording from the camera and run
the OF to check for passes/fails
… The report is automatically updated based on the
OF results
Louay: Technical
requirements. We support all OSs: Linux, Windows, Mac. We recommend
Linux as it has native Docker support, it's the fastest and easiest
way to deploy
… Docker Desktop means additional steps, but that's
documented
… Need TLS server certs, it's important for EME
tests, which requires HTTPS
… The camera for recording, to analyse 50 or 60Hz,
you need double the frame rate, so 120Hz or higher
… We use Samsung S23+ smartphones, happy with the
results
Louay: The test report
is the same format as WPT
… OF shows additional assertions added to the test
results. It shows the reason for any test failures
Louay: We did
validation in our lab, and in HbbTV plugfests. We tested on the
latest TVs coming into the market in the next year
… At DTG in London, in Fraunhofer, next event in
October
Louay: HbbTV test
setup. At plugfests we take a DVB modulator, where we configured a
transport stream with HbbTV app pointing to the test suite landing
page
… HbbTV developers will be famiilar with
this
Louay: Testing on many
devices. For each result session, we have tools to get results
across test runs, summarise the information into a single
table
… Easy to see which tests are performing well, and
which need checking
Louay: Thank you. Happy to take questions
Discussion
Alicia: You mentioned MPEG-TS, that part is separate from MSE testing? You have tests for broadcast applications?
Louay: This part is just to launch the landing page on the TV. Tests only rely on MSE and EME, no broadcast dependency
<jpiesing> I run the tests on a mobile phone when exercising them.
Louay: If you have a smart TV app, you could have another way to launch the landing page
Alicia: What protocol is used to get the tests into the TV?
Louay: The test runner uses a REST API to the test server, HTTP. When the test runs on the TV, results reported locally, then results sent over the REST API to the server
<jpiesing> Some SmartTVs allow URLs to be pushed to the TV using a home network protocol like DIAL. That can also be used to push the URL of the test runner to the TV.
Alicia: Does the DVB modulator just send the URL?
Louay: Yes
Wolfgang: You said you run the test suite at the HbbTV plugfest. Any area where you find failures, any hot spots?
Louay: Focusing on the
test runner and process for running tests, it's stable, no issues
on TVs we tested
… But a group of tests, initial frames skipped or
not displayed in the video, something we've observerd
… Working on audio observations. One challenge is
getting the audio from the TV. We use line-out, but not supported
on all TVs
… Now use HDMI for capture
Piers: Have you tried using DIAL for launching the app?
Louay: You can also use
DIAL, but we use the DVB modulator, as in the plug fests we run
other tests as well
… If you have control on the local network. There
might be limitations on device discovery at plugfests
… So we rely on having our own setup
Piers: Is the media configurable, whether it comes from the CTA media streaming site, or can you host locally?
Louay: We try to avoid
dependencies on the open internet. Streams are available on a CDN,
but we download them and run locally
… It's important when you deploy, the device under
test and test runner should be as close as possible. We're not
testing content delivery over the internet. We want to avoid
dependencies on network conditions
… Can be a server in your orgsisation. We have a
dedicated server for the TVs in the lab. You could use a
laptop.
… One challenge is having a TLS cert, for JS
tests
… and a domain name
Chris: I found some
limitations with the WPT tests, e.g., EME persistent usage record
tests are still there despite the feature being dropped by Media
WG.
… Also the Media Playback Quality tests only test
for the API surface being present, not the functionality. Any
suggestions for improving WPT?
Louay: Can't use JS to
check the results, e.g., video plays but is delayed, you don't get
this from the API. So you need an external system to check the
playback starts from a given event
… It's something we solved with the test framework
and mezzanine content, and external observations. Things you can't
capture via APIs
… Needs an external system to give more
detail
… How to do more automatically? Currently need to
copy file and upload to a server, could make a fully automated
pipeline, but that needs additional resources
Chris: Any specific tests for VideoPlaybackQuality, e.g., dropped frames?
Louay: Not specifically, but the QR code gives you info about quality level with MSE, but analysing that it's displaying as it should, e.g., when upscaling is applied. Could be a future discussion, we just make sure the bitrate representation is displayed now from the info in the QR code
Alicia: I have this
problem about the audio, writing WPT, but didn't work well. I put
in single tones, to seeif the audio changed. But then you also
depend on Web Audio API, but the test ended up flaky
… Not sure if it was an MSE but or with Web Audio.
You could do something similar, where yuo can make beeps that are
easy to analyse with FFT
Louay: We rely on external framework to measure those things
Jon: The audio is a pattern based on pseudo-random noise. The OF can analyse the pattern and confirm it's the audio that's expected
Alicia: That could test
if it's not in sync
… Also about the domain for TLS certs. Is that
configurable?
Louay: You can configure it, but need to create your own domain
Alicia: That's very cool
Chris: Lots of value here. WPT focuses on automatic execution, API interactions, things not needing human review
Louay: Some need user interaction, which can break the automation of the tests
Chris: Improvments in WPTs or web APIs
Thomas: It's about
dissemination of the information. Make a bit of a promotion. We're
open to collecting feedback, e..g, do we need new tests? We focused
on basic codecs, so look at more advanced codecs?
… Things could be stimulate. Needs to people to
understand the value. Any W3C platforms we can use for
dissemination, including demos
Alicia: I can use something like this. We've been trying to get more tests running WPE. There's a need for tooling. Personally, I'll advocate to have people look at this in my company
ChrisL: The BBC Tech
Meetup could be another way to promote: https://
Chris: There's also the video-dev Slack
Kaz: Could have a TPAC breakout session?
Louay: I'll try to join remotely, but happy to support
Chris: Look forward to future meetings to continue the discussion
[adjourned]