<Travis> .. joining the meeting in progress

<Travis> SubTopic: Support for certs not anchored to WEb PKI (Self-signed)

<Travis> ..

<Travis> SubTopic: provide a way to specify a subprotocol

<Travis> It's not trival to find a place to put this in the protocol.

<Travis> Thought: put in first string, but not block data.

<Travis> Could do this without losing RTT

<Travis> But don't want to add extra complexity

<Travis> Two other issues not covered: conjestion control and stream prioritization

Topic is now WebCodecs

Peter Thatcher: does anyone want an intro?

<Travis> Scribe: jbroman

(response is yes)

low-level API for media encode/decode

based on transform streams

useful for live streaming, faster-than-realtime encoding, realtime comms, ...

VideoEncoder takes image data and makes a bitstream

typically gotten from a MediaStreamTrack via VideoTrackReader

not strictly part of web codecs but pulled in as a dependency

slide example of constructing audio/video decoders from some readable streams

and plugging that into a <video> element

slide example of using this to transcode media by piping between decoder and encoder

slide comparing to MediaRecoder: more low-level, ...

similar comparison to MSE

comparison to WebRTC: more low-level, not tied to network transport, can choose what kind of transport you want

buffering is done by the application, not browser

all of this can be done by wasm but this is faster and more power-efficient

access to hw encode/decode

track reader/writer would be useful for wasm codecs too, avoids having to use <canvas>

proposal, explainer and webidl but no spec

impl starting soon in chromium

<Zakim> Travis, you wanted to ask about codec discovery if not covered in the presentation...

Travis: can you enumerate supported codecs?

Peter Thatcher: that's the first issue under capabilities

we want to use media capabilities; don't want to duplicate it

there are specific needs to add to capabilities

e.g. low latency vs b-frames

a lot of people want to explicitly choose hw or sw encoder, since some hw encoders are bonkers

for low-level things like svc, there is no support inside MediaRecorder or MSE but we want it

concerned about expanding the fingerprinting surface by telling more to the app

Travis: have you approached them?

Peter Thatcher: yes, and they're engaging

Travis: agree with general sentiment

<Zakim> yoav, you wanted to comment on images

yoav: you mentioned image encode/decode; I suspect this will be an ask

what is the current status?

Peter Thatcher: there was a lengthy discussion; even made a pr

the more we dug into the use cases people wanted, it became unclear whether it was worth it vs wasm or having a one-frame video

currently consider it out of scope

it occurred after a <img>/<video> discussion yesterday that if we could take decoder output and put it in an <img> that would be good

yoav: I think img/video discussion is different because if app developers are involved, they can just have a muted <video> in all cases except background-image

in background-image case this solution wouldn't help anyway

I don't think that covers the use cases people have when they want to put videos into <img> elements

you're probably correct that people can do things with one-frame videos

at least in image formats today

there might be an image format in the future that is not 100% identical to video format even if inspired

why does it matter if it's one-frame video or image format?

Peter Thatcher: one interesting case is gifs

somebody wanted to encode (animated) gifs

yoav: as an encoding output?

Peter Thatcher: yes

would we call that a video?

there is no containerization right now

is there such a thing as a containerized gif?

images always have containerization; for video it's nice to have bistream w/o container

yoav: ok

Steve Anton: in summary it seemed like scope creep

can do image encode/decode with current apis

yoav: would be interested to hear the use case for gif; do we really want more animated gifs?

???: lots of internet traffic is memes

Travis: drill into answer about existing way to do image encode

Steve Anton: you can write it to a canvas

Travis: want to be able to bring image payload down from server and decode it later, and recapture the memory later if it goes out of view

want to manage that at a fine-grained level

rbyers: didn't we recently ship something like that (jsbell?)
... HTMLImageElement.decode; forces eager decode

Travis: can you provide user-defined codec?

Peter Thatcher: yes, injectable codecs

can use simd extensions to wasm etc, but need to get it into/out efficiently

that's where track reader/writer comes into play

we discussed this at a breakout session

want to allow injectable codecs in WebRTC API

Travis: can this be done off-thread?

Peter Thatcher: should be possible, in a worklet

Steve Anton: Web Codecs is more about exposing the platform codecs

though there is interest in allowing injectable codecs in MSE, etc

it is more relevant to those APIS than to Web Codecs

Amit Hilbuch: don't you want it to work the same way as web codecs?

Peter Thatcher: similar API shape, could use Web Codecs or custom codecs

Steve Anton: could imagine constructing a codec on a worker, using it on main thread, etc

Peter Thatcher: Next issue: some codecs require extra out-of-band data, like H.264 SPS/PPS

???: isn't this usually embedded in the container?

Steve Anton: could be, yes

Peter Thatcher: usually extracted from container format and then passed to decoder

(example shows it passed as options to the VideoDecoder ctor

)

Steve Anton: this is data necessary for decoder itself

Amit Hilbuch: should these be specified in IDL or be freeform?

Peter Thatcher: in WebRTC we have key-value pairs; would probably mimic that

Steve Anton: settings are codec-specific

Amit Hilbuch: agree, wouldn't want to bake specific codec parameters into web standard

Steve Anton: is it fine to specify statically at decoder construction, or would they need to change later on, e.g. per-chunk

Amit Hilbuch: how are these different from ???

Steve Anton: distinction between things needed to understand bitstream vs encoder hints

Steve Anton: are there parameters needed to understand the bitstream that are not known when the decoder is constructed?

need to consult codec experts

Peter Thatcher: next issue: decoding b-frames

in video codecs there are dependencies between frames

b-frames depend on future frames

need to decode out of order

how do we support this in the decoder api?

yoav: buffering?

Peter Thatcher: require JS to handle it by getting it in decode order; JS can reorder after

but hardware decoders reorder internally

Steve Anton: feels like most decoders could efficiently reorder for you

even sw decoders

Peter Thatcher: do we make the API such that decoders can do it more efifciently?

so far we've added a timestamp to the info going into the decoder

so that it can do the buffering

but if you opt into low latency

then this timestamp would be unimportant

Steve Anton: requiring timestamps doesn't seem too onerous if you don't need them

can just increment by 1

Travis: seems like surrogate pairs

where it's useless to get half of it

almost always just want full character in e.g. HTML parser

is the b-frame an implementation detail, or does it need to be exposed?

Steve Anton: wouldn't get partial images; just get frames out of order

would get keyframe and then partial frames that come before

would the decoder expose the keyframe or wait until after the delta frames have been emitted

yoav: I would ask developers (a) is there an actual use case to get the frames out of order? (b) what is their preference? do they want the API to magically handle that or not?

Peter Thatcher: that's a good idea to ask on discourse

pretty sure I know what they will say

"do you want to do extra work or do you want us to?"

yoav: would it be more efficient in some way if it's done in js? doesn't seem so

???: which timestamp?

Peter Thatcher: presentation timestamp

Peter Thatcher: next issue: complex coding modes

e.g. SVC

SVC is Scalable Video Coding

progressive images, but for video

different "layers" that depend on each other

useful in real-time communication, e.g. video conferencing

surprised it hasn't been popular in livestreaming

Steve Anton: also useful in offline playback

Peter Thatcher: with only a portion of the bitstream, can decode a lower-quality version

yoav: how does that work wrt video length?

is it "for every x frames"?

(overlapped discussion about how this is implemented, and complicated diagrams in a spec)

Peter Thatcher: simple case has bigger frames dependent on smaller frames

yoav: okay, so each frame is progressive

Peter Thatcher: but it can get more complicated

Steve Anton: SVC is similar to JPEG progressive decode

Peter Thatcher: from encoder API standpoint, put in 1 frame and get out n frames for same timestamp

dependencies between them

similar to time dependencies in normal delta frames

but this impacts the api

when decoding, get different frames for same timestamp

maybe you get a higher-quality frame after having rendered the lower-quality frame

1 frame in - 1 frame out no longer applies

have tried to integrate into the api

but expressing how you want it set up can be complex

expressing what is possible through media capabilities is also complex

yoav: is there a preference

e.g. I'm a low-power device, only want partial layers?

but client and server could have communicated that

Peter Thatcher: in real-time comms, client asks for higher res version that is available according to network conditions

yoav: device could say max quality it can handle

Amit: that's in signaling

yoav: maybe in media capabilities you want to know what's possible

Peter Thatcher: might have a limit in how many layers you can do

not just for video

in audio, opus fec

particular packet can contain current time's high quality audio and previuos time's low quality audio

so if you miss a packet you can fill in with the low quality version

so from an opus bitstream you can decode either high or low quality data

Steve Anton: might be able to figure it out automatically

based on gaps in timestamps

Peter Thatcher: good point

could treat it like SVC

if you get the earlier one first and high quality one later, get a second audio frame at the same time

Steve Anton: at encode side, just need a way to turn it on

Peter Thatcher: agreed that it's easy

decode is what's interesting

Steve Anton: with SVC, multiple layers make up one frame

if you expect to get all the layers you want to wait

might need a way to flush what's currently available

yoav: how can we facilitate this in the futuree?

Peter Thatcher: might not want to implement all of this now

but don't want to paint ourselves into a corner

Steve Anton: can't assume frame in, frame out

yoav: at the same time, a future application using those coding modes will be aware of it

they will know that multiple frames can come out, etc

Peter Thatcher: skipping Timestamps and time domains because Paul isn't here

short version: when we go from readable streams to tracks, for audio want to give it to OS as needed; for video to keep it smooth you want to say when a video frame is presented

if you just take frames from decoder as fast as they come out, it might not be smooth

???: might need to drop frames to keep smooth?

Peter Thatcher: how do you decide which frame goes out on a particular vsync?

if we put JS in complete control, prone to JS pauses

Steve Anton: concern already exists e.g. with WebGL

so might not be intractable

Peter Thatcher: might not be clear until we try to implement

Paul has better ideas

Todd: expected to be done off render thread?

Peter Thatcher: yes; still may want to control what frame goes out on what vsync

Todd: sounds like it might be more efficient

Peter Thatcher: if latency isn't as important, buffering to provide smoothness might be more important

vs real-time use cases

<Travis> Thanks jbroman !

information pipeline between WICG and web developers

yoav: don't know if this is interesting to non-chairs

Todd: what is the core of the question?

yoav: I think discourse today is mostly used as a "bring us your proposal" forum

not the most effective way to get stuff done

would be interesting to use it more, or use another means more, to get use cases from developers

and then iterate over API shape questions

Todd: when you say use cases, you mean problems that are difficult or impossible to solve?

and exactly what are you trying to do?

rather than "I dreamed of this API shape"

yoav: is that worthwhile to discuss in this forum?

cwilso: is discourse the right entry point to wicg, or would something else make more sense? I drew the parallel in the AC meeting; there is 1:1 overlap between ???. The way we start things up is similar to WICG. We ask to see evidence that >1 person cares about this problem to give it legitimacy. Immersive Web Group has specific "proposals" repo. File an issue; everyone follows this repo. Sometimes we still have competing repos about interesting

ideas. That's how we do it; we barely use our mailing list. Would that be easier than using discourse?

I don't go to discourse often

Travis: but you do go to github often

cwilso: I do, but don't necessarily dig into the right repo every day

Todd: how to keep web developers engaged, understand what are the issues?

cwilso: process for starting in WICG is intended to be easy but still have a minimum bar

we do ask that every incubation proposal has >1 organization

if 15 Google employees say it's a good idea, that's not good enough

but 1 web dev + 1 MS employee, that's good enough

some measure of cross-industry interest

are we using the right mechanism for that?

yoav: there's the question of engagement, and beyond that, once you've got people engaged, what do you actually want from them?

I think that I traditionally try to get web devs more involved in standards, but it may have been a mistake in the sense that people don't necessarily have time to do that on top of their job

at the same time, an engagement that says "here is a problem space we're looking at; give us your use cases and pain points"

and they just have to write 100 words and leave and don't have to follow up as a long-term commitment

that is something that developers are actually likely to be interested in doing

if we make it easy/welcoming enough

Travis: I wonder how much a developer feels like they get to be a part of it if they just pop in, make a suggestion, and never come up

what's missing is the follow-up

that we turned your random idea into something complete

wdyt? does it still work?

How can we make sure that's working unless we keep them engaged?

yoav: discourse, github

once we have some API shape, we can ask them if it would address their use case or not

we can try to have long-term engagement in terms of revalidating use cases, API shape, etc over time

until shipped

but doesn't require them to do a ton of work

Travis: could be done through discourse or github, orthogonal to the question?

yoav: not a tooling question

do we want to take that approach? can pick a tool after that

chairs, does that make sense as a browser developer?

Steve Anton: web developers have a hard time getting excited about proposals

they seem so far off

first step is use cases

devs happy to talk about use cases

there's a gap where it's hard to get them involved in middle stages

until origin trial

cwilso: early part of "Web dev has a use case" is outside of this

more like "web we want" effort

let's turn early use cases into actionable problem statements for further incubation

so many people come up with problems

but thinking about how to solve in a broader space is hard sometimes

I'm more concerned about having an active set of people, how do we get more people engaged in solution efforts?

yoav: define "engage in solution efforts"

cwilso: iterating on problem statements, solutions in incubation process

a number of incubations had people saying it was interesting, turns into one person creating, proposing, implementing, experimenting w/o a lot of engagement

that's tough

not saying that's a totally broken pattern but shouldn't be the usual pattern

Todd: if there are 5 people on discourse saying it's a good idea and one person begins work, how do all 5 people contribute?

cwilso: even continually engaging after that

one downside of discourse is that it's for initial stages only

once there's a repo, it's disconnected from discourse

conversation moves into a different tool altogether

that seems odd

Peter Thatcher: with Web Transport, had some people interested; wanted to have community discussion

wasn't sure whether to use discourse, github, mailing list, ?

lack of a way to keep interested people in the loop

Todd: does an individual effort have a single discourse thread?

cwilso: usually one, but once repo created there's no conversation there

Todd: a bot to update discourse thread with # issues created etc?

yoav: there are also labels

theoretically you could have multiple threads and have them labelled

Todd: do we think the discourse thread don't go to the github

or do they not know? not have time?

do we know why/

cwilso: we don't truly know why

part of it is that it's a barrier that we need to see engagement to create the repo

engagement beyond that is a long-term interest thing

human nature to have limited time/effort to expend

superstar is co-chair Marco (sp?) who comments on everything which is great

suspect if we used GitHub as that comm tool it might be easier to get interest from other people

we could manage tagging/labelling things in the proposals repo

so people could follow e.g. styling or networking proposals

Todd: replacing discourse with github?

cwilso: yeah

Peter Thatcher: when people were interested in Web Transport/Codecs, had to tell people to come to discourse page and people weren't familiar

maybe GitHub would be better

yoav: one way to resolve question would be to ask it

do a survey

provide multiple engagement mechanisms and ask that they would prefer

discourse vs github vs ??? with examples

Peter Thatcher: have you talked to FOMS: http://www.foms-workshop.org/foms2019SF (sp?); they have engagement from web devs about media stuff

whatever they're doing seems to do a good job

Todd: web performance wg has attended velocity conference (now renamed)

that did get some engagement

yoav: view source next week would get a good place to collect anecdata

cwilso: we should wrap up

<Travis> RRSAgent: make minutes

This is scribe.perl Revision: 1.154  of Date: 2018/09/25 16:35:56  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)

Succeeded: s/Amit/Amit Hilbuch/
Succeeded: s/disocurse/discourse/
Succeeded: s/foms/FOMS: http://www.foms-workshop.org/foms2019SF/

WARNING: No "Present: ... " found!
You can indicate people for the Present list like this:
        <dbooth> Present: dbooth jonathan mary
        <dbooth> Present+ amy
        <amy> Present+

Found Scribe: jbroman
Inferring ScribeNick: jbroman

WARNING: No meeting title found!
You should specify the meeting title like this:
<dbooth> Meeting: Weekly Baking Club Meeting


WARNING: No meeting chair found!
You should specify the meeting chair like this:
<dbooth> Chair: dbooth


WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.


WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)