Meeting minutes
Welcome, agenda bashing
anssik: Welcome! New co-chair for this group, Louay, who has been involved since the beginning.
… A couple of lightning talk speakers invited today.
anssik: Michiel, Hyojin, Pavlo.
… We will start with the Open Screen Protocol, continue with the Presentation API, Remote Playback API.
… That should last one hour, then break, then presentations
Open Screen Protocol
anssik: Do we have a label for editorial issues blocking publication as FPWD?
mfoltzgoogle: No. I was thinking of a meta-issue with checkboxes.
<anssik> [meta] Publish First Public Working Draft
mfoltzgoogle: Let me take a quick step back and describe where we are
… The Open Screen Protocol was an incubation in the CG for a couple of years. Before TPAC, we agreed to move it to the WG as a deliverable.
… Anssi proposed to prepare it for publication as First Public Working Draft.
… I'm fine with it, I'm just trying to bring in resolutions that we took but are not yet reflected in the draft.
… There are some outstanding issues, but my understanding is that it's ok to have some for publication as FPWD.
anssik: That's correct.
… My take is that the spec far exceeds the expectations for FPWD.
… One option could be to link to issues within the spec.
<anssik> Editorial issues for OSP FPWD
mfoltzgoogle: I would like that any issue that have been identified as v1 issue should be linked to the document.
… I don't anticipate it will take very substantial changes to address editorial issues.
… I would then be fine publishing the spec before the outstanding issues are addressed.
anssik: You'll likely get more eyeballs, and potentially contributors when you publish. From that perspective, I would like to publish earlier rather than later.
mfoltzgoogle: Let's have a two-week deadline to address the editorial issues.
anssik: Question to people, do you think the spec should be advertized to groups such as Media WG, WebRTC WG, Media & Entertainment IG?
Louay: Yes, I think that would be valuable.
mfoltzgoogle: I think these groups should be interested by the spec indeed.
… Worthwhile to do, I'll just want to check that the introduction sets the context correctly.
anssik: What about the TAG review? Not done yet, I believe
kenchris: Whenever you have something to share, the TAG would be happy to review.
anssik: Along with the FPWD, let's aim at TAG review as well.
mfoltzgoogle: Again, for TAG review, I want to make sure that there's enough context around it. The way this came along, there was no explainer process.
… I just want to make sure that there is a little more context to this spec, maybe the introduction.
… So that people can understand why certain choices were done this way.
kenchris: If you write a small explainer, that's the right place to do that
mfoltzgoogle: Yes, I'd like to take a stab at it
Louay: I believe there's a pull request with a Draft of Open Screen Protocol explainer document #168.
mfoltzgoogle: OK, action me to take a look and adjust that, then
Action: mfoltzgoogle to adjust and merge explainer document for the Open Screen Protocol
mfoltzgoogle: 6 main issues remaining. First is Make a required/default remote playback state table #148, how do we synchronize the state from one agent with another.
… It's just making a table to make it more readable. More reformatting than anything else.
mfoltzgoogle: Second is Meaning of receives-audio and receives-video capabilities #200: when an agent talks to another agent and wants to tell video capabilities.
… Again, I think that's just a clarification
mfoltzgoogle: Do agents need to support the Cookies extension (used in HelloRetryRequest)? #219 and Determine if there is an advantage to session resumption outside of early data #220: agent makes a connection to another agent using TLS, there is some discussion in the spec about what TLS features can be used. Feedback we got was to go with default and remove those parts of the spec which overspecifies things.
… And also that does not account to some allowed mechanisms.
mfoltzgoogle: mv value needs encoding specified #239: I think the spec talks about that, we just need to add wording
… Use the same codec names as the media APIs #233: This is where we allow an agent to describe its decoding capabibilities. What values it may use. Some discussion about reusing same codec names as in other media APIs. If it's a simple change, I may do that right away. If it's more involved, I'll address that later.
… Those are the editorial issues to address.
anssik: Any thoughts on these issues?
… In general, if they are quick to address, we should do it within the next couple of weeks. Otherwise, it's fine to leave them open and address later after publication as FPWD.
Louay: Do you need any support from people in the group?
… e.g. for media capabilities for #233?
mfoltzgoogle: If you are aware of the current way that Media Capabilities addresses codecs and can share that, that would certainly be useful.
… If that looks like further investigation needed, no need to worry about that right away.
Louay: Chris, would you be able to help, perhaps?
cpn: I don't know the specifics on this. I would recommend to talk with Chris Cunningham, as editor of the spec.
mfoltzgoogle: Right. The codec string can contain a lot of information depending on how detailed you want it to be. I may not be able to figure out the right trade off there.
… H.264 is not quite specific enough to tell another device how it should encode a stream.
anssik: Right, we should figure out what is the state of the art.
mfoltzgoogle: Web specs tend not to define these strings very concretely, but there are IETF documents that define them for codec families. We may be able to refer to those.
cpn: Media Capabilities refer to WHATWG MIME Sniffing spec as far as I can tell.
tidoust: It may be worth looking at WebRTC. I believe it does not use the same codec strings as Media Capabilities, it would be good to investigate.
Louay: Also, canPlayType has some specific details on codecs, used in MSE to create a SourceBuffer.
anssik: I'm hearing that this may take some time, so I think that we should not block on this issue.
mfoltzgoogle: That's fine with me.
anssik: About remaining issues?
mfoltzgoogle: The two open pull requests are waiting for feedback from the group. Not blocking, but feedback is welcome.
<anssik> Reference CSS Color Module Level 4 for color spaces PR #225
mfoltzgoogle: First one is about advertising what color spaces an agent supports. We were waiting for the discussion on Media Capabilities to be resolved. That should be now done. I updated the PR as a result.
… How to advertise the color gamut encoded in the media that it can receive.
… I invite feedback on the pull request to see if that's the proper way.
cpn: Are you interested purely in color space or also dynamic range?
mfoltzgoogle: There's a different open item to also enable a device to query the dynamic range and the HDR decoding capabilities.
… With a combination of the three, that should address all needs. Does that make sense?
cpn: That makes sense.
<anssik> Adds a backpressure signal to the streaming protocol PR #248
mfoltzgoogle: The second open PR has been open for a while. This is when one agent is streaming media to another, for remoting case. How do we allow the receiver to tell the sender to adjust its send rate?
… This PR adds three value enums to the remoting state, from the receiver to the sender: enough data, too much data, not enough data.
… That seems to be the simplest way to introduce a backpressure mechanism.
… That probably would benefit from implementation experience. We may revisit that in the future based on that, but the PR should be good enough in the meantime.
anssik: Specific folks in mind for review?
mfoltzgoogle: For the first one, Chris or other Media WG folks could have the right feedback. I wasn't involved in discussions around color gamut. We would benefit from having people with expertise there have a look at it.
… On the second one, I'm not sure, I'll have to go back to previous discussions and see who was involved.
cpn: I asked my colleague Nigel for review of the PAKE algorithm. We have some feedback on that. He seems to say that the CPACE algorithm is more efficient than the OPAQUE one.
… Security proofs and efficiency guarantees.
… I don't know how it compares with the other ones we've already reviewed. That seems to be the outcome from the IETF groups though.
… I can post something under #242.
mfoltzgoogle: Yes, that's the right issue to track this discussion. Feedback there would be great.
Presentation API
<anssik> Possibility for UA to close the connection when it is no longer operational #485
mfoltzgoogle: This was an issue raised by Francois about whether we should have the user agent try to detect the liveness of a presentation connection. I presented some of the reasons why we allow applications to make their own decisions.
<anssik> connected definition
mfoltzgoogle: Francois seems to agree with me and suggests that we add a note to the spec to clarify that "connected" gives no guarantees that message delivery will effectively work.
… Provided that does not change the meaning of "connected", that's good to me.
tidoust: I'll prepare a PR to add an informative note for developers.
anssik: Daniel is not around today to discuss the same-screen presentation issues. Defer to tomorrow?
mfoltzgoogle: Sounds good to me.
<anssik> Add support for allowing same-screen display PR #479
mfoltzgoogle: I think that the use case that is being proposed is being able to open a new window (window.open) and being to be able to place it fullscreen on the screen display as the current content.
… Currently, it can work with existing APIs but requires a couple of clicks: to open the window and make it fullscreen. They are wondering whether the Presentation API could be used for this.
… Strictly speaking, it could. However, we discussed a little bit the overlap with Window Placement API. Daniel said he would loop with the right folks.
anssik: Mike, did you coordinate with Daniel?
Mike: I think I did. I think they wanted to use a single API for local/remote use cases.
anssik: Let's see with Daniel tomorrow when he joins.
mfoltzgoogle: No other issue on the Presentation API that I think would benefit from the group's input.
<anssik> Presentation API W3C Candidate Recommendation 05 November 2020
<anssik> Changes since 01 June 2017
anssik: Small announcement that we published a Candidate Recommendation Draft of the Presentation API since last TPAC
mfoltzgoogle: Implementation-wise, we had a serious bug that we've hopefully now fixed.
anssik: Good thing to have feedback from users, it shows that the API is being used!
mfoltzgoogle: I'm trying to make sure that test coverage is improved. It was supposed to be a non-functional change in Chromium, and it wasn't.
Remote Playback API
anssik: Just one issue on test automation.
… I would like to check whether we have people who can take a look at it.
<anssik> Remote Playback API test automation #92
anssik: Louay, you've looked into it in the past. Do you potentially have students or ideas?
Louay: It was too late at TPAC for the semester. Should be doable for upcoming semester in April.
… Question about the level of automation.
<anssik> WebDriver Extension API for Remote Playback proposal
Louay: Best would be to use WebDriver, but that requires extended WebDrive protocols.
… If we follow the same approach as for the Presentation API, most of the tests would be manual tests.
… I would start with manual, then look at automation.
… I believe automation would take more time.
<anssik> [meta] Publish Proposed Recommendation #130
mfoltzgoogle: I'm not aware of any work that is being done in Chrome to support this at this point. I can ping Mounir. As far as I know, there aren't well-defined automation hooks to automate testing of this API at this point.
<anssik> Remote Playback API: All Results
mfoltzgoogle: We have internal tests, but they are not Web Platform Tests.
Louay: I would say that we can take a decision by the end of March on the automated/manual direction.
… Then I can get support from students in April. Support from other people in the group is of course also welcome.
tidoust: No specific requirements from a W3C perspective on test automation. Better for implementers obviously. We need an implementation report.
Louay: I note the observation framework, developed in CTA WAVE, to run tests and observe the results, e.g. for video remoting. If, in the future, it makes sense to make sure that the video is really playing at the appropriate resolution, we could look into it.
Deskreen intro
Deskreen - turn any device into a secondary screen for your computer
Paul: Project turns any device with a web browser into a second screen.
… 65K downloads as of today. Pretty good success so far!
… For second screen, an HDMI plug is needed, lots of discussion on that, different os-es have different approaches and hacks there.
… Deskreen can be used for a number of features, including teleprompter on any device. It works with WiFi and can support multiple connected devices at the same time.
… It has advanced video quality, reducing the video quality when needed.
Paul: I implemented end-to-end encryption. It uses WebRTC.
… Depending on WiFi speed, it can be fast. Maybe not as fast as a commercial solution for now but I'm looking at ways to improve performances.
Paul: It really helps Linux users who did not have many options in that field.
… How Deskreen works: home LAN or WiFi networks, no Internet access needed.
… Main programming language is TypeScript. Computer app is written using ElectronJS and ReactJS.
… Client viewer is implemented using ReactJS.
… System requirements: Windows, Linux, MacOS. 210MB of disk space (due to use of Electron.js).
… 250MB of RAM. New screen session can take up to 100MB of extra memory.
… Regarding use of CPU, sluggish performances on Windows, not on MacOS and Linux, probably due to the way Electron runs on Windows.
… Deskreen limitations: slide links to benchmarks.
… Key things for performance: lower virtual display resolutions. Video auto quality change mode gives better performance while watching video.
… Mouse pointer movements can be a pain, and may require restarting the WebRTC connection.
Paul: More powerful hardware gives better performance. Sluggish on small and old machines for now.
… Some missing limitations collected from Reddit and GitHub:
… - notion of trusted devices to auto re-connect
… - support for IPv6
… - ability to change network cards
… - autostart on system login
… - system tray functionality
… - audio sharing
… Deskreen is built with Electron.js. Some bugs are due to Electron or to underlying browser rendering engines.
… The TODO list includes missing features I mentioned.
… Questions for this group: Anssi and Louay suggested that Presentation API and Open Screen Protocol could be integrated, I wonder how.
anssik: Very interesting project. Lots of demand for this, given the excitment and attention in the community.
… That means you found the sweet spot of requirements.
… Mark, any advise from your experience in that space?
mfoltzgoogle: This really sounds like a WebRTC-based application. That certainly provides the ability to send video streams from Web applications. If there were a way to discover screen points, they could use that to bootstrap the connection with WebRTC.
… We talked about the possibility for applications to add their own endpoints in the past.
… That is one avenue where we could offer support.
… Right now, we did not really have the use cases for that, but that could be one.
anssik: How did you come up with this idea?
Paul: I wanted to solve it for me. I had an old Mac book, iPad Pro. Solutions performed bad.
… First generation of these devices don't support WebRTC though.
… There are many solutions to create virtual displays even without dongles, check GitHub discussions. That involves tweaking system settings though.
anssik: If you'd like to be engaged, we could consider having you as an Invited Expert in this group.
… I encourage people here to have a look at the intro video.
… Thanks for joining us. This is amazing!
Michiel: I worked on a native implementation of WebRTC in the past Pion-WebRTC. If you ever want to have full control and switch to a native solution, you may want to look at that.
Second Screen use cases in webOS
Second screen use cases in webOS
hyojin: I'm working in LG Electronics. I've been involved in standardization work for a decade, but cannot be directly engaged these days unfortunately.
… The LG webOS platform has been used in different products. There is an open source edition. In this talk, I want to focus on TV.
… [showing Home Dashboard in LG TV]
… Several connectivity features are running for providing services pictured here.
… According to our statistics, local network is widely used. In my experience, if I buy a washing machine, the TV can display a "laundry done" notification.
… The LG ThinQ app may be used on one's smartphone to control the TV.
… In webOS, there is a second screen gateway.
… SSDP and WebSocket or SSAP (Simple Service Access Protocol) developed in LG for internal usage.
… Through SSAP, a controller can send commands to the gateway.
… LG controls both sides of the communication.
… But we have requirements for an open source set of protocols for other usages.
… Some possible use cases (not official ones):
… - remote control for TV apps using a mobile app.
… - Promote TV app from mobile web apps
… - Third and fourth use cases around web app/media beaming
anssik: Thank you for the presentation
Louay: Two features already implemented in this group: Web app/media beaming mapping to Presentation API and Remote Playback API.
… SSAP, is that something defined on your side? Never heard of that before.
hyojin: Right, SSAP is proprietary LG protocol.
Louay: Do you think that the Open Screen Protocol provide a useful solution to replace this in the future?
hyojin: Yes.
Louay: This is great. I hope that LG in the future can participate in the group and provide feedback on the protocol, at least from a receiver perspective.
hyojin: Priority is low in webOS for this for the time being.
… If we could expand our web ecosystem for WebOS, we could embrace these APIs. Currently, the webOS ecosystem is not quite large.
anssik: I understand, product decision.
Local Devices Protocol - Building Blocks for the Local Web
Local Devices Protocol - Presentation slides
backkem: My project goes beyond what this group is trying to achieve, not restricted to screens.
backkem: The LAN is not a first-class citizen of the web world.
… You'll ever connect through HTTP or self-signed SSL certificates, and the browser will tell you that you're essentially doing something wrong.
… Lots of use cases where local devices currently require cloud access, which seems silly.
… Lots of tricks. IoT is a bit more complicated because there are so many protocols to deal with.
… For LAN WebRTC signaling, one HTTP server running on one machine. But again, that means that you cannot use HTTPS easily due to the use of local IP addresses.
… People have tried addressing that before:
… Network Service Discovery API, FlyWeb, Raw-sockets, TCP and UDP sockets.
… Most of these projects have stalled.
… Security issues, e.g. breaking CORS if you give access to the TCP/UDP level.
… These things never land in practice.
… Concept: establish trust by pairing à la Chromecast with a pairing code.
… Without the pairing step, you can't avoid MITM attacks.
… Some of the design goals:
… LAN only, offline first (no server/cloud required), security first, user friendly (no red check mark in the browser).
… Tried to favor low level building blocks over all-inclusive higher level APIs, reusing existing protocols as much as practical.
… What does it look like: same basis as Open Screen Protocol
… mDNS discovery, minimal messaging protocol. JSON messaging could be layered on top of this.
… Some comments on trusted devices: allows you to skip pairing step in the future. If we exchange self-signed certificates between agents, they could perhaps be considered for other usages (e.g. WebSockets, QUICTransport, etc.)
… Pairing providing the trust.
… Management of the trusted devices would be handled by the UA.
… Access would be granted per domain.
… I came up with a JS API similar to the MediaDevices API to discover devices on the LAN, filtering what devices you need.
… Enumeration of devices would require user permission, obviously.
… For the messaging part of it, I thought I'd leverage the WebTransport API. Two ways to create it:
… 1. giving it the device instance you'd get through discovery.
… 2. or because the certificate would be in the certificate store, you could use the address directly.
… Optional: the browser could act as a virtual device on the network, for LAN games.
… Fingerprinting is one of the security risks that I see here: If it's never unsolicited, then that should be mitigated.
… Next steps: scrutinize the design & security concerns.
… And find community support base for the idea.
mfoltzgoogle: One of the main concerns that makes it difficult to expose local devices directly to the Web is that the argument that the Web gives access to a lot of services that were not thought about for local devices. Example: UPnP access, drive by web could do terrible things such as read/wipe things out on your NAS storage.
backkem: Pairing would be a mitigation measure there.
… That will help the users select which devices can be discovered directly.
… Plus user consent will always be required.
… I don't see how you could run into drive by issues.
mfoltzgoogle: The trust is between an origin and a device.
backkem: You establish a trust between a browser and a device, and then user gives trust to a domain.
… I use the same thing as getUserMedia to filter by name. A device has an ID which you could store locally to re-connect to the exact same device directly.
… It may be something that the UA does, not an ID that the device exposes.
mfoltzgoogle: It sounds that this was inspired in part by OSP. Two parts to this spec: discovery and negotiation of capabilities, and then implementation of specific APIs (Presentation API / Remote Playback API)
backkem: I'm proposing to reuse directly the entire first part.
… Then have a minimal application protocol, and the rest will be at the application level.
mfoltzgoogle: One reason we chose CBOR is that applications could perhaps speak CBOR directly if we expose that in the end.
… These things are aligned with the direction we're heading, so excited to see this.
… I see that you've been using GO. There is a prototype of the Open Screen Protocol written in GO.
… It does not have the PAKE part though.
backkem: Great. I'd also like to get the WebRTC community informed on this work.
<anssik> Pure Go implementation of the WebRTC API
tidoust: I note that enumeration of devices in general is hard to achieve because it's hard for browsers to display a meaningful message to users if the enumeration mixes screens with fridges and health devices
backkem: One posssibility would be to query a specific type of devices perhaps.
anssik: My thought is that we need to find a place to incubate that idea.
tidoust: Michiel proposed that in the WICG, that seems like a good place to start.
backkem: Right, I'm trying to build a community here.
mfoltzgoogle: I see connections with what we're doing with WebRTC and WebTransport. Incubation in the WICG is probably fine for now. It shouldn't be hard to move it later if a specific community is needed.
<anssik> https://
<anssik> https://
anssik: Feel free to request a GitHub repo under WICG.
anssik: You can loop us in, happy to help.
anssik: Thanks all for participating, see you all tomorrow!