Meeting minutes
Multi-Screen Window Placement API
Fullscreen Companion Windows
[group reviews agenda, security related issues for OSP would be on the agenda for TPAC meeting]
msw: Goal is to initiate compelling user experience from a single user gesture.
… Main extension: permitting a site to open a popup window after requesting fullscreen. Requiring the window placement permission and requiring for multiple displays and then permitting a window.open call on a targeted display to create the popup in the targeted display.
msw: Exploration is to look at an incremental way to support the request from our partners, that would otherwise require multiple user prompts or too broad permissions in Chrome.
anssik: Is this feature in latest version of Chrome?
msw: Disabled by default but available behind a flag.
… We do have a spec PR up for review. PR against the Multi-Screen Window placement API.
… Our goal for this is to permit a site to either create a window or initiating a fullscreen content for each display of the user.
… I imagine clicking a single button that allows a site to create fullscreen content across displays with popup windows on some of them.
… Main partner is a web-based partner that is looking to achieve parity with native experience.
msw: The explainer contains details, and you can test in Chrome by enabling the flag.
"The implementation is complete, and available in M-103 with a feature flag: --enable-features=WindowPlacementFullscreenCompanionWindow"
mfoltzgoogle: One question I had, specifying the behavior: in the explainer, you mention mitigations that browser may put in place for new popups above/below the new window.
… I was wondering whether that puts constraints on the window placement API to move windows on top or below fullscreen windows.
msw: Yes. E.g. prevent a popup from appearing on top of a fullscreen window. It's a sensitive area, we're trying to make sure we handle all cases appropriately in a way that would work for our security team.
"Pre-existing protections prevent placing popups over fullscreen
- Chrome exits fullscreen (see ForSecurityDropFullscreen)
- Firefox exits fullscreen
- Safari opens the popup in a separate fullscreen space"
msw: Some related history in the explainer on how other browsers handle similar use cases.
mfoltzgoogle: One more observation. One thing that's a little bit different is that user normally gets to see the omni bar. With this, I'm unclear that user has control on this.
msw: That's a good question for extensions we have in mind.
… Requested by our partners, so we're looking into it.
anssik: What is the review story on top of TAG?
Mozilla's Standards Positions > Multi-Screen Window Placement: Fullscreen Companion Window
msw: We requested review from Mozilla. They requested details about security considerations. More a general comment on the prose and phrasing of the considerations.
… I pinged the original TAG review for additional guidance here. We haven't heard back yet.
anssik: This is great exploration. This feature potentially is something that other browsers might want to implement a bit differently from a UX perspective.
… In general, it would be good to accommodate different UX concepts.
… Reminds me of yesterday's discussion, e.g. to avoid user prompts.
msw: I hope we can get some involvment from Mozilla so that we can converge on a mechanism.
Fullscreen Capability Delegation
msw: Capability delegation is a proposal incubated in WICG.
… Initially targeted at the Payment Request capability.
msw: A frame receiving a user activation signal may wish to delegate its capability granted by that signal to another frame.
msw: A parent frame might wish to delegate a capability to show a payment request to a sub-frame for instance.
… Mozilla and others expressed an interest to generalize that to other capabilities.
… One of the capabilities that appeared here is fullscreen request.
… More specifically between a popup and an opener.
… You may have a control window that has a button that can make a window fullscreen. Currently, it's not possible to request fullscreen in a separate frame.
… Capability delegation offers a solution there. The work that we're undertaking here is to add fullscreen request to the list of capabilities that can be delegated.
… Chrome launched capability delegation for payment request in M100.
… I've been partnering closely with Mustak.
anssik: I recall seeing positive signal from Mozilla. I believe we don't have signals from Webkit, is that correct?
msw: That matches my understanding.
… It's definitely worth prototyping. I'd like to see more hollistic thinking about how capabilities can be decomposed and shared across specs. Currently, it's being retro-actively added to specs.
Accurate Screen Labels
msw: Something that we had considered from the outset with the multi-screen window placement API.
… Internal screen 1, internal screen 2 to describe the screens in a way that is more meaningful for users.
… This effort is exposing more accurate screen labels, surfacing label provided by EDID or OS APIs.
… Fair amount of prototyping by Brad.
btriebw: Implementing different platforms, running in some issues such as blank EDIDs (we'll return empty strings in such cases, I created a PR for that)
… We're also working on implementations for Linux and Mac. Available behind a flag.
<ghurlbot> Pull Request 102 [closed] Add note about empty string in screen labels. (bradtriebwasser)
anssik: User will see these strings and there may be privacy implications, so important issue to discuss.
mfoltzgoogle: I don't know if this is part of the spec. Given not all displays have valid EDID or you may have missing data for some displays, would the idea be to fallback to generic "display 1" labels?
btriebw: That was the initial idea.
… But our partners had a preference for them to control the labels when data is unavailable, hence the idea to return empty strings so that they can act on it.
mfoltzgoogle: The other thing we've run onto before. The order by which the platform enumerates the displays may not match the physical setting. Maybe you can re-order the set. Just curious about the order the displays are returned to the API.
msw: We did implement as part of the base API an ordering based on the screen's top-left coordinate.
… I'll need to check but I'm pretty sure that we're using the same ordering.
… We got a comment from someone saying "I have two identical displays, I'd love them to appear as 'left' and 'right'".
… I think we can iterate over this, and figure out what the best labels are.
… We also want to permit user agents to hide the information if they don't wish to expose it.
Opening child window in fullscreen / window.open 'fullscreen' option
<ghurlbot> Issue 7 window.open should support the 'fullscreen' option (cterefinko) enhancement
<ghurlbot> Issue 92 Feature request: Fullscreen support on multiple screens (shizukanaqun) enhancement
msw: This is a bit more open-ended. In the pursuit of incremental solutions using existing Web API surfaces to give sites more multi-screen capabilities, one of the low-hanging fruits is the possibility to open a child window in fullscreen.
… Mark had a good comment about this earlier. There's a number of usability concerns when it comes to opening separate windows, esp. multiple windows, in fullscreen.
… It's definitely something that is being requested by a number of different partners.
… I think that's something that we're bound to explore in the mid-term.
… I would like to see more thoughtful exploration of what it would like to exposing an API to allow multiple child windows in fullscreen across multiple displays.
… Requesting popup windows to be created in fullscreen seems one of the most tractable thing that we can explore.
… I hope we can explore this further and meet our partner request.
mfoltzgoogle: One question. For the use cases where you may want fullscreen content on multiple displays. How many require separate windows? How many would be content with a window that spans multiple displays?
msw: Just because the layout of displays might not be contiguous and rectangular, I imagine multiple windows is a more tangible approach.
… That is an interesting idea though. I'd love to see how this has been approached by other existing software solutions.
<msw> https://
anssik: I put a link to an image in the issue discussion that describes this idea.
… Am I hearing that your current partner feedback does not request this?
msw: Definitely been requested but in less urgent manner.
Other ways to extend User Activation concepts for multi-screen/window interactions
msw: Another request that we started seeing. The ability to open N windows on N displays, perhaps making a subset or all of them fullscreen, and perhaps being able to delegate a fullscreen request to each of the N windows given a single user activation.
… It sort of builds on requests that come again and again. Try to understand how to allow sites to use multiple displays simultaneously is an interesting challenge.
… Interesting to explore how user interaction models might be limiting site capabilities.
anssik: Similar to 15+ years ago when browsing on mobile became a thing. Many web features had to be catered for smaller screen.
… I hope the group will be part of the solution.
… I encourage people, including those who are not on this call, to review the issues.
Virtual Display Exploration
Virtual Display use case exploration
anssik: I started putting down some notes, figuring out how virtual displays could work.
… Videos help understand what virtual displays are. First video from Deskreen, project who was presented to the group during our last F2F.
[going through Deskreen video]
anssik: WebRTC connection between the laptop and the iPad running Safari. Virtual display created through a dummy HDMI plug. You can also create virtual displays through software.
… The key thing is that the OS manages the virtual display creation.
… Once done, you can consider the virtual display like an offscreen. That's a way to create an ad-hoc screen space to extend your screen area.
[going through Sidecar video]
anssik: That gives you a rough idea of what a virtual display use case is.
… Sidecar is more integrated with the OS but basically the same idea.
… I started thinking about whether there were some learnings here that we could apply to our work.
… Some UX considerations:
… 1. Display arrangement, currently OS is in charge.
… 2. Display Source picker: some browser implement this for getDisplayMedia, or captureStream
… 3. Virtual display creation: A fantastic UX would be that bringing a tablet close to a computer would be enough for magic to happen. Currently, it is managed by the OS.
… 4. Window placement: OS handles this currently, but Multi-Screen Window Placement API could handle part of it, though virtual displays are a non-goal currently.
… [going through illustrations for each of these considerations]
… For window placement, drag and drop are the implicit granting permission mechanism. OSes have keyboard shortcuts as well.
… The traditional Web API approach is to prompt users.
… There may be an OS-level control built in window manager.
… Idea here is to show that there are multiple ways to ask user consent. In some cases, integrating with OS might be preferred by browser vendors.
… I'm running out of time, I'll share some version of that with the group for further discussions.
… Is anyone else using such virtual displays? Deskreen is a one-man project which hit the home page of Hacker News last year.
msw: I remember seeing the Deskreen video at our prior meeting. I like that idea a lot. Valuable in many contexts where you may wish to co-op devices to create more displays.
… That seems pretty valuable to me. Further, being able to share the aggregate display space in a more seamless manner would be interesting.
… Lots of good opportunities here. It's tricky to explore as well because each individual is going to have different setups. A deskreen-style solution is a good generic solution to present the multi-display setting.
anssik: Surface across OSes is the challenge. Different concepts depending on the OS, e.g. "virtual desk", etc.
msw: It's interesting that you brought up the idea of separate spaces or workspaces or desks. It's somewhat tangential to having multiple displays. It's more like a toggable environment for each of the displays or for the aggregate display space
Louay: I'm not sure if we had in the past a discussion about allowing a web page to advertise itself into a receiver.
anssik: Maybe we can create a repo story and discuss these points.
… It's just a collection of ideas for now. I'll start something online.
mfoltzgoogle: I'll look at the slides later. It looks like in most of these scenarios, the user has to connect these virtual displays through the OS before they become available for use on the web
… If the OS could tell us that there has been a virtual display connected with the device in the past, maybe the browser could leverage this to suggest re-connecting the virtual display if it is currently disconnected.
… It might be something to think about.
anssik: That resonates with me. I'll put the material in a repo.
Open Screen Protocol
mfoltzgoogle: First thing I flagged on the OSP is around expressing HDR rendering capabilities.
… We had a meeting with the Media WG, incorporated the Media Capabilities, merged the PR and that closed that out.
… There are two separate considerations: transfer function and the format of the HDR metadata.
… We need to finish this off. Currently the issue only covers the color gamut piece.
… I propose to address the remaining two pieces of information separately.
… i.e. merge PR 225, and prepare additional PRs for the remaining transfer function and metadata format.
<ghurlbot> Issue 233 Use the same codec names as the media APIs. (pthatcherg) F2F, v1-spec
mfoltzgoogle: The other capability we discussed with the Media WG was around ways to describe codecs with a string.
<cpn> WebCodecs Registry: https://
mfoltzgoogle: Since we discussed, the WebCodecs spec has a note that links to a registry spec with strings that link to various specs describing the codecs.
… The WebCodecs doc looks like it has entries for the codecs we care about.
… The proposal is to create a PR to specifically refer to that as the source of codec capabilities for the Open Screen Protocol.
cpn: There has not been more discussion in the Media WG around this.
… You mentioned that the registry contains all the codecs you're interested in. These are all the codecs supported by WebCodecs, but list may be different for media playback or remote playback.
… Different list from Media Capabilities in short.
mfoltzgoogle: Does Media Capabilities have a different way to refer to this?
cpn: It just refers to MIME type definitions somewhere.
… Media Capabilities itself does not define any codec string.
mfoltzgoogle: We may need something a little more nuanced as in: use the registry for codecs listed in WebCodecs, and for codecs not listed, it may depend on a different specification.
cpn: And we may need to find a reference.
mfoltzgoogle: I still would like to use WebCodecs as a baseline.
… Maybe we can discuss in the issue how we can go beyond that.
cpn: Yes, I'll happily follow up on an issue.
Open Screen Protocol - security issues
mfoltzgoogle: There are a few other issues that are not security related per se, but still have some security implications. As said earlier, I'm more targeting TPAC for these together with core security related issues.
Connectivity Standards Alliance coordination
Link to published slides on Matter and Open Screen Protocol
anssik: In our latest charter, we added the Connectivity Standards Alliance to the coordination section, for their work on Matter.
mfoltzgoogle: I may skip a few details in the interest of time. Will remain a high level presentation in any case.
… Matter is an evolution of a few different efforts in the smart home and IoT industry to resolve interoperability issues between different manufacturers.
… Main emphasis on finding mechanisms for users to introduce new devices that can communicate with one another.
… There have been some public presentations on Matter and announcement, but the spec itself remains is confidential until later this year.
… I'll stick to public considerations here.
… More of the technical details will be available later this year.
mfoltzgoogle: There is also an open source implementation of Matter, so you can go and look at that to better understand what it covers.
… The scope of Matter covers a bunch of device types, including lighting, sensors, as well as media devices.
… Most of these are fixed devices that don't run apps.
… Fixed set of functions, you don't load your software on the devices.
… Goal is to create common infrastructure to provision, manage, handle security aspects.
… The main outputs are the spec (end of the year), the open source sdk available on GitHub and certification tools
… Matter comes out of an industry group called the Connectivity Standards Alliance, previously the Zigbee Alliance. They do have members across the industry (see CSA web site)
… I don't know a lot on the inner workings of the CSA. They have different working groups and spec tracks.
… Zigbee was an earlier effort to build a low-power network. As far as I can tell, that continues to exist in parallel with Matter. There may not be immediate interconnection between the two.
… Matter timeline: They had some internal test events to evaluate whether Matter performs as planned.
… They're wrapping up these events and in the fall are planning to release the specification.
… Hopefully, we can take a look at the spec at TPAC.
… Some technical info release: IP-based set of protocols. Supports TCP and UDP as underlying transport.
… Reliable mode.
… They only use IPv6 for IP and then provide routing data between IPv6 and underlying networks, including Thread, low-power network.
… I'd be interested to learn more about Thread.
… A lot of emphasis on security and authentication.
… The main way for a device to become trusted is to get two certificates: one from the the manufacturer, probably relying on a core list of certificates by manufacturers.
… The way that the device gets added to your home is to go through a Commissioning mechanism.
… This creates additional certificates to validate permissions.
… The protocol can do access control using that additional authentication.
… In terms of comparing with the Open Screen Protocol, in many ways there are complimentary.
… We've chosen QUIC, they have their own transport protocol that uses certificates and leveraging Sigma (which I need to research more)
… Below that, the OSP does not really care as long as you can do UDP, whereas Matter looks more into details on network topologies.
… The main difference is how authentication is handle. We use SPAKE-2 (possibly C-PAKE in the future).
… Matter goes through this commissioning ceremony.
… In terms of the certificates used, there are some similarities and differences. It could be worth looking into extending support in OSP to cover Matter certificates.
… There are some published papers but spec would be the best to look at once available.
… Areas of investigation: pairing ceremony, interop between Matter and open screen transports, third is data model for media streaming and media/application control and whether we can leverage those.
… Between now and TPAC, I would like to address security issues in Open Screen Protocol, so that we can better compare.
tidoust: do we know whether media streaming capabilities are in Matter, in a sense of 1-UA mode?
… anything more about the media capabilities
mfoltzgoogle: I think data models have not been published yet
… source code is available however, so that can be used to infer the capabilitites
tidoust: last time I checked the things were more comparable with the 2-UA mode
mfoltzgoogle: on a high level, more message oriented, and not streaming oriented
TPAC 2022
anssik: TPAC will be hybrid meeting in Vancouver, mid september.
anssik: I'd like to evaluate people's feelings towards traveling to Vancouver, and if so if there are conflicts that you would like to avoid.
mfoltzgoogle: I would like to attend the WebRTC WG meeting. I may also want to attend the Media WG. It really depends on the agenda.
anssik: WebRTC is monday/tuesday currently (preliminary draft schedule). Media WG would be Friday, I think.
mfoltzgoogle: I'm planning to be in-person in Vancouver.
anssik: I'll be there as well unless the world significantly changes as well.
cpn: Likely to be there.
Louay: Some conflict with IBC fair in September but I'll probably be around.
<msw> very likely to attend in-person; Second Screen is my primary WG of interest; I'm also interested in Web Apps WG and WICG
anssik: I'm hearing that at least some people are interested in traveling if situation is reasonable.
takumif: I would like to be there as well.
[I'll be in Vancouver as well]
anssik: I'm hearing that CSA Matter would be a topic on the agenda. WoT people may be interested to attend that part of the discussions.
… Also Second Screen WG PAG should have published final report by then too.
… Multi-screen window placement will probably have made progress.
… Anything else on your side?
msw: Multi-screen window placement is my main focus. I'd be interested to engage with CSS folks for implications with multi-screen.
… notably on CSS OM.
… related with fullscreen.
anssik: I'll work with Louay and Francois to propose a schedule for TPAC meetings
mfoltzgoogle: In previous meetings we had people interested in exploring work on window segments. Should that be on the agenda?
anssik: It's transitioning to CSSWG, in a previous meeting we decided to move parts of it to Visual Viewports
Window Segments Enumeration API explainer
anssik: Need to check on current status
<msw> https://
anssik: If you have suggestions for the CG agenda in TPAC, we can use that to bring attention to ideas
cpn: would like to be in your meetings, but often have a conflict
… in my submission wanted to avoid conflicts with Second Screen WG/CG
anssik: we'll be remote friendly, US+Europe and US+APAC times
Thank You!
anssik: Thank you to everyone
<msw> ty Anssi and all!