W3C

Second Screen F2F - Day 2/2

21 October 2020

Attendees

Present
Alexis Menard, Anssi Kostiainen, Arno Mandy, Chris Needham, Eric Carson, Francois Daoust, Kenneth Christiansen, Laura Morinigo, Mark Foltz, Masaya Ikeo, Mike Wasserman, Olivier Yiptong, Srikanth, Takumi Fujimoto, Victor Costan, Wooglae Kim, Zouhir Chahoud
Regrets
Chris Needham, Francois Daoust
Chair
Anssi
Scribe
Mark

Meeting minutes

Foldable use cases and requirements

Web support for new device features

anssik: Today's meeting will be focused on exploratory work in the SSCG and in related groups.
… I have asked Srikant to talk to us about new form factors in upcoming devices,
… how they should be considered in new Web APIs.

Srikanth: Presentating on Web Support for New Device Features
… Long history in VESA, HDMI, Wi-Fi display, other bodies.
… There are multiple new form factors. Foldable displays with multiple independent display panels.
… Ranging from laptop size to smaller sizes, like a mobile phone.

[ Srikanth presenting slides ]

[ MID = Multiple Internal Displays ]

Srikanth: Intel Horsehoe Bend, at CES earlier this year. Supports multiple configurations: laptop, table-top, flat, canvas or book
… Depending on the amount of incline, you can lay it all the way flat, for input or drawing.
… Your user experience will differ, and the camera and speaker are reconfigured accordingly.

Srikanth: shows a video demonstrating capabilities of the foldable laptop.
… This is not a new concept. [Shows a slide of the Lenovo X1 Fold on the MID platform]
… Even if you have a MID, the emphasis is on fold. Both are posture-based. User experience and resources will differ.
… Many of these are supported in W3C today, but how to provide a seamless, consistent experience.
… Consistency in how Web applications determine what the posture is. Posture enforcement as necessary.
… Want this regardless of foldable vs. MID.
… [Shows a device with a physical hinge]. This device can with a sharper fold; can have a 360 degree bend.
… Devices without hinge, there is a less acute fold because of the physical layers of the display. Can't completely fold (0 or 360)
… Number of displays, we are seeing two, but there could be more. Could be panels of unequal size, or three different panels.
… Do the interface once, have these webapps work and be foreward compatible.
… Foldables, there is typically one display. It depends on the display configuration, could be one or more depending on the posture.
… Seam is always present on a MID. In a foldable, there is a virtual hinge, it's a user experience decision.
… The beauty of a virtual hinge is it's a software abstraction.
… Touch controller is associated with each panel. Is configurable to generate various HID reports.

Srikanth: [Table showing attributes of MID vs. Foldables]
… If you have an enlightened app, aware of the seam, it can adjust layout according to the seams in the display.
… This is an example of how a seam can help the web app to choose.
… [Slide shows how apps could customize themselves for different foldable postures]
… Key point is, apps can benefit from other information, posture, occlusion, etc.

Alexis: New UX patterns as you use different postures on the device.
… Show you kinds of problems you might get on web applications. We designed a gallery to respond to multiple displays.
… Not to just adapt their app, but based on posture, provide a better experience.
… Don't want to position across the hinge.
… The current thing that we see, is covering the new form factors.I need the screen fold API to determine
… If I am in tent mode. They are complementary, but I can't use them in a consistent manner, across the OS.
… Next steps, MID are different from foldables. Different from an externally connected displays.
… Whatever specification we put up, should cover these types of devices.

anssik: The future looks quite exciting, when it comes to new form factors. We should be designing new APIs in a future-proof manner.
… This is the beginning of the discussion, but want it to be a kick-off for this type of exercise.

Mark: certain type of single displays can do virtual multiple displays, how does this work out with that, has that been considered a use case?

Srikanth: That is a use case as well. For example, you can use docked devices.
… You can have software abstractions that virtualize the hardware. I fully expect the platforms to have multiple docked scenarios.

zouhir_: We have an API to get the geometry of each physical display, an API with an enum of postures.
… Developer has to use two APIs to get geometry and posture.

Alexis: Edge doesn't allow me to get the posture, that's the inconstency.
… : Incomplete implementations, by having them separate, some browsers may not implement one or the other.

Srikanth: Many low-level details are exposed to the application, applications have to put them together to get the big picture.
… Different applications may interpret them differently. What sort of things are okay to allow some inconsistency.

msw: Do you wish to generalize the foldables API to handle additional use cases? Like MID / 360 hinges?

Srikanth: Given all the possibilities in the underlying hardware. What is the right abstraction to expose to webapps?
… Tomorrow, if there are 4 internal panels, foldable in different ways, do we want to re-do some APIs, or create full abstractions?

kenchris: It has to be simple as possible for the common use cases, or developers won't adopt it.

anssik: Try to understand what are the experiences they want to build, and optimize for the common scenarios.
… Make the less common thing possible, but maybe not as easy.

Multi-Screen Window Placement

Multi-Screen Window Placement TPAC 2020 update (also available in PDF)

msw: [Shares slides]

msw: This proposal is related to foldables. Addresses a generalized set of use cases.
… Generalize Web Platform for multi-screen displays.
… [Slide comparing existing Web APIs and goals of MSWP]
… [Slides showing examples of multiple-display devices, projectors, signage, lots of use cases]

Goals: Increasing initial information about additional displays; adding a change event when they change;
… Request additional per-screen info; sites can request fullscreen on another window
… Slide shows current properties exposed through CSSOM
… Proposal is to allow sites to request additional information about connected screens.
… Currently you can use window.open to create new windows, with left/top coordinates
… Can move to/by (x,y) coordinates
… Crux of the proposal is to allow cross-screen coordinates for these operations.

msw: One set of updates is that we have an ongoing origin trial with prototype code.
… Addressing some good feedback, including some API shape changes.
… There is a demo: https://‌window-placement.glitch.me
… More info: https://‌web.dev/‌multi-screen-window-placement
… Github: https://‌michaelwasserman.github.io/‌window-placement-demo

msw: We have interested partners in this, they want to explore a new space.
… We have Chrome apps APIs and OS APIs, we want to offer the same affordances to the Web.

anssik: You are catering for the established usages. Your use cases are more clearly defined.
… For foldables, it's not as clear.
… How much we can unify MSWP, and some of these new proposals. It might not be 100% overlap.

Eric: What you've done to mitigate the many bits of entropy about users with non-standard configurations?

msw: Similar information is exposed if windows on placed on multiple screens. We are also proposing a permission that user agents can use to gate access.

Eric: What is exposed without a prompt?

msw: A single bit whether there is an external screen, so sites know whether to prompt multiscreen users.

<anssik> Mark: Presentation API is slightly different in terms of availability detection, the Presentation API does not expose any info on external display, Remote Playback is similar

Zouhir: Want to work to create an API surface that is minimal, can talk again about how make these APIs work together.
… Regarding privacy, if permissions is the way to protect customers, and I want to make sure my YouTube controls are laid out right.
… Don't want to show a bad experience, then require a permissions prompt to show a good experience.

msw: Good opportunity to collaborate on an API surface.
… Permissions are tricky, Web platform was built so that users can see screen info. If you install an app, it can access screens and system info.
… Possibly we could expose screens that a application is spread across, without requesting permissions.

anssik: Let's create follow-up actions through GitHub

Window Segments

Window Segment Enumeration Script API (PDF)

Zouhir: [Presents slides]
… Window Segment Enumeration Script API
… Targeting dual screen and foldable devices.
… Surface Duo/Neo. 2 physical screens, 1 fold.
… Samsung Galaxy, 1 screen, 1 fold, 2 logical screens.
… Email app example, responsive on a dual-screen, the content is not laid out in a user-friendly way. Snap content to fold.
… CSS API to help developer lay out. In case CSS is not available for games and canvas, the JS API is handy.
… Simple, one method. getWindowSegments() returns a list of segments.
… API is an immutable snapshot, need to listen to resize events to know if it's invalid.
… Difference between dual monitors and foldable. App launches is not consuming the full screen on desktop.
… Maximizes, does not span dual monitors, but does on foldable.
… Users want to avoid alerts, dialogs on fold. Want to move it to one side or the other.
… Customers want to focus on portable form factors.

Zouhir: Don't want to create new, redundant APIs
… Don't want a lot of API surface for developers.
… Outlook, e.g. not requesting multi-monitor spanning behavior.
… Want the Web to be prepared for new form factors, versus sniffing around to optimize for them.

anssik: Start with the core use cases, and incrementally ship new features. "When in doubt, leave it out"

Zouhir: If use cases are different, e.g. multi-window applications versus app layout. Think about naming it carefully if we decide to keep APIs separate.

msw: The premise of segments, is potentially polyfillable given the intersections of screens and windows.
… Are foldables doing something different than the desktop display paradigm?
… There is a more general set of primitives, that could provide more information, that people could build libraries for different use cases.

Zouhir: We can use one API with more complex information, developers can optimize it. If I have to do calculations, developers will build a support library.
… What if I am on desktop, and I span my browser. I feel like users don't expect layouts to change. Would there be surprises for users?
… Looking into how we can make information available, without requiring permissions.

kenchris: It is hard to polyfill CSS. You might want to use an iPad as a secondary screen, and not be cut by a seam.

Eric: What do you do to mitigate the entropy that this API exposes about a non-standard configuration.

Zouhir: Right now we don't have permissions, developers can know a lot about the device form factor already. We have a policy to prevent iframes from getting info.
… Gating it by a permission is providing a bad user experience at the end.

Eric: Agree that a user prompt isn't useful, but for perspective, we have device enumeration for getUserMedia.
… When we looked at how they were being used, we found that enumerateDevices was used more than 2 orders of magnitude times more often than getUserMedia.
… It was being used solely as a means of fingerprinting. Now, enumerateDevices returns a lie, until the user has given consent to capture.
… We got feedback that it was a change that had to be made. You'll have to have a compelling argument that the bits of entropy are worth the cost.

Zouhir: We don't want developers to be able to achieve something natural for this class of device. Exposing the device hinge is not a big risk.
… We can't lie to developers about the hinge position. We have to tell them something useful, position and dimension of hinge.

anssik: What extra info does surface duo share, versus e.g. iPhone max?

Zouhir: You can infer device from geometry, because there are few foldables on market. For iPhones, developers need to know size e.g. of notch, to avoid rendering text on it.

anssik: We need to differentiate between multi-monitor setups, and other use cases.
… (regarding fingerprinting possibilities)

Zouhir: Foldables are another responsive Web design target, I think it's something developers need to be aware of.
… As a website developer, I should be able to make it look nice. But if I have an ad iframe, it does not need this information.

Alexis: In the case of a dual screen, you have only one screen, so the viewport size is the sum, so without this API you can get the viewport size and aspect ratio.
… We are not exposing anything, because it's viewport / 2.
… We are already exposing much of this information in other ways.

Action: zouhir_ to document fingerprinting concerns and mitigations in the spec

Zouhir: Want to respond to the discussion about libraries. Whether to have something simple, or require a library/NPM.

Action: zouhir_ to open a GitHub issue to discuss use of libraries to provide window segments information

Screen Fold API

kenchris: Some of these devices have a physical hinge, with dual screens.
… Want to change behavior based on posture. Or animate based on fold angle.

kenchris: Expose as a ScreenFold object with a change event.
… Look at different postures that might exist.
… Some devices might have a second fold, or different screen sizes.
… The hinge is not in the middle. Or maybe there are two hinges.
… Looking at what OEMs are thinking of, dual fold devices.
… Want this to be integrated into other specs. Could be used as a secondary screen.
… Want to use the screen on the table for video editing.

kenchris: Important to standardize when things are in certain postures. What angles are they using? Talked to Samsung, Intel.

kenchris: Can be used in combination with other APIs.

anssik: Are we able to handle multiple, external monitors, which might be foldable?
… To test how prepared we are for the future.

kenchris: We need to integrate with Window Placement, and handle multiple folds.
… I could imagine having the Intel Horseshoe, and multiple external monitors.

anssik: We may be able to handle this API in the future, how does the future look like? Don't paint ourselves into a corner.

msw: When there are multiple integrated displays, if they are separate Screen objects, how would a single fold between those screens be supported?

kenchris: We have seen this as one virtual screen. Are there any devices where the hinge separates the screen?
… See if there is a hinge, but not screens right next to each other.

Alexis: On the surface Duo, it will be two physical screens, two logical screens.

anssik: Does the screen fold API address traditional form factors, that might have a fold sensor?
… Does this API report the angle?

kenchris: Not sure if there is a sensor for these devices.

kenchris: Are there use cases for supporting this?

msw: Anecdotally, laptops and Chromebooks transition into tablet mode, when they fold beyond 180?

Zouhir: Window Segment / Screen Enumeration have a similar concept, but different use case. Foldables API, complements these use cases.
… Which way do we want to think about merging Window Segment.

kenchris: If we have a laptop with a hinge, that knows when it is converted. It's a different screen size, is there a use case for traditional laptops.
… If so, it doesn't make sense for foldables.

anssik: Do native APIs expose hinge angle? If no, then Web content doesn't require it either.

Alexis: There is a sensor for the Pixelbook. It it's used to resolve the media query for hover/touch.
… Web application can adapt to be touch friendly.

kenchris: That seems like a way to optimize for primary input. Also, on some devices, the granularity of the sensor varies.

Alexis: Some devices with a stylus always report that a stylus is available, so input signals are not a good way to detect tablet mode.

Aligning Window Segments, Screen Fold, Multi-Screen Window Placement APIs

Zouhir: We have three APIs, two similar in concepts, two similar in use cases.
… Don't rush and commit, open to ideas.

anssik: Discuss opportunities for aligning. Make sure that these people are watching each others' work.
… Useful to cross-link specs, that Screenfold and Window Segments are addressing similar applications.

Zouhir: Agree. We should add the foldables APIs to keep them in those use cases.

Action: zouhir_ to cross-link foldable-related specs together

Zouhir: When I am dragging my window across two windows, how can web developers decide not to opt-in the foldable layout?

<kenchris> https://‌github.com/‌SamsungInternet/‌Explainers/‌blob/‌master/‌Foldables/‌FoldState.md

<kenchris> https://‌w3c.github.io/‌screen-fold/

msw: There are currently no window events on move. It would probably be valuable to have a move events on window. Maybe expose segments after the move loop is complete.
… Maybe only surface segment information if the window is maximized and spanning both monitors.

Zoushir: Different devices have different granularity, with respect to resizing. Also only maximizing, may exclude Samsung use cases.
… Desktop always be opting in, unless you opt out with media queries, viewport size.

msw: Part of the screen information exposed, could be whether the screens are physically adjacent.

Action: msw to reference complementary proposals of Window Segments and Screen Fold, Visual Viewport, open appropriate GH issues

<tidoust> [As part of cross-referencing specs, alignment on terms could be useful (and easy) too. I heard Alexis talk in terms of logical/physical displays. Screen Fold uses "screen", Window Segment uses "display" and "screen", Window placement uses "connected screens"]

msw: A breakout session could be the best way that we have something concrete to look forward to.

Action: msw to (with help from kenneth, zouhir, dlibby) organize a TPAC breakout session on aligning Window Segments, Screen Fold, Multi-Screen Window Placement APIs

<msw> TPAC 2020 breakout session ideas

Summary of action items

  1. zouhir_ to document fingerprinting concerns and mitigations in the spec
  2. zouhir_ to open a GitHub issue to discuss use of libraries to provide window segments information
  3. zouhir_ to cross-link foldable-related specs together
  4. msw to reference complementary proposals of Window Segments and Screen Fold, Visual Viewport, open appropriate GH issues
  5. msw to (with help from kenneth, zouhir, dlibby) organize a TPAC breakout session on aligning Window Segments, Screen Fold, Multi-Screen Window Placement APIs
Minutes manually created (not a transcript), formatted by scribe.perl version 123 (Tue Sep 1 21:19:13 2020 UTC).