Skip ⬇

Cloud Gaming on the Web

Facilitator: Zhaoming Li

Speakers: Qingqian Tao, Alicia Nie

Minutes (including discussions that were not audio-recorded)

Previous: Learning from MiniApps All breakouts Next: Memory copies & zero-copy operations on the Web

Skip ⬇

Skip ⬇

Transcript

Can you hear me?

It's very clear.

Okay fine, lets get started .

Basic knowledge and some of the technologies we are using on Web now as well as the future outlook.

I will probably talk about four points.

First I will talk about what is cloud gaming I believe everyone has a better understanding of this today - because time is limited so I will talk a little faster and briefly.

The second point is about the reason why we should run the cloud game on the web and its advantages.

The third point is about some of the technologies that we will use when we serve cloud gaming.

And then finally how our browsers will help the development of cloud gaming and cloud game operating better in our browsers.

First of all ， the concept of cloud gaming is that an online game running on the server and streaming back to local server something like that.

It is understood by everyone/ The main point I want to mention here is that some small Edge computing may used in the game and the concept of edge computing.

In the practice of cloud gaming in China our game servers are usually running behind NAT.

In this way this server may not have the public access.

In terms of network latency because our edge servers are generally deployed second and third tier cities in this way compared with the centralized servers which may be deployed in some large cities.

In this way our edge servers may have lower network latency.

In the game scene because our network delay is a very important part so some of our technologies can be used to reduce some of the delay effectively which is some of our practices.

Then I will tell you an example of our practice of cloud gaming in China which is done by Netease.

Cloud gaming done by NetEase is on PC and Android two platforms but not on host platforms like XBox and play station.

These PC games we have developed run in QEMU virtual machine with NVIDIA graphic cards.

The interesting point is that their PC users and iOS users who use the client are using browsers but not through the independent native app not app.

They have an app on Android but they can also use a browser to connect on Android.

This means that companies who develop cloud gaming have a mature product running on web and it runs quite well.

We can see that in the first tier cities the network delay may be less than 10ms.

Besides the experience of playing RPG is very good when playing some FPS games experience may become a little worse but basically it is still acceptable.

Such an situation in fact has been verified in the commercialization they now charge about 1.8 yuan per hour.

This is a true case of cloud gaming in China.

This case actually has one point that we talked about in the middle is that it only uses the browsers as the client.

So the whole experience of the cloud gaming is based on the web.

Why would they used web?

First the most important point is that it has an click-to-run experience.

It can be used as long as there is a web.

The second point is about ability to distribute on the advertising network.

It is in fact that there are actually some promotions on Netease.

For example like Netease which is also a game company.

Their games for example YI Shuihan have a trial button on their official website there is a trial button on it.

If you click it you will directly open their cloud game and then start this game to try some content.

They will also put such advertisements on other platforms such as some advertising alliances on PC, then use the web as a cloud game platform.

There are some other important points: for example we actually have very good things on the browser including video decoding and keyboard and mouse events and the gamepad API.

These things have good encapsulation in the browsers; good upper encapsulation.

If it is native this thing will become more complicated.

Finally it is crossplatform and it can even be used on mobile phones.

These technologies are very mature and tested so its development cost is much smaller than native one.

The technology we use now in cloud games, Web technology, is mainly like this: for example webRTC the connection ， the video streaming and controls streaming in webRTC.

Then we use h264 or VP8 for video coding and then the mouse event is pointer events and we have pointer lock API which we can invent some FPS games and relative position of the mouse including the fullscreen API and gamepad API.

With these APIs in fact we have already achieved to give a better experience of playing cloud games on the web but in fact we still have some shortcomings in the browser whether in experience or in development.

For example the buffer for our playback are uncontrollable, videos must have a certain playback delay.

It is very difficult for us to optimize on the Javascript level.

The second point is that even if we use the pointer lock API it is difficult for us to get the real movement of the mouse because there may be acceleration of the mouse related to the operating system and its pointer may not be the same as the real movement which will be more obvious in the game scene.

In addition if we use webRTC it actually does not have the the video decoding of H265/AV1 so we are now using h264.

Another point about experience is the realization of google chrome now: when he presses esc key he may get some message reminder.

I have a picture here of playing this game on Netease platform: this is a window of chrome Then if you use the script inside to get the pointer lock API to achieve the rotation of perspective.

In this way ，a message will pop out on the head saying press esc to show your cursor.

Every time you use this pointer lock API to lock a mouse the message will be displayed.

For the user although if we use pointer lock we will hide the mouse.

For users，they need a way to understand how to display the mouse but for users who often play or are familiar with this platform in fact this prompt is relatively redundant and will effect operation.

For example this is a fullscreen scene and under this scene we also use pointer lock under this scene that the way to exit becomes press and hold.

But when the toasts comes up every time you press esc it will come up on google chrome.

Now it's an 84 version.

Then for example there is an operation bar on the top of the interface of my interface game and there is a UI that can be operated but the message that pops out will block the mouse click filed.

If you click it with the mouse the UI at the bottom will not be clicked.

You have to wait for this message to disappear before you can click it.

And generally in the game we use esc to open the menu so as long as you open the menu tips will pop up and then to block your operation.

In our user survey the experience is relatively poor But Edge in chromium base it will on display once when entering fullscreen it will not display after different time due to different browser.

I have looked up for standard from W3C in fact there is no rule that how this thing must become something after displayed.

It just said that the a given gesture to turn it off and exit the fullscreen state but it didn't say how the fullscreen state should be prompted.

About some of the shortcomings we have mentioned before we may have some improvements on the standard or some capabilities on the browser like coding mentioned before.

There is an WebCodecs which is a report of the game designing group : I feel like theres been some progress.

In addition as for the mouse acceleration at the operating system level mentioned before there are also some issues in the W3C standard and then this is our current situation.

Now there will be some progress on it and we will still want to have some new API or ability to do more suitable things for the game scene.

So to sum up because there are some things that we skipped before we want to say browsers do well on cloud gaming.

And the web has become the most important platform for cloud gaming.

Then as it was mentioned before that there are some good abilities to do these things on web the browsers actually did not achieve the best and ideal experience and we can do more things on the browser.

Next we want to have a discuss about how we can do these things better.

This part is led by my colleagues today.

Hello, everyone.

So I will use the English to share in this part content.

And so Monica, I will use the English, so could you help me translate to the Chinese, to the Chinese friends?

Thank you.

Oh, okay.

Oh, okay.

Okay, let's start.

The content we want to discuss in this presentation and we have three three parts to share.

So the first is a way we try to find the balance between the user-friendly and the safety toasts and immersive experience in the cloud gaming on the web.

So we also have to point to sharing and we can discuss if it is efficient way to resolve the problem.

So the first is the ability to cancel popup toasts when pressing functional keys.

So we're jamming just to share in the picture or some browser, or maybe use the browser UI to hide the cloud gaming, gaming UI.

So this maybe not a browser functional ability and just the browser UI function, but we try to find a good, a better user experience to resolve this problem.

The second is when we use the pointer lock API to handle the Esc key and sometimes it not works.

We just to discuss was it it is a web standard problem.

And in the second part, we want to share in a user case found some new capabilities in the web.

We just use the mouse point to some positions and maybe the browser need to provide some new API to move the cursor to some point, this is use case in the cloud gaming.

Okay.

The last is a way how to how can we use the HTTP/3 and or the QUIC in the cloud gaming to transport the streaming better.

So this is three-part content, and anybody can share in share your ideas.