Any particular reason we wouldn't just overload .src for this?

(In reply to Ian 'Hixie' Hickson from comment #1)
> Any particular reason we wouldn't just overload .src for this?

I'd be fine with an overload of .src. What would .currentSrc return then? An object instead of DOMString?

I originally did that and we changed it because people (Adam Barth) wanted the 'src' DOM attribute to remain a simple reflection of the 'src' content attribute. See the thread "MediaStreams and src attributes" on public-media-capture.

(In reply to Robert O'Callahan (Mozilla) from comment #3)
> I originally did that and we changed it because people (Adam Barth) wanted
> the 'src' DOM attribute to remain a simple reflection of the 'src' content
> attribute. See the thread "MediaStreams and src attributes" on
> public-media-capture.

Oh right. I forgot about that discussion. It seems like the interaction between .src & .srcObject might be tricky. Here are a few questions that come to mind.

1. How does setting one clear the other? .src = "" should probably clear .srcObject eventhough the load fails.

2. Should currentSrc only reflect .src still or should it reflect either .src or .srcObject based on which one is actually playing.

3. How should we handle v1.srcObject = v2.srcObject? Objects may not be capable of sourcing 2 elements. Should the load in v1 simply fail as if the "resource is not available"?

Questions 1 and 2 in comment 4 are why I think I'd prefer .src, but maybe Adam can elaborate?

Is this already shipping anywhere, by the way? If so, then this discussion is moot and I should just reverse-engineer whatever shipped.

(In reply to Aaron Colwell from comment #4)
> (In reply to Robert O'Callahan (Mozilla) from comment #3)
> > I originally did that and we changed it because people (Adam Barth) wanted
> > the 'src' DOM attribute to remain a simple reflection of the 'src' content
> > attribute. See the thread "MediaStreams and src attributes" on
> > public-media-capture.
> 
> Oh right. I forgot about that discussion. It seems like the interaction
> between .src & .srcObject might be tricky. Here are a few questions that
> come to mind.
> 
> 1. How does setting one clear the other? .src = "" should probably clear
> .srcObject eventhough the load fails.

The point of separating src from srcObject is that .src always reflects the "src" content attribute, so we'd want 'src' to be unaffected by setting srcObject.

To answer the obvious question, if both .src and .srcObject are set, .srcObject takes priority.

> 2. Should currentSrc only reflect .src still or should it reflect either
> .src or .srcObject based on which one is actually playing.

In our implementation, currentSrc returns the empty string when srcObject is non-null. I don't see any better way to do this, since a srcObject may not have a corresponding URI.

> 3. How should we handle v1.srcObject = v2.srcObject? Objects may not be
> capable of sourcing 2 elements. Should the load in v1 simply fail as if the
> "resource is not available"?

That is outside the scope of this discussion. It really belongs to the specs that define the sources. I presume in your case this is a question for the MSE spec. In the Blob and MediaStream cases there is no problem.

(In reply to Ian 'Hixie' Hickson from comment #5)
> Is this already shipping anywhere, by the way? If so, then this discussion
> is moot and I should just reverse-engineer whatever shipped.

We've shipped it as mozSrcObject.

(In reply to Robert O'Callahan (Mozilla) from comment #6)
> (In reply to Aaron Colwell from comment #4)
> > (In reply to Robert O'Callahan (Mozilla) from comment #3)
> > > I originally did that and we changed it because people (Adam Barth) wanted
> > > the 'src' DOM attribute to remain a simple reflection of the 'src' content
> > > attribute. See the thread "MediaStreams and src attributes" on
> > > public-media-capture.
> > 
> > Oh right. I forgot about that discussion. It seems like the interaction
> > between .src & .srcObject might be tricky. Here are a few questions that
> > come to mind.
> > 
> > 1. How does setting one clear the other? .src = "" should probably clear
> > .srcObject eventhough the load fails.
> 
> The point of separating src from srcObject is that .src always reflects the
> "src" content attribute, so we'd want 'src' to be unaffected by setting
> srcObject.
> 
> To answer the obvious question, if both .src and .srcObject are set,
> .srcObject takes priority.

It seems like this could lead to surprising behavior for existing users.

Say I have <video autoplay src="awesome.ogv"></video>

At some point JavaScript assigns a Blob to .srcObject. IIUC .src will equal "awesome.ogv", but the Blob will start playing.

Then JavaScript sets .src = "even-more-awesome.ogv". IIUC playback of the new URL won't happen because .srcObject is still set.

Do I understand this correctly? 

I suppose having 2 attributes interact with eachother might be more surprising though. I guess if I squint the right way, this is kind of like how <source> elements are handled.

> 
> > 2. Should currentSrc only reflect .src still or should it reflect either
> > .src or .srcObject based on which one is actually playing.
> 
> In our implementation, currentSrc returns the empty string when srcObject is
> non-null. I don't see any better way to do this, since a srcObject may not
> have a corresponding URI.

This is why I was wondering if it should return the srcObject or a DOMString. Is this not possible or is it not ideal for some reason?

> 
> > 3. How should we handle v1.srcObject = v2.srcObject? Objects may not be
> > capable of sourcing 2 elements. Should the load in v1 simply fail as if the
> > "resource is not available"?
> 
> That is outside the scope of this discussion. It really belongs to the specs
> that define the sources. I presume in your case this is a question for the
> MSE spec. In the Blob and MediaStream cases there is no problem.

Fair enough. I already define this in the MSE spec to deal with the same object URL being assigned to multiple media elements so it isn't a big deal.

> 
> (In reply to Ian 'Hixie' Hickson from comment #5)
> > Is this already shipping anywhere, by the way? If so, then this discussion
> > is moot and I should just reverse-engineer whatever shipped.
> 
> We've shipped it as mozSrcObject.

(In reply to Aaron Colwell from comment #7)
> It seems like this could lead to surprising behavior for existing users.
> 
> Say I have <video autoplay src="awesome.ogv"></video>
> 
> At some point JavaScript assigns a Blob to .srcObject. IIUC .src will equal
> "awesome.ogv", but the Blob will start playing.
> 
> Then JavaScript sets .src = "even-more-awesome.ogv". IIUC playback of the
> new URL won't happen because .srcObject is still set.
> 
> Do I understand this correctly? 
> 
> I suppose having 2 attributes interact with eachother might be more
> surprising though. I guess if I squint the right way, this is kind of like
> how <source> elements are handled.

Right.

> > > 2. Should currentSrc only reflect .src still or should it reflect either
> > > .src or .srcObject based on which one is actually playing.
> > 
> > In our implementation, currentSrc returns the empty string when srcObject is
> > non-null. I don't see any better way to do this, since a srcObject may not
> > have a corresponding URI.
> 
> This is why I was wondering if it should return the srcObject or a
> DOMString. Is this not possible or is it not ideal for some reason?

It could be overloaded to return the srcObject. I'm not sure that's useful in practice.

I don't feel all that strongly about this issue --- after all, I initially supported overloading the 'src' attribute --- but I *do* have a strong feeling of annoyance about having had this discussion a long time ago, being pulled in one direction by one set of people, and now being pulled in a different direction by a different set of people (some of whom work at the same company as some of the previous set of people).

(In reply to Robert O'Callahan (Mozilla) from comment #8)
> (In reply to Aaron Colwell from comment #7)
> > It seems like this could lead to surprising behavior for existing users.
> > 
> > Say I have <video autoplay src="awesome.ogv"></video>
> > 
> > At some point JavaScript assigns a Blob to .srcObject. IIUC .src will equal
> > "awesome.ogv", but the Blob will start playing.
> > 
> > Then JavaScript sets .src = "even-more-awesome.ogv". IIUC playback of the
> > new URL won't happen because .srcObject is still set.
> > 
> > Do I understand this correctly? 
> > 
> > I suppose having 2 attributes interact with eachother might be more
> > surprising though. I guess if I squint the right way, this is kind of like
> > how <source> elements are handled.
> 
> Right.
> 
> > > > 2. Should currentSrc only reflect .src still or should it reflect either
> > > > .src or .srcObject based on which one is actually playing.
> > > 
> > > In our implementation, currentSrc returns the empty string when srcObject is
> > > non-null. I don't see any better way to do this, since a srcObject may not
> > > have a corresponding URI.
> > 
> > This is why I was wondering if it should return the srcObject or a
> > DOMString. Is this not possible or is it not ideal for some reason?
> 
> It could be overloaded to return the srcObject. I'm not sure that's useful
> in practice.
> 
> I don't feel all that strongly about this issue --- after all, I initially
> supported overloading the 'src' attribute --- but I *do* have a strong
> feeling of annoyance about having had this discussion a long time ago, being
> pulled in one direction by one set of people, and now being pulled in a
> different direction by a different set of people (some of whom work at the
> same company as some of the previous set of people).

I'm sorry. I don't have a strong opinion one way or the other about this. I'm just trying to understand and participate constructively.

I'm not trying to pull particularly, I'm just trying to understand the design, and make sure we end up with the best one. This seems like my job, as editor of the spec in question. :-)

Who should we be cc'ing to get the explanation as to why it's better to have two attributes than one, here?

I think you may need to corner Adam Barth in his office :-)

Adam said he didn't have time to think about this.

roc, what's your implementation experience been like? Do you have any insight on whether authors find srcObject intuitive?

srcObject is definitely better for developers than createObjectURL which it replaced. We didn't get a chance to compare it to overloaded 'src' because we never shipped that in a release. srcObject is at least adequate; AFAIK no developers have asked "why didn't you just overload 'src'?".

One thing to think about is how this would extend to other element types. For example there are use-cases where having the functionality of srcObject on <img> would be useful, so you can use a Blob with an <img> without the GC hassles of createObjectURL. Would we be OK with overloading 'src' on <img> and other such places? Would we prefer to avoid adding an extra attribute in those cases?

One thing in favor of overloading 'src' is that back when this was first considered our only option in WebIDL was to make 'src' an "any" type, and now WebIDL has union types which makes this a lot cleaner.

Whatever you want to do here, please run it by public-media-capture and any other group that might be interested to reduce the likelihood of this being controversial or being reversed again. If you go with overloading 'src', no doubt I will again find myself having to explain and defend the decision in the future, but I'd at least like to try to minimize that.

It's probably a bit crazy, but one possibility would be to have Blob objects toString to a URL, so you could do:

   video.src = blob;

...and it would really be equivalent to:

   video.src = URL.createObjectURL(blob);

We could even make this URL autorevoke in the script cleanup steps. It wouldn't handle use cases like srcset="" where you need the URL later, but then neither does srcObject.

Probably a bit too crazy though.


I think I'm leaning towards just speccing srcObject as-is.

Media elements can have an "assigned MediaStream source object".

Wherever we call "concept-media-load-algorithm", skip the call if the media element has an "assigned MediaStream source object".

On getting, media.srcObject returns the "assigned MediaStream source object".

On setting, media.srcObject sets the "assigned MediaStream source object" and then invokes "concept-media-load-algorithm".

Add a new mode to "concept-media-load-algorithm", /object/.

The rest of the logic from here should be adopted:
http://dev.w3.org/2011/webrtc/editor/getusermedia.html#mediastreams-as-media-elements

(Though probably not verbatim, there's lots of stuff there that makes no sense as written, e.g. "must fail" is not meaningful.)


Why does MediaStream disallow buffering? It seems like if a browser wants to implement buffering, it should be allowed. What if I want to see what someone was saying a few seconds ago?

We want srcObject to take Blobs and MediaSources as well as MediaStreams.

(In reply to Ian 'Hixie' Hickson from comment #15)
> Why does MediaStream disallow buffering? It seems like if a browser wants to
> implement buffering, it should be allowed. What if I want to see what
> someone was saying a few seconds ago?

AFAIK no-one wants to implement that. It's tricky to implement that and also hit the minimize latency requirements. Maybe you could spec it in a way that lets the UA choose no buffering.

Yeah, I'm certainly not suggesting we should require it. But if one day a browser decides that they should allow it, seems weird for the spec to just ban it. The whole API is designed around making this a UA decision for all streams.

Yeah. FWIW I didn't write any of the text in that spec.

I preferred the overloading design. I think at this point I'm fine with either. As a data point, URLs now have .search (string) and .searchParams (object).

Can the editors of the Media Capture and Streams specification confirm that they want me to specify these features in the HTML spec and that they'll subsequently be removed from the Media Capture and Streams spec?

(In reply to Ian 'Hixie' Hickson from comment #20)
> Can the editors of the Media Capture and Streams specification confirm that
> they want me to specify these features in the HTML spec and that they'll
> subsequently be removed from the Media Capture and Streams spec?

Please do. We'll remove the feature when we are able to reference it from it's new location.

I started working on this, but:

Why is MSE defining a mechanism for attaching to an object that defines a media resource that is different than the mechanism Media Capture and Streams defines? It seems there's a huge amount of overlap in these two mechanisms.

It seems like it'd be ideal if these two mechanisms were merged into one, and that srcObject worked for the union of those.

Can you be more specific? Honestly I don't think there's that much overlap between MediaStreams and MediaSources.

Maybe you're suggesting a MediaSource should generate a MediaStream to be fed into srcObject? That doesn't work since, among other things, a MediaStream is not seekable.

There are HTTP-delivered streams that aren't seekable too, I don't see why the that would be an issue.

MediaStreams and MediaSources are both objects that consist of multiple tracks, and that can be plugged into a <video> element. That seems like a lot of commonality right there. Shouldn't they inherit from a common interface? Why should you not be able to take a video stream from a local camera and an audio stream from a static file and merge them into one object that you then play back? Or send to a remote endpoint?

(In reply to Ian 'Hixie' Hickson from comment #24)
> There are HTTP-delivered streams that aren't seekable too, I don't see why
> the that would be an issue.

The problem is that MediaSources *are* seekable so going MediaSource -> MediaStream -> HTMLMediaElement would break seeking.

> MediaStreams and MediaSources are both objects that consist of multiple
> tracks, and that can be plugged into a <video> element. That seems like a
> lot of commonality right there. Shouldn't they inherit from a common
> interface?

Since MediaSource references AudioTrack/VideoTrack/TextTrack instead of MediaStreamTrack, if we add any common interface for track-owners it should first be a common interface for HTMLMediaElement and MediaSource. If we made MediaStream share that interface, that'd be great, but we'd have to unify MediaStreamTrack with AudioTrack/VideoTrack/TextTrack first.

> Why should you not be able to take a video stream from a local
> camera and an audio stream from a static file and merge them into one object
> that you then play back? Or send to a remote endpoint?

You can, you just have to go MediaSource -> HTMLMediaElement -> MediaStream to get the MediaStream; then you can merge tracks and send to a remote endpoint or to another media element for combined playback (if you need to; in this case it would be simpler to just play the audio in one element and play the video in another element and not use MediaStreams at all).

HTMLMediaElement is considerably richer than MediaStream. As well as seeking which I already mentioned, it manages compressed resource data, decodes that data, and exposes state related to that (e.g. readystate, loadedmetadata, buffered). MediaSource is meant to represent a compressed media resource that happens to be provided by script instead of by normal resource fetching. MediaStream

Oops.

MediaSource represents a pre-decoding media resource that is fetched/generated dynamically by script and can be manipulated in most of the ways any media resource can (e.g. seeking). MediaStream represents a post-decoding stream of media data that is produced and consumed in real time only.

ISTM there are only two kinds of commonality here between MediaSource and MediaStream:
1) They both manage lists of tracks. But the track interfaces currently aren't the same. I agree we should maybe fix that and then MediaSource, MediaStream and HTMLMediaElement could all have a common "TrackOwner" interface.
2) They can both be used as the input to an HTMLMediaElement. We could have a common interface for that, but I can't think of any API it would offer other than being usable for srcObject. So I think a WebIDL union for srcObject is simpler.

Of course, I'm not actually responsible for either MediaSource or MediaStream, so the people you need to convince are not on this bug.

> MediaStream represents a post-decoding stream of media data that is produced and 
> consumed in real time only

I don't think that's accurate. There's nothing that should prevent a browser from buffering content it receives on a MediaStream, just like it could DVR content received from a TV receiver card (with src="tv:4" or something).


> Of course, I'm not actually responsible for either MediaSource or
> MediaStream, so the people you need to convince are not on this bug.

Ok. I sent a mail to the relevant lists.

http://lists.w3.org/Archives/Public/public-html-media/2014Jun/thread.html#msg14

I've tried to spec srcObject. Please file new bugs for any issues with the definition. Thanks. As specced, it allows MediaStream, MediaSource, File, and Blob.

Checked in as WHATWG revision r8716.
Check-in comment: Define <video>.srcObject
http://html5.org/tools/web-apps-tracker?from=8715&to=8716