HTML Speech Incubator Group Final Report (Internal Draft)

3.1.1 Strong Interest

A requirement was classified as having "strong interest" if at least 80% of the group believed it needs to be addressed by any specification developed based on the work of this group. These requirements are:

• FPR40. Web applications must be able to use barge-in (interrupting audio and TTS output when the user starts speaking).
• FPR4. It should be possible for the web application to get the recognition results in a standard format such as EMMA.
• FPR24. The web app should be notified when recognition results are available.
• FPR50. Web applications must not be prevented from integrating input from multiple modalities.
• FPR59. While capture is happening, there must be a way for the web application to abort the capture and recognition process.
• FPR52. The web app should be notified when TTS playback finishes.
• FPR60. Web application must be able to programatically abort tts output.
• FPR38. Web application must be able to specify language of recognition.
• FPR45. Applications should be able to specify the grammars (or lack thereof) separately for each recognition.
• FPR1. Web applications must not capture audio without the user's consent.
• FPR19. User-initiated speech input should be possible.
• FPR21. The web app should be notified that capture starts.
• FPR22. The web app should be notified that speech is considered to have started for the purposes of recognition.
• FPR23. The web app should be notified that speech is considered to have ended for the purposes of recognition.
• FPR25. Implementations should be allowed to start processing captured audio before the capture completes.
• FPR26. The API to do recognition should not introduce unneeded latency.
• FPR34. Web application must be able to specify domain specific custom grammars.
• FPR35. Web application must be notified when speech recognition errors or non-matches occur.
• FPR42. It should be possible for user agents to allow hands-free speech input.
• FPR48. Web application author must be able to specify a domain specific statistical language model.
• FPR54. Web apps should be able to customize all aspects of the user interface for speech recognition, except where such customizations conflict with security and privacy requirements in this document, or where they cause other security or privacy problems.
• FPR51. The web app should be notified when TTS playback starts.
• FPR53. The web app should be notified when the audio corresponding to a TTS element is played back.
• FPR5. It should be easy for the web appls to get access to the most common pieces of recognition results such as utterance, confidence, and nbests.
• FPR39. Web application must be able to be notified when the selected language is not available.
• FPR13. It should be easy to assign recognition results to a single input field.
• FPR14. It should not be required to fill an input field every time there is a recognition result.
• FPR15. It should be possible to use recognition results to multiple input fields.
• FPR16. User consent should be informed consent.
• FPR18. It must be possible for the user to revoke consent.
• FPR11. If the web apps specify speech services, it should be possible to specify parameters.
• FPR12. Speech services that can be specified by web apps must include network speech services.
• FPR2. Implementations must support the XML format of SRGS and must support SISR.
• FPR27. Speech recognition implementations should be allowed to add implementation specific information to speech recognition results.
• FPR3. Implementation must support SSML.
• FPR46. Web apps should be able to specify which voice is used for TTS.
• FPR7. Web apps should be able to request speech service different from default.
• FPR9. If browser refuses to use the web application requested speech service, it must inform the web app.
• FPR17. While capture is happening, there must be an obvious way for the user to abort the capture and recognition process.
• FPR37. Web application should be given captured audio access only after explicit consent from the user.
• FPR49. End users need a clear indication whenever microphone is listening to the user.

3.1.2 Moderate Interest

A requirement was classified as having "moderate interest" if less than 80% but at least 50% of the group believed it needs to be addressed by any specification developed based on the work of this group. These requirements are:

• FPR33. There should be at least one mandatory-to-support codec that isn't encumbered with IP issues and has sufficient fidelity & low bandwidth requirements.
• FPR28. Speech recognition implementations should be allowed to fire implementation specific events.
• FPR41. It should be easy to extend the standard without affecting existing speech applications.
• FPR36. User agents must provide a default interface to control speech recognition.
• FPR44. Recognition without specifying a grammar should be possible.
• FPR61. Aborting the TTS output should be efficient.
• FPR32. Speech services that can be specified by web apps must include local speech services.
• FPR47. When speech input is used to provide input to a web app, it should be possible for the user to select alternative input methods.
• FPR56. Web applications must be able to request NL interpretation based only on text input (no audio sent).
• FPR30. Web applications must be allowed at least one form of communication with a particular speech service that is supported in all UAs.
• FPR55. Web application must be able to encrypt communications to remote speech service.
• FPR58. Web application and speech services must have a means of binding session information to communications.
• FPR6. Browser must provide default speech resource.
• FPR20. The spec should not unnecessarily restrict the UA's choice in privacy policy.

3.1.3 Mild Interest

A requirement was classified as having "mild interest" if less than 50% of the group believed it needs to be addressed by any specification developed based on the work of this group. These requirements are:

• FPR29. Speech synthesis implementations should be allowed to fire implementation specific events.
• FPR31. User agents and speech services may agree to use alternate protocols for communication.
• FPR43. User agents should not be required to allow hands-free speech input.
• FPR10. If browser uses speech services other than the default one, it must inform the user which one(s) it is using.
• FPR8. User agent (browser) can refuse to use requested speech service.
• FPR57. Web applications must be able to request recognition based on previously sent audio.

3.3.1 General Design Decisions

1. There are three aspects to the solution which must be addressed: communication with and control of speech services, a script-level API, and markup-level hooks and capabilities.
2. The script API will be Javascript.
3. The scripting API is the primary focus, with all key functionality available via scripting. Any HTML markup capabilities, if present, will be based completely on the scripting capabilities.
4. Notifications from the user agent to the web application should be in the form of Javascript events/callbacks.
5. For ASR, there must at least be these three logical functions:
   1. start speech input and start processing
   2. stop speech input and get result
   3. cancel (stop speech input and ignore result)
6. For TTS, there must be at least these two logical functions:
   1. play
   2. pause
   There is agreement that it should be possible to stop playback, but there is not agreement on the need for an explicit stop function.
7. It must be possible for a web application to specify the speech engine.
8. Speech service implementations must be referenceable by URI.
9. It must be possible to reference ASR grammars by URI.
10. It must be possible to select the ASR language using language tags.
11. It must be possible to leave the ASR grammar unspecified. Behavior in this case is not yet defined.
12. The XML format of SRGS 1.0 is mandatory to support, and it is the only mandated grammar format. Note in particular that this means we do not have any requirement for SLM support or SRGS ABNF support.
13. For TTS, SSML 1.1 is mandatory to support, as is UTF-8 plain text. These are the only mandated formats.
14. SISR 1.0 support is mandatory, and it is the only mandated semantic interpretation format.
15. There must be no technical restriction that would prevent using only TTS or only ASR.
16. There must be no technical restriction that would prevent implementing only TTS or only ASR. There is *mostly* agreement on this.
17. There will be a mandatory set of capabilities with stated limitations on interoperability.
18. For reco results, both the DOM representation of EMMA and the XML text representation must be provided.
19. For reco results, a simple Javascript representation of a list of results must be provided, with each result containing the recognized utterance, confidence score, and semantic interpretation. Note that this may need to be adjusted based on any decision regarding support for continuous recognition.
20. For grammar URIs, the "HTTP" and "data" protocol schemes must be supported.
21. A standard set of common-task grammars must be supported. The details of what those are is TBD.
22. The API should be able to start speech reco without having to select a microphone, i.e., there must be a notion of a "default" microphone.
23. There should be a default user interface.
24. The user agent must notify the user when audio is being captured. Web applications must not be able to override this notification.
25. It must be possible to customize the user interface to control how recognition start is indicated.
26. If the HTML standard has an audio capture API, we should be able to use it for ASR. If not, we should not create one, and we will not block waiting for one to be created.
27. We will collect requirements on audio capture APIs and relay them to relevant groups.
28. A low-latency endpoint detector must be available. It should be possible for a web app to enable and disable it, although the default setting (enabled/disabled) is TBD. The detector detects both start of speech and end of speech and fires an event in each case.
29. The API will provide control over which portions of the captured audio are sent to the recognizer.
30. We expect to have the following six audio/speech events: onaudiostart/onaudioend, onsoundstart/onsoundend, onspeechstart/onspeechend. The onsound* events represent a "probably speech but not sure" condition, while the onspeech* events represent the recognizer being sure there's speech. The former are low latency. An end event can only occur after at least one start event of the same type has occurred. Only the user agent can generate onaudio* events, the energy detector can only generate onsound* events, and the speech service can only generate onspeech* events.
31. There are 3 classes of codecs: audio to the web-app specified ASR engine, recognition from existing audio (e.g., local file), and audio from the TTS engine. We need to specify a mandatory-to-support codec for each.
32. It must be possible to specify and use other codecs in addition to those that are mandatory-to-implement.
33. Support for streaming audio is required -- in particular, that ASR may begin processing before the user has finished speaking.
34. It must be possible for the recognizer to return a final result before the user is done speaking.
35. We will require support for http for all communication between the user agent and any selected engine, including chunked http for media streaming, and support negotiation of other protocols (such as WebSockets or whatever RTCWeb/WebRTC comes up with).
36. Maxresults should be an ASR parameter representing the maximum number of results to return.
37. The user agent will use the URI for the ASR engine exactly as specified by the web application, including all parameters, and will not modify it to add, remove, or change parameters.
38. The scripting API communicates its parameter settings by sending them in the body of a POST request as Media Type "multipart". The subtype(s) accepted (e.g., mixed, formdata) are TBD.
39. If an ASR engine allows parameters to be specified in the URI in addition to in the POST body, when a parameter is specified in both places the one in the body takes precedence. This has the effect of making parameters set in the URI be treated as default values.
40. We cannot expect consistency in language support and performance/quality.
41. We agree that there must be API-level consistency regardless of user agent and engine.
42. We agree on having the same level of consistency across all four of the following categories:
    1. consistency between different UAs using their default engine
    2. consistency between different UAs using web app specified engine
    3. consistency between different UAs using different web specified engines
    4. consistency between default engine and specified engines
    With exception that #4 may have limitations due to privacy issues.
43. From this point on we will use "service" rather than "engine" because a service may be a proxy for more than one engine.
44. We will not support selection of service by characteristics.
45. Add to list of expected inconsistency (change from existing wording of interoperability): reco performance including maximum size on parameters, microphone characteristics, semantics and exact values of sensitivity and confidence, time need to perform ASR/TTS, latencies, endpoint sensitivity and latency, result contents, presence/absence of optional events, recorded waveform
46. For continuous recognition, we must support the ability to change grammars and parameters for each chunk/frame/result
47. If the user's device is emitting other sounds than those produced by the current HTML page, there is no particular requirement that the User Agent be required to detect/reduce/eliminate it.
48. If a web app specifies a speech service and it is not available, an error is thrown. No automatic fallback to another service or the default service takes place.
49. The API should provide a way to determine if a service is available before trying to use the service; this applies to the default service as well.
50. The API must provide a way to query the availability of a specific configuration of a service.
51. The API must provide a way to ask the user agent for the capabilities of a service. In the case of private information that the user agent may have when the default service is selected, the user agent may choose to answer with "no comment" (or equivalent).
52. Informed user consent is required for all use of private information. This includes list of languages for ASR and voices for TTS. When such information is requested by the web app or speech service and permission is refused, the API must return "no comment" (or equivalent).
53. It must be possible for user permission to be granted at the level of specific web apps and/or speech services.
54. User agents, acting on behalf of the user, may deny the use of specific web apps and/or speech services.
55. The API will support multiple simultaneous grammars, any combination of allowed grammar formats. It will also support a weight on each grammar.
56. The API will support multiple simultaneous requests to speech services (same or different, ASR and TTS).
57. We disagree about whether there needs to be direct API support for a single ASR request and single TTS request that are tied together.
58. It must be possible to individually control ASR and TTS.
59. It must be possible for the web app author to get timely information about recognition event timing and about TTS playback timing. It must be possible for the web app author to determine, for any specific UA local time, what the previous TTS mark was and the offset from that mark.
60. It must be possible for the web app to stop/pause/silence audio output directly at the client/user agent.
61. When audio corresponding to TTS mark location begins to play, a Javascript event must be fired, and the event must contain the name of the mark and the UA timestamp for when it was played.
62. It must be possible to specify service-specific parameters in both the URI and the message body. It must be clear in the API that these parameters are service-specific, i.e., not standard.
63. Every message from UA to speech service should send the UA-local timestamp.
64. API must have ability to set service-specific parameters using names that clearly identify that they are service-specific, e.g., using an "x-" prefix. Parameter values can be arbitrary Javascript objects.
65. EMMA already permits app-specific result info, so there is no need to provide other ways for service-specific information to be returned in the result.
66. The API must support DOM 3 extension events as defined (which basically require vendor prefixes). See http://www.w3.org/TR/2009/WD-DOM-Level-3-Events-20090908/#extending_events-Vendor_Extensions. It must allow the speech service to fire these events.
67. The protocol must send its current timestamp to the speech service when it sends its first audio data.
68. It must be possible for the speech service to instruct the UA to fire a vendor-specific event when a specific offset to audio playback start is reached by the UA. What to do if audio is canceled, paused, etc. is TBD.
69. HTTPS must also be supported.
70. Using web app in secure communication channel should be treated just as when working with all secured sites (e.g., with respect to non-secured channel for speech data).
71. Default speech service implementations are encouraged not to use unsecured network communication when started by a web app in a secure communication channel
72. In Javascript, speech reco requests should have an attribute for a sequence of grammars, each of which can have properties, including weight (and possibly language, but that is TBD).
73. In Javascript will be able to set parameters as dot properties and also via a getParameters method. Browser should also allow service-specific parameters to be set this way.
74. Bjorn's email on continuous recognition represents our decisions regarding continuous recognition, except that there needs to be a feedback mechanism which could result in the service sending replaces. We may refer to "intermediate" as "partial", but naming changes such as this are TBD.
75. There will be an API method for sending text input rather than audio. There must also be a parameter to indicate how text matching should be done, including at least "strict" and "fuzzy". Other possible ways could be defined as vendor-specific additions.
76. It must be possible to do one or more re-recognitions with any request that you have indicated before first use that it can be re-recognized later. This will be indicated in the API by setting a parameter to indicate re-recognition. Any parameter can be changed, including the speech service.
77. In the protocol, the client must store the audio for re-recognition. It may be possible for the server to indicate that it also has stored the audio so it doesn't have to be resent.
78. Once there is a way (defined by another group) to get access to some blob of stored audio, we will support re-recognition of it.
79. No explicit need for JSON format of EMMA, but we might use it if it existed.
80. Candidate codecs to consider are Speex, FLAC, and Ogg Vorbis, in addition to plain old mu-law/a-law/linear PCM.
81. Protocol design should not prevent implementability of low-latency event delivery.
82. Protocol should support the client to begin TTS playback before receipt of all of the audio.
83. We will not require support for video codecs. However, protocol design must not prohibit transmission of codecs that have the same interface requirements as audio codecs.
84. Every event from speech service to the user agent must include timing information that the UA can convert into a UA-local timestamp. This timing info must be for the occurrence represented by the event, not the event time itself. For example, an end-of-speech event would contain timing for the actual end of speech, not the time when the speech service realizes end of speech occurred or when the event is sent.

3.3.2 Speech Service Communication and Control Design Decisions

This is where design decisions regarding control of and communication with remote speech services, including media negotiation and control, will be recorded.

3.3.3 Script API Design Decisions

This is where design decisions regarding the script API capabilities and realization will be recorded.

• It must be possible to define at least the following handlers (names TBD):
  • onspeechstart (not yet clear precisely what start of speech means)
  • onspeechend (not yet clear precisely what end of speech means)
  • onerror (one or more handlers for errors)
  • a handler for when the recognition result is available
  Note: significant work is needed to get interoperability here.

3.3.4 Markup API Design Decisions

This is where design decisions regarding the markup changes and/or enhancements will be recorded.