Blog

So, one of the cool things we aimed for with Buttercup was ensuring that we not only worked well with screen readers, but, that we worked well without screen readers too. If you think about scenarios like the public library, chances are they will not have a screen reader installed but Silverlight is a much better probability.

To achieve this we wanted to support the idea of a self voicing UI. We did this, but, it comes with quite a few caveats. This includes the fact that it only works in Internet Explorer and even then you’ve got to do some security acrobatics to enable the Active X calls that we make.

I’ve been kicking around a few ideas of how we might improve the self voicing experience. I’m big on doing this because it’s inherently re-usable functionality. if we can get cross browser self voicing working it’ll allow us to ‘Self Voice’ not only Silverlight applications but also plain old HTML apps as well.

So, some of the thoughts I’ve had:

• Pre recorded MP3s for UI elements. Basically we’ll create an MP3 to be played by the UI when you button hover or field focus.
  This is pretty easy to achieve, but, it has a few draw backs too. We will only be able to support static speech- if you run a search in Buttercup today we’ll actually speak the result count back to you. Using pre-recorded MP3s is also a fairly time consuming process- we’ll either need to use voice talent, or generate each MP3 using a synthesizer and package it up. Sure we could probably build a tool that used reflection to interrogate the UI Automation tags and then created the speech files. But, overall this still feels a bit fugly to me.
• Calling up to the server. We could make a web service call to the server passing up the speech we’d like generated. We could then use one of the variety of Text to Speech generators server side to create a speech MP3 to be returned to the client. We’d obviously do some caching on the client and probably push this cached data into isolated storage to persist across runs. This gets us around the dynamic generation limitations from above, but, gives us a server side dependency which is something we’ve thus far managed to avoid in Buttercup- we really want people to just be able to grab the Buttercup *.xap file and stick it on their site. Making server calls doesn’t really suit this model. I’m also not 100% on whether the latency will be tolerable- realistically we’re probably talking 3000ms at a minimum. I think if we did go this way it’s probably only be a stop gap and we might just run the speech gen service in say Windows Azure.
• Generating the Speech in Silverlight. This is kind of the nirvana. In Silverlight 2.0 (which is what Buttercup is currently built in) we just can’t do it. But, Silverlight 3 we have the ability to use the new MediaStreamSource and this might just let us do some sort of Audio generation. I think that we’d probably need to look at porting an existing open source TTS engine to C# managed code and then surfacing this into Silverlight. I’d love to get feedback from anyone who has had good or bad experiences with any of the open source TTS engines.

Now that 3rd point got me thinking about another commonly requested feature- variable speed playback. The Silverlight 3 Beta doesn’t support variable speed MP3 playback and it doesn’t look like it’s a feature that we’ll get in this release. It might be possible, though, to build a custom MP3 decoder to support this feature- maybe a port of MP3Sharp.

Anyway, that’s my braindump for today. I know that Andrew Tokeley will have some interesting samples to post after his Twilight Session tomorrow. I’d love to hear in the comments if people have other ideas for the above problems.