Apr 16, 2010

HTML5 Video on the iPad—Overcoming Missing Events

Ken Hanson's picture
Ken Hanson
Design Engineer
24 comments

This is the second post in a series about how the Hot Studio engineering team tackled a set of interactive content modules for Zinio’s new iPad application.

In the previous post, I gave you a little bit of background—we were essentially creating an entirely new magazine experience—and detailed some of the key challenges we faced.

In this post, I’ll dive into more details by talking about volume controls and how we got creative with the events Mobile Safari gives you.

About Volume Controls

Our user-experience designers are always concerned with how a user will interact with our applications and what sort of scenarios they will run into, no matter the device or medium.

The iPad is no exception. Matthew Carlson, the design lead, immediately felt we needed to have volume controls, even though the original player did not. His logic was that you could be in any orientation when a video starts. This didn’t matter on the iPhone and iPod Touch because they were so small, but now we’re talking about a fairly large device and having to rapidly hit those volume controls on a plane or in a doctor’s office or some other closed, quiet space. And those controls could be on the bottom, side, top, or natural position. Instead of having to find them, you should just be able to use your finger to adjust the volume right where your eyes are already looking. Sound logic, Matt!

So we designed it, developed it, and then tested it. And tested it. And double triple tested it. In the end, the volume property was definitely being adjusted. My math was sound. But no matter what, even on the real device, we could not get the iPad volume to adjust.

Edit: This is what Apple's Vicki Murley responded to me with.

Sorry, it's not possible to programmatically change the volume for <audio> or <video> in Safari on iPad. This is by design.

Hopefully, someday the Mobile Safari team will add the ability to have volume controls. When or if that’s possible, here are the basic things you need to address so you can roll your own:

The property you’re going to touch is:

volume (float): The current volume - from 0.0 to 1.0

Here are the basics:

var player = document.getElementById('player')
player.volume = .5; // Half the full volume. Volume is 0.0 - 1.0.

It can definitely get more difficult, but you just talk to the volume attribute. It’s simple enough. It’s so simple that you really can’t mess it up. But like I said, even though it’s so simple, no matter what we tried, the iPad volume would not adjust. And nobody is talking about why Edit: See above for Apple's response. So we ended up bailing on the idea, resigning ourselves to seeing users frantically search for the volume rocker in the stone quiet reception areas.

Creating the Scrubber

You’re probably thinking, “A scrubber? Ah, well, there are definitely events that are perfect for that! Safari Dev says so!” I thought so, too! Turns out that quite a few of these great events that the player will radio back and forth about don’t actually work in Mobile Safari. This became especially challenging for the buffer amount indicator (shown below).

Video on the iPad: Show custom buffer bars.

In particular, I wanted to use the “progress” event:

progress - Browser loads data

It seemed perfect! “Browser, you tell me when you load new data, and I’ll adjust the loaded scrubber ok?” But Mobile Safari never actually responds back about this. You’ll wait all day for that one. So we ended up watching for this event initially:

canplay - Video can be played, but possibly must stop to buffer content

But we found that this could result in a delay, and nothing happening makes users impatient. And when they get impatient, they start smashing things (at least, I do). And we definitely don’t want that.

So we went another route and started watching for the “durationchange” event:

durationchange - Duration has changed (for streams)

This event actually fires once the first scrap of (meta) data is in the pipe, and from there we can monitor it, albeit with polling, as inefficient as that is. (Polling wouldn’t be necessary if Apple had hooked us up with the “progress” event like we had hoped. But, alas, it was not to be.)

So here’s the code block where we used “durationchange,” monitored it with polling, and also checked for an internet connection.

player.addEventListener('durationchange', function(event){
updateDuration(); // we use this function to talk to the duration numbers
checkSourceForConnect();
}, false);

The checkSourceForConnect() function will fire off a check for internet connections (all videos are typically streamed from Zinio’s CDN), and then, if all goes well, we move on to this function:

function getBuffered() {
var bufferedPercent = parseInt(((player.buffered.end(0) / player.duration) * 100));
$('#currentTrack').css({'width' : bufferedPercent + '%', '-webkit-transition' : 'width .3s ease-out'});
$('body').delay(300, function(){
if(bufferedPercent < 100) {
getBuffered();
}
});
}

Here’s what’s happening inside of getBuffered():

Now that we’re ready for retrieving a buffer amount, we grab the current buffered percentage. This is represented by the var bufferedPercent. The math translates it out into a round 0 – 100 number. You can read about the various properties inside that math here: HTML 5 Video—DOM Attributes and Events

Next, we apply the bufferedPercent to the CSS, so that our buffer bar stretches across. At the same time, we apply a CSS transition to width, and every time we update the transition property, it will animate out to the new width.

After that’s done, we relax for 300 milliseconds (the time it takes for the original animation to finish), and give it another go if we’re not completely finished buffering. Note: This is done with the delay plugin for jQuery. We liked it more than setTimeouts.

The effect ends up being pretty classy. It’s kind of a ease-in / ease-out animation that springs into action. It makes it seem like your download is having bursts toward the finish line. Or… that’ s how I thought it looked. I’m sure our designers have some better buzzwords to describe the effect.

Here's a quick clip of it in action (on the iPad simulator):

Whew! If you’ve made it this far, congrats! Before your head explodes (these posts are getting longer and longer), I’ll save some for the next post, where I’ll talk about how we got fullscreen (or pseudo fullscreen), track down exactly why Apple’s not a fan of autoplay (and why it sort of makes sense), and how we went about making our audio player from the video player.

I’m particularly excited to write about the fullscreen jazz we’ve done, as it let’s us use an unbridled amount of CSS3 Transitions and Transforms that Mobile Safari supports.

Until next time!

This post is the second post in a series. Check out the first post here:
HTML5 Video on the iPad—5 Essential Lessons

Because you're a Hottie, please log in before commenting:

24 comments

Bob's picture

Hey, did you ever post the third article?

DouG Molidor's picture

Thank you for the post. It really helped me understand the mobile safari personality.

Have you been able to capture 'volumechange'? It seems you've dug deep enough to know and I would love to read about more of your discoveries.

I too am having trouble wrangling events and manipulations surrounding volume on iOS devices. I can't for the life of me capture the 'volumechange' event in Mobile Safari. I found a very helpful demo set of files, written by Apple, that demos the HTML5 Video Event flow, and that can't even capture 'volumechange' on iOS. Apple's docu claims that the event is available in iOS 3.0 and up, but if their own demo, and my 8 hours of testing, can't capture anything. I've run out of hope.

I've even tried polling the volume property from the video element in the 'progress' event handler, but that always comes back as 1, no matter where the volume is set.

Andrew's picture

Great articles - I'm struggling with the same issues this week. Any plans to release that promised 3rd article?

I'd love to hear about how you got a pseudo-fullscreen player to work.

HTML5 Video on the iPad – 5 Essential Lessons | ItPak.net's picture

[...] it for now. In the next post, I’ll walk you through our experience with the volume controls and talk about the challenges we [...]

Mat Says » HTML5 Video for iPad (p3)'s picture

[...] video is not supported on iPad at all (thanks Ken Hanson for helping me out of the frustration and leading me back in the right direction). iStreamPlanet HTML5 Video for iPad solution with dynamic stream [...]

Brad's picture

Haha, I hear you. I've actually been playing around a lot today and noticed that 'durationchange' and 'loadeddata' were not always firing on page load. It was quite strange. Turned out, that if you run the load() method too soon, nothing ends up firing. From my tests, the safest time to run the load() method is when the networkstate is 1 (IDLE) and the readyState is 0 (Have Nothing). So I just run a little poll until those two states hit 1 and 0 and then fire my load method off. Seems to be working every time now, despite using 'durationchange' or 'loadeddata'.

Totally agree with you about the wild wild west though!

Ken Hanson's picture

I dunno, I think it seems to be firing when it should Brad.

It's built for streaming, so that when the length of the available stream changes, you know about it.

If it can tell me when its stream length has changed, then I can check and make sure its not zero, and do some work on the visuals of the controls.

A great example would be if we actually lose some length. We need to recheck everything and find out how long that buffer bar is actually supposed to be.
[Edit: changed my response here to actually make sense ;)]

Your right to be concerned, but in that regard, everything we're doing right now should be causing concern, this is the wild west after all ha!

Nothings actually got a standard right now lol. We just have best guesses from Apple =p

Brad's picture

Hey Ken, I did some testing as well and you're right about the duration change firing first. I like it, my only concern is does it seem like an event that should be firing on page load? I'm worried that in later version the code may break as it doesn't seem like something that should fire when it currently is. I dunno, just some of my thoughts.

Ken Hanson's picture

@Caleb definitely gonna check out that wired app (how horrible is it that I haven't!), but my guess is that they're setting width / height and such.

The way we do it in a nutshell, is to use -webkit-transform: scale() on the video, and then to use scaling math to keep the container proportionate, and then absolute position all four edges (you can do that when you do have to worry about IE6!!), so that all four points get 0 values (top: 0px; left: 0px; right:0px; bottom:0px;) for the overlay div (the guy that hides the content behind it). Clear as mud right? ha =p

Whats great about the absolute position jazz is that we don't have to worry about the width / height of the body, because it just sticks to the edges the entire time!

@Yogi Thanks so much for the kind words! I'm currently in the middle of a career shift, and am heading out to open up my own engineering shop next month, but one of my top priorities before heading out is to get that post #3 written and posted! It's a good problem to have, but we've been SO slammed with mobile work! Keep an eye out, I'll update here so you know it happened and thanks for being so patient!

Ken Hanson's picture

@Brad great suggestion Brad! The problem still exists though, because loadeddata only fires once the data is loaded. Doesn't really give us a good snapshot of buffering, and doesn't happen more often than right at the beginning.

The event in fact registers though, so its definitely swappable with our duration change.

I did some testing and durationchange happens way before loadeddata (because well... data needs to finish loading of course!)

I like using durationchange because with either event listener, we're going to need to poll, and if we gotta poll no matter what, we may as well run with something that starts earlier so we see progress sooner.

Check out: http://drp.ly/1t7s1F (pretty much my bible for events and such involving html5 video). The ideal event from there is "progress", because it actually fires multiple times. It'd be way more accurate than our delay and check polling.

I think that loadeddata, and in fact our solution (durationchange), miss the mark because its not truly representative of well... loading data! haha

Was hoping to see 4.0 bring "progress" events, but still no luck.

Thanks again @Brad, always appreciate the suggestions!

Brad's picture

Have you guys looked into using 'loadeddata' as your event listener? I know this was posted back on 4.16.2010, but you're getting comments in the month of July so people are turning to your resource.

I think that event will trigger properly if you do:

window.onload = function(){iVideo = document...;iVideo.load();};

Yogi's picture

Great series guys, thanks for confirming some of my supposed bugs! When's post #3? I hope you guys have the time to share more of your iPad experience. Thanks!

Caleb's picture

Great insights guys. Thanks for sharing the knowledge. I'm very interested to see how you do full-screen mode for ipad on load though, like how they do it on the ads within wired magazine.

Ken Hanson's picture

Hey Rodger, thanks for swinging by and letting us know! I'll definitely edit the post to include this information.

The event your specifically talking about is the progress event (as opposed to the volume controls), correct?

Thanks!

Rodger Combs's picture

The missing events are actually not a MobileSafari issue. The version of WebKit that 3.2's Safari is based on didn't contain those events. They have been added in newer versions, but 3.2 still has the old version. Wait for 4.0 (or 3.2.1, if it's to exist)

MArk's picture

Thanks for suggestions. I already tried -webkit-transform-style before but it didnt help. At this point im just starting with a very simple test case. I use jquery to create video object and menu div consisting div cells for buttons. video and menu div is added on the fly depending on set parameter. Menu suppose to appear higher in z axis than the video object. Right now im using just a few simple buttons to cycle thru my set of videos/menus. On click previous video and associated menu is erased from parent container and then new video and menu objects are constructed and placed in the same parent container. In every browser even in Safari on my desktop the menu is displayed on top of the video and works as intended but on iPad it gets stuck behind video frame.

Ken Hanson's picture

@Mark, try using translate3d, it gives you the z-index like effect (but really because its actually in 3d space) and works really reliably.

I'm stabbing in the dark, because I haven't reproduced your problem, but lemme know if that works out.

It's really weird that the generated dom element doesn't pick up the z-index. Are you sure that your injecting the div in the right spot?

Here's the translate3d bit.

-webkit-transform-style: preserve-3d;
-webkit-transform: translate3d(0, 0, 10px);

The 3rd number is the Z-Axis. You'll push your overlay 10px "above" the video tag, and then do the same for the video tag, just with 0 so its resting on the initial baseline.

MArk's picture

Cant wait for the 3rd part..very nice series. I have a question: did you use any div containers as overlays on top of dynamically created video tags. Currently i am struggling to get them to show up on IPad on top of video elements when i use javascript to manipulate html objects. It looks like it is possible to have properly visible overlays on top of video tags when you hard code them to be like that with html and css, but once you start injecting dynamically created video tags into existing html and try to create overlaid elements IPad doesn't want to respect z-order and serves video always on top of everything else. Any solution to that?

Ray Matos's picture

@Ken Hanson : Awesome. I also use the JS width and height, i found the biggest issue was having the video return to the original space when the user rotated the device. Since the landscape vs portrait view were styled differently.

Ken Hanson's picture

Hey Thanks all!

@Jeroen & @Ray Matos : The quick and dirty was JS Width & Height / CSS Transitions first. Overcoming the crazy animation jitters though was another problem entirely. We found that, nearly all the CSS Transforms performed leaps and bounds better. Eye poppingly better. It went from a stuttery jittery mess, to smooooth as butter ;) There are loads of challenges surrounding the transforms though.

I'll be posting code samples of that this week in the 3rd post in the series. I'll also post back here so you know its up if you subscribe to comment notifications.

Thanks so much everyone! Glad you like the limited info I was able to share haha.

Renato's picture

Great to see more documentation on iPad and Html5 video. Very helpful post! Please continue!

limewire's picture

dang sweet info dude.

Jeroen's picture

Thanks for the interesting articles in this series. I'm particularly interested in how you solved the missing fullscreen functionality. Really looking forward to your next article !

Ray Matos's picture

Great work. I have too had to deal with custom controls and making my own full screen, i'm curious to see how you did it. You can see mine in the textplus app.

Post new comment