Automated regression testing of Chromecast using Cucumber, JRuby and Rukuli

As you may know, i work as a Developer In Test for Media Playout at the BBC. We enable audio and video playout on the web (desktop, tablet and mobile) using our Standard Media Player.

We recently added support for Chromecast, and i want to write a bit about how i automated testing for it.

Chromecast HDMI dongle

Chromecast HDMI dongle

Chromecast is a HDMI dongle that plugs into a normal television and adds connected TV functionality. The neat thing is you can control it from your laptop, tablet or mobile using the media apps that you already know, such as YouTube, Netflix and now BBC iPlayer. Google also provides an API so that new Chromecast apps can be written all the time.

So how do you test a Chromecast? Well, personally i like to test things from an user’s point of view, actually clicking things and observing what happens as a result. But there’s only so far you can take this. I don’t want to rely on a television having a Chromecast being plugged in all the time and on the same network so that i can talk to it.

Though i’ve got to say .. it would be quite awesome to use a television 2 metres wide for my automated regression testing! :p

About to start casting

About to start casting

I didn’t need to test the Chromecast receiver app; that had already been written and tested by the guys in Salford for use by the iOS iPlayer app. I realised, what we actually care about is the communication between our player and the Chromecast. And there is a well documented Chromecast API. So with the help of colleagues Tim Hewitt and Wyell Hanna, i set about creating a fake Chromecast. Pretty much a bit of javascript that would respond in the same way as a real Chromecast, so that we could test how our player would respond to it.

And now i present to you .. the fake Chromecast!

https://gist.github.com/sermoa/10988494

A lot of it is empty methods that are neccessary to the API. But there are some neat tricks here too. I’ll talk you through a few of the more interesting bits.

Firstly we need to load the fake Chromecast into the page. We can achieve this with a @chromecast tag and a Before hook.

Before('@chromecast') do
  $chromecast = Chromecast.new
end

My Chromecast class has an initialize method that inserts the javascript into the page.

class Chromecast

  def initialize
    $capybara.execute_script <<-EOF
      var el=document.createElement("script");
      el.type="text/javascript";
      el.src = "https://gist.githubusercontent.com/sermoa/10988494/raw/82e08c5a29b5689b5e9f3d03c191b8c981102d85/fakeChromecast.js";
      document.getElementsByTagName("head")[0].appendChild(el);
    EOF
    @media_alive = true
  end

end

With this in place, so long as i set up our player with the relevant business logic that says casting is available, when i hover over the player, i get a cast button!

  @chromecast
  Scenario: Chromecast button appears
    Given Chromecast is available
    When I play media
    Then I should see the Chromecast button
A cast button appears!

A cast button appears!

So how does that work? I assure you, i don’t have a Chromecast on the network right now. I’m not using the Google Chrome extension. I’m actually running it in Firefox! What is this voodoo?!!

Have a look at the fakeChromecast.js file:

window.chrome = {};
window.chrome.cast = {};
window.chrome.cast.isAvailable = true;

See that? We’ve set up a Chromecast and said it’s available! Sneaky, hey? It’s that easy, folks! :)

The Standard Media Player will now attempt to set up a session request with its application ID. That’s fine: we’ll happily enable it to do its thing.

window.chrome.cast.SessionRequest = function(a) { }

Obviously a real Chromecast does something rather significant here. But we don’t have to care. This is a fake Chromecast. We only provide the function so that it doesn’t break.

Next, the media player makes a new ApiConfig. It’ll pass the session request it just obtained (in our case it’s null, but that doesn’t matter), and two callbacks, the second being a receiver listener. That’s the important one. We want to convince it that a Chromecast is available, so we trigger this callback with the special string “available”.

window.chrome.cast.ApiConfig = function(a, b, c) {
  c("available");
}

So Chromecast is available. Now suppose the user clicks the button to begin casting. This should request a session.

  @chromecast
  Scenario: Click Chromecast button and connect to a session
    Given I am playing media
    When I click the Chromecast button
    Then I should see Chromecast is connecting
Connecting to Chromecast

Connecting to Chromecast

How did we do this? Easy! The media player requests a session, sending a success callback. Fake Chromecast store a references to that callback – on the window so that we can trigger it any time we like! The callback function is expected to provide a session. We send it a reference to a fake session, which is entirely within our control. Oh it’s all so much fun!

window.chrome.cast.requestSession = function(a, b) {
  window.triggerConnectingToCC = a;
  window.triggerConnectingToCC(window.fakeSession);
}

As the documentation says, “The Session object also has a Receiver which you can use to display the friendly name of the receiver.” We want to do exactly that. We decided to call our fake Chromecast Dave. Because Dave is a friendly name! :)

window.fakeSession = {};
window.fakeSession.receiver = {friendlyName:"dave", volume:{level:0.7}};

I think i found our app expected the session receiver to supply a volume too, so i added that.

The media player does some shenanigans with media that we don’t need to care about, but when it sends a request to load media it passes its callback to trigger when media is discovered by Chromecast. That’s another important one for us to keep, so we store that one. We wait 1 second for a semi-realistic connection time, and then trigger it, passing a reference to .. fake media, woo!

window.fakeSession.loadMedia = function(a, b, c) {
  window.pretendMediaDiscovered = b;
  setTimeout(function() {
    window.pretendMediaDiscovered(window.fakeMedia);
  }, 1000);
}

And now we are almost there. The last clever trick is the communication of status updates. The media player sends us a callback it wants triggered when there is a media status update. So we store that.

window.fakeMedia.addUpdateListener = function(a) {
  window.updateCCmediaStatus = a;
}

The magic of this is, we stored the references to fakeMedia and fakeSession on the browser’s window object, as well as the callbacks. This means the test script has access to them. Therefore the test script can control the fake Chromecast.

So you want Chromecast to report playing? Make it so!

  @chromecast
  Scenario: Begin casting
    Given I am playing media
    When I click the Chromecast button
    And Chromecast state is "PLAYING"
    Then I should see Chromecast is casting
We are casting!

We are casting!

What does that mean, Chromecast state is “PLAYING”? It’s pretty straightforward now, using the objects and callback functions that the fake Chromecast has set up:

When(/^Chromecast state is "(.*?)"$/) do |state|
  $chromecast.state = state
  $chromecast.update!
end

Those two methods get added to the Chromecast Ruby class:

class Chromecast

  def initialize
    $capybara.execute_script <<-EOF
      var el=document.createElement("script");
      el.type="text/javascript";
      el.src = "https://gist.githubusercontent.com/sermoa/10988494/raw/82e08c5a29b5689b5e9f3d03c191b8c981102d85/fakeChromecast.js";
      document.getElementsByTagName("head")[0].appendChild(el);
    EOF
    @media_alive = true
  end

  def state=(state)
    @media_alive = false if state == 'IDLE'
    $capybara.execute_script "window.fakeMedia.playerState = '#{state}';"
  end

  def update!
    $capybara.execute_script "window.updateCCmediaStatus(#{@media_alive});"
  end

end

Note that i had to do something special for the “IDLE” case because the media status update expects a true/false to indicate whether the media is alive.

So, using these techniques, sending and capturing events and observing reactions, i was able to verify all the two-way communication between the Standard Media Player and Chromecast. I verified playing and pausing, seeking, changing volume, turning subtitles on and off, ending casting, and loading new media.

This fake Chromecast is a lot faster than a real Chromecast, which has to make connections to real media. It’s also possible to put the fake Chromecast into situations that are very difficult to achieve with a real Chromecast. The “IDLE” state, for example, is really hard to achieve. But we need to know what would happen if it occurrs, and by testing it with a fake Chromecast i was able to identify a bug that would likely have gone unnoticed otherwise.

For the curious, here’s a demo of my Chromecast automated regression test in action!

This is all a lot of fun, but there is nothing quite like testing for real with an actual Chromecast plugged into a massive television!

Testing Chromecast for real

Testing Chromecast for real

So that’s how i test Chromecast. If you want to test Chromecast too, please feel free to copy and tweak my fakeChromecast.js script, and ask me questions if you need any help.

Speak the phone’s language

I have long had a prejudice against tools that help you to write apps for mobile phones, especially the ones that claim you can write an app once and deploy it to multiple different platforms. My argument was that iPhone, Blackberry, Android, Windows Phone, etc, all have different user bases who expect different things. How can you write one app that works well across all platforms?

I recognised this as a prejudice, and realised that i hadn’t actually tried out the things that i was criticising. After watching a talk by coder, hacker and lecturer Syd Lawrence about PhoneGap and Sencha Touch, i realised that the time had come to test my assumptions.

Syd Lawrence – Mr Mobbles Magical Emporium from Event Handler on Vimeo.

An opportunity came along to develop an Android version of an app that already exists for iPhone. It’s quite a simple app, and on Monday last week i made a start.

I downloaded PhoneGap and installed the library into a new Android application. It provides you a very fast way to get up and running with an HTML5 app that runs locally on the phone. There are numerous javascript functions which translate into the phone’s native language to access things like the camera, GPS, accelerometer. You can also write native code and expose it to your javascript, for example i wrote a little bit of Java code to hide the soft keyboard, which i could call from javascript at the appropriate time.

To style the app, i used Sencha Touch which is an extension of Ext.js. It’s a very nice way of declaring components in javascript that look visually somewhat like a native app. I got the basic functionality together quite quickly: tabs, network connectivity with JSONP, a scrolling gallery of pictures.

It was nice to use tools i am familiar with: jQuery and CSS3. You can run your app in a browser as you are building it, and periodically just check it looks okay on a phone.

Progress was fast. I encountered a few obstacles, most notably cross-site-scripting limitations, and limitations of the pre-release Sencha Touch version 2 which meant i had to go back to the slower and older 1.1. But i managed to overcome every obstacle and within 4 days i had a basic functioning app.

All was great apart from one thing: performance. On my Android tablet the app took 3 seconds to load. On my entry-level phone it took 7 seconds initialising PhoneGap, Sencha Touch and jQuery. Just to load a page that does basically nothing until you start typing in it. I was also disappointed by the refresh when you rotate the phone’s orientation and it jerkily redraws the screen. It’s great as a proof-of-concept prototype, but it’s not something i’d be happy to release and put my name to it.

So on Friday i thought to myself, you know what? I’m going to see what it would take to rewrite this as a native Android app in Java. I spent a good deal of the weekend working on it and by Monday i had got it to approximately the same point as the PhoneGap/Sencha Touch version. I gave my client the option which to continue with, and together we chose to go ahead with the native app. We lost the ability to deploy to multiple platforms, which was something we were hoping for, but in return, we get a far more responsive, native look and feel high quality app that we can both be proud of.

There is a place for Sencha Touch: if i’m writing a mobile friendly version of a website i’ll definitely give it another try. But for an app that has been downloaded and installed, i feel that the users are justified in expecting something better. And to achieve the quality expected, at least for Android, you have to speak to the phone it its native language.

Cucumber running headless Selenium with Jenkins (the easy way)

Selenium is a great way to test your javascript from Cucumber. It opens up a browser and you actually see it clicking around, filling in fields and submitting forms right before your eyes. It’s cool.

Jenkins is a brilliant continuous integration and deployment tool. I recently set it up for a new project i’ve started. It checks for git commits, pulls the latest code, runs all the specs and cucumber scenarios, and if they all pass, it deploys the code to five different websites. It’s great because we just have to git push and all five websites will get updated.

But Jenkins runs all its commands on the command line, and it doesn’t actually have a display on which it can run a browser. This isn’t too big a problem. You can use Xvfb virtual frame buffer which can emulate a display for you.

The old way to do this was to configure Xvfb to make a display (we usually use 99 to avoid conflicts) and write little shell scripts to start and stop this display, and export an environment variable to make sure the display is used ………… YAWN!

Now enter the gem headless. It does all those boring things behind the scenes, making it much simpler. I got this tip from 8th light’s blog post Jenkins, RVM, and Selenium. To get headless Selenium you simply have to do the following:

Install Xvfb:

sudo apt-get install xvfb

Require headless in your Gemfile:

gem 'headless'

Add this little snippet to features/support/env.rb:

if ENV['HEADLESS'] == 'true'
  require 'headless'

  headless = Headless.new
  headless.start

  at_exit do
    headless.destroy
  end
end

Tag your cucumber scenarios with @javascript for those that require selenium.

Call cucumber like this:

HEADLESS=true cucumber

It Just Works™!

As an added bonus, when i’m working remotely i’m often tunelling via SSH to a shared screen on a remote server. I can use the same HEADLESS=true trick to run my selenium scenarios remotely.

Software Craftsmanship 2010

Today i had a great day at the Software Craftsmanship 2010 conference at Bletchley Park. Despite getting up ridiculously early, and feeling somewhat ill during the day, i managed to have an excellent time, learn a few things, and chat to some wonderful, interesting people.

I enjoyed Jason Gorman’s enlightening tutorial on refuctoring – the art of incrementally changing code a little at a time, so that the tests still pass but it becomes less readable and less maintainable, to the point where only you have a clue what the code does. Sometimes known as Mortgage Driven Development, because it claims to ensure you keep your job. Jason encouraged to look for Code Smiles (well-written, elegant code) and refuctor them away effectively. I enjoyed pairing with Gavin on making the ugliest “Hello, World!” we possibly could!

As an antidote to the refuctoring, several people went to the workshop on beautiful, maintainable, Designer Code by Chris, however i chose to attend Michael Hunger’s Game of Life session. After the recent code retreat at eden, i was eager to learn more about the Game of Life and Michael showed us a video of John Conway talking about the Game of Life, and an implementation of the Game of Life in APL which is mind-blowingly awesome! I worked with Chris and Iain to do a version of the Game of Life in Java. I think we had an hour to do it, and we very nearly got something that worked, except we weren’t calculating the neighbours correctly so every cell died, prompting us to rebrand it as The Game of Death!

A special highlight of the day was the guided tour of Bletchley Park where we saw the places where secret codes were intercepted and decoded in World War II, and saw the impressive machines that were built to apply brute force to crack the encoding strategies. What impressed me most was how innovation was driven forward by necessity, and it made me smile to learn that the codebreaking was usually made possible thanks to stupid mistakes that were made by those transmitting the codes.

With a nod to the Lorenz encoder and the Colossus code breaker, eden launched the Minisculus Challenge today. I hope a few people have already tried the challenge and are enjoying it as much as i did! It was hard keeping it a secret last week, when i really wanted to tweet about it!

In the afternoon I went to David Laing’s workshop on Functional Kōans. I paired with Adam and together we did the kōans in javascript, learning quite a few tricks along the way.

I had a very good day, and i thank Jason, all the workshop facilitators, and everyone who helped to make the day such a success!

Javascript BDD with Cucumber and Harmony

Inspired by tooky‘s recent post Exploring Harmony for javascript BDD with RSpec i have become determined to become better at writing javascript in a BDD kind of way.

On Monday we were privileged to have Corey Haines visit Eden for the day. I paired with Corey for a couple of hours learning about BlueRidge for unit-testing javascript classes in a Rails application. I liked what i learned but i am disappointed by the need to maintain HTML fixtures. I would like to run the specs directly on my application.

I was pairing with Tris today and we really wanted to run javascript straight from Cucumber scenarios, so we had another look at Harmony. The first thing we noticed that it needed to load a file (rather like a fixture), which we didn’t like very much. We went to see how HolyGrail does it.

Maybe it’s because we’re using Capybara not Webrat, or maybe it’s because we did something wrong, but we couldn’t get HolyGrail to work right. However, a quick peek at the source code gave us a few clues. We pulled out and tweaked the following:

def js(code)
  @__page ||= Harmony::Page.new(page.body.to_s)
  @__page.execute_js(code)
end

This worked pretty nicely for a step involving simple javascript, like:

Then /^the page is titled "([^\"]*)"$/ do |title|
  js('document.title').should == title
end

It was actually pretty exciting when we first saw that working! On to something more meaningful …

  Scenario: Use javascript to hide a box
    Given I am on a page with a box that can be hidden
    When I use javascript to click "Hide"
    Then the box should be hidden

Clicking the link with javascript actually wasn’t hard:

When /^I use javascript to click "([^\"]*)"$/ do |text|
  js("$('a:contains(#{text})')").click
end

We used jQuery to find the right link and click it using Harmony. Harmony sends the ‘click’ method straight through to the javascript object, which is pretty cool.

Ensuring the box disappears was a little more tricky. For the moment we’re just checking that the contents becomes empty.

Then 'the box should be hidden' do
  js("$('.hide_box')").html.should be_nil
end

Sure enough, when we plug in the unobtrusive jQuery code to bind the click and remove the box, the feature passes! Alright, this is a very small step down a long road, but to me it is very exciting indeed!

Next steps: we have already begun writing a Capybara driver to run Harmony. So that we don’t have to write a separate step definition When I use javascript to click “Hide” but we simply use the standard When I follow “Hide” and we tag the scenario with @harmony to use the alternative driver. That will first attempt to click it with javascript and, if nothing happens, it will follow the link in the normal way. We will also use the driver to handle when the @__page cached copy gets refreshed.

An important next step is make it a proper integration test, interacting with the full stack, such as clicking something which triggers an AJAX request to the server which must run a bit of code and send the response back to the page. I can foresee this presenting some challenges, we’ll see!

Exciting things are happening, and this is just the beginning! Web development these days is all about standing on the shoulders of giants. This is only possible thanks to the great work already being done by Harmony, Johnson and SpiderMonkey. I’m hoping that other people will become inspired by the possibilities and take this further. Watch this space for more news!