Automated regression testing of Chromecast using Cucumber, JRuby and Rukuli

As you may know, i work as a Developer In Test for Media Playout at the BBC. We enable audio and video playout on the web (desktop, tablet and mobile) using our Standard Media Player.

We recently added support for Chromecast, and i want to write a bit about how i automated testing for it.

Chromecast HDMI dongle

Chromecast HDMI dongle

Chromecast is a HDMI dongle that plugs into a normal television and adds connected TV functionality. The neat thing is you can control it from your laptop, tablet or mobile using the media apps that you already know, such as YouTube, Netflix and now BBC iPlayer. Google also provides an API so that new Chromecast apps can be written all the time.

So how do you test a Chromecast? Well, personally i like to test things from an user’s point of view, actually clicking things and observing what happens as a result. But there’s only so far you can take this. I don’t want to rely on a television having a Chromecast being plugged in all the time and on the same network so that i can talk to it.

Though i’ve got to say .. it would be quite awesome to use a television 2 metres wide for my automated regression testing! :p

About to start casting

About to start casting

I didn’t need to test the Chromecast receiver app; that had already been written and tested by the guys in Salford for use by the iOS iPlayer app. I realised, what we actually care about is the communication between our player and the Chromecast. And there is a well documented Chromecast API. So with the help of colleagues Tim Hewitt and Wyell Hanna, i set about creating a fake Chromecast. Pretty much a bit of javascript that would respond in the same way as a real Chromecast, so that we could test how our player would respond to it.

And now i present to you .. the fake Chromecast!

https://gist.github.com/sermoa/10988494

A lot of it is empty methods that are neccessary to the API. But there are some neat tricks here too. I’ll talk you through a few of the more interesting bits.

Firstly we need to load the fake Chromecast into the page. We can achieve this with a @chromecast tag and a Before hook.

Before('@chromecast') do
  $chromecast = Chromecast.new
end

My Chromecast class has an initialize method that inserts the javascript into the page.

class Chromecast

  def initialize
    $capybara.execute_script <<-EOF
      var el=document.createElement("script");
      el.type="text/javascript";
      el.src = "https://gist.githubusercontent.com/sermoa/10988494/raw/82e08c5a29b5689b5e9f3d03c191b8c981102d85/fakeChromecast.js";
      document.getElementsByTagName("head")[0].appendChild(el);
    EOF
    @media_alive = true
  end

end

With this in place, so long as i set up our player with the relevant business logic that says casting is available, when i hover over the player, i get a cast button!

  @chromecast
  Scenario: Chromecast button appears
    Given Chromecast is available
    When I play media
    Then I should see the Chromecast button
A cast button appears!

A cast button appears!

So how does that work? I assure you, i don’t have a Chromecast on the network right now. I’m not using the Google Chrome extension. I’m actually running it in Firefox! What is this voodoo?!!

Have a look at the fakeChromecast.js file:

window.chrome = {};
window.chrome.cast = {};
window.chrome.cast.isAvailable = true;

See that? We’ve set up a Chromecast and said it’s available! Sneaky, hey? It’s that easy, folks! :)

The Standard Media Player will now attempt to set up a session request with its application ID. That’s fine: we’ll happily enable it to do its thing.

window.chrome.cast.SessionRequest = function(a) { }

Obviously a real Chromecast does something rather significant here. But we don’t have to care. This is a fake Chromecast. We only provide the function so that it doesn’t break.

Next, the media player makes a new ApiConfig. It’ll pass the session request it just obtained (in our case it’s null, but that doesn’t matter), and two callbacks, the second being a receiver listener. That’s the important one. We want to convince it that a Chromecast is available, so we trigger this callback with the special string “available”.

window.chrome.cast.ApiConfig = function(a, b, c) {
  c("available");
}

So Chromecast is available. Now suppose the user clicks the button to begin casting. This should request a session.

  @chromecast
  Scenario: Click Chromecast button and connect to a session
    Given I am playing media
    When I click the Chromecast button
    Then I should see Chromecast is connecting
Connecting to Chromecast

Connecting to Chromecast

How did we do this? Easy! The media player requests a session, sending a success callback. Fake Chromecast store a references to that callback – on the window so that we can trigger it any time we like! The callback function is expected to provide a session. We send it a reference to a fake session, which is entirely within our control. Oh it’s all so much fun!

window.chrome.cast.requestSession = function(a, b) {
  window.triggerConnectingToCC = a;
  window.triggerConnectingToCC(window.fakeSession);
}

As the documentation says, “The Session object also has a Receiver which you can use to display the friendly name of the receiver.” We want to do exactly that. We decided to call our fake Chromecast Dave. Because Dave is a friendly name! :)

window.fakeSession = {};
window.fakeSession.receiver = {friendlyName:"dave", volume:{level:0.7}};

I think i found our app expected the session receiver to supply a volume too, so i added that.

The media player does some shenanigans with media that we don’t need to care about, but when it sends a request to load media it passes its callback to trigger when media is discovered by Chromecast. That’s another important one for us to keep, so we store that one. We wait 1 second for a semi-realistic connection time, and then trigger it, passing a reference to .. fake media, woo!

window.fakeSession.loadMedia = function(a, b, c) {
  window.pretendMediaDiscovered = b;
  setTimeout(function() {
    window.pretendMediaDiscovered(window.fakeMedia);
  }, 1000);
}

And now we are almost there. The last clever trick is the communication of status updates. The media player sends us a callback it wants triggered when there is a media status update. So we store that.

window.fakeMedia.addUpdateListener = function(a) {
  window.updateCCmediaStatus = a;
}

The magic of this is, we stored the references to fakeMedia and fakeSession on the browser’s window object, as well as the callbacks. This means the test script has access to them. Therefore the test script can control the fake Chromecast.

So you want Chromecast to report playing? Make it so!

  @chromecast
  Scenario: Begin casting
    Given I am playing media
    When I click the Chromecast button
    And Chromecast state is "PLAYING"
    Then I should see Chromecast is casting
We are casting!

We are casting!

What does that mean, Chromecast state is “PLAYING”? It’s pretty straightforward now, using the objects and callback functions that the fake Chromecast has set up:

When(/^Chromecast state is "(.*?)"$/) do |state|
  $chromecast.state = state
  $chromecast.update!
end

Those two methods get added to the Chromecast Ruby class:

class Chromecast

  def initialize
    $capybara.execute_script <<-EOF
      var el=document.createElement("script");
      el.type="text/javascript";
      el.src = "https://gist.githubusercontent.com/sermoa/10988494/raw/82e08c5a29b5689b5e9f3d03c191b8c981102d85/fakeChromecast.js";
      document.getElementsByTagName("head")[0].appendChild(el);
    EOF
    @media_alive = true
  end

  def state=(state)
    @media_alive = false if state == 'IDLE'
    $capybara.execute_script "window.fakeMedia.playerState = '#{state}';"
  end

  def update!
    $capybara.execute_script "window.updateCCmediaStatus(#{@media_alive});"
  end

end

Note that i had to do something special for the “IDLE” case because the media status update expects a true/false to indicate whether the media is alive.

So, using these techniques, sending and capturing events and observing reactions, i was able to verify all the two-way communication between the Standard Media Player and Chromecast. I verified playing and pausing, seeking, changing volume, turning subtitles on and off, ending casting, and loading new media.

This fake Chromecast is a lot faster than a real Chromecast, which has to make connections to real media. It’s also possible to put the fake Chromecast into situations that are very difficult to achieve with a real Chromecast. The “IDLE” state, for example, is really hard to achieve. But we need to know what would happen if it occurrs, and by testing it with a fake Chromecast i was able to identify a bug that would likely have gone unnoticed otherwise.

For the curious, here’s a demo of my Chromecast automated regression test in action!

This is all a lot of fun, but there is nothing quite like testing for real with an actual Chromecast plugged into a massive television!

Testing Chromecast for real

Testing Chromecast for real

So that’s how i test Chromecast. If you want to test Chromecast too, please feel free to copy and tweak my fakeChromecast.js script, and ask me questions if you need any help.

Advertisements

Testing apps that depend on external services or APIs

A lot of web programming these days doesn’t live in isolation. We consume services over the internet via APIs, and we send information to other services, such as Google Analytics, to track what users are doing.

I’m currently back at the BBC, where we have a lot of teams working on components that cooperate together: programme scheduling and publishing, video encoding, media streaming, DRM verification, content delivery networks. Testing becomes a great challenge when dealing with so many moving parts, across multiple environments (dev, int, stage and live) .. and even when it works, integration tests can get slow.

What do we really mean by integration testing?

I think automated integration testing should test the integration points .. not whether they actually do all line up and integrate correctly. There is a place for both, of course. Sometimes you need full end-to-end integration tests. But they are slow and brittle. I think we can add a little bit more control here.

Example of a dependency on an external API

Suppose I want to test an app that relies on an external service. In this case it’s a weather app.

Weather in Bristol

You can visit this now if you like: aimeerivers.com/weather/Bristol

Change the city in the URL and you’ll find out the weather for that city.

The way it does this is by parsing an API provided by OpenWeatherMap.

http://api.openweathermap.org/data/2.5/weather?q=Bristol

It gives us something like this:

{
  "coord": {
    "lon": -2.59665,
    "lat": 51.45523
  },
  "sys": {
    "country": "GB",
    "sunrise": 1368850387,
    "sunset": 1368907233
  },
  "weather": [
    {
      "id": 800,
      "main": "Clear",
      "description": "Sky is Clear",
      "icon": "01d"
    }
  ],
  "base": "global stations",
  "main": {
    "temp": 286.78,
    "pressure": 1008,
    "humidity": 81,
    "temp_min": 284.15,
    "temp_max": 289.35
  },
  "wind": {
    "speed": 2.6,
    "deg": 230,
    "var_beg": 200,
    "var_end": 270
  },
  "rain": {
    "3h": 0
  },
  "clouds": {
    "all": 0
  },
  "dt": 1368907010,
  "id": 2654675,
  "name": "Bristol",
  "cod": 200
}

If we didn’t know anything about the API dependency, we might write the following feature file:

Feature: Weather report
  As a visitor to Bristol
  I want a page that tells me what the weather will be
  So that I can decide what clothes to wear

  Scenario: Report says it's going to be clear
    When I look at the weather report for "Bristol"
    Then I should see the weather is "Sky is Clear"

We could use Capybara to test, and step definitions would look like this:

When /^I look at the weather report for "(.*?)"$/ do |city|
  visit "/weather/#{city}"
end

Then /^I should see the weather is "(.*?)"$/ do |expected_weather|
  page.should have_css '.description', text: expected_weather
end

I’m sure the problem here should be immediately obvious: the weather changes! So how do we allow for changing weather?

One way is to make the step definition more vague: “Then I should see the correct weather” .. the step definition would have to make a call to the same API, parse the JSON, and make the expectation based on the response it finds. If it comes up with the same answer as the app, we can assume the app is behaving as expected.

It’s is a viable solution, but i don’t think it’s the best.

Stub out the API

Ideally, i should be able to run the app locally. There are a number of good reasons for that anyway: i can debug the code, i am able to trace log files and get more meaningful error messages, it’s faster to run, and i can run it offline (once i have stubbed out the API dependencies!)

If i can’t run the app locally, it’s okay, but i have to be able to configure the app somehow to fetch data from a different source, one that is under my control. We did this a lot in the Olympics, using REST-assured to return responses to API requests. It was a good way to gain control over the ever-changing data.

But if i can run the app locally, i can reroute api.openweathermap.org to localhost, and then it’s mine to play with as i wish.

sudo vim /etc/hosts

Add a line:

127.0.0.1 api.openweathermap.org

Now my machine is going to reroute requests for api.openweathermap.org to localhost. Hurrah! Successfully intercepted!

Intercepted API

And of course, my app, running locally, encounters the same thing.

App broken because i have intercepted the API

Now i can put the data that i want there. Oh, there’s a catch: we have to run on port 80. But that’s okay, we can do that. Considering it means the app will be completely unaware of the change, i think that’s a small price to pay.

Introducing RackDoubles

RackDoubles allows you to create, change and remove endpoints at runtime.

Start off with a very simple config.ru file to run a Rack app that uses RackDoubles. You have to define a run method but it’s as simple as it could possibly be.

require 'rack_doubles'
use RackDoubles::Middleware

run lambda { |e|
  [404, {'Content-Type' => 'text/plain'}, ['Not found']]
}

Because we need it on port 80, we have to use sudo to run it, or in my case rvmsudo because i use and love Ruby Version Manager.

rvmsudo rackup -p 80

And just leave it running. The magic is about to happen.

Now, watch this. Here’s what we’re going for:

  Scenario: Report says it's going to be clear
    Given the weather API is stubbed to return "Bristol.json"
    When I look at the weather report for "Bristol"
    Then I should see the weather is "Sky is Clear"

I have saved the JSON i want into a file features/support/stubs/Bristol.json

So let’s use RackDoubles to set up the stub response:

def client
  RackDoubles::Client.new('http://127.0.0.1')
end

Given /^the weather API is stubbed to return "(.*?)"$/ do |filename|
  response = File.read(File.join(File.dirname(__FILE__),
                       '..', 'support', 'stubs', filename))
  client.stub('/data/2.5/weather')
  .to_return(200,
             {'Content-Type' => 'application/json'},
             response)
end

Now the data is entirely under my control. The test passes.

The best news is, now i have the opportunity to simulate bad data from the API. And did you spot the 200 status code there? I can easily write a step definition to stub the API to return a 503 Service Unavailable, to see how the app handles that.

  Scenario: API is unavailable
    Given the weather API is unavailable
    When I look at the weather report for "Bristol"
    Then I should see "Please try again later"

Here’s the step definition:

Given /^the weather API is unavailable$/ do
  client.stub('/data/2.5/weather')
  .to_return(503,
             {'Content-Type' => 'text/plain'},
             'Service unavailable')
end

Now if that test fails i have something to tell the developers about. I can easily demonstrate a scenario in which the API is unavailable. These are things that a manual tester cannot easily do. A developer might simulate them with unit tests, but i am able to test it in a reliable, repeatable way that may not be a full end-to-end test, but i think it’s a pretty good integration test.

Of course, there is still a need for full end-to-end test coverage, but i think that’s best covered by manual exploratory and smoke testing, which are still necessary and highly valuable. As Developers in Test, we have the skills to make life a bit easier for ourselves and our teams, so let’s use them.

Automating Cucumber test scripts on a Mac

I’m really enjoying my work at the BBC on the Olympics team! I’m primarily implementing Cucumber scenarios to verify the work that the developers are doing. I’m also using other tools for stress testing and performance monitoring.

My work here is leading me to use Cucumber in weird and wonderful ways! Here’s an example. Yesterday i was asked to run a script every half an hour that tests the Olympics video player to see if it’s playing successfully, and if not, to email people to say that it’s broken. Not your usual Cucumber scenario, but amazingly enough, it can be used to do that!

Forgive the jargon below. PID is a programme identifier, and IVP is the Interactive Video Player that we are building.

  Scenario: Check LIVE video player
    Given a live PID 
    When I check the IVP 
    Then I should email people if anything goes wrong

I can use a combination of javascript hooks and taking screenshots a few seconds apart and comparing them to tell whether the video is running. I decided to run some step definitions that i’ve already defined and rescue any errors to be emailed. There are a few more steps in here but i’ve stripped it down a bit for simplicity.

When /^I check the IVP$/ do                                                                         
  if @pid
    begin
      step 'I attempt to view the IVP host page with that PID'
      step 'the video should be ready'
      step 'the video should be playing'
    rescue Exception => e
      @error = e.message
    end
  end
end

Notice that there might not actually be a live PID at the moment. It feels very weird, to put conditional logic in Cucumber step definitions like this!

Now comes the email part. I used Pony and sendmail as a simple email sending mechanism.

Then /^I should email people if anything goes wrong$/ do
  if @pid && @error
    subject = "Assurance test failed in #{ENV['ENVIRONMENT']}!"
    body = "#{page.current_url} caused this error: #{@error}"
    recipients = ['send_to@me.co.uk']

    Pony.mail(:to => recipients,
              :subject => subject,
              :body => body,
              :via => :sendmail)
  end
end

So, with that working, i set about putting it on a schedule. I thought i could just call the script from a cron task, but it turns out that wouldn’t work because i need to launch Firefox with Selenium to test the flash player.

I found this cool built-in Mac tool called Automator. You can set up all sorts of tasks into a workflow. I just needed a simple shell script:

And now, we can call that from a cron job every 30 minutes! You might have to hunt around a bit for where it saves the file. For me it’s in my Library/Services.

*/30 * * * * automator /Users/daniea16/Library/Services/IVP\ Test.workflow

Another creative alternative to the cron job is to set up recurring calendar appointments in iCal. You can set an alarm that runs a script. Pretty awesome, hey?! :)

I know Cucumber is meant to be a BDD framework, and was never intended to be used for automated regression testing or assurance testing, but isn’t it cool that with a little bit of imagination you can bend it to fit these alternative uses? :)

Cucumber running headless Selenium with Jenkins (the easy way)

Selenium is a great way to test your javascript from Cucumber. It opens up a browser and you actually see it clicking around, filling in fields and submitting forms right before your eyes. It’s cool.

Jenkins is a brilliant continuous integration and deployment tool. I recently set it up for a new project i’ve started. It checks for git commits, pulls the latest code, runs all the specs and cucumber scenarios, and if they all pass, it deploys the code to five different websites. It’s great because we just have to git push and all five websites will get updated.

But Jenkins runs all its commands on the command line, and it doesn’t actually have a display on which it can run a browser. This isn’t too big a problem. You can use Xvfb virtual frame buffer which can emulate a display for you.

The old way to do this was to configure Xvfb to make a display (we usually use 99 to avoid conflicts) and write little shell scripts to start and stop this display, and export an environment variable to make sure the display is used ………… YAWN!

Now enter the gem headless. It does all those boring things behind the scenes, making it much simpler. I got this tip from 8th light’s blog post Jenkins, RVM, and Selenium. To get headless Selenium you simply have to do the following:

Install Xvfb:

sudo apt-get install xvfb

Require headless in your Gemfile:

gem 'headless'

Add this little snippet to features/support/env.rb:

if ENV['HEADLESS'] == 'true'
  require 'headless'

  headless = Headless.new
  headless.start

  at_exit do
    headless.destroy
  end
end

Tag your cucumber scenarios with @javascript for those that require selenium.

Call cucumber like this:

HEADLESS=true cucumber

It Just Works™!

As an added bonus, when i’m working remotely i’m often tunelling via SSH to a shared screen on a remote server. I can use the same HEADLESS=true trick to run my selenium scenarios remotely.

Welcome to Rohit, my new apprentice!

I have spent the last 8 weeks working at a consulting, design and development company in London. I’ve had a fantastic time, felt very welcome and i know that together we did some amazing work.

While i was there i had the pleasure of getting to know Rohit Gogna, who is employed there as a Quality Assurance Tester. Straight away i knew that there was something different about Rohit. In fact i commented early on that i’d never known such an enthusiastic tester. The following quote shows how Ro thinks of testing:

Software Testing is not a job … it’s a passion that turns into a career.

Rohit is keen to create a community of testers in London. LondonTester.com is Ro’s initiative to help testers find jobs and share knowledge. Ro is very keen to improve, learn new techniques and share with other people. In short, perfect apprenticeship material.

I encouraged Ro a lot over the 8 weeks. We spoke about my other apprentices, and what apprenticing and mentoring mean. I encouraged Ro to seek mentors who can guide and channel this enthusiasm, because i can see so much potential, provided Ro receives the right support.

I did not offer myself as a mentor; i waited for Ro to ask me, something like the moment in Avatar when Jake has become one of the Na’vi people and may choose a partner. Jake wants Neytiri but says “she must also choose me”. Neytiri says, “She already has”. If you are an apprentice and want a mentor, you should be the one to ask. Chances are that person might already want to mentor you but is waiting to see if you ask for it!

I take mentoring very seriously, so i made sure Ro knew exactly what would be involved, and knew that it would be hard work and painful at times. Ro understood and was still keen, so we shook hands to begin the formal arrangement, and went to skype call my mentor Enrique to share the great news!

Initially, i want to get Ro learning and blogging about automated test tools, starting with Selenium. I also want Ro to have a second mentor: someone who is more knowledgeable about testing and mentor Ro on these things that i don’t know so well. We’ll get networking and meeting people to see if we can find someone just right.

So please, follow Ro on twitter: Testerpedia, and watch with me, with pride, as we see Ro’s full potential begin to emerge!