Testing apps that depend on external services or APIs

A lot of web programming these days doesn’t live in isolation. We consume services over the internet via APIs, and we send information to other services, such as Google Analytics, to track what users are doing.

I’m currently back at the BBC, where we have a lot of teams working on components that cooperate together: programme scheduling and publishing, video encoding, media streaming, DRM verification, content delivery networks. Testing becomes a great challenge when dealing with so many moving parts, across multiple environments (dev, int, stage and live) .. and even when it works, integration tests can get slow.

What do we really mean by integration testing?

I think automated integration testing should test the integration points .. not whether they actually do all line up and integrate correctly. There is a place for both, of course. Sometimes you need full end-to-end integration tests. But they are slow and brittle. I think we can add a little bit more control here.

Example of a dependency on an external API

Suppose I want to test an app that relies on an external service. In this case it’s a weather app.

Weather in Bristol

You can visit this now if you like: aimeerivers.com/weather/Bristol

Change the city in the URL and you’ll find out the weather for that city.

The way it does this is by parsing an API provided by OpenWeatherMap.

http://api.openweathermap.org/data/2.5/weather?q=Bristol

It gives us something like this:

{
  "coord": {
    "lon": -2.59665,
    "lat": 51.45523
  },
  "sys": {
    "country": "GB",
    "sunrise": 1368850387,
    "sunset": 1368907233
  },
  "weather": [
    {
      "id": 800,
      "main": "Clear",
      "description": "Sky is Clear",
      "icon": "01d"
    }
  ],
  "base": "global stations",
  "main": {
    "temp": 286.78,
    "pressure": 1008,
    "humidity": 81,
    "temp_min": 284.15,
    "temp_max": 289.35
  },
  "wind": {
    "speed": 2.6,
    "deg": 230,
    "var_beg": 200,
    "var_end": 270
  },
  "rain": {
    "3h": 0
  },
  "clouds": {
    "all": 0
  },
  "dt": 1368907010,
  "id": 2654675,
  "name": "Bristol",
  "cod": 200
}

If we didn’t know anything about the API dependency, we might write the following feature file:

Feature: Weather report
  As a visitor to Bristol
  I want a page that tells me what the weather will be
  So that I can decide what clothes to wear

  Scenario: Report says it's going to be clear
    When I look at the weather report for "Bristol"
    Then I should see the weather is "Sky is Clear"

We could use Capybara to test, and step definitions would look like this:

When /^I look at the weather report for "(.*?)"$/ do |city|
  visit "/weather/#{city}"
end

Then /^I should see the weather is "(.*?)"$/ do |expected_weather|
  page.should have_css '.description', text: expected_weather
end

I’m sure the problem here should be immediately obvious: the weather changes! So how do we allow for changing weather?

One way is to make the step definition more vague: “Then I should see the correct weather” .. the step definition would have to make a call to the same API, parse the JSON, and make the expectation based on the response it finds. If it comes up with the same answer as the app, we can assume the app is behaving as expected.

It’s is a viable solution, but i don’t think it’s the best.

Stub out the API

Ideally, i should be able to run the app locally. There are a number of good reasons for that anyway: i can debug the code, i am able to trace log files and get more meaningful error messages, it’s faster to run, and i can run it offline (once i have stubbed out the API dependencies!)

If i can’t run the app locally, it’s okay, but i have to be able to configure the app somehow to fetch data from a different source, one that is under my control. We did this a lot in the Olympics, using REST-assured to return responses to API requests. It was a good way to gain control over the ever-changing data.

But if i can run the app locally, i can reroute api.openweathermap.org to localhost, and then it’s mine to play with as i wish.

sudo vim /etc/hosts

Add a line:

127.0.0.1 api.openweathermap.org

Now my machine is going to reroute requests for api.openweathermap.org to localhost. Hurrah! Successfully intercepted!

Intercepted API

And of course, my app, running locally, encounters the same thing.

App broken because i have intercepted the API

Now i can put the data that i want there. Oh, there’s a catch: we have to run on port 80. But that’s okay, we can do that. Considering it means the app will be completely unaware of the change, i think that’s a small price to pay.

Introducing RackDoubles

RackDoubles allows you to create, change and remove endpoints at runtime.

Start off with a very simple config.ru file to run a Rack app that uses RackDoubles. You have to define a run method but it’s as simple as it could possibly be.

require 'rack_doubles'
use RackDoubles::Middleware

run lambda { |e|
  [404, {'Content-Type' => 'text/plain'}, ['Not found']]
}

Because we need it on port 80, we have to use sudo to run it, or in my case rvmsudo because i use and love Ruby Version Manager.

rvmsudo rackup -p 80

And just leave it running. The magic is about to happen.

Now, watch this. Here’s what we’re going for:

  Scenario: Report says it's going to be clear
    Given the weather API is stubbed to return "Bristol.json"
    When I look at the weather report for "Bristol"
    Then I should see the weather is "Sky is Clear"

I have saved the JSON i want into a file features/support/stubs/Bristol.json

So let’s use RackDoubles to set up the stub response:

def client
  RackDoubles::Client.new('http://127.0.0.1')
end

Given /^the weather API is stubbed to return "(.*?)"$/ do |filename|
  response = File.read(File.join(File.dirname(__FILE__),
                       '..', 'support', 'stubs', filename))
  client.stub('/data/2.5/weather')
  .to_return(200,
             {'Content-Type' => 'application/json'},
             response)
end

Now the data is entirely under my control. The test passes.

The best news is, now i have the opportunity to simulate bad data from the API. And did you spot the 200 status code there? I can easily write a step definition to stub the API to return a 503 Service Unavailable, to see how the app handles that.

  Scenario: API is unavailable
    Given the weather API is unavailable
    When I look at the weather report for "Bristol"
    Then I should see "Please try again later"

Here’s the step definition:

Given /^the weather API is unavailable$/ do
  client.stub('/data/2.5/weather')
  .to_return(503,
             {'Content-Type' => 'text/plain'},
             'Service unavailable')
end

Now if that test fails i have something to tell the developers about. I can easily demonstrate a scenario in which the API is unavailable. These are things that a manual tester cannot easily do. A developer might simulate them with unit tests, but i am able to test it in a reliable, repeatable way that may not be a full end-to-end test, but i think it’s a pretty good integration test.

Of course, there is still a need for full end-to-end test coverage, but i think that’s best covered by manual exploratory and smoke testing, which are still necessary and highly valuable. As Developers in Test, we have the skills to make life a bit easier for ourselves and our teams, so let’s use them.

Advertisements

2 comments on “Testing apps that depend on external services or APIs

  1. I was on your old blog, then followed the link to get here. This is an interesting and informative post. Not my reason for visiting, but I figured since it was a useful post that took you a long time to create the least I could do was say thanks for taking the time to do so.

  2. I have been having API issues on one of my blogs but not all and they used the same script. Weird. I think you may have provided the solution. Thanks.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s