Getting chromedriver to work with Ruby selenium-webdriver under OSX Mavericks

I interrupt my soprano-saxophone-and-bass-guitar-playing evening to write very quickly about something that annoyed me today. I can’t find the fix anywhere else on the internet, so i guess it’s up to me to provide the solution.

Yesterday i upgraded to OS X Mavericks. Which is nice. Free upgrade, woo! Most things work well, though i had a few Java version issues.

Today i found that selenium-webdriver was not able to launch Chrome with chromedriver. It would open and immediately quit.

A quick google reveals that the workaround is to start chromedriver with the --verbose flag and then it magically works again. Which is true in my experience. But all the examples are for selenium wedriver called by Java or Python and i couldn’t find one for Ruby.

The first thing i did was upgrade selenium-webdriver gem to 2.37.0 and chromedriver to 2.4. I checked my version of Chrome; it is 30.0.1599.101 which is apparently up to date. When none of those things made any difference, I investigated the verbose option.

I thought i would just have to pass --verbose to the Capybara::Selenium::Driver via the args or switches option. But no. That will send the flag to Chrome when it launches, but i need the flag to be set on chromedriver.

Ruby open source to the rescue!

I poked around in the source code for service args and found this:

# selenium/webdriver/chrome/bridge.rb

module Selenium
  module WebDriver
    module Chrome

      class Bridge < Remote::Bridge

        # ...

        def extract_service_args(opts)
          args = []

          if opts.has_key?(:service_log_path)
            args << "--log-path=#{opts.delete(:service_log_path)}"
          end

          args
        end

        # ...

      end
    end
  end
end

Hmm, so the only service arg you can set is the --log-path. That’s no good.

Ruby monkey patching to the rescue!

In my cucumber features folder i made a support file to monkey patch that method, and add a possibility of sending any service args i like.

# features/support/selenium_webdriver_monkey_patch.rb

module Selenium::WebDriver::Chrome
  class Bridge
    def extract_service_args(opts)
      args = []
      args += opts.delete(:service_args) if opts.has_key?(:service_args)
      args << "--log-path=#{opts.delete(:service_log_path)}" if opts.has_key?(:service_log_path)
      args
    end
  end
end

Now in my env.rb file i just have to require the monkey patch file and i’m able to send the verbose flag.

# features/support/env.rb

# ...

require 'selenium-webdriver'
require 'selenium_webdriver_monkey_patch'

# ...

Capybara.register_driver :selenium_chrome do |app|
  Capybara::Selenium::Driver.new(
    app,
    browser: :chrome,
    service_args: ["--verbose"])
end

# ...

Hey presto, now i can set the verbose flag on chromedriver, and sure enough, it works. I am testing in Chrome again.

It seems a really horrible hack, and i hope a better solution is found soon, either from a Mavericks update, or chromedriver, or maybe selenium-webdriver will add this option for service args in a future version.

I still don’t really know why verbose works when normal mode does not. But there we go. I can test again tomorrow, and that makes me happy!

Now, back to my bass guitar! :)

Advertisements

Testing apps that depend on external services or APIs

A lot of web programming these days doesn’t live in isolation. We consume services over the internet via APIs, and we send information to other services, such as Google Analytics, to track what users are doing.

I’m currently back at the BBC, where we have a lot of teams working on components that cooperate together: programme scheduling and publishing, video encoding, media streaming, DRM verification, content delivery networks. Testing becomes a great challenge when dealing with so many moving parts, across multiple environments (dev, int, stage and live) .. and even when it works, integration tests can get slow.

What do we really mean by integration testing?

I think automated integration testing should test the integration points .. not whether they actually do all line up and integrate correctly. There is a place for both, of course. Sometimes you need full end-to-end integration tests. But they are slow and brittle. I think we can add a little bit more control here.

Example of a dependency on an external API

Suppose I want to test an app that relies on an external service. In this case it’s a weather app.

Weather in Bristol

You can visit this now if you like: aimeerivers.com/weather/Bristol

Change the city in the URL and you’ll find out the weather for that city.

The way it does this is by parsing an API provided by OpenWeatherMap.

http://api.openweathermap.org/data/2.5/weather?q=Bristol

It gives us something like this:

{
  "coord": {
    "lon": -2.59665,
    "lat": 51.45523
  },
  "sys": {
    "country": "GB",
    "sunrise": 1368850387,
    "sunset": 1368907233
  },
  "weather": [
    {
      "id": 800,
      "main": "Clear",
      "description": "Sky is Clear",
      "icon": "01d"
    }
  ],
  "base": "global stations",
  "main": {
    "temp": 286.78,
    "pressure": 1008,
    "humidity": 81,
    "temp_min": 284.15,
    "temp_max": 289.35
  },
  "wind": {
    "speed": 2.6,
    "deg": 230,
    "var_beg": 200,
    "var_end": 270
  },
  "rain": {
    "3h": 0
  },
  "clouds": {
    "all": 0
  },
  "dt": 1368907010,
  "id": 2654675,
  "name": "Bristol",
  "cod": 200
}

If we didn’t know anything about the API dependency, we might write the following feature file:

Feature: Weather report
  As a visitor to Bristol
  I want a page that tells me what the weather will be
  So that I can decide what clothes to wear

  Scenario: Report says it's going to be clear
    When I look at the weather report for "Bristol"
    Then I should see the weather is "Sky is Clear"

We could use Capybara to test, and step definitions would look like this:

When /^I look at the weather report for "(.*?)"$/ do |city|
  visit "/weather/#{city}"
end

Then /^I should see the weather is "(.*?)"$/ do |expected_weather|
  page.should have_css '.description', text: expected_weather
end

I’m sure the problem here should be immediately obvious: the weather changes! So how do we allow for changing weather?

One way is to make the step definition more vague: “Then I should see the correct weather” .. the step definition would have to make a call to the same API, parse the JSON, and make the expectation based on the response it finds. If it comes up with the same answer as the app, we can assume the app is behaving as expected.

It’s is a viable solution, but i don’t think it’s the best.

Stub out the API

Ideally, i should be able to run the app locally. There are a number of good reasons for that anyway: i can debug the code, i am able to trace log files and get more meaningful error messages, it’s faster to run, and i can run it offline (once i have stubbed out the API dependencies!)

If i can’t run the app locally, it’s okay, but i have to be able to configure the app somehow to fetch data from a different source, one that is under my control. We did this a lot in the Olympics, using REST-assured to return responses to API requests. It was a good way to gain control over the ever-changing data.

But if i can run the app locally, i can reroute api.openweathermap.org to localhost, and then it’s mine to play with as i wish.

sudo vim /etc/hosts

Add a line:

127.0.0.1 api.openweathermap.org

Now my machine is going to reroute requests for api.openweathermap.org to localhost. Hurrah! Successfully intercepted!

Intercepted API

And of course, my app, running locally, encounters the same thing.

App broken because i have intercepted the API

Now i can put the data that i want there. Oh, there’s a catch: we have to run on port 80. But that’s okay, we can do that. Considering it means the app will be completely unaware of the change, i think that’s a small price to pay.

Introducing RackDoubles

RackDoubles allows you to create, change and remove endpoints at runtime.

Start off with a very simple config.ru file to run a Rack app that uses RackDoubles. You have to define a run method but it’s as simple as it could possibly be.

require 'rack_doubles'
use RackDoubles::Middleware

run lambda { |e|
  [404, {'Content-Type' => 'text/plain'}, ['Not found']]
}

Because we need it on port 80, we have to use sudo to run it, or in my case rvmsudo because i use and love Ruby Version Manager.

rvmsudo rackup -p 80

And just leave it running. The magic is about to happen.

Now, watch this. Here’s what we’re going for:

  Scenario: Report says it's going to be clear
    Given the weather API is stubbed to return "Bristol.json"
    When I look at the weather report for "Bristol"
    Then I should see the weather is "Sky is Clear"

I have saved the JSON i want into a file features/support/stubs/Bristol.json

So let’s use RackDoubles to set up the stub response:

def client
  RackDoubles::Client.new('http://127.0.0.1')
end

Given /^the weather API is stubbed to return "(.*?)"$/ do |filename|
  response = File.read(File.join(File.dirname(__FILE__),
                       '..', 'support', 'stubs', filename))
  client.stub('/data/2.5/weather')
  .to_return(200,
             {'Content-Type' => 'application/json'},
             response)
end

Now the data is entirely under my control. The test passes.

The best news is, now i have the opportunity to simulate bad data from the API. And did you spot the 200 status code there? I can easily write a step definition to stub the API to return a 503 Service Unavailable, to see how the app handles that.

  Scenario: API is unavailable
    Given the weather API is unavailable
    When I look at the weather report for "Bristol"
    Then I should see "Please try again later"

Here’s the step definition:

Given /^the weather API is unavailable$/ do
  client.stub('/data/2.5/weather')
  .to_return(503,
             {'Content-Type' => 'text/plain'},
             'Service unavailable')
end

Now if that test fails i have something to tell the developers about. I can easily demonstrate a scenario in which the API is unavailable. These are things that a manual tester cannot easily do. A developer might simulate them with unit tests, but i am able to test it in a reliable, repeatable way that may not be a full end-to-end test, but i think it’s a pretty good integration test.

Of course, there is still a need for full end-to-end test coverage, but i think that’s best covered by manual exploratory and smoke testing, which are still necessary and highly valuable. As Developers in Test, we have the skills to make life a bit easier for ourselves and our teams, so let’s use them.

How BDD focusses on the ‘why’

Today i got talking with somebody who was unconvinced of the benefits of behaviour driven development. I hope i’m not misquoting them, but they seemed to be saying that BDD doesn’t tell you why something happens, only what happens.

This person also said that BDD doesn’t encourage you to code correctly. That i agree with. You can code poorly with any test tools or development framework. The only thing that’s going to encourage you to code cleanly is your own standards, or those that are instilled into you by requirement, or pair programming, or code review.

To me, the focus on the why is a key difference of BDD as opposed to TDD. Here’s an example for comparison:

Test First Development tells you that you’re going to use an Array of names and it’s going to receive the shuffle method. You write the test because you’ve already decided that’s what you’re going to do. It is brittle because a change in the code will require you to rewrite the test.

Test Driven Development says that somehow some names are going to be input, and after the process they’ll come out in a random order. It doesn’t necessarily mind whether you use an Array, Hash, a shuffle method or write your own randomising function. You let the tests drive you to make those decisions when you need to.

Behaviour Driven Development tells you about me as a key stakeholder, and that i want to draw names in a random order, so that i can assign prizes fairly. It tells a story about the people entering, explaining in natural language that the names are drawn out of a hat, and when they are, the first randomly chosen person picked gets the first prize, and their name is put aside so that they don’t win another prize. How you implement the details to achieve this behaviour is entirely up to you as a developer. But you should understand quite well what you’re expecting to achieve, and why.

Okay, i know, it’s a horrible example because it’s quite difficult to test for randomness. Given an unbiased coin / When i flip the coin / Then i should get … either heads or tails … Anyway, hopefully those examples show how i consider the test methodologies to differ.

My mentor Enrique recently helped me to understand BDD, particularly Cucumber scenarios, as documentation of the understanding between the client and the developer. It just so happens that you can execute this documentation to help you write your code, and continue to use it for regression testing.

Now i pass over to you, my readers! Are your opinions different from mine? What have i missed out, misunderstood, or misrepresented? I’m not an expert here! Fortunately, you don’t need to be a BDD expert to benefit from using it! :)

Celebrating a fantastic Code Retreat day

Today, Saturday 7th May 2011, there were six code retreats around the world! They were in Belgium, Spain, UK, USA, and two in Romania! We all did the same challenge: implementing Conway’s Game of Life in pairs, using test-driven development, in iterations of 45 minutes, discarding code at the end of every iteration and pairing with somebody different for the next iteration. We introduced a few extra challenges in every session! ;)

I co-facilitated the code retreat at Sheffield University. We had 18 participants: a good mix of computer science students, professional programmers, people who code for fun, and someone who seemed to have been dragged along but got thoroughly stuck into solving the problem along with everyone else! :)

Tom Crayford co-facilitated with me, and Chris Murray ran around making sure everything went extremely smoothly.

Code Retreat Sheffield - Coding

In iteration 1 we gave everyone chance to explore the problem without heckling to see how they would do. The only special challenge was to pair with someone they didn’t know. In the retrospective we talked about the languages and classes used.

For iteration 2 we talked about TDD as in, letting the tests really drive the code in the direction they want to go. We challenged people to do the very simplest thing they could possibly do at each stage. Many people found themselves writing unnecessary code, so we kept bringing them back to the rules of the game. The retrospective brought up questions of where to start, and whether a cell should be responsible for its own state, or whether a separate entity should know where all the cells are.

In iteration 3 we encouraged people to try a different starting point, to see how that worked. We stipulated that the code could not contain any booleans, like true or false or strings or numbers representing boolean states. The third iteration was sneakily only 30 minutes long, which we didn’t tell anyone until after the retrospective! There was a good discussion about using a grid to hold the cells, versus having an array of cells who know their coordinate position.

Code Retreat Sheffield - Skype link to Timisoara

Throughout the day, at various points, we were able to have Skype video calls with other code retreats. We found this very enjoyable: we discussed the different implementation options with other people who were also tackling the same problems. During the day, Sheffield managed to communicate with all the other code retreats: Valladolid in Spain, Cronos in Belgium, Bucharest and Timisoara in Romania, and finally, towards the end of our day, Pittsburgh in Pennsylvania, USA, just as they were at the end of their second iteration.

Code Retreat Sheffield - live updates!

We also had two projectors constantly showing the twitter #coderetreat hashtag on tweetviz.com and a slideshow of pictures on hashalbum.com. These things really helped us to feel connected to a bigger community of people sharing a similar experience at the same time.

Code Retreat Sheffield - Lunch

After lunch we introduced Keith Braithwaite’s TDD as if you meant it and for iteration 4 we challenged people to obey the strict structure given. The purpose of this was to really break everyone’s normal processes to avoid mental shortcuts. It really ensures that the solution is exactly what the tests require, and nothing more. It was a tough challenge but it was taken well. A lot of people found they came to a better solution when these restrictions were applied.

In iteration 5 nobody was allowed to use if statements. (Or case statements, ternary operators, etc.) We also focussed on ping pong pair programming. This is where one person writes the test, the other person writes just enough code to make it pass, then writes the next test and hands back. During the day i had seen a lot of rather uneven pairing, where one person was doing most of the work, and the other was mostly just watching and commenting. Ping pong helps to address that by bouncing back equally between the two. Comments in the retrospective were that you had to concentrate harder, and felt more engaged because of it. I was surprised that many people had not paired that way before.

Iteration 6 was the last one for us. (Unlike Valladolid who went on to 8 iterations!) We gave people a bit longer and told them to just enjoy this one. The restrictions were that no method could be more than 3 lines long, and the code you wrote had to be the best you could possibly do. You had to write something you could be totally proud of. For most people, they said their code wasn’t the best they could possibly do, but they were definitely proud of it! :)

Code Retreat Sheffield - Retrospective

At the end of the day we did a raffle. Entry to the raffle was by way of a donation to Bletchley Park, for which we raised £200. Names were shuffled in a Ruby array and winners received either a copy of Apprenticeship Patterns or a software craftsmanship motivational calendar.

With that done, all that was left was a trip to the pub to relax after a good day’s work! :D

Thanks and kisses

I thank Tom for being a fantastic co-facilitator, and Chris for doing an excellent job of organising everything: food, refreshments, sponsorship, venue, my hotel last night, and probably a whole lot more that went so smoothly it was unnoticed.

Thank you very much indeed to the sponsors and supporters: epiGenesys, University of Sheffield Enterprise, O’Reilly UK and The GIST Foundation. It would have not been nearly as awesome without your help.

Thanks to all the people around the world who talked to us over Skype – it was great to feel part of something so much bigger than just us.

Thank you so very much to everybody who participated today. You really made it a great day for everyone.

Code Retreat in Winchester

All day today there was a Code Retreat at Eden Development. About 20 people came, from all over the country, and we spent an excellent Saturday pair programming together, sharing ideas, having stimulating conversations and getting to know each other much better.

At a code retreat everybody works on a single problem: Conway’s Game of Life. You pair with different people in each 45 minute session. You have to TDD the development, but you can use whichever programming language you choose as a pair. At the end of each session all the code must be deleted. You do a retrospective as a group, have a little break, then go and pair with somebody else.

We got through 6 sessions, which means i pair programmed with 6 people i didn’t know very well before today. It was great! The first two sessions i paired with Paul and then Quentin. To start with, we were pretty much feeling our way through the problem, implementing the rules of the game, and then getting stuck with determining the position of each cell.

In the third session i paired with Ben and we decided to try a completely different approach: dealing purely with coordinates to represent the positions of live cells. It actually worked quite nicely, although implementing the rule logic became quite difficult.

After lunch we were invited to try “TDD as if you mean it” which has very strict rules about where and when you should create your code, and when you can move it. It’s quite restricting the rules seemed to hold us back from getting very far. I was with Louis that time, who was very good at helping me to program in Java, which i’m not so familiar with.

At the 5th session we were allowed to just do whatever we want, so long as it was still test driven. I paired with Paweł and we just went a bit wild with linking cells to each other and getting them to link back respectively. We were going with North, South, East, West, but we eventually realised that you could really link any cell to any other, and the direction doesn’t matter. At that point it becomes more of a network problem. You could say that as a software coder i live or die depending on the community and people i know. That was an interesting insight.

In the final session I was with Clive and it turned out to be our best session of all. We were extremely controlled in our TDD, making sure we understood the code at each stage, and only taking small steps, and predicting at each test run whether it would pass or fail and what the failure message would be. We had cells that knew their positions, and could receive other cells and decide whether or not to attach to them as neighbours. Meaning we could pass every cell to every other cell and get them to connect based on their positions, then we could count how many neighbours they each have.

I mostly programmed in Ruby today, but there were people coding in Java, C++, C#, Python, and i think Scala, Clojure and PHP. We learnt a lot about how to program the game of life, and more importantly, how not to! A code retreat is a great opportunity to have fun, get to know other people, and code in an environment where it’s safe to try things out and make mistakes.

To find a code retreat near you, look on coderetreat.com or consider running your own!

Using Cucumber to test concurrency issues

Recently i encountered a concurrency problem of the type where there is a queue of things to do, and users press a button to be automatically assigned the next item in the queue. The bug report was that two users could get assigned the same item.

My pair programmer and i tried to reproduce the problem using two computers, but we couldn’t. We were only running one Rails instance, but we know that in the production environment there are multiple load-balanced servers pointing to one database, so we had an inkling that we’d be able to produce it using multi-threading.

To give it a test, we wrote a Rake task which we ran in two terminal windows to mimic the simultaneous access. The Rake task looked something like this:

require File.join(File.dirname(__FILE__), '..', '..', 'config', 'environment.rb')
namespace :test do
  task :take_next_for, :login do |t, args|
    user = User.find_by_login(args[:login])
    user.take_next_item
    puts user.item.inspect
  end
end

This is easily called by running:

rake test:take_next_for['ann']

We ran it for two users simultaneously and inspected the output. Sure enough they were being assigned the same item.

Since there is only one database, we knew that we could fix it with a carefully placed transaction and lock on the database. But we wanted to add a Cucumber feature so that we could be sure it was working, and to give us confidence that the bug would not come back again in the future.

  Scenario: Two users take next item simultaneously
    Given a user with login "ann"
    And a user with login "bob"
    And an available item called "Item 1"
    And an available item called "Item 2"
    When two users attempt to take the next item at the same time
    Then they should each have taken different items

Notice we can’t actually say who gets which item – it’s a race condition. We can only check that both of them have an item and that they are not the same item. We could alternatively check that both of the items have successfully been taken.

Testing this concurrency issue in Cucumber turned out to be somewhat tricky. We tried using simple Ruby threads in Cucumber, but it wasn’t properly simultaneously. I guess the single Cucumber environment still only does one thing at a time. So it was back to the Rake task.

When /^two users attempt to take the next item at the same time$/ do
  t1 = Thread.new { `RAILS_ENV=cucumber rake test:take_next_for['ann']` }
  t2 = Thread.new { `RAILS_ENV=cucumber rake test:take_next_for['bob']` }
  t1.join
  t2.join
end

We ‘join’ the two threads to make sure they’ve both finished before carrying on.

It’s slow because it loads up a whole new Rails environment for each of the Rake tasks, but that is exactly what we want to do, to mimic the concurrency of the production system.

The next problem we encountered was that Cucumber scenarios are run inside a transaction which means that a Rake task running outside of it cannot see the users and items we just created. So we had to tag the scenario as @no-txn so that they would be available externally and @clean-up-afterwards so that we could remove them from the database.

After "@clean-up-afterwards" do
  User.destroy_all
  Item.destroy_all
end

With this in place the Cucumber scenario failed as we hoped it would! Then it was simply a matter of creating a transaction from the moment we find the next item (with a database lock) until we have successfully assigned the item. This is a simplified version of what we ended up with:

class User < ActiveRecord::Base
  has_one :item

  def take_next_item
    transaction do
      item = Item.available.by_priority.find(:first, :lock => true)
      self.item = item
    end
  end

end

The Cucumber scenario passed and the problem was solved. In the live system, if two users now try to take an item at the same time, one of them has to wait a moment until the database has finished assigning to the first user so that it can assign a different item to the second user.

How would you have tested a concurrency issue like this? Are there better ways of imitating a multi-server production environment than the solution we came up with?

Licky wiki news

Just a quick update on licky, my little wiki apprenticeship task i’m doing. I realise that time has gone on and i haven’t talked about it as much as i might, plus i actually have some quite encouraging news to report now!

Last time i wrote, i was just starting to work on Pages and a PageFactory to find them or create them as necessary. Since then i’ve added a persistence strategy which saves wiki pages to the filesystem. I pass the desired directory to the strategy so that test pages can be kept separate from real pages. This was necessary as i wanted to clear out the test directory every time after use and i didn’t want to accidentally delete real pages.

After that little burst of progress i stalled for a long time as i knew i needed to start writing the web interface and i didn’t know where to begin. James gave me a good idea which was to set up a simple Webserver class that would wrap around the web server of my choice. I have chosen WEBrick for the time being, but it shouldn’t be too difficult to swap in another one later, probably with another strategy pattern. James suggested i start by visiting a page and looking for the text ‘Hello world’. I thought this was a good idea and so i started finding out about WEBrick servlets.

With that start i was suddenly able to leap ahead, testing and implementing everything i need so that my wiki is now able to create, show and edit pages through the web interface. I feel as if i am very close now. Possibly the only thing left to do is add a way to link between pages and then i can truly call it a wiki.

Test driving the development of this whole project has been extremely educational. I know i started off getting too involved in writing a test framework, so Enrique pulled me back and made me think of the simplest thing that could possibly work. I began with one method: assert_equal. That was enough in the beginning. As time went on i added two more methods: assert_true and assert_contains. When necessary i added useful debugging information to report failures. At the appropriate time i refactored the testing framework into a module that could be included into multiple test classes.

I know i also started thinking about the web server too early. I started off trying to work with ERB and templates, as i am familiar with Rails, and in the end all i needed was a few servlets. As Enrique made me realise, the most important part of a wiki is not that it runs on a web server. The most important thing is content. Pages, with content, saved to a file system. Having focussed on that and got a real solid, well tested mechanism for saving and retrieving pages, i knew that the WEBrick servlets could rely on the underlying framework, and when the time came, it did not let me down.