Ariejan de Vroom – 2 November 2015
967 words in about 4 minutes

As a professional software engineer you have every intention of making your project the best. You religiously practice TDD/BDD, write clean code, engage in pair programming and perform thorough code reviews. Inevitably you get to a point where you notice technical debt gathering in the small corners of your code base. A few refactoring rounds and a some upgrading of libraries and you’re back on track. Until you notice this big hairy monster glaring you right in the face.

The wonderful world of cucumbers and capybaras

Cucumber is advertised as Simple, human collaboration. Simply write your features and scenarios in plain English (or whatever language floats your boat):

Feature: Addition
  In order to avoid silly mistakes
  As a math idiot
  I want to be told the sum of two numbers
  Scenario: Add two numbers
    Given I have entered 50 into the calculator
    And I have entered 70 into the calculator
    When I press add
    Then the result should be 120 on the screen

The idea is that you can easily discuss features with customers, focussing on the behaviour or the application instead of the nitty gritty technical details. Sounds good! The next step involves a developer writing some regular expression powered steps to actually perform these steps somewhere. In the case of Ruby on Rails it’s common to use a library named Capybara. This acceptance test framework virtualizes a browser window and can take scripted instructions to click on links, fill out forms and inspect page content.

When you start out development for a new project this works great and at Sprint Demo 1 you can proudly show the customer that their user stories, along with several usage scenarios, is automatically tested and reported as working.

Fast Forward six months.

Trouble in paradise

Development is going fast. You’re adding features every sprint like a boss and your team is happy and the customer satisfied. Your cucumber tests have helped you tremendously, preventing you from deploying broken code to production several times. Yah for TDD/BDD!

The time has come that you need to do some optimalisation in your application. Several pages have become a bit overweight and need to be optimized for a better user experience. One of the things you consider is loading parts of the page on demand with AJAX. This is where things start to break down.

From your entire set of features, there’s this one scenario that now needs to load part of the page on demand with AJAX. Simply put, the page now shows a button, when clicked, the browser will fetch some HTML and update the page with the new content.

The first problem is that the default driver from Capybara will not work as it’s only dealing in HTML, not JavaScript. Following the documentation you add the @javascript tag to your scenario and configure Capybara to use another virtual browser that does support JavaScript. You quickly update your scenario and add the relevant JavaScript code. Hmm, the feature still fails?

You quickly add a save_and_open_page to inspect the page your test is running against and try to spot something obviously wrong. Nope, it all looks like it should. After banging our head onto your keyboard several times you learn that save_and_open_page, the way to inspect the HTML your tests runs against, does not work with that new JavaScript-enabled virtual browser. It does show the right page, but your JavaScript DOM-manipulations are not visible at all. Digging around Stack Overflow yields a code snippet that create a screenshot of the virtual browser, it quickly turns out that the AJAX content loads in the wrong place, which resolves the issue.

You run the entire feature suite to make sure you didn’t break anything. Hmz, 73 failures?! An obscure error message about some element not being found, but "" was found? Wait, what? Turns out that other features expect to see some content you’re now loading with AJAX. All those features need to be updated as well.

Now you have another problem! Some of those features check content that is hidden by default, but is made visible by a simple jQuery toggle() call.

The original Capybara framework had not trouble with this, as the elements are in the DOM-tree. Disregarding visibility of the element when checking for it. The new JavaScript-enabled tools do take visibility of elements into account. So out of the 74 scenarios tagged with @javascript to handle that AJAX call, 34 are now broken because visibility has now become a requirement.

Where to go from here?

This is just one of the issues you’ll likely run into when your app evolves and starts to use more JavaScript features as it grows.

Besides the fact that I rarely see Cucumber used for simple, human interaction, there’s nothing inherently wrong with using either Cucumber or JavaScript. However, there are a lot of implied choices made for you when you first include Cucumber and Capybara into your project. Choices that you need to be aware of and choices I think should be made differently from the start. There’s always a trade-off between performance (of features running) and features (of the test framework).

In my opinion we’d be much better off if the defaults for acceptance test frameworks would include:

I have no clear solution for the above, but it’s something I have (and have seen people) run into several times and it costs a lot of time, money and happiness to get it fixed. JavaScript is almost never an optional addition – with Ruby on Rails jQuery is loaded by default, so why not set the same convention for the acceptance test framework?

Ariejan de Vroom

Software Engineer • CodeRetreat Facilitator • Ruby, Go and C Programmer • Electronics Apprentice