Tuesday, September 25, 2007

Using the wrong tool

I just saw a Selenium test script that was supposed to test routing system (routing as in finding a driving route between places on the map, not network routing).
The script would open the routing web-page, enter a source and destination into the appropriate inputs, click "search" and then assert on some key properties of the resulting route (such as route length, total time, etc).

All's nice, except the script repeated this sequence for about twenty-something different routes.

So what's wrong with this?
1. Wrong focus:
The test is in fact about checking results of different route-requests. It has nothing to do with the web-GUI, and having the test repeat the GUI actions (enter this into that textbox, enter that into this textbox, click that button, wait for this panel to be visible etc etc) is just creating "noise" in the test which is totally irrelevant to the actual "beef" of it which is "for a given route request, assert that the total time is XXX, that the route length is YYY" etc.

2. Slow test:
Testing the routing server from the web GUI is is just making the test run slow, which makes the feedback cycles for this test larger (more time take between running the test and getting feedback about the system-under-test), and makes the likeliness of running the test on a continual basis much lower.

3. Test Brittleness:
Since the test is repeating a sequence of web-GUI actions (click this link, type into that textbox, wait for this panel to be visible etc.), any change to the web-GUI will force us to repeat a fix in this test pertaining to that GUI-change. Like I mentioned before, these GUI change have nothing to do with the "beef" of this test, and having to apply repeated and tedious fixes to the test-script makes it brittle (i.e. breaks easily due to irrelevant changes in the system-under-test) and hard to maintain.

The solution?
The same assertions that are made by this test can be made at a lower-level, such as sending raw requests to the routing server and asserting on the results. I think something like a Fitness ColumnFixture which sends requests with different source and destination points and expects certain key-values to be returned. This should focus the test on what it's actually about, run faster (since t won't go through the web-GUI) and be more maintainable (since it won't have to fixed due to changes in the web GUI).

Balanced breakfast:
In addition to being misplaced, the bad Selenium test actually did indirectly test the web-GUI, which should not be discarded. I guess having a focused Fitness-test for the routing service should be complimented with a (much shorter) Selenium test for the web-GUI, just repeating the original sequence once or twice to be sure that the web-GUI actually works.

No comments: