Using the wrong tool

I just saw a Selenium test script that was supposed to test routing system (routing as in finding a driving route between places on the map, not network routing).
The script would open the routing web-page, enter a source and destination into the appropriate inputs, click "search" and then assert on some key properties of the resulting route (such as route length, total time, etc).

All's nice, except the script repeated this sequence for about twenty-something different routes.

So what's wrong with this?
1. Wrong focus:
The test is in fact about checking results of different route-requests. It has nothing to do with the web-GUI, and having the test repeat the GUI actions (enter this into that textbox, enter that into this textbox, click that button, wait for this panel to be visible etc etc) is just creating "noise" in the test which is totally irrelevant to the actual "beef" of it which is "for a given route request, assert that the total time is XXX, that the route length is YYY" etc.

2. Slow test:
Testing the routing server from the web GUI is is just making the test run slow, which makes the feedback cycles for this test larger (more time take between running the test and getting feedback about the system-under-test), and makes the likeliness of running the test on a continual basis much lower.

3. Test Brittleness:
Since the test is repeating a sequence of web-GUI actions (click this link, type into that textbox, wait for this panel to be visible etc.), any change to the web-GUI will force us to repeat a fix in this test pertaining to that GUI-change. Like I mentioned before, these GUI change have nothing to do with the "beef" of this test, and having to apply repeated and tedious fixes to the test-script makes it brittle (i.e. breaks easily due to irrelevant changes in the system-under-test) and hard to maintain.

The solution?
The same assertions that are made by this test can be made at a lower-level, such as sending raw requests to the routing server and asserting on the results. I think something like a Fitness ColumnFixture which sends requests with different source and destination points and expects certain key-values to be returned. This should focus the test on what it's actually about, run faster (since t won't go through the web-GUI) and be more maintainable (since it won't have to fixed due to changes in the web GUI).

Balanced breakfast:
In addition to being misplaced, the bad Selenium test actually did indirectly test the web-GUI, which should not be discarded. I guess having a focused Fitness-test for the routing service should be complimented with a (much shorter) Selenium test for the web-GUI, just repeating the original sequence once or twice to be sure that the web-GUI actually works.

Testing "Everything"

Roy Osherove has posted about how to test-drive composite-methods such as validation, This got me triggered to ramble about my own experience with this kind of task.

Here's the short background:
(I'll be using Roy's example to demonstrate my own solution here).
We start test-driving a Validate() method which in fact involves validating several rules (HasAtLeast6Chars, HasDigits, HasMax10Chars etc).
We start by test-driving the first validation rule. Easy.
Next, we want to test-drive the second rule, but for this we need to "assume" that the first rule is valid, otherwise we might be getting false-negatives.
Once we get to the third rule, we wish we could "assume" that "everything is valid except the rule-under-test".

Solution: Everything = Abstraction
I find that words like "everyhing", "all", "anything" and other generalization-words in the specifications can be like a big neon-sign above our heads flashing "abstract it!". Imagine that we could actually write the tests for the 3rd rule like this:

//arrange - assume all rules are valid except HasMaxChars:
validator.AnyRuleShouldReturnTrueExcept(HasMaxChars);

//act:
validator.Validate("this is longert than ten chars but has no digits");

// assert:
Assert.IsFalse(validator.IsValid);

Hmmm... wait - we're practicing TDD, so maybe we can get the code to be like that? Remember - the tests should "tell" us what the code should look like (hence the word "Driven" in "Test-Driven-Development" - the tests "drive" the design of the code).

Let's try refactoring a bit, replacing each of the "other" validation-rules with stub-rules that are always valid:

//arrange - assume all rules are valid except HasMaxChars:
validator.HasAtLeast6CharsRule = new Stubs.ValidRule();
validator.HasDigitsRule = new Stub.ValidRule();
...

Whereas the validator class now relies on extracted collaborators to perform each rule. Each of these rules implement an IRule interface which can be stubbed or mocked.

class Validator
{
  public IRule HasAtLeast6CharsRule = new HasAtLeast6CharsRule();
  public IRule HasMax10CharsRule = new HasMax10CharsRule();
  public IRule HasDigitsRule = new HasDigitsRule();
  ...

  public void Validate()
  {
    if (this.HasAtLeast6CharsRule.IsValid &&
        this.HasMax10CharsRule.IsValid...)
    ...
  }
}

Now this is a little better since we can really neutralize "everything except" when testing a specific rule by replacing all the other rules with stubs, but it'll still be a hassle dealing with "all" these rules in each test's "arrange" section. Let's generalize a bit further - how do we generalize "neutralizing all the rules"? with a collection of course. We can run through each rule, replacing it with a stub positive:

//arrange - assume all rules pass....
string[] ruleNames = validator.Rules.GetKeys();
foreach (string ruleName in ruleNames)
{
  validator.Rules[ruleName] = new Stubs.AlwaysPassingRule();
}

// ...except HasMax10Chars:
validator.Rules["HasMax10Chars"] = new HasMax10CharsRule();

Where the Validator class now uses a collection of IRules (a dictionary of named rules, in fact), like so:

class Validator
{
  public Dictionary<string,IRule> Rules = new Dictionary<string,IRule>();

  ...

  public void Validate()
  {
    if (
        this.Rules["HasAtLeast6Chars"].IsValid &&
        this.Rules["HasMax10Chars"].IsValid...)
    ...
  }
}

But there must be a better way to make "all the rules" valid... How?
"The truth is there is no spoon" - we can make all the rules valid by simply elminating all the rules:

//arrange - assume all rules pass....
validator.Rules.Clear();

// ...except HasMax10Chars:
validator.Rules["HasMax10Chars"] = new HasMax10CharsRule();

And we quickly realize that the rule-names aren't doing us much good either, so we throw them away too:

//arrange - assume all rules pass....
validator.Rules.Clear();

// ...except HasMax10Chars:
validator.Rules.Add(new HasMax10CharsRule());

...And in the validator:

class Validator
{
 public List<IRule> Rules = new List<IRule>();

 ...

 public void Validate()
 {
  foreach (IRule rule in this.Rules)
  {
   if (!rule.IsValid)...
  }
 }
}

Viola. we've got generalized composition through refactoring.

Testing this AND that
Roy also mentioned how we need to be able to test what happens if the first rule is satisfied but the second is not, etc etc. Notice that in the above example, we can very easily test this too through generalization.

When we're asking "what happens when the first rule is satisfied and the second rule is not satisfied", this can be reduced to "assume there are only two rules; the first is satisfied and the second isn't". In codespeak this would read:

// arrange - assume we have 2 rules, the first one is satisfied and the second one isn't:
validator.Rules.Clear();
validator.Rules.Add(new Stubs.AlwaysPassingRule());
validator.Rules.Add(new Stubs.AlwaysFailingRule());

Notice how (again) the actual rules don't matter? we're testing the composition-logic decoupled from the actual rules that compose the whole. Asbtraction at it's finest hour.

Agile israel is back

I was glad to read that Roy Osherove is re-gathering the Agile-Israel group after a year or so with no meetings. Next meeting is this week, but unfortunately the last week of August is the the longest weekend ever for us preschool-parents, so no soup for me until next time...
http://weblogs.asp.net/rosherove/archive/2007/08/20/be-sure-to-register-for-agile-israel-august-meeting.aspx

Selenium and CruiseControl

Surprisingly, running Selenium tests under CruiseControl seems to be one of the least-covered topics in the agile community.. After going through the few articles on the matter (a pretty good one here and a not so promising one on the CruiseControl wiki here) I decided to roll my own.

It seems that most solutions (well, I only found two so "most" might be a harsh word here :-) require some knowledge of NAnt (which I lack) and incorporate lot's of temporary files, logs, xsl-transformations, server scripts and other such cuss-words which all make me feel a bit sea-sick.

I prefer to keep things encapsulated and clean, so my idea was to make a simple task (say a command-line EXE that can be run through a simple CruiseControl <exec> task). This task would open up some kind of in-process HTTP-listener waiting for the Selenium results to come in, and then run the Selenium TestRunner in a browser, telling it to report back to our listener when it's done. Once the results come in - we just need to analyze them, return some Cruise-Control-friendly result code and close the browser. Simple, right?

Well in fact it *was* that simple, and worked like a charm! The highlight was using DotNet 2.0's HttpListener class with help from Rick Strahl's excellent wrapper posted here. The code is still all too spiky to show it's face in public, but I'll be sure to post it soon, so stay tuned!

I am lazy therefore I am

OK, so the balloon has popped.
Over the last several days I've been trying to write down some brilliant posts about the best subjects ever, just to realize It's not all fun and games - so here it is. A short realization of one's self. I am lazy, lazy am I.
Ahh.. Much better now. I bet the next post shall be really brilliant. Oh yes, it shall.

Hello World!

Ummm... Hello world?
So you've guessed it - this is my first dip into the blog'o-puddle.
For some time now I've been pondering about a blog, and I guess pondering time is over - so here goes..

Why, oh why another blog?
One of my goals for this blog is as a Professional Development tool.
I've been learning and implementing new ideas throughout my professional life, learning as I go and trying to get myself ahead one step at a time. For the past several years my main learning subject was Agile development in general and Test-Driven Development in particular.

Although I've been able to learn and even incorporate these methodologies in my work, after reading this article by Andy Warren about professional development I realized something was missing - a greater plan, a defined goal. So under the influence it all came together in a flash of white light - "A blog! I need a blog! I MUST have a blog!!!". Yes, it seems like posting my rants and ideas on a blog might not only make a good outlet for my sleepless idea-infested mind, but might even commit me into molding my learning path into something more useful, if not to myself then at least to others (or was that the other way around?)

If you're reading this, you're too close
...Or maybe I'm too naive. Either way it'll be interesting to see what feedback I get from this blog apart from my own..

Agile Shmagile
Yes, I know it's probably the most over-hyped subject these days, nevertheless Agile and Test Driven Development will probably be the main subject-matter for this blog simply because.. umm.. well because I say so :-)

Enough is enough
So enough ranting for now, let's get this iteration wrapped up.
Hit F5, "Hello World" tests all green, release time!