Solving the Mystery Behind Alternating Failed Tests in QUnit

I’ve recently started using QUnit for my JavaScript testing. It’s a unit testing framework for JavaScript. It’s quite an easy framework to learn and I strongly encourage everyone who is doing any decent JavaScript coding to use it or any one of the myriad frameworks out there.

One of the strange problems I faced with QUnit is a strange phenomenon where a custom written plugin fails the test on every alternate invocations of the test.

You cannot imagine the frustration of debugging the code and the testing framework for one whole day. Everything looks fine. So when I ran it the first time, the test fails. But when I refresh the page, the test passes. So I refresh the page again and the test fails! But press F5 (refresh), voila, it passes!

So, being the stubborn fella that I am, I refuse to accept this as a reality. After taking a good hard look, I finally accept that my code is fine. So the issue must lie with the testing framework. Of course searching for “QUnit fails on alternate invocations” yields no straight answers.

Going back to good ol’ research, I began looking at each page to look for some clues as to the cause. Funny enough, my breakthrough came from a Microsoft page ( http://msdn.microsoft.com/en-us/magazine/gg749824.aspx) of all places.

The page titled “Automating JavaScript Testing with Qunit” didn’t seem helpful at first. It looked like it is another page teaching first timers how to use QUnit. So I quickly browsed through the page half heartedly. The article was longer than most tutorials out there so I thought the page must have some useful stuff inside. I decided to take a second look. Slowly.

Of course the first few sections didn’t have any thing that I didn’t already know. Then I reached the section “More Efficient TDD”. Under it, I found out that QUnit is designed such that on a page refresh it runs the failed test first in an attempt to be more efficient in testing.

A simple concept. But it is also the part of the reason why my test fails and passes on alternate page refreshes.

To explain why, I need to first explain what I’m testing. I’ve written a plugin that automatically binds to existing elements on the page. So when the file containing the plugin is included in the page, the elements are binded automatically.

So when I run the tests the first time, it fails. Why?! The reason is that when the tests are run, the elements in the test harness are seemingly re-created on each run of the tests. So when they are re-created, the automatic binding does not take place since the JavaScript file has already been included.

If so, then why does it work on alternate invocations? Ah… therein lies the magic. To make tests more efficient, QUnit will run the failed tests before other tests upon page refresh. So, when the failed tests are executed before other tests, the elements in the test harness that are already present are binded, and thus, the tests PASS! But because these tests now pass, when the page is refreshed they go back to the same order as they originally was, and thus they now FAIL!

Of course with this realization, it prompted me to rewrite the plugin more efficiently. Hope this self-discovery is helpful to someone out there! Have fun testing!