Image comparison testing

hotels.com
At hotels.com I was working on a project where the team wanted to change the data provider in various sections of the website. While changing the data provider it was important that the page and the content displayed remained exactly the same as before.

I explored different options on how to test this change, but finally decided to create an automated image comparison test which would be able to compare the pages to ensure they are both the same.

Using Selenium I would be able to navigate to the required webpages and take screenshots. I would save the screenshots with an appropriate name so I could differentiate whether it was the old data provider or new.
Using a tool called ImageMagick, I would run the image comparison task, which would compare the two images and output a diff image which highlighted the difference between the captured screenshots in red.

diff

I ran the test and soon found a bug in the old data provider which was fixed in the new system. This would mean all the screen comparisons test would fail as the data would be different.

I then extended the image comparison framework further. It would now output a diff.txt file showing a numeric value based on the percentage of pixels that have changed for each of the images compared. I then ran each test multiple times and found the exact number which indicated the data issue. This number was used as the threshold. The test would only pass if the number in the diff.txt file was lower than the ‘threshold’ number.

Whilst running the tests I noticed the screenshots were only from the section displayed in the browser view. Swiping down and taking multiple screenshots would be impractical as some pages were very long. After some research I found a PhantomJSDriver which is a headless browser which can take screenshots of the whole page.

Running tests on a headless browser makes it almost impossible to debug a failed test. Using logback, I added debug logging inside each method.

logback