Imagine a website with dozens of web templates. Someone needs to change margin and padding of a navigation element. You make this change, check some templates with navigation in every possible browser and send changes to production server. What if this change break a template you haven't checked yet? It would cost a lot of time to check all the templates in every browser with every change. That's exactly why automatic regression tests can help save your time. This article contains my notes about installation of regression testing based on BackstopJS. A lot of useful information can be found in article by Garris Shipon (author of BackstopJS) at Automating CSS regression testing. Let's do it.

First of all create a directory for your new project.

 GULP

Gulp is a fast and intuitive streaming build tool built on Node.js. It's similar to Grunt

 Installation

  1. Globally install Gulp CLI

    $ npm install gulp -g
    
  2. Go to your project directory and init NPM

    $ npm init
    

PhantomJS

PhantomJS is a headless WebKit scriptable with a JavaScript API.

Installation

  1. From your project directory install via NPM

    $ sudo npm install phantomjs
    
  2. Try to run

    $ phantomjs -v
    

CasperJS

CasperJS is an open source navigation scripting & testing utility written in Javascript for the PhantomJS.

Installation

  1. From your project directory install via NPM

    $ sudo npm install -g casperjs
    
  2. try

    $ casperjs
    

BackstopJS

BackstopJS is a config-driven automated screenshot test component.

 Installation

  1. From your project directory install via Bower

    $ bower install backstopjs
    
  2. Install NPM dependencies

    $ cd bower_components/backstopjs
    $ npm install
    
  3. Generate BackstopJS config template (do it in bower_components/backstopjs directory). It will create backstop.json config template in project root.

    $ gulp genConfig
    

Content of backstop.json template config

{
"viewports" : [
    {
     "name": "phone",
     "viewport": {"width": 320, "height": 480}
    }
    ,{
     "name": "tablet_v",
     "viewport": {"width": 568, "height": 1024}
    }
    ,{
     "name": "tablet_h",
     "viewport": {"width": 1024, "height": 768}
    }
]
,"grabConfigs" : [
    {
        "testName":"http://getbootstrap.com"
        ,"url":"http://getbootstrap.com"
        ,"hideSelectors": [
        ]
        ,"removeSelectors": [
            "#carbonads-container"
        ]
        ,"selectors":[
            "header"
            ,"main"
            ,"body .bs-docs-featurette:nth-of-type(1)"
            ,"body .bs-docs-featurette:nth-of-type(2)"
            ,"footer"
            ,"body"
        ]   
        ,"readyEvent": null
        ,"delay":500
    }
]
}

 Test BackstopJS server

  1. Go to bower_components/backstopjs and run

    $ gulp start
    
  2. Open browser and go to http://localhost:3001/compare

  3. You should see something like this:

Create a basic test website

This is how our basic website looks like.

Test website

Edit test configuration

The test website is located at http://localhost so we have to change config file accordingly.

{
"viewports" : [
    {
     "name": "phone",
     "viewport": {"width": 320, "height": 480}
    }
    ,{
     "name": "tablet_v",
     "viewport": {"width": 568, "height": 1024}
    }
    ,{
     "name": "tablet_h",
     "viewport": {"width": 1024, "height": 768}
    }
]
,"grabConfigs" : [
    {
        "testName":"http://localhost"
        ,"url":"http://localhost"
        ,"hideSelectors": [
        ]
        ,"removeSelectors": [
        ]
        ,"selectors":[
            "body"
        ]   
        ,"readyEvent": ""
        ,"delay":0
    }
  ]
}

Create test website reference screenshots

Go to bower_components/backstopjs and start

$ gulp reference

This will create bower_components/backstopjs/bitmaps_reference directory with reference screenshots according to settings.

Website references

First test

We didn't change anything yet so first test should pass. Let's try it

$ gulp test

This will create bower_components/backstopjs/bitmaps_test directory with test screenshots according to settings.

First test

 Let's change something

Let's change padding, text color and run the test again.

Regression test failed

Test failed. Let's look at the differences. We simply see our changes in a report.

Differences

Creating new references

Let's say that the website version with a yellow font and changed padding bellow is a new development version. We have to create a new reference screenshots so the test can pass. Start

$ gulp reference 

and then

$ gulp test

new references

The test passed, everything is unicorn.

new references screenshots

Testing elements separately

In more complex projects, we will need to check elements separately (for example header, navigation, footer etc.). It's simple to do it with BackstopJS. Just open backstop.json, go to section:

,"selectors":[
    "body"
]

and add the element you want to test. In my case it's h1 element:

,"selectors":[
    "h1",
    "body"
]

Let's create a new reference screenshots using

$ gulp reference 

and test using

$ gulp test 

body and h1 elements are now being tested using BackstopJS

Testing elements

Removing and hiding elements

In some cases we will need to remove or hide elements (banners, other ads, frequently changing content etc.). That's why backstop.json contains removeSelectors and hideSelectors sections in backstop.json. Let's say I want to remove a paragraph on my test page

,"removeSelectors": [
     "p"
] 

Again

$ gulp reference
$ gulp test

and see the result.

Removing selectors

The paragraph is now gone.

Creating screenshots at the right time

Creating screenshots of websites that contains content loaded using AJAX etc may be little bit harder. A screenshot has to be taken after the site's content is loaded. BackstopJS has two options - readyEvent and delay.

readyEvent is fired when your app sends predefined string to output. Let's simulate it on my test page. Wait 5 seconds and then log page_loaded string to output.

<script>
    setTimeout(function(){console.log("page_loaded")}, 5000 );
</script>

Change readyEvent option in backstop.json:

"readyEvent": "page_loaded"

And create new reference screenshots using

$ gulp reference

Console output:

CasperJS: CREATING NEW REFERENCE FILES
CasperJS: remote console > page_loaded
CasperJS: page_loaded
CasperJS: Ready event received.
CasperJS: Current location is   http://localhost/index.html
CasperJS: Screenshots for phone (320x480)
CasperJS: remote console > page_loaded
CasperJS: page_loaded
CasperJS: Ready event received.
CasperJS: Current location is http://localhost/index.html
CasperJS: Screenshots for tablet_v (568x1024)
CasperJS: remote console > page_loaded
CasperJS: page_loaded
CasperJS: Ready event received.
CasperJS: Current location is http://localhost/index.html
CasperJS: Screenshots for tablet_h (1024x768)
CasperJS: Comparison config file updated.

Bitmap file generation completed.

Delay takes effect after readyEvent. If we change it to 3000 now, backstop will wait to receive ready event (5 seconds) and then will capture screenshots after 3 seconds delay.

gulp-chug

The trouble is that BackstopJS has it's own gulpfile.js and it's pretty complicated. If you want to control backstop's gulp within your own project's gulpfile.js, there is an easy way out using gulp-chug package.

Installation

Go to the project directory and type

$ npm install gulp-chug --save-dev

Configuration

Load gulp-chug

var chug = require('gulp-chug');

and create the first task. I want to call gulp reference to create a new reference screenshots.

//gulp chug (BackstopJS: $ gulp reference)
gulp.task( 'reference', function () {
    gulp.src( './bower_components/backstopjs/gulpfile.js' )
        .pipe( chug({
        tasks:  [ 'reference' ]
    }));
});

Now i may go to my project folder and type:

$ gulp reference

to create new reference screenshots.

Useful resources

Automating CSS Regression Testing at CSS-tricks.com

Visual Regresssion Testing at Sonniesedge.co.uk

BackstopJS

PhantomJS

CasperJS

Gulp