vbot - JSON based visual regression tester

A command line tool to do visual regression testings, support interaction flows by JSON configurations without custom scripts.

Posted by Katat Choi on January 9, 2017

Motivations

How long would you take to spot all the differences? diff

During the front end development, quite often, even small updates to the code, may requires a lot of interactions, clicks and typings, on a page in order to check the changes reflected in the view as expected. Doing the repeated front end tests in the manual way, certainly kills productivity, consumes a huge amount of time and mental energy from developers.

I tried several existing automated visual regression test tools to see if it can remedy this painful manual front end tests and adhere to agile development practices, but they seem either too complicated to start with or difficult to sustain the tests in a long run. Then I created vbot with the aims to help developers quickly create tests with minimum development overhead, and automate the testings “headlessly” at local dev environment or on a continuous integration server, supports End-to-End testings which involve front end interactions covering the backend processes.

This post discuss a bit more on the motivations behind vbot, and gives a simple demo.

Agile process

The key of agile development process is to code up a working software, enabling users to interact with the software, and collecting the feedbacks as soon as possible. Then we repeat this process as a series of sprints until all the requirements have been satisfied. The process would introduce lots of changes to the code, as the requirements keep evolving, and probably need refactors to the source codes. With the tests covering the functionalities built in earlier sprints, it will keep the refactoring be more stable and give developers confidences to make necessary changes to maintain the health status of the source codes.

Visual regression test facilitates quickly gathering visual feedbacks for changes, making it easier to test the components used in different parts of the web applications. Also it can be used as a part of End-to-End test, which does user interaction tests at the front end to make sure the user experienced as expected while monitoring the backend system processes.

Tedious testings in SPA

Single Page Application or any complicated web applications, normally includes many interactions in different views, and the states of a view could depend on the previous views’s states. Manual testing on this type of application could involve a lot of clicks and typings just for a single function. No creative developers like to do this kind of dull testing work, wasting a huge amount of time in clicks and typings. The worse is that eventually it kills the developers’ motivation for doing the job without a proper automated testing tool, leaving the projects with a decent quality.

Collaboration

It is usual to see different developers work on the same code for the time being, and mess up the code at some point. It is also common to see there are codes at different places having the same functionality, and motivated to refactor it into a component to be reusable, but fear of the uncertain amount of time needed for the testings at different places.

Automated tests is an efficient way to pass on the works among team members. It eases the learning curve for the developers at the code written by others, and encourage them to make bold changes to the code whenever necessary to balance the technical debt in a long run.

Whenever there is a pull request, visual regression test can be used as a testing component, reducing the manual testing efforts for every pull request and be more confident on the release cycles.

Open source VRT tools

So what solutions are there to help?

BackstopJS

The idea of writing testing definitions in a JSON file is great, just need to define the different screen sizes and what parts of the web page to take screenshots by CSS selector without writing JS code. It is easy to share and communicate with, having a clear information about what to test, compared with lengthy JS codes. It also supports simulating user interactions using CasperJS custom scripts, referenced from the JSON file. However, it is more suitable for static web pages, as the creator states in its github page.

BackstopJS may be just the thing if you develop custom WordPress, Drupal or other CMS templates

PhantomCSS

The library is designed to integrate with the CasperJS scripts, flexible to create screenshots at any part of the web page by CSS selector, at any point of time during the UI test flow.

VBot, a new player

The goal of vbot is to help developers create test quickly with minimum overhead, save their dev time and mental energy, most importantly, sustain the quality of the work in a longer run.

Key features/goals of vbot:

  • JSON based
    Tests can be defined in JSON with simple schema, without any JS codes.
  • Interaction flow
    Support interaction tests, particularly useful for feature-rich web app or SPA.
  • Screenshots
    Sceenshots can be taken at any point during the testing flow, then compare with baseline and generate reports.
  • Portable
    Run on local dev environment or on remote server to facilitate continuous integration process.

Why JSON based?

JSON based schema makes it much easier to share tests among the team members, as the schema helps them have a common senses over the JSON definitions. When trying to make senses on how the tests work, the expected JSON syntax layout helps developers quickly skim through and locate to the spot on where they want to check more details. The natural of JSON syntax also helps highlight the test definition differences between current and previous commit.

VBot in action

So enough with the theory about the visual regression tests, let’s take a look at how vbot can ease the VRT process. Suppose we want to test this TODO web app using vbot. We define the interaction flow in a JSON file as below:

{
    "viewWidth": 375,
    "viewHeight": 677,
    "captureSelector": "html",
    "host": "http://todomvc.com",
    "scenarios": [{
        "name": "todo test",
        "path": "/examples/react",
        "actions": [{
            "type": "assert",
            "waitFor": ".new-todo",
            "shot": true,
            "comment": "wait for the input element .new-todo appear and take a screenshot"
        },{
            "type": "enter",
            "waitFor": ".new-todo",
            "value": "drink a cup of coffee",
            "enter": true,
            "comment": "enter task message and press 'enter' key"
        },{
            "type": "assert",
            "waitFor": "ul.todo-list li:nth-child(1)",
            "shot": true,
            "comment": "make sure there is a newly created task in the list and take screenshot"
        },{
            "type": "enter",
            "waitFor": ".new-todo",
            "value": "drink another cup of coffee",
            "enter": true,
            "comment": "create second task"
        },{
            "type": "assert",
            "waitFor": "ul.todo-list li:nth-child(2)",
            "shot": true,
            "comment": "ensure the second task is created"
        },{
            "type": "click",
            "waitFor": "ul.todo-list li:nth-child(1) .toggle",
            "comment": "mark the first task as completed"
        },{
            "type": "assert",
            "waitFor": "ul.todo-list li.completed:nth-child(1)",
            "shot": true,
            "commend": "ensure the completed status is reflected in the view and take screenshot"
        }]
    }]
}

The interaction flow is defined step by step, and straightforward. You probably could guess the meanings of the most parts in the definitions above. To execute the tests, run the tests targeting the json file:

vbot --f=schema.json

It should execute the tests and generate the screenshots as shown below:


{
    "type": "assert",
    "waitFor": ".new-todo",
    "shot": true,
    "comment": "wait for the input element .new-todo appear and take a screenshot"
}

0_assert-.new-todo.png


{
    "type": "enter",
    "waitFor": ".new-todo",
    "value": "drink a cup of coffee",
    "enter": true,
    "comment": "enter task message and press 'enter' key"
}
{
    "type": "assert",
    "waitFor": "ul.todo-list li:nth-child(1)",
    "shot": true,
    "comment": "make sure there is a newly created task in the list and take screenshot"
}

2_assert-ul.todo-list_li:nth-child(1).png


{
    "type": "enter",
    "waitFor": ".new-todo",
    "value": "drink another cup of coffee",
    "enter": true,
    "comment": "create second task"
}
{
    "type": "assert",
    "waitFor": "ul.todo-list li:nth-child(2)",
    "shot": true,
    "comment": "ensure the second task is created"
}

4_assert-ul.todo-list_li:nth-child(2).png


{
    "type": "click",
    "waitFor": "ul.todo-list li:nth-child(1) .toggle",
    "comment": "mark the first task as completed"
}
{
    "type": "assert",
    "waitFor": "ul.todo-list li.completed:nth-child(1)",
    "shot": true,
    "commend": "ensure the completed status is reflected in the view and take screenshot"
}

6_assert-ul.todo-list_li.completed:nth-child(1).png


If this is the first time vbot run the tests for this JSON definition file, the images generated are the baselines. The subsequent tests will compare the newly generated image file with the baselines and highlight the differences if it is not 100% match. For example, let’s change the action definition below:

{
    "type": "enter",
    "waitFor": ".new-todo",
    "value": "drink a cup of coffee",
    "enter": true
}

to

{
    "type": "enter",
    "waitFor": ".new-todo",
    "value": "drink a cup of milk",
    "enter": true
}

After re-run the command line above, it should report there is test result mismatch the baseline and generated a image that highlights the difference:


2_assert-ul.todo-list_li:nth-child(1).fail.png


When rebase is needed, run the following command to rebase the images:

vbot --f=schema.json --rebase

There are other features, such as preset localStorage values, set maximum timeout for waiting an element to appear, scroll element, etc. For more details about vbot, please see github repo.

I hope vbot could help a bit within your development processes. Certainly I welcome any suggestions or complaints, so feel free to leave comments.