On a way to find the perfect e2e testing tool — part 2
Welcome back to our journey in search of the best JavaScript testing tools. If you missed our previous article go back to catch up what our goal is and to see our review of the Cypress library.
Read part 1 of “On a way to find the perfect e2e testing tool”
Our desire is to take a different approach for the testing process of one of our projects. Instead of manually testing the website occasionally we wanted to reduce the possibility for regression issues by building an automated workflow. This way we can ensure that all of the main features are working as expected.. or at least that some of our latest changes are not breaking any existing functionalities. Based on this we decided to use an automatic standalone user journey performed by an e2e testing tool, as part of our build pipeline. And since we want to extend our tech stack we wanted to try something new so we have to identify the perfect test runner by subjecting it to a series of checks which have to be met.
Our requirements list includes the following:
- easy to learn, clean and descriptive syntax
- well detailed documentation and support community
- rich abilities to debug and inspect issues
- extendable to work with custom UI components (modals, select2, etc.)
- as fast and lightweight as possible
Up next in our list is Nightwatch.js.
Nightwatch.js
Nightwatch.js is an integrated, easy to use e2e testing solution for web applications and websites, written in Node.js. It uses the W3C WebDriver API to drive browsers in order to perform commands and assertions on DOM elements. It comes with simple but powerful syntax which enables you to write tests very quickly, using only JavaScript and CSS or Xpath selectors. Uses a built-in command-line test runner which runs the tests either sequentially or in parallel, with retries and implicit waits. Sounds promising so we decided to give it a try.
Getting started with Nightwatch.js
Note: The purpose of this article is to review the different testing libraries and provide you some feedback in order to help you choose the right tool for your case. Thus you cannot rely on this article as a tutorial or step-by-step guide, instead you can go and read the official docs.
Our first job is to initialize a new project and set up the required configurations and dependencies. It’s a simple process if you already have installed Node.js and npm on your environment. Since there are multiple WebDriver options we’ll leave you to choose for yourself so better to follow the official installation guide on your own. Once you’re ready with the configuration of your new test environment we can move on to our first test with Nightwatch.js.
module.exports = {
'Access the Home page': function (browser) {
browser
.url(browser.launchUrl)
.assert.containsText('.description h1', 'Hello world!')
.end();
}
};As promised, the code is clean and descriptive so you shouldn’t have any troubles understanding the different actions and their execution order simply by reading the code. Finding elements on a page is by far one of the most common functions during an e2e test and Nightwatch provides several techniques for locating elements and also an extensible assertion framework to perform verifications on them. Of course, our first test is really simple so we can move on to see it in action.
[Guest/Home] Test Suite
=======================
ℹ Connected to localhost on port 4444 (6579ms).
Using: firefox (77.0.1) on linux 5.3.0-59-generic platform.Running: Access the Home page✔ Testing if element <.description h1=""> contains text 'Hello world!' (179ms)OK. 1 assertions passed. (3.797s)
There are no fancy UI wrappers, just a browser window while the tests are running and a console output with the results after that. Also you can omit the browser popup if you want to include the job in your pipeline, there is a config option to start the runner in headless mode.
Let’s add some real world tests
We’ve got the basics so we can try to add some real world scenarios to the tests suite. One of the most common web page elements are the forms and it’s the best start for us (our app depends on multiple webforms with a variety of input elements). For this example we’ll use our Sign Up form because of its rich set of input types.
'[Sign Up page] should report form errors on submit if provided empty form': function (browser) {
browser
.url(browser.launchUrl + '/signup')
.click('button[type=submit]')
.assert.visible('.alert.alert-danger');
browser.expect.elements('.alert.alert-danger ul > li').count.to.equal(8);
browser.expect.element('.alert.alert-danger ul > li:nth-of-type(1)').text.to.contain('The first name field is required.');
browser.expect.element('.alert.alert-danger ul > li:nth-of-type(2)').text.to.contain('The last name field is required.');
browser.expect.element('.alert.alert-danger ul > li:nth-of-type(3)').text.to.contain('The company name field is required.');
// ...
browser.end();
}First thing to do is to submit an empty form and verify whether the validation is working. In the code snippet above you can see how we interact with the form, by clicking the submit button. Then we use some standard assertions to check the visibility of the alert elements and their messages.
[Guest/Signup] Test Suite
=========================
ℹ Connected to localhost on port 4444 (6051ms).
Using: firefox (77.0.1) on linux 5.3.0-59-generic platform.Running: [Sign Up page] should report form errors on submit if provided empty form✔ Testing if element <.alert.alert-danger> is visible (137ms)
✔ Expected elements <.alert.alert-danger ul=""> li> count to equal: "8" (40ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(1)> text to contain: "The first name field is required." (60ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(2)> text to contain: "The last name field is required." (32ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(3)> text to contain: "The company name field is required." (23ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(4)> text to contain: "The email field is required." (26ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(5)> text to contain: "The phone field is required." (25ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(6)> text to contain: "The city field is required." (23ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(7)> text to contain: "The state field is required." (28ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(8)> text to contain: "The marketing strategies field is required." (30ms)OK. 10 assertions passed. (7.664s)
Once we run the tests you can see the interactions with your webpage and the complete log of all assertions in the console output. Next up we have to test the valid state of the form, so let’s try to fill in some details. We’ve got a few basic inputs and a bunch of select2 components so it should be a straightforward task, right?
'[Sign Up page] should report form errors on submit if provided invalid data': function (browser) {
browser
.url(browser.launchUrl + '/signup')
.setValue('#registration--first-name', faker.name.firstName())
.setValue('#registration--last-name', faker.name.lastName())
.setValue('#registration--email', 'invalid-email')
.click('select[id="registration--state"] option[value="California"]')
.click('input[type=submit]')
.assert.visible('.alert.alert-danger');
browser.expect.elements('.alert.alert-danger ul > li').count.to.equal(1);
browser.expect.element('.alert.alert-danger ul > li:nth-of-type(1)').text.to.contain('The email must be a valid email address.');
browser.end();
}Well, if we want to change the value of the standard HTML elements (input, textarea, select, etc) it’s not a problem at all, there are a bunch of predefined interceptors. Unfortunately the same is not valid for a bit more complex elements, as it is the select2 box in our case. We couldn’t find a way to change the selected value both searching for a solution in the docs and the community forums or by attempting to solve the issue by our own working with the UI. After a few hours of unsuccessful tryouts our disappointment took over and we decided to give up. We stuck so early on a problem that was hard to be resolved so we concluded that maybe this won’t be the best tool for us and we have to move on.
Note: Select2 provides a customizable select box with support for searching, tagging, remote data sets, infinite scrolling, and many other highly used options. We find it really handy and we use it a lot in our project so it’s a key point to find a way how to interact within the tests. If you’re still not familiar with this library we strongly recommend you to give it a try.
Yet, we don’t want to put any negative view over Nightwatch.js. It’a a well developed tool with great stats (according to Github and NPM) so you shouldn’t pass it away, it may serve you better than us. Also there is an official support for cloud integrations so you can automate even more your test environment. If you already have any experience with Nightwatch feel free to share it with us, we would be glad to hear from you. We hope you’re still interested in what will be the next tool in our review list so stay tuned for the next chapter of our journey.
Originally published on the MTR Design company website.
MTR Design is a small Bulgarian web development company with expertise in a wide range of technologies (PHP, Python, Golang, NodeJS, React and React Native, Angular, VueJS, iOS and Android development). We are currently open for new projects and cooperations, so if there are any projects we can help with, please react out (use the contact form on our website, or email us at office@mtr-design.com) and we will be happy to discuss them.
