At the very beginning of Claro theme’s development Lauri Eskola (front end framework manager of Drupal) declared that the theme has to support Chrome, Firefox, Internet Explorer 11, Edge and Safari. The theme has to work properly with these browsers on their most common operating systems. He also wanted me to create screenshots about the actual component to prove that it looks as designed on every target platform.

The problem

Fortunately, we already realized at the beginning of the development that the most expedient working method is to write dummy modules that display the components we are about to create. These dummy modules make the development a lot easier. After I thought I finished the task, it was time to create the screenshots.

For the first task, I tried to take these screenshots by hand. This proved to be a very big challenge: it wasn’t just about repeating the scenario on five different systems, in five different browsers (11 variations overall), but it was really easy to make a mistake. In addition, there were several cases where the appropriate input element state (:hover + focus) was only capturable with serious tricks1.

  Ubuntu macOS Win. Android iOS 12
Chrome ⬤ ⬤ ⬤ ⬤  
Firefox ⬤ ⬤ ⬤    
Safari   ⬤     ⬤
IE 11     ⬤    
Edge     ⬤    

Once I spent 4–6 hours on taking and combining screenshots, it became evident that we have to find a solution to automate the process (at least partially).

The idea

Since we have all the fundamentals in the picture of the dummy modules, we only need a tool being able to perform the same runbook on the browser instances we need:

  1. We need a Drupal 8 instance that has the actual dummy module being installed.
  2. Then this Drupal instance gets tested in a real browser by a tool…
  3. …Which tool is able to create screenshots (let’s be precise: browser viewport shots).

Nightwatch.js was added to Drupal core this summer, let’s check it out!

What is Nightwatch.js?

Nightwatch.js is an E2E testing framework made for testing websites and web applications on “real” browsers.

I read that Drupal people are talking about it, it is using W3C WebDriver API to interact with browsers. And I also see that I can ask it to take a screenshot. I played with it a bit and made sure it could work not only with WebDriver middlewares (like chromedriver or geckodriver) but also Selenium server.

So our hard requirements above are met!

The code

Test as a screenshot script

Basically, I’m writing Nightwatch.js tests which are testing the UI that the demo modules provide. The first few tasks are about form elements (HTML inputs), where it is particularly important to check the appearance of element states.

What happens if I check a checkbox with the keyboard? What does the element look like when it is in focus and also the pointer is over it? And what if this element also has an error warning, and its border color is different?

Well, after a couple of tries, I think I can write a script as a JS test like a pro now!

Taking screenshots

Another challenging task is to create full page screenshots. I wasn’t able (because it is impossible2) to resize the browser window on each system to be able to capture the whole webpage we are interacting with. So I had to write a custom script, a Nightwatch.js command, which scrolls to the top of the page, saves the visible viewport to a file, then it scrolls downwards, takes another screenshot, and repeats these steps until it reaches the bottom of the page.

The merged version of the screenshot pieces can be created with a cli JS script which is using Jimp

And this is an example image that shows the textarea and editor error style updates:

'Textarea and editor error styles. Seven theme on the left-hand side, Claro on the right' Textarea and editor error styles. Seven theme on the left-hand side, Claro on the right.

The infrastructure

I spent most of my time assembling the infrastructure. The final solution — the tool for launching browsers, and managing communication between the Nightwatch.js tests and the browsers — eventually became a Selenium grid.

The first challenge was to identify the latest version of Selenium that the ancient version of Nightwatch.js used in Drupal can properly communicate with. Well, it wasn’t the latest Selenium release!

I’ve set up a Selenium grid, which consists of a central communication server (this is called hub), and an unlimited number of “sub” servers (nodes). The hub acts as a router between the Nightwatch.js tests and the nodes; and the node is responsible for starting and controlling the actual web browser instances, and performing the interactive actions we ask for in the tests.

I have four Selenium nodes:

  • The first one I configured is the node I’m using for controlling the browsers on my macOS localhost: Safari, Firefox, and Google Chrome.
  • I have another node running in a Ubuntu virtual machine with Firefox and Google Chrome.
  • I have yet-another two Selenium nodes, one in a Windows 8.1 VM, and another one in a Windows 10 VM; with Google Chrome, Firefox, Internet Explorer 11 and Microsoft Edge.
  • And for being able to test on mobile devices, I also have an Appium server which controls an emulated Android device with Chrome for Android, and an iOS device with Safari.

Local Selenium node setup

Since I am on macOs, Safari browser was already available. I only needed an installed Java SE, Google Chrome, Firefox and Safari Webdriver (latter should be enabled, instructions are here).

And besides these, I also had to get:

To simplify the maintenance, I always rename the downloaded and extracted Selenium server and the webdriver executables to a name that includes their version number and place them into the same folder.

Configuration and DX

Selenium hub

I wrote a shell script hub-up.sh to start the Selenium hub (because this way I don’t have to remember the parameters):



1
2
3
4
5
6
7



#!/bin/bash
HOST=${1:-"$(ipconfig getifaddr en0)"}
java \
  -jar "${HOME}/selenium/selenium-server-standalone-3.4.0.jar" \
  -port 4444 \
  -role hub \
  -host "${HOST}"

Selenium node

For the local Selenium node instance, we have to provide the paths to the webdriver executables, the endpoint of the Selenium hub and also define our node’s capabilities in a JSON config file, e.g. selenium-node-config--3.4.0--osx.json:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34



{
  "capabilities": [
    {
      "browserName": "firefox",
      "maxInstances": 5,
      "seleniumProtocol": "WebDriver"
    },
    {
      "browserName": "chrome",
      "maxInstances": 5,
      "seleniumProtocol": "WebDriver"
    },
    {
      "browserName": "safari",
      "maxInstances": 1,
      "seleniumProtocol": "WebDriver"
    }
  ],
  "proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
  "maxSession": 5,
  "port": 5501,
  "register": true,
  "registerCycle": 5000,
  "hub": "http://localhost:4444",
  "nodeStatusCheckTimeout": 5000,
  "nodePolling": 5000,
  "role": "node",
  "unregisterIfStillDownAfter": 60000,
  "downPollingLimit": 2,
  "debug": false,
  "servlets" : [],
  "withoutServlets": [],
  "custom": {}
}

The Selenium node can be started with the following script:



1
2
3
4
5
6
7
8
9



#!/bin/bash
HOST=${1:-"$(ipconfig getifaddr en0)"}
java \
  -Dwebdriver.gecko.driver="${HOME}/selenium/geckodriver-0.22.0" \
  -Dwebdriver.chrome.driver="${HOME}/selenium/chromedriver-chrome-75" \
  -jar "${HOME}/selenium/selenium-server-standalone-3.4.0.jar" \
  -role node \
  -hub "http://${HOST}:4444/grid/register" \
  -nodeConfig "${HOME}/selenium/selenium-node-config--3.4.0--osx.json"

I hope you enjoyed reading and found this post useful. I’ll be coming up soon with a recipe for nodes running on virtual machines.

Sources:

Footnotes:

  1. For example because the key combination for taking a screenshot stole the focus from that item. ↩

  2. Nothing happens if you try to resize the browser window of Internet Explorer, headless Firefox (on any system), or on mobile devices (latter is obvious at least) ↩

  3. Selenium 3.4.0 is the highest version which is compatible with the ancient Nightwatch.js version) used by Drupal core. ↩