The script `benchmark.js` will collect page load metrics from the
extension, and print them to a file or the console. A method for
collecting metrics was added to the web driver to help with this.
This script will calculate the min, max, average, and standard
deviation for four metrics: 'firstPaint', 'domContentLoaded', 'load',
and 'domInteractive'. The variation between samples is sometimes high,
with the results varying between samples if only 3 were taken. However,
all tests I've done locally with 5 samples have produced results within
one standard deviation of each other. The default number of samples has
been set to 10, which should be more than enough to produce consistent
results.
The benchmark can be run with the npm script `benchmark:chrome` or
`benchmark:firefox`, e.g. `yarn benchmark:chrome`.
There have been intermittent test failures at the beginning of various
e2e test runs. Most tests start with waiting for the 'Welcome' button
to be visible and enabled, which means waiting for the loading screen
to go away.
It looks like the reason the test intermittently fails is that
sometimes the loading screen doesn't appear until a few moments _after_
the page loads (or that it vanishes and comes back).
It was rather difficult to track down each possible cause for the
loading screens, so in the meantime a pause has been added at the start
of each run. This should hopefully suffice to ensure the momentary gap
in loading has been passed by the time the first test starts up.
The driver now has a page navigation function that can navigate to any
of the three primary pages used in the extension. Additional pages and
support of paths can be added later as needed.
The switch case has been moved to a separate function so that the
initialization steps following the web driver instantiation could more
easily be deduplicated.
* Wait until element is clickable before clicking in e2e tests
A new `findClickableElement` has been added to the webdriver to allow
finding an element and blocking until it's both visible and enabled.
This is now used by the pre-existing `clickElement` method as well.
All cases where something is clicked in the e2e tests have been
updated to use one of these methods, to ensure we don't run into
intermittent failures when loading screens take longer than usual.
* Specify type before parameter name
Various JSDoc `@param` entries were specified as `name {type}` rather
than `{type} name`.
A couple of `@return` entries have been given types as well.
* Use JSDoc optional syntax rather than Closure syntax
* Use @returns rather than @return
* Use consistent built-in type capitalization
Primitive types are lower-case, and Object is upper-case.
* Separate param/return description with a dash
The Selenium webdriver is difficult to use, and easy to misuse. To help
use the driver and make it easier to maintain our e2e tests, all driver
interactions are now performed via a `driver` module. This is basically
a wrapper class around the `selenium-webdriver` that exposes only the
methods we want to use directly, along with all of our helper methods.
Update `selenium-webdriver` to v4.0.0-alpha.5. Despite the fact that
this version has "alpha" in the name, the maintainer of
`selenium-webdriver` has described this release as stable [1].
A few APIs were removed or changed in v4, which required changes to our
Firefox webdriver.
The port used for webdriver communication can now be specified
manually. This was required to ensure the threebox tests kept working,
because they used two different driver instances. This new version of
`selenium-webdriver` now uses the same port for each instance of the
webdriver (unlike the old version, which generated a new port for each
one), so it was necessary to manually specify the port to prevent the
same port from being used for both instances.
`chromedriver` required an update, as the version we were using was not
compatible with the new W3C WebDriver protocol. I've updated
`geckodriver` as well, just to bring it in line with the version of
Firefox we are using.
[1]: https://github.com/SeleniumHQ/selenium/issues/5617#issuecomment-373446249
Whymarrh Whitby
Mark Stacey