The benchmark script can now be set to retry upon failure, like the E2E
tests do. The default is zero, just as with the E2E tests. A retry of 2
has been set in CI to match the E2E tests as well.
The `retry` module had to be adjusted to throw an error in the case of
failure. Previously it just set the exit code, but that only worked
because it was the last thing called before the process ended. That is
no longer the case.
Our benchmark script now uses `yargs`. Functionally it should be nearly
the same as before, except that now it has more documentation and
validation. The one functional difference aside from that is that the
`--pages` flag now takes space-separated arguments rather than comma-
separated.
Previously the benchmark script would throw an error if asked to take
just 1 sample. Now it works, though the stats returned are of
dubious use.
The problem was that it was impossible to calculate the standard
deviation or margin of error of a set of 1. Instead it now returns
zero for both of those values in the single-sample case, which is what
it would return for two identical samples.
* Add `--leave-running` flag to E2E test script
The `--leave-running` flag has been added to the E2E test runner. This
ensures the browser, ganache, and everything else stays running upon
test failure. This is useful for local debugging, for investigating
what state the extension was in when it failed.
* Add `--leave-running` support to `metamask-ui.spec.js`
This script makes it easier to run an individual E2E test. In the past
I've run individual scripts by editing `run-all.sh` manually, but now
that can be done more easily with this script. It also allows setting
the number of retries to use and the browser to use from the CLI.
This script has been added as an npm script as well, called
'test:e2e:single'.
The `run-all.sh` script was rewritten in JavaScript to make it easier
to pass through a `--retries` argument.
The default number of retries has been set to zero to make local
testing easier. It has been set to 2 on CI.
This was mainly done to consolidate the code used to run an E2E test in
one place, to make later improvements easier.
Chrome logs are now enabled for E2E tests when the 'ENABLE_CHROME_LOGS'
environment variable is set to anything other than `false`.
This was helpful to me in debugging Chrome crashes on CI, the ones with
the error "unknown error: DevToolsActivePort file doesn't exist". This
was the only way to discover the cause of the error. It's also useful
for discovering console errors from the background process or from the
UI.
It's disabled by default because it makes the test output quite noisy
and difficult to read.
The dapp is now started directly from the `metamask-ui.spec.js` test
module. This makes it easier to run independently, and brings it in-
line with our other E2E tests.
The `--no-timeouts` flag is now used as well, rather than setting the
timeout to `0` within the test. This also brings it in-line with our
other tests.
Mainly this was done to facilitate further refactors which will come in
later PRs.
Dan J Miller
Mark Stacey
ryanml
Brad Decker
Alex Donesky
sooyoung
PeterYinusa
David Walsh
Etienne Dusseault
MetaMask Bot
Niranjana Binoy