A margin of error metric has been added, which is calculated from a 95%
confidence interval. This confidence interval is calculated using
Student's t-distribution, which is generally preferred for smaller
sample sizes (< ~30) of populations following a normal distribution.
The script `benchmark.js` will collect page load metrics from the
extension, and print them to a file or the console. A method for
collecting metrics was added to the web driver to help with this.
This script will calculate the min, max, average, and standard
deviation for four metrics: 'firstPaint', 'domContentLoaded', 'load',
and 'domInteractive'. The variation between samples is sometimes high,
with the results varying between samples if only 3 were taken. However,
all tests I've done locally with 5 samples have produced results within
one standard deviation of each other. The default number of samples has
been set to 10, which should be more than enough to produce consistent
results.
The benchmark can be run with the npm script `benchmark:chrome` or
`benchmark:firefox`, e.g. `yarn benchmark:chrome`.
The 'can retype the seed phrase' test would fail sometimes when one of
the words in the seed phrase was a subset of another word (e.g. 'issue'
and 'tissue'). This is because the selector used to find the word
looked for the first element that contained the text, rather than an
exact match.
To simplify the selector and make it more reliable, test ids were added
to each seed phrase word. The selector now uses CSS instead of XPath,
and it only finds exact matches.
A test id was also added to the div containing the shuffled seed words
to select from, so that the chosen seed words wouldn't be selected
in place of the real target when the same word appears twice.
There have been intermittent test failures at the beginning of various
e2e test runs. Most tests start with waiting for the 'Welcome' button
to be visible and enabled, which means waiting for the loading screen
to go away.
It looks like the reason the test intermittently fails is that
sometimes the loading screen doesn't appear until a few moments _after_
the page loads (or that it vanishes and comes back).
It was rather difficult to track down each possible cause for the
loading screens, so in the meantime a pause has been added at the start
of each run. This should hopefully suffice to ensure the momentary gap
in loading has been passed by the time the first test starts up.
The `withFixtures` helper will instantiate ganache, a web driver, and
a fixture server initialized with the given set of fixtures. It is
meant to facilitating writing small, isolated e2e tests.
The first example test has been added: simple-send. It ensures that the
user can send 1 ETH to another account.
These new e2e tests will run during the normal e2e test run.
Closes#6548
The driver now has a page navigation function that can navigate to any
of the three primary pages used in the extension. Additional pages and
support of paths can be added later as needed.
As of #7663, an in-memory store was used in place of local storage
during e2e tests, to facilitate the use of state fixtures. However,
this made it difficult to export state during a test run. The
instructions for exporting state to create fixtures assumed that local
storage was being used.
A new global function has been added to the background context to allow
exporting state. This method is available during testing and
development, and it works with either local storage or the in-memory
store. The fixture instructions have been updated to reference this new
function.
The signature request e2e tests were previously using ropsten. This
expectation was even hard-coded into the test contract dapp.
Instead the contract-dapp has been updated to use the current `chainId`
when calling `signTypedData` (falling back to the `networkId` if
`chainId` is not set). The fixture used by `signature-request` has been
updated to use ganache.
The switch case has been moved to a separate function so that the
initialization steps following the web driver instantiation could more
easily be deduplicated.
The e2e tests were failing intermittently after removing an account
because the account was shown as not deleted after the removal. I
suspect this was because the account _had_ been removed, but that
change to the background state hadn't yet propagated to the UI.
The background state is now synced before the loading overlay for
removing the account is removed, ensuring that the removed account
cannot be seen in the UI after removal.
* Remove unused functions from `mapDispatchToProps`
The actions import was also updated to import only the two actions
used, rather than all actions.
* Remove unused container component
Well, technically it was the props injected by this container that were
unused. The container served no purpose, so the component it surrounded
is now used directly instead.
* Remove both unused `getCurrentViewContext` selectors
* Remove unused SHOW_CONFIG_PAGE action
* Remove checks for `currentView` with name `config`
Now that the SHOW_CONFIG_PAGE action has been removed, it's no longer
possible for `currentView.name` to be set to that value.
* Remove unused `wallet-view` container props
* Delete unused SHOW_SEND_PAGE and SHOW_ADD_TOKEN_PAGE actions
* Remove unused `account-menu.container` props
* Remove unused SHOW_INFO_PAGE action
* Remove unused SET_NEW_ACCOUNT_FORM action
* Wait until element is clickable before clicking in e2e tests
A new `findClickableElement` has been added to the webdriver to allow
finding an element and blocking until it's both visible and enabled.
This is now used by the pre-existing `clickElement` method as well.
All cases where something is clicked in the e2e tests have been
updated to use one of these methods, to ensure we don't run into
intermittent failures when loading screens take longer than usual.
These rows on the Advanced Settings page were being looked up in the
e2e tests by the order they appeared in. Instead they're now referenced
by data id, so that we can add new settings and re-arrange them without
breaking the e2e tests.
The transaction navigation in the e2e tests has been made simpler with
the addition of data attributes to help with finding the navigation
buttons. Each button is now labelled according to its purpose.
* Specify type before parameter name
Various JSDoc `@param` entries were specified as `name {type}` rather
than `{type} name`.
A couple of `@return` entries have been given types as well.
* Use JSDoc optional syntax rather than Closure syntax
* Use @returns rather than @return
* Use consistent built-in type capitalization
Primitive types are lower-case, and Object is upper-case.
* Separate param/return description with a dash
The Selenium webdriver is difficult to use, and easy to misuse. To help
use the driver and make it easier to maintain our e2e tests, all driver
interactions are now performed via a `driver` module. This is basically
a wrapper class around the `selenium-webdriver` that exposes only the
methods we want to use directly, along with all of our helper methods.
These tests were updated in #7473 to navigate in a different order,
because the transaction order changed. Unfortunately this meant that
a second contract deployment was being confirmed, where it was
previously being rejected.
This updates the test to ensure the same transaction is rejected and
confirmed as prior to the change in #7473
The account details close button is difficult to click from the e2e
tests because it has a size of zero. The actual icon is added via CSS
as an `::after` pseudo-element.
The CSS has been adjusted to give the icon a size, and it the markup
is now a `button` rather than a `div`.
* Make gas estimate update on debounced token amount change, not just on blur after change
* Updated tests
* Ensure `updateGas` is bound early
Co-authored-by: Mark Stacey <markjstacey@gmail.com>
Typically the fullscreen UI will open upon installation, though this
behaviour was suppressed in development. This was dealt with in the e2e
tests by waiting for it to open, then closing it.
Instead this behaviour is now suppressed for test builds as well.
* Improve `openNewPage` helper function
The two delays were removed, and the window handle for the new page is
now returned. This was made possible with the new `newWindow` function
added in `v4.0.0-alpha.3` of `selenium-webdriver`.
* Replace recursion with loops
This should result in far more pleasant stack traces for any
exceptions in these functions. It might also be faster. These functions
seem easier to understand as loops as well.
* Remove unused string parameter
The `closeAllWindowHandlesExcept` function has been simplified by
removing a branch that handles the case where the `exceptions`
parameter given is a string. That parameter is never a string.
Update `selenium-webdriver` to v4.0.0-alpha.5. Despite the fact that
this version has "alpha" in the name, the maintainer of
`selenium-webdriver` has described this release as stable [1].
A few APIs were removed or changed in v4, which required changes to our
Firefox webdriver.
The port used for webdriver communication can now be specified
manually. This was required to ensure the threebox tests kept working,
because they used two different driver instances. This new version of
`selenium-webdriver` now uses the same port for each instance of the
webdriver (unlike the old version, which generated a new port for each
one), so it was necessary to manually specify the port to prevent the
same port from being used for both instances.
`chromedriver` required an update, as the version we were using was not
compatible with the new W3C WebDriver protocol. I've updated
`geckodriver` as well, just to bring it in line with the version of
Firefox we are using.
[1]: https://github.com/SeleniumHQ/selenium/issues/5617#issuecomment-373446249
* Remove unused onboarding stream
* Pass `sender` through to `setupProviderEngine`
The Port `sender` has been passed down a few more layers. This allows
us to get more information from the sender deeper in the stack, but
also simplifies things a bit as well. For example, now the "fake"
URL object with the `metamask` hostname is no longer needed.
* Create onboarding middleware
This middleware intercepts `wallet_registerOnboarding` RPC messages. It
will register the sender as an oboarding initiator if possible, and
otherwise ignores the message.
* Add network store for testing
An alternative persistent state store has been created for use with e2e
tests. Instead of reading state from disk, it tries to load state from
a local fixture server running on port `12345` and serving state from
the path `/state.json`, and returns a blank state otherwise.
* Add e2e test fixture server
A fixture server has been added for serving background state, which the
background will read upon startup as part of restoring persisted state.
The `signature-request` e2e test has been updated to use a fixture to
bypass the registration step. The fixture used (`imported-account`) was
generated by pausing midway through that test run
* Add Get Accounts button
This button calls `eth_accounts`. This button is always enabled, even
when not connected.
* Disable account buttons by default
The buttons that require you to have first connected have been disabled
by default. Previously they would be enabled until the JavaScript
finished initializing the page, at which point they'd be disabled. This
resulted in a distracting flash as the page loaded and the buttons
changed.
The `signTypedData` button was added to the accounts button set as
well, rather than being left enabled regardless of connected status.
* Allow connect button to become re-enabled
The Connect button was broken previously in that after being disabled,
it would stay disabled even if the dapp lost access to MetaMask. The
button will now be enabled whenever not connected.
* Stringify signTypedData results
* Defer metamask onboarding bundle to speed up page load
The `ganache.js` helper module uses `ganache-core` to start `ganache`
instead of `ganache-cli`, and allows all of the same customization.
Using `ganache` programmatically from our e2e tests is much faster, as
we don't have to wait that arbitrary 5 seconds before each test as we
wait for `ganache-cli` to start up.
Mark Stacey
Whymarrh Whitby
Erik Marks
Dan J Miller
Frankie