* Add benchmark to CI
The page load benchmark for Chrome is now run during CI, and the
results are collected and summarized in the `metamaskbot` comment.
Closes#6881
* Double default number of samples
The number of default samples was changed from 10 to 20. The results
from 10 samples would show statistically significant changes in page
load times between builds, so weren't a sufficiently useful metric.
A margin of error metric has been added, which is calculated from a 95%
confidence interval. This confidence interval is calculated using
Student's t-distribution, which is generally preferred for smaller
sample sizes (< ~30) of populations following a normal distribution.
The script `benchmark.js` will collect page load metrics from the
extension, and print them to a file or the console. A method for
collecting metrics was added to the web driver to help with this.
This script will calculate the min, max, average, and standard
deviation for four metrics: 'firstPaint', 'domContentLoaded', 'load',
and 'domInteractive'. The variation between samples is sometimes high,
with the results varying between samples if only 3 were taken. However,
all tests I've done locally with 5 samples have produced results within
one standard deviation of each other. The default number of samples has
been set to 10, which should be more than enough to produce consistent
results.
The benchmark can be run with the npm script `benchmark:chrome` or
`benchmark:firefox`, e.g. `yarn benchmark:chrome`.
Erik Marks
Whymarrh Whitby
Mark Stacey