The Firefox extension version format does not support the version
format we use (SemVer), so we have to specially format the extension
version to be compatible. The format we chose was
`[major].[minor].[patch].[buildType][buildVersion]`. But when we tried
to submit a build with a version in that format, it was rejected as
invalid for unknown reasons.
The Firefox extension format has been updated to
`[major].[minor].[patch][buildType][buildVersion]`. This seems to pass
validation.
The `version_name` manifest field was being used on Chrome to store the
build type. However, Chrome intended this field to be a full
representation of the version, for display purposes. This was evident
when uploading this version to the Chrome Web Store, because it used
`flask` as the entire version.
Instead the `version_name` field now includes the full SemVer version
string. The version parsing code within the build script and in the
wallet itself have been updated accordingly.
The build script only allowed prerelease versions for the "beta" build
type (e.g. `X.Y.Z-beta.0`). Now it allows Flask prerelease versions as
well.
This is required for the Flask release, where the prerelease version
helps distinguish the Flask error reports and metrics.
The ESLint config has been updated to v8. The breaking changes are:
* The Prettier rule `quoteProps` has been changed from `consistent` to
`as-needed`, meaning that if one key requires quoting, only that key is
quoted rather than all keys.
* The ESLint rule `no-shadow` has been made more strict. It now
prevents globals from being shadowed as well.
Most of these changes were applied with `yarn lint:fix`. Only the
shadowing changes required manual fixing (shadowing variable names were
either replaced with destructuring or renamed).
The dependency `globalThis` was added to the list of dynamic
dependencies in the build system, where it should have been already.
This was causing `depcheck` to fail because the new lint rules required
removing the one place where `globalThis` had been erroneously imported
previously.
A rule requiring a newline between multiline blocks and expressions has
been disabled temporarily to make this PR smaller and to avoid
introducing conflicts with other PRs.
`remote-redux-devtools` is now explicitly excluded and disabled in non-
dev builds, and in the `testDev` build. This was causing console errors
in the `testDev` build during e2e tests, which would cause certain
tests to fail.
This was already only supposed to be enabled for development builds,
but this library used the `NODE_ENV` environment variable to make that
determination. This gives us more control over when it's disabled.
The React dev tools can result in console errors if dev tools is not
open during the test. Some of our e2e tests fail if there are any
console errors, so these errors break those tests.
`react-devtools` has been completely disabled for `testDev` builds to
make debugging e2e tests easier. The React dev tools can still be used
from development builds.
A propType error was showing up during e2e tests with a `testDev`
build. It was caused by `process.env.IN_TEST` being treated as a
boolean, when in fact it is either the string `'true'` or a boolean.
`IN_TEST` has been updated to always be a boolean. `loose-envify` has
no trouble injecting boolean values, so there's no reason to treat this
as a string.
The coverage reporter for the console has been changed from `text` to
`text-summary` because `text` was too long. It exceeded the maximum
length of the CircleCI terminal output shown on their jobs page, making
it very difficult to see why the unit test coverage job failed.
The text summary only displays overall coverage metrics, but that is
usually enough to indicate when the test fails due to a drop in
coverage. The more detailed breakdown is still available as a HTML
report linked in the `metamaskbot` comment, and in the `jest-coverage`
directory when run locally.
The three on-disk coverage reports used previously (`lcov`, `json`, and
`clover`) have been replaced with just `html`. The HTML report was part
of the `lcov` report, and it was the only one we were using.
The environment variables used for test builds was wrong for certain
bundles because the `testing` flag wasn't passed through to the
function that determines which environment variables to inject.
Effectively this means that test builds on `master` were going to the
production `metamask` Sentry project rather than the `test-metamask`
project. This has been the case since #11080.
The `testing` flag is now included for all bundles, and test builds now
use the `test-metamask` Sentry project in all cases.
The LavaMoat policy generation script would sporadically fail because
it ran the build concurrently three times, and the build includes
steps that delete the `dist` directory and write to it. So if one build
process tried to write to the directory after another deleted it, it
would fail.
This was solved by adding a new `--policy-only` flag to the build
script, and a new `scripts:prod` task. The `scripts:prod` task only
runs the script tasks for prod, rather than the entire build process.
The `--policy-only` flag stops the script tasks once the policy has
been written, and stops any other files from being written to disk.
This prevents the three concurrent build processes from getting in each
others way, and it dramatically speeds up the process.
The LavaMoat policy generation script would sporadically fail because
it ran the build concurrently three times, and the build includes
steps that delete the `dist` directory and write to it. So if one build
process tried to write to the directory after another deleted it, it
would fail.
This was solved by adding a new `--policy-only` flag to the build
script, and a new `scripts:prod` task. The `scripts:prod` task only
runs the script tasks for prod, rather than the entire build process.
The `--policy-only` flag stops the script tasks once the policy has
been written, and stops any other files from being written to disk.
This prevents the three concurrent build processes from getting in each
others way, and it dramatically speeds up the process.
The environment variables used for test builds was wrong for certain
bundles because the `testing` flag wasn't passed through to the
function that determines which environment variables to inject.
Effectively this means that test builds on `master` were going to the
production `metamask` Sentry project rather than the `test-metamask`
project. This has been the case since #11080.
The `testing` flag is now included for all bundles, and test builds now
use the `test-metamask` Sentry project in all cases.
This PR improves the error handling of the code fence removal transform stream by catching errors thrown by the `removeFencedCode` function and passing them to the `end` callback. This appears to resolve a problem where watched builds would blow up whenever a file with fences was reloaded.
This PR adds one LavaMoat background script policy or each build type. It also renames the build system policy directory from `node` to `build-system` to make its purpose more clear. Each build type has the original `policy-override.json` for `main` builds. The `.prettierignore` file has been updated to match the locations of the new auto-generated policy files.
We need to maintain separate policies for each build type because each type will produce different bundles with different internal and external modules.
Co-authored-by: Mark Stacey <markjstacey@gmail.com>
The build system now supports platform-specific modifications to the
manifest for each build type. The need to customize the `id` on Firefox
motivated this change.
To support this, a new directory was made in each build type directory
for manifest changes. The images currently in this directory were moved
into an `images` subdirectory.
This new `manifest` directory can include each manifest file currently
in `app/manifest`. The `_base.json` file is assumed to exist, but the
platform manifest modifications are optional.
The build system now supports platform-specific modifications to the
manifest for each build type. The need to customize the `id` on Firefox
motivated this change.
To support this, a new directory was made in each build type directory
for manifest changes. The images currently in this directory were moved
into an `images` subdirectory.
This new `manifest` directory can include each manifest file currently
in `app/manifest`. The `_base.json` file is assumed to exist, but the
platform manifest modifications are optional.
* GridPlus: Adds support for GridPlus Lattice1 hardware wallet
* Fixes issue with switching hardware HD path
The main `Select HD Path` piece of the account selection component was not
properly hooked up to the state manager (`onPathChange`) and the extra
`Popover` component was being used instead.
I'm not sure what the origin of this is, but I don't see why the Popover
is needed at all. I have remove it and hooked `onPathChange` directly into
the HD path selector dropdown.
This was an issue that nearly every Lattice user who had come from Ledger
has contacted us about.
* GridPlus: Addresses QA issues
* Adds Lattice tutorial + image
* Cleans up connectivity issues (see: https://github.com/GridPlus/eth-lattice-keyring/pull/16)
* GridPlus: Adds Firefox support
To connect to the Lattice you need to open a new tab/window and get
login data from it. We were not able to do this for Firefox because
we relied on the `window` API. This is now fixed.
See corresponding changes:
* `eth-lattice-keyring`: https://github.com/GridPlus/eth-lattice-keyring/pull/17
* Lattice connector: https://github.com/GridPlus/wallet-web/pull/152
* GridPlus: Adds missing error path for Firefox
See: 242a93f559
The beta and Flask builds are now built on CI and included in the
metamask bot comment alongside the main builds. The same sourcemap
linter and mozilla linter used for the prod builds is also run on the
beta and Flask builds.
Closes#12426
The beta and Flask builds are now built on CI and included in the
metamask bot comment alongside the main builds. The same sourcemap
linter and mozilla linter used for the prod builds is also run on the
beta and Flask builds.
Closes#12426
This PR enables the exclusion of JavaScript and JSON source by `buildType`, and enables the running of `eslint` under LavaMoat. 80-90% of the changes in this PR are `.patch` files and LavaMoat policy additions.
The file exclusion is designed to work in conjunction with our code fencing. If you forget to fence an import statement of an excluded file, the application will now error on boot. **This PR commits us to a particular naming convention for files intended only for certain builds.** Continue reading for details.
### Code Fencing and ESLint
When a file is modified by the code fencing transform, we run ESLint on it to ensure that we fail early for syntax-related issues. This PR adds the first code fences that will be actually be removed in production builds. As a consequence, this was also the first time we attempted to run ESLint under LavaMoat. Making that work required a lot of manual labor because of ESLint's use of dynamic imports, but the manual changes necessary were ultimately quite minor.
### File Exclusion
For all builds, any file in `app/`, `shared/` or `ui/` in a sub-directory matching `**/${otherBuildType}/**` (where `otherBuildType` is any build type except `main`) will be added to the list of excluded files, regardless of its file extension. For example, if we want to add one or more pages to the UI settings in Flask, we'd create the folder `ui/pages/settings/flask`, add any necessary files or sub-folders there, and fence the import statements for anything in that folder. If we wanted the same thing for Beta, we would name the directory `ui/pages/settings/beta`.
As it happens, we already organize some of our source files in this way, namely the logo JSON for Beta and Flask builds. See `ui/helpers/utils/build-types.js` to see how this works in practice.
Because the list of ignored filed is only passed to `browserify.exclude()`, any files not bundled by `browserify` will be ignored. For our purposes, this is mostly relevant for `.scss`. Since we don't have anything like code fencing for SCSS, we'll have to consider how to handle our styles separately.
The code fence transform was including contents after the final END directive twice. That was not covered by the tests, because none of the examples contained any content after the final END directive, and concatenating the empty string twice does not produce an observable difference in the test results.
This bug was due to an off-by-one error in the loop of the multiSplice function. The error has been fixed, and more test cases have been added.
Static files have been added for the Flask build. This includes logos
of each size and variety that we use, and it includes the 3D model JSON
file.
Closes#12427
The production build was accidentally broken in #12440 because of a
merge conflict with a #12441 that wasn't initially noticed. The
conflict was the renaming of the `BuildTypes` variable to `BuildType`.
This variable is used to check the current build type, but only for
production builds. `BuildTypes` is `undefined`, so this would result in
a crash when that enum was used.
The build script has been updated to embed the correct Infura project
ID and Segment write key for beta and Flask builds. These are set via
environment variable or config file. They have already been added in CI
as environment variables.
The Segment production write key has also been moved into the set of
environment variables that can be set in the configuration file. This
was to make the way we reference it more consistent.
The new project IDs and keys are only used in the "production"
environment, which right now is the merge step into the `master`
branch. This is appropriate for Flask, but it doesn't match our plan
for how the beta release would get created. In a future PR, when the
beta release automation work is completed, the conditions for when
the beta secrets are used should be updated to ensure they're used only
for the beta builds.
Closes#11896
Recently validation was added for our build configuration as part of
the PR #12438. This had the unintended consequence of making all builds
from forks fail because they don't get secrets injected. Specifically
it was the missing `INFURA_PROJECT_ID` that made the builds fail.
The Infura project ID is no longer required for building. In practice
it's still required for doing anything with a build but running e2e
tests, but that's all we need to do in CI anyway.
The build type (i.e. the distribution) is now included in the Sentry
environment during setup, for all builds except the "main" build. This
will allow us to track Flask and beta errors separately from other
errors.
A constant was created for the build types. The equivalent constant in
our build scripts was updated to match it more closely, for
consistency. We can't use the same constant in both places because our
shared constants are in modules that use ES6 exports, and our build
script does not yet support ES6 exports.
The singular `BuildType` was used rather than `BuildTypes` to match our
naming conventions elsewhere for enums. We name them like classes or
types, rather than like a collection.
Relates to #11896
We now use two separate Infura project IDs for production builds, and
for all other builds. Previously all CI builds used the production
Infura project ID. Separating them will make our Infura dashboard
metrics more representative of real production usage.
The new environment variable for production has been setup in CI
already, but the old environment variable will remain set to the
production project ID until this commit is included in a release.
We can't switch the old environment variable out until we're confident
that it won't get used for a production build.
We now use constants for the various different build environments. This
was done to improve the JSDoc types of the `getInfuraProjectId` helper
method.
The `getConfigValue` function was added to make it easier to validate
that required config values are set. This should ensure builds fail
early with an informative error message when they are missing the
necessary configuration.
The main `version` field in `package.json` will now include the beta
version (if present) rather than it being passed in via the CLI when
building. The `version` field is now a fully SemVer-compatible version,
with the added restriction that any prerelease portion of the version
must match the format `<build type>.<build version>`.
This brings the build in-line with the future release process we will
be using for the beta version. The plan is for each future release to
enter a "beta phase" where the version would get updated to reflect
that it's a beta, and we would increment this beta version over time as
we update the beta. The manifest gives us a place to store this beta
version. It was also important to replace the automatic minor bump
logic that was being used previously, because the version in beta might
not be a minor bump.
Additionally, the filename logic used for beta builds was updated to
be generic across all build types rather than beta-specific. This will
be useful for Flask builds in the future.
* lavamoat - add lavamoat to webapp background
* test:e2e - add delay to resolve failure
* test:e2e - add delay to resolve failure
* build - add a switch for applying lavamoat, currently off for all
* test/e2e - remove delays added for lavamoat
* Revert "test/e2e - remove delays added for lavamoat"
This reverts commit 79c3479f15c072ed362ba1d4f1af41ea11a17d63.
* lockdown - breakout making globalThis properties non-writable into lockdown-more.js
* Update app/scripts/lockdown-more.js
Co-authored-by: David Walsh <davidwalsh83@gmail.com>
* Update app/scripts/lockdown-more.js
Co-authored-by: Erik Marks <25517051+rekmarks@users.noreply.github.com>
Co-authored-by: David Walsh <davidwalsh83@gmail.com>
Co-authored-by: Erik Marks <25517051+rekmarks@users.noreply.github.com>
The MetaMask logo used for beta development builds was wrong. The lock
screen (and any other place using the `@metamask/logo` logo) showed the
correct logo, but all of our static assets used the "regular" logo.
Now the beta logo should be used everywhere for beta development
builds.
This is a refactor to replace the `isBeta` boolean with `buildType`
throughout the build system. This will allow us to modify the behaviour
of each step of the build process for Flask as well.
This should result in no functional changes.
This PR adds build-time code exclusion by means of code fencing. For details, please see the README in `./development/build/transforms`. Note that linting of transformed files as a form of validation is added in a follow-up, #12075.
Hopefully exhaustive tests are added to ensure that the transform works according to its specification. Since these tests are Node-only, they required their own Jest config. The recommended way to work with multiple Jest configs is using the `projects` field in the Jest config, however [that feature breaks coverage collection](https://github.com/facebook/jest/issues/9628). That being the case, I had to set up two separate Jest configs. In order to get both test suites to run in parallel, Jest is now invoked via a script, `./test/run-jest.sh`.
By way of example, this build system feature allows us to add fences like this:
```javascript
this.store.updateStructure({
...,
GasFeeController: this.gasFeeController,
TokenListController: this.tokenListController,
///: BEGIN:ONLY_INCLUDE_IN(beta)
PluginController: this.pluginController,
///: END:ONLY_INCLUDE_IN
});
```
Which at build time are transformed to the following if the build type is not `beta`:
```javascript
this.store.updateStructure({
...,
GasFeeController: this.gasFeeController,
TokenListController: this.tokenListController,
});
```
Co-authored-by: Mark Stacey <markjstacey@gmail.com>
This rationalizes how arguments are passed to and parsed by the build system. To accomplish this, everything that isn't an environment variable from `.metamaskrc` or our CI environment is now passed as an argument on the command line.
Of such arguments, the `entryTask` is still expected as a positional argument in the first position (i.e. `process.argv[2]`), but everything else must be passed as a named argument. We use `minimist` to parse the arguments, and set defaults to preserve existing behavior.
Arguments are parsed in a new function, `parseArgv`, in `development/build/index.js`. They are assigned to environment variables where convenient, and otherwise returned from `parseArgv` to be passed to other functions invoked in the same file.
This change is motivated by our previous inconsistent handling of arguments to the build system, which will grow increasingly problematic as the build system grows in complexity. (Which it will very shortly, as we introduce Flask builds.)
Miscellaneous changes:
- Adds a build system readme at `development/build/README.md`
- Removes the `beta` package script. Now, we can instead call: `yarn dist --build-type beta`
- Fixes the casing of some log messages and reorders some parameters in the build system
There are a few issues encountered when running `yarn setup` on new
Apple Silicon (aka M1, aka arm64) Macs:
* The script halts when attempting to run the install step for
the `chromedriver` package with the message "Only Mac 64 bits
supported". This is somewhat misleading as it seems to indicate that
chromedriver can only be installed on a 64-bit Mac. However, what I
think is happening is that the installation script for `chromedriver`
is not able to detect that an arm64 CPU *is* a 64-bit CPU. After
looking through the `chromedriver` repo, it appears that 87.0.1 is the
first version that adds a proper check ([1]).
Note that upgrading chromedriver caused the Chrome-specific tests to
fail intermittently on CI. I was not able to 100% work out the reason
for this, but ensuring that X (which provides a way for Chrome to run
in a GUI setting from the command line) is available seems to fix
these issues.
* The script also halts when attempting to run the install step for
the `electron` package. This happens because for the version of
`electron` we are using (9.4.2), there is no available binary for
arm64. It appears that Electron 11.x was the first version to support
arm64 Macs ([2]). This is a bit trickier to resolve because we don't
explicitly rely on `electron` — that's brought in by `react-devtools`.
The first version of `react-devtools` that relies on `electron` 11.x
is 4.11.0 ([3]).
[1]: 469dd0a6ee
[2]: https://www.electronjs.org/blog/apple-silicon
[3]: https://github.com/facebook/react/blob/main/packages/react-devtools/CHANGELOG.md#4110-april-9-2021
Mark Stacey
kumavis
Erik Marks
Alex Miller
Jyoti Puri
ryanml
David Walsh
Alex Donesky
Daniel
Elliot Winkler