* Create `.zip` files deterministically
Our build system now creates `.zip` archives deterministically.
Previously the `.zip` file would differ between builds even when the
files being archived were identical. This was because the order the
files were passed in was non-deterministic, and the `mtime` for each
file was different between builds.
The files are now sorted before being zipped, and the `mtime` for each
file has been set to the unix epoch.
* Update lavamoat build policy
* Create `.zip` files deterministically
Our build system now creates `.zip` archives deterministically.
Previously the `.zip` file would differ between builds even when the
files being archived were identical. This was because the order the
files were passed in was non-deterministic, and the `mtime` for each
file was different between builds.
The files are now sorted before being zipped, and the `mtime` for each
file has been set to the unix epoch.
* Update lavamoat build policy
This commit modifies the build system so that TypeScript files can be
transpiled into ES5 just like JavaScript files.
Note that this commit does NOT change the build system to run TypeScript
files through the TypeScript compiler. In other words, no files will be
type-checked at the build stage, as we expect type-checking to be
handled elsewhere (live, via your editor integration with `tsserver`,
and before a PR is merged, via `yarn lint`). Rather, we merely instruct
Babel to strip TypeScript-specific syntax from any files that have it,
as if those files had been written using JavaScript syntax alone.
Why take this approach? Because it prevents the build process from being
negatively impacted with respect to performance (as TypeScript takes a
significant amount of time to run).
It's worth noting the downside of this approach: because we aren't
running files through TypeScript, but relying on Babel's [TypeScript
transform][1] to identify TypeScript syntax, this transform has to keep
up with any syntax changes that TypeScript adds in the future. In fact
there are a few syntactical forms that Babel already does not recognize.
These forms are rare or are deprecated by TypeScript, so I don't
consider them to be a blocker, but it's worth noting just in case it
comes up later. Also, any settings we place in `tsconfig.json` will be
completely ignored by Babel. Again, this isn't a blocker because there
are some analogs for the most important settings reflected in the
options we can pass to the transform. These and other caveats are
detailed in the [documentation for the transform][2].
[1]: https://babeljs.io/docs/en/babel-plugin-transform-typescript
[2]: https://babeljs.io/docs/en/babel-plugin-transform-typescript#caveats
This commit allows developers to write TypeScript files and lint them
(either via a language server in their editor of choice or through the
`yarn lint` command).
The new TypeScript configuration as well as the updated ESLint
configuration not only includes support for parsing TypeScript files,
but also provides some compatibility between JavaScript and TypeScript.
That is, it makes it possible for a TypeScript file that imports a
JavaScript file or a JavaScript file that imports a TypeScript file to
be linted.
Note that this commit does not integrate TypeScript into the build
system yet, so we cannot start converting files to TypeScript and
pushing them to the repo until that final step is complete.
ESLint rules have been added to enforce our JSDoc conventions. These
rules were introduced by updating `@metamask/eslint-config` to v9.
Some of the rules have been disabled because the effort to fix all lint
errors was too high. It might be easiest to enable these rules one
directory at a time, or one rule at a time.
Most of the changes in this PR were a result of running
`yarn lint:fix`. There were a handful of manual changes that seemed
obvious and simple to make. Anything beyond that and the rule was left
disabled.
The ESLint config has been updated to v8. The breaking changes are:
* The Prettier rule `quoteProps` has been changed from `consistent` to
`as-needed`, meaning that if one key requires quoting, only that key is
quoted rather than all keys.
* The ESLint rule `no-shadow` has been made more strict. It now
prevents globals from being shadowed as well.
Most of these changes were applied with `yarn lint:fix`. Only the
shadowing changes required manual fixing (shadowing variable names were
either replaced with destructuring or renamed).
The dependency `globalThis` was added to the list of dynamic
dependencies in the build system, where it should have been already.
This was causing `depcheck` to fail because the new lint rules required
removing the one place where `globalThis` had been erroneously imported
previously.
A rule requiring a newline between multiline blocks and expressions has
been disabled temporarily to make this PR smaller and to avoid
introducing conflicts with other PRs.
# Permission System 2.0
## Background
This PR migrates the extension permission system to [the new `PermissionController`](https://github.com/MetaMask/snaps-skunkworks/tree/main/packages/controllers/src/permissions).
The original permission system, based on [`rpc-cap`](https://github.com/MetaMask/rpc-cap), introduced [`ZCAP-LD`](https://w3c-ccg.github.io/zcap-ld/)-like permissions to our JSON-RPC stack.
We used it to [implement](https://github.com/MetaMask/metamask-extension/pull/7004) what we called "LoginPerSite" in [version 7.7.0](https://github.com/MetaMask/metamask-extension/releases/tag/v7.7.0) of the extension, which enabled the user to choose which accounts, if any, should be exposed to each dapp.
While that was a worthwhile feature in and of itself, we wanted a permission _system_ in order to enable everything we are going to with Snaps.
Unfortunately, the original permission system was difficult to use, and necessitated the creation of the original `PermissionsController` (note the "s"), which was more or less a wrapper for `rpc-cap`.
With this PR, we shake off the yoke of the original permission system, in favor of the modular, self-contained, ergonomic, and more mature permission system 2.0.
Note that [the `PermissionController` readme](https://github.com/MetaMask/snaps-skunkworks/tree/main/packages/controllers/src/permissions/README.md) explains how the new permission system works.
The `PermissionController` and `SubjectMetadataController` are currently shipped via `@metamask/snap-controllers`. This is a temporary state of affairs, and we'll move them to `@metamask/controllers` once they've landed in prod.
## Changes in Detail
First, the changes in this PR are not as big as they seem. Roughly half of the additions in this PR are fixtures in the test for the new migration (number 68), and a significant portion of the remaining ~2500 lines are due to find-and-replace changes in other test fixtures and UI files.
- The extension `PermissionsController` has been deleted, and completely replaced with the new `PermissionController` from [`@metamask/snap-controllers`](https://www.npmjs.com/package/@metamask/snap-controllers).
- The original `PermissionsController` "domain metadata" functionality is now managed by the new `SubjectMetadataController`, also from [`@metamask/snap-controllers`](https://www.npmjs.com/package/@metamask/snap-controllers).
- The permission activity and history log controller has been renamed `PermissionLogController` and has its own top-level state key, but is otherwise functionally equivalent to the existing implementation.
- Migration number 68 has been added to account for the new state changes.
- The tests in `app/scripts/controllers/permissions` have been migrated from `mocha` to `jest`.
Reviewers should focus their attention on the following files:
- `app/scripts/`
- `metamask-controller.js`
- This is where most of the integration work for the new `PermissionController` occurs.
Some functions that were internal to the original controller were moved here.
- `controllers/permissions/`
- `selectors.js`
- These selectors are for `ControllerMessenger` selector subscriptions. The actual subscriptions occur in `metamask-controller.js`. See the `ControllerMessenger` implementation for details.
- `specifications.js`
- The caveat and permission specifications are required by the new `PermissionController`, and are used to specify the `eth_accounts` permission and its JSON-RPC method implementation.
See the `PermissionController` readme for details.
- `migrations/068.js`
- The new state should be cross-referenced with the controllers that manage it.
The accompanying tests should also be thoroughly reviewed.
Some files may appear new but have just moved and/or been renamed:
- `app/scripts/lib/rpc-method-middleware/handlers/request-accounts.js`
- This was previously implemented in `controllers/permissions/permissionsMethodMiddleware.js`.
- `test/mocks/permissions.js`
- A truncated version of `test/mocks/permission-controller.js`.
Co-authored-by: Mark Stacey <markjstacey@gmail.com>
This PR adds one LavaMoat background script policy or each build type. It also renames the build system policy directory from `node` to `build-system` to make its purpose more clear. Each build type has the original `policy-override.json` for `main` builds. The `.prettierignore` file has been updated to match the locations of the new auto-generated policy files.
We need to maintain separate policies for each build type because each type will produce different bundles with different internal and external modules.
Co-authored-by: Mark Stacey <markjstacey@gmail.com>
This PR enables the exclusion of JavaScript and JSON source by `buildType`, and enables the running of `eslint` under LavaMoat. 80-90% of the changes in this PR are `.patch` files and LavaMoat policy additions.
The file exclusion is designed to work in conjunction with our code fencing. If you forget to fence an import statement of an excluded file, the application will now error on boot. **This PR commits us to a particular naming convention for files intended only for certain builds.** Continue reading for details.
### Code Fencing and ESLint
When a file is modified by the code fencing transform, we run ESLint on it to ensure that we fail early for syntax-related issues. This PR adds the first code fences that will be actually be removed in production builds. As a consequence, this was also the first time we attempted to run ESLint under LavaMoat. Making that work required a lot of manual labor because of ESLint's use of dynamic imports, but the manual changes necessary were ultimately quite minor.
### File Exclusion
For all builds, any file in `app/`, `shared/` or `ui/` in a sub-directory matching `**/${otherBuildType}/**` (where `otherBuildType` is any build type except `main`) will be added to the list of excluded files, regardless of its file extension. For example, if we want to add one or more pages to the UI settings in Flask, we'd create the folder `ui/pages/settings/flask`, add any necessary files or sub-folders there, and fence the import statements for anything in that folder. If we wanted the same thing for Beta, we would name the directory `ui/pages/settings/beta`.
As it happens, we already organize some of our source files in this way, namely the logo JSON for Beta and Flask builds. See `ui/helpers/utils/build-types.js` to see how this works in practice.
Because the list of ignored filed is only passed to `browserify.exclude()`, any files not bundled by `browserify` will be ignored. For our purposes, this is mostly relevant for `.scss`. Since we don't have anything like code fencing for SCSS, we'll have to consider how to handle our styles separately.
On an M1 Mac, when running `yarn start`, CPU can spike to 100% CPU, and
sometimes a bunch of `mdworker` instances will get spawned. This seems
to be caused by the file-watching mechanism used in dev to automatically
regenerate the build when something is changed. More specifically, we
are using an older version of `watchify`, which uses an older version of
`chokidar`, which is the package that actually does the watching. v4.0.0
of `watchify` upgrades `chokidar` to v3.x ([1]), which comes with
"massive CPU & RAM consumption improvements" ([2]). After the upgrade,
CPU usage decreases to 20-40%.
[1]: https://github.com/browserify/watchify/blob/master/CHANGELOG.md#400
[2]: https://github.com/paulmillr/chokidar/releases/tag/3.0.0
The main `version` field in `package.json` will now include the beta
version (if present) rather than it being passed in via the CLI when
building. The `version` field is now a fully SemVer-compatible version,
with the added restriction that any prerelease portion of the version
must match the format `<build type>.<build version>`.
This brings the build in-line with the future release process we will
be using for the beta version. The plan is for each future release to
enter a "beta phase" where the version would get updated to reflect
that it's a beta, and we would increment this beta version over time as
we update the beta. The manifest gives us a place to store this beta
version. It was also important to replace the automatic minor bump
logic that was being used previously, because the version in beta might
not be a minor bump.
Additionally, the filename logic used for beta builds was updated to
be generic across all build types rather than beta-specific. This will
be useful for Flask builds in the future.
* lavamoat - add lavamoat to webapp background
* test:e2e - add delay to resolve failure
* test:e2e - add delay to resolve failure
* build - add a switch for applying lavamoat, currently off for all
* test/e2e - remove delays added for lavamoat
* Revert "test/e2e - remove delays added for lavamoat"
This reverts commit 79c3479f15c072ed362ba1d4f1af41ea11a17d63.
The warnings about use of the unsafe Buffer constructor have been
addressed by package updates and patches.
The updates were:
* `gulp-sourcemaps` was updated from v2 to v3, and was patched to
replace remaining uses of the `Buffer` constructor
* Upstream PR: https://github.com/gulp-sourcemaps/gulp-sourcemaps/pull/388
* The transitive dependency `yazl` was updated from v2.4.3 to v2.5.1
in the lockfile.
* The abandoned packages `combine-source-map` and `inline-source-map`
were patched.
kumavis
Mark Stacey
Frederik Bolding
Alex Donesky
Shane
Elliot Winkler
Erik Marks
VSaric
Dan J Miller
Daniel
dragana8
ryanml
Erik Marks