hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Daniel Savu	27aabf238c	feat: Relayer tokio task instrumentation (#3760 ) ### Description - Started off adding tokio-metrics but then realised those are quite general, so while we do have instrumentation it's not exposed in our metrics endpoint - switched to adding [tokio-console](https://github.com/tokio-rs/console/tree/main), which does give insight into the lifetime of specific tasks, so we can check which ones take up a long time during relayer startup. These are only visible at the `dependencyTrace` log level, so don't affect performance in the `hyperlane` context. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Helps debug https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3454 and any future performance issues - Does half the work for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3239 (still need to expose these in the metrics endpoint and import the grafana template) ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	1c4cf44c91	fix: allow all log levels in release build (#3824 ) ### Description Found the reason why we never had `trace` level logs in release builds - the `tracing` crate had a feature flag enabled that would wipe out all trace calls at compile time :( ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3823 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	5aeb83a2e5	Relayer Message Batching to EVM (#3656 ) ### Description Status: - [x] op-queue scaffolding - [x] hyperlane-ethereum integration with multicall - [x] new config items - [x] actually build batch in the op-queue submit step - [x] e2e tests with batching configured - [x] dynamic batch sizes (batch as many messages as the prepare step has ready) - [x] parallelize metadata building - [ ] proportional IGP deduction ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes #3628 - Fixes #3630 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	7 months ago
Daniel Savu	6e39cf01af	Relayer retry endpoints (#3476 ) ### Description Adds a `/message_retry` relayer endpoint, which supports retrying OpQueue operations either by ID or by destination domain. Such a request will be sent to an `mpmc` channel's receiving end in the OpQueue, which is emptied when `OpQueue::pop` is called. Example calls: ``` GET http://127.0.0.1:60843/message_retry?destination_domain=42 GET http://127.0.0.1:60843/message_retry?message_id=0x46910b1329ee53c86a023b322e9ca1c17e5f9f0bee789c77b0abced0a173d714 ``` If the endpoint is called specifying both filters, two requests will be sent across the channel, one for each condition. Efficiency note: The entire queue is iterated over if there's at least one retry request in the channel. The good thing is that rebuilding the heap is `O(n)` so not too bad. ### Drive-by changes - Removed the usage of `enum_dispatch`, because it would have made unit tests messy. The drawback is that we have to work around object safety restrictions in `PendingOperation` - for instance it can't be cloned (but can be stored behind an `Arc` if we ever need this). - Added new methods to `PendingOperation`, either to match by ID, or to implement heap element prioritization without depending on the concrete `PendingMessage` type (as was done before). Additions: `id()`, `origin_domain()`, `priority()`. Don't think this conflicts with any operations we may add in the future (e.g. gas oracle updates). - Although `tokio`'s broadcast channel is multi-producer-multi-consumer by default, the consumer end isn't `Clone` - you instead need to have a producer to call `.subscribe()` on to get a new consumer, so I added an `MpmcChannel` struct to encapsulate keeping a producer around to get new consumers. - OpQueue is moved into its own file and has unit tests added for retries (covering broadcast to multi-queue, and both retry types). - Did a bit of agent `server `cleanup, including `EigenNodeApi` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3098 ### Backward compatibility Yes ### Testing Unit tests for the server route (channel transmitter logic) and for the OpQueue retries (channel receiver logic).	8 months ago
Daniel Savu	e737d998ff	Cosmos block-by-block indexing (#3245 ) ### Description Implements cosmos block-by-block indexing, parallelizing where possible: the `block` and `block_results` calls, and the individual block queries within a range. Also reuses the existing `handle_txs` logic, to improve readability. ### Drive-by changes Adds `tokio::task::JoinError` to the `hyperlane-core` error enum ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3234 - Added some hacky retrying logic to `get_logs_in_block`, since we were seeing lots of `503` http errors. In https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 we should make sure to remove this and propagate the error instead, to retry via the cursor / indexer logic. I edited the description of 3264 to reflect this ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	9 months ago
Trevor Porter	9a83eb4d57	Sequence-aware indexing refactor (#3262 ) ### Description Part 1 of an indexing refactor. This mostly focuses on the sequence-aware cursors. Changes: - Some renames, e.g. `SequenceIndexer` -> `SequenceAwareIndexer`, `sequence_and_tip` -> `latest_sequence_count_and_tip` - Moved cursors out of `contract_sync/cursor.rs` into `contract_sync/cursors/*` - The goal was to make the the Forward sequence-aware cursor, the Backward sequence-aware, and the ForwardBackward sequence aware cursors more resilient, easier to understand, and better tested. - General strategy for doing this was to: - Only move onto a new range to index if the logs passed into the cursor's `update` function justify this - If updated with any weird data, like gaps in logs, rewind the cursor and retry previously queried ranges I'd recommend starting the review in the following order: 1. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/forward.rs` - where `ForwardSequenceAwareSyncCursor` & tests live 2. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/backward.rs` - where `BackwardSequenceAwareSyncCursor` & tests live 3. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/mod.rs` - where `ForwardBackwardSequenceAwareSyncCursor` lives ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Kunal Arora	13fb461888	feat:add eigen node endpoints to agent API (#3095 ) ### Description - compatibility with eigenlayer's node spec here: https://eigen.nethermind.io/docs/spec/api/ - changing from hyper to axum as a preferred choice of server handling - adding server trait to validator, relayer, and server (to customize behavior for each in the future) ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - fixes https://github.com/hyperlane-xyz/issues/issues/853 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Ivan Temchenko	65e2ea50f9	Google Cloud Storage CheckpointSyncer implementation (#3156 ) ### Description GSC builder and implementer classes; Documentation comments; Examples; Anonymous test on public bucket access; ### Drive-by changes `cargo doc` build fixes: * replaced square braces of non-existing items with `'` ; * raw URL surrounded by `<>`; ### Related issues Fixes #2242 Part 1 ### Backward compatibility Yes ### Testing Unit Tests	10 months ago
Yorke Rhodes	0ff3e2b171	Deploy injective and inEVM (#3140 ) ### Description - Deploy core to injective and inEVM networks ### Drive-by changes - Update cw-hyperlane dependency ### Backward compatibility Yes ### Testing - Fork tests - e2e tests --------- Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	10 months ago
Daniel Savu	f2a0e92c7f	Cosmos grpc fallbackprovider (#3139 ) ### Description Implements grpc fallback provider logic for cosmos - initially tried implementing the fallback provider deprioritization logic at middleware level like in the EVM. The difference between ethers and cosmrs is that in the latter, middleware can only live at the transport layer (`tower` crate level). - based on this github [issue](https://github.com/hyperium/tonic/issues/733), that actually doesn't look possible, because the http::Request type isn't `Clone` so it can't be submitted to multiple providers - ended up implementing the fallback provider at the application layer, by keeping an array of grpc channels - There is now a `call` method in `hyperlane_core::FallbackProvider` which I'm actually really happy with. This method handles the fallbackprovider-specific logic by taking in an async closure, running it on each provider, and iterating providers if the closure call fails. In `grpc.rs` you can see how this is slightly verbose but I think it's quite manageable. The only part that bugs me is having to duplicate `Pin::from(Box::from(future))`, but that's need afaict because the regular closure returns an anonymous type - adds `grpcUrls` and `customGrpcUrls` config items - tests the cosmos fallback provider e2e ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes: https://github.com/hyperlane-xyz/issues/issues/998 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	c9ef41c109	Fallback Provider Refactor (#3127 ) ### Description - Generalizes the fallback provider. Now any provider type can be used in it, as long as it implements `Into<Box<dyn BlockNumberGetter>> + Clone` (aka it can be wrapped into a struct that reads the current block #, and it can be cloned). - moves the fallback provider implementation to hyperlane-core - had serious issues with getting versions to successfully resolve - next time I run into issues, strongly inclined to create a new crate with dependencies for all chains, but which are only depended on by the agents codebase (rather than the contract as well). Chatted with @tkporter about this. - now fallbackprovider needs to be wrapped in a newtype struct for chain-specific trait implementations - If `Deref` is implemented for the newtype struct, all the function calls will still work (although using Deref for newtypes is [debatable](https://users.rust-lang.org/t/understanding-the-perils-of-deref/47958/18)) - the `request` function required by the `JsonRpcClient` still has some reusable logic but I left it there until it's clearer what can be reused by the cosmos fallback provider - the fallback provider unit tests still live in hyperlane-ethereum because of the logic in `require` ### Drive-by changes - needed to use `async-rwlock` instead of `tokio` for the `RwLock` structure used by the fallback provider for interior mutability. This is because the solana program requirements had rust pinned too far into the past for `tokio` to resolve to a version. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/892 ### Backward compatibility Yes ### Testing Unit tests that were already there for the fallback provider	10 months ago
Trevor Porter	5b4af6bf1d	Injective-specific account info protobuf (#3144 ) ### Description Only change needed was to correctly handle the account info. Because we may want to handle injective testnet, added `is_injective` fn to HyperlaneDomain instead of just doing the match directly. Happy to import some of the longer fully qualified names instead ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	d18f7ae8e1	Cosmos gas price config (#3042 ) ### Description Adds a cosmos-specific config item for setting the minimum gas price, in the format returned by the `cosmos.base.node.v1beta1.Service/Config` cosmos-sdk grpc endpoint. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/810 ### Backward compatibility No. Will break existing cosmos configs. ### Testing None yet, will test in e2e	11 months ago
Daniel Savu	8e44bc1b81	feat: sealevel merkle indexer + e2e (#3053 ) Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3048 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3046 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2754	11 months ago
Trevor Porter	ffd4de4110	Use new ethers-rs version that should work with BSC (#3000 ) ### Description Uses a version of our ethers fork with https://github.com/hyperlane-xyz/ethers-rs/pull/10, which adds back a change that I had reverted from https://github.com/hyperlane-xyz/ethers-rs/pull/8 This is required to fix bsc / bsctestnet txs, where the base fee is 0 and we want to estimate the priority fee ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility ye ### Testing deployed and saw successful bsc / bsctestnet msgs	1 year ago
Trevor Porter	c2288eb317	Update ethers-rs with polygon gas oracle fix (#2998 ) ### Description Update to latest ethers-rs (forked) version with https://github.com/hyperlane-xyz/ethers-rs/pull/9 ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility n/a ### Testing Planning to deploy with it, but the ethers-rs version is well tested	1 year ago
Trevor Porter	9b8ad55a6a	Update ethers-rs to use lower max priority fee (#2986 ) ### Description Cherry-picked the commit from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2903, which is now out of date ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility yes ### Testing we're running the relayer with an image off of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2903 Co-authored-by: -f <kunalarora1729@gmail.com>	1 year ago
Trevor Porter	ece2be507a	Cosmos / CosmWasm agents (#2865 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: hashableric <hashableric@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz>	1 year ago
Yorke Rhodes	1d18549755	Merge main to v3 (#2812 )	1 year ago
Daniel Savu	c6cea785ae	V3 agents feedback (#2780 ) ### Description Part of the remediations from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2746 --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Daniel Savu	af5bd88aa8	V3 agents rebase (#2746 ) ### Description It's your favourite PR coming right back... V3 agents! Closes https://github.com/hyperlane-xyz/issues/issues/561 Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2742 Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2681 for e2e testing This PR includes: - [x] Merkle tree hook indexer - [x] Merkle tree builder task - [x] Update submitter to trigger retries if no proof is available yet Slightly more detailed overview of the work here: https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2720#issuecomment-1724038643 <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: -f <kunalarora1729@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Guillaume Bouvignies <guillaumebouvignies@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: Guillaume Bouvignies <guillaume.bouvignies@kurtosistech.com>	1 year ago
Trevor Porter	3f3306c35e	Helloworld program and tooling (#2744 ) ### Description * HelloWorld program added to `rust/sealevel/programs/helloworld` * Some changes to the Sealevel tooling * Refactored the old warp-route-specific deploy management into a more generic framework for idempotently deploying and managing router apps in Sealevel * Added helloworld deploy tooling * Added ISM & owner checking to this router deployment tooling * Added `--require-tx-approval` to prevent txs from being called without first prompting * Added a bunch of new commands for creating txs with certain instructions that were needed along the way -- e.g. setting an ISM, deploying a specific multisig ISM (previously only one would be deployed as a part of `core deploy` * Added a command for configuring multisig ISMs, `multisig-ism-message-id configure`, that takes in a JSON file of multisig ISM configs and applies them onchain * A bit of cleanup / refactor - e.g. removed some old commands like `mailbox receive` * Added foreignDeployments into `RouterApp` * Because RouterApp takes in `contractsMap: HyperlaneContractsMap<Factories>,`, which require attached contracts, a new `readonly foreignDeployments: ChainMap<Address> = {},` is added to the constructor * These foreignDeployments are considered in the return value of `remoteChains(chainName: string)`, but not in `chains()` -- this means that `chains()` now concretely means "chains that can be deployed to / interacted with and that there is an entry in `contractsMap` for, and `remoteChains(chainName: string)` returns any and all remote chains, regardless of whether they can be deployed to / interacted with * Added complete ISM support to the HyperlaneRouterChecker * when checking the ISM, if there's not a match and the ISM is a config, then the ISM will be deployed * Also added RouterViolation, before it'd just throw if there was a violation * Updated the Helloworld, IGP, and core tooling to work when AEE deployments are also configured * Moved to Routing ISM -> Aggregation ISM -> Merkle / Message ID multisig setup Some things to note: * atm there are a few places that have a TODO saying to remove something after some multisig txs are executed, I plan to revisit these after we get some sigs * I've deployed the mainnet sealevel version of helloworld, but haven't been able to enroll it in the EVM chains yet. Waiting for some multisig activity here ### Drive-by changes ### Related issues #2502 ### Backward compatibility I believe it should all be backward compatible ### Testing Deployed, ran checkers, etc	1 year ago
Mattie Conover	741c9bb553	Update Rust (#2765 ) ### Description Regular rust version and package updates. ### Drive-by changes None ### Related issues ### Backward compatibility Yes ### Testing Unit Tests	1 year ago
Mattie Conover	7c989e7eaa	Agent Config Parsing Casing Fixes (#2666 ) ### Description This is a fix for an issue in the arguments and environment where we are unable to correctly support attribute names due to the casing. The main issue cannot be fully fixed practically by this PR because it would cause some configuration cases to break. For now it adds manual exceptions for the command line argument case and leaves envs as they were. There are then two new parsers that can be used with the new config format. ### Drive-by changes Also adds this to argument parser since it was trivial to copy/paste it and it will allow us to harden parsing when we switch over. ### Related issues - Fixes #2662 - Fixes #2663 - Progress on #2215 ### Backward compatibility Yes ### Testing Unit Tests	1 year ago
Mattie Conover	644db5eee1	Remove unused tracing logic (#2639 ) ### Description When I was working on #2215 I noticed that we are not using either zipkin or jaeger at all and possibly never have. Ripping these out to reduce dead code and also simplify our package dependencies. ### Drive-by changes None ### Related issues Discovered through #2215 ### Backward compatibility Yes - we were not using these features and did not document their existence ### Testing Manual - Verified that when running deploy-agents, the generated configs do not have jaeger or zipkin enabled on `main` and then tested to make sure they still work in this branch	1 year ago
Trevor Porter	c53c1f7059	Sealevel IGP connection client support, warp routes support gas payments (#2613 ) ### Description * Adds IGP configuration to the connection client * Adds gas router, which are used by warp routes to have configured gas amounts * Adds resizing of the HyperlaneToken account to warp routes (useful for remote routers & gas routers which can change size) * Adds IGP configuration to warp routes * Pays for gas in warp routes is an IGP is configured * Updates deploy tooling for warp routes to allow the IGP to be set, defaulting to the overhead IGP if one isn't configured ### Drive-by changes * Removed the CLI command `token init`, which we never really expect to be used directly anymore and is just dead weight. This instead happens in `warp-route deploy` behind the scenes * Moved IGP tests into their own crate to avoid a cyclical dependency situation between the IGP crate and test-utils. This is why `mailbox-test` exists too ### Related issues * Fixes #2494 ### Backward compatibility The storage layout of the warp route programs changes with this. So this is incompatible with existing sealevel testnet warp route deployments ### Testing e2e test, unit tests	1 year ago
Mattie Conover	a5e1815746	Rust updates (#2619 ) ### Description Let this one slip a while. Updates our version of rust and as many packages as did not conflict with solana. I think we are going to need to figure something better out going forward with solana since if we have a couple more envs like this, it is going to make updates very hard. ### Drive-by changes None ### Related issues General Maintenance ### Backward compatibility Yes ### Testing Manual & Unit Tests	1 year ago
Mattie Conover	9045f8362e	IGP Tooling (#2602 ) ### Description Adds support for deploying IGPs and initial setup. Also extends the E2E tests to support these changes. - Deploy an IGP - Configure gas oracle data on IGPs - Configure gas overhead amounts for overhead IGP - Adds new config files to support the above - Adds support for u128 in `serde_json` ### Drive-by changes - Minor code cleanup around unused build cmd args - Refactored transaction submission to prevent common errors. - Cleaned up arg passing in the core commands ### Related issues Fixes #2500 ### Backward compatibility Yes-ish, slightly changes some sealevel commands but mostly the same. See the e2e test if in doubt. ### Testing Manual Unit Tests --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Trevor Porter	21acccac3d	Sealevel IGP program (#2574 ) ### Description * Sealevel IGP program * Includes tests * Minor changes to some other Sealevel program tests to use some common test utils ### Drive-by changes n/a ### Related issues #2217 ### Backward compatibility Yes ### Testing Unit / functional	1 year ago
Trevor Porter	8c5983933b	Sealevel (#2404 ) ### Description Opening for that sweet sweet agent image ### Drive-by changes _Are there any minor or drive-by changes also included?_ ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ Yes No _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None Yes ### Testing _What kind of testing have these changes undergone?_ None Manual Unit Tests --------- Co-authored-by: Steven Sloboda <steven@eclipse.builders> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Asa Oines <asaoines@gmail.com> Co-authored-by: Sergei Patrikeev <serejke.best@gmail.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: yorhodes <yorkerhodesiv@gmail.com> Co-authored-by: Yorke Rhodes <yorke@useabacus.network> Co-authored-by: Yorke Rhodes <email@yorke.dev> Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local> Co-authored-by: Kunal Arora <kuarora@ucsd.edu> Co-authored-by: Anett <44020788+anettrolikova@users.noreply.github.com> Co-authored-by: Yaqub Mahmoud <yaqub320@gmail.com> Co-authored-by: Alex <alex@alexbh.dev>	1 year ago
Yorke Rhodes	2deb9b8973	Fix AWS KMS Signer connection issues (#2319 ) ### Description - Update ethers-rs fork - Create new KMS client instead of using OnceCell ### Backward compatibility Yes No ### Testing Running validator locally with replica of validator DB	2 years ago
Yorke Rhodes	50f04db1fa	Make merkle proofs optional on multisig ISM (#2173 ) ### Description Validators currently sign `(root, index)` checkpoints and during verification, a `message` is passed as calldata, an `id()` is derived, and a `proof` of `id()` at `index` in `root` is verified This provides “all or nothing” censorship resistance guarantees because a validator can only sign roots to allow any contained messages to be processed. We have considered alternatives where validators sign `message` directly and we lose censorship resistance in exchange for eliminating merkle proof verification gas costs. However, if validators sign `(root, index, message)` tuples, we can skip merkle proof verification on the destination chain while still maintaining censorship resistance by providing two valid metadata formats: 1. existing validator signatures and merkle proof verification of inclusion 2. including merkle proof verification for pathway where validators are censoring `message` It’s worth noting the validator is required to index event data to produce this new signature format. However, this does not require historical indexing and new validators being spun up can simply begin indexing from tip. See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2187 for validator changes See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2248 for relayer and e2e test changes ### Drive-by changes Merkle index also optional ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2192 ### Backward compatibility - new ISM deployment is necessary (we could upgrade implementation in theory) - Validator and relayer upgrades ### Testing Unit (fuzz) Tests, E2E tests	2 years ago
Mattie Conover	5e38cdf31a	Omniscient Relayer (#2195 ) ### Description Adds support for running a relayer which delivers messages from multiple origin chains instead of just one origin chain. ### Drive-by changes _Are there any minor or drive-by changes also included?_ - The E2E pipeline got a major re-work. It both runs faster and also tests the new changes. - SerialSubmitter has been abstracted to support additional operation types such as oracle updates - SerialSubmitter has been updated to actually confirm a transaction was included and is safe from re-orgs - SerialSubmitter has been updated to have higher throughput by pipelining operations ### Related issues - Blocked by #2148 - Fixes #1899 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	b56b022f88	Hyperlane-Core WASM support (#2110 ) ### Description This shuffles some code around and changes dependencies to make `hyperlane-core` WASM compatible and adds a check to the pipeline to ensure it stays that way. To do this it does refactor hyperlane db to the `base` crate, which makes more sense IMO anyway as that is an implementation detail of the agents and not part of the core protocol. ### Drive-by changes ### Related issues - Fixes #1977 ### Backward compatibility _Are these changes backward compatible?_ Yes-ish - It does change where things are but it does not change support. Should only impact people who directly use the libraries. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	1722f4ee7e	Better Agent Configuration Parsing (#2070 ) ### Description This is a significant change to how we parse user configs. The main goal of this PR is to give a clear path and env key to any errors that are encountered when parsing configuration. In order to accomplish this, I have crated a new trait `FromRawConf` which defines a set of steps to validate and convert individual fields. Also for every major config struct, there is now a `Raw` variant of it which is used by the `config` library to reduce errors that occur during deserialization itself. `Raw` types should always use the most basic form of a value, such as `String`, `StrOrInt`, `bool`, and ideally optional in all cases. Missing values should be handled during the raw conversion and not by `config`. I also needed to make changes to a number of types stored in the parsed config types to force validation forward to the parsing step instead of doing it when we read the config value. This also required me to create a filter to prevent trying to validate certain configs that we would not ever need to load. These changes can also be built on later to support something other than `config` if we choose to, or add support for merging configs from multiple sources since everything is optional. ### Drive-by changes - Default to `http` for connection type - Default to no gas payment enforcement if not specified instead of failing - `ChainSetup` -> `ChainConf` - `GasPaymentEnforcementConfig` -> `GasPaymentEnforcementConf` - Made ethereum connection configuration more forgiving - Moved hyperlane base settings from `mod.rs` to `base.rs` - Moved config chain tests in hyperlane core to `tests` dir to fix a cyclical import problem - Extension traits to help with config in hyperlane core - Moved `StrOrInt` to new hyperlane core `config` module - Support for parsing `U256` from `StrOrInt` - Removed `HexString` type which is now redundant - Updated base settings to use hyperlane domain - Use `heyKey` as signer type if `type` is not specified and `key` is - Moved hyperlane ethereum chain config to a new module ### Related issues - Fixes #2033 - Fixes #2012 ### Backward compatibility _Are these changes backward compatible?_ Yes - This should take in configs in the same shape but be more forgiving in a few places _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Test	2 years ago
Mattie Conover	7d6e5d5281	Rust Fuel Updates (#2029 ) ### Description This brings fuel up to the latest version. The main change required was to to abigen process. After some digging I figured out they just moved it to a new crate. ### Drive-by changes - Updated an error message I ran into when trying to run the validator to be more descriptive - Found an unlinked file which I was able to delete - Add `*.ignore` files to gitignore. This allows me to block out some configs I was using for testing. ### Related issues - Fixes hyperlane-xyz/issues#410 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual - Ran the validator against a local node	2 years ago
Asa Oines	f25feecfa7	Scaffolding for multiple ISM types in the relayer (#2028 ) ### Description This PR adds scaffolding in the relayer to support building metadata for multiple ISM types. To add a new ISM type: - Add a new implementation of MetadataBuilder for that type - Add the type to the SupportedIsmTypes enum - Add the type to the match statement in BaseMetadataBuilder ### Drive-by changes None ### Related issues - #1757 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E tests	2 years ago
Trevor Porter	5bf8aedbfc	Use legacy feature for ethers-contract (#2001 ) ### Description Was accidentally removed in https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/1975/files#diff-7bdd9f71c229927ffff28bd337aa4ab3ab960426cd17fe857afe922014dfe791L13 Using the legacy feature allows us to opt into EIP 1559 txs. We should clean this up at some point, this is so fragile Decided to put the legacy feature at the workspace lvl because we'll probably not want to miss this again by accident ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ None	2 years ago
Mattie Conover	4657fcc737	Better rust package management (#1975 ) ### Description At the suggestion of @tkporter this moves common dependencies and configuration to use the new(ish) cargo workspace configuration feature. This was added in July 2022 and allows us to ensure all versions and configurations are the same across different crates. It does not decrease the typing, but it does reduce the updates required when we go to update a package and keeps duplicates next to each other. This PR does not change any package versions or add/remove any packages. It only moves the configuration. ### Drive-by changes Also sorted packages by name. ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Mattie Conover	1c004f4b10	Remove gelato support (#1946 ) ### Description We have not been using Gelato anymore so thought it would be nice to remove this code and avoid continuing to maintain it. I know I have had to update it a few times to maintain parity. As an added perk it should slightly speed up build times since it removes an entire crate. ### Drive-by changes None ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes - As we do not use this code _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Mattie Conover	7353d605df	Fuel scaffolding (#1401 ) * Fuel scaffolding * Fix primitive type import * Remove unused dep	2 years ago
Asa Oines	f15632f26c	Finalize rename from abacus->hyperlane (#1312 ) * Rename abacus->hyperlane in agents * Rename in typescript * Rename abacus->hyperlane in dockerfile	2 years ago
Mattie Conover	b4d06f9639	Message Explorer Inbox Scraping (#1159 ) * Better hex processing * Cleanup clippy errors * Fix parsing of 160 bit values * Support multiple contracts with cursors * InboxIndexer for getting message processed events * Work on inbox indexing and delivered message linking * Use message hashes to link deliveries and also add hash indexes for common lookups * split up settings struct * Support scraper config for main fn * Multiple config parsing for scraper * Better docs and more reliant config loading * Refactoring building of contract items * Cut down on duplication * Support inbox indexer building * fmt * Scraper settings loading re-work * Fix errors, warnings, and cleanup * Fix clippy warnings * Fix hash index not supporting uniqueness * Fix after entity update * Remove special case * Better error messages for missing config values * Env prefix filtering * Improve error context for url without base error * Fix inbox domain bug * Actually I was wrong about domains, misinterpreted config * Fix loading settings method * Fix upsert error * Debugged and fixed incorrect txn linking with messages and deliveries * Cleanup * Update config for scraper * Update config for scraper * Delivered message linking; again. * Reduce call frequency of message linking query * Fixed typo * Fix init mistake * changed to agent_prefix * Domain settings * Use a loop * Add more docs * Remove semicolen * Increase linking frequency * Add moonbeam domain * Fmt * Renamed settings struct * Add docs * Add test for FROM_HEX_CHARS array	2 years ago
Mattie Conover	5b06d0280c	Scraper DB generation and schema (#991 ) * Cleanup cargo config * BaseAgent trait * Pull `run_all` fn out of BaseAgent into static fn * Add `run` to `BaseAgent` * Scraper crate init * Create tables * WIP * data init * Init script * Fix * Attempting to run migration * fix ordering * fix domain creation * fix block table creation * fix checkpoint table * fix table cursor * fix table transaction + cleanup * enum type * progress * gas payments * message * delivered message * message state * cleanup * cleanup warnings * undo * Unique by domain and height * update index names * Change to OriginTxId * Add timestamp to messages * comment	2 years ago
webbhorn	02f9589fdc	Drop rust-Kathy (#713 )	2 years ago
Mattie Conover	0644e4b22d	E2E testing workflow (#679 ) * Work on new e2e script * new e2e mimics old * Support CI mode * Kathy CI mode * remove old script * e2e workflow * Update readme * fix workflow * split build from run * change build target * Use lighter-weight http client * fix typo * better handling for stopping children * Fix url * Support multiple filters * cache node modules * cache yarn build * ci cleanup * Cleanup and updates * requested changes	2 years ago
webbhorn	dd63f70da1	Gelato Plumbing (#637 ) This PR prepares for the introduction of Gelato ForwardRequest API usage. However, it only refactors current code and introduces a prerequisite interface. In particular, the existing MessageProcessor was previously responsible both for message discovery via AbacusDB, as well as managing the submission of that message to the destination chain via some RPC provider. Since we want to have the option to deliver those messages over Gelato, a natural place to split things up might be between the AbacusDB message discovery phase, and the message delivery phase. This CL formalizes that notion by moving the submission logic into a SerialSubmitter submodule. The MessageProcessor and SerialSubmitter only communicate via a channel now. A subsequent PR will introduce GelatoSubmitter, which will route messages from the MessageProcessor to the Gelato API according to the new configuration options introduced in this PR.	2 years ago
Mattie Conover	1f7beb3b9c	Better Abigen (#633 ) * Abigen tools * Generate abis with buildscript * Generate abis with buildscript for middleware * Simplify code * update lock * Fix pipeline order, must run check first new to generate files	2 years ago
Mattie Conover	aa92873088	Oneline Backtraces (#627 ) * add backtrace lib * Pull over code for printing backtraces * Oneline backtraces * code shuffle * fmt * remove stale comment * cleanup * refactoring	2 years ago
webbhorn	018c73206b	Gelato: ForwardRequest and TaskStatus calls, and EIP-712 signing for ForwardRequests (#511 ) Goal of this PR is to add support for posting ForwardRequest messages to Gelato and obtaining the status of those requests post-submission-to-Gelato. The key abstractions introduced in this PR are: ForwardRequestCall (in fwd_req_call.rs): Logic for making a single, individual RPC to the Gelato endpoint responsible for accepting the initial forward request submission, which it then enqueues and handles asynchronously. fwd_req_sig.rs, which contains an impl of the ethers::types::transaction::eip712::Eip712 trait for ForwardRequestOpArgs. This allows an ethers::signers::Signer to sign our message before we submit it to the relevant Gelato endpoint. TaskStatusCall in task_status_call.rs: Analogous to ForwardRequestCall in that it encapsulates the logic and state required to make an individual call to a Gelato endpoint to get the status of a previously-submitted request. Chain in chains.rs: Represents chains that are relevant for Abacus and/or supported by Gelato. Handles conversions between the chain_id integer representation expected by Gelato and other types. Also provides the per-chain addresses we need when signing gelato requests for supported chains.	2 years ago

1 2

72 Commits (683a96257e446c4901022d589b5a618ef9c9f9d8)