79 Commits (707db4a2793004053fc6be963672db3f16f28e6b)

Author	SHA1	Message	Date
Daniel Savu	b643fba90a	feat: use multiple `mpsc` channels instead of `broadcast` for async interface (#4190) ... ### Description Follow up to https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4180 Lowers txid broadcast channel capacity from 1M to 30k and refactors it to use a vec of `mpsc` under the hood. ### Context The heap profiler showed that each new chain takes up 100MB of memory because of the txid broadcast channel. In addition to this, the tokio broadcast channel is a ring buffer where new items simply overwrite old items, making it hard to safely lower the capacity. As such, to be able to safely lower the capacity of this channel, usage of `tokio`'s `broadcast` channel is replaced with a `Vec` of `mpsc`s, which offer an async `send` interface for backpressure. So instead of overwriting old items, senders will block on sends, which makes sense in our case as we absolutely don't want to miss items. The submitter is slower than the indexer based on empirical observation, so blocking on channel size doesn't bottleneck throughput. I noticed the max number of channel items we can reach is 29k, by running the RC relayer after an 8 day break, during which around 160k messages were submitted. Lowering the capacity to 30k should cause us to only encounter backpressure in very rare cases, and the footprint of this channel would lower from 100MB to 3MB. ### Drive-by changes The delivery indexer in the scraper is no longer passed a txid receiver, since message deliveries are not expected to occur in the same tx as message dispatches. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Testing E2E, by adding invariants to make sure txid indexing works --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	4 months ago
Daniel Savu	6140e6f397	feat: Relayer heap profiling (#4180) ... ### Description During the scalability audit we've identified relayer memory usage as a cause of concern. This PR runs the profiler each time e2e is run and outputs a report called `dhat-heap.json` in the current dir. That file can be visualised with a dhat viewer, such as the one hosted by the author [here](https://nnethercote.github.io/dh_view/dh_view.html). It adds a conditionally compiled heap memory profiler ([dhat-rs](https://docs.rs/dhat/latest/dhat/)), which is only included when both the `memory-profiling` feature is enabled and the `memory-profiling` cargo profile is used. The report can order the profiling results by total lifetime allocated bytes, peak allocated bytes, total allocation count, etc. Below is the result of sorting by total lifetime allocated bytes, which reveals that the biggest memory hogs are the hook indexing channel [senders](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/hyperlane-base/src/contract_sync/mod.rs#L52) (one per origin chain), taking up 100+ MB. Its size can most likely be reduced from 1M to 1k.  ### Drive-by changes Adds the remaining IDs in the channel as a field to the hook indexing log, to know if it's safe to lower the channel size. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Backward compatibility Yes, since everything is conditionally compiled / isolated in a new cargo profile. ### Testing Manual	4 months ago
Daniel Savu	a5e5f70610	chore: Revert "feat(hyp-ethereum): gas escalator middleware (#3852)" (#4098) ... This reverts commit c3c002a8fe. ### Description The gas escalator took up to 28gb of memory, and even though it was only ever released on RC, it must have taken up too many cluster resources, ending up starving the `hyperlane` relayer. Reverting for now until we have a better idea ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Daniel Savu	8c8f39abfe	feat(relayer): ListOperations api endpoint (#4089) ... ### Description Adds a `list_operations` API endpoint to the relayer, which returns all operations in its prepare queue. Example output from `GET http://0.0.0.0:9092/list_operations?destination_domain=13371` ``` [ { "app_context": null, "message": { "body": [ 18, 52 ], "destination": 13371, "nonce": 1, "origin": 13373, "recipient": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "sender": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "version": 3 }, "num_retries": 13, "status": { "Retry": "CouldNotFetchMetadata" }, "submitted": false, "type": "PendingMessage" }, { "app_context": null, "message": { "body": [ 18, 52 ], "destination": 13371, "nonce": 2, "origin": 13372, "recipient": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "sender": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "version": 3 }, "num_retries": 13, "status": { "Retry": "CouldNotFetchMetadata" }, "submitted": false, "type": "PendingMessage" } ] ``` ### Drive-by changes Makes `PendingOperation` serializable, which means switching from JSON format to e.g. CSV (for easily filtering / aggregating in excel) should be very easy. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Manual and Unit Tests	5 months ago
Daniel Savu	c3c002a8fe	feat(hyp-ethereum): gas escalator middleware (#3852) ... ### Description Wraps evm providers in a gas escalator middleware as part of the `build_with_signer` step, so signed txs can have their gas price bumped if stuck in the mempool. Gas price is increased every 60s by 12.5%, and is capped at the `max_fee_per_gas` set in tx overrides. ### Drive-by changes - changes to our fork of `ethers`: - removes the `M: Clone` bound, since it's not actually required and conflicts with the middleware we use - increases the level of the `escalated` log in the middleware to `debug` (so we can see it working in prod without enabling `trace`) ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2959 ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	5 months ago
Daniel Savu	d213e32341	feat: relayer queue operation labels (#4003) ... ### Description - adds status labels to operations in the relayer queues. In practice, the most useful ones are those for when a message is retried due to an error. Labels are defined as enums to keep the cardinality finite, the size of prometheus metrics could get very large - whenever a message is pushed to a queue, the interface requires specifying the new status as a `Option<PendingOperationStatus>`. If `None`, the status is left as before (such as in the case when an operation isn't ready to be retried). - a limitation is that labels aren't stored in the DB, so messages are loaded with the `FirstPrepareAttempt` label. I just triggered retries for destinations I wanted to get labels for, but we could persist to the db in a follow-up PR - Example prep queue labels for inevm: <img width="621" alt="Screenshot 2024-06-21 at 18 05 58" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/23065004/f53b2858-371f-4b6b-86d9-5c21409d641f"> ### Drive-by changes - removes `make_op_try` as it doesn't really add anything that couldn't be done with regular Rust - bumps the `strum` version because there was a bug in the previous one, preventing inner enum fields from being converted to strings ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Yes ### Testing Manual	5 months ago
Trevor Porter	a109937f96	chore: some throughput tweaks (#4036) ... ### Description Context: https://discord.com/channels/935678348330434570/1254036693217054794 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Yorke Rhodes	49f41d9759	chore: Merge branch 'main' into cli-2.0 (#3863) ... Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Connor McEwen <connor.mcewen@gmail.com> Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Avi Atkin <103125634+avious00@users.noreply.github.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Ali Alaoui <aalaoui2001@gmail.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz> Co-authored-by: omahs <73983677+omahs@users.noreply.github.com>	6 months ago
Daniel Savu	27aabf238c	feat: Relayer tokio task instrumentation (#3760) ... ### Description - Started off adding tokio-metrics but then realised those are quite general, so while we do have instrumentation it's not exposed in our metrics endpoint - switched to adding [tokio-console](https://github.com/tokio-rs/console/tree/main), which does give insight into the lifetime of specific tasks, so we can check which ones take up a long time during relayer startup. These are only visible at the `dependencyTrace` log level, so don't affect performance in the `hyperlane` context. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Helps debug https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3454 and any future performance issues - Does half the work for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3239 (still need to expose these in the metrics endpoint and import the grafana template) ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	1c4cf44c91	fix: allow all log levels in release build (#3824) ... ### Description Found the reason why we never had `trace` level logs in release builds - the `tracing` crate had a feature flag enabled that would wipe out all trace calls at compile time :( ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3823 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	5aeb83a2e5	Relayer Message Batching to EVM (#3656) ... ### Description Status: - [x] op-queue scaffolding - [x] hyperlane-ethereum integration with multicall - [x] new config items - [x] actually build batch in the op-queue submit step - [x] e2e tests with batching configured - [x] dynamic batch sizes (batch as many messages as the prepare step has ready) - [x] parallelize metadata building - [ ] proportional IGP deduction ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes #3628 - Fixes #3630 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	7 months ago
Daniel Savu	6e39cf01af	Relayer retry endpoints (#3476) ... ### Description Adds a `/message_retry` relayer endpoint, which supports retrying OpQueue operations either by ID or by destination domain. Such a request will be sent to an `mpmc` channel's receiving end in the OpQueue, which is emptied when `OpQueue::pop` is called. Example calls: ``` GET http://127.0.0.1:60843/message_retry?destination_domain=42 GET http://127.0.0.1:60843/message_retry?message_id=0x46910b1329ee53c86a023b322e9ca1c17e5f9f0bee789c77b0abced0a173d714 ``` If the endpoint is called specifying both filters, **two** requests will be sent across the channel, one for each condition. Efficiency note: The entire queue is iterated over if there's at least one retry request in the channel. The good thing is that rebuilding the heap is `O(n)` so not too bad. ### Drive-by changes - Removed the usage of `enum_dispatch`, because it would have made unit tests messy. The drawback is that we have to work around object safety restrictions in `PendingOperation` - for instance it can't be cloned (but can be stored behind an `Arc` if we ever need this). - Added new methods to `PendingOperation`, either to match by ID, or to implement heap element prioritization without depending on the concrete `PendingMessage` type (as was done before). Additions: `id()`, `origin_domain()`, `priority()`. Don't think this conflicts with any operations we may add in the future (e.g. gas oracle updates). - Although `tokio`'s broadcast channel is multi-producer-multi-consumer by default, the consumer end isn't `Clone` - you instead need to have a producer to call `.subscribe()` on to get a new consumer, so I added an `MpmcChannel` struct to encapsulate keeping a producer around to get new consumers. - OpQueue is moved into its own file and has unit tests added for retries (covering broadcast to multi-queue, and both retry types). - Did a bit of agent `server `cleanup, including `EigenNodeApi` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3098 ### Backward compatibility Yes ### Testing Unit tests for the server route (channel transmitter logic) and for the OpQueue retries (channel receiver logic).	8 months ago
Daniel Savu	e737d998ff	Cosmos block-by-block indexing (#3245) ... ### Description Implements cosmos block-by-block indexing, parallelizing where possible: the `block` and `block_results` calls, and the individual block queries within a range. Also reuses the existing `handle_txs` logic, to improve readability. ### Drive-by changes Adds `tokio::task::JoinError` to the `hyperlane-core` error enum ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3234 - Added some hacky retrying logic to `get_logs_in_block`, since we were seeing lots of `503` http errors. In https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 we should make sure to remove this and propagate the error instead, to retry via the cursor / indexer logic. I edited the description of 3264 to reflect this ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	9 months ago
Trevor Porter	9a83eb4d57	Sequence-aware indexing refactor (#3262) ... ### Description Part 1 of an indexing refactor. This mostly focuses on the sequence-aware cursors. Changes: - Some renames, e.g. `SequenceIndexer` -> `SequenceAwareIndexer`, `sequence_and_tip` -> `latest_sequence_count_and_tip` - Moved cursors out of `contract_sync/cursor.rs` into `contract_sync/cursors/*` - The goal was to make the the Forward sequence-aware cursor, the Backward sequence-aware, and the ForwardBackward sequence aware cursors more resilient, easier to understand, and better tested. - General strategy for doing this was to: - Only move onto a new range to index if the logs passed into the cursor's `update` function justify this - If updated with any weird data, like gaps in logs, rewind the cursor and retry previously queried ranges I'd recommend starting the review in the following order: 1. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/forward.rs` - where `ForwardSequenceAwareSyncCursor` & tests live 2. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/backward.rs` - where `BackwardSequenceAwareSyncCursor` & tests live 3. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/mod.rs` - where `ForwardBackwardSequenceAwareSyncCursor` lives ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Kunal Arora	13fb461888	feat:add eigen node endpoints to agent API (#3095) ... ### Description - compatibility with eigenlayer's node spec here: https://eigen.nethermind.io/docs/spec/api/ - changing from hyper to axum as a preferred choice of server handling - adding server trait to validator, relayer, and server (to customize behavior for each in the future) ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - fixes https://github.com/hyperlane-xyz/issues/issues/853 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Ivan Temchenko	65e2ea50f9	Google Cloud Storage CheckpointSyncer implementation (#3156) ... ### Description GSC builder and implementer classes; Documentation comments; Examples; Anonymous test on public bucket access; ### Drive-by changes `cargo doc` build fixes: * replaced square braces of non-existing items with `'` ; * raw URL surrounded by `<>`; ### Related issues Fixes #2242 Part 1 ### Backward compatibility Yes ### Testing Unit Tests	10 months ago
Yorke Rhodes	0ff3e2b171	Deploy injective and inEVM (#3140) ... ### Description - Deploy core to injective and inEVM networks ### Drive-by changes - Update cw-hyperlane dependency ### Backward compatibility Yes ### Testing - Fork tests - e2e tests --------- Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	10 months ago
Daniel Savu	f2a0e92c7f	Cosmos grpc fallbackprovider (#3139) ... ### Description Implements grpc fallback provider logic for cosmos - initially tried implementing the fallback provider deprioritization logic at middleware level like in the EVM. The difference between ethers and cosmrs is that in the latter, middleware can only live at the transport layer (`tower` crate level). - based on this github [issue](https://github.com/hyperium/tonic/issues/733), that actually doesn't look possible, because the http::Request type isn't `Clone` so it can't be submitted to multiple providers - ended up implementing the fallback provider at the application layer, by keeping an array of grpc channels - There is now a `call` method in `hyperlane_core::FallbackProvider` which I'm actually really happy with. This method handles the fallbackprovider-specific logic by taking in an async closure, running it on each provider, and iterating providers if the closure call fails. In `grpc.rs` you can see how this is slightly verbose but I think it's quite manageable. The only part that bugs me is having to duplicate `Pin::from(Box::from(future))`, but that's need afaict because the regular closure returns an anonymous type - adds `grpcUrls` and `customGrpcUrls` config items - tests the cosmos fallback provider e2e ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes: https://github.com/hyperlane-xyz/issues/issues/998 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	c9ef41c109	Fallback Provider Refactor (#3127) ... ### Description - Generalizes the fallback provider. Now any provider type can be used in it, as long as it implements `Into<Box<dyn BlockNumberGetter>> + Clone` (aka it can be wrapped into a struct that reads the current block #, and it can be cloned). - moves the fallback provider implementation to hyperlane-core - had serious issues with getting versions to successfully resolve - next time I run into issues, strongly inclined to create a new crate with dependencies for all chains, but which are only depended on by the agents codebase (rather than the contract as well). Chatted with @tkporter about this. - now fallbackprovider needs to be wrapped in a newtype struct for chain-specific trait implementations - If `Deref` is implemented for the newtype struct, all the function calls will still work (although using Deref for newtypes is [debatable](https://users.rust-lang.org/t/understanding-the-perils-of-deref/47958/18)) - the `request` function required by the `JsonRpcClient` still has some reusable logic but I left it there until it's clearer what can be reused by the cosmos fallback provider - the fallback provider unit tests still live in hyperlane-ethereum because of the logic in `require` ### Drive-by changes - needed to use `async-rwlock` instead of `tokio` for the `RwLock` structure used by the fallback provider for interior mutability. This is because the solana program requirements had rust pinned too far into the past for `tokio` to resolve to a version. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/892 ### Backward compatibility Yes ### Testing Unit tests that were already there for the fallback provider	10 months ago
Trevor Porter	5b4af6bf1d	Injective-specific account info protobuf (#3144) ... ### Description Only change needed was to correctly handle the account info. Because we may want to handle injective testnet, added `is_injective` fn to HyperlaneDomain instead of just doing the match directly. Happy to import some of the longer fully qualified names instead ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	d18f7ae8e1	Cosmos gas price config (#3042) ... ### Description Adds a cosmos-specific config item for setting the minimum gas price, in the format returned by the `cosmos.base.node.v1beta1.Service/Config` cosmos-sdk grpc endpoint. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/810 ### Backward compatibility No. Will break existing cosmos configs. ### Testing None yet, will test in e2e	11 months ago
Daniel Savu	8e44bc1b81	feat: sealevel merkle indexer + e2e (#3053) ... Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3048 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3046 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2754	11 months ago
Trevor Porter	ffd4de4110	Use new ethers-rs version that should work with BSC (#3000) ... ### Description Uses a version of our ethers fork with https://github.com/hyperlane-xyz/ethers-rs/pull/10, which adds back a change that I had reverted from https://github.com/hyperlane-xyz/ethers-rs/pull/8 This is required to fix bsc / bsctestnet txs, where the base fee is 0 and we want to estimate the priority fee ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility ye ### Testing deployed and saw successful bsc / bsctestnet msgs	1 year ago
Trevor Porter	c2288eb317	Update ethers-rs with polygon gas oracle fix (#2998) ... ### Description Update to latest ethers-rs (forked) version with https://github.com/hyperlane-xyz/ethers-rs/pull/9 ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility n/a ### Testing Planning to deploy with it, but the ethers-rs version is well tested	1 year ago
Trevor Porter	9b8ad55a6a	Update ethers-rs to use lower max priority fee (#2986) ... ### Description Cherry-picked the commit from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2903, which is now out of date ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility yes ### Testing we're running the relayer with an image off of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2903 Co-authored-by: -f <kunalarora1729@gmail.com>	1 year ago
Trevor Porter	ece2be507a	Cosmos / CosmWasm agents (#2865) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: hashableric <hashableric@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz>	1 year ago
Yorke Rhodes	1d18549755	Merge main to v3 (#2812)	1 year ago
Daniel Savu	c6cea785ae	V3 agents feedback (#2780) ... ### Description Part of the remediations from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2746 --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Daniel Savu	af5bd88aa8	V3 agents rebase (#2746) ... ### Description It's your favourite PR coming right back... V3 agents! Closes https://github.com/hyperlane-xyz/issues/issues/561 Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2742 Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2681 for e2e testing This PR includes: - [x] Merkle tree hook indexer - [x] Merkle tree builder task - [x] Update submitter to trigger retries if no proof is available yet Slightly more detailed overview of the work here: https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2720#issuecomment-1724038643 <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: -f <kunalarora1729@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Guillaume Bouvignies <guillaumebouvignies@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: Guillaume Bouvignies <guillaume.bouvignies@kurtosistech.com>	1 year ago
Trevor Porter	3f3306c35e	Helloworld program and tooling (#2744) ... ### Description * HelloWorld program added to `rust/sealevel/programs/helloworld` * Some changes to the Sealevel tooling * Refactored the old warp-route-specific deploy management into a more generic framework for idempotently deploying and managing router apps in Sealevel * Added helloworld deploy tooling * Added ISM & owner checking to this router deployment tooling * Added `--require-tx-approval` to prevent txs from being called without first prompting * Added a bunch of new commands for creating txs with certain instructions that were needed along the way -- e.g. setting an ISM, deploying a specific multisig ISM (previously only one would be deployed as a part of `core deploy` * Added a command for configuring multisig ISMs, `multisig-ism-message-id configure`, that takes in a JSON file of multisig ISM configs and applies them onchain * A bit of cleanup / refactor - e.g. removed some old commands like `mailbox receive` * Added foreignDeployments into `RouterApp` * Because RouterApp takes in `contractsMap: HyperlaneContractsMap<Factories>,`, which require attached contracts, a new `readonly foreignDeployments: ChainMap<Address> = {},` is added to the constructor * These foreignDeployments are considered in the return value of `remoteChains(chainName: string)`, but not in `chains()` -- this means that `chains()` now concretely means "chains that can be deployed to / interacted with and that there is an entry in `contractsMap` for, and `remoteChains(chainName: string)` returns any and all remote chains, regardless of whether they can be deployed to / interacted with * Added complete ISM support to the HyperlaneRouterChecker * when checking the ISM, if there's not a match and the ISM is a config, then the ISM will be deployed * Also added RouterViolation, before it'd just throw if there was a violation * Updated the Helloworld, IGP, and core tooling to work when AEE deployments are also configured * Moved to Routing ISM -> Aggregation ISM -> Merkle / Message ID multisig setup Some things to note: * atm there are a few places that have a TODO saying to remove something after some multisig txs are executed, I plan to revisit these after we get some sigs * I've deployed the mainnet sealevel version of helloworld, but haven't been able to enroll it in the EVM chains yet. Waiting for some multisig activity here ### Drive-by changes ### Related issues #2502 ### Backward compatibility I believe it should all be backward compatible ### Testing Deployed, ran checkers, etc	1 year ago
Mattie Conover	741c9bb553	Update Rust (#2765) ... ### Description Regular rust version and package updates. ### Drive-by changes None ### Related issues ### Backward compatibility Yes ### Testing Unit Tests	1 year ago
Mattie Conover	7c989e7eaa	Agent Config Parsing Casing Fixes (#2666) ... ### Description This is a fix for an issue in the arguments and environment where we are unable to correctly support attribute names due to the casing. The main issue cannot be fully fixed practically by this PR because it would cause some configuration cases to break. For now it adds manual exceptions for the command line argument case and leaves envs as they were. There are then two new parsers that can be used with the new config format. ### Drive-by changes Also adds this to argument parser since it was trivial to copy/paste it and it will allow us to harden parsing when we switch over. ### Related issues - Fixes #2662 - Fixes #2663 - Progress on #2215 ### Backward compatibility Yes ### Testing Unit Tests	1 year ago
Mattie Conover	644db5eee1	Remove unused tracing logic (#2639) ... ### Description When I was working on #2215 I noticed that we are not using either zipkin or jaeger at all and possibly never have. Ripping these out to reduce dead code and also simplify our package dependencies. ### Drive-by changes None ### Related issues Discovered through #2215 ### Backward compatibility Yes - we were not using these features and did not document their existence ### Testing Manual - Verified that when running deploy-agents, the generated configs do not have jaeger or zipkin enabled on `main` and then tested to make sure they still work in this branch	1 year ago
Trevor Porter	c53c1f7059	Sealevel IGP connection client support, warp routes support gas payments (#2613) ... ### Description * Adds IGP configuration to the connection client * Adds gas router, which are used by warp routes to have configured gas amounts * Adds resizing of the HyperlaneToken account to warp routes (useful for remote routers & gas routers which can change size) * Adds IGP configuration to warp routes * Pays for gas in warp routes is an IGP is configured * Updates deploy tooling for warp routes to allow the IGP to be set, defaulting to the overhead IGP if one isn't configured ### Drive-by changes * Removed the CLI command `token init`, which we never really expect to be used directly anymore and is just dead weight. This instead happens in `warp-route deploy` behind the scenes * Moved IGP tests into their own crate to avoid a cyclical dependency situation between the IGP crate and test-utils. This is why `mailbox-test` exists too ### Related issues * Fixes #2494 ### Backward compatibility The storage layout of the warp route programs changes with this. So this is incompatible with existing sealevel testnet warp route deployments ### Testing e2e test, unit tests	1 year ago
Mattie Conover	a5e1815746	Rust updates (#2619) ... ### Description Let this one slip a while. Updates our version of rust and as many packages as did not conflict with solana. I think we are going to need to figure something better out going forward with solana since if we have a couple more envs like this, it is going to make updates very hard. ### Drive-by changes None ### Related issues General Maintenance ### Backward compatibility Yes ### Testing Manual & Unit Tests	1 year ago
Mattie Conover	9045f8362e	IGP Tooling (#2602) ... ### Description Adds support for deploying IGPs and initial setup. Also extends the E2E tests to support these changes. - Deploy an IGP - Configure gas oracle data on IGPs - Configure gas overhead amounts for overhead IGP - Adds new config files to support the above - Adds support for u128 in `serde_json` ### Drive-by changes - Minor code cleanup around unused build cmd args - Refactored transaction submission to prevent common errors. - Cleaned up arg passing in the core commands ### Related issues Fixes #2500 ### Backward compatibility Yes-ish, slightly changes some sealevel commands but mostly the same. See the e2e test if in doubt. ### Testing Manual Unit Tests --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Trevor Porter	21acccac3d	Sealevel IGP program (#2574) ... ### Description * Sealevel IGP program * Includes tests * Minor changes to some other Sealevel program tests to use some common test utils ### Drive-by changes n/a ### Related issues #2217 ### Backward compatibility Yes ### Testing Unit / functional	1 year ago
Trevor Porter	8c5983933b	Sealevel (#2404) ... ### Description Opening for that sweet sweet agent image ### Drive-by changes _Are there any minor or drive-by changes also included?_ ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ Yes No _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None Yes ### Testing _What kind of testing have these changes undergone?_ None Manual Unit Tests --------- Co-authored-by: Steven Sloboda <steven@eclipse.builders> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Asa Oines <asaoines@gmail.com> Co-authored-by: Sergei Patrikeev <serejke.best@gmail.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: yorhodes <yorkerhodesiv@gmail.com> Co-authored-by: Yorke Rhodes <yorke@useabacus.network> Co-authored-by: Yorke Rhodes <email@yorke.dev> Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local> Co-authored-by: Kunal Arora <kuarora@ucsd.edu> Co-authored-by: Anett <44020788+anettrolikova@users.noreply.github.com> Co-authored-by: Yaqub Mahmoud <yaqub320@gmail.com> Co-authored-by: Alex <alex@alexbh.dev>	1 year ago
Yorke Rhodes	2deb9b8973	Fix AWS KMS Signer connection issues (#2319) ... ### Description - Update ethers-rs fork - Create new KMS client instead of using OnceCell ### Backward compatibility Yes No ### Testing Running validator locally with replica of validator DB	2 years ago
Yorke Rhodes	50f04db1fa	Make merkle proofs optional on multisig ISM (#2173) ... ### Description Validators currently sign `(root, index)` checkpoints and during verification, a `message` is passed as calldata, an `id()` is derived, and a `proof` of `id()` at `index` in `root` is verified This provides “all or nothing” censorship resistance guarantees because a validator can only sign roots to allow any contained messages to be processed. We have considered alternatives where validators sign `message` directly and we lose censorship resistance in exchange for eliminating merkle proof verification gas costs. However, if validators sign `(root, index, message)` tuples, we can skip merkle proof verification on the destination chain while still maintaining censorship resistance by providing two valid metadata formats: 1. existing validator signatures and merkle proof verification of inclusion 2. including merkle proof verification for pathway where validators are censoring `message` It’s worth noting the validator is required to index event data to produce this new signature format. However, this does not require historical indexing and new validators being spun up can simply begin indexing from tip. See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2187 for validator changes See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2248 for relayer and e2e test changes ### Drive-by changes Merkle index also optional ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2192 ### Backward compatibility - new ISM deployment is necessary (we could upgrade implementation in theory) - Validator and relayer upgrades ### Testing Unit (fuzz) Tests, E2E tests	2 years ago
Mattie Conover	5e38cdf31a	Omniscient Relayer (#2195) ... ### Description Adds support for running a relayer which delivers messages from multiple origin chains instead of just one origin chain. ### Drive-by changes _Are there any minor or drive-by changes also included?_ - The E2E pipeline got a major re-work. It both runs faster and also tests the new changes. - SerialSubmitter has been abstracted to support additional operation types such as oracle updates - SerialSubmitter has been updated to actually confirm a transaction was included and is safe from re-orgs - SerialSubmitter has been updated to have higher throughput by pipelining operations ### Related issues - Blocked by #2148 - Fixes #1899 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	b56b022f88	Hyperlane-Core WASM support (#2110) ... ### Description This shuffles some code around and changes dependencies to make `hyperlane-core` WASM compatible and adds a check to the pipeline to ensure it stays that way. To do this it does refactor hyperlane db to the `base` crate, which makes more sense IMO anyway as that is an implementation detail of the agents and not part of the core protocol. ### Drive-by changes ### Related issues - Fixes #1977 ### Backward compatibility _Are these changes backward compatible?_ Yes-ish - It does change where things are but it does not change support. Should only impact people who directly use the libraries. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	1722f4ee7e	Better Agent Configuration Parsing (#2070) ... ### Description This is a significant change to how we parse user configs. The main goal of this PR is to give a clear path and env key to any errors that are encountered when parsing configuration. In order to accomplish this, I have crated a new trait `FromRawConf` which defines a set of steps to validate and convert individual fields. Also for every major config struct, there is now a `Raw*` variant of it which is used by the `config` library to reduce errors that occur during deserialization itself. `Raw*` types should always use the most basic form of a value, such as `String`, `StrOrInt`, `bool`, and ideally optional in all cases. Missing values should be handled during the raw conversion and not by `config`. I also needed to make changes to a number of types stored in the parsed config types to force validation forward to the parsing step instead of doing it when we read the config value. This also required me to create a filter to prevent trying to validate certain configs that we would not ever need to load. These changes can also be built on later to support something other than `config` if we choose to, or add support for merging configs from multiple sources since everything is optional. ### Drive-by changes - Default to `http` for connection type - Default to no gas payment enforcement if not specified instead of failing - `ChainSetup` -> `ChainConf` - `GasPaymentEnforcementConfig` -> `GasPaymentEnforcementConf` - Made ethereum connection configuration more forgiving - Moved hyperlane base settings from `mod.rs` to `base.rs` - Moved config chain tests in hyperlane core to `tests` dir to fix a cyclical import problem - Extension traits to help with config in hyperlane core - Moved `StrOrInt` to new hyperlane core `config` module - Support for parsing `U256` from `StrOrInt` - Removed `HexString` type which is now redundant - Updated base settings to use hyperlane domain - Use `heyKey` as signer type if `type` is not specified and `key` is - Moved hyperlane ethereum chain config to a new module ### Related issues - Fixes #2033 - Fixes #2012 ### Backward compatibility _Are these changes backward compatible?_ Yes - This should take in configs in the same shape but be more forgiving in a few places _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Test	2 years ago
Mattie Conover	7d6e5d5281	Rust Fuel Updates (#2029) ... ### Description This brings fuel up to the latest version. The main change required was to to abigen process. After some digging I figured out they just moved it to a new crate. ### Drive-by changes - Updated an error message I ran into when trying to run the validator to be more descriptive - Found an unlinked file which I was able to delete - Add `*.ignore` files to gitignore. This allows me to block out some configs I was using for testing. ### Related issues - Fixes hyperlane-xyz/issues#410 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual - Ran the validator against a local node	2 years ago
Asa Oines	f25feecfa7	Scaffolding for multiple ISM types in the relayer (#2028) ... ### Description This PR adds scaffolding in the relayer to support building metadata for multiple ISM types. To add a new ISM type: - Add a new implementation of MetadataBuilder for that type - Add the type to the SupportedIsmTypes enum - Add the type to the match statement in BaseMetadataBuilder ### Drive-by changes None ### Related issues - #1757 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E tests	2 years ago
Trevor Porter	5bf8aedbfc	Use legacy feature for ethers-contract (#2001) ... ### Description Was accidentally removed in https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/1975/files#diff-7bdd9f71c229927ffff28bd337aa4ab3ab960426cd17fe857afe922014dfe791L13 Using the legacy feature allows us to opt into EIP 1559 txs. We should clean this up at some point, this is so fragile Decided to put the legacy feature at the workspace lvl because we'll probably not want to miss this again by accident ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ None	2 years ago
Mattie Conover	4657fcc737	Better rust package management (#1975) ... ### Description At the suggestion of @tkporter this moves common dependencies and configuration to use the new(ish) cargo workspace configuration feature. This was added in July 2022 and allows us to ensure all versions and configurations are the same across different crates. It does not decrease the typing, but it does reduce the updates required when we go to update a package and keeps duplicates next to each other. This PR does not change any package versions or add/remove any packages. It only moves the configuration. ### Drive-by changes Also sorted packages by name. ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Mattie Conover	1c004f4b10	Remove gelato support (#1946) ... ### Description We have not been using Gelato anymore so thought it would be nice to remove this code and avoid continuing to maintain it. I know I have had to update it a few times to maintain parity. As an added perk it should slightly speed up build times since it removes an entire crate. ### Drive-by changes None ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes - As we do not use this code _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Mattie Conover	7353d605df	Fuel scaffolding (#1401) ... * Fuel scaffolding * Fix primitive type import * Remove unused dep	2 years ago
Asa Oines	f15632f26c	Finalize rename from abacus->hyperlane (#1312) ... * Rename abacus->hyperlane in agents * Rename in typescript * Rename abacus->hyperlane in dockerfile	2 years ago