96 Commits (683a96257e446c4901022d589b5a618ef9c9f9d8)

Author	SHA1	Message	Date
Daniel Savu	3bb9d0a767	fix(cosmos): validator checkpoint & agent indexing improvements (#3958) ... - propagates RPC errors to the indexing logic, so errors cause sleeps instead of overwhelming the RPCs. This also greatly reduces cosmos log noisiness: from 9.5k logs per min, to 2.5k logs per min. - checks for published checkpoints in reverse, to prioritize recent messages among those from before the validator was spun up - fixes a validator bug where errors in checkpoint publishing would cause the entire checkpoint queue to be reiterated through, from scratch - Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3887 so we can test this out on osmosis ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
overallteach	42a6dbc3e6	chore: remove repetitive words (#3644) ... ### Description <!-- What's included in this PR? --> remove repetitive words ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> Signed-off-by: overallteach <cricis@foxmail.com>	6 months ago
Daniel Savu	826b4ae57f	feat: batching igp (#3870) ... ### Description Deducts IGP cost from senders of messages within a batch, based on the total gas used by the batch and the estimates for submitting messages individually. The formula is `gas_used_by_operation = gas_used_by_tx * (operation_estimated_gas / total_estimated_cost)` Note that the individual estimate have a buffer value added to them (currently 75k), which will slightly skew proportions, though by a negligible amount. For example, given these estimates in a batch: 150k, 160k, 180k, if we add 80k to each, we go from e.g. 0.312 for the first message to 0.319 -> 2.25% error which isn't so bad. The error can be larger if operations within a batch have very different estimates, but realistically within a 5% range based on back-of-the-napkin calculations. ### Drive-by changes - The batching feature introduced an bug whereby gas expenditure would only be deducted in the `confirm` step. Now this is done in the `submit` step to account for txs that revert - Sealevel e2e can now be disabled for faster iteration! I've not done the cleanest job, but even the fact that we have this will reduce the time to run e2e locally by more than half. To use it, set `SEALEVEL_ENABLED=false` when running `run-locally` - e2e uses a new utility called `get_matching_lines`, that can be used to count (and in the future _parse_) logs, to reconstruct the state of the relayer and have more expressive correctness checks. This is used to make sure that gas is deducted for all messages, including those in batches. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3709 ### Backward compatibility Yes ### Testing E2E, using `get_matching_lines`	6 months ago
Daniel Savu	942aba8e55	feat(evm): collaborative indexing through txid sharing (#3833) ... ### Description - improved indexing reliability by sharing channels between indexing tasks, such that they inform one another when they come across a hyperlane-related transaction - the receiving end will then get the receipt of that tx and filter it for events relevant to it - this adds some flexibility to indexing since we can stop relying on watermark cursor entirely: as long as we have one task using sequence indexing, all others can rely on being informed by their channels about what txs to index - this PR is only for the EVM chains, but can be easily extended ### Drive-by changes - Enforces gas payments for all chains in e2e (previously this only used to be the case for sealevel) - The channel size is configured per type, similarly to the indexing cursor, in `contract_sync/cursors/mod.rs` - Reduces log verbosity is some places, e.g. by manually implementing `Debug` to exclude fields that aren't relevant to debugging (maybe there's even a macro annotation to exclude, but I haven't checked) - Renames `MpmcReceiver` -> `BroadcastReceiver` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3267 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing e2e tested with the watermark cursor manually disabled	6 months ago
Daniel Savu	be43b0fd91	feat: use batching configs on all execution environments (#3723) ... ### Description While we only have a tx batching implementation available for EVM chains, we still parallelize the `prepare` and `confirm` steps in the serial submitter according to the `maxBatchSize` config item. To reap the benefits of this change, we need to make these configs universal, because they were only defined within `h_eth::ChainConnectionConf`. This PR adds slight duplication because it adds the batching config on each individual connection config, but it's far simpler than including it in the `ChainConf` struct. The latter would've required changing all the trait impls of `BuildableWithProvider` to include the batching config. ### Drive-by changes Renames `serial_submitter` to `op_submitter` since it's no longer serial. ### Related issues Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3722 ### Backward compatibility Yes ### Testing Manually - added the configs to cosmos e2e and checked the logs	7 months ago
Daniel Savu	f46cf7708e	Indexing Type Wrapper (#3610) ... ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3544. Wraps all types to be indexed in an `Indexed<T>` struct in the relayer. This allows for using sequece-aware cursors for data that doesn't contain a sequence but can have it inferred from metadata (as is the case for the IGP on sealevel). `Indexed<T>` stores the sequence as an `Option`, so the db can have additional sequence data stored in it when the sequence is defined - this is how `process_indexed_gas_payment` is implemented now. In addition, the IGP payments and Deliveries are configured to be indexed with a `SequenceAware` cursor on sealevel (see `cursors/mod.rs`). ### Drive-by changes Implements the custom `Encode` and `Decode` traits for `IndexingDecorator`, which can be generalized to `Option<T>` where `T: Encode + Decode`. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing E2E	7 months ago
Daniel Savu	5aeb83a2e5	Relayer Message Batching to EVM (#3656) ... ### Description Status: - [x] op-queue scaffolding - [x] hyperlane-ethereum integration with multicall - [x] new config items - [x] actually build batch in the op-queue submit step - [x] e2e tests with batching configured - [x] dynamic batch sizes (batch as many messages as the prepare step has ready) - [x] parallelize metadata building - [ ] proportional IGP deduction ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes #3628 - Fixes #3630 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	7 months ago
Daniel Savu	f4626eb4fe	Allow protocol-specific indexing (#3544) ... ### Description - Generalizes contract syncer hashmaps to store trait objects instead of concrete structs, since both `SequencedDataContractSync` and `WatermarkContractSync` implement `ContractSyncer` - The nice thing this enables is being able to instantiate a different type of syncer for the same contract, for different chain protocols (e.g. for IGP, we can use a sequenced syncer on sealevel and cosmwasm, and a watermark syncer on EVM). - The problem is that `ContractSyncer` is implemented on different trait bounds on `SequencedDataContractSync` and `WatermarkContractSync`. Most notably, the `SequencedDataContractSync` impl bounds the indexed type by the `Sequenced` trait. This means that if a type is to be indexed by more than one syncer strategy, it needs to implement the union of bounds of those strategies - **Blocker:** In the IGP, this means that `InterchainGasPayment` needs to implement `Sequenced` and we have no obvious way of getting that just from the type. Even if the sequence is derived from the type, it has to be a sensible value, since indexing reliability depends on it. - Similar to the point above, we now have `contract_syncs`, a single function that instantiates contract syncs for all chains. Since `contract_syncs` must be able to build a `ContractSyncer` trait obj for all syncer types, it bounds both `T` (the indexed type) and `D` (the log store type) with a union of bounds, so it may be too strict for some types in the future. - The scraper now uses forward-backward indexing instead of just forward indexing, because that simplified the refactor - there's now a convenience trait called `TryFromWithMetrics` that allows adding this bound: `SequenceIndexer<T>: TryFromWithMetrics<ChainConf>,`. This isn't really extendable if we use non-sequence indexers, but it's fine for now since this is used everywhere - replaces the macro-based contract syncer builder with generic fns Still todo: - add new `IndexingDecorator<T>` struct that can wrap `InterchainGasPayment`, so we can use sequence-aware indexing for the IGP - remove sequence logic from the watermark cursor ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
J M Rossy	adab1bec0c	Add code spellcheck CI job (#3467) ... ### Description Fix spelling mistakes and create codespell CI job. An extension of #3422 with the remaining mistakes fixed ### Backward compatibility Yes --------- Co-authored-by: miles <66052478+miles-six@users.noreply.github.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz>	8 months ago
Daniel Savu	e737d998ff	Cosmos block-by-block indexing (#3245) ... ### Description Implements cosmos block-by-block indexing, parallelizing where possible: the `block` and `block_results` calls, and the individual block queries within a range. Also reuses the existing `handle_txs` logic, to improve readability. ### Drive-by changes Adds `tokio::task::JoinError` to the `hyperlane-core` error enum ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3234 - Added some hacky retrying logic to `get_logs_in_block`, since we were seeing lots of `503` http errors. In https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 we should make sure to remove this and propagate the error instead, to retry via the cursor / indexer logic. I edited the description of 3264 to reflect this ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	9 months ago
Yorke Rhodes	0ff3e2b171	Deploy injective and inEVM (#3140) ... ### Description - Deploy core to injective and inEVM networks ### Drive-by changes - Update cw-hyperlane dependency ### Backward compatibility Yes ### Testing - Fork tests - e2e tests --------- Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	10 months ago
Daniel Savu	f2a0e92c7f	Cosmos grpc fallbackprovider (#3139) ... ### Description Implements grpc fallback provider logic for cosmos - initially tried implementing the fallback provider deprioritization logic at middleware level like in the EVM. The difference between ethers and cosmrs is that in the latter, middleware can only live at the transport layer (`tower` crate level). - based on this github [issue](https://github.com/hyperium/tonic/issues/733), that actually doesn't look possible, because the http::Request type isn't `Clone` so it can't be submitted to multiple providers - ended up implementing the fallback provider at the application layer, by keeping an array of grpc channels - There is now a `call` method in `hyperlane_core::FallbackProvider` which I'm actually really happy with. This method handles the fallbackprovider-specific logic by taking in an async closure, running it on each provider, and iterating providers if the closure call fails. In `grpc.rs` you can see how this is slightly verbose but I think it's quite manageable. The only part that bugs me is having to duplicate `Pin::from(Box::from(future))`, but that's need afaict because the regular closure returns an anonymous type - adds `grpcUrls` and `customGrpcUrls` config items - tests the cosmos fallback provider e2e ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes: https://github.com/hyperlane-xyz/issues/issues/998 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Trevor Porter	0cad4f7407	Support cosmos contract byte lengths other than 32 (#3147) ... ### Description fixes #3143 Initially went down a path that would use protocol-specific types for addresses as suggested by #3143. I had made a `HyperlaneConnectionConf` enum with protocol-specific variants whose values were `addresses: CoreContractAddresses<ProtocolSpecificAddressType>` and `connection: ProtocolSpecificConnectionConf`. This worked pretty well until I hit the ISM logic. Because hyperlane-core is where the Mailbox trait is defined, the return type of `recipient_ism` in the trait cannot involve protocol specific types. It can't be moved to hyperlane-base because hyperlane-base imports the chain crates, so we'd have a cyclic dependency. I experimented with moving away from H256 to something like a Vec<u8> or string, but this felt a bit weird. In the end we decided to keep H256s as the global representation for contract addresses for now, with the intent of eventually changing this, and to support the varying length situation in a cosmos config ### Drive-by changes - Added some cosmos specific agent configurations into the sdk - Moved to bech32_prefix in the agents for consistency with what the SDK's chain metadata already does - I guess no one's ran cargo test in a while so vectors/message.json got a new v3 message lol ### Related issues Fixes #3143 ### Backward compatibility Changes prefix to bech32_prefix in the agent config, and now requires `contractAddressBytes` ### Testing Tested merged with #3144 and all worked --------- Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com>	10 months ago
Daniel Savu	c2cf7be8f2	feat(cosmos): ism dry-run (#3077) ... ### Description Implements `dry_run_verify` for the aggregation ISM on cosmwasm. One remaining issue is that the estimated gas is hardcoded to `1`, because we're actually [just querying](37fea49429/contracts/isms/aggregate/src/lib.rs (L108)) via rpc rather than simulating a tx. The `verify` tx isn't marked as a contract [entrypoint](https://book.cosmwasm.com/basics/entry-points.html) so it can't be called from outside iiuc. (here's the [verify](37fea49429/contracts/isms/aggregate/src/lib.rs (L124)) fn for reference). Worth mentioning that the query interface for the aggregation ISM is named incorrectly - it should return fields called `threshold` and `modules`, but instead copies the response from the multisig ISM and returns `threshold` and `validators`. This can be particularly misleading because validators have 20-bytes long addresses, whereas modules (contracts) have 32-bytes long addresses. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/807 ### Backward compatibility yes ### Testing E2E. The ISM setup is `routing` -> `aggregation (1/1)` -> `multisig (1/1)`	11 months ago
Daniel Savu	d18f7ae8e1	Cosmos gas price config (#3042) ... ### Description Adds a cosmos-specific config item for setting the minimum gas price, in the format returned by the `cosmos.base.node.v1beta1.Service/Config` cosmos-sdk grpc endpoint. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/810 ### Backward compatibility No. Will break existing cosmos configs. ### Testing None yet, will test in e2e	12 months ago
Daniel Savu	8e44bc1b81	feat: sealevel merkle indexer + e2e (#3053) ... Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3048 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3046 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2754	12 months ago
Daniel Savu	77aa58c581	Relayer balance metrics (#2976) ... Done: - scaffolding for fetching custom agent metrics - abstractions for building a metrics fetcher for a given VM - querying cosmos balances; e2e tested. - querying evm balances; e2e tested. - **Note that as a result, evm addresses are now no longer zero-padded when printed in the logs. This may break existing log queries** - fixed a nasty bug on ubuntu where wasmd (osmosisd dependency, part of the grpc query flow) would panic when a block is specified via `x-cosmos-block-height`. The fix was to bump the version of osmosisd from `19.0.0` to `20.5.0`. **Note that when running e2e on Mac OS, the osmosis version in use is still 19.0.0**. That's because we need a fork that publishes a darwin target binary (currently pointing [here](https://github.com/hashableric/osmosis/releases/download/v19.0.0-mnts/osmosisd-19.0.0-mnts-darwin-arm64.tar.gz)) For follow up PR: - sealevel balance querying I'm open to all renaming suggestions, I just tried to speed through and didn't ponder names too much Relates to https://github.com/hyperlane-xyz/issues/issues/701 Closes https://github.com/hyperlane-xyz/issues/issues/702 (because the balance becomes available in the metrics endpoint for polling)	12 months ago
ByeongSu Hong	50aed86517	fix: change domain id of cosmos localnet to random (#2977) ... ### Description We, Mitosis team will be using 26657 as the chain id, so it's good to change now ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	1 year ago
Trevor Porter	ece2be507a	Cosmos / CosmWasm agents (#2865) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: hashableric <hashableric@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz>	1 year ago
Daniel Savu	b0870b1113	V3 E2E agent testing scenarios (#2815) ... ### Description - Adds E2E testing scenarios to the agents: zero, single and double merkle tree hook insertion - zero insertion messages are assumed to not be deliverable for now, because that would require additional changes to e2e logic (deploying more one `TestSendReceiver.sol` instance per destination network) - `TestSendReceiver.sol` is extended to dynamically specify the default hook to use - the kathy hardhat script is extended with two params: the required and default hooks. If these are not specified via CLI, they default to `protocolFee` and `aggregationHook` respectively and are still set in the mailbox, to ensure idempotency. - sealevel tests commented out for now ### Drive-by changes ### Related issues - Closes https://github.com/hyperlane-xyz/issues/issues/561 - Closes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2720 - Closes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2755 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz>	1 year ago
Yorke Rhodes	1d18549755	Merge main to v3 (#2812)	1 year ago
Daniel Savu	c6cea785ae	V3 agents feedback (#2780) ... ### Description Part of the remediations from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2746 --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Daniel Savu	af5bd88aa8	V3 agents rebase (#2746) ... ### Description It's your favourite PR coming right back... V3 agents! Closes https://github.com/hyperlane-xyz/issues/issues/561 Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2742 Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2681 for e2e testing This PR includes: - [x] Merkle tree hook indexer - [x] Merkle tree builder task - [x] Update submitter to trigger retries if no proof is available yet Slightly more detailed overview of the work here: https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2720#issuecomment-1724038643 <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: -f <kunalarora1729@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Guillaume Bouvignies <guillaumebouvignies@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: Guillaume Bouvignies <guillaume.bouvignies@kurtosistech.com>	1 year ago
Trevor Porter	3f3306c35e	Helloworld program and tooling (#2744) ... ### Description * HelloWorld program added to `rust/sealevel/programs/helloworld` * Some changes to the Sealevel tooling * Refactored the old warp-route-specific deploy management into a more generic framework for idempotently deploying and managing router apps in Sealevel * Added helloworld deploy tooling * Added ISM & owner checking to this router deployment tooling * Added `--require-tx-approval` to prevent txs from being called without first prompting * Added a bunch of new commands for creating txs with certain instructions that were needed along the way -- e.g. setting an ISM, deploying a specific multisig ISM (previously only one would be deployed as a part of `core deploy` * Added a command for configuring multisig ISMs, `multisig-ism-message-id configure`, that takes in a JSON file of multisig ISM configs and applies them onchain * A bit of cleanup / refactor - e.g. removed some old commands like `mailbox receive` * Added foreignDeployments into `RouterApp` * Because RouterApp takes in `contractsMap: HyperlaneContractsMap<Factories>,`, which require attached contracts, a new `readonly foreignDeployments: ChainMap<Address> = {},` is added to the constructor * These foreignDeployments are considered in the return value of `remoteChains(chainName: string)`, but not in `chains()` -- this means that `chains()` now concretely means "chains that can be deployed to / interacted with and that there is an entry in `contractsMap` for, and `remoteChains(chainName: string)` returns any and all remote chains, regardless of whether they can be deployed to / interacted with * Added complete ISM support to the HyperlaneRouterChecker * when checking the ISM, if there's not a match and the ISM is a config, then the ISM will be deployed * Also added RouterViolation, before it'd just throw if there was a violation * Updated the Helloworld, IGP, and core tooling to work when AEE deployments are also configured * Moved to Routing ISM -> Aggregation ISM -> Merkle / Message ID multisig setup Some things to note: * atm there are a few places that have a TODO saying to remove something after some multisig txs are executed, I plan to revisit these after we get some sigs * I've deployed the mainnet sealevel version of helloworld, but haven't been able to enroll it in the EVM chains yet. Waiting for some multisig activity here ### Drive-by changes ### Related issues #2502 ### Backward compatibility I believe it should all be backward compatible ### Testing Deployed, ran checkers, etc	1 year ago
Kunal Arora	f0a45bd6b1	Merkle Tree Hook Deployer changes (#2747) ... ### Description - Interceptor is the pair <hook, ism> - `MerkleTreeInterceptorDeployer` for deploying merkle root hook and the `StaticMerkleRootMultisigIsm` on chain ### Drive-by changes - None ### Related issues - addresses https://github.com/hyperlane-xyz/issues/issues/621 ### Backward compatibility Yes ### Testing Manual	1 year ago
Trevor Porter	62ab488db9	Sealevel IGP debugging, temporarily change Sealevel gas payment policy (#2758) ... ### Description - Some small changes to help with debugging sealevel IGP - Turns out the GasPaymentAccount data format onchain is different from the one in `main`. This causes IGP indexing to not work in the wild, but it works in e2e. Because we were planning to change the IGP anyways as we transfer ownership to Zebec, this isn't so bad - so temporarily just not enforcing ZBC warp route Solana -> Nautilus gas payments. In practice, we're still effectively enforcing gas payments because the warp route will pay for gas ### Drive-by changes - Also added the new BSC POSE addy ### Related issues n/a ### Backward compatibility yes ### Testing Deployed, poked around, used cmds	1 year ago
Daniel Savu	71e8988ccd	Sealevel igp indexing (#2585) ... ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2583 Indexes IGP payments related to the relayer's data pda address. Unless this address is specified in the config (`sealevel.relayer_account `), no filtering is applied and all IGP payments are stored in the local database. <!-- What's included in this PR? --> ### Drive-by changes - Sets `HYP_BASE_GASPAYMENTENFORCEMENT` in `run-locally` for the relayer, to test that it correctly indexes the IGP payment before submitting the message - A new config section (`sealevel`) is added to the relayer - The `MessageIndexer` trait is replaced with `SequenceIndexer<HyperlaneMessage>`, renaming `fetch_count_at_tip` to `sequence_at_tip`. `SequenceIndexer` is now common to both the message and igp indexers. - The `parse_addr` macro is modified so it can be reused when parsing the sealevel relayer address config too - `rust/utils/sealevel-test.bash` is included because I was using it to test locally, but I can remove it if the sealevel e2e test already does all the steps there @mattiecnvr - Performs a `try_into` conversion that can be removed once https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2610 is done ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2501 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? --> e2e tests but the pipeline is failing, likely fixed by https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2602 --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Daniel Savu	51b81b84fa	fix(sealevel): data encoding in `set_return_data` (#2635) ... ### Description Suffixes `set_return_data` bytes with a non-zero byte to prevent a truncation bug ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2589 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Mattie Conover	041cf92d57	Fix insertion errors for txn and block (#2629) ... ### Description This resolves the insertion conflict that is being caused by #2246 that introduced multiple concurrent tasks to scrape. This does not address the underlying concurrency issue I discovered, however. Please review the issue comments for more details. ### Drive-by changes - Cleans up logs a bit (This was useful during testing and I think is useful globally) - Removed an unused file ### Related issues Fixes hyperlane-xyz/issues#493 ### Backward compatibility Yes ### Testing Manual	1 year ago
Mattie Conover	a5e1815746	Rust updates (#2619) ... ### Description Let this one slip a while. Updates our version of rust and as many packages as did not conflict with solana. I think we are going to need to figure something better out going forward with solana since if we have a couple more envs like this, it is going to make updates very hard. ### Drive-by changes None ### Related issues General Maintenance ### Backward compatibility Yes ### Testing Manual & Unit Tests	1 year ago
Mattie Conover	19838b11c6	E2E test `pkill` anvil (#2607) ... ### Description Fixes a minor bug when testing locally where sometimes a failure in the e2e test will not correctly clean up anvil. This just runs `pkill -SIGKILL anvil` to clean it up if it was running. ### Drive-by changes None ### Related issues ### Backward compatibility Yes ### Testing Manual	1 year ago
Mattie Conover	9045f8362e	IGP Tooling (#2602) ... ### Description Adds support for deploying IGPs and initial setup. Also extends the E2E tests to support these changes. - Deploy an IGP - Configure gas oracle data on IGPs - Configure gas overhead amounts for overhead IGP - Adds new config files to support the above - Adds support for u128 in `serde_json` ### Drive-by changes - Minor code cleanup around unused build cmd args - Refactored transaction submission to prevent common errors. - Cleaned up arg passing in the core commands ### Related issues Fixes #2500 ### Backward compatibility Yes-ish, slightly changes some sealevel commands but mostly the same. See the e2e test if in doubt. ### Testing Manual Unit Tests --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Mattie Conover	9ed3a24902	Solana e2e tests (#2578) ... ### Description Adds support for testing solana with the e2e tests. This involves 1) Downloading the pre-built solana cli tools 2) Setting up the solana configuration 3) Downloading the pre-built solana programs we need (see https://github.com/hyperlane-xyz/solana-program-library/releases/) 4) The solana programs in the repo 5) Start the solana validator 6) Deploy the solana programs 7) Deploy a warp route 8) Initialize the multisig ism and validator announce 9) Initialize a transfer 10) Wait for the message to be delivered In addition these were "woven" in with the existing E2E test logic, so for instance we only run one new validator and the existing relayer was extended to support these additional chains. This will make it much easier later on to support testes between environments. ### Drive-by changes - Fix a bug in the relayer when trying to index the mailbox - Add support in the hyperlane sealevel CLI for custom solana config paths - Fix a linker error on linux causes by not specifying `no-entrypoint` - Cleaned up unnecessary `no-entrypoint` features in libraries - Minor refactor to how we run commands to avoid arg passing - Created an easy way to define tasks that run asyncly `as_task` - Split up main logic flow into a couple files for easier reading - Use mold linker to speed up builds (well, the linking part) - Removed support for `log_all` to simplify code pathways - Added context when a child process ends unexpectedly for easier debugging - Fixed a bug in the validator where it would infinitely retry to send an announcement on failure without waiting - Cleaned up solana configs - Fixed processes hanging on exit (very annoying when testing locally since you have to manually go kill them) - Added stderr logging to the hyperlane sealevel CLI subprocess calls ### Related issues Fixes #2415 ### Backward compatibility Yes ### Testing Manual	1 year ago
Trevor Porter	8c5983933b	Sealevel (#2404) ... ### Description Opening for that sweet sweet agent image ### Drive-by changes _Are there any minor or drive-by changes also included?_ ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ Yes No _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None Yes ### Testing _What kind of testing have these changes undergone?_ None Manual Unit Tests --------- Co-authored-by: Steven Sloboda <steven@eclipse.builders> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Asa Oines <asaoines@gmail.com> Co-authored-by: Sergei Patrikeev <serejke.best@gmail.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: yorhodes <yorkerhodesiv@gmail.com> Co-authored-by: Yorke Rhodes <yorke@useabacus.network> Co-authored-by: Yorke Rhodes <email@yorke.dev> Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local> Co-authored-by: Kunal Arora <kuarora@ucsd.edu> Co-authored-by: Anett <44020788+anettrolikova@users.noreply.github.com> Co-authored-by: Yaqub Mahmoud <yaqub320@gmail.com> Co-authored-by: Alex <alex@alexbh.dev>	1 year ago
Mattie Conover	c1c86c7953	E2E Sealevel Prep (#2551) ... ### Description This PR hardens the e2e test pipeline to failure cases we have seen frequently and also makes the code more generic to better support future environments such as sealevel. The original ticket describes making one or more traits to help with this but after much experimentation and consideration, I feel that doing so would make the code unnecessarily complex and less flexible to future changes. My solution instead after talking with @tkporter regarding the sealvel e2e test pathway is to: 1) get utility code out of the way to make it more clear what the execution path is 2) simplify the process of running build tasks in the background so that as we add more envs we can have multiple things running concurrently without difficulty 3) standardize our program execution pattern to reduce the understanding overhead of the main execution path The main thing I kept running into every time I tried to create traits is that we ultimately need a lot of env and context which means either arg passing or even more structures to hold them or we statically code it into that module. The last option was the cleanest but at that point there isn't much difference between creating a trait and just defining a function to call. And even then there is still some live data we define as we go in main that is bulky to pass around but it might become worth it depending. Also the amount of support code for traits kept trying to ballon because we would need something that constructs the "components", and then runs each of the stages, but it also would need to still know what order to run things in, possibly even within each of those steps. ### Drive-by changes - Now watches agents to detect early termination - Standardizes log prefixes to distinguish between e2e logs and sub process logs - Now logs what commands are run - Standardized the program args concept to reduce param passing and better support reusable configurations - Has all build commands run asynchronously by default to prevent `spawn` commands in main - Fixed an issue where some rust components were being built twice, once as debug and again as release ### Related issues - Fixes hyperlane-xyz/issues#501 - Should help debug recent flakiness if not resolve it ### Backward compatibility Yes ### Testing Manual	1 year ago
Yorke Rhodes	638b3f5474	Unify lint configs	1 year ago
Yorke Rhodes	eb17a9af15	Block validator submitter on nonzero mailbox count (#2341) ... ### Description Ensures all checkpoint submitters wait for nonzero mailbox count ### Drive-by Changes Notably always spawns backfill submitter after seeing nonzero mailbox count ### Related issues Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2340 ### Backward compatibility Yes ### Testing Updated E2E Tests	2 years ago
Mattie Conover	efad4e1326	Omniscient relayer infra (#2299) ... ### Description - Adds support for omniscient relayers - Updates config formats to allow better overideability by agent role - Enables deploying selected agents rather than having to deploy them all at once - Merges scraper scripts and configuration ### Drive-by changes Lots of cleanup was made in the infrastructure scripts and source as it was already being touched by the above. While this PR touches a lot of files, many of the changes are just because we have so many scripts and a small change to something they all use instantly can lead to many files changed. ### Related issues - Fixes hyperlane-xyz/issues#438 - Fixes hyperlane-xyz/issues#447 - Fixes #1664 ### Backward compatibility _Are these changes backward compatible?_ No _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Asa Oines	90191d43d0	Fix bug in MessageSyncCursor rewind logic (#2297) ... ### Description This PR fixes a bug in the MessageSyncCursor rewind logic that causes the cursor to get stuck. The cursor would look for the message with nonce `n`, find it, and move its `next_block` to the block `b_n` at which `n` was dispatched. It would then index starting at `b_n`, find `n` again, interpret that as a "discontinuity", and rewind back to `b_n`, getting itself stuck permanently. We *should* be moving to `b_n` and not e.g. `b_n + 1` because we have no guarantee that `n+1` was not also present in `b_n` but just dropped by a flaky `eth_getLogs` implementation. The fix is to only consider messages with nonce > `n` when looking for discontinuities when our target nonce is `n + 1`. ### Drive-by changes - Modifies the E2E tests to use a chunk size of 1, allowing us to reproduce this failure mode without the filtering fix. - Debug log on rewind - Comments to clarify fast forward logic	2 years ago
Yorke Rhodes	50f04db1fa	Make merkle proofs optional on multisig ISM (#2173) ... ### Description Validators currently sign `(root, index)` checkpoints and during verification, a `message` is passed as calldata, an `id()` is derived, and a `proof` of `id()` at `index` in `root` is verified This provides “all or nothing” censorship resistance guarantees because a validator can only sign roots to allow any contained messages to be processed. We have considered alternatives where validators sign `message` directly and we lose censorship resistance in exchange for eliminating merkle proof verification gas costs. However, if validators sign `(root, index, message)` tuples, we can skip merkle proof verification on the destination chain while still maintaining censorship resistance by providing two valid metadata formats: 1. existing validator signatures and merkle proof verification of inclusion 2. including merkle proof verification for pathway where validators are censoring `message` It’s worth noting the validator is required to index event data to produce this new signature format. However, this does not require historical indexing and new validators being spun up can simply begin indexing from tip. See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2187 for validator changes See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2248 for relayer and e2e test changes ### Drive-by changes Merkle index also optional ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2192 ### Backward compatibility - new ISM deployment is necessary (we could upgrade implementation in theory) - Validator and relayer upgrades ### Testing Unit (fuzz) Tests, E2E tests	2 years ago
Asa Oines	8aa7b62bea	Minor logging improvements to agents (#2271) ... ### Description Removes some unnecessary clutter, adds domain to some spans ### Drive-by changes None	2 years ago
Asa Oines	ec63ed19c5	Fix CI flakiness (#2269) ... ### Description - Don't panic if E2E test fails to fetch a metric - Run coverage commands sequentially ### Drive-by changes - Cleans up some lint errors in forge tests	2 years ago
Asa Oines	63562c7211	Refactor agent event indexing (#2246) ... ### Description This PR refactors event indexing in the agents, allowing similar logic to be shared across multiple event types (i.e. messages, deliveries, gas payments) and database types (i.e. the Relayer rocks DB and Scraper SQL DB). Furthermore, it adds new syncing modes by way of `MessageSyncCursors` that take advantage of the monotonically increasing dispatched message nonce to sync more intelligently. ### Drive-by changes - Fixes a bug in the existing cursor that caused the same block range to be indexed three times - Modifies kathy to get rid of the idea of "rounds", just sends messages with a sleep in between - Minor modifications to the e2e test for performance - Expand macros in settings - Add scraper to e2e test ### Opportunities for improvement - We can further reduce RPC usage (or improve latency) by sharing the view of the latest finalized block number between cursors - We can speed up the effective time for (a relayer to start deliving messages | the scraper to scraper recent events) by creating forward/backward cursors for gas payments and deliveries where the backwards cursor terminates at index_settings.from - We can remove the need for index_settings.from by terminating backwards cursors based on the block number that the first message was dispatched at ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E tests --------- Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Mattie Conover	5e38cdf31a	Omniscient Relayer (#2195) ... ### Description Adds support for running a relayer which delivers messages from multiple origin chains instead of just one origin chain. ### Drive-by changes _Are there any minor or drive-by changes also included?_ - The E2E pipeline got a major re-work. It both runs faster and also tests the new changes. - SerialSubmitter has been abstracted to support additional operation types such as oracle updates - SerialSubmitter has been updated to actually confirm a transaction was included and is safe from re-orgs - SerialSubmitter has been updated to have higher throughput by pipelining operations ### Related issues - Blocked by #2148 - Fixes #1899 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Asa Oines	f6ad1e9a95	Validator automatically self-announces (#2177) ... ### Description This PR adds the ability for validators to self-announce themselves, so long as the validator key has a large enough balance to pay for the tx. This required setting the retry provider to skip retrying reverting transactions, since with a quorum provider and a very speedy blockchain (like hardhat) the transaction is broadcast multiple times and the later transactions conflict with the first one. ### Drive-by changes None ### Related issues - Fixes #2120 ### Backward compatibility _Are these changes backward compatible?_ Yes --------- Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Mattie Conover	0cc4600931	Update Agents (#2159) ... ### Description Update rust toolchain and packages. ### Drive-by changes No ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Mattie Conover	ed17ddc32f	Agent Argument Parsing (#2132) ... ### Description All configs can now be set by arguments as well as env and config files. The arguments will take the highest priority and overwrite the others. We should also update the docs to include this new support when it gets released, but since it is an entirely new feature it also can be a stealth release which we document after the fact. ### Drive-by changes ### Related issues - Fixes #2034 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Asa Oines	2900737c8f	Add HyperlaneIsmFactoryDeployer (#2068) ... ### Description This PR separates ISM deployment from HyperlaneCoreDeployer. Furthermore, it adds support for Routing and Aggregation ISM deployment via a recursive IsmConfig type. The MultisigIsm address is removed from the artifacts. Instead, the default ISM will be fetched at run time and compared against a provided IsmConfig. If there are *any* differences, a new defaultIsm will be deployed. ### Drive-by changes None ### Related issues None ### Backward compatibility _Are these changes backward compatible?_ No _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes, the default LegacyMultisigIsms will no longer be supported by the checker. The governance script will instead deploy a routing ISM and set that as the default ISM. ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Mattie Conover	1722f4ee7e	Better Agent Configuration Parsing (#2070) ... ### Description This is a significant change to how we parse user configs. The main goal of this PR is to give a clear path and env key to any errors that are encountered when parsing configuration. In order to accomplish this, I have crated a new trait `FromRawConf` which defines a set of steps to validate and convert individual fields. Also for every major config struct, there is now a `Raw*` variant of it which is used by the `config` library to reduce errors that occur during deserialization itself. `Raw*` types should always use the most basic form of a value, such as `String`, `StrOrInt`, `bool`, and ideally optional in all cases. Missing values should be handled during the raw conversion and not by `config`. I also needed to make changes to a number of types stored in the parsed config types to force validation forward to the parsing step instead of doing it when we read the config value. This also required me to create a filter to prevent trying to validate certain configs that we would not ever need to load. These changes can also be built on later to support something other than `config` if we choose to, or add support for merging configs from multiple sources since everything is optional. ### Drive-by changes - Default to `http` for connection type - Default to no gas payment enforcement if not specified instead of failing - `ChainSetup` -> `ChainConf` - `GasPaymentEnforcementConfig` -> `GasPaymentEnforcementConf` - Made ethereum connection configuration more forgiving - Moved hyperlane base settings from `mod.rs` to `base.rs` - Moved config chain tests in hyperlane core to `tests` dir to fix a cyclical import problem - Extension traits to help with config in hyperlane core - Moved `StrOrInt` to new hyperlane core `config` module - Support for parsing `U256` from `StrOrInt` - Removed `HexString` type which is now redundant - Updated base settings to use hyperlane domain - Use `heyKey` as signer type if `type` is not specified and `key` is - Moved hyperlane ethereum chain config to a new module ### Related issues - Fixes #2033 - Fixes #2012 ### Backward compatibility _Are these changes backward compatible?_ Yes - This should take in configs in the same shape but be more forgiving in a few places _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Test	2 years ago
Mattie Conover	7d6e5d5281	Rust Fuel Updates (#2029) ... ### Description This brings fuel up to the latest version. The main change required was to to abigen process. After some digging I figured out they just moved it to a new crate. ### Drive-by changes - Updated an error message I ran into when trying to run the validator to be more descriptive - Found an unlinked file which I was able to delete - Add `*.ignore` files to gitignore. This allows me to block out some configs I was using for testing. ### Related issues - Fixes hyperlane-xyz/issues#410 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual - Ran the validator against a local node	2 years ago