hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Daniel Savu	cdbaf9ef02	fix: dedup eth logs (#3108 ) ### Description Deduplicates ethereum logs by collecting them into a HashSet before returning a Vec. I couldn't find a way to deduplicate using an ethers config, but since `fetch_logs` is only called once in hyperlane-base, I'm doing the deduplication there "globally". I also noted in doc comments that duplicates may be returned. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3070 ### Backward compatibility Yes ### Testing Will rely on e2e passing but haven't added unit tests for this scenario	11 months ago
Daniel Savu	d18f7ae8e1	Cosmos gas price config (#3042 ) ### Description Adds a cosmos-specific config item for setting the minimum gas price, in the format returned by the `cosmos.base.node.v1beta1.Service/Config` cosmos-sdk grpc endpoint. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/810 ### Backward compatibility No. Will break existing cosmos configs. ### Testing None yet, will test in e2e	12 months ago
Daniel Savu	8e44bc1b81	feat: sealevel merkle indexer + e2e (#3053 ) Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3048 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3046 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2754	12 months ago
Daniel Savu	77aa58c581	Relayer balance metrics (#2976 ) Done: - scaffolding for fetching custom agent metrics - abstractions for building a metrics fetcher for a given VM - querying cosmos balances; e2e tested. - querying evm balances; e2e tested. - Note that as a result, evm addresses are now no longer zero-padded when printed in the logs. This may break existing log queries - fixed a nasty bug on ubuntu where wasmd (osmosisd dependency, part of the grpc query flow) would panic when a block is specified via `x-cosmos-block-height`. The fix was to bump the version of osmosisd from `19.0.0` to `20.5.0`. Note that when running e2e on Mac OS, the osmosis version in use is still 19.0.0. That's because we need a fork that publishes a darwin target binary (currently pointing [here](https://github.com/hashableric/osmosis/releases/download/v19.0.0-mnts/osmosisd-19.0.0-mnts-darwin-arm64.tar.gz)) For follow up PR: - sealevel balance querying I'm open to all renaming suggestions, I just tried to speed through and didn't ponder names too much Relates to https://github.com/hyperlane-xyz/issues/issues/701 Closes https://github.com/hyperlane-xyz/issues/issues/702 (because the balance becomes available in the metrics endpoint for polling)	1 year ago
ByeongSu Hong	50aed86517	fix: change domain id of cosmos localnet to random (#2977 ) ### Description We, Mitosis team will be using 26657 as the chain id, so it's good to change now ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	1 year ago
Trevor Porter	ece2be507a	Cosmos / CosmWasm agents (#2865 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: hashableric <hashableric@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz>	1 year ago
Yorke Rhodes	3501557d38	Remove legacy multisig from agents (#2839 ) Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2837	1 year ago
Yorke Rhodes	1d18549755	Merge main to v3 (#2812 )	1 year ago
Kunal Arora	9168cca6d2	validator deploy tweaks (#2784 ) ### Description - fixes to deploy testnet4 for agents ### Drive-by changes - truncating labels to length 63 - adding `.relayer` to `agentConfig.contextChainNames` ### Related issues - for https://github.com/hyperlane-xyz/issues/issues/574 ### Backward compatibility Yes ### Testing Manual --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	1 year ago
Yorke Rhodes	401b32dddd	Put f64 in hyperlane-core behind float flag (#2788 )	1 year ago
Daniel Savu	c6cea785ae	V3 agents feedback (#2780 ) ### Description Part of the remediations from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2746 --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Kunal Arora	35c176fe90	Testnet4 deployment (#2781 ) fixes https://github.com/hyperlane-xyz/issues/issues/574	1 year ago
Daniel Savu	af5bd88aa8	V3 agents rebase (#2746 ) ### Description It's your favourite PR coming right back... V3 agents! Closes https://github.com/hyperlane-xyz/issues/issues/561 Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2742 Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2681 for e2e testing This PR includes: - [x] Merkle tree hook indexer - [x] Merkle tree builder task - [x] Update submitter to trigger retries if no proof is available yet Slightly more detailed overview of the work here: https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2720#issuecomment-1724038643 <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: -f <kunalarora1729@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Guillaume Bouvignies <guillaumebouvignies@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: Guillaume Bouvignies <guillaume.bouvignies@kurtosistech.com>	1 year ago
Trevor Porter	3f3306c35e	Helloworld program and tooling (#2744 ) ### Description * HelloWorld program added to `rust/sealevel/programs/helloworld` * Some changes to the Sealevel tooling * Refactored the old warp-route-specific deploy management into a more generic framework for idempotently deploying and managing router apps in Sealevel * Added helloworld deploy tooling * Added ISM & owner checking to this router deployment tooling * Added `--require-tx-approval` to prevent txs from being called without first prompting * Added a bunch of new commands for creating txs with certain instructions that were needed along the way -- e.g. setting an ISM, deploying a specific multisig ISM (previously only one would be deployed as a part of `core deploy` * Added a command for configuring multisig ISMs, `multisig-ism-message-id configure`, that takes in a JSON file of multisig ISM configs and applies them onchain * A bit of cleanup / refactor - e.g. removed some old commands like `mailbox receive` * Added foreignDeployments into `RouterApp` * Because RouterApp takes in `contractsMap: HyperlaneContractsMap<Factories>,`, which require attached contracts, a new `readonly foreignDeployments: ChainMap<Address> = {},` is added to the constructor * These foreignDeployments are considered in the return value of `remoteChains(chainName: string)`, but not in `chains()` -- this means that `chains()` now concretely means "chains that can be deployed to / interacted with and that there is an entry in `contractsMap` for, and `remoteChains(chainName: string)` returns any and all remote chains, regardless of whether they can be deployed to / interacted with * Added complete ISM support to the HyperlaneRouterChecker * when checking the ISM, if there's not a match and the ISM is a config, then the ISM will be deployed * Also added RouterViolation, before it'd just throw if there was a violation * Updated the Helloworld, IGP, and core tooling to work when AEE deployments are also configured * Moved to Routing ISM -> Aggregation ISM -> Merkle / Message ID multisig setup Some things to note: * atm there are a few places that have a TODO saying to remove something after some multisig txs are executed, I plan to revisit these after we get some sigs * I've deployed the mainnet sealevel version of helloworld, but haven't been able to enroll it in the EVM chains yet. Waiting for some multisig activity here ### Drive-by changes ### Related issues #2502 ### Backward compatibility I believe it should all be backward compatible ### Testing Deployed, ran checkers, etc	1 year ago
Mattie Conover	741c9bb553	Update Rust (#2765 ) ### Description Regular rust version and package updates. ### Drive-by changes None ### Related issues ### Backward compatibility Yes ### Testing Unit Tests	1 year ago
Daniel Savu	d65c4e7e0a	Eta calculator fixes (#2735 ) ## Bug 1 Closes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2723 The relayer panic is caused by an overflow, bc of dividing by ~`6.540888459481895e-211`. On my local, the effective rate of indexing starts at `0.61`. ``` {"timestamp":"2023-09-15T09:57:10.746276Z","level":"INFO","fields":{"message":"~~~ blocks_processed: 2508, tip_progression: 2042, elapsed: 757.10340475, old_rate: Some(0.6155037701275111), effective_rate: 0.6155037701275111"},"target":"hyperlane_base::contract_sync::eta_calculator","span":{"domain":"solanadevnet","label":"gas_payments","name":"ContractSync"},"spans":[{"domain":"solanadevnet","label":"gas_payments","name":"ContractSync"}]} ``` But then both the `blocks_processed` and the `tip_progression` are consistently zero, which makes the `new_rate` be zero (`eea423ad04/rust/hyperlane-base/src/contract_sync/eta_calculator.rs (L41)`), and over time it takes over the entire moving average window to make it almost zero, leading to an overflow. 15 mins after that initial log, the effective rate already became `0.00038`. The reason for blocks_processed and tip_progression consistently being zero after the first log is that `eta_calculator.calculate(from, tip)` is always called with the same from and tip although it expects to get the latest values. ### The fix the tip wasn't being set after the sequence_and_tip query here: `eea423ad04/rust/hyperlane-base/src/contract_sync/cursor.rs (L565)` And then the to and from are calculated based on it: `eea423ad04/rust/hyperlane-base/src/contract_sync/cursor.rs (L550)` So even though the sync_state internal variables were kept up-to-date, the min(...) would cause the issue. The first PR commit fixes this. ## Bug 2 There was another bug in the eta calculator, caused by it only using `next_block` to approximate synced state, which doesn't apply to sequence indexing. The way the eta calculator is called had to be changed to use abstract measures of sync progression (which could be blocks or sequences). The second PR commit fixes this, afaict.	1 year ago
Kunal Arora	f0e4f2b89b	Prevent invalid destination griefing for the relayer (#2703 ) ### Description - Adds destinationDomain to the `GasPayment` event - recording destination domain as `destination` while reading events for the relayer ### Drive-by changes - none ### Related issues - fixes https://github.com/hyperlane-xyz/issues/issues/475 ### Backward compatibility No, change in event emitted and relayer indexing ### Testing - Unit --------- Signed-off-by: -f <kunalarora1729@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz>	1 year ago
Daniel Savu	71e8988ccd	Sealevel igp indexing (#2585 ) ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2583 Indexes IGP payments related to the relayer's data pda address. Unless this address is specified in the config (`sealevel.relayer_account `), no filtering is applied and all IGP payments are stored in the local database. <!-- What's included in this PR? --> ### Drive-by changes - Sets `HYP_BASE_GASPAYMENTENFORCEMENT` in `run-locally` for the relayer, to test that it correctly indexes the IGP payment before submitting the message - A new config section (`sealevel`) is added to the relayer - The `MessageIndexer` trait is replaced with `SequenceIndexer<HyperlaneMessage>`, renaming `fetch_count_at_tip` to `sequence_at_tip`. `SequenceIndexer` is now common to both the message and igp indexers. - The `parse_addr` macro is modified so it can be reused when parsing the sealevel relayer address config too - `rust/utils/sealevel-test.bash` is included because I was using it to test locally, but I can remove it if the sealevel e2e test already does all the steps there @mattiecnvr - Performs a `try_into` conversion that can be removed once https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2610 is done ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2501 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? --> e2e tests but the pipeline is failing, likely fixed by https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2602 --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Mattie Conover	17a6e79ea5	Rust agent config parsing (#2687 ) ### Description This does two things: 1) It parses the agent-specific configs for the relayer, validator, and scraper. 2) It removes the rigid structures that had been in place for parsing that were called out earlier and instead parses the serde json values directly. This PR does not cut over to the new versions just yet. That can happen once we start integrating it into the testing and deployment pipelines. ### Drive-by changes - Derive more was added and used where it made sense - The old config macro was removed ### Related issues - Progress on #2215 ### Backward compatibility Yes ### Testing None	1 year ago
Mattie Conover	389029dafb	Agent Config Base Parsing (#2665 ) ### Description This is a big one for #2215... 1. Splits the parsing of configs from the config definitions to support both the old, new, and future parsers simultaneously. (We might want to rename it from parser to config format or something; not set on this name). 2. Implements the new config format without switching to it or enabling it. This means this PR does introduce dead code that is going to be used in the next PR. ### Drive-by changes - Deleted some unused (or single use) macros defined in hyperlane base - Updated the definition of the config parsing trait to simplify implementation - Added some more ergonomic methods of handling errors in config parsing - Removed any raw `unwrap()` I noticed since these cause difficult to debug errors in prod. - Added some additional validation to catch code mistakes in config path definitions ### Related issues - Progress on #2215 - Comes after #2658 ### Backward compatibility Yes ### Testing Manual (E2E only for now to make sure I did not break the old config format)	1 year ago
Daniel Savu	326eced987	Revert "refactor(IndexMode): remove `IndexRange`… (#2562 )" (#2655 ) This reverts commit `127294decc`, to include a cursor bugfix in main and remove an IGP indexing bug added by that PR. ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	1 year ago
Mattie Conover	8cbfd3cbc0	Support 64-byte txn ids (attempt 2) (#2612 ) ### Description Adds support to TxOutcome for 512-bit ids and to log meta to support Solana. Now that we have primitive types in our crate we were able to create a better solution than before which just stores the EVM 256bit hashes inside of the 512bit one with leading zeros rather than needing to over engineer an enum container for it. ### Drive-by changes - Removed some dead code - Improved primitive type support for 512bit types - Improved serialization support for 512bit types ### Related issues - Fixes #2252 - Replaces #2256 ### Backward compatibility _Are these changes backward compatible?_ Yes-ish, changes one of the internal database key strings. At standup we discussed the possibility of migrating the data and concluded that was a can of worms we did not want to open. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit tests	1 year ago
Mattie Conover	a5e1815746	Rust updates (#2619 ) ### Description Let this one slip a while. Updates our version of rust and as many packages as did not conflict with solana. I think we are going to need to figure something better out going forward with solana since if we have a couple more envs like this, it is going to make updates very hard. ### Drive-by changes None ### Related issues General Maintenance ### Backward compatibility Yes ### Testing Manual & Unit Tests	1 year ago
Mattie Conover	9045f8362e	IGP Tooling (#2602 ) ### Description Adds support for deploying IGPs and initial setup. Also extends the E2E tests to support these changes. - Deploy an IGP - Configure gas oracle data on IGPs - Configure gas overhead amounts for overhead IGP - Adds new config files to support the above - Adds support for u128 in `serde_json` ### Drive-by changes - Minor code cleanup around unused build cmd args - Refactored transaction submission to prevent common errors. - Cleaned up arg passing in the core commands ### Related issues Fixes #2500 ### Backward compatibility Yes-ish, slightly changes some sealevel commands but mostly the same. See the e2e test if in doubt. ### Testing Manual Unit Tests --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Daniel Savu	127294decc	refactor(IndexMode): remove `IndexRange`, use default index mode per env (#2562 ) ### Description Removes the `index_mode` config field, default to whichever mode is currently supported by our existing envs. Doing this allows for safely removing runtime checks to see if the correct index mode was configured. Additional work is required to support multiple indexing modes per env. Check this discord thread for the full discussion: https://discord.com/channels/935678348330434570/961711527092682783/1133347733411987568 ### Drive-by changes - `IndexRange` is removed entirely and replaced by `RangeInclusive<u32>` - Moves `is_arbitrum_nitro` into a bigger existing `impl HyperlaneDomain` block for clarity ### Related issues This is part 1 of https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2511 ### Backward compatibility Yes ### Testing Relies on existing e2e tests	1 year ago
Trevor Porter	8c5983933b	Sealevel (#2404 ) ### Description Opening for that sweet sweet agent image ### Drive-by changes _Are there any minor or drive-by changes also included?_ ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ Yes No _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None Yes ### Testing _What kind of testing have these changes undergone?_ None Manual Unit Tests --------- Co-authored-by: Steven Sloboda <steven@eclipse.builders> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Asa Oines <asaoines@gmail.com> Co-authored-by: Sergei Patrikeev <serejke.best@gmail.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: yorhodes <yorkerhodesiv@gmail.com> Co-authored-by: Yorke Rhodes <yorke@useabacus.network> Co-authored-by: Yorke Rhodes <email@yorke.dev> Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local> Co-authored-by: Kunal Arora <kuarora@ucsd.edu> Co-authored-by: Anett <44020788+anettrolikova@users.noreply.github.com> Co-authored-by: Yaqub Mahmoud <yaqub320@gmail.com> Co-authored-by: Alex <alex@alexbh.dev>	1 year ago
Alex	da25b06254	CCIP Read ISM (#2398 ) ### Description CCIP Read ISM implementation, - [x] Contracts - [x] Relayer - [x] Example HTTP backend ### Drive-by changes _Are there any minor or drive-by changes also included?_ ### Related issues - Fixes #[2211] ### Backward compatibility _Are these changes backward compatible?_ Yes, relayers will skip messages they can't handle. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes, relayers won't be able to submit messages that have a CCIP Read ISM. ### Testing _What kind of testing have these changes undergone?_ - [x] Manual - [x] Unit Tests ### General approach - [x] Get a basic [CCIP Read implementation working](https://github.com/AlexBHarley/permissionless-chainlink-feeds/tree/main/contracts) - [x] Build a [mock service that serves CCIP Read compatible data](https://github.com/AlexBHarley/permissionless-chainlink-feeds/tree/main/apps/api) - [x] Update the Relayer to support CCIP Read ISMs	1 year ago
Kunal Arora	d69a3f639a	No metadata relayer support for hooks (#2495 ) ### Description - Adding `NoMetadataRelayer` moduleType which returns an empty `vec![]` on `MetadataBuilder::build()` - Adding `HyperlaneHookDeployer` which lets you deploy the hook, ISM and a testRecipient which specifies to use the `AbtractNativeISM` ### Drive-by changes - none ### Related issues - Accompanies #2316 - Fixes https://github.com/hyperlane-xyz/issues/issues/379 for Optimism - Related to adding native hooks to Mailbox V3 https://github.com/hyperlane-xyz/issues/issues/486 ### Backward compatibility Backward compatible? Yes Tooling changes needed? Need to update node to ^18 ### Testing - Manual on testnet3	1 year ago
Daniel Savu	6ee5868353	fix(validator): announce directly if no gas estimate (#2454 ) ### Description It has been reported that `contract_call.tx.max_cost()` can fail / not exist on some networks. When that's the case, default to assuming no additional validator gas is required and just send the tx. If the tx fails, report the failure and suggest that gas may be insufficient in the logged error. ### Drive-by changes None ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2426 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ None --------- Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	1 year ago
Daniel Savu	7889385501	Relayer aggregation ism (#2411 ) ### Description Adds AggregationISM support to the relayer, by adding the scaffolding, sub-ism metadata fetching, and custom metadata encoding. ### Drive-by changes - adds `dry_run_verify(...)` to the `InterchainSecurityModule` trait and implements it for `EthereumInterchainSecurityModule` - changes the default ISM in the test environment from `routingIsm(multisigIsm)` to `routingIsm(aggregationIsm([multisigIsm, messageIdIsm], threshold = 1))` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/1757 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing The AggregationISM metadata encoding is tested using unit tests whose input was taken from the tests in `AggregationIsm.t.sol`, by printing the individual ISM metadatas and the final metadata. The overall AggregationISM is tested with the `run-locally` script.	1 year ago
Daniel Savu	fdc86f687c	Reconstruct relayer `OpQueue` from HyperlaneDB (#2382 ) ### Description Stores the `num_retries` field of `PendingMessage` at its own prefix in HyperlaneDB, so that `PendingMessage` can be reconstructed with the same exponential backoff in the `next_attempt_after` field and avoid making unnecesary RPC calls on restart. To use persisted data, `PendingMessage` is constructed with `from_persisted_retries` instead of the default constructor. If reading from the db fails, the new constructor just defaults to the regular constructor, so it's worth considering replacing the regular constructor with this one entirely. ### Drive-by changes A new db prefix is added (`PENDING_MESSAGE_RETRY_COUNT_FOR_MESSAGE_ID `), as well as related getter / setter methods. Using the new prefix helps avoid a migration. I do see a `processed_by_nonce` prefix, which could be migrated to store data related to `PendingMessage` in general so let me know what you think. Could be a future improvement to merge its prefix with the prefix added in this PR. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/1902 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ A processor is run on some dummy database entries. The `SerialSubmitter`'s end of the channel is used to ensure that the incoming `PendingMessage`s have the correct backoff in `_next_attempt_after`.	1 year ago
Asa Oines	aaddba776b	Fix latest indexed block gauge (#2387 ) ### Description This PR fixes a bug that affected the latest indexed block gauge for message_dispatch. This allows the gauge to be updated to the "correct" value whether or not a getLogs query actually takes place. ### Drive-by changes None	1 year ago
Yorke Rhodes	50f04db1fa	Make merkle proofs optional on multisig ISM (#2173 ) ### Description Validators currently sign `(root, index)` checkpoints and during verification, a `message` is passed as calldata, an `id()` is derived, and a `proof` of `id()` at `index` in `root` is verified This provides “all or nothing” censorship resistance guarantees because a validator can only sign roots to allow any contained messages to be processed. We have considered alternatives where validators sign `message` directly and we lose censorship resistance in exchange for eliminating merkle proof verification gas costs. However, if validators sign `(root, index, message)` tuples, we can skip merkle proof verification on the destination chain while still maintaining censorship resistance by providing two valid metadata formats: 1. existing validator signatures and merkle proof verification of inclusion 2. including merkle proof verification for pathway where validators are censoring `message` It’s worth noting the validator is required to index event data to produce this new signature format. However, this does not require historical indexing and new validators being spun up can simply begin indexing from tip. See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2187 for validator changes See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2248 for relayer and e2e test changes ### Drive-by changes Merkle index also optional ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2192 ### Backward compatibility - new ISM deployment is necessary (we could upgrade implementation in theory) - Validator and relayer upgrades ### Testing Unit (fuzz) Tests, E2E tests	2 years ago
Asa Oines	63562c7211	Refactor agent event indexing (#2246 ) ### Description This PR refactors event indexing in the agents, allowing similar logic to be shared across multiple event types (i.e. messages, deliveries, gas payments) and database types (i.e. the Relayer rocks DB and Scraper SQL DB). Furthermore, it adds new syncing modes by way of `MessageSyncCursors` that take advantage of the monotonically increasing dispatched message nonce to sync more intelligently. ### Drive-by changes - Fixes a bug in the existing cursor that caused the same block range to be indexed three times - Modifies kathy to get rid of the idea of "rounds", just sends messages with a sleep in between - Minor modifications to the e2e test for performance - Expand macros in settings - Add scraper to e2e test ### Opportunities for improvement - We can further reduce RPC usage (or improve latency) by sharing the view of the latest finalized block number between cursors - We can speed up the effective time for (a relayer to start deliving messages \| the scraper to scraper recent events) by creating forward/backward cursors for gas payments and deliveries where the backwards cursor terminates at index_settings.from - We can remove the need for index_settings.from by terminating backwards cursors based on the block number that the first message was dispatched at ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E tests --------- Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Mattie Conover	5e38cdf31a	Omniscient Relayer (#2195 ) ### Description Adds support for running a relayer which delivers messages from multiple origin chains instead of just one origin chain. ### Drive-by changes _Are there any minor or drive-by changes also included?_ - The E2E pipeline got a major re-work. It both runs faster and also tests the new changes. - SerialSubmitter has been abstracted to support additional operation types such as oracle updates - SerialSubmitter has been updated to actually confirm a transaction was included and is safe from re-orgs - SerialSubmitter has been updated to have higher throughput by pipelining operations ### Related issues - Blocked by #2148 - Fixes #1899 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Asa Oines	f6ad1e9a95	Validator automatically self-announces (#2177 ) ### Description This PR adds the ability for validators to self-announce themselves, so long as the validator key has a large enough balance to pay for the tx. This required setting the retry provider to skip retrying reverting transactions, since with a quorum provider and a very speedy blockchain (like hardhat) the transaction is broadcast multiple times and the later transactions conflict with the first one. ### Drive-by changes None ### Related issues - Fixes #2120 ### Backward compatibility _Are these changes backward compatible?_ Yes --------- Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Mattie Conover	24f9642ea5	Agent docs and config improvements (#2148 ) ### Description Bunch of minor tweaks and updates based on what we learned from EthTokyo. - Updated Rust readme and the root readme to align with the hyperlane docs - Defaults to a 5 sec validator interval now - Makes the metrics port default to `9090` and have a soft failure if it can't start - Now will automatically create the checkpointsyncer directory if it is missing - Hyperlane database now officially uses the domain name and id as a prefix internally to support multiple relayers using the same database - ### Drive-by changes Refactored and cleaned up the database code. Removed a lot of duplicate/unnecessary/redundant functions. ### Related issues - Fixes #2113 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - This changes the database prefixes internally to use domain name and domain id; also prepares us for #1899 ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	ed17ddc32f	Agent Argument Parsing (#2132 ) ### Description All configs can now be set by arguments as well as env and config files. The arguments will take the highest priority and overwrite the others. We should also update the docs to include this new support when it gets released, but since it is an entirely new feature it also can be a stealth release which we document after the fact. ### Drive-by changes ### Related issues - Fixes #2034 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	3e5b20c88e	Better validation of signers during config parsing (#2139 ) ### Description This improves the validation during config parsing to ensure that all the expected signers are configured which means the errors will have the correct config path. ### Drive-by changes Also updated the checks for missing chain configurations to be more clear and use the chain config path rather than the config that indicated the config was required and removes a now redundant check. ### Related issues - Fixes #2101 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Mattie Conover	b56b022f88	Hyperlane-Core WASM support (#2110 ) ### Description This shuffles some code around and changes dependencies to make `hyperlane-core` WASM compatible and adds a check to the pipeline to ensure it stays that way. To do this it does refactor hyperlane db to the `base` crate, which makes more sense IMO anyway as that is an implementation detail of the agents and not part of the core protocol. ### Drive-by changes ### Related issues - Fixes #1977 ### Backward compatibility _Are these changes backward compatible?_ Yes-ish - It does change where things are but it does not change support. Should only impact people who directly use the libraries. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Asa Oines	1d0e067166	Add relayer support for MultisigIsm and RoutingIsm (#2074 ) ### Description This PR adds support in the relayer for MultisigIsm (non-legacy) and RoutingIsm metadata building. AggregationIsm support is not yet implemented. ### Drive-by changes None ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ end-to-end	2 years ago
Mattie Conover	1b7d41ad8e	Fix docs link (#2097 ) ### Description Minor fix to docs URL linked to by configuration errors. ### Drive-by changes None ### Related issues - https://discord.com/channels/935678348330434570/1096400530722537513 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ None	2 years ago
Mattie Conover	d36cf6141a	Add link to reference (#2092 ) ### Description Forgot this in the other PR, just adds a link to the reference when there is a parsing error. ### Drive-by changes None ### Related issues - From #2070 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ None	2 years ago
Mattie Conover	1722f4ee7e	Better Agent Configuration Parsing (#2070 ) ### Description This is a significant change to how we parse user configs. The main goal of this PR is to give a clear path and env key to any errors that are encountered when parsing configuration. In order to accomplish this, I have crated a new trait `FromRawConf` which defines a set of steps to validate and convert individual fields. Also for every major config struct, there is now a `Raw` variant of it which is used by the `config` library to reduce errors that occur during deserialization itself. `Raw` types should always use the most basic form of a value, such as `String`, `StrOrInt`, `bool`, and ideally optional in all cases. Missing values should be handled during the raw conversion and not by `config`. I also needed to make changes to a number of types stored in the parsed config types to force validation forward to the parsing step instead of doing it when we read the config value. This also required me to create a filter to prevent trying to validate certain configs that we would not ever need to load. These changes can also be built on later to support something other than `config` if we choose to, or add support for merging configs from multiple sources since everything is optional. ### Drive-by changes - Default to `http` for connection type - Default to no gas payment enforcement if not specified instead of failing - `ChainSetup` -> `ChainConf` - `GasPaymentEnforcementConfig` -> `GasPaymentEnforcementConf` - Made ethereum connection configuration more forgiving - Moved hyperlane base settings from `mod.rs` to `base.rs` - Moved config chain tests in hyperlane core to `tests` dir to fix a cyclical import problem - Extension traits to help with config in hyperlane core - Moved `StrOrInt` to new hyperlane core `config` module - Support for parsing `U256` from `StrOrInt` - Removed `HexString` type which is now redundant - Updated base settings to use hyperlane domain - Use `heyKey` as signer type if `type` is not specified and `key` is - Moved hyperlane ethereum chain config to a new module ### Related issues - Fixes #2033 - Fixes #2012 ### Backward compatibility _Are these changes backward compatible?_ Yes - This should take in configs in the same shape but be more forgiving in a few places _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Test	2 years ago
Asa Oines	1edbecf9e3	Lag count in Validator agent (#2073 ) ### Description This PR adds a lag to `mailbox.count()` in the validator agent. Without this lag, validators can shut down after the first message because the latestCheckpoint call is lagged and will fail due to the lagged count still being zero. ### Drive-by changes None ### Related issues - Fixes #1764 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ None --------- Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Mattie Conover	e91c3100c2	Better Indexing (#2036 ) ### Description * Logs the distance from the tip * Calculates and logs the time to fully sync * Changes indexing timeout when catching up from 1s to 100ms ### Drive-by changes * Cleans up some log messages at the `info` level. * Refactors the indexing logic to be a bit cleaner/simpler * Updated all indexers to use the eta calculations ### Related issues - Fixes #1872 - Fixes #1851 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Asa Oines	f25feecfa7	Scaffolding for multiple ISM types in the relayer (#2028 ) ### Description This PR adds scaffolding in the relayer to support building metadata for multiple ISM types. To add a new ISM type: - Add a new implementation of MetadataBuilder for that type - Add the type to the SupportedIsmTypes enum - Add the type to the match statement in BaseMetadataBuilder ### Drive-by changes None ### Related issues - #1757 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E tests	2 years ago
Mattie Conover	1c2eb1bc07	Scraper Refresh (#1932 ) ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.	2 years ago
Mattie Conover	4657fcc737	Better rust package management (#1975 ) ### Description At the suggestion of @tkporter this moves common dependencies and configuration to use the new(ish) cargo workspace configuration feature. This was added in July 2022 and allows us to ensure all versions and configurations are the same across different crates. It does not decrease the typing, but it does reduce the updates required when we go to update a package and keeps duplicates next to each other. This PR does not change any package versions or add/remove any packages. It only moves the configuration. ### Drive-by changes Also sorted packages by name. ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Mattie Conover	1c004f4b10	Remove gelato support (#1946 ) ### Description We have not been using Gelato anymore so thought it would be nice to remove this code and avoid continuing to maintain it. I know I have had to update it a few times to maintain parity. As an added perk it should slightly speed up build times since it removes an entire crate. ### Drive-by changes None ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes - As we do not use this code _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago

1 2

90 Commits (f3b7ddb699fda050af7c90bd34ad91c4f6864021)