hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Daniel Savu	12840964b9	chore: add all new chains to `KnownHyperlaneDomain` struct (#3960 ) ### Description So validators can be run with `--originChainName <chainname>`, like in the docs [here](https://docs.hyperlane.xyz/docs/operate/validators/run-validators#running-the-binary) <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Daniel Savu	5a31e7b5d3	chore: debug slow relayer startup (#3932 ) ### Description Opened this PR to debug slow relayer startup, but it actually looks like that's already been fixed by hook indexing. The requirement for fast startup is that the processor sees new messages and then one of the following happens: - (1) if the message is meant for an unknown domain, it drops it ([this log](`6d132b02a3/rust/agents/relayer/src/msg/processor.rs (L269)`)), otherwise - it forwards the message to the submitter, which then: - (2) adds it straight to the confirm queue because it's already been submitted ([here](`6d132b02a3/rust/agents/relayer/src/msg/pending_message.rs (L159)`)) - (3) successfully builds metadata ([here](`6d132b02a3/rust/agents/relayer/src/msg/op_submitter.rs (L231)`)) - (4) in some cases building metadata fails because the ism doesn't exist The relayer logs either of the four cases (in most cases (2), because we're running a second relayer for now). To be even more confident in fast startup, I ran a single relayer and restarted it. New messages were successfully processed according to one of the four cases above, and the prep queues didn't increase ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
overallteach	42a6dbc3e6	chore: remove repetitive words (#3644 ) ### Description <!-- What's included in this PR? --> remove repetitive words ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> Signed-off-by: overallteach <cricis@foxmail.com>	6 months ago
Paul Balaji	65e4b05f77	chore: fix mainnet3 agent config (#3931 ) chore: fix mainnet3 agent config follow-on from https://github.com/hyperlane-xyz/hyperlane-registry/pull/49 Signed-off-by: Paul Balaji <paul@hyperlane.xyz>	6 months ago
Daniel Savu	826b4ae57f	feat: batching igp (#3870 ) ### Description Deducts IGP cost from senders of messages within a batch, based on the total gas used by the batch and the estimates for submitting messages individually. The formula is `gas_used_by_operation = gas_used_by_tx * (operation_estimated_gas / total_estimated_cost)` Note that the individual estimate have a buffer value added to them (currently 75k), which will slightly skew proportions, though by a negligible amount. For example, given these estimates in a batch: 150k, 160k, 180k, if we add 80k to each, we go from e.g. 0.312 for the first message to 0.319 -> 2.25% error which isn't so bad. The error can be larger if operations within a batch have very different estimates, but realistically within a 5% range based on back-of-the-napkin calculations. ### Drive-by changes - The batching feature introduced an bug whereby gas expenditure would only be deducted in the `confirm` step. Now this is done in the `submit` step to account for txs that revert - Sealevel e2e can now be disabled for faster iteration! I've not done the cleanest job, but even the fact that we have this will reduce the time to run e2e locally by more than half. To use it, set `SEALEVEL_ENABLED=false` when running `run-locally` - e2e uses a new utility called `get_matching_lines`, that can be used to count (and in the future _parse_) logs, to reconstruct the state of the relayer and have more expressive correctness checks. This is used to make sure that gas is deducted for all messages, including those in batches. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3709 ### Backward compatibility Yes ### Testing E2E, using `get_matching_lines`	6 months ago
Daniel Savu	942aba8e55	feat(evm): collaborative indexing through txid sharing (#3833 ) ### Description - improved indexing reliability by sharing channels between indexing tasks, such that they inform one another when they come across a hyperlane-related transaction - the receiving end will then get the receipt of that tx and filter it for events relevant to it - this adds some flexibility to indexing since we can stop relying on watermark cursor entirely: as long as we have one task using sequence indexing, all others can rely on being informed by their channels about what txs to index - this PR is only for the EVM chains, but can be easily extended ### Drive-by changes - Enforces gas payments for all chains in e2e (previously this only used to be the case for sealevel) - The channel size is configured per type, similarly to the indexing cursor, in `contract_sync/cursors/mod.rs` - Reduces log verbosity is some places, e.g. by manually implementing `Debug` to exclude fields that aren't relevant to debugging (maybe there's even a macro annotation to exclude, but I haven't checked) - Renames `MpmcReceiver` -> `BroadcastReceiver` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3267 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing e2e tested with the watermark cursor manually disabled	6 months ago
Daniel Savu	4261df74bb	chore: restrict for how long the backward syncer can block (#3918 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Noah Bayindirli 🥂	ba946f1eff	fix(config): switch to recommended 'rust-analyzer' from deprecated 'cargo' (#3886 ) ### Description - switch to recommended `rust-analyzer` since `cargo` is deprecated	6 months ago
Paul Balaji	d10b035ec2	fix: testnet4 holesky config (#3900 ) ran `yarn tsx ./scripts/agents/update-agent-config.ts -e testnet4` and updated testnet4 config Signed-off-by: Paul Balaji <paul@hyperlane.xyz>	6 months ago
Kunal Arora	554f875ebf	feat: core deployment on Holesky (#3776 ) ### Description Core deployment on Holesky ### Drive-by changes None ### Related issues - fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3597 ### Backward compatibility Yes ### Testing Manual	6 months ago
Trevor Porter	db9f733e4f	fix: use secret RPC urls in infra, apply environment-specific registry overrides (#3875 ) ### Description A while back, it seems secret RPC URLs stopped being used by monorepo services (see https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3179/files#diff-ba3b7bea2b5c046acd88d25ed382df875b49d77e061f2ef8347ffcc741c0353eR34). This adds them back. Some things to note on the approach: - Sadly, due to the way some of the infra config generation works, we're forced to use synchronous interfaces for getting chain metadata in a bunch of places. This is why `getRegistry` returns a `FileSystemRegistry`, which is a `SynchronousRegistry`. - Applying secrets / overrides requires us to do things async for two reasons -- one is just that fetching from GCP secrets is an async operation, the other is that a `MergedRegistry` is not a `SynchronousRegistry`. The approach I've taken is that there are two ways of getting a registry - using `getRegistry` will give the synchronous version that can continue to be used by our sync config generation, but it won't have any environment-specific overrides (tx overrides or secrets). To get an async registry with environment-specific overrides, the `getRegistry` function on the EnvironmentConfig is used. - Overrides are now more succinct as the merging happens later in the pipeline ### Drive-by changes - Started moving away from using the gcloud CLI under the hood for GCP operations and toward using the GCP secret SDK - Updated the mainnet3 key funder to fix an incident, but will follow up with a fresh deploy once this is mergd ### Related issues - These are the code changes for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3774, but I'll follow up with fresh deploys for the monorepo services ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Connor McEwen <connor.mcewen@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: J M Rossy <jm.rossy@gmail.com>	6 months ago
Yorke Rhodes	49f41d9759	chore: Merge branch 'main' into cli-2.0 (#3863 ) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Connor McEwen <connor.mcewen@gmail.com> Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Avi Atkin <103125634+avious00@users.noreply.github.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Ali Alaoui <aalaoui2001@gmail.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz> Co-authored-by: omahs <73983677+omahs@users.noreply.github.com>	6 months ago
omahs	806f9842de	fix: duplicate word typos (#3854 )	6 months ago
Daniel Savu	214f503e53	feat: forward-backward message processor (#3775 ) ### Description - adds logic for iterating message nonces in the processor task in forward-backward fashion - removes all manual calls that were updating the next nonce to query. This is done automatically in the iterator struct now - stores the highest known message nonce in the db, which is used to initialize the iterator ### Drive-by changes - Converts the concrete HyperlaneDb type stored in the processor to a trait obj, to enable mocking db responses ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3796 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3816 ### Backward compatibility Yes - if there is no db entry for the new prefix, the processor starts from nonce zero, so no migration is required ### Testing Unit testing + e2e	6 months ago
Trevor Porter	37af3dc6ba	feat: deploy to zetachain and redstone (#3788 ) ### Description - Deploys to zetachain and redstone - Some learnings can be found in https://discord.com/channels/935678348330434570/1240168332813144084 Chains not yet fully connected: - inevm (Mailbox ownership situation makes this hard) - mantapacific (waiting on an update to the multisig ownership set there, then will create the tx) - non-EVM chains ### Drive-by changes - decreased ethereum cost from $2 to $1.5 following some feedback - Added Superform to the blast set - Allowed for concurrent deploys I originally had some ugly caching to avoid checking storage gas oracles so much (we check them for each route in a routing hook configuration), but I've removed this. Calling it out for the future though `232695dd6c` ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	27aabf238c	feat: Relayer tokio task instrumentation (#3760 ) ### Description - Started off adding tokio-metrics but then realised those are quite general, so while we do have instrumentation it's not exposed in our metrics endpoint - switched to adding [tokio-console](https://github.com/tokio-rs/console/tree/main), which does give insight into the lifetime of specific tasks, so we can check which ones take up a long time during relayer startup. These are only visible at the `dependencyTrace` log level, so don't affect performance in the `hyperlane` context. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Helps debug https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3454 and any future performance issues - Does half the work for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3239 (still need to expose these in the metrics endpoint and import the grafana template) ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	1c4cf44c91	fix: allow all log levels in release build (#3824 ) ### Description Found the reason why we never had `trace` level logs in release builds - the `tracing` crate had a feature flag enabled that would wipe out all trace calls at compile time :( ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3823 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Trevor Porter	335a0d7479	feat: add agent config for Osmosis Mainnet (#3825 ) ### Description - Added agent config description for Osmosis Mainnet - Updated infra test to allow agent config chains to be a superset of infra chains Taken from @byeongsu-hong's https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3766, but with a fixed test ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: byeongsu-hong <hong@byeongsu.dev>	6 months ago
Trevor Porter	03ca2d1df7	Update gas oracles, deploy agents (#3755 ) ### Description - Updates gas oracles to make messages to Scroll slightly cheaper - After running into issues configuring non-EVM remotes in the IGP / gas oracle tooling (see https://discord.com/channels/935678348330434570/1238439687165579264), added a temporary band aid of just try/catching. We'll want to fix this at some point in the future - Deploys agents with a new image - Updates the agent JSON configs with some params we were running the agents with in prod via env vars - Adjusts kathy frequency on testnet to send every 1.5hrs instead of 7hrs - Stops running eclipsetestnet agents - apparently there was a reset ### Drive-by changes Some drive-by logging changes ### Related issues n/a ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	5d96847ffb	Multithreaded relayer (#3748 ) ### Description - Sets the tokio runtime flavor to be multithreaded and hardcodes the worker threads to 20. The latter usually defaults to the number of CPUs in the system so 20 may be high, but it worked fine in RC. Improved performance metrics: - time to first `Ingesting leaf` log: 1:30 mins (multithreaded) vs 5:51 mins (single threaded) - time to first message in the submit queue of Optimism (one of the slowest chains to boot up for): 8:30 mins vs 20 mins ### Drive-by changes Sets more descriptive `span`s when instrumenting the contract syncs ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Manually, by checking the RC logs	7 months ago
Daniel Savu	be43b0fd91	feat: use batching configs on all execution environments (#3723 ) ### Description While we only have a tx batching implementation available for EVM chains, we still parallelize the `prepare` and `confirm` steps in the serial submitter according to the `maxBatchSize` config item. To reap the benefits of this change, we need to make these configs universal, because they were only defined within `h_eth::ChainConnectionConf`. This PR adds slight duplication because it adds the batching config on each individual connection config, but it's far simpler than including it in the `ChainConf` struct. The latter would've required changing all the trait impls of `BuildableWithProvider` to include the batching config. ### Drive-by changes Renames `serial_submitter` to `op_submitter` since it's no longer serial. ### Related issues Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3722 ### Backward compatibility Yes ### Testing Manually - added the configs to cosmos e2e and checked the logs	7 months ago
Daniel Savu	f46cf7708e	Indexing Type Wrapper (#3610 ) ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3544. Wraps all types to be indexed in an `Indexed<T>` struct in the relayer. This allows for using sequece-aware cursors for data that doesn't contain a sequence but can have it inferred from metadata (as is the case for the IGP on sealevel). `Indexed<T>` stores the sequence as an `Option`, so the db can have additional sequence data stored in it when the sequence is defined - this is how `process_indexed_gas_payment` is implemented now. In addition, the IGP payments and Deliveries are configured to be indexed with a `SequenceAware` cursor on sealevel (see `cursors/mod.rs`). ### Drive-by changes Implements the custom `Encode` and `Decode` traits for `IndexingDecorator`, which can be generalized to `Option<T>` where `T: Encode + Decode`. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing E2E	7 months ago
Daniel Savu	301239265e	WIP: don't hardcode multicall gas limit (#3710 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	8e5ef272a6	fix: agent config check, run relayer on mantapacific (#3714 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	22f367ff6c	fix: sleep after submitting to cosmos chains (#3699 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	5aeb83a2e5	Relayer Message Batching to EVM (#3656 ) ### Description Status: - [x] op-queue scaffolding - [x] hyperlane-ethereum integration with multicall - [x] new config items - [x] actually build batch in the op-queue submit step - [x] e2e tests with batching configured - [x] dynamic batch sizes (batch as many messages as the prepare step has ready) - [x] parallelize metadata building - [ ] proportional IGP deduction ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes #3628 - Fixes #3630 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	7 months ago
J M Rossy	3528b281e3	Migrate SDK consts to Registry and use new registry utilities (#3615 ) ### Description - Remove most consts from SDK - Refactor tests that relied on chainMetadata - Remove JSON imports/exports in the SDK - Update the HelloWorld package to read metadata from the registry lib - Update the Infra package to read/write to a LocalRegistry - Update the CLI to read and write from registries ### Drive-by changes - Fix typo in `EvmHypCollateralVault` token standard - Use consistent, newest yaml lib version ### Related issues Fixes https://github.com/hyperlane-xyz/issues/issues/917 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2810 ### Backward compatibility No: - Several SDK exports like `chainMetadata` have been removed - Common args for the CLI like `--chain` have been replaced with `--registry` and `--overrides`	7 months ago
Trevor Porter	c4b7ad3196	Update Mode and Blast validator sets & gas oracles (#3669 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: nambrot <nambrot@googlemail.com>	7 months ago
Trevor Porter	21cd38c2f3	Update IGP gas oracles with new heuristics, get Cosmos gas prices from registry (#3662 ) ### Description The original and main motivation of this PR was to feel confident when running `deploy.ts -m igp` off of main. Prior to this, running it off main could result in misconfiguring gas oracles, particularly when it came to non-EVM remotes. See https://github.com/hyperlane-xyz/issues/issues/1178 for some context. This PR does a number of things: - Fetches Cosmos gas prices from the Cosmos registry - the `gasPrices.json` now lets you specify the decimals relating to the amount in there. There are 2 reasons for this: it's easier for us to deal with gwei values for EVM chains (as we often end up manually changing these instead of just relying on the script). The second reason is covered in the next bullet - Handles decimals when it comes to gas prices. The existing tooling assumed gas prices would always be denominated in wei. An extra weird situation arises with non-EVM chains: chains like Neutron have gas prices that are quotes in the smallest denomination (ie the equivalent of "wei" for NTRN), but the gas price is not an integer and is very small, e.g. `0.0053`. This poses issues because the IGP contracts require the gas price that's configured onchain to be an integer. Simply rounding up from 0.0053 to 1 results in almost 2 orders of magnitude difference :(. So the solution here is to have some logic that accommodates really tiny gas prices by scaling up the configured gas price and scaling down the configured token exchange rate. - Applies some "non-linear" gas pricing heuristics. See https://github.com/hyperlane-xyz/issues/issues/1204 and https://discord.com/channels/935678348330434570/935678739663192184/1232293417309044818 for some context - Running `update-agent-config.ts` will fetch gas prices for Cosmos chains from the Cosmos registry and ensure they're accurate ### Drive-by changes Some logging messages when applying IGP gas oracle changes to make it more clear what the change actually does ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3386 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3484 - Fixes https://github.com/hyperlane-xyz/issues/issues/1178 - Fixes https://github.com/hyperlane-xyz/issues/issues/1206 - Fixes https://github.com/hyperlane-xyz/issues/issues/1204 ### Backward compatibility Fully backward compatible ### Testing Ran deploy.ts for the IGP on all chains! --------- Co-authored-by: nambrot <nambrot@googlemail.com>	7 months ago
Trevor Porter	d6f25ed035	Deploy to Mode & Blast (#3653 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	f2cb6ebc1b	refactor: organize files in `hyperlane-ethereum` (#3633 ) ### Description Some cleanup along my way to integrating with `Multicall3` in hyp-ethereum. - Categorizes most files from `src` into `ism`, `contracts`, `signer` modules - Renames the previous `contracts` dir (that held the abi interfaces) to `interfaces`, so we can use `contracts` for the actual contracts (e.g. mailbox, igp)	7 months ago
Trevor Porter	700240eb8a	Ancient8 new ism, misc infra fixes (#3606 ) ### Description - Deploys ancient8 with a new ISM where we are not quorum - Pulls out warp config to live in a single file / function so that deploy.ts and check-deploy.ts are in sync. Otherwise fork.sh is broken for warp routes - Shuffles around the aw multisig serialization stuff - now they're segmented by environment and context and are written to `config/{env}/aw-validators/{context}.json`. This was in response to some confusing merging logic, see https://discord.com/channels/935678348330434570/1228377321082327141/1229464197297868910 ### Drive-by changes Needed to specify pino to work in pretty mode for warp monitoring :/ this is necessary because pino doesn't work otherwise. After #3582 we should change this ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	f4626eb4fe	Allow protocol-specific indexing (#3544 ) ### Description - Generalizes contract syncer hashmaps to store trait objects instead of concrete structs, since both `SequencedDataContractSync` and `WatermarkContractSync` implement `ContractSyncer` - The nice thing this enables is being able to instantiate a different type of syncer for the same contract, for different chain protocols (e.g. for IGP, we can use a sequenced syncer on sealevel and cosmwasm, and a watermark syncer on EVM). - The problem is that `ContractSyncer` is implemented on different trait bounds on `SequencedDataContractSync` and `WatermarkContractSync`. Most notably, the `SequencedDataContractSync` impl bounds the indexed type by the `Sequenced` trait. This means that if a type is to be indexed by more than one syncer strategy, it needs to implement the union of bounds of those strategies - Blocker: In the IGP, this means that `InterchainGasPayment` needs to implement `Sequenced` and we have no obvious way of getting that just from the type. Even if the sequence is derived from the type, it has to be a sensible value, since indexing reliability depends on it. - Similar to the point above, we now have `contract_syncs`, a single function that instantiates contract syncs for all chains. Since `contract_syncs` must be able to build a `ContractSyncer` trait obj for all syncer types, it bounds both `T` (the indexed type) and `D` (the log store type) with a union of bounds, so it may be too strict for some types in the future. - The scraper now uses forward-backward indexing instead of just forward indexing, because that simplified the refactor - there's now a convenience trait called `TryFromWithMetrics` that allows adding this bound: `SequenceIndexer<T>: TryFromWithMetrics<ChainConf>,`. This isn't really extendable if we use non-sequence indexers, but it's fine for now since this is used everywhere - replaces the macro-based contract syncer builder with generic fns Still todo: - add new `IndexingDecorator<T>` struct that can wrap `InterchainGasPayment`, so we can use sequence-aware indexing for the IGP - remove sequence logic from the watermark cursor ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Noah Bayindirli 🥂	aea79c686e	feat: add dry-run support for warp deploy + gas stats (#3586 ) ### Description * New feature to allow users to dry-run the warp deploy script against a forked (base/origin) network of their choice * To run: `yarn build && yarn hyperlane deploy warp --dry-run` \|\| `yarn build && yarn hyperlane deploy warp -d` * Externally enables: `hyperlane deploy warp --dry-run` \|\| `hyperlane deploy warp -d` * Also adds gas usage util for both warp & core deployments in all contexts, e.g. * When running a vanilla core deploy, for example, between alfajores and fuji, you will now see: ``` ⛽️ Gas Usage Statistics - Gas required for core deploy on alfajores: 0.0058686745 CELO - Gas required for core deploy on fuji: 0.0239308515 AVAX ``` ### Drive-by changes * None ### Related issues * Fixes https://github.com/hyperlane-xyz/issues/issues/819 ### Backward compatibility * Yes ### Testing * Note: `hl` == `yarn build && yarn hyperlane`. The below tests are only a sample and are not inclusive. #### Manual Testing * With `anvil` NOT running in separate instance: * `hl deploy warp -d` * Throws: ``` Error: No active anvil node detected. Please run `anvil` in a separate instance. ``` * With `anvil` running in separate instance: * `hl deploy warp -d -k c0052e22df5d1f4ae7c51e254Xx00Xx0eb833453eaed6301xXxxx8a30d92d10a` (any private key) * Throws `Error: Invalid address length. Please ensure you are passing an address and not a private key.` * `hl deploy warp -d -k 0x16F4898F47c085C41d7Cc6b1dc0xX0xXX017dcBb` (any public address) * Output: ``` 🔎 Verifying anvil node is running... ✅ Successfully verified anvil node is running Using warp route deployment config at ./configs/warp-route-deployment.yaml No chain config file provided ? Do you want to use some core deployment address artifacts? This is required for PI chains (non-core chains). no Forking alfajores for dry-run... ✅ Successfully forked alfajores for dry-run Impersonating account (0x16F4898F47c085C41d7Cc6b1dc72B91EA617dcBb)... ✅ Successfully impersonated account (0x16F4898F47c085C41d7Cc6b1dc72B91EA617dcBb) ... ✅ Hyp token deployments complete Writing deployment artifacts Deployment is complete! Contract address artifacts are in artifacts/dry-run_warp-route-deployment-2024-04-16-12-27-02.json Warp config is in artifacts/dry-run_warp-config-2024-04-16-12-27-02.json ⛽️ Gas Usage Statistics - Gas required for warp dry-run on alfajores: 0.013310162514578124 CELO Resetting forked network... ✅ Successfully reset forked network ✅ Warp dry-run completed successfully ``` #### CI Testing * Successful CI-backed integration/regression testing via `ci-test.sh`	7 months ago
Trevor Porter	d3aa3f5cc9	Set prio fee override for moonbeam (#3587 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Trevor Porter	64c8c8ad6d	Update key funder deployments w/ RpcConsensusType.Fallback, fix with ESM (#3583 ) ### Description Making testnet4 and mainnet3 key funder use fallback for consistency & robustness ### Drive-by changes Updated the deployed key funder images	8 months ago
Trevor Porter	26a26f4073	Agent transaction overrides, ensure agent configs are updated in CI (#3563 ) ### Description - Adds some basic transaction overrides for Ethereum chains to be used by the agents. This follows the existing config schema in chain metadata that we have been using on the TS side for a while. The overrides are as follows: - `transactionOverrides.gasPrice` - if specified, non EIP-1559 txs are used, and they use this gasPrice - `transactionOverrides.gasLimit` - if specified, this takes precedence over any gas estimation that's done and is the gas limit that's used for txs. (This is pretty niche and I don't imagine we'll ever wanna use this internally, but happens to cover the use case described [here](https://discord.com/channels/935678348330434570/984123861144600587/1224617234580766741), which is nice to cover) - `transactionOverrides.maxFeePerGas` - if specified and if the chain supports EIP 1559 txs, is used for the `maxFeePerGas` - `transactionOverrides.maxPriorityFeePerGas` - if specified and if the chain supports EIP 1559 txs, is used for the `maxPriorityFeePerGas` - Updates some of the transaction overrides in mainnet3 / testnet4 to be more reasonable now that we'll be using them to submit transactions pretty frequently - Note that atm the transaction overrides are most easily applied to the agents via the config JSONs, and not via our K8s configuration. The upshot of this is if we wanna change a transaction override right now, the easiest way to do it will be to run `update-agent-config.ts` and then build a new image. Created https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3565 to track making this experience better - Part of this change means that it's more important to keep the `transactionOverrides` specified in the SDK / infra up to date with the agent config. So I ended up doing https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3325 along the way, which also required doing https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3311. - To fix the issue with arbitrum blocks, we now require the index.from to be specified in the chain metadata if the chain's technical stack is arbitrum nitro - A new script, `update-agent-config.ts` is added, and it's ran for mainnet3 and testnet4 in CI to make sure there's no diff (similar idea to what we do with prettier) - Along the way I renamed mainnet3_config.json and testnet4_config.json to mainnet_config.json and testnet_config.json. This has been a source of confusion with ppl in the past - in v2 we named them without any numeric suffixes, so we now just do that again ### Drive-by changes - There's a drive-by to poll more frequently for pending transactions (2 seconds). The default is 7 seconds in Ethers, and for many chains that have really quick block times, this may be a way for us to speed up us learning that a transaction is included - Don't enforce gas on the inevm routes, looks like there's some weirdness there that was reported by Nam ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3562 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3325 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3311 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Trevor Porter	526d27db04	Ancient8 deploy (#3472 ) ### Description - Deploys ancient8 core - Deploys a USDC warp route from Ethereum to ancient8 - Some changes to the key generation step - specifically: - `scripts/create-keys.ts` now expects `--newChainValidators chainname=1` instead of the flag `--newChains` - fixed the logic around adding new validators, which previously was broken - Adds a hook module to deploy.ts - this is meant to be used in a similar fashion to the warp module, where configs are built ad-hoc - changes `aw-multisig.json` to be `aw-multisig-${context}.json`, as previously the entire aw-multisig.json would be overridden no matter what the current context was	8 months ago
Noah Bayindirli 🥂	38358ececd	feat: deprecate Mumbai testnet (#3521 ) ### Description - Removes all reference to Mumbai testnet, since it is soon being [deprecated](https://polygon.technology/blog/polygon-pos-is-cooking-the-napoli-upgrade-means-better-ux-the-mumbai-testnet-takes-a-bow) by the Polygon team - Monorepo docker tag: https://console.cloud.google.com/gcr/images/abacus-labs-dev/global/hyperlane-monorepo@sha256:d4b44359cae9c4e3bdeb6f965c59d056f7fce72c42069adc0dea038ac600785a/details?authuser=5&project=abacus-labs-dev&tab=info - Agent docker tag: https://console.cloud.google.com/gcr/images/abacus-labs-dev/global/hyperlane-agent@sha256:a578d96f7f06721d0b6cd8095d6a1fa5d22f01b59eba3320ee3b1fe34ca3c5df/details?authuser=5&project=abacus-labs-dev&tab=info ### Drive-by changes - Adds missing Testnet domains, Fuji & BSC Testnet - Fix for failing `test-rs` CI check with ``` error[E0004]: non-exhaustive patterns: `chain::KnownHyperlaneDomain::Fuji` and `chain::KnownHyperlaneDomain::BinanceSmartChainTestnet` not covered --> hyperlane-core/src/utils.rs:226:15 \| 226 \| match $v { \| ^^ patterns `chain::KnownHyperlaneDomain::Fuji` and `chain::KnownHyperlaneDomain::BinanceSmartChainTestnet` not covered ``` ### Related issues - Dependency on #3523 - Fixes #1181 ### Backward compatibility - [x] Backwards compatible - [x] Requires docker image tags to be added post-build - [x] Requires agent artifact cleanup in cloud account: https://www.notion.so/hyperlanexyz/Runbook-83c755f2652943289cf98cd1309487a8?pvs=4#77730653a12f409dacfbd427cab545d3 ### Testing * Build success * `yarn hyperlane chains list -e testnet` does not return `mumbai` * fork.sh script success locally	8 months ago
Daniel Savu	9b11f327c5	Range retries in watermark cursor (#3500 ) ### Description Changes the logic in the watermark cursor, to update the range in the `update` method rather than in `get_next_range`. This means that RPC errors won't result in ranges being skipped, so IGP indexing should be more reliable. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - First in a sequence of many, for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Daniel Savu	00fcaf74db	fix: warn on tx reverts; log cosmos tx results (#3497 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Daniel Savu	ff6b06079e	Half the relayer timeout on v3 (#3508 ) ### Description V3 counterpart of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3431 Co-authored-by: nambrot <nambrot@googlemail.com>	8 months ago
Daniel Savu	6e39cf01af	Relayer retry endpoints (#3476 ) ### Description Adds a `/message_retry` relayer endpoint, which supports retrying OpQueue operations either by ID or by destination domain. Such a request will be sent to an `mpmc` channel's receiving end in the OpQueue, which is emptied when `OpQueue::pop` is called. Example calls: ``` GET http://127.0.0.1:60843/message_retry?destination_domain=42 GET http://127.0.0.1:60843/message_retry?message_id=0x46910b1329ee53c86a023b322e9ca1c17e5f9f0bee789c77b0abced0a173d714 ``` If the endpoint is called specifying both filters, two requests will be sent across the channel, one for each condition. Efficiency note: The entire queue is iterated over if there's at least one retry request in the channel. The good thing is that rebuilding the heap is `O(n)` so not too bad. ### Drive-by changes - Removed the usage of `enum_dispatch`, because it would have made unit tests messy. The drawback is that we have to work around object safety restrictions in `PendingOperation` - for instance it can't be cloned (but can be stored behind an `Arc` if we ever need this). - Added new methods to `PendingOperation`, either to match by ID, or to implement heap element prioritization without depending on the concrete `PendingMessage` type (as was done before). Additions: `id()`, `origin_domain()`, `priority()`. Don't think this conflicts with any operations we may add in the future (e.g. gas oracle updates). - Although `tokio`'s broadcast channel is multi-producer-multi-consumer by default, the consumer end isn't `Clone` - you instead need to have a producer to call `.subscribe()` on to get a new consumer, so I added an `MpmcChannel` struct to encapsulate keeping a producer around to get new consumers. - OpQueue is moved into its own file and has unit tests added for retries (covering broadcast to multi-queue, and both retry types). - Did a bit of agent `server `cleanup, including `EigenNodeApi` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3098 ### Backward compatibility Yes ### Testing Unit tests for the server route (channel transmitter logic) and for the OpQueue retries (channel receiver logic).	8 months ago
J M Rossy	adab1bec0c	Add code spellcheck CI job (#3467 ) ### Description Fix spelling mistakes and create codespell CI job. An extension of #3422 with the remaining mistakes fixed ### Backward compatibility Yes --------- Co-authored-by: miles <66052478+miles-six@users.noreply.github.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz>	8 months ago
Paul Balaji	d535e66297	fix: proxy factory impl names (#3471 )	8 months ago
Daniel Savu	140e3c4bb5	fix: neutron grpc url default; url parsing error message (#3466 ) ### Description The default neturon grpc url is plaintext but it was using a `http` prefix so all requests would fail ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Trevor Porter	176e710cae	Fix BSC gas price issues by treating chains with a base fee of 0 as non-1559 (#3462 ) ### Description - Fixes #3396 - See v2 backport here #3463 So turns out this is because: - BSC nodes tend to enforce a minimum gas price of 3 gwei when you submit a tx - it's possible for validators to include privileged txs at a lower gas price, so in practice (probably some mev stuff ?) there are a bunch of txs in blocks with transactions lower than 3 gwei - BSC's 1559 situation is they have a base fee of zero, so it's basically like they just use legacy txs - when we try to figure out what to pay in prio fees, we are suggested a 1 gwei priority fee: ``` $ cast rpc eth_feeHistory 10 latest "[5.0]" --rpc-url https://rpc.ankr.com/bsc {"oldestBlock":"0x23706f7","reward":[["0x3b9aca00"]],"baseFeePerGas":["0x0","0x0"],"gasUsedRatio":[0.1149265]} $ cast td 0x3b9aca00 1000000000 ``` So options are: 1. tx overrides for bsc, set it to 3 gwei 2. use a different eip 1559 estimator and set the percentile used in eth_feeHistory to something higher. Atm the percentile is 5%, if you change this to 50% we seem to get 3 gwei. This would have consequences for other chains tho 3. another heuristic, to fall back to legacy txs (where we can trust the eth_gasPrice to give us an accurate price) whenever the base fee is 0 I'm a tx override hater and we don't have this in the agents so far so I'm gonna go with 3. It's the least intrusive change	8 months ago
Daniel Savu	8ea0bde7c9	Use app context classifier in relayer submitter queues (#3385 ) ### Description Includes the `app_context` classification in `PendingMessage`, and adds trait methods on `PendingOperation` to require always having such a label on `OpQueue` operations. This is done by reusing the matching list logic from the validator checkpoint labels (https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3057). The nice thing is that this enables later support for retrying a group of `OpQueue` operations just by specifying the `app_context` label, without adding any new logic, since these labels are essentially matching list results. One downside to using `app_context` for retries is that the endpoint caller is constrained to only the matching lists defined by the relayer operator - however imo only the relayer operator that should be able to trigger retries. ### Drive-by changes The `OpQueue` type alias is converted to an actual struct, that stores the queue label (for metrics purposes), and also the `IntGaugeVec` metric: the generic group of metrics associated with that queue (basically only `submitter_queue_length` currently). ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3240 ### Backward compatibility Yes ### Testing Manual, by spinning up a relayer for injective and inevm. Sample metrics, from `--metricAppContexts '[{"name": "injectivelabel", "matchingList": [{"destination_domain": 6909546 }] }, {"name": "inevmlabel", "matchingList": [{"destination_domain": 2525 }] }]'` ``` hyperlane_submitter_queue_length{agent="relayer",app_context="inevmlabel",hyperlane_baselib_version="0.1.0",queue_name="confirm_queue",remote="inevm"} 11 hyperlane_submitter_queue_length{agent="relayer",app_context="inevmlabel",hyperlane_baselib_version="0.1.0",queue_name="prepare_queue",remote="inevm"} 0 hyperlane_submitter_queue_length{agent="relayer",app_context="injectivelabel",hyperlane_baselib_version="0.1.0",queue_name="confirm_queue",remote="injective"} 63 hyperlane_submitter_queue_length{agent="relayer",app_context="injectivelabel",hyperlane_baselib_version="0.1.0",queue_name="prepare_queue",remote="injective"} 13281 ```	9 months ago
Paul Balaji	76bd730103	feat: remove goerli testnets (#3389 )	9 months ago
Daniel Savu	ee1f11b4e4	fix(agents): add inevm to known domains and make igp indexing log war… (#3387 ) ### Description adds inevm to known domains and makes igp indexing log warn level ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago

1 2 3 4 5 ...

858 Commits (d274c6b3f879215611f8940ca350fb144a525422)