218 Commits (d274c6b3f879215611f8940ca350fb144a525422)

Author	SHA1	Message	Date
Paul Balaji	7625bf8dce	fix: typo in trait builder log (#4292) ... - fix: typo in trait builder log - noticed while looking at scraper logs	4 months ago
Trevor Porter	fe9365fdb3	chore: agent metric log tweaks (#4244) ... ### Description - some small tweaks as I looked into a recent issue with metrics ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Daniel Savu	b643fba90a	feat: use multiple `mpsc` channels instead of `broadcast` for async interface (#4190) ... ### Description Follow up to https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4180 Lowers txid broadcast channel capacity from 1M to 30k and refactors it to use a vec of `mpsc` under the hood. ### Context The heap profiler showed that each new chain takes up 100MB of memory because of the txid broadcast channel. In addition to this, the tokio broadcast channel is a ring buffer where new items simply overwrite old items, making it hard to safely lower the capacity. As such, to be able to safely lower the capacity of this channel, usage of `tokio`'s `broadcast` channel is replaced with a `Vec` of `mpsc`s, which offer an async `send` interface for backpressure. So instead of overwriting old items, senders will block on sends, which makes sense in our case as we absolutely don't want to miss items. The submitter is slower than the indexer based on empirical observation, so blocking on channel size doesn't bottleneck throughput. I noticed the max number of channel items we can reach is 29k, by running the RC relayer after an 8 day break, during which around 160k messages were submitted. Lowering the capacity to 30k should cause us to only encounter backpressure in very rare cases, and the footprint of this channel would lower from 100MB to 3MB. ### Drive-by changes The delivery indexer in the scraper is no longer passed a txid receiver, since message deliveries are not expected to occur in the same tx as message dispatches. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Testing E2E, by adding invariants to make sure txid indexing works --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	4 months ago
Daniel Savu	d864c6a939	fix(evm): only create signer provider when needed (#4208) ... ### Description Adds an associated const to all ethereum provider builders to specify whether a signer is needed. For an e2e run with 20 messages, we now have 12 providers built with signers at the beginning (3 of which are validator announce), and 708 providers built without a signer. This should hopefully fix the gas escalator memory leak. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Daniel Savu	97588b6fc1	fix(relayer): don't submit reverting messages in batches (#4169) ... ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4159 Some messages in a Multicall would revert because previous messages would invalidate them temporarily. The IGP allowance of those messages would be deducted, and if this happened more than once, those messages would become stuck. This PR fixes those unnecesary spends on reverting `process` txs by using `.call()` to simulate the outcome of a Multicall tx. Reverting messages are removed from the batch and submitted serially, and if the batch is only left with one non-reverting message, even that one is submitted serially (rather than through a batch of one message). Messages that did get submitted have their igp allowance deducted and status updated in `update_sent_ops_state`. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1294 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing Tested by manually setting the result of `call` to failed for all messages, to see if they are submitted serially. In prod, we can track the `Submitted transaction batch` log, which now includes any excluded messages from the batch.	4 months ago
Daniel Savu	bb16dfa728	fix: more conservative batching gas estimates (#4159) ... ### Description Estimating gas isn't very accurate for Multicall txs that allow failures. This PR now estimates gas as: `max(batch_tx_estimate, individual_tx_estimates_sum)`, where `individual_tx_estimates_sum` estimates each individual tx and adds a conservative overhead that `Multicall.aggregate3` adds per subtx. The aggregate3 overhead is calcualted based on some observed tenderly gas profiling results as follows. - For [4 txs](https://dashboard.tenderly.co/shared/simulation/63e85ac7-3ea9-475c-8218-a7c1dd508366/gas-usage): - aggregate3 gas: 805,160 - process costs: 290,091 + 168,711 + 168,711 + 168,711 = 796,224 - overhead: 805,160 - 796,224 = 8,936 => 2,234 per subtx - For [2 txs](https://dashboard.tenderly.co/tx/arbitrum/0xad644e431dc53c3fc0a074a749d118ff5517346c3f28d8e2513610cc9ab5c91a/gas-usage): - aggregate3 gas: 201,710 - process costs: 103,011 + 92,512 = 195,523 - overhead: 201,710 - 195,523 = 6,187 => 3,093 per subtx - So will assume an overhead of 3500 `aggregate3` gas per message in batch ### Drive-by changes This slightly changes the semantics of gas overrides. If a gas limit is configured, it will only guarantee that that's the minimum limit a tx can have. Now we use the `max(estimated_gas, gas_limit_override)` as the actual gas limit. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1295 ### Backward compatibility Yes ### Testing I think it's ok to release to release and check the new log, to see the individual estimate vs the batch estimate and double-check which one is picked in the tx explorer	4 months ago
Daniel Savu	a5e5f70610	chore: Revert "feat(hyp-ethereum): gas escalator middleware (#3852)" (#4098) ... This reverts commit c3c002a8fe. ### Description The gas escalator took up to 28gb of memory, and even though it was only ever released on RC, it must have taken up too many cluster resources, ending up starving the `hyperlane` relayer. Reverting for now until we have a better idea ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Daniel Savu	c3c002a8fe	feat(hyp-ethereum): gas escalator middleware (#3852) ... ### Description Wraps evm providers in a gas escalator middleware as part of the `build_with_signer` step, so signed txs can have their gas price bumped if stuck in the mempool. Gas price is increased every 60s by 12.5%, and is capped at the `max_fee_per_gas` set in tx overrides. ### Drive-by changes - changes to our fork of `ethers`: - removes the `M: Clone` bound, since it's not actually required and conflicts with the middleware we use - increases the level of the `escalated` log in the middleware to `debug` (so we can see it working in prod without enabling `trace`) ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2959 ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	5 months ago
Snoppy	f1fe882e3b	chore: fix typos (#3645) ... fix minor typos Signed-off-by: snoppy <michaleli@foxmail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz>	5 months ago
Daniel Savu	3bb9d0a767	fix(cosmos): validator checkpoint & agent indexing improvements (#3958) ... - propagates RPC errors to the indexing logic, so errors cause sleeps instead of overwhelming the RPCs. This also greatly reduces cosmos log noisiness: from 9.5k logs per min, to 2.5k logs per min. - checks for published checkpoints in reverse, to prioritize recent messages among those from before the validator was spun up - fixes a validator bug where errors in checkpoint publishing would cause the entire checkpoint queue to be reiterated through, from scratch - Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3887 so we can test this out on osmosis ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	942aba8e55	feat(evm): collaborative indexing through txid sharing (#3833) ... ### Description - improved indexing reliability by sharing channels between indexing tasks, such that they inform one another when they come across a hyperlane-related transaction - the receiving end will then get the receipt of that tx and filter it for events relevant to it - this adds some flexibility to indexing since we can stop relying on watermark cursor entirely: as long as we have one task using sequence indexing, all others can rely on being informed by their channels about what txs to index - this PR is only for the EVM chains, but can be easily extended ### Drive-by changes - Enforces gas payments for all chains in e2e (previously this only used to be the case for sealevel) - The channel size is configured per type, similarly to the indexing cursor, in `contract_sync/cursors/mod.rs` - Reduces log verbosity is some places, e.g. by manually implementing `Debug` to exclude fields that aren't relevant to debugging (maybe there's even a macro annotation to exclude, but I haven't checked) - Renames `MpmcReceiver` -> `BroadcastReceiver` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3267 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing e2e tested with the watermark cursor manually disabled	6 months ago
Yorke Rhodes	49f41d9759	chore: Merge branch 'main' into cli-2.0 (#3863) ... Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Connor McEwen <connor.mcewen@gmail.com> Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Avi Atkin <103125634+avious00@users.noreply.github.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Ali Alaoui <aalaoui2001@gmail.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz> Co-authored-by: omahs <73983677+omahs@users.noreply.github.com>	6 months ago
Daniel Savu	214f503e53	feat: forward-backward message processor (#3775) ... ### Description - adds logic for iterating message nonces in the processor task in forward-backward fashion - removes all manual calls that were updating the next nonce to query. This is done automatically in the iterator struct now - stores the highest known message nonce in the db, which is used to initialize the iterator ### Drive-by changes - Converts the concrete HyperlaneDb type stored in the processor to a trait obj, to enable mocking db responses ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3796 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3816 ### Backward compatibility Yes - if there is no db entry for the new prefix, the processor starts from nonce zero, so no migration is required ### Testing Unit testing + e2e	6 months ago
Daniel Savu	be43b0fd91	feat: use batching configs on all execution environments (#3723) ... ### Description While we only have a tx batching implementation available for EVM chains, we still parallelize the `prepare` and `confirm` steps in the serial submitter according to the `maxBatchSize` config item. To reap the benefits of this change, we need to make these configs universal, because they were only defined within `h_eth::ChainConnectionConf`. This PR adds slight duplication because it adds the batching config on each individual connection config, but it's far simpler than including it in the `ChainConf` struct. The latter would've required changing all the trait impls of `BuildableWithProvider` to include the batching config. ### Drive-by changes Renames `serial_submitter` to `op_submitter` since it's no longer serial. ### Related issues Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3722 ### Backward compatibility Yes ### Testing Manually - added the configs to cosmos e2e and checked the logs	7 months ago
Daniel Savu	f46cf7708e	Indexing Type Wrapper (#3610) ... ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3544. Wraps all types to be indexed in an `Indexed<T>` struct in the relayer. This allows for using sequece-aware cursors for data that doesn't contain a sequence but can have it inferred from metadata (as is the case for the IGP on sealevel). `Indexed<T>` stores the sequence as an `Option`, so the db can have additional sequence data stored in it when the sequence is defined - this is how `process_indexed_gas_payment` is implemented now. In addition, the IGP payments and Deliveries are configured to be indexed with a `SequenceAware` cursor on sealevel (see `cursors/mod.rs`). ### Drive-by changes Implements the custom `Encode` and `Decode` traits for `IndexingDecorator`, which can be generalized to `Option<T>` where `T: Encode + Decode`. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing E2E	7 months ago
Daniel Savu	301239265e	WIP: don't hardcode multicall gas limit (#3710) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	5aeb83a2e5	Relayer Message Batching to EVM (#3656) ... ### Description Status: - [x] op-queue scaffolding - [x] hyperlane-ethereum integration with multicall - [x] new config items - [x] actually build batch in the op-queue submit step - [x] e2e tests with batching configured - [x] dynamic batch sizes (batch as many messages as the prepare step has ready) - [x] parallelize metadata building - [ ] proportional IGP deduction ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes #3628 - Fixes #3630 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	7 months ago
Daniel Savu	f2cb6ebc1b	refactor: organize files in `hyperlane-ethereum` (#3633) ... ### Description Some cleanup along my way to integrating with `Multicall3` in hyp-ethereum. - Categorizes most files from `src` into `ism`, `contracts`, `signer` modules - Renames the previous `contracts` dir (that held the abi interfaces) to `interfaces`, so we can use `contracts` for the actual contracts (e.g. mailbox, igp)	7 months ago
Trevor Porter	26a26f4073	Agent transaction overrides, ensure agent configs are updated in CI (#3563) ... ### Description - Adds some basic transaction overrides for Ethereum chains to be used by the agents. This follows the existing config schema in chain metadata that we have been using on the TS side for a while. The overrides are as follows: - `transactionOverrides.gasPrice` - if specified, non EIP-1559 txs are used, and they use this gasPrice - `transactionOverrides.gasLimit` - if specified, this takes precedence over any gas estimation that's done and is the gas limit that's used for txs. (This is pretty niche and I don't imagine we'll ever wanna use this internally, but happens to cover the use case described [here](https://discord.com/channels/935678348330434570/984123861144600587/1224617234580766741), which is nice to cover) - `transactionOverrides.maxFeePerGas` - if specified and if the chain supports EIP 1559 txs, is used for the `maxFeePerGas` - `transactionOverrides.maxPriorityFeePerGas` - if specified and if the chain supports EIP 1559 txs, is used for the `maxPriorityFeePerGas` - Updates some of the transaction overrides in mainnet3 / testnet4 to be more reasonable now that we'll be using them to submit transactions pretty frequently - Note that atm the transaction overrides are most easily applied to the agents via the config JSONs, and not via our K8s configuration. The upshot of this is if we wanna change a transaction override right now, the easiest way to do it will be to run `update-agent-config.ts` and then build a new image. Created https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3565 to track making this experience better - Part of this change means that it's more important to keep the `transactionOverrides` specified in the SDK / infra up to date with the agent config. So I ended up doing https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3325 along the way, which also required doing https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3311. - To fix the issue with arbitrum blocks, we now require the index.from to be specified in the chain metadata if the chain's technical stack is arbitrum nitro - A new script, `update-agent-config.ts` is added, and it's ran for mainnet3 and testnet4 in CI to make sure there's no diff (similar idea to what we do with prettier) - Along the way I renamed mainnet3_config.json and testnet4_config.json to mainnet_config.json and testnet_config.json. This has been a source of confusion with ppl in the past - in v2 we named them without any numeric suffixes, so we now just do that again ### Drive-by changes - There's a drive-by to poll more frequently for pending transactions (2 seconds). The default is 7 seconds in Ethers, and for many chains that have really quick block times, this may be a way for us to speed up us learning that a transaction is included - Don't enforce gas on the inevm routes, looks like there's some weirdness there that was reported by Nam ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3562 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3325 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3311 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Noah Bayindirli 🥂	38358ececd	feat: deprecate Mumbai testnet (#3521) ... ### Description - Removes all reference to Mumbai testnet, since it is soon being [deprecated](https://polygon.technology/blog/polygon-pos-is-cooking-the-napoli-upgrade-means-better-ux-the-mumbai-testnet-takes-a-bow) by the Polygon team - Monorepo docker tag: https://console.cloud.google.com/gcr/images/abacus-labs-dev/global/hyperlane-monorepo@sha256:d4b44359cae9c4e3bdeb6f965c59d056f7fce72c42069adc0dea038ac600785a/details?authuser=5&project=abacus-labs-dev&tab=info - Agent docker tag: https://console.cloud.google.com/gcr/images/abacus-labs-dev/global/hyperlane-agent@sha256:a578d96f7f06721d0b6cd8095d6a1fa5d22f01b59eba3320ee3b1fe34ca3c5df/details?authuser=5&project=abacus-labs-dev&tab=info ### Drive-by changes - Adds missing Testnet domains, Fuji & BSC Testnet - Fix for failing `test-rs` CI check with ``` error[E0004]: non-exhaustive patterns: `chain::KnownHyperlaneDomain::Fuji` and `chain::KnownHyperlaneDomain::BinanceSmartChainTestnet` not covered --> hyperlane-core/src/utils.rs:226:15 | 226 | match $v { | ^^ patterns `chain::KnownHyperlaneDomain::Fuji` and `chain::KnownHyperlaneDomain::BinanceSmartChainTestnet` not covered ``` ### Related issues - Dependency on #3523 - Fixes #1181 ### Backward compatibility - [x] Backwards compatible - [x] Requires docker image tags to be added post-build - [x] Requires agent artifact cleanup in cloud account: https://www.notion.so/hyperlanexyz/Runbook-83c755f2652943289cf98cd1309487a8?pvs=4#77730653a12f409dacfbd427cab545d3 ### Testing * Build success * `yarn hyperlane chains list -e testnet` does not return `mumbai` * fork.sh script success locally	8 months ago
Daniel Savu	9b11f327c5	Range retries in watermark cursor (#3500) ... ### Description Changes the logic in the watermark cursor, to update the range in the `update` method rather than in `get_next_range`. This means that RPC errors won't result in ranges being skipped, so IGP indexing should be more reliable. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - First in a sequence of many, for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Daniel Savu	00fcaf74db	fix: warn on tx reverts; log cosmos tx results (#3497) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Daniel Savu	ff6b06079e	Half the relayer timeout on v3 (#3508) ... ### Description V3 counterpart of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3431 Co-authored-by: nambrot <nambrot@googlemail.com>	8 months ago
J M Rossy	adab1bec0c	Add code spellcheck CI job (#3467) ... ### Description Fix spelling mistakes and create codespell CI job. An extension of #3422 with the remaining mistakes fixed ### Backward compatibility Yes --------- Co-authored-by: miles <66052478+miles-six@users.noreply.github.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz>	8 months ago
Trevor Porter	176e710cae	Fix BSC gas price issues by treating chains with a base fee of 0 as non-1559 (#3462) ... ### Description - Fixes #3396 - See v2 backport here #3463 So turns out this is because: - BSC nodes tend to enforce a minimum gas price of 3 gwei when you submit a tx - it's possible for validators to include privileged txs at a lower gas price, so in practice (probably some mev stuff ?) there are a bunch of txs in blocks with transactions lower than 3 gwei - BSC's 1559 situation is they have a base fee of zero, so it's basically like they just use legacy txs - when we try to figure out what to pay in prio fees, we are suggested a 1 gwei priority fee: ``` $ cast rpc eth_feeHistory 10 latest "[5.0]" --rpc-url https://rpc.ankr.com/bsc {"oldestBlock":"0x23706f7","reward":[["0x3b9aca00"]],"baseFeePerGas":["0x0","0x0"],"gasUsedRatio":[0.1149265]} $ cast td 0x3b9aca00 1000000000 ``` So options are: 1. tx overrides for bsc, set it to 3 gwei 2. use a different eip 1559 estimator and set the percentile used in eth_feeHistory to something higher. Atm the percentile is 5%, if you change this to 50% we seem to get 3 gwei. This would have consequences for other chains tho 3. another heuristic, to fall back to legacy txs (where we can trust the eth_gasPrice to give us an accurate price) whenever the base fee is 0 I'm a tx override hater and we don't have this in the agents so far so I'm gonna go with 3. It's the least intrusive change	8 months ago
Paul Balaji	76bd730103	feat: remove goerli testnets (#3389)	9 months ago
Daniel Savu	9736164392	fix: log level order, cosmos logging nit (#3374) ... ### Description Our logging prio order was off, double checked with [this crate](https://docs.rs/log/latest/log/enum.Level.html). Debug logs would be visible at `Info` level according to the previous rules. ### Drive-by changes Removed the debug formatting of a cosmsos log field, so it's no longer surrounded in quotes (bc we'd get `cursor_label: ""InterchainGasPaymentCursor""`) ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	2c2f79cbb6	Add log for cosmos cursors (#3366) ... ### Description IGP indexing seemed to not be working. Adding these logs revealed that igp indexing was just super slow... After waiting for 15 mins to do a full local sync based on the hyperlane context db, turns out we're actually successfully indexing igp! Keeping the log for improved troubleshooting in the future. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3372 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	dd8ac43c97	Validator task retries (#3361) ... ### Description - Makes validator tasks infallible by adding retries - This fixes a bug where certain tasks would return an error an shut down, affecting liveness. This was desired in the relayer since we don't want individual chain failures to affect the liveness of other chains. Now validator tasks either terminate or panic, and panicking will be propagated by `try_join_all`, causing the agent to shut down. - A thing to keep in mind in general is that agents will only terminate if a task panics. If it returns an `Err` but doesn't panic, the task won't be respawned. We should consider the implications of this in the scraper too. If this isn't desired, should consider using `select_all!` ### Drive-by changes - retry logic is moved from `rust/chains/hyperlane-cosmos/src/providers/rpc.rs` into `rust/hyperlane-core/src/rpc_clients/retry.rs`, so we're even closer to turning it into a retrying provider - changes several fn signatures within the validator to now return `ChainCommunicationError`s instead of `eyre::Report`s, for compatibility with the retry logic. Also makes `DbError` convertible to `ChainCommunicationError`, for the same reason. This achieves some progress on https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2878 - allows the backfill checkpoint submitter task to terminate, since `try_join_all` is tolerant of this (as described above) ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3349 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Trevor Porter	332826c5a6	Friendlier error when an agent has insufficient funds for a Cosmos tx (#3359) ... ### Description An error that will be easier to debug in the case of the signer not having sufficient funds to cover a tx that's being sent. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues Fixes #3358 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	e737d998ff	Cosmos block-by-block indexing (#3245) ... ### Description Implements cosmos block-by-block indexing, parallelizing where possible: the `block` and `block_results` calls, and the individual block queries within a range. Also reuses the existing `handle_txs` logic, to improve readability. ### Drive-by changes Adds `tokio::task::JoinError` to the `hyperlane-core` error enum ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3234 - Added some hacky retrying logic to `get_logs_in_block`, since we were seeing lots of `503` http errors. In https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 we should make sure to remove this and propagate the error instead, to retry via the cursor / indexer logic. I edited the description of 3264 to reflect this ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	9 months ago
Trevor Porter	9a83eb4d57	Sequence-aware indexing refactor (#3262) ... ### Description Part 1 of an indexing refactor. This mostly focuses on the sequence-aware cursors. Changes: - Some renames, e.g. `SequenceIndexer` -> `SequenceAwareIndexer`, `sequence_and_tip` -> `latest_sequence_count_and_tip` - Moved cursors out of `contract_sync/cursor.rs` into `contract_sync/cursors/*` - The goal was to make the the Forward sequence-aware cursor, the Backward sequence-aware, and the ForwardBackward sequence aware cursors more resilient, easier to understand, and better tested. - General strategy for doing this was to: - Only move onto a new range to index if the logs passed into the cursor's `update` function justify this - If updated with any weird data, like gaps in logs, rewind the cursor and retry previously queried ranges I'd recommend starting the review in the following order: 1. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/forward.rs` - where `ForwardSequenceAwareSyncCursor` & tests live 2. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/backward.rs` - where `BackwardSequenceAwareSyncCursor` & tests live 3. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/mod.rs` - where `ForwardBackwardSequenceAwareSyncCursor` lives ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	f11878ad08	CCIP Read metadata builder fix (#3246) ... ### Description `H160`'s `ToString` implementation adds ellipsis points instead of outputting the full hex string, for some reason (e.g. `0xc66a…7b6f`). This will result in an error when `Mailbox.process` is called on the CCIP Read ISM. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	3cbb06a07a	feat: fetch chain-specific metrics in standalone task (#3214) ... ### Description Spawning a tokio task to fetch chain-specific metrics in provider middleware was wasteful because we ended up with 15 tasks doing the same job. Now there is a single task that is spawned inside the relayer, which I added in the struct that used to only fetch relayer balance. Some questions: - I'm realising in the current state I'm probably removing these metrics from validators. Lmk if yes and I'll add back - the chain-specific metrics were moved out of `MiddlewareMetrics` because they're no longer part of middleware, but I wasn't sure where to place them. Maybe `CoreMetrics` is a better place than the current `custom_metrics.rs` file? `tokio-metrics` turned out not to be useful because we'd need to instrument every call site of `tokio::spawn` with it. We should still do it but as a separate task imo. ### Drive-by changes - This PR also makes it very easy to add the metrics tasks for new chains, by extending the `HyperlaneProvider` trait with a `get_chain_metrics` call which seems reasonably general to me to have implemented by all providers. (Bc we currently only track these for evm chains) - the description, naming and logic of the `gas_price` metric was also changed to support new chains ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3047 ### Backward compatibility Yes ### Testing Still need to manually test	10 months ago
Yorke Rhodes	0ff3e2b171	Deploy injective and inEVM (#3140) ... ### Description - Deploy core to injective and inEVM networks ### Drive-by changes - Update cw-hyperlane dependency ### Backward compatibility Yes ### Testing - Fork tests - e2e tests --------- Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	10 months ago
Daniel Savu	f2a0e92c7f	Cosmos grpc fallbackprovider (#3139) ... ### Description Implements grpc fallback provider logic for cosmos - initially tried implementing the fallback provider deprioritization logic at middleware level like in the EVM. The difference between ethers and cosmrs is that in the latter, middleware can only live at the transport layer (`tower` crate level). - based on this github [issue](https://github.com/hyperium/tonic/issues/733), that actually doesn't look possible, because the http::Request type isn't `Clone` so it can't be submitted to multiple providers - ended up implementing the fallback provider at the application layer, by keeping an array of grpc channels - There is now a `call` method in `hyperlane_core::FallbackProvider` which I'm actually really happy with. This method handles the fallbackprovider-specific logic by taking in an async closure, running it on each provider, and iterating providers if the closure call fails. In `grpc.rs` you can see how this is slightly verbose but I think it's quite manageable. The only part that bugs me is having to duplicate `Pin::from(Box::from(future))`, but that's need afaict because the regular closure returns an anonymous type - adds `grpcUrls` and `customGrpcUrls` config items - tests the cosmos fallback provider e2e ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes: https://github.com/hyperlane-xyz/issues/issues/998 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Huulu	3ea5dd7b66	chore(rust): proofreading (#3152) ... Came across few typos while browsing ./rust. One of them is an error message displayed to the user, rest are comments. Hope I could be of any help. Have a nice week-end Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com>	10 months ago
Trevor Porter	3bb4d87dda	Injective / inevm agent code & config changes (#3185) ... ### Description Taken from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3155 but without any of the ancillary things like changes to the SDK or infra. Merging to get an image off `main` that works w/ injective. Branch plan is: - after this is merged, an image off main can be used to run an agent against injective/inevm - https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3140 will have all the SDK changes from the contract deployments - https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3155 will have the infra changes to start deploying injective / inevm validators / relayer ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	c9ef41c109	Fallback Provider Refactor (#3127) ... ### Description - Generalizes the fallback provider. Now any provider type can be used in it, as long as it implements `Into<Box<dyn BlockNumberGetter>> + Clone` (aka it can be wrapped into a struct that reads the current block #, and it can be cloned). - moves the fallback provider implementation to hyperlane-core - had serious issues with getting versions to successfully resolve - next time I run into issues, strongly inclined to create a new crate with dependencies for all chains, but which are only depended on by the agents codebase (rather than the contract as well). Chatted with @tkporter about this. - now fallbackprovider needs to be wrapped in a newtype struct for chain-specific trait implementations - If `Deref` is implemented for the newtype struct, all the function calls will still work (although using Deref for newtypes is [debatable](https://users.rust-lang.org/t/understanding-the-perils-of-deref/47958/18)) - the `request` function required by the `JsonRpcClient` still has some reusable logic but I left it there until it's clearer what can be reused by the cosmos fallback provider - the fallback provider unit tests still live in hyperlane-ethereum because of the logic in `require` ### Drive-by changes - needed to use `async-rwlock` instead of `tokio` for the `RwLock` structure used by the fallback provider for interior mutability. This is because the solana program requirements had rust pinned too far into the past for `tokio` to resolve to a version. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/892 ### Backward compatibility Yes ### Testing Unit tests that were already there for the fallback provider	10 months ago
Trevor Porter	0cad4f7407	Support cosmos contract byte lengths other than 32 (#3147) ... ### Description fixes #3143 Initially went down a path that would use protocol-specific types for addresses as suggested by #3143. I had made a `HyperlaneConnectionConf` enum with protocol-specific variants whose values were `addresses: CoreContractAddresses<ProtocolSpecificAddressType>` and `connection: ProtocolSpecificConnectionConf`. This worked pretty well until I hit the ISM logic. Because hyperlane-core is where the Mailbox trait is defined, the return type of `recipient_ism` in the trait cannot involve protocol specific types. It can't be moved to hyperlane-base because hyperlane-base imports the chain crates, so we'd have a cyclic dependency. I experimented with moving away from H256 to something like a Vec<u8> or string, but this felt a bit weird. In the end we decided to keep H256s as the global representation for contract addresses for now, with the intent of eventually changing this, and to support the varying length situation in a cosmos config ### Drive-by changes - Added some cosmos specific agent configurations into the sdk - Moved to bech32_prefix in the agents for consistency with what the SDK's chain metadata already does - I guess no one's ran cargo test in a while so vectors/message.json got a new v3 message lol ### Related issues Fixes #3143 ### Backward compatibility Changes prefix to bech32_prefix in the agent config, and now requires `contractAddressBytes` ### Testing Tested merged with #3144 and all worked --------- Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com>	10 months ago
Trevor Porter	5b4af6bf1d	Injective-specific account info protobuf (#3144) ... ### Description Only change needed was to correctly handle the account info. Because we may want to handle injective testnet, added `is_injective` fn to HyperlaneDomain instead of just doing the match directly. Happy to import some of the longer fully qualified names instead ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	80f1a5688e	fix: support TLS gRPC endpoints (#3122) ... ### Description Supporting TLS gRPC endpoints is as simple as enabling the `tonic` crate features in this PR. However to maintain compatiblity with the plaintext endpoints, we need to always use `http` instead of `https`, otherwise `tonic` will try to establish a TLS connection to them and fail. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/893 ### Backward compatibility No, in the sense that all prior plaintext urls need to be changed from `https` to `http` ### Testing Manual	11 months ago
Daniel Savu	cdbaf9ef02	fix: dedup eth logs (#3108) ... ### Description Deduplicates ethereum logs by collecting them into a HashSet before returning a Vec. I couldn't find a way to deduplicate using an ethers config, but since `fetch_logs` is only called once in hyperlane-base, I'm doing the deduplication there "globally". I also noted in doc comments that duplicates may be returned. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3070 ### Backward compatibility Yes ### Testing Will rely on e2e passing but haven't added unit tests for this scenario	11 months ago
Trevor Porter	3f88aa6f6e	Observability of validators for relayers (#3057) ... ### Description Goal of this was to have insight into validators of important sets being "up" Introduces a new metric used by relayers: `hyperlane_observed_validator_latest_index`, e.g.: ``` hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test1",hyperlane_baselib_version="0.1.0",origin="test2",validator="0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc"} 664 hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test1",hyperlane_baselib_version="0.1.0",origin="test3",validator="0x976ea74026e726554db657fa54763abd0c3a0aa9"} 641 hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test2",hyperlane_baselib_version="0.1.0",origin="test1",validator="0x15d34aaf54267db7d7c367839aaf71a00a2c6a65"} 670 hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test2",hyperlane_baselib_version="0.1.0",origin="test3",validator="0x976ea74026e726554db657fa54763abd0c3a0aa9"} 665 hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test3",hyperlane_baselib_version="0.1.0",origin="test1",validator="0x15d34aaf54267db7d7c367839aaf71a00a2c6a65"} 652 hyperlane_observed_validator_latest_index{agent="relayer",app_context="default_ism",destination="test3",hyperlane_baselib_version="0.1.0",origin="test2",validator="0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc"} 664 hyperlane_observed_validator_latest_index{agent="relayer",app_context="testapp",destination="test1",hyperlane_baselib_version="0.1.0",origin="test2",validator="0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc"} 658 hyperlane_observed_validator_latest_index{agent="relayer",app_context="testapp",destination="test1",hyperlane_baselib_version="0.1.0",origin="test3",validator="0x976ea74026e726554db657fa54763abd0c3a0aa9"} 641 ``` Tapping into metadata building for multisig ISMs, the relayer will update the metric with the latest indices for the validators in a set. In order to prevent the cardinality being ridiculously high, only certain validator sets are tracked. This is done by introducing an `app_context` label (I'm very open to other names here, for some reason whenever idk how to name some kind of identifier I end up calling it a context 😆) The app context can either be: - if a new setting, --metricAppContexts, is specified, a message will be classified based off the first matching list it matches. E.g. `--metricAppContexts '[{"name": "testapp", "matchingList": [{"recipient_address": "0xd84379ceae14aa33c123af12424a37803f885889", "destination_domain": 13371 }] }]'`. This is nice for e.g. warp route deployments, where the ISM is maybe not a default ISM, and can be changed - if a message doesn't get classified this way, it can also be classified with the "default_ism" app context, which is just for any message that happens to use the default ISM as its "root" ISM This way we have insight in to the default ISM and any application-specific ISMs. Some things to note: - it's possible for a message to actually have more than one validator set, e.g. if it's using an aggregation ISM. In this case, we'll have metrics on the union of all validator sets for that app context - Some effort is required to make sure that metrics don't stick around for a validator that has actually been removed from the set. To handle this, we cache the validator set for an app context and clear out the entire set each time we set the metrics ### Drive-by changes - Zod's nonempty function for strings is deprecated, moves to `.min(1)` instead ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/1762 ### Backward compatibility yes ### Testing Ran locally - I think i'll probably add something in e2e tests, but opening now	11 months ago
Daniel Savu	c2cf7be8f2	feat(cosmos): ism dry-run (#3077) ... ### Description Implements `dry_run_verify` for the aggregation ISM on cosmwasm. One remaining issue is that the estimated gas is hardcoded to `1`, because we're actually [just querying](37fea49429/contracts/isms/aggregate/src/lib.rs (L108)) via rpc rather than simulating a tx. The `verify` tx isn't marked as a contract [entrypoint](https://book.cosmwasm.com/basics/entry-points.html) so it can't be called from outside iiuc. (here's the [verify](37fea49429/contracts/isms/aggregate/src/lib.rs (L124)) fn for reference). Worth mentioning that the query interface for the aggregation ISM is named incorrectly - it should return fields called `threshold` and `modules`, but instead copies the response from the multisig ISM and returns `threshold` and `validators`. This can be particularly misleading because validators have 20-bytes long addresses, whereas modules (contracts) have 32-bytes long addresses. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/807 ### Backward compatibility yes ### Testing E2E. The ISM setup is `routing` -> `aggregation (1/1)` -> `multisig (1/1)`	12 months ago
Daniel Savu	d18f7ae8e1	Cosmos gas price config (#3042) ... ### Description Adds a cosmos-specific config item for setting the minimum gas price, in the format returned by the `cosmos.base.node.v1beta1.Service/Config` cosmos-sdk grpc endpoint. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/810 ### Backward compatibility No. Will break existing cosmos configs. ### Testing None yet, will test in e2e	12 months ago
Daniel Savu	8e44bc1b81	feat: sealevel merkle indexer + e2e (#3053) ... Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3048 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3046 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2754	12 months ago
Daniel Savu	86b7f98850	fix: cosmos balance parsing (#3036) ... @tkporter reported that cosmos balance strings were being interpreted as hex when parsed, causing them to appear much larger. This fixes by parsing as decimal. Tested manually by checking the metrics endpoint.	12 months ago
Daniel Savu	98bb228a96	Sealevel balance metrics (#3025) ... Publishes sealevel balance as a relayer metric **Drive-by Changes** - Creates `HyperlaneSealevelError`, a `hyperlane-sealevel` specific type to avoid including sealevel errors in `hyperlane-core`'s `ChainCommunicationError`. It's pretty empty now, but we have a starting point now. This follows the approach used in `hyperlane-cosmos` too. - Includes an rpc provider instance in `SealevelProvider`, and uses it to add `get_balance` on `SealevelProvider`. **Testing** None, as sealevel e2e is currently disabled	1 year ago
Daniel Savu	77aa58c581	Relayer balance metrics (#2976) ... Done: - scaffolding for fetching custom agent metrics - abstractions for building a metrics fetcher for a given VM - querying cosmos balances; e2e tested. - querying evm balances; e2e tested. - **Note that as a result, evm addresses are now no longer zero-padded when printed in the logs. This may break existing log queries** - fixed a nasty bug on ubuntu where wasmd (osmosisd dependency, part of the grpc query flow) would panic when a block is specified via `x-cosmos-block-height`. The fix was to bump the version of osmosisd from `19.0.0` to `20.5.0`. **Note that when running e2e on Mac OS, the osmosis version in use is still 19.0.0**. That's because we need a fork that publishes a darwin target binary (currently pointing [here](https://github.com/hashableric/osmosis/releases/download/v19.0.0-mnts/osmosisd-19.0.0-mnts-darwin-arm64.tar.gz)) For follow up PR: - sealevel balance querying I'm open to all renaming suggestions, I just tried to speed through and didn't ponder names too much Relates to https://github.com/hyperlane-xyz/issues/issues/701 Closes https://github.com/hyperlane-xyz/issues/issues/702 (because the balance becomes available in the metrics endpoint for polling)	1 year ago