hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Daniel Savu	b72c881a93	feat: alert instead of crashing relayer when building cursors (#4811 ) ### Description - we should make sure there are no `unwrap`s outside a `tokio::spawn`, otherwise they'll take down the entire relayer. The offenders have so far been the indexing cursors, so this PR just logs an error and sets a new metric flag instead of panicking: `critical_error`, an `IntGauge` type because prometheus doesn't have boolean gauges. For now this is only used for origin chains, but I envision this as being set when a critical error occurs in the submitter too, in the future - before setting the flag, cursor instantiation is now retried 10 times (to avoid short lived RPC hiccups) ### Drive-by changes Removes usage of `Builder` derive macro in metric types, since they're not complex enough to warrant it ### Related issues ### Backward compatibility Yes ### Testing the existing e2e	3 weeks ago
Danil Nemirovsky	14e4719f00	feat: Use H512 as transaction hash in HyperlaneProvider and Scraper (#4759 ) ### Description Use H512 as transaction hash in HyperlaneProvider and Scraper. ### Related issues - Contributes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4272 ### Backward compatibility Yes ### Testing Manual run of Scraper for Ethereum and Neutron --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	1 month ago
Daniel Savu	efd438f9b2	chore: fix queue metric juggedness (#4689 ) ### Description See https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4068 for the problem description. In this fix, whenever an operation is moved from one queue to another, its metric count is decremented from the old queue and incremented for the new one. My initial implementation approach was to update these metrics inside `queue.push(op)`, but the metrics for the operation's previous queue aren't accessible there. #4068 suggests updating them in `op.set_status`, which can't be done for the same reason, even if `op` has a pointer to the current queue's metric internally. So the fix I went for does store a pointer to the current queue metric internally in `op`, but also adds a new `op.set_status_and_update_metrics(status, new_queue_metric)` method, which must be used if the queue metrics are to be correctly calculated. This works well except for when ops are removed from the confirm queue, because in the absence of a call to `set_status_and_update_metrics`, no metric decrementing is done. I considered using the `Drop` trait to decrement, but it'd have to be implemented individually for each `PendingOperation` type, which isn't very maintainable. I ended up decrementing the metric in `confirm_operation`, which is called for both batches and single submissions and, of course, all implementations of `PendingOperation`. Here's a screenshot of my local grafana server showing no jaggedness in the e2e run, with prometheus configured to scrape every 2s: ![Screenshot 2024-10-15 at 17 26 56](https://github.com/user-attachments/assets/26004e0e-2ccf-4cec-aa23-ee2d032df25a) ### Drive-by changes Adds the `prepare_queue` arg of `submit_single_operation` to the `instrument(skip(...))` list so it no longer pollutes logs. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4068 ### Backward compatibility Yes ### Testing Manually, by checking the queue length metric of an e2e run in grafana	1 month ago
Danil Nemirovsky	e490e77445	feat: Generate Cosmos address for Injective (#4618 ) ### Description Injective uses different logic compared to Neutron and Osmosis to generate account addresses from public key. It is implemented in this PR so that sender address is correctly generated from signer info of transaction. Depending on the type of public key communicated in the transaction, we shall generate either Bitcoin-like or Ethereum-like address for Injective. We can also specify the type of account id for the signer. Bitcoin-like will be chosen by default, so, we don't need to change the configuration of Relayer. ### Drive-by changes Made some errors to be warnings. We'll add metrics to measure how frequent the issues are and decide how to fix them: 1. transaction contains multiple contract execution messages 2. transaction contains fees in unsupported denominations Made Scraper to refrain from recording transaction which it cannot parse properly. ### Backward compatibility Yes ### Testing Added unit tests Manual testing with Injective blockchain --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	2 months ago
Daniel Savu	8809f37271	feat: checkpoint syncer fraud flag (#4587 ) Upon detection of a merkle root mismatch between the local merkle tree (built from indexed events) and the onchain one, reorg diagnostic data is posted to checkpoint storage. This PR focuses on the submitter logic and leaves two things for a follow up PR: - adding logic to not spin up the validator if a fraud flag has been posted - integration testing with S3 and GCP I assume rolling out to validators will happen after the follow up PR, but running with logic from this PR will already mean the reorg diagnostic flag is posted to S3 (though not tested).	2 months ago
Paul Balaji	3c819c777e	revert: "feat: Generate cosmos account addresses for Injective" (#4615 ) Reverts hyperlane-xyz/hyperlane-monorepo#4576 context: https://discord.com/channels/935678348330434570/1291354470730436680/1291357041624875010 > The problem is that Injective supports both "normal cosmos" signing with the bitcoin style address, and also ethereum style addresses. They disambiguate depending on the type of the pubkey that's signed in the tx > The relayer signs things using a bitcoin style address on Injective bc this was easier than making code changes to have it do ethereum-style (which isn't supported easily out of the box with the cosmos libs we use) > So we want the relayer account balance metric to look at the bitcoin-style address relating to our relayer key, not the Ethereum-style one at least this is the theory behind what happened	2 months ago
Danil Nemirovsky	6e787a9b39	feat: Generate cosmos account addresses for Injective (#4576 ) ### Description Injective uses different logic on generating account addresses from public key. It is implemented in this PR so that sender address is correctly generated from signer info of transaction. ### Backward compatibility Yes ### Testing Added unit tests Manual testing with Injective blockchain --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	2 months ago
Daniel Savu	0744137f38	feat: v3 Jito submission (#3429 ) (#4528 ) Ports this v2 fix to v3 https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3434 originally authored by @tkporter --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	2 months ago
Danil Nemirovsky	040f7efd7f	chore: Remove pined dependencies on cargo-platform and predicates-core (#4502 ) ### Description Remove pined dependencies on cargo-platform and predicates-core since the pins are not needed. ### Backward compatibility Yes ### Testing Run E2E Tests Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	2 months ago
Mantas-M	096aeff13e	feat: Splitting `rust` workspace into `main` and `sealevel` (#4438 ) ### Description The single rust workspace has been split into 2 workspaces. The sealevel workspace includes the contents of the previous `sealevel` folder and the main workspace includes everything else. The workspaces differ in rust versions as the sealevel has the old one to ensure compatability, the main workspace has and updated `rustc` version of `1.80.1`. Some dependencies and sub-dependencies have been locked in versions of the old pre-split workspace to ensure no breaking code changes. The rust CI workflow has been updated to include both workspaces in the tests. ### Drive-by changes N/A ### Related issues Relates to https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4361 ### Backward compatibility No. ### Testing Rust Unit tests from the CI	3 months ago
Danil Nemirovsky	b6cea10b63	feat: Add scraper to Cosmos E2E test (#4387 ) ### Description Add scraper to Cosmos E2E test * Add minimal functionality to scraper for cosmos so that scraper does not crash * Add scraper to Cosmos E2E test ### Backward compatibility Yes ### Testing E2E Tests for Cosmos --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 months ago
Daniel Savu	2909e81a55	feat(sealevel): charge protocol fees (#4355 ) ### Description Adds protocol fees to the Sealevel implementation, by embedding the logic / custody into the mailbox program & PDA. The logic is very similar to what already exists in the IGP implementation. The instructions to interact with the protocol fees implementation are: - `Init` (now requires specifying the `protocol_fee` and `beneficiary` in addition to the other mailbox params) - `ClaimProtocolFees` - invokable by anyone, and sending accumulated lamports to the `beneficiary` - `SetProtocolFeeConfig` - invokable by the owner, sets new values for `protocol_fee` and / or `beneficiary` ### Drive-by changes - The restriction applied on max message size has been removed to mirror the Solidity V3 implementation ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4356 ### Backward compatibility No - requires redeploying the mailbox ### Testing Sealevel program integration tests, as is done for the other programs	3 months ago
Danil Nemirovsky	d1bf212bf7	feat: Report git commit hash as agent metadata (#4282 ) ### Description * We are storing git commit hash as an environment variable when we compile the source code of an agent. * Validator uses git commit hash as a agent version in `metadata_latest.json` file. * File `metadata_latest.json` is overwritten every time when validator is restarted. * Git commit hash is extracted by `git` util, so, it should be available during build time. * If the directory with the source code does not contain `.git` folder, git commit hash will be defaulted to `VERGEN_IDEMPOTENT_OUTPUT`. Agent Metadata is propagated to Relayer and Scraper, but is not used there. We can use it in the future, reporting git hash commit in the logs, for example. ### Backward compatibility Yes ### Testing * Tested locally with E2E Tests. * Tested against S3 by deploying docker image into k8s --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	4 months ago
Daniel Savu	b643fba90a	feat: use multiple `mpsc` channels instead of `broadcast` for async interface (#4190 ) ### Description Follow up to https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4180 Lowers txid broadcast channel capacity from 1M to 30k and refactors it to use a vec of `mpsc` under the hood. ### Context The heap profiler showed that each new chain takes up 100MB of memory because of the txid broadcast channel. In addition to this, the tokio broadcast channel is a ring buffer where new items simply overwrite old items, making it hard to safely lower the capacity. As such, to be able to safely lower the capacity of this channel, usage of `tokio`'s `broadcast` channel is replaced with a `Vec` of `mpsc`s, which offer an async `send` interface for backpressure. So instead of overwriting old items, senders will block on sends, which makes sense in our case as we absolutely don't want to miss items. The submitter is slower than the indexer based on empirical observation, so blocking on channel size doesn't bottleneck throughput. I noticed the max number of channel items we can reach is 29k, by running the RC relayer after an 8 day break, during which around 160k messages were submitted. Lowering the capacity to 30k should cause us to only encounter backpressure in very rare cases, and the footprint of this channel would lower from 100MB to 3MB. ### Drive-by changes The delivery indexer in the scraper is no longer passed a txid receiver, since message deliveries are not expected to occur in the same tx as message dispatches. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Testing E2E, by adding invariants to make sure txid indexing works --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	4 months ago
Daniel Savu	6140e6f397	feat: Relayer heap profiling (#4180 ) ### Description During the scalability audit we've identified relayer memory usage as a cause of concern. This PR runs the profiler each time e2e is run and outputs a report called `dhat-heap.json` in the current dir. That file can be visualised with a dhat viewer, such as the one hosted by the author [here](https://nnethercote.github.io/dh_view/dh_view.html). It adds a conditionally compiled heap memory profiler ([dhat-rs](https://docs.rs/dhat/latest/dhat/)), which is only included when both the `memory-profiling` feature is enabled and the `memory-profiling` cargo profile is used. The report can order the profiling results by total lifetime allocated bytes, peak allocated bytes, total allocation count, etc. Below is the result of sorting by total lifetime allocated bytes, which reveals that the biggest memory hogs are the hook indexing channel [senders](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/hyperlane-base/src/contract_sync/mod.rs#L52) (one per origin chain), taking up 100+ MB. Its size can most likely be reduced from 1M to 1k. ![Screenshot 2024-07-23 at 14 32 21](https://github.com/user-attachments/assets/f551ebfe-3ee0-40de-b53f-e5de1bc56dfa) ### Drive-by changes Adds the remaining IDs in the channel as a field to the hook indexing log, to know if it's safe to lower the channel size. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Backward compatibility Yes, since everything is conditionally compiled / isolated in a new cargo profile. ### Testing Manual	4 months ago
Daniel Savu	97588b6fc1	fix(relayer): don't submit reverting messages in batches (#4169 ) ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4159 Some messages in a Multicall would revert because previous messages would invalidate them temporarily. The IGP allowance of those messages would be deducted, and if this happened more than once, those messages would become stuck. This PR fixes those unnecesary spends on reverting `process` txs by using `.call()` to simulate the outcome of a Multicall tx. Reverting messages are removed from the batch and submitted serially, and if the batch is only left with one non-reverting message, even that one is submitted serially (rather than through a batch of one message). Messages that did get submitted have their igp allowance deducted and status updated in `update_sent_ops_state`. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1294 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing Tested by manually setting the result of `call` to failed for all messages, to see if they are submitted serially. In prod, we can track the `Submitted transaction batch` log, which now includes any excluded messages from the batch.	4 months ago
Daniel Savu	a5e5f70610	chore: Revert "feat(hyp-ethereum): gas escalator middleware (#3852 )" (#4098 ) This reverts commit `c3c002a8fe`. ### Description The gas escalator took up to 28gb of memory, and even though it was only ever released on RC, it must have taken up too many cluster resources, ending up starving the `hyperlane` relayer. Reverting for now until we have a better idea ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Daniel Savu	8c8f39abfe	feat(relayer): ListOperations api endpoint (#4089 ) ### Description Adds a `list_operations` API endpoint to the relayer, which returns all operations in its prepare queue. Example output from `GET http://0.0.0.0:9092/list_operations?destination_domain=13371` ``` [ { "app_context": null, "message": { "body": [ 18, 52 ], "destination": 13371, "nonce": 1, "origin": 13373, "recipient": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "sender": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "version": 3 }, "num_retries": 13, "status": { "Retry": "CouldNotFetchMetadata" }, "submitted": false, "type": "PendingMessage" }, { "app_context": null, "message": { "body": [ 18, 52 ], "destination": 13371, "nonce": 2, "origin": 13372, "recipient": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "sender": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "version": 3 }, "num_retries": 13, "status": { "Retry": "CouldNotFetchMetadata" }, "submitted": false, "type": "PendingMessage" } ] ``` ### Drive-by changes Makes `PendingOperation` serializable, which means switching from JSON format to e.g. CSV (for easily filtering / aggregating in excel) should be very easy. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Manual and Unit Tests	5 months ago
Daniel Savu	c3c002a8fe	feat(hyp-ethereum): gas escalator middleware (#3852 ) ### Description Wraps evm providers in a gas escalator middleware as part of the `build_with_signer` step, so signed txs can have their gas price bumped if stuck in the mempool. Gas price is increased every 60s by 12.5%, and is capped at the `max_fee_per_gas` set in tx overrides. ### Drive-by changes - changes to our fork of `ethers`: - removes the `M: Clone` bound, since it's not actually required and conflicts with the middleware we use - increases the level of the `escalated` log in the middleware to `debug` (so we can see it working in prod without enabling `trace`) ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2959 ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	5 months ago
Daniel Savu	aecb65a986	feat: persist status labels (#4043 ) ### Description Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4003 to persist updates to an operation's status to the db, so we no longer default to `FirstPrepareAttempt` upon restarting the relayer ### Drive-by changes Moves `hyperlane_db` from `hyperlane_base` to `hyperlane_core` so it can be added to the `PendingOperation` trait. `PendingMessage` already stores a reference to the db in `ctx`, so I figured it doesn't break any assumptions. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Unit tests encoding of `PendingOperationStatus`, need to manually tests e2e	5 months ago
Trevor Porter	a109937f96	chore: some throughput tweaks (#4036 ) ### Description Context: https://discord.com/channels/935678348330434570/1254036693217054794 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Daniel Savu	826b4ae57f	feat: batching igp (#3870 ) ### Description Deducts IGP cost from senders of messages within a batch, based on the total gas used by the batch and the estimates for submitting messages individually. The formula is `gas_used_by_operation = gas_used_by_tx * (operation_estimated_gas / total_estimated_cost)` Note that the individual estimate have a buffer value added to them (currently 75k), which will slightly skew proportions, though by a negligible amount. For example, given these estimates in a batch: 150k, 160k, 180k, if we add 80k to each, we go from e.g. 0.312 for the first message to 0.319 -> 2.25% error which isn't so bad. The error can be larger if operations within a batch have very different estimates, but realistically within a 5% range based on back-of-the-napkin calculations. ### Drive-by changes - The batching feature introduced an bug whereby gas expenditure would only be deducted in the `confirm` step. Now this is done in the `submit` step to account for txs that revert - Sealevel e2e can now be disabled for faster iteration! I've not done the cleanest job, but even the fact that we have this will reduce the time to run e2e locally by more than half. To use it, set `SEALEVEL_ENABLED=false` when running `run-locally` - e2e uses a new utility called `get_matching_lines`, that can be used to count (and in the future _parse_) logs, to reconstruct the state of the relayer and have more expressive correctness checks. This is used to make sure that gas is deducted for all messages, including those in batches. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3709 ### Backward compatibility Yes ### Testing E2E, using `get_matching_lines`	6 months ago
Yorke Rhodes	49f41d9759	chore: Merge branch 'main' into cli-2.0 (#3863 ) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Connor McEwen <connor.mcewen@gmail.com> Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Avi Atkin <103125634+avious00@users.noreply.github.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Ali Alaoui <aalaoui2001@gmail.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz> Co-authored-by: omahs <73983677+omahs@users.noreply.github.com>	6 months ago
Daniel Savu	214f503e53	feat: forward-backward message processor (#3775 ) ### Description - adds logic for iterating message nonces in the processor task in forward-backward fashion - removes all manual calls that were updating the next nonce to query. This is done automatically in the iterator struct now - stores the highest known message nonce in the db, which is used to initialize the iterator ### Drive-by changes - Converts the concrete HyperlaneDb type stored in the processor to a trait obj, to enable mocking db responses ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3796 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3816 ### Backward compatibility Yes - if there is no db entry for the new prefix, the processor starts from nonce zero, so no migration is required ### Testing Unit testing + e2e	6 months ago
Daniel Savu	27aabf238c	feat: Relayer tokio task instrumentation (#3760 ) ### Description - Started off adding tokio-metrics but then realised those are quite general, so while we do have instrumentation it's not exposed in our metrics endpoint - switched to adding [tokio-console](https://github.com/tokio-rs/console/tree/main), which does give insight into the lifetime of specific tasks, so we can check which ones take up a long time during relayer startup. These are only visible at the `dependencyTrace` log level, so don't affect performance in the `hyperlane` context. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Helps debug https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3454 and any future performance issues - Does half the work for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3239 (still need to expose these in the metrics endpoint and import the grafana template) ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	5aeb83a2e5	Relayer Message Batching to EVM (#3656 ) ### Description Status: - [x] op-queue scaffolding - [x] hyperlane-ethereum integration with multicall - [x] new config items - [x] actually build batch in the op-queue submit step - [x] e2e tests with batching configured - [x] dynamic batch sizes (batch as many messages as the prepare step has ready) - [x] parallelize metadata building - [ ] proportional IGP deduction ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes #3628 - Fixes #3630 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	7 months ago
Daniel Savu	9b11f327c5	Range retries in watermark cursor (#3500 ) ### Description Changes the logic in the watermark cursor, to update the range in the `update` method rather than in `get_next_range`. This means that RPC errors won't result in ranges being skipped, so IGP indexing should be more reliable. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - First in a sequence of many, for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Daniel Savu	6e39cf01af	Relayer retry endpoints (#3476 ) ### Description Adds a `/message_retry` relayer endpoint, which supports retrying OpQueue operations either by ID or by destination domain. Such a request will be sent to an `mpmc` channel's receiving end in the OpQueue, which is emptied when `OpQueue::pop` is called. Example calls: ``` GET http://127.0.0.1:60843/message_retry?destination_domain=42 GET http://127.0.0.1:60843/message_retry?message_id=0x46910b1329ee53c86a023b322e9ca1c17e5f9f0bee789c77b0abced0a173d714 ``` If the endpoint is called specifying both filters, two requests will be sent across the channel, one for each condition. Efficiency note: The entire queue is iterated over if there's at least one retry request in the channel. The good thing is that rebuilding the heap is `O(n)` so not too bad. ### Drive-by changes - Removed the usage of `enum_dispatch`, because it would have made unit tests messy. The drawback is that we have to work around object safety restrictions in `PendingOperation` - for instance it can't be cloned (but can be stored behind an `Arc` if we ever need this). - Added new methods to `PendingOperation`, either to match by ID, or to implement heap element prioritization without depending on the concrete `PendingMessage` type (as was done before). Additions: `id()`, `origin_domain()`, `priority()`. Don't think this conflicts with any operations we may add in the future (e.g. gas oracle updates). - Although `tokio`'s broadcast channel is multi-producer-multi-consumer by default, the consumer end isn't `Clone` - you instead need to have a producer to call `.subscribe()` on to get a new consumer, so I added an `MpmcChannel` struct to encapsulate keeping a producer around to get new consumers. - OpQueue is moved into its own file and has unit tests added for retries (covering broadcast to multi-queue, and both retry types). - Did a bit of agent `server `cleanup, including `EigenNodeApi` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3098 ### Backward compatibility Yes ### Testing Unit tests for the server route (channel transmitter logic) and for the OpQueue retries (channel receiver logic).	8 months ago
Daniel Savu	dd8ac43c97	Validator task retries (#3361 ) ### Description - Makes validator tasks infallible by adding retries - This fixes a bug where certain tasks would return an error an shut down, affecting liveness. This was desired in the relayer since we don't want individual chain failures to affect the liveness of other chains. Now validator tasks either terminate or panic, and panicking will be propagated by `try_join_all`, causing the agent to shut down. - A thing to keep in mind in general is that agents will only terminate if a task panics. If it returns an `Err` but doesn't panic, the task won't be respawned. We should consider the implications of this in the scraper too. If this isn't desired, should consider using `select_all!` ### Drive-by changes - retry logic is moved from `rust/chains/hyperlane-cosmos/src/providers/rpc.rs` into `rust/hyperlane-core/src/rpc_clients/retry.rs`, so we're even closer to turning it into a retrying provider - changes several fn signatures within the validator to now return `ChainCommunicationError`s instead of `eyre::Report`s, for compatibility with the retry logic. Also makes `DbError` convertible to `ChainCommunicationError`, for the same reason. This achieves some progress on https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2878 - allows the backfill checkpoint submitter task to terminate, since `try_join_all` is tolerant of this (as described above) ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3349 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	e737d998ff	Cosmos block-by-block indexing (#3245 ) ### Description Implements cosmos block-by-block indexing, parallelizing where possible: the `block` and `block_results` calls, and the individual block queries within a range. Also reuses the existing `handle_txs` logic, to improve readability. ### Drive-by changes Adds `tokio::task::JoinError` to the `hyperlane-core` error enum ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3234 - Added some hacky retrying logic to `get_logs_in_block`, since we were seeing lots of `503` http errors. In https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 we should make sure to remove this and propagate the error instead, to retry via the cursor / indexer logic. I edited the description of 3264 to reflect this ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	9 months ago
Trevor Porter	9a83eb4d57	Sequence-aware indexing refactor (#3262 ) ### Description Part 1 of an indexing refactor. This mostly focuses on the sequence-aware cursors. Changes: - Some renames, e.g. `SequenceIndexer` -> `SequenceAwareIndexer`, `sequence_and_tip` -> `latest_sequence_count_and_tip` - Moved cursors out of `contract_sync/cursor.rs` into `contract_sync/cursors/*` - The goal was to make the the Forward sequence-aware cursor, the Backward sequence-aware, and the ForwardBackward sequence aware cursors more resilient, easier to understand, and better tested. - General strategy for doing this was to: - Only move onto a new range to index if the logs passed into the cursor's `update` function justify this - If updated with any weird data, like gaps in logs, rewind the cursor and retry previously queried ranges I'd recommend starting the review in the following order: 1. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/forward.rs` - where `ForwardSequenceAwareSyncCursor` & tests live 2. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/backward.rs` - where `BackwardSequenceAwareSyncCursor` & tests live 3. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/mod.rs` - where `ForwardBackwardSequenceAwareSyncCursor` lives ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Kunal Arora	13fb461888	feat:add eigen node endpoints to agent API (#3095 ) ### Description - compatibility with eigenlayer's node spec here: https://eigen.nethermind.io/docs/spec/api/ - changing from hyper to axum as a preferred choice of server handling - adding server trait to validator, relayer, and server (to customize behavior for each in the future) ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - fixes https://github.com/hyperlane-xyz/issues/issues/853 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	155c85eda2	Infallible relayer tasks (#3257 ) ### Description The intended behavior of agents is described by @tkporter's comment here: https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3257#discussion_r1489233847 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3209 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	3cbb06a07a	feat: fetch chain-specific metrics in standalone task (#3214 ) ### Description Spawning a tokio task to fetch chain-specific metrics in provider middleware was wasteful because we ended up with 15 tasks doing the same job. Now there is a single task that is spawned inside the relayer, which I added in the struct that used to only fetch relayer balance. Some questions: - I'm realising in the current state I'm probably removing these metrics from validators. Lmk if yes and I'll add back - the chain-specific metrics were moved out of `MiddlewareMetrics` because they're no longer part of middleware, but I wasn't sure where to place them. Maybe `CoreMetrics` is a better place than the current `custom_metrics.rs` file? `tokio-metrics` turned out not to be useful because we'd need to instrument every call site of `tokio::spawn` with it. We should still do it but as a separate task imo. ### Drive-by changes - This PR also makes it very easy to add the metrics tasks for new chains, by extending the `HyperlaneProvider` trait with a `get_chain_metrics` call which seems reasonably general to me to have implemented by all providers. (Bc we currently only track these for evm chains) - the description, naming and logic of the `gas_price` metric was also changed to support new chains ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3047 ### Backward compatibility Yes ### Testing Still need to manually test	10 months ago
Ivan Temchenko	65e2ea50f9	Google Cloud Storage CheckpointSyncer implementation (#3156 ) ### Description GSC builder and implementer classes; Documentation comments; Examples; Anonymous test on public bucket access; ### Drive-by changes `cargo doc` build fixes: * replaced square braces of non-existing items with `'` ; * raw URL surrounded by `<>`; ### Related issues Fixes #2242 Part 1 ### Backward compatibility Yes ### Testing Unit Tests	10 months ago
Yorke Rhodes	0ff3e2b171	Deploy injective and inEVM (#3140 ) ### Description - Deploy core to injective and inEVM networks ### Drive-by changes - Update cw-hyperlane dependency ### Backward compatibility Yes ### Testing - Fork tests - e2e tests --------- Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	10 months ago
Daniel Savu	f2a0e92c7f	Cosmos grpc fallbackprovider (#3139 ) ### Description Implements grpc fallback provider logic for cosmos - initially tried implementing the fallback provider deprioritization logic at middleware level like in the EVM. The difference between ethers and cosmrs is that in the latter, middleware can only live at the transport layer (`tower` crate level). - based on this github [issue](https://github.com/hyperium/tonic/issues/733), that actually doesn't look possible, because the http::Request type isn't `Clone` so it can't be submitted to multiple providers - ended up implementing the fallback provider at the application layer, by keeping an array of grpc channels - There is now a `call` method in `hyperlane_core::FallbackProvider` which I'm actually really happy with. This method handles the fallbackprovider-specific logic by taking in an async closure, running it on each provider, and iterating providers if the closure call fails. In `grpc.rs` you can see how this is slightly verbose but I think it's quite manageable. The only part that bugs me is having to duplicate `Pin::from(Box::from(future))`, but that's need afaict because the regular closure returns an anonymous type - adds `grpcUrls` and `customGrpcUrls` config items - tests the cosmos fallback provider e2e ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes: https://github.com/hyperlane-xyz/issues/issues/998 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	c9ef41c109	Fallback Provider Refactor (#3127 ) ### Description - Generalizes the fallback provider. Now any provider type can be used in it, as long as it implements `Into<Box<dyn BlockNumberGetter>> + Clone` (aka it can be wrapped into a struct that reads the current block #, and it can be cloned). - moves the fallback provider implementation to hyperlane-core - had serious issues with getting versions to successfully resolve - next time I run into issues, strongly inclined to create a new crate with dependencies for all chains, but which are only depended on by the agents codebase (rather than the contract as well). Chatted with @tkporter about this. - now fallbackprovider needs to be wrapped in a newtype struct for chain-specific trait implementations - If `Deref` is implemented for the newtype struct, all the function calls will still work (although using Deref for newtypes is [debatable](https://users.rust-lang.org/t/understanding-the-perils-of-deref/47958/18)) - the `request` function required by the `JsonRpcClient` still has some reusable logic but I left it there until it's clearer what can be reused by the cosmos fallback provider - the fallback provider unit tests still live in hyperlane-ethereum because of the logic in `require` ### Drive-by changes - needed to use `async-rwlock` instead of `tokio` for the `RwLock` structure used by the fallback provider for interior mutability. This is because the solana program requirements had rust pinned too far into the past for `tokio` to resolve to a version. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/892 ### Backward compatibility Yes ### Testing Unit tests that were already there for the fallback provider	10 months ago
Daniel Savu	528a19022b	Deploy eclipse testnet (#3172 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	10 months ago
Trevor Porter	5b4af6bf1d	Injective-specific account info protobuf (#3144 ) ### Description Only change needed was to correctly handle the account info. Because we may want to handle injective testnet, added `is_injective` fn to HyperlaneDomain instead of just doing the match directly. Happy to import some of the longer fully qualified names instead ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	80f1a5688e	fix: support TLS gRPC endpoints (#3122 ) ### Description Supporting TLS gRPC endpoints is as simple as enabling the `tonic` crate features in this PR. However to maintain compatiblity with the plaintext endpoints, we need to always use `http` instead of `https`, otherwise `tonic` will try to establish a TLS connection to them and fail. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/893 ### Backward compatibility No, in the sense that all prior plaintext urls need to be changed from `https` to `http` ### Testing Manual	11 months ago
Daniel Savu	d18f7ae8e1	Cosmos gas price config (#3042 ) ### Description Adds a cosmos-specific config item for setting the minimum gas price, in the format returned by the `cosmos.base.node.v1beta1.Service/Config` cosmos-sdk grpc endpoint. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/810 ### Backward compatibility No. Will break existing cosmos configs. ### Testing None yet, will test in e2e	12 months ago
Daniel Savu	8e44bc1b81	feat: sealevel merkle indexer + e2e (#3053 ) Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3048 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3046 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2754	12 months ago
Daniel Savu	42ea929785	Sealevel v3 contracts (#3048 ) ### Description MVP contracts for sealevel v3 - note this doesn't include hooks so IGP payments still have to be made separately. However contracts will now be compatible with the metadata format used by the other V3 deployments. ### Drive-by changes The unit tests in `rust/sealevel/programs/ism/multisig-ism-message-id/src/metadata.rs` are refactored to reuse struct creation logic. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3044 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3045 ### Backward compatibility No, since this upgrades sealevel v2 to v3. ### Testing Unit tests	12 months ago
Daniel Savu	77aa58c581	Relayer balance metrics (#2976 ) Done: - scaffolding for fetching custom agent metrics - abstractions for building a metrics fetcher for a given VM - querying cosmos balances; e2e tested. - querying evm balances; e2e tested. - Note that as a result, evm addresses are now no longer zero-padded when printed in the logs. This may break existing log queries - fixed a nasty bug on ubuntu where wasmd (osmosisd dependency, part of the grpc query flow) would panic when a block is specified via `x-cosmos-block-height`. The fix was to bump the version of osmosisd from `19.0.0` to `20.5.0`. Note that when running e2e on Mac OS, the osmosis version in use is still 19.0.0. That's because we need a fork that publishes a darwin target binary (currently pointing [here](https://github.com/hashableric/osmosis/releases/download/v19.0.0-mnts/osmosisd-19.0.0-mnts-darwin-arm64.tar.gz)) For follow up PR: - sealevel balance querying I'm open to all renaming suggestions, I just tried to speed through and didn't ponder names too much Relates to https://github.com/hyperlane-xyz/issues/issues/701 Closes https://github.com/hyperlane-xyz/issues/issues/702 (because the balance becomes available in the metrics endpoint for polling)	1 year ago
Trevor Porter	ffd4de4110	Use new ethers-rs version that should work with BSC (#3000 ) ### Description Uses a version of our ethers fork with https://github.com/hyperlane-xyz/ethers-rs/pull/10, which adds back a change that I had reverted from https://github.com/hyperlane-xyz/ethers-rs/pull/8 This is required to fix bsc / bsctestnet txs, where the base fee is 0 and we want to estimate the priority fee ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility ye ### Testing deployed and saw successful bsc / bsctestnet msgs	1 year ago
Trevor Porter	c2288eb317	Update ethers-rs with polygon gas oracle fix (#2998 ) ### Description Update to latest ethers-rs (forked) version with https://github.com/hyperlane-xyz/ethers-rs/pull/9 ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility n/a ### Testing Planning to deploy with it, but the ethers-rs version is well tested	1 year ago
Trevor Porter	9b8ad55a6a	Update ethers-rs to use lower max priority fee (#2986 ) ### Description Cherry-picked the commit from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2903, which is now out of date ### Drive-by changes n/a ### Related issues n/a ### Backward compatibility yes ### Testing we're running the relayer with an image off of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2903 Co-authored-by: -f <kunalarora1729@gmail.com>	1 year ago
Trevor Porter	ece2be507a	Cosmos / CosmWasm agents (#2865 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: hashableric <hashableric@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz>	1 year ago
Yorke Rhodes	1d18549755	Merge main to v3 (#2812 )	1 year ago

10 Commits (fa424826c0132c135511e5316a6e559b574b3a8f)