hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Danil Nemirovsky	57346defe3	refactor: Refactor Hyperlane store for Scraper (#4855 ) ### Description Refactor Hyperlane store for Scraper. Just moved source code into submodules. ### Backward compatibility Yes ### Testing E2E test Ethereum and Sealevel --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 weeks ago
Trevor Porter	4f8245a1ed	chore: apply 10% gas buffer for Arbitrum Nitro txs, some drivebys (#4854 ) ### Description - Applies a 10% buffer to arbitrum nitro txs, see https://discord.com/channels/935678348330434570/1306264065110310987 for context ### Drive-by changes - Some drive-bys -- imo the Display impl of HyperlaneMessage isn't useful at all and should be the same as Debug (or we should stop using the Display impl) - Updates a sealevel comment that was inaccurate as flagged by some partners - makes sure that relayer pods update when there's a configmap change ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 weeks ago
Danil Nemirovsky	8e50c2af21	feat: Remove unique chain id index from db migration script (#4834 ) ### Description Remove unique chain id index from db migration script ### Related issues - Contributes into https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4798 ### Backward compatibility Yes ### Testing Manual run of DB migration script --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	4 weeks ago
Daniel Savu	b72c881a93	feat: alert instead of crashing relayer when building cursors (#4811 ) ### Description - we should make sure there are no `unwrap`s outside a `tokio::spawn`, otherwise they'll take down the entire relayer. The offenders have so far been the indexing cursors, so this PR just logs an error and sets a new metric flag instead of panicking: `critical_error`, an `IntGauge` type because prometheus doesn't have boolean gauges. For now this is only used for origin chains, but I envision this as being set when a critical error occurs in the submitter too, in the future - before setting the flag, cursor instantiation is now retried 10 times (to avoid short lived RPC hiccups) ### Drive-by changes Removes usage of `Builder` derive macro in metric types, since they're not complex enough to warrant it ### Related issues ### Backward compatibility Yes ### Testing the existing e2e	4 weeks ago
Danil Nemirovsky	38bd1ae49f	feat: Scraper stores input data from Ethereum transactions (#4794 ) ### Description Scraper stores input data from Ethereum transactions ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4778 ### Backward compatibility Yes ### Testing Manual run of Scraper against 1. local database 2. database restored from backup of production database Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	1 month ago
Mantas-M	fc82b30a2e	feat: Refactor Relayer Retry Server (#4693 ) ### Description Refactor of the Relayer Retry Server to utilize matching lists. This allows for a single POST request to be able to retry a wide range of messages. Since a matching list is used it is now possible to retry messages depending on: - Message Id - Origin Domain - Destination Domain - Recipient Address - Sender Address ## Example Instead of the old request format ``` GET http://127.0.0.1:60843/message_retry?destination_domain=42 GET http://127.0.0.1:60843/message_retry?message_id=0x46910b1329ee53c86a023b322e9ca1c17e5f9f0bee789c77b0abced0a173d714 ``` The new one should be used ``` POST http://127.0.0.1:60843/message_retry ``` With a JSON body like so: ``` [{"messageid": "", "origindomain": "", "senderaddress": "", "destinationdomain": "", "recipientaddress": "*"}] ``` Retry params can be both specific values and wildcards. ### Drive-by changes - Moved `MockPendingOperation` to `hyperlane-test` - Moved certain test utils for the relayer server to `hyperlane-test` - Added functions to retrieve recipient and sender address from the `PendingOperation` - Added a `message_id` to the `MatchingList` struct to allow retries by message id - Removed `MessageRetryRequest` enum since it was no longer in use ### Related issues Fixes #3571 #4079 ### Backward compatibility Yes ### Testing Unit tests	1 month ago
Danil Nemirovsky	14e4719f00	feat: Use H512 as transaction hash in HyperlaneProvider and Scraper (#4759 ) ### Description Use H512 as transaction hash in HyperlaneProvider and Scraper. ### Related issues - Contributes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4272 ### Backward compatibility Yes ### Testing Manual run of Scraper for Ethereum and Neutron --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	1 month ago
Danil Nemirovsky	ffbe1dd82e	feat: Make HyperlaneProvider to request block by height, not by hash (#4727 ) ### Description Make HyperlaneProvider to request block by height, not by hash. Since it is hard to find a block by hash for Solana, we switch to use block height. We have to do it for all type of chains since we need to change signature of chain-agnostic method ### Drive-by changes * Small method rename ### Related issues - Contributes into https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4272 ### Backward compatibility Yes ### Testing Manual run of Scraper for Ethereum and Neutron chains. --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	1 month ago
Paul Balaji	7f3e0669d2	feat: add stride to relayed chains (#4716 ) ### Description - feat: add stride to relayer - feat: add stride to scraper ### Drive-by changes - fix filtering non-evm addresses in appFromAddressesMapHelper - fallback to manual `yarn install` in `yarn-build-with-cache` action - fix bug in app governor to make inevm env-test pass ### Related issues na ### Backward compatibility yes ### Testing manual --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	1 month ago
Alexander Pastushenka	c3e9268f1e	feat: Add block tag support in `reorgPeriod` parameter (#4629 ) ### Description <!-- What's included in this PR? --> Allows setting an arbitrary string in `reorgPeriod`, which is used as a block tag to get the finalized block. Currently, only the Ethereum connector supports this feature with the following tags: - finalized - safe - pending - latest - earliest This is currently useful for EVM-compatible chains that don't have a fixed reorganization period, but instead rely on the block tag `Finalized` to indicate finality. For such chains, you should set `reorgPeriod` to `finalized` instead of a number of blocks. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> Yes ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com>	1 month ago
Trevor Porter	013d3211c0	fix: use hyp_message as tracing field instead of message (#4721 ) ### Description https://docs.rs/tracing/latest/tracing/#shorthand-macros > unlike other fields, `message`'s shorthand initialization is just the string itself. When using tracing::info etc, passing in an explicit `message` field takes precedence over the message string that's supplied as the second positional param! So we end up in situations where we see `message: HyperlaneMessage { id: ....` instead of the log message we want! e.g. https://cloudlogging.app.goo.gl/Nr8Q5W5KviD6sT8W9 Open to other field names, like `msg` or something, but this felt explicit ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	1 month ago
Daniel Savu	777ef084a6	fix: queue length metric bug (#4691 ) ### Description Fix for https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4689, where `Dropped` status operations weren't decremented from the metric. I noticed there's still a small difference on a handful of chains, such as celo, where even if Dropped ones are considered, the prep queue metric reports an extra 53 ops compared to the actual queue. Could be that I manually miscounted, I expect releasing this will be a big improvement anyway. ### Backward compatibility Yes ### Testing No, just checked all places where we `.pop_many` but don't `push`	2 months ago
Daniel Savu	efd438f9b2	chore: fix queue metric juggedness (#4689 ) ### Description See https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4068 for the problem description. In this fix, whenever an operation is moved from one queue to another, its metric count is decremented from the old queue and incremented for the new one. My initial implementation approach was to update these metrics inside `queue.push(op)`, but the metrics for the operation's previous queue aren't accessible there. #4068 suggests updating them in `op.set_status`, which can't be done for the same reason, even if `op` has a pointer to the current queue's metric internally. So the fix I went for does store a pointer to the current queue metric internally in `op`, but also adds a new `op.set_status_and_update_metrics(status, new_queue_metric)` method, which must be used if the queue metrics are to be correctly calculated. This works well except for when ops are removed from the confirm queue, because in the absence of a call to `set_status_and_update_metrics`, no metric decrementing is done. I considered using the `Drop` trait to decrement, but it'd have to be implemented individually for each `PendingOperation` type, which isn't very maintainable. I ended up decrementing the metric in `confirm_operation`, which is called for both batches and single submissions and, of course, all implementations of `PendingOperation`. Here's a screenshot of my local grafana server showing no jaggedness in the e2e run, with prometheus configured to scrape every 2s: ![Screenshot 2024-10-15 at 17 26 56](https://github.com/user-attachments/assets/26004e0e-2ccf-4cec-aa23-ee2d032df25a) ### Drive-by changes Adds the `prepare_queue` arg of `submit_single_operation` to the `instrument(skip(...))` list so it no longer pollutes logs. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4068 ### Backward compatibility Yes ### Testing Manually, by checking the queue length metric of an e2e run in grafana	2 months ago
Danil Nemirovsky	bb82b1cc01	feat: Store as much as possible from every range (#4660 ) ### Description Currently, if Scraper cannot fetch and parse a block or a transaction, it will fail the whole chunk of logs. It means that a transaction which can be fetched and parsed, but happens to be in the same chunk as unparseable transaction, won't be inserted into database. This PR fixes it. Scraper will do its best to insert as many as possible blocks and transactions into database from the chunk. ### Related issues https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4632 ### Backward compatibility Yes ### Testing Local run of E2E tests. Local run of Scraper --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	2 months ago
Daniel Savu	7dccf80b57	feat: do not start up validator if fraud was reported (#4627 ) ### Description Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4587 Adds a check in the `CheckpointSubmitter` builder which emits a panic if a reorg flag was posted, since this potentially signals fraud ### Drive-by changes `CheckpointSubmitter::build` was made private, so any use of it is now replaced by the `build_and_submit` flow ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Unit Tests - which is enough as long as `CheckpointSubmitter::build` is kept private	2 months ago
Daniel Savu	8809f37271	feat: checkpoint syncer fraud flag (#4587 ) Upon detection of a merkle root mismatch between the local merkle tree (built from indexed events) and the onchain one, reorg diagnostic data is posted to checkpoint storage. This PR focuses on the submitter logic and leaves two things for a follow up PR: - adding logic to not spin up the validator if a fraud flag has been posted - integration testing with S3 and GCP I assume rolling out to validators will happen after the follow up PR, but running with logic from this PR will already mean the reorg diagnostic flag is posted to S3 (though not tested).	2 months ago
Daniel Savu	e2b5a4c785	feat: HyperlaneDb trait (#4609 ) ### Description Defines a `HyperlaneDb` trait for use in the validator submitter, so a mock can be passed in to simulate indexed data from reorged state. Although the `HyperlaneDb` trait can't be `Clone` to be object safe, as long as we use `Arc`s everywhere, all will be ok. Eventually the aim is to replace all direct usage of `HyperlaneRocksDb` with `Arc<dyn HyperlaneDb>`.	2 months ago
Daniel Savu	4f92871c2b	feat: include `id` field in `list_operations` endpoint (#4610 ) ### Description The `id` field was missing if the serialized `PendingOperation` was a `PendingMessage`, which made it hard to look up the message in relayer logs or the explorer. This PR takes advantage of `id()` being a required method on `PendingOperation` to derive and display the ID as part of endpoint logic. ### Testing Unit Tests	2 months ago
Daniel Savu	bf53f087c3	fix(relayer): dry-run before sending txs (#4565 ) ### Description Fixes a bug in the individual submission logic. There is a race condition where a message can become undeliverable between when it was simulated in the `prepare` step and when it's actually submitted. ### Context The bug was discovered as part of batched messages but affects chains without batching too. - when delivery fails inside batch submission simulation, we fall back to individual submission - individual submission doesn't simulate the tx outcome, and instead just submits directly - this is wrong, because the batch may cause messages to become undeliverable for long periods of time (such as when a mint cap is reached) - I believe simulation used to happen in the individual submission step, but we switched to using the already computed gas estimate from the IGP check in the `prepare` step, to save an RPC call - this results in senders being charged from their IGP allowance unnecessarily, and messages becoming undeliverable due to insufficient gas The correct flow requires three delivery simulations: 1. in the `prepare` step, to estimate gas and check the IGP policy 2. in the batch submission logic, to filter messages that are made invalid by earlier messages in the batch 3. right before submitting again, in case the batch caused the message to not be deliverable ### Solution Dry-run right before submitting an operation, and if it fails send it back to the prep queue	2 months ago
Daniel Savu	96ee728788	feat: turbo eth warp route (#4514 )	3 months ago
Mantas-M	096aeff13e	feat: Splitting `rust` workspace into `main` and `sealevel` (#4438 ) ### Description The single rust workspace has been split into 2 workspaces. The sealevel workspace includes the contents of the previous `sealevel` folder and the main workspace includes everything else. The workspaces differ in rust versions as the sealevel has the old one to ensure compatability, the main workspace has and updated `rustc` version of `1.80.1`. Some dependencies and sub-dependencies have been locked in versions of the old pre-split workspace to ensure no breaking code changes. The rust CI workflow has been updated to include both workspaces in the tests. ### Drive-by changes N/A ### Related issues Relates to https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4361 ### Backward compatibility No. ### Testing Rust Unit tests from the CI	3 months ago

21 Commits (0410b4e4b1e8d4043ffa6ae04802f352befa22fc)