hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Trevor Porter	57bbb16cc1	feat: panics upon startup when RPC issues occur include the offending chain (#4323 ) ### Description Some quick low hanging fruit to make debugging easier in the future. Today (https://discord.com/channels/935678348330434570/1273582453889699843) some chains were having RPC issues and it caused an agent deploy to panic upon startup. Debugging this was hard because the panic message just indicates an error is occurring, but doesn't say what the offending chain is! Examples: - https://cloudlogging.app.goo.gl/puQk7hm4fXyJhYKs8 - https://cloudlogging.app.goo.gl/5qZrbc2Ww4LWQXWC6 - https://cloudlogging.app.goo.gl/ZL7mKbBzzJaePtty9 All these panic happen when constructing the cursor, which isn't yet done in an indexing task. The quick and easy fix is to just use expect instead of unwrap We still unwrap in a few places, but mostly in places that aren't susceptible to flaky RPC issues, e.g. reading from a HashMap that we know should always have an entry, etc ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Trevor Porter	beab39c43c	chore: update sei gas price, temporarily stop deploying to osmosis (#4321 ) ### Description Following https://seitrace.com/proposal/83?chain=pacific-1, need a higher gas price ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Paul Balaji	279516f280	chore: enroll worldchain, xlayer, cheesechain, zircuit validators (#4313 ) - chore: enroll worldchain, xlayer, cheesechain, zircuit validators - fix: update cosmos zod schema - drive-by: update core checker to check if the ism in registry matches the expected config	4 months ago
Trevor Porter	9db5871f27	fix: use correct Neutron IGP (#4303 ) ### Description - We've been operating Neutron warp routes with a whitelist so far. It seems like in the initial shuffle of things, we configured the agents to index an IGP that was actually replaced 😬 by another one - This starts indexing the new one. Because of watermark indexing, we won't backfill old IGP payments, because from its perspective it thinks it's already indexed those. I wonder if we should tie watermark indexing cursors to the IGP address? Either way, this is probably fine as to date only a couple non-warp messages from Neutron have occurred ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Paul Balaji	d274c6b3f8	chore: update registry to v2.5.0 (#4302 ) chore: update registry to v2.5.0	4 months ago
Paul Balaji	8be75b6569	feat: core deploy to new chains (#4281 ) Deploy to new chains: 1. cyber 1. degenchain 1. kroma 1. lisk 1. lukso 1. merlin 1. metis 1. mint 1. proofofplay 1. real 1. sanko 1. tangle 1. xai drive-by: - fix bug in `set-rpc-urls-from-registry.ts` - add script to generate sql to add multiple chains to scraper db	4 months ago
Paul Balaji	2d922805c0	feat: add arbitrumsepolia, basesepolia, ecotestnet, optimismsepolia, polygonamoy to core chains (#4297 ) Deploy to arbitrumsepolia, basesepolia, ecotestnet, optimismsepolia, polygonamoy	4 months ago
Danil Nemirovsky	d1bf212bf7	feat: Report git commit hash as agent metadata (#4282 ) ### Description * We are storing git commit hash as an environment variable when we compile the source code of an agent. * Validator uses git commit hash as a agent version in `metadata_latest.json` file. * File `metadata_latest.json` is overwritten every time when validator is restarted. * Git commit hash is extracted by `git` util, so, it should be available during build time. * If the directory with the source code does not contain `.git` folder, git commit hash will be defaulted to `VERGEN_IDEMPOTENT_OUTPUT`. Agent Metadata is propagated to Relayer and Scraper, but is not used there. We can use it in the future, reporting git hash commit in the logs, for example. ### Backward compatibility Yes ### Testing * Tested locally with E2E Tests. * Tested against S3 by deploying docker image into k8s --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	4 months ago
Paul Balaji	7625bf8dce	fix: typo in trait builder log (#4292 ) - fix: typo in trait builder log - noticed while looking at scraper logs	4 months ago
Paul Balaji	103e6b3e12	feat: deploy to `zircuit` (#4265 ) - feat: deploy to `zircuit` - update registry + agent config	4 months ago
Daniel Savu	a49f52cc9c	feat: deploy solana, eclipsemainnet (#4238 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: pbio <10051819+paulbalaji@users.noreply.github.com>	4 months ago
Danil Nemirovsky	21da4861d0	docs: Add hyperlane registry as prerequisite for e2e tests (#4263 ) ### Description E2E tests will fail if `hyperlane-registry` repo is not cloned next to the monorepo. This change adds it to the `rust/README.md` file. ### Testing Checked how `rust/README.md` looks like in GitHub. --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com> Co-authored-by: Trevor Porter <tkporter4@gmail.com>	4 months ago
Paul Balaji	ff3c9897ea	chore: migrate `test1` from `13371` to `133711` (#4252 ) - chore: migrate `test1` from `13371` to `133711` - due to clash with immutablezkevm - drive-by registry update + agent config update --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	4 months ago
Paul Balaji	c6bf71d1b3	chore: update registry to v2.4.0 (#4247 ) chore: update registry to v2.4.0 - pinned CI version updated to `d9fea91f16949a51b2c8e776bac26712ba1fef3d`, which includes some testnet updates after 2.4.0 (https://github.com/hyperlane-xyz/hyperlane-registry/pull/126) --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	4 months ago
Trevor Porter	fe9365fdb3	chore: agent metric log tweaks (#4244 ) ### Description - some small tweaks as I looked into a recent issue with metrics ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Mohammed Hussan	8e942d3c6b	feat: deploy cheesechain, worldchain, xlayer (#4207 ) ### Description - Support deployment to cheesechain, worldchain and zlayer ### Drive-by changes - update `CosmosChainSchema` to add `chain_type` to fix breaking change introduced by `6854133f98`	4 months ago
Daniel Savu	b643fba90a	feat: use multiple `mpsc` channels instead of `broadcast` for async interface (#4190 ) ### Description Follow up to https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4180 Lowers txid broadcast channel capacity from 1M to 30k and refactors it to use a vec of `mpsc` under the hood. ### Context The heap profiler showed that each new chain takes up 100MB of memory because of the txid broadcast channel. In addition to this, the tokio broadcast channel is a ring buffer where new items simply overwrite old items, making it hard to safely lower the capacity. As such, to be able to safely lower the capacity of this channel, usage of `tokio`'s `broadcast` channel is replaced with a `Vec` of `mpsc`s, which offer an async `send` interface for backpressure. So instead of overwriting old items, senders will block on sends, which makes sense in our case as we absolutely don't want to miss items. The submitter is slower than the indexer based on empirical observation, so blocking on channel size doesn't bottleneck throughput. I noticed the max number of channel items we can reach is 29k, by running the RC relayer after an 8 day break, during which around 160k messages were submitted. Lowering the capacity to 30k should cause us to only encounter backpressure in very rare cases, and the footprint of this channel would lower from 100MB to 3MB. ### Drive-by changes The delivery indexer in the scraper is no longer passed a txid receiver, since message deliveries are not expected to occur in the same tx as message dispatches. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Testing E2E, by adding invariants to make sure txid indexing works --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	4 months ago
Daniel Savu	d864c6a939	fix(evm): only create signer provider when needed (#4208 ) ### Description Adds an associated const to all ethereum provider builders to specify whether a signer is needed. For an e2e run with 20 messages, we now have 12 providers built with signers at the beginning (3 of which are validator announce), and 708 providers built without a signer. This should hopefully fix the gas escalator memory leak. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Daniel Savu	6140e6f397	feat: Relayer heap profiling (#4180 ) ### Description During the scalability audit we've identified relayer memory usage as a cause of concern. This PR runs the profiler each time e2e is run and outputs a report called `dhat-heap.json` in the current dir. That file can be visualised with a dhat viewer, such as the one hosted by the author [here](https://nnethercote.github.io/dh_view/dh_view.html). It adds a conditionally compiled heap memory profiler ([dhat-rs](https://docs.rs/dhat/latest/dhat/)), which is only included when both the `memory-profiling` feature is enabled and the `memory-profiling` cargo profile is used. The report can order the profiling results by total lifetime allocated bytes, peak allocated bytes, total allocation count, etc. Below is the result of sorting by total lifetime allocated bytes, which reveals that the biggest memory hogs are the hook indexing channel [senders](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/hyperlane-base/src/contract_sync/mod.rs#L52) (one per origin chain), taking up 100+ MB. Its size can most likely be reduced from 1M to 1k. ![Screenshot 2024-07-23 at 14 32 21](https://github.com/user-attachments/assets/f551ebfe-3ee0-40de-b53f-e5de1bc56dfa) ### Drive-by changes Adds the remaining IDs in the channel as a field to the hook indexing log, to know if it's safe to lower the channel size. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Backward compatibility Yes, since everything is conditionally compiled / isolated in a new cargo profile. ### Testing Manual	4 months ago
Daniel Savu	97588b6fc1	fix(relayer): don't submit reverting messages in batches (#4169 ) ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4159 Some messages in a Multicall would revert because previous messages would invalidate them temporarily. The IGP allowance of those messages would be deducted, and if this happened more than once, those messages would become stuck. This PR fixes those unnecesary spends on reverting `process` txs by using `.call()` to simulate the outcome of a Multicall tx. Reverting messages are removed from the batch and submitted serially, and if the batch is only left with one non-reverting message, even that one is submitted serially (rather than through a batch of one message). Messages that did get submitted have their igp allowance deducted and status updated in `update_sent_ops_state`. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1294 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing Tested by manually setting the result of `call` to failed for all messages, to see if they are submitted serially. In prod, we can track the `Submitted transaction batch` log, which now includes any excluded messages from the batch.	4 months ago
Daniel Savu	bb16dfa728	fix: more conservative batching gas estimates (#4159 ) ### Description Estimating gas isn't very accurate for Multicall txs that allow failures. This PR now estimates gas as: `max(batch_tx_estimate, individual_tx_estimates_sum)`, where `individual_tx_estimates_sum` estimates each individual tx and adds a conservative overhead that `Multicall.aggregate3` adds per subtx. The aggregate3 overhead is calcualted based on some observed tenderly gas profiling results as follows. - For [4 txs](https://dashboard.tenderly.co/shared/simulation/63e85ac7-3ea9-475c-8218-a7c1dd508366/gas-usage): - aggregate3 gas: 805,160 - process costs: 290,091 + 168,711 + 168,711 + 168,711 = 796,224 - overhead: 805,160 - 796,224 = 8,936 => 2,234 per subtx - For [2 txs](https://dashboard.tenderly.co/tx/arbitrum/0xad644e431dc53c3fc0a074a749d118ff5517346c3f28d8e2513610cc9ab5c91a/gas-usage): - aggregate3 gas: 201,710 - process costs: 103,011 + 92,512 = 195,523 - overhead: 201,710 - 195,523 = 6,187 => 3,093 per subtx - So will assume an overhead of 3500 `aggregate3` gas per message in batch ### Drive-by changes This slightly changes the semantics of gas overrides. If a gas limit is configured, it will only guarantee that that's the minimum limit a tx can have. Now we use the `max(estimated_gas, gas_limit_override)` as the actual gas limit. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1295 ### Backward compatibility Yes ### Testing I think it's ok to release to release and check the new log, to see the individual estimate vs the batch estimate and double-check which one is picked in the tx explorer	4 months ago
Daniel Savu	bf1dad7f77	feat(relayer): new status label for IGP not found (#4106 ) ### Description Adds new status label to relayer opqueue for operations whose gas payments were not found. This now creates a distinction between operations without a payment and those with insufficient payment. Helps better uncover cases where hook indexing misses events because there is no receipt yet	4 months ago
Paul Balaji	0dedbf5a02	feat: deploy to new chains (endurance, fuse, zora) (#4148 ) - feat: deploy to new chains (endurance, fuse, zora) - https://github.com/hyperlane-xyz/issues/issues/1308 - also includes https://github.com/hyperlane-xyz/issues/issues/1305 - registry PR https://github.com/hyperlane-xyz/hyperlane-registry/pull/104	5 months ago
Trevor Porter	dcd6dc502e	chore: remove static plumetestnet gas price (#4135 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Paul Balaji	2c0ae3cf37	feat: deploy to new testnets (connextsepolia, superpositiontestnet) (#4150 ) - feat: deploy to new testnets (connextsepolia, superpositiontestnet) - https://github.com/hyperlane-xyz/issues/issues/1308 - also includes https://github.com/hyperlane-xyz/issues/issues/1305 - registry pr https://github.com/hyperlane-xyz/hyperlane-registry/pull/105	5 months ago
Paul Balaji	f83b492deb	feat: implement `update()` for `EvmHookModule` (#3888 ) resolves https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3577 - enables updates of hooks - includes functionality for updating IGP hook and gas oracles - drive-by fix to ism module TODO: - [x] tests (non-igp) - [x] tests (igp) --------- Signed-off-by: Paul Balaji <paul@hyperlane.xyz> Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com> Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Paul Balaji	0a40dcb8b1	fix(sdk): update cosmos chain schema (#4142 ) - update cosmos chain schema to match latest spec `c6cfe11be2/chain.schema.json (L451-L460)` - drive-by fixing of agg test --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Caio	4ec872c573	chore: Reorder variants of `KnownHyperlaneDomain` (#4083 ) ## Description Fix #3961	5 months ago
Daniel Savu	a5e5f70610	chore: Revert "feat(hyp-ethereum): gas escalator middleware (#3852 )" (#4098 ) This reverts commit `c3c002a8fe`. ### Description The gas escalator took up to 28gb of memory, and even though it was only ever released on RC, it must have taken up too many cluster resources, ending up starving the `hyperlane` relayer. Reverting for now until we have a better idea ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Daniel Savu	8c8f39abfe	feat(relayer): ListOperations api endpoint (#4089 ) ### Description Adds a `list_operations` API endpoint to the relayer, which returns all operations in its prepare queue. Example output from `GET http://0.0.0.0:9092/list_operations?destination_domain=13371` ``` [ { "app_context": null, "message": { "body": [ 18, 52 ], "destination": 13371, "nonce": 1, "origin": 13373, "recipient": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "sender": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "version": 3 }, "num_retries": 13, "status": { "Retry": "CouldNotFetchMetadata" }, "submitted": false, "type": "PendingMessage" }, { "app_context": null, "message": { "body": [ 18, 52 ], "destination": 13371, "nonce": 2, "origin": 13372, "recipient": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "sender": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "version": 3 }, "num_retries": 13, "status": { "Retry": "CouldNotFetchMetadata" }, "submitted": false, "type": "PendingMessage" } ] ``` ### Drive-by changes Makes `PendingOperation` serializable, which means switching from JSON format to e.g. CSV (for easily filtering / aggregating in excel) should be very easy. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Manual and Unit Tests	5 months ago
Assaf Morami	1022e3858e	chore: add shebang to build.sh to specify bash as the interpreter (#4066 ) the `[[` command isn't available on e.g. zsh Co-authored-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Paul Balaji	d316772243	feat: deploy to BOB, Mantle, Taiko (#4051 ) resolves https://github.com/hyperlane-xyz/issues/issues/1289 - feat: deploy to BOB, Mantle, Taiko - drive-by: fix deploying to specific chain - drive-by: adding AVS operators for eth - drive-by: support non-evm remotes in mainnet3 core config --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Daniel Savu	c3c002a8fe	feat(hyp-ethereum): gas escalator middleware (#3852 ) ### Description Wraps evm providers in a gas escalator middleware as part of the `build_with_signer` step, so signed txs can have their gas price bumped if stuck in the mempool. Gas price is increased every 60s by 12.5%, and is capped at the `max_fee_per_gas` set in tx overrides. ### Drive-by changes - changes to our fork of `ethers`: - removes the `M: Clone` bound, since it's not actually required and conflicts with the middleware we use - increases the level of the `escalated` log in the middleware to `debug` (so we can see it working in prod without enabling `trace`) ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2959 ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	5 months ago
Daniel Savu	d90d8f86b4	chore: reduce op-queue verbosity (#4092 ) ### Description The gcp bill has been shooting up, and it's most likely due to the increased number of undeliverable messages being added to and removed from the queue, causing lots of `debug` level logs on `push` and `pop_many` to be emitted. This PR changes the verbosity to `trace` - we'll have lower visibility over the lifetime of a message, but there should be sufficient logs remaining to tell us what's going on (such as [operation prepared](`30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L240)`), [confirming a pre-submitted one](`30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L260)`), [submitting single op](`30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L313)`), [submitting batch](`30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L456)`), and [op confirmed](`30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L386)`)). @paulbalaji suggested measuring log counts, so here they are: - in the last 6h, we had 42M `hyperlane` relayer logs, of which 22M where `push` logs and 1.1M were `pop_many` (no `pop`s) - if we make push ones trace, 5.5% of the remaining logs would be pop_many ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Trevor Porter	6f4ef05d41	feat: faster way of figuring out how many new scraper rows were inserted (#4073 ) ### Description ### Context Full context / thread: https://discord.com/channels/935678348330434570/1254871503825141923 Copied here: so this is kinda interesting basically the gas payment indexing in the scraper task is consistently prioritizing hook-based indexing work over block range indexing work this only manifests in the scraper and for arbitrum because: there are a lot of arbitrum origin messages on the scraper, each gas payment indexing tick takes a very long time because of some very suboptimal queries. store_payments can take like 10-15 seconds! You can see batches of logs Found log(s) in index range https://cloudlogging.app.goo.gl/Bo9Q7YwyziSqjsEu7 that correspond with the gas payment contract sync block height (green line) advancing https://abacusworks.grafana.net/goto/poW4KOQIg?orgId=1 ![image](https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/65da2e44-5728-4ecc-9c1f-52741805eb5b) but we're still indexing gas payments bc of hook indexing - there are tons of Found log(s) for tx id https://cloudlogging.app.goo.gl/QvDaBuf67SjjKviw6 In the scraper's store_payments we bizarrely end up performing 2 very expensive queries with the calls to payments_count - it gets the count of gas payments for the provided domain so it can figure out how many new gas payments were actually inserted https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/agents/scraper/src/db/payment.rs#L27-L64 These queries vary a bunch in how expensive they can be - sometimes each one is like 1-2 seconds, sometimes it gets up to 10 seconds each though https://cloudlogging.app.goo.gl/cDFNH3oxh6WWV3Go8 ![image](https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/612c7999-21b2-43f4-861f-8b42580e4cb7) easiest fix is probably to figure out how to make store_payments less expensive - we probably don't need to be making these calls to payments_count to figure out which ones are new which ill do atm there's maybe a decent argument to having hook indexing and range indexing not contend with one another but we should be able to kick that can also an annoying thing is https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/hyperlane-base/src/contract_sync/mod.rs#L155 this log not showing up, probably some weird instrumentation setup or something makes debugging harder ### The fix We want to keep supporting the possibility that we've double-indexed a data type, and to only return the number of new rows inserted. We want to keep each indexing task as non-conflicting as possible - we know that only one scraper is running at a time, but that a task for each chain and data type is executing concurrently. It's possible for e.g. two gas payment indexing tasks to try to insert into the same table concurrently, but it'll be for different domains. Insert statement concurrency issues aren't an issue because the only chance for conflict is the auto-incrementing primary key id, but this is all handled properly by database internals. Insert results only give the latest ID resulting from the insertion. E.g. if no matter how many rows were inserted, it gives the latest ID from that batch, but no info about how many were in the batch. We could do something like get the latest id in the table for all domains (this is cheap) before, then do the insert, and then get the latest id again, but this is susceptible to race conditions if concurrent tasks get an insertion in during that time as well. Shoving all these into a transaction doesn't seem to solve the problem either unless there's a lock on the table. However, sea-orm doesn't support table level locks, just select locks. We could also use a CTE to perform as a way to perform the select & insertion atomically and return the value of the select - but again sea-orm doesn't seem to provide a nice way of doing this from what I could tell. Instead, my suggested fix is: 1. Get the latest id in the table relating to the specific domain (this is cheap, < 100ms) 2. Perform the insertion 3. Count how many new rows for the specific domain now exist whose id is > the id from (1). This is also cheap, < 100ms. If there is a very long time since the last insertion for the domain, this could take longer, but in practice this doesn't seem to be the case. If this couple order of magnitude improvement still poses issues in the future, we can consider other alternatives, probably involving locking or fancier CTE queries. Some example queries and how long they take: Before (2x of these would occur!): <img width="477" alt="Screen Shot 2024-06-28 at 12 36 41 PM" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/5cc8ac35-74fb-4847-9c6f-448deb253a0c"> Now: 1. Max id for a given domain <img width="494" alt="Screen Shot 2024-06-28 at 12 37 00 PM" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/56dbb3ff-b540-458c-809c-56ebfbfd3b0f"> 2. Subsequent count but with an ID filter - note I actually changed this query to look for IDs > 3600000, which includes 55300 rows and it's still just 102ms! <img width="632" alt="Screen Shot 2024-06-28 at 12 37 49 PM" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/11593180-682f-4867-9867-97fd93512802"> ### Drive-by changes - Updated the neutron image along with a deploy there ### Related issues ### Backward compatibility ### Testing Ran some sorta ad-hoc unit tests to test my new query by running against the prod db with a tokio::task. Tested for a new domain and an existing one	5 months ago
Daniel Savu	befc38daba	fix: agent log verbosity (#4054 ) ### Description tbh I don't know why, but this makes all the logs we were previously missing show up in e2e: - `b99920ecca/rust/hyperlane-base/src/contract_sync/mod.rs (L155)` - `f68319f46e/rust/hyperlane-base/src/db/rocks/hyperlane_db.rs (L93)` ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3915 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing E2E, by counting occurrences of the two logs linked above in relayer log output. Before this change, those logs would be missing completely --------- Co-authored-by: Paul Balaji <paul@hyperlane.xyz>	5 months ago
Daniel Savu	aecb65a986	feat: persist status labels (#4043 ) ### Description Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4003 to persist updates to an operation's status to the db, so we no longer default to `FirstPrepareAttempt` upon restarting the relayer ### Drive-by changes Moves `hyperlane_db` from `hyperlane_base` to `hyperlane_core` so it can be added to the `PendingOperation` trait. `PendingMessage` already stores a reference to the db in `ctx`, so I figured it doesn't break any assumptions. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Unit tests encoding of `PendingOperationStatus`, need to manually tests e2e	5 months ago
Daniel Savu	d213e32341	feat: relayer queue operation labels (#4003 ) ### Description - adds status labels to operations in the relayer queues. In practice, the most useful ones are those for when a message is retried due to an error. Labels are defined as enums to keep the cardinality finite, the size of prometheus metrics could get very large - whenever a message is pushed to a queue, the interface requires specifying the new status as a `Option<PendingOperationStatus>`. If `None`, the status is left as before (such as in the case when an operation isn't ready to be retried). - a limitation is that labels aren't stored in the DB, so messages are loaded with the `FirstPrepareAttempt` label. I just triggered retries for destinations I wanted to get labels for, but we could persist to the db in a follow-up PR - Example prep queue labels for inevm: <img width="621" alt="Screenshot 2024-06-21 at 18 05 58" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/23065004/f53b2858-371f-4b6b-86d9-5c21409d641f"> ### Drive-by changes - removes `make_op_try` as it doesn't really add anything that couldn't be done with regular Rust - bumps the `strum` version because there was a bug in the previous one, preventing inner enum fields from being converted to strings ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Yes ### Testing Manual	5 months ago
Daniel Savu	718d2c27da	chore: don't reset retry count when processing retry API reqs (#4037 ) ### Description When messages are retried due to the Message Retry API, we bring messages to the front of the queue (by changing their `next_attempt_after`), but also reset their retry count. For messages that are unprocessable due to various reasons (and hence have very high `next_attempt_after`), resetting the retry count will cause the relayer to waste time instead of trying to process newer messages. This PR makes it such that the retry API will cause messages to be retried immediately once, but won't boost their priority in the queue afterwards (because `num_retries` isn't changed). ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Paul Balaji	cc0ea73360	feat: pin monorepo to a specific registry version (#4041 ) - feat: pin monorepo to a specific registry version (cherry-pick #4031) - drive-by agent config update --------- Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Trevor Porter	a109937f96	chore: some throughput tweaks (#4036 ) ### Description Context: https://discord.com/channels/935678348330434570/1254036693217054794 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Snoppy	f1fe882e3b	chore: fix typos (#3645 ) fix minor typos Signed-off-by: snoppy <michaleli@foxmail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz>	5 months ago
Paul Balaji	c5a5e264b8	chore: enable `osmosis` on relayer (#4028 ) - chore: enable `osmosis` on relayer - drive-by add default chain signer/key/type to agent config - `signer.key` is copied from `neutron` --------- Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Paul Balaji	39f66df326	feat(ci): rust caching in `test` workflow (#4021 ) - brings https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4015 to cli-2.0 branch - cherry-pick e2e flake fix - ensure tests run on pushes to `cli-2.0` --------- Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com>	5 months ago
Trevor Porter	c84a1dc73b	feat: address blacklisting in the relayer (#4000 ) ### Description See context here https://discord.com/channels/935678348330434570/1236041564673806427/1252918894755319920 and https://discord.com/channels/935678348330434570/1252953507833708605/1252954456400592997 This is a rudimentary first step toward blacklisting addresses. Future options are outlined in https://discord.com/channels/935678348330434570/1236041564673806427/1252918894755319920. Eventually we can also fetch these addresses directly within the agent from an updated list that we cache and re-fetch occasionally. For now, this just relies on configured addresses. The approach is: - Allow hex addresses to be configured. These are intentionally treated as a Vec<u8> instead of an H160 / H256. This is because treating them as H160 is a leak of Ethereum-specific logic into non-chain-specific code in the relayer, and treating them as H256 would pad Ethereum addresses with a bunch of zeroes which would diminish the accuracy of our heuristic implemented in this. The subsequence stuff is a bit unfortunate because it's O(n^2) but 🤷‍♂️ - The heuristic is as follows - if any of the configured addresses are found as a subsequence in the message sender, recipient, or body, it's thrown out by the message processor, and the message will never make its way to the op_submitter. This heuristic for the body is to prevent warp route transfers that include that address. We could maybe convert Vec<u8> to H256 for the sender and recipient, I'm open to this - I just went with the substring approach bc it felt easiest tbh. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1285 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Daniel Savu	f227eec892	fix: e2e invariant flake (#4017 ) ### Description Context is explained in the comment, copying it here: ``` // TODO: Sometimes we find more logs than expected. This may either mean that gas is deducted twice for the same message due to a bug, // or that submitting the message transaction fails for some messages. Figure out which is the case and convert this check to // strict equality. // EDIT: Having had a quick look, it seems like there are some legitimate reverts happening in the confirm step // (`Transaction attempting to process message either reverted or was reorged`) // in which case more gas expenditure logs than messages are expected. ``` ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Paul Balaji	8ce2a71f20	chore: reconcile mainnet config json with registry (#4016 ) chore: reconcile mainnet config json with registry Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Daniel Savu	3bb9d0a767	fix(cosmos): validator checkpoint & agent indexing improvements (#3958 ) - propagates RPC errors to the indexing logic, so errors cause sleeps instead of overwhelming the RPCs. This also greatly reduces cosmos log noisiness: from 9.5k logs per min, to 2.5k logs per min. - checks for published checkpoints in reverse, to prioritize recent messages among those from before the validator was spun up - fixes a validator bug where errors in checkpoint publishing would cause the entire checkpoint queue to be reiterated through, from scratch - Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3887 so we can test this out on osmosis ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Trevor Porter	bbbe93206f	chore: bump default ISM max depth from 5 to 7 (#3989 ) ### Description Needed for some Renzo messages that hit the max depth ### Drive-by changes - drive by to update the agent config with the updated registry ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Paul Balaji	b0828b3d0e	feat: reintroduce `ism read` and `hook read` commands (#3990 ) the original functionality of these commands got lost across a couple of refactors 1. `ism read`/`hook read` was merged to become `core read`, which still had the purpose of "gimme config at this address" 2. then with a recent change, `core read` became "gimme the ism/hook configs relating to this mailbox address" This PR adds the original functionality back of returning the ISM/Hook configs for any ISM/Hook address (on EVM) drive-by: - update mainnet config - fix typescript linting errors/warnings	5 months ago

1 2 3 4 5 ...

862 Commits (3449fc76b10953b6cb5f7b1ec8fb34fa139ff8a6)