hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Paul Balaji	73c232b3aa	feat: deploy to oort mainnet (#4473 ) feat: deploy to oort mainnet - registry PR https://github.com/hyperlane-xyz/hyperlane-registry/pull/190 - builds on top of the everclear PR https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4471	3 months ago
L.Y	b326d4bfce	fix: incorrect use of blacklist variable (#4099 ) ### Description Otherwise, the whitelist will not work; any whitelist parameter will function as a blocklist. Co-authored-by: Trevor Porter <tkporter4@gmail.com>	3 months ago
Roman V	7f099419e3	fix: gracefully handle missing objects for GCS checkpoints (#4453 ) ### Description The issue with the validator was that it wasn't handling the bucket's 404 response as a correct response when the object is not found when fetching a specific checkpoint or latest index, thus it was getting stuck in a loop failing to submit a checkpoint or get the latest stored index. I fixed it by catching the 404 response in `gcs_storage.rs` and returning `Ok(None)` instead of raising an error, the same way it's handled in S3 Storage. ### Drive-by changes Bumped ya-gcp to the latest 0.11.3 ### Related issues - Fixes #4451 ### Backward compatibility Yes ### Testing Manual	3 months ago
Paul Balaji	2e6176f675	feat: deploy to everclear mainnet (#4471 ) feat: deploy to everclear mainnet - registry PR https://github.com/hyperlane-xyz/hyperlane-registry/pull/187	3 months ago
Daniel Savu	a75e84af4d	chore: rm leftover printlines (#4448 ) ### Description Noticed these when I was debugging the scroll delivery issues ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Daniel Savu	d4204c6f0a	chore: revert to using a 10 min relayer `CONFIRM_DELAY` (#4452 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Yorke Rhodes	7af7431bd0	chore: Update agent config (#4443 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: pbio <10051819+paulbalaji@users.noreply.github.com>	3 months ago
Daniel Savu	3e90734310	chore: update scroll reorgperiod in agents config (#4442 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Daniel Savu	39eef5fb66	feat: solana <> eclipse with protocol fees (#4414 ) ### Description Artifacts from the core deployment, as well as the SOL warp route between Solana and Eclipse. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Paul Balaji	f30ba0efa4	fix: increase wait time for postgres to be ready in cosmos e2e (#4411 ) ## fix: increase wait time for postgres to be ready in cosmos e2e increase time waiting for postgres to be ready, to reduce this sort of flake https://github.com/hyperlane-xyz/hyperlane-monorepo/actions/runs/10665491633/job/29559169324?pr=4410#step:13:2998 based on the logs it looks like 10s is just on the border, adding an extra 5s to give headroom ## drive-by fix: disable fail-fast on e2e-matrix currently only the non-cosmwasm tests are required to merge, therefore we should not fail the entire matrix if the cosmos job fails. This also means we can test that the issues we're seeing with the `cosmwasm` job do not affect the `non-cosmwasm` one.	3 months ago
Daniel Savu	8adcd60402	chore: add `mainnet` suffix to solana and eclipse (#4410 ) ### Description Because we're doing a new core deployment that includes protocol fees ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Danil Nemirovsky	42ccf23678	feat: add error handling and refactoring (#4409 ) ### Description * Add error handling for getting transaction by hash * Add error handing for extracting contract address * Extract a method for ### Backward compatibility Yes ### Testing Cosmos E2E Testing (without invariant for scraper) --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 months ago
Danil Nemirovsky	c808e5f184	feat: add error handling for blocks and proper sender (#4406 ) ### Description * Use correct signer as sender * Add error handling in getting block by hash ### Backward compatibility Yes ### Testing E2E Test for Cosmos (no implementation in invariant) --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 months ago
Daniel Savu	b8f4ed3bfa	fix: set profile overrides in root manifest (#4402 ) ### Description Fixes `cargo build-sbf` warnings by moving all profile overrides to `rust/Cargo.toml`, so they actually take effect. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1368 ### Backward compatibility Yes ### Testing Manually, by initializing `inbox.processed_count` to `u64::MAX - 1`, and then observing logs in e2e: ``` 13: Program log: About to increment processed count. Current count: 18446744073709551615 14: Program log: panicked at 'attempt to add with overflow', sealevel/programs/mailbox/src/processor.rs:386:5 ``` The panic occurs at `inbox.processed_count += 1;`	3 months ago
Paul Balaji	70ab4b21c0	chore: update .registryrc and agent config (#4403 ) chore: update .registryrc and agent config --------- Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Trevor Porter <tkporter4@gmail.com>	3 months ago
Danil Nemirovsky	6f867a5974	feat: populate more fields in scraper for Cosmos (#4401 ) ### Description Populate gas price, nonce and contract as a recipient of a transaction (still naive handling). ### Backward compatibility Yes ### Testing E2E Test for Cosmos (no check in the invariant yet) --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 months ago
Paul Balaji	ef813b9810	chore: deduplicate verification jsons (#4392 ) - script to deduplicate verification jsons with jq - can be used as a break-glass-in-case-of-emergency option - do the deduplication - deduplicate the verification artifact before writing it back to infra landfill - drive-by fix to special-case getting `index.from` for arb nitro chains --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	3 months ago
Danil Nemirovsky	3fc6e3c7f9	refactor: Cosmos indexers (#4394 ) ### Description Refactoring Cosmos related indexers: * Separate dispatch and deliver indexers * Re-use repeated code on log parsing ### Backward compatibility Yes ### Testing E2E Test for Cosmos --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 months ago
Kunal Arora	791deab55c	fix: revert "support `IWeightedMultisigIsm` in relayer (#4360 )" (#4399 ) This reverts commit `24c8188e55`. ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Kunal Arora	24c8188e55	feat: support `IWeightedMultisigIsm` in relayer (#4360 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - fixes https://github.com/hyperlane-xyz/issues/issues/1345 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Paul Balaji	707db4a279	feat: deploy new core chains (#4386 ) Deploy new core chains: - astar - astarzkevm - bitlayer - coredao - dogechain - flare - molten - shibarium drive-by fixes: - key-funder config is only strict where necessary - update prettier ignore	3 months ago
Kunal Arora	535eeb192b	fix: remove redundant ``yarn install && yarn build`` from e2e (#4391 ) ### Description - test.yaml workflow already installs yarn dependencies and runs `yarn build` so it's redundant and wasteful to do the same in the main.rs file for run-locally which is dependent on yarn build as it is. So, I made it conditional if E2E_CI_MODE="false". yarn install ~= 19s yarn build ~= 1m33s It should save us 1m52s in the usual longest running CI item. ### Drive-by changes None ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing e2e	3 months ago
Danil Nemirovsky	b6cea10b63	feat: Add scraper to Cosmos E2E test (#4387 ) ### Description Add scraper to Cosmos E2E test * Add minimal functionality to scraper for cosmos so that scraper does not crash * Add scraper to Cosmos E2E test ### Backward compatibility Yes ### Testing E2E Tests for Cosmos --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 months ago
Daniel Savu	38a52deac2	fix: improve reliability of hook indexing (#4214 ) ### Description After deploying this, should no longer see logs like https://cloudlogging.app.goo.gl/gK4ByT1qLXTG7VN19 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4212 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Roman V	ae6cd6232a	feat: Fix GCS support: setting checkpointSyncer.type=gcs (#4382 ) ### Description Fixing the issue I described [earlier](https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3156#issuecomment-2285659584) where the usage of GCS as an alternative to AWS although supported by Hyperlane Agents, in practice didn't work because the settings still didn't accept "gcs" as a checkpoint syncer type. ### Backward compatibility Yes ### Testing Manual	3 months ago
Daniel Savu	dac2fb5210	feat: SOL (solana <> eclipse) warp route (#4268 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	3 months ago
Danil Nemirovsky	86b1c7646c	feat: Add check if an address is a contract in Cosmos (#4369 ) ### Description Currently, integration into Cosmos make an assumption that the address where a message should be delivered is a contract. This change allows Relayer to actually check if the recipient is a contract. ### Backward compatibility Yes ### Testing * Tested using E2E tests for Cosmos. * Tested using auxiliary unit test (it is marked as ignored since it connects to real gRPC of Neutron). --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 months ago
Paul Balaji	e51cd1d438	feat: misc improvements to infra deployment (#4374 ) feat: misc improvements to infra deployments These are all things that I've done somewhat ad-hoc before, but now upstreaming into the monorepo. - write the deploy plan to a file instead of stdout - to avoid cluttering terminal with 1000+ line configs - `getMultiProviderForRole` should only be populated with the requested set of chains or the supported chains for given environmen - to avoid fetching 50+ more keys than you need to on any given deploy - remove the agent config write-back from the post-deploy - this is already covered by the update-agent-config.ts script which is usually ran manually anyway - also moved the helper function into the update-agent-config.ts file, as that is the only place it is being used right now - in `infra/scripts` instead of `infra/src` to avoid a dependency issue - add `writeYamlAtPath` helper method to utils - with small refactor of existing `writeJsonAtPath` - streamline hit detection on `readCache` in legacy deployer - improve logging of legacy hook deployer when deploying an aggregation hook - better config matching detection when deploying routing hooks - extension of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4305 --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	3 months ago
Daniel Savu	feb1d390c4	chore: lower log noisiness by removing `ret` from `PendingOperation::… (#4372 ) …perpare` ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Paul Balaji	240d24d06d	chore: add zircuit safe address (#4340 ) - chore: add zircuit safe address - drive-by update to include latest zircuit metadata --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	3 months ago
Daniel Savu	2909e81a55	feat(sealevel): charge protocol fees (#4355 ) ### Description Adds protocol fees to the Sealevel implementation, by embedding the logic / custody into the mailbox program & PDA. The logic is very similar to what already exists in the IGP implementation. The instructions to interact with the protocol fees implementation are: - `Init` (now requires specifying the `protocol_fee` and `beneficiary` in addition to the other mailbox params) - `ClaimProtocolFees` - invokable by anyone, and sending accumulated lamports to the `beneficiary` - `SetProtocolFeeConfig` - invokable by the owner, sets new values for `protocol_fee` and / or `beneficiary` ### Drive-by changes - The restriction applied on max message size has been removed to mirror the Solidity V3 implementation ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4356 ### Backward compatibility No - requires redeploying the mailbox ### Testing Sealevel program integration tests, as is done for the other programs	3 months ago
Trevor Porter	57bbb16cc1	feat: panics upon startup when RPC issues occur include the offending chain (#4323 ) ### Description Some quick low hanging fruit to make debugging easier in the future. Today (https://discord.com/channels/935678348330434570/1273582453889699843) some chains were having RPC issues and it caused an agent deploy to panic upon startup. Debugging this was hard because the panic message just indicates an error is occurring, but doesn't say what the offending chain is! Examples: - https://cloudlogging.app.goo.gl/puQk7hm4fXyJhYKs8 - https://cloudlogging.app.goo.gl/5qZrbc2Ww4LWQXWC6 - https://cloudlogging.app.goo.gl/ZL7mKbBzzJaePtty9 All these panic happen when constructing the cursor, which isn't yet done in an indexing task. The quick and easy fix is to just use expect instead of unwrap We still unwrap in a few places, but mostly in places that aren't susceptible to flaky RPC issues, e.g. reading from a HashMap that we know should always have an entry, etc ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Trevor Porter	beab39c43c	chore: update sei gas price, temporarily stop deploying to osmosis (#4321 ) ### Description Following https://seitrace.com/proposal/83?chain=pacific-1, need a higher gas price ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Paul Balaji	279516f280	chore: enroll worldchain, xlayer, cheesechain, zircuit validators (#4313 ) - chore: enroll worldchain, xlayer, cheesechain, zircuit validators - fix: update cosmos zod schema - drive-by: update core checker to check if the ism in registry matches the expected config	4 months ago
Trevor Porter	9db5871f27	fix: use correct Neutron IGP (#4303 ) ### Description - We've been operating Neutron warp routes with a whitelist so far. It seems like in the initial shuffle of things, we configured the agents to index an IGP that was actually replaced 😬 by another one - This starts indexing the new one. Because of watermark indexing, we won't backfill old IGP payments, because from its perspective it thinks it's already indexed those. I wonder if we should tie watermark indexing cursors to the IGP address? Either way, this is probably fine as to date only a couple non-warp messages from Neutron have occurred ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Paul Balaji	d274c6b3f8	chore: update registry to v2.5.0 (#4302 ) chore: update registry to v2.5.0	4 months ago
Paul Balaji	8be75b6569	feat: core deploy to new chains (#4281 ) Deploy to new chains: 1. cyber 1. degenchain 1. kroma 1. lisk 1. lukso 1. merlin 1. metis 1. mint 1. proofofplay 1. real 1. sanko 1. tangle 1. xai drive-by: - fix bug in `set-rpc-urls-from-registry.ts` - add script to generate sql to add multiple chains to scraper db	4 months ago
Paul Balaji	2d922805c0	feat: add arbitrumsepolia, basesepolia, ecotestnet, optimismsepolia, polygonamoy to core chains (#4297 ) Deploy to arbitrumsepolia, basesepolia, ecotestnet, optimismsepolia, polygonamoy	4 months ago
Danil Nemirovsky	d1bf212bf7	feat: Report git commit hash as agent metadata (#4282 ) ### Description * We are storing git commit hash as an environment variable when we compile the source code of an agent. * Validator uses git commit hash as a agent version in `metadata_latest.json` file. * File `metadata_latest.json` is overwritten every time when validator is restarted. * Git commit hash is extracted by `git` util, so, it should be available during build time. * If the directory with the source code does not contain `.git` folder, git commit hash will be defaulted to `VERGEN_IDEMPOTENT_OUTPUT`. Agent Metadata is propagated to Relayer and Scraper, but is not used there. We can use it in the future, reporting git hash commit in the logs, for example. ### Backward compatibility Yes ### Testing * Tested locally with E2E Tests. * Tested against S3 by deploying docker image into k8s --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	4 months ago
Paul Balaji	7625bf8dce	fix: typo in trait builder log (#4292 ) - fix: typo in trait builder log - noticed while looking at scraper logs	4 months ago
Paul Balaji	103e6b3e12	feat: deploy to `zircuit` (#4265 ) - feat: deploy to `zircuit` - update registry + agent config	4 months ago
Daniel Savu	a49f52cc9c	feat: deploy solana, eclipsemainnet (#4238 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: pbio <10051819+paulbalaji@users.noreply.github.com>	4 months ago
Danil Nemirovsky	21da4861d0	docs: Add hyperlane registry as prerequisite for e2e tests (#4263 ) ### Description E2E tests will fail if `hyperlane-registry` repo is not cloned next to the monorepo. This change adds it to the `rust/README.md` file. ### Testing Checked how `rust/README.md` looks like in GitHub. --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com> Co-authored-by: Trevor Porter <tkporter4@gmail.com>	4 months ago
Paul Balaji	ff3c9897ea	chore: migrate `test1` from `13371` to `133711` (#4252 ) - chore: migrate `test1` from `13371` to `133711` - due to clash with immutablezkevm - drive-by registry update + agent config update --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	4 months ago
Paul Balaji	c6bf71d1b3	chore: update registry to v2.4.0 (#4247 ) chore: update registry to v2.4.0 - pinned CI version updated to `d9fea91f16949a51b2c8e776bac26712ba1fef3d`, which includes some testnet updates after 2.4.0 (https://github.com/hyperlane-xyz/hyperlane-registry/pull/126) --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	4 months ago
Trevor Porter	fe9365fdb3	chore: agent metric log tweaks (#4244 ) ### Description - some small tweaks as I looked into a recent issue with metrics ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Mohammed Hussan	8e942d3c6b	feat: deploy cheesechain, worldchain, xlayer (#4207 ) ### Description - Support deployment to cheesechain, worldchain and zlayer ### Drive-by changes - update `CosmosChainSchema` to add `chain_type` to fix breaking change introduced by `6854133f98`	4 months ago
Daniel Savu	b643fba90a	feat: use multiple `mpsc` channels instead of `broadcast` for async interface (#4190 ) ### Description Follow up to https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4180 Lowers txid broadcast channel capacity from 1M to 30k and refactors it to use a vec of `mpsc` under the hood. ### Context The heap profiler showed that each new chain takes up 100MB of memory because of the txid broadcast channel. In addition to this, the tokio broadcast channel is a ring buffer where new items simply overwrite old items, making it hard to safely lower the capacity. As such, to be able to safely lower the capacity of this channel, usage of `tokio`'s `broadcast` channel is replaced with a `Vec` of `mpsc`s, which offer an async `send` interface for backpressure. So instead of overwriting old items, senders will block on sends, which makes sense in our case as we absolutely don't want to miss items. The submitter is slower than the indexer based on empirical observation, so blocking on channel size doesn't bottleneck throughput. I noticed the max number of channel items we can reach is 29k, by running the RC relayer after an 8 day break, during which around 160k messages were submitted. Lowering the capacity to 30k should cause us to only encounter backpressure in very rare cases, and the footprint of this channel would lower from 100MB to 3MB. ### Drive-by changes The delivery indexer in the scraper is no longer passed a txid receiver, since message deliveries are not expected to occur in the same tx as message dispatches. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Testing E2E, by adding invariants to make sure txid indexing works --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	4 months ago
Daniel Savu	d864c6a939	fix(evm): only create signer provider when needed (#4208 ) ### Description Adds an associated const to all ethereum provider builders to specify whether a signer is needed. For an e2e run with 20 messages, we now have 12 providers built with signers at the beginning (3 of which are validator announce), and 708 providers built without a signer. This should hopefully fix the gas escalator memory leak. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	4 months ago
Daniel Savu	6140e6f397	feat: Relayer heap profiling (#4180 ) ### Description During the scalability audit we've identified relayer memory usage as a cause of concern. This PR runs the profiler each time e2e is run and outputs a report called `dhat-heap.json` in the current dir. That file can be visualised with a dhat viewer, such as the one hosted by the author [here](https://nnethercote.github.io/dh_view/dh_view.html). It adds a conditionally compiled heap memory profiler ([dhat-rs](https://docs.rs/dhat/latest/dhat/)), which is only included when both the `memory-profiling` feature is enabled and the `memory-profiling` cargo profile is used. The report can order the profiling results by total lifetime allocated bytes, peak allocated bytes, total allocation count, etc. Below is the result of sorting by total lifetime allocated bytes, which reveals that the biggest memory hogs are the hook indexing channel [senders](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/hyperlane-base/src/contract_sync/mod.rs#L52) (one per origin chain), taking up 100+ MB. Its size can most likely be reduced from 1M to 1k. ![Screenshot 2024-07-23 at 14 32 21](https://github.com/user-attachments/assets/f551ebfe-3ee0-40de-b53f-e5de1bc56dfa) ### Drive-by changes Adds the remaining IDs in the channel as a field to the hook indexing log, to know if it's safe to lower the channel size. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Backward compatibility Yes, since everything is conditionally compiled / isolated in a new cargo profile. ### Testing Manual	4 months ago

1 2 3 4 5 ...

893 Commits (1a8cc278183188bb0f31cb40fa16b131f606da5d)