12 Commits (4c74affecf569d656b401ec3bcf458694b911fd6)

Author	SHA1	Message	Date
Daniel Savu	6140e6f397	feat: Relayer heap profiling (#4180) ... ### Description During the scalability audit we've identified relayer memory usage as a cause of concern. This PR runs the profiler each time e2e is run and outputs a report called `dhat-heap.json` in the current dir. That file can be visualised with a dhat viewer, such as the one hosted by the author [here](https://nnethercote.github.io/dh_view/dh_view.html). It adds a conditionally compiled heap memory profiler ([dhat-rs](https://docs.rs/dhat/latest/dhat/)), which is only included when both the `memory-profiling` feature is enabled and the `memory-profiling` cargo profile is used. The report can order the profiling results by total lifetime allocated bytes, peak allocated bytes, total allocation count, etc. Below is the result of sorting by total lifetime allocated bytes, which reveals that the biggest memory hogs are the hook indexing channel [senders](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/hyperlane-base/src/contract_sync/mod.rs#L52) (one per origin chain), taking up 100+ MB. Its size can most likely be reduced from 1M to 1k.  ### Drive-by changes Adds the remaining IDs in the channel as a field to the hook indexing log, to know if it's safe to lower the channel size. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Backward compatibility Yes, since everything is conditionally compiled / isolated in a new cargo profile. ### Testing Manual	4 months ago
Noah Bayindirli 🥂	aea79c686e	feat: add dry-run support for warp deploy + gas stats (#3586) ... ### Description * New feature to allow users to dry-run the warp deploy script against a forked (base/origin) network of their choice * To run: `yarn build && yarn hyperlane deploy warp --dry-run` || `yarn build && yarn hyperlane deploy warp -d` * Externally enables: `hyperlane deploy warp --dry-run` || `hyperlane deploy warp -d` * Also adds gas usage util for both warp & core deployments in all contexts, e.g. * When running a vanilla core deploy, for example, between alfajores and fuji, you will now see: ``` ⛽️ Gas Usage Statistics - Gas required for core deploy on alfajores: 0.0058686745 CELO - Gas required for core deploy on fuji: 0.0239308515 AVAX ``` ### Drive-by changes * None ### Related issues * Fixes https://github.com/hyperlane-xyz/issues/issues/819 ### Backward compatibility * Yes ### Testing * Note: `hl` == `yarn build && yarn hyperlane`. The below tests are only a sample and are not inclusive. #### Manual Testing * With `anvil` NOT running in separate instance: * `hl deploy warp -d` * Throws: ``` Error: No active anvil node detected. Please run `anvil` in a separate instance. ``` * With `anvil` running in separate instance: * `hl deploy warp -d -k c0052e22df5d1f4ae7c51e254Xx00Xx0eb833453eaed6301xXxxx8a30d92d10a` (any private key) * Throws `Error: Invalid address length. Please ensure you are passing an address and not a private key.` * `hl deploy warp -d -k 0x16F4898F47c085C41d7Cc6b1dc0xX0xXX017dcBb` (any public address) * Output: ``` 🔎 Verifying anvil node is running... ✅ Successfully verified anvil node is running Using warp route deployment config at ./configs/warp-route-deployment.yaml No chain config file provided ? Do you want to use some core deployment address artifacts? This is required for PI chains (non-core chains). no Forking alfajores for dry-run... ✅ Successfully forked alfajores for dry-run Impersonating account (0x16F4898F47c085C41d7Cc6b1dc72B91EA617dcBb)... ✅ Successfully impersonated account (0x16F4898F47c085C41d7Cc6b1dc72B91EA617dcBb) ... ✅ Hyp token deployments complete Writing deployment artifacts Deployment is complete! Contract address artifacts are in artifacts/dry-run_warp-route-deployment-2024-04-16-12-27-02.json Warp config is in artifacts/dry-run_warp-config-2024-04-16-12-27-02.json ⛽️ Gas Usage Statistics - Gas required for warp dry-run on alfajores: 0.013310162514578124 CELO Resetting forked network... ✅ Successfully reset forked network ✅ Warp dry-run completed successfully ``` #### CI Testing * Successful CI-backed integration/regression testing via `ci-test.sh`	8 months ago
Daniel Savu	e737d998ff	Cosmos block-by-block indexing (#3245) ... ### Description Implements cosmos block-by-block indexing, parallelizing where possible: the `block` and `block_results` calls, and the individual block queries within a range. Also reuses the existing `handle_txs` logic, to improve readability. ### Drive-by changes Adds `tokio::task::JoinError` to the `hyperlane-core` error enum ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3234 - Added some hacky retrying logic to `get_logs_in_block`, since we were seeing lots of `503` http errors. In https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 we should make sure to remove this and propagate the error instead, to retry via the cursor / indexer logic. I edited the description of 3264 to reflect this ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	9 months ago
Asa Oines	97d17570cc	Refactor IGP related code out of HyperlaneCore (#1907) ... ### Description A number of changes intended to make the SDK more easily consumed by hyperlane-deploy - Splits the IGP related code out of HyperlaneCore* into HyperlaneIgp* - Moves agent config building from infra to SDK so that it can be accessed by hyperlane-deploy - Modifies CoreConfig to specify a ChainMap<MultisigIsmConfig> (i.e. specify everything that you need to know to deploy to a chain in that chain's CoreConfig) - Removes core.extendWithConnectionClientConfig - Moves common logic into HyperlaneAppGovernor to be shared between HyperlaneCoreGovernor and HyperlaneIgpGovernor - Adds TestRecipient contract addresses to the SDK addresses for consumption by hyperlane-deploy - Allow buildContracts to build when address keys are a superset of factory keys (via filterContracts). This is useful as it allows us to just throw all the addresses at a HyperlaneApp constructor without needing to remove the ones that aren't relevant - Removes InterchainGasCalculator, the logic we want to keep now lives in HyperlaneIgp - Allows chains to be added to a MultiProvider ### Drive-by changes - Merges the infra govern script into the check script (to dedupe) - Some minor renaming where "Core" was being used liberally - Default to using whatever is configured in `owners.ts` in the router configs for infra (for ICA, IQS, HelloWorld, and LL deployment) ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ No, existing consumers of the SDK may need to adjust. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual / Unit Tests	2 years ago
Mattie Conover	599722a7ca	Load all configs by default (#1887) ... ### Description This removes `RUN_ENV` and `BASE_CONFIG` from the Rust config loading process and also from the helm charts and TS configs as appropriate. ### Drive-by changes - Flattens config dir structure - Removes the test config from the git repo. - Fixed a bug in the deploy script that caused the e2e pipeline to break when the config file was removed - Removes `environment` from the config files - Updates TS code to use `runEnv` for deployment - Updates TS code to type environment with `DeployEnvironment` instead of `string` ### Related issues - Relates to #1850 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual - Updated e2e test to verify these don't cause a config loading issue.	2 years ago
webbhorn	274efcd34d	Restore reverted PR to .gitignore run-locally-induced diffs. (#746) ... This reverts commit 023eacd44b. Also introduce two minor changes: * End rust/.gitignore with a newline. * End test.json with newline, otherwise run-locally removes it, introducing an annoying diff.	2 years ago
Mattie Conover	023eacd44b	Revert "stop checking in test configs since they are generated automatically (#685)" (#687) ... This reverts commit 5ec2d82740.	2 years ago
Mattie Conover	5ec2d82740	stop checking in test configs since they are generated automatically (#685)	2 years ago
webbhorn	6b3822f664	Misc. agent cleanup and dead code deletion (#470) ... * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in comment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool. * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in coment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool.	3 years ago
Trevor Porter	fec07d21fe	Remove docs & all optics-related rust (#322) ... * optics -> abacus in infra * rm docs * rename contract-metrics prom metrics optics-* -> abacus-* * Rm watcher, updater, a bunch of optics things in abacus-base * rm agents/processor * Removing a whole lot of updater / home / replica stuff from abacus-core * Deleted a lot throughout -- some still commented out and a lot of cleanup needed * Slowly cleaning up, kathy is where the remaining errors are * Delete some commented out code * makes kathy an abacus agent * compiling woo * cargo fmt * compiling with all features * removing some update references * Ignore handles_missing_rpc_checkpoints * cargo fmt * Add signedCheckpoint.json * rm some references to updates, delete unnecessary abis * rm all references to home or replica * compiling, remove som e unused code * fix test warning * nit * New README - needs a better pass, but no optics references for now * mv rust/helm/optics-agent rust/helm/abacus-agent * Note on us not indexing updates * Get checkpointer working from local env var output	3 years ago
James Prestwich	ce79bb1c5a	refactor: improve db and test roundtrip store/load (#774) ... * refactor: improve db and test roundtrip store/load * refactor: improve tracing messages in processor * refactor: home event getters rely only on db now * feature: more tx info in report_tx tracing	3 years ago
James Prestwich	2afd32391a	the last syllable of recorded time	4 years ago