hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Daniel Savu	528f3dad9a	fix: e2e passes with escalator enabled	5 months ago
Trevor Porter	e79cda7873	attempted poc for fix	5 months ago
Daniel Savu	d213e32341	feat: relayer queue operation labels (#4003 ) ### Description - adds status labels to operations in the relayer queues. In practice, the most useful ones are those for when a message is retried due to an error. Labels are defined as enums to keep the cardinality finite, the size of prometheus metrics could get very large - whenever a message is pushed to a queue, the interface requires specifying the new status as a `Option<PendingOperationStatus>`. If `None`, the status is left as before (such as in the case when an operation isn't ready to be retried). - a limitation is that labels aren't stored in the DB, so messages are loaded with the `FirstPrepareAttempt` label. I just triggered retries for destinations I wanted to get labels for, but we could persist to the db in a follow-up PR - Example prep queue labels for inevm: <img width="621" alt="Screenshot 2024-06-21 at 18 05 58" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/23065004/f53b2858-371f-4b6b-86d9-5c21409d641f"> ### Drive-by changes - removes `make_op_try` as it doesn't really add anything that couldn't be done with regular Rust - bumps the `strum` version because there was a bug in the previous one, preventing inner enum fields from being converted to strings ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Yes ### Testing Manual	5 months ago
Daniel Savu	718d2c27da	chore: don't reset retry count when processing retry API reqs (#4037 ) ### Description When messages are retried due to the Message Retry API, we bring messages to the front of the queue (by changing their `next_attempt_after`), but also reset their retry count. For messages that are unprocessable due to various reasons (and hence have very high `next_attempt_after`), resetting the retry count will cause the relayer to waste time instead of trying to process newer messages. This PR makes it such that the retry API will cause messages to be retried immediately once, but won't boost their priority in the queue afterwards (because `num_retries` isn't changed). ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Trevor Porter	a109937f96	chore: some throughput tweaks (#4036 ) ### Description Context: https://discord.com/channels/935678348330434570/1254036693217054794 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Snoppy	f1fe882e3b	chore: fix typos (#3645 ) fix minor typos Signed-off-by: snoppy <michaleli@foxmail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz>	6 months ago
Paul Balaji	c5a5e264b8	chore: enable `osmosis` on relayer (#4028 ) - chore: enable `osmosis` on relayer - drive-by add default chain signer/key/type to agent config - `signer.key` is copied from `neutron` --------- Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	6 months ago
Trevor Porter	c84a1dc73b	feat: address blacklisting in the relayer (#4000 ) ### Description See context here https://discord.com/channels/935678348330434570/1236041564673806427/1252918894755319920 and https://discord.com/channels/935678348330434570/1252953507833708605/1252954456400592997 This is a rudimentary first step toward blacklisting addresses. Future options are outlined in https://discord.com/channels/935678348330434570/1236041564673806427/1252918894755319920. Eventually we can also fetch these addresses directly within the agent from an updated list that we cache and re-fetch occasionally. For now, this just relies on configured addresses. The approach is: - Allow hex addresses to be configured. These are intentionally treated as a Vec<u8> instead of an H160 / H256. This is because treating them as H160 is a leak of Ethereum-specific logic into non-chain-specific code in the relayer, and treating them as H256 would pad Ethereum addresses with a bunch of zeroes which would diminish the accuracy of our heuristic implemented in this. The subsequence stuff is a bit unfortunate because it's O(n^2) but 🤷‍♂️ - The heuristic is as follows - if any of the configured addresses are found as a subsequence in the message sender, recipient, or body, it's thrown out by the message processor, and the message will never make its way to the op_submitter. This heuristic for the body is to prevent warp route transfers that include that address. We could maybe convert Vec<u8> to H256 for the sender and recipient, I'm open to this - I just went with the substring approach bc it felt easiest tbh. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1285 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	f227eec892	fix: e2e invariant flake (#4017 ) ### Description Context is explained in the comment, copying it here: ``` // TODO: Sometimes we find more logs than expected. This may either mean that gas is deducted twice for the same message due to a bug, // or that submitting the message transaction fails for some messages. Figure out which is the case and convert this check to // strict equality. // EDIT: Having had a quick look, it seems like there are some legitimate reverts happening in the confirm step // (`Transaction attempting to process message either reverted or was reorged`) // in which case more gas expenditure logs than messages are expected. ``` ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Paul Balaji	8ce2a71f20	chore: reconcile mainnet config json with registry (#4016 ) chore: reconcile mainnet config json with registry Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	6 months ago
Daniel Savu	3bb9d0a767	fix(cosmos): validator checkpoint & agent indexing improvements (#3958 ) - propagates RPC errors to the indexing logic, so errors cause sleeps instead of overwhelming the RPCs. This also greatly reduces cosmos log noisiness: from 9.5k logs per min, to 2.5k logs per min. - checks for published checkpoints in reverse, to prioritize recent messages among those from before the validator was spun up - fixes a validator bug where errors in checkpoint publishing would cause the entire checkpoint queue to be reiterated through, from scratch - Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3887 so we can test this out on osmosis ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Trevor Porter	bbbe93206f	chore: bump default ISM max depth from 5 to 7 (#3989 ) ### Description Needed for some Renzo messages that hit the max depth ### Drive-by changes - drive by to update the agent config with the updated registry ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Paul Balaji	5cc64eb095	feat: deploy to new chains (fraxtal, linea, sei) (#3887 ) https://github.com/hyperlane-xyz/issues/issues/1279 - adds `fraxtal`, `linea`, `sei` - supports `osmosis` as a remote - ~drive-by hack to skip reconfiguring the same gas oracle (cc @tkporter)~ - drive-by fix to bring the logic of estimating tx + adding 10% buffer into the ism factory (as originally seen in the multiprovider handleDeploy function) - drive-by fix for creating custom safes - drive-by support of `osmosis` as a remote chain - drive-by addition of `osmosis` to the rust agent --------- Signed-off-by: Paul Balaji <paul@hyperlane.xyz> Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: nambrot <nambrot@googlemail.com>	6 months ago
Ayene	6171d82c82	chore: fix broken links (#3970 ) ### Description changed links pointing to Solana docs due to migration to different repository	6 months ago
Daniel Savu	12840964b9	chore: add all new chains to `KnownHyperlaneDomain` struct (#3960 ) ### Description So validators can be run with `--originChainName <chainname>`, like in the docs [here](https://docs.hyperlane.xyz/docs/operate/validators/run-validators#running-the-binary) <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	630ad40a92	wip	6 months ago
Daniel Savu	5a31e7b5d3	chore: debug slow relayer startup (#3932 ) ### Description Opened this PR to debug slow relayer startup, but it actually looks like that's already been fixed by hook indexing. The requirement for fast startup is that the processor sees new messages and then one of the following happens: - (1) if the message is meant for an unknown domain, it drops it ([this log](`6d132b02a3/rust/agents/relayer/src/msg/processor.rs (L269)`)), otherwise - it forwards the message to the submitter, which then: - (2) adds it straight to the confirm queue because it's already been submitted ([here](`6d132b02a3/rust/agents/relayer/src/msg/pending_message.rs (L159)`)) - (3) successfully builds metadata ([here](`6d132b02a3/rust/agents/relayer/src/msg/op_submitter.rs (L231)`)) - (4) in some cases building metadata fails because the ism doesn't exist The relayer logs either of the four cases (in most cases (2), because we're running a second relayer for now). To be even more confident in fast startup, I ran a single relayer and restarted it. New messages were successfully processed according to one of the four cases above, and the prep queues didn't increase ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
overallteach	42a6dbc3e6	chore: remove repetitive words (#3644 ) ### Description <!-- What's included in this PR? --> remove repetitive words ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> Signed-off-by: overallteach <cricis@foxmail.com>	6 months ago
Paul Balaji	65e4b05f77	chore: fix mainnet3 agent config (#3931 ) chore: fix mainnet3 agent config follow-on from https://github.com/hyperlane-xyz/hyperlane-registry/pull/49 Signed-off-by: Paul Balaji <paul@hyperlane.xyz>	6 months ago
Daniel Savu	826b4ae57f	feat: batching igp (#3870 ) ### Description Deducts IGP cost from senders of messages within a batch, based on the total gas used by the batch and the estimates for submitting messages individually. The formula is `gas_used_by_operation = gas_used_by_tx * (operation_estimated_gas / total_estimated_cost)` Note that the individual estimate have a buffer value added to them (currently 75k), which will slightly skew proportions, though by a negligible amount. For example, given these estimates in a batch: 150k, 160k, 180k, if we add 80k to each, we go from e.g. 0.312 for the first message to 0.319 -> 2.25% error which isn't so bad. The error can be larger if operations within a batch have very different estimates, but realistically within a 5% range based on back-of-the-napkin calculations. ### Drive-by changes - The batching feature introduced an bug whereby gas expenditure would only be deducted in the `confirm` step. Now this is done in the `submit` step to account for txs that revert - Sealevel e2e can now be disabled for faster iteration! I've not done the cleanest job, but even the fact that we have this will reduce the time to run e2e locally by more than half. To use it, set `SEALEVEL_ENABLED=false` when running `run-locally` - e2e uses a new utility called `get_matching_lines`, that can be used to count (and in the future _parse_) logs, to reconstruct the state of the relayer and have more expressive correctness checks. This is used to make sure that gas is deducted for all messages, including those in batches. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3709 ### Backward compatibility Yes ### Testing E2E, using `get_matching_lines`	6 months ago
Daniel Savu	942aba8e55	feat(evm): collaborative indexing through txid sharing (#3833 ) ### Description - improved indexing reliability by sharing channels between indexing tasks, such that they inform one another when they come across a hyperlane-related transaction - the receiving end will then get the receipt of that tx and filter it for events relevant to it - this adds some flexibility to indexing since we can stop relying on watermark cursor entirely: as long as we have one task using sequence indexing, all others can rely on being informed by their channels about what txs to index - this PR is only for the EVM chains, but can be easily extended ### Drive-by changes - Enforces gas payments for all chains in e2e (previously this only used to be the case for sealevel) - The channel size is configured per type, similarly to the indexing cursor, in `contract_sync/cursors/mod.rs` - Reduces log verbosity is some places, e.g. by manually implementing `Debug` to exclude fields that aren't relevant to debugging (maybe there's even a macro annotation to exclude, but I haven't checked) - Renames `MpmcReceiver` -> `BroadcastReceiver` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3267 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing e2e tested with the watermark cursor manually disabled	6 months ago
Daniel Savu	4261df74bb	chore: restrict for how long the backward syncer can block (#3918 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Noah Bayindirli 🥂	ba946f1eff	fix(config): switch to recommended 'rust-analyzer' from deprecated 'cargo' (#3886 ) ### Description - switch to recommended `rust-analyzer` since `cargo` is deprecated	6 months ago
Paul Balaji	d10b035ec2	fix: testnet4 holesky config (#3900 ) ran `yarn tsx ./scripts/agents/update-agent-config.ts -e testnet4` and updated testnet4 config Signed-off-by: Paul Balaji <paul@hyperlane.xyz>	6 months ago
Kunal Arora	554f875ebf	feat: core deployment on Holesky (#3776 ) ### Description Core deployment on Holesky ### Drive-by changes None ### Related issues - fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3597 ### Backward compatibility Yes ### Testing Manual	6 months ago
Trevor Porter	db9f733e4f	fix: use secret RPC urls in infra, apply environment-specific registry overrides (#3875 ) ### Description A while back, it seems secret RPC URLs stopped being used by monorepo services (see https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3179/files#diff-ba3b7bea2b5c046acd88d25ed382df875b49d77e061f2ef8347ffcc741c0353eR34). This adds them back. Some things to note on the approach: - Sadly, due to the way some of the infra config generation works, we're forced to use synchronous interfaces for getting chain metadata in a bunch of places. This is why `getRegistry` returns a `FileSystemRegistry`, which is a `SynchronousRegistry`. - Applying secrets / overrides requires us to do things async for two reasons -- one is just that fetching from GCP secrets is an async operation, the other is that a `MergedRegistry` is not a `SynchronousRegistry`. The approach I've taken is that there are two ways of getting a registry - using `getRegistry` will give the synchronous version that can continue to be used by our sync config generation, but it won't have any environment-specific overrides (tx overrides or secrets). To get an async registry with environment-specific overrides, the `getRegistry` function on the EnvironmentConfig is used. - Overrides are now more succinct as the merging happens later in the pipeline ### Drive-by changes - Started moving away from using the gcloud CLI under the hood for GCP operations and toward using the GCP secret SDK - Updated the mainnet3 key funder to fix an incident, but will follow up with a fresh deploy once this is mergd ### Related issues - These are the code changes for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3774, but I'll follow up with fresh deploys for the monorepo services ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Connor McEwen <connor.mcewen@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: J M Rossy <jm.rossy@gmail.com>	6 months ago
omahs	806f9842de	fix: duplicate word typos (#3854 )	6 months ago
Daniel Savu	1864f776ec	feat: use `max_fee_per_gas` from tx-overrides in the escalator	6 months ago
Daniel Savu	64803be90b	feat(hyp-ethereum): gas escalator middleware	6 months ago
Daniel Savu	214f503e53	feat: forward-backward message processor (#3775 ) ### Description - adds logic for iterating message nonces in the processor task in forward-backward fashion - removes all manual calls that were updating the next nonce to query. This is done automatically in the iterator struct now - stores the highest known message nonce in the db, which is used to initialize the iterator ### Drive-by changes - Converts the concrete HyperlaneDb type stored in the processor to a trait obj, to enable mocking db responses ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3796 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3816 ### Backward compatibility Yes - if there is no db entry for the new prefix, the processor starts from nonce zero, so no migration is required ### Testing Unit testing + e2e	6 months ago
Trevor Porter	37af3dc6ba	feat: deploy to zetachain and redstone (#3788 ) ### Description - Deploys to zetachain and redstone - Some learnings can be found in https://discord.com/channels/935678348330434570/1240168332813144084 Chains not yet fully connected: - inevm (Mailbox ownership situation makes this hard) - mantapacific (waiting on an update to the multisig ownership set there, then will create the tx) - non-EVM chains ### Drive-by changes - decreased ethereum cost from $2 to $1.5 following some feedback - Added Superform to the blast set - Allowed for concurrent deploys I originally had some ugly caching to avoid checking storage gas oracles so much (we check them for each route in a routing hook configuration), but I've removed this. Calling it out for the future though `232695dd6c` ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	27aabf238c	feat: Relayer tokio task instrumentation (#3760 ) ### Description - Started off adding tokio-metrics but then realised those are quite general, so while we do have instrumentation it's not exposed in our metrics endpoint - switched to adding [tokio-console](https://github.com/tokio-rs/console/tree/main), which does give insight into the lifetime of specific tasks, so we can check which ones take up a long time during relayer startup. These are only visible at the `dependencyTrace` log level, so don't affect performance in the `hyperlane` context. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Helps debug https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3454 and any future performance issues - Does half the work for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3239 (still need to expose these in the metrics endpoint and import the grafana template) ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	1c4cf44c91	fix: allow all log levels in release build (#3824 ) ### Description Found the reason why we never had `trace` level logs in release builds - the `tracing` crate had a feature flag enabled that would wipe out all trace calls at compile time :( ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3823 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Trevor Porter	335a0d7479	feat: add agent config for Osmosis Mainnet (#3825 ) ### Description - Added agent config description for Osmosis Mainnet - Updated infra test to allow agent config chains to be a superset of infra chains Taken from @byeongsu-hong's https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3766, but with a fixed test ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: byeongsu-hong <hong@byeongsu.dev>	7 months ago
Trevor Porter	03ca2d1df7	Update gas oracles, deploy agents (#3755 ) ### Description - Updates gas oracles to make messages to Scroll slightly cheaper - After running into issues configuring non-EVM remotes in the IGP / gas oracle tooling (see https://discord.com/channels/935678348330434570/1238439687165579264), added a temporary band aid of just try/catching. We'll want to fix this at some point in the future - Deploys agents with a new image - Updates the agent JSON configs with some params we were running the agents with in prod via env vars - Adjusts kathy frequency on testnet to send every 1.5hrs instead of 7hrs - Stops running eclipsetestnet agents - apparently there was a reset ### Drive-by changes Some drive-by logging changes ### Related issues n/a ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	5d96847ffb	Multithreaded relayer (#3748 ) ### Description - Sets the tokio runtime flavor to be multithreaded and hardcodes the worker threads to 20. The latter usually defaults to the number of CPUs in the system so 20 may be high, but it worked fine in RC. Improved performance metrics: - time to first `Ingesting leaf` log: 1:30 mins (multithreaded) vs 5:51 mins (single threaded) - time to first message in the submit queue of Optimism (one of the slowest chains to boot up for): 8:30 mins vs 20 mins ### Drive-by changes Sets more descriptive `span`s when instrumenting the contract syncs ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Manually, by checking the RC logs	7 months ago
Daniel Savu	be43b0fd91	feat: use batching configs on all execution environments (#3723 ) ### Description While we only have a tx batching implementation available for EVM chains, we still parallelize the `prepare` and `confirm` steps in the serial submitter according to the `maxBatchSize` config item. To reap the benefits of this change, we need to make these configs universal, because they were only defined within `h_eth::ChainConnectionConf`. This PR adds slight duplication because it adds the batching config on each individual connection config, but it's far simpler than including it in the `ChainConf` struct. The latter would've required changing all the trait impls of `BuildableWithProvider` to include the batching config. ### Drive-by changes Renames `serial_submitter` to `op_submitter` since it's no longer serial. ### Related issues Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3722 ### Backward compatibility Yes ### Testing Manually - added the configs to cosmos e2e and checked the logs	7 months ago
Daniel Savu	f46cf7708e	Indexing Type Wrapper (#3610 ) ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3544. Wraps all types to be indexed in an `Indexed<T>` struct in the relayer. This allows for using sequece-aware cursors for data that doesn't contain a sequence but can have it inferred from metadata (as is the case for the IGP on sealevel). `Indexed<T>` stores the sequence as an `Option`, so the db can have additional sequence data stored in it when the sequence is defined - this is how `process_indexed_gas_payment` is implemented now. In addition, the IGP payments and Deliveries are configured to be indexed with a `SequenceAware` cursor on sealevel (see `cursors/mod.rs`). ### Drive-by changes Implements the custom `Encode` and `Decode` traits for `IndexingDecorator`, which can be generalized to `Option<T>` where `T: Encode + Decode`. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing E2E	7 months ago
Daniel Savu	301239265e	WIP: don't hardcode multicall gas limit (#3710 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	8e5ef272a6	fix: agent config check, run relayer on mantapacific (#3714 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	22f367ff6c	fix: sleep after submitting to cosmos chains (#3699 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	5aeb83a2e5	Relayer Message Batching to EVM (#3656 ) ### Description Status: - [x] op-queue scaffolding - [x] hyperlane-ethereum integration with multicall - [x] new config items - [x] actually build batch in the op-queue submit step - [x] e2e tests with batching configured - [x] dynamic batch sizes (batch as many messages as the prepare step has ready) - [x] parallelize metadata building - [ ] proportional IGP deduction ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes #3628 - Fixes #3630 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	7 months ago
J M Rossy	3528b281e3	Migrate SDK consts to Registry and use new registry utilities (#3615 ) ### Description - Remove most consts from SDK - Refactor tests that relied on chainMetadata - Remove JSON imports/exports in the SDK - Update the HelloWorld package to read metadata from the registry lib - Update the Infra package to read/write to a LocalRegistry - Update the CLI to read and write from registries ### Drive-by changes - Fix typo in `EvmHypCollateralVault` token standard - Use consistent, newest yaml lib version ### Related issues Fixes https://github.com/hyperlane-xyz/issues/issues/917 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2810 ### Backward compatibility No: - Several SDK exports like `chainMetadata` have been removed - Common args for the CLI like `--chain` have been replaced with `--registry` and `--overrides`	7 months ago
Trevor Porter	c4b7ad3196	Update Mode and Blast validator sets & gas oracles (#3669 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: nambrot <nambrot@googlemail.com>	7 months ago
Trevor Porter	21cd38c2f3	Update IGP gas oracles with new heuristics, get Cosmos gas prices from registry (#3662 ) ### Description The original and main motivation of this PR was to feel confident when running `deploy.ts -m igp` off of main. Prior to this, running it off main could result in misconfiguring gas oracles, particularly when it came to non-EVM remotes. See https://github.com/hyperlane-xyz/issues/issues/1178 for some context. This PR does a number of things: - Fetches Cosmos gas prices from the Cosmos registry - the `gasPrices.json` now lets you specify the decimals relating to the amount in there. There are 2 reasons for this: it's easier for us to deal with gwei values for EVM chains (as we often end up manually changing these instead of just relying on the script). The second reason is covered in the next bullet - Handles decimals when it comes to gas prices. The existing tooling assumed gas prices would always be denominated in wei. An extra weird situation arises with non-EVM chains: chains like Neutron have gas prices that are quotes in the smallest denomination (ie the equivalent of "wei" for NTRN), but the gas price is not an integer and is very small, e.g. `0.0053`. This poses issues because the IGP contracts require the gas price that's configured onchain to be an integer. Simply rounding up from 0.0053 to 1 results in almost 2 orders of magnitude difference :(. So the solution here is to have some logic that accommodates really tiny gas prices by scaling up the configured gas price and scaling down the configured token exchange rate. - Applies some "non-linear" gas pricing heuristics. See https://github.com/hyperlane-xyz/issues/issues/1204 and https://discord.com/channels/935678348330434570/935678739663192184/1232293417309044818 for some context - Running `update-agent-config.ts` will fetch gas prices for Cosmos chains from the Cosmos registry and ensure they're accurate ### Drive-by changes Some logging messages when applying IGP gas oracle changes to make it more clear what the change actually does ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3386 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3484 - Fixes https://github.com/hyperlane-xyz/issues/issues/1178 - Fixes https://github.com/hyperlane-xyz/issues/issues/1206 - Fixes https://github.com/hyperlane-xyz/issues/issues/1204 ### Backward compatibility Fully backward compatible ### Testing Ran deploy.ts for the IGP on all chains! --------- Co-authored-by: nambrot <nambrot@googlemail.com>	7 months ago
Trevor Porter	d6f25ed035	Deploy to Mode & Blast (#3653 ) ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	f2cb6ebc1b	refactor: organize files in `hyperlane-ethereum` (#3633 ) ### Description Some cleanup along my way to integrating with `Multicall3` in hyp-ethereum. - Categorizes most files from `src` into `ism`, `contracts`, `signer` modules - Renames the previous `contracts` dir (that held the abi interfaces) to `interfaces`, so we can use `contracts` for the actual contracts (e.g. mailbox, igp)	8 months ago
Trevor Porter	700240eb8a	Ancient8 new ism, misc infra fixes (#3606 ) ### Description - Deploys ancient8 with a new ISM where we are not quorum - Pulls out warp config to live in a single file / function so that deploy.ts and check-deploy.ts are in sync. Otherwise fork.sh is broken for warp routes - Shuffles around the aw multisig serialization stuff - now they're segmented by environment and context and are written to `config/{env}/aw-validators/{context}.json`. This was in response to some confusing merging logic, see https://discord.com/channels/935678348330434570/1228377321082327141/1229464197297868910 ### Drive-by changes Needed to specify pino to work in pretty mode for warp monitoring :/ this is necessary because pino doesn't work otherwise. After #3582 we should change this ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Daniel Savu	f4626eb4fe	Allow protocol-specific indexing (#3544 ) ### Description - Generalizes contract syncer hashmaps to store trait objects instead of concrete structs, since both `SequencedDataContractSync` and `WatermarkContractSync` implement `ContractSyncer` - The nice thing this enables is being able to instantiate a different type of syncer for the same contract, for different chain protocols (e.g. for IGP, we can use a sequenced syncer on sealevel and cosmwasm, and a watermark syncer on EVM). - The problem is that `ContractSyncer` is implemented on different trait bounds on `SequencedDataContractSync` and `WatermarkContractSync`. Most notably, the `SequencedDataContractSync` impl bounds the indexed type by the `Sequenced` trait. This means that if a type is to be indexed by more than one syncer strategy, it needs to implement the union of bounds of those strategies - Blocker: In the IGP, this means that `InterchainGasPayment` needs to implement `Sequenced` and we have no obvious way of getting that just from the type. Even if the sequence is derived from the type, it has to be a sensible value, since indexing reliability depends on it. - Similar to the point above, we now have `contract_syncs`, a single function that instantiates contract syncs for all chains. Since `contract_syncs` must be able to build a `ContractSyncer` trait obj for all syncer types, it bounds both `T` (the indexed type) and `D` (the log store type) with a union of bounds, so it may be too strict for some types in the future. - The scraper now uses forward-backward indexing instead of just forward indexing, because that simplified the refactor - there's now a convenience trait called `TryFromWithMetrics` that allows adding this bound: `SequenceIndexer<T>: TryFromWithMetrics<ChainConf>,`. This isn't really extendable if we use non-sequence indexers, but it's fine for now since this is used everywhere - replaces the macro-based contract syncer builder with generic fns Still todo: - add new `IndexingDecorator<T>` struct that can wrap `InterchainGasPayment`, so we can use sequence-aware indexing for the IGP - remove sequence logic from the watermark cursor ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	8 months ago
Noah Bayindirli 🥂	aea79c686e	feat: add dry-run support for warp deploy + gas stats (#3586 ) ### Description * New feature to allow users to dry-run the warp deploy script against a forked (base/origin) network of their choice * To run: `yarn build && yarn hyperlane deploy warp --dry-run` \|\| `yarn build && yarn hyperlane deploy warp -d` * Externally enables: `hyperlane deploy warp --dry-run` \|\| `hyperlane deploy warp -d` * Also adds gas usage util for both warp & core deployments in all contexts, e.g. * When running a vanilla core deploy, for example, between alfajores and fuji, you will now see: ``` ⛽️ Gas Usage Statistics - Gas required for core deploy on alfajores: 0.0058686745 CELO - Gas required for core deploy on fuji: 0.0239308515 AVAX ``` ### Drive-by changes * None ### Related issues * Fixes https://github.com/hyperlane-xyz/issues/issues/819 ### Backward compatibility * Yes ### Testing * Note: `hl` == `yarn build && yarn hyperlane`. The below tests are only a sample and are not inclusive. #### Manual Testing * With `anvil` NOT running in separate instance: * `hl deploy warp -d` * Throws: ``` Error: No active anvil node detected. Please run `anvil` in a separate instance. ``` * With `anvil` running in separate instance: * `hl deploy warp -d -k c0052e22df5d1f4ae7c51e254Xx00Xx0eb833453eaed6301xXxxx8a30d92d10a` (any private key) * Throws `Error: Invalid address length. Please ensure you are passing an address and not a private key.` * `hl deploy warp -d -k 0x16F4898F47c085C41d7Cc6b1dc0xX0xXX017dcBb` (any public address) * Output: ``` 🔎 Verifying anvil node is running... ✅ Successfully verified anvil node is running Using warp route deployment config at ./configs/warp-route-deployment.yaml No chain config file provided ? Do you want to use some core deployment address artifacts? This is required for PI chains (non-core chains). no Forking alfajores for dry-run... ✅ Successfully forked alfajores for dry-run Impersonating account (0x16F4898F47c085C41d7Cc6b1dc72B91EA617dcBb)... ✅ Successfully impersonated account (0x16F4898F47c085C41d7Cc6b1dc72B91EA617dcBb) ... ✅ Hyp token deployments complete Writing deployment artifacts Deployment is complete! Contract address artifacts are in artifacts/dry-run_warp-route-deployment-2024-04-16-12-27-02.json Warp config is in artifacts/dry-run_warp-config-2024-04-16-12-27-02.json ⛽️ Gas Usage Statistics - Gas required for warp dry-run on alfajores: 0.013310162514578124 CELO Resetting forked network... ✅ Successfully reset forked network ✅ Warp dry-run completed successfully ``` #### CI Testing * Successful CI-backed integration/regression testing via `ci-test.sh`	8 months ago

1 2 3 4 5 ...

833 Commits (528f3dad9a03f269ed4a5d3b5d1c0b00ce4cc662)