833 Commits (8c8f39abfe59cac4f424cfbae62a4d32781d9757)

Author	SHA1	Message	Date
Daniel Savu	8c8f39abfe	feat(relayer): ListOperations api endpoint (#4089) ... ### Description Adds a `list_operations` API endpoint to the relayer, which returns all operations in its prepare queue. Example output from `GET http://0.0.0.0:9092/list_operations?destination_domain=13371` ``` [ { "app_context": null, "message": { "body": [ 18, 52 ], "destination": 13371, "nonce": 1, "origin": 13373, "recipient": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "sender": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "version": 3 }, "num_retries": 13, "status": { "Retry": "CouldNotFetchMetadata" }, "submitted": false, "type": "PendingMessage" }, { "app_context": null, "message": { "body": [ 18, 52 ], "destination": 13371, "nonce": 2, "origin": 13372, "recipient": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "sender": "0x000000000000000000000000927b167526babb9be047421db732c663a0b77b11", "version": 3 }, "num_retries": 13, "status": { "Retry": "CouldNotFetchMetadata" }, "submitted": false, "type": "PendingMessage" } ] ``` ### Drive-by changes Makes `PendingOperation` serializable, which means switching from JSON format to e.g. CSV (for easily filtering / aggregating in excel) should be very easy. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Manual and Unit Tests	5 months ago
Assaf Morami	1022e3858e	chore: add shebang to build.sh to specify bash as the interpreter (#4066) ... the `[[` command isn't available on e.g. zsh Co-authored-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Paul Balaji	d316772243	feat: deploy to BOB, Mantle, Taiko (#4051) ... resolves https://github.com/hyperlane-xyz/issues/issues/1289 - feat: deploy to BOB, Mantle, Taiko - drive-by: fix deploying to specific chain - drive-by: adding AVS operators for eth - drive-by: support non-evm remotes in mainnet3 core config --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Daniel Savu	c3c002a8fe	feat(hyp-ethereum): gas escalator middleware (#3852) ... ### Description Wraps evm providers in a gas escalator middleware as part of the `build_with_signer` step, so signed txs can have their gas price bumped if stuck in the mempool. Gas price is increased every 60s by 12.5%, and is capped at the `max_fee_per_gas` set in tx overrides. ### Drive-by changes - changes to our fork of `ethers`: - removes the `M: Clone` bound, since it's not actually required and conflicts with the middleware we use - increases the level of the `escalated` log in the middleware to `debug` (so we can see it working in prod without enabling `trace`) ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2959 ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	5 months ago
Daniel Savu	d90d8f86b4	chore: reduce op-queue verbosity (#4092) ... ### Description The gcp bill has been shooting up, and it's most likely due to the increased number of undeliverable messages being added to and removed from the queue, causing lots of `debug` level logs on `push` and `pop_many` to be emitted. This PR changes the verbosity to `trace` - we'll have lower visibility over the lifetime of a message, but there should be sufficient logs remaining to tell us what's going on (such as [operation prepared](30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L240)), [confirming a pre-submitted one](30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L260)), [submitting single op](30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L313)), [submitting batch](30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L456)), and [op confirmed](30630c4fa6/rust/agents/relayer/src/msg/op_submitter.rs (L386))). @paulbalaji suggested measuring log counts, so here they are: - in the last 6h, we had 42M `hyperlane` relayer logs, of which 22M where `push` logs and 1.1M were `pop_many` (no `pop`s) - if we make push ones trace, 5.5% of the remaining logs would be pop_many ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Trevor Porter	6f4ef05d41	feat: faster way of figuring out how many new scraper rows were inserted (#4073) ... ### Description ### Context Full context / thread: https://discord.com/channels/935678348330434570/1254871503825141923 Copied here: so this is kinda interesting basically the gas payment indexing in the scraper task is consistently prioritizing hook-based indexing work over block range indexing work this only manifests in the scraper and for arbitrum because: there are a lot of arbitrum origin messages on the scraper, each gas payment indexing tick takes a very long time because of some very suboptimal queries. store_payments can take like 10-15 seconds! You can see batches of logs Found log(s) in index range https://cloudlogging.app.goo.gl/Bo9Q7YwyziSqjsEu7 that correspond with the gas payment contract sync block height (green line) advancing https://abacusworks.grafana.net/goto/poW4KOQIg?orgId=1  but we're still indexing gas payments bc of hook indexing - there are tons of Found log(s) for tx id https://cloudlogging.app.goo.gl/QvDaBuf67SjjKviw6 In the scraper's store_payments we bizarrely end up performing 2 very expensive queries with the calls to payments_count - it gets the count of gas payments for the provided domain so it can figure out how many new gas payments were actually inserted https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/agents/scraper/src/db/payment.rs#L27-L64 These queries vary a bunch in how expensive they can be - sometimes each one is like 1-2 seconds, sometimes it gets up to 10 seconds each though https://cloudlogging.app.goo.gl/cDFNH3oxh6WWV3Go8  easiest fix is probably to figure out how to make store_payments less expensive - we probably don't need to be making these calls to payments_count to figure out which ones are new which ill do atm there's maybe a decent argument to having hook indexing and range indexing not contend with one another but we should be able to kick that can also an annoying thing is https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/hyperlane-base/src/contract_sync/mod.rs#L155 this log not showing up, probably some weird instrumentation setup or something makes debugging harder ### The fix We want to keep supporting the possibility that we've double-indexed a data type, and to only return the number of new rows inserted. We want to keep each indexing task as non-conflicting as possible - we know that only one scraper is running at a time, but that a task for each chain and data type is executing concurrently. It's possible for e.g. two gas payment indexing tasks to try to insert into the same table concurrently, but it'll be for different domains. Insert statement concurrency issues aren't an issue because the only chance for conflict is the auto-incrementing primary key id, but this is all handled properly by database internals. Insert results only give the latest ID resulting from the insertion. E.g. if no matter how many rows were inserted, it gives the latest ID from that batch, but no info about how many were in the batch. We could do something like get the latest id in the table for all domains (this is cheap) before, then do the insert, and then get the latest id again, but this is susceptible to race conditions if concurrent tasks get an insertion in during that time as well. Shoving all these into a transaction doesn't seem to solve the problem either unless there's a lock on the table. However, sea-orm doesn't support table level locks, just select locks. We could also use a CTE to perform as a way to perform the select & insertion atomically and return the value of the select - but again sea-orm doesn't seem to provide a nice way of doing this from what I could tell. Instead, my suggested fix is: 1. Get the latest id in the table relating to the specific domain (this is cheap, < 100ms) 2. Perform the insertion 3. Count how many new rows for the specific domain now exist whose id is > the id from (1). This is also cheap, < 100ms. If there is a very long time since the last insertion for the domain, this could take longer, but in practice this doesn't seem to be the case. If this couple order of magnitude improvement still poses issues in the future, we can consider other alternatives, probably involving locking or fancier CTE queries. Some example queries and how long they take: Before (2x of these would occur!): <img width="477" alt="Screen Shot 2024-06-28 at 12 36 41 PM" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/5cc8ac35-74fb-4847-9c6f-448deb253a0c"> Now: 1. Max id for a given domain <img width="494" alt="Screen Shot 2024-06-28 at 12 37 00 PM" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/56dbb3ff-b540-458c-809c-56ebfbfd3b0f"> 2. Subsequent count but with an ID filter - note I actually changed this query to look for IDs > 3600000, which includes 55300 rows and it's still just 102ms! <img width="632" alt="Screen Shot 2024-06-28 at 12 37 49 PM" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/11593180-682f-4867-9867-97fd93512802"> ### Drive-by changes - Updated the neutron image along with a deploy there ### Related issues ### Backward compatibility ### Testing Ran some sorta ad-hoc unit tests to test my new query by running against the prod db with a tokio::task. Tested for a new domain and an existing one	5 months ago
Daniel Savu	befc38daba	fix: agent log verbosity (#4054) ... ### Description tbh I don't know why, but this makes all the logs we were previously missing show up in e2e: - b99920ecca/rust/hyperlane-base/src/contract_sync/mod.rs (L155) - f68319f46e/rust/hyperlane-base/src/db/rocks/hyperlane_db.rs (L93) ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3915 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing E2E, by counting occurrences of the two logs linked above in relayer log output. Before this change, those logs would be missing completely --------- Co-authored-by: Paul Balaji <paul@hyperlane.xyz>	5 months ago
Daniel Savu	aecb65a986	feat: persist status labels (#4043) ... ### Description Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4003 to persist updates to an operation's status to the db, so we no longer default to `FirstPrepareAttempt` upon restarting the relayer ### Drive-by changes Moves `hyperlane_db` from `hyperlane_base` to `hyperlane_core` so it can be added to the `PendingOperation` trait. `PendingMessage` already stores a reference to the db in `ctx`, so I figured it doesn't break any assumptions. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Unit tests encoding of `PendingOperationStatus`, need to manually tests e2e	5 months ago
Daniel Savu	d213e32341	feat: relayer queue operation labels (#4003) ... ### Description - adds status labels to operations in the relayer queues. In practice, the most useful ones are those for when a message is retried due to an error. Labels are defined as enums to keep the cardinality finite, the size of prometheus metrics could get very large - whenever a message is pushed to a queue, the interface requires specifying the new status as a `Option<PendingOperationStatus>`. If `None`, the status is left as before (such as in the case when an operation isn't ready to be retried). - a limitation is that labels aren't stored in the DB, so messages are loaded with the `FirstPrepareAttempt` label. I just triggered retries for destinations I wanted to get labels for, but we could persist to the db in a follow-up PR - Example prep queue labels for inevm: <img width="621" alt="Screenshot 2024-06-21 at 18 05 58" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/23065004/f53b2858-371f-4b6b-86d9-5c21409d641f"> ### Drive-by changes - removes `make_op_try` as it doesn't really add anything that couldn't be done with regular Rust - bumps the `strum` version because there was a bug in the previous one, preventing inner enum fields from being converted to strings ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Yes ### Testing Manual	5 months ago
Daniel Savu	718d2c27da	chore: don't reset retry count when processing retry API reqs (#4037) ... ### Description When messages are retried due to the Message Retry API, we bring messages to the front of the queue (by changing their `next_attempt_after`), but also reset their retry count. For messages that are unprocessable due to various reasons (and hence have very high `next_attempt_after`), resetting the retry count will cause the relayer to waste time instead of trying to process newer messages. This PR makes it such that the retry API will cause messages to be retried immediately once, but won't boost their priority in the queue afterwards (because `num_retries` isn't changed). ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Paul Balaji	cc0ea73360	feat: pin monorepo to a specific registry version (#4041) ... - feat: pin monorepo to a specific registry version (cherry-pick #4031) - drive-by agent config update --------- Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Trevor Porter	a109937f96	chore: some throughput tweaks (#4036) ... ### Description Context: https://discord.com/channels/935678348330434570/1254036693217054794 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Snoppy	f1fe882e3b	chore: fix typos (#3645) ... fix minor typos Signed-off-by: snoppy <michaleli@foxmail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz>	5 months ago
Paul Balaji	c5a5e264b8	chore: enable `osmosis` on relayer (#4028) ... - chore: enable `osmosis` on relayer - drive-by add default chain signer/key/type to agent config - `signer.key` is copied from `neutron` --------- Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Paul Balaji	39f66df326	feat(ci): rust caching in `test` workflow (#4021) ... - brings https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4015 to cli-2.0 branch - cherry-pick e2e flake fix - ensure tests run on pushes to `cli-2.0` --------- Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com>	5 months ago
Trevor Porter	c84a1dc73b	feat: address blacklisting in the relayer (#4000) ... ### Description See context here https://discord.com/channels/935678348330434570/1236041564673806427/1252918894755319920 and https://discord.com/channels/935678348330434570/1252953507833708605/1252954456400592997 This is a rudimentary first step toward blacklisting addresses. Future options are outlined in https://discord.com/channels/935678348330434570/1236041564673806427/1252918894755319920. Eventually we can also fetch these addresses directly within the agent from an updated list that we cache and re-fetch occasionally. For now, this just relies on configured addresses. The approach is: - Allow hex addresses to be configured. These are intentionally treated as a Vec<u8> instead of an H160 / H256. This is because treating them as H160 is a leak of Ethereum-specific logic into non-chain-specific code in the relayer, and treating them as H256 would pad Ethereum addresses with a bunch of zeroes which would diminish the accuracy of our heuristic implemented in this. The subsequence stuff is a bit unfortunate because it's O(n^2) but 🤷‍♂️ - The heuristic is as follows - if any of the configured addresses are found as a subsequence in the message sender, recipient, or body, it's thrown out by the message processor, and the message will never make its way to the op_submitter. This heuristic for the body is to prevent warp route transfers that include that address. We could maybe convert Vec<u8> to H256 for the sender and recipient, I'm open to this - I just went with the substring approach bc it felt easiest tbh. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1285 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Daniel Savu	f227eec892	fix: e2e invariant flake (#4017) ... ### Description Context is explained in the comment, copying it here: ``` // TODO: Sometimes we find more logs than expected. This may either mean that gas is deducted twice for the same message due to a bug, // or that submitting the message transaction fails for some messages. Figure out which is the case and convert this check to // strict equality. // EDIT: Having had a quick look, it seems like there are some legitimate reverts happening in the confirm step // (`Transaction attempting to process message either reverted or was reorged`) // in which case more gas expenditure logs than messages are expected. ``` ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Paul Balaji	8ce2a71f20	chore: reconcile mainnet config json with registry (#4016) ... chore: reconcile mainnet config json with registry Signed-off-by: Paul Balaji <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Daniel Savu	3bb9d0a767	fix(cosmos): validator checkpoint & agent indexing improvements (#3958) ... - propagates RPC errors to the indexing logic, so errors cause sleeps instead of overwhelming the RPCs. This also greatly reduces cosmos log noisiness: from 9.5k logs per min, to 2.5k logs per min. - checks for published checkpoints in reverse, to prioritize recent messages among those from before the validator was spun up - fixes a validator bug where errors in checkpoint publishing would cause the entire checkpoint queue to be reiterated through, from scratch - Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3887 so we can test this out on osmosis ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Trevor Porter	bbbe93206f	chore: bump default ISM max depth from 5 to 7 (#3989) ... ### Description Needed for some Renzo messages that hit the max depth ### Drive-by changes - drive by to update the agent config with the updated registry ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	5 months ago
Paul Balaji	b0828b3d0e	feat: reintroduce `ism read` and `hook read` commands (#3990) ... the original functionality of these commands got lost across a couple of refactors 1. `ism read`/`hook read` was merged to become `core read`, which still had the purpose of "gimme config at this address" 2. then with a recent change, `core read` became "gimme the ism/hook configs relating to this mailbox address" This PR adds the original functionality back of returning the ISM/Hook configs for *any* ISM/Hook address (on EVM) drive-by: - update mainnet config - fix typescript linting errors/warnings	5 months ago
Paul Balaji	5cc64eb095	feat: deploy to new chains (fraxtal, linea, sei) (#3887) ... https://github.com/hyperlane-xyz/issues/issues/1279 - adds `fraxtal`, `linea`, `sei` - supports `osmosis` as a remote - ~drive-by hack to skip reconfiguring the same gas oracle (cc @tkporter)~ - drive-by fix to bring the logic of estimating tx + adding 10% buffer into the ism factory (as originally seen in the multiprovider handleDeploy function) - drive-by fix for creating custom safes - drive-by support of `osmosis` as a remote chain - drive-by addition of `osmosis` to the rust agent --------- Signed-off-by: Paul Balaji <paul@hyperlane.xyz> Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: nambrot <nambrot@googlemail.com>	5 months ago
Ayene	6171d82c82	chore: fix broken links (#3970) ... ### Description changed links pointing to Solana docs due to migration to different repository	5 months ago
Connor McEwen	897f627f09	fix: remove extra rust code from merge	5 months ago
Connor McEwen	ec5d43cc4c	chore: merge main to cli-2.0 (#3964)	5 months ago
Daniel Savu	12840964b9	chore: add all new chains to `KnownHyperlaneDomain` struct (#3960) ... ### Description So validators can be run with `--originChainName <chainname>`, like in the docs [here](https://docs.hyperlane.xyz/docs/operate/validators/run-validators#running-the-binary) <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	5a31e7b5d3	chore: debug slow relayer startup (#3932) ... ### Description Opened this PR to debug slow relayer startup, but it actually looks like that's already been fixed by hook indexing. The requirement for fast startup is that the processor sees new messages and then one of the following happens: - (1) if the message is meant for an unknown domain, it drops it ([this log](6d132b02a3/rust/agents/relayer/src/msg/processor.rs (L269))), otherwise - it forwards the message to the submitter, which then: - (2) adds it straight to the confirm queue because it's already been submitted ([here](6d132b02a3/rust/agents/relayer/src/msg/pending_message.rs (L159))) - (3) successfully builds metadata ([here](6d132b02a3/rust/agents/relayer/src/msg/op_submitter.rs (L231))) - (4) in some cases building metadata fails because the ism doesn't exist The relayer logs either of the four cases (in most cases (2), because we're running a second relayer for now). To be even more confident in fast startup, I ran a single relayer and restarted it. New messages were successfully processed according to one of the four cases above, and the prep queues didn't increase ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
overallteach	42a6dbc3e6	chore: remove repetitive words (#3644) ... ### Description <!-- What's included in this PR? --> remove repetitive words ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> Signed-off-by: overallteach <cricis@foxmail.com>	6 months ago
Paul Balaji	65e4b05f77	chore: fix mainnet3 agent config (#3931) ... chore: fix mainnet3 agent config follow-on from https://github.com/hyperlane-xyz/hyperlane-registry/pull/49 Signed-off-by: Paul Balaji <paul@hyperlane.xyz>	6 months ago
Daniel Savu	826b4ae57f	feat: batching igp (#3870) ... ### Description Deducts IGP cost from senders of messages within a batch, based on the total gas used by the batch and the estimates for submitting messages individually. The formula is `gas_used_by_operation = gas_used_by_tx * (operation_estimated_gas / total_estimated_cost)` Note that the individual estimate have a buffer value added to them (currently 75k), which will slightly skew proportions, though by a negligible amount. For example, given these estimates in a batch: 150k, 160k, 180k, if we add 80k to each, we go from e.g. 0.312 for the first message to 0.319 -> 2.25% error which isn't so bad. The error can be larger if operations within a batch have very different estimates, but realistically within a 5% range based on back-of-the-napkin calculations. ### Drive-by changes - The batching feature introduced an bug whereby gas expenditure would only be deducted in the `confirm` step. Now this is done in the `submit` step to account for txs that revert - Sealevel e2e can now be disabled for faster iteration! I've not done the cleanest job, but even the fact that we have this will reduce the time to run e2e locally by more than half. To use it, set `SEALEVEL_ENABLED=false` when running `run-locally` - e2e uses a new utility called `get_matching_lines`, that can be used to count (and in the future _parse_) logs, to reconstruct the state of the relayer and have more expressive correctness checks. This is used to make sure that gas is deducted for all messages, including those in batches. ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3709 ### Backward compatibility Yes ### Testing E2E, using `get_matching_lines`	6 months ago
Daniel Savu	942aba8e55	feat(evm): collaborative indexing through txid sharing (#3833) ... ### Description - improved indexing reliability by sharing channels between indexing tasks, such that they inform one another when they come across a hyperlane-related transaction - the receiving end will then get the receipt of that tx and filter it for events relevant to it - this adds some flexibility to indexing since we can stop relying on watermark cursor entirely: as long as we have one task using sequence indexing, all others can rely on being informed by their channels about what txs to index - this PR is only for the EVM chains, but can be easily extended ### Drive-by changes - Enforces gas payments for all chains in e2e (previously this only used to be the case for sealevel) - The channel size is configured per type, similarly to the indexing cursor, in `contract_sync/cursors/mod.rs` - Reduces log verbosity is some places, e.g. by manually implementing `Debug` to exclude fields that aren't relevant to debugging (maybe there's even a macro annotation to exclude, but I haven't checked) - Renames `MpmcReceiver` -> `BroadcastReceiver` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3267 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing e2e tested with the watermark cursor manually disabled	6 months ago
Daniel Savu	4261df74bb	chore: restrict for how long the backward syncer can block (#3918) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Noah Bayindirli 🥂	ba946f1eff	fix(config): switch to recommended 'rust-analyzer' from deprecated 'cargo' (#3886) ... ### Description - switch to recommended `rust-analyzer` since `cargo` is deprecated	6 months ago
Paul Balaji	d10b035ec2	fix: testnet4 holesky config (#3900) ... ran `yarn tsx ./scripts/agents/update-agent-config.ts -e testnet4` and updated testnet4 config Signed-off-by: Paul Balaji <paul@hyperlane.xyz>	6 months ago
Kunal Arora	554f875ebf	feat: core deployment on Holesky (#3776) ... ### Description Core deployment on Holesky ### Drive-by changes None ### Related issues - fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3597 ### Backward compatibility Yes ### Testing Manual	6 months ago
Trevor Porter	db9f733e4f	fix: use secret RPC urls in infra, apply environment-specific registry overrides (#3875) ... ### Description A while back, it seems secret RPC URLs stopped being used by monorepo services (see https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3179/files#diff-ba3b7bea2b5c046acd88d25ed382df875b49d77e061f2ef8347ffcc741c0353eR34). This adds them back. Some things to note on the approach: - Sadly, due to the way some of the infra config generation works, we're forced to use synchronous interfaces for getting chain metadata in a bunch of places. This is why `getRegistry` returns a `FileSystemRegistry`, which is a `SynchronousRegistry`. - Applying secrets / overrides requires us to do things async for two reasons -- one is just that fetching from GCP secrets is an async operation, the other is that a `MergedRegistry` is not a `SynchronousRegistry`. The approach I've taken is that there are two ways of getting a registry - using `getRegistry` will give the synchronous version that can continue to be used by our sync config generation, but it won't have any environment-specific overrides (tx overrides or secrets). To get an async registry with environment-specific overrides, the `getRegistry` function on the EnvironmentConfig is used. - Overrides are now more succinct as the merging happens later in the pipeline ### Drive-by changes - Started moving away from using the gcloud CLI under the hood for GCP operations and toward using the GCP secret SDK - Updated the mainnet3 key funder to fix an incident, but will follow up with a fresh deploy once this is mergd ### Related issues - These are the code changes for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3774, but I'll follow up with fresh deploys for the monorepo services ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Connor McEwen <connor.mcewen@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: J M Rossy <jm.rossy@gmail.com>	6 months ago
Yorke Rhodes	49f41d9759	chore: Merge branch 'main' into cli-2.0 (#3863) ... Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Connor McEwen <connor.mcewen@gmail.com> Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Avi Atkin <103125634+avious00@users.noreply.github.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Ali Alaoui <aalaoui2001@gmail.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz> Co-authored-by: omahs <73983677+omahs@users.noreply.github.com>	6 months ago
omahs	806f9842de	fix: duplicate word typos (#3854)	6 months ago
Daniel Savu	214f503e53	feat: forward-backward message processor (#3775) ... ### Description - adds logic for iterating message nonces in the processor task in forward-backward fashion - removes all manual calls that were updating the next nonce to query. This is done automatically in the iterator struct now - stores the highest known message nonce in the db, which is used to initialize the iterator ### Drive-by changes - Converts the concrete HyperlaneDb type stored in the processor to a trait obj, to enable mocking db responses ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3796 - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3816 ### Backward compatibility Yes - if there is no db entry for the new prefix, the processor starts from nonce zero, so no migration is required ### Testing Unit testing + e2e	6 months ago
Trevor Porter	37af3dc6ba	feat: deploy to zetachain and redstone (#3788) ... ### Description - Deploys to zetachain and redstone - Some learnings can be found in https://discord.com/channels/935678348330434570/1240168332813144084 Chains not yet fully connected: - inevm (Mailbox ownership situation makes this hard) - mantapacific (waiting on an update to the multisig ownership set there, then will create the tx) - non-EVM chains ### Drive-by changes - decreased ethereum cost from $2 to $1.5 following some feedback - Added Superform to the blast set - Allowed for concurrent deploys I originally had some ugly caching to avoid checking storage gas oracles so much (we check them for each route in a routing hook configuration), but I've removed this. Calling it out for the future though 232695dd6c ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	27aabf238c	feat: Relayer tokio task instrumentation (#3760) ... ### Description - Started off adding tokio-metrics but then realised those are quite general, so while we do have instrumentation it's not exposed in our metrics endpoint - switched to adding [tokio-console](https://github.com/tokio-rs/console/tree/main), which does give insight into the lifetime of specific tasks, so we can check which ones take up a long time during relayer startup. These are only visible at the `dependencyTrace` log level, so don't affect performance in the `hyperlane` context. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Helps debug https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3454 and any future performance issues - Does half the work for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3239 (still need to expose these in the metrics endpoint and import the grafana template) ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	1c4cf44c91	fix: allow all log levels in release build (#3824) ... ### Description Found the reason why we never had `trace` level logs in release builds - the `tracing` crate had a feature flag enabled that would wipe out all trace calls at compile time :( ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3823 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Trevor Porter	335a0d7479	feat: add agent config for Osmosis Mainnet (#3825) ... ### Description - Added agent config description for Osmosis Mainnet - Updated infra test to allow agent config chains to be a superset of infra chains Taken from @byeongsu-hong's https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3766, but with a fixed test ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: byeongsu-hong <hong@byeongsu.dev>	6 months ago
Trevor Porter	03ca2d1df7	Update gas oracles, deploy agents (#3755) ... ### Description - Updates gas oracles to make messages to Scroll slightly cheaper - After running into issues configuring non-EVM remotes in the IGP / gas oracle tooling (see https://discord.com/channels/935678348330434570/1238439687165579264), added a temporary band aid of just try/catching. We'll want to fix this at some point in the future - Deploys agents with a new image - Updates the agent JSON configs with some params we were running the agents with in prod via env vars - Adjusts kathy frequency on testnet to send every 1.5hrs instead of 7hrs - Stops running eclipsetestnet agents - apparently there was a reset ### Drive-by changes Some drive-by logging changes ### Related issues n/a ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	5d96847ffb	Multithreaded relayer (#3748) ... ### Description - Sets the tokio runtime flavor to be multithreaded and hardcodes the worker threads to 20. The latter usually defaults to the number of CPUs in the system so 20 may be high, but it worked fine in RC. Improved performance metrics: - time to first `Ingesting leaf` log: **1:30 mins (multithreaded) vs 5:51 mins (single threaded)** - time to first message in the submit queue of Optimism (one of the slowest chains to boot up for): **8:30 mins vs 20 mins** ### Drive-by changes Sets more descriptive `span`s when instrumenting the contract syncs ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility Yes ### Testing Manually, by checking the RC logs	7 months ago
Daniel Savu	be43b0fd91	feat: use batching configs on all execution environments (#3723) ... ### Description While we only have a tx batching implementation available for EVM chains, we still parallelize the `prepare` and `confirm` steps in the serial submitter according to the `maxBatchSize` config item. To reap the benefits of this change, we need to make these configs universal, because they were only defined within `h_eth::ChainConnectionConf`. This PR adds slight duplication because it adds the batching config on each individual connection config, but it's far simpler than including it in the `ChainConf` struct. The latter would've required changing all the trait impls of `BuildableWithProvider` to include the batching config. ### Drive-by changes Renames `serial_submitter` to `op_submitter` since it's no longer serial. ### Related issues Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3722 ### Backward compatibility Yes ### Testing Manually - added the configs to cosmos e2e and checked the logs	7 months ago
Daniel Savu	f46cf7708e	Indexing Type Wrapper (#3610) ... ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3544. Wraps all types to be indexed in an `Indexed<T>` struct in the relayer. This allows for using sequece-aware cursors for data that doesn't contain a sequence but can have it inferred from metadata (as is the case for the IGP on sealevel). `Indexed<T>` stores the sequence as an `Option`, so the db can have additional sequence data stored in it when the sequence is defined - this is how `process_indexed_gas_payment` is implemented now. In addition, the IGP payments and Deliveries are configured to be indexed with a `SequenceAware` cursor on sealevel (see `cursors/mod.rs`). ### Drive-by changes Implements the custom `Encode` and `Decode` traits for `IndexingDecorator`, which can be generalized to `Option<T>` where `T: Encode + Decode`. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing E2E	7 months ago
Daniel Savu	301239265e	WIP: don't hardcode multicall gas limit (#3710) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	8e5ef272a6	fix: agent config check, run relayer on mantapacific (#3714) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	22f367ff6c	fix: sleep after submitting to cosmos chains (#3699) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago