103 Commits (1a8cc278183188bb0f31cb40fa16b131f606da5d)

Author	SHA1	Message	Date
Daniel Savu	8adcd60402	chore: add `mainnet` suffix to solana and eclipse (#4410) ... ### Description Because we're doing a new core deployment that includes protocol fees ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Danil Nemirovsky	b6cea10b63	feat: Add scraper to Cosmos E2E test (#4387) ... ### Description Add scraper to Cosmos E2E test * Add minimal functionality to scraper for cosmos so that scraper does not crash * Add scraper to Cosmos E2E test ### Backward compatibility Yes ### Testing E2E Tests for Cosmos --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 months ago
Trevor Porter	57bbb16cc1	feat: panics upon startup when RPC issues occur include the offending chain (#4323) ... ### Description Some quick low hanging fruit to make debugging easier in the future. Today (https://discord.com/channels/935678348330434570/1273582453889699843) some chains were having RPC issues and it caused an agent deploy to panic upon startup. Debugging this was hard because the panic message just indicates an error is occurring, but doesn't say what the offending chain is! Examples: - https://cloudlogging.app.goo.gl/puQk7hm4fXyJhYKs8 - https://cloudlogging.app.goo.gl/5qZrbc2Ww4LWQXWC6 - https://cloudlogging.app.goo.gl/ZL7mKbBzzJaePtty9 All these panic happen when constructing the cursor, which isn't yet done in an indexing task. The quick and easy fix is to just use expect instead of unwrap We still unwrap in a few places, but mostly in places that aren't susceptible to flaky RPC issues, e.g. reading from a HashMap that we know should always have an entry, etc ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	3 months ago
Paul Balaji	2d922805c0	feat: add arbitrumsepolia, basesepolia, ecotestnet, optimismsepolia, polygonamoy to core chains (#4297) ... Deploy to arbitrumsepolia, basesepolia, ecotestnet, optimismsepolia, polygonamoy	3 months ago
Danil Nemirovsky	d1bf212bf7	feat: Report git commit hash as agent metadata (#4282) ... ### Description * We are storing git commit hash as an environment variable when we compile the source code of an agent. * Validator uses git commit hash as a agent version in `metadata_latest.json` file. * File `metadata_latest.json` is overwritten every time when validator is restarted. * Git commit hash is extracted by `git` util, so, it should be available during build time. * If the directory with the source code does not contain `.git` folder, git commit hash will be defaulted to `VERGEN_IDEMPOTENT_OUTPUT`. Agent Metadata is propagated to Relayer and Scraper, but is not used there. We can use it in the future, reporting git hash commit in the logs, for example. ### Backward compatibility Yes ### Testing * Tested locally with E2E Tests. * Tested against S3 by deploying docker image into k8s --------- Co-authored-by: Danil Nemirovsky <4614623+ameten@users.noreply.github.com>	3 months ago
Paul Balaji	103e6b3e12	feat: deploy to `zircuit` (#4265) ... - feat: deploy to `zircuit` - update registry + agent config	4 months ago
Daniel Savu	a49f52cc9c	feat: deploy solana, eclipsemainnet (#4238) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: pbio <10051819+paulbalaji@users.noreply.github.com>	4 months ago
Paul Balaji	ff3c9897ea	chore: migrate `test1` from `13371` to `133711` (#4252) ... - chore: migrate `test1` from `13371` to `133711` - due to clash with immutablezkevm - drive-by registry update + agent config update --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	4 months ago
Mohammed Hussan	8e942d3c6b	feat: deploy cheesechain, worldchain, xlayer (#4207) ... ### Description - Support deployment to cheesechain, worldchain and zlayer ### Drive-by changes - update `CosmosChainSchema` to add `chain_type` to fix breaking change introduced by 6854133f98	4 months ago
Daniel Savu	b643fba90a	feat: use multiple `mpsc` channels instead of `broadcast` for async interface (#4190) ... ### Description Follow up to https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4180 Lowers txid broadcast channel capacity from 1M to 30k and refactors it to use a vec of `mpsc` under the hood. ### Context The heap profiler showed that each new chain takes up 100MB of memory because of the txid broadcast channel. In addition to this, the tokio broadcast channel is a ring buffer where new items simply overwrite old items, making it hard to safely lower the capacity. As such, to be able to safely lower the capacity of this channel, usage of `tokio`'s `broadcast` channel is replaced with a `Vec` of `mpsc`s, which offer an async `send` interface for backpressure. So instead of overwriting old items, senders will block on sends, which makes sense in our case as we absolutely don't want to miss items. The submitter is slower than the indexer based on empirical observation, so blocking on channel size doesn't bottleneck throughput. I noticed the max number of channel items we can reach is 29k, by running the RC relayer after an 8 day break, during which around 160k messages were submitted. Lowering the capacity to 30k should cause us to only encounter backpressure in very rare cases, and the footprint of this channel would lower from 100MB to 3MB. ### Drive-by changes The delivery indexer in the scraper is no longer passed a txid receiver, since message deliveries are not expected to occur in the same tx as message dispatches. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/1293 ### Testing E2E, by adding invariants to make sure txid indexing works --------- Co-authored-by: Trevor Porter <tkporter4@gmail.com>	4 months ago
Paul Balaji	0dedbf5a02	feat: deploy to new chains (endurance, fuse, zora) (#4148) ... - feat: deploy to new chains (endurance, fuse, zora) - https://github.com/hyperlane-xyz/issues/issues/1308 - also includes https://github.com/hyperlane-xyz/issues/issues/1305 - registry PR https://github.com/hyperlane-xyz/hyperlane-registry/pull/104	4 months ago
Paul Balaji	d316772243	feat: deploy to BOB, Mantle, Taiko (#4051) ... resolves https://github.com/hyperlane-xyz/issues/issues/1289 - feat: deploy to BOB, Mantle, Taiko - drive-by: fix deploying to specific chain - drive-by: adding AVS operators for eth - drive-by: support non-evm remotes in mainnet3 core config --------- Signed-off-by: pbio <10051819+paulbalaji@users.noreply.github.com>	5 months ago
Trevor Porter	6f4ef05d41	feat: faster way of figuring out how many new scraper rows were inserted (#4073) ... ### Description ### Context Full context / thread: https://discord.com/channels/935678348330434570/1254871503825141923 Copied here: so this is kinda interesting basically the gas payment indexing in the scraper task is consistently prioritizing hook-based indexing work over block range indexing work this only manifests in the scraper and for arbitrum because: there are a lot of arbitrum origin messages on the scraper, each gas payment indexing tick takes a very long time because of some very suboptimal queries. store_payments can take like 10-15 seconds! You can see batches of logs Found log(s) in index range https://cloudlogging.app.goo.gl/Bo9Q7YwyziSqjsEu7 that correspond with the gas payment contract sync block height (green line) advancing https://abacusworks.grafana.net/goto/poW4KOQIg?orgId=1  but we're still indexing gas payments bc of hook indexing - there are tons of Found log(s) for tx id https://cloudlogging.app.goo.gl/QvDaBuf67SjjKviw6 In the scraper's store_payments we bizarrely end up performing 2 very expensive queries with the calls to payments_count - it gets the count of gas payments for the provided domain so it can figure out how many new gas payments were actually inserted https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/agents/scraper/src/db/payment.rs#L27-L64 These queries vary a bunch in how expensive they can be - sometimes each one is like 1-2 seconds, sometimes it gets up to 10 seconds each though https://cloudlogging.app.goo.gl/cDFNH3oxh6WWV3Go8  easiest fix is probably to figure out how to make store_payments less expensive - we probably don't need to be making these calls to payments_count to figure out which ones are new which ill do atm there's maybe a decent argument to having hook indexing and range indexing not contend with one another but we should be able to kick that can also an annoying thing is https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/hyperlane-base/src/contract_sync/mod.rs#L155 this log not showing up, probably some weird instrumentation setup or something makes debugging harder ### The fix We want to keep supporting the possibility that we've double-indexed a data type, and to only return the number of new rows inserted. We want to keep each indexing task as non-conflicting as possible - we know that only one scraper is running at a time, but that a task for each chain and data type is executing concurrently. It's possible for e.g. two gas payment indexing tasks to try to insert into the same table concurrently, but it'll be for different domains. Insert statement concurrency issues aren't an issue because the only chance for conflict is the auto-incrementing primary key id, but this is all handled properly by database internals. Insert results only give the latest ID resulting from the insertion. E.g. if no matter how many rows were inserted, it gives the latest ID from that batch, but no info about how many were in the batch. We could do something like get the latest id in the table for all domains (this is cheap) before, then do the insert, and then get the latest id again, but this is susceptible to race conditions if concurrent tasks get an insertion in during that time as well. Shoving all these into a transaction doesn't seem to solve the problem either unless there's a lock on the table. However, sea-orm doesn't support table level locks, just select locks. We could also use a CTE to perform as a way to perform the select & insertion atomically and return the value of the select - but again sea-orm doesn't seem to provide a nice way of doing this from what I could tell. Instead, my suggested fix is: 1. Get the latest id in the table relating to the specific domain (this is cheap, < 100ms) 2. Perform the insertion 3. Count how many new rows for the specific domain now exist whose id is > the id from (1). This is also cheap, < 100ms. If there is a very long time since the last insertion for the domain, this could take longer, but in practice this doesn't seem to be the case. If this couple order of magnitude improvement still poses issues in the future, we can consider other alternatives, probably involving locking or fancier CTE queries. Some example queries and how long they take: Before (2x of these would occur!): <img width="477" alt="Screen Shot 2024-06-28 at 12 36 41 PM" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/5cc8ac35-74fb-4847-9c6f-448deb253a0c"> Now: 1. Max id for a given domain <img width="494" alt="Screen Shot 2024-06-28 at 12 37 00 PM" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/56dbb3ff-b540-458c-809c-56ebfbfd3b0f"> 2. Subsequent count but with an ID filter - note I actually changed this query to look for IDs > 3600000, which includes 55300 rows and it's still just 102ms! <img width="632" alt="Screen Shot 2024-06-28 at 12 37 49 PM" src="https://github.com/hyperlane-xyz/hyperlane-monorepo/assets/20362969/11593180-682f-4867-9867-97fd93512802"> ### Drive-by changes - Updated the neutron image along with a deploy there ### Related issues ### Backward compatibility ### Testing Ran some sorta ad-hoc unit tests to test my new query by running against the prod db with a tokio::task. Tested for a new domain and an existing one	5 months ago
Paul Balaji	5cc64eb095	feat: deploy to new chains (fraxtal, linea, sei) (#3887) ... https://github.com/hyperlane-xyz/issues/issues/1279 - adds `fraxtal`, `linea`, `sei` - supports `osmosis` as a remote - ~drive-by hack to skip reconfiguring the same gas oracle (cc @tkporter)~ - drive-by fix to bring the logic of estimating tx + adding 10% buffer into the ism factory (as originally seen in the multiprovider handleDeploy function) - drive-by fix for creating custom safes - drive-by support of `osmosis` as a remote chain - drive-by addition of `osmosis` to the rust agent --------- Signed-off-by: Paul Balaji <paul@hyperlane.xyz> Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: nambrot <nambrot@googlemail.com>	5 months ago
Daniel Savu	942aba8e55	feat(evm): collaborative indexing through txid sharing (#3833) ... ### Description - improved indexing reliability by sharing channels between indexing tasks, such that they inform one another when they come across a hyperlane-related transaction - the receiving end will then get the receipt of that tx and filter it for events relevant to it - this adds some flexibility to indexing since we can stop relying on watermark cursor entirely: as long as we have one task using sequence indexing, all others can rely on being informed by their channels about what txs to index - this PR is only for the EVM chains, but can be easily extended ### Drive-by changes - Enforces gas payments for all chains in e2e (previously this only used to be the case for sealevel) - The channel size is configured per type, similarly to the indexing cursor, in `contract_sync/cursors/mod.rs` - Reduces log verbosity is some places, e.g. by manually implementing `Debug` to exclude fields that aren't relevant to debugging (maybe there's even a macro annotation to exclude, but I haven't checked) - Renames `MpmcReceiver` -> `BroadcastReceiver` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3267 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing e2e tested with the watermark cursor manually disabled	6 months ago
Yorke Rhodes	49f41d9759	chore: Merge branch 'main' into cli-2.0 (#3863) ... Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Connor McEwen <connor.mcewen@gmail.com> Co-authored-by: Trevor Porter <tkporter4@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Avi Atkin <103125634+avious00@users.noreply.github.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Ali Alaoui <aalaoui2001@gmail.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Paul Balaji <paul@hyperlane.xyz> Co-authored-by: omahs <73983677+omahs@users.noreply.github.com>	6 months ago
omahs	806f9842de	fix: duplicate word typos (#3854)	6 months ago
Daniel Savu	27aabf238c	feat: Relayer tokio task instrumentation (#3760) ... ### Description - Started off adding tokio-metrics but then realised those are quite general, so while we do have instrumentation it's not exposed in our metrics endpoint - switched to adding [tokio-console](https://github.com/tokio-rs/console/tree/main), which does give insight into the lifetime of specific tasks, so we can check which ones take up a long time during relayer startup. These are only visible at the `dependencyTrace` log level, so don't affect performance in the `hyperlane` context. ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Helps debug https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3454 and any future performance issues - Does half the work for https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3239 (still need to expose these in the metrics endpoint and import the grafana template) ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	6 months ago
Daniel Savu	f46cf7708e	Indexing Type Wrapper (#3610) ... ### Description Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3544. Wraps all types to be indexed in an `Indexed<T>` struct in the relayer. This allows for using sequece-aware cursors for data that doesn't contain a sequence but can have it inferred from metadata (as is the case for the IGP on sealevel). `Indexed<T>` stores the sequence as an `Option`, so the db can have additional sequence data stored in it when the sequence is defined - this is how `process_indexed_gas_payment` is implemented now. In addition, the IGP payments and Deliveries are configured to be indexed with a `SequenceAware` cursor on sealevel (see `cursors/mod.rs`). ### Drive-by changes Implements the custom `Encode` and `Decode` traits for `IndexingDecorator`, which can be generalized to `Option<T>` where `T: Encode + Decode`. ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing E2E	7 months ago
Trevor Porter	d6f25ed035	Deploy to Mode & Blast (#3653) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Trevor Porter	700240eb8a	Ancient8 new ism, misc infra fixes (#3606) ... ### Description - Deploys ancient8 with a new ISM where we are not quorum - Pulls out warp config to live in a single file / function so that deploy.ts and check-deploy.ts are in sync. Otherwise fork.sh is broken for warp routes - Shuffles around the aw multisig serialization stuff - now they're segmented by environment and context and are written to `config/{env}/aw-validators/{context}.json`. This was in response to some confusing merging logic, see https://discord.com/channels/935678348330434570/1228377321082327141/1229464197297868910 ### Drive-by changes Needed to specify pino to work in pretty mode for warp monitoring :/ this is necessary because pino doesn't work otherwise. After #3582 we should change this ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Daniel Savu	f4626eb4fe	Allow protocol-specific indexing (#3544) ... ### Description - Generalizes contract syncer hashmaps to store trait objects instead of concrete structs, since both `SequencedDataContractSync` and `WatermarkContractSync` implement `ContractSyncer` - The nice thing this enables is being able to instantiate a different type of syncer for the same contract, for different chain protocols (e.g. for IGP, we can use a sequenced syncer on sealevel and cosmwasm, and a watermark syncer on EVM). - The problem is that `ContractSyncer` is implemented on different trait bounds on `SequencedDataContractSync` and `WatermarkContractSync`. Most notably, the `SequencedDataContractSync` impl bounds the indexed type by the `Sequenced` trait. This means that if a type is to be indexed by more than one syncer strategy, it needs to implement the union of bounds of those strategies - **Blocker:** In the IGP, this means that `InterchainGasPayment` needs to implement `Sequenced` and we have no obvious way of getting that just from the type. Even if the sequence is derived from the type, it has to be a sensible value, since indexing reliability depends on it. - Similar to the point above, we now have `contract_syncs`, a single function that instantiates contract syncs for all chains. Since `contract_syncs` must be able to build a `ContractSyncer` trait obj for all syncer types, it bounds both `T` (the indexed type) and `D` (the log store type) with a union of bounds, so it may be too strict for some types in the future. - The scraper now uses forward-backward indexing instead of just forward indexing, because that simplified the refactor - there's now a convenience trait called `TryFromWithMetrics` that allows adding this bound: `SequenceIndexer<T>: TryFromWithMetrics<ChainConf>,`. This isn't really extendable if we use non-sequence indexers, but it's fine for now since this is used everywhere - replaces the macro-based contract syncer builder with generic fns Still todo: - add new `IndexingDecorator<T>` struct that can wrap `InterchainGasPayment`, so we can use sequence-aware indexing for the IGP - remove sequence logic from the watermark cursor ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	7 months ago
Noah Bayindirli 🥂	38358ececd	feat: deprecate Mumbai testnet (#3521) ... ### Description - Removes all reference to Mumbai testnet, since it is soon being [deprecated](https://polygon.technology/blog/polygon-pos-is-cooking-the-napoli-upgrade-means-better-ux-the-mumbai-testnet-takes-a-bow) by the Polygon team - Monorepo docker tag: https://console.cloud.google.com/gcr/images/abacus-labs-dev/global/hyperlane-monorepo@sha256:d4b44359cae9c4e3bdeb6f965c59d056f7fce72c42069adc0dea038ac600785a/details?authuser=5&project=abacus-labs-dev&tab=info - Agent docker tag: https://console.cloud.google.com/gcr/images/abacus-labs-dev/global/hyperlane-agent@sha256:a578d96f7f06721d0b6cd8095d6a1fa5d22f01b59eba3320ee3b1fe34ca3c5df/details?authuser=5&project=abacus-labs-dev&tab=info ### Drive-by changes - Adds missing Testnet domains, Fuji & BSC Testnet - Fix for failing `test-rs` CI check with ``` error[E0004]: non-exhaustive patterns: `chain::KnownHyperlaneDomain::Fuji` and `chain::KnownHyperlaneDomain::BinanceSmartChainTestnet` not covered --> hyperlane-core/src/utils.rs:226:15 | 226 | match $v { | ^^ patterns `chain::KnownHyperlaneDomain::Fuji` and `chain::KnownHyperlaneDomain::BinanceSmartChainTestnet` not covered ``` ### Related issues - Dependency on #3523 - Fixes #1181 ### Backward compatibility - [x] Backwards compatible - [x] Requires docker image tags to be added post-build - [x] Requires agent artifact cleanup in cloud account: https://www.notion.so/hyperlanexyz/Runbook-83c755f2652943289cf98cd1309487a8?pvs=4#77730653a12f409dacfbd427cab545d3 ### Testing * Build success * `yarn hyperlane chains list -e testnet` does not return `mumbai` * fork.sh script success locally	8 months ago
Daniel Savu	6e39cf01af	Relayer retry endpoints (#3476) ... ### Description Adds a `/message_retry` relayer endpoint, which supports retrying OpQueue operations either by ID or by destination domain. Such a request will be sent to an `mpmc` channel's receiving end in the OpQueue, which is emptied when `OpQueue::pop` is called. Example calls: ``` GET http://127.0.0.1:60843/message_retry?destination_domain=42 GET http://127.0.0.1:60843/message_retry?message_id=0x46910b1329ee53c86a023b322e9ca1c17e5f9f0bee789c77b0abced0a173d714 ``` If the endpoint is called specifying both filters, **two** requests will be sent across the channel, one for each condition. Efficiency note: The entire queue is iterated over if there's at least one retry request in the channel. The good thing is that rebuilding the heap is `O(n)` so not too bad. ### Drive-by changes - Removed the usage of `enum_dispatch`, because it would have made unit tests messy. The drawback is that we have to work around object safety restrictions in `PendingOperation` - for instance it can't be cloned (but can be stored behind an `Arc` if we ever need this). - Added new methods to `PendingOperation`, either to match by ID, or to implement heap element prioritization without depending on the concrete `PendingMessage` type (as was done before). Additions: `id()`, `origin_domain()`, `priority()`. Don't think this conflicts with any operations we may add in the future (e.g. gas oracle updates). - Although `tokio`'s broadcast channel is multi-producer-multi-consumer by default, the consumer end isn't `Clone` - you instead need to have a producer to call `.subscribe()` on to get a new consumer, so I added an `MpmcChannel` struct to encapsulate keeping a producer around to get new consumers. - OpQueue is moved into its own file and has unit tests added for retries (covering broadcast to multi-queue, and both retry types). - Did a bit of agent `server `cleanup, including `EigenNodeApi` ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3098 ### Backward compatibility Yes ### Testing Unit tests for the server route (channel transmitter logic) and for the OpQueue retries (channel receiver logic).	8 months ago
Paul Balaji	76bd730103	feat: remove goerli testnets (#3389)	9 months ago
Trevor Porter	9a83eb4d57	Sequence-aware indexing refactor (#3262) ... ### Description Part 1 of an indexing refactor. This mostly focuses on the sequence-aware cursors. Changes: - Some renames, e.g. `SequenceIndexer` -> `SequenceAwareIndexer`, `sequence_and_tip` -> `latest_sequence_count_and_tip` - Moved cursors out of `contract_sync/cursor.rs` into `contract_sync/cursors/*` - The goal was to make the the Forward sequence-aware cursor, the Backward sequence-aware, and the ForwardBackward sequence aware cursors more resilient, easier to understand, and better tested. - General strategy for doing this was to: - Only move onto a new range to index if the logs passed into the cursor's `update` function justify this - If updated with any weird data, like gaps in logs, rewind the cursor and retry previously queried ranges I'd recommend starting the review in the following order: 1. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/forward.rs` - where `ForwardSequenceAwareSyncCursor` & tests live 2. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/backward.rs` - where `BackwardSequenceAwareSyncCursor` & tests live 3. `rust/hyperlane-base/src/contract_sync/cursors/sequence_aware/mod.rs` - where `ForwardBackwardSequenceAwareSyncCursor` lives ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	c037206928	remove leftover basegoerli mentions (#3290) ... ### Description The scraper is still trying to connect to basegoerli although this network is now down ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Kunal Arora	13fb461888	feat:add eigen node endpoints to agent API (#3095) ... ### Description - compatibility with eigenlayer's node spec here: https://eigen.nethermind.io/docs/spec/api/ - changing from hyper to axum as a preferred choice of server handling - adding server trait to validator, relayer, and server (to customize behavior for each in the future) ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - fixes https://github.com/hyperlane-xyz/issues/issues/853 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Trevor Porter	6d4266ae93	Allow unknown domains in the scraper (#3276) ... ### Description Spotted a regression ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	155c85eda2	Infallible relayer tasks (#3257) ... ### Description The intended behavior of agents is described by @tkporter's comment here: https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3257#discussion_r1489233847 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3209 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	3cbb06a07a	feat: fetch chain-specific metrics in standalone task (#3214) ... ### Description Spawning a tokio task to fetch chain-specific metrics in provider middleware was wasteful because we ended up with 15 tasks doing the same job. Now there is a single task that is spawned inside the relayer, which I added in the struct that used to only fetch relayer balance. Some questions: - I'm realising in the current state I'm probably removing these metrics from validators. Lmk if yes and I'll add back - the chain-specific metrics were moved out of `MiddlewareMetrics` because they're no longer part of middleware, but I wasn't sure where to place them. Maybe `CoreMetrics` is a better place than the current `custom_metrics.rs` file? `tokio-metrics` turned out not to be useful because we'd need to instrument every call site of `tokio::spawn` with it. We should still do it but as a separate task imo. ### Drive-by changes - This PR also makes it very easy to add the metrics tasks for new chains, by extending the `HyperlaneProvider` trait with a `get_chain_metrics` call which seems reasonably general to me to have implemented by all providers. (Bc we currently only track these for evm chains) - the description, naming and logic of the `gas_price` metric was also changed to support new chains ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3047 ### Backward compatibility Yes ### Testing Still need to manually test	10 months ago
Trevor Porter	473b43f001	Injective & inevm agents (#3155) ... ### Description - Deploys injective & inevm validators, adds them to the hyperlane context relayer, and starts scraping inevm ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: nambrot <nambrot@googlemail.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com>	10 months ago
Yorke Rhodes	8fe24830ee	Deploy viction core and warp routes (#3067) ... ### Description Deploys core and warp to viction ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/850 - Fixes https://github.com/hyperlane-xyz/issues/issues/859 ### Backward compatibility Yes ### Testing Fork tests --------- Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	10 months ago
Daniel Savu	8e44bc1b81	feat: sealevel merkle indexer + e2e (#3053) ... Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3048 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3046 Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2754	11 months ago
Daniel Savu	77aa58c581	Relayer balance metrics (#2976) ... Done: - scaffolding for fetching custom agent metrics - abstractions for building a metrics fetcher for a given VM - querying cosmos balances; e2e tested. - querying evm balances; e2e tested. - **Note that as a result, evm addresses are now no longer zero-padded when printed in the logs. This may break existing log queries** - fixed a nasty bug on ubuntu where wasmd (osmosisd dependency, part of the grpc query flow) would panic when a block is specified via `x-cosmos-block-height`. The fix was to bump the version of osmosisd from `19.0.0` to `20.5.0`. **Note that when running e2e on Mac OS, the osmosis version in use is still 19.0.0**. That's because we need a fork that publishes a darwin target binary (currently pointing [here](https://github.com/hashableric/osmosis/releases/download/v19.0.0-mnts/osmosisd-19.0.0-mnts-darwin-arm64.tar.gz)) For follow up PR: - sealevel balance querying I'm open to all renaming suggestions, I just tried to speed through and didn't ponder names too much Relates to https://github.com/hyperlane-xyz/issues/issues/701 Closes https://github.com/hyperlane-xyz/issues/issues/702 (because the balance becomes available in the metrics endpoint for polling)	12 months ago
Trevor Porter	ece2be507a	Cosmos / CosmWasm agents (#2865) ... ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: hashableric <hashableric@gmail.com> Co-authored-by: byeongsu-hong <hong@byeongsu.dev> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz>	1 year ago
Kunal Arora	c44fd29914	Deploy contracts to mainnet3 (#2826) ... ### Description - Deploys mainnet3 (and removes mainnet2) ### Drive-by changes - Changes protocol fees to 0 - Changes owners to deployer key (temporarily) ### Related issues - closes https://github.com/hyperlane-xyz/issues/issues/655 ### Testing Some v2 related tests broken --------- Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	1 year ago
Yorke Rhodes	1d18549755	Merge main to v3 (#2812)	1 year ago
Kunal Arora	9168cca6d2	validator deploy tweaks (#2784) ... ### Description - fixes to deploy testnet4 for agents ### Drive-by changes - truncating labels to length 63 - adding `.relayer` to `agentConfig.contextChainNames` ### Related issues - for https://github.com/hyperlane-xyz/issues/issues/574 ### Backward compatibility Yes ### Testing Manual --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com>	1 year ago
Daniel Savu	c6cea785ae	V3 agents feedback (#2780) ... ### Description Part of the remediations from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2746 --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Kunal Arora	35c176fe90	Testnet4 deployment (#2781) ... fixes https://github.com/hyperlane-xyz/issues/issues/574	1 year ago
Daniel Savu	af5bd88aa8	V3 agents rebase (#2746) ... ### Description It's your favourite PR coming right back... V3 agents! Closes https://github.com/hyperlane-xyz/issues/issues/561 Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2742 Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2681 for e2e testing This PR includes: - [x] Merkle tree hook indexer - [x] Merkle tree builder task - [x] Update submitter to trigger retries if no proof is available yet Slightly more detailed overview of the work here: https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2720#issuecomment-1724038643 <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: -f <kunalarora1729@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Guillaume Bouvignies <guillaumebouvignies@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: Guillaume Bouvignies <guillaume.bouvignies@kurtosistech.com>	1 year ago
Mattie Conover	741c9bb553	Update Rust (#2765) ... ### Description Regular rust version and package updates. ### Drive-by changes None ### Related issues ### Backward compatibility Yes ### Testing Unit Tests	1 year ago
Mattie Conover	17a6e79ea5	Rust agent config parsing (#2687) ... ### Description This does two things: 1) It parses the agent-specific configs for the relayer, validator, and scraper. 2) It removes the rigid structures that had been in place for parsing that were called out earlier and instead parses the serde json values directly. This PR does not cut over to the new versions just yet. That can happen once we start integrating it into the testing and deployment pipelines. ### Drive-by changes - Derive more was added and used where it made sense - The old config macro was removed ### Related issues - Progress on #2215 ### Backward compatibility Yes ### Testing None	1 year ago
Mattie Conover	389029dafb	Agent Config Base Parsing (#2665) ... ### Description This is a big one for #2215... 1. Splits the parsing of configs from the config definitions to support both the old, new, and future parsers simultaneously. (We might want to rename it from parser to config format or something; not set on this name). 2. Implements the new config format without switching to it or enabling it. This means this PR does introduce dead code that is going to be used in the next PR. ### Drive-by changes - Deleted some unused (or single use) macros defined in hyperlane base - Updated the definition of the config parsing trait to simplify implementation - Added some more ergonomic methods of handling errors in config parsing - Removed any raw `unwrap()` I noticed since these cause difficult to debug errors in prod. - Added some additional validation to catch code mistakes in config path definitions ### Related issues - Progress on #2215 - Comes after #2658 ### Backward compatibility Yes ### Testing Manual (E2E only for now to make sure I did not break the old config format)	1 year ago
Daniel Savu	326eced987	Revert "refactor(IndexMode): remove `IndexRange`… (#2562)" (#2655) ... This reverts commit 127294decc, to include a cursor bugfix in main and remove an IGP indexing bug added by that PR. ### Description <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	1 year ago
Mattie Conover	644db5eee1	Remove unused tracing logic (#2639) ... ### Description When I was working on #2215 I noticed that we are not using either zipkin or jaeger at all and possibly never have. Ripping these out to reduce dead code and also simplify our package dependencies. ### Drive-by changes None ### Related issues Discovered through #2215 ### Backward compatibility Yes - we were not using these features and did not document their existence ### Testing Manual - Verified that when running deploy-agents, the generated configs do not have jaeger or zipkin enabled on `main` and then tested to make sure they still work in this branch	1 year ago
Mattie Conover	041cf92d57	Fix insertion errors for txn and block (#2629) ... ### Description This resolves the insertion conflict that is being caused by #2246 that introduced multiple concurrent tasks to scrape. This does not address the underlying concurrency issue I discovered, however. Please review the issue comments for more details. ### Drive-by changes - Cleans up logs a bit (This was useful during testing and I think is useful globally) - Removed an unused file ### Related issues Fixes hyperlane-xyz/issues#493 ### Backward compatibility Yes ### Testing Manual	1 year ago
Mattie Conover	8cbfd3cbc0	Support 64-byte txn ids (attempt 2) (#2612) ... ### Description Adds support to TxOutcome for 512-bit ids and to log meta to support Solana. Now that we have primitive types in our crate we were able to create a better solution than before which just stores the EVM 256bit hashes inside of the 512bit one with leading zeros rather than needing to over engineer an enum container for it. ### Drive-by changes - Removed some dead code - Improved primitive type support for 512bit types - Improved serialization support for 512bit types ### Related issues - Fixes #2252 - Replaces #2256 ### Backward compatibility _Are these changes backward compatible?_ Yes-ish, changes one of the internal database key strings. At standup we discussed the possibility of migrating the data and concluded that was a can of worms we did not want to open. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit tests	1 year ago
Mattie Conover	a5e1815746	Rust updates (#2619) ... ### Description Let this one slip a while. Updates our version of rust and as many packages as did not conflict with solana. I think we are going to need to figure something better out going forward with solana since if we have a couple more envs like this, it is going to make updates very hard. ### Drive-by changes None ### Related issues General Maintenance ### Backward compatibility Yes ### Testing Manual & Unit Tests	1 year ago