140 Commits (8aa7b62beae417323c4f60a2a0c315508a1fe598)

Author	SHA1	Message	Date
Asa Oines	8aa7b62bea	Minor logging improvements to agents (#2271) ... ### Description Removes some unnecessary clutter, adds domain to some spans ### Drive-by changes None	2 years ago
Asa Oines	73fb967ec2	Properly store gas payment expenditure data in agent rocks db (#2268) ... ### Description This PR fixes a bug that resulted in gas payment expenditures not being properly stored in the rocks DB, making them unretrievable. ### Drive-by changes - Introduces a macro to HyperlaneRocksDB to prevent similar bugs from happening in the future	2 years ago
Asa Oines	63562c7211	Refactor agent event indexing (#2246) ... ### Description This PR refactors event indexing in the agents, allowing similar logic to be shared across multiple event types (i.e. messages, deliveries, gas payments) and database types (i.e. the Relayer rocks DB and Scraper SQL DB). Furthermore, it adds new syncing modes by way of `MessageSyncCursors` that take advantage of the monotonically increasing dispatched message nonce to sync more intelligently. ### Drive-by changes - Fixes a bug in the existing cursor that caused the same block range to be indexed three times - Modifies kathy to get rid of the idea of "rounds", just sends messages with a sleep in between - Minor modifications to the e2e test for performance - Expand macros in settings - Add scraper to e2e test ### Opportunities for improvement - We can further reduce RPC usage (or improve latency) by sharing the view of the latest finalized block number between cursors - We can speed up the effective time for (a relayer to start deliving messages | the scraper to scraper recent events) by creating forward/backward cursors for gas payments and deliveries where the backwards cursor terminates at index_settings.from - We can remove the need for index_settings.from by terminating backwards cursors based on the block number that the first message was dispatched at ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E tests --------- Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Mattie Conover	5e38cdf31a	Omniscient Relayer (#2195) ... ### Description Adds support for running a relayer which delivers messages from multiple origin chains instead of just one origin chain. ### Drive-by changes _Are there any minor or drive-by changes also included?_ - The E2E pipeline got a major re-work. It both runs faster and also tests the new changes. - SerialSubmitter has been abstracted to support additional operation types such as oracle updates - SerialSubmitter has been updated to actually confirm a transaction was included and is safe from re-orgs - SerialSubmitter has been updated to have higher throughput by pipelining operations ### Related issues - Blocked by #2148 - Fixes #1899 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	24f9642ea5	Agent docs and config improvements (#2148) ... ### Description Bunch of minor tweaks and updates based on what we learned from EthTokyo. - Updated Rust readme and the root readme to align with the hyperlane docs - Defaults to a 5 sec validator interval now - Makes the metrics port default to `9090` and have a soft failure if it can't start - Now will automatically create the checkpointsyncer directory if it is missing - Hyperlane database now officially uses the domain name and id as a prefix internally to support multiple relayers using the same database - ### Drive-by changes Refactored and cleaned up the database code. Removed a lot of duplicate/unnecessary/redundant functions. ### Related issues - Fixes #2113 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - This changes the database prefixes internally to use domain name and domain id; also prepares us for #1899 ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Asa Oines	3adb6de5a5	Relayer warns if validator not announced (#2160) ... ### Description Explicitly call out if an ISM is configured with a validator that has not been announced. ### Related issues - Fixes #2120	2 years ago
Mattie Conover	3e5b20c88e	Better validation of signers during config parsing (#2139) ... ### Description This improves the validation during config parsing to ensure that all the expected signers are configured which means the errors will have the correct config path. ### Drive-by changes Also updated the checks for missing chain configurations to be more clear and use the chain config path rather than the config that indicated the config was required and removes a now redundant check. ### Related issues - Fixes #2101 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Mattie Conover	b56b022f88	Hyperlane-Core WASM support (#2110) ... ### Description This shuffles some code around and changes dependencies to make `hyperlane-core` WASM compatible and adds a check to the pipeline to ensure it stays that way. To do this it does refactor hyperlane db to the `base` crate, which makes more sense IMO anyway as that is an implementation detail of the agents and not part of the core protocol. ### Drive-by changes ### Related issues - Fixes #1977 ### Backward compatibility _Are these changes backward compatible?_ Yes-ish - It does change where things are but it does not change support. Should only impact people who directly use the libraries. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	910e5c84e9	Error message to indicate if there are no validators (#2099) ... ### Description Adds a more descriptive error message for failing to reach validator quorum when there are no validators defined. ### Drive-by changes No ### Related issues - Fixes #2031 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ None	2 years ago
Asa Oines	1d0e067166	Add relayer support for MultisigIsm and RoutingIsm (#2074) ... ### Description This PR adds support in the relayer for MultisigIsm (non-legacy) and RoutingIsm metadata building. AggregationIsm support is not yet implemented. ### Drive-by changes None ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ end-to-end	2 years ago
Mattie Conover	94e6b11f94	Fix payment enforcement config parsing bug (#2093) ... ### Description Fixes a bug where we were first parsing the matching list as a string and then trying to convert it which was impacting the expected config type. ### Drive-by changes None ### Related issues - https://discord.com/channels/935678348330434570/1096400530722537513 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Mattie Conover	91837d8a20	Remove txsubmission type (#2094) ... ### Description Removes a vestigial organ from the gelato days. We no longer need a config to denote txsubmission type. ### Drive-by changes No ### Related issues - https://discord.com/channels/935678348330434570/1096400530722537513 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Mattie Conover	1722f4ee7e	Better Agent Configuration Parsing (#2070) ... ### Description This is a significant change to how we parse user configs. The main goal of this PR is to give a clear path and env key to any errors that are encountered when parsing configuration. In order to accomplish this, I have crated a new trait `FromRawConf` which defines a set of steps to validate and convert individual fields. Also for every major config struct, there is now a `Raw*` variant of it which is used by the `config` library to reduce errors that occur during deserialization itself. `Raw*` types should always use the most basic form of a value, such as `String`, `StrOrInt`, `bool`, and ideally optional in all cases. Missing values should be handled during the raw conversion and not by `config`. I also needed to make changes to a number of types stored in the parsed config types to force validation forward to the parsing step instead of doing it when we read the config value. This also required me to create a filter to prevent trying to validate certain configs that we would not ever need to load. These changes can also be built on later to support something other than `config` if we choose to, or add support for merging configs from multiple sources since everything is optional. ### Drive-by changes - Default to `http` for connection type - Default to no gas payment enforcement if not specified instead of failing - `ChainSetup` -> `ChainConf` - `GasPaymentEnforcementConfig` -> `GasPaymentEnforcementConf` - Made ethereum connection configuration more forgiving - Moved hyperlane base settings from `mod.rs` to `base.rs` - Moved config chain tests in hyperlane core to `tests` dir to fix a cyclical import problem - Extension traits to help with config in hyperlane core - Moved `StrOrInt` to new hyperlane core `config` module - Support for parsing `U256` from `StrOrInt` - Removed `HexString` type which is now redundant - Updated base settings to use hyperlane domain - Use `heyKey` as signer type if `type` is not specified and `key` is - Moved hyperlane ethereum chain config to a new module ### Related issues - Fixes #2033 - Fixes #2012 ### Backward compatibility _Are these changes backward compatible?_ Yes - This should take in configs in the same shape but be more forgiving in a few places _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Test	2 years ago
Asa Oines	f25feecfa7	Scaffolding for multiple ISM types in the relayer (#2028) ... ### Description This PR adds scaffolding in the relayer to support building metadata for multiple ISM types. To add a new ISM type: - Add a new implementation of MetadataBuilder for that type - Add the type to the SupportedIsmTypes enum - Add the type to the match statement in BaseMetadataBuilder ### Drive-by changes None ### Related issues - #1757 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E tests	2 years ago
Mattie Conover	a19ad5c08e	Serial-submitter rework (#1978) ... ### Description Updates the serial submitter to use a priority queue (heap) to sort messages by when they should next be retried and avoid sifting through the entire queue which has become increasingly large in some cases. This also improves logging around the message that is currently being processed. ### Drive-by changes - Refactored `SubmitMessageArgs` -> `PendingMessage` - Cleaned up documentation which had severely rotted ### Related issues - Fixes #1894 - Fixes #1955 - Blocked by #1973 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Nam Chu Hoai	27798e0e0a	Set origin of metric (#1981) ... ### Description The metric sets it for the origin label of `any` which is not as useful as setting to the actual origin	2 years ago
Mattie Conover	b25fe7f355	Change relayer backoff scaling (#1973) ... ### Description This adds more backoff to earlier requests so that a request which is sent when there are no others won't be retired constantly until it ends up in the 5min backoff phase. ### Drive-by changes None ### Related issues - Fixes #1896 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Mattie Conover	4657fcc737	Better rust package management (#1975) ... ### Description At the suggestion of @tkporter this moves common dependencies and configuration to use the new(ish) cargo workspace configuration feature. This was added in July 2022 and allows us to ensure all versions and configurations are the same across different crates. It does not decrease the typing, but it does reduce the updates required when we go to update a package and keeps duplicates next to each other. This PR does not change any package versions or add/remove any packages. It only moves the configuration. ### Drive-by changes Also sorted packages by name. ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Mattie Conover	8fcb458062	Indicate why metadata could not be fetched on debug (#1905) ... ### Description - Changes metadata failure reasons to be DEBUG level. - Adds generic failure message as INFO level - Adds context to all errors returned by the metadata constructor to make them easier to trace. ### Drive-by changes None ### Related issues - Relates to #1856 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Mattie Conover	1c004f4b10	Remove gelato support (#1946) ... ### Description We have not been using Gelato anymore so thought it would be nice to remove this code and avoid continuing to maintain it. I know I have had to update it a few times to maintain parity. As an added perk it should slightly speed up build times since it removes an entire crate. ### Drive-by changes None ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes - As we do not use this code _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Trevor Porter	81ad2295fe	Move to estimateRetryableTicket to get L2 gas for arbitrum chains (#1949) ... ### Description * Moves to `estimateRetryableTicket` to get Arbitrum L2 gas ### Drive-by changes none ### Related issues - Fixes #1728 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit tests - haven't deployed yet tho	2 years ago
Mattie Conover	2c0b12528b	Support default signer configuration (#1908) ... ### Description - Adds a `defaultSigner` to settings which will automatically set all chains without their own signer configuration to use it. - Adds validation to relayers at startup to ensure that all destination chains have a signer configuration set. - Updates e2e test to validate this works ### Drive-by changes - Support `StrOrInt` or metric port deserialization. ### Related issues - Relates to #1856 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E Test	2 years ago
Nam Chu Hoai	80005f01e9	Add Sepolia (#1930) ... ### Description This PR deploys Hyperlane to Sepolia. Updates packages to 1.2.2 to include sepolia ### Drive-by changes - Fixes ContractVerifier - Adds multiple bytecode hashes that are valid in HyperlaneCoreChecker ### Related issues - Fixes #1878 ### Backward compatibility Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None	2 years ago
Trevor Porter	7fb935e801	Arbitrum gas amount handling in gas enforcement policies (#1912) ... ### Description See #1728 for detailed explanation Uses Arbitrum's NodeInterface to get the L1 cost of a tx so we can isolate the L2 gas and use that for gas enforcement policies. Very open to alternative ideas in the way I wrote it as well ### Drive-by changes No ### Related issues - Fixes #1728 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests - haven't deployed yet, but plan to deploy to rc testnet soon	2 years ago
Nam Chu Hoai	97c7f52801	Remove GasPaymentEnforcementPolicy::MeetsEstimatedCost (#1910) ... ### Description As discussed offline, to establish the social contract of the relayer accepting the quotes of its associated IGP on the origin chain, this PR removes the GasPaymentEnforcementPolicy that was assessing gas payments at tx submission time via off-chain oracles.	2 years ago
Mattie Conover	0049cca1ba	Support On Chain Fee Quoting (#1658) ... ### Description * New gas enforcement policy that uses the on-chain-fee-quoting values. * Change gas enforcement policies to return the gas limit that should be used to submit the transaction. * Change gelato submitter to use the gas limit provided by the gas enforcement policy. * Now indexes the gas amounts for the payments A couple of notes: * We might want to test this with the e2e test. * I took a look into the retying provider and it looks like it already does not retry on failed transaction submissions so I don't think anything needs to be done there after all. ### Drive-by changes ### Related issues - Fixes #1535 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - changes the internal database format for message payments ### Testing _What kind of testing have these changes undergone?_ Unit Tests --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	2 years ago
Trevor Porter	59a90b1bb6	Default to not allowing LocalStorage checkpoint syncers in relayer (#1900) ... ### Description Open to better names Adds a setting to the relayer, `allow_local_checkpoint_syncers`, which determines whether local storage based checkpoint syncers will be allowed by the metadata builder. Originally, I wanted something a bit more clever, like being able to specify `HYP_RELAYER_VALIDCHECKPOINTSYNCERS=LocalStorage,S3` or something, where I'd like those variants to be matched to the variants found in https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/hyperlane-base/src/types/checkpoint_syncer.rs#L14. But that enum requires values, so things get ugly. One option would be to create a new enum like: ``` enum CheckpointSyncerTypes { LocalStorage, S3, } ``` And another option is to use something like strum's [EnumString](https://docs.rs/strum/latest/strum/derive.EnumString.html) (shoutout to @mattiecnvr). But this still is a bit clunky, so for now just making this a bool and we can figure out something more elegant later if we ever get to a point where we're supporting multiple types of checkpoint syncers ### Drive-by changes none ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/402 ### Backward compatibility _Are these changes backward compatible?_ Yes - although if you ever want to run a relayer that uses local storage now, you'll need to set `HYP_RELAYER_ALLOWLOCALCHECKPOINTSYNCERS=true` _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None - we always expect to not be reading from the local fs in deployed relayers ### Testing _What kind of testing have these changes undergone?_ Ran e2e tests	2 years ago
Mattie Conover	836179c9ad	Better default stack traces (#1886) ... * Update to use color-eyre by default for stack traces * Use backtrace setting to define config formatting for debugging	2 years ago
Mattie Conover	6ad3b39114	Allow configs to use number or strings for numerics (#1885) ... * Allow strings or integers for configuration * Update TS for new integer change * Fixed finality blocks parsing	2 years ago
Mattie Conover	7a121a8d80	Make relayer require destination chains (#1884) ... * Make destination chains required for relayers * Update helm chart to include destination chains * Added notice for missing destination chain names	2 years ago
Mattie Conover	03d32f4009	Eliminate trivial constructors (#1847)	2 years ago
Asa Oines	0274c4f5ea	Better sanitization of announced validator locations (#1844)	2 years ago
Mattie Conover	29f0c78f5e	Fix message logs (#1825)	2 years ago
Asa Oines	207563b189	Add max message nonce gauge to relayer (#1819)	2 years ago
Mattie Conover	8105913dea	Improve agent logging (#1797)	2 years ago
Asa Oines	a836ce21d3	Use ValidatorAnnounce in relayer (#1569)	2 years ago
Mattie Conover	4b7e7ca9bb	Improve config errors (#1776) ... * Remove path to error as it was not working * Better domain parsing errors * Locked min version of config lib * Improve error messages for config loading	2 years ago
Trevor Porter	b55d7c5516	Log gas enforcement configuration (#1770) ... * Log gas enforcement configuration * fmt	2 years ago
Trevor Porter	1f88d0dfa0	Add whitelist to gas enforcement, whitelist interchain queries (#1732) ... * Add gas payment whitelist to relayer * Add interchain query routers to the gas enforcement whitelist * update testnet3 rc image * don't really wanna mess with rc * nit comment * pr comments	2 years ago
Trevor Porter	7daabfef35	Prevent calls to `Mailbox.recipientIsm` when the recipient of a message isn't a contract (#1763) ... * Add is_contract to avoid calls to recipientIsm when the recipient isn't a contract * trace -> debug	2 years ago
Mattie Conover	ce3e21fab4	Support relayers setting a maximum gas for transactions (#1670) ... * Add transaction gas limit for relayer * Don't run if it exceeds the limit * Allow skipping transaction gas limits for some chains * fmt * Update log * Update configs * Undo * disable max gas on main chains for now so we can test in rc first	2 years ago
Mattie Conover	d95d9b29db	v2 rc contexts (#1689) ... * Config for testnet3 * Config for mainnet2 * Removed newlines from debug print * Print full hashes for matching list debug * Deploy and config * Fixes from PR review * Fix compiler error * Deploy gnosis rc	2 years ago
Nam Chu Hoai	19dc142b1e	Deploy to Gnosis (#1675) ... * Gnosis deploy mainnet2 jan 26 * Fix rust * PR review * Set etheruem finality to 14 blocks * Fix testnet verification artifacts * fix artifact * Fix key funding logic * Deploy agents	2 years ago
Mattie Conover	759996b221	Whitelist/Blacklist consistent terms (#1584) ... * Standardize matching list terms * More consistent naming * Use originDomain and destinationDomain * Fix lint * Undo test_config changes	2 years ago
Mattie Conover	c163fce339	Fixes for run queue init delay (#1659) ... * Decrease wait between serial submitter ticks * Have processor sleep less time * Fix processor unprocessed message query * fmt	2 years ago
Mattie Conover	3ff3893a9a	Make timeout exponential (#1583)	2 years ago
Mattie Conover	f6aa1f6039	Fallback provider (#1571) ... * Created fallback provider * support fallback provider * WIP testing * Fix testing * Use http provider so we can better inspect errors * Undo change * Fix comment	2 years ago
Mattie Conover	ef9639ba3f	Simplify Hyperlane Agent (#1563) ... * Simplify Hyperlane Agent base settings validator validator relayer relayer settings undo change to relayer scraper agent validator submit fmt * update comment * Fixes after merge	2 years ago
Mattie Conover	64e26946e0	Dynamic Checkpoint Syncer (#1562) ... multisig validator submit validator relayer	2 years ago
Mattie Conover	65b10f1376	Signer changes (#1557) ... * Signer and signable ethereum signer changes move signers into chain config (chains.rs) Undo removal of traits in hyperlane-core/traits/mod.rs fix chains.rs validator submit remove signers from hyperlane core Announcements and checkpoint Remove test_output trait_builder.rs hyperlane base settings mod base settings signers base settings mod trait builder multisig Undo checkpoint syncer change for now local storage s3 storage validator metadata builder validator submit relayer run locally configs config and deployment fmt * WIP * some refactoring of signing * fmt * Null signer in config * Fix after merge	2 years ago