66 Commits (8aa7b62beae417323c4f60a2a0c315508a1fe598)

Author	SHA1	Message	Date
Asa Oines	8aa7b62bea	Minor logging improvements to agents (#2271) ... ### Description Removes some unnecessary clutter, adds domain to some spans ### Drive-by changes None	2 years ago
Asa Oines	63562c7211	Refactor agent event indexing (#2246) ... ### Description This PR refactors event indexing in the agents, allowing similar logic to be shared across multiple event types (i.e. messages, deliveries, gas payments) and database types (i.e. the Relayer rocks DB and Scraper SQL DB). Furthermore, it adds new syncing modes by way of `MessageSyncCursors` that take advantage of the monotonically increasing dispatched message nonce to sync more intelligently. ### Drive-by changes - Fixes a bug in the existing cursor that caused the same block range to be indexed three times - Modifies kathy to get rid of the idea of "rounds", just sends messages with a sleep in between - Minor modifications to the e2e test for performance - Expand macros in settings - Add scraper to e2e test ### Opportunities for improvement - We can further reduce RPC usage (or improve latency) by sharing the view of the latest finalized block number between cursors - We can speed up the effective time for (a relayer to start deliving messages | the scraper to scraper recent events) by creating forward/backward cursors for gas payments and deliveries where the backwards cursor terminates at index_settings.from - We can remove the need for index_settings.from by terminating backwards cursors based on the block number that the first message was dispatched at ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E tests --------- Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Mattie Conover	5e38cdf31a	Omniscient Relayer (#2195) ... ### Description Adds support for running a relayer which delivers messages from multiple origin chains instead of just one origin chain. ### Drive-by changes _Are there any minor or drive-by changes also included?_ - The E2E pipeline got a major re-work. It both runs faster and also tests the new changes. - SerialSubmitter has been abstracted to support additional operation types such as oracle updates - SerialSubmitter has been updated to actually confirm a transaction was included and is safe from re-orgs - SerialSubmitter has been updated to have higher throughput by pipelining operations ### Related issues - Blocked by #2148 - Fixes #1899 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Asa Oines	f6ad1e9a95	Validator automatically self-announces (#2177) ... ### Description This PR adds the ability for validators to self-announce themselves, so long as the validator key has a large enough balance to pay for the tx. This required setting the retry provider to skip retrying reverting transactions, since with a quorum provider and a very speedy blockchain (like hardhat) the transaction is broadcast multiple times and the later transactions conflict with the first one. ### Drive-by changes None ### Related issues - Fixes #2120 ### Backward compatibility _Are these changes backward compatible?_ Yes --------- Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Mattie Conover	24f9642ea5	Agent docs and config improvements (#2148) ... ### Description Bunch of minor tweaks and updates based on what we learned from EthTokyo. - Updated Rust readme and the root readme to align with the hyperlane docs - Defaults to a 5 sec validator interval now - Makes the metrics port default to `9090` and have a soft failure if it can't start - Now will automatically create the checkpointsyncer directory if it is missing - Hyperlane database now officially uses the domain name and id as a prefix internally to support multiple relayers using the same database - ### Drive-by changes Refactored and cleaned up the database code. Removed a lot of duplicate/unnecessary/redundant functions. ### Related issues - Fixes #2113 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - This changes the database prefixes internally to use domain name and domain id; also prepares us for #1899 ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	3e5b20c88e	Better validation of signers during config parsing (#2139) ... ### Description This improves the validation during config parsing to ensure that all the expected signers are configured which means the errors will have the correct config path. ### Drive-by changes Also updated the checks for missing chain configurations to be more clear and use the chain config path rather than the config that indicated the config was required and removes a now redundant check. ### Related issues - Fixes #2101 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Mattie Conover	1722f4ee7e	Better Agent Configuration Parsing (#2070) ... ### Description This is a significant change to how we parse user configs. The main goal of this PR is to give a clear path and env key to any errors that are encountered when parsing configuration. In order to accomplish this, I have crated a new trait `FromRawConf` which defines a set of steps to validate and convert individual fields. Also for every major config struct, there is now a `Raw*` variant of it which is used by the `config` library to reduce errors that occur during deserialization itself. `Raw*` types should always use the most basic form of a value, such as `String`, `StrOrInt`, `bool`, and ideally optional in all cases. Missing values should be handled during the raw conversion and not by `config`. I also needed to make changes to a number of types stored in the parsed config types to force validation forward to the parsing step instead of doing it when we read the config value. This also required me to create a filter to prevent trying to validate certain configs that we would not ever need to load. These changes can also be built on later to support something other than `config` if we choose to, or add support for merging configs from multiple sources since everything is optional. ### Drive-by changes - Default to `http` for connection type - Default to no gas payment enforcement if not specified instead of failing - `ChainSetup` -> `ChainConf` - `GasPaymentEnforcementConfig` -> `GasPaymentEnforcementConf` - Made ethereum connection configuration more forgiving - Moved hyperlane base settings from `mod.rs` to `base.rs` - Moved config chain tests in hyperlane core to `tests` dir to fix a cyclical import problem - Extension traits to help with config in hyperlane core - Moved `StrOrInt` to new hyperlane core `config` module - Support for parsing `U256` from `StrOrInt` - Removed `HexString` type which is now redundant - Updated base settings to use hyperlane domain - Use `heyKey` as signer type if `type` is not specified and `key` is - Moved hyperlane ethereum chain config to a new module ### Related issues - Fixes #2033 - Fixes #2012 ### Backward compatibility _Are these changes backward compatible?_ Yes - This should take in configs in the same shape but be more forgiving in a few places _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Test	2 years ago
Asa Oines	1edbecf9e3	Lag count in Validator agent (#2073) ... ### Description This PR adds a lag to `mailbox.count()` in the validator agent. Without this lag, validators can shut down after the first message because the latestCheckpoint call is lagged and will fail due to the lagged count still being zero. ### Drive-by changes None ### Related issues - Fixes #1764 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ None --------- Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
Mattie Conover	4657fcc737	Better rust package management (#1975) ... ### Description At the suggestion of @tkporter this moves common dependencies and configuration to use the new(ish) cargo workspace configuration feature. This was added in July 2022 and allows us to ensure all versions and configurations are the same across different crates. It does not decrease the typing, but it does reduce the updates required when we go to update a package and keeps duplicates next to each other. This PR does not change any package versions or add/remove any packages. It only moves the configuration. ### Drive-by changes Also sorted packages by name. ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Mattie Conover	836179c9ad	Better default stack traces (#1886) ... * Update to use color-eyre by default for stack traces * Use backtrace setting to define config formatting for debugging	2 years ago
Mattie Conover	6ad3b39114	Allow configs to use number or strings for numerics (#1885) ... * Allow strings or integers for configuration * Update TS for new integer change * Fixed finality blocks parsing	2 years ago
Mattie Conover	1b45f8cad1	Rust Updates (#1845) ... * Update rust lang version * Fix new clippy lint * Update packages * Update rocksdb * Updates	2 years ago
Mattie Conover	8105913dea	Improve agent logging (#1797)	2 years ago
Mattie Conover	4b7e7ca9bb	Improve config errors (#1776) ... * Remove path to error as it was not working * Better domain parsing errors * Locked min version of config lib * Improve error messages for config loading	2 years ago
Asa Oines	c0e2c512ce	Add ValidatorAnnounce to HyperlaneCoreDeploy (#1560)	2 years ago
Mattie Conover	e5e794eda4	Fuel Validators (#1474) ... * Fuel ABI gen * WIP * Progress * Use HyperlaneDomain * Progress on fuel init * Support unknown domains * Work on handling wallets and signers for fuel * Move signer output test * Work on signers * Fixing up compilation * Update to use HyperlaneDomain * Fixes * WIP * Update Mailbox ABI * Move signer to be in chain conf and change rpc style to implementation * Better loader debugging * WIP * Merge fixes and refactor * Fix "rpcStyle" after merge * Simplify chain mappings * Fix tests * Remove excess runtime deps from validator * Use agent core in scraper and remove DB assumptions * More core cleanup * Remove duplication for checkpoint syncers * Better error messages for debugging * Debugging * Cleanup * remove signers * fmt * Fix after merge * Fixes after merge * Fixes after merge * remove random inclusion * Remove unused files * Requested changes from PR * Simplify deps and update sha3 * Fix after merge	2 years ago
Mattie Conover	f6aa1f6039	Fallback provider (#1571) ... * Created fallback provider * support fallback provider * WIP testing * Fix testing * Use http provider so we can better inspect errors * Undo change * Fix comment	2 years ago
Mattie Conover	ef9639ba3f	Simplify Hyperlane Agent (#1563) ... * Simplify Hyperlane Agent base settings validator validator relayer relayer settings undo change to relayer scraper agent validator submit fmt * update comment * Fixes after merge	2 years ago
Mattie Conover	64e26946e0	Dynamic Checkpoint Syncer (#1562) ... multisig validator submit validator relayer	2 years ago
Mattie Conover	581ddf60e8	Deployed Contract Traits (#1561) ... * Deployed contract traits mailbox trait mailbox impls mailbox impls interchain gas multisig ism extract fn map chains validator submit validator submit * Fix after merge	2 years ago
Mattie Conover	65b10f1376	Signer changes (#1557) ... * Signer and signable ethereum signer changes move signers into chain config (chains.rs) Undo removal of traits in hyperlane-core/traits/mod.rs fix chains.rs validator submit remove signers from hyperlane core Announcements and checkpoint Remove test_output trait_builder.rs hyperlane base settings mod base settings signers base settings mod trait builder multisig Undo checkpoint syncer change for now local storage s3 storage validator metadata builder validator submit relayer run locally configs config and deployment fmt * WIP * some refactoring of signing * fmt * Null signer in config * Fix after merge	2 years ago
Asa Oines	be940a5ed4	Validator signed announcements (#1507)	2 years ago
Mattie Conover	9a56b7b35e	Update to use HyperlaneDomain (#1461) ... * Update to use HyperlaneDomain * Fixes	2 years ago
Mattie Conover	7353d605df	Fuel scaffolding (#1401) ... * Fuel scaffolding * Fix primitive type import * Remove unused dep	2 years ago
Mattie Conover	fcba422718	Better errors in core (#1404) ... * Better errors * Remove unused error types * fmt * Better error when object is not found from a provider	2 years ago
Asa Oines	253e5f81f2	Validators sign checkpoints with index 0 (#1368)	2 years ago
Mattie Conover	de4ed8bf2b	More post v2 cleanup (#1388) ... * Removed code files that are not part of project anymore * Removed unused metric * Refactor some obj names * Update string constants for outbox and inbox * Update comments * Changes for PR review * Fix typo	2 years ago
Mattie Conover	f814286998	Update packages (#1385) ... * Update ethers * Update packages * Fix build error	2 years ago
Mattie Conover	e8ed5f88ce	Deploy scraper for testnet3 (#1374) ... * How did those get there * abacus -> hyperlane * remove deprecated domains * Deleted dead code * Update sea-orm * Simplify settings post v2 changes * Split up settings * Update contexts * Fix scraper provider address regression during merge * Updated scraper deployment configs	2 years ago
Asa Oines	f15632f26c	Finalize rename from abacus->hyperlane (#1312) ... * Rename abacus->hyperlane in agents * Rename in typescript * Rename abacus->hyperlane in dockerfile	2 years ago
Asa Oines	a23c1a8d2a	V2 agents (#1227)	2 years ago
Mattie Conover	05e10b656d	Message Explorer Block and Txn Data from Providers (#1218) ... * Better hex processing * Cleanup clippy errors * Fix parsing of 160 bit values * Support multiple contracts with cursors * InboxIndexer for getting message processed events * Work on inbox indexing and delivered message linking * Use message hashes to link deliveries and also add hash indexes for common lookups * split up settings struct * Support scraper config for main fn * Multiple config parsing for scraper * Better docs and more reliant config loading * Refactoring building of contract items * Cut down on duplication * Support inbox indexer building * fmt * Scraper settings loading re-work * Fix errors, warnings, and cleanup * Fix clippy warnings * Fix hash index not supporting uniqueness * Fix after entity update * Remove special case * Better error messages for missing config values * Env prefix filtering * Improve error context for url without base error * Fix inbox domain bug * Actually I was wrong about domains, misinterpreted config * Fix loading settings method * Fix upsert error * Debugged and fixed incorrect txn linking with messages and deliveries * Cleanup * Update config for scraper * Update config for scraper * Delivered message linking; again. * Reduce call frequency of message linking query * Fixed typo * Fix init mistake * changed to agent_prefix * Domain settings * Use a loop * Add more docs * Remove semicolen * Increase linking frequency * Add moonbeam domain * Fmt * Renamed settings struct * Add docs * Add test for FROM_HEX_CHARS array * WIP * WIP * Work on provider building * WIP attempt to create different make provider with connection * Create AbacusProviders * Update tables * Update tables and revert seaorm version * Pull data from providers * Better span tracing * Fixes after merge * Apparently this file was excluded * fix addr len error * Better contexts for error decyphering during config parsing * Add optimismgoerli * Add other rollups from #1176 * correct gas used field * Fix build * Cleanup * Update transactions to have additional EIP-1559 data and also default times for easier manual ops * Fix warnings * Remove status from abacus core * ETH instead of AGOR * Remove block gas info * Make address optional for contract locator * Removed debug assert * Revert "Make address optional for contract locator" This reverts commit 5f8dd8f360. * chain agnostic zero address for provider construction * Fix var setting * Don't scale floats * Remove debug asserts	2 years ago
Mattie Conover	b4d06f9639	Message Explorer Inbox Scraping (#1159) ... * Better hex processing * Cleanup clippy errors * Fix parsing of 160 bit values * Support multiple contracts with cursors * InboxIndexer for getting message processed events * Work on inbox indexing and delivered message linking * Use message hashes to link deliveries and also add hash indexes for common lookups * split up settings struct * Support scraper config for main fn * Multiple config parsing for scraper * Better docs and more reliant config loading * Refactoring building of contract items * Cut down on duplication * Support inbox indexer building * fmt * Scraper settings loading re-work * Fix errors, warnings, and cleanup * Fix clippy warnings * Fix hash index not supporting uniqueness * Fix after entity update * Remove special case * Better error messages for missing config values * Env prefix filtering * Improve error context for url without base error * Fix inbox domain bug * Actually I was wrong about domains, misinterpreted config * Fix loading settings method * Fix upsert error * Debugged and fixed incorrect txn linking with messages and deliveries * Cleanup * Update config for scraper * Update config for scraper * Delivered message linking; again. * Reduce call frequency of message linking query * Fixed typo * Fix init mistake * changed to agent_prefix * Domain settings * Use a loop * Add more docs * Remove semicolen * Increase linking frequency * Add moonbeam domain * Fmt * Renamed settings struct * Add docs * Add test for FROM_HEX_CHARS array	2 years ago
Trevor Porter	8f656d1f34	Update ethers-rs to include QuorumProvider fix for methods like eth_estimateGas (#1201)	2 years ago
Mattie Conover	fe904f22cc	Runnable Scraper (#1140) ... * Fix metric ordering (#1043) * Remove redunant enums * WIP * block cursor and date time * add chrono * add mods * Work on scraper logic * Init plumbing and config parsing * Stub out inbox loading * Update metric type * Refactoring out weird opt struct * Minor refactoring * Sync logic for scraper outbox reading messages * Fix param names * Stubs and docs to lay out next steps * Add way to get a contract address * read latest message index and work on storing messages * Ported LogMeta and refactored CheckpointMeta out + moved RawCommittedMessage * Remove redundant CheckpointWithMeta struct * Update outbox indexer to include metadata when fetching messages * WIP storing messages * Work on processing blocks and txns * Work on processing blocks and txns * weave data through for message insertion * cleanup * Scraper config * Support multiple index setting configs * Fix panics loading config * Fix db generation * Fix loading wrong outbox from config * Removed deprecated testnets from scraper configs * Fix outbox loading using wrong outbox config * Refactoring out unused index config from ethereum outbox indexer * Fix `todo` panic * Fixed handling of block/txn fetching/storing * allow disabling scraper outboxes * Fix elusive unwrap error * Cleanup and fix `ensure_blocks_and_txns` * Logging that apparently does nothing rn * Refactoring to support initializing tracing before general agent initialization. * Simplify agent main fns * Include deprecated domains * regen entities * Update index names to be more consistent * Remove destination fkey for messages as it does not have to be a valid id * Better logging * remove constraint on block height for the moment * Fix reverse id issue * Fix initial cursor insertion * fix test * appease clippy for now * Get scraper bin in Dockerfile too * append only cursors * append only cursor fix * Fix handling of empty insertion * Reduce query load * Scraper config for mainnet * Remove metrics fn from agent definition * Remove unused checkpoint fn * fix idx name * Order by height instead of id for cursors * Don't fetch message leaf index for every tick * Better var name * Add comment * Removed dead code stub * Better docs * Updated param name * Comment for pr * Added comment * Added comment * Removed unused code * Add doc on domains * Add comment. Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	2 years ago
Trevor Porter	1a1be089b0	Info log in validator when polling checkpoints (#1112) ... * Info log in validator when polling checkpoints * rename * Info log every once in a while * Just log indices	2 years ago
Trevor Porter	9b66dfebbb	Decrease frequency of Outbox.state() RPCs (#1087) ... * Increase interval of metrics loop in processor * Do the same for validators * Don't spawn a new metrics loop updating the outbox state for each inbox	2 years ago
Mattie Conover	9cc91d9c51	Refactoring out redundant enums (#1060) ... * Fix metric ordering (#1043) * Remove redunant enums * Remove checkpoint fn that was not actually in use	2 years ago
Mattie Conover	c37aa61b91	Update ethers (#1058) ... * Update run-locally to test quorum provider better * Update ethers-rs	2 years ago
Mattie Conover	1a7e74ea7c	Update ethers (#1037)	2 years ago
Trevor Porter	000b67cf16	Upgrade ethers-rs to fix Optimsim gas estimation (#1023)	2 years ago
Nam Chu Hoai	03ebda6acd	Monitor validator checkpoints in CheckpointSyncer (#993) ... * Monitor validator checkpoints in CheckpointSyncer * PR review	2 years ago
Mattie Conover	d316ae68d5	update ethers-rs (#1004)	2 years ago
Asa Oines	ae5db869a5	Fix lint-rs (#989)	2 years ago
Mattie Conover	333b18ecfe	Quorum provider (#934) ... * Add quorum provider support * Update Rust * Pull Http logic out of retrying provider * first pass at retrying provider for quorum provider * change weight parsing * Better logging and handling for quorum retry logic * Better logging * Appease new clippy rules * Update packages in vein attempt to make things work * quorum provider testing * switch to our version of ethers-rs * Update config * Support quorum provider * Use quorum provider for testing * Update packages * Update	2 years ago
Mattie Conover	7be7183068	Mattie/message explorer/refactoring (#974) ... * Cleanup cargo config * BaseAgent trait * Pull `run_all` fn out of BaseAgent into static fn * Add `run` to `BaseAgent` * Fix merge	2 years ago
exdx	57bd7ab3e2	docs: Fix typo in validator.rs (#775)	2 years ago
Trevor Porter	954d4bc866	Remote checkpointer infra (#673) ... * Remote checkpointer infra * rm more values * fix	2 years ago
Nam Chu Hoai	c80426383c	Only pass non-zero reorg periods as lag to latest_checkpoint (#676)	2 years ago
Mattie Conover	02f32b61d0	Use current index for validator latest processed gauge init (#646)	2 years ago