8 Commits (c6cea785ae778f90d584ad5b6969ca335067b642)

Author	SHA1	Message	Date
Mattie Conover	1c2eb1bc07	Scraper Refresh (#1932) ... ### Description Since we need to re-scrape everything anyway, wanted to go ahead and make some improvements and bug fixes now. * Adds code to scrape gas payments from the IGP during the sync loop * Updates gas payments table to work with new gas payment format * Fixed support with scraper for new configuration format (adds configurable list of chains to scrape) * Adds a new view for messages to display all relevant information * Adds a new view for gas payments to show total amounts paid for a given message * Fixed a bug introduced by seaorm (0.10) where `last_inserted_id` switched from being the first insert id to being the last * Pulled in a fix from @serejke in #1925 * Updated seaorm to 0.11.1 * Adds support for sepolia * Improved logging output at various levels ### Drive-by changes * Updated fallback provider to still retry even if it only has one url * Adds support for postgres `numeric` types to store Wei values (supported as of seaorm 0.11) * Switches to `bytea` representations of binary values * Switched from the `chrono` library to `time` (`chrono` is outdated and uses an unpatched version of `time`) * Removed redundant timestamp from `message` table * Cleans up some unused code ### Related issues - Fixes #1925 - Fixes #1156 ### Backward compatibility _Are these changes backward compatible?_ No - this has a new database format and for this reason defines entirely new migrations. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes - I changed it to `scraper3` so that we can run it along side the old scraper. It now also lines up with the name in the database. There is a new env var to specify what chains are scraped `HYP_BASE_CHAINSTOSCRAPE`. ### Testing _What kind of testing have these changes undergone?_ Manual - Been running the scraper locally and also deployed a current version of it to testnet3 for a subset of chains.	2 years ago
Mattie Conover	f814286998	Update packages (#1385) ... * Update ethers * Update packages * Fix build error	2 years ago
Mattie Conover	e8ed5f88ce	Deploy scraper for testnet3 (#1374) ... * How did those get there * abacus -> hyperlane * remove deprecated domains * Deleted dead code * Update sea-orm * Simplify settings post v2 changes * Split up settings * Update contexts * Fix scraper provider address regression during merge * Updated scraper deployment configs	2 years ago
Mattie Conover	0b09754922	Message explorer cleanup (#1251) ... * Better hex processing * Cleanup clippy errors * Fix parsing of 160 bit values * Support multiple contracts with cursors * InboxIndexer for getting message processed events * Work on inbox indexing and delivered message linking * Use message hashes to link deliveries and also add hash indexes for common lookups * split up settings struct * Support scraper config for main fn * Multiple config parsing for scraper * Better docs and more reliant config loading * Refactoring building of contract items * Cut down on duplication * Support inbox indexer building * fmt * Scraper settings loading re-work * Fix errors, warnings, and cleanup * Fix clippy warnings * Fix hash index not supporting uniqueness * Fix after entity update * Remove special case * Better error messages for missing config values * Env prefix filtering * Improve error context for url without base error * Fix inbox domain bug * Actually I was wrong about domains, misinterpreted config * Fix loading settings method * Fix upsert error * Debugged and fixed incorrect txn linking with messages and deliveries * Cleanup * Update config for scraper * Update config for scraper * Delivered message linking; again. * Reduce call frequency of message linking query * Fixed typo * Fix init mistake * changed to agent_prefix * Domain settings * Use a loop * Add more docs * Remove semicolen * Increase linking frequency * Add moonbeam domain * Fmt * Renamed settings struct * Add docs * Add test for FROM_HEX_CHARS array * WIP * WIP * Work on provider building * WIP attempt to create different make provider with connection * Create AbacusProviders * Update tables * Update tables and revert seaorm version * Pull data from providers * Better span tracing * Fixes after merge * Apparently this file was excluded * fix addr len error * Better contexts for error decyphering during config parsing * Add optimismgoerli * Add other rollups from #1176 * correct gas used field * Fix build * Cleanup * Update transactions to have additional EIP-1559 data and also default times for easier manual ops * Fix warnings * Some splits * Flatten dir structure a bit * WIP refactoring * #1229 add index on message timestamp * Refactor out db txn fns * Refactor out db block fns * Remove the message linker * Split of db logic into sub mods * Update rust version and sea-orm and satisfy new clippy requirments * new rust version feature + cleanup * Split up syncing logic * Remove use of cell since it was making things more complicated * Fix tracing * Doc cleanup * Record data even if no messages were returned * Reduce tuple use * Keep sea-orm 0.9.3 for now * Update comment. * Remove old commentˆ * Store instead of record * Fix message leaf index metric * Move dead code allowance to fns * Remove redundant type def * Docs * fmt * Move delivery impl * More docs * Fmt * Fixes after testing * Reduce logging verbosity from span * Use IN instead of OR for finding lists of hashes * Remove unused code	2 years ago
Mattie Conover	05e10b656d	Message Explorer Block and Txn Data from Providers (#1218) ... * Better hex processing * Cleanup clippy errors * Fix parsing of 160 bit values * Support multiple contracts with cursors * InboxIndexer for getting message processed events * Work on inbox indexing and delivered message linking * Use message hashes to link deliveries and also add hash indexes for common lookups * split up settings struct * Support scraper config for main fn * Multiple config parsing for scraper * Better docs and more reliant config loading * Refactoring building of contract items * Cut down on duplication * Support inbox indexer building * fmt * Scraper settings loading re-work * Fix errors, warnings, and cleanup * Fix clippy warnings * Fix hash index not supporting uniqueness * Fix after entity update * Remove special case * Better error messages for missing config values * Env prefix filtering * Improve error context for url without base error * Fix inbox domain bug * Actually I was wrong about domains, misinterpreted config * Fix loading settings method * Fix upsert error * Debugged and fixed incorrect txn linking with messages and deliveries * Cleanup * Update config for scraper * Update config for scraper * Delivered message linking; again. * Reduce call frequency of message linking query * Fixed typo * Fix init mistake * changed to agent_prefix * Domain settings * Use a loop * Add more docs * Remove semicolen * Increase linking frequency * Add moonbeam domain * Fmt * Renamed settings struct * Add docs * Add test for FROM_HEX_CHARS array * WIP * WIP * Work on provider building * WIP attempt to create different make provider with connection * Create AbacusProviders * Update tables * Update tables and revert seaorm version * Pull data from providers * Better span tracing * Fixes after merge * Apparently this file was excluded * fix addr len error * Better contexts for error decyphering during config parsing * Add optimismgoerli * Add other rollups from #1176 * correct gas used field * Fix build * Cleanup * Update transactions to have additional EIP-1559 data and also default times for easier manual ops * Fix warnings * Remove status from abacus core * ETH instead of AGOR * Remove block gas info * Make address optional for contract locator * Removed debug assert * Revert "Make address optional for contract locator" This reverts commit 5f8dd8f360. * chain agnostic zero address for provider construction * Fix var setting * Don't scale floats * Remove debug asserts	2 years ago
Mattie Conover	b4d06f9639	Message Explorer Inbox Scraping (#1159) ... * Better hex processing * Cleanup clippy errors * Fix parsing of 160 bit values * Support multiple contracts with cursors * InboxIndexer for getting message processed events * Work on inbox indexing and delivered message linking * Use message hashes to link deliveries and also add hash indexes for common lookups * split up settings struct * Support scraper config for main fn * Multiple config parsing for scraper * Better docs and more reliant config loading * Refactoring building of contract items * Cut down on duplication * Support inbox indexer building * fmt * Scraper settings loading re-work * Fix errors, warnings, and cleanup * Fix clippy warnings * Fix hash index not supporting uniqueness * Fix after entity update * Remove special case * Better error messages for missing config values * Env prefix filtering * Improve error context for url without base error * Fix inbox domain bug * Actually I was wrong about domains, misinterpreted config * Fix loading settings method * Fix upsert error * Debugged and fixed incorrect txn linking with messages and deliveries * Cleanup * Update config for scraper * Update config for scraper * Delivered message linking; again. * Reduce call frequency of message linking query * Fixed typo * Fix init mistake * changed to agent_prefix * Domain settings * Use a loop * Add more docs * Remove semicolen * Increase linking frequency * Add moonbeam domain * Fmt * Renamed settings struct * Add docs * Add test for FROM_HEX_CHARS array	2 years ago
Mattie Conover	44361545f8	Message explorer entities (#998) ... * Make domain id the primary key * Generation code to build entities * generated entity code * update docs * Switch to hex strings for hashes and addresses * fix log config * Switch to varchar(64) * Fix stdio -> stderr	2 years ago
Mattie Conover	5b06d0280c	Scraper DB generation and schema (#991) ... * Cleanup cargo config * BaseAgent trait * Pull `run_all` fn out of BaseAgent into static fn * Add `run` to `BaseAgent` * Scraper crate init * Create tables * WIP * data init * Init script * Fix * Attempting to run migration * fix ordering * fix domain creation * fix block table creation * fix checkpoint table * fix table cursor * fix table transaction + cleanup * enum type * progress * gas payments * message * delivered message * message state * cleanup * cleanup warnings * undo * Unique by domain and height * update index names * Change to OriginTxId * Add timestamp to messages * comment	2 years ago