120 Commits (2af933cdac0f3ca019e1dd1eaa51c2b80490b26d)

Author	SHA1	Message	Date
Mattie Conover	fe904f22cc	Runnable Scraper (#1140) ... * Fix metric ordering (#1043) * Remove redunant enums * WIP * block cursor and date time * add chrono * add mods * Work on scraper logic * Init plumbing and config parsing * Stub out inbox loading * Update metric type * Refactoring out weird opt struct * Minor refactoring * Sync logic for scraper outbox reading messages * Fix param names * Stubs and docs to lay out next steps * Add way to get a contract address * read latest message index and work on storing messages * Ported LogMeta and refactored CheckpointMeta out + moved RawCommittedMessage * Remove redundant CheckpointWithMeta struct * Update outbox indexer to include metadata when fetching messages * WIP storing messages * Work on processing blocks and txns * Work on processing blocks and txns * weave data through for message insertion * cleanup * Scraper config * Support multiple index setting configs * Fix panics loading config * Fix db generation * Fix loading wrong outbox from config * Removed deprecated testnets from scraper configs * Fix outbox loading using wrong outbox config * Refactoring out unused index config from ethereum outbox indexer * Fix `todo` panic * Fixed handling of block/txn fetching/storing * allow disabling scraper outboxes * Fix elusive unwrap error * Cleanup and fix `ensure_blocks_and_txns` * Logging that apparently does nothing rn * Refactoring to support initializing tracing before general agent initialization. * Simplify agent main fns * Include deprecated domains * regen entities * Update index names to be more consistent * Remove destination fkey for messages as it does not have to be a valid id * Better logging * remove constraint on block height for the moment * Fix reverse id issue * Fix initial cursor insertion * fix test * appease clippy for now * Get scraper bin in Dockerfile too * append only cursors * append only cursor fix * Fix handling of empty insertion * Reduce query load * Scraper config for mainnet * Remove metrics fn from agent definition * Remove unused checkpoint fn * fix idx name * Order by height instead of id for cursors * Don't fetch message leaf index for every tick * Better var name * Add comment * Removed dead code stub * Better docs * Updated param name * Comment for pr * Added comment * Added comment * Removed unused code * Add doc on domains * Add comment. Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	2 years ago
Nam Chu Hoai	1fc65f3b7f	Add Moonbeam artifacts (#1153) ... * Add Moonbeam artificats * Chain metadata * More artifacts * Fix * Lint * Lint * Fix test * Fix * Deploy workloads to moonbeam	2 years ago
Trevor Porter	1a1be089b0	Info log in validator when polling checkpoints (#1112) ... * Info log in validator when polling checkpoints * rename * Info log every once in a while * Just log indices	2 years ago
Trevor Porter	08b4edecd8	Add timeout to inner HttpQuorum providers & all API requests, correctly record # of gas payment events processed (#1115) ... * Add timeout to inner HttpQuorum providers, correctly record # of gas payment events processed * Add timeouts to S3 requests * Add timeout everywhere	2 years ago
Trevor Porter	f2bb187cea	More friendly logs, mainnet rc using QuorumProvider (#1102) ... * More friendly log when tx estimation fails * Deploy mainnet rc with quorumprovider * Log params when retrying an RPC too	2 years ago
Trevor Porter	7040e5c8a4	Gas enforcement policy requiring payment to meet estimated costs (#1083) ... * All compiles * Rename files fwd request -> sponsored call * More scaffolding to get the API key in there * Cleaning up * Rename task_status_call to task_status * Finish rename * rm useForDisabledOriginChains * Introduce TransactionSubmitterType * submitter type -> submission type * Pass around gelato_config instead of the sponsor api key * Final cleanup * Nit * Use default tx submission type * Wip * wip * Getting there * Getting there * Refactor of serial submitter * Some renames * cargo fmt * Some testing * Nits after some testing * A bit of a refactor of policies * Adding some chain enums * Single source of truth for abacus domain IDs * Add local test chains * Get coingecko api key via external-secrets * Move some things around, rm gelato oracle api interaction * Gonna move to abacus-core * Move chain stuff to abacus-core * nit * comment nit * Deploy tooling * MeetsEstimatedCost policy test * PR comments, still need to merge domain enums * Single AbacusDomain enum	2 years ago
Trevor Porter	9b66dfebbb	Decrease frequency of Outbox.state() RPCs (#1087) ... * Increase interval of metrics loop in processor * Do the same for validators * Don't spawn a new metrics loop updating the outbox state for each inbox	2 years ago
Trevor Porter	ea5b584fe8	Support new Gelato relay API (#1041) ... * All compiles * Rename files fwd request -> sponsored call * More scaffolding to get the API key in there * Cleaning up * Rename task_status_call to task_status * Finish rename * rm useForDisabledOriginChains * Introduce TransactionSubmitterType * submitter type -> submission type * Pass around gelato_config instead of the sponsor api key * Final cleanup * Nit * Use default tx submission type * Some renames * Nits after some testing	2 years ago
Mattie Conover	9cc91d9c51	Refactoring out redundant enums (#1060) ... * Fix metric ordering (#1043) * Remove redunant enums * Remove checkpoint fn that was not actually in use	2 years ago
Nam Chu Hoai	00feef0a69	Chain Expansion 0: Goerli + Moonbase Alpha (#1053) ... * Add goerli/moonbase alpha to configs * Add artifacts from core deploy * Add artifacts from helloworld deploy * Update keyfunder config * Add from fields * Add missing inboxes * Add domain to chain.rs * Add to gelato mapping * Fix lint * Lint * New lines * Deploy * Add reorgPeriods * Deploy kathy	2 years ago
Mattie Conover	c37aa61b91	Update ethers (#1058) ... * Update run-locally to test quorum provider better * Update ethers-rs	2 years ago
Trevor Porter	f544a5df03	Add gas payment enforcement policy to require a minimum payment (#1031) ... * Add gas payment enforcement policy * Add infra, prettier * Better logs, add IGPs into configs * Fix e2e * Fix clippy	2 years ago
Mattie Conover	44361545f8	Message explorer entities (#998) ... * Make domain id the primary key * Generation code to build entities * generated entity code * update docs * Switch to hex strings for hashes and addresses * fix log config * Switch to varchar(64) * Fix stdio -> stderr	2 years ago
Mattie Conover	1a7e74ea7c	Update ethers (#1037)	2 years ago
Trevor Porter	2d9f729ac9	Process messages via Gelato ForwardRequests without charging gas yet (#967) ... * WIP, messy, just wanna test * Rename gelato_conf to gelato * Enabled is a str, not a bool * testing * working at least * closer * log * Things working pretty well * Cargo fmt * metrics * cleanin * cleaning * clippy * nit * deploy tooling * Fix disabled logic * Instrument * nits * More sane gelato config * nit * Reset rc * PR comment * Add ForwardRequestOpArgs * Add some chains to Gelato * Use Http * Fix test	2 years ago
Trevor Porter	000b67cf16	Upgrade ethers-rs to fix Optimsim gas estimation (#1023)	2 years ago
Nam Chu Hoai	03ebda6acd	Monitor validator checkpoints in CheckpointSyncer (#993) ... * Monitor validator checkpoints in CheckpointSyncer * PR review	2 years ago
Mattie Conover	d316ae68d5	update ethers-rs (#1004)	2 years ago
Mattie Conover	0f5e319d11	Make domain id the primary key (#997) ... * Make domain id the primary key * Comment	2 years ago
Mattie Conover	5b06d0280c	Scraper DB generation and schema (#991) ... * Cleanup cargo config * BaseAgent trait * Pull `run_all` fn out of BaseAgent into static fn * Add `run` to `BaseAgent` * Scraper crate init * Create tables * WIP * data init * Init script * Fix * Attempting to run migration * fix ordering * fix domain creation * fix block table creation * fix checkpoint table * fix table cursor * fix table transaction + cleanup * enum type * progress * gas payments * message * delivered message * message state * cleanup * cleanup warnings * undo * Unique by domain and height * update index names * Change to OriginTxId * Add timestamp to messages * comment	2 years ago
webbhorn	02f9589fdc	Drop rust-Kathy (#713)	2 years ago
Asa Oines	ae5db869a5	Fix lint-rs (#989)	2 years ago
Mattie Conover	333b18ecfe	Quorum provider (#934) ... * Add quorum provider support * Update Rust * Pull Http logic out of retrying provider * first pass at retrying provider for quorum provider * change weight parsing * Better logging and handling for quorum retry logic * Better logging * Appease new clippy rules * Update packages in vein attempt to make things work * quorum provider testing * switch to our version of ethers-rs * Update config * Support quorum provider * Use quorum provider for testing * Update packages * Update	2 years ago
Mattie Conover	7be7183068	Mattie/message explorer/refactoring (#974) ... * Cleanup cargo config * BaseAgent trait * Pull `run_all` fn out of BaseAgent into static fn * Add `run` to `BaseAgent` * Fix merge	2 years ago
Trevor Porter	0691b7b326	Gelato relayer scaffolding (#950) ... * Revive fwd_req_op. * Transform iter map into a for loop in relayer.rs. * cargo fmt * Plumb signer into GelatoSubmitter. * Provide inbox signer to GelatoSubmitter init. This might fail, in which case the relayer can't really do anything for that inbox if it is configured to use Gelato. In that case, bail!(..). Bailing requires that the boot/init boilerplate in the relayer handle errors, so plumb/wrap those all in Result<> return types and update accordingly. * Plumb deps up to ForwardRequestArgs construction. * Plumb actual target chain, and rename chain --> target_chain. * Use 0xEE for native token always. * Plumb outbox chain id * Plumb address of IVM on target chain * Just use zero bytes instead of todo! to satisfy clippy etc. * Plumb IVM BaseContract into GelatoSubmitter so that we can use it to encode calldata for IVM.process(). * cargo fmt again * How to marshal bytes. * Add interface for accessing inbox contract eth address * Plumb inbox address into the gelato submitter finally. * cargo fmt * Actually return the address instead of todo!()... * Actually build the byte array of calldata! * Better error msg to fit < 100c * defer introducing fwd_req_op until next PR * Handful of pre-review cleanups: - Only give GelatoSubmitter the inbox domain instead of the whole InboxContracts - Don't wrap http: reqwest::Client in an Arc. It's unnecessary and handled internally for you, and better to clone() so that cxn pool reused. - Shorter names from imports where noticed - stub op.run() impl - shorter submitter field names where appropriate - dont pass GelatoConf to GelatoSubmitter - get correct visibility in abigen'd build code (and allow no doc comments to satisfy clippy) * rename data to call_data * cargo fmt * Defer field name change for chain_id and target in FwdReqArgs * Cleanup of relayer.rs including more clear data validation and imoprt scoping. * cargo fmt * Improve (?) address handling for inbox and ivm. * for InboxValidatorManagerVariants::contract_address(), actually delegate to the Mock variant implementation's contract_address() call instead of over-eagerly returning None. * Stub impl of _contract_address in mockall mock! Inbox contract impl. * for Other variants, match arm does unimplemented!(), so we avoid having to wrap in an Option, making cleaner at call sites * Minimize namespace qualifications where possible for Address-like types * attempt to make the IVM contract_address() for abacus-ethereum impl parallel to inbox w/r/t types * Move abacus domain conversion out of gelato::Chain and into private helper in gelato_submitter * More logical ordering of GelatoSubmitter struct fields * cargo fmt * Drop extra comment, clarify log * Plumb sponsor address from configs into GelatoSubmitter. Interpreted as a source chain address. * cargo fmt --all * stop trying to wrap http client in an arc, it's counterproductive * Write Ok once instead of every line * formatting * Do not separate with empty line the inner attributes found in lib.rs post-abigen-buildscript-exec Otherwise cargo fmt finds a diff and presubmits halt. * spelling typo * Change GelatoSubmitter construction from new to helper in relayer.rs. Also use simpler construction for tokenizing the proof for merkle trees when generating the forward request args struct. Was needlessly convoluted before using default trait unnecessarily etc. Use writeln instead of write in abigen util since clippy complains otherwise. * Get ivm_base_contract from ABI string directly rather than plumbing. * pass GelatoConf by as_ref() which refs inner type rather than &Option<GelatoConf> * Drop unrelated comment in gelato/chains.rs, noted elsewhere * Revive fwd_req_op. * Transform iter map into a for loop in relayer.rs. * cargo fmt * Plumb signer into GelatoSubmitter. * Provide inbox signer to GelatoSubmitter init. This might fail, in which case the relayer can't really do anything for that inbox if it is configured to use Gelato. In that case, bail!(..). Bailing requires that the boot/init boilerplate in the relayer handle errors, so plumb/wrap those all in Result<> return types and update accordingly. * Plumb deps up to ForwardRequestArgs construction. * Plumb actual target chain, and rename chain --> target_chain. * Use 0xEE for native token always. * Plumb outbox chain id * Plumb address of IVM on target chain * Just use zero bytes instead of todo! to satisfy clippy etc. * Plumb IVM BaseContract into GelatoSubmitter so that we can use it to encode calldata for IVM.process(). * cargo fmt again * How to marshal bytes. * Add interface for accessing inbox contract eth address * Plumb inbox address into the gelato submitter finally. * cargo fmt * Actually return the address instead of todo!()... * Actually build the byte array of calldata! * Better error msg to fit < 100c * defer introducing fwd_req_op until next PR * Handful of pre-review cleanups: - Only give GelatoSubmitter the inbox domain instead of the whole InboxContracts - Don't wrap http: reqwest::Client in an Arc. It's unnecessary and handled internally for you, and better to clone() so that cxn pool reused. - Shorter names from imports where noticed - stub op.run() impl - shorter submitter field names where appropriate - dont pass GelatoConf to GelatoSubmitter - get correct visibility in abigen'd build code (and allow no doc comments to satisfy clippy) * rename data to call_data * cargo fmt * Defer field name change for chain_id and target in FwdReqArgs * Cleanup of relayer.rs including more clear data validation and imoprt scoping. * cargo fmt * Improve (?) address handling for inbox and ivm. * for InboxValidatorManagerVariants::contract_address(), actually delegate to the Mock variant implementation's contract_address() call instead of over-eagerly returning None. * Stub impl of _contract_address in mockall mock! Inbox contract impl. * for Other variants, match arm does unimplemented!(), so we avoid having to wrap in an Option, making cleaner at call sites * Minimize namespace qualifications where possible for Address-like types * attempt to make the IVM contract_address() for abacus-ethereum impl parallel to inbox w/r/t types * Move abacus domain conversion out of gelato::Chain and into private helper in gelato_submitter * More logical ordering of GelatoSubmitter struct fields * cargo fmt * Drop extra comment, clarify log * Plumb sponsor address from configs into GelatoSubmitter. Interpreted as a source chain address. * cargo fmt --all * stop trying to wrap http client in an arc, it's counterproductive * Write Ok once instead of every line * formatting * Do not separate with empty line the inner attributes found in lib.rs post-abigen-buildscript-exec Otherwise cargo fmt finds a diff and presubmits halt. * spelling typo * Change GelatoSubmitter construction from new to helper in relayer.rs. Also use simpler construction for tokenizing the proof for merkle trees when generating the forward request args struct. Was needlessly convoluted before using default trait unnecessarily etc. Use writeln instead of write in abigen util since clippy complains otherwise. * Get ivm_base_contract from ABI string directly rather than plumbing. * pass GelatoConf by as_ref() which refs inner type rather than &Option<GelatoConf> * Drop unrelated comment in gelato/chains.rs, noted elsewhere * Wrap Inbox::contract_address() result in an Option. * Don't over-zealously mark pub(crate) struct's members pub(crate). pub is good enough. * More natural iter map construction in Token byte packing for call_data on process. * More natural iter map construction in Token byte packing for call_data on process. * Restore fwd req op struct and fmt * rm unnecessary Results * add with_capacity * Make contract_address return Address, not Option<Address> * rm unimplemeneted * Rm address field of validator manager * Mid refactor of creating GelatoSubmitter * add inbox_gelato_chain * cleaning * Some shuffling around * some moving around * Finishing touches * Remove unnecessary type definition & some unused code Co-authored-by: webbhorn <webb.horn@gmail.com>	2 years ago
Trevor Porter	ebb0652b8d	Some clarifying comments following the new retry queue heuristic (#948) ... * Change queue heuristic * Add a todo * rm comment change * Drain without collecting and don't recreate the wait_queue * Add comments, oops	2 years ago
Trevor Porter	933dee6f7f	Push failing messages to the back of the run queue (#933) ... * Change queue heuristic * Add a todo * rm comment change * Drain without collecting and don't recreate the wait_queue	2 years ago
Nam Chu Hoai	95eb1c47e3	Add leaf index as a key on the log line (#929)	2 years ago
Mattie Conover	e5a72eaba2	Relayer blacklists (#807) ... * blacklists * fix not_matches logic to be more understandable * WIP * standardize src/dst * move docs around * Add tests for matches	2 years ago
exdx	57bd7ab3e2	docs: Fix typo in validator.rs (#775)	2 years ago
webbhorn	c819982db4	Reduce log severity for already-sent messages. (#735) ... It is expected for the 'processor' module to churn through the entire history of sent messages on relayer init, and it is expected/normal behavior to discover that the vast majority of those have already been sent. Therefore, it is cluttering / inappropriate to log those events at info! severity, and instead preferable to record them at debug severity. Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
webbhorn	87400211a5	Check delivery status on inbox before attempting delivery (#712) ... Restores functionality accidentally lost in #637 wherein we proactively check inbox message_status() call before attempting delivery of a message. * Check whether message has already been delivered before attempting delivery * s/deliver/process	2 years ago
Mattie Conover	8d0509a4d2	Operational cleanup (#707) ... * Reduce log level for common message * retry more of the errors and downgrade log level for non-stop cases * cleanup * reduce error logs	2 years ago
webbhorn	8b79a14b51	Set message_delivered phase correctly in serial_submitter (#695) ... Update two metrics upon successful delivery of message: * counter reflecting number of messages successfully delivered during process lifetime * gauge of highest leaf index of any message delivered to this inbox so far during process lifetime Also updates run-locally integration test to verify these metric values upon test termination.	2 years ago
webbhorn	dd63f70da1	Gelato Plumbing (#637) ... This PR prepares for the introduction of Gelato ForwardRequest API usage. However, it only refactors current code and introduces a prerequisite interface. In particular, the existing MessageProcessor was previously responsible both for message discovery via AbacusDB, as well as managing the submission of that message to the destination chain via some RPC provider. Since we want to have the option to deliver those messages over Gelato, a natural place to split things up might be between the AbacusDB message discovery phase, and the message delivery phase. This CL formalizes that notion by moving the submission logic into a SerialSubmitter submodule. The MessageProcessor and SerialSubmitter only communicate via a channel now. A subsequent PR will introduce GelatoSubmitter, which will route messages from the MessageProcessor to the Gelato API according to the new configuration options introduced in this PR.	2 years ago
Trevor Porter	954d4bc866	Remote checkpointer infra (#673) ... * Remote checkpointer infra * rm more values * fix	2 years ago
Nam Chu Hoai	c80426383c	Only pass non-zero reorg periods as lag to latest_checkpoint (#676)	2 years ago
Mattie Conover	02f32b61d0	Use current index for validator latest processed gauge init (#646)	2 years ago
Mattie Conover	de20710466	Rust updates (#644) ... * Update config and trim down macro code * Update rust * Package updates * Updates and cleanup * make clippy happy	2 years ago
Mattie Conover	fbf7130313	Provider request reduction (#638) ... * add contract function from call data to metric and no longer use histogram * fix warning * Get function names for metrics * Fix relayer metrics fetch interval * Fix validator metrics updates * initialize the latest processed checkpoint * Fix clippy errors	2 years ago
Nam Chu Hoai	8c9c38918e	Do not create inbox contracts if validator agent (#635)	2 years ago
Mattie Conover	f566bb40c8	Validator Metrics (#609) ... * minor refactoring * Add metrics to validator	2 years ago
Mattie Conover	4ec3e2124a	Even more metrics (#595) ... * Allow Raw messages to be parsed without cloning * Add hardcoded mapping between domain and chain * Remove extra clone * Report dst chain for `dispatch` phase of `last_known_message_leaf_index` * Update state call for outbox * Add outbox state metric	2 years ago
webbhorn	630e20bbc2	validator: Ensure non-empty outbox on init. (#586) ... validator: Ensure non-empty outbox on init. This addresses the issue described in #575 wherein validators crash after new contract deployment due to failed calls on outbox.latest_checkpoint(). The calls fail due to u256 underflow on count()-1, since count() is zero on init, and latest_checkpoint() wants to return an **index**, which is count() - 1. This PR adds a retry loop just prior to entering the existing main validator-submit-loop. I chose to poll outbox.count() rather than try to decode the error code or revert reason on failed calls to latest_checkpoint() so that we avoid tightly coupling this behavior to particular error codes or messages, and in fact may want to preserve exisitng behavior when calls to outbox.latest_checkpoint() fail for other reasons. Tested with a modified `rust/run-locally.sh` which asks Kathy to sleep for 5 minutes prior to sending messages. Without this change, the validator crashes. With this change, the validator does not crash, and after 5 minutes, starts signing checkpoints as designed.	2 years ago
Mattie Conover	4d9603c868	Relayer whitelist (#570) ... * Update ABIs, compiles. Removed Inbox indexer and cached checkpoints from the Inbox * Rm checkpoint indexing test * Update InboxValidatorManager, just need to connect the pieces now * Add channels * So close * Fix InboxValidatorManager deploy * Make message processor slightly more readable * Create loop-control crate * Try to bring some sanity to message processor * Fix bug where prover_sync wasn't in line with latest_signed_checkpoint * Rm immediate message processing, clean up * rm abacus-cli * more cleanup * rename / rm some settings * TS renames / rms * Lower some processing failure logs to info * Checkpoint fetcher doesn't need the CommittedMessages * Hardcode kathy dispatching * Move to watch channel * I'm sorry clippy * nits * rm some nonce related stuff * more refactoring * cleanup * Add whitelist settings * Whitelist filtering * fix docker build * use relayed terminology * Change whitelist tuple order * naming change * move whitelist check to before fetching proof * Update span event * make messages singular * Update error * minor cleanup * Update configs in scripts * Whitelist testing and parsing fixes Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	2 years ago
Trevor Porter	1793fffbeb	Testnet2 agent deployment (#568) ... * Deploy testnet2 contracts * Add testnet2 configs to rust * don't panic for pre-london chains * Don't checkpoint, retry fewer times, and configure kathy properly * clean up * cleaninngggg Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local>	2 years ago
Asa Oines	0a81c910a3	Move InterchainGasPaymaster from AbacusConnectionManager to AbacusConnectionClient (#558)	3 years ago
Trevor Porter	d120a76c1e	No more cached checkpoints in agents (#550) ... * Update ABIs, compiles. Removed Inbox indexer and cached checkpoints from the Inbox * Rm checkpoint indexing test * Update InboxValidatorManager, just need to connect the pieces now * Add channels * So close * Fix InboxValidatorManager deploy * Fix bug where prover_sync wasn't in line with latest_signed_checkpoint * Rm immediate message processing, clean up * rm abacus-cli * more cleanup * rename / rm some settings * TS renames / rms * Lower some processing failure logs to info * Checkpoint fetcher doesn't need the CommittedMessages * Hardcode kathy dispatching * Move to watch channel * I'm sorry clippy * nits * rm some nonce related stuff * PR comments	3 years ago
Trevor Porter	29970fe326	Index GasPayment events in Relayer (#504) ... * Add InterchainGasPaymaster traits and ethereum structs * Add abacuse-base interchain_gas * Actually setting up indexer * Setting up DB usage * wip * Add interchain gas paymaster to AbacusAgentCore * Sync gas payments in the relayer * It runs wooooooo * Use U256 for gas payments * nits, rm InterchainGasPaymasterEvents * Add chain_name, fix metrics * More name -> chain_name rename * Correctly rename contract_name => chain_name, move InterchainGasPayment from traits to types * split up contract_sync a bit * cargo fmt * nit * clean * Add metrics * to_string => encode_hex * make clippy happy * move contract_sync/common.rs stuff into mod.rs * string interpolation for logs * some nits * mv types/interchain_gas into types/mod * Idempotent GasPayment indexing * Use new contract ABI & update rust code * Rename RandomBadRecipient -> TestSendReceiver * use nit * only index finalized blocks * Run with finalityBlocks * Add InterchainGasPaymaster to update_abis * Some PR comments	3 years ago
Trevor Porter	5050eee9a4	Use new contract ABI & update rust code (#519)	3 years ago