98 Commits (19dc142b1e81029f347b0b75b49fe06a168b0fd6)

Author	SHA1	Message	Date
Trevor Porter	ebb0652b8d	Some clarifying comments following the new retry queue heuristic (#948) ... * Change queue heuristic * Add a todo * rm comment change * Drain without collecting and don't recreate the wait_queue * Add comments, oops	2 years ago
Trevor Porter	933dee6f7f	Push failing messages to the back of the run queue (#933) ... * Change queue heuristic * Add a todo * rm comment change * Drain without collecting and don't recreate the wait_queue	2 years ago
Nam Chu Hoai	95eb1c47e3	Add leaf index as a key on the log line (#929)	2 years ago
Mattie Conover	e5a72eaba2	Relayer blacklists (#807) ... * blacklists * fix not_matches logic to be more understandable * WIP * standardize src/dst * move docs around * Add tests for matches	2 years ago
webbhorn	c819982db4	Reduce log severity for already-sent messages. (#735) ... It is expected for the 'processor' module to churn through the entire history of sent messages on relayer init, and it is expected/normal behavior to discover that the vast majority of those have already been sent. Therefore, it is cluttering / inappropriate to log those events at info! severity, and instead preferable to record them at debug severity. Co-authored-by: Mattie Conover <git@mconover.dev>	2 years ago
webbhorn	87400211a5	Check delivery status on inbox before attempting delivery (#712) ... Restores functionality accidentally lost in #637 wherein we proactively check inbox message_status() call before attempting delivery of a message. * Check whether message has already been delivered before attempting delivery * s/deliver/process	2 years ago
Mattie Conover	8d0509a4d2	Operational cleanup (#707) ... * Reduce log level for common message * retry more of the errors and downgrade log level for non-stop cases * cleanup * reduce error logs	2 years ago
webbhorn	8b79a14b51	Set message_delivered phase correctly in serial_submitter (#695) ... Update two metrics upon successful delivery of message: * counter reflecting number of messages successfully delivered during process lifetime * gauge of highest leaf index of any message delivered to this inbox so far during process lifetime Also updates run-locally integration test to verify these metric values upon test termination.	2 years ago
webbhorn	dd63f70da1	Gelato Plumbing (#637) ... This PR prepares for the introduction of Gelato ForwardRequest API usage. However, it only refactors current code and introduces a prerequisite interface. In particular, the existing MessageProcessor was previously responsible both for message discovery via AbacusDB, as well as managing the submission of that message to the destination chain via some RPC provider. Since we want to have the option to deliver those messages over Gelato, a natural place to split things up might be between the AbacusDB message discovery phase, and the message delivery phase. This CL formalizes that notion by moving the submission logic into a SerialSubmitter submodule. The MessageProcessor and SerialSubmitter only communicate via a channel now. A subsequent PR will introduce GelatoSubmitter, which will route messages from the MessageProcessor to the Gelato API according to the new configuration options introduced in this PR.	2 years ago
Mattie Conover	de20710466	Rust updates (#644) ... * Update config and trim down macro code * Update rust * Package updates * Updates and cleanup * make clippy happy	2 years ago
Mattie Conover	fbf7130313	Provider request reduction (#638) ... * add contract function from call data to metric and no longer use histogram * fix warning * Get function names for metrics * Fix relayer metrics fetch interval * Fix validator metrics updates * initialize the latest processed checkpoint * Fix clippy errors	2 years ago
Nam Chu Hoai	8c9c38918e	Do not create inbox contracts if validator agent (#635)	2 years ago
Mattie Conover	4ec3e2124a	Even more metrics (#595) ... * Allow Raw messages to be parsed without cloning * Add hardcoded mapping between domain and chain * Remove extra clone * Report dst chain for `dispatch` phase of `last_known_message_leaf_index` * Update state call for outbox * Add outbox state metric	3 years ago
Mattie Conover	4d9603c868	Relayer whitelist (#570) ... * Update ABIs, compiles. Removed Inbox indexer and cached checkpoints from the Inbox * Rm checkpoint indexing test * Update InboxValidatorManager, just need to connect the pieces now * Add channels * So close * Fix InboxValidatorManager deploy * Make message processor slightly more readable * Create loop-control crate * Try to bring some sanity to message processor * Fix bug where prover_sync wasn't in line with latest_signed_checkpoint * Rm immediate message processing, clean up * rm abacus-cli * more cleanup * rename / rm some settings * TS renames / rms * Lower some processing failure logs to info * Checkpoint fetcher doesn't need the CommittedMessages * Hardcode kathy dispatching * Move to watch channel * I'm sorry clippy * nits * rm some nonce related stuff * more refactoring * cleanup * Add whitelist settings * Whitelist filtering * fix docker build * use relayed terminology * Change whitelist tuple order * naming change * move whitelist check to before fetching proof * Update span event * make messages singular * Update error * minor cleanup * Update configs in scripts * Whitelist testing and parsing fixes Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	3 years ago
Asa Oines	0a81c910a3	Move InterchainGasPaymaster from AbacusConnectionManager to AbacusConnectionClient (#558)	3 years ago
Trevor Porter	d120a76c1e	No more cached checkpoints in agents (#550) ... * Update ABIs, compiles. Removed Inbox indexer and cached checkpoints from the Inbox * Rm checkpoint indexing test * Update InboxValidatorManager, just need to connect the pieces now * Add channels * So close * Fix InboxValidatorManager deploy * Fix bug where prover_sync wasn't in line with latest_signed_checkpoint * Rm immediate message processing, clean up * rm abacus-cli * more cleanup * rename / rm some settings * TS renames / rms * Lower some processing failure logs to info * Checkpoint fetcher doesn't need the CommittedMessages * Hardcode kathy dispatching * Move to watch channel * I'm sorry clippy * nits * rm some nonce related stuff * PR comments	3 years ago
Trevor Porter	29970fe326	Index GasPayment events in Relayer (#504) ... * Add InterchainGasPaymaster traits and ethereum structs * Add abacuse-base interchain_gas * Actually setting up indexer * Setting up DB usage * wip * Add interchain gas paymaster to AbacusAgentCore * Sync gas payments in the relayer * It runs wooooooo * Use U256 for gas payments * nits, rm InterchainGasPaymasterEvents * Add chain_name, fix metrics * More name -> chain_name rename * Correctly rename contract_name => chain_name, move InterchainGasPayment from traits to types * split up contract_sync a bit * cargo fmt * nit * clean * Add metrics * to_string => encode_hex * make clippy happy * move contract_sync/common.rs stuff into mod.rs * string interpolation for logs * some nits * mv types/interchain_gas into types/mod * Idempotent GasPayment indexing * Use new contract ABI & update rust code * Rename RandomBadRecipient -> TestSendReceiver * use nit * only index finalized blocks * Run with finalityBlocks * Add InterchainGasPaymaster to update_abis * Some PR comments	3 years ago
Trevor Porter	5050eee9a4	Use new contract ABI & update rust code (#519)	3 years ago
Mattie Conover	71ce3e285b	Fix duplicate metric labels (#518)	3 years ago
Mattie Conover	d687d3e230	More Metrics (#495) ... * metrics refactoring * metrics refactoring * Make brain hurt less by using a damn function * patch through metrics to abacus_ethereum inbox and outbox * WIP * Minor cleanup * work on time update * added label arrays * refactoring * Metrics for sending txns * Metrics middleware * cleanup * Cleanup after merge * Add metrics config and setup prom metrics with core metrics * Minor fixes to docs * Better trait building * rename * Weave metrics through code * Use core metrics instead * Removed some unused code * fmt * change feature layout * fix docker build * Start scraping metrics on an interval * fmt * fix duplicate label error * change features * fix u256 conversion fn * refactor * Support default values in config * Minor fixes * Use f64 conversion from primitive-types * remove unused code * fix const name * improve docs * better var name * remove unused code * remove unused code	3 years ago
webbhorn	6b3822f664	Misc. agent cleanup and dead code deletion (#470) ... * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in comment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool. * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in coment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool.	3 years ago
Mattie Conover	f78e79ab9b	Better Agent Error Logging (#412) ... * Make errors print on one line * Replace color-eyre with eyre in core and base * Replace color-eyre with eyre other libs * Log should be imported without patch version * Update tracing and opentelemetry * Only need openssl dynamic libs * Fix no-default-feature build * Custom eyre handler * Removed unused log dep * Cleanup paste dep * More package cleanup * Update tokio * Minor refactor	3 years ago
Nam Chu Hoai	cc6c2db5d8	Testnet deploy (#369) ... * Got keys for testnet * Testnet deploy * Add rust configs * prettier * Nits, update governance config * new agent image * Rename secret reference * Have kathy also checkpoint * Only checkpoint when count > 1 * Move relayer to unique key for (source, remote) pair * prettier * Clean up get-aws-key-addresses * get GCP keys half working * let's clean this up * First steps at refactor * Migrate delete-keys over * support key rotation * Create and get key all working * update relayer external secret * Prettier * Update ethers-rs * Debugging testnet deployment (#370) * Dont fail on inbox checkpointing tx error * fix docker build * Debug * More instrumentation * Lint * Use the correct signers * Lint * prettier * suffix -> index, go back to 1 key per agent * rm relayer user * Create user and key identifiers * Move some fns to key-utils, fix gcp secret names * Correct kathy & checkpointer key names * Back to Promse.all in deploy-agents, clean * Don't require GCP keys to be created with agentconfig * fix * pure json response for get-key-addresses Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	3 years ago
Nam Chu Hoai	cee9a2d8ef	Add more tracing information (#362) ... * Add more tracing information * PR review	3 years ago
Nam Chu Hoai	5e254f0512	Add gauges to track agents (#360)	3 years ago
Nam Chu Hoai	e5ea4d42a9	0-indexed Checkpoints (#344) ... * Checkpoints are zero indexed * Checkpoints are not one-indexed anymore * Agent changes for 0-indexed checkpoints * Disallow checkpoint index 0 * Can just check indicies again * Proper isCheckpoint * Dispatch dummy message * fix before * PR review Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local>	3 years ago
Nam Chu Hoai	0216d28dcf	Merge Inbox prove and process (#339) ... * Merge Inbox prove and process * PR reviews * Package-lock	3 years ago
Trevor Porter	b5f121b837	Multisig validator rust agents (#342) ... * add MultisigCheckpointSyncer * cargo fmt * Nits * Some comments, clean up * rm inbox's submit_checkpoint, get settings compiling * Move some checkpoint structs to core, add inbox validator manager base & core * Update existing Inbox/Outbox ABIs, add InboxValidatorManager ABI * Add ethereum inbox validator manager * Add inbox validator manager to AbacusAgentCore * Put everything together * Updated some configs, nearly able to run the relayer * Update config to have InboxValidatorManager alongside Inbox * Able to run it * Remove now unnecessary AbacusAgentCore::new * Make MultisigSignedCheckpoint::try_from not take in a mut * Use InboxContracts in relayer * cleaning up * nit * clippy * Check all checkpoints are the same in the try_from for MultisigSignedCheckpoint	3 years ago
Trevor Porter	fec07d21fe	Remove docs & all optics-related rust (#322) ... * optics -> abacus in infra * rm docs * rename contract-metrics prom metrics optics-* -> abacus-* * Rm watcher, updater, a bunch of optics things in abacus-base * rm agents/processor * Removing a whole lot of updater / home / replica stuff from abacus-core * Deleted a lot throughout -- some still commented out and a lot of cleanup needed * Slowly cleaning up, kathy is where the remaining errors are * Delete some commented out code * makes kathy an abacus agent * compiling woo * cargo fmt * compiling with all features * removing some update references * Ignore handles_missing_rpc_checkpoints * cargo fmt * Add signedCheckpoint.json * rm some references to updates, delete unnecessary abis * rm all references to home or replica * compiling, remove som e unused code * fix test warning * nit * New README - needs a better pass, but no optics references for now * mv rust/helm/optics-agent rust/helm/abacus-agent * Note on us not indexing updates * Get checkpointer working from local env var output	3 years ago
Nam Chu Hoai	cc8a8b40ea	Message Processor Error Handling (#294) ... * Refactor message processor * Fetch Proofs directly instead of the DB (#295) * Fetch Proofs directly instead of the DB * Support retrying of message processes with expoenential backoff (#308) * Support retrying of message processes with expoenential backoff * Minor message processing todos (#309) * Minor message processing todos * Configure relayer for immediate message processing * Catch prover inconsistency * Remove TODO and rely on configuration for now * Only retry when time to retry as come * PR review * PR review * Lint * lint	3 years ago
Nam Chu Hoai	f6168cbe47	Add MessageProcessor (#287) ... * Add MessageProcessor * Wait for indexed messages * PR review * Fix subtle message processing bugs (#290) * Fix suble message processing bugs * Lint * Simplify	3 years ago
Nam Chu Hoai	a9b509e973	Ignore messages during message processing in TipProver (#281)	3 years ago
Nam Chu Hoai	a34858b1be	Read checkpoint syncer from relayer settings (#274)	3 years ago
Nam Chu Hoai	d3fbd1f9f9	Add relayer scaffold (#225) ... * Add relayer scaffold * Lint * Add Outbox indexing to relayer and determine messages destined for inbox (#227) * Add Outbox indexing to relayer and determine messages destined for inbox * Lint * PR review * Support message processing in relayer (#232) * Support message processing in relayer * Lint * lint * PR review * Add clarifying comment * Be more specific between polling_interval and submission_latency	3 years ago
Nam Chu Hoai	d4c8720700	Rename rust packages (#128) ... * Rename rust packages * rustfmt * More rename * Rename update_abis * Rename metrics.rs	3 years ago
Luke Tchang	40d0ceb253	refactor/feature: split up indexing/contract/db logic + make watcher functional again (#925) ... * refactor: prefixes all loads and stores with home_name or entity in AgentDB * fmt: cargo fmt * refactor: use AgentDB everywhere and scap HomeDB * refactor: rename AgentDB OpticsDB * fix: put watcher back into cargo.toml and fix OpticsDB there * feature: adds ReplicaIndexer and moves index() to Common * refactor: implement CachingHome and CachingReplica + chain agnostic indexer, agent still broken due to not using CachingHome * prog: agent with CachingHome still broken; plan is to provide metrics obj and index-time * fix: contract sync stores messages too * refactor: successfully builds with CachingHome on agent that can spawn sync task * feature: adds unused CachingReplica * refactor: remove contract_name from indexer and give field to contract_sync * refactor: replaces all Replicas usage with CachingReplica * refactor: only CachingHome/Replica reach out to db and implement Event traits * fix: watcher tests work again * fix: removes db references from ethereum home and replica * rebase: rebase big indexer refactor on TypedDB/OpticsDB changes from updater split PR * fix: update crates and silence abigen! warnings for lint * feature: make watcher functional again * rename: spawn_sync --> sync * refactor: fetch_messages returns empty vec by default and home implements method * fix: update watcher-partial config and rename index tasks sync tasks * refactor: HomeIndexer now separate trait than CommonIndexer * doc: document Indexer trait and ContractSync * refactor: Homes/Replicas serve as Arc wrappers * fix: watcher stores update but not dependent on latest root * doc: document watcher fraud behavior * fix: indexers return _sorted_ updates and messages	3 years ago
Luke Tchang	f007a299c8	feature: add metrics/tracking to remaining agents (#922) ... * feature: add num relayed messages metrics/tracking to relayer * refactor: remove processor-specific nonce storage method from optics db * fix: use countervec for relayer instead of guage * feature: updater uses intcounter vec for submitted update count * fix: remove db from relayer since it now uses countervec with no rocksdb * fix: updates relayed not messages oops * fix: update submitter takes the correct countervec	3 years ago
Conner Swann	7b2c4a21df	refactor: split updater tasks for safety (#909) ... * refactor: split updater tasks, refactor opticsdb * feature: asrefs for db types * refactor: refactor typeddb to hold entity and pair with all loads/stores * ci: cargo +nightly * ci: use nightly toolchain * fix: use 1.56 stable not nightly * fix: updater can never overwrite an existing update with a conflicting one * fix: also check suggested update for state change guard Co-authored-by: Luke Tchang <ltchang@stanford.edu>	3 years ago
ember arlynx	a240facc6d	Balance exporter tool skeleton (#793) ... * First pass at balance exporter tool * this was broken with windows unicode paths * promote optics_base to a facade * optics-ethereum has many fewer lines of code * remove a layer of indirection * update all the other code * fmt * post-mege * flatten core::types::\* * impl Chain for ethers * move balance-exporter into tools * post-merge fixes * post-merge resolution * put Default back	3 years ago
ember arlynx	e23625fbef	feature: record span durations as a metric (#759) ... * record span durations as a metric * kathy is just like the others * show rest of comment its contents were hiding in my brain * -Default	3 years ago
James Prestwich	0aba12bb9c	Prestiwch/polygon metrics fix (#783) ... * refactor: move block height tracking to the HomeIndexer, temporarily disable replica block height tracking * chore: enhance updater main comment * refactor: beta reduction	3 years ago
James Prestwich	39b7fc5964	Prover sync refactoring (#781) ... * refactor: display impl for ProverSync * refactor: make prover internal only * refactor: remove non-functional gas escalator * refactor: disable watcher, task management refactor * fixed Docker build Co-authored-by: Conner Swann <me@connerswann.me>	3 years ago
ember arlynx	d5745362c6	rust agents: expose prometheus metrics (#670) ... * expose prometheus metrics from agents currently not actually updated in the course of execution * cargo update * log when starting metrics server, or not * metricsport -> metrics * metrics ought to be string * set content-type	3 years ago
Anna Carroll	6f5f51d97a	Remove queue from Replica (#703) ... * Remove queue from Replica * Update rust * Lint * Fix test order	3 years ago
James Prestwich	3d72c2db63	refactor: polling_interval -> interval, updater_pause -> pause (#701) ... * refactor: polling_interval -> interval, updater_pause -> pause * refactor: message_interval -> interval * helm updates for interval and pause Co-authored-by: Conner Swann <me@connerswann.me>	3 years ago
Anna Carroll	5ba0bed73a	Final Contract Cleaning + Renaming (#698) ... * Rename Pending to Proven; make case for enums consistent * Replica docs * GovernanceRouter docs * Fix test error message * XAppConnectionManager docs * Remove is_owner from rust * Remove isOwner test * Rename enqueue => dispatch * Update tests to remove enqueue * Rename enqueue in rust * Fix: disapatch * Rename sequences => nonces * Rename sequences => nonces * fix lint * Rename current => committedRoot * Rename leaf => messageHash * Rename endBlock => recoveryActiveAt * Fix test gen	3 years ago
ember arlynx	5f041c9db1	drive-by cleanups (#592) ... * reorganize rust code into dirs * rust: ignore vendor dir * how did kathy go missing? * drive-by cleanups: abstraction in update_abis, hyperlink, shell scripts are mostly shellcheck clean * rust readme: note agents dir * cargo fmt --all * remove stray comment * cargo update * why am i fighting my computer rn * patch over sloppy rebase * shore up test vector file handling * clarify how to add a new chain * add rust-toolchain.toml for 1.54 * DRY abis_dir * chore: add clippy installation to CI * github actions will use rust-toolchain.yaml now * move rust-toolchain so the action will find it rustup seems to scan upwards for this file, should pick it up fine when using from within * use obsolete rust-toolchain file location, because the action is naive * chore: specify toolchain in rust.yml * chore: specify toolchain in rust.yml in all workflows * chore: return clippy to rust.yml Co-authored-by: ember arlynx <ember@lunar.software> Co-authored-by: James Prestwich <james@prestwi.ch>	3 years ago