188 Commits (97c7f528016904502e40fb031444f0a8c189ada0)

Author	SHA1	Message	Date
webbhorn	dd63f70da1	Gelato Plumbing (#637) ... This PR prepares for the introduction of Gelato ForwardRequest API usage. However, it only refactors current code and introduces a prerequisite interface. In particular, the existing MessageProcessor was previously responsible both for message discovery via AbacusDB, as well as managing the submission of that message to the destination chain via some RPC provider. Since we want to have the option to deliver those messages over Gelato, a natural place to split things up might be between the AbacusDB message discovery phase, and the message delivery phase. This CL formalizes that notion by moving the submission logic into a SerialSubmitter submodule. The MessageProcessor and SerialSubmitter only communicate via a channel now. A subsequent PR will introduce GelatoSubmitter, which will route messages from the MessageProcessor to the Gelato API according to the new configuration options introduced in this PR.	2 years ago
Trevor Porter	954d4bc866	Remote checkpointer infra (#673) ... * Remote checkpointer infra * rm more values * fix	2 years ago
Nam Chu Hoai	c80426383c	Only pass non-zero reorg periods as lag to latest_checkpoint (#676)	2 years ago
Mattie Conover	02f32b61d0	Use current index for validator latest processed gauge init (#646)	2 years ago
Mattie Conover	de20710466	Rust updates (#644) ... * Update config and trim down macro code * Update rust * Package updates * Updates and cleanup * make clippy happy	2 years ago
Mattie Conover	fbf7130313	Provider request reduction (#638) ... * add contract function from call data to metric and no longer use histogram * fix warning * Get function names for metrics * Fix relayer metrics fetch interval * Fix validator metrics updates * initialize the latest processed checkpoint * Fix clippy errors	2 years ago
Nam Chu Hoai	8c9c38918e	Do not create inbox contracts if validator agent (#635)	2 years ago
Mattie Conover	f566bb40c8	Validator Metrics (#609) ... * minor refactoring * Add metrics to validator	2 years ago
Mattie Conover	4ec3e2124a	Even more metrics (#595) ... * Allow Raw messages to be parsed without cloning * Add hardcoded mapping between domain and chain * Remove extra clone * Report dst chain for `dispatch` phase of `last_known_message_leaf_index` * Update state call for outbox * Add outbox state metric	2 years ago
webbhorn	630e20bbc2	validator: Ensure non-empty outbox on init. (#586) ... validator: Ensure non-empty outbox on init. This addresses the issue described in #575 wherein validators crash after new contract deployment due to failed calls on outbox.latest_checkpoint(). The calls fail due to u256 underflow on count()-1, since count() is zero on init, and latest_checkpoint() wants to return an **index**, which is count() - 1. This PR adds a retry loop just prior to entering the existing main validator-submit-loop. I chose to poll outbox.count() rather than try to decode the error code or revert reason on failed calls to latest_checkpoint() so that we avoid tightly coupling this behavior to particular error codes or messages, and in fact may want to preserve exisitng behavior when calls to outbox.latest_checkpoint() fail for other reasons. Tested with a modified `rust/run-locally.sh` which asks Kathy to sleep for 5 minutes prior to sending messages. Without this change, the validator crashes. With this change, the validator does not crash, and after 5 minutes, starts signing checkpoints as designed.	3 years ago
Mattie Conover	4d9603c868	Relayer whitelist (#570) ... * Update ABIs, compiles. Removed Inbox indexer and cached checkpoints from the Inbox * Rm checkpoint indexing test * Update InboxValidatorManager, just need to connect the pieces now * Add channels * So close * Fix InboxValidatorManager deploy * Make message processor slightly more readable * Create loop-control crate * Try to bring some sanity to message processor * Fix bug where prover_sync wasn't in line with latest_signed_checkpoint * Rm immediate message processing, clean up * rm abacus-cli * more cleanup * rename / rm some settings * TS renames / rms * Lower some processing failure logs to info * Checkpoint fetcher doesn't need the CommittedMessages * Hardcode kathy dispatching * Move to watch channel * I'm sorry clippy * nits * rm some nonce related stuff * more refactoring * cleanup * Add whitelist settings * Whitelist filtering * fix docker build * use relayed terminology * Change whitelist tuple order * naming change * move whitelist check to before fetching proof * Update span event * make messages singular * Update error * minor cleanup * Update configs in scripts * Whitelist testing and parsing fixes Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	3 years ago
Trevor Porter	1793fffbeb	Testnet2 agent deployment (#568) ... * Deploy testnet2 contracts * Add testnet2 configs to rust * don't panic for pre-london chains * Don't checkpoint, retry fewer times, and configure kathy properly * clean up * cleaninngggg Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local>	3 years ago
Asa Oines	0a81c910a3	Move InterchainGasPaymaster from AbacusConnectionManager to AbacusConnectionClient (#558)	3 years ago
Trevor Porter	d120a76c1e	No more cached checkpoints in agents (#550) ... * Update ABIs, compiles. Removed Inbox indexer and cached checkpoints from the Inbox * Rm checkpoint indexing test * Update InboxValidatorManager, just need to connect the pieces now * Add channels * So close * Fix InboxValidatorManager deploy * Fix bug where prover_sync wasn't in line with latest_signed_checkpoint * Rm immediate message processing, clean up * rm abacus-cli * more cleanup * rename / rm some settings * TS renames / rms * Lower some processing failure logs to info * Checkpoint fetcher doesn't need the CommittedMessages * Hardcode kathy dispatching * Move to watch channel * I'm sorry clippy * nits * rm some nonce related stuff * PR comments	3 years ago
Trevor Porter	29970fe326	Index GasPayment events in Relayer (#504) ... * Add InterchainGasPaymaster traits and ethereum structs * Add abacuse-base interchain_gas * Actually setting up indexer * Setting up DB usage * wip * Add interchain gas paymaster to AbacusAgentCore * Sync gas payments in the relayer * It runs wooooooo * Use U256 for gas payments * nits, rm InterchainGasPaymasterEvents * Add chain_name, fix metrics * More name -> chain_name rename * Correctly rename contract_name => chain_name, move InterchainGasPayment from traits to types * split up contract_sync a bit * cargo fmt * nit * clean * Add metrics * to_string => encode_hex * make clippy happy * move contract_sync/common.rs stuff into mod.rs * string interpolation for logs * some nits * mv types/interchain_gas into types/mod * Idempotent GasPayment indexing * Use new contract ABI & update rust code * Rename RandomBadRecipient -> TestSendReceiver * use nit * only index finalized blocks * Run with finalityBlocks * Add InterchainGasPaymaster to update_abis * Some PR comments	3 years ago
Trevor Porter	5050eee9a4	Use new contract ABI & update rust code (#519)	3 years ago
Mattie Conover	71ce3e285b	Fix duplicate metric labels (#518)	3 years ago
Mattie Conover	d687d3e230	More Metrics (#495) ... * metrics refactoring * metrics refactoring * Make brain hurt less by using a damn function * patch through metrics to abacus_ethereum inbox and outbox * WIP * Minor cleanup * work on time update * added label arrays * refactoring * Metrics for sending txns * Metrics middleware * cleanup * Cleanup after merge * Add metrics config and setup prom metrics with core metrics * Minor fixes to docs * Better trait building * rename * Weave metrics through code * Use core metrics instead * Removed some unused code * fmt * change feature layout * fix docker build * Start scraping metrics on an interval * fmt * fix duplicate label error * change features * fix u256 conversion fn * refactor * Support default values in config * Minor fixes * Use f64 conversion from primitive-types * remove unused code * fix const name * improve docs * better var name * remove unused code * remove unused code	3 years ago
webbhorn	6b3822f664	Misc. agent cleanup and dead code deletion (#470) ... * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in comment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool. * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in coment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool.	3 years ago
Mattie Conover	f78e79ab9b	Better Agent Error Logging (#412) ... * Make errors print on one line * Replace color-eyre with eyre in core and base * Replace color-eyre with eyre other libs * Log should be imported without patch version * Update tracing and opentelemetry * Only need openssl dynamic libs * Fix no-default-feature build * Custom eyre handler * Removed unused log dep * Cleanup paste dep * More package cleanup * Update tokio * Minor refactor	3 years ago
Nam Chu Hoai	cc6c2db5d8	Testnet deploy (#369) ... * Got keys for testnet * Testnet deploy * Add rust configs * prettier * Nits, update governance config * new agent image * Rename secret reference * Have kathy also checkpoint * Only checkpoint when count > 1 * Move relayer to unique key for (source, remote) pair * prettier * Clean up get-aws-key-addresses * get GCP keys half working * let's clean this up * First steps at refactor * Migrate delete-keys over * support key rotation * Create and get key all working * update relayer external secret * Prettier * Update ethers-rs * Debugging testnet deployment (#370) * Dont fail on inbox checkpointing tx error * fix docker build * Debug * More instrumentation * Lint * Use the correct signers * Lint * prettier * suffix -> index, go back to 1 key per agent * rm relayer user * Create user and key identifiers * Move some fns to key-utils, fix gcp secret names * Correct kathy & checkpointer key names * Back to Promse.all in deploy-agents, clean * Don't require GCP keys to be created with agentconfig * fix * pure json response for get-key-addresses Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	3 years ago
Nam Chu Hoai	cee9a2d8ef	Add more tracing information (#362) ... * Add more tracing information * PR review	3 years ago
Nam Chu Hoai	5e254f0512	Add gauges to track agents (#360)	3 years ago
Nam Chu Hoai	e5ea4d42a9	0-indexed Checkpoints (#344) ... * Checkpoints are zero indexed * Checkpoints are not one-indexed anymore * Agent changes for 0-indexed checkpoints * Disallow checkpoint index 0 * Can just check indicies again * Proper isCheckpoint * Dispatch dummy message * fix before * PR review Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local>	3 years ago
Nam Chu Hoai	0216d28dcf	Merge Inbox prove and process (#339) ... * Merge Inbox prove and process * PR reviews * Package-lock	3 years ago
Trevor Porter	b5f121b837	Multisig validator rust agents (#342) ... * add MultisigCheckpointSyncer * cargo fmt * Nits * Some comments, clean up * rm inbox's submit_checkpoint, get settings compiling * Move some checkpoint structs to core, add inbox validator manager base & core * Update existing Inbox/Outbox ABIs, add InboxValidatorManager ABI * Add ethereum inbox validator manager * Add inbox validator manager to AbacusAgentCore * Put everything together * Updated some configs, nearly able to run the relayer * Update config to have InboxValidatorManager alongside Inbox * Able to run it * Remove now unnecessary AbacusAgentCore::new * Make MultisigSignedCheckpoint::try_from not take in a mut * Use InboxContracts in relayer * cleaning up * nit * clippy * Check all checkpoints are the same in the try_from for MultisigSignedCheckpoint	3 years ago
Trevor Porter	fec07d21fe	Remove docs & all optics-related rust (#322) ... * optics -> abacus in infra * rm docs * rename contract-metrics prom metrics optics-* -> abacus-* * Rm watcher, updater, a bunch of optics things in abacus-base * rm agents/processor * Removing a whole lot of updater / home / replica stuff from abacus-core * Deleted a lot throughout -- some still commented out and a lot of cleanup needed * Slowly cleaning up, kathy is where the remaining errors are * Delete some commented out code * makes kathy an abacus agent * compiling woo * cargo fmt * compiling with all features * removing some update references * Ignore handles_missing_rpc_checkpoints * cargo fmt * Add signedCheckpoint.json * rm some references to updates, delete unnecessary abis * rm all references to home or replica * compiling, remove som e unused code * fix test warning * nit * New README - needs a better pass, but no optics references for now * mv rust/helm/optics-agent rust/helm/abacus-agent * Note on us not indexing updates * Get checkpointer working from local env var output	3 years ago
Nam Chu Hoai	cc8a8b40ea	Message Processor Error Handling (#294) ... * Refactor message processor * Fetch Proofs directly instead of the DB (#295) * Fetch Proofs directly instead of the DB * Support retrying of message processes with expoenential backoff (#308) * Support retrying of message processes with expoenential backoff * Minor message processing todos (#309) * Minor message processing todos * Configure relayer for immediate message processing * Catch prover inconsistency * Remove TODO and rely on configuration for now * Only retry when time to retry as come * PR review * PR review * Lint * lint	3 years ago
Nam Chu Hoai	ae3c26189f	Start from last signed checkpoint (#292) ... Co-authored-by: Asa Oines <asaoines@gmail.com>	3 years ago
Nam Chu Hoai	f6168cbe47	Add MessageProcessor (#287) ... * Add MessageProcessor * Wait for indexed messages * PR review * Fix subtle message processing bugs (#290) * Fix suble message processing bugs * Lint * Simplify	3 years ago
Nam Chu Hoai	a9b509e973	Ignore messages during message processing in TipProver (#281)	3 years ago
Nam Chu Hoai	a34858b1be	Read checkpoint syncer from relayer settings (#274)	3 years ago
Trevor Porter	328941fc2c	Checkpointer policy v1 (#217) ... * Add script for running local dev environment * Prettier * Move Checkpointer to (Abacus)Agent * Lint * Fix * Commit dev files to run checkpointer * Use @abacus-network/utils in solidity/abacus-core/scripts/abacus.ts * Compiles, some warnings to think of * rename checkpoint() -> create_checkpoint() * cargo fmt * Move to index based decision * More info * Move from (H256, u32) to Checkpoint * cargo fmt * Add some scripts for testing * Actually fix merge conflicts * Add latency * Nits * rm testing utils * rm 1000_config.json * nit * fix package-lock.json * Better names * Sleep instead of using last_checkpoint_time * cargo fmt * Rm comment * Add checkpointer to output-agent-env-vars Co-authored-by: Asa Oines <a@celo.org> Co-authored-by: nambrot <nambrot@googlemail.com>	3 years ago
Nam Chu Hoai	9ab0c16dd6	Add CheckpointSyncer configuration in Validator and support S3 (#266) ... * Add CheckpointSyncer configuration in Validator and support S3 * Lint * PR review	3 years ago
Nam Chu Hoai	d3fbd1f9f9	Add relayer scaffold (#225) ... * Add relayer scaffold * Lint * Add Outbox indexing to relayer and determine messages destined for inbox (#227) * Add Outbox indexing to relayer and determine messages destined for inbox * Lint * PR review * Support message processing in relayer (#232) * Support message processing in relayer * Lint * lint * PR review * Add clarifying comment * Be more specific between polling_interval and submission_latency	3 years ago
Nam Chu Hoai	4a28875af8	Add Validator scaffold (#213)	3 years ago
Nam Chu Hoai	72ae94e655	Make checkpointer the truly first Abacus agent (#197) ... * Move Checkpointer to (Abacus)Agent * Lint * Fix	3 years ago
Nam Chu Hoai	2cb643486f	Add checkpointer scaffold (#183) ... * Add checkpointer scaffold * Add Abacus ABIs to abacus-ethereum (#186) * Add Abacus ABIs * Build on all PRs * PR comments * lint * Lint * Add new Abacus Outbox/Inbox wrappers (#189) * Add new Abacus Outbox/Inbox wrappers * PR review * PR comments	3 years ago
Nam Chu Hoai	8261b2352b	Auto-heal missing updates (#175) ... * Auto-heal missing updates * Log the committed root for the UpdateSubmitter * Use the already exposed function * Compare against the suggested previous root	3 years ago
Nam Chu Hoai	23c3e4cd22	Add lookback for updates/messages in contract_sync (#144) ... * Apply contract_sync mod patch * Add IndexDataTypes * Fmt * Do not store updates/messages when encountering an invalid continuation If we store them we may end up invariably overwriting the latest_leaf_index database entry which keeps track of "contigousness". I.e. this would cause holes of size >1 to cause the agents being unable to recover * Index both messages and updates on the processor * remove comment * Followups for the contract-sync patch (#146) * Readd abacus_last_known_message_leaf_index to message indexing in the processor * Remove tipbuffer functionality * Additional contract_sync tests (#147) * Fix test case for when batch includes the already known updates/messages * Modify tests to include multiple ranges of missing updates/events	3 years ago
Nam Chu Hoai	e3c1b3bdcc	Support tip buffers for contract_sync (#138) ... * Support tip buffers for contract_sync * Pass IndexSettings * Add tipBuffer to deploy tooling * Run prettier * Deploy on dev	3 years ago
Nam Chu Hoai	d4c8720700	Rename rust packages (#128) ... * Rename rust packages * rustfmt * More rename * Rename update_abis * Rename metrics.rs	3 years ago
Nam Chu Hoai	91a411e2c9	Add gauge to track last_known_message_leaf_index (#117) ... * Add gauge to track last_known_message_leaf_index * Fix lint * Lint * Lint	3 years ago
Nam Chu Hoai	45d9ce9e9b	Repeat processor reverted txs (#48) ... * Repeat processor reverted txs * Adjust error message	3 years ago
nambrot	632cc53ee1	rustfmt	3 years ago
Nam Chu Hoai	8877a98aa8	Support selective index-only for processors (#17) ... * Support selective index-only for processors * Support no indexonly config	3 years ago
Nam Chu Hoai	b6caa0da55	Fix S3 proof pushing (#13) ... * Fix S3 proof pushing * Support AWS in getAgentVars * Add staging output-agent-env-vars	3 years ago
Nam Chu Hoai	a3c08207a3	Set default updater pause/interval for productionCommunity (#6) ... * Set default updater pause/interval for productionCommunity * Fix lint	3 years ago
Luke Tchang	40d0ceb253	refactor/feature: split up indexing/contract/db logic + make watcher functional again (#925) ... * refactor: prefixes all loads and stores with home_name or entity in AgentDB * fmt: cargo fmt * refactor: use AgentDB everywhere and scap HomeDB * refactor: rename AgentDB OpticsDB * fix: put watcher back into cargo.toml and fix OpticsDB there * feature: adds ReplicaIndexer and moves index() to Common * refactor: implement CachingHome and CachingReplica + chain agnostic indexer, agent still broken due to not using CachingHome * prog: agent with CachingHome still broken; plan is to provide metrics obj and index-time * fix: contract sync stores messages too * refactor: successfully builds with CachingHome on agent that can spawn sync task * feature: adds unused CachingReplica * refactor: remove contract_name from indexer and give field to contract_sync * refactor: replaces all Replicas usage with CachingReplica * refactor: only CachingHome/Replica reach out to db and implement Event traits * fix: watcher tests work again * fix: removes db references from ethereum home and replica * rebase: rebase big indexer refactor on TypedDB/OpticsDB changes from updater split PR * fix: update crates and silence abigen! warnings for lint * feature: make watcher functional again * rename: spawn_sync --> sync * refactor: fetch_messages returns empty vec by default and home implements method * fix: update watcher-partial config and rename index tasks sync tasks * refactor: HomeIndexer now separate trait than CommonIndexer * doc: document Indexer trait and ContractSync * refactor: Homes/Replicas serve as Arc wrappers * fix: watcher stores update but not dependent on latest root * doc: document watcher fraud behavior * fix: indexers return _sorted_ updates and messages	3 years ago
Luke Tchang	f007a299c8	feature: add metrics/tracking to remaining agents (#922) ... * feature: add num relayed messages metrics/tracking to relayer * refactor: remove processor-specific nonce storage method from optics db * fix: use countervec for relayer instead of guage * feature: updater uses intcounter vec for submitted update count * fix: remove db from relayer since it now uses countervec with no rocksdb * fix: updates relayed not messages oops * fix: update submitter takes the correct countervec	3 years ago