176 Commits (82ebc01054582d88196e0dfc101cf41c46624df1)

Author	SHA1	Message	Date
Asa Oines	0a81c910a3	Move InterchainGasPaymaster from AbacusConnectionManager to AbacusConnectionClient (#558)	3 years ago
Trevor Porter	d120a76c1e	No more cached checkpoints in agents (#550) ... * Update ABIs, compiles. Removed Inbox indexer and cached checkpoints from the Inbox * Rm checkpoint indexing test * Update InboxValidatorManager, just need to connect the pieces now * Add channels * So close * Fix InboxValidatorManager deploy * Fix bug where prover_sync wasn't in line with latest_signed_checkpoint * Rm immediate message processing, clean up * rm abacus-cli * more cleanup * rename / rm some settings * TS renames / rms * Lower some processing failure logs to info * Checkpoint fetcher doesn't need the CommittedMessages * Hardcode kathy dispatching * Move to watch channel * I'm sorry clippy * nits * rm some nonce related stuff * PR comments	3 years ago
Trevor Porter	29970fe326	Index GasPayment events in Relayer (#504) ... * Add InterchainGasPaymaster traits and ethereum structs * Add abacuse-base interchain_gas * Actually setting up indexer * Setting up DB usage * wip * Add interchain gas paymaster to AbacusAgentCore * Sync gas payments in the relayer * It runs wooooooo * Use U256 for gas payments * nits, rm InterchainGasPaymasterEvents * Add chain_name, fix metrics * More name -> chain_name rename * Correctly rename contract_name => chain_name, move InterchainGasPayment from traits to types * split up contract_sync a bit * cargo fmt * nit * clean * Add metrics * to_string => encode_hex * make clippy happy * move contract_sync/common.rs stuff into mod.rs * string interpolation for logs * some nits * mv types/interchain_gas into types/mod * Idempotent GasPayment indexing * Use new contract ABI & update rust code * Rename RandomBadRecipient -> TestSendReceiver * use nit * only index finalized blocks * Run with finalityBlocks * Add InterchainGasPaymaster to update_abis * Some PR comments	3 years ago
Trevor Porter	5050eee9a4	Use new contract ABI & update rust code (#519)	3 years ago
Mattie Conover	71ce3e285b	Fix duplicate metric labels (#518)	3 years ago
Mattie Conover	d687d3e230	More Metrics (#495) ... * metrics refactoring * metrics refactoring * Make brain hurt less by using a damn function * patch through metrics to abacus_ethereum inbox and outbox * WIP * Minor cleanup * work on time update * added label arrays * refactoring * Metrics for sending txns * Metrics middleware * cleanup * Cleanup after merge * Add metrics config and setup prom metrics with core metrics * Minor fixes to docs * Better trait building * rename * Weave metrics through code * Use core metrics instead * Removed some unused code * fmt * change feature layout * fix docker build * Start scraping metrics on an interval * fmt * fix duplicate label error * change features * fix u256 conversion fn * refactor * Support default values in config * Minor fixes * Use f64 conversion from primitive-types * remove unused code * fix const name * improve docs * better var name * remove unused code * remove unused code	3 years ago
webbhorn	6b3822f664	Misc. agent cleanup and dead code deletion (#470) ... * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in comment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool. * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in coment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool.	3 years ago
Mattie Conover	f78e79ab9b	Better Agent Error Logging (#412) ... * Make errors print on one line * Replace color-eyre with eyre in core and base * Replace color-eyre with eyre other libs * Log should be imported without patch version * Update tracing and opentelemetry * Only need openssl dynamic libs * Fix no-default-feature build * Custom eyre handler * Removed unused log dep * Cleanup paste dep * More package cleanup * Update tokio * Minor refactor	3 years ago
Nam Chu Hoai	cc6c2db5d8	Testnet deploy (#369) ... * Got keys for testnet * Testnet deploy * Add rust configs * prettier * Nits, update governance config * new agent image * Rename secret reference * Have kathy also checkpoint * Only checkpoint when count > 1 * Move relayer to unique key for (source, remote) pair * prettier * Clean up get-aws-key-addresses * get GCP keys half working * let's clean this up * First steps at refactor * Migrate delete-keys over * support key rotation * Create and get key all working * update relayer external secret * Prettier * Update ethers-rs * Debugging testnet deployment (#370) * Dont fail on inbox checkpointing tx error * fix docker build * Debug * More instrumentation * Lint * Use the correct signers * Lint * prettier * suffix -> index, go back to 1 key per agent * rm relayer user * Create user and key identifiers * Move some fns to key-utils, fix gcp secret names * Correct kathy & checkpointer key names * Back to Promse.all in deploy-agents, clean * Don't require GCP keys to be created with agentconfig * fix * pure json response for get-key-addresses Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	3 years ago
Nam Chu Hoai	cee9a2d8ef	Add more tracing information (#362) ... * Add more tracing information * PR review	3 years ago
Nam Chu Hoai	5e254f0512	Add gauges to track agents (#360)	3 years ago
Nam Chu Hoai	e5ea4d42a9	0-indexed Checkpoints (#344) ... * Checkpoints are zero indexed * Checkpoints are not one-indexed anymore * Agent changes for 0-indexed checkpoints * Disallow checkpoint index 0 * Can just check indicies again * Proper isCheckpoint * Dispatch dummy message * fix before * PR review Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local>	3 years ago
Nam Chu Hoai	0216d28dcf	Merge Inbox prove and process (#339) ... * Merge Inbox prove and process * PR reviews * Package-lock	3 years ago
Trevor Porter	b5f121b837	Multisig validator rust agents (#342) ... * add MultisigCheckpointSyncer * cargo fmt * Nits * Some comments, clean up * rm inbox's submit_checkpoint, get settings compiling * Move some checkpoint structs to core, add inbox validator manager base & core * Update existing Inbox/Outbox ABIs, add InboxValidatorManager ABI * Add ethereum inbox validator manager * Add inbox validator manager to AbacusAgentCore * Put everything together * Updated some configs, nearly able to run the relayer * Update config to have InboxValidatorManager alongside Inbox * Able to run it * Remove now unnecessary AbacusAgentCore::new * Make MultisigSignedCheckpoint::try_from not take in a mut * Use InboxContracts in relayer * cleaning up * nit * clippy * Check all checkpoints are the same in the try_from for MultisigSignedCheckpoint	3 years ago
Trevor Porter	fec07d21fe	Remove docs & all optics-related rust (#322) ... * optics -> abacus in infra * rm docs * rename contract-metrics prom metrics optics-* -> abacus-* * Rm watcher, updater, a bunch of optics things in abacus-base * rm agents/processor * Removing a whole lot of updater / home / replica stuff from abacus-core * Deleted a lot throughout -- some still commented out and a lot of cleanup needed * Slowly cleaning up, kathy is where the remaining errors are * Delete some commented out code * makes kathy an abacus agent * compiling woo * cargo fmt * compiling with all features * removing some update references * Ignore handles_missing_rpc_checkpoints * cargo fmt * Add signedCheckpoint.json * rm some references to updates, delete unnecessary abis * rm all references to home or replica * compiling, remove som e unused code * fix test warning * nit * New README - needs a better pass, but no optics references for now * mv rust/helm/optics-agent rust/helm/abacus-agent * Note on us not indexing updates * Get checkpointer working from local env var output	3 years ago
Nam Chu Hoai	cc8a8b40ea	Message Processor Error Handling (#294) ... * Refactor message processor * Fetch Proofs directly instead of the DB (#295) * Fetch Proofs directly instead of the DB * Support retrying of message processes with expoenential backoff (#308) * Support retrying of message processes with expoenential backoff * Minor message processing todos (#309) * Minor message processing todos * Configure relayer for immediate message processing * Catch prover inconsistency * Remove TODO and rely on configuration for now * Only retry when time to retry as come * PR review * PR review * Lint * lint	3 years ago
Nam Chu Hoai	ae3c26189f	Start from last signed checkpoint (#292) ... Co-authored-by: Asa Oines <asaoines@gmail.com>	3 years ago
Nam Chu Hoai	f6168cbe47	Add MessageProcessor (#287) ... * Add MessageProcessor * Wait for indexed messages * PR review * Fix subtle message processing bugs (#290) * Fix suble message processing bugs * Lint * Simplify	3 years ago
Nam Chu Hoai	a9b509e973	Ignore messages during message processing in TipProver (#281)	3 years ago
Nam Chu Hoai	a34858b1be	Read checkpoint syncer from relayer settings (#274)	3 years ago
Trevor Porter	328941fc2c	Checkpointer policy v1 (#217) ... * Add script for running local dev environment * Prettier * Move Checkpointer to (Abacus)Agent * Lint * Fix * Commit dev files to run checkpointer * Use @abacus-network/utils in solidity/abacus-core/scripts/abacus.ts * Compiles, some warnings to think of * rename checkpoint() -> create_checkpoint() * cargo fmt * Move to index based decision * More info * Move from (H256, u32) to Checkpoint * cargo fmt * Add some scripts for testing * Actually fix merge conflicts * Add latency * Nits * rm testing utils * rm 1000_config.json * nit * fix package-lock.json * Better names * Sleep instead of using last_checkpoint_time * cargo fmt * Rm comment * Add checkpointer to output-agent-env-vars Co-authored-by: Asa Oines <a@celo.org> Co-authored-by: nambrot <nambrot@googlemail.com>	3 years ago
Nam Chu Hoai	9ab0c16dd6	Add CheckpointSyncer configuration in Validator and support S3 (#266) ... * Add CheckpointSyncer configuration in Validator and support S3 * Lint * PR review	3 years ago
Nam Chu Hoai	d3fbd1f9f9	Add relayer scaffold (#225) ... * Add relayer scaffold * Lint * Add Outbox indexing to relayer and determine messages destined for inbox (#227) * Add Outbox indexing to relayer and determine messages destined for inbox * Lint * PR review * Support message processing in relayer (#232) * Support message processing in relayer * Lint * lint * PR review * Add clarifying comment * Be more specific between polling_interval and submission_latency	3 years ago
Nam Chu Hoai	4a28875af8	Add Validator scaffold (#213)	3 years ago
Nam Chu Hoai	72ae94e655	Make checkpointer the truly first Abacus agent (#197) ... * Move Checkpointer to (Abacus)Agent * Lint * Fix	3 years ago
Nam Chu Hoai	2cb643486f	Add checkpointer scaffold (#183) ... * Add checkpointer scaffold * Add Abacus ABIs to abacus-ethereum (#186) * Add Abacus ABIs * Build on all PRs * PR comments * lint * Lint * Add new Abacus Outbox/Inbox wrappers (#189) * Add new Abacus Outbox/Inbox wrappers * PR review * PR comments	3 years ago
Nam Chu Hoai	8261b2352b	Auto-heal missing updates (#175) ... * Auto-heal missing updates * Log the committed root for the UpdateSubmitter * Use the already exposed function * Compare against the suggested previous root	3 years ago
Nam Chu Hoai	23c3e4cd22	Add lookback for updates/messages in contract_sync (#144) ... * Apply contract_sync mod patch * Add IndexDataTypes * Fmt * Do not store updates/messages when encountering an invalid continuation If we store them we may end up invariably overwriting the latest_leaf_index database entry which keeps track of "contigousness". I.e. this would cause holes of size >1 to cause the agents being unable to recover * Index both messages and updates on the processor * remove comment * Followups for the contract-sync patch (#146) * Readd abacus_last_known_message_leaf_index to message indexing in the processor * Remove tipbuffer functionality * Additional contract_sync tests (#147) * Fix test case for when batch includes the already known updates/messages * Modify tests to include multiple ranges of missing updates/events	3 years ago
Nam Chu Hoai	e3c1b3bdcc	Support tip buffers for contract_sync (#138) ... * Support tip buffers for contract_sync * Pass IndexSettings * Add tipBuffer to deploy tooling * Run prettier * Deploy on dev	3 years ago
Nam Chu Hoai	d4c8720700	Rename rust packages (#128) ... * Rename rust packages * rustfmt * More rename * Rename update_abis * Rename metrics.rs	3 years ago
Nam Chu Hoai	91a411e2c9	Add gauge to track last_known_message_leaf_index (#117) ... * Add gauge to track last_known_message_leaf_index * Fix lint * Lint * Lint	3 years ago
Nam Chu Hoai	45d9ce9e9b	Repeat processor reverted txs (#48) ... * Repeat processor reverted txs * Adjust error message	3 years ago
nambrot	632cc53ee1	rustfmt	3 years ago
Nam Chu Hoai	8877a98aa8	Support selective index-only for processors (#17) ... * Support selective index-only for processors * Support no indexonly config	3 years ago
Nam Chu Hoai	b6caa0da55	Fix S3 proof pushing (#13) ... * Fix S3 proof pushing * Support AWS in getAgentVars * Add staging output-agent-env-vars	3 years ago
Nam Chu Hoai	a3c08207a3	Set default updater pause/interval for productionCommunity (#6) ... * Set default updater pause/interval for productionCommunity * Fix lint	3 years ago
Luke Tchang	40d0ceb253	refactor/feature: split up indexing/contract/db logic + make watcher functional again (#925) ... * refactor: prefixes all loads and stores with home_name or entity in AgentDB * fmt: cargo fmt * refactor: use AgentDB everywhere and scap HomeDB * refactor: rename AgentDB OpticsDB * fix: put watcher back into cargo.toml and fix OpticsDB there * feature: adds ReplicaIndexer and moves index() to Common * refactor: implement CachingHome and CachingReplica + chain agnostic indexer, agent still broken due to not using CachingHome * prog: agent with CachingHome still broken; plan is to provide metrics obj and index-time * fix: contract sync stores messages too * refactor: successfully builds with CachingHome on agent that can spawn sync task * feature: adds unused CachingReplica * refactor: remove contract_name from indexer and give field to contract_sync * refactor: replaces all Replicas usage with CachingReplica * refactor: only CachingHome/Replica reach out to db and implement Event traits * fix: watcher tests work again * fix: removes db references from ethereum home and replica * rebase: rebase big indexer refactor on TypedDB/OpticsDB changes from updater split PR * fix: update crates and silence abigen! warnings for lint * feature: make watcher functional again * rename: spawn_sync --> sync * refactor: fetch_messages returns empty vec by default and home implements method * fix: update watcher-partial config and rename index tasks sync tasks * refactor: HomeIndexer now separate trait than CommonIndexer * doc: document Indexer trait and ContractSync * refactor: Homes/Replicas serve as Arc wrappers * fix: watcher stores update but not dependent on latest root * doc: document watcher fraud behavior * fix: indexers return _sorted_ updates and messages	3 years ago
Luke Tchang	f007a299c8	feature: add metrics/tracking to remaining agents (#922) ... * feature: add num relayed messages metrics/tracking to relayer * refactor: remove processor-specific nonce storage method from optics db * fix: use countervec for relayer instead of guage * feature: updater uses intcounter vec for submitted update count * fix: remove db from relayer since it now uses countervec with no rocksdb * fix: updates relayed not messages oops * fix: update submitter takes the correct countervec	3 years ago
Conner Swann	7b2c4a21df	refactor: split updater tasks for safety (#909) ... * refactor: split updater tasks, refactor opticsdb * feature: asrefs for db types * refactor: refactor typeddb to hold entity and pair with all loads/stores * ci: cargo +nightly * ci: use nightly toolchain * fix: use 1.56 stable not nightly * fix: updater can never overwrite an existing update with a conflicting one * fix: also check suggested update for state change guard Co-authored-by: Luke Tchang <ltchang@stanford.edu>	3 years ago
Luke Tchang	ea32adf3af	refactor: scrap HomeDB for generalized OpticsDB (#902) ... * refactor: comment out homedb and have typeddb use arg prefixes * refactor: prefixes all loads and stores with home_name or entity in AgentDB * fmt: cargo fmt * fix: keyed encodable/decodable * refactor: use AgentDB everywhere and scap HomeDB * refactor: rename AgentDB OpticsDB * fix: put watcher back into cargo.toml and fix OpticsDB there	3 years ago
ember arlynx	a240facc6d	Balance exporter tool skeleton (#793) ... * First pass at balance exporter tool * this was broken with windows unicode paths * promote optics_base to a facade * optics-ethereum has many fewer lines of code * remove a layer of indirection * update all the other code * fmt * post-mege * flatten core::types::\* * impl Chain for ethers * move balance-exporter into tools * post-merge fixes * post-merge resolution * put Default back	3 years ago
Luke Tchang	eb685bdbf8	refactor: watcher home indexing rewrite (#809) ... * refactor: add HomeDB struct (raw db not replaced yet) * refactor: EthereumHome and HomeIndexer use HomeDB instead of DB * refactor: UpdateHandler uses HomeDB * refactor: processor/prover-sync use HomeDB * refactor: error cleanup * refactor: implements run_all without metrics * refactor: watcher uses oneshot to receive double update * fix: update comments * fix: panic on run and run_many * tracing: add more info when double update found * rebase: fix store_latest_update rename * test: add default impls to make watcher unit tests compile * fix: run_loop takes HomeDB, watcher struct holds DB * run_watch_tasks borrows &self instead of many params * rebase: fix unused imports	3 years ago
Luke Tchang	0e5b04bd1f	fix: increment next_message_nonce for ReplicaProcessor and temporarily fix stuck processor (#879) ... * metrics: increment next_nonce for ReplicaProcessor and add info spans * obsv: info spans in home indexer * fix linter * obsv: add origin chain to message retrieval info span * actually copy clis into release image * FIX: proversync always stores proofs during spawn loop (TEMP FIX) * fix: debug -> info span * tracing: add info span for processing new message Co-authored-by: Conner Swann <me@connerswann.me>	3 years ago
James Prestwich	985bd7850c	feature: configuration to push proofs to s3 buckets (#821) ... * feature: configuration to push proofs to s3 buckets * refactor: key s3 proofs by their index, not their leaf hash * chore: improve processor bootup logging * fix: add name and bucket to debug for Pusher * bug: check that content-length is gt 0 * bug: expect 3 labels in next_message_nonce metric * bug: properly handle non-existent key * fix: log messages for proof pushing * feature: upload message to s3 with proof	3 years ago
Luke Tchang	1248604f5d	fix: fix kathy sync bug related to `ethers-rs` concurrency (#859) ... * fix: have kathy acquire lock so only one home.dispatch at a time * nit: drop(guard)	3 years ago
Luke Tchang	07bd730d4e	fix: fix processor index/nonce conflation and add per-replica nonce storage (#850) ... * docs/fix: fixes nonce/index conflation in processor and adds kv store docs * fix: rename next_index next_nonce and add new IntGaugeVec for each Replica processor * fix: Replica processors share single int guage vec for next_nonces * fix: change db keys to be prefixed with value indicator + standardize use of leaf instead of leaf_hash * docs: fix out of date docs * fix: rebase on main and fix cli to use leaf instead of leaf_hash * fix: replica processor label matches int gauge vec labels	3 years ago
Conner Swann	6275eaa3f5	fix: fix overloaded index config var (#843)	3 years ago
James Prestwich	eed0b96c51	feature: optionally skip replica tasks (#839) ... * feature: optionally skip replica tasks * refactor: use index_only in processor	3 years ago
Luke Tchang	890c5495bf	fix: fix potential out-of-order updates when `HomeIndexer` syncing (#824) ... * fix: order update events and add race condition TODO * docs: remove race condition comment * doc: clarify edge case where we try to prove leaf with no update yet * fix: proversync check stops at prover.count() * rebase: fix merge conflicts after rebasing on HomeDB merged PR * fix: proversync check stops at prover.count()	3 years ago
Luke Tchang	8e54b713bd	refactor: deduplicate db keys by adding `HomeDB` (#803)	3 years ago