hyperlane-monorepo

Commit Graph

Author	SHA1	Message	Date
Daniel Savu	e737d998ff	Cosmos block-by-block indexing (#3245 ) ### Description Implements cosmos block-by-block indexing, parallelizing where possible: the `block` and `block_results` calls, and the individual block queries within a range. Also reuses the existing `handle_txs` logic, to improve readability. ### Drive-by changes Adds `tokio::task::JoinError` to the `hyperlane-core` error enum ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3234 - Added some hacky retrying logic to `get_logs_in_block`, since we were seeing lots of `503` http errors. In https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3264 we should make sure to remove this and propagate the error instead, to retry via the cursor / indexer logic. I edited the description of 3264 to reflect this ### Backward compatibility Yes ### Testing e2e --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	9 months ago
Daniel Savu	155c85eda2	Infallible relayer tasks (#3257 ) ### Description The intended behavior of agents is described by @tkporter's comment here: https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3257#discussion_r1489233847 ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3209 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	9 months ago
Daniel Savu	f2a0e92c7f	Cosmos grpc fallbackprovider (#3139 ) ### Description Implements grpc fallback provider logic for cosmos - initially tried implementing the fallback provider deprioritization logic at middleware level like in the EVM. The difference between ethers and cosmrs is that in the latter, middleware can only live at the transport layer (`tower` crate level). - based on this github [issue](https://github.com/hyperium/tonic/issues/733), that actually doesn't look possible, because the http::Request type isn't `Clone` so it can't be submitted to multiple providers - ended up implementing the fallback provider at the application layer, by keeping an array of grpc channels - There is now a `call` method in `hyperlane_core::FallbackProvider` which I'm actually really happy with. This method handles the fallbackprovider-specific logic by taking in an async closure, running it on each provider, and iterating providers if the closure call fails. In `grpc.rs` you can see how this is slightly verbose but I think it's quite manageable. The only part that bugs me is having to duplicate `Pin::from(Box::from(future))`, but that's need afaict because the regular closure returns an anonymous type - adds `grpcUrls` and `customGrpcUrls` config items - tests the cosmos fallback provider e2e ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues - Fixes: https://github.com/hyperlane-xyz/issues/issues/998 ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->	10 months ago
Daniel Savu	d18f7ae8e1	Cosmos gas price config (#3042 ) ### Description Adds a cosmos-specific config item for setting the minimum gas price, in the format returned by the `cosmos.base.node.v1beta1.Service/Config` cosmos-sdk grpc endpoint. ### Related issues - Fixes https://github.com/hyperlane-xyz/issues/issues/810 ### Backward compatibility No. Will break existing cosmos configs. ### Testing None yet, will test in e2e	11 months ago
Daniel Savu	c6cea785ae	V3 agents feedback (#2780 ) ### Description Part of the remediations from https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2746 --------- Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	1 year ago
Daniel Savu	af5bd88aa8	V3 agents rebase (#2746 ) ### Description It's your favourite PR coming right back... V3 agents! Closes https://github.com/hyperlane-xyz/issues/issues/561 Builds on top of https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2742 Depends on https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2681 for e2e testing This PR includes: - [x] Merkle tree hook indexer - [x] Merkle tree builder task - [x] Update submitter to trigger retries if no proof is available yet Slightly more detailed overview of the work here: https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2720#issuecomment-1724038643 <!-- What's included in this PR? --> ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests --> --------- Co-authored-by: -f <kunalarora1729@gmail.com> Co-authored-by: Trevor Porter <trkporter@ucdavis.edu> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Guillaume Bouvignies <guillaumebouvignies@gmail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: Guillaume Bouvignies <guillaume.bouvignies@kurtosistech.com>	1 year ago
Mattie Conover	17a6e79ea5	Rust agent config parsing (#2687 ) ### Description This does two things: 1) It parses the agent-specific configs for the relayer, validator, and scraper. 2) It removes the rigid structures that had been in place for parsing that were called out earlier and instead parses the serde json values directly. This PR does not cut over to the new versions just yet. That can happen once we start integrating it into the testing and deployment pipelines. ### Drive-by changes - Derive more was added and used where it made sense - The old config macro was removed ### Related issues - Progress on #2215 ### Backward compatibility Yes ### Testing None	1 year ago
Mattie Conover	644db5eee1	Remove unused tracing logic (#2639 ) ### Description When I was working on #2215 I noticed that we are not using either zipkin or jaeger at all and possibly never have. Ripping these out to reduce dead code and also simplify our package dependencies. ### Drive-by changes None ### Related issues Discovered through #2215 ### Backward compatibility Yes - we were not using these features and did not document their existence ### Testing Manual - Verified that when running deploy-agents, the generated configs do not have jaeger or zipkin enabled on `main` and then tested to make sure they still work in this branch	1 year ago
Mattie Conover	a5e1815746	Rust updates (#2619 ) ### Description Let this one slip a while. Updates our version of rust and as many packages as did not conflict with solana. I think we are going to need to figure something better out going forward with solana since if we have a couple more envs like this, it is going to make updates very hard. ### Drive-by changes None ### Related issues General Maintenance ### Backward compatibility Yes ### Testing Manual & Unit Tests	1 year ago
Trevor Porter	8c5983933b	Sealevel (#2404 ) ### Description Opening for that sweet sweet agent image ### Drive-by changes _Are there any minor or drive-by changes also included?_ ### Related issues - Fixes #[issue number here] ### Backward compatibility _Are these changes backward compatible?_ Yes No _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None Yes ### Testing _What kind of testing have these changes undergone?_ None Manual Unit Tests --------- Co-authored-by: Steven Sloboda <steven@eclipse.builders> Co-authored-by: Nam Chu Hoai <nambrot@googlemail.com> Co-authored-by: Yorke Rhodes <yorke@hyperlane.xyz> Co-authored-by: J M Rossy <jm.rossy@gmail.com> Co-authored-by: Mattie Conover <git@mconover.dev> Co-authored-by: Asa Oines <asaoines@gmail.com> Co-authored-by: Sergei Patrikeev <serejke.best@gmail.com> Co-authored-by: Daniel Savu <23065004+daniel-savu@users.noreply.github.com> Co-authored-by: Kunal Arora <55632507+aroralanuk@users.noreply.github.com> Co-authored-by: yorhodes <yorkerhodesiv@gmail.com> Co-authored-by: Yorke Rhodes <yorke@useabacus.network> Co-authored-by: Yorke Rhodes <email@yorke.dev> Co-authored-by: Asa Oines <asaoines@Asas-MacBook-Pro.local> Co-authored-by: Kunal Arora <kuarora@ucsd.edu> Co-authored-by: Anett <44020788+anettrolikova@users.noreply.github.com> Co-authored-by: Yaqub Mahmoud <yaqub320@gmail.com> Co-authored-by: Alex <alex@alexbh.dev>	1 year ago
Alex	da25b06254	CCIP Read ISM (#2398 ) ### Description CCIP Read ISM implementation, - [x] Contracts - [x] Relayer - [x] Example HTTP backend ### Drive-by changes _Are there any minor or drive-by changes also included?_ ### Related issues - Fixes #[2211] ### Backward compatibility _Are these changes backward compatible?_ Yes, relayers will skip messages they can't handle. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ Yes, relayers won't be able to submit messages that have a CCIP Read ISM. ### Testing _What kind of testing have these changes undergone?_ - [x] Manual - [x] Unit Tests ### General approach - [x] Get a basic [CCIP Read implementation working](https://github.com/AlexBHarley/permissionless-chainlink-feeds/tree/main/contracts) - [x] Build a [mock service that serves CCIP Read compatible data](https://github.com/AlexBHarley/permissionless-chainlink-feeds/tree/main/apps/api) - [x] Update the Relayer to support CCIP Read ISMs	1 year ago
Yorke Rhodes	50f04db1fa	Make merkle proofs optional on multisig ISM (#2173 ) ### Description Validators currently sign `(root, index)` checkpoints and during verification, a `message` is passed as calldata, an `id()` is derived, and a `proof` of `id()` at `index` in `root` is verified This provides “all or nothing” censorship resistance guarantees because a validator can only sign roots to allow any contained messages to be processed. We have considered alternatives where validators sign `message` directly and we lose censorship resistance in exchange for eliminating merkle proof verification gas costs. However, if validators sign `(root, index, message)` tuples, we can skip merkle proof verification on the destination chain while still maintaining censorship resistance by providing two valid metadata formats: 1. existing validator signatures and merkle proof verification of inclusion 2. including merkle proof verification for pathway where validators are censoring `message` It’s worth noting the validator is required to index event data to produce this new signature format. However, this does not require historical indexing and new validators being spun up can simply begin indexing from tip. See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2187 for validator changes See https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2248 for relayer and e2e test changes ### Drive-by changes Merkle index also optional ### Related issues - Fixes https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2192 ### Backward compatibility - new ISM deployment is necessary (we could upgrade implementation in theory) - Validator and relayer upgrades ### Testing Unit (fuzz) Tests, E2E tests	2 years ago
Mattie Conover	5e38cdf31a	Omniscient Relayer (#2195 ) ### Description Adds support for running a relayer which delivers messages from multiple origin chains instead of just one origin chain. ### Drive-by changes _Are there any minor or drive-by changes also included?_ - The E2E pipeline got a major re-work. It both runs faster and also tests the new changes. - SerialSubmitter has been abstracted to support additional operation types such as oracle updates - SerialSubmitter has been updated to actually confirm a transaction was included and is safe from re-orgs - SerialSubmitter has been updated to have higher throughput by pipelining operations ### Related issues - Blocked by #2148 - Fixes #1899 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	b56b022f88	Hyperlane-Core WASM support (#2110 ) ### Description This shuffles some code around and changes dependencies to make `hyperlane-core` WASM compatible and adds a check to the pipeline to ensure it stays that way. To do this it does refactor hyperlane db to the `base` crate, which makes more sense IMO anyway as that is an implementation detail of the agents and not part of the core protocol. ### Drive-by changes ### Related issues - Fixes #1977 ### Backward compatibility _Are these changes backward compatible?_ Yes-ish - It does change where things are but it does not change support. Should only impact people who directly use the libraries. _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Tests	2 years ago
Mattie Conover	1722f4ee7e	Better Agent Configuration Parsing (#2070 ) ### Description This is a significant change to how we parse user configs. The main goal of this PR is to give a clear path and env key to any errors that are encountered when parsing configuration. In order to accomplish this, I have crated a new trait `FromRawConf` which defines a set of steps to validate and convert individual fields. Also for every major config struct, there is now a `Raw` variant of it which is used by the `config` library to reduce errors that occur during deserialization itself. `Raw` types should always use the most basic form of a value, such as `String`, `StrOrInt`, `bool`, and ideally optional in all cases. Missing values should be handled during the raw conversion and not by `config`. I also needed to make changes to a number of types stored in the parsed config types to force validation forward to the parsing step instead of doing it when we read the config value. This also required me to create a filter to prevent trying to validate certain configs that we would not ever need to load. These changes can also be built on later to support something other than `config` if we choose to, or add support for merging configs from multiple sources since everything is optional. ### Drive-by changes - Default to `http` for connection type - Default to no gas payment enforcement if not specified instead of failing - `ChainSetup` -> `ChainConf` - `GasPaymentEnforcementConfig` -> `GasPaymentEnforcementConf` - Made ethereum connection configuration more forgiving - Moved hyperlane base settings from `mod.rs` to `base.rs` - Moved config chain tests in hyperlane core to `tests` dir to fix a cyclical import problem - Extension traits to help with config in hyperlane core - Moved `StrOrInt` to new hyperlane core `config` module - Support for parsing `U256` from `StrOrInt` - Removed `HexString` type which is now redundant - Updated base settings to use hyperlane domain - Use `heyKey` as signer type if `type` is not specified and `key` is - Moved hyperlane ethereum chain config to a new module ### Related issues - Fixes #2033 - Fixes #2012 ### Backward compatibility _Are these changes backward compatible?_ Yes - This should take in configs in the same shape but be more forgiving in a few places _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual Unit Test	2 years ago
Asa Oines	f25feecfa7	Scaffolding for multiple ISM types in the relayer (#2028 ) ### Description This PR adds scaffolding in the relayer to support building metadata for multiple ISM types. To add a new ISM type: - Add a new implementation of MetadataBuilder for that type - Add the type to the SupportedIsmTypes enum - Add the type to the match statement in BaseMetadataBuilder ### Drive-by changes None ### Related issues - #1757 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ E2E tests	2 years ago
Mattie Conover	4657fcc737	Better rust package management (#1975 ) ### Description At the suggestion of @tkporter this moves common dependencies and configuration to use the new(ish) cargo workspace configuration feature. This was added in July 2022 and allows us to ensure all versions and configurations are the same across different crates. It does not decrease the typing, but it does reduce the updates required when we go to update a package and keeps duplicates next to each other. This PR does not change any package versions or add/remove any packages. It only moves the configuration. ### Drive-by changes Also sorted packages by name. ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual	2 years ago
Mattie Conover	1c004f4b10	Remove gelato support (#1946 ) ### Description We have not been using Gelato anymore so thought it would be nice to remove this code and avoid continuing to maintain it. I know I have had to update it a few times to maintain parity. As an added perk it should slightly speed up build times since it removes an entire crate. ### Drive-by changes None ### Related issues ### Backward compatibility _Are these changes backward compatible?_ Yes - As we do not use this code _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Unit Tests	2 years ago
Nam Chu Hoai	97c7f52801	Remove GasPaymentEnforcementPolicy::MeetsEstimatedCost (#1910 ) ### Description As discussed offline, to establish the social contract of the relayer accepting the quotes of its associated IGP on the origin chain, this PR removes the GasPaymentEnforcementPolicy that was assessing gas payments at tx submission time via off-chain oracles.	2 years ago
Mattie Conover	836179c9ad	Better default stack traces (#1886 ) * Update to use color-eyre by default for stack traces * Use backtrace setting to define config formatting for debugging	2 years ago
Mattie Conover	03d32f4009	Eliminate trivial constructors (#1847 )	2 years ago
Mattie Conover	8105913dea	Improve agent logging (#1797 )	2 years ago
Mattie Conover	4b7e7ca9bb	Improve config errors (#1776 ) * Remove path to error as it was not working * Better domain parsing errors * Locked min version of config lib * Improve error messages for config loading	2 years ago
Mattie Conover	f6aa1f6039	Fallback provider (#1571 ) * Created fallback provider * support fallback provider * WIP testing * Fix testing * Use http provider so we can better inspect errors * Undo change * Fix comment	2 years ago
Mattie Conover	fcba422718	Better errors in core (#1404 ) * Better errors * Remove unused error types * fmt * Better error when object is not found from a provider	2 years ago
Mattie Conover	f814286998	Update packages (#1385 ) * Update ethers * Update packages * Fix build error	2 years ago
Mattie Conover	e8ed5f88ce	Deploy scraper for testnet3 (#1374 ) * How did those get there * abacus -> hyperlane * remove deprecated domains * Deleted dead code * Update sea-orm * Simplify settings post v2 changes * Split up settings * Update contexts * Fix scraper provider address regression during merge * Updated scraper deployment configs	2 years ago
Asa Oines	f15632f26c	Finalize rename from abacus->hyperlane (#1312 ) * Rename abacus->hyperlane in agents * Rename in typescript * Rename abacus->hyperlane in dockerfile	2 years ago
Trevor Porter	8f656d1f34	Update ethers-rs to include QuorumProvider fix for methods like eth_estimateGas (#1201 )	2 years ago
Mattie Conover	fe904f22cc	Runnable Scraper (#1140 ) * Fix metric ordering (#1043) * Remove redunant enums * WIP * block cursor and date time * add chrono * add mods * Work on scraper logic * Init plumbing and config parsing * Stub out inbox loading * Update metric type * Refactoring out weird opt struct * Minor refactoring * Sync logic for scraper outbox reading messages * Fix param names * Stubs and docs to lay out next steps * Add way to get a contract address * read latest message index and work on storing messages * Ported LogMeta and refactored CheckpointMeta out + moved RawCommittedMessage * Remove redundant CheckpointWithMeta struct * Update outbox indexer to include metadata when fetching messages * WIP storing messages * Work on processing blocks and txns * Work on processing blocks and txns * weave data through for message insertion * cleanup * Scraper config * Support multiple index setting configs * Fix panics loading config * Fix db generation * Fix loading wrong outbox from config * Removed deprecated testnets from scraper configs * Fix outbox loading using wrong outbox config * Refactoring out unused index config from ethereum outbox indexer * Fix `todo` panic * Fixed handling of block/txn fetching/storing * allow disabling scraper outboxes * Fix elusive unwrap error * Cleanup and fix `ensure_blocks_and_txns` * Logging that apparently does nothing rn * Refactoring to support initializing tracing before general agent initialization. * Simplify agent main fns * Include deprecated domains * regen entities * Update index names to be more consistent * Remove destination fkey for messages as it does not have to be a valid id * Better logging * remove constraint on block height for the moment * Fix reverse id issue * Fix initial cursor insertion * fix test * appease clippy for now * Get scraper bin in Dockerfile too * append only cursors * append only cursor fix * Fix handling of empty insertion * Reduce query load * Scraper config for mainnet * Remove metrics fn from agent definition * Remove unused checkpoint fn * fix idx name * Order by height instead of id for cursors * Don't fetch message leaf index for every tick * Better var name * Add comment * Removed dead code stub * Better docs * Updated param name * Comment for pr * Added comment * Added comment * Removed unused code * Add doc on domains * Add comment. Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	2 years ago
Trevor Porter	7040e5c8a4	Gas enforcement policy requiring payment to meet estimated costs (#1083 ) * All compiles * Rename files fwd request -> sponsored call * More scaffolding to get the API key in there * Cleaning up * Rename task_status_call to task_status * Finish rename * rm useForDisabledOriginChains * Introduce TransactionSubmitterType * submitter type -> submission type * Pass around gelato_config instead of the sponsor api key * Final cleanup * Nit * Use default tx submission type * Wip * wip * Getting there * Getting there * Refactor of serial submitter * Some renames * cargo fmt * Some testing * Nits after some testing * A bit of a refactor of policies * Adding some chain enums * Single source of truth for abacus domain IDs * Add local test chains * Get coingecko api key via external-secrets * Move some things around, rm gelato oracle api interaction * Gonna move to abacus-core * Move chain stuff to abacus-core * nit * comment nit * Deploy tooling * MeetsEstimatedCost policy test * PR comments, still need to merge domain enums * Single AbacusDomain enum	2 years ago
Mattie Conover	c37aa61b91	Update ethers (#1058 ) * Update run-locally to test quorum provider better * Update ethers-rs	2 years ago
Mattie Conover	1a7e74ea7c	Update ethers (#1037 )	2 years ago
Trevor Porter	000b67cf16	Upgrade ethers-rs to fix Optimsim gas estimation (#1023 )	2 years ago
Mattie Conover	d316ae68d5	update ethers-rs (#1004 )	2 years ago
Mattie Conover	333b18ecfe	Quorum provider (#934 ) * Add quorum provider support * Update Rust * Pull Http logic out of retrying provider * first pass at retrying provider for quorum provider * change weight parsing * Better logging and handling for quorum retry logic * Better logging * Appease new clippy rules * Update packages in vein attempt to make things work * quorum provider testing * switch to our version of ethers-rs * Update config * Support quorum provider * Use quorum provider for testing * Update packages * Update	2 years ago
Trevor Porter	0691b7b326	Gelato relayer scaffolding (#950 ) * Revive fwd_req_op. * Transform iter map into a for loop in relayer.rs. * cargo fmt * Plumb signer into GelatoSubmitter. * Provide inbox signer to GelatoSubmitter init. This might fail, in which case the relayer can't really do anything for that inbox if it is configured to use Gelato. In that case, bail!(..). Bailing requires that the boot/init boilerplate in the relayer handle errors, so plumb/wrap those all in Result<> return types and update accordingly. * Plumb deps up to ForwardRequestArgs construction. * Plumb actual target chain, and rename chain --> target_chain. * Use 0xEE for native token always. * Plumb outbox chain id * Plumb address of IVM on target chain * Just use zero bytes instead of todo! to satisfy clippy etc. * Plumb IVM BaseContract into GelatoSubmitter so that we can use it to encode calldata for IVM.process(). * cargo fmt again * How to marshal bytes. * Add interface for accessing inbox contract eth address * Plumb inbox address into the gelato submitter finally. * cargo fmt * Actually return the address instead of todo!()... * Actually build the byte array of calldata! * Better error msg to fit < 100c * defer introducing fwd_req_op until next PR * Handful of pre-review cleanups: - Only give GelatoSubmitter the inbox domain instead of the whole InboxContracts - Don't wrap http: reqwest::Client in an Arc. It's unnecessary and handled internally for you, and better to clone() so that cxn pool reused. - Shorter names from imports where noticed - stub op.run() impl - shorter submitter field names where appropriate - dont pass GelatoConf to GelatoSubmitter - get correct visibility in abigen'd build code (and allow no doc comments to satisfy clippy) * rename data to call_data * cargo fmt * Defer field name change for chain_id and target in FwdReqArgs * Cleanup of relayer.rs including more clear data validation and imoprt scoping. * cargo fmt * Improve (?) address handling for inbox and ivm. * for InboxValidatorManagerVariants::contract_address(), actually delegate to the Mock variant implementation's contract_address() call instead of over-eagerly returning None. * Stub impl of _contract_address in mockall mock! Inbox contract impl. * for Other variants, match arm does unimplemented!(), so we avoid having to wrap in an Option, making cleaner at call sites * Minimize namespace qualifications where possible for Address-like types * attempt to make the IVM contract_address() for abacus-ethereum impl parallel to inbox w/r/t types * Move abacus domain conversion out of gelato::Chain and into private helper in gelato_submitter * More logical ordering of GelatoSubmitter struct fields * cargo fmt * Drop extra comment, clarify log * Plumb sponsor address from configs into GelatoSubmitter. Interpreted as a source chain address. * cargo fmt --all * stop trying to wrap http client in an arc, it's counterproductive * Write Ok once instead of every line * formatting * Do not separate with empty line the inner attributes found in lib.rs post-abigen-buildscript-exec Otherwise cargo fmt finds a diff and presubmits halt. * spelling typo * Change GelatoSubmitter construction from new to helper in relayer.rs. Also use simpler construction for tokenizing the proof for merkle trees when generating the forward request args struct. Was needlessly convoluted before using default trait unnecessarily etc. Use writeln instead of write in abigen util since clippy complains otherwise. * Get ivm_base_contract from ABI string directly rather than plumbing. * pass GelatoConf by as_ref() which refs inner type rather than &Option<GelatoConf> * Drop unrelated comment in gelato/chains.rs, noted elsewhere * Revive fwd_req_op. * Transform iter map into a for loop in relayer.rs. * cargo fmt * Plumb signer into GelatoSubmitter. * Provide inbox signer to GelatoSubmitter init. This might fail, in which case the relayer can't really do anything for that inbox if it is configured to use Gelato. In that case, bail!(..). Bailing requires that the boot/init boilerplate in the relayer handle errors, so plumb/wrap those all in Result<> return types and update accordingly. * Plumb deps up to ForwardRequestArgs construction. * Plumb actual target chain, and rename chain --> target_chain. * Use 0xEE for native token always. * Plumb outbox chain id * Plumb address of IVM on target chain * Just use zero bytes instead of todo! to satisfy clippy etc. * Plumb IVM BaseContract into GelatoSubmitter so that we can use it to encode calldata for IVM.process(). * cargo fmt again * How to marshal bytes. * Add interface for accessing inbox contract eth address * Plumb inbox address into the gelato submitter finally. * cargo fmt * Actually return the address instead of todo!()... * Actually build the byte array of calldata! * Better error msg to fit < 100c * defer introducing fwd_req_op until next PR * Handful of pre-review cleanups: - Only give GelatoSubmitter the inbox domain instead of the whole InboxContracts - Don't wrap http: reqwest::Client in an Arc. It's unnecessary and handled internally for you, and better to clone() so that cxn pool reused. - Shorter names from imports where noticed - stub op.run() impl - shorter submitter field names where appropriate - dont pass GelatoConf to GelatoSubmitter - get correct visibility in abigen'd build code (and allow no doc comments to satisfy clippy) * rename data to call_data * cargo fmt * Defer field name change for chain_id and target in FwdReqArgs * Cleanup of relayer.rs including more clear data validation and imoprt scoping. * cargo fmt * Improve (?) address handling for inbox and ivm. * for InboxValidatorManagerVariants::contract_address(), actually delegate to the Mock variant implementation's contract_address() call instead of over-eagerly returning None. * Stub impl of _contract_address in mockall mock! Inbox contract impl. * for Other variants, match arm does unimplemented!(), so we avoid having to wrap in an Option, making cleaner at call sites * Minimize namespace qualifications where possible for Address-like types * attempt to make the IVM contract_address() for abacus-ethereum impl parallel to inbox w/r/t types * Move abacus domain conversion out of gelato::Chain and into private helper in gelato_submitter * More logical ordering of GelatoSubmitter struct fields * cargo fmt * Drop extra comment, clarify log * Plumb sponsor address from configs into GelatoSubmitter. Interpreted as a source chain address. * cargo fmt --all * stop trying to wrap http client in an arc, it's counterproductive * Write Ok once instead of every line * formatting * Do not separate with empty line the inner attributes found in lib.rs post-abigen-buildscript-exec Otherwise cargo fmt finds a diff and presubmits halt. * spelling typo * Change GelatoSubmitter construction from new to helper in relayer.rs. Also use simpler construction for tokenizing the proof for merkle trees when generating the forward request args struct. Was needlessly convoluted before using default trait unnecessarily etc. Use writeln instead of write in abigen util since clippy complains otherwise. * Get ivm_base_contract from ABI string directly rather than plumbing. * pass GelatoConf by as_ref() which refs inner type rather than &Option<GelatoConf> * Drop unrelated comment in gelato/chains.rs, noted elsewhere * Wrap Inbox::contract_address() result in an Option. * Don't over-zealously mark pub(crate) struct's members pub(crate). pub is good enough. * More natural iter map construction in Token byte packing for call_data on process. * More natural iter map construction in Token byte packing for call_data on process. * Restore fwd req op struct and fmt * rm unnecessary Results * add with_capacity * Make contract_address return Address, not Option<Address> * rm unimplemeneted * Rm address field of validator manager * Mid refactor of creating GelatoSubmitter * add inbox_gelato_chain * cleaning * Some shuffling around * some moving around * Finishing touches * Remove unnecessary type definition & some unused code Co-authored-by: webbhorn <webb.horn@gmail.com>	2 years ago
webbhorn	dd63f70da1	Gelato Plumbing (#637 ) This PR prepares for the introduction of Gelato ForwardRequest API usage. However, it only refactors current code and introduces a prerequisite interface. In particular, the existing MessageProcessor was previously responsible both for message discovery via AbacusDB, as well as managing the submission of that message to the destination chain via some RPC provider. Since we want to have the option to deliver those messages over Gelato, a natural place to split things up might be between the AbacusDB message discovery phase, and the message delivery phase. This CL formalizes that notion by moving the submission logic into a SerialSubmitter submodule. The MessageProcessor and SerialSubmitter only communicate via a channel now. A subsequent PR will introduce GelatoSubmitter, which will route messages from the MessageProcessor to the Gelato API according to the new configuration options introduced in this PR.	2 years ago
Mattie Conover	de20710466	Rust updates (#644 ) * Update config and trim down macro code * Update rust * Package updates * Updates and cleanup * make clippy happy	2 years ago
Mattie Conover	4d9603c868	Relayer whitelist (#570 ) * Update ABIs, compiles. Removed Inbox indexer and cached checkpoints from the Inbox * Rm checkpoint indexing test * Update InboxValidatorManager, just need to connect the pieces now * Add channels * So close * Fix InboxValidatorManager deploy * Make message processor slightly more readable * Create loop-control crate * Try to bring some sanity to message processor * Fix bug where prover_sync wasn't in line with latest_signed_checkpoint * Rm immediate message processing, clean up * rm abacus-cli * more cleanup * rename / rm some settings * TS renames / rms * Lower some processing failure logs to info * Checkpoint fetcher doesn't need the CommittedMessages * Hardcode kathy dispatching * Move to watch channel * I'm sorry clippy * nits * rm some nonce related stuff * more refactoring * cleanup * Add whitelist settings * Whitelist filtering * fix docker build * use relayed terminology * Change whitelist tuple order * naming change * move whitelist check to before fetching proof * Update span event * make messages singular * Update error * minor cleanup * Update configs in scripts * Whitelist testing and parsing fixes Co-authored-by: Trevor Porter <trkporter@ucdavis.edu>	2 years ago
Mattie Conover	d687d3e230	More Metrics (#495 ) * metrics refactoring * metrics refactoring * Make brain hurt less by using a damn function * patch through metrics to abacus_ethereum inbox and outbox * WIP * Minor cleanup * work on time update * added label arrays * refactoring * Metrics for sending txns * Metrics middleware * cleanup * Cleanup after merge * Add metrics config and setup prom metrics with core metrics * Minor fixes to docs * Better trait building * rename * Weave metrics through code * Use core metrics instead * Removed some unused code * fmt * change feature layout * fix docker build * Start scraping metrics on an interval * fmt * fix duplicate label error * change features * fix u256 conversion fn * refactor * Support default values in config * Minor fixes * Use f64 conversion from primitive-types * remove unused code * fix const name * improve docs * better var name * remove unused code * remove unused code	3 years ago
webbhorn	6b3822f664	Misc. agent cleanup and dead code deletion (#470 ) * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in comment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool. * Fix path to agent::Agent in comment. Otherwise `cargo doc` emits a rustdoc::broken_intra_doc_links warning. * Use automatic link in coment for rustdoc. Otherwise 'cargo doc' emits a warning. * Delete checkpointer agent. It is not used in current deployment. * Delete kms-cli. Not used in current deployment (no references). * Include "--all" in precommit cargo fmt -- --check invocation. * Delete unused and hidden src/bin programs. These appear to have been introduced over a year ago and appear to be dead code now. Unhide them from vscode workspace too -- since they were hidden, automated refactoring tools wouldn't find them until compiler failed. * Update github CODEOWNERS to current Abacus team. Drop optional authors field from Cargo.tomls. * Drop processordb from .gitignore. Processor crate has been deleted. * Delete balance-exporter crate. It is not used / no references. * Hoist abacus-cli into its own top-level crate. It is the only tool now, no reason to hide it inside of a tools directory. * Delete old relaese.sh, which only relates to Celo Optics. * Cleanup top-level rust README.md. - Architecture paragraphs outdated, delete. - Point to run-locally.sh instead of documentation. * Explain a bit more about crate deps in README. * Dockerfile fix after tools directory restructuring. We had previously hoisted abacus-cli out since it was the only remaining tool.	3 years ago
Mattie Conover	f78e79ab9b	Better Agent Error Logging (#412 ) * Make errors print on one line * Replace color-eyre with eyre in core and base * Replace color-eyre with eyre other libs * Log should be imported without patch version * Update tracing and opentelemetry * Only need openssl dynamic libs * Fix no-default-feature build * Custom eyre handler * Removed unused log dep * Cleanup paste dep * More package cleanup * Update tokio * Minor refactor	3 years ago
Nam Chu Hoai	d4c8720700	Rename rust packages (#128 ) * Rename rust packages * rustfmt * More rename * Rename update_abis * Rename metrics.rs	3 years ago
Luke Tchang	f007a299c8	feature: add metrics/tracking to remaining agents (#922 ) * feature: add num relayed messages metrics/tracking to relayer * refactor: remove processor-specific nonce storage method from optics db * fix: use countervec for relayer instead of guage * feature: updater uses intcounter vec for submitted update count * fix: remove db from relayer since it now uses countervec with no rocksdb * fix: updates relayed not messages oops * fix: update submitter takes the correct countervec	3 years ago
Conner Swann	7b2c4a21df	refactor: split updater tasks for safety (#909 ) * refactor: split updater tasks, refactor opticsdb * feature: asrefs for db types * refactor: refactor typeddb to hold entity and pair with all loads/stores * ci: cargo +nightly * ci: use nightly toolchain * fix: use 1.56 stable not nightly * fix: updater can never overwrite an existing update with a conflicting one * fix: also check suggested update for state change guard Co-authored-by: Luke Tchang <ltchang@stanford.edu>	3 years ago
ember arlynx	5f041c9db1	drive-by cleanups (#592 ) * reorganize rust code into dirs * rust: ignore vendor dir * how did kathy go missing? * drive-by cleanups: abstraction in update_abis, hyperlink, shell scripts are mostly shellcheck clean * rust readme: note agents dir * cargo fmt --all * remove stray comment * cargo update * why am i fighting my computer rn * patch over sloppy rebase * shore up test vector file handling * clarify how to add a new chain * add rust-toolchain.toml for 1.54 * DRY abis_dir * chore: add clippy installation to CI * github actions will use rust-toolchain.yaml now * move rust-toolchain so the action will find it rustup seems to scan upwards for this file, should pick it up fine when using from within * use obsolete rust-toolchain file location, because the action is naive * chore: specify toolchain in rust.yml * chore: specify toolchain in rust.yml in all workflows * chore: return clippy to rust.yml Co-authored-by: ember arlynx <ember@lunar.software> Co-authored-by: James Prestwich <james@prestwi.ch>	3 years ago
James Prestwich	a38f4f5153	chore: bump ethers to main in all cargo.toml files	3 years ago
James Prestwich	aff52ce2a9	bug: tracing and pending txn fixes - use branch of ethers-rs with pendingtx poll delay - use tracing::Instrumented in most agent trait methods - add in_current_span to most spawns - add an id generator to jaeger tracing	3 years ago
James Prestwich	a37a6ed776	Prestwich/rust config (#419 ) * improve configuration handling of env vars - improve decl_settings macro to name the declared structs - change domains and timers to String - rename chat_gen to chat in kathy settings - panic on failed uint parsing in settings deser * bug: update typescript deploy to output string domains * chore: update new deploy conf to match new format * chore: set kathy destination to 3000 for now * refactor: ensure kathy dispatches messages to each comnfigured replica	3 years ago

47 Commits (258bf85e4336098386b40f69992ca1e8629791a6)