drive-by cleanups (#592)

* reorganize rust code into dirs

* rust: ignore vendor dir

* how did kathy go missing?

* drive-by cleanups:

abstraction in update_abis, hyperlink, shell scripts are mostly shellcheck clean

* rust readme: note agents dir

* cargo fmt --all

* remove stray comment

* cargo update

* why am i fighting my computer rn

* patch over sloppy rebase

* shore up test vector file handling

* clarify how to add a new chain

* add rust-toolchain.toml for 1.54

* DRY abis_dir

* chore: add clippy installation to CI

* github actions will use rust-toolchain.yaml now

* move rust-toolchain so the action will find it

rustup seems to scan upwards for this file, should pick it up fine when using from
within

* use obsolete rust-toolchain file location, because the action is naive

* chore: specify toolchain in rust.yml

* chore: specify toolchain in rust.yml in all workflows

* chore: return clippy to rust.yml

Co-authored-by: ember arlynx <ember@lunar.software>
Co-authored-by: James Prestwich <james@prestwi.ch>

.github/workflows/rust.yml

@@ -18,6 +18,9 @@ jobs:
 
     steps:
       - uses: actions/checkout@v2
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: 1.54.0
       - uses: Swatinem/rust-cache@v1
         with:
           working-directory: ./rust
@@ -30,6 +33,9 @@ jobs:
 
     steps:
       - uses: actions/checkout@v2
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: 1.54.0
       - uses: Swatinem/rust-cache@v1
         with:
           working-directory: ./rust
@@ -42,14 +48,16 @@ jobs:
 
     steps:
       - uses: actions/checkout@v2
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: 1.54.0
       - uses: Swatinem/rust-cache@v1
         with:
           working-directory: ./rust
-
+      - name: Rustfmt
+        run: cd rust && cargo fmt -- --check
       - name: Clippy
         run: cd rust && cargo clippy -- -D warnings
-      - name: Lint
-        run: cd rust && cargo fmt -- --check
 
   complete:
     runs-on: ubuntu-latest

.gitignore

@@ -3,6 +3,7 @@ test_deploy.env
 
 typescript/optics-deploy/.env
 
+rust/vendor/
 rust/tmp_db
 rust/tmp.env
 tmp.env

README.md

@@ -129,8 +129,7 @@ Note: In the event you need to bypass the pre-commit hooks, pass the
 
 ### Rust
 
-- install `rustup`
-  - [link here](https://rustup.rs/)
+- install [`rustup`](https://rustup.rs)
 - see `rust/README.md`
 
 #### Building Agent Images

pre-commit.sh

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 
 # Specifying minimum versions
 MIN_GIT=2.25.0
@@ -18,7 +18,7 @@ echo_abort ()
 
 ensure()
 {
-    if ! command -v $1 &> /dev/null
+    if ! command -v "$1" &> /dev/null
     then
         echo_abort
         echo "command '$1' could not be found."
@@ -79,7 +79,7 @@ if ! git diff-index --quiet HEAD -- ./solidity/optics-core; then
     npm run compile
     # add abis, typechain
     cd ../..
-    git add rust/optics-ethereum/abis
+    git add rust/chains/optics-ethereum/abis
     git add typescript/typechain
     # add linter modified files
     git add solidity/optics-core/contracts
@@ -153,9 +153,10 @@ else
 fi
 
 # Git add abis if updated
-if ! git diff-index --quiet HEAD -- ./rust/optics-ethereum/abis/; then
-    echo '+git add ./rust/optics-ethereum/abis/*'
-    git add ./rust/optics-ethereum/abis/*
+abis_dir=./rust/chains/optics-ethereum/abis/
+if ! git diff-index --quiet HEAD -- $abis_dir; then
+    echo "+git add $abis_dir\*"
+    git add $abis_dir/*
 else
     echo "+Skipping git add ABIs"
 fi

rust-toolchain

@@ -0,0 +1,3 @@
+[toolchain]
+channel = "1.54"
+profile = "default"

rust/Cargo.lock

@@ -1885,9 +1885,9 @@ checksum = "624a8340c38c1b80fd549087862da4ba43e08858af025b236e509b6649fc13d5"
 
 [[package]]
 name = "openssl"
-version = "0.10.35"
+version = "0.10.36"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "549430950c79ae24e6d02e0b7404534ecf311d94cc9f861e9e4020187d13d885"
+checksum = "8d9facdb76fec0b73c406f125d44d86fdad818d66fef0531eec9233ca425ff4a"
 dependencies = [
  "bitflags",
  "cfg-if",
@@ -1905,9 +1905,9 @@ checksum = "28988d872ab76095a6e6ac88d99b54fd267702734fd7ffe610ca27f533ddb95a"
 
 [[package]]
 name = "openssl-sys"
-version = "0.9.65"
+version = "0.9.66"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7a7907e3bfa08bb85105209cdfcb6c63d109f8f6c1ed6ca318fff5c1853fbc1d"
+checksum = "1996d2d305e561b70d1ee0c53f1542833f4e1ac6ce9a6708b6ff2738ca67dc82"
 dependencies = [
  "autocfg",
  "cc",

rust/Cargo.toml

@@ -3,12 +3,12 @@
 members = [
     "optics-core",
     "optics-base",
-    "optics-ethereum",
     "optics-test",
-    "kathy",
-    "updater",
-    "relayer",
-    "watcher",
-    "processor",
-    "kms-cli"
+    "chains/optics-ethereum",
+    "agents/kathy",
+    "agents/updater",
+    "agents/relayer",
+    "agents/watcher",
+    "agents/processor",
+    "kms-cli",
 ]

rust/Dockerfile

@@ -10,16 +10,13 @@ RUN apt-get update && \
     rustup target add x86_64-unknown-linux-musl 
 
 # Add workspace to workdir 
-COPY optics-core ./optics-core
-COPY optics-ethereum ./optics-ethereum
+COPY agents ./agents
+COPY chains ./chains
+COPY kms-cli ./kms-cli
 COPY optics-base ./optics-base 
+COPY optics-core ./optics-core
 COPY optics-test ./optics-test
-COPY kms-cli ./kms-cli
-COPY processor ./processor
-COPY relayer ./relayer
-COPY watcher ./watcher
-COPY updater ./updater
-COPY kathy ./kathy
+
 COPY Cargo.toml .
 COPY Cargo.lock .
 

rust/README.md

@@ -98,9 +98,10 @@ We use the tokio async runtime environment. Please see the docs
     - trait implementations for different chains
     - shared configuration file formats
     - basic setup for an off-chain agent
-- `optics-ethereum`
+- `chains/optics-ethereum`
   - interfaces to the ethereum contracts
-
+- `agents`
+  - each of the off-chain agents implemented thus far
 ### High-level guide to building an agent
 
 - `cargo new $AGENT_NAME`

rust/agents/kathy/Cargo.toml

@@ -22,6 +22,6 @@ tracing-futures = "0.2.4"
 tracing-subscriber = "0.2.15"
 rand = "0.8.3"
 
-optics-base = {path = "../optics-base"}
-optics-core = {path = "../optics-core"}
+optics-base = {path = "../../optics-base"}
+optics-core = {path = "../../optics-core"}
 paste = "1.0.5"

rust/agents/kathy/src/main.rs

@@ -16,6 +16,7 @@ use crate::{kathy::Kathy, settings::KathySettings as Settings};
 async fn _main() -> Result<()> {
     color_eyre::install()?;
     let settings = Settings::new()?;
+    // there needs to be an HTTP server here??? for health agent endpoint stuff.
     settings.base.tracing.start_tracing()?;
 
     let agent = Kathy::from_settings(settings).await?;

rust/agents/processor/Cargo.toml

@@ -20,10 +20,10 @@ tracing-futures = "0.2.4"
 tracing-subscriber = "0.2.15"
 rocksdb = { git = "https://github.com/rust-rocksdb/rust-rocksdb" }
 
-optics-core = { path = "../optics-core" }
-optics-base = { path = "../optics-base" }
+optics-core = { path = "../../optics-core" }
+optics-base = { path = "../../optics-base" }
 paste = "1.0.5"
 
 [dev-dependencies]
-optics-test = { path = "../optics-test" }
+optics-test = { path = "../../optics-test" }
 

rust/agents/processor/src/main.rs

@@ -10,6 +10,7 @@
 
 mod processor;
 mod prover;
+mod prover_sync;
 mod settings;
 
 use color_eyre::Result;

rust/agents/processor/src/processor.rs

@@ -18,6 +18,7 @@ use tokio::{
 };
 use tracing::{error, info, instrument, instrument::Instrumented, Instrument};
 
+use crate::{prover::Prover, prover_sync::ProverSync, settings::ProcessorSettings as Settings};
 use optics_base::{
     agent::{AgentCore, OpticsAgent},
     cancel_task, decl_agent,
@@ -31,11 +32,6 @@ use optics_core::{
     traits::{CommittedMessage, Common, Home, MessageStatus, Replica},
 };
 
-use crate::{
-    prover::{Prover, ProverSync},
-    settings::ProcessorSettings as Settings,
-};
-
 #[derive(Debug)]
 pub(crate) struct ReplicaProcessor {
     interval: u64,

rust/agents/processor/src/prover.rs

@@ -1,6 +1,6 @@
-/// Struct responsible for syncing Prover
-pub mod prover_sync;
-pub use prover_sync::ProverSync;
+//! Prover process: generate proofs in the tree.
+//!
+//! Struct responsible for syncing Prover
 
 use ethers::core::types::H256;
 use rocksdb::DB;

rust/agents/relayer/Cargo.toml

@@ -19,10 +19,10 @@ tracing = "0.1.22"
 tracing-futures = "0.2.4"
 tracing-subscriber = "0.2.15"
 
-optics-core = { path = "../optics-core" }
-optics-base = { path = "../optics-base" }
+optics-core = { path = "../../optics-core" }
+optics-base = { path = "../../optics-base" }
 paste = "1.0.5"
 
 [dev-dependencies]
 tokio-test = "0.4.0"
-optics-test = { path = "../optics-test" }
+optics-test = { path = "../../optics-test" }

rust/agents/updater/Cargo.toml

@@ -20,11 +20,11 @@ tracing-futures = "0.2.4"
 tracing-subscriber = "0.2.15"
 rocksdb = { git = "https://github.com/rust-rocksdb/rust-rocksdb" }
 
-optics-core = { path = "../optics-core" }
-optics-base = { path = "../optics-base" }
-optics-ethereum = { path = "../optics-ethereum" }
+optics-core = { path = "../../optics-core" }
+optics-base = { path = "../../optics-base" }
+optics-ethereum = { path = "../../chains/optics-ethereum" }
 paste = "1.0.5"
 
 [dev-dependencies]
 mockall = "0.9.1"
-optics-test = { path = "../optics-test" }
+optics-test = { path = "../../optics-test" }

rust/agents/watcher/Cargo.toml

@@ -20,11 +20,11 @@ tracing-futures = "0.2.4"
 tracing-subscriber = "0.2.15"
 rocksdb = { git = "https://github.com/rust-rocksdb/rust-rocksdb" }
 
-optics-core = { path = "../optics-core" }
-optics-base = { path = "../optics-base" }
-optics-ethereum = { path = "../optics-ethereum" }
+optics-core = { path = "../../optics-core" }
+optics-base = { path = "../../optics-base" }
+optics-ethereum = { path = "../../chains/optics-ethereum" }
 paste = "1.0.5"
 
 [dev-dependencies]
 tokio-test = "0.4.0"
-optics-test = { path = "../optics-test" }
+optics-test = { path = "../../optics-test" }

rust/chains/README.md

@@ -0,0 +1,4 @@
+To add a new chain:
+
+    - add a new crate to this directory, implementing the appropriate traits.
+    - add a new enum variant in `optics_base::homes::Homes` using the new crate and fix all the compilation errors.

rust/chains/optics-ethereum/Cargo.toml

@@ -17,6 +17,6 @@ async-trait = { version = "0.1.42", default-features = false }
 tracing = "0.1.22"
 color-eyre = "0.5.0"
 
-optics-core = { path = "../optics-core" }
+optics-core = { path = "../../optics-core" }
 tokio = "1.7.1"
 hex = "0.4.3"

rust/chains/optics-ethereum/src/home.rs

@@ -15,7 +15,10 @@ use std::{convert::TryFrom, error::Error as StdError, sync::Arc};
 use crate::report_tx;
 
 #[allow(missing_docs)]
-abigen!(EthereumHomeInternal, "./optics-ethereum/abis/Home.abi.json");
+abigen!(
+    EthereumHomeInternal,
+    "./chains/optics-ethereum/abis/Home.abi.json"
+);
 
 /// A reference to a Home contract on some Ethereum chain
 #[derive(Debug)]

rust/chains/optics-ethereum/src/replica.rs

@@ -17,7 +17,7 @@ use crate::report_tx;
 #[allow(missing_docs)]
 abigen!(
     EthereumReplicaInternal,
-    "./optics-ethereum/abis/Replica.abi.json",
+    "./chains/optics-ethereum/abis/Replica.abi.json",
      methods {
         initialize(address) as initialize_common;
         initialize(uint32, address, bytes32, uint256, uint32) as initialize;

rust/chains/optics-ethereum/src/xapp.rs

@@ -14,7 +14,7 @@ use crate::report_tx;
 #[allow(missing_docs)]
 abigen!(
     EthereumConnectionManagerInternal,
-    "./optics-ethereum/abis/XAppConnectionManager.abi.json"
+    "./chains/optics-ethereum/abis/XAppConnectionManager.abi.json"
 );
 
 /// A reference to a XAppConnectionManager contract on some Ethereum chain

rust/kms-cli/Cargo.toml

@@ -6,7 +6,7 @@ edition = "2018"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
-clap = "3.0.0-beta.2"
+clap = "3.0.0-beta.4"
 color-eyre = "0.5.11"
 ethers = { git = "https://github.com/gakonst/ethers-rs", branch = "master" }
 ethers-signers = { git = "https://github.com/gakonst/ethers-rs", branch = "master", features = ["aws"] }
@@ -16,4 +16,3 @@ rusoto_core = "0.47.0"
 rusoto_kms = "0.47.0"
 tokio = "1.9.0"
 serde_json = "1.0.66"
-

rust/optics-base/Cargo.toml

@@ -22,7 +22,7 @@ tracing-subscriber = "0.2.15"
 rocksdb = { git = "https://github.com/rust-rocksdb/rust-rocksdb" }
 
 optics-core = { path = "../optics-core" }
-optics-ethereum = { path = "../optics-ethereum"}
+optics-ethereum = { path = "../chains/optics-ethereum"}
 optics-test = { path = "../optics-test" }
 paste = "1.0.5"
 tracing-error = "0.1.2"

rust/optics-core/src/test_output.rs

@@ -3,6 +3,7 @@ use crate::{
         merkle::{merkle_root_from_branch, MerkleTree},
         TREE_DEPTH,
     },
+    test_utils::find_vector,
     utils::{destination_and_sequence, home_domain_hash},
     FailureNotification, OpticsMessage, Update,
 };
@@ -52,7 +53,7 @@ pub mod output_functions {
             .write(true)
             .create(true)
             .truncate(true)
-            .open("../../vectors/message.json")
+            .open(find_vector("message.json"))
             .expect("Failed to open/create file");
 
         file.write_all(json.as_bytes())
@@ -89,7 +90,7 @@ pub mod output_functions {
             .write(true)
             .create(true)
             .truncate(true)
-            .open("../../vectors/proof.json")
+            .open(find_vector("proof.json"))
             .expect("Failed to open/create file");
 
         file.write_all(json.as_bytes())
@@ -113,7 +114,7 @@ pub mod output_functions {
             .write(true)
             .create(true)
             .truncate(true)
-            .open("../../vectors/homeDomainHash.json")
+            .open(find_vector("homeDomainHash.json"))
             .expect("Failed to open/create file");
 
         file.write_all(json.as_bytes())
@@ -139,7 +140,7 @@ pub mod output_functions {
             .write(true)
             .create(true)
             .truncate(true)
-            .open("../../vectors/destinationSequence.json")
+            .open(find_vector("destinationSequence.json"))
             .expect("Failed to open/create file");
 
         file.write_all(json.as_bytes())
@@ -182,7 +183,7 @@ pub mod output_functions {
                 .write(true)
                 .create(true)
                 .truncate(true)
-                .open("../../vectors/signedUpdate.json")
+                .open(find_vector("signedUpdate.json"))
                 .expect("Failed to open/create file");
 
             file.write_all(json.as_bytes())
@@ -232,7 +233,7 @@ pub mod output_functions {
                 .write(true)
                 .create(true)
                 .truncate(true)
-                .open("../../vectors/signedFailure.json")
+                .open(find_vector("signedFailure.json"))
                 .expect("Failed to open/create file");
 
             file.write_all(json.as_bytes())

rust/optics-core/src/test_utils.rs

@@ -1,6 +1,6 @@
 use crate::accumulator::merkle::Proof;
 use ethers::core::types::H256;
-use std::{fs::File, io::Read};
+use std::{fs::File, io::Read, path::PathBuf};
 
 /// Struct representing a single merkle test case
 #[derive(serde::Deserialize, serde::Serialize)]
@@ -16,9 +16,21 @@ pub struct MerkleTestCase {
     pub expected_root: H256,
 }
 
+/// Find a vector file assuming that a git checkout exists
+// TODO: look instead for the workspace `Cargo.toml`? use a cargo env var?
+pub fn find_vector(final_component: &str) -> PathBuf {
+    let cwd = std::env::current_dir().expect("no cwd?");
+    let git_dir = cwd
+        .ancestors() // . ; ../ ; ../../ ; ...
+        .find(|d| d.join(".git").is_dir())
+        .expect("could not find .git somewhere! confused about workspace layout");
+
+    git_dir.join("vectors").join(final_component)
+}
+
 /// Reads merkle test case json file and returns a vector of `MerkleTestCase`s
 pub fn load_merkle_test_json() -> Vec<MerkleTestCase> {
-    let mut file = File::open("../../vectors/merkle.json").unwrap();
+    let mut file = File::open(find_vector("merkle.json")).unwrap();
     let mut data = String::new();
     file.read_to_string(&mut data).unwrap();
     serde_json::from_str(&data).unwrap()

rust/optics-test/Cargo.toml

@@ -19,4 +19,4 @@ rand = "0.8.3"
 rocksdb = { git = "https://github.com/rust-rocksdb/rust-rocksdb" }
 
 optics-core = { path = "../optics-core" }
-optics-ethereum = { path = "../optics-ethereum"}
+optics-ethereum = { path = "../chains/optics-ethereum"}

scripts/test-all.sh

@@ -2,43 +2,48 @@
 # Script for testing entire optics monorepo unconditionally
 # Run from root (./scripts/test-all.sh)
 
+set -xe
+
 # update ABIs
-cd ./solidity
+(
+    cd ./solidity
 
-echo "+Lint and compile core"
-cd ./optics-core
-npm run lint
-npm run compile
+    echo "+Lint and compile core"
+    cd ./optics-core
+    npm run lint
+    npm run compile
 
-echo "+Lint and compile xApps"
-cd ../optics-xapps
-npm run lint
-npm run compile
-cd ..
+    echo "+Lint and compile xApps"
+    cd ../optics-xapps
+    npm run lint
+    npm run compile
+)
 
 # run Rust bins to output into vector JSON files
-cd ../rust/optics-core
-
-echo "+Running lib vector generation"
-echo '+cargo run --bin lib_test_output --features output'
-cargo run --bin lib_test_output --features output
-
-echo "+Running utils vector generation"
-echo '+cargo run --bin utils_test_output --features output'
-cargo run --bin utils_test_output --features output
-cd ..
-
-# Run rust tests, clippy, and formatting
-echo "+Running rust tests"
-echo '+cargo fmt -- --check'
-cargo fmt -- --check
-echo '+cargo clippy -- -D warnings'
-cargo clippy -- -D warnings
-echo '+cargo test -- -q'
-cargo test -- -q
-cd ..
+(
+    cd ./rust/optics-core
+
+    echo "+Running lib vector generation"
+    echo '+cargo run --bin lib_test_output --features output'
+    cargo run --bin lib_test_output --features output
+
+    echo "+Running utils vector generation"
+    echo '+cargo run --bin utils_test_output --features output'
+    cargo run --bin utils_test_output --features output
+
+    cd ..
+
+    # Run rust tests, clippy, and formatting
+    echo "+Running rust tests"
+    echo '+cargo fmt -- --check'
+    cargo fmt -- --check
+    echo '+cargo clippy -- -D warnings'
+    cargo clippy -- -D warnings
+    echo '+cargo test -- -q'
+    cargo test -- -q
+)
 
 # Run solidity tests
 echo "+Running solidity tests"
 cd ./typescript/optics-tests
-npm run testNoCompile
+npm run testNoCompile

solidity/optics-core/update_abis.sh

@@ -1,3 +1,7 @@
-cat artifacts/contracts/Replica.sol/Replica.json| jq .abi > ../../rust/optics-ethereum/abis/Replica.abi.json && \
-cat artifacts/contracts/Home.sol/Home.json| jq .abi > ../../rust/optics-ethereum/abis/Home.abi.json && \
-cat artifacts/contracts/XAppConnectionManager.sol/XAppConnectionManager.json | jq .abi > ../../rust/optics-ethereum/abis/XAppConnectionManager.abi.json
+#!/bin/sh 
+
+copy() {
+    jq .abi < artifacts/contracts/"$1".sol/"$1".json > ../../rust/chains/optics-ethereum/abis/"$1".abi.json
+}
+
+copy Replica && copy Home && copy XAppConnectionManager