hyperlane-monorepo/docs/failure-cases.md

## Optics Failure Cases

Optics is a robust system, resistant to all sorts of problems. However, there are a set of failure cases that require human intervention and need to be enumerated

### Agent State/Config

#### Updater
- *Two `updater`s deployed with the same config*
    - (See Double Update)
- *Extended updater downtime*
    - **Effect:**
        - Updates stop being sent for a period of time
    - **Mitigation:**
        - `Updater` Rotation (not implemented)
- *Fraudulent `updater`*
    - **Effect:**
        - Invalid or fraudulent update is sent  
    - **Mitigation:**
        - `Watcher` detects fraud, submits fraud proof (see Improper Update)
#### Relayer
- *`relayer` "relays" the same update more than once*
    - **Effect:**
        - Only the first one works
        - Subsequent transactions are rejected by the replicas
    - **Mitigation:**
        - Mempool scanning
            - "is there a tx in the mempool already that does what I want to do?"
            
            If so, do nothing, pick another message to process. 
        - __If minimizing gas use:__ Increase polling interval (check less often)
#### Processor
- *`processor` "processes" the same message more than once*
    - **Effect:**
        - Only the first one works
        - Subsequent transactions are rejected by the smart contracts
#### Watcher
- *Watcher and Fraudulent Updater Collude*
  - **Effect:**
      - Fraud is possible
  - **Mitigation:**
      - Distribute watcher operations to disparate entities. Anyone can run a watcher. 

#### General
- *Transaction Wallets Empty* 
    - **Effect:**
        - Transactions cease to be sent
    - **Mitigation:**
        - Monitor and top-up wallets on a regular basis
### Contract State
- *Double Update*
    - Happens if `Updater` (single key), submits two updates building off the "old root" with different "new root"
    - If two `updater`s were polling often but message volume was low, would likely result in the "same update" 
    - If two `updater`s were polling often but message volume was high, would likely result in a "double update" 
    - Doesn't necessarily need to be the __two updaters__, edge case could occur where the updater is submitting a transaction, crashes, and then reboots and submits a double update
    - **Effect:**
        - Home and Replicas go into a **Failed** state (stops working)
    - **Mitigation:**
        - Agent code has the ability to check its Database for a signed update, check whether it is going to submit a double update, and prevent itself from doing so 
        - Need to improve things there
        - Updater wait time
            - `Updater` doesn't want to double-update, so it creates an update and sits on it for some interval. If still valid after the interval, submit. __(Reorg mitigation)__
        - __"Just don't run multiple updaters with the same config"__
- *Improper Update*
    - Should only occur if the chain has a "deep reorg" that is longer than the `Updater`'s __pause period__ OR if the `Updater` is actively committing fraud. 
    - **Effect:**
        - `Home` goes into a **FAILED** state (stops working)
            - No plan for dealing with this currently
        - `Updater` gets slashed
            - (not implemented currently)
    - **Mitigation:**
        - `Watcher`(s) unenroll `xapps` 
        - Humans look at the situation, determine if the `Updater` was committing fraud or just the victim of poor consensus environment. 
### Network Environment
- *Network Partition*
    - When multiple nodes split off on a fork and break consensus 
    - Especially bad if the `updater` is off on the least-power chain (results in __Improper Update__)
    - **Effect:**
        - Manifests as a double-update
        - Manifests as an improper update
        - Messages simply stop 
    - **Mitigation:**
        - Pay attention and be on the right fork
        - **Stop signing updates when this occurs!**
        - Have a reliable mechanism for determining this is happening and pull the kill-switch. 
- *PoW Chain Reorg (See Network Partition)*
    - What happens when a __network partition__ ends
    - **Mitigation:**
- *PoS Chain Reorg (See Network Partition)*
    - Safety failure (BPs producing conflicting blocks)
    - Liveness Failure (no new blocks, chain stops finalizing new blocks)
    - **Effect:**
        - Slows down finality 
        - Blocks stop being produced 
    - How would this manifest in Celo? 
        - Celo would stop producing blocks.
        - Agents would __pause__ and sit there 
        - When agents see new blocks, they continue normal operations.