Added options for metrics push mode (#674)

Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
6 years ago · e785d77d6d
parent 0fe372a22f
commit e785d77d6d
2 changed files with 164 additions and 19 deletions
--- a/docs/Reference/Pantheon-CLI-Syntax.md
+++ b/docs/Reference/Pantheon-CLI-Syntax.md
@ -210,9 +210,12 @@ The default is 25.
 metrics-enabled=true
 ```

-Set to `true` to enable the [Prometheus](https://prometheus.io/) monitoring service to access [Pantheon metrics](../Using-Pantheon/Debugging.md#monitor-node-performance-using-third-party-clients).
+Set to `true` to enable the [metrics exporter](../Using-Pantheon/Debugging.md#monitor-node-performance-using-prometheus).
 The default is `false`.

+`--metrics-enabled` cannot be specified with `--metrics-push-enabled`. That is, either Prometheus polling or Prometheus 
+push gateway support can be enabled but not both at once. 
+
 ### metrics-host

 ```bash tab="Syntax"
@ -227,8 +230,10 @@ The default is `false`.
 metrics-host="127.0.0.1"
 ```

-Specifies the host on which the [Prometheus](https://prometheus.io/) monitoring service accesses Pantheon
-metrics. The default is `127.0.0.1`. The metrics server respects the [`--host-whitelist` option](#host-whitelist).
+Specifies the host on which [Prometheus](https://prometheus.io/) accesses [Pantheon metrics](../Using-Pantheon/Debugging.md#monitor-node-performance-using-prometheus). 
+The metrics server respects the [`--host-whitelist` option](#host-whitelist).
+
+The default is `127.0.0.1`. 

 ### metrics-port

@ -244,8 +249,98 @@ metrics. The default is `127.0.0.1`. The metrics server respects the [`--host-wh
 metrics-port="6174"
 ```

-Specifies the port on which the [Prometheus](https://prometheus.io/) monitoring service accesses Pantheon
-metrics. The default is `9545`. The metrics server respects the [`--host-whitelist` option](#host-whitelist).
+Specifies the port on which [Prometheus](https://prometheus.io/) accesses [Pantheon metrics](../Using-Pantheon/Debugging.md#monitor-node-performance-using-prometheus).
+The default is `9545`. 
+
+### metrics-push-enabled 
+
+```bash tab="Syntax"
+--metrics-push-enabled[=<true|false>]
+```
+
+```bash tab="Example Command Line"
+--metrics-push-enabled
+```
+
+```bash tab="Example Configuration File"
+metrics-push-enabled="true"
+```
+
+Set to `true` to start the [push gateway integration](../Using-Pantheon/Debugging.md#running-prometheus-with-pantheon-in-push-mode).
+
+`--metrics-push-enabled` cannot be specified with `--metrics-enabled`. That is, either Prometheus polling or Prometheus 
+push gateway support can be enabled but not both at once.
+
+### metrics-push-host
+
+```bash tab="Syntax"
+--metrics-push-host=<HOST>
+```
+
+```bash tab="Example Command Line"
+--metrics-push-host=127.0.0.1
+```
+
+```bash tab="Example Configuration File"
+metrics-push-host="127.0.0.1"
+```
+
+Host of the [Prometheus Push Gateway](https://github.com/prometheus/pushgateway).
+The default is `127.0.0.1`. 
+The metrics server respects the [`--host-whitelist` option](#host-whitelist).
+
+!!! note
+    When pushing metrics, ensure `--metrics-push-host` is set to the machine on which the push gateway is. 
+    Generally, this will be a different machine to the machine on which Pantheon is running.  
+
+### metrics-push-interval
+
+```bash tab="Syntax"
+--metrics-push-interval=<INTEGER>
+```
+
+```bash tab="Example Command Line"
+--metrics-push-interval=30
+```
+
+```bash tab="Example Configuration File"
+metrics-push-interval=30
+```
+
+Interval in seconds to push metrics when in `push` mode. The default is 15.
+
+### metrics-push-port
+
+```bash tab="Syntax"
+--metrics-push-port=<PORT>
+```
+
+```bash tab="Example Command Line"
+--metrics-push-port=6174
+```
+
+```bash tab="Example Configuration File"
+metrics-push-port="6174"
+```
+
+Port of the [Prometheus Push Gateway](https://github.com/prometheus/pushgateway).
+The default is `9001`. 
+
+### metrics-push-prometheus-job
+
+```bash tab="Syntax"
+--metrics-prometheus-job=<metricsPrometheusJob>
+```
+
+```bash tab="Example Command Line"
+--metrics-prometheus-job="my-custom-job"
+```
+
+```bash tab="Example Configuration File"
+metrics-prometheus-job="my-custom-job"
+```
+
+Job name when in `push` mode. The default is `pantheon-client`. 

 ### miner-coinbase

--- a/docs/Using-Pantheon/Debugging.md
+++ b/docs/Using-Pantheon/Debugging.md
@ -13,26 +13,29 @@ not been automatically converted to smart quotes or that double hyphens have not
 You can monitor node performance using the [`debug_metrics`](../Reference/JSON-RPC-API-Methods.md#debug_metrics)
 JSON-RPC API method.

-## Monitor Node Performance Using Third-Party Clients
+## Monitor Node Performance Using Prometheus

 Use the [`--metrics-enabled` option](../Reference/Pantheon-CLI-Syntax.md#metrics-enabled) to enable the [Prometheus](https://prometheus.io/) monitoring and 
 alerting service to access Pantheon metrics. You can also visualize the collected data using [Grafana](https://grafana.com/).

-While Prometheus is running, it consumes the Pantheon data directly for monitoring. To specify the host and port on which 
-Prometheus accesses Pantheon, use the [`--metrics-listen` option](../Reference/Pantheon-CLI-Syntax.md#metrics-listen). 
-The default host and port are 127.0.0.1:9545.
+To specify the host and port on which Prometheus accesses Pantheon, use the [`--metrics-host`](../Reference/Pantheon-CLI-Syntax.md#metrics-host) and 
+[`--metrics-port`](../Reference/Pantheon-CLI-Syntax.md#metrics-port) options. 
+The default host and port are 127.0.0.1 and 9545.

 You can install other Prometheus components such as the Alert Manager. Additional configuration
 is not required for these components because Prometheus handles and analyzes data directly from the feed.

-Here's an example of setting up and running Prometheus with Pantheon:
+To use Prometheus with Pantheon, install the [prometheus main component](https://prometheus.io/download/). On MacOS, install with [Homebrew](https://formulae.brew.sh/formula/prometheus): 

-1. Install the [prometheus main component](https://prometheus.io/download/). On MacOS you can install with [Homebrew](https://brew.sh/): 
- ```bash
+ ```
 brew install prometheus
 ```

-2. Configure Prometheus to poll Pantheon. For example, add the following yaml fragment to the `scrape_configs`
+###  Setting up and Running Prometheus with Pantheon
+
+To configure Prometheus and run with Pantheon: 
+
+1. Configure Prometheus to poll Pantheon. For example, add the following yaml fragment to the `scrape_configs`
 block of the `prometheus.yml` file:
 
    ```yml tab="Example"
@ -52,7 +55,7 @@ block of the `prometheus.yml` file:
        using [`--host-whitelist`](../Reference/Pantheon-CLI-Syntax.md#host-whitelist) when starting Pantheon. 


-3. Start Pantheon with the [`--metrics-enabled` option](../Reference/Pantheon-CLI-Syntax.md#metrics-enabled). To start
+1. Start Pantheon with the [`--metrics-enabled` option](../Reference/Pantheon-CLI-Syntax.md#metrics-enabled). To start
 a single node for testing with metrics enabled:

    ```bash tab="Example"
@ -60,18 +63,65 @@ block of the `prometheus.yml` file:
    --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled
    ```

-4. In another terminal, run Prometheus specifying the `prometheus.yml` file: 
+1. In another terminal, run Prometheus specifying the `prometheus.yml` file: 

    ```bash tab="Example"
    prometheus --config.file=config.yml 
    ```

-5. Open a web browser to `http://localhost:9090` to view the Prometheus graphical interface.
+1. Open a web browser to `http://localhost:9090` to view the Prometheus graphical interface.

-6. Choose **Graph** from the menu bar and click the **Console** tab below.
+1. Choose **Graph** from the menu bar and click the **Console** tab below.

-7. From the **Insert metric at cursor** drop-down, select a metric such as `pantheon_blockchain_difficulty_total` or
+1. From the **Insert metric at cursor** drop-down, select a metric such as `pantheon_blockchain_difficulty_total` or
 `pantheon_blockchain_height` and click **Execute**. The values are displayed below.

    Click the **Graph** tab to view the data as a time-based graph. The query string is displayed below the graph. 
    For example: `{pantheon_blockchain_height{instance="localhost:9545",job="prometheus"}`
+
+### Running Prometheus with Pantheon in Push Mode 
+
+The [`--metrics-enabled`](../Reference/Pantheon-CLI-Syntax.md#metrics-enabled) option enables Prometheus polling 
+Pantheon but sometimes metrics are hard to poll (for example, when running inside Docker containers with varying IP addresses). 
+The [`--metrics-push-enabled`](../Reference/Pantheon-CLI-Syntax.md#metrics-push-enabled) option enables Pantheon 
+to push metrics to a [Prometheus Pushgateway](https://github.com/prometheus/pushgateway).   
+
+To configure Prometheus and run with Pantheon pushing to a push gateway: 
+
+1. Configure Prometheus to read from a push gateway. For example, add the following yaml fragment to the `scrape_configs`
+   block of the `prometheus.yml` file:
+    
+       ```yml tab="Example"
+        - job_name: push-gateway
+          metrics_path: /metrics
+          scheme: http
+          static_configs:
+          - targets:
+            - localhost:9091
+       ```
+   
+    !!! note
+        The [`--host-whitelist` option](../Reference/Pantheon-CLI-Syntax.md#host-whitelist) defaults to `localhost`.
+        If `127.0.0.1` is specified instead of `localhost` in the `prometheus.yml` file, add `127.0.0.1` to the host whitelist
+        using [`--host-whitelist`](../Reference/Pantheon-CLI-Syntax.md#host-whitelist) when starting Pantheon. 
+
+1. Start the push gateway. The push gateway can be deployed using the Docker image: 
+
+    ```bash tab="Example"
+    docker pull prom/pushgateway
+    docker run -d -p 9091:9091 prom/pushgateway
+    ```
+
+1. Start Pantheon specifying the `--metrics-push-enabled` option and port of the push gateway: 
+
+    ```bash tab="Example"
+    pantheon --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73 --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-push-enabled --metrics-push-port=9091 --metrics-push-host=127.0.0.1
+    ```
+
+1. In another terminal, run Prometheus specifying the `prometheus.yml` file: 
+   
+    ```bash tab="Example"
+    prometheus --config.file=config.yml 
+    ```
+
+1. View the Prometheus graphical interface as described in [Setting up and Running Prometheus with Pantheon](#setting-up-and-running-prometheus-with-pantheon).