From 05ff3c77904a364768d4bc9af0812ca48d68c429 Mon Sep 17 00:00:00 2001 From: Joshua Fernandes Date: Thu, 7 Apr 2022 08:08:15 +1000 Subject: [PATCH] updates for k8s monitoring (#1011) * updates for k8s monitoring Signed-off-by: Joshua Fernandes * updating the alerts note Signed-off-by: Joshua Fernandes * pr fixes Signed-off-by: Joshua Fernandes * pr fixes Signed-off-by: Joshua Fernandes * pr fixes Signed-off-by: Joshua Fernandes --- .lh/.lhignore | 6 + .../Kubernetes/Deploy-Charts.md.json | 122 ++++++++++++++++++ docs/Tutorials/Kubernetes/Deploy-Charts.md | 27 ++-- 3 files changed, 147 insertions(+), 8 deletions(-) create mode 100644 .lh/.lhignore create mode 100644 .lh/docs/Tutorials/Kubernetes/Deploy-Charts.md.json diff --git a/.lh/.lhignore b/.lh/.lhignore new file mode 100644 index 00000000..0145f592 --- /dev/null +++ b/.lh/.lhignore @@ -0,0 +1,6 @@ +# list file to not track by the local-history extension. comment line starts with a '#' character +# each line describe a regular expression pattern (search for "Javascript regex") +# it will relate to the workspace directory root. for example: +# ".*\.txt" ignores any file with "txt" extension +# "/test/.*" ignores all the files under the "test" directory +# ".*/test/.*" ignores all the files under any "test" directory (even under sub-folders) diff --git a/.lh/docs/Tutorials/Kubernetes/Deploy-Charts.md.json b/.lh/docs/Tutorials/Kubernetes/Deploy-Charts.md.json new file mode 100644 index 00000000..855b5d0f --- /dev/null +++ b/.lh/docs/Tutorials/Kubernetes/Deploy-Charts.md.json @@ -0,0 +1,122 @@ +{ + "sourceFile": "docs/Tutorials/Kubernetes/Deploy-Charts.md", + "activeCommit": 0, + "commits": [ + { + "activePatchIndex": 26, + "patches": [ + { + "date": 1649074473969, + "content": "Index: \n===================================================================\n--- \n+++ \n" + }, + { + "date": 1649074493114, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -79,9 +79,9 @@\n then deploy the chart using:\n \n ```bash\n cd dev/helm\n-helm install monitoring ./charts/quorum-monitoring --namespace besu --values ./values/monitoring.yaml\n+\n ```\n \n !!! warning\n \n" + }, + { + "date": 1649074538335, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -78,10 +78,14 @@\n Update the admin `username` and `password` in the [monitoring values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/monitoring.yml),\n then deploy the chart using:\n \n ```bash\n-cd dev/helm\n+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts\n+helm repo update\n+helm install monitoring prometheus-community/kube-prometheus-stack --version 34.6.0 --namespace=quorum --create-namespace --values ./values/monitoring.yml --wait\n+kubectl --namespace quorum apply -f ./values/monitoring/\n \n+# NOTE: please refer to values/monitoring.yml to configure the alerts per your requirements ie slack, email etc\n ```\n \n !!! warning\n \n" + }, + { + "date": 1649074543559, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -82,10 +82,8 @@\n helm repo add prometheus-community https://prometheus-community.github.io/helm-charts\n helm repo update\n helm install monitoring prometheus-community/kube-prometheus-stack --version 34.6.0 --namespace=quorum --create-namespace --values ./values/monitoring.yml --wait\n kubectl --namespace quorum apply -f ./values/monitoring/\n-\n-# NOTE: please refer to values/monitoring.yml to configure the alerts per your requirements ie slack, email etc\n ```\n \n !!! warning\n \n" + }, + { + "date": 1649074564314, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -86,9 +86,9 @@\n ```\n \n !!! warning\n \n- For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/).\n+ For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to configure \n \n Once complete, you can view deployments in the Kubernetes dashboard (or equivalent).\n \n ![k8s-monitoring](../../images/kubernetes-monitoring.png)\n" + }, + { + "date": 1649074569752, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -86,9 +86,10 @@\n ```\n \n !!! warning\n \n- For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to configure \n+ For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n+ configure \n \n Once complete, you can view deployments in the Kubernetes dashboard (or equivalent).\n \n ![k8s-monitoring](../../images/kubernetes-monitoring.png)\n" + }, + { + "date": 1649074584222, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -87,9 +87,9 @@\n \n !!! warning\n \n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n- configure \n+ configure alerts to the receiver of your choice i.e. \n \n Once complete, you can view deployments in the Kubernetes dashboard (or equivalent).\n \n ![k8s-monitoring](../../images/kubernetes-monitoring.png)\n" + }, + { + "date": 1649074589411, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -87,9 +87,9 @@\n \n !!! warning\n \n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n- configure alerts to the receiver of your choice i.e. \n+ configure alerts to the receiver of your choice i.e. email, slack etc\n \n Once complete, you can view deployments in the Kubernetes dashboard (or equivalent).\n \n ![k8s-monitoring](../../images/kubernetes-monitoring.png)\n" + }, + { + "date": 1649074636393, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -89,11 +89,9 @@\n \n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n configure alerts to the receiver of your choice i.e. email, slack etc\n \n-Once complete, you can view deployments in the Kubernetes dashboard (or equivalent).\n \n-![k8s-monitoring](../../images/kubernetes-monitoring.png)\n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n" + }, + { + "date": 1649074646883, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -89,10 +89,10 @@\n \n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n configure alerts to the receiver of your choice i.e. email, slack etc\n \n+Optionally you can also deploy \n \n-\n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n \n" + }, + { + "date": 1649074659683, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -89,9 +89,9 @@\n \n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n configure alerts to the receiver of your choice i.e. email, slack etc\n \n-Optionally you can also deploy \n+Optionally you can also deploy the Elastic Stack to view logs and metrics\n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n" + }, + { + "date": 1649074668785, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -89,9 +89,9 @@\n \n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n configure alerts to the receiver of your choice i.e. email, slack etc\n \n-Optionally you can also deploy the Elastic Stack to view logs and metrics\n+Optionally you can also deploy the Elastic Stack to view logs (and metrics)\n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n" + }, + { + "date": 1649074679316, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -89,9 +89,9 @@\n \n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n configure alerts to the receiver of your choice i.e. email, slack etc\n \n-Optionally you can also deploy the Elastic Stack to view logs (and metrics)\n+Optionally you can also deploy the [Elastic Stack to view logs (and metrics)\n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n" + }, + { + "date": 1649074694055, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -89,9 +89,9 @@\n \n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n configure alerts to the receiver of your choice i.e. email, slack etc\n \n-Optionally you can also deploy the [Elastic Stack to view logs (and metrics)\n+Optionally you can also deploy the [Elastic Stack]() to view logs (and metrics)\n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n" + }, + { + "date": 1649074704110, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -89,10 +89,11 @@\n \n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n configure alerts to the receiver of your choice i.e. email, slack etc\n \n-Optionally you can also deploy the [Elastic Stack]() to view logs (and metrics)\n+Optionally you can also deploy the [Elastic Stack](https://www.elastic.co/elastic-stack/) to view logs (and metrics).\n \n+\n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n \n" + }, + { + "date": 1649074710058, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -90,8 +90,14 @@\n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n configure alerts to the receiver of your choice i.e. email, slack etc\n \n Optionally you can also deploy the [Elastic Stack](https://www.elastic.co/elastic-stack/) to view logs (and metrics).\n+```bash\n+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts\n+helm repo update\n+helm install monitoring prometheus-community/kube-prometheus-stack --version 34.6.0 --namespace=quorum --create-namespace --values ./values/monitoring.yml --wait\n+kubectl --namespace quorum apply -f ./values/monitoring/\n+```\n \n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n" + }, + { + "date": 1649074722415, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -90,13 +90,15 @@\n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). Also remember to\n configure alerts to the receiver of your choice i.e. email, slack etc\n \n Optionally you can also deploy the [Elastic Stack](https://www.elastic.co/elastic-stack/) to view logs (and metrics).\n+\n ```bash\n-helm repo add prometheus-community https://prometheus-community.github.io/helm-charts\n+helm repo add elastic https://helm.elastic.co\n helm repo update\n-helm install monitoring prometheus-community/kube-prometheus-stack --version 34.6.0 --namespace=quorum --create-namespace --values ./values/monitoring.yml --wait\n-kubectl --namespace quorum apply -f ./values/monitoring/\n+helm install elasticsearch --version 7.16.3 elastic/elasticsearch --namespace quorum --create-namespace --values ./values/elasticsearch.yml\n+helm install kibana --version 7.16.3 elastic/kibana --namespace quorum --values ./values/kibana.yml\n+helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n ```\n \n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n" + }, + { + "date": 1649074728821, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -97,8 +97,10 @@\n helm repo update\n helm install elasticsearch --version 7.16.3 elastic/elasticsearch --namespace quorum --create-namespace --values ./values/elasticsearch.yml\n helm install kibana --version 7.16.3 elastic/kibana --namespace quorum --values ./values/kibana.yml\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n+\n+\n ```\n \n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n" + }, + { + "date": 1649074734328, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -97,10 +97,10 @@\n helm repo update\n helm install elasticsearch --version 7.16.3 elastic/elasticsearch --namespace quorum --create-namespace --values ./values/elasticsearch.yml\n helm install kibana --version 7.16.3 elastic/kibana --namespace quorum --values ./values/kibana.yml\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n+# to get\n \n-\n ```\n \n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n" + }, + { + "date": 1649074752609, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -97,9 +97,9 @@\n helm repo update\n helm install elasticsearch --version 7.16.3 elastic/elasticsearch --namespace quorum --create-namespace --values ./values/elasticsearch.yml\n helm install kibana --version 7.16.3 elastic/kibana --namespace quorum --values ./values/kibana.yml\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n-# to get\n+# to get metrics, please install metricbeat with config that is similar to filebeat \n \n ```\n \n \n" + }, + { + "date": 1649074758270, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -97,9 +97,9 @@\n helm repo update\n helm install elasticsearch --version 7.16.3 elastic/elasticsearch --namespace quorum --create-namespace --values ./values/elasticsearch.yml\n helm install kibana --version 7.16.3 elastic/kibana --namespace quorum --values ./values/kibana.yml\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n-# to get metrics, please install metricbeat with config that is similar to filebeat \n+# to get metrics, please install metricbeat with config that is similar to filebeat and once comp\n \n ```\n \n \n" + }, + { + "date": 1649074769224, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -97,9 +97,9 @@\n helm repo update\n helm install elasticsearch --version 7.16.3 elastic/elasticsearch --namespace quorum --create-namespace --values ./values/elasticsearch.yml\n helm install kibana --version 7.16.3 elastic/kibana --namespace quorum --values ./values/kibana.yml\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n-# to get metrics, please install metricbeat with config that is similar to filebeat and once comp\n+# to get metrics, please install metricbeat with config that is similar to filebeat and once complete create a `metricbeat` index in kibana\n \n ```\n \n \n" + }, + { + "date": 1649074828351, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -98,9 +98,8 @@\n helm install elasticsearch --version 7.16.3 elastic/elasticsearch --namespace quorum --create-namespace --values ./values/elasticsearch.yml\n helm install kibana --version 7.16.3 elastic/kibana --namespace quorum --values ./values/kibana.yml\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n # to get metrics, please install metricbeat with config that is similar to filebeat and once complete create a `metricbeat` index in kibana\n-\n ```\n \n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n@@ -203,9 +202,9 @@\n first validator was spun up, before the logs display blocks being created.\n \n ![k8s-validator-logs](../../images/kubernetes-validator-logs.png)\n \n-To add a validator into the network, deploy a normal RPC node (step 7) and then\n+**To add a validator into the network, deploy a normal RPC node (step 7) and then\n [vote](../../HowTo/Configure/Consensus-Protocols/IBFT.md#adding-and-removing-validators) it into the validator pool.\n \n ### 7. Deploy RPC or transaction nodes\n \n" + }, + { + "date": 1649074863621, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -100,9 +100,8 @@\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n # to get metrics, please install metricbeat with config that is similar to filebeat and once complete create a `metricbeat` index in kibana\n ```\n \n-\n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n \n@@ -202,9 +201,9 @@\n first validator was spun up, before the logs display blocks being created.\n \n ![k8s-validator-logs](../../images/kubernetes-validator-logs.png)\n \n-**To add a validator into the network, deploy a normal RPC node (step 7) and then\n+**To add a validator into the network**, deploy a normal RPC node (step 7) and then\n [vote](../../HowTo/Configure/Consensus-Protocols/IBFT.md#adding-and-removing-validators) it into the validator pool.\n \n ### 7. Deploy RPC or transaction nodes\n \n" + }, + { + "date": 1649074870469, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -100,8 +100,9 @@\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n # to get metrics, please install metricbeat with config that is similar to filebeat and once complete create a `metricbeat` index in kibana\n ```\n \n+\n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n \n" + }, + { + "date": 1649074885518, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -100,8 +100,9 @@\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n # to get metrics, please install metricbeat with config that is similar to filebeat and once complete create a `metricbeat` index in kibana\n ```\n \n+If you install `filebeat`, \n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n" + }, + { + "date": 1649074902562, + "content": "Index: \n===================================================================\n--- \n+++ \n@@ -100,9 +100,9 @@\n helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml\n # to get metrics, please install metricbeat with config that is similar to filebeat and once complete create a `metricbeat` index in kibana\n ```\n \n-If you install `filebeat`, please create a `filebeat-*` index pattern in `k\n+If you install `filebeat`, please create a `filebeat-*` index pattern in `ki\n \n You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\n and set the `database` and `secret_key_base` values.\n" + } + ], + "date": 1649074473969, + "name": "Commit-0", + "content": "---\ntitle: Besu Kubernetes - Deploying Charts\ndescription: Deploying Besu Helm Charts for a Kubernetes cluster\n---\n\n## Prerequisites\n\n* Clone the [Quorum-Kubernetes](https://github.com/ConsenSys/quorum-kubernetes) repository\n* A [running Kubernetes cluster](./Create-Cluster.md)\n* Install [Kubectl](https://kubernetes.io/docs/tasks/tools/)\n* Install [Helm3](https://helm.sh/docs/intro/install/)\n\n## Provision with Helm charts\n\nHelm allows you to package a collection of objects into a chart which can be deployed to the cluster. For the\nrest of this tutorial we use the **[Dev](https://github.com/ConsenSys/quorum-kubernetes/tree/master/dev)** Helm charts.\nAfter cloning the [Quorum-Kubernetes](https://github.com/ConsenSys/quorum-kubernetes) repository, change\nto the `dev` directory for the rest of this tutorial.\n\n```bash\ncd dev/helm\n```\n\nIf you're running the cluster on AWS or Azure, update the `values.yml` with `provider: aws` or\n`provider: azure` as well.\n\n!!! note\n\n You can customize any of the charts in this repository to suit your requirements, and make pull requests to\n extend functionality.\n\n### 1. Check that you can connect to the cluster with `kubectl`\n\nVerify kubectl is connected to cluster using:\n\n```bash\nkubectl version\n```\n\nThe result looks similar to:\n\n```bash\nClient Version: version.Info{Major:\"1\", Minor:\"23\", GitVersion:\"v1.23.1\", GitCommit:\"86ec240af8cbd1b60bcc4c03c20da9b98005b92e\", GitTreeState:\"clean\", BuildDate:\"2021-12-16T11:41:01Z\", GoVersion:\"go1.17.5\", Compiler:\"gc\", Platform:\"linux/amd64\"}\nServer Version: version.Info{Major:\"1\", Minor:\"22\", GitVersion:\"v1.22.3\", GitCommit:\"c92036820499fedefec0f847e2054d824aea6cd1\", GitTreeState:\"clean\", BuildDate:\"2021-10-27T18:35:25Z\", GoVersion:\"go1.16.9\", Compiler:\"gc\", Platform:\"linux/amd64\"}\n```\n\n### 2. Deploy the network\n\nThis tutorial isolates groups of resources (for example, StatefulSets and Services) within a single cluster.\n\n!!! note\n\n The rest of this tutorial uses `besu` as the namespace,\n but you're free to pick any name when deploying, as long as it's consistent across the\n [infrastructure scripts](./Create-Cluster.md) and charts.\n\nRun the following in a terminal window:\n\n```bash\nkubectl create namespace besu\n```\n\n### 3. Deploy the metrics chart\n\nThis chart deploys Prometheus and Grafana to monitor the cluster, nodes, and state of the network.\nEach Besu pod has [`annotations`](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)\nwhich allow Prometheus to scrape metrics from the pod at a specified port and path. For example:\n\n```bash\n template:\n metadata:\n annotations:\n prometheus.io/scrape: \"true\"\n prometheus.io/port: 9545\n prometheus.io/path: \"/metrics\"\n```\n\nUpdate the admin `username` and `password` in the [monitoring values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/monitoring.yml),\nthen deploy the chart using:\n\n```bash\ncd dev/helm\nhelm install monitoring ./charts/quorum-monitoring --namespace besu --values ./values/monitoring.yaml\n```\n\n!!! warning\n\n For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/).\n\nOnce complete, you can view deployments in the Kubernetes dashboard (or equivalent).\n\n![k8s-monitoring](../../images/kubernetes-monitoring.png)\n\nYou can optionally deploy BlockScout to aid with monitoring the network. To do this, update the\n[BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml)\nand set the `database` and `secret_key_base` values.\n\n!!! important\n\n Changes to the database requires changes to both the `database` and the `blockscout` dictionaries.\n\nOnce completed, deploy the chart using:\n\n```bash\nhelm dependency update ./charts/blockscout\nhelm install blockscout ./charts/blockscout --namespace besu --values ./values/blockscout-besu.yaml\n```\n\n### 4. Deploy the genesis chart\n\nThe genesis chart creates the genesis file and keys for the validators and bootnodes.\n\n!!! warning\n\n It's important to keep the release names of the bootnodes and validators as per this tutorial, that is `bootnode-n` and\n `validator-n` (for the initial validator pool), where `n` is the node number. Any validators created after the initial\n pool can be named to anything you like.\n\nUpdate the number of validators, accounts, chain ID, and any parameters for the genesis file in the\n[`genesis-besu` values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/genesis-besu.yml), then\ndeploy the chart using:\n\n```bash\nhelm install genesis ./charts/besu-genesis --namespace besu --create-namespace --values ./values/genesis-besu.yml\n```\n\nOnce completed, view the genesis and enodes (the list of static nodes) configuration maps that every Besu node uses, and\nthe validator and bootnode node keys as secrets.\n\n![k8s-genesis-configmaps](../../images/kubernetes-genesis-configmaps.png)\n\n![k8s-genesis-secrets](../../images/kuberenetes-genesis-secrets.png)\n\n### 5. Deploy the bootnodes\n\nThe Dev charts use two bootnodes to replicate best practices for a production network. Each Besu node has flags\nthat tell the StatefulSet what to deploy and how to clean up.\nThe default `values.yml` for the StatefulSet define the following flags which are present in all the\noverride values files.\n\n```bash\nnodeFlags:\n bootnode: true\n generateKeys: false\n privacy: false\n removeKeysOnDeletion: false\n```\n\nWe don't generate keys for the bootnodes and initial validator pool. To create a Tessera pod paired to Besu\nfor private transactions, set the `privacy` flag to `true`. Optionally remove the secrets for the node if you\ndelete the StatefulSet (for example removing a member node) by setting the `removeKeysOnDeletion` flag to `true`.\n\nFor the bootnodes, set the `bootnode` flag to `true` to indicate they are bootnodes. All the other nodes\n(for example, validators, and members) wait for the bootnodes to be up before proceeding, and have this flag set to `false`.\n\n```bash\nhelm install bootnode-1 ./charts/besu-node --namespace besu --values ./values/bootnode.yml\nhelm install bootnode-2 ./charts/besu-node --namespace besu --values ./values/bootnode.yml\n```\n\n!!! warning\n\n It's important to keep the release names of the bootnodes the same as it is tied to the keys that the genesis chart\n creates. So we use `bootnode-1` and `bootnode-2` in the previous command.\n\nOnce complete, you see two StatefulSets, and the two bootnodes discover themselves and peer.\nBecause there are no validators present yet, there are no blocks created, as seen in the following logs.\n\n![k8s-bootnode-logs](../../images/kubernetes-bootnode-logs.png)\n\n### 6. Deploy the validators\n\nThe validators peer with the bootnodes and themselves, and when a majority of the validators have peered, blocks\nare proposed and created on the chain.\n\nFor the initial validator pool we set all the node flags to `false` and then deploy.\n\n```bash\nhelm install validator-1 ./charts/besu-node --namespace besu --values ./values/validator.yml\nhelm install validator-2 ./charts/besu-node --namespace besu --values ./values/validator.yml\nhelm install validator-3 ./charts/besu-node --namespace besu --values ./values/validator.yml\nhelm install validator-4 ./charts/besu-node --namespace besu --values ./values/validator.yml\n```\n\n!!! warning\n\n As with the bootnodes, it's important to keep the release names of the initial validators the same as it is tied\n to the keys that the genesis chart creates. So we use `validator-1`, `validator-2`, etc. in the previous command.\n\nOnce completed, you may need to give the validators a few minutes to peer and for round changes, depending on when the\nfirst validator was spun up, before the logs display blocks being created.\n\n![k8s-validator-logs](../../images/kubernetes-validator-logs.png)\n\nTo add a validator into the network, deploy a normal RPC node (step 7) and then\n[vote](../../HowTo/Configure/Consensus-Protocols/IBFT.md#adding-and-removing-validators) it into the validator pool.\n\n### 7. Deploy RPC or transaction nodes\n\nThese nodes need their own node keys, so set the `generateKeys` flag to `true` for a standard RPC node.\nFor a transaction node (Besu paired with Tessera for private transactions), set the `privacy` flag to `true` and\ndeploy in the same manner as before.\n\nFor an RPC node with the release name `rpc-1`:\n\n```bash\nhelm install rpc-1 ./charts/besu-node --namespace besu --values ./values/reader.yml\n```\n\nFor a transaction node release name `tx-1`:\n\n```bash\nhelm install tx-1 ./charts/besu-node --namespace besu --values ./values/txnode.yml\n```\n\nLogs for `tx-1` resemble the following for Tessera:\n\n![`k8s-tx-tessera-logs`](../../images/kubernetes-tx-tessera-logs.png)\n\nLogs for Besu resemble the following:\n\n![`k8s-tx-Besu-logs`](../../images/kubernetes-tx-Besu-logs.png)\n\n### 8. Connect to the node from your local machine via an ingress\n\nTo view Grafana dashboards or connect to the nodes to make transactions from your local machine, you can\ndeploy an ingress controller with rules. We use the `ingress-nginx` ingress controller which can be deployed as follows:\n\n```bash\nhelm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\nhelm repo update\nhelm install besu-ingress ingress-nginx/ingress-nginx \\\n --namespace besu \\\n --set controller.replicaCount=1 \\\n --set controller.nodeSelector.\"beta\\.kubernetes\\.io/os\"=linux \\\n --set defaultBackend.nodeSelector.\"beta\\.kubernetes\\.io/os\"=linux \\\n --set controller.admissionWebhooks.patch.nodeSelector.\"beta\\.kubernetes\\.io/os\"=linux \\\n --set controller.service.externalTrafficPolicy=Local\n```\n\nUse [pre-defined rules](https://github.com/ConsenSys/quorum-kubernetes/blob/master/ingress/ingress-rules-besu.yml)\nto test functionality, and alter to suit your requirements (for example, to connect to multiple nodes via different paths).\n\nEdit the [rules](https://github.com/ConsenSys/quorum-kubernetes/blob/master/ingress/ingress-rules-besu.yml) file so that the\nservice names match your release name. In the example, we deployed a transaction node with the release name `member-1`\nso the corresponding service is called `besu-node-member-1` for the `rpc` and `ws` path prefixes. Once you have settings\nthat match your deployments, deploy the rules as follows:\n\n```bash\nkubectl apply -f ../../ingress/ingress-rules-besu.yml\n```\n\nOnce complete, view the IP address under the `Ingress` section if you're using the Kubernetes Dashboard\nor equivalent `kubectl` command.\n\n![`k8s-ingress`](../../images/kubernetes-ingress-ip.png)\n\nYou can view the Grafana dashboard by going to:\n\n```bash\n# For Besu's grafana address:\nhttp:///d/XE4V0WGZz/besu-overview?orgId=1&refresh=10s\n```\n\nThe following is an example RPC call, which confirms that the node running the JSON-RPC service is syncing:\n\n=== \"curl HTTP request\"\n\n ```bash\n curl -v -X POST -H \"Content-Type: application/json\" --data '{\"jsonrpc\":\"2.0\",\"method\":\"eth_blockNumber\",\"params\":[],\"id\":1}' http:///rpc\n ```\n\n=== \"JSON result\"\n\n ```json\n {\n \"jsonrpc\" : \"2.0\",\n \"id\" : 1,\n \"result\" : \"0x4e9\"\n }\n ```\n" + } + ] +} \ No newline at end of file diff --git a/docs/Tutorials/Kubernetes/Deploy-Charts.md b/docs/Tutorials/Kubernetes/Deploy-Charts.md index 6dda62b8..2184d527 100644 --- a/docs/Tutorials/Kubernetes/Deploy-Charts.md +++ b/docs/Tutorials/Kubernetes/Deploy-Charts.md @@ -60,7 +60,7 @@ Run the following in a terminal window: kubectl create namespace besu ``` -### 3. Deploy the monitoring chart +### 3. Deploy the metrics chart This chart deploys Prometheus and Grafana to monitor the cluster, nodes, and state of the network. Each Besu pod has [`annotations`](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) @@ -75,21 +75,32 @@ which allow Prometheus to scrape metrics from the pod at a specified port and pa prometheus.io/path: "/metrics" ``` -Update the admin `username` and `password` in the [monitoring values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/monitoring.yml), -then deploy the chart using: +Update the admin `username` and `password` in the [monitoring values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/monitoring.yml). +Configure alerts to the receiver of your choice (for example, email or Slack), then deploy the chart using: ```bash -cd dev/helm -helm install monitoring ./charts/quorum-monitoring --namespace besu --values ./values/monitoring.yaml +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update +helm install monitoring prometheus-community/kube-prometheus-stack --version 34.6.0 --namespace=quorum --create-namespace --values ./values/monitoring.yml --wait +kubectl --namespace quorum apply -f ./values/monitoring/ ``` !!! warning For production use cases, configure Grafana with one of the supported [native auth mechanisms](https://grafana.com/docs/grafana/latest/auth/). -Once complete, you can view deployments in the Kubernetes dashboard (or equivalent). +Optionally you can also deploy the [Elastic Stack](https://www.elastic.co/elastic-stack/) to view logs (and metrics). + +```bash +helm repo add elastic https://helm.elastic.co +helm repo update +helm install elasticsearch --version 7.16.3 elastic/elasticsearch --namespace quorum --create-namespace --values ./values/elasticsearch.yml +helm install kibana --version 7.16.3 elastic/kibana --namespace quorum --values ./values/kibana.yml +helm install filebeat elastic/filebeat --namespace quorum --values ./values/filebeat.yml +# to get metrics, please install metricbeat with config that is similar to filebeat and once complete create a `metricbeat` index in kibana +``` -![k8s-monitoring](../../images/kubernetes-monitoring.png) +If you install `filebeat`, please create a `filebeat-*` index pattern in `kibana`. All the logs from the nodes are sent to the `filebeat` index. You can optionally deploy BlockScout to aid with monitoring the network. To do this, update the [BlockScout values file](https://github.com/ConsenSys/quorum-kubernetes/blob/master/dev/helm/values/blockscout-besu.yml) @@ -192,7 +203,7 @@ first validator was spun up, before the logs display blocks being created. ![k8s-validator-logs](../../images/kubernetes-validator-logs.png) -To add a validator into the network, deploy a normal RPC node (step 7) and then +**To add a validator into the network**, deploy a normal RPC node (step 7) and then [vote](../../HowTo/Configure/Consensus-Protocols/IBFT.md#adding-and-removing-validators) it into the validator pool. ### 7. Deploy RPC or transaction nodes