用kubernetes資源物件建立Grafana Dashboard

首頁>技術>風月笑今生2020-03-30 13:28

用kubernetes資源物件建立Grafana Dashboard

我們在使用 Grafana Dashboard 來展示我們的監控圖表的時候，很多時候我們都是去找別人已經做好的 Dashboard 拿過來改一改，但是這樣也造成了很多使用 Grafana 的人員壓根不知道如何去自定義一個 Dashboard，雖然這並不是很困難。這裡我們介紹一個比較新穎（騷）的工具：[DARK](https://github.com/K-Phoen/dark)，全稱 `Dashboards As Resources in Kubernetes.`，意思就是通過 Kubernetes 的資源物件來定義 Grafana Dashboard，實現原理也很簡單，也就是通過 CRD 來定義 Dashboard，然後通過和 Grafana 的 API Token 進行互動實現 Dashboard 的 CRUD。

下面我們來看下如何使用 `DARK` 定義 Grafana Dashboard。首先 Clone 專案程式碼：

```shell

$ git clone https://github.com/K-Phoen/dark.git

```

然後安裝 CRD 資源：

```shell

$ kubectl apply -f k8s/crd.yaml

```

然後通過 Secret 物件建立 Grafana 的 API KEYS，在 Grafana 主介面中，選擇左側的配置選單 -> `API Keys` 建立 API Keys，選擇 `Editor` 的角色：

![api keys](/file/2020/03/30/20200330132500_67571.jpg.png)api keys

建立完成後會彈出一個對話方塊顯示對應的 `API Keys`，使用這個 KEY 來建立一個對應的 Secret 物件：

```yaml

$ kubectl create secret generic dark-tokens --from-literal=grafana=<替換成APIKEY>

```

然後修改 `k8s/cluster-role.yaml` 檔案，如下所示：

```yaml

apiVersion: v1

kind: ServiceAccount

metadata:

---

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

rules:

- apiGroups: ["k8s.kevingomez.fr"]

resources: ["grafanadashboards"]

verbs: ["get", "watch", "list"]

- apiGroups: [""]

resources: ["events"]

verbs: ["create", "patch"]

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

subjects:

- kind: ServiceAccount

namespace: default

roleRef:

kind: ClusterRole

apiGroup: rbac.authorization.k8s.io

```

然後建立上面的資源物件：

```shell

$ kubectl apply -f k8s/cluster-role.yaml

```

修改 `k8s/deployment.yaml` 檔案，將 `GRAFANA_HOST` 環境變數修改成自己的 Grafana 的地址，由於我這裡 Grafana 也安裝在 Kubernetes 叢集中的，所以直接用 DNS 形式配置，然後加上上面建立的 `dark` 這個 ServiceAccount：

```yaml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: dark

spec:

selector:

matchLabels:

app: dark

template:

metadata:

labels:

app: dark

spec:

volumes:

- name: dark-tokens

secret:

secretName: dark-tokens

serviceAccountName: dark

containers:

- name: dark

image: kphoen/dark:latest

env:

- name: GRAFANA_HOST

value: http://grafana.kube-mon:3000

- name: GRAFANA_TOKEN

valueFrom:

secretKeyRef:

key: grafana

```

修改完成後直接建立上面的 Controller：

```shell

$ kubectl apply -f k8s/deployment.yaml

$ kubectl get pods -l app=dark

NAME READY STATUS RESTARTS AGE

dark-6bd956b8d6-755p2 1/1 Running 0 36m

```

現在 Controller 定義好過後，實際上我們就可以去通過 CRD 物件來定義 Grafana Dashboard 了，如下所示定義了一個 `GrafanaDashboard` 物件，在物件中我們完全就可以根據自己的需求去定義內容了，比如定義 `annotations`、`variables`、`graph`、`table` 都可以，當然最重要的還是資料來源要正確，以及查詢語句：（example-dashboards.yaml）

```yaml

apiVersion: k8s.kevingomez.fr/v1

kind: GrafanaDashboard

metadata:

folder: "Test folder"

spec:

title: Awesome dashboard

editable: true

shared_crosshair: true

tags: [generated, yaml]

auto_refresh: 10s

tags_annotations:

- name: Deployments

datasource: "Prometheus"

color: "#5794F2"

tags: ["deploy", "production"]

variables:

- interval:

label: Interval

values: ["30s", "1m", "5m", "10m", "30m", "1h", "6h", "12h"]

- query:

label: HTTP status

datasource: Prometheus

request: "label_values(prometheus_http_requests_total, code)"

- const:

label: Percentile

default: 80

values_map:

50th: "50"

75th: "75"

80th: "80"

85th: "85"

90th: "90"

95th: "95"

99th: "99"

- custom:

default: v2

values_map:

v1: v1

v2: v2

rows:

- name: Prometheus

panels:

- graph:

title: HTTP Rate

height: 400px

datasource: Prometheus

targets:

- prometheus:

query: "rate(promhttp_metric_handler_requests_total[$interval])"

legend: "{{handler}} - {{ code }}"

- graph:

title: Heap allocations

height: 400px

datasource: Prometheus

targets:

- prometheus:

query: "go_memstats_heap_alloc_bytes"

legend: "{{job}}"

ref: A

- table:

title: Threads

datasource: Prometheus

targets:

- prometheus:

query: "go_threads"

hidden_columns: ["Time"]

time_series_aggregations:

- label: AVG

type: avg

- label: Current

type: current

- single_stat:

title: Heap Allocations

datasource: Prometheus

targets:

- prometheus:

query: 'go_memstats_heap_alloc_bytes{job="prometheus"}'

unit: bytes

thresholds: ["26000000", "28000000"]

color: ["value"]

- name: "Some text, because it might be useful"

panels:

- text:

title: Some awesome text?

markdown: "Markdown syntax help: [commonmark.org/help](https://commonmark.org/help/)\\n${percentile}"

- text:

title: Some awesome html?

html: "Some <b>awesome</b> html?"

```

同樣直接建立上面的示例檔案：

```shell

$ kubectl apply -f example-dashboards.yaml

$ kubectl get dashboards

NAME AGE

example-dashboard 35m

$ kubectl logs -f dark-6bd956b8d6-755p2

W0327 11:10:24.356194 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

I0327 11:10:24.360886 1 controller.go:87] Setting up event handlers

I0327 11:10:24.362305 1 controller.go:118] Starting dark-controller

I0327 11:10:24.362341 1 controller.go:121] Waiting for informer caches to sync

I0327 11:10:24.462733 1 controller.go:126] Starting workers

I0327 11:10:24.462820 1 controller.go:132] Started workers

I0327 11:13:22.641706 1 controller.go:197] Successfully synced 'default/example-dashboard'

I0327 11:13:22.643061 1 event.go:278] Event(v1.ObjectReference{Kind:"GrafanaDashboard", Namespace:"default", Name:"example-dashboard", UID:"efc6f96f-c7fc-40b5-8b8f-831a95b0a042", APIVersion:"k8s.kevingomez.fr/v1", ResourceVersion:"48490732", FieldPath:""}): type: 'Normal' reason: 'Synced' GrafanaDashboard synced successfully

```

在 Controller 中也可以看到對應的日誌資訊，資源物件建立成功以後，現在去 Grafana 頁面上檢視可以看到已經新增了一個 `Test folder` 的資料夾以及 `Awesome dashboard`：

![img](/file/2020/03/30/20200330132500_67572.jpg.png)

檢視 Dashboard 就可以看到和上面 CRD 中定義的各種圖表資訊了：

![grafana dashboard](/file/2020/03/30/20200330132500_67573.jpg.png)grafana dashboard

這樣我們就使用 Kubernetes 資源物件去定義了 Grafana Dashboard 了，這種方式比直接在頁面上去手動配置顯然要更優雅，也符合 `everything as code` 的思想。

Dashboard

YAML

∨ 深入淺出掌握React 與 React Native這兩個框架

劇多