摘要:監控告警原型圖原型圖解釋與作為運行在同一個中并交由控制器管理,默認開啟端口,因為我們的與是處于同一個中,所以直接使用就可以與通信用于發送告警通知,告警規則配置以的形式掛載到容器供使用,告警通知對象配置也通過掛載到容器供使用,這里我們使用郵件
監控告警原型圖 原型圖解釋
prometheus與alertmanager作為container運行在同一個pods中并交由Deployment控制器管理,alertmanager默認開啟9093端口,因為我們的prometheus與alertmanager是處于同一個pod中,所以prometheus直接使用localhost:9093就可以與alertmanager通信(用于發送告警通知),告警規則配置rules.yml以Configmap的形式掛載到prometheus容器供prometheus使用,告警通知對象配置也通過Configmap掛載到alertmanager容器供alertmanager使用,這里我們使用郵件接收告警通知,具體配置在alertmanager.yml中
測試環境環境:Linux 3.10.0-693.el7.x86_64 x86_64 GNU/Linux
平臺:Kubernetes v1.10.5
Tips:prometheus與alertmanager完整的配置在文檔末尾
在prometheus中指定告警規則的路徑, rules.yml就是用來指定報警規則,這里我們將rules.yml用ConfigMap的形式掛載到/etc/prometheus目錄下面即可:
rule_files: - /etc/prometheus/rules.yml
這里我們指定了一個InstanceDown告警,當主機掛掉1分鐘則prometheus會發出告警
rules.yml: | groups: - name: example rules: - alert: InstanceDown expr: up == 0 for: 1m labels: severity: page annotations: summary: "Instance {{ $labels.instance }} down" description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes."配置prometheus與alertmanager通信(用于prometheus向alertmanager發送告警信息)
alertmanager默認開啟9093端口,又因為我們的prometheus與alertmanager是處于同一個pod中,所以prometheus直接使用localhost:9093就可以與alertmanager通信
alerting: alertmanagers: - static_configs: - targets: ["localhost:9093"]alertmanager配置告警通知對象
我們這里舉了一個郵件告警的例子,alertmanager接收到prometheus發出的告警時,alertmanager會向指定的郵箱發送一封告警郵件,這個配置也是通過Configmap的形式掛載到alertmanager所在的容器中供alertmanager使用
alertmanager.yml: |- global: smtp_smarthost: "smtp.exmail.qq.com:465" smtp_from: "xin.liu@woqutech.com" smtp_auth_username: "xin.liu@woqutech.com" smtp_auth_password: "xxxxxxxxxxxx" smtp_require_tls: false route: group_by: [alertname] group_wait: 30s group_interval: 5m repeat_interval: 10m receiver: default-receiver receivers: - name: "default-receiver" email_configs: - to: "1148576125@qq.com"原型效果展示
在prometheus web ui中可以看到配置的告警規則
為了看測試效果,關掉一個主機節點:
在prometheus web ui中可以看到一個InstanceDown告警被觸發
在alertmanager web ui中可以看到alertmanager收到prometheus發出的告警
指定接收告警的郵箱收到alertmanager發出的告警郵件
全部配置node_exporter_daemonset.yaml
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: node-exporter namespace: kube-system labels: app: node_exporter spec: selector: matchLabels: name: node_exporter template: metadata: labels: name: node_exporter spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: node-exporter image: alery/node-exporter:1.0 ports: - name: node-exporter containerPort: 9100 hostPort: 9100 volumeMounts: - name: localtime mountPath: /etc/localtime - name: host mountPath: /host readOnly: true volumes: - name: localtime hostPath: path: /usr/share/zoneinfo/Asia/Shanghai - name: host hostPath: path: /
alertmanager-cm.yaml
kind: ConfigMap apiVersion: v1 metadata: name: alertmanager namespace: kube-system data: alertmanager.yml: |- global: smtp_smarthost: "smtp.exmail.qq.com:465" smtp_from: "xin.liu@woqutech.com" smtp_auth_username: "xin.liu@woqutech.com" smtp_auth_password: "xxxxxxxxxxxx" smtp_require_tls: false route: group_by: [alertname] group_wait: 30s group_interval: 5m repeat_interval: 10m receiver: default-receiver receivers: - name: "default-receiver" email_configs: - to: "1148576125@qq.com"
prometheus-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: prometheus namespace: kube-system rules: - apiGroups: [""] resources: - nodes - nodes/proxy - services - endpoints - pods verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: v1 kind: ServiceAccount metadata: name: prometheus namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: prometheus namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: kube-system
prometheus-cm.yaml
kind: ConfigMap apiVersion: v1 data: prometheus.yml: | rule_files: - /etc/prometheus/rules.yml alerting: alertmanagers: - static_configs: - targets: ["localhost:9093"] scrape_configs: - job_name: "node" kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_ip] action: replace target_label: __address__ replacement: $1:9100 - source_labels: [__meta_kubernetes_pod_host_ip] action: replace target_label: instance - source_labels: [__meta_kubernetes_pod_node_name] action: replace target_label: node_name - action: labelmap regex: __meta_kubernetes_pod_label_(name) - source_labels: [__meta_kubernetes_pod_label_name] regex: node_exporter action: keep rules.yml: | groups: - name: example rules: - alert: InstanceDown expr: up == 0 for: 5m labels: severity: page annotations: summary: "Instance {{ $labels.instance }} down" description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes." - alert: APIHighRequestLatency expr: api_http_request_latencies_second{quantile="0.5"} > 1 for: 10m annotations: summary: "High request latency on {{ $labels.instance }}" description: "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)" metadata: name: prometheus-config-v0.1.0 namespace: kube-system
prometheus.yaml
apiVersion: extensions/v1beta1 kind: Deployment metadata: namespace: kube-system name: prometheus labels: name: prometheus spec: replicas: 1 selector: matchLabels: app: prometheus template: metadata: name: prometheus labels: app: prometheus spec: serviceAccountName: prometheus nodeSelector: node-role.kubernetes.io/master: "" tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master operator: Exists securityContext: runAsUser: 0 fsGroup: 0 containers: - name: prometheus image: prom/prometheus:v2.4.0 args: - "--config.file=/etc/prometheus/prometheus.yml" ports: - name: web containerPort: 9090 volumeMounts: - name: prometheus-config mountPath: /etc/prometheus - name: prometheus-storage mountPath: /prometheus - name: localtime mountPath: /etc/localtime - name: alertmanager image: prom/alertmanager:v0.14.0 args: - "--config.file=/etc/alertmanager/alertmanager.yml" - "--log.level=debug" ports: - containerPort: 9093 protocol: TCP name: alertmanager volumeMounts: - name: alertmanager-config mountPath: /etc/alertmanager - name: alertmanager-storage mountPath: /alertmanager - name: localtime mountPath: /etc/localtime volumes: - name: prometheus-config configMap: name: prometheus-config-v0.1.0 - name: alertmanager-config configMap: name: alertmanager - name: localtime hostPath: path: /usr/share/zoneinfo/Asia/Shanghai - name: prometheus-storage hostPath: path: /gaea/prometheus type: DirectoryOrCreate - name: alertmanager-storage hostPath: path: /gaea/alertmanager type: DirectoryOrCreate --- apiVersion: v1 kind: Service metadata: labels: name: prometheus kubernetes.io/cluster-service: "true" name: prometheus namespace: kube-system spec: ports: - name: prometheus nodePort: 30065 port: 9090 protocol: TCP targetPort: 9090 selector: app: prometheus sessionAffinity: None type: NodePort --- apiVersion: v1 kind: Service metadata: labels: name: prometheus kubernetes.io/cluster-service: "true" name: alertmanager namespace: kube-system spec: ports: - name: alertmanager nodePort: 30066 port: 9093 protocol: TCP targetPort: 9093 selector: app: prometheus sessionAffinity: None type: NodePort
文章版權歸作者所有,未經允許請勿轉載,若此文章存在違規行為,您可以聯系管理員刪除。
轉載請注明本文地址:http://m.specialneedsforspecialkids.com/yun/32736.html
摘要:同時有權限控制日志審計整體配置過期時間等功能。將成為趨勢前置條件要求的版本應該是因為和支持的限制的核心思想是將的部署與它監控的對象的配置分離,做到部署與監控對象的配置分離之后,就可以輕松實現動態配置。 一.單獨部署 二進制安裝各版本下載地址:https://prometheus.io/download/ Docker運行 運行命令:docker run --name promet...
摘要:集群三步安裝概述應當是使用監控系統的最佳實踐了,首先它一鍵構建整個監控系統,通過一些無侵入的手段去配置如監控數據源等故障自動恢復,高可用的告警等。。 kubernetes集群三步安裝 概述 prometheus operator應當是使用監控系統的最佳實踐了,首先它一鍵構建整個監控系統,通過一些無侵入的手段去配置如監控數據源等故障自動恢復,高可用的告警等。。 不過對于新手使用上還是有一...
摘要:集群三步安裝概述應當是使用監控系統的最佳實踐了,首先它一鍵構建整個監控系統,通過一些無侵入的手段去配置如監控數據源等故障自動恢復,高可用的告警等。。 kubernetes集群三步安裝 概述 prometheus operator應當是使用監控系統的最佳實踐了,首先它一鍵構建整個監控系統,通過一些無侵入的手段去配置如監控數據源等故障自動恢復,高可用的告警等。。 不過對于新手使用上還是有一...
閱讀 3086·2023-04-26 00:53
閱讀 3536·2021-11-19 09:58
閱讀 1700·2021-09-29 09:35
閱讀 3290·2021-09-28 09:46
閱讀 3869·2021-09-22 15:38
閱讀 2698·2019-08-30 15:55
閱讀 3016·2019-08-23 14:10
閱讀 3831·2019-08-22 18:17