实操Install Victoriametrics in K8s
背景
之前给大家介绍了victoriametrics以及安装中的一些注意事项,今天来给大家实操一下,如何在k8s中进行安装。本次是基于云上的k8s上安装一个cluster版本的victoriametrics,需要使用到云上的负载均衡。
注:victoriametrics后续简称vm
安装准备 一个k8s集群,我的k8s版本是v1.20.6 在集群上准备好一个storageclass,我这里用的NFS来做的 operator镜像tag为v0.17.2,vmstorage、vmselect和vminsert镜像tag为v1.63.0。可提前拉取镜像保存到本地镜像仓库 安装须知
vm可以通过多种方式安装,如二进制、docker镜像以及源码。可根据场景进行选择。如果在k8s中进行安装,我们可以直接使用operator来进行安装。下面重点说一下安装过程中的一些注意事项。
一个最小的集群必须包含以下节点:
一个vmstorage单节点,另外要指定-retentionPeriod和-storageDataPath两个参数 一个vminsert单节点,要指定-storageNode= 一个vmselect单节点,要指定-storageNode=注:高可用情况下,建议每个服务至少有个两个节点
在vmselect和vminsert前面需要一个负载均衡,比如vmauth、nginx。这里我们使用云上的负载均衡。同时要求:
以/insert开头的请求必须要被路由到vminsert节点的8480端口 以/select开头的请求必须要被路由到vmselect节点的8481端口注:各服务的端口可以通过-httpListenAddr进行指定
建议为集群安装监控
如果是在一个主机上进行安装测试集群,vminsert、vmselect和vmstorage各自的-httpListenAddr参数必须唯一,vmstorage的-storageDataPath、-vminsertAddr、-vmselectAddr这几个参数必须有唯一的值。
当vmstorage通过-storageDataPath目录大小小于通过-storage.minFreeDiskSpaceBytes指定的可用空间时,会切换到只读模式;vminsert停止像这类节点发送数据,转而将数据发送到其他可用vmstorage节点
安装过程 安装vm
1、创建crd
# 下载安装文件 export VM_VERSION=`basename $(curl -fs -o/dev/null -w %{redirect_url} https://github.com/VictoriaMetrics/operator/releases/latest)` wget https://github.com/VictoriaMetrics/operator/releases/download/$VM_VERSION/bundle_crd.zip unzip bundle_crd.zip kubectl apply -f release/crds # 检查crd [root@test opt]# kubectl get crd |grep vm vmagents.operator.victoriametrics.com 2022-01-05T07:26:01Z vmalertmanagerconfigs.operator.victoriametrics.com 2022-01-05T07:26:01Z vmalertmanagers.operator.victoriametrics.com 2022-01-05T07:26:01Z vmalerts.operator.victoriametrics.com 2022-01-05T07:26:01Z vmauths.operator.victoriametrics.com 2022-01-05T07:26:01Z vmclusters.operator.victoriametrics.com 2022-01-05T07:26:01Z vmnodescrapes.operator.victoriametrics.com 2022-01-05T07:26:01Z vmpodscrapes.operator.victoriametrics.com 2022-01-05T07:26:01Z vmprobes.operator.victoriametrics.com 2022-01-05T07:26:01Z vmrules.operator.victoriametrics.com 2022-01-05T07:26:01实操Z vmservicescrapes.operator.victoriametrics.com 2022-01-05T07:26:01Z vmsingles.operator.victoriametrics.com 2022-01-05T07:26:01Z vmstaticscrapes.operator.victoriametrics.com 2022-01-05T07:26:01Z vmusers.operator.victoriametrics.com 2022-01-05T07:26:01Z
2、安装operator
# 安装operator。记得提前修改operator的镜像地址 kubectl apply -f release/operator/ # 安装后检查operator是否正常 [root@test opt]# kubectl get po -n monitoring-system vm-operator-76dd8f7b84-gsbfs 1/1 Running 0 25h
3、安装vmcluster operator安装完成后,需要根据自己的需求去构建自己的的cr。我这里安装一个vmcluster。先看看vmcluster安装文件
# cat vmcluster-install.yaml apiVersion: operator.victoriametrics.com/v1beta1 kind: VMCluster metadata: name: vmcluster namespace: monitoring-system spec: replicationFactor: 1 retentionPeriod: "4" vminsert: image: pullPolicy: IfNotPresent repository: images.huazai.com/release/vminsert tag: v1.63.0 podMetadata: labels: victoriaMetrics: vminsert replicaCount: 1 resources: limits: cpu: "1" memory: 1000Mi requests: cpu: 500m memory: 500Mi vmselect: cacheMountPath: /select-cache image: pullPolicy: IfNotPresent repository: images.huazai.com/release/vmselect tag: v1.63.0 podMetadata: labels: victoriaMetrics: vmselect replicaCount: 1 &nbsInp; resources: limits: cpu: "1" memory: 1000Mi requests: cpu: 500m memory: 500Mi storage: volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 2G storageClassName: nfs-csi volumeMode: Filesystem vmstorage: image: pullPolicy: IfNotPresent repository: images.huazai.com/release/vmstorage tag: v1.63.0 podMetadata: labels: victoriaMetrics: vmstorage replicaCount: 1 resources: limits: cpu: "1" memory: 1500Mi requests: cpu: 500m memory: 750Mi storage: volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 20G storageClassName: nfs-csi volumeMode: Filesystem storageDataPath:&nbsstp;/vm-data # install vmcluster kubectl apply -f vmcluster-install.yaml # 检查vmcluster install结果 [root@test opt]# kubectl get po -n monitoring-system NAME READY STATUS RESTARTS AGE vm-operator-76dd8f7b84-gsbfs 1/1 Running 0 26h vminsert-vmcluster-main-69766c8f4-r795w 1/1 Running 0 25h vmselect-vmcluster-main-0 1/1 Running 0 25h vmstorage-vmcluster-main-0 1/1 Running 0 25h
4、创建vminsert和vmselect service
# 查看创建的svc [root@test opt]# kubectl get svc -n monitoring-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vminsert-vmcluster-main ClusterIP 10.0.182.73 8480/TCP 25h vmselect-vmcluster-main ClusterIP None 8481/TCP 25h vmstorage-vmcluster-main ClusterIP None &alnbsp; 8482/TCP,8400/TCP,8401/TCP 25h # 这里为了方便不同k8s集群的数据都可以存储到该vm来,同时方便后续查询数据, # 重新创建两个svc,类型为nodeport,分别为vminsert-lbsvc和vmselect-lbsvc.同时配置云上的lb监听8480和8481端口,后端服务器为vm所在集群的节点ip, # 端口为vminsert-lbsvc和vmsleect-lbsvc两个service暴露出来的nodeport # 但与vm同k8s集群的比如opentelemetry需要存储数据时,仍然可以用: # vminsert-vmcluster-main.kube-system.svc.cluster.local:8480 # 与vm不同k8s集群的如opentelemetry存储数据时使用lb:8480 # cat vminsert-lb-svc.yaml apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: monitoring app.kubernetes.io/instance: vmcluster-main app.kubernetes.io/name: vminsert name: vminsert-vmcluster-main-lbsvc namespace: monitoring-system spec: externalTrafficPolicy: Cluster ports: - name: http nodePort: 30135 port: 8480 protocol: TCP targetPort: 8480 selector: app.kubernetes.io/component: monitoring app.kubernetes.io/instance: vmcluster-main app.kubernetes.io/name: vminsert sessionAffinity: None type: NodePort # cat vmselect-lb-svc.yaml apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: monitoring app.kubernetes.io/instance: vmcluster-main app.kubernetes.io/name: vmselect name: vmselect-vmcluster-main-lbsvc namespace: monitoring-system spec: externalTrafficPolicy: Cluster ports: - name: http nodePort: 31140 port: 8481 protocol: TCP targetPort: 8481 selector: app.kubernetes.io/component: monitoring app.kubernetes.io/instance: vmcluster-main app.kubernetes.io/name: vmselect sessionAffinity: None type: NodePort # 创建svc kubectl apply -f vmselect-lb-svc.yaml kubectl apply -f vminsert-lb-svc.yaml &l nbsp;# !!配置云上lb, 自行配置 # 最后检查vm相关的pod和svc [root@test opt]# kubectl get po,svc -n monitoring-system NAME READY STATUS RESTARTS AGE pod/vm-operator-76dd8f7b84-gsbfs 1/1 Running 0 30h pod/vminsert-vmcluster-main-69766c8f4-r795w 1/1 Running 0 29h pod/vmselect-vmcluster-main-0 1/1 Running 0 29h pod/vmstorage-vmcluster-main-0 1/1 Running 0 29h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/vminsert-vmcluster-main ClusterIP 10.0.182.73 8480/TCP 29h service/vminsert-vmcluster-main-lbsvc NodePort 10.0.255.212 8480:30135/TCP 7h54m service/vmselect-vmcluster-main ClusterIP None 8481/TCP 29h&nVibsp;service/vmselect-vmcluster-main-lbsvc NodePort 10.0.45.239 8481:31140/TCP 7h54m service/vmstorage-vmcluster-main ClusterIP None 8482/TCP,8400/TCP,8401/TCP 29h 安装prometheus-expoter
这里还是来安装node exporter,暴露k8s节点数据,由后续的opentelemetry来采集,并通过vminsert存储到vmstorage。数据通过vmselect来进行查询
# kubectl apply -f prometheus-node-exporter-install.yaml apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: prometheus-node-exporter release: prometheus-node-exporter name: prometheus-node-exporter namespace: kube-system spec: revisionHistoryLimit: 10 selector: matchLabels: app: prometheus-node-exporter release: prometheus-node-exporter template: metadata: labels: app: prometheus-node-exporter release: prometheus-node-exporter spec: containers: - args: - --path.procfs=/host/proc - --path.sysfs=/host/sys - --path.rootfs=/host/root - --web.listen-address=$(HOST_IP):9100 env: - name: HOST_IP value: 0.0.0.0 image: images.huazai.com/release/node-exporter:v1.1.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: / &nctbsp; port: 9100 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: node-exporter ports: - containerPort: 9100 hostPort: 9100 name: metrics protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: / port: 9100 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: 200m memory: 50Mi requests: cpu: 100m memory: 30Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /host/proc name: proc readOnly: true - mountPath: /host/sys name: sys &nbsorp; readOnly: true - mountPath: /host/root mountPropagation: HostToContainer name: root readOnly: true dnsPolicy: ClusterFirst hostNetwork: true hostPID: true restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 65534 runAsGroup: 65534 runAsNonRoot: true runAsUser: 65534 serviceAccount: prometheus-node-exporter serviceAccountName: prometheus-node-exporter terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /proc type: "" name: proc - hostPath: path: /sys type: "" name: sys - hostPath: path: / type: "" name: root updateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate # 检查node-exporter [root@test ~]# kubectl get po -n kube-system |grep prometheus prometheus-node-exporter-89wjk 1/1  ia; Running 0 31h prometheus-node-exporter-hj4gh 1/1 Running 0 31h prometheus-node-exporter-hxm8t 1/1 Running 0 31h prometheus-node-exporter-nhqp6 1/1 Running 0 31h 安装opentelemetry
prometheus node exporter安装好之后,再来安装opentelemetry(以后有机会再介绍)
# opentelemetry 配置文件。定义数据的接收、处理、导出 # 1.receivers即从哪里获取数据 # 2.processors即对获取的数据的处理 # 3.exporters即将处理过的数据导出到哪里,本次数据通过vminsert最终写入到vmstorage # kubectl apply -f opentelemetry-install-cm.yaml apiVersion: v1 data: relay: | exporters: prometheusremotewrite: # 我这里配置lb_ip:8480,即vminsert地址 endpoint: http://lb_ip:8480/insert/0/prometheus # 不同的集群添加不同的label,比如cluster: uat/prd external_labels: cluster: uat extensions: health_check: {} processors: batch: {} memory_limiter: ballast_size_mib: 819 check_interval: 5s limit_mib: 1638 spike_limit_mib: 512 receivers: prometheus: config: scrape_configs: - job_name: opentelemetry-collector scrape_interval: 10s static_configs: &menbsp; - targets: - localhost:8888 ...省略... - job_name: kube-state-metrics kubernetes_sd_configs: - namespaces: names: - kube-system role: service metric_relabel_configs: - regex: ReplicaSet;([w|-]+)-[0-9|a-z]+ replacement: $$1 source_labels: - created_by_kind - created_by_name target_label: created_by_name - regex: ReplicaSet replacement: Deployment source_labels: - created_by_kind target_label: created_by_kind relabel_configs: - action: keep regex: kube-state-metrics source_labels: - __meta_kubernetes_service_name - job_name: node-exporter kubernetes_sd_configs: &nbtrsp; - namespaces: names: - kube-system role: endpoints relabel_configs: - action: keep regex: node-exporter source_labels: - __meta_kubernetes_service_name - source_labels: - __meta_kubernetes_pod_node_name target_label: node - source_labels: - __meta_kubernetes_pod_host_ip target_label: host_ip ...省略... service: # 上面定义的receivors、processors、exporters以及extensions需要在这里配置,不然不起作用 extensions: - health_check pipelines: metrics: exporters: - prometheusremotewrite processors: - memory_limiter - batch receivers: - prometheus kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: opentelemetry-collector-hua meta.helm.sh/release-namespace: kube-system labels: app.kubernetes.io/instance: opentelemetry-collectoicr-hua app.kubernetes.io/name: opentelemetry-collector-hua name: opentelemetry-collector-hua namespace: kube-system # 安装opentelemetry # kubectl apply -f opentelemetry-install.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/instance: opentelemetry-collector-hua app.kubernetes.io/name: opentelemetry-collector-hua name: opentelemetry-collector-hua namespace: kube-system spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/instance: opentelemetry-collector-hua app.kubernetes.io/name: opentelemetry-collector-hua strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app.kubernetes.io/instance: opentelemetry-collector-hua app.kubernetes.io/name: opentelemetry-collector-hua spec: containers: - command: - /otelcol - --config=/conf/relay.yaml - --metrics-addr=0.0.0.0:8888 - --mem-ballast-size-mib=819 env: - name: MY_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP image: images.huazai.com/release/opentelemetry-collector:0.27.0 imagePullPolicy: IfNotPresent livenessProbe:&nbss p; failureThreshold: 3 httpGet: path: / port: 13133 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: opentelemetry-collector-hua ports: - containerPort: 4317 name: otlp protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: / port: 13133 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "1" memory: 2Gi requests: cpu: 500m memory: 1Gi volumeMounts: - mountPath: /conf # 上面创建的给oepntelnemetry用的configmap name: opentelemetry-collector-configmap-hua - mountPath: /etc/otel-collector/secrets/etcd-cert/ &nbsinp; name: etcd-tls readOnly: true dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} # sa这里自行创建吧 serviceAccount: opentelemetry-collector-hua serviceAccountName: opentelemetry-collector-hua terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 420 items: - key: relay path: relay.yaml # 上面创建的给oepntelnemetry用的configmap name: opentelemetry-collector-hua name: opentelemetry-collector-configmap-hua - name: etcd-tls secret: defaultMode: 420 secretName: etcd-tls # 检查opentelemetry运行情况。如果opentelemetry与vm在同一个k8s集群,请写service那一套,不要使用lb(受制于云上 # 4层监听器的后端服务器暂不能支持同时作为客户端和服务端) [root@kube-control-1 ~]# kubectl get po -n kube-system |grep opentelemetry-collector-hua opentelemetry-collector-hua-647c6c64c7-j6p4b 1/1 Running 0 8h 安装检查
所有的组件安装完成后,在浏览器输入http://lb:8481/select/0/vmui,然后在server url输入;http://lb:8481/select/0/prometheus。最后再输入对应的指标就可以查询数据了,左上角还可以开启自动刷新!
总结
整个安装过程还是比较简单的。一旦安装完成后,即可存储多个k8s集群的监控数据。vm是支持基于PromeQL的MetricsQL的,也能够作为grafana的数据源。想想之前需要手动在每个k8s集群单独安装prometheus,还要去配置存储,需要查询数据时,要单独打开每个集群的prometheus UI是不是显得稍微麻烦一点呢。如果你也觉得vm不错,动手试试看吧!
全文参考 https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster https://docs.victoriametrics.com/ https://opentelemetry.io/docs/ https://prometheus.io/docs/prometheus/latest/configuration/configuration/
-
标签: OpenAI Sora Disney Anthropic爱立信将以62亿美元全现金收购全球云通信供应商Vonage美图秀秀即将发布全新 AI 智能助手Tilde AI 发布开源 TildeOpen LLM:支持多种欧洲语言的超大语言模型Twitter鏀惰喘鏂伴椈搴旂敤Brief寮€鍙戝洟闃�标签: 尼康 ASML 浸没式光刻机 EUV光刻机标签: SpaceX Cursor Anysphere 600亿美元上海AI Lab推出Lumina-DiMOO,开创多模态生成与理解新局面5.6万次下载冲进前三!OpenAI的Sora还没全面开放就杀疯了游戏公司MIXI与OpenAI合作,成功部署ChatGPT Enterprise
- ·苹果将重新批准社交平台Parler上架:曾因冲击国会事件遭删除
- ·团建主持人开场白
- ·通用汽车扩展氢燃料电池业务 可为工地 / 音乐节 / 数据中心供能
- ·2020年全国工业机器人出货量同比增长19.1%
- ·消息称亚马逊接近达成协议:拟90亿美元收购米高梅
- ·标签: 菜鸟 淘天集团 阿里中国电商事业群 物流科技
- ·标签: Uber Robotaxi 香港 巴拉吉·克里希纳穆尔蒂
- ·一路上有你歌曲
- ·云计算核心技术Docker教程:Docker 使用模板化配置
- ·标签: TikTok 周受资 美国数据安全合资公司 Adam Presser
- ·云原生架构日志监控优秀实践
- ·标签: 任庚 郎咸朋 具身智能 特斯拉人形机器人
- ·全面开源的 LLaVA-OneVision-1.5,超越 Qwen2.5-VL 的多模态模型登场
- ·鑻规灉楂樼�瑙h�2021璐㈠勾Q3璐㈡姤锛氫腑鍥界敤鎴峰�iPhone 12 Pro鍙嶅搷寰堢儹鐑�
- ·OpenAI加速推进GPT-6开发,奥尔特曼承诺打造"有记忆"的个性化AI助手
- ·IBM第四季度营收204亿美元 其中云计算业务营收为75亿美元
- ·标签: Vbot维他动力 秦海龙 超能机器狗 具身智能
- ·云安全日报210818:Ubuntu Linux内核发现任意代码执行漏洞,需要尽快升级
- ·标签: REDMI K90 Max 鸿蒙智行 智界 乐道L90 微信支付
- ·云计算核心技术Docker教程:Docker 守护进程dockerd运行多个守护程序
- ·前微软高管创办AI公司Maximor ,打破财务管理对 Excel 的依赖
- ·标签: 华为Mate80 HarmonyOS
- ·标签: 佰维存储 AI眼镜 Meta 智能穿戴
- ·标签: Arm Holdings 人工智能芯片 数据中心芯片
- ·这20个Pandas函数,堪称"数据清洗"杀手!
- ·腾讯提出无训练优化方法:120元成本实现传统7万元微调效果
