Upgrade Kubernetes Monitoring
Choose the following upgrade instructions appropriate for your existing configuration of Kubernetes Monitoring.
Update Kubernetes Monitoring and Grafana Agent features
The engineering team regularly pushes updates to Kubernetes Monitoring and the Grafana Agent. When a new version becomes available, an Update button on the Settings tab appears.
Click Update to install any updates to Grafana dashboards, alerting and recording rules, and new Grafana Agent features.
If you receive an error that the has failed, refer to troubleshooting instructions.
Upgrade from Kubernetes Monitoring Helm chart
If you previously configured with the Kubernetes Monitoring Helm chart, copy and run the following to upgrade with the newer version of the Helm chart:
helm repo update
helm upgrade grafana-k8s-monitoring grafana/k8s-monitoring --namespace "${NAMESPACE}" --values <your values file>
Upgrade from static Agent configuration
If you previously deployed the Grafana Agent in static mode, complete the following steps to remove the agent and supporting systems before deploying the Kubernetes Monitoring Helm chart.
Persist custom configuration
If you customized your Grafana Agent configuration to add metric sources, log sources, relabeling rules, or any other changes, save your config file outside of your cluster.
Step 1: Export common variables
Copy and run the following to use these variables throughout the remaining steps.
# Set this to the namespace where Grafana Agent was deployed
export NAMESPACE="default"
# This will extract the installed Grafana Agent version (e.g. "v0.34.1")
export AGENT_VERSION=$(kubectl get statefulset grafana-agent -n "${NAMESPACE}" -o jsonpath='{$.spec.template.spec.containers[:1].image}' | sed -e 's/grafana\/agent:\(v[.0-9]*\.1\)/\1/')
Step 2: Remove the Grafana Agent for Metrics
Copy and run the following to remove the grafana-agent
StatefulSet and associated ConfigMap that was used for scraping metrics.
MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/${AGENT_VERSION}/production/kubernetes/agent-bare.yaml /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/${AGENT_VERSION}/production/kubernetes/install-bare.sh)" | kubectl delete -f -
kubectl delete configmap grafana-agent -n "${NAMESPACE}"
Step 3: Remove the Grafana Agent for Logs
Copy and run the following to remove the grafana-agent-logs
DaemonSet and associated ConfigMap that was used for scraping metrics.
MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/${AGENT_VERSION}/production/kubernetes/agent-loki.yaml /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent//${AGENT_VERSION}/production/kubernetes/install-bare.sh)" | kubectl delete -f -
kubectl delete configmap grafana-agent-logs -n "${NAMESPACE}"
Step 4: Remove supporting metric systems
Copy and run the following to be redeployed with the Kubernetes Monitoring Helm chart:
helm delete ksm -n "${NAMESPACE}"
helm delete nodeexporter -n "${NAMESPACE}"
helm delete opencost -n "${NAMESPACE}"
Upgrade from Agent Operator configuration
If you previously deployed the Grafana Agent in Operator mode, complete the following to clean up before deploying the Kubernetes Monitoring Helm chart.
Persist a custom configuration
If you customized your Agent configuration with additional Integration objects, refer to the Agent Flow mode documentation for the Flow-equivalent components and add them to the Kubernetes Monitoring Helm chart’s .extraConfig
value.
If you are using additional PodMonitor or ServiceMonitor objects, no change is necessary.
The Grafana Agent deployed by the Kubernetes Monitoring Helm chart will still detect and utilize those objects.
Step 1: Export common variables
Copy and run the following to use this variable throughout the remaining steps.
# Set this to the namespace where Grafana Agent was deployed
export NAMESPACE="default"
Step 2: Remove monitoring objects
Next, you must remove all the objects deployed during the Agent Operator deployment process:
Kind | Name |
---|---|
ServiceAccount | grafana-agent |
ClusterRole | grafana-agent |
ClusterRoleBinding | grafana-agent |
GrafanaAgent | grafana-agent |
Secret | metrics-secret |
Integration | node-exporter |
MetricsInstance | grafana-agent-metrics |
ServiceMonitor | cadvisor-monitor |
ServiceMonitor | kubelet-monitor |
ServiceMonitor | kube-state-metrics |
ClusterRole | kube-state-metrics |
ClusterRoleBinding | kube-state-metrics |
Service | kube-state-metrics |
Deployment | kube-state-metrics |
ServiceMonitor | ksm-monitor |
Secret | logs-secret |
LogsInstance | grafana-agent-logs |
PodLogs | kubernetes-logs |
PersistentVolumeClaim | agent-eventhandler |
Integration | agent-eventhandler |
ServiceAccount | opencost |
ClusterRole | opencost |
ClusterRoleBinding | opencost |
Service | opencost |
Deployment | opencost |
ServiceMonitor | opencost |
Copy and run the following:
kubectl delete -n "${NAMESPACE}" \
serviceaccount/grafana-agent \
clusterrole/grafana-agent \
clusterrolebinding/grafana-agent \
grafanaagent/grafana-agent \
secret/metrics-secret \
integration/node-exporter \
metricsinstance/grafana-agent-metrics \
servicemonitor/cadvisor-monitor \
servicemonitor/kubelet-monitor \
serviceaccount/kube-state-metrics \
clusterrole/kube-state-metrics \
clusterrolebinding/kube-state-metrics \
service/kube-state-metrics \
deployment/kube-state-metrics \
servicemonitor/ksm-monitor \
secret/logs-secret \
logsinstance/grafana-agent-logs \
podlogs/kubernetes-logs \
persistentvolumeclaim/agent-eventhandler \
integration/agent-eventhandler \
serviceaccount/opencost \
secret/opencost \
clusterrole/opencost \
clusterrolebinding/opencost \
service/opencost \
deployment/opencost \
servicemonitor/opencost \
Step 3: Remove the Grafana Agent Operator and associated CRDs
Configured with Helm
If you previously deployed the Grafana Agent operator with Helm, copy and run the following remove the Grafana Agent Operator along with the associated Operator CRDs:
helm delete -n "${NAMESPACE}" grafana-agent-operator
Configured without Helm
If you deployed without Helm, you must manually remove the Grafana Agent Operator and the associated Operator CRDs:
Kind | Name |
---|---|
ServiceAccount | grafana-agent-operator |
ClusterRole | grafana-agent-operator |
ClusterRoleBinding | grafana-agent-operator |
Deployment | grafana-agent-operator |
| |
Copy and run the following to do so: |
kubectl delete -n "${NAMESPACE}" \
deployment/grafana-agent-operator serviceaccount/grafana-agent-operator \
clusterrole/grafana-agent-operator clusterrolebinding/grafana-agent-operator
kubectl delete
-f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.coreos.com_podmonitors.yaml \
-f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.coreos.com_probes.yaml \
-f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.coreos.com_servicemonitors.yaml \
-f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.grafana.com_grafanaagents.yaml \
-f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.grafana.com_integrations.yaml \
-f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.grafana.com_logsinstances.yaml \
-f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.grafana.com_metricsinstances.yaml \
-f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.grafana.com_podlogs.yaml
Upgrade to use cost monitoring
If you have already deployed Kubernetes Monitoring using Agent or Agent Operator, follow the instructions on this page to upgrade from Agent or from Agent Operator.
Create an access policy token
Create an Access Policy Token that has the scope of read metrics
.
OpenCost needs the ability to read metrics from your hosted Prometheus.
Deploy OpenCost
Deploy OpenCost via its Helm chart by copying this code into your terminal and running it:
helm repo add opencost https://opencost.github.io/opencost-helm-chart &&
helm repo update && \
helm install opencost opencost/opencost -n "default" -f - <<EOF
fullnameOverride: opencost
opencost:
exporter:
defaultClusterId: "REPLACE_WITH_CLUSTER_NAME"
extraEnv:
CLOUD_PROVIDER_API_KEY: AIzaSyD29bGxmHAVEOBYtgd8sYM2gM2ekfxQX4U
EMIT_KSM_V1_METRICS: "false"
EMIT_KSM_V1_METRICS_ONLY: "true"
PROM_CLUSTER_ID_LABEL: cluster
prometheus:
password: "REPLACE_WITH_ACCESS_POLICY_TOKEN"
username: "REPLACE_WITH_GRAFANA_CLOUD_PROMETHEUS_USER_ID"
external:
enabled: true
url: "REPLACE_WITH_GRAFANA_CLOUD_PROMETHEUS_QUERY_ENDPOINT"
internal: { enabled: false }
ui: { enabled: false }
EOF
CLOUD_PROVIDER_API_KEY
is supplied for evaluation. For example, the GCP pricing API requires a key.password
is the access policy token that was just created.username
is the Prometheus user ID (which is a number).url
is the Prometheus query endpoint.
Related resources from Grafana Labs


