20 KiB
Grafana Helm Chart
The leading tool for querying and visualizing time series and metrics.
Source Code
Requirements
Kubernetes: ^1.25.0-0
Installing the Chart
OCI Registry
OCI registries are preferred in Helm as they implement unified storage, distribution, and improved security.
helm install RELEASE-NAME oci://ghcr.io/grafana-community/helm-charts/grafana
HTTP Registry
helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
helm install RELEASE-NAME grafana-community/grafana
Uninstalling the Chart
To remove all of the Kubernetes objects associated with the Helm chart release:
helm delete RELEASE-NAME
Changelog
See the changelog.
Upgrading
A major chart version change (like v1.2.3 -> v2.0.0) indicates that there is an incompatible breaking change needing manual actions.
To 4.0.0 (And 3.12.1)
This version requires Helm >= 2.12.0.
To 5.0.0
You have to add --force to your helm upgrade command as the labels of the chart have changed.
To 6.0.0
This version requires Helm >= 3.1.0.
To 7.0.0
For consistency with other Helm charts, the global.image.registry parameter was renamed
to global.imageRegistry. If you were not previously setting global.image.registry, no action
is required on upgrade. If you were previously setting global.image.registry, you will
need to instead set global.imageRegistry.
To 10.0.0
Static alerting resources now support Helm templating. This means that alerting resources loaded from external files (alerting.*.files) are now processed by the Helm template engine.
If you already use template expressions intended for Alertmanager (for example, {{ $labels.instance }}), these must now be escaped to avoid unintended Helm or Go template evaluation. To escape them, wrap the braces with an extra layer like this:
{{ "{{" }} $labels.instance {{ "}}" }}
This ensures the expressions are preserved for Alertmanager instead of being rendered by Helm.
To 11.0.0
The minimum required Kubernetes version is now 1.25. All references to deprecated APIs have been removed.
Configuration
Example ingress with path
With grafana 6.3 and above
grafana.ini:
server:
domain: monitoring.example.com
root_url: "%(protocol)s://%(domain)s/grafana"
serve_from_sub_path: true
ingress:
enabled: true
hosts:
- "monitoring.example.com"
path: "/grafana"
Example of extraVolumeMounts and extraVolumes
Configure additional volumes with extraVolumes and volume mounts with extraVolumeMounts.
Example for extraVolumeMounts and corresponding extraVolumes:
extraVolumeMounts:
- name: plugins
mountPath: /var/lib/grafana/plugins
subPath: configs/grafana/plugins
readOnly: false
- name: dashboards
mountPath: /var/lib/grafana/dashboards
hostPath: /usr/shared/grafana/dashboards
readOnly: false
extraVolumes:
- name: plugins
existingClaim: existing-grafana-claim
- name: dashboards
hostPath: /usr/shared/grafana/dashboards
Volumes default to emptyDir. Set to persistentVolumeClaim,
hostPath, csi, or configMap for other types. For a
persistentVolumeClaim, specify an existing claim name with
existingClaim.
Import dashboards
There are a few methods to import dashboards to Grafana. Below are some examples and explanations as to how to use each method:
dashboards:
default:
some-dashboard:
json: |
{
"annotations":
...
# Complete json file here
...
"title": "Some Dashboard",
"uid": "abcd1234",
"version": 1
}
custom-dashboard:
# This is a path to a file inside the dashboards directory inside the chart directory
file: dashboards/custom-dashboard.json
prometheus-stats:
# Ref: https://grafana.com/dashboards/2
# title: My Custom Title # optional; when set for a downloaded dashboard (gnetId or url), overrides the title displayed in Grafana
gnetId: 2
revision: 2
datasource: Prometheus
loki-dashboard-quick-search:
gnetId: 12019
revision: 2
datasource:
- name: DS_PROMETHEUS
value: Prometheus
- name: DS_LOKI
value: Loki
local-dashboard:
url: https://github.com/cloudnative-pg/grafana-dashboards/blob/main/charts/cluster/grafana-dashboard.json
# redirects to:
# https://raw.githubusercontent.com/cloudnative-pg/grafana-dashboards/refs/heads/main/charts/cluster/grafana-dashboard.json
# default: -skf
# -s - silent mode
# -k - allow insecure (eg: non-TLS) connections
# -f - fail fast
# -L - follow HTTP redirects
curlOptions: -Lf
BASE64 dashboards
Dashboards could be stored on a server that does not return JSON directly and instead of it returns a base64 encoded file (e.g. Gerrit) A new parameter has been added to the URL use case so if you specify a b64content value equals to true after the URL entry a base64 decoding is applied before save the file to disk. If this entry is not set or is equals to false not decoding is applied to the file before saving it to disk.
Gerrit use case
Gerrit API for download files has the following schema: https://yourgerritserver/a/{project-name}/branches/{branch-id}/files/{file-id}/content where {project-name} and {file-id} usually has '/' in their values and so they MUST be replaced by %2F so if project-name is user/repository, branch-id is master and file-id is equals to dir1/dir2/dashboard the URL value is https://yourgerritserver/a/user%2Frepo/branches/master/files/dir1%2Fdir2%2Fdashboard/content
Sidecar for dashboards
If the parameter sidecar.dashboards.enabled is set, a sidecar container is deployed in the grafana
pod. This container watches all configmaps (or secrets) in the cluster and filters out the ones with
a label as defined in sidecar.dashboards.label. The files defined in those configmaps are written
to a folder and accessed by grafana. Changes to the configmaps are monitored and the imported
dashboards are deleted/updated.
A recommendation is to use one configmap per dashboard, as a reduction of multiple dashboards inside one configmap is currently not properly mirrored in grafana.
Example dashboard config:
apiVersion: v1
kind: ConfigMap
metadata:
name: sample-grafana-dashboard
labels:
grafana_dashboard: "1"
data:
k8s-dashboard.json: |-
[...]
Sidecar for datasources
If the parameter sidecar.datasources.enabled is set, an init container is deployed in the grafana
pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and
filters out the ones with a label as defined in sidecar.datasources.label. The files defined in
those secrets are written to a folder and accessed by grafana on startup. Using these YAML files,
the data sources in grafana can be imported.
Should you aim for reloading datasources in Grafana each time the config is changed, set sidecar.datasources.skipReload: false and adjust sidecar.datasources.reloadURL to http://<svc-name>.<namespace>.svc.cluster.local/api/admin/provisioning/datasources/reload.
Secrets are recommended over configmaps for this usecase because datasources usually contain private data like usernames and passwords. Secrets are the more appropriate cluster resource to manage those.
Example values to add a postgres datasource as a kubernetes secret:
apiVersion: v1
kind: Secret
metadata:
name: grafana-datasources
labels:
grafana_datasource: 'true' # default value for: sidecar.datasources.label
stringData:
pg-db.yaml: |-
apiVersion: 1
datasources:
- name: My pg db datasource
type: postgres
url: my-postgresql-db:5432
user: db-readonly-user
secureJsonData:
password: 'SUperSEcretPa$$word'
jsonData:
database: my_datase
sslmode: 'disable' # disable/require/verify-ca/verify-full
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
postgresVersion: 1000 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10
timescaledb: false
# <bool> allow users to edit datasources from the UI.
editable: false
Example values to add a datasource adapted from Grafana:
datasources:
datasources.yaml:
apiVersion: 1
datasources:
# <string, required> name of the datasource. Required
- name: Graphite
# <string, required> datasource type. Required
type: graphite
# <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
access: proxy
# <int> org id. will default to orgId 1 if not specified
orgId: 1
# <string> url
url: http://localhost:8080
# <string> database password, if used
password:
# <string> database user, if used
user:
# <string> database name, if used
database:
# <bool> enable/disable basic auth
basicAuth:
# <string> basic auth username
basicAuthUser:
# <string> basic auth password
basicAuthPassword:
# <bool> enable/disable with credentials headers
withCredentials:
# <bool> mark as default datasource. Max one per org
isDefault:
# <map> fields that will be converted to json and stored in json_data
jsonData:
graphiteVersion: "1.1"
tlsAuth: true
tlsAuthWithCACert: true
# <string> json object of data that will be encrypted.
secureJsonData:
tlsCACert: "..."
tlsClientCert: "..."
tlsClientKey: "..."
version: 1
# <bool> allow users to edit datasources from the UI.
editable: false
Sidecar for notifiers
If the parameter sidecar.notifiers.enabled is set, an init container is deployed in the grafana
pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and
filters out the ones with a label as defined in sidecar.notifiers.label. The files defined in
those secrets are written to a folder and accessed by grafana on startup. Using these YAML files,
the notification channels in grafana can be imported. The secrets must be created before
helm install so that the notifiers init container can list the secrets.
Secrets are recommended over configmaps for this usecase because alert notification channels usually contain private data like SMTP usernames and passwords. Secrets are the more appropriate cluster resource to manage those.
Example datasource config adapted from Grafana:
notifiers:
- name: notification-channel-1
type: slack
uid: notifier1
# either
org_id: 2
# or
org_name: Main Org.
is_default: true
send_reminder: true
frequency: 1h
disable_resolve_message: false
# See `Supported Settings` section for settings supporter for each
# alert notification type.
settings:
recipient: 'XXX'
token: 'xoxb'
uploadImage: true
url: https://slack.com
delete_notifiers:
- name: notification-channel-1
uid: notifier1
org_id: 2
- name: notification-channel-2
# default org_id: 1
Sidecar for alerting resources
If the parameter sidecar.alerts.enabled is set, a sidecar container is deployed in the grafana
pod. This container watches all configmaps (or secrets) in the cluster (namespace defined by sidecar.alerts.searchNamespace) and filters out the ones with
a label as defined in sidecar.alerts.label (default is grafana_alert). The files defined in those configmaps are written
to a folder and accessed by grafana. Changes to the configmaps are monitored and the imported alerting resources are updated, however, deletions are a little more complicated (see below).
This sidecar can be used to provision alert rules, contact points, notification policies, notification templates and mute timings as shown in Grafana Documentation.
To fetch the alert config which will be provisioned, use the alert provisioning API (Grafana Documentation). You can use either JSON or YAML format.
Example config for an alert rule:
apiVersion: v1
kind: ConfigMap
metadata:
name: sample-grafana-alert
labels:
grafana_alert: "1"
data:
k8s-alert.yml: |-
apiVersion: 1
groups:
- orgId: 1
name: k8s-alert
[...]
To delete provisioned alert rules is a two step process, you need to delete the configmap which defined the alert rule and then create a configuration which deletes the alert rule.
Example deletion configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: delete-sample-grafana-alert
namespace: monitoring
labels:
grafana_alert: "1"
data:
delete-k8s-alert.yml: |-
apiVersion: 1
deleteRules:
- orgId: 1
uid: 16624780-6564-45dc-825c-8bded4ad92d3
Statically provision alerting resources
If you don't need to change alerting resources (alert rules, contact points, notification policies and notification templates) regularly you could use the alerting config option instead of the sidecar option above.
This will grab the alerting config and apply it statically at build time for the helm file.
There are two methods to statically provision alerting configuration in Grafana. Below are some examples and explanations as to how to use each method:
alerting:
team1-alert-rules.yaml:
file: alerting/team1/rules.yaml
team2-alert-rules.yaml:
file: alerting/team2/rules.yaml
team3-alert-rules.yaml:
file: alerting/team3/rules.yaml
notification-policies.yaml:
file: alerting/shared/notification-policies.yaml
notification-templates.yaml:
file: alerting/shared/notification-templates.yaml
contactpoints.yaml:
apiVersion: 1
contactPoints:
- orgId: 1
name: Slack channel
receivers:
- uid: default-receiver
type: slack
settings:
# Webhook URL to be filled in
url: ""
# We need to escape double curly braces for the tpl function.
text: '{{ `{{ template "default.message" . }}` }}'
title: '{{ `{{ template "default.title" . }}` }}'
The two possibilities for static alerting resource provisioning are:
- Inlining the file contents as shown for contact points in the above example.
- Importing a file using a relative path starting from the chart root directory as shown for the alert rules in the above example.
Important notes on file provisioning
- The format of the files is defined in the Grafana documentation on file provisioning.
- The chart supports importing YAML and JSON files.
- The filename must be unique, otherwise one volume mount will overwrite the other.
- Alerting configurations support Helm templating. Double curly braces that arise from the Grafana configuration format and are not intended as templates for the chart must be escaped.
- The number of total files under
alerting:is not limited. Each file will end up as a volume mount in the corresponding provisioning folder of the deployed Grafana instance. - The file size for each import is limited by what the function
.Files.Getcan handle, which suffices for most cases.
How to serve Grafana with a path prefix (/grafana)
In order to serve Grafana with a prefix (e.g., http://example.com/grafana), add the following to your values.yaml.
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/use-regex: "true"
path: /grafana/?(.*)
hosts:
- k8s.example.dev
grafana.ini:
server:
root_url: http://localhost:3000/grafana # this host can be localhost
How to securely reference secrets in grafana.ini
This example uses Grafana file providers for secret values and the extraSecretMounts configuration flag (Additional grafana server secret mounts) to mount the secrets.
In grafana.ini:
grafana.ini:
[auth.generic_oauth]
enabled = true
client_id = $__file{/etc/secrets/auth_generic_oauth/client_id}
client_secret = $__file{/etc/secrets/auth_generic_oauth/client_secret}
Existing secret, or created along with helm:
---
apiVersion: v1
kind: Secret
metadata:
name: auth-generic-oauth-secret
type: Opaque
stringData:
client_id: <value>
client_secret: <value>
Include in the extraSecretMounts configuration flag:
extraSecretMounts:
- name: auth-generic-oauth-secret-mount
secretName: auth-generic-oauth-secret
defaultMode: 0440
mountPath: /etc/secrets/auth_generic_oauth
readOnly: true
extraSecretMounts using a Container Storage Interface (CSI) provider
This example uses a CSI driver e.g. retrieving secrets using Azure Key Vault Provider
extraSecretMounts:
- name: secrets-store-inline
mountPath: /run/secrets
readOnly: true
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "my-provider"
nodePublishSecretRef:
name: akv-creds
Image Renderer Plug-In
This chart supports enabling remote image rendering
imageRenderer:
enabled: true
Image Renderer NetworkPolicy
By default the image-renderer pods will have a network policy which only allows ingress traffic from the created grafana instance
High Availability for unified alerting
If you want to run Grafana in a high availability cluster you need to enable
the headless service by setting headlessService: true in your values.yaml
file.
As next step you have to setup the grafana.ini in your values.yaml in a way
that it will make use of the headless service to obtain all the IPs of the
cluster. For example, use {{ .Release.Name }} to refer to the Helm release name in your values.
grafana.ini:
...
unified_alerting:
enabled: true
ha_peers: {{ .Release.Name }}-headless:9094
ha_listen_address: ${POD_IP}:9094
ha_advertise_address: ${POD_IP}:9094
rule_version_record_limit: "5"
alerting:
enabled: false
Installing plugins
If you want to install a Grafana plugin using the helm chart, you can do so by using the identifier of the plugin, for example digirich-bubblechart-panel will install Bubble Chart.
You can also install a plugin and a specific version by specifying the version and URL of the download file as shown in the example below :
plugins:
- digrich-bubblechart-panel
- grafana-clock-panel
## You can also use other plugin download URL, as long as they are valid zip files,
## and specify the name of the plugin as prefix, with an version. Like this:
# - marcusolsson-json-datasource@1.3.24@https://grafana.com/api/plugins/marcusolsson-json-datasource/versions/1.3.24/download
Generic documentation about plugins can be found in the official documentation.