# Grafana Helm Chart The leading tool for querying and visualizing time series and metrics. ## Source Code * ## Requirements Kubernetes: `^1.25.0-0` ## Installing the Chart ### OCI Registry OCI registries are preferred in Helm as they implement unified storage, distribution, and improved security. ```console helm install RELEASE-NAME oci://ghcr.io/grafana-community/helm-charts/grafana ``` ### HTTP Registry ```console helm repo add grafana-community https://grafana-community.github.io/helm-charts helm repo update helm install RELEASE-NAME grafana-community/grafana ``` ## Uninstalling the Chart To remove all of the Kubernetes objects associated with the Helm chart release: ```console helm delete RELEASE-NAME ``` ## Changelog See the [changelog](https://grafana-community.github.io/helm-charts/changelog/?chart=grafana). --- ## Upgrading A major chart version change (like v1.2.3 -> v2.0.0) indicates that there is an incompatible breaking change needing manual actions. ### To 4.0.0 (And 3.12.1) This version requires Helm >= 2.12.0. ### To 5.0.0 You have to add --force to your helm upgrade command as the labels of the chart have changed. ### To 6.0.0 This version requires Helm >= 3.1.0. ### To 7.0.0 For consistency with other Helm charts, the `global.image.registry` parameter was renamed to `global.imageRegistry`. If you were not previously setting `global.image.registry`, no action is required on upgrade. If you were previously setting `global.image.registry`, you will need to instead set `global.imageRegistry`. ### To 10.0.0 Static alerting resources now support Helm templating. This means that alerting resources loaded from external files (`alerting.*.files`) are now processed by the Helm template engine. If you already use template expressions intended for Alertmanager (for example, `{{ $labels.instance }}`), these must now be escaped to avoid unintended Helm or Go template evaluation. To escape them, wrap the braces with an extra layer like this: `{{ "{{" }} $labels.instance {{ "}}" }}` This ensures the expressions are preserved for Alertmanager instead of being rendered by Helm. ### To 11.0.0 The minimum required Kubernetes version is now 1.25. All references to deprecated APIs have been removed. ## Configuration ### Example ingress with path With grafana 6.3 and above ```yaml grafana.ini: server: domain: monitoring.example.com root_url: "%(protocol)s://%(domain)s/grafana" serve_from_sub_path: true ingress: enabled: true hosts: - "monitoring.example.com" path: "/grafana" ``` ### Example of extraVolumeMounts and extraVolumes Configure additional volumes with `extraVolumes` and volume mounts with `extraVolumeMounts`. Example for `extraVolumeMounts` and corresponding `extraVolumes`: ```yaml extraVolumeMounts: - name: plugins mountPath: /var/lib/grafana/plugins subPath: configs/grafana/plugins readOnly: false - name: dashboards mountPath: /var/lib/grafana/dashboards hostPath: /usr/shared/grafana/dashboards readOnly: false extraVolumes: - name: plugins existingClaim: existing-grafana-claim - name: dashboards hostPath: /usr/shared/grafana/dashboards ``` Volumes default to `emptyDir`. Set to `persistentVolumeClaim`, `hostPath`, `csi`, or `configMap` for other types. For a `persistentVolumeClaim`, specify an existing claim name with `existingClaim`. ## Import dashboards There are a few methods to import dashboards to Grafana. Below are some examples and explanations as to how to use each method: ```yaml dashboards: default: some-dashboard: json: | { "annotations": ... # Complete json file here ... "title": "Some Dashboard", "uid": "abcd1234", "version": 1 } custom-dashboard: # This is a path to a file inside the dashboards directory inside the chart directory file: dashboards/custom-dashboard.json prometheus-stats: # Ref: https://grafana.com/dashboards/2 # title: My Custom Title # optional; when set for a downloaded dashboard (gnetId or url), overrides the title displayed in Grafana gnetId: 2 revision: 2 datasource: Prometheus loki-dashboard-quick-search: gnetId: 12019 revision: 2 datasource: - name: DS_PROMETHEUS value: Prometheus - name: DS_LOKI value: Loki local-dashboard: url: https://github.com/cloudnative-pg/grafana-dashboards/blob/main/charts/cluster/grafana-dashboard.json # redirects to: # https://raw.githubusercontent.com/cloudnative-pg/grafana-dashboards/refs/heads/main/charts/cluster/grafana-dashboard.json # default: -skf # -s - silent mode # -k - allow insecure (eg: non-TLS) connections # -f - fail fast # -L - follow HTTP redirects curlOptions: -Lf ``` ## BASE64 dashboards Dashboards could be stored on a server that does not return JSON directly and instead of it returns a base64 encoded file (e.g. Gerrit) A new parameter has been added to the URL use case so if you specify a b64content value equals to true after the URL entry a base64 decoding is applied before save the file to disk. If this entry is not set or is equals to false not decoding is applied to the file before saving it to disk. ### Gerrit use case Gerrit API for download files has the following schema: where {project-name} and {file-id} usually has '/' in their values and so they MUST be replaced by %2F so if project-name is user/repository, branch-id is master and file-id is equals to dir1/dir2/dashboard the URL value is ## Sidecar for dashboards If the parameter `sidecar.dashboards.enabled` is set, a sidecar container is deployed in the grafana pod. This container watches all configmaps (or secrets) in the cluster and filters out the ones with a label as defined in `sidecar.dashboards.label`. The files defined in those configmaps are written to a folder and accessed by grafana. Changes to the configmaps are monitored and the imported dashboards are deleted/updated. A recommendation is to use one configmap per dashboard, as a reduction of multiple dashboards inside one configmap is currently not properly mirrored in grafana. Example dashboard config: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: sample-grafana-dashboard labels: grafana_dashboard: "1" data: k8s-dashboard.json: |- [...] ``` ## Sidecar for datasources If the parameter `sidecar.datasources.enabled` is set, an init container is deployed in the grafana pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and filters out the ones with a label as defined in `sidecar.datasources.label`. The files defined in those secrets are written to a folder and accessed by grafana on startup. Using these YAML files, the data sources in grafana can be imported. Should you aim for reloading datasources in Grafana each time the config is changed, set `sidecar.datasources.skipReload: false` and adjust `sidecar.datasources.reloadURL` to `http://..svc.cluster.local/api/admin/provisioning/datasources/reload`. Secrets are recommended over configmaps for this usecase because datasources usually contain private data like usernames and passwords. Secrets are the more appropriate cluster resource to manage those. Example values to add a postgres datasource as a kubernetes secret: ```yaml apiVersion: v1 kind: Secret metadata: name: grafana-datasources labels: grafana_datasource: 'true' # default value for: sidecar.datasources.label stringData: pg-db.yaml: |- apiVersion: 1 datasources: - name: My pg db datasource type: postgres url: my-postgresql-db:5432 user: db-readonly-user secureJsonData: password: 'SUperSEcretPa$$word' jsonData: database: my_datase sslmode: 'disable' # disable/require/verify-ca/verify-full maxOpenConns: 0 # Grafana v5.4+ maxIdleConns: 2 # Grafana v5.4+ connMaxLifetime: 14400 # Grafana v5.4+ postgresVersion: 1000 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10 timescaledb: false # allow users to edit datasources from the UI. editable: false ``` Example values to add a datasource adapted from [Grafana](http://docs.grafana.org/administration/provisioning/#example-datasource-config-file): ```yaml datasources: datasources.yaml: apiVersion: 1 datasources: # name of the datasource. Required - name: Graphite # datasource type. Required type: graphite # access mode. proxy or direct (Server or Browser in the UI). Required access: proxy # org id. will default to orgId 1 if not specified orgId: 1 # url url: http://localhost:8080 # database password, if used password: # database user, if used user: # database name, if used database: # enable/disable basic auth basicAuth: # basic auth username basicAuthUser: # basic auth password basicAuthPassword: # enable/disable with credentials headers withCredentials: # mark as default datasource. Max one per org isDefault: # fields that will be converted to json and stored in json_data jsonData: graphiteVersion: "1.1" tlsAuth: true tlsAuthWithCACert: true # json object of data that will be encrypted. secureJsonData: tlsCACert: "..." tlsClientCert: "..." tlsClientKey: "..." version: 1 # allow users to edit datasources from the UI. editable: false ``` ## Sidecar for notifiers If the parameter `sidecar.notifiers.enabled` is set, an init container is deployed in the grafana pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and filters out the ones with a label as defined in `sidecar.notifiers.label`. The files defined in those secrets are written to a folder and accessed by grafana on startup. Using these YAML files, the notification channels in grafana can be imported. The secrets must be created before `helm install` so that the notifiers init container can list the secrets. Secrets are recommended over configmaps for this usecase because alert notification channels usually contain private data like SMTP usernames and passwords. Secrets are the more appropriate cluster resource to manage those. Example datasource config adapted from [Grafana](https://grafana.com/docs/grafana/latest/administration/provisioning/#alert-notification-channels): ```yaml notifiers: - name: notification-channel-1 type: slack uid: notifier1 # either org_id: 2 # or org_name: Main Org. is_default: true send_reminder: true frequency: 1h disable_resolve_message: false # See `Supported Settings` section for settings supporter for each # alert notification type. settings: recipient: 'XXX' token: 'xoxb' uploadImage: true url: https://slack.com delete_notifiers: - name: notification-channel-1 uid: notifier1 org_id: 2 - name: notification-channel-2 # default org_id: 1 ``` ## Sidecar for alerting resources If the parameter `sidecar.alerts.enabled` is set, a sidecar container is deployed in the grafana pod. This container watches all configmaps (or secrets) in the cluster (namespace defined by `sidecar.alerts.searchNamespace`) and filters out the ones with a label as defined in `sidecar.alerts.label` (default is `grafana_alert`). The files defined in those configmaps are written to a folder and accessed by grafana. Changes to the configmaps are monitored and the imported alerting resources are updated, however, deletions are a little more complicated (see below). This sidecar can be used to provision alert rules, contact points, notification policies, notification templates and mute timings as shown in [Grafana Documentation](https://grafana.com/docs/grafana/next/alerting/set-up/provision-alerting-resources/file-provisioning/). To fetch the alert config which will be provisioned, use the alert provisioning API ([Grafana Documentation](https://grafana.com/docs/grafana/next/developers/http_api/alerting_provisioning/)). You can use either JSON or YAML format. Example config for an alert rule: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: sample-grafana-alert labels: grafana_alert: "1" data: k8s-alert.yml: |- apiVersion: 1 groups: - orgId: 1 name: k8s-alert [...] ``` To delete provisioned alert rules is a two step process, you need to delete the configmap which defined the alert rule and then create a configuration which deletes the alert rule. Example deletion configuration: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: delete-sample-grafana-alert namespace: monitoring labels: grafana_alert: "1" data: delete-k8s-alert.yml: |- apiVersion: 1 deleteRules: - orgId: 1 uid: 16624780-6564-45dc-825c-8bded4ad92d3 ``` ## Statically provision alerting resources If you don't need to change alerting resources (alert rules, contact points, notification policies and notification templates) regularly you could use the `alerting` config option instead of the sidecar option above. This will grab the alerting config and apply it statically at build time for the helm file. There are two methods to statically provision alerting configuration in Grafana. Below are some examples and explanations as to how to use each method: ```yaml alerting: team1-alert-rules.yaml: file: alerting/team1/rules.yaml team2-alert-rules.yaml: file: alerting/team2/rules.yaml team3-alert-rules.yaml: file: alerting/team3/rules.yaml notification-policies.yaml: file: alerting/shared/notification-policies.yaml notification-templates.yaml: file: alerting/shared/notification-templates.yaml contactpoints.yaml: apiVersion: 1 contactPoints: - orgId: 1 name: Slack channel receivers: - uid: default-receiver type: slack settings: # Webhook URL to be filled in url: "" # We need to escape double curly braces for the tpl function. text: '{{ `{{ template "default.message" . }}` }}' title: '{{ `{{ template "default.title" . }}` }}' ``` The two possibilities for static alerting resource provisioning are: * Inlining the file contents as shown for contact points in the above example. * Importing a file using a relative path starting from the chart root directory as shown for the alert rules in the above example. ### Important notes on file provisioning * The format of the files is defined in the [Grafana documentation](https://grafana.com/docs/grafana/next/alerting/set-up/provision-alerting-resources/file-provisioning/) on file provisioning. * The chart supports importing YAML and JSON files. * The filename must be unique, otherwise one volume mount will overwrite the other. * Alerting configurations support Helm templating. Double curly braces that arise from the Grafana configuration format and are not intended as templates for the chart must be escaped. * The number of total files under `alerting:` is not limited. Each file will end up as a volume mount in the corresponding provisioning folder of the deployed Grafana instance. * The file size for each import is limited by what the function `.Files.Get` can handle, which suffices for most cases. ## How to serve Grafana with a path prefix (/grafana) In order to serve Grafana with a prefix (e.g., ), add the following to your values.yaml. ```yaml ingress: enabled: true annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/rewrite-target: /$1 nginx.ingress.kubernetes.io/use-regex: "true" path: /grafana/?(.*) hosts: - k8s.example.dev grafana.ini: server: root_url: http://localhost:3000/grafana # this host can be localhost ``` ## How to securely reference secrets in grafana.ini This example uses Grafana [file providers](https://grafana.com/docs/grafana/latest/administration/configuration/#file-provider) for secret values and the `extraSecretMounts` configuration flag (Additional grafana server secret mounts) to mount the secrets. In grafana.ini: ```yaml grafana.ini: [auth.generic_oauth] enabled = true client_id = $__file{/etc/secrets/auth_generic_oauth/client_id} client_secret = $__file{/etc/secrets/auth_generic_oauth/client_secret} ``` Existing secret, or created along with helm: ```yaml --- apiVersion: v1 kind: Secret metadata: name: auth-generic-oauth-secret type: Opaque stringData: client_id: client_secret: ``` Include in the `extraSecretMounts` configuration flag: ```yaml extraSecretMounts: - name: auth-generic-oauth-secret-mount secretName: auth-generic-oauth-secret defaultMode: 0440 mountPath: /etc/secrets/auth_generic_oauth readOnly: true ``` ### extraSecretMounts using a Container Storage Interface (CSI) provider This example uses a CSI driver e.g. retrieving secrets using [Azure Key Vault Provider](https://github.com/Azure/secrets-store-csi-driver-provider-azure) ```yaml extraSecretMounts: - name: secrets-store-inline mountPath: /run/secrets readOnly: true csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "my-provider" nodePublishSecretRef: name: akv-creds ``` ## Image Renderer Plug-In This chart supports enabling [remote image rendering](https://github.com/grafana/grafana-image-renderer/blob/master/README.md#run-in-docker) ```yaml imageRenderer: enabled: true ``` ### Image Renderer NetworkPolicy By default the image-renderer pods will have a network policy which only allows ingress traffic from the created grafana instance ### High Availability for unified alerting If you want to run Grafana in a high availability cluster you need to enable the headless service by setting `headlessService: true` in your `values.yaml` file. As next step you have to setup the `grafana.ini` in your `values.yaml` in a way that it will make use of the headless service to obtain all the IPs of the cluster. For example, use ``{{ .Release.Name }}`` to refer to the Helm release name in your values. ```yaml grafana.ini: ... unified_alerting: enabled: true ha_peers: {{ .Release.Name }}-headless:9094 ha_listen_address: ${POD_IP}:9094 ha_advertise_address: ${POD_IP}:9094 rule_version_record_limit: "5" alerting: enabled: false ``` ### Installing plugins If you want to install a Grafana plugin using the helm chart, you can do so by using the identifier of the plugin, for example `digirich-bubblechart-panel` will install [Bubble Chart](https://grafana.com/grafana/plugins/digrich-bubblechart-panel/). You can also install a plugin and a specific version by specifying the version and URL of the download file as shown in the example below : ```yaml plugins: - digrich-bubblechart-panel - grafana-clock-panel ## You can also use other plugin download URL, as long as they are valid zip files, ## and specify the name of the plugin as prefix, with an version. Like this: # - marcusolsson-json-datasource@1.3.24@https://grafana.com/api/plugins/marcusolsson-json-datasource/versions/1.3.24/download ``` Generic documentation about plugins can be found in the [official documentation](https://grafana.com/docs/grafana/latest/administration/plugin-management/).