K8s Cluster Platform — Services

Bootstraps platform/application services (vault, kube-prometheus-stack, harbor, jenkins, sonarqube, headlamp) onto the cluster using ArgoCD app-of-apps pattern.

Run this after cluster-bootstrap finishes (ArgoCD, MetalLB, Envoy Gateway, NFS storage must already be up).

Architecture

Git repo (cluster-platform)
  └── ArgoCD watches platform/apps/*/application.yaml (recursive) → syncs all Applications

platform-app.yaml           ← root "main" Application, applied once by hand (kubectl apply)
platform/apps/<service>/    ← one self-contained folder per service:
  application.yaml            ArgoCD Application CRD (multi-source: chart + values + extras)
  values.yaml                  Helm values for the upstream chart
  *.yaml                       any extra raw manifests (HTTPRoute, RBAC, ClusterSecretStore)

Each service's application.yaml is a single multi-source Application: upstream Helm chart + this repo's values.yaml (via ref: values) + a third source pointing at the same folder (excluding application.yaml/values.yaml) for any extra raw manifests. The root platform-app.yaml only watches platform/apps/*/application.yaml (directory.recurse: true + include filter) — it never touches values.yaml or the extras directly.

Ordering uses two independent layers:

• Application-level sync-wave (on application.yaml's metadata) — orders services relative to each other.
• Resource-level sync-wave (on the extra manifests themselves, e.g. httproute.yaml) — orders a service's own extras (wave "1") after its Helm chart's resources (implicit wave "0"), within the same Application.

Wave	Service	Purpose
0	vault	Secrets engine (standalone, manual init/unseal) (+ HTTPRoute, wave 1 internally)
0	headlamp	K8s dashboard (+ RBAC for login token, wave 1 internally)
1	external-secrets	Vault → K8s Secret operator (+ ClusterSecretStore, wave 1 internally)
1	kube-prometheus-stack	Prometheus + Grafana + Alertmanager (+ HTTPRoute, wave 1 internally)
1	harbor	Image registry (+ HTTPRoute, wave 1 internally)
2	jenkins	CI (+ HTTPRoute, wave 1 internally)
2	sonarqube	Code quality, embedded H2 (+ HTTPRoute, wave 1 internally)

⚠️ Chart targetRevision pins in each application.yaml are best-effort and marked TODO: verify latest — this session had no live access to the Helm repos to confirm current versions. Run helm repo add <name> <url> && helm search repo <name>/<chart> --versions before or after first sync and bump if a pin doesn't resolve.

Bootstrap Sequence

Apply the root platform Application

kubectl apply -f platform-app.yaml

Everything else is reached by ArgoCD syncing platform/apps/ from here.

Post-sync manual steps

Vault — initialize + unseal (standalone mode, not auto-unseal):

kubectl exec -n vault vault-0 -- vault operator init -key-shares=1 -key-threshold=1
# save the unseal key + root token shown, then:
kubectl exec -n vault vault-0 -- vault operator unseal <unseal-key>

Wire Vault up for external-secrets (kv-v2 mount + kubernetes auth method — this is Vault-internal config, not a k8s resource, so it can't go through ArgoCD; do it once after unseal):

kubectl exec -it -n vault vault-0 -- sh
export VAULT_TOKEN=<root-token-from-init>

vault secrets enable -path=kv kv-v2
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc"

vault policy write external-secrets - <<EOF
path "kv/data/*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/external-secrets \
  bound_service_account_names=external-secrets-vault-auth \
  bound_service_account_namespaces=external-secrets \
  policies=external-secrets \
  ttl=1h

Once this is done, the ClusterSecretStore (vault-backend, part of the external-secrets Application) should show Valid — check with kubectl get clustersecretstore vault-backend -o yaml.

Per-service ExternalSecret resources (harbor-credentials, gitea-credentials, sonarqube-token, Jenkins creds) aren't created yet — that's a follow-up once you've actually put those secrets into Vault under kv/.

Grafana admin password (chart auto-generates, never in Git):

kubectl get secret -n monitoring kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo

Jenkins admin password (chart auto-generates, never in Git):

kubectl exec -n jenkins deploy/jenkins -c jenkins -- cat /run/secrets/additional/chart-admin-password; echo

Harbor admin password — chart ships a default (Harbor12345). Log in to http://harbor.fireflylab.local and rotate it immediately; Harbor's password is set inside its own database on first boot, so it cannot be swapped via a kubectl patch the way ArgoCD's can.

Grafana → actual-http-api datasource (Infinity plugin, queries homelab-docker-compose-prod/actual-http-api's ActualQL run-query endpoint directly — see homelab-services README for why the Prometheus exporter alone can't answer arbitrary group-by/amount questions). The chart install just gets the plugin onto the pod; the datasource itself is added by hand in the Grafana UI (not provisioned/GitOps) so the API key never needs its own K8s Secret:

1. Grafana → Connections → Data sources → Add data source → search Infinity → select it.
2. URL: http://192.168.1.41:8003.
3. Auth → enable Custom HTTP Headers → add header x-api-key = <actual-http-api API_KEY, same value as its compose file>. Grafana stores this encrypted in its own DB (uses kube-prometheus-stack-grafana's secret key), not in git.
4. Save & test.

Datasource then shows up in Grafana as whatever name you gave it — build a panel with method POST, URL path /budgets/<sync-id>/run-query, and an ActualQLquery JSON body, e.g. sum of $ per category:

{
  "ActualQLquery": {
    "table": "transactions",
    "groupBy": ["category"],
    "select": ["category", { "total": { "$sum": "amount" } }]
  }
}

Keep the panel's refresh interval long (≥1m) — each call is a real login against actual_server, same rate-limit risk hit by the exporter earlier.

Headlamp login token (ServiceAccount created by headlamp/rbac.yaml):

kubectl create token headlamp-admin -n headlamp

Paste the token into the Headlamp UI login screen.

Apply HTTPRoutes note

Each service creates its own HTTPRoute as part of the same Application (unlike cluster-bootstrap's ArgoCD route, which had to be applied by hand to avoid a chicken-and-egg problem before Envoy existed) — Envoy Gateway is already up by the time this repo syncs, so these are fully GitOps/auto-synced.

Backend service names in each httproute.yaml are best-effort based on each chart's naming convention and marked with a verify with: kubectl get svc comment — confirm and adjust if a route doesn't resolve.

Domains (add to local DNS, all → 192.168.1.30)

Service	Hostname
Vault	vault.fireflylab.local
Grafana	grafana.fireflylab.local
Harbor	harbor.fireflylab.local
Jenkins	jenkins.fireflylab.local
SonarQube	sonarqube.fireflylab.local

Headlamp has no HTTPRoute — accessed via kubectl port-forward until/unless you add a route for it.

⚠️ Vault's UI/API is now reachable externally via Envoy Gateway (HTTP, no TLS, same as every other service here). Since Vault holds secrets, consider whether that's acceptable for your threat model versus keeping it kubectl port-forward/internal-only.