cluster-platform/README.md

# K8s Cluster Platform — Services

Bootstraps platform/application services (vault, kube-prometheus-stack, harbor,
jenkins, sonarqube, headlamp) onto the cluster using ArgoCD app-of-apps pattern.

Run this after `cluster-bootstrap` finishes (ArgoCD, MetalLB, Envoy Gateway, NFS
storage must already be up).

## Architecture

```
Git repo (cluster-platform)
  └── ArgoCD watches platform/apps/*/application.yaml (recursive) → syncs all Applications

platform-app.yaml           ← root "main" Application, applied once by hand (kubectl apply)
platform/apps/<service>/    ← one self-contained folder per service:
  application.yaml            ArgoCD Application CRD (multi-source: chart + values + extras)
  values.yaml                  Helm values for the upstream chart
  *.yaml                       any extra raw manifests (HTTPRoute, RBAC, ClusterSecretStore)
```

Each service's `application.yaml` is a single multi-source Application:
upstream Helm chart + this repo's `values.yaml` (via `ref: values`) + a third
source pointing at the same folder (excluding `application.yaml`/`values.yaml`)
for any extra raw manifests. The root `platform-app.yaml` only watches
`platform/apps/*/application.yaml` (`directory.recurse: true` + `include`
filter) — it never touches `values.yaml` or the extras directly.

Ordering uses two independent layers:
- **Application-level** `sync-wave` (on `application.yaml`'s `metadata`) —
  orders services relative to each other.
- **Resource-level** `sync-wave` (on the extra manifests themselves, e.g.
  `httproute.yaml`) — orders a service's own extras (wave `"1"`) after its
  Helm chart's resources (implicit wave `"0"`), within the same Application.

| Wave | Service | Purpose |
|------|---------|---------|
| 0 | vault | Secrets engine (standalone, manual init/unseal) (+ HTTPRoute, wave 1 internally) |
| 0 | headlamp | K8s dashboard (+ RBAC for login token, wave 1 internally) |
| 1 | external-secrets | Vault → K8s Secret operator (+ ClusterSecretStore, wave 1 internally) |
| 1 | kube-prometheus-stack | Prometheus + Grafana + Alertmanager (+ HTTPRoute, wave 1 internally) |
| 1 | harbor | Image registry (+ HTTPRoute, wave 1 internally) |
| 2 | jenkins | CI (+ HTTPRoute, wave 1 internally) |
| 2 | sonarqube | Code quality, embedded H2 (+ HTTPRoute, wave 1 internally) |

⚠️ Chart `targetRevision` pins in each `application.yaml` are best-effort and
marked `TODO: verify latest` — this session had no live access to the Helm
repos to confirm current versions. Run `helm repo add <name> <url> && helm
search repo <name>/<chart> --versions` before or after first sync and bump
if a pin doesn't resolve.

## Bootstrap Sequence

**Apply the root platform Application**

```bash
kubectl apply -f platform-app.yaml
```

Everything else is reached by ArgoCD syncing `platform/apps/` from here.

### Post-sync manual steps

**Vault — initialize + unseal** (standalone mode, not auto-unseal):

```bash
kubectl exec -n vault vault-0 -- vault operator init -key-shares=1 -key-threshold=1
# save the unseal key + root token shown, then:
kubectl exec -n vault vault-0 -- vault operator unseal <unseal-key>
```

**Wire Vault up for external-secrets** (kv-v2 mount + kubernetes auth method —
this is Vault-internal config, not a k8s resource, so it can't go through
ArgoCD; do it once after unseal):

```bash
kubectl exec -it -n vault vault-0 -- sh
export VAULT_TOKEN=<root-token-from-init>

vault secrets enable -path=kv kv-v2
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc"

vault policy write external-secrets - <<EOF
path "kv/data/*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/external-secrets \
  bound_service_account_names=external-secrets-vault-auth \
  bound_service_account_namespaces=external-secrets \
  policies=external-secrets \
  ttl=1h
```

Once this is done, the `ClusterSecretStore` (`vault-backend`, part of the
`external-secrets` Application) should show `Valid` — check with
`kubectl get clustersecretstore vault-backend -o yaml`.

Per-service `ExternalSecret` resources (harbor-credentials, gitea-credentials,
sonarqube-token, Jenkins creds) aren't created yet — that's a follow-up once
you've actually put those secrets into Vault under `kv/`.

**Grafana admin password** (chart auto-generates, never in Git):

```bash
kubectl get secret -n monitoring kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo
```

**Jenkins admin password** (chart auto-generates, never in Git):

```bash
kubectl exec -n jenkins deploy/jenkins -c jenkins -- cat /run/secrets/additional/chart-admin-password; echo
```

**Harbor admin password** — chart ships a default (`Harbor12345`). Log in to
`http://harbor.fireflylab.local` and rotate it immediately; Harbor's password
is set inside its own database on first boot, so it cannot be swapped via a
`kubectl patch` the way ArgoCD's can.

**Grafana → actual-http-api datasource** (Infinity plugin, queries
`homelab-docker-compose-prod/actual-http-api`'s ActualQL `run-query` endpoint
directly — see `homelab-services` README for why the Prometheus exporter
alone can't answer arbitrary group-by/amount questions). The chart install
just gets the plugin onto the pod; the datasource itself is added by hand in
the Grafana UI (not provisioned/GitOps) so the API key never needs its own
K8s Secret:

1. Grafana → **Connections → Data sources → Add data source** → search
   **Infinity** → select it.
2. URL: `http://192.168.1.41:8003`.
3. **Auth** → enable **Custom HTTP Headers** → add header
   `x-api-key` = `<actual-http-api API_KEY, same value as its compose
   file>`. Grafana stores this encrypted in its own DB (uses
   `kube-prometheus-stack-grafana`'s secret key), not in git.
4. **Save & test**.

Datasource then shows up in Grafana as whatever name you gave it — build a panel
with method `POST`, URL path `/budgets/<sync-id>/run-query`, and an
`ActualQLquery` JSON body, e.g. sum of $ per category:

```json
{
  "ActualQLquery": {
    "table": "transactions",
    "groupBy": ["category"],
    "select": ["category", { "total": { "$sum": "amount" } }]
  }
}
```

Keep the panel's refresh interval long (≥1m) — each call is a real login
against `actual_server`, same rate-limit risk hit by the exporter earlier.

**Headlamp login token** (ServiceAccount created by `headlamp/rbac.yaml`):

```bash
kubectl create token headlamp-admin -n headlamp
```
Paste the token into the Headlamp UI login screen.

### Apply HTTPRoutes note

Each service creates its own HTTPRoute as part of the same Application
(unlike `cluster-bootstrap`'s ArgoCD route, which had to be applied by hand to
avoid a chicken-and-egg problem before Envoy existed) — Envoy Gateway is
already up by the time this repo syncs, so these are fully GitOps/auto-synced.

Backend service names in each `httproute.yaml` are best-effort based on each
chart's naming convention and marked with a `verify with: kubectl get svc`
comment — confirm and adjust if a route doesn't resolve.

## Domains (add to local DNS, all → 192.168.1.30)

| Service | Hostname |
|---------|----------|
| Vault | vault.fireflylab.local |
| Grafana | grafana.fireflylab.local |
| Harbor | harbor.fireflylab.local |
| Jenkins | jenkins.fireflylab.local |
| SonarQube | sonarqube.fireflylab.local |

Headlamp has no HTTPRoute — accessed via `kubectl port-forward` until/unless
you add a route for it.

⚠️ Vault's UI/API is now reachable externally via Envoy Gateway (HTTP, no TLS,
same as every other service here). Since Vault holds secrets, consider whether
that's acceptable for your threat model versus keeping it `kubectl
port-forward`/internal-only.