duynguyen/cluster-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add vault, external-secrets, monitoring, harbor, jenkins, sonarqube, headlamp

Browse Source 
Two-source ArgoCD Applications (upstream chart + values from this repo),
same pattern as cluster-bootstrap. Each service paired with a *-config
Application for HTTPRoute/RBAC/ClusterSecretStore that must exist
after the main install (separate sync wave).

Chart targetRevision pins and HTTPRoute backendRef service names are
best-effort - marked TODO/verify in-file since no live helm repo access
this session to confirm current versions or actual rendered svc names.
This commit is contained in:
duynguyen
2026-07-29 23:49:59 +07:00
parent b3c9539eaf
commit a1cff1dfb9
28 changed files with 674 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+119 -3
View File
@@ -1,7 +1,7 @@
# K8s Cluster Platform — Services
Bootstraps platform/application services (vault, external-secrets, harbor, jenkins,
sonarqube, etc.) onto the cluster using ArgoCD app-of-apps pattern.
Bootstraps platform/application services (vault, kube-prometheus-stack, harbor,
jenkins, sonarqube, headlamp) onto the cluster using ArgoCD app-of-apps pattern.
Run this after `cluster-bootstrap` finishes (ArgoCD, MetalLB, Envoy Gateway, NFS
storage must already be up).
@@ -14,9 +14,34 @@ Git repo (cluster-platform)
platform-app.yaml ← root Application, applied once by hand (kubectl apply)
platform/apps/ ← ArgoCD watches this path; one Application CRD per service
manifests/ ← Helm values referenced by those Applications
manifests/ ← Helm values + raw manifests referenced by those Applications
```
Sync waves (no hard interdependencies between services yet — grouped for
readability / staggered rollout):
| Wave | Service | Purpose |
|------|---------|---------|
| 0 | vault | Secrets engine (standalone, manual init/unseal) |
| 0 | headlamp | K8s dashboard |
| 1 | headlamp-config | ClusterRoleBinding for login token |
| 1 | external-secrets | Vault → K8s Secret operator |
| 1 | kube-prometheus-stack | Prometheus + Grafana + Alertmanager |
| 1 | harbor | Image registry |
| 2 | external-secrets-config | ClusterSecretStore wired to Vault (k8s auth) |
| 2 | kube-prometheus-stack-config | Grafana HTTPRoute |
| 2 | harbor-config | Harbor HTTPRoute |
| 2 | jenkins | CI |
| 2 | sonarqube | Code quality (embedded H2, no external Postgres) |
| 3 | jenkins-config | Jenkins HTTPRoute |
| 3 | sonarqube-config | SonarQube HTTPRoute |
⚠️ Chart `targetRevision` pins in `platform/apps/*.yaml` are best-effort and
marked `TODO: verify latest` — this session had no live access to the Helm
repos to confirm current versions. Run `helm repo add <name> <url> && helm
search repo <name>/<chart> --versions` before or after first sync and bump
if a pin doesn't resolve.
## Bootstrap Sequence
**Apply the root platform Application**
@@ -26,3 +51,94 @@ kubectl apply -f platform-app.yaml
```
Everything else is reached by ArgoCD syncing `platform/apps/` from here.
### Post-sync manual steps
**Vault — initialize + unseal** (standalone mode, not auto-unseal):
```bash
kubectl exec -n vault vault-0 -- vault operator init -key-shares=1 -key-threshold=1
# save the unseal key + root token shown, then:
kubectl exec -n vault vault-0 -- vault operator unseal <unseal-key>
```
**Wire Vault up for external-secrets** (kv-v2 mount + kubernetes auth method —
this is Vault-internal config, not a k8s resource, so it can't go through
ArgoCD; do it once after unseal):
```bash
kubectl exec -it -n vault vault-0 -- sh
export VAULT_TOKEN=<root-token-from-init>
vault secrets enable -path=kv kv-v2
vault auth enable kubernetes
vault write auth/kubernetes/config \
kubernetes_host="https://kubernetes.default.svc"
vault policy write external-secrets - <<EOF
path "kv/data/*" {
capabilities = ["read"]
}
EOF
vault write auth/kubernetes/role/external-secrets \
bound_service_account_names=external-secrets-vault-auth \
bound_service_account_namespaces=external-secrets \
policies=external-secrets \
ttl=1h
```
Once this is done, `external-secrets-config`'s `ClusterSecretStore` (`vault-backend`)
should show `Valid` — check with `kubectl get clustersecretstore vault-backend -o yaml`.
Per-service `ExternalSecret` resources (harbor-credentials, gitea-credentials,
sonarqube-token, Jenkins creds) aren't created yet — that's a follow-up once
you've actually put those secrets into Vault under `kv/`.
**Grafana admin password** (chart auto-generates, never in Git):
```bash
kubectl get secret -n monitoring kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo
```
**Jenkins admin password** (chart auto-generates, never in Git):
```bash
kubectl exec -n jenkins deploy/jenkins -c jenkins -- cat /run/secrets/additional/chart-admin-password; echo
```
**Harbor admin password** — chart ships a default (`Harbor12345`). Log in to
`http://harbor.fireflylab.local` and rotate it immediately; Harbor's password
is set inside its own database on first boot, so it cannot be swapped via a
`kubectl patch` the way ArgoCD's can.
**Headlamp login token** (ServiceAccount created by `headlamp-config`):
```bash
kubectl create token headlamp-admin -n headlamp
```
Paste the token into the Headlamp UI login screen.
### Apply HTTPRoutes note
Each `*-config` Application creates its own HTTPRoute (unlike
`cluster-bootstrap`'s ArgoCD route, which had to be applied by hand to avoid
a chicken-and-egg problem before Envoy existed) — Envoy Gateway is already up
by the time this repo syncs, so these are fully GitOps/auto-synced.
Backend service names in each `httproute.yaml` are best-effort based on each
chart's naming convention and marked with a `verify with: kubectl get svc`
comment — confirm and adjust if a route doesn't resolve.
## Domains (add to local DNS, all → 192.168.1.30)
| Service | Hostname |
|---------|----------|
| Grafana | grafana.fireflylab.local |
| Harbor | harbor.fireflylab.local |
| Jenkins | jenkins.fireflylab.local |
| SonarQube | sonarqube.fireflylab.local |
Vault and Headlamp have no HTTPRoute — Vault stays internal-only
(`vault.vault.svc.cluster.local:8200`); Headlamp is accessed via
`kubectl port-forward` until/unless you add a route for it.
manifests/external-secrets-config/clustersecretstore.yaml
+17
View File
@@ -0,0 +1,17 @@
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "http://vault.vault.svc.cluster.local:8200"
path: kv
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: external-secrets
serviceAccountRef:
name: external-secrets-vault-auth
namespace: external-secrets
manifests/external-secrets-config/serviceaccount.yaml
+5
View File
@@ -0,0 +1,5 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-secrets-vault-auth
namespace: external-secrets
manifests/external-secrets/values.yaml
+1
View File
@@ -0,0 +1 @@
installCRDs: true
manifests/harbor-config/httproute.yaml
+20
View File
@@ -0,0 +1,20 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: harbor
namespace: harbor
spec:
parentRefs:
- name: envoy-gateway
namespace: envoy-gateway-system
hostnames:
- "harbor.fireflylab.local"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
# verify with: kubectl get svc -n harbor (nginx frontend, combines portal+core+registry)
- name: harbor-nginx
port: 80
manifests/harbor/values.yaml
+30
View File
@@ -0,0 +1,30 @@
expose:
type: clusterIP
tls:
enabled: false
externalURL: http://harbor.fireflylab.local
persistence:
imageChartStorage:
disableRedirect: true
type: filesystem
filesystem:
rootdirectory: /storage
persistentVolumeClaim:
registry:
storageClass: nfs-delete
size: 50Gi
jobservice:
jobLog:
storageClass: nfs-delete
size: 5Gi
database:
storageClass: nfs-delete
size: 5Gi
redis:
storageClass: nfs-delete
size: 2Gi
trivy:
storageClass: nfs-delete
size: 5Gi
manifests/headlamp-config/rbac.yaml
+18
View File
@@ -0,0 +1,18 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: headlamp-admin
namespace: headlamp
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: headlamp-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: headlamp-admin
namespace: headlamp
manifests/headlamp/values.yaml
+2
View File
@@ -0,0 +1,2 @@
service:
type: ClusterIP
manifests/jenkins-config/httproute.yaml
+20
View File
@@ -0,0 +1,20 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: jenkins
namespace: jenkins
spec:
parentRefs:
- name: envoy-gateway
namespace: envoy-gateway-system
hostnames:
- "jenkins.fireflylab.local"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
# verify with: kubectl get svc -n jenkins
- name: jenkins
port: 8080
manifests/jenkins/values.yaml
+14
View File
@@ -0,0 +1,14 @@
controller:
serviceType: ClusterIP
persistence:
storageClass: nfs-delete
size: 20Gi
resources:
requests:
cpu: 250m
memory: 1Gi
limits:
memory: 2Gi
agent:
enabled: true
manifests/kube-prometheus-stack-config/httproute.yaml
+20
View File
@@ -0,0 +1,20 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: grafana
namespace: monitoring
spec:
parentRefs:
- name: envoy-gateway
namespace: envoy-gateway-system
hostnames:
- "grafana.fireflylab.local"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
# verify with: kubectl get svc -n monitoring
- name: kube-prometheus-stack-grafana
port: 80
manifests/kube-prometheus-stack/values.yaml
+26
View File
@@ -0,0 +1,26 @@
grafana:
persistence:
enabled: true
storageClassName: nfs-delete
size: 5Gi
prometheus:
prometheusSpec:
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: nfs-delete
resources:
requests:
storage: 20Gi
retention: 15d
alertmanager:
alertmanagerSpec:
storage:
volumeClaimTemplate:
spec:
storageClassName: nfs-delete
resources:
requests:
storage: 2Gi
manifests/sonarqube-config/httproute.yaml
+20
View File
@@ -0,0 +1,20 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: sonarqube
namespace: sonarqube
spec:
parentRefs:
- name: envoy-gateway
namespace: envoy-gateway-system
hostnames:
- "sonarqube.fireflylab.local"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
# verify with: kubectl get svc -n sonarqube
- name: sonarqube-sonarqube
port: 9000
manifests/sonarqube/values.yaml
+17
View File
@@ -0,0 +1,17 @@
postgresql:
enabled: false
persistence:
enabled: true
storageClass: nfs-delete
size: 20Gi
service:
type: ClusterIP
resources:
requests:
cpu: 500m
memory: 2Gi
limits:
memory: 4Gi
manifests/vault/values.yaml
+15
View File
@@ -0,0 +1,15 @@
server:
dataStorage:
enabled: true
storageClass: nfs-delete
size: 10Gi
standalone:
enabled: true
ha:
enabled: false
ui:
enabled: true
injector:
enabled: true
platform/apps/external-secrets-config.yaml
+22
View File
@@ -0,0 +1,22 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: external-secrets-config
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "2"
spec:
project: default
source:
repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
path: manifests/external-secrets-config
destination:
server: https://kubernetes.default.svc
namespace: external-secrets
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- SkipDryRunOnMissingResource=true
platform/apps/external-secrets.yaml
+29
View File
@@ -0,0 +1,29 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: external-secrets
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
project: default
sources:
- repoURL: https://charts.external-secrets.io
chart: external-secrets
targetRevision: "0.10.0" # TODO: verify latest via `helm search repo external-secrets/external-secrets --versions`
helm:
valueFiles:
- $values/manifests/external-secrets/values.yaml
- repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: external-secrets
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
platform/apps/harbor-config.yaml
+22
View File
@@ -0,0 +1,22 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: harbor-config
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "2"
spec:
project: default
source:
repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
path: manifests/harbor-config
destination:
server: https://kubernetes.default.svc
namespace: harbor
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- SkipDryRunOnMissingResource=true
platform/apps/harbor.yaml
+28
View File
@@ -0,0 +1,28 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: harbor
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
project: default
sources:
- repoURL: https://helm.goharbor.io
chart: harbor
targetRevision: "1.16.0" # TODO: verify latest via `helm search repo harbor/harbor --versions`
helm:
valueFiles:
- $values/manifests/harbor/values.yaml
- repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: harbor
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
platform/apps/headlamp-config.yaml
+22
View File
@@ -0,0 +1,22 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: headlamp-config
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
project: default
source:
repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
path: manifests/headlamp-config
destination:
server: https://kubernetes.default.svc
namespace: headlamp
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- SkipDryRunOnMissingResource=true
platform/apps/headlamp.yaml
+28
View File
@@ -0,0 +1,28 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: headlamp
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
project: default
sources:
- repoURL: https://kubernetes-sigs.github.io/headlamp/
chart: headlamp
targetRevision: "0.31.0" # TODO: verify latest via `helm search repo headlamp/headlamp --versions`
helm:
valueFiles:
- $values/manifests/headlamp/values.yaml
- repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: headlamp
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
platform/apps/jenkins-config.yaml
+22
View File
@@ -0,0 +1,22 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: jenkins-config
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "3"
spec:
project: default
source:
repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
path: manifests/jenkins-config
destination:
server: https://kubernetes.default.svc
namespace: jenkins
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- SkipDryRunOnMissingResource=true
platform/apps/jenkins.yaml
+28
View File
@@ -0,0 +1,28 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: jenkins
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "2"
spec:
project: default
sources:
- repoURL: https://charts.jenkins.io
chart: jenkins
targetRevision: "5.8.0" # TODO: verify latest via `helm search repo jenkins/jenkins --versions`
helm:
valueFiles:
- $values/manifests/jenkins/values.yaml
- repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: jenkins
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
platform/apps/kube-prometheus-stack-config.yaml
+22
View File
@@ -0,0 +1,22 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kube-prometheus-stack-config
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "2"
spec:
project: default
source:
repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
path: manifests/kube-prometheus-stack-config
destination:
server: https://kubernetes.default.svc
namespace: monitoring
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- SkipDryRunOnMissingResource=true
platform/apps/kube-prometheus-stack.yaml
+29
View File
@@ -0,0 +1,29 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kube-prometheus-stack
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
project: default
sources:
- repoURL: https://prometheus-community.github.io/helm-charts
chart: kube-prometheus-stack
targetRevision: "65.0.0" # TODO: verify latest via `helm search repo prometheus-community/kube-prometheus-stack --versions`
helm:
valueFiles:
- $values/manifests/kube-prometheus-stack/values.yaml
- repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: monitoring
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
platform/apps/sonarqube-config.yaml
+22
View File
@@ -0,0 +1,22 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sonarqube-config
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "3"
spec:
project: default
source:
repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
path: manifests/sonarqube-config
destination:
server: https://kubernetes.default.svc
namespace: sonarqube
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- SkipDryRunOnMissingResource=true
platform/apps/sonarqube.yaml
+28
View File
@@ -0,0 +1,28 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sonarqube
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "2"
spec:
project: default
sources:
- repoURL: https://SonarSource.github.io/helm-chart-sonarqube
chart: sonarqube
targetRevision: "10.6.0" # TODO: verify latest via `helm search repo sonarqube/sonarqube --versions`
helm:
valueFiles:
- $values/manifests/sonarqube/values.yaml
- repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: sonarqube
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
platform/apps/vault.yaml
+28
View File
@@ -0,0 +1,28 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: vault
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
project: default
sources:
- repoURL: https://helm.releases.hashicorp.com
chart: vault
targetRevision: "0.30.0" # TODO: verify latest via `helm search repo hashicorp/vault --versions`
helm:
valueFiles:
- $values/manifests/vault/values.yaml
- repoURL: https://gitea.fireflylab.cc/duynguyen/cluster-platform.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: vault
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true