duynguyen
c2c265c898
Drop .env/envsubst/bake-hash-into-values.yaml flow entirely. Chart's configs.secret block is removed for good, so the rendered manifest never declares admin.password/admin.passwordMtime. Password hash is set once directly on the live argocd-secret via kubectl patch, and survives every self-heal sync since ArgoCD never owns those fields.
279 lines
7.7 KiB
Markdown
279 lines
7.7 KiB
Markdown
# K8s Cluster Bootstrap — Platform Services
|
|
|
|
Bootstraps platform services onto the k8s cluster using ArgoCD app-of-apps pattern.
|
|
Run this after cluster-init finishes provisioning and configuring nodes.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Git repo (cluster-bootstrap)
|
|
└── ArgoCD watches bootstrap/apps/ → syncs all Applications
|
|
|
|
Bootstrap order (sync waves):
|
|
Wave -1 → argocd (self-managed, once bootstrapped)
|
|
Wave 0 → metallb (LoadBalancer IPs)
|
|
Wave 1 → metallb-config (IPAddressPool + L2Advertisement)
|
|
Wave 2 → envoy-gateway (HTTP gateway controller)
|
|
Wave 2 → nfs-provisioner (dynamic PVC provisioner from xpen NAS)
|
|
Wave 3 → envoy-gateway-config (GatewayClass + EnvoyProxy + Gateway)
|
|
|
|
External access:
|
|
MetalLB assigns 192.168.1.30 to Envoy Gateway LoadBalancer service
|
|
All HTTP traffic → Envoy Gateway (192.168.1.30:80) → HTTPRoutes → services
|
|
DNS: *.fireflylab.local → 192.168.1.30 (configure in your local DNS/router)
|
|
```
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
- k8s cluster running (see cluster-init repo)
|
|
- `kubectl` configured on client machine (kubeconfig at `~/.kube/config`)
|
|
- Client machine can reach `192.168.1.31` (master01)
|
|
- xpen NAS NFS export accessible from all k8s nodes
|
|
|
|
---
|
|
|
|
## Phase 1 — Install Helm
|
|
|
|
```bash
|
|
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
|
|
```
|
|
|
|
```bash
|
|
helm version
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 2 — Install tools
|
|
|
|
```bash
|
|
sudo dnf install -y httpd-tools
|
|
```
|
|
|
|
`htpasswd` is used in Phase 3.1 to generate the admin password hash. No `.env` file, no `envsubst` — the password never touches Git.
|
|
|
|
---
|
|
|
|
## Phase 3 — Install ArgoCD
|
|
|
|
Add Helm repo:
|
|
|
|
```bash
|
|
helm repo add argo https://argoproj.github.io/argo-helm
|
|
```
|
|
|
|
```bash
|
|
helm repo update
|
|
```
|
|
|
|
Install ArgoCD:
|
|
|
|
```bash
|
|
helm install argocd argo/argo-cd -n argocd --create-namespace -f manifests/argocd/values.yaml
|
|
```
|
|
|
|
`values.yaml` deliberately has no `configs.secret` block — the chart auto-generates a random admin password, stored in `argocd-initial-admin-secret`. That's replaced with your own in 3.1.
|
|
|
|
Wait for ArgoCD to be ready:
|
|
|
|
```bash
|
|
kubectl wait --for=condition=available deployment/argocd-server -n argocd --timeout=120s
|
|
```
|
|
|
|
Check pods are up:
|
|
|
|
```bash
|
|
kubectl get pods -n argocd
|
|
```
|
|
|
|
> UI login isn't needed for bootstrap (Phase 4 applies the root Application via `kubectl`). Only port-forward if you want to inspect ArgoCD manually: `kubectl port-forward svc/argocd-server -n argocd 8080:443`, then open `https://localhost:8080` with `admin` / your chosen password.
|
|
|
|
### 3.1 Set your own admin password directly on the Secret
|
|
|
|
Set it once, straight on `argocd-secret` (the Secret ArgoCD actually reads) — never in `values.yaml`, never committed to Git:
|
|
|
|
```bash
|
|
read -s -p "ArgoCD admin password: " PW; echo
|
|
HASH=$(htpasswd -nbBC 12 "" "$PW" | tr -d ':\n' | sed 's/$2y/$2a/')
|
|
kubectl patch secret argocd-secret -n argocd --type merge -p \
|
|
"{\"stringData\":{\"admin.password\":\"$HASH\",\"admin.passwordMtime\":\"$(date -u +%FT%TZ)\"}}"
|
|
unset PW HASH
|
|
```
|
|
|
|
**Why this stays stable across self-management:** `values.yaml` never declares `configs.secret.argocdServerAdminPassword`, so the Helm chart's rendered manifest never includes `admin.password`/`admin.passwordMtime`. Once ArgoCD self-manages (wave -1, `selfHeal: true`), it only reconciles fields present in its own rendered output — it has no opinion on keys it never declared, so your patched hash survives every sync, forever. If that block is ever added back to `values.yaml`, the next self-heal overwrites it — don't add it.
|
|
|
|
To rotate later: repeat the same `kubectl patch` with a new hash and a new `admin.passwordMtime` (ArgoCD only accepts the change if the Mtime also changes).
|
|
|
|
---
|
|
|
|
## Phase 4 — Apply the root bootstrap Application
|
|
|
|
`bootstrap-app.yaml` (repo root) is the one seed manifest — it lives in Git, not created ad-hoc via UI/CLI. Per GitOps practice, every Application ArgoCD runs must be recreatable from Git; a UI-created app that's stored nowhere can't survive a rebuild. This is the only manifest ever applied by hand — everything downstream (including ArgoCD managing itself, wave -1) is reached by ArgoCD syncing `bootstrap/apps/` from here.
|
|
|
|
```bash
|
|
kubectl apply -f bootstrap-app.yaml
|
|
```
|
|
|
|
ArgoCD syncs all child Applications in wave order automatically, including self-managing itself via `bootstrap/apps/argocd.yaml` (wave -1).
|
|
|
|
Monitor progress:
|
|
|
|
```bash
|
|
kubectl get applications -n argocd
|
|
```
|
|
|
|
```bash
|
|
kubectl get pods -n metallb-system
|
|
```
|
|
|
|
```bash
|
|
kubectl get pods -n envoy-gateway-system
|
|
```
|
|
|
|
```bash
|
|
kubectl get pods -n nfs-provisioner
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 5 — Verify Envoy Gateway has external IP
|
|
|
|
```bash
|
|
kubectl get svc -n envoy-gateway-system
|
|
```
|
|
|
|
`EXTERNAL-IP` should be `192.168.1.30` (assigned by MetalLB).
|
|
|
|
If it stays `<pending>`, check MetalLB:
|
|
|
|
```bash
|
|
kubectl get ipaddresspool -n metallb-system
|
|
```
|
|
|
|
```bash
|
|
kubectl get l2advertisement -n metallb-system
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 6 — Apply ArgoCD HTTPRoute
|
|
|
|
Once Envoy Gateway has the external IP, expose ArgoCD via hostname:
|
|
|
|
```bash
|
|
kubectl apply -f manifests/argocd/httproute.yaml
|
|
```
|
|
|
|
ArgoCD UI now accessible at `http://argocd.fireflylab.local` — no more port-forward needed.
|
|
|
|
---
|
|
|
|
## Phase 7 — Verify StorageClasses
|
|
|
|
```bash
|
|
kubectl get storageclass
|
|
```
|
|
|
|
Expected:
|
|
|
|
```
|
|
NAME PROVISIONER RECLAIMPOLICY
|
|
nfs-delete cluster.local/nfs-subdir-external-provisioner Delete
|
|
nfs-retain cluster.local/nfs-subdir-external-provisioner Retain
|
|
```
|
|
|
|
Test PVC provisioning:
|
|
|
|
```bash
|
|
kubectl apply -f - <<'EOF'
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: nfs-test
|
|
namespace: default
|
|
spec:
|
|
accessModes: [ReadWriteOnce]
|
|
storageClassName: nfs-delete
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
EOF
|
|
```
|
|
|
|
```bash
|
|
kubectl get pvc nfs-test
|
|
```
|
|
|
|
Status should be `Bound`. Clean up:
|
|
|
|
```bash
|
|
kubectl delete pvc nfs-test
|
|
```
|
|
|
|
---
|
|
|
|
## Full verification
|
|
|
|
```bash
|
|
kubectl get applications -n argocd
|
|
```
|
|
|
|
```bash
|
|
kubectl get pods -n metallb-system
|
|
```
|
|
|
|
```bash
|
|
kubectl get pods -n envoy-gateway-system
|
|
```
|
|
|
|
```bash
|
|
kubectl get pods -n nfs-provisioner
|
|
```
|
|
|
|
```bash
|
|
kubectl get svc -n envoy-gateway-system
|
|
```
|
|
|
|
```bash
|
|
kubectl get storageclass
|
|
```
|
|
|
|
All Applications should be `Synced` / `Healthy`.
|
|
|
|
---
|
|
|
|
## File reference
|
|
|
|
```
|
|
cluster-bootstrap/
|
|
├── README.md
|
|
├── bootstrap-app.yaml # root seed Application — the only manifest applied by hand
|
|
├── bootstrap/
|
|
│ └── apps/ # ArgoCD Application CRDs
|
|
│ ├── argocd.yaml # wave -1 — self-managed ArgoCD
|
|
│ ├── metallb.yaml # wave 0 — Helm chart
|
|
│ ├── metallb-config.yaml # wave 1 — IPAddressPool + L2Advertisement
|
|
│ ├── envoy-gateway.yaml # wave 2 — Helm chart
|
|
│ ├── nfs-provisioner.yaml # wave 2 — Helm chart
|
|
│ └── envoy-gateway-config.yaml # wave 3 — GatewayClass + EnvoyProxy + Gateway
|
|
└── manifests/
|
|
├── argocd/
|
|
│ ├── values.yaml # ArgoCD Helm values
|
|
│ └── httproute.yaml # ArgoCD HTTPRoute (applied after Envoy is up)
|
|
├── metallb/
|
|
│ └── values.yaml
|
|
├── metallb-config/
|
|
│ ├── ipaddresspool.yaml # IP pool: 192.168.1.30/32
|
|
│ └── l2advertisement.yaml
|
|
├── envoy-gateway/
|
|
│ └── values.yaml
|
|
├── envoy-gateway-config/
|
|
│ ├── gatewayclass.yaml
|
|
│ ├── envoy-proxy.yaml # DaemonSet, LoadBalancer service
|
|
│ └── gateway.yaml # HTTP :80 listener
|
|
└── nfs-provisioner/
|
|
└── values.yaml # ⚠ fill in nfs.server + nfs.path before push
|
|
```
|