K8s Cluster Bootstrap — Platform Services

Bootstraps platform services onto the k8s cluster using ArgoCD app-of-apps pattern. Run this after cluster-init finishes provisioning and configuring nodes.

Architecture

Git repo (cluster-bootstrap)
  └── ArgoCD watches bootstrap/apps/ → syncs all Applications

Bootstrap order (sync waves):
  Wave -1 → argocd          (self-managed, once bootstrapped)
  Wave 0 → metallb          (LoadBalancer IPs)
  Wave 1 → metallb-config   (IPAddressPool + L2Advertisement)
  Wave 2 → envoy-gateway    (HTTP gateway controller)
  Wave 2 → nfs-provisioner  (dynamic PVC provisioner from xpen NAS)
  Wave 3 → envoy-gateway-config (GatewayClass + EnvoyProxy + Gateway)

External access:
  MetalLB assigns 192.168.1.30 to Envoy Gateway LoadBalancer service
  All HTTP traffic → Envoy Gateway (192.168.1.30:80) → HTTPRoutes → services
  DNS: *.fireflylab.local → 192.168.1.30  (configure in your local DNS/router)

---

Prerequisites

• k8s cluster running (see cluster-init repo)
• kubectl configured on client machine (kubeconfig at ~/.kube/config)
• Client machine can reach 192.168.1.31 (master01)
• xpen NAS NFS export accessible from all k8s nodes

---

Phase 1 — Install Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

helm version

---

Phase 2 — Install tools

sudo dnf install -y httpd-tools

htpasswd generates the admin password hash used in Phase 3.1.

---

Phase 3 — Install ArgoCD

Add Helm repo:

helm repo add argo https://argoproj.github.io/argo-helm

helm repo update

Install ArgoCD (values.yaml has no configs.secret block — chart auto-generates a random admin password):

helm install argocd argo/argo-cd -n argocd --create-namespace -f manifests/argocd/values.yaml

Wait for ready:

kubectl wait --for=condition=available deployment/argocd-server -n argocd --timeout=120s

(Optional) inspect manually: kubectl port-forward svc/argocd-server -n argocd 8080:443 → https://localhost:8080. Not needed for bootstrap.

3.1 Set admin password on the Secret

Patched directly on argocd-secret, never in values.yaml/Git — keeps configs.secret absent so self-heal (wave -1) never touches it, password stays stable forever:

read -s -p "ArgoCD admin password: " PW; echo
HASH=$(htpasswd -nbBC 12 "" "$PW" | tr -d ':\n' | sed 's/$2y/$2a/')
kubectl patch secret argocd-secret -n argocd --type merge -p \
  "{\"stringData\":{\"admin.password\":\"$HASH\",\"admin.passwordMtime\":\"$(date -u +%FT%TZ)\"}}"
unset PW HASH

Rotate: repeat with new hash + new admin.passwordMtime (required for ArgoCD to accept the change).

---

Phase 4 — Apply the root bootstrap Application

bootstrap-app.yaml (repo root) is the seed manifest, committed to Git — not created via UI/CLI. Only manifest ever applied by hand:

kubectl apply -f bootstrap-app.yaml

ArgoCD syncs all child Applications in wave order automatically, including self-managing itself (bootstrap/apps/argocd.yaml, wave -1).

Monitor progress:

kubectl get applications -n argocd

kubectl get pods -n metallb-system

kubectl get pods -n envoy-gateway-system

kubectl get pods -n nfs-provisioner

---

Phase 5 — Verify Envoy Gateway has external IP

kubectl get svc -n envoy-gateway-system

EXTERNAL-IP should be 192.168.1.30 (assigned by MetalLB).

If it stays <pending>, check MetalLB:

kubectl get ipaddresspool -n metallb-system

kubectl get l2advertisement -n metallb-system

---

Phase 6 — Apply ArgoCD HTTPRoute

Once Envoy Gateway has the external IP, expose ArgoCD via hostname:

kubectl apply -f manifests/argocd/httproute.yaml

ArgoCD UI now accessible at http://argocd.fireflylab.local — no more port-forward needed.

---

Phase 7 — Verify StorageClasses

kubectl get storageclass

Expected:

NAME                   PROVISIONER                                                     RECLAIMPOLICY
nfs-delete (default)   cluster.local/nfs-provisioner-nfs-subdir-external-provisioner   Delete

Test PVC provisioning:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-test
  namespace: default
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: nfs-delete
  resources:
    requests:
      storage: 1Gi
EOF

kubectl get pvc nfs-test

Status should be Bound.

Test a pod actually writing to it:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: nfs-test-pod
  namespace: default
spec:
  containers:
    - name: writer
      image: busybox:1.36
      command: ["sh", "-c", "echo hello-from-$(hostname)-$(date -u +%FT%TZ) > /data/test.txt && sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: nfs-test
EOF
kubectl wait --for=condition=Ready pod/nfs-test-pod --timeout=60s
kubectl exec nfs-test-pod -- cat /data/test.txt

Verify data persists across pod delete/recreate (proves it's NFS-backed, not local emptyDir):

kubectl delete pod nfs-test-pod
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: nfs-test-pod2
  namespace: default
spec:
  containers:
    - name: reader
      image: busybox:1.36
      command: ["sleep", "3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: nfs-test
EOF
kubectl wait --for=condition=Ready pod/nfs-test-pod2 --timeout=60s
kubectl exec nfs-test-pod2 -- cat /data/test.txt   # same content, different pod

Clean up:

kubectl delete pod nfs-test-pod2
kubectl delete pvc nfs-test

---

Full verification

kubectl get applications -n argocd

kubectl get pods -n metallb-system

kubectl get pods -n envoy-gateway-system

kubectl get pods -n nfs-provisioner

kubectl get svc -n envoy-gateway-system

kubectl get storageclass

All Applications should be Synced / Healthy.

---

File reference

cluster-bootstrap/
├── README.md
├── bootstrap-app.yaml                 # root seed Application — the only manifest applied by hand
├── bootstrap/
│   └── apps/                         # ArgoCD Application CRDs
│       ├── argocd.yaml               # wave -1 — self-managed ArgoCD
│       ├── metallb.yaml              # wave 0 — Helm chart
│       ├── metallb-config.yaml       # wave 1 — IPAddressPool + L2Advertisement
│       ├── envoy-gateway.yaml        # wave 2 — Helm chart
│       ├── nfs-provisioner.yaml      # wave 2 — Helm chart
│       └── envoy-gateway-config.yaml # wave 3 — GatewayClass + EnvoyProxy + Gateway
└── manifests/
    ├── argocd/
    │   ├── values.yaml               # ArgoCD Helm values
    │   └── httproute.yaml            # ArgoCD HTTPRoute (applied after Envoy is up)
    ├── metallb/
    │   └── values.yaml
    ├── metallb-config/
    │   ├── ipaddresspool.yaml        # IP pool: 192.168.1.30/32
    │   └── l2advertisement.yaml
    ├── envoy-gateway/
    │   └── values.yaml
    ├── envoy-gateway-config/
    │   ├── gatewayclass.yaml
    │   ├── envoy-proxy.yaml          # DaemonSet, LoadBalancer service
    │   └── gateway.yaml             # HTTP :80 listener
    └── nfs-provisioner/
        └── values.yaml              # ⚠ fill in nfs.server + nfs.path before push

---

Next: platform services

This repo only brings up the minimal layer needed to make the cluster usable (ArgoCD, LoadBalancer IPs, HTTP gateway, storage). Application/platform services (Vault, external-secrets, Harbor, Jenkins, SonarQube, etc.) live in the separate cluster-platform repo, applied after this one finishes:

kubectl apply -f ../cluster-platform/platform-app.yaml