cluster-init/RUNBOOK.md

K8s Cluster Runbook — Build From Scratch

Architecture

Client machine (192.168.1.39)
  ├── Terraform  → Proxmox API → provision VMs
  └── Ansible    → SSH         → configure k8s

Proxmox cluster
  ├── proxmox    (node 1)
  ├── proxmox02  (node 2)
  └── proxmox03  (node 3, 192.168.1.18) ← all VMs here

VMs (CentOS Stream 10, cloud-init)
  ├── master01   192.168.1.31   2 vCPU  8GB RAM   100GB disk
  ├── worker01   192.168.1.35   2 vCPU  12GB RAM  100GB disk
  ├── worker02   192.168.1.36   2 vCPU  12GB RAM  100GB disk
  └── worker03   192.168.1.37   2 vCPU  12GB RAM  100GB disk

Storage: xpen (NFS NAS) — all VM disks
         local           — cloud image download + cloud-init drive
Pool:    IaC

---

Phase 1 — Client machine setup

1.1 Install tools

# Terraform
yum install -y yum-utils
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
yum install -y terraform
terraform version

# Ansible
pip3 install ansible
# or: dnf install ansible

# Git
yum install -y git

1.2 Clone repo

git clone <your-gitea-repo-url> ~/k8s
cd ~/k8s

1.3 Set up Terraform credentials

cp cluster-init/terraform/.env.example cluster-init/terraform/.env
# edit cluster-init/terraform/.env — fill in endpoint + token

cluster-init/terraform/.env is gitignored — never commit it.

1.4 Generate SSH keypair for VM access

Skip if cluster-init/terraform/k8s_terraform already exists.

ssh-keygen -t ed25519 -f ~/k8s/cluster-init/terraform/k8s_terraform -N "" -C "terraform-k8s"

Private key: cluster-init/terraform/k8s_terraform (gitignored — do not commit) Public key: cluster-init/terraform/k8s_terraform.pub (gitignored — injected into VMs via cloud-init)

---

Phase 2 — Proxmox preparation

2.1 Enable local storage content types

Proxmox UI → Datacenter → Storage → local → Edit → Content

Enable:

• Disk image (import) — for cloud image download via Terraform
• Snippets — for cloud-init drive

2.2 Create Terraform user, role, token

SSH into any Proxmox node as root:

# Create role
pveum role add TerraformRole --privs \
  "VM.Allocate,VM.Clone,VM.Config.Disk,VM.Config.CPU,\
VM.Config.Memory,VM.Config.Network,VM.Config.Options,\
VM.Config.CloudInit,VM.Config.CDROM,VM.PowerMgmt,VM.Audit,\
Datastore.AllocateSpace,Datastore.AllocateTemplate,\
Datastore.Allocate,Datastore.Audit,Sys.Audit,Sys.Modify,\
Pool.Allocate,SDN.Use"

# Create user + token — SAVE the secret, shown only once
pveum user add terraform@pve --comment "Terraform IaC"
pveum user token add terraform@pve terraform --privsep 0

# ACLs
pveum acl modify /nodes              --user terraform@pve --role TerraformRole
pveum acl modify /nodes/proxmox03   --user terraform@pve --role TerraformRole
pveum acl modify /storage/local     --user terraform@pve --role TerraformRole
pveum acl modify /storage/xpen      --user terraform@pve --role TerraformRole
pveum acl modify /pool/IaC          --user terraform@pve --role TerraformRole
pveum acl modify /sdn/zones/localnetwork/vmbr0 --user terraform@pve --role TerraformRole

# Required for proxmox_download_file — Sys.Audit + Sys.Modify must be at root
pveum role add SysDownloadRole --privs "Sys.Audit,Sys.Modify"
pveum acl modify / --user terraform@pve --role SysDownloadRole

# SDN — token needs explicit ACL (separate from user ACL)
pveum acl modify /sdn/zones/localnetwork/vmbr0 \
  --token terraform@pve!terraform --role TerraformRole

Update cluster-init/terraform/.env with the token secret.

---

Phase 3 — Provision VMs with Terraform

3.1 Init

cd ~/k8s/cluster-init/terraform
source .env
terraform init

3.2 Plan

source .env
terraform plan

Expected: 5 resources to create

• proxmox_download_file.centos10
• proxmox_virtual_environment_vm.master
• proxmox_virtual_environment_vm.workers["worker01"]
• proxmox_virtual_environment_vm.workers["worker02"]
• proxmox_virtual_environment_vm.workers["worker03"]

3.3 Apply

terraform apply -parallelism=1

Must use -parallelism=1 — parallel disk clones to xpen NFS cause storage lock timeout.

Takes ~10-15 min (cloud image download ~1 min + 4 VMs × ~3 min each).

3.4 Post-apply state sync

After first apply, run once to sync pool membership state:

terraform apply -refresh-only

Type yes. Then verify no further changes:

terraform plan

3.5 Verify VMs

ssh -i cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
hostname    # master01
free -h     # Swap: 0 (disabled by cloud-init)

---

Phase 4 — Configure k8s with Ansible

Playbooks in cluster-init/ansible/

4.1 Run all playbooks

cd ~/k8s/cluster-init/ansible

# All nodes — OS prerequisites (swap, kernel modules, sysctl)
ansible-playbook -i inventory.ini playbooks/01-common.yml

# All nodes — /etc/hosts (nodes know each other by hostname)
ansible-playbook -i inventory.ini playbooks/02-hosts.yml

# All nodes — container runtime (containerd)
ansible-playbook -i inventory.ini playbooks/03-containerd.yml

# All nodes — kubeadm/kubelet/kubectl
ansible-playbook -i inventory.ini playbooks/04-kube-tools.yml

# master01 only — init cluster + CNI
ansible-playbook -i inventory.ini playbooks/05-master-init.yml

# worker01/02/03 — join cluster
ansible-playbook -i inventory.ini playbooks/06-worker-join.yml

4.2 Verify cluster (from master01)

ssh -i ../terraform/k8s_terraform cloud-user@192.168.1.31
kubectl get nodes

All 4 nodes should be Ready.

---

Phase 5 — kubectl from client machine (192.168.1.39)

5.1 Install kubectl

# Match the cluster version (1.32)
curl -LO "https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl
kubectl version --client

5.2 Copy kubeconfig from master01

mkdir -p ~/.kube
scp -i ~/k8s/cluster-init/terraform/k8s_terraform \
    cloud-user@192.168.1.31:/home/cloud-user/.kube/config \
    ~/.kube/config

5.3 Add cluster nodes to client /etc/hosts

sudo tee -a /etc/hosts <<'EOF'

# k8s cluster nodes
192.168.1.31  master01
192.168.1.35  worker01
192.168.1.36  worker02
192.168.1.37  worker03
EOF

5.4 Verify

kubectl get nodes
kubectl get pods -A

---

Teardown

cd ~/k8s/cluster-init/terraform
source .env
terraform destroy -parallelism=1

Destroys all 4 VMs + cloud image file. Re-run Phase 3 to rebuild from scratch.

---

SSH access

ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31   # master01
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.35   # worker01
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.36   # worker02
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.37   # worker03

Or add to ~/.ssh/config:

Host master01
  HostName 192.168.1.31
  User cloud-user
  IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform

Host worker01
  HostName 192.168.1.35
  User cloud-user
  IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform

Host worker02
  HostName 192.168.1.36
  User cloud-user
  IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform

Host worker03
  HostName 192.168.1.37
  User cloud-user
  IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform

Then: ssh master01, ssh worker01, etc.

---

File reference

k8s/cluster-init/
├── RUNBOOK.md
├── terraform/
│   ├── .env                 # Terraform credentials (gitignored)
│   ├── .env.example         # template — safe to commit
│   ├── main.tf              # provider config
│   ├── variables.tf         # input variables
│   ├── terraform.tfvars     # non-secret defaults
│   ├── cloud_image.tf       # CentOS Stream 10 cloud image download
│   ├── vms.tf               # master01 + worker01/02/03
│   ├── outputs.tf           # IPs + SSH command
│   ├── k8s_terraform        # SSH private key (gitignored)
│   ├── k8s_terraform.pub    # SSH public key (gitignored)
│   └── .gitignore
└── ansible/
    ├── inventory.ini
    ├── ansible.cfg
    └── playbooks/
        ├── 01-common.yml
        ├── 02-hosts.yml
        ├── 03-containerd.yml
        ├── 04-kube-tools.yml
        ├── 05-master-init.yml
        └── 06-worker-join.yml