# K8s Cluster Runbook — Build From Scratch ## Architecture ``` Client machine (192.168.1.39) ├── Terraform → Proxmox API → provision VMs └── Ansible → SSH → configure k8s Proxmox cluster ├── proxmox (node 1) ├── proxmox02 (node 2) └── proxmox03 (node 3, 192.168.1.18) ← all VMs here VMs (CentOS Stream 10, cloud-init) ├── master01 192.168.1.31 2 vCPU 8GB RAM 100GB disk ├── worker01 192.168.1.35 2 vCPU 12GB RAM 100GB disk ├── worker02 192.168.1.36 2 vCPU 12GB RAM 100GB disk └── worker03 192.168.1.37 2 vCPU 12GB RAM 100GB disk Storage: xpen (NFS NAS) — all VM disks local — cloud image download + cloud-init drive Pool: IaC ``` --- ## Phase 1 — Client machine setup ### 1.1 Install tools ```bash # Terraform yum install -y yum-utils yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo yum install -y terraform terraform version # Ansible pip3 install ansible # or: dnf install ansible # Git yum install -y git ``` ### 1.2 Clone repo ```bash git clone ~/k8s cd ~/k8s ``` ### 1.3 Set up Terraform credentials ```bash cp cluster-init/terraform/.env.example cluster-init/terraform/.env # edit cluster-init/terraform/.env — fill in endpoint + token ``` `cluster-init/terraform/.env` is gitignored — never commit it. ### 1.4 Generate SSH keypair for VM access > Skip if `cluster-init/terraform/k8s_terraform` already exists. ```bash ssh-keygen -t ed25519 -f ~/k8s/cluster-init/terraform/k8s_terraform -N "" -C "terraform-k8s" ``` Private key: `cluster-init/terraform/k8s_terraform` (gitignored — do not commit) Public key: `cluster-init/terraform/k8s_terraform.pub` (gitignored — injected into VMs via cloud-init) --- ## Phase 2 — Proxmox preparation ### 2.1 Enable local storage content types **Proxmox UI → Datacenter → Storage → local → Edit → Content** Enable: - `Disk image (import)` — for cloud image download via Terraform - `Snippets` — for cloud-init drive ### 2.2 Create Terraform user, role, token SSH into any Proxmox node as root: ```bash # Create role pveum role add TerraformRole --privs \ "VM.Allocate,VM.Clone,VM.Config.Disk,VM.Config.CPU,\ VM.Config.Memory,VM.Config.Network,VM.Config.Options,\ VM.Config.CloudInit,VM.Config.CDROM,VM.PowerMgmt,VM.Audit,\ Datastore.AllocateSpace,Datastore.AllocateTemplate,\ Datastore.Allocate,Datastore.Audit,Sys.Audit,Sys.Modify,\ Pool.Allocate,SDN.Use" # Create user + token — SAVE the secret, shown only once pveum user add terraform@pve --comment "Terraform IaC" pveum user token add terraform@pve terraform --privsep 0 # ACLs pveum acl modify /nodes --user terraform@pve --role TerraformRole pveum acl modify /nodes/proxmox03 --user terraform@pve --role TerraformRole pveum acl modify /storage/local --user terraform@pve --role TerraformRole pveum acl modify /storage/xpen --user terraform@pve --role TerraformRole pveum acl modify /pool/IaC --user terraform@pve --role TerraformRole pveum acl modify /sdn/zones/localnetwork/vmbr0 --user terraform@pve --role TerraformRole # Required for proxmox_download_file — Sys.Audit + Sys.Modify must be at root pveum role add SysDownloadRole --privs "Sys.Audit,Sys.Modify" pveum acl modify / --user terraform@pve --role SysDownloadRole # SDN — token needs explicit ACL (separate from user ACL) pveum acl modify /sdn/zones/localnetwork/vmbr0 \ --token terraform@pve!terraform --role TerraformRole ``` Update `cluster-init/terraform/.env` with the token secret. --- ## Phase 3 — Provision VMs with Terraform ### 3.1 Init ```bash cd ~/k8s/cluster-init/terraform source .env terraform init ``` ### 3.2 Plan ```bash source .env terraform plan ``` Expected: 5 resources to create - `proxmox_download_file.centos10` - `proxmox_virtual_environment_vm.master` - `proxmox_virtual_environment_vm.workers["worker01"]` - `proxmox_virtual_environment_vm.workers["worker02"]` - `proxmox_virtual_environment_vm.workers["worker03"]` ### 3.3 Apply ```bash terraform apply -parallelism=1 ``` > Must use `-parallelism=1` — parallel disk clones to xpen NFS cause storage lock timeout. Takes ~10-15 min (cloud image download ~1 min + 4 VMs × ~3 min each). ### 3.4 Post-apply state sync After first apply, run once to sync pool membership state: ```bash terraform apply -refresh-only ``` Type `yes`. Then verify no further changes: ```bash terraform plan ``` ### 3.5 Verify VMs ```bash ssh -i cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31 hostname # master01 free -h # Swap: 0 (disabled by cloud-init) ``` --- ## Phase 4 — Configure k8s with Ansible > Playbooks in `cluster-init/ansible/` ### 4.1 Run all playbooks (one command) ```bash cd ~/k8s/cluster-init/ansible ansible-playbook -i inventory.ini cluster.yml ``` Or run individual playbooks: ```bash cd ~/k8s/cluster-init/ansible # All nodes — OS prerequisites (swap, kernel modules, sysctl) ansible-playbook -i inventory.ini playbooks/01-common.yml # All nodes — /etc/hosts (nodes know each other by hostname) ansible-playbook -i inventory.ini playbooks/02-hosts.yml # All nodes — container runtime (containerd) ansible-playbook -i inventory.ini playbooks/03-containerd.yml # All nodes — kubeadm/kubelet/kubectl ansible-playbook -i inventory.ini playbooks/04-kube-tools.yml # master01 only — init cluster + CNI ansible-playbook -i inventory.ini playbooks/05-master-init.yml # worker01/02/03 — join cluster ansible-playbook -i inventory.ini playbooks/06-worker-join.yml ``` ### 4.2 Verify cluster (from master01) ```bash ssh -i ../terraform/k8s_terraform cloud-user@192.168.1.31 kubectl get nodes ``` All 4 nodes should be `Ready`. --- ## Phase 5 — kubectl from client machine (192.168.1.39) ### 5.1 Install kubectl ```bash # Match the cluster version (1.32) curl -LO "https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl" chmod +x kubectl sudo mv kubectl /usr/local/bin/kubectl kubectl version --client ``` ### 5.2 Copy kubeconfig from master01 ```bash mkdir -p ~/.kube scp -i ~/k8s/cluster-init/terraform/k8s_terraform \ cloud-user@192.168.1.31:/home/cloud-user/.kube/config \ ~/.kube/config ``` ### 5.3 Add cluster nodes to client /etc/hosts ```bash sudo tee -a /etc/hosts <<'EOF' # k8s cluster nodes 192.168.1.31 master01 192.168.1.35 worker01 192.168.1.36 worker02 192.168.1.37 worker03 EOF ``` ### 5.4 Verify ```bash kubectl get nodes kubectl get pods -A ``` --- ## Teardown ```bash cd ~/k8s/cluster-init/terraform source .env terraform destroy -parallelism=1 ``` Destroys all 4 VMs + cloud image file. Re-run Phase 3 to rebuild from scratch. --- ## SSH access ```bash ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31 # master01 ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.35 # worker01 ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.36 # worker02 ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.37 # worker03 ``` Or add to `~/.ssh/config`: ``` Host master01 HostName 192.168.1.31 User cloud-user IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform Host worker01 HostName 192.168.1.35 User cloud-user IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform Host worker02 HostName 192.168.1.36 User cloud-user IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform Host worker03 HostName 192.168.1.37 User cloud-user IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform ``` Then: `ssh master01`, `ssh worker01`, etc. --- ## File reference ``` k8s/cluster-init/ ├── RUNBOOK.md ├── terraform/ │ ├── .env # Terraform credentials (gitignored) │ ├── .env.example # template — safe to commit │ ├── main.tf # provider config │ ├── variables.tf # input variables │ ├── terraform.tfvars # non-secret defaults │ ├── cloud_image.tf # CentOS Stream 10 cloud image download │ ├── vms.tf # master01 + worker01/02/03 │ ├── outputs.tf # IPs + SSH command │ ├── k8s_terraform # SSH private key (gitignored) │ ├── k8s_terraform.pub # SSH public key (gitignored) │ └── .gitignore └── ansible/ ├── inventory.ini ├── ansible.cfg └── playbooks/ ├── 01-common.yml ├── 02-hosts.yml ├── 03-containerd.yml ├── 04-kube-tools.yml ├── 05-master-init.yml └── 06-worker-join.yml ```