# K8s Cluster Runbook — Build From Scratch ## Architecture ``` Client machine (192.168.1.39) ├── Terraform → Proxmox API → provision VMs └── Ansible → SSH → configure k8s Proxmox cluster ├── proxmox (node 1) ├── proxmox02 (node 2) └── proxmox03 (node 3, 192.168.1.18) ← all VMs here VMs (CentOS Stream 10, cloud-init) ├── master01 192.168.1.31 2 vCPU 8GB RAM 100GB disk ├── worker01 192.168.1.35 2 vCPU 12GB RAM 100GB disk ├── worker02 192.168.1.36 2 vCPU 12GB RAM 100GB disk └── worker03 192.168.1.37 2 vCPU 12GB RAM 100GB disk Storage: xpen (NFS NAS) — all VM disks local — cloud image download + cloud-init drive Pool: IaC ``` --- ## Phase 1 — Client machine setup ### 1.1 Install tools **Terraform:** ```bash yum install -y yum-utils ``` ```bash yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo ``` ```bash yum install -y terraform ``` **Ansible:** ```bash pip3 install ansible ``` **Git:** ```bash yum install -y git ``` ### 1.2 Clone repo ```bash git clone ~/k8s ``` ### 1.3 Set up Terraform credentials ```bash cp cluster-init/terraform/.env.example cluster-init/terraform/.env ``` Edit `cluster-init/terraform/.env` — fill in endpoint + token. This file is gitignored — never commit it. ### 1.4 Generate SSH keypair for VM access > Skip if `cluster-init/terraform/k8s_terraform` already exists. ```bash ssh-keygen -t ed25519 -f ~/k8s/cluster-init/terraform/k8s_terraform -N "" -C "terraform-k8s" ``` Private key: `cluster-init/terraform/k8s_terraform` (gitignored) Public key: `cluster-init/terraform/k8s_terraform.pub` (gitignored — injected into VMs via cloud-init) --- ## Phase 2 — Proxmox preparation ### 2.1 Enable local storage content types **Proxmox UI → Datacenter → Storage → local → Edit → Content** Enable: - `Disk image (import)` — for cloud image download via Terraform - `Snippets` — for cloud-init drive ### 2.2 Create Terraform user, role, token SSH into any Proxmox node as root, then run each command: **Create role:** ```bash pveum role add TerraformRole --privs "VM.Allocate,VM.Clone,VM.Config.Disk,VM.Config.CPU,VM.Config.Memory,VM.Config.Network,VM.Config.Options,VM.Config.CloudInit,VM.Config.CDROM,VM.PowerMgmt,VM.Audit,Datastore.AllocateSpace,Datastore.AllocateTemplate,Datastore.Allocate,Datastore.Audit,Sys.Audit,Sys.Modify,Pool.Allocate,SDN.Use" ``` **Create user:** ```bash pveum user add terraform@pve --comment "Terraform IaC" ``` **Create token** — SAVE the secret, shown only once: ```bash pveum user token add terraform@pve terraform --privsep 0 ``` **ACLs:** ```bash pveum acl modify /nodes --user terraform@pve --role TerraformRole ``` ```bash pveum acl modify /nodes/proxmox03 --user terraform@pve --role TerraformRole ``` ```bash pveum acl modify /storage/local --user terraform@pve --role TerraformRole ``` ```bash pveum acl modify /storage/xpen --user terraform@pve --role TerraformRole ``` ```bash pveum acl modify /pool/IaC --user terraform@pve --role TerraformRole ``` ```bash pveum acl modify /sdn/zones/localnetwork/vmbr0 --user terraform@pve --role TerraformRole ``` **Required for proxmox_download_file — must be at root:** ```bash pveum role add SysDownloadRole --privs "Sys.Audit,Sys.Modify" ``` ```bash pveum acl modify / --user terraform@pve --role SysDownloadRole ``` **SDN — token needs explicit ACL (separate from user ACL):** ```bash pveum acl modify /sdn/zones/localnetwork/vmbr0 --token terraform@pve!terraform --role TerraformRole ``` Update `cluster-init/terraform/.env` with the token secret. --- ## Phase 3 — Provision VMs with Terraform ### 3.1 Init ```bash cd ~/k8s/cluster-init/terraform ``` ```bash source .env ``` ```bash terraform init ``` ### 3.2 Plan ```bash source .env ``` ```bash terraform plan ``` Expected: 5 resources to create - `proxmox_download_file.centos10` - `proxmox_virtual_environment_vm.master` - `proxmox_virtual_environment_vm.workers["worker01"]` - `proxmox_virtual_environment_vm.workers["worker02"]` - `proxmox_virtual_environment_vm.workers["worker03"]` ### 3.3 Apply > Must use `-parallelism=1` — parallel disk clones to xpen NFS cause storage lock timeout. > Takes ~10-15 min (cloud image download ~1 min + 4 VMs × ~3 min each). ```bash terraform apply -parallelism=1 ``` ### 3.4 Post-apply state sync After first apply, run once to sync pool membership state: ```bash terraform apply -refresh-only ``` Type `yes`. Then verify no further changes: ```bash terraform plan ``` ### 3.5 Verify VMs ```bash ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31 ``` ```bash hostname ``` ```bash free -h ``` --- ## Phase 4 — Configure k8s with Ansible ### 4.1 Run all playbooks (one command) ```bash cd ~/k8s/cluster-init/ansible ``` ```bash ansible-playbook -i inventory.ini cluster.yml ``` ### 4.2 Or run individual playbooks ```bash cd ~/k8s/cluster-init/ansible ``` All nodes — OS prerequisites (swap, kernel modules, sysctl): ```bash ansible-playbook -i inventory.ini playbooks/01-common.yml ``` All nodes — /etc/hosts: ```bash ansible-playbook -i inventory.ini playbooks/02-hosts.yml ``` All nodes — container runtime (containerd): ```bash ansible-playbook -i inventory.ini playbooks/03-containerd.yml ``` All nodes — kubeadm/kubelet/kubectl: ```bash ansible-playbook -i inventory.ini playbooks/04-kube-tools.yml ``` master01 only — init cluster + CNI: ```bash ansible-playbook -i inventory.ini playbooks/05-master-init.yml ``` worker01/02/03 — join cluster: ```bash ansible-playbook -i inventory.ini playbooks/06-worker-join.yml ``` ### 4.3 Verify cluster (from master01) ```bash ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31 ``` ```bash kubectl get nodes ``` All 4 nodes should be `Ready`. --- ## Phase 5 — kubectl from client machine (192.168.1.39) ### 5.1 Install kubectl ```bash curl -LO "https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl" ``` ```bash chmod +x kubectl ``` ```bash sudo mv kubectl /usr/local/bin/kubectl ``` ```bash kubectl version --client ``` ### 5.2 Copy kubeconfig from master01 ```bash mkdir -p ~/.kube ``` ```bash scp -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31:/home/cloud-user/.kube/config ~/.kube/config ``` ### 5.3 Add cluster nodes to client /etc/hosts ```bash sudo tee -a /etc/hosts <<'EOF' # k8s cluster nodes 192.168.1.31 master01 192.168.1.35 worker01 192.168.1.36 worker02 192.168.1.37 worker03 EOF ``` ### 5.4 Verify ```bash kubectl get nodes ``` ```bash kubectl get pods -A ``` --- ## Teardown ```bash cd ~/k8s/cluster-init/terraform ``` ```bash source .env ``` ```bash terraform destroy -parallelism=1 ``` Destroys all 4 VMs + cloud image file. Re-run Phase 3 to rebuild from scratch. --- ## SSH access ```bash ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31 ``` ```bash ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.35 ``` ```bash ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.36 ``` ```bash ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.37 ``` Or add to `~/.ssh/config` for shorthand (`ssh master01`, `ssh worker01`, etc.): ``` Host master01 HostName 192.168.1.31 User cloud-user IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform Host worker01 HostName 192.168.1.35 User cloud-user IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform Host worker02 HostName 192.168.1.36 User cloud-user IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform Host worker03 HostName 192.168.1.37 User cloud-user IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform ``` --- ## File reference ``` k8s/cluster-init/ ├── README.md ├── terraform/ │ ├── .env # Terraform credentials (gitignored) │ ├── .env.example # template — safe to commit │ ├── main.tf # provider config │ ├── variables.tf # input variables │ ├── terraform.tfvars # non-secret defaults │ ├── cloud_image.tf # CentOS Stream 10 cloud image download │ ├── vms.tf # master01 + worker01/02/03 │ ├── outputs.tf # IPs + SSH command │ ├── k8s_terraform # SSH private key (gitignored) │ ├── k8s_terraform.pub # SSH public key (gitignored) │ └── .gitignore └── ansible/ ├── cluster.yml # run all playbooks in one command ├── inventory.ini ├── ansible.cfg └── playbooks/ ├── 01-common.yml ├── 02-hosts.yml ├── 03-containerd.yml ├── 04-kube-tools.yml ├── 05-master-init.yml └── 06-worker-join.yml ```