Files

duynguyen eade8553f6 add phase 6 reference to cluster-bootstrap repo

2026-06-14 16:55:45 +07:00

9.2 KiB

Raw Permalink Blame History

K8s Cluster Runbook — Build From Scratch

Architecture

Client machine (192.168.1.39)
  ├── Terraform  → Proxmox API → provision VMs
  └── Ansible    → SSH         → configure k8s

Proxmox cluster
  ├── proxmox    (node 1)
  ├── proxmox02  (node 2)
  └── proxmox03  (node 3, 192.168.1.18) ← all VMs here

VMs (CentOS Stream 10, cloud-init)
  ├── master01   192.168.1.31   2 vCPU  8GB RAM   100GB disk
  ├── worker01   192.168.1.35   2 vCPU  12GB RAM  100GB disk
  ├── worker02   192.168.1.36   2 vCPU  12GB RAM  100GB disk
  └── worker03   192.168.1.37   2 vCPU  12GB RAM  100GB disk

Storage: xpen (NFS NAS) — all VM disks
         local           — cloud image download + cloud-init drive
Pool:    IaC

Phase 1 — Client machine setup

1.1 Install tools

Terraform:

yum install -y yum-utils

yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo

yum install -y terraform

Ansible:

pip3 install ansible

Git:

yum install -y git

1.2 Clone repo

git clone https://gitea.fireflylab.cc/duynguyen/cluster-init.git

cd cluster-init

1.3 Set up Terraform credentials

cp cluster-init/terraform/.env.example cluster-init/terraform/.env

Edit cluster-init/terraform/.env — fill in endpoint + token. This file is gitignored — never commit it.

1.4 Generate SSH keypair for VM access

Skip if cluster-init/terraform/k8s_terraform already exists.

ssh-keygen -t ed25519 -f ~/k8s/cluster-init/terraform/k8s_terraform -N "" -C "terraform-k8s"

Private key: cluster-init/terraform/k8s_terraform (gitignored) Public key: cluster-init/terraform/k8s_terraform.pub (gitignored — injected into VMs via cloud-init)

Phase 2 — Proxmox preparation

2.1 Enable local storage content types

Proxmox UI → Datacenter → Storage → local → Edit → Content

Enable:

Disk image (import) — for cloud image download via Terraform
Snippets — for cloud-init drive

2.2 Create Terraform user, role, token

SSH into any Proxmox node as root, then run each command:

Create role:

pveum role add TerraformRole --privs "VM.Allocate,VM.Clone,VM.Config.Disk,VM.Config.CPU,VM.Config.Memory,VM.Config.Network,VM.Config.Options,VM.Config.CloudInit,VM.Config.CDROM,VM.PowerMgmt,VM.Audit,Datastore.AllocateSpace,Datastore.AllocateTemplate,Datastore.Allocate,Datastore.Audit,Sys.Audit,Sys.Modify,Pool.Allocate,SDN.Use"

Create user:

pveum user add terraform@pve --comment "Terraform IaC"

Create token — SAVE the secret, shown only once:

pveum user token add terraform@pve terraform --privsep 0

ACLs:

pveum acl modify /nodes --user terraform@pve --role TerraformRole

pveum acl modify /nodes/proxmox03 --user terraform@pve --role TerraformRole

pveum acl modify /storage/local --user terraform@pve --role TerraformRole

pveum acl modify /storage/xpen --user terraform@pve --role TerraformRole

pveum acl modify /pool/IaC --user terraform@pve --role TerraformRole

pveum acl modify /sdn/zones/localnetwork/vmbr0 --user terraform@pve --role TerraformRole

Required for proxmox_download_file — must be at root:

pveum role add SysDownloadRole --privs "Sys.Audit,Sys.Modify"

pveum acl modify / --user terraform@pve --role SysDownloadRole

SDN — token needs explicit ACL (separate from user ACL):

pveum acl modify /sdn/zones/localnetwork/vmbr0 --token terraform@pve!terraform --role TerraformRole

Update cluster-init/terraform/.env with the token secret.

Phase 3 — Provision VMs with Terraform

3.1 Init

cd ~/k8s/cluster-init/terraform

source .env

terraform init

3.2 Plan

source .env

terraform plan

Expected: 5 resources to create

proxmox_download_file.centos10
proxmox_virtual_environment_vm.master
proxmox_virtual_environment_vm.workers["worker01"]
proxmox_virtual_environment_vm.workers["worker02"]
proxmox_virtual_environment_vm.workers["worker03"]

3.3 Apply

Must use -parallelism=1 — parallel disk clones to xpen NFS cause storage lock timeout. Takes ~10-15 min (cloud image download ~1 min + 4 VMs × ~3 min each).

terraform apply -parallelism=1

3.4 Post-apply state sync

After first apply, run once to sync pool membership state:

terraform apply -refresh-only

Type yes. Then verify no further changes:

terraform plan

3.5 Verify VMs

ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31

hostname

free -h

Phase 4 — Configure k8s with Ansible

4.1 Run all playbooks (one command)

cd ~/k8s/cluster-init/ansible

ansible-playbook -i inventory.ini cluster.yml

4.2 Or run individual playbooks

cd ~/k8s/cluster-init/ansible

All nodes — OS prerequisites (swap, kernel modules, sysctl):

ansible-playbook -i inventory.ini playbooks/01-common.yml

All nodes — /etc/hosts:

ansible-playbook -i inventory.ini playbooks/02-hosts.yml

All nodes — container runtime (containerd):

ansible-playbook -i inventory.ini playbooks/03-containerd.yml

All nodes — kubeadm/kubelet/kubectl:

ansible-playbook -i inventory.ini playbooks/04-kube-tools.yml

master01 only — init cluster + CNI:

ansible-playbook -i inventory.ini playbooks/05-master-init.yml

worker01/02/03 — join cluster:

ansible-playbook -i inventory.ini playbooks/06-worker-join.yml

4.3 Verify cluster (from master01)

ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31

kubectl get nodes

All 4 nodes should be Ready.

Phase 5 — kubectl from client machine (192.168.1.39)

5.1 Install kubectl

curl -LO "https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl"

chmod +x kubectl

sudo mv kubectl /usr/local/bin/kubectl

kubectl version --client

5.2 Copy kubeconfig from master01

mkdir -p ~/.kube

scp -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31:/home/cloud-user/.kube/config ~/.kube/config

5.3 Add cluster nodes to client /etc/hosts

sudo tee -a /etc/hosts <<'EOF'

# k8s cluster nodes
192.168.1.31  master01
192.168.1.35  worker01
192.168.1.36  worker02
192.168.1.37  worker03
EOF

5.4 Verify

kubectl get nodes

kubectl get pods -A

Phase 6 — Bootstrap platform services

Cluster is now running with bare k8s only. To install ArgoCD, MetalLB, Envoy Gateway, NFS storage, and all platform services — continue to the cluster-bootstrap repo:

https://gitea.fireflylab.cc/duynguyen/cluster-bootstrap

Teardown

cd ~/k8s/cluster-init/terraform

source .env

terraform destroy -parallelism=1

Destroys all 4 VMs + cloud image file. Re-run Phase 3 to rebuild from scratch.

SSH access

ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31

ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.35

ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.36

ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.37

Or add to ~/.ssh/config for shorthand (ssh master01, ssh worker01, etc.):

Host master01
  HostName 192.168.1.31
  User cloud-user
  IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform

Host worker01
  HostName 192.168.1.35
  User cloud-user
  IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform

Host worker02
  HostName 192.168.1.36
  User cloud-user
  IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform

Host worker03
  HostName 192.168.1.37
  User cloud-user
  IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform

File reference

k8s/cluster-init/
├── README.md
├── terraform/
│   ├── .env                 # Terraform credentials (gitignored)
│   ├── .env.example         # template — safe to commit
│   ├── main.tf              # provider config
│   ├── variables.tf         # input variables
│   ├── terraform.tfvars     # non-secret defaults
│   ├── cloud_image.tf       # CentOS Stream 10 cloud image download
│   ├── vms.tf               # master01 + worker01/02/03
│   ├── outputs.tf           # IPs + SSH command
│   ├── k8s_terraform        # SSH private key (gitignored)
│   ├── k8s_terraform.pub    # SSH public key (gitignored)
│   └── .gitignore
└── ansible/
    ├── cluster.yml          # run all playbooks in one command
    ├── inventory.ini
    ├── ansible.cfg
    └── playbooks/
        ├── 01-common.yml
        ├── 02-hosts.yml
        ├── 03-containerd.yml
        ├── 04-kube-tools.yml
        ├── 05-master-init.yml
        └── 06-worker-join.yml

9.2 KiB Raw Permalink Blame History Unescape Escape