449 lines
8.8 KiB
Markdown
449 lines
8.8 KiB
Markdown
# K8s Cluster Runbook — Build From Scratch
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Client machine (192.168.1.39)
|
||
├── Terraform → Proxmox API → provision VMs
|
||
└── Ansible → SSH → configure k8s
|
||
|
||
Proxmox cluster
|
||
├── proxmox (node 1)
|
||
├── proxmox02 (node 2)
|
||
└── proxmox03 (node 3, 192.168.1.18) ← all VMs here
|
||
|
||
VMs (CentOS Stream 10, cloud-init)
|
||
├── master01 192.168.1.31 2 vCPU 8GB RAM 100GB disk
|
||
├── worker01 192.168.1.35 2 vCPU 12GB RAM 100GB disk
|
||
├── worker02 192.168.1.36 2 vCPU 12GB RAM 100GB disk
|
||
└── worker03 192.168.1.37 2 vCPU 12GB RAM 100GB disk
|
||
|
||
Storage: xpen (NFS NAS) — all VM disks
|
||
local — cloud image download + cloud-init drive
|
||
Pool: IaC
|
||
```
|
||
|
||
---
|
||
|
||
## Phase 1 — Client machine setup
|
||
|
||
### 1.1 Install tools
|
||
|
||
**Terraform:**
|
||
|
||
```bash
|
||
yum install -y yum-utils
|
||
```
|
||
|
||
```bash
|
||
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
|
||
```
|
||
|
||
```bash
|
||
yum install -y terraform
|
||
```
|
||
|
||
**Ansible:**
|
||
|
||
```bash
|
||
pip3 install ansible
|
||
```
|
||
|
||
**Git:**
|
||
|
||
```bash
|
||
yum install -y git
|
||
```
|
||
|
||
### 1.2 Clone repo
|
||
|
||
```bash
|
||
git clone <your-gitea-repo-url> ~/k8s
|
||
```
|
||
|
||
### 1.3 Set up Terraform credentials
|
||
|
||
```bash
|
||
cp cluster-init/terraform/.env.example cluster-init/terraform/.env
|
||
```
|
||
|
||
Edit `cluster-init/terraform/.env` — fill in endpoint + token. This file is gitignored — never commit it.
|
||
|
||
### 1.4 Generate SSH keypair for VM access
|
||
|
||
> Skip if `cluster-init/terraform/k8s_terraform` already exists.
|
||
|
||
```bash
|
||
ssh-keygen -t ed25519 -f ~/k8s/cluster-init/terraform/k8s_terraform -N "" -C "terraform-k8s"
|
||
```
|
||
|
||
Private key: `cluster-init/terraform/k8s_terraform` (gitignored)
|
||
Public key: `cluster-init/terraform/k8s_terraform.pub` (gitignored — injected into VMs via cloud-init)
|
||
|
||
---
|
||
|
||
## Phase 2 — Proxmox preparation
|
||
|
||
### 2.1 Enable local storage content types
|
||
|
||
**Proxmox UI → Datacenter → Storage → local → Edit → Content**
|
||
|
||
Enable:
|
||
- `Disk image (import)` — for cloud image download via Terraform
|
||
- `Snippets` — for cloud-init drive
|
||
|
||
### 2.2 Create Terraform user, role, token
|
||
|
||
SSH into any Proxmox node as root, then run each command:
|
||
|
||
**Create role:**
|
||
|
||
```bash
|
||
pveum role add TerraformRole --privs "VM.Allocate,VM.Clone,VM.Config.Disk,VM.Config.CPU,VM.Config.Memory,VM.Config.Network,VM.Config.Options,VM.Config.CloudInit,VM.Config.CDROM,VM.PowerMgmt,VM.Audit,Datastore.AllocateSpace,Datastore.AllocateTemplate,Datastore.Allocate,Datastore.Audit,Sys.Audit,Sys.Modify,Pool.Allocate,SDN.Use"
|
||
```
|
||
|
||
**Create user:**
|
||
|
||
```bash
|
||
pveum user add terraform@pve --comment "Terraform IaC"
|
||
```
|
||
|
||
**Create token** — SAVE the secret, shown only once:
|
||
|
||
```bash
|
||
pveum user token add terraform@pve terraform --privsep 0
|
||
```
|
||
|
||
**ACLs:**
|
||
|
||
```bash
|
||
pveum acl modify /nodes --user terraform@pve --role TerraformRole
|
||
```
|
||
|
||
```bash
|
||
pveum acl modify /nodes/proxmox03 --user terraform@pve --role TerraformRole
|
||
```
|
||
|
||
```bash
|
||
pveum acl modify /storage/local --user terraform@pve --role TerraformRole
|
||
```
|
||
|
||
```bash
|
||
pveum acl modify /storage/xpen --user terraform@pve --role TerraformRole
|
||
```
|
||
|
||
```bash
|
||
pveum acl modify /pool/IaC --user terraform@pve --role TerraformRole
|
||
```
|
||
|
||
```bash
|
||
pveum acl modify /sdn/zones/localnetwork/vmbr0 --user terraform@pve --role TerraformRole
|
||
```
|
||
|
||
**Required for proxmox_download_file — must be at root:**
|
||
|
||
```bash
|
||
pveum role add SysDownloadRole --privs "Sys.Audit,Sys.Modify"
|
||
```
|
||
|
||
```bash
|
||
pveum acl modify / --user terraform@pve --role SysDownloadRole
|
||
```
|
||
|
||
**SDN — token needs explicit ACL (separate from user ACL):**
|
||
|
||
```bash
|
||
pveum acl modify /sdn/zones/localnetwork/vmbr0 --token terraform@pve!terraform --role TerraformRole
|
||
```
|
||
|
||
Update `cluster-init/terraform/.env` with the token secret.
|
||
|
||
---
|
||
|
||
## Phase 3 — Provision VMs with Terraform
|
||
|
||
### 3.1 Init
|
||
|
||
```bash
|
||
cd ~/k8s/cluster-init/terraform
|
||
```
|
||
|
||
```bash
|
||
source .env
|
||
```
|
||
|
||
```bash
|
||
terraform init
|
||
```
|
||
|
||
### 3.2 Plan
|
||
|
||
```bash
|
||
source .env
|
||
```
|
||
|
||
```bash
|
||
terraform plan
|
||
```
|
||
|
||
Expected: 5 resources to create
|
||
- `proxmox_download_file.centos10`
|
||
- `proxmox_virtual_environment_vm.master`
|
||
- `proxmox_virtual_environment_vm.workers["worker01"]`
|
||
- `proxmox_virtual_environment_vm.workers["worker02"]`
|
||
- `proxmox_virtual_environment_vm.workers["worker03"]`
|
||
|
||
### 3.3 Apply
|
||
|
||
> Must use `-parallelism=1` — parallel disk clones to xpen NFS cause storage lock timeout.
|
||
> Takes ~10-15 min (cloud image download ~1 min + 4 VMs × ~3 min each).
|
||
|
||
```bash
|
||
terraform apply -parallelism=1
|
||
```
|
||
|
||
### 3.4 Post-apply state sync
|
||
|
||
After first apply, run once to sync pool membership state:
|
||
|
||
```bash
|
||
terraform apply -refresh-only
|
||
```
|
||
|
||
Type `yes`. Then verify no further changes:
|
||
|
||
```bash
|
||
terraform plan
|
||
```
|
||
|
||
### 3.5 Verify VMs
|
||
|
||
```bash
|
||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
|
||
```
|
||
|
||
```bash
|
||
hostname
|
||
```
|
||
|
||
```bash
|
||
free -h
|
||
```
|
||
|
||
---
|
||
|
||
## Phase 4 — Configure k8s with Ansible
|
||
|
||
### 4.1 Run all playbooks (one command)
|
||
|
||
```bash
|
||
cd ~/k8s/cluster-init/ansible
|
||
```
|
||
|
||
```bash
|
||
ansible-playbook -i inventory.ini cluster.yml
|
||
```
|
||
|
||
### 4.2 Or run individual playbooks
|
||
|
||
```bash
|
||
cd ~/k8s/cluster-init/ansible
|
||
```
|
||
|
||
All nodes — OS prerequisites (swap, kernel modules, sysctl):
|
||
|
||
```bash
|
||
ansible-playbook -i inventory.ini playbooks/01-common.yml
|
||
```
|
||
|
||
All nodes — /etc/hosts:
|
||
|
||
```bash
|
||
ansible-playbook -i inventory.ini playbooks/02-hosts.yml
|
||
```
|
||
|
||
All nodes — container runtime (containerd):
|
||
|
||
```bash
|
||
ansible-playbook -i inventory.ini playbooks/03-containerd.yml
|
||
```
|
||
|
||
All nodes — kubeadm/kubelet/kubectl:
|
||
|
||
```bash
|
||
ansible-playbook -i inventory.ini playbooks/04-kube-tools.yml
|
||
```
|
||
|
||
master01 only — init cluster + CNI:
|
||
|
||
```bash
|
||
ansible-playbook -i inventory.ini playbooks/05-master-init.yml
|
||
```
|
||
|
||
worker01/02/03 — join cluster:
|
||
|
||
```bash
|
||
ansible-playbook -i inventory.ini playbooks/06-worker-join.yml
|
||
```
|
||
|
||
### 4.3 Verify cluster (from master01)
|
||
|
||
```bash
|
||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
|
||
```
|
||
|
||
```bash
|
||
kubectl get nodes
|
||
```
|
||
|
||
All 4 nodes should be `Ready`.
|
||
|
||
---
|
||
|
||
## Phase 5 — kubectl from client machine (192.168.1.39)
|
||
|
||
### 5.1 Install kubectl
|
||
|
||
```bash
|
||
curl -LO "https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl"
|
||
```
|
||
|
||
```bash
|
||
chmod +x kubectl
|
||
```
|
||
|
||
```bash
|
||
sudo mv kubectl /usr/local/bin/kubectl
|
||
```
|
||
|
||
```bash
|
||
kubectl version --client
|
||
```
|
||
|
||
### 5.2 Copy kubeconfig from master01
|
||
|
||
```bash
|
||
mkdir -p ~/.kube
|
||
```
|
||
|
||
```bash
|
||
scp -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31:/home/cloud-user/.kube/config ~/.kube/config
|
||
```
|
||
|
||
### 5.3 Add cluster nodes to client /etc/hosts
|
||
|
||
```bash
|
||
sudo tee -a /etc/hosts <<'EOF'
|
||
|
||
# k8s cluster nodes
|
||
192.168.1.31 master01
|
||
192.168.1.35 worker01
|
||
192.168.1.36 worker02
|
||
192.168.1.37 worker03
|
||
EOF
|
||
```
|
||
|
||
### 5.4 Verify
|
||
|
||
```bash
|
||
kubectl get nodes
|
||
```
|
||
|
||
```bash
|
||
kubectl get pods -A
|
||
```
|
||
|
||
---
|
||
|
||
## Teardown
|
||
|
||
```bash
|
||
cd ~/k8s/cluster-init/terraform
|
||
```
|
||
|
||
```bash
|
||
source .env
|
||
```
|
||
|
||
```bash
|
||
terraform destroy -parallelism=1
|
||
```
|
||
|
||
Destroys all 4 VMs + cloud image file. Re-run Phase 3 to rebuild from scratch.
|
||
|
||
---
|
||
|
||
## SSH access
|
||
|
||
```bash
|
||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
|
||
```
|
||
|
||
```bash
|
||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.35
|
||
```
|
||
|
||
```bash
|
||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.36
|
||
```
|
||
|
||
```bash
|
||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.37
|
||
```
|
||
|
||
Or add to `~/.ssh/config` for shorthand (`ssh master01`, `ssh worker01`, etc.):
|
||
|
||
```
|
||
Host master01
|
||
HostName 192.168.1.31
|
||
User cloud-user
|
||
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
|
||
|
||
Host worker01
|
||
HostName 192.168.1.35
|
||
User cloud-user
|
||
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
|
||
|
||
Host worker02
|
||
HostName 192.168.1.36
|
||
User cloud-user
|
||
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
|
||
|
||
Host worker03
|
||
HostName 192.168.1.37
|
||
User cloud-user
|
||
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
|
||
```
|
||
|
||
---
|
||
|
||
## File reference
|
||
|
||
```
|
||
k8s/cluster-init/
|
||
├── README.md
|
||
├── terraform/
|
||
│ ├── .env # Terraform credentials (gitignored)
|
||
│ ├── .env.example # template — safe to commit
|
||
│ ├── main.tf # provider config
|
||
│ ├── variables.tf # input variables
|
||
│ ├── terraform.tfvars # non-secret defaults
|
||
│ ├── cloud_image.tf # CentOS Stream 10 cloud image download
|
||
│ ├── vms.tf # master01 + worker01/02/03
|
||
│ ├── outputs.tf # IPs + SSH command
|
||
│ ├── k8s_terraform # SSH private key (gitignored)
|
||
│ ├── k8s_terraform.pub # SSH public key (gitignored)
|
||
│ └── .gitignore
|
||
└── ansible/
|
||
├── cluster.yml # run all playbooks in one command
|
||
├── inventory.ini
|
||
├── ansible.cfg
|
||
└── playbooks/
|
||
├── 01-common.yml
|
||
├── 02-hosts.yml
|
||
├── 03-containerd.yml
|
||
├── 04-kube-tools.yml
|
||
├── 05-master-init.yml
|
||
└── 06-worker-join.yml
|
||
```
|