rename RUNBOOK.md to README.md
This commit is contained in:
339
README.md
Normal file
339
README.md
Normal file
@@ -0,0 +1,339 @@
|
||||
# K8s Cluster Runbook — Build From Scratch
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Client machine (192.168.1.39)
|
||||
├── Terraform → Proxmox API → provision VMs
|
||||
└── Ansible → SSH → configure k8s
|
||||
|
||||
Proxmox cluster
|
||||
├── proxmox (node 1)
|
||||
├── proxmox02 (node 2)
|
||||
└── proxmox03 (node 3, 192.168.1.18) ← all VMs here
|
||||
|
||||
VMs (CentOS Stream 10, cloud-init)
|
||||
├── master01 192.168.1.31 2 vCPU 8GB RAM 100GB disk
|
||||
├── worker01 192.168.1.35 2 vCPU 12GB RAM 100GB disk
|
||||
├── worker02 192.168.1.36 2 vCPU 12GB RAM 100GB disk
|
||||
└── worker03 192.168.1.37 2 vCPU 12GB RAM 100GB disk
|
||||
|
||||
Storage: xpen (NFS NAS) — all VM disks
|
||||
local — cloud image download + cloud-init drive
|
||||
Pool: IaC
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Client machine setup
|
||||
|
||||
### 1.1 Install tools
|
||||
|
||||
```bash
|
||||
# Terraform
|
||||
yum install -y yum-utils
|
||||
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
|
||||
yum install -y terraform
|
||||
terraform version
|
||||
|
||||
# Ansible
|
||||
pip3 install ansible
|
||||
# or: dnf install ansible
|
||||
|
||||
# Git
|
||||
yum install -y git
|
||||
```
|
||||
|
||||
### 1.2 Clone repo
|
||||
|
||||
```bash
|
||||
git clone <your-gitea-repo-url> ~/k8s
|
||||
cd ~/k8s
|
||||
```
|
||||
|
||||
### 1.3 Set up Terraform credentials
|
||||
|
||||
```bash
|
||||
cp cluster-init/terraform/.env.example cluster-init/terraform/.env
|
||||
# edit cluster-init/terraform/.env — fill in endpoint + token
|
||||
```
|
||||
|
||||
`cluster-init/terraform/.env` is gitignored — never commit it.
|
||||
|
||||
### 1.4 Generate SSH keypair for VM access
|
||||
|
||||
> Skip if `cluster-init/terraform/k8s_terraform` already exists.
|
||||
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -f ~/k8s/cluster-init/terraform/k8s_terraform -N "" -C "terraform-k8s"
|
||||
```
|
||||
|
||||
Private key: `cluster-init/terraform/k8s_terraform` (gitignored — do not commit)
|
||||
Public key: `cluster-init/terraform/k8s_terraform.pub` (gitignored — injected into VMs via cloud-init)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Proxmox preparation
|
||||
|
||||
### 2.1 Enable local storage content types
|
||||
|
||||
**Proxmox UI → Datacenter → Storage → local → Edit → Content**
|
||||
|
||||
Enable:
|
||||
- `Disk image (import)` — for cloud image download via Terraform
|
||||
- `Snippets` — for cloud-init drive
|
||||
|
||||
### 2.2 Create Terraform user, role, token
|
||||
|
||||
SSH into any Proxmox node as root:
|
||||
|
||||
```bash
|
||||
# Create role
|
||||
pveum role add TerraformRole --privs \
|
||||
"VM.Allocate,VM.Clone,VM.Config.Disk,VM.Config.CPU,\
|
||||
VM.Config.Memory,VM.Config.Network,VM.Config.Options,\
|
||||
VM.Config.CloudInit,VM.Config.CDROM,VM.PowerMgmt,VM.Audit,\
|
||||
Datastore.AllocateSpace,Datastore.AllocateTemplate,\
|
||||
Datastore.Allocate,Datastore.Audit,Sys.Audit,Sys.Modify,\
|
||||
Pool.Allocate,SDN.Use"
|
||||
|
||||
# Create user + token — SAVE the secret, shown only once
|
||||
pveum user add terraform@pve --comment "Terraform IaC"
|
||||
pveum user token add terraform@pve terraform --privsep 0
|
||||
|
||||
# ACLs
|
||||
pveum acl modify /nodes --user terraform@pve --role TerraformRole
|
||||
pveum acl modify /nodes/proxmox03 --user terraform@pve --role TerraformRole
|
||||
pveum acl modify /storage/local --user terraform@pve --role TerraformRole
|
||||
pveum acl modify /storage/xpen --user terraform@pve --role TerraformRole
|
||||
pveum acl modify /pool/IaC --user terraform@pve --role TerraformRole
|
||||
pveum acl modify /sdn/zones/localnetwork/vmbr0 --user terraform@pve --role TerraformRole
|
||||
|
||||
# Required for proxmox_download_file — Sys.Audit + Sys.Modify must be at root
|
||||
pveum role add SysDownloadRole --privs "Sys.Audit,Sys.Modify"
|
||||
pveum acl modify / --user terraform@pve --role SysDownloadRole
|
||||
|
||||
# SDN — token needs explicit ACL (separate from user ACL)
|
||||
pveum acl modify /sdn/zones/localnetwork/vmbr0 \
|
||||
--token terraform@pve!terraform --role TerraformRole
|
||||
```
|
||||
|
||||
Update `cluster-init/terraform/.env` with the token secret.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Provision VMs with Terraform
|
||||
|
||||
### 3.1 Init
|
||||
|
||||
```bash
|
||||
cd ~/k8s/cluster-init/terraform
|
||||
source .env
|
||||
terraform init
|
||||
```
|
||||
|
||||
### 3.2 Plan
|
||||
|
||||
```bash
|
||||
source .env
|
||||
terraform plan
|
||||
```
|
||||
|
||||
Expected: 5 resources to create
|
||||
- `proxmox_download_file.centos10`
|
||||
- `proxmox_virtual_environment_vm.master`
|
||||
- `proxmox_virtual_environment_vm.workers["worker01"]`
|
||||
- `proxmox_virtual_environment_vm.workers["worker02"]`
|
||||
- `proxmox_virtual_environment_vm.workers["worker03"]`
|
||||
|
||||
### 3.3 Apply
|
||||
|
||||
```bash
|
||||
terraform apply -parallelism=1
|
||||
```
|
||||
|
||||
> Must use `-parallelism=1` — parallel disk clones to xpen NFS cause storage lock timeout.
|
||||
|
||||
Takes ~10-15 min (cloud image download ~1 min + 4 VMs × ~3 min each).
|
||||
|
||||
### 3.4 Post-apply state sync
|
||||
|
||||
After first apply, run once to sync pool membership state:
|
||||
```bash
|
||||
terraform apply -refresh-only
|
||||
```
|
||||
|
||||
Type `yes`. Then verify no further changes:
|
||||
```bash
|
||||
terraform plan
|
||||
```
|
||||
|
||||
### 3.5 Verify VMs
|
||||
|
||||
```bash
|
||||
ssh -i cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
|
||||
hostname # master01
|
||||
free -h # Swap: 0 (disabled by cloud-init)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Configure k8s with Ansible
|
||||
|
||||
> Playbooks in `cluster-init/ansible/`
|
||||
|
||||
### 4.1 Run all playbooks
|
||||
|
||||
```bash
|
||||
cd ~/k8s/cluster-init/ansible
|
||||
|
||||
# All nodes — OS prerequisites (swap, kernel modules, sysctl)
|
||||
ansible-playbook -i inventory.ini playbooks/01-common.yml
|
||||
|
||||
# All nodes — /etc/hosts (nodes know each other by hostname)
|
||||
ansible-playbook -i inventory.ini playbooks/02-hosts.yml
|
||||
|
||||
# All nodes — container runtime (containerd)
|
||||
ansible-playbook -i inventory.ini playbooks/03-containerd.yml
|
||||
|
||||
# All nodes — kubeadm/kubelet/kubectl
|
||||
ansible-playbook -i inventory.ini playbooks/04-kube-tools.yml
|
||||
|
||||
# master01 only — init cluster + CNI
|
||||
ansible-playbook -i inventory.ini playbooks/05-master-init.yml
|
||||
|
||||
# worker01/02/03 — join cluster
|
||||
ansible-playbook -i inventory.ini playbooks/06-worker-join.yml
|
||||
```
|
||||
|
||||
### 4.2 Verify cluster (from master01)
|
||||
|
||||
```bash
|
||||
ssh -i ../terraform/k8s_terraform cloud-user@192.168.1.31
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
All 4 nodes should be `Ready`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — kubectl from client machine (192.168.1.39)
|
||||
|
||||
### 5.1 Install kubectl
|
||||
|
||||
```bash
|
||||
# Match the cluster version (1.32)
|
||||
curl -LO "https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl"
|
||||
chmod +x kubectl
|
||||
sudo mv kubectl /usr/local/bin/kubectl
|
||||
kubectl version --client
|
||||
```
|
||||
|
||||
### 5.2 Copy kubeconfig from master01
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.kube
|
||||
scp -i ~/k8s/cluster-init/terraform/k8s_terraform \
|
||||
cloud-user@192.168.1.31:/home/cloud-user/.kube/config \
|
||||
~/.kube/config
|
||||
```
|
||||
|
||||
### 5.3 Add cluster nodes to client /etc/hosts
|
||||
|
||||
```bash
|
||||
sudo tee -a /etc/hosts <<'EOF'
|
||||
|
||||
# k8s cluster nodes
|
||||
192.168.1.31 master01
|
||||
192.168.1.35 worker01
|
||||
192.168.1.36 worker02
|
||||
192.168.1.37 worker03
|
||||
EOF
|
||||
```
|
||||
|
||||
### 5.4 Verify
|
||||
|
||||
```bash
|
||||
kubectl get nodes
|
||||
kubectl get pods -A
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Teardown
|
||||
|
||||
```bash
|
||||
cd ~/k8s/cluster-init/terraform
|
||||
source .env
|
||||
terraform destroy -parallelism=1
|
||||
```
|
||||
|
||||
Destroys all 4 VMs + cloud image file. Re-run Phase 3 to rebuild from scratch.
|
||||
|
||||
---
|
||||
|
||||
## SSH access
|
||||
|
||||
```bash
|
||||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31 # master01
|
||||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.35 # worker01
|
||||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.36 # worker02
|
||||
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.37 # worker03
|
||||
```
|
||||
|
||||
Or add to `~/.ssh/config`:
|
||||
|
||||
```
|
||||
Host master01
|
||||
HostName 192.168.1.31
|
||||
User cloud-user
|
||||
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
|
||||
|
||||
Host worker01
|
||||
HostName 192.168.1.35
|
||||
User cloud-user
|
||||
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
|
||||
|
||||
Host worker02
|
||||
HostName 192.168.1.36
|
||||
User cloud-user
|
||||
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
|
||||
|
||||
Host worker03
|
||||
HostName 192.168.1.37
|
||||
User cloud-user
|
||||
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
|
||||
```
|
||||
|
||||
Then: `ssh master01`, `ssh worker01`, etc.
|
||||
|
||||
---
|
||||
|
||||
## File reference
|
||||
|
||||
```
|
||||
k8s/cluster-init/
|
||||
├── RUNBOOK.md
|
||||
├── terraform/
|
||||
│ ├── .env # Terraform credentials (gitignored)
|
||||
│ ├── .env.example # template — safe to commit
|
||||
│ ├── main.tf # provider config
|
||||
│ ├── variables.tf # input variables
|
||||
│ ├── terraform.tfvars # non-secret defaults
|
||||
│ ├── cloud_image.tf # CentOS Stream 10 cloud image download
|
||||
│ ├── vms.tf # master01 + worker01/02/03
|
||||
│ ├── outputs.tf # IPs + SSH command
|
||||
│ ├── k8s_terraform # SSH private key (gitignored)
|
||||
│ ├── k8s_terraform.pub # SSH public key (gitignored)
|
||||
│ └── .gitignore
|
||||
└── ansible/
|
||||
├── inventory.ini
|
||||
├── ansible.cfg
|
||||
└── playbooks/
|
||||
├── 01-common.yml
|
||||
├── 02-hosts.yml
|
||||
├── 03-containerd.yml
|
||||
├── 04-kube-tools.yml
|
||||
├── 05-master-init.yml
|
||||
└── 06-worker-join.yml
|
||||
```
|
||||
Reference in New Issue
Block a user