split all commands into individual code blocks for copy buttons

2026-06-14 14:34:09 +07:00
parent 65c8592aa1
commit e091584f79
1 changed files with 162 additions and 60 deletions
--- a/README.md
+++ b/README.md
@@ -29,18 +29,29 @@ Pool:    IaC

 ### 1.1 Install tools

+**Terraform:**
+
 ```bash
-# Terraform
 yum install -y yum-utils
+```
+
+```bash
 yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
+```
+
+```bash
 yum install -y terraform
-terraform version
+```

-# Ansible
+**Ansible:**
+
+```bash
 pip3 install ansible
-# or: dnf install ansible
+```

-# Git
+**Git:**
+
+```bash
 yum install -y git
 ```

@@ -48,17 +59,15 @@ yum install -y git

 ```bash
 git clone <your-gitea-repo-url> ~/k8s
-cd ~/k8s
 ```

 ### 1.3 Set up Terraform credentials

 ```bash
 cp cluster-init/terraform/.env.example cluster-init/terraform/.env
-# edit cluster-init/terraform/.env — fill in endpoint + token
 ```

-`cluster-init/terraform/.env` is gitignored — never commit it.
+Edit `cluster-init/terraform/.env` — fill in endpoint + token. This file is gitignored — never commit it.

 ### 1.4 Generate SSH keypair for VM access

@@ -68,7 +77,7 @@ cp cluster-init/terraform/.env.example cluster-init/terraform/.env
 ssh-keygen -t ed25519 -f ~/k8s/cluster-init/terraform/k8s_terraform -N "" -C "terraform-k8s"
 ```

-Private key: `cluster-init/terraform/k8s_terraform` (gitignored — do not commit)
+Private key: `cluster-init/terraform/k8s_terraform` (gitignored)
 Public key:  `cluster-init/terraform/k8s_terraform.pub` (gitignored — injected into VMs via cloud-init)

 ---
@@ -85,37 +94,66 @@ Enable:

 ### 2.2 Create Terraform user, role, token

-SSH into any Proxmox node as root:
+SSH into any Proxmox node as root, then run each command:
+
+**Create role:**

 ```bash
-# Create role
-pveum role add TerraformRole --privs \
-  "VM.Allocate,VM.Clone,VM.Config.Disk,VM.Config.CPU,\
-VM.Config.Memory,VM.Config.Network,VM.Config.Options,\
-VM.Config.CloudInit,VM.Config.CDROM,VM.PowerMgmt,VM.Audit,\
-Datastore.AllocateSpace,Datastore.AllocateTemplate,\
-Datastore.Allocate,Datastore.Audit,Sys.Audit,Sys.Modify,\
-Pool.Allocate,SDN.Use"
+pveum role add TerraformRole --privs "VM.Allocate,VM.Clone,VM.Config.Disk,VM.Config.CPU,VM.Config.Memory,VM.Config.Network,VM.Config.Options,VM.Config.CloudInit,VM.Config.CDROM,VM.PowerMgmt,VM.Audit,Datastore.AllocateSpace,Datastore.AllocateTemplate,Datastore.Allocate,Datastore.Audit,Sys.Audit,Sys.Modify,Pool.Allocate,SDN.Use"
+```

-# Create user + token — SAVE the secret, shown only once
+**Create user:**
+
+```bash
 pveum user add terraform@pve --comment "Terraform IaC"
+```
+
+**Create token** — SAVE the secret, shown only once:
+
+```bash
 pveum user token add terraform@pve terraform --privsep 0
+```

-# ACLs
+**ACLs:**
+
+```bash
 pveum acl modify /nodes --user terraform@pve --role TerraformRole
+```
+
+```bash
 pveum acl modify /nodes/proxmox03 --user terraform@pve --role TerraformRole
+```
+
+```bash
 pveum acl modify /storage/local --user terraform@pve --role TerraformRole
+```
+
+```bash
 pveum acl modify /storage/xpen --user terraform@pve --role TerraformRole
+```
+
+```bash
 pveum acl modify /pool/IaC --user terraform@pve --role TerraformRole
+```
+
+```bash
 pveum acl modify /sdn/zones/localnetwork/vmbr0 --user terraform@pve --role TerraformRole
+```

-# Required for proxmox_download_file — Sys.Audit + Sys.Modify must be at root
+**Required for proxmox_download_file — must be at root:**
+
+```bash
 pveum role add SysDownloadRole --privs "Sys.Audit,Sys.Modify"
-pveum acl modify / --user terraform@pve --role SysDownloadRole
+```

-# SDN — token needs explicit ACL (separate from user ACL)
-pveum acl modify /sdn/zones/localnetwork/vmbr0 \
-  --token terraform@pve!terraform --role TerraformRole
+```bash
+pveum acl modify / --user terraform@pve --role SysDownloadRole
+```
+
+**SDN — token needs explicit ACL (separate from user ACL):**
+
+```bash
+pveum acl modify /sdn/zones/localnetwork/vmbr0 --token terraform@pve!terraform --role TerraformRole
 ```

 Update `cluster-init/terraform/.env` with the token secret.
@@ -128,7 +166,13 @@ Update `cluster-init/terraform/.env` with the token secret.

 ```bash
 cd ~/k8s/cluster-init/terraform
+```
+
+```bash
 source .env
+```
+
+```bash
 terraform init
 ```

@@ -136,6 +180,9 @@ terraform init

 ```bash
 source .env
+```
+
+```bash
 terraform plan
 ```

@@ -148,22 +195,23 @@ Expected: 5 resources to create

 ### 3.3 Apply

+> Must use `-parallelism=1` — parallel disk clones to xpen NFS cause storage lock timeout.
+> Takes ~10-15 min (cloud image download ~1 min + 4 VMs × ~3 min each).
+
 ```bash
 terraform apply -parallelism=1
 ```

-> Must use `-parallelism=1` — parallel disk clones to xpen NFS cause storage lock timeout.
-
-Takes ~10-15 min (cloud image download ~1 min + 4 VMs × ~3 min each).
-
 ### 3.4 Post-apply state sync

 After first apply, run once to sync pool membership state:
+
 ```bash
 terraform apply -refresh-only
 ```

 Type `yes`. Then verify no further changes:
+
 ```bash
 terraform plan
 ```
@@ -171,52 +219,80 @@ terraform plan
 ### 3.5 Verify VMs

 ```bash
-ssh -i cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
-hostname    # master01
-free -h     # Swap: 0 (disabled by cloud-init)
+ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
+```
+
+```bash
+hostname
+```
+
+```bash
+free -h
 ```

 ---

 ## Phase 4 — Configure k8s with Ansible

-> Playbooks in `cluster-init/ansible/`
-
 ### 4.1 Run all playbooks (one command)

 ```bash
 cd ~/k8s/cluster-init/ansible
+```
+
+```bash
 ansible-playbook -i inventory.ini cluster.yml
 ```

-Or run individual playbooks:
+### 4.2 Or run individual playbooks

 ```bash
 cd ~/k8s/cluster-init/ansible
+```

-# All nodes — OS prerequisites (swap, kernel modules, sysctl)
+All nodes — OS prerequisites (swap, kernel modules, sysctl):
+
+```bash
 ansible-playbook -i inventory.ini playbooks/01-common.yml
+```

-# All nodes — /etc/hosts (nodes know each other by hostname)
+All nodes — /etc/hosts:
+
+```bash
 ansible-playbook -i inventory.ini playbooks/02-hosts.yml
+```

-# All nodes — container runtime (containerd)
+All nodes — container runtime (containerd):
+
+```bash
 ansible-playbook -i inventory.ini playbooks/03-containerd.yml
+```

-# All nodes — kubeadm/kubelet/kubectl
+All nodes — kubeadm/kubelet/kubectl:
+
+```bash
 ansible-playbook -i inventory.ini playbooks/04-kube-tools.yml
+```

-# master01 only — init cluster + CNI
+master01 only — init cluster + CNI:
+
+```bash
 ansible-playbook -i inventory.ini playbooks/05-master-init.yml
+```

-# worker01/02/03 — join cluster
+worker01/02/03 — join cluster:
+
+```bash
 ansible-playbook -i inventory.ini playbooks/06-worker-join.yml
 ```

-### 4.2 Verify cluster (from master01)
+### 4.3 Verify cluster (from master01)
+
+```bash
+ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
+```

 ```bash
-ssh -i ../terraform/k8s_terraform cloud-user@192.168.1.31
 kubectl get nodes
 ```

@@ -229,10 +305,18 @@ All 4 nodes should be `Ready`.
 ### 5.1 Install kubectl

 ```bash
-# Match the cluster version (1.32)
 curl -LO "https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl"
+```
+
+```bash
 chmod +x kubectl
+```
+
+```bash
 sudo mv kubectl /usr/local/bin/kubectl
+```
+
+```bash
 kubectl version --client
 ```

@@ -240,9 +324,10 @@ kubectl version --client

 ```bash
 mkdir -p ~/.kube
-scp -i ~/k8s/cluster-init/terraform/k8s_terraform \
-    cloud-user@192.168.1.31:/home/cloud-user/.kube/config \
-    ~/.kube/config
+```
+
+```bash
+scp -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31:/home/cloud-user/.kube/config ~/.kube/config
 ```

 ### 5.3 Add cluster nodes to client /etc/hosts
@@ -262,6 +347,9 @@ EOF

 ```bash
 kubectl get nodes
+```
+
+```bash
 kubectl get pods -A
 ```

@@ -271,7 +359,13 @@ kubectl get pods -A

 ```bash
 cd ~/k8s/cluster-init/terraform
+```
+
+```bash
 source .env
+```
+
+```bash
 terraform destroy -parallelism=1
 ```

@@ -282,13 +376,22 @@ Destroys all 4 VMs + cloud image file. Re-run Phase 3 to rebuild from scratch.
 ## SSH access

 ```bash
-ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31   # master01
-ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.35   # worker01
-ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.36   # worker02
-ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.37   # worker03
+ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
 ```

-Or add to `~/.ssh/config`:
+```bash
+ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.35
+```
+
+```bash
+ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.36
+```
+
+```bash
+ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.37
+```
+
+Or add to `~/.ssh/config` for shorthand (`ssh master01`, `ssh worker01`, etc.):

 ```
 Host master01
@@ -312,15 +415,13 @@ Host worker03
  IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
 ```

-Then: `ssh master01`, `ssh worker01`, etc.
-
 ---

 ## File reference

 ```
 k8s/cluster-init/
-├── RUNBOOK.md
+├── README.md
 ├── terraform/
 │   ├── .env                 # Terraform credentials (gitignored)
 │   ├── .env.example         # template — safe to commit
@@ -334,6 +435,7 @@ k8s/cluster-init/
 │   ├── k8s_terraform.pub    # SSH public key (gitignored)
 │   └── .gitignore
 └── ansible/
+    ├── cluster.yml          # run all playbooks in one command
    ├── inventory.ini
    ├── ansible.cfg
    └── playbooks/