duynguyen/cluster-init
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
duynguyen
2026-06-14 14:21:09 +07:00
commit 054835e2a7
17 changed files with 771 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

339
RUNBOOK.md Normal file
View File

@@ -0,0 +1,339 @@
# K8s Cluster Runbook — Build From Scratch
## Architecture
```
Client machine (192.168.1.39)
├── Terraform → Proxmox API → provision VMs
└── Ansible → SSH → configure k8s
Proxmox cluster
├── proxmox (node 1)
├── proxmox02 (node 2)
└── proxmox03 (node 3, 192.168.1.18) ← all VMs here
VMs (CentOS Stream 10, cloud-init)
├── master01 192.168.1.31 2 vCPU 8GB RAM 100GB disk
├── worker01 192.168.1.35 2 vCPU 12GB RAM 100GB disk
├── worker02 192.168.1.36 2 vCPU 12GB RAM 100GB disk
└── worker03 192.168.1.37 2 vCPU 12GB RAM 100GB disk
Storage: xpen (NFS NAS) — all VM disks
local — cloud image download + cloud-init drive
Pool: IaC
```
---
## Phase 1 — Client machine setup
### 1.1 Install tools
```bash
# Terraform
yum install -y yum-utils
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
yum install -y terraform
terraform version
# Ansible
pip3 install ansible
# or: dnf install ansible
# Git
yum install -y git
```
### 1.2 Clone repo
```bash
git clone <your-gitea-repo-url> ~/k8s
cd ~/k8s
```
### 1.3 Set up Terraform credentials
```bash
cp cluster-init/terraform/.env.example cluster-init/terraform/.env
# edit cluster-init/terraform/.env — fill in endpoint + token
```
`cluster-init/terraform/.env` is gitignored — never commit it.
### 1.4 Generate SSH keypair for VM access
> Skip if `cluster-init/terraform/k8s_terraform` already exists.
```bash
ssh-keygen -t ed25519 -f ~/k8s/cluster-init/terraform/k8s_terraform -N "" -C "terraform-k8s"
```
Private key: `cluster-init/terraform/k8s_terraform` (gitignored — do not commit)
Public key: `cluster-init/terraform/k8s_terraform.pub` (gitignored — injected into VMs via cloud-init)
---
## Phase 2 — Proxmox preparation
### 2.1 Enable local storage content types
**Proxmox UI → Datacenter → Storage → local → Edit → Content**
Enable:
- `Disk image (import)` — for cloud image download via Terraform
- `Snippets` — for cloud-init drive
### 2.2 Create Terraform user, role, token
SSH into any Proxmox node as root:
```bash
# Create role
pveum role add TerraformRole --privs \
"VM.Allocate,VM.Clone,VM.Config.Disk,VM.Config.CPU,\
VM.Config.Memory,VM.Config.Network,VM.Config.Options,\
VM.Config.CloudInit,VM.Config.CDROM,VM.PowerMgmt,VM.Audit,\
Datastore.AllocateSpace,Datastore.AllocateTemplate,\
Datastore.Allocate,Datastore.Audit,Sys.Audit,Sys.Modify,\
Pool.Allocate,SDN.Use"
# Create user + token — SAVE the secret, shown only once
pveum user add terraform@pve --comment "Terraform IaC"
pveum user token add terraform@pve terraform --privsep 0
# ACLs
pveum acl modify /nodes --user terraform@pve --role TerraformRole
pveum acl modify /nodes/proxmox03 --user terraform@pve --role TerraformRole
pveum acl modify /storage/local --user terraform@pve --role TerraformRole
pveum acl modify /storage/xpen --user terraform@pve --role TerraformRole
pveum acl modify /pool/IaC --user terraform@pve --role TerraformRole
pveum acl modify /sdn/zones/localnetwork/vmbr0 --user terraform@pve --role TerraformRole
# Required for proxmox_download_file — Sys.Audit + Sys.Modify must be at root
pveum role add SysDownloadRole --privs "Sys.Audit,Sys.Modify"
pveum acl modify / --user terraform@pve --role SysDownloadRole
# SDN — token needs explicit ACL (separate from user ACL)
pveum acl modify /sdn/zones/localnetwork/vmbr0 \
--token terraform@pve!terraform --role TerraformRole
```
Update `cluster-init/terraform/.env` with the token secret.
---
## Phase 3 — Provision VMs with Terraform
### 3.1 Init
```bash
cd ~/k8s/cluster-init/terraform
source .env
terraform init
```
### 3.2 Plan
```bash
source .env
terraform plan
```
Expected: 5 resources to create
- `proxmox_download_file.centos10`
- `proxmox_virtual_environment_vm.master`
- `proxmox_virtual_environment_vm.workers["worker01"]`
- `proxmox_virtual_environment_vm.workers["worker02"]`
- `proxmox_virtual_environment_vm.workers["worker03"]`
### 3.3 Apply
```bash
terraform apply -parallelism=1
```
> Must use `-parallelism=1` — parallel disk clones to xpen NFS cause storage lock timeout.
Takes ~10-15 min (cloud image download ~1 min + 4 VMs × ~3 min each).
### 3.4 Post-apply state sync
After first apply, run once to sync pool membership state:
```bash
terraform apply -refresh-only
```
Type `yes`. Then verify no further changes:
```bash
terraform plan
```
### 3.5 Verify VMs
```bash
ssh -i cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31
hostname # master01
free -h # Swap: 0 (disabled by cloud-init)
```
---
## Phase 4 — Configure k8s with Ansible
> Playbooks in `cluster-init/ansible/`
### 4.1 Run all playbooks
```bash
cd ~/k8s/cluster-init/ansible
# All nodes — OS prerequisites (swap, kernel modules, sysctl)
ansible-playbook -i inventory.ini playbooks/01-common.yml
# All nodes — /etc/hosts (nodes know each other by hostname)
ansible-playbook -i inventory.ini playbooks/02-hosts.yml
# All nodes — container runtime (containerd)
ansible-playbook -i inventory.ini playbooks/03-containerd.yml
# All nodes — kubeadm/kubelet/kubectl
ansible-playbook -i inventory.ini playbooks/04-kube-tools.yml
# master01 only — init cluster + CNI
ansible-playbook -i inventory.ini playbooks/05-master-init.yml
# worker01/02/03 — join cluster
ansible-playbook -i inventory.ini playbooks/06-worker-join.yml
```
### 4.2 Verify cluster (from master01)
```bash
ssh -i ../terraform/k8s_terraform cloud-user@192.168.1.31
kubectl get nodes
```
All 4 nodes should be `Ready`.
---
## Phase 5 — kubectl from client machine (192.168.1.39)
### 5.1 Install kubectl
```bash
# Match the cluster version (1.32)
curl -LO "https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl
kubectl version --client
```
### 5.2 Copy kubeconfig from master01
```bash
mkdir -p ~/.kube
scp -i ~/k8s/cluster-init/terraform/k8s_terraform \
cloud-user@192.168.1.31:/home/cloud-user/.kube/config \
~/.kube/config
```
### 5.3 Add cluster nodes to client /etc/hosts
```bash
sudo tee -a /etc/hosts <<'EOF'
# k8s cluster nodes
192.168.1.31 master01
192.168.1.35 worker01
192.168.1.36 worker02
192.168.1.37 worker03
EOF
```
### 5.4 Verify
```bash
kubectl get nodes
kubectl get pods -A
```
---
## Teardown
```bash
cd ~/k8s/cluster-init/terraform
source .env
terraform destroy -parallelism=1
```
Destroys all 4 VMs + cloud image file. Re-run Phase 3 to rebuild from scratch.
---
## SSH access
```bash
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.31 # master01
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.35 # worker01
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.36 # worker02
ssh -i ~/k8s/cluster-init/terraform/k8s_terraform cloud-user@192.168.1.37 # worker03
```
Or add to `~/.ssh/config`:
```
Host master01
HostName 192.168.1.31
User cloud-user
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
Host worker01
HostName 192.168.1.35
User cloud-user
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
Host worker02
HostName 192.168.1.36
User cloud-user
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
Host worker03
HostName 192.168.1.37
User cloud-user
IdentityFile ~/k8s/cluster-init/terraform/k8s_terraform
```
Then: `ssh master01`, `ssh worker01`, etc.
---
## File reference
```
k8s/cluster-init/
├── RUNBOOK.md
├── terraform/
│ ├── .env # Terraform credentials (gitignored)
│ ├── .env.example # template — safe to commit
│ ├── main.tf # provider config
│ ├── variables.tf # input variables
│ ├── terraform.tfvars # non-secret defaults
│ ├── cloud_image.tf # CentOS Stream 10 cloud image download
│ ├── vms.tf # master01 + worker01/02/03
│ ├── outputs.tf # IPs + SSH command
│ ├── k8s_terraform # SSH private key (gitignored)
│ ├── k8s_terraform.pub # SSH public key (gitignored)
│ └── .gitignore
└── ansible/
├── inventory.ini
├── ansible.cfg
└── playbooks/
├── 01-common.yml
├── 02-hosts.yml
├── 03-containerd.yml
├── 04-kube-tools.yml
├── 05-master-init.yml
└── 06-worker-join.yml
```

13
ansible/ansible.cfg Normal file
View File

@@ -0,0 +1,13 @@
[defaults]
inventory = inventory.ini
remote_user = cloud-user
private_key_file = ../terraform/k8s_terraform
host_key_checking = False
stdout_callback = default
result_format = yaml
interpreter_python = /usr/bin/python3.12
deprecation_warnings = False
[privilege_escalation]
become = True
become_method = sudo

11
ansible/inventory.ini Normal file
View File

@@ -0,0 +1,11 @@
[masters]
master01 ansible_host=192.168.1.31
[workers]
worker01 ansible_host=192.168.1.35
worker02 ansible_host=192.168.1.36
worker03 ansible_host=192.168.1.37
[k8s:children]
masters
workers

40
ansible/playbooks/01-common.yml Normal file
View File

@@ -0,0 +1,40 @@
---
- name: Common OS prerequisites for all k8s nodes
hosts: k8s
become: true
tasks:
- name: Disable swap now
command: swapoff -a
when: ansible_facts['swaptotal_mb'] > 0
- name: Remove swap from /etc/fstab
replace:
path: /etc/fstab
regexp: '^([^#].*\s+swap\s+.*)$'
replace: '# \1'
- name: Load kernel modules at boot
copy:
dest: /etc/modules-load.d/k8s.conf
content: |
overlay
br_netfilter
- name: Load kernel modules now
modprobe:
name: "{{ item }}"
loop:
- overlay
- br_netfilter
- name: Set sysctl for k8s networking
copy:
dest: /etc/sysctl.d/k8s.conf
content: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
- name: Apply sysctl
command: sysctl --system

15
ansible/playbooks/02-hosts.yml Normal file
View File

@@ -0,0 +1,15 @@
---
- name: Configure /etc/hosts for all nodes
hosts: k8s
become: true
tasks:
- name: Add cluster nodes to /etc/hosts
blockinfile:
path: /etc/hosts
marker: "# {mark} k8s cluster nodes"
block: |
192.168.1.31 master01
192.168.1.35 worker01
192.168.1.36 worker02
192.168.1.37 worker03

35
ansible/playbooks/03-containerd.yml Normal file
View File

@@ -0,0 +1,35 @@
---
- name: Install and configure containerd
hosts: k8s
become: true
tasks:
- name: Add Docker CE repo (provides containerd.io)
get_url:
url: https://download.docker.com/linux/centos/docker-ce.repo
dest: /etc/yum.repos.d/docker-ce.repo
- name: Install containerd
dnf:
name: containerd.io
state: present
- name: Create containerd config directory
file:
path: /etc/containerd
state: directory
- name: Generate default containerd config (overwrites Docker CE default which disables CRI)
shell: containerd config default > /etc/containerd/config.toml
- name: Enable SystemdCgroup in containerd config
replace:
path: /etc/containerd/config.toml
regexp: 'SystemdCgroup = false'
replace: 'SystemdCgroup = true'
- name: Enable and start containerd
systemd:
name: containerd
enabled: true
state: restarted

35
ansible/playbooks/04-kube-tools.yml Normal file
View File

@@ -0,0 +1,35 @@
---
- name: Install kubeadm, kubelet, kubectl
hosts: k8s
become: true
vars:
kube_version: "1.32"
tasks:
- name: Add Kubernetes repo
copy:
dest: /etc/yum.repos.d/kubernetes.repo
content: |
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v{{ kube_version }}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v{{ kube_version }}/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
- name: Install kubeadm, kubelet, kubectl
dnf:
name:
- kubelet
- kubeadm
- kubectl
state: present
disable_excludes: kubernetes
- name: Enable kubelet
systemd:
name: kubelet
enabled: true
state: started

67
ansible/playbooks/05-master-init.yml Normal file
View File

@@ -0,0 +1,67 @@
---
- name: Initialize Kubernetes master
hosts: masters
become: true
vars:
pod_network_cidr: "10.244.0.0/16" # Flannel default
tasks:
- name: Check if cluster already initialized
stat:
path: /etc/kubernetes/admin.conf
register: kubeadm_init_done
- name: Run kubeadm init
command: >
kubeadm init
--apiserver-advertise-address={{ ansible_host }}
--pod-network-cidr={{ pod_network_cidr }}
when: not kubeadm_init_done.stat.exists
register: kubeadm_output
- name: Create .kube dir for cloud-user
file:
path: /home/cloud-user/.kube
state: directory
mode: "0700"
owner: cloud-user
group: cloud-user
- name: Copy admin.conf to cloud-user kubeconfig
copy:
src: /etc/kubernetes/admin.conf
dest: /home/cloud-user/.kube/config
remote_src: true
owner: cloud-user
group: cloud-user
mode: "0600"
- name: Create .kube dir for root (enables sudo kubectl)
file:
path: /root/.kube
state: directory
mode: "0700"
- name: Copy admin.conf to root kubeconfig
copy:
src: /etc/kubernetes/admin.conf
dest: /root/.kube/config
remote_src: true
mode: "0600"
- name: Install Flannel CNI
command: kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
environment:
KUBECONFIG: /etc/kubernetes/admin.conf
- name: Get join command
command: kubeadm token create --print-join-command
register: join_command
- name: Save join command to local file
local_action:
module: copy
content: "{{ join_command.stdout }}"
dest: /tmp/k8s_join_command.sh
mode: "0600"

21
ansible/playbooks/06-worker-join.yml Normal file
View File

@@ -0,0 +1,21 @@
---
- name: Join workers to Kubernetes cluster
hosts: workers
become: true
tasks:
- name: Check if already joined
stat:
path: /etc/kubernetes/kubelet.conf
register: kubelet_conf
- name: Read join command from local file
local_action:
module: slurp
src: /tmp/k8s_join_command.sh
register: join_command_b64
when: not kubelet_conf.stat.exists
- name: Join cluster
command: "{{ join_command_b64.content | b64decode | trim }}"
when: not kubelet_conf.stat.exists

2
terraform/.env.example Normal file
View File

@@ -0,0 +1,2 @@
export TF_VAR_proxmox_endpoint=https://<proxmox-node-ip>:8006
export TF_VAR_proxmox_api_token=terraform@pve!terraform=<token-secret>

10
terraform/.gitignore vendored Normal file
View File

@@ -0,0 +1,10 @@
.env
.terraform/
.terraform.lock.hcl
terraform.tfstate
terraform.tfstate.backup
*.tfstate
*.tfstate.*
k8s_terraform
k8s_terraform.pub
*.tfvars.local

8
terraform/cloud_image.tf Normal file
View File

@@ -0,0 +1,8 @@
resource "proxmox_download_file" "centos10" {
content_type = "import"
datastore_id = "local"
node_name = var.proxmox_node
url = "https://cloud.centos.org/centos/10-stream/x86_64/images/CentOS-Stream-GenericCloud-10-latest.x86_64.qcow2"
file_name = "centos-stream-10-cloud.qcow2"
overwrite = false
}

14
terraform/main.tf Normal file
View File

@@ -0,0 +1,14 @@
terraform {
required_providers {
proxmox = {
source = "bpg/proxmox"
version = "~> 0.107"
}
}
}
provider "proxmox" {
endpoint = var.proxmox_endpoint
api_token = var.proxmox_api_token
insecure = true
}

11
terraform/outputs.tf Normal file
View File

@@ -0,0 +1,11 @@
output "master_ip" {
value = "192.168.1.31"
}
output "worker_ips" {
value = { for k, v in proxmox_virtual_environment_vm.workers : k => v.initialization[0].ip_config[0].ipv4[0].address }
}
output "ssh_command" {
value = "ssh -i ${path.module}/k8s_terraform cloud-user@192.168.1.31"
}

5
terraform/terraform.tfvars Normal file
View File

@@ -0,0 +1,5 @@
proxmox_endpoint = "https://192.168.1.18:8006"
proxmox_node = "proxmox03"
proxmox_pool = "IaC"
gateway = "192.168.1.1"
dns_server = "192.168.1.1"

24
terraform/variables.tf Normal file
View File

@@ -0,0 +1,24 @@
variable "proxmox_endpoint" {
description = "Proxmox API endpoint URL"
}
variable "proxmox_api_token" {
description = "Proxmox API token (user@realm!tokenid=secret)"
sensitive = true
}
variable "proxmox_node" {
default = "proxmox03"
}
variable "proxmox_pool" {
default = "IaC"
}
variable "gateway" {
default = "192.168.1.1"
}
variable "dns_server" {
default = "192.168.1.1"
}

121
terraform/vms.tf Normal file
View File

@@ -0,0 +1,121 @@
resource "proxmox_virtual_environment_vm" "master" {
name = "master01"
vm_id = 131
node_name = var.proxmox_node
pool_id = var.proxmox_pool
stop_on_destroy = true
cpu {
cores = 2
type = "host"
}
memory {
dedicated = 8192
}
scsi_hardware = "virtio-scsi-single"
disk {
datastore_id = "xpen"
import_from = proxmox_download_file.centos10.id
interface = "scsi0"
iothread = true
size = 100
}
network_device {
bridge = "vmbr0"
model = "virtio"
}
boot_order = ["scsi0"]
agent {
enabled = true
}
initialization {
datastore_id = "local"
ip_config {
ipv4 {
address = "192.168.1.31/24"
gateway = var.gateway
}
}
dns {
servers = [var.dns_server]
}
user_account {
username = "cloud-user"
keys = [trimspace(file("${path.module}/k8s_terraform.pub"))]
}
}
}
resource "proxmox_virtual_environment_vm" "workers" {
for_each = {
worker01 = { vm_id = 135, ip = "192.168.1.35" }
worker02 = { vm_id = 136, ip = "192.168.1.36" }
worker03 = { vm_id = 137, ip = "192.168.1.37" }
}
name = each.key
vm_id = each.value.vm_id
node_name = var.proxmox_node
pool_id = var.proxmox_pool
stop_on_destroy = true
cpu {
cores = 2
type = "host"
}
memory {
dedicated = 12288
}
scsi_hardware = "virtio-scsi-single"
disk {
datastore_id = "xpen"
import_from = proxmox_download_file.centos10.id
interface = "scsi0"
iothread = true
size = 100
}
network_device {
bridge = "vmbr0"
model = "virtio"
}
boot_order = ["scsi0"]
agent {
enabled = true
}
initialization {
datastore_id = "local"
ip_config {
ipv4 {
address = "${each.value.ip}/24"
gateway = var.gateway
}
}
dns {
servers = [var.dns_server]
}
user_account {
username = "cloud-user"
keys = [trimspace(file("${path.module}/k8s_terraform.pub"))]
}
}
}