티스토리 뷰
※ Kubernetes the hard way의 목적
Kubernetes the hard way의 핵심은 "자동화 도구(Managed Services나 Installer)를 전혀 사용하지 않고,
모든 구성 요소를 밑바닥부터 직접 설치하고 설정하는 것. kubeadm 도구조차 사용하지 않고 설치하는 것이 특징이며 각각의 구성요소들을 직접 한땀한땀 설치하므로써 kubernetes의 내부 구조를 이해하고 분석할 수 있는 능력을 키우기 위함
본 자료는 Cloud@net 스터디 자료로 실제 교육 내용을 토대로 현재 프로덕션 환경에 적용하기 위한 목적이어서 실제 물리서버로 설치하는 방식으로 실습을 진행하였음
▶ 사전 준비물
☞ 하이퍼바이저 물리 서버 1식 (VM 4식)
- 하이퍼바이저 : Proxmox
- 서버 모델 : HP Gen G7 (CPU : 4 core, MEM : 250G. HDD : 2T)
- VM : 총 4식 (운영체제 : Debian 12)
| NAME | Description | CPU | MEM | NIC | HOSTNAME |
| jumpbox | Administration host | 2 | 1536 MB | 192.168.7.245 | jumpbox |
| server | Kubernetes server | 2 | 2GB | 192.168.7.246 | server.kubernetes.local server |
| node-0 | Kubernetes worker | 2 | 2GB | 192.168.7.247 | node-0.kubernetes.local node-0 |
| node-1 | Kubernetes worker | 2 | 2GB | 192.168.7.248 | node-1.kubernetes.local node-1 |
☞ 스터디 수업에서는 NIC 이 외부와 내부로 분리하여 구성하고 있으나
실제 프로덕션 환경에서 구축하는 방식으로 실습을 진행하였기 때문에
VirtualBox와 Vegrant를 사용하지 않고 베어메탈 서버의 단일 네트워크
환경으로 구축하였음
▶ 실습 구성도
☞ 우분투 로컬 서버 (KIND 설치) 노드 - 1식
우분투 노드 1식은 kubernetes the hardway 의 설치 및 구성 정보를 KIND를 이용하여 서로 비교하기 위해 필요하며 본인은 테스트를 위한 베어메탈 서버 자원이 이미 확보된 상황이어서 실습할 때 WSL을 사용할 필요가 없음. 우분투가 설치된 프러덕션 환경을 이용하여 실습할 경우 스터디 내용 처럼 VirtualBox와 baygrant 그리고 WSL 관련 내용이 불필요하므로 제외시켰음
▶ 우분투 가상화 노드에 KIND 설치
더보기
# 도커 설치
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh ./get-docker.sh
...
# 설치 확인
docker info
docker ps
sudo systemctl status docker
cat /etc/group | grep docker
# KIND 및 관리 툴 설치
# 기본 사용자 디렉터리 이동
cd $PWD
pwd
#
sudo systemctl stop apparmor && sudo systemctl disable apparmor
#
sudo apt update && sudo apt-get install bridge-utils net-tools jq tree unzip kubectx kubecolor -y
# Install Kind
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.31.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
kind --version
# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv ./kubectl /usr/bin
sudo kubectl version --client=true
# Install Helm
curl -s https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
helm version
# Source the completion
source <(kubectl completion bash)
echo 'source <(kubectl completion bash)' >> ~/.bashrc
# Alias kubectl to k
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -o default -F __start_kubectl k' >> ~/.bashrc
# Install Kubeps & Setting PS1
git clone https://github.com/jonmosco/kube-ps1.git
echo -e "source $PWD/kube-ps1/kube-ps1.sh" >> ~/.bashrc
cat <<"EOT" >> ~/.bashrc
KUBE_PS1_SYMBOL_ENABLE=true
function get_cluster_short() {
echo "$1" | cut -d . -f1
}
KUBE_PS1_CLUSTER_FUNCTION=get_cluster_short
KUBE_PS1_SUFFIX=') '
PS1='$(kube_ps1)'$PS1
EOT
# .bashrc 적용을 위해서 logout 후 터미널 다시 접속 하자
exit
# KIND로 K8S 배포
# 클러스터 배포 전 확인
docker ps
# Create a cluster with kind
kind create cluster --name myk8s --image kindest/node:v1.32.8 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30000
hostPort: 30000
- containerPort: 30001
hostPort: 30001
- role: worker
EOF
# 확인
kind get nodes --name myk8s
kubens default
# kind 는 별도 도커 네트워크 생성 후 사용 : 기본값 172.18.0.0/16
docker network ls
docker inspect kind | jq
# k8s api 주소 확인 : 어떻게 로컬에서 접속이 되는 걸까?
kubectl cluster-info
# 노드 정보 확인 : CRI 는 containerd 사용
kubectl get node -o wide
# 파드 정보 확인 : CNI 는 kindnet 사용
kubectl get pod -A -o wide
# 네임스페이스 확인 >> 도커 컨테이너에서 배운 네임스페이스와 다릅니다!
kubectl get namespaces
# 컨트롤플레인/워커 노드(컨테이너) 확인 : 도커 컨테이너 이름은 myk8s-control-plane , myk8s-worker 임을 확인
docker ps
docker images
docker exec -it myk8s-control-plane ss -tnlp
# 디버그용 내용 출력에 ~/.kube/config 권한 인증 로드
kubectl get pod -v6
# kube config 파일 확인 : "server: https://127.0.0.1:YYYYY" 127.0.0.1:Port로 접속 가능을 확인!
cat ~/.kube/config
ls -l ~/.kube/config
(참고) 클러스터 삭제
# 클러스터 삭제
kind delete cluster --name myk8s
docker ps
cat ~/.kube/config
K8S the hard way 클러스터 설치
1. 노드 공통
#!/usr/bin/env bash
echo ">>>> Proxmox VM Initial Config Start <<<<"
# [TASK 1] 사용자 설정 및 시간대
# Cloud-init 기본 사용자인 'debian' 기준 (vagrant 대신 debian 사용)
echo "alias vi=vim" >> /etc/profile
ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
# [TASK 2] AppArmor 비활성화 (K8s 호환성)
systemctl stop apparmor && systemctl disable apparmor >/dev/null 2>&1
# [TASK 3] SWAP 비활성화 (K8s 필수 설정)
swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab
# [TASK 4] 필수 패키지 설치
apt update -qq
apt install tree git jq yq unzip vim sshpass -y -qq
# [TASK 5] Root 및 Debian 사용자 패스워드 설정
echo "root:qwe123" | chpasswd
echo "debian:qwe123" | chpasswd
# [TASK 6] SSH 설정 수정 (Root 접속 허용)
sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
systemctl restart sshd
# [TASK 7] 로컬 DNS 설정 (/etc/hosts) - 요청하신 IP 반영
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
cat << EOF >> /etc/hosts
192.168.7.245 jumpbox
192.168.7.246 server.kubernetes.local server
192.168.7.247 node-0.kubernetes.local node-0
192.168.7.248 node-1.kubernetes.local node-1
EOF
echo ">>>> Initial Config End <<<<"
2. jumpbox 가상머신
# 사용자 확인
whoami
pwd
# OS version 확인
cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
# AppArmor 상태 확인
aa-status
systemctl is-active apparmor
inactive
# /etc/hosts 파일 내용 확인
# https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_hostname_resolution
cat /etc/hosts
Set Up The Jumpbox
설치에 필요한 파일 등 준비 - Docs
설치 항목버전
| 설치 항목 | 버전 |
| k8s 관련 (kube-apiserver, kubelet 등) | 1.32.2 |
| etcd | 3.6.0 |
| containerd | 2.1.0 |
| runc | 1.3.0 |
# 툴 설치 : 이미 적용되어 있음
apt-get update && apt install tree git jq yq unzip vim sshpass -y
# Sync GitHub Repository
pwd
git clone --depth 1 https://github.com/kelseyhightower/kubernetes-the-hard-way.git
cd kubernetes-the-hard-way
tree
pwd
(확인)
debian@jumpbox:~$ pwd
git clone --depth 1 https://github.com/kelseyhightower/kubernetes-the-hard-way.git
cd kubernetes-the-hard-way
tree
pwd
/home/debian
Cloning into 'kubernetes-the-hard-way'...
remote: Enumerating objects: 41, done.
remote: Counting objects: 100% (41/41), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 41 (delta 3), reused 14 (delta 1), pack-reused 0 (from 0)
Receiving objects: 100% (41/41), 29.27 KiB | 3.66 MiB/s, done.
Resolving deltas: 100% (3/3), done.
.
├── CONTRIBUTING.md
├── COPYRIGHT.md
├── LICENSE
├── README.md
├── ca.conf
├── configs
│ ├── 10-bridge.conf
│ ├── 99-loopback.conf
│ ├── containerd-config.toml
│ ├── encryption-config.yaml
│ ├── kube-apiserver-to-kubelet.yaml
│ ├── kube-proxy-config.yaml
│ ├── kube-scheduler.yaml
│ └── kubelet-config.yaml
├── docs
│ ├── 01-prerequisites.md
│ ├── 02-jumpbox.md
│ ├── 03-compute-resources.md
│ ├── 04-certificate-authority.md
│ ├── 05-kubernetes-configuration-files.md
│ ├── 06-data-encryption-keys.md
│ ├── 07-bootstrapping-etcd.md
│ ├── 08-bootstrapping-kubernetes-controllers.md
│ ├── 09-bootstrapping-kubernetes-workers.md
│ ├── 10-configuring-kubectl.md
│ ├── 11-pod-network-routes.md
│ ├── 12-smoke-test.md
│ └── 13-cleanup.md
├── downloads-amd64.txt
├── downloads-arm64.txt
└── units
├── containerd.service
├── etcd.service
├── kube-apiserver.service
├── kube-controller-manager.service
├── kube-proxy.service
├── kube-scheduler.service
└── kubelet.service
4 directories, 35 files
/home/debian/kubernetes-the-hard-way
debian@jumpbox:~/kubernetes-the-hard-way$ pwd
/home/debian/kubernetes-the-hard-way
# Download Binaries : k8s 구성을 위한 컴포넌트 다운로드
# CPU 아키텍처 확인
dpkg --print-architecture
arm64 # macOS 사용자
amd64 # Windows 사용자
debian@jumpbox:~/kubernetes-the-hard-way$ dpkg --print-architecture
amd64
# CPU 아키텍처 별 다운로드 목록 정보 다름
ls -l downloads-*
-rw-r--r-- 1 root root 839 Jan 4 10:30 downloads-amd64.txt
-rw-r--r-- 1 root root 839 Jan 4 10:30 downloads-arm64.txt
# https://kubernetes.io/releases/download/
cat downloads-$(dpkg --print-architecture).txt
https://dl.k8s.io/v1.32.3/bin/linux/amd64/kubectl
https://dl.k8s.io/v1.32.3/bin/linux/amd64/kube-apiserver
https://dl.k8s.io/v1.32.3/bin/linux/amd64/kube-controller-manager
https://dl.k8s.io/v1.32.3/bin/linux/amd64/kube-scheduler
https://dl.k8s.io/v1.32.3/bin/linux/amd64/kube-proxy
https://dl.k8s.io/v1.32.3/bin/linux/amd64/kubelet
https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.32.0/crictl-v1.32.0-linux-amd64.tar.gz
https://github.com/opencontainers/runc/releases/download/v1.3.0-rc.1/runc.amd64
https://github.com/containernetworking/plugins/releases/download/v1.6.2/cni-plugins-linux-amd64-v1.6.2.tgz
https://github.com/containerd/containerd/releases/download/v2.1.0-beta.0/containerd-2.1.0-beta.0-linux-amd64.tar.gz
https://github.com/etcd-io/etcd/releases/download/v3.6.0-rc.3/etcd-v3.6.0-rc.3-linux-amd64.tar.gz
# wget 으로 다운로드 실행 : 500MB Size 정도
wget -q --show-progress \
--https-only \
--timestamping \
-P downloads \
-i downloads-$(dpkg --print-architecture).txt
kubectl 100%[=================================================================================================================>] 54.67M 4.27MB/s in 13s
kube-apiserver 100%[=================================================================================================================>] 88.94M 4.13MB/s in 21s
kube-controller-manager 100%[=================================================================================================================>] 82.00M 3.98MB/s in 21s
kube-scheduler 100%[=================================================================================================================>] 62.79M 3.61MB/s in 16s
kube-proxy 100%[=================================================================================================================>] 63.75M 4.02MB/s in 15s
kubelet 100%[=================================================================================================================>] 73.82M 3.85MB/s in 19s
crictl-v1.32.0-linux-amd64.tar.gz 100%[=================================================================================================================>] 18.21M 4.18MB/s in 4.4s
runc.amd64 100%[=================================================================================================================>] 11.30M 4.53MB/s in 2.5s
cni-plugins-linux-amd64-v1.6.2.tgz 100%[=================================================================================================================>] 50.35M 3.55MB/s in 13s
containerd-2.1.0-beta.0-linux-amd64.tar.gz 100%[=================================================================================================================>] 37.01M 3.38MB/s in 10s
etcd-v3.6.0-rc.3-linux-amd64.tar.gz 100%[=================================================================================================================>] 22.48M 3.84MB/s in 5.9s
# 확인
ls -oh downloads
total 566M
-rw-r--r-- 1 debian 51M Jan 7 2025 cni-plugins-linux-amd64-v1.6.2.tgz
-rw-r--r-- 1 debian 38M Mar 18 2025 containerd-2.1.0-beta.0-linux-amd64.tar.gz
-rw-r--r-- 1 debian 19M Dec 9 2024 crictl-v1.32.0-linux-amd64.tar.gz
-rw-r--r-- 1 debian 23M Mar 28 2025 etcd-v3.6.0-rc.3-linux-amd64.tar.gz
-rw-r--r-- 1 debian 89M Mar 12 2025 kube-apiserver
-rw-r--r-- 1 debian 83M Mar 12 2025 kube-controller-manager
-rw-r--r-- 1 debian 64M Mar 12 2025 kube-proxy
-rw-r--r-- 1 debian 63M Mar 12 2025 kube-scheduler
-rw-r--r-- 1 debian 55M Mar 12 2025 kubectl
-rw-r--r-- 1 debian 74M Mar 12 2025 kubelet
-rw-r--r-- 1 debian 12M Mar 4 2025 runc.amd64
# Extract the component binaries from the release archives and organize them under the downloads directory.
ARCH=$(dpkg --print-architecture)
echo $ARCH
mkdir -p downloads/{client,cni-plugins,controller,worker}
tree -d downloads
downloads
├── client
├── cni-plugins
├── controller
└── worker
5 directories
# 압축 풀기
tar -xvf downloads/crictl-v1.32.0-linux-${ARCH}.tar.gz \
-C downloads/worker/ && tree -ug downloads
(확인)
crictl
[debian debian ] downloads
├── [debian debian ] client
├── [debian debian ] cni-plugins
├── [debian debian ] cni-plugins-linux-amd64-v1.6.2.tgz
├── [debian debian ] containerd-2.1.0-beta.0-linux-amd64.tar.gz
├── [debian debian ] controller
├── [debian debian ] crictl-v1.32.0-linux-amd64.tar.gz
├── [debian debian ] etcd-v3.6.0-rc.3-linux-amd64.tar.gz
├── [debian debian ] kube-apiserver
├── [debian debian ] kube-controller-manager
├── [debian debian ] kube-proxy
├── [debian debian ] kube-scheduler
├── [debian debian ] kubectl
├── [debian debian ] kubelet
├── [debian debian ] runc.amd64
└── [debian debian ] worker
└── [debian debian ] crictl
5 directories, 12 files
tar -xvf downloads/containerd-2.1.0-beta.0-linux-${ARCH}.tar.gz \
--strip-components 1 \
-C downloads/worker/ && tree -ug downloads
bin/containerd-shim-runc-v2
bin/containerd
bin/containerd-stress
bin/ctr
[debian debian ] downloads
├── [debian debian ] client
├── [debian debian ] cni-plugins
├── [debian debian ] cni-plugins-linux-amd64-v1.6.2.tgz
├── [debian debian ] containerd-2.1.0-beta.0-linux-amd64.tar.gz
├── [debian debian ] controller
├── [debian debian ] crictl-v1.32.0-linux-amd64.tar.gz
├── [debian debian ] etcd-v3.6.0-rc.3-linux-amd64.tar.gz
├── [debian debian ] kube-apiserver
├── [debian debian ] kube-controller-manager
├── [debian debian ] kube-proxy
├── [debian debian ] kube-scheduler
├── [debian debian ] kubectl
├── [debian debian ] kubelet
├── [debian debian ] runc.amd64
└── [debian debian ] worker
├── [debian debian ] containerd
├── [debian debian ] containerd-shim-runc-v2
├── [debian debian ] containerd-stress
├── [debian debian ] crictl
└── [debian debian ] ctr
5 directories, 16 files
tar -xvf downloads/cni-plugins-linux-${ARCH}-v1.6.2.tgz \
-C downloads/cni-plugins/ && tree -ug downloads
./
./ipvlan
./tap
./loopback
./host-device
./README.md
./portmap
./ptp
./vlan
./bridge
./firewall
./LICENSE
./macvlan
./dummy
./bandwidth
./vrf
./tuning
./static
./dhcp
./host-local
./sbr
[debian debian ] downloads
├── [debian debian ] client
├── [debian debian ] cni-plugins
│ ├── [debian debian ] LICENSE
│ ├── [debian debian ] README.md
│ ├── [debian debian ] bandwidth
│ ├── [debian debian ] bridge
│ ├── [debian debian ] dhcp
│ ├── [debian debian ] dummy
│ ├── [debian debian ] firewall
│ ├── [debian debian ] host-device
│ ├── [debian debian ] host-local
│ ├── [debian debian ] ipvlan
│ ├── [debian debian ] loopback
│ ├── [debian debian ] macvlan
│ ├── [debian debian ] portmap
│ ├── [debian debian ] ptp
│ ├── [debian debian ] sbr
│ ├── [debian debian ] static
│ ├── [debian debian ] tap
│ ├── [debian debian ] tuning
│ ├── [debian debian ] vlan
│ └── [debian debian ] vrf
├── [debian debian ] cni-plugins-linux-amd64-v1.6.2.tgz
├── [debian debian ] containerd-2.1.0-beta.0-linux-amd64.tar.gz
├── [debian debian ] controller
├── [debian debian ] crictl-v1.32.0-linux-amd64.tar.gz
├── [debian debian ] etcd-v3.6.0-rc.3-linux-amd64.tar.gz
├── [debian debian ] kube-apiserver
├── [debian debian ] kube-controller-manager
├── [debian debian ] kube-proxy
├── [debian debian ] kube-scheduler
├── [debian debian ] kubectl
├── [debian debian ] kubelet
├── [debian debian ] runc.amd64
└── [debian debian ] worker
├── [debian debian ] containerd
├── [debian debian ] containerd-shim-runc-v2
├── [debian debian ] containerd-stress
├── [debian debian ] crictl
└── [debian debian ] ctr
5 directories, 36 files
## --strip-components 1 : etcd-v3.6.0-rc.3-linux-amd64/etcd 경로의 앞부분(디렉터리)을 제거
tar -xvf downloads/etcd-v3.6.0-rc.3-linux-${ARCH}.tar.gz \
-C downloads/ \
--strip-components 1 \
etcd-v3.6.0-rc.3-linux-${ARCH}/etcdctl \
etcd-v3.6.0-rc.3-linux-${ARCH}/etcd && tree -ug downloads
# 확인
tree downloads/worker/
tree downloads/cni-plugins
ls -l downloads/{etcd,etcdctl}
debian@jumpbox:~/kubernetes-the-hard-way$ tree downloads/worker/
downloads/worker/
├── containerd
├── containerd-shim-runc-v2
├── containerd-stress
├── crictl
└── ctr
1 directory, 5 files
debian@jumpbox:~/kubernetes-the-hard-way$
debian@jumpbox:~/kubernetes-the-hard-way$
debian@jumpbox:~/kubernetes-the-hard-way$ tree downloads/cni-plugins
downloads/cni-plugins
├── LICENSE
├── README.md
├── bandwidth
├── bridge
├── dhcp
├── dummy
├── firewall
├── host-device
├── host-local
├── ipvlan
├── loopback
├── macvlan
├── portmap
├── ptp
├── sbr
├── static
├── tap
├── tuning
├── vlan
└── vrf
1 directory, 20 files
debian@jumpbox:~/kubernetes-the-hard-way$ ls -l downloads/{etcd,etcdctl}
-rwxr-xr-x 1 debian debian 25219224 Mar 28 2025 downloads/etcd
-rwxr-xr-x 1 debian debian 16421016 Mar 28 2025 downloads/etcdctl
# 파일 이동
mv downloads/{etcdctl,kubectl} downloads/client/
mv downloads/{etcd,kube-apiserver,kube-controller-manager,kube-scheduler} downloads/controller/
mv downloads/{kubelet,kube-proxy} downloads/worker/
mv downloads/runc.${ARCH} downloads/worker/runc
# 확인
tree downloads/client/
tree downloads/controller/
tree downloads/worker/
tree downloads/controller/
tree downloads/worker/
downloads/client/
├── etcdctl
└── kubectl
1 directory, 2 files
downloads/controller/
├── etcd
├── kube-apiserver
├── kube-controller-manager
└── kube-scheduler
1 directory, 4 files
downloads/worker/
├── containerd
├── containerd-shim-runc-v2
├── containerd-stress
├── crictl
├── ctr
├── kube-proxy
├── kubelet
└── runc
1 directory, 8 files
# 불필요한 압축 파일 제거
ls -l downloads/*gz
rm -rf downloads/*gz
-rw-r--r-- 1 debian debian 52794236 Jan 7 2025 downloads/cni-plugins-linux-amd64-v1.6.2.tgz
-rw-r--r-- 1 debian debian 38807044 Mar 18 2025 downloads/containerd-2.1.0-beta.0-linux-amd64.tar.gz
-rw-r--r-- 1 debian debian 19100418 Dec 9 2024 downloads/crictl-v1.32.0-linux-amd64.tar.gz
-rw-r--r-- 1 debian debian 23577153 Mar 28 2025 downloads/etcd-v3.6.0-rc.3-linux-amd64.tar.gz
# Make the binaries executable.
ls -l downloads/{client,cni-plugins,controller,worker}/*
sudo chmod +x downloads/{client,cni-plugins,controller,worker}/*
ls -l downloads/{client,cni-plugins,controller,worker}/*
# 일부 파일 소유자 변경
tree -ug downloads # cat /etc/passwd | grep vagrant && cat /etc/group | grep vagrant
sudo chown root:root downloads/client/etcdctl
sudo chown root:root downloads/controller/etcd
sudo chown root:root downloads/worker/crictl
tree -ug downloads
# kubernetes client 도구인 kubectl를 설치
ls -l downloads/client/kubectl
cp downloads/client/kubectl /usr/local/bin/
# can be verified by running the kubectl command:
kubectl version --client
Client Version: v1.32.3
Kustomize Version: v5.5.0
[K8S Docs] Release* : 1년에 3개의 마이너 버전 출시 → 최근 3개 버전 release branches(패치) 지원 (그이상 힘듬..) - Link
- Download - Link
- Release Cycle - Link
- Patch Releases - Link
- Kubernetes 1.32가 유지보수 모드로 전환되었습니다.2025년 12월 28일.
- Kubernetes 1.32의 지원 종료일은 다음과 같습니다.2026년 2월 28일.
- Release Managers - Link
- Release Notes - Link
- Version Skew Policy - Link*
- kube-apiserver : HA apiserver 경우 newest 와 oldest 가 1개 마이너 버전 가능
- newest kube-apiserver is at 1.32
- other kube-apiserver instances are supported at 1.32 and 1.31
- kubelet/kube-proxy : kubelet 는 apiserver 보다 높을수 없음. 추가로 apiserver 보다 3개 older 가능
- kube-apiserver is at 1.32
- kubelet is supported at 1.32, 1.31, 1.30, and 1.29
- 만약 apiserver 가 HA로 1.32, 1.31 있다면 kubelet 는 1.32는 안되고, 1.31, 1.30, 1.29 가능
- kube-controller-manager, kube-scheduler, and cloud-controller-manager : skip
- kubectl : one minor version (older or newer) of kube-apiserver
- kube-apiserver is at 1.32
- kubectl is supported at 1.33, 1.32, and 1.31
- kube-apiserver : HA apiserver 경우 newest 와 oldest 가 1개 마이너 버전 가능
Provisioning Compute Resources
☞ To ssh jumpbox
# Machine Database (서버 속성 저장 파일) : IPV4_ADDRESS FQDN HOSTNAME POD_SUBNET
## 참고) server(controlplane)는 kubelet 동작하지 않아서, 파드 네트워크 대역 설정 필요 없음
cat <<EOF > machines.txt
192.168.7.246 server.kubernetes.local server
192.168.7.247 node-0.kubernetes.local node-0 10.200.0.0/24
192.168.7.248 node-1.kubernetes.local node-1 10.200.1.0/24
EOF
cat machines.txt
while read IP FQDN HOST SUBNET; do
echo "${IP} ${FQDN} ${HOST} ${SUBNET}"
done < machines.txt
192.168.7.246 server.kubernetes.local server
192.168.7.247 node-0.kubernetes.local node-0 10.200.0.0/24
192.168.7.248 node-1.kubernetes.local node-1 10.200.1.0/24
# Configuring SSH Access 설정
# sshd config 설정 파일 확인 : 이미 암호 기반 인증 접속 설정 되어 있음
grep "^[^#]" /etc/ssh/sshd_config
PasswordAuthentication yes
KbdInteractiveAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
ClientAliveInterval 120
PermitRootLogin yes
...
PasswordAuthentication yes
PermitRootLogin yes
# Generate a new SSH key
ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa
debian@jumpbox:~/kubernetes-the-hard-way$ sudo ls -l /root/.ssh
total 16
-rw------- 1 root root 0 Jan 5 19:55 authorized_keys
-rw------- 1 root root 2602 Jan 8 16:20 id_rsa
-rw-r--r-- 1 root root 566 Jan 8 16:20 id_rsa.pub
# Copy the SSH public key to each machine
while read IP FQDN HOST SUBNET; do
sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no root@${IP}
done < machines.txt
sshpass -p '1' ssh-copy-id -o StrictHostKeyChecking=no root@${IP}
done < machines.txt
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.7.246'"
and check to make sure that only the key(s) you wanted were added.
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.7.247'"
and check to make sure that only the key(s) you wanted were added.
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' 'root@192.168.7.248'"
and check to make sure that only the key(s) you wanted were added.
while read IP FQDN HOST SUBNET; do
ssh -n root@${IP} cat /root/.ssh/authorized_keys
done < machines.txt
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeZbf084vEL9Whhj7FWB6NGxTA7n1eHjTrxG6+gyof6ANcvfKEzTPwHOPZlkyQS+SxizkTz/QU0tNUbyEf0YthLO95VrtF/obB2Yh6TifECug6QNNlxn901hbttU8TQgffsY/Zwqnl7amdqN6/RNu5iRFmGLrE9hjmg8nuJ1uAxtAz2ifbWwgIuyRANbycFLQ7u/Hm+gfOcRPwlFRojcLq2HEPn8LiZkfRnQTTtsXfM7pmFmacHHdB/Xvx+8ZAQhBMMBFl5EPF6kdu56VHl/0cmJ4fwzsfEgKRUPMBW9Xns3BZ42W4UfTpPzoZS9klZxttHKiJNy00TBVRJW0UUVjUJUgNA4T1bstd8OlMvjqYmmKTgZQxbKkU5TLguZFYujS4bP4VqGO8lxBErUpZqPNVaYhQjIaJnLZjj9nbxjZzp2tv6ckYWDmjN94VV1Rpcq4USPI+Q9xQiJ0VW2Nye4Mip/vGqZvPmb4nFlO91LhLRJ3ZuATE4SVBffi1R5WUny0= root@jumpbox
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0YRbcxtxXdfdGp6AL/pHNgu5WvGbqF+adax8VXfg/IJC/iPWHE2srBjb0TyEA2fqOX8Y97c67cpCFLmbEk7gDHzzo+gHc5zgA6cBg3DCg6q0oPpaxlhIm9uE7XzVevPSe6NbmCNPaDPefToJ9GAsOtGzcc8gz7X5C9YgvChstHbSuDW9EZJvKJkN3MO0cPy5LQempuRjpme2LEHTR+L4YPpnT/bil6hVglbx4JZOAetdgogUb4aczAFYGPYQIS8xn9J8u1GtC09Tjq8cIOBac4NDeiRHI5wT1xCPakc6o/faEpLDIduaVDKKrneBlYu5SfEEHwJDSoaTfY/dORk24PgNV2WflzuaNGb1zF+EicPopGmQMvU2p70fa4ZtX4wTdYsGXlUSd22aogqH3Qv7WINdN6SOjyKc9wIQL7p7rbcfxzbU9UWnGHPN/ze8N0hwlMoO1wMWaPXe7lXgrRJw88FDio67Vipu0lloAOdWS1zCDjzH2UAp4IZ/Vc7FMGYM= root@jumpbox
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeZbf084vEL9Whhj7FWB6NGxTA7n1eHjTrxG6+gyof6ANcvfKEzTPwHOPZlkyQS+SxizkTz/QU0tNUbyEf0YthLO95VrtF/obB2Yh6TifECug6QNNlxn901hbttU8TQgffsY/Zwqnl7amdqN6/RNu5iRFmGLrE9hjmg8nuJ1uAxtAz2ifbWwgIuyRANbycFLQ7u/Hm+gfOcRPwlFRojcLq2HEPn8LiZkfRnQTTtsXfM7pmFmacHHdB/Xvx+8ZAQhBMMBFl5EPF6kdu56VHl/0cmJ4fwzsfEgKRUPMBW9Xns3BZ42W4UfTpPzoZS9klZxttHKiJNy00TBVRJW0UUVjUJUgNA4T1bstd8OlMvjqYmmKTgZQxbKkU5TLguZFYujS4bP4VqGO8lxBErUpZqPNVaYhQjIaJnLZjj9nbxjZzp2tv6ckYWDmjN94VV1Rpcq4USPI+Q9xQiJ0VW2Nye4Mip/vGqZvPmb4nFlO91LhLRJ3ZuATE4SVBffi1R5WUny0= root@jumpbox
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0YRbcxtxXdfdGp6AL/pHNgu5WvGbqF+adax8VXfg/IJC/iPWHE2srBjb0TyEA2fqOX8Y97c67cpCFLmbEk7gDHzzo+gHc5zgA6cBg3DCg6q0oPpaxlhIm9uE7XzVevPSe6NbmCNPaDPefToJ9GAsOtGzcc8gz7X5C9YgvChstHbSuDW9EZJvKJkN3MO0cPy5LQempuRjpme2LEHTR+L4YPpnT/bil6hVglbx4JZOAetdgogUb4aczAFYGPYQIS8xn9J8u1GtC09Tjq8cIOBac4NDeiRHI5wT1xCPakc6o/faEpLDIduaVDKKrneBlYu5SfEEHwJDSoaTfY/dORk24PgNV2WflzuaNGb1zF+EicPopGmQMvU2p70fa4ZtX4wTdYsGXlUSd22aogqH3Qv7WINdN6SOjyKc9wIQL7p7rbcfxzbU9UWnGHPN/ze8N0hwlMoO1wMWaPXe7lXgrRJw88FDio67Vipu0lloAOdWS1zCDjzH2UAp4IZ/Vc7FMGYM= root@jumpbox
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeZbf084vEL9Whhj7FWB6NGxTA7n1eHjTrxG6+gyof6ANcvfKEzTPwHOPZlkyQS+SxizkTz/QU0tNUbyEf0YthLO95VrtF/obB2Yh6TifECug6QNNlxn901hbttU8TQgffsY/Zwqnl7amdqN6/RNu5iRFmGLrE9hjmg8nuJ1uAxtAz2ifbWwgIuyRANbycFLQ7u/Hm+gfOcRPwlFRojcLq2HEPn8LiZkfRnQTTtsXfM7pmFmacHHdB/Xvx+8ZAQhBMMBFl5EPF6kdu56VHl/0cmJ4fwzsfEgKRUPMBW9Xns3BZ42W4UfTpPzoZS9klZxttHKiJNy00TBVRJW0UUVjUJUgNA4T1bstd8OlMvjqYmmKTgZQxbKkU5TLguZFYujS4bP4VqGO8lxBErUpZqPNVaYhQjIaJnLZjj9nbxjZzp2tv6ckYWDmjN94VV1Rpcq4USPI+Q9xQiJ0VW2Nye4Mip/vGqZvPmb4nFlO91LhLRJ3ZuATE4SVBffi1R5WUny0= root@jumpbox
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0YRbcxtxXdfdGp6AL/pHNgu5WvGbqF+adax8VXfg/IJC/iPWHE2srBjb0TyEA2fqOX8Y97c67cpCFLmbEk7gDHzzo+gHc5zgA6cBg3DCg6q0oPpaxlhIm9uE7XzVevPSe6NbmCNPaDPefToJ9GAsOtGzcc8gz7X5C9YgvChstHbSuDW9EZJvKJkN3MO0cPy5LQempuRjpme2LEHTR+L4YPpnT/bil6hVglbx4JZOAetdgogUb4aczAFYGPYQIS8xn9J8u1GtC09Tjq8cIOBac4NDeiRHI5wT1xCPakc6o/faEpLDIduaVDKKrneBlYu5SfEEHwJDSoaTfY/dORk24PgNV2WflzuaNGb1zF+EicPopGmQMvU2p70fa4ZtX4wTdYsGXlUSd22aogqH3Qv7WINdN6SOjyKc9wIQL7p7rbcfxzbU9UWnGHPN/ze8N0hwlMoO1wMWaPXe7lXgrRJw88FDio67Vipu0lloAOdWS1zCDjzH2UAp4IZ/Vc7FMGYM= root@jumpbox
# Once each key is added, verify SSH public key access is working
# 아래는 IP 기반으로 접속 확인
while read IP FQDN HOST SUBNET; do
ssh -n root@${IP} hostname
done < machines.txt
server
node-0
node-1
# Hostnames 설정
# 확인 : init_cfg.sh 로 이미 설정되어 있음
while read IP FQDN HOST SUBNET; do
ssh -n root@${IP} cat /etc/hosts
done < machines.txt
while read IP FQDN HOST SUBNET; do
ssh -n root@${IP} hostname --fqdn
done < machines.txt
# 아래는 hostname 으로 ssh 접속 확인
cat /etc/hosts
while read IP FQDN HOST SUBNET; do
sshpass -p 'qwe123' ssh -n -o StrictHostKeyChecking=no root@${HOST} hostname
done < machines.txt
while read IP FQDN HOST SUBNET; do
sshpass -p 'qwe123' ssh -n root@${HOST} uname -o -m -n
done < machines.txt
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
# /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.7.245 jumpbox
192.168.7.246 server.kubernetes.local server
192.168.7.247 node-0.kubernetes.local node-0
192.168.7.248 node-1.kubernetes.local node-1
Provisioning a CA and Generating TLS Certificates
☞ CA 설정 및 TLS 인증서 생성 - Docs
| 개인키 | CSR | 인증서 | 참고 정보 | |
| Root CA | ca.key | X | ca.crt | |
| admin | admin.key | admin.csr | admin.crt | CN = admin, O = system:masters |
| node-0 | node-0.key | node-0.csr | node-0.crt | CN = system:node:node-0, O = system:nodes |
| node-1 | node-1.key | node-1.csr | node-1.crt | CN = system:node:node-1, O = system:nodes |
| kube-proxy | kube-proxy.key | kube-proxy.csr | kube-proxy.crt | CN = system:kube-proxy, O = system:node-proxier |
| kube-scheduler | kube-scheduler.key | kube-scheduler | kube-scheduler.crt | CN = system:kube-scheduler, O = system:kube-scheduler |
| kube-controller-manager | kube-controller-manager.key | kube-controller-manager.csr | kube-controller-manager.crt | CN = system:kube-controller-manager, O = system:kube-controller-manager |
| kube-api-server | kube-api-server.key | kube-api-server.csr | kube-api-server.crt | CN = kubernetes, SAN: IP(127.0.0.1, 10.32.0.1), DNS(kubernetes,..) |
| service-accounts | service-accounts.key | service-accounts.csr | service-accounts.crt | CN = service-accounts |
| 항목 | 네트워크 대역 |
| clusterCIDR | 10.200.0.0/16 |
| → node-0 PodCIDR | 10.200.0.0/24 |
| → node-1 PodCIDR | 10.200.1.0/24 |
| ServiceCIDR | 10.32.0.0/24 |
| → api clusterIP | 10.32.0.1 |
(참고) kind k8s 에서 인증서 관련 파일 확인
더보기
# 툴 설치
docker exec -it myk8s-control-plane sh -c 'apt update && apt install tree yq jq -y'
# Check certificates expiration for a Kubernetes cluster
docker exec -i myk8s-control-plane kubeadm certs check-expiration
[check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[check-expiration] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jan 06, 2027 06:59 UTC 363d ca no
apiserver Jan 06, 2027 06:59 UTC 363d ca no
apiserver-etcd-client Jan 06, 2027 06:59 UTC 363d etcd-ca no
apiserver-kubelet-client Jan 06, 2027 06:59 UTC 363d ca no
controller-manager.conf Jan 06, 2027 06:59 UTC 363d ca no
etcd-healthcheck-client Jan 06, 2027 06:59 UTC 363d etcd-ca no
etcd-peer Jan 06, 2027 06:59 UTC 363d etcd-ca no
etcd-server Jan 06, 2027 06:59 UTC 363d etcd-ca no
front-proxy-client Jan 06, 2027 06:59 UTC 363d front-proxy-ca no
scheduler.conf Jan 06, 2027 06:59 UTC 363d ca no
super-admin.conf Jan 06, 2027 06:59 UTC 363d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jan 04, 2036 06:59 UTC 9y no
etcd-ca Jan 04, 2036 06:59 UTC 9y no
front-proxy-ca Jan 04, 2036 06:59 UTC 9y no
#
docker exec -it myk8s-control-plane tree /etc/kubernetes
/etc/kubernetes
|-- admin.conf
|-- controller-manager.conf
|-- kubelet.conf
|-- manifests
| |-- etcd.yaml
| |-- kube-apiserver.yaml
| |-- kube-controller-manager.yaml
| `-- kube-scheduler.yaml
|-- pki
| |-- apiserver-etcd-client.crt
| |-- apiserver-etcd-client.key
| |-- apiserver-kubelet-client.crt
| |-- apiserver-kubelet-client.key
| |-- apiserver.crt
| |-- apiserver.key
| |-- ca.crt
| |-- ca.key
| |-- etcd
| | |-- ca.crt
| | |-- ca.key
| | |-- healthcheck-client.crt
| | |-- healthcheck-client.key
| | |-- peer.crt
| | |-- peer.key
| | |-- server.crt
| | `-- server.key
| |-- front-proxy-ca.crt
| |-- front-proxy-ca.key
| |-- front-proxy-client.crt
| |-- front-proxy-client.key
| |-- sa.key
| `-- sa.pub
|-- scheduler.conf
`-- super-admin.conf
4 directories, 31 files
☞ cat ca.conf 파일 내용 확인
| 구분 | 역할 |
| [req] | OpenSSL 요청 기본 동작 |
| [ca_*] | CA 인증서 |
| [admin] | 관리자 (kubectl) |
| [service-accounts] | ServiceAccount 토큰 서명 |
| [node-*] | 워커 노드(kubelet) |
| [kube-proxy] | kube-proxy |
| [kube-controller-manager] | 컨트롤러 |
| [kube-scheduler] | 스케줄러 |
| [kube-api-server] | API Server |
| [default_req_extensions] | 공통 CSR 옵션 |
[req]
distinguished_name = req_distinguished_name
prompt = no # CSR 생성 시 대화형 입력 없음
x509_extensions = ca_x509_extensions # CA 인증서 생성 시 사용할 확장
[ca_x509_extensions] # CA 인증서 설정 (Root of Trust)
basicConstraints = CA:TRUE # CA 권한 인증서
keyUsage = cRLSign, keyCertSign # 다른 인증서를 서명 가능, Kubernetes 모든 인증의 신뢰 루트
[req_distinguished_name]
C = US
ST = Washington
L = Seattle
CN = CA # 클러스터 CA
[admin] # Admin 사용자 (kubectl)
distinguished_name = admin_distinguished_name
prompt = no
req_extensions = default_req_extensions
[admin_distinguished_name]
CN = admin # [K8S] CN → user
O = system:masters # [K8S] O → group , system:masters - Kubernetes 슈퍼유저 그룹, 모든 RBAC 인가 우회
# Service Accounts
#
# The Kubernetes Controller Manager leverages a key pair to generate
# and sign service account tokens as described in the
# [managing service accounts](https://kubernetes.io/docs/admin/service-accounts-admin/)
# documentation.
[service-accounts] # Service Account 서명자
distinguished_name = service-accounts_distinguished_name
prompt = no
req_extensions = default_req_extensions
[service-accounts_distinguished_name]
CN = service-accounts # controller-manager가 사용하는 ServiceAccount 토큰 서명용 인증서 , apiserver에서 --service-account-key-file 로 사용
# Worker Nodes
#
# Kubernetes uses a [special-purpose authorization mode](https://kubernetes.io/docs/admin/authorization/node/)
# called Node Authorizer, that specifically authorizes API requests made
# by [Kubelets](https://kubernetes.io/docs/concepts/overview/components/#kubelet).
# In order to be authorized by the Node Authorizer, Kubelets must use a credential
# that identifies them as being in the `system:nodes` group, with a username
# of `system:node:<nodeName>`.
[node-0] # Worker Node 인증서 (kubelet)
distinguished_name = node-0_distinguished_name
prompt = no
req_extensions = node-0_req_extensions
[node-0_req_extensions]
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth, serverAuth # clientAuth: apiserver → kubelet & serverAuth: kubelet HTTPS 서버(10250)
keyUsage = critical, digitalSignature, keyEncipherment
nsCertType = client
nsComment = "Node-0 Certificate"
subjectAltName = DNS:node-0, IP:127.0.0.1
subjectKeyIdentifier = hash
[node-0_distinguished_name]
CN = system:node:node-0 # kubelet 사용자 , CN = system:node:<nodeName>
O = system:nodes # Node Authorizer 그룹 ,O = system:nodes
C = US
ST = Washington
L = Seattle
[node-1]
distinguished_name = node-1_distinguished_name
prompt = no
req_extensions = node-1_req_extensions
[node-1_req_extensions]
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth, serverAuth
keyUsage = critical, digitalSignature, keyEncipherment
nsCertType = client
nsComment = "Node-1 Certificate"
subjectAltName = DNS:node-1, IP:127.0.0.1
subjectKeyIdentifier = hash
[node-1_distinguished_name]
CN = system:node:node-1
O = system:nodes
C = US
ST = Washington
L = Seattle
# Kube Proxy Section
[kube-proxy] # kube-proxy
distinguished_name = kube-proxy_distinguished_name
prompt = no
req_extensions = kube-proxy_req_extensions
[kube-proxy_req_extensions]
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth, serverAuth
keyUsage = critical, digitalSignature, keyEncipherment
nsCertType = client
nsComment = "Kube Proxy Certificate"
subjectAltName = DNS:kube-proxy, IP:127.0.0.1
subjectKeyIdentifier = hash
[kube-proxy_distinguished_name]
CN = system:kube-proxy
O = system:node-proxier # system:node-proxier ClusterRoleBinding 존재 , 서비스 네트워크 제어 가능
C = US
ST = Washington
L = Seattle
# Controller Manager
[kube-controller-manager]
distinguished_name = kube-controller-manager_distinguished_name
prompt = no
req_extensions = kube-controller-manager_req_extensions
[kube-controller-manager_req_extensions]
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth, serverAuth
keyUsage = critical, digitalSignature, keyEncipherment
nsCertType = client
nsComment = "Kube Controller Manager Certificate"
subjectAltName = DNS:kube-controller-manager, IP:127.0.0.1
subjectKeyIdentifier = hash
[kube-controller-manager_distinguished_name] # 클러스터 상태 관리, Node, ReplicaSet, SA 토큰 등 관리
CN = system:kube-controller-manager
O = system:kube-controller-manager
C = US
ST = Washington
L = Seattle
# Scheduler
[kube-scheduler]
distinguished_name = kube-scheduler_distinguished_name
prompt = no
req_extensions = kube-scheduler_req_extensions
[kube-scheduler_req_extensions]
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth, serverAuth
keyUsage = critical, digitalSignature, keyEncipherment
nsCertType = client
nsComment = "Kube Scheduler Certificate"
subjectAltName = DNS:kube-scheduler, IP:127.0.0.1
subjectKeyIdentifier = hash
[kube-scheduler_distinguished_name]
CN = system:kube-scheduler
O = system:kube-scheduler # Pod 스케줄링 전용 권한
C = US
ST = Washington
L = Seattle
# API Server
#
# The Kubernetes API server is automatically assigned the `kubernetes`
# internal dns name, which will be linked to the first IP address (`10.32.0.1`)
# from the address range (`10.32.0.0/24`) reserved for internal cluster
# services.
[kube-api-server] # API Server 인증서
distinguished_name = kube-api-server_distinguished_name
prompt = no
req_extensions = kube-api-server_req_extensions
[kube-api-server_req_extensions]
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth, serverAuth
keyUsage = critical, digitalSignature, keyEncipherment
nsCertType = client, server
nsComment = "Kube API Server Certificate"
subjectAltName = @kube-api-server_alt_names
subjectKeyIdentifier = hash
[kube-api-server_alt_names] # SAN (Subject Alternative Name) : 모든 내부/외부 접근 주소
IP.0 = 127.0.0.1
IP.1 = 10.32.0.1
DNS.0 = kubernetes
DNS.1 = kubernetes.default
DNS.2 = kubernetes.default.svc
DNS.3 = kubernetes.default.svc.cluster
DNS.4 = kubernetes.svc.cluster.local
DNS.5 = server.kubernetes.local
DNS.6 = api-server.kubernetes.local
[kube-api-server_distinguished_name]
CN = kubernetes
C = US
ST = Washington
L = Seattle
[default_req_extensions] # 공통 CSR 확장 : 대부분 클라이언트 인증서 -> kubelet / apiserver만 serverAuth 추가
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth
keyUsage = critical, digitalSignature, keyEncipherment
nsCertType = client
nsComment = "Admin Client Certificate"
subjectKeyIdentifier = hash
☞ Certificate Authority (인증서 설치)
# Generate the CA configuration file, certificate, and private key
# Root CA 개인키 생성 : ca.key
openssl genrsa -out ca.key 4096
ls -l ca.key
-rw------- 1 root root 3272 Jan 8 16:34 ca.key
cat ca.key
openssl rsa -in ca.key -text -noout # 개인키 구조 확인
# Root CA 인증서 생성 : ca.crt
## -x509 : CSR을 만들지 않고 바로 인증서(X.509) 생성, 즉, Self-Signed Certificate
## -noenc : 개인키를 암호화하지 않음, 즉, CA 키(ca.key)에 패스프레이즈 없음
## -config ca.conf : 인증서 세부 정보는 설정 파일에서 읽음 , [req] 섹션 사용됨 - DN 정보 → [req_distinguished_name] , CA 확장 → [ca_x509_extensions]
openssl req -x509 -new -sha512 -noenc \
-key ca.key -days 3653 \
-config ca.conf \
-out ca.crt
ls -l ca.crt
-rw-r--r-- 1 root root 1899 Jan 8 16:36 ca.crt
# ca.conf 관련 내용
cat ca.conf
-------------------------------------------
[req]
distinguished_name = req_distinguished_name
prompt = no
x509_extensions = ca_x509_extensions
[ca_x509_extensions]
basicConstraints = CA:TRUE # 이 인증서는 CA 역할 가능
keyUsage = cRLSign, keyCertSign # cRLSign: 인증서 폐기 목록(CRL) 서명 가능, keyCertSign: 다른 인증서를 서명할 수 있음
[req_distinguished_name]
C = US
ST = Washington
L = Seattle
CN = CA
-------------------------------------------
cat ca.crt
openssl x509 -in ca.crt -text -noout # 인증서 전체 내용 확인
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
56:fb:42:82:5e:2f:96:cf:f5:83:2e:78:46:98:6e:3f:08:ee:99:67
Signature Algorithm: sha512WithRSAEncryption
Issuer: C = US, ST = Washington, L = Seattle, CN = CA
Validity
Not Before: Jan 3 00:46:22 2026 GMT
Not After : Jan 4 00:46:22 2036 GMT
Subject: C = US, ST = Washington, L = Seattle, CN = CA
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (4096 bit)
Modulus:
00:ae:ce:95:7e:db:50:a2:4a:71:8b:99:54:0d:b4:
ce:1e:6f:13:3c:a6:54:30:0f:5b:0a:76:56:8c:44:
75:98:58:a6:57:7d:d2:38:e8:05:3c:cc:a5:e9:86:
57:73:98:c5:17:52:7c:7e:c8:48:6c:6b:86:13:1c:
7a:72:5d:10:3a:15:72:8d:66:35:e3:55:06:3e:f7:
44:7f:1b:fc:9e:4a:2b:4a:28:dd:2c:34:63:8d:26:
cc:39:50:3b:44:e5:f8:fe:68:c8:c0:a5:94:ba:b1:
d5:e3:55:1d:d9:98:0c:03:23:3f:9d:d9:a0:79:2c:
e9:ce:c9:92:b8:1f:6e:83:cb:08:1e:e6:28:cf:55:
29:b3:f3:19:1b:fe:c2:d8:30:6e:ee:68:7e:80:c3:
9a:53:77:d1:ae:2a:21:ee:82:94:d7:b5:f3:8f:a3:
98:f8:85:c6:c9:94:72:f3:1e:61:45:84:97:e4:25:
69:c8:5e:11:2c:75:2a:85:a6:b4:75:50:5f:a1:6c:
0a:54:1e:78:ab:25:a3:2e:04:18:21:68:86:11:3d:
90:09:95:02:aa:fc:32:2c:c5:ed:ac:d1:14:3c:d7:
fc:c3:a3:9f:dd:52:07:eb:2f:a7:fc:22:5e:2c:23:
ad:f6:f5:1e:90:db:3f:32:eb:10:38:34:c3:f0:40:
5d:c9:0b:d0:01:fd:78:73:0b:80:92:75:0c:24:76:
c1:6d:93:42:86:4b:a0:6d:99:7b:72:46:b6:52:b1:
2f:47:90:a0:ed:d8:93:71:23:c4:20:c8:63:04:a1:
f6:b6:d8:6e:6b:20:1a:2b:56:43:02:47:5e:77:ae:
4e:00:d5:ec:05:f6:e8:a4:ab:aa:8b:14:8b:b9:da:
d4:a6:e6:c6:c2:35:5a:fd:24:51:7d:29:bb:3c:d3:
fd:a7:bf:a9:5b:77:5a:e1:b1:7b:51:ab:29:a4:15:
e7:ac:f6:2a:1e:38:68:bb:f6:6f:60:e4:26:34:cc:
45:08:2b:e0:71:9a:8e:67:e3:0d:d4:67:63:0b:76:
27:bd:ff:8d:9c:78:e5:b8:55:f0:ce:c1:35:b7:b6:
e7:44:60:60:25:ae:f1:0f:3d:c6:7e:25:03:a3:c8:
87:f8:3d:cd:4b:06:1b:d1:94:63:31:50:33:5b:3f:
3c:66:a8:4d:df:2a:b4:76:a4:fa:54:73:43:09:ac:
6c:21:0b:9c:35:e9:14:ca:25:cd:f1:72:c1:fe:0f:
aa:56:59:1d:ea:45:a7:ab:f5:41:a5:d1:50:3d:da:
f0:71:ff:8b:d2:3b:04:0a:d2:80:e9:17:d6:9a:a3:
1a:5f:19:9b:a0:ef:08:36:d4:88:65:2b:50:42:10:
14:e1:a9
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:TRUE
X509v3 Key Usage:
Certificate Sign, CRL Sign
X509v3 Subject Key Identifier:
B3:5D:82:13:B6:1C:44:59:8C:0A:4E:DB:2B:18:98:77:0D:7A:2F:5B
(참고) kind k8s ca.crt 정보
더보기
docker exec -i myk8s-control-plane cat /etc/kubernetes/pki/ca.crt | openssl x509 -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2963297175414940201 (0x291fbf14b086a229)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes
Validity
Not Before: Jan 8 04:37:34 2026 GMT
Not After : Jan 6 04:42:34 2036 GMT
Subject: CN = kubernetes
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:d2:b2:32:b5:5a:28:f4:fc:d4:0b:4a:f8:46:e8:
a5:cf:5a:55:56:aa:23:6b:f7:f5:12:0b:30:2e:3a:
b3:63:b6:1c:52:a6:2e:fa:01:12:bb:9c:8e:5f:55:
f1:e0:a3:25:d5:44:b7:ae:21:57:7c:a4:50:4d:8a:
22:d1:c3:8d:4a:03:89:5b:c2:5f:ba:f5:12:ce:b3:
60:43:e0:3b:ed:1f:45:89:c8:2a:e1:17:a7:06:9b:
dd:16:c2:61:78:c6:20:03:02:20:1f:f0:b7:46:a7:
c8:87:ce:6f:c1:ce:ff:93:22:cf:7d:77:c0:15:a4:
1e:8a:36:94:c0:34:5f:c0:aa:eb:9a:b2:07:97:b6:
89:c8:41:95:95:36:3c:7f:a4:97:6a:a0:3c:08:47:
3b:d7:75:f8:bd:3d:4d:70:ac:a6:81:2a:a4:9f:fb:
33:d3:84:a6:63:c7:92:1f:7a:d6:07:d8:b9:70:fa:
6d:b6:4e:15:17:33:4f:64:91:64:80:66:8f:37:e9:
87:45:b6:d3:18:06:98:f8:e3:94:e9:32:61:c8:ea:
3a:d4:5b:eb:b0:5c:20:9c:86:64:8a:39:08:cc:62:
c0:32:ed:49:5c:4d:56:08:85:e9:97:71:e0:01:78:
39:d5:a4:5d:66:95:e9:3f:e7:42:cf:52:34:55:8e:
6b:b5
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
C2:48:9F:C8:3A:EE:CC:BE:0E:A9:A8:AA:5E:1B:16:3F:78:76:7B:D0
X509v3 Subject Alternative Name:
DNS:kubernetes
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
b0:65:db:82:8f:e8:be:65:fd:5d:d2:e0:0f:fd:df:9d:fc:b9:
01:66:21:01:66:12:56:fd:7b:a5:e7:c7:59:b1:ec:c2:9c:a9:
ee:ac:ac:07:26:b9:b4:9a:0f:63:2f:07:a2:63:73:4c:cd:67:
31:4d:da:98:bc:bf:51:0c:26:36:4e:9e:72:15:b0:5e:26:44:
00:77:7f:2e:41:80:dc:92:32:30:92:7b:64:39:22:a3:b0:fd:
9b:8b:8a:10:56:73:fa:1d:20:47:0e:31:54:93:23:70:4d:8b:
3f:9f:9b:bb:2e:cc:69:eb:ea:04:c7:f1:14:db:8d:47:ba:c6:
1e:2a:f1:b0:5a:9a:0b:f4:52:35:32:64:33:92:c4:02:cc:4c:
fc:a9:a7:a2:a1:86:0a:da:f9:c9:17:f1:81:22:cf:0d:1e:0c:
e4:78:ff:69:7d:ed:7b:d3:ab:7c:1c:50:07:e0:2a:57:15:aa:
b5:e7:68:28:68:2e:59:13:fa:f0:39:1d:4c:ff:e6:69:4f:52:
c6:b8:02:65:10:4a:20:af:e0:56:5d:01:03:b8:91:ab:76:aa:
f7:31:b1:4a:73:36:41:54:d6:30:c8:79:54:86:f8:dc:7b:7d:
65:52:80:78:0d:df:77:0e:a0:5e:22:37:47:46:80:f6:b0:67:
4d:29:80:d0
☞ Create Client and Server Certificates : admin
# Create Client and Server Certificates : admin
openssl genrsa -out admin.key 4096
ls -l admin.key
# ca.conf 에 admin 섹션
cat ca.conf
-------------------------------------------
[admin]
distinguished_name = admin_distinguished_name
prompt = no
req_extensions = default_req_extensions
[admin_distinguished_name]
CN = admin
O = system:masters
[default_req_extensions] # 공통 CSR 확장
basicConstraints = CA:FALSE
extendedKeyUsage = clientAuth
keyUsage = critical, digitalSignature, keyEncipherment
nsCertType = client
nsComment = "Admin Client Certificate"
subjectKeyIdentifier = hash
-------------------------------------------
# csr 파일 생성 : admin.key 개인키를 사용해 'CN=admin, O=system:masters'인 Kubernetes 관리자용 클라이언트 인증서 요청(admin.csr) 생성
openssl req -new -key admin.key -sha256 \
-config ca.conf -section admin \
-out admin.csr
ls -l admin.csr
openssl req -in admin.csr -text -noout # CSR 전체 내용 확인
# ca에 csr 요청을 통한 crt 파일 생성
## -req : CSR를 입력으로 받아 인증서를 생성, self-signed 아님, CA가 서명하는 방식
## -days 3653 : 인증서 유효기간 3653일 (약 10년)
## -copy_extensions copyall : CSR에 포함된 모든 X.509 extensions를 인증서로 복사
## -CAcreateserial : CA 시리얼 번호 파일 자동 생성, 다음 인증서 발급 시 재사용, 기본 생성 파일(ca.srl)
openssl x509 -req -days 3653 -in admin.csr \
-copy_extensions copyall \
-sha256 -CA ca.crt \
-CAkey ca.key \
-CAcreateserial \
-out admin.crt
Certificate request self-signature ok
subject=CN = admin, O = system:masters
ls -l admin.crt
openssl x509 -in admin.crt -text -noout
...
Issuer: C = US, ST = Washington, L = Seattle, CN = CA
Validity
Not Before: Jan 3 01:01:40 2026 GMT
Not After : Jan 4 01:01:40 2036 GMT
Subject: CN = admin, O = system:masters
...
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
Netscape Cert Type:
SSL Client
Netscape Comment:
Admin Client Certificate
...
(참고) system:masters 그룹에 속한 사용자 특징 : 인증 후 → 인가 우회. 즉, 인증 시 바로 슈퍼유저 권한
더보기
- RBAC 기반 권한검사(Role/ClusterRole)를 거치지 않음. 웹훅 기반 인가도 거치지 않음.
- 즉, API 서버에서 모든 동작을 무조건 허용함. 실질적으로 클러스터 슈퍼유저(superuser) 권한
- system:masters 그룹은 API 서버에 대한 무제한(superuser) 접근 권한을 부여하는 내장(built-in) 그룹.
- 이 그룹에 속한 사용자는 모든 요청에 대해 RBAC이나 Webhook 같은 인가(authorization) 검사를 받지 않고 API 접근을 허용받음.
- 따라서 일반적인 RBAC 정책(ClusterRole, ClusterRoleBinding)과 상관 없이 전체 클러스터에 대한 관리자 권한을 갖습니다 → 즉 인가 검증이 사실상 우회됨.
- 공식 체크리스트에서도 이 그룹의 사용은 최소화하라고 권고. (부트스트랩 이후에는 사용자/컴포넌트에 system:masters 인증 정보로 접근하지 않아야 함) → AWS Root 계정과 유사하게 관리
- 관리자 인증서가 탈취당하면 복구가 매우 어려움
☞ Create Client and Server Certificates : 나머지 전부
# ca.conf 수정
cat ca.conf | grep system:kube-scheduler
CN = system:kube-scheduler
O = system:system:kube-scheduler
sed -i 's/system:system:kube-scheduler/system:kube-scheduler/' ca.conf
cat ca.conf | grep system:kube-scheduler
CN = system:kube-scheduler
O = system:kube-scheduler
# 변수 지정
certs=(
"node-0" "node-1"
"kube-proxy" "kube-scheduler"
"kube-controller-manager"
"kube-api-server"
"service-accounts"
)
# 확인
echo ${certs[*]}
node-0 node-1 kube-proxy kube-scheduler kube-controller-manager kube-api-server service-accounts
# 개인키 생성, csr 생성, 인증서 생성
for i in ${certs[*]}; do
openssl genrsa -out "${i}.key" 4096
openssl req -new -key "${i}.key" -sha256 \
-config "ca.conf" -section ${i} \
-out "${i}.csr"
openssl x509 -req -days 3653 -in "${i}.csr" \
-copy_extensions copyall \
-sha256 -CA "ca.crt" \
-CAkey "ca.key" \
-CAcreateserial \
-out "${i}.crt"
done
Certificate request self-signature ok
subject=CN = system:node:node-0, O = system:nodes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:node:node-1, O = system:nodes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-proxy, O = system:node-proxier, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-scheduler, O = system:kube-scheduler, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-controller-manager, O = system:kube-controller-manager, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = kubernetes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = service-accounts
ls -1 *.crt *.key *.csr
admin.crt
admin.csr
admin.key
ca.crt
ca.key
kube-api-server.crt
kube-api-server.csr
kube-api-server.key
kube-controller-manager.crt
kube-controller-manager.csr
kube-controller-manager.key
kube-proxy.crt
kube-proxy.csr
kube-proxy.key
kube-scheduler.crt
kube-scheduler.csr
kube-scheduler.key
node-0.crt
node-0.csr
node-0.key
node-1.crt
node-1.csr
node-1.key
service-accounts.crt
service-accounts.csr
service-accounts.key
# 인증서 정보 확인
openssl x509 -in node-0.crt -text -noout
Issuer: C = US, ST = Washington, L = Seattle, CN = CA
Validity
Not Before: Jan 3 02:41:09 2026 GMT
Not After : Jan 4 02:41:09 2036 GMT
Subject: CN = system:node:node-0, O = system:nodes, C = US, ST = Washington, L = Seattle
...
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
Netscape Cert Type:
SSL Client
Netscape Comment:
Node-0 Certificate
X509v3 Subject Alternative Name:
DNS:node-0, IP Address:127.0.0.1
openssl x509 -in node-1.crt -text -noout
Subject: CN = system:node:node-1, O = system:nodes, C = US, ST = Washington, L = Seattle
X509v3 Subject Alternative Name:
DNS:node-1, IP Address:127.0.0.1
openssl x509 -in kube-proxy.crt -text -noout
Issuer: C = US, ST = Washington, L = Seattle, CN = CA
Validity
Not Before: Jan 3 02:41:13 2026 GMT
Not After : Jan 4 02:41:13 2036 GMT
Subject: CN = system:kube-proxy, O = system:node-proxier, C = US, ST = Washington, L = Seattle
...
X509v3 extensions:
Netscape Cert Type:
SSL Client
Netscape Comment:
Kube Proxy Certificate
X509v3 Subject Alternative Name:
DNS:kube-proxy, IP Address:127.0.0.1
openssl x509 -in kube-scheduler.crt -text -noout
Issuer: C = US, ST = Washington, L = Seattle, CN = CA
Validity
Not Before: Jan 3 02:41:13 2026 GMT
Not After : Jan 4 02:41:13 2036 GMT
Subject: CN = system:kube-scheduler, O = system:kube-scheduler, C = US, ST = Washington, L = Seattle
Netscape Cert Type:
SSL Client
Netscape Comment:
Kube Scheduler Certificate
X509v3 Subject Alternative Name:
DNS:kube-scheduler, IP Address:127.0.0.1
openssl x509 -in kube-controller-manager.crt -text -noout
Validity
Not Before: Jan 3 02:41:16 2026 GMT
Not After : Jan 4 02:41:16 2036 GMT
Subject: CN = system:kube-controller-manager, O = system:kube-controller-manager, C = US, ST = Washington, L = Seattle
Netscape Cert Type:
SSL Client
Netscape Comment:
Kube Controller Manager Certificate
X509v3 Subject Alternative Name:
DNS:kube-controller-manager, IP Address:127.0.0.1
# api-server : SAN 정보에 10.32.0.1 은 kubernetes (Service) ClusterIP. 다른 인증서와 다르게 SSL Server 역할 추가 확인
openssl x509 -in kube-api-server.crt -text -noout
Issuer: C = US, ST = Washington, L = Seattle, CN = CA
Validity
Not Before: Jan 3 02:41:17 2026 GMT
Not After : Jan 4 02:41:17 2036 GMT
Subject: CN = kubernetes, C = US, ST = Washington, L = Seattle
Netscape Cert Type:
SSL Client, SSL Server
Netscape Comment:
Kube API Server Certificate
X509v3 Subject Alternative Name:
IP Address:127.0.0.1, IP Address:10.32.0.1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.svc.cluster.local, DNS:server.kubernetes.local, DNS:api-server.kubernetes.local
# service-accounts
openssl x509 -in service-accounts.crt -text -noout
Issuer: C = US, ST = Washington, L = Seattle, CN = CA
Validity
Not Before: Jan 3 02:41:17 2026 GMT
Not After : Jan 4 02:41:17 2036 GMT
Subject: CN = service-accounts
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
Netscape Cert Type:
SSL Client
Netscape Comment:
Admin Client Certificate
(참고) kind k8s 에 관련 정보 : 인증서 등
#
docker exec -it myk8s-control-plane tree /etc/kubernetes
docker exec -it myk8s-control-plane tree /var/lib/kubelet/pki
# node 인증서 정보
docker exec -it myk8s-control-plane tree /var/lib/kubelet/pki
/var/lib/kubelet/pki
|-- kubelet-client-2026-01-04-01-20-27.pem
|-- kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2026-01-04-01-20-27.pem
|-- kubelet.crt
`-- kubelet.key
docker exec -i myk8s-control-plane cat /var/lib/kubelet/pki/kubelet.crt | openssl x509 -text -noout
Issuer: CN=myk8s-control-plane-ca@1767489627
Validity
Not Before: Jan 4 00:20:27 2026 GMT
Not After : Jan 4 00:20:27 2027 GMT
Subject: CN=myk8s-control-plane@1767489627
...
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
5A:E4:1B:29:0F:79:7E:EB:5A:A2:BC:5B:B2:51:70:5F:90:D2:EC:9B
X509v3 Subject Alternative Name:
DNS:myk8s-control-plane
docker exec -i myk8s-worker cat /var/lib/kubelet/pki/kubelet.crt | openssl x509 -text -noout
Subject: CN=myk8s-worker@1767489640
...
X509v3 Subject Alternative Name:
DNS:myk8s-worker
# kube-proxy 인증서 정보
kubectl get cm -n kube-system kube-proxy -o yaml
data:
config.conf: |-
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
bindAddressHardFail: false
clientConnection:
acceptContentTypes: ""
burst: 0
contentType: ""
kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
...
kubeconfig.conf: |-
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
server: https://myk8s-control-plane:6443
name: default
contexts:
- context:
cluster: default
namespace: default
user: default
name: default
current-context: default
users:
- name: default
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
# scheduler 인증서 정보
docker exec -i myk8s-control-plane cat /etc/kubernetes/scheduler.conf
docker exec -i myk8s-control-plane cat /etc/kubernetes/scheduler.conf | grep client-certificate-data | cut -d ':' -f 2 | base64 -d | openssl x509 -text -noout
docker exec -i myk8s-control-plane cat /etc/kubernetes/scheduler.conf | grep client-certificate-data | cut -d ':' -f 2 | tr -d ' ' | base64 -d | openssl x509 -text -noout
Issuer: CN = kubernetes
Validity
Not Before: Jan 3 02:46:18 2026 GMT
Not After : Jan 3 02:51:18 2027 GMT
Subject: CN = system:kube-scheduler
...
X509v3 Extended Key Usage:
TLS Web Client Authentication
kubectl get pod -n kube-system -l component=kube-scheduler
kubectl describe pod -n kube-system -l component=kube-scheduler
kubectl rolesum -k User system:kube-scheduler
kubectl rbac-tool lookup system:kube-scheduler
# kcm 인증서 정보
docker exec -i myk8s-control-plane cat /etc/kubernetes/controller-manager.conf
docker exec -i myk8s-control-plane cat /etc/kubernetes/controller-manager.conf | grep client-certificate-data | cut -d ':' -f 2 | base64 -d | openssl x509 -text -noout
docker exec -i myk8s-control-plane cat /etc/kubernetes/controller-manager.conf | grep client-certificate-data | cut -d ':' -f 2 | tr -d ' ' | base64 -d | openssl x509 -text -noout
Subject: CN=system:kube-controller-manager
kubectl rolesum -k User system:kube-controller-manager
kubectl rbac-tool lookup system:kube-controller-manager
# api-server 인증서 정보 : IP Address:10.96.0.1, IP Address:192.168.97.3, IP Address:127.0.0.1
docker exec -i myk8s-control-plane cat /etc/kubernetes/pki/apiserver.crt | openssl x509 -text -noout
Issuer: CN=kubernetes
Validity
Not Before: Jan 3 03:31:30 2026 GMT
Not After : Jan 3 03:36:30 2027 GMT
Subject: CN=kube-apiserver
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
3D:E2:06:BE:D3:96:A8:C6:46:48:DC:91:47:DB:54:0D:5B:12:91:01
X509v3 Subject Alternative Name:
DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, DNS:myk8s-control-plane, IP Address:10.96.0.1, IP Address:192.168.97.3, IP Address:127.0.0.1
kubectl get svc,ep
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 74m
NAME ENDPOINTS AGE
endpoints/kubernetes 192.168.97.3:6443 74m
# apiserver 파드 args 정보
kubectl describe pod -n kube-system kube-apiserver-myk8s-control-plane
...
Command:
kube-apiserver
--advertise-address=192.168.97.3 # 클러스터 내 다른 컴포넌트가 접근할 API 서버 주소
--allow-privileged=true # privileged: true 파드 허용, CNI, kube-proxy, CSI, Cilium 같은 시스템 파드 필수
--authorization-mode=Node,RBAC # [인가] Node Authorizer(system:node:* , kubelet 전용 권한 검증) -> RBAC (사용자 / 그룹 기반 권한)
--client-ca-file=/etc/kubernetes/pki/ca.crt # [인증] X.509 클라이언트 인증 : kubectl, controller, scheduler 인증서 모두 여기로 검증
--enable-admission-plugins=NodeRestriction # Admission Controller : kubelet이 자기 노드/파드만 조작 가능하게 제한, Node Authorizer와 함께 작동
--enable-bootstrap-token-auth=true # [인증] Bootstrap 토큰 인증 : 신규 노드 join 시 bootstrap.kubernetes.io/token 사용
--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt # etcd 연동 : kube-apiserver → etcd mTLS , apiserver는 etcd의 클라이언트
--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt # 상동
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key # 상동
--etcd-servers=https://127.0.0.1:2379 # etcd 연동 : etcd 엔드포인트
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt # kubelet 통신 : API 서버 → kubelet 호출 시 사용
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key # kubelet 통신 : 로그, exec, metrics, stats 접근
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname # kubelet 통신 : 노드 접근 주소 우선순위
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt # Aggregated API / Front Proxy : API Aggregation Layer 인증
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key # 상동
--requestheader-allowed-names=front-proxy-client # 프록시 인증서 CN 제한
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt # 프록시가 전달한 사용자 정보 신뢰
--requestheader-extra-headers-prefix=X-Remote-Extra- # metrics-server, aggregated apiserver 사용 시 필수
--requestheader-group-headers=X-Remote-Group # 상동
--requestheader-username-headers=X-Remote-User # 상동
--runtime-config= # 비어 있음 → 기본 API 활성화
--secure-port=6443 # HTTPS API 포트 , HTTP(8080)는 기본 비활성화
--service-account-issuer=https://kubernetes.default.svc.cluster.local # ServiceAccount 토큰 관리 : SA 토큰 issuer (OIDC 호환), BoundServiceAccountTokenVolume 필수
--service-account-key-file=/etc/kubernetes/pki/sa.pub # 상동 : 토큰 검증용 공개키
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key # 상동 : 토큰 서명용 개인키
--service-cluster-ip-range=10.96.0.0/16
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt # kube-apiserver 서버 인증서(tls) : 클라이언트(kubectl, kubelet 등)가 이 인증서 검증
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key # 상동
# (참고) api -> kubelet 호출 시 : api client crt 정보
docker exec -it myk8s-control-plane ls -l /etc/kubernetes/pki | grep apiserver
-rw-r--r-- 1 root root 1123 Jan 4 01:20 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Jan 4 01:20 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1176 Jan 4 01:20 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Jan 4 01:20 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1326 Jan 4 01:20 apiserver.crt
-rw------- 1 root root 1679 Jan 4 01:20 apiserver.key
docker exec -i myk8s-control-plane cat /etc/kubernetes/pki/apiserver-kubelet-client.crt | openssl x509 -text -noout
...
# ServiceAccount 토큰 검증용 공개키/개인키 확인 : api 파드 service-account-key-file, service-account-signing-key-file 에서 사용
docker exec -it myk8s-control-plane ls -l /etc/kubernetes/pki/sa.{key,pub}
-rw------- 1 root root 1679 Jan 4 01:20 /etc/kubernetes/pki/sa.key
-rw------- 1 root root 451 Jan 4 01:20 /etc/kubernetes/pki/sa.pub
☞ Distribute the Client and Server Certificates
# Copy the appropriate certificates and private keys to the node-0 and node-1 machines
for host in node-0 node-1; do
ssh root@${host} mkdir /var/lib/kubelet/
scp ca.crt root@${host}:/var/lib/kubelet/
scp ${host}.crt \
root@${host}:/var/lib/kubelet/kubelet.crt
scp ${host}.key \
root@${host}:/var/lib/kubelet/kubelet.key
done
# 확인
ssh node-0 ls -l /var/lib/kubelet
ssh node-1 ls -l /var/lib/kubelet
ssh node-1 ls -l /var/lib/kubelet
total 60
-rw-r--r-- 1 root root 1899 Jan 8 16:42 ca.crt
drwx------ 2 root root 4096 Jan 6 16:34 checkpoints
-rw------- 1 root root 62 Jan 6 16:34 cpu_manager_state
drwxr-xr-x 2 root root 4096 Jan 6 16:34 device-plugins
-rw------- 1 root root 10161 Jan 6 16:16 kubeconfig
-rw-r--r-- 1 root root 610 Jan 6 16:31 kubelet-config.yaml
-rw-r--r-- 1 root root 2147 Jan 8 16:42 kubelet.crt
-rw------- 1 root root 3272 Jan 8 16:42 kubelet.key
-rw------- 1 root root 61 Jan 6 16:34 memory_manager_state
drwxr-x--- 2 root root 4096 Jan 6 16:34 plugins
drwxr-x--- 2 root root 4096 Jan 6 16:34 plugins_registry
drwxr-x--- 2 root root 4096 Jan 6 16:34 pod-resources
drwxr-x--- 3 root root 4096 Jan 6 16:44 pods
total 60
-rw-r--r-- 1 root root 1899 Jan 8 16:42 ca.crt
drwx------ 2 root root 4096 Jan 6 16:36 checkpoints
-rw------- 1 root root 62 Jan 6 16:36 cpu_manager_state
drwxr-xr-x 2 root root 4096 Jan 6 16:36 device-plugins
-rw------- 1 root root 10161 Jan 6 16:16 kubeconfig
-rw-r--r-- 1 root root 610 Jan 6 16:31 kubelet-config.yaml
-rw-r--r-- 1 root root 2147 Jan 8 16:42 kubelet.crt
-rw------- 1 root root 3268 Jan 8 16:42 kubelet.key
-rw------- 1 root root 61 Jan 6 16:36 memory_manager_state
drwxr-x--- 2 root root 4096 Jan 6 16:36 plugins
drwxr-x--- 2 root root 4096 Jan 6 16:36 plugins_registry
drwxr-x--- 2 root root 4096 Jan 6 16:36 pod-resources
drwxr-x--- 3 root root 4096 Jan 6 16:44 pods
# Copy the appropriate certificates and private keys to the server machine
scp \
ca.key ca.crt \
kube-api-server.key kube-api-server.crt \
service-accounts.key service-accounts.crt \
root@server:~/
ca.key 100% 3272 2.4MB/s 00:00
ca.crt 100% 1899 1.6MB/s 00:00
kube-api-server.key 100% 3268 2.9MB/s 00:00
kube-api-server.crt 100% 2354 2.0MB/s 00:00
service-accounts.key 100% 3272 2.9MB/s 00:00
service-accounts.crt 100% 2004 2.1MB/s 00:00
# 확인
ssh server ls -l /root
-rw------- 1 root root 3272 Jan 6 15:31 admin.key
-rw------- 1 root root 9953 Jan 6 16:16 admin.kubeconfig
-rw-r--r-- 1 root root 1899 Jan 8 16:42 ca.crt
-rw------- 1 root root 3272 Jan 8 16:42 ca.key
-rw-r--r-- 1 root root 2354 Jan 8 16:42 kube-api-server.crt
-rw------- 1 root root 3268 Jan 8 16:42 kube-api-server.key
-rw-r--r-- 1 root root 727 Jan 6 16:20 kube-apiserver-to-kubelet.yaml
-rw-r--r-- 1 root root 2004 Jan 8 16:42 service-accounts.crt
-rw------- 1 root root 3272 Jan 8 16:42 service-accounts.key
☞ Generating Kubernetes Configuration Files for Authentication
목표 : API Server와 통신을 위한 Client 인증 설정 파일 작성 - Docs
Kubernetes client configuration files
The kubelet Kubernetes Configuration File
- When generating kubeconfig files for Kubelets the client certificate matching the Kubelet's node name must be used.
- This will ensure Kubelets are properly authorized by the Kubernetes Node Authorizer.
- Node Authorization : Kubelet이 수행하는 API 요청에 대한 권한을 특별히 부여하는 특수 목적의 권한 부여 모드
# apiserver 파드 args 정보
kubectl describe pod -n kube-system kube-apiserver-myk8s-control-plane
Command:
kube-apiserver
--authorization-mode=Node,RBAC
# Generate a kubeconfig file for the node-0 and node-1 worker nodes
# config set-cluster
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://server.kubernetes.local:6443 \
--kubeconfig=node-0.kubeconfig && ls -l node-0.kubeconfig && cat node-0.kubeconfig
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://server.kubernetes.local:6443 \
--kubeconfig=node-1.kubeconfig && ls -l node-1.kubeconfig && cat node-1.kubeconfig
# config set-credentials
kubectl config set-credentials system:node:node-0 \
--client-certificate=node-0.crt \
--client-key=node-0.key \
--embed-certs=true \
--kubeconfig=node-0.kubeconfig && cat node-0.kubeconfig
kubectl config set-credentials system:node:node-1 \
--client-certificate=node-1.crt \
--client-key=node-1.key \
--embed-certs=true \
--kubeconfig=node-1.kubeconfig && cat node-1.kubeconfig
# set-context : default 추가
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=system:node:node-0 \
--kubeconfig=node-0.kubeconfig && cat node-0.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=system:node:node-1 \
--kubeconfig=node-1.kubeconfig && cat node-1.kubeconfig
...
contexts:
- context:
cluster: kubernetes-the-hard-way
user: system:node:node-0
name: default
# use-context : current-context 에 default 추가
kubectl config use-context default \
--kubeconfig=node-0.kubeconfig
kubectl config use-context default \
--kubeconfig=node-1.kubeconfig
...
current-context: default
#
ls -l *.kubeconfig
-rw------- 1 root root 10157 Jan 3 14:55 node-0.kubeconfig
-rw------- 1 root root 10068 Jan 3 14:50 node-1.kubeconfig
<실습 검증>
ls -l *.kubeconfig
(참고) kind system:node 관련 정보
#
kubectl describe clusterroles system:node
kubectl get clusterrolebindings.rbac.authorization.k8s.io system:node
# 일반적인 ClusterRoleBinding 다르게 subjects 필드가 없음
# system:node 는 특수한 bootstrap RBAC
# 노드 인증서의 O = system:nodes 가 이 ClusterRoleBinding 과 매칭됨 , 참고로 CN = system:node:<node명>
kubectl get clusterrolebindings.rbac.authorization.k8s.io system:node -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: "2026-01-03T03:36:37Z"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:node
resourceVersion: "146"
uid: a451c617-1577-499a-9b04-59af6789e3ee
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
#
docker exec -i myk8s-worker cat /etc/kubernetes/kubelet.conf
docker exec -i myk8s-worker cat /var/lib/kubelet/pki/kubelet-client-current.pem | openssl x509 -text -noout
Subject: O=system:nodes, CN=system:node:myk8s-worker
☞ The kube-proxy Kubernetes Configuration File
# Generate a kubeconfig file for the kube-proxy service
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://server.kubernetes.local:6443 \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials system:kube-proxy \
--client-certificate=kube-proxy.crt \
--client-key=kube-proxy.key \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=system:kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default \
--kubeconfig=kube-proxy.kubeconfig
# 확인
cat kube-proxy.kubeconfig
☞ The kube-controller-manager Kubernetes Configuration File
# Generate a kubeconfig file for the kube-controller-manager service
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://server.kubernetes.local:6443 \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \
--client-certificate=kube-controller-manager.crt \
--client-key=kube-controller-manager.key \
--embed-certs=true \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=system:kube-controller-manager \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config use-context default \
--kubeconfig=kube-controller-manager.kubeconfig
# 확인
cat kube-controller-manager.kubeconfig
☞ The admin Kubernetes Configuration File
# Generate a kubeconfig file for the admin user
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://127.0.0.1:6443 \
--kubeconfig=admin.kubeconfig
kubectl config set-credentials admin \
--client-certificate=admin.crt \
--client-key=admin.key \
--embed-certs=true \
--kubeconfig=admin.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=admin \
--kubeconfig=admin.kubeconfig
kubectl config use-context default \
--kubeconfig=admin.kubeconfig
# 확인
cat admin.kubeconfig
☞ Distribute the Kubernetes Configuration Files
#
ls -l *.kubeconfig
-rw------- 1 root root 9949 Jan 3 15:00 admin.kubeconfig
-rw------- 1 root root 10305 Jan 3 14:59 kube-controller-manager.kubeconfig
-rw------- 1 root root 10187 Jan 3 14:58 kube-proxy.kubeconfig
-rw------- 1 root root 10231 Jan 3 14:59 kube-scheduler.kubeconfig
-rw------- 1 root root 10157 Jan 3 14:55 node-0.kubeconfig
-rw------- 1 root root 10068 Jan 3 14:50 node-1.kubeconfig
# Copy the kubelet and kube-proxy kubeconfig files to the node-0 and node-1 machines
for host in node-0 node-1; do
ssh root@${host} "mkdir -p /var/lib/{kube-proxy,kubelet}"
scp kube-proxy.kubeconfig \
root@${host}:/var/lib/kube-proxy/kubeconfig \
scp ${host}.kubeconfig \
root@${host}:/var/lib/kubelet/kubeconfig
done
# 확인
ssh node-0 ls -l /var/lib/*/kubeconfig
ssh node-1 ls -l /var/lib/*/kubeconfig
# Copy the kube-controller-manager and kube-scheduler kubeconfig files to the server machine
scp admin.kubeconfig \
kube-controller-manager.kubeconfig \
kube-scheduler.kubeconfig \
root@server:~/
# 확인
ssh server ls -l /root/*.kubeconfig
☞ Generating the Data Encryption Config and Key
목표 : ETCD 에 Secret 저장 시, 암호화 저장 설정 - Docs , Encrypting Confidential Data at Rest
# The Encryption Key
# Generate an encryption key
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
echo $ENCRYPTION_KEY
JMnUP1PUUORZE9iadPdzYifnvPVIniSzOW6NUoMofVc=
# The Encryption Config File
# Create the encryption-config.yaml encryption config file
# (참고) 실제 etcd 값에 기록되는 헤더 : k8s:enc:aescbc:v1:key1:<ciphertext>
cat configs/encryption-config.yaml
kind: EncryptionConfiguration # kube-apiserver가 etcd에 저장할 리소스를 어떻게 암호화할지 정의
apiVersion: apiserver.config.k8s.io/v1 # --encryption-provider-config 플래그로 참조
resources:
- resources:
- secrets # 암호화 대상 Kubernetes 리소스 : 여기서는 Secret 리소스만 암호화
providers: # 지원 providers(위 부터 적용됨) : identity, aescbc, aesgcm, kms v2, secretbox
- aescbc: # etcd에 저장될 Secret을 AES-CBC 방식으로 암호화
keys:
- name: key1 # 키 식별자 (etcd 데이터에 기록됨)
secret: ${ENCRYPTION_KEY}
- identity: {} # 암호화하지 않음 (Plaintext), 주로 하위 호환성 / 점진적 암호화 목적
envsubst < configs/encryption-config.yaml > encryption-config.yaml
cat encryption-config.yaml
# Copy the encryption-config.yaml encryption config file to each controller instance:
scp encryption-config.yaml root@server:~/
ssh server ls -l /root/encryption-config.yaml
☞ Bootstrapping the etcd Cluster
목표 : server 노드에 etcd 서비스 기동
# Prerequisites
# hostname 변경 : controller -> server
# http 평문 통신!
# Each etcd member must have a unique name within an etcd cluster.
# Set the etcd name to match the hostname of the current compute instance:
cat units/etcd.service | grep controller
ETCD_NAME=server
cat > units/etcd.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
--name ${ETCD_NAME} \\
--initial-advertise-peer-urls http://127.0.0.1:2380 \\
--listen-peer-urls http://127.0.0.1:2380 \\
--listen-client-urls http://127.0.0.1:2379 \\
--advertise-client-urls http://127.0.0.1:2379 \\
--initial-cluster-token etcd-cluster-0 \\
--initial-cluster ${ETCD_NAME}=http://127.0.0.1:2380 \\
--initial-cluster-state new \\
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
cat units/etcd.service | grep server
# Copy etcd binaries and systemd unit files to the server machine
scp \
downloads/controller/etcd \
downloads/client/etcdctl \
units/etcd.service \
root@server:~/
# 아래는 server 가상머신 접속 후 명령 실행
# The commands in this lab must be run on the server machine. Login to the server machine using the ssh command. Example:
ssh root@server
-------------------------------------------------------------------
# Bootstrapping an etcd Cluster
# Install the etcd Binaries
# Extract and install the etcd server and the etcdctl command line utility
pwd
mv etcd etcdctl /usr/local/bin/
# Configure the etcd Server
mkdir -p /etc/etcd /var/lib/etcd
chmod 700 /var/lib/etcd
cp ca.crt kube-api-server.key kube-api-server.crt /etc/etcd/
# Create the etcd.service systemd unit file:
mv etcd.service /etc/systemd/system/
tree /etc/systemd/system/
# Start the etcd Server
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
# 확인
systemctl status etcd --no-pager
ss -tnlp | grep etcd
LISTEN 0 4096 127.0.0.1:2380 0.0.0.0:* users:(("etcd",pid=2829,fd=3))
LISTEN 0 4096 127.0.0.1:2379 0.0.0.0:* users:(("etcd",pid=2829,fd=6))
# List the etcd cluster members
etcdctl member list
702b0a34e2cfd39, started, server, http://127.0.0.1:2380, http://127.0.0.1:2379, false
etcdctl member list -w table
etcdctl endpoint status -w table
exit
-------------------------------------------------------------------
(참고) kind k8s etcd 관련 인증서 정보
# https 암호 통신 설정 정보 확인
kubectl describe pod -n kube-system etcd-myk8s-control-plane
...
Command:
etcd
--advertise-client-urls=https://192.168.97.3:2379
--cert-file=/etc/kubernetes/pki/etcd/server.crt
--client-cert-auth=true
--data-dir=/var/lib/etcd
--experimental-initial-corrupt-check=true
--experimental-watch-progress-notify-interval=5s
--initial-advertise-peer-urls=https://192.168.97.3:2380
--initial-cluster=myk8s-control-plane=https://192.168.97.3:2380
--key-file=/etc/kubernetes/pki/etcd/server.key
--listen-client-urls=https://127.0.0.1:2379,https://192.168.97.3:2379
--listen-metrics-urls=http://127.0.0.1:2381
--listen-peer-urls=https://192.168.97.3:2380
--name=myk8s-control-plane
--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
--peer-client-cert-auth=true
--peer-key-file=/etc/kubernetes/pki/etcd/peer.key
--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
--snapshot-count=10000
--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
#
docker exec -it myk8s-control-plane tree /etc/kubernetes/pki/etcd
/etc/kubernetes/pki/etcd
|-- ca.crt
|-- ca.key
|-- healthcheck-client.crt
|-- healthcheck-client.key
|-- peer.crt
|-- peer.key
|-- server.crt
`-- server.key
# etcd 별도 CA 확인
docker exec -i myk8s-control-plane cat /etc/kubernetes/pki/etcd/ca.crt | openssl x509 -text -noout
Issuer: CN=etcd-ca
Validity
Not Before: Jan 4 01:15:25 2026 GMT
Not After : Jan 2 01:20:25 2036 GMT
Subject: CN=etcd-ca
...
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
8C:D1:86:C6:90:7E:1E:9F:16:2B:FE:7C:6F:3F:3F:1A:6A:5F:B9:AD
X509v3 Subject Alternative Name:
DNS:etcd-ca
# kubeadm 에 인증서 정보 확인
docker exec -i myk8s-control-plane kubeadm certs check-expiration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
apiserver-etcd-client Jan 04, 2027 01:20 UTC 364d etcd-ca no
etcd-healthcheck-client Jan 04, 2027 01:20 UTC 364d etcd-ca no
etcd-peer Jan 04, 2027 01:20 UTC 364d etcd-ca no
etcd-server Jan 04, 2027 01:20 UTC 364d etcd-ca no
...
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jan 02, 2036 01:20 UTC 9y no
etcd-ca Jan 02, 2036 01:20 UTC 9y no
front-proxy-ca Jan 02, 2036 01:20 UTC 9y no
# apiserver(client) -> etcd(server) 호출 시 사용하는 클라이언트 인증서 정보 확인
docker exec -i myk8s-control-plane cat /etc/kubernetes/pki/apiserver-etcd-client.crt | openssl x509 -text -noout
Issuer: CN=etcd-ca
Validity
Not Before: Jan 4 01:15:25 2026 GMT
Not After : Jan 4 01:20:25 2027 GMT
Subject: CN=kube-apiserver-etcd-client
...
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
☞ Bootstrapping the Kubernetes Control Plane
Server 노드에 ‘api server, scheduler, kcm’ 서비스 기동 - Docs
| 항목 | 네트워크 대역 or IP |
| clusterCIDR | 10.200.0.0/16 |
| → node-0 PodCIDR | 10.200.0.0/24 |
| → node-1 PodCIDR | 10.200.1.0/24 |
| ServiceCIDR | 10.32.0.0/24 |
| → api clusterIP | 10.32.0.1 |
설정 파일 작성 후 server 에 전달
# Prerequisites
# kube-apiserver.service 수정 : service-cluster-ip-range 추가
# https://github.com/kelseyhightower/kubernetes-the-hard-way/issues/905
# service-cluster-ip 값은 ca.conf 에 설정한 [kube-api-server_alt_names] 항목의 Service IP 범위
cat ca.conf | grep '\[kube-api-server_alt_names' -A2
[kube-api-server_alt_names]
IP.0 = 127.0.0.1
IP.1 = 10.32.0.1
cat units/kube-apiserver.service
cat << EOF > units/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
--allow-privileged=true \\
--apiserver-count=1 \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/var/log/audit.log \\
--authorization-mode=Node,RBAC \\
--bind-address=0.0.0.0 \\
--client-ca-file=/var/lib/kubernetes/ca.crt \\
--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
--etcd-servers=http://127.0.0.1:2379 \\
--event-ttl=1h \\
--encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
--kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \\
--kubelet-client-certificate=/var/lib/kubernetes/kube-api-server.crt \\
--kubelet-client-key=/var/lib/kubernetes/kube-api-server.key \\
--runtime-config='api/all=true' \\
--service-account-key-file=/var/lib/kubernetes/service-accounts.crt \\
--service-account-signing-key-file=/var/lib/kubernetes/service-accounts.key \\
--service-account-issuer=https://server.kubernetes.local:6443 \\
--service-cluster-ip-range=10.32.0.0/24 \\
--service-node-port-range=30000-32767 \\
--tls-cert-file=/var/lib/kubernetes/kube-api-server.crt \\
--tls-private-key-file=/var/lib/kubernetes/kube-api-server.key \\
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
cat units/kube-apiserver.service
# kube-apiserver가 kubelet(Node)에 접근할 수 있도록 허용하는 '시스템 내부용 RBAC' 설정
cat configs/kube-apiserver-to-kubelet.yaml ; echo
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true" # Kubernetes가 업그레이드 시 자동 관리
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- "" # Core API group (v1) : Node 관련 서브리소스는 core group에 속함
resources: # 아래 처럼, kubelet API 대부분을 포괄
- nodes/proxy ## apiserver → kubelet 프록시 통신
- nodes/stats ## 노드/파드 리소스 통계 (cAdvisor)
- nodes/log ## metrics-server / top 명령
- nodes/spec ## kubectl logs
- nodes/metrics ## metrics-server / top 명령
verbs:
- "*" # 대상은 “nodes 하위 리소스”로 한정 + 모든 동작 허용 (get, list, watch, create, proxy 등)
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver # 누가 이 권한을 쓰는가? → kube-apiserver 자신
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes # 사용자 kubernetes ,이 사용자는 kube-apiserver가 사용하는 클라이언트 인증서의 CN
# api-server : Subject CN 확인
openssl x509 -in kube-api-server.crt -text -noout
Subject: CN = kubernetes,
# api -> kubelet 호출 시 Flow
kube-apiserver (client)
|
| (TLS client cert, CN=kubernetes)
↓
kubelet API Server 역할 (/stats, /log, /metrics)
|
↓
RBAC 평가:
User = kubernetes
→ ClusterRoleBinding system:kube-apiserver 매칭
→ ClusterRole system:kube-apiserver-to-kubelet 권한 부여
# kube-scheduler
cat units/kube-scheduler.service ; echo
cat configs/kube-scheduler.yaml ; echo
# kube-controller-manager : cluster-cidr 는 POD CIDR 포함하는 대역, service-cluster-ip-range 는 apiserver 설정 값 동일 설정.
cat units/kube-controller-manager.service ; echo
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
--bind-address=0.0.0.0 \
--cluster-cidr=10.200.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/var/lib/kubernetes/ca.crt \
--cluster-signing-key-file=/var/lib/kubernetes/ca.key \
--kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \
--root-ca-file=/var/lib/kubernetes/ca.crt \
--service-account-private-key-file=/var/lib/kubernetes/service-accounts.key \
--service-cluster-ip-range=10.32.0.0/24 \
--use-service-account-credentials=true \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
# Connect to the jumpbox and copy Kubernetes binaries and systemd unit files to the server machine
scp \
downloads/controller/kube-apiserver \
downloads/controller/kube-controller-manager \
downloads/controller/kube-scheduler \
downloads/client/kubectl \
units/kube-apiserver.service \
units/kube-controller-manager.service \
units/kube-scheduler.service \
configs/kube-scheduler.yaml \
configs/kube-apiserver-to-kubelet.yaml \
root@server:~/
# 확인
ssh server ls -l /root
☞ Provision the Kubernetes Control Plane : kubectl 확인
ssh root@server
---------------------------------------------------------------
# Create the Kubernetes configuration directory:
pwd
mkdir -p /etc/kubernetes/config
# Install the Kubernetes binaries:
mv kube-apiserver \
kube-controller-manager \
kube-scheduler kubectl \
/usr/local/bin/
ls -l /usr/local/bin/kube-*
# Configure the Kubernetes API Server
mkdir -p /var/lib/kubernetes/
mv ca.crt ca.key \
kube-api-server.key kube-api-server.crt \
service-accounts.key service-accounts.crt \
encryption-config.yaml \
/var/lib/kubernetes/
ls -l /var/lib/kubernetes/
## Create the kube-apiserver.service systemd unit file:
mv kube-apiserver.service \
/etc/systemd/system/kube-apiserver.service
tree /etc/systemd/system
# Configure the Kubernetes Controller Manager
## Move the kube-controller-manager kubeconfig into place:
mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
## Create the kube-controller-manager.service systemd unit file:
mv kube-controller-manager.service /etc/systemd/system/
# Configure the Kubernetes Scheduler
## Move the kube-scheduler kubeconfig into place:
mv kube-scheduler.kubeconfig /var/lib/kubernetes/
## Create the kube-scheduler.yaml configuration file:
mv kube-scheduler.yaml /etc/kubernetes/config/
## Create the kube-scheduler.service systemd unit file:
mv kube-scheduler.service /etc/systemd/system/
# Start the Controller Services : Allow up to 10 seconds for the Kubernetes API Server to fully initialize.
systemctl daemon-reload
systemctl enable kube-apiserver kube-controller-manager kube-scheduler
systemctl start kube-apiserver kube-controller-manager kube-scheduler
# 확인
ss -tlp | grep kube
LISTEN 0 4096 *:6443 *:* users:(("kube-apiserver",pid=3071,fd=3))
LISTEN 0 4096 *:10257 *:* users:(("kube-controller",pid=3072,fd=3))
LISTEN 0 4096 *:10259 *:* users:(("kube-scheduler",pid=3073,fd=3))
systemctl is-active kube-apiserver
systemctl status kube-apiserver --no-pager
journalctl -u kube-apiserver --no-pager
systemctl status kube-scheduler --no-pager
systemctl status kube-controller-manager --no-pager
# Verify this using the kubectl command line tool:
kubectl cluster-info dump --kubeconfig admin.kubeconfig
kubectl cluster-info --kubeconfig admin.kubeconfig
Kubernetes control plane is running at https://127.0.0.1:6443
kubectl get node --kubeconfig admin.kubeconfig
kubectl get pod -A --kubeconfig admin.kubeconfig
kubectl get service,ep --kubeconfig admin.kubeconfig
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 110m
NAME ENDPOINTS AGE
endpoints/kubernetes 10.0.2.15:6443 110m
# clusterroles 확인
kubectl get clusterroles --kubeconfig admin.kubeconfig
NAME CREATED AT
admin 2026-01-03T08:42:54Z
cluster-admin 2026-01-03T08:42:54Z
edit 2026-01-03T08:42:54Z
system:aggregate-to-admin 2026-01-03T08:42:54Z
system:aggregate-to-edit 2026-01-03T08:42:54Z
system:aggregate-to-view 2026-01-03T08:42:54Z
system:auth-delegator 2026-01-03T08:42:54Z
system:basic-user 2026-01-03T08:42:54Z
system:certificates.k8s.io:certificatesigningrequests:nodeclient 2026-01-03T08:42:54Z
system:certificates.k8s.io:certificatesigningrequests:selfnodeclient 2026-01-03T08:42:54Z
system:certificates.k8s.io:kube-apiserver-client-approver 2026-01-03T08:42:54Z
system:certificates.k8s.io:kube-apiserver-client-kubelet-approver 2026-01-03T08:42:54Z
system:certificates.k8s.io:kubelet-serving-approver 2026-01-03T08:42:54Z
system:certificates.k8s.io:legacy-unknown-approver 2026-01-03T08:42:54Z
system:controller:attachdetach-controller 2026-01-03T08:42:54Z
system:controller:certificate-controller 2026-01-03T08:42:54Z
system:controller:clusterrole-aggregation-controller 2026-01-03T08:42:54Z
system:controller:cronjob-controller 2026-01-03T08:42:54Z
system:controller:daemon-set-controller 2026-01-03T08:42:54Z
system:controller:deployment-controller 2026-01-03T08:42:54Z
system:controller:disruption-controller 2026-01-03T08:42:54Z
system:controller:endpoint-controller 2026-01-03T08:42:54Z
system:controller:endpointslice-controller 2026-01-03T08:42:54Z
system:controller:endpointslicemirroring-controller 2026-01-03T08:42:54Z
system:controller:ephemeral-volume-controller 2026-01-03T08:42:54Z
system:controller:expand-controller 2026-01-03T08:42:54Z
system:controller:generic-garbage-collector 2026-01-03T08:42:54Z
system:controller:horizontal-pod-autoscaler 2026-01-03T08:42:54Z
system:controller:job-controller 2026-01-03T08:42:54Z
system:controller:legacy-service-account-token-cleaner 2026-01-03T08:42:54Z
system:controller:namespace-controller 2026-01-03T08:42:54Z
system:controller:node-controller 2026-01-03T08:42:54Z
system:controller:persistent-volume-binder 2026-01-03T08:42:54Z
system:controller:pod-garbage-collector 2026-01-03T08:42:54Z
system:controller:pv-protection-controller 2026-01-03T08:42:54Z
system:controller:pvc-protection-controller 2026-01-03T08:42:54Z
system:controller:replicaset-controller 2026-01-03T08:42:54Z
system:controller:replication-controller 2026-01-03T08:42:54Z
system:controller:resourcequota-controller 2026-01-03T08:42:54Z
system:controller:root-ca-cert-publisher 2026-01-03T08:42:54Z
system:controller:route-controller 2026-01-03T08:42:54Z
system:controller:service-account-controller 2026-01-03T08:42:54Z
system:controller:service-controller 2026-01-03T08:42:54Z
system:controller:statefulset-controller 2026-01-03T08:42:54Z
system:controller:ttl-after-finished-controller 2026-01-03T08:42:54Z
system:controller:ttl-controller 2026-01-03T08:42:54Z
system:controller:validatingadmissionpolicy-status-controller 2026-01-03T08:42:54Z
system:discovery 2026-01-03T08:42:54Z
system:heapster 2026-01-03T08:42:54Z
system:kube-aggregator 2026-01-03T08:42:54Z
system:kube-controller-manager 2026-01-03T08:42:54Z
system:kube-dns 2026-01-03T08:42:54Z
system:kube-scheduler 2026-01-03T08:42:54Z
system:kubelet-api-admin 2026-01-03T08:42:54Z
system:monitoring 2026-01-03T08:42:54Z
system:node 2026-01-03T08:42:54Z
system:node-bootstrapper 2026-01-03T08:42:54Z
system:node-problem-detector 2026-01-03T08:42:54Z
system:node-proxier 2026-01-03T08:42:54Z
system:persistent-volume-provisioner 2026-01-03T08:42:54Z
system:public-info-viewer 2026-01-03T08:42:54Z
system:service-account-issuer-discovery 2026-01-03T08:42:54Z
system:volume-scheduler 2026-01-03T08:42:54Z
view 2026-01-03T08:42:54Z
kubectl describe clusterroles system:kube-scheduler --kubeconfig admin.kubeconfig
# kube-scheduler subject 확인
kubectl get clusterrolebindings --kubeconfig admin.kubeconfig
kubectl describe clusterrolebindings system:kube-scheduler --kubeconfig admin.kubeconfig
Role:
Kind: ClusterRole
Name: system:kube-scheduler
Subjects:
Kind Name Namespace
---- ---- ---------
User system:kube-scheduler
---------------------------------------------------------------
▶ RBAC for Kubelet Authorization
이 섹션에서는 Kubernetes API 서버가 각 작업자 노드에서 Kubelet API에 액세스할 수 있도록 RBAC 권한을 구성합니다. Kubelet API에 대한 액세스 권한은 메트릭, 로그를 검색하고 포드에서 명령을 실행하는 데 필요합니다. 이 튜토리얼에서는 Kubelet --authorization-mode 플래그를 Webhook으로 설정합니다. Webhook 모드에서는 SubjectAccessReview API를 사용하여 권한을 결정합니다.
ssh root@server # 이미 server 에 ssh 접속 상태
---------------------------------------------------------------
# api -> kubelet 접속을 위한 RBAC 설정
# Create the system:kube-apiserver-to-kubelet ClusterRole with permissions to access the Kubelet API and perform most common tasks associated with managing pods:
cat kube-apiserver-to-kubelet.yaml
kubectl apply -f kube-apiserver-to-kubelet.yaml --kubeconfig admin.kubeconfig
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
# 확인
kubectl get clusterroles system:kube-apiserver-to-kubelet --kubeconfig admin.kubeconfig
kubectl get clusterrolebindings system:kube-apiserver --kubeconfig admin.kubeconfig
---------------------------------------------------------------
(참고) kind k8s 에서 kubelet config 설정
ssh root@server # 이미 server 에 ssh 접속 상태
---------------------------------------------------------------
# api -> kubelet 접속을 위한 RBAC 설정
# Create the system:kube-apiserver-to-kubelet ClusterRole with permissions to access the Kubelet API and perform most common tasks associated with managing pods:
cat kube-apiserver-to-kubelet.yaml
kubectl apply -f kube-apiserver-to-kubelet.yaml --kubeconfig admin.kubeconfig
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
# 확인
kubectl get clusterroles system:kube-apiserver-to-kubelet --kubeconfig admin.kubeconfig
kubectl get clusterrolebindings system:kube-apiserver --kubeconfig admin.kubeconfig
---------------------------------------------------------------
jumpbox 서버에서 k8s controlplane 정상 동작 확인
curl -s -k --cacert ca.crt https://server.kubernetes.local:6443/version | jq
{
"major": "1",
"minor": "32",
"gitVersion": "v1.32.3",
"gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
"gitTreeState": "clean",
"buildDate": "2025-03-11T19:52:21Z",
"goVersion": "go1.23.6",
"compiler": "gc",
"platform": "linux/arm64"
}
'Kubernetes' 카테고리의 다른 글
| [3주차] Kubeadm & K8S Upgrade (0) | 2026.01.20 |
|---|---|
| [2주차] Ansible 기초 (0) | 2026.01.12 |
| [kind] 쿠버네티스 경량화 서비스 설치 (1) | 2025.10.31 |
| [kubespray] Ansible로 k8s 자동 설치 (0) | 2025.10.31 |
| Gemini CLI 를 통한 k3s 쿠버네티스 제어 (0) | 2025.07.11 |