티스토리 뷰
이 포스팅은 Cloud@net의 기시다님의 온라인 강좌 내용을 정리한 스터디 요약 자료입니다.
Kubespray offline 설치
폐쇄망 환경 소개
보안이 요구되는 일반적인 기업망 구성
내부망 내부에서는 외부 인터넷 접속이 불가능하며, 필요 시 방화벽 정책 승인 후 Bastion Server 를 통해서 다운로드 가능하다.
폐쇄망에서 서비스 동작을 위해 필요한 주요 구성요소
- NTP (Network Time Protocol) Server
- 설명: 서버 간의 시간을 동기화하는 서비스
- 폐쇄망 내 역할: 로그 분석, 보안 인증(Certificate), 데이터 정합성 유지에 필수적입니다. 외부 표준시를 받아올 수 없으므로 내부 전용 장비(GPS 연동 등)를 통해 시간을 공급
- DNS (Domain Name System) Server
- 설명: 도메인 주소(예: registry.internal.local)를 실제 IP 주소로 변환
- 폐쇄망 내 역할: 하드코딩된 IP 대신 유연한 서비스 연결을 지원하며, 내부 서비스들의 위치를 식별하는 전화번호부 역할
- Network Gateway (IGW, NATGW)
- 설명: 서로 다른 네트워크 대역 간의 통신 경로를 지정하는 관문입니다.
- 폐쇄망 내 역할: 내부망(Private)과 완충 구역(DMZ) 사이의 트래픽을 제어하며, 보안 정책에 따라 허용된 통신만 라우팅
- Local (Mirror) YUM/DNF Repository
- 설명: 리눅스(CentOS, RHEL, Rocky 등) 소프트웨어 패키지(.rpm) 저장소의 로컬 복사
- 주요 도구: reposync(외부 저장소 동기화), createrepo(메타데이터 생성).
- 폐쇄망 내 역할: 인터넷 연결 없이도 서버에 새로운 패키지를 설치하거나 보안 업데이트를 수행
- Helm Artifact Repository (Server)
- 설명: 쿠버네티스용 패키지 관리자인 Helm의 '차트(Chart, 설정 묶음)' 저장소
- 주요 도구: ChartMuseum, zot (OCI 규격 지원).
- 폐쇄망 내 역할: 복잡한 쿠버네티스 리소스 배포 설정을 버전별로 관리하여 일관된 배포를 지원
- Private PyPI (Python Package Index) Mirror
- 설명: Python 라이브러리(pip install) 저장소의 사설 미러
- 주요 도구: Devpi.
- 폐쇄망 내 역할: 외부 인터넷의 PyPI에 접속할 수 없는 개발자들이 필요한 라이브러리를 내부망에서 즉시 다운 가능
- Private Go Module Proxy
- 설명: Go 언어의 모듈(라이브러리)을 캐싱하고 제공하는 프록시 서버
- 주요 도구: Athens.
- 폐쇄망 내 역할: Go 빌드 시 필요한 외부 모듈들을 내부에서 관리하여 빌드 속도를 높이고 외부 의존성을 제거
폐쇄망 서비스 실습
일반적으로 DMZ 서버 → 내부 admin서버(파일 가져오기, kubespary 실행)이지만, PC Spec과 시간 단축을 위해 1대 서버로 구성
Vagrantfile : admin 서버 120GB 용량 증설, k8s-node 2대(각각 Ctrl, Wk 1대씩)
더보기
# Base Image https://portal.cloud.hashicorp.com/vagrant/discover/bento/rockylinux-10.0
BOX_IMAGE = "bento/rockylinux-10.0" # "bento/rockylinux-9"
BOX_VERSION = "202510.26.0"
N = 2 # max number of Node
Vagrant.configure("2") do |config|
# Nodes
(1..N).each do |i|
config.vm.define "k8s-node#{i}" do |subconfig|
subconfig.vm.box = BOX_IMAGE
subconfig.vm.box_version = BOX_VERSION
subconfig.vm.provider "virtualbox" do |vb|
vb.customize ["modifyvm", :id, "--groups", "/Kubespary-offline-Lab"]
vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
vb.name = "k8s-node#{i}"
vb.cpus = 4
vb.memory = 2048
vb.linked_clone = true
end
subconfig.vm.host_name = "k8s-node#{i}"
subconfig.vm.network "private_network", ip: "192.168.10.1#{i}"
subconfig.vm.network "forwarded_port", guest: 22, host: "6000#{i}", auto_correct: true, id: "ssh"
subconfig.vm.synced_folder "./", "/vagrant", disabled: true
subconfig.vm.provision "shell", path: "init_cfg.sh" , args: [ N ]
end
end
# Admin Node
config.vm.define "admin" do |subconfig|
subconfig.vm.box = BOX_IMAGE
subconfig.vm.box_version = BOX_VERSION
subconfig.vm.provider "virtualbox" do |vb|
vb.customize ["modifyvm", :id, "--groups", "/Kubespary-offline-Lab"]
vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
vb.name = "admin"
vb.cpus = 4
vb.memory = 2048
vb.linked_clone = true
end
subconfig.vm.host_name = "admin"
subconfig.vm.network "private_network", ip: "192.168.10.10"
subconfig.vm.network "forwarded_port", guest: 22, host: "60000", auto_correct: true, id: "ssh"
subconfig.vm.synced_folder "./", "/vagrant", disabled: true
subconfig.vm.disk :disk, size: "120GB", primary: true # https://developer.hashicorp.com/vagrant/docs/disks/usage
subconfig.vm.provision "shell", path: "admin.sh" , args: [ N ]
end
end
admin.sh
더보기
#!/usr/bin/env bash
echo ">>>> Initial Config Start <<<<"
echo "[TASK 1] Change Timezone and Enable NTP"
timedatectl set-local-rtc 0
timedatectl set-timezone Asia/Seoul
echo "[TASK 2] Disable firewalld and selinux"
systemctl disable --now firewalld >/dev/null 2>&1
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
echo "[TASK 3] Setting Local DNS Using Hosts file"
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
echo "192.168.10.10 admin" >> /etc/hosts
for (( i=1; i<=$1; i++ )); do echo "192.168.10.1$i k8s-node$i" >> /etc/hosts; done
echo "[TASK 4] Delete default routing - enp0s9 NIC" # setenforce 0 설정 필요
nmcli connection modify enp0s9 ipv4.never-default yes
nmcli connection up enp0s9 >/dev/null 2>&1
echo "[TASK 5] Config net.ipv4.ip_forward"
cat << EOF > /etc/sysctl.d/99-ipforward.conf
net.ipv4.ip_forward = 1
EOF
sysctl --system >/dev/null 2>&1
echo "[TASK 6] Install packages"
dnf install -y python3-pip git sshpass cloud-utils-growpart >/dev/null 2>&1
echo "[TASK 7] Install Helm"
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | DESIRED_VERSION=v3.20.0 bash >/dev/null 2>&1
echo "[TASK 8] Increase Disk Size"
growpart /dev/sda 3 >/dev/null 2>&1 # lsblk
xfs_growfs /dev/sda3 >/dev/null 2>&1 # df -hT /
echo "[TASK 9] Setting SSHD"
echo "root:qwe123" | chpasswd
cat << EOF >> /etc/ssh/sshd_config
PermitRootLogin yes
PasswordAuthentication yes
EOF
systemctl restart sshd >/dev/null 2>&1
echo "[TASK 10] Setting SSH Key"
ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa >/dev/null 2>&1
sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no root@192.168.10.10 >/dev/null 2>&1 # cat /root/.ssh/authorized_keys
ssh -o StrictHostKeyChecking=no root@admin-lb hostname >/dev/null 2>&1
for (( i=1; i<=$1; i++ )); do sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no root@192.168.10.1$i >/dev/null 2>&1 ; done
for (( i=1; i<=$1; i++ )); do sshpass -p 'qwe123' ssh -o StrictHostKeyChecking=no root@k8s-node$i hostname >/dev/null 2>&1 ; done
echo "[TASK 11] Install K9s"
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
wget -P /tmp https://github.com/derailed/k9s/releases/latest/download/k9s_linux_${CLI_ARCH}.tar.gz >/dev/null 2>&1
tar -xzf /tmp/k9s_linux_${CLI_ARCH}.tar.gz -C /tmp
chown root:root /tmp/k9s
mv /tmp/k9s /usr/local/bin/
chmod +x /usr/local/bin/k9s
echo "[TASK 12] ETC"
echo "sudo su -" >> /home/vagrant/.bashrc
echo ">>>> Initial Config End <<<<"
init_cfg.sh
더보기
#!/usr/bin/env bash
echo ">>>> Initial Config Start <<<<"
echo "[TASK 1] Change Timezone and Enable NTP"
timedatectl set-local-rtc 0
timedatectl set-timezone Asia/Seoul
echo "[TASK 2] Disable firewalld and selinux"
systemctl disable --now firewalld >/dev/null 2>&1
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
echo "[TASK 3] Disable and turn off SWAP & Delete swap partitions"
swapoff -a
sed -i '/swap/d' /etc/fstab
sfdisk --delete /dev/sda 2 >/dev/null 2>&1
partprobe /dev/sda >/dev/null 2>&1
echo "[TASK 4] Config kernel & module"
cat << EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
vxlan
EOF
modprobe overlay >/dev/null 2>&1
modprobe br_netfilter >/dev/null 2>&1
cat << EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system >/dev/null 2>&1
echo "[TASK 5] Setting Local DNS Using Hosts file"
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
echo "192.168.10.10 admin" >> /etc/hosts
for (( i=1; i<=$1; i++ )); do echo "192.168.10.1$i k8s-node$i" >> /etc/hosts; done
echo "[TASK 6] Delete default routing - enp0s9 NIC" # setenforce 0 설정 필요
nmcli connection modify enp0s9 ipv4.never-default yes
nmcli connection up enp0s9 >/dev/null 2>&1
echo "[TASK 7] Setting SSHD"
echo "root:qwe123" | chpasswd
cat << EOF >> /etc/ssh/sshd_config
PermitRootLogin yes
PasswordAuthentication yes
EOF
systemctl restart sshd >/dev/null 2>&1
echo "[TASK 8] Install packages"
dnf install -y python3-pip git >/dev/null 2>&1
echo "[TASK 9] ETC"
echo "sudo su -" >> /home/vagrant/.bashrc
echo ">>>> Initial Config End <<<<"mkdir k8s-offline
cd k8s-offline
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-kubespary-offline/Vagrantfile
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-kubespary-offline/admin.sh
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-kubespary-offline/init_cfg.sh
vagrant up
vagrant status
# admin, k8s-node1, k8s-node2 각각 접속 : 호스트 OS에 sshpass가 없을 경우 ssh로 root로 접속 후 암호 qwe123 입력
sshpass -p 'qwe123' ssh root@192.168.10.10 # ssh root@192.168.10.10
sshpass -p 'qwe123' ssh root@192.168.10.11 # ssh root@192.168.10.11
sshpass -p 'qwe123' ssh root@192.168.10.12 # ssh root@192.168.10.12
☞ [참고] NetworkManager 기본 정보
더보기
# admin-lb
ip -c route
default via 10.0.2.2 dev enp0s8 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev enp0s8 proto kernel scope link src 10.0.2.15 metric 100
192.168.10.0/24 dev enp0s9 proto kernel scope link src 192.168.10.10 metric 101
# enp0s8
cat /etc/NetworkManager/system-connections/enp0s8.nmconnection
[connection]
id=enp0s8
uuid=7f94e839-e070-4bfe-9330-07090381d89f
type=ethernet
interface-name=enp0s8
[ethernet]
[ipv4]
method=auto # DHCP 서버로부터 IP 주소, 서브넷 마스크, 게이트웨이 정보를 자동으로 할당받습니다.
...
# enp0s9
cat /etc/NetworkManager/system-connections/enp0s9.nmconnection
[connection]
id=enp0s9
uuid=74ceb4c4-b514-4afd-bc84-7bdc7767b649
type=ethernet
autoconnect-priority=-100 # 자동 연결 우선순위가 낮게 설정되어 있습니다.
autoconnect-retries=1 # 실패 시 1회만 재시도
interface-name=enp0s9
timestamp=1769701094
[ethernet]
mac-address=08:00:27:86:00:C2
[ipv4]
address1=192.168.10.10/24
method=manual
never-default=true # 절대 Default Route 생성 안 함
...
# NetworkManager.service : 실제 네트워크 장치를 관리하고 IP를 할당하는 핵심 데몬입니다. D-Bus를 통해 시스템과 통신합니다.
systemctl status NetworkManager.service --no-pager
cat /usr/lib/systemd/system/NetworkManager.service
[Unit]
Description=Network Manager
Documentation=man:NetworkManager(8)
Wants=network.target
After=network-pre.target dbus.service # DBus 준비 후 NetworkManager 시작
Before=network.target # network.target 이전에 네트워크 구성 완료
BindsTo=dbus.service
[Service]
Type=dbus
BusName=org.freedesktop.NetworkManager
ExecReload=/usr/bin/busctl call org.freedesktop.NetworkManager /org/freedesktop/NetworkManager org.freedesktop.NetworkManager Reload u 0
ExecStart=/usr/sbin/NetworkManager --no-daemon
Restart=on-failure
KillMode=process
TimeoutStartSec=600
CapabilityBoundingSet=CAP_NET_ADMIN CAP_DAC_OVERRIDE CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_SYS_MODULE CAP_AUDIT_WRITE CAP_KILL CAP_SYS_CHROOT
ProtectSystem=true
ProtectHome=read-only
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Also=NetworkManager-dispatcher.service
Also=NetworkManager-wait-online.service
# NetworkManager-wait-online.service : 시스템 부팅 시 네트워크가 완전히 연결될 때까지 다른 서비스들의 시작을 잠시 대기시키는 역할을 합니다. (oneshot 타입)
systemctl status NetworkManager-wait-online.service --no-pager
cat /usr/lib/systemd/system/NetworkManager-wait-online.service
[Unit]
Description=Network Manager Wait Online
Documentation=man:NetworkManager-wait-online.service(8)
Requires=NetworkManager.service
After=NetworkManager.service
Before=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/bin/nm-online -s -q
RemainAfterExit=yes
Environment=NM_ONLINE_TIMEOUT=60
[Install]
WantedBy=network-online.target
# NetworkManager-dispatcher.service : 인터페이스가 UP(활성화)되거나 DOWN(비활성화)될 때, /etc/NetworkManager/dispatcher.d/에 있는 사용자 정의 스크립트를 자동으로 실행.
## 발동 조건 : 인터페이스 up/down, IP 변경, DHCP 갱신
systemctl status NetworkManager-dispatcher.service --no-pager
cat /usr/lib/systemd/system/NetworkManager-dispatcher.service
[Unit]
Description=Network Manager Script Dispatcher Service
Documentation=man:NetworkManager-dispatcher.service(8)
[Service]
Type=dbus
BusName=org.freedesktop.nm_dispatcher
ExecStart=/usr/libexec/nm-dispatcher
NotifyAccess=main
KillMode=process
[Install]
Alias=dbus-org.freedesktop.nm-dispatcher.service
# 디렉터리 내에 파일 없는 상태
tree /etc/NetworkManager/dispatcher.d/
/etc/NetworkManager/dispatcher.d/
├── no-wait.d
├── pre-down.d
└── pre-up.d
4 directories, 0 files
kubespary offline 설치 소개
kubespray는 오프라인 환경에서 k8s를 배포를 위한 편의성을 지원
- 다운로드될 파일 목록과 컨테이너 이미지 목록을 파일별로 생성
- 오프라인 배포를 위한 컨테이너 이미지 다운로드 및 이미지 레지스트리(저장소)에 등록(업로드)
- 파일(목록)을 다운로드 하고 Nginx 컨테이너를 실행하여, 파일 다운로드 기능 제공
kubespary-offline 설치 실습
기본환경준비
☞ git clone 후 download-all.sh 로 설치에 필요한 파일들 다운로드 수행 (3.3GB 정도) 17분 소요
- download-all.sh : 아래 후속 실행되는 스크립트들 확인
#!/bin/bash
run() {
echo "=> Running: $*"
$* || {
echo "Failed in : $*"
exit 1
}
}
source ./config.sh
#run ./install-docker.sh
#run ./install-nerdctl.sh
run ./precheck.sh
run ./prepare-pkgs.sh || exit 1
run ./prepare-py.sh
run ./get-kubespray.sh
if $ansible_in_container; then
run ./build-ansible-container.sh
else
run ./pypi-mirror.sh
fi
run ./download-kubespray-files.sh
run ./download-additional-containers.sh
run ./create-repo.sh
run ./copy-target-scripts.sh
echo "Done."
config.sh → target-scripts/config.sh : 설치되는 버전 정보 변수 설정 : 버전 변경 시에는 이 단계에서 수정 필요!, 버전 변수에 파일 다운로드 됨
더보기
#!/bin/bash
source ./target-scripts/config.sh
# container runtime for preparation node
docker=${docker:-podman}
#docker=${docker:-docker}
#docker=${docker:-/usr/local/bin/nerdctl}
# Run ansible in container?
ansible_in_container=${ansible_in_container:-false}cat ./target-scripts/config.sh
#!/bin/bash
# Kubespray version to download. Use "master" for latest master branch.
KUBESPRAY_VERSION=${KUBESPRAY_VERSION:-2.30.0}
#KUBESPRAY_VERSION=${KUBESPRAY_VERSION:-master}
# Versions of containerd related binaries used in `install-containerd.sh`
# These version must be same as kubespray.
# Refer `roles/kubespray_defaults/vars/main/checksums.yml` of kubespray.
RUNC_VERSION=1.3.4
CONTAINERD_VERSION=2.2.1
NERDCTL_VERSION=2.2.1
CNI_VERSION=1.8.0
# Some container versions, must be same as ../imagelists/images.txt
NGINX_VERSION=1.29.4
REGISTRY_VERSION=3.0.0
# container registry port
REGISTRY_PORT=${REGISTRY_PORT:-35000}
# Additional container registry hosts
ADDITIONAL_CONTAINER_REGISTRY_LIST=${ADDITIONAL_CONTAINER_REGISTRY_LIST:-"myregistry.io"}
# Architecture of binary files
# Detect OS type and get architecture accordingly
map_arch() {
case "$1" in
x86_64)
echo "amd64"
;;
aarch64)
echo "arm64"
;;
*)
echo "$1"
;;
esac
}
if [ -e /etc/redhat-release ]; then
# RHEL/AlmaLinux/Rocky Linux
ARCH=$(uname -m)
IMAGE_ARCH=$(map_arch "$ARCH")
elif command -v dpkg >/dev/null 2>&1; then
# Ubuntu/Debian
IMAGE_ARCH=$(dpkg --print-architecture)
else
# Fallback: use uname -m
ARCH=$(uname -m)
IMAGE_ARCH=$(map_arch "$ARCH")
fi
precheck.sh
더보기
#!/bin/bash
source /etc/os-release
source ./config.sh
if [ "$docker" != "podman" ]; then
if ! command -v $docker >/dev/null 2>&1; then
echo "No $docker installed"
exit 1
fi
fi
if [ -e /etc/redhat-release ] && [[ "$VERSION_ID" =~ ^7.* ]]; then
if [ "$(getenforce)" == "Enforcing" ]; then
echo "You must disable SELinux for RHEL7/CentOS7"
exit 1
fi
fiprepare-pkgs.sh : dnf install -y rsync gcc libffi-devel createrepo git podman createrepo_c
더보기
#!/bin/bash
echo "==> prepare-pkgs.sh"
. /etc/os-release
. ./scripts/common.sh
# Select python version
. ./target-scripts/pyver.sh
# Install required packages
if [ -e /etc/redhat-release ]; then
echo "==> Install required packages"
$sudo dnf check-update
$sudo dnf install -y rsync gcc libffi-devel createrepo git podman || exit 1
case "$VERSION_ID" in
7*)
# RHEL/CentOS 7
echo "FATAL: RHEL/CentOS 7 is not supported anymore."
exit 1
;;
8*)
# RHEL/CentOS 8
if ! command -v repo2module >/dev/null; then
echo "==> Install modulemd-tools"
$sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
$sudo dnf copr enable -y frostyx/modulemd-tools-epel
$sudo dnf install -y modulemd-tools
fi
;;
9*)
# RHEL 9
if ! command -v repo2module >/dev/null; then
$sudo dnf install -y modulemd-tools
fi
;;
10*)
# RHEL 9
if ! command -v repo2module >/dev/null; then
$sudo dnf install -y createrepo_c
fi
;;
*)
echo "Unknown version_id: $VERSION_ID"
exit 1
;;
esac
# Install python
$sudo dnf install -y python${PY} python${PY}-pip python${PY}-devel || exit 1
else
$sudo apt update
if [ "$1" == "--upgrade" ]; then
$sudo apt upgrade
fi
$sudo apt -y install lsb-release curl gpg gcc libffi-dev rsync git software-properties-common || exit 1
case "$VERSION_ID" in
20.04)
# Prepare for podman
echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | $sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
curl -SL https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_${VERSION_ID}/Release.key | $sudo apt-key add -
# Prepare for latest python3
sudo add-apt-repository ppa:deadsnakes/ppa -y || exit 1
$sudo apt update
;;
esac
$sudo apt install -y python${PY} python${PY}-venv python${PY}-dev python3-pip python3-selinux podman || exit 1
fiprepare-py.sh → target-scripts/venv.sh : pip install -U pip setuptools & pip install -r requirements.txt
더보기
#!/bin/bash
# Create python3 env
echo "==> prepare-py.sh"
. /etc/os-release
. ./target-scripts/venv.sh
source ./scripts/set-locale.sh
echo "==> Update pip, etc"
pip install -U pip setuptools
#if [ "$(getenforce)" == "Enforcing" ]; then
# pip install -U selinux
#fi
echo "==> Install python packages"
pip install -r requirements.txtcat target-scripts/venv.sh
#!/bin/bash
source /etc/os-release
# Select python version
source "$(dirname "${BASH_SOURCE[0]}")/pyver.sh"
python3=python${PY}
VENV_DIR=${VENV_DIR:-~/.venv/${PY}}
echo "python3 = $python3"
echo "VENV_DIR = ${VENV_DIR}"
if [ ! -e ${VENV_DIR} ]; then
$python3 -m venv ${VENV_DIR}
fi
source ${VENV_DIR}/bin/activate
get-kubespray.sh
더보기
#!/bin/bash
CURRENT_DIR=$(cd $(dirname $0); pwd)
source config.sh
KUBESPRAY_TARBALL=kubespray-${KUBESPRAY_VERSION}.tar.gz
KUBESPRAY_DIR=./cache/kubespray-${KUBESPRAY_VERSION}
umask 022
mkdir -p ./cache
mkdir -p outputs/files/
remove_kubespray_cache_dir() {
if [ -e ${KUBESPRAY_DIR} ]; then
/bin/rm -rf ${KUBESPRAY_DIR}
fi
}
# If KUBESPRAY_VERSION looks like a git commit hash, check out that commit
if [[ $KUBESPRAY_VERSION =~ ^[0-9a-f]{7,40}$ ]]; then
remove_kubespray_cache_dir
echo "===> Checkout kubespray commit: $KUBESPRAY_VERSION"
git clone https://github.com/kubernetes-sigs/kubespray.git ${KUBESPRAY_DIR}
cd ${KUBESPRAY_DIR}
git checkout $KUBESPRAY_VERSION || {
echo "Error: commit $KUBESPRAY_VERSION not found"
exit 1
}
cd - >/dev/null
tar czf outputs/files/${KUBESPRAY_TARBALL} -C ./cache kubespray-${KUBESPRAY_VERSION}
echo "Done (commit checkout)."
exit 0
fi
if [ $KUBESPRAY_VERSION == "master" ] || [[ $KUBESPRAY_VERSION =~ ^release- ]]; then
remove_kubespray_cache_dir
echo "===> Checkout kubespray branch : $KUBESPRAY_VERSION"
if [ ! -e ${KUBESPRAY_DIR} ]; then
git clone -b $KUBESPRAY_VERSION https://github.com/kubernetes-sigs/kubespray.git ${KUBESPRAY_DIR}
tar czf outputs/files/${KUBESPRAY_TARBALL} -C ./cache kubespray-${KUBESPRAY_VERSION}
fi
exit 0
fi
if [ ! -e outputs/files/${KUBESPRAY_TARBALL} ]; then
echo "===> Download ${KUBESPRAY_TARBALL}"
curl -SL https://github.com/kubernetes-sigs/kubespray/archive/refs/tags/v${KUBESPRAY_VERSION}.tar.gz >outputs/files/${KUBESPRAY_TARBALL} || exit 1
remove_kubespray_cache_dir
fi
if [ ! -e ${KUBESPRAY_DIR} ]; then
echo "===> Extract ${KUBESPRAY_TARBALL}"
tar xzf outputs/files/${KUBESPRAY_TARBALL}
mv kubespray-${KUBESPRAY_VERSION} ${KUBESPRAY_DIR}
sleep 1 # ad hoc, for vagrant shared directory
# Apply patches
patch_dir=${CURRENT_DIR}/target-scripts/patches/${KUBESPRAY_VERSION}
if [ -d $patch_dir ]; then
for patch in ${patch_dir}/*.patch; do
echo "===> Apply patch $patch"
(cd $KUBESPRAY_DIR && patch -p1 < $patch) || exit 1
done
fi
fi
echo "Done."
pypi-mirror.sh : pip install -U pip python-pypi-mirror
더보기
#!/bin/bash
source config.sh
KUBESPRAY_DIR=./cache/kubespray-${KUBESPRAY_VERSION}
if [ ! -e $KUBESPRAY_DIR ]; then
echo "No kubespray dir at $KUBESPRAY_DIR"
exit 1
fi
source /etc/os-release
source ./target-scripts/venv.sh
source ./scripts/set-locale.sh
umask 022
echo "==> Create pypi mirror for kubespray"
#set -x
pip install -U pip python-pypi-mirror
DEST="-d outputs/pypi/files"
PLATFORM="--platform manylinux2014_$(uname -m)" # PEP-599
#PLATFORM="--platform manylinux_2_17_$(uname -m)" # PEP-600
REQ=requirements.tmp
#sed "s/^ansible/#ansible/" ${KUBESPRAY_DIR}/requirements.txt > $REQ # Ansible does not provide binary packages
cp ${KUBESPRAY_DIR}/requirements.txt $REQ
echo "PyYAML" >> $REQ # Ansible dependency
echo "ruamel.yaml" >> $REQ # Inventory builder
for pyver in 3.11 3.12; do
echo "===> Download binary for python $pyver"
pip download $DEST --only-binary :all: --python-version $pyver $PLATFORM -r $REQ || exit 1
done
/bin/rm $REQ
echo "===> Download source packages"
pip download $DEST --no-binary :all: -r ${KUBESPRAY_DIR}/requirements.txt
echo "===> Download pip, setuptools, wheel, etc"
pip download $DEST pip setuptools wheel || exit 1
pip download $DEST pip setuptools==40.9.0 || exit 1 # For RHEL...
echo "===> Download additional packages"
PKGS=selinux # need for SELinux (#4)
PKGS="$PKGS flit_core" # build dependency of pyparsing (#6)
PKGS="$PKGS cython<3" # PyYAML requires Cython with python 3.10 (ubuntu 22.04)
pip download $DEST pip $PKGS || exit 1
pypi-mirror create $DEST -m outputs/pypi
echo "pypi-mirror.sh done"
download-kubespray-files.sh* : 파일과 이미지 목록 작성 → download-images.sh 실행 ⇒ kubespary에 contrib/offline/generate_list.sh 사용
더보기
#!/bin/bash
umask 022
source ./config.sh
source scripts/common.sh
source scripts/images.sh
KUBESPRAY_DIR=./cache/kubespray-${KUBESPRAY_VERSION}
if [ ! -e $KUBESPRAY_DIR ]; then
echo "No kubespray dir at $KUBESPRAY_DIR"
exit 1
fi
FILES_DIR=outputs/files
# Decide relative directory of file from URL
#
# kubernetes/vx.x.x : kubeadm/kubectl/kubelet
# kubernetes/etcd : etcd
# kubernetes/cni : CNI plugins
# kubernetes/cri-tools : crictl
# kubernetes/calico/vx.x.x : calico
# kubernetes/calico : calicoctl
# runc/vx.x.x : runc
# cilium-cli/vx.x.x : cilium-cli
# gvisor/{ver}/{arch} : gvisor (sunrc, containerd-shim)
# scopeo/vx.x.x : scopeo
# yq/vx.x.x : yq
#
decide_relative_dir() {
local url=$1
local rdir
rdir=$url
rdir=$(echo $rdir | sed "s@.*/\(v[0-9.]*\)/.*/kube\(adm\|ctl\|let\)@kubernetes/\1@g")
rdir=$(echo $rdir | sed "s@.*/etcd-.*.tar.gz@kubernetes/etcd@")
rdir=$(echo $rdir | sed "s@.*/cni-plugins.*.tgz@kubernetes/cni@")
rdir=$(echo $rdir | sed "s@.*/crictl-.*.tar.gz@kubernetes/cri-tools@")
rdir=$(echo $rdir | sed "s@.*/\(v.*\)/calicoctl-.*@kubernetes/calico/\1@")
rdir=$(echo $rdir | sed "s@.*/\(v.*\)/runc.${IMAGE_ARCH}@runc/\1@")
rdir=$(echo $rdir | sed "s@.*/\(v.*\)/cilium-linux-.*@cilium-cli/\1@")
rdir=$(echo $rdir | sed "s@.*/\([^/]*\)/\([^/]*\)/runsc@gvisor/\1/\2@")
rdir=$(echo $rdir | sed "s@.*/\([^/]*\)/\([^/]*\)/containerd-shim-runsc-v1@gvisor/\1/\2@")
rdir=$(echo $rdir | sed "s@.*/\(v[^/]*\)/skopeo-linux-.*@skopeo/\1@")
rdir=$(echo $rdir | sed "s@.*/\(v[^/]*\)/yq_linux_*@yq/\1@")
if [ "$url" != "$rdir" ]; then
echo $rdir
return
fi
rdir=$(echo $rdir | sed "s@.*/calico/.*@kubernetes/calico@")
if [ "$url" != "$rdir" ]; then
echo $rdir
else
echo ""
fi
}
get_url() {
url=$1
filename="${url##*/}"
rdir=$(decide_relative_dir $url)
if [ -n "$rdir" ]; then
if [ ! -d $FILES_DIR/$rdir ]; then
mkdir -p $FILES_DIR/$rdir
fi
else
rdir="."
fi
if [ ! -e $FILES_DIR/$rdir/$filename ]; then
echo "==> Download $url"
for i in {1..3}; do
curl --location --show-error --fail --output $FILES_DIR/$rdir/$filename $url && return
echo "curl failed. Attempt=$i"
done
echo "Download failed, exit : $url"
exit 1
else
echo "==> Skip $url"
fi
}
# execute offline generate_list.sh
generate_list() {
#if [ $KUBESPRAY_VERSION == "2.18.0" ]; then
# export containerd_version=${containerd_version:-1.5.8}
# export host_os=linux
# export image_arch=amd64
#fi
LANG=C /bin/bash ${KUBESPRAY_DIR}/contrib/offline/generate_list.sh || exit 1
#if [ $KUBESPRAY_VERSION == "2.18.0" ]; then
# # check roles/download/default/main.yml to decide version
# snapshot_controller_tag=${snapshot_controller_tag:-v4.2.1}
# sed -i "s@\(.*/snapshot-controller:\)@\1${snapshot_controller_tag}@" ${KUBESPRAY_DIR}/contrib/offline/temp/images.list || exit 1
#fi
}
. ./target-scripts/venv.sh
generate_list # 목록 작성 스크립트 실행
mkdir -p $FILES_DIR
cp ${KUBESPRAY_DIR}/contrib/offline/temp/files.list $FILES_DIR/
cp ${KUBESPRAY_DIR}/contrib/offline/temp/images.list $IMAGES_DIR/
# download files : 파일 다운로드
files=$(cat ${FILES_DIR}/files.list)
for i in $files; do
get_url $i
done
# download images : 컨테너 (이미지) 다운로드
./download-images.sh || exit 1
download-images.sh : 기본 (컨테이너) 이미지 다운로드 실행
더보기
#!/bin/bash
source ./config.sh
source scripts/common.sh
source scripts/images.sh
if [ "$SKIP_DOWNLOAD_IMAGES" = "true" ]; then
exit 0
fi
if [ ! -e "${IMAGES_DIR}/images.list" ]; then
echo "${IMAGES_DIR}/images.list does not exist. Run download-kubespray-files.sh first."
exit 1
fi
# download images
images=$(cat ${IMAGES_DIR}/images.list)
for i in $images; do
get_image $i
done
download-additional-containers.sh : 추가 (컨테이너) 이미지 다운로드 실행 - nginx, registry
더보기
#!/bin/bash
if [ "$SKIP_DOWNLOAD_IMAGES" = "true" ]; then
exit 0
fi
echo "==> Pull additional container images"
umask 022
source ./config.sh
source scripts/common.sh
source scripts/images.sh
cat imagelists/*.txt | sed "s/#.*$//g" | sort -u > $IMAGES_DIR/additional-images.list
cat $IMAGES_DIR/additional-images.list
IMAGES=$(cat $IMAGES_DIR/additional-images.list)
for image in $IMAGES; do
image=$(expand_image_repo $image)
get_image $image
done
create-repo.sh : scripts/create-repo-rhel.sh 후속 실행
더보기
#!/bin/bash
if [ -e /etc/redhat-release ]; then
./scripts/create-repo-rhel.sh || exit 1
else
./scripts/create-repo-ubuntu.sh || exit 1
fi
scripts/create-repo-rhel.sh : createrepo outputs/rpms/local
더보기
#!/bin/bash
umask 022
. /etc/os-release
REQUIRE_MODULE=false
VERSION_MAJOR=$VERSION_ID
case "${VERSION_MAJOR}" in
8*)
REQUIRE_MODULE=true
VERSION_MAJOR=8
;;
9*)
REQUIRE_MODULE=true
VERSION_MAJOR=9
;;
10*)
VERSION_MAJOR=10
;;
*)
echo "Unsupported version: $VERSION_MAJOR"
;;
esac
# packages
PKGS=$(cat pkglist/rhel/*.txt pkglist/rhel/${VERSION_MAJOR}/*.txt | grep -v "^#" | sort | uniq)
CACHEDIR=cache/cache-rpms
mkdir -p $CACHEDIR
RT="sudo dnf download --resolve --alldeps --downloaddir $CACHEDIR"
echo "==> Downloading: " $PKGS
$RT $PKGS || {
echo "Download error"
exit 1
}
# create rpms dir
RPMDIR=outputs/rpms/local
if [ -e $RPMDIR ]; then
/bin/rm -rf $RPMDIR || exit 1
fi
mkdir -p $RPMDIR
/bin/cp $CACHEDIR/*.rpm $RPMDIR/
/bin/rm $RPMDIR/*.i686.rpm
echo "==> createrepo"
createrepo $RPMDIR || exit 1
#Wait a second to avoid error on Vagrant
sleep 1
if $REQUIRE_MODULE; then
cd $RPMDIR
#createrepo_c . || exit 1
echo "==> repo2module"
LANG=C repo2module -s stable . modules.yaml || exit 1
echo "==> modifyrepo"
modifyrepo_c --mdtype=modules modules.yaml repodata/ || exit 1
fi
echo "create-repo done."
copy-target-scripts.sh : cp -f -r target-scripts/* outputs/
더보기
# 기본 60G -> 120G 증설 확인
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 120G 0 disk
├─sda1 8:1 0 600M 0 part /boot/efi
├─sda2 8:2 0 3.8G 0 part [SWAP]
└─sda3 8:3 0 115.6G 0 part /
df -hT /
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda3 xfs 116G 5.5G 111G 5% /
# git clone
git clone https://github.com/kubespray-offline/kubespray-offline
tree kubespray-offline/
cd kubespray-offline/
# 변수 정보 확인 : nginx 와 registry 는 각각 1.29.4 와 3.0.0 변수 선언 확인 <- 버전 변경 시에는 이 단계에서 수정 필요!
source ./config.sh
echo -e "kubespary $KUBESPRAY_VERSION"
echo -e "runc $RUNC_VERSION"
echo -e "containerd $CONTAINERD_VERSION"
echo -e "nercdtl $NERDCTL_VERSION"
echo -e "cni $CNI_VERSION"
echo -e "nginx $NGINX_VERSION"
echo -e "registry $REGISTRY_VERSION"
echo -e "registry_port: $REGISTRY_PORT"
echo -e "Additional container registry hosts: $ADDITIONAL_CONTAINER_REGISTRY_LIST"
echo -e "cpu arch: $IMAGE_ARCH"
# 17분 소요
cat download-all.sh
./download-all.sh
...
(230/230): container-selinux-2.240.0-1.el10.noarch.rpm 4.6 kB/s | 56 kB 00:12
/bin/rm: cannot remove 'outputs/rpms/local/*.i686.rpm': No such file or directory # 32비트 RPM 파일이 애초에 없어서 지울 게 없음. 오류 아니여서 무시해도됨.
==> createrepo
Directory walk started
Directory walk done - 230 packages
Temporary output repo path: outputs/rpms/local/.repodata/
Pool started (with 5 workers)
Pool finished
create-repo done.
=> Running: ./copy-target-scripts.sh
==> Copy target scripts
Done.
# venv 디렉터리 확인
du -sh ~/.venv
490M /root/.venv
tree ~/.venv | more
/root/.venv
└── 3.12
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── Activate.ps1
│ ├── ansible
...
# /root/.cache 디렉터리 확인
tree ~/.cache | more
du -sh ~/.cache
814M /root/.cache
# 다운로드 될 파일과 이미지 생성 스크립트는 kubespary repo 에 offline 참고 확인
tree /root/kubespray-offline/cache/kubespray-2.30.0/contrib/offline/
cache/kubespray-2.30.0/contrib/offline/
├── docker-daemon.json
├── generate_list.sh
├── generate_list.yml
├── manage-offline-container-images.sh
├── manage-offline-files.sh
├── nginx.conf
├── README.md
├── registries.conf
├── temp
│ ├── files.list*
│ ├── files.list.template
│ ├── images.list*
│ └── images.list.template
└── upload2artifactory.py
# 용량 확인
du -sh /root/kubespray-offline/outputs/
3.3G outputs/
# 디렉터리/파일 구조 확인
tree /root/kubespray-offline/outputs/ | more
tree /root/kubespray-offline/outputs/ -L 1
outputs/
├── config.sh
├── config.toml
├── containerd.service
├── extract-kubespray.sh
├── files # kubectl/kubelet/kubeadm, containerd 등 바이너리 파일들
├── images # 컨테이너 이미지를 .tar.gz 압축파일
├── install-containerd.sh
├── load-push-all-images.sh
├── nginx-default.conf
├── patches
├── playbook # 노드들에 offline repo 설정을 위한 playbook/role
├── pypi # python 패키지 파일들 - index.html, *.whl, *.tar.gz
├── pyver.sh
├── rpms # rpm 패키지 파일들
├── setup-all.sh
├── setup-container.sh
├── setup-offline.sh
├── setup-py.sh
├── start-nginx.sh
├── start-registry.sh
└── venv.sh
7 directories, 15 files
[1] outputs 디렉터리 이동 후 setup-container.sh 실행 : 추가로 install-containerd.sh 실행됨
더보기
# setup-container.sh
#!/bin/bash
# install containerd
./install-containerd.sh
# Load images
echo "==> Load registry, nginx images"
NERDCTL=/usr/local/bin/nerdctl
cd ./images
for f in docker.io_library_registry-*.tar.gz docker.io_library_nginx-*.tar.gz; do
sudo $NERDCTL load -i $f || exit 1
done
if [ -f kubespray-offline-container.tar.gz ]; then
sudo $NERDCTL load -i kubespray-offline-container.tar.gz || exit 1
fi
# install-containerd.sh
# outputs 확인
cd /root/kubespray-offline/outputs
ls -l *.sh
-rw-r--r--. 1 root root 1371 Feb 3 19:46 config.sh
-rwxr-xr-x. 1 root root 719 Feb 3 19:46 extract-kubespray.sh
-rwxr-xr-x. 1 root root 2544 Feb 3 19:46 install-containerd.sh
-rwxr-xr-x. 1 root root 1141 Feb 3 19:46 load-push-all-images.sh
-rw-r--r--. 1 root root 607 Feb 3 19:46 pyver.sh
-rwxr-xr-x. 1 root root 394 Feb 3 19:46 setup-all.sh
-rwxr-xr-x. 1 root root 408 Feb 3 19:46 setup-container.sh
-rwxr-xr-x. 1 root root 2106 Feb 3 19:46 setup-offline.sh
-rwxr-xr-x. 1 root root 1213 Feb 3 19:46 setup-py.sh
-rwxr-xr-x. 1 root root 654 Feb 3 19:46 start-nginx.sh
-rwxr-xr-x. 1 root root 445 Feb 3 19:46 start-registry.sh
-rw-r--r--. 1 root root 322 Feb 3 19:46 venv.sh
# Install containerd from local files. Load nginx and registry images to containerd.
./setup-container.sh
==> Install runc
==> Install nerdctl
==> Install containerd
==> Start containerd
==> Install CNI plugins
==> Load registry, nginx images
# 설치된 바이너리 파일 및 버전 확인
which runc && runc --version
which containerd && containerd --version
which nerdctl && nerdctl --version
tree -ug /opt/cni/bin/
# containerd systemd unit 확인
cat /etc/containerd/config.toml
cat /etc/systemd/system/containerd.service
systemctl status containerd.service --no-pager
# 다운받은 이미지를 압축해제 후 로컬에 load 확인 : CPU Arch = PLATFORM 확인!
nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
nginx 1.29.4 4c333d291372 About a minute ago linux/arm64 190.7MB 184MB
nginx 1.28.0-alpine bcb5257f77e1 About a minute ago linux/arm64 52.73MB 51.23MB
registry 3.0.0 496d3637ba81 About a minute ago linux/arm64 57.52MB 57.34MB
registry 2.8.1 b1524398e0af About a minute ago linux/arm64 25.85MB 25.68MB
[2] start-nginx.sh 실행 : 웹 서버로 files, images, pypi, rpms 제공
더보기
# start-nginx.sh
# (옵션) nginx conf 파일 수정 : 디렉터리 목록 표시되게
cp nginx-default.conf nginx-default.bak
cat << EOF > nginx-default.conf
server {
listen 80;
listen [::]:80;
server_name localhost;
location / {
root /usr/share/nginx/html;
# index index.html index.htm;
autoindex on; # 디렉터리 목록 표시
autoindex_exact_size off; # 파일 크기 KB/MB/GB 단위로 보기 좋게
autoindex_localtime on; # 서버 로컬 타임으로 표시
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
# Force sendfile to off
sendfile off;
}
EOF
# Start nginx container.
./start-nginx.sh
# (옵션) 실행 명령 참고
echo "===> Start nginx"
$NERDCTL container run -d \
--network host \
--restart always \
--name nginx \
-v ${BASEDIR}:/usr/share/nginx/html \
-v ${BASEDIR}/nginx-default.conf:/etc/nginx/conf.d/default.conf \
${NGINX_IMAGE} || exit 1
# nginx 컨테이너 확인
nerdctl ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0b71e6989724 docker.io/library/nginx:1.29.4 "/docker-entrypoint.…" 18 seconds ago Up nginx
ss -tnlp | grep nginx
LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=19771,fd=6),("nginx",pid=19770,fd=6),("nginx",pid=19769,fd=6),("nginx",pid=19768,fd=6),("nginx",pid=19735,fd=6))
LISTEN 0 511 [::]:80 [::]:* users:(("nginx",pid=19771,fd=7),("nginx",pid=19770,fd=7),("nginx",pid=19769,fd=7),("nginx",pid=19768,fd=7),("nginx",pid=19735,fd=7))
# nginx 웹 접속
open http://192.168.10.10
[3] setup-offline.sh 실행 : offline repo 설정, pypi mirror 전역 설정
더보기
# setup-offline.sh
# 스크립트 실행 전 기본 정보 확인
dnf repolist
repo id repo name
appstream Rocky Linux 10 - AppStream
baseos Rocky Linux 10 - BaseOS
extras Rocky Linux 10 - Extras
...
cat /etc/redhat-release
Rocky Linux release 10.0 (Red Quartz)
# 스크립트 실행 : Setup yum/deb repo config and PyPI mirror config to use local nginx server.
./setup-offline.sh
===> Disable all yumrepositories
===> Setup local yum repository
===> Setup PyPI mirror
# 기존 repo 이름이 .original로 변경되고, offline.repo 추가 확인
tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── offline.repo
├── rocky-addons.repo.original
├── rocky-devel.repo.original
├── rocky-extras.repo.original
└── rocky.repo.original
# offline.repo 파일 확인
cat /etc/yum.repos.d/offline.repo
[offline-repo]
name=Offline repo
baseurl=http://localhost/rpms/local/
enabled=1
gpgcheck=0
# offline repo 확인
dnf clean all
dnf repolist
repo id repo name
offline-repo Offline repo
# pip 전역 설정 : pypi mirror 설정 확인
cat ~/.config/pip/pip.conf
[global]
index = http://localhost/pypi/
index-url = http://localhost/pypi/
trusted-host = localhost# 스크립트 실행 전 기본 정보 확인
dnf repolist
repo id repo name
appstream Rocky Linux 10 - AppStream
baseos Rocky Linux 10 - BaseOS
extras Rocky Linux 10 - Extras
...
cat /etc/redhat-release
Rocky Linux release 10.0 (Red Quartz)
# 스크립트 실행 : Setup yum/deb repo config and PyPI mirror config to use local nginx server.
./setup-offline.sh
===> Disable all yumrepositories
===> Setup local yum repository
===> Setup PyPI mirror
# 기존 repo 이름이 .original로 변경되고, offline.repo 추가 확인
tree /etc/yum.repos.d/
/etc/yum.repos.d/
├── offline.repo
├── rocky-addons.repo.original
├── rocky-devel.repo.original
├── rocky-extras.repo.original
└── rocky.repo.original
# offline.repo 파일 확인
cat /etc/yum.repos.d/offline.repo
[offline-repo]
name=Offline repo
baseurl=http://localhost/rpms/local/
enabled=1
gpgcheck=0
# offline repo 확인
dnf clean all
dnf repolist
repo id repo name
offline-repo Offline repo
# pip 전역 설정 : pypi mirror 설정 확인
cat ~/.config/pip/pip.conf
[global]
index = http://localhost/pypi/
index-url = http://localhost/pypi/
trusted-host = localhost
[4] setup-py.sh 실행 : offline repo 로 부터 python${PY} 설치 시도 → offline repo 동작 여부 확인
> setup-py.sh
더보기
#!/bin/bash
. /etc/os-release
IS_OFFLINE=${IS_OFFLINE:-true}
# Select python version
source "$(dirname "${BASH_SOURCE[0]}")/pyver.sh"
# Install python and dependencies
echo "===> Install python, venv, etc"
if [ -e /etc/redhat-release ]; then
# RHEL
DNF_OPTS=
if [[ $IS_OFFLINE = "true" ]]; then
DNF_OPTS="--disablerepo=* --enablerepo=offline-repo"
fi
#sudo dnf install -y $DNF_OPTS gcc libffi-devel openssl-devel || exit 1
if [[ "$VERSION_ID" =~ ^7.* ]]; then
echo "FATAL: RHEL/CentOS 7 is not supported anymore."
exit 1
fi
sudo dnf install -y $DNF_OPTS python${PY} || exit 1
#sudo dnf install -y $DNF_OPTS python${PY}-devel || exit 1
else
# Ubuntu
sudo apt update
case "$VERSION_ID" in
20.04)
if [ "${IS_OFFLINE}" = "false" ]; then
# Prepare for latest python3
sudo apt install -y software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa -y || exit 1
sudo apt update
fi
;;
esac
#sudo apt install -y python${PY}-dev gcc libffi-dev libssl-dev || exit 1
sudo apt install -y python${PY}-venv || exit 1
fi
> pyver.sh
더보기
# Install python3 and venv from local repo.
## sudo dnf install -y --disablerepo=* --enablerepo=offline-repo python${PY}
./setup-py.sh
===> Install python, venv, etc
Last metadata expiration check: 0:06:40 ago on Wed 04 Feb 2026 12:23:11 AM KST.
Package python3-3.12.12-3.el10_1.aarch64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
# 변수 확인
source pyver.sh
echo -e "python_version $python${PY}"
python_version 3.12
# offline-repo 에 패키지 파일 확인
dnf info python3
tree rpms/local/ | grep -i python
├── libcap-ng-python3-0.8.4-6.el10.aarch64.rpm
├── python3-3.12.12-3.el10_1.aarch64.rpm
...# Install python3 and venv from local repo.
## sudo dnf install -y --disablerepo=* --enablerepo=offline-repo python${PY}
./setup-py.sh
===> Install python, venv, etc
Last metadata expiration check: 0:06:40 ago on Wed 04 Feb 2026 12:23:11 AM KST.
Package python3-3.12.12-3.el10_1.aarch64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
# 변수 확인
source pyver.sh
echo -e "python_version $python${PY}"
python_version 3.12
# offline-repo 에 패키지 파일 확인
dnf info python3
tree rpms/local/ | grep -i python
├── libcap-ng-python3-0.8.4-6.el10.aarch64.rpm
├── python3-3.12.12-3.el10_1.aarch64.rpm
...
[5] start-registry.sh 실행 : (컨테이너) 이미지 저장소 컨테이너로 기동
> start-registry.sh
더보기
# Start docker private registry container.
./start-registry.sh
===> Start registry
# (옵션) 실행 명령 참고
echo "===> Start registry"
sudo /usr/local/bin/nerdctl run -d \
--network host \
-e REGISTRY_HTTP_ADDR=0.0.0.0:${REGISTRY_PORT} \
--restart always \
--name registry \
-v $REGISTRY_DIR:/var/lib/registry \
$REGISTRY_IMAGE || exit 1
# 관련 변수 확인
source config.sh
echo -e "registry_port: $REGISTRY_PORT"
registry_port: 35000
# 확인
nerdctl ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
778cd497fcaf docker.io/library/registry:3.0.0 "/entrypoint.sh /etc…" About a minute ago Up registry
0b71e6989724 docker.io/library/nginx:1.29.4 "/docker-entrypoint.…" 30 minutes ago Up nginx
ss -tnlp | grep registry
LISTEN 0 4096 *:35000 *:* users:(("registry",pid=20021,fd=7))
LISTEN 0 4096 *:5001 *:* users:(("registry",pid=20021,fd=3))
# 현재는 registry 서버 내부에 저장된 (컨테이너) 이미지 없는 상태 : (참고) REGISTRY_DIR=${REGISTRY_DIR:-/var/lib/registry}
tree /var/lib/registry/
/var/lib/registry/
# tcp 5001 port : debug, metrics
curl 192.168.10.10:5001/metrics
curl 192.168.10.10:5001/debug/pprof/# Start docker private registry container.
./start-registry.sh
===> Start registry
# (옵션) 실행 명령 참고
echo "===> Start registry"
sudo /usr/local/bin/nerdctl run -d \
--network host \
-e REGISTRY_HTTP_ADDR=0.0.0.0:${REGISTRY_PORT} \
--restart always \
--name registry \
-v $REGISTRY_DIR:/var/lib/registry \
$REGISTRY_IMAGE || exit 1
# 관련 변수 확인
source config.sh
echo -e "registry_port: $REGISTRY_PORT"
registry_port: 35000
# 확인
nerdctl ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
778cd497fcaf docker.io/library/registry:3.0.0 "/entrypoint.sh /etc…" About a minute ago Up registry
0b71e6989724 docker.io/library/nginx:1.29.4 "/docker-entrypoint.…" 30 minutes ago Up nginx
ss -tnlp | grep registry
LISTEN 0 4096 *:35000 *:* users:(("registry",pid=20021,fd=7))
LISTEN 0 4096 *:5001 *:* users:(("registry",pid=20021,fd=3))
# 현재는 registry 서버 내부에 저장된 (컨테이너) 이미지 없는 상태 : (참고) REGISTRY_DIR=${REGISTRY_DIR:-/var/lib/registry}
tree /var/lib/registry/
/var/lib/registry/
# tcp 5001 port : debug, metrics
curl 192.168.10.10:5001/metrics
curl 192.168.10.10:5001/debug/pprof/
[6] load-push-images.sh 실행 : (컨테이너) 이미지 저장소에 이미지 push
> load-push-images.sh
더보기
# 스크립트 실행 전 관련 변수 확인
echo -e "cpu arch: $IMAGE_ARCH"
cpu arch: arm64
## (옵션) 'registry.k8s.io k8s.gcr.io gcr.io ghcr.io docker.io quay.io' 이외에 추가로 필요한 저장소가 있다면 설정
echo -e "Additional container registry hosts: $ADDITIONAL_CONTAINER_REGISTRY_LIST"
Additional container registry hosts: myregistry.io
# nerdctl load 할 .tar.gz 파일들
ls -l images/*.tar.gz
-rw-r--r--. 1 root root 10068768 Feb 3 19:42 images/docker.io_amazon_aws-alb-ingress-controller-v1.1.9.tar.gz
-rw-r--r--. 1 root root 175407403 Feb 3 19:44 images/docker.io_amazon_aws-ebs-csi-driver-v0.5.0.tar.gz
-rw-r--r--. 1 root root 95707259 Feb 3 19:40 images/docker.io_cloudnativelabs_kube-router-v2.1.1.tar.gz
...
# Load all container images to containerd. Tag and push them to the private registry.
./load-push-all-images.sh
# FATA[0003] image might be filtered out (Hint: set `--platform=PLATFORM` or `--all-platforms`)
# aarch64) PLATFORM="linux/arm64" vs x86_64) PLATFORM="linux/amd64"
# (TS) mac사용자: 아래 내용 추가 후 스크립트 다시 실행
vi load-push-all-images.sh
# 아래 추가 -----------------------
...
load_images() {
for image in $BASEDIR/images/*.tar.gz; do
echo "===> Loading $image"
sudo $NERDCTL load --all-platforms -i $image || exit 1
done
...
push_images() {
...
echo "===> Push ${newImage}"
sudo $NERDCTL push --platform=linux/arm64 ${newImage} || exit 1 # 왼쪽 추가 안해도 됨
done
# -------------------------------
# 다시 실행 후 성공! : 2분 소요
./load-push-all-images.sh
# 로컬 이미지 load 확인
nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
localhost:35000/kube-proxy v1.34.3 fa5ed2c96dd3 46 seconds ago linux/arm64 78.05MB 75.94MB
localhost:35000/kube-scheduler v1.34.3 985575f183de 46 seconds ago linux/arm64 53.34MB 51.59MB
localhost:35000/kube-controller-manager v1.34.3 354700b61969 47 seconds ago linux/arm64 74.38MB 72.62MB
localhost:35000/kube-apiserver v1.34.3 dece5cf2dd3b 47 seconds ago linux/arm64 86.56MB 84.81MB
...
localhost:35000/flannel/flannel-cni-plugin v1.7.1-flannel1 332db17b4c4a About a minute ago linux/arm64 11.39MB 11.37MB
localhost:35000/flannel/flannel v0.27.3 3b36a8d4db19 About a minute ago linux/arm64 102.6MB 101.5MB
...
registry.k8s.io/pause 3.10.1 3f85f9d8a6bc About a minute ago linux/arm64 516.1kB 516.9kB
registry.k8s.io/metrics-server/metrics-server v0.8.0 87ccea7af925 About a minute ago linux/arm64 82.58MB 80.84MB
registry.k8s.io/kube-scheduler v1.34.3 985575f183de About a minute ago linux/arm64 53.34MB 51.59MB
registry.k8s.io/kube-proxy v1.34.3 fa5ed2c96dd3 About a minute ago linux/arm64 78.05MB 75.94MB
registry.k8s.io/kube-controller-manager v1.34.3 354700b61969 About a minute ago linux/arm64 74.38MB 72.62MB
registry.k8s.io/kube-apiserver v1.34.3 dece5cf2dd3b About a minute ago linux/arm64 86.56MB 84.81MB
registry.k8s.io/ingress-nginx/controller v1.13.3 68a587e5104f About a minute ago linux/arm64 336.3MB 334.2MB
registry.k8s.io/dns/k8s-dns-node-cache 1.25.0 7071feee8b70 About a minute ago linux/arm64 90.54MB 88.43MB
registry.k8s.io/cpa/cluster-proportional-autoscaler v1.8.8 4146047e636f About a minute ago linux/arm64 39.98MB 37.86MB
registry.k8s.io/coredns/coredns v1.12.1 e674cf21adf3 About a minute ago linux/arm64 74.94MB 73.19MB
...
quay.io/metallb/speaker v0.13.9 51f18d4f5d4d About a minute ago linux/arm64 111.1MB 111.1MB
quay.io/metallb/controller v0.13.9 b724b69a4c9b About a minute ago linux/arm64 63.12MB 63.11MB
quay.io/jetstack/cert-manager-webhook v1.15.3 2d91656807bb About a minute ago linux/arm64 58.15MB 56.39MB
quay.io/jetstack/cert-manager-controller v1.15.3 5114bfbeac23 About a minute ago linux/arm64 67.13MB 65.37MB
quay.io/jetstack/cert-manager-cainjector v1.15.3 a13418dc926e About a minute ago linux/arm64 44.65MB 42.89MB
quay.io/coreos/etcd v3.5.26 4b003fe9069c About a minute ago linux/arm64 66.06MB 63.34MB
...
rancher/local-path-provisioner v0.0.32 4a3d51575c84 2 minutes ago linux/arm64 61.37MB 61.35MB
mirantis/k8s-netchecker-server v1.2.2 8e0ef348cf54 2 minutes ago linux/amd64 125.8MB 123.7MB
mirantis/k8s-netchecker-agent v1.2.2 e07c83f8f083 2 minutes ago linux/amd64 5.681MB 5.856MB
flannel/flannel v0.27.3 3b36a8d4db19 2 minutes ago linux/arm64 102.6MB 101.5MB
flannel/flannel-cni-plugin v1.7.1-flannel1 332db17b4c4a 2 minutes ago linux/arm64 11.39MB 11.37MB
...
# (참고) kube-proxy 컨테이너가 localhost:35000 과 registry.k8s.io 로 push 된 상태 확인
nerdctl images | grep -i kube-proxy
localhost:35000/kube-proxy v1.34.3 fa5ed2c96dd3 24 minutes ago linux/arm64 78.05MB 75.94MB
registry.k8s.io/kube-proxy v1.34.3 fa5ed2c96dd3 24 minutes ago linux/arm64 78.05MB 75.94MB
# localhost 있는 이미지가 각각 registry, quay, rancher, flannel 등으로 push 된것을 알 수 있다
nerdctl images | grep localhost
nerdctl images | grep localhost | wc -l
55
nerdctl images | grep -v localhost
nerdctl images | grep -v localhost | wc -l
56
# 이미지 저장소 카탈로그 확인
curl -s http://localhost:35000/v2/_catalog | jq
{
"repositories": [
"amazon/aws-alb-ingress-controller",
"amazon/aws-ebs-csi-driver",
...
# kube-apiserver 정보 확인
curl -s http://localhost:35000/v2/kube-apiserver/tags
curl -s http://localhost:35000/v2/kube-apiserver/tags/list | jq
{
"name": "kube-apiserver",
"tags": [
"v1.34.3"
]
}
## Image Manifest
curl -s http://localhost:35000/v2/kube-apiserver/manifests/v1.34.3 | jq
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:cf65ae6c8f700cc27f57b7305c6e2b71276a7eed943c559a0091e1e667169896",
"size": 2906
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar",
"digest": "sha256:378b3db0974f7a5a8767b6329ad310983bc712d0e400ff5faa294f95f869cc8c",
"size": 327680
},
...(생략)
# 이미지 저장소의 저장 디렉터리 확인
tree /var/lib/registry/ -L 5
/var/lib/registry/
└── docker
└── registry
└── v2
├── blobs
│ └── sha256
└── repositories
├── amazon
├── calico
├── cilium
├── cloudnativelabs
├── coredns
├── coreos
├── cpa
├── dns
├── flannel
├── ingress-nginx
├── jetstack
├── k8snetworkplumbingwg
├── kube-apiserver
├── kube-controller-manager
├── kubeovn
├── kube-proxy
├── kubernetesui
├── kube-scheduler
├── kube-vip
├── library
├── metallb
├── metrics-server
├── mirantis
├── pause
├── provider-os
├── rancher
└── sig-storage
[7] extract-kubespary.sh 실행 : kubespary 저장소 압축 해제
> extract-kubespary.sh
더보기
#!/bin/bash
cd $(dirname $0)
CURRENT_DIR=$(pwd)
source ./config.sh
KUBESPRAY_TARBALL=files/kubespray-${KUBESPRAY_VERSION}.tar.gz
DIR=kubespray-${KUBESPRAY_VERSION}
if [ -d $DIR ]; then
echo "${DIR} already exists."
exit 0
fi
if [ ! -e $KUBESPRAY_TARBALL ]; then
echo "$KUBESPRAY_TARBALL does not exist."
exit 1
fi
tar xvzf $KUBESPRAY_TARBALL || exit 1
# apply patches
sleep 1 # avoid annoying patch error in shared folders.
if [ -d $CURRENT_DIR/patches/${KUBESPRAY_VERSION} ]; then
for patch in $CURRENT_DIR/patches/${KUBESPRAY_VERSION}/*.patch; do
if [[ -f "${patch}" ]]; then
echo "===> Apply patch: $patch"
(cd $DIR && patch -p1 < $patch)
fi
done
fi
kubespary-2.18.0 버전에서 patch 되는 내용으로, 2.30.0과 관계없음
From 24f1402a142bfd72c5c0a624fdeef0b30c43c490 Mon Sep 17 00:00:00 2001
From: Choi Yongbeom <59861163+mircyb@users.noreply.github.com>
Date: Wed, 5 Jan 2022 18:14:33 +0900
Subject: [PATCH] nerdctl insecure registry config (#8339)
* Update prep_download.yml
nerdctl insecure registry config
* Update prep_download.yml
* Update prep_download.yml
apply conversations advice
* Update prep_download.yml
* Update prep_download.yml
* Update prep_download.yml
* Update prep_download.yml
* Update prep_download.yml
* Update prep_download.yml
* Update main.yml
* Update main.yml
* Update prep_download.yml
* Update prep_download.yml
---
roles/download/defaults/main.yml | 3 +++
roles/download/tasks/prep_download.yml | 4 ++--
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/roles/download/defaults/main.yml b/roles/download/defaults/main.yml
index eecee3f5..a0cf9dbe 100644
--- a/roles/download/defaults/main.yml
+++ b/roles/download/defaults/main.yml
@@ -48,6 +48,9 @@ download_delegate: "{% if download_localhost %}localhost{% else %}{{ groups['kub
# Arch of Docker images and needed packages
image_arch: "{{host_architecture | default('amd64')}}"
+# Nerdctl insecure flag set
+nerdctl_extra_flags: '{%- if containerd_insecure_registries is defined and containerd_insecure_registries|length>0 -%}\" --insecure-registry"{%- else -%}{%- endif -%}'
+
# Versions
kubeadm_version: "{{ kube_version }}"
etcd_version: v3.5.0
diff --git a/roles/download/tasks/prep_download.yml b/roles/download/tasks/prep_download.yml
index 6fc84bc0..342f20c8 100644
--- a/roles/download/tasks/prep_download.yml
+++ b/roles/download/tasks/prep_download.yml
@@ -16,7 +16,7 @@
- name: prep_download | Set image pull/info command for containerd
set_fact:
image_info_command: "{{ bin_dir }}/nerdctl -n k8s.io images --format '{% raw %}{{ '{{' }} .Repository {{ '}}' }}:{{ '{{' }} .Tag {{ '}}' }}{% endraw %}' 2>/dev/null | grep -v ^:$ | tr '\n' ','"
- image_pull_command: "{{ bin_dir }}/nerdctl -n k8s.io pull --quiet"
+ image_pull_command: "{{ bin_dir }}/nerdctl -n k8s.io pull --quiet{{ nerdctl_extra_flags }}"
when: container_manager == 'containerd'
- name: prep_download | Set image pull/info command for crio
@@ -34,7 +34,7 @@
- name: prep_download | Set image pull/info command for containerd on localhost
set_fact:
image_info_command_on_localhost: "{{ bin_dir }}/nerdctl -n k8s.io images --format '{% raw %}{{ '{{' }} .Repository {{ '}}' }}:{{ '{{' }} .Tag {{ '}}' }}{% endraw %}' 2>/dev/null | grep -v ^:$ | tr '\n' ','"
- image_pull_command_on_localhost: "{{ bin_dir }}/nerdctl -n k8s.io pull --quiet"
+ image_pull_command_on_localhost: "{{ bin_dir }}/nerdctl -n k8s.io pull --quiet{{ nerdctl_extra_flags }}"
when: container_manager_on_localhost == 'containerd'
- name: prep_download | Set image pull/info command for crio on localhost
--
2.35.1
# 스크립트 실행 전 관련 파일 확인 : kubespary repo 압축된 파일
ls -lh files/kubespray-*
-rw-r--r--. 1 root root 2.5M Feb 3 19:30 files/kubespray-2.30.0.tar.gz
# patches 파일 : kubespary-2.18.0 버전에서 patch 되는 내용으로, kubespary-2.30.0과 관계없음
tree patches/
└── 2.18.0
├── 0001-nerdctl-insecure-registry-config-8339.patch
├── 0002-Update-config.toml.j2-8340.patch
└── 0003-generate-list-8537.patch
# Extract kubespray tarball and apply all patches.
./extract-kubespray.sh
# kubespary 저장소 압축해제된 파일들 확인
tree kubespray-2.30.0/ -L 1
kubespray-2.30.0/
├── ansible.cfg
├── CHANGELOG.md
├── cluster.yml
├── CNAME
├── code-of-conduct.md
├── _config.yml
├── contrib
├── CONTRIBUTING.md
├── Dockerfile
├── docs
├── extra_playbooks
├── galaxy.yml
├── index.html
├── inventory
├── library
├── LICENSE
├── logo
├── meta
├── OWNERS
├── OWNERS_ALIASES
├── pipeline.Dockerfile
├── playbooks
├── plugins
├── README.md
├── recover-control-plane.yml
├── RELEASE.md
├── remove-node.yml
├── remove_node.yml
├── requirements.txt
├── reset.yml
├── roles
├── scale.yml
├── scripts
├── SECURITY_CONTACTS
├── test-infra
├── tests
├── upgrade-cluster.yml
├── upgrade_cluster.yml
└── Vagrantfile
14 directories, 26 files
kubespary 설치
kubespary 설치 3분 소요
> offline.yml : etcd cpu arch 변수 수정 {{ etcd_version }}-linux-amd64 → {{ etcd_version }}-linux-{{ image_arch }}
더보기
# check python version
python --version
Python 3.12.12
# venv 실행
python3.12 -m venv ~/.venv/3.12
source ~/.venv/3.12/bin/activate
which ansible
tree ~/.venv/3.12/ -L 4
/root/.venv/3.12/
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── Activate.ps1
│ ├── ansible
...
# kubespary 디렉터리 이동
cd /root/kubespray-offline/outputs/kubespray-2.30.0
# Install ansible : 이미 설치 완료된 상태
pip install -U pip # update pip
pip install -r requirements.txt # Install ansible
# offline.yml 파일 복사 후 inventory 복사
cp ../../offline.yml .
cp -r inventory/sample inventory/mycluster
tree inventory/mycluster/
# 웹서버와 이미지 저장소 정보 수정 : http_server, registry_host
cat offline.yml
sed -i "s/YOUR_HOST/192.168.10.10/g" offline.yml
cat offline.yml | grep 192.168.10.10
http_server: "http://192.168.10.10"
registry_host: "192.168.10.10:35000"
# 수정 반영된 offline.yml 파일을 inventory 디렉터리 내부로 복사
\cp -f offline.yml inventory/mycluster/group_vars/all/offline.yml
cat inventory/mycluster/group_vars/all/offline.yml
# inventory 파일 작성
cat <<EOF > inventory/mycluster/inventory.ini
[kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
[etcd:children]
kube_control_plane
[kube_node]
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12
EOF
cat inventory/mycluster/inventory.ini
# ansible 연결 확인
ansible -i inventory/mycluster/inventory.ini all -m ping
# 각 노드에 offline repo 설정
tree ../playbook/
├── offline-repo.yml
└── roles
└── offline-repo
├── defaults
│ └── main.yml
├── files
│ └── 99offline
└── tasks
├── Debian.yml
├── main.yml
└── RedHat.yml
mkdir offline-repo
cp -r ../playbook/ offline-repo/
tree offline-repo/
ansible-playbook -i inventory/mycluster/inventory.ini offline-repo/playbook/offline-repo.yml
# k8s-node 확인
ssh k8s-node1 tree /etc/yum.repos.d/
ssh k8s-node1 dnf repolist
repo id repo name
appstream Rocky Linux 10 - AppStream
baseos Rocky Linux 10 - BaseOS
extras Rocky Linux 10 - Extras
offline-repo Offline repo for kubespray
ssh k8s-node1 cat /etc/yum.repos.d/offline.repo
[offline-repo]
baseurl = http://192.168.10.10/rpms/local
enabled = 1
gpgcheck = 0
name = Offline repo for kubespray
## 추가로 설치를 위해 기존 repo 제거 : 미실행할 경우, kubespary 실행 시 fail됨
for i in rocky-addons rocky-devel rocky-extras rocky; do
ssh k8s-node1 "mv /etc/yum.repos.d/$i.repo /etc/yum.repos.d/$i.repo.original"
ssh k8s-node2 "mv /etc/yum.repos.d/$i.repo /etc/yum.repos.d/$i.repo.original"
done
ssh k8s-node1 tree /etc/yum.repos.d/
ssh k8s-node1 dnf repolist
ssh k8s-node2 tree /etc/yum.repos.d/
ssh k8s-node2 dnf repolist
# admin-lb 에 kubectl 없는 것 확인
which kubectl
# group vars 실습 환경에 맞게 설정
echo "kubectl_localhost: true" >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml # 배포를 수행하는 로컬 머신의 bin 디렉토리에도 kubectl 바이너리를 다운로드
sed -i 's|kube_owner: kube|kube_owner: root|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|kube_network_plugin: calico|kube_network_plugin: flannel|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|kube_proxy_mode: ipvs|kube_proxy_mode: iptables|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|enable_nodelocaldns: true|enable_nodelocaldns: false|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
grep -iE 'kube_owner|kube_network_plugin:|kube_proxy_mode|enable_nodelocaldns:' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
echo "enable_dns_autoscaler: false" >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
echo "flannel_interface: enp0s9" >> inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
sed -i 's|helm_enabled: false|helm_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
sed -i 's|metrics_server_enabled: false|metrics_server_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
grep -iE 'metrics_server_enabled:' inventory/mycluster/group_vars/k8s_cluster/addons.yml
echo "metrics_server_requests_cpu: 25m" >> inventory/mycluster/group_vars/k8s_cluster/addons.yml
echo "metrics_server_requests_memory: 16Mi" >> inventory/mycluster/group_vars/k8s_cluster/addons.yml
# 지원 버전 정보 확인
cat roles/kubespray_defaults/vars/main/checksums.yml | grep -i kube -A40
# [macOS 사용자] (TS) 이슈 해결
# -----------------------------------------------
TASK [download : Download_file | Download item] **************************************************************************
fatal: [k8s-node1]: FAILED! => {"attempts": 4, "changed": false, "dest": "/tmp/releases/etcd-3.5.26-linux-arm64.tar.gz", "elapsed": 0, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "http://192.168.10.10/files/kubernetes/etcd/etcd-v3.5.26-linux-amd64.tar.gz"}
http://192.168.10.10/files/kubernetes/etcd/etcd-v3.5.26-linux-amd64.tar.gz
# vi roles/download/tasks/download_file.yml >> no_log: false # (참고) 실패 task 상세 로그 출력 설정하여 원인 파악
cat inventory/mycluster/group_vars/all/offline.yml | grep amd64
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-v{{ etcd_version }}-linux-amd64.tar.gz"
sed -i 's/amd64/arm64/g' inventory/mycluster/group_vars/all/offline.yml
# -----------------------------------------------
# 배포 : 설치 완료까지 3분!
ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml -e kube_version="1.34.3"
# 설치 후 NetworkManger 에 dns 설정 파일 추가 확인
ssh k8s-node2 cat /etc/NetworkManager/conf.d/dns.conf
[global-dns-domain-*]
servers = 10.233.0.3,192.168.10.10
[global-dns]
searches = default.svc.cluster.local,svc.cluster.local
options = ndots:2,timeout:2,attempts:2
# 하지만 '/etc/NetworkManager/conf.d/99-dns-none.conf' 파일로 인해, 위 설정이 resolv.conf 에 반영되지 않음, 즉 노드에서는 service명으로 도메인 질의는 불가능.
ssh k8s-node2 cat /etc/resolv.conf
nameserver 192.168.10.10
# 설치 후 NetworkManger 에서 특정 NIC은 관리하지 않게 설정 추가 확인
ssh k8s-node2 cat /etc/NetworkManager/conf.d/k8s.conf
[keyfile]
unmanaged-devices+=interface-name:kube-ipvs0;interface-name:nodelocaldns
# kubectl 바이너리 파일을 ansible-playbook 실행한 서버에 다운로드 확인
file inventory/mycluster/artifacts/kubectl
ls -l inventory/mycluster/artifacts/kubectl
tree inventory/mycluster/
inventory/mycluster/
├── artifacts
│ └── kubectl
├── credentials
│ └── kubeadm_certificate_key.creds
├── group_vars
...
cp inventory/mycluster/artifacts/kubectl /usr/local/bin/
kubectl version --client=true
Client Version: v1.34.3
Kustomize Version: v5.7.1
# k8s admin 자격증명 확인
mkdir /root/.kube
scp k8s-node1:/root/.kube/config /root/.kube/
sed -i 's/127.0.0.1/192.168.10.11/g' /root/.kube/config
k9s
# 자동완성 및 단축키 설정
source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k
echo 'source <(kubectl completion bash)' >> /etc/profile
echo 'alias k=kubectl' >> /etc/profile
echo 'complete -F __start_kubectl k' >> /etc/profile
# 이미지 저장소가 192.168.10.10:35000 임을 확인
kubectl get deploy,sts,ds -n kube-system -owide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/coredns 2/2 2 2 4m9s coredns 192.168.10.10:35000/coredns/coredns:v1.12.1 k8s-app=kube-dns
deployment.apps/metrics-server 1/1 1 1 4m5s metrics-server 192.168.10.10:35000/metrics-server/metrics-server:v0.8.0 app.kubernetes.io/name=metrics-server,version=0.8.0
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
daemonset.apps/kube-flannel 0 0 0 0 0 <none> 4m25s kube-flannel 192.168.10.10:35000/flannel/flannel:v0.27.3 app=flannel
...
daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 4m43s kube-proxy 192.168.10.10:35000/kube-proxy:v1.34.3 k8s-app=kube-proxy(TS) flannel 파드 최초 정상 기동을 위해 k8s-node 에 디폴트 라우팅 추가 필요 : 해결을 위해 k8s-node 에 디폴트 라우트 적용되어 있음
더보기
# (TS) k8s-node 에 디폴트 라우팅 추가 필요
# -----------------------------------------------
# Flannel 파드가 정상 기동 시, kube-flannel 컨테이너가 호스트에 /run/flannel/subnet.env 파일 생성함
# Flannel은 기본적으로 노드의 Default Route가 설정된 인터페이스를 찾아서 그 IP를 노드 간 통신(VxLAN 등)의 엔드포인트로 사용하려 합니다.
# 기본 라우팅이 없으면 Flannel이 어떤 인터페이스를 통해 다른 노드와 통신해야 할지 결정하지 못해 실행이 중단됨. 결국 파드가 정상 기동 실패됨.
# 파드 기동 실패가 되면, 호스트에 /run/flannel/subnet.env 파일 생성이 안됨. 결국 해당 Task 에서 실패하게 됨!
TASK [network_plugin/flannel : Flannel | Wait for flannel subnet.env file presence] **************************************
fatal: [k8s-node1]: FAILED! => {"changed": false, "elapsed": 600, "msg": "Timeout when waiting for file /run/flannel/subnet.env"}
#ansible-playbook -i inventory/mycluster/inventory.ini -v reset.yml -e kube_version="1.34.3"
[k8s-nodes]
nmcli connection modify enp0s9 +ipv4.routes "0.0.0.0/0 192.168.10.10 200"
nmcli connection up enp0s9
ip route
[admin-lb]
# NAT 설정 제거하여, 실제적으로 k8s-node 가 외부 인터넷이 안되는 상황을 만들고 실습 진행해보자
iptables -t nat -D POSTROUTING -o enp0s8 -j MASQUERADE
iptables -t nat -S
# -----------------------------------------------
[k8s-node] 이미지 저장소 관련 정보 확인
더보기
[k8s-node1 , k8s-node2]
# 로컬 이미지 확인
crictl images
IMAGE TAG IMAGE ID SIZE
192.168.10.10:35000/coredns/coredns v1.12.1 138784d87c9c5 73.2MB
192.168.10.10:35000/flannel/flannel-cni-plugin v1.7.1-flannel1 e5bf9679ea8c3 11.4MB
192.168.10.10:35000/flannel/flannel v0.27.3 cadcae92e6360 102MB
192.168.10.10:35000/kube-apiserver v1.34.3 cf65ae6c8f700 84.8MB
192.168.10.10:35000/kube-controller-manager v1.34.3 7ada8ff13e54b 72.6MB
192.168.10.10:35000/kube-proxy v1.34.3 4461daf6b6af8 75.9MB
192.168.10.10:35000/kube-scheduler v1.34.3 2f2aa21d34d2d 51.6MB
192.168.10.10:35000/library/nginx 1.28.0-alpine 5a91d90f47ddf 51.2MB
192.168.10.10:35000/metrics-server/metrics-server v0.8.0 bc6c1e09a843d 80.8MB
192.168.10.10:35000/pause 3.10.1 d7b100cd9a77b 517kB
# containerd 정보 확인
tree /etc/containerd/
/etc/containerd/
├── certs.d
│ └── 192.168.10.10:35000
│ └── hosts.toml
├── config.toml
└── cri-base.json
cat /etc/containerd/config.toml
...
[plugins."io.containerd.cri.v1.images".pinned_images]
sandbox = "192.168.10.10:35000/pause:3.10.1"
[plugins."io.containerd.cri.v1.images".registry]
config_path = "/etc/containerd/certs.d"
....
cat /etc/containerd/certs.d/192.168.10.10\:35000/hosts.toml
server = "https://192.168.10.10:35000"
[host."http://192.168.10.10:35000"]
capabilities = ["pull","resolve"]
skip_verify = true
override_path = false
kubespary 에 offline 설치를 위한 지원
더보기
다운로드 파일, 이미지 목록 작성* : generate_list.sh → generate_list.yml
> generate_list.sh
# generate_list.sh
#!/bin/bash
set -eo pipefail
CURRENT_DIR=$(cd $(dirname $0); pwd)
TEMP_DIR="${CURRENT_DIR}/temp"
REPO_ROOT_DIR="${CURRENT_DIR%/contrib/offline}"
: ${DOWNLOAD_YML:="roles/kubespray_defaults/defaults/main/download.yml"}
mkdir -p ${TEMP_DIR}
# generate all download files url template
grep 'download_url:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
| sed 's/^.*_url: //g;s/\"//g' > ${TEMP_DIR}/files.list.template
# generate all images list template
sed -n '/^downloads:/,/download_defaults:/p' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
| sed -n "s/repo: //p;s/tag: //p" | tr -d ' ' \
| sed 'N;s#\n# #g' | tr ' ' ':' | sed 's/\"//g' > ${TEMP_DIR}/images.list.template
# add kube-* images to images list template
# Those container images are downloaded by kubeadm, then roles/kubespray_defaults/defaults/main/download.yml
# doesn't contain those images. That is reason why here needs to put those images into the
# list separately.
KUBE_IMAGES="kube-apiserver kube-controller-manager kube-scheduler kube-proxy"
for i in $KUBE_IMAGES; do
echo "{{ kube_image_repo }}/$i:v{{ kube_version }}" >> ${TEMP_DIR}/images.list.template
done
# run ansible to expand templates
/bin/cp ${CURRENT_DIR}/generate_list.yml ${REPO_ROOT_DIR}
(cd ${REPO_ROOT_DIR} && ansible-playbook $* generate_list.yml && /bin/rm generate_list.yml) || exit 1
> generate_list.yml
---
- name: Collect container images for offline deployment
hosts: localhost
become: false
roles:
# Just load default variables from roles.
- role: kubespray_defaults
when: false
- role: download
when: false
tasks:
# Generate files.list and images.list files from templates.
- name: Collect container images for offline deployment
template:
src: ./contrib/offline/temp/{{ item }}.list.template
dest: ./contrib/offline/temp/{{ item }}.list
mode: "0644"
with_items:
- files
- images
> download.yml
#
cd /root/kubespray-offline/outputs/kubespray-2.30.0/contrib/offline
tree
├── docker-daemon.json
├── generate_list.sh
├── generate_list.yml
├── manage-offline-container-images.sh
├── manage-offline-files.sh
├── nginx.conf
├── README.md
├── registries.conf
└── upload2artifactory.py
# 다운받을 파일과 (컨테이너)이미지 목록 파일로 작성 스크립트(ansible generate_list.yml 실행 포함) 실행
# https://github.com/kubernetes-sigs/kubespray/blob/master/roles/kubespray_defaults/defaults/main/download.yml
cat ../../roles/kubespray_defaults/defaults/main/download.yml # 다운로드될 파일 목록과 컨테이너 이미지 목록 참고 파일
./generate_list.sh
PLAY [Collect container images for offline deployment] ****************************************************************************
TASK [Collect container images for offline deployment] ****************************************************************************
...
# temp 디렉터리 확인
tree temp
├── files.list
├── files.list.template
├── images.list
└── images.list.template
# 파일 목록 확인
cat temp/files.list.template
{{ dl_k8s_io_url }}/release/v{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet
{{ dl_k8s_io_url }}/release/v{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl
{{ dl_k8s_io_url }}/release/v{{ kube_version }}/bin/linux/{{ image_arch }}/kubeadm
...
cat temp/files.list | tee 1-files.list
https://dl.k8s.io/release/v1.34.3/bin/linux/arm64/kubelet
https://dl.k8s.io/release/v1.34.3/bin/linux/arm64/kubectl
https://dl.k8s.io/release/v1.34.3/bin/linux/arm64/kubeadm
...
# (컨테이너)이미지 목록 확인
cat temp/images.list | tee 1-images.list
tail temp/images.list -n 4 | tee 1-images.list
registry.k8s.io/kube-apiserver:v1.34.3
registry.k8s.io/kube-controller-manager:v1.34.3
registry.k8s.io/kube-scheduler:v1.34.3
registry.k8s.io/kube-proxy:v1.34.3
cat temp/images.list.template
tail temp/images.list.template -n 4
{{ kube_image_repo }}/kube-apiserver:v{{ kube_version }}
{{ kube_image_repo }}/kube-controller-manager:v{{ kube_version }}
{{ kube_image_repo }}/kube-scheduler:v{{ kube_version }}
{{ kube_image_repo }}/kube-proxy:v{{ kube_version }}
# 목록 정보 수정 필요 시: 필요한 변수명 확인 후 설정하여 반영
## cat temp/files.list.template
## cat temp/images.list.template
# cat ../../roles/kubespray_defaults/vars/main/checksums.yml | grep -i kube -A40
# arm64
# 다시 목록 생성 : ansible-playbook 실행이 되니, 스크립트 실행 시 -e 변수 지정 가능
./generate_list.sh -e kube_version=1.33.7 -e image_arch=amd64 # Windows 사용자분들은 arm64 사용할 것!
cat temp/files.list | tee 2-files.list
cat temp/images.list | tee 2-images.list
# 변경 비교
diff 1-files.list 2-files.list
vi -d 1-files.list 2-files.list
diff 1-images.list 2-images.list
vi -d 1-images.list 2-images.list
rm -rf *.list
manage-offline-container-images.sh : 오프라인 배포를 위한 컨테이너 이미지 수집 스크립트
더보기
#!/usr/bin/env bash
OPTION=$1
CURRENT_DIR=$(cd $(dirname $0); pwd)
TEMP_DIR="${CURRENT_DIR}/temp"
IMAGE_TAR_FILE="${CURRENT_DIR}/container-images.tar.gz"
IMAGE_DIR="${CURRENT_DIR}/container-images"
IMAGE_LIST="${IMAGE_DIR}/container-images.txt"
RETRY_COUNT=5
function create_container_image_tar() {
set -e
if [ -z "${IMAGES_FROM_FILE}" ]; then
echo "Getting images from current \"$(kubectl config current-context)\""
IMAGES=$(mktemp --suffix=-images)
trap 'rm -f "${IMAGES}"' EXIT
kubectl describe cronjobs,jobs,pods --all-namespaces | grep " Image:" | awk '{print $2}' | sort | uniq > "${IMAGES}"
# NOTE: etcd and pause cannot be seen as pods.
kubectl cluster-info dump | grep -E "quay.io/coreos/etcd:|registry.k8s.io/pause:" | sed s@\"@@g >> "${IMAGES}"
else
echo "Getting images from file \"${IMAGES_FROM_FILE}\""
if [ ! -f "${IMAGES_FROM_FILE}" ]; then
echo "${IMAGES_FROM_FILE} is not a file"
exit 1
fi
IMAGES=$(realpath $IMAGES_FROM_FILE)
fi
rm -f ${IMAGE_TAR_FILE}
rm -rf ${IMAGE_DIR}
mkdir ${IMAGE_DIR}
cd ${IMAGE_DIR}
sudo --preserve-env=http_proxy,https_proxy,no_proxy ${runtime} pull registry:latest
sudo ${runtime} save -o registry-latest.tar registry:latest
while read -r image
do
FILE_NAME="$(echo ${image} | sed s@"/"@"-"@g | sed s/":"/"-"/g | sed -E 's/\@.*//g')".tar
set +e
for step in $(seq 1 ${RETRY_COUNT})
do
sudo --preserve-env=http_proxy,https_proxy,no_proxy ${runtime} pull ${image}
if [ $? -eq 0 ]; then
break
fi
echo "Failed to pull ${image} at step ${step}"
if [ ${step} -eq ${RETRY_COUNT} ]; then
exit 1
fi
done
set -e
sudo ${runtime} save -o ${FILE_NAME} ${image}
# NOTE: Here removes the following repo parts from each image
# so that these parts will be replaced with Kubespray.
# - kube_image_repo: "registry.k8s.io"
# - gcr_image_repo: "gcr.io"
# - ghcr_image_repo: "ghcr.io"
# - docker_image_repo: "docker.io"
# - quay_image_repo: "quay.io"
FIRST_PART=$(echo ${image} | awk -F"/" '{print $1}')
if [ "${FIRST_PART}" = "registry.k8s.io" ] ||
[ "${FIRST_PART}" = "gcr.io" ] ||
[ "${FIRST_PART}" = "ghcr.io" ] ||
[ "${FIRST_PART}" = "docker.io" ] ||
[ "${FIRST_PART}" = "quay.io" ] ||
[ "${FIRST_PART}" = "${PRIVATE_REGISTRY}" ]; then
image=$(echo ${image} | sed s@"${FIRST_PART}/"@@ | sed -E 's/\@.*/\n/g')
fi
echo "${FILE_NAME} ${image}" >> ${IMAGE_LIST}
done < "${IMAGES}"
cd ..
sudo chown ${USER} ${IMAGE_DIR}/*
tar -zcvf ${IMAGE_TAR_FILE} ./container-images
rm -rf ${IMAGE_DIR}
echo ""
echo "${IMAGE_TAR_FILE} is created to contain your container images."
echo "Please keep this file and bring it to your offline environment."
}
function register_container_images() {
create_registry=false
REGISTRY_PORT=${REGISTRY_PORT:-"5000"}
if [ -z "${DESTINATION_REGISTRY}" ]; then
echo "DESTINATION_REGISTRY not set, will create local registry"
create_registry=true
DESTINATION_REGISTRY="$(hostname):${REGISTRY_PORT}"
fi
echo "Images will be pushed to ${DESTINATION_REGISTRY}"
if [ ! -f ${IMAGE_TAR_FILE} ]; then
echo "${IMAGE_TAR_FILE} should exist."
exit 1
fi
if [ ! -d ${TEMP_DIR} ]; then
mkdir ${TEMP_DIR}
fi
# To avoid "http: server gave http response to https client" error.
if [ -d /etc/docker/ ]; then
set -e
# Ubuntu18.04, RHEL7/CentOS7
cp ${CURRENT_DIR}/docker-daemon.json ${TEMP_DIR}/docker-daemon.json
sed -i s@"HOSTNAME"@"$(hostname)"@ ${TEMP_DIR}/docker-daemon.json
sudo cp ${TEMP_DIR}/docker-daemon.json /etc/docker/daemon.json
elif [ -d /etc/containers/ ]; then
set -e
# RHEL8/CentOS8
cp ${CURRENT_DIR}/registries.conf ${TEMP_DIR}/registries.conf
sed -i s@"HOSTNAME"@"$(hostname)"@ ${TEMP_DIR}/registries.conf
sudo cp ${TEMP_DIR}/registries.conf /etc/containers/registries.conf
elif [ "$(uname)" == "Darwin" ]; then
echo "This is a Mac, no configuration changes are required"
else
echo "runtime package(docker-ce, podman, nerctl, etc.) should be installed"
exit 1
fi
tar -zxvf ${IMAGE_TAR_FILE}
if ${create_registry}; then
sudo ${runtime} load -i ${IMAGE_DIR}/registry-latest.tar
set +e
sudo ${runtime} container inspect registry >/dev/null 2>&1
if [ $? -ne 0 ]; then
sudo ${runtime} run --restart=always -d -p "${REGISTRY_PORT}":"${REGISTRY_PORT}" --name registry registry:latest
fi
set -e
fi
while read -r line; do
file_name=$(echo ${line} | awk '{print $1}')
raw_image=$(echo ${line} | awk '{print $2}')
new_image="${DESTINATION_REGISTRY}/${raw_image}"
load_image=$(sudo ${runtime} load -i ${IMAGE_DIR}/${file_name} | head -n1)
org_image=$(echo "${load_image}" | awk '{print $3}')
# special case for tags containing the digest when using docker or podman as the container runtime
if [ "${org_image}" == "ID:" ]; then
org_image=$(echo "${load_image}" | awk '{print $4}')
fi
image_id=$(sudo ${runtime} image inspect --format "{{.Id}}" "${org_image}")
if [ -z "${file_name}" ]; then
echo "Failed to get file_name for line ${line}"
exit 1
fi
if [ -z "${raw_image}" ]; then
echo "Failed to get raw_image for line ${line}"
exit 1
fi
if [ -z "${org_image}" ]; then
echo "Failed to get org_image for line ${line}"
exit 1
fi
if [ -z "${image_id}" ]; then
echo "Failed to get image_id for file ${file_name}"
exit 1
fi
sudo ${runtime} load -i ${IMAGE_DIR}/${file_name}
sudo ${runtime} tag ${image_id} ${new_image}
sudo ${runtime} push ${new_image}
done <<< "$(cat ${IMAGE_LIST})"
echo "Succeeded to register container images to local registry."
echo "Please specify \"${DESTINATION_REGISTRY}\" for the following options in your inventry:"
echo "- kube_image_repo"
echo "- gcr_image_repo"
echo "- docker_image_repo"
echo "- quay_image_repo"
}
# get runtime command
if command -v nerdctl 1>/dev/null 2>&1; then
runtime="nerdctl"
elif command -v podman 1>/dev/null 2>&1; then
runtime="podman"
elif command -v docker 1>/dev/null 2>&1; then
runtime="docker"
else
echo "No supported container runtime found"
exit 1
fi
if [ "${OPTION}" == "create" ]; then
create_container_image_tar
elif [ "${OPTION}" == "register" ]; then
register_container_images
else
echo "This script has two features:"
echo "(1) Get container images from an environment which is deployed online, or set IMAGES_FROM_FILE"
echo " environment variable to get images from a file (e.g. temp/images.list after running the"
echo " ./generate_list.sh script)."
echo "(2) Deploy local container registry and register the container images to the registry."
echo ""
echo "Step(1) should be done online site as a preparation, then we bring"
echo "the gotten images to the target offline environment. if images are from"
echo "a private registry, you need to set PRIVATE_REGISTRY environment variable."
echo "Then we will run step(2) for registering the images to local registry, or to an existing"
echo "registry set by the DESTINATION_REGISTRY environment variable. By default, the local registry"
echo "will run on port 5000. This can be changed with the REGISTRY_PORT environment variable"
echo ""
echo "${IMAGE_TAR_FILE} is created to contain your container images."
echo "Please keep this file and bring it to your offline environment."
echo ""
echo "Step(1) can be operated with:"
echo " $ ./manage-offline-container-images.sh create"
echo ""
echo "Step(2) can be operated with:"
echo " $ ./manage-offline-container-images.sh register"
echo ""
echo "Please specify 'create' or 'register'."
echo ""
exit 1
fi
manage-offline-files.sh : 지정된 모든 파일을 다운로드하고, Nginx 컨테이너를 실행하여 파일 다운로드 기능 제공
더보기
#!/bin/bash
CURRENT_DIR=$( dirname "$(readlink -f "$0")" )
OFFLINE_FILES_DIR_NAME="offline-files"
OFFLINE_FILES_DIR="${CURRENT_DIR}/${OFFLINE_FILES_DIR_NAME}"
OFFLINE_FILES_ARCHIVE="${CURRENT_DIR}/offline-files.tar.gz"
FILES_LIST=${FILES_LIST:-"${CURRENT_DIR}/temp/files.list"}
NGINX_PORT=80
# download files
if [ ! -f "${FILES_LIST}" ]; then
echo "${FILES_LIST} should exist, run ./generate_list.sh first."
exit 1
fi
rm -rf "${OFFLINE_FILES_DIR}"
rm "${OFFLINE_FILES_ARCHIVE}"
mkdir "${OFFLINE_FILES_DIR}"
while read -r url; do
if ! wget -x -P "${OFFLINE_FILES_DIR}" "${url}"; then
exit 1
fi
done < "${FILES_LIST}"
tar -czvf "${OFFLINE_FILES_ARCHIVE}" "${OFFLINE_FILES_DIR_NAME}"
[ -n "$NO_HTTP_SERVER" ] && echo "skip to run nginx" && exit 0
# run nginx container server
if command -v nerdctl 1>/dev/null 2>&1; then
runtime="nerdctl"
elif command -v podman 1>/dev/null 2>&1; then
runtime="podman"
elif command -v docker 1>/dev/null 2>&1; then
runtime="docker"
else
echo "No supported container runtime found"
exit 1
fi
sudo "${runtime}" container inspect nginx >/dev/null 2>&1
if [ $? -ne 0 ]; then
sudo --preserve-env=http_proxy,https_proxy,no_proxy "${runtime}" run \
--restart=always -d -p ${NGINX_PORT}:80 \
--volume "${OFFLINE_FILES_DIR}":/usr/share/nginx/html/download \
--volume "${CURRENT_DIR}"/nginx.conf:/etc/nginx/nginx.conf \
--name nginx nginx:alpine
fi
upload2artifactory.py : 디렉터리 아래의 각 파일을 Artifactory의 일반 저장소에 재귀적으로 업로드
더보기
#!/usr/bin/env python3
"""This is a helper script to manage-offline-files.sh.
After running manage-offline-files.sh, you can run upload2artifactory.py
to recursively upload each file to a generic repository in Artifactory.
This script recurses the current working directory and is intended to
be started from 'kubespray/contrib/offline/offline-files'
Environment Variables:
USERNAME -- At least permissions'Deploy/Cache' and 'Delete/Overwrite'.
TOKEN -- Generate this with 'Set Me Up' in your user.
BASE_URL -- The URL including the repository name.
"""
import os
import urllib.request
import base64
def upload_file(file_path, destination_url, username, token):
"""Helper function to upload a single file"""
try:
with open(file_path, 'rb') as f:
file_data = f.read()
request = urllib.request.Request(destination_url, data=file_data, method='PUT') # NOQA
auth_header = base64.b64encode(f"{username}:{token}".encode()).decode()
request.add_header("Authorization", f"Basic {auth_header}")
with urllib.request.urlopen(request) as response:
if response.status in [200, 201]:
print(f"Success: Uploaded {file_path}")
else:
print(f"Failed: {response.status} {response.read().decode('utf-8')}") # NOQA
except urllib.error.HTTPError as e:
print(f"HTTPError: {e.code} {e.reason} for {file_path}")
except urllib.error.URLError as e:
print(f"URLError: {e.reason} for {file_path}")
except OSError as e:
print(f"OSError: {e.strerror} for {file_path}")
def upload_files(base_url, username, token):
""" Recurse current dir and upload each file using urllib.request """
for root, _, files in os.walk(os.getcwd()):
for file in files:
file_path = os.path.join(root, file)
relative_path = os.path.relpath(file_path, os.getcwd())
destination_url = f"{base_url}/{relative_path}"
print(f"Uploading {file_path} to {destination_url}")
upload_file(file_path, destination_url, username, token)
if __name__ == "__main__":
a_user = os.getenv("USERNAME")
a_token = os.getenv("TOKEN")
a_url = os.getenv("BASE_URL")
if not a_user or not a_token or not a_url:
print(
"Error: Environment variables USERNAME, TOKEN, and BASE_URL must be set." # NOQA
)
exit()
upload_files(a_url, a_user, a_token)
kubespary-offline 에 kube_version 변경 적용하여 관련 파일 다운로드
더보기
# 디렉터리 이동 후 기존 정보 백업
cd /root/kubespray-offline
tree cache/kubespray-2.30.0/contrib/offline/temp/
├── files.list
├── files.list.template
├── images.list
└── images.list.template
mv cache/kubespray-2.30.0/contrib/offline/temp/files.list cache/kubespray-2.30.0/contrib/offline/temp/files-2.list
mv cache/kubespray-2.30.0/contrib/offline/temp/images.list cache/kubespray-2.30.0/contrib/offline/temp/images-2.list
# (옵션) 다운로드 스크립트에 실제 다운로드 실행하는 부분 제거 시
# -----------------------------------------------
cat download-kubespray-files.sh
cp download-kubespray-files.sh download-kubespray-files.bak
sed -i '/generate_list$/,$ { /generate_list/!d }' download-kubespray-files.sh
diff download-kubespray-files.sh download-kubespray-files.bak
# 다운로드 목록 작성 스크립트(ansible-playbook 후속 실행)에 kube_version=1.33.7 추가
sed -i 's|offline/generate_list.sh|offline/generate_list.sh -e kube_version=1.33.7|g' download-kubespray-files.sh
cat download-kubespray-files.sh | grep kube_version
LANG=C /bin/bash ${KUBESPRAY_DIR}/contrib/offline/generate_list.sh -e kube_version=1.33.7 || exit 1
# -----------------------------------------------
# 스크립트 실행 후 확인
./download-kubespray-files.sh
cd cache/kubespray-2.30.0/contrib/offline/temp
# 최초 설치 버전과 비교 확인
diff files-2.list files.list
vi -d files-2.list files.list
diff images-2.list images.list
vi -d images-2.list images.list
# (옵션) kube_version 1.33.7 관련 버전을 적용하여 파일과 이미지 다운로드 실행 시
cp download-kubespray-files.bak download-kubespray-files.sh
sed -i 's|offline/generate_list.sh|offline/generate_list.sh -e kube_version=1.33.7|g' download-kubespray-files.sh
./download-kubespray-files.sh
K8S 관련 폐쇄망 서비스 실습
폐쇄망에서 서비스 동작을 위해 필요한 주요 구성요소
> 샘플 앱 배포 : nginx:alpine
더보기
# [k8s-node] 외부 통신은 안되는 상태
ping -c 1 -w 1 -W 1 8.8.8.8
ip route
crictl images
tree /etc/containerd/certs.d/
/etc/containerd/certs.d/
└── 192.168.10.10:35000
└── hosts.toml
# [admin] nginx 디플로이먼트 배포 시도
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:alpine # docker.io/library/nginx:alpine
ports:
- containerPort: 80
EOF
# 확인 : docker.io/library/nginx:alpine 이미지 가져오기 실패!
kubectl describe pod
...
Warning Failed 8s kubelet Failed to pull image "nginx:alpine": failed to pull and unpack image "docker.io/library/nginx:alpine": failed to resolve image: failed to do request: Head "https://registry-1.docker.io/v2/library/nginx/manifests/alpine": dial tcp [2600:1f18:2148:bc01:c6a5:489e:7233:242c]:443: connect: no route to host
...
> (컨테이너) 이미지 저장소에 이미지 push
더보기
# [admin] (컨테이너) 이미지 저장소에 이미지 push
podman images
# 로컬에 nginx:alpine 다운로드
podman pull nginx:alpine
docker.io/library/nginx:alpine 선택
podman images | grep nginx
# (컨테이너) 이미지 저장소에 이미지 push
podman tag nginx:alpine 192.168.10.10:35000/library/nginx:alpine
podman images | grep nginx
# 기본적으로 컨테이너 엔진들은 HTTPS를 요구합니다. 내부망에서 HTTP로 테스트하려면 Registry 주소를 '안전하지 않은 저장소'로 등록해야 합니다.
# (참고) registries.conf 는 containers-common 설정이라서, 'podman, skopeo, buildah' 등 전부 동일하게 적용됨.
cat <<EOF >> /etc/containers/registries.conf
[[registry]]
location = "192.168.10.10:35000"
insecure = true
EOF
grep "^[^#]" /etc/containers/registries.conf
# 프라이빗 레지스트리에 업로드 : 성공!
podman push 192.168.10.10:35000/library/nginx:alpine
# 업로드된 이미지와 태그 조회
curl -s 192.168.10.10:35000/v2/_catalog | jq
{
"repositories": [
...
"library/nginx",
curl -s 192.168.10.10:35000/v2/library/nginx/tags/list | jq
{
"name": "library/nginx",
"tags": [
"1.28.0-alpine",
"1.29.4",
"alpine"
]
}> 샘플 앱 이미지 업데이트 후 배포 확인
더보기
# 현재 파드 상태
kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-54fc99c8d-m6fl5 0/1 ImagePullBackOff 0 16m
# 이미지 정보
kubectl get deploy -owide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
nginx 0/1 1 0 17m nginx nginx:alpine app=nginx
# 디플로이먼트에 이미지 정보 업데이트
kubectl set image deployment/nginx nginx=192.168.10.10:35000/library/nginx:alpine
kubectl get deploy -owide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
nginx 1/1 1 1 19m nginx 192.168.10.10:35000/library/nginx:alpine app=nginx
# 현재 파드 상태
kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-5ff7dd7b8-t6b2n 1/1 Running 0 22s
> k8s-node 에서 저장소 미러 설정 후 배포 확인
더보기
# 삭제 후 다시 디플로이먼트 배포
kubectl delete deployments.apps nginx
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:alpine # docker.io/library/nginx:alpine
ports:
- containerPort: 80
EOF
# 현재 파드 상태
kubectl get pod
[k8s-node1, k8s-node2]
# docker.io 대신 내부 이미지 레지스트리 주소 설정
mkdir -p /etc/containerd/certs.d/docker.io
cat <<EOF > /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io" # [HTTPS] 원본 registry 주소 , docker.io 를 대상으로 할 때 이 설정을 참조한다는 의미
[host."http://192.168.10.10:35000"] # [HTTP] 내부 레지스트리를 미러로 지정 , docker.io 대신 실제 이미지를 가져올 내부 레지스트리 주소
capabilities = ["pull", "resolve"] # "pull": 이미지 다운로드 허용 , "resolve": 태그 → 다이제스트 해석
skip_verify = true # HTTPS 인증서 검증 스킵 (HTTP라서 사실상 의미 없는지 테스트 해보자)
EOF
systemctl restart containerd
# 이미지 가져오기 실행 후 확인 : k8s-nodes는 현재 외부 통신 불능 상태인데, 아래 처럼 docker.io 미러 설정되어서 가져오는 것을 확인!
nerdctl pull docker.io/library/nginx:alpine
crictl images | grep nginx
192.168.10.10:35000/library/nginx alpine aea88c29b151e 25.7MB
docker.io/library/nginx alpine aea88c29b151e 25.7MB
# 현재 파드 상태
kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-54fc99c8d-8jpqt 1/1 Running 0 12m
> kubespary 에 containerd_registries_mirrors values 설정 후 적용 --tags containerd
더보기
[admin]
# containerd registry 정보 확인
cat /root/kubespray-offline/outputs/kubespray-2.30.0/inventory/mycluster/group_vars/all/offline.yml | head -n 15
...
http_server: "http://192.168.10.10"
registry_host: "192.168.10.10:35000"
# Insecure registries for containerd
containerd_registries_mirrors:
- prefix: "{{ registry_host }}"
mirrors:
- host: "http://{{ registry_host }}"
capabilities: ["pull", "resolve"]
skip_verify: true
# 수정
nano inventory/mycluster/group_vars/all/offline.yml
-------------------------------------------------
# Insecure registries for containerd
containerd_registries_mirrors:
- prefix: "{{ registry_host }}"
mirrors:
- host: "http://{{ registry_host }}"
capabilities: ["pull", "resolve"]
skip_verify: true
- prefix: "docker.io"
mirrors:
- host: "http://192.168.10.10:35000"
capabilities: ["pull", "resolve"]
skip_verify: false
- prefix: "registry-1.docker.io"
mirrors:
- host: "http://192.168.10.10:35000"
capabilities: ["pull", "resolve"]
skip_verify: false
- prefix: "quay.io"
mirrors:
- host: "http://192.168.10.10:35000"
capabilities: ["pull", "resolve"]
skip_verify: false
-------------------------------------------------
cat inventory/mycluster/group_vars/all/offline.yml | head -n 30
# 설정 업데이트
ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml -e kube_version="1.34.3" --tags containerd
# 확인
ssh k8s-node2 tree /etc/containerd
/etc/containerd
├── certs.d
│ ├── 192.168.10.10:35000
│ │ └── hosts.toml
│ ├── docker.io
│ │ └── hosts.toml
│ ├── quay.io
│ │ └── hosts.toml
│ └── registry-1.docker.io
│ └── hosts.toml
├── config.toml
└── cri-base.json
ssh k8s-node2 cat /etc/containerd/certs.d/quay.io/hosts.toml
server = "https://quay.io"
[host."http://192.168.10.10:35000"]
capabilities = ["pull","resolve"]
skip_verify = false
override_path = false
* 이미지 사용하는 디플로이먼트 배포 테스트
더보기
# quay.io nginx 이미지 가져오기
podman pull quay.io/nginx/nginx-unprivileged
podman images | grep quay
podman tag quay.io/nginx/nginx-unprivileged 192.168.10.10:35000/nginx/nginx-unprivileged
podman images | grep nginx
# 프라이빗 레지스트리에 업로드
podman push 192.168.10.10:35000/nginx/nginx-unprivileged
# 업로드된 이미지와 태그 조회
curl -s 192.168.10.10:35000/v2/_catalog | jq
curl -s 192.168.10.10:35000/v2/nginx/nginx-unprivileged/tags/list | jq
# quay.io nginx 디플로이먼트 배포 시도
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-quay
labels:
app: nginx-quay
spec:
replicas: 1
selector:
matchLabels:
app: nginx-quay
template:
metadata:
labels:
app: nginx-quay
spec:
containers:
- name: nginx-quay
image: quay.io/nginx/nginx-unprivileged
ports:
- containerPort: 80
EOF
# 확인
kubectl get pod -l app=nginx-quay -owide
kubectl get deploy nginx-quay -owide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
nginx-quay 1/1 1 1 66s nginx-quay quay.io/nginx/nginx-unprivileged app=nginx-quay
ssh k8s-node2 crictl images | grep quay
quay.io/nginx/nginx-unprivileged latest 1805d678a9a4a 60.9MB
# 삭제
kubectl delete deploy nginx-quay
'Kubernetes' 카테고리의 다른 글
| [7주차] RKE2 & Cluster API (0) | 2026.02.22 |
|---|---|
| [5주차] Kubespary HA & Upgrade (0) | 2026.02.06 |
| [4주차] Kubespary 배포 분석 (0) | 2026.02.06 |
| [3주차] Kubeadm & K8S Upgrade (0) | 2026.01.20 |
| [2주차] Ansible 기초 (0) | 2026.01.12 |