[Kubernetes] GPU NodePool 사용하기

Azure Kubernetes는 계산 집약적인 Kubernetes 워크로드를 실행하기 위해 GPU 지원 Linux 노드 풀을 지원합니다. 관련된 내용은 다음 링크를 참조하면 됩니다.

 

 

AKS GPU 이미지를 사용하도록 클러스터 업데이트

1. aks-preview 명령을 사용하여 Azure CLI 확장을 설치합니다.

az extension add --name aks-preview

 

2. 명령을 사용하여 최신 버전의 확장으로 업데이트 합니다.

az extension update --name aks-preview

 

3. az feature register 명령을 사용하여 GPUDedicatedVHDPreview 기능 플래그를 등록합니다.

az feature register --namespace "Microsoft.ContainerService" --name "GPUDedicatedVHDPreview"

 

4. az feature show 명령을 사용하여 등록 상태를 확인합니다.

az feature show --namespace "Microsoft.ContainerService" --name "GPUDedicatedVHDPreview"

 

5. 상태가 Registered(등록됨)로 변경되면 az provider register 명령을 사용하여 Microsoft.ContainerService 리소스 공급자의 등록을 새로 고침 합니다.

az provider register --namespace Microsoft.ContainerService

 

 

GPU 노드에 대한 노드 풀 추가

이 때 node vm size가 zone을 지원하는지 확인이 필요합니다.

az aks nodepool add \
--resource-group <Cluster가 배포된 ResourceGroupName> \
--cluster-name <ClusterName> \
--name gpunp \
--node-vm-size Standard_NC6s_v3 \
--node-taints sku=gpu:NoSchedule \
--aks-custom-headers UseGPUDedicatedVHD=true \
--node-count 1 \
--max-pods 30 \
--kubernetes-version 1.27.3 \
--os-sku Ubuntu \
--vnet-subnet-id /subscriptions/fd5817.../resourceGroups/<ResourceGroupName>/providers/Microsoft.Network/virtualNetworks/prd-vnet/subnets/prd-ap-sb \
--pod-subnet-id /subscriptions/fd5817.../resourceGroups/<ResourceGroupName>/providers/Microsoft.Network/virtualNetworks/prd-vnet/subnets/prd-ap-secondary-sb

 

 

NVIDIA 플러그인 설치

MS 공식 문서에 있는 Daemonset은 버전이 오래되어 다음 Github 경로에 있는 yaml 파일로 배포합니다.

# Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      - key: "sku"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"
      # Mark this pod as a critical add-on; when enabled, the critical add-on
      # scheduler reserves resources for critical add-on pods so that they can
      # be rescheduled after a failure.
      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
      priorityClassName: "system-node-critical"
      containers:
      - image: nvcr.io/nvidia/k8s-device-plugin:v0.14.1
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

 

 

GPU 사용량 확인을 위한 워크로드 실행

apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app: samples-tf-mnist-demo
  name: samples-tf-mnist-demo
spec:
  template:
    metadata:
      labels:
        app: samples-tf-mnist-demo
    spec:
      containers:
        - name: nvidia-plugin-test-ctr
          image: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v1.10.1
          imagePullPolicy: IfNotPresent
          command: ['sh', '-c']
          args:
            - "while true;  do vectorAdd; sleep 30; done"
          securityContext:
            allowPrivilegeEscalation: false
      restartPolicy: OnFailure
      tolerations:
      - key: "sku"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"