Skip to content

Commit eb400b8

Browse files
hami-robot[bot]hami-robot[bot]
authored
Merge pull request #106 from Project-HAMi/kunlun_support
Add kunlunxin vxpu support
2 parents 2595742 + d5b7f27 commit eb400b8

File tree

12 files changed

+606
-15
lines changed

12 files changed

+606
-15
lines changed

‎OWNERS‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
reviewers:
2+
- archlitchi
3+
- wawa0210
4+
- windsonsea
15
approvers:
26
- archlitchi
37
- wawa0210
8+
- windsonsea

‎docs/userguide/Device-supported.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,5 @@ The table below lists the devices supported by HAMi:
1414
| GPU | Mthreads | MTT S4000 | ✅ | ✅ | ❌ |
1515
| GPU | Metax | MXC500 | ✅ | ✅ | ❌ |
1616
| GCU | Enflame | S60 | ✅ | ✅ | ❌ |
17+
| XPU | Kunlunxin | P800 | ✅ | ✅ | ❌ |
1718
| DPU | Teco | Checking | In progress | In progress | ❌ |

‎docs/userguide/Kunlunxin-device/enable-kunlunxin-schedule.md‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -55,10 +55,4 @@ spec:
5555
resources:
5656
limits:
5757
kunlunxin.com/xpu: 4 # requesting 4 XPUs
58-
```
59-
60-
:::note
61-
62-
You can find more examples in examples folder soon.
63-
64-
:::
58+
```

‎docs/userguide/Kunlunxin-device/enable-kunlunxin-vxpu.md‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
---
2+
title: Enable Kunlunxin VXPU
3+
---
4+
5+
## Introduction
6+
7+
This component supports multiplexing Kunlunxin XPU devices (P800-OAM) and provides the following vGPU-like multiplexing capabilities, Special thanks for rise-union and kunlunxin for contributing:
8+
9+
***XPU Sharing***: Each task can occupy only a portion of the device, allowing multiple tasks to share a single XPU
10+
11+
***Memory Allocation Limits***: You can now allocate XPUs using memory values (e.g., 24576M), and the component ensures that tasks do not exceed the allocated memory limit
12+
13+
***Device UUID Selection***: You can specify to use or exclude specific XPU devices through annotations
14+
15+
16+
## Prerequisites
17+
* driver version >= 5.0.21.16
18+
* xpu-container-toolkit >= xpu_container_1.0.2-1
19+
* XPU device type: P800-OAM
20+
21+
## Enable XPU-sharing Support
22+
23+
* Deploy [vxpu-device-plugin]
24+
```yaml
25+
apiVersion: rbac.authorization.k8s.io/v1
26+
kind: ClusterRole
27+
metadata:
28+
name: vxpu-device-plugin
29+
rules:
30+
- apiGroups: [""]
31+
resources: ["pods"]
32+
verbs: ["get", "list", "update", "watch", "patch"]
33+
- apiGroups: [""]
34+
resources: ["nodes"]
35+
verbs: ["get", "list", "watch", "update", "patch"]
36+
---
37+
apiVersion: rbac.authorization.k8s.io/v1
38+
kind: ClusterRoleBinding
39+
metadata:
40+
name: vxpu-device-plugin
41+
subjects:
42+
- kind: ServiceAccount
43+
name: vxpu-device-plugin
44+
namespace: kube-system
45+
roleRef:
46+
kind: ClusterRole
47+
name: vxpu-device-plugin
48+
apiGroup: rbac.authorization.k8s.io
49+
---
50+
apiVersion: v1
51+
kind: ServiceAccount
52+
metadata:
53+
name: vxpu-device-plugin
54+
namespace: kube-system
55+
labels:
56+
app.kubernetes.io/component: vxpu-device-plugin
57+
---
58+
apiVersion: apps/v1
59+
kind: DaemonSet
60+
metadata:
61+
name: vxpu-device-plugin
62+
namespace: kube-system
63+
labels:
64+
app.kubernetes.io/component: vxpu-device-plugin
65+
spec:
66+
selector:
67+
matchLabels:
68+
app.kubernetes.io/component: vxpu-device-plugin
69+
template:
70+
metadata:
71+
labels:
72+
app.kubernetes.io/component: vxpu-device-plugin
73+
hami.io/webhook: ignore
74+
spec:
75+
priorityClassName: "system-node-critical"
76+
serviceAccountName: vxpu-device-plugin
77+
containers:
78+
- image: projecthami/vxpu-device-plugin:v1.0.0
79+
name: device-plugin
80+
resources:
81+
requests:
82+
memory: 500Mi
83+
cpu: 500m
84+
limits:
85+
memory: 500Mi
86+
cpu: 500m
87+
args:
88+
- xpu-device-plugin
89+
- --memory-unit=MiB
90+
- --resource-name=kunlunxin.com/vxpu
91+
- -logtostderr
92+
securityContext:
93+
privileged: true
94+
capabilities:
95+
add: [ "ALL" ]
96+
volumeMounts:
97+
- name: device-plugin
98+
mountPath: /var/lib/kubelet/device-plugins
99+
- name: xre
100+
mountPath: /usr/local/xpu
101+
- name: dev
102+
mountPath: /dev
103+
env:
104+
- name: NODE_NAME
105+
valueFrom:
106+
fieldRef:
107+
fieldPath: spec.nodeName
108+
- name: KUBECONFIG
109+
value: /etc/kubernetes/kubelet.conf
110+
volumes:
111+
- name: device-plugin
112+
hostPath:
113+
path: /var/lib/kubelet/device-plugins
114+
- name: xre
115+
hostPath:
116+
path: /usr/local/xpu
117+
- name: dev
118+
hostPath:
119+
path: /dev
120+
nodeSelector:
121+
xpu: "on"
122+
```
123+
124+
125+
:::note
126+
Default resource names are as follows:
127+
128+
- `kunlunxin.com/vxpu` for VXPU count
129+
- `kunlunxin.com/vxpu-memory` for memory allocation
130+
131+
You can customize these names using the parameters above.
132+
:::
133+
134+
## Device Granularity Partitioning
135+
136+
XPU P800-OAM supports 2 levels of partitioning granularity: 1/4 card and 1/2 card, with memory allocation automatically aligned. The rules are as follows:
137+
> - Requested memory ≤ 24576M (24G) will be automatically aligned to 24576M (24G)
138+
> - Requested memory > 24576M (24G) and ≤ 49152M (48G) will be automatically aligned to 49152M (48G)
139+
> - Requested memory > 49152M (48G) will be allocated as full cards
140+
141+
## Running XPU Tasks
142+
143+
```yaml
144+
apiVersion: v1
145+
kind: Pod
146+
metadata:
147+
name: vxpu-pod-demo
148+
spec:
149+
containers:
150+
- name: vxpu-pod-demo
151+
image: pytorch:resnet50
152+
workingDir: /root
153+
command: ["sleep","infinity"]
154+
resources:
155+
limits:
156+
kunlunxin.com/vxpu: 1 # requesting a VXPU
157+
kunlunxin.com/vxpu-memory: 24576 # requesting a virtual XPU that requires 24576 MiB of device memorymemory
158+
```
159+
160+
## Device UUID Selection
161+
162+
You can specify to use or exclude specific XPU devices through Pod annotations:
163+
164+
```yaml
165+
apiVersion: v1
166+
kind: Pod
167+
metadata:
168+
name: poddemo
169+
annotations:
170+
# Use specific XPU devices (comma-separated list)
171+
hami.io/use-xpu-uuid: ""
172+
# Or exclude specific XPU devices (comma-separated list)
173+
hami.io/no-use-xpu-uuid: ""
174+
spec:
175+
# ... rest of Pod configuration
176+
```
177+
178+
> **Note:** Device ID format is `{BusID}`. You can find available device IDs in the node status.
179+
180+
### Finding Device UUIDs
181+
182+
You can use the following commands to find Kunlunxin P800-OAM XPU device UUIDs on nodes:
183+
184+
```bash
185+
kubectl get pod <pod-name> -o yaml | grep -A 10 "hami.io/xpu-devices-allocated"
186+
```
187+
188+
Or by checking node annotations:
189+
190+
```bash
191+
kubectl get node <node-name> -o yaml | grep -A 10 "hami.io/node-register-xpu"
192+
```
193+
194+
Look for annotations containing device information in the node annotations.
195+
196+
197+
## Important Notes
198+
199+
The current Kunlun chip driver supports a maximum of 32 handles. Eight XPU devices occupy 8 handles, so it is not possible to split all 8 devices into 4 each.
200+
```yaml
201+
# valid
202+
kunlunxin.com/vxpu: 8
203+
204+
# valid
205+
kunlunxin.com/vxpu: 6
206+
kunlunxin.com/vxpu-memory: 24576
207+
208+
# valid
209+
kunlunxin.com/vxpu: 8
210+
kunlunxin.com/vxpu-memory: 49152
211+
212+
# invalid
213+
kunlunxin.com/vxpu: 8 # not support
214+
kunlunxin.com/vxpu-memory: 24576
215+
```

‎docs/userguide/Kunlunxin-device/examples/allocate_vxpu.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
---
2+
title: Allocate vxpu device
3+
---
4+
5+
## Allocate vxpu device
6+
7+
To allocate a certain part of device core resource, you need only to assign the `kunlunxin.com/vxpu` along with the `kunlunxin.com/vxpu-memory`
8+
9+
```yaml
10+
apiVersion: v1
11+
kind: Pod
12+
metadata:
13+
name: xpu-pod
14+
spec:
15+
containers:
16+
- name: ubuntu-container
17+
image: ubuntu:22.04
18+
imagePullPolicy: IfNotPresent
19+
command: ["sleep","infinity"]
20+
resources:
21+
limits:
22+
kunlunxin.com/vxpu: 1 # requesting 1 XPU
23+
kunlunxin.com/vxpu-memory: 24576 # each XPU require 24576 MiB device memory
24+
```

‎docs/userguide/Kunlunxin-device/examples/allocate_whole_xpu.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
title: Allocate a whole xpu card
3+
---
4+
5+
## Allocate exclusive device
6+
7+
To allocate a whole xpu device, you need to only assign `kunlunxin.com/xpu` without other fields. You can allocate multiple XPUs for a container.
8+
9+
```yaml
10+
apiVersion: v1
11+
kind: Pod
12+
metadata:
13+
name: xpu-pod
14+
spec:
15+
containers:
16+
- name: ubuntu-container
17+
image: ubuntu:22.04
18+
imagePullPolicy: IfNotPresent
19+
command: ["sleep","infinity"]
20+
resources:
21+
limits:
22+
kunlunxin.com/xpu: 1 # requesting 1 XPU
23+
```

‎i18n/zh/docusaurus-plugin-content-docs/current/userguide/Device-supported.md‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,13 @@ HAMi支持的设备视图如下表所示：
77

88
| 生产商 | 制造商 | 类型 | 内存隔离 | 核心隔离 | 多卡支持 |
99
|-------------|------------|-------------|-----------|---------------|-------------------|
10-
| GPU | NVIDIA | 全部 | ✅ | ✅ | ✅ |
11-
| MLU | Cambricon | 370, 590 | ✅ | ✅ | ❌ |
12-
| DCU | Hygon | Z100, Z100L | ✅ | ✅ | ❌ |
13-
| Ascend | Huawei | 910B, 910B3, 310P | ✅ | ✅ | ❌ |
14-
| GPU | iluvatar | 全部 | ✅ | ✅ | ❌ |
15-
| GPU | Mthreads | MTT S4000 | ✅ | ✅ | ❌ |
16-
| DPU | Teco | 检查中 | 进行中 | 进行中 | ❌ |
10+
| GPU | NVIDIA | 全部 | ✅ | ✅ | ✅ |
11+
| MLU | Cambricon | 370, 590 | ✅ | ✅ | ❌ |
12+
| DCU | Hygon | Z100, Z100L | ✅ | ✅ | ❌ |
13+
| Ascend | Huawei | 910B, 910B3, 310P | ✅ | ✅ | ❌ |
14+
| GPU | iluvatar | 全部 | ✅ | ✅ | ❌ |
15+
| GPU | Mthreads | MTT S4000 | ✅ | ✅ | ❌ |
16+
| GPU | Metax | MXC500 | ✅ | ✅ | ❌ |
17+
| GCU | Enflame | S60 | ✅ | ✅ | ❌ |
18+
| XPU | Kunlunxin | P800 | ✅ | ✅ | ❌ |
19+
| DPU | Teco | 检查中 | 进行中 | 进行中 | ❌ |

‎i18n/zh/docusaurus-plugin-content-docs/current/userguide/Kunlunxin-device/enable-kunlunxin-schedule.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
---
2+
title: 启用昆仑芯 GPU 拓扑感知调度
3+
---
4+
5+
**昆仑芯 GPU 拓扑感知调度现在通过 `kunlunxin.com/xpu` 资源得到支持。**
6+
7+
当在单个P800服务器上配置多个XPU时，当XPU卡连接到同一NUMA节点或互相之间可以直接连接时，性能会显著提升。从而在服务器上的所有 XPU 之间形成拓扑，如下所示：
8+
9+
![img](../../resources/kunlunxin_topo.jpg)
10+
11+
当用户作业请求一定数量的 `kunlunxin.com/xpu` 资源时，
12+
Kubernetes 将 Pod 调度到适当的节点上，目标是减少碎片化
13+
并最大化性能。然后 `xpu-device` 在选定的节点上执行细粒度分配
14+
请求的资源，遵循以下规则：
15+
16+
1. 只允许 1、2、4 或 8 卡分配。
17+
2. 1、2 或 4 个 XPU 的分配不能跨越 NUMA 节点。
18+
3. 分配后应最小化碎片化。
19+
20+
## 重要说明
21+
22+
1. 这种模式**不支持**设备共享。
23+
2. 这些功能已在昆仑芯 P800 硬件上进行了测试。
24+
25+
## 前置条件
26+
27+
* Kunlunxin driver >= v5.0.21
28+
* Kubernetes >= v1.23
29+
* kunlunxin k8s-device-plugin
30+
31+
## 启用拓扑感知调度
32+
33+
- 在 P800 节点上部署昆仑芯设备插件。
34+
（请联系您的设备供应商获取相应的软件包和文档。）
35+
- 按照 `README.md` 中的说明部署 HAMi。
36+
37+
## 运行昆仑芯作业
38+
39+
昆仑芯 P800 GPU 可以通过容器使用 `kunlunxin.com/xpu` 资源类型来请求。
40+
以下是 Pod 规范示例：
41+
42+
```yaml
43+
apiVersion: v1
44+
kind: Pod
45+
metadata:
46+
name: gpu-pod1
47+
spec:
48+
containers:
49+
- name: ubuntu-container
50+
image: docker.io/library/ubuntu:latest
51+
imagePullPolicy: IfNotPresent
52+
command: ["sleep", "infinity"]
53+
resources:
54+
limits:
55+
kunlunxin.com/xpu: 4 # 请求 4 个 XPU
56+
```

0 commit comments

Comments
 (0)