Hao Liang's Blog

Embrace the World with Cloud Native and Open-source

Customize Kubernetes cpuset strategy in GPU container task

1. Brief description This article takes the NVIDIA L40s GPU device as an example to briefly describe its GPU topology and the current CPU core-binding capabilities of Kubernetes. To briefly review, the NVIDIA L40s GPU device topology is shown in the figure below: The CPU of this model has a total of 384 cores, distributed on 2 NUMA nodes. Each GPU has a mellanox high-speed RDMA network card and is hung under the same PCIe Bridge.

Posted by Hao Liang's Blog on Sunday, December 8, 2024

KubeCon AI_dev 中国香港 2024 回顾

1. KubeCon 简介 2024 年 8 月 21 日至 23 日，云原生计算基金会的旗舰会议联合开源峰会和 AI_dev 在香港汇聚来自领先开源和云原生社区的应用者和技术专家。 KubeCon 每年会在欧洲、

Posted by Hao Liang's Blog on Friday, August 30, 2024

Recap of KubeCon AI_dev HongKong 2024

1. Introduction to KubeCon The Cloud Native Computing Foundation’s flagship conference joins with Open Source Summit and AI_dev to gather adopters and technologists from leading open source and cloud native communities in Hong Kong from 21-23 August 2024. KubeCon is held every year in 3 places in Europe, America, and Asia. It has entered China since 2018. This is the third time I have attended in KubeCon. The first two times I attended as a speaker.

Posted by Hao Liang's Blog on Friday, August 30, 2024

Kubelet Streaming Server Port Closed Unexpectedly

1. Description Kernel version: 5.4.241 kubelet version: 1.22.5 nvidia driver version: 535.161.08 and 535.154.05 After the kubelet process on the node is started, it listens to a random port (46127) in the range of ip_local_port_range ss -lntpe |grep kubelet code snippets: After running for a while, the listen port suddenly disappeared The corresponding fd (fd=13) is also closed, but the kubelet process still exists 2. Analysis From the corresponding kubelet code snippets, we found that the streaming server is pulled up through a separate goroutine.

Posted by Hao Liang's Blog on Saturday, July 13, 2024

Kubelet Streaming Server 端口异常关闭

1. 问题描述内核版本：5.4.241 kubelet版本：1.22.5 nvidia驱动版本：535.161.08 和 535.154.05 节点上的 kubelet 进程启动后，监听了

Posted by Hao Liang's Blog on Saturday, July 13, 2024

Renaming Node Name without Resetting kubelet Environment

Goal Rename any node name in Kubernetes cluster. No need to reset the whole kubelet environment like most of the approaches. No need to drain any Pods running on the Node. Bootstrap Process of kubelet Doc refer to: https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/ Chinese version introduction refer to: https://cloud.tencent.com/developer/article/1656007 The kubelet process starts. Try to find kubeconfig file specified by arg --kubeconfig=xxx, if not found, try to find bootstrap-kubeconfig file specified by arg --bootstrap-kubeconfig=xxx instead.

Posted by Hao Liang's Blog on Friday, December 22, 2023

Something you might need to know when developing a CNI plugin

Introduction CNI, as in Container Networking Interface for kubernetes, dedicated to provide network solution for Kubernetes containers. There are tons of CNI plugin for kubernetes networking on the market, some of them are opensource projects.(e.g. flannel, calico, cilium) Besides, the CNI officially provides some sample cni demo for end users. How does kubelet interact with CNI Implemented by Dockershim In preview version of Kubernetes(less or equal 1.23), if the container runtime is specified to docker, CNI plugin will be called in dockershim#cni.

Posted by Hao Liang's Blog on Friday, December 15, 2023

【Troubleshooting】Reusable CPUs from initContainer were not being honored

1. Description In early version of kubernetes v1.18. Related Commit: Fix a bug whereby reusable CPUs and devices were not being honored #93289 Related PR: Fix a bug whereby reusable CPUs and devices were not being honored #93189 Refactor the algorithm used to decide CPU assignments in the CPUManager #102014 Previously, it was possible for reusable CPUs and reusable devices (i.e. those previously consumed by init containers) to not be reused by subsequent init containers or app containers if the TopologyManager was enabled.

Posted by Hao Liang's Blog on Monday, October 23, 2023

【Troubleshooting】Summary of kube-apiserver troubleshooting techniques (analysis of logs and caching principles)

1. Related background When troubleshooting apiserver issues, we found nodes that may have performance bottlenecks through monitoring. The next step is to further analyze the apiserver logs on the nodes. 2. APIServer log analysis skills Trace log Log printing conditions When the total request time exceeds the threshold (default 500ms), apiserver will print the trace log, and at each step of the trace, it will calculate a step time-consuming threshold

Posted by Hao Liang's Blog on Sunday, May 15, 2022

【Troubleshooting】Summary of kube-apiserver troubleshooting techniques (monitoring analysis)

1. Related background As the scale of a single K8s cluster continues to expand (the number of nodes reaches 4,000+), we found during the operation that the apiserver has gradually become the performance bottleneck of the cluster, prone to problems such as unresponsive requests, slow responses, and request rejections, and even causes cluster avalanches, causing Network failure found. The following details how to quickly locate and troubleshoot apiserver performance issues.

Posted by Hao Liang's Blog on Saturday, May 14, 2022