在使用nvidia官方源更新apt时可能会出现Could not handshake: Error in the pull function.的网络问题，比如下面这种：

首先配置apt仓库

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

或

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

执行apt update 

root@ubuntu:/# sudo apt update
...
Ign:2 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease
Ign:4 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease
Ign:5 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64  InRelease
Err:2 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease                                                                                                                                               
  Could not handshake: Error in the pull function. [IP: 185.199.111.153 443]
Err:4 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease                                                                                                                                          
  Could not handshake: Error in the pull function. [IP: 185.199.111.153 443]
Err:5 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64  InRelease                                                                                                                                                            
  Could not handshake: Error in the pull function. [IP: 185.199.111.153 443]                                                                                                                                             
Fetched 3,037 kB in 16s (195 kB/s)                                                                                                                                                                                                   
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64/InRelease  Could not handshake: Error in the pull function. [IP: 185.199.111.153 443]
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64/InRelease  Could not handshake: Error in the pull function. [IP: 185.199.111.153 443]
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64/InRelease  Could not handshake: Error in the pull function. [IP: 185.199.111.153 443]
W: Some index files failed to download. They have been ignored, or old ones used instead.

主要原因就是国内环境的网络，同时又没有可以替代的国内源，解决办法就是直接安装deb

1. nvidia官方github

这里提供了各种版本的nvidia工具

https://github.com/NVIDIA/libnvidia-container/tree/gh-pages/stable/ubuntu18.04/amd64

2. 下载相关deb软件包

找到自己需要的deb并下载

注意，总共要下载6个deb软件包。根据要安装的nvidia-docker2版本下载兼容的其他deb软件包，不然安装时会提示失败版本不兼容。

这里以安装nvidia-docker2_2.13.0-1_all.deb为例，需要下载如下的deb：

1. libnvidia-container1_1.13.5-1_amd64.deb
2. libnvidia-container-tools_1.13.5-1_amd64.deb
3. nvidia-container-toolkit-base_1.13.5-1_amd64.deb
4. nvidia-container-toolkit_1.13.5-1_amd64.deb
5. nvidia-docker2_2.13.0-1_all.deb
6. nvidia-container-runtime_3.13.0-1_all.deb

3. 安装deb 

注意，安装要顺序执行，不然也会提示安装失败

dpkg -i libnvidia-container1_1.13.5-1_amd64.deb
...
dpkg -i libnvidia-container-tools_1.13.5-1_amd64.deb
...
dpkg -i nvidia-container-toolkit-base_1.13.5-1_amd64.deb
...
dpkg -i nvidia-container-toolkit_1.13.5-1_amd64.deb
...
dpkg -i nvidia-docker2_2.13.0-1_all.deb
...
dpkg -i nvidia-container-runtime_3.13.0-1_all.deb
...

4. 验证

最后使用docker验证是否安装成功

root@ubuntu:/# sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi
Wed Feb 21 08:46:43 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090         Off| 00000000:01:00.0 Off |                  N/A |
| 30%   43C    P0               97W / 350W|      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

成功！

如果验证时提示unknown or invalid runtime name: nvidia：

root@ubuntu:/# sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi
docker: Error response from daemon: unknown or invalid runtime name: nvidia.

是因为docker的daemon配置错了，修改/etc/docker/daemon.json（需要管理员权限）

原文件内容：

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

新文件内容：

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

保存后重启docker：

sudo systemctl restart docker

问题解决了！