Last update: 16 November 2023

If you are doing deep learning AI research and/or development with GPUs, big chance you will be using graphics card from NVIDIA to perform the deep learning tasks. A vantage point with GPU computing is related with the fact that the graphics card occupies the PCI / PCIe slot. From the frugality point of view, it may be a brilliant idea to scavenge unused graphics cards from the fading PC world and line them up on another unused desktop motherboard to create a somewhat powerful compute node for AI tasks. Maybe not.

With the increasing popularity of container-based deployment, a system architect may consider creating several containers with each running different AI tasks. This means that that the underlying GPU resources should then be shared among the containers. NVIDIA provides a utility called NVIDIA Docker or nvidia-docker2 that enables the containerization of a GPU-accelerated machine. As the name suggests, the utility targets Docker container type.

This utility, however, cannot be immediately usable for all NVIDIA graphics card models. Only graphics card having GPUs with architecture newer than Fermi can benefit from this feature. In 2023, this means that NVIDIA GPUs with the following architecture support containerization of GPU resources:

• Kepler architecture
• Maxwell architecture
• Pascal architecture
• Volta architecture
• Turing architecture
• Ampere architecture
• Ada Lovelace architecture

CUDA-enabled vs Deep-Learning Ready GPUs

It is important to note that not all CUDA-enabled GPUs can perform deep learning tasks. NVIDIA introduced a terminology called CUDA compute capability that refers to the general specifications and available features of a CUDA-enabled GPU. The GPUs built with Fermi architecture has a maximum compute capability of 2.1, while the Kepler architecture has a minimum compute capability of 3.0.

The deep learning frameworks that rely on CUDA for GPU computing operate by invoking CUDA-specific GPU-accelerated deep learning methods to speed up the computation. These methods are provided in the cuDNN library. The library itself is compatible with CUDA-enabled GPUs with compute capability at least 3.0. It is suffice to say that only GPUs with Kepler architecture or newer are capable of accomplishing the deep learning tasks, or in other words, deep-learning ready.

Desktop GPUs vs Datacenter GPUs

When purchasing a GPU, especially NVDIA GPU, you may need to make sure the intended use of the GPU. Each GPU model has specific environment where it is supposed to be running. Desktop GPUs shall only be installed on PCs. Installing NVIDIA desktop GPUs (i.e. GeForce GPUs) on rack servers in a data center is not permitted and may render the GPU warranty void.

Datacenter GPUs come with higher specs and hence higher pricing point compared to a desktop GPU. It is also natural that datacenter GPUs produce higher performance and less bound to feature constraints. As an example, CUDA forward compatibility and virtual GPUs are only supported in datacenter GPUs.

If you are exploring deep learning and plans to experiment on popular algorithms such as CNN, RNN, or LSTM, desktop GPUs should suffice. Alternatively, if you are working on state-of-the-art computer vision model, transformers like BERT or GPT, or style transfer, datacenter GPU might be a better option especially from performance point of view.

Desktop GPUs for Deep Learning

The table below summarizes the list of NVIDIA desktop GPU models that serve as a better fit for building a deep learning AI system, especially through containerization support of AI tasks. The list is sourced from the Wikipedia entry of NVIDIA GPUs, and NVIDIA product pages and datasheets.

What about non-desktop GPUs that are also not datacenter GPUs such as those on workstation or laptop? Can we also use them for deep learning? NVIDIA provides the list for all products with CUDA support on this page: https://developer.nvidia.com/cuda-gpus.

GPU and CUDA Version

Each version of CUDA toolkit has a minimum compute capability that it supports. If you are using older GPU, you may be unable to run the latest version of CUDA. After obtaining the compute capability of your chosen NVIDIA GPU, you check against CUDA compatibility in this article.