NVIDIA tops new AI inference benchmarks

NVIDIA has posted the fastest results on new benchmarks measuring the performance of artificial intelligence (AI) inference workloads in data centres and at the edge. The results of the industry’s first independent suite of AI benchmarks for inference, called MLPerf Inference 0.5, build on the company’s equally strong position in recent benchmarks measuring AI training.

Source: NVIDIA. The EGX platform is a supercomputer capable of running AI inference workloads.

The new benchmarks demonstrate the performance of NVIDIA Turing GPUs for data centres and NVIDIA Jetson Xavier system-on-a-chip (SoC) for edge computing. MLPerf’s five inference benchmarks — applied across a range of form factors and four inferencing scenarios — cover established AI applications such as image classification, object detection and translation. NVIDIA topped all five benchmarks for both data centre scenarios (server and offline), with Turing GPUs providing the highest performance per processor among commercially available processors. Jetson Xavier provided the highest performance among commercially-available edge and mobile SoCs under both edge-focused scenarios (single stream and multistream).

“AI is at a tipping point as it moves swiftly from research to large-scale deployment for real applications,” said Ian Buck, GM and VP of accelerated computing at NVIDIA. “AI inference is a tremendous computational challenge. Combining the industry’s most advanced programmable accelerator, the CUDA-X suite of AI algorithms and our deep expertise in AI computing, NVIDIA can help data centres deploy their large and growing body of complex AI models.”

NVIDIA was the only company to submit results across all five MLPerf benchmarks. In July, NVIDIA won multiple MLPerf 0.6 benchmark results for AI training, setting eight records in training performance.

All of NVIDIA’s MLPerf results were achieved using NVIDIA TensorRT 6 high performance deep learning-inference software that optimises and deploys AI applications easily in production from the data centre to the edge. New TensorRT optimisations are also available as open source in the Github repository.

NVIDIA GPUs accelerate large-scale inference workloads in the world’s largest cloud infrastructures, including Alibaba Cloud, AWS, Google Cloud Platform, Microsoft Azure and Tencent. Major businesses and organisations, including Procter & Gamble, are using NVIDIA’s EGX edge computing platform and AI inference capabilities to run sophisticated AI workloads at the edge.

NVIDIA also introduced Jetson Xavier NX at the same time (November 7). The world’s smallest, most powerful AI supercomputer for robotic and embedded computing devices at the edge, Jetson Xavier NX is a low-power version of the Xavier SoC used in the MLPerf Inference 0.5 benchmarks.