MLPerf releases results for top machine-learning training systems

The MLPerf consortium has released the results for MLPerfTraining v0.7, the third round of results from their machine learning training performance benchmark suite.

MLPerf is a consortium of over 70 companies and researchers from leading universities, and the MLPerf Benchmark suites are the industry standard for measuring machine-learning performance.

The consortium said that the MLPerf Benchmark shows substantial industry progress and growing diversity in terms of systems undergoing the benchmark, including multiple new processors, accelerators, and software frameworks. Compared to the prior submission round, the fastest results on the five unchanged benchmarks improved by an average of ​2.7x​, showing substantial improvement in hardware, software, and system scale.

This latest training round encompasses 138 results on a wide variety of systems from nine organisations. The Closed division results all used the same model/optimiser(s), while Open division results may use more varied approaches - results include commercially-available systems, upcoming preview systems, as well as research, development and internal (RDI) systems.

The MLPerf Training Benchmark suite measures the time it takes to train one of eight machine-learning models to a standard quality target in tasks including image classification,recommendation, translation, and playing Go.

This version of MLPerf includes two new benchmarks and one substantially revised benchmark as follows:

● Bi-directional Encoder Representation from Transformers (BERT) trained with Wikipedia is a leading-edge language model that is used extensively in natural language-processing tasks.

Given a text input, language models predict related words and are employed as a building block for translation, search, text understanding, answering questions, and generating text.

● Mini-Go

The reinforcement learning is similar to Mini-Go from v0.5 and v0.6, but uses a full-size 19x19 Go board, which is more reflective of research.

● Deep Learning Recommendation Model (DLRM) trained with Criteo AI Lab’s Terabyte Click-Through-Rate (CTR) dataset. The terabyte-sized click logs of Criteo AI Lab’s Terabyte CTR dataset is the largest open recommendation dataset, containing click logs of 4 billion user and item interactions over 24 days.

The DLRM Benchmark is representative of a wide variety of commercial applications. The dataset includes recommendations for online shopping, search results, and social media content rankings.

MLPerf said it is committed to providing benchmarks that reflect the needs of machine learning customers, and is pioneering customer advisory boards to steer future benchmark construction. DLRM is the first benchmark produced using this process.

“The DLRM-Terabyte recommendation benchmark is representative of industry use cases and captures important characteristics of model architectures and user-item interactions in recommendation data sets,” said Carole-Jean Wu, MLPerf Recommendation Benchmark Advisory Board Chair from Facebook AI.

Explore:

See the results​.

Read more about the Training v0.7 benchmarks.