NVIDIA has released an open-source, high-quality dataset that can be used to teach autonomous robots and vehicles how to interact with the physical world. The commercial-grade, prevalidated dataset can help researchers and developers kickstart physical AI projects that can be prohibitively difficult to start from scratch, NVIDIA said.
The company explained that running a fleet of vehicles over months to gather data for autonomous vehicle AI is impractical and costly — and, since much of the footage collected is uneventful, typically just 10% of data is used for training. Yet this scale of data collection is essential to building safe, accurate, commercial-grade models. The NVIDIA DRIVE AV end-to-end AI model for autonomous vehicles requires tens of thousands of hours of driving data to develop, while the NVIDIA Isaac GR00T robotics models take thousands of hours of video clips for post-training.
 |
| Source: NVIDIA blog post. SimReady data. |
The NVIDIA Physical AI Dataset can help developers scale AI performance during pretraining, where more data helps build a more robust model — for testing and validation, and during post-training to improve performance for a specific use case. The dataset is slated to contain a subset of the real-world and synthetic data NVIDIA uses to train, test and validate physical AI for the NVIDIA Cosmos world model development platform, the NVIDIA DRIVE AV software stack, the NVIDIA Isaac AI robot development platform and the NVIDIA Metropolis application framework for smart cities.
Now available on Hugging Face, the dataset currently offers developers 15 TB of data comprising thousands of hours of multicamera video at unprecedented diversity, scale and geography. The data represents more than 320,000 trajectories for robotics training, plus up to 1,000 Universal Scene Description (OpenUSD) assets, including a SimReady (simulation-ready) collection. Dedicated data to support end-to-end autonomous vehicle (AV) development — which will include 20-second clips of diverse traffic scenarios in cities — is coming soon, NVIDIA said.
In addition to harnessing the NVIDIA Physical AI Dataset to help meet their data needs, developers can further boost AI development with tools like NVIDIA NeMo Curator, which processes vast datasets efficiently for model training and customisation. Using NeMo Curator, 20 million hours of video can be processed in two weeks on NVIDIA Blackwell GPUs, compared with 3.4 years on unoptimised CPU pipelines.
NVIDIA envisions the dataset to become the world’s largest unified and open dataset for physical AI development over time. NVIDIA said the dataset could enable new work on identifying outliers and assessing model generalisation performance, contributing to NVIDIA Halos’ full-stack AV safety system. Other use cases could include AI models that power robots that move safely in warehouse environments, humanoid robots that support surgeons during procedures and AVs that can navigate complex traffic scenarios like construction zones.
Robotics developers can further tap the new NVIDIA Isaac GR00T blueprint for synthetic manipulation motion generation, a reference workflow built on NVIDIA Omniverse and NVIDIA Cosmos that uses a small number of human demonstrations to create massive amounts of synthetic motion trajectories for robot manipulation.
Early adopters said the dataset can help to train predictive AI models that help autonomous vehicles better track the movements of vulnerable road users like pedestrians; develop semantic AI models that understand the context of spaces like homes, hotel rooms and hospitals; grasping which groceries need refrigeration for example, and create digital twins that simulate edge cases and challenging weather conditions to train and test autonomous driving models in rare real-world situations.
Details
Access the NVIDIA Physical AI dataset on Hugging Face at https://huggingface.co/collections/nvidia/physical-ai-67c643edbb024053dcbcd6d8
Hashtag: #GTC2025
No comments:
Post a Comment