LMDrive
Dataset
The LMDrive dataset consists of 64,000 instruction-sensor-control data segments collected in the CARLA simulator, each containing navigation instructions, multimodal multi-view sensor data, and control signals, lasting from 2 to 20 seconds, suitable for language-driven autonomous driving research.
Dataset Highlights
A large-scale multimodal benchmark dataset for language-driven autonomous driving
Language Navigation Instructions
Each data segment is accompanied by natural language navigation instructions, such as "turn right at the next intersection," supporting research on the alignment of language and driving behavior.
Multi-Angle Camera Data
Includes front, left, right, and other multi-angle RGB camera image sequences, providing rich visual perception information for scene understanding.
LiDAR Point Cloud
Each frame is accompanied by high-precision LiDAR 3D point cloud data, which can be used for obstacle detection, depth estimation, and 3D spatial perception tasks.
Control Signals
Records precise control signals such as steering angle, throttle, and brake, supporting the training and evaluation of end-to-end driving models.
CARLA Simulator Collection
Data is collected in the high-fidelity CARLA simulator, covering various weather, lighting, and traffic scenarios, safely reproducing various driving situations.
Diverse Driving Scenarios
Covers various road types and traffic scenarios, including urban roads, intersections, roundabouts, and highways, enhancing model generalization capabilities.
Applicable Scenarios
From academic research to industrial prototype validation, covering the core directions of autonomous driving
Language-Guided Driving
Training driving models that can understand natural language navigation instructions, achieving human-machine language interactive autonomous driving
End-to-End Autonomous Driving
End-to-end learning directly from sensor input to control output, exploring driving strategies without intermediate representations
Multimodal Fusion
Integrating various modal data such as camera images, LiDAR point clouds, and language instructions to enhance perception and decision-making capabilities
Instruction Following
Evaluating the model's understanding and execution capabilities of human navigation instructions, promoting the development of interactive autonomous driving systems
Data Preview
The following is a typical structure of a single data segment, including instructions, sensor paths, and control signals
{
"instruction": "Turn right at the next intersection.",
"segment_id": "route_00042_seg_0015",
"duration": 8.2,
"sensors": {
"camera_front": "data/route_00042/seg_0015/rgb_front/*.png",
"camera_left": "data/route_00042/seg_0015/rgb_left/*.png",
"camera_right": "data/route_00042/seg_0015/rgb_right/*.png",
"lidar": "data/route_00042/seg_0015/lidar/*.npy"
},
"control": [
{ "frame": 0, "steering": 0.00, "throttle": 0.60, "brake": 0.00 },
{ "frame": 10, "steering": 0.35, "throttle": 0.45, "brake": 0.00 },
{ "frame": 20, "steering": 0.72, "throttle": 0.30, "brake": 0.00 },
{ "frame": 30, "steering": 0.50, "throttle": 0.55, "brake": 0.00 },
{ "frame": 40, "steering": 0.05, "throttle": 0.65, "brake": 0.00 }
],
"weather": "ClearNoon",
"town": "Town03"
}
3 Steps to Get Started Quickly
From browsing to loading, you can start your autonomous driving research in just a few minutes
Browse the Dataset
View the details of the LMDrive dataset on the Ace Data Cloud platform, and learn about metadata such as data structure, scene distribution, and licensing agreements.
Download Data
Download 64,000 instruction-sensor-control data segments, including multi-view camera images, LiDAR point clouds, and control signals.
Load and Train
Use the official LMDrive toolkit to load the data and start training and evaluating the language-driven autonomous driving model.
Start Exploring the LMDrive Dataset
A large-scale language-driven autonomous driving dataset with open licensing, available for immediate download. Whether you are an autonomous driving researcher or a multimodal AI developer, this dataset is worth trying.