MIAM: Multimodal Industrial Activity Monitoring

Overview

The MIAM Dataset is a comprehensive multimodal dataset designed to facilitate research in human-robot collaboration, activity recognition, and engagement prediction within industrial environments. Captured during realistic assembly and disassembly workflows, the dataset offers a rich resource for advancing industrial automation, human behavior modeling, and collaborative robotics.

Request Access

Features

• Multimodal Data: Includes RGB, depth, and 9-axis IMU sensor data from both hands.
• Annotations: Provides action labels, engagement states, and timestamps for over 22 sessions.
• Realistic Scenarios: Simulates real-world tasks with natural operator variability.
• Gender-Diverse Subjects: Includes data from 8 participants across different demographics.
• Untrimmed Videos: Covers 220 minutes of industrial activity.

Meta Tasks

Our dataset supports a wide range of meta tasks designed to optimize performance and safety in industrial settings.

• Action Localization: This task identifies the temporal boundaries [t_start, t_end] of an operator's actions. Accurate action localization is crucial for precise monitoring and optimizing workflow sequences.
  • Total distinct assembly actions cataloged: 39
  • Total distinct disassembly actions cataloged: 30
• Active Object Localization: Monitors interactions between operators and tools throughout their tasks. This data is essential for detailed assessments of tool usage and operational efficiency.
• Engagement Prediction: Analyzes operator engagement levels by measuring metrics such as gaze tracking and manual interactions. These insights are vital for identifying opportunities for ergonomic improvements and increasing operational efficiency.

This dataset is meticulously designed to facilitate comprehensive analyses of action patterns, tool usage, and operator engagement, proving to be an invaluable resource for optimizing industrial and manufacturing processes.

Dataset Structure

Our dataset is organized to meet the diverse needs of industrial automation and human-robot interaction. It includes multi-view and multi-modal data, complete with preprocessing scripts available on GitHub to facilitate analysis.

• RGB: Comprehensive multi-view RGB video files from two distinct devices, C1 and C2, designed to capture a wide range of operational contexts.
• Depth: Depth frames from the C2 device, synchronized with RGB data to provide detailed 3D spatial analysis capabilities.
• IMU: Inertial Measurement Unit (IMU) data, crucial for capturing precise motion dynamics and orientations in complex industrial settings.
• Annotations: Extensively labeled for temporal actions, engagement states, and bounding boxes for active objects.

Citation

If you use the MIAM Dataset in your research, please cite our paper:

Naval Kishore Mehta, Arvind, Himanshu Kumar, Abeer Banerjee, Sumeet Saurav, Sanjay Singh. (2025). A Multimodal Dataset for Enhancing Industrial Task Monitoring and Engagement Prediction. arXiv preprint arXiv:2501.05936. Available at https://arxiv.org/abs/2501.05936.