Atomic actions
Describe the specific operation, manipulated objects, tools, and motion—not just the activity.
Every action.
Every hand.
Every moment.
Benchmarking Dense Action Captioning for Egocentric Human Videos
1 S-Lab, Nanyang Technological University 2 Shanghai Jiao Tong University
See what happens.
Then look closer.
What happens, when it happens,
and which hand performs it.
The video could not load. You can still explore its annotations using the timeline.
HAND
Select an action on the timeline to inspect its caption and time interval.
Click or drag a track to seek to any time. The frame buttons step through adjacent frames, numbered from 0. Use the action picker to jump to an action’s start, or 0.25× and 0.5× playback to inspect brief actions. On small screens, swipe the time ruler to scroll the timeline.
Beyond recognizing
the overall activity.
Understanding “cutting a pumpkin” is only the beginning. Models must recover the individual actions within continuous manipulation, locate them in time, and identify which hand performs them.
Frame-level boundaries
Action boundaries are annotated at the level of individual video frames.
Hand association
Distinguish independent left- and right-hand actions from coordinated, two-handed manipulation.
Read the benchmark overview
Learning from human data offers a scalable path toward embodied intelligence, yet it requires models to accurately understand fine-grained human actions from egocentric observations. However, existing benchmarks remain limited in action atomicity, temporal precision, and hand–action association. We introduce EgoActionBench, a benchmark that represents continuous human manipulation as temporally localized atomic actions associated with the left hand, right hand, or both hands. EgoActionBench contains 1,407 episodes with 41,132 human-refined action annotations. To evaluate such dense predictions, we propose a progressive evaluation protocol that retrieves temporally eligible candidates, selects a prediction by content agreement, and scores semantic fidelity and temporal overlap alongside action precision and recall, jointly capturing what happens, when it happens, and which hand performs it. We benchmark 19 vision-language models on EgoActionBench. Our experiments reveal substantial limitations of current models in action atomicity, temporal precision, and hand–action association, highlighting the gap between general video understanding and the action understanding required for embodied intelligence.
Match the action.
Then measure its precision.
We evaluate what happens, when it happens, and which hand performs it.
- 01
Temporal eligibility
Retrieve candidate predictions that satisfy the temporal eligibility criteria for a reference action.
- 02
Content matching
Select the prediction with the strongest content match for each reference action.
- 03
Temporal and semantic scoring
Measure temporal overlap, action coverage, and semantic fidelity.
EgoActionScore
The average of Temporal Overall and Semantic Overall. Higher is better.
Benchmark results.
EgoActionBench evaluates temporal localization and semantic fidelity through ten complementary metrics.
Results could not be loaded. Please reload the page.
19 baseline models, two EgoAction models, and a human reference.
| Model | EgoAction Score | Temporal | Semantic | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| m-tIoU | F1 | Overall | Hand | Verb | Objects | Tools | Motion | Overall | ||
Temporal Overall averages m-tIoU and F1. Semantic Overall averages the applicable field scores within each action, then across reference actions.
Performance across five action-duration ranges, with sub-second actions highlighted.
| Model | <0.5 s | [0.5, 1) s | [1, 2) s | [2, 5) s | ≥5 s |
|---|
Each cell averages the selected per-action metric within its ground-truth duration range.
Results reported in the accompanying paper.Download table data
Build on EgoActionBench.
Publication and release links will appear here when available. Full source videos, dataset files, and model weights are not included in this preview.
01Coming soon
Paper
Benchmark, evaluation protocol, and experiments.
02Coming soon
Code & evaluation
Tools to run the benchmark and reproduce the evaluation.
03Coming soon
Dataset
Annotations, access instructions, and usage conditions.
Citation
Preliminary project citation.
Publication details will be updated on release.
@misc{shen2026egoactionbench,
title = {EgoActionBench: Benchmarking Dense Action
Captioning for Egocentric Human Videos},
author = {Shen, Yujiao and Song, Jiahe and Li, Haitian
and Yang, Jingkang and Liu, Ziwei},
year = {2026},
url = {https://egoaction-bench.github.io}
}