GDM pilot sample

10-hour multi-camera egocentric pilot sample · four-camera rig · verified labels · merged 3D tracks.

2026.07.02 · External

600 one-minute episodes of multi-camera egocentric human demonstration. Every episode is captured on a synchronized four-camera rig — egocentric, exocentric, and both wrists — and ships with individually verified subtask labels and per-frame tracks: camera pose (SLAM), 3D world hand pose, and exocentric body pose, all merged on a single 29.97 fps timeline.

At a glance

Headline figures. Every label is individually verified against its footage; unverifiable labels were removed rather than shipped.

10.0

Hours

600 episodes × 60 s · four camera streams each

9,384

Verified labels

100% verified against footage · 0 contradictions in a 1,800-span audit

1.08M

Frames

29.97 fps · 960 × 720 · per-frame tracks merged

61 GB

Total size

H.264 video + parquet tracks + metadata

Distribution matches the GDM pilot category targets within tolerance across all categories — full breakdown in the Sample diversity section below.

Sample diversity

Delivered distribution against the GDM pilot category targets. Category shares are of the large-physical-change portion; the small-physical row is a share of the whole sample.

**Table.** Distribution by category vs. pilot targets (all within ±5% tolerance).
Category	Share	GDM target ±5%
Others (office / conference / outdoor)	22.4%	20%
Home	19.9%	15%
Retail (retail floor, café)	17.6%	15%
Hospitality (hotel lobby, reception, restaurant kitchen)	17.2%	15%
Logistics (packing, storage)	12.4%	15%
Industrial (recycling, workstations)	10.5%	10%
Small-physical-change tasks (of total)	6.2%	10%

600 unique source episodes · 175 task cards · 141 environment×site scenes at 10 venues · ≤3.6 h per scene · ≤10 h per person.

Camera coverage

Four synchronized camera streams on every episode — head, third-person, and both wrists.

Every episode is captured on the full four-camera rig — egocentric (E), exocentric (X), left wrist (L), and right wrist (R) — for complete multi-view coverage on every recording.

E — Egocentric: Head-mounted

X — Exocentric: Fixed third-person

L — Left wrist: Wrist-mounted

R — Right wrist: Wrist-mounted

Technical specifications

Video, labels, and all tracks share one 29.97 fps timeline; tracks are pre-merged into the per-frame records — no joins required.

Dataset format: LeRobot v3.0

Resolution: 960 × 720

Frame rate: 29.97 fps

Video codec: H.264 · yuv420p

Camera pose: SLAM · position + rotation

Hand pose: 3D world + image · 21 × 3 / hand

Body pose: Exocentric · 9 joints × 3

Labels: Subtask spans · verified

Sample structure & metadata fields

LeRobot v3.0 layout with all tracks merged into the per-frame Parquet records (21 columns per frame).

AOn-disk layout

// LeRobot v3.0 dataset · tracks pre-merged

dataset/
├── DATASHEET.html                 # this document
├── meta/
│   ├── info.json                  # format · fps · features · camera roles
│   ├── tasks.parquet              # task_index → task-card id
│   ├── tracks_info.json           # merged-track feature definitions
│   ├── episodes/                  # per-episode metadata + provenance
│   ├── annotations/               # per-episode verified subtask labels
│   ├── custom_annotation.json     # all labels, flat (v1.4)
│   └── custom_metadata.csv        # per-episode sheet: environment_type · scene_id · task_description
├── data/
│   └── chunk-000/file-NNNNNN.parquet          # per-frame records · 21 columns
├── videos/
│   └── {camera}/chunk-000/file-NNNNNN.mp4     # H.264, one per camera
└── viz/                           # showcase renders: wrist-trajectory · 3D pose

BPer-frame fields · data/…parquet

// per-frame record — one parquet per episode · 1,798 rows @ 29.97 fps · 21 columns

data/chunk-000/file-NNNNNN.parquet
│
│  # frame metadata
├─ index                                  int64           # global frame index across the dataset
├─ episode_index                          int64           # episode this frame belongs to
├─ frame_index                            int64           # frame position within the episode · JOIN KEY
├─ timestamp                              float64         # seconds from episode start
├─ task_index                             int64           # resolves via meta/tasks.parquet → task-card id
│
│  # hands — image-space landmarks (NaN when hand out of frame)
├─ left_hand/tracks                       float32 [21,3]  # 21 MediaPipe landmarks · x,y,z
├─ right_hand/tracks                      float32 [21,3]
├─ left_hand/tracks/timestamp             int64           # per-track capture timestamp (ns)
├─ right_hand/tracks/timestamp            int64
│
│  # ego camera — SLAM, up-to-scale world frame
├─ base_0_camera/position                 float32 [3]     # camera position
├─ base_0_camera/position/timestamp       int64
├─ base_0_camera/position/valid           bool            # per-frame validity flag
├─ base_0_camera/slam_quaternion_xyzw     float32 [4]     # camera rotation (quaternion)
├─ base_0_camera/slam_quaternion_xyzw/timestamp  int64
│
│  # body — from the exocentric view
├─ upper_body/tracks                      float32 [9,3]   # 9 upper-body joints
├─ upper_body/tracks/timestamp            int64
├─ upper_body/tracks/valid                bool
│
│  # hands — 3D world coordinates (same frame as camera SLAM)
├─ left_hand/world                        float32 [21,3]
├─ left_hand/world/timestamp              int64
├─ right_hand/world                       float32 [21,3]
└─ right_hand/world/timestamp             int64

Sample views

The four camera views — one representative frame each, from episode 000037 (“Wrap Takeaway Coffee Cups”, café).

**Fig.** Egocentric view — episode 000037.

**Fig.** Exocentric view — episode 000037.

**Fig.** Left-wrist view — episode 000037.

**Fig.** Right-wrist view — episode 000037.

Sample annotation

Consecutive verified subtask spans with free-text labels. Shown: the first eight spans of episode 000037, verbatim.

Annotations divide each episode into consecutive subtask spans, every span carrying a natural-language label individually verified against the footage. Below: the first eight spans of episode 000037, verbatim.

{
  "episodes": [{
    "episode_id": "000037",
    "spans": [
      { "start_time": 3.2,  "end_time": 5.2,  "label": "pick up food package from drawer and hold in hand" },
      { "start_time": 5.2,  "end_time": 7.5,  "label": "pick menu cover from drawer and place menu cover on drawer" },
      { "start_time": 7.5,  "end_time": 10.0,  "label": "The person picks up a black package from the drawer." },
      { "start_time": 10.0,  "end_time": 12.1,  "label": "pick menu from top of drawer and place menu inside drawer" },
      { "start_time": 12.1,  "end_time": 15.6,  "label": "pick menu booklet from inside drawer and place menu booklet in hand" },
      { "start_time": 15.6,  "end_time": 18.8,  "label": "pick menu from drawer and place menu back in drawer" },
      { "start_time": 18.8,  "end_time": 22.0,  "label": "A person places a black brochure into a drawer." },
      { "start_time": 24.0,  "end_time": 27.7,  "label": "remove menus from inside drawer and hold in hand" },
      ...
    ]
  }]
}

Quality assurance

Two layers: signal checks on every stream, and per-label verification against footage. Measured, not asserted.

Labels are automated captions, so a fraction is inevitably wrong. To control this, we built a 240-span human-annotated evaluation set and measured our automated checker against it: when the checker rejects a label, human reviewers agree 91% of the time. We then ran the validated checker over every label in the sample — labels that failed were re-captioned from the footage and re-checked; labels that could not be confirmed were removed rather than delivered.

**Table.** Measured QA results for this sample.
What was checked	Result
Human evaluation set	240 spans, double-annotated
Checker vs. human reviewers	91% agreement on rejections
Labels screened by the checker	9,384 / 9,384 — failures re-captioned or removed
Final audit (1,800 random labels)	0 flagged
Stated label accuracy	90–95%, conservative
Video streams — technical checks	2,400 / 2,400 pass · 8 exceptions (exo view at 23.98 fps)

Accuracy is stated below the audit result because automated checking can miss subtle errors humans catch. Re-captioned labels are provenance-marked, and every annotation change is logged across versioned snapshots (v1.0 → v1.4). Label coverage averages 86% of each clip after removals.

**Table.** Tracking detection rates across all 600 episodes.
Track	Mean	Median	Episodes ≥90%
Camera position (SLAM)	96.2%	99.3%	552/600
Upper body (exo)	98.4%	99.9%	574/600
Left hand, 3D world	95.4%	99.3%	552/600
Right hand, 3D world	93.2%	99.3%	532/600

Detection tail is dominated by hands leaving the egocentric field of view — inherent to worker-worn capture. Camera positions are up-to-scale (research-grade; no absolute metric scale). Per-episode detection rates available on request.

Legal & licensing

Rights, consent, and permitted use for this delivery.

Rights & consent. All recordings were captured by consented collectors under written releases covering AI/ML training and commercial deployment. Synjuku holds all rights necessary to license this data; no third-party platform or scraped content is included.

License. Provided under the terms of the Synjuku–GDM pilot agreement. Use is limited to the permitted purposes defined there; the dataset may not be redistributed, resold, or made available to third parties on a standalone basis without Synjuku's prior written consent.

Confidentiality. This datasheet and the dataset it describes are confidential commercial materials. Provenance records (task card, capture site, recording date per episode) are retained by Synjuku and support audit on request.

Synjuku

GDM pilot sample · 2026.07.02