IEEE-RAS Humanoids 2026

DexDirect

Direct Kinesthetic Arm Guidance for Efficient Dexterous Demonstration Collection

3.2×

more successful demos than TeleDex

17.2×

more successful demos than AnyTeleop

2.01

overall workload (Raw TLX), vs 3.08 and 3.81

90%

autonomous policy success, 18 of 20

1Mechanical and Aerospace Engineering 2Electrical and Computer Engineering 3Computer Science

University of California, Los Angeles · †Equal advising

Abstract

Scalable collection of dexterous manipulation demonstrations remains a major bottleneck for robot learning. High-fidelity interfaces often require costly hardware and extensive setup, while low-setup, low-cost alternatives tend to provide less precise control and impose greater cognitive workload on operators. We present DexDirect, a direct kinesthetic arm guidance for efficient dexterous demonstration collection. The operator drags a 6-DoF gravity-compensated robot arm directly by a handle, while a single webcam retargets operator’s other hand onto a 16 joints 13-DoF dexterous robot hand. User studies suggest DexDirect collects 17.2× and 3.2× more successful demonstrations compared to purely vision (AnyTeleop) and pose-tracking (TeleDex) baselines. An adapted NASA-TLX shows DexDirect greatly reduces mental demand, effort, and frustration, despite raising physical demand. A diffusion policy trained on DexDirect demonstrations reaches a 90% success rate on a cube pick-and-place task. These results suggest that direct kinesthetic arm guidance combined with vision-based hand retargeting provides an efficient low-setup and scalable interface for collecting dexterous manipulation demonstrations.

Two Hands. Two Pathways.

Arm motion needs global 6-DoF positioning and physical contact. Finger articulation is high-dimensional but lives in a hand-local frame. DexDirect gives each the interface that suits it, with no leader arm, no operator-to-robot calibration and no arm-side retargeting.

Data-collection diagram. The operator's left hand moves the arm through a handle in gravity-compensation mode. The right hand is filmed by a webcam, tracked with MediaPipe and retargeted with dex-retargeting at 30 Hz. The arm and hand joint encoders (200 Hz), tactile sensors (60 Hz) and wrist camera (30 Hz) are recorded while the pedal is pressed, then processed into aligned data.

Left hand

Drag the Arm

The arm runs in a gravity-compensated drag mode with zero position stiffness and low joint damping. A staged system identification corrects the model gravity torques for payload, torque scaling and friction. Trajectories come straight from the joint encoders, with no pose estimation, retargeting or IK, and the rigid handle lets the operator feel contact.

200 Hz
joint-level control
0
position stiffness
0.05 kg
3D-printed handle

Right hand

Track the Fingers

A palm-facing webcam feeds MediaPipe Hands, and wrist-relative landmarks are mapped to the MIDAS hand with the dex-retargeting framework released with AnyTeleop. The camera never estimates the wrist pose because the arm already sets it. A 3D-printed forearm support keeps the palm centered in view.

30 Hz
hand tracking
21
3D hand landmarks
16 / 13
robot joints / DoF

Foot pedal

Record the Robot

The pedal starts recording. Because the operator moves the robot itself, the robot’s own sensors are the record: arm and hand joint states, tactile readings and wrist-camera images, all stamped by one clock on the logging computer. For training, each stream is matched to the nearest wrist-camera frame, without interpolation, to form 30 Hz episodes.

200 Hz
arm & hand states
60 Hz
tactile
30 Hz
wrist camera & episodes

Try It in Simulation

Drag the arm by its orange handle as you would the real one, drive the hand with your webcam (or the grip slider), and play with the cubes. Reset puts everything back.

The simulation needs JavaScript, WebGL and WebAssembly.

Annotated DexDirect setup: a 6-DoF PIPER arm with the MIDAS robot hand, a wrist handle for kinesthetic guidance, a wrist support with a hand-tracking webcam, a wrist camera, and a foot switch that triggers data collection.

Hardware

Arm
Agilex PIPER, 6-DoF
Hand
MIDAS: 16 joints, 13 actuators, 283 tactile taxels
Handle
PLA, 0.05 kg, grip 65 mm off the flange
Wrist camera
Luxonis OAK-1, wide FOV
Finger tracking
Logitech C270, 45 cm below the palm
Recording
Foot switch; arm & hand 200 Hz, tactile 60 Hz, camera 30 Hz

System Overview

Joint, tactile and wrist-camera streams share one clock and are aligned to the camera frames, forming 30 Hz episodes for a multimodal diffusion policy.

System overview diagram: the left hand guides the arm, the right hand is tracked by a webcam and retargeted, joint, tactile and wrist-camera streams are recorded and synchronized, and a multimodal diffusion policy is trained and deployed.

Five Contact-Rich Tasks

Real-time excerpts of DexDirect evaluation sessions. Numbers are user-study means.

Compare Interfaces

Same robot, hand, finger retargeting, visual feedback and tasks. Only the way the operator commands the arm changes.

Side-by-side photos of the three operator interfaces: DexDirect, TeleDex and AnyTeleop.

DexDirect Ours

Drag the physical arm by a wrist handle; joint encoders record the motion.

481
successful demos
2.01
Raw TLX workload

TeleDex

A handheld smartphone in a 3D-printed wrist mount is tracked in free space.

151
successful demos
3.08
Raw TLX workload

AnyTeleop

An RGB-D camera estimates the wrist pose; CuRobo MPC at 120 Hz drives the arm.

28
successful demos
3.81
Raw TLX workload
Participants
10, new to all three interfaces
Order
Counterbalanced: ABC, BCA, CAB
Time budget
1 min practice, then 120 s per task (180 s for Jenga)

More Demonstrations. Less Workload.

DexDirect had the highest success rate on every task and finished successful trials 1.8–4.8× faster than TeleDex, so the gain does not come from simply attempting more often.

Successful demonstrations collected All participants and tasks, same time budget per interface
DexDirect 481
TeleDex 151
AnyTeleop 28

Per-task results (mean ± SEM; completion time over successful trials)

Task Success rate Completion time (s)
DexDirectTeleDexAnyTeleop DexDirectTeleDexAnyTeleop
Jenga extraction0.94 ± 0.020.69 ± 0.080.44 ± 0.1210.46 ± 0.8249.76 ± 5.9835.50 ± 5.42
USB unplugging0.95 ± 0.020.81 ± 0.040.39 ± 0.095.88 ± 0.6510.36 ± 1.9017.75 ± 5.74
Lazy Susan pick-and-place0.96 ± 0.020.85 ± 0.060.50 ± 0.138.36 ± 0.4921.96 ± 2.7347.14 ± 11.15
Whiteboard erasing0.91 ± 0.040.56 ± 0.120.00 ± 0.0013.85 ± 1.3939.10 ± 7.82—
Keyboard typing0.71 ± 0.040.32 ± 0.060.00 ± 0.006.60 ± 0.3217.50 ± 3.97—

— AnyTeleop produced no successful whiteboard or keyboard trials. Participants struggled to regulate depth against the board, and latency turned brief key presses into repeated registrations.

Lower Workload

On an adapted NASA-TLX, DexDirect’s overall Raw TLX score was 2.01, against 3.08 for TeleDex and 3.81 for AnyTeleop (paired Wilcoxon signed-rank, p = 0.0195 and p = 0.00195).

1.77
mental demand
TeleDex 3.19
1.47
frustration
TeleDex 2.79
2.17
effort
TeleDex 3.40

The one trade-off is physical demand, which rises to 3.23 (vs 2.72 TeleDex, 2.62 AnyTeleop). DexDirect swaps the cognitive effort of tracking and remapping for the physical effort of moving a gravity-compensated arm.

Bar chart of mean adapted NASA-TLX scores per dimension for AnyTeleop, TeleDex and DexDirect. DexDirect is lowest on every dimension except physical demand.
Mean adapted NASA-TLX over 10 participants and 5 tasks; 95% CIs; lower is better.

90% Autonomous Success

A diffusion policy trained on 200 DexDirect demonstrations, collected by one operator separately from the user study, lifts a cube and drops it onto a box 30 cm away in 18 of 20 trials.

Vision
Wrist RGB 240×240 through DINOv2 ViT-S/14
Proprioception
13-D absolute hand joint positions
Touch
4-D thumb and index tactile summary
Action
19-D chunks: 6-DoF end-effector + 13 fingers
  1. Rollout at t = 0 s: the hand above a cube on the table.0 s
  2. Rollout at t = 5 s: the hand grasping the cube.5 s
  3. Rollout at t = 10 s: the cube lifted above the box.10 s
  4. Rollout at t = 15 s: the cube placed on the box.15 s

Watch DexDirect

Supplementary video · 1:12

Citation

@inproceedings{kim2026dexdirect,
  title         = {DexDirect: Direct Kinesthetic Arm Guidance for Efficient
                   Dexterous Demonstration Collection},
  author        = {Kim, Beom Jun and Wang, Shiu-Jen and Liu, Jonathan and Zhu, Alvin and
                   Wang, Quanyou and Fang, Hanzhang and Xu, Feng and Zhu, Mingzhang and
                   Cui, Yuchen and Hong, Dennis W.},
  booktitle     = {IEEE-RAS International Conference on Humanoid Robots (Humanoids)},
  year          = {2026},
  eprint        = {2607.27784},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}