Code for ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling.
NeurIPS 2026
TLDR: ST-DiffEye is a diffusion model that generates continuous eye-trajectoy and aligned scanpath conditioned on a visual stimuli and an optional search target, using a per-point fixation that ties each trajectory point to human scanpath.
The code was tested with Python 3.11, PyTorch 2.3.1 and CUDA 12.1 on a single GPU.
Conda
conda env create -f environment.yml
conda activate stdiffeyepip (into an existing Python 3.11 environment)
pip install -r requirements.txtDownload the released checkpoints from Hugging Face into checkpoints/:
hf download briannlongzhao/ST-DiffEye --local-dir checkpointsWe use COCO-Search18, COCO-FreeView and MIT1003.
# Scanpaths, images and image features from the ScanDiff data release
wget https://ailb-web.ing.unimore.it/publicfiles/ScanDiff_ICCV2025/data.zip
unzip data.zip -d data && rm data.zip
# MIT1003 gaze recordings
wget https://people.csail.mit.edu/tjudd/WherePeopleLook/DATA.zip
unzip DATA.zip -d data/mit1003 && rm DATA.zipFor COCO-Search18 and COCO-FreeView, request the raw EyeLink .edf gaze recordings from the dataset authors, put them under data/cocosearch18/edfs/ and data/coco_freeview/edfs/, and convert them to .asc with edf2asc from the EyeLink Developers Kit:
find data/cocosearch18/edfs data/coco_freeview/edfs -name "*.edf" -exec edf2asc -input {} \;Every dataset has an experiment preset in configs/experiment/ that holds all of its settings, so each command only needs the preset name, the data folder and, for evaluation, a checkpoint. src/train.py trains a model; src/eval.py generates trajectories and scanpaths for the test set with a checkpoint and computes the metrics.
The default training batch sizes (128, or 64 for COCO-FreeView) need more than 24 GB of GPU memory. On smaller GPUs, lower it with data.batch_size_train=... (e.g. 32 for the COCO-Search18 and MIT1003 presets, 8 for COCO-FreeView on a 24 GB GPU).
# Train
python src/train.py experiment=cocosearch18_tp data_dir=/path/to/data
# Evaluate the released checkpoint
python src/eval.py experiment=cocosearch18_tp data_dir=/path/to/data ckpt_path=checkpoints/COCOSearch18-TP.pt# Train
python src/train.py experiment=cocosearch18_ta data_dir=/path/to/data
# Evaluate the released checkpoint
python src/eval.py experiment=cocosearch18_ta data_dir=/path/to/data ckpt_path=checkpoints/COCOSearch18-TA.pt# Train
python src/train.py experiment=cocofreeview data_dir=/path/to/data
# Evaluate the released checkpoint
python src/eval.py experiment=cocofreeview data_dir=/path/to/data ckpt_path=checkpoints/COCOFreeView.pt# Train
python src/train.py experiment=mit1003 data_dir=/path/to/data
# Evaluate the released checkpoint
python src/eval.py experiment=mit1003 data_dir=/path/to/data ckpt_path=checkpoints/MIT1003.ptsrc/demo.py generates trajectories and scanpaths for a single image with a released checkpoint, and saves a figure with the trajectories (left) and scanpaths (right) overlaid on the image. It needs the checkpoints and data/task_embeddings.npy from the data release.
# Visual search: target present / target absent
python src/demo.py --image path/to/image.jpg --task cocosearch18_tp --target clock
python src/demo.py --image path/to/image.jpg --task cocosearch18_ta --target "potted plant"
# Free viewing
python src/demo.py --image path/to/image.jpg --task cocofreeview
python src/demo.py --image path/to/image.jpg --task mit1003--task:cocosearch18_tp,cocosearch18_ta,cocofreeviewormit1003; selects the checkpoint incheckpoints/(override with--ckpt).--target: search target forcocosearch18_tp/cocosearch18_ta, one ofbottle,bowl,car,chair,clock,cup,fork,keyboard,knife,laptop,microwave,mouse,oven,potted plant,sink,stop sign,toilet,tv.--num_samples: number of generated samples (default 10);--seed: random seed for reproducible samples (default: none, so every run differs).--out: output figure (default<image name>_<task>[_<target>].pngin the current folder).
@article{zhao2026stdiffeye,
title = {ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling},
author
584C
= {Zhao, Brian Nlong and Kara, Ozgur and Kim, Junho and Rehg, James M.},
journal = {arXiv preprint arXiv:2606.15486},
year = {2026}
}