- Packages
conda install -c conda-forge tqdm conda install -c anaconda scipy conda install -c conda-forge scikit-learn conda install -c open3d-admin open3d conda install pytorch==1.4.0 torchvision==0.5.0 -c pytorch conda install -c huggingface transformers - Follow instructions from https://github.com/facebookresearch/SparseConvNet to download SparseConvNet.
For the Scannet Dataset please go to https://github.com/ScanNet/ScanNet and fill out the agreement form to download the dataset.
For the ScanRefer Dataset please go to https://github.com/daveredrum/ScanRefer and fill out the agreement form to download the dataset.
Download the preprocessed glove embeddings from ScanRefer.
scannet_data
|--scans
This Repository
|--glove.p
|--ScanRefer
|--Files from ScanRefer download
First store the point cloud data for each scene into pth files.
python prepare_data.py
Split the files into train and val folders.
python split_train_val.py
Please download the pretrained instance segmentation model and place into the folder like this.
This Repository
|--GRU
|--checkpoints
|--model_insseg-000000512.pth
|--BERT
|--checkpoints
|--model_insseg-000000512.pth
Pretrained model for referring model with gru encoder and place into the folder like this.
This Repository
|--GRU
|--checkpoints
|--gru
|--models
|--gru-000000032.pth
Pretrained model for referring model with bert encoder and place into the folder like this.
This Repository
|--BERT
|--checkpoints
|--bert
|--models
|--bert-000000064.pth
Train the referring model with GRU encoder. (Note that we train with 2 GTX 1080Tis and Batchsize 8)
cd GRU/
python unet_gru_CoP.py
Validate referring model with GRU encoder.
cd GRU/
python unet_gru_CoP_val.py
Validate referring model with BERT encoder.
cd BERT/
python unet_bert_val_CoP.py
Our dataloader and training implementations are modified from https://github.com/facebookresearch/SparseConvNet and https://github.com/daveredrum/ScanRefer, please go check out their repositories for sparseconvolution and 3D referring object localization implementations respectively. We would also like to thank the teams behind Scannet and ScanRefer for providing their dataset.