A repo containing tokenizer training code in java and a simple pytorch LM training setup.
First clone the repository, cd env to enter the environment folder.
run Scripts/activate.ps1 in the console (if using powershell) to activate the virtual environment.
To install dependencies, run pip install -r requirements.txt
To train a tokenizer, first download and prepare the data.
python3 src/training/download_data.py
python3 src/training/download_data_sft.py
python3 src/training/generate_tokenizer_train_samples.pyThen run the java script to train the tokenizer:
java ./src/tokenizer/TokenizerTrain.javaFinally, encode all the files with the trained tokenizer:
java ./src/tokenizer/EncodeFiles.javaTo train the model:
python3 src/training/train.py --config_file <your_config_here>
(config sample for training can be found at env/src/training/configs/sample_train.yaml)
And for inference:
python3 src/training/inference.py --config_file <your_config_here>