This repository contains the training and evaluation code accompanying the thesis "Learning with Less: Contrastive Weight Tying on the BabyLM Challenge" (Ino van de Wouw, VU Amsterdam, 2025). The project studies headless language models, models pretrained with Contrastive Weight Tying (CWT) (Godey et al., 2024) instead of a standard cross-entropy
glue slurm pytorch transformer computational-linguistics gpt bert blimp huggingface pretrained-language-model pytorch-lightning contrastive-learning sample-efficient-rl llm babylm sample-efficient-ml contrastive-weight-tying headless-language-models sample-efficient-learning
-
Updated
Jul 23, 2026 - Python