Fine-tuning on SSV2 with official 1B_ft_ssv2_f8.pth weights yields only ~2.78% Top-1 accuracy, loss barely decreases

[Wjqboss]

I downloaded the official weights (L14_ft_ssv2_f8.pth) and fine-tuned the model on the SSV2 dataset. After 8 epochs, the validation Top-1 accuracy is only ~2.78%, which is barely above random guessing (SSV2 has 174 classes, random baseline is ~0.57%). Both training and validation loss remain nearly flat throughout training (~4.22 → 4.04), indicating that the model is not learning.

{"train_lr": 2.1873697505525736e-06, "train_loss": 4.39070353931896, "val_loss": 4.047283397870281, "val_acc1": 2.550649960861434, "epoch": 0}
{"train_lr": 6.3125e-06, "train_loss": 4.2231926481143605, "val_loss": 4.047495159728945, "val_acc1": 2.046170143769742, "epoch": 1}
{"train_lr": 1.0437630249447425e-05, "train_loss": 4.223525819729209, "val_loss": 4.046330469506293, "val_acc1": 2.0582776594242813, "epoch": 2}
{"train_lr": 1.2100895909686002e-05, "train_loss": 4.223054990936006, "val_loss": 4.0448254477250725, "val_acc1": 2.7322626938320687, "epoch": 3}
{"train_lr": 9.890037348431687e-06, "train_loss": 4.22190657817833, "val_loss": 4.043751512584705, "val_acc1": 2.7161193368548253, "epoch": 4}
{"train_lr": 6.312681116098593e-06, "train_loss": 4.221108646998975, "val_loss": 4.045732998547919, "val_acc1": 2.3528938718372863, "epoch": 5}
{"train_lr": 2.7352557035717826e-06, "train_loss": 4.220008719731904, "val_loss": 4.041641456921588, "val_acc1": 2.691904306469476, "epoch": 6}
{"train_lr": 5.242160262188567e-07, "train_loss": 4.219491178569124, "val_loss": 4.0436078170377545, "val_acc1": 2.5748649875518628, "epoch": 7}
{"Final top-1": 2.7801303531317894, "Final Top-5": 12.541886069889898}