Join the conversation
Join the community of Machine Learners and AI enthusiasts.
Sign UpI think the emphasis on data quality is well placed. Smaller models seem much less forgiving of noisy or inconsistent training data, so improvements in curation may end up having a larger impact than simply adding more parameters.
That raises an interesting question: do you think the looping behavior is primarily caused by the training data, or do distillation and quantization play a larger role as models get smaller?
I think is the second thing. A small model that enters a strange context may produce a relatively flat distribution. It does not have a strong idea of what should come next but still it has to pick something up. Anti-repetition behavior from data consumes capacity, the less capacity, the more likely your model will suffer from this bias.
That's a simplified view I have on self-reinforcing degeneration mechanism which you can read about here btw: https://arxiv.org/abs/2109.08705