Textbooks Are All You Need
www.youtube.com
I discuss the power of the "Textbooks Are All You Need" methodology to build much more compact LLMs using higher quality data. I emphasize phi-1 (coding LLM ...

With the recent release of the phi-1 model weights on HuggingFace, co-author Bubeck explains the landmark paper “Textbooks Are All You Need” in this video.

The main takeaway is that researchers have realised astounding results in training smaller LLMs with carefully filtered and specially curated datasets:

The resulting 1.3 billion parameter phi-1 model scores 50.6% on HumanEval.

To see how other models score on HumanEval check out the Big Code Models Leaderboard while keeping in mind the model size.