This article provides a guide on fine-tuning the StarCoder2, 3B model, a code-focused large language model, specifically for SQL programming using Google Colab T4 GPU. It highlights the model's performance and training details, focusing on its application in code generation and completion.
From the article
StarCoder is a Code LLM (Large Language Model) by BigCode in collaboration with Hugging Face. It was trained on the Stack dataset containing more than 600 programming languages. StarCoder2 was trained with 3B, 7B and 15B parameters on 3.3 to 4.4 trillions tokens. The Starcoder2–3B outperforms Code LLMS of similar size. The largest model, StarCoder2–15B outperforms CodeLlama-34B, a model more than twice its size, and also DeepSeekCoder-33B, the best-performing model at code completion for high-resource languages. In this blog,we will learn to fine-tune the smallest LLM model of BigCode, StarCoder2–3B, exclusively in the SQL programming language. Large Language Models for Code have rapidly...
Skills
- codellms
- sql
- code
- large-language-models
- fine-tuning
