Small Language Model Ensembles to Improve AI Code Generation
Open Access DepositedIn recent years, transformer-based AI language models have gained prominence due to their powerful capabilities in a variety of tasks including generation of images, video, text and code. Large Language Models (LLMs) exist with parameters counts of a trillion parameters and greater. Such models are proprietary and unavailable for organizations to deploy privately. Even if such deployments were possible, the tremendous resource requirements of LLMs preclude their deployment on infrastructure smaller than enterprise and hyper-scale data centers. Small Language Models (SLMs), with far lower parameter counts of billions or fewer are a viable alternative for use on small servers and edge devices including PCs. While SLMs possess similar generative capabilities as LLMs, the reduction in model size is correlated with a decrease in accuracy when evaluated across a broad range of generative applications, including code generation in multiple languages. To mitigate this shortcoming, an SLM may be fine-tuned with a curated code dataset consisting of code examples in a target programming language. This praxis presents results illustrating how two fine-tuned SLMs variants have been created that improve average accuracy in C++ code generation by more than 9%, and Rust code generation by more than 14%. Fine-tuning a model to improve its accuracy on a task such as code generation in one language may reduce the output quality of the fine-tuned model in other areas, such as code generation in another language, or prose text generation. To overcome this tradeoff in generation quality, a novel SLM ensemble architecture was created for this praxis. This router-based ensemble encapsulates multiple fine-tuned specialist code models. At inference time, a model is selected as the most likely to produce optimal output for a given prompt. This model is then used to generate the final result. This praxis defines the problem space of code generation with SLMs and provides an in-depth review of the methods used to create the ensemble model, the experimental methodology, and the significant capability improvements shown in the results. A notable finding is that by employing an ensemble of fine-tuned coding SLMs, average code generation accuracy in C++ and Rust is improved by more than 10% compared to an off-the-shelf SLM.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.
| Thumbnail | Title | Date Uploaded | Visibility | Actions |
|---|---|---|---|---|
|
|
McEttrick_gwu_0075A_17744.pdf | 2026-02-26 | Open Access |
|