Artificial Intelligence-Based System for Boosting Automated Code Generation from Natural Language Descriptions
Open Access Depositedreflection adds +6 pp on average. Fine-tuning on MBPP delivers +3–5 pp improvements. In addition, latency and cost analyses reveal substantial practical advantages of SLMs.
Artificial Intelligence-Based System for Boosting Automated Code Generation from Natural Language Descriptions Automating code generation promises to have a significant impact on software development acceleration, reduction of costs, and human error minimization. This study evaluates the viability of small language models (SLMs)—augmented with agentic workflows—as a privacy-preserving, cost-effective alternative to proprietary large language models (LLMs) for generating Python code based on natural-language descriptions. 24 open-source SLMs (2.8B - 22B parameters) were compared across four benchmarks—HumanEval, MBPP, LBPP, and BigCodeBench—using a uniform inference pipeline implemented with PyTorch and HuggingFace. The experiments were conducted with three stages of post-processing (raw output, fence extraction, full cleaning) and a tiered prompt-engineering framework (basic, instructional, full prompts). Two agentic workflows were introduced—a two-stage reflection agent and a multi-agent collaboration chain—to iteratively refine generated code. Key hyperparameters (temperature, top-p) were systematically tuned, and selected SLMs underwent fine-tuning via QLoRA on the MBPP training dataset. Results demonstrate that full cleaning achieves the maximum mean pass@1 scores. Basic prompts outperform more elaborate prompts. Agentic workflows yield further gains
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.