A Machine Learning Approach to Enhanced Software Quality Through the Automated Creation of Unit Tests
Open Access DepositedThe escalating complexity of modern software systems has amplified the cost and difficulty of maintaining high-quality test suites. Conventional manual testing strategies have struggled to keep pace, prompting interest in leveraging large language models (LLMs) for automated unit test generation. This praxis investigates whether compact, memory-efficient LLMs can be repurposed for this task, and how various augmentation strategies can improve upon the effectiveness of LLMs alone. Three variants of LLMs based on Microsoft’s (MS) Phi-3 compact LLM were evaluated. The baseline consisted of the unmodified MS Phi3-mini and Phi-3.5-mini models, each having 3.8 billion parameters and a 128k token context. Two models optimized by Unsloth, Phi3-mini-4k-instruct (4k context) and Phi3.5-mini-instruct (128k context) were fine-tuned on a Python source dataset. A third model, the MS Phi-3-mini-4k-instruct, served as a comparator for the Unsloth 4k model. Four experimental conditions were applied to each model. First, no augmentation was done to establish each model’s base performance. Second, an Abstract Syntax Tree (AST) representation of the Python source code was added to the source code providing structural information to the model. Third, tests were generated by each model, then passed through a Python code linting tool looking for errors. Error messages, the original source code, and the generated tests were then fed back into the model with a prompt requesting that the model fix the errors. Fourth, the body of the Python reference guide was stored in a Retrieval Augmented Generator (RAG) version of each model and used to assist test case generation by referencing the language documentation based on the source code. This study shows that fine-tuning with AST augmentation produced the greatest improvement over any of the baseline models, while feedback and RAG augmentation degraded performance overall, especially on the models with 4k context length.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.