LLM unit test automation

Background

Writing unit tests for Spring Boot services is repetitive work that teams postpone, so coverage drifts down over time. Existing generators produced structurally valid but semantically empty tests, and none of them integrated with the Maven build the team already ran. A literature review of automated testing tools and LLM applications in software engineering found no approach that combined coverage analysis, test generation and the build lifecycle for Java Spring Boot.

Approach

• ✓Applied Design Science Research with its relevance, rigor and design cycles, so the artifact was built against a stated real-world problem rather than a hypothetical one.
• ✓Developed and compared prompt strategies systematically: zero-shot, one-shot and few-shot, evaluating which produced compilable and meaningful JUnit 5 tests.
• ✓Built UTestGenLLM, a Maven plugin that hooks into configurable build phases, analyses the project's classes and generates tests without leaving the normal build.
• ✓Integrated JaCoCo so the plugin reads existing coverage, identifies untested classes and methods, and prioritises generation by the actual gaps rather than generating blindly.
• ✓Ran the LLM against a private Azure-hosted GPT instance, so proprietary source code never reached a public endpoint.
• ✓Validated across multiple projects with quantitative coverage metrics and an expert evaluation.

Results

• ✓Test coverage rose from 9 to 42 percent across the evaluated projects
• ✓Effort for writing tests fell from 1,440 to 30 minutes per project compared with manual development
• ✓Cost per generated test suite fell from about 566 € of developer time to about 2.13 € in API cost
• ✓Prompt engineering was shown to measurably affect the quality of generated technical code
• ✓Produced a reusable framework for integrating LLMs into enterprise software development workflows

Technology stack

• Java
• Maven
• Spring Boot
• JUnit 5
• JaCoCo
• OpenAI GPT-3.5 and GPT-4
• Azure OpenAI