UNDIP, Semarang (September 23, 2025) – In the midst of rapid digital transformation, the appearance of an application is often underestimated. Yet the user interface (UI) is the first gateway that determines whether people feel comfortable using technology or choose not to return. From banking apps and online shopping to public service systems, visual interaction plays a decisive role in an application’s success.
Ensuring a truly user-friendly UI, however, is no simple task. UI testing has long relied on manual checks of each screen, a process that is slow, costly, and prone to human error. Minor but critical issues—such as tiny buttons, overlapping text, or confusing icons—often slip through unnoticed, negatively affecting user experience.
Addressing this challenge, a research team from Universitas Diponegoro introduced a breakthrough. Led by Dr. Aris Puji Widodo, S.Si., M.T., lecturer at the Department of Informatics, Faculty of Science and Mathematics, and Director of UNDIP’s Information Systems and Technology, the team developed an Artificial Intelligence (AI) technology based on few-shot learning. Together with Dr. Eng. Adi Wibowo, S.Si., M.Kom., and Dr. Kabul Kurniawan, S.Si., M.Cs., they leveraged AI to automate UI testing more efficiently and accurately.
“With only five examples per category, our model can already recognize screen types and automatically detect design flaws,” Dr. Aris Puji said. Their findings, published in the International Journal of Advanced Computer Science and Applications (2023), mark UNDIP’s global contribution to digital technology development.
The research used two main datasets. The first dataset, the Enrico Dataset, contains 1,460 UI screens from various apps, categorized into 20 design types, ranging from login pages and chat interfaces to media players and maps. The second, the Mistake Dataset, developed by UNDIP’s team, comprises 200 screens featuring 10 common errors, including low contrast, unclear icons, and poor typography.
The AI model, trained with EfficientNet-B1, showed promising results. Using just five-shot learning—or five examples per category—it achieved accuracy rates of up to 76.05%. Even on more challenging datasets, the model still reached 42.6%, considered high given the limited training data.
“The efficiency and accuracy we achieved are good news, especially for startups or small developer teams without access to large datasets,” Aris said. He noted the technology could significantly reduce testing time and production costs, allowing teams to focus on creative innovation.
For the public, the benefits can be felt in three key ways. First, apps become more user-friendly, with minor issues like hard-to-tap send buttons or overlapping text resolved more quickly. Second, the technology promotes digital inclusion by detecting designs that are unfriendly to vulnerable groups such as older people, the visually impaired, or users with motor limitations. Third, digital services in both the public and private sectors can be launched more quickly without compromising quality.
“Innovation is not just about technology, but also fairness and accessibility. Everyone deserves apps that are easy to use,” Dr. Aris emphasized.
In the industry, automating UI testing with AI has significant implications. In modern software development, where continuous integration and delivery (CI/CD) are key, speed and precision are essential. Integrating AI into DevOps pipelines enables UI testing to run automatically whenever a design change occurs. “Imagine that every revision instantly receives evaluations and recommendations for improvements. This innovation would be a revolution in workflow,” he said.
Still, challenges remain. Data diversity and AI result interpretation require attention. According to Aris, intelligent AI still needs human guidance, especially in complex design ethics and user experience. Future research will expand the datasets, involve end-users in training, and broaden detection to other UI aspects, including navigation and animation.
“We are optimistic. AI is not meant to replace designers or testers, but to act as a partner that accelerates and strengthens innovation,” Aris concluded.
This innovation demonstrates UNDIP’s commitment to its tagline, “Noble and Valuable UNDIP,” realized through impactful research under the “Diktisaintek Berdampak” program. By combining artificial intelligence with a spirit of innovation, UNDIP contributes to a more inclusive, reliable, and user-centered digital ecosystem that responds to today’s needs while delivering tangible benefits to society. (Public Communication/ UNDIP/ DHW)