UI interpretation system

About the client

A leading player in the high-tech industry.

How it all started

The client asked XPG Factor to create an intelligent LLM-based multimodal solution that could efficiently handle various system interfaces used within the company. The idea was to enable the solution to analyze screenshots uploaded by employees, identify interface elements such as fields, buttons and menus, and interpret the visual context. This would allow it to provide clear and accurate instructions for new employees and accelerate the onboarding process.

Challenges we faced

The main challenge for us was the high variability of interfaces. This meant that the system had to be flexible to recognize and interpret dozens of different screens and interface elements.

Implemented solution

Our team developed a custom AI solution with an architecture specifically designed for scaling and multimodal data processing. At its core is a FastAPI microservice running on a GPU-accelerated Azure virtual machine, which allows the system to process images and generate responses almost instantaneously. As a result, the system doesn’t just provide standardized prompts, but creates informed and contextually relevant recommendations that actually help employees make the right decisions in real time.

Solution’s key attributes

• Robust operation
  Thanks to the use of a vector knowledge database and Retrieval-Augmented Generation, the system quickly adapts to new interfaces and ensures stable performance during peak loads and simultaneous multi-user access.
• Logically structured guidance
  The system is designed to provide not just isolated facts, but context-aware instructions that match the employee’s workflow.
• Multiple roles and scenarios
  The system is scalable for different roles. Its functionality can be adapted for use by supervisors, technical administrators or even for training purposes.
• Live knowledge control
  The system supports real-time updates from admins: they can input new guidance, revise existing content or refine decision logic.
• User activity monitoring
  With the query history function, the system helps to identify typical user problems, refine recommendations and optimize training procedures.

Security and privacy

During the project, it was important for us to ensure that the system complied with strict restrictions on data transfer and storage, especially screen images, which could potentially contain confidential information. Therefore, our solution is designed to meet all data protection requirements, including environment isolation, encryption and access control.

Testing

The testing process consisted of three main stages: validating all components, running controlled tests using real system screenshots and a pilot deployment. Each stage was followed by relevant improvements, enabling us to deliver a fully operational and effective solution.