Search Quality Assurance with AI as a Judge

Deep dive into how Zalando builds a search quality assurance framework with LLM as a judge to evaluate the search quality at scale with high coverage and multi language support.

Tao Ruangyam۲۵ اسفند ۱۴۰۴ ۱۳ دقیقه مطالعهمتن اصلی

In 2024, Zalando research published a paper on LLM-as-a-judge for search quality assurance at scale. The framework allows scientists and developers to effectively evaluate the semantic relevance of search results of the given search queries at large scale with multi-language support. This capability has strong potential to help the Search engineering team to quickly identify and fix search issues which we will walk through in this post.

Real-world use case: Launching a new country¶

In 2025 Zalando expanded its fashion store business into 3 new countries: Luxembourg, Portugal and Greece. Ensuring these markets have a good search experience is critical for the success of the launch, but the challenge is how can we do that without any prior search data from real users?

Before using LLM-as-a-judge, the search quality assurance process was heavily reliant on human experts and a manual process as follows: