Search Quality Assurance with AI as a Judge
Deep dive into how Zalando builds a search quality assurance framework with LLM as a judge to evaluate the search quality at scale with high coverage and multi language support.
In 2024, Zalando research published a paper on LLM-as-a-judge for search quality assurance at scale. The framework allows scientists and developers to effectively evaluate the semantic relevance of search results of the given search queries at large scale with multi-language support. This capability has strong potential to help the Search engineering team to quickly identify and fix search issues which we will walk through in this post.
Real-world use case: Launching a new country¶
In 2025 Zalando expanded its fashion store business into 3 new countries: Luxembourg, Portugal and Greece. Ensuring these markets have a good search experience is critical for the success of the launch, but the challenge is how can we do that without any prior search data from real users?
Before using LLM-as-a-judge, the search quality assurance process was heavily reliant on human experts and a manual process as follows:
Keep reading Zalando Engineering with Pro
This source is part of the Pro library. Unlock every premium source, AI translation and custom feeds.
Originally published on
Zalando Engineering
Keep reading
Netflix TechBlog
Uber Engineering