Google announced the launch of the world's first double-blind evaluation for cutting-edge proprietary AI models, restricting external evaluation to an encrypted "black box" environment to prevent models from accessing test information in advance to optimize their performance.
Just as students cannot accurately demonstrate their abilities if they see exam questions before a test, current AI model evaluations face the same problem: if a model encounters the test questions in advance (that is, so-called "benchmark contamination"), the credibility of the evaluation results is significantly undermined.
Google worked with the Singapore Institute of Artificial Intelligence Safety, OpenMined, AVERI, and MLCommons to conduct confidential benchmarking of the Gemini Flash Lite model in a privacy-preserving environment, improving the integrity of the evaluation results.
Traditional high-risk external evaluations have long faced a dilemma: either the evaluator hands over the test prompts, creating a risk that the model provider could see the questions in advance, or the model provider hands over the model weights, risking intellectual property leaks.
Double-blind evaluation resolves this conflict. Through Confidential Space, part of Google Cloud's Confidential Computing product suite, cryptographic verification ensures that external evaluation data and the proprietary model remain private to their respective owners: the evaluator cannot see the Gemini model weights, and Google cannot see the evaluator's test prompts.
This encryption method can effectively prevent benchmark contamination while protecting sensitive data, making it especially important for highly sensitive evaluations conducted by cybersecurity or government organizations. Double-blind evaluation enables independent institutions to rigorously test advanced AI models without compromising data sovereignty or security.
Google said it hopes this pilot can open up a new direction for model oversight and help the industry as a whole build AI systems that are safer, more reliable, and more trusted by the public.