Back to AI Research

AI Research

Blind-Spots-Bench: Evaluating Blind Spots in Multim... | AI Research

Key Takeaways

  • Modern AI models often excel at complex tasks like coding or advanced mathematics, yet they frequently stumble on simple, everyday challenges—such as countin...
  • Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipulating a string or drawing a dog with five legs.
  • These examples suggest that existing benchmarks may under-measure persistent blind spots in current systems.
  • We introduce $\texttt{blind-spots-bench}$, a benchmark designed to expose such blind spots through tasks that appear simple for humans but remain challenging for modern AI.
  • We collect raw questions from students in an AI course, clean and annotate them with structured reference solutions, and propose a task taxonomy tailored to the resulting dataset of 235 samples.
Paper AbstractExpand

Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipulating a string or drawing a dog with five legs. These examples suggest that existing benchmarks may under-measure persistent blind spots in current systems. We introduce $\texttt{blind-spots-bench}$, a benchmark designed to expose such blind spots through tasks that appear simple for humans but remain challenging for modern AI. We collect raw questions from students in an AI course, clean and annotate them with structured reference solutions, and propose a task taxonomy tailored to the resulting dataset of 235 samples. We further develop an automated grading pipeline to evaluate a wide range of models, including open-weight and closed-source language, vision-language, and image-generation models. Our analysis on $\texttt{blind-spots-bench}$ reveals that closed-source frontier models can substantially outperform open-weight models with even $\approx10\%$ gap, even when they attain comparable performance on existing benchmarks. A more fine-grained analysis shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models. These results highlight the value of $\texttt{blind-spots-bench}$ as a diagnostic stress test for identifying concrete weaknesses in current modern models.

Modern AI models often excel at complex tasks like coding or advanced mathematics, yet they frequently stumble on simple, everyday challenges—such as counting objects in an image or manipulating a specific string of text. The paper Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models introduces a new benchmark designed to expose these persistent weaknesses. By focusing on tasks that are trivial for humans but difficult for AI, the researchers provide a diagnostic "stress test" to identify where current state-of-the-art models still fail.

Building a Diagnostic Benchmark

To create this benchmark, the researchers collected 235 questions from students in a graduate-level AI course. These questions were specifically chosen because they represent problems that frontier AI models failed to solve correctly. The team cleaned and annotated these samples with structured reference solutions and organized them into a taxonomy of three main categories: object-centric tasks (like counting and spatial reasoning), abstract reasoning (logic and math), and language-and-knowledge tasks. This structure allows for a more granular understanding of model performance than traditional, broad-spectrum benchmarks.

The Evaluation Pipeline

The researchers developed an automated, end-to-end evaluation pipeline to test 38 different models, including language models, vision-language models, and image-generation systems. The process uses a "solver" model to attempt the questions and a "grader" model (Gemini-3-flash) to verify the accuracy of the responses against the reference solutions. To ensure the reliability of this automated grading, the team manually audited the results and found a high level of agreement between the AI grader and human judgment, confirming the pipeline's effectiveness.

Key Findings and Model Performance

The analysis revealed several notable trends in how modern AI performs:

  • Closed-source vs. Open-weight: Frontier closed-source models generally outperform open-weight models, sometimes by a margin of approximately 10%, even when both types of models perform similarly on standard benchmarks.

  • Cost-Effectiveness: While frontier models are often more accurate, some open-weight models (such as GLM-5.2 and DeepSeek-V4) offer better performance relative to their evaluation cost.

  • Task-Specific Strengths: No single model dominates across all categories. For instance, some models excel at arithmetic reasoning while others perform better at character-level manipulation.

  • Persistent Challenges: Certain tasks, particularly fine-grained visual perception like counting objects, remain difficult for all evaluated models, with accuracy rates often failing to exceed 60%.

Important Considerations

The researchers noted that scaling up a model's size within the same family does not always lead to consistent improvements in these specific "blind spot" tasks. Additionally, they observed that the use of external tools—while helpful for some operations—does not uniformly improve accuracy and can sometimes lead to lower performance. These results suggest that current benchmarks may be masking specific, fundamental weaknesses in AI reasoning, and that this new benchmark serves as a necessary tool for identifying and addressing these concrete gaps in capability.

Comments (0)

No comments yet

Be the first to share your thoughts!