Back to AI Research

AI Research

KnowHal: A Knowledge-Driven Benchmark for Comprehen... | AI Research

Key Takeaways

  • KnowHal is a benchmark designed to evaluate how Multimodal Large Language Models (MLLMs) handle hallucinations across four distinct dimensions: entity, attri...
  • Hallucination remains a critical challenge for developing trustworthy Multimodal Large Language Models (MLLMs).
  • KnowHal constructs paired positive and negative questions over shared images and entities, enabling controlled comparisons among perceptual errors, knowledge-related errors, and false-premise acceptance.
  • The benchmark contains 1,800 samples across 10 domains and 50 categories, constructed through a semi-automated pipeline combining LLM assistance, CLIP-based filtering, and human verification.
  • We evaluate 14 representative MLLMs on KnowHal and conduct extensive analyses.
Paper AbstractExpand

Hallucination remains a critical challenge for developing trustworthy Multimodal Large Language Models (MLLMs). While existing benchmarks mainly focus on entity, attribute, and relation hallucinations, knowledge-related failures are often investigated separately, lacking a unified evaluation framework across different hallucination dimensions. To overcome this, we propose \textbf{KnowHal}, a benchmark that explicitly incorporates knowledge hallucination into multimodal hallucination evaluation spanning four dimensions: entity, attribute, relation, and knowledge. KnowHal constructs paired positive and negative questions over shared images and entities, enabling controlled comparisons among perceptual errors, knowledge-related errors, and false-premise acceptance. The benchmark contains 1,800 samples across 10 domains and 50 categories, constructed through a semi-automated pipeline combining LLM assistance, CLIP-based filtering, and human verification. We evaluate 14 representative MLLMs on KnowHal and conduct extensive analyses. Results show that the knowledge dimension consistently presents the greatest challenge for nearly all evaluated models, while most models exhibit substantial performance degradation on negative questions, revealing limited robustness to false premises. By unifying four hallucination dimensions with paired question design, KnowHal addresses an important gap in existing evaluation frameworks and enables a more comprehensive assessment of hallucinations in MLLMs.

KnowHal is a benchmark designed to evaluate how Multimodal Large Language Models (MLLMs) handle hallucinations across four distinct dimensions: entity, attribute, relation, and knowledge. By using a unified framework that pairs positive and negative questions for the same images, the researchers aim to measure both a model’s ability to identify correct information and its tendency to accept false premises.

A Unified Evaluation Framework

Existing benchmarks often treat perceptual errors (what a model sees) and knowledge-level failures (what a model knows) as separate issues. KnowHal integrates these by evaluating four dimensions:

  • Entity: Identifying objects present in an image.

  • Attribute: Assessing fine-grained visual properties like color or shape.

  • Relation: Evaluating spatial or behavioral interactions between objects.

  • Knowledge: Testing factual accuracy regarding entities that requires information beyond the visual input.
    The researchers constructed 1,800 samples across 10 domains and 50 categories. Each sample includes paired questions: positive questions query valid evidence, while negative questions introduce misleading cues—such as substituting objects or reversing spatial relationships—to test the model’s robustness.

Construction and Validation

The benchmark was built using a semi-automated pipeline. First, the team identified target entities and retrieved images using structured queries. They employed CLIP to filter images for semantic consistency. Finally, they used a combination of LLM-assisted generation and rigorous human verification to ensure the quality of the QA pairs. Three annotators reviewed the samples, achieving high inter-annotator agreement (Fleiss’ κ = 0.854), and a reconstruction audit found that 90.6% of the samples required no revisions.

Performance of MLLMs

The authors evaluated 14 representative MLLMs, including both closed-source and open-source models, in a zero-shot setting. Key findings include:

  • Knowledge is the hardest dimension: Nearly all models performed worst on the knowledge dimension compared to the other three.

  • Vulnerability to false premises: Most models showed a significant drop in accuracy when answering negative questions compared to positive ones. This suggests that even models with high factual accuracy often struggle to reject misleading or false information.

  • Closed-source vs. Open-source: Closed-source models generally achieved higher positive accuracy, though the Qwen series of open-source models remained competitive in maintaining robustness against negative questions.

  • Scaling effects: For the Qwen3-VL and InternVL3.5 model families, increasing model size consistently improved both positive and negative question accuracy.

Franklin Analysis

The data suggests that high performance on standard positive-question benchmarks does not guarantee a model’s reliability. The Hallucination Proneness Score (HPS) developed by the authors—which measures the relative performance gap between positive and negative questions—reveals that models like LLaVA-OneVision-1.5-8B-Instruct and MiniCPM-V-4.5 exhibit higher susceptibility to misleading inputs. This indicates that current MLLM development may be prioritizing the retrieval of correct information over the critical ability to identify and reject false, hallucination-inducing premises.

Comments (0)

No comments yet

Be the first to share your thoughts!