Back to AI Research

AI Research

V-FiLLM: Verified Financial LLM Reasoning Benchmark | AI Research

Key Takeaways

  • V-FiLLM is a framework designed to evaluate how well Large Language Models (LLMs) perform financial reasoning over structured data.
  • While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored.
  • We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction.
  • V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size.
  • This allows for the scalable testing of an LLM's ability to handle complex, multi-step financial calculations.
Paper AbstractExpand

While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.

V-FiLLM is a framework designed to evaluate how well Large Language Models (LLMs) perform financial reasoning over structured data. By using a deterministic pipeline to generate questions from synthetic financial spreadsheets, the researchers created a benchmark that provides ground-truth answers without the need for human annotation or model-based labeling. This allows for the scalable testing of an LLM's ability to handle complex, multi-step financial calculations.

How the Benchmark Works

The V-FiLLM pipeline generates questions by sampling "computation trees" grounded in synthetic financial tables. Each tree represents a specific mathematical operation—such as addition, subtraction, or growth rates—applied to data points like revenue or assets. Because the answers are derived directly from these executable trees, they are correct by construction.
The framework allows researchers to control four specific axes of difficulty:

  • Computation depth: The number of steps required to reach an answer.

  • Expression breadth: The number of sub-results that must be computed simultaneously.

  • Financial concept complexity: The use of derived concepts like gross profit or operating income.

  • Context size: The amount of data provided to the model.
    The researchers also applied linguistic augmentations to these questions, such as varying phrasing and introducing minor typos, to test how robust models are to superficial changes in input.

Evaluating Model Performance

The authors evaluated several open-source models using this framework. They found that model accuracy declines as the reasoning depth increases. Specifically, accuracy fell by up to 51% as the number of required reasoning steps grew. Furthermore, when the researchers introduced adversarial numerical perturbations—such as replacing digits with visually similar characters to simulate OCR errors—model accuracy dropped by up to 47 percentage points.
The study also tested the impact of Low-Rank Adaptation (LoRA) fine-tuning. By training models on verified chain-of-thought traces, the researchers improved accuracy on held-out problems from 81.1% to 85.6%. Additionally, this fine-tuning approach outperformed the base model by 5 percentage points on the existing FinQA dataset.

Multi-Turn Reasoning

V-FiLLM includes a multi-turn extension that breaks complex, single-turn questions into sequential sub-queries. The researchers observed that this approach significantly mitigates performance degradation for models that struggle with deep reasoning. For example, the Qwen3.5-9B model improved from 55.2% to 86.8% accuracy when the task was restructured into a multi-turn dialogue. This suggests that breaking down complex financial problems into smaller, explicit steps can help models manage the computational load required for accurate financial analysis.

Limitations and Considerations

The authors note that while their synthetic data allows for controlled testing, real-world financial documents present additional challenges. Real 10-Q filings contain messy layouts, footnotes, and mixed units that are not fully captured by the simplified synthetic sheets. The study found that models performed roughly 20 percentage points worse on real 10-Q data compared to simplified statements when evaluating the same number of reasoning steps. This indicates that while V-FiLLM provides a reproducible way to measure compositional reasoning, the gap between synthetic benchmarks and real-world document processing remains a significant hurdle for robust financial AI.

Comments (0)

No comments yet

Be the first to share your thoughts!