Evaluating Investment Logic in Large Language Model...

Evaluating Investment Logic in Large Language Model... | AI Research

Key Takeaways

Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents introduces a new benchmark, InvestLogicBen...
Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries.
Yet financial LLMs are evaluated either by static question answering or by terminal profit and loss.
The former omits agency; the latter cannot reveal whether a profitable action was grounded, profile-consistent, or merely lucky.
We ask whether the community is using the wrong ruler for consequential agents.

Paper AbstractExpand

Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries. Yet financial LLMs are evaluated either by static question answering or by terminal profit and loss. The former omits agency; the latter cannot reveal whether a profitable action was grounded, profile-consistent, or merely lucky. We ask whether the community is using the wrong ruler for consequential agents. We introduce \textsc{InvestLogicBench}, a process-native benchmark containing 201,247 documented decisions from 151 real-world investors. Each episode instantiates a \textbf{P$\rightarrow$E$\rightarrow$R$\rightarrow$D$\rightarrow$O} trace: investor \textit{Profile}, observable market \textit{Events}, investment \textit{Reasoning}, executable \textit{Decision}, and delayed \textit{Outcome}. The release includes profile construction, point-in-time event binding, structured logic, horizons, outcomes, and post-mortems, and supports comprehension, profile-conditioned generation, and end-to-end replay. Across four leading LLMs, logical plausibility remains near 4/5 while event grounding is only 0.8--2.8/5; return and process quality also disagree. These results expose polished but weakly grounded reasoning that outcome-only evaluation hides. We further argue that P$\rightarrow$E$\rightarrow$R$\rightarrow$D$\rightarrow$O should be a data-system interface, requiring versioned profiles, temporal provenance, inspectable retrieval, decision ledgers, and replayable outcomes. Finance is our stress test for a broader class of personalized, consequential agents.

Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents introduces a new benchmark, InvestLogicBench2026, to measure how Large Language Models (LLMs) perform in financial decision-making. The authors argue that current evaluation methods rely too heavily on static question-answering or simple profit-and-loss metrics, which fail to capture whether an AI agent’s investment decisions are based on sound, personalized logic or mere chance.

The P→E→R→D→O Framework

The researchers propose that financial competence requires an integrated chain of reasoning: Person (P), which includes an investor's unique goals and risk profile; Events (E), the market data; Reasoning (R), the synthesis of that data; Decision (D), the resulting action; and Outcome (O), the final result. InvestLogicBench2026 is designed to evaluate this entire chain. It contains 201,247 documented decisions from 151 real-world investors, including fund managers and key opinion leaders, to provide a standard for testing whether an LLM can replicate expert-level investment logic.

The Capabilities-Performance Paradox

The authors conducted a seven-week live U.S. equity trading competition to test LLM-based agents. They observed a "Capabilities-Performance Paradox," where models with high scores on general reasoning benchmarks (like GPT-5 and Claude-Sonnet-4.5) often underperformed in live markets compared to models with lower general scores. The researchers attribute this to two primary cognitive failures: "Consensus Bias," where models prioritize training data averages over independent analysis, and "Reasoning Fragmentation," where models struggle to maintain a coherent strategy when faced with complex, non-stationary market data.

Evaluating Logical Plausibility

When tested on the benchmark, leading LLMs showed a significant gap between their ability to generate profitable outcomes and their ability to provide grounded reasoning. While some models achieved high profitability scores, their "Event Coherence" scores—which measure how well they connect their decisions to specific market events—remained low (ranging from 0.8 to 2.8 out of 5). This suggests that even when models make profitable trades, they often lack the underlying logical foundation that defines professional investment management.

Limitations and Future Directions

The authors conclude that current evaluation paradigms are insufficient for consequential agents. They argue that the industry needs to move toward a data-system interface that includes versioned profiles, temporal provenance, and inspectable decision ledgers. By shifting the focus from passive knowledge recall to active, personalized reasoning, the researchers aim to provide a diagnostic tool that helps developers build more reliable, market-aligned financial agents.