BackTrend: Evaluating Scientific Weak-Signal Predic... | AI Research

What the paper is about

Scientific weak signals are early, low-visibility research directions that later become central to mature scientific topics, yet existing resources such as trend tracking, citation forecasting, and foresight reports rarely provide validated reference sets that link concrete early precursors to later paradigms. We introduce BackTrend, a retrospective benchmark in which, given a mature target topic and a temporal evidence constraint, systems must recover two types of precursors: problem-space signals, underrecognized research problems, and solution-space signals, emerging methods for known problems. BackTrend contains 25 mature target topics in artificial intelligence and machine learning and 66 human-validated weak signals, reconstructed from large-scale literature by grounding each candidate in its 2019-2024 publication-frequency trajectory. We evaluate frontier LLMs, RAG systems, and agentic research systems using semantic matching and coverage-based metrics. Current systems often generate plausible but misaligned precursors, exhibiting topic drift, granularity mismatch, near-miss matching, and incomplete coverage; the strongest system achieves only 10.1% F1, while Coverage10 reaches at most 18.5% of the reference signals. Our budget analyses show that additional retrieval and web-search evidence can improve performance up to a moderate budget, but does not by itself close the substantial performance gap.

What it covers

BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction Xiao Zhou 𝒀 \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} † † thanks: Equal contributions. Yilun Zhao 𝒀 \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} 1 1 footnotemark: 1 Owen Jiang 𝒀 \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} 1 1 footnotemark: 1 Tiansheng Hu 𝑵 \hskip 1.00006pt{}^{{\color[rgb]{0.3398,0.0234,0.5508}\boldsymbol{N}}} Affiliation: [5pt] Cai Xu Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Manasi Patwardhan 𝑻 \hskip 1.00006pt{}^{{\color[rgb]{0.0039,0.4922,0.7813}\boldsymbol{T}}} Arman Cohan 𝒀 \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Affiliation: [7pt] 𝒀 \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Yale NLP Lab 𝑵 \hskip 1.00006pt{}^{{\color[rgb]{0.3398,0.0234,0.5508}\boldsymbol{N}}} New York University 𝑻 \hskip 1.00006pt{}^{{\color[rgb]{0.0039,0.4922,0.7813}\boldsymbol{T}}} TCS Research Affiliation: [6pt] BackTrend Dataset BackTrend Code Abstract Scientific weak signals are early, low-visibility research directions that later become central to mature scientific topics, yet existing resources such as trend tracking, citation forecasting, and foresight reports rarely provide validated reference sets that link concrete early precursors to later paradigms. We introduce BackTrend, a retrospective benchmark in which, given a mature target topic and a temporal evidence constraint, systems must recover two types of precursors: problem-space signals , underrecognized research problems, and solution-space signals , emerging methods for known problems. BackTrend contains 25 mature target topics in artificial intelligence and machine learning and 66 human-validated weak signals, reconstructed from large-scale literature by grounding each candidate in its 2019–2024 publication-frequency trajectory. We evaluate frontier LLMs, RAG systems, and agentic research systems using semantic matching and coverage-based metrics. Current systems often generate plausible but misaligned precursors, exhibiting topic drift, granularity mismatch, near-miss matching, and incomplete coverage; the strongest system achieves only 10.1% F1, while Coverage 10 10 reaches at most 18.5% of the reference signals. Our budget analyses show that additional retrieval and web-search evidence can improve performance up to a moderate budget, but does not by itself close the substantial performance gap. 1 Introduction Figure 1: BackTrend overview for the 2024 mature target topic large language models . Top: the benchmark-construction pipeline (§ 3.2 ), in which candidate problem-space and solution-space precursors are mined from the topic’s 2019–2023 papers, scored by their 2019–2024 frequency trajectory, gated, and expert-validated. Lower-left: two of the topic’s eleven human-validated weak signals (the top-ranked problem- and solution-space precursor), each with its 2019–2024 frequency trajectory, which shows the low-then-rising signature that defines a weak signal. Lower-right: four real GPT-5.4 predictions, verbatim from the prediction outputs and produced under a pre-2024 evidence cutoff, each listed with its outcome and with the reference weak signal it was scored against. The full error taxonomy, which also covers granularity mismatch and coverage failure, is defined in § 5.2 . Identifying early indicators of transformative research, or weak signals, has long been a difficult goal across scientific disciplines ( Ansoff, 1975 ; Porter et al., 2004 ) . Detecting such signals is valuable for researchers, corporations, and policymakers because it informs long-term investment, research prioritization, and policy design ( Kajikawa et al., 2008 ; Yoon and Kim, 2012 ; Lee et al., 2014 ; Ogawa and Kajikawa, 2015 ) . As AI and machine learning accelerate the pace of scientific research, effective foresight depends on detecting low-visibility but potentially high-impact ideas within a large volume of background noise, and recent work continues to identify open issues in methods for detecting such emerging themes ( Ren and Zhao, 2021 ; Liu et al., 2023 ) . Across scientific domains, LLM-based research agents are increasingly deployed to search literature, synthesize emerging areas, and propose new methods ( Zhao et al., 2026a ; Hu et al., 2026 ; Yu et al., 2025 ) , and a growing body of benchmarks evaluates their scientific literature understanding and research capabilities ( Zhao et al., 2025a ; Xu et al., 2025 ; Chen et al., 2026a ; Zhao et al., 2026b ; Wang et al., 2025 ; Zhao et al., 2025b ; Chen et al., 2026b ) , yet they are rarely tested on whether they can recover low-visibility precursors that later become central, leaving an open challenge for long-context reasoning and abstraction control. We address this challenge with BackTrend , a retrospective benchmark for scientific weak-signal prediction ( Figure 1 ). Given a mature target topic and a temporally restricted evidence window, systems must recover early problem-space and solution-space weak signals that later became important precursors of that topic. Because prospective evaluation would require waiting years for outcomes to mature, BackTrend uses backward reconstruction as a controlled proxy for scientific foresight. We construct the reference set by reconstructing candidate precursors directly from large-scale Semantic Scholar literature: for each mature target topic, we mine candidate research directions from its 2019–2023 papers and score them by their 2019–2024 publication frequency, so that a candidate is retained only when a low-visibility early emergence is followed by a clear rise in its 2024 frequency. AI/ML researchers among the authors then validate every retained candidate for topical relevance, temporal consistency, and supporting evidence; at evaluation time, systems must recover the held-out signals using only evidence from publications available before the end-of-2023 prediction cutoff. Our results show that strong systems can produce plausible precursors but routinely miss the benchmark’s intended structure. Across all seven evaluated systems, semantic-judge F1 and Coverage K K remain low: F1 reaches at most 10.1% Coverage K K reaches at most 18.5%. The dominant errors are topic drift, granularity mismatch, lexical near-miss, and coverage failure. Additional evidence can improve performance, but not uniformly: deeper retrieval generally benefits both RAG models, with a stronger effect on Qwen3-8B, while DeepResearch performs best with a moderate search budget rather than unlimited search. Weak-signal prediction therefore tests semantic alignment, abstraction-level control, and coverage over concrete research precursors rather than fluency in generating scientific phrases. Our main contributions are summarized below:

• We formulate retrospective scientific weak-signal prediction : recovering early problem-space and solution-space precursors of mature research topics under a temporal evidence constraint that simulates research foresight.

• We construct BackTrend, a benchmark of 25 mature target topics in artificial intelligence and machine learning and 66 human-validated weak signals, reconstructed from large-scale literature by scoring each candidate’s 2019–2024 publication-frequency trajectory and verified by AI/ML researchers among the authors.

• We benchmark frontier LLMs, RAG systems, and DeepResearch agents on BackTrend, finding that current systems produce plausible but misaligned signals and that semantic-judge F1 reaches at most 10.1% for every system and Coverage@ 10 10 at most 18.5%, with errors dominated by topic drift, granularity mismatch, lexical near-miss, and coverage failure. 2 Related Work Benchmark Construction for Emerging Topics and Weak Signals. Existing benchmarks for emerging-topic analysis mainly evaluate whether systems can detect and track salient events or trends over time, rather than identify early scientific ideas that later develop into dominant paradigms ( Allan, 2002 ; Petrović et al., 2010 ; Deng et al., 2022 ) . In scientific domains, benchmark construction has also focused on trajectory labeling or future impact prediction, such as classifying topics as rising or declining or forecasting citations ( Prabhakaran et al., 2016 ; Moiseeva and Schütze, 2020 ; Ofer et al., 2024 ; Gu and Krenn, 2025 ; Ajith et al., 2026 ) . Foresight and horizon-scanning reports offer another related resource, but the signals they curate are usually broad thematic areas rather than concrete technical precursors ( Day and Schoemaker, 2005 ; Saritas, 2013 ; van Rij, 2010 ) . BackTrend addresses these limitations by constructing benchmark instances retrospectively from mature research topics, defining weak signals as empirically grounded early precursors rather than transient events, broad themes, or short-term impact patterns. Each instance decomposes into problem-space and solution-space precursors and is evaluated under a prediction cutoff that withholds post-cutoff information from the system. Weak Signal Detection and Emerging Trend Analysis. Weak-signal detection has long been studied in foresight, bibliometrics, and technology intelligence as the task of identifying faint early indicators of future change ( Ansoff, 1975 ; Hiltunen, 2008 ; Holopainen and Toivonen, 2012 ) . Early computational approaches relied on keyword frequencies, citation networks, and co-occurrence statistics, while later work adopted topic models and contextual embedding methods to better capture latent semantic structure and thematic evolution over time ( Small, 1973 ; Yoon, 2012 ; Song et al., 2018 ; Blei et al., 2003 ; Rudolph and Blei, 2018 ; Yao et al., 2018 ; Grootendorst, 2022 ; Boutaleb et al., 2024 ; Ebadi et al., 2026 ) . However, these methods are still typically evaluated through case studies or proxy trend metrics, because existing resources rarely provide verified examples of concrete early precursors to mature scientific themes. BackTrend addresses this gap with a precursor-recovery task featuring validated targets and a unified protocol. We benchmark LLM-based research agents and omit traditional bibliometric or topic-modeling systems because their outputs are not directly aligned with BackTrend ’s output format: given a mature topic, BackTrend requires an explicit, topic-conditioned set of precursor hypotheses, whereas these methods typically produce corpus-level trends, clusters, or topic trajectories. Adapting them would require an additional, non-standardized mapping from such outputs to target-specific precursor labels, potentially confounding direct comparison. We leave such adaptations to future work. Figure 2: Overview of the BackTrend construction pipeline. We first compile 25 mature target topics from the Artificial Intelligence and Machine Learning domain of a 2024 JRC report ( Eulaerts et al., 2025 ) and retrieve their 2019–2024 papers from Semantic Scholar. We then mine problem-space and solution-space candidate topics from the 2019–2023 abstracts, consolidate them by embedding clustering, and score each candidate by its 2019–2024 publication frequency, keeping only candidates that pass all four frequency gates. Finally, two AI/ML researchers among the authors validate the surviving candidates for topical relevance, correct problem/solution categorization, temporal consistency, and evidence faithfulness to produce the final BackTrend benchmark. 3 BackTrend Benchmark BackTrend is a retrospective weak-signal prediction benchmark in which systems recover early problem-space and solution-space precursors of mature research topics. Below we define the task and the two weak-signal categories (§ 3.1 ), detail the construction pipeline with expert validation (§ 3.2 ), and report per-topic statistics (§ 3.3 ). 3.1 BackTrend Task The BackTrend task frames scientific foresight as recovering a mature topic’s early, low-visibility precursors from evidence available before it matured, using backward reconstruction as a controlled proxy for prediction. We define a precursor as an early research direction that later becomes associated with or central to the mature topic, without implying a causal relationship. Mature Target Topics. Given a scientific domain, mature target topics serve as anchor concepts for weak-signal construction. Each mature target topic represents a research direction that has already achieved substantial visibility within its domain through sustained publication activity. For example, large language models constitute a mature target topic in 2024. In BackTrend, these topics are instantiated from externally validated AI and machine-learning topics in the 2024 JRC weak-signal report ( Eulaerts et al., 2025 ) ; we describe the selection process in § 3.2 . Problem-space and Solution-space Weak Signals. We define a weak signal as a concrete research direction that initially received limited attention but later grew exponentially in prominence. We distinguish two types: (1) problem-space weak signals are underrecognized research problems or problem formulations that later become central to the mature target topic in a given field; (2) solution-space weak signals are early or niche methods, techniques, or design principles that were not yet widely adopted but later became important solutions to already recognized problems. In short, problem-space signals surface new questions, while solution-space signals offer emerging answers to existing ones. Task Formulation. We formulate weak-signal discovery as the task of identifying historically grounded early research directions that later contribute to the emergence of a given mature target topic. Formally, given a domain-specific mature target topic M M , a maturity year y y , an evidence window t = [ y − k , y − 1 ] t=[y-k,y-1] , and a designated signal space (problem or solution), the model must retrieve or infer a set of candidate weak signals s = { s 1 , … , s n } s={s_{1},\dots,s_{n}} . A valid weak signal s i ∈ s s_{i}\in s must (i) exhibit low visibility during its early stage, (ii) demonstrate growth in prominence within the evidence window, and (iii) have this rise confirmed in the maturity-year corpus of M M , with its frequency in year y y exceeding its highest frequency within the evidence window by a factor of at least λ = 1.2 \lambda=1.2 (§ 3.2 ), where λ \lambda denotes the maturity-year frequency lift threshold. The year y − 1 y-1 is referred to as the prediction cutoff year : at evaluation time, systems are given the mature target topic M M but may only use evidence available up to the end of this year, simulating a research-foresight setting in which evidence from the maturity year about how M M emerged is unavailable to the model. In BackTrend, mature topics are drawn from the 2024 JRC report ( Eulaerts et al., 2025 ) , where y = 2024 y=2024 and k = 5 k=5 , resulting in an evidence window of 2019–2023 and a prediction cutoff year of 2023. 3.2 Benchmark Construction We next detail the BackTrend construction process, with an overview shown in Figure 2 . The pipeline is grounded in observed publication frequency: a candidate is retained only when a low-visibility, exponentially growing early emergence is followed by a rise in its frequency in the 2024 mature-topic corpus. Corpus Construction. To identify weak signals retrospectively, we begin with a set of research topics that have already reached maturity. We obtain these mature topics from the 2024 JRC weak-signal report ( Eulaerts et al., 2025 ) , which identifies science and technology topics through large-scale data analysis and expert validation. Specifically, we focus on topics within the Artificial Intelligence and Machine Learning domain and backtrack their historical development to uncover the weak signals that preceded them. This results in a benchmark comprising 25 mature topics spanning a diverse range of AI and machine learning research directions. For each mature target topic M M , we retrieve papers through the Semantic Scholar API using keyword queries derived from the topic name and its manually curated paraphrases, with one query per year from 2019 to 2024, to capture alternative terminology and improve retrieval coverage. These paraphrase queries expand corpus coverage and reduce the risk of missing historically relevant papers. The retrieved papers form the topic/paraphrase paper corpus, whose members we refer to as M M -papers. Papers from 2019–2023 serve as the historical corpus for candidate-topic discovery, while 2024 papers provide the later-year frequencies used to test whether an early candidate’s frequency subsequently rose. Candidate Topic Discovery. We next extract candidate topics from the 2019–2023 historical corpus. For each paper abstract retrieved for a mature target topic, we extract up to two reusable literature-level candidate topics that are explicitly grounded in the abstract and conceptually related to the target topic; the extraction prompts are shown in Figure 5 and Figure 6 . Each candidate is assigned to one of two spaces: problem-space candidates describe research problems, limitations, risks, gaps, evaluation failures, or scientific questions, whereas solution-space candidates describe reusable methods, system directions, defenses, benchmarks, datasets, or evaluation protocols. We require candidate labels to be neither overly broad field names nor paper-specific implementation details, and we avoid problem-solution phrases that conflate a method with the problem it addresses. The extracted candidates are then consolidated within each mature target topic and candidate type. We first exact-deduplicate normalized candidate strings while retaining their supporting source-paper IDs, years, evidence snippets, and mention counts. We then encode candidate-topic strings with text-embedding-3-large 1 1 1 https://platform.openai.com/docs/guides/embeddings and cluster semantically similar candidates using a cosine-similarity threshold of 0.85, chosen to balance semantic consolidation of related candidate topics with separation of distinct research directions. Clustering is performed separately for problem-space and solution-space candidates to avoid merging research problems with solution methods. For each cluster, we select a canonical candidate-topic label based on supporting evidence, prioritizing candidates with more source papers and mentions while favoring compact labels when support is comparable. The resulting clustered candidate topics are the units considered in the final weak-signal identification step. Weak Signal Identification and Validation. For each clustered candidate topic c c of mature target topic M M , we compute its yearly frequency over 2019–2024. Let n y ​ ( c ) n_{y}(c) be the number of non-survey source papers supporting c c in year y y , and N y ​ ( M ) N_{y}(M) the number of non-survey papers retrieved for M M in year y y ; the frequency is f y ​ ( c , M ) = n y ​ ( c ) N y ​ ( M ) . f_{y}(c;M)=\frac{n_{y}(c)}{N_{y}(M)}. We exclude survey papers from both counts: a single survey touches many topics at once and would spuriously inflate a candidate’s frequency, so excluding surveys reduces false-positive matches. However, we retain them during candidate discovery and clustering because surveys provide broad coverage of research directions. We further validate this design choice in Appendix A.4 . For 2019–2023, a paper supports c c when its abstract matches c c ’s cluster. For 2024, we instead count a paper as supporting c c when it cites at least one of c c ’s early 2019–2023 source papers: f 2024 ​ ( c , M ) = r 2024 ​ ( c ) N 2024 ​ ( M ) , f_{2024}(c;M)=\frac{r_{2024}(c)}{N_{2024}(M)}, where r 2024 ​ ( c ) r_{2024}(c) is the number of such non-survey 2024 M M -papers. We measure the 2024 frequency through citations rather than by re-matching c c ’s wording because a research direction is often renamed or rephrased as it matures, and a direct text or embedding match in 2024 would miss these drifted mentions; a citation to the candidate’s own early papers tracks the same line of work however it is now phrased. A candidate is selected only if it passes four gates g 1 g_{1} – g 4 g_{4} . Let τ c ∈ { 2019 , … , 2022 } \tau_{c}\in{2019,\dots,2022} be its onset year, selected as the maximizer of the growth score below, and ϵ \epsilon a small constant. The gates use four hyperparameters: a 2024-frequency lift λ = 1.2 \lambda=1.2 , a year-to-year retention ρ = 0.8 \rho=0.8 , a pre-onset tolerance δ = 0.6 \delta=0.6 , and a sparse-year skip κ = 1 \kappa=1 . We first define the auxiliary quantities F max \displaystyle F_{\max} = max 2019 ≤ y ≤ 2023 ⁡ f y ​ ( c , M ) , \displaystyle=\textstyle\max_{2019\leq y\leq 2023},f_{y}(c;M), F τ \displaystyle F_{\tau} = max τ c ≤ y ≤ 2023 ⁡ f y ​ ( c , M ) , \displaystyle=\textstyle\max_{\tau_{c}\leq y\leq 2023},f_{y}(c;M), F 0 ] , \displaystyle=\mathbf{1}[\mathrm{score}(c)>0], g 2 ​ ( c ) \displaystyle g_{2}(c) = [ f 2024 ( c ; M ) + ϵ ≥ λ F max ] , \displaystyle=\mathbf{1}!\left[f_{2024}(c;M)+\epsilon\geq\lambda F_{\max}\right], g 3 d ​ ( c ) \displaystyle g_{3}^{d}(c) = { 𝟏 [ f 2023 ( c ; M ) + ϵ ≥ ρ F τ ] , d = prob , [ ∏ y = τ c 2022 A y ( c ) = 1 ] , d = sol , \displaystyle=\begin{cases}\mathbf{1}[f_{2023}(c;M)+\epsilon\geq\rho F_{\tau}],&d=\mathrm{prob},\ \mathbf{1}!\left[\prod_{y=\tau_{c}}^{2022}A_{y}(c)=1\right],&d=\mathrm{sol},\end{cases} g 4 ​ ( c ) \displaystyle g_{4}(c) = [ F < τ ≤ max ( f τ c ( c ; M ) , δ f 2023 ( c ; M ) ) ] . \displaystyle=\mathbf{1}!\left[F_{<\tau}\leq\max!\left(f_{\tau_{c}}(c;M),\delta f_{2023}(c;M)\right)\right]. The four gates operationalize the three clauses of the definition in § 3.1 , each ruling out one failure mode. g 1 g_{1} requires the onset-window trajectory to fit a rising exponential, ruling out flat, declining, and single-point traces. g 2 g_{2} requires the rise to be confirmed in the maturity year, ruling out candidates whose 2024 frequency never clears their own early peak. g 3 g_{3} requires the growth to persist to the cutoff rather than collapse along the way, ruling out candidates that spike early and then fade. g 4 g_{4} requires the candidate to have been faint before its onset year, ruling out directions that were already established when they began to grow. Together, g 1 g_{1} , g 3 g_{3} , and g 4 g_{4} characterize the observable early-stage properties of a weak signal, while g 2 g_{2} validates whether the signal eventually contributes to the mature topic. A candidate is retained when g 1 ​ ( c ) ​ g 2 ​ ( c ) ​ g 3 d ​ ( c ) ​ g 4 ​ ( c ) = 1 g_{1}(c)g_{2}(c)g^{d}{3}(c)g{4}(c)=1 , with d = prob d=\mathrm{prob} for problem-space candidates and d = sol d=\mathrm{sol} for solution-space candidates. After this automatic filtering step, two AI/ML researchers among the authors conduct human validation: they inspect the candidate label, source papers, its 2024 citation evidence, and frequency trajectory, then verify topical relevance, correct problem/solution categorization, temporal consistency, and evidence faithfulness. Both experts independently label all 125 gate-passing candidates as valid weak signals or not, with an inter-annotator agreement of Cohen’s κ = 0.75 \kappa=0.75 . Candidates that fail validation are revised or removed through consensus adjudication, yielding the final 66 signals. Appendix A.2 traces a single candidate, retrieval-augmented language models under the mature target topic large language models , through every stage of the pipeline as a running example. 3.3 Data Statistics BackTrend covers 25 mature target topics and 66 human-validated weak signals (34 problem-space and 32 solution-space), all within the Artificial Intelligence and Machine Learning domain of the 2024 JRC report. Per-topic statistics are reported in Table 1 : the 66 signals are distributed over 18 of the 25 topics, while the remaining seven yield no candidate that survives both the four gates of § 3.2 and expert validation; § A.7 traces each empty set to an identifiable property of the topic’s literature. At evaluation time, systems are asked to recover the weak signals associated with each mature target topic under an end-of-2023 temporal search cut-off, using only historically available information, simulating prospective foresight. Mature Target Topic The same ai evaluation question is explored in Q&A on Any Spreadsheet Requires Interpreting..., which adds a research perspective.
# Papers
# P-WS
# S-WS
Mature Target Topic
# Papers
# P-WS
# S-WS
Artificial Intelligence of Things 25.8K 1 0 Machine Unlearning 1.9K 2 2 Asynchronous Federated Learning 3.0K 2 1 Masked Face Recognition 1.9K 1 1 Attention Mechanisms in CNN 29.5K 3 5 Masked Language Model 5.4K 0 3 Decentralized Federated Learning 4.3K 1 1 Multimodal AI 8.6K 0 0 Epistemic AI 1.2K 0 0 Multimodal Hate Speech 0.1K 0 0 Evolutionary Neural Arch. Search 1.1K 1 0 Privacy-Preserving Machine Learning 11.3K 1 0 Explainable AI 18.8K 2 1 Scientific Machine Learning 35.4K 0 1 Federated Deep Learning 5.5K 0 0 Self-Supervised CNN 2.6K 0 0 Federated Machine Learning 6.8K 4 6 Tiny Machine Learning 3.6K 1 0 Federated Reinforcement Learning 1.6K 0 0 Trustworthy AI 29.1K 4 2 Human–AI Interface 22.4K 1 1 Trustworthy Machine Learning 26.6K 0 0 Human-Centric AI 7.2K 1 1 Vertical Federated Learning 1.0K 4 1 Large Language Models 22.8K 5 6 Total (grand total across all 25 topics) 277.7K 34 32 Table 1: Data statistics of BackTrend. All 25 mature target topics fall in the Artificial Intelligence and Machine Learning domain. The same ai evaluation question is explored in How Good Are Frontier Models at..., which adds a research perspective.
# Papers : unique non-empty-abstract papers retrieved for the topic and its
paraphrases over 2019–2024 (thousands; entries are rounded to 0.1K and need not sum exactly to the total).
# P-WS / # S-WS : number of human-validated problem- /
solution-space weak signals. Seven topics yield no validated signal: five produce no gate-passing candidate, and for two, expert validation removes every gate-passing survivor (§ A.7 ). 4 Experiment Setup We next discuss the evaluated systems and our automated and human evaluation protocols. 4.1 Evaluated Systems We evaluate three categories of weak-signal prediction systems. All systems operate under the end-of-2023 prediction cutoff defined in § 3.1 , enforced through prompt instructions for parametric systems and through hard filtering of retrieval results for systems that can search external corpora at inference time. Frontier LLMs. We include GPT-5.4, Qwen3.5-397B-A17B (Qwen3.5-397B) ( Qwen Team, 2026 ) , and DeepSeek-R1-0528 ( Guo et al., 2025 ) . Temporal consistency is enforced via explicit prompt instructions specifying the emergence time of weak signals. Retrieval-Augmented LLMs (RAG). We evaluate Qwen3-8B and Qwen3-30B ( Qwen Team, 2025 ) with retrieval augmentation, which are equipped with retrieval mechanisms to better ground predictions in evidence from historical scientific literature. For each mature target topic, we retrieve a set of candidate papers from Semantic Scholar using the topic as a query. The retrieved papers are encoded using the bge-base-en-v1.5 embedding model ( Xiao et al., 2024 ) and ranked by cosine similarity. The top 50 most relevant papers are then selected and provided as external context to support weak-signal prediction. To ensure temporal consistency, retrieved papers are filtered on the client side to include only those published on or before the end of 2023. We deliberately keep this retrieval stage simple; stronger instruction-following or reasoning-intensive retrievers and rerankers ( Song et al., 2025a ; Song et al., 2025b ; Zhao et al., 2026a ) , as well as alternative ways of constructing retrieval queries when the mature target topic is unavailable, could be explored in future work. DeepResearch Systems. We evaluate two agentic LLM systems, DR-Tulu-8B ( Shao et al., 2025 ) and Tongyi-DeepResearch-30B-A3B (Tongyi-DR-30B-A3B) ( Tongyi DeepResearch Team et al., 2026 ) , which integrate multi-step reasoning with tool-augmented academic retrieval and evidence collection. Tongyi-DR-30B-A3B follows a ReAct-style framework, where the model iteratively performs reasoning and tool calls to query Semantic Scholar and refine its predictions. In contrast, DR-Tulu adopts a workflow-based agentic search pipeline that orchestrates model inference and retrieval through a modular multi-service architecture. In both systems, given a mature target topic, the model retrieves relevant papers and generates weak signals grounded in the collected academic evidence. To ensure temporal consistency, retrieval is restricted to papers published on or before the end of 2023, preventing information leakage from future publications during the multi-step search process. Model Set LLM Signal LLM Cov@10 Human Problem Solution All Problem Solution All Problem Solution All Frontier LLMs GPT-5.4 7.4 3.6 5.7 7.0 1.6 4.5 14.3 14.5 14.4 6.0 Qwen3.5-397B 6.5 4.0 5.3 4.7 1.1 3.0 7.5 9.5 8.4 5.3 DeepSeek-R1-0528 14.1 5.4 10.1 9.8 4.1 7.1 23.2 13.1 18.5 9.0 Retrieva The same large language models question is explored in FlashVector, which adds a research perspective. as detailed in the full paper on Arxiv