When Does Bigger Help? A Controlled Study of LLM Sc... | AI Research

When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning investigates whether increasing the size of Large Language Models (LLMs) systematically improves their ability to perform ontology learning (OL) tasks. The authors, Hamed Babaei Giglou, Sören Auer, and Jennifer D’Souza, address the lack of empirical evidence regarding how parameter count, architecture, and model lineage influence performance in specialized knowledge engineering.

Evaluating LLM Scale in Ontology Learning

The researchers conducted a controlled study using the OntoLearner pipeline to evaluate 13 models, including variants from the Qwen3.5 and Qwen3.6 lineages and proprietary GPT releases. By keeping variables such as the embedding model, retrieval configuration, prompt templates, and datasets constant, the study isolates the effect of model scale on three core tasks: term typing, taxonomy discovery, and non-taxonomic relationship extraction. The experiments covered four ontologies across biomedical and materials science domains.

Key Findings on Model Performance

The study reveals that model size is not a reliable indicator of performance for ontology learning. Within the dense Qwen3.5 lineage, increasing parameter count primarily improves precision rather than recall, with the most significant gains observed between 9B and 27B parameters. However, these improvements are neither monotonic nor uniform across different tasks.
The researchers found that architecture often outweighs nominal parameter count. For example, dense 27B models outperformed larger sparse Mixture-of-Experts (MoE) models in term typing, while larger MoE models achieved the best results for taxonomy discovery. Furthermore, performance differences between matched Qwen variants and proprietary GPT releases suggest that model lineage and architecture are critical factors that can supersede the benefits of simply increasing the number of parameters.

Challenges in Ontology Engineering

The study identifies non-taxonomic relationship extraction as a persistent difficulty across all model scales, particularly within the Materials Data Science ontology. The authors conclude that model size alone is an insufficient criterion for selecting an LLM for ontology engineering. Instead, the results suggest that practitioners should consider the specific task requirements and the underlying architecture of the model rather than assuming that larger models will inherently provide better results.

Implications for Reproducible Research

This research provides empirical guidance for the development of reproducible LLM-assisted ontology engineering. By demonstrating that scaling effects are task-dependent and influenced by architectural choices, the authors highlight the need for more nuanced model selection strategies. The study emphasizes that because ontology learning tasks impose different semantic and structural demands, the relationship between model capacity and performance remains complex and requires careful evaluation rather than reliance on general-purpose scaling laws.