pro-team at LLMs4OL 2026 Tasks Flagship and Reuse:... | AI Research

The paper "pro-team at LLMs4OL 2026 Tasks Flagship and Reuse: Retrieval-Augmented Generation and Vocabulary-Constrained Filtering for Ontology Learning" presents a framework designed to automate the construction and extension of ontologies from unstructured text. By combining retrieval-augmented generation with strict filtering, the authors aim to reduce common Large Language Model (LLM) issues such as hallucinations, inconsistent formatting, and the tendency to favor hierarchical relationships over associative ones.

The Approach

The researchers developed an offline pipeline using the Qwen2.5-14B-Instruct model. To provide the model with relevant context without fine-tuning, they use an offline retrieval system powered by the all-MiniLM-L6-v2 model. This system identifies and retrieves semantically similar training examples to guide the LLM during inference.
To manage long prompts, the team implemented a left-truncated context-windowing strategy. This ensures that essential task instructions remain at the end of the prompt, preventing them from being cut off when processing large amounts of data. For the Reuse Task (Task B), the authors added a deterministic vocabulary-constrained filter. This step checks generated triples against a predefined set of terms and types, discarding any that do not match the expected vocabulary to ensure the output remains consistent with the existing ontology.

Performance Results

The framework was evaluated on the LLMs4OL 2026 benchmark, which consists of two primary tracks: the End-to-End Flagship Task (Task A) and the Ontology Extension Reuse Task (Task B).
For Task B, the system achieved a Semantic Graph Similarity of 0.8692, a Term-Typing F1 score of 0.9200, and a Taxonomy Discovery F1 score of 0.8540. For Task A, the system achieved a Semantic Graph Similarity of 0.7416.

Limitations and Observations

Despite the performance on taxonomy-related tasks, the researchers noted a significant limitation: the system failed to extract any non-taxonomic relations. The authors attribute this to the use of closed, taxonomy-oriented relation vocabularies, which restrict the model's ability to identify more complex, non-hierarchical semantic connections.
Franklin analysis: The reliance on a closed-world assumption for Task B—while effective for maintaining consistency and reducing hallucinations—appears to be the primary factor in the system's inability to capture non-taxonomic relationships. The evidence suggests that while vocabulary-constrained filtering improves precision for known types, it simultaneously limits the model's flexibility in discovering broader semantic associations that fall outside the predefined schema.