Back to AI Research

AI Research

Schema-Guided Hierarchical Information Extraction a... | AI Research

Key Takeaways

  • Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI presents a framework for extracting complex, nested data from u...
  • Information extraction from a document is performed in a single call to the model, in zero-shot mode.
  • In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard.
  • We were able to extract 12 out of 14 attributes with an F1 score of $>$90\% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3.
  • The time needed to extract the attributes from a document was $\sim$30 times lower than the time taken by a human domain expert.
Paper AbstractExpand

We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of $>$90\% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was $\sim$30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.

Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI presents a framework for extracting complex, nested data from unstructured documents and automatically evaluating the accuracy of that data against a gold standard. The authors developed this system to address the challenges of processing Health Technology Assessment (HTA) reports, which contain specialized, hierarchical information—such as drug indications and reimbursement recommendations—that vary in structure and length across different organizations.

The Schema-Based Framework

The core of the approach is a JSON schema that acts as a unified information model. This schema encodes domain knowledge by defining the attributes to be extracted, their hierarchical relationships, and constraints such as data types, patterns, and allowed values (enums).
By providing this schema in the prompt, the researchers guide a generative AI model to perform the extraction in a single, zero-shot API call. This method replaces manual extraction by domain experts, who previously had to interpret and record these complex attributes by hand. The schema also provides the necessary context for the evaluation phase, ensuring that the extracted output remains consistent with the required structure.

Automated Semantic Evaluation

Because the extracted data is hierarchical and of variable length, comparing it to a gold standard is not a simple task. The authors introduced a path-based matching algorithm that traverses the nested structure of the JSON objects to align corresponding elements.
When an exact string match is not found, the framework uses a second generative AI model to perform a semantic comparison. This comparison classifies results into four categories:

  • Exact match: Minor formatting differences (e.g., different dash types).

  • Semantic match: Lexically different but identical in meaning (e.g., "fever in kids" vs. "pyrexia in children").

  • Useful match: Information that is not identical but remains valuable for HTA research (e.g., "fever" vs. "fever in children under 12").

  • Non-match: No similarity found.

Performance and Efficiency

In tests using documents from the National Institute for Health and Care Excellence (NICE), the researchers extracted 12 out of 14 attributes with an F1 score exceeding 90%. The process proved significantly faster than manual extraction, requiring approximately 30 times less time than a human domain expert.
The authors report that the framework is generalizable across different generative AI models and transferable to HTA documents from different organizations and in different languages.

Considerations for Implementation

The researchers note that the effectiveness of this framework relies on the quality of the schema and the ability of the chosen generative AI model to follow instructions. While the study achieved high accuracy, the process requires a human-generated gold standard for the initial evaluation phase. Additionally, the authors emphasize that the "useful match" category is specifically tailored to the needs of HTA research, where the goal is to support reimbursement decisions rather than clinical practice, suggesting that the definition of a "successful" extraction may vary depending on the specific domain requirements.

Comments (0)

No comments yet

Be the first to share your thoughts!