Heretic

Heretic is a specialized toolkit designed for the customization, ablation, and evaluation of dense and MoE/hybrid Large Language Models (LLMs).

What Heretic does

Heretic provides a framework for researchers and developers to modify and analyze the internal structures of LLMs. By facilitating ablation studies and structural customization, the toolkit allows users to investigate the performance and behavior of both dense models and Mixture-of-Experts (MoE) or hybrid architectures. It serves as a technical utility for those looking to perform rigorous evaluations of model components.

Who it helps

This tool is intended for AI researchers, machine learning engineers, and developers working on the architecture of large-scale language models. It is particularly useful for those focused on model interpretability, structural optimization, and the comparative analysis of different LLM design paradigms.

Notable capabilities

• Customization of dense and MoE/hybrid LLM architectures.
• Ablation of model components to study structural impact.
• Evaluation of model performance and behavior.