Measurement-Driven Sub-Network Selection for On-Pre... | AI Research

Measurement-Driven Sub-Network Selection for On-Premise Retrieval-Augmented Factory Agents
Factory workers often need quick access to technical machine documentation, but the large language models (LLMs) capable of providing this information are usually too big to run on standard shop-floor hardware. This paper introduces a pipeline to compress these models and deploy them on resource-constrained edge devices without sacrificing the quality of the answers provided. By using a "train-once, select-per-device" approach, the researchers ensure that each piece of hardware runs the most efficient version of the model possible, keeping sensitive data on-site while maintaining high performance.

The Challenge of Model Size

A key finding in this research is that model size is not a perfect indicator of how well an AI will perform once it is adapted for a specific task. While a model's general knowledge tends to drop as you remove parameters, its ability to answer questions based on specific factory documents does not decline at the same rate. Because of this, simply picking the largest model that fits in memory is not necessarily the best strategy. The researchers discovered that they need to evaluate models based on actual performance—measuring both the quality of the answers and the speed of the device—rather than relying on simple proxies like parameter count.

A Two-Stage Compression Pipeline

To solve the footprint problem, the team developed a multi-stage process. First, they train a "supernetwork" that contains many different sub-networks within a single set of shared weights. This is done using a technique called sandwich-style in-place distillation, which allows smaller versions of the model to learn from the larger, more capable version during training. Once this supernetwork is trained, the researchers extract specific sub-networks and further specialize them using retrieval-grounded distillation, which teaches the model to focus on factory-specific manuals and ignore irrelevant information.

Hardware-Aware Selection

Because different factory devices have different memory and processing limits, the researchers created a selection process that happens after the model has been adapted. They define a "general-capability floor" to ensure that any chosen sub-network remains smart enough to handle basic tasks. By benchmarking three specific "anchor" versions of the model on a new device, they can predict how all other versions will perform. This allows them to automatically select the best sub-network for any given piece of hardware, balancing the need for high-quality answers with the physical constraints of the machine.

Practical Results

In a case study involving a manufacturing manual, this pipeline successfully recovered two-thirds of the quality lost during the initial compression process. The resulting assistants were deployed across three different tiers of edge hardware, ranging from powerful GPUs to smaller embedded controllers. These models were able to operate efficiently, with standby power consumption as low as 1.3 to 5 watts, proving that sophisticated, document-grounded AI can be run locally on the shop floor without needing to offload data to the cloud.