Back to AI Research

AI Research

SPADE: Speculative Decoding for Precise and Low Cos... | AI Research

Key Takeaways

  • SPADE is a distributed inference framework designed to reduce the computational costs and latency of running Large Language Models (LLMs) by splitting the wo...
  • Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands.
  • Deploying smaller LLMs directly on the edge can circumvent this, but with degraded accuracy.
  • Deploying smaller cloud-based big LLMs preserves performance, but at the cost of expensive per-token computation.
  • We present a distributed inference framework, \our{}, that integrates speculative decoding (SD) across edge and cloud.
Paper AbstractExpand

Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands. Deploying smaller LLMs directly on the edge can circumvent this, but with degraded accuracy. Deploying smaller cloud-based big LLMs preserves performance, but at the cost of expensive per-token computation. We present a distributed inference framework, \our{}, that integrates speculative decoding (SD) across edge and cloud. A compact draft model deployed on the edge generates candidate tokens rapidly, and a large verifier model on the cloud validates these tokens in parallel. Accepted tokens are retained, while only rejections trigger verifier correction, substantially reducing the number of cloud queries. Our plug-and-play design shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement. Our approach demonstrates a practical path toward scalable, cost-efficient, and accurate deployment of LLMs in real-world environments. Experimental results across multiple Natural Language Processing tasks using SpecBench and CNN/Dailymail datasets demonstrate that \our{} reduces the cloud model calls by $76\%$ with zero loss in accuracy as compared to the full model.

SPADE is a distributed inference framework designed to reduce the computational costs and latency of running Large Language Models (LLMs) by splitting the workload between edge devices and cloud servers. By using a small model on the edge to generate draft text and a large model in the cloud to verify it, the system maintains the accuracy of the full-scale model while significantly decreasing the number of expensive cloud queries.

Balancing Edge and Cloud

Deploying LLMs presents a trade-off: running models entirely on edge devices often results in lower accuracy, while running them entirely in the cloud is expensive and slow due to the autoregressive nature of token generation. SPADE addresses this by using a "plug-and-play" architecture. A compact draft model (such as LLaMA-3.2-1B) resides on the edge to generate candidate tokens, while a larger, more accurate verifier model (such as LLaMA-3.1-8B) resides in the cloud. This setup shifts the bulk of the generative computation to the edge, reserving cloud resources only for parallel verification.

How the Verification Process Works

The framework utilizes speculative decoding to minimize cloud interaction. The edge model generates a block of $d$ candidate tokens, which are sent to the cloud in a single request. The cloud verifier evaluates these tokens in parallel. If the tokens match the verifier’s probability distribution, they are accepted. If a token is rejected, the verifier provides a corrected token, and the edge model resumes generation from that updated point. This process ensures the final output is statistically identical to the output of the full-scale cloud model, requiring no additional training or fine-tuning.

Performance and Efficiency

Experimental results using the SpecBench and CNN/DailyMail datasets indicate that SPADE reduces cloud model calls by 76% compared to running the full model alone. In these tests, the framework achieved performance scores comparable to the full-scale model across various tasks, including summarization, translation, and mathematical reasoning. The researchers note that the number of draft tokens ($d$) serves as a key hyperparameter; increasing $d$ reduces the frequency of cloud calls but must be balanced against the risk of higher token rejection rates and increased local computation.

Practical Considerations

The effectiveness of SPADE depends on the alignment between the edge draft model and the cloud verifier. If the two models are poorly aligned, the acceptance rate of the draft tokens drops, which leads to more frequent cloud interventions and higher costs. The researchers suggest that the optimal number of draft tokens should be determined empirically by monitoring acceptance rates on a small validation subset of data. This approach allows the system to adapt to different hardware constraints and latency requirements without needing to retrain the underlying models.

Comments (0)

No comments yet

Be the first to share your thoughts!