AI Research

Xiaomi-TabLDM: A Tabular Foundation Model Technical... | AI Research

Key Takeaways

Xiaomi-TabLDM is a foundation model designed to handle tabular data—the structured rows and columns common in fields like finance and healthcare—without the...
We introduce Xiaomi-TabLDM, a tabular large data foundation model for classification and regression via in-context learning, which delivers superior prediction accuracy without requiring task-specific fine-tuning.
Pretrained exclusively on synthetic data generated from structural causal models (SCMs), our model enables more flexible context utilization and more efficient capacity scaling.
Favorable performance--efficiency trade-off: Xiaomi-TabLDM combines strong predictive performance with substantially lower computational cost.
For example, on TabArena regression, it achieves the second-highest Elo while using 82% less training time and 68% less prediction time than the top-ranked TabFM.

Paper AbstractExpand

We introduce Xiaomi-TabLDM, a tabular large data foundation model for classification and regression via in-context learning, which delivers superior prediction accuracy without requiring task-specific fine-tuning. Pretrained exclusively on synthetic data generated from structural causal models (SCMs), our model enables more flexible context utilization and more efficient capacity scaling. i) A new performance standard. Strong regression performance across benchmarks: Xiaomi-TabLDM ranks 1st on OpenML-CTR23 and 2nd on regression across TALENT, TabArena, and BCCO, demonstrating consistently strong regression performance across four complementary benchmark suites. Favorable performance--efficiency trade-off: Xiaomi-TabLDM combines strong predictive performance with substantially lower computational cost. For example, on TabArena regression, it achieves the second-highest Elo while using 82% less training time and 68% less prediction time than the top-ranked TabFM. ii) Large-scale synthetic pretraining. Xiaomi-TabLDM expands the coverage and diversity of synthetic tabular data used for pretraining. We also adopt a three-stage training strategy together with dual-stream feature grouping, lightweight Attention Residual, and sparse Mixture-of-Experts, enabling Xiaomi-TabLDM to learn richer feature interactions and expert specialization across diverse tabular tasks. iii) Test-time scaling. Xiaomi-TabLDM further extends tabular prediction through test-time compute scaling, where allocating additional computation at inference time consistently improves predictive performance over the base model.

Xiaomi-TabLDM is a foundation model designed to handle tabular data—the structured rows and columns common in fields like finance and healthcare—without the need for task-specific fine-tuning. By leveraging in-context learning, the model can make predictions on new, unseen datasets simply by observing labeled examples provided as context. It aims to set a new standard for accuracy and efficiency in both classification and regression tasks by using a specialized architecture trained entirely on synthetic data.

Large-Scale Synthetic Pretraining

Rather than training on real-world datasets, Xiaomi-TabLDM is pretrained exclusively on synthetic data generated from structural causal models. This approach allows the researchers to create a vast, diverse range of tabular tasks with varying structures and relationships. The training process follows a three-stage curriculum that progressively increases the complexity and size of the datasets. This synthetic foundation enables the model to learn robust feature interactions and generalize across different types of tabular problems effectively. The same reasoning question is explored in GRAIN, which adds a research perspective.

Architectural Innovations

The model introduces several technical features to improve performance and efficiency:

Dual-stream feature grouping: This method breaks feature symmetries and creates more flexible column representations, making the model robust to how columns are ordered.
Lightweight Attention Residuals: These connections allow the model to selectively retrieve information from earlier layers, improving how data propagates through the network and stabilizing training.
Sparse Mixture-of-Experts: By replacing standard feed-forward layers with a sparse Mixture-of-Experts, the model can significantly increase its total capacity for complex tasks while keeping the actual computational cost per prediction low.

Performance and Efficiency

Xiaomi-TabLDM demonstrates strong predictive capabilities, particularly in regression. It ranks first on the OpenML-CTR23 benchmark and second on regression tasks across the TALENT, TabArena, and BCCO suites. Beyond raw accuracy, the model offers a significant efficiency advantage. For instance, on the TabArena regression benchmark, it achieves high performance while using 82% less training time and 68% less prediction time compared to top-ranked alternatives like TabFM. The ai search story also surfaces in Google AI Releases TimesFM 3 for..., adding another angle.

Test-Time Scaling

The model further improves its predictions through test-time compute scaling. Instead of modifying the model parameters, this strategy allocates additional computation during inference. By using techniques like feature shuffling, diverse preprocessing views, and combining multiple prediction paths through nonnegative least squares, the model can adapt to the specific characteristics of a new dataset. This allows for improved accuracy on heterogeneous tasks without the need for retraining or fine-tuning the base model. The same ai evaluation question is explored in Efficient Test-Time Adaptation through Human-AI Interaction, which adds a research perspective. as detailed in the full paper on Arxiv

Comments (0)

No comments yet

Be the first to share your thoughts!