Back to AI Research

AI Research

Random Cloud: Finding Minimal Neural Architectures... | AI Research

Key Takeaways

  • Random Cloud: Finding Minimal Neural Architectures Without Training The Random Cloud method is a training-free approach to neural architecture search designe...
  • I propose the \emph{Random Cloud} method, a training-free approach to neural architecture search that discovers minimal feedforward network topologies through stochastic exploration and progressive structural reduction.
  • I evaluate on 7 classification benchmarks against magnitude pruning and random pruning baselines.
  • The Random Cloud matches or outperforms both baselines in 6 of 7 datasets, achieving statistically significant improvements on Sonar ($+4.9$pp accuracy, $p{=}0.017$ vs magnitude pruning) with 87\% parameter reduction.
  • Crucially, the method is faster than both pruning baselines in 4 of 5 datasets (0.67--0.94$\times$ the cost of full training), since it avoids training the full-size network entirely.
Paper AbstractExpand

I propose the \emph{Random Cloud} method, a training-free approach to neural architecture search that discovers minimal feedforward network topologies through stochastic exploration and progressive structural reduction. Unlike post-training pruning methods that require a full train-prune-retrain cycle, this method evaluates randomly initialized networks without backpropagation, progressively reduces their topology, and only trains the best minimal candidate at the end. I evaluate on 7 classification benchmarks against magnitude pruning and random pruning baselines. The Random Cloud matches or outperforms both baselines in 6 of 7 datasets, achieving statistically significant improvements on Sonar ($+4.9$pp accuracy, $p{=}0.017$ vs magnitude pruning) with 87\% parameter reduction. Crucially, the method is faster than both pruning baselines in 4 of 5 datasets (0.67--0.94$\times$ the cost of full training), since it avoids training the full-size network entirely.

Random Cloud: Finding Minimal Neural Architectures Without Training
The Random Cloud method is a training-free approach to neural architecture search designed to identify the smallest possible neural network topology for a given task. Traditional methods for finding minimal networks often require multiple, computationally expensive training cycles—training a large network, pruning it, and then retraining it to recover performance. The Random Cloud method bypasses this by evaluating randomly initialized networks without using backpropagation, progressively reducing their size, and only performing a single training session on the best-performing minimal candidate.

How the Method Works

The process is divided into three distinct phases: exploration, selection, and refinement. During the exploration phase, the system generates a "cloud" of networks with random weights. Each network is evaluated on the training data using only forward passes. The system then iteratively removes neurons from the network, reconstructing it while preserving the existing weight structure, and re-evaluates its performance. This continues until the network can no longer be reduced while maintaining a specific accuracy threshold. In the selection phase, the system identifies the smallest network that achieved the highest accuracy during the exploration phase. Finally, this selected candidate is trained using standard backpropagation.

Performance and Efficiency

The Random Cloud method was tested against standard magnitude and random pruning baselines across seven classification datasets. It matched or outperformed these baselines in six of the seven datasets. Notably, on the Sonar dataset, the method achieved a statistically significant improvement in accuracy while reducing the number of parameters by 87%. Because the method avoids the need to train a full-sized network, it is also more computationally efficient; in four out of five datasets, it proved faster than traditional pruning methods, requiring only 0.67 to 0.94 times the cost of full training.

Key Considerations and Limitations

The success of the Random Cloud method relies on the observation that, among a sufficiently large group of randomly initialized networks, some will naturally exhibit non-trivial classification accuracy. The method is robust to its primary hyperparameters, such as the accuracy threshold and the size of the "cloud" of networks. However, the approach has limitations regarding scalability. In scenarios with high-dimensional input spaces, the signal provided by the training-free evaluation degrades, making the method less effective compared to traditional pruning. Consequently, the technique is currently best suited for tabular data with moderate dimensionality.

Comments (0)

No comments yet

Be the first to share your thoughts!