A CodeContests study finds that giving language models related competitive programming examples can improve their first-attempt pass rate. The effect depended on which examples were supplied. In the reported evaluation, retrieving similar problems helped on average, while adding random examples performed slightly worse than using the original problem alone. The preprint was submitted to arXiv on November 6, 2026.
How did the experiment compare retrieval methods?
The authors tested six models on 81 problems. They compared a prompt with only the problem, a prompt augmented with examples chosen at random and a prompt using similarity-based retrieval. The retrieval system searched embeddings of training-split problems. It varied the similarity cutoff based on the strength of the nearest match and returned at most three examples. That design tests whether retrieval-augmented generation can give the model useful context without filling the prompt with loosely related material.
How large was the difference?
The authors report average success of 20.37% with similar examples, compared with 17.90% without examples and 17.28% with random examples. The improvement over the prompt without examples was 2.47 percentage points. Model results varied: the authors report that DeepSeek-R1 went from 61.7% without examples to 64.2% with similar examples. The same model scored 59.3% with random retrieval.
What should readers take from the design?
The test focused on problems that had similar examples in the training split, so it cannot show the same gain on tasks with no useful neighbor. The authors also used a Min-K membership analysis to examine potential pretraining memorization, which is distinct from passing an executable test. An evaluation of a coding assistant could use this setup as a starting point: compare no examples with random and retrieved examples, then measure whether the generated programs actually solve the problems.
Comments