Style, Not Self: Surface Cues Explain Zero-Shot Cod... | AI Research

What the paper is about

If a language model can recognize code it wrote, it may favor that code as a judge, and instances of one model monitoring each other could collude. We test this zero-shot on current commercial models. Five LLMs generate solutions to MBPP, HumanEval, and DS-1000, seven more to MBPP, and models act as evaluators in four tasks: picking their own solution from a pair, judging whether a single solution is their own, identifying which of two solutions a named model wrote, and judging quality blind. In the single-solution task, balanced accuracy is 49-58% for all 15 model-benchmark combinations, while raw accuracy (38-67%) mostly reflects how readily a model claims authorship. In the pairwise task, accuracy across 14 evaluator-opponent combinations correlates at r=0.93 with how often the evaluator's solution is longer. Attribution to a named model succeeds on some pairs and is consistently inverted on others. A rule-based normalization that strips docstrings, comments, type hints, and local names preserves Pass@1 and leaves ten of twelve re-tested results at chance; the other two follow a length difference it leaves, although a trained classifier still separates most normalized pairs. Claude Haiku's self-preference also disappears. We recommend reporting balanced accuracy, heuristic baselines, and label consistency.

What it covers

Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models Ehsan Barkhordar Affiliation: Koç University Email: [email protected] Surendrabikram Thapa Affiliation: Virginia Tech Email: [email protected] Abstract If a language model can recognize code it wrote, it may favor that code as a judge, and instances of one model monitoring each other could collude. We test this zero-shot on current commercial models. Five LLMs generate solutions to MBPP, HumanEval, and DS-1000, seven more to MBPP, and models act as evaluators in four tasks: picking their own solution from a pair, judging whether a single solution is their own, identifying which of two solutions a named model wrote, and judging quality blind. In the single-solution task, balanced accuracy is 49–58% for all 15 model–benchmark combinations, while raw accuracy (38–67%) mostly reflects how readily a model claims authorship. In the pairwise task, accuracy across 14 evaluator–opponent combinations correlates at r = 0.93 r=0.93 with how often the evaluator’s solution is longer. Attribution to a named model succeeds on some pairs and is consistently inverted on others. A rule-based normalization that strips docstrings, comments, type hints, and local names preserves Pass@1 and leaves ten of twelve re-tested results at chance; the other two follow a length difference it leaves, although a trained classifier still separates most normalized pairs. Claude Haiku’s self-preference also disappears. We recommend reporting balanced accuracy, heuristic baselines, and label consistency. 1 Introduction Language models increasingly evaluate the outputs of other language models, as reward models, code reviewers, and monitors in AI-control pipelines ( Greenblatt et al., 2024 ) . Two failure modes threaten such oversight. LLM judges rate their own generations above others’ even when humans see no difference ( Panickssery et al., 2024 ; Wataoka et al., 2024 ) , and in untrusted monitoring a monitor that is an instance of the generator may collude with it, either through a hidden signal ( Greenblatt et al., 2024 ) or simply by recognizing the code as its own ( Gardner-Challis et al., 2026 ) . Both are often traced to a model recognizing where an output came from: self-preference in summarization grows with self-recognition ability ( Panickssery et al., 2024 ) , although it can also arise from familiarity alone ( Wataoka et al., 2024 ) . How much self-recognition do current models have? Code is a good domain for the question. Coding style is measurable, and trained classifiers attribute LLM-generated programs to their source model with high accuracy ( Bisztray et al., 2025 ; Tihanyi et al., 2026 ; Guo et al., 2026 ) . Whether a model can use such cues without training, when simply asked which code is its own, is a separate question, and the literature disagrees: some studies report substantial self-recognition ( Panickssery et al., 2024 ; Caiado and Hahsler, 2023 ) , Davidson et al. (2024) find none once position and length are controlled, Bai et al. (2025) find attribution near chance, and St. Amand et al. (2026) , whose domains include code, trace the disagreement to presentation format and a quality heuristic. We present a controlled study of zero-shot code attribution. Five widely used commercial models, GPT-5 and four fast or open-weight models, generate solutions to three Python benchmarks, and seven more generate MBPP solutions for cross-model experiments. In four tasks a model picks its own solution from a pair, says whether a single solution is its own, identifies which of two solutions a named model wrote, or judges which of two solutions is better without authorship information. Every pairwise result is compared not only with chance but with simple non-LLM rules (choose the longer solution, the one with a docstring) on the same pairs, with correction for multiple comparisons, and a trained classifier measures how much authorship signal the code itself carries. The results are largely negative for self-recognition. In the individual setting, balanced accuracy lies between 49% and 58% for all 15 model–benchmark combinations, and only two cells survive correction; the spread of raw accuracies, from 38% to 67%, reflects how readily a model claims authorship. In the pairwise setting, accuracy across 14 evaluator–opponent combinations correlates at r = 0.93 r=0.93 with how often the evaluator’s own solution is the longer one, and item by item most evaluators pick the longer solution, so verbose models appear to recognize themselves and terse models score well below chance. Attribution to a named model reaches at most 82%, on a pair that length alone separates at 89%, and on two other pairs every judge is below chance, attaching the wrong name to styles it can tell apart. A rule-based normalization that strips docstrings, comments, and type annotations and renames local identifiers preserves Pass@1. On normalized code ten of twelve re-tested results lie between 44% and 58%, none significant after correction; the other two follow a length difference the normalization leaves intact. A trained classifier still attributes most normalized pairs well above chance, so the judges lose the signal they were using, not all the signal in the code. The one clear self-preference, Claude Haiku favoring its own solutions, also disappears. Two methodological findings accompany these results. An earlier version of our attribution prompt listed the author names in the same order as the solutions, which let most judges score 86--99% from the name order alone. And response bias is invisible, and below-chance accuracy easily misread as absence of ability, when a single accuracy is reported against a 50% baseline. Code, generations, and judgments are released. 1 1 1 https://github.com/ebarkhordar/llm-collusion 2 Related Work Self-recognition and self-preference. Panickssery et al. (2024) introduced pairwise and individual self-recognition tests on summarization, found GPT-4 at 73.5% without training, and showed that self-preference grows as fine-tuning improves self-recognition. Caiado and Hahsler (2023) reported high self-detection for some providers and chance for others. Later work is more skeptical: Davidson et al. (2024) found no consistent self-recognition across ten models and attributed apparent successes to models picking the answer they judge best, Bai et al. (2025) found near-chance performance with predictions collapsing onto a few frontier families, and St. Amand et al. (2026) , whose domains include coding, trace the conflicting results to presentation format and task domain, with a quality heuristic as the dominant confound. Length normalization largely removes Llama-3-8B-Instruct’s ability to tell its text from other models’ ( Ackerman and Panickssery, 2025 ) , self-detection of education text degrades for some models under human-like prompting ( Burger et al., 2026 ) , and hidden states carry self-recognition signal that representation editing can surface ( Zhou et al., 2025 ) , outside our black-box setting. Recognizing one’s own text is one of the tasks of the Situational Awareness Dataset ( Laine et al., 2024 ) , and models fine-tuned to predict their own behavior do so better than other models trained on the same behavior ( Binder et al., 2025 ) . Biases of LLM judges. LLM judges show position and verbosity bias ( Zheng et al., 2023 ; Saito et al., 2023 ) , which length-controlled evaluation reduces ( Dubois et al., 2024 ) ; self-preference tracks the judge’s own perplexity ( Wataoka et al., 2024 ) , and code judges are swayed by the variable names and comments of functionally identical code ( Moon et al., 2026 ) . Code stylometry. Syntactic features identify human programmers ( Caliskan-Islam et al., 2015 ) , and semantics-preserving transformations can mislead such attribution ( Quiring et al., 2019 ) . Trained classifiers also attribute LLM-generated code well: 95% five-way on C, where removing comments costs 2–3 points in a two-way task and about 7 points five-way ( Bisztray et al., 2025 ) ; 88–96% on JavaScript, with minified variants above 90% five-way ( Tihanyi et al., 2026 ) ; and about 93% F1 without comments ( Guo et al., 2026 ) . Family-level attribution remains hard under distribution shift ( Orel et al., 2026 ) , simple lexical features fingerprint LLM text ( McGovern et al., 2025 ) , and machine-generated code can be detected zero-shot from a proxy model’s token probabilities ( Yang et al., 2023 ) . We ask whether models can use this signal when prompted, without training or probabilities. Obfuscation and collusion. Paraphrasing text lowers attribution accuracy but imprints the paraphraser’s style ( Tripto et al., 2024 ) , and perturbing interpretable style axes obfuscates authorship ( Fisher et al., 2024 ) . For code, semantics-preserving rewrites weaken watermark detectors ( Suresh et al., 2024 ) and obfuscation drives n-gram watermark detectors to chance ( Zhang et al., 2025 ) . In AI control, redacting code (removing comments, formatting, minifying) or paraphrasing it before an untrusted monitor sees it is a defense against collusion signals ( Greenblatt et al., 2024 ) ; steganographic collusion is a recognized risk ( Motwani et al., 2024 ) that paraphrasing only partly prevents ( Mathew et al., 2025 ) , and a monitor fine-tuned to recognize its generator’s code remains an effective colluder even with comments stripped, although un-elicited self-recognition was weak in the same setting ( Gardner-Challis et al., 2026 ) . We ask the zero-shot version across four attribution tasks, with rule-based baselines on the same items: does a normalization that removes the features those baselines find discriminative also remove the attribution signal of untrained models? 3 Experimental Setup 3.1 Benchmarks, Models, and Testing We use the sanitized test split of MBPP ( Austin et al., 2021 , 257 problems;) , HumanEval ( Chen et al., 2021 , 164;) , and the first 150 problems of DS-1000 ( Lai et al., 2023 ) , all from its Pandas subset. All pairwise experiments use MBPP; the individual task uses all three. Five core models generate solutions on all three benchmarks and serve as evaluators: GPT-5, Claude Haiku 4.5, Gemini 2.5 Flash, Grok 4 Fast, and DeepSeek-V3-0324 (hereafter Claude Haiku, Gemini, Grok, and DeepSeek-V3 or DeepSeek). GPT-5 is a flagship model; Claude Haiku, Gemini, and Grok are the small or fast tiers of their families, and DeepSeek-V3 is a non-reasoning open-weight model. For target identification we add seven MBPP generators (Claude Opus 4.6, Gemini 3.1 Flash Lite, DeepSeek-V3.2, GPT-5.3-Codex, Codestral 2508, Qwen3-Coder-Next, MiMo-V2-Pro) and the judges GPT-5, GPT-5.3-Codex, Grok-Code-Fast-1, and Gemini 2.5 Flash. All models were accessed through OpenRouter with temperature 0 and a 2,000-token output limit (identifiers in Appendix D ). Requests set nothing else, so each model ran with OpenRouter’s default reasoning settings (Gemini 2.5 Flash and Grok 4 Fast also support reasoning), and they were not pinned to a hosting provider. The OpenAI models are reasoning models that ignore temperature and count reasoning tokens toward the limit; GPT-5 returned no code on 45 of its 571 generation tasks (16 on MBPP). These count as failures for Pass@1 and are excluded from every attribution task. Generation and the original experiments ran in March 2026 and the later experiments in September 2026, by which time Grok 4 Fast and Grok-Code-Fast-1 had been withdrawn from the API. Comparisons between the two periods can therefore also reflect model updates; a September repeat of one March run reproduces its accuracy exactly (Section 4.2 ). Each benchmark has its own generation prompt (Appendix A ); the MBPP prompt permits comments without asking for them, while the HumanEval and DS-1000 prompts ask for a docstring or comments and for descriptive names, which makes the models’ styles on those benchmarks more alike. Code runs against the benchmark’s tests in an isolated subprocess with a five-second timeout and passes only if all tests pass. For HumanEval the prompt is loaded first so that its helper functions are available, the tests call the prompt’s entry point, and a body-only answer is scored as a completion of the prompt, as in the official protocol. DS-1000 was run with pandas 2.3.3. 3.2 Attribution Tasks Every task shows the original task description and one or two solutions and asks for a one-token answer. Prompts are given in Appendix A . A reply counts as an answer only if its first line is the answer alone; refusals (“Sorry, I can’t help with that.”) and explanations are non-answers and are excluded. Pairs whose two solutions are identical are excluded as well, since no judge or rule can attribute them. Task 1a: pairwise self-recognition. The evaluator sees its own solution and another model’s solution to the same problem in random order and answers “A” or “B” to the question of which one it wrote. Chance is 50%. In the original runs each core model was paired with GPT-5 and GPT-5 with Grok 4 Fast; we later ran all pairs among the four core models still available. Task 1b: individual self-recognition. Following the individual-presentation paradigm of Panickssery et al. (2024) , the evaluator sees one solution and answers “yes” or “no” to whether it generated it. For each evaluator we present all of its own solutions and all solutions of two other core models (Claude Haiku and DeepSeek for GPT-5 and Grok; DeepSeek and Gemini for Claude Haiku; Claude Haiku and Gemini for DeepSeek; Claude Haiku and DeepSeek for Gemini), so one third of the items are positives. Responses that are neither “yes” nor “no” are recorded as abstentions and excluded from the rates (raw accuracy counts them as errors). Because of this imbalance we report balanced accuracy, BA = 1 2 ​ ( P ⁡ ( yes ∣ own ) + P ⁡ ( no ∣ other ) ) \mathrm{BA}=\tfrac{1}{2}\big(P(\text{yes}\mid\text{own})+P(\text{no}\mid\text{other})\big) , whose chance level is 50% regardless of response bias, together with the two yes-rates. Raw accuracy is reported only to illustrate how misleading it is: always answering “no” scores 66.7%. Task 2: target identification. A judge, which need not be either author, sees two solutions, is told the two authors’ names, and must say which solution a named target wrote. Model names appear in the prompt in an order randomized independently of solution order (Appendix B explains why this matters). We ran 17 judge–pair–target combinations on MBPP, chosen to cover pairs with very different styles, a pair of recent open-weight models from two Chinese labs whose styles differ little (Qwen3-Coder-Next vs. MiMo-V2-Pro), and pairs involving models released close to or after the judges, whose code a judge can know only through the model family. Task 3: blind self-preference. A model sees two solutions, one its own, in random order and with no authorship information, and is asked which is better, judging correctness first and disregarding comments and formatting. For a pair X X vs. Y Y we run the same items with X X as judge and with Y Y as judge. The quantity Δ = P ⁡ ( X ​ chosen ∣ X ​ judges ) − P ⁡ ( X ​ chosen ∣ Y ​ judges ) \Delta=P(X\text{ chosen}\mid X\text{ judges})-P(X\text{ chosen}\mid Y\text{ judges}) is the combined self-preference of the two judges on the same items; a shared opinion about which solution is better cancels out. A third model Z Z outside the pair judges the same items in the same order, which splits Δ \Delta into Δ X = P ⁡ ( X ∣ X ) − P ⁡ ( X ∣ Z ) \Delta_{X}=P(X\mid X)-P(X\mid Z) and Δ Y = P ⁡ ( X ∣ Z ) − P ⁡ ( X ∣ Y ) \Delta_{Y}=P(X\mid Z)-P(X\mid Y) , each relative to Z Z ’s taste. All rates are computed on the items that every judge of the pair answered. Since test outcomes are known, we also check Δ \Delta on items on which both solutions pass or both fail. Four pairs were run on original code (all pairs among Claude Haiku, Gemini, and DeepSeek, and GPT-5 vs. DeepSeek) and three on normalized code. 3.3 Baselines and Statistics For every pairwise experiment we compute the accuracy of seven non-LLM rules on the same pairs: choose the longer (or shorter) solution by character count, the one with (or without) a docstring, the one with (or without) type hints, and the one with more comment lines, with ties scored as one half. Docstrings and type hints are detected with regular expressions (a triple-quoted string; a parameter or return annotation). These baselines answer whether a rule with no access to the model could do as well; since the best rule is chosen on the same items, it is an upper bound on what any one fixed rule achieves. As a reference for how much authorship signal the code itself carries, we also train a classifier (character 2–5-gram TF-IDF features and logistic regression, five-fold cross-validation grouped by task) and let it make the judges’ two-way choice: of two solutions to a task, the one it scores as more likely to be the target’s is assigned to the target. We report Wilson 95% confidence intervals and two-sided exact binomial tests against 50% for accuracies (a normal approximation for balanced accuracy), the rate at which each evaluator picks position A, and position-balanced accuracy, the mean of the accuracies with the evaluator’s own solution in position A and in position B. Stars in tables mark raw p p -values; because each table contains several tests, we also apply the Holm step-down correction within each table (and across all twelve normalized-code results) and state in the text which results survive it. For the self-preference comparison, whose two runs judge the same items, we use an exact McNemar test on the discordant items. For target identification we additionally check, when a judge has been asked about both targets of a pair, whether its two answers form a consistent partition (different positions) and, if so, whether the labels are correct or inverted. A judge answering at random is consistent on half the items; one that separates the two styles but holds the wrong association will be consistent and inverted. 3.4 Normalization To test whether the features identified by the baselines can be removed without harming the code, we apply a rule-based normalization to every MBPP solution: remove docstrings and comments, remove type annotations, rename local variables, parameters, and nested functions to v 0 , v 1 , … v_{0},v_{1},\dots , and re-serialize from the AST, which standardizes formatting (example in Appendix E ). Top-level function and class names (which the tests call), imported names, built-ins, attributes, and string literals are kept, so error messages, for example, survive. Four of the 3,084 solutions do not parse (none from a core model) and are only stripped of comments and docstrings; one answer written in Markdown is normalized from its code block. We then re-run the benchmark tests, recompute every heuristic baseline, and re-run the four original pairwise self-recognition runs whose evaluator was still available (all but Grok’s), six target-identification runs, and the self-preference task with the same evaluators, judges, and prompts. Because Grok-Code-Fast-1 had been withdrawn, we also ran a third-party judge, Gemini 2.5 Flash, on Claude Haiku vs. DeepSeek on both versions of the code. The target-identification and self-preference reruns keep each item’s A/B order; the pairwise reruns drew a new order, which position-balanced accuracy shows to be immaterial (Section 4.5 ). The reruns were first made with a normalizer that left lambda parameters and nested function names unrenamed, and then with one that broke imports inside function bodies; after each fix we re-judged the judgments whose code had changed (309 of 4,111, and later 58 of 5,396), so all normalized results use the final normalizer. 4 Results 4.1 Code Generation Table 9 (Appendix F ) reports Pass@1. GPT-5 is strongest on MBPP and DS-1000 and Grok 4 Fast on HumanEval; DS-1000 is hardest, with Claude Haiku at 36%. Of 797 failures across 2,855 executions, 67% are assertion errors, 9% syntax errors (mostly in DS-1000 snippets and spread evenly across models), and 46 are empty outputs, 45 of them from GPT-5 (Appendix C ). 4.2 Task 1a: Pairwise Self-Recognition Evaluator \ \backslash Other GPT-5 Claude Haiku 4.5 Gemini 2.5 Flash Grok 4 Fast DeepSeek-V3 GPT-5 – 47.9 (67) 45.1 (47) 81.9 ∗∗∗ (97) 78.8 ∗∗∗ (94) Claude Haiku 4.5 31.5 ∗∗∗ (33) – 39.5 ∗∗∗ (25) – 83.9 ∗∗∗ (89) Gemini 2.5 Flash 50.0 (53) 64.1 ∗∗∗ (75) – – 81.2 ∗∗∗ (95) Grok 4 Fast 41.1 ∗∗ (2) – – – – DeepSeek-V3 30.5 ∗∗∗ (6) 31.0 ∗∗∗ (11) 29.3 ∗∗∗ (5) – – Table 1: Pairwise self-recognition accuracy (%) for each evaluator (rows) against each other core model (columns) where the pair was run; in parentheses, the percentage of pairs in which the evaluator’s own solution is the longer one (ties count one half). N = 234 N=234 –256 per cell after removing items with an empty or identical solution and non-answers ( N N per cell in Table 15 ). Significance against 50%: ∗ p < 0.05 {}^{}p<0.05 , p ∗ ⁣ ∗ < 0.01 {}^{}p<0.01 , ∗ ∗ ∗ p < 0.001 {}^{}p<0.001 . Grok 4 Fast was withdrawn from the API before the additional pairs could be run. Confidence intervals, position rates, and position-balanced accuracy for the original runs are in Appendix F . Table 1 gives accuracy for every evaluator against every available opponent. In the original runs, in which each model was paired with GPT-5, only GPT-5 itself exceeded chance (81.9% against Grok, p < 10 − 14 p<10^{-14} ); Gemini was at chance, and Grok (41.1%), Claude Haiku (31.5%), and DeepSeek-V3 (30.5%) were significantly below it, attributing GPT-5’s code to themselves. The heuristic baselines account for both outcomes. Every non-empty GPT-5 MBPP solution has a docstring, and the solutions average 18 lines, against 17% and 7 lines for Grok. On GPT-5’s pairs with Grok the rule “choose the longer solution” scores 97.5% and “choose the solution with a docstring” 91.8%, both above GPT-5’s own accuracy, and on the 199 pairs where exactly one solution has a docstring GPT-5 chooses that solution 85.4% of the time. Claude Haiku’s and DeepSeek’s solutions are shorter than GPT-5’s and lack its type hints, and both models nevertheless claim GPT-5’s, which is consistent with the “choose the best-looking answer” behavior Davidson et al. (2024) describe for text. GPT-5 declines to answer on up to 3% of its pairs (“Sorry, I can’t help with that”), as it does more often in the individual task. The full matrix shows that this is largely a property of pairs rather than of evaluators. Claude Haiku scores 83.9% against DeepSeek and 31.5% against GPT-5; Gemini scores 81.2% against DeepSeek and 50.0% against GPT-5; DeepSeek, whose solutions are the shortest of the four models in the full matrix, is below chance against every opponent (29.3–31.0%). Eleven of the 14 cells differ from chance after Holm correction; the three that do not are GPT-5 against Claude Haiku and against Gemini, and Gemini against GPT-5, pairs in which both solutions are documented. Across the 14 cells, accuracy correlates with the fraction of pairs in which the evaluator’s own solution is the longer one at Pearson r = 0.93 r=0.93 , with a slope of 0.54: evaluators whose code is longer in 89–97% of pairs score 79–84%, and evaluators whose code is longer in 2–11% of pairs score 29–41%. Since cells share evaluators and items, we treat the correlation as descriptive (leaving out any one evaluator, r = 0.90 r=0.90 –0.96) and check it item by item (Appendix Table 15 ). In 12 of the 14 cells the evaluator picks the longer of the two solutions on 59–87% of items. The exceptions are GPT-5 against Claude Haiku and Gemini, whose solutions are documented like its own; there GPT-5 picks the longer solution about half the time, which suggests that it keys on documentation rather than length. The other four models lean on one rule, that the longer and better documented solution is their own. Within a cell, the evaluator is usually more accurate when its own solution is the longer one (11 of 14 cells; Claude Haiku against GPT-5: 50.6% versus 22.2%). Position bias is also present (DeepSeek chooses position A on 29% of its pairs with GPT-5 and Gemini on 75%), but position-balanced accuracy is within 1.5 points of raw accuracy in every run. A paraphrased prompt shifts the below-chance accuracies by up to ten points without changing their direction, and a repeat run with the original prompt six months later reproduces the aggregate exactly and the individual answer on 97.5% of items shown in the same order (Appendix F , Table 8 ). 4.3 Task 1b: Individual Self-Recognition Figure 1: Individual self-recognition: rate of answering “yes” to “did you generate this code?” when the code is the evaluator’s own (blue) and when it is another core model’s (orange). Recognition would appear as a gap between the two bars; the bars are similar for most models and benchmarks, the largest gap being Grok’s on DS-1000 (89% vs. 72%). Full numbers are in Appendix F . Figure 1 shows the individual-presentation results. For most models and benchmarks, the probability of claiming authorship is similar for the model’s own code and for other models’ code. Balanced accuracy ranges from 49.4% to 58.2% over the 15 combinations (Table 12 gives 95% intervals and raw accuracies). Four cells are above 50% before correction, Grok on HumanEval (54.0%) and DS-1000 (58.2%), Gemini on MBPP (53.8%), and DeepSeek on MBPP (54.4%), and the two Grok cells survive Holm correction. A small amount of signal may therefore exist; the largest value corresponds to yes-rates of 89% and 72%. This matches text, where without fine-tuning the individual setting was near chance for every model but GPT-4 ( Panickssery et al., 2024 ) . What differs across models is response policy. GPT-5 never answers “yes” (0 of 1,668 items) and declines to answer on 5–10% of items, typically with a statement that it cannot know whether it generated the code. Claude Haiku says “yes” to at most 6% of items. Grok and Gemini say “yes” to 58–97% of items. DeepSeek is the only model with an intermediate yes-rate on every benchmark, and its gap between own and other code never exceeds 9 points; its few abstentions are degenerate repetitive text rather than refusals. These policies produce raw accuracies from 38% to 67% that would be misread as an ability ranking; Claude Haiku’s 65.7% on MBPP, for example, is below the 66.7% obtained by always answering “no”. 4.4 Task 2: Target Identification Pair Judge Target N N Acc. (%) 95% CI P(A) Pos.-bal. (%) Best heuristic (%) Claude Haiku 4.5 vs. DeepSeek-V3 Gemini 2.5 Flash Claude Haiku 4.5 255 77.3 ∗∗∗ [71.7, 82.0] 53 77.2 Longer code (89.0) Claude Haiku 4.5 vs. DeepSeek-V3 Gemini 2.5 Flash DeepSeek-V3 255 27.1 ∗∗∗ [22.0, 32.8] 57 27.2 Shorter code (89.0) Claude Haiku 4.5 vs. DeepSeek-V3 Grok-Code-Fast-1 Claude Haiku 4.5 255 81.6 ∗∗∗ [76.4, 85.8] 55 81.5 Longer code (89.0) Claude Haiku 4.5 vs. DeepSeek-V3 Grok-Code-Fast-1 DeepSeek-V3 255 69.0 ∗∗∗ [63.1, 74.4] 52 69.1 Shorter code (89.0) Gemini 2.5 Flash vs. GPT-5 GPT-5.3-Codex Gemini 2.5 Flash 240 76.2 ∗∗∗ [70.5, 81.2] 31 77.3 Longer code (52.9) Gemini 2.5 Flash vs. GPT-5 Grok-Code-Fast-1 Gemini 2.5 Flash 240 40.4 ∗∗ [34.4, 46.7] 49 40.5 Longer code (52.9) Claude Haiku 4.5 vs. GPT-5 GPT-5.3-Codex Claude Haiku 4.5 241 67.2 ∗∗∗ [61.1, 72.8] 34 68.0 No type hints (76.1) Codestral 2508 vs. GPT-5.3-Codex GPT-5 Codestral 2508 239 57.3 ∗ [51.0, 63.4] 31 57.2 No docstring (51.0) DeepSeek-V3.2 vs. MiMo-V2-Pro GPT-5 DeepSeek-V3.2 256 52.0 [45.8, 58.0] 50 52.0 Has docstring (56.1) Codestral 2508 vs. Grok 4 Fast GPT-5.3-Codex Codestral 2508 246 49.6 [43.4, 55.8] 51 49.6 Longer code (61.0) Qwen3-Coder-Next vs. MiMo-V2-Pro GPT-5 Qwen3-Coder-Next 252 41.3 ∗∗ [35.4, 47.4] 49 41.3 No type hints (56.9) Qwen3-Coder-Next vs. MiMo-V2-Pro GPT-5 MiMo-V2-Pro 252 41.3 ∗∗ [35.4, 47.4] 72 41.6 Has type hints (56.9) Qwen3-Coder-Next vs. MiMo-V2-Pro GPT-5.3-Codex Qwen3-Coder-Next 252 41.7 ∗∗ [35.7, 47.8] 57 41.6 No type hints (56.9) Qwen3-Coder-Next vs. MiMo-V2-Pro GPT-5.3-Codex MiMo-V2-Pro 252 43.3 ∗ [37.3, 49.4] 64 43.5 Has type hints (56.9) Qwen3-Coder-Next vs. MiMo-V2-Pro Grok-Code-Fast-1 Qwen3-Coder-Next 252 39.7 ∗∗ [33.8, 45.8] 50 39.7 No type hints (56.9) Claude Opus 4.6 vs. Gemini 3.1 Flash Lite GPT-5.3-Codex Claude Opus 4.6 257 7.8 ∗∗∗ [5.1, 11.7] 54 7.7 No docstring (98.4) Claude Opus 4.6 vs. Gemini 3.1 Flash Lite GPT-5.3-Codex Gemini 3.1 Flash Lite 257 31.1 ∗∗∗ [25.8, 37.0] 39 31.0 Has docstring (98.4) Table 2: Target identification on MBPP. The judge is told both authors’ names and asked which solution the target wrote. N N excludes items with an empty or identical solution. P(A) is the rate of choosing position A, and Pos.-bal. the mean of the accuracies with the target’s solution in position A and in position B; the last column is the best non-LLM heuristic on the same pairs. Starred rows below 50% are significantly below chance. Significance: ∗ p < 0.05 {}^{}p<0.05 , p ∗ ⁣ ∗ < 0.01 {}^{}p<0.01 , ∗ ∗ ∗ p < 0.001 {}^{}p<0.001 . Table 2 reports attribution to a named model by a third-party judge. The easiest pair is Claude Haiku vs. DeepSeek-V3, on which Grok-Code-Fast-1 identifies Claude’s code at 81.6% and DeepSeek’s at 69.0%; the two models also differ strongly in verbosity, and choosing the longer solution scores 89.0% on the same items. Gemini 2.5 Flash on this pair shows what a purely cue-driven judge looks like: it names Claude’s solution as Claude’s (77.3%) and also as DeepSeek’s (DeepSeek accuracy 27.1%), gi To see anthropic in practice, The Ultimate Vibe Coding Guide (2026... walks through a concrete example. as detailed in the full paper on Arxiv The anthropic story also surfaces in Anthropic Researcher Estimates Over 10% Chance..., adding another angle.