code-embed-v3-src: results and logs
Raw outputs for the paper Does Execution-Verified Near-Miss Training Teach Code Embeddings to Prefer Working Code? A Negative Result (preprint, doi:10.5281/zenodo.24402953). Every number in the paper can be recomputed from the files here. Code and the research log that explains each run (R0-R16): https://github.com/divakantpandey/code-embed-v3
This repo holds results, not a model. The models are code-embed-v3-150m-ft-behaviour (pilot) and code-embed-v3-150m-ft-control (replay-only control).
Layout
| Path | Contents | Research log |
|---|---|---|
pilot/pilot-1/ |
Pilot: status, configs, logs, behaviour eval and MTEB(Code, v1) of base and pilot (A40) | R15 |
pilot/pilot-1-merge/ |
Weight-averaged pilot/base models (0.4, 0.5, 0.7): behaviour eval, MTEB, paired tests | R15 |
pilot/pilot-1-control/ |
Replay-only control; base, pilot and control re-scored on one machine; results/compare/ paired tests (McNemar, bootstrap CIs) |
R16 |
ablate/20261031-092215/ |
Three-seed small-scale comparison (paper Table 1) | R9 |
ablate/20261031-164208/ |
Batch-size bridge (paper Table 2) | R15 |
ablate/<other stamps>/ |
Earlier ablation rounds; stamps are launch times (UTC) | R2-R10 |
dryrun/<stamp>/ |
End-to-end dry-run attempts | R8, R11, R14 |
baselines/ |
UniXcoder-base baseline | |
v2-zeroshot/ |
Zero-shot re-evaluation of the earlier code-embed-v2 model | |
code-embed-v3-src.tar.gz, control/code.tar.gz |
Code snapshots that ran on the GPU machines |
Per-query behaviour outputs are in results/behaviour/*.json; MTEB per-task scores in results/mteb-*/.
Training runs before the float16 master-weight fix (log R6, 2026-10-30) are not valid evidence about data or recipe.
License
Results and logs: CC BY 4.0. Code snapshots: Apache-2.0, as in the GitHub repo.
Citation
@misc{pandey2026nearmiss,
title = {Does Execution-Verified Near-Miss Training Teach Code Embeddings to Prefer Working Code? A Negative Result},
author = {Pandey, Divakant},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.24402953},
url = {https://doi.org/10.5281/zenodo.24402953}
}