code-embed-v3-src: results and logs

Raw outputs for the paper Does Execution-Verified Near-Miss Training Teach Code Embeddings to Prefer Working Code? A Negative Result (preprint, doi:10.5281/zenodo.24402953). Every number in the paper can be recomputed from the files here. Code and the research log that explains each run (R0-R16): https://github.com/divakantpandey/code-embed-v3

This repo holds results, not a model. The models are code-embed-v3-150m-ft-behaviour (pilot) and code-embed-v3-150m-ft-control (replay-only control).

Layout

Path Contents Research log
pilot/pilot-1/ Pilot: status, configs, logs, behaviour eval and MTEB(Code, v1) of base and pilot (A40) R15
pilot/pilot-1-merge/ Weight-averaged pilot/base models (0.4, 0.5, 0.7): behaviour eval, MTEB, paired tests R15
pilot/pilot-1-control/ Replay-only control; base, pilot and control re-scored on one machine; results/compare/ paired tests (McNemar, bootstrap CIs) R16
ablate/20261031-092215/ Three-seed small-scale comparison (paper Table 1) R9
ablate/20261031-164208/ Batch-size bridge (paper Table 2) R15
ablate/<other stamps>/ Earlier ablation rounds; stamps are launch times (UTC) R2-R10
dryrun/<stamp>/ End-to-end dry-run attempts R8, R11, R14
baselines/ UniXcoder-base baseline
v2-zeroshot/ Zero-shot re-evaluation of the earlier code-embed-v2 model
code-embed-v3-src.tar.gz, control/code.tar.gz Code snapshots that ran on the GPU machines

Per-query behaviour outputs are in results/behaviour/*.json; MTEB per-task scores in results/mteb-*/.

Training runs before the float16 master-weight fix (log R6, 2026-10-30) are not valid evidence about data or recipe.

License

Results and logs: CC BY 4.0. Code snapshots: Apache-2.0, as in the GitHub repo.

Citation

@misc{pandey2026nearmiss,
  title     = {Does Execution-Verified Near-Miss Training Teach Code Embeddings to Prefer Working Code? A Negative Result},
  author    = {Pandey, Divakant},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.24402953},
  url       = {https://doi.org/10.5281/zenodo.24402953}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support