add NL->Code and Code->Code category averages, fix row highlighting
Browse files
README.md
CHANGED
|
@@ -53,13 +53,17 @@ potion-code-16M is created using the following pipeline:
|
|
| 53 |
|
| 54 |
Results on the [CoIR benchmark](https://github.com/CoIR-team/coir) (NDCG@10, `mteb>=2.10`):
|
| 55 |
|
| 56 |
-
| Model | Params |
|
| 57 |
-
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| 58 |
-
| CodeRankEmbed | 137M |
|
| 59 |
-
| BM25 | — |
|
| 60 |
-
| **potion-code-16M** | **16M** | **37.05** | **3.97** | **42.99** | **36.26** | **50.27** | **43.40** | **39.76** | **31.72** | **21.37** | **57.47** | **43.34** |
|
| 61 |
-
| potion-retrieval-32M | 32M |
|
| 62 |
-
| potion-base-32M | 32M |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
|
| 64 |
## Model Details
|
| 65 |
|
|
|
|
| 53 |
|
| 54 |
Results on the [CoIR benchmark](https://github.com/CoIR-team/coir) (NDCG@10, `mteb>=2.10`):
|
| 55 |
|
| 56 |
+
| Model | Params | AVG | NL→Code | Code→Code | AppsRetrieval | COIRCodeSearchNet | CodeFeedbackMT | CodeFeedbackST | CodeSearchNetCC | CodeTransContest | CodeTransDL | CosQA | StackOverflow | Text2SQL |
|
| 57 |
+
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| 58 |
+
| CodeRankEmbed | 137M | 59.14 | 58.60 | 59.68 | 23.46 | 94.70 | 42.61 | 78.11 | 76.39 | 66.43 | 34.84 | 35.92 | 80.53 | 58.37 |
|
| 59 |
+
| BM25 | — | 39.11 | 30.07 | 48.16 | 4.76 | 32.45 | 59.69 | 67.85 | 33.00 | 47.29 | 32.97 | 15.53 | 69.54 | 28.07 |
|
| 60 |
+
| **potion-code-16M** | **16M** | **37.05** | **33.83** | **40.28** | **3.97** | **42.99** | **36.26** | **50.27** | **43.40** | **39.76** | **31.72** | **21.37** | **57.47** | **43.34** |
|
| 61 |
+
| potion-retrieval-32M | 32M | 32.10 | 27.58 | 36.61 | 4.22 | 31.80 | 36.71 | 45.11 | 38.64 | 29.97 | 32.62 | 8.70 | 56.26 | 36.93 |
|
| 62 |
+
| potion-base-32M | 32M | 31.42 | 27.96 | 34.88 | 3.37 | 29.58 | 34.77 | 42.69 | 37.88 | 28.51 | 30.55 | 14.61 | 53.36 | 38.88 |
|
| 63 |
+
|
| 64 |
+
**NL→Code** tasks: AppsRetrieval, COIRCodeSearchNetRetrieval, CosQA, StackOverflowQA, SyntheticText2SQL
|
| 65 |
+
|
| 66 |
+
**Code→Code** tasks: CodeFeedbackMT, CodeFeedbackST, CodeSearchNetCCRetrieval, CodeTransOceanContest, CodeTransOceanDL
|
| 67 |
|
| 68 |
## Model Details
|
| 69 |
|