krho/openadmet-cyp-tdi-method-report

Predicting CYP Time-Dependent Inhibition: Rank-Averaged Tree Ensembles, with the Operating Point Calibrated from Leaderboard Aggregates

OpenADMET CYP Inhibition Blind Challenge — Method Report Classification (TDI) track · Leaderboard alias KR LAB Bio · KR LAB Bio Inc., Republic of Korea Draft v4, 2026-10-30. No proprietary data was used. No test-set labels were used at any step.

---

1. Summary

This report covers our classification submission, predicting time-dependent inhibition (TDI) of CYP3A4 and CYP2D6 for the 750 blinded test compounds. The submission is an equal-weight rank average of tree-model variants that differ in their fingerprint representation, with a separate ensemble, a separate combination rule, and a separate operating point for each isoform.

Cross-validated macro-averaged MCC is 0.3092, from CYP3A4 MCC 0.4193 (ROC-AUC 0.8262) and CYP2D6 MCC 0.1991 (ROC-AUC 0.6307). On the blinded test set the same ensemble scored MA-MCC 0.3212 at our cross-validated operating point and 0.3384 after that operating point was recalibrated, which is the single largest gain we obtained from any intervention in this challenge. Our v1 draft left open whether the blinded set, whose compounds sit closer to the training data than our scaffold-split folds do, would score more favourably than cross-validation, and noted that only submitting could answer it. It did: 0.3384 against 0.3092, about +0.029. We attribute that to the test set's construction (section 2) rather than to anything in the model, and the same fact is what makes its positive rate recoverable (section 7).

Five findings account for the work, and we consider the last two the contribution worth reading. The first three are what took us from a tuned single model (MA-MCC 0.2927) to the submitted ensemble:

1. The operating point is a call rate, selected outside the scored fold. For an MCC metric on an imbalanced label, where to cut matters as much as how well the model ranks.
2. Ensemble members differ in representation, not in hyperparameters. Five to eleven variants over four additional fingerprint blocks each improve the rank ordering slightly; individually most of them are worse than the baseline.
3. The two isoforms are treated as two separate problems. CYP2D6 has roughly 40% of CYP3A4's labelled compounds, and every selection step that helps CYP3A4 hurts CYP2D6 — including the member selection that is essential on CYP3A4 and harmful on CYP2D6 (section 6).
4. Per-isoform confusion matrices can be recovered from the leaderboard's own aggregate columns, without any label access (section 7). Five macro-averaged figures, plus two submissions differing in exactly one isoform's call rate, determine both isoforms' positives, true positives and scored counts. The recovered positive rates are 1.46 times the training rate on CYP3A4 and 0.31 times on CYP2D6 — opposite directions, which a single macro-averaged figure hides. Lowering the call rate accordingly moved MA-MCC from 0.3212 to 0.3384; every modelling change we tried over the same week moved less than 0.01 on cross-validation. A pre-registered third submission that raised one isoform's call rate back up cost −0.0201, confirming the direction from the other side.
5. An improvement must be located, not just measured. Splitting the out-of-fold gain by nearest-training-neighbour similarity (section 8) shows that our strongest new component improves the stratum holding 0.3% of the test set and degrades the stratum holding 72.3% of it. The same predictions give +0.0062 or −0.0018 depending only on how the strata are weighted. A single cross-validation number cannot distinguish these.

Our negative results are reported in full in section 9: eighteen directions, each with the measurement that closed it. Several looked promising on discrimination (ROC-AUC) yet did not transfer to MCC, and several looked promising on three seeds and vanished on six. We report the measurement resolution that makes the second class of error possible (section 5), because without it a negative result is indistinguishable from an unmeasured one.

2. Task and data

Item	Value
Predicted endpoints	CYP3A4_is_TDI, CYP2D6_is_TDI (binary)
Metric	Macro-averaged Matthews correlation coefficient
Test set	750 compounds, blinded
Training labels	Organisers' cyp-challenge-TRAIN_TDI.csv
CYP3A4 labelled compounds	3,583 (764 positive, 21.3%)
CYP2D6 labelled compounds	1,494 (324 positive, 21.7%)

Training uses the organisers' released data only. We prepared public auxiliary data (PubChem AID 1851, ChEMBL 37 CYP assays, the Octant blog release; 85,491 rows over 29,798 compounds after removing any compound overlapping the test set) and it did improve our regression models, but it did not improve TDI classification in any configuration we measured, so no external data enters this submission. Section 9 gives the numbers.

Test set construction, from the organisers' own description. The challenge documentation states that hit expansion was performed on the top 25 hits from CYP1A2, CYP2C9 and CYP3A4 by purchasing the top 10 ECFP4-nearest catalogue analogues per hit, and that the resulting 750-compound set was then assayed against all four CYPs including CYP2D6. Three consequences shaped our work:

• 25 × 3 × 10 = 750, so the test set is 75 parent series of 10 analogues, not a diverse set. Its compounds sit closer to the training data than a scaffold split implies (nearest-neighbour Tanimoto median 0.470 against 0.338 for our CYP2D6 folds).
• CYP2D6 was not used to select the parents. The test set is therefore expected to be depleted in CYP2D6 actives relative to a CYP2D6-selected library. The recovery in section 7 puts a number on it: a CYP2D6 positive rate of 0.067 against 0.215 in training, while CYP3A4 moves the other way, to 0.312.
• The organisers state the set contains activity cliffs. We tested whether that structure could be exploited directly (section 9, items 12 and 13); it could not.

3. Features

Two blocks are always present:

• 217 RDKit descriptors via useful_rdkit_utils.RDKitDescriptors, matching the organisers' tutorial baseline.
• Morgan count fingerprint, radius 2, 2048 bits, kept as counts. Bits present in fewer than five training compounds are pruned.

Four further blocks are available to individual ensemble members, each describing a molecule on a different axis:

Block	Width	Rationale
FCFP4 (feature Morgan, radius 2)	2048 counts	Atoms typed by pharmacophoric role rather than element, which is closer to what a CYP active site discriminates: hydrophobic bulk and basic nitrogen.
Morgan radius 3	2048 counts	Separates substituted scaffolds that radius 2 collapses onto shared bits.
Avalon	1024 counts	Path- and ring-based rather than circular, so its errors fall elsewhere.
MACCS keys	167 binary	Hand-curated substructures, which name groups (nitro, sulfonamide, quaternary nitrogen) that a count fingerprint only reaches indirectly.

Computing all four blocks for 35,391 compounds takes 7.2 seconds; no molecule failed.

4. Models

Two families, each a member of the ensemble under several feature configurations:

• ExtraTrees — 600 trees, min_samples_leaf=2, max_features="sqrt".
• LightGBM in two deliberately different shapes: a tuned setting (600 estimators, learning rate 0.03, 31 leaves, min_child_samples=15, colsample_bytree=0.4, subsample=0.8, reg_lambda=1.0) and a deep setting (800 estimators, learning rate 0.02, 127 leaves, min_child_samples=5, colsample_bytree=0.25, subsample=0.7, reg_lambda=3.0). The second exists to be wrong in different places, not to be better.

A third family, a shared-representation multi-task MLP, was built and measured but is not in the submission; section 9 item 15 reports why.

Class balance is applied through sample_weight at fit time only. An earlier version of our code applied both sample_weight and the estimator's own class_weight; at a positive rate of 0.217 each is about a 3.6-fold up-weight, so the two together reached roughly 13-fold. Removing the duplication changed the ranking of our candidate settings.

One classifier is trained per scored isoform. Pooling isoform rows into one shared model helps the regression track and hurts TDI, so we do not use it here.

5. Validation protocol and measurement resolution

Everything is measured with a protocol that mirrors what a submission actually does. Two corrections during development came from exactly this comparison, so we state it explicitly.

• Split: 5-fold, grouped by Murcko scaffold at the compound level, shared across isoforms. Seeds 0 to 5 are available for CYP3A4 and seeds 0 to 2 for CYP2D6; we report the mean and the spread.
• Scoring: out-of-fold predictions are pooled and scored once, because the organisers score all 750 test compounds at once.
• Operating point: for each fold, the call rate maximising MCC on the other folds is selected from a grid of 0.04 to 0.60 in steps of 0.01. The threshold is then taken as that quantile of the held-out fold's own score distribution, because the submission cuts the test set at its own quantile. Taking the training folds' quantile instead reads about 0.01 MA-MCC high on CYP2D6, where the two distributions differ most.
• Member selection (CYP3A4) is likewise fitted on the other folds within each fold, so the reported figure includes the cost of choosing.
• Three evaluation populations. Beyond the fold-uniform pooled score we also report two reweighted populations, obtained by matching the similarity-stratum proportions of the test set (population A) and by restricting to nearest-neighbour Tanimoto at or above 0.45 (population B). Population A is the one the submission is scored against; section 8 shows that it and the fold-uniform score can disagree in sign.

Measurement resolution. Paired per-seed differences between two configurations have a standard error of 0.0066 at six seeds. Any claimed gain below about 0.02 is therefore not separable from seed noise, and we adopted the following pass criteria before measuring any of the later candidates: mean ΔMA-MCC ≥ 0.020, positive in at least five of six seeds, paired t ≥ 3, and no isoform worse than −0.010.

Candidates were written down before being measured, rather than selected after a sweep, because with more than twenty candidates and a seed standard deviation near 0.01 the maximum of a sweep is not an estimate of anything. Where a comparison involves a new degree of freedom we say so explicitly and treat the result as unproven: section 8's stratum boundary is such a case.

This is not a formality. Our boundary-weighting candidate measured +0.0100 on three seeds, with the spread halved, and looked like the best idea we had had in a week. On six seeds it measured +0.0019, t = 0.29, with per-seed differences of −0.0056, +0.0187, +0.0171, −0.0246, +0.0055 and +0.0002 — three positive, three negative. Had we stopped at three seeds we would have reported a false positive in this document. We state the resolution so that a reader can see which of our negative results are "measured to be zero" and which are only "too small for us to see".

Neural-network runs must record the thread count. With the same seed, the same data and the same number of epochs, changing the PyTorch thread count moved fixed-call-rate MCC by 0.0057, comparable to the resolution above. All MLP results here use 7 threads.

6. The submitted ensemble

Isoform	Members	Combination	Out-of-fold ROC-AUC	Out-of-fold MCC	Call rate (CV)	Call rate (submitted)	Positives called
CYP3A4	11 of 21 candidates	Top-k by out-of-fold ROC-AUC, then equal-weight percentile-rank average	0.8262	0.4193 ± 0.0083	0.36	0.30	270 at 0.36, 225 at 0.30
CYP2D6	5 ExtraTrees variants	Equal-weight percentile-rank average	0.6307	0.1991 ± 0.0197	0.12	0.12	91 at 0.12, 75 at 0.10

Positives called are given because section 7 inverts them; note that a call rate of 0.12 on 750 compounds yields 91 rather than 90 calls, because the score distribution has ties at the quantile.

Cross-validated MA-MCC 0.3092. Ranks use average ranks for ties, so the result does not depend on input order; an earlier argsort(argsort(x)) implementation broke ties by array position, which matters because tree models emit identical probabilities often.

The CYP3A4 members span both model families and several fingerprint configurations; the CYP2D6 members are ExtraTrees over five fingerprint configurations. Member selection is deliberately absent on CYP2D6: top-k raises its ROC-AUC (0.6301 to 0.6361) and lowers its MCC (0.2045 to 0.1876), because with roughly 1,490 labels the selected k changes from fold to fold, the score distribution changes with it, and the cut point becomes unstable. The chosen call rates concentrate between 0.09 and 0.15 under rank averaging and scatter between 0.04 and 0.34 under top-k.

The submitted CYP3A4 call rate differs from the cross-validated one; section 7 explains why.

Thresholding ceiling. Choosing each fold's call rate with that fold's own labels, which no submission can do, gives MA-MCC 0.3318 against 0.3092. The remaining 0.0226 is what a better operating point could still be worth on this population; anything beyond that requires better ranking, not better thresholding.

7. Calibrating the operating point from leaderboard aggregates

This section describes the only intervention that produced a leaderboard gain larger than our measurement resolution, and the arithmetic that let us check, after three submissions, whether the operating point we had reached was the right one. It uses no test-set labels and no information the organisers have not published for every participant.

What the leaderboard publishes. Every entry carries five macro-averaged figures: MCC, Accuracy, Precision, Recall and F1. Macro-averaging here is the unweighted mean of the two isoforms' values. Two of our submissions differ in exactly one isoform's call rate — CYP3A4 held at 225 calls while CYP2D6 went from 75 to 91 — so the CYP3A4 term is common to both and cancels in any difference between them. That is enough to pull the average apart.

The arithmetic. Write c for calls, t for true positives, p for positives and n for the number of scored compounds, with subscript 2 for CYP2D6 and a prime for the third submission. Each difference isolates one unknown:

ΔPrecision = t₂·(1/c₂ − 1/c₂′) / 2                  → t₂
ΔF1        = t₂·(c₂′ − c₂) / ((c₂+p₂)·(c₂′+p₂))     → p₂
ΔAccuracy  = −(c₂′ − c₂) / (2·n₂)                   → n₂

The first and third hold because t₂ is the same in both submissions, which Recall tells us directly (below). Precision's absolute value then gives CYP3A4's true positives, Recall gives its positives, and Accuracy's absolute value gives its scored count. Four of the five published figures are used, and the fifth is left over as a check.

What it recovers.

	Scored	Positives	Positive rate	Calls	True pos.	False pos.	False neg.	MCC
CYP3A4	628	196	0.312	225	125	100	71	0.393
CYP2D6	727	49	0.067	75	20	55	29	0.270

Three checks, because an inversion this convenient deserves them.

• The unused fifth figure. MCC took no part in the inversion. The recovered cells give MA-MCC 0.3358 for submission 2 and 0.3156 for submission 3, against 0.3384 and 0.3183 published — within 0.0026 in both cases. An independent equation agreeing to 0.8% is not coincidence.
• The scored set is a subset of the 750, of unknown size. The organisers state that "only compounds whose label can be assigned with confidence contribute to the score", so some compounds we called may not be scored at all and our c values are upper bounds. Redoing the inversion with 85% to 100% of the submitted calls assumed to be scored moves the recovered positive rates by less than 0.004, and leaves the ratio of calls to positives on CYP2D6 at 1.54–1.55 throughout. That ratio is what the decision below rests on, and the scored-set size cancels out of it.
• No exact integer solution exists, and that is expected. An exhaustive integer search over all nine unknowns returns nothing. The reason is in the organisers' own code: the leaderboard column is {metric}_mean (leaderboards.py), the mean over 1,000 bootstrap resamples, not the point estimate. Bootstrap means of ratio statistics do not land on rationals with small denominators. This is therefore a recovery to three significant figures, not an exact decode, and we describe it as such.

The two isoforms move in opposite directions, and our earlier reading was wrong. The recovered positive rates are 0.312 for CYP3A4 and 0.067 for CYP2D6, against a training positive rate near 0.215 for both: 1.46× and 0.31×. Earlier in this work we read a single figure of "about 0.73× overall" from the ratio of macro Precision to macro Recall. That step does not hold. For one isoform Precision/Recall = positives/calls, but the ratio of the two macro means is not the mean of the two ratios, and here the two ratios are far apart. The 0.73× figure was an artefact of averaging across isoforms whose positive rates differ by a factor of five. The construction described in section 2 explains the direction: parent compounds were selected on CYP3A4 activity, so CYP3A4 is enriched for inhibitors while CYP2D6, along for the ride, is depleted.

Where the optimum lies. Three methods — the one we had been using and two added after submission 3 — now agree that the submission-2 operating point is already at the optimum.

Method	CYP3A4 optimum	CYP2D6 optimum	Implied gain over 0.30/0.10
Out-of-fold simulation, positives thinned to the recovered rate	0.36–0.40	0.09	+0.009 to +0.020
Binormal model at the discriminability recovered from the test set	0.282	0.106	+0.0002
The three submissions themselves	0.30 ≈ 0.36	0.10 > 0.12	—

The first two disagree, and the reason settles which to trust. The simulation carries the models' cross-validated discriminability — AUC 0.826 on CYP3A4 and 0.631 on CYP2D6 — into a thinned population. But the recovered confusion matrices imply AUC 0.779 and 0.795 on the test set: CYP3A4 ranks slightly worse than cross-validation promised, and CYP2D6 ranks very much better. The simulation's CYP2D6 optimum of 0.09 is thus an answer to the wrong question, calibrated to a model that ranks far worse than ours actually does on this set. Feeding the recovered discriminability into a binormal model instead, the MCC-versus-call-rate curve is flat near its peak: 0.26 to 0.33 lies within 0.002 on CYP3A4, and 0.08 to 0.12 within 0.002 on CYP2D6. There is nothing further to win here, and we stopped turning the lever.

Outcome. Lowering both call rates (submission 2, 0.30/0.10) moved the leaderboard score from 0.3212 to 0.3384, which is +0.0172 — about 2.6 times our measurement resolution, and larger than the sum of every representation and model-family gain we found (+0.0098). Precision rose from 0.3761 to 0.4135 while Recall fell from 0.5667 to 0.5290, the expected signature of cutting less deeply on a set with fewer positives. The simulation had predicted +0.0021 against +0.0172 measured, underestimating by a factor of eight — for the reason now identified: it had CYP2D6's discriminability badly wrong.

The pre-registered experiment came back negative. Raising the CYP2D6 call rate from 0.10 to 0.12 cost −0.0201 MA-MCC (0.3384 to 0.3183), three times our measurement resolution and enough to drop fifteen leaderboard places. The decomposition is unambiguous because only one isoform changed:

	2nd (0.30/0.10)	3rd (0.30/0.12)	Change
MA-MCC	0.3384	0.3183	−0.0201
Macro Precision	0.4135	0.3896	−0.0239
Macro Recall	0.5290	0.5290	0.0000

Recall did not move at all, to four decimal places, while sixteen further compounds were called. Recall's numerator is the true-positive count, so those sixteen calls returned no additional positives: every one of them was a false positive. We pre-registered that a decline here would mean reverting to the submission-2 configuration and closing the call-rate lever, and that is what we did.

One remark on how much the negative result proves. At the recovered discriminability, the binormal model expects raising CYP2D6 from 0.10 to 0.12 to cost about 0.0002 MA-MCC, not 0.0201. Sixteen calls yielding zero positives where two or three were expected has probability near 7% — uncommon, not impossible. The measured decline is larger than the mechanism can account for, so part of it is sampling noise. This does not change the decision, which was pre-registered and which both independent estimates support; it changes what we claim from it. The defensible statement is that the operating point is flat near 0.30/0.10 and that moving away from it in either direction is not worth a submission slot — not that 0.12 is specifically bad.

On the propriety of this. The leaderboard's aggregate columns are published by the organisers for every entry, and the submission interval (12 hours) is theirs. We did not access, infer or approximate any individual test-set label, and no test compound's identity enters any fitted model. What we used is a published summary statistic about our own submissions, inverted with arithmetic. A challenge that publishes five macro-averaged figures per entry is publishing enough to recover per-isoform confusion matrices from any two entries that differ in one isoform; we think it more useful to say so plainly in a method report than to leave it as an advantage for whoever notices.

8. Where a gain lands: similarity-stratified contribution

A cross-validation score answers whether a change helps on the validation population. It does not answer whether it helps on the population being scored. We found that these can differ in sign, and the diagnostic that revealed it is simple enough to recommend generally.

Stratify every training compound by its nearest-training-neighbour Tanimoto similarity, score the submission's predictions within each stratum, and compare the strata's share of the training data against their share of the test set. For CYP3A4, adding ten bagged multi-task MLP members to the eleven-member ensemble gives:

Stratum	Training compounds	Positives	Share of test set	MCC before	MCC after	Difference	t
0.00–0.35	308	51	0.1%	0.3503	0.3432	−0.0071	−0.53
0.35–0.45	1,177	277	0.3%	0.3540	0.3725	+0.0184	3.43
0.45–0.55	1,030	279	27.3%	0.3928	0.3991	+0.0064	3.78
0.55–1.01	1,068	157	72.3%	0.5360	0.5301	−0.0059	−1.96

The stratum that improves most is 32.8% of the training data and 0.3% of the test set. The stratum that degrades is 72.3% of the test set. Weighting the same four differences by training count reconstructs +0.00551 against a measured fold-uniform contribution of +0.0062; weighting them by test-set share reconstructs −0.00250 against a measured population-A contribution of −0.0018. MCC does not decompose exactly as a weighted mean, yet both signs and magnitudes come out right and the residual is +0.0007 in both cases.

So the same predictions yield +0.0062 (t = 13.0) or −0.0018 (t = −0.71) according to weighting alone. Reporting only the first would be a true statement about a population nobody is scored on.

The mechanism is dilution where the baseline is already strong. The 0.55+ stratum has the highest baseline MCC of the four (0.5360): the tree ensemble already performs well on compounds with close training neighbours, and adding ten MLP members at 10/21 weight dilutes it. At 1/12 weight (a single MLP member) all four strata move by less than 0.0007 — too small a weight to help or hurt.

We checked whether our earlier null results were cancellations of this kind, and they were not. Three previously closed candidates, measured under the identical protocol, show a maximum |t| of 1.39, 0.96 and 1.16 across the four strata. They did not move anything anywhere; only this one moved two strata with t > 3 and then cancelled. "Cancelled out" and "no signal" are different diagnoses with different remedies, and conflating them would have sent us looking for stratum weighting in cases where there was nothing to weight.

Acting on the diagnosis closed the direction rather than opening it. Using the MLP members only below the 0.55 boundary and the original eleven above it turns all three populations positive: fold-uniform +0.0076 (t 4.44), population A +0.0033 (t 2.33), population B +0.0046 (t 4.43). This is a real effect, 1.8 times larger than the weighted-mean approximation predicted, because using different predictions in different strata also redistributes the global call budget, which the approximation cannot capture. We still did not submit it: ΔMA-MCC is +0.0016, one twelfth of our pass criterion; the boundary at 0.55 is a new degree of freedom that invites exactly the overfitting we had been avoiding; and on the live leaderboard +0.0016 moves us zero places. We record the measured figure rather than the approximation so that the next person reads "measurable but twelve times too small" instead of "not even measurable".

The practical recommendation. When screening ensemble members for a blinded set whose similarity profile differs from the validation split, score the high-similarity stratum separately. A member with t = 13 on the pooled score and a negative sign in the stratum holding 72% of the test set is not a candidate.

9. What did not work

Eighteen directions, each with the measurement that closed it. Reported in full, since several cost more to learn than the method itself. Where a paired t is known it is given; where only a sign is given, the measurement predates the six-seed protocol of section 5 and should be read as "not shown to help" rather than "shown to hurt".

#	Direction	What was done	Result
1	Call-rate tuning	Full 0.04–0.60 sweep	Existing values optimal: CYP3A4 0.33–0.38, CYP2D6 0.12–0.14
2	Member and combination reselection (fold-uniform)	40 candidates, six combination rules, 15-run seed bagging, larger forests, RandomForest members	No configuration beat the incumbent. Log-odds averaging and the per-compound median were about 0.009 below rank averaging; learned logistic weights were below top-k; AUC-proportional weighting moved +0.0021. RandomForest members raise CYP3A4 slightly and collapse CYP2D6 (MCC 0.2243 → 0.1903, under an earlier scoring scope that reads about 0.02 high there)
3	Member and combination reselection (test-matched)	Nested selection under populations A and B	All four cases worse: CYP3A4 −0.0133 and −0.0156, CYP2D6 −0.0581 and −0.0958
4	Another model family (TabICL)	Added as a member	Worse on all three populations at fixed call rate
5	Cross-isoform transfer	CYP3A4 knowledge applied to CYP2D6	Ruled out before training: 259 compounds carry both labels, CYP3A4 scores predict CYP2D6 labels at ROC-AUC 0.5519, phi 0.1605
6	External data (ChEMBL)	4,155 preincubation and time-dependence rows as auxiliary labels	MA-MCC 0.2767 → 0.2544. ChEMBL's assays do not share the challenge's 2-fold-shift definition
7	Pretrained representations	Three representations on three axes (ten TDI comparisons, four regression)	None beat descriptors plus fingerprints. CYP3A4 ROC-AUC 0.812 → 0.806 at best, CYP2D6 0.597 → 0.586
8	Structural alerts	42 literature alerts	Enrichment is real (benzodioxole, 2.64-fold on CYP2D6 positives) but does not reach model performance: only 21 compounds carry that motif
9	Objective: lambdarank	Learning to rank	CYP3A4 −0.0062, CYP2D6 −0.0758 (three seeds)
10	Objective: rank_xendcg	Learning to rank	CYP3A4 −0.0119, CYP2D6 −0.0281 (three seeds)
11	Boundary weighting	2× weight near the top-k boundary, boundary found by inner 5-fold	Three seeds +0.0100 → six seeds +0.0019, t 0.29
12	Matched-pair exchange	Analogue-pair comparison model, swapping across the call boundary	Pair-direction accuracy 65.3% against the ensemble's 70.6%. Oracle ceiling, swapping with labels visible, is MA-MCC +0.0067
13	Series-wise positive allocation	Exploit the 75×10 test structure	Not possible. Training data has no series structure: Butina clustering at distance 0.40 gives 2,792 clusters of which 89.2% are singletons
14	CYP2D6 selection-bias residual	Learn the residual on single-concentration non-hits	Not possible. Zero CYP2D6 TDI labels exist among non-hits
15	Single-concentration multi-task MLP	Shared trunk, six heads (2 TDI classification + 4 single-concentration regression) over 17,504 unused organiser rows	Auxiliary heads raise CYP3A4 MCC +0.0386, t 4.90, taking the MLP above the LGBM baseline (0.3926 against 0.3830), yet ensemble contribution is +0.0009, t 0.76. CYP2D6 −0.0124, t −0.89, not distinguishable from zero
16	Bagging the MLP member	Ten training seeds as ten members	Bagging gain +0.0167 (t 4.14); ensemble contribution +0.0062 fold-uniform (t 13.01) but −0.0018 on population A (t −0.71, not distinguishable from zero)
17	Stratum-gated weighting	Use the new members only below Tanimoto 0.55	Measured +0.0033 on population A (t 2.33), ΔMA-MCC +0.0016 → zero places on the leaderboard
18	Call-rate recombination	Grid over call-rate pairs under each positive-rate assumption, then tested on the leaderboard	The simulation preferred 0.30/0.12 and the leaderboard disagreed by −0.0201 (section 7). The simulation holds discrimination at its cross-validated value, which on CYP2D6 is AUC 0.631 against 0.795 recovered from the test set — so its optimum answers the wrong question. At the recovered discriminability the curve is flat from 0.08 to 0.12 and the lever is closed

Following the label definition. is_TDI is defined by a pIC50 shift exceeding log10(2) = 0.301. Predicting both pIC50s and applying that rule scored 0.0571 against 0.2485 for direct classification at the time, because our pIC50 error (0.5 to 0.65 log) exceeds the threshold the rule tests.

Why the largest unused dataset did not help the isoform that needed it. Item 15 deserves more than a table row, because it failed for a reason that is a property of the data rather than of the model. The organisers' single-concentration screen covers 4,372 compounds across four enzymes (17,504 rows) and never entered any public baseline we saw; CYP2D6 has 2.9 times as many rows there as it has TDI labels, and CYP2D6 is our bottleneck. The auxiliary heads did learn their task (out-of-fold R² of +0.36 for CYP2D6 single-concentration inhibition, Pearson 0.61), so the shared representation did absorb the signal. But every compound carrying a CYP2D6 TDI label is already a single-concentration hit. Labelled rows have log2fc −1.795 ± 0.745 and unlabelled rows +0.308 ± 0.273, and the fraction of labelled rows exceeding the 99th percentile of unlabelled rows is 0.000: the two populations do not overlap. The auxiliary task teaches "which molecules inhibit CYP2D6", and within the labelled set that question is constant. Selection bias in which compounds were sent for TDI measurement makes the extra data uninformative about the labels we must predict, no matter how well it is learned. CYP3A4, whose labelled and unlabelled distributions do overlap (−1.689 ± 1.384 against −2.067 ± 1.325), is the isoform the auxiliary heads helped.

The recurring pattern, and its two causes. Four separate candidates improved a single model and contributed nothing to the ensemble: predicted direct-inhibition pIC50 (+0.0069 alone, 0 in ensemble), TabICL, boundary weighting, and the multi-task MLP. Section 8's stratified diagnostic separates these into two kinds. For the first three, no stratum shows a signal at all (maximum |t| of 1.39, 0.96, 1.16): there was nothing to add, and the remedy would be a genuinely more different model. For the MLP, two strata move with t > 3 and cancel against a third: the remedy is stratum weighting, which works (item 17) and is too small to matter. Diagnosing which case one is in costs one script and changes what to try next.

Discrimination gains that do not become MCC gains. Several changes raised CYP2D6 ROC-AUC (0.6301 to 0.6337) without raising MCC, because the improvement lands in the middle of the ranking rather than in the top 12% where we cut. Reaching the intermediate leaderboard's top statistical tier from our cross-validated position would require about +0.060 CYP2D6 ROC-AUC or +0.045 on CYP3A4, against a measurement resolution of 0.02 and a total found gain of +0.0098. We state plainly that our modelling did not reach that tier, and that the one intervention which did move the leaderboard materially was an operating-point correction (section 7), not a better model.

What the whole set of negative results says about our validation. Our cross-validation read 0.3092 where the test set gave 0.3384 at the same operating point and the same weights. A scaffold split designed to be pessimistic was pessimistic, by about 0.029 — more than four times the resolution at which we were adjudicating candidates. That mismatch is not a nuisance; it is the reason eighteen directions closed. We were selecting components against a population harder and differently composed than the one being scored (section 8 makes the composition explicit), and on that population the available gains are all smaller than the noise. The two things that did work both came from characterising the scored population instead of improving the model on the validation population: recovering its per-isoform composition, and locating where a component's gain falls within it. If we were to start again with the same week, we would spend the first day on the second kind of question.

10. Reproducibility and scope

Fixed seeds (0 to 5 for CYP3A4, 0 to 2 for CYP2D6) throughout; metric implementations were checked against the organisers' custom_scoring_functions.py on 200 random inputs and on the training data, with a maximum difference of 0 and identical bootstrap index matrices. Fold assignments and labels were asserted identical to earlier artefacts character for character before any comparison was reported, because a silent fold change would invalidate every paired test in this document. MLP runs fix the PyTorch thread count at 7 (section 5). All computation ran on commodity hardware: feature computation 73 seconds and 545 MB, one cross-validation run 13 to 480 seconds, peak memory 1.5 GB.

Amendment, 2026-10-30: we have since entered the regression track. Earlier drafts of this report said we had not, and gave as the reason that we would rather report a measured gap than occupy a rank we cannot interpret. The gap is unchanged, and we still report it: our cross-validated MA-ST-RAE is 0.7227 against a live-leaderboard leading entry of 0.3432. What changed is our view that the measurement behind that entry is worth publishing on its own terms. It has its own submission and its own method report: https://huggingface.co/krho/openadmet-cyp-regression-method-report. Section 3 of that report is the part we would point a reader to: it separates ST-RAE's numerator from its denominator per isoform and finds that our macro-averaged score is carried almost entirely by CYP3A4, for a reason that lies in the width of the measurement intervals rather than in the model. We did not enter the structure track.

No proprietary data, no undisclosed measurements, and no test-set labels enter any step: member selection, combination weights and call rates are all derived from out-of-fold predictions on the organisers' training data, except for the call-rate adjustment of section 7, which is derived from the organisers' own published leaderboard aggregates for our own prior submissions.

11. Contact

KR LAB Bio Inc., Republic of Korea. Leaderboard alias: KR LAB Bio.