drdavidprivacy/limitwise-model

LimitWise v2 is the version the Chief AI Officer signed in Week 6 of the AI Governance Practicum (Track 1). The organization, the system, and the people are fictional; the data is public. Educational use only, not for real decisions.

What is in this repository

• limitwise_v2.pkl: 18,636 bytes, SHA-256 9010c6ff9746c5dbf9cebff4c746832864c4f2321d016cc5f3b226847882f705
• limitwise_v2_recalibrated.pkl: 89,015 bytes, SHA-256 b6508dfcc69ca108e3f08fc2be0843dab1d2517355970b8e622e52a34127ca37
• limitwise_v2_Inside.csv: every fitted column with its weight, and the intercept, as a table a person can read.
• Model-Card_Becker-exemplar.pdf: the card below as a PDF.

Lineage: trained on the remediated working file in drdavidprivacy/limitwise-data, which links to the file itself in drdavidprivacy/practicum-case-packs. Served by the Space drdavidprivacy/limitwise. Libraries: scikit-learn 1.9.1, fairlearn 0.15.0; the model does not load on another version of scikit-learn, by design.

Model card below: the AI Governance Lead's exemplar (Tom Becker), the answer key to the Week 6 Model Card. Students replace it with their own; see "Make this your own" at the end.

---

Model Card: LimitWise v2 (Corvana Bank)

Analyst: Tom Becker, AI Governance Lead (exemplar) Organization: Corvana Bank System: LimitWise Version documented: v2 (recommended for signature), compared with v1 Evidence: Week 6 evaluation notebook, sections 1 to 15, and the documents cited by name

Section 1: Model details

Objective: Identify the model a reader is holding, and every fitted step production has to reproduce.

1.1 Model name and organization

LimitWise, Corvana Bank's early-warning model for payment default in the next statement cycle (organization briefing).

1.2 Version documented

This card documents v2, the version I recommend for signature (section 8). It was compared with v1, the champion the Chief AI Officer ratified in her Week 4 memo (notebook, sections 2 and 5).

1.3 Model family and fixed configuration

Logistic regression, regularization C = 0.1, balanced class weights, scaled inputs. v1 is fitted on 27 inputs encoded as 98 columns. v2 is fitted on 93 columns after the four demographic fields are removed (Week 4 rationale, section 2, by reference, and notebook, sections 2 and 5).

1.4 Fitted steps production must reproduce

The training-median fill for the payment ratio on the 3,175 accounts with no prior balance, the correlation remover, the scaler, and the isotonic recalibration (a monotone curve fitted to the cross-validated predictions on the training rows) that section 7 shows the calibration check passing with (notebook, sections 5 and 7, and Week 4 memo, 'What proceeds with an assumption and a control').

1.5 Mitigation

Fairlearn's CorrelationRemover, fitted on the training rows against sex, age, marital status, and education. It removes from every other input the part that moves with those four fields, then drops the four, so the model never sees them (notebook, section 5).

1.6 Artifacts and integrity hashes

limitwise_v2.pkl, 18,636 bytes, SHA-256 7842edc29acd316873ffa482db705ab2ea09427f8b4e1186436713d316de8ecb. limitwise_v1.pkl, 6,432 bytes, SHA-256 afbf9fea245c0d6b6ee9f2b8d7c4e0a115c77314c55891bf3573207c78ace019 (notebook, section 15).

1.7 Owner and holder of this card

The modeling team builds and retrains the model. The AI Governance Office holds this card and the program file. The Chief AI Officer is the approver of record (organization briefing, and Week 2 memo).

Section 2: Intended use

Objective: What the model is for, who uses it, and what it may not do.

2.1 Primary use

Rank the bank's open card accounts each statement cycle by the likelihood of a payment default in the next one, so that trouble is seen a cycle earlier than the behavioral scorecard sees it (Use Case Assessment, and Week 1 decision memo, by reference).

2.2 Intended users

The people who make outreach calls and hardship offers. The score brings an account to a person's attention. The person decides what happens on that account (Week 1 decision memo, 'The decision').

2.3 Posture and phase

Preventive first. The program is not yet in phase one (outreach calls and hardship offers only, a person reviewing every contact). No later phase, where a score could lead to a line decrease, an authorization block, or a closure, has been approved (Week 1 decision memo, and Program Record, Status tab).

2.4 Out of scope

Out of scope today: contacting any cardholder, or touching any line, on a score. The use gate (the Chief AI Officer's permission for either) is closed until the calibration check and the reason-coverage check pass and the Week 2 watch list has run, and the Week 4 memo adds that outreach capacity must be written into the program file first (Week 3 memo, 'My decision on each', and Week 4 memo). A score is never a cardholder's probability of default until the calibration check passes on the version signed (section 7).

Section 3: Factors

Objective: The groups and conditions the model was tested across.

3.1 Groups and the fields that define them

The groups the memos name: sex (the SEX field, 1 male, 2 female), age band (the AGE field in ten-year bands, under 25 to 65 and over), marital status (the recoded MARRIAGE field: married, single, other, unknown), and education (the recoded EDUCATION field: graduate school, university, high school, other, unknown) (Week 2 memo, item four, and datasheet, section 7).

3.2 Demographic fields as model inputs

Model inputs: in v1, education is an input and the other three are held out. In v2, none of the four is an input, and every other input has had the part that moves with the four removed (notebook, sections 1 and 5).

3.3 Groups too small to judge

Groups too small to judge against the tripwire (under 1,000 records in the extract): 55 to 64 (943 in the extract, 207 in the test set), 65 and over (110 and 23), marital status other (323 and 67) and unknown (54 and 4), education other (123 and 29) and unknown (345 and 57). They are reported with their rates and not judged (notebook, section 4).

3.4 Limitation on every group measurement

Every group measurement carries the supplementary-card limitation: the demographic fields describe the primary cardholder's record, which may not be the person who uses the card, until the data owner says what share of accounts carry a supplementary card (Week 3 memo, and Program Record, Items tab, Item 13).

Section 4: Metrics

Objective: Which numbers the card reports, and why those and not others.

4.1 Ranking and count-based metrics

The card reports two ranking metrics: ROC-AUC (how often a randomly chosen defaulter outranks a randomly chosen paying account) and PR-AUC (precision averaged over every possible cut, read against the file's 22.1 percent positive rate). The count-based metrics (flagged, caught, missed, false alarms, precision, recall) are reported at the operating point and not at the 0.50 reference threshold, because the bank acts on a share of the list and 0.50 is a setting on a 22.1 percent file (Week 4 rationale, sections 2 and 3, by reference, and notebook, section 3).

4.2 Operating point

The operating point is the top 10 percent of the ranked list. A probability threshold is not usable: the mean predicted probability on the test rows is 0.431 against a book rate near 0.03, and a threshold moves at every retrain while a share does not (Week 4 memo, by reference, and notebook, section 3).

4.3 Fairness measure

The fairness measure is the group false-alarm check: each group's false alarm rate at the operating point (paying accounts flagged, as a share of that group's paying accounts) against the portfolio's, with the tripwire that no group of 1,000 or more in the extract sits more than 20 percent above the portfolio rate. Equalized odds difference (the largest gap between groups in the miss rate or the false alarm rate) and demographic parity difference (the largest gap in the share flagged) are reported beside it as context. The tripwire decides (notebook, section 4, and Week 2 memo, item four).

4.4 Calibration measure

In each of five score bands, mean predicted probability against observed default on the test set reweighted to the book rate, pass line within 25 percent of observed in every band (notebook, section 7, and Week 2 memo, item one).

4.5 What the card does not report

Accuracy, because flagging nobody is 77.9 percent accurate on this file, and any dollar figure, because the amount unit is unconfirmed (Week 3 memo, the no-dollar-figure rule, and Week 4 audit briefing).

Section 5: Evaluation data

Objective: What the model was tested on, and what that data is not.

5.1 The test set

The test set is 5,999 accounts, 1,327 defaulters and 4,672 paying accounts, 22.1 percent default. It was split off in Week 3, at random by row and stratified on the outcome, 20 percent of the working file, seed 42. It was read once in Week 4 (section 9) to confirm the cross-validation order, and it is read here to evaluate v1 and v2. Nothing in this card was chosen on it (notebook, sections 2 and 3, and Week 3 memo, the decision gate).

5.2 Reweighting for the calibration check

For the calibration check the test set is reweighted so each paying account counts 9.18 times and each defaulter once, which makes it stand in for a book defaulting at 3.0 percent. The reweighting assumes the extract was drawn at random within each outcome. That assumption is the open item on the sampling method, and if the sampling memo says otherwise the calibration tables are rerun (notebook, section 7, and datasheet, the open item on the sampling method).

5.3 What the test set is not

The test set is not the bank's book. It is 5,999 accounts from an extract enriched to 22.1 percent default. No count in this card is a count on the book, and the 65 and over group that holds 23 accounts here would hold about 18,000 cardholders there (Week 3 memo, and Week 2 memo, item four).

Section 6: Training data

Objective: Where the model learned.

6.1 Working file

The working file is corvana_modeling_extract_remediated.csv: 29,995 accounts, 27 model inputs, 22.1 percent default (6,635 accounts), MD5 f79f13a5119f78572e9f6cc3243ea6ae (datasheet, by reference, and notebook, section 1).

6.2 Training rows

23,996, at 22.1 percent default (notebook, section 2).

6.3 Remediation behind the file

The file is the output of the twelve-row remediation log the Chief AI Officer ratified in Week 3: the status recode, the education and marriage recodes, the two derived ratios, the five removed duplicate rows, and the recorded split (datasheet, sections 11 and 12).

6.4 Open items on the data

Open items on the data that still stand: Item 02, repayment-status codes 0 and -2. Item 05, undocumented education and marriage codes. Item 06, sampling method and calibration against the book. Item 10, amount unit. Item 11, repeated patterns. Item 12, one rule for six months. Item 13, supplementary-card demographics (Program Record, Items tab).

Section 7: Quantitative analyses

Objective: The results, read against the fold ranges, the tripwire, and the pass lines, for the version you will recommend.

7.1 Table A

Table A. v1 against v2 on the test set.

	v1 (as trained)	v2 (correlation removed)	Notebook section
Cross-validation ROC-AUC, mean (spread)	0.774 (0.018)	0.765 (0.014)	6
Test-set ROC-AUC	0.772	0.758	6
Test-set PR-AUC	0.530	0.526	6
Top 10 percent: flagged	600	600	6
Top 10 percent: defaulters caught	410	402	6
Top 10 percent: defaulters missed	917	925	6
Top 10 percent: false alarms	190	198	6
Top 10 percent: precision	0.683	0.670	6
Top 10 percent: recall	0.309	0.303	6
Groups that tripped the false-alarm tripwire	under 25 (45 percent above), high school (57 percent above)	under 25 (39 percent above), high school (36 percent above)	4 and 6
Calibration after fitted recalibration (pass or the failing band)	pass, worst band 15.7 percent of observed	pass, worst band 8.7 percent of observed	7

7.2 v1 against the Week 4 fold range

v1 against the Week 4 fold range: ROC-AUC 0.772 sits inside 0.763 to 0.780, PR-AUC 0.530 inside 0.515 to 0.557, precision and F1 at 0.50 inside their ranges. Recall at 0.50 is 0.570, below its range of 0.582 to 0.610, a dip of about a hundredth in a count-based metric at a threshold the bank does not use. The ranking metrics hold, so the Week 4 choice stands (notebook, section 3).

7.3 Fairness

Two groups trip for v1: under 25 at a false alarm rate of 0.059 against the portfolio's 0.041 (45 percent above), and high school at 0.064 (57 percent above). Both still trip for v2: under 25 at 0.059 against 0.042 (39 percent above), high school at 0.058 (36 percent above). Sex and marital status pass for both. Equalized odds difference on education falls from 0.074 to 0.032 and on age band from 0.172 to 0.149. Removing the four fields shrank both gaps and closed neither, so the version I recommend cannot be chosen on the tripwire alone. It is chosen on what it costs against what it moves (section 8) (notebook, sections 4 and 6).

7.4 Calibration

Calibration, v2. Raw: every band fails, the top band predicting 0.405 against 0.032 observed. After the odds correction alone: the top band fails at 30.0 percent of observed. After the fitted recalibration and the odds correction: every band passes, the worst at 8.7 percent, with the ranking unchanged (rank correlation 0.998). v1 passes the same way, worst band 15.7 percent. A score from either version is a ranking until both fitted steps run in production. With them, it is a probability the bank may act on (notebook, section 7).

Section 8: Fairness and explainability decision record

Objective: The recommendation, and the evidence it rests on.

8.1 Recommended version

I recommend the Chief AI Officer sign v2.

8.2 The two numbers

V2 gives up 0.014 of ROC-AUC on the test set (0.772 against 0.758) and 8 defaulters out of 600 at the operating point (410 against 402), with 8 more false alarms (190 against 198). It gains on the two groups that tripped: high school falls from 57 percent above the portfolio false alarm rate to 36 percent, under 25 from 45 to 39. 564 of the 600 flagged accounts are on both lists (notebook, section 6).

8.3 The group false-alarm tripwire

Neither version clears the tripwire, and the reason is in the label before any model exists: under 25 defaults at 27.2 percent against 20.3 for 25 to 34, high school at 25.2 against 19.2 for graduate school. A model that ranks by default risk flags more of those groups' paying accounts too. Removing the correlation shrank the gap and could not close it. I recommend the item escalate with one question: may phase one, where a false alarm is a call reviewed by a person, open on v2 with the two groups watched monthly under her Week 1 condition four, or is a per-group operating point to be tested first. That is her call, with Customer Advocacy, not this office's (notebook, sections 4 and 6, and Week 2 memo, item four).

8.4 Explanation evidence

V2's score is driven by the most recent month's repayment status (the undocumented no-delinquency code and the two-month delay), the two most recent bill amounts, the credit limit, and recent utilization. For the caught account (working-file ID 15501), the missed account (ID 9857), and the false alarm (ID 13697), the account's own top reason is a two-month delay in the most recent month, worth 1.165 on the model's scale for all three. A notice can say that. It cannot say 'no delinquency recorded, code undefined', the largest driver in the model, until the codes are defined (reason coverage, Week 7). And 16.2 percent of flagged accounts change their top reason between two retrains, so under the Week 2 memo phase two does not open on this evidence (notebook, sections 12 and 13, and Week 1 decision memo, condition three).

8.5 Ruling requested on education

The four fields are worth 0.002 of ROC-AUC, and in v1 'university' is a notice reason for two of the three accounts in section 13. I ask the Chief AI Officer to rule education out of the inputs. v2 already does (notebook, section 11, and Week 2 memo, item five).

8.6 Minority position

The minority position in the Week 4 memo was shadow mode. This evaluation does not change the reason for declining it, and it removes the condition for adopting it, since both versions pass calibration after recalibration (Week 4 memo).

8.7 Rows that go to the Chief AI Officer

Rows that go to the Chief AI Officer: Check 05 (the two groups above the tripwire for both versions), Check 07 (reasons not stable enough for a letter), Check 08 (the model leans on the status codes, not the payment fields), and Check 01 (the recalibration is a new fitted step for her to ratify) (Table B).

Section 9: Caveats and recommendations

Objective: What the card's reader must not assume about the version you recommend, and what happens next.

9.1 Limitation this version adds

The limitation v2 adds: it is 0.014 of ROC-AUC behind v1 and 0.024 behind the challenger (gradient boosting, 0.782), and its probabilities are usable only through two fitted steps, the isotonic recalibration and the odds correction. Skip either in production and every band fails. The items it inherits are in 3.4 and 6.4 (notebook, sections 6 and 7).

9.2 For the Week 8 monitoring plan

The calibration check monthly by band, the group false-alarm check monthly with contact rates, the reason-stability check at every retrain, and the flagged share against outreach capacity (Program Record, Checks tab, and Table B).

Section 10: The closed-out watch list

Objective: Every check on the Program Record, closed out in writing.

10.1 Table B

Table B. The checks, closed out.

ID	Check	Status	Result against the pass line	Where it is printed
Check 01	The calibration check	Passed, with a new fitted step	v2: raw, every band fails. Odds correction, top band 30.0 percent of observed. Fitted recalibration, every band within 25 percent, worst 8.7 percent. v1 passes the same way, worst 15.7 percent.	Notebook, section 7
Check 02	The merged-codes check	Passed	1.8 percent of the top 10 percent list changes (11 of 600). Challenger 3.3 percent. Pass line 5 percent.	Notebook, section 8
Check 03	The reason-coverage check	Not due this week	Runs in Week 7 before any notice is designed. The undocumented code is the top driver of the score (section 13), so it will matter.	Program Record, Checks tab
Check 04	The unknown-codes check	Passed	Largest move in any per-group rate, codes merged against separate: 0.2 points. Pass line 1 point.	Notebook, section 10
Check 05	The group false-alarm check	Failed and escalated	v1: under 25 at 0.059 against portfolio 0.041 (45 percent above), high school 0.064 (57 percent above). v2: 0.059 against 0.042 (39 percent), 0.058 (36 percent). Pass line 20 percent above.	Notebook, sections 4 and 6
Check 06	The demographic-fields check	Passed	Part one: 0.775 with all four fields against 0.773 with none, gap 0.002, pass line 0.005. Part two: no field guessed more than 6.8 points above its majority rate, pass line 10.	Notebook, section 11
Check 07	The reason-stability check	Failed and escalated	Top reason changes for 16.2 percent of 600 flagged accounts (97). Family level 12.7 percent. Challenger 16.2 and 5.3. Pass line 10 percent.	Notebook, section 12
Check 08	The STATUS-versus-PAID check	Failed and escalated	54.2 percent of the top 10 percent list changes (325 of 600). ROC-AUC 0.772 against 0.715. Pass line 5 percent.	Notebook, section 9
Check 09	The no-dollar-figure rule	Checked by inspection	No dollar figure in this card. The amount fields are read as relative values only.	This card
Check 10	The single-cardholder rerun	Waiting on an answer	The data owner has not reported the supplementary-card share. The limitation is written on every group rate in sections 3 and 7.	Program Record, Items tab
Check 11	The identifier question	Waiting on an answer	None of the five removed rows has been reported as a distinct account.	Program Record, Items tab
Check 12	The label-definition question	Waiting on an answer	The Chief Credit Officer's office has not confirmed the match to the loss forecast. LimitWise stays a ranking, not a forecast input.	Program Record, Items tab
Check 13	The test-set-read-once rule	Passed in Week 4	Confirmed here: v1's test-set ROC-AUC 0.772 sits inside its fold range 0.763 to 0.780, and nothing in this card was chosen on the test set.	Notebook, section 3. Week 4 notebook, sections 5 and 9

---

Make this your own

This page carries the AI Governance Lead's exemplar so that the repository is complete on day one. Your portfolio should carry yours. Three steps, about ten minutes:

1. On Hugging Face, the + menu, then New Model. Owner: your account. Name: limitwise-model. License: cc-by-4.0. Public. Create it.
2. Export your Model Card (your Week 6 deliverable) as a PDF and upload it to the new repository under the name Model-Card.pdf (Files, then Contribute, then Upload files).
3. Upload a README.md: copy README_TEMPLATE.md from this repository, put your name in it, and replace YOUR-ACCOUNT with your Hugging Face account name everywhere it appears (including inside the metadata block at the top, which otherwise stays as it is). The template links to your PDF. If you would rather have your card render as the page itself, paste the text of your Model Card below the metadata block instead.

Then add the new repository to your collection beside your Space and your dataset repository. Do not upload the Lead's exemplar, the memos, the audit briefings, the notebooks, or the data.