Worth Mailing
A charity sends 82,000 mailers at $2 each to households that might donate, ranked by a model's predicted probability. That only works if a score of 0.8 really means an 80% chance of donating. This project measures that gap on the KDD98 charity mailing dataset, corrects it with three calibration methods across three model families, and checks whether the fix is worth the compute it costs to produce.
Same dataset, three models, three fixes
Logistic regression, random forest, and XGBoost are trained on the same 82,318-household KDD Cup 1998 dataset, then each is corrected with Platt scaling, isotonic regression, and temperature scaling, a 3x4 grid of before-and-after Expected Calibration Error. Logistic regression comes out well-calibrated on its own; the tree-based models don't, and not by the same amount or in the same shape.
Isotonic regression cut the random forest's ECE from 0.0334 to 0.0046.Platt scaling helped the same model, but made XGBoost's calibration worse.
Pricing the fix, not just scoring it
A lower ECE number doesn't mean anything to whoever approves the mailing budget, so every calibration result gets translated into a simulated mailing run. At 10,000 pieces, the isotonic-calibrated contact list cost $84.61 less than the uncalibrated one for the same donation revenue; at 100,000 pieces that gap scales to $846.10. Every training and calibration run is also logged with CodeCarbon, so the calibration gain is weighed against the energy it cost to produce, not just reported on its own.