FIX stabilize weighted confusion_matrix_at_thresholds on float32-only devices - #34827
FIX stabilize weighted confusion_matrix_at_thresholds on float32-only devices#34827ogrisel wants to merge 9 commits into
confusion_matrix_at_thresholds on float32-only devices#34827Conversation
On float32-only Array API devices, float cumsum saturates past 2**24. Weight normalization is not sufficient at that scale; use a fixed-point integer cumulative sum instead when float64 is unavailable. Refs scikit-learn#34813. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop _max_precision_int_dtype; float32-only devices of interest (e.g. torch MPS) provide int64. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Rely on _array_api_for_tests for skipping when array-api-strict or SCIPY_ARRAY_API is unavailable. Co-authored-by: Cursor <cursoragent@cursor.com>
|
cc @david-cortes-intel as we discussed this during a meeting. |
Add sklearn.metrics/34827.fix.rst for PR scikit-learn#34827. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
| # Micro-unit fixed point: enough resolution for typical weights while | ||
| # keeping scaled totals inside int64 for very large n. int64 is assumed | ||
| # available on float32-only devices of interest (e.g. torch MPS). |
There was a problem hiding this comment.
Thanks. This could be mentioned in the documentation so that users would know that granularity of the weights is limited.
| # available on float32-only devices of interest (e.g. torch MPS). | ||
| scale = 1_000_000 | ||
| y_true_i = xp.astype(y_true, xp.int64) | ||
| w_scaled = xp.astype(xp.round(weight * scale), xp.int64) |
There was a problem hiding this comment.
weight = weight / weight.sum() gives mean of 1/n that get rounded to zero so
round(weight * 1e6) returns zero for any n>4e6.
I measured fo U(0,1) weights only 75% survive at n=1e6 and at n=1e7 tps[-1] is 0 and roc_curve returns a NaN where main returns 7.4e-4 relative error.
There was a problem hiding this comment.
@Fazel94 I don't understand what you mean. This code does not divide by weight.sum(). Could you give a reproducer where this PR fails?
There was a problem hiding this comment.
I think I got what you meant. If the weights are normalized ahead of time (e.g. by the user, then using the 1e7 fixed scale can fail).
I iterated on this problem with an LLM in ogrisel#24. Let me merge this iteration into this PR as I actually think it's an improvement.
There was a problem hiding this comment.
Yes that is exactly what i meant
Choose the micro-unit scale from the mean sample weight (capped by int64 headroom via n * max(weight)) so pre-normalized or uniformly tiny weights do not round to zero under a fixed 1e6 scale, while O(1) weights keep the previous micro-unit resolution. Extend the float32-only large-n regression test to cover those failure modes. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
…-4d84 FIX adaptive fixed-point scale for weighted float32-only cumsums
❌ Linting issuesThis PR is introducing linting issues. Here's a summary of the issues. Note that you can avoid having linting issues by enabling You can see the details of the linting issues under the
|
|
In my tests your adaptive scale idea works great in all but one easily rectifiable case. The case goes as tps would be wrong for the starting ones. scale = max(int(0.9 * (2**63 - 1) / (n_samples * weight_max)), 1)Generally summing in descending order is numerically problematic, the idea to use def compensated_cumsum(x):
c = np.cumsum(x, dtype=f32)
a = np.concatenate([np.zeros(1, dtype=f32), c[:-1]])
s = a + x
bb = s - a
e = (a - (s - bb)) + (x - bb)
d = (s - c) + e
return c + np.cumsum(d, dtype=f32)[1]: Higham, Nicholas J. 4.6 In Accuracy and Stability of Numerical Algorithms, 2nd ed. |
Fixes the weighted half of #34813.
Complements #34817, which handles the unweighted case with integer cumsum.
On float32-only array API devices (e.g. array-api-strict's
no_float64, torch MPS),confusion_matrix_at_thresholdsaccumulates weighted counts in float32 when float64 is unavailable. Float32 cumsums saturate past 2**24, which breaks ROC/PR curves and related metrics on large datasets. Weight normalization alone is not enough at that scale.When
sample_weightis set and the device only supportsfloat32, this PR accumulates in fixed-pointint64(round(weight * 1e6)), then converts back tofloat32at the output boundary.The
float64path is unchanged, so this PR should result in a net numerical stability improvement forfloat32devices. We could explore using the fixed-point code path also for devices that supportfloat64operations but I am not sure if this is a good idea or not.Note to reviewers: I think we should review and merge the simpler #34817 fix first and then I can rebase this branch to review it more naturally.