FIX MLP squared_error loss now matches the gradient for multi-output targets - #34830

Reference Issues/PRs

Closes #8349. Detailed evidence and the comparison of the two candidate fixes are in this comment.

What does this implement/fix? Explain your changes.

_loss_grad_lbfgs returns a (loss, grad) pair, and lbfgs line-searches on the loss while stepping along the gradient. For multi-output regression with squared_error these are not the same function.

squared_loss reduces over outputs with .mean(), while _backprop computes deltas[last] = activations[-1] - y and divides only by sw_sum. That is the canonical loss-link derivative of a per-sample squared error summed over outputs, so the loss is n_outputs times smaller than the function the gradient belongs to.

Central-difference check of the returned pair against itself:

The factor is exact in all 24 configurations I tried (hidden_layer_sizes (5,) and (6, 4), alpha 0 and 1e-3, 3 seeds), so it is not a conditioning artefact.

squared_loss is the only loss that does this. poisson_loss, log_loss and binary_log_loss all reduce with .sum(), and all of them pass the same check. This PR makes squared_loss use .sum() too.

The gradient is not touched, so sgd and adam produce bit-for-bit identical coefficients. Only lbfgs, which actually consumes the loss, is affected. The alternative fix, scaling deltas[last] down by n_outputs, also restores consistency, but it changes the effective learning rate for every solver and needs loss-specific special-casing inside _backprop, so I did not take it. Happy to switch if maintainers prefer it.

Introduce yourself

I have been using scikit-learn for several years. It was one of the first libraries I worked with when I started doing AI and data analysis work, and I have kept using it since. Over that time I have made a point of understanding the libraries I depend on rather than treating them as black boxes, and I decided I would like to help with development instead of only consuming it. This is my first contribution here. I went through the open issues looking for numerical correctness bugs, found that this one still reproduces on main nine years after it was reported, and worked it up.

AI usage disclosure

I used AI assistance for:

• Code generation
• Test/benchmark generation
• Documentation
• Research and understanding

I reviewed all of it before pushing. I have read the change, the test and the changelog entry, I understand why summing over outputs is the correct reduction here, and I can explain any part of it on request. Every number quoted in this description came from running the code against a build of main rather than being asserted, and all of it is reproducible from what is written above.

Any other comments?

This does not improve predictive accuracy, and I would rather say so than oversell it. Over 12 seeds on multi-output regression, test MSE improved on 6 of 12. Excluding one outlier seed the mean is marginally worse, 0.03258 against 0.03369. The case for the change is correctness: right now the optimiser follows a gradient that does not belong to the objective it is evaluating.

There is one behaviour change to weigh. loss_, loss_curve_ and best_loss_ scale by n_outputs for multi-output squared_error. Since tol is compared against loss changes, convergence becomes effectively n_outputs times stricter, so lbfgs tends to run longer, median 536 against 594 iterations over 15 seeds. A changelog entry is included. Let me know if this also warrants a changed-models note.

On testing: test_gradient only exercises MLPClassifier, whose loss already uses .sum(), which is why the regressor path went unchecked for so long. This PR adds test_gradient_regressor, parametrised over loss and n_outputs.

• The new test fails on main for squared_error with n_outputs 2 and 3, and passes for poisson and for single output, so it is specific to the defect.
• pytest --pyargs sklearn.neural_network: 91 to 97 passed (6 new cases), 2 skipped, no regressions.
• Common estimator checks for MLP: 124 passed, 13 skipped.
• Everything matching mlp or MLP or neural: 222 passed, 18 skipped.
• MLPRegressor and MLPClassifier doctests are unchanged, verified against a pristine control, since the regressor example is single output and the two reductions coincide there.