feat(constraint.py): Auto-convert hard to soft constraints through soften method

[isanchez-ng]

Closes #782
Hi! This is my first contribution. Let me know if there's something I should work on!

---

Context: Building soft constraints requires a lot of manual work that can be avoided through a method that modifies both the variables (to add the slacks) and the objective function (to add the penalty term). This PR intents to built this new feature.

Changes proposed in this Pull Request

Implementation

I followed the proposed structure for the method, but changed some details that I mention on the next section.
Also, I changed slightly the typehints for objective.__add__. to avoid some mypy errors that weren't actually errors.

How did I test this new feature?

• Built unit tests for both the feature and the wrapper
• Used some manual inspection using this code here:

import linopy
import pandas as pd

add_budget_slack = True
add_risk_slack = False
budget_penalty = 2
penalty_risk = 0

# --- Toy data: just 3 investments ---
investments = pd.Index(["Investment A", "Investment B", "Investment C"], name="investments")
risk = pd.Series([0.5, 0.2, 0.8], index=investments, name="risk")
maximum_risk = 0.1 # 0.1 forces the slack on the budget to act.
expected_return = pd.Series([0.08, 0.03, 0.1], index=investments, name="expected_return")

# --- Model ---
m = linopy.Model()

# --- Variables ---
# Fraction of the portfolio allocated to each asset, between 0 and 1
w = m.add_variables(lower=0, upper=1, coords=[investments], name="weights")
m.add_objective((expected_return * w).sum(), sense="max")


# ---- Constraints ----
# Budget constraint: allocations must sum to 1 (fully invested). Option with and without slack.
if add_budget_slack:
    # new easy version:
    # budget_constraint = m.add_constraints(w.sum() == 1, name="budget")
    # budget_slack = budget_constraint.soften(penalty=budget_penalty)

    # OR fastest version:
    budget_constraint = m.add_constraints(w.sum() == 1, name="budget_constraint", penalty=budget_penalty)
else:
    budget_constraint = m.add_constraints(w.sum() == 1, name="budget_constraint")

# Risk constraint: keep the risk to be lower than 0.5. Option with and without slack.
if add_risk_slack:
    risk_constraint = m.add_constraints((w * risk).sum() <= maximum_risk, name="total_risk", penalty=penalty_risk)
else:
    risk_constraint = m.add_constraints((w * risk).sum() <= maximum_risk, name="total_risk")

m.solve(solver_name="highs", output_flag=False)

w.solution

Open discussions:

• I decided to change the proposed behaviour of returning either a Variable or a tuple of variables, because in my experience this behaviour gets messy when the codebase grows (for example, people would have to manage the doubled behaviour through isinstance(var_name, tuple) through their own codes. Instead of that, I proposed a NamedTuple.
• I decided to raise an assertion at the start of soften() if the model's objective hasn't been defined yet, since soften() adds a penalty term to the existing objective rather than replacing it.
  • It would be technically possible to let soften() run before add_objective(). But doing so creates a weird condition where the line model.objective += penalty things asserts the objective is still empty, so calling a second model.add_objective after soften() would raise an error telling the user to pass overwrite=True. Doing that, however, replaces the whole objective expression, and silently discards the penalty term soften() had already added.
  • soften() also relies on model.sense to pick the correct sign for the penalty term. Since sense defaults to "min" until model.add_objective() is called with a different value, calling soften() first risks silently penalizing in the wrong direction if the user later sets sense="max".
• The .soften method doesn't work for freeze constraints! I propose handling this on another PR if it's strongly needed.

I'm open to discussion on these bullet points if someone else has a better proposal.

Questions that still have to be answered:

• Shoudld I create an example of this in one of the notebooks, or create a new notebook? I don't know exactly what to do here so, any guidance from the mantainers will be appreciated!
• One of the "to-dos" says: A note for the release notes doc/release_notes.rst of the upcoming release is included.. I don't know exactly what they mean with this 🤔. Does this just mean to add a small phrase of what's this doing? (it's my first time colllaborating on this repo). I'd appreaciate a small guidance.

To-Dos:

• Build unit tests for .soften method
• Build wrapper of soften inside model.add_constraint

Checklist

• AI-generated content is marked (see AGENTS.md).
• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Unit tests for new features were added (if applicable).
• I consent to the release of this PR's code under the MIT license.

[codspeed]

Merging this PR will regress 3 benchmarks

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 3 regressed benchmarks
✅ 176 untouched benchmarks
⏩ 181 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
❌	test_to_lp[nodal_balance_sparse-severity=50]	2.8 MB	3.7 MB	-24.14%
❌	test_to_lp[merge_balance-severity=0]	2.7 MB	3.3 MB	-18.3%
❌	test_to_lp[nodal_balance-severity=50]	3.3 MB	3.7 MB	-10.48%
⚡	test_to_lp[expression_arithmetic-n=10]	1,338.6 KB	733.7 KB	+82.45%
⚡	test_to_lp[expression_arithmetic-n=250]	46.7 MB	41.1 MB	+13.64%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing isanchez-ng:master (99908a1) with master (718c0c1)}

Footnotes

1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[FabianHofmann]

@isanchez-ng wonderful initiative and I like the approach! let me know as soon as I should review

[isanchez-ng]

@codspeedbot fix this regression

[isanchez-ng]

@FabianHofmann I think this is ready for review! The one thing I couldn't fix was the performance analysis. If you could guide me a little bit with this I'd be happy to fix them as well.

Heads up that it's a fair amount of new code (+370 lines). Happy to hop on a quick 10-min call to walk you through it if that helps for a faster review :)

[FabianHofmann]

@isanchez-ng great job, the changes look totally reviewable! here is what my agent found (easy to tackle):

---

[F1] Orphan slack variable on mixed-sign constraints (correctness, should fix).
In soften, positive_slack is created via model.add_variables(...) before the mixed-sign check runs:

positive_slack = model.add_variables(...)   # side effect on the model
negative_slack = None
sign_values = pd.unique(self.sign.values.ravel())
if len(sign_values) > 1:
    raise NotImplementedError(...)           # too late

So softening a mixed-sign constraint raises, but leaves a dangling {name}_pos variable registered in model.variables, with the constraint and objective untouched. That breaks fail-fast: it fails and pollutes the model. Fix: read and validate sign_values at the very top of the method, before creating any variable. Cheap and complete.

[F2] penalty=0 is allowed but the docstring and message say it must be positive (minor).
The check is np.all(np.asarray(penalty) >= 0), so 0 passes. But the docstring says "Must be bigger than 0" and the error says "Penalty is not positive." A zero penalty adds a free slack, which silently lets the solver break the constraint at no cost. That is a footgun. Pick one: either forbid 0 with > 0 (matches the docs), or allow it and fix the docs and message. I recommend > 0.

[F3] The add_constraints(..., penalty=...) shortcut throws away the Slack return (API gap, minor).
model.py calls constraint.soften(penalty=penalty) and discards the result, then returns the constraint. So a user of the shortcut cannot get the slack variables except by guessing the derived name f"{name}_slack_pos". That is acceptable for the "fast path", but the derived name is not documented in the add_constraints docstring. Add one sentence naming the convention, so the slacks are reachable.

---

Beyond that:

[F4] should we add a .slack property returning a hidden _slack attribute to Constraint?
[F5] pleas also add release notes :)

Then this is good to go!

[isanchez-ng]

@FabianHofmann sorry for the late response, I'll address this now!
Just a small question though: Is there a proper way of adding release notes? I took a look into release_notes.rst, but I'm not sure about how to add this feature properly. Should I add a small phrase under the "Other" subtitle? As mentioned on the PR body, a little bit of guidance on this would be helpful :)

Also, there's another question that I left there: Should I create an example of this new feature in one of the notebooks, or create a new notebook?

[isanchez-ng]

@FabianHofmann I was addressing what you mentioned, and now I have another question: Do you think we should support adding more than one pair of slack variable on a constraint? What I mean is something like this:

# this line adds the first 2 slacks (positive and negative, because it's an equality)
budget_constraint = m.add_constraints(w.sum() == 1, name="first", penalty=budget_penalty)

# then this other line, adds a second pair of slacks:
budget_constraint.soften(penalty=100, name='second')

What finally gives you a constraint with 4 slack variables (see the image below). I'm not sure if this is something used in the field of optimization (maybe to model two types of costs?). I would say it's redundant, and the code shouldn't allow it, but maybe I'm missing something. Let me know your thoughts on this to correctly address this situation on the code. For now I will explicitly raise an error if the user tries to soften twice.

[FabianHofmann]

no worry about the release note, I can improve them if necessary. about the notebook, this is up to you. and the second order slack, Idk honestly. probably we can leave it and the user and decide whether they want to use it :)

[isanchez-ng]

about the second order slack, Idk honestly. probably we can leave it and the user and decide whether they want to use it :)

Thinking of it, i think there will be two trade offs if we decide to allow the possibility to add more than one pair of slacks for a constraint:

1. the design for the return of these slacks get slighly more complicated. Instead of returning only the positive and negative slack, I'd have to return something like a dictionary of slacks, keyed by the name given by the user. Also, I would have to add an extra guard to avoid the user adding a pair of slacks with the same name.
2. I think this might lead to degenerate solutions in some cases (two identical slack variables tie for absorbing the same violation at equal cost, so the total penalty is right but which slack "took" it is arbitrary and can flip between solver runs). And I'm not sure how to avoid that.

For these reasons, I think it would be better to leave this feature for another issue (if the community asks for it on the first place).

If you agree with this, the PR is ready for review! :)