add qp support for highs #3531

quantresearch1 · 2025-03-21T00:03:37Z

Fixes #3381 .

Summary/Motivation:

HiGHS can solve quadratic programming (QP) models, which contain an objective term x.T @ Q @ x where the Hessian matrix Q is positive semi-definite.

Changes proposed in this PR:

Allow quadratic terms in the objective by changing quadratic to True in generate_standard_repn

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

emma58

@quantresearch1, thank you for the contribution! As written, this does not yet add QP support for Highs, though: By changing the quadratic flag to True in generate_standard_repn, you move the nonlinear term in your objective from repn.nonlinear_expr to repn.quadratic_vars and repn.quadratic_coefs (and hence bypass the error in line 622 of appsi/solvers/highs.py) . To properly send quadratic expressions to Highs, you will need to parse this part of the StandardRepn object and send the result to Highs.

(Your test appears to get lucky--it is actually sending a constant objective to Highs, but the variables happen to land on the values you are expecting. You can see this by adding an assertion that the results object returned by opt.solve(m) agrees with your expected objective value: self.assertEqual(results.best_feasible_objective, 2).)

quantresearch1 · 2025-03-24T17:26:23Z

@emma58 Thanks for the advice. Looking into the Highs C++ code, I did not find a quadratic equivalent to changeColCost so I think each coefficient update would have to be a call to passHessian which is not ideal in a persistent setting (assuming many updates). I will think about this a bit more and reraise if I can come up with a decent solution

quantresearch1 · 2025-03-25T03:54:45Z

pyomo/contrib/solver/tests/solvers/test_highs.py

+    raise unittest.SkipTest
+
+
+class TestBugs(unittest.TestCase):


@mrmundt I have copied the test cases that are passing from the legacy appsi_highs. I am not sure if you wanted to leave the test cases for a later stage, let me know if I should remove them (I do need the new qp test cases though)

quantresearch1 · 2025-03-25T04:01:37Z

pyomo/contrib/solver/solvers/highs.py


        self._solver_model.changeObjectiveSense(sense)
-        self._solver_model.changeColsCost(n, indices, costs)
+        self._mutable_objective = _MutableObjective(


the idea here is we collect all the mutable objective terms and update them once through _mutable_objective.update()

quantresearch1 · 2025-03-25T04:05:01Z

pyomo/contrib/solver/solvers/highs.py

+        self.col_idx = col_idx
+
+
+class _MutableObjective:


@emma58 I have tried to emulate the new gurobi_persistent and use a similar objective data structure. However, highs quadratic objective handling is too different so I could not make it fit as nicely as I would have hoped. To avoid always calling passHessian at each update I first check against the last set of coefficients, let me know if you think there's a better way

I can't think of a better way. I think this is okay.

quantresearch1 · 2025-03-25T13:38:37Z

I need to look into the highs unit tests failure, they look genuine

quantresearch1 · 2025-03-27T16:51:14Z

I need to look into the highs unit tests failure, they look genuine

I have had a look at the remaining failures and I think it's unrelated to my PR:

osx
INFO: The following extensions were downloaded:
[FAIL] gjh
[ OK ] mcpp
linux
Message: 'Significant delay observed waiting to join reader threads, possible output stream deadlock'
Arguments: ()
Significant delay observed waiting to join reader threads, possible output stream deadlock
------- generated xml file: /home/runner/work/pyomo/pyomo/TEST-pyomo.xml -------
=========================== short test summary info ============================
FAILED pyomo/contrib/solver/tests/solvers/test_solvers.py::TestSolvers::test_domain_0_gurobi - RuntimeError: TeeStream: deadlock observed joining reader threads

jsiirola · 2025-03-27T17:08:13Z

I need to look into the highs unit tests failure, they look genuine

I have had a look at the remaining failures and I think it's unrelated to my PR:

I believe that those two test failures will be resolved by #3537.

quantresearch1 · 2025-03-28T15:17:30Z

@jsiirola Thanks, I have synced with main and that has helped with a few failures. I am just left with this one failure for python 3.13 on linux. Not clear to me if this branch is the culprit but will try to install linux over the weekend and debug. Let me know if you think it's unrelated to the branch
self = <pyomo.common.tee.capture_output object at 0x7f900cbbd6d0>, et = None
ev = None, tb = None

def __exit__(self, et, ev, tb):
    # Check that we were nested correctly
    FAIL = []
    if self.tee.STDOUT is not sys.stdout:
        FAIL.append('Captured output does not match sys.stdout.')
    if self.tee.STDERR is not sys.stderr:
        FAIL.append('Captured output does not match sys.stderr.')
    # Exit all context managers.  This includes
    #  - Restore any file descriptors we commandeered
    #  - Close / join the TeeStream
    #  - Close any opened files
    FAIL.extend(self._exit_context_stack(et, ev, tb))
    sys.stdout, sys.stderr = self.old
    self.old = None
    self.tee = None
    self.output_stream = None
    if FAIL:

      raise RuntimeError("\n".join(FAIL))

E RuntimeError: TeeStream: deadlock observed joining reader threads

pyomo/common/tee.py:380: RuntimeError
----------------------------- Captured stderr call -----------------------------
Significant delay observed waiting to join reader threads, possible output stream deadlock
TeeStream: deadlock observed joining reader threads
------- generated xml file: /home/runner/work/pyomo/pyomo/TEST-pyomo.xml -------
=========================== short test summary info ============================
FAILED pyomo/contrib/solver/tests/solvers/test_solvers.py::TestSolvers::test_equality_4_highs - RuntimeError: TeeStream: deadlock observed joining reader threads

pyomo/contrib/solver/solvers/highs.py

pyomo/contrib/solver/tests/solvers/test_highs.py

michaelbynum

Overall, this looks great. I have one major concern and a couple of minor concerns.

michaelbynum · 2025-06-24T20:08:51Z

pyomo/contrib/solver/solvers/highs.py

+
+        for ndx, coef in enumerate(self.quadratic_coefs):
+            current_val = value(coef.expr)
+            if current_val != self.last_quadratic_coef_values[ndx]:


If the set of quadratic variables changes (e.g., the objective changes from x**2 + y**2 to x**2 + z**2) will ndx be in self.last_quadratic_coef_values?

Oh, this should be fine because _set_objective should have been called when the objective changed. Can you just make sure there is a test that validates this? There might already be one. I don't remember.

I have changed that logic. The test that deletes x1 should cover this now.

michaelbynum · 2025-06-24T20:13:54Z

pyomo/contrib/solver/solvers/highs.py

+        )
+
+        if status != highspy.HighsStatus.kOk:
+            logger.warning(


I think this should be an exception. Thoughts?

Thinking a bit more about it, pretty much every single function in highspy returns a HighsStatus (https://github.com/ERGO-Code/HiGHS/blob/364c83a51e44ba6c27def9c8fc1a49b1daf5ad5c/highs/Highs.h#L962),
However, if I look at the current usage in this file, we don't capture the status when calling changeRowBounds, changeCoeff, changeColCost, etc... Just for the sake of consistency, there's an argument to remove this check

Good point. I guess I don't feel strongly one way or the other.

michaelbynum · 2025-06-24T20:14:36Z

pyomo/contrib/solver/solvers/highs.py

+        self.col_idx = col_idx
+
+
+class _MutableObjective:


I can't think of a better way. I think this is okay.

michaelbynum · 2025-06-24T20:21:03Z

pyomo/contrib/solver/solvers/highs.py

+
+                    mutable_quadratic_coefficients.append(
+                        _MutableQuadraticCoefficient(
+                            expr=coef, row_idx=v1_ndx, col_idx=v2_ndx


I don't think you should store the variable indices here. If a variable gets removed from the model, these indices will become incorrect. Note how the _MutableObjectiveCoefficient stores the variable id and the _pyomo_var_to_solver_var_map. That way, it can get the correct index at the time update is called.

I realize this is very convoluted, but I'm not sure how to avoid it.

It would be great to add a test for this scenario.

Ah yes you're right, let me add one.

I have changed the code such that the dictionary index is the id of variable 1 and 2, I think that should address your concern. These indices only get turned to row/col indices when we build the hessian. I am not sold on the Coefficient class having the map itself, it's definitely a model property in my mind.

pyomo/contrib/solver/tests/solvers/test_highs.py

quantresearch1 · 2025-06-24T21:50:08Z

pyomo/contrib/solver/tests/solvers/test_solvers.py

+        del m.x1
+        results = opt.solve(m)
+        self.assertAlmostEqual(m.x2.value, 1, places=4)
+        self.assertAlmostEqual(results.incumbent_objective, 1, 4)


@michaelbynum following your suggestion, I have added a test that deletes one of the variables. GurobiPersistent is failing this test as it finds the incumbent objective to be 2. I have not really played around with deleting variables before, is it the same as fixing that variable to 0 ? Because if I remove x1 from 2* x1^2 + x2^2 -x1*x2 I would expect the objective to be 1 since we have x2 >= 1 as a constraint. Not sure what I am missing

Oops. I think I essentially answered this in a new thread. Sorry.

pyomo/contrib/appsi/solvers/tests/test_highs_persistent.py

michaelbynum · 2025-07-29T01:25:53Z

pyomo/contrib/solver/tests/solvers/test_solvers.py

+        del m.x1
+        results = opt.solve(m)
+        self.assertAlmostEqual(m.x2.value, 1, places=4)
+        self.assertAlmostEqual(results.incumbent_objective, 2, 4)


This does not do what you think. Deleting a variable from a model does is not like removing a column from an LP. The convention in Pyomo is that the model is defined by the set of active components (normally constraints and objectives, but also things like disjunctions). If the objective still uses x1, then deleting x1 from the model actually has no effect (except that you cannot easily access x1 anymore). I'll make a minor modification to this test real quick, and then we can get it merged.

Thanks for explaining how deleting works and updating the tests for me !

michaelbynum

Assuming tests pass, I think this is ready.

michaelbynum · 2025-07-29T12:19:32Z

@emma58, your review is blocking, but I think it is outdated.

Unblocking merge

add qp support for highs

39f4392

emma58 previously requested changes Mar 24, 2025

View reviewed changes

quantresearch1 closed this Mar 24, 2025

quantresearch1 added 6 commits March 24, 2025 20:59

wrong result in test qp

823d882

pass qp tests

2621b3e

rename to quadratic_coefs for consistency with gurobi

0815a03

rename to quadratic_coefs for consistency with gurobi 2

9bba7d0

introduce _MutableObjective

1b42058

fix persistency delta bug

c659ccc

quantresearch1 reopened this Mar 25, 2025

quantresearch1 and others added 3 commits March 25, 2025 00:00

or polynomial degree > 2

38a0a85

Merge branch 'main' into feature/3381_quadratic_obj_highs

b289dac

remove changes to legacy appsi_highs

6c4b7b1

quantresearch1 commented Mar 25, 2025

View reviewed changes

blnicho requested a review from michaelbynum March 25, 2025 19:13

quantresearch1 added 2 commits March 25, 2025 21:10

add back mutable vs non mutable handling

891483e

fix more unit tests

6d6c2d2

quantresearch1 requested a review from emma58 March 28, 2025 00:04

Merge branch 'main' into feature/3381_quadratic_obj_highs

4a92411

Merge branch 'main' into feature/3381_quadratic_obj_highs

8902562

mrmundt self-requested a review April 8, 2025 18:39

Merge branch 'main' into feature/3381_quadratic_obj_highs

720a5ed

mrmundt requested changes Apr 10, 2025

View reviewed changes

pyomo/contrib/solver/solvers/highs.py Outdated Show resolved Hide resolved

pyomo/contrib/solver/solvers/highs.py Outdated Show resolved Hide resolved

pyomo/contrib/solver/tests/solvers/test_highs.py Outdated Show resolved Hide resolved

quantresearch1 added 2 commits April 10, 2025 20:00

standardize pyomo.environ

3d71b39

rename q_value to hessian_value

04eb847

Merge branch 'main' into feature/3381_quadratic_obj_highs

203c6a9

blnicho added this to August 2025 Release Jun 23, 2025

github-project-automation bot moved this to Todo in August 2025 Release Jun 23, 2025

michaelbynum requested changes Jun 24, 2025

View reviewed changes

quantresearch1 and others added 3 commits June 24, 2025 16:48

Merge branch 'main' into feature/3381_quadratic_obj_highs

3f7c74f

move qp tests to test_solvers

c1422b7

delete variable x in test

be02bba

quantresearch1 commented Jun 24, 2025

View reviewed changes

quantresearch1 added 2 commits June 24, 2025 18:45

addressing michael's comments MutableQuadraticCoefficient

5823583

remove commented bit in test

3fb1476

quantresearch1 requested a review from michaelbynum June 26, 2025 01:13

blnicho moved this from Todo to Review In Progress in August 2025 Release Jun 27, 2025

mrmundt added 2 commits June 30, 2025 07:56

Merge branch 'main' into feature/3381_quadratic_obj_highs

4489372

Merge branch 'main' into feature/3381_quadratic_obj_highs

83784fc

mrmundt reviewed Jul 2, 2025

View reviewed changes

pyomo/contrib/appsi/solvers/tests/test_highs_persistent.py Outdated Show resolved Hide resolved

quantresearch1 and others added 2 commits July 3, 2025 19:24

add back TerminationCondition

a5317fa

Merge branch 'main' into feature/3381_quadratic_obj_highs

a1410b8

michaelbynum requested changes Jul 29, 2025

View reviewed changes

michaelbynum added 2 commits July 29, 2025 05:54

update mutable qp test

c4b08aa

run black

de53637

michaelbynum added the AT: PRE-TEST INSPECTED label Jul 29, 2025

typo

0d45025

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Jul 29, 2025

michaelbynum approved these changes Jul 29, 2025

View reviewed changes

michaelbynum added the AT: PRE-TEST INSPECTED label Jul 29, 2025

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Jul 29, 2025

michaelbynum merged commit 8f07c5a into Pyomo:main Jul 29, 2025
35 checks passed

github-project-automation bot moved this from Review In Progress to Done in August 2025 Release Jul 29, 2025

add qp support for highs #3531

add qp support for highs #3531

Uh oh!

Conversation

quantresearch1 commented Mar 21, 2025

Fixes #3381 .

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Uh oh!

emma58 left a comment

Choose a reason for hiding this comment

Uh oh!

quantresearch1 commented Mar 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quantresearch1 commented Mar 25, 2025

Uh oh!

quantresearch1 commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsiirola commented Mar 27, 2025

Uh oh!

quantresearch1 commented Mar 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelbynum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quantresearch1 Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

quantresearch1 Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelbynum Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quantresearch1 Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

quantresearch1 commented Mar 27, 2025 •

edited

Loading

quantresearch1 Jun 24, 2025 •

edited

Loading

quantresearch1 Jun 24, 2025 •

edited

Loading

michaelbynum Jul 29, 2025 •

edited

Loading

quantresearch1 Jul 29, 2025 •

edited

Loading