Python API¶

This page describes what each verb takes, returns, guarantees and refuses, for anyone who runs a spec from Python. A spec is the YAML file; what it may contain is the language.

import specsolve as sps

sps.check('spec.yaml')  # compiles? no data needed

result = sps.solve('spec.yaml', sources)
result.objective
result.primal('p')  # a polars.DataFrame
result.dual('power_balance')

The verbs¶

Every verb takes the spec first and, except check, the sources second: the tables that carry its numbers. The glossary defines model, result, sink and the other house terms this page uses.

Errors and warnings¶

Every error is one tree, rooted at SpecsolveError. LanguageError (with SchemaError, DimensionError) is a fault in the spec. DataError is a fault in the data attached to it. LayoutError is a directory or an archive that is not a layout this package reads. NoSolutionError is a solve that left nothing to read. A spec the language accepts and specsolve cannot build raises SpecsolveError itself, and its message names the rewrite that builds the same model. Which one you get: errors.

SpecsolveWarning is the one warning category, and carries check's advice. warnings.simplefilter('error', sps.SpecsolveWarning) makes a spec repository fail CI on it.

The spec argument¶

Every verb takes the spec as a path, a str, a dict or a Spec: what mathspec.to_spec takes, which is never a lowered Program. So a framework that emits declarations never writes a temporary file to run them:

spec = {'dimensions': ..., 'variables': ..., 'constraints': ..., 'objective': ...}

sps.solve(spec, sources)  # a dict runs like a file
kept = to_spec(spec)  # ...or read once and keep the document
sps.solve(kept, sources)  # a Spec is not read again

to_spec(spec).to_yaml()  # the review copy — a dict-built spec still gets a file

Keep the Spec, not the Program. sps.check hands back a lowered Program for reading the plan, and no verb takes one. A Spec handed back to a verb is not read again.

A formulation is written out before a verb reads it. A piecewise: block states rows nothing lowers, so specsolve refuses a spec still carrying one, at every verb, rather than expanding it unasked, and names the way in: to_spec(spec).expand('piecewise') writes each curve out as the variables and constraints it states and keeps every sos: block, for a sink that branches on a set; to_spec(spec).expand() writes the sets out too, as binaries and linking rows, which every sink takes. Which of the two is the caller's to say, because the sinks disagree: HiGHS has no SOS concept and refuses a set, Gurobi and Xpress take one whole.

from mathspec import to_spec

sps.solve(to_spec('piecewise.yaml').expand(), sources)  # curves and sets written out — any sink
sps.solve(to_spec('sos.yaml').expand('piecewise'), sources, 'gurobi')  # the set reaches the solver as a set

A framework emits data, not YAML text, and never merges files. A generated spec must be able to show you a file. Hand-written math still starts as one.

A dict-built spec still gets a file. to_dict() and to_yaml() are the language's, and what they write is its page.

The sources argument¶

sources maps each declared name to its data, and a dimension's own key supplies its labels. What each value may be, and what attaching refuses, is the data contract; the type is specsolve.lanes.Source, which every verb annotates sources with.

result = sps.solve(
    'dispatch.yaml',
    {'load': 'load.parquet', 'cost': cost_frame, 'p_max': p_max_frame},
)

sources is the whole of the build's input: parameters and dimension indexes in one mapping. solver_options is not a build knob. It is forwarded to the solver verbatim.

Checking a spec¶

check is the CI verb. It parses, resolves and lowers the spec and attaches nothing, so a spec repository can validate every commit without the data. It returns the program: the spec lowered to the plan a build reads its rows off.

Names that differ only by case¶

Two declarations of one namespace whose names differ only by case are refused, whichever verb lowers the spec. Every declaration is written to disk as a file named after it, and a case-insensitive filesystem, which a stock macOS or Windows volume is, folds p and P into one file.

variable 'P' and variable 'p' differ only by case, and one answer on disk
cannot hold both: ... Tell them apart by a suffix rather than a capital:
'p_rated' beside 'p'.

The namespaces are the language's own: one flat namespace holding dimensions, relations, parameters, variables and named expressions, and constraints beside it. A constraint may carry a variable's name already, so a constraint P beside a variable p is accepted. The two are written under dual/ and primal/, which nothing folds together.

Checking against a sink¶

Whether a spec is sayable does not depend on the solver. Where it can land is a separate question, and sink= asks it:

sps.check('spec.yaml')  # sayable?
sps.check('spec.yaml', sink='highs')  # ...and will HiGHS take it?
sps.check('spec.yaml', sink='.lp')  # ...will the LP writer?

sink is a solver name (highs, gurobi) or an output suffix (.lp). It is optional and silent by default. With a sink named, you get back one of:

• A refusal (SpecsolveError) if the sink has no such concept, or refuses the combination. The message names the construct, the sink, and the sinks that do take it. Only Gurobi and the LP writer take a quadratic row, and HiGHS refuses a quadratic objective beside integrality while taking either alone. HiGHS has no SOS concept, so a set on it is refused and the message names Spec.expand(), which writes the set out as binaries and linking rows every sink takes.

check answers off a declared table, with no data and no installed solver. check(m, sink='gurobi') answers on a machine that has never had gurobipy.

solve and write read the same table, so a refusal comes whether or not you asked. sps.write(m, sources, 'model.mps') on a model carrying a quadratic term is refused by name rather than written with its quadratic rows missing.

What each sink takes¶

The four quadratic rows, and the two sections HiGHS writes but will not read back, are probed against the shipped solvers by tests/test_sink_capability_probes.py and tests/test_gurobi_capability_probes.py. The rest are read off the APIs.

• HiGHS excludes quadratic twice: by convexity, and by conjunction with integrality.
• The lp_file column says what can be written, not what reads back. The same HiGHS parser takes the quadratic-objective section and refuses the sos and quadratic-constraint sections.
• "No path here" describes this package, not Xpress. The Optimizer takes a Hessian; the sink in solvers/xpress.py never hands it one.

Building a model¶

sps.build returns a Model: the math with your data on it. Build once when one model feeds more than one sink, or is solved more than once:

model = sps.build('spec.yaml', sources)
model.write('model.lp')
result = model.solve()
model.diagnostics()  # what the build and its solves did that the answer does not show
model.row('balance', snapshot=17)  # what one row actually says

Questions about the model are build's, not solve's. How big the model is, what it did not build, what one row says and how its re-solves went are the Model's to answer.

Reading one row¶

row says what one constraint says at one coordinate, once the data is on it. to_latex renders the spec before any data, and result.dual('balance') gives a row's number without its terms; row is the third question, and the one a wrong model is debugged by.

print(model.row('balance', snapshot=1))
# balance[snapshot=1]: +1 p[1, wind] +50 p[1, gas] +30 p[1, coal] >= 60

The line is linopy's format, as Constraint.print() renders it, with the row's identity on the same line where linopy prints a header.

The same content is a table, for a row too wide to read and for anything that filters or joins:

row = model.row('balance', snapshot=17)
row.terms  # (variable, coordinate, coefficient), one row per term
row.sense  # '=='
row.rhs  # 80.0

A row too wide to spell out is summarised, not truncated:

print(model.row('balance', t=0))
# balance[t=0]: 301 terms — p: 300 (|coef| 0.001…0.3), slack: 1 (|coef| 1000) >= 5

The line says how much of the row each declaration contributes, and whether its coefficients span an order of magnitude. diagnostics().coefficient_range reports that spread per declaration; nothing reports it per row. display_terms sets where a line stops spelling terms out.

row reads the built row.

• A coefficient is the number the data produced, every digit of it.
• A term whose variable a where masked out is not there.
• A term whose coefficient the data made exactly zero is not there either: the build prunes it.
• A row a where removed raises, and the message names the three things that cause it.

row needs no solve.

The coordinate names every dimension of the declaration. A partial one names a set of rows rather than one. The constraint is positional, so a dimension may be called name and still be named in the coordinate. A label the dimension cannot hold (a string against an integer dimension, a stranger against a declared label set) is refused naming the dimension, not the dtypes.

There is no verb for a column. A variable's bounds are in to_yaml(); its coefficients are the transpose of row, which nothing exposes.

Reading a result¶

result.status, result.termination_condition, result.objective
result.spec_digest  # a digest of the spec this answered
result.record  # all of it as one Record: the row save writes, and one row of sweep.record
result.is_ok  # rolled-up verdict: not an error, abort or refusal
result.has_primal  # narrower: are there values to read
result.kept  # how much of the session this solve kept: 'nothing', 'solver' or 'progress'

result.primal('p')  # tidy table (dims…, value) in label order — the native shape
result.dual('power_balance')  # shadow prices, same shape, same join
result.activity('power_balance')  # each row's left-hand side at the solution
result.dual_ray('power_balance')  # why an infeasible model has no solution at all
result.evaluate('co2')  # a named expression at the solution, over its own dims
result.evaluate('sum(p * rate)')  # a quantity the file never named, same shape

result.to_pandas('p')  # the same, as a DataFrame
result.to_dataarray('p')  # the same, labelled: .sel / resample / plot
result.to_dataarray('power_balance', 'dual')  # a price, labelled — every bridge takes kind=
result.to_dataset()  # every variable by default; names for a subset
result.to_dataset(kind='dual')  # every dual; one kind per dataset
result.save(directory)  # the whole answer to disk: record.parquet, primal/ dual/ activity/ expression/, reasons.parquet
sps.load_result(directory)  # and back whole, every reader answering what it answered
sps.scan_result(directory)  # the same, read off the directory as you ask for it

primal returns a polars.DataFrame, one row per coordinate: a frame. It is Arrow-backed, so it exports the protocol the loader recognises. to_pandas and to_dataarray are the bridges out; they need pandas and xarray, which specsolve does not install.

Nothing has to be released. primal and the to_* readers stay valid for as long as the Result does. close() and the context-manager protocol hand a large model back early.

Writing a file instead of solving¶

sps.write('spec.yaml', sources, 'model.lp')

The suffix picks the writer: .lp or .mps. Anything else is a ValueError listing what can be written, raised before the build.

The two formats describe one model, and name their columns and rows the same way. LP is the one a person diffs; MPS is the one a decade-old toolchain accepts.

Re-solving with new numbers¶

update puts new data on a model that is already built, so a loop over the same math pays for the YAML, the plan and the build once:

model = sps.build('sub.yaml', sources)
for capacity in search:
    result = model.update({'cap_hat': capacity}).solve()
    price = result.dual('capacity')

For a sweep, a rolling horizon or a myopic pathway, solve_over is this loop written for you. update is the primitive underneath, for when the next set of numbers depends on the last answer. Where it depends on you, Change a model is the loop written out.

How much of the session a solve keeps¶

A session holds two things: the solver with the model on it, and the work that solver did. An update keeps the first. keep= says whether it keeps the second. The two can only be dropped in that order.

result = model.update({'load': load}).solve()
result.kept  # 'solver' — reused, and the work it did discarded

again = model.update({'load': more}).solve(keep='progress')
again.kept  # 'progress' — it carried on from where the last solve got to

baseline = model.solve(keep='nothing')  # whatever the session held, gone
baseline.kept  # 'nothing'

keep='progress' can lose by an order of magnitude and win by a factor of two. Over six updates on HiGHS (#815), carrying the solver's work cost 76.6 s against 4.3 s on a dispatch model whose presolve cracks the problem outright, an 18× loss, and 111.2 s against 213.9 s on a storage model whose cyclic recurrence presolve cannot crack, a 1.9× win.

Which one a model wants is measured (timing a loop). The answer does not change either way: across both models above the objectives agreed to 2e-15 relative.

result.kept reports what happened, not what was asked. An update that had to rebuild reports 'nothing', whatever it asked for, and loads ticks on exactly those solves. 'nothing' on every iteration means the session is being rebuilt away.

What progress is made of stays the solver's business. kept says how much was kept, not what it was. No solver option reaches the same thing; on both solvers that ship, an option asking for it did not produce it (#815).

A rebuild carries no progress. A cutting-plane master re-solved after gaining a cut has gained a row, and a basis spans the model it was read from. #382 tracks that case.

Archiving a model¶

sps.solve('spec.yaml', sources, archive='case.zip')

case = sps.load_archive('case.zip', 'case/')
case.answer.primal('p')  # what came back
sps.solve(case.spec, case.sources)  # the same question, asked again

An archive is the spec, its data and its answer: spec.yaml, sources/<key>.parquet for every key the file declares, sources.parquet digesting those members, answer/ holding what result.save or sweep.save writes plus answer/metrics.parquet, and axis.json for a sweep.

The suffix decides the container, as sps.write's does. .zip packs the members into one file; anything else lays them out in a directory, which is read where it lies:

sps.solve('spec.yaml', sources, archive='case/')  # a directory
sps.load_archive('case/')  # read where it lies — no into=

sps.solve, model.solve and sps.solve_over take archive=, and nothing else writes one. Each writes the spec, the data and the answer it holds at that moment, so the three cannot be paired up wrongly.

The sources go in through the door that reads them, so what build refuses is refused here and nothing is written. A parquet path is copied as its own bytes; a table, a bare label range, a {label: value} map or a single number is written as the tidy parquet table it stands for. Members are stored uncompressed.

The recipes are archiving a solve and reading a directory of runs.

Loading or scanning¶

Three saved things read back, and each reads two ways. load_ reads it whole: the frames are in memory when the call returns, so what comes back owes the directory nothing. scan_ leaves them where they lie and reads each at the call that asks for it, so the files have to outlive the value. A load reads every name; a scan reads only the ones asked for.

case = sps.load_archive('case.zip')  # whole, and nowhere to unpack
case = sps.scan_archive('case.zip', 'case/')  # read as asked for, off 'case/'

The pairs are load_archive / scan_archive, load_result / scan_result and load_sweep / scan_sweep. Each pair takes the same arguments, hands back the same type, and refuses the same things: a directory holding no answer, and an archive whose answer names another spec. The one difference is the into= a zip needs, which the table above gives.

A loaded value is fixed and a scanned one is not. A load leaves nothing to be read later. A scan re-reads the file at every collect, so a frame rewritten underneath it comes back changed.

Diagnostics¶

model.diagnostics() reports what a build and its solves did that the answer does not show. Every field is advisory. Nothing about an answer depends on any of them.

diagnostics() answers after close() too. A sweep's diagnostics are sweep.metrics, one row per slice (sweeps).

metrics() is the scalars as one row, a Metrics. The frames are not in it — a range is a table per declaration, which does not fold into a row beside a count. This is what archive= records and what archive.metrics hands back, and what a caller feeding its own store reads off a model it solved. It is eleven attributes and they are every column of answer/metrics.parquet:

Every clock names its unit, and every one is cumulative over the solves the row covers. A phase that never ran writes zero rather than no column, so rows written by runs that never met concatenate into one table. What writing the archive cost is in no column: time the call.

Choosing a solver¶

The caller chooses the solver, not the file. solver_name is highs (ships with the package), gurobi (the [gurobi] extra) or xpress (the [xpress] extra). Nothing in the YAML names one. A name outside the three is an error listing them, never a quiet fallback.

Options travel in the chosen solver's own vocabulary, forwarded verbatim. A time limit is three different words:

sps.solve('spec.yaml', sources, solver_options={'time_limit': 60})
sps.solve('spec.yaml', sources, solver_name='gurobi', solver_options={'TimeLimit': 60})
sps.solve('spec.yaml', sources, solver_name='xpress', solver_options={'timelimit': 60})

Gurobi's remote and licensing options travel the same way, so Compute Server, Instant Cloud and WLS need nothing from this package:

options = {'ComputeServer': 'srv:61000', 'ServerPassword': '…'}
sps.solve('spec.yaml', sources, solver_name='gurobi', solver_options=options)

The options are applied when Gurobi's environment is created, which ComputeServer, TokenServer and WLSAccessID require.