FFFF feat: override list, open options and prepared limit setters (minigraf#872) by adityamukho · Pull Request #18 · project-minigraf/minigraf-python · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
8000
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm, macos-14, windows-latest]
os: [ubuntu-latest, ubuntu-24.04-arm, macos-15, windows-latest]

steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -75,7 +75,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm, macos-14, windows-latest]
os: [ubuntu-latest, ubuntu-24.04-arm, macos-15, windows-latest]
steps:
- uses: actions/checkout@v4
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
- os: ubuntu-24.04-arm
build_args: --release --manylinux manylinux_2_34 --out dist
rust_targets: ''
- os: macos-14
- os: macos-15
build_args: --release --target universal2-apple-darwin --out dist
rust_targets: x86_64-apple-darwin
- os: windows-latest
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/verify-published.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm, macos-14, windows-latest]
os: [ubuntu-latest, ubuntu-24.04-arm, macos-15, windows-latest]
steps:
- uses: actions/checkout@v4
with:
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ path = "src/uniffi_bindgen.rs"

[dependencies]
# Unreleased v3 core (format v8). Switches to minigraf = "=3.0.0" at the v3.0.0 release.
minigraf = { git = "https://github.com/project-minigraf/minigraf", rev = "02243f1656eedd987611a9babdfee4a8c6e22b6b" }
minigraf = { git = "https://github.com/project-minigraf/minigraf", rev = "d9755c02d19a1538ca03fd0b80c56a752bbc6504" }
uniffi = { version = "0.31.1", features = ["cli"] }
thiserror = "2.0.18"
serde_json = "1.0.149"
Expand Down
44 changes: 43 additions & 1 deletion 10BC8 README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ src = minigraf.open("app.graph", read_only=True, page_cache_size=4096)

# Every option: read_only, page_cache_size, allow_unlocked, wal_checkpoint_threshold,
# max_derived_facts, max_results, synchronous (minigraf.SyncMode.FULL / NORMAL),
# query_memory_limit.
# query_memory_limit, allow_overrides, transaction_chunk_facts, max_iterations.
db = minigraf.MiniGrafDb.open_with_options("app.graph", minigraf.OpenOptions(max_derived_facts=10_000))
```

Expand All @@ -58,6 +58,33 @@ limit. Rows a cursor returns are not counted, so read a large answer with `db.qu
and the checkpoint when the handle closes, for applications that call `checkpoint()` on
their own schedule.

## Override list and limits

Query text may override some settings with keywords such as `:max-derived-facts 1000`.
`allow_overrides` lists the `minigraf.Overridable` keywords it may use
(`MAX_DERIVED_FACTS`, `MAX_RESULTS`, `QUERY_MEMORY_LIMIT`, `MAX_ITERATIONS`). Left out, the
defaults apply: `minigraf.OVERRIDABLE_DEFAULTS`, read from the core library (today
`MAX_DERIVED_FACTS`, `MAX_RESULTS` and `MAX_ITERATIONS`). A list replaces the defaults, it
does not add to them, and `[]` refuses every keyword: a query that uses one raises
`MiniGrafError.Api` with `[API-025]` from `execute()`, `query()` and `prepare()`.

```python
from minigraf import Overridable

# Query text may set only :query-memory-limit.
db = minigraf.open("app.graph", allow_overrides=[Overridable.QUERY_MEMORY_LIMIT])
# The defaults plus :query-memory-limit.
db = minigraf.open(
"app.graph",
allow_overrides=[*minigraf.OVERRIDABLE_DEFAULTS, Overridable.QUERY_MEMORY_LIMIT],
)
```

`max_iterations` is the most rounds each stratum of a rule query may run (default
1,000,000; more raise `MiniGrafError` with `[INT-020]`). `transaction_chunk_facts` is how
many facts one transaction on a file database stages before it is written in chunks
(default 65,536, clamped to 1..=1,000,000). A negative count raises `[API-017]`.

## Query cursors

`db.query()` returns a cursor whose answer is fixed when it opens. Iterating gives
Expand Down Expand Up @@ -101,6 +128,21 @@ elements is written `minigraf.Value.SET([minigraf.Value.KEYWORD(":role/admin")])
inside a set raises `MiniGrafError.Api` with `[API-021]`; a `list` is not a set. Set values
come back as JSON arrays from `execute()` and as `Value.SET([...])` in fact records.

### Limits of one prepared query

`with_max_derived_facts(n)`, `with_query_memory_limit(bytes)` and `with_max_iterations(n)`
each return a new prepared query with that limit; the one they are called on is
unchanged. They are application code, so `allow_overrides` does not apply to them, and an
allowed keyword in the query text still overrides them. `2**63 - 1` means no limit and a
negative value raises `[API-017]`. A derived query keeps its database open, as any
prepared query does.

```python
reach = db.prepare("(query [:find ?x ?y :where (reach ?x ?y)])")
small = reach.with_max_derived_facts(10_000).with_query_memory_limit(16 << 20)
small.execute() # [INT-020] or [QRY-012] past a limit; reach.execute() is unchanged
```

## User functions

A predicate subclasses `minigraf.Predicate` and is called in Datalog as `[(name ?v)]`.
Expand Down
35 changes: 35 additions & 0 deletions consumer-test/contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import os
import sys

import minigraf
from minigraf import Accumulator, Aggregate, MiniGrafDb, MiniGrafError, Predicate, UdfError, Value

QUERY = "(query [:find ?n :where [?e :name ?n]])"
Expand Down Expand Up @@ -95,3 +96,37 @@ def finish(self, count):
again = MiniGrafDb.open(path)
assert names(again) == ["Bob"], "query after reopen returns Bob"
print("ok: file-backed survives reopen")

# minigraf#872: the override list, the new open options and a prepared setter.
assert minigraf.OVERRIDABLE_DEFAULTS == (
minigraf.Overridable.MAX_DERIVED_FACTS,
minigraf.Overridable.MAX_RESULTS,
minigraf.Overridable.MAX_ITERATIONS,
), "the default override list comes from core"
limited = minigraf.open(
os.path.join(sys.argv[1], "overrides.graph"),
allow_overrides=[],
transaction_chunk_facts=2,
max_iterations=1000,
)
limited.execute("(transact [[:a :edge :b] [:b :edge :c] [:c :edge :d]])")
limited.execute("(rule [(reach ?x ?y) [?x :edge ?y]])")
limited.execute("(rule [(reach ?x ?y) [?x :edge ?z] (reach ?z ?y)])")
reach = "(query [:find ?x ?y :where (reach ?x ?y)])"
try:
limited.execute("(query [:find ?x ?y :where (reach ?x ?y) :max-derived-facts 10])")
except MiniGrafError.Api as e:
assert e.msg.startswith("[API-025]"), "an empty list refuses the keyword"
else:
sys.exit("an empty override list did not refuse :max-derived-facts")
pq = limited.prepare(reach)
derived = pq.with_max_iterations(1)
try:
derived.execute()
except MiniGrafError as e:
assert e.msg.startswith("[INT-020]"), "the derived query has its own limit"
else:
sys.exit("with_max_iterations(1) did not limit the derived query")
assert len(json.loads(pq.execute())["results"]) == 6, "the receiver is unchanged"
print("ok: override list, open options and a prepared setter")

21 changes: 21 additions & 0 deletions minigraf/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@
MiniGrafUdfError,
MiniGrafValue,
OpenOptions,
Overridable,
SyncMode,
overridable_defaults,
)

# Shorter names for the objects.
Expand All @@ -40,6 +42,10 @@
#: and none when the handle closes. Call ``checkpoint()`` yourself.
WAL_CHECKPOINT_NEVER = 2**63 - 1

#: The override list that applies when ``allow_overrides`` is not given:
#: the Rust ``Overridable::DEFAULTS``, read from the core library.
OVERRIDABLE_DEFAULTS = tuple(overridable_defaults())


def open(
path,
Expand All @@ -52,13 +58,23 @@ def open(
max_results=None,
synchronous=None,
query_memory_limit=None,
allow_overrides=None,
transaction_chunk_facts=None,
max_iterations=None,
):
"""Open a file-backed database. An option left as ``None`` keeps its default.

``minigraf.open("app.graph", read_only=True)`` opens the file read-only:
any number of read-only handles can share it, and writes raise
``MiniGrafError`` with code API-014. ``query_memory_limit`` caps the bytes
one query may hold; a query that would hold more raises QRY-012.

``allow_overrides`` lists the ``Overridable`` keywords query text may use
to override these settings; it replaces ``OVERRIDABLE_DEFAULTS`` (``None``)
and ``[]`` refuses them all (API-025). ``max_iterations`` limits the
rounds of each stratum of a rule query (INT-020 beyond), and
``transaction_chunk_facts`` the facts a transaction stages before it is
written in chunks.
"""
return MiniGrafDb.open_with_options(
path,
Expand All @@ -71,6 +87,9 @@ def open(
max_results=max_results,
synchronous=synchronous,
query_memory_limit=query_memory_limit,
allow_overrides=None if allow_overrides is None else list(allow_overrides),
transaction_chunk_facts=transaction_chunk_facts,
max_iterations=max_iterations,
),
)

Expand Down Expand Up @@ -253,7 +272,9 @@ def _writer_exit(self, exc_type, exc, tb):
"MiniGrafPreparedQuery",
"MiniGrafUdfError",
"MiniGrafValue",
"OVERRIDABLE_DEFAULTS",
"OpenOptions",
"Overridable",
"Predicate",
"PreparedQuery",
"SyncMode",
Expand Down
Loading
Loading
0