Debugging#

LEAPP provides multiple lines of defense against an incorrectly traced graph. The goal is confidence that the traced graph is numerically correct.

When	Use
You want numeric confidence in an export	`compile_graph(validate=True)`
Validation failed and you need the recorded ops	`log.txt` and the FX dump
You need to tell tracing from export from patching	`dry_run`, `non_traced`, `global_patching=False`

Confirm the export#

compile_graph(validate=True) checks each exported node against all its cached Python input/output samples, including exported state inputs and updates. Repeated execution can capture consecutive stateful transitions; max_cached_io controls how many samples are retained per node. LEAPP runs the exported model on each sample, compares outputs with torch.allclose(..., rtol=..., atol=...), logs any deviation, and returns a dict mapping node names to results.

leapp.compile_graph(
    validate=True,
    rtol=1e-3,
    atol=1e-5,
    strict=True,
)

strict=True raises if any node fails. strict=False still runs validation, but leaves the result dict available. Nodes with no compiled model, such as non_traced or dry-run cases, are skipped and treated as successful.

A mismatch log includes the node and output names, the sample index, tolerances, shapes and dtypes, value ranges, and absolute-difference percentiles. Several samples are what make those numbers meaningful: one pair of tensors cannot characterize error, and it cannot show whether a value was captured because it is computed or because it happened to be constant on the first step.

Run more than one step#

Warning

Loop the annotated policy. A graph that matches the first observation can still have inlined a live input as a constant, and a single sample says nothing about error on any other input.

leapp.start(name="policy_graph", max_cached_io=5)
for _ in range(5):
    obs = annotate.input_tensors("policy", {"obs": next_obs()})
    action = policy(obs)
    annotate.output_tensors(
        "policy", {"action": action}, export_with="jit",
    )
leapp.stop()
leapp.compile_graph(validate=True, strict=True)

Each iteration is stored as a sample, up to max_cached_io. compile_graph(validate=True) replays the export against every one. Use inputs that look like deployment: different state, commands, and timing, not copies of the first frame. If a later step fails while the first passes, the log names that sample N.

Re-entry also checks names, shapes, dtypes, and connectivity against the first step. Those failures show up during tracing, not at compile_graph().

Stateful transitions and sequential replay#

When consecutive calls carry state forward, cached samples preserve the original trajectory, including each state’s next value:

(observation_0, state_0) -> (action_0, state_1)
(observation_1, state_1) -> (action_1, state_2)
(observation_2, state_2) -> (action_2, state_3)

LEAPP supplies each recorded observation and state to the exported node and compares both its action and its next state with the recorded outputs. This validates the state-transition computation across the captured trajectory. State must be included in the exported interface as described in State capture and feedback; unannotated Python state is outside this check.

For deterministic computation with all relevant state represented, exact agreement on every transition also implies agreement when those transitions are chained with the same observations, initial state, and reset boundaries: each exported next state equals the next recorded input state.

Validation accepts numerical differences within rtol and atol. Cached validation supplies the recorded reference state again at each step, so small accepted errors do not accumulate between samples. Sequential runtime replay feeds the exported state outputs into subsequent calls and can reveal whether those errors grow, as well as problems in feedback wiring, initialization, or reset handling. Increasing the cache size expands recorded transition coverage; it does not change how validation supplies state.

Use both checks when assessing stateful deployment: capture representative consecutive transitions, then compare a sequence through LEAPP Runtime against the original pipeline from matching initial state. Include episode resets and fresh observations beyond the capture set.

Inspect what was captured#

Every leapp.start() call writes a LEAPP log file in the graph output directory:

leapp.start("debug_policy", save_path="exports")

creates exports/debug_policy/log.txt. The file includes DEBUG output. The console is quieter and shows warnings and errors. Pass verbose=True to stream the same detail to the console:

leapp.start("debug_policy", save_path="exports", verbose=True)
# ... run your traced code ...
leapp.stop()
leapp.compile_graph(validate=True, verbose=True)

leapp.start(..., verbose=True) covers trace-time diagnostics. leapp.compile_graph(..., verbose=True) covers compilation, export, and validation.

FX graph#

For each traced node, LEAPP writes the full torch.fx.Graph to the log after building the node’s fx.GraphModule. Search for Compiled graph module for <node name> in log.txt:

[DEBUG]: Compiled graph module for policy:
graph():
    %obs : [num_users=1] = placeholder[target=obs]
    %linear : [num_users=1] = call_module[target=linear](args = (%obs,), kwargs = {})
    %tanh : [num_users=1] = call_function[target=torch.tanh](args = (%linear,), kwargs = {})
    return tanh
[DEBUG]: Graph module inputs: [...]
[DEBUG]: Graph module outputs: [...]

This is the graph LEAPP hands to the export backend. Use it to check that annotated inputs became placeholders, that operations recorded as the operators you expected, that unused inputs were trimmed, and that the graph outputs match what you passed to annotate.output_tensors().

Graph image#

On Python 3.11+, leapp.compile_graph(visualize=True) writes a PNG of the node graph, which is the fastest check that nodes connected the way you intended and the dtypes are as you expect.

Isolate the failure#

These flags turn pieces of a LEAPP session off so you can tell whether a bad result comes from tracing, from export, or from LEAPP’s own patches.

Skip export#

Tracing is what produces the values that carry graph connectivity, so it cannot be skipped while still producing a graph. Export can.

leapp.start(..., dry_run=True) traces every node and writes YAML, but exports no models. Use it to inspect boundaries, I/O, and connectivity without paying export cost.
leapp.start(..., non_traced=["some_node"]) does the same for listed nodes only: they still appear in the graph and still connect to neighbors, but they produce no model artifact.
export_with=None on a single output_tensors() call skips export for that node only.

Warp graphs still run the annotated path a second time and still report a segment that diverges between the two runs. Only APIC capture is skipped, because nothing consumes the captured bundle when the node is not exported.

leapp.start("debug_graph", dry_run=True)
# ... run your traced code ...
leapp.stop()
leapp.compile_graph()

Disable session patches#

leapp.start(..., global_patching=False) turns off the conversions LEAPP patches for the tracing session (for example NumPy and torch interop). Use this only to test whether those patches are involved in a failure.