Provenance and reuse

Provenance and reuse#

This is the focused guide to artifact mechanics. It shows how to reopen one exact result, branch a parameter, identify reused work, force recomputation, and compare lineage. Read Why Scarf records provenance first for the short mental model.

Prerequisites#

  • Scarf installed with the extra optional dependencies

What you will learn#

  • Reuse normalization, PCA, and ANN when only neighbor k changes

  • Rebuild reduction and everything downstream when dims changes

  • Force a new artifact with invalidate_cache=True

  • Compare upstream lineage for neighbour-count and dimensionality forks

Dataset#

The rebuilt PBMC store carries a completed pipeline run labelled docs_default. Its exact selection, normalization, PCA, neighbour, and graph refs provide the baseline. This page creates only the parameter forks needed to demonstrate reuse.

import scarf
scarf.configure_output(level="WARNING", progress=False)

dataset = scarf.cytebase.connect("scarf_docs").download_dataset(
    "tenx_5K_pbmc_rnaseq",
    destination="scarf_datasets",
    zarr=True,
)
ds = scarf.DataStore(f"{dataset}/data.zarr", nthreads=4)
baseline_run = ds.pipeline.open(label="docs_default")
cell_selection = baseline_run["analysis_cell_selection"]
hvg_ref = baseline_run["highly_variable_features"]

1. Open and inspect the baseline chain#

The completed run retains every immutable reference needed to keep side comparisons separate.

normalized = baseline_run["normalized"]
pca = baseline_run["pca"]
ann = baseline_run["ann_index"]
neighbors_k11 = baseline_run["neighbors"]
graph_k11 = baseline_run["connectivity_map"]

The catalog can also find results by exact provenance predicates. It returns every match and never chooses a latest result, so one-item destructuring is an explicit cardinality check:

[reopened_graph] = ds.list_artifacts(
    from_assay="RNA",
    kind="connectivity_map",
    operation="build_connectivity_map",
    inputs={"neighbors": neighbors_k11},
    complete_only=True,
)
assert reopened_graph == graph_k11

status = ds.inspect_artifact(reopened_graph)
{
    "operation": status.operation,
    "parameters": status.parameters,
    "inputs": status.inputs,
    "complete": status.complete,
}

{'operation': 'build_connectivity_map',
 'parameters': {'local_connectivity': 1.0, 'bandwidth': 1.5},
 'inputs': {'neighbors': {'type': 'artifact',
   'scope': 'assay',
   'kind': 'neighbors',
   'artifact_id': '4b3a82191fcc29dd22f1fa6720ab6dae3c775976b13ae1c9d7349de5496d02f2',
   'assay': 'RNA'}},
 'complete': True}

2. Vary k: reuse upstream#

A new neighbor count changes only the neighbors and connectivity provenance. The normalization, PCA, and ANN references are unchanged.

neighbors_k15 = ds.query_neighbors(ann, k=15)
graph_k15 = ds.build_connectivity_map(neighbors_k15)

{
    "normalization reused": ds.run_normalization(cell_selection, hvg_ref) == normalized,
    "PCA reused": ds.run_pca(normalized, dims=15) == pca,
    "ANN index reused": ds.build_ann_index(pca) == ann,
    "neighbors recomputed": neighbors_k15 != neighbors_k11,
    "graph recomputed": graph_k15 != graph_k11,
}

{'normalization reused': True,
 'PCA reused': True,
 'ANN index reused': True,
 'neighbors recomputed': True,
 'graph recomputed': True}

The graph-construction guide owns the scientific effect of changing k. Here the important result is where the immutable branch begins.

3. Vary dims: invalidate downstream#

A new PCA dimensionality creates a new reduction. ANN, neighbors, and connectivity that depend on the old reduction are not reused for the new chain.

pca_dims20 = ds.run_pca(normalized, dims=20)
ann_dims20 = ds.build_ann_index(pca_dims20)
neighbors_dims20 = ds.query_neighbors(ann_dims20, k=11)
graph_dims20 = ds.build_connectivity_map(neighbors_dims20)

{
    "PCA recomputed": pca_dims20 != pca,
    "ANN index recomputed": ann_dims20 != ann,
    "neighbors recomputed": neighbors_dims20 != neighbors_k11,
    "graph recomputed": graph_dims20 != graph_k11,
}

{'PCA recomputed': True,
 'ANN index recomputed': True,
 'neighbors recomputed': True,
 'graph recomputed': True}

4. Force recompute#

invalidate_cache=True skips reuse even when the parameters match. Previously completed artifacts remain on disk. The new reference has a different id and path. The operation and parameters stay the same.

For normalization, the call below would write a fresh normalized artifact while retaining the exact immutable cell_selection and feature_selection inputs. It is not executed here because a throwaway duplicate adds no evidence to the lineage figure.

forced = ds.run_normalization(
    cell_selection,
    hvg_ref,
    invalidate_cache=True,
)
status = ds.inspect_artifact(forced)
baseline_status = ds.inspect_artifact(normalized)
baseline_inputs = baseline_status.inputs or {}
forced_inputs = status.inputs or {}

{
    "new artifact": forced != normalized,
    "complete": status.complete,
    "operation": status.operation,
    "path": status.path,
    "baseline path": baseline_status.path,
    "same parameters": status.parameters == baseline_status.parameters,
    "same inputs": forced_inputs == baseline_inputs,
}

5. Compare lineage#

Build one read-only report from both neighbour-count branches and the dims=20 fork. Shared upstream nodes appear once; the forks show where each branch diverged.

lineage = ds.lineage(
    {
        "k11 graph": graph_k11,
        "k15 graph": graph_k15,
        "dims20 graph": graph_dims20,
    }
)
lineage

        flowchart LR
    artifact0["RNA / metadata_snapshot | snapshot_run_metadata | 293ca94d65b9"]
    artifact1["datastore / cell_selection | snapshot_pipeline_input_selection | 819afea150f0"]
    artifact2["datastore / metadata_snapshot | snapshot_run_metadata | 15d9edf057e4"]
    artifact3["datastore / cell_selection | filter_pipeline_cells | 03538281953b"]
    artifact4["RNA / feature_summary | summarize_rna_features | 64e42a796a82"]
    artifact5["RNA / feature_selection | select_hvgs | 53792654cf05"]
    artifact6["RNA / normalized | run_normalization | 839be6a0613c"]
    artifact7["RNA / feature_scaling | calculate_feature_scaling | 5b9d66b1bdb7"]
    artifact8["RNA / reduction | run_pca | 6d5f89422f43"]
    artifact9["RNA / ann_index | build_ann_index | bfa5752b81b3"]
    artifact10["RNA / neighbors | query_neighbors | 4b3a82191fcc"]
    artifact11["RNA / connectivity_map | build_connectivity_map | 5cdc62e75929 | outputs: k11 graph"]
    artifact12["RNA / neighbors | query_neighbors | 7acd8d15b998"]
    artifact13["RNA / connectivity_map | build_connectivity_map | 4b19e7dd51c2 | outputs: k15 graph"]
    artifact14["RNA / reduction | run_pca | cfe572c06d65"]
    artifact15["RNA / ann_index | build_ann_index | 088b96edf1a2"]
    artifact16["RNA / neighbors | query_neighbors | 52cc8fab597a"]
    artifact17["RNA / connectivity_map | build_connectivity_map | 0b3f5ad13b3b | outputs: dims20 graph"]
    artifact8 -->|"coordinates"| artifact10
    artifact9 -->|"ann_index"| artifact10
    artifact10 -->|"neighbors"| artifact11
    artifact8 -->|"coordinates"| artifact12
    artifact9 -->|"ann_index"| artifact12
    artifact12 -->|"neighbors"| artifact13
    artifact3 -->|"pca_cell_selection"| artifact14
    artifact6 -->|"normalized"| artifact14
    artifact7 -->|"feature_scaling"| artifact14
    artifact14 -->|"coordinates"| artifact15
    artifact14 -->|"coordinates"| artifact16
    artifact15 -->|"ann_index"| artifact16
    artifact16 -->|"neighbors"| artifact17
    artifact1 -->|"input_cell_selection"| artifact3
    artifact2 -->|"cell_snapshot"| artifact3
    artifact3 -->|"cell_selection"| artifact4
    artifact0 -->|"feature_snapshot"| artifact5
    artifact4 -->|"feature_summary"| artifact5
    artifact3 -->|"cell_selection"| artifact6
    artifact5 -->|"feature_selection"| artifact6
    artifact6 -->|"normalized"| artifact7
    artifact3 -->|"pca_cell_selection"| artifact8
    artifact6 -->|"normalized"| artifact8
    artifact7 -->|"feature_scaling"| artifact8
    artifact8 -->|"coordinates"| artifact9

Artifact details

RNA / metadata_snapshot / 293ca94d65b9

  • Status: complete

  • Path: RNA/artifacts/metadata_snapshot/293ca94d65b9605db12213217631264ddf301efc641275018add9b8da0810f90

  • Operation: snapshot_run_metadata

  • Parameters: assay="RNA"; axis="feature"; ordered_columns=["names"]

  • Other inputs: column_fingerprints={"names":"25b8d03a2f9b2b01183cc9742301f3cf9cd4762c4fe962fe3ad5189013d8c857"}; ordered_row_ids_fingerprint="4548c9820ab9ed0afbc48cbb6503bdcdc725b57c65dbfe3c4da8bb480b7fc4b1"

datastore / cell_selection / 819afea150f0

  • Status: complete

  • Path: artifacts/cell_selection/819afea150f0e598fde34df719a1be70d788d91354a9497a79f345791c3dac5a

  • Operation: snapshot_pipeline_input_selection

  • Parameters: assay="RNA"

  • Execution options: source_column="I"

  • Other inputs: ordered_row_ids_fingerprint="f5f05615ffc8833f75752b75152ccd728702f42950f81e9c28402880a4dd2ae5"; values_fingerprint="94f668fa448559768ea628c836d7ed1b683636c280cd42e9a6c9b73ba0179a38"

datastore / metadata_snapshot / 15d9edf057e4

  • Status: complete

  • Path: artifacts/metadata_snapshot/15d9edf057e452f709c66b65665aa58f371c1e0d23ed4f332d1151a4670d5412

  • Operation: snapshot_run_metadata

  • Parameters: assay=null; axis="cell"; ordered_columns=["names","RNA_nCounts","RNA_nFeatures","RNA_percentMito"]

  • Other inputs: column_fingerprints={"RNA_nCounts":"a8a7520000e32d28bcf97a8977290bcc7185570098e1fe95739c74687b435843","RNA_nFeatures":"a3df08addee7271194b0ebdfac85ad92ec93f3801031e65316776453eb...; ordered_row_ids_fingerprint="f5f05615ffc8833f75752b75152ccd728702f42950f81e9c28402880a4dd2ae5"

datastore / cell_selection / 03538281953b

  • Status: complete

  • Path: artifacts/cell_selection/03538281953b535b6237742b94c463d2b04fe7fa15f45420fa242f3b642b3a98

  • Operation: filter_pipeline_cells

  • Parameters: attrs=["RNA_nCounts","RNA_nFeatures","RNA_percentMito"]; enabled=true; highs=[15000,4000,15]; keepBounds=false; lows=[1000,500,0]; method="manual"

  • Execution options: source_column="I"

  • Other inputs: ordered_row_ids_fingerprint="f5f05615ffc8833f75752b75152ccd728702f42950f81e9c28402880a4dd2ae5"; values_fingerprint="f994b6b61f3cddbb8b890fca9fee8f9b5e02f530ccf7645f21bf49719a79c39d"

RNA / feature_summary / 64e42a796a82

  • Status: complete

  • Path: RNA/artifacts/feature_summary/64e42a796a823c31d2030cb99f8f7111b6ea3fbdf8c92c7372b033f936735006

  • Operation: summarize_rna_features

  • Parameters: normalization_method={"module":"scarf.assay","qualname":"norm_lib_size"}; size_factor=1000

  • Execution options: nthreads=2

  • Other inputs: dataset_fingerprint="cf9c200795fa3b1c7645f5ef830af6fd277d319cf56c7ef4bae6e2da9bba4b5a"

RNA / feature_selection / 53792654cf05

  • Status: complete

  • Path: RNA/artifacts/feature_selection/53792654cf05f97f09fdfecb14e7561a84821047d0573824bedef41037f3f4d6

  • Operation: select_hvgs

  • Parameters: bin_strategy="adaptive"; blacklist="^MT-|^RPS|^RPL|^MRPS|^MRPL|^CCN|^HLA-|^H2-|^HIST|^XIST$|^DDX3Y$|^USP9Y$|^EIF1AY$|^KDM5D$|^SRY$|^ZFY$|^UTY$|^TMSB4Y$|^NLGN4Y$"; blacklist_fingerprint="40cb83e4a3e9a81770a7ef6b7841595d91720fc1928e937503f6ca23c089175e"; keep_bounds=false; lowess_frac=0.1; max_cells=3928; max_mean={"special_float":"inf"}; max_var={"special_float":"inf"}; min_cells=20; min_mean={"special_float":"-inf"}; min_var={"special_float":"-inf"}; n_bins=200; ... 3 more

  • Execution options: invalidate_cache=false; nthreads=2; plot_kwargs={}; show_plot=false

RNA / normalized / 839be6a0613c

  • Status: complete

  • Path: RNA/artifacts/normalized/839be6a0613c943017b9a22806e920bfabe32953015bf552fb0237760008a3c2

  • Operation: run_normalization

  • Parameters: log_transform=true; normalization_method={"external_hook":true,"module":"scarf.assay","qualname":"norm_lib_size"}; renormalize_subset=true; size_factor=1000.0

  • Execution options: invalidate_cache=false

  • Other inputs: dataset_fingerprint="cf9c200795fa3b1c7645f5ef830af6fd277d319cf56c7ef4bae6e2da9bba4b5a"

RNA / feature_scaling / 5b9d66b1bdb7

  • Status: complete

  • Path: RNA/artifacts/feature_scaling/5b9d66b1bdb76066883e29f5f00b03b13bf61ed02e2a4738777d2de2a3fb0c9e

  • Operation: calculate_feature_scaling

  • Parameters: enabled=true

  • Execution options: batch_size=3948; invalidate_cache=false; local_cache="auto"

RNA / reduction / 6d5f89422f43

  • Status: complete

  • Path: RNA/artifacts/reduction/6d5f89422f43325772fa24e2abbfc985707cffe929f962fae7aa3d475f4e76f1

  • Operation: run_pca

  • Parameters: dims=15; feat_scaling=true

  • Execution options: batch_size=3948; invalidate_cache=false; local_cache="auto"; show_elbow_plot=false

RNA / ann_index / bfa5752b81b3

  • Status: complete

  • Path: RNA/artifacts/ann_index/bfa5752b81b3ea12d267afddb07914f97629a5eed78bbe0208e169ac237f0a4d

  • Operation: build_ann_index

  • Parameters: ann_ef=50; ann_efc=50; ann_m=48; ann_metric="l2"; ann_parallel=false; parallel_threads=null; rand_state=4466

  • Execution options: batch_size=3948; invalidate_cache=false

RNA / neighbors / 4b3a82191fcc

  • Status: complete

  • Path: RNA/artifacts/neighbors/4b3a82191fcc29dd22f1fa6720ab6dae3c775976b13ae1c9d7349de5496d02f2

  • Operation: query_neighbors

  • Parameters: distance_metric="l2"; k=11

  • Execution options: batch_size=3948; invalidate_cache=false

RNA / connectivity_map / 5cdc62e75929

  • Status: complete

  • Path: RNA/artifacts/connectivity_map/5cdc62e75929a73b516656bd812e4ab104140f6a7d1e0d86c2e13382692572bd

  • Operation: build_connectivity_map

  • Outputs: k11 graph

  • Parameters: bandwidth=1.5; local_connectivity=1.0

  • Execution options: invalidate_cache=false

RNA / neighbors / 7acd8d15b998

  • Status: complete

  • Path: RNA/artifacts/neighbors/7acd8d15b998167fa10998126506e53adc9e76f2e8009ae2a20581bffaead308

  • Operation: query_neighbors

  • Parameters: distance_metric="l2"; k=15

  • Execution options: batch_size=3948; invalidate_cache=false

RNA / connectivity_map / 4b19e7dd51c2

  • Status: complete

  • Path: RNA/artifacts/connectivity_map/4b19e7dd51c2b095104e9090e16897b5af00f64a1e7698045422f6ff480ab0f7

  • Operation: build_connectivity_map

  • Outputs: k15 graph

  • Parameters: bandwidth=1.5; local_connectivity=1.0

  • Execution options: invalidate_cache=false

RNA / reduction / cfe572c06d65

  • Status: complete

  • Path: RNA/artifacts/reduction/cfe572c06d65bd7528b738d0ed57d4f23bf21e1bbde7c4474381adefaecd1ec6

  • Operation: run_pca

  • Parameters: dims=20; feat_scaling=true

  • Execution options: batch_size=3948; invalidate_cache=false; local_cache="auto"; show_elbow_plot=false

RNA / ann_index / 088b96edf1a2

  • Status: complete

  • Path: RNA/artifacts/ann_index/088b96edf1a2e8580c70745839a82b9e55d3798bd0100452614c2b4dbf73f70f

  • Operation: build_ann_index

  • Parameters: ann_ef=50; ann_efc=50; ann_m=48; ann_metric="l2"; ann_parallel=false; parallel_threads=null; rand_state=4466

  • Execution options: batch_size=3948; invalidate_cache=false

RNA / neighbors / 52cc8fab597a

  • Status: complete

  • Path: RNA/artifacts/neighbors/52cc8fab597afb8ba4dab427c753b706952cdf5be1b6eaccc17741de22a82e1c

  • Operation: query_neighbors

  • Parameters: distance_metric="l2"; k=11

  • Execution options: batch_size=3948; invalidate_cache=false

RNA / connectivity_map / 0b3f5ad13b3b

  • Status: complete

  • Path: RNA/artifacts/connectivity_map/0b3f5ad13b3ba0c680542f9e5c3af6b8a02e95d034384b933fce4ed390eeddcf

  • Operation: build_connectivity_map

  • Outputs: dims20 graph

  • Parameters: bandwidth=1.5; local_connectivity=1.0

  • Execution options: invalidate_cache=false

Notebook display renders the Mermaid dependency graph and the artifact details beneath it. The k branches should diverge after the ANN index. The dims=20 branch should fork earlier, at PCA, then carry its own ANN, neighbours, and graph.

Export the same report when it needs to travel with an analysis. to_markdown() is what notebook display uses. Inspect a short preview before writing or sending the complete string elsewhere:

lineage_markdown = lineage.to_markdown()
lineage_markdown.splitlines()[:12]

['```mermaid',
 'flowchart LR',
 '    artifact0["RNA / metadata_snapshot | snapshot_run_metadata | 293ca94d65b9"]',
 '    artifact1["datastore / cell_selection | snapshot_pipeline_input_selection | 819afea150f0"]',
 '    artifact2["datastore / metadata_snapshot | snapshot_run_metadata | 15d9edf057e4"]',
 '    artifact3["datastore / cell_selection | filter_pipeline_cells | 03538281953b"]',
 '    artifact4["RNA / feature_summary | summarize_rna_features | 64e42a796a82"]',
 '    artifact5["RNA / feature_selection | select_hvgs | 53792654cf05"]',
 '    artifact6["RNA / normalized | run_normalization | 839be6a0613c"]',
 '    artifact7["RNA / feature_scaling | calculate_feature_scaling | 5b9d66b1bdb7"]',
 '    artifact8["RNA / reduction | run_pca | 6d5f89422f43"]',
 '    artifact9["RNA / ann_index | build_ann_index | bfa5752b81b3"]']

lineage.to_mermaid() returns only the diagram source when a tooling pipeline needs that form alone.