Assays and metadata API reference

Assays and metadata API reference#

Assay.score_features(feature_names, cell_key, ctrl_size, n_bins, rand_seed) is computation-only. It computes full-row-order averages blockwise in memory and is safe on read-only counts; it does not plan artifacts or write metadata. Persistent cell-cycle outputs belong to DataStore.run_cell_cycle_scoring.

Feature-count percentages belong to DataStore.run_feature_percentage(cell_selection, features). It derives the assay from the exact feature-selection ref and returns an assay-scoped quality_metric ref whose values array is the per-cell percentage. It does not add a metadata column.

Persisted normalization belongs to DataStore.run_normalization(cell_selection, features), which returns an immutable artifact ref. Assay normalization methods are computation-only and require explicit feature indexes where they do not accept an ArtifactRef.

class scarf.assay.Assay(z, workspace, name, cell_data, nthreads, matrix_root=None, resources=None, storageIo=None)#

A generic Assay class that contains methods to calculate feature level statistics and stream normalized values for downstream computation.

Parameters:

  • z (Group) – Zarr hierarchy where raw data is located

  • workspace (str | None) – Workspace name when assays live under matrices/ (None for legacy layout)

  • name (str) – A label/name for assay.

  • cell_data (MetaData) – Metadata class object for the cell attributes.

  • nthreads (int) – number of threads to use for parallel computations

name#

A label for the assay instance

z#

Zarr group that contains the assay

cells#

A Metadata class object for cell attributes

nthreads#

number of threads to use for computations

rawData#

chunked array containing the raw data

feats#

a MetaData class object for feature attributes

attrs#

Zarr attributes for the zarr group of the assay

normMethod#

normalization method to use.

sf#

scaling factor for doing library-size normalization

iter_normed_feature_wise(cell_idx, feat_idx, batch_size, msg, as_dataframe=True, **norm_params)#

Iterate over explicitly selected normalized features in batches.

Parameters:

  • cell_idx (ndarray) – Ordered physical cell indices to include.

  • feat_idx (ndarray) – Ordered physical feature indices to include.

  • batch_size (int | None) – Number of genes loaded at a time. When None, selected features are grouped into chunk-aligned blocks that fit the operation memory budget.

  • msg (str | None) – Message to be displayed in the progress bar

  • as_dataframe (bool) – If true (default) then the yielded matrices are pandas dataframe

  • **norm_params (Any) – Extra keyword arguments forwarded to normed.

Return type:

Generator[DataFrame | tuple[ndarray, ndarray], None, None]

Returns:

Generator yielding DataFrames or (matrix, feature index) tuples.

mean_features(feature_names, cell_key='I', *, missing='error')#

Per-cell mean normalized expression over named features.

Returns one value per active cell under cell_key. Does not write cell metadata. Distinct from score_features, which subtracts a control-gene background.

Return type:

ndarray

normed(cell_idx=None, feat_idx=None, **kwargs)#

This function normalizes the raw and returns a delayed chunked array of the normalized data.

Parameters:

  • cell_idx (ndarray | None) – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table)

  • feat_idx (ndarray | None) – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis.

  • **kwargs (Any)

Return type:

ChunkedArray

Returns: A chunked array (delayed matrix) containing normalized data.

score_features(feature_names, cell_key, ctrl_size, n_bins, rand_seed, *, log_transform=False)#

Calculates the scores (mean values) of selection of features over a randomly sampled selected feature set in given cells (as marked by cell_key)

Parameters:

  • feature_names (list[str]) – Names (as in ‘names’ column of the feature attribute table) of features to be used for scoring

  • cell_key (str) – Name of the key (column) from cell attribute table.

  • ctrl_size (int) – Number of reference features to be sampled from each bin.

  • n_bins (int) – Number of bins for sampling.

  • rand_seed (int) – The seed to use for the random number generation.

Return type:

ndarray

Returns: Numpy array of the calculated scores

to_raw_sparse(cell_key)#
Parameters:

cell_key (str) – A column from cell attribute table. This column must be a boolean type. The data will be exported for only those that have a True value in this column.

Return type:

csr_matrix

Returns: A sparse matrix containing raw data.

class scarf.assay.RNAassay(z, name, cell_data, *, workspace=None, nthreads=1, **kwargs)#

This subclass of Assay is designed for feature selection and normalization of scRNA-Seq data.

Parameters:

  • z (Group) – Zarr hierarchy where raw data is located

  • name (str) – A label/name for assay.

  • cell_data (MetaData) – Metadata class object for the cell attributes.

  • **kwargs (Any) – kwargs to be passed to the Assay class

normMethod#

A pointer to the function to be used for normalization of the raw data

sf#

scaling factor for doing library-size normalization

scalar#

This is used to cache the library size of the cells. It is set to None until normed method is called.

iter_normed_feature_wise(cell_idx, feat_idx, batch_size, msg, as_dataframe=True, **norm_params)#

Iterate over explicitly selected normalized features in batches.

Parameters:

  • cell_idx (ndarray) – Ordered physical cell indices to include.

  • feat_idx (ndarray) – Ordered physical feature indices to include.

  • batch_size (int | None) – Number of genes loaded at a time. When None, selected features are grouped into chunk-aligned blocks that fit the operation memory budget.

  • msg (str | None) – Message to be displayed in the progress bar

  • as_dataframe (bool) – If true (default) then the yielded matrices are pandas dataframe

  • **norm_params (Any) – Extra keyword arguments forwarded to normed.

Return type:

Generator[DataFrame | tuple[ndarray, ndarray], None, None]

Returns:

Generator yielding DataFrames or (matrix, feature index) tuples.

normed(cell_idx=None, feat_idx=None, renormalize_subset=False, log_transform=False, **kwargs)#

This function normalizes the raw and returns a delayed chunked array of the normalized data. Unlike the normed method in the generic Assay class this method is optimized for scRNA-Seq data and takes additional parameters that will be used by norm_lib_size (default normalization method for this class).

Parameters:

  • cell_idx (ndarray | None) – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table)

  • feat_idx (ndarray | None) – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis.

  • renormalize_subset (bool) – If true, normalize using only feat_idx rather than total expression across all features in a cell. (Default value: False)

  • log_transform (bool) – If True, then the normalized data is log-transformed (Default value: False).

  • **kwargs (Any) – kwargs have no effect here.

Return type:

ChunkedArray

Returns:

A chunked array (delayed matrix) containing normalized data.

class scarf.assay.ATACassay(z, name, cell_data, *, workspace=None, nthreads=1, **kwargs)#

This subclass of Assay is designed for feature selection and normalization of scATAC-Seq data.

normed(cell_idx=None, feat_idx=None, **kwargs)#

This function normalizes the raw and returns a delayed chunked array of the normalized data. Unlike the normed method in the generic Assay class this method is optimized for scATAC-Seq data. This method uses the normalization indicated by attribute self.normMethod which by default is set to norm_tf_idf. Document frequency is learned from cell_idx. The returned matrix contains feat_idx, while term frequency uses total ATAC counts unless subset renormalization is requested.

Parameters:

  • cell_idx (ndarray | None) – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table)

  • feat_idx (ndarray | None) – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis.

  • **kwargs (Any) – log_transform must be false. renormalize_subset uses counts among feat_idx as the term-frequency denominator.

Return type:

ChunkedArray

Returns: A chunked array (delayed matrix) containing normalized data.

class scarf.assay.ADTassay(z, name, cell_data, *, workspace=None, nthreads=1, **kwargs)#

This subclass of Assay is designed for normalization of ADT/HTO (feature-barcodes library) data from CITE-Seq experiments.

Parameters:

  • z (Group) – Zarr hierarchy where raw data is located

  • name (str) – A label/name for assay.

  • cell_data (MetaData) – Metadata class object for the cell attributes.

  • **kwargs (Any)

normMethod#

Pointer to the function to be used for normalization of the raw data

normed(cell_idx=None, feat_idx=None, **kwargs)#

This function normalizes the raw and returns a delayed chunked array of the normalized data. This method uses the normalization indicated by attribute self.normMethod which by default is set to norm_clr. The centered log-ratio normalization is performed using only the cells and features indicated by the ‘cell_idx’ and ‘feat_idx’ parameters.

Parameters:

  • cell_idx (ndarray | None) – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table)

  • feat_idx (ndarray | None) – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis.

  • **kwargs (Any)

Return type:

ChunkedArray

Returns: A chunked array (delayed matrix) containing normalized data.

scarf.assay.norm_dummy(_, counts)#

A dummy normalizer. Doesn’t perform any normalization. This is useful when the ‘raw data’ is already normalized.

Parameters:

  • _ (Assay)

  • counts (ChunkedArray) – A chunked array with ‘raw’ counts data

Return type:

ChunkedArray

Returns: A chunked array

class scarf.metadata.MetaData(zgrp)#

Metadata table for cells and features backed by Zarr arrays.

Changes made through this class are synchronized with the backing store.

active_index(key)#

Return global row indices selected by a boolean column.

Return type:

ndarray

property columns: list[str]#

Return all mounted metadata column names.

default_block_rows(column='I')#

Prefer the Zarr chunk length of column for row iteration.

Return type:

int

drop(column)#

Delete an unprotected metadata column.

Return type:

None

fetch(column, key='I')#

Return column values for rows selected by key.

Return type:

ndarray

fetch_all(column)#

Return all values from a metadata column.

Return type:

ndarray

fetch_all_columns(columns)#

Return several whole columns; object stores read them concurrently.

Return type:

list[ndarray]

get_dtype(column)#

Return the dtype of a metadata column.

Return type:

dtype[Any]

get_index_by(value_targets, column, key=None)#

Return row indices for requested values in a metadata column.

Return type:

ndarray

grep(pattern, only_valid=False)#

Return feature names matching a case-insensitive regex.

Return type:

list[str]

head(n=5)#

Return the first n rows of all metadata columns.

Return type:

DataFrame

index_to_bool(idx, invert=False)#

Convert row indices into a table-sized boolean array.

Return type:

ndarray

insert(column_name, values, fill_value=nan, key='I', overwrite=False, location='primary', force=False)#

Insert a column into the table.

Return type:

None

iter_row_blocks(*, cell_key='I', columns=None, block_rows=None)#

Yield contiguous row blocks over this table.

Each block covers a half-open global index range [start, stop). active_global_indices lists rows in that range that pass cell_key. Column arrays are aligned to those active indices only.

Return type:

Iterator[MetaDataRowBlock]

multi_sift(columns, lows, highs, keep_bounds=False)#

Return a boolean mask where all column filters are satisfied.

Return type:

ndarray

remove_trend(x, y, n_bins=200, lowess_frac=0.1, fill_value=0)#

Remove a LOWESS trend of column y with respect to column x.

Return type:

ndarray

reset_key(key)#

Set every value in a boolean metadata key to true.

Return type:

None

sift(column, min_v=-inf, max_v=inf, keep_bounds=False)#

Return rows whose values fall within the requested bounds.

Return type:

ndarray

to_pandas_dataframe(columns, key=None)#

Return requested columns as a DataFrame, optionally filtered by key.

Return type:

DataFrame

update_key(values, key)#

Restrict a boolean metadata key using the supplied values.

Return type:

None

ATAC coordinate melding#

class scarf.GffReader(gff_fn, up_offset=1000, down_offset=500, chunk_size=100000)#

Reader for a GFF3 format file.

static d_apply(d, func)#

Apply a function over dataframe rows and return an array.

Return type:

ndarray

fetch_header_lines()#

Fetch header lines starting with # from a GFF file.

Return type:

list[str]

get_body(v)#

Create strand-aware gene body and promoter coordinates.

Return type:

tuple[int, int]

static get_ids_names(v)#

Extract gene ID and gene name values from a GFF record.

Return type:

tuple[str | None, str | None]

get_promoter(v)#

Create strand-aware promoter coordinates.

Return type:

tuple[int, int]

stream()#

Stream the GFF file in chunks as pandas dataframes.

Return type:

DataFrame

to_bed(out_bed_fn, flavour='body')#

Convert gene annotations from GFF to a six-column BED file.

Return type:

None

scarf.coordinate_melding(assay, workspace, feature_bed, new_assay_name, peaks_col='ids', scalar_coeff=100000.0, renormalization=True, peaks_coords=None, idf_cell_idx=None)#

Transfer coordinate-based assay values to overlapping external features.

Parameters:

  • assay (Assay) – Source assay whose features have genomic coordinates.

  • workspace (str | None) – Workspace name. None uses the legacy layout.

  • feature_bed (DataFrame) – External interval table used as the meld target.

  • new_assay_name (str) – Name of the assay group to create.

  • peaks_col (str) – Feature-metadata column holding source coordinates.

  • scalar_coeff (float) – Scaling coefficient applied during melding.

  • renormalization (bool) – If True, rescale melded values after mapping.

  • peaks_coords (ndarray | None) – Optional precomputed source coordinates. When None, values are read from peaks_col.

  • idf_cell_idx (ndarray | None) – Optional cell indices used for IDF statistics.

Return type:

None

Returns:

None