squidpy.gr.calculate_niche_cellcharter

squidpy.gr.calculate_niche_cellcharter(data, *, distance=3, aggregation='mean', rng=None, spatial_connectivities_key='spatial_connectivities', n_clusters=10, n_pca_components=None, n_jobs=None, use_rep=None, embedding_key_added='niche_embedding', key_added='cellcharter_niche', min_niche_size=None, cluster_mask=None, copy=False, table_key=None)[source]

Compute spatial niches using a CellCharter-style embedding and GMM [Varrone et al., 2023].

CellCharter recommends a dimensionality-reduced embedding such as scVI, passed as use_rep; PCA of adata.X is the fallback when it is not given. The mixture model is scikit-learn’s, not CellCharter’s torchgmm, so partitions will not match theirs.

This method identifies niches by clustering an embedding that represents each observation together with information from its surrounding spatial neighborhood. Unlike calculate_niche_neighborhood(), which builds an embedding from categorical cell-type composition, this approach uses a continuous feature representation and a Gaussian mixture model (GMM) for clustering.

A spatial connectivity graph must already be present in adata.obsp[spatial_connectivities_key]. This function does not construct the graph itself.

The method:

  1. Takes a reduced representation of every observation: adata.obsm[use_rep] if given, otherwise the first n_pca_components principal components of adata.X, restricted to adata.var["highly_variable"] when that column is present.

  2. Aggregates that representation over each disjoint hop ring of the spatial graph, from direct neighbors through distance graph hops, according to aggregation.

  3. Concatenates the observation’s own representation with every ring’s aggregate.

  4. Fits a Gaussian mixture model with n_clusters mixture components.

  5. Uses the GMM component assignments as niche labels.

Consequently, each niche corresponds to a probabilistic cluster in a feature space that encodes both an observation’s features and its broader spatial context. Increasing distance allows the embedding to reflect larger tissue-scale neighborhoods, whereas smaller values emphasize local microenvironments.

Parameters:

  • adata – Annotated data object.

  • distance (int) – Maximum number of graph hops included when constructing the CellCharter-style spatial embedding. distance=1 emphasizes direct neighbors; larger values incorporate progressively more distal observations in the spatial graph.

  • aggregation (str) –

    Aggregation statistic used to summarize features across spatial neighborhoods. Typical options include "mean" and "variance".

    "mean" emphasizes the average local feature state, such as the average local expression program. "variance" emphasizes local heterogeneity in the feature representation.

  • rng (int | integer | Sequence[int] | SeedSequence | Generator | BitGenerator | None) – Pseudorandom number generator state, following SPEC 7. When None, a new numpy.random.Generator is created using entropy from the operating system. Types other than numpy.random.Generator are passed to numpy.random.default_rng() to instantiate a generator. Seeds the Gaussian mixture clustering step.

  • spatial_connectivities_key (str) – Key in anndata.AnnData.obsp containing the spatial connectivity matrix.

  • n_clusters (int) – Number of Gaussian mixture components, and therefore the number of niche labels produced per library or dataset.

  • n_pca_components (int | None) – Number of principal components of adata.X aggregated over the hop rings. None uses 10, the latent width scVI defaults to, or one fewer than the smallest dimension of adata.X when that is narrower. Named for the PCA because GaussianMixture spells its cluster count n_components too; here that is n_clusters. Rejected together with use_rep, which supplies an already reduced embedding and so skips the PCA.

  • n_jobs (int | None) – Number of parallel threads to use. None and -1 use numba’s default thread count (NUMBA_NUM_THREADS); asking for more threads than that warns and falls back to it, while 0 and values below -1 raise, since scanpy only supports n_jobs >= -1.

  • use_rep (str | None) – Key in anndata.AnnData.obsm holding a reduced representation, such as an scVI latent, or 'X' for X. Aggregated over the hop rings in place of the PCA of adata.X.

  • cluster_mask (Series | None) – Boolean pandas.Series indexed like anndata.AnnData.obs. False observations are labeled 'not_a_niche' and take no part in the clustering, though they still reach their neighbors through the graph. Observations it omits are kept. To drop them from the graph as well, subset and rebuild it with spatial_neighbors().

  • key_added (str) – Name of the anndata.AnnData.obs column the labels are written to.

  • embedding_key_added (str) – Controls the name of the key to be added in anndata.AnnData.obsm, which will hold the niche embedding computed.

  • min_niche_size (int | None) – Minimum number of observations required for a niche. Niches with fewer observations are relabeled 'not_a_niche'.

  • copy (bool) – If True, return the result, otherwise save it to the adata object.

  • table_key (str | None) – Key in spatialdata.SpatialData.tables where the table is stored. Required when adata is a spatialdata.SpatialData object and ignored otherwise.

Return type:

AnnData | None

Returns:

If copy=True, returns a copy of adata with the embedding stored in .obsm[embedding_key_added] and GMM-based niche assignments added to .obs. Otherwise, modifies adata in place and returns None.