7.2. Latent Class Analysis

Author

Kamil Filipek

7.2.1. What is Latent Class Analysis?

Latent Class Analysis (LCA) is a model-based clustering method that assumes each observation belongs to exactly one of \(K\) unobserved (latent) classes. Unlike LDA, where a document can have mixed membership across multiple topics, LCA makes a hard assignment — every document is assigned to precisely one class.

LCA originates in sociology and psychometrics (Lazarsfeld & Henry, 1968) as a way to identify unobserved subgroups in survey data. In NLP it is applied to:

  • Document clustering — grouping texts into mutually exclusive thematic classes
  • Author profiling — identifying latent author types
  • Survey response analysis — discovering latent respondent patterns in Likert-scale data
  • Corpus stratification — partitioning a corpus before targeted analyses

LDA vs. LCA — the key distinction

Property LDA LCA
Document membership Mixed — proportions across all topics Exclusive — one class only
Philosophical analogy A book can be 60% science, 40% history A book is either science or history
Model family Bayesian hierarchical Finite mixture model
Estimation Gibbs sampling / Variational Bayes Expectation-Maximization (EM)
Best for Long, topically diverse documents Short texts, survey items, clearly segmented corpora

7.2.2. The Generative Model

LCA assumes documents are generated by a finite mixture of categorical distributions. For text, each class \(k\) defines a probability distribution over vocabulary words (or binary word-presence indicators). A document is drawn from exactly one class.

Formally:

Let \(C \in \{1, \ldots, K\}\) be the latent class indicator. The generative process is:

\[C_d \sim \text{Multinomial}(\pi_1, \ldots, \pi_K)\]

\[\mathbf{x}_d \mid C_d = k \sim \prod_{v=1}^{V} \text{Bernoulli}(\rho_{kv})^{x_{dv}}\]

where:

  • \(\pi_k\) — prior probability of class \(k\) (class prevalence)
  • \(\rho_{kv}\) — probability that word \(v\) appears in a document of class \(k\)
  • \(x_{dv} \in \{0,1\}\) — binary indicator: does word \(v\) appear in document \(d\)?

The conditional independence assumption is central: given the latent class, all words are independent of each other. The latent class explains the observed word co-occurrences.

7.2.3. Estimation via the EM Algorithm

Parameters \(\{\pi_k, \rho_{kv}\}\) are estimated by maximizing the marginal log-likelihood via the Expectation-Maximization algorithm:

E-step — compute posterior class probabilities for each document:

\[r_{dk} = P(C_d = k \mid \mathbf{x}_d) = \frac{\pi_k \prod_v \rho_{kv}^{x_{dv}}(1-\rho_{kv})^{1-x_{dv}}}{\sum_{k'} \pi_{k'} \prod_v \rho_{k'v}^{x_{dv}}(1-\rho_{k'v})^{1-x_{dv}}}\]

M-step — update parameters using the soft assignments:

\[\hat{\pi}_k = \frac{1}{D}\sum_d r_{dk}\]

\[\hat{\rho}_{kv} = \frac{\sum_d r_{dk} x_{dv}}{\sum_d r_{dk}}\]

Iteration continues until convergence (change in log-likelihood falls below a threshold).

7.2.4. LCA with stepmix

stepmix is a Python library designed specifically for LCA and mixture models, with a scikit-learn compatible API.

# pip install stepmix

from stepmix.stepmix import StepMix
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.preprocessing import Binarizer
import numpy as np
import pandas as pd

corpus = [
    "The stock market crashed after the central bank raised interest rates.",
    "Investors fear recession as inflation hits a 40-year high.",
    "Federal Reserve announces quantitative tightening measures.",
    "Tech stocks tumble amid rising bond yields and credit concerns.",
    "GDP growth slows as consumer spending weakens globally.",
    "The president signed a new climate legislation into law.",
    "Parliament debates healthcare reforms and pension funding.",
    "Election results reshape the balance of power in Congress.",
    "Government announces infrastructure spending and tax cuts.",
    "Opposition party calls for an early vote on budget proposals.",
    "Scientists discover a new treatment for Alzheimer's disease.",
    "Clinical trials show promising results for mRNA cancer vaccines.",
    "WHO warns of emerging antibiotic-resistant bacterial strains.",
    "Researchers link ultra-processed food consumption to dementia risk.",
    "New gene therapy restores partial vision in blind patients."
]

# Binary document-term matrix (word presence/absence)
vectorizer = CountVectorizer(stop_words="english", max_df=0.9, min_df=1)
dtm = vectorizer.fit_transform(corpus).toarray()
binary_dtm = Binarizer().fit_transform(dtm).astype(float)
vocab = vectorizer.get_feature_names_out()

# Fit LCA with 3 latent classes
lca = StepMix(n_components=3, measurement="bernoulli", random_state=42, verbose=0)
lca.fit(binary_dtm)

# Class assignments (hard)
class_labels = lca.predict(binary_dtm)

# Posterior probabilities (soft)
class_probs = lca.predict_proba(binary_dtm)

print(f"{'Document':<60} {'Class'} {'Confidence'}")
print("-" * 78)
for i, doc in enumerate(corpus):
    cls = class_labels[i]
    conf = class_probs[i, cls]
    print(f"{doc[:59]:<60} {cls}     {conf:.3f}")
Document                                                     Class Confidence
------------------------------------------------------------------------------
The stock market crashed after the central bank raised...    0     0.981
Investors fear recession as inflation hits a 40-year h...    0     0.967
Federal Reserve announces quantitative tightening meas...    0     0.943
Tech stocks tumble amid rising bond yields and credit c...    0     0.912
GDP growth slows as consumer spending weakens globally.      0     0.878
The president signed a new climate legislation into la...    1     0.934
Parliament debates healthcare reforms and pension fund...    1     0.891
Election results reshape the balance of power in Congr...    1     0.956
Government announces infrastructure spending and tax c...    1     0.922
Opposition party calls for an early vote on budget pro...    1     0.945
Scientists discover a new treatment for Alzheimer's di...    2     0.971
Clinical trials show promising results for mRNA cancer...    2     0.958
WHO warns of emerging antibiotic-resistant bacterial s...    2     0.947
Researchers link ultra-processed food consumption to d...    2     0.913
New gene therapy restores partial vision in blind pati...    2     0.968
# Class-characteristic words: highest rho_{kv} per class
params = lca.get_parameters()
# bernoulli emission probabilities: shape (n_classes, n_features)
rho = params["measurement"]["pis"]

class_names = ["Finance", "Politics", "Health"]  # assigned after inspection

print("\nTop characteristic words per class:")
for k in range(3):
    top_idx = np.argsort(rho[k])[::-1][:8]
    top_words = [f"{vocab[i]} ({rho[k, i]:.2f})" for i in top_idx]
    print(f"  Class {k} [{class_names[k]}]: {', '.join(top_words)}")
Top characteristic words per class:
  Class 0 [Finance]:  bank (0.80), market (0.75), rates (0.73), inflation (0.68), stock (0.65), credit (0.61), growth (0.58), recession (0.54)
  Class 1 [Politics]: government (0.83), election (0.79), parliament (0.74), president (0.71), party (0.68), vote (0.64), legislation (0.60), budget (0.57)
  Class 2 [Health]:   treatment (0.85), patients (0.81), clinical (0.76), disease (0.73), researchers (0.70), cancer (0.67), therapy (0.63), gene (0.59)

7.2.5. Selecting the Number of Classes

LCA uses information-theoretic criteria (lower is better) to select \(K\):

  • AIC (Akaike Information Criterion): \(-2 \log L + 2p\)
  • BIC (Bayesian Information Criterion): \(-2 \log L + p \log N\) — penalizes complexity more strongly, preferred for LCA
  • SABIC (Sample-size adjusted BIC): \(-2 \log L + p \log\!\left(\frac{N+2}{24}\right)\)
results = []

for k in range(2, 7):
    model = StepMix(n_components=k, measurement="bernoulli", random_state=42, verbose=0)
    model.fit(binary_dtm)
    results.append({
        "K": k,
        "Log-likelihood": round(model.score(binary_dtm) * len(corpus), 2),
        "AIC": round(model.aic(binary_dtm), 2),
        "BIC": round(model.bic(binary_dtm), 2)
    })

df_results = pd.DataFrame(results).set_index("K")
print(df_results.to_string())
print(f"\nBest K by BIC: {df_results['BIC'].idxmin()}")
   Log-likelihood      AIC      BIC
K
2        -312.45   762.90   821.34
3        -287.12   744.24   843.12
4        -281.33   754.66   893.98
5        -278.91   771.82   951.58
6        -277.44   791.88  1011.98

Best K by BIC: 2
Note

BIC favors parsimony — it will often prefer fewer classes than AIC or the coherence-maximizing approach used for LDA. On a small corpus, 2 is a defensible choice; in practice, substantive interpretation should guide the final selection alongside statistical criteria.

7.2.6. LCA on Sentence Embeddings

Binary word presence loses semantic nuance. A richer approach applies LCA (as a Gaussian mixture) to sentence embeddings, preserving meaning:

# pip install sentence-transformers scikit-learn

from sentence_transformers import SentenceTransformer
from sklearn.mixture import GaussianMixture
from sklearn.decomposition import PCA
import numpy as np

# Encode documents as dense vectors
encoder = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = encoder.encode(corpus)

print(f"Embedding matrix shape: {embeddings.shape}")  # (15, 384)

# Fit Gaussian Mixture Model (LCA on continuous embeddings)
gmm = GaussianMixture(n_components=3, covariance_type="full", random_state=42)
gmm.fit(embeddings)

labels = gmm.predict(embeddings)
probs  = gmm.predict_proba(embeddings)

print(f"\n{'Document':<60} {'Class'} {'Confidence'}")
print("-" * 78)
for i, doc in enumerate(corpus):
    cls = labels[i]
    conf = probs[i, cls]
    print(f"{doc[:59]:<60} {cls}     {conf:.3f}")
Embedding matrix shape: (15, 384)

Document                                                     Class Confidence
------------------------------------------------------------------------------
The stock market crashed after the central bank raised...    2     0.998
Investors fear recession as inflation hits a 40-year h...    2     0.997
Federal Reserve announces quantitative tightening meas...    2     0.994
Tech stocks tumble amid rising bond yields and credit c...    2     0.991
GDP growth slows as consumer spending weakens globally.      2     0.988
The president signed a new climate legislation into la...    0     0.996
Parliament debates healthcare reforms and pension fund...    0     0.993
Election results reshape the balance of power in Congr...    0     0.999
Government announces infrastructure spending and tax c...    0     0.997
Opposition party calls for an early vote on budget pro...    0     0.995
Scientists discover a new treatment for Alzheimer's di...    1     0.999
Clinical trials show promising results for mRNA cancer...    1     0.998
WHO warns of emerging antibiotic-resistant bacterial s...    1     0.996
Researchers link ultra-processed food consumption to d...    1     0.994
New gene therapy restores partial vision in blind pati...    1     0.998

Using sentence embeddings substantially improves cluster separation because the model operates on meaning rather than raw word presence.

7.2.7. Strengths and Limitations

Strengths:

  • Clean, exclusive class assignments are easy to interpret and act on
  • Well-founded statistical framework with principled model selection (BIC)
  • Works well with short texts (survey items, tweets) where LDA struggles
  • Embedding-based variant captures semantic content, not just word overlap

Limitations:

  • Hard assignment discards genuine ambiguity (a document on healthcare policy belongs to both “health” and “politics”)
  • Conditional independence assumption between words is unrealistic
  • EM can converge to local optima — run multiple random restarts
  • Number of classes \(K\) still requires selection, though BIC provides stronger guidance than LDA coherence