4.6. Similarity Measures in NLP

Author

Kamil Filipek

4.6.1. Why Similarity Matters in NLP

Many NLP tasks reduce to a single question: how alike are two pieces of text? Whether we are ranking search results, deduplicating documents, finding paraphrases, or clustering sentences, we need a principled way to assign a score to that question.

Similarity measures fall into three broad families:

Family	Works on	Examples
String-based	Raw character or token sequences	Levenshtein, Jaccard, Dice
Vector-based	Numerical embeddings	Cosine, Euclidean, Dot-product
Fuzzy / approximate	Strings with tolerance for noise	Jaro-Winkler, fuzzy matching (FuzzyWuzzy / RapidFuzz)

4.6.2. Cosine Similarity

Cosine similarity is by far the most common measure in NLP. Given two vectors \(\mathbf{u}, \mathbf{v} \in \mathbb{R}^d\), it measures the angle between them, ignoring their magnitudes:

\[ \cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \cdot \|\mathbf{v}\|} = \frac{\sum_{i=1}^{d} u_i v_i}{\sqrt{\sum_{i=1}^{d} u_i^2} \cdot \sqrt{\sum_{i=1}^{d} v_i^2}} \]

Range: \([-1, 1]\) for general vectors; \([0, 1]\) for non-negative embeddings (TF-IDF, most word vectors).
A value of 1 means the vectors point in the same direction (maximally similar); 0 means orthogonal (no shared direction); −1 means opposite.

Why cosine and not Euclidean distance?

For text embeddings the magnitude of a vector often reflects document length, not meaning. Two short and long versions of the same sentence would look very different under Euclidean distance but almost identical under cosine similarity because the direction stays the same.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import TfidfVectorizer

docs = [
    "The cat sat on the mat.",
    "A cat is sitting on the mat.",
    "Dogs are running in the park.",
]

vec = TfidfVectorizer()
X = vec.fit_transform(docs)

sim = cosine_similarity(X)
print("Cosine similarity matrix:")
print(np.round(sim, 3))
# Expected: docs 0 and 1 are much more similar to each other than to doc 2

4.6.3. Jaccard Similarity

Jaccard similarity treats documents as sets of tokens and measures their overlap:

\[ J(A, B) = \frac{|A \cap B|}{|A \cup B|} \]

Range: \([0, 1]\).
Insensitive to word frequency — a word either is or is not in the document.
Useful for short texts (tweets, keywords) and deduplication tasks.

def jaccard(text_a: str, text_b: str) -> float:
    a, b = set(text_a.lower().split()), set(text_b.lower().split())
    return len(a & b) / len(a | b) if a | b else 0.0

print(jaccard("the cat sat on the mat", "a cat is sitting on the mat"))  # ~0.33
print(jaccard("the cat sat on the mat", "dogs run in the park"))         # 0.0

Dice coefficient is a close relative — it is \(2|A \cap B| / (|A| + |B|)\) — giving slightly more weight to the intersection.

4.6.4. Edit-Distance Based Measures

Edit-distance measures count the minimum number of character-level operations needed to transform one string into another.

Levenshtein Distance

\[ \text{lev}(a, b) = \begin{cases} |a| & \text{if } |b| = 0 \\ |b| & \text{if } |a| = 0 \\ \text{lev}(a[1:], b[1:]) & \text{if } a[0] = b[0] \\ 1 + \min\begin{cases}\text{lev}(a[1:], b) \\ \text{lev}(a, b[1:]) \\ \text{lev}(a[1:], b[1:])\end{cases} & \text{otherwise} \end{cases} \]

Operations allowed: insert, delete, substitute (each costs 1).

import Levenshtein  # pip install python-Levenshtein

a, b = "kitten", "sitting"
print(Levenshtein.distance(a, b))       # 3
print(Levenshtein.ratio(a, b))          # normalised similarity in [0,1]

Jaro-Winkler Similarity

Designed for short strings and proper names. It gives extra weight to a common prefix, making it robust to OCR errors and typos at the start of a word. Range: \([0, 1]\).

import jellyfish  # pip install jellyfish

print(jellyfish.jaro_winkler_similarity("martha", "marhta"))   # ~0.96
print(jellyfish.jaro_winkler_similarity("jones",  "johnson"))  # ~0.83

4.6.5. Fuzzy Matching

Fuzzy matching is a practical, engineering-oriented approach to approximate string comparison. The most popular library is RapidFuzz (a fast re-implementation of the older FuzzyWuzzy).

from rapidfuzz import fuzz, process  # pip install rapidfuzz

# Simple ratio — longest common substring approach
print(fuzz.ratio("New York Mets", "New York Meats"))        # ~96
print(fuzz.partial_ratio("Mets", "New York Mets"))          # 100  (substring)

# Token sort ratio — order-insensitive
print(fuzz.token_sort_ratio("fuzzy wuzzy", "wuzzy fuzzy"))  # 100

# Find best match in a list
choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
print(process.extractOne("new york jets", choices))          # ('New York Jets', 100, ...)

When to use which fuzzy scorer:

Scorer	Best for
`ratio`	Similar-length strings, typos
`partial_ratio`	Short query inside a longer string
`token_sort_ratio`	Same words in different order
`token_set_ratio`	Superset/subset token relationships

4.6.6. Semantic Similarity with Sentence Embeddings

All the measures above work on the surface form of text. Semantic similarity operates on the meaning by comparing embedding vectors from a pre-trained model.

from sentence_transformers import SentenceTransformer, util
# pip install sentence-transformers

model = SentenceTransformer("all-MiniLM-L6-v2")

sentences = [
    "The bank can guarantee deposits will eventually cover future tuition costs.",
    "An investment can guarantee future tuition costs will be covered by bank deposits.",
    "The dog played in the garden.",
]

emb = model.encode(sentences, convert_to_tensor=True)
scores = util.cos_sim(emb, emb)

for i in range(len(sentences)):
    for j in range(i + 1, len(sentences)):
        print(f"[{i},{j}] {scores[i][j]:.4f}")
# Sentences 0 and 1 are semantically very similar despite different word order;
# sentence 2 is unrelated.

Surface vs. Semantic Similarity

	Jaccard	Cosine (TF-IDF)	Cosine (SBERT)
“cat on mat” vs “feline on rug”	Low	Low	High
“bank deposit” vs “river bank”	High	Medium	Low

Surface measures are fast and domain-agnostic. Semantic measures require a model but capture paraphrases and synonymy.

4.6.7. BM25 — Ranking Similarity

BM25 (Best Match 25) is the industry-standard relevance ranking function used in search engines (Elasticsearch, Solr, Lucene). It extends TF-IDF with term saturation and document-length normalisation:

\[ \text{BM25}(d, q) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t,d) \cdot (k_1 + 1)}{f(t,d) + k_1 \cdot \left(1 - b + b \cdot \frac{|d|}{\text{avgdl}}\right)} \]

where \(k_1 \approx 1.2\)–\(2.0\) controls term-frequency saturation and \(b \approx 0.75\) controls length normalisation.

from rank_bm25 import BM25Okapi  # pip install rank-bm25

corpus = [
    "the cat sat on the mat",
    "a cat is sitting on the mat",
    "dogs are running in the park",
    "the park is full of dogs and cats",
]

tokenized = [doc.split() for doc in corpus]
bm25 = BM25Okapi(tokenized)

query = "cat mat"
scores = bm25.get_scores(query.split())
print(dict(zip(corpus, scores.round(3))))

4.6.8. Choosing the Right Measure

Task                              Recommended measure
─────────────────────────────────────────────────────────
Keyword / token overlap           Jaccard, Dice
Typo / OCR correction             Levenshtein, Jaro-Winkler
Fuzzy name matching               RapidFuzz token_sort_ratio
Document retrieval / ranking      BM25
Embedding-space similarity        Cosine (TF-IDF or Word2Vec)
Paraphrase / semantic search      Cosine (SBERT / sentence-transformers)

In practice, hybrid pipelines are common: BM25 for fast candidate retrieval followed by a SBERT re-ranker for final scoring.