5.1. ELMo

Author

Kamil Filipek

5.1.1. Contextual Embeddings

From static embeddings we move to dynamic ones, called “contextual embeddings”. Contextual embeddings are word representations that change depending on the surrounding context, unlike static embeddings (Word2Vec, GloVe) where each word has a single fixed vector regardless of how it’s used.

The core insight is simple: the word “bank” should have different representations in “river bank” vs. “bank account.” Static embeddings can’t do this - they assign one vector to “bank.” Contextual embeddings solve this by running the entire sentence through a deep neural network and producing per-token representations that are informed by all the other tokens.

5.1.2. Historical Context

ELMo was introduced by Peters et al. (2018) at the Allen Institute for AI in the paper “Deep Contextualized Word Representations” (NAACL 2018). It was the first widely adopted model to produce word-level representations that are a function of the entire input sentence, rather than individual word lookup tables.

Before ELMo, the dominant paradigm was to use fixed pre-trained vectors (Word2Vec, GloVe) as input features to task-specific models. ELMo changed this by showing that deep language model representations — not just token embeddings — could be extracted and reused. The word “representation” now depended on the whole sentence context.

The key conceptual shift ELMo introduced:

  1. From lookup tables to language model activations — embeddings are computed by a neural network, not stored in a table.
  2. From single-layer to multi-layer — all intermediate representations are exposed, not just the final layer.
  3. From context-free to context-sensitive — the same word gets a different vector in different sentences.

This approach set the stage for the full pre-training paradigm that followed with BERT and GPT.

5.1.3. ELMo Architecture

ELMo (Embeddings from Language Models) uses a bidirectional LSTM. It trains two language models - one reading left-to-right, one right-to-left (bi-directional LSTM) - and concatenates their hidden states at each layer. The final embedding for a word is a learned weighted combination of all layer representations. This was a breakthrough because it showed that different layers capture different linguistic properties: lower layers tend to encode syntax, higher layers encode semantics.

Key characteristics are:

  • Context-aware: Word meaning changes with context.

  • Deep Representations: Uses multiple layers from the language model.

  • Pre-trained + Task-specific: ELMo embeddings are integrated into downstream models and fine-tuned accordingly.

The model uses two separate LSTMs:

  • The forward LSTM reads the sentence from left to right and predicts the next word.

  • The backward LSTM reads from right to left and predicts the previous word.

Formally, for a token at position \(k\) in a sequence of \(N\) tokens, ELMo produces a representation:

\[ \text{ELMo}_k^{task} = \gamma^{task} \sum_{j=0}^{L} s_j^{task} \mathbf{h}_{k,j}^{LM} \]

where \(\mathbf{h}_{k,j}^{LM}\) are the hidden-state vectors from layer \(j\), \(s_j^{task}\) are softmax-normalised weights learned per task, and \(\gamma^{task}\) is a global scale factor. This means the downstream task learns which layers are most useful rather than just using the final layer.

5.1.4. Performance and Benchmarks

ELMo produced large gains over the previous state of the art on every major NLP benchmark at the time of publication (Peters et al., 2018):

Task Metric Previous SOTA ELMo Improvement
SQuAD (reading comprehension) F1 81.1 85.8 +4.7
SRL (semantic role labelling) F1 81.7 84.6 +2.9
NER (named entity recognition) F1 91.93 92.22 +0.29
SST-5 (sentiment, 5 classes) Accuracy 53.7 54.7 +1.0
Coreference resolution F1 67.2 70.4 +3.2

These improvements were achieved simply by replacing fixed GloVe/Word2Vec embeddings with ELMo representations in the existing model architecture — no architecture changes required.

5.1.5. ELMo Usage in Python

If you have a version conflict on macOS:

# Uninstall conflicting packages
pip uninstall tensorflow tensorflow-hub tf-keras protobuf -y

# Install compatible versions together
pip install tensorflow==2.15.0 tensorflow-hub==0.15.0 protobuf==3.20.3

Simple Python implementation using TensorFlow Hub:

import tensorflow as tf
import tensorflow_hub as hub

elmo = hub.load("https://tfhub.dev/google/elmo/3")

def get_elmo_embedding(sentences):
    embeddings = elmo.signatures["default"](tf.constant(sentences))["elmo"]
    return embeddings

sentences = [
    "The bank will approve your loan.",
    "He sat by the bank of the river."
]

embeddings = get_elmo_embedding(sentences)
print(embeddings.shape)   # (2, max_seq_len, 1024)

Comparing context-dependent embeddings for the word “bank”:

import numpy as np

# Get token-level embeddings
result = elmo.signatures["default"](tf.constant(sentences))

# "bank" is token index 1 in first sentence, index 4 in second
bank_financial = result["elmo"][0, 1, :].numpy()
bank_river     = result["elmo"][1, 4, :].numpy()

cosine_sim = np.dot(bank_financial, bank_river) / (
    np.linalg.norm(bank_financial) * np.linalg.norm(bank_river)
)
print(f"Cosine similarity between two senses of 'bank': {cosine_sim:.3f}")
# Expected: ~0.6–0.8 — similar but not identical

5.1.6. Limitations and Legacy

Despite its impact, ELMo has several limitations that motivated subsequent models:

Limitation Description How BERT/GPT addressed it
Sequential computation BiLSTM processes tokens one by one — slow to train Transformers process all tokens in parallel via self-attention
Shallow bidirectionality The two LSTMs are trained independently BERT fuses both directions in every layer via masked language modelling
Long-sequence degradation LSTM hidden states degrade over very long sequences Transformers with relative position encodings handle longer contexts
Feature-based only ELMo produces features used as inputs; downstream model still needs task-specific design BERT/GPT can be fine-tuned end-to-end with a single head
Computational cost Requires a large BiLSTM at inference time DistilBERT and other lighter models achieve similar performance

ELMo’s true legacy is conceptual: it demonstrated that pre-trained language model representations transfer across tasks — a principle that underlies every state-of-the-art NLP system today. It directly inspired BERT, which replaced the BiLSTM with a Transformer encoder and achieved even larger gains.

Note

Key reference

Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep Contextualized Word Representations. Proceedings of NAACL-HLT 2018. https://arxiv.org/abs/1802.05365