IMDB Reviews

The Large Movie Review Dataset (aclImdb v1.0): 50,000 polar movie reviews from IMDB, split evenly for training and testing. For over a decade the default benchmark for binary sentiment classification — simple enough to load with one function call, large enough that word order and negation actually matter.

Provenance

Collected at Stanford and released alongside Maas, Daly, Pham, Huang, Ng and Potts, Learning Word Vectors for Sentiment Analysis, ACL 2011. The dataset page (and download) lives at ai.stanford.edu/~amaas/data/sentiment. The original motivation was learning sentiment-aware word representations; the review corpus it shipped with promptly outlived the method.

Schema

  • instances: 50,000 labelled reviews — 25,000 train and 25,000 test, each perfectly balanced (12,500 positive / 12,500 negative) — plus a further 50,000 unlabelled reviews for unsupervised or semi-supervised use.
  • labels: derived from star ratings — a review scoring at most 4/10 is negative, at least 7/10 is positive. Neutral reviews (5-6) are excluded entirely, so the task is deliberately polar.
  • anti-leakage design: at most 30 reviews per movie (same-movie ratings correlate), and the train and test sets draw from disjoint movie sets, so memorising movie-specific vocabulary buys nothing.
  • format: raw .txt files in train/pos, train/neg, test/pos, test/neg, train/unsup; filenames encode the original star rating.

Benchmarks

Honest reference points for test accuracy:

model familyaccuracy
bag-of-words + naive bayes / logistic regression~83-88%
Maas et al. 2011 (word vectors + bow)88.9%
NB-SVM with bigrams (Wang and Manning 2012)91.2%
pretrained language models, fine-tuned (ULMFiT, BERT)~95%
leaderboard-topping transformers (e.g. XLNet)~96%

Two cautions. First, the ceiling is soft: a few percent of labels are arguably wrong or genuinely ambiguous, so the mid-90s is near saturation and decimal-point comparisons are noise. Second, the test set has been public since 2011 and the community has collectively tuned against it for years — treat published SOTA as optimistic. The enduring lesson from the linear-model era is that a well-smoothed bigram baseline is embarrassingly hard to beat; always run one before reaching for a transformer.

On this site

  • The tagged model is BERT: fine-tuning a pretrained encoder on IMDB is the canonical first transfer-learning exercise in NLP — see the transformers section for the architecture.

Loading

The archive is an 80 MB download that unpacks to one text file per review, so it is not bundled with sklearn — but load_files reads the unpacked tree directly:

from sklearn.datasets import load_files

# after: curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
#        tar xzf aclImdb_v1.tar.gz
train = load_files("aclImdb/train", categories=["pos", "neg"], encoding="utf-8")
test = load_files("aclImdb/test", categories=["pos", "neg"], encoding="utf-8")
print(len(train.data), len(test.data))
import numpy as np
print(np.bincount(train.target), np.bincount(test.target))  # perfectly balanced
print(train.data[0][:80])

(keras.datasets.imdb ships the corpus pre-tokenised as integer sequences; huggingface datasets.load_dataset("imdb") ships the raw text — both skip the tarball.)