WebText

WebText is a collection of web documents assembled for training OpenAI’s GPT-2 language model. Researchers selected outbound links from Reddit posts that had received at least three karma points. This provided a way to choose pages that people had already shared and endorsed, rather than collecting an unrestricted sample of every website. The training material came from the linked pages. [GPT-2 research paper, Section 2.1](https://cdn.OpenAI.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf).

The researchers cleaned and deduplicated that material before training. Their preliminary collection contained more than eight million documents. They removed Wikipedia pages to reduce overlap with evaluation datasets commonly used to test language models. WebText’s identity rests on this selection and preparation method: it is a particular training collection assembled from online writing, with choices about which sources and documents remain.