Overview

This lesson provides an introduction to text normalization.

Outcomes

define text normalization
explain how text normalization is used
present examples of text normalization

Prerequisites

Before starting this tutorial, ensure that …

you are familiar with tokens and types
you are familiar with parts of speech

Background

Though it isn't frequently discussed in the literature, data cleaning (ex. text normalization) and preprocessing is often the most time-consuming step in any analysis or machine learning pipeline.

What is text normalization?

Real-world, unstructured text data is messy. It's often full of typos and inconsistently formatted. If you're scraping data from a website, you may find yourself needing to sift through a soup of broken HTML and JavaScript. Text normalization is a set of techniques designed to increase the uniformity of text.

Why normalize text?

Text normalization is an essential preprocessing step in many natural language processing applications, such as…

data canonicalization¹
information retrieval (IR)[^ir]
query normalization (ex. expanding acronyms, correcting spelling errors, case folding, canonicalizing emoticons, etc.) [^ir]: You'll learn about IR in LING 531
text to speech systems (TTS)
"$3.50" → "three dollars and fifty cents"
machine learning (ML)
for example, if we're detecting spam, knowing that an email contains a URL could be an informative feature, but perhaps not the attached query strings. Similarly, the domain of a URL make for a good feature (e.g., "https://domain.org/stuff" → "domain.org").
enforcing consistent input for training and using statistical classifiers (ex. part of speech taggers, syntactic dependency parsers, etc.)
reducing the feature space
- in a bag of words classifier[^bow-classifier], each distinct word (token) is treated as a feature. Words that occur extremely infrequently[^hapax] overall are unlikely to be reliable features. Some normalization might help to collapse distinct categories.
[^bow-classifier]: You'll learn how to build a bag of words (BoW) classifier in LING 539 [^hapax]: The most extreme case being hapax legomenon
probability estimates
- this will come up in later units, but a lack of normalization can lead to underestimating probabilities. This can have a deterimental effect on probabilistic language models[^lang-model] and classifiers that use probabilities as features
[^lang-model]: Predicting which word is likely to follow
spelling correction
detecting and correcting typographical errors

Why not normalize text?

Some differences could be informative to a task. For example, use of punctuation (ex. !!!!) or the dialectal distinction between color and colour might be useful features in certain classification tasks such as authorship detection. Similarly, making the case uniform through case folding (all uppercase or all lowercase) might make it more difficult to distinguish between certain proper and non-proper nouns.

Original	Case-folded version
Please call about ABE²	please call about abe
Shinzo Abe called Trudeau	shinzo abe called trudeau

Techniques

Here we'll examine a few common techniques for text normalization.

Case folding

The process of make the case of all text uniform (ex. uppercase or lowercase).

Raw	case-folded version
I LiKe TuRtLeS	i like turtles
I LIKE TURTLES	i like turtles
I LIKE turtles	i like turtles

Replacement

It can be useful to standardize text by replacing elements of a shared type with some categorical label. For an example, see th table below:

Raw	Replacement URL
"https://parsertongue.org"	URL
"https://arizona.edu"	URL

The replacement can be at varying levels of specificity. For example, …

Raw	Replacement URL
"https://parsertongue.org"	parsertongue.org
"https://parsertongue.org/about"	parsertongue.org
"https://arizona.edu"	arizona.edu
"https://linguistics.arizona.edu/ma-native-american-languages-linguistics"	linguistics.arizona.edu

Lemmatization

Lemmatization is the assignment of a canonical³ to all members of a group sharing the same lexeme. For each token sharing a lexeme, replace the token with its lemma form (i.e., the form you'd see in the dictionary).

$\{fish, fishing, fishes\} \rightarrow fish$

$\{sneak, sneaking, snuck, sneaked\} \rightarrow sneak$ ⁴

$\{sneaker, sneakers\} \rightarrow sneaker$

$\{fast, faster, fastest\} \rightarrow fast$

If performed correctly, lemmatization should not change a word's coarse-grained grammatical category:

$\{studious\} \rightarrow studious$

$\{studied, studying\} \rightarrow study$

This means that lemmatization relies on correctly identifying the part of speech assigned to a token:

Word in context	Coarse POS category	Lemma
the carpenter's saw is shiny	NOUN	saw
I saw the boy with the telescope	VERB	see

Removal

It is not uncommon to remove certain text such as punctuation to make the data more uniform. If processing many documents from the same source, it may be useful to remove metadata. For example, if you were processing text from Project Gutenberg, you might want to remove or ignore content such as the table of contents and chapter headings.

Frequency-based thresholding

Depending on how you plan to use the text (ex. training classifiers), you may wish to outright discard tokens below some frequency or replace them with some common symbol. For example, you may want to replace all singularly occurring tokens with the symbol UNK ("unknown"). This will be covered in LING 539.

Stemming

Stemming is a string transformation process that aims to reduce a word down to some base form. Unlike lemmatization, stemming may not return real words.

Typically, stemming is performed using a series of string transformation rules to iteratively whittle a token down to some simplified representation:

enthusiastically
ally $\rightarrow$ al
enthusiastical
al $\rightarrow$ $\epsilon$
enthusiastic iastic $\rightarrow$ istic
enthusistic
istic $\rightarrow$ ist
enthusist
ist $\rightarrow$ $\epsilon$
enthus

There are two types of errors assocated with stemming: overstemming and understemming.

Overstemming is when two words with distinct meanings are reduced to the same stem:

Original	over-stemmed form
universal	univers
university	univers

Understemming is when two words with highly related meanings are not reduced to the same stem:

Original	under-stemmed form
alumna	alumna
alumni	alumni
alumnus	alumnus

You'll learn more about stemming algorithms in LING 538.

Next steps

You've learned the motivation behind text normalization as well as places where it may cause issues. It's time to apply what you've learned…

Practice

Imagine you are working with data from Twitter. Describe some text normalization/cleanup procedures you might use to preprocess the data.
Imagine you are working with emails. Describe some text normalization/cleanup procedures you might use to preprocess the data.

Footnotes

i.e., ensuring the same format is used for all data belonging to some category
- email addresses (ex. "username AT domain DOT org" vs. "username@domain.org")
- phone numbers (ex. "1.111.111.1111" vs. "1 (111) 111-1111")
- etc.
↩
As in Application for Benefits Eligibility ↩
the form you'd see in the dictionary ↩
"Wait! Is snuck really a word?" If people use it, it's a word! 😄 ↩