Overview

In this tutorial, we're going to step through training a naïve Bayes classifier with $n$ -gram features using a popular machine learning library in Python called scikit-learn. This library provides implementations and an easy-to-use interface for many supervised and unsupervised¹ classifiers, as well as utilities for …

data preprocessing
feature extraction
feature selection
dimensionality reduction
classifier evaluation

… among other things!

Outcomes

After completing this lesson, you'll be able to …

use the CountVectorizer to generate $n$ -grams counts for documents
use the LabelEncoder to string representations of class labels to integers
train a naïve Bayes classifier and use it to make predictions

Prerequisites

Before starting this tutorial, you should be comfortable with …

You can follow along and run these snippets interactively in the IPython REPL using following docker image:

docker run -it "uazhlt/sklearn-demo:latest" ipython

Alternatively, to launch a jupyter notebook server on port 7777, run the following command:

docker run -it \
  -p 7777:9999 \
  -v "${PWD}/notebooks:/app/notebooks" \
  uazhlt/sklearn-demo:latest

First, let's import some things that we'll be using throughout this lesson…

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.preprocessing import LabelEncoder
from sklearn.naive_bayes import MultinomialNB
import typing
import re

Getting started

We're going to step through training a naïve Bayes classifier using a toy dataset with $n$ -gram features.

First, we'll need some documents …

docs_train = [
  "LOL. I love this site http://scam.com/enter/to/win.",
  "My mother loves the lotto",
  "Dear Sir or Madam, you have been named in a will to receive a sum of $2000000",
  "Will you please stop using that site",
  "hot and lonely tortoises in your neighborhood XXX",
  "It's lonely being an outlier",
  "Announcing the WaffleCOIN ICO",
  "Important message from the IRS.  Send your bank account number and SSN immediately to irs-staff@scam.com"
]
 
docs_test = [
  "OMG. Have you seen this site http://scam.com/p0wnd",
  "Dear Sir or Madam, I am contacting you about an urgent financial matter. I have taken possession of an abandoned account with a large sum of money ($1,000,000)",
  "Did you remember to buy lettuce for the tortoise?",
  "UPCOMING ICO! Get ready for MuffinCOIN",
]

… and some labels for those documents

y_train = [
  "SPAM",
  "HAM",
  "SPAM",
  "HAM",
  "SPAM",
  "HAM",
  "SPAM",
  "SPAM"
]
 
y_test = [
  "SPAM",
  "SPAM",
  "HAM",
  "SPAM"
]

Generating and counting $n$ -ngrams

Now that we have a set of documents reserved for training purposes, let's look at how we can use scikit-learn's CountVectorizer class to generate feature vectors of word or character $n$ -gram counts.

First, we need to create an instance of the CountVectorizer class that is configured for our use case.

Let's look at a subset of parameters we can use to customize the behavior of CountVectorizer …

vectorizer = CountVectorizer(
  # case fold all text 
  # before generating n-grams
  lowercase=True,
  # optionally apply the specified function
  # before counting n-grams
  preprocessor=None,
  # optionally provide a list of tokens to remove/ignore before generating n-grams
  stop_words=None,
  # specify a range of n-grams as (min_n, max_n). 
  # (1, 1) means unigrams.
  # (1, 2) means unigrams and bigrams
  # (4, 5) means 4-grams and 5-grams
  ngram_range=(1, 1),
  # "word", "char" (character), or "char_wb" n-grams
  analyzer="word",
  # whether or not to use binary counts
  binary=False
)

The IPython REPL makes it easy to review the documentation for a class or function by typing the class or function name followed by ??. To learn more about how to configure and use CountVectorizer, type CountVectorizer?? in the IPython terminal to view its docstring. Use the arrow keys to scroll forward and backward through the documentation. Type q to quit or exit the documentation screen.

Now that we've created an instance of CountVectorizer, let's see how it's used to turn text into count-based feature vectors.

Determine the feature vocabulary

We can determine our feature vocabulary using the .fit() method. In other words, we'll provide a set of documents (our training data) to come up with a set of $n$ -gram features (the columns in our feature vectors). This will allow us to transform unseen data to a familiar form that we can feed into a classifier or use to compute similarity to other text data we transform using the same process (we must maintain the column count and order!).

Let's use the toy training data we defined earlier to derive our feature vocabulary:

vectorizer.fit(docs_train)

.fit() assigns each unique feature an ID that corresponds to its position in any feature vector we later generate using the .transform() method of our vectorizer.

Once we've fit our CountVectorizer, we can access a vocabulary mapping (item $\rightarrow$ index):

vectorizer.vocabulary_

These (key, value) mappings tell us the column index (value) of each feature (key)in every feature vector we generate using this vectorizer.

Now that we've determined our vocabulary, let's apply our vectorizer to some data using .transform().

vectorizer.transform(
    [
        "It's going to cost you $23,030.12 or more.", 
        "Pay here: http://super-sketchy-site.info"
    ]
).todense()

Notice that the matrix that is returned is very sparse. Most of the values are zero. Only a few are non-zero. For a very large feature space, this is an inefficient way to store this information. We really only need to keep track of the dimensions of the matrix and the values of any non-zero entries. By default, scikit-learn returns just such a representation. the .todense() call above coverted the low-memory sparse matrix format to an unabreviated format that includes all the zeros. Since most of our feature vectors are likely to be very sparse, you'll typically want to work with the sparse matrix format.

Values to features

For ease of inspection, let's create a reverse mapping from (index $\rightarrow$ item).

i2v = dict((i, v) for (v, i) in vectorizer.vocabulary_.items())

What is the feature in the first position (index = 0)?

# what is the feature in the first position (index=0)?
i2v[0]

What is the feature in the sixth position (index = 5)?

# what is the feature in the sixth position (index = 5)?
i2v[5]

Alternatively, we can transform some data and then map it back to feature names (note the use of .todense() here) to guarantee that the first element in each array corresponds to the index = 0 feature in our vocabulary.

vectorizer.inverse_transform(
    vectorizer.transform(["It's going to cost you $23,030.12 or more."]).todense()
)

We've configured our vectorizer to use unigram features. How many features do have in our vocabulary?

len(vectorizer.vocabulary_)

Encoding labels

Now that we have a utility to generate our feature vectors, let's prepare our labels for training.

lbl_encoder = LabelEncoder()
# map distinct strings to integers
lbl_encoder.fit(y_train)
 
# prints  array(['HAM', 'SPAM'], dtype='<U4')
lbl_encoder.classes_

Like the vectorizer, we must fit our label encoder to our data in order to establish a consistent string $\rightarrow$ int mapping. Once we've fit the label encoder, we can access an array of distinct class labels using the .classes_ attribute. The index of each label corresponds to how that label will be represented as an integer.

Let's try transforming some string representations of labels to integer form…

# prints array([0])
lbl_encoder.transform(["HAM"])

Classifying documents

Training

Now that we can transform documents into feature vectors and represent class labels as vector of integers, let's train a classifier. Here we'll use a naïve Bayes classifier, but the API we'll use to train and predict is shared by all classifiers defined in scikit-learn.

clf = MultinomialNB(
  # should we learn p(y) from our data?
  fit_prior=True
)
 
# train the classifier by fitting it to some X and y
clf.fit(vectorizer.transform(docs_train), lbl_encoder.transform(y_train))
 
# prints array([0, 1])
clf.classes_

Predicting

Now that we've trained our classifier, we can use it to make predictions:

yhats       = clf.predict(vectorizer.transform(docs_test))
# convert ints to strings for readability
predictions = lbl_encoder.inverse_transform(yhats)
# let's compare our predictions to our true labels...
for pred, gold in zip(predictions, y_test):
  print(f"PRED: {pred} \tGOLD: {gold}")

Of course, this is just a toy dataset without any form of preprocessing or normalization being applied.

Next steps

You now have a sense of how to use scikit-learn to extract $n$ -gram features and train a very simple classifier. In future lessons, we'll look at ways of improving this pipeline. Before moving on, though, let's practice …

Practice

What values for $n$ are we using with the our CountVectorizer instance?
Is "pay" represented in the matrix that is returned by .transform()? Why or why not?
Why shouldn't you use .fit() or .fit_transform() at prediction time?

Unknown features

What happens if we pass a datum composed soley of unseen/unknown features?

vectorizer.transform(["ZAMBORTANI DIEMPO"]).todense()

vectorizer.transform(["Kltpzyxm"]).todense()

vectorizer.transform(["$20.00"]).todense()

What happens when we pass the .transform() method of our label encoder instance an unseen label after having called .fit()?
Why is it that calling .transform() on either ZAMBORTANI DIEMPO or Kltpzyxm results in vectors of the same length?
Why are all values in both ZAMBORTANI DIEMPO and Kltpzyxm vectors 0?

Footnotes

clustering methods ↩