Natural Language Processing (NLP)

11.1 Introduction

Natural Language Processing (NLP) teaches computers to work with human language such as text and speech.

NLP is used for emails, search, translation, voice assistants, captions, sentiment analysis, document processing and many other applications. A collection of text is called a corpus; the plural is corpora.

11.2 TextBlob

TextBlob is an object-oriented NLP library built on NLTK and pattern. It provides easier interfaces for many common NLP tasks.

The chapter uses TextBlob together with NLTK and several other NLP libraries. Its examples use Shakespeare's Romeo and Juliet text.

Installation

conda install -c conda-forge textblob
ipython -m textblob.download_corpora

11.2.1 Create a TextBlob

A TextBlob is the fundamental object used for many TextBlob NLP operations.

from textblob import TextBlob

text = 'Today is a beautiful day. Tomorrow looks like bad weather.'
blob = TextBlob(text)

blob

TextBlob objects, Sentences and Words support useful string operations as well as NLP methods.

11.2.2 Tokenizing Text into Sentences and Words

Tokenization means breaking text into smaller useful pieces.

blob.sentences
blob.words

The sentences property gives Sentence objects. The words property gives a WordList containing the words, with punctuation separated out.

11.2.3 Parts-of-Speech Tagging

Parts-of-speech (POS) tagging determines how each word is being used in its context.

blob.tags

POS tagging is useful because a word can have multiple meanings and grammatical roles depending on context.

11.2.4 Extracting Noun Phrases

A noun phrase is a group of words that represents a noun idea, such as water ski or beautiful day.

blob.noun_phrases

Noun phrase extraction can help applications understand important groups of words, for example when processing search queries.

11.2.5 Sentiment Analysis with TextBlob's Default Sentiment Analyzer

Sentiment analysis estimates whether text expresses positive, neutral or negative sentiment. TextBlob's default analyzer returns polarity and subjectivity.

blob.sentiment
blob.sentiment.polarity
blob.sentiment.subjectivity

The chapter demonstrates why simply looking for words such as "good" and "bad" is not enough. For example, "The food is not good" is negative even though it contains "good".

for sentence in blob.sentences:
    print(sentence.sentiment)

Sentiment can be analyzed for the whole TextBlob or separately for each sentence.

11.2.6 Sentiment Analysis with the NaiveBayesAnalyzer

TextBlob also provides NaiveBayesAnalyzer, a sentiment analyzer trained using movie reviews. It uses the Naive Bayes text-classification algorithm.

from textblob.sentiments import NaiveBayesAnalyzer

blob = TextBlob(
    text,
    analyzer=NaiveBayesAnalyzer()
)

blob.sentiment

The output format differs from the default analyzer: instead of polarity and subjectivity, it reports a positive/negative classification and corresponding values.

11.2.7 Language Detection and Translation

TextBlob can use Google Translate support to detect a language and translate TextBlobs, Sentences and Words. These operations require an Internet connection in the source chapter.

blob.detect_language()

spanish = blob.translate(to='es')
spanish

spanish.detect_language()

Language codes such as en, es and zh identify languages. You can also specify from_lang when the source language is known.

11.2.8 Inflection: Pluralization and Singularization

Inflection means different forms of a word, such as singular/plural forms and different verb forms.

from textblob import Word

index = Word('index')
index.pluralize()

cacti = Word('cacti')
cacti.singularize()

animals = TextBlob('dog cat fish bird').words
animals.pluralize()

Notice that pluralization and singularization are not always as simple as adding or removing s.

11.2.9 Spell Checking and Correction

Text analysis can be affected by spelling mistakes. TextBlob's spellcheck() returns possible corrections and confidence values.

from textblob import Word

word = Word('theyr')
word.spellcheck()
word.correct()

correct() chooses the highest-confidence correction. The highest-confidence choice is not guaranteed to be correct for every context.

sentence = TextBlob('Ths sentense has missplled wrds.')
sentence.correct()

11.2.10 Normalization: Stemming and Lemmatization

Normalization prepares words for analysis. One common goal is to treat related forms as related words.

from textblob import Word

word = Word('varieties')
word.stem()
word.lemmatize()

For example, an application may want program, programs, programmer, programming and programmed to be treated as related forms.

11.2.11 Word Frequencies

Word frequencies tell us how often words occur in a corpus. TextBlob provides a word_counts dictionary.

from pathlib import Path
from textblob import TextBlob

blob = TextBlob(Path('RomeoAndJuliet.txt').read_text())

blob.word_counts['romeo']
blob.word_counts['juliet']
blob.words.count('joy')

Frequency information is useful in document analysis, search and similarity techniques.

11.2.12 Getting Definitions, Synonyms and Antonyms from WordNet

WordNet is a lexical database created at Princeton University. TextBlob uses NLTK's WordNet interface to access definitions, synonyms and antonyms.

from textblob import Word

happy = Word('happy')
happy.definitions
happy.synsets

A synset represents a set of synonyms. Each synset can provide Lemma objects, whose names are synonymous words.

synonyms = set()

for synset in happy.synsets:
    for lemma in synset.lemmas():
        synonyms.add(lemma.name())

synonyms

Antonyms can be obtained from a Lemma using antonyms(). For example, a WordNet antonym for happy is unhappy.

11.2.13 Deleting Stop Words

Stop words are very common words such as the, a, is and you. They are often removed before analysis because they may provide little useful information.

import nltk
nltk.download('stopwords')

from nltk.corpus import stopwords
stops = stopwords.words('english')

blob = TextBlob('Today is a beautiful day.')
[word for word in blob.words if word not in stops]

NLTK provides stop-word lists for several languages.

11.2.14 n-grams

An n-gram is a sequence of n text items. When the items are words, an n-gram is a consecutive group of words.

blob = TextBlob(
    'Today is a beautiful day. Tomorrow looks like bad weather.'
)

blob.ngrams()
blob.ngrams(n=5)

n-grams can help with word prediction, speech-to-text and finding words that frequently occur next to each other.

11.3 Visualizing Word Frequencies with Bar Charts and Word Clouds

Visualizations make word-frequency analysis easier to understand. The chapter uses two approaches:

A bar chart displays frequencies as bar heights. A word cloud displays frequent words in larger fonts.

11.3.1 Visualizing Word Frequencies with Pandas

The chapter processes Romeo and Juliet, removes stop words, sorts words by frequency and puts the top 20 into a pandas DataFrame.

from pathlib import Path
from textblob import TextBlob
from nltk.corpus import stopwords
from operator import itemgetter
import pandas as pd

blob = TextBlob(Path('RomeoAndJuliet.txt').read_text())
stop_words = stopwords.words('english')

items = blob.word_counts.items()
items = [item for item in items if item[0] not in stop_words]

sorted_items = sorted(
    items, key=itemgetter(1), reverse=True
)

top20 = sorted_items[1:21]
df = pd.DataFrame(top20, columns=['word', 'count'])

axes = df.plot.bar(x='word', y='count', legend=False)

The source example's top words include romeo, thou, juliet, thy, capulet and nurse. plt.gcf().tight_layout() can improve the chart layout when labels are truncated.

11.3.2 Visualizing Word Frequencies with Word Clouds

The wordcloud library can generate a visual word cloud. More frequent words appear larger.

conda install -c conda-forge wordcloud

from pathlib import Path
import imageio
from wordcloud import WordCloud

text = Path('RomeoAndJuliet.txt').read_text()
mask_image = imageio.imread('mask_heart.png')

wordcloud = WordCloud(
    mask=mask_image,
    background_color='white'
)

wordcloud = wordcloud.generate(text)
wordcloud.to_file('RomeoAndJulietHeart.png')

A mask image controls the shape. The source chapter uses a heart-shaped mask. generate() calculates word frequencies and removes stop words using the wordcloud library's built-in list. fit_words() can be used when you already have a dictionary of word counts.

11.4 Readability Assessment with Textatistic

Readability describes how easy text is to understand. It can depend on vocabulary, sentence length, sentence structure and topic.

The Textatistic library calculates several well-known readability measures.

pip install textatistic

from pathlib import Path
from textatistic import Textatistic

text = Path('RomeoAndJuliet.txt').read_text()
readability = Textatistic(text)

readability.dict()

The chapter reports these statistics for the processed Romeo and Juliet text and explains that different formulas map text complexity to different kinds of readability information.

11.5 Named Entity Recognition with spaCy

Named Entity Recognition (NER) locates and categorizes important entities such as dates, people, organizations, places and quantities.

conda install -c conda-forge spacy
python -m spacy download en

import spacy

nlp = spacy.load('en')

document = nlp(
    'In 1994, Tim Berners-Lee founded the '
    'World Wide Web Consortium (W3C), devoted to '
    'developing web technologies'
)

for entity in document.ents:
    print(f'{entity.text}: {entity.label_}')

The Doc object's ents property contains the recognized entities. Each entity has useful properties such as text and label_.

11.6 Similarity Detection with spaCy

Similarity detection compares documents to estimate how alike they are.

import spacy
from pathlib import Path

nlp = spacy.load('en')

document1 = nlp(Path('RomeoAndJuliet.txt').read_text())
document2 = nlp(Path('EdwardTheSecond.txt').read_text())

document1.similarity(document2)

The value is between 0.0 (not similar) and 1.0 (identical). The chapter also compares a current news story with Romeo and Juliet, producing a much lower similarity.

11.7 Other NLP Libraries and Tools

The chapter encourages exploring multiple NLP tools because different tasks may be better suited to different libraries.

11.8 Machine Learning and Deep Learning Natural Language Applications

Many advanced NLP applications use machine learning or deep learning. The chapter previews applications that will be explored further later in the book.

11.9 Natural Language Datasets

NLP requires text data. The chapter lists many sources that can be used for learning and experiments.

• Wikipedia datasets
• IMDB movie and TV datasets
• UCI text datasets
• Project Gutenberg e-books
• Jeopardy! dataset
• NLTK data
• Sentiment-labeled sentence datasets
• Registry of Open Data on AWS
• Amazon Customer Reviews Dataset
• Pitt.edu corpora

11.10 Wrap-Up

This chapter introduced a broad range of NLP tasks. You learned how TextBlob can create text objects, tokenize sentences and words, perform POS tagging, extract noun phrases and analyze sentiment.

You also learned language detection and translation, pluralization and singularization, spelling correction, stemming, lemmatization, word frequencies, WordNet definitions/synonyms/antonyms, stop-word removal and n-grams.

Then you visualized word frequencies with pandas bar charts and word clouds, assessed readability with Textatistic, and used spaCy for named entity recognition and document similarity.