Hands-On Python for Natural Language Processing 101 ? Part 1

Updated November 2026. This was first published on 24 May 2020. The idea is the same. Any number below is what the original piece said then. It is not a new count.

Key takeaways

In our previous posts, it took us just 4 tutorials to be able to write a basic supervised learning algorithm that was able to predict if a car driver was an Excellent/Good/Poor driver.
Much of the prediction was based on the analysis of numeric data.
The fields of data that were strings (alphabetical data) also got converted to numeric data.

A How-To Guide for Non-Programmers #

In our previous posts, it took us just 4 tutorials to be able to write a basic supervised learning algorithm that was able to predict if a car driver was an Excellent/Good/Poor driver. Much of the prediction was based on the analysis of numeric data. The fields of data that were strings (alphabetical data) also got converted to numeric data.

In this next series of posts, let’s examine how we can use Python and some of its pre-built libraries to understand text data. We can find Text data in many different places: on web pages (your own or your competitors), a repository of customer reviews, customer emails, and even transcriptions of your call center data. If you think Machine Learning for strings (text) is secondary, think again. In 2016, IBM estimated 80% of all business data to be text data.

Before we dive into an actual example, let’s understand the key concept of ‘vectorizatione.

Vectorization — How Machines View Words #

Computers may appear to read and understand words, as evidenced by the Google searches we do each day. the truth is far different from what we may think.

Let’s dispel a few myths before we start on Text Analysis by clarifying a few definitions and ground rules.

A. Word Vector #

They understand representations of words as a numeric value. These numeric values can be thought of as a Word Vector.

B. Word Vector values are specific to a Corpus #

Each word vector value is unique to the training data it operates on. This training data or corpus may differ from one context to another, e.g., a sports-related search where the word ‘sprint’ means something different from the word ‘sprint’ in the corporate world (the Phone company).

The big take away here is that the Word Vector values differ for the same word, when the word is used for different purposes. For synonyms, thought the values are close to each other within the same corpus. And naturally, if the overall context of the words is different, then the vector values represented by the same words should differ greatly.

C. Focus More on Co-Occurrence of words than the actual Word Itself #

Imagine of you had hundreds of thousands of customer emails to analyze for insight. For sure you can have your Customer Service agents classify the email on the type of emails coming in e.g. Password Reset, Tech Issues with App, Shipment Delay, Broken/Damaged product etc. but will you actually be able to much deeper than that.

The customer service representatives are able to listen to the caller and easily confirm that ‘password resete and ’locked oute are probably the same issue. Computers and programs need to see these phrases occurring closely and learn this by plotting the conversation as numeric values.

With the 3 concepts above established, let’s examine how Natural Language Processing (NLP) in Python can assist us in getting insights from a large body of text. We will need to work our way through to such an example.

Much like, how we started our Machine Learning journey, let’s start with a straightforward dataset. A single url on the history of the doughnut from the Smithsonian - https://www.smithsonianmag.com/history/the-history-of-the-doughnut-150405177/.

The goal of our exercise will be to find the 20 most relevant keywords on this webpage. The thinking here is to get a sense of what this page is about without actually reading it.

As we did before with our first post on Machine Learning, let’s follow the following steps.

Step 1 ? Setting Up the Environment #

BeautifulSoup & Urllib #

Python has three key libraries that will help us get started.

BeautifulSoup - parse the HTML of a page.

B. Urllib ? get the contents of any url.

Nltk — Pythones Natural Language Tool Kit that will assist us in the analysis of the words on the target page.

With the 3 lines of code above (not counting the comments), we have called the libraries we know of at this point in time that we will use. We can always call in additional libraries and modules at a later point in the program.

When you run this program for the first time you are likely to see this popup below.

Give the download some time as it is a large file.

Please also keep in mind that as Python executes the program in discrete steps working from the top to the bottom, this pop up will be triggered each time you run the program. But once you have downloaded the ‘collectione, you can hit X the next time and dismiss the pop-up and there is no need to download the package repeatedly.

Step 2 ? Loading the dataset #

Getting Python to read and parse the url #

to do so we use the urlib library to crawl and read our target page - https://www.smithsonianmag.com/history/the-history-of-the-doughnut-150405177/.

to do so we invoke the request and url open command in the following manner.

We name this variable as ‘pagee. You code now should look like this.

Now that Python know which url to read, we need to equip it with an object to hold the contents of the page and a parser with which we can exclude elements of the page we do not need. At this point if you run your program, you will not be able to view any output as it has not been stored into memory or any database.

to do that we name an object called ‘soupe and use a html5 parser.

The print statement at the end is a way to validate of the output of the program is what we expect it to be. Upon running this program, you should see the full html of the page be reproduced in the right-hand pane of your Spyder screen.

Once the print statement has confirmed the verification you are looking for, you can either delete it or comment it out.

We have confirmed that all the contents of the page that are visible to us when we use a view source command in our browser have been retrieved and are being held in an object named soup.

Step 3 ? Verifying the data #

In any project, creating and running the program/algorithm is the easiest part. The hard part is the data cleansing, standardization and preparation. Right now, we have the test we wish to isolate and the code all mixed up in a single entity.

At a high level, most HTML documents are coded in the following structure.

Of course you can have multiple div tags nested between each other, multiple scripts and links.

to provide our program to extract the text we are interested in, we need to specify the tags we wish to extract.

As an example, we wish to extract the h1 title text. to do so, we highlight the text and right click on our mouse choosing the ‘inspecte. The screenshot below is from the Chrome browser which now displays that the highlighted text in is the h1 tag.

Similarly, we identify, the text in the h2 tag and the 20 paragraphs of text on the page in the p tags.

To specify that this is the text we wish to extract, we write the following lines of code.

At this point your code should look like this.

You can use the print statement to verify if the correct h1, h2 and p tag text is being retrieved. In the case of the h2 tag as an example, you will note that some of the html is also being pulled in.

The same is true in the case of the p tags.

To get rid of all of these html characters, we create a single new list where we will import just the text from all the tags we have identified.

append the h1 to tags.txt #

append the h2 to tags.txt #

append the ptags to tags.txt #

When you run this program, you will see a list of words that includes the h1, h2 and the paragraph text.

And we can see the h1 text, h2 text and all the text from the paragraph in a single new file, new_list.

Now that we have all of our target text in a single list, we can break this single list into individual words. We do by writing this function, where def defines this new function being created.

return ([i for item in full_list for i in item.split()]).

To verify the output we run a print statement.

The right hand side of your Spyder pane should now show individual words confirming that the words have been tokenized (evidenced by the words being in single quotes as compared to the earlier screenshot above).

The Python NTLK contains some stop words. Stop words are commonly used word (such as ’thee, ‘ae, ‘ane, ‘ine). As you can see these words may make sense to use as part of our language but they would cloud any analysis we would run so its best to remove them.

We now create a new list named filtered_words where we add all words from word_tokens provided a match is not found in stop_words.

filtered_words = [w for w in word_tokens if not w in stop_words].

If you wish to view the kind of words that are being excluded, you can run the following code in a new pane.

Your new code should now look like as follows.

While eye-balling the filtered_words, let’s say you decided that you wanted to standardize all words to lower case so that we did not have any case of an upper case word being treated as the same as a lower case word.

Python has a handy function for that too.

lower_list = [x.lower() for x in filtered_words].

With the line of code above, we can create a new list named lower_list where x (meaning all characters) in filtered_words are converted to a lower case.

Step 4 - Running the Program #

We have now completed a fair amount of pre-processing. Let’s try and chart the top 20 most common words used on this page. to do so, we use the Frequency Distribution (fq) function.

print(fq) #

Upon running this program, you should see an output along the lines below.

When you navigate to the ‘plotse pane of your screen, you should see a chart that you can copy and paste.

As you can see, our pre-processing and data cleansing tasks are not complete yet.

We need to account for singular and plural uses of the word.

We also need to remove words like ‘ine, ’theye etc that donet really provide any insight.

We will cover that and more in the next post.

Also you can access the code written here on the following link https://colab.research.google.com/drive/12C3WMO4CR6cM5-LXJJ_TErtGIIsGTXeneusp=sharing.

Questions and answers #

What is the main point? #

In our previous posts, it took us just 4 tutorials to be able to write a basic supervised learning algorithm that was able to predict if a car driver was an Excellent/Good/Poor driver.

What else did the original piece say? #

Much of the prediction was based on the analysis of numeric data.