Tokenization 101 with R (Tidyverse)

4 min read · Feb 19, 2023

Tokenization is the process of breaking down text into individual units called tokens, which are typically words, but can also be phrases, subwords, or other linguistic units. Tokenization is a fundamental step in natural language processing (NLP) and is used in many NLP tasks, such as text classification, sentiment analysis, and machine translation. In R, there are several packages and functions within the tidyverse that can be used for tokenization, including stringr(), and tidytext.

1. Using stringr() from tidyverse:

The stringr() function from tidyverse provides a range of functions for manipulating strings, including tokenization. The str_split() function can be used to split a string into tokens based on a specified delimiter.

Example:

library(stringr)

text <- "This is a sentence."
tokens <- str_split(text, " ")
print(tokens)

Output:

[[1]]
[1] "This"     "is"       "a"        "sentence."

In this example, the str_split() function takes the text string and splits it into multiple pieces based on the space delimiter. The result is a list where each element contains the tokens from the original string.

str_split() can also be used with regular expressions to split a string based on more complex patterns. For example, to split a string into sentences, you could use the regular expression "[.!?]+", which matches one or more periods, exclamation marks, or question marks.

library(stringr)

text <- "This is a sentence. This is another sentence! And this is a third sentence?"
sentences <- str_split(text, "[.!?]+")
print(sentences)

Output:

[[1]]
[1] "This is a sentence"           " This is another sentence"   
[3] " And this is a third sentence"

In this example, the str_split() function uses the regular expression "[.!?]+" to split the text string into multiple sentences. The result is a list where each element contains a sentence from the original string.

2. Using tidytext:

The tidytext package is a package for text mining using tidy data principles. It provides a range of functions for text processing, including tokenization. The unnest_tokens() function can be used to split a string into tokens based on a specified delimiter.

Example:

library(tidytext)

text <- data.frame(text = "This is a sentence.")
tokens <- unnest_tokens(text, word, text)
print(tokens)

Output:

# A tibble: 4 x 1
  word     
  <chr>    
1 This     
2 is       
3 a        
4 sentence.

In tidytext, tokenization is typically done using the unnest_tokens() function. This function takes a data frame and a column containing text data as input and creates a new data frame where each row corresponds to a single token. The tokens are typically stored in a column called word.

Here’s another example of how to use unnest_tokens() for tokenization:

library(tidyverse)
library(tidytext)

# Define some text data
text_data <- tibble(
  id = c(1, 2, 3),
  text = c("This is some text.",
           "Here is another sentence.",
           "And yet another one.")
)

# Tokenize the text data
tokens <- text_data %>%
  unnest_tokens(word, text)

# Print the resulting tokens
print(tokens)

Output:

# A tibble: 11 x 2
      id word    
   <dbl> <chr>   
 1     1 this    
 2     1 is      
 3     1 some    
 4     1 text    
 5     2 here    
 6     2 is      
 7     2 another 
 8     2 sentence
 9     3 and     
10     3 yet     
11     3 another 

In this example, we first defined a data frame called text_data that contains three rows of text data. We then used the unnest_tokens() function to tokenize the text data, resulting in a new data frame called tokens that contains each token as a separate row, along with an ID that indicates which original row it came from.

By default, unnest_tokens() tokenizes text using whitespace as the delimiter. However, it’s possible to specify a different delimiter by passing it as a second argument to the function. For example, to tokenize text using commas as the delimiter, we could use the following code:

tokens <- text_data %>%
  unnest_tokens(word, text, token = "regex", pattern = ", ")

In this case, we used the token = "regex" argument to specify that we want to use a regular expression to split the text into tokens. We then used the pattern = ", " argument to specify that we want to split the text at each comma followed by a space.

In summary, tidytext is a powerful tool for tokenization in R. The unnest_tokens() function makes it easy to tokenize text data and prepare it for further analysis using other tools in the tidyverse.

Conclusion

tidytext provides a simple and flexible way to tokenize text data in R using the principles of the tidyverse. By leveraging the unnest_tokens() function, it is possible to quickly and easily split text into individual tokens and prepare it for further analysis using other tidyverse tools.

Using the tidyverse collection of packages, including dplyr and ggplot2, it is possible to perform a wide range of text analysis tasks, such as sentiment analysis, topic modeling, and text classification. With tidytext as a part of the tidyverse, R users have a powerful set of tools at their disposal for exploring and analyzing text data.

By incorporating tokenization into a larger text analysis workflow, R users can gain deeper insights into the patterns and relationships within their data, and ultimately make more informed decisions based on those insights. Overall, the tidytext package and other tidyverse functions for text analysis provide a comprehensive and user-friendly framework for working with text data in R.