Lab 8: Text Mining and NLP

Overview

In this lab, we will use tidy text techniques to analyze a dataset of amazon reviews. Each problem utilizes the tidy text mining techniques described in either chapter 2 (Problem 1), chapter 3 (Problem 2), or chapter 4 (Problem 3) of the tidy text mining with r textbook. Note: the dataset for this assignment is a bit bigger than what we have typically worked with in the class. On my computer everything worked fast enough, but if your computer is older and you find the computations intolerably slow you may reduce the size of the dataset by 90% by taking only the first 10% of reviews. If you do this make sure it is clearly stated. I have also listed a second shorter version of the file.

Problem 1: Sentiment and Review Score

  1. Download “simple_reviews.json” from the following google drive link: https://drive.google.com/drive/folders/1bk_2ihR5gQ8k6Tkn0NNpB58K1efruKnX?usp=sharing, read it into R, and rectangle it so that it is a dataframe where each row contains a single amazon review. If this file is too large, click here to download a shorter version where 90% of the reviews have been dropped].

  2. To make sentiment analysis possible, add an index variable to the review data frame so that each review is uniquely identified by an integer. Then tokenize the review data frame using words as the tokens, and remove all stop words from the data set.

  3. Does sentiment correlate with reviews? Use the afinn lexicon to calculate a sentiment score for each review, normalizing by the number of lexicon words in each review. Visualize the distribution of sentiment scores for each rating and calculate the mean sentiment score for each review category. What do you observe?

  4. Reviewer Personalities: For each reviewer, compute the number of ratings, the mean sentiment, and the mean review score. Filter for reviewers who have written more than 10 reviews, and plot the relationship between mean rating and mean sentiment. What do you observe?

Problem 2: Words with high relative frequency

  1. As your starting point, take the tokenized data frame that has been filtered to remove stop words, but hasn’t been joined with the sentiment lexicon data. For each item (asin), use the bind_tf_idf function to find the word that occurs in the reviews of that item with the highest frequency relative to the frequency of words in the entire review test dataset.

  2. Select five items from the dataset (either at random or by hand) and look up the asin code for those items on Amazon.com. In each of these cases, does the highest relative frequency word correspond to the identify or type of the item that you chose? You may not be able to find every single item, but I was able to find a solid majority of the ones I searched for by searching amazon.com for the asin.

Problem 3: Bigrams and Sentiment

  1. Consider the two negative words not and don't. Starting from the original dataset, tokenize the data into bigrams. Then calculate the frequency of bigrams that start with either not or don't. What are the 10 most common words occurring after not and after don't? What are their sentiment values according to the afinn lexicon?

  2. Pick the most commonly occurring bigram where the first word is not or don't and the afinn sentiment of the second word is 2 or greater. Compute the mean rating of the reviews containing this bigram. How do they compare the average review score over the entire dataset?