Week 11 Info: Tidy Text and Natural Language Processing
Hello Class,
Week 11 has begun. In week 11, we will build on our previous module on strings and learn how to work with text data. Text data comes in many different forms, from books and news articles to technical manuals, legal documents, or product reviews. This data lacks the rectangular structure that many other datasets naturally posses, and requires processing in order to generate features that enable us to run analyses or generate insights.
We will read the first 4 chapters of the online book text mining with R: a tidy approach, which cover how to wrangle textual data and convert it into a tidy format, how to work with lexicons to peform basic sentiment analysis, how to interpret the frequency of words occuring in text, and how to make use of word combinations (n-grams) and to compute correlations. The primary tool will be a tidyverse package called tidytext which has built in functions to tokenize text data into a tidy format and to analyze the resulting dataset.
This week we will have a lab assignment which involves extracting data stored in a hierarchical format, and combining webscraping tools with functions and iteration to automatically download many files at once. Find Lab 8 here. Note: I have added an alternative, shorter data file for those whoser personal computers struggle with the original one short_reviews.json
For your reading assignments and more details about this week, head over to Module 11..