Lab3: Exploratory Data Analysis

Overview

This is a two part lab where each part will focus on a different dataset: the first part will use a dataset containing a series of diagnostic measurements taken on members of the Akimel O’odham people (an indigenous group living in the Southwestern United States who are also called the Pima) to understand diabetes risk (click here to download diabetes.csv), and the second dataset contains information on traffic accidents in New York City in the months of July and August of this year, and was compiled by NYC Open Data (click here to download crashes.csv).

For this problem set you will need to install the skimr and GGally packages, and in particular the functions skim and ggpairs.

We will also explore the concept of an inlier, which is an erroneous value that occurs in the interior of the distribution of a variable, rather than in the tails of the variable. The US Census published an article on the problem of inliers here

Part 1: Health Diagnostics and Diabetes Incidence

Problem 1: Data Description and Outliers.

Load diabetes.csv into R and take a look at the data using the skimr package (make sure to install it if you don’t have it). Skimr provides a tidy summary function called skim. Use skim on the data frame that you loaded from diabetes.csv.

Skim will list several variables. Pregnancies is the past number of pregnancies (this dataset includes women 21 years or older), glucose describes the concentration of glucose in the blood after an oral glucose tolerance test (drinking a sugary drink and measuring two hours later), skin thickness is the result of a skinfold thickness test taken at the triceps (upper arm), Insulin is the insulin concentration in the blood taken at the same time as the glucose measurement (Insulin is a hormone that transports glucose into cells), BMI is “Body Mass Index”, Diabetes Pedigree Function is a measure of diabetes risk based on the family history of diabetes for each patient (this is an engineered feature) and outcome is equal to 1 if the patient was diagnosed with diabetes with 5 years and 0 otherwise.

  1. Skim should show no missing data, but should indicate potential data issues. Do any of the percentile ranges (p0, p25, p50, p75, or p100) for the reported variables suggest a potential problem?

  2. Further investigate the dataset to find potentially problematic variables using a qq-plot (geom_qq) or group_by combined with count and arrange. For which variables do you find repeated values and what are those values? Do you believe these values represent real measurements or could they correspond to missing data? Do the repeated variables occur in the same rows or different rows?

Write an overview of which values are missing and replace all missing values with NA for the next stage of analysis.

  1. Perform Tukey Box plots on each variable to identify potential outliers. Which variables have the most outliers? Are there any outliers that you think come from measurement error? If so remove them.

Problem 2: Pair Plots

  • Use the GGally package and its function ggpair on both the original dataset and the cleaned dataset. Which correlations change the most? What are the strongest correlations between variables overall and with the Outcome?

  • LLM Prompting Exercise: Pair plots can often format poorly. Look at your pair plot and identify any deficiencies, which could be issues with legibility of variable names, axis labels, or the visibility of the visual elements itself. Try to see if you can improve some of these issues using an LLM of your choice. Report which LLM you used and what prompt you gave in addition to showing the changes to the figure.

  • Remark: This dataset has been used as a model dataset for the construction of binary classifiers using machine learning and there are a large number of published studies showing these analyses. However, many of these analyses did not exclude the missing values erroneously coded as zero, as is discussed in this interesting paper by Breault, leading to highly degraded accuracy.

Part 2: Car Crashes in NYC

Problem 3: Finding Inliers and Missing Data

Load the NYC car crash dataset using read_csv. You can download the data from the course website by clicking here.

  1. Which variables have missing data (use skim or another tool of your choosing)? Some missing values have a different interpretation than others- what does it mean when VEHICLE TYPE CODE 2 is missing compared to LATITUDE?

  2. Latitude and Longitude have the same number of missing values. Verify that they always occur in the same row. Check the counts of latitude and longitude values- do you find any hidden missing values? If so recode them as NA.

  3. Many of the geographic values are missing, but geographic information is redundant in multiple variables in the dataset. For example, with effort you could determine the borough of an accident from the zip code, the latitude and longitude, or the streets (not part of the assignment for this week). Consider the borough variable- what percentage of the missing values of borough have values present of at least one of zip code or latitude. What about if we include all the street name variables? What fraction of rows don’t have any detailed location information (latitude, zip code, or street names)?

  4. The CRASH TIME variable has no missing values. Compute the count of how many times each individual time occurs in the crash data set. This will suggest that there are some inliers in the data. Compute summary statistics on the count data, and determine how many inliers there are (define an inlier as a data value where the count is an outlier, i.e. the count of that value is greater than 1.5*IQR + P75, i.e. 1.5 times the interquartile range past the 75th percentile for the distribution of counts for values of that variable.) For which inliers do you believe the time is most likely to be accurate? For which is it least likely to be accurate and why do you think so?

Problem 4: Finding Patterns in the Data

Formulate a question about crash data in NYC and make visualizations to explore your question. It could be related to the geographical distribution of accidents, the timing of accidents, which types of vehicles lead to more or less dangerous accidents, or anything else you want. Write comments/notes describing your observations in each visualizations you create and mention how these observations impact your initial hypotheses.

Useful questions to consider when you observe a pattern (not all of them might apply to your case):

  • Could this pattern be due to coincidence (i.e. random chance)?
  • How can you describe the relationship implied by the pattern?
  • How strong is the relationship implied by the pattern?
  • What other variables might affect the relationship?
  • Does the relationship change if you look at individual subgroups of the data?