Lab 2: The Linear Model

Overview

Complete this assignment by answering the following questions using code, text descriptions, and mathematics in a quarto markdown document. Render your .qmd to a pdf and submit both the qmd and pdf files on brightspace.

The purpose of this lab is for you to practice making linear regression models for inference and for prediction using statsmodels (my suggestion) or another library of your choosing. statsmodels is well featured for implementing statistical models and will automatically calculate most of the statistics that you need. scikit-learn is also possible to use. If you are feeling adventurous and want more of a challenge, you are welcome to perform the lab using pymc and do the regressions in a Bayesian framework (I would replace AIC with WAIC if you do so).

We will use two datasets, one for each problem. For problem 1 we will use a dataset called earnings. It can be found on the website for the textbook Regression and Other Stories, and if you are struggling this book has a good amount of discussion on this dataset. Use the version of the dataset that I provided, however, as I found some errors in the data processing in the originals on the website.

The second dataset is the UCI concrete compressive strength dataset, and we will use it for problem two. I have provided a training and testing dataset.

Problem 1: Confounds in Earnings Data

The dataset earnings.csv contains samples from the Work, Family, and Well Being survey, which was collected by a group led by Catherine Ross in 1990. We are working with a processed version of the dataset, which contains information on the height, weight, age, gender, ethnicity, earnings, education (for the respondent and their mother and father), and their health habits (levels of walk, smoking, exercise, anger, and tension). You can read a dataset description at earnings_dictionary.qmd or read it on this link. The goal of this problem is to understand which socioeconomic factors have an effect on earnings.

  1. Missing Values and Transformations: An important decision is how to handle earnk, the target variable, and whether to log transform it or not. Look at the distribution of earnk and discuss the pros and cons of using the log transform, and make a decision about whether to perform the transformation. Describe and justify how you handle missing the missing values in this problem. If you drop data with 0 earnings, desribe how that changes the population that your model describes. A helpful article to avoid some pitfalls if you use the ‘log’ transform is here: The log-0 problem. Two more complex and arguably better approaches that are beyond the scope of week 3 are the Two-Stage Model and the Heckman Correction.

  2. Height and Weight Confounds: Fit separate linear regression models predicting the effect of either height or weight on your earnings target variable (log_earnk or earnk). Report the model fit coefficients and their confidence intervals for each. Make a linear regression model that includes both height and weight and compare the confidence intervals for the coefficients. What is your explanation for what happened? Find another variable that could confound the relationship between height/weight and earnings, include it in a linear regression model, and explain the outcome.

  3. Potential Colliders: Consider each variable carefully, as well as your data processing steps. Are there any variables that you can identify which have the potential to cause collider bias in your regression model? You do not need to perform any regressions, just describe your reasoning.

Problem 2: Optimizing a Predictive Model by Engineering Features

The background goal of this problem is to devise a linear regression model which optimally predicts strength. You should use ‘AIC’ (or you may use a related quantity called ‘WAIC’ if you want to complete the problem set using ‘pymc’) to select models. If you use a very large number of engineered features, make sure to use the corrected version of the formula. You can download the data here:

  1. concrete_train.csv
  2. concrete_test.csv
  3. concrete_dictionary.txt

  1. Brawn Model: It is common in some domains to fit a linear regression model with a large number of interaction terms. This is called response surface modeling. Fit a model that uses all of the data variables, including all main effects, all pairwise interactions, and all squared terms to predict the variable ‘strength’. Calculate and report the AIC of this model on the training set.

  2. Feature Engineering and Variable Selection: Build a simple model that aims to improves on the Brawn Model by engineering a small number of features. These features can come from a few sources, including the patterns of residuals on the brawn model. Two useful sources are this short chapter Fly Ash Facts for Highway Engineers and the models presented in the ‘Modeling of Strength of High-Performance Concrete Using Artificial Neural Networks’ paper, in particular the introduction section and the section on statistical models. Concrete in lab conditions improves in strength over time as it sets. I recommend developing several time features by examining plots of how strength increases with age in the training set and/or the residuals of the brawn model. You are also encouraged to experiment with generating other features which use attributes of the mixture. For each feature, provide a brief rationale (including relevant plots of the residuals in (a) or of the data) for why you developed it, and fit a small number of models using your engineered features, reporting the AIC for your candidates.

  3. Evaluate the brawn model and the engineered models on the test set. Do the rankings out of sample agree with the rankings based on AIC? If it does not, redo the ranking with corrected AIC formula:

\[ \mathrm{AIC}_c = \mathrm{AIC} + \frac{2k(k+1)}{n-k-1} \]

where \(k\) is the total number of parameters and \(n\) is the number of data points and check if that changes the outcome.