Lab 9: Using and Evaluating LLMs

Author

George I. Hagstrom

Overview

Data

In this assignment you will be working with several different datasets.

Each contains a text string which corresponds to a review and the numerical rating given to that movie.

  • The second dataset contains randomly selected negative Amazon reviews that have been labeled according to whether they identify certain flaws in a given product. The values of the human response are None, Quality (the product is of low overall quality, is not fit for purpose), Quality Control (the product sometimes arrives broken or differs from batch to batch), Durability (the product breaks too easily), False Advertising (the product does not match expectations based on the picture or product description). There is a “test” set and a “training” set.

  • ‘amazon_training_examples.csv’

  • ‘amazon_testing_examples.csv’

  • The third dataset (for the extra credit question) consists of a subset of negative (below 4 stars) product reviews from a selection of different products, which are identified by their SKU. Each row contains the SKU for a single product and a string containing the text of 10 negative Amazon reviews for that product, concatenated together.

  • ’amazon_reviews_by_product.csv”

Software

To complete this lab assignment, you will need to be able to interact with LLMs using the R packages ellmer and ’vitals](https://vitals.tidyverse.org/). You may also want to use the [mall`. There are two ways to connect LLMs with R:

  1. Acquiring an API key from one of the major providers, such as anthropic, openAI, or google amongst others
  2. Installing open weights models on your computer, for instance using ollama

The downside of (1) is that it can cost a small amount of money or have otherwise limited usage. Google provides one year of free AI Pro to all college students. If you subscribe to this you will be able to obtain an API key with likely sufficient usage to complete this assignment (though do be cautious about what models you use). I will provide API keys for anthropic for those who request them, but credits are limited. The downside of (2) is that the models are inferior and run more slowly. However in all cases there are cheap/fast models that can be used for this assignment (flash,flash-lite,haiku, llama3, nanogpt etc). Installation instructions and API key acquisition depends on what you choose, and there are links to compatible LLMs on the ellmer page. Ollama can be installed by clicking here. You can read about google gemini API keys here.

Problem 1: LLMs for Sentiment Analysis

In this problem you are going to use either ellmer (with parrallel_chat_structured) or mall (with llm_sentiment) to perform sentiment analysis on the large movie reviews dataset at IMDB.

  1. Select three different LLMs (don’t use Claude Opus or Gemini Pro, and I recommend against expensive deep thinking models but you can try things versions of sonnet, haiku, nanogpt, llama-3, or even deepseek if you have the processing power for it). Use either the mall package’s llm_sentiment function or the ellmer package’s parallel_chat_structured function. Report the system prompt that you use for each LLM. Ensure that the output of the LLMs is restricted to a integer values between 1 and 10, using options if you are using mall or type_enum if you are using ellmer. The reviews are contained in the file ‘imdb_test.csv’. You will need to put the reviews into a list where each element contains a single review in a character format to use the functions in these packages. Compute the correlation coefficient of each of the models output and create a table which shows the correlation and the cost for each model. Do the more expensive models perform better? When I performed a sentiment analysis using standard methods on this dataset, the correlation coefficient was 0.53. Compare that to the LLM performance.

  2. Update the system prompts that you used in (b) by adding examples of reviews from the imdb_train.md file (you can add text to the beginning of this file to prompt the LLM and then use the function interpolate_file to add a file to the system prompt), or by modifying the chat object to include instances of receiving movie reviews as text and outputing the correct rating. If you do include additional text in the file as a system prompt, be sure to report what text you used and to upload the file you used for your prompt. Repeat the sentiment analysis of the part (b) using these modified prompts, and again compute the correlation and the cost, outputing them in a table. How did enhancing the prompts change the results? Was it model dependent?

Note: I was shocked at the high correlation achievable with some models and wondered if there was a data leakage issue (data leakage is a term from machine learning that describes a situation where data for testing the algorithm is unexpectedly part of the data set that was used to train the algorithm), because LLMs are trained using a very large corpus of data which may include the reviews and the ratings. I did a small experiment on reviews gathered from the past few weeks and found roughly the same answer so I think the result of this homework is not explained by data laekage.

Problem 2: LLMs for identifying flaws in products from user reviews.

For this problem, I have selected a subset of negative amazon reviews taken from a public dataset of amazon reviews. The goal is to extract information about potential flaws in products from these reviews automatically using LLMs. These reviews are divided into two files, a training set which may be incorporated into prompts, and a test set. I have labeled the Amazon reviews into several categories:

  • None indicates no specific flaw was identified with the product
  • Quality indicates that the product is poorly designed for the intended purpose
  • Quality Control indicates that different instances of the product are inconsistent with each other or that the product sometimes arrives broken
  • Durability indicates that the product is damaged easily from use
  • False Advertising indicates that the product doesn’t correspond to the images or online descriptions on Amazon

Your goal is to determine the accuracy of LLMs at classifying text into one of these categories, compared against a human labeler.

  1. Again select three LLMs (they do not have to be the same as in problem 1, but it would make sense to use the same ones to observe if the same accuracy relationships hold on a new task). The human labeled reviews for training are contained in the file ‘amazon_training_examples.csv’, and the labeled testing reviews are contained in ‘amazon_testing_examples.csv’. Use parallel_chat_structured with a type_enum corresponding to the categories described above. Again you should create a list of prompts containing each review, and modify the system prompt or the chat object history with the training reviews. Compute the accuracy and cost of each LLM in comparison to the labels, and comment on the results. If you selected the same models as in problem 1, do the same relationships hold in this experiment?

  2. Determine which questions have the highest error and examine them. Do you agree with the human label? Compare the error patterns across models- note if there are some models which tend to make mistakes on a distinct set of reviews compared to other questions.

Problem 3 (Extra Credit- up to 10 points): Using a LLM judge and the vitals package to screen products for problems.

The vitals package has a number of features which enable comparison and evaluation of LLMs. These include the ability to use an LLM to score the outputs of models, which is called an “LLM as a judge” evaluation. For this problem, I have assembled another dataset of amazon reviews, tabulated by product. The reviews are contained in the file ’amazon_reviews_by_product.csv”. For each product I have assembled 10 negative reviews, with the text concatenated together for each product so that it forms a single string per product. Create a vitals dataset where each input contains all of the negative reviews, and each target contains criteria for the llm-judge to evaluate the outputs by. Create a solver for your chosen LLMs and use model_graded_qa as the scorer. Use a strong model as the judge (not a “thinking model” but something like sonnet or flash , and use cheaper models to compare against each-other (make sure the judge model isn’t also one of the models being used for a comparison). Perform the comparison of the different models at the task of identifying the primary problems with each product. I recommend using the “Incorrect, Correct, Partial Credit” scheme to score the models. Output the results of the model comparison using vitals. Which models perform best? Examine some of the product reviews for which model performance varies, or which are very challenging for all models. Can you understand why those problems were challenging or differentiators between models (I realize this is a subjective question- I just want your thoughts on what happened in your experiment).