Evaluating LLMs for R Coding using Vitals

Click here for a video coding vignette using the vitals package to evaluate different large language models on the are ggplot2 coding problem dataset

In this coding vignette I go through the introductory vignette from the vitals webpage, which shows how to use vitals to evaluate the performance on different coding challenges. The dataset has both text questions and also a description of the answer in text with samples of allowable code. We will use the “LLM-as-a-judge” approach, in which a grader LLM compares the output of each LLM on each question either to an answer or evaluates it using a description of answer criteria. My video follows the vitals example closely except for a few choices of prompt and the LLMs used.

Click here to watch the video

The live code that was used for this vignette can be found on github at Vitals_Live.R.