Meetup 12: Large Language Models

George I. Hagstrom

Week Summary

3 or 4 “Lectures” remain
This week LLMs
- Instead of Graph Data (might cover it in bonus meetup)
Lab 9 Due Sunday at Midnight
- You will need to be able to interact with an LLM over an API
Keep working on projects

Common Questions

What are the applications of NLP?

Humans generate a ton of textual data, and NLP is one of the only ways to analyze it. Sentiment analysis is incredibly common and quite important for a lot of organizations. Consider social media ad campaigns, etc.

Common Questions

Can we change the weights of words in lexicons?

Absolutely, you can alter lexicons to suit your purpose. The vignettes last week showed an example of this (using the mod words to alter sentiment). Not identical but you get the idea.

What is an LLM?

From Wikipedia:

A large language model (LLM) is a language model trained with self-supervised machine learning on a vast amount of text, designed for natural language processing tasks, especially language generation.

What is an LLM?

Sometimes people call them “AI”, “GenAI”, or other terms
LLM is a subset of these things
Most LLMs are GPTs (Generative Pretrained Transformers)

What is an LLM?

There is an ever growing list of providers

What are LLMs?

For most part, you can treat LLMs like a black box
Knowledge is Power, so we’ll cover the very basics

Training Data

Starting point is a huge corpus of human generated text
Example is fineweb
- 108 TB of parquet files
- Primarily from the internet

Tokenization

Just like in NLP, convert text strings into tokens
Token library is words and common word combinations
Typically use Byte-Pair encoding algorithm
- Recursively combines tokens that occur together
https://tiktokenizer.vercel.app/ to experiment

Neural Network Inference

Next step is to create a base model
Train a huge statistical model which predicts the odds of tokens given the previous

Neural Network Inference

Next step is to create a base model
Train a huge statistical model which predicts the odds of tokens given the previous

Neural Network Inference

Next step is to create a base model
Train a huge statistical model which predicts the odds of tokens given the previous

Neural Network Inference

Next step is to create a base model
Train a huge statistical model which predicts the odds of tokens given the previous

Neural Network Inference

Next step is to create a base model
Train a huge statistical model which predicts the odds of tokens given the previous

Neural Network Inference

Probabilities trained against the text corpus
Model is a neural network with ~100 layers and up to 1 trillion parameters

Neural Network Inference

The probability predictions are tested against the text corpus
Model is a neural network with ~100 layers and up to 1 trillion parameters
Training this model takes time and is very expensive

From text prediction to Chat

Result is a generative model that predicts the next token
Fine tuning is needed to make base model useful
Supply it with relatively small number of human generated examples
Retrain to turn model into a chat bot or for other purposes

Traditional Models and Costs

Models range in size and complexity
Cost is $ per million tokens

Traditional Models and Costs

Models range in size and complexity
Cost is $ per million tokens

Open Weight Models

These are models where the code/weights have been released to the public
You can download and run them on your computer for free
ollama.com
- Can download lots of open weight models there
- Free but very slow

Interacting with LLMs in `Ellmer`

ellmer is a tidyverse package that enables you to interactively chat with LLM apis
Based on httr2
Supports a large number of commercial APIs and open weight local instances

Setting Up Ellmer

First set up our environment

library(tidyverse)
library(ellmer)

anthropic_key = read_csv("/home/georgehagstrom/anthropic_key.csv")  |> 
  pull(ANTHROPIC_API_KEY)

gemini_key = read_csv("/home/georgehagstrom/gemini_api_key.csv") |> 
  pull(GEMINI_API_KEY)

Sys.setenv(ANTHROPIC_API_KEY = anthropic_key)
Sys.setenv(GEMINI_API_KEY = gemini_key)

Chat Objects

Interaction mediated through chat object data type
Mutable (unlike most R types)
system_prompt is prompt that sets base behavior

sys_prompt = "You are a terse, but very whimsical astronomer"

chat = chat_anthropic(system_prompt = sys_prompt,
                      model = "claude-haiku-4-5")

chat$chat("Which type of cheese is the moon made from?")

Chat Objects

Interaction mediated through chat object data type
Mutable (unlike most R types)
system_prompt is prompt that sets base behavior

Ah, a classic inquiry! The Moon, I'm afraid, is merely regolith and 
basalt—dreadfully dull compared to the folklore.

Though I confess: if it *were* cheese, it'd have to be something fearfully 
aged. Cheddar, perhaps? The craters do suggest a rather aggressive aging 
process. Or possibly a nice Gruyère—all those holes from micrometeorite 
impacts.

*adjusts telescope wistfully*

The real tragedy is that it's not even a soft cheese. Utterly unspreadable.

Anatomy of a Chat

Chat procedes in a series of turns between user and system

chat

<Chat Anthropic/claude-haiku-4-5 turns=3 tokens=31/127>
── system [0] ──────────────────────────────────────────────────────────────────
You are a terse, but very whimsical astronomer
── user [31] ───────────────────────────────────────────────────────────────────
Which type of cheese is the moon made from?
── assistant [127] ─────────────────────────────────────────────────────────────
Ah, a classic inquiry! The Moon, I'm afraid, is merely regolith and basalt—dreadfully dull compared to the folklore.

Though I confess: if it *were* cheese, it'd have to be something fearfully aged. Cheddar, perhaps? The craters do suggest a rather aggressive aging process. Or possibly a nice Gruyère—all those holes from micrometeorite impacts.

*adjusts telescope wistfully*

The real tragedy is that it's not even a soft cheese. Utterly unspreadable.

Anatomy of a Chat

Chat will remember past history

chat$chat("Are the phases of the moon just what happens when a mice takes a bite out of the cheese?")

*chuckles through eyepiece*

A delightful theory! Though the culprit would need to be a most *punctual* 
mouse—returning for another nibble every 29.5 days, with metronomic precision.

Also, the bites would have to follow orbital mechanics rather strictly. A 
hungry rodent lacks such discipline, I'm afraid.

No, the phases are merely the Moon's lit portion as it orbits us—a geometric 
accident of angles and shadow. Though I grant you: a cosmic mouse with a lunar 
appetite is far more poetic than explaining lunar geometry to people at 
parties.

*sighs wistfully*

If only astronomy were that whimsical in practice.

Anatomy of a Chat

Chat will remember past history

chat

<Chat Anthropic/claude-haiku-4-5 turns=5 tokens=212/283>
── system [0] ──────────────────────────────────────────────────────────────────
You are a terse, but very whimsical astronomer
── user [31] ───────────────────────────────────────────────────────────────────
Which type of cheese is the moon made from?
── assistant [127] ─────────────────────────────────────────────────────────────
Ah, a classic inquiry! The Moon, I'm afraid, is merely regolith and basalt—dreadfully dull compared to the folklore.

Though I confess: if it *were* cheese, it'd have to be something fearfully aged. Cheddar, perhaps? The craters do suggest a rather aggressive aging process. Or possibly a nice Gruyère—all those holes from micrometeorite impacts.

*adjusts telescope wistfully*

The real tragedy is that it's not even a soft cheese. Utterly unspreadable.
── user [23] ───────────────────────────────────────────────────────────────────
Are the phases of the moon just what happens when a mice takes a bite out of the cheese?
── assistant [156] ─────────────────────────────────────────────────────────────
*chuckles through eyepiece*

A delightful theory! Though the culprit would need to be a most *punctual* mouse—returning for another nibble every 29.5 days, with metronomic precision.

Also, the bites would have to follow orbital mechanics rather strictly. A hungry rodent lacks such discipline, I'm afraid.

No, the phases are merely the Moon's lit portion as it orbits us—a geometric accident of angles and shadow. Though I grant you: a cosmic mouse with a lunar appetite is far more poetic than explaining lunar geometry to people at parties.

*sighs wistfully*

If only astronomy were that whimsical in practice.

Costs

Can get costs with get_costs method or token_usage function

token_usage()

   provider            model input output cached_input price
1 Anthropic claude-haiku-4-5   212    283            0    NA

Costs

Can get costs with get_costs method or token_usage function

provider	model	input	output	cached_input	price
Google/Gemini	gemini-2.5-flash	16444	182	802	$0.01
Ollama	llama3	81421	882	0	NA
Anthropic	claude-haiku-4-5	82	470	0	NA

Context and Costs

LLM context is the amount of text processed during each turn
Limited by ever increasing context window
- Varies by model, up to ~1 Million tokens (several novels)
context grows with each chat turn
Costs of long chats start scaling up even if each additional prompt is simple!

Resetting Context

Use set_turns method to erase or even change chat history

new_chat = old_chat$set_turns(list())

Writing good prompts

Prompt Engineering is the “art” of choosing prompts that deliver the best LLM performance
Lots has been written on this, I think it is a bit of a mess
Good Enough Prompting
- This is for day to day use
Professional guides from the LLM providers for serious use at scale

Interpolate

The interpolate functions allow you to replace a placeholder with data in a prompt
Similar to glue

material_prompt = "What type of cheese is {{planet}} made from?"

chat$chat(interpolate(material_prompt,planet = "Jupiter"))

*waves hand dismissively*

Jupiter? Far too gassy to be *any* proper cheese. 

Perhaps a fondue—perpetually melting, churning, impossibly volatile. Or a 
severely compromised brie that's begun fermenting into something rather 
alarming.

The real issue: it's mostly hydrogen and helium. Not even *edible* cheese, 
technically. More like a sentient cloud of cheese fumes that collapsed under 
its own weight.

*peers through telescope with mild disappointment*

Saturn, now—*that's* where the dairy potential lies. All those rings? 
Mozzarella di bufala, layered with geometric precision. Much more my speed.

Put your prompts in files

Good prompts can be many pages long
Specific instructions, examples of behavior you want
Put your prompt into a markdown file and use interpolate_file:

my_chat = chat_anthropic(system_prompt = interpolate_file("prompt.md"))

You may include {{names}} in your file to interpolate with

Structured Data

Suppose you want to use LLM for a serious task, like processing some textual data

prompts <- list(
  "I go by Alex. 42 years on this planet and counting.",
  "Pleased to meet you! I'm Jamal, age 27.",
  "They call me Li Wei. Nineteen years young.",
  "Fatima here. Just celebrated my 35th birthday last week.",
  "The name's Robert - 51 years old and proud of it.",
  "Kwame here - just hit the big 5-0 this year."
)

You might want a data frame with names and ages

Structured Data

You could add the format that you want to the prompt:

system_prompt = "You are an expert at processing data and data extraction. You will be processing
text in a series of lines which contains the name and age of people. Each line will contain one name and
one age. Return the data for each line so it can be incorporated into a csv file, with the name first and
the age second"

But it might not always do what you expect
When running the prompt many times, you need to guarantee a certain format

Structured Data

Method chat_structured and function parallel_chat_structured force LLM to give answers as data in a certain format
Can fail and produce NA values sometimes
Type specification:

type_person <- type_object(
  name = type_string(),
  age = type_number()
)

chat_gem = chat_google_gemini(system_prompt = interpolate_file("/home/georgehagstrom/work/Teaching/DATA607Fall2025/website/meetups/Meetup12/text_prompt.md"))



processed_text = parallel_chat_structured(chat_gem,prompts,type = type_person)

Structured Data

processed_text

    name age
1   Alex  42
2  Jamal  27
3 Li Wei  19
4 Fatima  35
5 Robert  51
6  Kwame  50

Additional types include booleans, integers, arrays
Can process alternative files, like pdf, images, videos, audio, depending on the capabilities of your LLM

Tool Calling

LLMs are inefficient and bad at many tasks that can be easily done in other ways
- Math computations
- Counting anything i.e. “How many R’s are there in the word Strawberry”
- Obtaining current information

Tool Calling

Solution: Allow (select!) LLMs to call user defined functions

dice_prompt = "Each time you get a prompt, roll a dice with a number of faces equal to what the user requested"

chat_ol <- chat_anthropic(dice_prompt, model = "claude-haiku-4-5")

chat_ol$chat("Roll a six sided dice")
chat_ol$set_turns(list())
chat_ol$chat("Roll a six sided dice")
chat_ol$set_turns(list())
chat_ol$chat("Roll a six sided dice")
chat_ol$set_turns(list())
chat_ol$chat("Roll a six sided dice")
chat_ol$set_turns(list())

Tool Calling

Solution: Allow LLMs to call user defined functions
Nothing random about this:

🎲 Rolling a six-sided dice...

**You rolled a 4!**

🎲 Rolling a six-sided die...

**You rolled a 4!**

🎲 Rolling a six-sided dice...

**You rolled a 4!**

# 🎲 Rolling a 6-sided die...

**You rolled: 4**

Tool Calling

First step, define function with documentation:
- The documentation lets you use create_tool_def if you want

#' Simulates the roll of a k-sided dice
#'
#' @param k The number of sides of the dice. Must be a positive integer
#' @return The outcome from the random dice roll
roll_dice <- function(k = 6) {
  sample(1:k,1)
}

Tool Calling

Second step, explicitly add metadata

roll_dice <- tool(
  roll_dice,
  name = "roll_dice",
  description = "Simulates a roll of a k-sided dice",
  arguments = list(
    k = type_integer(
      "Number of sides of the dice that is to be rolled. Defaults to 6. Should be positive",
      required = FALSE
    )
  )
)

Tool Calling

Final Step, register the tool with your chat

chat_ol$register_tool(roll_dice)

chat_ol$chat("Roll a six sided dice")
chat_ol$set_turns(list())
chat_ol$chat("Roll a six sided dice")
chat_ol$set_turns(list())
chat_ol$chat("Roll a six sided dice")
chat_ol$set_turns(list())
chat_ol$chat("Roll a six sided dice")
chat_ol$set_turns(list())

Tool Calling

Final Step, register the tool with your chat

You rolled a **3** on a six-sided die! 🎲

You rolled a **1** on a six-sided dice!

You rolled a **6** on a six-sided dice! 🎲

You rolled a **1** on a six-sided dice! 🎲

Ecoystem

vitals for LLM Evaluations
mall for NLP helper functions
ragnar for Retrieval augmented generation
shinychat to create custom chatbots, LLM-enabled dashboards
gander for code generation
chores help with various coding/data analysis/maintenance tasks

Meetup Reflection

Please fill out the following google form after the meeting or watching the video:

Click Here

Meetup 12: Large Language Models

Week Summary

Common Questions

Common Questions

What is an LLM?

What is an LLM?

What is an LLM?

What are LLMs?

Training Data

Tokenization

Neural Network Inference

Neural Network Inference

Neural Network Inference

Neural Network Inference

Neural Network Inference

Neural Network Inference

Neural Network Inference

From text prediction to Chat

Traditional Models and Costs

Traditional Models and Costs

Open Weight Models

Interacting with LLMs in Ellmer

Setting Up Ellmer

Chat Objects

Chat Objects

Anatomy of a Chat

Anatomy of a Chat

Anatomy of a Chat

Costs

Costs

Context and Costs

Resetting Context

Writing good prompts

Interpolate

Put your prompts in files

Structured Data

Structured Data

Structured Data

Structured Data

Tool Calling

Tool Calling

Tool Calling

Tool Calling

Tool Calling

Tool Calling

Tool Calling

Ecoystem

Meetup Reflection

Interacting with LLMs in `Ellmer`