✨ Mini-Series: The Creative Shell – What Makes LLMs “Creative”? (Part 1/5)

Our first stop inside the “Creative Shell” is: 🌡️Temperature! (see Part 0 for context)

If you’ve worked with an LLM API, you’ve probably seen this parameter, but what does it actually do?

When we prompt the model with: “The sky is …”

Behind the scenes, the model first converts your text into numbers that feed into its artificial neural network (ANN). Then, the ANN produces “logits”… raw numerical scores for every possible next word (often 50,000+ options in GPT-2, see Python code 🐍 below).

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "gpt2"

# Load the tokenizer corresponding to the model
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Select the device to run on:
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load the pretrained causal language model (for text generation)
model = AutoModelForCausalLM.from_pretrained(model_name).to(device)
model.eval()

# Define your input text (the prompt)
prompt = "The sky is"

# Tokenize the input text into tensors compatible with the model
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Disable gradient computation since we’re only doing inference 
with torch.no_grad():
    # Forward pass through the model
    outputs = model(**inputs)

# Access the logits for each token position
# [batch_size, sequence_length, vocab_size]
# Take only the logits for the last token in the sequence
next_token_logits = outputs.logits[:, -1, :]  # shape: [1, 50257]

# Get the top 10 logits and their indices
topk_logits, topk_indices = torch.topk(next_token_logits, k=5)

# Decode the indices to get the corresponding words (tokens)
topk_words = [tokenizer.decode([idx.item()]) for idx in topk_indices[0]]

# Display the words and their raw logits
print("Top 5 words and their raw logits:")
for word, logit in zip(topk_words, topk_logits[0]):
    print(f"Word: '{word.strip()}'\tLogit: {logit.item():.4f}")

These logits form a fixed vector: a numerical score for each word. In our example, the model scores potential next words as:

• “the” → −93.7
• “blue” → −94.3
• “falling” → −94.5
• “a” → −94.8

Think of logits as the model’s confidence levels for what the next word should be. They’re all negative … the less negative, the more likely a word will appear next.

These values come directly from training, representing the model’s learned sense of what fits best (visualized in the left plot below … all plots coded in Jupyter notebook).

BUT, every time you type “The sky is …”, the model produces identical logits. It’s deterministic 😒… nothing creative about it yet!

So why do you sometimes get different answers to the same prompt in ChatGPT?

This is where creativity begins: Temperature. 🌡️

The raw logits (left plot) that the ANN produces are divided by Temperature to create scaled logits (middle plot), before converting to probabilities (right plot).

Once these probabilities are calculated, the model samples from them to decide the next word. This is what introduces variation and creativity… and we’ll see this process in detail in the next post 🤩.

Notice in the plots below how changing the temperature slider reshapes the probability distribution:
Here’s what happens:
👉 Temp = 1.0 → logits unchanged (left and middle plots match)
👉 Temp = 0.5 → divide by 0.5 → scaled logits become more negative (amplified) → probabilities concentrate on fewer dominant options
👉 Temp = 2.0 → divide by 2.0 → scaled values approach zero (squashed) → probabilities flatten, letting more options compete

So, when you adjust temperature:
➡️ Low Temp (<1) → fewer words dominate → predictable, systematic output with limited variation
➡️ High Temp (>1) → many words compete → diverse, creative, wilder responses

Temperature is literally the creativity dial, but it’s only half the story!

We’ve scaled the scores. But how does the model actually choose? 🤔
Tomorrow, we’ll look at the Softmax function – the dice inside every LLM 🎲.

The shell is just beginning to open. 🐚