Made-up shop inbox, decide real (D8)

Muse Glimmer LoRA · Muse Glimmer 30B · finished Nov 1, 04:23 UTC

The run

Recipe: Muse Glimmer LoRA
Model: Muse Glimmer 30B
Dataset: Made-up shop inbox (D8 test)
GPU: 1x A100 SXM4 (40 GB)
GPU time: 15 min
GPU cost: $0.50
LoRA rank: 16
Learning rate: 0.0002
Mode: decide
Epochs: 2
Score only: 0

Both models were scored on the same held-out test rows by reading how much probability each gave every option's letter. No judge and no model call: the scores are counted, not graded.

Decision results

78 held-out test rows · scored, no judge

Tuned accuracy

100%

78 of 78 · 95% interval 95% to 100%

Base Glimmer accuracy

100%

78 of 78 · 95% interval 95% to 100%

Macro F1

1.00

base 1.00 · each label counts the same

Calibration error

0.0%

0.0% before T = 0.25

The tuned model was right on 78 of 78 held-out examples (100%); base Glimmer was right on 78 (100%).

T = 0.25 was fit on 38 dev rows to bring confidence closer to how often the model was right on them. On the held-out test rows the calibration error is 0.0% with T (0.0% before).

The tuned model is a small LoRA adapter on Muse Glimmer 30B, trained on 274 of this dataset's examples. The test rows were held out before training, and both models were scored on the same ones.
It can be wrong: check anything that matters.

Per label

Tuned · base Glimmer

Precision, recall and F1 for each label on the test rows, tuned then base Glimmer
Label	Test rows	Precision	Recall	F1
Billing	20	1.00 · 1.00	1.00 · 1.00	1.00 · 1.00
Shipping	20	1.00 · 1.00	1.00 · 1.00	1.00 · 1.00
Returns	20	1.00 · 1.00	1.00 · 1.00	1.00 · 1.00
Account	18	1.00 · 1.00	1.00 · 1.00	1.00 · 1.00

Confusion matrix

Tuned: rows are the right label, columns the pick

Tuned's picks on the test rows: each row is the right label, each column what it picked
Right label	Billing	Shipping	Returns	Account
Billing	20	0	0	0
Shipping	0	20	0	0
Returns	0	0	20	0
Account	0	0	0	18

Base Glimmer's matrix

Base Glimmer's picks on the test rows: each row is the right label, each column what it picked
Right label	Billing	Shipping	Returns	Account
Billing	20	0	0	0
Shipping	0	20	0	0
Returns	0	0	20	0
Account	0	0	0	18

Reliability

After T, on the test rows

Tuned (T = 0.25)
Base Glimmer (T = 0.25)
Right as often as it says

Each dot is the test rows whose confidence fell in one tenth of the range: across, how sure the model said it was; up, how often it was right. On the dashed line it is right as often as it says. Calibration error tuned 0.0% with T, 0.0% before; base Glimmer 0.0% with T, 4.5% before.

The bins as a table

Reliability bins after T
Confidence	Tuned: rows, right, mean confidence	Base Glimmer: rows, right, mean confidence
0% to 10%	none	none
10% to 20%	none	none
20% to 30%	none	none
30% to 40%	none	none
40% to 50%	none	none
50% to 60%	none	none
60% to 70%	none	none
70% to 80%	none	none
80% to 90%	none	none
90% to 100%	78, 100%, 100%	78, 100%, 100%

Baselines

All on the same test rows

Accuracy of each model and baseline on the same test rows
Approach	Right	Accuracy
Tunedthe trained adapter	78 of 78	100%95% to 100%
Base Glimmerzero-shot, same prompts	78 of 78	100%95% to 100%
Always Returnsthe commonest training label	20 of 78	26%17% to 36%
Word overlapa stand-in, not a model: the label sharing most words	61 of 78	78%68% to 86%

Health

Letter mass, median and lowest: tuned 1.00, 1.00 · base Glimmer 0.90, 0.82
Answered "none of the listed options": tuned 0% · base Glimmer 0%
Decline check (right label taken out): tuned 78 of 78 (100%) · base Glimmer 72 of 78 (92%)
Renderings too long to score: 0
Test rows with no usable score: 0 of 78
Prompts per row: 1

Letter mass is the share of the first answer token's probability that lands on the shown letters; near 1 the model answers in the format. Base Glimmer sometimes starts with a word instead of a letter, so its letter mass is lower and this first-token read understates what a prompted base model could do.