Made-up shop inbox, decide score (D8)

Muse Glimmer LoRA · Muse Glimmer 30B · finished Nov 1, 04:07 UTC

The run

Recipe: Muse Glimmer LoRA
Model: Muse Glimmer 30B
Dataset: Made-up shop inbox (D8 test)
GPU: 1x A100 SXM4 (40 GB)
GPU time: 9 min
GPU cost: $0.30
LoRA rank: 16
Learning rate: 0.0002
Mode: decide
Epochs: 2
Score only: 1

Both models were scored on the same held-out test rows by reading how much probability each gave every option's letter. No judge and no model call: the scores are counted, not graded.

Decision results

78 held-out test rows · scored, no judge

Base Glimmer accuracy

100%

78 of 78 · 95% interval 95% to 100%

Macro F1

1.00

each label counts the same

Calibration error

0.0%

4.4% before T = 0.25

Base Glimmer was right on 78 of 78 held-out examples (100%), with nothing trained.

T = 0.25 was fit on 38 dev rows to bring confidence closer to how often the model was right on them. On the held-out test rows the calibration error is 0.0% with T (4.4% before).

It can be wrong: check anything that matters.

Per label

Base Glimmer

Precision, recall and F1 for each label on the test rows
Label	Test rows	Precision	Recall	F1
Billing	20	1.00	1.00	1.00
Shipping	20	1.00	1.00	1.00
Returns	20	1.00	1.00	1.00
Account	18	1.00	1.00	1.00

Confusion matrix

Base Glimmer: rows are the right label, columns the pick

Base Glimmer's picks on the test rows: each row is the right label, each column what it picked
Right label	Billing	Shipping	Returns	Account
Billing	20	0	0	0
Shipping	0	20	0	0
Returns	0	0	20	0
Account	0	0	0	18

Reliability

After T, on the test rows

Base Glimmer (T = 0.25)
Right as often as it says

Each dot is the test rows whose confidence fell in one tenth of the range: across, how sure the model said it was; up, how often it was right. On the dashed line it is right as often as it says. Calibration error base Glimmer 0.0% with T, 4.4% before.

The bins as a table

Reliability bins after T
Confidence	Base Glimmer: rows, right, mean confidence
0% to 10%	none
10% to 20%	none
20% to 30%	none
30% to 40%	none
40% to 50%	none
50% to 60%	none
60% to 70%	none
70% to 80%	none
80% to 90%	none
90% to 100%	78, 100%, 100%

Baselines

All on the same test rows

Accuracy of each model and baseline on the same test rows
Approach	Right	Accuracy
Base Glimmerzero-shot, same prompts	78 of 78	100%95% to 100%
Always Returnsthe commonest training label	20 of 78	26%17% to 36%
Word overlapa stand-in, not a model: the label sharing most words	61 of 78	78%68% to 86%

Health

Letter mass, median and lowest: base Glimmer 0.90, 0.82
Answered "none of the listed options": base Glimmer 0%
Decline check (right label taken out): base Glimmer 72 of 78 (92%)
Renderings too long to score: 0
Test rows with no usable score: 0 of 78
Prompts per row: 1

Letter mass is the share of the first answer token's probability that lands on the shown letters; near 1 the model answers in the format. Base Glimmer sometimes starts with a word instead of a letter, so its letter mass is lower and this first-token read understates what a prompted base model could do.