~/work/melody-transformer/README.md
Melody Transformer
I taught a GPT to write melodies. You can listen to one below.
Twelve pitch classes in one octave, so timing and note length are mine, not the model's: every step is played as an even eighth note.
Problem
Language models predict the next character. Melodies are sequences too. How far can a small GPT get if notes are its alphabet?
Approach
- Music as text. Melodies from the BiMMuDa MIDI collection are reduced to one octave, with each note written as one of twelve pitch-class characters (lowercase for sharps),
Rfor a rest and a newline between melodies. - Baseline: the course’s character-level GPT (6 blocks, 6 heads, 384-dim, ReLU), re-pointed at the 14-token musical alphabet. Its train loss fell to 0.46 while validation stalled at 1.56, a clear sign of memorisation.
- Extended model: wider and deeper (8 blocks, 8 heads, 512-dim) with GELU, more dropout (0.3) and a lower learning rate (2 × 10⁻⁴). Train and validation loss ended close together, at 0.96 and 1.01.
- Back to sound. A converter turns generated text into MIDI. The player above does the same in your browser with the site’s Web Audio synth.
Honest caveats
The two runs used slightly different preprocessing, so the loss comparison is indicative rather than controlled. The split was sequential rather than by song, and the data was already transposition-augmented, so related melodies can appear on both sides. A stricter follow-up would hold out whole songs. The model only knows pitch classes: rhythm, note length and octave in the playback are my choices, not its.
Why I like it
It sits between two of my other projects: Melody Masters synthesises music by hand, and my dissertation is about what Transformer-based models get wrong. Here a Transformer gets to be creative instead.
~/architecture
- BiMMuDa MIDI → one-octave strings (C c D … B, R)
- 14-token vocabulary · 256-step context
- decoder-only Transformer (masked self-attention)
- autoregressive sampling
- text → MIDI converter