Cadencia: Automatic Beatmap Generator for osu!mania

I like to play rhythm games, but most rhythm game songs are Japanese and I want to play other songs.

Beatmaps or charts, is a term for a collection of actions a player is required to perform during a song on specific timings. Different rhythm games have different ways, some more creative than others, to represent these beatmaps. For example, Project Sekai has falling notes on a lane-based playfield, where notes drop toward a judgement line at the bottom of the screen and the player has to tap, hold, or flick each one as it lands on the line.

Many rhythm games are very creative in how they make the experience enjoyable for players, this makes it quite difficult to generate beatmaps for these games. Therefore, for the sake of simplicity let’s consider creating maps for osu!mania.

osu!mania is a game mode inside the popular rhythm game osu!, which I played often in highschool and much less often now. osu!mania has multiple lanes and a judgement line at the bottom. A map with 4 lanes are called 4K maps and a map with 7 lanes are called 7K maps.

So I tried creating a beatmap automatic generator so that I can enjoy playing songs I like.

Architecture

I collected data for 1000 charts of varying difficulty. For each chart, we keep the audio features describing the music, note times, lanes, and hold lengths, difficulty (measured using osu!mania star ratings), and timing information from original chart.

We divide the song into 10-millisecond steps. At each step, the model receives nearby audio, previous notes, difficulty, and beat information. Its target describes what the human mapper placed there.

In particular, the question asked is: “At time $t$, given this music and previous notes, should something happen? If so, which lanes should contain taps or hold starts?” Many steps contain no notes so we sample some of these empty steps during training and weight them to account for the sampling.

I present the architecture for Cadencia. I use MERT-v2-30s as the audio encoder to extract features from the music. Its weights remain frozen during training, so we can use its existing musical representations.

I also added a mel-spectrogram branch. A mel spectrogram shows how the sound’s frequency content changes over time. The idea is to give the model additional information about short musical attacks that may help with note timing.

Then, we have a separate beat tracker that estimates the song’s beats and bar positions. This gives the model a sense of where it is within the rhythm, including when the tempo changes.

The model also needs to understand how activity changes throughout the song. A quiet introduction should usually feel different from an energetic chorus. For this, I use a bar planner: a small transformer that looks across the song and predicts four properties for each bar:

• How likely the bar is to contain no notes.
• How many notes it should contain per beat.
• How often notes should form chords.
• How much of the bar should be occupied by holds.

These predictions then guide the note decoder. The note decoder’s responsibility is to actually make the following predictions:

1. Timing - Should an event happen at this timestamp?
2. Placement - Which four-lane pattern should happen?
3. Duration - How long should each hold last?

The decoder itself is pretty small, just two layers with approximately 0.8 M parameters. Its self-attention combines information from previous events, while cross-attention lets it read the audio features. Before training with audio, I first trained parts of the decoder using chart sequences. Given previous events, it learned to predict the next event’s timing gap, lane pattern, and hold lengths. Those weights then provided a starting point for audio-conditioned training.

During training, the model receives feedback on three things: whether it places an event at the right time, whether it chooses the correct four-lane pattern, and whether its hold lengths match the human chart.

However, individually correct notes do not necessarily form a natural pattern. To address this, I also train on four-beat windows and compare transitions between neighboring patterns. This considers which lane combinations follow each other and how far apart they occur. An additional loss encourages these local transitions to resemble those in human charts.

This comparison uses predicted probabilities, allowing it to guide learning through backpropagation. It captures local pattern statistics, rather than judging the musical quality of an entire phrase.

The balance between these training signals matters. Most timestamps contain no notes, and some windows are much quieter than others. I combine several complete windows into each training batch, sample some empty timestamps, and weight the losses to account for the sampling. The pattern-transition loss receives a small weight that increases gradually at the start of training.

During training, the decoder receives previous notes from the human chart. During generation, it instead uses its own previous predictions. This makes playtesting important: matching a reference chart during training does not guarantee that a complete generated chart will feel good to play.

I probably should have created a decent baseline for comparison reasons, unfortunately we’re way past that. The next time I do research, having a good baseline model to produce iterative improvements is important.

Code

• Github
• Huggingface Model - WIP