7 ton shark

Simple interactive music with SoundBox

For my 2022 js13k entry, I used the SoundBox tracker to compose my background music -- really just an ambient background loop, a few long wind noises and some random percussion. I've wanted for a while to make a game with music that was more interactive, and this year's entry is the perfect time to try it.

There are different ways to make music interactive, but in this article I'm making use of "layers" -- the full song will have some tracks that are muted by default, and we'll bring them in and out depending on the situation.

Tweaking the player #

By default, the SoundBox player builds the entire song and gives it to us as an audio buffer. If we're going to control the volume (gain) on individual tracks, we need to change that.

In the top section of player-small.js, we'll add a new array to hold our channel buffers:

    var mSong, mLastRow, mCurrentCol, mNumWords, mMixBuf, mChannelBufs;

We'll initialize it in the init() function:

        mChannelBufs = [];

        for (let i = 0; i < song.numChannels; i++) {
            mChannelBufs.push(new Int32Array(mNumWords));
        }

And then down in the generate() function, we'll save the output to these buffers in addition to the "full" song buffer:

                    // ...and add to stereo mix buffer
                    mMixBuf[k] += lsample | 0;
                    mMixBuf[k+1] += rsample | 0;

                    // ...and channel (this is new)
                    mChannelBufs[mCurrentCol][k] += lsample | 0;
                    mChannelBufs[mCurrentCol][k+1] += rsample | 0;

Now, down in the createAudioBuffer() function, we can optionally take a channel number, to get just one track instead of all of the tracks mixed together:

    this.createAudioBuffer = function(context, channelNumber) {
        let source = channelNumber === undefined ? mMixBuf : mChannelBufs[channelNumber];

        var buffer = context.createBuffer(2, mNumWords / 2, 44100);
        for (var i = 0; i < 2; i ++) {
            var data = buffer.getChannelData(i);
            for (var j = i; j < mNumWords; j += 2) {
                data[j >> 1] = mMixBuf[j] / 65536;
                data[j >> 1] = source[j] / 65536;
            }
        }

        return buffer;
    };

Check out the commit on GitHub to see these changes in context.

Loading a song track-by-track #

Now that the SoundBox player is able to generate track-by-track buffers, we can use them to create individual buffer audio sources attached to individual gain nodes (a gain node is what we'll use to control volume for each track).

Previously, my song-loading code looked like this:

    let buffer = this.player.createAudioBuffer(Audio.ctx);
    this.songSource = Audio.ctx.createBufferSource();
    this.songSource.buffer = buffer;
    this.songSource.loop = true;
    this.songSource.connect(Audio.gain_);
    this.songSource.start();

We're going to change this to loop through each track ("channel", as SoundBox calls it), make an audio buffer and a gain node, and wire them up. After all of the buffers and nodes are created, then we'll pick a single time to start playing all the buffers simultaenously. (We don't start each buffer as we create it, because we don't know exactly how long these buffers will take to create -- even if each buffer only takes 40-50ms, the end result would be a subtle "staggering" of our different tracks that would make it sound out-of-sync).

    this.musicGainNodes = [];
    this.songSources = [];

    for (let i = 0; i < song.numChannels; i++) {
        let buffer = this.player.createAudioBuffer(Audio.ctx, i);
        let bufferSource = Audio.ctx.createBufferSource();

        let gainNode = Audio.ctx.createGain();
        gainNode.connect(Audio.gain_);
        this.musicGainNodes.push(gainNode);

        bufferSource.buffer = buffer;
        bufferSource.loop = true;
        bufferSource.connect(gainNode);
        this.songSources.push(bufferSource);
    }

    this.musicStartTime = Audio.ctx.currentTime + 0.1;

    for (let i = 0; i < song.numChannels; i++) {
        this.songSources[i].start(this.musicStartTime);
    }

There! Our song is playing in the background, just like it was before, but now we can interact with those individual gain nodes. For example:

// Example: mute first track of the song
this.musicGainNodes[0].gain.value = 0;

Adding some interactive tracks #

Time to jump into the actual tracker. I'm going to add two tracks: the first track will start when an enemy wave begins, and the second track will kick in the first time an enemy becomes visible on screen (due to where the enemies randomly spawn, the difference may be 1 second or it could be 10+ seconds, so the first track will hopefully help build up suspense).

To make things easy on myself, I've written out the same short series of notes in both tracks -- you can open it in SoundBox to see it for yourself. Tracks 1-5 make up my original song, and new tracks 6 and 7 are now a repeating series of notes (F# F E D#). Track 7 is the "wave" music (using the pipe hit instrument), and track 6 is the "combat" music (using the classic 8-bit instrument).

If I were to save this song and load it up in the game, all 7 tracks would immediately start playing, so we'll want to make some adjustments.

First, a couple of helpful constants:

export const TRACK_COMBAT = 5;
export const TRACK_WAVE = 6;

In the for loop where we create our audio buffers and gain nodes, we'll mute the two new tracks:

            if (i === TRACK_COMBAT || i === TRACK_WAVE) {
                gainNode.gain.value = 0;
            }

And now we need a way to start and stop our new tracks:

    startWave() {
        if (!this.trackWavePlaying) {
            this.musicGainNodes[TRACK_WAVE].gain.linearRampToValueAtTime(1, Audio.ctx.currentTime + 3);
            this.trackWavePlaying = true;
        }
    },

    startCombat() {
        if (!this.trackCombatPlaying) {
            this.musicGainNodes[TRACK_COMBAT].gain.linearRampToValueAtTime(1, Audio.ctx.currentTime + 2);
            this.trackCombatPlaying = true;
        }
    },

    stopWave() {
        this.musicGainNodes[TRACK_WAVE].gain.linearRampToValueAtTime(0, Audio.ctx.currentTime + 3);
        this.musicGainNodes[TRACK_COMBAT].gain.linearRampToValueAtTime(0, Audio.ctx.currentTime + 2);
        this.trackWavePlaying = this.trackCombatPlaying = false;
    }

Now all we need to do is hook up the appropriate function calls to the game logic that controls the wave and enemy behavior.

Try it out below!

(Music may take 3-4 seconds to start playing, please be patient.)

Aligning track start times #

If you play with the example above for long enough, you'll notice that it doesn't always sound very good when the new track kicks in. Yes, we're fading in the notes, but because we're fading in from a random point in the pattern, our ears get confused -- it might kick in in the middle of a note (this results in an unpleasant click as the track starts). Or, it might sound like a note, but it's the wrong note -- if the first notes we hear are "E D# F# F", it takes the ear another second or two to realize that it's a repeating series of decreasing notes. Instead of an exciting, suspense-building sound, it just momentarily confuses the player.

We can fix this if we make sure the new tracks only kick in when we want them to -- at the next beginning of our "natural pattern" (in my case, the 4 note run). We can use the value rowLen, which is available on the song exported from SoundBox, to calculate this.

The value rowLen represents the number of samples in an individual note (row) in the tracker. The songs SoundBox exports play at 44100hz, or 44100 samples per second, so dividing the rowLen into 44100 gives you how long an individual note takes to play, in fractions of a second.

Inside SoundBox this is related to the BPM (beats per minute) setting -- BPM represents how many quarter notes (i.e., every fourth note) plays in a given minute. Because my song is an ambient background track, I have an unusually low setting of 63BPM, which makes my song's rowLen an unusually high 10500.

Either value can be used to calculate the length of notes in your song.

From BPM:
60 seconds / (63 BPM * 4 notes per beat)   = 0.238 seconds

From Samples:
44100 samples per second / 10400 samples   = 0.238 seconds

Back when we changed up our player code, we saved the time the music started playing in the variable musicStartTime. We're going to use that now to help calculate how far into the sequence we currently are, and when the next one will start.

    startWave() {
        if (!this.trackWavePlaying) {
            // Calculate the best start time (aka: the F# note in our sequence)
            const sequenceLength = song.rowLen * 4 / 44100;
            const sequencePoint = (Audio.ctx.currentTime - this.musicStartTime) % sequenceLength;
            const startTime = Audio.ctx.currentTime - sequencePoint + sequenceLength;

            // Don't fade in, just cut to full volume for that F# note
            this.musicGainNodes[TRACK_WAVE].gain.setValueAtTime(1, startTime);
            this.trackWavePlaying = true;
        }
    }

Let's try it!

(Music may take 3-4 seconds to start playing, please be patient.)

Nice... now every time an enemy appears on screen, the combat music is going to start out with that strong F# note.

Going further #

This is where I ended my experiment, but there are other cool things you could do here.

Right now, all of my tracks are playing all the time in the background. You don't have to do it this way -- in theory we could have the wave track ready to play, but not started yet. Because tracks 1-5 are ambient, they don't have to line up pattern-to-pattern with Track 7 (the "wave") track -- we could get away with only lining up on note length. (You would still want to use the sequence calculation we did above when starting Track 6, the combat track).

This type of approach would work great for a game with enemy encounters -- think of Final Fantasy style combat. You could cut from a peaceful exploration song to a fight song, and as long as both are the same BPM and you calculate a note start time, you can do a dramatic instant cutover and it won't sound jarring.

Play with it and have fun!