Procedural rhythm game based on audio analysis

Procedural Rhythm Game

By Laurens Cnossen – 500821318

Goal

The goal of this project is to create a rhythm game that can procedurally create levels for itself based on the audio you give it.

Index

  1. Separating frequencies
  2. Scriptable Template
  3. Rhythm matching
  4. Rhythm matching 2.0
  5. Visuals
  6. Global values
  7. Highscore
  8. Delay problems
  9. Result
  10. Continuing
  11. Sources

1. Separating frequencies

The first thing I wanted to do was see if I can distinguish frequencies based on if they’re high or low. I started by looking at the Unity documentation about spectrum data. There appears to be a method that can be called that automatically analyzes spectrum data (Unity, n.d.).

This method returns 3 values:

Result of the example code given in the Unity documentation.
  • Samples – The array to populate with audio samples. These are the values given to frequencies based on the audio given to the AudioSource component.
  • Channel – Which AudioSource to analyze (only needed when you use more than one AudioSource).
  • FFT Window – The spectrum analysis algorithm used to output the sample data. examples of these are Rectangular, Blackman, Hanning, Hamming and more.
 if (SpectrumAnalysis.samples[i] > DetectionThreshold)
                    {
                        _cubeList[i].GetComponent<MeshRenderer>().material.color = Color.red;
                    }
                    else if (SpectrumAnalysis.samples[i] > DetectionThreshold / 2)
                    {
                        _cubeList[i].GetComponent<MeshRenderer>().material.color = Color.yellow;
                    }
                    else
                    {
                        _cubeList[i].GetComponent<MeshRenderer>().material.color = Color.grey;
                    }

I used a tutorial to visualize the spectrumdata I found earlier to check what range of data I was getting (Peer play, 2016). Based on the amplitude on a given sample index, I made a script that measures how well a frequency is detected on that given audiosample. In the screenshot above, everything that is red has a value over the threshold, which in this case is .03. Everything that is yellow is half of the threshold. Based on inputting drum sounds like kicks, hats, snares, etc. I can measure at what sample[i] they have the highest amplitude. For example: snares seem to peak around i⋍230 range. Below is another representation of how only samples above a certain amplitude threshold get picked up (red).

Visual representation of checking amplitude threshold

2. Scriptable Template

To create playable songs with the spectrumdata, I created a template creator that can send rhyhthmpoints to a scriptable that can be replayed whenever you want.

        private void SetTimers()
        {
            timer += Time.deltaTime;
            if (TimingCooldown > 0)
            {
                TimingCooldown -= Time.deltaTime;
            }
        }
private void SendToScriptable()
        {
            if (overrideTemplate)
            {
                for (var i = (int) analysisRange.x; i < (int) analysisRange.y; i++)
                {
                    if (!(TimingCooldown <= 0)) continue;
                    if (!(SpectrumAnalysis.instance.samples[i] >= detectionThreshold)) continue;
                    TimingCooldown = secondsBetweenAdding;
                    newTemplateList.Add(timer);
                    songTemplate.peakPoints = newTemplateList;
                }
            }
        }

In this script, I set a range of frequency bands (samples) to check (analysisRange.XY) in a for-loop. At the same time a timer is running, every frequency band within the loop that peaks above a certain amplitude threshold the current time gets added to a list. That way, I end up with a list full of time floats. finally, Because a peak is usually longer than 0.01 second, I also added a cooldown (secondsBetweenAdding), so the list doesn’t contain a bunch of values that are within milliseconds of each other that cant be matched.

template to scriptable creator

3. Rhythm matching 1.0

Now that I have the data, I need to actually be able to play it.

     private float DelayCalculation()
        {
            var delay = timer - rhythmpoint;
            return delay;
        }

To do this, I compare the data saved in the scriptable to the current timer when you press the spacebar. Based on that delay, you get a certain amount of points. In this case lower score is better because the closer the timer is to the rhythmpoint the better.

if (timer >= rhythmPoints[rhythmPointIndex]) //when timer exceeds index go next
            {
                _scoringEnabled = true;
                rhythmPointIndex++;
            }

To make sure the player matches the right rhythmpoint, it automatically goes to the next index whenever the timer exceeds the rhythmpoint.

4. Rhythm matching 2.0

In rhythm matching 2.0 I rewrote the previous score calculation from rhythm Matching 1.0 as it had a big flaw, it would hand out absolute points instead of a percentage of how close you are to matching the rhythmpoint perfectly. This means that if there are 10 seconds between points you could get a lot more points (10 points max, but lower is better) than if there were only 2 seconds between points (2 points max). Now the score is a percentage of the maximum available score. The percentage is how close you are to the timing of the current beat compared to how close you are to the previous beat.

        private void CalculateScore()
        {
            var calculatedScore = rhythmPointIndex switch
            {
                > 0 => GlobalValues.BaseScore - RhythmPointIndexDifference() * GlobalValues.BaseScore,
                _ => GlobalValues.BaseScore - (rhythmPoints[rhythmPointIndex] - timer) /
                    rhythmPoints[rhythmPointIndex] * GlobalValues.BaseScore
            };

            _score += (int) calculatedScore;
            delaySnapText.text = $"+ {RoundToInt(calculatedScore).ToString()}";
        }

In this case, the closer you are matching the rhythmpoint perfectly, the closer you are to getting 500 points, meaning that more points is now actually what you want (as it should be). now you dont have to worry much time is inbetween two rhythmpoints.

Because I need to check the difference between the timer and the rhythmpoints a lot, I also decided to create a method that just checks that difference, instead of checking it in every method that uses it.

       public float RhythmPointIndexDifference()
        {
            float indexDiff;
            if (rhythmPointIndex > 1)
            {
                indexDiff = (rhythmPoints[rhythmPointIndex] - timer) /
                            (rhythmPoints[rhythmPointIndex] - rhythmPoints[rhythmPointIndex - 1]);
            }
            else
            {
                indexDiff = (rhythmPoints[rhythmPointIndex] - timer) / rhythmPoints[rhythmPointIndex];
            }

            return indexDiff;
        }

To calculate this difference, I simply calculate: part divided by whole.

Because the calculation is being made between the current and the previous rhyhtmpoint I also needed a slightly different calculation for i=0 because otherwise it would create an OOB error.

Debug visuals

Whilst testing this system I made some debug visuals: the top text number the active timer going up, the left number is the next time/beat you need to match, which is based on the earlier created scriptable template that contains the timing points. Finally, the right number is the score, which goes up when pressing spacebar at the appropriate time.

5. Visuals

I wanted to finally work on my visuals, because right now it’s just text that tells my if what I’m doing is working. The first thing I wanted to create visuals for is the actual timing.

Timing visuals
        private void RhythmVisualizer()
        {
            foreach (var point in rhythmPoints)
            {
                var newCube = Instantiate(cube, cubeContainer);
                newCube.transform.localPosition = new Vector3(newCube.transform.localPosition.x, point);
                _cubeList.Add(newCube);
            }

            cubeContainer.localPosition = new Vector3(cubeContainer.localPosition.x, rhythmPoints[^1]);
        }

To do this, I created a rectangle for every datapoint and set it to the y-axis. Then I put all of them in a parent container and decrease the container y-value by the timer. This way, the rectangles perfectly line up with the middle of the screen (y=0) when they need to be hit.

Final looks

Next up I added more post-processing, in this case a moving background based on the particle system, some scanlines (Michal_, 2014), chromatic aberration and bloom. I also use particles every time you press spacebar, the color, and amount is decided by how close to the rhythmpoint you are.

All the post-processing that was added

6. Global Values

Because a lot of the values I use for visuals and more are used multiple times in different classes, I made a static GlobalValues class that keeps track of these variables including, colors

public static class GlobalValues
    {
        public static readonly string
            VeryEarly = "Very Early",
            Early = "Early",
            Good = "Good",
            Great = "Great",
            Perfect = "Perfect";

        public static readonly Color
            ColorVeryEarly = Color.red,
            ColorEarly = new Color32(255, 100, 0, 255),
            ColorGood = Color.yellow,
            ColorGreat = Color.cyan,
            ColorPerfect = Color.green;

        public const float
            ThresholdVeryEarly = .99f,
            ThresholdEarly = .75f,
            ThresholdGood = .5f,
            ThresholdGreat = .25f,
            ThresholdPerfect = .1f;

        public const int BaseScore = 500;
    }

Resulting in most of the code controlling the visuals looking close to a variation of this:

switch (_rhythmController.RhythmPointIndexDifference())
            {
                case < GlobalValues.ThresholdPerfect:
                    _particleSystemMain.startColor = GlobalValues.ColorPerfect;
                    _particleSystemEmission.rateOverTime = GlobalValues.BaseScore;

                    delayHint.text = GlobalValues.Perfect;
                    delayHint.color = GlobalValues.ColorPerfect;
                    _rhythmController.delaySnapText.color = GlobalValues.ColorPerfect;

                    cListMat.SetColor(EmissionColor, GlobalValues.ColorPerfect);
                    break;

7. Highscore

if (rhythmPointIndex == rhythmPoints.Count && songTemplate.highScore < _score)
{
    songTemplate.highScore = _score;
    highScore.text = _score.ToString();
}

I added was a simple highscore that gets saved to the scriptable whenever the current score is higher than the saved highscore.

8. Delay problems

        private void AdjustPointsForDelay()
        {
            foreach (var point in songTemplate.peakPoints)
            {
                rhythmPoints.Add(point + songTemplate.delay);
            }
        }

To create a scriptable template, I have to run template creator script with a song running realtime, this means that creating a template is prone to getting delays based on performance. To fix this, I added a delay variable to the scriptable that gets added to every index value to compensate for the performance delay. I obviously built this program for my laptop, this does mean that depending on your pc or if you run the program in playmode or in build, there is another delay difference for which I currently do not have dynamic a solution.

Final content of the scriptable

9. Result

Final Gameplay

I created a fully playable, rhythm game that can procedurally create its own levels and store them for replayability.

10. Continuing

If I were to continue with this project, I would implement a streak scoremultiplier that goes up when you match multiple points in a row. I would also look for a way to make the game less reliant on pc performance. Right now both the template creation process and just playing the game are reliant on pc performance, this means that depending on how fast your pc is you could have more or less delay, resulting in more or less points, during gameplay. Finally, I would want to add a second template to match, so you can play with two hands, both matching the rhythm based on different frequencies.

11. Sources

  1. Unity. (n.d.). Unity – Scripting API: AudioSource.GetSpectrumData. Unity Documentation. Retrieved November 4, 2022, from https://docs.unity3d.com/ScriptReference/AudioSource.GetSpectrumData.html
  2. Peer play. (2016, September 17). Audio Visualization – Unity/C# Tutorial [Part 1 – FFT/Spectrum Theory] [Video]. YouTube. https://www.youtube.com/watch?v=4Av788P9stk&feature=youtu.be
  3. Michal_. (2014). Need some help writing a simple scanlines shader: Unity Forum. https://forum.unity.com/threads/need-some-help-writing-a-simple-scanlines-shader.375868/

Posted

in

by

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *