Creating levels for video game franchise Dance Dance Revolution is a time-consuming task. The step charts used by the game are essentially choreographic scores, instructing players where to place their feet in time with the music. Usually they’re written by hand, either by the game’s developers or fans using open source DDR port StepMania. Now, though, computer engineers have created a quicker way to generate step charts for any song — using the power of neural networks.
In a paper published this week (with the quite brilliant title Dance Dance Convolution), a trio of researchers from the University of California describe training a neural network to generate new step charts. Neural networks study data to analyze patterns and then create similar-looking outputs, and in this case, there was an abundant source of data in the form of fan-written step charts.
One of the study’s co-authors (and a longtime DDR fan) Chris Donahue told The Outline: “It sort of dawned on me one day that I had somewhere buried deep in my hard drive, gigabytes and gigabytes of data from this game StepMania, from a folder I’d been transferring from computer to computer since I was a teenager.” The researchers used two main datasets from different choreographers, with the total training data spanning 35 hours of annotated music and more than 350,000 steps.
/cdn0.vox-cdn.com/uploads/chorus_asset/file/8215301/ddr.jpg)
The trained neural net generates new step charts using a series of simple actions. First, songs are converted into spectrograms — visual representations of audio frequencies, which the neural network uses to identify features like pitch and rhythm. Next, a step placement algorithm slices the song into 10 millisecond samples, and decides whether or not any single sample could contain a step based on the presence (or absence) of relevant audio features. Then, a step selection algorithm maps these choices out using different moves, creating a fully-fledged DDR step chart.
You can see the sort of choreography the network comes up with below. On the left is a step chart written by a human; on the right, a computer-generated version for the same song.
The results are perfectly human-playable, but, as with many creative forays by artificial intelligence, professionals can still tell the difference. Speaking to The Register, step chart creator Fraxtil, who made many of the charts used to train the neural network, said “it’s pretty easy to tell that its output is synthetic.”
“There’s a lot of creativity involved in step charting, mainly selective use of repetition and contrast, that the AI either can’t learn or can’t apply effectively,” said Fraxtil. But, he added, of all the attempts he’s seen to auto-generate step charts, this one was by far “the most successful iteration.” Now if we could only create a robot that can perform DDR moves as well as humans.
