Jason Page and Michael Kelly spoke to GDC attendees about creating next-generation audio on the PS3. In a surprising discussion, Sony's Jason Page admitted that the future of video game music may be based in technology that's considered antiquated by most: MIDI. Many developers have been moving away from MIDI to pre-recorded, orchestrated background music, but in the process, the interactivity of game music has been sacrificed.
The incredible processing power of the next-generation consoles has changed the rules of MIDI: gone are the days where MIDI sounds like R2-D2 singing. MIDI samples used by consoles can be just as good, if not better, than the samples used on dedicated synthesizers. Because MIDI loads in real-time, it retains the interactivity that composers like Koji Kondo would need, and it would allow games to load more quickly.
MIDI in the next-generation could potentially retain the same fidelity that an orchestrated score might have. With the increasing need for interactive 5.1 and 7.1 music and audio in games, the sound of "chip music" may change quite drastically in this new console generation.
[Update 1: Corrected name source: Jason Page is from Sony, not Dolby.]
{ Mar 8th 2007 at 4:24PM } 
Reader Comments (Page 1 of 1)
Player1 @ Mar 8th 2007 4:31PM
Ironically enough, I would imagine that a lot of that pre-recorded background music is originally produced using midi.
wwwiii @ Mar 8th 2007 4:35PM
I've always thought that MIDI should stay in the mainstream, it allows for a lot more than standard audio does. Anyone remember the IMUSE engine in X-wing and Tie Fighter?
Agent MOO @ Mar 8th 2007 4:36PM
It does make sense as a way of removing redundant data. Why load 3 hours of pre-recorded music when you could do a minute worth of samples driven by midi files that have everything you need for dynamic music?
The only limitation I see is that techno/hip hop and other genres won't make their transition into this structure with certain effects (I.E. record scratch) intact.
arrakisman @ Mar 8th 2007 4:37PM
Back in the DOS day a AWE 32/64 was unbeatable. Games like X-Wing or Tie-Fighter had better music using MIDI than when they switched to the recorded stuff on XvT. I also had fond memories of Arena and Daggerfall music. For whatever reason I find that a lot of the newer game's sound tracks are not quite as memarable as the older, simpler stuff.
J.Goodwin @ Mar 8th 2007 4:38PM
How is this different from DirectMusic?
Microsoft's been using this for years.
Zertoss @ Mar 8th 2007 4:46PM
"The only limitation I see is that techno/hip hop and other genres won't make their transition into this structure with certain effects (I.E. record scratch) intact."
Those effects could make the transition actually. MIDI can be programmed with special samples, or it can even read new samples from a file (like Soundfonts).
I never would've imagined that MIDI, the past of gaming, would also be the future. But the argument here is certainly compelling. The main problem I have with it is the loss of quality. By that I mean that the instruments sound synthesized in MIDIs (since they are), and that really bugs me. If they could be made to sound as close to their real counterparts as possible, then I'd be all for shifting to MIDI.
ShineDog @ Mar 8th 2007 5:07PM
Not so sure, lots of games have been achieving dynamic soundtracks through channel mixing, and doing it very well.
its just much harder and rarer.
Infil @ Mar 8th 2007 5:10PM
Do you remember IGN's review of Twilight Princess? They slam the audio for being MIDI and not orchestrated, even though the music in that game was absolutely incredible as far as I'm concerned. I was extremely ticked off; you can't just write off a soundtrack because it's not orchestrated. The catchiest and most memorable videogame tunes of all time are ALL MIDI. Twilight Princess did a great job of creating a moving atmosphere with its soundtrack, despite not boasting 110 people holding an instrument.
Sometimes MIDI is just better. Orchestrated stuff is nice for some things, and not so nice for others.
Crono @ Mar 8th 2007 4:59PM
Thus nintendo looks simultaneously backwards and ahead of their time by using MIDI in twilight princess.
bill pullman @ Mar 8th 2007 5:05PM
"The main problem I have with it is the loss of quality. By that I mean that the instruments sound synthesized in MIDIs (since they are), and that really bugs me. If they could be made to sound as close to their real counterparts as possible, then I'd be all for shifting to MIDI."
actually, it comes down to which samples are being used. clearly we're not talking about synthesizing the waveforms themselves, but using sample sets (soundfonts, as mentioned earlier) of actual multisampled instruments. this would be similar to a high-end keyboard/workstation or it's PC equivalent.
bad synthesizer patches have been a thing of the past for several years now. in fact, a lot of commercial recordings (music/movies/etc) that we enjoy today are often created from samples of prerecorded instruments, and not live instruments themselves.
still, with audio compression being as widespread and prevalent as it is, i don't really see a big push for MIDI in gaming. the "dynamic score" concept might make it useful, but the fact is that many games today just use commercial tracks for their soundtrack anyway. also consider that a dynamic score may force the creator to set aside any specific flow or direction in a song in order to drop in specific pieces at key moments.
Rob Price @ Mar 9th 2007 6:07AM
The main problem is that while a lot of the compositions that are used are MIDI based in the first place, they come from expensive libraries that the developers would have to license separately for usage in the game - as the copyright belongs to the owners of the sample libraries.
Poisoned Al @ Mar 8th 2007 5:17PM
Gah, people forget that all MIDI is, is a set of instructions like sheet music for hardware. It is NOT a audio format like .wav or .mp3. MIDI sounds as good as the hardware that plays it and these days it's petty damn good. Sure your 360 can't compete with a full real life orchestra in terms of quality, but what it can do is change on the fly to match the action on screen. You can only switch between pre-recored stuff, but with midi music you can up the tempo, change the pitch and add more cowbell smoothly.
Megaqwerty @ Mar 8th 2007 5:19PM
Multi-gigabyte soundsets would be a lovely way to flesh out those Blu-ray DVD's...
But, yeah, go to Native Instrument's site and check out the demos for Kontakt 4 or for Bandstand. It's all MIDI and most of it sounds insane. Granted, the soundsets are HUGE, but the elasticity that MIDI provides is also quite big.
Now, pardon me while I listen to E1M1.
Vercin @ Mar 8th 2007 5:23PM
To Megaqwerty:
I was just going to direct interested readers to a demo page of that sort of thing, but you beat me to it. I work with Garritan Virtual Orchestra, and it's pretty slick what you can do with MIDI these days.
Rubang B @ Mar 8th 2007 5:29PM
Anybody who slams audio just for being MIDI has no idea what MIDI is and how much it can do.
REUYL @ Mar 8th 2007 5:39PM
I should get back into MIDI I guess... I've got a Roland Fantom-X8 that'll do the job well.
bill pullman @ Mar 8th 2007 5:53PM
aww.... and here i am on my lowly Fantom-S. still, it's more machine than i need.
AirIntake @ Mar 8th 2007 5:58PM
Or you can use prerecorded music and let the game splice it together differently depending on what's happening in the game, thus removing the advantage of MIDI. Oh wait, they already do that.
I mean really, do you want to listen to a MIDI guitar solo? Would a MIDI of "Layla" (no matter how good the sound samples) sound as good as Eric Clapton playing it? There are many things that instruments can do that you just can't translate to sheet music.
theLoneYoshi @ Mar 8th 2007 6:01PM
I agree to what Infil (#10) said about Twilight Princess's soundtrack. To tell you the truth, if someone told me that the soundtrack was orchestrated, I'd believe it. MIDI has advanced as far as I can tell (compare Twilight Princess's Hyrule Field with Ocarina of Time's Hyrule Field, and obviously you can tell which one sounds better), and I doubt an orchestra can make it sound significantly better.
As for MIDI being the future... I'll say it really depends on the game. Games like the Tony Hawk franchise use licensed music because it goes with the type of game. I know I'd rather have Hip Hop and Rap in the background of NBA Street instead of something MIDI. The Mario's and Zelda's can live off of the MIDI Soundtracks because their soundtracks were meant for the type of games that they are. And I like Final Fantasy games with orchestrated pieces.
Rookie @ Mar 8th 2007 6:12PM
MIDI files are only as good as the sound samples you have and the hardware. If you have really good samples and hardware, then MIDI can sound almost as good, if not as good as orchestrated music.
jabbertrack @ Mar 8th 2007 6:27PM
People mention Twilight Princess... and rightly so.
It's a bit of a shame that one of the biggest games of the year sounded like it was one of the biggest games of 10 years ago.
Maybe I'm just stubborn but having an "epic soundtrack" sound like it was played on a $99 Casio keyboard really pulls me out of the experience.
... and NO... just because Super Mario Bros. used beeps and boops for really great music doesn't mean all future games get a free pass to skimp on sound quality which is makes for an AWESOME experience when done well
kevinski @ Mar 8th 2007 7:12PM
I've had a renewed respect for MIDI ever since I discovered that much of the music in Illbleed (my favorite Dreamcast game) is composed as such. It sounds absolutely incredible, and most people probably wouldn't know the difference between MIDI and pre-recorded music.
Thomas @ Mar 8th 2007 8:35PM
The first time I heard an orchestrated soundtrack, I was absolutely amazed. Not by the "quality" of the sound, but by the fact that they would waste memory by storing a full soundtrack rather than generating it in-game! I've gotten used to the idea now (I even bought a copy of the Shadow of the Colossus soundtrack!). But not *so* used to it that I consider it the only way, or the "modern" way.
Of course we should continue to use MIDI. We should have a variety of tools on our belts, using the right tool for the right job. It's just as silly to use recorded audio in a Zelda game as it is to use MIDI in Guitar Hero. But there's a caveat.
Wavetable synthesis (using samples) has come a long way, and a lot of research is still going on to digitally synthesize realistic instruments (including the human voice). But there seems to be the equivalent of an "uncanny valley" that has yet to be crossed (and can it ever?). The human ear is very sensitive, and the brain is well-trained in natural sounds, and together they tend to be very discriminatory when an artificial sound is presented, no matter how complex the model was that generated it.
And so there's a difference between games like Zelda: Twilight Princess and Shadow of the Colossus. Synthesized sounds are perfect for the kind of music in Zelda. Shadow of the Colossus has a loud and epic soundtrack that can't easily be replicated by the same technique. It's a design decision. The trade-off, of course, is interactivity. I thought everyone knew this.
Dave Silva @ Mar 8th 2007 8:50PM
Everyone's saying things about MIDI, but we've had a perfectly good format for years: MOD/IT files.
Thomas @ Mar 8th 2007 9:05PM
MOD, IT, etc. are file formats that store short waveform samples for playback at various pitches to create melody, rhythm, etc. They have to be interpreted and rendered in a particular way, and I don't think they lend themselves to on-the-fly rearrangements.
On the other hand, as someone said above, MIDI is not a file format but a communication protocol. It is independent of the technology that generates the sounds (whether it's samples of recorded instruments, physical modeling synthesis, or old-school FM synthesis).
Module files have their benefits. They're great for distributing over the Internet since they sound the same on whatever device they're played on. But for game consoles this isn't an issue, because a PS2 running a particular game will generate the same sounds as any other PS2 running the same game. And all things equal, a good synthesizer can render a MIDI sequence with much better fidelity than a MOD file.
Fullman @ Mar 8th 2007 10:09PM
I don't agree with this at all. Unless samples got significantly better and effects like reverb, delay, chorus and flange were introduced in a way where it didn't sound like 1996, bringing MIDI back in full-force would be a disaster.
Dynamic/micro-scoring has been around for a while now, and games like 'Tomb Raider: Legend' show just how interactive music in games can be. It doesn't require extra loading time because the songs are loaded/streamed in segments, and those segments are then switched out depending on what's happening in the game.
But don't worry, we're still in an age where Nintendo insists HD/advanced graphics isn't necessary to enhance the gameplay. That and there isn't a shortage of Wii's. Riiiiiiiight.
mocax @ Mar 9th 2007 6:43AM
Problem with PC MIDI, is that different manufacturers use different sound samples. There's no standard sound even in General Midi.
It's a composer's nightmare.
Things will be different with consoles. You get the same hardware everywhere. Makes things a little easier to manage.
t_m @ Mar 9th 2007 6:50AM
iMuse is the ultimate example of this (as others have said). When they moved from iMuse (in tie fighter) to cd soundtracks in XWvTF it killed the atmosphere...
ssuk @ Mar 9th 2007 8:43AM
Chipmusic never died, Nintendo has it in the DS, okay, it doesn't sound as good as an orchestrated, pre-rendered music file, but there's also the option for that, ADX runs perfectly on the DS and uses a fraction of the processing power, games like Resident Evil Deadly Silence use it. But games like Sonic Rush on the DS also shown something, that the DS soundchip can work in harmony with custom samples to create wonderful music, but we already knew this from the days of Sonic the Hedgehog 3 back on the Megadrive where custom samples were all over Ice Cap zone, Launch Base and the mid-boss battle tunes.
A new form of sound chip, so long it doesn't sound like blips and blops and could be used to create realistic music would be great, along with the custom samples it would be a fantastic thing to do, quality music with little file size... Though, then again with games soon reaching 50GB maybe (with this Tripple layer HD-DVD) what's the point, they're making a soundchip which, in reality, will be used for a TINY percentage of games.
Joshua @ Mar 9th 2007 1:54PM
MIDI's are great. They allow for more creatively and pricesly the sounds a composer is looking for.
uppertoe @ Mar 9th 2007 8:26PM
MIDI can sound pretty nice when used well, but can never quite capture the subtleties of music that a musician can. Sure, it can play the notes and many samples sound almost exactly like the real thing, but as any musician will tell you, there is more to music than what is written on the score. Compare any high quality midi with an orchestra, and you'll know what I mean.
Sigma 7 @ Mar 9th 2007 8:59PM
The main problem with MIDIs is that they are different depending on the player used. (e.g. The midi used in Descent may sound great in-game, but not as well in WMP or Winamp.)
I tended to like the Impulse Tracker files used within the Unreal engine - while the files are larger, it will sound the same regardless of where it's played (although a player for it isn't as common.)
JPage @ Mar 9th 2007 9:29PM
Hi all
Two things:
1) Dolby?! I work for Sony! Must pay more attention. See me.. ;-)
2) The point I was making is that the PS3 is powerful enough so that the user could play the original MIDI tracks, along with the softsynths / samples and DSP effects which they would have originally created the track on, say, Cubase in the first place. The word MIDI could therefore be replaced with "Tracker" or whatever. So, the point isn't that MIDI could be used as a format, but that all the bells and whistles (and VST's) which go into a track could also be processed in real time - and still leave more than enough power to run the game too..
As the first post says:
"Ironically enough, I would imagine that a lot of that pre-recorded background music is originally produced using midi."
Indeed, thats my point. Just now you could process it in real time and allow the user to jump around the track, retaining any reverb trail-off's and so on.
Cheers
Jason
Jompa @ Mar 11th 2007 5:06PM
Excellent thinking, Jason! I can envision a complex environment - like DirectMusic combined with all the DSP (like any VST plug, instrument or effect) and other technologies you'd need for a particular score. Here you, as a composer, could choose the sound sources you want at the same time as controlling the level of musical adaptivity preferred. This could include fully synthesized sounds (using any kind of today’s available synthesis technologies) - or just triggering a complete recording of Beethoven's 5th with the London Philharmonics with a single MIDI note-on command. Or layering Clapton's guitar solo on top of whatever background you choose.
The level of musical adaptivity would thus be fully decided by the composer and/or producer by using MIDI in whatever capacity you want. MIDI (as many of you have already stated, but some don't yet seem to understand) does not in itself put any restrictions on sound quality or performance 'authenticity'.
I started doing research in this area many years ago and have come to pretty much the same conclusions as Jason. Now we're finally approaching the hardware necessary for (again) making MIDI (along with other technologies) a working tool for achieving adaptive game music of the highest artistic quality! Looking forward to some exciting years!
iocomposer @ Mar 11th 2007 3:17PM
Jason Page is with Sony, not Dolby. Get it right.
Daniel Joseph @ May 14th 2007 6:37PM
Just to get this out there. MIDI orchestra synthesis will be a part of game soundtracks for as long as it remains easier to do than hire an orchestra and record it live. This means forever.
Also, I'm planning a MIDI sequencing of all or part of the Shadow of the Colossus OST. Just FYI ;)