Archive for May, 2012
If you have been following my blog, I have laid out a number of fairly obvious possibilities for the evolution of music-making. In order to fully take advantage of the possibilities, I believe there are two cross-company initiatives that would transform the marketplace for everyone:
1. Define a standard plug-in data model and API for parameter passing. Today every manufacture has their own data model and writes to specs like VST, AU, etc. This is great and works well, but all of these technologies involve local processing. Vienna has figured out how to remote VST and AU plugins, but that is just the beginning. What we need is a specification that allows processing to be outsourced to internet-based engines. Instrument models, incredibly rich reverb effects, and powerful synth engines that would overwhelm a desktop PC can be envisioned and created by existing instrument and effect designers. The problem is not the algorithms. The problem is the communications infrastructure and data model for passing the parameters. A group with representation from DAW manufacturers, plugin vendors, digital instrument makers (like Eigenharp), and sample vendors/modeling companies could produce a rich, extensible data model that could facilitate an on-line/offline render model and incorporate the kind of rich data than next generation musical controllers can produce in a post-MIDI world. The spec needs to be open, and not tied to any one vendor. The Internet Engineering Task Force and dozens of other high-tech standards bodies show how this works to benefit of all participants.
2. Create a standards-body work for virtual performance space. Like the plugin specification above, this would specify a Data model, interfaces, etc for a virtual concert space. As I lay out in my previous post on the future of music, there will be entirely digital performances in the future. What we need is a model that allows all the vendors to plug in seamlessly to a cloud-based concert hosting service. Everyone from mix engines, FX providers, lighting technicians, instrument models, etc need to collaborate to make a virtual concert happen in a virtual space. Provision must be made to use rich controller and body-suit information to create 3D performers, and avatars that can actually play instruments on virtual stages. Even 30 minutes of reflection on the inputs and outputs for a virtual concert experience point directly to the need for a standards-body to coordinate industry input and define interfaces that dozens of vendors can use to meet customer demand. We don’t need proprietary digital islands – that will delay progress by 5-10 years. Existing enterprise technology adoption illustrates this – closed development is a dead-end, and technology will route around road-blocks.
Both of these efforts are multi-year efforts that will require a lot of work. As the CPU and RAM vendors push forward, the timeline for broad distribution of the requisite horsepower and consumer devices will roughly coincide. A 3-5 year timeframe for the development of this technology is reasonable and most MI companies would stand to benefit tremendously. Even DSP-based companies like AVID or UAD could easily port their algorithms and control surfaces to this world.
Organization of these bodies will create a rising tide that lifts all boats within the industry, and vastly expand what artists can deliver.
Orchestral mockups are presently a LOT of work – as much work as actually composing the piece. On the one hand, this is reasonable – one person is trying to simulate 50-100 players. It stands to reason that this will be hard, and fraught with complexity. As processor power increases, and instrument models evolve, we should expect this area to be transformed. Much of the variation that has to be hand-entered today, should be performed via a “rich-data” controller like an Eigenharp Alpha or some other string/wind variant geared toward those kinds of players. These controllers output exponentially more information than MIDI could ever handle, but will provide the kind of expression data that makes an analog instrument so rewarding. Think about the possibility of feeding a violin model parameters like: bow pressure, speed, angle, distance from the bridge, string friction coefficient, string bending force, etc. Once this rich data is in a suitable DAW, it will be possible to use algorithmic means to vary this data, run it through different models, and truly make whole sections of related, but individual performers. This has the potential to revolutionize orchestral mockups and completely eliminate manual sample switching.
It would be ideal if rich and very expensive instrument models were accessible in the cloud – in two part form – smaller models to write with, and crazy rich models to render with. The history of all samples is that they get cheaper over time, no matter how real they seem initially. In the future, one or a handful of highly skilled players in each section will play rich-data controllers in real-time. Sample switching is not necessary since the models record all string/bow movement, pressure, bow speed, pluck point, fretting point, breath parameters, etc. All data goes to the cloud as inputs to the model. Algorithms can create interpolated players using the rich data to fill out the sections and avoid repeating sounds identically; different instrument models can be mixed as well to further the ensemble. Incredibly expensive reverb can put every player into a common space. This reverb can effectively be almost infinitely expensive computationally because of the power available in large internet data centers. Cloud0-based CPU complexes can render the sound and output individual tracks for mixing in 24 bit surround-sound, or provide direct output as a simple stereo wave from modeled microphones in the modeled room.
Whoever figures this out, will have most of commercial music beating a path to the door. If someone like the VSL folk figure out how to take their samples, apply modeling over the top and come up with a fast data interface (think Eigenharp level data feeds) in cooperation with a digital instrument maker, they will be an incredible force to reckon with. The SampleModeling stuff is already amazing. Put this in the cloud where computation is limitless for final render, and the local composer’s responsibility is just to get rich data off a digital controller.
First, allow me to thank you for the incredibly rich capabilities you already deliver. Being able to run 30-50 tracks off my MacBookPro is most excellent! This letter, however, concerns the future and how you might re-invent the DAW to take advantage of next gen computing horsepower.
Today, all audio plugins have to operate in real-time. This means that the average laptop drives the quality level for the whole industry since that is what most people use. More computationally expensive algorithms exist, but are not compatible with real-time playback on a laptop. We need the ability to “render” our final audio, just like the video folk. Video is massively more than a desktop PC can handle, so all previews are of lower quality, but are fine for editing, color correction, etc. The final render, however, runs at full resolution, full frame rate, and full color bit depth. In the music world we don’t have this option. The DAW company that figures this out will have a massive advantage. If the software is smart, it can tell when a full version vs a draft version will do and automatically render in the background what is needed. Video companies do this, and some even use GPU cards to accelerate algorithsm – another viable strategy.
What we need is an online/offline model. I can use existing plugins to work – they are real-time and have great sound. But, I want my plugins to have access to algorithms so powerful they would halt my computer for days. This kind of power is available in the cloud, but I can’t send data to the cloud in real-time; there is too much latency. What I need is for my plugins to be able to stream audio to the cloud in the background, process it using very expensive algorithms and return my processed audio to the track as a “freeze” track. Even as good as a UAD or Protools card is, more quality can be had – particularly for reverb and highly involved synthesis engines.
You could leave this up to the plugin vendors and simply make a way for them to “play a track” in the background through their plugin (a great use for all the cores in a modern multi-core machine). Or, you could make a torrent server that centralizes these requests, schedules them and maximizes the communication, keeping track of which tracks are edited and need to be updated. Or, you could allow plugin vendors to make a “render” algorithm that runs on the final bounce/mix-down. Because this doesn’t have to be real-time, more expensive processing can be used. Plugin vendors can get their real-time tools close enough to the render tools for engineers to have confidence that it will be “the same, only better”.
If you do this, your plugin vendors will love you. All software synth makers should be requesting this from you. This is how they can keep their best code in the cloud and not deal with copy-protection, etc. it also opens up the idea of an online plugin marketplace. They can give lower quality versions away and charge for the renders. Surely you don’t want Propellerheads to own this by themselves (they already have an online plugin marketplace)? Your users will also love you. This will let me access “the high end stuff” on a project by project basis. If I am just writing demos for my band, I may not care, but if I am polishing our album, I would definitely spend extra to get the last bit of great sound via some cloud-based plugins that are so expensive they don’t even fit in a 5k Bricasti box or a $3k Access-Virus chassis.
Dear algorithm magicians,
You work on the most CPU and memory intensive effect in the business. A desktop PC, even a fast one, is massively compromised for real-time reverb processing. The trick is that many of us don’t need real-time processing all the time. Whoever builds the most expensive, best sounding algorithm in the cloud stands to win huge business. What most of us need is a solid “draft quality” algorithm for working, and then access to an offline-final-rendering plugin of the highest quality. As an orchestral composer, the best option today is something like a Bricasti box, TC6000, Lexicon, or Vienna’s “MIR” (which consumes much of single PC). I don’t need that to write, or even mix. I need it for the final output. Someone needs to make a plugin that handles this. Rendering is OK for video, and it is OK for audio too. Let me make a better than fantastic mix by running my PC all night, or sending data to the cloud. Here’s the feature list:
1. A reverb plugin I insert in my DAW needs to deliver a competent base reverb. Think Logic’s Sound Designer or something similar. Not exquisite ear candy, but credible and competent – give this away. It is the “shopping cart” for your big algorithms. You don’t have to worry about piracy because your big algorithm never lives on their computer.
2. The plug is torrent enabled behind the scenes to handle the up/download to the cloud. The cloud uses massive parallel processing to crunch things at least as well as Bricasti, but why not 100 Bricasti’s? Cheap and on-demand. Your business scales with demand, you only pay for what you use. It’s a cash business.
3. When I am ready to bounce, I enable the reverb “sends” to go to the cloud, and play my piece. The data goes through the plugins, up to the cloud and comes back as tracks with reverb.
4. Mix these in to taste. Freeze tracks can preserve the render
5. If the DAW companies can be involved, then someone can make it so that the track contents constantly stream in the background up to the cloud. Returned tracks are “frozen”. Usually as a mix comes together, many tracks are not changed. If edits are done, background sending gets the changes rendered. Audio tracks for a 3-10 min song are small bandwidth in a broadband world.
Done. Mega-expensive sound accomplished. Easy to use. $1 per instance per song. Why buy a $5k box, when I just want to polish one to a dozen tracks? Make it so everyone can afford the best algorithm and win big. Be known as the best room modeler and make interfaces to all platforms. Brand the room algorithms, not hardware. Past algorithms will only be interesting to date a sound. Buying dedicated hardware is great for live shows, and constant daily use, but there is a whole market of folk who would happily pay a few bucks to access “killer reverb” for a special project. These folk are NEVER going to spend $2-5k for a high-end box . The folks who want a dedicated box are still going to buy one. I want a Bricasti for the studio, would use it every day, and still pay to render out if something better was available there! You can’t audio process a live show in the cloud, so you will still sell expensive reverb boxes with DSP in them. Network latency is too great. This is about market expansion and the long, long tail of the whole internet.
Ultimately having the most expensive algorithm will win as CPU power becomes ever more free. In a few years, music made on cell phones will be uploaded to the cloud for processing. This is the early footprint – build the brand now for customers that will come online 5 years from now when cell phones are more powerful than current loaded Mac Pros and have better broadband than a cable modem today.