What Happens When AI Learns From 40 Years of Your Own Music?

I have spent most of my life making music, and like a lot of people who have been doing this for decades, I have accumulated a pretty substantial archive. In my case, it goes all the way back to 1984.

Some of those recordings started on analog multitrack tape. Then came ADAT, hard disk recording, Pro Tools and eventually the completely digital recording environment we have today. Fortunately, I kept a lot of it. Over the years, the older tape and ADAT recordings were transferred and digitized. Add those to everything recorded digitally since then and I ended up with more than 500 original compositions, many with the individual tracks still intact.

About two years ago, I started looking at that archive differently. There was already a lot of discussion about AI music and the massive amounts of music these systems were learning from. That made me wonder what would happen if I approached the idea from the opposite direction. Instead of asking AI to learn from millions of songs, what if I could build a private machine learning system around my own musical history?

Not just the finished recordings, but everything behind them. Drums, bass, guitars, keyboards, vocals, background vocals, MIDI, samples, alternate takes, arrangements, mixes and masters. More than 40 years of making records.

That question turned into a two-year experiment.

The first part really wasn't about AI at all. Before a machine could learn anything from the archive, I had to organize it into something it could understand. Songs were cataloged, individual tracks identified and information such as tempo, key, time signature, year, instrumentation and recording format added wherever possible. Song structures were mapped into intros, verses, choruses, bridges, solos and outros. Lyrics could be connected to vocal performances, MIDI preserved where it existed and instruments and voices identified separately.

Suddenly I didn't just have hundreds of recording sessions sitting on hard drives. I had a dataset.

Because those recordings stretched across more than four decades, the dataset also had a timeline. It included music recorded on analog tape in the 1980s, recordings from the ADAT years and projects created entirely inside modern digital studios. That meant the system wasn't just looking at songs. It could begin identifying relationships between songwriting, arrangement, performance, sound and production across different periods of my own musical history.

One thing I learned pretty quickly was that trying to make one giant AI system do everything wasn't necessarily the best approach. Music is made up of different relationships. There is song structure, harmony, melody, rhythm, bass movement, instrumentation, arrangement, performance and production. Breaking those elements apart made much more sense.

The multitrack recordings became incredibly valuable because a finished stereo master only tells you what the final record sounds like. The individual tracks tell you how it got there. You can see when the bass enters, what the drums are doing underneath a chorus, when another guitar appears, how background vocals are used and how an arrangement grows or pulls back throughout a song.

MIDI was equally valuable because it already describes music as information. Notes, timing, velocity, duration and rhythm are there without having to extract them from a finished recording. The goal wasn't to teach a machine to copy one of my old songs. It was to see whether machine learning could recognize the musical relationships buried across hundreds of them.

Keeping the human at the center became probably the most important part of the experiment. I never wanted to build something where you type "write me a hit song" and wait for a finished recording to appear. Songs don't always begin the same way. Sometimes I start with a lyric. Sometimes it's a melody. Sometimes it's a chord progression. Sometimes it's just a title or an idea that refuses to leave me alone.

So the system needed to work the same way.

I could provide original lyrics, sing or play a melody, enter a chord structure or provide all three. If I already had the lyrics, melody and chords, I wasn't asking AI to write my song. I was asking it to help me develop the record.

That distinction became fundamental to the project. If I supplied a melody, I could keep that melody. If I supplied the chords, I could keep the chords. If I wrote the lyrics, I controlled the words. But I could also ask the system for ideas. What happens if we change the chord under this line? What if the bass moves differently going into the chorus? What if we strip the first verse down and wait until the chorus to bring in the full band?

Now AI wasn't replacing the creative decision. It was giving me another way to explore it.

Another important breakthrough came when I stopped thinking of composition and sound as the same problem. There are really two questions: What should be played, and what should it sound like?

My archive contained decades of instruments, drums, amplifiers, effects, samples, rooms, voices and performances. Those sounds could become part of the musical vocabulary available to the system, while machine learning worked with the relationships between lyrics, chords, melody, rhythm and arrangement.

There was one part of the process I eventually decided I didn't need to reinvent: final generative audio.

Producing high-quality generative audio, especially complete performances with realistic instruments and vocals, requires significant computing power. I could have invested considerably more time, money and hardware trying to build that technology myself, but that wasn't really the problem I was trying to solve. My interest was everything happening before the final render.

So I connected the system to a third-party generative audio API. The external engine could handle the computationally expensive job of generating the finished audio, while my system concentrated on developing and preparing the musical direction being sent to it within the controls the external API supported.

That distinction is important. The third-party system isn't my machine learning model, and I make no claim about what material that external model may have originally been trained on. My private system is the part built around my own archive.

The easiest way I can explain it is this:

The human creates the song. My system develops the production. The external engine renders the audio.

A new project might begin with lyrics, a melody and a chord progression. I could add some direction: 92 BPM, G major, start with acoustic guitar, keep the first verse intimate, bring the full band into the chorus and pull it back again for the second verse.

My system could take those human decisions, combine them with what it had learned from my archive and develop the arrangement and production information. That information could then be passed to the external audio engine for rendering.

The basic process became:

Human Idea → Lyrics / Melody / Chords → Private ML System → Arrangement / Production Direction → Generative Audio API → Finished Recording

But I didn't want the finished recording to necessarily be the end. I wanted to be able to go back, change the arrangement, try another bass approach, replace an instrument, rewrite the second verse, modify the melody or try another production direction and render it again.

I didn't want AI to hand me something and say, "Here's your song." I wanted a creative process.

That makes the system much closer to an intelligent production environment than an automatic song generator.

None of this happened overnight. It took nearly two years of organizing recordings, preparing data, experimenting with machine learning, testing ideas, listening, making mistakes and changing direction. Somewhere along the way, something happened that I hadn't really expected. I started discovering things about my own music.

How did my chord choices change over the years? How did I approach choruses in 1987 compared with 2007? How did my arrangements change? How did moving from analog tape to ADAT and eventually Pro Tools affect the records I was making? Were there melodic or rhythmic ideas I had unknowingly returned to over and over again?

Things that had been scattered across tapes, hard drives and recording sessions for decades could suddenly be looked at together.

In a strange way, I had turned my recording archive into a musical memory. And now that memory could become part of creating something new.

The funny thing is that none of this was remotely on my mind when I started recording those songs back in 1984. We were musicians standing in a studio playing into microphones connected to a tape machine. There was no machine learning. There were no generative audio APIs. There wasn't even a computer sitting next to the console.

We were just making records.

Those performances became multitrack tapes. The tapes eventually became digital files. The digital files became structured data. The data became part of a private machine learning system. More than 40 years later, those recordings could become part of the creative process again.

There is a lot of conversation right now about whether AI is going to replace songwriters, musicians and producers. After spending two years looking at it from a completely different direction, I think there is another question worth asking.

What happens when we stop asking AI to create for us and start teaching it how to create with us?

For me, that's where this gets interesting.

I wasn't trying to teach AI to replace my creativity. I was teaching it how to work with it.