Slightly off-topic. Now that 1920s jazz music is falling into public domain, has anyone tried to reinvigorate the music using AI and generative adversarial approaches? Pre-1940s music didn't have high-fidelity sound, so the strong bass lines weren't captured. In theory, we could "downgrade" modern recordings to sound like 1920s recordings, then use adversarial techniques to train the machine on how to restore the antique recordings. Anyone know of any work being done in this area?
It might be easier than that. Are the bass lines totally missing or are they just very weak? If you can capture a recording using vintage equipment and the placing of it, you can get the system response. Run the original recordings through an inversion of the response and you should get really close. Another possible method is to find the transform between an identical modern recording of the song and use the difference between the two recordings to make your transform.
So the idea would be to reconstruct the low frequency components from whatever upper harmonics are left in the recording? If you know the instruments and positioning of the recording device and something of its(the instruments, recorder, environment, etc.) characteristics, it might be possible to solve that using classic methods. There would be huge numbers of parameters, it is an interesting thought. Is there a large easily/freely available corpus of those recordings?
To do this, I think you are right that you would need to 'downgrade' modern recordings to sound old so that you have both sides of the training data covered.
This would be a cool project to work on. Ideally you would buy some vintage gear and then run the audio through both, but that would be very expensive. You could may be find some vst emulations though and get decent results.
People who are interested in this application should check synplant[0]. It has a ML technology called "Genopatch" which gives you 2 functionality:
1. you can try to describe a sound with some tags and it will try to generate a sound to capture the feeling of these tags
2.you can feed it with a sound sample and it will try to re-synthesize the sound with its synth engine. Though the end result will usually be just a "re-imagined" version of your input sample.
My guess is the underlying model is not a "deep" model. The main benefit is that the end result is not a wave file, but a list of generated parameters that can be synthesized by the synthplant engine. And now it comes the interesting part: you can tweak these parameters to finetune the generated sound. These parameters have actual meanings (FM ratio, reverb etc.)
How far are we from getting a general model that can resynthesize any instrumental audio sound without fiddling with any knobs, so that we can recreate instruments we hear from any song? Seems like it should exist by now?
For me creating the exact sound is not very interesting from sound designing perspective. You can always sample the real instrument.
Like physical modeling synthesis, the interesting part is to compress the sound to some parameters that you can tweak and generate new sounds
Another approach is VAE, which also you give your some latent embedding, you can tweak the embedding to generate new sound. However the meaning of this embedding is not explicit.
This doesn't really work on instruments like guitars. Open D sounds way different than fretted D on the E string. Timbre changes with position and it's one of the ways I determine where a player's hands are on the neck when I'm trying to play their song.
That is not something inherent in guitars themselves, it is the norm in steel string guitars and the fan-braced/Spanish guitar but mostly because that is the norm for all those mass produced guitars which make up the bulk of guitars. On steel string you can often greatly decrease this quality just by switching to flatwounds, this is part of the flatwound sound, it shifts the timbrel content into the players technique but if you want much timbrel content with flatwounds you need heavy strings and a high action, and the hand strength and technique that sort of setup requires.
Before the rise of the steel string and the Spanish guitar, guitars tended to be more even across their range and also had less bass which helped even them out, and now that sound is what we are used to. There have always been niches that wanted that more even sound, but for most that just makes it more difficult to play all that music that developed around these quirks, so they remain niches.
I'm not doing fancy AI stuff but I have worked a lot with my own bespoke supercollider system where I record whole fretboards of guitars and then play alternative notes based off of certain rules. For whatever dumb reason though, the most natural sounding thing is really just playing, e.g., any random D4 from its possibilities at any given moment.
Timbral differences also exist depending on force, the manner plucked, the already ringing overtones... It's hard to know what you want, but the most natural thing is always going to be some organic variation in the notes in general.
If you have a good ear, you aren't, I don't think, hearing so much the timbral diff in the individual open or fretted notes as much as the fact that a barre chord and an open chord is a different voicing of the same harmony.
No, I'm going off the timbral differences - same way I identify which pickup position is being used. There's a specific 'thickness' I cue in on to determine pickup and specific note placement.
SUNO is pretty close. It still has some weird things going on with high frequency artifacts and phase between left and right channels but if you aren't listening on a good system (like a phone) most people probably wont notice.
I sound design a lot of stuff (in fact I made some of the default kicks in the app), but this is just a different tool, and I wanted some practice training and deploying a generative AI model.
Interesting! I had not seen this. On their website they mention diffusion but not the other models so it might not be identical but its definitely similar.
Articles like this are why I come back to HN. Interesting technically, kinda novel and fun. Got me thinking about datasets that may be sitting on old HDD, got TBs of old video and audio from projects of past. Blogs like this help point the way.. Now if only I had the time..
The compression is the OTT which stands for Over The Top compression. It was originally a multiband compressor preset in ableton and is now used widely throughout dance music.
Did you save any of the "failed" results? I'd love to hear what kind of weird sounds it makes out of distribution (e.g. on the keywords it didn't have much data for).
Not sure if there's something wrong with the player, or if it's just me, but they both sound like noise. I guess the first sounds vaguely kick drum-like (but distorted), the second is just noise.
the spectrograms are 128x173 (128 mel frequency bins by 173 time frames)
the encoder is downsampling 4 stages of stride 2 convolutions so it halves dimensions 4 times
0: 128 x 173
1: 64 x 87
2: 32 x 44
3: 16 x 22
4: 8 x 11
Then i used 4 separate channels.
This was somewhat arbitrary due to the local training constraint. This would be a hyper parameter worth tuning if I had time to dig into this more.
I trained this a few month ago and don't remember exactly what I tried before I arrived here, but I only ran the whole process 2 or 3 times because of how long it took to train.
Hope this answers your question!
I wouldn't exactly say it's trying to solve a problem. It's to explore and see what happens which is what music is all about. It's also a unique niche model I haven't seen before.
Decomposing sounds from (fully produced?) tracks into underlying components, and then giving the user the option to synthesize them with different parameter settings. I think.
I was trying (and failing) to do this the other day. It’s a really interesting problem space and I love to see someone with a more solid foundation give it a try.
If you are committed the model should work about the same on any type of one shot sample. The code is public and documented so if you have the snare collection and a macbook you could probably point claude/chatgpt at it and it would be able to train on your laptop.
This is a really really fun sounding project - ironically, because there are no audio samples provided at all. I would have thought a music producer creating samples for music would naturally let you listen to what they were making.
I always roll my eyes when I see LLM weirdos talk about getting models to run on "old" hardware and finding out it's hardware that's still better than what most people have access to.
It doesn't make it any less impressive to those who know what hardware requirements for LLMs usually is/are but for those with no idea it usually ends up reinforcing bitterness towards it as they feel annoyed that their own hardware is somehow worse and yet are unable to upgrade because of said LLMs stealing all the hardware in the world all while RAM/memory/storage manufacturers manipulate the market(s) against them.
The Geforce GTX 1060 launched 10 years ago with a MSRP of $249. It spend 5 years and 4 months as the #1 card according to the Steam Hardware Survey. That makes it hard to feel that it is fair to accuse it of still being better than what most people have access to, unless you are asserting that most people have access to no GPU at all, which is likely accurate, but not likely to be accurate here, nor in any sort of enthusiast circumstance. If you lump the Intel Xe built in graphics (started with the 11th gen Core Is) and the Intel UHD (launched with 8th gen Core Is) together, the combined group would come in 6th place, with the 5 places above that in commonness for people who are actively playing steam games all being considerably faster than the Geforce GTX 1060 or Geforce GTX 1660 cards.
Interestingly, now the #1 GPU is the GeForce RTX 4060 Mobile version, which I believe is the first time the top has been a laptop chip instead of desktop chip.Items #2 and #3 on the list are the 2 generation old RTX 3060, followed by the 1 generation newer RTX 4060. 4th and 5th are RTX 5070 and RTX 3050.
If you are curious I used a NVIDIA GeForce GTX 1660 SUPER
So to be exact, it came out 7 years ago (I upgraded at some point on this desktop a long time ago and didn't remember the exact year) (I updated the article to reflect this now)
This cost $230 new and you can get one now for $100 which I don't think is too out of reach.
> but for those with no idea it usually ends up reinforcing bitterness towards it as they feel annoyed that their own hardware is somehow worse
I don't think "those with no idea" spend much time thinking about their hardware at all. They respond to marketing and peer-pressure influences, but most of them are not upgrading phones or laptops because they can't run AI on it.
Most people I know have been wanting upgrade cycles to slow down for quite some time, now. I think that those people will survive deferred retail therapy for a few years.
Slightly off-topic. Now that 1920s jazz music is falling into public domain, has anyone tried to reinvigorate the music using AI and generative adversarial approaches? Pre-1940s music didn't have high-fidelity sound, so the strong bass lines weren't captured. In theory, we could "downgrade" modern recordings to sound like 1920s recordings, then use adversarial techniques to train the machine on how to restore the antique recordings. Anyone know of any work being done in this area?
It might be easier than that. Are the bass lines totally missing or are they just very weak? If you can capture a recording using vintage equipment and the placing of it, you can get the system response. Run the original recordings through an inversion of the response and you should get really close. Another possible method is to find the transform between an identical modern recording of the song and use the difference between the two recordings to make your transform.
So the idea would be to reconstruct the low frequency components from whatever upper harmonics are left in the recording? If you know the instruments and positioning of the recording device and something of its(the instruments, recorder, environment, etc.) characteristics, it might be possible to solve that using classic methods. There would be huge numbers of parameters, it is an interesting thought. Is there a large easily/freely available corpus of those recordings?
To do this, I think you are right that you would need to 'downgrade' modern recordings to sound old so that you have both sides of the training data covered.
This would be a cool project to work on. Ideally you would buy some vintage gear and then run the audio through both, but that would be very expensive. You could may be find some vst emulations though and get decent results.
People who are interested in this application should check synplant[0]. It has a ML technology called "Genopatch" which gives you 2 functionality:
1. you can try to describe a sound with some tags and it will try to generate a sound to capture the feeling of these tags
2.you can feed it with a sound sample and it will try to re-synthesize the sound with its synth engine. Though the end result will usually be just a "re-imagined" version of your input sample.
My guess is the underlying model is not a "deep" model. The main benefit is that the end result is not a wave file, but a list of generated parameters that can be synthesized by the synthplant engine. And now it comes the interesting part: you can tweak these parameters to finetune the generated sound. These parameters have actual meanings (FM ratio, reverb etc.)
[0]: https://soniccharge.com/synplant
How far are we from getting a general model that can resynthesize any instrumental audio sound without fiddling with any knobs, so that we can recreate instruments we hear from any song? Seems like it should exist by now?
Fiddling with the knobs is the fun part. https://www.youtube.com/watch?v=la2u4VlGwbQ
For me creating the exact sound is not very interesting from sound designing perspective. You can always sample the real instrument.
Like physical modeling synthesis, the interesting part is to compress the sound to some parameters that you can tweak and generate new sounds
Another approach is VAE, which also you give your some latent embedding, you can tweak the embedding to generate new sound. However the meaning of this embedding is not explicit.
"You can always sample the real instrument."
This doesn't really work on instruments like guitars. Open D sounds way different than fretted D on the E string. Timbre changes with position and it's one of the ways I determine where a player's hands are on the neck when I'm trying to play their song.
That is not something inherent in guitars themselves, it is the norm in steel string guitars and the fan-braced/Spanish guitar but mostly because that is the norm for all those mass produced guitars which make up the bulk of guitars. On steel string you can often greatly decrease this quality just by switching to flatwounds, this is part of the flatwound sound, it shifts the timbrel content into the players technique but if you want much timbrel content with flatwounds you need heavy strings and a high action, and the hand strength and technique that sort of setup requires.
Before the rise of the steel string and the Spanish guitar, guitars tended to be more even across their range and also had less bass which helped even them out, and now that sound is what we are used to. There have always been niches that wanted that more even sound, but for most that just makes it more difficult to play all that music that developed around these quirks, so they remain niches.
I'm not doing fancy AI stuff but I have worked a lot with my own bespoke supercollider system where I record whole fretboards of guitars and then play alternative notes based off of certain rules. For whatever dumb reason though, the most natural sounding thing is really just playing, e.g., any random D4 from its possibilities at any given moment.
Timbral differences also exist depending on force, the manner plucked, the already ringing overtones... It's hard to know what you want, but the most natural thing is always going to be some organic variation in the notes in general.
If you have a good ear, you aren't, I don't think, hearing so much the timbral diff in the individual open or fretted notes as much as the fact that a barre chord and an open chord is a different voicing of the same harmony.
No, I'm going off the timbral differences - same way I identify which pickup position is being used. There's a specific 'thickness' I cue in on to determine pickup and specific note placement.
Huh, got it. That's pretty cool!
SUNO is pretty close. It still has some weird things going on with high frequency artifacts and phase between left and right channels but if you aren't listening on a good system (like a phone) most people probably wont notice.
synplant is a great synth!
Confused. Why not just make the kick drum from a sine? Seconds
I sound design a lot of stuff (in fact I made some of the default kicks in the app), but this is just a different tool, and I wanted some practice training and deploying a generative AI model.
A sine is only a part of most kicks
This has been done years ago. See https://audialab.com/products/emergent-drums-2/ for instance.
Interesting! I had not seen this. On their website they mention diffusion but not the other models so it might not be identical but its definitely similar.
Articles like this are why I come back to HN. Interesting technically, kinda novel and fun. Got me thinking about datasets that may be sitting on old HDD, got TBs of old video and audio from projects of past. Blogs like this help point the way.. Now if only I had the time..
If you know what you want to achieve you can asks claude/codex/glm whatever, to do the proof of concept first and Dave some time like that
Thanks!
Modeled reverb yet no modeled compressor, hrmm, is compression not used on kick drums (or not a big part of the sound)?
The compression is the OTT which stands for Over The Top compression. It was originally a multiband compressor preset in ableton and is now used widely throughout dance music.
I just wish it had samples! I want to hear it
For sure! I just added a couple
Did you save any of the "failed" results? I'd love to hear what kind of weird sounds it makes out of distribution (e.g. on the keywords it didn't have much data for).
I just added one with the "techno" keyword in the keywords section. Its pretty weird lol
They sound cool! Add a few more! :P
I was hoping to hear some songs using these samples!
Not sure if there's something wrong with the player, or if it's just me, but they both sound like noise. I guess the first sounds vaguely kick drum-like (but distorted), the second is just noise.
Chrome 149.0.7827.200 (Official Build) (arm64), macOS Tahoe 26.0.1
Excellent article! I think it has the right level of detail, one question though: why the shape of the tensor? 4x8x11.
That I didn't get from the text.
the spectrograms are 128x173 (128 mel frequency bins by 173 time frames) the encoder is downsampling 4 stages of stride 2 convolutions so it halves dimensions 4 times
0: 128 x 173
1: 64 x 87
2: 32 x 44
3: 16 x 22
4: 8 x 11
Then i used 4 separate channels.
This was somewhat arbitrary due to the local training constraint. This would be a hyper parameter worth tuning if I had time to dig into this more.
I trained this a few month ago and don't remember exactly what I tried before I arrived here, but I only ran the whole process 2 or 3 times because of how long it took to train. Hope this answers your question!
Yeah, thanks!!
For a moment I thought Gen AI meant the current generation of kids. It's a fitting moniker
I have to admit I don't understand what exactly the problem is we're trying to solve with ML here...?
I wouldn't exactly say it's trying to solve a problem. It's to explore and see what happens which is what music is all about. It's also a unique niche model I haven't seen before.
Decomposing sounds from (fully produced?) tracks into underlying components, and then giving the user the option to synthesize them with different parameter settings. I think.
I was trying (and failing) to do this the other day. It’s a really interesting problem space and I love to see someone with a more solid foundation give it a try.
someone needs to take care of the snares
If you are committed the model should work about the same on any type of one shot sample. The code is public and documented so if you have the snare collection and a macbook you could probably point claude/chatgpt at it and it would be able to train on your laptop.
You could save so much time and processing power by just learning how to sidechain.
This is a really really fun sounding project - ironically, because there are no audio samples provided at all. I would have thought a music producer creating samples for music would naturally let you listen to what they were making.
Good point! I just added a couple at the end of the intro
I always roll my eyes when I see LLM weirdos talk about getting models to run on "old" hardware and finding out it's hardware that's still better than what most people have access to.
It doesn't make it any less impressive to those who know what hardware requirements for LLMs usually is/are but for those with no idea it usually ends up reinforcing bitterness towards it as they feel annoyed that their own hardware is somehow worse and yet are unable to upgrade because of said LLMs stealing all the hardware in the world all while RAM/memory/storage manufacturers manipulate the market(s) against them.
The Geforce GTX 1060 launched 10 years ago with a MSRP of $249. It spend 5 years and 4 months as the #1 card according to the Steam Hardware Survey. That makes it hard to feel that it is fair to accuse it of still being better than what most people have access to, unless you are asserting that most people have access to no GPU at all, which is likely accurate, but not likely to be accurate here, nor in any sort of enthusiast circumstance. If you lump the Intel Xe built in graphics (started with the 11th gen Core Is) and the Intel UHD (launched with 8th gen Core Is) together, the combined group would come in 6th place, with the 5 places above that in commonness for people who are actively playing steam games all being considerably faster than the Geforce GTX 1060 or Geforce GTX 1660 cards.
Interestingly, now the #1 GPU is the GeForce RTX 4060 Mobile version, which I believe is the first time the top has been a laptop chip instead of desktop chip.Items #2 and #3 on the list are the 2 generation old RTX 3060, followed by the 1 generation newer RTX 4060. 4th and 5th are RTX 5070 and RTX 3050.
For sure.
If you are curious I used a NVIDIA GeForce GTX 1660 SUPER So to be exact, it came out 7 years ago (I upgraded at some point on this desktop a long time ago and didn't remember the exact year) (I updated the article to reflect this now)
This cost $230 new and you can get one now for $100 which I don't think is too out of reach.
Saying 10yo 6GB will make most people think you are talking about a Geforce GTX 1060.
That was an oversight. The tower is more than 10 years old. I updated the article to be exact now
English adjectives are highly contextual. In this context, the author's clear meaning is in context to the current generative AI boom.
> but for those with no idea it usually ends up reinforcing bitterness towards it as they feel annoyed that their own hardware is somehow worse
I don't think "those with no idea" spend much time thinking about their hardware at all. They respond to marketing and peer-pressure influences, but most of them are not upgrading phones or laptops because they can't run AI on it.
Most people I know have been wanting upgrade cycles to slow down for quite some time, now. I think that those people will survive deferred retail therapy for a few years.