Synthetic 3D Sound

I was working with a buddy for about a year who had one of those Neumann head microphones. Pretty weird to listen with headphones on after recording with that thing.

Interesting article, but I’m not really sure what they are getting at or if they are even close to being the first to do this? Binaural mic’ing has been around for quite some time now.

Forget the video part - how do they achieve the spacial positioning? It doesn't sound like basic panning. There are products/reverbs out there that move the apparent position of a given source ..say in an x,y matrix. My first thought, especially given the amount of reverb present and the quality of the recordings was that's what they're using. It would work fine with a single source, but there are a number of clips in the video with two instruments.. What kind of product can do that with a mono source?  Again, sans the video part.

As I understand it, you do NOT need a binaural signal present for the algorithms to work. Otherwise, what would be the point?

That's the whole idea behind training it ahead of time, as discussed in the article. They taught the AI program to recognize what it is seeing, via video, and correlate that with the mono signal source and turn it into the 2.5D sound output.

So it's not something that has to be trained for every signal source. They've already got the AI trained now to recognize things in future video feeds. All that would be needed is a mono sound source and corresponding video. That's it.

I'm not saying whether or not this currently or ever will have real world applicability, but it might. It's an interesting concept either way.

I'd agree, something like a gaggle of geese would be pretty difficult. Though, in fairness, they did seem to indicate that this was only geared towards musical performance and not any and all sounds.

15 years ago, I listened to a 5.1 demo of a thunderstorm that was recorded off a camcorder's mono microphone. It sounded very good. The decoder was some hybrid of Dolby tech. Don't remember the company. But the thought then was to allow for surround sound audio decoding even on small portable recording devices, without the need for a surround mic array.

Hey Mark, nice to see another Sci-fi fan on the forum. Just love those types of shows.

Hahaha perfect response, ericn!

I should note, too, that I am *mostly* kidding in my robot post.
But not entirely. Not entirely at all.

I have some trouble understanding what the benefit to humanity is of the tool from the original post. Besides, "wow, that sounds cool, doesn't it?!?!" I can't see what exactly it is achieving that is of any real positive benefit in the long run.

But then again, AI kind of freaks me out. I feel freaked out when I get an ad for a product that I would never purchase, but that my wife and I had a discussion about two days ago. My wife would never buy the hypothetical product, either, FWIW. Neither of us will have searched for it online. I do believe our phones are listening to us. And that freaks me out.

Bingo. Terrifying.

I get the fear of AI, and don't think it's entirely unfounded, though I do think it can do some incredibly useful things and is misunderstood, at least in some cases.

If you want to read some really interesting developments on this front:

spectrum.ieee.org/tech-talk/transportation/self-driving/cheap-centimeterprecision-gps-for-cars-and-drones

I went to a talk on this particular research at the Texas GIS forum a few years ago. It was kind of mind blowing what this sort of advancement would allow. The word "Skynet" came out of my mouth more than once. It was both incredibly interesting and also kind of scary.

It's not that this particular research involved AI itself, but it IS the kind of advancement that would give AI a truly global reach as far as the kind of information net it would need to cast to do the very thing that is being fearfully discussed in this thread.

At this conference, they were discussing the ability of everyone's phone to cheaply and easily create a centimeter accurate, georeferenced, 3D image (with corresponding photos) of the entire world. Think Google Street View, but EVERYWHERE, not just what can be seen from a Google car driving down the street. This includes the inside of buildings. Anywhere a smart phone can go, so can this network. Both interesting and scary stuff.

arxiv.org/pdf/1812.04204.pdf

Read the original paper and it will answer most of your questions (or contact the author. They will be more than willing to tell you about their project. At least that was my experience when I was working on my project based on www.ofai.at/~jan.schlueter/pubs/2017_eusipco.pdf).

I really liked their idea of using video frames to provide visual spatial information. I wonder how they have implemented that feature in their code (pity they didn't make their code open to public..).


p.s. Don't trash talk someone else's work if you don't know what you're talking about.

I actually built a binaural head about 40 years ago.  It was a wig dummy that I covered in putty (to have a skin-like consistency).  A couple of cardboard pinnae and clips for the old Sony EC-1s (all I could afford).  The pinnae stuck out a bit, so I named the head Lyndon (after President Johnson, with whom the head had some similarities).

The thing actually worked pretty well--at least well enough for my experiments.  The problem was the players.  They couldn't stop laughing at the thing.