How to set up a proper AD conversion shoot out?

There's a thread on GS that does a round trip loop-back file and the guy uses Audio Diffmaker. Not sure the veracity or the results. I think he has them ranked according to a parameter he labels as "difference"...but I'm not really sure what it is measuring. I would think, though, this would really only measure the transparency of the AD, right?

hmm..wonder if I could build that? Not interested enough to drop any money into this. How would something like that affect the impedance loading on the mic pre?

Would we actually hear a difference in a single track across these different AD converters? Let's say I track something like an acoustic guitar. Are we going to be able to hear enough of a difference to say that one is better than the other? Or is this a case of a cumulative effect?

Do you have an ATTY handy? You're basically going to mult the preamp output simultaneously to the various converters. The problem comes in if they're calibrated to different levels on their analog side--at that point, you have to either pad the level to hit the less hot ones in their designed spot....or just find a compromise erring on the low side--like if you have a -18dbfs and -12dbfd, you might just pick -15 to be the level to feed them--slightly low for one, slightly hot for the other....

Then afterwards, your gonna match level by reversing phase and getting as much to cancel as you can.

Then for one aspect of the listening tests, apply absurd "clean"/boring digital EQ. I like to Highshelf +20db at 3khz. Or crazy high pass filters--say 250hz. Not usually the same pass--but, you're looking for it to retain body with the lows gutted and with the high shelf, a good converter will sound laughably bright--a cheaper one will be like ice picks in your ears. Because again--too bright isn't the same as BAD BRIGHT....they're extreme for a reason--they're showing you how malleable the signal is--not that you need it that malleable, but if it can handle a huge shelf and not get "bad bright", then you'll know it takes digital DSP gracefully. The difference in other algos and real world use will obviously BE smaller....but, will be cumulative.

I would stick to solo and ACOUSTIC instruments. I find various distortions and frequency losses can end up being flattering on amps and bass DI and such...

Test single rate and double rate as knowing how well they work and how evenly they sound is something you'd want to know. End of the day for tracking, you will use whatever a given converter sounds better at...which I've never NOT had be double rate, but it's a value add if the company shifts it's filters nicely for single rates for professional "who know what you'll be given today" purposes.

If you want to get out of the scientific realm and more into what you actually hear, you can do what I did once. Make a 3 or 4 way split cable (whatever), come out of a pre amp with it into all the different converters with the same exact take. No way to influence the results this way. What you hear on the other end is what they do.

Very telling.

I'm with Cowboy on this.

Loopbacks, analyzers, matching levels, EQing, all that stuff lends itself to adding biases to the results.  They are tools to help you figure out issues, but they don't tell you anything about how a box sounds..

Case in point.. We all know and love vintage Neve gear..  But measurement wise, no two modules sound/measure quite the same, and they are typically considered distortion machines.. But they sound great.

So over all, I'd mult the signal and listen.  Loud, soft, percussive, smooth, male and female vocals, strings, etc.  Listen to as many sources as you can, switching between the tracks from each box quickly and you'll hear the differences pop up.

But be aware that splitting the signal will load the preamp and the AD boxes differently and they'll react differently to how they would if they were the sole receiving device..  Again, introducing a bias to the test.

This is where I start to become cynical about things like this, because going looking for differences leads ordinary people to overlook scientific biases that are accounted for in extremely rigorous testing.

Because of this, I'd say that you have a handful of mighty fine converters in your hands right now.  If you can't obviously hear a difference while using them in a normal setting, then perhaps there really isn't that big of a difference to worry about..

I did it once with two converters and the bias (psychology) was the biggest problem.
Most of the differences faded away in a blind test, as long we knew the brands our brains did calculated things to the sound which have not been there.
It was easy one was a Behringer and with this it had to sound bad in our believes.

If you can choose between three great units in the same league, use the one you like the most and go on with your work.
We are sometimes too picky about the gear. We forget too often - its just tools to reach the goal.
Why wasting life time with hours of setting up a test??

Even cheap converters of today are more than usable...

Just my 2 cents...

I remember myself believing that I found the holy grail because the soundstage was wider.
Later my Tech told me that Metric Halo manipulates the Outputs with some kind of limiting.