What's the difference between two microphones, or two preamps? Between an MP3 and a WAV? Mix A and Mix B? First you need to find out whether there is an audible difference. Casual listening comparisons are almost never up to this job, because they leave too much room for error. All of us are subject to listening biases (expectation bias, confirmation bias, loudness bias, and more). We receive cues and influences from what we see, from the person running the test, and from what others think. Casual comparisons are full of these problems, so they can't give a reliable answer. We need a scientific method, and the ABX Comparator can help!
In an ABX test you can listen to files A and B as often as you like, knowing which is which. Then you listen to X, which is secretly either A or B, chosen at random each trial, and decide which one it is. The ABX Comparator removes most of these test problems and tells you when the test is complete: either an audible difference was detected, with statistical significance, or no audible difference was detected. You might be surprised at what you learn!
Scientific methods for measuring just-noticeable differences, and for double-blind testing, go back nearly 200 years. The ABX method for listening tests was described in 1950 by W. A. Munson and Mark B. Gardner of Bell Telephone Laboratories. In the late 1970s, Arnold Krueger, David Clark and colleagues in the Southeastern Michigan Woofer and Tweeter Marching Society built a switch box that made double-blind ABX tests practical for hi-fi equipment, which led to the ABX Company. Today, software makes ABX testing available to anyone with a pair of files.
After the last trial, the app compares your score with what guessing would produce. The p-value is the chance that someone who hears no difference at all would score at least as well as you by luck. If it's below 0.05 (5%), the result is called statistically significant: you can hear a difference.
A test can't quite prove that two things sound identical. A score near half tells you that you, with this material, on this system, couldn't hear a difference. That's still a useful answer: a difference you can't hear in a fair blind test doesn't matter much to you. More trials, better listening conditions or a more revealing passage can potentially turn up a small difference (or not).
Jon Boley and Michael Lester, Statistical Analysis of ABX Results Using Signal Detection Theory, Audio Engineering Society 127th Convention, New York, 2009.
Questions, ideas, or bugs? edulab@wondersmithaudio.com
Try the other Wondersmith EduLab apps, or bring a hands-on workshop to your school or team.
Visit the Edu Lab ↗The test is by ear only: the time bar shows where you are, not what the audio looks like. If you can't tell, guess. Guesses are part of the test.
A B X play that source · Space play or stop · 1 X is A · 2 X is B
Pressing a Play button while it's already playing stops playback.
The fewest matches needed for a significant result (p < 0.05):
| Trials | 10 | 16 | 20 | 25 | 50 |
|---|---|---|---|---|---|
| Matches | 9 | 12 | 15 | 18 | 32 |
The app comes without example sounds, so use your own files. Make each pair from the same source, so the only difference is the one you're testing.
Export the same mix as a WAV and as an MP3 at 320, 192 and 128 kbps. Which bit rate can you reliably pick out? Leave Time-align on: MP3 encoders add a short delay at the start, and the app will report it.
Make a copy of a file turned down by 0.5 dB. With Match levels off, the difference is usually easy to spot, and many people describe the louder copy as fuller or clearer, not just louder. Turn Match levels on and the difference disappears. This is why level-matched comparisons matter.
Print a track through an analog-modeled EQ or a "warmth" plug-in set flat, against the unprocessed file. Or compare sample-rate conversions, dither types, or two mastering limiters at the same loudness.