My Stand On Objective Vs Subjective Reviews

As I'll be posting up reviews and quick impressions of audio equipment from time to time, I feel that it's important to let you know my stand on objective and subjective reviews.

As a recent chemical engineering graduate, data and measurements are pretty much everything to me. If the measurements don't make sense, you analysed it and find out where the problem is. The same can be applied to the audio industry. The audio industry has been around for years and sound engineers have been making measurements and graphs for an equally long time. I'm fairly confident that the techniques and measurement equipment used today are able to paint a more accurate picture of what our audio equipment sound like than our ears (There are exceptions that I'll get to and elaborate later on.). Below I get started on my the objective vs. subjective section, I'll like to touch on and clarify these topics of interests: snake oil, expectation bias, neutrality, and double blind tests.

Snake Oils
Snake oils I will define snake oils as such: advertisement of a product that is either false and exaggerated, or misleading (there is a difference btw). As a quick note, I should probably point out that just because a product is based on snake oil will definitely have a subpar performance, or its price will be ridiculously expensive; It just is the case most of the time, and you'll be hard pressed to find the exception to the rule, but they are out there. A product with false advertising will not perform feats that it is claimed capable of (for example, it having a 24 bits DAC when it has only a 20 bits DAC). Fortunately this is blatant fraud and regardless of which camp you belong to, you won't stand for it. So such products aren't that common.

 A product with misleading advertisement is more tricky. This is simply down to whether you believe the snake oil to be true, or you believe it to be false. I will give 2 examples. It is common now to see many products advertising the use of boutique opoamps or DAC chips in their circuits, hence leading you to think that the products I are well designed, will sound good and hence worth paying what they are worth. However, I will like to point out that just because a good chip is used, the performance will be good; a chip is just one part of the circuit,just like a chain's strength is not dependent on one link, and a V12 engine will not guarantee that a car will be fast; you have to consider the material, suspensions, aerodynamics, tires etc... My point is that the presence of good components does not immediately equate outstanding performance; you need to see how they're placed together. My 2nd example is much more controversial: Audio cables. Most audiophiles (including the old me) believe that the "sound" of a cable depends on the material, purity and whether it has been cryo treated (I had never believed this....). Afterall, different materials have different resitivity,which will affect the electrical signal running through it in a way. Right? While that is true, the issue lies in the fact that such differences are really really small, and thus not audible, even to the most sensitive of ears, or when the cables you're using is really really long. In a typical desktop headphone setup, you can't hear the difference.Any differences that you may hear are mostly due to expectation bias. A simple way to determine if there are indeed any audible differences is to do a double blind test. These will be further elaborated in the next section.

Expectation Bias
If you've been reading NwAvGuy's blog, you'll know that one of the main, HUGE reasons why subjective reviews aren't very reliable (especially those done without ABX and double blind tests, again it will be elaborated on later) is due to expectation bias. We humans are EXTREMELY susceptible to bias of all kinds. For those of you who watch Mythbusters, in the Battle Of The Sexes Episode, there was a myth that said that women were worse drivers than men. In order to prove/disprove this myth, they had 10 men and women covered from head to toes in clothes. They wore helmets with reflective screens so that the test examiner couldn't look in, and the men even had to wear padded bras so that the test examiner will not be able to tell the men from the women by the lack of a chest. Obviously, if the test examiner was able to tell the gender of the driver, the preconception (that women are worse drivers than men) will come into play, and affect his grading. This gives a simple example of expectation bias and the steps taken to remove it. The same can be applied to audio. When comparing two different equipment, if you know that one is alot more expensive than the other, you will naturally EXPECT the former to sound better than the latter, even though that may not be the case. For more information on expectation bias, I recommend that you head over to NwAvGuy's blog. He did a fantastic article on this topic.

Besides expectation bias, there are other form of bias that can come into play and affect the result of the test. In terms of online reviews, the most common bias is $$. Normally when you read any review, there will be a disclaimer stating if the product reviewed is bought by the review, or given to him by the manufacturer. If you are given a product, you will UNCONSCIOUSLY feel the need to give a good review to repay the goodwill you just received. For some review sites, they contain advertisements by manufacturers. Headfonia is the most well-known example in the audio world. Looking at their website, I can immediately spot 7 advertisements on the top right hand corner. Putting yourself in their shoes, will you be able to give a negative review of a product, knowing that the manufacturer is paying you for advertisements, and that they can easily pull out all advertisements? I've read alot of their reviews and generally, they are happy with the products reviewed, and that the performances are well worth the money you would spend on it (for eg, the FiiO XX might not perform as well as a Benchmark DAC, but for it's price, you're getting performance twice of it). The only exception to this is NwAvGuy's Objective 2 amp. It is unique in the sense that the design is open source (meaning everyone can become the manufacturer of the amp) and that NwAvGuy does not earn a single cent from any amount of Objective 2 amps sold. The Objective 2 amp is a serious threat to many manufacturers' products, it having been measured to be completely neutral, but at a fraction of the cost of most amps with similar performances. When Headfonia reviewed the O2 amp, they faced two possible scenarios: a) They give the O2 amp a good review, consumers start to buy O2 amps instead of manufacturers, resulting in the manufacturers losing profits. The natural thing for them to do is to pull advertisements from Headfonia. b) They give the O2 amp a bad review, it makes the manufacturers' products look good, naturally, manufacturers will continue to advertise on their website. If I'm the owner of Headfonia, I will give the O2 amp a bad review to ensure a steady stream of income from advertisements. Of course, it's possible that the O2 amp is in fact, an inferior amp that sounds as bad as what Headfonia had made it out to be. I will like to point out though, that the O2 amp has been measured to be audibly identical to the Violectric V100, so why is the Violectric V100 given a glowing review, while the O2 amp given a bad review? (I will explain later on why when two equipment have the same measurement, they sound the same.) Hence, you WILL NOT find a single advertisement on this blog, and that every equipment reviewed will contain a disclaimer at the beginning, so you can judge for yourself, the bias present in my review.

Neutrality
I will define neutrality to be the same as transparency: Audio equipment that are 100% neutral will present the music to you exactly the same as how it was mixed or mastered by the sound engineer. A simple analogy is if you think of your audio equipment as windows; If your windows are transparent, what you see through your windows will be the same as when the windows are not there. Of course, everyone has different tastes in music (just like how everyone has their own sense of beauty, one man's poison is another man's meat etc...) so not everyone will like the way the music is originally presented. For instance, I like my music to be very clear, with tamed highs, and a strong bass which doesn't spill over to the mids. Hence, I might find the original mix to have highs that are too shrill, and bass that is not strong enough. So I won't look for a 100% neutral audio setup, but one that will give me the sound that I want. There is no right and wrong in your music preference. It isn't wrong to prefer music that is not "neutral", it's just a matter of preference.

With regards to amps and DACs, NwAvGuy recommends that you get the most neutral setups you can find. His logic is actually very simple and doesn't go against my "doctrine" of " Getting the sound that you like". There are a few ways of achieving that:

1) Go around testing different combinations of headphones, amps, and DACs till you hit upon the right combination and synergy that will give you your holy grail sound. The first downside to this method is that it is very time-consuming and can be expensive. Afterall, if the amps are designed by ear, it is very likely that you will hit upon 2 amps that sounds the same. So it's very likely that you will not be able to find a cheaper alternative. The other downside is that if you ever change any one piece in your setup, you will have to change your entire setup (very expensive) or look for a piece with similar sonic performances (very limited choices).

2) Get the most transparent amp and DAC, and choose the headphones/iems/speakers that sound the best to your ears. After that, play with EQ till you've achieved your holy grail sound. The upside to this is that there is no point in changing/upgrading your amp and DAC because, while it's possible to design more and more transparent equipment, there is a limitation to what your ears can perceive, and in today's market, you can already find equipments that are audibly transparent. So mostly, the only upgrade you'll have to do is your headphones.

Both methods can ultimately give you the best (in your opinion) sound but NwAvGuy heavily argues for the 2nd method because it is just so much cheaper.

ABX and Double Blind Tests
I'll like to state upfront that I'm no expert on ABX and double blind tests so I may state several wrong facts. For a more informative read on this subject, I will suggest that you check out NwAvGuy's blog. From my limited understanding, and from a logical and scientific point of view, I believe that ABX and double blind tests are extremely useful when done correctly. Mythbusters use them all the time on the time on their show and I can't begin the backlash they'll get if they haven't done so.

Let me just list the following experiement: Comparing 2 cables, whether it's between 2 cables of different materials or of different makes. As mentioned before, expectation bias will wreck havoc on how you perceive the cables to sound like. You will expect the silver to sound bright and analytical (probably because of how shiny silver is) and copper to be warm (because of the orangey color of copper). However, in a double blind test, where 2 cables are connected to the same headphones, like 2 DT880, plugged into the same headphone amp, like a Violectric V100, and the cables are wrapped in the same black covering, the expectation bias are removed. You are not able to tell which cable is which, both cables are powered by the same amp, and you're listening to the same headphones. To make the test fair, the cables should be swapped with each headphones because it is likely that while the headphones are the same model, they might sound slightly different from each other.

There are many AES paper done on this subject and the following link is one of the easier reads: http://seanolive.blogspot.sg/2009/04/dishonesty-of-sighted-audio-product.html

Subjective Reviews
Subjective reviews are most of the reviews that you see online. Most, if not all, of my reviews will be subjective reviews too. I will define it as a review that is based mostly or solely on the thoughts and opinions of the reviewer, without any hard unbiased results or measurements. Straightaway, you can see a few pros and cons with such reviews:

The big pro is that anyone can write a subjective review. This means that one can easily find many reviews of the same equipment, and compare and contrast among the different reviews.

The big con is that bias will ever be present, since such reviews only contain the thoughts and opinions of the reviewer. Some reviews will be more biased than others: As mentioned before, blog sites like Headfonia which accepts advertisements has a incentive not to write bad reviews. For that reason, my blog will not contain any advertisements, I am not making any money off of my blog, and any equipment that I review will contain a disclaimer or information on whether I bought the product, just quickly audition it and etc.

Beauty is in the eye of the beholder and the same applies to sound too. What I like may not be what you like and vice versus hence it is common to find opposing reviews. This also means that every subjective reviews should be taken with a grain of salt and where all possible, you should try the product out for yourself first before making a decision on whether to buy it or not.

Objective Reviews
Reviews that consist of mostly data measurements, data analysis and ABX/Double blind tests are objective reviews. Some comments or conclusions are usually made based on the data collected.

The big pro here is that data measurements are easily reproducible. Anyone else with a similar or accurate measuring device should be able to obtain similar results. Human bias is not present, except maybe in the final comments or conclusions. However, someone with enough knowledge can easily draw his own conclusions based on the data and charts collected.

The big con here is that you need the technical knowledge, and equipment to collect sufficient data to write an objective review. Furthermore, most readers do not have the knowledge to be able to understand the data presented. Hence, objective reviews are very rare and hard to find and more often than not, you might not be able to find an objective review of the product you're interested in.

On a side note, ABX and double blind tests are the exception here. You do not need to be a sound engineer to conduct your own ABX/double blind tests or to understand the results of one. I believe that anyone with a high school education will be able to follow the article that is linked above under the ABX section of this article. However, you do need to take an extraordinary amount of time, and be particularly meticulous to conduct a proper test and I understand that can be off-putting to most, since 1 small misstep can render your entire experiment meaningless.

First Impression of the Aurisonics Generic AS-1

On Friday, a friend and I decided to go all the way down to 112 Katong and audition the Aurisonics Generic AS-1. I read up on it beforehand, and I was pretty impressed by the specs (15mm dynamic driver!!), the design and the reviews that I found on Head-Fi. Similarly, my hard-to-impress friend was interested in trying out too.

I first tried it without an amp, straight out from my iPod Nano. Instantly, I was baffled by the sound I was hearing. It was so.... muffled that I couldn't believe it was coming from a S$350 earphones. My Soundmagic PL30, priced below S$50, sounds better than this! I then tried it with the O2 amp and although the sound cleared up a little, it was still really really muffled. I passed it over to my friend without a comment (no expectation bias) and after he tried it, he kinda agreed with me that the AS-1 was really really bad. There were no good points at all: The highs, mids, bass, soundstage... Nothing was good. (Btw, I was using the JVC FXT90, and my friend was using the Futuresonics Atrios, so our ears were pretty used to low to mid tier earphones. The AS-1 is about twice the price of the JVC FXT90.)

So... all in all, a vastly disappointing audition. Generally, with a bigger driver, you will come to expect a better sound, and in this case, the sound is much worse than other dynamic drivers earphones with smaller drivers. Incidently, my FXT90 has 2 dynamic drivers and the Atrios has a dynamic driver too. In this short dynamic earphones shootout, size really doesn't matters.