News: Speaker fidelity tests

I wanted to try out something new to allow for an even better judgement of a speaker's sound quality. Please let me know if you have some further ideas suggestions or wishes, so that this test could evolve to something helpful maybe.

With this test, which I will simply call "fidelity test" I will try to focus purely on the fidelity of a speaker and how it sounds compared to the original recording. There are lots of speaker tests and speaker videos available on Youtube which will play you some song through a speaker to show off its sound qualities. Without any additional reference (another speaker or whatever), it is hard to tell if the shown speaker really sounds that bad, or if the used recording setup, microphone or even the room, where the recording was done is just that bad, so that it makes the speaker sound bad.
Other videos are trying to compare several speakers at the same time, similar to what I was doing until now, but when switching over to the next speaker, there is hardly any difference noticeable. Such tests are pretty useless and wasted time for the creation part and for the watching part if you ask me.

I don't claim that my videos I did so far to be good, but it really took an effort to make my setup as solid as possible and allow for direct speaker comparisons without having them at home at the same time thanks to referenced volume levels and a standardized recording setup with exact distances etc.
Despite all the effort I meanwhile noticed that I might still get differences in sound when I try to record the same speaker another time. Despite the same setup both recordings might still sound a little bit different, probably because of slight differences in angle between microphone and speaker, slight differences in distance (although +/-1cm shouldn't really matter), maybe even slight differences in outfit of the room, if one shirt is hanging at the left side instead of right might have an influence hahaha...

Instead of comparing 2 or more speakers against each other like I previously did, now I want to show how close a speaker manages to sound compared to the original source. This is not really easy to achieve, as my room is acoustically not perfect. It is not a typcial room with typical acoustics. But I noticed that recordings done in this room give purer results than recordings in bigger undampened rooms. I think that the best results might be achieved outdoors, due to missing interreflections with any walls (with expection of the ground), you will get an even purer sound. Unfortunately all my outdoor videos I did so far, were a real pain to do. Despite dead kittens and locations protected against wind, I always had to fight with wind noise, plane noise, etc...
An optimized room maybe even close to anechoic would be probably the best way to go, but currently I don't have ressources for such a thing, despite my wife who already thinks I got crazy with all those stupid speakers...

My idea is to record a song through each speaker in my usual recording room at one single fixed level. This level should be set not too low and not too high to simulate a comfortable listening volume and which should give a representative result for a speaker's sound quality. I currently fixed this level to exactly half volume of the Sonos Play:1, this corresponds to about maximum volume of the Mini Jambox, or step 9/16 on the JBL Charge 2, 67% on the Soundlink Mini etc. To bring this into some measurable SPL readings the recorded music gives me the following values at a distance of 50cm:
LAFmax: ~75dB
LZpeak: ~95dB

For some speakers this level might already be too loud causing distortion or introducing compression artefacts, noticeable bass reduction etc, other speakers might sound better at even higher levels. Thanks to the dynamic sound processing of many modern active speakers, the overall sound character can differ quite much between low or high levels, as you can see here from the example measuring of the Bose Soundlink III where I overlaid both frequency responses for a rather low level (~25% blue) and a higher level (~75% red):

Both would sound completely different if both were played back with the same loudness, with only the mids still resembling the same speaker. But due to equal loudness contours of the human hearing in reality the differences might not appear that obvious as the human ear hears low and high frequencies differently at different volume levels. Everything is not that easy, still the goal is to give the best representation of the speakers sound character and bring that into relation to the original recording.

Of course with this test I cannot cover other aspects of a speaker's quality like directionality, loudness etc, as this would lead too far and make it too complicated. This fidelity test will only provide the soundquality exactly on-axis at optimum position which should give the best possible result for every speaker, I might vary the angle between speaker and microphone slightly if this gives a better result. Especially on Bose speakers the sound can change a lot if you turn them only a little bit. While many speakers might still sound good from higher above or from the side, I won't be able to give any statement on this like I already tried with some of my previous videos, where I also included sound recordings off-axis for comparison. 

The used recorder and microphone will also have an influence on sound. My first preliminary tests are done with the Zoom H1, it is definitely not the most linear recorder, but I found it to record low frequencies better than the H2n, or the Olympus LS10 which I had also tried before. It seems to have some problems with treble though. It sounds too harsh and metallic compared to the original sound of the speaker.
I am in the process of trying out other recorders with a microphone that has hopefully a flatter overall response. Of course I am open for any suggestions. If other readers have more experience than me on this topic, please let me know. I really want realistic and representative results.

So far I have chosen a funky recording from Fusion-bassist Gerald Veasley, I like this track and I think it is a pretty nice recording with good definition in both bass and treble, only mising some vocals. If you have better suggestions for a testtrack please let me know. It should be a rather non-commercial track to avoid copyright issues. Maybe there are even some real musicians among the readers who would like to contribute an own song, this could also be used as promotion for their own site etc. The premise is that it has to be a recording of high quality which allows conclusions to be drawn on a speaker's sound. When executed well, this could become a big database for sound quality of various (portable) speakers that I was thinking of from the beginning.

Here you can find some first videos with speakers I currently had at hand. If you listen to them over headphones, you will immediately notice the lack of any stereo imaging from the speaker recordings compared to the original track. Maybe for better judgement I should also convert the original track to mono? When listening to these videos through speakers the lack of stereo is not as obvious though.
I am open to any suggestions, let me know if all this makes sense, and if the differences between all these speakers are noticeable at all. Please leave me a comment if you would like to contribute! Let's start with the Sonos Play:1 which is probably the most faithful speaker of the recorded ones: