Thursday, 13 September 2007

Its all lies...

In this post from the excellent IWR Blog the author points out a short post on the CMS Watch blog. There are several great quotes from the IWR Post which demonstrate the difficulties in benchmarking any sort of service;

"In many fields of computing, benchmarking is broken. In server microprocessors, companies like Intel, AMD, IBM and Sun can make merry with numbers but they don’t mean much, at least not without a ton of accompanying explanation. The fault is not so much with the design of the simulations themselves or their accuracy, but with the multiple, and often mutually contradictory, nature of the metrics being applied. For example, in servers, some buyers will want raw performance, some energy efficiency, and an increasing number will want the perfect virtualisation host."

"Too often, benchmarks are cited to impress even though the numbers are meaningless."

No comments: