UPDATE: AMD's website paints Barcelona performance pictures not seen in the real world

Posted by Rick C. Hodgin

Indianapolis (IN) - If you visit AMD's website and look at Barcelona's performance numbers you'll see a very rosy picture.  Benchmarks there show performance numbers which most would desire.  And yet, if you turn to the real-world benchmarks published today, the situation is much different.

AMD's benchmarks


The first benchmark listed on AMD's website is called VMmark Performance.  It shows a 3.2 GHz Opteron 2224 SE compared to a non-existent Barcelona Opteron 2360 SE.  They also use a "VMware ESX 3.5 experimental build 55263" release which is not available to the public.  VMware's website shows VMware ESX Server 3i as their current product, which on build 52542.

The increase in performance for this benchmark is 124% over the much faster clocked 880 system.  That's more than double the performance and it shows just how advanced Barcelona is over their previous architecture.  And yet to obtain these high performance numbers, AMD is using a publicly unavailable processor on an build of VMware ESX, which is also not available.  These numbers are not something we can duplicate in the real world, or even take advantage of yet.  They show only what is possible with Barcelona, not what's available.

The next benchmarks actually show the available 2350 part that was launched today.  The performance increase through AMD's Rapid Virtualization Indexing (RVI) is shown in these benchmarks.  RVI is the same technology previously called Nested Paging.  There are no doubts that RVI speeds up virtualization performance.  And that fact is the reason AMD introduced it into this generation of architecture in the first place.  MS Terminal Services sees a 23% speedup and SQL DB sees 14%.  These speedups shown are very powerful and real-world benefits will be had.  Again, hardware improvements relative to a prior architecture show AMD's increases with Barcelona.

When we get to the benchmarks against Intel's Xeon, they use the 5345.  It's two steps below the highest-clocked Xeon part, but is comparable in power.  Here we find the FLUENT 6.4.3 (beta release) benchmark showing Barcelona exceeding the 5345 by 10%, 28%, 46% and 67% in four different tests.  AMD's Barcelona is also shown scaling at a ratio of 6.45 on this test.  The ideal increase would be 8.0 and Intel's 5345 is shown at only 3.54.

If we look at energy consumption, AMD sees a 45% advantage with four cores, and an 83% advantage with eight cores, over the 5345.  Again, significant increases across the board, though they are using a beta release of the benchmark software in all tests.  The beta release has not been fully validated and could be changed before the final version of this benchmark can be trusted.  They show relative increases in performance, as well as those against Intel.  And they show AMD in a very positive light (up to 67% better than the 5345).

Next we get to the test which prompted me to write this article.  It's also the one which will be the focus for the AMD-benchmark-to-real-world comparison.  AMD's STREAM memory comparison.  In this test they show MB/sec throughput around 17,000 (it's graphical, no hard numbers are shown).  These numbers are 261% to 289% faster than Intel's 5345.  That's nearly 3x as fast.

I should also point out that for the test they use two different memory sizes.  Barcelona used 8x 1GB DDR2 while Xeon use 8x 2GB FB-DIMMs.  They also used two different hard drive sizes.  Barcelona used 150 GB Seagate SATA while Xeon uses 80 GB Seagate SATA.  And the SuSE Linux versions that are listed are not identical.  It does not specifically say they are different, but they are not listed identically.  And to go even one step further, the graphic showing the benchmark results is labeled "Xeon 5345" while the "Configuration Information" link pointing to the details shows two 5355s being used.  So which is it?

How can we trust a benchmark that is so poorly labeled and directed?  With this different hardware and potentially software, are they not comparing apples to oranges?  Shouldn't tests like this, especially those published by AMD itself, use the same equipment and the same Linux install?

AMD has not done its work on these benchmarks on their own website.  They are not complete, yet they indicate numbers which look very promising.  Still, not all of those numbers are even found in the real-world.

Available benchmarks


It's worth noting again that not every reviewer was able to get a Barcelona CPU for review.  And the highest-end 2350 part was not even handed out to all of them.  Some were forced to use 2347 parts at 1.9 GHz.  And even more, some reviewer had to go through special channels outside of AMD, even to other countries.  And those that did receive parts from AMD received them so close to the release date that a whole host of proper benchmarks could not be run.  They wanted to do more, but they didn't have the time because of the way AMD has handled this launch.

By not giving reviewers products with enough lead time, AMD has cast all of us into a position of having to comment on that fact.  We are also required to release lesser results than we might've been able to do otherwise.  Or to paint part of the picture to day, only to find the truer picture in a few days.  And that very fact has been prominent in the reviews so far.  And, whether AMD realizes it or not it does raise suspicion by consumers.  Why did they wait so long to send out Barcelona samples?

 

Read on the next page: Real world benchmarking and results published by review sites 


 

 

Real-world benchmarking

 

AMD's new memory subsystem contains optimizer technology which AMD claims will show up to 50% increases in throughput.  The benchmarks which stress the memory subsystem do show performance increases.  These memory improvements are also targeted primarily at high performance computing applications, though desktop Phenoms will also see benefits.

Still, even with the memory enhancements a LINPACK benchmark at AnandTech.com shows Xeon 5345 besting Barcelona by about 17%.  The 5345 was chosen because of its commensurate power consumption, even though there are also two faster Xeons with greater performance than this one.

Of all the tests performed at AnandTech, Barcelona won 5 and Xeon won 3.  The total percentages by which each one beat the other were 27.13 and 73.94.  This means that when Barcelona won, it won by much less percentage-wise.  And when Xeon won it, won by a a lot more.  These tests do not demonstrate the performance AMD's website indicated they should, nor do they include the fastest Intel parts available today.

If we then look to a much more comprehensive benchmark at The Tech Report we find Barcelona winning 3 tests, and Xeon winning 23 tests.  The total percentages were 123.63% and 634.26%.  It's also worth noting that the bulk of the large Barcelona percentage shown here comes from a single test which included a 121.14% improvement over Xeon in memory bandwidth using a 1 GB test set.  If we remove that test, then Barcelona's three wins only total a 2.49% over Xeon's.  And if that memory test had used data sets of anything at 64MB or below, then it would've shown Xeon winning by similar percentages at various data set sizes.

[Update:  In reviewing the numbers, there were two mistakes made regarding the Barcelona versus Xeon wins.  The new wins would be 5 and 21, and the new percentages would be 136.21% and 623.04%.  If we factor out the 121.14%, then Barcelona's total win becomes 15.07%.]

All told at both sites, Barcelona wins 10 and Xeon wins 24, with one tie.  The total percentages across 38 benchmarks were 163.34% and 696.17% values.  The average winning percentages are 16.33% for Barcelona and 29.04% for Xeon.  If we remove the one benchmark which had Barcelona winning by 121.14%, then the results are average winning percentages for Barcelona of 4.22% on 29.4% of the benchmarks.  And 29.04% for Xeon on 70.6%.  This indicates that in those instances where Barcelona wins, it wins by a much smaller margin than Xeon.  So small that it's hardly worth mentioning, especially when you consider there are two faster clocked processors available today from Intel.  [Update:  This paragraph has been adjusted to reflect the revised numbers from the previous paragraph.]

It's also interesting to note that AMD emphasizes Barcelona's power efficiency.  And yet, they introduced a new method of rating their power consumption which shows better values than their old method.  And in addition, in the real-world AnandTech.com was forced to conclude that the performance-per-watt advantage goes to Xeon 5345 by 13% over Barcelona 2350.

[Update: The following table explains one of the benchmarks which has been credited to Xeon as a win. The results within the benchmark indicate Barcelona winning half and Xeon winning half. Only the top-most value was included previously as the maximum stress on the system. However, some more telling spects come from within the test. The benchmark is SPECjbb2005.  It is a Java-based simulation benchmark which performs similar workloads to those found in real-world three-tier data systems. Based on the number of data warehouses the performance graphs out as follows:

Warehouses Barcelona 2350 Xeon 5345 Win %
1 37517 41655 11.03%
2 58758 72551 23.47%
3 74758 87448 16.97%
4 93241 89379 4.32%
5 91586 89655 2.15%
6 91310 89931 1.53%
7 90482 89931 0.61%
8 88275 90482 2.5%
Total values 625,931 651,034 4.01%
Total % 8.62% 53.98%  

Note: Numbers were rounded or truncated to nearest integer. End update.]

 

Conclusion

With Barcelona, there is a lot of data to analyze before you can make heads or tails out of performance claims.  But in such a wide ranging set of tests a prominent picture begins to surface.  AMD's website benchmarks just don't match the real world's.

If cost is key, then Barcelona can offer greater performance with more value (today).  If performance is key, then unless you have specific performance needs and it can be shown that Barcelona will perform better in those particular applications, you'd be better off going with the higher-end Xeons.  They perform proportionally better when they perform better than Barcelona.  And in those cases where they lose, it's only marginally.  In the alternative, it might be worth waiting a few months for the upcoming 45nm Harpertowns.  They will have much less power consumption and will likely be priced to compete very handidly with Barcelona on the high-end.

What should've been AMD's biggest release ever, their native quad-core design, has fallen significantly short of the hope and hype.  The theoretical 2.6 GHz chip would've been wonderful if released today.  The Tech Report was able to benchmark one of those chips and it performed quite well as the 2360 SE.  That chip would've lived up to the expectations.  But it was not released today.  Of the ones available, it shows a processor that does not have a lot of performance-winning life ahead of it when we consider Intel's upcoming 45nm releases just two months and one day away.

Barcelona needed to be a slam dunk for AMD.  It has turned out to be much less.  AMD now needs to focus solely on solving their manufacturing issues and releasing faster clocked Barcelonas.  AMD's customers need to be knowledgeable of the fact that several of Intel's upcoming 45nm products will be here in a few months.  These will likely deliver better performance on less power.

If AMD continues as they have this year.  If they continue to lose large amounts of money each quarter.  And if they are not able to achieve high clock speeds with their 65nm SOI technology at a pace consistent with Intel's anticipated ramping at 45nm, then this launch will be the turning point.  It will prove out to be the beginning of the end for AMD.

Until we saw Barcelona numbers there was always hope.  AMD knew this and kept their cards very close to their chest, not even releasing products for review until late last week.  Unless AMD can turn it around and significantly ramp up the clock speed to compete with and win against Intel's 45 nm competition, then AMD may be headed into a life-threatening storm.  The next six months will be telling for AMD.