While the issues have been mostly found in the Intel server and workstation CPUs, about 2,600 official Intel CPU benchmarks have been invalidated.
Hardware testing and benchmarks remain among the most important part in showing the capabilities of a computer product. It doesn’t matter whether the product is from AMD, Nvidia, Intel or others.
These benchmarks are done via various different software which test these hardware products efficiently. Testing hardware products ranges from CPU to graphics cards to RAM to even SSD and others.
One of the more commonly used benchmarking software for CPUs, especially for server CPUs, is SPEC CPU 2017.
In Intel’s own words, “SPEC CPU 2017 is one of those common, industry-standard benchmark suites that measures and scores general compute-intensive performance.”
Now it seems that Intel is found cheating in the same SPEC CPU 2017 benchmarks.
2,600 Official Intel CPU Benchmarks Invalidated
Phoronix and ServeTheHome reports (via Tom’s Hardware) that SPEC CPU 2017 has stopped publishing results of some of the Intel CPU benchmarks done via the SPEC CPU 2017.
It has also added notice to about 2,600 benchmarks while invalidating them. These benchmark invalidations mostly effect the Intel 4th Gen Sapphire Rapids CPUs. They are server and workstation CPUs and not consumer ones.
For what. Apparently, it is found that Intel CPUs were using a compiler which was found to be wrongly boosting performance specifically only in certain part of SPEC CPU 2017 benchmarks, rather than improve performance in actual workloads. This essentially means that Intel CPUs were cheating in the benchmarks.
Phoronix says that a these benchmark submissions now mention this notice on them:
“SPEC has ruled that the compiler used for this result was performing a compilation that specifically improves the performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks using a priori knowledge of the SPEC code and dataset to perform a transformation that has narrow applicability.
In order to encourage optimizations that have wide applicability (see rule 1.4 https://www.spec.org/cpu2017/Docs/runrules.html#rule_1.4), SPEC will no longer publish results using this optimization.
This result is left in the SPEC results database for historical reference.”
Phoronix also says that performance boost in the benchmarks can be as much as 9% in some case and 4% in another.
It also mentions that the said compiler is an open source one and Intel may or may not have any role in this. It can also be oversight from Intel’s side during the review process.
Conclusion
Intel doing this should not come as a surprise. Two years ago, Intel published a whole PDF trying to essentially say that SPEC CPU 2017 isn’t good enough and everyone should move ahead from it.
It’s not the first time that a combination of Intel CPUs and compilers have found to be in the news for the wrong reasons.
Years ago, Intel was accused of making a compiler that made sure that any software created using that compiler would perform and run slower on AMD CPUs than on Intel ones.
Anyway. For now, this is a developing story. It would be interesting to see what Intel replies.
