David Ness |
Mind / Matter |
By David Ness
Saturday, January 26, 2002The data presented here is very preliminary. I am currently analyzing and checking it.
It is often useful for me to have an idea about how fast different machines are, and how much various software combinations may contribute to or detract from, effective computational speed. There are invariably enough variables involved to make this a non-trivial consideration.
In this study several different pieces of hardware, some of which contain multiple operating systems, each of which can run a number of different pieces of software. The complexities inherent in having several different pieces of hardware, software and basic programs involved.
The Test Systems
Monicker Name Type Speed RAM OS Disk Vaio800 Sony Vaio Desktop Pentium III 800Mhz 512Mb Win ME 30Gb Vaio650 Sony Vaio Portable Celeron III 512Mhz 320Mb Win ME 15Gb Dell200 Dell Dimension Pentium Pro 200Mhz 64Mb NT 4.0 4Gb 2K-500 Dell Inspiron Celeron III 500Mhz 448Mb Win 2K 25Gb 98-500 Dell Inspiron Celeron III 500Mhz 448Mb Win 98 25Gb NT-500 Dell Inspiron Celeron III 500Mhz 448Mb NT 4.0 25Gb 2K-IDot IDot Desktop AMD-K6 333Mhz 384Mb Win 2K 30Gb 98-IDot IDot Desktop AMD-K6 333Mhz 384Mb Win 98 30Gb NT-IDot IDot Desktop AMD-K6 333Mhz 384Mb NT 4.0 30Gb Lin-IDot IDot Desktop AMD-K6 333Mhz 384Mb Linux 30Gb Compaq Compac X86 75Mhz 32Mb NT 4.0 530Mb Libretto Toshiba Libretto Pentium I 75Mhz 16Mb Win 95 800Mb Titanium Apple Titanium Power PC 500Mhz 256Mb OS 9.2 ? The Test Software
J and K allow us to run tests on both computational and disk-oriented problems. J has the particular advantage that it is implemented for a large number of different system/OS combinations.While the implementations of a particular language may differ across environments, the implementations of J are quite likely to be of similar (high) quality in all cases.
The J used for these tests is J4.06d, except for the Linux test which used J4.02. The K is K 2.9.
The Test OS
Two of the pieces of hardware (-500 and -IDot) have several versions of Windows on them. This allows some comparison of effectiveness of the operating systems themselves. Indeed there is a J implementation that runs under Linux on both of the systems, and we have tested it on one of the two systems so far.
The Exponential / Log Test Problem
The mathematical statement of the calculation is straightforward. This calculation involves taking the first n positive integers, raising them to a non-integral power, and then summing the result. The purpose of this calculation is to simply provoke a considerable number of logarithm and exponential calculations which, in turn, implies lots of multiplications and additions and some divisions. No disk activity, other than that which might be provoked by paging, enters the picture. Here is the code:
In J:
ExpLog =: 3 : 0 6!:2 'z=:+/1.0001^~>:i.y.' )In K:
ExpLog:{+/(1+!x)^1.0001'}Now we perform the calculations over the range from 100,000 operations to 30,000,000.
Special results are OOM for Out of Memory and Full for Workspace Full. In addition K will produce a Perm result indicating a Permission Error. This arises when one exceeds the internal limits that are imposed on the evaluation version of K.
In developing these results, it also becomes quite obvious, in some circumstances, that the scale of problem is forcing paging activity, swapping information into and out of the RAM of the machine in question. Since this behavior is highly dependent on RAM availability it does not contribute anything useful to our measurement data, and as a result we will rather arbitrarily remove these data points, when we recognize them, from the analysis. These data points are indicated in the subsequent tables by enclosing them in parenthesis.
Benchmarking K
Moniker 100k 300k 1m 3m 10m 30m Vaio800 0.220 0.610 1.879 3.130 Perm Vaio650 0.719 0.990 1.650 3.840 Perm Dell200 0.310 0.911 3.154 Full 2K-500 0.120 0.370 1.201 3.595 Perm 98-500 0.219 0.939 1.809 4.449 Perm NT-500 0.110 0.350 1.181 3.515 Perm 2K-IDot 0.290 0.901 3.014 9.123 Perm 98-IDot 0.979 1.869 4.890 13.560 Perm NT-IDot 0.260 0.811 2.703 8.622 Perm Compaq 2.022 6.188 29.051 Full Libretto 1.430 9.440 Full Benchmarking J
Moniker 100k 300k 1m 3m 10m 30m Vaio800 0.057 0.188 0.680 1.779 5.728
OOM Vaio650 0.066 0.301 0.762 2.128 6.853 OOM Dell200 0.189 0.595 3.709 52.502 OOM 2K-500 0.075 0.225 0.750 2.265 7.509 (59.258) 98-500 0.079 0.249 0.806 2.423 8.193 OOM NT-500 0.074 0.223 0.745 2.233 7.726 (42.513) 2K-IDot 0.165 0.495 1.717 5.340 17.357 (147.056) 98-IDot 0.209 0.761 2.105 6.395 21.286 OOM NT-IDot 0.163 0.488 1.921 5.448 17.056 (95.140) Lin-IDot 0.246 0.748 2.519 7.352 24.178 OOM Compaq 1.181 4.176 24.205 172.187 OOM Libretto 0.561 2.331 13.250 OOM Commentary on the tests might prove to be useful.
The Load Data File Test Problem
This test problem involved reading a file each line of which contains 10 numbers. The files range in size from 10,000 lines up to 1,000,000 lines.The problem is to read the lines and convert the ASCII file into a matrix of the appropriate size.
In J:
dn =: 3 : 0 6!:2 'zz=:". '' '' ((# i.@#)a e. 26 13 10{a.)}a=.1!:1<''c:\'',y.' )In K:
c:_i \t zz:.:'0:c[0]Benchmarking J
Moniker 10k 30k 100k 300k 1m Vaio800 0.820 1.539 4.290 11.869 Perm Vaio650 0.623 1.630 4.137 11.514 OOM Dell200 1.141 3.329 OOM 2K-500 0.393 1.204 3.994 11.937 OOM 98-500 0.749 1.444 4.126 13.838 OOM NT-500 0.803 1.666 4.764 13.658 OOM 2K-IDot 1.156 2.513 8.457 25.364 (131.068) 98-IDot 0.947 2.795 9.670 28.672 OOM NT-IDot 1.054 2.695 8.391 25.249 OOM Lin-IDot 0.699 2.237 8.290 23.987 OOM Compaq 10.625 27.569 OOM Libretto 4.955 OOM Benchmarking K
Moniker 10k 30k 100k 300k 1m Vaio800 0.284 0.866 2.902 8.708 OOM Vaio650 0.540 1.980 5.770 14.889 Perm Dell200 1.371 4.216 13.299 64.562 Full 2K-500 0.550 1.642 5.497 16.503 Perm 98-500 1.039 2.200 6.260 17.739 Perm NT-500 0.440 1.321 4.416 13.390 Perm 2K-IDot 1.502 4.456 14.921 45.986 Perm 98-IDot 1.589 4.070 12.530 43.890 Perm NT-IDot 1.992 6.198 20.519 61.858 Perm Compaq 10.925 33.147 109.577 (527.097) Libretto 6.369 21.099 73.109 Full Analysis of the Data
As a first pass at analysis, a simple linear best fit was performed after eliminating (by eye) some of the data which rather obviously involved swapping memory to disk. The analysis which is present in these tables indicates the correlation (R) in fitting the equation
Time(in seconds) = Alpha + Beta * (number of millions of items)to the observed calculation times.
J: Math Problem
Name Alpha Beta R Factor mVaio800 0.0 0.57 1.00 455.5 mVaio650 0.1 0.68 1.00 441.9 mDell200 -0.4 4.05 0.99 810.3 m2K500 0.0 0.75 1.00 375.5 m98500 -0.0 0.82 1.00 409.8 mNT500 -0.0 0.77 1.00 386.8 m2KIDot 0.0 1.74 1.00 578.7 m98IDot 0.0 2.12 1.00 707.6 mNTIDot 0.1 1.70 1.00 566.6 mLinIDot 0.1 2.41 1.00 803.9 mCompaq -2.5 26.37 1.00 1978.0 mLibretto -1.4 14.49 1.00 1086.8 K: Math Problem
Name Alpha Beta R Factor mVaio800 0.4 0.96 0.96 768.2 mVaio650 0.6 1.07 1.00 694.5 mDell200 -0.0 3.17 1.00 634.3 m2K500 0.0 1.20 1.00 598.3 m98500 0.3 1.39 0.99 694.1 mNT500 -0.0 1.17 1.00 586.5 m2KIDot -0.0 3.05 1.00 1014.3 m98IDot 0.6 4.33 1.00 1443.5 mNTIDot -0.1 2.89 1.00 962.5 mLinDot --- --- --- --- mCompaq -1.9 30.72 1.00 2303.9 mLibretto -2.6 40.05 1.00 3003.8 J: File Problem
Name Alpha Beta R Factor mVaio800 1.0 3.59 1.00 2871.1 mVaio650 1.0 3.49 1.00 2267.2 mDell200 0.0 109.40 1.00 21880.0 m2K500 0.6 3.74 1.00 1869.9 m98500 0.7 4.28 0.99 2139.3 mNT500 1.0 4.17 1.00 2085.6 m2KIDot 1.4 7.90 1.00 2630.6 m98IDot 1.4 9.00 1.00 2996.4 mNTIDot 1.4 7.85 1.00 2613.5 mLinIDot 1.2 7.56 1.00 2517.5 mCompaq 2.2 847.20 1.00 63540.0 mLibretto -0.1 502.25 1.00 37668.8 K: File Problem
Name Alpha Beta R Factor mVaio800 0.4 2.73 1.00 2184.5 mVaio650 1.2 4.58 1.00 2974.2 mDell200 -0.1 20.75 0.99 4149.7 m2K500 0.8 5.17 1.00 2586.5 m98500 1.3 5.41 1.00 2706.1 mNT500 0.6 4.20 1.00 2100.1 m2KIDot 2.1 14.44 1.00 4808.6 m98IDot 1.6 13.79 0.99 4591.7 mNTIDot 3.1 19.39 1.00 6458.3 mLinIDot NA NA NA NA mCompaq 20.1 89.65 0.98 6723.5 mLibretto 12.4 60.80 0.98 4559.8 Cross-Problem Analysis
Cross-Language Analysis
Cross-OS Analysis
Cross-Architecture Analysis
David Ness' summary of work can be found at http://mywebpages.comcast.net/dness