David Ness
Mind / Matter

A Collection of Benchmarks

By David Ness
Saturday, January 26, 2002

The data presented here is very preliminary. I am currently analyzing and checking it.

It is often useful for me to have an idea about how fast different machines are, and how much various software combinations may contribute to or detract from, effective computational speed. There are invariably enough variables involved to make this a non-trivial consideration.

In this study several different pieces of hardware, some of which contain multiple operating systems, each of which can run a number of different pieces of software. The complexities inherent in having several different pieces of hardware, software and basic programs involved.

The Test Systems

Monicker Name Type Speed RAM OS Disk
Vaio800 Sony Vaio Desktop Pentium III 800Mhz 512Mb Win ME 30Gb
Vaio650 Sony Vaio Portable Celeron III 512Mhz 320Mb Win ME 15Gb
Dell200 Dell Dimension Pentium Pro 200Mhz 64Mb NT 4.0 4Gb
2K-500 Dell Inspiron Celeron III 500Mhz 448Mb Win 2K 25Gb
98-500 Dell Inspiron Celeron III 500Mhz 448Mb Win 98 25Gb
NT-500 Dell Inspiron Celeron III 500Mhz 448Mb NT 4.0 25Gb
2K-IDot IDot Desktop AMD-K6 333Mhz 384Mb Win 2K 30Gb
98-IDot IDot Desktop AMD-K6 333Mhz 384Mb Win 98 30Gb
NT-IDot IDot Desktop AMD-K6 333Mhz 384Mb NT 4.0 30Gb
Lin-IDot IDot Desktop AMD-K6 333Mhz 384Mb Linux 30Gb
Compaq Compac X86 75Mhz 32Mb NT 4.0 530Mb
Libretto Toshiba Libretto Pentium I 75Mhz 16Mb Win 95 800Mb
Titanium Apple Titanium Power PC 500Mhz 256Mb OS 9.2 ?

The Test Software

J and K allow us to run tests on both computational and disk-oriented problems. J has the particular advantage that it is implemented for a large number of different system/OS combinations.While the implementations of a particular language may differ across environments, the implementations of J are quite likely to be of similar (high) quality in all cases.

The J used for these tests is J4.06d, except for the Linux test which used J4.02. The K is K 2.9.

The Test OS

Two of the pieces of hardware (-500 and -IDot) have several versions of Windows on them. This allows some comparison of effectiveness of the operating systems themselves. Indeed there is a J implementation that runs under Linux on both of the systems, and we have tested it on one of the two systems so far.

The Exponential / Log Test Problem

The mathematical statement of the calculation is straightforward. This calculation involves taking the first n positive integers, raising them to a non-integral power, and then summing the result. The purpose of this calculation is to simply provoke a considerable number of logarithm and exponential calculations which, in turn, implies lots of multiplications and additions and some divisions. No disk activity, other than that which might be provoked by paging, enters the picture. Here is the code:

In J:

ExpLog =: 3 : 0
   6!:2 'z=:+/1.0001^~>:i.y.'
)

In K:

ExpLog:{+/(1+!x)^1.0001'}

Now we perform the calculations over the range from 100,000 operations to 30,000,000.

Special results are OOM for Out of Memory and Full for Workspace Full. In addition K will produce a Perm result indicating a Permission Error. This arises when one exceeds the internal limits that are imposed on the evaluation version of K.

In developing these results, it also becomes quite obvious, in some circumstances, that the scale of problem is forcing paging activity, swapping information into and out of the RAM of the machine in question. Since this behavior is highly dependent on RAM availability it does not contribute anything useful to our measurement data, and as a result we will rather arbitrarily remove these data points, when we recognize them, from the analysis. These data points are indicated in the subsequent tables by enclosing them in parenthesis.

Benchmarking K

Moniker 100k 300k 1m 3m 10m 30m
Vaio800 0.220  0.610  1.879  3.130  Perm   
Vaio650 0.719  0.990  1.650  3.840  Perm   
Dell200 0.310  0.911  3.154  Full     
2K-500 0.120  0.370  1.201  3.595  Perm   
98-500 0.219  0.939  1.809  4.449  Perm   
NT-500 0.110  0.350  1.181  3.515  Perm   
2K-IDot 0.290  0.901  3.014  9.123  Perm   
98-IDot 0.979  1.869  4.890  13.560  Perm   
NT-IDot 0.260  0.811  2.703  8.622  Perm   
Compaq 2.022  6.188  29.051  Full     
Libretto 1.430  9.440  Full       

Benchmarking J

Moniker 100k 300k 1m 3m 10m 30m
Vaio800 0.057  0.188  0.680  1.779 

5.728 

OOM 
Vaio650 0.066  0.301  0.762  2.128  6.853  OOM 
Dell200 0.189  0.595  3.709  52.502  OOM   
2K-500 0.075  0.225  0.750  2.265  7.509  (59.258)
98-500 0.079  0.249  0.806  2.423  8.193  OOM 
NT-500 0.074  0.223  0.745  2.233  7.726  (42.513)
2K-IDot 0.165  0.495  1.717  5.340  17.357  (147.056)
98-IDot 0.209  0.761  2.105  6.395  21.286  OOM 
NT-IDot 0.163  0.488  1.921  5.448  17.056  (95.140)
Lin-IDot 0.246  0.748  2.519  7.352  24.178  OOM 
Compaq 1.181  4.176  24.205  172.187  OOM   
Libretto 0.561  2.331  13.250  OOM     

Commentary on the tests might prove to be useful.

The Load Data File Test Problem

This test problem involved reading a file each line of which contains 10 numbers. The files range in size from 10,000 lines up to 1,000,000 lines.The problem is to read the lines and convert the ASCII file into a matrix of the appropriate size.

In J:

dn =: 3 : 0
   6!:2 'zz=:". '' '' ((# i.@#)a e. 26 13 10{a.)}a=.1!:1<''c:\'',y.'
)

In K:

c:_i
\t zz:.:'0:c[0]

Benchmarking J

Moniker 10k 30k 100k 300k 1m
Vaio800 0.820  1.539  4.290  11.869  Perm 
Vaio650 0.623  1.630  4.137  11.514  OOM
Dell200 1.141  3.329  OOM     
2K-500 0.393  1.204  3.994  11.937  OOM 
98-500 0.749  1.444  4.126  13.838  OOM 
NT-500 0.803  1.666  4.764  13.658  OOM 
2K-IDot 1.156  2.513  8.457  25.364  (131.068)
98-IDot 0.947  2.795  9.670  28.672  OOM 
NT-IDot 1.054  2.695  8.391  25.249  OOM 
Lin-IDot 0.699  2.237  8.290  23.987  OOM 
Compaq 10.625  27.569  OOM     
Libretto 4.955  OOM       

Benchmarking K

Moniker 10k 30k 100k 300k 1m
Vaio800 0.284  0.866  2.902  8.708  OOM 
Vaio650 0.540  1.980  5.770  14.889  Perm 
Dell200 1.371  4.216  13.299  64.562  Full 
2K-500 0.550  1.642  5.497  16.503  Perm 
98-500 1.039  2.200  6.260  17.739  Perm 
NT-500 0.440  1.321  4.416  13.390  Perm 
2K-IDot 1.502  4.456  14.921  45.986  Perm 
98-IDot 1.589  4.070  12.530  43.890  Perm 
NT-IDot 1.992  6.198  20.519  61.858  Perm 
Compaq 10.925  33.147  109.577  (527.097)  
Libretto 6.369  21.099  73.109  Full   

Analysis of the Data

As a first pass at analysis, a simple linear best fit was performed after eliminating (by eye) some of the data which rather obviously involved swapping memory to disk. The analysis which is present in these tables indicates the correlation (R) in fitting the equation

Time(in seconds) = Alpha + Beta * (number of millions of items)

to the observed calculation times.

J: Math Problem

Name Alpha Beta R Factor
mVaio800 0.0  0.57  1.00  455.5 
mVaio650 0.1  0.68  1.00  441.9 
mDell200 -0.4  4.05  0.99  810.3 
m2K500 0.0  0.75  1.00  375.5 
m98500 -0.0  0.82  1.00  409.8 
mNT500 -0.0  0.77  1.00  386.8 
m2KIDot 0.0  1.74  1.00  578.7 
m98IDot 0.0  2.12  1.00  707.6 
mNTIDot 0.1  1.70  1.00  566.6 
mLinIDot 0.1  2.41  1.00  803.9 
mCompaq -2.5  26.37  1.00  1978.0 
mLibretto -1.4  14.49  1.00  1086.8 

K: Math Problem

Name Alpha Beta R Factor
mVaio800 0.4  0.96  0.96  768.2 
mVaio650 0.6  1.07  1.00  694.5 
mDell200 -0.0  3.17  1.00  634.3 
m2K500 0.0  1.20  1.00  598.3 
m98500 0.3  1.39  0.99  694.1 
mNT500 -0.0  1.17  1.00  586.5 
m2KIDot -0.0  3.05  1.00  1014.3 
m98IDot 0.6  4.33  1.00  1443.5 
mNTIDot -0.1  2.89  1.00  962.5 
mLinDot ---  ---  ---  --- 
mCompaq -1.9  30.72  1.00  2303.9 
mLibretto -2.6  40.05  1.00  3003.8 

J: File Problem

Name Alpha Beta R Factor
mVaio800 1.0  3.59  1.00  2871.1 
mVaio650 1.0  3.49  1.00  2267.2 
mDell200 0.0  109.40  1.00  21880.0 
m2K500 0.6  3.74  1.00  1869.9 
m98500 0.7  4.28  0.99  2139.3 
mNT500 1.0  4.17  1.00  2085.6 
m2KIDot 1.4  7.90  1.00  2630.6 
m98IDot 1.4  9.00  1.00  2996.4 
mNTIDot 1.4  7.85  1.00  2613.5 
mLinIDot 1.2  7.56  1.00  2517.5 
mCompaq 2.2  847.20  1.00  63540.0 
mLibretto -0.1  502.25  1.00  37668.8 

K: File Problem

Name Alpha Beta R Factor
mVaio800 0.4  2.73  1.00  2184.5 
mVaio650 1.2  4.58  1.00  2974.2 
mDell200 -0.1  20.75  0.99  4149.7 
m2K500 0.8  5.17  1.00  2586.5 
m98500 1.3  5.41  1.00  2706.1 
mNT500 0.6  4.20  1.00  2100.1 
m2KIDot 2.1  14.44  1.00  4808.6 
m98IDot 1.6  13.79  0.99  4591.7 
mNTIDot 3.1  19.39  1.00  6458.3 
mLinIDot NA  NA  NA  NA 
mCompaq 20.1  89.65  0.98  6723.5 
mLibretto 12.4  60.80  0.98  4559.8 

Cross-Problem Analysis

Cross-Language Analysis

Cross-OS Analysis

Cross-Architecture Analysis

 

David Ness' summary of work can be found at http://mywebpages.comcast.net/dness

Home