Next: Sources used in
Up: pseudoknot.html
Previous: Operational behaviour
The Pseudoknot program had to be translated by hand from the Scheme or C
versions to the various other languages. All versions were
hand tuned to exhibit the best performance available with the compiler
being used. The following set of guidelines has been used to make the
comparison as fair as possible:
-
Algorithmic changes are forbidden but slight modifications to the code
to better use a particular feature of the language or programming
system are allowed. A modification that would be valid for all
languages involved is not permitted.
-
Only small changes to the data structures are permitted (e.g. a tuple
may be turned into an array or a list).
-
Annotations are permitted, for example strictness annotations, or
annotations for inlining and specialisation of code.
-
All changes and annotations should be documented.
-
Anyone should be able to repeat the experiments. So all sources and
measurement procedures should be made public (by ftp somewhere).
-
All programs must produce the same output (the number 33.7976 to 6
significant figures).
The optimisations and annotations made to obtain best performance with
each of the compilers are discussed in the subsections to follow. Often
we will make reference to particular parts of the program text. As the
program is relatively large it would be difficult to reproduce it in
full here, and in many versions. The reader is invited to consult the
archive that contains most of the versions of the program at
ftp.fwi.uva.nl, file /pub/computer-systems/functional/pseudoknot.tar.Z.
The guidelines above were designed on the basis of the experience
gained at the Dagstuhl workshop with a small subset of the present,
much larger set of implementations. We tried to make the guidelines as
clear and concise as possible, yet they were open to different
interpretations. The problems we had were aggravated by the use of
different terminology, in particular when one term means different
things to people from different backgrounds. During the process of
interpreting and integrating the benchmarking results in the paper we
have made every effort to iron out the differences that we found.
Clearly there are some left that we are not aware of.
In addition to these unintentional differences, there are intentional
differences: some experimentors spent more time and effort in
improving their version of Pseudoknot than others. These efforts have
been documented in the sections to follow.
The impact of this combined variability of efforts, intentional and
unintentional, is very difficult to assess. A formal method for
performing benchmarks is clearly needed.
Next: Sources used in
Up: pseudoknot.html
Previous: Operational behaviour