Tests

Testing

Next to version control and bug tracking automated testing is important to collaborative software development.

We initiated a test framework in the test subdirectory of the source tree. Invoking {make} in the test directory compiles the test binaries and executes them. Tests are designed such that they write a entry to a xml file. A test is an assertion if the output of some program part meets the specification. There are 3 defined outcomes:

  • failed
  • passed
  • unspecified

We have to distinguish white box and black box tests which both are useful and possible.

White box tests

White box tests is a test written in Fortran and linked with the original source code of the executable. Such the test assertions have access to the data structures, subroutines and parameters of the life program. For reporting the outcome of such a tests a simple Fortran Framework is delivered.
in test/src/testframework

Black box tests

Black box tests merely analyses program output. This analyses is preferably done with a perl script. The test framework includes Perl modules that provide functions for asserting and reporting. The output of a series of tests is a report xml file.

XML

We use xml files because there are simple powerful and everywhere available tools to process the files and the formats can be extended with new fields without braking any of the scripts. In addition to that every Web browser can display and process xml files very efficiently.

a test report looks like

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet href="./report.xsl" type="text/xsl"?>
<report>
  <test>
    <name>EQATOMS.OUT</name>
    <description>EQATOMS.OUT         EQATOMS.diff        </description>
    <directory>test01</directory>
    <status>passed</status>
  </test>
 
 ... 
 
</report>
insert the code here

How to write a test

Write white box test

Tests should be as independent as possible. This this is sometimes difficult to do. A test is passed if the result of a program unit e.g. subroutine matches with the predefined reference data. The test program is a fortran program or routine that calls the to be tested subroutine and verifies the result. It is additional code which is not part of the production tool.

to report the results the {testreport(status} routine writes the report to the test-results xml file.

testunitname="LINENGY.OUT"
inputf="LINENGY.OUT"
outputf="LINENGY.diff"
call testreport(passed)

== Write black box test ==
use lib "../perl/";
use lib "../perl/lib/";
use XML::Simple;
use XML::Writer;
use IO::File;
use Test;
$writer= Test::initreport("report.xml");
 
#do some tests opening files comparing values ...
 
 Test::writetestreport({
         directory=>"test02/runarp",
         name=>"arpack run",
         description=>"The test run  using arpack finished without errors",
         status=>$status
         }, $writer);
 
 Test::closereport($writer);

Generate summary

the summary script in test/perl/summary.pl collects all xml files containing reports, generates the statistics,singles out the failed tests and writes them into separate files

Web view of test results

Today's web browsers have powerful xml processing features which allow to format the raw xml data into nice looking pages. The generated xml files in ./tests/reports are associated with xsl style sheets that generate graphs and generate coloured sorted lists of the test reports.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License