# A testing system for Boost.Build

Introduction for users
Introduction for developers
  Changing the working directory
  Examining the working directory and changing it
  Test result
Reference documentation
  Method __init__
  Method `set_tree`
  Method `write`
  Method `copy`
  Method `touch`
  Method `run_build_system`
  Method `read`
  Method `read_and_strip`
  Methods for declaring expectations
  Methods for ignoring changes
  Methods for explicitly specifying results
  Helper class `List`

## Introduction for users

The testing system for Boost.Build is a small set of Python modules and scripts for automatically testing user-obversable behaviour. It uses components from testing systems of Scons and Subverion, together with some additional functionality.

To run the tests you'd need:

1. Get the source tree of Boost.Build (located at `tools/build` in Boost)
2. Have Python installed. Version 2.1 is known to work.
3. Build Boost.Jam. See $boost_build_root/jam/index.html for instructions.
4. Configure at least one toolset. You can edit `site-config.jam` or `user-config.jam` to add new toolsets. Or you can create file `test-config.jam` in `$boost_build_root/test` directory. In this case, `site-config.jam` and `user-config.jam` will be ignored for testing.

When all is done, you can run the tests with

    python test_all.py

which will use the default toolset, or you can specify toolset on the command line, for example:

    python test_all.py borland

If everything's OK, you'll see a list of passed tests. Otherwise, a failure will be reported.

It is possible to run a specific test, for example:

    python generators_test.py

## Introduction for developers

It is suggested that every new functionality come together with tests, and that bugfixes are accompanied by tests. There's no need to say that tests are good, but two points are extremely important:

- For an interpreted language like Jam, without any static checks, testing is simply the only sefeguard we can have.
- Good tests allow to change internal design much more safely, and we didn't nailed everything down yet.

Adding a new test is simple:

1. Go to `$boost_build_root/test/test_all.py` and add new test name to the list at the end of file. Suppose the test name is "hello".
2. Add a new python module, in this example "hello.py", to do actual testing.

The module, in general will perform these basic actions:

1. Setting the initial working directory state
2. Running the build system and checking:
   1. generated output,
   2. changes made to the working directory,
   3. new content of the working directory.
3. Adding, removing or touching files, or changing their content and then repeating the previous step, until satisfied.
4. Cleaning up

The "hello.py" module might contain:

    from BoostBuild import Tester, List

    # Create a temporary working directory
    t = Tester()

    # Create the needed files
    t.write("project-root.jam", "")
    t.write("Jamfile", """
    exe hello : hello.cpp ;
    """)
    t.write("hello.cpp", """
    int main()
    {
        return 0;
    }

    """)

    t.run_build_system()

    # First, create a list of three pathnames
    file_list = List("bin/$toolset/debug/") * List("hello.exe hello.obj")
    # Second, assert that those files were added as result of the last build system invocation.
    t.expect_addition(file_list)

    # Invoke the build system once again
    t.run_build_system("clean")
    # Check if the files added previously were removed.
    t.expect_removal(file_list)

    # Remove temporary directories
    t.cleanup()

The `test` directory contains a file "template.py" which can be used as a start for your own tests.

Overview of the most important methods of class `TestBoostBuild` follows.

### Changing the working directory

The class `TestBoostBuild` creates a temporary directory in its constructor and changes to that directory. It can be modified by calling these methods:

- `set_tree` -- sets the content of the working directory to be equal to the content of the specified directory. This method is preferrable when directory tree for testing is large.
- `write` -- sets the content of file in a working directory. This is optimal if you want to create a directory tree with 3-4 small files.
- `touch` -- changes the modification times of a file

### Examining the working directory and changing it

The method `read`, inherited from the `TestCmd` class, can be used to read any file in the working directory and check its content. `TestBoostBuild` adds another method for tracking changes. Whenever build system is run (via `run_build_system`), the state of working dir before and after running is recorded. In addition, difference between the two states -- i.e. lists of files that were added, removed, modified or touched -- is stored in two member variables, `tree_difference` and `unexpected_difference`.

After than, the test author may specify that some change is expected, for example, by calling `expect_addition("foo")`. This call will check if the file was indeed added, and if so, will remove its name from the list of added files in `unexpected_difference`. Likewise, it's possible to specify that some changes are not interesting, for example a call `ignore("*.obj")` will just remove every files with ".obj" extension from `unexpected_difference`.

When test has finished with expectations and ignoring, the member `unexpected_difference` will contain the list of all changes not yet accounted for. It is possible to assure that this list is empty by calling `expect_nothing_more` member function.

### Test result

Any of the `expect*` methods below will fail the test if the expectation is not met. It is also possible to perform manually arbitrary test and explicitly cause the test to either pass or fail. Ordinary filesystem functions can be used to work with the directory tree. Methods `pass_test` and `fail_test` are used to explicitly give the test outcome.

Typically, after test termination, the working directory is erased. To debug a failed test, you should add "--preserve" option when invoking test. On failure, the working directory will be copied to "failed_test" directory in the current dir.

## Reference documentation

The test system is composed of class `Tester`, derived form `TestCmd.TestCmd`, and helper class `List`. The methods of `Tester`, and the class `List` are described below.

The documentation frequently refer to filename. In all cases, files are specified in unix style: a sequence of components, separated by "/". This is true on all platforms. In some contexts, a list of files is allowed. In that case any object with sequence interface is allowed.

### Method `__init__(self, workdir='', arguments='', executable='bjam')`

Effects:

1. Remembers the current working directory in member `original_workdir`.
2. Determines the location of executable (`bjam` by default) and build system files, assuming that the current directory is `tools/build/test`. Formulates jam invocation command, which will include explicit setting for `BOOST_BUILD_PATH` variable and arguments passed to this methods, if any. This command will be used by subsequent invocation of `run_build_system`. Finally, initializes the base class.
3. Changes current working dir to the temporary working directory created by the base constructor.
4. If you want to run a test in a existing directory, pass it to `workdir`.

### Method `set_tree(self, tree_location)`

Effects:

Replaces the content of the current working directory with the content of directory at `tree_location`. If `tree_location` is not absolute pathname, it will be treated as relative to `self.original_workdir`. This methods also explicitly makes the copied files writeable.

### Method `write(self, name, content)`

Effects:

Writes the specified content to the file given by `name` under the temporary working directory. If the file already exists, it is overwritten. Any required directories are automatically created.

### Method `copy(self, src, dst)`

Effects:

Equvivalent to `self.write(self.read(src), dst)`.

### Method `touch(self, names)`

Effects:

Sets the access and modification times for all files in `names` to the current time. All the elements in `names` should be relative paths.

### Method `run_build_system(self, subdir='', extra_args='', stdout=None, stderr='', status=0, **kw)`

Effects:

1. Stores the state of the working directory in `self.previous_tree`.
2. Changes to `subdir`, if it is specified. It is relative to the `original_workdir` or the workdir specified in `__init`.
3. Invokes the `bjam` executable, passing `extra_args` to it. The binary should be located under `<test_invocation_dir>/../jam/src/bin.<platform>`. This is to make sure tests use the version of jam build from CVS.
4. Compares the stdout, stderr and exit status of build system invocation with values to appropriate parameters, if they are not `None`. If any difference is found, the test fails.

5. Stores the new state of the working directory in `self.tree`. Computes the difference between previous and current trees and store them in variables `self.tree_difference` and `self.unexpected_difference`.

   Both variables are instances of class `tree.Trees_different`, which have four attributes: `added_files`, `removed_files`, `modified_files` and `touched_files`. Each is a list of strings.

### Method `read(self, name)`

Effects:

Read the specified file and returns it content. Raises an exception is the file is absent.

### Method `read_and_strip(self, name)`

Effects:

Read the specified file and returns it content, after removing trailing whitespace from every line. Raises an exception is the file is absent.

Rationale:

Althought this method is questionable, there are a lot of cases when jam or shells it uses insert spaces. It seems that introducing this method is much simpler than dealing with all those cases.

### Methods for declaring expectations

Accordingly to the number of changes kinds that are detected, there are four methods that specify that test author expects a specific change to occur. They check `self.unexpected_difference`, and if the change is present there, it is removed. Otherwise, test fails.

Each method accepts a list of names. Those names use `/` path separator on all systems. Additionaly, the test system translates suffixes appropriately. For the test to be portable, suffixes should use Windows convention: `exe` for executables, `dll` for dynamic libraries and `lib` for static libraries. Lastly, the string "$toolset" in file names is replaced by the name of tested toolset.

Note: The `List` helper class might be useful to create lists of names.

Note: The file content can be examined using `TestCmd.read` function.

The members are:

- expect_addition
- expect_removal
- expect_modification
- expect_nothing

Note that `expect_modification` is used to check that a either file content or timestamp has changed. The rationale is that some compilers change content even if sources does not change, and it's easier to have a method which checks for both content and time changes.

There's also a member `expect_nothing_more`, which checks that all the changes are either expected or ignored, in other words that `unexpected_difference` is empty by now.

Lastly, there's a method to compare file content with expected content:

`expect_content(self, name, content, exact=0)`

The method fails the test if the content of file identified by 'name' is different from 'content'. If 'exact' is true, the file content is used as-is, otherwise, two transformations are applied:

- The `read_and_strip` method is used to read the file, which removes trailing whitespace
- Each backslash in the file content is converted to forward slash.

### Methods for ignoring changes

There are five methods which ignore changes made to the working tree. They silently remove elements from `self.unexpected_difference`, and don't generate error if element is not found. They accept shell style wildcard.

The following methods correspond to four kinds of changes:

- ignore_addition(self, wildcard)
- ignore_removal(self, wildcard)
- ignore_modification(self, wildcard)
- ignore_touch(self, wilcard)

The method `ignore(self, wildcard)` ignores all the changes made to files that match a wildcard.

### Methods for explicitly specifying results

#### Method `pass_test(self, condition=1)`

At this moment, the method should not be used.

#### Method `fail_test(self, condition=1)`

Effects: Cause the test to fail if `condition` is true.

### Helper class `List`

The class has sequence interface and two additional methods.

#### Method `__init__(self, string)`

Effects: Splits the string on unescaped spaces and tabs. The split components can further be retrieved using standard sequence access.

#### Method `__mul__(self, other)`

Effects: Returns an `List` instance, which elements are all possible concatenations of two string, first of which is from `self`, and second of which is from `other`.

The class also defines `__str__` and `__repr__` methods. Finally, there's `__coerce__` method which allows to convert strings to instances of `List`.

Example:

        l = "a b" * List("c d")
        for e in l:
            print e

will output

        ac
        ad
        bc
        bd

Last modified: Mar 11, 2005

© Copyright Vladimir Prus 2002, 2003, 2004, 2005. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided ``as is'' without express or implied warranty, and with no claim as to its suitability for any purpose.