Introduction

Atheist is a simple framework for running test cases. You write small files in Python language using a set of predefined functions and classes. In many senses, the concept is quite similar to make or the SCons framework although Atheist is not a building system at all.

Features:

ToDo:

  • Per-project plugins.
  • Limit/perf testing.
  • Remote testing.
  • Test deployment.
  • Remote management.
  • Distributed testing.

Test objects

The Test object is the minimal testing unit. Each Test instance defines an individual execution (a shell command) that may be checked for success upon termination. The Test constructor accepts many parameters that may change the test’s exception behavior in several ways. The only mandatory parameter is cmd which is the command to execute.

The Test object is responsible for executing the command and checking its termination value. A very basic example:

Test('true')

Test files (TestCases)

Test instances need to be defined in text source files (with .test extension). Although these files are written in a subset of the Python language, they may be seen as declarative programs. You tell Atheist what you want to test and even the order, but the decision about when to run the corresponding action is taken by Atheist; some of them may be never done.

The file does not define sequential imperative sentences. For example, if you write this in a .test file:

Test('ls /')
print "hello"

the print statement will be executed when the test is LOADED and the ls command will run later, when the test is EXECUTED. You must take in mind that the atheist .test file defines a set of tests. It is not a conventional python program.

Key-args

Any Test constructor accepts the next key-val parameters. All of them are optional. Beside the parameter’s name appear its type and default value.

check – type: bool, default: True

If ‘check’ is False there is no consequences when the Task fails and it is not considered in the final stats.

cwd – type: str

Directory for the task execution.

delay – type: int, default: 0

Waits for ‘delay’ seconds before launching the task actions.

desc – type: str

One-liner textual task description.

detach – type: bool, default: False

When detach is False the next task does not start until the current one ends. When detach is True the next task is executed even if the current one is still running.

env – type: str:str map

A dictionary with shell environment variables.

expected – type: int

Expected return code for the command. It may be negative if the process is killed by a signal.

tid – type: str

It is a unique string Task IDentifier. You can get a reference to the task later by giving this value to the get_task() function.

must_fail – type: bool, default: False

When you expect the program to end with an error but the return code is not known (i.e: is different from zero). You should check other conditions (stdout, stderr, generated files, etc) to differentiate alternative fails.

parent – type: CompositeTask

Use this to aggregate the current test to an already defined CompositeTask.

template – type: Template list

A list of templates. See Templates.

timeout – type: int, default: 5

The maximum task execution time (in seconds). When the timeout finishes, Atheist sends the programmed signal to the process. To avoid timed termination (daemonic task) give timeout=0.

save_stderr – type: bool, default: False

Store the process’ stderr in a text file. If the stderr parameter is not set, Atheist will create an unique name for the file.

save_stdout – type: bool, default: False

Store the process’ stdout in a text file. If the stdout parameter is not set, Atheist will create an unique name for the file.

shell – type: bool, default: False

Execute the command within a shell session. bash is the used shell.

signal – type: int, default: SIGKILL

It is the signal that Atheist sends to the process when the timeout finishes.

stdout – type: str

It is the name of the file where to save the process’ stdout. Setting this parameters implies save_stdout = True.

todo – type: bool, default: False

It indicates that the task is not fully verified and it is possible that it fail unduly. This has no effect when the task ends successfully.

Not all of these key-args are available for all Task classes. See Task’s, Test’s, Commands, Daemons....

Task results

The execution of any task returns a value, which can be:

  • FAIL: The task ran normally but user requirements or conditions were not met. The test failed.
  • OK: The task ran successfully and all required conditions and/or return values were correct.
  • NOEXEC: The task was skipped or it was not executed.
  • ERROR: The implementation of the task is wrong and the task execution failed itself.
  • UNKNOWN: The task was executed but its result is not known.
  • TODO: The task implementation is unstable and it may produce false failures.

Templates

The template is a set of predefined values for Test key-values. You may use the same configuration for many tests avoiding to repeat them. This is an example:

t1 = Template(timeout=6, expected=-9)
Test('foo', templates=[t1])
Test('bar', templates=[t1])

Both tests will be automatically killed after 6 seconds and the expected return value is -9. This means that these processes receive the SIGKILL(9) signal. You may specify several templates as a list.

Conditions

Conditions are predicates (actually, functors) that check for specific conditions. Conditions may be specified to be checked before (pre-conditions) or after (post-conditions) the task execution. If any of the conditions fail, then the task fails. This is an example:

t = Test('foo')
t.pre  += FileExists('/path/to/foofile')
t.post += FileContains('path/to/barfile', 'some text')

Available builtin conditions

AtheistVersion(version)

Checks that the installed version of atheist is equal or newer than the given number. This is useful to assure recent or experimental features.

Callback(*args)

Call the specified function with the given arguments. You must avoid the use of this condition as much as possible.

DirExists(path)

Checks that directory path exists.

EnvVarDefined(name[, value])

Checks that the environment variable name exists, and optionally has the value value.

FileExists(filename)

Checks that file filename exists.

FileContains(val[, filename=task.stdout, strip='', whole=False, times=1])

Checks that file filename exists and contains val, which may be a string or a string list.

If whole is False (default) val must be a strip and the whole content of file must be equal to val.

Otherwise, the file must contain at least times occurrences of val. The default value for filename is the stdout of the corresponding task, and it implies save_stdout=True. This implies also the automatic creation of a FileExists condition for that file.

The file content may be stripped by means of strip argument. No stripping by default.

FileEquals(filename1[, filename2=task.stdout])

Checks that the contents of filename1 and filename2 are identical. The default value for filename2 is the stdout of the current task, and it implies save_stdout=True.

OpenPort(port[, host='localhost'[, proto='tcp']])

Checks that port number port is open, i.e., a process is listening to it.

ProcessRunning(pid)

Checks that the given PID belongs to a running process.

There are other available conditions as plugins.

Condition decorators

Not(condition)

It is True when condition is evaluated as False.

Example:

t = Test('rm foo_file')
t.pre += Not(FileExists(foo_file))
Poll(condition[, interval=1[, timeout=5]])

Checks condition every interval seconds, stopping after timeout seconds or when its value becomes True.

In the next example, the task waits (a maximum of 5 seconds) for the nc server to become ready before continuing:

t = Test('nc -l -p 2000')
t.post += Poll(OpenPort(2000))

Condition decorators may be combined. The following example shows a task that waits for an environment variable to be removed before executing the command:

t = Test('command')
t.pre += Poll(Not(EnvVarDefined('WAIT_FLAG')), timeout=10)

Note that the effect of Poll(Not(condition)) is not the same that Not(Poll(condition)).

Task’s, Test’s, Commands, Daemons...

Task is the base class for all executable items. Test is-a Task that runs a shell command but other kind of Task are possible:

Command

It is a non-checked Test. Command is exactly the same that a Test with a check=False parameter.

The Commands (or other non-checked Tasks) are not considered in results counting.

Daemon

Command shortcut for detached commands. Predefined parameters are:

  • detach = True
  • expected = -9 (sigkilled)
  • timeout = None (runs “forerver”)
  • check = False
TestFunc
Check the return value of arbitrary Python functions or methods. A return value of 0 means success, otherwise is a error code. For unit testing, prefer `unittestcase`_ instead of TestFunc.

There are other available Task subclasses as plugins.

Function utilities

get_task(name)

Returns the task whose tid attribute is name. ToDo: [include a sample here]

load(filename)

Makes possible to reuse atheist or python code in other files. load() returns a module-like object that may be used to access to functions, classes and variables defined in the “loaded” module. All atheist classes are available within the loaded modules:

common = load("files/common.py")

Test("./server %s" % common.server_params())

Warning

Previous include() function is not supported any more.

Variable substitutions

Test files may include some substitutable variable. This is useful to locate test-relevant related files. You must write the symbol ‘$’ preceding each one of the next words:

basedir
is the name of the directory where atheist was executed. Usually this is a tests directory into your project.
dirname
is the name of the directory where the testfile is.
fname
is the path to the testfile without its extension (.test).
testname
is just the name of the testfile, without extension nor directory path.

For example, for the following vars.test file:

Test("echo $basedir $dirname $fname $testname")

When you run atheist, you get:

~/sample$ atheist -i2 tests/vars.test
[ OK ] Test case: ./tests/substitutions.test
[ OK ] +- T-1   ( 0: 0)  echo . ./tests ./tests/vars vars
Total:  ALL OK!!  - 0.24s - 1 test

hooks: setup and teardown

You may write tasks to execute before and after each test file. To do so, just put this tasks in files called _setup.test and _teardown.test.

Clean-up

When your task creates files you may track them for automatic cleaning. Just add their filenames to the task gen attribute. Here’s an example:

t = Test('touch foo')
t.gen += 'foo'

You may specify one o more filenames (as a string list).

If you want the generated files not to be automatically removed for manual inspection of the results, you must specify the --dirty option (see below). To clean-up these files later, specify the -C option.

Invoking Atheist

-h, --help

Show basic help information.

-a ARGS, --task-args=ARGS

Colon-separated options for the tests.

-b PATH, --base-dir=PATH

Change working directory.

-C, --clean-only

Don’t execute anything, only remove generated files.

-d, --describe

Describe tasks but don’t execute anything.

-e, --stderr

Print the test process’ stderr.

-f, --out-on-fail

Print task output, but only if it fails.

-g, --gen-template

Generate a test file template with default values.

-i LEVEL, --report-detail=LEVEL

Verbosity level (0:nothing, [1:case], 2:task, 3:composite, 4:condition)

-j, --skip-hooks

Skip _setup and _teardown files.

-k, --keep-going

Continue despite failed tests.

-l, --list

List tests but do not execute them.

-o, --stdout

Print the test process’ stdout.

-p PATH, --plugin-dir=PATH

Load plugins from that directory (may be given several times).

Print the test process’ stdout.

-q, --quiet

Do not show result summary nor warnings, only totals.

-r RANDOM, --random=RANDOM

Run testcases in random order using the specified seed.

-s INLINE, --script=INLINE

Specifies command line script.

-t, --time-tag

Include time info in the logs.

-u, --until-failure

Repeat tests until something fails.

-v, --verbose

Increase verbosity.

-w WORKERS, --workers=WORKERS

Number of simultaneous tasks. ‘0’ allows atheist to choose the number. Default is 1.

-x COMMAND, --exec=COMMAND

Exec COMMAND like in “Test(COMMAND, shell=True)”

--case-time

Print case execution time in reports

--cols=WIDTH

Set terminal width (in chars).

--config=FILE

Alternate config file.

--dirty

Don’t remove generated files after test execution.

--disable-bar

Don’t show progress bar.

--ignore=PATTERN

Files to ignore (glob patterns) separated with semicolon.

--log=LOG

List tasks but do not execute them.

--plain

Avoid color codes in console output.

--save-stdout

Save stdout of all tasks

--save-stderr

Save stderr of all tasks

--version

Atheist version

--notify-jabber=JABBER

Notify failed tests to the given jabber account (may be given several times).

--notify-smtp=MAIL

Notify failed tests to the given email address (may be given several times).

Logging control

[ToDo]

Result report

[ToDo]

Config files

[ToDo]

Notifications (other reporters)

You may use Atheist to monitor the proper working of any application. It may send you notifications when something is wrong. The easiest way is to run a testcase with cron specifying one --notify-* command-line argument. Currently, two notificators are implemented:

Email

The destination is a email account using the SMTP protocol. Atheist does not require an SMTP server or smarthost. You only need to configure an email account that will be used by Atheist to send mail to any destination. That information must be written in the ~/.atheist/config configuration file. This is an example:

[smtp]
host = smtp.server.org
port = 587
user = atheist.notify@server.org
pasw = somesecret
Jabber

You must indicate a jabber account that will be used by Atheist to send notification messages to any other jabber account. The destination account must accept this contact previously. The following is an example for the configuration in the ~/.atheist/config file:

[jabber]
user = atheist.notify@server.org
pasw = anothersecret

To ask for a notification you just need to specify a test file and one or more destinations. For example:

$ atheist --notify-jabber John.Doe@jabber.info test/some_test.test

It is possible to give several –notify-* arguments in the same command-line to send notifications to different destinations.