dtest
A C++ testing framework for performing unit tests, distributed tests, and performance tests.
Usage
- Add Makefile.template in your project's test directory.
- Write your test modules (must have extension .dtest.cpp and #include <dtest.h>).
- Build and run tests.
- Check the test results in the output file dtest.log.json.
Syntax
1. Test Modules
Tests for a given project are divided into modules. Each module can contain one or more tests. Modules can have dependencies (i.e. an entire test module cannot run unless its module dependencies are run without errors).
module("module-name")
.dependsOn({
"module-a",
"module-b"
});
2. Test Status
Each test run has a status. The status describes the general category of the result of the test. The output file dtest.log.json will have more detailed information on every test.
Status | Description |
---|---|
Status:: PASS | Test passed with no errors detected. |
Status:: PASS_WITH_MEMORY_LEAK | Test passed but has a memory leak. |
Status:: MEMORY_LIMIT_EXCEEDED | Test passed but a set memory limit was exceeded. |
Status:: TOO_SLOW | Test did not meet performance requirements. |
Status:: TIMEOUT | Test duration exceeded the specified timeout value. |
Status:: FAIL | Test has failed (assertion error, uncaught exception, segfault, etc.). |
3. Unit Tests
Unit tests provide an easy and fast way to verify that your components behave as expected. You can write unit tests as follows:
unit("module-name", "test-name")
.option()
.body([] {
// test code here
assert(1 == 1);
});
The macro assert() can accept any C++ expression that can be evaluated as a boolean. If the expression evaluates to false, an assertion error is generated.
In addition to assertions, unit tests check for memory leaks, timeouts, segfaults, etc. The framework runs the tests inside a sandbox that measures cpu time, memory allocations, and network activity.
Each test can have any number of options set to control its behavior. The available options are as follows:
Option | Description |
---|---|
.body | The main body of the test is defined using this option. This accepts either a (void)->void lambda or a function of the same signature. |
.onInit | Adds a code block to run before the beginning of the main test body. This accepts either a (void)->void lambda or a function of the same signature. |
.onComplete | Adds a code block to run after the end of the main test body. This accepts either a (void)->void lambda or a function of the same signature. |
.timeout | Specifies a timeout duration in seconds. If the test takes longer than this duration, it is terminated and considered to fail. (default = 10 seconds) |
.timeoutMillis | Specifies a timeout duration in milliseconds. If the test takes longer than this duration, it is terminated and considered to fail. (default = 10 seconds) |
.timeoutMicros | Specifies a timeout duration in microseconds. If the test takes longer than this duration, it is terminated and considered to fail. (default = 10 seconds) |
.timeoutNanos | Specifies a timeout duration in nanoseconds. If the test takes longer than this duration, it is terminated and considered to fail. (default = 10 seconds) |
.memoryBytesLimit | Sets a limit on the maximum amount of memory (in bytes) allocated. |
.memoryBlocksLimit | Sets a limit on the maximum number of memory blocks allocated. |
.expect | Sets the expected test status. If the test status is different from the expected, it is considered as a failed test. (default = Status::PASS) |
.disable | Disables the test. |
.enable | Enables the test. |
.dependsOn | Adds extra dependencies for this test (in addition to the module-wide dependencies). |
.ignoreMemoryLeak | Does not perform a memory leak check at the end of the test. |
.inProcess | Runs the test in a local sandbox for debugging. The default behavior is to run the test in a separate process to ensure the best possible isolation between tests. |
.input | Sets an input string to be fed to the test through stdin. |
4. Distributed Unit Tests
Distributed unit tests run tests to validate components that typically depend on the interaction between multiple processes or machines. Instead of a single test body, the test is divided into two blocks of code: driver and worker. A single instance of the driver test code is spawn, and there can be one or more instances of the worker test code running.
dunit("module-name", "test-name")
.option()
.driver([] {
// test code here
assert(1 == 1);
})
.worker([] {
// test code here
assert(1 == 1);
});
In addition to the options available for unit tests, distributed unit tests have the following options:
Option | Description |
---|---|
.driver | The driver code is defined using this option. This accepts either a (void)->void lambda or a function of the same signature. |
.worker | The worker code is defined using this option. This accepts either a (void)->void lambda or a function of the same signature. |
.workers | Sets the number of worker instances. (default = 4) |
.faultyNetwork(chance, duration) | Simulates a faulty network by introducing holes during which send operations are ignored. |
5. Performance Tests
Performance tests provide a means to measure the runtime of a piece of code in comparison to some baseline. The test requires that the runtime of the main body is less than the baseline. An optional margin can be set to specify the minimum reduction in runtime allowed (as an absolute value or ratio) to consider this test successful.
perf("module-name", "test-name")
.option()
.body([] {
// test code here
})
.baseline([] {
// test code here
});
In addition to the options available for unit tests, performance tests have the following options:
Option | Description |
---|---|
.baseline | Provides the body of some baseline to be used in comparison to the code block in .body(). This accepts either a (void)->void lambda or a function of the same signature. |
.performanceMargin | Specifies the absolute time difference in seconds required between the main body and baseline to consider this test as an improvement over the baseline. (default = 1ms) |
.performanceMarginMillis | Specifies the absolute time difference in milliseconds required between the main body and baseline to consider this test as an improvement over the baseline. (default = 1ms) |
.performanceMarginMicros | Specifies the absolute time difference in microseconds required between the main body and baseline to consider this test as an improvement over the baseline. (default = 1ms) |
.performanceMarginNanos | Specifies the absolute time difference in nanoseconds required between the main body and baseline to consider this test as an improvement over the baseline. (default = 1ms) |
.performanceMarginAsBaselineRatio | Sets the required ratio of body/baseline runtime. If set, this ignores the absolute performance margin. |
6. Utilities
The framework provides a number of utilities that help facilitate a number of frequently used operations. These utilities are provided as macros and functions that can be used inside the test body.
Function/Macro | Description |
---|---|
dtest_random() | Returns a random number between 0.0 and 1.0 from a uniform distribution. This can be substantially faster than standard C/C++ methods in a multithreaded context. |
dtest_num_workers() | Returns the set number of workers for a distributed unit test. |
dtest_worker_id() | Returns the unique id of the worker. The driver is always guaranteed to have id = 0. |
dtest_is_driver() | Returns true for the driver, false for any worker. |
dtest_notify() | Sends a notification to the driver or workers. |
dtest_wait([n]) | Waits for a notification from the driver or worker(s). An optional parameter n can be set to specify the number of notify messages required. The default value is 1 on workers. On the driver, the default is the number of workers set for the test. |
dtest_send_msg(msg) | Sends a message to the driver/worker. The parameter msg can be any series of variables separated by "<<" (e.g. var1 << var2 << ...) |
dtest_recv_msg(msg) | Receives a message from the driver/worker. The parameter msg can be any series of variables separated by ">>" (e.g. var1 >> var2 >> ...) |