tests/README.md
1 Running and creating tests {#running-and-creating-tests}
2 ==========================
3 
4 There are a number of tests included in RIOT. They are located in the
5 [tests folder](https://github.com/RIOT-OS/RIOT/tree/master/tests). These tests
6 allow basic functionality to be verified as well as provide an example of
7 usage.
8 
9 
10 Implementing automated tests
11 ----------------------------
12 
13 The goal is to be able to run all tests in a sequential way for as many targets
14 as possible.
15 
16 As some board can't be reset without a manual trigger tests should be implemented
17 with some kind of `synchronization`. This can be done in two ways:
18 
19 - use `test_utils_interactive_sync` when uart input/output does not need to be
20  disabled for the test. This is enabled by default.
21 - set up the test in a loop so the test script will be able so sync with some kind
22  of start condition in the test.
23 
24 The module for the first option is `test_utils_interactive_sync` and is set as a
25 default module in `Makefile.tests_common`. It can be disabled by setting in the
26 application makefile `DISABLE_MODULE += test_utils_interactive_sync`. The python
27 test script will adapt to it automatically.
28 
29 When using the `shell` module, `test_utils_interactive_sync` will use the shell
30 itself to synchronize, and will not use `test_utils_interactive_sync();` function
31 to synchronize. Some times you will want to synchronize before the start of the
32 script and use `test_utils_interactive_sync();` function (e.g.:
33 [tests/ps_schedstatistics](tests/ps_schedstatistics/main.c)). For these cases
34 you can disable `test_utils_interactive_sync_shell` module in the application
35 `Makefile`: `DISABLE_MODULE += test_utils_interactive_sync_shell`.
36 
37 
38 Running automated tests
39 -----------------------
40 
41 Some tests can be performed automatically. The test automation scripts are
42 defined in the `<test_application>/tests/` folder. They are written in python
43 and interact through the uart with the test application code running on a
44 board to do the validation. It is recommended to flash the board with the
45 test just before running it because some platforms cannot be reset while
46 testing.
47 
48 From the test application directory run:
49 
50  BOARD=<board_of_your_choice> make flash test
51 
52 
53 An automated way of knowing if a test is available is to execute the
54 'test/available' target from the test application directory.
55 It executes without error if tests run by 'make test' are present.
56 
57  make test/available
58 
59 
60 Automated Tests Guidelines
61 --------------------------
62 
63 
64 When using `pexpect` `$` is useless for matching the end of a line, instead use
65 `\r\n`([pexpect end-of-line](https://pexpect.readthedocs.io/en/stable/overview.html#find-the-end-of-line-cr-lf-conventions)).
66 
67 Beware of `+` and `*` at the end of patterns. These patterns will always get
68 a minimal match (non-greedy).([pexpect end-of-patterns](https://pexpect.readthedocs.io/en/stable/overview.html#beware-of-and-at-the-end-of-patterns))
69 This can be an issue when matching groups and using the matched groups to verify
70 some kind of behavior since `*` could return an empty match and `+` only a subset.
71 
72 This is especially prevalent since `printf()` is buffered so the output might not
73 arrive in a single read to `pexpect`.
74 
75 To avoid this make sure to match a non-ambiguous character at the end of the
76 pattern like `\r\n`, `\s`, `\)`, etc..
77 
78 **don't**:
79 
80 ~~~~
81  child.expect(r'some string: (\d+)')
82 ~~~~
83 
84 **do**:
85 
86 ~~~
87  child.expect(r'some string: (\d+)\r\n')
88 ~~~
89 ~~~
90  child.expect(r'some string: (\d+)\s')
91 ~~~
92 ~~~
93  child.expect(r'some string: (\d+) ,')
94 ~~~
95 
96 Use expect() instead of assert()
97 --------------------------------
98 
99 In order to make a test application functional in all cases, use `expect()`
100 instead of `assert()`. The former works like the latter, but will still be
101 compiled in if `NDEBUG` is defined. This is useful to keep a test application
102 working even when compiling with -DNDEBUG, allowing for the code-under-test to
103 be compiled with that flag. Otherwise, the application would force compiling
104 all tested code with assertions enabled.
105 `expect()` is defined in the header `test_utils/expect.h`.
106 
107 Interaction through the uart
108 ----------------------------
109 
110 Tests implemented with `testrunner` use the `cleanterm` target that
111 provides an interaction without adding extra text output or input handling.
112 It can currently be expected to have unmodified line based interaction with the
113 board.
114 
115 The expected behavior is verified with the test in `tests/test_tools`.
116 
117 Tests cannot rely on having on all boards and terminal programs:
118 * unbuffered input
119 * allowing sending special characters like `ctrl+c/ctrl+d`