summaryrefslogtreecommitdiffstats
path: root/support/testing
Commit message (Collapse)AuthorAgeFilesLines
...
* support/testing: fix run-tests -jRicardo Martincoski2017-07-241-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since commit cf3cd4388a652c9af27ef1c35622e2d0a55b99a9 the -j option is silently ignored. The configuration lines are processed using '\n'.join(). This function adds intervening occurrences of the separator, but the resulting string does not end at a separator. >>> "n".join(["a","b"]) 'anb' It results in a defconfig that does not end in a newline. When BR2_JLEVEL is added by -j logic to the defconfig it ends up concatenated to the last line of the defconfig. BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=yBR2_JLEVEL=7 The resulting .config has the default BR2_JLEVEL=0. Instead of just workaround this problem by adding a newline before BR2_JLEVEL when -j is used, make the defconfig to end in a newline since it is a more future-proof solution. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Yann E. MORIN <yann.morin.1998@free.fr> Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* testing/infra/builder: dump defconfig to logRicardo Martincoski2017-07-241-0/+4
| | | | | | | | | | | The defconfig is composed on-the-fly by test infra + tests. Dump it to the logfile before running 'make olddefconfig' so it can easily analysed when debugging. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* testing/tests/package: add basic unit test for IPythonAndrey Smirnov2017-07-221-0/+38
| | | | | | Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com> [Thomas: update .gitlab-ci.yml.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* testing/tests/package/test_python: allow to change timeoutAndrey Smirnov2017-07-221-8/+8
| | | | | | | | | Depending on Python implementation used for testing, time it takes to perform a given test can vary pretty significantly. To accout for that allow individual test functions to specify different timeout value. Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* testing/tests/package/test_python: do not hardcode interpreter nameAndrey Smirnov2017-07-221-4/+6
| | | | | | | | | | In order to be able to leverage the same test code for testing different python interpreters (or wrappers around CPython) allow child classes of TestPythonBase to override the name of the executable used to run tests. Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* testing/tests/package/test_python: add TestPython3Andrey Smirnov2017-07-221-2/+15
| | | | | | | | | | | | Add Python3 version of TestPython2 to make sure both versions of Python get unit-tested. Modify the code of libc_time_test() to support that change (convert the code to use Python3-style "print"). Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com> [Thomas: update .gitlab-ci.yml.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* testing/tests/package/test_python: refactor TestPythonBaseAndrey Smirnov2017-07-221-4/+19
| | | | | | | | | | | | | | | | Convert TestPythonBase to a true base class that only provides code implementing various tests without defining tests themselves in a "discoverable" form. To retain correct testing functionality, add TestPython2 derived class that uses code from TestPythonBase to define actual runnable test. Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> [Thomas: fix typo in commit log, update .gitlab-ci.yml, both pointed by Ricardo.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* testing/infra/emulator: allow to specify pexpect timeoutAndrey Smirnov2017-07-221-3/+3
| | | | | | | | | | | Some commands take more than 5 seconds to complete under QEMU, so add provisions to allow individual unit-test to specify different duration to avoid false negative test failures. Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/tests: allow properly indented config fragmentYann E. MORIN2017-07-181-0/+1
| | | | | | | | | | | | | | | | | | | | | | Currently, defining a config fragment in the runtime test infra requires that the fragment not to be indented. This is beark, and causes grievance when looking at the code (e.g. to fix it). Just strip out all leading spaces/tabs when writing the configuration lines into the config file, allowing in-line indented config fragments, like so: class TestFoo(bla): config = bla.config + \ """ FOO=y # BAR is not set """ Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Arnout Vandecappelle <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/tests: fix ext4 runtime testYann E. MORIN2017-07-141-1/+1
| | | | | | | | | | | | The current test fails because of a legacy option, renamed during the recent ext overhaul. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Sébastien Szymanski <sebastien.szymanski@armadeus.com> Cc: Samuel Martin <s.martin49@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Arnout Vandecappelle <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: unbreak run-tests -lArnout Vandecappelle2017-07-101-1/+1
| | | | | | | | | | | | | | | | | | | In commit b78b50465c20c1733753a8dd47945cf80c9155f8, the initialisation of BRTest.builddir was moved to the __init__ function. However, it is set based on BRTest.outputdir and that is only set when the -o argument is given to run-tests. When called as "run-tests -l", there is no -o argument so BRTest.outputdir remains unset. To fix, keep BRTest.builddir at None when BRTest.outputdir is None. While we're at it, drop the direct access to the class member. If a subclass wishes to set outputdir to something else before calling BRTest.__init__, they are free to do so. Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Reported-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: move BRTest initialisation to __init__Arnout Vandecappelle2017-07-101-4/+7
| | | | | | | | | | | | BRTest's setUp() method contains a few assignments that initialize its member variables. Since we will want to use these in test case overrides, move them to the __init__ function. Also allow the config member to be overridden, rather than always taking the class member. Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: strip /usr/ part from HOST_DIRArnout Vandecappelle2017-07-101-2/+2
| | | | | Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: remove references to host/usrArnout Vandecappelle2017-07-053-3/+3
| | | | | | | The tools are now installed in host/bin instead of host/usr/bin. Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: add tests for musl and uclibc toolchainsThomas Petazzoni2017-07-051-0/+75
| | | | | | | | | | | | | | These tests simply build a system with musl and uclibc toolchains, and boot them under qemu. It allows to minimally validate that our support for musl/uclibc external toolchains is working. We already had some tests covering glibc toolchains, so we can now easily test that all three C libraries are supported. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> --- This commit is part of the series, as I've written/used those tests to validate that things are still working correctly with all of glibc, uclibc and musl toolchains.
* support/testing: large timeout for login promptRicardo Martincoski2017-07-011-1/+3
| | | | | | | | | | | When running multiple instances of emulator in parallel, the login prompt can take some time to appear. Use a large timeout when waiting for the prompt to avoid random failures. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: run testcases in parallelRicardo Martincoski2017-07-012-1/+18
| | | | | | | | | | | | | | | | | | | Let the user to pass -t to set the number of testcases to run simultaneously. When -j is not specified, calculate it to split the available cores between the simultaneous testcases. Example of auto calculated -j for cpu_count 8: -t -j total 1 9 9 2 4 8 3 3 9 4 2 8 >=5 1 t Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: allow to set BR2_JLEVELRicardo Martincoski2017-07-012-1/+15
| | | | | | | Let the user to override the default BR2_JLEVEL used for each testcase. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: fix code style in emulatorRicardo Martincoski2017-07-011-1/+1
| | | | | | | | Remove unused import. Use 2 empty lines before a class. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: use pexpect.expect directlyRicardo Martincoski2017-07-011-22/+13
| | | | | | | | When using pexpect there is no need for a helper function. Just use expect() directly everywhere. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: use pexpect.sendline directlyRicardo Martincoski2017-07-011-7/+4
| | | | | | | | When using pexpect there is no need for a helper function. Just use sendline() directly everywhere. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: let pexpect write stdout to logRicardo Martincoski2017-07-011-1/+2
| | | | | | | | | | | | | | | When the parameter logfile is passed to spawn(), pexpect sends both stdin and stdout to the logfile and it creates a double echo effect. One way to avoid the double echo in the logfile would be to disable the echo on the terminal just after login ("stty -echo"), but double echo of user and password would remain. Instead of that, send only the stdout to the logfile using the logfile_read property. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: use qemu stdio in emulatorRicardo Martincoski2017-07-011-16/+8
| | | | | | | | | | | | | Instead of redirecting qemu serial to telnet, redirect it to stdio. It allows to run testcases in parallel without random failing caused by two emulators trying to use the same telnet port (1234). 'qemu -serial stdio' returns some extra <CR> characters, so remove them from the log. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: use pexpect in emulatorRicardo Martincoski2017-07-011-15/+13
| | | | | | | | | | Replace subprocess + telnetlib with pexpect. Use the telnet installed on the host machine instead of telnetlib, while the serial from qemu is not yet redirected to stdio. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: rename variables for readabilityLuca Ceresoli2017-05-281-4/+4
| | | | | | | | | | | | | The 'lines' variable is overwritten with its own fields. Thus it contains a line first, and then a list of fields -- it never contains 'lines'. Use two different variables named 'line' and 'fields' to make the code more readable. Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Acked-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: fs/ext: add a negative test for the "extent" featureLuca Ceresoli2017-05-281-0/+1
| | | | | | | | | We only have a positive test for it, in ext4. Let's have a negative one as well. Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: simplify logging by keeping the log file openLuca Ceresoli2017-05-173-33/+15
| | | | | | | | | | | | | | | We currently call infra.smart_open() to open log files each time we need to write to them. Opening the file once in the constructor of Builder and Emulator and writing to it whenever needed is simpler and slightly more efficient. Remove smart_open and instead create a new open_log_file() function which just opens the logfile. Also let it compute the filename, in order to simplify even further the Builder and Emulator code. Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing/run-tests: help: put the one-letter form before the long formLuca Ceresoli2017-05-171-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | This is what the manpages usually do, and what Python does with the automatically-added -h/--help parameter: Before the change: $ ./support/testing/run-tests [...] optional arguments: -h, --help show this help message and exit --list, -l list of available test cases --all, -a execute all test cases After the change: $ ./support/testing/run-tests [...] optional arguments: -h, --help show this help message and exit -l, --list list of available test cases -a, --all execute all test cases Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: document get_elf_prog_interpreterLuca Ceresoli2017-05-171-0/+10
| | | | | Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: rename check_broken_links to has_broken_linksLuca Ceresoli2017-05-171-3/+3
| | | | | | | | | has_broken_links makes it self-explanatory that this is a predicate function, and that the return value tells whether there _are_ broken links, not the opposite. Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: remove unused variableLuca Ceresoli2017-05-101-1/+0
| | | | | Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: show messages in a more readable formatLuca Ceresoli2017-05-101-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently messages from run-tests are quite unpleasant: [br-tests/TestPostScripts/2017-05-09 15:51:57] Building [br-tests/TestPostScripts/2017-05-09 15:52:23] Building done [br-tests/TestPostScripts/2017-05-09 15:52:23] Cleaning up .[br-tests/TestNoTimezone/2017-05-09 15:52:23] Starting [br-tests/TestNoTimezone/2017-05-09 15:52:23] Building [br-tests/TestNoTimezone/2017-05-09 15:53:17] Building done [br-tests/TestNoTimezone/2017-05-09 15:53:22] Cleaning up .[br-tests/TestGlibcNonDefaultLimitedTimezone/2017-05-09 15:53:22] Starting [br-tests/TestGlibcNonDefaultLimitedTimezone/2017-05-09 15:53:22] Building [br-tests/TestGlibcNonDefaultLimitedTimezone/2017-05-09 15:54:33] Building done [br-tests/TestGlibcNonDefaultLimitedTimezone/2017-05-09 15:54:37] Cleaning up [...] Change them in a more readable way by removing the date and using a columnar style: 15:12:22 TestPostScripts Starting 15:12:25 TestPostScripts Building 15:12:48 TestPostScripts Building done 15:12:48 TestPostScripts Cleaning up .15:12:48 TestNoTimezone Starting 15:12:54 TestNoTimezone Building 15:13:44 TestNoTimezone Building done 15:13:49 TestNoTimezone Cleaning up .15:13:49 TestGlibcNonDefaultLimitedTimezone Starting 15:14:00 TestGlibcNonDefaultLimitedTimezone Building 15:14:56 TestGlibcNonDefaultLimitedTimezone Building done 15:15:01 TestGlibcNonDefaultLimitedTimezone Cleaning up [...] Note the '.' and other characters presented by nose2 are still printed. They are not affected by this change. Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: add toolchain testsThomas Petazzoni2017-05-072-0/+156
| | | | | | | | This commit adds an initial toolchain test case, testing the ARM CodeSourcery toolchain, just checking that the proper sysroot is used, and that a minimal Linux system boots fine under Qemu. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: add package testsThomas Petazzoni2017-05-073-0/+63
| | | | | | | | This commit adds some basic tests for two Buildroot packages: python and dropbear. These tests are by no mean meant to be exhaustive, but mainly to serve as initial examples for other tests. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: add fs testsThomas Petazzoni2017-05-0711-0/+469
| | | | | | | | | | This commit adds a number of test cases for various filesystem formats: ext2/3/4, iso9660, jffs2, squashfs, ubi/ubifs and yaffs2. All of them except yaffs2 are runtime tested. The iso9660 set of test cases is particularly rich, testing the proper operation of the iso9660 support with all of grub, grub2 and isolinux. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: add core testsThomas Petazzoni2017-05-078-0/+159
| | | | | | | | | | | This commit adds a few Buildroot "core" tests, testing functionalities such as: - post-build and post-image scripts - root filesystem overlays - timezone support Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: core testing infrastructureThomas Petazzoni2017-05-077-0/+428
This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
OpenPOWER on IntegriCloud