summaryrefslogtreecommitdiffstats
path: root/support/testing/infra/basetest.py
Commit message (Collapse)AuthorAgeFilesLines
* testing/infra: Add BRConfigTest as superclass of BRTestRicardo Martincoski2019-02-061-7/+19
| | | | | | | | | | | | | | | | | The git tests don't need to do a full build, they only need to do a configure and download and/or legal-info. More tests of that type will be added in the future. Therefore, we want to have a test base class that doesn't automatically do a full build in the setUp(). Add this new class as a superclass of the existing BRTest class, so we don't need to update existing tests. Only the code in run-tests that iterates over all subclasses of BRTest has to be adapted to use BRConfigTest instead. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br> Cc: Arnout Vandecappelle <arnout@mind.be> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/testing/infra/basetest: support br2-externalRicardo Martincoski2019-01-291-1/+2
| | | | | | | | | | | | | | | | | | Some upcoming test cases can use one or more br2-external trees as fixtures that provide packages used only in runtime tests. Add support for br2-external into the BRTest class. Any test case can then provide a list of paths for being used as br2-external trees during the build of the image to test. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br> Cc: Arnout Vandecappelle <arnout@mind.be> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com> Signed-off-by: Daniel J. Leach <dleach@belcan.com> Signed-off-by: Adam Duskett <Aduskett@gmail.com> Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com> [Thomas: use named argument for make_extra_opts.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/testing: fix python syntaxYann E. MORIN2018-06-101-2/+2
| | | | | | | | | | | | Fix three issues with code style in our test infra: - 'print' is now a function, - exceptions need to be caught-assigned with the 'as' keyword, - old-style "%s"%() formatting is deprecated. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com> [Thomas: drop indices in format strings.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* testing/infra/builder: split configure() from build()Ricardo Martincoski2018-04-011-0/+1
| | | | | | | | | | | | | | | | | Some test cases don't use a full build as setup, so split the build() method into configure() and build(). It allows a test case to perform configuration at the setup stage and the build inside the test itself. Call this new method just before build in the BRTest base class, to keep the current behavior for existing test cases. This change will be needed when adding a common class to test the git download infra. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* testing/infra/builder: call make with empty envRicardo Martincoski2018-04-011-0/+1
| | | | | | | | | | | | | | | | | | | | | Use an empty environment when calling make, but import PATH so the buildroot tree under test can find binaries from the host machine. Since environment variables are now ignored, move the handling of BR2_DL_DIR to the defconfig to keep the current precedence of -d: BR2_DL_DIR | -d DIR | test downloads | BR downloads ------------+----------+------------------+-------------- unset | unset | [error] | [error] unset | set | in $(DIR) | in $(DIR) set | unset | in $(BR2_DL_DIR) | in $(BR2_DL_DIR) set | set | in $(DIR) | in $(DIR) Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/testing: use more recent toolchainsThomas Petazzoni2017-10-211-3/+2
| | | | | | | With the hard disk crash of autobuild.b.o, we lost old toolchains, so use the latest toolchains, which have been restored. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: fix code styleRicardo Martincoski2017-10-061-0/+2
| | | | | | | | | | | Fix the trivial warnings from flake8: - remove modules imported but unused; - use 2 lines before class or module level method; - remove blank line at end of file. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/testing: standardize defconfig fragments styleRicardo Martincoski2017-10-061-19/+19
| | | | | | | | | | | | | | Change all defconfig fragments to take advantage of "cf3cd4388a support/tests: allow properly indented config fragment". Make each defconfig fragment: - start after a backslash; - be declared as a multi-line string literal; - be indented one level more than the variable that contains it. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/testing: allow to indent ccache defconfig fragmentRicardo Martincoski2017-10-061-3/+1
| | | | | | | | | | | | | | | | | | Move the strip out of leading spaces in defconfig fragments from the BRTest class to the Builder class. It actually postpones the strip out, consequentially allowing test cases to post-process the defconfig in their own __init__ before calling the __init__ method from BRTest. Moving this code to the Builder class also allows any new test class that inherits from BRTest to reuse the same code even if the new class overrides the setUp method. At same time, prepend a newline to the jlevel handling otherwise it would stop working for defconfig fragments that don't end in a newline. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/testing: allow to use a multiplier for timeoutsRicardo Martincoski2017-08-101-1/+3
| | | | | | | | | | | | | Add a parameter to run-tests to act as a multiplier for all timeouts of emulator. It can be used to avoid sporadic failures on slow host machines as well in elastic runners on the cloud. Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> [Arnout: rename multiplier to timeout_multiplier everywhere] Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* testing/infra/basetest: move jlevel logic to constructorRicardo Martincoski2017-07-241-5/+3
| | | | | | | | | | | | | | | | | As suggested by Arnout in [1]. While at it, simplify the logic by always appending the BR2_JLEVEL and defaulting to 0 (the value copied from Config.in is used for 5 years now and is very unlikely to change). [1] http://patchwork.ozlabs.org/patch/790525/ Suggested-by: Arnout Vandecappelle <arnout@mind.be> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: fix run-tests -jRicardo Martincoski2017-07-241-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since commit cf3cd4388a652c9af27ef1c35622e2d0a55b99a9 the -j option is silently ignored. The configuration lines are processed using '\n'.join(). This function adds intervening occurrences of the separator, but the resulting string does not end at a separator. >>> "n".join(["a","b"]) 'anb' It results in a defconfig that does not end in a newline. When BR2_JLEVEL is added by -j logic to the defconfig it ends up concatenated to the last line of the defconfig. BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=yBR2_JLEVEL=7 The resulting .config has the default BR2_JLEVEL=0. Instead of just workaround this problem by adding a newline before BR2_JLEVEL when -j is used, make the defconfig to end in a newline since it is a more future-proof solution. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Yann E. MORIN <yann.morin.1998@free.fr> Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/tests: allow properly indented config fragmentYann E. MORIN2017-07-181-0/+1
| | | | | | | | | | | | | | | | | | | | | | Currently, defining a config fragment in the runtime test infra requires that the fragment not to be indented. This is beark, and causes grievance when looking at the code (e.g. to fix it). Just strip out all leading spaces/tabs when writing the configuration lines into the config file, allowing in-line indented config fragments, like so: class TestFoo(bla): config = bla.config + \ """ FOO=y # BAR is not set """ Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Arnout Vandecappelle <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: unbreak run-tests -lArnout Vandecappelle2017-07-101-1/+1
| | | | | | | | | | | | | | | | | | | In commit b78b50465c20c1733753a8dd47945cf80c9155f8, the initialisation of BRTest.builddir was moved to the __init__ function. However, it is set based on BRTest.outputdir and that is only set when the -o argument is given to run-tests. When called as "run-tests -l", there is no -o argument so BRTest.outputdir remains unset. To fix, keep BRTest.builddir at None when BRTest.outputdir is None. While we're at it, drop the direct access to the class member. If a subclass wishes to set outputdir to something else before calling BRTest.__init__, they are free to do so. Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Reported-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: move BRTest initialisation to __init__Arnout Vandecappelle2017-07-101-4/+7
| | | | | | | | | | | | BRTest's setUp() method contains a few assignments that initialize its member variables. Since we will want to use these in test case overrides, move them to the __init__ function. Also allow the config member to be overridden, rather than always taking the class member. Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: allow to set BR2_JLEVELRicardo Martincoski2017-07-011-1/+5
| | | | | | | Let the user to override the default BR2_JLEVEL used for each testcase. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: remove unused variableLuca Ceresoli2017-05-101-1/+0
| | | | | Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: show messages in a more readable formatLuca Ceresoli2017-05-101-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently messages from run-tests are quite unpleasant: [br-tests/TestPostScripts/2017-05-09 15:51:57] Building [br-tests/TestPostScripts/2017-05-09 15:52:23] Building done [br-tests/TestPostScripts/2017-05-09 15:52:23] Cleaning up .[br-tests/TestNoTimezone/2017-05-09 15:52:23] Starting [br-tests/TestNoTimezone/2017-05-09 15:52:23] Building [br-tests/TestNoTimezone/2017-05-09 15:53:17] Building done [br-tests/TestNoTimezone/2017-05-09 15:53:22] Cleaning up .[br-tests/TestGlibcNonDefaultLimitedTimezone/2017-05-09 15:53:22] Starting [br-tests/TestGlibcNonDefaultLimitedTimezone/2017-05-09 15:53:22] Building [br-tests/TestGlibcNonDefaultLimitedTimezone/2017-05-09 15:54:33] Building done [br-tests/TestGlibcNonDefaultLimitedTimezone/2017-05-09 15:54:37] Cleaning up [...] Change them in a more readable way by removing the date and using a columnar style: 15:12:22 TestPostScripts Starting 15:12:25 TestPostScripts Building 15:12:48 TestPostScripts Building done 15:12:48 TestPostScripts Cleaning up .15:12:48 TestNoTimezone Starting 15:12:54 TestNoTimezone Building 15:13:44 TestNoTimezone Building done 15:13:49 TestNoTimezone Cleaning up .15:13:49 TestGlibcNonDefaultLimitedTimezone Starting 15:14:00 TestGlibcNonDefaultLimitedTimezone Building 15:14:56 TestGlibcNonDefaultLimitedTimezone Building done 15:15:01 TestGlibcNonDefaultLimitedTimezone Cleaning up [...] Note the '.' and other characters presented by nose2 are still printed. They are not affected by this change. Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/testing: core testing infrastructureThomas Petazzoni2017-05-071-0/+66
This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
OpenPOWER on IntegriCloud