summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
* [lit] Try using process pools by default againReid Kleckner2017-04-071-1/+1
| | | | | | | | | | | Both pickling errors encountered on clang bots and Darwin compiler-rt should now be fixed. This has no impact on testing time on Linux, and on Windows goes from 88s to 63s for 'check'. The tests pass on Mac, but I haven't compared execution time. llvm-svn: 299775
* [lit] Implement timeouts and max_time for process pool testingReid Kleckner2017-04-063-50/+83
| | | | | | | | | | | | | | | | | | | This is necessary to pass the lit test suite at llvm/utils/lit/tests. There are some pre-existing failures here, but now switching to pools doesn't regress any tests. I had to change test-data/lit.cfg to import DummyConfig from a module to fix pickling problems, but I think it'll be OK if we require test formats to be written in real .py modules outside lit.cfg files. I also discovered that in some circumstances AsyncResult.wait() will not raise KeyboardInterrupt in a timely manner, but you can pass a non-zero timeout to work around this. This makes threading.Condition.wait use a polling loop that runs through the interpreter, so it's capable of asynchronously raising KeyboardInterrupt. llvm-svn: 299605
* [lit] Revert to old execution strategy while I debug these pickling errorsReid Kleckner2017-04-051-1/+1
| | | | llvm-svn: 299565
* [lit] Use Python 3 style print to satisfy some botsReid Kleckner2017-04-051-1/+1
| | | | llvm-svn: 299564
* [lit] Use process pools for test execution by defaultReid Kleckner2017-04-052-42/+173
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This drastically reduces lit test execution startup time on Windows. Our previous strategy was to manually create one Process per job and manage the worker pool ourselves. Instead, let's use the worker pool provided by multiprocessing. multiprocessing.Pool(jobs) returns almost immediately, and initializes the appropriate number of workers, so they can all start executing tests immediately. This avoids the ramp-up period that the old implementation suffers from. This appears to speed up small test runs. Here are some timings of the llvm-readobj tests on Windows using the various execution strategies: # multiprocessing.Pool: $ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-process-pool |& grep real: ; done real: 0m1.156s real: 0m1.078s real: 0m1.094s # multiprocessing.Process: $ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-processes |& grep real: ; done real: 0m6.062s real: 0m5.860s real: 0m5.984s # threading.Thread: $ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-threads |& grep real: ; done real: 0m9.438s real: 0m10.765s real: 0m11.079s I kept the old code to launch processes in case this change doesn't work on all platforms that LLVM supports, but at some point I would like to remove both the threading and old multiprocessing execution strategies. Reviewers: modocache, rafael Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31677 llvm-svn: 299560
* [lit] Add a minimum export implementation.Rafael Espindola2017-04-041-9/+21
| | | | llvm-svn: 299475
* Rename variable.Rafael Espindola2017-03-311-1/+1
| | | | | | Requested on post commit code review. llvm-svn: 299232
* Add a %basename substitution.Rafael Espindola2017-03-311-1/+4
| | | | | | This will be used to avoid various call to basename in the asan tests. llvm-svn: 299216
* Use the current working directory in the glob expansionRafael Espindola2017-03-312-8/+13
| | | | | | | | | | | This fixes tests that do things like mkdir <dir> cd <dir> .. <cmd> *.foo llvm-svn: 299209
* Use os.path.realpath when tracking the cwd.Rafael Espindola2017-03-301-1/+1
| | | | | | | | | | | | | | | | | | | | | This is needed by TestCases/Posix/coverage-direct.cc The problem is that the test does: mkdir <dir> cd <dir> cd .. rm -rf <dir> <more commands> the current directory currently looks like "/.../<dir>/../" which doesn't exist when dir is deleted. at some point we should probably switch to using the os current directory (specially if we want to add subshell), but this is a small incremental improvement. llvm-svn: 299113
* lit: support redirect from globsRafael Espindola2017-03-301-7/+13
| | | | | | | | | | | This adds support for commands like FileCheck < foobar* which is used by some asan tests because the file they want to read has a pid in the name. llvm-svn: 299111
* Remove unused argument.Rafael Espindola2017-03-291-2/+2
| | | | llvm-svn: 298994
* lit: remove python2-ismsBrian Gesiak2017-03-221-2/+2
| | | | | | | | | | | | | | | | | Summary: `assert.assertItemEqual` went away in Python 3. Seeing how lists are ordered, comparing a list against each other should work just as well. Patch by @jbergstroem (Johan Bergström). Reviewers: modocache, gparker42 Reviewed By: modocache Differential Revision: https://reviews.llvm.org/D31229 llvm-svn: 298479
* Teach lit to expand glob expressions.Zachary Turner2017-03-033-9/+58
| | | | | | | | | | | This will enable removing hacks throughout the codebase in clang and compiler-rt that feed multiple inputs to a testing utility by globbing, all of which are either disabled on Windows currently or using xargs / find hacks. Differential Revision: https://reviews.llvm.org/D30380 llvm-svn: 296904
* Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILGreg Parker2017-01-2515-84/+566
| | | | | | | | and UNSUPPORTED" This reverts the revert in r292942. llvm-svn: 293007
* Revert "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILAlex Lorenz2017-01-2415-566/+84
| | | | | | | | | | | and UNSUPPORTED" After r292904 llvm-lit fails to emit the test results in the XML format for Apple's internal buildbots. rdar://30164800 llvm-svn: 292942
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-2415-84/+566
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292904
* Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED"Greg Parker2017-01-2418-571/+89
| | | | | | This change needs to be better-coordinated with libc++. llvm-svn: 292900
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-2418-89/+571
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292896
* [lit] Limit parallelism of sanitizer tests on Darwin [llvm part, take 2]Kuba Mracek2017-01-204-5/+23
| | | | | | | | | | Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests. This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests. Differential Revision: https://reviews.llvm.org/D28420 llvm-svn: 292548
* [lit] Support sharding testsuites, for parallel execution.Graydon Hoare2017-01-182-0/+121
| | | | | | | | | | | | | | | | | | | | | | | Summary: This change equips lit.py with two new options, --num-shards=M and --run-shard=N (set by default from env vars LIT_NUM_SHARDS and LIT_RUN_SHARD). The options must be used together, and N must be in 1..M. Together these options effect only test selection: they partition the testsuite into M equal-sized "shards", then select only the Nth shard. They can be used in a cluster of test machines to achieve a very crude (static) form of parallelism, with minimal configuration work. Reviewers: modocache, ddunbar Reviewed By: ddunbar Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D28789 llvm-svn: 292417
* [LIT] Make util.executeCommand python3 friendlyEric Fiselier2017-01-181-0/+4
| | | | | | | | | | | | | | Summary: The parameter `input` to `subprocess.Popen.communicate(...)` must be an object of type `bytes` . This is strictly enforced in python3. This patch (1) allows `to_bytes` to be safely called redundantly. (2) Explicitly convert `input` within `executeCommand`. This allows for usages like `executeCommand(['clang++', '-'], input='int main() {}\n')`. Reviewers: ddunbar, BinaryKhaos, modocache, dim, EricWF Reviewed By: EricWF Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D28736 llvm-svn: 292308
* Revert r292231.Kuba Mracek2017-01-174-28/+4
| | | | llvm-svn: 292237
* [lit] Limit parallelism of sanitizer tests on Darwin [llvm part]Kuba Mracek2017-01-174-4/+28
| | | | | | | | | | Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests. This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests. Differential Revision: https://reviews.llvm.org/D28420 llvm-svn: 292231
* [gtest] Upgrade googletest to version 1.8.0, minimizing local changes.Chandler Carruth2017-01-041-1/+7
| | | | | | | | | This required re-working the streaming support and lit's support for '--gtest_list_tests' but otherwise seems to be a clean upgrade. Differential Revision: https://reviews.llvm.org/D28154 llvm-svn: 291029
* [AVR] Whitelist the avrlit config environment variablesDylan McKay2016-12-151-1/+1
| | | | | | This allows us to use `lit` to run on-target execution tests. llvm-svn: 289769
* Revert "[AVR] Add the very first on-target test"Renato Golin2016-12-141-1/+1
| | | | | | | This reverts commit r289648, as it's an execution test and relies on the emulator/dispatcher being available on all builders. llvm-svn: 289651
* [AVR] Add the very first on-target testDylan McKay2016-12-141-1/+1
| | | | | | This test runs on actual AVR hardware. llvm-svn: 289648
* [lit] Support custom parsers in parseIntegratedTestScriptEric Fiselier2016-12-054-50/+298
| | | | | | | | | | | | | | | | | | | | | | | Summary: Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers. This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively. As an example after this change libc++ can implement the `MODULES-DEFINES` simply using: ``` mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST) parseIntegratedTestScript(test, additional_parsers=[mparser]) macro_list = mparser.getValue() ``` Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs Subscribers: mgrang, llvm-commits, cfe-commits Differential Revision: https://reviews.llvm.org/D27005 llvm-svn: 288694
* Recommit r287403 (reverted in r287804): [lit] When setting SDKROOT on ↵Kuba Mracek2016-12-011-1/+1
| | | | | | | | Darwin, use '--sdk macosx' to find the right SDK path. This shouls now be safe and not break any more bots. It's strictly better to use '--sdk macosx', otherwise xcrun can return weird things for example when you have Command Line Tools or the SDK installed into '/'. llvm-svn: 288385
* Revert "[lit] When setting SDKROOT on Darwin, use '--sdk macosx' to find the ↵Vedant Kumar2016-11-231-1/+1
| | | | | | | | | right SDK path." This reverts commit r287403. It breaks an internal asan bot. According to Kuba, a fix is up for review here: https://reviews.llvm.org/D26929 llvm-svn: 287804
* [lit] When setting SDKROOT on Darwin, use '--sdk macosx' to find the right ↵Kuba Mracek2016-11-181-1/+1
| | | | | | | | SDK path. This will make sure that we find an actual path in case you have Command Line Tools installed. llvm-svn: 287403
* [lit] Print negative exit codes on Windows in hexReid Kleckner2016-11-071-2/+8
| | | | | | | Negative exit codes are usually exceptions. They're easier to recognize in hex. Compare -1073741502 to 0xc0000142. llvm-svn: 286150
* [lit] Remove TODOBrian Gesiak2016-11-032-175/+3
| | | | | | | | | | | | | | | | | | | | | | Summary: Instead of keeping track of TODOs for lit in a file checked into source control, use LLVM's bug tracker. The TODOs have been migrated to the following bugs: * https://llvm.org/bugs/show_bug.cgi?id=30666 * https://llvm.org/bugs/show_bug.cgi?id=30667 * https://llvm.org/bugs/show_bug.cgi?id=30668 * https://llvm.org/bugs/show_bug.cgi?id=30669 * https://llvm.org/bugs/show_bug.cgi?id=30670 * https://llvm.org/bugs/show_bug.cgi?id=30671 Reviewers: ddunbar, beanz, echristo, delcypher Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25496 llvm-svn: 285973
* [lit] Work around Windows MSys command line tokenization bugReid Kleckner2016-10-261-0/+61
| | | | | | | | | | | | | | Summary: This will allow us to revert LLD r284768, which added spaces to get MSys echo to print what we want. Reviewers: ruiu, inglorion, rafael Subscribers: modocache, llvm-commits Differential Revision: https://reviews.llvm.org/D26009 llvm-svn: 285237
* [lit] Add more testing instructions to READMEBrian Gesiak2016-10-221-0/+15
| | | | | | | | | | | | | | | Summary: r283710 introduced two regressions, one to llvm-lit, and the other to lit executables that were installed via setuptools. Add instructions on how to test for these regressions in the future. Reviewers: ddunbar, delcypher, beanz, chapuni, cmatthews, echristo Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25459 llvm-svn: 284919
* [lit] Fix test shtest-timeout.py for modern outputMichal Gorny2016-10-121-10/+9
| | | | | | | | | | Update the CHECK lines in the shtest-timeout.py lit test to account for the current output. The output has been changed in r271610 without adjusting the tests. Differential Revision: https://reviews.llvm.org/D25236 llvm-svn: 284057
* [lit] Fix FormatError on individual test timeoutMichal Gorny2016-10-121-1/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D25195 llvm-svn: 284056
* [lit] Run unit tests as part of lit test suiteBrian Gesiak2016-10-122-108/+112
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: The Python file `utils/lit/lit/ShUtil.py` contains: 1. Logic used by lit itself 2. A set of unit tests for that logic, which can be run by invoking `python utils/lit/lit/ShUtil.py` Move these unit tests to a `tests/unit` subdirectory of lit, and run the tests as part of lit's test suite. This ensures that, should the lit test suite be included in LLVM's own regression test suite, these unit tests will also be run. (Instructions on how to run lit's test suite can be found in `utils/lit/README.txt`.) Reviewers: ddunbar, echristo, delcypher, beanz Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D25411 llvm-svn: 283968
* Fix issue which cases lit installed with setup.py to not resolve mainChris Matthews2016-10-101-0/+2
| | | | llvm-svn: 283818
* [lit] Remove (or allow specific) unused importsBrian Gesiak2016-10-104-10/+12
| | | | | | | | | | | | | | | | | | | | | Summary: Using Python linter flake8 on the utils/lit reveals several linter warnings designated "F401: Unused import". Fix or silence these warnings. Some of these unused imports are legitimate, while some are part of lit's API. For example, users of lit expect to be able to access `lit.formats.ShTest` in their `lit.cfg`, despite the module hierarchy for that symbol actually being `lit.formats.shtest.ShTest`. To silence linter errors for these lines, include a "noqa" directive. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D25407 llvm-svn: 283710
* [lit] Remove unused TestingProgressDisplay attrBrian Gesiak2016-10-101-1/+0
| | | | | | | | | | | | | | Summary: `TestingProgressDisplay` initializes its `current` attribute to `None`, but never reads or writes the value again. Remove it. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25415 llvm-svn: 283709
* [lit] Fix undefined symbol ArgumentErrorBrian Gesiak2016-10-101-2/+2
| | | | | | | | | | | | | | | | | | | Summary: `ArgumentError` is not defined by the Python standard library. Executing this line of code would throw a exception, but not the intended one. It would throw a `NameError` exception, since `ArgumentError` is undefined. Use `ValueError` instead, which is defined by the Python standard library. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25410 llvm-svn: 283708
* [lit] Remove semicolons in Python codeBrian Gesiak2016-10-102-3/+3
| | | | | | | | | | | | | | | Summary: Semicolons aren't necessary as statement terminators in Python, and each of these uses are superfluous as they appear at the end of a line. The convention is to not use semicolons where not needed, so remove them. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25409 llvm-svn: 283707
* [lit] Remove unused variable in googletest formatBrian Gesiak2016-10-101-1/+0
| | | | | | | | | | | | Summary: `prefix` is written to but never read. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25408 llvm-svn: 283706
* [lit] Remove Python 2.6 and below exec workaroundBrian Gesiak2016-10-101-12/+7
| | | | | | | | | | | | | | Summary: The minimum version of Python required to run LLVM's test suite is 2.7. Remove a workaround for older Python versions. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25400 llvm-svn: 283705
* [lit] Remove workaround for Python 2.5Brian Gesiak2016-10-041-8/+1
| | | | | | | | | | | | | | Summary: The minimum version of Python necessary to run the LLVM test suite is 2.7. Code to work around Python 2.5 and lower isn't necessary. Reviewers: ddunbar, echristo, delcypher, beanz Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25209 llvm-svn: 283169
* [lit] Use argparse instead of optparseChris Bieneman2016-10-032-46/+45
| | | | | | | | | | | | | | | | | | | | | | | | Summary: optparse is deprecated in Python 2.7, which is the minimum version of Python required to run the LLVM test suite. Replace its usage in lit with argparse, optparse's 2.7 replacement module. argparse has several benefits over optparse, but this commit does not make use of those benefits yet. Instead, it simply uses the new API, and attempts to keep the number of changes to a minimum. Confirmed that lit's test suite, as well as LLVM's regression test suite, still pass with these changes. Patch By Brian Gesiak! Reviewers: ddunbar, echristo, beanz, delcypher Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25173 llvm-svn: 283152
* [lit] Throw in unimplemented method (NFC)Chris Bieneman2016-10-031-1/+1
| | | | | | | | | | | | | | | | | | Summary: lit's `OneCommandFileTest` class implements an abstract method that raises if called. However, it raises by referencing an undefined symbol. Instead, raise explicitly by throwing a `NotImplementedError`. This is clearer, and appeases Python linters. Patch By Brian Gesiak! Reviewers: ddunbar, echristo, beanz Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25170 llvm-svn: 283090
* [lit] Remove unused imports (NFC)Chris Bieneman2016-10-034-5/+6
| | | | | | | | | | | | Reviewers: ddunbar, echristo, beanz Patch by Brian Gesiak! Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25169 llvm-svn: 283089
OpenPOWER on IntegriCloud