summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
* Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILGreg Parker2017-01-2515-84/+566
| | | | | | | | and UNSUPPORTED" This reverts the revert in r292942. llvm-svn: 293007
* Revert "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILAlex Lorenz2017-01-2415-566/+84
| | | | | | | | | | | and UNSUPPORTED" After r292904 llvm-lit fails to emit the test results in the XML format for Apple's internal buildbots. rdar://30164800 llvm-svn: 292942
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-2415-84/+566
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292904
* Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED"Greg Parker2017-01-2418-571/+89
| | | | | | This change needs to be better-coordinated with libc++. llvm-svn: 292900
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-2418-89/+571
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292896
* [lit] Limit parallelism of sanitizer tests on Darwin [llvm part, take 2]Kuba Mracek2017-01-204-5/+23
| | | | | | | | | | Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests. This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests. Differential Revision: https://reviews.llvm.org/D28420 llvm-svn: 292548
* [lit] Support sharding testsuites, for parallel execution.Graydon Hoare2017-01-182-0/+121
| | | | | | | | | | | | | | | | | | | | | | | Summary: This change equips lit.py with two new options, --num-shards=M and --run-shard=N (set by default from env vars LIT_NUM_SHARDS and LIT_RUN_SHARD). The options must be used together, and N must be in 1..M. Together these options effect only test selection: they partition the testsuite into M equal-sized "shards", then select only the Nth shard. They can be used in a cluster of test machines to achieve a very crude (static) form of parallelism, with minimal configuration work. Reviewers: modocache, ddunbar Reviewed By: ddunbar Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D28789 llvm-svn: 292417
* [LIT] Make util.executeCommand python3 friendlyEric Fiselier2017-01-181-0/+4
| | | | | | | | | | | | | | Summary: The parameter `input` to `subprocess.Popen.communicate(...)` must be an object of type `bytes` . This is strictly enforced in python3. This patch (1) allows `to_bytes` to be safely called redundantly. (2) Explicitly convert `input` within `executeCommand`. This allows for usages like `executeCommand(['clang++', '-'], input='int main() {}\n')`. Reviewers: ddunbar, BinaryKhaos, modocache, dim, EricWF Reviewed By: EricWF Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D28736 llvm-svn: 292308
* Revert r292231.Kuba Mracek2017-01-174-28/+4
| | | | llvm-svn: 292237
* [lit] Limit parallelism of sanitizer tests on Darwin [llvm part]Kuba Mracek2017-01-174-4/+28
| | | | | | | | | | Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests. This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests. Differential Revision: https://reviews.llvm.org/D28420 llvm-svn: 292231
* [gtest] Upgrade googletest to version 1.8.0, minimizing local changes.Chandler Carruth2017-01-041-1/+7
| | | | | | | | | This required re-working the streaming support and lit's support for '--gtest_list_tests' but otherwise seems to be a clean upgrade. Differential Revision: https://reviews.llvm.org/D28154 llvm-svn: 291029
* [AVR] Whitelist the avrlit config environment variablesDylan McKay2016-12-151-1/+1
| | | | | | This allows us to use `lit` to run on-target execution tests. llvm-svn: 289769
* Revert "[AVR] Add the very first on-target test"Renato Golin2016-12-141-1/+1
| | | | | | | This reverts commit r289648, as it's an execution test and relies on the emulator/dispatcher being available on all builders. llvm-svn: 289651
* [AVR] Add the very first on-target testDylan McKay2016-12-141-1/+1
| | | | | | This test runs on actual AVR hardware. llvm-svn: 289648
* [lit] Support custom parsers in parseIntegratedTestScriptEric Fiselier2016-12-054-50/+298
| | | | | | | | | | | | | | | | | | | | | | | Summary: Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers. This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively. As an example after this change libc++ can implement the `MODULES-DEFINES` simply using: ``` mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST) parseIntegratedTestScript(test, additional_parsers=[mparser]) macro_list = mparser.getValue() ``` Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs Subscribers: mgrang, llvm-commits, cfe-commits Differential Revision: https://reviews.llvm.org/D27005 llvm-svn: 288694
* Recommit r287403 (reverted in r287804): [lit] When setting SDKROOT on ↵Kuba Mracek2016-12-011-1/+1
| | | | | | | | Darwin, use '--sdk macosx' to find the right SDK path. This shouls now be safe and not break any more bots. It's strictly better to use '--sdk macosx', otherwise xcrun can return weird things for example when you have Command Line Tools or the SDK installed into '/'. llvm-svn: 288385
* Revert "[lit] When setting SDKROOT on Darwin, use '--sdk macosx' to find the ↵Vedant Kumar2016-11-231-1/+1
| | | | | | | | | right SDK path." This reverts commit r287403. It breaks an internal asan bot. According to Kuba, a fix is up for review here: https://reviews.llvm.org/D26929 llvm-svn: 287804
* [lit] When setting SDKROOT on Darwin, use '--sdk macosx' to find the right ↵Kuba Mracek2016-11-181-1/+1
| | | | | | | | SDK path. This will make sure that we find an actual path in case you have Command Line Tools installed. llvm-svn: 287403
* [lit] Print negative exit codes on Windows in hexReid Kleckner2016-11-071-2/+8
| | | | | | | Negative exit codes are usually exceptions. They're easier to recognize in hex. Compare -1073741502 to 0xc0000142. llvm-svn: 286150
* [lit] Remove TODOBrian Gesiak2016-11-032-175/+3
| | | | | | | | | | | | | | | | | | | | | | Summary: Instead of keeping track of TODOs for lit in a file checked into source control, use LLVM's bug tracker. The TODOs have been migrated to the following bugs: * https://llvm.org/bugs/show_bug.cgi?id=30666 * https://llvm.org/bugs/show_bug.cgi?id=30667 * https://llvm.org/bugs/show_bug.cgi?id=30668 * https://llvm.org/bugs/show_bug.cgi?id=30669 * https://llvm.org/bugs/show_bug.cgi?id=30670 * https://llvm.org/bugs/show_bug.cgi?id=30671 Reviewers: ddunbar, beanz, echristo, delcypher Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25496 llvm-svn: 285973
* [lit] Work around Windows MSys command line tokenization bugReid Kleckner2016-10-261-0/+61
| | | | | | | | | | | | | | Summary: This will allow us to revert LLD r284768, which added spaces to get MSys echo to print what we want. Reviewers: ruiu, inglorion, rafael Subscribers: modocache, llvm-commits Differential Revision: https://reviews.llvm.org/D26009 llvm-svn: 285237
* [lit] Add more testing instructions to READMEBrian Gesiak2016-10-221-0/+15
| | | | | | | | | | | | | | | Summary: r283710 introduced two regressions, one to llvm-lit, and the other to lit executables that were installed via setuptools. Add instructions on how to test for these regressions in the future. Reviewers: ddunbar, delcypher, beanz, chapuni, cmatthews, echristo Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25459 llvm-svn: 284919
* [lit] Fix test shtest-timeout.py for modern outputMichal Gorny2016-10-121-10/+9
| | | | | | | | | | Update the CHECK lines in the shtest-timeout.py lit test to account for the current output. The output has been changed in r271610 without adjusting the tests. Differential Revision: https://reviews.llvm.org/D25236 llvm-svn: 284057
* [lit] Fix FormatError on individual test timeoutMichal Gorny2016-10-121-1/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D25195 llvm-svn: 284056
* [lit] Run unit tests as part of lit test suiteBrian Gesiak2016-10-122-108/+112
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: The Python file `utils/lit/lit/ShUtil.py` contains: 1. Logic used by lit itself 2. A set of unit tests for that logic, which can be run by invoking `python utils/lit/lit/ShUtil.py` Move these unit tests to a `tests/unit` subdirectory of lit, and run the tests as part of lit's test suite. This ensures that, should the lit test suite be included in LLVM's own regression test suite, these unit tests will also be run. (Instructions on how to run lit's test suite can be found in `utils/lit/README.txt`.) Reviewers: ddunbar, echristo, delcypher, beanz Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D25411 llvm-svn: 283968
* Fix issue which cases lit installed with setup.py to not resolve mainChris Matthews2016-10-101-0/+2
| | | | llvm-svn: 283818
* [lit] Remove (or allow specific) unused importsBrian Gesiak2016-10-104-10/+12
| | | | | | | | | | | | | | | | | | | | | Summary: Using Python linter flake8 on the utils/lit reveals several linter warnings designated "F401: Unused import". Fix or silence these warnings. Some of these unused imports are legitimate, while some are part of lit's API. For example, users of lit expect to be able to access `lit.formats.ShTest` in their `lit.cfg`, despite the module hierarchy for that symbol actually being `lit.formats.shtest.ShTest`. To silence linter errors for these lines, include a "noqa" directive. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D25407 llvm-svn: 283710
* [lit] Remove unused TestingProgressDisplay attrBrian Gesiak2016-10-101-1/+0
| | | | | | | | | | | | | | Summary: `TestingProgressDisplay` initializes its `current` attribute to `None`, but never reads or writes the value again. Remove it. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25415 llvm-svn: 283709
* [lit] Fix undefined symbol ArgumentErrorBrian Gesiak2016-10-101-2/+2
| | | | | | | | | | | | | | | | | | | Summary: `ArgumentError` is not defined by the Python standard library. Executing this line of code would throw a exception, but not the intended one. It would throw a `NameError` exception, since `ArgumentError` is undefined. Use `ValueError` instead, which is defined by the Python standard library. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25410 llvm-svn: 283708
* [lit] Remove semicolons in Python codeBrian Gesiak2016-10-102-3/+3
| | | | | | | | | | | | | | | Summary: Semicolons aren't necessary as statement terminators in Python, and each of these uses are superfluous as they appear at the end of a line. The convention is to not use semicolons where not needed, so remove them. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25409 llvm-svn: 283707
* [lit] Remove unused variable in googletest formatBrian Gesiak2016-10-101-1/+0
| | | | | | | | | | | | Summary: `prefix` is written to but never read. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25408 llvm-svn: 283706
* [lit] Remove Python 2.6 and below exec workaroundBrian Gesiak2016-10-101-12/+7
| | | | | | | | | | | | | | Summary: The minimum version of Python required to run LLVM's test suite is 2.7. Remove a workaround for older Python versions. Reviewers: echristo, delcypher, beanz, ddunbar Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25400 llvm-svn: 283705
* [lit] Remove workaround for Python 2.5Brian Gesiak2016-10-041-8/+1
| | | | | | | | | | | | | | Summary: The minimum version of Python necessary to run the LLVM test suite is 2.7. Code to work around Python 2.5 and lower isn't necessary. Reviewers: ddunbar, echristo, delcypher, beanz Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25209 llvm-svn: 283169
* [lit] Use argparse instead of optparseChris Bieneman2016-10-032-46/+45
| | | | | | | | | | | | | | | | | | | | | | | | Summary: optparse is deprecated in Python 2.7, which is the minimum version of Python required to run the LLVM test suite. Replace its usage in lit with argparse, optparse's 2.7 replacement module. argparse has several benefits over optparse, but this commit does not make use of those benefits yet. Instead, it simply uses the new API, and attempts to keep the number of changes to a minimum. Confirmed that lit's test suite, as well as LLVM's regression test suite, still pass with these changes. Patch By Brian Gesiak! Reviewers: ddunbar, echristo, beanz, delcypher Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25173 llvm-svn: 283152
* [lit] Throw in unimplemented method (NFC)Chris Bieneman2016-10-031-1/+1
| | | | | | | | | | | | | | | | | | Summary: lit's `OneCommandFileTest` class implements an abstract method that raises if called. However, it raises by referencing an undefined symbol. Instead, raise explicitly by throwing a `NotImplementedError`. This is clearer, and appeases Python linters. Patch By Brian Gesiak! Reviewers: ddunbar, echristo, beanz Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25170 llvm-svn: 283090
* [lit] Remove unused imports (NFC)Chris Bieneman2016-10-034-5/+6
| | | | | | | | | | | | Reviewers: ddunbar, echristo, beanz Patch by Brian Gesiak! Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25169 llvm-svn: 283089
* [lit] Compare to None using identity, not equalityChris Bieneman2016-10-032-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | Summary: In Python, `None` is a singleton, so checking whether a variable is `None` may be done with `is` or `is not`. This has a slight advantage over equiality comparisons `== None` and `!= None`, since `__eq__` may be overridden in Python to produce sometimes unexpected results. Using `is None` and `is not None` is also recommended practice in https://www.python.org/dev/peps/pep-0008: > Comparisons to singletons like `None` should always be done with `is` or > `is not`, never the equality operators. Patch by Brian Gesiak! Reviewers: ddunbar, echristo, beanz Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25168 llvm-svn: 283088
* [lit] Add instructions to run lit's test suiteDaniel Dunbar2016-09-271-0/+16
| | | | | | | | - Patch by Brian Gesiak. - https://reviews.llvm.org/D24968 llvm-svn: 282525
* [lit] Fix refacto introduced by rL282479.Daniel Dunbar2016-09-271-1/+3
| | | | llvm-svn: 282501
* Trying to fix lldb build breakage probably caused by rL282452Dimitar Vlahovski2016-09-271-11/+11
| | | | llvm-svn: 282479
* [lit] Add a --max-failures option.Daniel Dunbar2016-09-264-3/+41
| | | | | | | | | - This is primarily useful as a "fail fast" mode for lit, where it will stop running tests after the first failure. - Patch by Max Moiseev. llvm-svn: 282452
* [lit] Downgrade error to warning on gtest crashes during discovery.Ahmed Bougacha2016-09-051-2/+2
| | | | | | | | | | Lots of unittests started failing under asan after r280455. It seems they've been failing for a long time, but lit silently ignored them. Downgrade the error so we can figure out what is going on. Filed http://llvm.org/PR30285. llvm-svn: 280674
* lit/util.py: Another fix for py3.NAKAMURA Takumi2016-09-051-0/+2
| | | | | | 'str' object has no attribute 'decode'. llvm-svn: 280641
* Make lit/util.py py3-compatible.NAKAMURA Takumi2016-09-031-1/+1
| | | | llvm-svn: 280579
* lit: print process output, if getting the list of google-tests failed.Ivan Krasin2016-09-022-6/+11
| | | | | | | | | | | | | Summary: This is a follow up to r280455, where a check for the process exit code was introduced. Some ASAN bots throw this error now, but it's impossible to understand what's wrong with them, and the issue is not reproducible. Reviewers: vitalybuka Differential Revision: https://reviews.llvm.org/D24210 llvm-svn: 280550
* Quick fix to make LIT_PRESERVES_TMP work againReid Kleckner2016-09-021-1/+2
| | | | llvm-svn: 280502
* [lit] Clean up temporary files created by testsReid Kleckner2016-09-022-11/+28
| | | | | | | | | | | | Do this by creating a temp directory in the normal system temp directory, and cleaning it up on exit. It is still possible for this temp directory to leak if Python exits abnormally, but this is probably good enough for now. Fixes PR18335 llvm-svn: 280501
* [lit] Fail testing if a googletest executable crashes during test discoveryGreg Parker2016-09-022-2/+6
| | | | | | | | | googletest formatted tests are discovered by running the test executable. Previously testing would silently succeed if the test executable crashed during the discovery process. Now testing fails with "error: unable to discover google-tests ..." if the test executable exits with a non-zero status. llvm-svn: 280455
* [lit] Use multiprocessing by default on WindowsReid Kleckner2016-09-011-6/+3
| | | | | | | | | | | | | | | | Apparently nobody evaluated multiprocessing on Windows since Daniel enabled multiprocessing on Unix in r193279. It works so far as I can tell. Today this is worth about an 8x speedup (631.29s to 73.25s) on my 24 core Windows machine. Hopefully this will improve Windows buildbot cycle time, where currently it takes more time to run check-all than it does to self-host with assertions enabled: http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/20 build stage 2 ninja all ( 28 mins, 22 secs ) ninja check 2 stage 2 ( 37 mins, 38 secs ) llvm-svn: 280382
* [lit] Use full config path in diagnostics.Daniel Dunbar2016-07-211-1/+1
| | | | | | | | | - This allows tools like emacs to automatically find the config file path when you step through errors. - Patch by Dave Abrahams. llvm-svn: 276357
OpenPOWER on IntegriCloud