summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit/tests
Commit message (Collapse)AuthorAgeFilesLines
* Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILGreg Parker2017-01-2512-7/+109
| | | | | | | | and UNSUPPORTED" This reverts the revert in r292942. llvm-svn: 293007
* Revert "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILAlex Lorenz2017-01-2412-109/+7
| | | | | | | | | | | and UNSUPPORTED" After r292904 llvm-lit fails to emit the test results in the XML format for Apple's internal buildbots. rdar://30164800 llvm-svn: 292942
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-2412-7/+109
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292904
* Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED"Greg Parker2017-01-2415-122/+13
| | | | | | This change needs to be better-coordinated with libc++. llvm-svn: 292900
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-2415-13/+122
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292896
* [lit] Support sharding testsuites, for parallel execution.Graydon Hoare2017-01-181-0/+90
| | | | | | | | | | | | | | | | | | | | | | | Summary: This change equips lit.py with two new options, --num-shards=M and --run-shard=N (set by default from env vars LIT_NUM_SHARDS and LIT_RUN_SHARD). The options must be used together, and N must be in 1..M. Together these options effect only test selection: they partition the testsuite into M equal-sized "shards", then select only the Nth shard. They can be used in a cluster of test machines to achieve a very crude (static) form of parallelism, with minimal configuration work. Reviewers: modocache, ddunbar Reviewed By: ddunbar Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D28789 llvm-svn: 292417
* [lit] Support custom parsers in parseIntegratedTestScriptEric Fiselier2016-12-053-0/+141
| | | | | | | | | | | | | | | | | | | | | | | Summary: Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers. This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively. As an example after this change libc++ can implement the `MODULES-DEFINES` simply using: ``` mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST) parseIntegratedTestScript(test, additional_parsers=[mparser]) macro_list = mparser.getValue() ``` Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs Subscribers: mgrang, llvm-commits, cfe-commits Differential Revision: https://reviews.llvm.org/D27005 llvm-svn: 288694
* [lit] Fix test shtest-timeout.py for modern outputMichal Gorny2016-10-121-10/+9
| | | | | | | | | | Update the CHECK lines in the shtest-timeout.py lit test to account for the current output. The output has been changed in r271610 without adjusting the tests. Differential Revision: https://reviews.llvm.org/D25236 llvm-svn: 284057
* [lit] Run unit tests as part of lit test suiteBrian Gesiak2016-10-121-0/+112
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: The Python file `utils/lit/lit/ShUtil.py` contains: 1. Logic used by lit itself 2. A set of unit tests for that logic, which can be run by invoking `python utils/lit/lit/ShUtil.py` Move these unit tests to a `tests/unit` subdirectory of lit, and run the tests as part of lit's test suite. This ensures that, should the lit test suite be included in LLVM's own regression test suite, these unit tests will also be run. (Instructions on how to run lit's test suite can be found in `utils/lit/README.txt`.) Reviewers: ddunbar, echristo, delcypher, beanz Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D25411 llvm-svn: 283968
* [lit] Use argparse instead of optparseChris Bieneman2016-10-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | Summary: optparse is deprecated in Python 2.7, which is the minimum version of Python required to run the LLVM test suite. Replace its usage in lit with argparse, optparse's 2.7 replacement module. argparse has several benefits over optparse, but this commit does not make use of those benefits yet. Instead, it simply uses the new API, and attempts to keep the number of changes to a minimum. Confirmed that lit's test suite, as well as LLVM's regression test suite, still pass with these changes. Patch By Brian Gesiak! Reviewers: ddunbar, echristo, beanz, delcypher Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25173 llvm-svn: 283152
* [lit] Remove unused imports (NFC)Chris Bieneman2016-10-032-3/+0
| | | | | | | | | | | | Reviewers: ddunbar, echristo, beanz Patch by Brian Gesiak! Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25169 llvm-svn: 283089
* [lit] Add a --max-failures option.Daniel Dunbar2016-09-261-0/+14
| | | | | | | | | - This is primarily useful as a "fail fast" mode for lit, where it will stop running tests after the first failure. - Patch by Max Moiseev. llvm-svn: 282452
* [lit] Improve logging with file redirection.Daniel Dunbar2016-06-073-5/+7
| | | | | | | | | - This will cause lit to automatically include the first 1K of data in redirected output files when a command fails (previously if the command failed, but the main point of the test was, say, a `FileCheck` later on, then the log wasn't helpful in showing why the command failed). llvm-svn: 272021
* [lit] Improve readability of failing scripts.Daniel Dunbar2016-06-025-6/+38
| | | | | | | | | | | | | | | | | | | | | | | - This only applies to scripts executed by the _internal_ shell script interpreter. - This patch reworks the log to look more like a shell transcript, and be less verbose (but in the interest of calling attention to the important parts). Here is an example of the new format, for commands with/without failures and with/without output: ``` $ true $ echo hi hi $ false note: command had no output on stdout or stderr error: command failed with exit status 1 ``` llvm-svn: 271610
* Add "REQUIRES-ANY" feature testEric Fiselier2016-06-023-2/+8
| | | | | | | | | | | | | | | | Summary: This patch adds a "REQUIRES-ANY" feature test that is disjunctive. This marks a test as `UNSUPPORTED` if none of the specified features are available. Libc++ has the need to write feature test such as `// REQUIRES-ANY: c++98, c++03` when testing of behavior that is specific to older dialects but has since changed. Reviewers: rnk, ddunbar Subscribers: ddunbar, probinson, llvm-commits, cfe-commits Differential Revision: http://reviews.llvm.org/D20757 llvm-svn: 271468
* [lit][googletest] Handle upstream gtest outputDaniel Dunbar2016-03-313-0/+61
| | | | | | | | | | | | | | | | | | | Summary: Upstream googletest prints "Running main() from gtest_main.cc" to stdout prior to running tests. LLVM removed that print statement in r61540. If a user were to use lit to run tests that use upstream googletest, however, lit reports "Running main()" as an invalid test name. To avoid such a failure, add an extra conditional to `formats/googletest.py`. Also add tests to demonstrate the modified behavior. Reviewers: abdulras, ddunbar Subscribers: ddunbar, llvm-commits, kastiglione Differential Revision: http://reviews.llvm.org/D18606 llvm-svn: 265034
* Revert "Teach the CMake build system to run lit's test suite. These can be run"Dan Liew2016-01-092-24/+2
| | | | | | | | | | | | | | This reverts r257221. This caused several build bot failures * It looks like some of the tests don't work correctly under Windows * It looks like the lit per test timeout tests fail So I'm reverting for now. Once the above failures are fixed running lit's tests can be enabled again. llvm-svn: 257268
* Teach the CMake build system to run lit's test suite. These can be runDan Liew2016-01-082-2/+24
| | | | | | | | | | | | | | | directy with ``make check-lit`` and are run as part of ``make check-all``. In principle we should run lit's testsuite before testing LLVM using lit so that any problems with lit get discovered before testing LLVM so we can bail out early. However this implementation (``check-all`` runs all tests together) seemed simpler and will still report failing lit tests. Note that the tests and the configured ``lit.site.cfg`` have to be copied into the build directory to avoid polluting the source tree. llvm-svn: 257221
* [lit] Implement support of per test timeout in lit.Dan Liew2015-12-2710-0/+279
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
* [LIT] Fix discovery.py test once againEric Fiselier2015-07-211-4/+4
| | | | llvm-svn: 242758
* [LIT] Fix failing LIT testsEric Fiselier2015-06-134-6/+8
| | | | | | | | | | | | | | | | | | | | | | | | Summary: I spend some time trying to get the LIT test suite passing. Here are the changes that I needed to make on my machine. I made the following changes for the following reasons. 1. google-test.py: The Google test format now checks for "[ PASSED ] 1 test." to check if a test passes. 2. discovery.py: The output appears in a different order on my machine than it did in the test. 3. unittest-adaptor.py: The output appears in a different order on my machine than it did in the test. 4. The classname is now formed differently in `getJUnitXML(...)`. I'm not sure what is causing the output order to differ in discovery.py and unittest-adaptor.py. Does anybody have any thoughts? Reviewers: ddunbar, danalbert, jroelofs Reviewed By: jroelofs Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9864 llvm-svn: 239663
* Require python 2.7.Rafael Espindola2014-12-121-2/+0
| | | | | | | | | | | | | We were already requiring 2.5, which meant that people on old linux distros had to upgrade anyway. Requiring python 2.6 will make supporting 3.X easier as we can use the 3.X exception syntax. According to the discussion on llvmdev, there is not much value is requiring just 2.6, we may as well just require 2.7. llvm-svn: 224129
* Add a test-case for lit xunit outputChris Matthews2014-12-031-0/+10
| | | | llvm-svn: 223307
* [lit] Add an --output option, for writing results in a machine readable form.Daniel Dunbar2013-09-143-2/+27
| | | | llvm-svn: 190738
* [lit] Add support for attach arbitrary metrics to test results.Daniel Dunbar2013-09-113-0/+63
| | | | | | | | | - This is a work-in-progress and all details are subject to change, but I am trying to build up support for allowing lit to be used as a driver for performance tests (or other tests which might want to record information beyond simple PASS/FAIL). llvm-svn: 190535
* [lit] Allow config files to pass arbitrary values to child configs.Daniel Dunbar2013-09-032-0/+6
| | | | | | | - This aligns with how existing test suites end up wanting to use the local config files, conceptually it makes sense to consider them to be inherited. llvm-svn: 189885
* [lit] [tests] Add missing test input file.Daniel Dunbar2013-08-291-0/+6
| | | | llvm-svn: 189561
* [lit] Fix internal shell's argv[0] handling.Daniel Dunbar2013-08-291-1/+2
| | | | | | | - At least on OS X, it is important for correct behavior of /bin/[ that argv[0] is passed as written, and not as the full executable path. llvm-svn: 189559
* [lit] Lift XFAIL handling to core infrastructure.Daniel Dunbar2013-08-211-0/+5
| | | | llvm-svn: 188949
* [lit] Add test coverage of gtest format.Daniel Dunbar2013-08-143-0/+57
| | | | llvm-svn: 188417
* [lit] Ensure test output is converted to strings where possible.Daniel Dunbar2013-08-145-4/+43
| | | | | | - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
* [lit] Support parsing scripts with inconsistent or invalid encodings.Daniel Dunbar2013-08-141-0/+3
| | | | | | | | | | | | - For whatever reason, we have a lot of test files with bogus unicode characters. This patch allows those scripts to still be parsed on Python3 by changing the parsing logic to work on binary files, and only require the actual script commands to be convertible to ascii. - This patch has been tweaked to now ensure that the command strings are not of unicode type on Python 2.6-7. llvm-svn: 188398
* Revert r188376, "[lit] Support parsing scripts with inconsistent or invalid ↵Daniel Dunbar2013-08-141-3/+0
| | | | | | encodings.", this doesn't work yet for bots using the internal shell. llvm-svn: 188379
* [lit] Support parsing scripts with inconsistent or invalid encodings.Daniel Dunbar2013-08-141-0/+3
| | | | | | | | | - For whatever reason, we have a lot of test files with bogus unicode characters. This patch allows those scripts to still be parsed on Python3 by changing the parsing logic to work on binary files, and only require the actual script commands to be convertible to ascii. llvm-svn: 188376
* [lit] Fix tests to execute lit with same python as invoked with.Daniel Dunbar2013-08-143-3/+6
| | | | llvm-svn: 188358
* [lit] Update lit's own tests to use lit_config and lit package, as appropriate.Daniel Dunbar2013-08-0911-4/+15
| | | | llvm-svn: 188107
* [lit] Change --show-{tests,suites} to exit after printing.Daniel Dunbar2013-08-081-10/+5
| | | | | | - This is a more sensible behavior than printing and also running tests. llvm-svn: 188009
* [lit] tests: Change discovery test to use --show-tests.Daniel Dunbar2013-08-071-25/+31
| | | | llvm-svn: 187851
* lit: Fix infinite recursion when an out-of-tree test root is located inside ↵Daniel Dunbar2013-04-124-0/+27
| | | | | | the source test root. llvm-svn: 179402
* lit: Add a test for discovery w/ test_exec_root (out-of-tree test root).Daniel Dunbar2013-04-123-9/+60
| | | | llvm-svn: 179401
* lit: Add a test for discovery when exact test names are given.Daniel Dunbar2013-04-111-0/+15
| | | | llvm-svn: 179247
* lit: Add a trivial test of the basic progress bar.Daniel Dunbar2013-04-116-0/+22
| | | | llvm-svn: 179243
* [lit] Add a test for internal shell execution behaviors.Daniel Dunbar2013-01-3110-0/+125
| | | | llvm-svn: 174102
* [lit] Add a test for the various ShTest format features.Daniel Dunbar2013-01-3116-0/+72
| | | | llvm-svn: 174072
* [lit] Move unittest adaptor code into discovery module.Daniel Dunbar2013-01-314-0/+25
| | | | | | - Also, add a test for it. llvm-svn: 174019
* [lit] Run the ShUtil tests as part of the test suite.Daniel Dunbar2013-01-312-0/+4
| | | | llvm-svn: 174018
* [lit] Set parallel mode for coverage tests.Daniel Dunbar2013-01-311-1/+1
| | | | llvm-svn: 174017
* [lit] Add a basic discovery test.Daniel Dunbar2013-01-319-0/+41
| | | | llvm-svn: 174001
* [lit] Add a script for checking test coverage.Daniel Dunbar2013-01-311-0/+11
| | | | llvm-svn: 174000
* [lit] Add a test suite for lit itself.Daniel Dunbar2013-01-312-0/+41
llvm-svn: 173999
OpenPOWER on IntegriCloud