summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit/tests
Commit message (Collapse)AuthorAgeFilesLines
...
* Revert "[lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configs"Reid Kleckner2017-08-023-30/+1
| | | | | | | This reverts r309602, check-lit still leaves Output directories in the source directory. llvm-svn: 309833
* [lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configsReid Kleckner2017-07-313-1/+30
| | | | | | | | | | | | | | | | | | | Summary: This is an alternative solution to running the lit test suite on bots without polluting the source directory. Each input test suite gets an auto-generated site config in the build directory that points back to the test input source directory. This adds some cmake comlexity, but now we don't need to remove and re-copy the test input directory before every test. Reviewers: delcypher, modocache Subscribers: mgorny, llvm-commits Differential Revision: https://reviews.llvm.org/D36026 llvm-svn: 309602
* lit::shtest-format.py: Make write-bad-encoding.py py3-aware.NAKAMURA Takumi2017-07-291-1/+1
| | | | | | | | | | Traceback (most recent call last): File "llvm/utils/lit/tests/Inputs/shtest-format/external_shell/write-bad-encoding.py", line 5, in <module> sys.stdout.write(b"a line with bad encoding: \xc2.") sys.stdout.write doesn't accept bytes but sys.stdout.buffer.write accepts. llvm-svn: 309473
* [lit] Use a %{python} substitution to avoid relying on python being on PATHReid Kleckner2017-07-285-9/+7
| | | | llvm-svn: 309434
* [lit] Dump some FileCheck inputs to try to debug some failing testsReid Kleckner2017-07-282-0/+6
| | | | llvm-svn: 309400
* [lit] Fix shtest-format external_shell failuresReid Kleckner2017-07-283-4/+7
| | | | | | | | | | When using win32 cmd.exe, turn off command echoing at the beginning of the script (@echo off). Replace a bash shell script with a python script for the fail_with_bad_encoding test. llvm-svn: 309399
* [lit] Port googletest lit tests to WindowsReid Kleckner2017-07-286-22/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: The technique of directly calling subprocess.Popen on a python script doesn't work on Windows. The executable path of the command must refer to a valid win32 executable. Instead, rename all the python scripts masquerading as gtest executables to have .py extensions, so we can easily detect then and call the python executable for them. Do this on Linux as well as Windows for consistency. The test suite directory names also come out in lower-case on Windows. We can consider removing that in a later patch. This change just updates the FileCheck lines to match on Windows. Fixes PR33933 Reviewers: modocache, mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35909 llvm-svn: 309347
* Un-revert "Teach the CMake build system to run lit's test suite. These can ↵Brian Gesiak2017-07-272-2/+24
| | | | | | | | | | | | | | | | | | | | | | be run" Summary: Depends on https://reviews.llvm.org/D35879. This reverts rL257268, which in turn was a revert of rL257221. https://reviews.llvm.org/D35879 marks the tests in the lit test suite that fail on Windows as XFAIL, which should allow these tests to pass on Windows-based buildbots. Reviewers: delcypher, beanz, mgorny, jroelofs, rnk Reviewed By: mgorny Subscribers: rnk, ddunbar, george.karpenkov, llvm-commits Differential Revision: https://reviews.llvm.org/D35880 llvm-svn: 309310
* [lit] Fix order of checks in shtest-shell.py testBrian Gesiak2017-07-272-0/+10
| | | | | | | | | | | | | | | | | | Summary: An expectation in `utils/lit/tests/Inputs/shtest-shell/redirects.txt` expects that first a string printed to stdout is seen, and then a string printed to stderr. Add `flush()` calls to ensure that stdout is printed before stderr, as expected. Reviewers: rnk, mgorny, jroelofs Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35947 llvm-svn: 309292
* [lit] Fix race between shtest-shell and max-failures testsReid Kleckner2017-07-262-4/+10
| | | | | | | Previously these tests would use the same Output directory leading to flaky non-deterministic failures. llvm-svn: 309227
* [lit] Fix shtest-shell and max-failures lit tests on WindowsReid Kleckner2017-07-267-16/+10
| | | | | | | | | Rewrite the write-to-stderr.sh and write-to-stdout-and-stderr.sh shell scripts as python scripts and call python on them. Fixes PR33940 llvm-svn: 309200
* [lit] Fix shtest-output-printing.py on Windows by matching either / or \\Reid Kleckner2017-07-261-4/+1
| | | | | | Fixes PR33938 llvm-svn: 309198
* [lit] Fix discovery.py on Windows by matching backslashes when necessaryReid Kleckner2017-07-261-22/+19
| | | | | | Fixes PR33932 llvm-svn: 309194
* [lit] Un-XFAIL selecting.py test on WindowsReid Kleckner2017-07-261-3/+0
| | | | | | | | This passes locally for me, which fails the overall lit test suite. I can't debug a passing test, but I will try to help debug the test when we get some failing logs. llvm-svn: 309190
* Un-XFAIL some internal lit tests on Windows, they pass for me locallyReid Kleckner2017-07-263-9/+0
| | | | llvm-svn: 309144
* Fix LIT test breakageGeorge Karpenkov2017-07-261-1/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D35867 llvm-svn: 309140
* [lit] Mark several of lit's tests XFAIL on WindowsBrian Gesiak2017-07-2613-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: rL257221 attempted to run lit's own test suite continuously, but that commit was reverted because lit's test suite does not pass on Windows. Because lit's tests do not run continuously, they often regress. In order to un-revert rL257221, mark lit tests that fail as XFAIL for Windows platforms. Test Plan: On a Windows development environment, follow the instructions in utils/lit/README.txt to run lit's test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test suite is run and a successful exit code is returned. Reviewers: mgorny, rnk, delcypher, beanz Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35879 llvm-svn: 309123
* [lit] Modify LIT to accept environment variable LIT_FILTER to select tests.George Karpenkov2017-07-071-0/+5
| | | | | | | | | This is especially useful when lit is invoked indirectly by the build system, and additional arguments can not be easily specified. Differential Revision: https://reviews.llvm.org/D35091 llvm-svn: 307339
* [lit] Implement timeouts and max_time for process pool testingReid Kleckner2017-04-062-38/+42
| | | | | | | | | | | | | | | | | | | This is necessary to pass the lit test suite at llvm/utils/lit/tests. There are some pre-existing failures here, but now switching to pools doesn't regress any tests. I had to change test-data/lit.cfg to import DummyConfig from a module to fix pickling problems, but I think it'll be OK if we require test formats to be written in real .py modules outside lit.cfg files. I also discovered that in some circumstances AsyncResult.wait() will not raise KeyboardInterrupt in a timely manner, but you can pass a non-zero timeout to work around this. This makes threading.Condition.wait use a polling loop that runs through the interpreter, so it's capable of asynchronously raising KeyboardInterrupt. llvm-svn: 299605
* lit: remove python2-ismsBrian Gesiak2017-03-221-2/+2
| | | | | | | | | | | | | | | | | Summary: `assert.assertItemEqual` went away in Python 3. Seeing how lists are ordered, comparing a list against each other should work just as well. Patch by @jbergstroem (Johan Bergström). Reviewers: modocache, gparker42 Reviewed By: modocache Differential Revision: https://reviews.llvm.org/D31229 llvm-svn: 298479
* Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILGreg Parker2017-01-2512-7/+109
| | | | | | | | and UNSUPPORTED" This reverts the revert in r292942. llvm-svn: 293007
* Revert "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILAlex Lorenz2017-01-2412-109/+7
| | | | | | | | | | | and UNSUPPORTED" After r292904 llvm-lit fails to emit the test results in the XML format for Apple's internal buildbots. rdar://30164800 llvm-svn: 292942
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-2412-7/+109
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292904
* Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED"Greg Parker2017-01-2415-122/+13
| | | | | | This change needs to be better-coordinated with libc++. llvm-svn: 292900
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-2415-13/+122
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292896
* [lit] Support sharding testsuites, for parallel execution.Graydon Hoare2017-01-181-0/+90
| | | | | | | | | | | | | | | | | | | | | | | Summary: This change equips lit.py with two new options, --num-shards=M and --run-shard=N (set by default from env vars LIT_NUM_SHARDS and LIT_RUN_SHARD). The options must be used together, and N must be in 1..M. Together these options effect only test selection: they partition the testsuite into M equal-sized "shards", then select only the Nth shard. They can be used in a cluster of test machines to achieve a very crude (static) form of parallelism, with minimal configuration work. Reviewers: modocache, ddunbar Reviewed By: ddunbar Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D28789 llvm-svn: 292417
* [lit] Support custom parsers in parseIntegratedTestScriptEric Fiselier2016-12-053-0/+141
| | | | | | | | | | | | | | | | | | | | | | | Summary: Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers. This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively. As an example after this change libc++ can implement the `MODULES-DEFINES` simply using: ``` mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST) parseIntegratedTestScript(test, additional_parsers=[mparser]) macro_list = mparser.getValue() ``` Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs Subscribers: mgrang, llvm-commits, cfe-commits Differential Revision: https://reviews.llvm.org/D27005 llvm-svn: 288694
* [lit] Fix test shtest-timeout.py for modern outputMichal Gorny2016-10-121-10/+9
| | | | | | | | | | Update the CHECK lines in the shtest-timeout.py lit test to account for the current output. The output has been changed in r271610 without adjusting the tests. Differential Revision: https://reviews.llvm.org/D25236 llvm-svn: 284057
* [lit] Run unit tests as part of lit test suiteBrian Gesiak2016-10-121-0/+112
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: The Python file `utils/lit/lit/ShUtil.py` contains: 1. Logic used by lit itself 2. A set of unit tests for that logic, which can be run by invoking `python utils/lit/lit/ShUtil.py` Move these unit tests to a `tests/unit` subdirectory of lit, and run the tests as part of lit's test suite. This ensures that, should the lit test suite be included in LLVM's own regression test suite, these unit tests will also be run. (Instructions on how to run lit's test suite can be found in `utils/lit/README.txt`.) Reviewers: ddunbar, echristo, delcypher, beanz Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D25411 llvm-svn: 283968
* [lit] Use argparse instead of optparseChris Bieneman2016-10-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | Summary: optparse is deprecated in Python 2.7, which is the minimum version of Python required to run the LLVM test suite. Replace its usage in lit with argparse, optparse's 2.7 replacement module. argparse has several benefits over optparse, but this commit does not make use of those benefits yet. Instead, it simply uses the new API, and attempts to keep the number of changes to a minimum. Confirmed that lit's test suite, as well as LLVM's regression test suite, still pass with these changes. Patch By Brian Gesiak! Reviewers: ddunbar, echristo, beanz, delcypher Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25173 llvm-svn: 283152
* [lit] Remove unused imports (NFC)Chris Bieneman2016-10-032-3/+0
| | | | | | | | | | | | Reviewers: ddunbar, echristo, beanz Patch by Brian Gesiak! Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25169 llvm-svn: 283089
* [lit] Add a --max-failures option.Daniel Dunbar2016-09-261-0/+14
| | | | | | | | | - This is primarily useful as a "fail fast" mode for lit, where it will stop running tests after the first failure. - Patch by Max Moiseev. llvm-svn: 282452
* [lit] Improve logging with file redirection.Daniel Dunbar2016-06-073-5/+7
| | | | | | | | | - This will cause lit to automatically include the first 1K of data in redirected output files when a command fails (previously if the command failed, but the main point of the test was, say, a `FileCheck` later on, then the log wasn't helpful in showing why the command failed). llvm-svn: 272021
* [lit] Improve readability of failing scripts.Daniel Dunbar2016-06-025-6/+38
| | | | | | | | | | | | | | | | | | | | | | | - This only applies to scripts executed by the _internal_ shell script interpreter. - This patch reworks the log to look more like a shell transcript, and be less verbose (but in the interest of calling attention to the important parts). Here is an example of the new format, for commands with/without failures and with/without output: ``` $ true $ echo hi hi $ false note: command had no output on stdout or stderr error: command failed with exit status 1 ``` llvm-svn: 271610
* Add "REQUIRES-ANY" feature testEric Fiselier2016-06-023-2/+8
| | | | | | | | | | | | | | | | Summary: This patch adds a "REQUIRES-ANY" feature test that is disjunctive. This marks a test as `UNSUPPORTED` if none of the specified features are available. Libc++ has the need to write feature test such as `// REQUIRES-ANY: c++98, c++03` when testing of behavior that is specific to older dialects but has since changed. Reviewers: rnk, ddunbar Subscribers: ddunbar, probinson, llvm-commits, cfe-commits Differential Revision: http://reviews.llvm.org/D20757 llvm-svn: 271468
* [lit][googletest] Handle upstream gtest outputDaniel Dunbar2016-03-313-0/+61
| | | | | | | | | | | | | | | | | | | Summary: Upstream googletest prints "Running main() from gtest_main.cc" to stdout prior to running tests. LLVM removed that print statement in r61540. If a user were to use lit to run tests that use upstream googletest, however, lit reports "Running main()" as an invalid test name. To avoid such a failure, add an extra conditional to `formats/googletest.py`. Also add tests to demonstrate the modified behavior. Reviewers: abdulras, ddunbar Subscribers: ddunbar, llvm-commits, kastiglione Differential Revision: http://reviews.llvm.org/D18606 llvm-svn: 265034
* Revert "Teach the CMake build system to run lit's test suite. These can be run"Dan Liew2016-01-092-24/+2
| | | | | | | | | | | | | | This reverts r257221. This caused several build bot failures * It looks like some of the tests don't work correctly under Windows * It looks like the lit per test timeout tests fail So I'm reverting for now. Once the above failures are fixed running lit's tests can be enabled again. llvm-svn: 257268
* Teach the CMake build system to run lit's test suite. These can be runDan Liew2016-01-082-2/+24
| | | | | | | | | | | | | | | directy with ``make check-lit`` and are run as part of ``make check-all``. In principle we should run lit's testsuite before testing LLVM using lit so that any problems with lit get discovered before testing LLVM so we can bail out early. However this implementation (``check-all`` runs all tests together) seemed simpler and will still report failing lit tests. Note that the tests and the configured ``lit.site.cfg`` have to be copied into the build directory to avoid polluting the source tree. llvm-svn: 257221
* [lit] Implement support of per test timeout in lit.Dan Liew2015-12-2710-0/+279
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
* [LIT] Fix discovery.py test once againEric Fiselier2015-07-211-4/+4
| | | | llvm-svn: 242758
* [LIT] Fix failing LIT testsEric Fiselier2015-06-134-6/+8
| | | | | | | | | | | | | | | | | | | | | | | | Summary: I spend some time trying to get the LIT test suite passing. Here are the changes that I needed to make on my machine. I made the following changes for the following reasons. 1. google-test.py: The Google test format now checks for "[ PASSED ] 1 test." to check if a test passes. 2. discovery.py: The output appears in a different order on my machine than it did in the test. 3. unittest-adaptor.py: The output appears in a different order on my machine than it did in the test. 4. The classname is now formed differently in `getJUnitXML(...)`. I'm not sure what is causing the output order to differ in discovery.py and unittest-adaptor.py. Does anybody have any thoughts? Reviewers: ddunbar, danalbert, jroelofs Reviewed By: jroelofs Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9864 llvm-svn: 239663
* Require python 2.7.Rafael Espindola2014-12-121-2/+0
| | | | | | | | | | | | | We were already requiring 2.5, which meant that people on old linux distros had to upgrade anyway. Requiring python 2.6 will make supporting 3.X easier as we can use the 3.X exception syntax. According to the discussion on llvmdev, there is not much value is requiring just 2.6, we may as well just require 2.7. llvm-svn: 224129
* Add a test-case for lit xunit outputChris Matthews2014-12-031-0/+10
| | | | llvm-svn: 223307
* [lit] Add an --output option, for writing results in a machine readable form.Daniel Dunbar2013-09-143-2/+27
| | | | llvm-svn: 190738
* [lit] Add support for attach arbitrary metrics to test results.Daniel Dunbar2013-09-113-0/+63
| | | | | | | | | - This is a work-in-progress and all details are subject to change, but I am trying to build up support for allowing lit to be used as a driver for performance tests (or other tests which might want to record information beyond simple PASS/FAIL). llvm-svn: 190535
* [lit] Allow config files to pass arbitrary values to child configs.Daniel Dunbar2013-09-032-0/+6
| | | | | | | - This aligns with how existing test suites end up wanting to use the local config files, conceptually it makes sense to consider them to be inherited. llvm-svn: 189885
* [lit] [tests] Add missing test input file.Daniel Dunbar2013-08-291-0/+6
| | | | llvm-svn: 189561
* [lit] Fix internal shell's argv[0] handling.Daniel Dunbar2013-08-291-1/+2
| | | | | | | - At least on OS X, it is important for correct behavior of /bin/[ that argv[0] is passed as written, and not as the full executable path. llvm-svn: 189559
* [lit] Lift XFAIL handling to core infrastructure.Daniel Dunbar2013-08-211-0/+5
| | | | llvm-svn: 188949
* [lit] Add test coverage of gtest format.Daniel Dunbar2013-08-143-0/+57
| | | | llvm-svn: 188417
OpenPOWER on IntegriCloud