summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
* lit googletest.py: Don't raise StopIteration in generatorHans Wennborg2020-06-161-1/+1
| | | | | | | | | | The intention here seems to be to end the generator function, but with modern Python, raising StopIteration causes a runtime error (https://www.python.org/dev/peps/pep-0479/). Differential revision: https://reviews.llvm.org/D79169 (cherry picked from commit 88aad9b9f05702585eb823d0188996f4cf62070a)
* Bump version to 10.0.1Tom Stellard2020-04-131-1/+1
|
* Make check-llvm run 50% faster on macOS, 18% faster on Windows.Nico Weber2020-01-061-6/+30
| | | | | | | | | | | | | | | | | | | | While looking at cycle time graphs of some of my bots, I noticed that 327894859cc made check-llvm noticeably slower on macOS and Windows. As it turns out, the 5 substitutions added in that change were enough to cause lit to thrash the build-in cache in re.compile() (re.sub() is implemented as re.compile().sub()), and apparently applySubstitutions() is on the cricital path and slow when all regexes need to compile all the time. (See `_MAXCACHE = 512` in cpython/Lib/re.py) Supporting full regexes for lit substitutions seems a bit like overkill, but for now add a simple unbounded cache to recover the lost performance. No intended behavior change.
* [lit] Add "from .main import main" back into lit/__init__.pyJessica Paquette2019-12-191-0/+1
| | | | | | | | | | | | | | | A refactoring commit (cf252240) removed this line. Removing it broke installing lit with pip and setup.py. This adds the line back in so that we can install lit again. For an example of how this appeared, see: http://green.lab.llvm.org/green/job/LNT_Tests/5853/ File "/Users/buildslave/jenkins/...s/__init__.py", line 2453, in resolve raise ImportError(str(exc)) ImportError: 'module' object has no attribute 'main'
* Fix buildbot failures after removing REQUIRES-ANYNemanja Ivanovic2019-12-171-10/+3
| | | | | | | It would appear that the removal of this lit feature was incomplete and there is a test case that still tests for this. This patch removes the remaining tests to bring the bots back to green. I would encourage the author to do a post-commit review on this in case there is a more desirable fix.
* [lit] Fix internal diff newlines for -w/-bJoel E. Denny2019-12-175-3/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For example, without this patch: ``` $ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt *** /tmp/foo.txt --- /tmp/bar.txt *************** *** 1,2 **** 1! 2--- 1,2 ---- 1! 20 ``` With this patch: ``` $ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt *** /tmp/foo.txt --- /tmp/bar.txt *************** *** 1,2 **** 1 ! 2 --- 1,2 ---- 1 ! 20 ``` Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D71577
* [lit] Remove lit's REQUIRES-ANY directiveThomas Preud'homme2019-12-173-21/+0
| | | | | | | | | | | | | | | | | Summary: Remove REQUIRES-ANY alias lit directive since it is hardly used and can be easily implemented using an OR expression using REQUIRES. Fixup remaining testcases still using REQUIRES-ANY. Reviewers: probinson, jdenny, gparker42 Reviewed By: gparker42 Subscribers: eugenis, asb, rbar, johnrusso, simoncook, sabuasal, niosHD, delcypher, jrtc27, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, lenary, s.egerton, pzheng, sameer.abuasal, apazos, luismarques, cfe-commits, #sanitizers, llvm-commits Tags: #llvm, #clang, #sanitizers Differential Revision: https://reviews.llvm.org/D71408
* [lit] [windows] Make sure to convert all path separators to backslashes in ↵Martin Storsjö2019-12-171-0/+4
| | | | | | | | | | | | NT style \\?\... paths E.g. the mingw python distributed in msys2 (the mingw one, which is a normal win32 application and doesn't use the msys2 runtime itself), despite being a normal win32 python, still uses forward slashes. This works fine for other cases (many, but not all), but when constructing a raw NT path, all path separators must be backslashes. Differential Revision: https://reviews.llvm.org/D71490
* [lit] max_failures does not need to be stored in LitConfigJulian Lettner2019-12-164-22/+24
|
* [lit] Small cleanups. NFCIJulian Lettner2019-12-137-53/+53
| | | | | Remove unnecessary (argument same as default), cleanup imports, use "pythonic" names for variables, and general formatting.
* [lit] Improve formatting of error messages. NFCJulian Lettner2019-12-112-10/+10
|
* Fix `sed -e s@FOO@%/S@` and similar when there's @'s in the working directoryDaniel Sanders2019-12-031-0/+14
| | | | | | | | | | | | | | | | | | | | | | Jenkins sometimes starts a new working directory by appending @2 (or incrementing the number if the @n suffix is already there). This causes several clang tests to fail as: s@INPUT_DIR@%/S/Inputs@g gets expanded to the invalid: s@INPUT_DIR@/path/to/workdir@2/Inputs@g ~~~~~~~~~~ where the part marked with ~'s is interpreted as the flags. These are invalid and the test fails. Previous fixes simply exchanged the @ character for another like | but that's just moving the problem. Address it by adding an expansion that escapes the @ character we're using as a delimiter as well as other magic characters in the replacement of sed's s@@@. There's still room for expansions to cause trouble though. One I ran into while testing this was that having a directory called foo@bar causes lots of `CHECK-NOT: foo` directives to match. There's also things like directories containing `\1`
* [lit] Be more explicit about the state of testsJulian Lettner2019-12-021-17/+19
| | | | | | | | | | | | | | Tests go through the following stages: *) discovered *) filtered *) executed Only executed tests have a result (e.g., PASS, FAIL, XFAIL, etc.). See "result codes" in Test.py. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D70612
* [lit] Attempt to print test summary on CTRL+CJulian Lettner2019-11-223-20/+20
|
* Fix up lit's tests to run in a multi-config build environment.Paul Robinson2019-11-141-0/+3
| | | | Differential Revision: https://reviews.llvm.org/D70239
* Update lit infra to detect "MemoryWithOrigins' sanitizer build.Sumanth Gundapaneni2019-11-141-1/+1
| | | | Differential Revision: https://reviews.llvm.org/D68399
* [lit] Better/earlier errors for empty runsJulian Lettner2019-11-124-9/+41
| | | | | | | | | | | | | | | | | | | | | | | | | Fail early, when we discover no tests at all, or filter out all of them. There is also `--allow-empty-runs` to disable test to allow workflows like `LIT_FILTER=abc ninja check-all`. Apparently `check-all` invokes lit multiple times if certain projects are enabled, which would produce unwanted "empty runs". Specify via `LIT_OPTS=--allow-empty-runs`. There are 3 causes for empty runs: 1) No tests discovered. This is always an error. Fix test suite config or command line. 2) All tests filtered out. This is an error by default, but can be suppressed via `--alow-empty-runs`. Should prevent accidentally passing empty runs, but allow the workflow above. 3) The number of shards is greater than the number of tests. Currently, this is never an error. Personally, I think we should consider making this an error by default; if this happens, you are doing something wrong. I added a warning but did not change the behavior, since this warrants more discussion. Reviewed By: atrick, jdenny Differential Revision: https://reviews.llvm.org/D70105
* [lit] Protect full test suite from FILECHECK_OPTSJoel E. Denny2019-11-062-3/+10
| | | | | | | | | | | | | | | | | | lit's test suite calls lit multiple times for various sample test suites. `FILECHECK_OPTS` is safe for FileCheck calls in lit's test suite. It's not safe for FileCheck calls in the sample test suites, whose output affects the results of lit's test suite. Without this patch, only one such sample test suite is protected from `FILECHECK_OPTS`, and currently `shtest-shell.py` breaks with `FILECHECK_OPTS=-vv`. Moreover, it's hard to predict the future, especially false passes. Thus, this patch protects all existing and future sample test suites from `FILECHECK_OPTS` (and the deprecated `FILECHECK_DUMP_INPUT_ON_FAILURE`). Reviewed By: probinson Differential Revision: https://reviews.llvm.org/D65156
* Revert "[analyzer] Add test directory for scan-build."Volodymyr Sapsai2019-11-051-1/+0
| | | | | | | | This reverts commit 0aba69eb1a01c44185009f50cc633e3c648e9950 with subsequent changes to test files. It caused test failures on GreenDragon, e.g., http://green.lab.llvm.org/green/job/clang-stage1-cmake-RA-incremental/
* Revert "[lit] Better/earlier errors when no tests are executed"Julian Lettner2019-11-054-29/+9
| | | | This reverts commit d8f2bff75126c6dde694ad245f9807fa12ad5630.
* [lit] Fix `not` calling internal commandsJoel E. Denny2019-11-0521-33/+325
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Without this patch, when using lit's internal shell, if `not` on a lit RUN line calls `env`, `diff`, or any of the other in-process shell builtins that lit implements, lit accidentally searches for the latter as an external executable. What's worse is that works fine when a developer is testing on a platform where those executables are available and behave as expected, but it then breaks on other platforms. `not` seems useful for some builtins, such as `diff`, so this patch supports such uses. `not --crash` does not seem useful for builtins, so this patch diagnoses such uses. In all cases, this patch ensures shell builtins are found behind any sequence of `env` and `not` commands. `not` calling `env` calling an external command appears useful when the `env` and external command are part of a lit substitution, as in D65156. This patch supports that by looking through any sequence of `env` and `not` commands, building the environment from the `env`s, and storing the `not`s. The `not`s are then added back to the command line without the `env`s to execute externally. This avoids the need to replicate the `not` implementation, in particular the `--crash` option, in lit. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D66531
* [analyzer] Add test directory for scan-build.Devin Coughlin2019-11-041-0/+1
| | | | | | | | | | The static analyzer's scan-build script is critical infrastructure but is not well tested. To start to address this, add a new test directory under tests/Analysis for scan-build lit tests and seed it with several tests. The goal is that future scan-build changes will be accompanied by corresponding tests. Differential Revision: https://reviews.llvm.org/D69781
* [lit] Move measurement of testing time out of Run.executeJulian Lettner2019-11-042-10/+8
|
* [lit] Better/earlier errors when no tests are executedJulian Lettner2019-11-044-9/+29
| | | | Fail early, when we discover no tests at all, or filter out all of them.
* [lit] Fix internal env calling envJoel E. Denny2019-11-014-8/+64
| | | | | | | | | | | | | | | | Without this patch, when using lit's internal shell, if `env` on a lit RUN line calls `env`, lit accidentally searches for the latter as an external executable. What's worse is that works fine when a developer is testing on a platform where `env` is available and behaves as expected, but it then breaks on other platforms. `env` calling `env` can make sense if one such `env` is within a lit substitution, as in D65156 and D65121. This patch ensures that lit executes both as internal commands. Reviewed By: probinson, mgorny, rnk Differential Revision: https://reviews.llvm.org/D65697
* [lit] Extract Display.print_header functionJulian Lettner2019-10-313-16/+19
|
* [lit] Always print newline before test time/summaryJulian Lettner2019-10-311-2/+1
| | | | Slightly decreases the time I need to parse the test summary.
* [lit] Fix internal env calling other internal commandsJoel E. Denny2019-10-318-63/+95
| | | | | | | | | | | | | | | | | | Without this patch, when using lit's internal shell, if `env` on a lit RUN line calls `cd`, `mkdir`, or any of the other in-process shell builtins that lit implements, lit accidentally searches for the latter as an external executable. This patch puts such builtins in a map so that boilerplate for them need be implemented only once. This patch moves that handling after processing of `env` so that `env` calling such a builtin can be detected. Finally, because such calls appear to be useless, this patch takes the safe approach of diagnosing them rather than supporting them. Reviewed By: probinson, mgorny, rnk Differential Revision: https://reviews.llvm.org/D66506
* [lit] Rename ProgressDisplay -> DisplayJulian Lettner2019-10-311-21/+23
|
* [lit] Add missing importReid Kleckner2019-10-301-0/+1
| | | | Apparently llvm-lit.py does not execute this path
* [lit] Silence warning about importing the resource module on WindowsReid Kleckner2019-10-301-1/+3
| | | | | lit was printing this warning on every test run on Windows, and that is not necessary.
* [lit] Change progress bar color to red on first failureJulian Lettner2019-10-302-3/+8
|
* [lit] Add helper for `test.result.code.isFailure`Julian Lettner2019-10-303-12/+13
|
* [lit] Extract `_install_win32_signal_handler` functionJulian Lettner2019-10-301-9/+12
|
* [lit] Refactor ordering of testsJulian Lettner2019-10-292-22/+28
|
* [lit] Small improvements in cl_arguments.pyJulian Lettner2019-10-292-58/+32
| | | | | | *) `--max-tests` should be positive integer *) `--max-time` should be positive integer *) Remove unnecessary defaults for command line option parsing
* [lit] Extend internal diff to support `-` argumentJoel E. Denny2019-10-297-5/+146
| | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-` as a command-line option. This patch adds support for `-` to mean stdin. Reviewed By: probinson, rnk Differential Revision: https://reviews.llvm.org/D67643
* [lit] Make internal diff work in pipelinesJoel E. Denny2019-10-299-271/+313
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - # RUN: not diff file1 file2 | FileCheck %s ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` cannot currently be used in pipelines and doesn't recognize `-` as a command-line option. To enable pipelines, this patch moves lit's `diff` implementation into an out-of-process script, similar to lit's `cat` implementation. A follow-up patch will implement `-` to mean stdin. Also, when lit's `diff` prints differences to stdout in Windows, this patch ensures it always terminate lines with `\n` not `\r\n`. That way, strict FileCheck directives checking the `diff` output succeed in both Linux and Windows. This wasn't an issue when `diff` was internal to lit because `diff` didn't then write to the true stdout, which is where the `\n` -> `\r\n` conversion happened in Python. Reviewed By: probinson, stella.stamenova Differential Revision: https://reviews.llvm.org/D66574
* [lit] Remove callback indirectionJulian Lettner2019-10-281-3/+3
| | | | | The callback provides no benefits since `run.execute()` does not take any arguments anymore.
* [lit] Refactor merging of user parametersJulian Lettner2019-10-282-13/+11
|
* [lit] Remove redundant comments from main functionJulian Lettner2019-10-281-19/+9
| | | | | Hopefully the functionality is now clear due to the use of small, well-named helper functions.
* [lit] Make main.py a pure Python moduleJulian Lettner2019-10-282-9/+3
| | | | Running it directly as a tool, that is what lit.py is for.
* [lit] Drop the user-site packages directory from search paths when running testsAlex Lorenz2019-10-271-0/+4
| | | | | | | Do not add user-site packages directory to the python search path. This avoids test failures if there's an incompatible lit module installed inside the user-site packages directory, as it gets prioritized over the lit from the PYTHONPATH.
* [lit] Move sharding logic into separate functionJulian Lettner2019-10-254-23/+36
|
* [lit] Don't fail when printing test output with special charsJoel E. Denny2019-10-254-3/+35
| | | | | | | | | This addresses a UnicodeEncodeError when using Python 3.6.5 in Windows 10. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D69207
* [lit] Move increase_process_limit to ParallelRunJulian Lettner2019-10-222-25/+24
| | | | | | | Increasing the process limit only makes sense when we use multiple processes. llvm-svn: 375474
* [lit] Simplify test scheduling via multiprocessing.PoolJulian Lettner2019-10-211-24/+17
| | | | llvm-svn: 375458
* [lit] Remove redundancy from names and commentsJulian Lettner2019-10-213-31/+26
| | | | llvm-svn: 375456
* [LLDB] [Windows] Initial support for ARM register contextsMartin Storsjo2019-10-211-0/+2
| | | | | | Differential Revision: https://reviews.llvm.org/D69226 llvm-svn: 375392
* [lit] Reduce value of synthesized timeoutsJulian Lettner2019-10-181-2/+3
| | | | | | | | | | | Large timeout values (one year, positive infinity) trip up Python on Windows with "OverflowError: timeout value is too large". One week seems to work and is still large enough in practice. Thanks to Simon Pilgrim for helping me test this. https://reviews.llvm.org/rL375171 llvm-svn: 375264
OpenPOWER on IntegriCloud