summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit/tests
Commit message (Collapse)AuthorAgeFilesLines
* Fix buildbot failures after removing REQUIRES-ANYNemanja Ivanovic2019-12-171-10/+3
| | | | | | | It would appear that the removal of this lit feature was incomplete and there is a test case that still tests for this. This patch removes the remaining tests to bring the bots back to green. I would encourage the author to do a post-commit review on this in case there is a more desirable fix.
* [lit] Fix internal diff newlines for -w/-bJoel E. Denny2019-12-174-2/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For example, without this patch: ``` $ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt *** /tmp/foo.txt --- /tmp/bar.txt *************** *** 1,2 **** 1! 2--- 1,2 ---- 1! 20 ``` With this patch: ``` $ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt *** /tmp/foo.txt --- /tmp/bar.txt *************** *** 1,2 **** 1 ! 2 --- 1,2 ---- 1 ! 20 ``` Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D71577
* [lit] Remove lit's REQUIRES-ANY directiveThomas Preud'homme2019-12-172-4/+0
| | | | | | | | | | | | | | | | | Summary: Remove REQUIRES-ANY alias lit directive since it is hardly used and can be easily implemented using an OR expression using REQUIRES. Fixup remaining testcases still using REQUIRES-ANY. Reviewers: probinson, jdenny, gparker42 Reviewed By: gparker42 Subscribers: eugenis, asb, rbar, johnrusso, simoncook, sabuasal, niosHD, delcypher, jrtc27, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, lenary, s.egerton, pzheng, sameer.abuasal, apazos, luismarques, cfe-commits, #sanitizers, llvm-commits Tags: #llvm, #clang, #sanitizers Differential Revision: https://reviews.llvm.org/D71408
* [lit] Small cleanups. NFCIJulian Lettner2019-12-131-4/+4
| | | | | Remove unnecessary (argument same as default), cleanup imports, use "pythonic" names for variables, and general formatting.
* [lit] Improve formatting of error messages. NFCJulian Lettner2019-12-111-4/+4
|
* Fix up lit's tests to run in a multi-config build environment.Paul Robinson2019-11-141-0/+3
| | | | Differential Revision: https://reviews.llvm.org/D70239
* [lit] Better/earlier errors for empty runsJulian Lettner2019-11-121-4/+18
| | | | | | | | | | | | | | | | | | | | | | | | | Fail early, when we discover no tests at all, or filter out all of them. There is also `--allow-empty-runs` to disable test to allow workflows like `LIT_FILTER=abc ninja check-all`. Apparently `check-all` invokes lit multiple times if certain projects are enabled, which would produce unwanted "empty runs". Specify via `LIT_OPTS=--allow-empty-runs`. There are 3 causes for empty runs: 1) No tests discovered. This is always an error. Fix test suite config or command line. 2) All tests filtered out. This is an error by default, but can be suppressed via `--alow-empty-runs`. Should prevent accidentally passing empty runs, but allow the workflow above. 3) The number of shards is greater than the number of tests. Currently, this is never an error. Personally, I think we should consider making this an error by default; if this happens, you are doing something wrong. I added a warning but did not change the behavior, since this warrants more discussion. Reviewed By: atrick, jdenny Differential Revision: https://reviews.llvm.org/D70105
* [lit] Protect full test suite from FILECHECK_OPTSJoel E. Denny2019-11-062-3/+10
| | | | | | | | | | | | | | | | | | lit's test suite calls lit multiple times for various sample test suites. `FILECHECK_OPTS` is safe for FileCheck calls in lit's test suite. It's not safe for FileCheck calls in the sample test suites, whose output affects the results of lit's test suite. Without this patch, only one such sample test suite is protected from `FILECHECK_OPTS`, and currently `shtest-shell.py` breaks with `FILECHECK_OPTS=-vv`. Moreover, it's hard to predict the future, especially false passes. Thus, this patch protects all existing and future sample test suites from `FILECHECK_OPTS` (and the deprecated `FILECHECK_DUMP_INPUT_ON_FAILURE`). Reviewed By: probinson Differential Revision: https://reviews.llvm.org/D65156
* Revert "[lit] Better/earlier errors when no tests are executed"Julian Lettner2019-11-051-15/+4
| | | | This reverts commit d8f2bff75126c6dde694ad245f9807fa12ad5630.
* [lit] Fix `not` calling internal commandsJoel E. Denny2019-11-0520-2/+262
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Without this patch, when using lit's internal shell, if `not` on a lit RUN line calls `env`, `diff`, or any of the other in-process shell builtins that lit implements, lit accidentally searches for the latter as an external executable. What's worse is that works fine when a developer is testing on a platform where those executables are available and behave as expected, but it then breaks on other platforms. `not` seems useful for some builtins, such as `diff`, so this patch supports such uses. `not --crash` does not seem useful for builtins, so this patch diagnoses such uses. In all cases, this patch ensures shell builtins are found behind any sequence of `env` and `not` commands. `not` calling `env` calling an external command appears useful when the `env` and external command are part of a lit substitution, as in D65156. This patch supports that by looking through any sequence of `env` and `not` commands, building the environment from the `env`s, and storing the `not`s. The `not`s are then added back to the command line without the `env`s to execute externally. This avoids the need to replicate the `not` implementation, in particular the `--crash` option, in lit. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D66531
* [lit] Better/earlier errors when no tests are executedJulian Lettner2019-11-041-4/+15
| | | | Fail early, when we discover no tests at all, or filter out all of them.
* [lit] Fix internal env calling envJoel E. Denny2019-11-013-3/+53
| | | | | | | | | | | | | | | | Without this patch, when using lit's internal shell, if `env` on a lit RUN line calls `env`, lit accidentally searches for the latter as an external executable. What's worse is that works fine when a developer is testing on a platform where `env` is available and behaves as expected, but it then breaks on other platforms. `env` calling `env` can make sense if one such `env` is within a lit substitution, as in D65156 and D65121. This patch ensures that lit executes both as internal commands. Reviewed By: probinson, mgorny, rnk Differential Revision: https://reviews.llvm.org/D65697
* [lit] Fix internal env calling other internal commandsJoel E. Denny2019-10-317-2/+41
| | | | | | | | | | | | | | | | | | Without this patch, when using lit's internal shell, if `env` on a lit RUN line calls `cd`, `mkdir`, or any of the other in-process shell builtins that lit implements, lit accidentally searches for the latter as an external executable. This patch puts such builtins in a map so that boilerplate for them need be implemented only once. This patch moves that handling after processing of `env` so that `env` calling such a builtin can be detected. Finally, because such calls appear to be useless, this patch takes the safe approach of diagnosing them rather than supporting them. Reviewed By: probinson, mgorny, rnk Differential Revision: https://reviews.llvm.org/D66506
* [lit] Extend internal diff to support `-` argumentJoel E. Denny2019-10-296-2/+135
| | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-` as a command-line option. This patch adds support for `-` to mean stdin. Reviewed By: probinson, rnk Differential Revision: https://reviews.llvm.org/D67643
* [lit] Make internal diff work in pipelinesJoel E. Denny2019-10-297-43/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - # RUN: not diff file1 file2 | FileCheck %s ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` cannot currently be used in pipelines and doesn't recognize `-` as a command-line option. To enable pipelines, this patch moves lit's `diff` implementation into an out-of-process script, similar to lit's `cat` implementation. A follow-up patch will implement `-` to mean stdin. Also, when lit's `diff` prints differences to stdout in Windows, this patch ensures it always terminate lines with `\n` not `\r\n`. That way, strict FileCheck directives checking the `diff` output succeed in both Linux and Windows. This wasn't an issue when `diff` was internal to lit because `diff` didn't then write to the true stdout, which is where the `\n` -> `\r\n` conversion happened in Python. Reviewed By: probinson, stella.stamenova Differential Revision: https://reviews.llvm.org/D66574
* [lit] Drop the user-site packages directory from search paths when running testsAlex Lorenz2019-10-271-0/+4
| | | | | | | Do not add user-site packages directory to the python search path. This avoids test failures if there's an incompatible lit module installed inside the user-site packages directory, as it gets prioritized over the lit from the PYTHONPATH.
* [lit] Don't fail when printing test output with special charsJoel E. Denny2019-10-253-2/+21
| | | | | | | | | This addresses a UnicodeEncodeError when using Python 3.6.5 in Windows 10. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D69207
* Revert r375114: "[lit] Make internal diff work in pipelines"Joel E. Denny2019-10-177-65/+43
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 375121
* Revert r375116: "[lit] Extend internal diff to support `-` argument"Joel E. Denny2019-10-176-135/+2
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 375120
* [lit] Extend internal diff to support `-` argumentJoel E. Denny2019-10-176-2/+135
| | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-` as a command-line option. This patch adds support for `-` to mean stdin. Reviewed By: probinson, rnk Differential Revision: https://reviews.llvm.org/D67643 llvm-svn: 375116
* [lit] Make internal diff work in pipelinesJoel E. Denny2019-10-177-43/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - # RUN: not diff file1 file2 | FileCheck %s ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` cannot currently be used in pipelines and doesn't recognize `-` as a command-line option. To enable pipelines, this patch moves lit's `diff` implementation into an out-of-process script, similar to lit's `cat` implementation. A follow-up patch will implement `-` to mean stdin. Reviewed By: probinson, stella.stamenova Differential Revision: https://reviews.llvm.org/D66574 llvm-svn: 375114
* [lit] Remove unnecessary usage of lit.RunJulian Lettner2019-10-161-4/+3
| | | | llvm-svn: 375056
* [lit] Fix internal diff's --strip-trailing-cr and use itJoel E. Denny2019-10-165-2/+72
| | | | | | | | | | | | | | | | | | | | | | | | | Using GNU diff, `--strip-trailing-cr` removes a `\r` appearing before a `\n` at the end of a line. Without this patch, lit's internal diff only removes `\r` if it appears as the last character. That seems useless. This patch fixes that. This patch also adds `--strip-trailing-cr` to some tests that fail on Windows bots when D68664 is applied. Based on what I see in the bot logs, I think the following is happening. In each test there, lit diff is comparing a file with `\r\n` line endings to a file with `\n` line endings. Without D68664, lit diff reads those files in text mode, which in Windows causes `\r\n` to be replaced with `\n`. However, with D68664, lit diff reads the files in binary mode instead and thus reports that every line is different, just as GNU diff does (at least under Ubuntu). Adding `--strip-trailing-cr` to those tests restores the previous behavior while permitting the behavior of lit diff to be more like GNU diff. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68839 llvm-svn: 375020
* [lit] Clean up internal diff's encoding handlingJoel E. Denny2019-10-166-2/+66
| | | | | | | | | | | | | | | | | | | | | As suggested by rnk at D67643#1673043, instead of reading files multiple times until an appropriate encoding is found, read them once as binary, and then try to decode what was read. For Python >= 3.5, don't fail when attempting to decode the `diff_bytes` output in order to print it. Avoid failures for Python 2.7 used on some Windows bots by transforming diff output with `lit.util.to_string` before writing it to stdout. Finally, add some tests for encoding handling. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68664 llvm-svn: 375018
* [lit] Add back LitTestCaseJulian Lettner2019-10-154-0/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This essentially reverts a commit [1] that removed the adaptor for Python unittests. The code has been slightly refactored to make it more additive: all code is contained in LitTestCase.py. Usage sites will require a small adaption: ``` [old] import lit.discovery ... test_suite = lit.discovery.load_test_suite(...) [new] import lit.LitTestCase ... test_suite = lit.LitTestCase.load_test_suite(...) ``` This was put back on request by Daniel Dunbar, since I wrongly assumed that the functionality is unused. At least llbuild still uses this [2]. [1] 70ca752ccf6a8f362aea25ccd3ee2bbceca93b20 [2] https://github.com/apple/swift-llbuild/blob/master/utils/Xcode/LitXCTestAdaptor/LitTests.py#L16 Reviewed By: ddunbar Differential Revision: https://reviews.llvm.org/D69002 llvm-svn: 374947
* [lit] Extend internal diff to support -UJoel E. Denny2019-10-145-2/+122
| | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff -U1 file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-U` as a command-line option. This patch adds `-U` support. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68668 llvm-svn: 374814
* Revert r374648: "Reland r374388: [lit] Make internal diff work in pipelines"Joel E. Denny2019-10-123-47/+18
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 374683
* Revert r374649: "Reland r374389: [lit] Clean up internal diff's encoding ↵Joel E. Denny2019-10-126-66/+2
| | | | | | | | handling" This series of patches still breaks a Windows bot. llvm-svn: 374682
* Revert r374650: "Reland r374390: [lit] Extend internal diff to support `-` ↵Joel E. Denny2019-10-126-135/+2
| | | | | | | | argument" This series of patches still breaks a Windows bot. llvm-svn: 374681
* Revert 374651: "Reland r374392: [lit] Extend internal diff to support -U"Joel E. Denny2019-10-123-116/+2
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 374680
* Revert r374652: "[lit] Fix internal diff's --strip-trailing-cr and use it"Joel E. Denny2019-10-125-72/+3
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 374679
* Revert r374653: "[lit] Fix a few oversights in r374651 that broke some bots"Joel E. Denny2019-10-121-1/+1
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 374678
* Revert r374657: "[lit] Try again to fix new tests that fail on Windows bots"Joel E. Denny2019-10-121-6/+6
| | | | llvm-svn: 374664
* [lit] Try again to fix new tests that fail on Windows botsJoel E. Denny2019-10-121-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Based on the bot logs, when lit's internal diff runs on Windows, it looks like binary diffs must be decoded also for Python 2.7. Otherwise, writing the diff to stdout fails with: ``` UnicodeEncodeError: 'ascii' codec can't encode characters in position 7-8: ordinal not in range(128) ``` I did not need to decode using Python 2.7.15 under Ubuntu. When I do it anyway in that case, `errors="backslashreplace"` fails for me: ``` TypeError: don't know how to handle UnicodeDecodeError in error callback ``` However, `errors="ignore"` works, so this patch uses that, hoping it'll work on Windows as well. This patch leaves `errors="backslashreplace"` for Python >= 3.5 as there's no evidence yet that doesn't work and it produces more informative binary diffs. This patch also adjusts some lit tests to succeed for either error handler. This patch adjusts changes introduced by D68664. llvm-svn: 374657
* Revert r374654: "[lit] Try to fix new tests that fail on Windows bots"Joel E. Denny2019-10-121-20/+20
| | | | llvm-svn: 374656
* [lit] Try to fix new tests that fail on Windows botsJoel E. Denny2019-10-121-20/+20
| | | | llvm-svn: 374654
* [lit] Fix a few oversights in r374651 that broke some botsJoel E. Denny2019-10-121-1/+1
| | | | llvm-svn: 374653
* [lit] Fix internal diff's --strip-trailing-cr and use itJoel E. Denny2019-10-125-3/+72
| | | | | | | | | | | | | | | | | | | | | | | | | Using GNU diff, `--strip-trailing-cr` removes a `\r` appearing before a `\n` at the end of a line. Without this patch, lit's internal diff only removes `\r` if it appears as the last character. That seems useless. This patch fixes that. This patch also adds `--strip-trailing-cr` to some tests that fail on Windows bots when D68664 is applied. Based on what I see in the bot logs, I think the following is happening. In each test there, lit diff is comparing a file with `\r\n` line endings to a file with `\n` line endings. Without D68664, lit diff reads those files with Python's universal newlines support activated, causing `\r` to be dropped. However, with D68664, lit diff reads the files in binary mode instead and thus reports that every line is different, just as GNU diff does (at least under Ubuntu). Adding `--strip-trailing-cr` to those tests restores the previous behavior while permitting the behavior of lit diff to be more like GNU diff. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68839 llvm-svn: 374652
* Reland r374392: [lit] Extend internal diff to support -UJoel E. Denny2019-10-123-2/+116
| | | | | | | | | | To avoid breaking some tests, D66574, D68664, D67643, and D68668 landed together. However, D68664 introduced an issue now addressed by D68839, with which these are now all relanding. Differential Revision: https://reviews.llvm.org/D68668 llvm-svn: 374651
* Reland r374390: [lit] Extend internal diff to support `-` argumentJoel E. Denny2019-10-126-2/+135
| | | | | | | | | | To avoid breaking some tests, D66574, D68664, D67643, and D68668 landed together. However, D68664 introduced an issue now addressed by D68839, with which these are now all relanding. Differential Revision: https://reviews.llvm.org/D67643 llvm-svn: 374650
* Reland r374389: [lit] Clean up internal diff's encoding handlingJoel E. Denny2019-10-126-2/+66
| | | | | | | | | | To avoid breaking some tests, D66574, D68664, D67643, and D68668 landed together. However, D68664 introduced an issue now addressed by D68839, with which these are now all relanding. Differential Revision: https://reviews.llvm.org/D68664 llvm-svn: 374649
* Reland r374388: [lit] Make internal diff work in pipelinesJoel E. Denny2019-10-123-18/+47
| | | | | | | | | | To avoid breaking some tests, D66574, D68664, D67643, and D68668 landed together. However, D68664 introduced an issue now addressed by D68839, with which these are now all relanding. Differential Revision: https://reviews.llvm.org/D66574 llvm-svn: 374648
* [lit] Change regex filter to ignore caseJulian Lettner2019-10-111-7/+3
| | | | | | | | | | | Make regex filter `--filter=REGEX` option more lenient via `re.IGNORECASE`. Reviewed By: yln Differential Revision: https://reviews.llvm.org/D68834 llvm-svn: 374601
* Revert r374388: "[lit] Make internal diff work in pipelines"Joel E. Denny2019-10-103-47/+18
| | | | | | This breaks a Windows bot. llvm-svn: 374429
* Revert r374389: "[lit] Clean up internal diff's encoding handling"Joel E. Denny2019-10-106-66/+2
| | | | | | This breaks a Windows bot. llvm-svn: 374427
* Revert r374390: "[lit] Extend internal diff to support `-` argument"Joel E. Denny2019-10-106-135/+2
| | | | | | This breaks a Windows bot. llvm-svn: 374426
* Revert r374392: "[lit] Extend internal diff to support -U"Joel E. Denny2019-10-103-116/+2
| | | | | | This breaks a Windows bot. llvm-svn: 374425
* [lit] Leverage argparse features to remove some codeJulian Lettner2019-10-102-2/+2
| | | | | | | | Reviewed By: rnk, serge-sans-paille Differential Revision: https://reviews.llvm.org/D68589 llvm-svn: 374405
* [lit] Extend internal diff to support -UJoel E. Denny2019-10-103-2/+116
| | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff -U1 file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-U` as a command-line option. This patch adds `-U` support. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68668 llvm-svn: 374392
* [lit] Extend internal diff to support `-` argumentJoel E. Denny2019-10-106-2/+135
| | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-` as a command-line option. This patch adds support for `-` to mean stdin. Reviewed By: probinson, rnk Differential Revision: https://reviews.llvm.org/D67643 llvm-svn: 374390
OpenPOWER on IntegriCloud