summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit/tests/max-failures.py
Commit message (Collapse)AuthorAgeFilesLines
* [lit] Fix internal diff newlines for -w/-bJoel E. Denny2019-12-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For example, without this patch: ``` $ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt *** /tmp/foo.txt --- /tmp/bar.txt *************** *** 1,2 **** 1! 2--- 1,2 ---- 1! 20 ``` With this patch: ``` $ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt *** /tmp/foo.txt --- /tmp/bar.txt *************** *** 1,2 **** 1 ! 2 --- 1,2 ---- 1 ! 20 ``` Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D71577
* [lit] Extend internal diff to support `-` argumentJoel E. Denny2019-10-291-1/+1
| | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-` as a command-line option. This patch adds support for `-` to mean stdin. Reviewed By: probinson, rnk Differential Revision: https://reviews.llvm.org/D67643
* [lit] Make internal diff work in pipelinesJoel E. Denny2019-10-291-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - # RUN: not diff file1 file2 | FileCheck %s ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` cannot currently be used in pipelines and doesn't recognize `-` as a command-line option. To enable pipelines, this patch moves lit's `diff` implementation into an out-of-process script, similar to lit's `cat` implementation. A follow-up patch will implement `-` to mean stdin. Also, when lit's `diff` prints differences to stdout in Windows, this patch ensures it always terminate lines with `\n` not `\r\n`. That way, strict FileCheck directives checking the `diff` output succeed in both Linux and Windows. This wasn't an issue when `diff` was internal to lit because `diff` didn't then write to the true stdout, which is where the `\n` -> `\r\n` conversion happened in Python. Reviewed By: probinson, stella.stamenova Differential Revision: https://reviews.llvm.org/D66574
* [lit] Don't fail when printing test output with special charsJoel E. Denny2019-10-251-1/+1
| | | | | | | | | This addresses a UnicodeEncodeError when using Python 3.6.5 in Windows 10. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D69207
* Revert r375114: "[lit] Make internal diff work in pipelines"Joel E. Denny2019-10-171-1/+1
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 375121
* Revert r375116: "[lit] Extend internal diff to support `-` argument"Joel E. Denny2019-10-171-1/+1
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 375120
* [lit] Extend internal diff to support `-` argumentJoel E. Denny2019-10-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-` as a command-line option. This patch adds support for `-` to mean stdin. Reviewed By: probinson, rnk Differential Revision: https://reviews.llvm.org/D67643 llvm-svn: 375116
* [lit] Make internal diff work in pipelinesJoel E. Denny2019-10-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - # RUN: not diff file1 file2 | FileCheck %s ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` cannot currently be used in pipelines and doesn't recognize `-` as a command-line option. To enable pipelines, this patch moves lit's `diff` implementation into an out-of-process script, similar to lit's `cat` implementation. A follow-up patch will implement `-` to mean stdin. Reviewed By: probinson, stella.stamenova Differential Revision: https://reviews.llvm.org/D66574 llvm-svn: 375114
* [lit] Fix internal diff's --strip-trailing-cr and use itJoel E. Denny2019-10-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | Using GNU diff, `--strip-trailing-cr` removes a `\r` appearing before a `\n` at the end of a line. Without this patch, lit's internal diff only removes `\r` if it appears as the last character. That seems useless. This patch fixes that. This patch also adds `--strip-trailing-cr` to some tests that fail on Windows bots when D68664 is applied. Based on what I see in the bot logs, I think the following is happening. In each test there, lit diff is comparing a file with `\r\n` line endings to a file with `\n` line endings. Without D68664, lit diff reads those files in text mode, which in Windows causes `\r\n` to be replaced with `\n`. However, with D68664, lit diff reads the files in binary mode instead and thus reports that every line is different, just as GNU diff does (at least under Ubuntu). Adding `--strip-trailing-cr` to those tests restores the previous behavior while permitting the behavior of lit diff to be more like GNU diff. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68839 llvm-svn: 375020
* [lit] Clean up internal diff's encoding handlingJoel E. Denny2019-10-161-1/+1
| | | | | | | | | | | | | | | | | | | | | As suggested by rnk at D67643#1673043, instead of reading files multiple times until an appropriate encoding is found, read them once as binary, and then try to decode what was read. For Python >= 3.5, don't fail when attempting to decode the `diff_bytes` output in order to print it. Avoid failures for Python 2.7 used on some Windows bots by transforming diff output with `lit.util.to_string` before writing it to stdout. Finally, add some tests for encoding handling. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68664 llvm-svn: 375018
* [lit] Extend internal diff to support -UJoel E. Denny2019-10-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff -U1 file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-U` as a command-line option. This patch adds `-U` support. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68668 llvm-svn: 374814
* Revert r374649: "Reland r374389: [lit] Clean up internal diff's encoding ↵Joel E. Denny2019-10-121-1/+1
| | | | | | | | handling" This series of patches still breaks a Windows bot. llvm-svn: 374682
* Revert r374650: "Reland r374390: [lit] Extend internal diff to support `-` ↵Joel E. Denny2019-10-121-1/+1
| | | | | | | | argument" This series of patches still breaks a Windows bot. llvm-svn: 374681
* Revert 374651: "Reland r374392: [lit] Extend internal diff to support -U"Joel E. Denny2019-10-121-1/+1
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 374680
* Revert r374652: "[lit] Fix internal diff's --strip-trailing-cr and use it"Joel E. Denny2019-10-121-1/+1
| | | | | | This series of patches still breaks a Windows bot. llvm-svn: 374679
* [lit] Fix internal diff's --strip-trailing-cr and use itJoel E. Denny2019-10-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | Using GNU diff, `--strip-trailing-cr` removes a `\r` appearing before a `\n` at the end of a line. Without this patch, lit's internal diff only removes `\r` if it appears as the last character. That seems useless. This patch fixes that. This patch also adds `--strip-trailing-cr` to some tests that fail on Windows bots when D68664 is applied. Based on what I see in the bot logs, I think the following is happening. In each test there, lit diff is comparing a file with `\r\n` line endings to a file with `\n` line endings. Without D68664, lit diff reads those files with Python's universal newlines support activated, causing `\r` to be dropped. However, with D68664, lit diff reads the files in binary mode instead and thus reports that every line is different, just as GNU diff does (at least under Ubuntu). Adding `--strip-trailing-cr` to those tests restores the previous behavior while permitting the behavior of lit diff to be more like GNU diff. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68839 llvm-svn: 374652
* Reland r374392: [lit] Extend internal diff to support -UJoel E. Denny2019-10-121-1/+1
| | | | | | | | | | To avoid breaking some tests, D66574, D68664, D67643, and D68668 landed together. However, D68664 introduced an issue now addressed by D68839, with which these are now all relanding. Differential Revision: https://reviews.llvm.org/D68668 llvm-svn: 374651
* Reland r374390: [lit] Extend internal diff to support `-` argumentJoel E. Denny2019-10-121-1/+1
| | | | | | | | | | To avoid breaking some tests, D66574, D68664, D67643, and D68668 landed together. However, D68664 introduced an issue now addressed by D68839, with which these are now all relanding. Differential Revision: https://reviews.llvm.org/D67643 llvm-svn: 374650
* Reland r374389: [lit] Clean up internal diff's encoding handlingJoel E. Denny2019-10-121-1/+1
| | | | | | | | | | To avoid breaking some tests, D66574, D68664, D67643, and D68668 landed together. However, D68664 introduced an issue now addressed by D68839, with which these are now all relanding. Differential Revision: https://reviews.llvm.org/D68664 llvm-svn: 374649
* Revert r374389: "[lit] Clean up internal diff's encoding handling"Joel E. Denny2019-10-101-1/+1
| | | | | | This breaks a Windows bot. llvm-svn: 374427
* Revert r374390: "[lit] Extend internal diff to support `-` argument"Joel E. Denny2019-10-101-1/+1
| | | | | | This breaks a Windows bot. llvm-svn: 374426
* Revert r374392: "[lit] Extend internal diff to support -U"Joel E. Denny2019-10-101-1/+1
| | | | | | This breaks a Windows bot. llvm-svn: 374425
* [lit] Leverage argparse features to remove some codeJulian Lettner2019-10-101-1/+1
| | | | | | | | Reviewed By: rnk, serge-sans-paille Differential Revision: https://reviews.llvm.org/D68589 llvm-svn: 374405
* [lit] Extend internal diff to support -UJoel E. Denny2019-10-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff -U1 file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-U` as a command-line option. This patch adds `-U` support. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68668 llvm-svn: 374392
* [lit] Extend internal diff to support `-` argumentJoel E. Denny2019-10-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | When using lit's internal shell, RUN lines like the following accidentally execute an external `diff` instead of lit's internal `diff`: ``` # RUN: program | diff file - ``` Such cases exist now, in `clang/test/Analysis` for example. We are preparing patches to ensure lit's internal `diff` is called in such cases, which will then fail because lit's internal `diff` doesn't recognize `-` as a command-line option. This patch adds support for `-` to mean stdin. Reviewed By: probinson, rnk Differential Revision: https://reviews.llvm.org/D67643 llvm-svn: 374390
* [lit] Clean up internal diff's encoding handlingJoel E. Denny2019-10-101-1/+1
| | | | | | | | | | | | | | | | | As suggested by rnk at D67643#1673043, instead of reading files multiple times until an appropriate encoding is found, read them once as binary, and then try to decode what was read. For python >= 3.5, don't fail when attempting to decode the `diff_bytes` output in order to print it. Finally, add some tests for encoding handling. Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D68664 llvm-svn: 374389
* [lit] Set --single-process for single tests and --threads=1Julian Lettner2019-02-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: Automatically upgrade debugging experience (single process, no thread pool) when: 1) we only run a single test 2) user specifies `-j1` Details: Fix `--max-failures` in single process mode. Option did not have an effect in single process mode. Add display feedback for single process mode. Adapted test. Improve argument checking (require positive integers). `--single-process` is now essentially an alias for `-j1`. Should we remove it? Reviewers: rnk Differential Revision: https://reviews.llvm.org/D58249 llvm-svn: 354068
* [lit] Report line number for failed RUN commandJoel E. Denny2018-05-311-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Relands r333584, reverted in 333592.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333614
* Revert r333584: [lit] Report line number for failed RUN commandJoel E. Denny2018-05-301-1/+1
| | | | | | It breaks test-suite. llvm-svn: 333592
* [lit] Report line number for failed RUN commandJoel E. Denny2018-05-301-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Relands r330755 (reverted in r330848) with fix for PR37239.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333584
* Revert r330755 "[lit] Report line number for failed RUN command"Reid Kleckner2018-04-251-1/+1
| | | | | | | It is causing many tests to fail on Windows buildbots: http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211 llvm-svn: 330848
* [lit] Report line number for failed RUN commandJoel E. Denny2018-04-241-1/+1
| | | | | | | | | | | | | | | | | | | | | When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. Reviewed By: asmith, delcypher Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 330755
* [lit] Implement 'cat' command for internal shellReid Kleckner2018-03-261-1/+1
| | | | | | | | | | Fixes PR36449 Patch by Chamal de Silva Differential Revision: https://reviews.llvm.org/D43501 llvm-svn: 328563
* [lit] Implement "-r" option for builtin "diff" command + a test using that.Max Moroz2018-01-091-1/+1
| | | | | | | | | | | | | | | | Summary: That would allow to recursively compare directories in tests using "diff -r" on Windows in a similar way as it can be done on Linux or Mac. Reviewers: zturner, morehouse, vsk Reviewed By: zturner Subscribers: kcc, llvm-commits Differential Revision: https://reviews.llvm.org/D41776 llvm-svn: 322102
* [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands ↵Ying Yi2017-12-011-1/+1
| | | | | | | | | | | | | | | | | internally Summary: The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands as the internal shell builtins. Reviewed by: Zachary Turner, Reid Kleckner Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39567 llvm-svn: 319528
* Reverted rL318911 since it broke the sanitizer-windows.Ying Yi2017-11-231-1/+1
| | | | llvm-svn: 318914
* [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands ↵Ying Yi2017-11-231-1/+1
| | | | | | | | | | | | | | | | | internally Summary: The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands as the internal shell builtins. Reviewers: Zachary Turner, Reid Kleckner Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39567 llvm-svn: 318911
* [lit] Fix race between shtest-shell and max-failures testsReid Kleckner2017-07-261-4/+4
| | | | | | | Previously these tests would use the same Output directory leading to flaky non-deterministic failures. llvm-svn: 309227
* [lit] Fix shtest-shell and max-failures lit tests on WindowsReid Kleckner2017-07-261-3/+0
| | | | | | | | | Rewrite the write-to-stderr.sh and write-to-stdout-and-stderr.sh shell scripts as python scripts and call python on them. Fixes PR33940 llvm-svn: 309200
* [lit] Mark several of lit's tests XFAIL on WindowsBrian Gesiak2017-07-261-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: rL257221 attempted to run lit's own test suite continuously, but that commit was reverted because lit's test suite does not pass on Windows. Because lit's tests do not run continuously, they often regress. In order to un-revert rL257221, mark lit tests that fail as XFAIL for Windows platforms. Test Plan: On a Windows development environment, follow the instructions in utils/lit/README.txt to run lit's test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test suite is run and a successful exit code is returned. Reviewers: mgorny, rnk, delcypher, beanz Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35879 llvm-svn: 309123
* [lit] Add a --max-failures option.Daniel Dunbar2016-09-261-0/+14
- This is primarily useful as a "fail fast" mode for lit, where it will stop running tests after the first failure. - Patch by Max Moiseev. llvm-svn: 282452
OpenPOWER on IntegriCloud