summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit/tests/shtest-format.py
Commit message (Collapse)AuthorAgeFilesLines
* Fix buildbot failures after removing REQUIRES-ANYNemanja Ivanovic2019-12-171-10/+3
| | | | | | | It would appear that the removal of this lit feature was incomplete and there is a test case that still tests for this. This patch removes the remaining tests to bring the bots back to green. I would encourage the author to do a post-commit review on this in case there is a more desirable fix.
* [lit] Fix UnicodeEncodeError when test commands contain non-ASCII charsMichal Gorny2019-06-141-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | Ensure that the bash script written by lit TestRunner is open with UTF-8 encoding when using Python 3. Otherwise, attempt to write non-ASCII characters causes UnicodeEncodeError. This happened e.g. with the following LLD test: UNRESOLVED: lld :: ELF/format-binary-non-ascii.s (657 of 2119) ******************** TEST 'lld :: ELF/format-binary-non-ascii.s' FAILED ******************** Exception during script execution: Traceback (most recent call last): File "/home/mgorny/llvm-project/llvm/utils/lit/lit/worker.py", line 63, in _execute_test result = test.config.test_format.execute(test, lit_config) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/formats/shtest.py", line 25, in execute self.execute_external) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1644, in executeShTest res = _runShTest(test, litConfig, useExternalSh, script, tmpBase) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1590, in _runShTest res = executeScript(test, litConfig, tmpBase, script, execdir) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1157, in executeScript f.write('{ ' + '; } &&\n{ '.join(commands) + '; }') UnicodeEncodeError: 'ascii' codec can't encode character '\xa3' in position 274: ordinal not in range(128) Differential Revision: https://reviews.llvm.org/D63254 llvm-svn: 363388
* [lit][tests][AIX] Update expected form of diagnostic messages; use `not` to ↵Hubert Tong2019-05-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | normalize non-zero exit values Summary: Various tests in the `lit` testing suite expect specific return codes and forms of diagnostic message from utility programs. As per POSIX.1-2017 XCU Section 1.4, Utility Description Defaults, "[the] format of diagnostic messages for most utilities is unspecified". The STDERR subsections of the `cat` and `wc` utilities merely indicate that "[the] standard error shall be used only for diagnostic messages". The corresponding EXIT STATUS subsections merely indicate, with regard to errors, an exit value of >0. The affected tests are updated to accept the applicable diagnostic message as produced by the utilities on AIX. The exit value is normalized using `not` as necessary. Reviewers: xingxue, sfertile, jasonliu Reviewed By: xingxue Subscribers: delcypher, jsji, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D60553 llvm-svn: 359690
* [lit] Move the shtest-xunit-output check lines into shtest-formatJustin Bogner2018-07-231-2/+86
| | | | | | | | These two tests are operating on the same test suite, which causes them to be racy about writing temporary files and can cause spurious failures. Merge them into one test to avoid the issue. llvm-svn: 337718
* [lit] Report line number for failed RUN commandJoel E. Denny2018-05-311-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Relands r333584, reverted in 333592.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333614
* Revert r333584: [lit] Report line number for failed RUN commandJoel E. Denny2018-05-301-1/+0
| | | | | | It breaks test-suite. llvm-svn: 333592
* [lit] Report line number for failed RUN commandJoel E. Denny2018-05-301-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Relands r330755 (reverted in r330848) with fix for PR37239.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333584
* Revert r330755 "[lit] Report line number for failed RUN command"Reid Kleckner2018-04-251-1/+0
| | | | | | | It is causing many tests to fail on Windows buildbots: http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211 llvm-svn: 330848
* [lit] Report line number for failed RUN commandJoel E. Denny2018-04-241-0/+1
| | | | | | | | | | | | | | | | | | | | | When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. Reviewed By: asmith, delcypher Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 330755
* Make check-lit tests respect LLVM_LIT_TOOLS_DIRGreg Bedwell2017-11-291-1/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D40520 llvm-svn: 319329
* Make lit :: shtest-format.py supported on Windows againVictor Leschuk2017-08-231-1/+0
| | | | | | | | | | | | It was marked as unsupported on Windows in r311230 because on some Win10 machines it failed or caused hang. The problem was that on these machines system bash (C:\Windows\System32\bash.exe) was used which requires paths to be passed like '/mnt/c/path/to/my/script' instead of 'C:\path\to\my\script'. TODO: we should make lit detect if system bash is used instead of msys and set appropriate path format. llvm-svn: 311558
* Temporary mark lit :: shtest-format as unsupported on windowsVictor Leschuk2017-08-191-0/+1
| | | | | | When run manually it fails, but when run under buildbot it causes hang. llvm-svn: 311230
* Un-XFAIL some internal lit tests on Windows, they pass for me locallyReid Kleckner2017-07-261-3/+0
| | | | llvm-svn: 309144
* [lit] Mark several of lit's tests XFAIL on WindowsBrian Gesiak2017-07-261-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: rL257221 attempted to run lit's own test suite continuously, but that commit was reverted because lit's test suite does not pass on Windows. Because lit's tests do not run continuously, they often regress. In order to un-revert rL257221, mark lit tests that fail as XFAIL for Windows platforms. Test Plan: On a Windows development environment, follow the instructions in utils/lit/README.txt to run lit's test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test suite is run and a successful exit code is returned. Reviewers: mgorny, rnk, delcypher, beanz Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35879 llvm-svn: 309123
* Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILGreg Parker2017-01-251-4/+11
| | | | | | | | and UNSUPPORTED" This reverts the revert in r292942. llvm-svn: 293007
* Revert "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILAlex Lorenz2017-01-241-11/+4
| | | | | | | | | | | and UNSUPPORTED" After r292904 llvm-lit fails to emit the test results in the XML format for Apple's internal buildbots. rdar://30164800 llvm-svn: 292942
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-241-4/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292904
* Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED"Greg Parker2017-01-241-13/+6
| | | | | | This change needs to be better-coordinated with libc++. llvm-svn: 292900
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-241-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292896
* [lit] Improve readability of failing scripts.Daniel Dunbar2016-06-021-3/+2
| | | | | | | | | | | | | | | | | | | | | | | - This only applies to scripts executed by the _internal_ shell script interpreter. - This patch reworks the log to look more like a shell transcript, and be less verbose (but in the interest of calling attention to the important parts). Here is an example of the new format, for commands with/without failures and with/without output: ``` $ true $ echo hi hi $ false note: command had no output on stdout or stderr error: command failed with exit status 1 ``` llvm-svn: 271610
* Add "REQUIRES-ANY" feature testEric Fiselier2016-06-021-2/+4
| | | | | | | | | | | | | | | | Summary: This patch adds a "REQUIRES-ANY" feature test that is disjunctive. This marks a test as `UNSUPPORTED` if none of the specified features are available. Libc++ has the need to write feature test such as `// REQUIRES-ANY: c++98, c++03` when testing of behavior that is specific to older dialects but has since changed. Reviewers: rnk, ddunbar Subscribers: ddunbar, probinson, llvm-commits, cfe-commits Differential Revision: http://reviews.llvm.org/D20757 llvm-svn: 271468
* [lit] Fix internal shell's argv[0] handling.Daniel Dunbar2013-08-291-1/+2
| | | | | | | - At least on OS X, it is important for correct behavior of /bin/[ that argv[0] is passed as written, and not as the full executable path. llvm-svn: 189559
* [lit] Lift XFAIL handling to core infrastructure.Daniel Dunbar2013-08-211-0/+5
| | | | llvm-svn: 188949
* [lit] Ensure test output is converted to strings where possible.Daniel Dunbar2013-08-141-4/+32
| | | | | | - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
* [lit] Add a test for the various ShTest format features.Daniel Dunbar2013-01-311-0/+43
llvm-svn: 174072
OpenPOWER on IntegriCloud