summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit/tests
Commit message (Collapse)AuthorAgeFilesLines
* Add Support for Creating and Deleting Unicode Files and Directories in LitSerge Guelton2019-02-282-0/+8
| | | | | | | | | | | This enables lit to work with unicode file names via mkdir, rm, and redirection. Lit still uses utf-8 internally, but converts to utf-16 on Windows, or just utf-8 bytes on everything else. Committed on behalf of Jason Mittertreiner Differential Revision: https://reviews.llvm.org/D56754 llvm-svn: 355122
* Fixed typos in a test: s/CEHCK/CHECK/Dmitri Gribenko2019-02-281-7/+7
| | | | | | | | | | | | Reviewers: ilya-biryukov, serge-sans-paille Subscribers: delcypher, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D58613 llvm-svn: 355080
* [lit] Allow setting parallelism groups to NoneJulian Lettner2019-02-264-0/+34
| | | | | | | | | | | | | | | | | | | Check that we do not crash if a parallelism group is explicitly set to None. Permits usage of the following pattern. [lit.common.cfg] lit_config.parallelism_groups['my_group'] = None if <condition>: lit_config.parallelism_groups['my_group'] = 3 [project/lit.cfg] config.parallelism_group = 'my_group' Reviewers: rnk Differential Revision: https://reviews.llvm.org/D58305 llvm-svn: 354912
* [lit] Remove LitTestCaseJulian Lettner2019-02-164-26/+0
| | | | | | | | | | | | | From the docs: `class LitTestCase(unittest.TestCase)` LitTestCase is an adaptor for providing a 'unittest' compatible interface to 'lit' tests so that we can run lit tests with standard python test runners. It does not seem to be used anywhere. Differential Revision: https://reviews.llvm.org/D58264 llvm-svn: 354188
* [lit][NFC] Cleanup copy&paste naming mistakeJulian Lettner2019-02-152-5/+5
| | | | llvm-svn: 354095
* [lit] Remove --single-process option (use -j1 instead)Julian Lettner2019-02-141-1/+1
| | | | | | | | Remove `--single-process` command line option. Use `-j1` instead. Also see commit: 96adb78b120b6aa9739eb714534dc8e819f7bc52 llvm-svn: 354073
* [lit] Set --single-process for single tests and --threads=1Julian Lettner2019-02-142-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: Automatically upgrade debugging experience (single process, no thread pool) when: 1) we only run a single test 2) user specifies `-j1` Details: Fix `--max-failures` in single process mode. Option did not have an effect in single process mode. Add display feedback for single process mode. Adapted test. Improve argument checking (require positive integers). `--single-process` is now essentially an alias for `-j1`. Should we remove it? Reviewers: rnk Differential Revision: https://reviews.llvm.org/D58249 llvm-svn: 354068
* make XFAIL, REQUIRES, and UNSUPPORTED support multi-line expressionsEric Fiselier2019-01-202-1/+29
| | | | llvm-svn: 351668
* [lit] Make it possible for the lit test suite to pass withDan Liew2019-01-101-1/+1
| | | | | | | | | `FILECHECK_OPTS=-v` set in the environment. Follow up to r350850 as requested by Joel E. Denny in https://reviews.llvm.org/D56541 . llvm-svn: 350854
* [lit] Respect PYTHONPATHDavid Greene2019-01-072-2/+18
| | | | | | | | | | | If a user has PYTHONPATH set in the environment, append new entries to it rather than blindly setting PYTHONPATH to a fixed string. This allows tests to, for example, find psutil if it is in PYTHONPATH. Without this change, lit will detect psutil but then various tests will fail because PYTHONPATH has been overwritten and psutil cannot be found. llvm-svn: 350536
* Python compat - print statementSerge Guelton2019-01-032-1/+4
| | | | | | | | | Make sure all print statements are compatible with Python 2 and Python3 using the `from __future__ import print_function` statement. Differential Revision: https://reviews.llvm.org/D56249 llvm-svn: 350307
* [lit] Disable shtest-timeout on WindowsStella Stamenova2018-09-102-5/+3
| | | | | | | | | | | | Summary: This is the only test that is still failing on Windows - or rather, it is expected to fail on the bots, but passes on the new bot that we're preparing causing a failure, so I'm going to disable it. Since the test has rarely, if ever, passed on the bots, this should have the same effect and it will unblock the creation of the new bot. Reviewers: asmith, delcypher, zturner Subscribers: stella.stamenova, llvm-commits Differential Revision: https://reviews.llvm.org/D51871 llvm-svn: 341856
* [lit, shtest-timeout] Always use an internal shell for the shtest-timeout to ↵Stella Stamenova2018-08-281-1/+3
| | | | | | | | | | | | | | | | | | | diagnose buildbot failures Summary: Right now this test is failing on the builtbots on Windows but we have a very similar setup where the test passes. The test is meant to test that specifying a timeout works correctly by running an infnite loop and having it timeout - on the buildbot, the infinite loop doesn't actually execute. This change runs all of the tests in the set using an internal shell rather than an external shell. I expect this will make the test pass which means that either the way the external shell is invoked or the external shell setup on the buildbots is not correct. Regardless of whether the test passes with this change, we'll need to undo this change and have a real fix. @gkistanova was able to get logs from the buildbot to rule out a number of theories as to why this test is failing, but they didn't have enough information to confirm exactly what the issue is. The purpose of this change is to narrow it down, but if someone has a local repro and can aid in debugging, that would make it much speedier (and less prone to making the bots fail). Reviewers: gkistanova, asmith, zturner, modocache, rnk, delcypher Reviewed By: rnk Subscribers: delcypher, llvm-commits, gkistanova Differential Revision: https://reviews.llvm.org/D51326 llvm-svn: 340840
* [lit] Disable shtest-timeout on WindowsStella Stamenova2018-08-071-0/+3
| | | | | | This test passes on Windows when using Python 3 but fails when using Python 2, so it needs more investigation before it can be enabled as the bots use Python 2. llvm-svn: 339184
* [lit, python3] Update lit error logging to work correctly in python3 and ↵Stella Stamenova2018-08-075-7/+4
| | | | | | | | | | | | | | | | | other test fixes Summary: In Python2 'unicode' is a distinct type from 'str', but in Python3 'unicode' does not exist and instead all 'str' objects are Unicode string. This change updates the logic in the test logging for lit to correctly process each of the types, and more importantly, to not just fail in Python3. This change also reverses the use of quotes in several of the cfg files. By using '""' we are guaranteeing that the resulting path will work correctly on Windows while "''" only works correctly sometimes. This also fixes one of the failing tests. Reviewers: asmith, zturner Subscribers: stella.stamenova, delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D50397 llvm-svn: 339179
* [lit, tests] Fix failing lit test: shtest-format.pyStella Stamenova2018-08-071-1/+1
| | | | | | | | | | | | | | | Summary: The problem here is that on windows double quotes are used for paths (usually) while single quotes are not. This is not generally a problem for the tests because the lit infrastructure tends to treat both the same. One (and possibly only) exception is when some tests are run in an external shell such as some of the shtest-format tests. In this case on windows the path to python was not created correctly because it had single quotes and the test failed. This same test is already failing with python 3 which is why our testing missed the new failure. This patch will take care of the immediate failure with python 2 and I'll send a follow up for the python 3 failure. Reviewers: asmith, zturner Subscribers: delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D50373 llvm-svn: 339091
* [lit, python] Always add quotes around the python path in litStella Stamenova2018-08-069-27/+28
| | | | | | | | | | | | | | | | | Summary: The issue with the python path is that the path to python on Windows can contain spaces. To make the tests always work, the path to python needs to be surrounded by quotes. This change updates several configuration files which specify the path to python as a substitution and also remove quotes from existing tests. Reviewers: asmith, zturner, alexshap, jakehehrlich Reviewed By: zturner, alexshap, jakehehrlich Subscribers: mehdi_amini, nemanjai, eraman, kbarton, jakehehrlich, steven_wu, dexonsmith, stella.stamenova, delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D50206 llvm-svn: 339073
* [lit] Move the shtest-xunit-output check lines into shtest-formatJustin Bogner2018-07-232-86/+86
| | | | | | | | These two tests are operating on the same test suite, which causes them to be racy about writing temporary files and can cause spurious failures. Merge them into one test to avoid the issue. llvm-svn: 337718
* Relax shtest-run-at-line.pyHans Wennborg2018-06-062-18/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The test was failing on Windows machines which had bash.exe on PATH (but not in the so called lit tools dir, containing cmp.exe, grep.exe etc.). The problem was that the outer lit invocation would load LLVMConfig from utils/lit/lit/llvm/config.py, which looks up the tools path with getToolsPath(). That has a surprising side effect of also setting bashPath, in our case setting it to empty. The outer lit invocation would thus configure the pdbg0 and pdbg1 substitutions based on not running with bash. But the inner lit invocation would not load LLVMConfig, so bash would be found on PATH, that would be used as external shell, and so the output wouldn't match pdbg0 and pdbg1. It seems weird to me that getBashPath() will return different results depending on whether getToolsPath() has been called before, but I also don't know how to fix it properly. This commit just relaxes the test case, because there doesn't seem to be much point in testing for the exact syntax of the run file as long as it works. (See https://crbug.com/850023) llvm-svn: 334100
* [lit] Fix windows cmd.exe test config for r333620Joel E. Denny2018-05-312-8/+8
| | | | llvm-svn: 333630
* [lit] Terminate ": RUN at line N" with ";" not "&&"Joel E. Denny2018-05-313-16/+16
| | | | | | | | This fixes projects/compiler-rt/test/fuzzer/sigusr.test, which was broken by r333614. The trouble was that "&&" changes the command for which "$!" gives the pid. llvm-svn: 333620
* [lit] Report line number for failed RUN commandJoel E. Denny2018-05-3115-6/+140
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Relands r333584, reverted in 333592.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333614
* Revert r333584: [lit] Report line number for failed RUN commandJoel E. Denny2018-05-3015-140/+6
| | | | | | It breaks test-suite. llvm-svn: 333592
* [lit] Report line number for failed RUN commandJoel E. Denny2018-05-3015-6/+140
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Relands r330755 (reverted in r330848) with fix for PR37239.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333584
* Use quoteattr to ensure we make well formed attributesChris Matthews2018-05-252-26/+26
| | | | | | | | We were making malformed XML on tests with ' in the name. Switch to using saxutils to set all of our attributes, so it can handle quotes etc correctly. llvm-svn: 333249
* [lit] Try to make `shtest-timeout.py` test more reliable by using aDan Liew2018-05-222-6/+11
| | | | | | | | | | | | | larger timeout value. This really isn't very good because it will still be susceptible to machine performance. While we are here also fix a bug in validation of `maxIndividualTestTime` where previously it wasn't checked if the type was an int. rdar://problem/40221572 llvm-svn: 332987
* [lit] Don't run `slow.py` in `shtest-timeout.py` test.Dan Liew2018-05-222-27/+2
| | | | | | | | | | | The program used to be used in `quick_then_slow.py` but that was removed in r328702. The tests always run `slow.py` on its own but this doesn't really test additional code so we'll just drop running `slow.py` so the tests run faster. rdar://problem/40221572 llvm-svn: 332986
* [lit] Don't check output of commands used in `shtest-timeout.py` test.Dan Liew2018-05-221-6/+0
| | | | | | | | | | If the system is under heavy load 1 second might not be long enough for it to produce output which could lead to spurious test failures. What matters is that the right test cases reach a timeout. rdar://problem/40221572 llvm-svn: 332985
* Mark test with "REQUIRES: shell" since it directly invokes "sh" and was ↵Douglas Yung2018-05-171-0/+2
| | | | | | failing on Windows. llvm-svn: 332563
* Escape ]]> in xunit xml outputAlexander Richardson2018-05-162-3/+5
| | | | | | | | | | | | | | Summary: This sequence ends the CDATA block so any characters after that are no longer escaped. This can be fixed by replacing "]]>" with "]]]]><![CDATA[>". Reviewers: cmatthews Reviewed By: cmatthews Differential Revision: https://reviews.llvm.org/D46886 llvm-svn: 332440
* remove output xml incase it is leftover from another runChris Matthews2018-05-161-0/+1
| | | | llvm-svn: 332424
* Use not to catch unexpected pass as well as remove old test resultsChris Matthews2018-05-161-1/+2
| | | | | | | As per review feedback, make sure we rm temp files, and make the return code checking for lit more specific. llvm-svn: 332423
* Add the message attribute to skippedChris Matthews2018-05-111-5/+5
| | | | | | | JUnit xml allows for a message attribute to be displayed on skips. Lets populate that with an analysis of why we skipped the test. llvm-svn: 332156
* [LIT] replace output escapes wit a cdata blockChris Matthews2018-05-111-3/+1
| | | | | | | CDATA blocks don't need to have XML stuff escaped. Makes sense to wrap output in them instead of escaping. llvm-svn: 332116
* Support Unsupported Tests in xunit outputChris Matthews2018-05-112-7/+12
| | | | | | | | We were reporting "Unsupported" tests in xunit as passes, however since they are not run, it make more sense to mark them as skipped. The Junit xml standard has support for that, so lets use it. llvm-svn: 332065
* [LIT] Move xunit tests tests into their own location, and and add failuresChris Matthews2018-05-105-9/+59
| | | | | | Failures will increase coverage. llvm-svn: 332056
* [LIT] Add the missing fileChris Matthews2018-05-101-0/+7
| | | | | | I forgot to commit this file. llvm-svn: 331946
* Refactor test incase results are backwardsChris Matthews2018-05-101-14/+3
| | | | | | Looks like results can come in either way in this file. Loosen the ordering constraints. llvm-svn: 331945
* [LIT] Handle xml characters in test namesChris Matthews2018-05-092-1/+13
| | | | | | | | | | Lit creates malformed xml when the test case has an & in the name. Escape those correctly. This also adds a test case which I will add other nasty encoding issues to in some followup commits. llvm-svn: 331942
* Revert r330755 "[lit] Report line number for failed RUN command"Reid Kleckner2018-04-2514-131/+7
| | | | | | | It is causing many tests to fail on Windows buildbots: http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211 llvm-svn: 330848
* [lit] Report line number for failed RUN commandJoel E. Denny2018-04-2414-7/+131
| | | | | | | | | | | | | | | | | | | | | When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. Reviewed By: asmith, delcypher Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 330755
* [lit] Remove spurious `-` in invocation of lit inDan Liew2018-04-241-1/+1
| | | | | | | | | | `shtest-xunit-output.py` test. Although there is no `-` file Jeremy Morse has reported to me that it causes problems in their setup because lit tries to find it and ends up loading an out of tree lit configuration file. llvm-svn: 330728
* [lit] Fix a bug where UNRESOLVED tests were not handled in the XUnitDan Liew2018-04-201-0/+78
| | | | | | | | | | | XML printer. A test has been added that tries to comprehensively test emitting XUnit XML output for shell tests. Differential Revision: https://reviews.llvm.org/D45567 llvm-svn: 330409
* Revert "[lit] Temporarily disable shtest-timeout.py on darwin"Dan Liew2018-03-281-4/+1
| | | | | | | | | | This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598) Hopefully the test will now pass on the bots. rdar://problem/38774530 llvm-svn: 328703
* [lit] Remove a timing senstive part of `shtest-timeout.py`Dan Liew2018-03-282-52/+2
| | | | | | | | | | | | | | | | | | | | | | | The `shtest-timeout.py` test was failing intermittently. It looks like the issue is that on a resource constrained system lit is unable to run `quick_then_slow.py` twice and print out the messages the tests expects within the one second timeout. The underlying issue is that the test is dependent on the performance of the host machine is a rather fragile way. This is due to hardcoding timeout values and having assumptions that the host machine is able to perform a certain amount of work within the hardcoded timeout values. We could increase the timeout values but that doesn't really fix the underlying issue. Instead this patch removes one of fragile assumptions in the hope that this will be enough to fix the bots. There are other fragile assumptions in this test (e.g. `quick.py` can be executed in less than 1 second). If the bots continue to fail we'll have to revisit this. rdar://problem/38774530 llvm-svn: 328702
* Revert "Revert "[lit] Generalized /dev/null support on Windows.""Mircea Trofin2018-03-272-0/+28
| | | | | | | | | | | | | | | | | Summary: This reverts commit r328596. Checking if the arguments are strings before testing if they contain "/dev/null". Reviewers: rnk Reviewed By: rnk Subscribers: delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D44914 llvm-svn: 328603
* [lit] Temporarily disable shtest-timeout.py on darwinJan Korous2018-03-271-1/+4
| | | | | | | | Disabled until fixed in order to avoid random failures on green dragon. rdar://problem/38774530 llvm-svn: 328598
* Revert "[lit] Generalized /dev/null support on Windows."Mircea Trofin2018-03-262-28/+0
| | | | | | This reverts commit ca7fdbb974384ce5a05528b22a41d46b1cc13e92. llvm-svn: 328596
* [lit] Generalized /dev/null support on Windows.Mircea Trofin2018-03-262-0/+28
| | | | | | | | | | | | | | Generalized /dev/null remapping on Windows, and added test. Reviewers: rnk Reviewed By: rnk Subscribers: amccarth, zturner, delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D44771 llvm-svn: 328589
* [lit] Implement 'cat' command for internal shellReid Kleckner2018-03-266-2/+107
| | | | | | | | | | Fixes PR36449 Patch by Chamal de Silva Differential Revision: https://reviews.llvm.org/D43501 llvm-svn: 328563
OpenPOWER on IntegriCloud