summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit/tests
Commit message (Collapse)AuthorAgeFilesLines
* Losen time contraint to accommodate system loadsLei Huang2018-02-281-1/+1
| | | | llvm-svn: 326359
* [lit] Implement "-r" option for builtin "diff" command + a test using that.Max Moroz2018-01-0910-2/+132
| | | | | | | | | | | | | | | | Summary: That would allow to recursively compare directories in tests using "diff -r" on Windows in a similar way as it can be done on Linux or Mac. Reviewers: zturner, morehouse, vsk Reviewed By: zturner Subscribers: kcc, llvm-commits Differential Revision: https://reviews.llvm.org/D41776 llvm-svn: 322102
* [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands ↵Ying Yi2017-12-0118-2/+279
| | | | | | | | | | | | | | | | | internally Summary: The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands as the internal shell builtins. Reviewed by: Zachary Turner, Reid Kleckner Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39567 llvm-svn: 319528
* Make check-lit tests respect LLVM_LIT_TOOLS_DIRGreg Bedwell2017-11-293-5/+9
| | | | | | Differential Revision: https://reviews.llvm.org/D40520 llvm-svn: 319329
* Reverted rL318911 since it broke the sanitizer-windows.Ying Yi2017-11-2318-273/+2
| | | | llvm-svn: 318914
* [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands ↵Ying Yi2017-11-2318-2/+273
| | | | | | | | | | | | | | | | | internally Summary: The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands as the internal shell builtins. Reviewers: Zachary Turner, Reid Kleckner Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39567 llvm-svn: 318911
* Revert "lit.py: Add new %{shared_output(LABEL)} substitution"Jordan Rose2017-10-145-25/+0
| | | | | | | This reverts r315697 and my ill-fated attempts to fix it on Windows. I'll try again when I get access to a Windows machine. llvm-svn: 315793
* lit.py: Give up and disable the new shared-output.py test on WindowsJordan Rose2017-10-141-0/+3
| | | | | | | | | | "No such file or directory: C:\\...\\tests\\Output\\shared-output.py.tmp/Output/Shared/SHARED.tmp" And yet other forward-slashes don't seem to be causing the same problem. I'll see if I can get ahold of a Windows machine to poke at this directly later. llvm-svn: 315792
* lit.py: Previous test fix was a red herring; backslashes are escapesJordan Rose2017-10-141-1/+1
| | | | | | | I don't have access to a Windows machine at the moment, so if this doesn't fix it I'll just revert for now. llvm-svn: 315782
* lit.py: Fix new test harder for systems that don't use / as os.path.sepJordan Rose2017-10-141-1/+1
| | | | | | | I didn't think about '%{inputs}' having the same problem. This one should be a fully Windows path name. llvm-svn: 315779
* lit.py: Fix new test for systems that don't use / as os.path.sepJordan Rose2017-10-131-1/+1
| | | | llvm-svn: 315773
* lit.py: Add new %{shared_output(LABEL)} substitutionJordan Rose2017-10-135-0/+22
| | | | | | | | | | | | This refers to a temporary path that can be shared across all tests, identified by a particular label. This can be used for things like caches. At the moment, the character set for the LABEL is limited to C identifier characters, plus '-', '+', '=', and '.'. This is the same set of characters currently allowed in REQUIRES clause identifiers. llvm-svn: 315697
* [lit] Fix running lit tests in unconfigured source dirMichal Gorny2017-10-011-1/+1
| | | | | | | | | | | | | Fix llvm_tools_dir attribute access not to fail when the variable is not present. This directory is not really necessary to run lit tests, and the code already accounts for it being None. The reference was added in r313407, and it breaks the stand-alone lit package in Gentoo. Differential Revision: https://reviews.llvm.org/D38442 llvm-svn: 314620
* [lit] Actually do normalize the case of files in the config map.Zachary Turner2017-09-212-2/+4
| | | | | | | | | | | | | | | This has gone back and forth, but it seems this is necessary after all. realpath is not sufficient because if you have a file named 'C:\foo.txt', then both realpath('c:\foo.txt') and realpath(C:\foo.txt') return the string that was passed to them exactly as is, meaning the case of the drive-letter won't match. The problem before was not that we were normalizing the case of items going into the config map, but rather that we were normalizing the case of something we needed to print. The value that is used to key on the config map should never be printed. llvm-svn: 313918
* [lit] Don't norm case when inserting into the config map.Zachary Turner2017-09-213-3/+3
| | | | | | | | This makes all paths lowercase on Windows, which seemed like a good idea at the time, but it means that tests can't properly use FileCheck to match expected path names. llvm-svn: 313889
* [lit] Add a test for the builtin config map.Zachary Turner2017-09-217-0/+46
| | | | | | | | | | | | | | | | | | | Config map is not exposed through the command line, so testing this is somewhat tricky. But basically we need a test that if a custom driver builds a config map and passes it to main, it gets respected. A config map allows config files in the source tree to be mapped to alternate config files in the build tree. This particular test works by having two config files in separate directories, and setting up a config map to have that redirects A/lit.site.cfg to B/altconfig. Then, we print a message in A/lit.site.cfg and B/altconfig and check that we do see the output from B but don't see the output from A. Additionally we test that the test suite specified by A's config map is properly discovered. Differential Revision: https://reviews.llvm.org/D38105 llvm-svn: 313887
* [lit] Make lit support config files with .py extension.Zachary Turner2017-09-212-0/+33
| | | | | | | | | | | | | | | | | Many editors and Python-related diagnostics tools such as debuggers break or fail in mysterious ways when python files don't end in .py. This is especially true on Windows, but still exists on other platforms. I don't want to be too heavy handed in changing everything across the board, but I do want to at least *allow* lit configs to have .py extensions. This patch makes the discovery process first look for a config file with a .py extension, and if one is not found, then looks for a config file using the old method. So for existing users, there should be no functional change. Differential Revision: https://reviews.llvm.org/D37838 llvm-svn: 313849
* [lit] Fix the lit unit tests.Zachary Turner2017-09-161-0/+1
| | | | | | | | A few tests were manually constructing a LitConfig object, since I added a new argument to it this was triggering some failures I didn't detect. `ninja check-lit` passes now. llvm-svn: 313461
* Resubmit "[lit] Force site configs to run before source-tree configs"Zachary Turner2017-09-151-3/+2
| | | | | | | | | | | | | | | | | | | | This is a resubmission of r313270. It broke standalone builds of compiler-rt because we were not correctly generating the llvm-lit script in the standalone build directory. The fixes incorporated here attempt to find llvm/utils/llvm-lit from the source tree returned by llvm-config. If present, it will generate llvm-lit into the output directory. Regardless, the user can specify -DLLVM_EXTERNAL_LIT to point to a specific lit.py on their file system. This supports the use case of someone installing lit via a package manager. If it cannot find a source tree, and -DLLVM_EXTERNAL_LIT is either unspecified or invalid, then we print a warning that tests will not be able to run. Differential Revision: https://reviews.llvm.org/D37756 llvm-svn: 313407
* Revert "[lit] Force site configs to run before source-tree configs"Zachary Turner2017-09-151-2/+3
| | | | | | | | This patch is still breaking several multi-stage compiler-rt bots. I already know what the fix is, but I want to get the bots green for now and then try re-applying in the morning. llvm-svn: 313335
* [lit] Force site configs to be run before source-tree configsZachary Turner2017-09-141-3/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch simplifies LLVM's lit infrastructure by enforcing an ordering that a site config is always run before a source-tree config. A significant amount of the complexity from lit config files arises from the fact that inside of a source-tree config file, we don't yet know if the site config has been run. However it is *always* required to run a site config first, because it passes various variables down through CMake that the main config depends on. As a result, every config file has to do a bunch of magic to try to reverse-engineer the location of the site config file if they detect (heuristically) that the site config file has not yet been run. This patch solves the problem by emitting a mapping from source tree config file to binary tree site config file in llvm-lit.py. Then, during discovery when we find a config file, we check to see if we have a target mapping for it, and if so we use that instead. This mechanism is generic enough that it does not affect external users of lit. They will just not have a config mapping defined, and everything will work as normal. On the other hand, for us it allows us to make many simplifications: * We are guaranteed that a site config will be executed first * Inside of a main config, we no longer have to assume that attributes might not be present and use getattr everywhere. * We no longer have to pass parameters such as --param llvm_site_config=<path> on the command line. * It is future-proof, meaning you don't have to edit llvm-lit.in to add support for new projects. * All of the duplicated logic of trying various fallback mechanisms of finding a site config from the main config are now gone. One potentially noteworthy thing that was required to implement this change is that whereas the ninja check targets previously used the first method to spawn lit, they now use the second. In particular, you can no longer run lit.py against the source tree while specifying the various `foo_site_config=<path>` parameters. Instead, you need to run llvm-lit.py. Differential Revision: https://reviews.llvm.org/D37756 llvm-svn: 313270
* [lit] Fix some issues with short paths in test discovery introduced in r312254Reid Kleckner2017-08-311-0/+7
| | | | llvm-svn: 312288
* Make lit :: shtest-format.py supported on Windows againVictor Leschuk2017-08-231-1/+0
| | | | | | | | | | | | It was marked as unsupported on Windows in r311230 because on some Win10 machines it failed or caused hang. The problem was that on these machines system bash (C:\Windows\System32\bash.exe) was used which requires paths to be passed like '/mnt/c/path/to/my/script' instead of 'C:\path\to\my\script'. TODO: we should make lit detect if system bash is used instead of msys and set appropriate path format. llvm-svn: 311558
* Temporary mark lit :: shtest-format as unsupported on windowsVictor Leschuk2017-08-191-0/+1
| | | | | | When run manually it fails, but when run under buildbot it causes hang. llvm-svn: 311230
* [lit] support unsetting env variables (again!)Ben Dunbobbin2017-08-186-0/+76
| | | | | | | | | | | | This is an updated version of https://reviews.llvm.org/D22144 by @jlpeyton. The patch was accepted but not landed. This is useful functionality and I would like to use this to enable lit tests for environment variable behaviour. Differential Revision: https://reviews.llvm.org/D36403 llvm-svn: 311180
* Fix check-lit compatibility with multi-config CMake generatorsGreg Bedwell2017-08-031-3/+14
| | | | | | | | | | | | | | | Multi-configuration CMake generators such as those for Visual Studio or Xcode do not specify a build config at configure time, but let the user choose at build time. In these cases binaries go into build/${Configuration}/bin rather than build/bin. Prior to this commit, check-lit would fail when using multi-configuration generators as it did not know how to resolve ${Configuration} in order to find tools such as FileCheck. This commit teaches it to resolve llvm_tools_dir within lit using the value specified with --param build_mode. Differential Revision: https://reviews.llvm.org/D36263 llvm-svn: 309967
* Revert "[lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configs"Reid Kleckner2017-08-023-30/+1
| | | | | | | This reverts r309602, check-lit still leaves Output directories in the source directory. llvm-svn: 309833
* [lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configsReid Kleckner2017-07-313-1/+30
| | | | | | | | | | | | | | | | | | | Summary: This is an alternative solution to running the lit test suite on bots without polluting the source directory. Each input test suite gets an auto-generated site config in the build directory that points back to the test input source directory. This adds some cmake comlexity, but now we don't need to remove and re-copy the test input directory before every test. Reviewers: delcypher, modocache Subscribers: mgorny, llvm-commits Differential Revision: https://reviews.llvm.org/D36026 llvm-svn: 309602
* lit::shtest-format.py: Make write-bad-encoding.py py3-aware.NAKAMURA Takumi2017-07-291-1/+1
| | | | | | | | | | Traceback (most recent call last): File "llvm/utils/lit/tests/Inputs/shtest-format/external_shell/write-bad-encoding.py", line 5, in <module> sys.stdout.write(b"a line with bad encoding: \xc2.") sys.stdout.write doesn't accept bytes but sys.stdout.buffer.write accepts. llvm-svn: 309473
* [lit] Use a %{python} substitution to avoid relying on python being on PATHReid Kleckner2017-07-285-9/+7
| | | | llvm-svn: 309434
* [lit] Dump some FileCheck inputs to try to debug some failing testsReid Kleckner2017-07-282-0/+6
| | | | llvm-svn: 309400
* [lit] Fix shtest-format external_shell failuresReid Kleckner2017-07-283-4/+7
| | | | | | | | | | When using win32 cmd.exe, turn off command echoing at the beginning of the script (@echo off). Replace a bash shell script with a python script for the fail_with_bad_encoding test. llvm-svn: 309399
* [lit] Port googletest lit tests to WindowsReid Kleckner2017-07-286-22/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: The technique of directly calling subprocess.Popen on a python script doesn't work on Windows. The executable path of the command must refer to a valid win32 executable. Instead, rename all the python scripts masquerading as gtest executables to have .py extensions, so we can easily detect then and call the python executable for them. Do this on Linux as well as Windows for consistency. The test suite directory names also come out in lower-case on Windows. We can consider removing that in a later patch. This change just updates the FileCheck lines to match on Windows. Fixes PR33933 Reviewers: modocache, mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35909 llvm-svn: 309347
* Un-revert "Teach the CMake build system to run lit's test suite. These can ↵Brian Gesiak2017-07-272-2/+24
| | | | | | | | | | | | | | | | | | | | | | be run" Summary: Depends on https://reviews.llvm.org/D35879. This reverts rL257268, which in turn was a revert of rL257221. https://reviews.llvm.org/D35879 marks the tests in the lit test suite that fail on Windows as XFAIL, which should allow these tests to pass on Windows-based buildbots. Reviewers: delcypher, beanz, mgorny, jroelofs, rnk Reviewed By: mgorny Subscribers: rnk, ddunbar, george.karpenkov, llvm-commits Differential Revision: https://reviews.llvm.org/D35880 llvm-svn: 309310
* [lit] Fix order of checks in shtest-shell.py testBrian Gesiak2017-07-272-0/+10
| | | | | | | | | | | | | | | | | | Summary: An expectation in `utils/lit/tests/Inputs/shtest-shell/redirects.txt` expects that first a string printed to stdout is seen, and then a string printed to stderr. Add `flush()` calls to ensure that stdout is printed before stderr, as expected. Reviewers: rnk, mgorny, jroelofs Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35947 llvm-svn: 309292
* [lit] Fix race between shtest-shell and max-failures testsReid Kleckner2017-07-262-4/+10
| | | | | | | Previously these tests would use the same Output directory leading to flaky non-deterministic failures. llvm-svn: 309227
* [lit] Fix shtest-shell and max-failures lit tests on WindowsReid Kleckner2017-07-267-16/+10
| | | | | | | | | Rewrite the write-to-stderr.sh and write-to-stdout-and-stderr.sh shell scripts as python scripts and call python on them. Fixes PR33940 llvm-svn: 309200
* [lit] Fix shtest-output-printing.py on Windows by matching either / or \\Reid Kleckner2017-07-261-4/+1
| | | | | | Fixes PR33938 llvm-svn: 309198
* [lit] Fix discovery.py on Windows by matching backslashes when necessaryReid Kleckner2017-07-261-22/+19
| | | | | | Fixes PR33932 llvm-svn: 309194
* [lit] Un-XFAIL selecting.py test on WindowsReid Kleckner2017-07-261-3/+0
| | | | | | | | This passes locally for me, which fails the overall lit test suite. I can't debug a passing test, but I will try to help debug the test when we get some failing logs. llvm-svn: 309190
* Un-XFAIL some internal lit tests on Windows, they pass for me locallyReid Kleckner2017-07-263-9/+0
| | | | llvm-svn: 309144
* Fix LIT test breakageGeorge Karpenkov2017-07-261-1/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D35867 llvm-svn: 309140
* [lit] Mark several of lit's tests XFAIL on WindowsBrian Gesiak2017-07-2613-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: rL257221 attempted to run lit's own test suite continuously, but that commit was reverted because lit's test suite does not pass on Windows. Because lit's tests do not run continuously, they often regress. In order to un-revert rL257221, mark lit tests that fail as XFAIL for Windows platforms. Test Plan: On a Windows development environment, follow the instructions in utils/lit/README.txt to run lit's test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test suite is run and a successful exit code is returned. Reviewers: mgorny, rnk, delcypher, beanz Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35879 llvm-svn: 309123
* [lit] Modify LIT to accept environment variable LIT_FILTER to select tests.George Karpenkov2017-07-071-0/+5
| | | | | | | | | This is especially useful when lit is invoked indirectly by the build system, and additional arguments can not be easily specified. Differential Revision: https://reviews.llvm.org/D35091 llvm-svn: 307339
* [lit] Implement timeouts and max_time for process pool testingReid Kleckner2017-04-062-38/+42
| | | | | | | | | | | | | | | | | | | This is necessary to pass the lit test suite at llvm/utils/lit/tests. There are some pre-existing failures here, but now switching to pools doesn't regress any tests. I had to change test-data/lit.cfg to import DummyConfig from a module to fix pickling problems, but I think it'll be OK if we require test formats to be written in real .py modules outside lit.cfg files. I also discovered that in some circumstances AsyncResult.wait() will not raise KeyboardInterrupt in a timely manner, but you can pass a non-zero timeout to work around this. This makes threading.Condition.wait use a polling loop that runs through the interpreter, so it's capable of asynchronously raising KeyboardInterrupt. llvm-svn: 299605
* lit: remove python2-ismsBrian Gesiak2017-03-221-2/+2
| | | | | | | | | | | | | | | | | Summary: `assert.assertItemEqual` went away in Python 3. Seeing how lists are ordered, comparing a list against each other should work just as well. Patch by @jbergstroem (Johan Bergström). Reviewers: modocache, gparker42 Reviewed By: modocache Differential Revision: https://reviews.llvm.org/D31229 llvm-svn: 298479
* Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILGreg Parker2017-01-2512-7/+109
| | | | | | | | and UNSUPPORTED" This reverts the revert in r292942. llvm-svn: 293007
* Revert "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAILAlex Lorenz2017-01-2412-109/+7
| | | | | | | | | | | and UNSUPPORTED" After r292904 llvm-lit fails to emit the test results in the XML format for Apple's internal buildbots. rdar://30164800 llvm-svn: 292942
* [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTEDGreg Parker2017-01-2412-7/+109
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 llvm-svn: 292904
* Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED"Greg Parker2017-01-2415-122/+13
| | | | | | This change needs to be better-coordinated with libc++. llvm-svn: 292900
OpenPOWER on IntegriCloud