summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
...
* Try to fix some failing bots.Zachary Turner2017-09-161-6/+8
| | | | | | | | | It doesn't make sense to me why these bots are failing as the traceback does not agree with the source code. It's possible something is stale or there is some other mysterious error, but in any case hopefully this fixes it. llvm-svn: 313469
* Resubmit "Add a shared llvm.lit module that all test suites can use."Zachary Turner2017-09-163-0/+149
| | | | | | | | There were some issues surrounding Py2 / Py3 compatibility, but I've now tested with both Py2 and Py3 and everything seems to work. llvm-svn: 313467
* [lit] Fix some Python 3 compatibility issues.Zachary Turner2017-09-161-7/+12
| | | | llvm-svn: 313466
* [lit] Fix the lit unit tests.Zachary Turner2017-09-162-0/+2
| | | | | | | | A few tests were manually constructing a LitConfig object, since I added a new argument to it this was triggering some failures I didn't detect. `ninja check-lit` passes now. llvm-svn: 313461
* [lit] Add a single process mode.Zachary Turner2017-09-163-45/+58
| | | | | | | | | This is helpful for debugging test failures since it removes the multiprocessing pool from the picture. This will obviously slow down the test suite by a few orders of magnitude, so it should only be used for debugging specific failures. llvm-svn: 313460
* Revert lit changes related to lit.llvm module.Zachary Turner2017-09-163-150/+0
| | | | | | | | It looks like this is going to be non-trivial to get working in both Py2 and Py3, so for now I'm reverting until I have time to fully test it under Python 3. llvm-svn: 313429
* [lit] Fix another Python 3 error.Zachary Turner2017-09-161-1/+8
| | | | | | | Apparently we have a buildbot running Python 3. This is going to be fun :-/ llvm-svn: 313428
* [lit] Better check for integral value.Zachary Turner2017-09-161-1/+2
| | | | | | | Some versions of python don't have 'long'. Use numbers.Number instead. llvm-svn: 313427
* Resubmit "[lit] Add a lit.llvm module that all llvm projects can use"Zachary Turner2017-09-163-0/+142
| | | | | | | This was reverted alongside the revert of the lit/llvm-lit refactor, but now that that has re-landed, I'm relanding this as well. llvm-svn: 313426
* Resubmit "[lit] Force site configs to run before source-tree configs"Zachary Turner2017-09-152-4/+17
| | | | | | | | | | | | | | | | | | | | This is a resubmission of r313270. It broke standalone builds of compiler-rt because we were not correctly generating the llvm-lit script in the standalone build directory. The fixes incorporated here attempt to find llvm/utils/llvm-lit from the source tree returned by llvm-config. If present, it will generate llvm-lit into the output directory. Regardless, the user can specify -DLLVM_EXTERNAL_LIT to point to a specific lit.py on their file system. This supports the use case of someone installing lit via a package manager. If it cannot find a source tree, and -DLLVM_EXTERNAL_LIT is either unspecified or invalid, then we print a warning that tests will not be able to run. Differential Revision: https://reviews.llvm.org/D37756 llvm-svn: 313407
* Revert "[lit] Force site configs to run before source-tree configs"Zachary Turner2017-09-152-17/+4
| | | | | | | | This patch is still breaking several multi-stage compiler-rt bots. I already know what the fix is, but I want to get the bots green for now and then try re-applying in the morning. llvm-svn: 313335
* [lit] Revert "Add a lit.llvm module that all llvm projects can use"Zachary Turner2017-09-153-142/+0
| | | | | | | This is breaking due to some changes I forgot to merge in, so I'm temporarily reverting them until I can re-test that this works. llvm-svn: 313328
* [lit] Remove some code that I forgot to remove.Zachary Turner2017-09-151-8/+0
| | | | llvm-svn: 313326
* [lit] Add a lit.llvm module that all test suites can use.Zachary Turner2017-09-153-0/+150
| | | | | | | | | | | | | | To further reduce duplicate code, this patch introduces a module that configs can simply import and get access to a lot of useful functionality such as setting up paths, adding features that are useful across all projects, and other utility-type functions. For now this only updates llvm's suite to use this new library, but subsequent patches will update other projects. Differential Revision: https://reviews.llvm.org/D37778 llvm-svn: 313325
* [lit] Fix some windows line endings that snuck in.Zachary Turner2017-09-141-2/+2
| | | | llvm-svn: 313301
* [lit] Force site configs to be run before source-tree configsZachary Turner2017-09-142-4/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch simplifies LLVM's lit infrastructure by enforcing an ordering that a site config is always run before a source-tree config. A significant amount of the complexity from lit config files arises from the fact that inside of a source-tree config file, we don't yet know if the site config has been run. However it is *always* required to run a site config first, because it passes various variables down through CMake that the main config depends on. As a result, every config file has to do a bunch of magic to try to reverse-engineer the location of the site config file if they detect (heuristically) that the site config file has not yet been run. This patch solves the problem by emitting a mapping from source tree config file to binary tree site config file in llvm-lit.py. Then, during discovery when we find a config file, we check to see if we have a target mapping for it, and if so we use that instead. This mechanism is generic enough that it does not affect external users of lit. They will just not have a config mapping defined, and everything will work as normal. On the other hand, for us it allows us to make many simplifications: * We are guaranteed that a site config will be executed first * Inside of a main config, we no longer have to assume that attributes might not be present and use getattr everywhere. * We no longer have to pass parameters such as --param llvm_site_config=<path> on the command line. * It is future-proof, meaning you don't have to edit llvm-lit.in to add support for new projects. * All of the duplicated logic of trying various fallback mechanisms of finding a site config from the main config are now gone. One potentially noteworthy thing that was required to implement this change is that whereas the ninja check targets previously used the first method to spawn lit, they now use the second. In particular, you can no longer run lit.py against the source tree while specifying the various `foo_site_config=<path>` parameters. Instead, you need to run llvm-lit.py. Differential Revision: https://reviews.llvm.org/D37756 llvm-svn: 313270
* [lit] Fix some issues with short paths in test discovery introduced in r312254Reid Kleckner2017-08-312-1/+8
| | | | llvm-svn: 312288
* [lit] Make symlinks in test paths work a different wayReid Kleckner2017-08-311-3/+4
| | | | | | | | | | | | | Use os.path.normpath instead of realpath to collapse '..' and '.' path components. Use realpath when caching search results about a path for good measure. I considered rigging up a test involving symlinks for this, but I doubt I can check a symlink into SVN. The test would have to conditionally create a symlink at runtime if the host OS supports it. This sounds too fragile and complicated to me to be worth it. llvm-svn: 312254
* Revert "[lit] Don't call realpath on the path used for test suite search"Reid Kleckner2017-08-311-0/+3
| | | | | | This reverts r312250, it breaks the lit test suite. llvm-svn: 312251
* [lit] Don't call realpath on the path used for test suite searchReid Kleckner2017-08-311-3/+0
| | | | | | | | | This preserves symlinks in paths, so that someone can symlink more tests into a larger test suite. For example, debuginfo-tests is currently designed to be checked out into clang/test. With this change, it can be symlinked into place instead, which works better with the monorepo. llvm-svn: 312250
* Make lit :: shtest-format.py supported on Windows againVictor Leschuk2017-08-231-1/+0
| | | | | | | | | | | | It was marked as unsupported on Windows in r311230 because on some Win10 machines it failed or caused hang. The problem was that on these machines system bash (C:\Windows\System32\bash.exe) was used which requires paths to be passed like '/mnt/c/path/to/my/script' instead of 'C:\path\to\my\script'. TODO: we should make lit detect if system bash is used instead of msys and set appropriate path format. llvm-svn: 311558
* Temporary mark lit :: shtest-format as unsupported on windowsVictor Leschuk2017-08-191-0/+1
| | | | | | When run manually it fails, but when run under buildbot it causes hang. llvm-svn: 311230
* [lit] support unsetting env variables (again!)Ben Dunbobbin2017-08-187-0/+89
| | | | | | | | | | | | This is an updated version of https://reviews.llvm.org/D22144 by @jlpeyton. The patch was accepted but not landed. This is useful functionality and I would like to use this to enable lit tests for environment variable behaviour. Differential Revision: https://reviews.llvm.org/D36403 llvm-svn: 311180
* Fix check-lit compatibility with multi-config CMake generatorsGreg Bedwell2017-08-032-4/+15
| | | | | | | | | | | | | | | Multi-configuration CMake generators such as those for Visual Studio or Xcode do not specify a build config at configure time, but let the user choose at build time. In these cases binaries go into build/${Configuration}/bin rather than build/bin. Prior to this commit, check-lit would fail when using multi-configuration generators as it did not know how to resolve ${Configuration} in order to find tools such as FileCheck. This commit teaches it to resolve llvm_tools_dir within lit using the value specified with --param build_mode. Differential Revision: https://reviews.llvm.org/D36263 llvm-svn: 309967
* Revert "[lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configs"Reid Kleckner2017-08-024-30/+28
| | | | | | | This reverts r309602, check-lit still leaves Output directories in the source directory. llvm-svn: 309833
* [lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configsReid Kleckner2017-07-314-28/+30
| | | | | | | | | | | | | | | | | | | Summary: This is an alternative solution to running the lit test suite on bots without polluting the source directory. Each input test suite gets an auto-generated site config in the build directory that points back to the test input source directory. This adds some cmake comlexity, but now we don't need to remove and re-copy the test input directory before every test. Reviewers: delcypher, modocache Subscribers: mgorny, llvm-commits Differential Revision: https://reviews.llvm.org/D36026 llvm-svn: 309602
* lit::shtest-format.py: Make write-bad-encoding.py py3-aware.NAKAMURA Takumi2017-07-291-1/+1
| | | | | | | | | | Traceback (most recent call last): File "llvm/utils/lit/tests/Inputs/shtest-format/external_shell/write-bad-encoding.py", line 5, in <module> sys.stdout.write(b"a line with bad encoding: \xc2.") sys.stdout.write doesn't accept bytes but sys.stdout.buffer.write accepts. llvm-svn: 309473
* [lit] Use a %{python} substitution to avoid relying on python being on PATHReid Kleckner2017-07-285-9/+7
| | | | llvm-svn: 309434
* [lit] Remove stale test inputs before running check-litReid Kleckner2017-07-281-0/+1
| | | | | | | | This should fix googletest-format test failures on the clang modules buildbots, which have a stale copy of the OneTest script in the build directory. llvm-svn: 309432
* [lit] Dump some FileCheck inputs to try to debug some failing testsReid Kleckner2017-07-282-0/+6
| | | | llvm-svn: 309400
* [lit] Fix shtest-format external_shell failuresReid Kleckner2017-07-284-4/+8
| | | | | | | | | | When using win32 cmd.exe, turn off command echoing at the beginning of the script (@echo off). Replace a bash shell script with a python script for the fail_with_bad_encoding test. llvm-svn: 309399
* [lit] Port googletest lit tests to WindowsReid Kleckner2017-07-287-26/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: The technique of directly calling subprocess.Popen on a python script doesn't work on Windows. The executable path of the command must refer to a valid win32 executable. Instead, rename all the python scripts masquerading as gtest executables to have .py extensions, so we can easily detect then and call the python executable for them. Do this on Linux as well as Windows for consistency. The test suite directory names also come out in lower-case on Windows. We can consider removing that in a later patch. This change just updates the FileCheck lines to match on Windows. Fixes PR33933 Reviewers: modocache, mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35909 llvm-svn: 309347
* [lit] Fix TestRunner unit test on WindowsBrian Gesiak2017-07-271-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Normally Python converts all newline characters, Windows or Unix, to Unix newlines when opening a file. However, lit opens files in binary mode, which does not perform this conversion. As a result, trailing Windows newlines are not stripped from test input, which caused a failure in the TestRunner unit test: ``` FAIL: test_custom (__main__.TestIntegratedTestKeywordParser) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\bgesiak\src\llvm\llvm\utils\lit\tests\unit\TestRunner.py", line 109, in test_custom self.assertItemsEqual(value, ['a', 'b', 'c']) AssertionError: Element counts were not equal: First has 1, Second has 0: 'c\r' First has 0, Second has 1: 'c' ``` Fix the discrepancy in behavior across the two platforms by manually stripping Windows newlines before yielding each line in the test file. Reviewers: echristo, beanz, ddunbar, delcypher, rnk Reviewed By: rnk Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D27746 llvm-svn: 309312
* Un-revert "Teach the CMake build system to run lit's test suite. These can ↵Brian Gesiak2017-07-273-2/+50
| | | | | | | | | | | | | | | | | | | | | | be run" Summary: Depends on https://reviews.llvm.org/D35879. This reverts rL257268, which in turn was a revert of rL257221. https://reviews.llvm.org/D35879 marks the tests in the lit test suite that fail on Windows as XFAIL, which should allow these tests to pass on Windows-based buildbots. Reviewers: delcypher, beanz, mgorny, jroelofs, rnk Reviewed By: mgorny Subscribers: rnk, ddunbar, george.karpenkov, llvm-commits Differential Revision: https://reviews.llvm.org/D35880 llvm-svn: 309310
* [lit] Fix order of checks in shtest-shell.py testBrian Gesiak2017-07-272-0/+10
| | | | | | | | | | | | | | | | | | Summary: An expectation in `utils/lit/tests/Inputs/shtest-shell/redirects.txt` expects that first a string printed to stdout is seen, and then a string printed to stderr. Add `flush()` calls to ensure that stdout is printed before stderr, as expected. Reviewers: rnk, mgorny, jroelofs Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35947 llvm-svn: 309292
* [lit] Fix race between shtest-shell and max-failures testsReid Kleckner2017-07-262-4/+10
| | | | | | | Previously these tests would use the same Output directory leading to flaky non-deterministic failures. llvm-svn: 309227
* [lit] Fix shtest-shell and max-failures lit tests on WindowsReid Kleckner2017-07-267-16/+10
| | | | | | | | | Rewrite the write-to-stderr.sh and write-to-stdout-and-stderr.sh shell scripts as python scripts and call python on them. Fixes PR33940 llvm-svn: 309200
* [lit] Fix shtest-output-printing.py on Windows by matching either / or \\Reid Kleckner2017-07-261-4/+1
| | | | | | Fixes PR33938 llvm-svn: 309198
* [lit] Fix discovery.py on Windows by matching backslashes when necessaryReid Kleckner2017-07-261-22/+19
| | | | | | Fixes PR33932 llvm-svn: 309194
* [lit] Un-XFAIL selecting.py test on WindowsReid Kleckner2017-07-261-3/+0
| | | | | | | | This passes locally for me, which fails the overall lit test suite. I can't debug a passing test, but I will try to help debug the test when we get some failing logs. llvm-svn: 309190
* Un-XFAIL some internal lit tests on Windows, they pass for me locallyReid Kleckner2017-07-263-9/+0
| | | | llvm-svn: 309144
* Fix LIT test breakageGeorge Karpenkov2017-07-261-1/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D35867 llvm-svn: 309140
* [lit] Mark several of lit's tests XFAIL on WindowsBrian Gesiak2017-07-2613-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: rL257221 attempted to run lit's own test suite continuously, but that commit was reverted because lit's test suite does not pass on Windows. Because lit's tests do not run continuously, they often regress. In order to un-revert rL257221, mark lit tests that fail as XFAIL for Windows platforms. Test Plan: On a Windows development environment, follow the instructions in utils/lit/README.txt to run lit's test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test suite is run and a successful exit code is returned. Reviewers: mgorny, rnk, delcypher, beanz Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35879 llvm-svn: 309123
* [lit] Fix type error for parallelism groupsBrian Gesiak2017-07-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Whereas rL299560 and rL309071 call `parallelism_groups.items()`, under the assumption that `parallelism_groups` is a `dict` type, the default parameter for that attribute is a `list`. Change the default to a `dict` for type correctness. This regression in the unit tests would have been caught if the unit tests were being run continously. It also would have been caught if the lit project used a Python type checker such as `mypy`. Test Plan: As per the instructions in `utils/lit/README.txt`, run the lit unit test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test `lit :: unit/TestRunner.py` fails before applying this patch, but passes once this patch is applied. Reviewers: mgorny, rnk, rafael Reviewed By: mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35878 llvm-svn: 309122
* Revert "[lit] Remove dead code not referenced in the LLVM SVN repo."Brian Gesiak2017-07-263-74/+123
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This reverts rL306623, which removed `FileBasedTest`, an abstract base class, but did not also remove the usages of that class in the lit unit tests. The revert fixes four test failures in the lit unit test suite. Test plan: As per the instructions in `utils/lit/README.txt`, run the lit unit test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the following tests fail before applying this patch, and pass once the patch is applied: ``` lit :: test-data.py lit :: test-output.py lit :: xunit-output.py ``` In addition, run `check-llvm` to make sure the existing LLVM test suite executes normally. Reviewers: george.karpenkov, mgorny, dlj Reviewed By: mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35877 llvm-svn: 309120
* [lit] Attempt to fix Python unittest adaptor logicReid Kleckner2017-07-261-7/+6
| | | | llvm-svn: 309071
* [lit] Fix UnboundLocalError for invalid shtest redirectsMichal Gorny2017-07-251-1/+1
| | | | | | | | | | | | | | | | | Replace the incorrect variable reference when invalid redirect is used. This fixes the following issue: File "/usr/src/llvm/utils/lit/lit/TestRunner.py", line 316, in processRedirects raise InternalShellError(cmd, "Unsupported redirect: %r" % (r,)) UnboundLocalError: local variable 'r' referenced before assignment which in turn broke shtest-shell.py and max-failures.py lit tests. The breakage was introduced during refactoring in rL307310. Differential Revision: https://reviews.llvm.org/D35857 llvm-svn: 309044
* [lit] add a -vv option to echo all executed commands.George Karpenkov2017-07-133-2/+16
| | | | | | | | | | | | | | | | | | | Debugging LIT scripts can be rather painful, as LIT directly does not specify which line has failed. Rather, FileCheck is expected to report the failing location, but it can be often ambiguous if multiple commands are tested against the same prefix. This change adds a -vv option, which echoes all output. Then detecting the error becomes straightforward: last printed line is the failing one. Of course, it could be desired to try to get failing line number directly from bash, but it involves excessive hacks on older bash versions (cf. https://stackoverflow.com/questions/24398691/how-to-get-the-real-line-number-of-a-failing-bash-command) Differential Revision: https://reviews.llvm.org/D35330 llvm-svn: 307938
* [lit] Fix import StringIO errors in Python 3Reid Kleckner2017-07-111-3/+3
| | | | | | | Remove the cStringIO micro-optimization, as it isn't portable to Python 3. llvm-svn: 307669
* [lit] Implement non-pipelined echo commands internallyReid Kleckner2017-07-111-0/+74
| | | | | | | | | | | | | | | Summary: This speeds up the LLD test suite on Windows by 3x. Most of the time is spent on lld/test/ELF/linkerscript/diagnostics.s, which repeatedly constructs linker scripts with appending echo commands. Reviewers: dlj, zturner, modocache Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35093 llvm-svn: 307668
OpenPOWER on IntegriCloud