summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
...
* [lit] Fix some issues with short paths in test discovery introduced in r312254Reid Kleckner2017-08-312-1/+8
| | | | llvm-svn: 312288
* [lit] Make symlinks in test paths work a different wayReid Kleckner2017-08-311-3/+4
| | | | | | | | | | | | | Use os.path.normpath instead of realpath to collapse '..' and '.' path components. Use realpath when caching search results about a path for good measure. I considered rigging up a test involving symlinks for this, but I doubt I can check a symlink into SVN. The test would have to conditionally create a symlink at runtime if the host OS supports it. This sounds too fragile and complicated to me to be worth it. llvm-svn: 312254
* Revert "[lit] Don't call realpath on the path used for test suite search"Reid Kleckner2017-08-311-0/+3
| | | | | | This reverts r312250, it breaks the lit test suite. llvm-svn: 312251
* [lit] Don't call realpath on the path used for test suite searchReid Kleckner2017-08-311-3/+0
| | | | | | | | | This preserves symlinks in paths, so that someone can symlink more tests into a larger test suite. For example, debuginfo-tests is currently designed to be checked out into clang/test. With this change, it can be symlinked into place instead, which works better with the monorepo. llvm-svn: 312250
* Make lit :: shtest-format.py supported on Windows againVictor Leschuk2017-08-231-1/+0
| | | | | | | | | | | | It was marked as unsupported on Windows in r311230 because on some Win10 machines it failed or caused hang. The problem was that on these machines system bash (C:\Windows\System32\bash.exe) was used which requires paths to be passed like '/mnt/c/path/to/my/script' instead of 'C:\path\to\my\script'. TODO: we should make lit detect if system bash is used instead of msys and set appropriate path format. llvm-svn: 311558
* Temporary mark lit :: shtest-format as unsupported on windowsVictor Leschuk2017-08-191-0/+1
| | | | | | When run manually it fails, but when run under buildbot it causes hang. llvm-svn: 311230
* [lit] support unsetting env variables (again!)Ben Dunbobbin2017-08-187-0/+89
| | | | | | | | | | | | This is an updated version of https://reviews.llvm.org/D22144 by @jlpeyton. The patch was accepted but not landed. This is useful functionality and I would like to use this to enable lit tests for environment variable behaviour. Differential Revision: https://reviews.llvm.org/D36403 llvm-svn: 311180
* Fix check-lit compatibility with multi-config CMake generatorsGreg Bedwell2017-08-032-4/+15
| | | | | | | | | | | | | | | Multi-configuration CMake generators such as those for Visual Studio or Xcode do not specify a build config at configure time, but let the user choose at build time. In these cases binaries go into build/${Configuration}/bin rather than build/bin. Prior to this commit, check-lit would fail when using multi-configuration generators as it did not know how to resolve ${Configuration} in order to find tools such as FileCheck. This commit teaches it to resolve llvm_tools_dir within lit using the value specified with --param build_mode. Differential Revision: https://reviews.llvm.org/D36263 llvm-svn: 309967
* Revert "[lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configs"Reid Kleckner2017-08-024-30/+28
| | | | | | | This reverts r309602, check-lit still leaves Output directories in the source directory. llvm-svn: 309833
* [lit] Avoid copying llvm/utils/lit/tests/Inputs with lit site configsReid Kleckner2017-07-314-28/+30
| | | | | | | | | | | | | | | | | | | Summary: This is an alternative solution to running the lit test suite on bots without polluting the source directory. Each input test suite gets an auto-generated site config in the build directory that points back to the test input source directory. This adds some cmake comlexity, but now we don't need to remove and re-copy the test input directory before every test. Reviewers: delcypher, modocache Subscribers: mgorny, llvm-commits Differential Revision: https://reviews.llvm.org/D36026 llvm-svn: 309602
* lit::shtest-format.py: Make write-bad-encoding.py py3-aware.NAKAMURA Takumi2017-07-291-1/+1
| | | | | | | | | | Traceback (most recent call last): File "llvm/utils/lit/tests/Inputs/shtest-format/external_shell/write-bad-encoding.py", line 5, in <module> sys.stdout.write(b"a line with bad encoding: \xc2.") sys.stdout.write doesn't accept bytes but sys.stdout.buffer.write accepts. llvm-svn: 309473
* [lit] Use a %{python} substitution to avoid relying on python being on PATHReid Kleckner2017-07-285-9/+7
| | | | llvm-svn: 309434
* [lit] Remove stale test inputs before running check-litReid Kleckner2017-07-281-0/+1
| | | | | | | | This should fix googletest-format test failures on the clang modules buildbots, which have a stale copy of the OneTest script in the build directory. llvm-svn: 309432
* [lit] Dump some FileCheck inputs to try to debug some failing testsReid Kleckner2017-07-282-0/+6
| | | | llvm-svn: 309400
* [lit] Fix shtest-format external_shell failuresReid Kleckner2017-07-284-4/+8
| | | | | | | | | | When using win32 cmd.exe, turn off command echoing at the beginning of the script (@echo off). Replace a bash shell script with a python script for the fail_with_bad_encoding test. llvm-svn: 309399
* [lit] Port googletest lit tests to WindowsReid Kleckner2017-07-287-26/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: The technique of directly calling subprocess.Popen on a python script doesn't work on Windows. The executable path of the command must refer to a valid win32 executable. Instead, rename all the python scripts masquerading as gtest executables to have .py extensions, so we can easily detect then and call the python executable for them. Do this on Linux as well as Windows for consistency. The test suite directory names also come out in lower-case on Windows. We can consider removing that in a later patch. This change just updates the FileCheck lines to match on Windows. Fixes PR33933 Reviewers: modocache, mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35909 llvm-svn: 309347
* [lit] Fix TestRunner unit test on WindowsBrian Gesiak2017-07-271-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Normally Python converts all newline characters, Windows or Unix, to Unix newlines when opening a file. However, lit opens files in binary mode, which does not perform this conversion. As a result, trailing Windows newlines are not stripped from test input, which caused a failure in the TestRunner unit test: ``` FAIL: test_custom (__main__.TestIntegratedTestKeywordParser) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\bgesiak\src\llvm\llvm\utils\lit\tests\unit\TestRunner.py", line 109, in test_custom self.assertItemsEqual(value, ['a', 'b', 'c']) AssertionError: Element counts were not equal: First has 1, Second has 0: 'c\r' First has 0, Second has 1: 'c' ``` Fix the discrepancy in behavior across the two platforms by manually stripping Windows newlines before yielding each line in the test file. Reviewers: echristo, beanz, ddunbar, delcypher, rnk Reviewed By: rnk Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D27746 llvm-svn: 309312
* Un-revert "Teach the CMake build system to run lit's test suite. These can ↵Brian Gesiak2017-07-273-2/+50
| | | | | | | | | | | | | | | | | | | | | | be run" Summary: Depends on https://reviews.llvm.org/D35879. This reverts rL257268, which in turn was a revert of rL257221. https://reviews.llvm.org/D35879 marks the tests in the lit test suite that fail on Windows as XFAIL, which should allow these tests to pass on Windows-based buildbots. Reviewers: delcypher, beanz, mgorny, jroelofs, rnk Reviewed By: mgorny Subscribers: rnk, ddunbar, george.karpenkov, llvm-commits Differential Revision: https://reviews.llvm.org/D35880 llvm-svn: 309310
* [lit] Fix order of checks in shtest-shell.py testBrian Gesiak2017-07-272-0/+10
| | | | | | | | | | | | | | | | | | Summary: An expectation in `utils/lit/tests/Inputs/shtest-shell/redirects.txt` expects that first a string printed to stdout is seen, and then a string printed to stderr. Add `flush()` calls to ensure that stdout is printed before stderr, as expected. Reviewers: rnk, mgorny, jroelofs Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35947 llvm-svn: 309292
* [lit] Fix race between shtest-shell and max-failures testsReid Kleckner2017-07-262-4/+10
| | | | | | | Previously these tests would use the same Output directory leading to flaky non-deterministic failures. llvm-svn: 309227
* [lit] Fix shtest-shell and max-failures lit tests on WindowsReid Kleckner2017-07-267-16/+10
| | | | | | | | | Rewrite the write-to-stderr.sh and write-to-stdout-and-stderr.sh shell scripts as python scripts and call python on them. Fixes PR33940 llvm-svn: 309200
* [lit] Fix shtest-output-printing.py on Windows by matching either / or \\Reid Kleckner2017-07-261-4/+1
| | | | | | Fixes PR33938 llvm-svn: 309198
* [lit] Fix discovery.py on Windows by matching backslashes when necessaryReid Kleckner2017-07-261-22/+19
| | | | | | Fixes PR33932 llvm-svn: 309194
* [lit] Un-XFAIL selecting.py test on WindowsReid Kleckner2017-07-261-3/+0
| | | | | | | | This passes locally for me, which fails the overall lit test suite. I can't debug a passing test, but I will try to help debug the test when we get some failing logs. llvm-svn: 309190
* Un-XFAIL some internal lit tests on Windows, they pass for me locallyReid Kleckner2017-07-263-9/+0
| | | | llvm-svn: 309144
* Fix LIT test breakageGeorge Karpenkov2017-07-261-1/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D35867 llvm-svn: 309140
* [lit] Mark several of lit's tests XFAIL on WindowsBrian Gesiak2017-07-2613-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: rL257221 attempted to run lit's own test suite continuously, but that commit was reverted because lit's test suite does not pass on Windows. Because lit's tests do not run continuously, they often regress. In order to un-revert rL257221, mark lit tests that fail as XFAIL for Windows platforms. Test Plan: On a Windows development environment, follow the instructions in utils/lit/README.txt to run lit's test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test suite is run and a successful exit code is returned. Reviewers: mgorny, rnk, delcypher, beanz Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35879 llvm-svn: 309123
* [lit] Fix type error for parallelism groupsBrian Gesiak2017-07-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Whereas rL299560 and rL309071 call `parallelism_groups.items()`, under the assumption that `parallelism_groups` is a `dict` type, the default parameter for that attribute is a `list`. Change the default to a `dict` for type correctness. This regression in the unit tests would have been caught if the unit tests were being run continously. It also would have been caught if the lit project used a Python type checker such as `mypy`. Test Plan: As per the instructions in `utils/lit/README.txt`, run the lit unit test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test `lit :: unit/TestRunner.py` fails before applying this patch, but passes once this patch is applied. Reviewers: mgorny, rnk, rafael Reviewed By: mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35878 llvm-svn: 309122
* Revert "[lit] Remove dead code not referenced in the LLVM SVN repo."Brian Gesiak2017-07-263-74/+123
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This reverts rL306623, which removed `FileBasedTest`, an abstract base class, but did not also remove the usages of that class in the lit unit tests. The revert fixes four test failures in the lit unit test suite. Test plan: As per the instructions in `utils/lit/README.txt`, run the lit unit test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the following tests fail before applying this patch, and pass once the patch is applied: ``` lit :: test-data.py lit :: test-output.py lit :: xunit-output.py ``` In addition, run `check-llvm` to make sure the existing LLVM test suite executes normally. Reviewers: george.karpenkov, mgorny, dlj Reviewed By: mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35877 llvm-svn: 309120
* [lit] Attempt to fix Python unittest adaptor logicReid Kleckner2017-07-261-7/+6
| | | | llvm-svn: 309071
* [lit] Fix UnboundLocalError for invalid shtest redirectsMichal Gorny2017-07-251-1/+1
| | | | | | | | | | | | | | | | | Replace the incorrect variable reference when invalid redirect is used. This fixes the following issue: File "/usr/src/llvm/utils/lit/lit/TestRunner.py", line 316, in processRedirects raise InternalShellError(cmd, "Unsupported redirect: %r" % (r,)) UnboundLocalError: local variable 'r' referenced before assignment which in turn broke shtest-shell.py and max-failures.py lit tests. The breakage was introduced during refactoring in rL307310. Differential Revision: https://reviews.llvm.org/D35857 llvm-svn: 309044
* [lit] add a -vv option to echo all executed commands.George Karpenkov2017-07-133-2/+16
| | | | | | | | | | | | | | | | | | | Debugging LIT scripts can be rather painful, as LIT directly does not specify which line has failed. Rather, FileCheck is expected to report the failing location, but it can be often ambiguous if multiple commands are tested against the same prefix. This change adds a -vv option, which echoes all output. Then detecting the error becomes straightforward: last printed line is the failing one. Of course, it could be desired to try to get failing line number directly from bash, but it involves excessive hacks on older bash versions (cf. https://stackoverflow.com/questions/24398691/how-to-get-the-real-line-number-of-a-failing-bash-command) Differential Revision: https://reviews.llvm.org/D35330 llvm-svn: 307938
* [lit] Fix import StringIO errors in Python 3Reid Kleckner2017-07-111-3/+3
| | | | | | | Remove the cStringIO micro-optimization, as it isn't portable to Python 3. llvm-svn: 307669
* [lit] Implement non-pipelined echo commands internallyReid Kleckner2017-07-111-0/+74
| | | | | | | | | | | | | | | Summary: This speeds up the LLD test suite on Windows by 3x. Most of the time is spent on lld/test/ELF/linkerscript/diagnostics.s, which repeatedly constructs linker scripts with appending echo commands. Reviewers: dlj, zturner, modocache Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35093 llvm-svn: 307668
* [lit] Modify LIT to accept environment variable LIT_FILTER to select tests.George Karpenkov2017-07-072-1/+7
| | | | | | | | | This is especially useful when lit is invoked indirectly by the build system, and additional arguments can not be easily specified. Differential Revision: https://reviews.llvm.org/D35091 llvm-svn: 307339
* [lit] Factor out some shell input/output redirection logic, NFCReid Kleckner2017-07-061-70/+97
| | | | | | | This is a very light refactoring aimed at improving readability. There is definitely still room for improvement here. llvm-svn: 307310
* [lit] Fix unit test discovery for Visual Studio builds.David L. Jones2017-07-061-1/+4
| | | | | | | | | | | | | | | | | | | | Fix by Andrew Ng! The Visual Studio build can contain output for multiple configuration types ( e.g. Debug, Release & RelWithDebInfo) within the same build output directory. Therefore when discovering unit tests, the "build mode" sub directory containing the appropriate configuration is included in the search. This sub directory may not always be present, so a test for its existence is required. Reviewers: zturner, modocache, dlj Reviewed By: zturner, dlj Subscribers: grimar, bd1976llvm, gbreynoo, edd, jhenderson, llvm-commits Differential Revision: https://reviews.llvm.org/D34976 llvm-svn: 307235
* [lit] Factor out listdir logic shared by different test formats.David L. Jones2017-06-303-44/+58
| | | | | | | | | | | | | | | | | | | | Summary: The lit test formats use largely the same logic for discovering tests. There are some superficial differences in the logic, which seem reasonable enough to handle in a single routine. At a high level, the common goal is "look for files that end with one of these suffixes, and skip anything starting with a dot." The balance of the logic specific to ShTest and GoogleTest collapses quite a bit, so that getTestsInDirectory is only a couple of lines around a call to the new function. Reviewers: zturner, MatzeB, modocache Subscribers: sanjoy, llvm-commits Differential Revision: https://reviews.llvm.org/D34855 llvm-svn: 306895
* Revert "[lit] Clean output directories before running tests."Zachary Turner2017-06-303-51/+39
| | | | | | | | | This reverts commit da6318a92fba793e4f2447ec478b001392d57d43. This is causing failures on some build bots due to what appears to be some kind of lit ordering dependency. llvm-svn: 306833
* [lit] Clean output directories before running tests.Zachary Turner2017-06-303-39/+51
| | | | | | | | | | | | | | | | | | Presently lit leaks files in the tests' output directories. Specifically, if a test creates output files, lit makes no effort to remove them prior to the next test run. This is problematic because it leads to false positives whenever a test passes because stale files were present. In general it is a source of flakiness that should be removed. This patch addresses this by building the list of all test directories that are part of the current run set, and then deleting those directories and recreating them anew. This gives each test a clean baseline to start from. Differential Revision: https://reviews.llvm.org/D34732 llvm-svn: 306832
* [lit] Re-apply: Fix some convoluted logic around Unicode encoding, and ↵David L. Jones2017-06-292-51/+69
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | de-duplicate across modules that used it. (Take 2: this patch re-applies r306625, which was reverted in r306629. This patch includes only trivial fixes.) In Python2 and Python3, the various (non-)?Unicode string types are sort of spaghetti. Python2 has unicode support tacked on via the 'unicode' type, which is distinct from 'str' (which are bytes). Python3 takes the "unicode-everywhere" approach, with 'str' representing a Unicode string. Both have a 'bytes' type. In Python3, it is the only way to represent raw bytes. However, in Python2, 'bytes' is an alias for 'str'. This leads to interesting problems when an interface requires a precise type, but has to run under both Python2 and Python3. The previous logic appeared to be correct in all cases, but went through more layers of indirection than necessary. This change does the necessary conversions in one shot, with documentation about which paths might be taken in Python2 or Python3. Changes from r306625: some tests just print binary outputs, so in those cases, fall back to str() in Python3. For googletests, add one missing call to to_string(). (Tested by verifying the visible breakage with Python3. Verified that everything works in py2 and py3.) llvm-svn: 306643
* Revert "[lit] Fix some convoluted logic around Unicode encoding, and ↵David L. Jones2017-06-292-62/+51
| | | | | | | | de-duplicate across modules that used it." This reverts r306625. llvm-svn: 306629
* Fix spelling: uncode -> unicode.David L. Jones2017-06-291-1/+1
| | | | | | | Remember kids: there is no 'I' in str or bytes, but there is ALWAYS an 'I' in unicode. llvm-svn: 306626
* [lit] Fix some convoluted logic around Unicode encoding, and de-duplicate ↵David L. Jones2017-06-292-51/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | across modules that used it. Summary: In Python2 and Python3, the various (non-)?Unicode string types are sort of spaghetti. Python2 has unicode support tacked on via the 'unicode' type, which is distinct from 'str' (which are bytes). Python3 takes the "unicode-everywhere" approach, with 'str' representing a Unicode string. Both have a 'bytes' type. In Python3, it is the only way to represent raw bytes. However, in Python2, 'bytes' is an alias for 'str'. This leads to interesting problems when an interface requires a precise type, but has to run under both Python2 and Python3. The previous logic appeared to be correct in all cases, but went through more layers of indirection than necessary. This change does the necessary conversions in one shot, with documentation about which paths might be taken in Python2 or Python3. Reviewers: zturner, modocache Subscribers: llvm-commits, sanjoy Differential Revision: https://reviews.llvm.org/D34793 llvm-svn: 306625
* [lit] Remove dead code not referenced in the LLVM SVN repo.David L. Jones2017-06-293-120/+90
| | | | | | | | | | | | | | | | | | | | | | | Summary: This change removes the intermediate 'FileBasedTest' format from lit. This format is only ever used by the ShTest format, so the logic can be moved into ShTest directly. In order to better clarify what the TestFormat subclasses do, I fleshed out the TestFormat base class with Python's notion of abstract methods, using @abc.abstractmethod. This gives a convenient way to document the expected interface, without the risk of instantiating an abstract class (that's what ABCMeta does -- it raises an exception if you try to instantiate a class which has abstract methods, but not if you instantiate a subclass that implements them). Reviewers: zturner, modocache Subscribers: sanjoy, llvm-commits Differential Revision: https://reviews.llvm.org/D34792 llvm-svn: 306623
* [lit] Remove dead code (not referenced anywhere), and clarify some function ↵David L. Jones2017-06-281-177/+41
| | | | | | | | | | | | | | | | | | | | | | | names. Summary: The dead code seems to be unreferenced, according to textual search across the LLVM SVN repo. The clarification part of this change alters the name of a module-level function so that it is different from the name of the class-methods that call it. Currently, there are no erroneous references, but stylistically (c.f. PEP-8), internal "helper" functions should generally be named accordingly by prepending an underscore. (I also chose to add '_impl', which isn't necessary, but helps me at least to mentally disambiguate the interface and implementation functions.) Reviewers: zturner, modocache Subscribers: sanjoy, llvm-commits Differential Revision: https://reviews.llvm.org/D34775 llvm-svn: 306600
* [lit][macOS] Add a utility function to find the platform SDK versionAlex Lorenz2017-06-021-0/+14
| | | | | | | | | on macOS This function will be used to tie Clang's Integeration tests to a particular SDK version. See https://reviews.llvm.org/D32178 for more context. llvm-svn: 304541
* Return a lit.Test.Result object from TestRunner's executeShTest()Dimitry Andric2017-05-251-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: For various clang analyzer tests, which were unsupported, I got lit exceptions, similar to the following: Exception during script execution: Traceback (most recent call last): File "utils/lit/lit/run.py", line 190, in execute_test result = test.config.test_format.execute(test, lit_config) File "tools/clang/test/Analysis/analyzer_test.py", line 11, in execute if result.code == lit.Test.FAIL: AttributeError: 'tuple' object has no attribute 'code' This is because executeShTest() in utils/lit/lit/TestRunner.py is supposed to return a lit.Test.Result object, but in case of unsupported tests, it returns a plain tuple. Fix this by returning a properly initialized lit.Test.Result object instead. Reviewers: rnk, rafael, modocache Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D33579 llvm-svn: 303943
* [lit] Take the last error when executing pipelines.Zachary Turner2017-05-191-6/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This seems to have been present since the beginning of time, which is quite surprising. The symptom was this: Suppose you have a test with a run line that looks like this: RUN: foo | FileCheck %s foo prints some output and then due to a bug in the program it asserts. On Windows this results in the program returning a negative exit code. But if enough output had been printed already by the tool so that the FileCheck match would succeed then FileCheck would return 0, and because of bad logic in lit this 0 return value would overwrite the failed return value from previous items in the pipeline. This only happened with negative exit codes. The most sensible behavior is to just take whatever the first exit code is. There is no logical ordering defined on exit codes, so comparing with < and > does not make a lot of sense. Instead, as soon as we find the first non-successful return value, that should be the result of the entire expression. This fixes the issue, as now tests which fail on non-Windows platforms also fail for me on Windows as well. llvm-svn: 303440
* Add back a dummy --use-processes.Rafael Espindola2017-05-171-0/+3
| | | | | | Some bots are using it. llvm-svn: 303282
OpenPOWER on IntegriCloud