[lit] Better/earlier errors for empty runs

Fail early, when we discover no tests at all, or filter out all of them.

There is also `--allow-empty-runs` to disable test to allow workflows
like `LIT_FILTER=abc ninja check-all`.  Apparently `check-all` invokes
lit multiple times if certain projects are enabled, which would produce
unwanted "empty runs". Specify via `LIT_OPTS=--allow-empty-runs`.

There are 3 causes for empty runs:
1) No tests discovered.  This is always an error.  Fix test suite config
   or command line.
2) All tests filtered out.  This is an error by default, but can be
   suppressed via `--alow-empty-runs`.  Should prevent accidentally
   passing empty runs, but allow the workflow above.
3) The number of shards is greater than the number of tests.  Currently,
   this is never an error.  Personally, I think we should consider
   making this an error by default; if this happens, you are doing
   something wrong. I added a warning but did not change the behavior,
   since this warrants more discussion.

Reviewed By: atrick, jdenny

Differential Revision: https://reviews.llvm.org/D70105

1 files changed, 18 insertions, 4 deletions

diff --git a/llvm/utils/lit/tests/selecting.py b/llvm/utils/lit/tests/selecting.py
index 0921cdd31ac..638c1088a26 100644
--- a/llvm/utils/lit/tests/selecting.py
+++ b/llvm/utils/lit/tests/selecting.py
@@ -1,6 +1,21 @@
 # RUN: %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-BASIC %s
 # CHECK-BASIC: Testing: 5 tests
 
+
+# Check that we exit with an error if we do not discover any tests, even with --allow-empty-runs.
+#
+# RUN: not %{lit} %{inputs}/nonexistent                    2>&1 | FileCheck --check-prefix=CHECK-BAD-PATH %s
+# RUN: not %{lit} %{inputs}/nonexistent --allow-empty-runs 2>&1 | FileCheck --check-prefix=CHECK-BAD-PATH %s
+# CHECK-BAD-PATH: Did not disover any tests for provided path(s).
+
+# Check that we exit with an error if we filter out all tests, but allow it with --allow-empty-runs.
+#
+# RUN: not %{lit} --filter 'nonexistent'                    %{inputs}/discovery 2>&1 | FileCheck --check-prefixes=CHECK-BAD-FILTER,CHECK-BAD-FILTER-ERROR %s
+# RUN:     %{lit} --filter 'nonexistent' --allow-empty-runs %{inputs}/discovery 2>&1 | FileCheck --check-prefixes=CHECK-BAD-FILTER,CHECK-BAD-FILTER-ALLOW %s
+# CHECK-BAD-FILTER: Filter did not match any tests (of 5 discovered).
+# CHECK-BAD-FILTER-ERROR: Use '--allow-empty-runs' to suppress this error.
+# CHECK-BAD-FILTER-ALLOW: Suppressing error because '--allow-empty-runs' was specified.
+
 # Check that regex-filtering works, is case-insensitive, and can be configured via env var.
 #
 # RUN: %{lit} --filter 'o[a-z]e' %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
@@ -8,6 +23,7 @@
 # RUN: env LIT_FILTER='o[a-z]e' %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
 # CHECK-FILTER: Testing: 2 of 5 tests
 
+
 # Check that maximum counts work
 #
 # RUN: %{lit} --max-tests 3 %{inputs}/discovery | FileCheck --check-prefix=CHECK-MAX %s
@@ -68,15 +84,13 @@
 #
 # RUN: %{lit} --num-shards 100 --run-shard 6 %{inputs}/discovery >%t.out 2>%t.err
 # RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-ERR2 < %t.err %s
-# RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-OUT2 < %t.out %s
 # CHECK-SHARD-BIG-ERR2: note: Selecting shard 6/100 = size 0/5 = tests #(100*k)+6 = []
-# CHECK-SHARD-BIG-OUT2: Testing: 0 of 5 tests
+# CHECK-SHARD-BIG-ERR2: Shard does not contain any tests.  Consider decreasing the number of shards.
 #
 # RUN: %{lit} --num-shards 100 --run-shard 50 %{inputs}/discovery >%t.out 2>%t.err
 # RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-ERR3 < %t.err %s
-# RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-OUT3 < %t.out %s
 # CHECK-SHARD-BIG-ERR3: note: Selecting shard 50/100 = size 0/5 = tests #(100*k)+50 = []
-# CHECK-SHARD-BIG-OUT3: Testing: 0 of 5 tests
+# CHECK-SHARD-BIG-ERR3: Shard does not contain any tests.  Consider decreasing the number of shards.
 
 
 # Check that range constraints are enforced