Add ability to import std module into expression parser to improve C++ debugging

Summary:
This patch is the MVP version of importing the std module into the expression parser to improve C++ debugging.

What happens in this patch is that we inject a `@import std` into our expression source code. We also
modify our internal Clang instance for parsing this expression to work with modules and debug info
at the same time (which is the main change in terms of LOC). We implicitly build the `std` module on the first use. The
C++ include paths for building are extracted from the debug info, which means that this currently only
works if the program is compiled with `-glldb -fmodules` and uses the std module. The C include paths
are currently specified by LLDB.

I enabled the tests currently only for libc++ and Linux because I could test this locally. I'll enable the tests
for other platforms once this has landed and doesn't break any bots (and I implemented the platform-specific
C include paths for them).

With this patch we can now:
* Build a libc++ as a module and import it into the expression parser.
* Read from the module while also referencing declarations from the debug info. E.g. `std::abs(local_variable)`.

What doesn't work (yet):
* Merging debug info and C++ module declarations. E.g. `std::vector<CustomClass>` doesn't work.
* Pretty much anything that involves the ASTImporter and templated code. As the ASTImporter is used for saving the result declaration, this means that we can't
call yet any function that returns a non-trivial type.
* Use libstdc++ for this, as it requires multiple include paths and Clang only emits one include path per module. Also libstdc++ doesn't support Clang modules without patches.

Reviewers: aprantl, jingham, shafik, friss, davide, serge-sans-paille

Reviewed By: aprantl

Subscribers: labath, mgorny, abidh, jdoerfert, lldb-commits

Tags: #c_modules_in_lldb, #lldb

Differential Revision: https://reviews.llvm.org/D58125

llvm-svn: 355939

16 files changed, 241 insertions, 2 deletions

diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/basic/Makefile b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/basic/Makefile
new file mode 100644
index 00000000000..01718d86aa7
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/basic/Makefile
@@ -0,0 +1,5 @@
+LEVEL = ../../../make
+USE_LIBCPP := 1
+CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
+CXX_SOURCES := main.cpp
+include $(LEVEL)/Makefile.rules
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/basic/TestImportStdModule.py b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/basic/TestImportStdModule.py
new file mode 100644
index 00000000000..2b87c2e5711
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/basic/TestImportStdModule.py
@@ -0,0 +1,56 @@
+"""
+Test importing the 'std' C++ module and evaluate expressions with it.
+"""
+
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+
+class ImportStdModule(TestBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    # FIXME: This should work on more setups, so remove these
+    # skipIf's in the future.
+    @add_test_categories(["libc++"])
+    @skipIf(compiler=no_match("clang"))
+    @skipIf(oslist=no_match(["linux"]))
+    @skipIf(debug_info=no_match(["dwarf"]))
+    def test(self):
+        self.build()
+
+        lldbutil.run_to_source_breakpoint(self,
+            "// Set break point at this line.", lldb.SBFileSpec("main.cpp"))
+
+        # Activate importing of std module.
+        self.runCmd("settings set target.import-std-module true")
+        # Calling some normal std functions that return non-template types.
+        self.expect("expr std::abs(-42)", substrs=['(int) $0 = 42'])
+        self.expect("expr std::div(2, 1).quot", substrs=['(int) $1 = 2'])
+        # Using types from std.
+        self.expect("expr (std::size_t)33U", substrs=['(size_t) $2 = 33'])
+        # Calling templated functions that return non-template types.
+        self.expect("expr char a = 'b'; char b = 'a'; std::swap(a, b); a",
+                    substrs=["(char) $3 = 'a'"])
+
+    # FIXME: This should work on more setups, so remove these
+    # skipIf's in the future.
+    @add_test_categories(["libc++"])
+    @skipIf(compiler=no_match("clang"))
+    @skipIf(oslist=no_match(["linux"]))
+    @skipIf(debug_info=no_match(["dwarf"]))
+    def test_non_cpp_language(self):
+        self.build()
+
+        lldbutil.run_to_source_breakpoint(self,
+            "// Set break point at this line.", lldb.SBFileSpec("main.cpp"))
+
+        # Activate importing of std module.
+        self.runCmd("settings set target.import-std-module true")
+        # These languages don't support C++ modules, so they shouldn't
+        # be able to evaluate the expression.
+        self.expect("expr -l C -- std::abs(-42)", error=True)
+        self.expect("expr -l C99 -- std::abs(-42)", error=True)
+        self.expect("expr -l C11 -- std::abs(-42)", error=True)
+        self.expect("expr -l ObjC -- std::abs(-42)", error=True)
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/basic/main.cpp b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/basic/main.cpp
new file mode 100644
index 00000000000..2f6d078daa3
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/basic/main.cpp
@@ -0,0 +1,7 @@
+// We need to import any std module. It doesn't matter which one.
+#include <iostream>
+
+int main(int argc, char **argv) {
+  std::cout << "Test" << std::endl;
+  return 0; // Set break point at this line.
+}
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/conflicts/Makefile b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/conflicts/Makefile
new file mode 100644
index 00000000000..01718d86aa7
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/conflicts/Makefile
@@ -0,0 +1,5 @@
+LEVEL = ../../../make
+USE_LIBCPP := 1
+CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
+CXX_SOURCES := main.cpp
+include $(LEVEL)/Makefile.rules
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/conflicts/TestStdModuleWithConflicts.py b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/conflicts/TestStdModuleWithConflicts.py
new file mode 100644
index 00000000000..8be474e39ce
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/conflicts/TestStdModuleWithConflicts.py
@@ -0,0 +1,36 @@
+"""
+Test importing the 'std' C++ module and check if we can handle
+prioritizing the conflicting functions from debug info and std
+module.
+
+See also import-std-module/basic/TestImportStdModule.py for
+the same test on a 'clean' code base without conflicts.
+"""
+
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+
+class TestImportStdModuleConflicts(TestBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    # FIXME: This should work on more setups, so remove these
+    # skipIf's in the future.
+    @add_test_categories(["libc++"])
+    @skipIf(compiler=no_match("clang"))
+    @skipIf(oslist=no_match(["linux"]))
+    @skipIf(debug_info=no_match(["dwarf"]))
+    def test(self):
+        self.build()
+
+        lldbutil.run_to_source_breakpoint(self,
+            "// Set break point at this line.", lldb.SBFileSpec("main.cpp"))
+
+        self.runCmd("settings set target.import-std-module true")
+        self.expect("expr std::abs(-42)", substrs=['(int) $0 = 42'])
+        self.expect("expr std::div(2, 1).quot", substrs=['(int) $1 = 2'])
+        self.expect("expr (std::size_t)33U", substrs=['(size_t) $2 = 33'])
+        self.expect("expr char a = 'b'; char b = 'a'; std::swap(a, b); a",
+                    substrs=["(char) $3 = 'a'"])
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/conflicts/main.cpp b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/conflicts/main.cpp
new file mode 100644
index 00000000000..e49b862a36c
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/conflicts/main.cpp
@@ -0,0 +1,10 @@
+#include <cstdlib>
+#include <utility>
+
+int main(int argc, char **argv) {
+  std::size_t f = argc;
+  f = std::abs(argc);
+  f = std::div(argc * 2, argc).quot;
+  std::swap(f, f);
+  return f; // Set break point at this line.
+}
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/no-std-module/Makefile b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/no-std-module/Makefile
new file mode 100644
index 00000000000..01718d86aa7
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/no-std-module/Makefile
@@ -0,0 +1,5 @@
+LEVEL = ../../../make
+USE_LIBCPP := 1
+CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
+CXX_SOURCES := main.cpp
+include $(LEVEL)/Makefile.rules
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/no-std-module/TestMissingStdModule.py b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/no-std-module/TestMissingStdModule.py
new file mode 100644
index 00000000000..f4d3d1fd492
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/no-std-module/TestMissingStdModule.py
@@ -0,0 +1,40 @@
+"""
+Test that importing the std module on a compile unit
+that doesn't use the std module will not break LLDB.
+
+It's not really specified at the moment what kind of
+error we should report back to the user in this
+situation. Currently Clang will just complain that
+the std module doesn't exist or can't be loaded.
+"""
+
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+
+class STLTestCase(TestBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    # FIXME: This should work on more setups, so remove these
+    # skipIf's in the future.
+    @add_test_categories(["libc++"])
+    @skipIf(compiler=no_match("clang"))
+    @skipIf(oslist=no_match(["linux"]))
+    @skipIf(debug_info=no_match(["dwarf"]))
+    def test(self):
+        self.build()
+
+        lldbutil.run_to_source_breakpoint(self,
+            "// Set break point at this line.", lldb.SBFileSpec("main.cpp"))
+
+        # Activate importing of std module.
+        self.runCmd("settings set target.import-std-module true")
+
+        # Run some commands that should all fail without our std module.
+        self.expect("expr std::abs(-42)", error=True)
+        self.expect("expr std::div(2, 1).quot", error=True)
+        self.expect("expr (std::size_t)33U", error=True)
+        self.expect("expr char a = 'b'; char b = 'a'; std::swap(a, b); a",
+                    error=True)
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/no-std-module/main.cpp b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/no-std-module/main.cpp
new file mode 100644
index 00000000000..c93e9d0ec8a
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/no-std-module/main.cpp
@@ -0,0 +1,5 @@
+// We don't import any std module here.
+
+int main(int argc, char **argv) {
+  return 0; // Set break point at this line.
+}
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/Makefile b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/Makefile
new file mode 100644
index 00000000000..e04d73ea487
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/Makefile
@@ -0,0 +1,10 @@
+LEVEL = ../../../make
+# We don't have any standard include directories, so we can't
+# parse the test_common.h header we usually inject as it includes
+# system headers.
+NO_TEST_COMMON_H := 1
+
+CXXFLAGS += $(MANDATORY_CXXMODULE_BUILD_CFLAGS)
+CXXFLAGS += -I $(SRCDIR)/root/usr/include/c++/include/ -I $(SRCDIR)/root/usr/include/ -nostdinc -nostdinc++ -nostdlib++
+CXX_SOURCES := main.cpp
+include $(LEVEL)/Makefile.rules
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/TestStdModuleSysroot.py b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/TestStdModuleSysroot.py
new file mode 100644
index 00000000000..dfd90e80e45
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/TestStdModuleSysroot.py
@@ -0,0 +1,34 @@
+"""
+Test that we respect the sysroot when building the std module.
+"""
+
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+import os
+
+class ImportStdModule(TestBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    # FIXME: This should work on more setups, so remove these
+    # skipIf's in the future.
+    @skipIf(compiler=no_match("clang"))
+    @skipIf(oslist=no_match(["linux"]))
+    @skipIf(debug_info=no_match(["dwarf"]))
+    def test(self):
+        self.build()
+
+        sysroot = os.path.join(os.getcwd(), "root")
+
+        # Set the sysroot.
+        self.runCmd("platform select --sysroot '" + sysroot + "' host", CURRENT_EXECUTABLE_SET)
+
+        lldbutil.run_to_source_breakpoint(self,
+            "// Set break point at this line.", lldb.SBFileSpec("main.cpp"))
+
+        self.runCmd("settings set target.import-std-module true")
+
+        # Call our custom function in our sysroot std module.
+        # If this gives us the correct result, then we used the sysroot.
+        self.expect("expr std::myabs(-42)", substrs=['(int) $0 = 42'])
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/main.cpp b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/main.cpp
new file mode 100644
index 00000000000..2fbc76b9a76
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/main.cpp
@@ -0,0 +1,6 @@
+#include <algorithm>
+
+int main(int argc, char **argv) {
+  libc_struct s;
+  return 0; // Set break point at this line.
+}
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/root/usr/include/c++/include/algorithm b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/root/usr/include/c++/include/algorithm
new file mode 100644
index 00000000000..e8cbcca8e84
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/root/usr/include/c++/include/algorithm
@@ -0,0 +1,7 @@
+#include "libc_header.h"
+
+namespace std {
+  int myabs(int i) {
+    return i < 0 ? -i : i;
+  }
+}
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/root/usr/include/c++/include/module.modulemap b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/root/usr/include/c++/include/module.modulemap
new file mode 100644
index 00000000000..0eb48492a65
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/root/usr/include/c++/include/module.modulemap
@@ -0,0 +1,3 @@
+module std {
+  module "algorithm" { header "algorithm" export * }
+}
diff --git a/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/root/usr/include/libc_header.h b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/root/usr/include/libc_header.h
new file mode 100644
index 00000000000..47525c9db34
--- /dev/null
+++ b/lldb/packages/Python/lldbsuite/test/expression_command/import-std-module/sysroot/root/usr/include/libc_header.h
@@ -0,0 +1 @@
+struct libc_struct {};
diff --git a/lldb/packages/Python/lldbsuite/test/make/Makefile.rules b/lldb/packages/Python/lldbsuite/test/make/Makefile.rules
index fc72bceb195..9cf5d9ddba2 100644
--- a/lldb/packages/Python/lldbsuite/test/make/Makefile.rules
+++ b/lldb/packages/Python/lldbsuite/test/make/Makefile.rules
@@ -272,7 +272,12 @@ else
 	CFLAGS += $(ARCHFLAG)$(ARCH) $(FRAMEWORK_INCLUDES) -I$(LLDB_BASE_DIR)include
 endif
 
-CFLAGS += -I$(SRCDIR) -include $(THIS_FILE_DIR)test_common.h -I$(THIS_FILE_DIR)
+CFLAGS += -I$(SRCDIR) -I$(THIS_FILE_DIR)
+
+ifndef NO_TEST_COMMON_H
+  CFLAGS += -include $(THIS_FILE_DIR)test_common.h
+endif
+
 CFLAGS += $(NO_LIMIT_DEBUG_INFO_FLAGS) $(ARCH_CFLAGS) $(CFLAGS_EXTRAS)
 
 # If the OS is Windows, we need to pass -gdwarf to clang, otherwise it will build
@@ -300,7 +305,11 @@ CLANG_MODULE_CACHE_DIR := $(BUILDDIR)/module-cache
 $(warning failed to set the shared clang module cache dir)
 endif
 
-MANDATORY_MODULE_BUILD_CFLAGS := -fmodules -gmodules -fmodules-cache-path=$(CLANG_MODULE_CACHE_DIR)
+MODULE_BASE_FLAGS := -fmodules -gmodules -fmodules-cache-path=$(CLANG_MODULE_CACHE_DIR)
+MANDATORY_MODULE_BUILD_CFLAGS := $(MODULE_BASE_FLAGS) -gmodules
+# Build flags for building with C++ modules.
+# -glldb is necessary for emitting information about what modules were imported.
+MANDATORY_CXXMODULE_BUILD_CFLAGS := $(MODULE_BASE_FLAGS) -fcxx-modules -glldb
 
 ifeq "$(OS)" "Darwin"
 	MANDATORY_MODULE_BUILD_CFLAGS += -fcxx-modules