[Support] - Add bad alloc error handler for handling allocation malfunctions

Summary:
Patch by Klaus Kretzschmar

We would like to introduce a new type of llvm error handler for handling
bad alloc fault situations.  LLVM already provides a fatal error handler
for serious non-recoverable error situations which by default writes
some error information to stderr and calls exit(1) at the end (functions
are marked as 'noreturn').

For long running processes (e.g. a server application), exiting the
process is not an acceptable option, especially not when the system is
in a temporary resource bottleneck with a good chance to recover from
this fault situation. In such a situation you would rather throw an
exception to stop the current compilation and try to overcome the
resource bottleneck. The user should be aware of the problem of throwing
an exception in bad alloc situations, e.g. you must not do any
allocations in the unwind chain. This is especially true when adding
exceptions in existing unfamiliar code (as already stated in the comment
of the current fatal error handler)

So the new handler can also be used to distinguish from general fatal
error situations where recovering is no option.  It should be used in
cases where a clean unwind after the allocation is guaranteed.

This patch contains:
- A report_bad_alloc function which calls a user defined bad alloc
  error handler. If no user handler is registered the
  report_fatal_error function is called. This function is not marked as
  'noreturn'.
- A install/restore_bad_alloc_error_handler to install/restore the bad
  alloc handler.
- An example (in Mutex.cpp) where the report_bad_alloc function is
  called in case of a malloc returns a nullptr.

If this patch gets accepted we would create similar patches to fix
corresponding malloc/calloc usages in the llvm code.

Reviewers: chandlerc, greened, baldrick, rnk

Reviewed By: rnk

Subscribers: llvm-commits, MatzeB

Differential Revision: https://reviews.llvm.org/D34753

llvm-svn: 307673

1 files changed, 44 insertions, 1 deletions

diff --git a/llvm/lib/Support/ErrorHandling.cpp b/llvm/lib/Support/ErrorHandling.cpp
index a7d3a18003e..20dc46b0bda 100644
--- a/llvm/lib/Support/ErrorHandling.cpp
+++ b/llvm/lib/Support/ErrorHandling.cpp
@@ -29,6 +29,7 @@
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <cstdlib>
+#include <new>
 
 #if defined(HAVE_UNISTD_H)
 # include <unistd.h>
@@ -42,9 +43,12 @@ using namespace llvm;
 
 static fatal_error_handler_t ErrorHandler = nullptr;
 static void *ErrorHandlerUserData = nullptr;
-
 static ManagedStatic<sys::Mutex> ErrorHandlerMutex;
 
+static fatal_error_handler_t BadAllocErrorHandler = nullptr;
+static void *BadAllocErrorHandlerUserData = nullptr;
+static ManagedStatic<sys::Mutex> BadAllocErrorHandlerMutex;
+
 void llvm::install_fatal_error_handler(fatal_error_handler_t handler,
                                        void *user_data) {
   llvm::MutexGuard Lock(*ErrorHandlerMutex);
@@ -104,6 +108,45 @@ void llvm::report_fatal_error(const Twine &Reason, bool GenCrashDiag) {
   exit(1);
 }
 
+void llvm::install_bad_alloc_error_handler(fatal_error_handler_t handler,
+                                           void *user_data) {
+  MutexGuard Lock(*BadAllocErrorHandlerMutex);
+  assert(!ErrorHandler && "Bad alloc error handler already registered!\n");
+  BadAllocErrorHandler = handler;
+  BadAllocErrorHandlerUserData = user_data;
+}
+
+void llvm::remove_bad_alloc_error_handler() {
+  MutexGuard Lock(*BadAllocErrorHandlerMutex);
+  BadAllocErrorHandler = nullptr;
+  BadAllocErrorHandlerUserData = nullptr;
+}
+
+void llvm::report_bad_alloc_error(const char *Reason, bool GenCrashDiag) {
+  fatal_error_handler_t Handler = nullptr;
+  void *HandlerData = nullptr;
+  {
+    // Only acquire the mutex while reading the handler, so as not to invoke a
+    // user-supplied callback under a lock.
+    MutexGuard Lock(*BadAllocErrorHandlerMutex);
+    Handler = BadAllocErrorHandler;
+    HandlerData = BadAllocErrorHandlerUserData;
+  }
+
+  if (Handler) {
+    Handler(HandlerData, Reason, GenCrashDiag);
+    llvm_unreachable("bad alloc handler should not return");
+  }
+
+#ifdef LLVM_ENABLE_EXCEPTIONS
+  // If exceptions are enabled, make OOM in malloc look like OOM in new.
+  throw std::bad_alloc();
+#else
+  // Otherwise, fall back to the normal fatal error handler.
+  report_fatal_error("out of memory: " + Twine(Reason));
+#endif
+}
+
 void llvm::llvm_unreachable_internal(const char *msg, const char *file,
                                      unsigned line) {
   // This code intentionally doesn't call the ErrorHandler callback, because