[GlobalOpt] Don't apply fastcc if it would break inalloca invariants

The inalloca parameter has to be the only parameter passed in memory.
Changing the convention to fastcc can break that.

At some point we should teach global opt how to optimize ABI attributes
like inalloca and maybe byval. These attributes are mainly used to match
C ABIs. They are harder for LLVM to optimize and they don't always
generate the best code.

Fixes PR38487

llvm-svn: 339360

2 files changed, 16 insertions, 0 deletions

diff --git a/llvm/lib/Transforms/IPO/GlobalOpt.cpp b/llvm/lib/Transforms/IPO/GlobalOpt.cpp
index 1761d7faff5..5518ef8fce9 100644
--- a/llvm/lib/Transforms/IPO/GlobalOpt.cpp
+++ b/llvm/lib/Transforms/IPO/GlobalOpt.cpp
@@ -2107,6 +2107,13 @@ static bool hasChangeableCC(Function *F) {
   if (CC != CallingConv::C && CC != CallingConv::X86_ThisCall)
     return false;
 
+  // Don't break the invariant that the inalloca parameter is the only parameter
+  // passed in memory.
+  // FIXME: GlobalOpt should remove inalloca when possible and hoist the dynamic
+  // alloca it uses to the entry block if possible.
+  if (F->getAttributes().hasAttrSomewhere(Attribute::InAlloca))
+    return false;
+
   // FIXME: Change CC for the whole chain of musttail calls when possible.
   //
   // Can't change CC of the function that either has musttail calls, or is a
diff --git a/llvm/test/Transforms/GlobalOpt/fastcc.ll b/llvm/test/Transforms/GlobalOpt/fastcc.ll
index 26398d2bb90..64c4a268ff3 100644
--- a/llvm/test/Transforms/GlobalOpt/fastcc.ll
+++ b/llvm/test/Transforms/GlobalOpt/fastcc.ll
@@ -26,12 +26,20 @@ define internal i32 @j(i32* %m) {
   ret i32 %v
 }
 
+define internal i32 @inalloca(i32* inalloca %p) {
+; CHECK-LABEL: define internal i32 @inalloca(i32* inalloca %p)
+  %rv = load i32, i32* %p
+  ret i32 %rv
+}
+
 define void @call_things() {
   %m = alloca i32
   call i32 @f(i32* %m)
   call x86_thiscallcc i32 @g(i32* %m)
   call coldcc i32 @h(i32* %m)
   call i32 @j(i32* %m)
+  %args = alloca inalloca i32
+  call i32 @inalloca(i32* inalloca %args)
   ret void
 }
 
@@ -44,3 +52,4 @@ define void @call_things() {
 ; CHECK: call fastcc i32 @g
 ; CHECK: call coldcc i32 @h
 ; CHECK: call i32 @j
+; CHECK: call i32 @inalloca(i32* inalloca %args)