[LV] Move insertelement sequence after scalar definitions

After r279649 when getting a vector value from VectorLoopValueMap, we create an
insertelement sequence on-demand if the value has been scalarized instead of
vectorized. We previously inserted this insertelement sequence before the
value's first vector user. However, this insert location is problematic if that
user is the phi node of a first-order recurrence. With this patch, we move the
insertelement sequence after the last scalar instruction we created when
scalarizing the value. Thus, the value's vector definition in the new loop will
immediately follow its scalar definitions. This should fix PR30183.

Reference: https://llvm.org/bugs/show_bug.cgi?id=30183
llvm-svn: 280001

1 files changed, 11 insertions, 0 deletions

diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 00a30b704cd..62b1339138f 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -2377,6 +2377,16 @@ InnerLoopVectorizer::getVectorValue(Value *V) {
       return VectorLoopValueMap.initVector(V, Entry);
     }
 
+    // Get the last scalarized instruction. This corresponds to the instruction
+    // we created for the last vector lane on the last unroll iteration.
+    auto *LastInst = cast<Instruction>(getScalarValue(V, UF - 1, VF - 1));
+
+    // Set the insert point after the last scalarized instruction. This ensures
+    // the insertelement sequence will directly follow the scalar definitions.
+    auto OldIP = Builder.saveIP();
+    auto NewIP = std::next(BasicBlock::iterator(LastInst));
+    Builder.SetInsertPoint(&*NewIP);
+
     // However, if we are vectorizing, we need to construct the vector values
     // using insertelement instructions. Since the resulting vectors are stored
     // in VectorLoopValueMap, we will only generate the insertelements once.
@@ -2387,6 +2397,7 @@ InnerLoopVectorizer::getVectorValue(Value *V) {
             Insert, getScalarValue(V, Part, Width), Builder.getInt32(Width));
       Entry[Part] = Insert;
     }
+    Builder.restoreIP(OldIP);
     return VectorLoopValueMap.initVector(V, Entry);
   }