summaryrefslogtreecommitdiffstats
path: root/llvm/test/Transforms/SLPVectorizer/X86
Commit message (Collapse)AuthorAgeFilesLines
...
* [SLP] Fix for PR32086: Count InsertElementInstr of the same elements as shuffle.Alexey Bataev2018-01-233-24/+24
| | | | | | | | | | | | | | | | | Summary: If the same value is going to be vectorized several times in the same tree entry, this entry is considered to be a gather entry and cost of this gather is counter as cost of InsertElementInstrs for each gathered value. But we can consider these elements as ShuffleInstr with SK_PermuteSingle shuffle kind. Reviewers: spatel, RKSimon, mkuper, hfinkel Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D38697 llvm-svn: 323246
* [SLP] Fix vectorization for tree with trunc to minimum required bit width.Alexey Bataev2018-01-192-19/+13
| | | | | | | | | | | | | | | | | Summary: If the vectorized tree has truncate to minimum required bit width and the vector type of the cast operation after the truncation is the same as the vector type of the cast operands, count cost of the vector cast operation as 0, because this cast will be later removed. Also, if the vectorization tree root operations are integer cast operations, do not consider them as candidates for truncation. It will just create extra number of the same vector/scalar operations, which will be removed by instcombiner. Reviewers: RKSimon, spatel, mkuper, hfinkel, mssimpso Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D41948 llvm-svn: 322946
* [SLP] Fix test checks, NFC.Alexey Bataev2018-01-181-12/+22
| | | | llvm-svn: 322865
* [SLP] Fix for PR32164: Improve vectorization of reverse order of extract ↵Alexey Bataev2018-01-161-20/+16
| | | | | | | | | | | | | | operations. Summary: Sometimes vectorization of insertelement instructions with extractelement operands may produce an extra shuffle operation, if these operands are in the reverse order. Patch tries to improve this situation by the reordering of the operands to remove this extra shuffle operation. Reviewers: mkuper, hfinkel, RKSimon, spatel Subscribers: mzolotukhin, llvm-commits Differential Revision: https://reviews.llvm.org/D33954 llvm-svn: 322579
* [SLP] Add/update tests for SLP vectorizer, NFC.Alexey Bataev2018-01-102-21/+69
| | | | llvm-svn: 322225
* [COST]Fix PR35865: Fix cost model evaluation for shuffle on X86.Alexey Bataev2018-01-091-0/+27
| | | | | | | | | | | | | | Summary: If the vector type is transformed to non-vector single type, the compile may crash trying to get vector information about non-vector type. Reviewers: RKSimon, spatel, mkuper, hfinkel Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D41862 llvm-svn: 322106
* [SLP] Fix PR35777: Incorrect handling of aggregate values.Alexey Bataev2018-01-084-38/+84
| | | | | | | | | | | | | | | | Summary: Fixes the bug with incorrect handling of InsertValue|InsertElement instrucions in SLP vectorizer. Currently, we may use incorrect ExtractElement instructions as the operands of the original InsertValue|InsertElement instructions. Reviewers: mkuper, hfinkel, RKSimon, spatel Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D41767 llvm-svn: 321994
* [SLP] Fix PR35628: Count external uses on extra reduction arguments.Alexey Bataev2018-01-082-0/+138
| | | | | | | | | | | | | | Summary: If the vectorized value is marked as extra reduction argument, its users are not considered as external users. Patch fixes this. Reviewers: mkuper, hfinkel, RKSimon, spatel Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D41786 llvm-svn: 321993
* [SLPVectorizer] Reintroduce std::stable_sort(properlyDominates()).Davide Italiano2018-01-071-153/+0
| | | | | | | | The approach was never discussed, I wasn't able to reproduce this non-determinism, and the original author went AWOL. After a discussion on the ML, Philip suggested to revert this. llvm-svn: 321974
* [SLP] Update more test checks, NFC.Alexey Bataev2018-01-052-111/+143
| | | | llvm-svn: 321872
* [SLP] Update test checks, NFC.Alexey Bataev2018-01-051-22/+140
| | | | llvm-svn: 321870
* [SLP] Update tests checks, NFC.Alexey Bataev2018-01-051-5/+9
| | | | llvm-svn: 321869
* Revert r320548:[SLP] Vectorize jumbled memory loadsMohammad Shahid2017-12-205-384/+52
| | | | llvm-svn: 321181
* [SLPVectorizer] Don't ignore scalar extraction instructions of aggregate valueGuozhi Wei2017-12-141-0/+19
| | | | | | | | | In SLPVectorizer, the vector build instructions (insertvalue for aggregate type) is passed to BoUpSLP.buildTree, it is treated as UserIgnoreList, so later in cost estimation, the cost of these instructions are not counted. For aggregate value, later usage are more likely to be done in scalar registers, either used as individual scalars or used as a whole for function call or return value. Ignore scalar extraction instructions may cause too aggressive vectorization for aggregate values, and slow down performance. So for vectorization of aggregate value, the scalar extraction instructions are required in cost estimation. Differential Revision: https://reviews.llvm.org/D41139 llvm-svn: 320736
* [SLP] Vectorize jumbled memory loads.Mohammad Shahid2017-12-135-52/+384
| | | | | | | | | | | | | | | | | | | | | | Summary: This patch tries to vectorize loads of consecutive memory accesses, accessed in non-consecutive or jumbled way. An earlier attempt was made with patch D26905 which was reverted back due to some basic issue with representing the 'use mask' of jumbled accesses. This patch fixes the mask representation by recording the 'use mask' in the usertree entry. Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df Reviewers: mkuper, loladiro, Ayal, zvi, danielcdh Reviewed By: Ayal Subscribers: mgrang, dcaballe, hans, mzolotukhin Differential Revision: https://reviews.llvm.org/D36130 llvm-svn: 320548
* Revert r319531 "[SLPVectorizer] Failure to beneficially vectorize 'copyable' ↵Hans Wennborg2017-12-012-95/+132
| | | | | | | | | | | | | | | | | | | | | | | | | | | | elements in integer binary ops." It causes builds to fail with "Instruction does not dominate all uses" (PR35497). > Patch tries to improve vectorization of the following code: > > void add1(int * __restrict dst, const int * __restrict src) { > *dst++ = *src++; > *dst++ = *src++ + 1; > *dst++ = *src++ + 2; > *dst++ = *src++ + 3; > } > Allows to vectorize even if the very first operation is not a binary add, but just a load. > > Fixed issues related to previous commit. > > Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev > > Reviewed By: ABataev, RKSimon > > Subscribers: llvm-commits, RKSimon > > Differential Revision: https://reviews.llvm.org/D28907 llvm-svn: 319550
* [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in ↵Dinar Temirbulatov2017-12-012-132/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | integer binary ops. Patch tries to improve vectorization of the following code: void add1(int * __restrict dst, const int * __restrict src) { *dst++ = *src++; *dst++ = *src++ + 1; *dst++ = *src++ + 2; *dst++ = *src++ + 3; } Allows to vectorize even if the very first operation is not a binary add, but just a load. Fixed issues related to previous commit. Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev Reviewed By: ABataev, RKSimon Subscribers: llvm-commits, RKSimon Differential Revision: https://reviews.llvm.org/D28907 llvm-svn: 319531
* SLPVectorizer.cpp: Avoid std::stable_sort(properlyDominates()).NAKAMURA Takumi2017-11-211-0/+153
| | | | | | | properlyDominates() shouldn't be used as sort key. It causes different output between stdlibc++ and libc++. Instead, I introduced RPOT. In most cases, it works for CSE. llvm-svn: 318743
* [SLP] Added more missed optimization remarksAdam Nemet2017-11-154-0/+215
| | | | | | | | | | | | | | | | | | | | Summary: Added more remarks to SLP pass, in particular "missed" optimization remarks. Also proposed several tests for new functionality. Patch by Vladimir Miloserdov! For reference you may look at: https://reviews.llvm.org/rL302811 Reviewers: anemet, fhahn Reviewed By: anemet Subscribers: javed.absar, lattner, petecoup, yakush, llvm-commits Differential Revision: https://reviews.llvm.org/D38367 llvm-svn: 318307
* Revert r318193 "[SLPVectorizer] Failure to beneficially vectorize 'copyable' ↵Hans Wennborg2017-11-151-68/+132
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | elements in integer binary ops." It crashes building sqlite; see reply on the llvm-commits thread. > [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in integer binary ops. > > Patch tries to improve vectorization of the following code: > > void add1(int * __restrict dst, const int * __restrict src) { > *dst++ = *src++; > *dst++ = *src++ + 1; > *dst++ = *src++ + 2; > *dst++ = *src++ + 3; > } > Allows to vectorize even if the very first operation is not a binary add, but just a load. > > Fixed issues related to previous commit. > > Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev > > Reviewed By: ABataev, RKSimon > > Subscribers: llvm-commits, RKSimon > > Differential Revision: https://reviews.llvm.org/D28907 llvm-svn: 318239
* [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in ↵Dinar Temirbulatov2017-11-141-132/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | | integer binary ops. Patch tries to improve vectorization of the following code: void add1(int * __restrict dst, const int * __restrict src) { *dst++ = *src++; *dst++ = *src++ + 1; *dst++ = *src++ + 2; *dst++ = *src++ + 3; } Allows to vectorize even if the very first operation is not a binary add, but just a load. Fixed issues related to previous commit. Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev Reviewed By: ABataev, RKSimon Subscribers: llvm-commits, RKSimon Differential Revision: https://reviews.llvm.org/D28907 llvm-svn: 318193
* [SLP] Fix PR23510: Try to find best possible vectorizable stores.Alexey Bataev2017-11-091-15/+10
| | | | | | | | | | | | | | | | | | Summary: The analysis of the store sequence goes in straight order - from the first store to the last. Bu the best opportunity for vectorization will happen if we're going to use reverse order - from last store to the first. It may be best because usually users have some initialization part + further processing and this first initialization may confuse SLP vectorizer. Reviewers: RKSimon, hfinkel, mkuper, spatel Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39606 llvm-svn: 317821
* Revert rL317618Mitch Phillips2017-11-081-68/+132
| | | | | | | | | | The implemented pass fails and is breaking a large number of unit tests. Example: http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/5777/steps/build-stage3-compiler/logs/stdio This reverts commit rL317618 llvm-svn: 317641
* [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in ↵Dinar Temirbulatov2017-11-071-132/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | integer binary ops. Patch tries to improve vectorization of the following code: void add1(int * __restrict dst, const int * __restrict src) { *dst++ = *src++; *dst++ = *src++ + 1; *dst++ = *src++ + 2; *dst++ = *src++ + 3; } Allows to vectorize even if the very first operation is not a binary add, but just a load. Fixed PR34619 and other issues related to previous commit. Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev Reviewed By: ABataev, RKSimon Subscribers: llvm-commits, RKSimon Differential Revision: https://reviews.llvm.org/D28907 llvm-svn: 317618
* [SLP] Fix PR35047: Fix default cost model for cast op in X86.Alexey Bataev2017-11-072-38/+29
| | | | | | | | | | | | | | | Summary: The cost calculation for default case on X86 target does not always follow correct wayt because of missing 4-th argument in `BaseT::getCastInstrCost()` call. Added this missing parameter. Reviewers: hfinkel, mkuper, RKSimon, spatel Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39687 llvm-svn: 317576
* [ValueTracking] readonly (const) is a requirement for converting sqrt to ↵Sanjay Patel2017-11-061-7/+38
| | | | | | | | | | | | | | | | | | | | | | llvm.sqrt; nnan is not As discussed in D39204, this is effectively a revert of rL265521 which required nnan to vectorize sqrt libcalls based on the old LangRef definition of llvm.sqrt. Now that the definition has been updated so the libcall and intrinsic have the same semantics apart from potentially setting errno, we can remove the nnan requirement. We have the right check to know that errno is not set: if (!ICS.onlyReadsMemory()) ...ahead of the switch. This will solve https://bugs.llvm.org/show_bug.cgi?id=27435 assuming that's being built for a target with -fno-math-errno. Differential Revision: https://reviews.llvm.org/D39642 llvm-svn: 317519
* [SLP] Test for PR35047, NFC.Alexey Bataev2017-11-061-0/+50
| | | | llvm-svn: 317482
* [SLPVectorizer] minimize tests and auto-generate full checks; NFCSanjay Patel2017-11-051-111/+105
| | | | llvm-svn: 317437
* [SLP] Test for PR23510, NFC.Alexey Bataev2017-11-031-0/+84
| | | | llvm-svn: 317334
* Revert rL315894, "SLPVectorizer.cpp: Try to appease stage2-3 difference. ↵NAKAMURA Takumi2017-10-161-153/+0
| | | | | | (D38586)" llvm-svn: 315896
* SLPVectorizer.cpp: Try to appease stage2-3 difference. (D38586)NAKAMURA Takumi2017-10-161-0/+153
| | | | llvm-svn: 315894
* [SLP] Add test for reversed load, NFC.Alexey Bataev2017-10-091-0/+30
| | | | llvm-svn: 315232
* [SLP] Test for wrongly vectorized set of extractelements, NFC.Alexey Bataev2017-10-091-0/+30
| | | | llvm-svn: 315217
* Revert r314806 "[SLP] Vectorize jumbled memory loads."Hans Wennborg2017-10-034-102/+52
| | | | | | | | | | | | | | | | | | | | | | | | | All the buildbots are red, e.g. http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/2436/ > Summary: > This patch tries to vectorize loads of consecutive memory accesses, accessed > in non-consecutive or jumbled way. An earlier attempt was made with patch D26905 > which was reverted back due to some basic issue with representing the 'use mask' of > jumbled accesses. > > This patch fixes the mask representation by recording the 'use mask' in the usertree entry. > > Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df > > Reviewers: mkuper, loladiro, Ayal, zvi, danielcdh > > Reviewed By: Ayal > > Subscribers: hans, mzolotukhin > > Differential Revision: https://reviews.llvm.org/D36130 llvm-svn: 314824
* [SLP] Vectorize jumbled memory loads.Mohammad Shahid2017-10-034-52/+102
| | | | | | | | | | | | | | | | | | | | | | Summary: This patch tries to vectorize loads of consecutive memory accesses, accessed in non-consecutive or jumbled way. An earlier attempt was made with patch D26905 which was reverted back due to some basic issue with representing the 'use mask' of jumbled accesses. This patch fixes the mask representation by recording the 'use mask' in the usertree entry. Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df Reviewers: mkuper, loladiro, Ayal, zvi, danielcdh Reviewed By: Ayal Subscribers: hans, mzolotukhin Differential Revision: https://reviews.llvm.org/D36130 llvm-svn: 314806
* [SLP] Fix crash on propagate IR flags for undef operands of min/maxAlexey Bataev2017-09-271-0/+86
| | | | | | | | | | | reductions. If both operands of the newly created SelectInst are Undefs the resulting operation is also Undef, not SelectInst. It may cause crashes when trying to propagate IR flags because function expects exactly SelectInst instruction, nothing else. llvm-svn: 314323
* [SLP] Add a test for PR32086, NFC.Alexey Bataev2017-09-251-0/+32
| | | | llvm-svn: 314137
* [SLP] Support for horizontal min/max reduction.Alexey Bataev2017-09-252-1231/+1470
| | | | | | | | | | | | | | | Summary: SLP vectorizer supports horizontal reductions for Add/FAdd binary operations. Patch adds support for horizontal min/max reductions. Function getReductionCost() is split to getArithmeticReductionCost() for binary operation reductions and getMinMaxReductionCost() for min/max reductions. Patch fixes PR26956. Reviewers: spatel, mkuper, hfinkel, RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D27846 llvm-svn: 314101
* Revert r313771 "[SLP] Vectorize jumbled memory loads."Hans Wennborg2017-09-204-102/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This broke the buildbots, e.g. http://bb.pgr.jp/builders/test-llvm-i686-linux-RA/builds/391 > Summary: > This patch tries to vectorize loads of consecutive memory accesses, accessed > in non-consecutive or jumbled way. An earlier attempt was made with patch D26905 > which was reverted back due to some basic issue with representing the 'use mask' > jumbled accesses. > > This patch fixes the mask representation by recording the 'use mask' in the usertree entry. > > Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df > > Subscribers: mzolotukhin > > Reviewed By: ayal > > Differential Revision: https://reviews.llvm.org/D36130 > > Review comments updated accordingly > > Change-Id: I22ab0a8a9bac9d49d74baa81a08e1e486f5e75f0 > > Added a TODO for sortLoadAccesses API > > Change-Id: I3c679bf1865422d1b45e17ea28f1992bca660b58 > > Modified the TODO for sortLoadAccesses API > > Change-Id: Ie64a66cb5f9e2a7610438abb0e750c6e090f9565 > > Review comment update for using OpdNum to insert the mask in respective location > > Change-Id: I016d0c1b29874e979efc0205bbf078991f92edce > > Fixes '-Wsign-compare warning' in LoopAccessAnalysis.cpp and code rebase > > Change-Id: I64b2ea5e68c1d7b6a028f5ef8251c5a97333f89b llvm-svn: 313781
* [SLP] Vectorize jumbled memory loads.Mohammad Shahid2017-09-204-52/+102
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This patch tries to vectorize loads of consecutive memory accesses, accessed in non-consecutive or jumbled way. An earlier attempt was made with patch D26905 which was reverted back due to some basic issue with representing the 'use mask' jumbled accesses. This patch fixes the mask representation by recording the 'use mask' in the usertree entry. Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df Subscribers: mzolotukhin Reviewed By: ayal Differential Revision: https://reviews.llvm.org/D36130 Review comments updated accordingly Change-Id: I22ab0a8a9bac9d49d74baa81a08e1e486f5e75f0 Added a TODO for sortLoadAccesses API Change-Id: I3c679bf1865422d1b45e17ea28f1992bca660b58 Modified the TODO for sortLoadAccesses API Change-Id: Ie64a66cb5f9e2a7610438abb0e750c6e090f9565 Review comment update for using OpdNum to insert the mask in respective location Change-Id: I016d0c1b29874e979efc0205bbf078991f92edce Fixes '-Wsign-compare warning' in LoopAccessAnalysis.cpp and code rebase Change-Id: I64b2ea5e68c1d7b6a028f5ef8251c5a97333f89b llvm-svn: 313771
* Revert r313736: "[SLP] Vectorize jumbled memory loads."Alexander Kornienko2017-09-204-102/+52
| | | | | | | The revision breaks buildbots: http://lab.llvm.org:8011/builders/clang-x86_64-debian-fast/builds/6694/steps/test/logs/stdio llvm-svn: 313758
* [SLP] Vectorize jumbled memory loads.Mohammad Shahid2017-09-204-52/+102
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: This patch tries to vectorize loads of consecutive memory accesses, accessed in non-consecutive or jumbled way. An earlier attempt was made with patch D26905 which was reverted back due to some basic issue with representing the 'use mask' of jumbled accesses. This patch fixes the mask representation by recording the 'use mask' in the usertree entry. Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df Reviewers: mkuper, loladiro, Ayal, zvi, danielcdh Reviewed By: Ayal Subscribers: mzolotukhin Differential Revision: https://reviews.llvm.org/D36130 Commit after rebase for patch D36130 Change-Id: I8add1c265455669ef288d880f870a9522c8c08ab llvm-svn: 313736
* [SLP] Reduce test, NFC.Alexey Bataev2017-09-191-134/+56
| | | | llvm-svn: 313630
* [SLP] Add a test for PR34635, NFC.Alexey Bataev2017-09-181-0/+176
| | | | llvm-svn: 313559
* [SLP] Revert r312791 and other necessary commits, except for TTI andChandler Carruth2017-09-152-1823/+1182
| | | | | | | | | | | | | | | | | | | | | | | | | | CostModel. The original patch added support for horizontal min/max reductions to the SLP vectorizer. This patch causes LLVM to miscompile fairly simple signed min reductions. I have attached a test progrom to http://llvm.org/PR34635 that shows the behavior change after this patch. We found this in a test for the open source Eigen library, but also in other code. Unfortunately, the revert is moderately challenging. It required reverting: r313042: [SLP] Test with multiple uses of conditional op and wrong parent. r312853: [SLP] Fix buildbots, NFC. r312793: [SLP] Fix the warning about paths not returning the value, NFC. r312791: [SLP] Support for horizontal min/max reduction. And even then, I had to completely skip reverting the changes to TTI and CostModel because r312832 rewrote so much of this code. Plus, the cost modeling changes aren implicated in the miscompile, so they should be fine and will just not be used until this gets re-introduced. llvm-svn: 313409
* Revert "[SLPVectorizer] Failure to beneficially vectorize 'copyable' ↵Ilya Biryukov2017-09-151-68/+132
| | | | | | | | | elements in integer binary ops." This reverts commit r313348. Reason: it caused buildbot failures. llvm-svn: 313352
* [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in ↵Dinar Temirbulatov2017-09-151-132/+68
| | | | | | | | | | | | | | | | | | | | | | integer binary ops. Patch tries to improve vectorization of the following code: void add1(int * __restrict dst, const int * __restrict src) { *dst++ = *src++; *dst++ = *src++ + 1; *dst++ = *src++ + 2; *dst++ = *src++ + 3; } Allows to vectorize even if the very first operation is not a binary add, but just a load. Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev, davide Subscribers: llvm-commits, RKSimon Differential Revision: https://reviews.llvm.org/D28907 llvm-svn: 313348
* [SLP] Test with mutiple uses of conditional op and wrong parent.Alexey Bataev2017-09-121-270/+1702
| | | | llvm-svn: 313042
* [SLP] Support for horizontal min/max reduction.Alexey Bataev2017-09-082-1181/+390
| | | | | | | | | | | | | SLP vectorizer supports horizontal reductions for Add/FAdd binary operations. Patch adds support for horizontal min/max reductions. Function getReductionCost() is split to getArithmeticReductionCost() for binary operation reductions and getMinMaxReductionCost() for min/max reductions. Patch fixes PR26956. Differential revision: https://reviews.llvm.org/D27846 llvm-svn: 312791
* [SLPVectorizer] Tighten up VLeft, VRight declaration, remove unnecessary ↵Dinar Temirbulatov2017-08-191-42/+0
| | | | | | testcase test/Transforms/SLPVectorizer/X86/reorder.ll, NFCI. llvm-svn: 311223
OpenPOWER on IntegriCloud