summaryrefslogtreecommitdiffstats
path: root/compiler-rt/lib/sanitizer_common/sanitizer_allocator_local_cache.h
Commit message (Collapse)AuthorAgeFilesLines
* [sanitizer][NFC] Remove unneeded SizeClassAllocatorLocalCacheVitaly Buka2019-05-011-7/+0
| | | | llvm-svn: 359729
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* [Sanitizers] Remove OOM/BadRequest allocator error handling policies.Alex Shlyapnikov2018-06-201-2/+5
| | | | | | | | | | | | | | | | | | | Summary: Remove the generic error nadling policies and handle each allocator error explicitly. Although more verbose, it allows for more comprehensive, precise and actionable allocator related failure reports. This finishes up the series of changes of the particular sanitizer allocators, improves the internal allocator error reporting and removes now unused policies. Reviewers: vitalybuka, cryptoad Subscribers: kubamracek, delcypher, #sanitizers, llvm-commits Differential Revision: https://reviews.llvm.org/D48328 llvm-svn: 335147
* [sanitizer] Remove unused 32-bit allocator TransferBatch parameterKostya Kortchinsky2018-05-031-2/+1
| | | | | | | | | | | | | | | | | Summary: NFC. Remove an unused parameter in `SizeClassAllocator32::TransferBatch::SetFromArray`, and thus get rid of the compilation warning. Reviewers: alekseyshl, vitalybuka Reviewed By: vitalybuka Subscribers: kubamracek, delcypher, #sanitizers, llvm-commits Differential Revision: https://reviews.llvm.org/D46397 llvm-svn: 331477
* [sanitizer] Size class map & local cache improvementsKostya Kortchinsky2018-02-121-10/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: - Reland rL324263, this time allowing for a compile-time decision as to whether or not use the 32-bit division. A single test is using a class map covering a maximum size greater than 4GB, this can be checked via the template parameters, and allows SizeClassAllocator64PopulateFreeListOOM to pass; - `MaxCachedHint` is always called on a class id for which we have already computed the size, but we still recompute `Size(class_id)`. Change the prototype of the function to work on sizes instead of class ids. This also allows us to get rid of the `kBatchClassID` special case. Update the callers accordingly; - `InitCache` and `Drain` will start iterating at index 1: index 0 contents are unused and can safely be left to be 0. Plus we do not pay the cost of going through an `UNLIKELY` in `MaxCachedHint`, and touching memory that is otherwise not used; - `const` some variables in the areas modified; - Remove an spurious extra line at the end of a file. Reviewers: alekseyshl, tl0gic, dberris Reviewed By: alekseyshl, dberris Subscribers: dberris, kubamracek, delcypher, llvm-commits, #sanitizers Differential Revision: https://reviews.llvm.org/D43088 llvm-svn: 324906
* [sanitizer] Allocator local cache improvementsKostya Kortchinsky2018-02-051-34/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Here are a few improvements proposed for the local cache: - `InitCache` always read from `per_class_[1]` in the fast path. This was not ideal as we are working with `per_class_[class_id]`. The latter offers the same property we are looking for (eg: `max_count != 0` means initialized), so we might as well use it and keep our memory accesses local to the same `per_class_` element. So change `InitCache` to take the current `PerClass` as an argument. This also makes the fast-path assembly of `Deallocate` a lot more compact; - Change the 32-bit `Refill` & `Drain` functions to mimic their 64-bit counterparts, by passing the current `PerClass` as an argument. This saves some array computations; - As far as I can tell, `InitCache` has no place in `Drain`: it's either called from `Deallocate` which calls `InitCache`, or from the "upper" `Drain` which checks for `c->count` to be greater than 0 (strictly). So remove it there. - Move the `stats_` updates to after we are done with the `per_class_` accesses in an attempt to preserve locality once more; - Change some `CHECK` to `DCHECK`: I don't think the ones changed belonged in the fast path and seemed to be overly cautious failsafes; - Mark some variables as `const`. The overall result is cleaner more compact fast path generated code, and some performance gains with Scudo (and likely other Sanitizers). Reviewers: alekseyshl Reviewed By: alekseyshl Subscribers: kubamracek, delcypher, #sanitizers, llvm-commits Differential Revision: https://reviews.llvm.org/D42851 llvm-svn: 324257
* [sanitizer] Re-introduce kUseSeparateSizeClassForBatch for the 32-bit PrimaryKostya Kortchinsky2017-08-281-45/+51
| | | | | | | | | | | | | | | | | | | | | | | Summary: Currently `TransferBatch` are located within the same memory regions as "regular" chunks. This is not ideal for security: they make for an interesting target to overwrite, and are not protected by the frontend (namely, Scudo). To solve this, we re-introduce `kUseSeparateSizeClassForBatch` for the 32-bit Primary allowing for `TransferBatch` to end up in their own memory region. Currently only Scudo would use this new feature, the default behavior remains unchanged. The separate `kBatchClassID` was used for a brief period of time previously but removed when the 64-bit ended up using the "free array". Reviewers: alekseyshl, kcc, eugenis Reviewed By: alekseyshl Subscribers: llvm-commits, kubamracek Differential Revision: https://reviews.llvm.org/D37082 llvm-svn: 311891
* [Sanitizers] 64 bit allocator respects allocator_may_return_null flagAlex Shlyapnikov2017-06-261-5/+9
| | | | | | | | | | | | | | | | Summary: Make SizeClassAllocator64 return nullptr when it encounters OOM, which allows the entire sanitizer's allocator to follow allocator_may_return_null=1 policy (LargeMmapAllocator: D34243, SizeClassAllocator64: D34433). Reviewers: eugenis Subscribers: srhines, kubamracek, llvm-commits Differential Revision: https://reviews.llvm.org/D34540 llvm-svn: 306342
* [Sanitizers] 32 bit allocator respects allocator_may_return_null flagAlex Shlyapnikov2017-06-221-3/+12
| | | | | | | | | | | | | | | | | | | Summary: Make SizeClassAllocator32 return nullptr when it encounters OOM, which allows the entire sanitizer's allocator to follow allocator_may_return_null=1 policy, even for small allocations (LargeMmapAllocator is already fixed by D34243). Will add a test for OOM in primary allocator later, when SizeClassAllocator64 can gracefully handle OOM too. Reviewers: eugenis Subscribers: kubamracek, llvm-commits Differential Revision: https://reviews.llvm.org/D34433 llvm-svn: 305972
* [sanitizer] Cache SizeClassForTransferBatch in the 32-bit local cacheKostya Kortchinsky2017-04-241-16/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: `SizeClassForTransferBatch` is expensive and is called for every `CreateBatch` and `DestroyBatch`. Caching it means `kNumClasses` calls in `InitCache` instead. This should be a performance gain if more than `kNumClasses / 2` batches are created and destroyed during the lifetime of the local cache. I have chosen to fully remove the function and putting the code in `InitCache`, which is a debatable choice. In single threaded benchmarks leveraging primary backed allocations, this turns out to be a sizeable gain in performances (greater than 5%). In multithreaded benchmarks leveraging everything, it is less significant but still an improvement (about 1%). Reviewers: kcc, dvyukov, alekseyshl Reviewed By: dvyukov Subscribers: kubamracek, llvm-commits Differential Revision: https://reviews.llvm.org/D32365 llvm-svn: 301184
* Cache size per class size in SizeClassAllocatorXLocalCache.Alex Shlyapnikov2017-04-131-4/+8
| | | | | | | | | | | | | | | Summary: Allocator::ClassIdToSize() is not free and calling it in every Allocate/Deallocate has noticeable impact on perf. Reapplying D31991 with the appropriate fixes. Reviewers: cryptoad Subscribers: kubamracek, llvm-commits Differential Revision: https://reviews.llvm.org/D32024 llvm-svn: 300216
* Revert "Cache size per class size in SizeClassAllocatorXLocalCache."Diana Picus2017-04-131-8/+4
| | | | | | | This reverts commit r300107 because it broke the ARM and AArch64 buildbots. llvm-svn: 300180
* Cache size per class size in SizeClassAllocatorXLocalCache.Alex Shlyapnikov2017-04-121-4/+8
| | | | | | | | | | | | | | Summary: Allocator::ClassIdToSize() is not free and calling it in every Allocate/Deallocate has noticeable impact on perf. Reviewers: eugenis, kcc Subscribers: llvm-commits, kubamracek Differential Revision: https://reviews.llvm.org/D31991 llvm-svn: 300107
* Avoid calling SizeClassMap::MaxCachedHint on hot path, it's not free.Alex Shlyapnikov2017-04-121-1/+1
| | | | | | | | | | | | Summary: Remove unecessary SizeClassMap::MaxCachedHint call. Reviewers: eugenis Subscribers: llvm-commits, kubamracek Differential Revision: https://reviews.llvm.org/D31989 llvm-svn: 300103
* [sanitizer] remove kBatchClassID that is not used any more; NFCKostya Serebryany2016-08-311-14/+0
| | | | llvm-svn: 280185
* [sanitizer] re-apply r279572 and r279595 reverted in r279643: change the ↵Kostya Serebryany2016-08-241-75/+30
| | | | | | 64-bit allocator to use a single array for free-d chunks instead of a lock-free linked list of tranfer batches. This change simplifies the code, makes the allocator more 'hardened', and will allow simpler code to release RAM to OS. This may also slowdown malloc stress tests due to lock contension, but I did not observe noticeable slowdown on various real multi-threaded benchmarks. llvm-svn: 279664
* Revert r279572 "[sanitizer] change the 64-bit..." because of failures on ubsanVitaly Buka2016-08-241-30/+75
| | | | | | This reverts commit r279572 and r279595. llvm-svn: 279643
* [sanitizer] change the 64-bit allocator to use a single array for free-d ↵Kostya Serebryany2016-08-231-75/+30
| | | | | | chunks instead of a lock-free linked list of tranfer batches. This change simplifies the code, makes the allocator more 'hardened', and will allow simpler code to release RAM to OS. This may also slowdown malloc stress tests due to lock contension, but I did not observe noticeable slowdown on various real multi-threaded benchmarks. llvm-svn: 279572
* [sanitizer] allocator: split the local cache class into two, one for 32-bit ↵Kostya Serebryany2016-08-231-1/+150
| | | | | | allocator and one for 64-bit one. NFC. The two imlementations will diverge in the following changes. llvm-svn: 279495
* [sanitizer] use 32-bit offset instead of 64-bit pointers in the 64-bit ↵Kostya Serebryany2016-08-091-6/+6
| | | | | | allocator's transfer batches. This saves 2x memory for the transfer batches (up to ~1.5% overall in some cases) llvm-svn: 278179
* [sanitizer] minor refactoring in the allocator, NFCKostya Serebryany2016-08-091-2/+1
| | | | llvm-svn: 278163
* [sanitizer] allocator: move TransferBatch into ↵Kostya Serebryany2016-08-061-13/+43
| | | | | | SizeClassAllocator64/SizeClassAllocator32 because we actually need different iplementations for the 64- and 32-bit case. NFC; the following patches will make the TransferBatch implementations differ llvm-svn: 277899
* [sanitizer] remove one redundant loop from the allocatorKostya Serebryany2016-08-031-5/+4
| | | | llvm-svn: 277563
* [sanitizer] refactor TransferBatch to hide the implementation. NFC expected. ↵Kostya Serebryany2016-08-031-9/+7
| | | | | | Second attempt after failed r276383 which was reverted. llvm-svn: 277554
* [sanitizer] revert r276383 while investigating failures on botKostya Serebryany2016-07-221-7/+9
| | | | llvm-svn: 276456
* [sanitizer] refactor TransferBatch to hide the implementation. NFCKostya Serebryany2016-07-221-9/+7
| | | | llvm-svn: 276383
* [sanitizer] allocator: introduce kUseSeparateSizeClassForBatch (false by ↵Kostya Serebryany2016-07-221-4/+4
| | | | | | default). When true, it will cause all TransferBatches to be allocated on a separate dedicated size class, which improves security and may potentially simplify memory reclamation. However in the current state this may cause up to 3% extra memory usage. Subsequent changes should bring this overhead down llvm-svn: 276377
* [sanitizer] allocator: remove kPopulateSize and only use ↵Kostya Serebryany2016-07-211-9/+19
| | | | | | SizeClassMap::MaxCached; ensure that TransferBatch size is a power of two, refactor TransferBatch creation/destruction into separate functions. llvm-svn: 276318
* [sanitizers] split sanitizer_allocator.h into a number of smaller .h files; NFCKostya Serebryany2016-07-201-0/+123
llvm-svn: 276195
OpenPOWER on IntegriCloud