summaryrefslogtreecommitdiffstats
path: root/llvm/lib/LTO/Caching.cpp
Commit message (Collapse)AuthorAgeFilesLines
* [llvm] Migrate llvm::make_unique to std::make_uniqueJonas Devlieghere2019-08-151-2/+2
| | | | | | | | Now that we've moved to C++14, we no longer need the llvm::make_unique implementation from STLExtras.h. This patch is a mechanical replacement of (hopefully) all the llvm::make_unique instances across the monorepo. llvm-svn: 369013
* Open native file handles to avoid converting from FDs, NFCReid Kleckner2019-07-111-9/+10
| | | | | | Follow up to r365588. llvm-svn: 365820
* [Support] Move llvm::MemoryBuffer to sys::fs::file_tReid Kleckner2019-07-101-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: On Windows, Posix integer file descriptors are a compatibility layer over native file handles provided by the C runtime. There is a hard limit on the maximum number of file descriptors that a process can open, and the limit is 8192. LLD typically doesn't run into this limit because it opens input files, maps them into memory, and then immediately closes the file descriptor. This prevents it from running out of FDs. For various reasons, I'd like to open handles to every input file and keep them open during linking. That requires migrating MemoryBuffer over to taking open native file handles instead of integer FDs. Reviewers: aganea, Bigcheese Reviewed By: aganea Subscribers: smeenai, silvas, mehdi_amini, hiraditya, steven_wu, dexonsmith, dang, llvm-commits, zturner Tags: #llvm Differential Revision: https://reviews.llvm.org/D63453 llvm-svn: 365588
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* [ThinLTO] Update ThinLTO cache file atimes when on WindowsAndrew Ng2018-07-041-8/+23
| | | | | | | | | | | | | | | | | | ThinLTO cache file access times are used for expiration based pruning and since Vista, file access times are not updated by Windows by default: https://blogs.technet.microsoft.com/filecab/2006/11/07/disabling-last-access-time-in-windows-vista-to-improve-ntfs-performance This means on Windows, cache files are currently being pruned from creation time. This change manually updates cache files that are accessed by ThinLTO, when on Windows. Patch by Owen Reynolds. Differential Revision: https://reviews.llvm.org/D47266 llvm-svn: 336276
* LTO: Keep file handles open for memory mapped files.Peter Collingbourne2018-06-131-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On Windows we've observed that if you open a file, write to it, map it into memory and close the file handle, the contents of the memory mapping can sometimes be incorrect. That was what we did when adding an entry to the ThinLTO cache using the TempFile and MemoryBuffer classes, and it was causing intermittent build failures on Chromium's ThinLTO bots on Windows. More details are in the associated Chromium bug (crbug.com/786127). We can prevent this from happening by keeping a handle to the file open while the mapping is active. So this patch changes the mapped_file_region class to duplicate the file handle when mapping the file and close it upon unmapping it. One gotcha is that the file handle that we keep open must not have been created with FILE_FLAG_DELETE_ON_CLOSE, as otherwise the operating system will prevent other processes from opening the file. We can achieve this by avoiding the use of FILE_FLAG_DELETE_ON_CLOSE altogether. Instead, we use SetFileInformationByHandle with FileDispositionInfo to manage the delete-on-close bit. This lets us remove the hack that we used to use to clear the delete-on-close bit on a file opened with FILE_FLAG_DELETE_ON_CLOSE. A downside of using SetFileInformationByHandle/FileDispositionInfo as opposed to FILE_FLAG_DELETE_ON_CLOSE is that it prevents us from using CreateFile to open the file while the flag is set, even within the same process. This doesn't seem to matter for almost every client of TempFile, except for LockFileManager, which calls sys::fs::create_link to create a hard link from the lock file, and in the process of doing so tries to open the file. To prevent this change from breaking LockFileManager I changed it to stop using TempFile by effectively reverting r318550. Differential Revision: https://reviews.llvm.org/D48051 llvm-svn: 334630
* [LTO] Remove unused Path parameter to AddBufferFnTeresa Johnson2018-02-201-2/+2
| | | | | | | | | | | | | | | Summary: With D43396, no clients use the Path parameter anymore. Depends on D43396. Reviewers: pcc Subscribers: mehdi_amini, inglorion, llvm-commits Differential Revision: https://reviews.llvm.org/D43400 llvm-svn: 325619
* LTO: clarify why we need to gracefully handle sys::fs::rename failuresBob Haarman2017-11-161-7/+9
| | | | | | | | | | | | Reviewers: pcc, rafael Reviewed By: pcc Subscribers: mehdi_amini, llvm-commits, hiraditya Differential Revision: https://reviews.llvm.org/D40094 llvm-svn: 318362
* Use TempFile in lto caching.Rafael Espindola2017-11-151-25/+34
| | | | | | | | | | This requires a small change to TempFile: allowing a discard after a failed keep. With this the cache now handles signals and reuses a fd instead of reopening the file. llvm-svn: 318322
* LTO: don't fatal when value for cache key already existsBob Haarman2017-11-101-2/+15
| | | | | | | | | | | | | | | | | | | | Summary: LTO/Caching.cpp uses file rename to atomically set the value for a cache key. On Windows, this fails when the destination file already exists. Previously, LLVM would report_fatal_error in such cases. However, because the old and the new value for the cache key are supposed to be equivalent, it actually doesn't matter which one we keep. This change makes it so that failing the rename when an openable file with the desired name already exists causes us to report success instead of fataling. Reviewers: pcc, hans Subscribers: mehdi_amini, llvm-commits, hiraditya Differential Revision: https://reviews.llvm.org/D39874 llvm-svn: 317899
* LTO: Improve error reporting when adding a cache entry.Peter Collingbourne2017-10-031-4/+6
| | | | | | | | Move error handling code next to the code that returns the error, and change the error message in order to distinguish it from a similar error message elsewhere in this file. llvm-svn: 314745
* LTO: Remove unnecessary Windows support code.Peter Collingbourne2017-09-071-15/+0
| | | | | | | I empirically verified that open files can in fact be renamed on Windows with sys::fs::rename, so remove the incorrect code and comment. llvm-svn: 312683
* LTO: Try to open cache files before renaming them.Peter Collingbourne2017-09-051-6/+21
| | | | | | | | | | | | | | It appears that a potential race between the cache client and the cache pruner that I thought was unlikely actually happened in practice [1]. Try to avoid the race condition by opening the temporary file before renaming it. Do this only on non-Windows platforms because we cannot rename open files on Windows using the sys::fs::rename function. [1] https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.memory%2FLinux_CFI%2F1610%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout Differential Revision: https://reviews.llvm.org/D37410 llvm-svn: 312567
* Try using llvm::errc, should hopefully fix version mismatch problem on ↵Peter Collingbourne2017-03-201-4/+2
| | | | | | clang-s390x-linux bot. llvm-svn: 298285
* Support, LTO: When pruning a directory, ignore files matching a prefix.Peter Collingbourne2017-03-201-2/+4
| | | | | | | | | | This is a safeguard against data loss if the user specifies a directory that is not a cache directory. Teach the existing cache pruning clients to create files with appropriate names. Differential Revision: https://reviews.llvm.org/D31109 llvm-svn: 298271
* LTO: Work around libstdc++ version mismatch bug, see D31063 review thread.Peter Collingbourne2017-03-171-1/+4
| | | | llvm-svn: 298127
* LTO: Fix a potential race condition in the caching API.Peter Collingbourne2017-03-171-9/+24
| | | | | | | | | | | | | | | | | | | | | After the call to sys::fs::exists succeeds, indicating a cache hit, we call AddFile and the client will open the file using the supplied path. If the client is using cache pruning, there is a potential race between the pruner and the client. To avoid this, change the caching API so that it provides a MemoryBuffer to the client, and have clients use that MemoryBuffer where possible. This scheme won't work with the gold plugin because the plugin API expects a file path. So we have the gold plugin use the buffer identifier as a path and live with the race for now. (Note that the gold plugin isn't actually affected by the problem at the moment because it doesn't support cache pruning.) This effectively reverts r279883 modulo the change to use the existing path in the gold plugin. Differential Revision: https://reviews.llvm.org/D31063 llvm-svn: 298020
* LTO: Create temporary cache files in the cache directory instead of $TMPDIR.Peter Collingbourne2017-03-161-28/+8
| | | | | | | | | | This fixes a race condition where another linker process can observe a partially written file if we copy it from another file system, and allows the link to be independent of the amount of free disk space in $TMPDIR. Differential Revision: https://reviews.llvm.org/D31045 llvm-svn: 297970
* LTO: When creating a local cache, create the cache directory if it does not ↵Peter Collingbourne2017-03-021-2/+5
| | | | | | | | already exist. Differential Revision: https://reviews.llvm.org/D30519 llvm-svn: 296726
* Apply clang-tidy's performance-unnecessary-value-param to LLVM.Benjamin Kramer2017-01-131-3/+4
| | | | | | | With some minor manual fixes for using function_ref instead of std::function. No functional change intended. llvm-svn: 291904
* Add qualification to fix MSVC build.Peter Collingbourne2016-09-231-1/+1
| | | | llvm-svn: 282313
* LTO: Simplify caching interface.Peter Collingbourne2016-09-231-50/+51
| | | | | | | | | | | | | | | | | The NativeObjectOutput class has a design problem: it mixes up the caching policy with the interface for output streams, which makes the client-side code hard to follow and would for example make it harder to replace the cache implementation in an arbitrary client. This change separates the two aspects by moving the caching policy to a separate field in Config, replacing NativeObjectOutput with a NativeObjectStream class which only deals with streams and does not need to be overridden by most clients and introducing an AddFile callback for adding files (e.g. from the cache) to the link. Differential Revision: https://reviews.llvm.org/D24622 llvm-svn: 282299
* [ThinLTO] Move loading of cache entry to clientTeresa Johnson2016-08-261-8/+2
| | | | | | | | | | | | | | | | | | | | | | Summary: Have the cache pass back the path to the cache entry when it is ready to be loaded, instead of a buffer. For gold-plugin we can simply pass this file back to gold directly, which avoids expensive writing of a separate tmp file. Ensure the cache entry is not deleted on cleanup by adjusting the setting of the IsTemporary flags. Moved the loading of the buffer into llvm-lto2 to maintain current behavior. Reviewers: mehdi_amini Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D23946 llvm-svn: 279883
* [ThinLTO] Add caching to the new LTO APIMehdi Amini2016-08-231-0/+104
Add the ability to plug a cache on the LTO API. I tried to write such that a linker implementation can control the cache backend. This is intrusive and I'm not totally happy with it, but I can't figure out a better design right now. Differential Revision: https://reviews.llvm.org/D23599 llvm-svn: 279576
OpenPOWER on IntegriCloud