summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Support/FileOutputBuffer.cpp
Commit message (Collapse)AuthorAgeFilesLines
* [LLD][ELF] Support --[no-]mmap-output-file with F_no_mmapNick Terrell2019-10-291-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Add a flag `F_no_mmap` to `FileOutputBuffer` to support `--[no-]mmap-output-file` in ELF LLD. LLD currently explicitly ignores this flag for compatibility with GNU ld and gold. We need this flag to speed up link time for large binaries in certain scenarios. When we link some of our larger binaries we find that LLD takes 50+ GB of memory, which causes memory pressure. The memory pressure causes the VM to flush dirty pages of the output file to disk. This is normally okay, since we should be flushing cold pages. However, when using BtrFS with compression we need to write 128KB at a time when we flush a page. If any page in that 128KB block is written again, then it must be flushed a second time, and so on. Since LLD doesn't write sequentially this causes write amplification. The same 128KB block will end up being flushed multiple times, causing the linker to many times more IO than necessary. We've observed 3-5x faster builds with -no-mmap-output-file when we hit this scenario. The bad scenario only applies to compressed filesystems, which group together multiple pages into a single compressed block. I've tested BtrFS, but the problem will be present for any compressed filesystem on Linux, since it is caused by the VM. Silently ignoring --no-mmap-output-file caused a silent regression when we switched from gold to lld. We pass --no-mmap-output-file to fix this edge case, but since lld silently ignored the flag we didn't realize it wasn't being respected. Benchmark building a 9 GB binary that exposes this edge case. I linked 3 times with --mmap-output-file and 3 times with --no-mmap-output-file and took the average. The machine has 24 cores @ 2.4 GHz, 112 GB of RAM, BtrFS mounted with -compress-force=zstd, and an 80% full disk. | Mode | Time | |---------|-------| | mmap | 894 s | | no mmap | 126 s | When compression is disabled, BtrFS performs just as well with and without mmap on this benchmark. I was unable to reproduce the regression with any binaries in lld-speed-test. Reviewed By: ruiu, MaskRay Differential Revision: https://reviews.llvm.org/D69294
* [llvm] Migrate llvm::make_unique to std::make_uniqueJonas Devlieghere2019-08-151-3/+3
| | | | | | | | Now that we've moved to C++14, we no longer need the llvm::make_unique implementation from STLExtras.h. This patch is a mechanical replacement of (hopefully) all the llvm::make_unique instances across the monorepo. llvm-svn: 369013
* [Support] Move llvm::MemoryBuffer to sys::fs::file_tReid Kleckner2019-07-101-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: On Windows, Posix integer file descriptors are a compatibility layer over native file handles provided by the C runtime. There is a hard limit on the maximum number of file descriptors that a process can open, and the limit is 8192. LLD typically doesn't run into this limit because it opens input files, maps them into memory, and then immediately closes the file descriptor. This prevents it from running out of FDs. For various reasons, I'd like to open handles to every input file and keep them open during linking. That requires migrating MemoryBuffer over to taking open native file handles instead of integer FDs. Reviewers: aganea, Bigcheese Reviewed By: aganea Subscribers: smeenai, silvas, mehdi_amini, hiraditya, steven_wu, dexonsmith, dang, llvm-commits, zturner Tags: #llvm Differential Revision: https://reviews.llvm.org/D63453 llvm-svn: 365588
* [Support] Renamed member 'Size' to 'AllocatedSize' in MemoryBlock and ↵Lang Hames2019-05-201-7/+11
| | | | | | | | | | | | | | | OwningMemoryBlock. Rename member 'Size' to 'AllocatedSize' in order to provide a hint that the allocated size may be different than the requested size. Comments are added to clarify this point. Updated the InMemoryBuffer in FileOutputBuffer.cpp to track the requested buffer size. Patch by Machiel van Hooren. Thanks Machiel! https://reviews.llvm.org/D61599 llvm-svn: 361195
* FileOutputBuffer: handle mmap(2) failureRui Ueyama2019-01-221-2/+6
| | | | | | | | | | | | | | | | If the underlying filesystem does not support mmap system call, FileOutputBuffer may fail when it attempts to mmap an output temporary file. This patch handles such situation. Unfortunately, it looks like it is very hard to test this functionality without a filesystem that doesn't support mmap using llvm-lit. I tested this locally by passing an invalid parameter to mmap so that it fails and falls back to the in-memory buffer. Maybe that's all what we can do. I believe it is reasonable to submit this without a test. Differential Revision: https://reviews.llvm.org/D56949 llvm-svn: 351883
* FileOutputBuffer: Handle "-" as stdout.Rui Ueyama2019-01-221-0/+10
| | | | | | | | | | | | | | | | | I was honestly a bit surprised that we didn't do this before. This patch is to handle "-" as the stdout so that if you pass `-o -` to lld, for example, it writes an output to stdout instead of file `-`. I thought that we might want to handle this at a higher level than FileOutputBuffer, because if we land this patch, we can no longer create a file whose name is `-` (there's a workaround though; you can pass `./-` instead of `-`). However, because raw_fd_ostream already handles `-` as a special file name, I think it's okay and actually consistent to handle `-` as a special name in FileOutputBuffer. Differential Revision: https://reviews.llvm.org/D56940 llvm-svn: 351852
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* Remove F_modify flag from FileOutputBuffer.Rui Ueyama2019-01-191-26/+11
| | | | | | | | This code is dead. There is no use of the feature in the entire LLVM codebase. Differential Revision: https://reviews.llvm.org/D56939 llvm-svn: 351613
* [Support] Allow discarding a FileOutputBuffer without removing the memory ↵Martin Storsjo2018-08-241-0/+6
| | | | | | | | mapping Differential Revision: https://reviews.llvm.org/D51095 llvm-svn: 340634
* Add a flag to FileOutputBuffer that allows modification.Zachary Turner2018-06-281-11/+26
| | | | | | | | | | | | | FileOutputBuffer creates a temp file and on commit atomically renames the temp file to the destination file. Sometimes we want to modify an existing file in place, but still have the atomicity guarantee. To do this we can initialize the contents of the temp file from the destination file (if it exists), that way the resulting FileOutputBuffer can have only selective bytes modified. Committing will then atomically replace the destination file as desired. llvm-svn: 335902
* Fix unused private variable.Zachary Turner2018-06-071-1/+2
| | | | | | This parameter got lost in the refactor. Add it back. llvm-svn: 334223
* [FileSystem] Split up the OpenFlags enumeration.Zachary Turner2018-06-071-1/+2
| | | | | | | | | | | | | | | | | This breaks the OpenFlags enumeration into two separate enumerations: OpenFlags and CreationDisposition. The first controls the behavior of the API depending on whether or not the target file already exists, and is not a flags-based enum. The second controls more flags-like values. This yields a more easy to understand API, while also allowing flags to be passed to the openForRead api, where most of the values didn't make sense before. This also makes the apis more testable as it becomes easy to enumerate all the configurations which make sense, so I've added many new tests to exercise all the different values. llvm-svn: 334221
* s/LLVM_ON_WIN32/_WIN32/, llvmNico Weber2018-04-291-1/+1
| | | | | | | | | | | | | | LLVM_ON_WIN32 is set exactly with MSVC and MinGW (but not Cygwin) in HandleLLVMOptions.cmake, which is where _WIN32 defined too. Just use the default macro instead of a reinvented one. See thread "Replacing LLVM_ON_WIN32 with just _WIN32" on llvm-dev and cfe-dev. No intended behavior change. This moves over all uses of the macro, but doesn't remove the definition of it in (llvm-)config.h yet. llvm-svn: 331127
* Make helpers static. NFC.Benjamin Kramer2017-11-241-0/+2
| | | | llvm-svn: 318953
* Create a TempFile class.Rafael Espindola2017-11-131-20/+19
| | | | | | | | | | | | | | | This just adds a TempFile class and replaces the use in FileOutputBuffer with it. The only difference for now is better error handling. Followup work includes: - Convert other user of temporary files to it. - Add support for automatically deleting on windows. - Add a createUnnamed method that returns a potentially unnamed file. It would be actually unnamed on modern linux and have a unknown name on windows. llvm-svn: 318069
* [FileOutputBuffer] Move factory methods out of their classes.Rui Ueyama2017-11-081-17/+14
| | | | | | | | | | InMemoryBuffer and OnDiskBuffer classes have both factory methods and public constructors, and that looks a bit odd. This patch makes factory methods non-member function to fix it. Differential Revision: https://reviews.llvm.org/D39693 llvm-svn: 317739
* Convert FileOutputBuffer::commit to Error.Rafael Espindola2017-11-081-5/+5
| | | | llvm-svn: 317656
* Convert FileOutputBuffer to Expected. NFC.Rafael Espindola2017-11-081-9/+9
| | | | llvm-svn: 317649
* Rewrite FileOutputBuffer as two separate classes.Rui Ueyama2017-11-011-82/+120
| | | | | | | | | | | | | | | | | | | | This patch is to rewrite FileOutputBuffer as two separate classes; one for file-backed output buffer and the other for memory-backed output buffer. I think the new code is easier to follow because two different implementations are now actually separated as different classes. Unlike the previous implementation, the class that does not replace the final output file using rename(2) does not create a temporary file at all. Instead, it allocates memory using mmap(2) and use it. I think this is an improvement because it is now guaranteed that the temporary memory region doesn't trigger any I/O and there's now zero chance to leave a temporary file behind. Also, it shouldn't impose new restrictions because were using mmap IO too. Differential Revision: https://reviews.llvm.org/D39449 llvm-svn: 317127
* Do not remove a target file in FileOutputBuffer::create().Rui Ueyama2017-09-271-8/+1
| | | | | | | | | | | | | | | | | | FileOutputBuffer::create() attempts to remove a target file if the file is a regular one, which results in an unexpected result in a failure scenario. If something goes wrong and the user of FileOutputBuffer decides to not call commit(), it leaves nothing. An existing file is removed, and no new file is created. What we should do is to atomically replace an existing file with a new file using rename(), so that it wouldn't remove an existing file without creating a new one. Differential Revision: https://reviews.llvm.org/D38283 llvm-svn: 314345
* Make FileOutputBuffer fail early if you pass a directory.Rui Ueyama2017-03-131-0/+2
| | | | | | | | | | Previously, it created a temporary directory and then failed when FileOutputBuffer tried to rename that file to the destination file (which is actually a directory name). Differential Revision: https://reviews.llvm.org/D30912 llvm-svn: 297679
* Support outputting to /dev/null.Rafael Espindola2017-01-091-22/+44
| | | | | | | | When writing to a non regular file we cannot rename to it. Since we have to write, we may as well create a temporary file to avoid trying to create an unique file in /dev when trying to write to /dev/null. llvm-svn: 291485
* Fix a real temp file leak in FileOutputBufferReid Kleckner2016-09-021-0/+3
| | | | | | | | | | | | | If we failed to commit the buffer but did not die to a signal, the temp file would remain on disk on Windows. Having an open file mapping and file handle prevents the file from being deleted. I am choosing not to add an assertion of success on the temp file removal, since virus scanners and other environmental things can often cause removal to fail in real world tools. Also fix more temp file leaks in unit tests. llvm-svn: 280445
* Remove temporary file on signal.Rafael Espindola2015-09-181-1/+6
| | | | | | Without this lld leaves temporary files behind when it crashes. llvm-svn: 247994
* Return ErrorOr from FileOutputBuffer::create. NFC.Rafael Espindola2015-08-131-7/+4
| | | | llvm-svn: 244848
* Revert r240137 (Fixed/added namespace ending comments using clang-tidy. NFC)Alexander Kornienko2015-06-231-1/+1
| | | | | | Apparently, the style needs to be agreed upon first. llvm-svn: 240390
* Fixed/added namespace ending comments using clang-tidy. NFCAlexander Kornienko2015-06-191-1/+1
| | | | | | | | | | | | | The patch is generated using this command: tools/clang/tools/extra/clang-tidy/tool/run-clang-tidy.py -fix \ -checks=-*,llvm-namespace-comment -header-filter='llvm/.*|clang/.*' \ llvm/lib/ Thanks to Eugene Kosov for the original patch! llvm-svn: 240137
* Purge unused includes throughout libSupport.Benjamin Kramer2015-03-231-4/+3
| | | | | | NFC. llvm-svn: 232976
* Support: Improve performance of FileOutputBuffer on WindowsRui Ueyama2015-03-061-0/+7
| | | | | | | | | | We extend an underlying file before mmap'ing it, but it's not needed on Windows. Extending file is slow on Windows, so we should avoid doing that. The difference gets larger as the size of an output file gets larger. It shove off 2 seconds out of 25 seconds when linking chrome.dll with LLD, for example. llvm-svn: 231452
* [cleanup] Re-sort all the #include lines in LLVM usingChandler Carruth2015-01-141-2/+2
| | | | | | | | | | | utils/sort_includes.py. I clearly haven't done this in a while, so more changed than usual. This even uncovered a missing include from the InstrProf library that I've added. No functionality changed here, just mechanical cleanup of the include order. llvm-svn: 225974
* Move the resize file feature from mapped_file_region to the only user.Rafael Espindola2014-12-121-0/+4
| | | | | | This removes a duplicated stat on every file that llvm-ar looks at. llvm-svn: 224138
* Remove unused feature. NFC.Rafael Espindola2014-12-121-7/+1
| | | | llvm-svn: 224135
* Remove a convoluted way of calling close by moving the call to the only caller.Rafael Espindola2014-12-111-1/+10
| | | | | | As a bonus we can actually check the return value. llvm-svn: 224046
* unique_ptrify FileOutputBuffer::FileOutputBufferDavid Blaikie2014-09-021-10/+7
| | | | llvm-svn: 216921
* Remove uses of the redundant ".reset(nullptr)" of unique_ptr, in favor of ↵David Blaikie2014-07-191-1/+1
| | | | | | | | | | | ".reset()" It's also possible to just write "= nullptr", but there's some question of whether that's as readable, so I leave it up to authors to pick which they prefer for now. If we want to discuss standardizing on one or the other, we can do that at some point in the future. llvm-svn: 213438
* Finishing touch for the std::error_code transition.Rafael Espindola2014-06-131-1/+2
| | | | | | | | | | | While std::error_code itself seems to work OK in all platforms, there are few annoying differences with regards to the std::errc enumeration. This patch adds a simple llvm enumeration, which will hopefully avoid build breakages in other platforms and surprises as we get more uses of std::error_code. llvm-svn: 210920
* Remove 'using std::errro_code' from lib.Rafael Espindola2014-06-131-9/+8
| | | | llvm-svn: 210871
* Don't use 'using std::error_code' in include/llvm.Rafael Espindola2014-06-121-0/+1
| | | | | | This should make sure that most new uses use the std prefix. llvm-svn: 210835
* Remove system_error.h.Rafael Espindola2014-06-121-1/+1
| | | | | | | This is a minimal change to remove the header. I will remove the occurrences of "using std::error_code" in a followup patch. llvm-svn: 210803
* Don't import make_error_code into the llvm namespace.Rafael Espindola2014-06-121-1/+1
| | | | llvm-svn: 210772
* Use std::error_code instead of llvm::error_code.Rafael Espindola2014-06-111-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The idea of this patch is to turn llvm/Support/system_error.h into a transitional header that just brings in the erorr_code api to the llvm namespace. I will remove it shortly afterwards. The cases where the general idea needed some tweaking: * std::errc is a namespace in msvc, so we cannot use "using std::errc". I could add an #ifdef, but there were not that many uses, so I just added std:: to them in this patch. * Template specialization had to be moved to the std namespace in this patch set already. * The msvc implementation of default_error_condition doesn't seem to provide the same transformations as we need. Not too surprising since the standard doesn't actually say what "equivalent" means. I fixed the problem by keeping our old mapping and using it at error_code construction time. Despite these shortcomings I think this is still a good thing. Some reasons: * The different implementations of system_error might improve over time. * It removes 925 lines of code from llvm already. * It removes 6313 bytes from the text segment of the clang binary when it is built with gcc and 2816 bytes when building with clang and libstdc++. llvm-svn: 210687
* Use error_code() instead of error_code::succes()Rafael Espindola2014-05-311-1/+1
| | | | | | | There is no std::error_code::success, so this removes much of the noise in transitioning to std::error_code. llvm-svn: 209952
* Remove last uses of OwningPtr from llvm. As far as I can tell these method ↵Craig Topper2014-05-181-11/+0
| | | | | | versions are not used by lldb, lld, or clang. llvm-svn: 209103
* [C++11] Make use of 'nullptr' in the Support library.Craig Topper2014-04-071-1/+1
| | | | llvm-svn: 205697
* [C++11] Add overloads for externally used OwningPtr functions.Ahmed Charles2014-03-051-3/+13
| | | | | | | | This will allow external callers of these functions to switch over time rather than forcing a breaking change all a once. These particular functions were determined by building clang/lld/lldb. llvm-svn: 202959
* Use the simpler version of sys::fs::remove when possible.Rafael Espindola2014-01-101-4/+2
| | | | llvm-svn: 198958
* Create files with the correct permission instead of changing it afterwards.Rafael Espindola2013-07-081-22/+7
| | | | | | Not intended functionality change. llvm-svn: 185830
* Add a createUniqueFile function and switch llvm's users of unique_file.Rafael Espindola2013-07-051-2/+2
| | | | | | | | | | | | | | This function is complementary to createTemporaryFile. It handles the case were the unique file is *not* temporary: we will rename it in the end. Since we will rename it, the file has to be in the same filesystem as the final destination and we don't prepend the system temporary directory. This has a small semantic difference from unique_file: the default mode is 0666. This matches the behavior of most unix tools. For example, with this change lld now produces files with the same permissions as ld. I will add a test of this change when I port clang over to createUniqueFile (next commit). llvm-svn: 185726
* [Support] Fix lifetime of file descriptors when using MemoryBuffer.Michael J. Spencer2013-03-141-2/+2
| | | | | | | Clients of MemoryBuffer::getOpenFile expect it not to take ownership of the file descriptor passed in. So don't. llvm-svn: 176995
* [Support] Make FileOutputBuffer work on Windows.Michael J. Spencer2012-12-031-55/+27
| | | | llvm-svn: 169167
OpenPOWER on IntegriCloud