summaryrefslogtreecommitdiffstats
path: root/clang/lib/Serialization/ModuleManager.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Close FileEntries of cached files in ModuleManager::addModule().Adrian Prantl2018-08-201-5/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | While investigating why LLDB (which can build hundreds of clang modules during one debug session) was getting "too many open files" errors, I found that most of them are .pcm files that are kept open by ModuleManager. Pretty much all of the open file dscriptors are FileEntries that are refering to `.pcm` files for which a buffer already exists in a CompilerInstance's PCMCache. Before PCMCache was added it was necessary to hold on to open file descriptors to ensure that all ModuleManagers using the same FileManager read the a consistent version of a given `.pcm` file on disk, even when a concurrent clang process overwrites the file halfway through. The PCMCache makes this practice unnecessary, since it caches the entire contents of a `.pcm` file, while the FileManager caches all the stat() information. This patch adds a call to FileEntry::closeFile() to the path where a Buffer has already been created. This is necessary because even for a freshly written `.pcm` file the file is stat()ed once immediately after writing to generate a FileEntry in the FileManager. Because a freshly-generated file's contents is stored in the PCMCache, it is fine to close the file immediately thereafter. The second change this patch makes is to set the `ShouldClose` flag to true when reading a `.pcm` file into the PCMCache for the first time. [For reference, in 1 Clang instance there is - 1 FileManager and - n ModuleManagers with - n PCMCaches.] rdar://problem/40906753 Differential Revision: https://reviews.llvm.org/D50870 llvm-svn: 340188
* Remove trailing spaceFangrui Song2018-07-301-5/+5
| | | | | | sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h} llvm-svn: 338291
* [Serialization] Fix some Clang-tidy modernize and Include What You Use ↵Eugene Zelenko2017-11-111-18/+29
| | | | | | warnings; other minor fixes (NFC). llvm-svn: 317953
* [modules] Add ability to specify module name to module file mapping (reapply)Boris Kolpackov2017-08-311-3/+13
| | | | | | | | | | | | | | | | | | | | Extend the -fmodule-file option to support the [<name>=]<file> value format. If the name is omitted, then the old semantics is preserved (the module file is loaded whether needed or not). If the name is specified, then the mapping is treated as just another prebuilt module search mechanism, similar to -fprebuilt-module-path, and the module file is only loaded if actually used (e.g., via import). With one exception: this mapping also overrides module file references embedded in other modules (which can be useful if module files are moved/renamed as often happens during remote compilation). This override semantics requires some extra work: we now store the module name in addition to the file name in the serialized AST representation. Reviewed By: rsmith Differential Revision: https://reviews.llvm.org/D35020 llvm-svn: 312220
* Revert r312105 [modules] Add ability to specify module name to module file ↵Victor Leschuk2017-08-301-13/+3
| | | | | | | | mapping Looks like it breaks win10 builder. llvm-svn: 312112
* [modules] Add ability to specify module name to module file mappingBoris Kolpackov2017-08-301-3/+13
| | | | | | | | | | | | | | | | | | | | Extend the -fmodule-file option to support the [<name>=]<file> value format. If the name is omitted, then the old semantics is preserved (the module file is loaded whether needed or not). If the name is specified, then the mapping is treated as just another prebuilt module search mechanism, similar to -fprebuilt-module-path, and the module file is only loaded if actually used (e.g., via import). With one exception: this mapping also overrides module file references embedded in other modules (which can be useful if module files are moved/renamed as often happens during remote compilation). This override semantics requires some extra work: we now store the module name in addition to the file name in the serialized AST representation. Reviewed By: rsmith Differential Revision: https://reviews.llvm.org/D35020 llvm-svn: 312105
* Spelling mistakes in comments. NFCI. (PR27635)Simon Pilgrim2017-03-301-1/+1
| | | | llvm-svn: 299083
* Reapply "Modules: Cache PCMs in memory and avoid a use-after-free"Duncan P. N. Exon Smith2017-03-201-7/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit r298185, effectively reapplying r298165, after fixing the new unit tests (PR32338). The memory buffer generator doesn't null-terminate the MemoryBuffer it creates; this version of the commit informs getMemBuffer about that to avoid the assert. Original commit message follows: ---- Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298278
* Revert "Modules: Cache PCMs in memory and avoid a use-after-free"Renato Golin2017-03-181-20/+7
| | | | | | This reverts commit r298165, as it broke the ARM builds. llvm-svn: 298185
* Modules: Cache PCMs in memory and avoid a use-after-freeDuncan P. N. Exon Smith2017-03-171-7/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298165
* [modules] Load the ModuleOffsetMap from the module header lazily.Richard Smith2017-02-181-4/+3
| | | | | | | | If we never need to map any ID within the module to its global ID, we don't need the module offset map. If a compilation transitively depends on lots of unused module files, this can result in a modest performance improvement. llvm-svn: 295517
* Modules: Fix a minor performance bug from r293393Duncan P. N. Exon Smith2017-01-291-3/+4
| | | | | | | | | | | | Oops... r293393 started calling ReadSignature in ModuleManager::addModule even when there was no ExpectedSignature. Whether or not this would have a measurable performance impact (I spotted this by inspection, and ReadSignature should be fairly fast), we might as well get what we can. Add an extra check against ExpectedSignature to avoid the hit. llvm-svn: 293415
* Modules: Return early in ModuleManager::addModule; NFCDuncan P. N. Exon Smith2017-01-281-71/+72
| | | | | | | | | Invert the main branch in ModuleManager::addModule to return early and reduce indentation, and clean up a bunch of logic as a result. I split out a function called updateModuleImports to avoid triggering code duplication. llvm-svn: 293400
* Modules: Clean up ModuleFile::Imports in ModuleManager::removeModulesDuncan P. N. Exon Smith2017-01-281-3/+3
| | | | | | | | | | I don't have a testcase for this (and I'm not sure if it's an observable bug), but it seems obviously wrong that ModuleManager::removeModules is failing to clean up deleted modules from ModuleFile::Imports. See the code in ModuleManager::addModule that inserts into ModuleFile::Imports; we need the inverse operation. llvm-svn: 293399
* Modules: Enforce that ModuleManager::removeModules deletes the tailDuncan P. N. Exon Smith2017-01-281-11/+13
| | | | | | | | | | | | | | | ModuleManager::removeModules always deletes a tail of the ModuleManager::Chain. Change the API to enforce that so that we can simplify the code inside. There's no real functionality change, although there's a slight performance hack to loop to the First deleted module instead of the final module in the chain (skipping the about-to-be-deleted tail). Also document something suspicious: we fail to clean deleted modules out of ModuleFile::Imports. llvm-svn: 293398
* Modules: Clarify ownership of ModuleFile instances in ModuleManager, NFCDuncan P. N. Exon Smith2017-01-281-34/+25
| | | | | | | Use std::unique_ptr to clarify the ownership of the ModuleFile instances in ModuleManager. llvm-svn: 293395
* Modules: Return ModuleFile& from ModuleManager::begin, etc.; NFCDuncan P. N. Exon Smith2017-01-281-16/+19
| | | | | | | | Hide the pointer indirection in ModuleManager::begin, ModuleManager::end, ModuleManager::rbegin, and ModuleManager::rend. Besides tidying up the call sites, this is preparation for making ownership of ModuleFile explicit. llvm-svn: 293394
* Modules: Separate out a checkSignature helper, almost NFCDuncan P. N. Exon Smith2017-01-281-13/+16
| | | | | | | | | | | | The main point is to move the delete-the-new-module logic into the same block that creates it, so I can simplify the memory management in a follow-up, but I think it's clearer to use use a checkSignature helper here anyway. There is a minor functionality change: we now scan ahead to pull the signature out of the control block *only* if this is a new ModuleFile. For old ones, ASTReader::ReadControlBlock will have already read the signature. llvm-svn: 293393
* [VFS] Replace TimeValue usage with std::chronoPavel Labath2016-11-091-1/+1
| | | | | | | | | | | | Summary: NFCI Reviewers: benlangmuir, zturner Subscribers: cfe-commits Differential Revision: https://reviews.llvm.org/D25948 llvm-svn: 286356
* Bitcode: Decouple block info block state from reader.Peter Collingbourne2016-11-081-3/+2
| | | | | | | | | | | | | | | | | As proposed on llvm-dev: http://lists.llvm.org/pipermail/llvm-dev/2016-October/106630.html Move block info block state to a new class, BitstreamBlockInfo. Clients may set the block info for a particular cursor with the BitstreamCursor::setBlockInfo() method. At this point BitstreamReader is not much more than a container for an ArrayRef<uint8_t>, so remove it and replace all uses with direct uses of memory buffers. Differential Revision: https://reviews.llvm.org/D26259 llvm-svn: 286207
* Clean up handling of reading module files from stdin. Don't bother trying toRichard Smith2016-09-021-3/+6
| | | | | | | | | look for a corresponding file, since we're not going to read it anyway. No observable behavior change (though we now avoid pointlessly trying to stat or open a file named "-"). llvm-svn: 280436
* Refactor to avoid holding a reference to a container element that could go awayRichard Smith2016-09-021-40/+36
| | | | | | | | during this function, and to avoid rolling back changes to the module manager's data structures. Instead, we defer registering the module file until after we have successfully finished loading it. llvm-svn: 280434
* [GraphTraits] Replace all NodeType usage with NodeRefTim Shen2016-08-221-4/+3
| | | | | | | | This should finish the GraphTraits migration. Differential Revision: http://reviews.llvm.org/D23730 llvm-svn: 279475
* Module: add -fprebuilt-module-path to support loading prebuilt modules.Manman Ren2016-08-181-1/+1
| | | | | | | | | | | | | In this mode, there is no need to load any module map and the programmer can simply use "@import" syntax to load the module directly from a prebuilt module path. When loading from prebuilt module path, we don't support rebuilding of the module files and we ignore compatible configuration mismatches. rdar://27290316 Differential Revision: http://reviews.llvm.org/D23125 llvm-svn: 279096
* [GraphWriter] Change GraphWriter to use NodeRef in GraphTraitsTim Shen2016-08-171-0/+1
| | | | | | | | | | | | Summary: Corresponding LLVM patch: D23580 Reviewers: dblaikie Subscribers: cfe-commits Differential Revision: https://reviews.llvm.org/D23581 llvm-svn: 278963
* [NFC] Header cleanupMehdi Amini2016-07-181-2/+1
| | | | | | | | | | Summary: Removed unused headers, replaced some headers with forward class declarations Patch by: Eugene <claprix@yandex.ru> Differential Revision: https://reviews.llvm.org/D20100 llvm-svn: 275882
* Use ranges to concisely express iterationDavid Majnemer2016-06-231-4/+4
| | | | | | | No functional change is intended, this should just clean things up a little. llvm-svn: 273522
* Fix use-after-free in ModuleManagerBen Langmuir2015-10-211-0/+3
| | | | | | | | | | | | | When removing out-of-date modules we might have left behind a VisitOrder that contains pointers to freed ModuleFiles. This was very rarely seen, because it only happens when modules go out of date and the VisitOrder happens to have the right size to not be recomputed. Thanks ASan! rdar://23181512 llvm-svn: 250963
* [modules] Remove now-dead code for lazy loading of files specified by ↵Richard Smith2015-08-091-11/+0
| | | | | | -fmodule-file=. llvm-svn: 244417
* [modules] Remove unused ModuleManager::visitDepthFirst function.Richard Smith2015-08-061-65/+0
| | | | llvm-svn: 244289
* [Modules] Wrap the main ModuleManager visitor in a function_ref.Benjamin Kramer2015-07-251-5/+3
| | | | | | | Avoids the awkward passing of an opaque void *UserData argument. No functional change intended. llvm-svn: 243213
* Fix dumb use-after-free bug introduced in r242868.Richard Smith2015-07-221-9/+9
| | | | llvm-svn: 242960
* [modules] Stop performing PCM lookups for all identifiers when building with ↵Richard Smith2015-07-221-0/+13
| | | | | | C++ modules. Instead, serialize a list of interesting identifiers and mark those ones out of date on module import. Avoiding the identifier lookups here gives a 20-30% speedup in builds with large numbers of modules. No functionality change intended. llvm-svn: 242868
* [modules] Change module manager visitation order to be a bit more stable whenRichard Smith2015-07-221-13/+9
| | | | | | | | | | | more modules are added: visit modules depth-first rather than breadth-first. The visitation is still (approximately) oldest-to-newest, and still guarantees that a module is visited before anything it imports, so modules that are imported by others sometimes need to jump to a later position in the visitation order when more modules are loaded, but independent module trees don't interfere with each other any more. llvm-svn: 242863
* Make the clang module container format selectable from the command line.Adrian Prantl2015-07-171-3/+3
| | | | | | | | | | | | | - introduces a new cc1 option -fmodule-format=[raw,obj] with 'raw' being the default - supports arbitrary module container formats that libclang is agnostic to - adds the format to the module hash to avoid collisions - splits the old PCHContainerOperations into PCHContainerWriter and a PCHContainerReader. Thanks to Richard Smith for reviewing this patch! llvm-svn: 242499
* Revert r240270 ("Fixed/added namespace ending comments using clang-tidy").Alexander Kornienko2015-06-221-1/+1
| | | | llvm-svn: 240353
* Fixed/added namespace ending comments using clang-tidy. NFCAlexander Kornienko2015-06-221-1/+1
| | | | | | | | | | | | The patch is generated using this command: $ tools/extra/clang-tidy/tool/run-clang-tidy.py -fix \ -checks=-*,llvm-namespace-comment -header-filter='llvm/.*|clang/.*' \ work/llvm/tools/clang To reduce churn, not touching namespaces spanning less than 10 lines. llvm-svn: 240270
* Introduce a PCHContainerOperations interface (NFC).Adrian Prantl2015-06-201-6/+8
| | | | | | | | | | | | | | | | A PCHContainerOperations abstract interface provides operations for creating and unwrapping containers for serialized ASTs (precompiled headers and clang modules). The default implementation is RawPCHContainerOperations, which uses a flat file for the output. The main application for this interface will be an ObjectFilePCHContainerOperations implementation that uses LLVM to wrap the module in an ELF/Mach-O/COFF container to store debug info alongside the AST. rdar://problem/20091852 llvm-svn: 240225
* Allow skipping imports in the module visitor.Manuel Klimek2015-05-201-22/+57
| | | | | | | Skip imports when we know that we do not need to visit any imports because we've already deserialized the redecls from a module. llvm-svn: 237782
* A couple of readASTFileSignature improvements (NFC)Ben Langmuir2015-03-241-2/+1
| | | | | | | | | | * Strength reduce a std::function to a function pointer, * Factor out checking the AST file magic number, * Add a brief doc comment to readAStFileSignature Thanks to Chandler for spotting these oddities. llvm-svn: 233050
* Make module files passed to a module build via -fmodule-file= available toRichard Smith2015-03-181-0/+11
| | | | | | | | | consumers of that module. Previously, such a file would only be available if the module happened to actually import something from that module. llvm-svn: 232583
* Revert "Wrap clang module files in a Mach-O, ELF, or COFF container."Adrian Prantl2015-02-251-5/+4
| | | | llvm-svn: 230454
* Wrap clang module files in a Mach-O, ELF, or COFF container.Adrian Prantl2015-02-251-4/+5
| | | | | | | | | | | | | | | | | This is a necessary prerequisite for debugging with modules. The .pcm files become containers that hold the serialized AST which allows us to store debug information in the module file that can be shared by all object files that were built importing the module. This reapplies r230044 with a fixed configure+make build and updated dependencies and testcase requirements. Over the last iteration this version adds - missing target requirements for testcases that specify an x86 triple, - a missing clangCodeGen.a dependency to libClang.a in the make build. rdar://problem/19104245 llvm-svn: 230423
* Revert "Wrap clang module files in a Mach-O, ELF, or COFF container."Adrian Prantl2015-02-241-5/+4
| | | | | | | This reverts commit r230305. Off to fix another round of missing dependencies on various platforms. llvm-svn: 230309
* Wrap clang module files in a Mach-O, ELF, or COFF container.Adrian Prantl2015-02-241-4/+5
| | | | | | | | | | | | | | This is a necessary prerequisite for debugging with modules. The .pcm files become containers that hold the serialized AST which allows us to store debug information in the module file that can be shared by all object files that were built importing the module. rdar://problem/19104245 This reapplies r230044 with a fixed configure+make build and updated dependencies. Take 3. llvm-svn: 230305
* Revert "Wrap clang module files in a Mach-O, ELF, or COFF container."Adrian Prantl2015-02-211-5/+4
| | | | | | | | This reverts commit 230099. The Linux configure+make build variant still needs some work. llvm-svn: 230103
* Wrap clang module files in a Mach-O, ELF, or COFF container.Adrian Prantl2015-02-201-4/+5
| | | | | | | | | | | | | | This is a necessary prerequisite for debugging with modules. The .pcm files become containers that hold the serialized AST which allows us to store debug information in the module file that can be shared by all object files that were built importing the module. rdar://problem/19104245 This reapplies r230044 with a fixed configure+make build and updated dependencies. Take 2. llvm-svn: 230089
* Revert "Wrap clang module files in a Mach-O, ELF, or COFF container."Adrian Prantl2015-02-201-5/+4
| | | | | | | | This reverts commit r230067. Investigating another batch of problems found by the bots. llvm-svn: 230073
* Wrap clang module files in a Mach-O, ELF, or COFF container.Adrian Prantl2015-02-201-4/+5
| | | | | | | | | | | | | | This is a necessary prerequisite for debugging with modules. The .pcm files become containers that hold the serialized AST which allows us to store debug information in the module file that can be shared by all object files that were built importing the module. rdar://problem/19104245 This reapplies r230044 with a fixed configure+make build and updated dependencies. llvm-svn: 230067
* Revert "Wrap clang module files in a Mach-O, ELF, or COFF container."Adrian Prantl2015-02-201-5/+4
| | | | | | | | | This reverts commit r230044 while dealing with buildbot breakage. Conflicts: test/Modules/module_container.m llvm-svn: 230052
OpenPOWER on IntegriCloud