summaryrefslogtreecommitdiffstats
path: root/clang/unittests/Lex/LexerTest.cpp
Commit message (Collapse)AuthorAgeFilesLines
* [clang] Add test for FindNextToken in Lexer.Utkarsh Saxena2019-10-071-2/+23
| | | | | | | | | | | | Reviewers: ilya-biryukov Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D68565 llvm-svn: 373910
* [Clang] Migrate llvm::make_unique to std::make_uniqueJonas Devlieghere2019-08-141-1/+1
| | | | | | | | | | Now that we've moved to C++14, we no longer need the llvm::make_unique implementation from STLExtras.h. This patch is a mechanical replacement of (hopefully) all the llvm::make_unique instances across the monorepo. Differential revision: https://reviews.llvm.org/D66259 llvm-svn: 368942
* Remove cache for macro arg stringizationReid Kleckner2019-07-301-5/+8
| | | | | | | | | | | | | | | | | | | Summary: The cache recorded the wrong expansion location for all but the first stringization. It seems uncommon to stringize the same macro argument multiple times, so this cache doesn't seem that important. Fixes PR39942 Reviewers: vsk, rsmith Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D65428 llvm-svn: 367337
* Re-apply r357823 "[Lexer] NFC: Fix an off-by-one bug in getAsCharRange()."Artem Dergachev2019-04-231-0/+19
| | | | | | | | | It now comes with a follow-up fix for the clients of this API in clangd and clang-tidy. Differential Revision: https://reviews.llvm.org/D59977 llvm-svn: 359035
* Revert "[Lexer] NFC: Fix an off-by-one bug in getAsCharRange()."Artem Dergachev2019-04-051-19/+0
| | | | | | | | | | This reverts commit r357823. Was breaking clang-tidy! Differential Revision: https://reviews.llvm.org/D59977 llvm-svn: 357827
* [Lexer] NFC: Fix an off-by-one bug in getAsCharRange().Artem Dergachev2019-04-051-0/+19
| | | | | | | | | | | | | | As the unit test demonstrates, subtracting 1 from the offset was unnecessary. The only user of this function was the plist file emitter (in Static Analyzer and ARCMigrator). It means that a lot of Static Analyzer's plist arrows are in fact off by one character. The patch carefully preserves this completely incorrect behavior and causes no functional change, i.e. no plist format breakage. Differential Revision: https://reviews.llvm.org/D59977 llvm-svn: 357823
* Modules: Rename MemoryBufferCache to InMemoryModuleCacheDuncan P. N. Exon Smith2019-03-091-3/+1
| | | | | | | | | | | | | Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to Serialization. Another patch will start using it to manage module build more explicitly, but this is split out because it's mostly mechanical. Because of the move to Serialization we can no longer abuse the Preprocessor to forward it to the ASTReader. Besides the rename and file move, that means Preprocessor::Preprocessor has one fewer parameter and ASTReader::ASTReader has one more. llvm-svn: 355777
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* PR37189 Fix incorrect end source location and spelling for a split '>>' token.Richard Smith2018-04-301-5/+4
| | | | | | | | | | | | | | | | | | When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an extension) when we do the same for other tokens starting with a '>', we can't just use a location pointing to the first '>' as the location of the split token, because that would result in our miscomputing the length and spelling for the token. As a consequence, for example, a refactoring replacing 'A<X>' with something else would sometimes replace one character too many, and similarly diagnostics highlighting a template-id source range would highlight one character too many. Fix this by creating an expansion range covering the first character of the '>>' token, whose spelling is '>'. For this to work, we generalize the expansion range of a macro FileID to be either a token range (the common case) or a character range (used in this new case). llvm-svn: 331155
* [Lex] Avoid out-of-bounds dereference in LexAngledStringLiteral.Volodymyr Sapsai2018-01-121-0/+2
| | | | | | | | | | | | | | | | | | | | | | Fix makes the loop in LexAngledStringLiteral more like the loops in LexStringLiteral, LexCharConstant. When we skip a character after backslash, we need to check if we reached the end of the file instead of reading the next character unconditionally. Discovered by OSS-Fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3832 rdar://problem/35572754 Reviewers: arphaman, kcc, rsmith, dexonsmith Reviewed By: rsmith, dexonsmith Subscribers: cfe-commits, rsmith, dexonsmith Differential Revision: https://reviews.llvm.org/D41423 llvm-svn: 322390
* [Lex] Inline a variable in test in preparation for more similar tests. NFC.Volodymyr Sapsai2018-01-101-2/+1
| | | | llvm-svn: 322240
* Stringizing raw string literals containing newlineTaewook Oh2017-12-061-1/+39
| | | | | | | | | | | | | | Summary: This patch implements 4.3 of http://open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4220.pdf. If a raw string contains a newline character, replace each newline character with the \n escape code. Without this patch, included test case (macro_raw_string.cpp) results compilation failure. Reviewers: rsmith, doug.gregor, jkorous-apple Reviewed By: jkorous-apple Subscribers: jkorous-apple, vsapsai, cfe-commits Differential Revision: https://reviews.llvm.org/D39279 llvm-svn: 319904
* [Lex] Avoid out-of-bounds dereference in SkipLineCommentAlex Lorenz2017-10-141-0/+5
| | | | | | | | | Credit to OSS-Fuzz for discovery: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3145 rdar://34526482 llvm-svn: 315785
* [Lexer] Finding beginning of token with escaped new lineAlexander Kornienko2017-08-101-0/+53
| | | | | | | | | | | | | | | | | | | | Summary: Lexer::GetBeginningOfToken produced invalid location when backtracking across escaped new lines. This fixes PR26228 Reviewers: akyrtzi, alexfh, rsmith, doug.gregor Reviewed By: alexfh Subscribers: alexfh, cfe-commits Patch by Paweł Żukowski! Differential Revision: https://reviews.llvm.org/D30748 llvm-svn: 310576
* [NFC] Refactor the Preprocessor function that handles Macro definitions and ↵Faisal Vali2017-07-171-3/+3
| | | | | | | | | | | | | | | | | rename Arguments to Parameters in Macro Definitions. - Extracted the reading of the tokens out into a separate function. - Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation). This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible. I will also directly update some extra clang tooling that is broken by the change from Argument to Parameter. Hopefully the bots will stay appeased. Thanks! llvm-svn: 308190
* Revert changes from my previous refactoring - will need to fix dependencies ↵Faisal Vali2017-07-171-3/+3
| | | | | | | | in clang's extra tooling (such as clang-tidy etc.). Sorry about that. llvm-svn: 308158
* [NFC] Refactor the Preprocessor function that handles Macro definitions and ↵Faisal Vali2017-07-171-3/+3
| | | | | | | | | | | | | | rename Arguments to Parameters in Macro Definitions. - Extracted the reading of the tokens out into a separate function. - Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation). This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible. Thanks! llvm-svn: 308157
* LexerTest memory leak fix-Erich Keane2017-06-151-1/+3
| | | | | | | | A new LexerTest unittest introduced a memory leak. This patch uses a unique_ptr with a custom deleter to ensure it is properly deleted. llvm-svn: 305491
* Fix LexerTest signed/unsigned comparison.Erich Keane2017-06-151-1/+1
| | | | | | | | Werror was catching a signed/unsigned compare in an assert, correct the signed 'expected' value to be unsigned. llvm-svn: 305435
* [Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, Erich Keane2017-06-141-9/+62
| | | | | | | | | | | | | | | | | | | | | correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
* Add #pragma clang module build/endbuild pragmas for performing a module buildRichard Smith2017-06-091-19/+1
| | | | | | | | | | | | | | | as part of a compilation. This is intended for two purposes: 1) Writing self-contained test cases for modules: we can now write a single source file test that builds some number of module files on the side and imports them. 2) Debugging / test case reduction. A single-source testcase is much more amenable to reduction, compared to a VFS tarball or .pcm files. llvm-svn: 305101
* Reapply "Modules: Cache PCMs in memory and avoid a use-after-free"Duncan P. N. Exon Smith2017-03-201-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit r298185, effectively reapplying r298165, after fixing the new unit tests (PR32338). The memory buffer generator doesn't null-terminate the MemoryBuffer it creates; this version of the commit informs getMemBuffer about that to avoid the assert. Original commit message follows: ---- Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298278
* Revert "Modules: Cache PCMs in memory and avoid a use-after-free"Renato Golin2017-03-181-4/+1
| | | | | | This reverts commit r298165, as it broke the ARM builds. llvm-svn: 298185
* Modules: Cache PCMs in memory and avoid a use-after-freeDuncan P. N. Exon Smith2017-03-171-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298165
* shared_ptrify (from InclusiveRefCntPtr) HeaderSearchOptionsDavid Blaikie2017-01-061-2/+2
| | | | llvm-svn: 291202
* Move PreprocessorOptions to std::shared_ptr from IntrusiveRefCntPtrDavid Blaikie2017-01-051-2/+2
| | | | llvm-svn: 291160
* [Lexer] Don't merge macro args from different macro filesVedant Kumar2016-05-191-2/+24
| | | | | | | | | | | | | | | | | | The lexer sets the end location of macro arguments incorrectly *if*, while merging consecutive args to fit into a single SLocEntry, it finds args which come from different macro files. Fix the issue by using separate SLocEntries in this situation. This fixes a code coverage crasher (rdar://problem/26181005). Because the lexer reported end locations for certain macro args incorrectly, we would generate bogus coverage mappings with negative line offsets. Reviewed-by: akyrtzi Differential Revision: http://reviews.llvm.org/D20401 llvm-svn: 270160
* Make TargetInfo store an actual DataLayout instead of a string.James Y Knight2016-03-041-2/+2
| | | | | | | | | | | | | | Use it to calculate UserLabelPrefix, instead of specifying it (often incorrectly). Note that the *actual* user label prefix has always come from the DataLayout, and is handled within LLVM. The main thing clang's TargetInfo::UserLabelPrefix did was to set the #define value. Having these be different from each-other is just silly. Differential Revision: http://reviews.llvm.org/D17183 llvm-svn: 262737
* Fix -Wextra-semi warnings.Hans Wennborg2015-07-221-1/+1
| | | | | | | | Patch by Eugene Zelenko! Differential Revision: http://reviews.llvm.org/D11401 llvm-svn: 242931
* [modules] Start moving the module visibility information off the Module itself.Richard Smith2015-05-011-2/+1
| | | | | | | It has no place there; it's not a property of the Module, and it makes restoring the visibility set when we leave a submodule more difficult. llvm-svn: 236300
* unique_ptrify SourceManager::createFileIDDavid Blaikie2014-08-291-1/+1
| | | | llvm-svn: 216715
* Update for LLVM api change.Rafael Espindola2014-08-271-2/+2
| | | | llvm-svn: 216585
* Use non-intrusive refcounting for TargetOptionsAlp Toker2014-07-061-2/+2
| | | | llvm-svn: 212388
* Track IntrusiveRefCntPtr::get() changes from LLVM r212366Alp Toker2014-07-051-1/+1
| | | | llvm-svn: 212369
* [C++11] Use 'nullptr'. Unittests edition.Craig Topper2014-06-081-2/+2
| | | | llvm-svn: 210423
* Remove the last remaining llvm/Config/config.h includesAlp Toker2014-06-041-1/+0
| | | | | | | | | | | | This corrects long-standing misuses of LLVM's internal config.h. In most cases the public llvm-config.h header was intended and we can now remove the old hacks thanks to LLVM r210144. The config.h header is private, won't be installed and should no longer be included by clang or other modules. llvm-svn: 210145
* SourceManager: Use setMainFileID() consistentlyAlp Toker2014-05-211-1/+1
| | | | | | | | | | | Eliminate createMainFileID() / createMainFileIDForMemBuffer() utility functions. These didn't add much convenience and conflated two distinct operations. This change makes things easier to follow by providing a consistent interface and getting rid of a bunch of cast-to-voids. llvm-svn: 209266
* Reformat code following Preprocessor constructor updatesAlp Toker2014-05-021-3/+3
| | | | | | Landing this separately to make the previous commits easy to follow at home. llvm-svn: 207826
* Factor TargetInfo pointer/DelayInitialization bool pair out of Preprocessor ctorAlp Toker2014-05-021-3/+3
| | | | | | | | The Preprocessor::Initialize() function already offers a clear interface to achieve this, further reducing the confusing number of states a newly constructed preprocessor can have. llvm-svn: 207825
* Quick fix for layering that broke shared library build.John Thompson2014-04-231-9/+11
| | | | llvm-svn: 207011
* Initial implementation of -modules-earch-all option, for searching for ↵John Thompson2014-04-231-0/+3
| | | | | | symbols in non-imported modules. llvm-svn: 206977
* Use the same SourceManager for ModuleMaps and compilations.Manuel Klimek2013-10-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows using virtual file mappings on the original SourceManager to map in virtual module.map files. Without this patch, the ModuleMap search will find a module.map file (as the FileEntry exists in the FileManager), but will be unable to get the content from the SourceManager (as ModuleMap previously created its own SourceManager). Two problems needed to be fixed which this patch exposed: 1. Storing the inferred module map When writing out a module, the ASTWriter stores the names of the files in the main source manager; when loading the AST again, the ASTReader errs out if such a file is found missing, unless it is overridden. Previously CompilerInstance's compileModule method would store the inferred module map to a temporary file; the problem with this approach is that now that the module map is handled by the main source manager, the ASTWriter stores the name of the temporary module map as source to the compilation; later, when the module is loaded, the temporary file has already been deleted, which leads to a compilation error. This patch changes the inferred module map to instead inject a virtual file into the source manager. This both saves some disk IO, and works with how the ASTWriter/ASTReader handle overridden source files. 2. Changing test input in test/Modules/Inputs/* Now that the module map file is handled by the main source manager, the VerifyDiagnosticConsumer will not ignore diagnostics created while parsing the module map file. The module test test/Modules/renamed.m uses -I test/Modules/Inputs and triggers recursive loading of all module maps in test/Modules/Inputs, some of which had conflicting names, thus leading errors while parsing the module maps. Those diagnostics already occur on trunk, but before this patch they would not break the test, as they were ignored by the VerifyDiagnosticConsumer. This patch thus changes the module maps that have been recently introduced which broke the invariant of compatible modules maps in test/Modules/Inputs. llvm-svn: 193314
* [Lexer] Improve Lexer::getSourceText() when the given range deals with ↵Argyrios Kyrtzidis2013-05-161-55/+225
| | | | | | | | function macro arguments. This is a modified version of a patch by Manuel Klimek. llvm-svn: 182055
* <rdar://problem/12368093> Extend module maps with a 'conflict' declaration, ↵Douglas Gregor2013-03-201-1/+2
| | | | | | and warn when a newly-imported module conflicts with an already-imported module. llvm-svn: 177577
* For ModuleLoader::makeModuleVisible() also pass the source location where theArgyrios Kyrtzidis2013-02-011-1/+2
| | | | | | module import occurred. llvm-svn: 174191
* clang/unittests: Fixup corresponding to r172290.NAKAMURA Takumi2013-01-121-0/+3
| | | | llvm-svn: 172295
* Really sort the #include lines in unittests/...Chandler Carruth2012-12-041-1/+1
| | | | | | I forgot to re-sort after fixing main module headers. llvm-svn: 169244
* Sort the #include lines for unittests/...Chandler Carruth2012-12-041-5/+5
| | | | | | I've tried to place sensible headers at the top as main-module headers. llvm-svn: 169243
* Fix unit tests for ModuleLoader change in r168961.Douglas Gregor2012-11-301-4/+5
| | | | llvm-svn: 168962
* clang/unittests: Fixup corresponding to Doug's r168136.NAKAMURA Takumi2012-11-161-1/+1
| | | | llvm-svn: 168137
OpenPOWER on IntegriCloud