summaryrefslogtreecommitdiffstats
path: root/clang/unittests/Lex
Commit message (Collapse)AuthorAgeFilesLines
* [clang-scan-deps] do not skip empty #if/#elif in the minimizer to avoid ↵Alex Lorenz2019-12-021-3/+14
| | | | | | | | | | missing `__has_include` dependencies This patch makes the minimizer more conservative to avoid missing dependency files that are brought in by __has_include PP expressions that occur in a condition of an #if/#elif that was previously skipped. The __has_include PP expressions can be used in an #if/#elif either directly, or through macro expansion, so we can't detect them at the time of minimization. Differential Revision: https://reviews.llvm.org/D70936
* [clang-scan-deps] Improve string/character literal skippingAlex Lorenz2019-10-081-0/+44
| | | | | | | | | | | | | | | | The existing string/character literal skipping code in the dependency directives source minimizer has two issues: - It doesn't stop the scanning when a newline is reached before the terminating character, unlike the lexer which considers the token to be done (even if it's invalid) at the end of the line. - It doesn't support whitespace between '\' and the newline when looking if the '\' is used as a line continuation character. This commit fixes both issues. Differential Revision: https://reviews.llvm.org/D68436 llvm-svn: 374127
* [clang] Add test for FindNextToken in Lexer.Utkarsh Saxena2019-10-071-2/+23
| | | | | | | | | | | | Reviewers: ilya-biryukov Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D68565 llvm-svn: 373910
* [clang-scan-deps] Allow continuation line backslashes followed by whitespaceAlex Lorenz2019-09-261-4/+15
| | | | | | | | | | | | in the dependency source minimizer Clang allows continuations that have whitespace between the backslash and the newline. This patch ensures that the dependency source minimizer can handle the whitespace between the backslash and the newline when looking for a line continuation. Differential Revision: https://reviews.llvm.org/D68052 llvm-svn: 373007
* Fix for stringized function-macro args continued across linesAlex Lorenz2019-09-191-8/+18
| | | | | | | | | | | In case of certain #define'd macros, there's a space just before line continuation that the minimized-source lexer was missing to include, resulting in invalid stringize. Patch by: kousikk (Kousik Kumar) Differential Revision: https://reviews.llvm.org/D67635 llvm-svn: 372360
* [clang-scan-deps] add skip excluded conditional preprocessor block ↵Alex Lorenz2019-09-111-0/+42
| | | | | | | | | | | | | | | preprocessing optimization This commit adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preprocessor blocks by bumping the lexer pointer, and not lexing the tokens until reaching appropriate #else/#endif directive. The skip positions and lexer offsets are computed when the file is minimized, directly from the minimized tokens. On an 18-core iMacPro with macOS Catalina Beta I got 10-15% speed-up from this optimization when running clang-scan-deps on the compilation database for a recent LLVM and Clang (3511 files). Differential Revision: https://reviews.llvm.org/D67127 llvm-svn: 371656
* Introduce a DirectoryEntryRef that stores both a reference and anAlex Lorenz2019-08-312-2/+2
| | | | | | | | | | | | | | accessed name to the directory entry This commit introduces a parallel API that returns a DirectoryEntryRef to the FileManager, similar to the parallel FileEntryRef API. All uses will have to be update in follow-up patches. The immediate use of the new API in this patch fixes the issue where a file manager was reused in clang-scan-deps, but reported an different file path whenever a framework lookup was done through a symlink. Differential Revision: https://reviews.llvm.org/D67026 llvm-svn: 370562
* [Clang] Migrate llvm::make_unique to std::make_uniqueJonas Devlieghere2019-08-141-1/+1
| | | | | | | | | | Now that we've moved to C++14, we no longer need the llvm::make_unique implementation from STLExtras.h. This patch is a mechanical replacement of (hopefully) all the llvm::make_unique instances across the monorepo. Differential revision: https://reviews.llvm.org/D66259 llvm-svn: 368942
* [clang-scan-deps] Add minimizer support for C++20 modules.Michael J. Spencer2019-08-091-2/+51
| | | | | | This only adds support to the minimizer, it doesn't actually capture the dependencies yet. llvm-svn: 368381
* [clang] Adopt new FileManager error-returning APIsHarlan Haskins2019-08-012-4/+4
| | | | | | | Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods. Signed-off-by: Harlan Haskins <harlan@apple.com> llvm-svn: 367616
* Remove cache for macro arg stringizationReid Kleckner2019-07-301-5/+8
| | | | | | | | | | | | | | | | | | | Summary: The cache recorded the wrong expansion location for all but the first stringization. It seems uncommon to stringize the same macro argument multiple times, so this cache doesn't seem that important. Fixes PR39942 Reviewers: vsk, rsmith Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D65428 llvm-svn: 367337
* [clang-scan-deps] Dependency directives source minimizer: handle #pragma onceAlex Lorenz2019-07-181-0/+24
| | | | | | | | | We should re-emit `#pragma once` to ensure the preprocessor will still honor it when running on minimized sources. Differential Revision: https://reviews.llvm.org/D64945 llvm-svn: 366509
* [clang-scan-deps] Dependency directives source minimizer:Alex Lorenz2019-07-101-0/+37
| | | | | | | | | | | | single quotes are not digit separators after a valid character literal prefix The single quote character can act as a c++ digit separator. However, the minimizer shouldn't treat it as such when it's actually following a valid character literal prefix, like L, U, u, or u8. Differential Revision: https://reviews.llvm.org/D64525 llvm-svn: 365700
* cmake: Add CLANG_LINK_CLANG_DYLIB optionTom Stellard2019-07-031-1/+1
| | | | | | | | | | | | | | | | Summary: Setting CLANG_LINK_CLANG_DYLIB=ON causes clang tools to link against libclang_shared.so instead of the individual component libraries. Reviewers: mgorny, beanz, smeenai, phosek, sylvestre.ledru Subscribers: arphaman, cfe-commits, llvm-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D63503 llvm-svn: 365092
* [clang][HeaderSearch] Shorten paths for includes in mainfile's directoryKadir Cetinkaya2019-07-031-9/+31
| | | | | | | | | | | | | | | | | | Summary: Currently HeaderSearch only looks at SearchDir's passed into it, but in addition to those paths headers can be relative to including file's directory. This patch makes sure that is taken into account. Reviewers: gribozavr Subscribers: jkorous, arphaman, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D63295 llvm-svn: 365005
* Fix for r42230, MSVC test failure in DependencyDirectivesSourceMinimizerTest.cppErich Keane2019-06-111-4/+6
| | | | | | | | | | | | r362459 introduced DependencyDirectivesSourceMinimizerTest.cpp, which hits an MSVC bug: developercommunity.visualstudio.com/content/problem/67300/stringifying-raw-string-literal.html This only happens when the parameter to a macro is stringified in the macro. This patch removes the string from the assert so that the warning no longer happens. llvm-svn: 363074
* [clang][HeaderSearch] Consider all path separators equalKadir Cetinkaya2019-06-061-0/+7
| | | | | | | | | | | | Reviewers: ilya-biryukov, sammccall Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D62965 llvm-svn: 362731
* Add clang source minimizer that reduces source to directivesAlex Lorenz2019-06-032-0/+509
| | | | | | | | | | | | | | | | | | | | | that might affect the dependency list for a compilation This commit introduces a dependency directives source minimizer to clang that minimizes header and source files to the minimum necessary preprocessor directives for evaluating includes. It reduces the source down to #define, #include, The source minimizer works by lexing the input with a custom fast lexer that recognizes the preprocessor directives it cares about, and emitting those directives in the minimized source. It ignores source code, comments, and normalizes whitespace. It gives up and fails if seems any directives that it doesn't recognize as valid (e.g. #define 0). In addition to the source minimizer this patch adds a -print-dependency-directives-minimized-source CC1 option that allows you to invoke the minimizer from clang directly. Differential Revision: https://reviews.llvm.org/D55463 llvm-svn: 362459
* [clang][HeaderSuggestion] Handle the case of dotdot with an absolute pathKadir Cetinkaya2019-04-241-0/+7
| | | | | | | | | | | | | | | | | Summary: Include insertion in clangd was inserting absolute paths when the include directory was an absolute path with a double dot. This patch makes sure double dots are handled both with absolute and relative paths. Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, jkorous, arphaman, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D60873 llvm-svn: 359078
* [clang][HeaderSearch] Make sure there are no backslashes in suggestedPathKadir Cetinkaya2019-04-241-0/+9
| | | | | | | | Reviewers: sammccall Differential Revision: https://reviews.llvm.org/D60995 llvm-svn: 359075
* Re-apply r357823 "[Lexer] NFC: Fix an off-by-one bug in getAsCharRange()."Artem Dergachev2019-04-231-0/+19
| | | | | | | | | It now comes with a follow-up fix for the clients of this API in clangd and clang-tidy. Differential Revision: https://reviews.llvm.org/D59977 llvm-svn: 359035
* Revert "[Lexer] NFC: Fix an off-by-one bug in getAsCharRange()."Artem Dergachev2019-04-051-19/+0
| | | | | | | | | | This reverts commit r357823. Was breaking clang-tidy! Differential Revision: https://reviews.llvm.org/D59977 llvm-svn: 357827
* [Lexer] NFC: Fix an off-by-one bug in getAsCharRange().Artem Dergachev2019-04-051-0/+19
| | | | | | | | | | | | | | As the unit test demonstrates, subtracting 1 from the offset was unnecessary. The only user of this function was the plist file emitter (in Static Analyzer and ARCMigrator). It means that a lot of Static Analyzer's plist arrows are in fact off by one character. The patch carefully preserves this completely incorrect behavior and causes no functional change, i.e. no plist format breakage. Differential Revision: https://reviews.llvm.org/D59977 llvm-svn: 357823
* Modules: Rename MemoryBufferCache to InMemoryModuleCacheDuncan P. N. Exon Smith2019-03-094-16/+7
| | | | | | | | | | | | | Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to Serialization. Another patch will start using it to manage module build more explicitly, but this is split out because it's mostly mechanical. Because of the move to Serialization we can no longer abuse the Preprocessor to forward it to the ASTReader. Besides the rename and file move, that means Preprocessor::Preprocessor has one fewer parameter and ASTReader::ASTReader has one more. llvm-svn: 355777
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-195-20/+15
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* Fix -Wsign-compare in new testsReid Kleckner2019-01-171-7/+7
| | | | llvm-svn: 351478
* Revert r351209 (which was a revert of r350891) with a fix.Aaron Ballman2019-01-171-2/+55
| | | | | | The test case had a parse error that was causing the condition string to be misreported. We now have better fallback code for error cases. llvm-svn: 351470
* Revert "Correct the source range returned from preprocessor callbacks."Benjamin Kramer2019-01-151-50/+8
| | | | | | | This reverts commit r350891. Also add a test case that would return an empty string with r350891. llvm-svn: 351209
* Silence -Wsign-compare in unittestsBjorn Pettersson2019-01-111-7/+7
| | | | llvm-svn: 350933
* Correct the source range returned from preprocessor callbacks.Aaron Ballman2019-01-101-1/+118
| | | | | | This adjusts the source range passed in to the preprocessor callbacks to only include the condition range itself, rather than all of the conditionally skipped tokens. llvm-svn: 350891
* Lift VFS from clang to llvm (NFC)Jonas Devlieghere2018-10-102-23/+23
| | | | | | | | | | | | | | | | | | | This patch moves the virtual file system form clang to llvm so it can be used by more projects. Concretely the patch: - Moves VirtualFileSystem.{h|cpp} from clang/Basic to llvm/Support. - Moves the corresponding unit test from clang to llvm. - Moves the vfs namespace from clang::vfs to llvm::vfs. - Formats the lines affected by this change, mostly this is the result of the added llvm namespace. RFC on the mailing list: http://lists.llvm.org/pipermail/llvm-dev/2018-October/126657.html Differential revision: https://reviews.llvm.org/D52783 llvm-svn: 344140
* Reland '[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective'Julie Hockett2018-05-101-11/+47
| | | | | | | | | | | | | | This commit relands r331904. Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective in PPCallbacks, and updating calls to that function. This will be useful in https://reviews.llvm.org/D43778 to determine which includes are system headers. Differential Revision: https://reviews.llvm.org/D46614 llvm-svn: 332021
* Revert "[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective"Julie Hockett2018-05-091-30/+11
| | | | | | This reverts commit r331904 because of a memory leak. llvm-svn: 331932
* [clang] Adding CharacteristicKind to PPCallbacks::InclusionDirectiveJulie Hockett2018-05-091-11/+30
| | | | | | | | | | | Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective in PPCallbacks, and updating calls to that function. This will be useful in https://reviews.llvm.org/D43778 to determine which includes are system headers. Differential Revision: https://reviews.llvm.org/D46614 llvm-svn: 331904
* PR37189 Fix incorrect end source location and spelling for a split '>>' token.Richard Smith2018-04-301-5/+4
| | | | | | | | | | | | | | | | | | When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an extension) when we do the same for other tokens starting with a '>', we can't just use a location pointing to the first '>' as the location of the split token, because that would result in our miscomputing the length and spelling for the token. As a consequence, for example, a refactoring replacing 'A<X>' with something else would sometimes replace one character too many, and similarly diagnostics highlighting a template-id source range would highlight one character too many. Fix this by creating an expansion range covering the first character of the '>>' token, whose spelling is '>'. For this to work, we generalize the expansion range of a macro FileID to be either a token range (the common case) or a character range (used in this new case). llvm-svn: 331155
* [Lexer] Support adding working directory to relative search dir for #include ↵Eric Liu2018-01-292-0/+97
| | | | | | | | | | | | shortening in HeaderSearch. Reviewers: bkramer Subscribers: mgorny, hintonda, cfe-commits Differential Revision: https://reviews.llvm.org/D42577 llvm-svn: 323647
* [Lex] Avoid out-of-bounds dereference in LexAngledStringLiteral.Volodymyr Sapsai2018-01-121-0/+2
| | | | | | | | | | | | | | | | | | | | | | Fix makes the loop in LexAngledStringLiteral more like the loops in LexStringLiteral, LexCharConstant. When we skip a character after backslash, we need to check if we reached the end of the file instead of reading the next character unconditionally. Discovered by OSS-Fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3832 rdar://problem/35572754 Reviewers: arphaman, kcc, rsmith, dexonsmith Reviewed By: rsmith, dexonsmith Subscribers: cfe-commits, rsmith, dexonsmith Differential Revision: https://reviews.llvm.org/D41423 llvm-svn: 322390
* [Lex] Inline a variable in test in preparation for more similar tests. NFC.Volodymyr Sapsai2018-01-101-2/+1
| | | | llvm-svn: 322240
* Stringizing raw string literals containing newlineTaewook Oh2017-12-061-1/+39
| | | | | | | | | | | | | | Summary: This patch implements 4.3 of http://open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4220.pdf. If a raw string contains a newline character, replace each newline character with the \n escape code. Without this patch, included test case (macro_raw_string.cpp) results compilation failure. Reviewers: rsmith, doug.gregor, jkorous-apple Reviewed By: jkorous-apple Subscribers: jkorous-apple, vsapsai, cfe-commits Differential Revision: https://reviews.llvm.org/D39279 llvm-svn: 319904
* [CMake] Use PRIVATE in target_link_libraries for executablesShoaib Meenai2017-12-051-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We currently use target_link_libraries without an explicit scope specifier (INTERFACE, PRIVATE or PUBLIC) when linking executables. Dependencies added in this way apply to both the target and its dependencies, i.e. they become part of the executable's link interface and are transitive. Transitive dependencies generally don't make sense for executables, since you wouldn't normally be linking against an executable. This also causes issues for generating install export files when using LLVM_DISTRIBUTION_COMPONENTS. For example, clang has a lot of LLVM library dependencies, which are currently added as interface dependencies. If clang is in the distribution components but the LLVM libraries it depends on aren't (which is a perfectly legitimate use case if the LLVM libraries are being built static and there are therefore no run-time dependencies on them), CMake will complain about the LLVM libraries not being in export set when attempting to generate the install export file for clang. This is reasonable behavior on CMake's part, and the right thing is for LLVM's build system to explicitly use PRIVATE dependencies for executables. Unfortunately, CMake doesn't allow you to mix and match the keyword and non-keyword target_link_libraries signatures for a single target; i.e., if a single call to target_link_libraries for a particular target uses one of the INTERFACE, PRIVATE, or PUBLIC keywords, all other calls must also be updated to use those keywords. This means we must do this change in a single shot. I also fully expect to have missed some instances; I tested by enabling all the projects in the monorepo (except dragonegg), and configuring both with and without shared libraries, on both Darwin and Linux, but I'm planning to rely on the buildbots for other configurations (since it should be pretty easy to fix those). Even after this change, we still have a lot of target_link_libraries calls that don't specify a scope keyword, mostly for shared libraries. I'm thinking about addressing those in a follow-up, but that's a separate change IMO. Differential Revision: https://reviews.llvm.org/D40823 llvm-svn: 319840
* [Lex] Avoid out-of-bounds dereference in SkipLineCommentAlex Lorenz2017-10-141-0/+5
| | | | | | | | | Credit to OSS-Fuzz for discovery: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3145 rdar://34526482 llvm-svn: 315785
* [Lexer] Finding beginning of token with escaped new lineAlexander Kornienko2017-08-101-0/+53
| | | | | | | | | | | | | | | | | | | | Summary: Lexer::GetBeginningOfToken produced invalid location when backtracking across escaped new lines. This fixes PR26228 Reviewers: akyrtzi, alexfh, rsmith, doug.gregor Reviewed By: alexfh Subscribers: alexfh, cfe-commits Patch by Paweł Żukowski! Differential Revision: https://reviews.llvm.org/D30748 llvm-svn: 310576
* [NFC] Refactor the Preprocessor function that handles Macro definitions and ↵Faisal Vali2017-07-171-3/+3
| | | | | | | | | | | | | | | | | rename Arguments to Parameters in Macro Definitions. - Extracted the reading of the tokens out into a separate function. - Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation). This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible. I will also directly update some extra clang tooling that is broken by the change from Argument to Parameter. Hopefully the bots will stay appeased. Thanks! llvm-svn: 308190
* Revert changes from my previous refactoring - will need to fix dependencies ↵Faisal Vali2017-07-171-3/+3
| | | | | | | | in clang's extra tooling (such as clang-tidy etc.). Sorry about that. llvm-svn: 308158
* [NFC] Refactor the Preprocessor function that handles Macro definitions and ↵Faisal Vali2017-07-171-3/+3
| | | | | | | | | | | | | | rename Arguments to Parameters in Macro Definitions. - Extracted the reading of the tokens out into a separate function. - Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation). This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible. Thanks! llvm-svn: 308157
* LexerTest memory leak fix-Erich Keane2017-06-151-1/+3
| | | | | | | | A new LexerTest unittest introduced a memory leak. This patch uses a unique_ptr with a custom deleter to ensure it is properly deleted. llvm-svn: 305491
* Fix LexerTest signed/unsigned comparison.Erich Keane2017-06-151-1/+1
| | | | | | | | Werror was catching a signed/unsigned compare in an assert, correct the signed 'expected' value to be unsigned. llvm-svn: 305435
* [Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, Erich Keane2017-06-141-9/+62
| | | | | | | | | | | | | | | | | | | | | correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
* Add #pragma clang module build/endbuild pragmas for performing a module buildRichard Smith2017-06-093-59/+4
| | | | | | | | | | | | | | | as part of a compilation. This is intended for two purposes: 1) Writing self-contained test cases for modules: we can now write a single source file test that builds some number of module files on the side and imports them. 2) Debugging / test case reduction. A single-source testcase is much more amenable to reduction, compared to a VFS tarball or .pcm files. llvm-svn: 305101
* Reapply "Modules: Cache PCMs in memory and avoid a use-after-free"Duncan P. N. Exon Smith2017-03-203-4/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit r298185, effectively reapplying r298165, after fixing the new unit tests (PR32338). The memory buffer generator doesn't null-terminate the MemoryBuffer it creates; this version of the commit informs getMemBuffer about that to avoid the assert. Original commit message follows: ---- Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298278
OpenPOWER on IntegriCloud