summaryrefslogtreecommitdiffstats
path: root/clang/lib/Lex
Commit message (Collapse)AuthorAgeFilesLines
* [Stats] Convert some ad-hoc header search stats to ALWAYS_ENABLED_STATISTIC.Volodymyr Sapsai2019-10-111-11/+22
| | | | | | | | | | | | | | | | rdar://problem/55715134 Reviewers: dsanders, bogner, rtereshin Reviewed By: dsanders Subscribers: hiraditya, jkorous, dexonsmith, ributzka, cfe-commits, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D68252 llvm-svn: 374581
* [clang-scan-deps] Improve string/character literal skippingAlex Lorenz2019-10-081-11/+29
| | | | | | | | | | | | | | | | The existing string/character literal skipping code in the dependency directives source minimizer has two issues: - It doesn't stop the scanning when a newline is reached before the terminating character, unlike the lexer which considers the token to be done (even if it's invalid) at the end of the line. - It doesn't support whitespace between '\' and the newline when looking if the '\' is used as a line continuation character. This commit fixes both issues. Differential Revision: https://reviews.llvm.org/D68436 llvm-svn: 374127
* Fix for expanding __pragmas in macro argumentsAmy Huang2019-10-071-32/+47
| | | | | | | | | | | | | | | | | Summary: Avoid parsing __pragma into an annotation token when macro arguments are pre-expanded. This is what clang currently does when parsing _Pragmas. Fixes https://bugs.llvm.org/show_bug.cgi?id=41128, where clang crashed when trying to get the length of an annotation token. Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D68114 llvm-svn: 373950
* [clang-scan-deps] Allow continuation line backslashes followed by whitespaceAlex Lorenz2019-09-261-11/+22
| | | | | | | | | | | | in the dependency source minimizer Clang allows continuations that have whitespace between the backslash and the newline. This patch ensures that the dependency source minimizer can handle the whitespace between the backslash and the newline when looking for a line continuation. Differential Revision: https://reviews.llvm.org/D68052 llvm-svn: 373007
* Move normalization of `\` in #includes from -fms-compatibility to ↵Reid Kleckner2019-09-261-15/+21
| | | | | | | | | -fms-extensions Handling backslashes in include paths in the implementation isn't non-conforming. llvm-svn: 372999
* Simplify -fms-compatibility include lookup logic, NFCReid Kleckner2019-09-251-10/+11
| | | | | | | | | | | This include search logic has an extra parameter to deal with Windows includes with backslashes, which get normalized to forward slashes on non-Windows under -fms-compatibility. Hoist the conditional operator out of LookupHeaderIncludeOrImport and pass the result in instead of repeating the ?: expression everywhere. llvm-svn: 372926
* Fix for stringized function-macro args continued across linesAlex Lorenz2019-09-191-2/+5
| | | | | | | | | | | In case of certain #define'd macros, there's a space just before line continuation that the minimized-source lexer was missing to include, resulting in invalid stringize. Patch by: kousikk (Kousik Kumar) Differential Revision: https://reviews.llvm.org/D67635 llvm-svn: 372360
* do not emit -Wunused-macros warnings in -frewrite-includes mode (PR15614)Lubos Lunak2019-09-161-1/+2
| | | | | | | | | | | -frewrite-includes calls PP.SetMacroExpansionOnlyInDirectives() to avoid macro expansions that are useless in that mode, but this can lead to -Wunused-macros false positives. As -frewrite-includes does not emit normal warnings, block -Wunused-macros too. Differential Revision: https://reviews.llvm.org/D65371 llvm-svn: 372026
* [NFCI]Create CommonAttributeInfo Type as base type of *Attr and ParsedAttr.Erich Keane2019-09-133-9/+10
| | | | | | | | | | | | In order to enable future improvements to our attribute diagnostics, this moves info from ParsedAttr into CommonAttributeInfo, then makes this type the base of the *Attr and ParsedAttr types. Quite a bit of refactoring took place, including removing a bunch of redundant Spelling Index propogation. Differential Revision: https://reviews.llvm.org/D67368 llvm-svn: 371875
* [clang-scan-deps] add skip excluded conditional preprocessor block ↵Alex Lorenz2019-09-114-0/+98
| | | | | | | | | | | | | | | preprocessing optimization This commit adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preprocessor blocks by bumping the lexer pointer, and not lexing the tokens until reaching appropriate #else/#endif directive. The skip positions and lexer offsets are computed when the file is minimized, directly from the minimized tokens. On an 18-core iMacPro with macOS Catalina Beta I got 10-15% speed-up from this optimization when running clang-scan-deps on the compilation database for a recent LLVM and Clang (3511 files). Differential Revision: https://reviews.llvm.org/D67127 llvm-svn: 371656
* Fix -Wnonportable-include-path suppression for header maps with absolute paths.Volodymyr Sapsai2019-09-111-10/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | In `DirectoryLookup::LookupFile` parameter `HasBeenMapped` doesn't cover the case when clang finds a file through a header map but doesn't remap the lookup filename because the target path is an absolute path. As a result, -Wnonportable-include-path suppression for header maps introduced in r301592 wasn't triggered. Change parameter `HasBeenMapped` to `IsInHeaderMap` and use parameter `MappedName` to track the filename remapping. This way we can handle both relative and absolute paths in header maps, and account for their specific properties, like filename remapping being a property preserved across lookups in multiple directories. rdar://problem/39516483 Reviewers: dexonsmith, bruno Reviewed By: dexonsmith Subscribers: jkorous, cfe-commits, ributzka Differential Revision: https://reviews.llvm.org/D58094 llvm-svn: 371655
* Introduce a DirectoryEntryRef that stores both a reference and anAlex Lorenz2019-08-312-2/+3
| | | | | | | | | | | | | | accessed name to the directory entry This commit introduces a parallel API that returns a DirectoryEntryRef to the FileManager, similar to the parallel FileEntryRef API. All uses will have to be update in follow-up patches. The immediate use of the new API in this patch fixes the issue where a file manager was reused in clang-scan-deps, but reported an different file path whenever a framework lookup was done through a symlink. Differential Revision: https://reviews.llvm.org/D67026 llvm-svn: 370562
* Re-land [clang-scan-deps] Minimizer: Correctly handle multi-line content ↵Alexandre Ganea2019-08-281-14/+35
| | | | | | | | | | with CR+LF line endings Previously, an #error directive with quoted, multi-line content, along with CR+LF line endings wasn't handled correctly. Differential Revision: https://reviews.llvm.org/D66556 llvm-svn: 370129
* [preprocessor] Add an opportunity to retain excluded conditional blocksEvgeny Mankov2019-08-271-4/+16
| | | | | | | | | | | | | | It is handy for clang tooling, for instance, in source to source transformation. Reviewers: vpykhtin (Valery Pykhtin), erichkeane (Erich Keane) Subscribers: rsmith (Richard Smith), akyrtzi (Argyrios Kyrtzidis) Tags: #clang Differential Revision: https://reviews.llvm.org/D66597 llvm-svn: 370123
* Use FileEntryRef for PPCallbacks::HasIncludeAlex Lorenz2019-08-271-2/+1
| | | | | | | | This fixes the issue where a filename dependendency was missing if the file that was referenced with __has_include() was accessed through a symlink in an earlier run, if the file manager was reused between runs. llvm-svn: 370081
* Revert "[clang-scan-deps] Minimizer: Correctly handle multi-line content ↵Richard Smith2019-08-271-35/+14
| | | | | | | | | | | | with CR+LF line endings" This reverts commit r369986. This change added a dependency on the 'dos2unix' tool, which is not one of our accepted test dependencies and may not exist on all machines that build Clang. llvm-svn: 370000
* Use FileEntryRef for PPCallbacks::FileSkippedAlex Lorenz2019-08-271-1/+1
| | | | | | | | This fixes the issue where a filename dependendency was missing if the file that was skipped was included through a symlink in an earlier run, if the file manager was reused between runs. llvm-svn: 369998
* [clang-scan-deps] Skip UTF-8 BOM in source minimizerAlexandre Ganea2019-08-271-0/+7
| | | | | | Differential Revision: https://reviews.llvm.org/D66511 llvm-svn: 369993
* [clang-scan-deps] Minimizer: Correctly skip over double slashes in angle ↵Alexandre Ganea2019-08-261-3/+4
| | | | | | | | | | bracket #include Previously, double slashes (//) occurring in angle brackets #include were incorrectly interpreted as comments. eg. #include <dir//file.h> Differential Revision: https://reviews.llvm.org/D66550 llvm-svn: 369988
* [clang-scan-deps] Minimizer: Correctly handle multi-line content with CR+LF ↵Alexandre Ganea2019-08-261-14/+35
| | | | | | | | | | line endings Previously, an #error directive with quoted, multi-line content, along with CR+LF line endings wasn't handled correctly. Differential Revision: https://reviews.llvm.org/D66556 llvm-svn: 369986
* FileManager: Use llvm::Expected in new getFileRef APIDuncan P. N. Exon Smith2019-08-262-13/+9
| | | | | | | | | | | | | | | | | | | | | `FileManager::getFileRef` is a modern API which we expect to convert to over time. We should modernize the error handling as well, using `llvm::Expected` instead of `llvm::ErrorOr`, to help clients that care about errors to ensure nothing is missed. However, not all clients care. I've also added another path for those that don't: - `FileEntryRef` is now copy- and move-assignable (using a pointer instead of a reference). - `FileManager::getOptionalFileRef` returns an `llvm::Optional` instead of `llvm::Expected`. - Added an `llvm::expectedToOptional` utility in case this is useful elsewhere. https://reviews.llvm.org/D66705 llvm-svn: 369943
* Introduce FileEntryRef and use it when handling includes to report correct ↵Alex Lorenz2019-08-226-252/+280
| | | | | | | | | | | | | | | | | | | | | | | dependencies when the FileManager is reused across invocations This commit introduces a parallel API to FileManager's getFile: getFileEntryRef, which returns a reference to the FileEntry, and the name that was used to access the file. In the case of a VFS with 'use-external-names', the FileEntyRef contains the external name of the file, not the filename that was used to access it. The new API is adopted only in the HeaderSearch and Preprocessor for include file lookup, so that the accessed path can be propagated to SourceManager's FileInfo. SourceManager's FileInfo now can report this accessed path, using the new getName method. This API is then adopted in the dependency collector, which now correctly reports dependencies when a file is included both using a symlink and a real path in the case when the FileManager is reused across multiple Preprocessor invocations. Note that this patch does not fix all dependency collector issues, as the same problem is still present in other cases when dependencies are obtained using FileSkipped, InclusionDirective, and HasInclude. This will be fixed in follow-up commits. Differential Revision: https://reviews.llvm.org/D65907 llvm-svn: 369680
* clang: Fix typo in commentNico Weber2019-08-211-1/+1
| | | | llvm-svn: 369539
* [Clang] Migrate llvm::make_unique to std::make_uniqueJonas Devlieghere2019-08-145-8/+8
| | | | | | | | | | Now that we've moved to C++14, we no longer need the llvm::make_unique implementation from STLExtras.h. This patch is a mechanical replacement of (hopefully) all the llvm::make_unique instances across the monorepo. Differential revision: https://reviews.llvm.org/D66259 llvm-svn: 368942
* Add __has_builtin support for builtin function-like type traits.Richard Smith2019-08-141-13/+30
| | | | | | | | | | | | | | | | | | | | | | | | Summary: Previously __has_builtin(__builtin_*) would return false for __builtin_*s that we modeled as keywords rather than as functions (because they take type arguments). With this patch, all builtins that are called with function-call-like syntax return true from __has_builtin (covering __builtin_* and also the __is_* and __has_* type traits and the handful of similar builtins without such a prefix). Update the documentation on __has_builtin and on type traits to match. While doing this I noticed the type trait documentation was out of date and incomplete; that's fixed here too. Reviewers: aaron.ballman Subscribers: jfb, kristina, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D66100 llvm-svn: 368785
* Don't use std::errcAlexey Bataev2019-08-131-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: As noted on Errc.h: // * std::errc is just marked with is_error_condition_enum. This means that // common patters like AnErrorCode == errc::no_such_file_or_directory take // 4 virtual calls instead of two comparisons. And on some libstdc++ those virtual functions conclude that ------------------------ int main() { std::error_code foo = std::make_error_code(std::errc::no_such_file_or_directory); return foo == std::errc::no_such_file_or_directory; } ------------------------- should exit with 0. Reviewers: thakis, rnk, jfb Reviewed By: thakis Subscribers: lebedev.ri, dexonsmith, xbolva00, cfe-commits, caomhin Tags: #clang Differential Revision: https://reviews.llvm.org/D66143 llvm-svn: 368739
* Don't diagnose errors when a file matches an include componentReid Kleckner2019-08-091-1/+2
| | | | | | | | | This regressed in r368322, and was reported as PR42948 and on the mailing list. The fix is to ignore the specific error code for this case. The problem doesn't seem to reproduce on Windows, where a different error code is used instead. llvm-svn: 368475
* [clang-scan-deps] Add minimizer support for C++20 modules.Michael J. Spencer2019-08-091-1/+70
| | | | | | This only adds support to the minimizer, it doesn't actually capture the dependencies yet. llvm-svn: 368381
* Fix up fd limit diagnosis codeReid Kleckner2019-08-081-2/+3
| | | | | | | | | | | Apparently Windows returns the "invalid argument" error code when the path contains invalid characters such as '<'. The test/Preprocessor/include-likely-typo.c test does this, so it was failing after r368322. Also, the diagnostic requires two arguments, so add the filename. llvm-svn: 368348
* clang: Diag running out of file handles while looking for filesNico Weber2019-08-081-2/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | clang would only print "file not found" when it's unable to find a header file. If the reason for that is a file handle leak, that's not a very useful error message. For errors that aren't in a small whitelist ("file not found", "file is directory"), print an error with the strerror() output. This changes behavior in corner cases: If clang was out of file handles while looking in one -I dir but then suddenly wasn't when looking in the next -I dir, and both directories contained a file with the desired name, previously we'd silently return the file from the second directory. For this reason, it's important to ignore "is a directory" for this new diag: if a file foo/foo exists and -I -Ifoo are passed, an include of "foo" should successfully open file "foo" in directory "foo/" instead of complaining that "./foo" is a directory. No test since we mostly hit this when there's a handle leak somewhere, and currently there isn't one. I manually tested this with the repro steps in comment 2 on the bug below. Fixes PR42524. Differential Revision: https://reviews.llvm.org/D65956 llvm-svn: 368322
* [clang] Adopt new FileManager error-returning APIsHarlan Haskins2019-08-015-82/+119
| | | | | | | Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods. Signed-off-by: Harlan Haskins <harlan@apple.com> llvm-svn: 367616
* [Preprocessor] Always discard body of #define if we failed to parse itIlya Biryukov2019-08-011-5/+9
| | | | | | | | | | | | | | | | | | | | Summary: Preivously we would only discard it if we failed to parse parameter lists. If we do not consume the body, parser sees tokens inside directive. In turn, this leads to spurious diagnostics and a crash in TokenBuffer, see the added tests. Reviewers: sammccall Reviewed By: sammccall Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D65517 llvm-svn: 367530
* Remove cache for macro arg stringizationReid Kleckner2019-07-302-32/+4
| | | | | | | | | | | | | | | | | | | Summary: The cache recorded the wrong expansion location for all but the first stringization. It seems uncommon to stringize the same macro argument multiple times, so this cache doesn't seem that important. Fixes PR39942 Reviewers: vsk, rsmith Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D65428 llvm-svn: 367337
* [clang-scan-deps] Dependency directives source minimizer: handle #pragma onceAlex Lorenz2019-07-181-1/+15
| | | | | | | | | We should re-emit `#pragma once` to ensure the preprocessor will still honor it when running on minimized sources. Differential Revision: https://reviews.llvm.org/D64945 llvm-svn: 366509
* Fix parameter name comments using clang-tidy. NFC.Rui Ueyama2019-07-165-14/+14
| | | | | | | | | | | | | | | | | | | | | This patch applies clang-tidy's bugprone-argument-comment tool to LLVM, clang and lld source trees. Here is how I created this patch: $ git clone https://github.com/llvm/llvm-project.git $ cd llvm-project $ mkdir build $ cd build $ cmake -GNinja -DCMAKE_BUILD_TYPE=Debug \ -DLLVM_ENABLE_PROJECTS='clang;lld;clang-tools-extra' \ -DCMAKE_EXPORT_COMPILE_COMMANDS=On -DLLVM_ENABLE_LLD=On \ -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ../llvm $ ninja $ parallel clang-tidy -checks='-*,bugprone-argument-comment' \ -config='{CheckOptions: [{key: StrictMode, value: 1}]}' -fix \ ::: ../llvm/lib/**/*.{cpp,h} ../clang/lib/**/*.{cpp,h} ../lld/**/*.{cpp,h} llvm-svn: 366177
* Delete dead storesFangrui Song2019-07-121-2/+1
| | | | llvm-svn: 365901
* [clang-scan-deps] Dependency directives source minimizer:Alex Lorenz2019-07-101-1/+8
| | | | | | | | | | | | single quotes are not digit separators after a valid character literal prefix The single quote character can act as a c++ digit separator. However, the minimizer shouldn't treat it as such when it's actually following a valid character literal prefix, like L, U, u, or u8. Differential Revision: https://reviews.llvm.org/D64525 llvm-svn: 365700
* [modules] Add PP callbacks for entering and leaving a submodule.Vassil Vassilev2019-07-041-0/+12
| | | | llvm-svn: 365153
* Change std::{lower,upper}_bound to llvm::{lower,upper}_bound or ↵Fangrui Song2019-07-032-17/+11
| | | | | | llvm::partition_point. NFC llvm-svn: 365006
* [clang][HeaderSearch] Shorten paths for includes in mainfile's directoryKadir Cetinkaya2019-07-031-16/+28
| | | | | | | | | | | | | | | | | | Summary: Currently HeaderSearch only looks at SearchDir's passed into it, but in addition to those paths headers can be relative to including file's directory. This patch makes sure that is taken into account. Reviewers: gribozavr Subscribers: jkorous, arphaman, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D63295 llvm-svn: 365005
* [C++2a] Add __builtin_bit_cast, used to implement std::bit_castErik Pilkington2019-07-021-0/+1
| | | | | | | | | | | | | | | | | | This commit adds a new builtin, __builtin_bit_cast(T, v), which performs a bit_cast from a value v to a type T. This expression can be evaluated at compile time under specific circumstances. The compile time evaluation currently doesn't support bit-fields, but I'm planning on fixing this in a follow up (some of the logic for figuring this out is in CodeGen). I'm also planning follow-ups for supporting some more esoteric types that the constexpr evaluator supports, as well as extending __builtin_memcpy constexpr evaluation to use the same infrastructure. rdar://44987528 Differential revision: https://reviews.llvm.org/D62825 llvm-svn: 364954
* Fix __has_cpp_attribute expansion to produce trailing L and (whereRichard Smith2019-06-211-2/+10
| | | | | | | | necessary) leading whitespace. Simplify unit test and extend to cover no_unique_address attribute. llvm-svn: 364102
* [clang][HeaderSearch] Consider all path separators equalKadir Cetinkaya2019-06-061-0/+5
| | | | | | | | | | | | Reviewers: ilya-biryukov, sammccall Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D62965 llvm-svn: 362731
* Fix windows build for commit r362459Alex Lorenz2019-06-031-6/+6
| | | | llvm-svn: 362463
* Add clang source minimizer that reduces source to directivesAlex Lorenz2019-06-032-0/+757
| | | | | | | | | | | | | | | | | | | | | that might affect the dependency list for a compilation This commit introduces a dependency directives source minimizer to clang that minimizes header and source files to the minimum necessary preprocessor directives for evaluating includes. It reduces the source down to #define, #include, The source minimizer works by lexing the input with a custom fast lexer that recognizes the preprocessor directives it cares about, and emitting those directives in the minimized source. It ignores source code, comments, and normalizes whitespace. It gives up and fails if seems any directives that it doesn't recognize as valid (e.g. #define 0). In addition to the source minimizer this patch adds a -print-dependency-directives-minimized-source CC1 option that allows you to invoke the minimizer from clang directly. Differential Revision: https://reviews.llvm.org/D55463 llvm-svn: 362459
* Make __has_builtin work with __builtin_LINE and friends.Eric Fiselier2019-05-291-0/+4
| | | | | | | | | | The source location builtins are implemented as keywords, but __has_builtin should still report true for them. This patch also fixes a test failure on systemz where the alignment of string literals is 2 not 1. llvm-svn: 361920
* [Preprocessor] Fix crash emitting note with framework location for "file not ↵Volodymyr Sapsai2019-05-271-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | found" error. A filename can be remapped with a header map to point to a framework header and we can find the corresponding framework without the header. But if the original filename doesn't have a remapped framework name, we'll fail to find its location and will dereference a null pointer during diagnostics emission. Fix by tracking remappings better and emit the note only if a framework is found before any of the remappings. rdar://problem/48883447 Reviewers: arphaman, erik.pilkington, jkorous Reviewed By: arphaman Subscribers: dexonsmith, cfe-commits Differential Revision: https://reviews.llvm.org/D61707 llvm-svn: 361779
* [PragmaHandler] Expose `#pragma` locationJoel E. Denny2019-05-212-31/+29
| | | | | | | | | | | | | | | | | | | | | | | Currently, a pragma AST node's recorded location starts at the namespace token (such as `omp` in the case of OpenMP) after the `#pragma` token, and the `#pragma` location isn't available. However, the `#pragma` location can be useful when, for example, rewriting a directive using Clang's Rewrite facility. This patch makes `#pragma` locations available in any `PragmaHandler` but it doesn't yet make use of them. This patch also uses the new `struct PragmaIntroducer` to simplify `Preprocessor::HandlePragmaDirective`. It doesn't do the same for `PPCallbacks::PragmaDirective` because that changes the API documented in `clang-tools-extra/docs/pp-trace.rst`, and I'm not sure about backward compatibility guarantees there. Reviewed By: ABataev, lebedev.ri, aaron.ballman Differential Revision: https://reviews.llvm.org/D61643 llvm-svn: 361335
* [Lex] Allow to consume tokens while preprocessingIlya Biryukov2019-05-178-23/+36
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: By adding a hook to consume all tokens produced by the preprocessor. The intention of this change is to make it possible to consume the expanded tokens without re-runnig the preprocessor with minimal changes to the preprocessor and minimal performance penalty when preprocessing without recording the tokens. The added hook is very low-level and reconstructing the expanded token stream requires more work in the client code, the actual algorithm to collect the tokens using this hook can be found in the follow-up change. Reviewers: rsmith Reviewed By: rsmith Subscribers: eraman, nemanjai, kbarton, jsji, riccibruno, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D59885 llvm-svn: 361007
* Reland "[Clang][PP] Add the __FILE_NAME__ builtin macro"Kristina Brooks2019-05-161-2/+17
| | | | | | | | | | | | | | | | | | | | | | | | | This relands commit rL360833 which caused issues on Win32 bots due to path handling/normalization differences. Now this uses `sys::path::filename` which should handle additional edge cases on Win32. Original commit: "[Clang][PP] Add the __FILE_NAME__ builtin macro" This patch adds the __FILE_NAME__ macro that expands to the last component of the path, similar to __FILE__ except with a guarantee that only the last path component (without the separator) will be rendered. I intend to follow through with discussion of this with WG14 as a potential inclusion in the C standard or failing that, try to discuss this with GCC developers since this extension is desired by GCC and Clang users/developers alike. Differential Revision: https://reviews.llvm.org/D61756 llvm-svn: 360938
OpenPOWER on IntegriCloud