summaryrefslogtreecommitdiffstats
path: root/clang/lib/Lex/TokenLexer.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Remove trailing spaceFangrui Song2018-07-301-10/+10
| | | | | | sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h} llvm-svn: 338291
* Remove \brief commands from doxygen comments.Adrian Prantl2018-05-091-4/+4
| | | | | | | | | | | | | | | | | | | This is similar to the LLVM change https://reviews.llvm.org/D46290. We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\@brief'); do perl -pi -e 's/\@brief //g' $i & done for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46320 llvm-svn: 331834
* PR37189 Fix incorrect end source location and spelling for a split '>>' token.Richard Smith2018-04-301-2/+2
| | | | | | | | | | | | | | | | | | When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an extension) when we do the same for other tokens starting with a '>', we can't just use a location pointing to the first '>' as the location of the split token, because that would result in our miscomputing the length and spelling for the token. As a consequence, for example, a refactoring replacing 'A<X>' with something else would sometimes replace one character too many, and similarly diagnostics highlighting a template-id source range would highlight one character too many. Fix this by creating an expansion range covering the first character of the '>>' token, whose spelling is '>'. For this to work, we generalize the expansion range of a macro FileID to be either a token range (the common case) or a character range (used in this new case). llvm-svn: 331155
* Fix typos in clangAlexander Kornienko2018-04-061-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Found via codespell -q 3 -I ../clang-whitelist.txt Where whitelist consists of: archtype cas classs checkk compres definit frome iff inteval ith lod methode nd optin ot pres statics te thru Patch by luzpaz! (This is a subset of D44188 that applies cleanly with a few files that have dubious fixes reverted.) Differential revision: https://reviews.llvm.org/D44188 llvm-svn: 329399
* [Lex] Fix some Clang-tidy modernize and Include What You Use warnings; other ↵Eugene Zelenko2017-12-081-4/+14
| | | | | | minor fixes (NFC). llvm-svn: 320207
* [Lex] Remove unused variables. No functionality change.Benjamin Kramer2017-10-151-6/+0
| | | | llvm-svn: 315845
* [c++2a] Implement P0306 __VA_OPT__ (Comma omission and comma deletion)Faisal Vali2017-10-151-7/+166
| | | | | | | | | | | | | | | | | | | | | | | | | This patch implements an extension to the preprocessor: __VA_OPT__(contents) --> which expands into its contents if variadic arguments are supplied to the parent macro, or behaves as an empty token if none. - Currently this feature is only enabled for C++2a (this could be enabled, with some careful tweaks, for other dialects with the appropriate extension or compatibility warnings) - The patch was reviewed here: https://reviews.llvm.org/D35782 and asides from the above (and moving some of the definition and expansion recognition logic into the corresponding state machines), I believe I incorporated all of Richard's suggestions. A few technicalities (most of which were clarified through private correspondence between rsmith, hubert and thomas) are worth mentioning. Given: #define F(a,...) a #__VA_OPT__(a ## a) a ## __VA_OPT__(__VA_ARGS__) - The call F(,) Does not supply any tokens for the variadic arguments and hence VA_OPT behaves as a placeholder. - When expanding VA_OPT (for e.g. F(,1) token pasting occurs eagerly within its contents if the contents need to be stringified. - A hash or a hashhash prior to VA_OPT does not inhibit expansion of arguments if they are the first token within VA_OPT. - When a variadic argument is supplied, argument substitution occurs within the contents as does stringification - and these resulting tokens are inserted back into the macro expansions token stream just prior to the entire stream being rescanned and concatenated. See wg21.link/P0306 for further details on the feature. Acknowledgment: This patch would have been poorer if not for Richard Smith's usual thoughtful analysis and feedback. llvm-svn: 315840
* Fix warnings. [-Wdocumentation]NAKAMURA Takumi2017-10-121-2/+1
| | | | llvm-svn: 315573
* Add parens around the boolean condition of one of the added asserts in ↵Faisal Vali2017-10-031-2/+2
| | | | | | | | | | | | | | | r314747 ... ... in the hopes of teaching the bots the gift of silence ;) For quick reference: https://reviews.llvm.org/rL314747 llvm-svn: 314753
* Remove an assertion I added from the refactoring of pasteTokens ↵Faisal Vali2017-10-031-4/+0
| | | | | | | | | | | | (https://reviews.llvm.org/rL314747). - it made the bots v angry! I'm not exactly sure why the assertion doesn't hold - if anyone has any insight - would appreciate it. Thanks! llvm-svn: 314748
* [NFC] Refactor PasteTokens so that it can be passed the Token Stream and ↵Faisal Vali2017-10-031-38/+58
| | | | | | | | | | | | | | Index to start concatenating at. In passing: - change the name of the function to pasteTokens c/w coding standards - rename CurToken to CurTokenIdx (since it is not the token, but the index) - add doxygen comments to document some of pasteTokens' functionality - use parameter names different from the data member names. This will be useful for implementing __VA_OPT__ (https://reviews.llvm.org/D35782#inline-322587) llvm-svn: 314747
* [NFC] Remove superfluous parameter Faisal Vali2017-09-301-1/+1
| | | | | | - MacroArgs already knows the maximum number of arguments that can be supplied to the macro. No need to pass MacroInfo (information about the macro definition) to the call to getPreExpArgument (which by the way might benefit from being called getExpandedArgument() ?) for it to compute the number of arguments. llvm-svn: 314593
* [NFC] Update local variable names to upper-case as per LLVM Coding Standards.Faisal Vali2017-07-201-9/+9
| | | | llvm-svn: 308574
* [NFC] Refactor the Preprocessor function that handles Macro definitions and ↵Faisal Vali2017-07-171-6/+6
| | | | | | | | | | | | | | | | | rename Arguments to Parameters in Macro Definitions. - Extracted the reading of the tokens out into a separate function. - Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation). This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible. I will also directly update some extra clang tooling that is broken by the change from Argument to Parameter. Hopefully the bots will stay appeased. Thanks! llvm-svn: 308190
* Revert changes from my previous refactoring - will need to fix dependencies ↵Faisal Vali2017-07-171-6/+6
| | | | | | | | in clang's extra tooling (such as clang-tidy etc.). Sorry about that. llvm-svn: 308158
* [NFC] Refactor the Preprocessor function that handles Macro definitions and ↵Faisal Vali2017-07-171-6/+6
| | | | | | | | | | | | | | rename Arguments to Parameters in Macro Definitions. - Extracted the reading of the tokens out into a separate function. - Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation). This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible. Thanks! llvm-svn: 308157
* Fix whitespace before token-paste of an argument.James Y Knight2017-05-041-12/+18
| | | | | | | | | | | | | | | The whitespace should come from the argument name in the macro expansion, rather than from the token passed to the macro (same as it does when not pasting). Added a new test case for the change in behavior to stringize_space.c. FileCheck'ized macro_paste_commaext.c, tweaked the test case, and added a comment; no behavioral change to this test. Differential Revision: https://reviews.llvm.org/D30427 llvm-svn: 302195
* [PP] Replace some index based for loops with range based onesErik Verbruggen2016-10-261-6/+5
| | | | | | | | | While in the area, also change some unsigned variables to size_t, and introduce an LLVM_FALLTHROUGH instead of a comment stating that. Differential Revision: http://reviews.llvm.org/D25982 llvm-svn: 285193
* [Lex] Speed up updateConsecutiveMacroArgTokens (NFC)Vedant Kumar2016-07-071-3/+4
| | | | | | | SM.isWrittenInSameFile() calls getFileID(), which can be expensive. Move this check behind some cheaper filters. llvm-svn: 274800
* [Lexer] Don't merge macro args from different macro filesVedant Kumar2016-05-191-0/+3
| | | | | | | | | | | | | | | | | | The lexer sets the end location of macro arguments incorrectly *if*, while merging consecutive args to fit into a single SLocEntry, it finds args which come from different macro files. Fix the issue by using separate SLocEntries in this situation. This fixes a code coverage crasher (rdar://problem/26181005). Because the lexer reported end locations for certain macro args incorrectly, we would generate bogus coverage mappings with negative line offsets. Reviewed-by: akyrtzi Differential Revision: http://reviews.llvm.org/D20401 llvm-svn: 270160
* Fix remaining Clang-tidy readability-redundant-control-flow warnings; other ↵Eugene Zelenko2016-02-121-7/+1
| | | | | | | | minor fixes. Differential revision: http://reviews.llvm.org/D17218 llvm-svn: 260757
* [MSVC Compat] Accept elided commas in macro function argumentsEhsan Akhgari2016-01-221-6/+11
| | | | | | | | | | | | | | | Summary: This fixes PR25875. When the trailing comma in a macro argument list is elided, we need to treat it similarly to the case where a variadic macro misses one actual argument. Reviewers: rnk, rsmith Subscribers: cfe-commits Differential Revision: http://reviews.llvm.org/D15670 llvm-svn: 258530
* When copying whitespace flags from the token naming a macro argument onto theRichard Smith2016-01-151-0/+1
| | | | | | | | first token of the expansion, don't forget to copy the "is at the start of a line" token (which is always false, as newlines cannot appear within a macro body); otherwise, stringizing the result can insert spurious whitespace. llvm-svn: 257863
* Emit a -Wmicrosoft warning when pasting /##/ into a comment token in MS mode.Nico Weber2015-12-291-7/+10
| | | | llvm-svn: 256595
* [clang] Refactoring of conditions so they use isOneOf() instead of multiple ↵Daniel Marjamaki2015-06-181-1/+1
| | | | | | is(). llvm-svn: 240008
* [MSVC] Mimic MSVC whitespace collapse for incompatible token pastingWill Wilson2015-04-171-0/+7
| | | | | | | | | | In public MS headers for XAudio, clang would fail to generate a valid UUID due to the UUID components being combined with the '-' UUID separators. Clang would attempting to recover but would preserve the leading whitespace from the tokens after each failed paste leading to spaces creeping into the UUID and causing an error in the __declspace(uuid()) parsing. Reference: Microsoft DirectX SDK (June 2010)\Include\XAudio2.h(51) Resolves http://llvm.org/pr23071 llvm-svn: 235186
* Remove many superfluous SmallString::str() calls.Yaron Keren2015-03-181-1/+1
| | | | | | | | | | | | | | | Now that SmallString is a first-class citizen, most SmallString::str() calls are not required. This patch removes a whole bunch of them, yet there are lots more. There are two use cases where str() is really needed: 1) To use one of StringRef member functions which is not available in SmallString. 2) To convert to std::string, as StringRef implicitly converts while SmallString do not. We may wish to change this, but it may introduce ambiguity. llvm-svn: 232622
* MSVC: A wide string literal from L#macro_arg in a macroAlexey Bataev2014-12-151-3/+19
| | | | | | | | Clang should form a wide string literal from L#macro_arg in a function-like macro in -fms-compatibility mode. Fix for http://llvm.org/PR9984. Differential Revision: http://reviews.llvm.org/D6604 llvm-svn: 224228
* Lex: Fix an invalid access into a SmallStringDavid Majnemer2014-10-251-3/+4
| | | | | | | | | | | | | We would crash because we used operator[] to access past the end of a SmallString. This occured because our token had length zero. Instead, form the pointer using .data() and arithmetic. This is safe because this forms a one-past-the-end pointer and it is only used to compare with another one-past-the-end pointer. This fixes PR21379. llvm-svn: 220614
* [C++11] Use 'nullptr'. Lex edition.Craig Topper2014-05-171-5/+5
| | | | llvm-svn: 209083
* Wrap to 80 columns, no code change.Nico Weber2014-05-091-4/+3
| | | | llvm-svn: 208386
* If the first token in a macro that appears at the start of a line expands toRichard Smith2014-02-241-0/+1
| | | | | | | nothing, be sure to inform the *next* token expanded from the macro that it is now at the start of a line. Patch by Harald van Dijk! llvm-svn: 202068
* Fix a non-error diagnostic that had an err_ name. Also move it from Warning toRichard Smith2014-02-181-5/+4
| | | | | | ExtWarn, since it's an extension. llvm-svn: 201540
* Clean up whitespace checksJustin Bogner2014-02-041-11/+5
| | | | | | | | | | | | | In TokenLexer::ExpandFunctionArguments(), CurTok.hasLeadingSpace() is checked in multiple locations, each time subtly differently. Checking it early, when the token is seen, and using NextTokGetsSpace exclusively after that makes the code simpler. No change in behaviour is intended. Patch by Harald van Dijk! llvm-svn: 200788
* Fix whitespace handling in empty macro expansionsJustin Bogner2014-02-041-2/+6
| | | | | | | | | | | | | When a macro expansion does not result in any tokens, and the macro name is preceded by whitespace, the whitespace should be passed to the first token that follows the macro expansion. Similarly when a macro expansion ends with a placemarker token, and that placemarker token is preceded by whitespace. This worked already for top-level macro expansions, but is now extended to also work for nested macro expansions. Patch by Harald van Dijk! llvm-svn: 200787
* Fix whitespace handling in empty macro argumentsJustin Bogner2014-02-041-18/+13
| | | | | | | | | | | | | When a function-like macro definition ends with one of the macro's parameters, and the argument is empty, any whitespace before the parameter name in the macro definition needs to be preserved. Promoting the existing NextTokGetsSpace to a preserved bit-field and checking it at the end of the macro expansion allows it to be moved to the first token following the macro expansion result. Patch by Harald van Dijk! llvm-svn: 200786
* Fix whitespace handling in ## operatorJustin Bogner2014-02-041-13/+16
| | | | | | | | | | | | | | | | | In x ## y, where x and y are regular tokens, whitespace between x and ## is ignored, and whitespace between ## and y is also ignored. When either x or y is a function argument, whitespace was preserved, but it should not be. This patch removes the checks for whitespace before ## and before y, and in the special case where x is an empty macro argument and y is a regular token, actively removes whitespace before y. One existing test is affected by that change, but as clang's output now matches the standard's requirements and that of GCC, I've tweaked the testcase. Patch by Harald van Dijk! llvm-svn: 200785
* Rename language option MicrosoftMode to MSVCCompatAlp Toker2014-01-141-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | There's been long-standing confusion over the role of these two options. This commit makes the necessary changes to differentiate them clearly, following up from r198936. MicrosoftExt (aka. fms-extensions): Enable largely unobjectionable Microsoft language extensions to ease portability. This mode, also supported by gcc, is used for building software like FreeBSD and Linux kernel extensions that share code with Windows drivers. MSVCCompat (aka. -fms-compatibility, formerly MicrosoftMode): Turn on a special mode supporting 'heinous' extensions for drop-in compatibility with the Microsoft Visual C++ product. Standards-compilant C and C++ code isn't guaranteed to work in this mode. Implies MicrosoftExt. Note that full -fms-compatibility mode is currently enabled by default on the Windows target, which may need tuning to serve as a reasonable default. See cfe-commits for the full discourse, thread 'r198497 - Move MS predefined type_info out of InitializePredefinedMacros' No change in behaviour. llvm-svn: 199209
* Sort all the #include lines with LLVM's utils/sort_includes.py whichChandler Carruth2014-01-071-1/+1
| | | | | | | encodes the canonical rules for LLVM's style. I noticed this had drifted quite a bit when cleaning up LLVM, so wanted to clean up Clang as well. llvm-svn: 198686
* Make Preprocessor::Lex non-recursive.Eli Friedman2013-09-191-12/+17
| | | | | | | | | | | | | | | Before this patch, Lex() would recurse whenever the current lexer changed (e.g. upon entry into a macro). This patch turns the recursion into a loop: the various lex routines now don't return a token when the current lexer changes, and at the top level Preprocessor::Lex() now loops until it finds a token. Normally, the recursion wouldn't end up being very deep, but the recursion depth can explode in edge cases like a bunch of consecutive macros which expand to nothing (like in the testcase test/Preprocessor/macro_expand_empty.c in this patch). <rdar://problem/14569770> llvm-svn: 190980
* Use pop_back_val() instead of both back() and pop_back().Robert Wilhelm2013-08-231-4/+2
| | | | | | No functionality change intended. llvm-svn: 189112
* Use SmallVectorImpl& for function arguments instead of SmallVector.Craig Topper2013-07-051-1/+1
| | | | llvm-svn: 185715
* Match MSVC's handling of commas during macro argument expansionReid Kleckner2013-06-261-0/+8
| | | | | | | | | | | | This allows clang to parse the type_traits header in Visual Studio 2012, which is included widely in practice. This is a rework of r163022 by João Matos. The original patch broke preprocessing of gtest headers, which this patch addresses. Patch by Will Wilson! llvm-svn: 184968
* [Preprocessor] Prevent expansion of y in x ## y when x is emptyArgyrios Kyrtzidis2013-05-251-7/+13
| | | | | | | | | | | | When x is empty, x ## is suppressed, and when y gets expanded, the fact that it follows ## is not available in the macro expansion result. The macro definition can be checked instead, the ## will be available there regardless of what x expands to. Fixes http://llvm.org/PR12767 Patch by Harald van Dijk! llvm-svn: 182699
* [Preprocessor] For the MacroExpands preprocessor callback, also pass the ↵Argyrios Kyrtzidis2013-05-031-1/+1
| | | | | | | | MacroArgs object that provides information about the argument tokens for a function macro. llvm-svn: 181065
* [preprocessor] When "merging" macro argument tokens into one SLocEntry chunk,Argyrios Kyrtzidis2012-12-191-2/+6
| | | | | | | | make sure they came from the same kind of FileIDs. Thanks to Abramo Bagnara for providing the test case. llvm-svn: 170616
* Sort all of Clang's files under 'lib', and fix up the broken headersChandler Carruth2012-12-041-2/+2
| | | | | | | | | | | | | uncovered. This required manually correcting all of the incorrect main-module headers I could find, and running the new llvm/utils/sort_includes.py script over the files. I also manually added quite a few missing headers that were uncovered by shuffling the order or moving headers up to be main-module-headers. llvm-svn: 169237
* Try to make the source location information for token pastes a bit more ↵Eli Friedman2012-12-011-0/+6
| | | | | | | | | consistent. Fixes a crash printing diagnostics on the gcc testsuite, and also makes diagnostic range printing print nicer results for token pastes. llvm-svn: 169068
* Improved support for removing the comma preceding __VA_ARGS__ where __VA_ARGS__Andy Gibbs2012-11-091-17/+64
| | | | | | | | | | is empty in a variadic macro expansion. This fixes a divergence in support for the ", ## __VA_ARGS__" GCC extension which differed in behaviour when in strict C99 mode (note: there is no change in behaviour has been made in the gnu99 mode that clang uses by default). In addition, there is improved support for the Microsoft alternative extension ", __VA_ARGS__". llvm-svn: 167613
* Revert r163022, it caused PR13924.Nico Weber2012-09-261-6/+0
| | | | | | | Add a test for PR13924. Do not revert the test added in r163022, it surprisingly still passes even after reverting the code changes. llvm-svn: 164672
OpenPOWER on IntegriCloud