summaryrefslogtreecommitdiffstats
path: root/clang/lib/Lex/TokenLexer.cpp
Commit message (Collapse)AuthorAgeFilesLines
* [clang] Refactoring of conditions so they use isOneOf() instead of multiple ↵Daniel Marjamaki2015-06-181-1/+1
| | | | | | is(). llvm-svn: 240008
* [MSVC] Mimic MSVC whitespace collapse for incompatible token pastingWill Wilson2015-04-171-0/+7
| | | | | | | | | | In public MS headers for XAudio, clang would fail to generate a valid UUID due to the UUID components being combined with the '-' UUID separators. Clang would attempting to recover but would preserve the leading whitespace from the tokens after each failed paste leading to spaces creeping into the UUID and causing an error in the __declspace(uuid()) parsing. Reference: Microsoft DirectX SDK (June 2010)\Include\XAudio2.h(51) Resolves http://llvm.org/pr23071 llvm-svn: 235186
* Remove many superfluous SmallString::str() calls.Yaron Keren2015-03-181-1/+1
| | | | | | | | | | | | | | | Now that SmallString is a first-class citizen, most SmallString::str() calls are not required. This patch removes a whole bunch of them, yet there are lots more. There are two use cases where str() is really needed: 1) To use one of StringRef member functions which is not available in SmallString. 2) To convert to std::string, as StringRef implicitly converts while SmallString do not. We may wish to change this, but it may introduce ambiguity. llvm-svn: 232622
* MSVC: A wide string literal from L#macro_arg in a macroAlexey Bataev2014-12-151-3/+19
| | | | | | | | Clang should form a wide string literal from L#macro_arg in a function-like macro in -fms-compatibility mode. Fix for http://llvm.org/PR9984. Differential Revision: http://reviews.llvm.org/D6604 llvm-svn: 224228
* Lex: Fix an invalid access into a SmallStringDavid Majnemer2014-10-251-3/+4
| | | | | | | | | | | | | We would crash because we used operator[] to access past the end of a SmallString. This occured because our token had length zero. Instead, form the pointer using .data() and arithmetic. This is safe because this forms a one-past-the-end pointer and it is only used to compare with another one-past-the-end pointer. This fixes PR21379. llvm-svn: 220614
* [C++11] Use 'nullptr'. Lex edition.Craig Topper2014-05-171-5/+5
| | | | llvm-svn: 209083
* Wrap to 80 columns, no code change.Nico Weber2014-05-091-4/+3
| | | | llvm-svn: 208386
* If the first token in a macro that appears at the start of a line expands toRichard Smith2014-02-241-0/+1
| | | | | | | nothing, be sure to inform the *next* token expanded from the macro that it is now at the start of a line. Patch by Harald van Dijk! llvm-svn: 202068
* Fix a non-error diagnostic that had an err_ name. Also move it from Warning toRichard Smith2014-02-181-5/+4
| | | | | | ExtWarn, since it's an extension. llvm-svn: 201540
* Clean up whitespace checksJustin Bogner2014-02-041-11/+5
| | | | | | | | | | | | | In TokenLexer::ExpandFunctionArguments(), CurTok.hasLeadingSpace() is checked in multiple locations, each time subtly differently. Checking it early, when the token is seen, and using NextTokGetsSpace exclusively after that makes the code simpler. No change in behaviour is intended. Patch by Harald van Dijk! llvm-svn: 200788
* Fix whitespace handling in empty macro expansionsJustin Bogner2014-02-041-2/+6
| | | | | | | | | | | | | When a macro expansion does not result in any tokens, and the macro name is preceded by whitespace, the whitespace should be passed to the first token that follows the macro expansion. Similarly when a macro expansion ends with a placemarker token, and that placemarker token is preceded by whitespace. This worked already for top-level macro expansions, but is now extended to also work for nested macro expansions. Patch by Harald van Dijk! llvm-svn: 200787
* Fix whitespace handling in empty macro argumentsJustin Bogner2014-02-041-18/+13
| | | | | | | | | | | | | When a function-like macro definition ends with one of the macro's parameters, and the argument is empty, any whitespace before the parameter name in the macro definition needs to be preserved. Promoting the existing NextTokGetsSpace to a preserved bit-field and checking it at the end of the macro expansion allows it to be moved to the first token following the macro expansion result. Patch by Harald van Dijk! llvm-svn: 200786
* Fix whitespace handling in ## operatorJustin Bogner2014-02-041-13/+16
| | | | | | | | | | | | | | | | | In x ## y, where x and y are regular tokens, whitespace between x and ## is ignored, and whitespace between ## and y is also ignored. When either x or y is a function argument, whitespace was preserved, but it should not be. This patch removes the checks for whitespace before ## and before y, and in the special case where x is an empty macro argument and y is a regular token, actively removes whitespace before y. One existing test is affected by that change, but as clang's output now matches the standard's requirements and that of GCC, I've tweaked the testcase. Patch by Harald van Dijk! llvm-svn: 200785
* Rename language option MicrosoftMode to MSVCCompatAlp Toker2014-01-141-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | There's been long-standing confusion over the role of these two options. This commit makes the necessary changes to differentiate them clearly, following up from r198936. MicrosoftExt (aka. fms-extensions): Enable largely unobjectionable Microsoft language extensions to ease portability. This mode, also supported by gcc, is used for building software like FreeBSD and Linux kernel extensions that share code with Windows drivers. MSVCCompat (aka. -fms-compatibility, formerly MicrosoftMode): Turn on a special mode supporting 'heinous' extensions for drop-in compatibility with the Microsoft Visual C++ product. Standards-compilant C and C++ code isn't guaranteed to work in this mode. Implies MicrosoftExt. Note that full -fms-compatibility mode is currently enabled by default on the Windows target, which may need tuning to serve as a reasonable default. See cfe-commits for the full discourse, thread 'r198497 - Move MS predefined type_info out of InitializePredefinedMacros' No change in behaviour. llvm-svn: 199209
* Sort all the #include lines with LLVM's utils/sort_includes.py whichChandler Carruth2014-01-071-1/+1
| | | | | | | encodes the canonical rules for LLVM's style. I noticed this had drifted quite a bit when cleaning up LLVM, so wanted to clean up Clang as well. llvm-svn: 198686
* Make Preprocessor::Lex non-recursive.Eli Friedman2013-09-191-12/+17
| | | | | | | | | | | | | | | Before this patch, Lex() would recurse whenever the current lexer changed (e.g. upon entry into a macro). This patch turns the recursion into a loop: the various lex routines now don't return a token when the current lexer changes, and at the top level Preprocessor::Lex() now loops until it finds a token. Normally, the recursion wouldn't end up being very deep, but the recursion depth can explode in edge cases like a bunch of consecutive macros which expand to nothing (like in the testcase test/Preprocessor/macro_expand_empty.c in this patch). <rdar://problem/14569770> llvm-svn: 190980
* Use pop_back_val() instead of both back() and pop_back().Robert Wilhelm2013-08-231-4/+2
| | | | | | No functionality change intended. llvm-svn: 189112
* Use SmallVectorImpl& for function arguments instead of SmallVector.Craig Topper2013-07-051-1/+1
| | | | llvm-svn: 185715
* Match MSVC's handling of commas during macro argument expansionReid Kleckner2013-06-261-0/+8
| | | | | | | | | | | | This allows clang to parse the type_traits header in Visual Studio 2012, which is included widely in practice. This is a rework of r163022 by João Matos. The original patch broke preprocessing of gtest headers, which this patch addresses. Patch by Will Wilson! llvm-svn: 184968
* [Preprocessor] Prevent expansion of y in x ## y when x is emptyArgyrios Kyrtzidis2013-05-251-7/+13
| | | | | | | | | | | | When x is empty, x ## is suppressed, and when y gets expanded, the fact that it follows ## is not available in the macro expansion result. The macro definition can be checked instead, the ## will be available there regardless of what x expands to. Fixes http://llvm.org/PR12767 Patch by Harald van Dijk! llvm-svn: 182699
* [Preprocessor] For the MacroExpands preprocessor callback, also pass the ↵Argyrios Kyrtzidis2013-05-031-1/+1
| | | | | | | | MacroArgs object that provides information about the argument tokens for a function macro. llvm-svn: 181065
* [preprocessor] When "merging" macro argument tokens into one SLocEntry chunk,Argyrios Kyrtzidis2012-12-191-2/+6
| | | | | | | | make sure they came from the same kind of FileIDs. Thanks to Abramo Bagnara for providing the test case. llvm-svn: 170616
* Sort all of Clang's files under 'lib', and fix up the broken headersChandler Carruth2012-12-041-2/+2
| | | | | | | | | | | | | uncovered. This required manually correcting all of the incorrect main-module headers I could find, and running the new llvm/utils/sort_includes.py script over the files. I also manually added quite a few missing headers that were uncovered by shuffling the order or moving headers up to be main-module-headers. llvm-svn: 169237
* Try to make the source location information for token pastes a bit more ↵Eli Friedman2012-12-011-0/+6
| | | | | | | | | consistent. Fixes a crash printing diagnostics on the gcc testsuite, and also makes diagnostic range printing print nicer results for token pastes. llvm-svn: 169068
* Improved support for removing the comma preceding __VA_ARGS__ where __VA_ARGS__Andy Gibbs2012-11-091-17/+64
| | | | | | | | | | is empty in a variadic macro expansion. This fixes a divergence in support for the ", ## __VA_ARGS__" GCC extension which differed in behaviour when in strict C99 mode (note: there is no change in behaviour has been made in the gnu99 mode that clang uses by default). In addition, there is improved support for the Microsoft alternative extension ", __VA_ARGS__". llvm-svn: 167613
* Revert r163022, it caused PR13924.Nico Weber2012-09-261-6/+0
| | | | | | | Add a test for PR13924. Do not revert the test added in r163022, it surprisingly still passes even after reverting the code changes. llvm-svn: 164672
* StringRef'ize Preprocessor::CreateString().Dmitri Gribenko2012-09-241-1/+1
| | | | llvm-svn: 164555
* Emulate MSVC's preprocessor macro argument separator behavior by not ↵Joao Matos2012-08-311-0/+6
| | | | | | considering commas from nested macro expansions as argument separators. Fixes parsing of VS 2012 headers. llvm-svn: 163022
* Make preprocessor act in a GCC-compatible fashion when a macro is redefinedRichard Smith2012-08-301-2/+3
| | | | | | | | within its own argument list. The original definition is used for the immediate expansion, but the new definition is used for any subsequent occurences within the argument list or after the expansion. llvm-svn: 162906
* Fix a typo (the the => the)Sylvestre Ledru2012-07-231-1/+1
| | | | llvm-svn: 160622
* Minor improvements to some C99 variadic-macro-related diagnostics.Richard Smith2012-06-221-3/+3
| | | | llvm-svn: 159054
* Fix issue where a token paste which forms a /* or // would discard the rest ofRichard Smith2012-06-131-2/+2
| | | | | | | | the input: token-pasting was producing a tok::eof. Patch by Andy Gibbs! llvm-svn: 158412
* Unify naming of LangOptions variable/get function across the Clang stack ↵David Blaikie2012-03-111-4/+4
| | | | | | | | | | (Lex to AST). The member variable is always "LangOpts" and the member function is always "getLangOpts". Reviewed by Chris Lattner llvm-svn: 152536
* Basic: import SmallString<> into clang namespaceDylan Noblesmith2012-02-051-1/+1
| | | | | | | (I was going to fix the TODO about DenseMap too, but that would break self-host right now. See PR11922.) llvm-svn: 149799
* Move a method from IdentifierTable.h out of line and remove the SmallString ↵Benjamin Kramer2012-02-041-1/+1
| | | | | | | | include. Fix all the transitive include users. llvm-svn: 149783
* Fixed exapnsion range for # and ##.Abramo Bagnara2011-10-031-8/+18
| | | | llvm-svn: 141012
* Rename SourceLocation::getFileLocWithOffset -> getLocWithOffset.Argyrios Kyrtzidis2011-09-191-2/+2
| | | | | | It already works (and is useful with) macro locs as well. llvm-svn: 140057
* Rename LangOptions::Microsoft to LangOptions::MicrosoftExt to make it clear ↵Francois Pichet2011-09-171-3/+3
| | | | | | | | that this flag must be used only for Microsoft extensions and not emulation; to avoid confusion with the new LangOptions::MicrosoftMode flag. Many of the code now under LangOptions::MicrosoftExt will eventually be moved under the LangOptions::MicrosoftMode flag. llvm-svn: 139987
* Silence 'may be used uninitialized' warnings.Argyrios Kyrtzidis2011-08-241-2/+2
| | | | llvm-svn: 138475
* Amend r138129 (reduction of SLocEntries) which introduced performance ↵Argyrios Kyrtzidis2011-08-231-18/+46
| | | | | | | | | | | | | | | | | | | | | | regression due to increased calls to SourceManager::getFileID. (rdar://9992664) Use a slightly different approach that is more efficient both in terms of speed (no extra getFileID calls) and in SLocEntries reduction. Comparing pre-r138129 and this patch we get: For compiling SemaExpr.cpp reduction of SLocEntries by 26%. For the boost enum library: -SLocEntries -34% (note that this was -5% for r138129) -Memory consumption -50% -PCH size -31% Reduced SLocEntries also benefit the hot function SourceManager::getFileID, evident by the reduced "FileID scans". llvm-svn: 138380
* Introduce SourceManager::isInSLocAddrSpace and use it in TokenLexer instead ↵Argyrios Kyrtzidis2011-08-231-12/+9
| | | | | | | | of isInFileID since it is a bit more efficient. llvm-svn: 138379
* Rename SourceManager::isBeforeInSourceLocationOffset -> isBeforeInSLocAddrSpace.Argyrios Kyrtzidis2011-08-231-2/+1
| | | | llvm-svn: 138378
* TokenLexer::getExpansionLocForMacroDefLoc doesn't need to return an invalid ↵Argyrios Kyrtzidis2011-08-231-29/+16
| | | | | | SourceLocation. llvm-svn: 138377
* For assigning SourceLocations to macro arg tokens, reserve a single SLocEntryArgyrios Kyrtzidis2011-08-191-22/+69
| | | | | | | | | | | | | | for tokens that are lexed consecutively from the same FileID, instead of creating a SLocEntry for each token. e.g for assert(foo == bar); there will be a single SLocEntry for the "foo == bar" chunk and locations for the 'foo', '==', 'bar' tokens will point inside that chunk. For parsing SemaExpr.cpp, this reduced the number of SLocEntries by 25%. llvm-svn: 138129
* Rename TokenLexer::getMacroExpansionLocation -> ↵Argyrios Kyrtzidis2011-08-191-8/+10
| | | | | | getExpansionLocForMacroDefLoc, no functionality change. llvm-svn: 138128
* Rename create(MacroArg)InstantiationLoc to create(MacroArg)ExpansionLoc.Chandler Carruth2011-07-261-20/+19
| | | | llvm-svn: 136054
* Mechanically rename SourceManager::getInstantiationLoc andChandler Carruth2011-07-251-1/+1
| | | | | | | | FullSourceLoc::getInstantiationLoc to ...::getExpansionLoc. This is part of the API and documentation update from 'instantiation' as the term for macros to 'expansion'. llvm-svn: 135914
* remove unneeded llvm:: namespace qualifiers on some core types now that ↵Chris Lattner2011-07-231-1/+1
| | | | | | | | LLVM.h imports them into the clang namespace. llvm-svn: 135852
* Revamp the SourceManager to separate the representation of parsedDouglas Gregor2011-07-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | source locations from source locations loaded from an AST/PCH file. Previously, loading an AST/PCH file involved carefully pre-allocating space at the beginning of the source manager for the source locations and FileIDs that correspond to the prefix, and then appending the source locations/FileIDs used for parsing the remaining translation unit. This design forced us into loading PCH files early, as a prefix, whic has become a rather significant limitation. This patch splits the SourceManager space into two parts: for source location "addresses", the lower values (growing upward) are used to describe parsed code, while upper values (growing downward) are used for source locations loaded from AST/PCH files. Similarly, positive FileIDs are used to describe parsed code while negative FileIDs are used to file/macro locations loaded from AST/PCH files. As a result, we can load PCH/AST files even during parsing, making various improvemnts in the future possible, e.g., teaching #include <foo.h> to look for and load <foo.h.gch> if it happens to be already available. This patch was originally written by Sebastian Redl, then brought forward to the modern age by Jonathan Turner, and finally polished/finished by me to be committed. llvm-svn: 135484
* Switch the TokenLexer's terminology from various forms of 'instantiate'Chandler Carruth2011-07-141-21/+21
| | | | | | | | | to 'expand' for macros. Only comments and uses local to the TokenLexer are updated. No functionality changed. llvm-svn: 135137
OpenPOWER on IntegriCloud