summaryrefslogtreecommitdiffstats
path: root/clang/lib/Lex/TokenLexer.cpp
Commit message (Collapse)AuthorAgeFilesLines
...
* Mechanically rename SourceManager::getInstantiationLoc andChandler Carruth2011-07-251-1/+1
| | | | | | | | FullSourceLoc::getInstantiationLoc to ...::getExpansionLoc. This is part of the API and documentation update from 'instantiation' as the term for macros to 'expansion'. llvm-svn: 135914
* remove unneeded llvm:: namespace qualifiers on some core types now that ↵Chris Lattner2011-07-231-1/+1
| | | | | | | | LLVM.h imports them into the clang namespace. llvm-svn: 135852
* Revamp the SourceManager to separate the representation of parsedDouglas Gregor2011-07-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | source locations from source locations loaded from an AST/PCH file. Previously, loading an AST/PCH file involved carefully pre-allocating space at the beginning of the source manager for the source locations and FileIDs that correspond to the prefix, and then appending the source locations/FileIDs used for parsing the remaining translation unit. This design forced us into loading PCH files early, as a prefix, whic has become a rather significant limitation. This patch splits the SourceManager space into two parts: for source location "addresses", the lower values (growing upward) are used to describe parsed code, while upper values (growing downward) are used for source locations loaded from AST/PCH files. Similarly, positive FileIDs are used to describe parsed code while negative FileIDs are used to file/macro locations loaded from AST/PCH files. As a result, we can load PCH/AST files even during parsing, making various improvemnts in the future possible, e.g., teaching #include <foo.h> to look for and load <foo.h.gch> if it happens to be already available. This patch was originally written by Sebastian Redl, then brought forward to the modern age by Jonathan Turner, and finally polished/finished by me to be committed. llvm-svn: 135484
* Switch the TokenLexer's terminology from various forms of 'instantiate'Chandler Carruth2011-07-141-21/+21
| | | | | | | | | to 'expand' for macros. Only comments and uses local to the TokenLexer are updated. No functionality changed. llvm-svn: 135137
* Tweak formatting.Chandler Carruth2011-07-081-1/+1
| | | | llvm-svn: 134675
* Switch the token-paste source locations inside of function style macroChandler Carruth2011-07-081-3/+3
| | | | | | | | argument expansion to use the macro argument source locations as well. Add a few tests to exercise this. There is still a bit more work needed here though. llvm-svn: 134674
* Keep track of which source locations are part of a macro argumentChandler Carruth2011-07-071-3/+3
| | | | | | | | | | | | | | | | | | | | instantiation and improve diagnostics which are stem from macro arguments to trace the argument itself back through the layers of macro expansion. This requires some tricky handling of the source locations, as the argument appears to be expanded in the opposite direction from the surrounding macro. This patch provides helper routines that encapsulate the logic and explain the reasoning behind how we step through macros during diagnostic printing. This fixes the rest of the test cases originially in PR9279, and later split out into PR10214 and PR10215. There is still some more work we can do here to improve the macro backtrace, but those will follow as separate patches. llvm-svn: 134660
* Turn hashhash into tok::unkwown when it comes from expanding an argument, ↵Argyrios Kyrtzidis2011-07-071-8/+19
| | | | | | per Chris' suggestion. llvm-svn: 134621
* When expanding macro arguments, treat '##' coming from an argument as a ↵Argyrios Kyrtzidis2011-07-071-5/+12
| | | | | | | | | | | normal token. e.g. #define M(x) A x B M(##) // should expand to 'A ## B', not 'AB' llvm-svn: 134588
* Make the Preprocessor more memory efficient and improve macro instantiation ↵Argyrios Kyrtzidis2011-07-071-10/+119
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | diagnostics. When a macro instantiation occurs, reserve a SLocEntry chunk with length the full length of the macro definition source. Set the spelling location of this chunk to point to the start of the macro definition and any tokens that are lexed directly from the macro definition will get a location from this chunk with the appropriate offset. For any tokens that come from argument expansion, '##' paste operator, etc. have their instantiation location point at the appropriate place in the instantiated macro definition (the argument identifier and the '##' token respectively). This improves macro instantiation diagnostics: Before: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:5:11: note: instantiated from: int y = M(/); ^ After: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:3:20: note: instantiated from: \#define M(op) (foo op 3); ~~~ ^ ~ t.c:5:11: note: instantiated from: int y = M(/); ^ The memory savings for a candidate boost library that abuses the preprocessor are: - 32% less SLocEntries (37M -> 25M) - 30% reduction in PCH file size (900M -> 635M) - 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M) llvm-svn: 134587
* Introduce a caching mechanism for macro expanded tokens.Argyrios Kyrtzidis2011-06-291-9/+5
| | | | | | | | | | | | | | | Previously macro expanded tokens were added to Preprocessor's bump allocator and never released, even after the TokenLexer that were lexing them was finished, thus they were wasting memory. A very "useful" boost library was causing clang to eat 1 GB just for the expanded macro tokens. Introduce a special cache that works like a stack; a TokenLexer can add the macro expanded tokens in the cache, and when it finishes, the tokens are removed from the end of the cache. Now consumed memory by expanded tokens for that library is ~ 1.5 MB. Part of rdar://9327049. llvm-svn: 134105
* revert r133003 and fix the bug properly: the issue was that ## in a tokenChris Lattner2011-06-141-5/+5
| | | | | | | | | | lexer is not a paste operator, it is a normal token. This fixes a conformance issue shown here: http://p99.gforge.inria.fr/c99-conformance/c99-conformance-clang-2.9.html and it defines away the crash from before. llvm-svn: 133005
* Fix a crash on the testcase in PR9981 / rdar://9486765.Chris Lattner2011-06-141-3/+4
| | | | llvm-svn: 133003
* Parsing/AST support for Structured Exception HandlingJohn Wiegley2011-04-281-5/+1
| | | | | | | | Patch authored by Sohail Somani. Provide parsing and AST support for Windows structured exception handling. llvm-svn: 130366
* Rename tok::eom to tok::eod.Peter Collingbourne2011-02-281-1/+1
| | | | | | | | The previous name was inaccurate as this token in fact appears at the end of every preprocessing directive, not just macro definitions. No functionality change, except for a diagnostic tweak. llvm-svn: 126631
* Make TokenLexer capable of storing preprocessor directive tokensPeter Collingbourne2011-02-221-0/+5
| | | | llvm-svn: 126220
* Introduced raw_identifier token kind.Abramo Bagnara2010-12-221-4/+5
| | | | llvm-svn: 122394
* Rename alignof -> alignOf to avoid irritating C++'0x compilers,Chris Lattner2010-10-301-1/+1
| | | | | | PR8423 llvm-svn: 117775
* fix PR7943, a corner case with the GNU __VA_ARGS__ comma Chris Lattner2010-08-211-0/+7
| | | | | | swallowing extension. llvm-svn: 111701
* Random temporary string cleanup.Benjamin Kramer2010-08-111-1/+1
| | | | llvm-svn: 110807
* Add another terrible VC++ compatibility hack: allow users toChris Lattner2010-07-171-2/+7
| | | | | | | allow invalid token pastes (when in -fms-extensions mode) with -Wno-invalid-token-paste llvm-svn: 108624
* Audit all Preprocessor::getSpelling() callers, improving failureDouglas Gregor2010-03-161-3/+8
| | | | | | recovery for those that need it. llvm-svn: 98689
* Let SourceManager::getBufferData return StringRef instead of a pair of two ↵Benjamin Kramer2010-03-161-1/+1
| | | | | | const char*. llvm-svn: 98630
* Give SourceManager a Diagnostic object with which to report errors,Douglas Gregor2010-03-161-2/+3
| | | | | | and start simplifying the interfaces in SourceManager that can fail. llvm-svn: 98594
* Introduce a new BufferResult class to act as the return type ofDouglas Gregor2010-03-151-1/+4
| | | | | | | | | | | | | | SourceManager's getBuffer() (and similar) operations. This abstract can be used to force callers to cope with errors in getBuffer(), such as missing files and changed files. Fix a bunch of callers to use the new interface. Add some very basic checks for file consistency (file size, modification time) into ContentCache::getBuffer(), although these checks don't help much until we've updated the main callers (e.g., SourceManager::getSpelling()). llvm-svn: 98585
* Use SmallString instead of SmallVectorKovarththanan Rajaratnam2010-03-131-1/+1
| | | | llvm-svn: 98436
* No need to call setIdentifierInfo() after LookUpIdentifierInfo() which ↵Kovarththanan Rajaratnam2010-03-131-3/+2
| | | | | | LookUpIdentifierInfo() will automatically do llvm-svn: 98435
* The PreExpArgTokens array is indexed with an argument #,Chris Lattner2009-12-281-1/+1
| | | | | | | not a token number. Fix the reserve logic to get the right amount of space. llvm-svn: 92202
* comment tweakChris Lattner2009-12-231-1/+1
| | | | llvm-svn: 92055
* move the VarargsElided member of MacrosArgs to shrink the MacroArgs structChris Lattner2009-12-141-1/+1
| | | | | | | on 64-bit targets. Pass Preprocessor into create/destroy methods of MacroArgs even though it isn't used yet. llvm-svn: 91345
* minor tidy.Chris Lattner2009-12-041-6/+5
| | | | llvm-svn: 90543
* Remove tabs, and whitespace cleanups.Mike Stump2009-09-091-60/+60
| | | | llvm-svn: 81346
* fix the "pasting formed 'a]', an invalid preprocessing token"Chris Lattner2009-05-281-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | diagnostic to include the full instantiation location for the invalid paste. For: #define foo(a, b) a ## b #define bar(x) foo(x, ]) bar(a) bar(zdy) Instead of: t.c:3:22: error: pasting formed 'a]', an invalid preprocessing token #define foo(a, b) a ## b ^ t.c:3:22: error: pasting formed 'zdy]', an invalid preprocessing token we now produce: t.c:7:1: error: pasting formed 'a]', an invalid preprocessing token bar(a) ^ t.c:4:16: note: instantiated from: #define bar(x) foo(x, ]) ^ t.c:3:22: note: instantiated from: #define foo(a, b) a ## b ^ t.c:8:1: error: pasting formed 'zdy]', an invalid preprocessing token bar(zdy) ^ t.c:4:16: note: instantiated from: #define bar(x) foo(x, ]) ^ t.c:3:22: note: instantiated from: #define foo(a, b) a ## b ^ llvm-svn: 72519
* Make the bad paste diagnostic print the entire pasted token.Eli Friedman2009-05-271-1/+1
| | | | llvm-svn: 72497
* improve comment, no functionality change.Chris Lattner2009-05-251-10/+15
| | | | llvm-svn: 72386
* Make sure an invalid concatentaion doesn't insert whitespace before Eli Friedman2009-05-241-1/+5
| | | | | | the RHS. Fixes assembler-with-cpp issue reported on cfe-dev. llvm-svn: 72370
* The TokenLexer may encounter annotations if the parser enters them using ↵Argyrios Kyrtzidis2009-05-221-1/+2
| | | | | | | | Preprocessor::EnterTokenStream. So check for annotation before using the Token's IdentifierInfo. llvm-svn: 72278
* Fix for PR4132: make sure to insert whitespace consistently before a Eli Friedman2009-05-031-1/+1
| | | | | | pasted token. llvm-svn: 70793
* Silence gcc warnings.Eli Friedman2009-04-251-2/+2
| | | | llvm-svn: 70086
* move token paste poisoning diagnostics to after the instantiation locChris Lattner2009-04-191-11/+16
| | | | | | | for a token is set, this makes the diagnostic "expanded from stack" work for this diagnostic. Add a testcase for PR3918. llvm-svn: 69544
* Fix PR3918: Invalid use of __VA_ARGS__ not diagnosed,Chris Lattner2009-04-191-2/+13
| | | | | | | by rejecting invalid poisoned tokens in the token pasting path. llvm-svn: 69536
* The individual pieces of an invalid paste as still candidates for expansion.Chris Lattner2009-03-231-1/+3
| | | | | | | This matters in assembler mode, where this is silently allowed. This fixes rdar://6709206. llvm-svn: 67539
* constructs like:Chris Lattner2009-03-181-3/+4
| | | | | | | | | #define Y X ## . Y are ok in .S files. llvm-svn: 67231
* simplify some logic by making ScratchBuffer handle the application of trailingChris Lattner2009-03-081-35/+28
| | | | | | | \0's to created tokens instead of making all clients do it. No functionality change. llvm-svn: 66373
* make the token lexer allocate its temporary token buffers forChris Lattner2009-03-041-2/+9
| | | | | | | preexpanded macro arguments from the preprocessor's bump pointer. This reduces # mallocs from 12444 to 11792. llvm-svn: 66025
* track "just a little more" location information for macro instantiations.Chris Lattner2009-02-151-8/+10
| | | | | | | | | | | | Now instead of just tracking the expansion history, also track the full range of the macro that got replaced. For object-like macros, this doesn't change anything. For _Pragma and function-like macros, this means we track the locations of the ')'. This is required for PR3579 because apparently GCC uses the line of the ')' of a function-like macro as the location to expand __LINE__ to. llvm-svn: 64601
* move library-specific diagnostic headers into library private dirs. ReduceChris Lattner2009-01-291-1/+1
| | | | | | redundant #includes. Patch by Anders Johnsen! llvm-svn: 63271
* Split the single monolithic DiagnosticKinds.def file into oneChris Lattner2009-01-271-1/+1
| | | | | | | | | .def file for each library. This means that adding a diagnostic to sema doesn't require all the other libraries to be rebuilt. Patch by Anders Johnsen! llvm-svn: 63111
* remove my hacks that aggressively threw away multiple Chris Lattner2009-01-261-6/+0
| | | | | | | | | | | instantiation history in an effort to speed up c99-intconst-1.c. Now that multiple nested instantiations are allowed, we just make them and don't pay the cost of lookups. With the other changes that went in before this, reverting this is actually a speedup for c99-intconst-1.c, speeding it up from 1.96s to 1.80s, and preserves much better loc info. llvm-svn: 63036
* This change refactors some of the low-level lexer interfaces a bit.Chris Lattner2009-01-261-17/+22
| | | | | | | | | | | | | Token now has a class of kinds for "literals", which include numeric constants, strings, etc. These tokens can optionally have a pointer to the start of the token in the lexer buffer. This makes it faster to get spelling and do other gymnastics, because we don't have to go through source locations. This change is performance neutral, but will make other changes more feasible down the road. llvm-svn: 63028
OpenPOWER on IntegriCloud