summaryrefslogtreecommitdiffstats
path: root/clang/lib/Lex/Lexer.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Improve performance during cursor traversal when a region of interestDouglas Gregor2010-07-221-0/+53
| | | | | | | | | | | | | | | | | | | | | is present. Rather than using clang_getCursorExtent(), which requires us to lex the token at the ending position to determine its length. Then, we'd be comparing [a, b) source ranges that cover the characters in the range rather than the normal behavior for Clang's source ranges, which covers the tokens in the range. However, relexing causes us to read the source file (which may come from a precompiled header), which is rather unfortunate and affects performance. In the new scheme, we only use Clang-style source ranges that cover the tokens in the range. At the entry points where this matters (clang_annotateTokens, clang_getCursor), we make sure to move source locations to the start of the token. Addresses most of <rdar://problem/8049381>. llvm-svn: 109134
* Introduce a new lexer function to compute the "preamble" of a file,Douglas Gregor2010-07-201-0/+125
| | | | | | | | | which is the part of the file that contains all of the initial comments, includes, and preprocessor directives that occur before any of the actual code. Added a new -print-preamble cc1 action that is only used for testing. llvm-svn: 108913
* fix PR4499, patch by Kyle Dean!Chris Lattner2010-07-071-24/+16
| | | | llvm-svn: 107836
* simpler fix for rdar://8044135 - escaped newlines have alreadyChris Lattner2010-05-301-10/+7
| | | | | | been processed, so they don't have to be tip-toed around. llvm-svn: 105182
* Improve our handling of NULL after an escaping '\' in a stringDouglas Gregor2010-05-301-2/+7
| | | | | | literal. Fixes <rdar://problem/8044135>. llvm-svn: 105181
* Improve code completion in failure cases in two ways:Douglas Gregor2010-05-251-0/+3
| | | | | | | | | | | 1) Suppress diagnostics as soon as we form the code-completion token, so we don't get any error/warning spew from the early end-of-file. 2) If we consume a code-completion token when we weren't expecting one, go into a code-completion recovery path that produces the best results it can based on the context that the parser is in. llvm-svn: 104585
* robustify the conflict marker stuff. Don't add 7 twice, which wouldChris Lattner2010-05-171-1/+2
| | | | | | | | | | | | | | | | | | | | | make it miss (invalid) things like: <<<<<<< >>>>>>> and crash if <<<<<<< was at the end of the line. When we find a >>>>>>> that is not at the end of the line, make sure to reset Pos so we don't crash on something like: <<<<<<< >>>>>>> This isn't worth making testcases for, since each would require a new file. rdar://7987078 - signal 11 compiling "<<<<<<<<<<" llvm-svn: 103968
* when code completing inside a C-style block comment, don't emit errors aboutChris Lattner2010-05-161-2/+3
| | | | | | | | a missing */ since we truncated the file. This fixes rdar://7948776 llvm-svn: 103913
* fix a minor bug I noticed while work with Jordy's patch for PR6101,Chris Lattner2010-04-121-2/+4
| | | | | | | | | | | | | | | | | in an input file like this: # 42 int x; we were emitting: # <something> int x; (with a space before the int) because we weren't clearing the leading whitespace flag properly after the \n from the directive was handled. llvm-svn: 101084
* Reinstate my CodeModificationHint -> FixItHint renaming patch, withoutDouglas Gregor2010-03-311-2/+1
| | | | | | the C-only "optimization". llvm-svn: 100022
* Revert r100008, which inexplicably breaks the clang-i686-darwin10 builderDouglas Gregor2010-03-311-1/+2
| | | | llvm-svn: 100018
* Rename CodeModificationHint to FixItHint, since we've been using theDouglas Gregor2010-03-311-2/+1
| | | | | | | term "fix-it" everywhere and even *I* get tired of long names sometimes. No functionality change. llvm-svn: 100008
* Remove unused variableDouglas Gregor2010-03-161-1/+0
| | | | llvm-svn: 98691
* Audit all Preprocessor::getSpelling() callers, improving failureDouglas Gregor2010-03-161-1/+5
| | | | | | recovery for those that need it. llvm-svn: 98689
* Audit all callers of SourceManager::getCharacterData(); update some ofDouglas Gregor2010-03-161-0/+1
| | | | | | them to recover more gracefully on failure. llvm-svn: 98672
* Let SourceManager::getBufferData return StringRef instead of a pair of two ↵Benjamin Kramer2010-03-161-5/+4
| | | | | | const char*. llvm-svn: 98630
* Give SourceManager a Diagnostic object with which to report errors,Douglas Gregor2010-03-161-5/+3
| | | | | | and start simplifying the interfaces in SourceManager that can fail. llvm-svn: 98594
* Introduce a new BufferResult class to act as the return type ofDouglas Gregor2010-03-151-1/+8
| | | | | | | | | | | | | | SourceManager's getBuffer() (and similar) operations. This abstract can be used to force callers to cope with errors in getBuffer(), such as missing files and changed files. Fix a bunch of callers to use the new interface. Add some very basic checks for file consistency (file size, modification time) into ContentCache::getBuffer(), although these checks don't help much until we've updated the main callers (e.g., SourceManager::getSpelling()). llvm-svn: 98585
* don't inform comment handlers about comments in #if 0 blocks,Chris Lattner2010-02-031-7/+9
| | | | | | | | | doing so invalidates the file guard optimization and is not in the spirit of "#if 0" because it is supposed to completely skip everything, even if it isn't lexically valid. Patch by Abramo Bagnara! llvm-svn: 95253
* Teach CIndex's cursor visitor to restrict its traversal to a specificDouglas Gregor2010-01-221-0/+4
| | | | | | | | | | | | | | region of interest (if provided). Implement clang_getCursor() in terms of this traversal rather than using the Index library; the unified cursor visitor is more complete, and will be The Way Forward. Minor other tweaks needed to make this work: - Extend Preprocessor::getLocForEndOfToken() to accept an offset from the end, making it easy to move to the last character in the token (rather than just past the end of the token). - In Lexer::MeasureTokenLength(), the length of whitespace is zero. llvm-svn: 94200
* allow the HandlerComment callback to push tokens into theChris Lattner2010-01-181-14/+24
| | | | | | | preprocessor. This could be used by an OpenMP implementation or something. Patch by Abramo Bagnara! llvm-svn: 93795
* add a TODO for a perf improvement in LexIdentifier.Chris Lattner2010-01-111-2/+5
| | | | llvm-svn: 93141
* Do not parse hexadecimal floating point literals in C++0x mode because they areAlexis Hunt2010-01-101-1/+2
| | | | | | | | | | | | | | | | | | incompatible with user-defined literals, specifically with the following form: 0x1p+1 The preprocessing-number token extends only as far as the 'p'; the '+' is not included. Previously we could get away with this extension as p was an invalid suffix, but now with user-defined literals, 'p' might well be a valid suffix and we are forced to consider it as such. This patch also adds a warning in non-0x C++ modes telling the user that this extension is incompatible with C++0x that is enabled by default (previously and with other languages, we warn only with a compliance option such as -pedantic). llvm-svn: 93135
* reimplement r90860, fixing a couple of problems:Chris Lattner2009-12-171-9/+21
| | | | | | | | | 1. Don't make a copy of LangOptions every time a lexer is created. 2. Don't make CharInfo global mutable state. 3. Fix the implementation to properly treat ^Z as EOF instead of as horizontal whitespace, which matches the semantic implemented by VC++. llvm-svn: 91586
* teach clang to recover gracefully from conflict markers left in sourceChris Lattner2009-12-141-14/+133
| | | | | | files: PR5238. llvm-svn: 91270
* Integrate the following from the 'objective-rewrite' branch:Steve Naroff2009-12-081-4/+9
| | | | | | http://llvm.org/viewvc/llvm-project?view=rev&revision=80043 llvm-svn: 90860
* Extend the source manager with the ability to override the contents ofDouglas Gregor2009-12-021-22/+9
| | | | | | | | | | files with the contents of an arbitrary memory buffer. Use this new functionality to drastically clean up the way in which we handle file truncation for code-completion: all of the truncation/completion logic is now encapsulated in the preprocessor where it belongs (<rdar://problem/7434737>). llvm-svn: 90300
* Fix PR5633 by making the preprocessor handle the case where we canChris Lattner2009-11-301-6/+5
| | | | | | | | | | stat a file but where mmaping it fails. In this case, we emit an error like: t.c:1:10: fatal error: error opening file '../../foo.h' instead of "cannot find file". llvm-svn: 90110
* Move DISABLE_INLINE to the front of the decl so MSVC can parse it. Patch by ↵Benjamin Kramer2009-11-141-4/+4
| | | | | | Amine Khaldi! llvm-svn: 88797
* Teach Lexer::MeasureTokenLength to be able to measure theChris Lattner2009-10-141-0/+1
| | | | | | length of comment tokens. Patch by Abramo Bagnara! llvm-svn: 84100
* Replace the -code-completion-dump option with Douglas Gregor2009-09-221-9/+21
| | | | | | | | | | | -code-completion-at=filename:line:column which performs code completion at the specified location by truncating the file at that position and enabling code completion. This approach makes it possible to run multiple tests from a single test file, and gives a more natural command-line interface. llvm-svn: 82571
* Refactor and simplify the CodeCompleteConsumer, so that all of theDouglas Gregor2009-09-211-12/+13
| | | | | | | real work is performed within Sema. Addresses Chris's comments, but still retains the heavyweight list-of-multimaps data structure. llvm-svn: 82459
* Initial implementation of a code-completion interface in Clang. InDouglas Gregor2009-09-171-1/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | essence, code completion is triggered by a magic "code completion" token produced by the lexer [*], which the parser recognizes at certain points in the grammar. The parser then calls into the Action object with the appropriate CodeCompletionXXX action. Sema implements the CodeCompletionXXX callbacks by performing minimal translation, then forwarding them to a CodeCompletionConsumer subclass, which uses the results of semantic analysis to provide code-completion results. At present, only a single, "printing" code completion consumer is available, for regression testing and debugging. However, the design is meant to permit other code-completion consumers. This initial commit contains two code-completion actions: one for member access, e.g., "x." or "p->", and one for nested-name-specifiers, e.g., "std::". More code-completion actions will follow, along with improved gathering of code-completion results for the various contexts. [*] In the current -code-completion-dump testing/debugging mode, the file is truncated at the completion point and EOF is translated into "code completion". llvm-svn: 82166
* Remove tabs, and whitespace cleanups.Mike Stump2009-09-091-148/+148
| | | | llvm-svn: 81346
* Convert the CharInfo table to be statically initialized, instead of ↵Chris Lattner2009-07-071-15/+86
| | | | | | dynamically initialized. Patch by Ryan Flynn! llvm-svn: 74919
* fix an out-of-date comment.Chris Lattner2009-07-071-5/+3
| | | | llvm-svn: 74894
* Add support for retrieving the Doxygen comment associated with a givenDouglas Gregor2009-07-021-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | declaration in the AST. The new ASTContext::getCommentForDecl function searches for a comment that is attached to the given declaration, and returns that comment, which may be composed of several comment blocks. Comments are always available in an AST. However, to avoid harming performance, we don't actually parse the comments. Rather, we keep the source ranges of all of the comments within a large, sorted vector, then lazily extract comments via a binary search in that vector only when needed (which never occurs in a "normal" compile). Comments are written to a precompiled header/AST file as a blob of source ranges. That blob is only lazily loaded when one requests a comment for a declaration (this never occurs in a "normal" compile). The indexer testbed now supports comment extraction. When the -point-at location points to a declaration with a Doxygen-style comment, the indexer testbed prints the associated comment block(s). See test/Index/comments.c for an example. Some notes: - We don't actually attempt to parse the comment blocks themselves, beyond identifying them as Doxygen comment blocks to associate them with a declaration. - We won't find comment blocks that aren't adjacent to the declaration, because we start our search based on the location of the declaration. - We don't go through the necessary hops to find, for example, whether some redeclaration of a declaration has comments when our current declaration does not. Similarly, we don't attempt to associate a \param Foo marker in a function body comment with the parameter named Foo (although that is certainly possible). - Verification of my "no performance impact" claims is still "to be done". llvm-svn: 74704
* Fix our check for "random whitespace between a \ and newline" to workChris Lattner2009-06-231-2/+3
| | | | | | | | | | | | | | | with dos style newlines. I have a trivial test for this: // RUN: clang-cc %s -verify #define test(x, y) \ x ## y but I don't know how to get svn to not change newlines and testrunner doesn't work with dos style newlines either, so "not worth it". :) rdar://6994000 llvm-svn: 73945
* Fix rdar://6880630 - # in _Pragma does not start a preprocessor directive.Chris Lattner2009-05-131-2/+2
| | | | llvm-svn: 71643
* Get rid of some useless uses of NoExtensions. The philosophy here is Eli Friedman2009-04-281-2/+1
| | | | | | | | | | | | that if we're going to print an extension warning anyway, there's no point to changing behavior based on NoExtensions: it will only make error recovery worse. Note that this doesn't cause any behavior change because NoExtensions isn't used by the current front-end. I'm still considering what to do about the remaining use of NoExtensions in IdentifierTable.cpp. llvm-svn: 70273
* fix rdar://6816766 - Crash with function-like macro test at end of directive.Chris Lattner2009-04-241-0/+2
| | | | llvm-svn: 69964
* add a new Lexer::SkipEscapedNewLines method.Chris Lattner2009-04-181-0/+23
| | | | llvm-svn: 69483
* factor escape newline measuring out into its own helper function.Chris Lattner2009-04-181-40/+45
| | | | llvm-svn: 69482
* remove unneeded scopes.Chris Lattner2009-04-181-45/+41
| | | | llvm-svn: 69481
* Fix two problems from PR3916, and one problem I noticed while hackingChris Lattner2009-04-171-5/+5
| | | | | | on the code. llvm-svn: 69404
* Change Lexer::MeasureTokenLength to take a LangOptions reference.Chris Lattner2009-04-141-6/+2
| | | | | | | | | | | | | | | | | | This allows it to accurately measure tokens, so that we get: t.cpp:8:13: error: unknown type name 'X' static foo::X P; ~~~~~^ instead of the woefully inferior: t.cpp:8:13: error: unknown type name 'X' static foo::X P; ~~~~ ^ Most of this is just plumbing to push the reference around. llvm-svn: 69099
* fix rdar://6757323, where an escaped newline in a // commentChris Lattner2009-04-051-0/+8
| | | | | | was causing the char after the newline to get eaten. llvm-svn: 68430
* A code modification hint for files that don't end in a newline.Mike Stump2009-04-021-1/+3
| | | | | | | Eventually, would be nice to be able to run these modifications even when we don't want the warning or errors for the actual diagnostic. llvm-svn: 68272
* silence some errors that should not apply to .S files on code like:Chris Lattner2009-03-181-4/+4
| | | | | | | | '' ' ' llvm-svn: 67237
* properly form a full token for # before calling HandleDirective.Chris Lattner2009-03-181-6/+7
| | | | llvm-svn: 67235
OpenPOWER on IntegriCloud