summaryrefslogtreecommitdiffstats
path: root/llvm/lib/XRay/Trace.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Switch LLVM to use 64-bit offsets (2/5)Igor Kudrin2019-08-061-18/+25
| | | | | | | | | This updates all libraries and tools in LLVM Core to use 64-bit offsets which directly or indirectly come to DataExtractor. Differential Revision: https://reviews.llvm.org/D65638 llvm-svn: 368014
* Open native file handles to avoid converting from FDs, NFCReid Kleckner2019-07-111-7/+6
| | | | | | Follow up to r365588. llvm-svn: 365820
* [Support] Move llvm::MemoryBuffer to sys::fs::file_tReid Kleckner2019-07-101-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: On Windows, Posix integer file descriptors are a compatibility layer over native file handles provided by the C runtime. There is a hard limit on the maximum number of file descriptors that a process can open, and the limit is 8192. LLD typically doesn't run into this limit because it opens input files, maps them into memory, and then immediately closes the file descriptor. This prevents it from running out of FDs. For various reasons, I'd like to open handles to every input file and keep them open during linking. That requires migrating MemoryBuffer over to taking open native file handles instead of integer FDs. Reviewers: aganea, Bigcheese Reviewed By: aganea Subscribers: smeenai, silvas, mehdi_amini, hiraditya, steven_wu, dexonsmith, dang, llvm-commits, zturner Tags: #llvm Differential Revision: https://reviews.llvm.org/D63453 llvm-svn: 365588
* Use llvm::stable_sortFangrui Song2019-04-231-4/+3
| | | | | | While touching the code, simplify if feasible. llvm-svn: 358996
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* [XRay] Use TSC delta encoding for custom/typed eventsDean Michael Berris2018-11-071-1/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This change updates the version number for FDR logs to 5, and update the trace processing to support changes in the custom event records. In the runtime, since we're already writing down the record preamble to handle CPU migrations and TSC wraparound, we can use the same TSC delta encoding in the custom event and typed event records that we use in function event records. We do the same change to typed events (which were unsupported before this change in the trace processing) which now show up in the trace. Future changes should increase our testing coverage to make custom and typed events as first class entities in the FDR mode log processing tools. This change is also a good example of how we end up supporting new record types in the FDR mode implementation. This shows the places where new record types are added and supported. Depends on D54139. Reviewers: mboerger Subscribers: hiraditya, arphaman, jfb, llvm-commits Differential Revision: https://reviews.llvm.org/D54140 llvm-svn: 346293
* [XRay] Update XRayRecord to support Custom/Typed EventsDean Michael Berris2018-11-061-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: This change cuts across LLVM and compiler-rt to add support for rendering custom events in the XRayRecord type, to allow for including user-provided annotations in the output YAML (as raw bytes). This work enables us to add custom event and typed event records into the `llvm::xray::Trace` type for user-provided events. This can then be programmatically handled through the C++ API and can be included in some of the tooling as well. For now we support printing the raw data we encounter in the custom events in the converted output. Future work will allow us to start interpreting these custom and typed events through a yet-to-be-defined API for extending the trace analysis library. Reviewers: mboerger Subscribers: hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D54139 llvm-svn: 346214
* [XRay] Add CPU ID in Custom Event FDR RecordsDean Michael Berris2018-11-011-7/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This change cuts across compiler-rt and llvm, to increment the FDR log version number to 4, and include the CPU ID in the custom event records. This is a step towards allowing us to change the `llvm::xray::Trace` object to start representing both custom and typed events in the stream of records. Follow-on changes will allow us to change the kinds of records we're presenting in the stream of traces, to incorporate the data in custom/typed events. A follow-on change will handle the typed event case, where it may not fit within the 15-byte buffer for metadata records. This work is part of the larger effort to enable writing analysis and processing tools using a common in-memory representation of the events found in traces. The work will focus on porting existing tools in LLVM to use the common representation and informing the design of a library/framework for expressing trace event analysis as C++ programs. Reviewers: mboerger, eizan Subscribers: hiraditya, mgrang, llvm-commits Differential Revision: https://reviews.llvm.org/D53920 llvm-svn: 345798
* Use the container form llvm::sort(C, ...)Fangrui Song2018-09-301-1/+1
| | | | | | | There are a few leftovers in rL343163 which span two lines. This commit changes these llvm::sort(C.begin(), C.end, ...) to llvm::sort(C, ...) llvm-svn: 343426
* [XRay] Bug fixes for FDR custom event and arg-loggingDean Michael Berris2018-09-131-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This change has a number of fixes for FDR mode in compiler-rt along with changes to the tooling handling the traces in llvm. In the runtime, we do the following: - Advance the "last record" pointer appropriately when writing the custom event data in the log. - Add XRAY_NEVER_INSTRUMENT in the rewinding routine. - When collecting the argument of functions appropriately marked, we should not attempt to rewind them (and reset the counts of functions that can be re-wound). In the tooling, we do the following: - Remove the state logic in BlockIndexer and instead rely on the presence/absence of records to indicate blocks. - Move the verifier into a loop associated with each block. Reviewers: mboerger, eizan Subscribers: llvm-commits, hiraditya Differential Revision: https://reviews.llvm.org/D51965 llvm-svn: 342122
* [XRay] Ensure lambda outlives llvm::function_refDean Michael Berris2018-09-111-2/+2
| | | | | | Follow-up to D51912. llvm-svn: 341912
* [XRay] Use FDR Records+Visitors for Trace LoadingDean Michael Berris2018-09-111-545/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: In this change, we overhaul the implementation for loading `llvm::xray::Trace` objects from files by using the combination of specific FDR Record types and visitors breaking up the logic to reconstitute an execution trace from flight-data recorder mode traces. This change allows us to handle out-of-temporal order blocks as written in files, and more consistently recreate an execution trace spanning multiple blocks and threads. To do this, we use the `WallclockRecord` associated with each block to maintain temporal order of blocks, before attempting to recreate an execution trace. The new addition in this change is the `TraceExpander` type which can be thought of as a decompression/decoding routine. This allows us to maintain the state of an execution environment (thread+process) and create `XRayRecord` instances that fit nicely into the `Trace` container. We don't have a specific unit test for the TraceExpander type, since the end-to-end tests for the `llvm-xray convert` tools already cover precisely this codepath. This change completes the refactoring started with D50441. Depends on D51911. Reviewers: mboerger, eizan Subscribers: mgorny, hiraditya, mgrang, llvm-commits Differential Revision: https://reviews.llvm.org/D51912 llvm-svn: 341906
* [XRay] Use correct type for PID recordsDean Michael Berris2018-08-311-1/+1
| | | | | | | | | Previously we've been reading and writing the wrong types which only worked in little endian implementations. This time we're writing the same typed values the runtime is using, and reading them appropriately as well. llvm-svn: 341241
* [XRay] Use correct type for thread ID parsingDean Michael Berris2018-08-311-3/+3
| | | | | | | Previously we were reading only a uint16_t when we really needed to read an int32_t from the log. llvm-svn: 341239
* [XRay] Change function record reader to be endian-awareDean Michael Berris2018-08-311-12/+13
| | | | | | | | | | | | | | | | | This change allows us to let the compiler do the right thing for when handling big-endian and little-endian records for FDR mode function records. Previously, we assumed that the encoding was little-endian that reading the first byte to look for the function id and function record types was ordered in a little-endian manner. This change allows us to better handle function records where the first four bytes may actually be encoded in big-endian thus giving us the wrong bytes where we're seeking the function information from. This is a follow-up to D51210 and D51289. llvm-svn: 341236
* [XRay] Make Trace loading endian-awareDean Michael Berris2018-08-311-12/+21
| | | | | | | | | | | This change makes the XRay Trace loading functions first use a little-endian data extractor, then on failures try a big-endian data extractor. Without this change, the trace loading facility will not work with data written from a big-endian machine. Follow-up to D51210 and D51289. llvm-svn: 341226
* [XRay] Refactor loadTraceFile(...) into two (NFC)Dean Michael Berris2018-08-241-5/+8
| | | | | | | | | | This patch splits the file trace loading function into two versions, one that takes a filename and one that takes a `DataExtractor`. This change is a precursor to larger changes to increase test coverage for the trace loading implementation. llvm-svn: 340603
* [XRay] Refactor file header reading (NFC)Dean Michael Berris2018-08-221-64/+10
| | | | | | | | | | | | | | | | Summary: This patch moves out the definition of the XRay log file header from binary logs into its own header and implementation file. This is one part of the refactoring being done in D50441. Reviewers: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51086 llvm-svn: 340389
* [XRay] Improve error reporting when loading tracesDean Michael Berris2018-08-071-152/+364
| | | | | | | | | | | | | | | | | | | | | | | Summary: This change uses a single offset pointer used throughout the implementation of the individual record parsers. This allows us to report where in a trace file parsing failed. We're still in an intermediate step here as we prepare to refactor this further into a set of types and use object-oriented design principles for a cleaner implementation. The next steps will be to allow us to parse/dump files in a streaming fashion and incrementally build up the structures in memory instead of the current all-or-nothing approach. Reviewers: kpw, eizan Reviewed By: kpw Subscribers: hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D50169 llvm-svn: 339092
* [XRay][compiler-rt] Add PID field to llvm-xray tool and add PID metadata ↵Dean Michael Berris2018-07-131-11/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | record entry in FDR mode Summary: llvm-xray changes: - account-mode - process-id {...} shows after thread-id - convert-mode - process {...} shows after thread - parses FDR and basic mode pid entries - Checks version number for FDR log parsing. Basic logging changes: - Update header version from 2 -> 3 FDR logging changes: - Update header version from 2 -> 3 - in writeBufferPreamble, there is an additional PID Metadata record (after thread id record and tsc record) Test cases changes: - fdr-mode.cc, fdr-single-thread.cc, fdr-thread-order.cc modified to catch process id output in the log. Reviewers: dberris Reviewed By: dberris Subscribers: hiraditya, llvm-commits, #sanitizers Differential Revision: https://reviews.llvm.org/D49153 llvm-svn: 336974
* [XRay] clarify error messages when parsing broken tracesMartin Pelikan2018-01-301-3/+4
| | | | | | | | | | | | | | | Summary: When there's a mismatch of a function argument being right after the wrong function, print an offset into the file where that happened, to ease further debugging. Reviewers: dberris, eizan, kpw Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D42492 llvm-svn: 323758
* [XRay] Use optimistic logging model for FDR modeDean Michael Berris2017-11-211-22/+122
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Before this change, the FDR mode implementation relied on at thread-exit handling to return buffers back to the (global) buffer queue. This introduces issues with the initialisation of the thread_local objects which, even through the use of pthread_setspecific(...) may eventually call into an allocation function. Similar to previous changes in this line, we're finding that there is a huge potential for deadlocks when initialising these thread-locals when the memory allocation implementation is also xray-instrumented. In this change, we limit the call to pthread_setspecific(...) to provide a non-null value to associate to the key created with pthread_key_create(...). While this doesn't completely eliminate the potential for the deadlock(s), it does allow us to still clean up at thread exit when we need to. The change is that we don't need to do more work when starting and ending a thread's lifetime. We also have a test to make sure that we actually can safely recycle the buffers in case we end up re-using the buffer(s) available from the queue on multiple thread entry/exits. This change cuts across both LLVM and compiler-rt to allow us to update both the XRay runtime implementation as well as the library support for loading these new versions of the FDR mode logging. Version 2 of the FDR logging implementation makes the following changes: * Introduction of a new 'BufferExtents' metadata record that's outside of the buffer's contents but are written before the actual buffer. This data is associated to the Buffer handed out by the BufferQueue rather than a record that occupies bytes in the actual buffer. * Removal of the "end of buffer" records. This is in-line with the changes we described above, to allow for optimistic logging without explicit record writing at thread exit. The optimistic logging model operates under the following assumptions: * Threads writing to the buffers will potentially race with the thread attempting to flush the log. To avoid this situation from occuring, we make sure that when we've finalized the logging implementation, that threads will see this finalization state on the next write, and either choose to not write records the thread would have written or write the record(s) in two phases -- first write the record(s), then update the extents metadata. * We change the buffer queue implementation so that once it's handed out a buffer to a thread, that we assume that buffer is marked "used" to be able to capture partial writes. None of this will be safe to handle if threads are racing to write the extents records and the reader thread is attempting to flush the log. The optimism comes from the finalization routine being required to complete before we attempt to flush the log. This is a fairly significant semantics change for the FDR implementation. This is why we've decided to update the version number for FDR mode logs. The tools, however, still need to be able to support older versions of the log until we finally deprecate those earlier versions. Reviewers: dblaikie, pelikan, kpw Subscribers: llvm-commits, hiraditya Differential Revision: https://reviews.llvm.org/D39526 llvm-svn: 318733
* [XRay] Stable sort XRayRecord to remove non-deterministic orderingMandeep Singh Grang2017-11-141-1/+1
| | | | | | | | | | | | | | | | Summary: This fixes failure in tools/llvm-xray/X86/graph-zero-latency-calls.yaml uncovered by D39245. Reviewers: dberris Reviewed By: dberris Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39943 llvm-svn: 318163
* [XRay][tools] Support arg1 logging entries in the basic logging modeDean Michael Berris2017-10-051-21/+51
| | | | | | | | | | | | | | | | | | | | | Summary: The arg1 logging handler changed in compiler-rt to start writing a different type for entries encountered when logging the first argument of XRay-instrumented functions. This change allows the trace loader to support reading these record types as well as prepare for when the basic (naive) mode implementation starts writing down the argument payloads. Without this change, binaries with arg1 logging support enabled start writing unreadable logs for any of the XRay tracing tools. Reviewers: pelikan Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D38550 llvm-svn: 314967
* [XRay] initialize all members of YAMLXRayRecord for -Wmissing-field-initializersMartin Pelikan2017-09-271-1/+1
| | | | llvm-svn: 314278
* [XRay] convert FDR arg1 log entriesMartin Pelikan2017-09-271-2/+27
| | | | | | | | | | | | | | | | | | | | Summary: A new FDR metadata record will support logging a function call argument; appending multiple metadata records will represent a sequence of arguments meaning that "holes" are not representable by the buffer format. Each call argument is currently a 64-bit value (useful for "this" pointers and synchronization objects). If present, we put this argument to the function call "entry" record it belongs to, and alter its type to notify the user of its presence. Reviewers: dberris Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D32840 llvm-svn: 314269
* [XRay][tools] Support tail-call exits before we write them in the runtimeDean Michael Berris2017-09-181-6/+10
| | | | | | | | | | | | | | | | | Summary: This change adds support for explicit tail-exit records to be written by the XRay runtime. This lets us differentiate the tail exit records/events in the log, and allows us to treat those exit events especially in the future. For now we allow printing those out in YAML (and reading them in). Reviewers: kpw, pelikan Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D37964 llvm-svn: 313514
* [XRay] fix and clarify comments in the log file decoderMartin Pelikan2017-09-151-30/+24
| | | | | | | | | | | | | | | | | | | Summary: For readers unfamiliar with the XRay code base, reference the compiler-rt implementation even though we're not allowed to share any code and explain our little-endian views more clearly. For code clarity either get rid of obvious comments or explain their intentions, fix typos, correct coding style according to LLVM's standards and manually CSE long expressions to point out it is the same expression. Reviewers: dberris Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34339 llvm-svn: 313340
* Move helper classes into anonymous namespaces.Benjamin Kramer2017-08-201-1/+3
| | | | | | No functionality change intended. llvm-svn: 311288
* Xray docs with description of Flight Data Recorder binary format.Keith Wyss2017-08-021-5/+6
| | | | | | | | | | | | | | | | | | | Summary: Adding a new restructuredText file to document the trace format produced with an FDR mode handler and read by llvm-xray toolset. Fixed two problems in the documentation from differential review. One bad table and a missing link in the toc. Original commit was e97c5836a77db803fe53319c53f3bf8e8b26d2b7. Reviewers: dberris, pelikan Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D36041 llvm-svn: 309891
* Revert "Xray docs with description of Flight Data Recorder binary format."Keith Wyss2017-08-021-6/+5
| | | | | | | | This reverts commit 3462b8ad41a840fd54dbbd0d3f2a514c5ad6f656. The docs-llvm-html target failed. llvm-svn: 309842
* Xray docs with description of Flight Data Recorder binary format.Keith Wyss2017-08-021-5/+6
| | | | | | | | | | | | | | Summary: Adding a new restructuredText file to document the trace format produced with an FDR mode handler and read by llvm-xray toolset. Reviewers: dberris, pelikan Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D36041 llvm-svn: 309836
* [XRay][lib] Support and temporarily skip over CustomEvent recordsDean Michael Berris2017-05-121-2/+32
| | | | | | | | | | | | | | | | Summary: In D30630 we will start writing custom event records. To avoid breaking the tools that read the FDR mode records, we skip over these records. To support these custom event records more effectively, we will have to expose them in the trace loading API. Those changes will be forthcoming. Reviewers: kpw, pelikan Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D33032 llvm-svn: 302856
* [XRay] - Fix spelling error to test commit access.Keith Wyss2017-04-061-1/+1
| | | | | | Just a spelling change in a comment intended to test svn commit access. llvm-svn: 299616
* Spelling mistakes in comments. NFCI.Simon Pilgrim2017-03-301-1/+1
| | | | | | Based on corrections mentioned in patch for clang for PR27635 llvm-svn: 299072
* [XRay] Update FDR log reader to be aware of buffer sizes per thread.Dean Michael Berris2017-03-291-19/+67
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: It is problematic for this reader that it expects to read data from several threads, but the header or message format does not define framing. Since the buffers are reused, we can't rely on skipping zeroed out data as a synchronization method either. There is an argument that this is not version compatible with the format the reader expected previously. I argue that since the writer wrote garbage past the end of buffer record, there is no currently working reader to compromise. The corresponding writer change is posted to D31384. Reviewers: dberris, pelikan Reviewed By: dberris Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31385 llvm-svn: 298983
* [XRAY] [x86_64] Adding a Flight Data filetype reader to the llvm-xray Trace ↵Dean Michael Berris2017-02-171-19/+304
| | | | | | | | | | | | | | | | | | | | | | | | implementation. Summary: The file type packs function trace data onto disk from potentially multiple threads that are aggregated and flushed during the course of an instrumented program's runtime. It is named FDR mode or Flight Data recorder as an analogy to plane blackboxes, which instrument a running system without access to IO. The writer code is defined in compiler-rt in xray_fdr_logging.h/cc Reviewers: rSerge, kcc, dberris Reviewed By: dberris Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D29697 llvm-svn: 295397
* Move symbols from the global namespace into (anonymous) namespaces. NFC.Benjamin Kramer2017-02-111-4/+4
| | | | llvm-svn: 294837
* Avoid std::errc::protocol_* to appease mingwHans Wennborg2017-01-121-1/+1
| | | | | | Like r291636 and r285261. llvm-svn: 291786
* [XRay] Define the library for XRay trace logsDean Michael Berris2017-01-111-0/+196
Summary: In this change we move the definition of the log reading routines from the tools directory in LLVM to {include/llvm,lib}/XRay. We improve the documentation a little bit for the publicly accessible headers, and adjust the top-matter. This also leads to some refactoring and cleanup in the tooling code. In particular, we do the following: - Rename the class from LogReader to Trace, as it better represents the logical set of records as opposed to a log. - Use file type detection instead of asking the user to say what format the input file is. This allows us to keep the interface simple and encapsulate the logic of loading the data appropriately. In future changes we increase the API surface and write dedicated unit tests for the XRay library. Depends on D24376. Reviewers: dblaikie, echristo Subscribers: mehdi_amini, mgorny, llvm-commits, varno Differential Revision: https://reviews.llvm.org/D28345 llvm-svn: 291652
OpenPOWER on IntegriCloud