1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
|
# DExTer (Debugging Experience Tester)
## Introduction
DExTer is a suite of tools used to evaluate the "User Debugging Experience". DExTer drives an external debugger, running on small test programs, and collects information on the behavior at each debugger step to provide quantitative values that indicate the quality of the debugging experience.
## Supported Debuggers
DExTer currently supports the Visual Studio 2015 and Visual Studio 2017 debuggers via the [DTE interface](https://docs.microsoft.com/en-us/dotnet/api/envdte.dte), and LLDB via its [Python interface](https://lldb.llvm.org/python-reference.html). GDB is not currently supported.
The following command evaluates your environment, listing the available and compatible debuggers:
dexter.py list-debuggers
## Dependencies
[TODO] Add a requirements.txt or an install.py and document it here.
### Python 3.6
DExTer requires python version 3.6 or greater.
### pywin32 python package
This is required to access the DTE interface for the Visual Studio debuggers.
<python-executable> -m pip install pywin32
### clang
DExTer is current compatible with 'clang' and 'clang-cl' compiler drivers. The compiler must be available for DExTer, for example the following command should successfully build a runnable executable.
<compiler-executable> tests/nostdlib/fibonacci/test.cpp
## Running a test case
The following DExTer commands build the test.cpp from the tests/nostdlib/fibonacci directory and quietly runs it on the Visual Studio debugger, reporting the debug experience heuristic. The first command builds with no optimizations (/Od) and scores 1.0000. The second command builds with optimizations (/Ox) and scores 0.2832 which suggests a worse debugging experience.
dexter.py test --builder clang-cl_vs2015 --debugger vs2017 --cflags="/Od /Zi" --ldflags="/Zi" -- tests/nostdlib/fibonacci
fibonacci = (1.0000)
dexter.py test --builder clang-cl_vs2015 --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi" -- tests/nostdlib/fibonacci
fibonacci = (0.2832)
## An example test case
The sample test case (tests/nostdlib/fibonacci) looks like this:
1. #ifdef _MSC_VER
2. # define DEX_NOINLINE __declspec(noinline)
3. #else
4. # define DEX_NOINLINE __attribute__((__noinline__))
5. #endif
6.
7. DEX_NOINLINE
8. void Fibonacci(int terms, int& total)
9. {
0. int first = 0;
11. int second = 1;
12. for (int i = 0; i < terms; ++i)
13. {
14. int next = first + second; // DexLabel('start')
15. total += first;
16. first = second;
17. second = next; // DexLabel('end')
18. }
19. }
20.
21. int main()
22. {
23. int total = 0;
24. Fibonacci(5, total);
25. return total;
26. }
27.
28. /*
29. DexExpectWatchValue('i', '0', '1', '2', '3', '4',
30. from_line='start', to_line='end')
31. DexExpectWatchValue('first', '0', '1', '2', '3', '5',
32. from_line='start', to_line='end')
33. DexExpectWatchValue('second', '1', '2', '3', '5',
34 from_line='start', to_line='end')
35. DexExpectWatchValue('total', '0', '1', '2', '4', '7',
36. from_line='start', to_line='end')
37. DexExpectWatchValue('next', '1', '2', '3', '5', '8',
38. from_line='start', to_line='end')
39. DexExpectWatchValue('total', '7', on_line=25)
40. DexExpectStepKind('FUNC_EXTERNAL', 0)
41. */
[DexLabel][1] is used to give a name to a line number.
The [DexExpectWatchValue][2] command states that an expression, e.g. `i`, should
have particular values, `'0', '1', '2', '3','4'`, sequentially over the program
lifetime on particular lines. You can refer to a named line or simply the line
number (See line 39).
At the end of the test is the following line:
DexExpectStepKind('FUNC_EXTERNAL', 0)
This [DexExpectStepKind][3] command indicates that we do not expect the debugger
to step into a file outside of the test directory.
[1]: Commands.md#DexLabel
[2]: Commands.md#DexExpectWatchValue
[3]: Commands.md#DexExpectStepKind
## Detailed DExTer reports
Running the command below launches the tests/nostdlib/fibonacci test case in DExTer, using clang-cl as the compiler, Visual Studio 2017 as the debugger, and producing a detailed report:
$ dexter.py test --builder clang-cl_vs2015 --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi" -v -- tests/nostdlib/fibonacci
The detailed report is enabled by `-v` and shows a breakdown of the information from each debugger step. For example:
fibonacci = (0.2832)
## BEGIN ##
[1, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 23, 1, "BREAKPOINT", "FUNC", {}]
[2, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 24, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
[3, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 25, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
. [4, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "FUNC", {}]
. [5, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [6, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [7, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [8, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [9, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "0"}]
. [10, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [11, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [12, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "1"}]
. [13, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [14, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [15, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "2", "total": "0", "first": "1"}]
. [16, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [17, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [18, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "3", "total": "0", "first": "2"}]
. [19, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [20, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [21, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "5", "total": "0", "first": "3"}]
. [22, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
. [23, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
. [24, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 20, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
[25, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 26, 1, "BREAKPOINT", "FUNC", {"total": "7"}]
## END (25 steps) ##
step kind differences [0/1]
FUNC_EXTERNAL:
0
test.cpp:15-18 [first] [9/21]
expected encountered values:
0
1
2
3
missing values:
5 [-6]
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 7 (Variable is optimized away and not available.)
step 8 (Variable is optimized away and not available.)
step 11 (Variable is optimized away and not available.)
step 14 (Variable is optimized away and not available.)
step 17 (Variable is optimized away and not available.)
step 20 (Variable is optimized away and not available.)
step 23 (Variable is optimized away and not available.)
test.cpp:15-18 [i] [15/21]
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 7 (Variable is optimized away and not available.) [-3]
step 8 (Variable is optimized away and not available.) [-3]
step 9 (Variable is optimized away and not available.) [-3]
step 11 (Variable is optimized away and not available.) [-3]
step 12 (Variable is optimized away and not available.)
step 14 (Variable is optimized away and not available.)
step 15 (Variable is optimized away and not available.)
step 17 (Variable is optimized away and not available.)
step 18 (Variable is optimized away and not available.)
step 20 (Variable is optimized away and not available.)
step 21 (Variable is optimized away and not available.)
step 23 (Variable is optimized away and not available.)
test.cpp:15-18 [second] [21/21]
expected encountered values:
1
2
3
5
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 7 (Variable is optimized away and not available.) [-3]
step 8 (Variable is optimized away and not available.) [-3]
step 11 (Variable is optimized away and not available.) [-3]
step 14 (Variable is optimized away and not available.) [-3]
step 17 (Variable is optimized away and not available.) [-3]
step 20 (Variable is optimized away and not available.) [-3]
step 23 (Variable is optimized away and not available.)
test.cpp:15-18 [total] [21/21]
expected encountered values:
0
missing values:
1 [-6]
2 [-6]
4 [-6]
7 [-3]
test.cpp:16-18 [next] [15/21]
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 8 (Variable is optimized away and not available.) [-3]
step 11 (Variable is optimized away and not available.) [-3]
step 14 (Variable is optimized away and not available.) [-3]
step 17 (Variable is optimized away and not available.) [-3]
step 20 (Variable is optimized away and not available.)
step 23 (Variable is optimized away and not available.)
test.cpp:26 [total] [0/7]
expected encountered values:
7
The first line
fibonacci = (0.2832)
shows a score of 0.2832 suggesting that unexpected behavior has been seen. This score is on scale of 0.0000 to 1.000, with 0.000 being the worst score possible and 1.000 being the best score possible. The verbose output shows the reason for any scoring. For example:
test.cpp:15-18 [first] [9/21]
expected encountered values:
0
1
2
3
missing values:
5 [-6]
result optimized away:
step 5 (Variable is optimized away and not available.) [-3]
step 7 (Variable is optimized away and not available.)
step 8 (Variable is optimized away and not available.)
step 11 (Variable is optimized away and not available.)
step 14 (Variable is optimized away and not available.)
step 17 (Variable is optimized away and not available.)
step 20 (Variable is optimized away and not available.)
step 23 (Variable is optimized away and not available.)
shows that for `first` the expected values 0, 1, 2 and 3 were seen, 5 was not. On some steps the variable was reported as being optimized away.
## Writing new test cases
Each test requires a `test.cfg` file. Currently the contents of this file are not read, but its presence is used to determine the root directory of a test. In the future, configuration variables for the test such as supported language modes may be stored in this file. Use the various [commands](Commands.md) to encode debugging expectations.
## Additional tools
For clang-based compilers, the `clang-opt-bisect` tool can be used to get a breakdown of which LLVM passes may be contributing to debugging experience issues. For example:
$ dexter.py clang-opt-bisect tests/nostdlib/fibonacci --builder clang-cl --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi"
pass 1/211 = (1.0000) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]
pass 2/211 = (0.7611) (-0.2389) [SROA on function (?Fibonacci@@YAXHAEAH@Z)]
pass 3/211 = (0.7611) (0.0000) [Early CSE on function (?Fibonacci@@YAXHAEAH@Z)]
pass 4/211 = (0.7611) (0.0000) [Simplify the CFG on function (main)]
pass 5/211 = (0.7611) (0.0000) [SROA on function (main)]
pass 6/211 = (0.7611) (0.0000) [Early CSE on function (main)]
pass 7/211 = (0.7611) (0.0000) [Infer set function attributes on module (c:\dexter\tests\fibonacci\test.cpp)]
pass 8/211 = (0.7611) (0.0000) [Interprocedural Sparse Conditional Constant Propagation on module (c:\dexter\tests\fibonacci\test.cpp)]
pass 9/211 = (0.7611) (0.0000) [Called Value Propagation on module (c:\dexter\tests\fibonacci\test.cpp)]
pass 10/211 = (0.7611) (0.0000) [Global Variable Optimizer on module (c:\dexter\tests\fibonacci\test.cpp)]
pass 11/211 = (0.7611) (0.0000) [Promote Memory to Register on function (?Fibonacci@@YAXHAEAH@Z)]
pass 12/211 = (0.7611) (0.0000) [Promote Memory to Register on function (main)]
pass 13/211 = (0.7611) (0.0000) [Dead Argument Elimination on module (c:\dexter\tests\fibonacci\test.cpp)]
pass 14/211 = (0.7611) (0.0000) [Combine redundant instructions on function (?Fibonacci@@YAXHAEAH@Z)]
pass 15/211 = (0.7611) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]a
pass 16/211 = (0.7345) (-0.0265) [Combine redundant instructions on function (main)]
pass 17/211 = (0.7345) (0.0000) [Simplify the CFG on function (main)]
pass 18/211 = (0.7345) (0.0000) [Remove unused exception handling info on SCC (?Fibonacci@@YAXHAEAH@Z)]
pass 19/211 = (0.7345) (0.0000) [Function Integration/Inlining on SCC (?Fibonacci@@YAXHAEAH@Z)]
pass 20/211 = (0.7345) (0.0000) [Deduce function attributes on SCC (?Fibonacci@@YAXHAEAH@Z)]
pass 21/211 = (0.7345) (0.0000) [SROA on function (?Fibonacci@@YAXHAEAH@Z)]
pass 22/211 = (0.7345) (0.0000) [Early CSE w/ MemorySSA on function (?Fibonacci@@YAXHAEAH@Z)]
pass 23/211 = (0.7345) (0.0000) [Speculatively execute instructions if target has divergent branches on function (?Fibonacci@@YAXHAEAH@Z)]
pass 24/211 = (0.7345) (0.0000) [Jump Threading on function (?Fibonacci@@YAXHAEAH@Z)]
pass 25/211 = (0.7345) (0.0000) [Value Propagation on function (?Fibonacci@@YAXHAEAH@Z)]
pass 26/211 = (0.7345) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]
pass 27/211 = (0.7345) (0.0000) [Combine redundant instructions on function (?Fibonacci@@YAXHAEAH@Z)]
pass 28/211 = (0.7345) (0.0000) [Tail Call Elimination on function (?Fibonacci@@YAXHAEAH@Z)]
pass 29/211 = (0.7345) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]
pass 30/211 = (0.7345) (0.0000) [Reassociate expressions on function (?Fibonacci@@YAXHAEAH@Z)]
pass 31/211 = (0.8673) (0.1327) [Rotate Loops on loop]
pass 32/211 = (0.5575) (-0.3097) [Loop Invariant Code Motion on loop]
pass 33/211 = (0.5575) (0.0000) [Unswitch loops on loop]
pass 34/211 = (0.5575) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]
pass 35/211 = (0.5575) (0.0000) [Combine redundant instructions on function (?Fibonacci@@YAXHAEAH@Z)]
pass 36/211 = (0.5575) (0.0000) [Induction Variable Simplification on loop]
pass 37/211 = (0.5575) (0.0000) [Recognize loop idioms on loop]
<output-snipped>
|
