criterion performance measurements
overview
want to understand this report?
| 100% dups | |
| 99% dups | |
| 95% dups | |
| 90% dups | |
| 80% dups | |
| 75% dups | |
| 70% dups | |
| 60% dups | |
| 50% dups | |
| 40% dups | |
| 30% dups | |
| 25% dups | |
| 20% dups | |
| 10% dups | |
| 5% dups | |
| 1% dups | |
| 0% dups |
100% dups/sort . nub 10000
6.3 6.4 6.5 6.60 6.7 6.80 6.9
mean |
10 15 20 25 30 35 40 45 5 iters 100 150 200 250 300 350 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 6.39 ms | 6.44 ms | 6.48 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 6.50 ms | 6.53 ms | 6.58 ms |
| Standard deviation | 88.3 μs | 115 μs | 157 μs |
Outlying measurements have slight (2.6%) effect on estimated standard deviation.
100% dups/nub . sort 10000
6.52 6.54 6.56 6.58 6.6
mean |
10 15 20 25 30 35 40 45 5 iters 100 150 200 250 300 350 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 6.55 ms | 6.57 ms | 6.58 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 6.55 ms | 6.56 ms | 6.57 ms |
| Standard deviation | 15.3 μs | 18.8 μs | 24.7 μs |
Outlying measurements have slight (2.6%) effect on estimated standard deviation.
99% dups/sort . nub 10000
110 115 120 125 130
mean |
2 3 4 5 6 7 8 1 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 114 ms | 119 ms | 122 ms |
| R² goodness-of-fit | 0.994 | 0.998 | 1.000 |
| Mean execution time | 116 ms | 119 ms | 124 ms |
| Standard deviation | 2.45 ms | 5.83 ms | 9.35 ms |
Outlying measurements have moderate (11.2%) effect on estimated standard deviation.
99% dups/nub . sort 10000
25.1 25.2 25.3 25.4 25.5 25.6 25.7
mean |
5 8 10 13 15 18 2.5 iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 25.5 ms | 25.6 ms | 25.8 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 25.3 ms | 25.3 ms | 25.4 ms |
| Standard deviation | 173 μs | 211 μs | 255 μs |
Outlying measurements have slight (5.0%) effect on estimated standard deviation.
95% dups/sort . nub 10000
145 150 155 160 165 170 175 180 185
mean |
2 3 4 5 6 7 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 145 ms | 151 ms | 166 ms |
| R² goodness-of-fit | 0.986 | 0.995 | 0.999 |
| Mean execution time | 151 ms | 157 ms | 170 ms |
| Standard deviation | 3.75 ms | 12.3 ms | 19.0 ms |
Outlying measurements have moderate (13.5%) effect on estimated standard deviation.
95% dups/nub . sort 10000
48 47.2 47.4 47.6 47.8 48.2
mean |
4 6 8 10 12 2 iters 200 300 400 500 600 700 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 47.8 ms | 48.1 ms | 48.5 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 47.5 ms | 47.6 ms | 47.8 ms |
| Standard deviation | 202 μs | 291 μs | 417 μs |
Outlying measurements have slight (7.1%) effect on estimated standard deviation.
90% dups/sort . nub 10000
170 180 190 200 210 220 230
mean |
2 3 4 5 6 7 1 iters 500 750 0 s 250 ms 1 s 1.25
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 157 ms | 161 ms | 166 ms |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 175 ms | 184 ms | 206 ms |
| Standard deviation | 8.69 ms | 20.2 ms | 29.7 ms |
Outlying measurements have moderate (27.1%) effect on estimated standard deviation.
90% dups/nub . sort 10000
69 70 68.5 69.5
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 68.5 ms | 69.2 ms | 69.8 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 68.9 ms | 69.1 ms | 69.4 ms |
| Standard deviation | 329 μs | 499 μs | 765 μs |
Outlying measurements have slight (8.3%) effect on estimated standard deviation.
80% dups/sort . nub 10000
210 220 230 240 250 260 270 280 290
mean |
2 3 4 5 6 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 192 ms | 195 ms | 199 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 216 ms | 230 ms | 261 ms |
| Standard deviation | 8.85 ms | 28.6 ms | 40.7 ms |
Outlying measurements have moderate (31.5%) effect on estimated standard deviation.
80% dups/nub . sort 10000
106 107 107 108 108 109 109
mean |
2 3 4 5 6 7 8 9 1 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 105 ms | 107 ms | 108 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 107 ms | 107 ms | 108 ms |
| Standard deviation | 617 μs | 1.01 ms | 1.52 ms |
Outlying measurements have slight (9.9%) effect on estimated standard deviation.
75% dups/sort . nub 10000
250 260 270 280 290
mean |
2 3 4 5 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 221 ms | 237 ms | 250 ms |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 253 ms | 264 ms | 277 ms |
| Standard deviation | 7.21 ms | 15.5 ms | 22.3 ms |
Outlying measurements have moderate (16.4%) effect on estimated standard deviation.
75% dups/nub . sort 10000
124 124 125 125
mean |
2 3 4 5 6 7 8 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 124 ms | 124 ms | 126 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 124 ms | 124 ms | 125 ms |
| Standard deviation | 191 μs | 566 μs | 888 μs |
Outlying measurements have moderate (10.9%) effect on estimated standard deviation.
70% dups/sort . nub 10000
240 250 260 270 280 290 300
mean |
2 3 4 5 1 iters 500 750 0 s 250 ms 1 s 1.25
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 217 ms | 225 ms | 229 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 244 ms | 258 ms | 283 ms |
| Standard deviation | 6.75 ms | 23.3 ms | 31.8 ms |
Outlying measurements have moderate (18.5%) effect on estimated standard deviation.
70% dups/nub . sort 10000
141 142 143 144 145 146 147 148 149
mean |
2 3 4 5 6 7 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 144 ms | 150 ms | 154 ms |
| R² goodness-of-fit | 0.997 | 0.999 | 1.000 |
| Mean execution time | 143 ms | 145 ms | 147 ms |
| Standard deviation | 2.51 ms | 3.15 ms | 3.81 ms |
Outlying measurements have moderate (12.2%) effect on estimated standard deviation.
60% dups/sort . nub 10000
280 290 300 310 320 330 340
mean |
2 3 4 5 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 258 ms | 267 ms | 276 ms |
| R² goodness-of-fit | 0.997 | 0.999 | 1.000 |
| Mean execution time | 287 ms | 299 ms | 320 ms |
| Standard deviation | 6.15 ms | 21.8 ms | 30.9 ms |
Outlying measurements have moderate (17.6%) effect on estimated standard deviation.
60% dups/nub . sort 10000
185 190 195 200
mean |
2 3 4 5 6 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 202 ms | 205 ms | 207 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 189 ms | 197 ms | 201 ms |
| Standard deviation | 1.57 ms | 7.27 ms | 10.5 ms |
Outlying measurements have moderate (13.9%) effect on estimated standard deviation.
50% dups/sort . nub 10000
300 310 320 330 340
mean |
2 3 4 5 1 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 277 ms | 286 ms | 293 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 297 ms | 307 ms | 325 ms |
| Standard deviation | 965 μs | 18.3 ms | 23.6 ms |
Outlying measurements have moderate (16.5%) effect on estimated standard deviation.
50% dups/nub . sort 10000
220 223 225 228 230 233 235
mean |
2 3 4 5 6 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 209 ms | 227 ms | 238 ms |
| R² goodness-of-fit | 0.989 | 0.997 | 1.000 |
| Mean execution time | 222 ms | 226 ms | 230 ms |
| Standard deviation | 4.49 ms | 5.77 ms | 7.19 ms |
Outlying measurements have moderate (13.9%) effect on estimated standard deviation.
40% dups/sort . nub 10000
335 340 345 350 355 360
mean |
1 2 2 3 3 4 4 0.5 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 328 ms | 328 ms | 357 ms |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 335 ms | 343 ms | 355 ms |
| Standard deviation | 1.25 ms | 11.6 ms | 14.8 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
40% dups/nub . sort 10000
257 257 258 258 259 259
mean |
2 3 4 5 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 254 ms | 257 ms | 260 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 257 ms | 258 ms | 259 ms |
| Standard deviation | 824 μs | 1.11 ms | 1.38 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
30% dups/sort . nub 10000
376 378 380 382 384
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 349 ms | 376 ms | 394 ms |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 377 ms | 380 ms | 383 ms |
| Standard deviation | 2.07 ms | 3.68 ms | 5.20 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
30% dups/nub . sort 10000
292 294 296 298 300 302 304
mean |
2 3 4 5 1 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 294 ms | 306 ms | 317 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 293 ms | 297 ms | 300 ms |
| Standard deviation | 3.58 ms | 4.80 ms | 6.13 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
25% dups/sort . nub 10000
360 365 370 375 380 385 390 395
mean |
1 2 2 3 3 4 4 0.5 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 341 ms | 354 ms | 371 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 365 ms | 375 ms | 391 ms |
| Standard deviation | 995 μs | 15.0 ms | 18.9 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
25% dups/nub . sort 10000
314 316 318 320 322
mean |
2 3 4 5 1 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 316 ms | 322 ms | 331 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 315 ms | 316 ms | 319 ms |
| Standard deviation | 1.02 ms | 3.03 ms | 4.40 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
20% dups/sort . nub 10000
370 375 380 385 390 395 400 405 410
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 345 ms | 359 ms | 373 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 374 ms | 384 ms | 401 ms |
| Standard deviation | 4.20 ms | 16.1 ms | 21.6 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
20% dups/nub . sort 10000
330 335 340 345 350 355 360
mean |
1 2 2 3 3 4 4 0.5 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 298 ms | 324 ms | 340 ms |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 335 ms | 347 ms | 355 ms |
| Standard deviation | 5.91 ms | 11.4 ms | 15.9 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
10% dups/sort . nub 10000
405 410 415 420 425
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 384 ms | 400 ms | 411 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 409 ms | 416 ms | 423 ms |
| Standard deviation | 3.67 ms | 8.50 ms | 11.6 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
10% dups/nub . sort 10000
370 371 372 373 374
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 368 ms | 375 ms | 388 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 370 ms | 371 ms | 373 ms |
| Standard deviation | 293 μs | 2.15 ms | 2.71 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
5% dups/sort . nub 10000
410 415 420 425 430 435 440
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 385 ms | 402 ms | 426 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 411 ms | 420 ms | 435 ms |
| Standard deviation | 3.74 ms | 13.9 ms | 18.1 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
5% dups/nub . sort 10000
388 388 388 388 388
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 387 ms | 387 ms | 388 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 388 ms | 388 ms | 388 ms |
| Standard deviation | 83.9 μs | 401 μs | 531 μs |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
1% dups/sort . nub 10000
425 430 435 440 445 450
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 398 ms | 417 ms | 442 ms |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 428 ms | 439 ms | 447 ms |
| Standard deviation | 4.55 ms | 10.7 ms | 13.6 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
1% dups/nub . sort 10000
406 406 407 407 408
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 404 ms | 406 ms | 410 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 406 ms | 406 ms | 407 ms |
| Standard deviation | 299 μs | 979 μs | 1.31 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
0% dups/sort . nub 10000
425 430 435 440 445 450 455 460 465 470
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 399 ms | 413 ms | 434 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 429 ms | 445 ms | 459 ms |
| Standard deviation | 8.36 ms | 17.8 ms | 22.6 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
0% dups/nub . sort 10000
410 415 420 425
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 378 ms | 410 ms | 444 ms |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 410 ms | 417 ms | 424 ms |
| Standard deviation | 5.36 ms | 8.49 ms | 10.0 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.