criterion performance measurements
overview
want to understand this report?
| sort . nub | |
| nub . sort |
sort . nub/100(1->536870911)
54 55 56 57 54.5 55.5 56.5
mean |
500 750 1000 1250 1500 250 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 54.4 μs | 54.6 μs | 54.8 μs |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 54.5 μs | 54.7 μs | 54.9 μs |
| Standard deviation | 377 ns | 640 ns | 935 ns |
Outlying measurements have no (0.9%) effect on estimated standard deviation.
sort . nub/200(1->536870911)
197 198 199 200 201 202
mean |
100 150 200 250 300 350 400 450 50 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 199 μs | 199 μs | 200 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 199 μs | 199 μs | 199 μs |
| Standard deviation | 912 ns | 1.10 μs | 1.45 μs |
Outlying measurements have slight (1.2%) effect on estimated standard deviation.
sort . nub/400(1->536870911)
738 740 743 745 748 750
mean |
40 60 80 100 120 20 iters 40 60 80 100 120 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 740 μs | 742 μs | 745 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 741 μs | 743 μs | 744 μs |
| Standard deviation | 2.50 μs | 3.32 μs | 4.34 μs |
Outlying measurements have slight (1.7%) effect on estimated standard deviation.
sort . nub/600(1->536870911)
1.64 1.65 1.65 1.66 1.66 1.67 1.67 1.68
mean |
20 30 40 50 60 10 iters 40 60 80 100 120 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.65 ms | 1.66 ms | 1.66 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.65 ms | 1.65 ms | 1.66 ms |
| Standard deviation | 6.56 μs | 8.31 μs | 12.2 μs |
Outlying measurements have slight (2.2%) effect on estimated standard deviation.
sort . nub/800(1->536870911)
2.83 2.84 2.85 2.86 2.87 2.88 2.89 2.9 2.91
mean |
10 15 20 25 30 35 40 5 iters 50 75 100 125 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.84 ms | 2.85 ms | 2.87 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 2.85 ms | 2.86 ms | 2.86 ms |
| Standard deviation | 11.8 μs | 17.7 μs | 25.8 μs |
Outlying measurements have slight (2.8%) effect on estimated standard deviation.
sort . nub/1000(1->536870911)
4.35 4.40 4.45 4.5
mean |
10 15 20 25 30 5 iters 50 75 100 125 150 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.40 ms | 4.43 ms | 4.48 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 4.38 ms | 4.40 ms | 4.42 ms |
| Standard deviation | 35.2 μs | 48.5 μs | 60.0 μs |
Outlying measurements have slight (3.3%) effect on estimated standard deviation.
sort . nub/1250(1->536870911)
6.60 6.65 6.70 6.75 6.80 6.85
mean |
10 15 20 5 iters 100 150 200 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 6.66 ms | 6.69 ms | 6.71 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 6.69 ms | 6.70 ms | 6.73 ms |
| Standard deviation | 37.9 μs | 52.4 μs | 75.7 μs |
Outlying measurements have slight (4.2%) effect on estimated standard deviation.
sort . nub/1500(1->536870911)
9.70 9.73 9.75 9.78 9.8 9.83
mean |
5 8 10 13 15 18 2.5 iters 100 150 200 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 9.70 ms | 9.74 ms | 9.78 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 9.75 ms | 9.76 ms | 9.78 ms |
| Standard deviation | 29.7 μs | 35.7 μs | 46.5 μs |
Outlying measurements have slight (5.0%) effect on estimated standard deviation.
sort . nub/2000(1->536870911)
17.1 17.2 17.3 17.4 17.5 17.6 17.7
mean |
4 6 8 10 12 14 2 iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 17.1 ms | 17.3 ms | 17.4 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 17.3 ms | 17.3 ms | 17.4 ms |
| Standard deviation | 82.4 μs | 144 μs | 218 μs |
Outlying measurements have slight (6.6%) effect on estimated standard deviation.
sort . nub/3000(1->536870911)
39.0 39.2 39.4 39.6 39.8 40.0 40.2
mean |
2 3 4 5 6 7 8 9 1 iters 100 150 200 250 300 350 400 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 39.1 ms | 39.4 ms | 39.7 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 39.3 ms | 39.5 ms | 39.8 ms |
| Standard deviation | 197 μs | 319 μs | 491 μs |
Outlying measurements have slight (9.9%) effect on estimated standard deviation.
sort . nub/4000(1->536870911)
69 70 71 68.5 69.5 70.5 71.5
mean |
2 3 4 5 6 7 1 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 70.4 ms | 71.0 ms | 73.2 ms |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 69.2 ms | 70.1 ms | 70.8 ms |
| Standard deviation | 813 μs | 1.09 ms | 1.35 ms |
Outlying measurements have moderate (12.2%) effect on estimated standard deviation.
sort . nub/5000(1->536870911)
108 108 109 109 110 110 111
mean |
2 3 4 5 1 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 103 ms | 107 ms | 111 ms |
| R² goodness-of-fit | 0.998 | 1.000 | 1.000 |
| Mean execution time | 108 ms | 109 ms | 110 ms |
| Standard deviation | 1.09 ms | 1.27 ms | 1.50 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
nub . sort/100(1->536870911)
57 58 59 60 57.5 58.5 59.5 60.5
mean |
500 750 1000 1250 1500 250 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 57.8 μs | 58.0 μs | 58.2 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 57.8 μs | 58.0 μs | 58.3 μs |
| Standard deviation | 361 ns | 631 ns | 900 ns |
Outlying measurements have no (0.9%) effect on estimated standard deviation.
nub . sort/200(1->536870911)
204 206 208 210 212
mean |
100 150 200 250 300 350 400 450 50 iters 40 60 80 100 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 207 μs | 208 μs | 210 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 207 μs | 207 μs | 209 μs |
| Standard deviation | 1.78 μs | 2.33 μs | 2.87 μs |
Outlying measurements have slight (1.2%) effect on estimated standard deviation.
nub . sort/400(1->536870911)
795 800 805 810 815
mean |
40 60 80 100 120 20 iters 40 60 80 100 120 0 s 20 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 796 μs | 799 μs | 803 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 798 μs | 800 μs | 804 μs |
| Standard deviation | 4.92 μs | 6.94 μs | 8.80 μs |
Outlying measurements have slight (1.7%) effect on estimated standard deviation.
nub . sort/600(1->536870911)
1.77 1.78 1.78 1.79 1.79 1.8 1.80 1.81 1.81
mean |
20 30 40 50 60 10 iters 50 75 100 125 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.78 ms | 1.79 ms | 1.80 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.79 ms | 1.79 ms | 1.79 ms |
| Standard deviation | 5.67 μs | 8.60 μs | 12.2 μs |
Outlying measurements have slight (2.3%) effect on estimated standard deviation.
nub . sort/800(1->536870911)
3.1 3.12 3.14 3.16 3.18 3.2
mean |
10 15 20 25 30 35 40 5 iters 50 75 100 125 150 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.09 ms | 3.10 ms | 3.11 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 3.11 ms | 3.11 ms | 3.12 ms |
| Standard deviation | 11.7 μs | 20.8 μs | 33.3 μs |
Outlying measurements have slight (2.9%) effect on estimated standard deviation.
nub . sort/1000(1->536870911)
4.8 4.85 4.9 4.95
mean |
10 15 20 25 30 5 iters 50 75 100 125 150 0 s 25 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.78 ms | 4.82 ms | 4.86 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 4.81 ms | 4.83 ms | 4.85 ms |
| Standard deviation | 37.1 μs | 47.1 μs | 64.7 μs |
Outlying measurements have slight (3.4%) effect on estimated standard deviation.
nub . sort/1250(1->536870911)
7.25 7.3 7.35 7.4 7.45 7.5 7.55 7.60 7.65
mean |
10 15 20 5 iters 100 150 200 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 7.18 ms | 7.22 ms | 7.26 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 7.35 ms | 7.39 ms | 7.43 ms |
| Standard deviation | 83.9 μs | 102 μs | 145 μs |
Outlying measurements have slight (4.3%) effect on estimated standard deviation.
nub . sort/1500(1->536870911)
10.4 10.5 10.5 10.6 10.6
mean |
5 8 10 13 15 18 2.5 iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 10.4 ms | 10.5 ms | 10.6 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 10.5 ms | 10.5 ms | 10.5 ms |
| Standard deviation | 65.1 μs | 74.2 μs | 96.8 μs |
Outlying measurements have slight (5.0%) effect on estimated standard deviation.
nub . sort/2000(1->536870911)
18.0 18.1 18.2 18.3 18.4 18.5
mean |
4 6 8 10 12 14 2 iters 100 150 200 250 300 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 18.1 ms | 18.2 ms | 18.4 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 18.1 ms | 18.2 ms | 18.3 ms |
| Standard deviation | 81.5 μs | 163 μs | 209 μs |
Outlying measurements have slight (6.6%) effect on estimated standard deviation.
nub . sort/3000(1->536870911)
40 39.2 39.4 39.6 39.8 40.2 40.4 40.6
mean |
2 3 4 5 6 7 8 9 1 iters 100 150 200 250 300 350 400 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 38.9 ms | 39.5 ms | 40.0 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 39.5 ms | 39.7 ms | 40.0 ms |
| Standard deviation | 231 μs | 379 μs | 516 μs |
Outlying measurements have slight (9.9%) effect on estimated standard deviation.
nub . sort/4000(1->536870911)
69 70 68.5 68.8 69.3 69.5 69.8 70.3
mean |
2 3 4 5 6 7 1 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 68.7 ms | 69.8 ms | 71.5 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 68.8 ms | 69.1 ms | 69.7 ms |
| Standard deviation | 249 μs | 537 μs | 755 μs |
Outlying measurements have moderate (12.2%) effect on estimated standard deviation.
nub . sort/5000(1->536870911)
105 106 107 108 109 110 111 112 113
mean |
2 3 4 5 1 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 99.9 ms | 109 ms | 117 ms |
| R² goodness-of-fit | 0.995 | 0.998 | 1.000 |
| Mean execution time | 107 ms | 109 ms | 112 ms |
| Standard deviation | 1.38 ms | 2.78 ms | 3.99 ms |
Outlying measurements have moderate (16.0%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.