criterion performance measurements
overview
want to understand this report?
| Words l=6-12 | |
| Words l=6-18 | |
| Words l=12-18 |
Words l=6-12/levenshteinMax (1)
500 510 520 530 540 550 560
mean |
200 300 400 500 600 700 800 100×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 512 ns | 514 ns | 517 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 509 ns | 512 ns | 515 ns |
| Standard deviation | 9.06 ns | 11.1 ns | 13.7 ns |
Outlying measurements have moderate (28.9%) effect on estimated standard deviation.
Words l=6-12/levenshteinMaxO (1)
450 460 470 480 490 500 510
mean |
400 600 800 200×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 457 ns | 459 ns | 462 ns |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 461 ns | 464 ns | 468 ns |
| Standard deviation | 9.01 ns | 12.0 ns | 17.3 ns |
Outlying measurements have moderate (37.1%) effect on estimated standard deviation.
Words l=6-12/levenshteinMax (2)
750 775 800 825 850 875
mean |
200 300 400 500 600 100×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 772 ns | 778 ns | 781 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 773 ns | 777 ns | 782 ns |
| Standard deviation | 11.5 ns | 18.5 ns | 27.9 ns |
Outlying measurements have moderate (32.4%) effect on estimated standard deviation.
Words l=6-12/levenshteinMaxO (2)
720 730 740 750 760
mean |
200 300 400 500 600 100×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 721 ns | 723 ns | 726 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 728 ns | 730 ns | 733 ns |
| Standard deviation | 10.4 ns | 12.0 ns | 13.5 ns |
Outlying measurements have moderate (18.5%) effect on estimated standard deviation.
Words l=6-12/levenshteinMax (3)
1.26 1.28 1.3 1.32 1.34
mean |
100 150 200 250 300 350 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.30 μs | 1.31 μs | 1.32 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.30 μs | 1.30 μs | 1.31 μs |
| Standard deviation | 23.9 ns | 26.8 ns | 29.6 ns |
Outlying measurements have moderate (25.2%) effect on estimated standard deviation.
Words l=6-12/levenshteinMaxO (3)
1.12 1.14 1.16 1.18 1.20 1.22 1.24
mean |
100 150 200 250 300 350 400 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.13 μs | 1.14 μs | 1.14 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.14 μs | 1.14 μs | 1.15 μs |
| Standard deviation | 17.7 ns | 23.2 ns | 31.9 ns |
Outlying measurements have moderate (25.3%) effect on estimated standard deviation.
Words l=6-12/levenshteinMax (4)
1.28 1.29 1.3 1.31 1.32 1.33 1.34 1.35 1.36
mean |
100 150 200 250 300 350 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.30 μs | 1.31 μs | 1.31 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.30 μs | 1.31 μs | 1.31 μs |
| Standard deviation | 14.8 ns | 17.4 ns | 20.3 ns |
Outlying measurements have moderate (12.5%) effect on estimated standard deviation.
Words l=6-12/levenshteinMaxO (4)
1.14 1.16 1.18 1.20 1.22
mean |
100 150 200 250 300 350 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.16 μs | 1.17 μs | 1.18 μs |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 1.16 μs | 1.16 μs | 1.17 μs |
| Standard deviation | 15.4 ns | 20.3 ns | 24.6 ns |
Outlying measurements have moderate (19.7%) effect on estimated standard deviation.
Words l=6-12/levenshtein
2.85 2.90 2.95 3.00 3.05 3.10
mean |
50 75 100 125 150 25×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.88 μs | 2.90 μs | 2.92 μs |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 2.89 μs | 2.90 μs | 2.92 μs |
| Standard deviation | 36.6 ns | 59.2 ns | 79.1 ns |
Outlying measurements have moderate (23.4%) effect on estimated standard deviation.
Words l=6-18/levenshteinMax (1)
500 510 520 530 540
mean |
200 300 400 500 600 700 800 100×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 509 ns | 513 ns | 516 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 509 ns | 511 ns | 514 ns |
| Standard deviation | 6.85 ns | 8.15 ns | 10.3 ns |
Outlying measurements have moderate (17.9%) effect on estimated standard deviation.
Words l=6-18/levenshteinMaxO (1)
475 500 525 550 575
mean |
400 600 800 200×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 468 ns | 471 ns | 475 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 473 ns | 477 ns | 486 ns |
| Standard deviation | 11.1 ns | 23.1 ns | 39.4 ns |
Outlying measurements have severe (67.6%) effect on estimated standard deviation.
Words l=6-18/levenshteinMax (2)
770 780 790 800 810
mean |
200 300 400 500 100×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 784 ns | 788 ns | 793 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 783 ns | 786 ns | 789 ns |
| Standard deviation | 9.71 ns | 11.4 ns | 13.4 ns |
Outlying measurements have moderate (14.9%) effect on estimated standard deviation.
Words l=6-18/levenshteinMaxO (2)
710 720 730 740 750 760 770 780 790
mean |
200 300 400 500 600 100×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 728 ns | 736 ns | 742 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 727 ns | 730 ns | 735 ns |
| Standard deviation | 8.91 ns | 14.1 ns | 19.7 ns |
Outlying measurements have moderate (23.9%) effect on estimated standard deviation.
Words l=6-18/levenshteinMax (3)
1.03 1.04 1.05 1.06 1.07 1.08
mean |
100 150 200 250 300 350 400 450 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.04 μs | 1.04 μs | 1.05 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.04 μs | 1.05 μs | 1.05 μs |
| Standard deviation | 12.6 ns | 14.2 ns | 17.8 ns |
Outlying measurements have moderate (13.1%) effect on estimated standard deviation.
Words l=6-18/levenshteinMaxO (3)
920 930 940 950 960 970 980 990 1 μs
mean |
200 300 400 500 100×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 932 ns | 934 ns | 938 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 934 ns | 937 ns | 941 ns |
| Standard deviation | 7.08 ns | 11.1 ns | 18.1 ns |
Outlying measurements have moderate (10.4%) effect on estimated standard deviation.
Words l=6-18/levenshteinMax (4)
1.26 1.28 1.3 1.32 1.34 1.36
mean |
100 150 200 250 300 350 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.30 μs | 1.31 μs | 1.32 μs |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 1.32 μs | 1.32 μs | 1.33 μs |
| Standard deviation | 20.0 ns | 22.8 ns | 28.2 ns |
Outlying measurements have moderate (19.0%) effect on estimated standard deviation.
Words l=6-18/levenshteinMaxO (4)
1.42 1.43 1.44 1.45 1.46
mean |
100 150 200 250 300 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.44 μs | 1.44 μs | 1.44 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.44 μs | 1.44 μs | 1.44 μs |
| Standard deviation | 8.11 ns | 9.66 ns | 11.5 ns |
Outlying measurements have no (0.5%) effect on estimated standard deviation.
Words l=6-18/levenshtein
4.35 4.40 4.45 4.5 4.55 4.6
mean |
40 60 80 100 20×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.41 μs | 4.42 μs | 4.45 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 4.40 μs | 4.41 μs | 4.42 μs |
| Standard deviation | 35.7 ns | 51.0 ns | 67.2 ns |
Outlying measurements have slight (8.7%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (1)
920 940 960 980 1 μs 1.02
mean |
200 300 400 500 100×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 940 ns | 943 ns | 949 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 943 ns | 946 ns | 953 ns |
| Standard deviation | 13.1 ns | 17.2 ns | 25.0 ns |
Outlying measurements have moderate (21.4%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (1)
840 850 860 870 880
mean |
200 300 400 500 100×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 844 ns | 846 ns | 849 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 848 ns | 851 ns | 854 ns |
| Standard deviation | 8.35 ns | 9.93 ns | 12.6 ns |
Outlying measurements have moderate (10.3%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (2)
1.38 1.40 1.42 1.44 1.46
mean |
100 150 200 250 300 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.39 μs | 1.40 μs | 1.41 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.39 μs | 1.39 μs | 1.40 μs |
| Standard deviation | 11.3 ns | 15.2 ns | 20.8 ns |
Outlying measurements have slight (8.6%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (2)
1.32 1.34 1.36 1.38 1.40 1.42
mean |
100 150 200 250 300 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.35 μs | 1.36 μs | 1.37 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.35 μs | 1.35 μs | 1.36 μs |
| Standard deviation | 10.8 ns | 14.3 ns | 19.5 ns |
Outlying measurements have slight (8.2%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (3)
1.90 1.92 1.94 1.96 1.98
mean |
100 150 200 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.91 μs | 1.92 μs | 1.92 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 1.91 μs | 1.91 μs | 1.92 μs |
| Standard deviation | 11.7 ns | 16.0 ns | 24.9 ns |
Outlying measurements have no (0.5%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (3)
2.22 2.24 2.26 2.28 2.30 2.32
mean |
100 150 200 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.24 μs | 2.26 μs | 2.27 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 2.25 μs | 2.25 μs | 2.26 μs |
| Standard deviation | 18.1 ns | 23.7 ns | 31.3 ns |
Outlying measurements have slight (7.7%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (4)
2.40 2.50 2.60 2.7 2.80 2.90 3.00
mean |
50 75 100 125 150 175 25×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.44 μs | 2.46 μs | 2.48 μs |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 2.44 μs | 2.46 μs | 2.49 μs |
| Standard deviation | 50.8 ns | 93.9 ns | 174 ns |
Outlying measurements have severe (52.6%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (4)
2.15 2.2 2.25 2.30 2.35 2.40 2.45 2.5
mean |
100 150 200 50×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.24 μs | 2.25 μs | 2.27 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 2.21 μs | 2.23 μs | 2.25 μs |
| Standard deviation | 32.9 ns | 58.4 ns | 82.6 ns |
Outlying measurements have moderate (34.5%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (5)
2.85 2.90 2.95 3.00 3.05 3.10 3.15 3.2
mean |
50 75 100 125 150 25×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.90 μs | 2.92 μs | 2.94 μs |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 2.92 μs | 2.94 μs | 2.95 μs |
| Standard deviation | 44.5 ns | 62.9 ns | 93.6 ns |
Outlying measurements have moderate (25.4%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (5)
3 2.65 2.7 2.75 2.80 2.85 2.90 2.95 3.05
mean |
50 75 100 125 150 25×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.72 μs | 2.75 μs | 2.77 μs |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 2.74 μs | 2.76 μs | 2.80 μs |
| Standard deviation | 68.5 ns | 85.9 ns | 113 ns |
Outlying measurements have moderate (41.9%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (6)
4 3.80 3.90 4.10 4.2 4.30 4.4
mean |
40 60 80 100 20×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.89 μs | 3.91 μs | 3.93 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 3.92 μs | 3.94 μs | 3.96 μs |
| Standard deviation | 67.9 ns | 98.3 ns | 140 ns |
Outlying measurements have moderate (31.1%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (6)
4.2 4.30 4.4 4.50 4.60 4.70
mean |
40 60 80 100 20×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.31 μs | 4.35 μs | 4.40 μs |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 4.31 μs | 4.33 μs | 4.36 μs |
| Standard deviation | 61.5 ns | 85.7 ns | 121 ns |
Outlying measurements have moderate (22.0%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (7)
4 3.80 4.2 4.4 4.6 4.80
mean |
40 60 80 100 20×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.94 μs | 3.96 μs | 3.98 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 3.94 μs | 3.96 μs | 4.02 μs |
| Standard deviation | 66.0 ns | 146 ns | 250 ns |
Outlying measurements have moderate (49.7%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (7)
5 4.6 4.7 4.8 4.9 5.1
mean |
40 60 80 100 20×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.67 μs | 4.70 μs | 4.73 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 4.69 μs | 4.71 μs | 4.75 μs |
| Standard deviation | 57.5 ns | 83.0 ns | 124 ns |
Outlying measurements have moderate (18.1%) effect on estimated standard deviation.
Words l=12-18/levenshteinMax (8)
5.7 5.80 5.9 6.00 6.10 6.20 6.30 6.4
mean |
20 30 40 50 60 70 10×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 5.78 μs | 5.80 μs | 5.85 μs |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 5.86 μs | 5.89 μs | 5.94 μs |
| Standard deviation | 129 ns | 159 ns | 203 ns |
Outlying measurements have moderate (33.5%) effect on estimated standard deviation.
Words l=12-18/levenshteinMaxO (8)
4.65 4.7 4.75 4.80 4.85 4.9
mean |
40 60 80 100 20×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.73 μs | 4.76 μs | 4.79 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 4.71 μs | 4.72 μs | 4.74 μs |
| Standard deviation | 44.5 ns | 52.7 ns | 67.8 ns |
Outlying measurements have slight (8.3%) effect on estimated standard deviation.
Words l=12-18/levenshtein
9 8.6 8.8 9.20 9.4
mean |
20 30 40 50 10×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 8.60 μs | 8.63 μs | 8.67 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 8.64 μs | 8.66 μs | 8.72 μs |
| Standard deviation | 66.1 ns | 149 ns | 224 ns |
Outlying measurements have moderate (16.3%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.