criterion performance measurements
overview
want to understand this report?
100% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.578796313730418e-3 | 3.5882921596052464e-3 | 3.600797320003547e-3 |
| Standard deviation | 2.4888376337760478e-5 | 3.3750639128894564e-5 | 4.5689387858710815e-5 |
Outlying measurements have slight (2.0823902218198177e-2%) effect on estimated standard deviation.
100% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.936649220093976e-3 | 6.948517053711997e-3 | 6.979765765954166e-3 |
| Standard deviation | 2.7893242204098173e-5 | 5.346436517428512e-5 | 1.0336987768218694e-4 |
Outlying measurements have slight (2.7006172839506043e-2%) effect on estimated standard deviation.
99% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.343569505832393e-2 | 8.371423616028188e-2 | 8.39089408794503e-2 |
| Standard deviation | 2.856800444935138e-4 | 3.988485891246123e-4 | 5.736873807493342e-4 |
Outlying measurements have slight (9.000000000000001e-2%) effect on estimated standard deviation.
99% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.765154012820557e-2 | 2.7750066598675547e-2 | 2.7867921249028982e-2 |
| Standard deviation | 1.7002728024301852e-4 | 2.2663332297138904e-4 | 2.9298943521698583e-4 |
Outlying measurements have slight (5.2469135802469015e-2%) effect on estimated standard deviation.
95% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1203173128035737 | 0.12102643417678066 | 0.1215652597348114 |
| Standard deviation | 6.01831596771325e-4 | 9.235747807972163e-4 | 1.2584972751700802e-3 |
Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.
95% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.1968154355944325e-2 | 5.2113814549520424e-2 | 5.2241918026806666e-2 |
| Standard deviation | 1.6386304651816553e-4 | 2.648378656807976e-4 | 3.940087935203919e-4 |
Outlying measurements have slight (7.100591715976332e-2%) effect on estimated standard deviation.
90% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.15076829635714667 | 0.15255474941733968 | 0.15907404579997025 |
| Standard deviation | 7.310920346158671e-4 | 4.277929982389772e-3 | 6.317527117180351e-3 |
Outlying measurements have moderate (0.12244897959183672%) effect on estimated standard deviation.
90% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.562226865388413e-2 | 7.607612640360557e-2 | 7.71590986910994e-2 |
| Standard deviation | 5.490568784776749e-4 | 1.172806101722295e-3 | 1.8863588801276048e-3 |
Outlying measurements have slight (9.0e-2%) effect on estimated standard deviation.
80% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.18978674334445664 | 0.19067553462219722 | 0.19237116583322708 |
| Standard deviation | 6.386180981302681e-4 | 1.6275515936698332e-3 | 2.3539415389896823e-3 |
Outlying measurements have moderate (0.13888888888888876%) effect on estimated standard deviation.
80% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.11712004103569841 | 0.11789203688584551 | 0.11862015717172408 |
| Standard deviation | 8.457168507592079e-4 | 1.1990078774353837e-3 | 1.7777919688301178e-3 |
Outlying measurements have moderate (0.10937500000000001%) effect on estimated standard deviation.
75% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2036814813888649 | 0.20489903642222204 | 0.2071866742999873 |
| Standard deviation | 8.501255544714542e-4 | 2.2213842585654e-3 | 3.157790069170674e-3 |
Outlying measurements have moderate (0.13888888888888867%) effect on estimated standard deviation.
75% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.13949622017858126 | 0.14073332242347816 | 0.14292512930613027 |
| Standard deviation | 1.0147484820005796e-3 | 2.338070139361078e-3 | 3.4018582117658477e-3 |
Outlying measurements have moderate (0.12244897959183675%) effect on estimated standard deviation.
70% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.21740093848336842 | 0.2181187628722638 | 0.21862362828895243 |
| Standard deviation | 4.182077100468951e-4 | 8.17236632045901e-4 | 1.115354200974326e-3 |
Outlying measurements have moderate (0.13888888888888887%) effect on estimated standard deviation.
70% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.15851437954280692 | 0.15910889049929297 | 0.15948543918569189 |
| Standard deviation | 5.141944912344865e-4 | 7.677365306894377e-4 | 1.0308910650048758e-3 |
Outlying measurements have moderate (0.12244897959183672%) effect on estimated standard deviation.
60% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.24388038983336174 | 0.24443824593333377 | 0.24531268983331303 |
| Standard deviation | 3.5474259824992346e-4 | 9.140385931061461e-4 | 1.298293068724513e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
60% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.19720013669999237 | 0.19788125336387705 | 0.1984841904166501 |
| Standard deviation | 5.157671633858416e-4 | 8.73430719770769e-4 | 1.1637748185708385e-3 |
Outlying measurements have moderate (0.13888888888888887%) effect on estimated standard deviation.
50% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2736780636667845 | 0.2742921206166875 | 0.27482369508333704 |
| Standard deviation | 5.65826667183869e-4 | 7.274961493713877e-4 | 8.909547238525465e-4 |
Outlying measurements have moderate (0.16000000000000003%) effect on estimated standard deviation.
50% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.23771551887995884 | 0.23982025189664302 | 0.24061863842664327 |
| Standard deviation | 1.90604294305493e-4 | 1.575148787327973e-3 | 2.0364014311948053e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
40% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2991912794001109 | 0.3011444086267132 | 0.3023813604933427 |
| Standard deviation | 1.0402526224122634e-3 | 2.0271107209897223e-3 | 3.08653300252846e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
40% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.27862661400004074 | 0.2802829206666653 | 0.2816171475333552 |
| Standard deviation | 1.1116149288953095e-3 | 1.9356247305050579e-3 | 2.8833961251934193e-3 |
Outlying measurements have moderate (0.16000000000000003%) effect on estimated standard deviation.
30% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3263137681332713 | 0.32667928837327354 | 0.32766835709990116 |
| Standard deviation | 4.412462322186592e-5 | 7.527293134956019e-4 | 9.663206174852151e-4 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
30% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3270253651332041 | 0.32814065743327775 | 0.32925827110002504 |
| Standard deviation | 9.02252200897758e-4 | 1.4833463481752286e-3 | 2.2050885681296644e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
25% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3367010797500143 | 0.3392140465416749 | 0.34067461720835956 |
| Standard deviation | 1.1872094801822827e-3 | 2.4695257024373307e-3 | 3.373043470323439e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
25% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3463549901873364 | 0.349372705812461 | 0.35094861156251034 |
| Standard deviation | 2.681917499482156e-4 | 2.942748119009894e-3 | 3.6393984360339046e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
20% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3490763352501176 | 0.35177495387507446 | 0.3544005966250836 |
| Standard deviation | 2.68104954763913e-3 | 3.102531828036489e-3 | 3.1851839886842957e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
20% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.369023374187492 | 0.37357693106250167 | 0.37652162537500544 |
| Standard deviation | 2.7251910411811033e-3 | 4.815659683649929e-3 | 6.800480216020847e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
10% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.373606083833162 | 0.3758666555207621 | 0.3772064348333212 |
| Standard deviation | 9.676088846522425e-4 | 2.2418965482029238e-3 | 3.0940877870428234e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
10% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4139026017502374 | 0.4158733812501036 | 0.4178441607499698 |
| Standard deviation | 1.4883506442334826e-3 | 2.5518736749496978e-3 | 3.417199988571679e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
5% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3846389042500391 | 0.38580239766669894 | 0.3870994472500797 |
| Standard deviation | 6.062254805722776e-4 | 1.5314243711557756e-3 | 2.080747468663435e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
5% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.43211802587489956 | 0.4356209523124335 | 0.43764210499995215 |
| Standard deviation | 1.1072200563098803e-3 | 3.4620310898267207e-3 | 4.667652192848979e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
1% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3972150631249747 | 0.39870397083329107 | 0.4000005536249205 |
| Standard deviation | 1.2781064626758458e-3 | 1.5685750519355448e-3 | 1.7701951851264424e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
1% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4495732185000918 | 0.45311930187503435 | 0.45525610637500336 |
| Standard deviation | 1.5222810681041993e-3 | 3.559098168614373e-3 | 4.93473861305083e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
0% dups/sort . nub 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3956989184999884 | 0.4013889370416924 | 0.40456421766670775 |
| Standard deviation | 1.5606780440824012e-3 | 5.473187780673031e-3 | 7.3329964944209205e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
0% dups/nub . sort 10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.45325647375011613 | 0.455058892604247 | 0.4571826752500101 |
| Standard deviation | 9.152536250098819e-4 | 2.4456137463835666e-3 | 3.1056641889471706e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.