criterion performance measurements

overview

want to understand this report?

sort . nub/100(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.803016898317925e-6 5.8427581176719815e-6 5.96242641585806e-6
Standard deviation 6.064463735953833e-8 1.8165213657997392e-7 3.1967893919400536e-7

Outlying measurements have moderate (0.35747111256485814%) effect on estimated standard deviation.

sort . nub/200(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0616921492212371e-5 1.0637277548724143e-5 1.0671494246423912e-5
Standard deviation 5.69118184841315e-8 7.118890344143704e-8 9.313921242446776e-8

Outlying measurements have no (6.8962191358024685e-3%) effect on estimated standard deviation.

sort . nub/400(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.167285639504089e-5 2.1789497473419878e-5 2.198955664415223e-5
Standard deviation 2.7747886078607825e-7 3.65060201833674e-7 4.5850805043775924e-7

Outlying measurements have moderate (0.11739271639359747%) effect on estimated standard deviation.

sort . nub/600(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.127896995414949e-5 3.139884539397163e-5 3.1503455682373745e-5
Standard deviation 1.8942877274331762e-7 2.619049280644739e-7 3.6995798168662687e-7

Outlying measurements have no (8.12953507121768e-3%) effect on estimated standard deviation.

sort . nub/800(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.1949089224115354e-5 4.2204705494077584e-5 4.247967711040388e-5
Standard deviation 5.126467914586057e-7 6.96978880952439e-7 9.259082059833913e-7

Outlying measurements have moderate (0.10586577280381143%) effect on estimated standard deviation.

sort . nub/1000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.142714401411551e-5 5.160488992454314e-5 5.1802439613984636e-5
Standard deviation 4.5772945378397795e-7 5.508480317680219e-7 7.040972387735487e-7

Outlying measurements have no (8.848852040816457e-3%) effect on estimated standard deviation.

sort . nub/1250(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.566610572877019e-5 6.591565785481325e-5 6.627752900899735e-5
Standard deviation 5.275816330413659e-7 8.094254294384727e-7 1.1259302062540059e-6

Outlying measurements have slight (4.02597011273025e-2%) effect on estimated standard deviation.

sort . nub/1500(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.799596255779397e-5 7.841190871146876e-5 7.891515284186254e-5
Standard deviation 1.0592914300072955e-6 1.2716907894269749e-6 1.5880563548117364e-6

Outlying measurements have slight (9.165100433021929e-2%) effect on estimated standard deviation.

sort . nub/2000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0351633746412174e-4 1.0396880978678948e-4 1.0442834242361642e-4
Standard deviation 9.873549827837924e-7 1.2301613606544101e-6 1.6656762297607875e-6

Outlying measurements have slight (1.009995835068729e-2%) effect on estimated standard deviation.

sort . nub/3000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.5289291542882453e-4 1.537715740700973e-4 1.5501513842563726e-4
Standard deviation 1.9497654302636497e-6 2.3305452326334025e-6 3.022380966382574e-6

Outlying measurements have slight (7.32055735074276e-2%) effect on estimated standard deviation.

sort . nub/4000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.0281718885753616e-4 2.0363027873923276e-4 2.0513571312063186e-4
Standard deviation 2.353524256061563e-6 3.1652486317674437e-6 4.325325837623405e-6

Outlying measurements have slight (6.750073204285821e-2%) effect on estimated standard deviation.

sort . nub/5000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.5663059378257827e-4 2.5784606898289383e-4 2.6093249094415476e-4
Standard deviation 2.4334466761006336e-6 4.155917928477889e-6 6.325677279259126e-6

Outlying measurements have slight (7.155771065316555e-2%) effect on estimated standard deviation.

nub . sort/100(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8789441390388496e-5 1.8897559774152387e-5 1.9017746084731045e-5
Standard deviation 2.4134381448032934e-7 3.1508904316484904e-7 4.289416889935044e-7

Outlying measurements have moderate (0.12172566390385064%) effect on estimated standard deviation.

nub . sort/200(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.216475057444917e-5 4.2317400734772936e-5 4.246978092192471e-5
Standard deviation 3.7062944224911167e-7 4.3351137076525807e-7 5.089678674216461e-7

Outlying measurements have no (8.546373365041474e-3%) effect on estimated standard deviation.

nub . sort/400(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.282245277015957e-5 9.320961990839635e-5 9.355723657418552e-5
Standard deviation 8.201438284319131e-7 1.0506020037703956e-6 1.4108922473546788e-6

Outlying measurements have no (9.899999999999907e-3%) effect on estimated standard deviation.

nub . sort/600(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.4832539255063426e-4 1.492731058294397e-4 1.5042820345400224e-4
Standard deviation 2.29003584960499e-6 2.6726065105320724e-6 3.191635743251117e-6

Outlying measurements have slight (9.458410032858765e-2%) effect on estimated standard deviation.

nub . sort/800(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.9772263832346315e-4 1.9884930315183615e-4 2.0170872280043264e-4
Standard deviation 3.3832443230180613e-6 4.739789145417122e-6 6.484894287969775e-6

Outlying measurements have moderate (0.14692040786180915%) effect on estimated standard deviation.

nub . sort/1000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.674649207399116e-4 2.6904871211779223e-4 2.701282698317177e-4
Standard deviation 3.0907013294925793e-6 3.7025180941590646e-6 4.8972765880329105e-6

Outlying measurements have slight (1.265614727153205e-2%) effect on estimated standard deviation.

nub . sort/1250(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.666235836384875e-4 3.686917554876405e-4 3.741375068870112e-4
Standard deviation 3.597770609980744e-6 1.023396860661694e-5 1.8299030310450773e-5

Outlying measurements have moderate (0.17137333050247522%) effect on estimated standard deviation.

nub . sort/1500(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.346983162821604e-4 4.367096435763704e-4 4.41128227664025e-4
Standard deviation 4.592231019632417e-6 6.3263607061376705e-6 8.329612715083202e-6

Outlying measurements have slight (1.4282713715605806e-2%) effect on estimated standard deviation.

nub . sort/2000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.006059545172231e-4 6.047946163595983e-4 6.076365673088377e-4
Standard deviation 7.499819691258609e-6 9.695987404426651e-6 1.2441309132311332e-5

Outlying measurements have slight (4.9553393377783576e-2%) effect on estimated standard deviation.

nub . sort/3000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.012643199449107e-3 1.0182876870939695e-3 1.0252284007625304e-3
Standard deviation 1.4490086932630799e-5 1.6553110208887063e-5 1.9431924437269065e-5

Outlying measurements have slight (1.8511925952296015e-2%) effect on estimated standard deviation.

nub . sort/4000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.4569972433267314e-3 1.463655787261681e-3 1.4686469020670464e-3
Standard deviation 1.220352386340164e-5 1.6516676004065828e-5 2.1868215093035805e-5

Outlying measurements have slight (2.1266540642721862e-2%) effect on estimated standard deviation.

nub . sort/5000(1->10)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.0887440677281066e-3 2.1014172460069587e-3 2.120051430518218e-3
Standard deviation 2.4420399549278726e-5 3.953901637408655e-5 6.16988966374302e-5

Outlying measurements have slight (2.4374999999999973e-2%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.