criterion performance measurements

overview

want to understand this report?

sort . nub/100(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.453850938638037e-5 5.471412776027625e-5 5.491529161853362e-5
Standard deviation 3.7697879722947886e-7 6.397775743462969e-7 9.345004082372795e-7

Outlying measurements have no (9.008264462809999e-3%) effect on estimated standard deviation.

sort . nub/200(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.986990388169876e-4 1.9893862236542727e-4 1.9949136964973537e-4
Standard deviation 9.115951843069287e-7 1.100605800286499e-6 1.4545431593816167e-6

Outlying measurements have slight (1.1763038548752698e-2%) effect on estimated standard deviation.

sort . nub/400(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.414636715580551e-4 7.427698084659206e-4 7.43885625014405e-4
Standard deviation 2.5024575305449925e-6 3.322815556794531e-6 4.337696387376704e-6

Outlying measurements have slight (1.6661878770468253e-2%) effect on estimated standard deviation.

sort . nub/600(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.6494096933717607e-3 1.6517972690098944e-3 1.6566223102468516e-3
Standard deviation 6.564182223480334e-6 8.309000909112374e-6 1.219031405293991e-5

Outlying measurements have slight (2.2210743801652746e-2%) effect on estimated standard deviation.

sort . nub/800(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.8510772774250426e-3 2.857573422871714e-3 2.864390102895223e-3
Standard deviation 1.1792448225060534e-5 1.7650915483578528e-5 2.580523301562686e-5

Outlying measurements have slight (2.7755102040816326e-2%) effect on estimated standard deviation.

sort . nub/1000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.379310997747846e-3 4.399213493398461e-3 4.420124758815521e-3
Standard deviation 3.5175638481365103e-5 4.8487678212819074e-5 6.003832417670799e-5

Outlying measurements have slight (3.329369797859691e-2%) effect on estimated standard deviation.

sort . nub/1250(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.686882756046457e-3 6.70409519324151e-3 6.728038679921771e-3
Standard deviation 3.791903654905298e-5 5.244010187695931e-5 7.574583101925404e-5

Outlying measurements have slight (4.158790170132312e-2%) effect on estimated standard deviation.

sort . nub/1500(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.745521974270319e-3 9.762268140557563e-3 9.779747425291372e-3
Standard deviation 2.9679734900136248e-5 3.573388119203744e-5 4.652984228011401e-5

Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.

sort . nub/2000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.728268893333036e-2 1.733569587261118e-2 1.7421017955464583e-2
Standard deviation 8.238709178367203e-5 1.4427494873396114e-4 2.1805909969071688e-4

Outlying measurements have slight (6.632653061224489e-2%) effect on estimated standard deviation.

sort . nub/3000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.93191138256386e-2 3.9453436002884514e-2 3.977344160762788e-2
Standard deviation 1.9712448106069073e-4 3.191652420889177e-4 4.90559024418983e-4

Outlying measurements have slight (9.876543209876541e-2%) effect on estimated standard deviation.

sort . nub/4000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.919462500236694e-2 7.01031441894613e-2 7.082037386804867e-2
Standard deviation 8.132177476423067e-4 1.0878608090100369e-3 1.3520919552770234e-3

Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.

sort . nub/5000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.10772653821998346 0.10869444035999928 0.10961440770000383
Standard deviation 1.0921661338497296e-3 1.272291576062404e-3 1.5007757340956254e-3

Outlying measurements have moderate (0.16%) effect on estimated standard deviation.

nub . sort/100(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.782080260462084e-5 5.801871818512889e-5 5.829150454067115e-5
Standard deviation 3.610371358394735e-7 6.305845363754592e-7 8.995785105012933e-7

Outlying measurements have no (9.090143927278908e-3%) effect on estimated standard deviation.

nub . sort/200(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.0676286569605522e-4 2.0744929452215518e-4 2.0851782726882289e-4
Standard deviation 1.7755214999875027e-6 2.3265783107110517e-6 2.866223371903915e-6

Outlying measurements have slight (1.1763038548752802e-2%) effect on estimated standard deviation.

nub . sort/400(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.984157388769116e-4 8.001211422436365e-4 8.036483998075468e-4
Standard deviation 4.923649004721395e-6 6.941073010214901e-6 8.799929766039026e-6

Outlying measurements have slight (1.7236072637734613e-2%) effect on estimated standard deviation.

nub . sort/600(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.786612626774282e-3 1.7892369885313209e-3 1.7924610015618186e-3
Standard deviation 5.665312783320385e-6 8.598010387484495e-6 1.2227604128828813e-5

Outlying measurements have slight (2.2714981070848923e-2%) effect on estimated standard deviation.

nub . sort/800(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.1076592921578812e-3 3.1137830440364517e-3 3.124371019525439e-3
Standard deviation 1.1675437932457002e-5 2.0764853400867385e-5 3.3315622474381944e-5

Outlying measurements have slight (2.8546712802768166e-2%) effect on estimated standard deviation.

nub . sort/1000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.811401245419065e-3 4.831423841666399e-3 4.851152603042935e-3
Standard deviation 3.708778400199269e-5 4.706829127055399e-5 6.474347707097212e-5

Outlying measurements have slight (3.4438775510204085e-2%) effect on estimated standard deviation.

nub . sort/1250(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 7.352561775574147e-3 7.38812984659878e-3 7.430993909336748e-3
Standard deviation 8.389934010196237e-5 1.0243771394567226e-4 1.446370703493651e-4

Outlying measurements have slight (4.338842975206612e-2%) effect on estimated standard deviation.

nub . sort/1500(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0451049497016387e-2 1.048616622610532e-2 1.05330847608525e-2
Standard deviation 6.512161026936831e-5 7.423145283043214e-5 9.681006542142066e-5

Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.

nub . sort/2000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8106190535114013e-2 1.818871288436092e-2 1.8265146392153028e-2
Standard deviation 8.146107424708149e-5 1.6272357012631335e-4 2.0887544853779569e-4

Outlying measurements have slight (6.632653061224476e-2%) effect on estimated standard deviation.

nub . sort/3000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.950679114939697e-2 3.971894308964747e-2 3.999238516113379e-2
Standard deviation 2.311581340160376e-4 3.787073661153276e-4 5.160695501832153e-4

Outlying measurements have slight (9.876543209876536e-2%) effect on estimated standard deviation.

nub . sort/4000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.881639110713747e-2 6.907053715814523e-2 6.969499453738177e-2
Standard deviation 2.4863740114935994e-4 5.36914520320153e-4 7.546363427730072e-4

Outlying measurements have moderate (0.12244897959183675%) effect on estimated standard deviation.

nub . sort/5000(1->536870911)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.10705690508002134 0.10864958899666817 0.11229230031665338
Standard deviation 1.377542827549194e-3 2.782372863500341e-3 3.9910287973872905e-3

Outlying measurements have moderate (0.16000000000000003%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.