criterion performance measurements

overview

want to understand this report?

sort . nub/100(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.9890056929707976e-5 4.9979490027449354e-5 5.007660122773075e-5
Standard deviation 1.92146822909029e-7 2.480548047712669e-7 2.925134379121018e-7

Outlying measurements have no (8.848852040816277e-3%) effect on estimated standard deviation.

sort . nub/200(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.774573058842562e-4 1.7774412061217614e-4 1.7827654148536544e-4
Standard deviation 7.84277115272655e-7 1.277278365245899e-6 1.834047558259831e-6

Outlying measurements have slight (1.1362135024442036e-2%) effect on estimated standard deviation.

sort . nub/400(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.363998986549905e-4 6.392523068558972e-4 6.427235244901397e-4
Standard deviation 6.146796688892394e-6 7.877905823774261e-6 1.057424999053296e-5

Outlying measurements have slight (1.586888657648289e-2%) effect on estimated standard deviation.

sort . nub/600(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.3015176707254841e-3 1.3057868178207727e-3 1.3112017823320482e-3
Standard deviation 1.0751754877774367e-5 1.396035927209839e-5 1.8337945946467235e-5

Outlying measurements have slight (2.0399305555555452e-2%) effect on estimated standard deviation.

sort . nub/800(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.053731065198546e-3 2.0586154035162818e-3 2.0640378013787697e-3
Standard deviation 9.282312985330934e-6 1.4211522167369824e-5 1.9371484465654823e-5

Outlying measurements have slight (2.3795359904818465e-2%) effect on estimated standard deviation.

sort . nub/1000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.877682553234292e-3 2.882623491184915e-3 2.8923031770271713e-3
Standard deviation 1.3050844142931901e-5 2.218605685588604e-5 3.1168413523922906e-5

Outlying measurements have slight (2.7755102040816267e-2%) effect on estimated standard deviation.

sort . nub/1250(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.98752590500976e-3 3.995297317442978e-3 4.005253782858267e-3
Standard deviation 1.3041111022993212e-5 2.2410081626275807e-5 2.9871519550842418e-5

Outlying measurements have slight (3.222222222222209e-2%) effect on estimated standard deviation.

sort . nub/1500(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.288364872967205e-3 5.29493121357799e-3 5.306462418427028e-3
Standard deviation 1.3854640084822788e-5 2.075517788594891e-5 2.7886371939557197e-5

Outlying measurements have slight (3.698224852070992e-2%) effect on estimated standard deviation.

sort . nub/2000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.099849215402785e-3 8.134179290395723e-3 8.193092123664893e-3
Standard deviation 5.425376132504736e-5 8.71866426788114e-5 1.4303445037847342e-4

Outlying measurements have slight (4.535147392290227e-2%) effect on estimated standard deviation.

sort . nub/3000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.3802131424208637e-2 1.3866087134532685e-2 1.391453226793713e-2
Standard deviation 7.974300580571309e-5 1.0996136478146354e-4 1.4790569874233712e-4

Outlying measurements have slight (5.859375e-2%) effect on estimated standard deviation.

sort . nub/4000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.0450751970954622e-2 2.050668744763688e-2 2.064067211457241e-2
Standard deviation 6.303875092385447e-5 1.9387075822774848e-4 2.496448231087852e-4

Outlying measurements have slight (7.100591715976319e-2%) effect on estimated standard deviation.

sort . nub/5000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.596486569921882e-2 2.6050706814547908e-2 2.6198448710965477e-2
Standard deviation 1.606869881729483e-4 2.0867163850032407e-4 2.695705960475899e-4

Outlying measurements have slight (8.264462809917353e-2%) effect on estimated standard deviation.

nub . sort/100(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.252482346368002e-5 5.2634138226322096e-5 5.29140273311206e-5
Standard deviation 3.5800971293832767e-7 4.814900172810157e-7 6.461100353737091e-7

Outlying measurements have no (8.927846765684318e-3%) effect on estimated standard deviation.

nub . sort/200(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8867566211567245e-4 1.8914061373470202e-4 1.8958516100844153e-4
Standard deviation 9.373885262695321e-7 1.244509361726705e-6 1.6698632444061457e-6

Outlying measurements have slight (1.1626297577854841e-2%) effect on estimated standard deviation.

nub . sort/400(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 6.691975352736602e-4 6.712392876586298e-4 6.733395637983126e-4
Standard deviation 4.754519935504887e-6 6.145931828188291e-6 7.34287733993525e-6

Outlying measurements have slight (1.6124697661918895e-2%) effect on estimated standard deviation.

nub . sort/600(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.2182799320168672e-3 1.224868891510548e-3 1.2369129753949037e-3
Standard deviation 1.7970079505161785e-5 2.4842587124872056e-5 3.797246392452889e-5

Outlying measurements have slight (7.513799319759219e-2%) effect on estimated standard deviation.

nub . sort/800(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.8715961515452877e-3 1.8784255110054567e-3 1.885056385977592e-3
Standard deviation 1.5826285745489875e-5 1.7962130783047734e-5 2.0498091164475396e-5

Outlying measurements have slight (2.3242630385487528e-2%) effect on estimated standard deviation.

nub . sort/1000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 2.6543167520671988e-3 2.658903648444294e-3 2.6634689230478404e-3
Standard deviation 1.0531373485208126e-5 1.2205088769725036e-5 1.4638266594973236e-5

Outlying measurements have slight (2.700617283950602e-2%) effect on estimated standard deviation.

nub . sort/1250(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.658654728449784e-3 3.685673394791171e-3 3.7006878684978915e-3
Standard deviation 4.202864735405513e-5 5.098493883039992e-5 5.996567091751282e-5

Outlying measurements have slight (3.1217481789802187e-2%) effect on estimated standard deviation.

nub . sort/1500(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 4.360948524094164e-3 4.367836455957483e-3 4.375031176260478e-3
Standard deviation 1.5734142118918266e-5 1.921849653914224e-5 2.3676313522929544e-5

Outlying measurements have slight (3.329369797859687e-2%) effect on estimated standard deviation.

nub . sort/2000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.885489159079811e-3 5.892153007146299e-3 5.899997582895967e-3
Standard deviation 1.6051879758707154e-5 1.9977402655491634e-5 2.7695484473264988e-5

Outlying measurements have slight (3.839999999999995e-2%) effect on estimated standard deviation.

nub . sort/3000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 8.086584848944194e-3 8.09794260887604e-3 8.117769925867853e-3
Standard deviation 3.1140970681087047e-5 3.5733146199633344e-5 4.650131487726725e-5

Outlying measurements have slight (4.535147392290228e-2%) effect on estimated standard deviation.

nub . sort/4000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 9.448140238077623e-3 9.460618951190538e-3 9.473848974736339e-3
Standard deviation 2.0104058994367147e-5 2.6714645652664253e-5 3.9506918581402794e-5

Outlying measurements have slight (4.75e-2%) effect on estimated standard deviation.

nub . sort/5000(1->1400)

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 1.0706639107201267e-2 1.072723038470658e-2 1.0772809103532042e-2
Standard deviation 4.0178951174718254e-5 6.0624281580913094e-5 8.627761904047203e-5

Outlying measurements have slight (5.246913580246878e-2%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.