Module Expr Str ImportFrom alias Import alias identifier ImportFrom alias ImportFrom alias identifier Assign NameStore Num FunctionDef arguments args NameParam NameParam defaults NameLoad body If CompareIs NameLoad NameLoad body Assign NameStore SubscriptLoad Call NameLoad Call AttributeLoad NameLoad attr Index Num Assign NameStore Call NameLoad GeneratorExp TupleLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr CompareNotEq NameLoad NameLoad Assign NameStore SubscriptLoad NameLoad Index NameLoad Return TupleLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults Num body Expr Str Assign NameStore Call NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Assign NameStore BinOpDiv BinOpMult Num BinOpSub Num NameLoad Num Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Num Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr BinOpDiv BinOpAdd NameLoad NameLoad Num Assign SubscriptStore NameLoad Index NameLoad TupleLoad Call NameLoad NameLoad NameLoad NameLoad Call NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Assign NameStore AttributeLoad NameLoad attr Assign NameStore BinOpDiv BinOpSub Call AttributeLoad NameLoad attr NameLoad NameLoad Assign SubscriptStore NameLoad Index NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad Num NameLoad body Expr Str Assign NameStore Call NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr keyword NameLoad Assign NameStore BinOpDiv BinOpSub NameLoad NameLoad NameLoad Assign NameStore BinOpMult Num BinOpSub Num NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad BinOpAdd NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign SubscriptStore NameLoad Index NameLoad TupleLoad Call NameLoad NameLoad NameLoad NameLoad Call NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Assign NameStore Call NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad FunctionDef arguments args NameParam defaults body TryExcept body Return Call AttributeLoad NameLoad attr handlers ExceptHandler body Return CompareEq Call NameLoad NameLoad NameLoad decorator_list For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body If BoolOpAnd Call NameLoad AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr body Assign NameStore Num For NameStore Call NameLoad BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr body AugAssignAdd NameStore Call AttributeLoad NameLoad attr BinOpSub BinOpSub BinOpSub Call AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr NameLoad BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr BinOpAdd AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign SubscriptStore NameLoad Index NameLoad NameLoad orelse Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad NameLoad attr CompareGt NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Assign NameStore BinOpSub BinOpAdd BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore BinOpSub BinOpAdd BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore BinOpDiv Call NameLoad BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpDiv Call NameLoad BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpDiv BinOpSub NameLoad NameLoad NameLoad Assign SubscriptStore NameLoad Index NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Assign NameStore BinOpSub BinOpAdd BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore BinOpSub BinOpAdd BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore BinOpDiv Call NameLoad BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpDiv Call NameLoad BinOpSub AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpDiv BinOpMult NameLoad BinOpSub Num NameLoad NameLoad Assign NameStore BinOpDiv BinOpMult NameLoad BinOpSub Num NameLoad NameLoad Assign NameStore BinOpDiv BinOpSub NameLoad NameLoad Call AttributeLoad NameLoad attr BinOpAdd NameLoad NameLoad Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad Call AttributeLoad NameLoad attr attr NameLoad Return NameLoad decorator_list Assign NameStore Dict Str Str Str Str Str Str Str NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad
N/A N/A Useful statistics. collections OrderedDict N/A numpy np scipy stats scipy special spc N/A SAMPLE_SIZE 10000 _split N/A N/A variations control N/A None N/A N/A N/A control None N/A N/A control N/A N/A list N/A N/A variations keys N/A 0 N/A others N/A OrderedDict N/A N/A label variation N/A N/A label variation N/A N/A variations items N/A label control N/A control N/A variations N/A control N/A N/A control others N/A expected_posterior N/A N/A variations N/A N/A N/A Calculate expected posterior E[parameter | data] for each variation. N/A values N/A OrderedDict N/A N/A label variation N/A N/A variations items N/A N/A N/A values N/A label N/A N/A N/A variation posterior mean N/A values N/A posterior_credible_interval N/A N/A variations level N/A 95 N/A N/A Calculate posterior P(parameter | data) credible interval for each
    variation.

    Returns a 3-tuple (lower, median, upper)
     N/A values N/A OrderedDict N/A N/A label variation N/A N/A variations items N/A N/A left_percentile N/A N/A 0.5 N/A 100 level 100 N/A left N/A N/A N/A variation posterior ppf left_percentile N/A median N/A N/A N/A variation posterior ppf 0.5 N/A right N/A N/A N/A variation posterior ppf N/A N/A level left_percentile 100 N/A N/A values N/A label N/A N/A min left right median N/A max left right N/A values N/A expected_lift N/A N/A variations control N/A None N/A N/A Calculate expected lift E[(B-A)/A]. N/A values N/A OrderedDict N/A N/A control others N/A _split variations control N/A a N/A control posterior N/A m N/A N/A a mean N/A N/A label variation N/A N/A others items N/A N/A b N/A variation posterior N/A lift N/A N/A N/A N/A b mean m m N/A N/A values N/A label lift N/A values N/A lift_credible_interval N/A N/A variations control level sample_size N/A None 95 SAMPLE_SIZE N/A N/A Calculate credible interval for lift E[(B-A)/A] using MCMC.

    Returns a 3-tuple (lower, median, upper)
     N/A values N/A OrderedDict N/A N/A a others N/A _split variations control N/A N/A label b N/A N/A others items N/A N/A a_samples N/A N/A N/A a posterior rvs size sample_size N/A b_samples N/A N/A N/A b posterior rvs size sample_size N/A lift_samples N/A N/A b_samples a_samples a_samples N/A left_percentile N/A 0.5 N/A 100 level N/A right N/A N/A np percentile lift_samples N/A level left_percentile N/A median N/A N/A np percentile lift_samples 50 N/A left N/A N/A np percentile lift_samples left_percentile N/A N/A values N/A label N/A N/A min left right median N/A max left right N/A values N/A dominance N/A N/A variations control sample_size N/A None SAMPLE_SIZE N/A N/A Calculate P(A > B).

    Uses a modified Evan Miller closed formula if prior parameters are integers
    http://www.evanmiller.org/bayesian-ab-testing.html

    Uses scipy's MCMC otherwise

    TODO: The modified formula for informative prior has to be proved correct
     N/A values N/A OrderedDict N/A N/A a others N/A _split variations control is_integer N/A N/A x N/A N/A N/A N/A N/A N/A N/A x is_integer N/A N/A N/A N/A N/A N/A int x x N/A N/A N/A label b N/A N/A others items N/A N/A N/A N/A is_integer N/A a prior_alpha N/A is_integer N/A a prior_beta N/A is_integer N/A b prior_alpha N/A is_integer N/A b prior_beta N/A N/A total 0 N/A i N/A range N/A N/A b alpha N/A b prior_alpha N/A N/A total N/A N/A np exp N/A N/A N/A N/A N/A spc betaln N/A N/A a alpha i N/A N/A b beta N/A a beta N/A N/A np log N/A N/A b beta i N/A N/A spc betaln N/A N/A b prior_alpha i N/A b beta N/A N/A spc betaln N/A a alpha N/A a beta N/A N/A values N/A label total N/A N/A a_samples N/A N/A N/A a posterior rvs sample_size N/A b_samples N/A N/A N/A b posterior rvs sample_size N/A N/A values N/A label N/A N/A np mean N/A b_samples a_samples N/A values N/A empirical_lift N/A N/A variations control N/A None N/A N/A Calculate empirical lift E[(B-A)/A]. N/A values N/A OrderedDict N/A N/A a others N/A _split variations control N/A N/A label b N/A N/A others items N/A N/A total_a N/A N/A N/A N/A a alpha N/A a prior_alpha N/A a beta N/A a prior_beta N/A total_b N/A N/A N/A N/A b alpha N/A b prior_alpha N/A b beta N/A b prior_beta N/A p_a N/A N/A float N/A N/A a alpha N/A a prior_alpha total_a N/A p_b N/A N/A float N/A N/A b alpha N/A b prior_alpha total_b N/A lift N/A N/A p_b p_a p_a N/A N/A values N/A label lift N/A values N/A ztest_dominance N/A N/A variations control N/A None N/A N/A Calculate z-test for dominance. N/A values N/A OrderedDict N/A N/A a others N/A _split variations control N/A N/A label b N/A N/A others items N/A N/A total_a N/A N/A N/A N/A a alpha N/A a prior_alpha N/A a beta N/A a prior_beta N/A total_b N/A N/A N/A N/A b alpha N/A b prior_alpha N/A b beta N/A b prior_beta N/A p_a N/A N/A float N/A N/A a alpha N/A a prior_alpha total_a N/A p_b N/A N/A float N/A N/A b alpha N/A b prior_alpha total_b N/A sse_a N/A N/A p_a N/A 1 p_a total_a N/A sse_b N/A N/A p_b N/A 1 p_b total_b N/A zscore N/A N/A p_b p_a N/A N/A np sqrt N/A sse_a sse_b N/A N/A values N/A label N/A N/A N/A N/A stats norm cdf zscore N/A values N/A N/A statistic_funcs N/A expected posterior posterior CI expected lift lift CI empirical lift dominance z-test dominance expected_posterior posterior_credible_interval expected_lift lift_credible_interval empirical_lift dominance ztest_dominance