tttjjj/elt_bench huggingface dataset:
You could also split it into two models: the probability of buying a car (binary response), and the value of the car given a purchase. This is the standard practice in many fields, eg insurance, credit risk, etc.
@gung I want to understand the relevant ideas, just in relation to a specific example, and with specific results.  The first answer of the thread you link has literally 14 links to external resources.  It's got to be simpler to answer this question of narrow scope.  It's like answering "what is the confidence interval of a binomial proportion" with "read the following three books".  I'd answer instead: "1.96 * sqrt(p*(1-p)/n) for a 95% confidence interval.  Here are some refs."
Hi, its nice paper, thanx for giving pdf link.. I'll surely go through this paper..
@MichaelChernick I am talking about the smallest detectable difference in the mean of the base samples(A). Since i am unable to re-take this samples, I have to consider that the new samples i take will have a slightly different variance than the base samples. This leads me to think that I would need more samples to cover for this location error than if i could re-take the baseline samples. To investigate how big this location error is i have the B samples.
A formal explanation requires a formal statement of the assumptions of an underlying probability model. Typically PCA is used without such assumptions - a mere transformation. See [here](http://stats.stackexchange.com/questions/87198/pca-randomness-of-component) for some examples of the danger of interpreting smaller components as noise.
typo in "that a randomly **dram** observation" ;)
Nope, the prior is not necessarily incorrect, unless one didn't actually observe a value of $0$ prior to conducting the experiment (or obtain some equivalent knowledge).  It basically means that, as the true $\\theta$ becomes arbitrarily large, the probability of observing these implicit observations becomes arbitrarily small (like getting a "unlucky sample").
You're welcome. Note I edited the index of $\\beta_{z3}$ to $\\beta_{z31}$ and $\\beta_{z32}$ because these are two different coefficients.
@whuber I intentionally did not give such details as this is a self-study question. It is a hint, not a full answer. I find it usually helps to know where I should end up, in order to fill in the steps in between.
There is no $n\\to \\infty$. For each fixed $n$, the random variable $(x'_i y_m-1)^2/\\|y_m\\|^2$ has unbounded support when $i\\notin S_m$.  This answers your question to the negative.
@whuber Yep - I re-read the question after posting and realized that I was responding to an implicit comparison of MLE vs posterior inference rather than his substantive concern.
It might be a feature of your data; have you tried running other RF implementation on the same dataset to see if it reproduces this effect?
there are tools to increase the frequency using statistical tools
@Stauner can I have your email address.
I'm not quite sure you mean when you say *"The best we can hope for is maybe 54%"*. Assuming you're referring to correctly predicting the class labels and that the outcomes aren't nearly 50/50 coin flips, you can probably do better than that without a model at all. For example, if the proportion of $1$s in the data set is $p>1/2$ (if $p<1/2$ just flip the labels). Then you will get a prediction accuracy of $p$ just by guessing everything is a $1$. Maybe I've missed your point..
@Glen_b They are measuring different quantity, adhesion strength and relative humidity. Null hypothesis is that there is not relationship between these two different quantities. I want to know that are my measurements of adhesion strength result of chance or are they statistically significant which occurred mostly due to relative humidity.
“Failing to reject normality doesn't imply your data are normal.”
Since this is a possibility, then which test I really should use to test statistical significance?
The existence of the baseline is comforting, but without a control group, I think you would need to be able to estimate the prior *trend*. Think of it this way, what if the proportion of good results had been going down beforehand at a faster rate, & after the intervention it is still going down, but at a slower rate? Then, the intervention was positive, even though the post intervention slope is negative. I don't see how you can rule that out. Moreover, standard correlations & t-test will be subject to the same problems, so I don't think that will help you. I hope it works out.
@NickStauner That is only one example. In a comment probabilityislogic hints at further situations. I would like to know when to be concerned about this misinterpretation. For example, I suspect it is uncommon for unimodal distributions to accurately describe biological systems. What about in that case? I am wondering if there is a general rule though.
So the code would be like the following? 
GLM.1 <- glm(Group ~ continuousVar, family=binomial(logit), data=diagnosticData)

summary(GLM.1)

GLM.2 <- glm(Group ~ 1, family=binomial(logit), data=diagnosticData)

anova(GLM.2, GLM.1, test="Chisq")
@Whuber we are good ... I just wanted to clarify that the term "amount of variation" means the error variance NOT the variance of the observed series and the term(too vague for me) "underlying value" is more precisely the "expected value"
I don't understand your main paragraph. First of all, the convergence to a normal is only after a shift and rescale of $S_n$ and this is not by the Berry--Esseen theorem (which is a theorem on the *convergence rate* to normal), but the CLT. Furthermore, the shift and rescale will depend on the particular fixed parameter. Have you looked at a case where you have, say, a three point prior uniformly distributed on $\\{0,1/2,1\\}$?
Yes you are correct and in fact I meant multiply the p value by the number of comparisons or in fact divide alpha by the number of comparisons
@dmartin, you mean p value? the number of the measurements from the patients are not the same amount. One patient might have 30 measurements, other might 16.  Why the assumption of independence is violated? So, if I use mixed effects model, what should be the fixed variables and random variables in SPSS? Or how to apply the mixed-effects model in SPSS?
Hi Nick,

I just wanted to thank you for updating my question to make it more legible - apologies but I couldn't work out how to private message you so thought i'd message you on here. I don't suppose you have any idea with regards to my question do you?
Thanks Liran, this is the classical XOR example that a linear classifier cannot solve but union of classifiers can. Nevertheless, I was thinking more on algorithms which can learn ANY distribution such as Random Forest, Nearest Neighbour, Neural Networks, SVM + Kernel etc.
I am not sure but have you looked at functional data analysis? e.g [Ramsay and Silverman](http://www.amazon.com/Functional-Analysis-Springer-Series-Statistics/dp/1441923004/ref=sr_1_1?ie=UTF8&qid=1351970299&sr=8-1&keywords=functional+data+analysis)
Glad to expand if you can state a specific question.
You can figure out the value $\\sum_i(a_i-\\bar{a})^2$ and $\\sum_j(b_j-\\bar{b})^2$ from the given s.d.'s and known sizes of the two data sets, and therefore can figure out $\\sum_ia_i^2$ and $\\sum_jb_j^2$. What you are after is $\\sum_i (a_i-\\bar{a+b})^2+\\sum_j(b_j-\\bar{a+b})^2$ where you know $\\bar{a+b}$ and can creatively use $(\\alpha-\\beta)^2=\\alpha^2+\\beta^2+2\\alpha\\beta$ to get this quantity.  Divide by size of total data set $- 1$, take square root, and you have the s.d. of the combined data set.  Sorry, can't help you with the R code; I am not a statistician.
I think it's a legitimate question: If a phrase includes an adjective like “adaptive”, it makes sense to wonder what distinction is implied.
Because $\\lambda$ and $\\mu$ are Lagrange multipliers, it's trivial to find derivatives with respect to them.  How do you anticipate the functional form of $L$ changing so that the derivatives with respect to the $w_i$ -- which currently are dead simple to compute -- will become complicated? // Anyway, any symbolic mathematics program should be able to do the job well: MatLab, Mathematica, etc.
This is a very good tutorial, thank you. I am interested in the nonlinear case. I suspect that in this case, we never explicitly calculate the weights since we do not know the phi mapping, but we can still evaluate the kernel function using the training samples and the test sample and calculate the discriminant. Yet, it is not crystal clear to me.
I think this is a duplicate question. There are plenty of questions like that on CV. See this question here: http://stats.stackexchange.com/questions/95839/gini-decrease-and-gini-impurity-of-children-nodes/95842#95842
Dilip I've been checking and I think there's no typo because in the paper in the confusion matrix in page 236 the FAR denominator is the total number of events predicted to occur so it is the first row of the confusion matrix
Karl
2x2 matrix's upper left corner is only one element... Can you reformulate the question?
Just as a side note: One needs to consider a little more than just the partition argument. Namely, the associated limit must be shown to also be a normal random variable. This is true, but doesn't simply follow "naively" from the fact that each element of the sequence is normal. (+1)
I assume that this is homework/self-study related. Please see the following link: http://meta.stackexchange.com/questions/10811/how-to-ask-and-answer-homework-questions/10812#10812 Try to show us your train of though. Someone just spitting out `b`-`b`-`b` would be quite useless for you anyway. ( `b`-`b`-`b` is not valid answer)
Frank Harrell on this site has a pretty epic post on evaluating [contingency tables with small counts](http://stats.stackexchange.com/a/14230/1036). If you use the `N-1` correction you only need to have expected frequencies around 1 for nominal coverage rates.
If features vary from fold to fold it means that there isn't enough information to confidently identify the useful features, so I'd view that as an advantage of cross-validation (as just looking at the results from a single model is likely to have over-fit the feature selection criterion and hence be misleading).  For problems with many features and few observations, ridge regression often gives better performance, so unless identifying features is a key goal, it is often better not to do any feature selection.
I am not quite sure what you mean. I added coded showing how you get the smoothed means in each bin by hand. If that's not what you are looking for, please explain what you have in mind in more detail. As far as I can tell, these graphs usually show the raw data and the smoothed means.
How many values are in your sample and what level of significance do you need?
@MånsT thanks for making me aware of the procedure.
For (I), what's a good procedure for a split vote (1:a, 2:c, 3:b), or is that likely too rare to really matter?
I would suggest to normalize (to one) that vector before feeding into SVM.
It should be possible to derive the expected correlation given the distributions.... but it's not the sort of thing I know how to do!
@Glen_b do you mean if $\\alpha > p-value$ reject $H_0$ ?

Does $D statistic$ is the same as $alpha$
Won't there tend to be some bias in the case where there isn't really a correct answer.  For example, you give a choice of Cindy Crawford and Nell Carter to describe a picture of the Sun.. Most people might just pick Cindy Crawford cause they like her.  Just a thought.
I don't comprehend what the intended meaning of "*a bias model is more relaxed while a model with a lot of variance is more flexible*" might be. I can't identify a context which makes sense of it. What do you mean and why do you think so?
(1) I'd model first assuming not breakpoint. With stata there are a number of options (http://www.stata.com/statalist/archive/2008-07/msg00305.html) including GAM (generalized additive models), mkpline=splines, and smoothers. (2) for breakpoint, segmented has given best results. But could use MARS/EARTH package in R.
Thanks for the clarification.  it helps to know what you are driving at even though i don't think I can answer the question. The idea of using the bootstrap always perks my interest.
You've asked quite a few homework questions the past few days, @Ranabir. This is perfectly fine of course, but perhaps not advisable: if you actually want to learn statistics then it's often better to solve the problems on your own (this includes looking in lots of books)...
@user023472, the coefficients tell you about the additional contribution of each variables, given the others in the model. They work together. That's true in logistic regression too, so maybe you need to check back on some examples where you thought you could interpret the coefficients as correlations. You can assess the strength of each variable's contribution by the absolute value of each $t$ statistic. The one with the greatest $t$ makes the greatest contribution.
(ctd) ... then a reminder may be okay. But even then, the OP is the final arbiter of what they choose to accept, or whether they accept any answer at all.
Tukey did not design or use hinges for breaking data into mutually exclusive groups: he used them for analysis.  As such, his approach was to split a batch of data at the median into two groups that *overlap at the median itself* (when the batch has an odd count) and then recursively to split those two groups in the same fashion.  How *best* to group data depends on how the groups will be used.  In some applications it matters little; in other applications it matters a lot; and in a few, it's invalid to choose the group cutpoints based on the data.
I think it would be more accurate to rewrite your question as: "How do I interpret the *coefficients* of x and x^2". (Since your actual interpretation of "x" and "x^2" would be independent of the regression itself)
Several questions:  Please tell us N and the measurement level of your dependent variable(s)... Are your variables with >100,000 classes Xs or Ys?...For how many variables, or for how many levels of a given variable, are you trying to impute values?...Why do you need to impute values in order to make predictions?
The second is the one orthogonal to the first that expains the largest percentage of the remaining variation and so on. But the principla components are linear combinations of all your features.  if you use prinicpal components you can reduce the number of variables only in the sense that the number of principal components describing the dependent variable can be a lot smaller than the number of original features.  But they don't reduce the number of features used.
(+1) Your statement is correct--and, as you point out, trivial at the same time. You definitely could make the entire parameter space your confidence set in every problem you address, but in short order you would have no clients and no career as a statistician because your information would not be helpful!
(1) Are all your effects linear in the logit? If not the inflection point doesn't necessarily coincide with a 0.5 probability for the response - which do you want? (2) At what values will you fix the other six predictors when calculating the inflection point of the one remaining predictor?
Uh, I always thought the natural reason to code female = 0 and male = 1 was "anatomy"...
Some basic material is [here](http://www.iac.ethz.ch/edu/courses/master/electives/acwd/Trend.pdf) (also [here](http://en.wikipedia.org/wiki/Trend_estimation)) and slightly less basic [here](http://pubs.usgs.gov/twri/twri4a3/pdf/chapter12.pdf); these would be good background.
Check out [this paper](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CCIQFjAA&url=http%3A%2F%2Fwww.researchgate.net%2Fpublication%2F221112311_Applying_Support_Vector_Machines_to_Imbalanced_Datasets%2Ffile%2F5046351b0229e207bd.pdf&ei=47SPU-HnG5aWqAanhYCADQ&usg=AFQjCNHyZ23NvEq87WMeizGN0I88mmVeZw&sig2=XisZd_z8rURfzJi01tCrcg&bvm=bv.68235269,d.b2k). It shines some light on ideas on how to deal with unbalanced data.
@StasK Interesting stuff (I just had a quick scan of the abstract and intro of the original paper). Thanks for the comment. By the way, if you edit for Wikipedia at all, feel free to improve [this entry on Berkson error](http://en.wikipedia.org/wiki/Berkson_error_model) as it is probably the most uninformative Wikipedia page I've ever seen :-)
@ Melon if you add your comment to your question above, you are far more likely to get better responses.
$P(Y\\leq y) = P(X_1 \\leq y).P(X_2 \\leq y)\\ldots P(X_n \\leq y)$. Can you do it now?
And in fact R's NegBinomial functions can be parametrized with `mu` rather than `prob`.
I do appreciate your attempt at helping, but unfortunately, this just made things 10x worse.  Are you really introducing trigonometry to explain r^2?  You're way too smart to be a good teacher!
I wouldn't bother comparing these models until after checking their goodness of fit.  I think you will find in the second one that neither the response nor its logarithm are linear functions of time.  This calls (seriously) into question any comparison of the slope estimates.
Thanks! This helps a lot.
Can you pls tell what is the size of your data set.
I've also read that block bootstrap is only consistent (or at least the method you appear to mention is) [when you have stationary procceses](http://arno.unimaas.nl/show.cgi?fid=15555) (some of them don't work propperly even in WSS procceses). For some special kind of non-stationary series, however, [some types of block-bootstrap appears to work well](http://arxiv.org/pdf/0711.4493.pdf).

With all of this in mind, my advice is to check what kind of model better suits your data (e.g. using a PACF). If you are sure of the stationarity of your data, I see no problem on your technique.
The 95% comes from the desired confidence level, so it has already been specified in the problem.  The burden on anyone suggesting an alternative confidence interval procedure is to demonstrate that their proposal guarantees 95% (or whatever is desired) coverage of the true mean--no matter what the true mean might happen to be--and that it does so efficiently: that is, its minimum coverage really is 95% or close to it.  Although your proposal might have this property, it's not apparent that it does.
"[T]o ensure independence of potential outcomes from treatment assignment, it suffices to ensure that each unit in the study has equal probability of assignment to treatment." **This is incorrect.** Suppose you have enrolled $x$ males and $x$ females in a study. Flip a fair coin: if heads, assign all females to the treatment group (and all males to the control group); if tails, all males will be in the treatment group and all females in the control group. Each subject (obviously) has 50% chance of assignment to the treatment group--but treatment is completely confounded with gender.
If your data is MCAR you could use only complete case analysis. Many papers report that using complete case analysis with MCAR data is the best solution. At least, some of the papers I found report this, even when comparing with other imputation methods
Amazing! Thank you so much for the reference and the explanation. I have edited your post somewhat, but the edit must be approved by an experienced member. Is http://www.jstor.org/stable/10.2307/2290793 the correct link to the Deville et al. paper? I understand that the multilevel case is hidden in the sentence starting with "Simple examples are" (line 13) in the introduction?
+1, sounds like exactly what I'm looking for. Plan to apply this one.
But this will affect the size of the overall test and such adjustment could be an intricate matter. I admit it's rather non-standard, but the null can be formulated like "$p_a$ is not the largest" and theoretically a p-value should exist telling us how extreme our outcome is given such $H_0$. Of course if this is overly difficult then a compromise will have to be made.
Global vector autoregression models deal with similar (but real world) applications.
Do the points in B have a one-to-one relationship with the points in A (eg each is a particular star) or is it more abstract than that?
@Matt Krause Indeed, the notation might be field-specific. Please note that in the notation I mentioned the standard error is not a subscript, but rather in parenthesis (e.g., $4.2(0.2)$).
Thanks a lot for your answer. It helped a lot!
In Python you can handle time series data using pandas. Try googling for `time series pandas`.
You need to define $t(a)$, link this to $Y$, $X$, and show where the two-sample problem comes from. Also, I would suspect that $\\| a \\| = 1$, which would make optimization problem somewhat more complicated.
@Cam The data didn't make it into the final cut :-).  (The context is rather tongue-in-cheek, I'm afraid.)
A structural break can affect seasonality as well, but it depends on the underlying data (e.g. military changing their intake numbers and time of year!). For seasonality, is there any theory behind those effects, e.g. you're selling winter clothing, or people tend to buy the products on the weekend, etc? The last time I used it, `stl` kept the same seasonality trend across all data - this may be an incorrect assumption.
Thank you so much for this thorough post (and accent on R).
And what about the first problem? What if the naive predictor is not the best one? then I want to compare against a random predictor. But I'm unsure how to calculate the chance in this case.
Please explain how you derive your last statement.  It seems to come from some assumption of additivity of likelihood but the context in which that would be true is not at all clear to me.
Something doesn't compute.  How can you have means or medians or CIs with categorical data?
Thanks to all the wonderful answers.  Each one added something important to the question!  The question is sure fraught with ambiguity and loose patterns, sometimes combining incompatible ideas that are well-defined in common statistics language.  The nugget of concreteness within was exposed through algebra and some good reasoning about upper and lower bounds.
I think what @Mark Lamias and Gray are hinting at, Zach, is that your question is perhaps a little too indefinite to answer.  If you could make it more specific (especially in the ways they suggest), it might be able to stand as a good (answerable) question.  Otherwise it risks being closed.
@user570593 Can you use the fact that $\\min(x) = \\max(-x)$? It sounds like you want to optimize a function involving a max function? Do you know about [Subderivatives](http://en.wikipedia.org/wiki/Subderivative), [Subgradient methods](http://en.wikipedia.org/wiki/Subgradient_method), and [Convex optimization](http://en.wikipedia.org/wiki/Convex_optimization)?
Correct me if I'm wrong, but the randomness of sampling simply affects the degree to which you can generalize findings. In contrast, random assignment is the more critical feature for causal inference.
Currently I use square approximation of circle. It works well but apparently involves some biases. You may also discuss the error being added by this approximation. Thanks
@PatrickCoulombe Thanks. I suppose you wanted to say A - 3.686.
You know, it's funny--if I posted on Stack Overflow I probably would have looked at the code first, but over here I didn't even think about it.  Thanks for your help!
@giovanna good question. You might want to ask a separate question about this specific issue. In general, I think determining dimensionality is a bit of an art. From a practical perspective I find that I tend to get similar results whether I do PCA or EFA, but in theory EFA aligns more with the concept of latent factors causing observed items.