tttjjj/elt_bench huggingface dataset:
There is so very little to be done to answer this question that one hardly knows what to write in response. Perhaps you could help us out by explaining the point of this exercise: what is it helping you to learn? What have you tried so far in your efforts to solve it? Where are you stuck?
"Confident" is an interesting word in this context, because linear regression provides information for constructing intervals of "confidence" around prediction: [prediction intervals](http://en.wikipedia.org/wiki/Prediction_interval#Regression_analysis). A wide interval reflects lack of "confidence" or trustworthiness in a prediction. You could use these widths to screen predictions for likely correctness. This could work well in conjunction with [goodness of fit tests](http://en.wikipedia.org/wiki/Lack-of-fit_sum_of_squares) (which would indicate whether to trust the prediction intervals!).
This has been very helpful, did you receive the bounty?
Hints towards the derivation: You already have reduced the problem to "If $Y\\sim N(0,\\theta_2)$, then show that $E[Y^4]=3\\theta_2^2$". So set up the integral, and integrate by parts using $u = y^3, \\mathrm dv = \\mathrm d(-e^{-y^2/2})$.
This was helpful, so I marked it as the answer. It seems as if the arima.rob() procedure in S+FinMetrics provides robust fitting of arima models and is able to detect the types and locations of any outliers. Unfortunately I haven't been able to find an equivalent package in R. Regrettably I have no budget for S+ or autobox or similar software :-( . And lack the expertise to program this myself.
Is this a homework question? If so, please change one of your existing tags for the `[self-study]` tag & read its wiki.
+1 Good finds.  However, most of the differential geometry applications to statistics do not use symplectic geometry.  I think our difficulty in finding references is testimony to how obscure and rare are the applications of symplectic geometry to statistics.
@DevinKing, Sampling with replacement means that once you select one gene, you could possibly select it again.  Sampling without replacement means that you don't put the ones sampled back and cannot sample them again.  I would suggest sampling without replacement because that is a better representation of your original data (I assume that your original group would not have had duplicates, so the samples should not).  Sampling with replacement is used in standard bootstrapping where the sample size is the same, so without replacment would not vary.
It's still bounded by (0,1), so all the above still applies.  The quasibinomial is presumably fine. The lower lines aren't exactly straight, but almost. I imagine the variable `days` has little association w/ the proportion 'alert'. I also guess there are fewer observations at later days for the light-bellied geese.
ANCOVA is necessarily *the* appropriate method: regressions (whether OLS, nonlinear, nonparametric) are also appropriate. Nice answer though.
Welcome to the site, @Nor Abd. I think this question has sufficient statistical / machine learning content to be on-topic here. But note that if you are only looking for code &/or debugging help, that would be off-topic for CV (see our [help center](http://stats.stackexchange.com/help/on-topic)). Be aware that you may get an answer w/o R code, or with pseudocode or code from another language. Some coding questions can be asked on [Stack Overflow](http://stackoverflow.com/).
I suggest you start from beta distribution and work from there. The Dirichlet for 3 is "just" a logical extension of beta, which is the Dirichlet for 2.
@Scortchi I don't have much idea about that. Could you please help by saying a little more?
I want to observe any change in the mean, variance or (shape of) distribution of the performance
Here is a good set of slides that show the exact calculation you are looking for: http://sites.stat.psu.edu/~jiali/course/stat597e/notes2/logit.pdf
Have you tried taking the logs of all your covariates (i.e., x's) and then fitting a linear model?
Just `tab2<-(tab1!=0)*1` is enough. Dimensions will survive comparison.
Do you mind elaborating on the statements "takes into account the correlation of the data" and "is scale invariant"? What pertinence do either of these things have to the question at hand?
Loss *could* be monetary, but is far more general. You could think of it as something akin to utility, but it doesn't have to be actual utility, or even very much like it. It's just some specifies measure of the 'badness' of the outcome in some sense.
Thanks for your answer! When I average the effects sizes of the subscores - how would you in this case derive the standard deviation of the averaged effect size? Just the mean of all standard deviations?
First, K-L is not a real distance, and that's a well known fact. Second, if the matrices are measured in different units, they cannot be equal.
If I understand your problem correctly, the bootstrapping approach you describe should be fine.
You've asked a very unclear question, and you certainly are not telling us the whole story. This makes it very hard to provide an answer. You would get better results if you explain what actual process is producing this data set, and what problem you are trying to solve.
Interesting idea - do you know of any R packages that would handle this? (or SPSS)
But I have not alwasys clear where is the difference between a twoppeny  "some deviation" and a little but meaningful deviation in a graph. On the other hand, normality test also take into account the number of data,  so I prefer a less subjective decision based in a p-value.
Actually not. I just want to have the end users rate some item's hotness, which I can sort from afterwards. Like 5 stars as max and 1 star as lowest. I could add the stars and divide by amount of votes...I was just wondering if there is a more efficient algorithm to deal with this stuff.
What is important mostly depends on what you're interested in finding out. For me it's usually the coefficients and their standard errors, and sometimes s. Sometimes the p-values are of interest to me as well. But other times I have interest in some particular part of the output.
The simplest thing that I can think of that tossing a coin n times and estimating the probability of a heads (denote by p). Suppose, we observe k heads. Then the probability of getting k heads is:

P (k heads in n trials) = (n, k) p^k (1-p)^(n-k)

Frequentist inference would maximize the above to arrive at an estimate of p = k / n.

Bayesian would say: Hey, I know that p ~ Beta(1,1) (which is equivalent to assuming that p is uniform on [0,1]). So, the updated inference would be:

p ~ Beta(1+k,1+n-k) and thus the bayesian estimate of p would be 

p = 1+k / (2+n)

I do not know R, sorry.
Thanks for your valuable advice.
Classical regression model. I updated the question accordingly.
I think the best way to about such a proof would be to watch the videos in the link, they are very informative and will provide a step by step explanation of the way this classification strategy works.
Good point - smoothing the data is the first problem, since I already have some methods that work for the second part. I'll give that a go and see how it does.
@Anand I know it’s a hard to follow advice, however I think you shouldn’t start by focusing on technical details but rather try to get *why* such a bound can exist... then the proof should appear easier. I tried to show you the *why* in the second part, added this morning (you need to sleep on a question like this – at least I need to). I think it’s terrible how this kind of intuitions doesn’t appear in most textbooks... even if you get the technical part, as long as you don’t have the ideas everything looks magic. Thank you and CrossV for giving me the opportunity to think to this in details!
One might argue that it increases "Type III" errors as well, which (generically and informally) are errors related to inapplicability of the underlying probability model.
This is too broad, & there are too many questions here for a simple Q&A site like CV. What you need is to take a regression course, or read a good regression textbook. If you can focus the question, we may be able to help you w/ specific issues. Regarding the results for non-numeric variables, read some of our threads on [tag:categorical-data]. For the missing data, try our threads on [tag:missing-data]. Also, you should not use stepwise selection, see [here](http://stats.stackexchange.com/a/20856/7290).
Thank you! Hope that one day I would be able to help others!
@whuber: Yes, it's the expectation of $G_{2}$ that is zero, of course. The $\\gamma_{2}$ was a relict from an earlier answer and should be $g_{2}$ (changed now); I've edited my answer quite heavily.
I have a question. Why did you say that the k-means performed poorly? I have passed your Matrix to k-means and it did perfect clustering. Did you not pass the value of k (number of clusters) to k-means?
$E(x)=\\sum_i x_ip(x_i)$
Thanks. i fixed the question.
(+1) This looks like exactly what I need.  I'm going to do a bit of verification this afternoon and then I'll hit the "accept" checkmark if everything works out.  Thank you!
Interested readers, and potential respondents, would like to know precisely what you mean by a "sample" and by the "shape" of a distribution. One reply already posted assumes a sample is a collection of $p$-tuples of real numbers and that "shape" is a similarity-invariant description of the resulting point cloud in $p$ dimensions. Many others, though, would take a "sample" to be a collection of *numbers,* its distribution to be their *empirical distribution function,* and its "shape" to be a description of the EDF up to changes of scale and location. Those are very different interpretations!
Alexis, thanks for your answer. However, and since I am applying backward stepwise regression, I already include all of the independent variables in my first model which yields the described results for the bootstrapped sample.
Kevin, it's better to split questioins into 'answerable' bites and the programming questions are probably better asked at stackoverflow or similar (though established stats.se members might disagree). But I found [this tutorial](http://www.stat.auckland.ac.nz/~paul/RGraphics/rgraphics.html) helpful for understanding base R graphics
@cafe876. Thks, let me a little time to think about it.
Similarity measure is selected mostly on the basis of theoretical/logical rationale, not empirically. Is it that you fail to work out the rationale that you go for distributional properties in order to select among measures?
Are you sure there is no zero or negative value in y ?
Great answer, thanks. I understand your answer, but `paired=FALSE` does not make sense to me in the first `wilcox.test()`. Is it required/true?
Thanks An, that will give me a good start.
My intuition says no:  I believe it is possible to construct a parametric distribution from which one can take infinitely many samples but there exist two distinct vector parameters for which the likelihood is maximized.
In this case it makes sense but most of the time M_i,j can be a rating and in that case the MSE is more useful. I'd say that the MSE is more general.
@jbowman Thanks. But if at the same time SSE/DFE also estimates sigma^2, then my question is how is SSE/DFE = (SSY - SSE)/(DFY - DEF).
You are not missing anything; your intuition is leading your astray. The mean and the median are not necessarily the same for asymmetric distributions. See Problem 4.8 in Stirzaker's _Elementary Probability_ which describes how the Royal Oak lottery of 18th century Britain exploited this fact.
This is a really wonderful answer, @gung
"Asymptotic" anything in "small samples" does not make sense: it's a contradiction in terms.  I suspect you are asking for the actual relative efficiency in small samples, period. The answer depends on the underlying distributions you are comparing and so will be complicated unless you have two specific distributions in mind. Many people choose the Normal for reference, but that might not necessarily be right for your applications.
apparently libsvm already does it, or at least some variant of the algorithm http://www.work.caltech.edu/~htlin/program/libsvm/
*I'm not sure I understand your answer...Is that correct?*

yes

The limitation with the logit is one of symmetry; going back to your example, you force the slope between 4 and 5 to be the same as the slope between 1 and 2.

If symmetry is a problem, you might want to try a complementary log-log link (not implemented in LMER).
One additional note - that "randomness" could come from pure measurement error. In the real world, nothing gets measured perfectly, so there will always be some correlation.
Thank you, comrade! I didn't know, that this probability binning is known as "Reliability Diagram".
I am not quite sure, but you seem to be implying that Spearman's $\\rho$ only applies to ordinal data. That is not true: it is a valid measure of monotonic association regardless of data coding.
It's a nice question. Presented abstractly, though, as a problem about arbitrary metric spaces, it strips away all the relationships that could be exploited to obtain efficient algorithms. For instance, $\\mathbb{F}_2^n$ with the Hamming distance enjoys a tremendous number of isometries. In the general case there's nothing to exploit, so you would just have to make some kind of exhaustive search of the balls, exploiting nothing but basic properties of a metric (symmetry and triangle inequality). What is the metric space you are *actually* dealing with in your practical problem?
Take it as accepted answer and vote it up! Help the community.
Thanks Glen ! I agree with you. Although 1 sample z test for proportions can be conducted it is difficult to justify this test
@steffen Yes, I am aware. I included in the code my invocation of t.test and I would like to know whether it is correct.
+1 too, thanks for your help.
Hi Tim, it appears that you are pleased with this answer - please consider accepting/upvoting it :)
@Glen_b This is part of a bigger problem which (ideally) requires a formal proof of the sampling distribution of $D$. This is because I would like to create test statistics for $D$ and conduct hypothesis testing.
@ryu576, I added some detail in the question.
Excellent answer - when I can +1 I will. Would this technique be called a Naive Bayes Classifier?
@wcampbell - Thanks for your answer.  The most sophisticated tool that I have to use for this is Excel (at least that I am aware of anyway)
Great! I now have the mantel statistic for various pairs of submatrices. To approximate the final correlation, do I compute the mean of these mantel statistics? Also, how do I know the final P-value for this statistic? [please excuse my ignorance]
have you looked into adapting the kNN approach of [Zhao (2000)](http://www.jstor.org/stable/3533180)?
Is there something you don't like about the solution you proposed?
Oops, my bad. I will correct
Hey Mike - welcome to the site. I'd suggest reframing your question  so that it's more statistical in tone.  Right now, it sounds like a computation problem - but what you're really asking is, "How do I predict performance for individuals who haven't taken any courses yet?"  I think you'll get better responses.
Could you expand your answer with contents from the linked article? Links may not work forever and then this answer would become less useful
I have n largest values from a sample of N values and I need to estimate the sample mean and deviation - or whatever statistic I can get. Sample size N may be either known or unknown.
+1 for being an excellent treatment of the question.
It is a daily sales data of a store. The country is US. I tried using holiday also. Still getting the same MAPE
Not much. It may be just related to the size of the time step.
*It means that The chain is not irreducible* - do you mean to say the chain is irreducible?
Hu ttnphns, thank you. What does it mean when you say "It is a test of distributional locations"? If I well understand (I'm only a user of statistics, not a statiscian), i shoudl be what I need. I have a  so called "level of compliance" to given recommendation and I want to check whether "the more you comply, the better". So, based on my interpretation of your sentence, the J-T test seems appropriate.
It usually makes a lot more sense when compiling descriptive statistics as opposed to running a linear model like you are. Sampling methods may have certain biases that would mess up your descriptive statistics, thus weights can be used to adjust. Let's say you were trying to profile your customer sales by country, but you knew your database has half as many European customers as you actually have (maybe half were deleted). So your random sample would underrepresent the Europeans by a factor of 2x. Thus you would upweigh each European customer by 2x.
There's no inherent problem with having lots of dummies - some programs have ways to handle this, for example in proc glm in SAS, there's the absorb option - which means here's a bunch of variables, don't show me their parameters. I've run models with hundreds of dummies.
@onestop There are covariates that they are interested in looking at but the actual test was only comparing the main effect.  If you would like to comment supposing that the test was based of a regression model or event assume we had time to event data to fit a Cox regression model feel free to comment.  I would love to hear your insights.  My question is addressed to the general problem and not just the specific example.
@AlecosPapadopoulos ... or I mean an event with respect to the Lebesgue measure
Censoring is perfectly ok, even though they're not "really" censored (you know that they won't develop the disease, as they're dead). Another approach is competing risks analysis, where you consider death as a competing event. If you're interested I can give you some references.
Do you know what the conditioning event is *a priori* (up to the values of the parameters)?
See http://cstheory.stackexchange.com/a/20759/1654 if you have no crossover you are missing the power of a genetic algorithm.  In your description you have no crossover (though maybe you just abstracted it away for simplicity).
Thanks for the answer and tips. For my purposes, I think this method will be sufficient.
I come from a 100% frequentist background in econometrics and am currently trying to get a grip on the Bayesian approach myself. I read on this site that many consider it intuitive. I disagree. It seems nobody can really define what it is and what is doing differently. You need some grounding in the terminology of information science to even be able to read the wikipedia article. It goes so far that it is so unclear defined that I literally don't know if I am doing Bayesian stuff already (as I clearly learned some of it in basic stats). There should be a book for people like me. I feel so dumb
@rrenaud For zero the error is not infinite (only Taylor expansion does not work here). It should be sth like $\\log(N)/N$ (or maybe even less, for this case, e.g. $\\log(\\min(n,N))/\\min(n,N)$, but I'm not sure). Of course, the problem arrises when you have a lot $p_i=0$ entries. In particular when you have infinite number of states $n=\\infty$ and finite number of total counts $N$, the error is infinite (unless you make some more assumptions on the distribution of probabilities).
@PeterFlom Because only `TRUE` and `FALSE` are reserved keywords (see `?reserved`). `F` and `T` are just variables assigned to your convenience. So you can easily use the following construction to produce horrible things: `T <- FALSE; F <- TRUE`.
You may be interested in reading this thread: [What is the difference between doing linear regression on y with x versus x with y](http://stats.stackexchange.com/questions/22718/).
You description seems a little ambiguous (or maybe it's just my own confusion). Can you show an example, either a histogram (preferably with a good number of bins) or better, a normal Q-Qplot, which might give some better sense of what you're asking. Note that there are many continuous distribution on the positive half-line.
@whuber if zip code is being used as a marker for distance from something or socioeconomic status (1) zip--> distance  (2) zip--> SES index. So only possible if you are using zip code was measured because it was thought to be a reasonable surrogate for distance or SES.
@Chris: you can change the maximum allowed delay (as in the Sakoe/Chiba bands) as you like
@PeterFlom In the case of the [Cauchy distribution](http://127.0.0.1:25031/library/stats/html/Cauchy.html) it is explicit that m and s are the location and scale. I agree the notation m and s suggests that they represent the mean and standard deviation, respectively. But this may just be a simplification of `\\mu` and `\\sigma` as well. +1 long ago, by the way.
Please either update your question, or post a new one mentioning this one. Answers are for answering. Judging by your text, 2 separate questions can be asked.
"*seems i can't decomposite it to 2 normal distribution, as 2 independent normal distribution's sum is still a normal distribution*" -- you're looking at something more like a mixture, not a convolution. Each change is not a sum of a deposit and a withdrawal, it's either one or the other.
Why wouldn't you just treat the 5 categories as 5 separate features?
If your dataset is balanced, and the output of the model still ends up as balanced (which in most cases I think it would), then of course there's not anything bad with accuracy.

But it's always good to report something like f-measure so people don't have to read the fine print of your methods to convince themselves you don't have an issue with imbalanced data.
@JohnK You're right, it should be clarified. The point was that the data are paired (before-after on each person); the OP seems to have realized this but to my mind didn't clearly articulate what the issue was. A paired t-test is merely a one sample test on the differences; the intent of my comment was to distinguish it from an unpaired test, which is what people mean when they say 'two-sample t-test' without further adjectives. Most of the OP's reasoning was correct, but needed to hone in on what I think is central to clearly stating what's happening there.
It would be nice to have a specific example you have seen or been confused about. That might help guide people better in their answers. As it stands, the question is a little on the vague side. Cheers.
+1, thanks for the additional contribution, @ttnphns. This is turning from what I originally thought was a fairly simple question into a fabulously informative thread.
Compare how?  Optimize what?  Loess in particular was designed as a visual aid and is "optimal" when it looks best.  As for the standard method of optimizing a smoothing spline, `mgcv` is the only one that comes to mind; it offers Generalized Cross Validation (GCV) and REML to optimize the curvature.  You could define your own loss function and use cross-validation to test different smoothers at different tuning values to see what is "optimal."  Be warned that using cross-validation to both tune and choose between multiple methods will overfit your training data without complicated nesting.
Is there a particular language you would *like* it in?
Could you expand on how would bootstrap provide a more interesting approach for factor analysis?
See `import.csv` to import your file into R.  The L means only that it's an integer; `import.csv` will automatically decide how best to store it (the defaults can be changed if you like). mpitkas used `dput` to output the data in a way that could easily be used by others, but that's usually something you would do only to help others recreate your code.
Can you report: summary(name_data_base\\$NumericVariable) and summary(name_data_base\\$FinalW)?? I just want to make sure everything looks like it is class numeric instead of a factor or something weird.
o|w∈f(o) this is basically for the summation (it should have shown under the sum symbol,sorry!) this depicts that the sum operation is done over all the words(w) predicted by the spell checker for the incorrect ocr word. |Docr| most probably represents the magnitude ie. the word length of the ocr document.@whuber