tttjjj/elt_bench huggingface dataset:
@Memming Edited my post. What do you mean by "How many do you have?"?
I wouldn't read too much into this. I haven't noticed an aversion among Bayesians to talking about statistical modeling. My impression is that probabilistic modeling is the more likely phrase if authors think of themselves as probabilists and have little or no concern with estimation or inference using real data, and conversely for statisticians and data-sensitive texts. Thus a probability-oriented text might emphasise the theory of Markov chains, Markov processes, etc., etc. but say little about confrontations with real data.
P.S. The similar controversy about (actually additional, not intrinsic) assumptions exist over Mann-Whitney or Kruskal-Wallis tests: see e.g. http://stats.stackexchange.com/q/76059/3277 with all the comments and especially @Glen_b answer. And, btw, Mann-Whitney's H0 is not "distributions are the same" but rather that there's no stochastical dominance (aka the so called "location" is the same).
See [What is a good resource on table design](http://stats.stackexchange.com/q/3542/1036) and [Some notes on making effective tables](http://stats.blogoverflow.com/2012/02/some-notes-on-making-effective-tables/) for references of interest. IMO for presentations you want to focus information much more than typical. Tables with more than 4~5 rows of information are too difficult to read in that venue.
+1, this is a nice answer: simple, clean & provides some guidance on *how* to accomplish this task.  NB, in addition to @Procrastinator's comment, the normality of the errors only matter for the validity of the statistical tests. It appears the OP isn't interested in that, & you haven't shown it in your code (appropriately), so it seems to be wholly irrelevant here. Even homoscedasticity probably doesn't matter, as that pertains to efficiency / the variance of the sampling distributions of the betas, but they are unbiased nonetheless.
+1 for addressing the apparent (rather than the stated) question, which is much more to the point.
Neither the X nor the Y variables need to be normally distributed (see here:[what-if-residuals-are-normally-distributed-but-y-is-not](http://stats.stackexchange.com/questions/12262/)). If your response variable is *ordinal* you should probably look into *ordinal logistic regression*; there are standard tests for this.
I used Levene's Test for Homogeneity of Variance. The result on my data is significant, which means the variances are significantly different.
In the options of Post-Hoc, down there are some options that you can selected as Games-Howell.
@user333 Wouldn't you agree that the information that the "scale of the measurements is very different" is an important information? You might want to update your question.
Let's continue in chat. Might get to this tomorrow.
Is there a statistical question here?  Perhaps this should be asked on math.SE?
Thank you for your answer. So if I am interested in the total change from the baseline to "married and black", I should consider the whole lot of coefficients and not only MARBLACK. That is what I thought.
Indeed, what even are the *data*?  What does it mean to say that "memory is lost"?  What actually gets recorded once the experiment is finished?
Dear Peter, Thanks you very much for your help.
A special note to get @jbowman 's attention as I can't add a comment to the question (too little rep).
Please check my editing of your question.
Ah, I see. I was also wondering how come k-means is so widely applied-for example, in customer profiling, where all sorts of categorical, binary, and continuous variables occur, and it is not clear what it means to compute Euclidean distance between variables such as amount of phone usage, age, salary, etc. Neither is it certain that such data will not have outliers, different shapes, densities, etc.

But now as I understand k-means more, I will be more careful in my analysis, hopefully, and also look for algorithms detecting structure (in my understanding hierarchical clustering)...Thank you!
I think we'll need some more information about what you're really interested in to be helpful. Here are some initial comments: (**1**) You've written the *squared* Frobenius norm on the RHS of your definition of $\\|A\\|$. (**2**) Consider the diagonal matrices. That should tell you everything you need to know about possible values that $\\mathrm{RSE}$ can take on (knowing nothing further about the structure of your problem). (**3**) Answering your question (b) will require knowing more about your interests and application.
Thanks. Suppose there are two big trucks in two different countries. Now big trucks tend to carry big loads. If we added a positive number to the stack every time each truck carried a big load we'd end up having to say that the behaviour of the two trucks was very similar. But actually the size of the load carried by one truck isn't related to the size of the load carried by the other at any particular time. They just happen to both be big trucks. So our measure of similarity wouldn't be useful. That's why we have to think about 'bigger than average'.
For a plot you can do whatever you want, of course.  As whuber points out, it's not at all clear that the relationship is heteroskedastic.  But even if it is, if there isn't much heteroskedasticity, a linear fit will be quite good, especially with so much data.  Otherwise, there are various methods you can use to (largely) correct for heteroskedasticity, e.g., generalized linear models.
What do you mean by "the general formula"? These lectures notes might be helpful: http://socserv.socsci.mcmaster.ca/jfox/Courses/soc740/lecture-3-notes.pdf
Do your observations represent the number of events over the same number of trials?
Welcome to the site, @Kevin Krumwiede. Your last sentence is ambiguous. It seems like you are making a common mistake that conflates a p-value of <.05 (eg) w/ a 95% probability that the null hypothesis is false.
I do not get the full picture of your question, as k-means is designed for Euclidean distance. Using it with e.g. pearson or covariance may in the worst case prevent k-means from converging.
Not an answer but one thing to keep in mind is that higher order moments require _a lot_ more observations to get the first sig-fig.
I think questions about how to implement a statistical analysis in SAS/R/whatever are ok. But, purely interface questions are off-topic.
@Harry That all depends on your null hypothesis. If your null hypothesis is to be $E[T]<2000$, if you reject, it is type 1 error, if you accept, it is type 2 error.
It's based on generalizing [a later section in the same Wikipedia entry](http://en.wikipedia.org/wiki/Sample_size_determination#By_cumulative_distribution_function) to the case of unknown $\\sigma$. If you follow through that you'll see there's no conceptual or practical problem for $n_1\\ne n_2$, apart from the need for much more extensive tables.
If you are dealing with time series, you should should test/model autocorrelation of residuals. This is currently not implemented in `lme4::lmer`, which is one of the main reasons why I still often use `nlme:lme`. I recommend reading "Mixed Effects Models and Extensions in Ecology with R" by Zuur et al., which is a nice introduction for beginners.
Thanks for the helpful response.
@Mehper: Google "TeX Manual" and take your pick.  I like the "gentle introduction" because it contains useful, readable tables: http://www.tex.ac.uk/tex-archive/info/gentle/gentle.pdf
I think your question was [asked before](http://stats.stackexchange.com/questions/16665/how-can-i-get-a-significant-overall-anova-but-no-significant-pairwise-difference).
Okay, I think that the difference in what I was thinking is that, rather than showing the prior to illustrate both $M_0$ and $T_0$, I was thinking about presenting the posterior of $M_0$ and $T_0$, even something like $\\mathbf{N}(M_0,T_0)$, but replace the variable names by figures.
Do you have a suggestion on any ranking methodology to rank the above between 1 to 5 say? The parameters vary, some are in seconds, while some are number of users etc.
Actually it is probably simpler than the above comment, just create the var1*var2 data frame, then create the var1*var3 data frame, sort both by var1 and use `cbind` to attach the var3 column from the 2nd df to the var1*var2 data frame.  If the marginals of var1 match (which they should if they came from the same original data set) then everything will now work out (possibly need to randomize df2 before sorting).
Please write something more about your problem -- this way this question will be closed.
Actually, as I has edited, $t_r^'$ should be defined as "the last OUT event prior to $t_r$." It means there is no other OUT event between $t_r^'$ and $t_r$. @MichaelChernick
A useful page is http://stats.stackexchange.com/questions/6498/seeking-certain-type-of-arima-explanation
Hi Stephane thanks for your response ! So you mean that E[NP] will be NP(1,1,1) + NP(1,1,0)+..................+NP(0,0,1) i.e. sum of 8  combinations ? If yes what is the theoretical backing/grounding for this ?
For those puzzled, specific syntax here is Stata syntax. Underlying principle is generic
I think your suggestion is still well-suited for my question. I was asking about choosing the maximum number of feature that has lower chance of overfitting the data, but I forgot to mention them. Also, now I realize that it depends on what algorithm I use, and characteristic of data.
Exactly, the weights become less arbitrary and you can look at the factor loadings which question explains a lot of variation.
Find maximum or minimum of a given function. I have \\nabla f. Hessian is [-2, -2; -2, -2]
The third window is arguably not so much noise as a relatively low-noise image. The distinction between the first two and the third one many be important - e.g. it may be necessary to distinguish "noise" from "high-information image that isn't text" in a classifier (e.g, it may be relatively easy to spot "just-noise" vs "there's information here" and it may take quite different criteria to distinguish "image" from "writing").
[Censored regression is different](http://stats.stackexchange.com/a/49456/919) from regression with a dependent variable that is confined to a fixed known range.  These data are not censored and censored regression will do nothing different with them than ordinary regression.
$R^{2}$ and $\\beta$ are dependent in that if $\\beta$ is zero, then $R$ is zero (leaving out nuances of $\\hat{\\beta}$ and "near zero" due to space), and if $\\beta$ is not zero, then $R^{2}$ is likewise not zero. If there *is not* a linear association, then a linear function of $X$ cannot "explain" *any* of the variation in *Y*; if there *is* a linear association then $X$ m must "explain" *some* variation in *Y*. (Also leaving out nuances about "approximately linear"; also leaving out the unidirectionality of OLS versus the bidirectionality of $r$).
From my neural network experience, could you make the 6 values into binary values such as 1000000, 0100000, 0010000, 0001000, etc and then use adaboost. just a thought.
I modified the answer, and included the name of the function. I don't write Fortran (I used f2c to use the code of Alan Genz), so no example :-)
@Procrastinator thanks for the link! I had been using the unbiased estimator, but the KRS estimator helps me somewhat. The MLE is also easy to compute by brute force.
Btw, none of these methods answer the question about normality of the errors.
Amen to that! `comment()` et al are handy for ad hoc notes and info, but a proper system is required for larger scale data handling. Interestingly we are having to address this now within the research group and consultancy I work for in regard to our chemistry data and needing to get it into a proper database.
@whuber: --this is a one sided filter: as far as i understood the option "online" makes sure it doesn't use  data from $t+i$, $i>0$ at time $t$. More generally, I agree with you: I also tough of asking the OP what was the end purpose (is he, for example, interested in the value of an  ar coefficient for a given lag)?
please look at feedback below. thank you.
I wish that sediment were like this! Instead, unless it is purposely well mixed, it tends to be well stratified and heterogeneous. How it is sampled can make a *tremendous* difference.  (Think about it: a fundamental archeological technique is to use vertical position within sedimentary strata to data artifacts: that alone shows that sediment cannot validly be considered a homogeneous medium for archeological purposes.)
I have seen--literally--many thousands of datasets in my career (and virtually none of them were collected according to a design reviewed by any statistician). Most of those were collected for formal purposes, such as satisfying regulatory requirements. I cannot recall *a single one* that did not have some design-related problems (although sometimes these were minor). This is not to say the datasets were useless or "dead": but in almost all cases my task was (to continue the medical analogy) first to resuscitate the dataset and then to apply it to its intended purpose, if at all possible.
There some (answered) questions dealing with this issue, such as http://stats.stackexchange.com/questions/5502/factor-scores-from-discrete-ordinal-responses or http://stats.stackexchange.com/questions/11899/factor-analysis-on-mixed-continuous-ordinal-nominal-data
The early answers to this thread are *so* off-topic I think this should be closed.  SE is not the place for lists of jokes, etc. (even though we all enjoy them.) See guideline #3 at http://blog.stackoverflow.com/2010/01/stack-overflow-where-we-hate-fun/, for example.  I anticipate complaints, though, so I'm holding back. But if the quality doesn't go up quickly, I won't hesitate to put in a close vote!
When you say "the overall decision", I think you mean "a decision to reject the null hypothesis or not". There are other decisions, for example involving a quantative loss function, which might be made using a confidence interval but not a $p$ value.
Now you should consider whether your statement is true or not. :)
+1, I was going to give as an answer the 2 point. Kudos for including the other 2.
Thank you for including mouseover text.
This was advice emailed to me by an academic (not an academic in mathematics or statistics). I think his reasoning related to the fact that my sample size is rather small and/or the fact that I'm sampling probabilities, and I was interested to see if any respondents cited those as reasons for using the median instead of the mean. Are they in fact good reasons?
That's *NOT* an inequality, it's an equality. There's a clear "$=$" sign between $\\mathbb{E}X$ and the $\\int$.
"*would it not be more useful to perform a one-way ANOVA*" -- that kind of depends on what you're interested in knowing. I'd think that "examining the relationship" would most usefully be achieved by choosing appropriate displays and perhaps via some form of estimation. It would seem prudent to consider the possibility that the relationship is not monotonic, and your suggestion is one way to look at that.
The KM is cumprod(1-D[t]/S[t]), while the cumulative hazard Lambda[t] is cumsum(D[t]/S[t]) and the Breslow estimator  cumprod( exp(-Lambda[t]) ). You ask "LTV/CLV?". I say, ... huh?
Thanks for providing the reasoning. I had a problem with non-identically-distributed variables, but the minimum logic still applied well :)
Do you have a R example on prostate data like in the paper i mentionned ?
What exactly are you doing to normalize? Just dividing by the smaller of the two numbers?
A common mistake. Density is not probability. The issue here effectively duplicates [this](http://stats.stackexchange.com/questions/4220/a-probability-distribution-value-exceeding-1-is-ok) or [this](http://stats.stackexchange.com/questions/9427/interpreting-gaussian-probabilities-greater-than-1) and a number of others.
@Dilip I don't think there's anything wrong with your answer, and I think there's a lot (especially the last bit) which are quite right (+1).  Absent a solution manual (which I don't think even exists), we can never know what was on Jaynes' mind.  But I think you're right that the final answer should have conditioning all the way through, and what would seem to be the natural conditioning event would be: $A \\cup B$.
@Zen Okay let's take that example. Why collect any data at all? We know g.  So we can numerically integrate it to determine c to whatever level of accuracy we desire without having to estimate anything!  The assumption that we can't compute c which means that even though we know g as a function of x we can't integrate it!  I think his example is weak and so is the argument and I like his books generally speaking.
This question appears to be off-topic because it is about how to use software.
To expand slightly on the answer (+1), your typed-in coefficients have only 4 and 5 digits of accuracy, but R stores them to a lot more; thus, you cannot expect the residuals to match beyond about the 5th decimal place.
I mean `stat::kmeans` might not be an option if you're using the `fpc` package for some functionality that is only in that package. If you paste in the output from `dput(mat2)` up there in your question then others can work with it also.
Aren't logit models heteroscedastic by definition? You explicitly model the mean-variance relationship through the logit link and binomial variance function.
@Bitwise, what would you do in this case then?
@COOLSerdash - since you've answered the question, you may as well move it to an answer and get credit for it!
It depends fundamentally on *why* one is fitting or assuming a distribution and what it is intended to represent. We field many questions on this site where it appears people feel they have to fit a distribution to data or derived quantities (like regression residuals) when in fact the exercise is pointless (or worse, deceptive) as far as solving the statistical problems they *really* have is concerned. Could you perhaps clarify the kinds of cases you have in mind?
(continued) Then I would get credit but not help the person anymore than I would by referring him to the source. This is much like some questions where it is easy to google for the answer. I do it and get credit or the OP could do it himself and be less dependent on us. A good thing I think although probably many members here would disagree.
It's not *quite* what you're asking, but - mostly in the case of regression - [optimal design](http://en.wikipedia.org/wiki/Optimal_design) addresses the issue of choosing the values of the IVs to optimize some outcome.
Thanks. Can you post the figure here? I don't have account to download the PDF file.
+1 A great question to have here. You can't define a uniform distribution over the rationals, not even restricted to [0,1], nor for any other countably infinite set. I wrote a little discussion about this once, I'll see if I can dig it up and look at it, but it probably doesn't add anything useful to the answer you have.
Many solutions and examples appear on the *Mathematica* site.  Start with a [search](http://mathematica.stackexchange.com/search?q=plotting+dimensions) and then explore!
@Wolfgang, thank you! I think I'll do as you suggested. Although I still do not understand how I've got such results (they seem strange for me). And also the difference between lme and lmer is rather big. I suppose all this happened because I have small N (37).
Sample size informs us about the *power* of any test but it is not needed to conduct a valid test.
@Pari - try looking up Paired-Sample or Dependent-Groups Proportions Tests (*Z*-Tests).
No i do not know how to use matlab debugger. I commented a few lines and added printf statements to check up till where the code is running fine. I believe after [m,n] is computed, there is an infinite loop for data. 
Can the infinite loop be because of the huge data set?
If the resulting scale is not an _interval scale_, that is, the meaning of a particular gap is not the same across the scale (e.g. is the difference between 1 and 3 the same as the difference between 101 and 103?) then treating it as a continuous measurement does not make any sense. So, this really translates to a question about whether or not you can make a rational argument that your outcome is an interval scale - if so, you can treat as you would any other continuous variable.
Seems to me that your data is categorical now. I'm not sure if methods that work with censored data could buy you anything here. Do you know the actual range of the temperature? If it was, say 0-50, then T=2 is obviously a much larger bucket than the other two.
@Jeromy:> I'm afraid not. On the other hand, i would pay attention to the issue of sample size and model complexity as 200 observations ain't a lot.
My study leader and I had this discussion, and I posted a question related to this discussion here http://stats.stackexchange.com/questions/43559/is-discriminant-analysis-supervised-learning. I came to understand that model fitting, training, estimation and learning just mean the application of some optimization method to find the  best parameters that defines a model of a specific form.
@Guillaume I am interested in both situations - imputing one when the other is missing, and also using both of them to impute the other covariates. That's where my concern about collinearity comes from)
(...) In sum, if you want to show that parental ratings are more reliable than children ones, then using ICC is fine; if, on the other hand, you want to study how parental ratings relate to children ratings, then you might resort to other kind of analysis (precisely, analysis of dyadic data).
I'm not sure I understand the question... Also, if your question is about a plot, you should show one.
In terms of prerequisites for Coursera classes, that info should be available somewhere in their materials. Outside of their classes / more generally, the question of what math you need for stat / ML / DM strikes me as a duplicate. There are several threads on CV that cover this material, including: [is-a-strong-background-in-maths-a-requisite-for-ml](http://stats.stackexchange.com/questions/40808/), & [skills-coursework-needed-to-be-a-data-analyst](http://stats.stackexchange.com/questions/30129/) (perhaps among others).
+1, this is a really nice contribution; short, simple, clear.
You can't readily assess normality in the presence of heteroskedasticity, since the hetero makes the residuals a mixture of errors with different distributions. As for whether hetero matters much depends on what he's using it for, and in exactly what manner. If he wants confidence intervals around predicted means, for example, he has a serious problem.
$h_1$ and $h_2$ are appropriate scalar functions, $f$ is a pdf, and $R$ is the correlation function, $X$ and $Y$ are random variables, $\\mathbf{x}$ and $\\mathbf{y}$ are vectors.
I agree with ERM, this is what I had in mind. The parallel with a decision tree it seems however a weaker one. There are some notable differences: a decision tree implies an automatic decision on how to split the input space, while what I think of is to propose the split as a learning bias, or as an apriori knowledge. Also how regions are split could by arbitrary (I might choose any boolean expression). I give a point for pointing out the ERM and thanks for answer
"The purpose of performing PCA is to reduce the dimensions of your variables." No, this is just one way PCA is used. This is not the only thing that PCA is good for.
Any thoughts on this one? I will appreciate any pointers. Thanks.
possible duplicate of [What is the meaning of the semicolon in $f(x;\\Theta)$?](http://stats.stackexchange.com/questions/30825/what-is-the-meaning-of-the-semicolon-in-fx-theta)
In my experience the required sample size for the $t$ distribution to be accurate is often larger than the sample size at hand.  The Wilcoxon signed-rank test is extremely efficient as you said, and it is robust, so I almost always prefer it over the $t$ test.
@Alexis "nominal" means two-level? I can't find any support for that definition. Is there a particular context for it?
Thanks a lot!  I found your answer very useful!
+1 Nice example of an appropriate analysis with the OP's data.