tttjjj/elt_bench huggingface dataset:
@static_rtti: Your comment is a little puzzling. With one or two points you can't say anything about smoothness or linearity, by definition. Starting at three points, you can talk about it, but your answer won't have much meaning. The more points you have in your time series, the more meaningful "linear" or "smooth" will be.
-1 As it stands, this answer is not correct. The correct answer was provided by @TooTone.
I fiddled with it a bit, the restriction is that you have an event sequence and the package has fewer options for working with that kind of data. Not sure what to suggest, specifically.
Note that the new output you posted suggests that some other variables might be involved as well (or that there is another issue I didn't see) because otherwise you would expect the SE to be much lower when only one of the two variables are included.
Frankly, I don't think there is any justifiable way to compute the standard errors. The bootstrap appears to be a commonly used method, but Abadie and Imbens (2008 Econometrica) paper demonstrated the problems the bootstrap runs into. For a method as complicated as PS matching, I am not entirely sure 100 is a sufficiently large sample size to give you the asymptotically normal distribution.
Weight recorded in micrograms is not going to be categorical data, as far as I can see; do you mean that *location* is categorical?
Identifiability won't fall out of a blue sky, so don't expect there will be any kind of a magic trick to render your structural form identified. You do need the excluded exogenous variables here to make any sense out of this. May be assuming zero correlations of $u$, $v$ and $w$ could help a tiny bit, but most of the identification should come from the $X$'s.
+1. Is the ks.test the best test for testing goodness of fit in such situations? I think you also answered one question of [mine](http://stats.stackexchange.com/questions/68700/test-for-measuring-pdf-goodness-of-fit).
LDA means linear discriminant analysis to many people here. Important to spell out what you mean. Locally famous terminology is not necessarily globally familiar.
If you sample points from either normal distribution, you get points on the Perikymata-axis rather than on the 2-dimensional area. Furthermore, the green zone is infinitely wide, so all values sampled from either distribution are under the green zone, so in this sense the probability would be 1.
Dear Dilip,

I did the same to prove f(x) is a PMF. But how m I suppose to find out the $\\mathbb Var[X]$ as the result I got is different from with the $\\mathbb E[X]$
Efficient in the meaning which are the most used or popular nowadays. Can you list at least few good algorithm candidates to dig into?
Ok, thanks for the explanation. Somehow when I add the personality as IV, it splits all the possible scores participants gave into separate variables instead of just taking the range of each personality trait. So for example, it analyzes extraversion score 1.25 separately, 1.50, 1,63 and so forth instead of taking everything as one
Don't know, but I'm pretty sure that everyone doing machine learning should learn statistics.
Normality of the residuals can support inferences about confidence intervals, but it is contentious to suggest that it is essential for using regression to make predictions. There are some relevant answers on http://stats.stackexchange.com/questions/31807/does-the-non-normality-matter-in-using-regression-for-prediction?rq=1
It's nice to see this simple case explained.  Did you notice, though, that the question specifically asks about *multivariate* regression?  It's not so easy to update the denominator of $\\beta$ in that case!
See the appendix on assumptions of PCA in [this paper](http://support.sas.com/publishing/pubcat/chaps/55129.pdf).
I think you also might be misinterpreting what your professor meant by "better," since you can always compare different models, linear or otherwise, based on their prediction accuracy to held out (or bootstrap sampled) data. You should definitely ask your professor for clarification.
Sorry, i'm not sure how does the multilevel analysis help in this situation. It's a bit thought to learn it alone i guess. I don't want to make average evaporation rate from all 32 test objects so i don't see where is the structure in this. I could as well take only one material, then I would have 4 Trendlines and an average of their slopes. Its just a case of multiple errors adding on top of each other.
aside: Anyone interested in this area might take a look at [Uplift modelling](http://en.wikipedia.org/wiki/Uplift_modelling)
great thanks to your answer. So if I understand it correctly, you mean that if the h most central observations are distributed symmetrically around a center, then the (Fast)-MCD is more likely to work; Otherwise, the (Fast)-MCD is more likely to fail, right?
I'm sure you're on the right track here, but I don't see understand why test and ID aren't treated the same way.  Can you clarify?
Do you mean that Excel doesn't recognize that your data are numerical? If you look at the cell format, is it defined as numbers? Just to make sure I understand your comment.
Thanks - Once the Markov chain stabilizes, does it remain stable? How does this related to the notion of mixing-time? Is it the same?
Stata is it? might worth adding a label or at least state it in the question.
Please provide more information. If the link doesn't work anymore, the answer should still be helpful.
I meant $x1 \\mapsto x1+ c_1$ where $c_1$ is a shift for every single component of $x_1$ and same for $x_2$
This graphic presents an interesting example of the effect of a poor choice of visualization: by shrinking the aspect ratio and extending the y-axis more than twice as far as it need be, the software has automatically suppressed the visual impression of any vertical scatter, making it difficult for the viewer to see much of anything. This is why a good exploration, although guided by graphical representation, must (a) use suitable methods of visualization that *reveal*, rather than suppress, data behavior, and (b) support them with additional analyses (such as shown in @Glen_b's post).
You're welcome. I'm glad I was able to help.
It would seem more useful to predict exact gestation time and then bin it at the end if you really want to. As for including college as an independent variable, the standard ways to start would be to include it as either a fixed or random effect. Any introductory book on linear models will help with that.
(2) You are right, I forgot about the 2 residual variances. By default the two of them are constrained to equality, and I don't think it matters much whether you leave you the `/REPEATED` line or not. You can look in the help files of SPSS for possible structures (DIAG, VC, AR1, ...). (3) You should treat time as a continuous variable and not a categorical variable (i.e. precede it with `WITH` and not `BY`), because in "real life" time is continuous, not separated into categories.
Yes approach 1 is the right one, and correspond to what Kaufman, Leonard, and Peter J. Rousseeuw are saying in the paragraphs I quoted in the answer. Approach 2 would be useless as the standardization removes the weights :)
@Tommaso: The problem may come from assuming independence and conditional independence between the symptoms, which if related to the disease is unlikely.
You forgot to say how you understood it and why you think cosine is good here.
@PeterFlom The issue is that the data set contains multiple records per  phone number , where the independent variable is the number of attempts in the past 60 days. Since the data covers a period much larger than 60 days, the variable "resets". I will post an example.
(What makes this question `multivariate analysis`?) The question is a little short on context to be giving an interpretation outside simply recounting the definitions of the terms. You can search for the meaning of terms like 'standard deviation' and 'perception' as easily as I could to give an answer.
Generally speaking I do not think that because you are capable of detecting a small difference through statistical analysis that it makes it worthwhile.  The word statistical significance just means that saying that the difference is at least x has a type I error of say 0.05.  It doesn't make x important.  What is a practical meaningful difference is a question for the investigator.  If you were able to collect a very large sample size you may be able to detect a small difference x that is very unimportant.
This does not really add to other answers to other threads, cited in comments on the question, especially as you (1) say in effect that's the definition (2) seem to confuse modulus and absolute value (3) end by guessing what the reason is.
@andrewniesen Not quite. It's true that a distribution with the same mean and lower standard deviation would be skinnier, but that won't be the case with these restrictions. The variance of a binomial distribution is $np(1-p)$. Assuming you keep $p$ equal to $\\frac{3}{n}$, which you must to keep the expected number of successes at 3, variance approaches 3 as $n$ grows. To see how this would look, try plotting a Poisson distribution with $\\lambda = 3$. (The Poisson can be thought of as the limit of the binomial as $n$ grows, $p$ shrinks, and $np$ remains constant.)
We've actually counteracted the issue of weeks not lining up by munging the weeks to line up. For example, this year there would actually be an additional week so the first week's information includes 8 weekdays instead of 5. It's not a perfect system but the holiday weeks should match up, the final week should roughly correspond every year, and so on.

What I'm interested in is, is there a good method to build a model based on the trends from previous years to apply to this year's data?
You need to say more about the context and the assumptions. Non-trivial 100% confidence intervals are usually impossible to achieve except when sampling from finite populations.
@AbhimanyuArora 's answer is the reason why p-value is more popular. Let's say that a reader wants to know if $H_0$ is still rejected under the 1%-rule, he will have to calculate another rejection region. With the p-value reported, he'll immediately know this.
1. This way you are predicting y2..yn-1 exactly, aren't you? 2. I think you are confusing things here. You have many variables as it stands, not one. If it is same thing measured at different times, then you need only to predict yn from previous y's. Clarify the question, now it contradicts itself.
The mean of a random variable is a *number* and therefore has no covariance.  It sounds like you might be asking about the difference between the distribution of a random variable and the distribution of the mean of a sample from that variable, but it is hard to tell: perhaps you could edit this question to clarify what you are looking for.  PCA, by the way, does not model the data as realizations of random variables: it is a descriptive procedure.
@suncoolsu, you are welcome.
This question appears to be off-topic because it is about rudimentary data cleaning and organization, not statistics.
The general rules on interpreting the observations and variables are summed, e.g, in http://onlinelibrary.wiley.com/doi/10.1002/9780470238004.app2/pdf. The interpretation depend on how the actual analysis was done, mainly whether the variables and observation were standardized before the analysis or not.
Pardon for asking, what define "dimensionality" in your case? And what the characteristics are you going to cluster snippets by? Similarity between snippets - then what similarity?
Can you tell us a bit about how the quantity ϕ comes about?
Presumably this is the PDF, because if it were used as a CDF your question would be trivially easy.  However, you need to tell us its domain, because its integral diverges.
+1 Nice summary.  I like that you point out the role of the loss function first.
This question is probably offtopic. Note that R is a programming language and you can code any algorithm you want. It would help a lot if you define a formula of what is a rolling annual return and be more precise of what kind of data you have (daily, hourly, equally spaced, etc.)
I've done something in the past similar, that is coloring cells, to produce trees (similar to the sort you would need for dendrograms). See page 7 [here](http://www.criminaljustice.ny.gov/crimnet/ojsa/2011-hate-crime.pdf). I've been in this boat before, although not quite as strict (some upper management only wanted Excel graphs because *they liked the way Excel graphs looked*). Good luck.
Actually they are not clustered very well. And if I perform ANOVA approximately no significant gene I would get. I think it should be because of fewer number of replicates.
@subhash: CV's for people to help each other learn statistics: there's no mechanism for senior statisticians to give their stamp of approval to answers, & little desire for one. Think for yourself: read the references, mull over what they say, check other sources, ask further questions on points you still don't understand. Accept answers that help you learn. Note also that an answer can be perfectly correct, yet your own understanding of its implications be dangerously flawed.
(+1): Or see here (shameless plug): [What statistical technique would be appropriate for optimising the weights?](http://stats.stackexchange.com/questions/7869/what-statistical-technique-would-be-appropriate-for-optimising-the-weights/7879#7879)
And see [here](http://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression) re separation. If you're using maximum likelihood parameter estimates & you're judging significance by Wald standard errors **they're badly wrong** when you have separation.
Are your observations out of sync? How come are the lenghts of your vectors different? (e.g. `list( c("a","b","c") ,c("a","b","c","c") )` )
Thank you for your answer. But I have no idea what $\\sigma(Y)$ and $A\\in\\sigma(Y):EX1_A=EZ1_A$ mean. Where can I learn more about them?
Thank you for the input. Ultimately, all I want from this analysis is to have some quantitative measure of the similarity of some species compared to others (I have two species which just based on gestalt appearance look like another closely related species, but genetically appear similar to a different species, suggesting ancient hybridization). The main point of this research question is to investigate the genetics of the group, and this morphological analysis will simply add to the whole biological story. Would this multidimensional scaling lead to visualization similar to PCA?
@Bill: That is basically how I'm dealing with it -- "Bob always has an A; that's just how he is, nothing to see here". My ultimate goal is to incorporate more information about how the sites (columns) and individuals (rows) relate to each other... but I feel like this adjustment is a major sticking point.
@Aaron Thanks for your answer. Perhaps you have the time to look at my follow-up question: http://stats.stackexchange.com/q/23874/442
It appears that advertising *lags* sales by about a month. A plausible model is that advertising is *reacting* to sales rather than (directly) driving it.
How many independent variables do you have?
@whuber Thanks for the reply. I do not choose the middle of the time as the origin because I need the origin to be the minimum of time (in other words, the starting point of time in the actual data), which is why there is the issue of collinearity. Regarding centering: you said that subtracting their means should be fine, does this mean that I could also subtract their respective minimums - since I need the intercept to be the mean of Y at the starting point in time. Thanks again for the help!
Could you indicate how a single number like $\\mu$ is presumed capable of expressing an entire "spatial distribution"?  One would expect an error map rather than a single value.
Hi user62117 - @whuber won't necessarily see the question in your comment. If you put "@whuber" in it, the system will notify whuber that you are responding. (Your comment is too old to edit though, you'd have to make a new one.)
@Frank, I suppose it depends on what type of models you are running, but standardization of predictor variables is often useful.  Centering them means that the intercept becomes interpretable as the mean predicted outcome and the relative importance of different predictors becomes more obvious.  I usually leave binary predictors alone, but sometimes other scaling options are worth considering.  Finally, in some cases having predictors with wildly different standard deviations can lead to computational/convergence problems.
One easy way to tell that the recursive partitioning does not fit the data is if it categorizes a continuous variable that has a smooth relationship with the outcome (i.e., the variable is not calendar or elapsed time).
AFAIK "dependent" and "independent" have **no meaning** when talking about line equation.
@ Matt: Maybe they should update that help file and say something like "this returns the sqrt of var" ?
In your case (if I have done the calculations correctly!), $\\sigma^2(\\alpha,F)=\\frac{1}{2\\alpha-1}+\\frac{2(\\pi \\alpha+1)\\cot(\\pi\\alpha)}{\\pi(2\\alpha-1)^2}.$ There's no closed form expression for its minimum value on $(0, 1/2)$ but it is straightforward to minimize numerically in the vicinity of $\\alpha=0.178691.$
@DrWho, you are welcome.
Do not use the comment section to give additional information that is not suited for this section. Use the edit button to edit your original post.
Thank you. My motivation here is that I would like to estimate the "true value" with the least possible error (on average), and to me the variance seems to be a good indicator here. I have two options, and the second, though it seems inappropriate to you, obviously (?) produces slightly better results. And I'm asking why. Also, the direct computation of the sampling variance gives me headaches, see the second part of my question.
I've used the R topicmodels package quite a bit. I'd certainly recommend it over rolling your own code - there's some documentation with a worked example at http://cran.r-project.org/web/packages/topicmodels/vignettes/topicmodels.pdf. The specific formatting of each document doesn't really matter, as everything's going to be reduced to a "bag of words" representation anyway. Just throw all the associated text into one string.
Can you provide a reference for a definition of restricted cubic splines?
(+1) I concur that this would make a good question for the main site and will upvote it. It is somewhat subjective, so it *might* be best as a CW question. (Also, there is *no* reason to be embarrassed.)
Thanks. I've updated it with a new link that does work.
I'm not so much worried yet about the fact that doing a lag plot makes no sense. My issue is far more fundamental. Your data are either observed over time (a time series) or they aren't. You have asserted that they aren't a time series. If your data are not "over time" how does one *lag them*? One would have to move a series by one in a dimension it doesn't even posses!
A few questions: 1) Are you interested in comparing all the different types of learning approaches you mention, or do you just need one to get a job done?  2) Does each document belong to 1, 1 or more, or 0 or more of the classes? 3) Do you specifically *want* to use feature selection for some reason, or did you just think it was necessary?  I agree with the other comments that this is a modest-sized problem by today's standards, and dimensionality reduction is not necessary.
Then how could i make simple design? I am still studying stat... Could anyone explain why do i need to use unpaired t test? and what will happen if i don't?
I'm not an expert for similarity of AST representations. I believe there is a notion of "simulation" in the sense that one tree is a special kind of subtree of the other. To compare the ASTs, you'd need to align them and count the relative differences, I guess. Maybe not taking order of branches into account, so trivial code reorderings do not change results. Be aware that you might get to the point where you get false positives because there just are n ways of solving the problem efficiently, and you get false positives just because they found the correct solution...
For every $n$ you can control the quality of the normal approximation using Berry-Esseen style bounds: http://en.wikipedia.org/wiki/Berry%E2%80%93Esseen_theorem
@Andrej I don't use R (coming from ML rather than stats :) but what mbq suggests seems fine.
Did you leave out the word hurdle?
Check out [this question](http://stats.stackexchange.com/questions/17174/a-problem-with-two-ways-of-calculating-aic) and the answer by Rob.
I see, many thanks.
Thanks for this, but I already tried this: the optimizer massively over estimates the variances?
about the plotting, for those encountering order problems, [this advice](http://stackoverflow.com/a/10008228/597069) is useful
ADF only tests for unit root stationary, this could be trend stationary. So you should use the KPSS test, see http://stats.stackexchange.com/questions/30569/whats-the-difference-between-stationary-test-and-unit-root-test

In general, there is a difference, between DS (difference-stationary) and TS (trend stationary) models. KPSS is the better test to distinguish between those models, see the link for more details.
Thank you for clarifying this. I just found a very good answer for a similar question here, with a nice worked example:

http://stats.stackexchange.com/questions/29479/difference-between-ci-of-anova-level-coefficients-vs-t-test-ci-which-one-is-c?rq=1
+1 - good answer but can you expand on condition index? I have yet to find a satisfactory means of dealing with collinearity in candidate explanatory variables.
... of course, power depends on the sample size and the exact alternative, so it can be arbitrarily close to 100%.
Hi Stephane. You might want to look at L. D. Brown, T. Cai, and A. DasGupta (2001), [Interval estimation for a binomial proportion](http://projecteuclid.org/euclid.ss/1009213286), *Statistical Science*, vol. 16, no. 2, 101-133. There, they point out a connection between the Clopper-Pearson (guaranteed coverage), mid-P, and Jeffreys (no guaranteed coverage, especially near end) intervals. See **Section 3.2** and **Section 4.3**. I have not gone through all the details, but I believe the effect you are seeing is real.
@StatTistician the idea is correct, but the implementation does have flaws. For example, you did not consider under-flow. Also, you looping variable k is confusing, you first set loglik[1] and loglik[2], after entering the while loop, you set loglik[1] again. This is not the natural way to do. My suggestion about initializing loglik[] means code: `loklik <- rep(NA, 100)`, that will pre-allocate loglik[1], loglik[2] ... loglik[100]. I raise that question because in your original code, I did not find the delcaration of loglik, maybe the code is truncated during pasting?
"*I have seen the future and it is very much like the present, only longer.*" (Kehlog Albran, The Profit)
Indeed not! Sampling more would not eliminate such intrinsic bias. That's the issue here. The poster is interested in power for estimating population association with a nonrandom sample, and my point is that you always have 0 power to estimate that (unless *very* careful and complex weighting mechanisms are employed).
You have demonstrated the calls are *not* uniformly distributed.  The $\\chi^2$ calculation is not "spoiled": it worked!
I might add the word "local" to make it clear that step 1 exists -- I agree with @Glen_b (another Glen -- hi!) that this is important. So I would tentatively suggest "local mean defect" where I just concatenated "deviation from the ideal" to "defect". Seems suitable.