tttjjj/elt_bench huggingface dataset:
when you say `What is the average growth rate over these three years?` what you need is a type of average suitable for averaging growth rates, rather than for averaging percentages. Percentages occur in a variety of ways and different kinds of averages are used depending on what you're averaging.
Or sets, if you wish. My real point was to try the obvious non-fancy things before spending a lot of time on fancier visualizations...
(1) Was your sample obtained with or without replacement?  (2) Selecting items "with probability $1/\\pi_i$" won't work (and is not part of the H-T estimator anyway), because obviously--since $\\pi_i \\le 1$--these reciprocals exceed unity.
Are you, for example, claiming that an ohm has a privileged physical meaning that, say, 1/ohm does not have?  (This is not true: 1/ohm has a physical meaning too, in terms of conductance instead of resistance.)  Often, a well-chosen Box-Cox parameter *reveals* a physical meaning that was not apparent in the original formulation.
The adjectives "parametric" and "nonparametric" don't apply to data. Can you explain what you mean without any technical terms? Your comment should be rewritten as a complete English sentence or sentences and included in your question.
@EmmaDixon Incidentaly, if you wanted to use the So-And-So test (a very famous test, don't you know ;) in Stata, and the So-And-So test *is not* built into the basic installation, you can type `findit So-And-So test` to potentially pull up a list of packages that *do* perform it (assuming you are connected to the Internet). If and when you do so, you can just click on the blue links to follow the leads to description and installation.
@Srikant Vadali: I wanted to post the formulas, but I couldn't find a way to embed them here, and ASCII-formulas would have made things worse instead of better. Also, the book I referenced is freely available as an ebook, so everyone can look it up.
@Matilda Are you sure the distributions of $y_n$ and $y_1$ are unknown? If $Z\\sim N(\\mu,\\sigma^2)$, then $Z+a\\sim N(a+\\mu,\\sigma^2)$. You can apply this property to both $y_n$ and $y_1$.
I'd be curious to know if you ever managed to do something about this problem. I had it marked for any update - so just touching base here.
Also, can you clarify if when the covariance matrix is used, if $\\mathbf{\\Sigma}$ is simply substituted for $\\mathbf{R}$ in the above definition of $\\mathbf{C}$?
@Andre To use the last part of this code you need to bind the x and y variables into an $n$ by $2$ matrix, as in `data <- cbind(temperature, pressure)`.  (This was done in order to match the way in which (x,y) data are usually stored in `R`, either as a matrix or data frame.)  I apologize for omitting that  step and have edited the answer to include it--see whether that works.
I'm confused about this problem set up. It sounds like if the spider reaches corner 2 or corner 3, the chain is over. So why are there valid transition probabilities from these two sstates to the other states? If the spider reaches corner two or three, for all intents and purposes doesn't it "stay" there? If not, the next state after two should always be "dead" and the next state after three should always be "exited," but it looks like these states aren't represented in your transition matrix since it's 4x4 and not 6x6.
By the way, all datasets have roughly the same median value.
You don't need to test for autocorrelation. It is there. The plot shows that. You could look at the autocorrelation function of these residuals (function `acf()`), but this will simply confirm what can be seen by plain eye: the correlations between lagged residuals are very high.
It seems to me that that the problem is one of micronumerosity (with respect to the number of model parameters -- you have thousands of them) rather than a problem of computation.  It's simply unreasonable to think you can estimate thousands of parameters with only a hundred observations or so.  You should note that the last paper you cited sets sample sizes on the basis of the dimension of the underlying multinomial.
@mpiktas: I am a newcomer to the field so I cannot give you any references.  However, it is easy to imagine where the regression might come in.  Suppose you measure the social networks of $n$ schools.  These networks take the form of adjacency matrices $X_n$.  You also know some demographic information about the schools, like average income, ethnic composition, etc.  How are those covariates related to features of the graphs?  And by the way, the eigenvalues capture a lot of the important properties of the network, such as average degree, connectivity, and clustering.
Have you looked into regression trees?  They seem like a good fit to your question.
Indeed. See http://en.wikipedia.org/wiki/Granger_causality.
You can't, at least not with the information you've given. Even if $F$ and $H$ are (marginally) independent, they may not be conditionally independent given $G$.
Just to clarify your "least squares solution" statement: one can use linear regression with number of heads in each experiment as the outcome variable, and the number of tosses of each coin as the predictors. The estimated coefficients will be the probability of heads for each coin.
And there isn't an alternative. You cannot sample without some distribution by default. Most imperative computer languages have a standard library function that produces random numbers in a uniform distribution between 0 and 1.
but ACF for MA(1) is zero when lag>1, isn't it?
Hint: to evaluate $P\\{U^2+V^2 \\leq w\\}$ switch to _polar_ coordinates.  But give some thought to the _uniform_ distribution and think of ways of computing this probability without the formality of integration.
We could help if you show the model.
I see! So you mean that the total average rate of unplanned pregnancies over longer time periods would evolve differently depending on variations between subgroups? Would you expect the evolution over longer time periods to be significantly different than using the technique I used (say, greater than 10 percentage points)?
I was thinking about a regression analysis. Which other analysis may I conduct with such a small sample?
Although this distribution can be described, it tends not to be useful. What to do in this situation depends on what you are planning to do with those sample means. Could you add some information about that to your question?
One example (as noted below) is if the cost of incorrect prediction is the same for all levels of profitability. Ie. when you have a continuous variable, but you are truly only interested in the discrete values. A spline with n knots regressed to the continuous variable may put many knots on the extreme values to accurately mode the shape of the data there, whereas a spline optimized for classification can put all its knots around 0.
What problem do *you* particularly face? Please tell us so that this question does not have to be closed as overly broad or ambiguous.
A strong case can be made for robust techniques that do the opposite of "punishing" outliers, because outliers in benchmarks often occur unavoidably due to other processes and external perturbations causing a process to wait from time to time.  This could be the reason the benchmarks do not appear to be normally distributed, too.  However, as cgillespie's answer points out, you can go a long way with the standard hypothesis testing machinery applied to SDs.  As a compromise, a lightly Winsorized version of the SD might be a good choice.  (http://en.wikipedia.org/wiki/Winsorising )
If $r=1$, then $Y=a+bX$, $b>0$.
What distribution do you want to test against & why?
A draft version (May 2003) of *Data Mining with R* can be found [here](http://datamining.dongguk.ac.kr/lectures/2009-2/dm/DataMiningWithR.pdf). (I don't have the book, so I can't say what's the gap between the two versions.)
@Tal "A poor choice" *for a particular purpose.*  What distinguishes us as statisticians from others is our understanding of the role of variability in data analysis and decision making, along with our ability to quantify that variability appropriately.  That is the approach exemplified (and implicitly advocated) in my answer here.  When it is carried out there is no such problem as you describe, because the user of the permutation procedure is led to understand its limitations and its strengths and will have the freedom to act according to his or her objectives.
Joel, Kendall's tau requires only that you be able to *order* each variable, so it applies here. I have some concerns that it might not be a very powerful procedure (especially because of all the ties it will have to cope with)--but that means if you apply it and it shows a clear and significant difference, then that difference would likely be even more significant with a better test.
Secondly, as you said, perhaps we should center the data before creating squared terms. Such a practice will mitigate the MC problem. However, it could lead to biased estimates, or more concretely, the omitted variable bias (OVB). To illustrate, see the following example: suppose the true specification is: y=b0+b1*x+b2*x^2+u. Centering the data beforehand will give: y=b0+b1*(x-xhar)+b2*(x-xbar)^2+v, where the new error term v=u+b1*xbar-b2*xbar^2+2b2*xbar*x. It is clear that cov(x-xbar, v)!=0. Thus, unfortunately, centering data beforehand would lead to biased estimates.
I thought I would also mention Tiku, et al, "Mahalanobis distance under non-normality", 2010 (which I am waiting for) and  Ekstrom, "Mahalanobis Distance Beyond Normal Distributions", 2011 (which didn't help me but could help someone else help me).
This may be related: http://stats.stackexchange.com/questions/961/statistical-test-for-n-x-m-contingency-tables
I have upvoted all three current answers to this question (Michael Chernick's, Patrick Caldon's, & whuber's). Taken together, I believe they provide a thorough understanding of the unit root, from the intuition to some of the underlying mathematics. +1 for a productive question.
I've edited my question to add more information
This may be a better fit for the Cross Validated site. If this is a programming question, you should post some of your code.
1. steffen, the lecture is available here: http://www.ml-class.org/course/video/preview_list　(Part X, Section "Learning Curves") 2. I see. I was thinking that when lots of features learned during training => model gets complex, and overfits the training set =>  Variance...
Related: http://www.jstor.org/pss/2684656
You need to find out how they find the fitted value for the first observation. One way is to remove the first observation. Then reverse the whole time series data that you have and then fit your AR(1) to this new data. Finally, forecast for just one step a head (i.e. h=1) to find the fitted value for the first observation. I am not sure if they implement this, but I have already read this method somewhere! I cannot remember the reference though!
Hi @DzungNguyen - I have made some edits to try to clarify your question, but you should check that I haven't changed your intent (e.g. with the replacement of 'rigorous' - which really didn't seem to fit the context - with 'rough'). The use of the abbreviation 'ML' is ambiguous here (it might mean one of several things). Do you mean 'maximum likelihood' or 'machine learning' or something else?
Perhaps you can clarify a bit more.  Are you interested in the type of errors?  Are the frequencies in the various categories equal.  What do you mean by "which rows attribute the most to the difference"?   Does this mean you are interested in testing the difference between rows?  Thanks.
Also: unless I have misunderstood the lecturer, he is not saying that you should substitute the prior with a delta function. He is saying that the belief "my parameter takes a certain value, which I don't know" can be translated into a delta function centred around an unknown point $a$. In practice you can use a proper prior (such as your uniform), but you have to be aware that this is not equivalent to the above statement of ignorance, but you are making some assumptions.
@chl -- I avoided it deliberately putting that as a comment because a comment which has a directly different idea all together tends to sometimes offend the original answer giver. Moreover, my question about doing it programitically was intended to be taken as one of the solution.
+1 for a thorough and interesting answer.
not R, but I use biogeme to estimate discrete choice models for transportation and private sector research: http://biogeme.epfl.ch/. Be glad to give you some tips to get up and running if this is the type of analysis you need to do.
(Taking the $X$'s to be independent, speak up if this is not reasonable), if you replace $w_i$ with the precision (inverse of the variance) of $X_i$, you will get a smaller-variance estimate of $A$. Of course you probably have no idea what the precision *is*, and $w_i$ may be your best notion of something like it, in which case you may not be able to do much.
I'm very interested in your SEM suggestion. I'll admit that my vague impression of SEM has been that it's like, say, game theory: very general, obstensibly can solve anything, but results in simplistic diagrams and not very satisfying answers. But your description, as simultaneously solving mixed models, makes sense. Can you point me to any papers/examples that seem particularly relevant to my situation? Thanks so much.
As far as post hoc power calculations - please see the paper I already mentioned.
@Anvaka what software did you use to calculate betweenness centrality? Do you have a script? Thanks!
Here I am thinking of all of frequentist/bayesian probability and all of descriptive/exploratory/inferential statistics.
thank you for the answer :) ! Now with the other's comments I'm OK :). Thank you :) !!!
I have found similar question, maybe it will be [helpfull](http://stats.stackexchange.com/questions/5935/anova-on-binomial-data).
(+1) Oops - I'll fix it. Thanks, my bad.
Sorry, do let me clarify. While the other question seems to focus on NHST doing things apparently backwards, I'm specifically interested in what the statistical tests are actually doing vs. what people assume/talk about them doing. Admittedly, it's a subtle distinction, but one that isn't addressed in the other question.
Thank you for the suggestion. However, I didn't see any video about the specific topic of large data analysis.
I have a feeling Luc Anselin wrote more than one paper in 2010! A more specific citation (plus a link) would be useful (although he has used these terms since his [Spatial Econometrics book](http://books.google.com/books?id=3dPIXClv4YYC) printed in 1988).
Once again I'll say: it's hard to speculate what causes the error without having the data. As for the properties of different extraction methods in SPSS, you may look [here](http://stats.stackexchange.com/q/50745/3277).
@Christopher For Mondrian, you have the "equivalent" R version through `iplots` cited by @Tal. About Paraview, you have the option to save a screenshot of your viz. `DescribeDisplay` is the way to go for exporting dynamic visualization from GGobi, http://cran.r-project.org/web/packages/DescribeDisplay/index.html.
For example, if I get a new sample every 10 minutes, and I'm doing an outlier detection of the network bandwidth usage of a company, basically at 6pm this measure will fall down (this is an expected an totaly normal pattern), and a standard deviation computed over a sliding window will fail (because it will trigger an alert for sure). At the same time, if the measure falls down at 4pm (deviating from the usual baseline), this is a real outlier.
In this context it is commonly understood that $\\left(X^\\prime X\\right)^{-1}$ is a *generalized inverse* (or, more accurately, a [constrained pseudoinverse](http://en.wikipedia.org/wiki/Bott%E2%80%93Duffin_inverse)); that is, 
$\\left(X^\\prime X\\right)^{-1} X^{*\\prime}$ refers to any solution $\\hat{\\beta^\\prime}$ of the equation $X^\\prime X \\hat{\\beta^\\prime} = X^{*\\prime}.$  The value of $X^{*}\\hat{\\beta^\\prime}$ is well defined if and only if $X^{*\\prime}$ lies within the span of the rows of $X$.
Although this question is couched in terms of R, I believe it may be appropriate for CV because there are several questions about the nature of these models, AIC vs SSE, how to interpret coefficients, etc, that make it a better fit here than SO.
You've posted quite a few routine bookwork questions. Please use the self-study tag for such questions, and read its [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) to see what to include when you ask one.
@EmreA - Unfortunately the categorical variables are not entirely independent.  Some combinations will be more likely than others...
A Wald test is a comparison of these models, as one is just a constrained version of the other. If you do se `log likelihood` instead of `log pseudo-likelihood`, you can use the likelihood ratio test to compare them, but in your case you can only do the Wald test.
@whuber Thanks Whuber, I have modified my question.
Of possible interest: [tag:biplot].
These aren't exactly nonparametric, but there are nonlinear mixed effect models that may suit your problem.
Remember, differentiation is the opposite of integration. In discrete terms, integration is like a cumsum, while differentiation is differencing.
I need *a* sample to have, with 95 % probability, the same distribution as the population has (the distribution between regions and their age groups). Sorry if my terminology or explanations are subpar.
@EngrStudent If the OP is unfamiliar with the Kalman filter, I don't see how that's going to be much help. Perhaps you could instead explain how to approach the problem without invoking any of the specifics (or the jargon) involved with the KF, though perhaps making use of your understanding of it to guide a response on the specifics here.
As user31264 and also glen_b said: You can't. Equivalence tests are only able to show that a parameter is *close* to a certain value but not that it has exactly this value.
This is akin to Winsorization which isn't too terrible.
I'm assuming you have some grouping in the data (e.g. repeated measurements on people) other than the two subgroups you were talking about. Call _that_ grouping `ID` and call the binary subgroup indicator `G`. Then, what I was suggesting was something like putting a `(1+G|ID)` term in your model.
What is x? what is y?  do you only have 2 data points? or data for each county in each year?  Your description suggests that you should have 3 or 4 predictor (independent) variables, but the model you show only has 1.  Can you give us a more detailed description of what you are trying to accomplish and what steps you have taken so far?
To be more specific, "exact" zeros are probably just too small to represent.
I don't think that you need to deal with λx at all as the optimal λy is conditional upon X and any ARIMA structure that might be necessary to render the error process Gaussian. The reason for this is that if you had a Z in addition to X you would have to independently asses a λz etc..
1st: The weight of 3 for the baseline case is that there is 3 times as many cases where {var1=0.03, var2=0}. So the results from SAS (which says that we need 762,112 total sample size to have 80% power of rejecting main effect var2=0, so that is the total sample size we need) would be allocated 37.5% to this baseline case.
That's right and I want to add that you can overcome this problem adding the option na.action = na.exclude into predict function.
Both excellent and thoughtful answers - thank you!
Now I think we're getting closer. I gather that what you want is to come up with an appropriate prior distribution to use with future analyses by examining the empirical distribution of a large number of existing test statistics. That seems reasonable. If you use R, you might want to check out the [fitdistrplus](http://cran.r-project.org/web/packages/fitdistrplus/index.html) package. You may also want to read a little about [meta-analysis](http://en.wikipedia.org/wiki/Meta-analysis).
+1 for the nice description. But, when the denominators of the rates are about the same, then modeling the counts and modeling the rates are effectively equivalent, right? When they are not the same, as you mentioned, it seems hard to justify modeling the counts unless, for example, you use an offset for the population size in a regression model (in which case you're effectively back to modeling the rates). Doesn't this seem to indicate that modeling the rates is always better?
Welcome to CrossValidated.  Thank you for attempting to help answer a question.  A small point: your answer could be much improved by briefly summarising the contents of the link as relevant to this question.  This is also important for the longevity of the answers, since if that link ever dies, your answer is useless.
Thanks for the pointer, I'll dig the PELT and Segment Neighbourhood things. Merely starting some R kind of makes me nervous.
Ok I think I have misused the word `uniform`. As I said I never really learned about statistics in detail. What I wanted to say was kind of a uniform distribution (don't nail me on these ;-)) between large and small numbers. I'm thinking that I could generate random numbers from a distribution function that decayes exponentially with increasing x by means of the inverse function method. Has this any chance of success? The distribution does not need to be statistically perfect, as long as there are nearly as many small as large numbers, all's well.
@Nick Counts *do* deserve special treatment for many reasons.  That is why, for instance, Tukey explicitly devoted the last quarter of *EDA* to analysis of count data.  They definitely are not ratio variables, at least not according to Stevens' original definitions.
Hi Kwak, The Huettmann paper is interesting but I have a few questions. 1) Under Assessing the link Functions, what is meant by the "mean coefficients derived from the 1000 comparisons" (p44)? 2) What do the "Predictions" in Table2 refer to? Predicted probability or predicted number of nesting sites, or what? 3) The marbled murrelet data are claimed to be public domain. Do you know the source? Thanks.
If the F value that is compued is "small" then that means (to me) that the null hypothesis is "very acceptable" since if the parameters for each group are identical then the computed F value would be identically 0.0 . I don't see a small F being considered as evidence that the null is wrong but rather "proof" that the null is true.
The median IS the middle value, but it is not N/2; in fact, since you have relative frequencies, you don't know what N is (and you don't need to know to answer the question).
@santiagozky: The description and graph are very helpful because they show this is a time-to-failure problem, not a regression problem.
Welcome to the site again, and thanks for posting a separate question, as suggested!
It is difficult to see how the second method could yield the correct solution as $\\Lambda\\to\\infty$. As $\\Lambda$ grows large, the solutions to $J$ usually would approach minima of $g^2$ alone and ultimately neglect $f$ altogether. Try it, for instance, with $f(x,y)=(x+y)^2$ and $g(x,y)=x+y+1$. The second method produces $x=0$ no matter what value $\\Lambda$ might have (and $y\\to -1$ as $\\Lambda\\to\\infty$), whereas the minimum of $f$ is attained at $(x,y)=(-1,0)$.
Consider how important it is to check for multicollinearity in your application, especially if prediction is your main goal. All variables related to outcome contain information for prediction, even if they do not turn out "significant" in a particular sample. Problems from multicollinearity often arise from attempts to eliminate individual predictor variables, leading to sometimes counter-intuitive effects on the relations of the remaining variables to outcome. For the management-related variables, you will have to do experiments in any event to validate your model.