tttjjj/elt_bench huggingface dataset:
@Cadu : What's the software you're using? If it's merely for calculation, you shouldn't use the theoretical functions. In R, there is a function called dnorm() that does the calculation for you. On this wiki page you get more information on the probability density function for the normal distribution : http://en.wikipedia.org/wiki/Normal_distribution
The clear answer will be somewhere along the lines of '*better is what you define it to be*', I'm afraid.
Can you have much Signal without variability?  Would'nt it be like an independent variable with a Slope close to zero.
I edited the existing answer to emphasize how it fully answers the question.
Note that ARIMA is usually more about representation rather than a different model to AR.  Usually, you can re-arrange an ARIMA model into an AR model.  I think you run into estimation problems if you can't (I'm pretty sure non-invertible times series models are hard to fit).  Also, much of your differences are more in terminology rather than models.  A conditional Random Field is basically a mixed model.  Hidden Markov models are very similar to Kalman Filters, etc. etc.
..thanks kjetil!
@Alec: Should work for small numbers as well. If you can do it either way, then I'd match for JBT and then take 10%, because you will really only have rounding error in the total estimation (e.g. 10% of 100 to 109 total tweets will be 10). At very high sampling rates the FPCF (finite population correction factor) may enter in, which will lower your variance.
this is complex enough to be worth asking on statalist, as well. (I would still be the one most likely to respond, though :))
I am trying to perform Tukey-type comparison using the glht package after fitting a GEE model. To do this I have to specify the vcov matrix in the glht function, which lead me to this post... Now, the problem is that the robust elements in the vcov matrix are smaller than the naive elements. Do you have inkling or former experience as to how this can be?
I love that blog!
Thanks, looking into that now.. (new world)
Closely related: http://stats.stackexchange.com/questions/54/why-do-us-and-uk-schools-teach-different-methods-of-calculating-the-standard-dev.
You need to find a good distance measure (often the most difficult task in clustering): if you can find a distance measure that correctly and accurately describes how similar (or not) your data items are, then you should not have any problems.
I added in a few missing hats on $\\beta$ terms.
@ Jon: you would be better of asking this question in CV: I will flag to migrate there.
+1 Very nice answer! Haven't yet read it completely, but your answer is the kind of I've been looking for. All the steps explained =)
Now I feel silly because I didn't even notice a warning until your comment. It does say "due to redundancies, degrees of freedom have been reduced for one or more variables". Is this bad thing, or just something I should note when I explain the models?
Logistic regression can certainly be used to test hypotheses.
Thanks, especially for your examples. I am very satisfied with the term "low N implies high variance", it was amost exactly what I was looking for.
This answer refers to a previous plot, now deleted.
Very thoughtful, thorough answer! Thanks very much.
"standard errors and p-values of interactions of standardized models are not reliable"? That is not true as far as I know. SE and p-values are equivalent between standardized and un-standardized models.
Why do you want to predict treatment? You know treatment, it's fixed by design.
Do you mean how to compute the probability of a sequence of possible latent states?  The forward algorithm conditions on the observations up to time $t$.
Peter Diggle has written a "model-based geostatistics". He also has an analysis of Lancashire crime data on this page: http://www.lancs.ac.uk/staff/diggle/MADE/ that might give you some good ideas.
@Gung I thought it was variance because I'm trying to see how far the participants vary from the mean either in that conversation or in all the conversations as a group.  Yes, it would be a count variable.  Is Poisson regression the same as logistic regression (see, I'm lost already!)?
I a couple of issues, but it's possible that these don't explain all of the differences in your example: One is that you didn't count all the parameters. The second is that likelihood is only defined up to an additive constant (so you can add or drop constants as you see fit, it's all still AIC), but to compare AICs you must use the *same* constants.
It concerns three different moderators to the same main relationship. Two are highly correlated.
I would have thought that as the Variance-Gamma distribution is a variance-mean mixture of normal distributions that MLE would be most easily done via the EM algorithm (at least for some of the parameters).
glm has weights, but does it have anythiong similar to STATA's "absorb" that allows me to include fixed effects but repress them in the output?
I agree with those points. Let's assume we do control for those things. I'm really here to learn how one would do the Bayesian analysis, not conduct a proper poll (although that is obviously also very important).
I don't think anyone could answer this till we get extra information as per my last comment...
@guest can you give us an example about `se.fit`? Thanks.
It depends on whether you are using the ACF to identify an MA model, or to test for white noise. Bartlett's limits are better for identifying a model, the default limits are appropriate for white noise tests.
+1 Good catch--thanks for your patience and time for looking this over so carefully.
@whuber: 1. the variation in y is due to random noise. 2. no. 4. very large, say 200 lines with 1000 points each. 5. usually yes, but some x values may have a few (say 1->10) y values  missing.
@Tomek Differentiation w.r.t. a single attribute ought to give its coefficient (when there are no interactions).  Of course, since the solutions are functions of all the data, you could differentiate w.r.t any particular $x_{ij}$.  It should be straightforward to work out.  (In fact, finding the solution for OLS would be sufficiently general.)  I haven't read B,K,& W in 20 years, so my memory is dim, but I don't recall that they went down to this level of detail and they definitely don't do this computation in the section I quoted.
+1. And I'd add a note of caution: lowess, in some implementations called loess or locfit, is perhaps better thought of not as a single precisely defined method, but as a family of methods depending on polynomial degree, kernel used, whether there is extra robustness built-in, etc. Different implementations won't agree unless they follow exactly the same recipe.
By using `drop1` you imply that dropping variables from a model is a good thing.  Instead, think about what hypothesis you want to test in the context of ANCOVA, and go and test it using the appropriate contrast within the context of the full model.  There are some equivalences to this in the normal case but dropping variables from the model is an inefficient way to proceed even when correct.
@Dason Yeah, that's true. Who said statistics isn't funny?
I would test both options and see which one works better, the dataset seems to be large enough. It may be worth trying to group selectively though, e.g. aggregate the breeds that have very few observations. On the other hand, there may be some popular breed (e.g. due to a movie) that people want in particular, in this case grouping would hide this information.
There's nothing special about negative AIC. Smaller (i.e. more negative, for negative values) is better.
One very good reason is because the working point was chosen previously: if the results in question are from the final model's validation, no further data-driven tuning (neither of the cutoff/ROC working point nor of any other hyperparameter) is allowed.
Do you agree that physics uses standard statistical hypothesis testing with a very low alpha (in this case, anyway). Or do they use some kind of Bayesian approach as Nestor said in a comment above?
As a side note, I'm getting very strange behaviour even trying to fit an exponential and linear term to the model!
You could use theoretical knowledge that you have of the subject matter, but to a first approximation there isn't a "best", at least that we will be able to discern. Note, eg, that the MSE *must* go down as you add variables, & R2 *must* go up.
@PeterEllis: I might not discount some utility of the normal distribution. Consider a two-person game in which Player 1 starts with $a$ chips and Player 2 with $b$ chips. Suppose they are equally skilled so that at the end of each turn, the result is that one player gives the other one chip with equal probability. Then Player 1 wins with probability $b/(a+b)$. The fortune of each Player follows a random walk until an absorbing state is hit, and the walk up to this time is approximated by a Brownian motion. If we introduce a third player, we get a diffusion inside the unit simplex.
The Ljung-Box test statistic (`X-squared`) gets larger as the sample auto-correlations of the residuals get larger (see its definition), & its p-value is the probability of getting a value as large as or larger than that observed under the null hypothesis that the true innovations are independent. Therefore a small p-value is evidence _against_ independence.
yes i think you're right about the cross validation.
You'd expect standard error (possibly) to get smaller, sd should get larger, because the spectra have different means.  sd(1,2) and sd(10, 11) are both smaller than sd(1, 2, 10, 11).  (If I've understood correctly).
As stated, both are wrong because $\\mu$ is a row vector while $T * invnorm(U)$ *must* be a column vector.  When you get your rows and columns straightened out, this question should answer itself simply by identifying which version of $(X-\\mu)'(X-\\mu)$ or  $(X-\\mu)(X-\\mu)'$ gives a matrix and which version gives just a number: check that you can compute the expectation of the matrix version and that it gives $S$.
@JMS: +1 Yes, Copulas are very appealing. I should try to study them in more details!
I've added an example, but it will always depend on how you've formatted your corpus (BoW, sequence of tokens, etc)
If you don't mind, I'll add tag `self-study`, failing to find tag `friend-study` on wiki.
+1 But, with "millions of data points," it seems highly likely that a t-test *would* be valid, almost regardless of the distribution. The real question here concerns what a t-test would mean when applied to "the full population (not a sample)."
In 2014 it can be presumed that you are using some program to do this: I would assert that's standard for software to do the ranking for you. Conversely, if you are doing this by hand, ranking first is a necessary part of the procedure.
The spreadsheet shows that under the assumptions of the question, Team1 (with a mean of $1.05$ goals per game) has a $9.14$% chance of beating Team2 (with a mean of $2.5$ goals per game) by one goal. The chance of a win is $13.98$%, a tie $17.07$%, and a loss $68.95$%, all found by summing appropriate ranges of chances in the spreadsheet (corresponding to the red, gray, and blue bars in the plot, respectively).
Hi, I just added the output. Thank you for your suggestion.g
"the statistician does not work on laborday" Is this some parochial reference? CV is an international site!
@ThierrySilbermann. Hidden = 10
Sorry, not a stats person ... $s^2$ is ... ?
If you do not have samples with a target label in training data set, for sure your model will not be able to predict it. Any variant of stratified sampling should be better than nothing
@Bakaburg You should consider using the p value as an index of evidence against the null hypothesis as Fisher intended it to be used rather than comparing it to a binary threshold for significance in the Neyman-Pearson manner. In my opinion, most people are more satisfied by the former even though they are made comfortable by the apparent rigor and certainty of the latter. You might like to read my new paper on the topic ;-)  (Brit J Pharmacol DOI: 10.1111/j.1476-5381.2012.01931.x)
@GaBorgulya About your first point, it seems to me that non-parametric tests make the assumption that there's a shift in central tendency, but that overall the distributions that are compared have the same shape.
Sure, when you said random walk, I assumed Brownian motion because the term is sometimes used synonymously, although strictly speaking, you are right random can be refer to any type of randomness a priori. You could still use moving averages and variances over a certain time window if the process is more exotic. But you can avoid that by transforming to the proper scale sometimes. Say, you have a geometrical Brownian motion, just plot the log of it.
Finally, somebody is here to talk about the Big Data Emperor's new clothes. Who needs the 600M Tweeter users if you can get all the answers from the college statistics sample size of 30.
Nobody? :(.....
How do you get a covariance matrix/SD/variance of presumably binary variables? And if you do get them what do they represent?
Thank you for your help and suggestions!
The collinearity is going to mean, however you do the analysis, that it's essentially impossible to determine if one variable is 'driving' the variation any more or less than a variable with which it is highly collinear. Bearing this limitation in mind, you could try [the lasso](http://cran.r-project.org/web/packages/lars/index.html) as a means of selecting a small number of variables that predict optimally, then reporting the set of variables it selects _and_ those with which that set is highly collinear. The grouped lasso is another option.
*Why* are you comparing "samples"? What is the objective?
@Glen_b The data is not discrete.  Values range from -2 to 2.
I don't think we can answer this thoroughly without seeing mymodel, but if there are zero df, then chi-square will be zero, and p will be 1 (or undefined), and the other fit indices are also correct. So yes, it probably has to do with the model specification.
@nico No one likes 0.055
Are the outliers "real"?  By this I mean are outliers caused by the inability of the model to predict extreme events or shocks?  Or are they "errors" such as data entry problems?
+1, this is very useful. Thanks.
I suspect NETLAB also works with octave, which solves the commercial license issue.  I prefer NETLAB as it has the basic tools for classification and regression set up quite straightforwardly and encourages the use of regularised networks, which is very important in avoiding over-fitting.  It also complements Chris Bishop's book very well, and that is the book I'd recommend to anybody who wanted to used neural networks in anger.  Gaussian processes are a more modern option though, the GPML toolbox and book are also highly recommended (http://www.gaussianprocess.org/gpml/).
(+1) Such "binomial" method is sometimes applied in neurophysiological studies, where people record a bunch of neurons in several conditions, and then for each neuron test if it is "tuned" to condition, i.e. if its firing rate significantly depends on condition. This is usually reported like e.g. "137 out of 978 neurons were significantly tuned with p<0.05", and sometimes a binomial test is performed to demonstrate that 137 out of 978 is a significant number (as an average 0.05*978=49 would be expected under null). I have never seen Fisher's or Stouffer's method applied in such a situation.
@HFBrowning: Less a rule of thumb; more an observation that when the effect of a unit change in the predictor $x_1$ depends on the value of another predictor, $x_2$, the effect of a unit change in $x_1$ in the particular case when $x_2=0$ is not usually of especial interest.
I have added violin plots and box plots http://www.win.tue.nl/~aserebre/2distrs.jpg
Hmm, I don't think I can help but perhaps if you posted your code there might be something about the way you are using packages that could be optimised?
+1 I agree. While it's not a direct answer to GKED's question, I think that users looking for tools like StatET are often the same as those who could benefit from RStudio (i.e., not necessarily the users diving into Emacs/ESS with gusto). While both R Studio and StatET offer similar functionality, RStudio wins in the ease of configuration stakes.
Actually, E(x) = 1*(1/3)+(-1/3)*2/3 = +1/9.
@gd047 Well, now that I've read the question, I can say that I didn't find my way with this in R.
Welcome to the site. :) How many months of data do you have?
Neptune, please put the link *in your question*, along with a proper reference, and then extract the information about what $K$ is from the article and explain that in your question so people don't have to go searching to even understand what you're asking.
Caveat a *time-series* multiple imputation approach would be better, as non-time series imputations fail in spectacular ways. See for example, Honaker, J. and King, G. (2010). What to do about missing values in time-series cross-section data. *American Journal of Political Science*, 54(2):561–581..
I can't advise easily; I can't see an interesting statistical hypothesis implicit in your question. You could reduce your matrices to a $2\\times2$ table: 0 or 1 in $A$ versus 0 or 1 in $B$ but that sounds a lousy way to compare $A$ and $B$ as virtually all the information is discarded.
But variables are usually *not* strictly ordinal or interval. The blind application of Stevens' taxonomy is just as bad as the blind application of any other rule. 

For strictly ordinal, you could recode it a 0, 2, 2.1, 2.2, 2.3, 19288191. But *would* you? And, other than physical measurements, what is *really* interval?
I think the model was intended to look like regression without a covariate which just makes the mle for alpha the sample mean for the ys and the residual variance estimate is just the sample variance of the ys.  Without a covariate it is not regression.  i should have referred to the estimate of beta as the least squares estimate and not the mle because the error term was not assumed to be normal and the properties of the estimatoe for beta is not relevant to the problem. I am editing the answer.
Note that this is not an $R^2$ because your "TSS" is not the (usual) total sum of squares, nor is it even a reduced sum of squares.
$p(\\mu,\\sigma) \\propto \\sigma^{-1}, \\,\\, \\mu \\in \\mathbb{R},\\, \\sigma \\in (0, \\infty)$
@Cam.Davidson.Pilon - I'm waiting on other submissions too! Because your model only has one predictor - sorting by the predicted proportion of defects would be synonymous as sorting by temperature  (assuming a monotonic effect - as it appears in your graph). Perhaps someone will come along though with another solution that effectively allows one to see both the predicted proportion and original temperature (or something completely different). This display is good for seeing bad predictions, but is not very good for things like seeing non-linear effects.
@BenBolker: forgot to, it was an older formula - now it is included
I believe a parametric bootstrap may be called for here in order to respect the covariance structure mentioned in the question.
what is your disciplinary background? what exactly are you looking for - theoretical problems, problems in an area of applied research etc?
Hello ash6. In fact it's a script for R software (for more info : http://www.r-project.org/), not syntax for PASW17. So this script can be run directly in the R console. This script does not calculate confidence intervals but give you the p-value (here precisely > pchisq(x, n, foo$maximum ==> [1] p-value=0.1190264) corresponding to the test of a difference between the 2 experiments (here between two stimuli, in the case of alternative hypothesis), and here we can't reject the null hypothesis that the two experiments gave the same results.
This may be the best answer, but my guess is it's way beyond the questioner's abilities or needs.
Yes. See help file: "model: Output from a previous call to Arima. If model is passed, this same model is fitted to x without re-estimating any parameters."
@Ladislav The method is called Transfer Function or Dynamic Regression. It incorporates Intervention Detection http://www.unc.edu/~jbhill/tsay.pdf procedures which have been extended to cover cases where there are known possible predictors (x=war).
(+1) I would give the same numerical solution anyway.
(+1) Nice find!
@drknexus I do not know R to comment in your question. Perhaps, you should just post your question with a comment that your qn may be lined to this one.
@gung thanks. maybe but it looks that `survival analysis` answers question like( correct me please) `what is the proportion of a tasks that will not finish after a certain time`? This is a part of an answer , I am looking also to characterize this tasks, can I do this using survival?
@Aksakal, a negative intercept would mean the the _actual_ (not testing) score has bias compared to target (predicted) scores. Furthermore, if "students are given a target score" "using _their_ previous results and the [I understand: _their_] results form an aptitude test", why should I compare schools in affluent ad poor areas? Furthermore, 'score' is _one_ variable; why should I imagine multiple variances?
If it were a multilevel model, I should shrink my estimates, but why bother if I'm comparing a single-school regression to a pooled regression?
This question does not appear to be about statistics within the scope defined in the [help center](http://stats.stackexchange.com/help).
I think I've fixed it up. $t \\neq \\bar{x}$, hm?
Thanks @gregsnow, helped a lot, even to make sure that I need to study a lot more!
By definition you have $\\sum_i L_{i,j}^2 = 1$. Since $L$ is real, you must have $-1 \\le L_{i,j} \\le 1$, as @whuber points out.
Thank your for your insightful comment. In some applications, coarse class comes from other contexts and may not necessarily similar in input space. If classification algorithm automatically recognize those data properties, that would help a lot.
I don't know what a Local Probability Distribution is, and a Google search shows up only a paper by an H. Rabbani (couldn't find a definition within). I am not an expert but in case this is not well-known terminology, would you post a definition please? We can help you unravel it.
1. the probability that the sequence is generated by component $k$ of the mixture, 2. for each cluster you have a set of sequences, from which you estimate the transition probabilities. Notice that for each component, you get different estimattes. When presented a sequence, you calculate the probability of that sequence for each of the clusters, i.e. the probability that the sequence has been generated by each of the components (as in the expression above), and pick up the index corresponding to the component yielding highest probability.
Not sure how I would work it with that new piece of information.
+1 Good answer. Just enough help given.
In this example the parameter would be the ratio of the two variances.  Hypothesis tests and confidence intervals serve different purposes.  A confidence interval tells you what the pluasible values for the parameter are.  The p-value for the hypothesis test just tells how unlikely it would be to observe a value for the test statistic that is as extreme or more extreme if the null hypothesis were true.  The conventional thing to do when testing that the variances are equal is to compute the p-value based on the test statistic for the given sample. I see no reason to do what you are suggesting.
I can't understand your question. Can you more clearly differentiate b/t the 3 pairs of variables? It sounds like you're correlating the same pair of variables 3 times & getting different results.
@whuber [Here's one](http://www.phil.vt.edu/dmayo/personal_website/). And [here's an econometrician](http://www.econ.vt.edu/faculty/facultybios/spanosbio.htm) who collaborates with her on scholarship in philosophy of statistic.
(+1) For the care you took to comment the approach proposed in the OP.
@MicheldeRuiter - I'll try to find it at my office and rehost it, thanks for mentioning this
I see a study group forming...
http://stats.stackexchange.com/q/17121/5509