tttjjj/elt_bench huggingface dataset:
@lucacerone, in theory you could use whatever test you want to assess equivalence. Equivalence is about what you are testing, not what test you use to do so. Off the top of my head, I'm not quite sure how you'd use the M-W U-test for tost, but I'm sure it can be done. You mostly need to work out the interval that you consider sufficiently close to be considered equivalent. [Here](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3019319/) is a tutorial publication I found.
What do you mean by finding the vector error by adding errors in quadrature?  Each of your directions is an error on a different quantity - adding errors in quadrature is for when you are combining multiple sources of error on one quantity.  What do you envisage the vector error to mean?
the kernel of the Inverse Gaussian Distribution, involves also the term $\\frac 1{x^{3/2}}$, which is critical -the title of your question is a bit misleading. If $k=0$ the integral has a solution involving a modified Bessel function.
UW ... you may want to accept my answer by clicking on the tick mark ;)
From the [help page](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/anova.lm.html) of `anova`: "Normally the F statistic is most appropriate, which compares the mean square for a row to the residual sum of squares for the largest model considered." In addition, see [this post](http://stats.stackexchange.com/a/8247/21054).
I gather this question is about interpreting SPSS output. Can you post an image of the output at issue? (Nb, CV may not let you, if so, you can post it to imgur & post a link here, then someone else can add them for you.) In general, when you have an interaction term (eg, X1*X2), the main effect is the change in Y associated w/ a change in X1 when X2=0.
@Gleb_b "An AIC is only high or low in comparison to another." - How can we think differently when we talk about model selection? We always look at low values rather than higher values. What's wrong with my forth sentence? I think it states rather clearly that we are talking about differences (e.g. within one or two of the minimum AIC). There is no mention of absolute values of AIC in the question.
I don't think you're getting what I'm saying.  What I've presented is the ad absurdum case for your argument that argues against it.
yep, not to mention thing will get really hairy/exciting/fun if we start to worry about the fact that some tests will be more powerful that other test - in an ideal world you want to listen most to the most powerful tests...
My answer at http://stats.stackexchange.com/a/71303 works out the conditional mean and the conditional variance, giving explicit formulas for both (skip to the end at the "Conclusions" section). The conditional mean is the "regression curve."  It is given by the standard least-squares regression formulas. Apply this result to the binormal distribution of $(Z,X)$, whose means and covariance matrix are easily computed.  (Although it is a lengthy post, the derivation in it is notable for not employing any integrals nor even any explicit representation of the PDF.)
I'm sure the observations are independent.
"ENTER" methods are not based on sound statistical principles.  Avoid that at all costs unless you use the bootstrap to find out how poorly the method performs and you are still satisfied with the result.
Sample sizes by class will have a big impact. You will get the same validity and generalization issues that arise when using stepwise regression methods.
Have you seen this? 

http://stats.stackexchange.com/questions/346/what-is-a-good-algorithm-for-estimating-the-median-of-a-huge-read-once-data-set?
@Wolfgang I do not see how your example pertains to confidence intervals.  You do not ask for anything related to a distributional parameter.  Your situation is most closely related to *prediction intervals.*  I think this entire discussion may have some interest in that context, but it doesn't belong in a thread about confidence intervals.
Incidentally, my business _is_ the sort of business that would gain significantly from one extra customer. Imagine a bespoke gent's outfitter, a boutique high-quality business. That sort of thing. Certainly a 10-fold increase in customers would be delightful, but won't happen for many, many years :)
The order of the observations should not matter. Can  you post a reproducible example? If you are using R consider debugging the kpss function of your choice or try a different one (e.g. kpss.test in tseries or ur.kpss in urca).
>you will update the weights by multiplying the existing weight by a value determined by backpropagation    - I don't think so, it's not just multiplication.
If I understand you correctly, you have two SMDs for this study, one based on a group of adolescents and one based on a group of adults. Since the two samples are independent (i.e., no overlap between the two groups), the two SMDs are also independent. So, you could include them both. However, one may still want to account for the multilevel structure of the data (SMDs nested within studies), which again requires more elaborate models. See, for example, this paper: http://onlinelibrary.wiley.com/doi/10.1002/jrsm.35/abstract
Is $a=1$ a requirement? Or does the same hold for some arbitrary constant $a=a_0$?
You should be careful; using the bootstrap with sample maxima doesn't always work (e.g. see [here](http://stats.stackexchange.com/questions/9664/what-are-examples-where-a-naive-bootstrap-fails/9722#9722), also see [this discussion](http://www.ms.uky.edu/~mai/biostat277/boot2ord.pdf) -- if you can bear to wade through the failure of the author to spellcheck). As such you should take great care to make sure that your suggested bootstrap has desirable properties.
@Simone So, if I understand you correctly, you suggest that the authors provide PPV and NPV values but gave the p-value corresponding to a global association test of the 2x2 table? Is it related to this recent question, http://stats.stackexchange.com/questions/9464/can-someone-give-me-details-about-a-particular-confidence-interval-formula?
How were participants sampled? I.e., are all participants from the one network, or are you using something like ego networks to calculate network measures?
Although it's not perfectly clear to me which rotation you are proposing, in my case given the fact that I don't have many observations for building the empirical distribution on which I calculate the entropy, I fear that the rotation will produce an almost equal situation, however I will try. Thank you.
Are you trying to make a conclusion about obliteration and seizure in individuals? If so, trying to use a study-level association to make that inference would be problematic.
Can you define DV and IV?  I'm sorry if I missed that.
@AlecosPapadopoulos Sorry I was not the one who downvoted but this time I do, because you're claiming there are some differences between two equivalent tests.
It sounds like you want to put the [entropy](http://en.wikipedia.org/wiki/Entropy_%28information_theory%29) on a 0 to 100 scale.
+1 In short, physical reasoning: two possible outcomes, perfectly symmetric, and probabilities of all possible outcomes must sum up to 1 -- the only answer can be 1/2 (-;
2. Which variables should (not) be used? You can use data reduction techniques such as PCA if you have many variables that seem to measure a more limited number of traits. I believe you usually **don't want highly correlated input variables** so check that too. But again, **the most important criterion is what's useful**.
@pepsimax - sorry about that. I have fixed the code - should work now.
books have many more things to tell. In correct sense, the random effects models presume (in case of your model) that certain factors behave randomly and carry zero-effect overall(this is what sampling theory implies). However, key moderators present in the data(effect-size here) for individuals continue to defy the randomness and hence are "nested" for particular groups. At Level 2(as you define) fixed effects assumption is active and thus you can identify whether public sector Vs. private sector classification matters.
I realize correlations aren't the way to go. I'm trying to figure out if the nonlinear relationships between my variables affect my ability to use tests like the condition index. Or, based on Maarten's answer, whether I even need to worry about collinearity if the relationships are non-linear.
As I understand, data dredging is something bad. Am I correct? And, in this case, we may not do such categorization, right?
What does your control run look like?  You know, the one where neither side had sugar (or where both sides had sugar).
Hey Alecos! It's very interessting! But I am laking many mathematical tools. That is why I ask so many questions here. But the good thing is that I understand the stuff after solving the exercises ;-)
You want to do the wrong thing. Factors are not implied to be different in level each vs another. It is gifferent _groups_ in the sample that you can compare by t-test (or other test) by this or that factor.
@Rachel, you should open a new question for this since it is unrelated to the original one :)
@Andy W. No as frailty is a the equivalent of a random effect in survival analysis. So the cathetar model is equivalent to a mixed model with both fixed and random coefficients. To show this I'll add the dummy variable example above. Perhaps it could be interpreted as an "order effect" - second infection versus first.
While everyone provided useful answers, I chose this one as the best because it made me think about the problem in a new way.  Thanks!
@Gavin, I doubt it myself, it looks like being from exponential or Gamma distribution, but more info would on the data source would be nice to know.
@gung it looks like trying to find which week had the most babies born, from daily data
This question appears to be off-topic because it is solely about software. It's not suitable for Stack Overflow either.
Quick look at http://en.wikipedia.org/wiki/Queueing_model says that the case N servers one queue is also well analyzed.
@Dail I don't think these residuals are normal: in the left panel they appear to have some negative skew and in the right panel, positive skew.  I mentioned normality only as an example.  This approach still works well for visual evaluation except when the groups have strongly different distributions (for then the proportion between the expected range and the expected SD in a random sample will be different).  For that reason, an essential part of evaluating residuals includes assessing the shape of their univariate distribution (using probability plots, for instance).
I might be missing something in your question, but both the `ltm` and `mirt` packages can handle missing data just fine for 2PL models. Have you tried either of those packages? Also, the reason I didn't post this response as an answer is that I'm worried about how the missingness occurred. Can the NA's be assumed to be MCAR or MAR?
+1 for this post. While non-parametric survival curves are far and away the most common, they are not the *only* method of constructing a survival curve. Caveats about extrapolation still apply however.
I dont think it's should be called "fundamental flaw." Everything's  relative to something. Welch's t-test came out in response to underestimated type I error of t-test of spooled variance, so it's an improved way of controlling type I error "compared to" spooled variance t-test. But when it comes to MCP and very low p-values, it sure has problems.
What does "similar" mean?  The chi-squared test and the KS test, for example, test whether two histograms are close to *identical.*  But "similar" might mean "have the same shape," *ignoring any differences of location and/or scale.*  Could you clarify your intent?
in a general sense: nothing at all. Think of it this way: a survey question is just a tool. A tool's usefulness depends on the context. A hammer is very useful when I want to hammer in a nail, but extremely useless if I want to change my son's diapers... So there is no such thing as "utitility in a general sense"
So when using L1 regularization, would you only count non-zero weights?
@PatrickBurns, thanks for the link but as you can see that the value of the beta for variable x (e.g. IBM) and y (e.g. S&P) is not stable (luckily +ve in this example but may even change the signs along with the magnitude) throughout time, I would like to find a stable beta for x and y using the betas I have for each of the 24 months.
It would really help if you were to describe what the "custom tables feature" in SPSS actually does... As it stands now, this question can't be answered unless we know by heart what the custom tables feature of SPSS does. What do you want to achieve exactly?
Nice answer. I was trying to pin down the OP, but your answer will serve much better. As an aside, Excel 2010 is much better, but I've still written my own VBA code for gammaln and other needed functions, and, of course, use R as often as I can.
Does anyone have a short answer?:)
Not a direct answer to your question, but an alternative method for your particular study: If you're interested in comparing your treatment groups to your control, you could also use a [Dunnett's test](http://www.jstor.org/pss/2281208). (Basically, it can be used after an ANOVA came up significant, that is at least one pair of means has been shown to be significantly different.)
@Itmar,that's definitely what I would try first. However, it seems like there must be some way to ensure that the examples you retain are representative of the larger data set.
Thank you for a very comprehensive answer. I cannot quite understand how the RMS formula can describe standard deviation without dividing by the number of balls. If you compare it to [http://en.wikipedia.org/wiki/Root-mean-square_deviation_(bioinformatics](http://en.wikipedia.org/wiki/Root-mean-square_deviation_(bioinformatics)) they have divided the sum by N. Should the sum be divided by N or N-1 (since 40 balls is just a selection from a population of balls?)
@AmigableClarkKent.  I noticed that you recently decided to unaccept my answer.  Although the additional answers are interesting and informative and tell you how to impliment corrections for leap seconds I really still question why you would want to do it.  Could you explain what it is that may be special about your data to make any adjustment for leap seconds useful to you in your statistical analysis of the data?
Do you know if $x$  and $n$ can be treated as independent random variables? If so, the _characteristic function_ $\\Psi_y$ of $y$ is the product of the characteristic functions $\\Psi_x$ and $\\Psi_n$ of $x$ and $n$ respectively, and so, since $\\Psi_n$ is nonzero everywhere, you get the characteristic function $\\Psi_x$ as being just $\\Psi_y/\\Psi_n$ from which the pdf of $y$ can be obtained.
The correlation between regressors and residuals (not errors) in a linear regression model estimated by least squares _is_ always zero. You cannot test exogeneity (conditional uncorrelatedness) without instrumental variables.
(+1) Nice piece of code.
I can almost "look" at the top of the answer and say "Whuber" :) Very Nice!
It won't be too good since R^2 doesn't apply to logistic regression models. There are "pseudo R^2's" of various sorts but they are all hokey & pander to the view that OLS is some sort of gold standard for multivariate testing. Best to use some form of likelihood ratio test that is based on agreement of predicted & observed values. Good discussion of options at http://faculty.chass.ncsu.edu/garson/PA765/logistic.htm
I also would suggest to investigate the existing data base for anomalies...
@Macro: (1) Until $\\lambda$ is introduced, $L_0$, $L_1$, and $L_2$ are explicitly named as "likelihoods."  After that, though, you appear to be using them as log likelihoods.  (2) What do you mean by "$\\sigma_1 =\\sigma_2 = 0$ ... has probability 0"?  Likelihoods are not probability distributions for parameters.
+1 Note that (1) not all combinations of $(p,q,m)$ are even possible (even when individually their values make sense for Beta distributions) and (2) in Excel, solutions may be imprecise.
Oh, that's you, Jarrett. Ah well. That's an odd and difficult problem, and there is no right solution to it, I am afraid. Let me go back and update my answer with the explicit formulae based on Johnson (1978), then.
I also got a relevant example via swiss {datasets}
For clarity, is it the case that your experiment is 2 (before vs. after) X 2 (condition A vs. condition B), eg?
@Mitten See the help for `set.seed`: it describes how to save the state of the RNG and restore it later.
Where possible you should avoid using the same symbol to mean more than one thing, and you should clearly define terms like 'accuracy' (is that just the proportion of correct answers?).
@Victor, the first vote was a downvote (so it's now 2 upvotes/1 downvote). Maybe someone didn't like this tool. Happy I could help :)
@MansT Thanks for the nice editing.
Hello James, I am testing  if categorical predictors X1 or X2 have any significant association with dependent variables (one dependent variable at a time).  Since there are several dependent variables I am running model for one dependent variable at a time. My question is question is if multiple testing should be accounted for?
"I also used an unobserved components model (UCM) and obtained a good forecast" looks very promissing. What R-library did You use ?
This reads like very slightly reworded bookwork. Is this a question for some subject? Please also try to make your question clearer.
Maybe as someone who has majored in CS I have a slightly skewed view on this, but if you can implement hold-out validation correctly (which already means splitting the dataset into 2 parts and using one for training and the other for testing), the only thing you need to change is the ratio of the split and put the whole thing into a loop. It just seems hard to believe that this would be a big problem.
Dilip, in that case since we know that $Y_i$ and $Z_i$ are iid, then $R_i$ would also be iid and we'd simply have $f(R_1,R_2)=f(R_1)f(R_2)$. I'm wondering how things change with the conditioning on dependent RVs? Surely it can't be something as simple as $f(R_1,R_2)=f(R_1|X_1)f(R_2|X_2)f(X_1,X_2)$?
Isn't it actually the case that sudden jumps in prices of essential commodities *cause* wars? :-)  In fact, I believe you will find that the price of oil increases *before* the onset of many wars.
Also, your first rendition of $r$, with a single sum, is the correct one. The $\\mathrm{S}$ notation is explained at the beginning of the book.
@GavinSimpson Thank you for correcting my understanding. This exchange has been very helpful to me. +1 (:
The graphviz tool you use makes a difference. Graphviz's sfdp or twopi might be good choices. You can also specify or override the locations of some nodes if you dive into the dot file.
$(22+15+10)/3$ would work but you might want to scale the scores (e.g. divide by the number of points possible) because the 45 point scale will contribute more weight than it should purely because there are more points possible
Hardin & Hilbe is quite good.
Can you provide some explanation/intuition with the code? Having the code only does not fully address the OP's concerns.
(+1) By the way.
I've bought a copy of "Pattern Recognition" by D, H, and S. It really is spectacular, and the whole thing is useful and easy to follow. Thanks again. This is exactly how SE is supposed to work :)
I really don't know why this post was down-voted because this answer actually helps. I will vote it up again. Thanks greg!
"market basket analysis" seems like it's what I'm looking for, thanks for input.
Please tell us more. How many variables do you have? What is your sample size? What are your hypotheses and research questions?
+1 Great idea about the subset selection! I think this is the link you're referring to @shoda: http://stats.stackexchange.com/a/554/9249
(+1) I admire the really careful and clear way in which you have posed your question.
That makes sense; approach one would work for ordinal data whereas approach two would be more suited for nominal data. Thanks for the insight! (Would upvote but not enough karma).
@Edo Or days. C'est la Gibbs.
If you come across more such publications, I would be glad if you could share them here.
I find it very unfortunate that terms get used in different ways; it makes it harder for people to understand these topics & communicate w/ each other. You seem to be using "nested" in the sense of *nested model tests* (checking if a version of your model w/o a covariate fits significantly worse than the full model), whereas more typically, "nested logit" might refer to something like modeling the probability of students passing a test, when those students are nested w/i classes, which are nested w/i schools, etc. It's worth being clear on this distinction b/c it can be quite slippery.
Thanks very for your answered! Everything clear!
I see by the looks of the graphs that you are using SPSS. Just open the graph to edit and find "Add fit line button" there you find some _nonlinear_ line-drawing options, e.g. Loess. Check if this option gives you a reasonably straight line.
@tseo So you may accept his answer
Have you tried asking Google itself?
on the R style, you don't need `;` on the ends of your statements. You did right to pre-allocate, but no need to fill with `NA`, `kvals <- numeric(length = n_trial)` would have sufficed. With `rep`, you only need arguments 1 and 3 from your call (e.g. `rep(NA, 10)`). In the `for` loop set-up, `1:n_trial` can be dangerous if programming; better is `seq_along(kvals)` or `seq_len(n_trial)` in this case. Finally, if you don't need to force printing, drop the `print()` round `summary()` - you'd only need it if you weren't working interactively with R. HTH.
When comparing models using likelihood-ratio tests, you can compare different *random effects* structures using REML (restricted/residual maximum likelihood, as above), but you must use ML (maximum likelihood) to compare different *fixed effect* models.
@Steven: You're welcome.  I'm not sure how to move questions; I think moderators can do it.
@cbeleites if I understand you right, you suggest to integrate the error in the classes' distribution functions. However, the explicit form of the distributions is unknown, and only the true class labels in the test set are given. Thus, this method is inapplicable. Please clarify if I misunderstood the comment.
I think that [this](http://stats.stackexchange.com/questions/5782/variance-of-a-function-of-one-random-variable) might help you.
@yoavram For any given number of trials I believe they contain the same information about the parameter.
Hmm, I think we are having different objective functions; let me stress that I agree: Elvis' answer is the "correct one". 
If one wants to find a new "Gaussian-like" sample something like: 
   E<-c(40,66,67,68,69,70,71,72,73,74,75, 90,91,92,93,94,100); 
   Cost<-function(pars,E){R <-bcPower(E +abs(pars[1])+abs(min(E)), pars[2]);
return(abs(skewness(R))+abs(kurtosis(R)-3));}
   Op<-optim(fn=Cost,E=E,method='SANN',par=rt(2,1));
   TranData<-bcPower(E+abs(Op\\$par[1])+abs(min(E)),Op\\$par[2])

(and then fixing SD & mean as Dimitriy said) will give a sample with the desired moment properties.
Could you tell us what your variables are? I could probably help you sort out which are independent and dependent!
Thanks! I was sure it was a well-known result, but I wasn't able to find it myself. I appreciate the reference too; academic writing was so much more colorful back then...
The answer's going to depend on what you want to do with them next.
+1, nicely explained answer. This reminds me of Jerry Dallal's [discussion of intention to treat analysis](http://www.jerrydallal.com/LHSP/itt.htm) (spoiler: he doesn't like it).
Could you share with us *why* you want to learn stats?  Curiosity?  Needed for a project or research?  Wanting to change jobs?  Need to teach some courses?  Want to collaborate with statisticians as the theoretical person?
Thank you so much for the answer, it was helpful.