tttjjj/elt_bench huggingface dataset:
No, this is not true. Try something like the following: Generate a predictor that is standard normal, calling them $X_1$. Generate a second predictor $X_2$ such that $X_2 = 0.99 X_1 + \\sqrt{1-0.99^2} Z$ where $Z$ is also standard normal. Now, make your model $Y = X_1 + 0.1 \\cdot X_2 + \\epsilon$. Everything is pairwise positively correlated, but you're likely to get a negative coefficient estimate on $X_1$ or $X_2$, particularly for small sample sizes like, say, 20.
you can find here a continuation of this discussion: http://stats.stackexchange.com/questions/19537/simultaneous-confidence-interval-for-multiple-proportions-in-r/19597#19597
Thanks for the code, I got the warning with my data, but it disappeared when I added in method="fn", even though it doesn't seem to make much difference to the results.

Defining linear models over the quantiles seems like a really nice approach and easy to interpret - eyeballing the summaries over the increasing quantiles I initially get a positive slope for x, then a flat slope in the middle, then a decreasing slope, exactly as I'd expect. 

Thanks for the idea, this is probably a good way forward and I never would have thought of it myself. Will read up on it :)
@Imran: I updated the answer accordingly. Hope it is enough to get started with coding.
Ah, and thanks for the book recommendation. Looks promising at a first glance.
I think that ou should just look at your model and think about it. The interpretation of the Decline coefficient is obvious.
@B_Miner I've never seen a good theoretical reason to add the eigenvalues into the calculation, but I've seen it improve predictive models, so I try it sometimes.
Are you just asking if there are alternative link functions (other than the logit) available for the multinomial case? Or, are you asking about different *types* of models (such as [tag:cart] models)? Or perhaps something else? (Side note, if the first, it might help you to read my answer here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909), for general information about this issue, although it was written in a slightly different context.)
@M.Berk  Although I did not downvote this answer, I believe it to be incorrect.  It *does* reflect a valuable insight, however: Monte-Carlo integration can be improved by removing a portion $T_0\\subset T$ of known measure from the target region $T$ and using Monte-Carlo methods to estimate the measure of what remains, $T-T_0$.
@ars. Thanks for that.
Thanks, I'll try this path
Welcome to the list. Are you  asking for statistical help (that is, what is the right way to find the association between these variables?) or Stata help? If the latter, then this would be off topic for this list, but it sounds like you want the former.
Can you solve the problem for the special case $a=0$?
Dear @niandra82, I decided to open a new question given the different problem I would like to explore: http://stats.stackexchange.com/questions/107656/gibbs-sampling-inserting-some-known-predictors
This question is now redundant. My statistics teacher sent me an answer saying that I was correct and he was wrong.
How many participants (i.e., animals) do you have? If the ratio of animals to missing data is too big, this is not that good. In all cases (e.g., after using mixed effect modeling as proposed by Mike Lawrence) you should compare the results of the data with listwise deleted cases (i.e., all animals with missing data omitted) with the modeled results...
+1 for Kolmogorov. Without him, a rigorous treatment of most fundamental statistical concepts would not be possible and therefore statistics not as reliable as they are now.
@whuber I agree, I posted that as a math trick how to use binomial->Poisson asymptotic to quickly approximate solution to such a problem. That's why its a comment and starts with a disclaimer.
+1 Very nice answer.  Nitpick: in the last line you need to convolve *scaled versions* of $F_X$.
Wow thank you so much Nick!
Agreeing with Dr. Flom, I'd just add that a category cannot be significant or not significant. It is difference between groups (categories) that can be significant. In case of regression, each category is compared to reference category.
Could you elaborate your hypothesis, e.g. by a formula?
"i.e. you accept a probability of 5% to call something significant while it isn't" is not completely clear. The test will still be significant, but you reject a true H0 in this case (given that H0 is indeed true). Otherwise "significant" would be a property on the population level... I also noticed that "random variation" or "chance" is often ill understood. Maybe it could help to specify in a form such as: the deviation of the sample statistic from the expected value under H0 is due to sampling error only...
I was considering simplifying the entire ordeal - my goal is to demonstrate a relationship between the frequency of the events and the polling results, and the issue is that the polls were taken at such irregular intervals. Could I reasonably just divide the number of events by the number of days since the last poll and plot the poll data against the number of attacks/day in a simple scatterplot and avoid this problem?
Yes, this looks like it's just a small typo.
Shep, this is probably the right site for such a question, but it lacks details needed to come up with an answer. Could you please tell us more about where these "arbitrary discrete distributions" come from, why you're trying to fit them to some (family) of distributions, what this family is specifically, and why you believe ML is the right approach.
To extend the rule of thumb: d should probably be small too as one ought not *over-difference* the data.
Depends what margins were fixed in the sampling scheme & whether you want to make arguments from (approximate) ancillarity to justify conditioning on those that weren't. Comparison of unconditional vs conditional power is not meaningful.
I suspect (based on not much information ;-) that the asymptotic distribution would be reached rather slowly for many underlying data distributions.  More importantly, the correlation simply is not the covariance; PCA on the two can differ radically, especially when the variances of the variables differ.
At bottom, the question is a great one: "What does an MoE mean for Likert scale responses?"  May I point out, though, that (1) this question is discussed extensively in many other questions here; and (2) the rest of the question is confusing.  For instance, it's mathematically impossible for an average of 7 integers to equal 6.67.  Second, what does "38%" mean as a MoE for responses from 1-7?  Third, where does this 38% value come from?  Fourth, perhaps this question is really about missing values rather than about MoE's?  (If missingness is not at random, it's hard to say much here...)
An alpha level of 0.025 would correspond to a 97.5% confidence interval -- an alpha of 0.05 would correspond to a 95% confidence interval (i.e. the 0.05 rejection area is split between the two tails of the t-distribution.)
@Diego Indeed. I'd say the statement "In neuroscience the true effect size is not known" is generalizable to most of the life and health sciences.
It would be great if you could say a little more about your data/design so we can really help you to decide which test is the best for your specific problem.
@MichaelChernick The name is the same, the game totally different. I wouldn't use data from 1980, let alone 100 years ago. Also the rules are different (for example you can't waste time by constantly returning the ball to goalkeeper hands) as well as the point system (2 points for a win, now it's 3) which makes a draw much less attractive outcome. As for additive/addictive, I do make such mistakes occasionally, unfortunately I can't edit the comment.
This question seems to belong to the Computer Science forum rather than the statistics forum.
I think reporting df is what to do; it's also what I usually see in "regular" reporting of $\\chi^2$
There is an old joke about this situation: it asserts the chance of "sharing some weird connection with any random person" is exactly $1$. The reason is that if you discover you do *not* have any weird connection, *that is itself a weird connection.*
@cardinal: I wanted to make the point that in case of non-independent data, Bonferonni _overcorrects_ and is way to conservative.
Could you be more specific what particular method I can use? I am not depended on linear regression only, I can use another method if is more suitable.
...And this is because we didn't have to deal with, explain and forecast, an economic crisis like the recent one.
I solved "a" as following but I do not know if it right or not :        if there are 4 particles in 1 ml then there are 12 particles in 3 ml . but in this case should I consider the value 12 lambda ?
This is a rewrite of http://stats.stackexchange.com/questions/77814/how-to-do-regression-between-likert-scale-data-and-time-series-data It is best to rewrite existing questions, not to post revised questions as new questions. Now that you have done this, please delete your first question.
thanks! very clear
but I've amended my question, the question was less "what is it" as "what else can you call it"
@oort, I am unsure what you are asking.  McNemar's test (or extensions to it) work on paired data (so 2 variables, in your case before and after).  The original test only dealt with 2 levels in each variable (yes/no, high/low, etc.) but the extensions allow for more than 2 levels (low/med/high/veryHigh, etc.).  Proportional odds logistic regression also uses a response variable that is an ordered categorical variable with 2 or more levels (again it can work with low/med/high/veryHigh).
Considering that the batches are not random, whatever test you perform on the outcomes will only tell if the difference between batches is statistically significant or not, but you won't be able to tell whether this is due to previous exposure to a specific diet or due to any other characteristic that is not equally distributed among batches - and this can be anything, from tongue length to temperature during first weeks of life... If you want to proceed anyway, google "Fisher's Exact Test".
+1 Thank you very much for your time. Can you also comment on the bootstrap technique explained in the post here: http://stats.stackexchange.com/questions/28873/goodness-of-fit-test-for-a-mixture-in-r When I tried this approach, I got a 0.111 p-value. I don't quite understand whether it is fair to use the bootstrap approach in the first place.  About the red/green curves - point taken. R base did something weird there. Thank you.
... guess I'll stop writing my answer (+1)!
@Chase I just rediscovered this very nice response from @Gavin about the [Interpretation of R's lm() output](http://stats.stackexchange.com/questions/5135/interpretation-of-rs-lm-output).
The first sentence is an excellent summary of my advice. The second, a question, brings in too many different aspects to be easy to answer, but within the space available: Plenty of accounts may imply otherwise, but in my experience there is usually a major element of judgment about which variables (factors, in your terminology) to feed in. I would distinguish sharply between PCAs and the classifications that might be based on them. PCAs often imply that classifications are of limited use because the pattern of variation is a continuum.
I mean $\\rho(x,y)=(x-\\bar{x})^T(y-\\bar{y})/\\sqrt{(x-\\bar{x})^T(x-\\bar{x})(y-\\bar{y})^T(y-\\bar{y})}$ where I wrote x rather than $x_i $ and y rather than $x_j$ for brevity.
Please see my comment to Tenali. The correct answer is independent of the distribution as long as it is continuous and the random variables are iid.
Here is the reference: Dunlap, W. P.; Cortina, J. M.; Vaslow, J. B. & Burke, M. J. Meta-Analysis of Experiments With Matched Groups or Repeated Measures Designs Psychological Methods, 1996, 1, 170-177. https://www.dropbox.com/s/1bbz20lehbff1cc/dunlap-1996%20Meta-Analysis%20of%20Experiments%20With%20Matched%20Groups%20or%20Repeated%20Measures%20Designs.pdf
Hey whuber, this method proved far more effective than I was expecting! I altered your code to find the volume signal(s) across the full stretch of expansion data (~40min = 60k data points @32hz = 180k @100hz) and it picks up each one with surprising precision. In the interest of speed (and now I'm curious) would you mind talking me through a Fourier transform implementation or even point me in the right direction? This is my first foray into signal processing but so far it's been fun!
This may explain why the normal distribution is often used in problems but as Douglas Zare commented nto all machine learning problems center around random components that are normal.  So there are problems where other parametric distributions should be used or nonparametric methods should be applied.  The question says "always".  I doubt that machine learning analysts always use the Gaussian distribution.
Nicolas, division of the data into "high" and "low" based on an underlying continuous quantity is not a good idea, and you will have residual variation hidden by that approach.  I would develop a combined model using the most raw continuous form of the moderator variable.
The minimum sample size *for what purpose* ?
Could you elaborate what the anomalies are?. April 7th and May 28th seems to me to be just following the trend.
thanks for your contirbutions
Another convenience item associated with normally distributed data is that 0 correlation implies independence.
@AlexWilliams if you share the data used for those plots I can try to answer your questions directly using the method in my answer below.
If I get you correctly, you're speaking about the amount of increase of the explained variance when the regressor is added to the rest ones. I believe they often call it "importance" of a regressor.
(+1) Looks great.
By "can't not", do you mean simply that you *cannot* use `PROC REG`?
"that the composite score reflects the concept it is measuring" That is much closer to _validity_ then to _reliability_. Perhaps you ought to rename/retag your question.
Could you explain more about why hypergeometric is the right distribution? It's not clear to me why what you describe is analogous to sampling without replacement.
Hi Greg:  1) Thank you for the answer.  2) I love your loess.demo function...
To explicate: we can consider three different scenarios: testing the frequency of P(H), testing the autocorrelation or testing the occurence of certain strings, such as HTHTHT. They're three different hypotheses. In the second case, 1010101010 is suspect, because of the perfect correlation between throw n and throw n-1.
Using zeros in place of missing values would be a mistake. Regression assumptions are likely to be violated; one would expect serial correlation and seasonality in the data.
You might more detail about what your goals are.  In general, like @rolando2 I encourage emphasizing predictions and confidence intervals rather than hypothesis tests.
You mix things up. The setting for clustering is this: you have n points (which you want to cluster) in d-dimensional space. In your case one point is observation vector of one site for one time in parameter space. So n=10*24 and d=18.
Why are you looking at the correlation matrix (which is mainly useful for diagnostics) and not at the coefficients and their standard errors? Your model specification seems correct to me.
Did you take a look at this question: [how-exactly-does-one-control-for-other-variables](http://stats.stackexchange.com/questions/17336/)?
Well, I do not know enough about your context to give a sensible comment but you have to model the velocity vector as a time series of some sort with an additive error term which you assume is iid. You estimate the model and test if the residuals are iid. Perhaps, you could ask another question with some context of the data and your goals are as far as data analysis is concerned. Perhaps, the issue of iid or not is not that relevant given what you want to achieve?
This is all fine. An early reference to a more general result which includes this one is Ellison (1964, J.Am.Stat.Assoc, 59, 89-95); see Corollary 1 of Theorem 2.
Another useful link: http://stats.stackexchange.com/questions/1385/techniques-for-handling-incomplete-missing-data
I'm using the riskratio function from the EpiTool package, manual attached here:
http://bm2.genes.nig.ac.jp/RGM2/R_current/library/epitools/man/riskratio.html

I'm still digesting your answer, thanks again!
The question is about correlated errors, not about correlated predictors. Could you try to reconsider the answer accordingly?
In general predictor selection is a bad thing to do. To help understand why, you may want to read my answer here: [algorithms-for-automatic-model-selection](http://stats.stackexchange.com/questions/20836//20856#20856).
I a point about exact methods that should be noted is that in the hypothesis teting framework they produce a sawtoothed power function.  This is because of the discrete nature of random variable like the binomial.  It also leads to erratic behavior for confidence intervals.  A number of methods that smooth thing out have been proposed.  Lots of articles on this in the American Statistician including mine with Christine Liu and a comment on it by Jeff Longmate.
This question is weird and I love it.
@Emily, data transcription errors aren't the only form of measurement error. More generally, Karl was probably also referring to errors in the measurement apparatus itself. If your measurement tool randomly had a very large mean zero value added to it, then you might want to delete extreme cases from the dataset. Where you get into trouble is when, for example, your measurement tool only reports high values inaccurately (in which case you would have bias).
One might take a radically empirical approach: forget all the nonsense about p-values and "significance;" for the Local Moran's I (and most local values) they aren't worth much anyway.  Throw $I$ into the mix of independent variables and see what happens!  If it proves to have good explanatory (or predictive) power, then why not use it?
Btw., you days should either be in R's `Date` type or integers. Personally, I usually prefer working with days since start of measurement.
@COOLSerdash Sorrt, somehow I missed the question about calculating the mean. If it's normal on the log scale, then conditioning on knowing the parameter values, you'd be computing the mean of a lognormal ($\\exp(\\mu + \\frac{1}{2} \\sigma^2)$). If you don't condition on at least the variance-parameter, the exponentiated estimate is instead log-t ... and then it doesn't have a mean.
Could you please clarify what you mean by "total uncertainty"? Are you referring to some measure of variation in $y_i$ or in the estimated parameters $\\hat{a}$ and $\\hat{b}$? What measure of variation are you using?
Yeah, but what does $P_t$ mean? Is it the t'th step transition matric or the transition matrix at time t? Or are these two things the same?
@Deathkill14 Yes I have, but how would other characteristics help me compare weekdays?
A few thoughts: (1) One measure of variable importance in RFs can be computed from a permuting scheme which is well described in any textbook or on Breiman's RF website. (2) It is safer to perform feature selection in a single shot (and within a proper cross-validation scheme), not using iterative search that will likely lead to overfitting; otherwise, consider using an hold-out sample. *Sidenote*: I imagine you already came across the following paper: [Evaluation of Decision Forests on Text Categorization](http://ect.bell-labs.com/who/tkh/publications/papers/textcat.pdf).
@Glen_b will keep that in mind
Internally studentized residual as defined in the formula I gave, yes. If you're unconvinced, you could compute an internally studentized residual from the available regression information directly and compare.
Yes. I later find the non-negativity constrained L1-regularized problem can be solved as a constrained quadratic program. I also find there is a solver written for such program as I update in the question description.
@petrelharp No, I know what the eigenfunctions can be used for (description of spatial or temporal pattern in multivariate data for one), but I was wondering if this approach had been developed elsewhere, under some name that I was not aware so I could read up more about this approach.
You have obtained confidence interval not for females (average spending), but for the difference between males and females (with respect to other features in the model).
Of possible interest: [A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment](http://biomet.oxfordjournals.org/content/83/4/835.abstract). In this paper the authors present a method for comparing two techniques for diagnosing melanoma. The problem I see in your context is that you have triplets $(x_i,y_i^1,y_i^2)$ and therefore it is not clear what is a fair comparison between the methods. I think you have to provide information about how are you constructing the survival curves.
This looks like a standard problem set for students. Is this homework, or otherwise related to coursework or a question from a text or test? Do you know the general approach for dealing with such questions? What region do you need to integrate?
Of possible interest: [Assessing reliability of a questionnaire: dimensionality, problematic items, and whether to use alpha, lambda6 or some other index?](http://stats.stackexchange.com/q/11628/930), [Omega vs. alpha reliability](http://stats.stackexchange.com/q/39182/930)
@John I would keep it as a statistical question
Just one last comment.  The one-sided version of equivalence is called noninferiority.  Often drug manufacturers can bring a so-called "me too" drug to market that does not even have to be as good in efficacy as the marketed competitor as long as it is not much worse (defined by a delta that is a window for noninferiority.  This can be done because the product offers some other advantage to the consumer, like fewer doses per day or as in the case of an anticoagulant not needing to have INR monitored.
C is the penalty factor introduced to control the trade-off between the hyperplane complexity and training errors. An other alternative could be the use of the bias b as a threshold since b is the distance from the centre of the feature space to the separating hyperplane. So varying b is like moving the hyperplane and thus having different TP and FP! This is my understanding!
Hi @Zach, I still do not understand, how can I **simulate** data from the conditional distribution given the values of the predictors. I want neither the predicted classes nor the probabilities. I want to simulate data. Thanks.
Attempts ending at an obstacle are probably very bad even without obstacle, leading to unhandy informative censoring. Otherwise good idea.
Count the equals signs, sometimes. The more there are, the more omnibus-like. However, I guess scope is the important thing. But I don't think there is consistency on this. Often names, as here, are just sticky labels: you can peel them off, or they drop off, and nothing's different underneath. I don't think there's an idea lurking there except that of an overarching test. You might find a definition lurking somewhere. This is just my impression: corrections and rebuttals may follow.
Gavin, in your first paragraph, do you mean "decision tree" rather than "random forest" in the first sentence? (Apologies if I'm daft.)
The probabilities are independent. I tried to come up with a scenario that would help explain the problem but I failed to think about a possible correlation of a person falling and then possibly falling again because they just fell.  Nice catch.  Just insert X and Y where needed, :-)