tttjjj/elt_bench huggingface dataset:
+1, just FYI, I notice you write your own function to standardize your data, you may find [?scale](http://stat.ethz.ch/R-manual/R-devel/library/base/html/scale.html) helpful.
Which post hoc test would you conduct? There are only two levels. Any statistically significant result would demonstrate difference between the two levels.
Nicely put, but it duplicates http://stats.stackexchange.com/a/1371 posted three years earlier.
Are $X$ and $Y$ independent? If so, what is the variance of $Z=X+Y$?
you should train the classifier on a different criterion. Since your number of instances of class a vs. b is so unsymmetrical, you should optimize the AUC criterion rather than accuracy.
It seems to me that you are describing an autocorrelated dataset.  Perhaps if you go looking with that keyword it will help you find an answer you feel is suitable.  An off the cuff approach might be to use the K previous values as predictors of your K+1 value, the residual would have drawn away most of the variance that is associated with the non-independence of values and yield a reasonable estimate of variance without dependence.
Have you cross-validated with and without it? What about holding a separate "calibration" set of data that learns the linear model from the $\\hat y_{held}$?
@ronaf Could you provide more detail about $D_n^{(r)}$?  What is R?  I don't see that permuting the categories does anything at all: notice that no permutation will change the sum of absolute differences of their counts.
@cryptron I see.  So what you need is a way to dynamically identify outliers.
ok thanks, stating the statistic was a mistake. I meant z or t table.
I hope you don't mind that I added an answer.  14.7 is correct, but the recurrence relationship is still flawed…
I think he really just wanted the standard deviation of s.
Do I need to do that to the target variables? I mean the normalization to the target variables? Is it necessary for the ys as well.
@EpiGrad the models are different, but run on the same data
I discuss the flip side of this issue in my answer here: [estimating b1x1 b2x2 instead of b1x1 b2x2 b3x3](http://stats.stackexchange.com/questions/58709//58712#58712).
@user333, your question was about logistic regression, if you want answers about ordinal regression too, please add that to the question.
@Imorin. I have included everything from the slides
See this question: http://stats.stackexchange.com/questions/8485/a-good-gibbs-sampling-tutorials-and-references
I also found one of the papers you reference "Learning from Crowds" to be very helpful  http://www.umiacs.umd.edu/labs/cvl/pirl/vikas/publications/raykar_JMLR_2010_crowds.pdf
Effect-size weights may be calculated as a function of their (the effect size) standard error. Bootstrapping is one way of [estimating the standard error of an effect size](http://en.wikipedia.org/wiki/Bootstrapping_(statistics)#Advantages).
I am curious why we should care how a model is derived when we are assessing its quality.  (Putting aside the problems with assessing model fit using the same data employed in computing that fit...).  It seems to me that if you view RMSE as a valid measure of fit, then that's all that matters: the inner workings of the model should be immaterial.
You may find this thread of interest: [Removal of statistically significant intercept term boosts $R^2$ in linear model](http://stats.stackexchange.com/questions/26176/).
Yes, quite separate.  Prediction means you are not needing to specify a loss (utility; cost) function and the end result for the analyst is the predicted probability of an outcome.  Classification means that you are explicitly or implicitly assuming a loss function (how?) and are making the decision for the consumer/reader.
Regarding a *physical* explanation of correlation, remember that it is always also possible for a correlation to be real, but [spurious](http://en.wikipedia.org/wiki/Spurious_relationship). Some of the examples here: [correlation-does-not-mean-causation](http://stats.stackexchange.com/questions/36/) may be interesting.
Also please tell us your current thinking on this question, c.f.: http://stats.stackexchange.com/faq
Is survival analysis not used also in the context of probability of reoccurrence of cancer in cancer patients? Would this not be an example of an event happening more than once and survival analysis still being used?
I admire the effort to present an elementary (Calculus-free) proof. A rigorous one-line demonstration is available via the [weighted AM-GM inequality](http://en.wikipedia.org/wiki/Inequality_of_arithmetic_and_geometric_means#Weighted_AM.E2.80.93GM_inequality) by noting that $\\exp(H)$ = $\\prod\\left(\\frac{1}{p_i}\\right)^{p_i}\\le\\sum p_i\\frac{1}{p_i}=n$ with equality holding iff all the $1/p_i$ are equal, QED.
thanks you! the data is sparse and do have high degree of self-correlation with median around 0.55.T here is open question for me how to measure the quality of randomization for such data. Is correlation enough? if we have the some degree of self correlation how to treat the prediction of the Y-scrambled data?  If we have R2 of normal data is 0.85 and Y-randomized 0.55 (median) is it good or bad? where is the threshold?
Fair enough, but your question is still unanswerable at present.
This related thread might help: [Multiple regression with no origin and mix of directly entered and stepwise entered variables using R](http://stats.stackexchange.com/questions/9503/multiple-regression-with-no-origin-and-mix-of-directly-entered-and-stepwise-enter).
I'm still not very clear about how the aim of the study, research questions and survey questions relate to each other. At a glance it does not seem like GLM, ANOVA, MANOVA would be relevant because you have few variables and no hypothesis. An internal consistency measure like Cronbach's alpha does not seem appropriate either because the 5 questions seem each to be measuring different things. Do you only have these 5 questions or is there other data from the same respondents?
John Tukey advocated a useful exploratory graphic for evaluating symmetry. It essentially plots $(\\log_2(i),(x_{[i]}+x_{[n+1-i]})/2)$ for $i=n/2,n/4,n/8,\\ldots,1$ where $x_{[i]}$ are order statistics. This is more powerful than any single test because a quick visual evaluation--looking for vertical deviations from the ideal horizontal plot achieved by a symmetric distribution--not only identifies asymmetry but can characterize its nature and localize its position within the distribution. Since your question sounds exploratory in spirit, this is something you might want to contemplate doing.
This is not exactly your question, but this thread: [interpretation-of-log-transformed-predictor](http://stats.stackexchange.com/questions/18480/) may be helpful in thinking about these issues.
So the `Deviation` is geometric, and you want to visualize it in arithmetic space? I see no problem with your obvious solution of multiplying deviation by traffic.   You could also plot the `Traffic` values with error bars.
It's a wonder you don't have this figured out yet, considering your name.
The problem with this use is the same as with other uses: The test will be dependent on sample size, so, it's essentially useless. It doesn't tell you whether you can use z scores.
Sure, I would report the lower and upper limit. For a real-world example, see this [BMJ article](http://www.bmj.com/content/342/bmj.d2496.full.pdf?sid=68bea816-b11e-4b8d-8c03-ed763306adb5) (p. 4, Table 2).
Appologies.. The data is the angle (degrees) of eye fixation away from a common point. My two categories are male and female. My hypothesis is that males look at this point more than females. Therefore, the male category having very small angles and the female category having larger numbers.
Thanks both for your answers. Could you please direct me to a book/reference where I could find a proof of the fact that for large samples the paired t-test works?
Since you haven't pre-specified equivalence regions, I'm not so sure about standard (Confidence Interval- or Two One-Sided T-Test based) equivalence testing. You could possibly use a Bayesian hypothesis test that can "accept the null", such as this thing which allows you to just enter the t value, n, and press the button: http://pcl.missouri.edu/bayesfactor

Alternatively, you could compute the 95% CI of the difference between 1 and 2 in standardised effect sizes (Cohen's d). If it only contains "small" values, you may say with some confidence that 1 and 2 are unlikely to differ "much".
Depends on the sample size too. For really small samples, a dotplot is really probably the best, because densities, bin frequencies, quantiles (or hinges), etc. are just not knowable.
@Nick Thanks for help with terminology. This is a problem of multiple modes in a distribution. The proposed constraint on minimum distance is an arbitrary idea to narrow the problem that can be dropped.  What matters is finding the "modes" which we were calling clusters.
@Macro, thanks for your edit. Can you please tell me how to use such equations on this website?
Please read again, the comparison you're asking for is not meaningful. The samples generated via MC are synthetic, artificial, only as good as the model they come from and therefore need not to adhere to the real data, let alone improve over it. It cannot predict anything more than the chosen model. You cannot compare MC to ML techniques: these generate descriptive/predictive models, MC *requires* a model from the start and does not give you one.
Is one of them a "null" value? If not, you could just directly compare likelihoods (if you don't have to formally test).
A worthy option for your consideration is a generalized linear model.  Please search our site for threads on GLMs.  If you still want to transform the response, then search for threads about transformations, logarithms, and regression: many of them explicitly discuss whether and how to add a "start value" to the data before re-expressing them.
Actually I think this is a model independent approach, you might indeed apply it to other classifiers than random forests. In Breiman's website there is a subtle remark about how you could compute the variable importance for a single case. http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#varimp (last sentence) I think that this hasn't been extensively studied yet, or at least extensively tested. The mean variable importance is not always what you want. For example it is not when you want to help a practitioner to take a decision on one case. This is a really interesting topic.
@octem Couldn't the investigator trust the statistician to do that part of the collaboration.  Why does the investigator need the tool.  I liken this with the physician to reversing the tables. How would he feel if I said give me a quick tutorial on how to do this surgery and I will go ahead and do it for you. I think shock and dsimay and it is illegal for me to practice mewdicine without a license. That is probably a good thing.  But doesn't the statistician deserve equal respect.  Why expect that I can just give him the tool and let him hack away with his lack of training.
"Good" for what purpose?
You also might like to see the recently-published: Ren, Shiquan , Lai, Hong , Tong, Wenjing , Aminzadeh, Mostafa , Hou, Xuezhang and Lai, Shenghan(2010) 'Nonparametric bootstrapping for hierarchical data', Journal of Applied Statistics, 37: 9, 1487 — 1498
@Glen_b : Thanks for the comment. Sure! The data I'm having is 2 month worth of tweets. Each tweet contains a timestamp, and the author's id. What I'm trying to model is the user's daily activity in terms of posting a tweet. E.G. the data for a single user would looks like [9:00, 9:12, 17:00, 17:01, 22:22, 22:37, 22:45, 22:47, 22:48...]. So this user post more tweets during night (around 10:30 pm) but very rare during work hours. I wish to model the probability P(user post a tweet|time).
Experimenting more, both with different feature sets and with different learning algorithms, is a good idea.  Also, based on your problem description, you may want to add features for appearance of each individual number (bag of words, not just bag of trigrams): if you use only trigrams, you are making it harder for the machine learning algorithm to learn facts like "sequences that contain 11 almost certainly do not have the property".
@DouglasZare Assuming that the unedited spelling was "indexes", then "indexes" actually gets a lot of use in economics literature. Quick browse of English forums on this matter reveals that both are correct.
SESTAT seems really cool!  It was able to answer my question very easily, and was exactly what I was after.  Thankyou.
If the Matrices are observed, does it mean the filter is still an optimal filter?
A Chi-square test pops up in all sorts of different situations.  For example, testing for a relationship between two categorical variables in a cross tab; or a more general case of comparing any observed distribution with a null hypothesis distribution.  Is there a particular use in a small sample case you're interested in?
[linear svm](https://en.wikipedia.org/wiki/Support_vector_machine#Linear_SVM)
It's not clear what you compared C with ... but more importantly, it reads like you worked out which hypotheses to test by looking at the data. If that's the case, then you can't say much about significance at all, because your p-values will be wrong.
thanks; very good idea, to compute the ranks on only one of the variable.
I [googled](https://www.google.co.uk/#hl=en&output=search&sclient=psy-ab&q=The+empirical+mode+decomposition+and+the+Hilbert+spectrum+for+nonlinear+and+non-stationary+time+series+analysis&oq=The+empirical+mode+decomposition+and+the+Hilbert+spectrum+for+nonlinear+and+non-stationary+time+series+analysis&gs_l=hp.12...357.357.0.1069.1.1.0.0.0.0.0.0..0.0...0.1.aaXYF4jcrp8&pbx=1&bav=on.2,or.r_gc.r_pw.r_qf.,cf.osb&fp=1deb3a27a4d9b1ab&biw=1440&bih=770) the [original paper](http://keck.ucsf.edu/~schenk/Huang_etal98.pdf), it has $5789$ citations, $1900+$ in 2010-2012. It seems to be quite popular.
In such a case, you might have to check your result to see whether such a random factor is important or not.  You can look at the variance between each subject and use a better  Covariance matrix if necessary.
@Steven See my response in my post. I would also take a look at Censored Least Absolute Deviations (CLAD) as a possible alternative.
Community wiki!
@Emlie You have a lot of variables where you could explore different relationships. I suggest asking another question on this site specifically focused on how to analyse a particular aspect of your data.
Yea this should be the accepted, really fantastic answer
Thank you very much, I have no idea why I couldn't get my head around it..haven't needed to use contingency tables for a while!
@ John : I understood that till the matrix of eigenvectors part, but after that how do you conclude $X^TU = VD$ ?
@radek I'm glad I was clear. I don't have an answer to the solution you've provided. I think the results from `margins` would make sense if linear combinations of parameters were first computed on the log scale then re-exponentiated to report hazard ratios. @andrea's suggestion of using the `lincom` command instead is something that I would echo. That command has never given me unexpected results.
,thank you for your comment.  
Yes, the replicate data will become a problem when it was labeled with wrong label.  I have search again the information about replicate data after read your comment.  Thanks again for your explanation. :)
I am trying to calculate the expected pass/fail rate for the entire population, and within each of the categories. As part of that, I need to know what the prediction interval is for the entire (un-audited) population, and within each category. "Appropriate vs inappropriate": minimum absolute number of samples audited in each category.
@B_Miner perhaps you could help me parse out what n(mYmZ) means in wuber's answer?
@Jeromy: Yes, I've changed it. @whuber: You're correct, I've tried to make the probability a bit clearer. BTW, integral->integer in your comment.
Can you amplify your question (by editing it: the edit link is in the lower left), by describing: (1) the *numerical values* taken by 'lead source' and your demographic variables; (2) the *study design* that produced these measurements; and (3) the *causal model* informing your hypotheses?
I was going to say "Shame on you" for using "affects" instead of "links with," "is associated with," etc.  but then I realized the variable certainly could affect what odds are *obtained* (i.e., printed out).
Can someone tell me why i would get a downvote for my response?
Google "Sheppard's corrections" for the standard solutions to this problem.
I'm not sure how to answer that question, but you could say that the methods are not the same but one of the underlying methods is.
Thank you, that was exactly what I needed. I thought there was an easy way out of this, and it helped me solve my actual problem that lead to this question.
Thanks! This is very helpful your plot especially is giving me new insight in the interpretation of my results, I appreciate you bearing with me as this is my first post..and yes this is a small sample of the dataset.
That's a first step but that is usually not the problem. At the moment I cannot find an error in the code. It is really strange that the activations become so big. Maybe there is a problem with your data? Do you have an unequal class distribution, e.g. 90% of your data are 1?
Could you show a scatterplot of the data? Maybe there are gross outliers that influence the estimated correlation.
If you are throwing $n$ balls into $10^6$ bins where each bin is equally likely then the number of balls that end up going into a specific bin has a $\\text{Binomial}(n,10^{-6})$ distribution.  This can be approximated by a $\\text{Poisson}(n/10^6)$ distribution, extremely well approximated if you are throwing millions of balls.
@MiroslavSabo has the essential answer. However, we can do even better by taking the pattern of relationships amongst the variables into account.  That is, you should calculate the [Mahalanobis distances](http://en.wikipedia.org/wiki/Mahalanobis_distance) & take the max of *that*.
+1 Great answer.  But did you read the penultimate paragraph about homework and the desire to avoid a complete solution?
Suppose one of the IVs is an affine transform of the DV itself, so that $R^2(1)=1$: that seems to contradict your progress so far. Now suppose all the IVs are orthogonal to the DV, so that $R^2(k)=0$ for all $k$. There doesn't seem to be much left to say.
Francesco First, see @Whuber 's question; he has a different take on it, one I hadn't thought of. If you mean it his way, then he may have more advice. However, if your situation is more like the one I envision (that is, you  actually *know* who went to which party) then there is no dependent variable and hence no regression.
So the idea is that under the lognormal setup the sigma indeed cancels out in theory, but for a finite sample it  doesn't. Even if the data generating process is homoskedastic, the observed variance is not going to be the same among cells unless there is a large number of observations per cell. To back out the observed ratio of arithmetic means from the parameters, one needs to perform manual estimation of cell variances by hand to plug them into the numerator and denominator.
Thanks. That's really useful.
"*I'm looking is a simple formula that would recursively dampened the trend like the exponential smoothing dampened trend method.*" -- again, *yes*, as soon as you tell me the method. Dampening forecasts is not one size fits all, as I have explained before. What's suitable for ordinary exponential smoothing is not necessarily suitable for  triple- exponential smoothing. What's suitable for seasonal series is not suitable for random walks, and so on. Give some kind of framework, and we can get somewhere.
Somewhat related: http://stats.stackexchange.com/q/6907/2970
@ars (+1) Many thanks! I missed this package... Always better to look at the code directly, I appreciate.
@Matthew I am looking for a measure that can ensure logistic regression model (made on an older data) fitting a new data-set. I want it to alert me when the model is not predicting accurately so that I can re-model my regression model to new data-set. 
What I meant by accuracy is that a new score should be able to indicate the error between observed (values of new data-set) and predicted values (by the model trained on older data-set)
I'm not familiar with this package but this appears to be saying that there are too many ties in terms of which neighbor is nearest. This can happen when you have a very discrete variable (i.e. it only takes on a few unique values) you're trying to cluster on - is that the case here?
What do you expect us to be able to say without seeing the data?
People sometimes do this - unknowingly (ie misuse Statistics, because they get the desired result) and knowingly (they did bootstrap and it did not affect the result substantially). Your point is valid, and Professor Harrell points this out in the Preface of his book that bootstrap is beneficial.
You'd probably like the **Poisson** videos over at Khan Academy--check them out.
Can you say something about the topic of this video, so that people don't have to watch it first to know what it's about & whether it was worth their time?
Re: "Linear Regression is used to establish a relationship between Dependent and Indipendent variables" - this is also true about logistic regression - it's just that the dependent variable is binary.
@NickStauner Okay, moved to an answer at your suggestion, and slightly edited.
Do you mean to ask what distribution it would follow *if* something about $x$ was true? (i.e. some null hypothesis).
It is possible that RMSE values for both training and testing are similar but bad (in some sense). So how to figure out based on data properties if the RMSE values really imply that our algorithm has learned something?
Thanks for your comment. I do realise this is not proper k-fold cross validation - I merely borrowed the title from that post. I'm not actually doing training/classification though so there's no standard error calculation - I use each fold to maximise a scoring function and take the parameters which results in that maximum score. I obviously need to prevent overfitting of these parameters though so need to apply some form of cross-validation.
Marc, positive definiteness of a real symmetric matrix is equivalent to having all positive eigenvalues. There is no close relationship with "gappy data," understanding that to mean sparse matrices with many zero entries. After all, the very sparsest among the nonsingular matrices (symmetric or not) are the diagonal ones, which exhibit their eigenvalues as their entries.
A quick comment on using offsets -- these are in most instances vital to having an appropriate model. I tell students that the need for the offset variable is to compare rates of events between groups [or covariate values] rather than comparing the absolute counts of events -- with the former almost always being our question of interest.
+1 for **EB ignores the uncertainty in the hyper-parameters**. Also, Bayesian fundamentalists consider EB *anti-Bayesian* because using the data for estimating the prior is *blasphemous*.
@whuber If the OP is looking for the ways the term is used ("is it just a catch-all" ... seems to suggest that kind of question), then pointing out that it is used in several different specific senses as you did there may constitute an actual answer, or at least partial answer.
I have thought of fitting a panel count data model with a DD specification but I wonder if it is feasible.  In a basic DD setup you basically compare two means. I do not know how this works in  a negative binominal model. Expecially I am unsure because I could not find any reference paper using a count data model with a DD specification. any hint on that?