tttjjj/elt_bench huggingface dataset:
@Charlie $\\operatorname{cov}(Y_t+W,Y_t)=\\operatorname{cov}(Y_t,Y_t) + \\operatorname{cov}(W,Y_t)$ **does not** require independence: it is a property of covariance that holds generally.  $W$ and $Y_t$ are independent because they are sums of two _mutually exclusive_ sets of independent random variables. Generally, if $X$ and $Y$ are independent, so are $g(X)$ and $h(Y)$ for _any_ (measurable) functions $g(\\cdot)$ and $h(\\cdot)$. If you do not already know these concepts. you are way in over your head when you look at covariance stationarity and random walks.
Is it that "no correlation could exist between possible unobserved factors and your measurement", or that the correlations that exist between other variables & your measurement in the population are accurately reflected (to the degree afforded by sampling error) in your data? Certainly *age* and *height* are related to each other whether your sample was random or not.
Graph and plots can actually be great to gauge the size of an effect in more intuitive ways than effect size measures. If you actually *see* the overlap between two groups on some measures (which would roughly correspond to a smaller *d*), it's perhaps easier to realize that a significant difference does not mean that member of one group all have lower scores than members of the other group, etc.
I happen to have detailed code (in R) showing how to do this precisely for the Nile data. I use it as an illustration for my students. Unfortunately it is in Spanish, but I hope the code is pretty clear (and I can translate the comments if not). You can grab this example from http://www.et.bs.ehu.es/~etptupaf/N4.html.
First is the issue of a mixture of skewed and normal data, should I use a test based on medians that is more robust to skewness for all variables?

Also, I'm thinking I should treat the observations (feedback/ no feedback) as paired samples for each subject at each timepoint, but I'd also like to quantify any significant change in the variance over time.
I've got a few ideas about how to do and implement this, but would appreciate having some data to play with. Any ideas on that?
(+1) I am unable to obtain the same conclusion from this information.  Is it possible the website includes additional information that might have a bearing on this calculation?
Graphs > Legacy Dialogs > Error Bar ...and... Define > Bar represents ... Standard Deviation, and Multiplier 2. However, it is clear that this question should migrate to Stackoverflow.
Sorry I dont think that is correct. The zeroinfl model has 2 sub-models: count and zero. When you indicate count in type you are not accounting for the zeros predicted from the zero submodel. From what I have read, type=response, accounts for both submodels and gives the prediction on the response scale.
@ran2, adf test is designed to test whether the stochastic trend is present. Although series with deterministic trend are non-stationary, regression analysis with them is "safe". Also detrending is meaningful. With stochastic strend the usual detrending methods will give misleading results.
@ttnphns This is done all the time is areas like psychological research (but much more widely than that); that's after all exactly what a [Likert scale](https://en.wikipedia.org/wiki/Likert_scale) is - a sum of items from ordinal items to produce a composite scale. It makes an implicit assumption (that the category intervals are equispaced), but in practice it's done when there's no particular reason to think it's the case. If there are many related items it seems to work quite well.
"*The reason you can't take the mean is because you don't know if the gaps between categories are equal.*" --- however, if you regard it as acceptable to add the individual items together, you have already assumed this. If you can add Likert items, you can calculate means of said items. If you regard calculating means as unacceptable, you have no basis on which to add items.
To add to@danas.zuokas's answer:  the eigenvalues describe the variance accounted for among the 50 variables used in PCA, but this is before the dependent variable (Y) was ever entered into the analysis.  The eigenvalues and loadings have nothing to do with relationships with Y.
(+1) That looks indeed like the right keyword to search for!
@AndrewCassidy yes!
Basically what you are trying to do is segmentation in your examples "leader"/"laggers" or risk taker/risk averse. Data analysis can only reveal only some behavioral insights, what you might want to do is consult a domain knowledge expert or do a qual/quant market research to determine attitudinal preference on "why" they did. Also it might help if you could provide a simple example for couple of customers.
Have you looked at either of the papers by Littlestone linked on the [Wikipedia page](http://en.wikipedia.org/wiki/Winnow_(algorithm))? There is definitely a mistake bound proof in them, although I don't remember if it is of the form your looking for.
I don't know the answers but would like to comment.  Relapse-free and recurrence-free sure sound the same. I of course agree that death can not be considered a recurrent event unless you count someone whose heart stopped and was revived.  I think death makes sense to use as a censoring time because it prevent the recurrence of many future events for the subject.
Thanks for pointing this out @Glen_b, I've changed the question statement
No, not quite. The problem is that transitions are not entirely independent so a change from 100->110 is much more likely than a change from 100->111 but a change from 110->111 is (probably) exactly as likely as a change from 100->101, so once a population contains members of type 110, the formation of members with type 111 becomes much more likely. This, I think, means that the _proportion_ of the final population that are of type 111 is not independent of the proportion of type 110.
Point (1) might be relevant: http://stats.stackexchange.com/a/55859/7071
I've considered this. The two variables are also constrained between 1 and 25, meaning there is a total of 625 possible solutions. It would make sense to do an exhaustive search over the entire solution space, but the cost function used to evaluate each pair of variables takes on average about 5 minutes (less for smaller pairs, more for bigger pairs). I thought I'd use the GA to try and selectively sample the best points in this solution space, so that the time taken to search it is reduced.
Can you add some matlab code for your issue? Usually matlab offers a way to use cross-validation within a classifier rather than just one train/test run. You should also check that test and train sets are randomly selected rather than referring to one portion of dataset
If your data represent number of items recalled of some total number of items presented, it would be better to represent your data in its raw binary format, labelling each item as remembered or not remembered and conducting a mixed effects model while specifying binomially distributed error.
Thanks for the explanation.  So then it seems that I have interpreted their results incorrectly as I thought they had run their experiment multiple times over the entire dataset.  I realize now that the confidence interval is based on a single experiment over the dataset.
I just noticed Dr G's comment [here](http://stats.stackexchange.com/questions/13326/can-i-use-kolmogorov-smirnov-to-compare-two-empirical-distributions) which suggests that ks.boot is what I need, but I still wonder whether dgof::ks.test's simulation option would work, and what "convervative" means.
[Other question](http://stats.stackexchange.com/questions/33639/what-am-i-measuring-when-i-apply-a-graded-response-model-to-the-hunting-of-the) is up.
Answers like this one are why I keep coming here.  Thank you!
@mpiktas That's correct, but SAS seems special IMHO, because it is *solely* for statistics (and, AFAIK, it is not ever used by non-statisticians: who could afford to?).  Where else on SE would one reasonably pose such a question and expect an answer?
When you add a constant to all values, that changes *only* the mean and not the SD.  When you rescale all values, that changes all moments but in a predictable way. In particular, it will not change the mean if the mean is already zero.  So, why not (a) shift the variable to make the mean zero, then (b) rescale it to make the SD equal to 10 (leaving the mean equal to zero), then (c) shift the mean to 5?
Interesting, there is a group from University of Alberta that claims the exact opposite. They think opponent modeling is key to writing a good AI, and right now they have the top Poker AI that I know of. Can you explain your reasoning for why you think otherwise a little deeper?
Thank you, karenu, Did I understand you right, you think that having one of variables as a vector with few thousdand elements will be ok for SVM? I usually use TF-IDF for clustering text, don't think I can use tf-idf in this case. -Alex
@svadali -  confidence intervals evaluate *data* for a fixed hypothesis.  Thus when changing the "fixed" part of the equation, if you fail to take account of the probability of the hypothesis prior to observing your data, then you are bound to come up with inconsistencies and incoherent results.  Conditional probability is not "constrained" when changing the conditions (e.g. by changing the conditions you can change a conditional probability from 0 to 1). The prior probability takes account of this arbitrariness. Conditioning on X is done because we are certain X has occured-we did observe X!
Superb...and FREE. What more could one want!
How about "An Interior-Point Method for Large-Scale L1-Regularized
Logistic Regression": http://web.stanford.edu/~boyd/papers/pdf/l1_logistic_reg.pdf. The abstract reports runtimes of "a few minutes" on a normal computer with a 1,000,000 x 1,000,000 model matrix.
This seems to give a little and ask a lot.
It is correct that scoring by average has problems (due to different variances of the averages), but **confidence interval scoring is not the solution.** From a statistical standpoint (which seems eminently appropriate for evaluating a purely statistical construct), it is an *ad hoc* abuse based on a misunderstanding of what CIs do. The financial people have a better, more flexible approach: they "score" based on a bidimensional "frontier" that captures both expected value ("alpha") in one dimension and a measure of uncertainty ("beta") in another. So maybe you're doing nothing wrong at all...
@Fiona.Ding Try the following link: http://stats.stackexchange.com/questions/13166/rs-lmer-cheat-sheet I think it will help you on how to structure your arguments for `lmer`.
@upul, don't reinvent the wheel (unigram ec), use alchemy's api - keyword extraction etc
... and there you are!
If you plan on interpreting the individual $p$-values, you should check for collinearity. Checking for collinearity is a frequently discussed topic on this website so doing a search for 'collinearity' will probably be fruitful. To look at dependence between categorical variables you can look at the usual $\\chi^2$ tests or something like that. In this thread,http://stats.stackexchange.com/questions/8088/assessing-multicollinearity-of-dichotomous-predictor-variables, the problem is discussed for binary predictors
Be careful!  You are fitting a model without a constant term: it's a poor fit to these data.  Also: what does it mean for you to "normalize" the error variance?  After all, if I tell you it's two miles from my house to work and somebody else tells you it's 3.2 kilometers, we're perfectly consistent: it's merely a change of units.  How should we reply if someone asks us to "normalize" how we report that distance?
As @Edward pointed out, please add some details.
Cross posting is typically discouraged, http://stackoverflow.com/q/12714591/604456
A form of gradiant descent (IRLS, for numerical stability). But, again, it will only converge to the local minimum that happens to be closest to your starting points. I also added some info to the answer...
One possible measure of smoothness is mentioned [here](http://stats.stackexchange.com/questions/69323/how-to-measure-the-smoothness-of-a-line/69432#69432)
(+1) Not too much handwaving going on here really. The geometric viewpoint *is* the most intuitive, in my view. There is likely to be a high-quality open-source figure out there that depicts things precisely this way.
That would be a very advanced analysis. If you aren't terribly familiar w/ stats, you may want to work w/ a statistical consultant.
(1) Removing the spaces between 1,2,N helps. (2) Beautiful pictures no longer take much effort or time; they only need facility with appropriate software. One thing about this site that initially impressed me was how *easily* people were able to offer great illustrations using `R`, typically needing just an extra (short) line of code. (To get the resulting image into a reply requires two short dialogs: one to save it on disk and another to load that onto the Web.) So, in practice, "extraordinary effort" often amounts just to an extra minute of work.
No answers or comments yet?  I will try to make this question in SO with a more specific coding focus....
KLR seems to be coming out of nowhere: I found the name KLR to be mentioned first around 2000 (for example in "kernel logistic regression and the import vector machine", Zhu and Hastie, 2001). But they say in there it is well known and reference smoothing spline literature. My Question is: When and where was KLR first introduced and why do they always reference smoothing-splines?
$ct$ is linear. $e^t$ is linearized if you present is as $ce^t$ with a linear constraint $c=1$.
One comment. We must pay attention to the parametrization because this algorithm crash near the boundaries of the parameters (alpha=2 or alpha=0 and beta=1 or beta=-1).
+1 Good points.  The *power* of the t-test with skewed alternatives, though, can degrade severely (to the point where it is essentially zero even for huge effect sizes).
It might help with the piurely numerical issues, not the statistical ones!  But, with modern numerical linear algebra algorithms, probably not.  Centering should be done if it helps interpretation of the model!
@pwerth: edited to respond.
@Cardinal Thanks for the clarification.
The selection of the hyper-parameters should not involve the test data in any way, as this will lead to an optimistically biased performance estimate.  Essentially choosing the hyper-parameters should be treated as an integral part of fitting the SVM, so the testing procedure needs to also test the error due to the selection of the hyper-parameters, see my paper that I reference in my answer to the question (it is open access).
The finite Dirichlet distribution is also a distribution over distributions, namely over multinomial distributions. That means if you draw a sample from it, you get a random distribution. A biased (weighted) die can be described by a multinomial distribution. A machine that makes biased die with some random error can be described by a Dirichlet distribution--a distribution over distributions.
Do you mean "Annual probability of at least one tornado"?
What happens in `R` when you initialize the solver with the solution found by Stata?
The main advantage of using `forecast::auto.arima` along with `tsoutliers` is that everything gets automated. 
However, it is advisable to run the automatic procedures with alternative options. You may first for example look at the ACF or unit root tests and then choose an ARIMA model to be passed to `tsoutliers`. If any outliers are found for your proposed model then you can repeat again the analysis for the adjusted series. It is an iterative process. The automatic procedure provides a helpful guide but it may not necessarily give the ultimate or unique solution.
How do you apply AdaBoost to SVM, do you change the weights of points in optimized function every time?
Here's a hint: A confidence interval is a spread of likely values. You are being asked for a threshold for unlikely values (5%)
I am accepting your answer for points 1-3. Much appreciated. On 4, I think you are right to say "does not require", but it is my understanding that increasing the number of baseline or endline measurements decreases the design effect and is therefore part of what makes the SW design efficient. Thoughts?
@Aniko I think we need to calculate p-values if the question involves inference about a parameter. But, the OP's qn is about the probability of observing a particular event. So, I think case 1 calculation is fine. If we want to compute the probability that we would see at least 39 such events then you would compute P(39 or 40) but perhaps I misinterpreted the OP's goals here.
@JonathanChristensen however, asymptotically it depends only on the height of the density at the median (well, and the sample size), so if the sample is large, it's not terribly complicated.
@PeterFlom No it's none you guessing. It's a confusion that if i consider the absolute value of t-statistic then my null-hypothesis will be rejected . And if i don't consider the absolute value of t-statistic, then i fail to reject null-hypothesis.
Actually my goal is to maximize the entropy of the output of an artificial neural network, by applying a multiplication to the weights, in order for the outputs to be more uniformly distributed. This weight-multiplication actually constitutes a one-to-one smooth transformation. My thoughts were that if the outputs are always nearly 0 or 1, the information content would be lower than when the outputs are uniformly distributed between 0 and 1. I don't share your intuition that $X$ and $Y$ contain the same information.
BUMP. Still hoping for an answer. Anyone?
How about using bootstrapping/permutation testing to test the null hypothesis:
H0: m1/m2 = m3/m4 
restated,
H0: m1/m2 - m3/m4 = 0;

I can see how I may bootstrap the difference in raio of means by sampling with replacement from each of m1,2,3,4 many times and recomputing the difference. But I can't see how to set this up as a permutation test to test the above null hypothesis.  What groups could I randomize across to assume the null hypothesis and obtain the distribution on which to test my observed value?
Thinking about it, I suspect, in real life, there is some correlation between the terms that you'd intuitively take into account (which is part of why you find the results strange), but which the naive Bayesian equation doesn't.  Ie, "news" negatively predicts (if that makes any sense) "football", but you use the same weights for "football" regardless of the presence of "news".
Are you aware of the [perturb](http://cran.r-project.org/web/packages/perturb/index.html) package in R? It seems to do exactly what you want.
(+1) Apologies. I'm not sure where my head was when I first looked at this. It's obviously (well, should have been!) correct.
I think there is some information missing from your question after the phrase "following format."
I think you'd be better off looking into the literature on telephone survey methods, market research, and how to increase response rates--topics that go far beyond statistics.  Also, your comment about a Bayesian approach is in "the spirit" of Bayesian thinking but does not strictly fit, since you would be using your knowledge as is, to guide implementation, rather than to be combined with observed data to yield a posterior distribution telling "what some quantity is likely to be".
this is the same as stepwise variable selection. see multiple posts on this topic elsewhere on this site
@Daniel; yes, given the Null device you have perfectly explained the response. Notice too the large standard errors, so that even though you predict the data perfectly, the Wald test is suggesting that the covariate is not significant. I believe the latter phenomenon is known as the Hauck Donner effect after the authors who noted this issue with the binomial GLM.
A very nice exposition on your thoughts working through the question, and you make the nk step explicit (my answer sorta just says 'clearly equal'). Cheers!
But aren't the values within each group autocorrelated and hence you have less information over the entire sample than you would if the data were independent?
I've updated my answer to address this.
What @sweezyjeezy means by linear thresholds, is linear decision boundaries. I think you misunderstood him. Thinking about it in 2-D, it may be that the observations that are labelled as `second category` are in the centre of the observations that are labelled `first category` so there is no straight line that can divide the two. Try running a decision tree and have a look at what the confusion matrix looks like. You can read [this](http://web.engr.oregonstate.edu/~xfern/classes/cs534/notes/decision-tree-7-11.pdf)
This question came up in comments to my answer - do all patients have revision times because revision is inevitable, or because only revision-needing patients are included? Those are very different problems.
@Frank Harrell Thanks. Overlaying is a good idea too. See e.g. http://www.stata-journal.com/sjpdf.html?articlenum=gr0018 for some examples in my work.
Intriguing question. I've know nothing about this so won't attempt an answer, but googling bimodality+test gives quite a few promising leads. 
http://www.google.co.uk/search?q=bimodality+test
Did you read [the paper](http://www.stata-journal.com/article.html?article=st0050) that introduced `rcal`? In Stata 13, I would do this using `sem`.
I suspect there is a misunderstanding here: "White" is the name of an econometrician (not the name of the color "white" used metaphorically in other contexts as in "whitening the series"), and "White standard errors" refer to the square roots of the variances of the estimated betas -NOT to some transformed residuals. So you don't need to "further correct" anything here. As the text says, the diagonal elements of this expression _are_ the "corrected" (heteroskedasicity robust) variances of the estimates, so you use them instead in t-tests etc. Does this help?
Thank you so much for your help.  I shoud add, I use Stata as well, but am not as familiar with the program.
Yes, but I mentioned it because I also think it throws light on some aspects of your discussion with whuber. When you condition on $x$, $x$ isn't regarded a parameter, the coefficients are. Unless you're holding the coefficients constant and varying the $x$ (rather than conditioning on a specific value). It's that need to be precise about what you're doing that may serve to clarify the issues.
Interesting paper, thanks for the link.
There is an answer already, so I'll close it for now so user4581 had a chance to access his answer when possibly answering your new questions. Then it will be deleted as you asked.
By the middle of the first paragraph you appear to confuse two separate things: the non-negativity of the responses $Y$ and the non-negativity of the parameter $\\beta$ of the regression  (I presume your "linear weight" is a parameter!).  In the second paragraph you assert that $Y$ is "latent" and then immediately claim it is observable!  Those inconsistencies appear to make this question unanswerable.
Could you provide your code and output? It is interesting to see `anova` or `reml` would produce negative variance estimates.
@chl, I meant every study. Assume the simplest case for my question. I don't know the difference between "continuous" and "categorical".
Samuel - usually \\` ... \\` is used when writing 'inline code' - inserting code in the middle of a sentence. If you want to put a chunk of code in, after you paste it in, you highlight it and then click the $\\{\\}$ button at the top of the edit window, which indents 4 spaces, to get the whole codeblock formatted as code.
+1 @guest, the code above is for R, which is freely available
... but with the "estimating a fixed effects regression", the fixed effects weren't in the original model as indicators, for example?  If you add new variables to a model, be they explicitly as indicators or implicitly as fixed effects, you're easily able to get sign changes in the coefficients of the variables that were there before.
I guess there is a way to do this with lmer, I just don't know.