tttjjj/elt_bench huggingface dataset:
Thanks for your reply. It seems lucene is a toolkit for search engine, while mahout includes some implementations for large-scale machine learning algorithms. I found both of them lack sufficient documentations, and also examples. :-(
Do you have a link that would describe how to do this in detail?
@AdamO makes a similar point to #2 here: [positive linear regression coefficient](http://stats.stackexchange.com/a/89775/7290), but develops the idea much more fully.
The frequentist null hypothesis does not express maximal ignorance, it starts of assuming that the null hypothesis is true and we should only proceed with the alternative hypothesis if the observations are sufficiently unlikely under H0.  It might be argued that null hypothesis testing encodes some prior, but it is a decidedly informative one.  In my opinion attempting to interpret frequentist hypothesis testing in Bayesian terms is misguided and a recipe for error; they are not answers to the same question.
Mr. Sarwate, if you want to help me you can check my another related question here: http://math.stackexchange.com/questions/234324/finding-probability-of-psw-at-bayesian-network-of-rain-problem
@user Well, that simplifies things considerably!  I thought that by "3D axes" you meant a full set of orthogonal local coordinate axes at each point.  Instead, you really just have an angle $\\phi$ to the local normal.  For uniform angles, the distribution has PDF equal to $\\sin(\\phi)$, $0\\le\\phi\\le \\pi/2$, and CDF equal to $1-\\cos(\\phi)$.  The mean is $1$ (radian) and the SD is $\\sqrt{\\pi-3}$ which equals $0.376288$.  Thus, you should apply your second method but compare $\\phi$ to *this* distribution.
+1 on Steyerberg and Vittinghoff. (Though I thought the Vittinghoff chapter of variable selection needed a bit more work and Steyerberg is focused on prediction modelling) I still have to get around to the Harrell book.
All other comparisons are meaningless.I also have 3 different regions that I want to consider all these comparisons in, but they are separate so it doesn't make sense to compare across them.
In fact SPSS calculates partial eta square for all ANOVAs. This will give the same value as eta squared in single IV Independent Groups Designs, but a different value in single IV repeated measures designs. This causes no end of problems with my students.
@Michael In this case, it is explained in this post: http://stats.stackexchange.com/questions/17022/the-plot-suggests-its-true-is-it  I am trying to come up with an alternate plan if I am unable to solve the relationship mathematically.
16 day data comprise data about every day? present a sample
I believe that a really crazy P will give a lower p-value half the time and a higher p-value half the time, and, moreover, the two p-values will be pretty near each other for data obtained in practice, very near if there really is a substantial difference in the means. The bottom line, I believe, is a general rule of thumb, that, provided the pairing is done in the way I specified in my question, one can, in practice, only profit from a paired t-test, and this is an entirely legitimate profit. Can anyone give me a counter-example?
Examples of 1 are shown in Wooldridge Introductory Econometrics section 13.2
Thanks - I am using MATLAB and have implemented a similar approach using `mle` and `ncx2pdf` etc but it is extremely slow so I am looking to implement a more efficient approach myself.
+1 great answer.
It should be noted that the statistical properties of real-world time series data (especially financial data) can vary depending on the sampling frequency. For many financial time series there is no straightforward $\\sqrt{T}$ relationship between the standard deviations sampled with period $p$ and with period $pT$. In fact the standard deviation tends to increase as the sampling frequency increases. Similarly correlation decreases as the sampling frequency increases (this is commonly know as the [Epps effect](http://en.wikipedia.org/wiki/Epps_effect))
@Bitwise We welcome questions here that ask for creative methods to visualize and analyze data. This one doesn't sound like it's just looking for some canned code.
Thank you guys for the suggestions.  Back to the drawing board...
About the permutation approach, see @whuber's answer to this related question, [Look and you shall find (a correlation)](http://stats.stackexchange.com/questions/5750/look-and-you-shall-find-a-correlation), which describes the basic idea of breaking the structure or link between your two series of measurements.
Please make your title a little more detailed
I doubt that a t-test is suitable here from several points of view: (i) It's *time series* data; you're unlikely to have the assumed independence; (ii) it looks like your data might actually be *counts* (they appear to be non-negative integers at least), not normally distributed random variables. What is the underlying research question you're trying to answer here?
I could use soemthing like this http://www.ats.ucla.edu/stat/stata/faq/proportion.htm but how can I correct for autocorrelation?
It depends. Naive Bayes models make conditional independence assumption between variables, while models trained with logistic regression don't.
There's been a bit of research on number of choice in Likert scales, but it's been a long time since I looked at it (15 years or more). Way back in the mid-90s I did a master's project comparing the factor structure of a test giving with two different scales and found that the factor structure was a bit different.
Dear @Pete - the Spotfire Analytics Server integrates into your existing information and analytics technology infrastructure, including databases like Oracle, IBM DB2, Microsoft SQL Server, Microsoft Access, MySQL, SAS/SHARE, Sybase, Informix or any custom JDBC data source
Are you familiar with compartmental models used in edpidemiology (http://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology)?  Your description sounds similar to SIR models (Susceptible, Infected, Recovered) albeit your data sounds more complex.
**(1)** Acquiring more knowledge and experience is only going to *improve your chances* of getting into a PhD program. That said, a finance gig is not gonna be your most efficient path to a PhD program. The most efficient way into a PhD program is to demonstrate an ability to do original research, by publishing papers or getting great recc letters from advisors . **(2)** Again it could only help but unless your finance job is to do original research then I wouldn't expect any dramatic benefit in this regard, so I wouldn't give this factor much weight in your decision process, if i were you.
(+1) 1: there is no evidence of a significant difference in means, so your first suggestion looks applicable. 2: the reason for using inverse-variance weighting is that this is the minimum-variance unbiased estimate of the mean.
(+1) Interesting! Which software do you use? Just a remark: There's no indication about absolute values for % or counts, so this seems to allow only a relative interpretation.
I tried your method, but the spread was bigger than my method above, and I wasn't real sure that I did it right.  I generated random sigmas using the following code `sigma <- sqrt((datpoi - 1) * (sd(ori)^2)/rchisq(n=newdatpoi, df=datpoi-1))` where `datpoi` is the number of original data points (in my example, datpoi=10).  And, `newdatpoi` is the number of random values wanted (in my example, newdatpoi=100). Those `sigmas` and some random means (centered around `mean(ori)` ) were fed into `rnorm()` to generate the new data points.  As far as the Bayes approach goes, that's out of my league.
@Aleksandr The chi-square ignores the ordering in the data, so loses a lot of power.
+1 great question. My instinct would be to use 16 as the maximum rather than 100 (`cbind(accuracy, 16-accuracy)`), but I'm concerned about whether it's mathematically justified.
Your *data* doesn't reject anything, it's just numbers. I assume you applied some test. Wouldn't a structural break invalidate the test assumptions? But yes, you can write models that have a unit root *and* a structural break, so data could have both.
One important point is that the moment-generating function is not always finite! (See [this question](http://stats.stackexchange.com/q/32706/2970), for example.) If you want to build a general theory, say, about convergence in distribution, you'd like to be able to have it work with as many objects as possible. The characteristic function is, of course, finite for any random variable since $|\\exp(itX)| \\leq 1$.
I appreciate all the comments on notation, but it is taken exactly from Sampling Techniques, Cochran(1977) which is the 'bible' for sampling theory. 

Also, all I am trying to do is work through the proof that the sample mean $\\bar{x}$ is an unbiased estimator of the population mean $\\bar{X}$.
I'm using a KDE as a plugin to a Bayesian classifier for the moment, though I've used the Naive variant too. Regarding why I'm trying to model this distribution at all, it's basically an exploration to see if the data can be summarised succinctly: maybe even used for subsequent prediction without having to resort to Bayesian methods.
(+1) Excellent point.  This dataset contains no external or objective indicator of true protein level, making it impossible to identify which is "better" in the sense of accurately reflecting the protein level.
I'm not sure I get your explanation in the comment. See, for example, the first diagram [here](http://en.wikipedia.org/wiki/Simpson%27s_paradox), which relates to omitted variable bias. What is it you're proposing to do to eliminate the problem?
predict is in the stats package in R.
The estimates are found as described at http://stats.stackexchange.com/questions/107597/is-there-a-way-to-use-the-covariance-matrix-to-find-coefficients-for-multiple-re.  With these in hand, any test you want is obtained in a standard fashion.
What happens when the $2$ comes from one group and the $3$ comes from the other group you are comparing it to?
I didnt realize this would be so difficult.  I recently read somewhere that lmer cannot do this type of heterogeneity and that I might need to use the nlme package instead. Can anyone confirm this?
Thanks, Joel. You're right, in that I could compare those. However, is there anything I can do with the other(s) (relating to total-household or adults-only)? Something to compare is better than nothing, but more would also be better.
(+1) Welcome to our site! I searched it for an existing answer to this question--which is pretty fundamental--but turned up nothing suitable.  All our questions on least squares seem to assume everybody knows exactly what it is or else they provide technical descriptions in the language of algebra and calculus.
The same also applies to the book *The Elements of Graphing Data* by the same author. Buy both of them; they are both excellent.
I fixed up the formatting, but do you really mean to have $C_0$ five times? This would be most unusual; I think you want $C_1$, $C_2$ etc.  Also, you didn't include an intercept; was that deliberate?
@MarioModesto, you might consider accepting Flask's answer (green checkmark under the vote arrows). Tks
LDA can work with 20 features, no problem. But when you ask "how can I apply LDA", what do you mean? How to do it in a particular program? If so, which one? Or are you interested in the mathematical procedure? Your question is not specific enough.
-1. Do not shout at a man trying to help you.
Alright. In order to show that the fraction $\\frac{\\bar{X}}{\\bar{Y}} $ follows an F-distribution, it suffices to write it as $ \\frac{\\frac{2\\sum X_i}{2 \\theta_1 n_1}}{\\frac{2\\sum Y_i}{2 \\theta_1 n_2 }}$ , right?
Thanks for all of the help. I've now gotten quite a bit of both practice data and match data. It's been interesting to see how the process has changed since I asked this question e.g. Now that we have averages for different stats/skills, we can use those to more accurately rank players.
I am not sure that reinforcement learning is appropriate here. Question for clarification: Do you can get feedback for an observation only once OR can you elect it for feedback multiple times ?
Expanding on @whuber 's question: If you have real data then you do not know what is an outlier and what isn't; you don't know if something is a data entry error or isn't; you don't know if that person whose height was recorded as 17.8 meters was really 1.78 meters or was some other error.

Simulating lets you control all this. Then you can test things and know if you get the right answer.
Perhaps if you stated why you're subtracting them and what makes subtracting the first two values of the columns any different from subtracting the first value in column A from the 6th value in column B.
Love LASSO. Worth a look. Will also help prevent overfitting.
@Wesley, note that I only claim that I found these books useful. Concerning Hamilton, it is a huge book, and probably some chapters are not good. On the other hand chapters on non-stationarity I found very good. In general I use this book as a reference, these type of books can be overwhelming at first, but in the end they are most useful, that is why I included it.
The key ideas are all clearly displayed.  There is a tiny (but interesting) error, though: your conclusion can be false when $x=\\theta$.  For instance, let $f_\\theta(x)=(x-\\theta)/2$ for $\\theta\\le x\\le\\theta+2$ and $f_\\theta(x)=0$ otherwise. Then $F(\\theta)=0,$ implying the limiting value of $F_{X_{[1]}}(x)$ is $0$, not $1$.  This is interesting because it highlights the importance of invoking the definition of *support* somewhere in the proof and because it reveals the nature of the convergence is a tiny bit trickier than you might think.
To put it another way, the MLE can be viewed as the maximum a posteriori estimator (MAP), just with an improper prior (when the integration allows for this), and MAP is a more compelling point estimate since it incorporates prior information. Now, instead of just picking the mode of the posterior... why not combine all values of the posterior according to their prior probability? It won't give you a point estimate of the parameters, but most often people don't really want a point estimate. Distributions over parameters are always more useful than point estimates when you can afford to get them
Hi @Lindon, your answer is more of a comment as it stands now. Perhaps you could try to give a little more detail instead of simply referring to a webpage.
Could you please point out some code to graph what you described?.  Your answer is really useful as I didn't know where to start.  I was thinking to use   
 `
library(ggplot2)
library(reshape2)

data <- read.table( "dta.txt" , header=T, sep='\\t', comment.char='-', quote='' )
p + geom_point( aes( x=day, y=RunningTime, colour=significant) )
dev.new()
tmp <- p + geom_histogram( aes( x=day ) )
print( tmp )`
Can you please say how the book is called where you took the example explaining the cointegration? Thanks!!
Whether a formula is needed depends on how you define correlation. Some of us define it as a property of a pair of variables that is invariant under (linear) changes in their units of measure. (That allows us to reduce the formula to the case where the variables are both standardized, for which the correlation is the expectation of their product.) From that it is immediate that the correlation of $X$ and $Y$ is the correlation of $X$ and $X$ and you're done. Thus, for this question to be answerable we need to be told what definition of correlation you use. (cc @Alecos)
+1, very good answer.
+1 Good points.  Welcome to our site, Dominik!
This agrees with my back-of-the-envelope simulation findings. Thanks for the thorough answers!
In this case this might be the best way to foster understanding of the problem, even though some would call it a little less "elegant" statistically speaking.
I will look into the reference range. Best means approaching the values distribution as bes as possible. What I get are flat and boring curves, almost straight lines. The expectation is to get more wild curves with expononential growth/decline, sine wave like or something in between. Some amount of random noise has also to be thrown in for good measure.
It is. Could you elaborate on the difference between medoids and component-wise medians?
@Chris: For the model I wrote down, the model residuals will be mean-zero for each rabbit, because each rabbit has its own factor. So the two methods would be equivalent.
A small example data-set might help to clarify things. And are you interested in predicting e.g. the duration of a mouse move given the task type, or the other way round. Are the three task types of interest in themselves, or just representative of various task types?
Dear Dan, Thank you for your answer.
Correlation is a weird statistic, highly nonlinear in both sample means. I seriously doubt that you can push through any exact conditioning results.
I don't really understand what you are asking. Could you try and reword your question?
What's your definition of "equivalence"? Is the SDE Ito/Stratonovich/what?
@Aksakal So??? Juho was answering a _slightly different_ and much more recently asked question. This new question has been merged with the one you see above, which I answered ten months ago.
This is awesome!! Thank you!
@mpiktas The data are perfectly random and mutually independent, *but the orderings of the rows and columns are not.*  In some sense this is a multivariate analog of what happens when you sort a sequence of iid variates: they are still random, but the sequence now has a pattern!
Aproaches 1) and 2) are both quite hard, as they require me to design either a meaningful distance or a feature space (which is essentially the same task). I'm awarding the bounty for the third answer, as recurrent neural networks and reservoir computing are unknown concepts to me.
@JacobMick, Yes, but could you please give more details? I don't know how to start.
About your search for "the correct" test: it's better to look for "a" correct test, or even better, a "reasonable" test. There may be several reasonable things one could do, each with their own disadvantages. That said, there are always "wrong" things to do, and it does sound like the CMH function is "wrong" for this data.
Your question is covered in many threads: try searching our site on [significant regression](http://stats.stackexchange.com/search?tab=votes&q=regression%20significant).  (ANOVA is an application of least squares regression.)  For instance, http://stats.stackexchange.com/questions/14500/how-can-a-regression-be-significant-yet-all-predictors-be-non-significant/14528#14528 provides an explicit example and some intuition.  Please research these and edit your question, if possible, to distinguish it from previous threads.
(1) Is the Normal distribution an assumption or do you have good evidence in support of it?  (2) Is the problem that you cannot accurately count the data near the center?  (That would be different than the usual meaning of "censoring," which is that you *can* count those data but you know only that their values lie within certain intervals.)
See http://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs for a previous question on a related topic. It should at least address all your questions related to subject A.
Stata manuals come bundled with the software in pdf format. Type "help st" and click on the blue link "[ST] st" at the top of the page that comes up.
You can't group subjects based on the timing of a future event. You have to do a forward-looking analysis. The fact that a subject could have gotten medication in 2005 but did not, and finally got it in 2011 is relevant to 2005 as well as 2011 (and all the years in between). You have a time-dependent predictor. It is difficult to make plots for it, but SAS can handle the analysis in PROC PHREG.
+1 Great answer and a good book recommendation. There are several other good books on extreme value theory including the classic by Gumbel and the books by Galambos and the one book Leadbetter, Lindgren and Rootzen on the extension to stationary stochastic processes. A new and very readable recent book is the one by Stuart Coles. It is worth mentioning that the cumulative cdf for the Gumbel distribution exp(-e$^-$$^x$).
I would compare with some other implementation, preferably part of a published package so it is more likely to be correct.
@Ben The reason why most panel data articles don't mention normality is that it is a very strong assumption turns out is not needed as long as you have a large number of observations.  I don't know of a reference that deals with this for the particular case of panel data models, but you could check out Bruce Hansen's Econometrics text book [link](http://www.ssc.wisc.edu/~bhansen/econometrics/Econometrics.pdf) and compare the assumptions made in the Normal regression model (Sec. 3.18) to the weaker OLS assumptions (Sec 4.3).  The same reasoning would apply to panel data models.
What kind of GEE is this (e.g. Gaussian, Binomial, Poisson, etc..)? Do you have access (or can you calculate) to the covariance matrix of the parameter estimates?
I see.  There should be another bracket around (1/score-1)
Maybe we're thinking about different meanings of standardize. I meant the one I linked to where, for instance, if one population has a stdev of 5 and the other has a stdev of 10, after standardization they would both have a stdev of 1. They could then be more fairly compared with the same bin size since each bin has a comparable amount of pixels and data. Or maybe you were getting at the larger issue that "appropriate bin size" is a bit of a black art and unique to every data set...
Right, so $f(x,y) = 0$ when $x \\neq y$. What about when $x=y$?
@wolfies This doesn't look like homework to me (for several reasons, including that the various questions reflect efforts to attack and simplify the problem). Regardless, *we shouldn't care whether problems are homework.* Policing student honesty is not our task. What matters for us are the nature and quality of the question: when a question is so obviously of a routine textbook nature we treat it a little differently than others. Although there's no clear dividing line between routine and non-routine, this one strikes me as being on the answerable side of that line.
Thanks for the reminder Galit, we will keep the big picture in mind.
@PeterFlom - Not all stat programs do create dummies for a regression procedure, nor do they all automatically create, for regression, all the cross-product variables that would be necessary to comprehensively test interactions.  But for ANOVA the procedure is often much simpler.  E.g., entering Age*Academic Discipline in SPSS ANOVA tests all interactions of a 3-dummy variable and a 2-dummy variable:  6 product terms that for regression per se would need to be computed and then separately entered.
(+1) raegtin, you ask nice, interesting questions. I am fond of BT models, but it seems a tad forced here. Are you familiar with the collaborative filtering literature? It's not the same problem, but some concepts and techniques could be borrowed. A question left unaddressed here is why one should believe the books can be given an unambiguous, well-defined ranking in the first place. (How would you handle the two-user, two-book case, for example?)
[Yes](http://mephisto.unige.ch/pub/TraMineR/doc/TraMineR-Users-Guide.pdf). Users guides are your friends.
which estimates?
Could you provide the lm() function from R that you used to estimate. How exactly did you use the contrasts function?
I somewhat get it now.  We want to minimize the negative exponent, so we can maximize the likelihood.  The requirement is $\\theta \\le x$, so we pick the smallest x possible, x1 in our case.  Am I anything near to be correct?
@Carol I've assumed that condition is within-subjects and groups is between subjects. is this correct?
i've added the r tag so the R code is highlighted in my comment