tttjjj/elt_bench huggingface dataset:
I have the same problem and I can't figure out a solution.
I started by doing my meta-analysis in PLINK, but I want the confidence intervals of the meta-analysis, which PLINK does not give.
I then tried metagen() function in the meta package, but I'm struggling to write a loop, because the output is a meta class object, and I don't know how to extract the results. I would be happy with any of these solutions: -A way of getting the CI from PLINK
or
-A way of extracting the results from the meta object to a table. Thanks!
Mike, *all* predictions, if they are any good, will exhibit less variation than actual data, because a prediction cannot hope to project random variation.  "Generating more dispersion" perhaps can be accomplished with a better model, but beyond a certain point requires making poorer predictions.
@user2036690: possibly, yes. Still I recommend that you check first whether instability is really the problem, and after regularization check that things actually improve. *If* the problem is instability and your regularization is successful, you should also see an increase in hit rate.
I figured it cost me enough hair...there's no need for someone else to go bald over this problem :)
Can you say anything about the positive noise, other than it's positive? What can you say about $f$, if anything? Is it smooth?
The answer is not correct. Intuitively, the difference of two poisson distribution cannot be another poisson distribution since it can take negative values. The correct pmf is named a Skellam distribution with a rather complicated analytic expression.
Welcome to the site.  

First, it would have been good to think about how to analyze the data *before* you started collecting it. Second, what are you trying to find out?
... keeping in mind when we say 'better power' and 'worse power' that we're generally referring to power against what people generally regard as interesting sorts of alternatives.
Sure. I would listen very closely to the advice of gung and glen_b though. You should really be clear about what it is you're comparing. Significance doesn't exist in a vacuum.
If you do the general case later, I will study it. Only do it if it would be fun for you, but I wanted to let you know that it would not be wasted effort. Thanks!
Define "comprehensive". There's a multivolume series just relating to continuous and discrete univariate distributions.
As far as I know, you have to compare the models - that's what any package would do, anyway
To begin with, I suggest reading: Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7, 105-125.
Hi @StasK: Thanks for those comments. I didn’t realize that a Bollen-Stine bootstrap could be produced by AMOS (good to learn that). So, can one use the mean chi-square value produced by the bootstrapping for each model to then compare the models? For instance, one could then calculate the probability of the chi-square change in R to distinguish models? And is the mean chi-square value from the bootstrapping - along with the corresponding p-value - the figure to report in a manuscript (i.e. in the same way one would for a regular chi-square)?
Since the RMSE is calculated as sqrt(RSS/n-k) and RSS=sum[(y-hat(y)^2], it is calculating the entire regression model's RMSE.  hat(y) is the predicted y, and you already have y in your data.
Tax's PhD thesis "One-class classification -- Concept-learning in the absence of counter-examples" is also available: http://homepage.tudelft.nl/n9d04/thesis.pdf
In fact, I prefer to have 10-20 general categories, and 100-200 sub-categories among them. Then I guess at least 100 documents among every sub-category will help to train a reliable model, to predict massive documents (1 million for example). The classifier I am using is LibSVM, it works fine but I lack suitable training data. :-( I'll have a look at the tool you proposed. Thanks !
I agree with @Russell (if I interpret his comment correctly): to the extent that answering this question requires reasoning about data, it can be construed as on topic.  The text of this question, though, could be appreciably improved by applying the recommendations in our [faq] for posting good questions.
Thanks! This was very helpful. I read somewhere about how the interpretation of logistic model output differed from linear, but couldn't find detail comprehensible to newbies.
As I did not know about it before, I am not sure if "age-period-cohort models" is applicable for the nature of my problem, which is:
my dataset represents 20 observations (samples) belong to two different groups (10 samples per group) and the variables represent the expression of genes in each sample (gene expression matrix). I do not know if the concept of "age-period-cohort models" is still applicable for such problem?
Hint:  Since $P\\{X < Y\\} = \\frac{1}{2}$ (think about why this must be so), $Z$ has value $0$ with probability $\\frac{1}{2}$. Thus, $Z$ is what is sometimes called a _mixed_ random variable which takes on some values with nonzero probability and behaves as a continuous random variable for some values. As @whuber does, I too ask whether you have mis-stated the problem.  It leads to more complications than one would expect from a typical end-of-chapter problem at the apparent level of the book you are using.
Ok, what have you tried and what are you having problems with?
Perhaps me missing the point of overfitting has to do with my perspective. Our end goal is not predictive, but rather to be able to describe relationships between predictors. However, I think I may have assumed that they both go together (a good predictive model also gives reliable relationships between predictors). Would you do anything different if your goal is to explore the nature of the relationships?
The answer is `1/3` not `2/3` surely? Only one out of `GB, BG, GG`
@Momo Thanks, that seems like exactly the kind of example I am thinking of. http://en.wikipedia.org/wiki/Negative_binomial_distribution#Gamma.E2.80.93Poisson_mixture
But I am still confused: In that case, what is $f_{Y|X}(y|x)$ and what is $f_Y(y)$ ?
Welcome to the site! Do you mean that you want to make animal, trial, and test day as random effects (intercepts)? Are animals nested in trials, and trials nested in test days?
Possibly related reference: Barabesi L & Greco L. 2002. A note on the exact computation of the Student t, Snedecor F, and sample correlation coefficient distribution functions. Journal of the Royal Statistical Society, 51D, 105-110. For pdf-implementations in C / FORTRAN, also see [DCDFLIB](http://people.sc.fsu.edu/~jburkardt/f_src/dcdflib/dcdflib.html).
Interesting hadn't thought of that with the variance.  New customers have a higher variance so more news means not only worse performance but higher variance and it is definitely heteroskedastic...
Could you please share why you are planning to put constraints on the cluster sizes? As most people are interested in seeing the nodes of the dendrogram to decide the clustering level, so your intention sounds to be interesting to me.
Eigenanalysis of the covariance matrix is called Principal Components Analysis, or PCA.  Because your questions are fully addressed at http://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues, I will vote to close this question and invite you to follow the link to the related question.
I would appreciate if you could share how you analyzed the data, and whether you could interpret any combination of arousal-valence-dominance to connote a specific emotion like fascination, pleasant surprise, disgust, indignation, contempt, unpleasant surprise, dissatisfaction etc.
Percentiling of $X$s is inconsistent with they way the $X$s have their effect.  A common error is to percentile weight or BMI when predicting a health outcome.  The physics of weight dictate that it is the physical dimensions of a person that relate to their body functions, not how many persons in the sample who are below the one subject's weight or BMI.
Thank you! I did add a quadratic term in my model. The actual model is more complicated with multiple experimental conditions, but the principle is the same.
That type of data would not meet the proportional hazards assumption of the Cox model and would be better off modelled using a different assumed distribution.
(4) survey post-processing, weighting and variance estimation, which is a return to statistics -- again, this is a rather poorly covered topic in terms of existing books. My brief learning of this took about two years of two classes per semester. If you are determined to do a survey on your own, you will likely screw up the first one completely, you will miss a lot of good practices on the second one, and your third one will probably be somewhat usable. As a statistical consultant, I've seen all of that, trust me.
Welcome to the site, @AJC. Would you mind expanding on your answer a little? EG, it would help readers if you could give readers a couple sentence overview of the paper & the answer contained therein.
I'm not quite sure what you mean, but whether or not to log the data before using lme() is a modeling choice.  If you do that, it's probably easier to just plot the logged data than to try to interpret the lme() parameters on the original scale.
My hypotheses are:Differences do not exist between early education teacher interns’ perceptions of their preparedness to teach in kindergarten and their supervisors’ collaborative evaluations of the interns’ ability to teach. 
H1: Differences do not exist between early education teacher interns’ perceptions of their preparedness to teach in primary grades (first through fourth grade) and their supervisors’ collaborative evaluations of the interns’ ability to teach. See if the correlation between intern's perception of how prepared they think they were aligns with the evaluation.
Yes, better as a separate question.  I just fit the model that I want the residuals for, as above; we'll see if others have other ideas.
i suggest everyone put their favourite image from the blog, so it's not just a collection of links...
According to Figure 2, they are total births per month. Earlier the article states, "We looked at the number of births ... We analysed monthly birth data... ." But this is all beside the point: the article concerns a *post hoc* test of an anomaly that happened to be noticed. As with any such anomaly, such tests are meaningless. After all, I can look at *any* sufficiently long time series, model it, pick out the largest residual, and find any number of "explanations" in contemporaneous newpapers. *By construction* this residual will appear "significant." The whole exercise is pointless.
The things you're arguing about aren't contradictory terms. In fact, "parametric data" isn't even a defined term. I suspect you want to ask whether the data needs to be analyzed through non-parametric statistical methods.
Thank you @cardinal. I agree with all your comments except perhaps (3).
I actually made an error since $c$ is $-1/\\Gamma(-a)$ (edited), but the exponent
of $n$ seems the right one. I used the representation
of $\\Gamma(z)$ as found e.g. on Wikipedia page on *infinite product*
and took $z:=-a$ which gives an equivalent for the product $\\prod_{k=1}^{n-1}$. I
would be more confident if you could check this.
Traditionally, "multivariate" regression models are models with more than one *dependent* variables, not independent variables. Regression with more than one independent variables are typically called **multiple** regression models.
$X_3$ is random, but a deterministic function of other random quantities in your model. It therefore should not occur explicitly in the likelihood. In the context of graphical modelling, you would refer to $X_3$ as a deterministic node.
What kind of measure are you looking for beyond the 98% returned by the PCA procedure itself? @Peter In this case PCA allows the data to be recreated fairly accurately in terms of just five variables, rather than 14, together with a (fixed) $14\\times 5$ matrix to transform them into 14 seemingly different variables. The PCA results tell us this works well, so the remaining issues concern goodness of fit: is this linear description of the data accurate? Are there outliers? How well might this work for reconstructing or simulating data in the future?
Also, many thanks - this was incredibly helpful both in my understanding of the problem and the general class of problems that I was dealing with :) .
There isn't enough information in this numerical summary to conclude, with any certainty, the shape of the distribution. We need a histogram and/or QQ plot of the data to answer this. The fact that there is little skew (although a skew of 0.9 in a sample size of 536 may indicate some problems) and the mean/median/mode are about about equal is a good sign but doesn't guarantee anything. This could be a very long-tailed (but roughly symmetric) distribution for all we know.
I guess it would be very helpful to show what you've tried already, and where you're stuck. This is one of the most basic tests in statistics, so you were surely given a text book that explains the methods used.
Hint: Focus on expectations only and add the self-sfudy tag.
Thank you.  What are the drawbacks to VAR relative to training multiple individual models?
How does Poisson regression and the permutation test compare to the MWW U-Test mentioned by Xian? Can it give me better results? (a have very small sample sizes)
Great, thanks for such a detailed answer.
Thank you for your answer, Peter.
A comment by the OP, "The y values are probabilities of being of a certain class, obtained from averaging classifications done manually by people," suggests that logistic regression would be inappropriate for these data--although it might be a great solution for the raw data (as suggested in your first paragraph), depending on what the "classifications" are and how the "averaging" occurred.  When applied to the data shown in the question, `glm` produces a relatively flat uncurved line that looks remarkably like the line shown in the question.
If $F$ is strictly increasing, can you solve it? If you can do that,you can get the answer in general by modifying your original answer.
@Cardinal I agree that *some* mechanism to relate a density to probability is needed. But historically such things could be done long before Calculus was invented, so perhaps a simple intuitive demonstration is still possible. Were we to draw the graph of the density $f$ and note its mirror symmetry around $x=a$, then would it not be immediate that the region to the left of $a+t$ is *congruent* to the region to the right of $a-t$? From that, the formula for the symmetry of the CDF follows easily. (Technically, we only need invariance of Lebesgue measure under reflections in $\\mathbb{R}$.)
+1 These principles are illustrated in some posts on our site that use histograms.  For instance, they show up in the generalization to 2D histograms I discuss at http://stats.stackexchange.com/a/24590.
In some sense, they might want to model the outliers. What are the outliers: measurement noise (add zero-mean random value), equipment failure (value will be outside of reasonable data range), recording error (swapped values, swapped digits, digit added or deleted), etc?
Not enough information - is anything unique about this type of figure not covered by tools in this question http://stats.stackexchange.com/questions/14437/software-needed-to-scrape-data-from-graph?
I don't want to ignore them, but I just don't want to plot them, just note that I've adjusted for them. Am I misunderstanding something?
how did you do to found the local maximum of this small peak, and the local minimum of the pit between two peaks?
It's the very same expansion as the one I gave on [your other question](http://stats.stackexchange.com/questions/60247/questions-on-definitions-and-notation-mse-sse-sxx). You need to pay more attention.
This is too simple to be an answer, but "Learning to Rank" is the keyword usually used in machine learning, and there's a rich literature on it. Ordinal Logistic Regression is problematic because the rank isn't an absolute level.  What's usually used is a classifier on the differences of the X variables:

so that you maximize $1/(1+exp(-((P1-P2) + (D1-D2) + (K1-K2)+\\alpha))$ in a logistic regression.
Leaving aside the normal approximation issue, the use of 'at least' is because a larger sample size will lead to a smaller expected interval size, which should also be sufficient. That is, it wants the minimum sample size that will achieve the probabilistic bound. More details about what standard error you were calculating would allow a more complete answer.
(+1) Very nice example.
It depends on what you mean by 'close'. Common choices would be mean absolute error, mean (absolute) relative error, mean square error, ...   $ \\quad $ what are the values measuring?
@sebastianb You specify that $X$ is Normal, and $Y$ is Normal, and that they are correlated $\\rho$, but you do not state whether $(X,Y)$ are jointly Normal ... or what their joint distribution is. I don't see how you can calculate the desired ratio expectation, given dependence, unless the joint pdf is known and stated.
Nice. It is worth mentioning that the variance of the sample median is not as easy to estimate as the one for the sample mean.
I don't think you need to be afraid of MCMC. There is a nice [vignette for MCMCglmm](http://cran.r-project.org/web/packages/MCMCglmm/vignettes/CourseNotes.pdf).
Thanks! Very helpful explanation.
Of possible interest: http://stats.stackexchange.com/a/20440/930.
Is the null that the distributions are identical apart from the possibility of a change in spread? Or is it that the centers might also be different? Or is it that the distributions might be differently shaped (such as two different t-distributions?)
+1 I fully agree with not being convinced. The bad thing about outliers is not if they are different from your data or don't belong in. The bad thing is that you won't know what causes them and how they look. Otherwise, you'd not call them outliers but give them a name (e.g. in Raman spectroscopy, we have "cosmic ray spikes") and develop a filter for that type of event ("cosmic ray filter"). "Outlier" IMHO is an extremely ill-defined class: the class of unknown things happening to your data that should not happen.
I'm no machine learning expert so I will defer to others to post answers but I do think that [The Elements of Statistical Learning](http://www.stanford.edu/~hastie/local.ftp/Springer/OLD//ESLII_print4.pdf) is considered a good text on the subject and is written by some of the biggest names in the field. I should add that this book is written at a high level and those I've heard recommend it did have PhDs in statistics.
@whuber, if it is a straw man, it is not intentional.  It's always difficult to make any attempt to *report* on others' opinions without accidentally including a judgement on it. And I don't claim to have a broad understanding of the many nuanced positions. I'll try to rethink my final paragraph. Also, you say I left out "different interpretations of probability altogether".  I'd rather say nothing than say something incorrect.  It's not possible to say everything.  I can try to give you the truth and nothing but the truth, but I can't give you the whole truth :-)
Once you prove that stationarity implies the $m^\\text{th}$ differences are stationary, you're done. That implication follows inductively by proving that the first differences of a stationary series are stationary. That proof is merely a direct application of your definition of stationary.
This the first thing I thought. But I hoped there is more effective way, especially for this goal. Thank you.
You should keep in mind that achieving what you want might be impossible.
Arts, Science should be Arts & Science
Thanks for the answer... I am working with the streams of bitstrings (256 bits). The similarity measure for this is similar to jaccard similarity, i.e. # of 1's in (ANDed of 2 bitstrings)/ # of 1's in (ORed of 2 bitstrings). As far as i have understood, i cannot permute to the different combinations of min-wise hashing (because the elements in the strings are just 1 or 0). So , is there is any hashing for this similarity and data type???
@Tomas: I am hoping to get some help on this problem.
@Glen_b: By "the null has more than one distributions", I mean the null hypothesis $H$ consists of more than one distributions of $X$.
Yeah sorry for framing it wrong. This is what I intended
Yes it does seem fishy... Your groupings seem amazing well-balanced (286 each.) Are you sure about that? Is the order of your nesting correct? In addition to this design comment: Do you get any convergence error messages?Have you tried changing the solver used by `lmer`? (so use `bobyqa` instead of the default simplex solver) If your data are not too large, you might also want to try fitting your model with `lme`'s from `nlme`.
Your statements seem contradictory to me. On the one hand, you say "This doesn't make much sense for me, since in reality he's running usually in 2:30 hours but he has some outliers which influences the mean downwards to 2:45 hours." And at the same time you say "I would like to predict how fast they are running considering outliers". Do you want the outliers to be considered  or not? Or maybe weighted down? Please clarify.
@whuber, I edited/posed the problem am facing. Would you prefer that I put forth the distributional assumptions? I'd like to see your perspectives, as my issue is concerned with the lack of a closed-form. But, there may be a different view, on the problem of estimating the bias..that is not concerned with the lack of a closed form?
Because (talking about properties of a distribution) skewness and kurtosis tell us nothing about the mean or variance (except that they exist), I take the opposite view: the first question is uninteresting because after your clarification there is nothing left to answer!  I do not know what the second question asks because I find it impossible to determine what you mean by "analogous approaches."
Connected question: http://stats.stackexchange.com/questions/812/what-is-the-difference-between-functional-data-analysis-and-high-dimensional-data
@RichardHerron Yes, this is called a control function approach.  The residual is there to "control for" the endogeneity in the error term.  The assumption that makes it work is that the conditional expectation of the error term in the Tobit (conditional on z and first equation error) is linear in first equation error.  That's true in the setup of this problem, and it is often true in the way problems like this are set up.  So the error term once we include the residual from the first equation is a *different* error term---it has had the endogeneity "sucked out."
For the Bayesian framework I think its easier to do if you use conditional distributions rather than equations.  You've done this for the random effects, but not for the linking model.  Also you've used $\\sigma$ twice - probably better to write $g(\\alpha_i,t_{ij})$ instead or use a different parameter for the error variance.
I doubt that NLTK will eat the `scipy.sparse` datastructures produced by scikit-learn, though.
This is a great answer. Thanks Matt! One question that I still have is when you said *"$F_1$ gives them [precision - recall] equal weight"*. But you still have to choose **a point** on the prec-recall curve to fill in the numbers in the $F_1$ formula, correct? $F_1$ gives them equal weight, but I may choose a point on the curve where precision is higher than recall (or the other way around). Is my understanding right?
Just to be clear: the input values in `test` that are the same as in `train_and_test` have not been removed by your filters, yet have different values, suggesting that they have not been processed the same? I'd agree that the whole point of -r and -s appears to be that the second data set would not be used in setting filtering coefficients, but would be processed with the same coefficients as the first data set.