tttjjj/elt_bench huggingface dataset:
Thanks for the reply! I centered on the mean value of X to get that interpretation of the intercept because. I'm interested in the intercept since I do not have a treatment variable and want to examine whether the group has changed when adjusting for baseline response. With the $$y−x=α+β(x-mean(x))$$ setup I get the same intercept. I was however deterred from using that setup by reading this thread: [link](http://stats.stackexchange.com/questions/15713/is-it-valid-to-include-a-baseline-measure-as-control-variable-when-testing-the-e)
Thanks. Could you be more precise?  I do not see any relationship between the two...
@cardinal good catch, the autocorrelation should be $\\rho = 2p - 1,$ not $1 - 2p$. correcting it...
This may be a question more appropriately asked on the stackoverflow forum, also you may be able to get a more concise answer if you describe what type of file the url is pointing to.
Anywhere else I could look for an example besides buying a $70 book ?
Could you explain why you divide the expectation by $p^2$ and how you then obtain the right hand side of the equation?
Do you expect some dropouts during follow-up? Are there any other covariates to be included in your model?
@Claire, this is inconsistent w/ the description in your question. There you say that X2 & X3 are dummies, and X1 & X4 are continuous. Which is it?
A prior depending (even if this is only through the support) on the data sounds wrong: **you cannot know the maximum of the sample before the sample has been generated**. Moreover, you claim that $\\theta = cX^*$ is an almost sure equality, with both $\\theta$ and $X^*$ random (thus there is correlation $1$). But this implies that the posterior distribution of $\\theta$ (which is the conditional distribution of $\\theta$ given the sample) is the Dirac mass at $cx^*$. And this contradicts your derivation of the posterior distribution. ... (no characters left...)
I am having a philosophical difficulty understanding what "actual" probabilities might mean, given that in any correct application of logistic regression all the response values are either $0$ or $1$.  Are these *frequencies*?  Of what, precisely?  (I am reminded of goodness of fit techniques in which a moving average of the response can be compared to the predicted probabilities.)
@bayer See the reply by @Charlie.  In the present example, *both* models are linear (whether or not R views them as such) because in both of them `a` is a linear function of the four coefficients.
I think this [page from UCLA ATS](http://www.ats.ucla.edu/stat/R/library/contrast_coding.htm) explains it quite well.
@Sergio, read the last two posts here: https://groups.google.com/forum/#!topic/stan-users/oSGUrVFCIVw
Log ratios are generally much better behaved than ratios, always assuming that you can take logs, i.e. all components are strictly positive. This is broadly true regardless of whether regression is intended.
I'd like to add that it actually makes more sense to identify the median when calculating two-sided p-values for asymmetric distributions than to identify the mode, as I stated on my original post. This is because of the very definition of a two-sided p-value: you actually search the probability of your data or more extreme values given the null hypothesis on both extremes, which you can define arbitrarly from the median. However, this isn't by any means the panacea for the problem, as it's still arbitrarily chosen (e.g., it wouldn't make sense for bimodal distributions).
Could you please explain why Gamma is not translation invariant.
Copied SO version; I'll ask mmyers to remove it from SO.
(Run out of chars in previous cmmt) The context is astronomy. Each _object_ is a star and the _parameters_ are properties of each star. Does this help? Please do tell me if I can/should add anything else to the question. Cheers.
They way you write it, $S(\\theta)$ is not the maximum likelihood  estimate, but the first derivative of the log-likelihood w.r.t. the unknown parameter. And what do we do with the first derivative in order to find a maximum? ... you should end up with a quadratic polynomial in $\\theta$.
Your first 2 plots are not going to show anything meaningful about the respective relationships unless you limit each X-axis to a much smaller range.
@lcl23 Listwise deletion is usually for subjects with missing data on any one of the covariates. Could you indicate how you define an outlier in your particular case?
Could you define alpha coefficient - there are many quantities that could be called "alpha"
I once had someone try to tell me that at least 60% of his data where outliers;... I said 'I think your model for the data is wrong'.
What about an example ? I think the OP might appreciate that (given he/she seems to have only basic knowledge of statistics). An example metric, some artificial data and an exemplary execution of a test ... this would make the answer really complete.
@whuber: You are right. This is not what I meant. I did an update.
I Have wrote a paper and i have only used (the combinition between PCA and wavelet analysis), the reviewer recommend to use some others approches and compare them with my method, i have sequence of profiles (taken for each unit of time) and each profile contains 1000 variables , also my profiles are nonlinear
Thanks and great package!
Of course, using a threshold of $p < 0.05$ is ridiculous in this case, but unfortunately many frequentists don't think about other thresholds.
The data of $Y$ were logarithmized.
Hi Diego, I went through the paper but I didn't understand the calculation of cross correlation distance metric ? Could you please   explain it. Thanks.
How are you defining SNR?
I didn't see both definitions in the references?
Be aware that "My interpretation is that I am looking at a graph y~x, where the slope changes due to the covariate, and the line shifts up or down based on the different levels of A and B (changes in intercepts)." is a little of: the way your fixed effects are set up (A*B*X), you're estimating a different intercept AND SLOPE IN X for each combination of the levels of A and B.
Hence the 83.4% CI as mentioned in the second reference (that's assuming independence and equal variance, of course).
Did you notice that p. 14 of that latest reference explicitly answers your question?
Please, add the self-study tag.
I am sorry, and I have added a general example for my original question.
This problem can be analyzed fully as a Markov chain on the Cayley graph of the permutation group.  Numerical calculations for k = 1 through 7 (a 5040 by 5040 matrix!) confirm that the largest eigenvalues in size (after 1 and -1) are exactly $(k-3)/(k-1) = 1 - 2/(k-1)$.  This implies that once you have coped with the problem of alternating the sign of the permutation (corresponding to the eigenvalue of -1), the errors in *all* probabilities decay at the rate $(1 - 2/(k-1))^n$ or faster.  I suspect this continues to hold for all larger $k$.
Can't thank you enough for that well explained answer !
For a classical design (balanced, nested, etc.) I think I can indeed argue that there's a real p-vaue, i.e. a probability of getting an estimate of beta of an observed magnitude or greater if the null hypothesis (beta=0) were false ... lme4 doesn't provide these denominator df, I believe, because it's harder to detect in general from an lme4 model structure when the model specified is one where some heuristic for computing a classical denominator df would work ...
Very good (+1). I immediately jumped on one interpretation, but you are absolutely right that the alternative definition is standard.
remember that a random forest is taking a decision 1 variable ( ie pixel ) at a time. So it is not very good for image processing -raw. You are better off first using some sort of preprocessing ( eg PCA, etc ) to develop more meaningful decision variables
$D$ is defined as the supremum of absolute differences in the two ECDFs; $Z$ is defined as $\\sqrt{n_1 n_2 / (n_1+n_2)} D $. I was just explaining how to compute $D$ from SPSS output. Hope that clarifies.
I think you are right, I may be asking the wrong question
Each plant has a different altitude (and different temperature). Many plants live in the same environment.
See http://en.wikipedia.org/wiki/Median_absolute_deviation, then read http://web.ipac.caltech.edu/staff/fmasci/home/statistics_refs/BetterThanMAD.pdf (which is also discussed at http://en.wikipedia.org/wiki/Robust_measures_of_scale).
What are "items" in your case? (I'm asking whether it has anything to do with psychometrics because if this is the case, I would recommend having a look at [item clustering](http://www.personality-project.org/r/html/ICLUST.html), or Revelle, W. [Hierarchical cluster analysis and tihe
internal structure of tests](http://personality-project.org/revelle/publications/iclust.pdf), MBR (1979) 14:57.)
@Nick Sabbe : can you explain in a bit more detail what this line means: $USG\\sim USG_s/r_s$
$USG_s=count number of google+ users with the given sample of 200 names. r_s$=dividing the number of US citizens with given surname by total number of US citizens. Does sim mean that you do this for every single name? and then you divide USG_s by r_s for every single name?
It is the first time I see the interpretation you mention in the second question. Could you give a reference?
The wiki article might help out:  http://en.wikipedia.org/wiki/Continuity_correction
It's not clear why you need to *approximate* the lognormal pdf and cdf in order to answer the first question (whether a sample distribution is consistent with the lognormal). Nor the second question, nor indeed the third question. For example, the first question might be pretty easily answered by performing normal goodness of fit test (some of which are fairly fast to compute) on the logs of the data.
What do you mean by the "quantitative overlap" of 2 species?
They won't be "standardized" though, will they?  I read the question as wanting to know the size of fixed effects if the explanatory variables were all on the same scale.
A binomial variable takes on values in ${0,1,2,\\dots,M}$, and as such can't be an "expected proportion of successes".  Do you perhaps mean that $Y/M$ is the expected proportion of successes?  If so, this is an interesting model; could you explain a little how it comes about?
Sure you can paste your output: copy it from the `R` window and paste it into your question-editing dialog as usual.  Format it using the tools found across the top of the window.
Can you please elaborate? What do you mean by ranking? Judging which model is better than other? If so, on what criteria? What do you mean by "based on historical data"? What kind of models do you want to "rank"?
Am I doing it right as glmm() fundtion expects data from two group of variables
A less popular method (but with increasing popularity) are [Scoring Rules](http://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf) which evaluate the predictive performance of the models.
To answer who saves objects: I do.  :)
Very cool and useful answer! I'll have to look into those packages. (+1)
How are you calculating the MSE in your code?
A related question with a nice answer: [How did Efron imagine the bootstrap?](http://stats.stackexchange.com/q/92344/32036)
Robust, as in reliable at producing correct results.
Yes, and don't forget the likelihood ratio $\\chi^2$ for each model, and its translation to AIC.  You might also consider the "adequacy index" which is related to the fraction of $\\chi^2$ each model generates when divided by the $\\chi^2$ of the supermodel containing all variables.
Soil sample data present some characteristic difficulties. First, it's almost always best to represent them as log concentrations, which will tend to have more symmetric distributions than the concentrations themselves. Second, unless all indicators are naturally occurring in detectable proportions, the dataset may exhibit substantial *left censoring* from the nondetect results. Your options depend on the extent to which that censoring occurs. A thread devoted to this issue appears at http://stats.stackexchange.com/questions/1781.
$s^2$ is unbiased for $\\sigma^2,$ so, yes, $s$ is biased for $\\sigma.$
(+1) This is a nice clear answer to the problem, as long as the normality assumption is reasonable.
can you show a picture?  do you know _anything_ about the underlying curve?
You cannot estimate MA part coefficients with OLS, and if there is no MA part, OLS estimate is MLE. So please clarify the question.
Hi, thank you so much. Your response confirms what i suspected but I wanted to get more opinions.
It seems unlikely that ID is relevant, but it depends on what, exactly it represents and whether that's something you want to model.
If someone asked you to justify your predictions you could do so if you used linear regression by explaining the relationships between the inputs and outputs your model assumes. A model attempts to explain the relationship b/w inputs and outputs. KNN doesn't attempt to explain the relationship b/w inputs and outputs, hence there's no model.
Could you elaborate on how you think a lower mean could "drive overall variance"?  What kinds of measurements are these?  Are they perhaps bounded below by 1 (and above by some other integer such as 7)?  Counts?  Responses on a Likert scale?  Digital instrument readings?  These three circumstances would indicate three different answers to the question!
Exactly, it gives you an idea of how the data is structured and, by that, it can show you if a given model would be relevant.
Im not sure what the r program is doing but we have

$$var(\\sum_iE[X_i])=a^Tvar(E[X])a$$

Where $a$ is a column vector of ones and $var(E[X])$ is the covariance matrix for $E[X]=(E[X_1],\\dots,E[X_n])^T$.  Does this help?
A CDF is increasing; it's graph can never look like that for any distribution. Do you mean that the graph should match that of a PDF?
See also: http://stats.stackexchange.com/q/22502/7828 - how to draw valid conclusions from big data.
More importantly, he considered `weight/height^3` which would be interpreted as a density (intuitively makes sense), but opted for the classic BMI because of its normal distribution as you said.
It seems to me that the quotation with which you begin your previous question nicely answers both (1) and (2). The distinction it explicitly draws is between a "family of inferences" specified in advance and a family that is not, or cannot, be so specified. The Wikipedia quotation is in the context of a clearly defined family: namely, all possible contrasts.
@whuber You're exactly right. I'm trying to compare my students' scores to their target scores. I don't aim to compare schools against schools or years against years.
If you pre-existing topic are just words, you can refer Tag LDA or Labeled LDA, just google it. As far as I known, in LDA you can't do that, even you pre-set the topics(distribution over words), it will converges with new topics set. But if you have topics already, why do you need LDA? or you can just fit document to topics?
+1 @JeromyAnglim I agree with the references.
I'll consider that, thank you
@whuber, my bad, corrected
(+1) An additional search term of interest is the *Rasch model* which is essentially your suggestion in item 2 but without the interaction. There are many extensions to the basic model, as I'm very sure you are aware. :)
50 out of 450 seems reasonable for 'regular` survival models. Some of your estimates may have large confidence intervals, but I don't see a way around that (although others may).
@Mascarpone Please read the text preceding the equation.  I do not think there is a mistake in the change of variable :-), but if you think it would clarify the exposition, I would be happy to point out that when $p=F(x)$, then $x=F^{-1}(p)$.  I just didn't think that was necessary.
Are you able to show the data for the example that didn't seem to work for you?
Do you mean that in the 1st data set there are only positive labels and no negatives at all?
I think your notation may be a bit off, or your probabilities are inconsistent.  For instance, there is only one link for Asia (A), and that leads to tuberculosis (T).  So this means that $P(A)\\geq P(T)$.  But you have assigned $P(A)=0.01<0.0104 = P(T)$.  So this is inconsistent.  I think this is actually $P(T|A)=0.0104$.  But then the rest is unclear, are these conditional probabilities?  Is $P(D)$ conditional on $B$ or $E$? or just $B$? or just $E$?
The smoothing parameter is not statistically estimable but is using chosen to maximize out-of-sample fit using, for example, cross validation. I think the standard packages for LASSO and Ridge Regression in R have built in functionality to do this for you - have you looked into that?
I would add a pointer to [this](http://www.jstor.org/stable/2289995) paper too.
Yes. Thanks a lot Bill. I didn't have any idea how I could paste it in. Is uppose I could have scanned it in to a file and then tried pasting it. That would have been a lot of trouble. Is that what you did? There are a few more scenes in that one that are also pretty funny.  But this gets the idea across.
Hi there, I'm not clear on what the application is doing. You say it is getting a list of strings, but then you later talk about sorting. I'm not seeing the link between your first paragraph - data collection - and your second paragraph - data sorting. Can you add more detail?
Thanks! I'll definitely look into Meinshausen's article.
A sufficient statistic is not necessarily an estimate of the parameter(s); e.g., the original sample doesn't estimate anything.  You have to do something to it to get an estimate.  The only requirement is that a sufficient statistic doesn't discard any information you could get about the parameter(s) that was in the original sample.  The factorization theorem shows sufficiency because it expresses the joint PDF conditioned on the parameter in such a way that the part that remains conditional on the parameter is only a function of the sufficient statistic.
Let me share with you some real data: I am  trying to set an equation which could predict well in advance whether the client will close the account with us depending upon certain parameters. I have assigned value as 1 for 1 account closed with corresponding independent variables (5 qualitative and 1 quantitative). For eg; 1st row has 1 as value for the 1 closed account user likewise I have 140 such users everyone has been assigned the value of 1. Please state which model should I use?
Are there any numerical heuristics about when you would want to apply something like King's procedure? Does the Manski book have something like this. For example, should I worry about the logit if my population has 40 million observations, with a 1% rate of $Y=1$ and I take a 10% sample?
Dear Patrick, I thank for your very prompt answer and helpful references! Best,
You are in some sense lucky that the cubic behaviour has already started at the beginning of your data set. If the actual execution time of the function would be approximately $10^{-3}n^3+1000n+10^6$, perfectly $O(n^3)$ behaviour that even shows up with the given inputs, then the R code above would not like the hypothesis much - for $n<1000$, the constant and linear terms dominate the $n^3$ term. Fitting to $an^b$ returns $29n^{1.7}$ with large residual square error. Not sure what would be an easy approach that would work for such data...
Boxplots should not have whiskers within their interiors, period.  Although that first figure may have some logical explanation, it's either an error or it's not a boxplot.
If you have different sites and more measurements for each site, I would use a multilevel model. http://en.wikipedia.org/wiki/Generalized_linear_mixed_model
Yes to what @AntoineVernet says, or if you do need to make the formula automatically, `xn <- cnames[1:(length(cnames)-1)]` (and similarly for dep) should be fine, if you add the `data=tmpData` parameter.  (Or just use ``[[` instead of `[`...)
It is a little hard to follow what you're trying to do. What is your dependent variable and which variables do you want to have as independent. Do you want to do $\\Delta BPD = \\beta_0+\\beta_1*work+\\beta_2*...\\beta_k$ or is it $Fancy\\; outcome = \\beta_0+\\beta_1*work+\\beta_2*\\Delta BPI+\\beta_3*...\\beta_k$?
Can you give a counterexample of what you mean? When would you ever expect single observations (i.e. the actual data) to be of interest? If you cannot answer this question, there is probably no need for the sentence in your research.
@Adalberto That misses the main point, which is that *any* definite, non-randomized procedure of assigning subjects to groups cannot be assured of having the desirable properties that a randomized procedure has. Suppose you spend years of time on such a study only afterwards to have a reviewer point out an unexpected but strong confounder between treatment and parity of birth day? Because we cannot anticipate all such confounding, we sidestep the problem by means of random assignment.
@CarlWitthoft Thanks for your comments. In the link, there is a formula for the least square regression method but not weighted least square regression method.
Please see the guidelines for [self-study questions](http://stats.stackexchange.com/tags/self-study/info). What is your attempt? Where do your difficulties lie? You should check all the details in your question, at least one of the pieces of information here is wrong.
I can't find the original paper, but some of the papers in the book "Neural Networks - Tricks of the Trade" suggest that tanh is better in the hidden layers as networks perform better if the hidden layer activations are centered (i.e. zero mean).
Edited my answer with your new information
Excellent response! This reminds me a few excellent responses user chl has provided on [interpreting interaction effects](http://stats.stackexchange.com/search?q=user:930+[interaction]). He gives article references in this response,[What are best practices in identifying interaction effects?](http://stats.stackexchange.com/q/4901/1036). And gives a wonderful example of graphically displaying interaction using coplots in this response, [Is interaction possible between two continuous variables?](http://stats.stackexchange.com/a/4965/1036).
Aren't there some cluster algorithms which do not rely on triangle inequality? Besides, there is a recent [master's thesis](http://uu.diva-portal.org/smash/record.jsf?pid=diva2:563483) which presents a modified k-means algorithm without triangle inequality. Maybe, this will be more interesting in the future if someone writes/updates a R-package with this modified k-means.
Thank you; it's better. But it is also confusing: this model underlying `lm` does *not* treat the data as bivariate normal, so you seem to be asking to do to different things at once. Do you want to predict income from education or do you want to model the bivariate distribution of the (income, distribution) pairs?
Unless $b$ and $c$ are both zero, June, $y$ could be *anything.* You have to make some restrictive assumptions about both $Z$ and $W$ in order to have an answerable question.
Answering your first question (finding the probability that the first sequence of observations are from $A$) requires a *prior probability* distribution for $(A,B)$. That might be the missing piece that causes your math to "break down." It also helps answer your final question.
You implicitly have an intercept term.
But is that possible? To transform a scalar in a way that gives you a matrix? The whole expression is derivative of scalar w.r.t. some other scalar, so I'm not sure how would I interpret that the result is a matrix. 
Thnx a lot for your help!
Here's a blog post on the topic: http://conjugateprior.org/2013/01/formulae-in-r-anova/
@Stephan This is a bit of a terminological nicety, but it can be important to know that confidence intervals are for *parameters* rather than their estimates. For instance, with a lognormal model two estimators of the mean are commonly used: the arithmetic average and the Land estimator.  The associated CIs in both cases would be termed CIs for the *mean*, since the purpose of both is to cover the *true* mean. It would be unusual to say that one is a CI for the arithmetic mean and the other is a CI for the Land mean.