tttjjj/elt_bench huggingface dataset:
It depends on what you want to express. The absolute strength of linear relationship would better be expressed with your first approach (absolute correlation coefficient) than with that suggested by Roman, as with his approach a strong negative correlation (near -1) tends towards 0 after transformation, and a strong positive correlation tends towards 1. Is this what you want?
+1 For instance, $\\phi(x)=-\\log(x)$ gives the Kullback-Leibler divergence and $\\phi(x)=(1-x)^2$ gives the $\\chi^2$ "distance" based on $(p-q)^2/p$. It is noteworthy that $-p\\log(q/p) $ $=p-q+\\frac{1}{2}(p-q)^2/p+O(1-\\frac{q}{p})^3$ (with the linear term $p-q$ vanishing upon summation), thereby connecting those two.
I should point out that I have left out any scale factors (e.g. $\\sqrt{N}$ is common in these problems) but they can be added without trouble. What you choose for a scale factor will depend on the distribution of $g'(p_0)$.
These are confidence limits around the estimate of the mean (1.96 * SE). Shouldn't have used the vague term error bars. Log scale would bring the 'bars' in for this one time point but would mean the others which are all nice and clear would be blown off their raw scale and I think doing that would sacrifice too much to not gain much. Cheers.
It's in Pinheiro and Bates (2000): see http://tinyurl.com/ntygq3 for the relevant page on Google Books.
It actually seems to be the opposite of normal form... for example, the dimensions are supposed to nonnormalized so that it is trivial to navigate and create conditions.  And the table structure is also extremely simple-- one central table, bunch of dimensions, but no connections between the dimensions and nothing else.
@ttnphns There are several equivalent definitions of a rotation matrix, such as a finite composition of reflections or the matrix of a linear transformation that preserves the usual inner product.  Factor analysis terminology abuses this convention by extending the term "rotation" to matrices which manifestly are not rotations.  The interest of a reflection lies in the fact that the parity of the number of reflections needed to express a (true) rotation is a property of that rotation.  In effect, it is an element of the quotient group O(n)/SO(n).  You need a similar context for your matrices.
Homework?  If so, please add the homework tag...  consider constructing an example, e.g., with a distribution on $\\{-1,1\\}$ to help develop some intuition.
Please explain the sense in which you are modifying sample sizes.  Are you trying to compare p-values for two independent experiments of different things or are you instead perhaps contemplating the possibility of augmenting a sample of size $45$ by collecting $120-45$ additional independent observations?
I am tempted to say that this strange peak is the more related to poor binning algorithm of histogram than to the features of data. Besides that since no context is given there is a lot of possible answers, but writing them up would not be interesting.
It might be very nice also to say about what is standardized version of the ECDF, since it is mentioned.
Fortunately for you, there's a thing called [one-class SVM](http://machinelearning.wustl.edu/mlpapers/paper_files/ManevitzY01.pdf).
+1 the Marsland book is quite good and filled a big gap in the existing selection of ML books.
+1 for the AOK reference. Have you played on the Zone? As for the question, I guess you will have to look at random sets: http://en.wikipedia.org/wiki/Random_compact_set if you don't want to limit yourself to a specific shape or some other restriction.
@AlecosPapadopoulos Thank you for your help.
@MarkGreenaway Glad I could help - its a pain, and believe me, I know it well. Currently dealing with not even being able to pool HRs if they were calculated using two different methods. Don't forget to accept the answer you eventually go with - more rep for everyone!
Thank you Charlie! That really cleared things up for me!
Minor note: you don't need to sign your answer: the software does it for you.
Thank you very much for your respnde, my research question is to identify the level of Knowledge sharing among emloyees based on thier demographic data. on of this demographic data is age.
Do you really mean consecutive .... or do you mean annual? Time series data have to be ordered in time ... and most methods assume equal spacing. But the interval can be anything you want. 1996-2000-2004 would be fine, so long as everything else is spaced 4 years apart.
AFAIK n.cond does not use the first few observations to fit. It did not help me there.
Nothing wrong with ML at all. I just would like to understand the differences.
Is this homework? If so please tag as such.
+1 for interpreting the question in a constructive manner and providing a good approach.
Do I need a state for each attribute of the variable or is it could be achieved in some other way...
(1) The `AED` package [has been discontinued](http://www.highstat.com/book2.htm); instead, just  `source("http://www.highstat.com/Book2/HighstatLibV6.R")` for the `corvif` function. (2) Hope to provide a real answer, but (a) I believe VIF takes **multicollinearity** into account (e.g. you may have three predictors, none of which has strong pairwise correlations, but the linear combination of A and B is strongly correlated with C) and (b) I have strong reservations about the wisdom of dropping collinear terms; see Graham *Ecology* 2003, doi:10.1890/02-3114
This question appears to be off-topic because it is about how to use SPSS.
I guess I'm wondering *why* the James-Stein estimator isn't widely used. Is it subsumed by these other techniques, or are the conditions of the theorem not met in practice?
Thanks for your response. I did it myself what you suggested and also had the same opinion. Just wanted to double check on it. That helps...thanks..

The independent variables could be inter correlated to each other which could give different results.
+1. I didn't take people to be rigidly saying 'never', but it's always nice to have another perspective & this is a very clear & thoughtful response. Welcome to CV, it'll be great to have you as part of the community.
For a start use a [proper scoring rule](http://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules) to assess model performance, unless you can justify that 0.5 cut-off.
Since you have $\\hat y_i$ on the left, your right hand side should have $\\hat \\beta$'s. While you're editing, please remove the "e" from the end of the word 'formula'; you have only one formula so you should not use the plural.
There are several "$\\chi^2$ tests", which one are you talking about ? I have in mind the $\\chi^2$ test for $2\\times 2$ table; this test assumes two independent binomial samples and the null hypothesis is the equality of the probabilities of success. Please precise your context.
Good point, U and V can't be binary! I think your second model is what I need, but I need to work out the inference. I will post another question on the Gibbs sampling of [that paper](http://www.cs.toronto.edu/~rsalakhu/papers/bpmf.pdf), hope you could answer. Many thanks!
please display your data if you have it. if you do not have data, explain your problem eg are you looking for an estimate of sample-size.
I think that is what I may end up doing, need to look at R implementation and reverse engineer. Thanks!
@curious_cat. They are numeric. I just updated the OP
only 3 reviews on Amazon...sketchy
"correlate to any given single date" is unclear to me. Wouldn't just tabulating the frequencies give you what you want? That is a list of the dates with the most events occurring on them.
@Daniel Please don't destroy your question only to ask a new variant of it.  Instead, you are able to edit your original question (this one) to reflect any improvements or changes you would like to make.
Ah. It was meant to be a general conceptual question, but I can give an example. Let A = {1, 3, 4, 2} and B = {50, 60, 65, 55}. Then, let A_g = {200%, 33%, -50%} and let B_g = {20%, 8%, -15%}, where A_g and B_g are the sequential growth rates for A and B. So, correlation of A and B is 1, but correlation of A_g and B_g is less than 1. I could make another example where the opposite is true. It seems that it should be a simple thing to understand, but for some reason I'm not getting it conceptually. Thanks so much for your help!
None of the three plots show correlation (at least not linear correlation, which is the relevant meaning of 'correlation' in the sense in which it is being used in "*the residuals and the fitted values are uncorrelated*").
Conversely there is also 'linear association'.
Says a statistican to his friend: "counting on me is pure frequentism".
Maybe some kind of lattice plot? Or stacked lines?
@DimitriyV.Masterov (which is essentially the same answer as Scortchi gives to this issue).
It appears this technique effectively ignores the information about preferences encoded by the 24 categories.  Obviously it's not going to work with just 13 observations.  But even in general it doesn't seem to address the question.
@Bitwise: why? For once that someone gives a `dput` he should rather be congratulated!
Average time a user spends on the website per unit: I am lost on what you seek here. My answer is: use the data; evidently you want something else.
Model selection and feature selection are very different things. Competing models may not necessarily be nested with the former, which is the case with AIC stepwise model selection or Bayesian Model Averaging, or so on. Feature selection from boosting, bagging, Lasso, has a much more regular space of nested models determined by inclusion or exclusion of specific variables.
Thanks. I will create another question.
Construct your matrix so that rows are observations and columns are predictor variables, and input this instead of your y ~ x model. You may want to check `help(randomforest)` to make sure you wouldn't have to transpose this (columns as observations and rows are variables), as I don't remember the format that `randomforest` likes to see.
Can you give more information about the magnitudes of $n$ and $k$ and the distributions of the $p_i$ that are of interest to you?
@caracal So can I look at it this way:  Under the null hypothesis the probability of success is 0 hence if you ever see a success you can reject the null hypothesis.  That is why you say the threshold is 1 because if the binomial sum ever gets to 1 you can reject with a type 2 error of 0! Now under the alternative the probability of the first success on the nth trial is (1-p)$^n$$^-$$^1$ p. This probability goes to 0 as n goes to infinity.  So a sequential rule going by stop when S$_n$=1 would have power 1 for any p>0.
And by guestimate I mean fit the model without the correlation term, fit the spherical model to the residuals, get those parameters and then refit with `gls()` but state the parameter values rather than use the formula using argument `value`. Without the nugget you'd only need to specify the range as a parameter but with the nugget you will need both the range and `1 - cor(i,j)` when `i` and `j` are two observations close together in space. See `?corSpher` for details. You don't even need to fit the model without the correlation structure if you can have reasonable guesses at those parameters.
In response to your clarification, let me correct this common misconception. OLS does not require your right-hand-side, explanatory variables to be normally distributed. Transformations of RHS variables may be helpful for theoretical reasons or analytical convenience, but least squares will work just fine without them.
(+1) for the code and the graphs together
Thanks Mike; that sort of paper was exactly what I was looking for!
Agreed that the chart looks busy. It would be useful however if the questions are grouped in some sort of an order e.g. Q1 - 10 asks about a certain dimension and so on. At a glance if the trends are obvious, the colours would tell.
Does this do it? xtmixed score a##b##c || s: R.a || s: R.b || s: R.c (I'm really just guessing, based on a very quick skim of the xtmixed manual [Examples 10 and 11 seem relevant])
Did you take a look at this related question: [Regression coefficients that flip sign after including other predictors](http://stats.stackexchange.com/q/1580/930)?
Can you spell out SOLS? What does it stand for?
1. Well, it's very rare that you want an interaction term without the main effects. 2. In general, high $R^2$ is good. 3. How to create a dummy variable depends on your software.
Welcome to the site, @DavidEpstein. You can make images w/ your preferred software & upload them into your answer by clicking the little picture icon (to launch the wizard) above the answer field. Unfortunately however, you need >=10 rep to do so. I'm sure you'll have that soon enough; in the interim, if you can post the image anywhere else on the internet & post a link to it in your answer, a higher rep user can fetch it & post it for you.
(+1) A very interesting question ...
Can you provide some more detail on what software/tools you used and any associated code?

At a guess, have you perhaps neglected to fit a seasonal term to the model? It really depends on what sort of model you are fitting as to the shape the forecast will take.
The QQ plot shows that @Maarten 's suspicions are confirmed. There is a long right tail.
@corone makes that a strict >
Thank you, Andy.
Even when $\\mu/\\sigma$ is huge, so that (arguably) we could neglect the chance of a negative value, the expectations are wrong: $X=\\exp(\\log(X))$ would have a mean of $\\exp(\\log(\\mu) + \\sigma^2/(2\\mu^2)),$ which is strictly greater than $\\mu$.  It *is* true that lognormal distributions with very small geometric standard deviations are approximately Normal, provided the parameters are appropriately chosen.
Welcome to the site, Jolle.
Do you have one measure per person? How does teacher/school come in? Can you describe the structure of your data?
Additionally whenever I run the code, The RMSE produced is different inspite of the same data set being used.
@MichaelChernick, I meant to say that the largest peak on the CCF is approximated by a gaussian. I'll edit that on my original post (sorry, just used to the astronomical jargon...).
$\\frac{\\alpha}{\\alpha + \\beta} = m = \\frac{1}{n}\\sum_{i=1}^n n_{acc}^i / n_{tot}^i$, $\\frac{\\alpha}{(\\alpha + \\beta)^2(\\alpha + \\beta + 1)} = v = \\frac{1}{n}\\sum_{i=1}^n (n_{acc}^i / n_{tot}^i - m)^2$. $v$ and $m$ can be calculated from your data. Then solve for $\\alpha$ and $\\beta$
The assumption is that observations within each group are normally distributed with different means, but the same variance, $\\sigma^2$. Therefore, if you subtract off the sample means from each group, you will have one set of $N(0,\\sigma^2)$ data, on which you can do a usual check for normality (e.g. qqplot), instead of 8 separate ones.
In your example, the condition is false. Whether to go left or right should be mentioned in any good software. There is no convention.
Even when you are interested in a "total system", incorporating covariates could be very useful. Say, for example, you are drawing up a contract for some work and you already know which team it will be assigned to. Incorporating this information into your model can potentially protect your financial interest when drawing up the SLA. In fact, you seem to be providing such an example yourself, though maybe I've misunderstood the desired use of the model.
(+6) Nice answer! And it's always nice to have references and reproducible code.
Thanks for your answer! After my calculation, if $X \\sim Laplace(\\mu_1, b_1)$ and $Y \\sim Laplace(\\mu_2, b_2)$, the PDF of $Z=X-Y$ will be
$$
\\begin{align}
f(z) &= \\frac{b_1}{2(b_1^2-b_2^2)} e^{-\\frac{|z-(\\mu_1-\\mu_2)|}{b_1} }-\\frac{b_2}{2(b_1^2-b_2^2)} e^{-\\frac{|z-(\\mu_1-\\mu_2)|}{b_2}}\\\\
&= \\frac{b_1^2}{(b_1^2-b_2^2)} \\frac{1}{2b_1}e^{-\\frac{|z-(\\mu_1-\\mu_2)|}{b_1}}- \\frac{b_2^2}{(b_1^2-b_2^2)} \\frac{1}{2b_2}e^{- \\frac{|z-(\\mu_1-\\mu_2)|}{b_2}}
\\end{align}
$$
Looks like a difference of two differently weighted Laplace distribution.
@MaxGordon What do you mean by _regular_ competing risk? Anyway, this _is_ a competing risk analysis.
I think you want it at the individual level. Alpha is a measure of test reliability, and the test is given to individuals.
Define a lot. Typical studies where one would have to recode the values themeselves would be the data you collected yourself or shared be colleague that would have a thousand or so cases, so you would have 8 or so cases who would endorse "Middle school". The data sets coming from large scale national surveys would very likely have larger sample sizes, and much better meta data and formatting routines already in place.
Thank you for the reference and the formula. I have a sample size of 300, using imputed data and by default MPLUS is using the ML estimator. The standardized loadings are .3 to .7 approx and the inter-item correlations are .1 to .3.
@mmacx Seems like he deleted it. I'm adding a comment to my answer.
I think this all depends on the goal of your analysis? By panel model I am assuming you mean a longitudinal auto regressive path model? Can you please add more information about the intent of your analysis?
I changed my answer to respond to your question.
This is very useful! Thanks Dikran.
Why do you think that the researchers should have used logistic regression? The response variable is clearly continous (although maybe with some censoring). Please elaborate, if we are to help you in providing a good answer to this question.
This shouldn't happen..... Training data and test data should be random selections from the same data set.
Out of curiosity, are people always weighed at the same location?
(+1) Great answer, very good points. I agree that we should call what Sweave and friends do *reproducible analysis*.
A little less abstractly, a change in X may change in response of more than one covariates. But only one of those covariates also leads to a change in Y.
(Continued) But, as a user, the choice of CV is a better site, for the simple reason that diligent statistical analysis is the responsibility of the user, not the developer.  If the user sees an erroneous $R^2$ relative to the magnitude of the RSS, then they should do their own post-processing before reporting further.  Programming-wise, I'd like to know how to avoid these numerical issues as much as possible, but I think that they can't be escaped, and that's where it's important to have  a diligent user and to educate others.
Came up with an alternative model, same expression but the levels of "Generation" are integers (1-6). (Null Deviance 757.95, 790 Df; resid deviance 528.94, 777df). This might actually be more valid, since cultural borders between generations are actually rather fuzzy. We think of them as discrete entities, but the difference between someone born 1960 vs. 1961 usually aren't all that large. Or have I decided to bark up not only the wrong tree but an entirely wrong forest?
@Pankaj More: ok but they all have 100 columns. Are these columns related? Can we think of each row as an observation and each column as variables/measurments?
"Given that GENMOD is generalized linear model code, and GLMs are (almost) always fit using ML, and given that REG is least squares regression, this conclusion seems well supported." - But even R's glm() with gaussian link (which should use ML estimation for the same reason) output matches that of R's lm() and that of PROC REG (exactly)! Any idea? Thanks.
@ cmoibenlepro the link is http://lowrank.net/nikos/fest/
While I have never conducted a similar study myself, this seems like the Netflix problem, which uses collaborative filtering (I believe).  Check out the package this package.  http://cran.r-project.org/web/packages/recommenderlab/vignettes/recommenderlab.pdf
But as I can see they are not overloading Microsoft's original, so they can break the [0,1) limit.
There are many, many posts on questions such as you are asking. In this case, search for posts on outliers.
It is, I suppose, worth mentioning that some methods that can predict propensities of exactly 0 or 1, ie deterministic methods, are going to lose the theoretical guarantees that p-scores have. I don't suppose anyone was really thinking of them though.
Fair enough, could you summarize the different methods you are wondering about, so that people don't have to read the pdf?
It is seldom an optimum way to analyze ratios.  If relative changes are relevant and you've verified the Bland-Altman conditions, then analysis of log ratios is more appropriate.  But even more frequently, analysis of covariance of the final measurement, adjusted for the initial measurement as a covariate, works better.  One example of this is when the baseline is measured with much error or is poorly correlated with the follow-up measurement; in that case you lose power by analyzing change instead of using analysis of covariance.
Regression, prediction, modelling: take your pick.
@dubhousing: == is not doing what you think it's doing. Try doing `SMOKE==1` and see what the result is. As Peter Ellis says, the == is doing a reasonable thing -- not what you're thinking it does, though -- and the issue arises in `sum`. That's a helpful tip: if a  statement isn't working as you expect, try the individual pieces to see where perhaps it's working differently.
+1 for highlighting the importance of model interpretation. I won't add anything about the uninformed ML approach (or ensemble methods) with more complex cross-validation schemes, because I feel you already said what's really matter here: (1) feature selection through resampling is hardly interpretable in isolation (i.e., by comparing one result after the other), and (2) it all depends if we are seeking for a predictive or an explanatory model.
@Jerry You might want to focus on a given $\\eta$ if you are interested in studying various residuals.  If you don't do anything differently, I am not sure it makes a big difference either way (see edited response above).
Can you tell us what you're studying and the exact model, and relevant results, you fitted each time? I can't follow what you did, or why you might do it. And I have no clue what "mushed cells" are either.
this site supports latex, please reformat your question. I am reluctant to do it myself, since the notation is not very clear.
I'm not sure I would start off with a PhD level course then, as it may assume too much. You might try: https://class.stanford.edu/courses/HumanitiesScience/StatLearning/Winter2014/about starting in a week, which is at an introductory level and which has a free PDF of the intro-level book by the course leaders (and two other coauthors) which will be used as the course text.
OK, I see the link now.  This is a fairly simple approach, and like Wikipedia mentions, there are advances upon this basic idea.  On reflection though, you're going to have to deal with some kind of trade-offs (convergence in this case).  I wonder if you're asking the right question here.   Are there really no good bindings to linalg libraries for D?
@Aksakal: Licensing costs & vendor lock-in aren't trifling concerns. You don't buy SAS software in a shrink-wrapped box off the shelf in Dixon's - you need to cut a deal with SAS. So I shouldn't be concerned in the least about working for a company that I'd heard decided against a SAS licence on cost grounds. (Nor of course, for one that had decided it was worth it to them.)
They are always equivalent whenever the model contains an intercept. See my answer.
The paper I referred to is now available at http://arxiv.org/abs/1311.0081
Alright, I see where we're going now. One last question: how would you recommend I examine this interaction? I have daily generation values over a year, which I have been using to determine the difference of generation between angles with module type held constant with a paired t-test. Would I use each of these daily values for the interaction, or just use the means over a year?
I am guessing you mean this:  http://www.stat.columbia.edu/~gelman/arm/  I just read a good review about it here: http://www.r-bloggers.com/bookshelf-remodelling/