tttjjj/elt_bench huggingface dataset:
@Amoeba and Uri: We need to be careful about is meant by "exponential decay." The PCA results will order the eigenvalues by decreasing magnitude, so of course the resulting scree plot will look initially like *some* kind of decay. It would be of interest to formally test the exponential model of decay for goodness of fit. I'm pretty sure that in most applications of PCA the goodness of fit will be poor overall, bringing us to a crucial issue: if you are allowed to "eyeball" the plot and perform the fit only for the first "few" eigenvalues, exponential decay might be a self-fulfilling prophecy.
@user4341 The SD of y is zero.
So what you are saying is:  the test statistic needs to have an assumed distribution, $H_0$ is the way to parameterize that distribution, and thus $H_0$ must use an $=$ to set that parameter. Correct?
+1 The `arima` option looks more different from Stata's `prais` at first glance, but it's more flexible and you can also use `tsdiag` to get a nice visual of how well your AR(1) assumption actually fits.
@IrishStat, can you explain to me , what u mean by a model ? 
Up, you'll find the data series that u asked, 
regards
In summary, the problem of selecting a method often is caused by not knowing what you actually want to achieve. Any comment or clarification on all this ?
It would also be sensible to make the value you do not give a dummy variable one of the common and specific answers, e.g. *single* and *female* rather than *unknown* and *unknown*.  Otherwise you may be tempted to report how much impact knowing your marital status or gender has.
[A Baumann](http://stats.stackexchange.com/a/40929) suggests a KS-test: read about it on Wikipedia and find an open-source implementation in `R`. [Ben Allison](http://stats.stackexchange.com/a/40935) recommends a chi-squared test and helpfully describes how to conduct it by dividing the range of data "into equally sized non-overlapping patches." Many sources, including [Wikipedia](http://en.wikipedia.org/wiki/Goodness_of_fit#Pearson.27s_chi-squared_test), describe this test and provide formulas. Open source software such as `R` will do the calculations automatically.
It looks like you just need to keep one of predictors (no matter which) unless you care for 0.5% gain in R-squared.
For grids, using MatLab: http://www.mathworks.com/help/toolbox/images/f21-17064.html.  For irregular point data, using R: http://stat.ethz.ch/R-manual/R-devel/library/spatial/html/correlogram.html.  (To make the example work, include the line `library(spatial)`.)
Ah ok, now I got it.
Thanks, @chl, you're right, the new title is better.
Whether the raters are acceptably similar is a domain-specific matter: what might be considered acceptable in one field might not be in another. There's no objective standard of similarity here that can be used to make a judgment against.
(+1) As usual, a nice, concise, informative answer.
If magnitude matters, than what you want is probably more like optimal stopping. http://en.wikipedia.org/wiki/Optimal_stopping. House selling is explicitly mentioned as an example application, and house buying amounts to the same problem (since advertising costs are assumed in the selling problem).
@whuber, by complete separation I mean that the largest value in the one group is less than the smallest in the other group.  This is the only way to get a p-value=0.05 with $6\\choose3$ possible permutations.  This was meant as a starting point where a specific distribution would not be needed (but still assumes exchangeability and the null is that both groups come from the same population/distribution).
I would agree with whuber though that the visualization of the variogram cloud would be really helpful in determining if you choice of model seems appropriate.
Alright, I think I have the full story up there now. When I first asked I thought it was a sort of silly question and someone would chime in right away and tell me it was super obvious that you do it a certain, but I have come to accept that is not the case. What factors go into deciding if you want to keep period, quarter and year if you already have a time variable that just increases by 1 for each period? Is it seasonality and controlling for differences between each interval?
Thank you (+1): could you please edit your question to remove the ambiguous language?  (The reference of "this distribution" could either be the Gamma or its conjugate prior.)
+1 now on both, thanks. (Among other things, this way all the search function features work as they should.)
@ Daniel: Your original question was "Estimating population average models in lmer". You changed it to "Estimating population average models in lmer or geepack". However, you did not modify the body of your question, i.e., you do not give any details of your problem with `geepack`. Also, please read the [FAQ](http://stats.stackexchange.com/faq), especially ["How do I ask questions here?"](http://stats.stackexchange.com/faq#howtoask) (please vote up helpful answers and accept answers).
I removed your comment replying to Frank because (1) it is counterproductive, from the point of view of this site, to split discussions off to some other location and (2) it's a bad idea to post your email addresses in such a public, searchable spot.
Okay, I will check that out. Thanks!
Not an answer to your question, but taking logs of zero inflated data 1) Will result in errors because log(0) is not defined. So you'd have to do something arbitrary.  2) Won't normalize the data.
@Macro Yes, but knowing the entire data set is not the same as knowing all the information in the sample.
Combining x and log x as predictors has been standard for some time in various fields. One heading is "fractional polynomials". See for example http://www.stata.com/manuals13/rfp.pdf and the references it cites.
Thanks @Stephan Kolassa, this will already help towards finding a solution.
+1, good point. I took it as an example, not the actual data, but reading again the OP certainly didn't say that.
@suncoolsu It might not look like it, but I did give an answer in a philosophical spirit!  The point is that there is no problem or paradox with someone noting that the confidence actually is 100% *in this particular case* when they used a procedure designed to assure *at least* 95% confidence in *any possible situation.*  In fact, something like this happens all the time with discrete data: in selecting a 95% CI procedure it may happen that the actual confidence is, say, *always* 97% or greater.  A careful statistician would recognize and report that fact.
@ttnphns Oh, I just mean let us say I run a k-means clustering, with k=3, for this example. Yes, clustering on only those two pcs, again, as an example. I am just making up numbers, what I am trying to ascertain really is insight into the two bullets on there - that is, given a new point, how do I go from there?
Lots of details are missing. What is the goal of your analysis? What have you tried? What precisely nlme was not able to do for you? From your questions it seems more that you need a statistics consultant. This site is more oriented to self-learning, so not putting any details (data is always welcome but in this particular case it is not enough) and waiting for complete answer will not give you any results.
yes, it is less prone to over-fitting than the resubstitution estimator, but in my experience it is still generally a big enough problem that exhaustive search is likely to be a bad idea.  Millar in his monograph on "subset selection in regression" advises to use regularisation rather than feature selection if predictive performance is the important criterion and identifying features is not a primary goal (paraphrasing somewhat).
I don't think such a value exists for a GLMM. The model you show that does have a `Residual` component is a *LMM* not a *GLMM*. In a GLMM there is a known mean-variance relationship and there isn't a parameter sigma to estimate. You can compute the residual deviance but this doesn't fit into the scheme of being a variance parameter (and hence can not be squared to give a standard deviation). That would be sufficient for it not to be shown in the output from the GLMM.
It would really help if you could post a reproducible example.  As currently stated, your question does not make sense.
After reading the [VI paper](http://www.sciencedirect.com/science/article/pii/S0047259X06002016) I believe it to be a better method. Thanks for the suggestion.
Adding `offset=log(population)` to the R model should make a big difference to the estimate of the constant, but little difference to the estimates of the other parameters, as the populations are all similar. If it makes a big difference to those as well, something's going wrong.
Another example from [Wikipedia](http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Two_normally_distributed_random_variables_need_not_be_jointly_bivariate_normal). Of course, if the variables are independent and marginally Gaussian, then they are jointly Gaussian.
It depends on what they are _exactly_, e.g. do you want simultaneous or pointwise bands? The method used in creating them can also matter. I like to create such bands using resampling, but the ones you show are too regular for that.
I am not sure I understand how are you using a binomial family with a response variable that is a percentage. Do you have the numerator and denominator counts?
"*My class was told to only include >0.6 for interpretation and omit the rest.*" -- such a rule will lead to some strange anomalies. Imagine you had loadings of 0.61 on variable A, 0.59 on variable B and 0.57 on variable C. The correct interpretation would be something like "about equal loading on A, B and C" but that rule would have you say 'it just loads on variable A' ... when A is much less important than B+C! In your case, the issue is a little less clear cut, but somewhat related to it. To answer the title question (absent the example), the answer is 'yes, certainly thats possible'.
If you were to replace each $x_i$ by $z_i=x_iU_i$ where $U_i$ are independently distributed with equiprobable values in $\\{-1,1\\},$ you would find that (a) the $z_i$ have a Normal$(0,\\theta)$ distribution while (b) $T(x)=T(z)$. Therefore you need to compute the distribution of the sum of squares of iid zero-mean Normal distributions. That should be familiar.
Could you please elaborate on why *subtract 1 because we center each observation by $\\bar x$*?
All of the models you've listed have the same general form: $$y = \\frac{\\beta_1 \\log\\left(\\beta_2 x\\right)+\\beta_3}{\\beta_4 \\log\\left(\\beta_5 x\\right)+\\beta_6}+\\epsilon$$ Might it be possible to fit the full model, then remove coefficients based on estimates and standard errors? Though that is a seven-parameter model and you have seven data points...
Thanks @chl. I'll take a look at them. Do you know if any of these packages provide calibrated outputs? (i.e. calibrated probabilities, rather than just scores)
JTT, thanks for the answer! yes, that's the experiment I'm using and the selected list of genes was selected based on epigenetic marks on the TSS of those genes. The approach that I ended up using was: 1) starting from FPKM values, sample 100k times (with replacement)the whole RNASeq dataset (let's call it A)and take the median of the 100k vectors 2)sample my genes dataset (let's call it B) and take the median 3)measure the 95th percentile and the mean of both 4)see if the average of B falls outside the 95th percentile of A 5)use Wilcoxon Rank Test to test the difference..makes sense?
Can you construct a simple, reproducible example of your problem? Just now I can't reproduce your difficultly since I don't have `mx`.
Closely related (but w/o the emphasis on ANNs): [When should you center your data & when should you standardize?](http://stats.stackexchange.com/questions/29781/)
@markovchain The cases are people who have sustained a particular type of injury, and the cases have not sustained that injury
Thank you! I use SPSS, which I see includes multinomial regression. And yes agree, I do need to consider and justify the direction of the relationship.
TMS: In statistics, context is never irrelevant!
Good answer! Highlights to _"It is better to understand the question and the science behind the data and use that information to help decide on a "Best" model"_ and all the paragraph that follows.
Perfect! Thank you so much for your help. :)
@Angelo: You should try it out.  Don't be afraid to look at the help and plug your own data in.  It won't bite.  :)  You should also know what a chi-squared test actually is, rather than just select an arbitrary test someone told you about on the internet.
Your first statement is not quite correct. If $X$ and $Y$ are independent normal random variables, then they necessarily **are** jointly normal random variables and the formula used in the second part of your answer applies to them too, and gives the same results as you stated separately because $\\rho = 0$.  A proof of the result for jointly normal random variables can be found [here](http://stats.stackexchange.com/a/19953/6633) on-site without needing to go to Wikipedia.
Thank you for the suggestion. I've added a reference that discusses this in detail, and that I found very valueable.
"the best way to predict Y from X" has nothing to do with cause and effect:  X could be the cause of Y or vice versa.  One can reason from causes to effects (deduction) or from effects to causes (abduction).
Thank you for the comment and the references!
I am sure we appreciate that. However, it also seems that you planned data collection without planning how the data would be analysed: why should that compromise  suggestions on what should be done to analyse the data?  Correspondence analysis seems a descriptive possibility here that should be familiar to you from ecological or ethological literature, and it's not perturbed by inferential nuances, as it tends to avoid them. Moreover, plotting the data to look at time dependence should take just a few minutes and seems to me to be something of scientific interest any way.
@gung: it's the most introductory book on SVM that i know.  It builds up very slowly (like: what is a vector)  completely self-contained. All the mathematics required is very well motivated
As the large SEs are in the binomial part of the model, I check if separation might be an issue - i.e. you can perfectly predict the response at some value of the predictor. Have you plotted the data as binary values (0,1) as a function of the seasons?
@Chris On the contrary, statistical significance is extremely useful here: it has *exactly* the same interpretation it would have for a huge dataset.  What you might be worried about is *power*: the ability of the data to discriminate between a null hypothesis and the alternative.  But you have no control over that.  The most you can hope for is to find a significant association, which would suggest that what you see in your scatterplots is not due to chance alone.  (Student, in his original paper of 1908, simulated datasets of just *four* values to demonstrate the t-test.)
@mpiktas: It probably means that most of the times that Google has come across the Latvian phrase, it has also come across *rms* in parallel English texts.  The obvious environment is European Union documentation.
+1. But I upvoted Martyn's answer, too: you don't lose any information about the distribution of $X$.  (There isn't any in the output in the first place.) And I think that other answer comes closer to the heart of the matter, which is that unstandardized coefficients scale with the units of measurement.
G. Jay Kerns:> Thanks, i've edited my post to adress your comment.
Good point Nick.  Why would identifiability not work?
@Jake Added a couple of lines at the end
No cut points, generally. If a variable is "good" for FA its communality should be moderate or above moderate. Small communality says that the variable is hardly driven by common factors. Who wants to keep such a variable? Too high communality may be considered as unrealistic. (If iterations inflate a communality over 1, this diiificulty is called Heywood case, and then it is recommended to reconsider and extract less factors).
$\\lambda=0$ is not allowed because division by zero is not allowed. $\\lambda < 0$ would give $f(x)<0$ for all $x$, which is not a probability density function. I believe it would be ok to take $f(x) = \\exp(-x/\\lambda)/|\\lambda|$ for $x<0$, with $\\lambda<0$, and I think this would be the density of -1 times an exponential random variable.
Linear correlation is defined without regard to the underlying distribution.  I do not know what you mean by "Pearson is not valid for non-normally distributed data."  Perhaps you are thinking of hypothesis testing?
I am aware of the following related question [Cross Validated 71511](http://stats.stackexchange.com/questions/71511/limit-on-number-of-time-varying-covariates-in-extended-cox-regression). However, my models are quite different (multi-level discrete event history) and unfortunately no one has provided an answer to the other question, so I am trying again here... Please help if anyone has an idea or opinion on this topic!
I don't have a full solution, but my guess is that solving this would involving inverting the problem. Call $P_{*n}$ the probability that it will visit the same state in exactly $n$ stops.

$P_{*2}$ = 1/12

$P_{*3}$ = 11/12*2/12 = 22/144 = 11/72
I don't know, exactly. It could depend any number of things. My first guess is that you're getting lucky w/ your 1st couple of betas, but then your luck runs out. I have to agree w/ @mark999 that "the estimates that we'd get were the original data suited to linear regression" doesn't actually make any sense; I wish it did & it sort of seems to at first blush, but unfortunately it doesn't. And it doesn't license any inferences on the original scale.
Unless the percentages *must* add to 100, you might want to include them all anyway.  (In the two examples given, the percentages ought to total less than 100 and the totals would likely vary.)
Perhaps the most famous--and likely the earliest--completely intuitive demonstration of this relation was obtained by Galton with his [quincunx](http://www1.kcn.ne.jp/~h-uchii/quinc.html).  (Don't visit Wikipedia or any of the top hits on this term, because they explain only its most trivial uses: the link I gave is the first I could find that reproduces one of Galton's illustrations and hints at the revelations this machine makes possible.)
ROC curves < 0.7 may be just as good as you can possibly get with just 0.2% positives. The base rate matters a lot in classification: http://en.wikipedia.org/wiki/Base_rate_fallacy Nevertheless, +1 for an interesting question.
I wonder whether, based on @Glen_b's comments here (http://stats.stackexchange.com/questions/362/what-is-the-difference-between-the-shapiro-wilk-test-of-normality-and-the-kolmog), one should not use the K-S test to compare empirical distributions because the K-S test shouldn't be used when parameters are estimated (?).
If you're asking about principal components regression, beta coefficients are effect size estimates. Are they not the ones you want?
*Ahem*. (i) The expression of the problem is nonsensical. How can 3=18? Surely the intent is something like $f(3) = 18$; (ii) if you can see enough to write $18=3\\times 6$, $32=4\\times 8$, etc., surely you can then see enough to split the second term in each of those ($6=3\\times 2$, $8=4\\times 2$, and so on) to then write: $18=3\\times 3\\times 2$, $32=4\\times 4\\times 2$, etc, and *instantly* spot the quadratic, $f(x) = 2x^2$. (You did the hard part, the next step is even simpler!)
please explain this "However, a scatterplot of this data does not make sense because the survival and the regrowth values are not paired" in more detail (without the use of subject specific jargon, of course). You say the uptake measurement is not from the same source, this is not enough detail.
This often happens when you've got dummy variables and some interaction.  It's a reminder that ANOVA's F-tests and the t-tests that come from a regression are not testing quite the same things. See also http://stats.stackexchange.com/questions/28938/why-do-linear-regression-and-anova-give-different-p-value-in-case-of-consideri
Take a look at the [`save`](http://stat.ethz.ch/R-manual/R-patched/library/base/html/save.html) and [`load`](http://stat.ethz.ch/R-manual/R-patched/library/base/html/load.html) functions.
Reading it more carefully, if you're using A as an input signal to predict event B, then your error types seem to be swapped - false positive would be when you predicted B (had seen A) but B didn't happen; and false negative would be when you didn't predict B (didn't see A), but B was actually present. So #3 and #4 would be opposite to what you have written now.
Sorry, my ambiguity, which I have fixed in an edit. I was referring to your response (dependent variable). (I did say "number of years".) Contrary to your assertion, counts as response are the canonical case for Poisson regression.
That doesn't seem to me to be a sufficiently well-defined problem; for example, if you add one feature, but it's highly informative w.r.t. the residuals of the "current" model, the std. errors on the existing coefficients will go down, perhaps substantially, even with no increase in sample size; if the additional feature is sufficiently close to orthogonal to the residuals, it may increase the std. errors of the existing coefficients.
Try method="SA", theoretically it should find the global minimum. Also try plotting contour plots of your function to get an idea of how it behaves. Note also that if the optimisation algorithm has stopped this does not mean that minimum exists, check the gradient and the hessian.
For that to be an ECDF, your sample size must be quite large. Ergo, given the large difference at low values, the two ECDFs differ.
welcome to the website. What is your analysis goal? I can read two options between the lines. (1) You want to select the variables that contribute most into explaining the outcome. (2) You want to have all variables in the model, as you know that all of them matter, but you are concerned about multicollinearity and large standard errors that result. For either of these tasks, yours is suboptimal procedure (and frankly the rest of the world uses R, rather than some 12-year old package).
1) is there a suitable Bayesian technique to apply to this very small forecast problem? If so what and how?
+1 If you literally followed the definition in your link, you would obtain $b$ immediately as the unit change in $\\log(Q)$ with respect to $\\log(P)$, with no algebraic rearrangement needed: in short, the elasticities are none other than the coefficients in any log-log regression.
Either that or you are somehow calling it wrong. I am usually hesitant to assume libraries are in error, especially if they are very popular, though it certainly does happen. In case of a bug you get the opportunity to help improve the library, which is nice.
rajah9's suggestion to have the stakeholders divvy up $100 is a variation of a range voting system (with a rating ballot).
The likelihood is bounded only for fixed variances. That is, in the binomial situation, the variance is $p(1-p)$; or in the Gaussian situation, if the variance is assumed known. If the variance is unknown, and has to be estimated, the likelihood is not bounded. Also, in the EM algorithm, there is a generic separation of the missing and parameters, at least for the frequentist statisticians, but the surfaces may indeed have saddles.
You should spell out "SVM", at least once.
(+1) Not all MLEs are consistent though: the general result is that there exists a consistent subsequence in the sequence of MLEs. For proper consistency a few additional requirements, e.g. identifiability, are needed. Examples of MLEs that aren't consistent are found in certain errors-in-variables models (where the "maximum" turns out to be a saddle-point).
@MvZB: no it doesn't.
While @lejohn's regression approach is probably more elegant, the Fisher's Z comparisons should work just fine and that's probably an easier way.