tttjjj/elt_bench huggingface dataset:
checked cross-validated and could not find anything that responds to this question.
It's on his webpage, see http://www.stat.unc.edu/postscript/rs/ISI89.pdf
Okay, but that's like saying that OLS is suitable for causal inference if we have the right data, i.e. without endogeneity. With ideal data as you describe it, GNC works perfectly fine for this purpose. The problem is that we rarely have this kind of ideal data which is why those microeconometric methods for causal inference were developed in the first place. The definition of GNC here is the standard textbook definition and I'm talking about it as a method for causal inference with minimum assumptions on the data.
Some context to this question or further explanation would help, because on the face of it the answer is obvious: because trigrams include all the information in the bigrams, then any reasonable use of the trigrams cannot possibly be worse. Having said that, it's not hard to imagine circumstances where trigrams appear to perform worse: one might over-fit a model with them, for instance.  But that depends on the application, the context, and the skills of the applier, among other things.
Your scheme never generates the value $0$. So it can't be right.
Thanks theomega. I was able to plot a useful tree diagram.
If, after considering some more sophisticated methods of comparing time series, you agree that Pearson's $r$ isn't appropriate to your analysis, Kendall's $\\tau$ and Spearman's $\\rho$ aren't likely to be much better. You may need to reformulate your question in that case, because you wouldn't just be talking about a simple correlation, I suspect.
I up-voted and agree with everything. In this case where it appears that there is a desire for risk assessment rather than classification I might emphasize a well calibrated model with Brier score/c-statistic/validation curve (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575184/) But these methods of assessment may not work well for non-likelihood based models (trees/tree ensembles etc) that the poster appears to be considering.
@whuber: Thanks: both points need qualifying. I'll edit my answer.
Ahh, I thought you were just trying to estimate the "underlying response rate of each zipcode"
@GusRustam This is exactly the physics aspect that I prefer not to get into. And what I am looking for is the mechanism, in the math or non-technical ways, of how it is done. Thanks for your answer
The [Wikipedia article on optimal design](http://en.wikipedia.org/wiki/Optimal_design) lists no fewer than *eight* criteria. Because many of these are stated in terms of the information matrix or its inverse--both of which have direct relationships to the principal components--then PCA is implicitly involved in most of them.
This solution requires no less calculation than the one I gave (unless $e^{rt}$ is less calculation than $(1+r)^{t_0-t}$, I hardly think so).  Another thing is that the interest rate in the PV and FV calculations are not consistent in *r* for in one place you are annually compounding, and another place you are continuously compounding.  your equations imply that $(1+r)^{t}=e^{rt}$.  $r$ in the exponential should be adjusted to $log(1+r)$.
PLease specify what language/program you are using in your question. It just looks like gobble-de-gook.
This reply is identical to http://stats.stackexchange.com/questions/44586. Can you provide more details so that your answers remain self-contained and indicate how they apply in the present context?
The British also drive on the wrong side
+1 for pointing out that, with very small sample sizes, sample correlations are usually quite high. As an extreme example, if $n=3$, you have a very good chance of getting nearly perfect correlation. Typing `cor( rnorm(3), rnorm(3) )` repeatedly in `R` will reveal that. Also, I noticed you reviewed an edit today - thanks for pitching in!
@Chiradeep: With three out of four pieces of information you can estimate  the fourth. Broadly speaking: the product numbers of rooms available times occupancy rate equals the number of rooms occupied each night, which in turn equals the product numbers of arrivals per day times length of stay.
@gelraen: this was written before you said "Sample size varying from 5 to few hundreds" and assumed the sample size was 1.
:re: your point #1, "You investigate a paired data solution," -- thank you for pointing that out. Using the bootstrap method, I **should** be taking _the difference of pre- & post- scores_, running the bootstrap, and then seeing if the 95% CI contains 0. Is that correct?
If your expected split is truly 100%/0% and you get *any* values in the 0% category, then you can instantly reject the null at every p-value, with no need to do a test: the chance of that occurring is zero. Indeed, this suggests that what underlies your 'expectation' doesn't conform to what the circumstances would need to be for a chi-squared test to apply in the first place. What makes your expected counts expected? Note that when you say 0%, you are saying there's *no chance whatever* to get anything but the other category.
@chl's thorough answer to a Q last December is relevant here http://stats.stackexchange.com/questions/5171/testing-paired-frequencies-for-independence/5258#5258
+1, your list provides a nice, clear way to understand the distinction.
The gain of sampling WOR depends on the fraction of the population that is sampled. This may or not be "very little". Moreover, in multi-stage samples, if PSUs are sampled WR, then a different set of second stage units should be selected each time a PSU is drawn.
Rather than throw a sequence of probability distributions at this problem (which I think is an utterly mistaken enterprise when the data is just guesses anyway), why not describe what the original underlying problem you're trying to solve is?
This question appears to be off-topic because seems to be about how a particular part of some software works rather than about a statistical issue. If this falls under the topics listed in the help, could you please edit to clarify.
Thanks for your comment, but actually i made a mistake.I have edited my second equation. I know that a>=0. sorry for my typo mistake.
I have a laser range finder imaging a target shape.  The points on the target closer to the sensor are more densely sampled than the points farther away.  We want to estimate the center of the target by the mean of the points.
This is explained at http://stats.stackexchange.com/questions/3734. That answer is long, because it asks for "intuition." It concludes, "This simple approximation, though, suggests how de Moivre might originally have suspected that there is a universal limiting distribution, that its logarithm is a quadratic function, and that the proper scale factor $s_n$ must be proportional to $\\sqrt{n}$... ."
I like these ideas.  Also, some less visually-oriented people will prefer to check (if normality holds) whether these 2 cases are an unusual number of standard deviations beyond the mean.  There are also tests for outliers, though it seems they're controversial.
+1 for the chutzpah
@Glen_b If it is possible to prove that no distribution exists with all of these conditions, I'll accept an answer with that proof.
(1) No, but it gives the right Type I error rate, in moderate-to-large samples (2) Thanks, but I respectfully disagree. Using t-tests over Wilcoxon makes it much easier to bridge the gap between testing and using confidence intervals. 

If one only wants to do testing, and never looks beyond the two groups in a study, Wilcoxon of course has situations where it works well. But often we don't want to do just testing, and want to help users generalize the results to other situations; the Wilcoxon test is then not helpful.
You have a fundamental problem: most proportions have a positive chance of being zero, whence the ratio (of independent proportions) has a positive chance of being undefined.  This can present severe difficulties for approximate methods (like the delta method) and suggests that normal approximations should be viewed more sceptically and tested more rigorously than usual.
I suppose I did :) However, I found the mistake and it had nothing to do with the choice of distribution. For anyone interested, I'll write an answer!
Am I just taking all of my x data points and applying ln to them? Giving me a new set of data points?
A way to derive the formulas is by Matrix Differential Calculus. If you look at Abadir, Magnus, "Matrix Algebra", Cambridge, pag. 388 you will see the expression of the 2nd order differential for the multivariate normal. If you set ${\\rm d}\\,\\mu=0$ and ${\\rm d}\\,\\Omega=0$ and use the identification formula at the beginning of the chapter you should be able to find expressions for the Hessians $H\\,  l(\\mu)$ and $H\\, l(\\Omega)$.
@onestop: Thanks for adding the DOIs/links.
How can we even have a guess what was done without knowing the paper? Please put a reference preferrably with link into the question.
Thank you Andy for your answer too, I will read the references! I did not know of the drawback you are mentioning!
$\\beta$ is not a root; if it were, $x_t$ could be complex which it is not. It is the coefficient that is used in the auxiliary model to test the presence of a unit root. But I think I see what you mean now, and yes the ADF test only tests for a traditional random walk, and not an oscillating one.
Could you ask a specific question, please? Your post contains only statements.
Thank you very much for this detailed answer and that working solution. I didn't suspect that a basic arithmetic task such as recovering errors from a MA(1) process could hide a matrix inversion... The issue is that my sample size will go way beyond 3000.
Thanks for the sound explanation. I was confusing myself with samples selection but now it is clear.
@iliasfl My positive set is around 2000 and negative set is around 3000. I have reduced the number of features to less than 100 using feature selection.
Hi Greg, this is a wonderful list of activities, thank you.  By the way, the last one (the helicopter experiment) was used in the first class I took (in the University, several good years ago) in the course "design of experiments and ANOVA".  I remember it as one of my earliest "fun" experiences with statistics.
You need the latter approach.
Thank you for your answer. Is there a way to display the values of the regularisation parameter? Additionally is there a way to then choose between the solutions based on this parameter? (Also is the parameter lambda?)
I was minimizing the -2loglikelihood @Glen_b
Answers to some of your questions have appeared in other threads, such as at http://stats.stackexchange.com/a/14528, which describes a situation in which *every* one of a set of only mildly correlated predictors appears to be insignificant even though they collectively belong in the model.  It's a good set of questions but leads to a large number of considerations and techniques; entire books have been written about them.  See, for instance, Frank Harrell's [Regression Modeling Strategies](http://www.amazon.com/Regression-Modeling-Strategies-Applications-Statistics/dp/0387952322).
Thank you for your detailed answer, very much appreciated. I am in the process of reading about LASSO, its intuition, and implementation in R. I am very new to the field of "model/variable selection" and still confused about the different methods.
Sorry that it took me a while to look into this. So far your solution worked great for a toy example, which I created for test purposes. Let's see if it also converges in a real world setting. Again, thanks for your help.
did you get cut off there?
@ran2, Yes the linear.hypothesis() function looks like it would work. So long as your getting a new package, this one might be simpler: http://hosho.ees.hokudai.ac.jp/~kubo/Rdoc/library/aod/html/wald.test.html

I assume that the R^2, F, etc output is from the LM function, right? Since you haven't specified the HAC standard errors at that point, you can't change that display.
@B_Miner, @Mikhil, added model description and code. Hope this helps.
@Vladimir Chupakhin  If your data consists of a  set of X vectors and a set of Y vectors, all of the same length, and the problem is to predict Y given X, then the procedure I outlined is sufficient.
For some thoughts on a closely related matter, see the thread at http://stats.stackexchange.com/questions/13858/how-can-i-improve-my-analysis-of-the-effects-of-reputation-on-voting.  (BTW, I haven't a clue what your example data mean.)
@Glen_b Thanks for replying. So the jist of your argument is, I simply don't have the information required to make any useful estimate?
Please clarify what you mean by "handling" an "input distribution."
Could you explain *why* you are carrying out this analysis? Are you looking for outliers? Changepoints? (If so, what kind of change?) Something else?
N in the hundreds of thousands, p in the 100-1000 range. Not sparse.
Did you miss some t()? Just get some  `non-conformable arguments` when I try to it with the example data provided by `AER::tobit`. Would you mind giving it a try with the example dataset?
OK, so you actually have a matrix of size 500 by (40 times 18 time 4).  That's a matrix of size 500 by 2880.  This is still very small.
Thanks Peter,
I am already comparing mean and median of the two distributions. However the distributions are non-standard, not shaped like gaussians, I was wondering if there was some non-parametric way of comparing two distributions.
I don't know about random forests, but if you have built these models from scratch, you definitely know how to work on extending things further. Have a look at recent papers for applications, etc.
because there is no linear combination of the dimensions but one that depends on a kernel function. Is it in this setting possible to determine the weight one dimension has for a (non-linear) principal component?
ok, thank you for your answer
Yup, I see you wrote your own answer but I reorganized mine to be more correct.
A further problem is that the center of the distribution can change over time - for example, it wouldn't make sense to set those bounds using data from the early 1900s, when Colorado was a haven for TB patients.  So what's an appropriate way to keep the lines up-to-date with long-term changes in the process, while still being able to identify deviations?
@CravingSpirit, what you need is a book on linear algebra that covers inner products and orthogonal projections. See e.g. [Linear Algebra Done Right](http://linear.axler.net/), Chapter 6.
@NeilG, yes, you're absolutely correct - I've edited my answer to reflect this. Its sometimes written $\\mathcal{L}(\\theta | D)$ because the likelihood can be viewed as a function of the parameters holding the data constant.
Thanks for your advice. But my real dataset is precisely a situation where it is sensible to drop the intercept. I am interested in the ratio y/x, I want to assess whether the group has an effect on this ratio. I have two alternative ideas: 1) ANOVA with the response y/x ; 2) ANOVA on the residuals of the regression without intercept. Do you have an opinion about these ideas ?
Hi Suncoolsu - you got it right, that's what I was wondering on how to do.  And my question was - is it a reasonable solution for that question.  From your answer, I deduce that you believe it is valid to do it this way (Also chl directed me on where to look for implementing this within R, which is great).  Thanks for you input.  Tal
See the comments at [this post](http://stats.stackexchange.com/questions/79778/how-does-one-show-that-there-is-no-unbiased-estimator-of-lambda-1-for-a-po), which might be of some help
"On the other hand since both models try to model one variable, they both produce the modelled values of this variable. So the question of model comparison is identical to comparison of modelled values to true values."  <---  I'm going to compare the MSE of modeled values compared to true values on an out-of-sample portion of the data.  It seems the best for me to do this.
Stask and @gui11aume, these three answers are nice but I think they miss a point: what about the *consistency* of the MLE for dependent data ?
Your answer has no relation or explicit link to the specific question.
Thanks Scortchi. Consistent with your observation, I myself have always used "model" in both cases.
I edited it, I feel it is quite clear.
No, because survivorship can vary over time.  A search of Google Images for "survival curve" will turn up plenty of plots of multiple survival curves that cross each other.  Consider two populations of people at birth, one of which has high infant mortality but otherwise is very long-lived.  The second population will start out with a higher survival rate, but eventually it will drop below the curve for the second population.
It's OK, it happens like that sometimes.  I think you are still thinking in terms of calculating probabilities; we don't do that with MCMC, we generate random numbers - and we don't need to calculate probabilities as part of the process.  It's a clever way to AVOID having to calculate probabilities, which, due to our (problem-specific) inability to calculate the constant of integration, we can't do anyway.
@Gaël_Laurans: sorry, I missed your comment - you were first.
@Lillian Yes you use the standard error of the difference but because of the negative correlation it is a more complicated calculation and is not the same as what you would get from comparing two independent binomial random variables.
@dikran marsupial -  Perhaps a better way to explain it is that integration is often well approximated by estimating a parameter by ML, and constraining that parameter to be equal its MLE.  The Laplace approximation provides a "correction factor" to this intuition - in the same way that REML does.
Coding colours with `1`, `2` and `3` will give the interpretation that `blue` is two times larger than `purple`, and `red` is three times larger than `purple`, which usually is not a case.
In my case the two models are a spatial error model $f(\\beta)= y = X\\beta+\\epsilon(I-\\rho W)^{-1}$ and a spatial Durbin model $g(\\beta,\\theta)= y = \\rho W y + X\\beta + WX\\theta + \\epsilon$. The SEM is nested in the SDM (under the case that $\\theta = -\\rho \\beta$, so I believe that @AdamO's joint test applies. I'm trying to make the argument that because the SDM is general, tests should try to reject it, not make the case for it. Is there statistical validity for this argument?
@AndyW Thanks for the heads-up.  I forgot about that thread (even though it's in my favorites).  I added this resource to that answer.
@Jakubee, I have edited my answer, see if that helps you.
@Wolfgang, I'll look at those papers when I get a chance. I imagine one of them (older ones especially) must come pretty close to what I did in my answer.
@Frank - why is it "not correct" to group BMI?  this seems perfectly reasonable, so long as the results are appropriately interpreted.  You could well be testing, for example, whether being "underweight" "healthy weight" "overweight" and "obese" are associated with smoking, where these terms are defined by the ranges of BMI.  I see no "wrong" here.
yes there exist ibm spss modeler and sas rapid rapid miner for datamining. Yes I will work on different locations, but I can also connect remotely on my PC in home.
Also tried that and gives the same error. Wish I could give you more info but it only gives that one line error.
We welcome questions like this, @user38208, but we treat them differently. Please tell us what you understand thus far, what you've tried & where you are stuck, & we'll try to provide hints to get you unstuck. To better understand the process, you should read the [wiki](http://stats.stackexchange.com/tags/self-study/info) for the `[self-study]` tag.
Outlier rejection is a very dangerous approach to data analysis especially when so many are removed.  What justification do you have external to the idea that statistics show that they are relatively extreme.  You could be throwing out some important and informative data. You might want to consider looking for other factors that could improve the prediction rather than an apparent arbitrary rejection of observations to the extent that the sample could now be biased.  If the data is biased the model can look good on the remaining data and still be a poor predictor!
Since you're using R, why not just use auto.arima() from the forecast package which will do the work for you.
*create latent varaible scores using these as independent predictors in a regression model (note that the standard errors here aren't correct because they don't account for uncertainty in the scores).*    



Which is what I thought I was doing by Analyze -> Dimension Reduction -> Factor and getting the resulting scores (see above). Is this something different?
@stats: See my edit.
Hi smillig, sorry for my late response. this is not close enough, this is exactly what i was looking for! thank you so much, I appreciate your helping hand! best marc
Could you please explain how your question does differ from the "Robust outlier detection" question you reference?  Based on what you have written, they appear identical.
@Cam.Davidson.Pilon no, not necessarily. They have arbitrary values.
Why did some people get test A and some test B?

In general, test equating for this type of test is pretty complex, and just z-transforming won't do it.
@Jake And, look at the differences between ratio / interval levels of measurement and compare the properties of those levels with ordinal and categorical.  This, along with theory of validity, determines what can and cannot be added together.
As far as I can tell, this has nothing to do with feature selection; can you update the title?
Look up (1) Flexible linear regression, (2) Kalman Filters.
It's not really clear to me from this what you want me to do.  Wikipedia's [ARFIMA](http://en.wikipedia.org/wiki/Autoregressive_fractionally_integrated_moving_average) page wasn't very helpful either. How does using a more complex model help things?
@RioRaider if I am discussing 1 way, are there others? Is there an alternative way (using AIC as a foundation) to get at the effect size/importance of each parameter in the model?
@whuber Great, thanks a lot, I did not know about offset. I will look into it. Is it possible to incorporate an offset in GAMM as well?
My PCA was based on the correlation matrix, and the weights were re-scaled already. Is that what you mean "using zero-centred data?"
@MichaelChernick well, I've edited my post to add a simple C++ implementation of a simple example. Can you please check if I understood it well, or there are some misunderstanding ?
As I mentioned in the preambule of my answer, there are various definition of what the 'Sdev' of a matrix means. I don't know which one of these definitions is the generally accepted one in the context of image processing.
Thanks for the reply, this is in line with my prior.  I guess I was just wondering whether there was something un-intuitive that I should be thinking about, that I'm not.  Maybe not!
I'm not sure why the software chose those two to omit - it could have just as easily omitted 'own'. Try removing the constant (_cons) and I guarantee you there will be one less 0. Can you help me understand why you only have 4 rows of data?
Of course Macro is correct: because x and y play equivalent roles in the question, which slope is more extreme is a matter of chance.  However, geometry suggests (incorrectly) that when we reverse x and y in the regression, we should get the *recipocal* of the original slope.  That never happens except when x and y are linearly dependent.  This question can be interpreted as asking why.