tttjjj/elt_bench huggingface dataset:
@Frank_Harrell Thanks for the help! To back up a bit and give some context for this test, I am analyzing how states' past experience with fossil fuel extraction is related to the stringency of their regulations for hydraulic fracturing (fracking). A scatterplot of this relationship shows that states with less experience have a wider range of regulatory stringency (some states have many regulations; others have very few), while experienced states all seem to have a "medium" number of regulations. I wanted to use the F-test to confirm that the pattern I'm eyeballing is statistically significant.
Sometimes you can get a hint by comparing your answer to the "official" one. For instance, when the answers differ by a factor of two, that's usually because one of you is performing a two-sided test and the other is performing a one-sided test. That's not the case here, though. In your situation $0.0356/0.00181 = 19.7$ is suggestively close to $20$, the number of males in the group.  That is a useful clue.
The problem with fitting many many straight lines (segments) is that you have to pre-specify how many segments you want. This is a fitting process not a modelling process. Furthermore one-time anomalies distort these line segments and more importantly auto-correlation is ignored not to mention seasonal pulses which may be present. Overall it appears to me that this is an inadequate treatment. Do I have this correct ?
@nik: I'm not sure what your x, a, b, c, d, stand for. Are a-d training data and x is a query against a trained SOM, or is x a fifth piece of training data? Or something else?
Can you support and/or explain these assertions in any way?
In line with your response, the [HSAUR](http://cran.r-project.org/web/packages/HSAUR/index.html) textbook deserves some attention.
First, you may never get a perfect model. Second, I've already explained why -- it is more general and flexible. Third, if you want to know how to use a function, read the help file.
The natural log is also useful in a semi-log time series regression since the estimated coefficients can be interpreted as continuously compounded growth rates.
Whiten, and then think in polar coordinate? Extreme values of Rayleigh distribution?
Are you talking about the code just before section 5.4? You might like to include the couple of lines of mathematics on which it is based, and define the variables..
the main flaw in your idea is that 64bit float is probably represented as IEEE 754 double precision floating point number, look it up in interweb. as you'll see it has matissa (fraction) and exponent parts. my gut feeling's that your scheme will not work for this format.
Yes, that is exactly the statement of the last sentence in parentheses in the second-to-last block.
Please do not post the same question again, rather edit or use comments.
I've been struggling with a related question, regarding priors on hyperparameters for negative binomials. I don't want to confuse you, but it might be helpful to look at a related model. http://stats.stackexchange.com/questions/94923/hyper-prior-for-negative-binomial-in-hierarchical-model-using-jags-bugs
This depends entirely on *your* objectives: why are you predicting? What are you going to do with the predictions later on? What's the cost of being wrong? How precise do you *need* the predictions to be to make use of them at all? ...
I was afraid of this.  Thanks for your input.
The z-test doesn't "break down" with large samples, it behaves exactly as it is supposed to. The problem is you're using hypothesis tests when they don't correspond to what you *actually* want to do. If you insist on whacking screws with a hammer, it's not the hammer's fault your cabinets look funny and the doors occasionally fall off. The fact that you only notice the problem with very long screws doesn't alter the fact that it's the wrong tool for the job even when the screw is short. Rather than "what should I do different in large samples", focus on answering the right question at any $n$.
Nothing is wrong, you just don't fully understand dummy variables. Check out http://socserv.mcmaster.ca/jfox/Courses/SPIDA/dummy-regression-notes.pdf
For your 1st question: The $\\hat{\\beta}_0=3.2$ is given, so you know this parameter. Does that help? If this is a homework, add a self-study tag.
Its a within subject design. With 4 slope values for each subject. Each slope value was calculated, in a different condition, over 5 independent variable values - and there are actually 4 measurements for each of these iv values.
good answer, it could be improved by writing out the steps to get each number, e.g. $6.67\\text{%}=\\frac{10}{10+100+40}\\times 100\\text{%}$.  similarly for second step $4.67\\text{%}=6.67\\text{%}\\times 70\\text{%}$.  If you have the equations as well as the words, people are less likely to get confused
NHANES - National Health and Nutrition Examination Survey http://www.cdc.gov/nchs/nhanes.htm
1) The i. operator does not work with xtoverid. You can just generate the dummies by using "qui tab year, gen(dyear)" and include them directly in the model instead of using an operator. 2) Yes, because it implies that RE is not consistent in your case. But try again with the complete model including the dummies.
Every variable is random, but can be observed.

(BTW, using this model to play with PyMC after reading your IPython book - it's great, thanks)
Ah, sorry about that. You can basically think of "magnitude" as a measure of brightness.
The prior month last year was a large value.
Thanks for the reply.  That makes sense.  I was basically attempting to replicate the SPSS output, but SPSS obviously wasn't applying the Rao & Scott adjustment.   I looked it up and honestly couldn't understand exactly what it was, but it does seem to be more appropriate for my uses.

Any thoughts regarding pair-wise analysis of factor levels after rejecting the null hypothesis of independence with a chi squared test?  Thanks.
I am not satisfied with this solution since it does not apply the additional information contained in the power calculation. Conceptually, power informs us how much our result depends on *n*; simply looking at the *p* value, it may just be far from 0 because a small sample is not reliably representative. Phrased differently, high *p* in a low power study only says that even if the nil-null is true, values different from 0 are not unexpected in small samples such as the given one. Right?
@FarazKhan I would suggest you ask a new question here on how to use K-means clustering on your vectors. This will better categorize each gap into "internal" and "discontinuity".
I have read that in the last few years, use of techniques such as use of rectified linear units and training with drop out has rendered the RBM layer obsolete. According to [FastML](http://fastml.com/deep-learning-these-days/) these techniques provide better results, faster, and adding RBM layers no longer improves things past that point.
How I'm handling over-fitting: Build a model with 75% of data. And, then fit the model with the same set of variables for each additional week of data. And, see how the coefficients for each variable vary across weeks. If the variance of coefficients is not very high, I conclude that the model is stable and responds well to new data/rejecting any fear of over-fitting.
Ordinarily, one evaluates estimators by looking at their *risk functions,* which plot the expected loss versus the parameters. Here, by fixing the parameters, you may have produced a misleading analysis.  After all, it's *always* the case that a stupid (constant, data-ignorant) estimator can produce very low expected loss: just set it equal to the correct parameter! This leaves me wondering what the simulation has really shown here.
Does each entry belong to exactly one class?  Or it is possible that some entry belongs to class 12 with certainty 1 and class 34 with certainty 2?
What's wrong with what you have so far? Why is what you have (the **k-means cluster analysis**) unacceptable?
I thought the consistency of an estimator is independent of the sample size. For instance, the sample mean of a regular probability distribution is $\\sqrt{n}$ consistent by the central limit theorem.
@whuber in hindsight i agree with you, i will fix the title and the text. i wasn't sure if the matrix of 1s were really a result or some sort of erroneous output due to the fact that the correlation could not be performed correctly.
+1. I dared to edit it slightly just to add emphasis where I thought it necessary. Please see now, whether you agree or not.
What do you mean by "finding which parameters of the model are likely"? Do you mean finding which **values** of parameters are likely? Or something else?
From my interpretation of the answer: Yes you can, but CUSUM model can model some more general situations as well.
+1.  Also, $h(x)$ is called the *carrier measure* because in the measure-theoretic formulation of exponential families (e.g. Shao's book), it is an actual measure.
Thanks, I'll take a peak at Blei's code and the Neal paper.
Your question appears to be more psychology than statistics. It might be more statistical if you could perhaps operationalize what you mean by 'fraudulent', since such scores are going to be subjective. Could you clarify more precisely what you want to avoid and what you want to achieve?
Hi Tiffany, If these are observational studies and therefore you don't have an separate effect estimate for the effect of A on Y (and B on Y) in the 7 studies then I would recommend you conduct a multivariate meta-regression (http://r-forge.r-project.org/projects/metasem/) with Y being the dependent variable and A and B being the independent variables.
(-1) This answer is wrong: each ROC includes a point with recall 0 (trivial model always predicting the negative class). AUC 1 does not imply perfect recall and precision, it implies only that a dichtomization threshold *exists* which yields perfect recall (true positive rate; sensitivity) and fall-out (true negative rate; specificity).
(+1) Nice, concise answer.
@cs I think the point is that there are several analytical options: one is to view the ratio as a continuous variable; another is to use a Poisson model for Lph* with Lph as an offset.
" the first sentence is misleading", do you mean the definition of complex Gaussian distribution shouldn't require independence between X and Y?
Are you asking about Mahalanobis distance?
Why do you need to "fit a distribution curve"?  It is rare that arbitrary curve fitting is of much help and when it is, that is because the fit is good, simple, and clear, which appears to be a highly unlikely outcome in this case.
That's a very good resource, thank you. In my case the models are based on different predictors data (not just different aggregation periods), but it looks like the methodology is still relevant.
@Michelle - unless I'm missing something, `count` is an integer (as it is fed into `vglm()`). See `> class(alligator$count)
[1] "integer"`
@ErikPukinskis The math checks out. That function is correct and should return 1. The coefficients though make me think that your variables are normalized first.
You may well have separation (based on the warning, & possibly on the z / p values), but it is not what you describe in your 1st paragraph. Since `supplementary food use` & `nest initiation date` are both covariates, the implication is that you have some collinearity / confounding. *Separation* pertains to the response variable. (Nb, collinearity also expands your SEs & reduces power.)
But what about multicollinearity. Elastic net does allow to select multi collinear features which is not good isn't it?
I forgot to mention that the GP is not modelling a time series, but it takes in (x, y) pairs with x being the location, that will be visited multiple times.

The changes will be abrupt, but they will only be at some locations $x$, whereas at the rest of the state space the distribution stays the same. I hope that is answering your question well enough.

I will look for the Box Jenkins models you mentioned, thanks for the pointer :)
I don't know enough about the power of Mauchly's test to tell whether it is good enough at picking up deviations that affect the validity of the p-value. The difference between the corrected and uncorrected p-values are tiny, and we are only talking about it because the magic threshold got crossed. I would stick with the same assumptions for the subset as for the whole set.
According to the part you pointed to, you compute this: $s_{\\overline{X}_1 - \\overline{X}_2} = \\sqrt{{s_1^2 \\over n_1} + {s_2^2  \\over n_2}}$. See how the $s^2$ terms are divided by those corresponding $n$'s? The square of the LHS is the variance of the difference in sample means. The two $s^2/n$ terms on the RHS are squares of standard errors of means. They are calculated from standard deviations of samples but they are different from those standard deviations (they're typically much smaller)
@AdamRyczkowski Actually, they have changed the name back again. They now call it "IBM SPSS".
The actual data *does* matter to make this reproducible. Why are you calling `t` on all your variables? That is not usual and would suggest your data is in a strange format (or shape)
I believe that value of 4.2 may be a misprint for -4.4, which is computed as -0.044/0.01: merely the conversion of the change in betas for AGE from model 1 to model 2 (from 0 to -0.044) to percent per year.
Exactly.  Theano is very alluring!
I don't work in this field, but I don't see why summing the observed values & then dividing each value by the total should make the algorithm unwieldy. It seems to me that if the models are too complex, slow, or numerically unstable (etc.), the problem is most likely elsewhere.
The 3 flavors of percentile behave the same way as quartile for me. For the 5-37 data, PERCENTILE.EXC(range,k) gives #NUM! for k={0,1}. For k=0.25, PERCENTILE.EXC gives 10. If I throw away 5 and 37, it gives 10.5, which agrees with the other 2 ways.
@user995434 All of this falls under a more general umbrella called the non-parametric bayes models. I didn't want to confuse you earlier than what you might already be at the moment :-). However, since you seem to be very keen on this, you may be willing to take the longer route - http://mlg.eng.cam.ac.uk/porbanz/talks/npb-tutorial.html
Thank you! If I leave out "identTILavg" from the regression, the coefficient of "fotball" becomes positive as well as almost significant. In other words, "identTILavg" greatly impacts the results of "fotball". These variables have a big correlation (0,53**). What I don't get, is how the coefficient of "fotball" becomes negative when the correlation between "identTILavg" and "fotball" is greatly positive.

Do you have a clue why "kampkvalitet" turns out negative in the regression as well? It has no negative correlations.. I know its non-significant, but I still have to explain it.
What is to be predicted is heat (the heat consumption of the house).
I discuss link functions extensively here: [Difference between logit and probit models](http://stats.stackexchange.com/questions/20523//30909#30909), focusing on regression for a binary response variable. Although only a little of that discussion focuses on the meaning of a link function's being 'canonical', it may nonetheless be helpful to read. Note that to understand the distinction b/t & advantages of a canonical vs non-canonical link function requires going fairly deep into the math underlying the GLiM.
Does it satisfy all the conditions to be a distribution function? They're not difficult to check. Shouldn't this be a self-study question?
In order to assign physical significance to the formula above, I would think you need to give some sort of physical meaning to your data.  No, X.0 is not a quadratic form.  Geometrically, the matrix $v1 %*% t(v1)$ is a rank 1 matrix that spans the column space of $v1$. Knowing what $X.center$ is might help expand this line of thought.
@BakakP Thank you so much, it would be greatly appreciated if you could also add in any relevant algebra steps too. Thanks again.
Thanks very much, Rob.
Fixed! I was posting from a phone earlier, which makes formatting difficult.
Clustering methods (at least classic ones) don't bother about the shape of clusters and don't have shape constraint. But they do differ in their ability or inclination to produce clusters more or less of a specific shape.
I can only add a direction. A more general theory that incorporates PCA is the http://en.wikipedia.org/wiki/Karhunen%E2%80%93Lo%C3%A8ve_theorem
This is heavily dependent on eigenvalues and eigenfunctions (which are themselves generalized eigenvectors). This is pretty math heavy, and over my own head, but maybe you can read it ;)
Missing values? `nrow(na.omit(x_cv)) == 1000`?
@VijayKotari This looks surprising but you don’t lose anything with this method, cf my edit!
It appears this argument would apply equally well to any discrete distribution with a countably infinite set of possible values, but then the conclusions are obviously wrong, so something is the matter.
"*I have the wage data so I can directly see that the raw data is log-normal.*" -- then you must be a lot cleverer than me. I seriously doubt I could just look at a set of numbers (or even a Q-Q plot) and distinguish lognormal from say a 50-50 mix of gamma and inverse gamma, let alone something even closer to lognormal. The rest of your comment is already answered by my first comment, is it not? If you don't think so, could you clarify what is lacking?
@Caroline, you are welcome.
Dr. Harrell, I will post more details with self-contained code soon. thanks!
Not to mention that every individual rater probably creates their own scale, with their own variations on the differences between levels... this stuff is absolutely loathsome. Gary King's papers on [anchoring vignettes in survey research](http://gking.harvard.edu/category/research-interests/methods/survey-research) provide a good overview of the problems here, and an interesting solution (that can't be applied to this particular case).
Recently, the TeX formatting in many posts seems to have gone a bit weird. I tried to edit this post to make `$\\text` into `$ \\text` which would apparently have fixed the problem but singe-space edits are not allowed. - at least I'm assuming that this isn't just an error on my end but happens everywhere.
More or less, yes. I don't know what methods you were using before, but for models like this it's generally accepted that the most accurate confidence intervals are obtained by bootstrapping, e.g., by using `lme4::confint()` with `method = "boot"`. That method will not care if your data is balanced or not.
First, better to try to figure out why the models 3 and 4 can't be estimated.

It is *possible* to account for interaction effects without the main effects, but it's rarely a good idea.
Thank you.  This is what I was looking for is that it's not appropriate to do this.

For the ETS() best fit forecasting by months, is there a minimum number of historical months that I should feed it for optimal/interpretable output?
@whuber No, the sum must start at $n=1$ (try the case when $P[X=42]=1$).
@Insanodag: so you suggest I can multiply data matrix with loadings matrix?
I will go over this more thoroughly later, but I believe this answers my question. Thank you!
It seems dassouki standardized the distributions, and that is why it ranges between -2 to 4.
Steve, glad you like the text.  It sounds like your assumptions under the null hypothesis fit well with a "permutation test" like whuber's answer (which uses simulation to get around the computational challenge--best option I think).  Depending on the categories of the top 3 responses, tests on the 3 categories (env, econ, soc) might be interesting too (like you said), in the same spirit as whuber--what's the distribution if everyone votes randomly, vs. how did everyone actually vote?
@Narendar reddy kalam:  Why have you chosen the quantile regression model? What are the property of the estimated slopes that you want and which lead you to choose this estimation technique?
As the comments note, simulation can show that, in a known situation, stepwise regression will overestimate $R^2$, and they can show how much. But it cannot show how much inflation there is in a situation where you don't know what the population values ought to be. That is, not only are results from stepwise biased, they are biased in ways that are very hard (if not impossible) to estimate.
What is the null hypothesis? That the population sizes are the same? I don't need a statistical test to see that the null is false. Do you mean testing whether the population growth is exponential, or..? With the second example, are you asking about whether the average income has increased? In that case, a $t$-test would work (possibly log transforming the values since incomes are known to be highly skewed).
Instead of putting in screenshots of the code output, it would be preferable to just copy/paste the output... as it is I cannnot read it.
+1 @gung thanks, yes (I pondered whether it was worth mentioning `anova.lm`) though it's perhaps a little more concise than what people might be used to in other packages. A simple example to see one in R for anyone interested is `anova(lm(dist~speed,cars))`
Can you explain the formulation of AR a little better here? My understanding is that an AR(1) model was just a linear model having a conditional covariance for the outcome structured as the following: $\\Sigma_{ i, j } = \\rho^{|i-j|}; \\rho \\in (0, 1)$.
@David: use Marsaglia 1965 instead as updated in the answer.
I do have a common intercept $\\beta_0$, but I have a number of factors in the model. I was just showing one factor in the example. Is it possible to suppress the intercept with more than one factor?  I thought that you could do it for one, but then it broke down with multiple.  That would fix my problem, as it would allocate the overall intercept to each factor.
Why shouldn't it be "chose"? I am pretty sure that is what it is.
I am not that familiar with that code. It might also be a bit outdated. Try WEKA or R, they provide recent random forest implementations.
Hello Olga, auto.arima actually estimates seasonal components too. For example: auto.arima(AirPassengers) returns a seasonal model...
Then it sounds more like you are in way over your head. I would recommend learning more about the individual parts. Trying to teach you how each of those work would require weeks of lecture and self study, and is in no way appropriate for a QA format.
Nice book you recommend there. Another one I could recommend is [Applied Linear Statistical Models](http://books.google.com/books/about/Applied_linear_statistical_models.html?id=0xqCAAAACAAJ) which contains a few sections on selection criteria, model selection and model comparison.
Thanks, indeed the order does not matter. I could use a measure like the one you proposed but one of my main problems is that I do not know on which framework I am. It is not "strict sensu" multivariate categorical data and I would like to get more info about this.
@Tomas Good point. In point $3.$ I would suggest the use of an appropriate untruncated prior.
The joint law of the $X_n$ is not important here. The result is a straightforward application of Chebyshev's inequality (and the definition of big O).
Can you be more clear about where passers by are counted? Is it cars that travel though the street segment (in between the crossroad)? Or is it where pedestrians actual cross the street (at multiple points or just within the street segment)?
Thanks Ben.  So, best way is to simulate?
This is "standard confusion" in econometrics!
Please notice that setting the derivative to zero only finds the critical points: to optimize the function, it is essential that you also evaluate it *along the boundary of the domain of definition.*  In the case $m/n\\lt 1/2,$ this means the MLE estimate of $x$ is $\\hat{x}=\\infty$.
(+1) Thanks for the links.
(+1) Thanks Matthias, and welcome to CV. It will take me some time to fully digest your answer, but it is a very interesting approach.