tttjjj/elt_bench huggingface dataset:
[Ripley's K](http://www.public.iastate.edu/~pcaragea/S40608/Notes/Dixon_Ripley_K.pdf) is popular metric, and is straightforward to apply in only one dimension (if its already available in your statistical software of choice for spatial point patterns you can simply pretend one of the dimensions is a constant). Quadrat stats can be underpowered (and ad-hoc). For nearest neighbor distances there are alot of different global tests, I would suggest checking out the [`spatstat` R package](http://cran.r-project.org/web/packages/spatstat/index.html) and all of the available tests.
(i) If you have no intercept ("`~0+`") the definition of (and meaning of) $R^2$ is different; (ii) even when you have an intercept, and the model is correct, and the mean *clearly does* change linearly with $x$, $R^2$ can still be almost zero; (iii) why does your question have the 'logistic' tag?
thanks caracal so much, I really appreciate it
This is untrue. The p-value of a parameter is not proportional to the importance of a corresponding feature, either, and neither is the p-value of the likelihood ratio test. Nor is there any guarantee that the variable importance as measured from the results of RF will match the p-value of the feature. Why? Because the p-value has nothing to do with out-of-sample performance whereas the RF importance score does!
I have never heard of this approach, but it seems somehow reasonable. But, "experiment" should rather be a random factor, than a fixed one. These two experiments are just two randomly drawn from the population of possible experiments with this design. They are in no way fixed.
+1 for a really good discussion.
Need clarification, by 70 attributes do you mean 70 variables? And can you actually give an example of what do you mean by 1-out-of-K coding? Did you observe the subject and make a new variable as soon as you spotted a new trait? And by "2-way interactions" do you mean association between each pair of attributes or do you mean the interaction between two independent variables? If it's the latter one your question is not clear because no dependent variable was mentioned. If you can supply an actual example, I believe it'd be easier for users here to help.
Interpolation? Maybe a Kalman filter can be used somehow.
Thank you for your reply. The blog you mentioned is really interesting study. I think I got the idea. Only concern is his cross entropy formula. It seems different with the one I found online. His using: cross.entropy <- function(target, predicted)
{
	predicted = pmax(1e-10, pmin(1-1e-10, predicted))
	- sum(target * log(predicted) + (1 - target) * log(1 - predicted))
}
This is a great answer. Very helpful and informative!
@ars: Yes. However A,B,C,D are not necessarily related.
That is inverse square root of Gamma distribution. See Bartlet (1964) am intrested in this topic too
It seems like you have a binary (failure/no failure) dependant variable? If so, you would rather need logistic regression to get the most out of your data.
Hi Ewan, welcome. Could you be a bit more precise about your question? What exactly do you want to test/show?
@levon9, you can get over-fitting that is just as serious when you choose the features, so feature selection does not guard against over-fitting.  Consider a logistic regression model used to predict the outcome of flipping a fair coin.  The potential inputs are the outcome of flipping a large number of other fair coins.  Some of these inputs will be positively correlated with the target, so the best all-subsets model will select inputs (even though they are useless) and you will get a model that appears to have skill, but in reality is no better than guessing.
Nothing: it has to be treated as nonsensical because it was derived from nonsensical input.
Could you maybe help me with interpreting the positive coefficient of the interaction effect MownDecline? Does it mean that with increasing mown during decline, R&D will increase or perhaps that the impact of R&D expenditures during decline (-0.852) "less" ?
It is a common misperception that $R^2$ (on an absolute basis) measures "goodness of fit." See [Is R^2 useful or dangerous?](http://stats.stackexchange.com/questions/13314/is-r2-useful-or-dangerous) for a discussion.  Another discussion closely related to your question has also appeared today at http://stats.stackexchange.com/questions/50425/what-is-the-relationship-between-r-squared-and-p-value-in-a-regression.
@Nestor - Arggh!  I was just a few characters short of finishing my reply to your earlier comment!  I wouldn't prove it; as with Fisher, it's completely informal.  The use of things like the 0.05 cutoff comes about by a happy coincidence that the p-value for many tests corresponds, numerically, to a test statistic in Neyman-Pearson hypothesis testing that can be compared directly to a significance level, but conceptually they are different.  If we wish to make a formal significance test, I agree, you can't, based on the p-value alone.
A bit shorter version: `M[sample.int(nrow(M)),]`
I quite like [this explanation](http://www.phdcomics.com/comics/archive.php?comicid=1489) from phdcomics. Of course, it is very basic.
Why? "Why" would matter here.
I once set up a scale: -2 Strongly Disagree, -1 Disagree, 0 Neutral, 1 Agree, 2 Strongly Agree.  That seemed more appealing to me since it was easier to grab statistics such as what percentage were negative/positive on a question.  It also creates a kind of internal consistency in which the sum is always positive if more people strongly agreed than strongly disagreed.
The "AIC" you have there looks like the small-sample corrected AIC (AICc). It's probably easier to see the relationship between the first statistic and the original, asymptotic version ($-2 log L + 2 N_p$ or $\\chi^2 + 2 N_p$ in cases where you can replace the first term that way). Divide both sides by $N_b - N_p$, and you can see the relationship clearly: AIC/($N_b - N_p$) is the reduced chi-square plus an additional term. I don't know if that helps you any.
I've edited your question to add a link to your previous question as further context.
Maybe you can precise your problem?
I think you are having confusion in only one part. If we pick two fruits from apple and orange, then there can be actually 4 combinations. You might wonder that 'Apple+Orange' and 'Orange+Apple' are the same things in 'combination-permutation' point of view. But if I assigned two numbers (say, randomly '2' and '5') along with selecting these combinations then are they the same? No they aren't. Then 'Apple+Orange' is actually 'Apple(2)+Orange(5)' and 'Orange+Apple' is actually 'Orange(2)+Apple(5)'. That's why 2^2 combinations are valid. Similarly 4^5 combinations for this problem are valid.
Since you have doubly nested data (students in schools in countries), I think you have to account for that in your model and use a 3 level multilevel model.
+1 for stating an unfortunate truth of our time.  "unreliable software should be the least of economist's concerns judging by the models and assumptions used and predictions made in this field."
Can you say more about your data? For example, if the groups are dependent (i.e. 1, 2, and 3 are all three different measurements observed on the same units), then you need to incorporate the covariances as well.
*Even today, I occasionally meet geneticists who ask me whether it is true that the great geneticist R. A. Fisher was also an important statistician*
-- Leonard Savage 
(Annals of Statistics, 1976 http://www.jstor.org/stable/2958221).
But I can also run the MIMIC model in two steps, right? The reason is that I dont want the loadings change with the structural model which in my case is quite a complex multivariate regression model (I have 3 factors, all are treated as dependent variables in the structural part). Btw, I dont think there is any limitation for the number of factors in MIMIC model.
Sorry--perhaps I made the question too general.  Yes this is a perfectly reasonable assumption.  In my actual problem I'm assuming that the variables are LogNormal, but I think I could adapt any technique you have in mind to some other distribution.
@Whuber: It can be assumed that $\\beta$ is far from $\\frac{1}{2}$ lets say $\\beta \\approx 0.1$. Case when n is big is simple, so I'm more interested in case where n is rather small: $10 \\le n \\le 20$
A common assumption when trying to estimate quantiles with binned data is to assume uniformity within bins. But when you know something about the way the data is likely to be distributed (as with incomes, which are right skew) assumptions that reflect that knowledge will tend to be better. Another alternative would be to assume that it's smooth, and then smooth the data (whether by KDE or some fitted distribution), redistribute points within bins according to the model [& possibly re-estimate (in somewhat EM-like fashion) the fit, &redistribute in bins again] then estimate quantiles from that.
For stata the command is just `estat vif` after you've run the model, as part of the many post estimation `estat` options (but seem like a subject knowledge question: there should be some underlying conceptual framework to assist in model building)
Don't worry @Michael, you'll get the hang of it soon! Some of us really appreaciate that you are now posting your answers with LaTeX code in it!
Sloppy writing: It should say "In general, **σ** is not known, but can be estimated from the data. This esti-
mate is known as the residual standard error". See also http://stats.stackexchange.com/questions/5135/interpretation-of-rs-lm-output
I'd HIGHLY recommend the `dlm` package. DLMs are not as easy to use as other solutions, but `dlm` makes it as easy as possible and has a nice vignette walking you through the process.
Is the distance between each adjacent pair equal? i.e. the jump from A to B is the same size as the jump from B to C? If so, you could just assign a numerical scale, so A=1, B=2, C=3, D=4 and then calculate the difference at each time stamp jump.
Hmm, the much-cited Benjamini-Hochberg JRSSB paper was published in 1995, so just *outside* the window I'm afraid! http://www.jstor.org/stable/2346101
Storey's paper that introduced $q$-values was 2002 though. http://dx.doi.org/10.1111/1467-9868.00346
I think you need to be more specific about the problem.  What is the model?  What factors were significant and which ones weren't?
Can you clarify your second question? By "choosing", do you mean arriving at a final value for the coefficients, or do you mean choosing which variables to include in the final model?
Exactly! Thank you!
I'm not convinced that pooling and resampling is the correct approach. The bootstrap is to estimate the SE, so it seems reasonable to keep the group sizes the same (which pooling would not do). I think the pooling you suggest is more appropriate for a permutation test...
+1 for the additional references. My favorite derivation for the change of variables formula starts with the cdf, like in your other answer.
A t-test for the logarithms is neither the same as a t-test for the untransformed data nor a nonparametric test.  The t-test on the logs compares *geometric* means, not the (usual) arithmetic means.  This is one of several important considerations in deciding whether using the logarithms is acceptable (which it can be, depending on the application).
@user1966337: FYI, in optical spectroscopy, the intensities at distinct wavelengths often have distinct meaning, so you can treat them as variates for a (physically meaningful) bilinear model with few components, leading to a more restricted model of the data. Sometimes you have effects though that do not allow this and where FDA would be more appropriate.
If your data is different each time, why would you expect a number calculated from that data to be the same each time?
I though it must satisfy the triangle inequality d(A,B)<d(A,C)+d(C,B). I think some NN algorithms such as KdTree based on that assumption.
For the discrete-discrete case, this answer to a somewhat related question [here](http://stats.stackexchange.com/a/56326/805), on plotting ordered categorical data may help (though possibly without the boxes in your case). I'm really not sure how you think this 'bias' arises; it would affect the visual impression of the data points (leading to use expecting the line to go somewhere other than where it should) but not the actual data itself. Can you explain your reasoning here?
I will try some visual methods, but don't agree with the statement "the median so much further from the maximum than the minimum" as reason that the "super" distribution is not the result of "sub" Gaussian distributions. This could be the case if one of the "sub" distributions (admittedly, assuming they exist) with high density is close to the minimum.
[The Art of Data Augmentation](http://j.mp/hiZ5ED), by van Dyk and Meng may be a good start. I also quickly found some R examples on Bayesian statistics on [Peter M. Lee](http://j.mp/fslE35) and [Brian Neelon](http://j.mp/h1yJyQ)'s websites. But I guess you can find other good tutorials with Google, or a more experienced user will have some good recommendations to offer.
Good question. Small note: The OR of the baseline drug compared to itself is always 1,  that's why it is typically not reported.
And just to reiterate what @MichealMayer said, if your goal is to predict, you probably don't care about the signs of the coefficients. You just want to fit the data the best you can. Therefore, you probably shouldn't take anything out. But, again, there's really no hard rules for these things. If you're goal was to think about causality, things would be different.
Linear and RBF kernels are good reasonable choices to look at first, however the choice of kernel is very much data-dependent.  The Nelder Mead method can optimise both parameters simultaneously, which is generally the best option.  I rarely use SVMs these days, as Kernel Logistic Regression gives estimates of probability, which I find more useful than a binary decision for most real-world problems.  Alternatively I use kernel ridge regression (AKA LS-SVM).  The training procedure is much more straightforward, and you can get cross-validation almost for free, which I use for model selection.
It may help you to read my answer here: [Difference between generalized linear models & GLMM in SPSS](http://stats.stackexchange.com/questions/32419//32421#32421).
"Normalization" typically means changing values to fall within a given range such as $0$ to $1$ or $0$% to $100$%.  You describe *standardization,* which consists of a linear transformation making the mean equal to zero and the standard deviation equal to unity.
Have you tried to ask yourself why the ICC is so small (excluding user mistakes in performing the analysis)? What's your level-1 variable, what are your level-2 units? How many level-2 units do you have?
@Bill - Let's put it this way:  to ask for help with a technical question that's got you stuck is a fairer use of people's time than asking them to start from scratch and think up your research questions for you.  The latter is really not what this site is designed for, or what many people are going to be willing to give you.  Sorry if that sounds cold.
How do you know which are errors? E.g. if the question was "What would did you eat for breakfast?" and the answers recorded were "sausage, egg, bacon, sausage, tomato", are you simply not interested whether the person really ate two sausages or only one sausage?
Sorry, that should be the other way round, first sum should be over $y_i=1$ and second should be over $y_i=0$.  So you measure how close to zero the probabilities for observed $1$s (which makes more sense, as these are residuals).
@GrayR If there is exceptional movement from Friday to Monday this ( in the absence of a "cause series" ) could be considered a Pulse or a one-time outlier/inlier. Intervention Detection is based upon an in-model evaluation as compared to an out-of-model  study of the current model residuals. The advantage of the in-model approach is that the "unusual value to be detected" doesn't adversely impact the standard deviation that is being used to test significance. I would be glad to talk to you about this further.
Welcome to the site. With just the information you have given, it isn't possible to give good advice. I wrote [how to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) on my blog. It may help you improve the question.
So those are raw residuals. You would expect patterns in those residuals wouldn't you? The model you've fitted allows for them (variance increasing with mean). You'll also get bands with discrete data. You might like to look at the `?residuals.zeroinfl` and use the `resid()` function to get the Pearson residuals. I would probably now predict from the model to see what sorts of predictions you get and how they relate to the observed data. Also start to think about whether you are missing key terms or functions by plotting residuals versus the predictors etc and look for pattern.
Search this site. There should be plenty.
Context, details, or a real or hypothetical example would help focus this discussion greatly.
Great comments.
To make it less complicated, let's assume the characteristics of the subsegment are similar to the population.
So then how can I apply the distribution to the subsegment?
Here is a link to the google spreadsheet for this problem: http://j.mp/II5PgF
Does the finger contact area in your expression for $PT_i$ vary with each touch?
+1 Thanks Fabians, You are totally right, the problem is not that I don't have enough data. Note that my question is especially about the interdependence structure. Scatterplots, heatmaps and hexbin plot are good tools if they are used for the good purpose.  I think general additive model can also be very powerfull there is a wonderfull paper of Brillinger providing good hints on how to use GAM.
Could you be more specific when you mentioned 'bayesian' setting. Is this a exam question?
Thank you.
Could I not instead use trainDescr <- trainDescr[,-nearZeroVar(trainDescr)] to remove zero variance predictors?
Again, thanks and happy holidays.
For other EF distributions I don't think the IRWLS algorithm gives equivalent results to simply performing OLS on a transformed response variable (such as log(y) for Poisson data), so I can't give such a precise answer.
Making categories out of continuous variables is worse than magical.  Diabolical may be a better word.  If you want to maximize model complexity, increase bias, and increase variance all at the same time, dichotomization is for you.  [It maximizes complexity because the lost information due to categorization requires more variables to be added to the model to achieve the same $R^2$.]
Random Forest is quite insensitive to correlated covariables, check out this answer: http://stats.stackexchange.com/questions/52016/non-independence-of-ivs-in-a-random-forest-model/52036#52036 . It's also quite robust to noise (for example data labeled incorrectly in classification). It is also immune to monotonic transformations of the covariables, so you don't really have to play around with this like in linear regression.
Cool, thanks a lot. I'll look into it.
You can Google general advice on complex interactions (but it probably won't be of much use).  It would be best if you could post or link to the table of condition means.  Also, report the magnitude and size of the lower order interactions and main effects. My first approach would be to graph it.  You can probably do it in 4 panel graphs with 3 levels within each graph.
I never stated that it was different, and if I implied it, I didn't mean to.
I guess that in the second case you have a more sophisticated random effects structure than you expect, specifically some correlations among the random effects that explain variance. Those effect help to achieve more accurate estimations of the fixed effects, hence the differences. Can you post the random effects output of all the models (i.e., the upper part of summary) so that we can confirm that?
@Germaniawerks I believe my post mentions both of those already.
@chl I refrained from commenting on the option of computing a common slope because that implicitly assumes homoscedasticity in the combined year 1 + year 2 datasets.  Given that the slope appears to have changed substantially I would want to check for a change in residual variance, which seems easiest to do by conducting a separate regression for each year--which the OP already has done.  The advantage of modeling the combined datasets comes from the additional power it offers of evaluating serial correlation of the residuals.
The determinant is not invariant with respect to reparameterization of the problem.  Thus, you cannot expect there to be any universal measure of flatness: you need to translate the "flatness" into something *meaningful for the problem.*  Looking at simultaneous confidence regions (which are typically estimated with the Hessian) might be a good place to start.
What is meant by $\\mu \\sim N(\\mu ; 0, \\xi^2)$? Is $\\mu$ a random variable or the mean of the random variable? What are the _three_ quantities inside the parentheses? Normal distributions are defined by _two_ parameters, not three. And what is meant by the distribution of the density at a fixed point $x$?
OK, then things are not going to simplify much.  If you assume that $B$ is uncorrelated with $A,C$, then $cov(A+B,C+B)=cov(A,C)+V(B)$.  This is because $V(B)=Cov(B,B)$.
But if this were my dataset, I'd create composite scores for each of the scales, and then treat the scales as measured variables. (And that's still a decent sized model for this sample).
Additionally, I'm having some problems getting models to converge with the development version (model failing to converge, computed covariance matrix is not non positive-definite). Any help would be appreciated, thanks!
You could approach this by supposing that the random class has 100 students. Let $X$ be the number of boys in that class.

Let $A$ be the event that the class is from program A; let $B$ be the event that the class is from program B. An "equal number of classes in each of the two programs" means that with no other information, the random class is equally likely to be from either program, so $P(A)=P(B)=0.5$.

Use binomial theory to get $P(X=55 | A)$ and $P(X=55 | B)$. You can then use Bayes's rule to get $P(B | X=55)$.
Does the edit make it clearer? To get $Y$ we want to take all the times $X_i=1$ and then among those count how many times $Z_i=1$ as well.
@caya - that's the point. $x(1+ \\epsilon_X)\\approx x e^x$ for small  $\\epsilon_X$ but using exponentials is much more mathematically tractable. which is precisely why people work with exponentials in mathematical finance rather than percentage change ( even though the ordinary language of finance is % change just as you have). In particular $\\log x_i$ is not heteroscedastic (using the exponential model)
Thanks for those thoughts, especially related to workplace accidents. I could imagine GC being used as a tool, and am wondering if anyone knows of it being used.
What are the numbers on the vertical axes? Codes for categories? Actual "distances"? Counts of something? Differences between two such values (as in the lower plot)? (Those four would suggest four different kinds of analysis.)
One way is to note that $Q$ does not change when rows $2$ and $3$ and columns $2$ and $3$ are simultaneously permuted.  However, that permutation interchanges $p_{21}(t)$ and $p_{31}(t).$
I can't really see the picture you posted, so I don't know.  Go back to the cat example.  If you only see cats that are wet when there are malevolent children around, you have no idea if they are unhappy because they are wet or unhappy because there are malevolent children around.  Are there variables that are either jointly or singly correlated with `LAW_SR`?  (If not, you're done:  null result.)  If so, you can't separate the effect of `LAW_SR` from things that it is correlated with.  And yes, you have the correct technical definition of the F stat, but you're missing the intuition/purpose.