tttjjj/elt_bench huggingface dataset:
I don't have a simple, clean, mathematical distribution. I have real data, which is messy by nature. No distribution whatsoever will be a perfect fit, because you cannot handle the situation analytically anymore. And the outliers are actually my interest. :-)
(+1) @whuber's edit was helpful in clarifying this answer.
Surely you have some source for this rule of thumb. I believe that if you were to consult it, it would not say anything about dividing by 10 or 100. Doesn't it just say to subtract one AIC from the other? If not, exactly what does it say?
Another way to look at this is to look for particular package *authors*; certain authors are very good at following best practices and writing clear code, in which case you can study all of their materials.
I got it wrong, let me edit the answer.
It's not a list of problems to work through, but you may find this: [statistics-interview-questions](http://stats.stackexchange.com/questions/5465/) helpful
@Anony-Mousse, I understand what you are suggesting on distanced being meaningless if your variables are at different scales. All of my 30 variables will fall in two categories. For one set (15 variables) values will be btw 0-50 and for other set values (dollar amounts) would be btw 0-100K. Is there way I can normalize variables so that both set of variables are assessed on same scale? For analysis, all variables are equally important, so analysis should treat distances at same scale.
This sounds like a homework question. It has to do with standard deviation which is used to calculate confidence intervals. I would say more but in the interest of facilitating your learning I'll direct you to the wikipedia page on standard deviation instead. http://en.wikipedia.org/wiki/Standard_deviation I'd be happy to discuss this further but please read that, make an attempt at solving the problem, and then ask some intelligent questions and we can go from there.
This is exactly what I was looking for. "Thank you, kind raccoon!"
It is highly unlikely you would succeed with this as stated: a *prediction* concerns trees not in your sample and thereby is subject to the same amount of variation seen in the population, which is going to exceed 5%. But if you want to *estimate* the *mean* volume for specified dbh values, you can succeed (because the more data you have, the more certain you can be about the mean). The formula necessarily involves an estimate of the residual variation of the error in the regression of log volume against some function of dbh: do you have such information available?
Some code is there now. I didn't give code for 3) because it's a simpler version of 4); you should be able to get it by cutting out lines from that.
@mpiktas The issue I am concerned about, as a moderator, is to identiy the source of this quotation. (I am not challenging its correctness or questioning its meaning or asking for further material to read). Borrowing materials is acceptable on this site, but using them without attribution is not. The quotation is unusual in that it appears in multiple places on the Web, but (IMHO) does not show up in any authoritative places (only in gray literature) and never with attribution. I wonder what the *original* source of this quotation is?
You're right that CV is less active than math.SE, but I suspect you're also right that this is a better fit. If you don't get a satisfactory answer here after a period of time, you can flag your Q & ask the moderators to migrate it to m.SE & give it another try.
Thank you! Your answer clarify my doubts. Thank you once again.
This was cross-posted to StackOverflow.
+1 this is an outstanding answer.  It takes a question which I thought wasn't really answerable (I had agreed with @jbowman), and offers real insight.  I wish I could upvote it more than once.
Yes. You can use a neural network to construct features from an image, $f:\\mathbb{R}^{n\\times n}\\mapsto\\mathbb{R}^m$. Neural networks are very good at representational learning.
$\\sigma$ is standard deviation. Will clarify, thanks.
You need to reverse the first two arguments. When $n > 100$ in your case, R is telling you that the cumulative probability is (flat at) 1 in that region.
(Added a tag for compound-distribution.  There are about 50 questions that might benefit from that tag if it makes sense to keep it.)
(+1) For the record--and as a counterpoint to the assertions in the question--I know of no reliable reference that defines (or even characterizes) non-parametric methods as "lacking assumptions."  *All* statistical procedures make assumptions.  Most non-parametric procedures actually do make restrictive quantitative assumptions about underlying probability distributions, but those assumptions do not narrow down the possible states of affairs to a set that has the structure of a finite dimensional real manifold.
cov(X,X) is another name for var(X). cov(X,Y) = 0 since X and Y are assumed to be uncorrelated (hence covariance = 0).
First thanks. Second, considering the central limit theorem, can't we calculate the standard error of the mean? Third, I want to focus on the difference $\\Delta_{x}$, that's the reason why I consider to use the Wilcoxon signed rank test. And finally, I want to find what method is the best: A or B. And the best method will have the lowest mean.
@ktdrv Some things make sense either way up. Say you want to document the ability of a fisherman. You can count the number of fish caught per day or you can measure the interval between successive catches. Either measurement makes sense but they are non-linearly related to each other. They are scaled reciprocals of each other and so can be one-to-one converted to the other. The log of the interval and the log of the number per day are linearly related to each other and differ by a constant (negative) factor.
@404Dreamer_ML I added some example code.  There are other, better ways to do this (i.e. use a separate test set to compare your models after you've fit them, or better yet cross-validation), but this code will at least give you a framework to work from.
I agree that its a gross over simplification.  I'm trying to start simple and work up, but maybe the simple will not be useful.  I'm open to any papers on clustering.  I am pretty naive but like to read.  thanks.
but can I use the KS to assign a value or number to each intervals for their "degree" of clustering? I believe that they KS test is simply use for hypothetical testing. - I would like to do a cross-sectional comparison. And that value of clustering that I assign to each interval, I will use it in cross-sectional regression.
This is a duplicate of a prior question, see http://stats.stackexchange.com/q/8505/1036. If you look at the related questions on the right, there are a host of very similar ones as well (although unfortunately for this specific question there does not appear to be a real satisfactory answer).
See http://stats.stackexchange.com/questions/61906/covariate-vs-factors In short, there isn't any important theoretical distinction to understand but only a bit of fuzziness in the terminology.
800 per hour is 800/60 per minute and you want the probable number in 2 minutes.
@whuber Very interesting. Knowing the context of the data is always helpful to propose a good model.
Great we are talking about the same thing now :-) I want to use that plot to explain some biological circumstances.
You can also try Fuzzy Set Theory, which allows for partial membership to non exclusive classes.
I know that I can do that, but I was under impression that using log base 2 should also yield entropy value between 0 and 1. Can log base 2 be used only when I have two probabilities?
@whuber - that is how I often feel when I see R code.  I will get to it later on (hopefully this evening) but I will try to annotate and describe.
@Tal, for **binary** data, that is, for 2x2 contingency table, _phi_ degenerates into abs(_r_). This can be shown by the fact that their classic formulas can be reduced to the common one shown [here](http://stats.stackexchange.com/questions/26105/what-is-the-difference-between-verifying-how-strong-is-the-relationship-of-varia/26111#26111)
I think you misunderstood my question or may be I wasn't clear. What I want is a subset of the ORIGINAL features, not a subset of the transformed features.
+1 Good response. I think [Cytoscape](http://www.cytoscape.org/) might be helpful for the 2nd part of your response. (It was used in a recent paper, [Comorbidity: A network perspective](http://www.waldorp.nl/lourens/literature/Cramer2010.pdf), BBS (2010) 33:137, in a spirit similar to the one you described here.)
I have some questions. Isn't the data always discrete? We can depict numbers only up to a number of digits. Can't the marks in a school be normal distributed?
@Damian Okay that gives you a good excuse but it is bad terminology for them too.  Otherwise their Biometry book is not too bad.
Information about tails would come from the Taylor expansion about $\\infty,$ not $0$.
To be honest, I do not know if they are equals in linear relationship. It's sure that pearson on ranks is spearman. However, during transformation to ranks some things happens: pearson becomes more robust to outliers, the covariance is irremediably altered, pearson incorporates possibly non-independent noise (perhaps generated by confounders). In general I use pearson for linear-based inference, spearman to check if there is something else other than linearity, plus for ordinals (which makes sense only for spearman).
You need to learn more about $M$-estimation (or extremum estimation, as it is sometimes called in econometrics). The sandwich estimator for regression is just a special case of a very general delta-method formula, and if you understand the latter, you won't have any issues with the former. There is no intuition in that the sandwich estimator does not try to model heteroskedasticity or do anything specific about it; it's just a different variance estimator that works under a more general set of assumptions than the standard OLS estimator.
Hello, and welcome ! Do you know whether it works in the case when there are only three r.v. $U$, $V_1$, $V_2$ ?
Interesting information in the related thread. I agree that in the context of data analysis the main term should probably be included, but the question is if the "background" variable should be modeled differently than adding one extra term for each IV (plus the background variable itself). My concern is that in a prediction problem, usually adding many new terms increases the out of sample error. Also I am curious to learn if there are other methods to analyze data with multiple IVs relationships similar to this example.
I think you can do this just by passing "NA" instead of the actual values for the target variable to JAGS, and of course asking JAGS to return the target variable.  Your sample would be multivariate, with one component for each observation, but there would be no estimation going on because JAGS has no target variable data.
Just read Drew's answer. He is right on the information loss.Having said that, in the end, what is "accepted" as measure really depends on the field you work in. I would refrain from re-inventing the wheel. Look at what papers in your field are using and do that. For SLAM research in robotics, the det and trace are being used as optimality criteria for a long time with varying success and Google returns many papers to prove this.
I assume $X$ and $M$ are matrices? Perhaps a $k \\times N$ matrix?
Hi, i think G*Power 3 provides power analysis for McNemar test too. You can take a look at it [here](http://www.gpower.hhu.de/en.html).
Does state 4 communicates with other states ?
+1 for citing a Canadian. :)
The specifics of SPSS commands can come at the very end. Mainly you'll want to look up partial correlation, a.k.a. conducting an analysis while controlling for/adjusting for/holding constant another variable.  (It's possible your SPSS Tutorial files have something useful on that topic.)  Also, be aware that mediation is different from moderation, which refers to interaction effects.
"Well" is hard to define here.  I'd suggest fitting a model that has the right number of parameters for the effective sample size (allowing multiple terms for apriori strong variables not known to act linearly), putting in subject-matter-driven interaction terms, then plotting predicted values from that model.  Partial effect plots are very helpful.  The R rms package makes this easy.  See http://biostat.mc.vanderbilt.edu/wiki/pub/Main/RmS/rms.pdf
You should provide some data so that people can reproduce what you've done. Also, you could consider adding some other tags, as you've only got the R tag right now, and no statistical ones.
To the best of my knowledge Business Analytics in Pharmaceutical/Insurance/Financial sector use SAS software. If you want an independent resource on type of tool, you can refer to Rexler analytics survey. They conduct surveys on software tools used by variety of industry/users which was published the results in kdnuggets see below link.

http://www.kdnuggets.com/2013/10/rexer-analytics-2013-data-miner-survey-highlights.html
@cardinal - I meant that I didn't particularly care about the two scale parameters, and as you say, the "true" regressor $X_{i}$.
As hinted by others in the answers, your model specification doesn't make sense. I recommend that you study the following section of the R users' GLMM FAQ: http://glmm.wikidot.com/faq#toc27
...treatment groups, one can entertain increasing the probability of the better performing arm to 50%:25%:25%. [This continuation of @StasK's comment was cut off in the conversion from reply format.]
@cardinal Why not make it an answer with a little elaboration?
Thanks, it's been very helpful.  I'll vote it up when I have enough rep.
+1 nicely worked up.
How is that a counter-argument? The situation $Z_i=UV_i$ is not general: it posits a particular structure on the $Z_i$. Greg Snow is right that copulas can effectively be used to construct marginal-logistic multivariate distributions. Moreover, paths through the space of copulas can be used to parameterize families of such distributions having correlations varying among the extremes that are mathematically possible (the Fréchet–Hoeffding bounds). The only aspect still up in the air is whether parameters exist that are a "simple function of a covariance matrix."
This is a perfectly good question, but because it's only about how to get `R` to do something, & not about any statistical issues (possibly w/ R), it belongs on Stack Overflow, not here.
In R you can get the same results using the more user friendly `power.t.test`, that gives you flexibility on what you keep fixed. E.g. the example above becomes `for (k in 0:11){cat(sprintf("%f %f\\n", k/2,  power.t.test(n=4, delta=k/2, sd=1, type="paired")$power))}`
@Suncoolsu, statement of work requires both
@Amol Pande: It'd be nice if you provide a reproducible example. I guess what you are talking about is the difference between standardized and unstandardized coefficients. The only way I know to get standardized coefficients is to use 'path.diagram' with 'standardize=TRUE'.
@Horst If $n$ is specified to be small, consistency (a property in the limit) isn't really a consideration.
You have uncovered a big problem with cross-validation: the choice of the fold sizes.  Use the bootstrap instead, which requires only the choice of the number of resamples (I suggest about 500 here).  You didn' explain why you need variable selection.  That is causing much of your problem.  Variable selection is arbitrary and does not usually help with overfitting.  For your sample size you badly need a penalized MLE method.
An when you consider the transition matrix the $a_{i,j}$ are the probabilities - not the values of $X$. The values are ${0,1}$ otherwise you would have a bigger matrix. The dimension is given by the number of states.
Any estimator of the form,

$$\\tilde \\theta = \\bar X + o(n)$$
like for example,


$$\\tilde \\theta = \\bar X + \\frac an$$

with $a$ a finite fixed number,

is a biased but consistent estimator. The question is why do you want if for?
The concern is that perhaps almost all of the numbers might lie in a few bins, such as from 0 through 999. Your solution will then destroy almost all information about the distribution. It would be helpful to be able to *adapt* the binning to the data distribution in order to reveal as much as possible about the distribution.
How many time points do you have? Are they evenly spaced?

The two main sorts of approaches are mixed model (aka HLM, multilevel models etc) and time series.
Many good real-world examples, with some of the linked projects providing the data sets (but most don't, unfortunately): http://infosthetics.com/
Ah, that makes a lot of sense. Especially now that I've gotten home and pulled out my old DeGroot & Schervish textbook, which has a nice description of what these sign tests are actually doing. Fantastic. And thanks for the general thoughts on hypothesis testing :)  I like the perspective.

One minor note:  the portion you note as misleading is actually from R, from the documentation for wilcox.test.  This is a little disturbing to me...
@mzalikhan, it looks very much like your working dataset is in split-file state. Undo it.
I think what is being said is that $N = 2n+1$ is an odd integer so that the median is just the $(n+1)$-th ordered sample value.  The efficiency is expressed as $4n/(\\pi (2n+1))$ in the reference provided and is always smaller than $1$, instead of diverging with $N$.  The $N$ is in the denominator, not the numerator, and the efficiency tends to $2/\\pi$ as $N$ (or $n$) becomes large, as love-stats (and the link provided) says.
@Kwak: A continuous mapping from R^n to R cannot induce a total order.  Technically, I was wrong, because any bijection from R^n to R will "order" R^n via the order on R, but this order will not be compatible with any of the metric structure on R^n (which is an essential part of the concept of a "median"), and that was the spirit of my comment.
I suspected floating point precision was the issue.  I was kind of hoping there would be a regression function that can deal with such issues - e.g. shifting the data if necessary.  Is there?
Univariate analysis is not usually helpful and is often misleading.
+1 for answer, (and suggestion in comment that something odd is happening in dataset) although I'd disagree with the opening of your comment "If a variables p-value is not small, the one would typically not include that variable in the model." This is not necessarily so -- often one wants to report the magnitude of a relationship, even if it is not "significant" (and more to the point, if you were interested in modelling a relationship to start with, then a null result is still important to report.)
Actually, both areas are (or at least *should*) be unity: one area is tall and thin while the other is short and wide.  Have you considered plotting the *logarithms* of the densities?
"yes/no" can be treated as ordered.
Indeed, very true.  Still, struggling to figure out how to connect this back to finding why there is such a drop in power.
@pualambagher I revised my response to address your second question.
Borel sets and measurable functions aside, the expected value of an Indicator Function is $P(X \\in A)$, same with $Y$ which are equal. But how does this generalize to functions?
Great, thanks Greg.
Not your question, but see e.g. http://biostat.mc.vanderbilt.edu/twiki/pub/Main/TatsukiRcode/Poster3.pdf for argument why these plots are poor choices.
@gabgoh I would like to hear more about which aspect(s) of the answer are weak. As far as scaling goes, you're stuck: you have already used up that possibility in standardizing the elements of the sequence. If (hypothetically) some form of scaling would keep the third moments from going to zero, then you would contradict the CLT because the limiting distribution would not be Normal. There's a related issue with asymptotics of estimators. Often you *can* adjust an estimator to kill higher moments asymptotically (e.g., with bootstrapping): but this still cannot be done by scaling alone.
Check out [this paper](http://www.ine.pt/revstat/pdf/rs080204.pdf). It doesn't give the R code you want, but it has many examples of nearly exact confidence intervals.
There is a mixture of point and interval observations in my data. The function `intReg` from the package `intReg` accepts only interval observations (it is confirmed by the author of `intReg`). So I can not use it for all of my data. I could make some small adjustments +/- delta to transform point observations to interval observations, but I am not sure what consequences it could make.
possible duplicate of [How to easily determine the results distribution for multiple dice?](http://stats.stackexchange.com/questions/3614/how-to-easily-determine-the-results-distribution-for-multiple-dice)
Palace, your test merely establishes that some correlations are nonzero.  Pairwise correlations will be problematic for regression only when their magnitudes are close to 1, which is not at all the same as being different from 0!  Moreover, p-values are completely meaningless: you're not trying to make inferences about how the independent variables are correlated within a population; all you care about for regression is how the IVs *in your particular sample* happen to be correlated.  The effects of their mutual correlations are better assessed with diagnostics like the VIF.
@svadali it is is more usual to use ratio here since the distribution of ratio of chi square is tabulated (Fisher's F). However, the problematic part of the question (i.e. the dependency between $X$ and $Y$) is still there whatever you use. It is not straightforward to build up a test with two dependent chi square... I tryed to give an answer with a solution on that point (see below).
+1 @Michael Chernick It seems like not everyone is lazy ;).
Most full-featured stats software does this with a single command.  What software are you using?
+1 This is what I was going to say, as well! As an aside, I'd also add that recursive partitioning is sometimes used to identify where to best make cuts in continuously-valued features, to partition them into bins.
If you would taste R, it is highly probable that you will resign from MATLAB (as in my case).
This is a great illustration of your point: http://xkcd.com/605/
Also, you're saying that you're getting worse than random chance (50%) accuracy?  So, based on your high level description: if you're doing a two-tailed t-test then you might be picking up significance in the wrong direction.  A one-tailed t-test might give you the proper (high) p-values you are expecting.
Note the problem I mentioned in my earlier [answer to your other question](http://math.stackexchange.com/questions/262311/confidence-intervals-for-two-proportions/262413#262413) -- about testing only against one variable when there are other important variables (I pointed to the wikipedia article on Simpson's Paradox) - Fisher's exact test doesn't get around that.
This question seems like a community wiki question.
Good post.  I think the distinction that the OP should bear in mind when choosing between the two approaches is whether the "outcome" in one polygon will influence the outcome of its neighbors.  If so, go with gmacfarlane's approach.  If not, the one I propose is simpler.
@naught101, if you are willing to assume that the regression line is perfectly straight, then the width of the prediction interval can be considered as a measure of the strength of the prediction, (w/ wider intervals indicating weaker predictions), but it's still contingent on that assumption.
In Vandermonde's identity (as given at your link), what happens, for example, when $r>m$? (It's quite simple ... and it's the same in the hypergeometric. Take care not to confuse the two different uses of $n$ in the two formulas though.)
Yes, i agree, and I've deleted my own answer.  I was confused because the OP sounded like the original model was fit with lm(), which turns out not to have been the case.
Did you mean validation set error and TEST set error? Otherwise, I'm confused. I dont even know train set error, cause I use validation set error to choose model and them check selected model on test set.