tttjjj/elt_bench huggingface dataset:
@whuber Thanks. Pertinent quotations from Breiman, Leo. 1973. _Statistics: With a view towards applications._ Boston: Houghton Mifflin (seemingly a little known text, but pleasingly incisive and individual): "The main thing to learn about statistics is what is sensible and honest and possible." (p.1) "Doubt and suspicion, as well as technical knowledge, are indispensable tools in statistics." (p.18)
you get priors for the *parameters* of the priors
If you want to know if a sample comes from a given probability distribution, you can use a Kolmogorov-Smirnov test: ks.test( c(20,25,15,20,15), pnorm, mean=1, sd=3 )
Of possible interest: [link](http://www.jstor.org/stable/2282440?seq=1).
I accepted the answer; it seems indeed to work. So thanks! Only problem: I have a headache today and don't quite understand how you adjusted for the covariance terms. Thinking about those rhos just makes my head spin...So it would be great if you could add a few thoughts how you got from my updated rho matrix to your formula.
One sample, three observations or data points.
@Arun: The four 0's are messing things up. If you change one of them to 1, things look great; with the 4 0's, you get estimated log ratios that are $\\pm \\infty$.
@Antonin: Do you mean for the discrete case? Because for the continuous case (the one in the question), you should get an answer of $E[X_2] = \\delta + 3b/4+a/4$, rather than such a complicated expression. This is because $X_2 \\sim U(X_1+\\delta, b+\\delta) = \\delta + U(X_1,b)$, so $$E[X_2|X_1] = \\delta + (X_1 + b)/2 = \\delta + b/2 + X_1/2$$ and so $$E[X_2] = E[E[X_2|X_1]] = \\delta + b/2 + E[X_1]/2 = \\delta + b/2 + ((a+b)/2)/2 = \\delta + 3b/4 + a/4.$$ In fact your expression simplifies to the same thing, if you replace the $2a\\delta$ with $-2a\\delta$ (which is probably what it should be).
Might you consider a two-sample Anderson-Darling like statistic (rather than Kolmogorov-Smirnov, because it puts more emphasis on the tail, where your interest seems to lie) -- that is, to look at the difference in distribution between a blank image and the other image without specifying the distribution of either?
Thanks for the attempt. Note the edit I made above--I goofed including the weight matrix $V$ in the normal equations for the logistic regression--it's not needed since the logit is the canonical link. I agree that if the $\\beta_{-k}$ are the same in equations 1 and 2, then (2) implies (1).  However, it's not true that the solution to 2 gives the solution to 1, in general.  See the edit for a numerical counter example.
Good point. I uploaded the R-code snapshot (it is not in text form) that demonstrates how to calculate transfer entropy
Please describe the problem you're trying to solve. Your approach seems like it is probably a poor solution to whatever problem that is. Seat-of-the-pants *ad-hoc*-ery can sometimes work but is usually much less useful (and certainly less easily justified) than something better-designed to suit the problem.
Thanks for the feedback, and glad I was able to help (even if confirming what you already knew :) ).  Good luck.
Replace the sum in the sum rule with an integral and then you should be able to derive your result (hint: product rule first, then sum/integral rule).
No, what you provided was very not normal like. When you take the logs, many of the points lie on the 45-degree line. This suggests that the center of the distributions look normal-ish, but the patterns shown in the QQ-plots of the logs are consistent with a positive skew.
This (unanswered) question is also relevant - http://stats.stackexchange.com/questions/20613/how-to-analyze-longitudinal-count-data-accounting-for-temporal-autocorrelation?rq=1
thanks this seems like a great way forward!
This is great Dason, thanks so much. Rob's code was great but your example made it obvious to me what was happening -- thanks for taking the time to do that!
@Glen_b: somebody else has already done it. But let me ask: is it in any way frowned upon here, when the question is edited several times (perhaps "too often") after posting? I realize that it bumps it to the top of the active list every time, so I was wondering if it is considered importunate (and I think I can detect it in the tone of your remark).
Thanks for a fantastic answer!
Perhaps - happy for it to be ported across to Stack Overflow, which is where I post most of my R questions. The reason I posted it here was that the last R - IPF question I saw got moved from SO to CV: http://stats.stackexchange.com/questions/59115/iterative-proportional-fitting-in-r . Wherever it's appropriate, I'm happy.
Thanks, that's all I've wanted to hear. Now I can permalink to your answer! =)
+1, nice, clear & informative.  @StéphaneLaurent, if you think this answers the question you had, you should click the check mark next to it.
@Innuo oh yes, I forgot that. Thank you very much !
@Ga Thanks for the thought.  I'm concerned that the correlation of the truncated distributions could vary markedly from the actual correlation.  There's ample evidence of this in the environmetrics and chemometrics literature dating back to the mid-80s.
From the [top-voted thread related to normal distributions](http://stats.stackexchange.com/a/4301): "Everybody believes in the exponential law of errors [i.e., the Normal distribution]: the experimenters, because they think it can be proved by mathematics; and the mathematicians, because they believe it has been established by observation."
Is it possible for you to post your data, or say a little bit more about what the data are, what your situation is, what you've found, and what your goals are?
@BabakP This isn't a question about programming; just because one uses R, which also happens to be a programming language, doesn't make such question programming ones. This is clearly about the use of statistical software.
2). Also, one might want to simulate potential impact of a rare but extreme short-term event that causes high mortality in the population (i.e., hurricane, chemical spill, disease outbreak, etc). Of course one would have make some assumptions about how those events might change the population’s growth rate but could use the simulation results to inform wildlife managers as to potential effects by magnitude and duration of declines in population growth rates on the population’s trajectory. Those results could then be used to help inform planning and management decisions for the population.
Actually, I am trying to make a decision like this: for one gene, if distribution is shifted to the right, assume 1 (gene is mostly 'on') and for a new value evaluate `cdf(z-score-of-value)`. If `cdf <0.2` then 1, else 0. If skewed to the left (gene mostly 'off'), then assume 0. For a new value evaluate 1-cdf(z-score-of-value) and if 1-cdf < 0.2, then assign 0, else 1. The extra example, really clarifies it. I think that if I use the Pearson moment, it should be OK for my purposes. I needed a measure that relates "most of the probability" to the mean.
Given that the question is primarily about R code you might be better off asking at StackOverflow, our programming website
Errr.... not quite. Your answer is correct if $X$ is a continuous random variable but not if $X$ is a discrete random variable. If the CDF is defined as $F(x) = P\\{X \\leq h\\}$, then $$E[I(x)] = \\lim_{\\delta \\to 0} F(x+h-\\delta) - F(x-h)$$ where $\\delta > 0$. Some people write the limiting value as $F((x+h)^-)$ meaning the limit as we approach the point $(x+h)$ from the left. These people also say that $F(x) = F(x^+)$ and emphasize that the CDF might be discontinuous in which case the limit from the right is larger than the limit from the left.
Hi Samuel, I have edited your question. If you feel that I have changed it away from your intended question, you should edit it to clarify your actual question. Can you define your terms, please, and perhaps give additional context for your question, or point to something that has more context?
Thank you Ben.  I had trouble getting the first solution to work properly.  But switching to the dev version of lme4 and using the predict/REform=NA function does the trick without having to mess with the details of the model matrix.  There are a couple of features that don't appear to be implemented yet in the dev version that I was using in my code (in particular, I can't seem to extract the random effect postVars as easily), but I'd rather be able to use the predict function and wait until the new lme4 is complete for the other pieces.
Isn't the dispersion parameter fixed at 1 for a Negative Binomial, in the same way that it is 1 in a Poisson.
First of all, get your notation correct. You are asked to show that $E[(\\hat{\\theta}-\\theta)^2] < \\operatorname{var}(\\bar{X})$ and not $E[\\hat\\theta -\\theta]^2$ as you have it. Your expression would be taken to mean $\\left(E[\\hat\\theta -\\theta]\\right)^2$ the square of the mean, whereas what is wanted is the mean of the square. Then argue that the difference between the distributions of $\\bar{X}$ and $\\hat{\\theta}$ is that all the probability mass to the left of $\\theta_0$ (to the right of $\\theta_1$) has been replaced by a point mass at $\\theta_0$ (at $\\theta_1$) and this reduces the variance.
As someone who works on developing machine learning algorithms, the performance of simple systems is a source of constant irritation! ;o)
SDM-Spatial Durbin Model. The text I am working from is "Introduction to Spatial Econometrics" by LeSage and Pace, who do an excellent job of organizing and presenting theory developed largely by Luc Anselin.
It is *certain* that the actual time of a rating will deviate from a single predicted value. For your recommendation to have any value, then, you would need to explain how to predict a useful *interval* of times.
Can you post more info about the data? There is a plethora of binary classifiers you can try, so some details are crucial to show you the right direction.
Fair enough! Actually I was still hoping for the "standard procedure" recipe :)
Thank you Peter. I agree this does qualify as a data driven argument. I guess I had a more quantitative approach in mind.
You should give more subject-area context to your first question and title.
@Anony-Mousee thanks, that book is £133.98 on amazon!! http://www.amazon.co.uk/DICTIONARY-DISTANCES-ELENA-AUTHOR-HARDBACK/dp/B0092J4HE0/ref=sr_1_3?ie=UTF8&qid=1374584199&sr=8-3&keywords=Dictionary+of+Distances
"Incorrect" is too strong a word; we're talking about opinions about how good an approximation is. There have been many papers (4 or 5 at least) since that have said that it's overly strict and suggested less stringent approaches. *Your mileage may vary*. But at some point you will encounter cases where the chi-square approximation is really not good. My answer gives suggestions about what to do then.
Thanks @whuber.
**Closely related:** http://stats.stackexchange.com/questions/23200 (concerning design of an object-oriented system based on measurement types) and http://stats.stackexchange.com/questions/539 (a discussion on "treating categorical data as continuous").
You can always include an offset term for the total number of cells and keep the response as the count, if you have that information? In R, you would add this in the formula as `+ offset(log(totalCount))` or via the `offset` argument as `offset = log(totalCount)`. You need to specify it on the log scale as this is the scale it will be used in as part of the linear predictor.
Except that your mixture might not be unimodal if you're not careful, I suppose.  Anyway, good luck!
+1 DouglasZare for a well thought out answer. But I would like to point out that with interleague play now in the MLB teams in opposing leagues can meet during the season.
Seems like a duplicate of  http://stats.stackexchange.com/questions/25229/correlation-between-dichotomous-and-continuous-variable    Can you tell us if the answers to that one helps you?
Interesting topic of your thesis. I think I'll check out Homogeneous Texture Descriptors.
One approach would be to run a **logistic regression** but with a **one-vs-all** classification set-up.
What quantity are you trying to compute the root-mean-square *of*?
I think it is to make it easy to see when the likelihood ratio is less than some threshold (eg 1/20th max) at some null hypothesis (eg zero).
As per the terminology, the "support" of the distribution of a random variable, is standard at least in mathematical statistics, for the _range_ of the random variable, which in turn is the _domain_ of the probability functions (and from where we "draw" realizations of random variables).
I understand the 2) point of exploiting any prior knoweldge. However, can u elaborate how point 1) a lot of classes can be useful. More classes could complicate the problem rather than simplifying it, as far as I know.
@Nick Post edited to include data.
The `by()` function is good for looping over subsets of the data. The code here would look something like `by(Data, factor(Data$Module), do.one.plot())`, where `do.one.plot()` does all the plotting commands you want, for a given data frame.
+1 Wow, it has a name...
Though beware - a test of skewness and kurtosis will be all but powerless to detect quite substantial deviations from normality, such as in the illustration [here](http://stats.stackexchange.com/a/62320/805)
+1 putting the home team first looks like a sensible approach to me.
In order for this not to be closed as "[too localized](http://stats.stackexchange.com/faq#close)", it needs to be made less localized and more general. In particular, the fact that the deadline has passed for this year is not relevant. (It still might be closed as too *geographically* localized.)
There are similar questions on StackOverflow (http://stackoverflow.com/q/8961586/1414455) and Quora (http://qr.ae/80zE4). The short answer is that you don't.
I don't believe this is a problem. The grouping structure can remain in the model even if you use PC coordinates in place of `x`and `z`.
You probably mean $MSE(y_0)$, not of $x_0$. Also, not sure what the expectation $E_T$ stands for -- what is the probability space over which you are integrating? Is your $x$ random or fixed?
"Mixture" refers to weighted average of probability density functions. Is that what you mean, or simply that $X = Signal + noise$?
SST usually stands for total sum of squares, so it better be positive, I don't see where the square is in your formula for SST
I tried weekly averages. The results are much different than my original method. This method suggests that protein has the biggest effect on weight gain... and that really doesn't make any sense.
I posted another question on the same paper - http://stats.stackexchange.com/questions/9396/why-are-cumulative-residuals-from-regression-on-stock-and-index-returns-mean-reve. Please look at it if possible. Thanks!
Thanks for the information. I will figure out how to export the data to Google Refine and see if that works well.
Ah, I did not think about this. Is it something standard?
Why does a sign change disturbs you more than a mere change of magnitude?
why did you use PCA in the first place? what was your goal?
I'll look into it, thanks! There are a few other indices and I think some of them have been used in analysis over time, so there may be other datasets that don't exhibit the problem you've highlighted.
I'm confused about several aspects of this question.  First, presumably your "$F(\\theta;x)$" is an abuse of notation to mean $F(x;\\theta)$ (for otherwise you have merely switched the names of the variables).  But why can you call $F(x;\\theta)$ *qua* function of $\\theta$ a "distribution" when it is usually not normalized (and sometimes not even integrable)?  Second, what is meant by $F^{-1}(\\theta; x)$?  If you are fixing $x$ and viewing this solely as a function of $\\theta$, it's certainly not a "quantile function"!  Third, what's the matter with the usual terminology, "likelihood"?
Actually, I have data for 1991 and 2011, and every variable is just a difference between the first and the last year in the period. it's just one region. I also excluded highly correlated variables (e.g. population density and aging population) before multiple regression. I was hoping that canonical correlation could solve my problems, because it works with two sets of variables, and that's exactly what I have.
Thank's for the suggestion Dave.
How would you weight the feature once you've got that result? (I've gone through a few approaches, and now I'm circling back to KL)
As a matter of fact the MCC is a Pearson product-moment correlation coefficient. The connection between MCC and Chi Square is obvious, so why should I use McNamar's Test?
Hi jay, do you have an example of how this phyl.pca function is used in nested data? I don't quite get the parameters, i.e. tree,Y,method,mode.
`The loadings are the weight each observation gets in the principal components` What do you mean?
Thanks . With decision trees you get the rule based classification which gives you the probable indicators of the event. I already have indicators with me. I want a sas syntax so that I can use those classifiers for 4 categories and score my dataset.
I used the kmeans R function with a random initialization so the results were different for each clustering attemp. Now I am reading the paper suggested above by Suresh.
@user1552372 I don't understand this question really but maybe this is what you want. Have you thought of using the distribution of changes from timepoint to timepoint rather than the distribution of values?
The ratio of two normal distributions has a Cauchy distribution only if the two distributions are independent and both have zero mean, which seems unlikely here. Else it's a more complex [Gaussian ratio distribution](http://en.wikipedia.org/wiki/Ratio_distribution#Gaussian_ratio_distribution).
Thanks, rstan sample code is great. We also tried Theil-Sen, but it only works if we exclude all the data points where the explanatory variable is close to zero (presumably because there are so many of them, that with the median approach, the informative points further away from zero have almost no influence on the beta estimation).
Can you make this more concrete?  It'd help to see more about the actual scenario, or a simplified or contrived one if that's too complex/sensitive to publicize.
Statistical crowd is clicking randomly in SPSS until desired p-value appears...
@Scortchi Indeed. But the latent-variable interpretation provides intuition. Also, it offers a rationale for choosing a symmetric around zero distribution for the error term (given the constant term), (like the logistic or the normal, or even others).
So, those operations with columns were only a way to make easier for us to see that those vectors are linear dependent. But we can always think in a single step starting by the first matrix of my answer: there exist scalars, not all equal to 0, that multiply each vector (column of the matrix of covariates) that will make their sum equal to zero, it'll just be more difficult to find them directly. In the last matrix, however, it's easy to see that any scalar multiplying the first column and the 0 multiplying the other columns will yield the 0 vector.
@Gleb_b just the lm.fit line actually, the index in the predict line doesn't matter.
Although I do not have this text to refer to, the reference to a "long memory model fit" in the description of the bottom ACF plot strongly suggests that the upper ACF plot would correspond to the "short memory" model. Surely you can confirm (or refute) this expectation by consulting the text yourself. I am even more confident that by its fifth chapter such a book would have included an explanation of how to interpret ACF plots.  What does it say in that regard? For instance, what does it tell you about the meanings of the dashed blue lines in those plots?
Okay this sounds like what I was looking for. My issue is that I can't see from their definition that both characteristic function and fourier transform are the same thing:
fourier transform is defined as $\\int \\exp(-itx) \\cdot f(x) \\; dx$ [link](http://en.wikipedia.org/wiki/Fourier_transform#Other_conventions)
and the characteristic function $\\int \\exp(itx) \\cdot f(x) \\; dx$ [link](http://en.wikipedia.org/wiki/Characteristic_function_%28probability_theory%29#Definition)
As we can see the definition differ with a minus. Why can we than say that both are the same thing? Is there any proof?
Do you mean you want to highlight cluster structure (i.e., its characteristics features wrt. external subject-specific variables, for example) or to provide insight into how clusters differ between the two methods?
@All: The source code of tt1st can be found at http://www.crcpress.com/product/isbn/9781439808184. I did not see there why $psi=n|\\theta_1 \\theta_2|$. (Also I might unintentionally have rejected your edit as I wanted to look it first up in the book.)
Are you reading the updated question? The Result is **61Gb** NOT 128Gb you can do it yourself just write: $Quantile[Table[Fold[Plus,0,RandomVariate[PERTDistribution[.25,5,1],40000]],10000],0.964]$
I am not comfortable  in statistics. But a general understanding of mine suggests that you are correct. hence you need an applause.
Thanks for the suggestions.  I suspect that there is some underlying statistics that can help address this general problem, still...shouldn't we be able to construct confidence intervals for packs smoked per day by age?
To do Gibbs sampling you must be able to evaluate the conditional posterior distribution of each variable conditioned on the other variables. Are you able to do that? If so, then yes can perform another iteration after adding new data, although Gibbs sampling is not an ideal online inference scheme. You might add some information about your model, so we can offer more helpful suggestions.
Straight lines on a log-log plot of the CCDF are not enough to saythe data is powerlaw, let alone heavy tailed. See Aaron Clauset et al paper: http://arxiv.org/abs/0706.1062. For methods that do work, start with Sid Resnick's book: http://www.springer.com/mathematics/probability/book/978-0-387-24272-9
I have corrected the definition of your thresholding rule to include the adjustment constant of the mad and the Tukey multiple --at Gaussian data-- for the threshold itself.
You're testing different hypotheses. Broadly speaking, both approaches do test whether $X$ has an effect on $Y$ but the nature of that effect is very different in the two models so it's not a surprise that you're getting different results. I guess my question would be: what is the substantive reason for conducting both tests? What are you trying to learn/prove by doing this? It seems to me you should use whichever model makes more sense in the context of the application, but not both.
1) I'm not seeing the comp stat distinction. There's not much that statisticians do that doesn't require a computer. I suppose you mean computationally intensive procedures such as iterative solutions, etc? But then, these are also common in modern statistical work that is not data mining. 2) In my own (stats) work, I've been interested in model building for explanation and for prediction, depending on the problem-I wouldn't have considered that data mining. 3) I'm left with the conclusion that modern DM is a particular application of statistics, which I think is a fine conclusion.
We just discussed that on [a very recent question](http://stats.stackexchange.com/q/22258/7224)!
The site http://codereview.stackexchange.com/ is a better place for this question.
A multilevel/mixed-effects model should be able to deal with some empty cells.
Thanks, Ari. Yes, I think you are right about the problem of unobserved port pairs; I think what I'll do is just include an area-area route dummy variable and see if it is significant, then apply this to the data with missing port names.
Can you explain a little bit more how you intend to compute the different contingency tables that make up your ROC curve?.  It will be important to know how you intend to compute the ROC curve.
I'm not familiar with SAS, but what you're describing sounds like a histogram. There must be a built-in function to produce a histogram somewhere in SAS.