tttjjj/elt_bench huggingface dataset:
The start of this reply looks good but I got lost in the second half, perhaps due to some lapses of grammar and the long sentence.  Could you clarify this part?
My data is related to protein interaction data. We can talk about it more in detail (privately) if you are interested. I will look into MCL—thanks!
Hi @Stephane. If you are in France, there is a [French office](http://www.sas.com/offices/europe/france/index.html). You can also e-mail at support@sas.com.

There are offices [all over the world, in fact](http://www.sas.com/offices/intro.html)
+1, good question & good answers. Perhaps it's worth remembering that, `"all models are wrong, but some are useful"` -[George Box](http://stats.stackexchange.com/questions/726/famous-statistician-quotes/730#730).
@chl thanks for your comment. Meanwhile, there is just one question left: Is there a built in function in R or do I have to use the given relations.
Sorry I don't know any formal ways to deal with these issues.
Great, thanks for the idea Seb. Checking it out now.
No worries about asking self-study questions. You probably ought to add the self-study tag though and see the [associated wiki](http://stats.stackexchange.com/tags/self-study/info).
Say that I measure volatility from Jan.00 to the end of Dec.00, I then have one value for volatility. Assume that on Jan 1st the following year the volatility spurs and I then compute volatility from Jan.00 to Jan1st 01....I do so for every day until the end of 2001. I would like to know how the CDF evolves every time I compute the new volatility.
Nowhere really. "Perfectly correlated" abundant however. Correlation is one measure of dependency. That's why probably makes more sense in that context. It may make sense in other measures of dependency as well I suppose, but not the dependency itself.
But shouldn't we in theory use the population mean and standard deviation for centering/scaling? In practice, is it as simple as using the sample mean/SD or is there more to it?
@Onur: I don't know anything about those true random number generators that isn't written on those web pages, sorry. But I noticed that the report linked at random.org has a discussion whether it is at all appropriate to use the same tests for true random number generators or pseudo random number generators.
I agree with this approach, though the code can be simplified, but the steps are clear. Also, the 0.5 describe in the question led me think that the average was taken *across columns* while if you take the average down the who colum (aka mean of X8), the mean should be the fraction of (Number of severe use / total sample).
How many variables do you actually have? If you have dozens, then you might consider categorical clustering algorithms, or a latent class model. With only two variables, then Tamas' answer below is great.
From what the `PDF` page says it does, it returns the same kind of thing I do, but the methods it uses in its calculation in this case are likely to be somewhat more accurate (for such a purpose additional accuracy has little value, however). For some discussion of the probability/density distinction, see [here](http://stats.stackexchange.com/questions/48109/what-does-the-y-axis-in-a-kernel-density-plot-mean) and [here](http://stats.stackexchange.com/questions/4220/a-probability-distribution-value-exceeding-1-is-ok).
[This question](http://stats.stackexchange.com/q/37944/11849) might be of interest.
... the overall individual variance can be improved. My question concerns the upper bound for the overall individual variance which is possible by any guess for the single individual variances. It's just to complete my mathematical models...
Could you please suggest a way of doing so? I have tried `car::Anova()` with no success as I am unable to use the same model (`DV ~ IV1 * IV2 + Error(subject/(IV1 * IV2))`).
If you're going to test whether the difference of slopes is zero (as in @Charlie's reply, to which you seem to be following up), you need an accurate, valid estimate of standard errors.  It doesn't matter whether you bootstrap that estimate or otherwise.
Is Bayesian **probability theory** any different from regular probability theory?  I agree that Bayesian _decision theory_, Bayesian _inference_,  Bayesian _estimation theory_ etc. do differ significantly from the frequentist versions of these two topics, but probability theory is the same for both camps, no?
@mark999, I would think there are both advantages and disadvantages of one-at-a-time presentation. The advantage is simplicity -- using cat() to display an ambiguous time, and scan() to record your interpretation of that time is easy to implement. The disadvantage is that you may miss the big picture of lots of entries that you could correct en masse with a single line of regex code. You might have a think about what you're hoping to get: if you just want to solve this problem, do it by hand. If you want to learn more about R, try to code a solution.
Wow. That's awesome. Many thanks.
This is standard material, explained on [Wikipedia](http://en.wikipedia.org/wiki/Likelihood-ratio_test) and in most textbooks (*q.v.*). If you like, you can ignore the theory altogether and just accept that it suggests a particular statistic. The theory shows that such tests often have certain desirable properties, such as high power among well-defined classes of tests, indicating further evaluation of this test would be worthwhile. To use it well, you would want to study its power. Absent an efficient way to compute the integrals, the power can still readily be estimated using simulation.
Incidentally, you can do even neater things if the individuals have serial numbers. Check out the story of the "German tank problem": http://www.theguardian.com/world/2006/jul/20/secondworldwar.tvandradio
I don't know what you want to do. Principal component analysis (PCA) is a dimensionality reduction technique. It is not a classification technique. So, what are the "classes"? What is "accuracy" (overall or not). Proportion of Variance explained is just what it says ... proportion of variance explained. That's not accuracy, that's a measure of how much variance has been explained.
And yes, higher scores on attitude represent more positive attitudes
(1)	Both the variance and mean can be informative for our study (e.g. migration departure dates may be significantly later for one population AND the range in departure dates is more variable).
Do you need to measure the discrepancy between these two curves on an *absolute* basis or on a *relative* one?  In the latter case you might, for instance, be willing to disregard a shift in the x- or y-directions.  These curves might match closely indeed if the left one were shifted about 3 meters right.  What about the possibility of overlooking arbitrary Euclidean motions, such as rotations?
I'm baffled as to how the least common denominator of any of these numbers would be relevant here.
Because it is not pure. It's really dirty data.
There are at least 2 sorts of explanation: 1) the "file drawer problem" - studies with p < .05 get published, those above don't, so it's really a mixture of two distributions 2) People are manipulating things, possibly subconciously, to get p < .05
You could simply have an extremely large n. Or (multi)collinearity. Hard to say. Could you edit your question to tell us a little more about your dataset and what you are trying to achieve?
(This is rather marginally related to data analysis.)  Were you not able to obtain answers with a Web search? Googling "social gaming metric churn arpu" turns up good results at http://blog.kontagent.com/2009/02/ , http://www.quora.com/What-are-the-most-relevant-metrics-for-social-games-and-how-are-they-calculated/answers/143000 , http://www.wavedash.net/2010/04/the-secret-glossary-of-social-games-analytics/ , http://playthisthing.com/social-network-games-arpu-over-design , and many other sites.
It's just that I found this (ab)use of notation to be extremely confusing (which makes me sympathetic to the O.P.).  Parameters and their distributions are such different mathematical objects that this overloading looks awfully ambiguous.
I could see Tukey writing this the year he was born.
Have you considered an intermediate variant where the covariance matrix is taken to be $\\text{diag}(\\sigma_1^2, \\sigma_2^2)$ so that the variances of the two data components are the same but the data are components are treated as uncorrelated or independent instead of having nonzero covariance $b$?
Welcome to the site, @Karianjahi Njeri. This question seems to be *only* about how to get something done in R. As such, it would be off-topic for CV (see our [help page](http://stats.stackexchange.com/help)), but on-topic on [Stack Overflow](http://stackoverflow.com/), or maybe even more appropriate on the r-help listserve. If you have a substantive question about these tests, please edit to clarify; if not, flag your Q & we'll migrate it for you (*please don't cross-post, though*).
Yes, you added in town correctly (see other answers for your new error).  Are you sure that child isn't a random effect as well.  Or perhaps it's not even something you need to model.  My guess is that this is modelling age and education of children in towns.  In that case you leave child out.
Different authors might use different notations, but the choice of symbols is just a matter of taste (or a convention by field).
And the final distribution can be found by the Panjer algorithm if the factors are integers.
This is a good answer but I think it, especially the last sentence, almost seems to indicate that you only use GLMs or GLMMs for non-normal data which probably wasn't intended, since the ordinary Gaussian linear (mixed) models also fall under the GL(M)M category.
@zbicyclist, I don't claim that the calculation is accurate. I wanted to give a **reasonable doubt** (as we are in the realm of the law with fraud detection), that the draw is not guilty.
Sorry, I meant I'm looking for a way to contain vector samples from a bi-variate normal-gamma/normal-inverse-gamma distribution in R. I looked at those packages and they are focused mainly on the regular inverse gamma or the normal inverse gaussian.
Thats tough to digest. Could you please simplify it with a simple example  !!!!
+1. The math comes down to this: after diagonalizing the covariance matrix, it becomes evident that the total variance accounted for by any $k$-dimensional subspace cannot exceed the sum of the $k$ largest eigenvalues.  Choosing the first $k$ principal components obviously achieves this upper bound, QED.
Splines can be written as an equation that is moderately easy to look at.  A similar approach to what Peter wrote is to fit a model in which every continuous variable is linear, then one in which every continuous variable has $k$ knots in the spline function, with $k$ varying over a sensible range (for restricted cubic splines, $k=3,4,5,6$ usually does the job).  Then pick the model using AIC.
I haven't encountered such a tool.  However, I thought iPython actually has some capabilities of interfacing with R.  Additionally, have you tried the nifty "Compile Notebook" button in RStudio (requires the knitr package FYI)?  I tried iPython briefly, but I just couldn't get used to the notebook style.  What work do you do that lends itself so well to that?
@Fernando I had a quick look at the paper, the Tetris environment the authors consider was deliberately developed to be “adversarial” and they explicitly do *not* assume a fixed probability of getting a given piece. They also suggest that the piece you get depends on the state of the board (what you need to fill the gaps) and not directly on the record of previous pieces (i.e. no balancing out frequent pieces as you initially hypothesized).
To (6): Compare the current model by 'anova' with the model adding the squared and the cubed regressor in question. But better go with a GLM.
In such a case, much more information is needed. At least include in the question the reference of what you are reading.
It depends on the purpose: why do you need a single scale?
Sxx is the distance from the sample (x) to the mean (x bar). Because Sxx is written with Sigma we say "For every x, sum the results of...". Thus, Sxx takes one sample at a time, subtracts the mean, and squares the result.

Data set
x = 3, 4.5, 6 
The mean of x (x bar) = 4.5

y = 1, 4, 12
The mean of y (y bar) = 5.67

Sxx = SQR(3 - 4.5) + SQR(4.5 - 4.5) + SQR(6 - 4.5)
Syy = SQR(1 - 5.67) + SQR(4 - 5.67) + SQR(12 - 5.67)
Sxy = (3 - 4.5)*(1 - 5.67) + (4.5 - 4.5)*(4 - 5.67) +(6 - 4.5)*(12 - 5.67)

I've edited the answer. See above. It's should be easier to read.
@ConjugatePrior: That's Ok. Obviously my answer was a little unclear and your comments on the subject added more body to the answer :-)
I take this as a correct answer as addressed my main question. Hats off to you for mentioning the fact that it should not depend on the current state. As you mentioned, the rejection/acceptance ratio can become impractical as the number of dimensions gets large.
Searching some more, I came across the following links regarding uniform sampling from simplex and I am including them here for future reference:<br/>
http://en.wikipedia.org/wiki/User:Skinnerd/Simplex_Point_Picking
<br/>http://geomblog.blogspot.de/2005/10/sampling-from-simplex.html
Thank you Henry, I'll be studying your answer! 
And as you suggested, the error behind this is so wide that to use a firm number is quite a crime.
While asking this I was also curious if there was some kind of answer.. if you're correct, well done.
Welcome to the site, @Mike. I don't understand your question. What does it mean to "test... for a time series"? Are you wondering if there is autocorrelation? something else?
+1 nice rolando2 (as usual).  This is a productive answer to a question that makes productive answers difficult to give.
(+1) In other words, it seems apparent that the (*many!*) statisticians at Google are in no way involved in this project.
I am pretty sure there has been question like this one already. However, you may have a look at papers to be published at AOS http://www.imstat.org/aos/ this should give you ideas of "hot" topics ?
Both regression and correlation (and other methods) compare variables. If you have IVs and DVs (and the question is pretty clear about that) then some form of regression/ANOVA analysis is probably what is needed.
@whuber - you give me too much credit.  My suggestion was based on looking at a graph of the pdf of $t_8$, a graph of the logistic pdf $f(x)=e^{-x}(1+e^{-x})^{-2}$, and a graph of standard normal pdf.  $5$ degrees of freedom matches the excess kurtosis, and may well be better.
When the groups are separate as we suggested the samples are independent.  The t test would be unpaired becuase the subgroups need not be equal and there is not a natural way to pair the samples even if the sample sizes were equal.  I mentioned the Wilcoxon test because the normality assumption may not be valid and the Wilcoxon test doesn't require normality.
+1 excellent as always
Hint: Conditioned on $Y = y$, the event $\\{Y \\geq X, Y \\leq Z\\}$ has conditional probability $$P\\{Y \\geq X, Y \\leq Z \\mid Y = y\\} = P\\{X \\leq y, Z \\geq y \\mid Y = y\\} = \\Phi((y-\\mu_X)/\\sigma_x)(1 - \\Phi((y-\\mu_z)/\\sigma_Z)).$$  Multiply by the density of $Y$ and integrate.
I should add I've been discarding new runners from both training and prediction data because of inherent uncertainty but would appreciate any better methods than 'Ignore'
@Macro is exactly right here (you could also just multiply your 1-unit effect by 10 each time, but that would get tedious). Note however, that logistic regression does not afford an effect size as a constant % change. That will depend on the starting & ending %'s. Rather, LR allows you to calculate an *odds ratio* as a constant effect size. That is, the odds ratio will be the same from $X_1=1$ to $X_1=2$, as from $X_1=2$ to $X_1=3$, whereas the % change will *not*.
From my analysis, I have noticed that the discontinuities in the output space cause a number of problems. I have considered using joint locations rather than joint angles to overcome this problem. By dimensionality reduction, did you have a particular method in mind? Unlike image-based approaches, I don't see how the differences in subjects (other than their movement patterns) would effect the training of the model, given that I am using orientations of IMU sensors which are consistently placed and post-processed to be aligned between subjects.
Thanks, Gael, for the thoughts. I may be having difficulty describing my problem, which may be indicative of there not actually being a satisfactory solution to what I'm trying to demonstrate.
Is that TUKEY not TUCKEY?
@Andy W's comments on the main question are spot on, and my answer above is less general than the ones to previous questions.  But note that my answer was posted on math.SE when this question was asked on that site.
The methods described [here](http://stats.stackexchange.com/q/10975) give one approach out of the many possible.
@I like The analysis in terms of tickets is similar to the analysis in terms of individual people having equal chances.  It's a little more complicated because the chance of someone winning can vary from week to week.  But since you don't know how many tickets each person has, there is very little you can do about predicting who will win in the future unless you make additional assumptions (such as each person consistently buys about the same number of tickets each week).
@Glen_b Thanks. Corrected.
In your first bullet you say that correlation is a necessary condition for causation - that's not true. If there's a non-monotonic relationship between two variables then they can be uncorrelated - this does not preclude causation.
I don't think it does. If you are choosing between logit and probit, this is a very superficial choice. If you are choosing between log and logit for proportions near zero, it does not make much difference, either. If you are choosing between the identity, log and inverse for positive data, you may want to try the Box-Cox transformation instead. At any rate, GEEs try to protect you from whether you did or did not make the "perfect" decision in model specification.
Does $y$ vary smoothly with $x$? If so you could try fitting a model including a smoother (say a GAM) and then compute first derivatives of the fitted smoother and their confidence interval. If the derivative is signif increasing then signif decreasing you have an answer.
"difference in means between two independent samples, and divide that number by their pooled standard deviation." can you clarify how this is different from the test statistic for the t-test.
A general method is to use maximum likelihood to fit a candidate distribution. What you mean by superimposing a distribution to obtain the parameters isn't clear, but if you mean guessing parameter values until you get a good fit that's a lousy method. Kolmogorov-Smirnov isn't useful here. You need decent software that gives you inferential results, so you need to indicate your software of choice so that people using that can try to help you. Your histogram isn't clear, but if there gaps then no distribution is likely to fit well.
thanks. not sure I want to see the non-gentle introduction though :)
@Herman : In your question you said you have N = 5, T = 61 but in the comments you said you have N=2 and T=15. Am I missing something here?
+1 for this. People not realizing significance is a function of sample size drives me nuts.
I believe you can do multivariate probit models with this ftp://218.193.60.3/software/rsoft/mprobit/html/00Index.html `R` package but I've never tried it, so I can't be of more help.
Thanks! So as I understood naive Bayes is not a robust classifier, that's why the results were poor. Ok, I'll try to create ensemble of NB models.
@macro Well, no; but nothing really can be trusted with that sample size. At least, nothing that I know of.
For each treatment, I tested several groups. And also, I have used different populations of two locations, so I need to nest the populations into the location. Is it doable?
@KHKim  Good to see that you are finding the information>
thank you very much i will give this a try and post an update on how it went. i am trying to remain unbiased dispite the common (yet unspoken) view in the industry that there is rampent collusion. hopefully i can put this issue to rest one way or the other :)
This question is rather unclear. Are you asking how the observed nodes in a Bayesian network affect the probability of the unobserved nodes? How to model the values for your specific problem (i.e. what distributions to use)? Clarifying your question will greatly increase our ability to provide a relevant answer.
Each element of the matrices can have a certain value, which represents the amount of one of the variables - say, the intensity of one of the signals - at the point in the spatial domain. So the variables are distributed across the same spatial domain, which happens to be a square "field" (exactly translatable to a matrix). I'm very interested in how the two variables co-localise across the domain. Sorry for the confusion and thank you for your interest.
Please observe that (*caeteris paribus*) (1) the widths of confidence intervals tend to shrink with increasing sample size and can be made arbitrarily small and (2) your question presents no constraints on sample size at all. Thus, for instance, if group A were large, it could easily happen that the group B mean would not lie within the CI for A because that CI could be very narrow compared to the sampling variation of the mean of B.  That answers your first question. For the second: what do you mean by "associated"?
@ssdecontrol I disagree, although there is usually a gray area between the two sites with R questions. There were no actual _coding_ problems to solved here. No errors, no unexpected or incorrect results. The OP simply didn't understand the statistical meaning of the output.
@BB01, the answer depends heavily on what do your values represent, i.e. what does -1 mean for example. Since your data is discrete you should use [Spearman rank correlation coefficient](http://en.wikipedia.org/wiki/Spearman's_rank_correlation_coefficient) or its alternatives. For the graph, again the answer depends on details of your study.
Please read the [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) for the `self-study` tag. With that in mind, what have you done so far, and where do your difficulties lie?
Actually primarily I thought I will place 0 for question number 6 because for those respondents who do not have formal work teams, the number of members in the work team (as no team exist for them) can be taken to be 0. But I am not sure if it is statistically correct to do that.

Besides, if you tell me how can I perform hot-deck imputation in R or SPSS and give me link to key words about this process I will be grateful.
I do not understand what multinomial distributions have to do with this situation. Could you explain?
@J.M.: not really, chi-square is sensitive to binning and tail fluctuations complicate that.  I think even with the KS, they reweigh the statistic for extremal points, and there's some discussion of other tests.  @onestop: I assumed the other way, and on re-reading, you could be right.  I'm not really sure ..
Thank you, it seems I haven't explained my problem well enough, see edit.
Thank you for your clarification. The reason why I posted this question is due to the introduction I got for neural networks. I was able to understand that neural network is available to overcome the disadvantages of logistic regression. Here it goes

_It is difficult to include higher order terms in logistic regression as the count of the independent variables drastically increases. So, in order to overcome this limitation we use neural network. Here we can determine which higher order terms to use._

Please throw some light.