tttjjj/elt_bench huggingface dataset:
This seems akin to a longstanding joke in which one person says "Suppose that there are $s$ sheep" and somebody asks "But what if the number of sheep is different?". If you denote some probability $p$, that is your notation: no more. Clearly what its magnitude is numerically and whether you can estimate it accurately are key questions; in abstraction all we can say is that some probabilities are more difficult to estimate than others. That can be said independently of any speculation about "true randomness".
For (e) the numerator is noncentral chi-square distribution, and the denomenator is another noncentral chi-square distribution, what's the distribution of the quotient? Is it F distribution? or noncentral F-distribution?
If your data is non stationnary, you might want to use the LTS-filter: it's the same approach as LTS, but as a filter. You will find more detail on page 23 of [this](http://cran.r-project.org/web/packages/robfilter/robfilter.pdf) document.
@Ben, I have tried several different parameter settings, and the   positive precision do not improve a lot. They tend to locate in the range of [9%  13%]. The classifier were libsvm, using either linear kernel or RBF kernel. I even got some results where the negative ones get the highest score values.
Non-parametric $\\neq$ no parameters.
What software are you using to fit the predictive model? Is it .NET, or are you just implementing the model in .NET that you fit elsewhere? There are strategies pre-programmed in statistical software such as R, SAS, SPSS, etc. that would be hard to do from scratch w/o a good deal of expertise. Also, are you going to need to do extrapolation, or will all future predictions be w/i the interval spanned by your training data? NB extrapolation raises difficult issues; see here: [using-a-regression-model-to-make-prediction-when-to-stop](http://stats.stackexchange.com/questions/26914//26923#26923).
You're asking if you should "use" Suff. Stats to find MLE [over] that [distribution]. What do you mean by use? Which distribution are you dealing with? Why is finding the MLE directly difficult?
Answers depend very much on the structure in the data, both in terms of what the trend consists of and how the dependence over time comes in. There are a number of tests that might be suitable (or might not!); an example would be the Mann-Kendall test for trend.
Have you checked the vignette for the dlm package: http://cran.r-project.org/web/packages/dlm/vignettes/dlm.pdf
Welcome to the site Corey. Please use the markdown formatting available (see the buttons at the top of editor when inserting your answer), or take a look at the advanced markdown editing help, http://stats.stackexchange.com/editing-help. In particular, it is much easier to read code when it is formatted appropriately.
@Artium" If you want to contact me directly please do so and I will try and help. Essentially the values that represent unusual activity increase the error sum of squares thus reducing the probability of finding anything non-random.  Please refer to my email address and send me  your contact info and I will try and help. Essentially the characteristics of the data can suggest a starting model which can then be iteratively re-constructed which can then be used to detect errant observations which can then result in a model that may  be useful in separating signal and noise.
@mpiktas I definitely appreciate the value of this site, but the editorial board has the potential to give the articles a more widely accepted currency of validity (e.g. C.V. value, pardon the pun) than the posts here. Furthermore, the format is fundamentally different, and the quality of answers on this site vary widely.
I recommend the reference Cabeza G., U. A.. (2013). La Media de la Distribución de Cauchy. In the blog [Apoyo en Matemáticas](http://apoyoenmatematicas.blogspot.com/2012/12/4-la-media-de-la-distribucion-de-cauchy.html) about the mean of Cauchy distribution.
As far as I know, 'glm()' and binomial family expects a binary response. Your 'tot_users' seems to be continuous. Is there some trick behind this?
If you were working out the mean residual lifetime for the 2742 teachers still there at $t=4$, you might add up their remaining lifetimes and divide by 2742.  The survival function $S(t)$ is essentially the number of teachers remaining at time $t$, divided by the 3941 teachers at the start.  So using the survivor function, you need to adjust the results by dividing by a factor of $\\frac{2742}{3941} \\approx 0.6958$ which is $S(4)$ though shown at the end of the row that starts "3 [3,4)"
Would be good to include some reasoning on-site.
+1 and thanks, But :-) what is "tl;dr" for me or Ion, both not native English speakers?
Your dependent variable is a count. It couldn't be normal. It could be Poisson, or some related count distribution where the variance wasn't equal to the mean (eg, the negative binomial). You will probably have to simulate to determine power (see my answer [here](http://stats.stackexchange.com/a/35994/7290)), but 1st, you need to determine the size of the differences b/t the different methods that you want to be able to detect.
We welcome questions like this, @JDAWG92, but we treat them differently. Please tell us what you understand thus far, what you've tried & where you are stuck, & we'll try to provide hints to get you unstuck. Please edit your question to add this information. To better understand the process, you should read the [wiki](http://stats.stackexchange.com/tags/self-study/info) for the `[self-study]` tag.
Hi and welcome to the site, @Anirudh Vemula. [This book](http://goo.gl/Jp7T4) contains an algorithm for generating RVs from a hyperexponential distribution (page 107). [This site](http://www.iro.umontreal.ca/~vazquez/SimSpiders/GenerRV/Methods/composition.html) also seems to offer some guidance. [Here](http://www.csee.usf.edu/~christen/tools/genhyp2.c) is an algorithm implemented in C.
Heteroscedasticity, non-normality, autocorrelation, non stationarity. Pearson correlation test unbiasedness is known to be extremely susceptible to violations of its assumptions.
It makes sense to me and I'd say the answer to your title question is "yes".
If you review the derivation of your first result, it will show you how to obtain the second--and it has nothing to do with the shape of the distribution itself.
+6000 for data.table. It really is so much faster than ddply, even for me on datasets smaller than 100k (I have one with just 20k rows). Must be something to do with the functions I am applying, but ddply will take minutes and data.table a few seconds.
It's just standard [change of variable](http://en.wikipedia.org/wiki/Probability_density_function#Dependent_variables_and_change_of_variables). Easiest way to see it is by beginning with $P(Y\\leq y)=P(1/X \\leq y) =...$ (taking great care with the case where $X$ can be zero). The case where $X>0$ is simple, perhaps try it first. Once you have the cdf, take the derivative to get the density. As whuber says, it's in pretty much any standard text that covers this stuff.
Thanks for the reference. Here's one from me: http://stats.stackexchange.com/faq#etiquette
Hi @BenBolker, thanks for your reply. I wanted to have a model for heteroscedasticity, but the codes I gave did not need that. The model would be

$$\\text{logDens}_{i(j_1, j_2)} = \\gamma_{000} + u_{0j_1} + v_{0j_2} + e_{i(j_1, j_2)},$$

where $u_{0j_1}$, $v_{0j_2}$, and $e_{i(j_1, j_2)}$ are independent normal, and are random effects (or residual terms) of, `sample` level, and `dilut` level, and within-level. Thanks again!
Presumably you also have the p-values :-).  But with only those and the coefficients, it's hard to imagine what else you might do besides a Bonferroni adjustment.  (I always make such an adjustment whenever reading any paper with multiple tests: it's a quick way to winnow out the results that are likely to be junk.)  Most people also provide summary statistics for the variables: you can use ranges or sds along with the coefficients to estimate how much effect each explanatory variable might have on the predictand.
Yes to both points.
From academic perspective, both cases are interesting. From practical one, second is more "fair". In production you will have fixed amount of raw data. If some method can extract more instances for training than another from the same raw data, it is fair advantage of that method.
You can also, as whuber mentions above, rely on the fact that $P_0+P_1+P_2=1$, especially if you know two of the three.
@andy - you can model "noise" - but it depends on what sort of noise you are modelling.  For example, one way to model noise is using a mixture model with lots of components, or some other "non-parametric" model.  The idea is to get a model with lots of parameters, so it can fit any data you give it.  you then set non-informative priors over these and you effectively introduce a penalty for each parameter (similar to BIC).
Accuracy of `0.802816901` (R) vs `0.745042493` (Python).
This exists only as a [formal power series](http://en.wikipedia.org/wiki/Log-normal_distribution#Characteristic_function_and_moment_generating_function) which has no closed-form expression.
`samples$grv <- log1p(samples[,resvar])` or `samples$grv <- log1p(samples[[resva]])`.
Excellent, thank you very much!
+1 for send an email and ask. In this particular case, it seems like the best approach. I am fairly sure they will give you that information. As to drawing a random sample: the urls are *most likely* generated through hashing, which means any potential url is equally likely. This suggests trying out random urls is indeed a valid approach.
One note. What is the effect of rotation in your 1st command, there being only 1 factor extracted?
Ethnicity is a fixed effect, not a random effect.
Certainly, whenever $m_x$ is unknown.
This is mostly irrelevant -- tree is not supposed to divide object into independent groups but just to make a good model. You are probably looking for *clustering*.
Should also point out that bias is present and potentially severe when too few dimensions are retained: e.g. trying to represent $n$ observations in three dimensions with $n$ observations in a single dimension.
thank you! Figured it out.
@AndyW there can be one or more treatments per site
Possible duplicate of a similar recent question that also didn't receive that much attention: http://stats.stackexchange.com/q/70783/442
@Macro This is a good point, but perhaps the best reply is, "of course they won't have the same distribution"! The distribution you want is the distribution *conditional* on the constraints. In general that will not be from the same family as the parent distribution. E.g., each element of a sample of size 4 with mean 0 and SD 1 drawn from a normal distribution is going to have nearly a *uniform* probability on [-1.5, 1.5], because the conditions place upper and lower bounds on the possible values.
I am try to understand how it's correlates with assumption that _test set_ has the same distribution as _train set_, so we can have relatively good model if train and test has the same distribution. But by over sampling I change the distribution, therefore the performance must be changed too. It's more philosophical question, but without doing over sampling I am not able successful recognize the target concept, which has a small probability.
That's exactly what I needed, thank you.
5 am thinking along the lines of Joel.  In ANOVA the overall F test indicates whether or not there is any effect.  If that is all you want to do there is no need to do multiplicity adjustment.  But if the F test rejects the null hypothesis of no effect the investigator wants to know which variables are contributing to the effect.  Statistical packages provide the option to test contrast in that situation.  Multiplicity adjustment is necessary and the adjustment will tend to be larger as the number of contrasts tested is increased.
No, it's not about t-test, these are just the variants of ANOVA. The difference between them the same as the difference between paired and non-paired t-tests. For example, if $X_{ij}$ is the blood pressure of the $i$th human participant under $j$th condition, then your samples are linked and you use within-subject ANOVA; if all the groups consist of different people, you use between-subjects ANOVA.
Welcome to CV! Thanks for the paper: it's a clear exposition of how to interpret logits. But I couldn't understand your answer at all: perhaps you could explain a little more fully or add an example to illustrate it ...
Yes a known distribution is of interest. I just used KDE for other reason (to fit a smoothed curve to all data) as I added in the question text.
@MichaelChernick I completely agree that a Poisson model is a terrible model for this situation. The question asked us to make that assumption though, so that's what I presented. I don't think the answer sheds any light on the real world question of who should be benched when.
I discussed this in my 2002 IJF paper.
+1 It's a good question. Note that it is pertinent even to the case of "continuous" dependent variables: in many situations it is not the case that numerical differences have the same meaning or interpretation regardless of the levels of the original values.
that's the hint I needed!
Are the various lines meant to be non-parallel or is that just the inherent imperfection of photoshop?
You could try "aggregating" the data set, so that you have 4 columns [X1 value] [X2 value] [outcome] [count] where [count] is the number of times the specific triple occurs in those couple of million rows.
Just a note regarding the submission of high-resolution figures for publication: I believe that charts should always be submitted as [vector graphics](http://en.wikipedia.org/wiki/Vector_graphics), not as [raster graphics](http://en.wikipedia.org/wiki/Raster_graphics), so questions of resolution are not relevant. Publishing fuzzy plots may cause readers to cast doubt on otherwise outstanding research. I sometimes wonder if some researchers simply crop screenshots of their Excel workspace.
Given that no one else has come up with an alternative I'll mark this correct. However Patrick it's entirely possible this is an (misapplied) $\\omega^2$. The $\\eta^2$ better suits a typo I think (upside down $\\mu$?).
@IrishStat, Not sure if I understand your comment.  In case my writing was unclear, I wasn't suggesting ARIMA, though that's something one could consider as well.  I was suggesting something much simpler: keep track of the average of all observations so far, and the standard deviation.  Each time you see a new observation, you can update the average and standard deviation (as long as you have kept track of the number of observations seen so far) with standard methods.  It may be simplistic, but I don't see why this would be inane.
This question appears to be off-topic because it is about programming, yet it needs further clarification to be suitable for [SO].
Even if the book is not in your local library, many libraries offer the option to borrow from other libraries through your own local library at no or nominal cost.
Read what Charlie Geyer on this topic (http://users.stat.umn.edu/~geyer/mcmc/burn.html) and convince yourself that burn-in is largely unnecessary.
@Randel You are right, thanks
@SeamusAbshere I imagine you're right, but I don't know for sure. Now that I think about it, I'm not sure how it could work any other way.
thank you for the answer! You've said it exactly - "_to obtain an expected survival time given a set of covariates_". I'll have to go with the neural networks and SVMs in my study.
@cardinal Data are in a 2D grid (original case is in 3D). The are too noisy so I applied a smoother, as the figure shows. Both contour data/grid data are available.
Trimming asymmetrically is a known and defensible tactic.
That's sad. I've updated the links to refer to wayback machine copies.
Ah, sorry. I misstyped. I have now changed it above.
Please explain, in more detail, what you mean by "variance ... due to" a variable. You do not seem to be concerned with random variables (for which there is a standard meaning of the term "variance") but rather with pairs of numbers. There is, however, no general algebraic relationship between $|N_1p_1-N_2p_2|$, $|N_1-N_2|,$ and $|p_1-p_2|$ for arbitrary numbers.
Thanks for all the advice, do you have any specific statistics/surveys analysis books?
This might be PCA if $\\mu$ were linear
I am a bit fried to write a complete answer right now but for what's worth check the paper "Regression Shrinkage and Selection via the Lasso" by Tibshirani. You are practically looking at how many regression coefficients $\\beta$ you are using (the # of solid vertical lines). More $\\beta$'s enter your regression equation sequentially as your $|\\beta|/max(|\\beta|)$ increases.
You can use any weights you like, but you are wrong that the standard approach is "nonsensical." Even a huge ratio of weights will not entirely suppress the influence of the low-weighted cases, especially if they have unique or unusual combinations of independent variable values. If introducing more data creates "major parameter changes," then you have a more fundamental problem: either the model is a bad fit and/or it changes over time. Fiddling with weights won't address that problem; it will only give you false confidence in the results.
@AdamO - context added.  Let me know if you need more details.
From a purely logical standpoint, note that #1 depends on #2 being true.
Many thanks for the answer. It is easy to implement too.
Cross posted at https://math.stackexchange.com/questions/837907/maximum-likelihood-estimation-for-state-space-models-using-bfgs
3 months of data it means 90 days
Thanks for your comments. Right now, I am trying XMeans and DBScan with $L_1$ distance. Btw, I am fairly new to this, so it would be useful if you could point me to some sources for parametric modelling.
I might be wrong but by calculating Percents you've already ranked them against each other? Without percents you would have say in EECS 90 people participated and in IEOR 36. When left in raw numbers EECS wins, when converted to percents IEOR wins. Percents is a way of comparing groups with different N's already. Are you looking for a weighting system so that with lower N's it becomes harder to win?
Does nobody knows or is the question not interesting? I think the problem arises each time you want to use the wavelet tool for online analysis. Does nobody do online wavelet analysis?
@mjburns: In dominic999's example, the averaging is of the coefficients that result from fitting the same linear model to each of the multiple versions of the multiply-imputed dataset. I don't think it makes sense to average the datasets themselves, since you'd lose the (hopefully justified and realistic) variability that multiple imputation provides. The summary statistics for the pooled (averaged) results are much the same as for a regular linear model (at least in terms of the coefficients themselves) where you have to look at the Pr (>|t|) for the significance of each.
I mean mainly everyday life decisions. Lets say "Should I buy this","Should I go there", "Should I do this". In this case I suppose the model would be made from "experience" data, that is data coming from previous experience or knowledge from other sources. I think the problem here would be to quantify this and put it into a mathematical perspective. Because decision taking is often influenced by many factors which we do not want  or are aware maybe this mathematical approach would be helpful. My question was marked as duplicate but I think it is not answered in the other question page.
Thanks for the thorough and educational answer. We ended up writing a stored procedure to record each error to a database and return the number of errors in the last X (we settled on 5) minutes. If that number was over our threshold, Y, a warning email was sent. We adjusted the threshold by experimentation until we were happy with it.

If I were doing it over, I would incorporate your suggestion of counting time between errors for greater granularity.
Hi (from the author) of the question, My whole post does not seem to have posted - prob my user error.  So my question is please:  Am I using the correct test to detect differences from 0.00 as I am concerned that my two groups (participants) and control have totally different variances (i.e. control has no variance - as the measurements are all zero) whilst the participant group does have variance.  Thank you
(+1) Nice answer. The [arules vignette](http://cran.r-project.org/web/packages/arules/vignettes/arules.pdf) also have some good references about *lift*.
Have you read about how W is computed?  If you're going to use it, you might as well know! E.g., see http://www.sussex.ac.uk/Users/grahamh/RM1web/WilcoxonHandoout2011.pdf.  That should *immediately* answer all your questions.
@Chl Why did you change my answer to a comment??  The question was to cite some sources for nonparametric methods on dependent data.  The citation was an answer.  It is different from my first asnwer because he changed the question from being about time series to any type of dependent data.
Thanks .. I need a stronger result for proving the convergence. I took the ratio of two integrals - $\\frac{\\int_{\\epsilon}^{1} f(x|y) dx}{\\int_0^{\\epsilon} f(x|y) dx} \\rightarrow 0 $ as $ y \\rightarrow 0$. From this, it is easy to show the convergence. 
Thanks for the counter-example.
(e.g. that the nominal 0.05 significance level is close to the actual size of the test), - but in the R code above we just saw that the bootstrap doesn't preserve the test size, right?
I don't know the scilab scripting language--and I suspect I am in a very large majority of readers--but nevertheless I cannot see anything that looks like it's trying to compute the sum of a large number of independent variables, which is what the [CLT is about](http://stats.stackexchange.com/questions/3734). Could you describe in English (or mathematics) what you think this code is doing?
Density estimation seems a real stretch here. Nice mosaic plot. The histograms can show proportions rather than raw counts.
@Macro I could, but my question is really more theoretical than a specific application. i.e., Theoretically, if I have a multiple regressions with several t-stats, can I compare the predictive powers based on that.
Are you sure you mean the *smallest* loglikelihood?
If you can't post data can you at least put up the model output rather than just the ANOVA?
I think the key aspect of his question is not just generating graphs; it is about generating them automatically (programatically) as the underlying data changes. Some tools are easier than other when it comes to connecting them up programmatically.
Homework? If so, should be marked as such.
I mean one that fits better.  In this particular case I am fitting a linear mixed effects model and had been using AIC as my model fit criterion.  The residuals would also be sensitive to the transformation of the DV.  What is reasonable, to transform the residuals to a common scale and then sum their absolute values to compare models?  I plan to use a discovered set of transformations as inputs for a subsequent analysis.
Could be a poster child fo argumentative and subjective.  At the least, need to define 'valuable'.
2SLS standard errors are wrong.  I forget why or how, but you'd probably find something if you google "IV 2SLS standard errors".  Most software packages implement 2sls with the solve(t(z)%*%(x)%*%t(z)%*%y method
Could you please clarify whether you want to test equality of *eigenvalues* (as stated in the first sentence), *eigenvectors* (as in the penultimate paragraph), or the full *"eigendecomposition"* (as in the title)?  (The latter would be equivalent to testing equality of the correlation matrices.)
@RiaGeorge: See my updated answer.