tttjjj/elt_bench huggingface dataset:
It looks like an inverse cdf for a discrete distribution. Normally I'd expect data like that plotted the other way around (giving an empirical cdf).
@Plumenator As Anony-Mousse has said, every language has its own specific constructs the lexical analyzer of the corresponding compiler would look for. I cannot imagine that a rule system based on that can be outperformed by an approach not using this domain knowledge. [Lexical Analysis](http://en.wikipedia.org/wiki/Lexical_analysis)
This answer seems to be the most focused yet informative.
There is one here http://theoval.cmp.uea.ac.uk/matlab/#optim but if you already have the optimisation toolbox then fminsearch is also an implementation of the Nelder-Mead method IIRC.
@Douglas That suggests what might be a considerable simplification: by taking a target like 0.99, you can convert the ratio of quadratic forms appearing in $r^2_{xy}$ into a *difference* (by clearing the denominator) and then investigate the chance that the difference is positive.  So now you're looking at the distribution of a quadratic form under uniformly random permutations: that's an accessible problem.
This may be clear to some, but I can't understand the distinction you are making. Empirical CDFs are necessarily calculated from the data. What is the modified CDF? Can you give an example of what the result would look like? Or an accessible reference for the term "modified CDF"?
Actually, python, with a JIT compiler like psyco, is quite fast, for basic operations (See http://stackoverflow.com/questions/2133031/is-matlab-faster-than-python). Python with Numpy is also comparable to matlab (see http://www.scipy.org/PerformancePython). Matlab is intended for different things than R: numerical/matrix computation, rather than statistical work. Python is more general, and that's a big plus.
@Skrikant The [mclust](http://cran.r-project.org/web/packages/mclust/index.html) package should work here. To reproduce the example on Wikipedia (two-cluster solution), you will have to fix parameter `modelNames` (shape of covariance matrices) or `G` (No. clusters). Otherwise, the returned solution (i.e. minimizes BIC) is a three-cluster solution. Maybe you could add it to your original answer, for illustration purpose.
I've read the link and "Statistics without psychology", I was like: Wut? Then I opened the book: "Statistics Without Maths for Psychology"
@Glen_b Would you prefer that we migrate your answer into the duplicate thread?
Beautiful reference--and spot on relevant.  Thank you.
This is something that should be easy to start. Theoretically you can link R to every programming language and use  it to build a graphic interface
There are many R GUI's available for Windows, so I'm not following your point.
I have seen them effectively and conveniently handle data that is 50k rows and 50k cols of mixed type (categorical, integer, continuous).  For going from 50k dimensions to 30 they are great.  Better tools are then used to go from 30 to below.
Thank you very much for your answers, really appreciated it. Well, it took me a long time, but I managed to analyse this (bloody) dataset. I was too ambitious I think, wanting to model everything at the same time. So I divided in the job in several models, for each problem (average frequency difference, increase in frequency, etc.). Conclusion: sometimes its better to split the job!
Hello Jane, may I ask you which test you applied? Was it 'nsdiffs' from 'forecast' package?
In section 11.1 of the "An Introduction to R" guide that comes with the installation, it says x*z specifies "cross classification" while x/z specifies "nested classification".  I'm not really sure what that means.  It also describes it in terms of matrix classifiers instead of covariates.  I'm not sure if that would change the interpretation.
Well, my F1 statistic was the same with or without correction, so the F' statistic didn't change. I wasn't able to double-check the corresponding p-value though. By now, I've found out that I didn't even need a F2 analysis (nor a minF') in my case, since the items weren't nested in the treatment. But thanks for your input, it's appreciated!
Correlation between covariates is not a problem unless it is so high as to create a near multicollinearity.  In fact if you believe one slope is different from the other then the IV interacts with gender and there is necessarily a correlation. Also correlation between a binary variable and and a continuous one is different from correlation between two continuous variables. Certainly a pearson correlation cannot be defined.
Hint: Do the combinatorics of how many regressions this implies.
Machine learning has shown us that we can get very far without having any idea about what to look for.
Andrej, are you interested only in the case where the components are iid standard normal (as your figure depicts) or are you interested in the more general case with an arbitrary covariance matrix $\\Sigma$?
Possibly related: [Why does t statistic increase with the sample size?](http://stats.stackexchange.com/q/13676/1909).
`The variance-covariance matrix rescales the variables` sounds strange, - it is not the correlation matrix.
You might find this review article helpful: http://www.iro.umontreal.ca/~lecuyer/myftp/papers/handstat.pdf
@Glen_b I am trying to analyze a dataset of logaritmic scores which are the result from a research institute, I am using windows 7 x64 but I can dual boot into ubuntu if needed. The amount of memory needed to run it in R is 32gb just to put it into a distance matrix before I can run a cmdscale.  That is why I ask for help here, I was hoping someone would know a program that could handle this amount of data with relatively little memory (10gb or so) which I know is a long shot.
Can you try to clarify this question? Title seems to be divergent from the contents. (@MTibbits Unfortunately nothing better than asking the OP do it.)
@JC22 You *should* report a test statistic (whatever statistic you bootstrap); a bootstrap test based on a mean will be different from one based on a trimmed mean, for example.
are `x` and `y` related somehow? Otherwise this problem is only two simpler problems.
@GaëlLaurans: I agree. Experimentalists often need or prefer a language that matches how they talk about their experiments. Some of them talk about factors; they are usually not people who also do factor analysis! People who prefer independent variables as terminology have to explain that independent variables can be highly correlated with each other, perhaps one of the smaller problems we have in teaching.
Concerning the variable importance you could try **Random Forests**. The party package, among others, in R implements measures for variable importance. For a reference see e.g. Strobl C, Boulesteix AL, Zeileis A, Hothorn T. Bias in random forest variable importance measures: illustrations, sources and a solution. _BMC Bioinformatics_. 2007;8:25.
This is not an offer board.
Yes, you are being specific, but your specific question is covered by all the answers to the duplicate: you are testing the hypothesis that the intercept equals zero. That is well defined and specific.
First, rather than looking and quantiles I would suggest considering the problem parametrically. The data looks rather close to normal with the exception of a group of outliers at 0. First remove those 0s, then find the mean and standard deviation. For choosing the number of sd's away from the mean that are "significant": you could use that distribution to compute a p-value for each point (probability you would see that extreme or more if it were actually normal). Then it would become a problem of multiple hypothesis testing to find which points are actually extreme given how many you have
I want to endup with intuitive peaks (if we plot the data as histogram). I was thinking about allowing slight deviations from normality, as you said, in order to have some stop criterion - threshold.
please check the following questions: [Supervised learning with “rare” events](http://stats.stackexchange.com/questions/9398/supervised-learning-with-rare-events-when-rarity-is-due-to-the-large-number-o) and [Best way to handle unbalanced multiclass dataset with SVM](http://stats.stackexchange.com/questions/20948/best-way-to-handle-unbalanced-multiclass-dataset-with-svm). Does this help ? Frankly, your questions sounds rather similar ;).
One quick dirty fix would be to square centered time series and then test for changes in mean.
Another good reference on when/why/whynot to adjust for baseline status. http://www.ncbi.nlm.nih.gov/pubmed/15987729
Hello Momo, thanks for the comment. "not touch the data" means not to remove any data or time-step or replace with 0 or the mean, it would compromise the information about the specific-time-lag linear dependence. I partially recoded the matlab (link above) autocorrelation and partial autocorrelation functions to deal with NaNs: any data couples including NaNs is excluded from the computation. This is done for each lag. It worked for me. Any suggestion is well accepted.
Appreciate the help thanks!
If your data are not too big, you can insert them (or a subset) as plain text here; otherwise, please avoid providing link to XLS file (txt or csv is to be preferred).
Yes. I was thinking about scaling myself actually but I wasn't quite sure if one can do that. I mean theoretically. It would quite precisely do what I am aiming for though...
Thanks for the useful starting point into the contemporary literature - I wasn't aware of dasymetric methods that did more than reweight population densities, so I'll look into this eagerly.
no, but with some [running median](http://stats.stackexchange.com/a/30362/603) it should be possible
No, you will find really bad tail coverage in both tails of the confidence interval using that approach.  The average coverage might by good luck be OK but both tail error rates will be wrong.
Very Clear! Thanks.
How are you imagining there would be samples to analyze? Would there be many rounds of competitions with A, B, and C in each (either the same or different indivduals)? Or, would there be many A's, B's, and C's in one competition?
Perhaps you could link to somewhere on the web where the debate is so we can see what the various parties are proposing, and the context?  See my comment to @StephanKolassa.
The same issue occurs with linear regression, which is also not stepwise. In the case of linear regression, you can see it explicitly if you fit your regression via Gram-Schmidt orthogonalization.
From the help files, `lmrob` fits a regression model which is not the same as correlation.  I would expect `lm(A~B)` to be different from `lm(B~A)`.  In which package is `cor.mve` and `cor.mcd`?
Many thanks for the answer and the pointer to the theory in PB. I had looked at 5.4 but like you didn't see anything there. Will take a look at the earlier chapter.
@user695652 Since the linked question only considers $R^2$ tests, i.e. only the second bullet of my question I don't think it's a duplicate
It is important to update the parameters of the Beta distribution as the y's are processed, because all y's share the same $\\theta$.  The equation at the end of the question assumes a separate $\\theta$ for each $y$, which is a different model.
I had to use `library(lattice)` to get `xyplot` to work.
"(...) no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas." (Fisher, 1956, p. 41) See also [Why P=0.05?](http://www.jerrydallal.com/LHSP/p05.htm)
Use conditional logistic regression for "fixed effects" logit. http://stats.stackexchange.com/q/87896/24000. R's "survival" package has `clogit()`
@gung Any model that using classical laws of physics is deterministic. You seem to be very quick to dismiss an entire, somewhat important(!), field of study as 'weird'...
I use the two terms interchangeably (I admit that may be confusing). For the sake of keeping with the turing mahine analogy, I will stick to the single term "symbol". The "data" represents the set of all of the symbols printed out by the turing machine. The symbols are strings (words if you like) drawn from a finite vocabulary. One could argue that the series of symbols actually form sentences, and that there could be an "implied grammar" to the series of symbols emitted by the turing machine - but I digress ...
@cbeleites thx a lot...
1. Right now my goal is to show the means and just draw the attention to the ones that are different. 2. The audience is not statistically aware and showing them a table of means with an emphasis on the ones that are different seems to be the right approach to me.
the package `nlreg` (for heteroskedastic nonlinear regression) has a function (`nlreg.diag`) that can return a variety of such things including studentized residuals; I don't know much about it but it might be able to do ordinary nonlinear regression for you.
Thanks for your answer. I am "accepting" the other answer as I'm allowed to pick only one and that seemed to fit my current situation slightly better. But your answer was very helpful too.
Ok, thanks for the explanation. Yeah, recurrence is usually taken for granted. I guess for collapsed Gibbs estimation of basic LDA (with sensible prior parameters), recurrence is quite trivially true, because the state space is finite, and all transitions with non-zero probability are reversible with non-zero probability.
"Measures based on percentage errors have the disadvantage of being infinite or undefined if yi=0 for any i in the period of interest, and having extreme values when any yi is close to zero." I think this will be issue in my case, because lot of cases actuals could be ZERO. I am thinking calculate MAE and change the results number to "percentage". Does it make sense?
Can you say exactly which document contains this advice? (e.g. by giving an URL)
Thanks. I don't understand it fully, but might try harder :) Will this test give me significantly better results than the mentioned MWW U-Test? I have very small sample sizes, maybe only slightly more than 5 per set.
And actually the sapply, could be coded into a multi core framework...
@Nick: My request is more in the sense of if comparing fuzzy and hierarchical clusterings  is a common enough problem in the machine-learning community so that there are established datasets or even adhoc validation procedures for it. Even bibliographical references would be very helpful.
Autocorrelation may be of little importance here.  Interest explicitly focuses on the trends: how do the underlying growth curves tend to differ between the two populations?  Autocorrelation parameters are nuisance parameters, to be introduced and dealt with only insofar as they might help improve the estimation of those growth curves.  The first priority is to adopt a scientific model of the growth, represent that model with parameters that are interpretable and of interest, and estimate *them.*  Automatic application of time series techniques is unlikely to accomplish that.
If you have your own process, you could always roll your own package. Or just a reusable function in some file loaded at the beginning of your script.
@conjectures I know how you feel, having been in similar situations many times. But these days I usually tend to go with the sentiment of the Ancient Simian Proverb, and assume most such things associated with techniques in standard practice are likely to be reasonably straightforward. It's usually right. The trick is figuring out what I've missed or done wrong that makes it seem harder.
It is continuous unless you put OUT events beyond as though they occurred at 2T.  Then it would be a mixed discrete and continuous with point mass at 2T for P(E>=2T).  But you don't have to approach it that way.
Thank for you response! :D
I had a look at that post already, and it didnt really help - I think due to my lack of experience with this I need instructions (or a very clear tutorial) - do you know of anything like this online please?
By the way, here is a Coursera course taught be Professor Hinton that goes through neural networks in detail. https://www.coursera.org/course/neuralnets
@Angelorf Have you good results for instance on MNIST reconstruction ?
Note also that there are lots of physical measurements/instrument output/lab procedures where the error will "naturally" come as a relative error rather than an absolute error (or a mix of different absolute and relative error sources).
I agree that the two-part model is the appropriate one.
I'm looking for a 95% prediction interval of a prediction which is in log-odds space. Later I transform that to probability space. A 100% prediction interval would never be interesting for any procedure, right? For example, a 100% prediction interval for linear regression would include -Inf to Inf... At any rate, as you can see in my code, the prediction interval is calculated in log odds space, which is then transformed into probability space later. So I don't think my question is pointless.
Check out [AICc](http://en.wikipedia.org/wiki/Akaike_information_criterion#AICc), which is probably relevant.
You mean *apart* from misspecification of functional forms, link functions, error distributions, heteroskadastic form and unobserved variables? No... if you have all these right, then mathematics guarantee your alpha levels. (Of course, there are people out there who keep claiming that the null hypothesis does not hold in any question you'd want to examine, anyway. Pay no attention to those people.)
Why don't you ask your supervisor?
I am sorry but I am not sure what you mean..
In my experience, a huge part of the problem of machine learning, is literally setting up the correct problem to be solved/optimized (i.e. features, feature representation, selection, etc). I'd love to see a book purely dedicated to empirical feature selection and pre-processing with many real life illustrations (like kaggle). If anyone knows of one, pls. post.  There are several books dedicated to things like data cleaning/data imputation, but a dedicated practical text on feature selection is sorely needed.
Categorizing doesn't always damage the information used for the fit in a (generalized) linear model. For example, if the relationship between $x$ and $y$ is highly non-linear, then you can use the data much more effectively by categorizing $x$ and then estimating it's effect as you would any factor in regression. This might more adequately capture the non-linear aspects of the relationship (if sufficiently many roughly homogeneous categories are made) than calculating a linear fit to the original continuous measurement.
Aha, so basically this is impossible with finite sample.  It's like looking for a needle in the haystack when the needle isn't even there.  @whuber: the link to the question on testing for variance is very helpful.  Thanks!
Take the logarithm of **x** and then plot that.
If you start with an empty queue and no-one being served, you don't need to wait for it to become stable; the system renews every time the queue becomes empty and no-one is being served, so the long run is just an endless repetition of short runs that begin in the "empty" state.  The wonders of the Poisson distribution at work.
How accurate do you need it? Could you just simulate the walk and avoid fully-functional solutions?
His paper about it got into *Biostatistics*: http://dx.doi.org/10.1093/biostatistics/kxp050
Great answer, thank you @usεr11852!
@ttnphns I want to measure the strength of the association. I was afraid that the warning may mean that a measure calculated for my sample can be very far off from the true measure for the whole population.
I think he mean to do a Generalized Linear Model, with a conditional Gaussian distribution, with a logit-link to the mean.  (At least I'm going to assume that's what he meant.)
In your opinion, does it seem reasonable (at least tentatively) to retain the two items regardless of their low correlation coefficients?
The first thing I would do is a lot of graphing. Then I'd look at why there is such high attrition. Then I'd probably go with some kind of multi-level model.
Bookmarked. I wish they offered PDFs, but you can't go wrong with NIST's documents.
I could also have another source of noise which means that I could have the true curve appearing outside the interval.
Do the $T_i$ have the same parameters or not?
Equivalence relation *on what set*?  There is no such thing as a "set of all random variables."
Can you explain what the two models are? It may well be that you can fit a model to all the data in such a way that you can explicitly test for a difference in the two samples.
On the surface, the model seems reasonable. But the right answer is to run a simulation with known "true" parameter values and see if your estimates are far from the truth. See the [Cook-Gelman-Rubin](http://stat.columbia.edu/~cook/Cook_Software_Validation.pdf) framework for doing this.
The 50% was just an assumption, the same 50% assumption mentioned earlier in your own question. When you assume that and then combine *independent* parts (beware - frequently that independence assumption is made when it is not appropriate to do so), then you should get the answer in your question.
Due to the time series structure of the data, such kind of analysis is usually quite a bit trickier than just selecting some measure of association. If it where cross-sectional data, you could e.g. look at Spearman's rank correlation.
@Procrastinator, John Salvatier is looking for problems where the unnormalized target density can be computed for a given state, but has a shape that makes MCMC convergence difficult. ABC is for problems so difficult that the unnormalized target density can't be computed for a given state, making standard MCMC impossible.
Isn't [SPSS TwoStep](http://www.spss.ch/upload/1122644952_The%20SPSS%20TwoStep%20Cluster%20Component.pdf) designed for handling large datasets? (I provide a response to a related question [here](http://stats.stackexchange.com/a/10170/930).) Otherwise, would my response to [Can principal component analysis be applied to datasets containing a mix of continuous and categorical variables?](http://stats.stackexchange.com/q/5774/930) be of any help?
thanks I have gone through the link. I am not sure whether we have a method that gives us the significance of pooled estimate of effect-size. And moreover, meta-analysis has been invented to reach at a valid estimate i.e. truth. Why should we test it further?
Is there more than just this paper?
Why do you need a *correlation*? Why not just run a Mann-Whitney U-test? (Btw, Spearman's correlation is fine if 1 variable is dichotomous.)
@Jawad, draw your rand #'s from a standardized dist (i.e., mean=0) and then adjust after everything is done. Ie, don't add numbers like 4 or 5, add +4 or -2. In the code above, the dists of alg quality, problem easiness and residual variability are *all* centered on 0.  You can adjust the SD's of these dists to influence the resulting correlations, eg, the smaller the ratio of prob SD to resid SD, the lower the ave cor. Likewise, broadening the range of the algs will influence how reliably they are ordered by quality, & increasing nProbs smooths the results.
Note that the [betareg](http://cran.r-project.org/web/packages/betareg/index.html) package in R uses an alternative parameterization (with the mean, $\\mu=\\alpha/\\alpha+\\beta$, & the precision, $\\phi=\\alpha+\\beta$--and hence the variance is $\\mu(1-\\mu)/(1+\\phi)$) which obviates the need for these calculations.
This looks to be quite an odd experiment I have to say.  I've had a go at answering because I think there is useful information in my answer on using inverse of variance to weight.  But to give more meaningful advice we'd need to know exactly what you're trying to do - for example, why do you know that coefficients with a SE >=X are completely useless whereas those just below that threshold presumeably aren't.  But most importantly what is the point of this whole thing?