tttjjj/elt_bench huggingface dataset:
I keep thinking of things that can go wrong with my original idea, but I have a new one. Can you give code for exactly how you detected the separators?
Speaking from experience, it helps if fitness functions are smooth. Just minimizing the empirical risk will probably fail.
What is Y? And what is "double log"? 

If Y is a count (non-negative integer) then log Y is not a count. Count models often need a negative binomial regression, or some other variation on count regression.
Early on in my exploration of distribution tests I was told that formal tests such as the KS are "notoriously unreliable."  I have certainly found that to be the case, and experienced analysts tend to treat such tests warily as well.  E.g., with N in the hundreds, KS results will point "over-eagerly" toward non-normality.  Sometimes a distribution being tested for normality will be "normal enough" for one's purposes regardless of what the KS test shows. Typically one needs to use one's judgment in interpreting the KS in conjunction with plots such as the histogram, Q-Q, and/or P-P.
@chl: Yes, I'm looking some R codes to generate random data for experimental designs like CRD, RCBD, Latin and other design situations. Any idea. Thanks
One question that concerns atoms appears at http://stats.stackexchange.com/questions/73623.
... a change in functional form would eliminate any heteroscedasticity.  Heteroscedasticity, strictly, is a property not of data but of a model applied to a population.
@GregSnow  If you include the 0 intercept cases I think you can't compare the R square or adjusted R-square to the models with intercepts.  So if you are really not sure that the intercept is not 0 you will need a different criteria for "best" to be able to compare all the models and find the minimum or maximum whatever the case may be.
If you treated your variable as continuous, yes, that would be the interpretation. However, categorical variables with more than 2 categories aren't modeled with a single covariate, but generally $n-1$ variables.
@DJ_ It would probably differ software to software. I used SPSS to create this; I just drag acid and drop into the "column" attribute, and then SPSS would split the graphs into columns by acid level. But if you look closer, panel 1 is just a clustered boxplot/error plot **with cases where acid==1**, with some repetition you can create three different graphs and just insert them into the document, side by side. The only trick is to make sure the y-axis min and max are the same, or they will be difficult to compare.
The more elementary one I'd recommend is [Steyerberg (2009), *Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating*](http://www.clinicalpredictionmodels.org/), for which *RMS* "provided the main inspiration". (I'm not a medical statistician either, but I tell all & sundry to read these books.)
Florian> Developed by Benzecri and heavily used by Bourdieu.
Just an idea: What about adding a classifier to predict whether a given question is english or italian ? This should be easy, isn't it ? Afterwards one does learn a separate classifier for  the domain, one for each language. This way you can filter stopwords etc. C++ and furniture are sooo different, that I'd expect a good performance based on 1-grams (without stopwords ofc). Maybe a closer look which instances have been classified incorrectly might help ?
This distribution has a simple (and pretty) geometric interpretation. Indeed, although Student (1908) first derived this form of the PDF through an intelligent guess (supported by Monte-Carlo simulation), Fisher (c. 1920) first obtained it with a geometric argument. The essence is that $f$ describes the distribution of the ratio of the *height* of a (uniformly distributed point) on the $\\nu+1$-sphere and its *radius* (distance from the axis): in other words, the tangent of its latitude. One account of this is provided at http://www.evolvedmicrobe.com/Literature/GeometricTDistribution.pdf.
@Macro technically you gave a complete answer to the question. Since Andrew is probably not familiar with the Bayesian methodology or at least the terminology i think it is appropriate to elaborate a little.  Bayesians do inference based on treating unknown models parameters as having probabilities. The likelihood is a probability density for the data given a value for the parameter.  The likelihood can be used by frequentists to do inference about the parameter without making assumptions about the parameter.
If I calculate the normal cumulative distribution for ALL the genes the right-hand tail is obscene. This is because there are lots of genes with extremely low values, some even negative. If I remove the genes that have negative values in either replicate then it looks more normal but still has a wide tail. This is what I want to improve upon. While in general the genes with low expression values have larger %diff and higher p-values (>0.05), that isn't the case for all of them. Thus, I want to have the soundest exclusionary criteria.
https://www.dropbox.com/s/c9loeh91us8lbu7/data.png
Nonzero correlation between two random variables $X,Y$ just means that $E(XY) \\neq E(X)E(Y)$. If the two variables are positively correlated, I think the ratio you defined would be $>1$ so I'm not sure how that captures causation but not correlation.
Thanks a lot. From the glmm-FAQ: "Researchers often want to know if there is a simple (or at least implemented-in-R) way to get an analogue of R2 or another simple goodness-of-fit metric for LMMs or GLMMs." Haha, true. ^^
That will work, but you should use only m-1 variables (11 in this case).  Variables 1-11 will be all zeroes for December (which is how you will know it is December). Variable 1 will be 1 in January, otherwise 0. Variable 2 will be 1 in February, else 0.  Excel isn't ideal, but will work.
Stratify by a top predictor and within each stratum compute the inverse transformation (logit, etc.) of the cumulative proportion $Y \\geq y$ for all $y$ above $min(Y)$.  Look for parallelism.
You'll need to provide more information on your dataset to get helpful answers. In particular, what do these percentages represent? Why are you interested in multiple comparisons?
I second @chl's recommendation of Tinsley and Brown's book.  It is expensive, though; see if you can get it from a library (as i first did).
But the level of measure theory will not be so high. Since also finance students from non-math faculties read it. So it should be appropriate, if you teach yourself the necessary math.
A disadvantage of the Beer Advocate method is that it does not take account of variability. Nevertheless, I prefer this line of thinking to the lower condifence limit idea.
Thanks for the answer. By building a portfolio --as long as it is good enough-- I can work around going to uni? It is definitely encouraging to hear, I missed my chance when I was younger because I wanted to travel and now want to make up for lost time.
For what purpose? What do you mean with reasonable? The answer will depend on which model you use. It absolutely makes sense if you use a model sensitive to meaningless features (such as regression). Then you could potentially have variables without any information in your model which only add noise to your predictions. The question is then how you should go about to find a good subset of features which are meaningful. If you use a model which is *not* sensitive to meaningless variables (such as a random forest) then there is absolutely no point in removing any variables.
Well I'd be thinking about using logistic regression with performance as the response, & with subject, condition, stimulus, & their interactions as the predictors.
@whuber: I think you might be thinking of the *sample* version of Kendall's $\\tau$. As a population quantity, this is defined as $\\tau = \\mathbb P((X-X')(Y-Y') > 0) - \\mathbb P((X-X')(Y-Y')<0)$ where $(X,Y)$ and $(X',Y')$ are iid bivariate vectors with continuous distributions. From this definition, it is clear that if $X$ and $Y$ are independent, then $\\tau = 0$. The OP *seems* to be asking for a characterization of the reverse implication.
Well, I need to read deeply! As my first impression it looks promising.
Both hcluster and k-means are decades old. You may want to try something more recent, actually **designed for text**. I have no idea what $O(n^2)$ you are talking about: k-means is in $O(nki)$ with $ki \\ll n$, whereas common hclust implementations are $O(n^3)$.
+1 Plotting is for statistics what voting is for Chicago: something everybody does early and often. ;-)
@user995434, the only way I can imagine that is to place a Gaussian at each cluster center and find the conditional for a point to belong to it.
+1. Sorry for the late reply, but this was perfect for what I needed. Thanks ttnphns!
(-1) You don't really answer the question, do you? Instead of the 'shopping-list' approach, let's wait for OP clarification and provide him with adequate solution.
Boxplots typically contain quantile-based measures (medians, quartiles etc...). Multiple imputation doesn't provide rules to create one "combined dataset" but rather a "combined estimate" (for the median, quartiles etc.). So in order to draw a box-plot, you have to calculate the medians, quartiles etc. for every imputed dataset and take the average of these. The "combined" median, quartiles etc. can then be used to construct the box-plot you want. Sounds difficult, yes, but MI is not intended to make nice plots.
Sorry for ignoring your answer, Stéphane, I unfortunately didn't see it until now. How would you apply the Jeffreys prior to my problem and in which ways would it be preferrable to the Inverse Gamma prior?
(+1) Someone who is contributing to Python tags on both SO and cooking has some exotic tastes in food :-)  I hear it tastes like chicken.
I can follow how that could be done, but is that what is wanted? If delta is meaningful, then you would want to detect not "is the difference greater than delta?" but "is the difference at least delta?" Perhaps a 1 sided test?

Or maybe make delta the "maximum meaningless difference" that is, the largest difference no one would care about.
A defect of this method is that `x` is ordered, you would need to shuffle it with `sample(x, length(x))`. The value of `sum(binary)` could be generated directly as `s <- rbinom(1,n,p)` and of course the value of `sum(!binary)` is `n-s`.
Both correlations and t-tests can be used to both test statistical hypotheses and as measures of effect size. They measure/test different things.
May be you need to specify `enter()` option as well, as Stata has excluded 2/3 of your data right away (and ignored a few more). Is that what you intended? Also, judging by the syntax diagram, you can specify an `exp` in the `exit()` option, so something like `exit( time new_end_age >= 11000 | Staph == 1)` can be tried. (I have never worked with `st` data in Stata, I have to admit.)
@yannick I'm not sure what you mean; are you looking for the likelihood in the case of unknown $N$? When you put down the likelihood it's conditional on any unknown parameters, including $N$, so the above expression should work. But you have more fundamental problems in that the data (that is, the sample mean and variance) carry *no* information about the sample size. You need more information about $N$ coming in through either your prior on $N$, or from some other data source.
Actually I've done that at first. I read about correlation analysis, factor analysis, Markivian and Bayesian networks (quite off-topic), assosiative dependencies and deterministic analysis (Chesnokov's book) and few other topics and books. But I wasn't able to find any working free software. I don't need it in commercial way - only comparison. So I can't pay about $100 for every library like Extreeme optimizations lib (http://www.extremeoptimization.com/) and similar tools.
Either that or you could break it down by hour or day of the week or something to that effect.
It sounds to me that what you have are the *steps* of a process that you want to test whether it is a *time-homogeneous* simple random walk (possibly asymmetric). So, can the problem be reasonably codified as follows? You observe $\\xi_1,\\xi_2,\\ldots$ each in $\\{+1,-1\\}$ with $\\mathbb P(\\xi_i = +1) = p(i)$. You want to test the hypothesis that $p(\\cdot)$ is actually a constant function. Is this right? If so, do you know (i.e., can you assume) anything further about the properties of $p(\\cdot)$, e.g., a single change in level or that it is smooth, etc?
I found these, don't know if it helps: http://www-01.ibm.com/support/docview.wss?uid=swg21477535 and http://www-01.ibm.com/support/docview.wss?uid=swg21480170
@A.Donda, by valid I mean statistically valid. Relationship in the sense of correlation, as detailed above.
if there is no trend (and seasonality and all that) over the years, then wouldn't the average mining rate (averaged over the years) be a pretty good estimate of the mean mining rate, upon which you can calculate your C.I.?
Crosstabulation is very simple. `PROC FREQ` in `SAS` or `table` in `R`. You don't need to test a hypothesis, just show that the table shows good agreement. I don't understand what you are asking in your last sentence; the logistic regression would have only one coefficient.
I believe a couple of people deleted their comments, so it looks like I'm responding to you directly in my question edit. Anyway, to simplify my motivation, I find it a confusing thing for reasons outlined above. I'm not for one moment trying to fight it; I'm just trying to understand it. It seems to me a most natural (even if wrong) response to think probability = 0 -> impossible.
http://stats.stackexchange.com/questions/58973/interpretation-of-the-ols-regression-coefficient-of-a-proportion-dependent-and-i is a duplicate of this question (but has no answers to date).  http://stats.stackexchange.com/questions/41379/how-do-i-interpret-the-results-of-a-regression-which-involves-interaction-terms asks exactly the same question but does not stipulate that one of the variables is categorical.  Nevertheless, its answer is relevant.
Exactly, I wonder if there are any references about the "smallest" n (wrt. to a high number of variables), or more precisely if any cross-validation techniques (or resampling strategy like in RFs) remain valid in such an extreme case.
Saying that a graph can be represented as an adjacency matrix or list is different from what is meant by the data matrix representation in which rows are samples and columns are features. The question is related to the data representation of the problem, rather than, the underlying 'data structures'. From your answer, we may generally, understand some tasks that can be applied nicely over graphs but it did not point specifically the prediction task of the question. I appreciate the answer but it is based on some confusion with the question. You may suggest some editting if that was the case.
(+1) Good answer.
@TheChymera: To tell the truth, this idea never crossed my mind. I agree that it could be a curious investigation (if it does both: analyzes a lot of real data sets from different fields, and presents some extensive modeling), but I don't think I am going to have time for this... I might think about it though. If you get any further ideas about your spectra, let me know, I am curious what you will settle on. Hope our exchange was useful for you.
Please consider accepting answers to some of your previous questions. This can be done by clicking on the checkmark next to the answer that best responds to your question. See the [FAQ](http://stats.stackexchange.com/faq) for more info.
I agree with @NickCox that this is not as bad as you think. You can try robust regression (`rlm` in the `MASS` package), or a permutation test (`lmPerm` package), if you're worried. Switching the roles of sample size and complexity in your first plot would probably give more informative results ...
@MichaelChernick See my comment [here](http://stats.stackexchange.com/q/27622/930) to understand why I converted your reply as a comment.
Nick gives a good answer with some good comments below. Among the first articles addressing the biases the result from aggregation is Gehlke, C. E. and Biehl, K. (1934). Certain Effects of Grouping Upon the Size of the Correlation Coefficient in Census Tract Material. *Journal of the American Statistical Association*, 29:169–170, which is an easy read, with easy to reproduce simulations using today's computers. Aggregation should make one think of [cross-level fallacies](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1508189/).
I gave it an uptick, but there is no visible check mark for acceptance. I think there is something wrong with my login. working on it.
"*I will pass my next exam in statistics*" isn't a null hypothesis, it's a prediction. http://en.wikipedia.org/wiki/Null_hypothesis
#4. Seems like a silly idea if you have too many variables...  This turns each continuous variable into a several categorical ones, which adds a lot more variables to your dataset.  Try a simple logistic regression using `glm` and see how long it takes to fit.  Also try taking a random sample of your dataset (maybe 10k rows) to test your modeling workflow.
This question appears to be off-topic because it is purely about programming.
@B_Miner What I had in mind is we know the average balance for periods 1-36 or reasonable expectations of these average balances (x1,x2,..x36).Let's call that x variable a covariate.Now we observe for a particular customer y1,y2 and y3. One can build a model between y and x exploiting whatever cross-correlative structure exists between y and x and any autoprojective structure evident in the lags of y.This might need augmentation using Pulses/Level/Step shifts,seasonal pulses.local tine trends reflecting omitted deterministic structure.THis is called a Dynamic Regression or a Transfer Function
In what sense do you want a *proof*? Are you just trying to understand the ideas? Is this a class assignment? What would having such a proof help you achieve?
The prime $'$ in $f'$ and $f'' = (f')'$ stands for the *derivative*.  [Wikipedia](http://en.wikipedia.org/wiki/Derivative#Computing_the_derivative) lists some basic formulas and rules for computing it, but if you're unfamiliar with this concept, it would be best to study some Calculus: the equivalent of a quarter of the first semester in a [standard course](http://ocw.mit.edu/courses/mathematics/18-01sc-single-variable-calculus-fall-2010/syllabus/) would be enough.
I think you wanted to say `we fail to reject the null hypothesis`. Just think about the the actual meaning of the words accept and fail to reject and how are they different.
Post-hoc analyses after this rely on "pair-wise" comparison, so are Mann-Whitney tests okay to look at where the significances are?
@Raaj, for approach 1 consider the point (0.71,0.71), is it in the quarter circle? what does approach 1 say?
..how does that reply not answer your question?
are you confounding observations and variables?
this one is just great :-)
In the next paragraph the question says "a uniform distribution over its range" which implied to me continuous uniform.  But at the top he probably meant discrete as you point out.  Either way independent does not mean (or contradict) uniform, either discrete or continuous.
how do you define a "neat pattern".  I see no rationale for why those blue points would be excluded besides visual inspection.  If you had a criteria for exclusion it would make more sense.
I'm having trouble understanding the equation; I can't find any way to justify it. Even a basic units analysis identifies problems. Suppose $y$ is measured in parsecs and $x$ in drams, for example, so that $f$ and its estimator (is that what the little dot over the $f$ means?) are also in parsecs. Then the lhs and $\\sigma^2$ are squared parsecs; the middle term on the rhs is the square of a difference between a parsec ($f(x)$) and parsecs per dram (due to the division by $x$); and the last term on the rhs is squared parsecs per dram. It's not valid to add any of these terms to one another!
have you seen the heritage health prize? It sounds uncommonly like your problem, so if you go to www.kaggle.com you may be able to find some suggestions as to how to deal with this kind of problem
@Dualinity sorry about that, link fixed. Questions like "which classifier will perform best for this given finite dataset" are always best answered empirically. However, I would say that if you have a reasonable amount of data which is relatively low dimensional, $k$-nn should at least be consider.
I would honestly just make a nice graph. It will probably tell as much of the story as there is to tell.
How can variance be "less than -1"?? Your post is unclear - please clarify the circumstances.
Thanks, Peter, for your plotting and interaction suggestions. If one quantifies the treatment effect as relative change from baseline, then the interaction between baseline biomarker value and treatment effect is simply equal to the post-treatment biomarker value—which I hadn't thought of before.
Thanks very much for that - had been fiddling with other "TABS" trying to get it to work. The interaction terms are spat out perfectly. I can go back to R knowing I have done my best for the unfortunate SPSS user.
"there is quite some variation for each time step across the individuals in my data set." There is variability in what? The outcome? The time-varying predictors? Should be both.
Although most of them don't seem relevant for the OP questions, still, this is a nice link - thanks Jeromy!
Byt the way, for more information about hierarchical agglomerative clustering:

Izenman, Modern Multivariate Statistical Techniques. Springer. 2008.
@Ariel, I really have no idea how much bias will result--it will depend on many factors. If your made up values are exactly correct, there won't be any, but that's not going to happen. 1 thing you could try is to run some simulations. Eg, you think they're really 1,2,3, (arithmetic sequence), but they're actually 1,2,4 (geometric sequence) and see how results differ, etc.
@Marco, @Didier Piau, it might be noted that @Didier's suggestion is another special case of an $f$-divergence where $f(x) = x \\log x - (1+x) \\log( \\frac{1+x}{2} )$.
@cardinal The application of symmetry needs to be a little subtler when an even number of balls is drawn.  BTW, a panel at the right of the review should show some statistics, beginning with a line of the form "Answers 1".
Models can be used for varying purposes.  Suppose that the goal is to have the narrowest (in the original, not-log scale) 95% prediction intervals.  One way to argue that comparison using R-sq is a bad idea is to find a simple example of data such that the log-transformed version has a higher R-sq and yet produces wider confidence intervals.  No such example comes to mind yet.
@RussLenth Right, I'm aware of that, just pointing out that the other possibility of seeing a chi-square based tested is from a LRT.
Welcome to the site.  If this is homework it should have the homework tag.
So using the adjusted boxplot, does this mean that the terms significantly-enriched in the given corpus of data will be those above the fence?
"How does our brain automatically correct the order of letters?" Maybe it's not correcting the order. Just because we write words in order doesn't mean that's how our brain recognizes them. People are starting to process information a couple of words ahead while they're still starting to process the word in front of them. From the example it seems that the first/last letters (probably combined with the overall context) are used to pare down the possible space of words, and each of the letters present inside the word narrow it down even further. But, that's just further speculation.
One time bump. Not sure if that's within the etiquette of this site.
+1, it's a nice contribution to have some better optimized code provided as well, & I appreciate your restraint in your comment about mine.