tttjjj/elt_bench huggingface dataset:
@BrashEquilibrium! That was very informative! Many thanks! Yes PCA is what I had begun with. Logistic regression after PCA gave poor results, but perhaps I should also try regression with LASSO. The whole purpose of posting this question was to check out if there is anything else that I should try (apart from several things I have already tried), before proceeding with further data collection. Thanks again.
Stephane I have edited your question to improve the English. i hope you don't mind. I think your English is good and the meaning of what you write is mostly very clear. I did the for two reasons: (a) I am a compulsive editor and (b) I think that maybe these edits can help you to improve your English a little.  If you don't like it I won't do it again.  Please just in case I misrepresented something that you were trying to say.
@lindelof followup point: how do you determine "decent coverage" for discrete predictors?
Friedman test is what comes to mind at once, because it is the repeated measures test for more than 2 related samples. But it is not very powerful: Friedman is _not_ the direct extension of Wilcoxon test, rather, it could be seen more as the extension of sign test (which is less powerful than Wilcoxon).
@NAUGHT101 . Critical values for the correlation coefficient are available under the assumption of joint-normality and undefined else wise.
Highly relevant discussion with several insightful answers and further links: [Best practice when analysing pre-post treatment-control designs](http://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs)
@Aniko, the OP's results are from linear regression, not logistic.  You may want to rewrite your answer because right now it's just not applicable, I'm afraid.  Also, I differ with you when you say "the first model is clearly wrong (because it assumes no interaction when there is one), so any inference from it is wrong as well."  It is a matter of judgment whether a set of 3 interactions that collectively take r-squared from .70 to .73 is worth preserving.
@Gavin You remember correctly.  The official dates recorded by SE are Private beta: 2010/07/19; Public beta: 2010/07/26; Graduation: 2010/11/04. Anyone here longer than 34.5 months would have participated in the public beta. Incidentally, [the first mod elections](http://meta.stats.stackexchange.com/questions/734) were held early in 2011.
ars is correct. It's AIC=LOO and BIC=K-fold where K is a complicated function of the sample size.
Hm. You could look into natural language summarization techniques. Often, automatic summarization of a document involves splitting the document into sentences, clustering the sentences, and then reporting one sentence from each cluster.

One basic way to do this is to represent each sentence as a [bag-of-words](http://en.wikipedia.org/wiki/Bag_of_words) vector. You define a high-dimension vector space where each dimension is a word. Sentences are vectors of the number of occurrences of each word. You can then cluster these vectors using common algorithms (e.g., k-means).

Does this help?
It may be relevant, and somewhat amusing, to note that actually measuring a six-digit probability would require a sample of $10^{12}$ people, which may be larger than the number of people who have ever lived.
The chi-square approximation doesn't rely on the number of cases being some number, but on the expected values.
@illiafl, yes that's exactly right. Based on the command per user, I need to figure out how many approximate users are there and probability.
Here's something that might [suggest a problem with the notion](http://stats.stackexchange.com/a/51753/805) - such a system will characterize those 4 plots differently, just as a human might. At the least, it suggests a need for a large caveat with such an automatic system.
No. The gamma varies smoothly around $k=e$. Even if the formula collapsed to something very simple, and it doesn't, that wouldn't be a reason why it should work particularly well with data. Sometimes numbers just are close to $e$, but without meaning anything. I'd guess the same story for negative binomial.
Sorry if I'm being unclear ; I assume that $Y_i = aX_i + \\beta(K_i) + \\sigma(K_i) u_i$, where $u_i$ is white noise, and I want to estimate $a$ and the $\\beta(k)$s. The $\\sigma(k)$s can be considered as "nuisance parameters" is this is the right expression.
(+1) woa, great references here :O !
Beware applying calculations that apply to events specified in advance to something observed *post hoc*. That is, if you see something that may be unlikely and say "what are the chances of *that*?", the calculations done as if it were an event specified before the draw are nonsensical (as simple thought experiments can show - e.g. the '[wheelbarrow full of distinguishable dice](http://stats.stackexchange.com/questions/67721/is-there-a-law-that-says-if-you-do-enough-trials-rare-things-happen/67723#67723)' experiment).
... and maybe a better topic?
Aha, indeed (+1).
Also to make a more general point, as you become knowledgeable in statistics (or data mining/machine learning etc) you'll find that answers to very general questions such as yours will often be a paraphrased version of 'it depends'. Knowing what 'it depends' on and why is the knowledge.
These data wouldn't be Poisson, if you have a known total who could be compliant. Instead they would be binomial. Are you wanting to compare A vs B, or T1 vs T2, or both? Are you essentially trying to do a power analysis?
I doubt you should do ANOVA at all; you probably want some form of multilevel model. The other papers might also be doing this, which could account for the df in the F test. Your data are not independent.
(+1) You are correct, Richard. I'm not sure what I was thinking about when I wrote that comment (perhaps [stability](http://en.wikipedia.org/wiki/Stable_distribution)), but, at any rate, I've removed it. Cheers.
The correlation coefficients are *numbers,* not random variables, and therefore do not have distributions.  They are properties of the joint distribution of $(X,Y,Z)$. As to your last question: the distribution of $Z$ is determined by a single number, say $\\Pr(Z=1)$. This depends on *four* independent values that determine the distributions of $X$ and $Y$ (say, $p_1, p_2, q_1, q_2$). Therefore you can recover one algebraic relationship among these four parameters from knowledge of $Z$, but that's all.
thank you @Anony-Mousse, I'll look into DBSCAN algorithm. I am quite new to large-scale clustering and familiar with only k-means and hierarchical clustering algorithms. Can you suggest me some related modern clustering algorithms for spare vectors to start with.
I think you are far from understanding the point, and I am not sure you should try to go further now. If you are motivated, you can read the paper from Bradley Efron I mentionned or that paper from Amari http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aos/1176345779.
I'm using the RBF covariance function. Squared exponention. When i exclude the x* covariances from the matrix everything is fine as well. It just happens in some cases. I guess it is really a problem with the implementation of the Commons Math solver.
More general version of the question [1](http://stats.stackexchange.com/q/32309/4485). Almost the same question [2](http://stats.stackexchange.com/q/30585/4485).
I bet that's meant to say "n=100"
Just to add a bit of concreteness to this answer and the one from @user12202013, the effect size should be something really meaningful to the people wanting the study. Talk to both the skin-care people and the marketing people; get them to tell you how big a difference is really meaningful -- to the customer, to serve the marketing needs, etc. And how big can a difference be and still be not worthwhile?    BTW, your previous study was *not* underpowered: you found significance, so obviously you had enough power for that to happen.
At present, this is more of a comment than an answer. Would you mind expanding it a little bit to make it more answer-ish?
Hint: there are [close connections](http://en.wikipedia.org/wiki/Chi-squared_distribution#Relation_to_other_distributions) between chi-squared (or, equivalently, Gamma) distributions and Beta distributions. F distributions [can be obtained from Beta distributions,](http://en.wikipedia.org/wiki/F-distribution#Related_distributions_and_properties) too.
When you say "restore", what do you actually mean?  Do you know the functional form of $\\rho$ and seek to estimate the parameters?  Do you seek a nonparametric estimate of $\\rho$?  Or something else? In addition, how large a sample do you have?
How is what you're asking about different from assessing seasonality (which I believe is standard in time-series analysis)?
+ I should have seen the symmetry. I still don't follow your BetaPrime argument (certainly due to my rusty algebra), but [*here*](http://en.wikipedia.org/wiki/Beta_distribution#Moments_of_linearly-transformed.2C_product_and_inverted_random_variables) is discussed the inverse of a Beta, which tells me most of what I need. Thanks.
Further: PARCOR coefficients are the same (except for a sign change) as the "reflection coefficients", which can be obtained from the AR coefficients via Levinson-Durbin recursion. And, essentially, testing for reflection coefficients having modulus less than one, is the same as the Schur-Cohn test in the other answer.
http://onlinelibrary.wiley.com/doi/10.1002/9780470611104.app6/pdf
How would I measure how similar they are?
+1 This is a great question because it appreciates that there's more to random sampling than some vague form of arbitrariness or lack of knowledge about the sampling procedure.
Agreed. You can also show that a hierarchical prior on sigma is inverse-gamma distributed, but that might be harder to work with. Also, check "Doing Bayesian Data Analysis" by Kruschke. Great book. Might have the example you're looking for.
What does your question mean?  Are you looking to show that the meanis infinite.  Isn't that obvious from the partial sums of the first n being n(n+1)/2 meaning that the average is (n+1)/2?
Ok I see you point. Thank you very much for your replies.
My first question, would be, what tool would you prefer to use to do a linear regression? Would love to help with a solution that can show you how to do it vs just doing it for you (which I don't think will help in the long run).
(ctd) ... but if the distribution were actually lighter tailed than normal, it may be that all three estimates are too high!
Some of the ideas of @gung's first paragraph are implemented in the function `SnowsPenultimateNormalityTest` in the `TeachingDemos` package for R.  I agree with @gung's idea of looking at a measure of closeness rather than focusing on a p-value.  One problem with using the correlation in the qq plot for this is that if your data has the correct shape, but different mean, variance, etc. you can still get really high correlation.  An alternative is to use the KS statistic or AD statistic as measures of difference from the theoretical.
Arguably this is not a question about data analysis at all, so PCA is practically irrelevant.  It is a question about *valuation.* Because the scores are intended to be used to compare objects, the weights given to the variables and how these weighted values are combined determine how much of one variable can be traded off for how much of another variable. This is the subject of multi-attribute valuation theory, as discussed in a concrete version of the same question at http://stats.stackexchange.com/questions/9358. (One comment thread there also discusses the inapplicability of PCA.)
Please vote up *any* and *all* reasonable answers: let's encourage people to contribute their ideas!  Remember, you have the additional ability to award an additional 15 points to the answer you eventually think is best.  Overall you will gain far more by *awarding* points than by *withholding* points.
Do you mean regression of y on time and time squared?  What is the test?
@ttnphns I don't seem to find any argument on why my nxk-means approach should be good or bad; or a good alternative, other than statements that k-means is prone for finding suboptimal minima.
A reasonable starting place to look would be the [CRAN Experimental Design Task View](http://cran.r-project.org/web/views/ExperimentalDesign.html). Don't know if it actually does hold what you need, though.
Consider adding a tag that directly references the R package you are using.
A missing value means no definite answer was obtained.  "Don't know" potentially says a lot about the respondent: it could be related to absentee fathers or non-communicative parents. So coding "don't know" as a missing value looks like a mistake, potentially a serious one.  **Some fundamental rules of data collection**: (1) Don't try to process the data while recording it. (2) Record exactly what is observed.
What is CEM? CV doesn't usually do tutorials & we aren't a software support site; be aware that although I think this question is sufficiently on-topic, you may not get guidance on doing this in Stata w/ your answer.
Could you please give an example for picking $v$-functions?
Duplicate of [this](http://stats.stackexchange.com/questions/74542/why-does-the-lasso-provide-variable-selection). Also see [here](http://stats.stackexchange.com/questions/86973/why-does-not-ridge-regression-perform-feature-selection-although-it-makes-use-of?lq=1) or [this diagram](http://www.astroml.org/_images/fig_lasso_ridge_1.png)
@Stat The OP is answering his own question (on the basis of some suggestions about how to approach it) - and providing a full answer at my invitation. People solving their own problems is perfectly okay on self-study questions.
Is it reasonable (in principle) for me to model all the data using logistic regression, if I'm planning to later collect another data set and attempt to use the model to predict the cancer status of people in that data set?
Although it is a basic question, it likely needs an original analysis to be answered. Computations with independent $t$ distributions will not do the trick because the two prediction intervals will be strongly correlated. So, although the formula is explicitly given at http://stats.stackexchange.com/questions/9131, it's only a start: I don't think you can use that directly to solve your problem.  A Bayesian analysis might be more tractable and perhaps would conform better to how you are thinking about this problem too, assuming you want a probability *conditional* on the existing data.
@Jeromy Anglim I would like to create a heatmap given dynamic data that should indicate lower and higher levels of a property. The intervals of lower and higher should be determined dynamically based on the dataset. The heatmap will have a legend indicating the interval that each color represents and the intervals should be "logical" to the person using the heatmap in analysis of information
@David In short, I feel this doesn't make sense -- this is a "wrong question" problem. The sparseness shows that the data is in extremely suboptimal form, and much more effective approach is to try to convert it. The paper you linked is a bit other problem.
Arun, my understanding is that Fisher's exact test is not limited to 2x2 tables, even if the original example was of that size. See for instance, C.R. Mehta and N.R. Patel(1986) Algorithm 643. FEXACT: A FORTRAN Subroutine for Fisher's Exact Test on Unordered r x c Contingency Tables, ACM Trans. Math. Soft. vol. 12, p. 154-161.
The book by Hastie et al. is offered as a free pdf download at the book's main website. "R in Action" is available at relatively low cost as a pdf download at manning.com. I believe you can also (legally) obtain the other two as pdf's.
Thanks for your edit.  I took the liberty of editing it further to make it clearer what Hmisc does (and some other smaller edits); if it makes it past peer review, you can decide if it still reflects your question accurately.
If you're not beholden to OpenBUGS, you can do this in R.
It tells you exactly the same thing as when the sample has many observations: the probability under the null hypothesis of equal means that the t-statistic would exceed that observed. Of course its power will be lower.
Wow that looks very helpful, I'll dig into it next week when i return home, thank you.
Are these plots of actual *data*? If so, you must have an enormous dataset which exhibits no appreciable scatter, making this purely a problem of mathematical discovery (and you might want to point [Eureqa](http://creativemachines.cornell.edu/eureqa) at it.) Otherwise, please describe your *raw* data.
I've incorporated your suggestions including checking for duplicate IPs and identifying the fastest respondents. In addition to duplicate IPs, I'm adding a check for the number of similar respondent's IP Addresses (matching the first three octets as shown in my original question) that took a survey within 30 minutes of a respondent as a warning flag. Thanks for your help lockedoff! I'm going to mark your answer as accepted but if anyone has any additional clever ideas please feel free to pass them along.
@Abe What do you mean by "single $N(\\mu, \\sigma^2)$ prior"? Assuming that you want to estimate both parameters, $\\mu$ and $\\sigma^2$, one has to assign appropriate prior distributions, e.g. dnorm and dgamma.
I don't know where you found this quote, but it sure is poorly phrased.
I guessed the question, since you probably hit post too soon. See if it is ok.
It is not duplicate of: http://math.stackexchange.com/questions/440902/bayes-network-example

It is the same example but the querstion is different. I simply took the same graph and values, it is a conceptual question anyways.
Also doing feature selection on the whole data set prior to classification can heavily bias results. See similar question here: http://stats.stackexchange.com/questions/27750/feature-selection-and-cross-validation?rq=1
+1 I think this is a good question not a stupid one.
@Oleic the Kaplan meier estimator does not extrapolate beyond your data. What you are looking to do is impute points. You will need to specify some model for the tails of your distribution (a common one is an exponential). Non parametrics cannot do this for you as *anything* is possible outside of your actual observations unless you use your contextual knowledge to bound things....but then you're no longer strictly nonparametric.
@mbq Actually, this is a simple GIS/time-series question with a straightforward answer.  Because the kernel is *square* and not circular, it's really a pair of independent "diffusions" in 1D, where its analysis (as a moving average) is a fundamental part of time series methodology.
This question appears to be off-topic because it is about how to do something in R, but without a [reproducible example](http://stackoverflow.com/q/5963269/1217536).
(0,1,0) may be a disappointing model, but it may also be right. Do the diagnostics on the residuals (e.g. ACF and PACF) look decent?
It may help you to read two of my answers to related questions: [Difference between logit and probit models](http://stats.stackexchange.com/questions/20523//30909#30909) (which discusses link functions & GLiMs in general--a comment at the end specifically addresses your 1 & 3), & [Difference between generalized linear models & generalized  linear mixed models](http://stats.stackexchange.com/questions/32419//32421#32421) (which discusses how your 4 is different from 1 & 3).
$\\log (1/x_2) = - \\log (x_2)$ which is what I found
This questions looks like a mixture of random statements... can you elaborate a bit more? Or post some link to suggest some context?
+1 Your second paragraph very nicely highlights something that is missing in my answer.
Sometimes I wonder if even weak consistency - outside its intuitive appeal - is in reality very important. If I have an estimator that behaves very sensibly at every finite sample size below $n=10^{1000}$ and in reality my biggest sample size will only ever be a miniscule fraction of that, I might have an inconsistent estimator that's nevertheless perfectly fine. It seems to me that the actual value in consistency at all is that it's usually (in practical cases rather than pathological ones) associated with estimators that still behave 'nicely' as sample sizes move past what we might ever see.
The Goldberg paper and your explanation clarified very much - thank you! I believe I can proceed then by comparing the out-of-sample predictions against the residuals (to calculate omega and w'). Also the variance will no longer be zero in this scenario.
Large p-values would be better, if they were capable of meaning anything. The linked thread has good material, especially the answer by @GregSnow. You might want to look into [diehard tests](http://en.wikipedia.org/wiki/Diehard_tests), & possibly the package [RDieHarder](http://dirk.eddelbuettel.com/code/rdieharder.html) (although I can't tell if it's available on CRAN).
Have you tried $y^{\\frac{1}{2}}$ or $y^{\\frac{1}{3}}$? These can be applied to zero (and negative values for the cube root), which is probably why you are loosing data. It might help if you post some summary statistics of your raw data (min, max, mean, median, kurtosis, skewness).
Not all samples are random - I think the participants are a sample, just not a random sample. It would be appropriate to call this a convenience sample, because it was drawn from those students available to you.
Direct yes, but the number of states (terms) grows like $2^N$, which can be computationally problematic.
Informedness is something you can easily visualize as you can see it on your ROC plot. It is the distance from (i.e. measured perpendicular to) the "Random line" joining (0,0) & (1,1). Then you can use Markedness as the second metric to define the overall quality of your Machine Learning system. Alternatively, you can combine both in a single metric as MCC = SQRT[Markedness * Informedness]. These 3 metrics are "unbiased" because they suffer much less from the bias of other metrics whose values depend not only on the ML itself but also on imbalances in the number of items in each class.
Well said.I'm looking a distribution of returns from a financial asset.  It is clearly not Guassian or T distributed.  I did find an R package that will fit a Generalized Lambda Distribution that looks fairly close. In order to develop any kind of filtering process I need to develop a proposal and a transition.
FYI, my paper on raking is coming up in Stata Journal.
@StasK: Please note the comment above and the edits to the answer.
I take it from your cross-post that you really want this question migrated. The effect of migration is to close the question on the original site and move it to the destination site. The move has been made; I'll take care of the close.  (Interesting question, though: +1.  I suspect it was simply overlooked here by many people. Our [faq] explains ways in which you can raise the visibility of a question that has languished.)
@Glen_b $\\theta$ is most likely the superset of the parameters of the two models, which would make sense if those are two equations of the same model.
Is that really the estimating equation of a trimmed mean? In your equation $c$ seems to be a constant that "discards" data that is $c$ away from the mean while in the usual version of a trimmed mean you define what proportion of the data points should be discarded from the tails from the data. Aren't these two different things or am I missing something?
I'm confused: your first paragraph suggests that you find a set of models somehow, though you don't specify how. Next, you find the best of those models through ER. Since you do not specify how you got to your first set, it's hard to answer that part. As per R2: there is much debate on its usefulness. If it is an option for you, use cross validation and a _real_ predictive measure like missclassification rate or AUC.