tttjjj/elt_bench huggingface dataset:
This appears decision trees, I am really looking for a clustering method
What made you use stepwise regression?  Do you know how to run simulation studies that demonstrate how poor these methods perform?
I have tried spectral clustering and it breaks down on large matrices, and the worst part is that in star graphs the nodes of the center come out on the ends of the spectrum.
Thank you, I clarified the question: I need to know *σ* as well, is this possible for every *y*?
And the answer is yes, there are ways to disprove the process is exhibiting CLT-like behavior: apply a distribution test to the data you have already collected.  Getting "general bell-like shapes" is common and often has little to do with the CLT.  But, once again, please note that you have not supplied any of the information about your process that would be needed for readers here to give you objective, informed, or relevant advice.
@whuber: Interesting remarks. Thanks for sharing! I always appreciate your insightful thoughts on such things. Regarding my previous comment on "random linear transformation", my point was that, at least through $\\mathbf{x}$, the transformation used depends on the sample point $\\omega$. Another way to think of it is there is a fixed, predetermined function $T: \\mathbb{R}^{n-1} \\to \\mathbb{R}^{n-1}$ such that $\\mathbf{w} = T(\\mathbf{x})$, but I wouldn't call that function linear, though it is linear on subsets that partition the $(n-1)$-cube. :)
With your above data in two columns in a data frame (here called `a`), I believe an appropriate analysis of your data is: `expec <- outer(rowSums(a),c(.5,.5));
(chisq <- sum((a-expec)^2/expec));
pchisq(chisq,df=dim(a)[1],lower=FALSE)` . Note that this is NOT a chisq test of independence.
@MarcClaesen looks like the last sentence should have had a ? at the end, so I added one.
(+1) You are really cleaning up the backlog on these problems! :-) Good work. An alternative argument for the first part: For $W \\geq 0$, $\\mathbb E W = 0$ implies $W = 0$ a.s. To see this, note that if $W \\neq 0$ almost surely then there is a $\\delta > 0$ and a set $A$ of positive measure such that $W > \\delta$ on $A$. Hence $\\mathbb E W > \\delta \\mathbb P(A) > 0$. Now, take $W = (Z - \\mathbb E Z)^2$ to get the result for the variance.
@user46343: was there anything else you wanted to know, or is my answer [acceptable](http://meta.stackexchange.com/a/5235/246931)?
I'm not sure whether to upvote this. I was thinking of exactly the same thing, but I can't tell if it's an answer to the question b/c I really can't understand what the goal is here (not that that's your fault).
@Wolfgang, later added EDIT of your reply states that Fisher is better. Comparison-of-slope-under-standardization approach is inadequate because "standard errors ... when you standardize ... do not take this standardization into consideration". Please explain me how they ought to take standardization into account so that comparison-of-slope-under-standardization approach become as valid as Fisher's test.
The result follows directly from the formula for elasticity in the [Wikipedia article](http://en.wikipedia.org/wiki/Elasticity_%28economics%29) because the form of the rhs shows $B_1$ is the derivative of $\\log(Y)$ with respect to $\\log(X)$.
Yes that's correct but if you assume the signal is real valued, you could take the IFFT of the periodogram square root (this may not hold for all signal properties given non-unique solution)
Could you explain a little more what you mean by 'differentiating' between them? Are you looking for a predictive model for which segment a consumer most likely belongs to based on these variables? Or just to characterize each group?
Heuristically, the sums should not be generalized Gaussian. (1) The shape parameter controls the heaviness of the tail, and the sums should in some sense have the same tail behavior (so $a$ must be the same) but (2) the CLT suggests that the mean will be asymptotically normal (i.e. Gaussian shaped near the mean) but the shape parameter *also* determines the shape of the distribution near the location parameter - so the shape parameter would need to change if you summed them up.
See the third illustration in the thread on [R^2](http://stats.stackexchange.com/a/13317), Julia: It slices a scatterplot (having high correlation) into skinny vertical pieces, each of which clearly has low correlation.  The mathematics will just describe what your eye clearly sees in this picture.
I think this is hazardous thinking. The AUROC is useful IMHO only because in the binary $Y$ case it equals the concordance probability ($c$-index; Wilcoxon statistic; Somers' $D_{xy}$ rank correlation coefficient).  Drawing the entire ROC leads to dichotomous thinking.
I rolled back an edit because it reflected a misunderstanding of the (conventional) notation: an empty product is, *by definition,* equal to $1$.  Thus when $k\\le 1$, $\\prod_{l=1}^{k-1}f(l) = 1$ for any function $f$.  Although it might seem like a technical nicety, including the possibility of such empty products in the formula reflects the process of computing the survival function for $k=0$ and $k=1$, which--because such values of $k$ are less than the support of $X$--is (axiomatically) equal to $1$.
@Srikant: Could you please elaborate on the following statement (i.e. how do I do what you suggested, in practise): "You can then assess the accuracy of the device by looking at the following probabilities: P(S=s|W=s)."
It may be worth pointing out that the cost of using risk ratios is that they are not constant over the conditioning variables, a problem worse than non-collapsibility in my view.  For example a risk ratio of 2 cannot apply to a high risk individual whose background risk is 0.51.
I would guess that your colleagues' recommendations are based either on intuition or some sort of market research. I am also guessing that the various recommendations wind up not making too much difference. Finally, the right recommendation might vary depending on the type of product.
The best thing I think for you to do right now is explain what you're trying to do in general (forget about removing outliers explain what your overall goal is). I highly encourage against removing data unless you know it was somehow recorded incorrectly.
**There are mistakes in the calculation of $p_0$:** it assumes *exactly* 1 "b", 2 "o"s, and 1 "t" will be chosen; and then it assumes the choice of the other three letters will be independent of those choices, which it is not. Assuming $n=100$ is the total number of tiles, the resulting value is larger than it should be (it equals $8/2585\\approx 0.0031$). The same mistake is propagated into the calculations of the wildcard probabilities.
In referencing this paper, the Marsaglia and Marsaglia paper (2004) states that this asymptotic expression "also suffers from lack of a method for its accurate determination."
aha! Turns out a better approximation when $\\mu=0$ is a $\\chi_1^2$ random variable, which makes perfect sense since a normal random variable squared is something like a $\\chi^2$ random variable.
Feel free to explore any of those possibilities and how they would affect your decision, but I can add some specifics to the problem if that would help. I would assume a random temporal distribution. The display is blank until the button is pressed, and the digit will appear for a short amount of time afterwards. It is blank when you see it. The button being pressed causes the display. But again, feel free to inform me on how a different constraint would change the problem significantly and use that as rationale.
So, conceptually, I am right (and thank you for the thorough explanation).  From a practical standpoint, however, our model still needs significant improvement in order to provide any kind of informed insight that couldn't be achieved by a much more simplistic strategy.
[Mining of Massive Datasets](http://infolab.stanford.edu/~ullman/mmds.html) by Anand Rajaram available for free ONLINE
Hmm, I try to guess what you mean. The sampling model for a mixture of Normals would be: randomly draw a normal distribution from the $K$ you got according to $\\pi_1, ...,\\pi_K$. Then draw a sample from that Normal distribution. This means that conditioned on the component your data point is multivariate Normal.
That all sounds like what I want to hear, and yes we're weighing the data against common sense, but what statement can I make with the data I have. I really just need some simple calculations, like (a*b)+(b/c)+r = something, to plug into the reporting engine.
I'll try to studied those sources. Thanks very much Patrick Coulombe.
You certainly can, but GNLMMs (generalized nonlinear mixed models) can be challenging to implement. [AD Model Builder](http://admb-project.org) is widely used in fisheries for nonlinear/mixed modeling, but has a steep learning curve ...
(+1) I've reworded things slightly to try to avoid a possible misinterpretation. We can get away with somewhat less in terms of the sampling scheme: exchangeability is enough.
Although it's not perfectly clear what you want, your request sounds remarkably like this question, which has answers: http://stats.stackexchange.com/questions/17109/measuring-entropy-information-patterns-of-a-2d-binary-matrix.
Given you want to look at the relationship between your factors and win/loss I think you should use logistic regression. This will tell you which factors are more likely to result in a win/loss.
... if there aren't any joints you are going to have to decide which conceptual joints make sense for your particular case.
You define three conditions & then talk about an undefined fourth.  What's with that? In any case, it sounds like reading this paper might help you: [Gelman (2006), The difference between 'significant' and 'not significant' is not itself statistically significant, *The American Statistician*, **60**, 4](http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf)
@cadamt, I wrote my comment independently of StasK, i.e. I saw his comment after I posted mine. As you see we ask similar question, but word it differently.
@James, can you clarify what you mean by $A/B$? $A$ and $B$ are events, not numbers - what does their ratio mean?
How "small" is your dataset? 1% of a population of say 100,000 is actually quite large and inference on a mean can often be done arithmetically.
Do you want to take sampling schemes into account, or do you just want to understand the basic idea of a confidence interval for a binomial distribution? If you want to take real people and do this, and you want to be able to generalize to the population of your country or the world, you will have to work with much more sophisticated analyses.
+1 for bringing me a fresh perspective of viewing PCA. Now as I understand it, both PCA and FA can explain variance of the observed variables, and since FA dictates that the error terms for each variable should not be correlated, whereas PCA does not make such dictation, so FA can capture all the covariance in the observed variables, but PCA fails to do that, because in PCA the error terms might also contain some covariance of the observed variables, unless we use all the PC to represent the observed variables, right?
+1, this is a nice list of some available options to explore.
ross, please consider upvoting and, if applicable, accepting helpful answers to questions you've asked.
(+1) This might be my single biggest problem with epidemiological research.
Thanks dmartin. I'm reading that web page. :-)
As it happens, I am using this text in a course that I am TAing. The students hate this book. Stock and Watson is a common intro book, with Wooldridge's Introductory Econometrics a bit more sophisticated, but very clear.
If you really want to calculate this by hand there is a ton of information about this on the web, for instance just look at (http://sciencefair.math.iit.edu/analysis/linereg/hand/) or (http://www.stat.rutgers.edu/home/buyske/401/RegHand.html). For the year just simply turn them into 1,2,3,.. or take the middle year for each interval such as 2002,2004,.. If you take the coefficients that I gave you and plot them with your data you will find a very good fit.
It doesn't matter how fancy the methods are, if they assume that you've measured everything relevant and you can't prove that you did, causal inference is problematic.  A sensitivity analysis can help, e.g., what properties must an unmeasured confounder have before the association would be explained away.
Both R and Weka have incredibly slow implementations of this algorithm. It's correctly written DBSCAN, btw. the "N" for example is for "noise".
I think it's in theory possible to assume a normal distribution for the students' results, but I don't see the teacher's input being normal nor independent of the pupil population (from experience on the battlefield). If you want to check the limitations of your equation, you need to go back to your problem more specifically.
Thank you. Well explanatory:)
If your software is well written it automatically standardizes internally to avoid numerical precision problems.  You shouldn't have to do anything special.
There are good questions in here but you're asking too much at once.  Focus on one issue at a time and formulate a specific question for it.  Specify what you do know and what efforts you have made towards answering the question.  "How do I even start" doesn't give us anything to go on--we would just refer you to a textbook or Wikipedia (which is not a bad idea, by the way).
Sounds like you want sum-to-zero coding, [here](http://www.ats.ucla.edu/stat/r/library/contrast_coding.htm#DEVIATION) called deviation coding.
Are you interested in Bayesian non-parametrics? If you do not want to make assumptions on the distribution of the $X_k$'s, you need a non-parametric framework. But then, given only $S_k$ you cannot say much...
Frank - These are great resources. I wonder how this information relates to what @Dan has provided in his answer. Perhaps I am misreading this, but it looks like the community is divided on this issue.
There's no point testing; every test of any use at all, a any reasonable significance level will clearly reject. Whatever guide you're reading has misled you. What do you mean by 'reliable' exactly. What 'limitation' of the Shapiro-Wilk do you refer to? I would very nearly agree with the statement in the answer you link to ... "I have never come across a situation where a normal test is the right thing to do" (I've seen at least once situation where I think it's the right thing to do, but people almost always do it for bad reasons).
Does any underlying theory predict the interaction ought to be there?
The question is too vague. What is the joint distribution of the two random variables? Are they independent or not? What are the two unknown parameters of interest? Are there any other parameters, and if yes, are they considered known or unknown?
I do think the question has improved, in that it more clearly reflects your situation & what you need to know, however, I note that the question now revolves, in large part, around how / why 2 R functions appear to give differing results. We can see how this Q fares in its current incarnation, but if you don't get a satisfactory answer it may do better on [Stack Overflow](http://stackoverflow.com/), or probably even as a submission to the R help listserv.
Are all the observations statistically independent? If so, your theory provides a way to collect a set of independent *standardized* observations by dividing the difference between each observation and its predicted value by the standard deviation. That reduces your question to "how can I compare a set of observations to a standard normal distribution," one that is answered in [many threads here](http://stats.stackexchange.com/search?q=fit+normal+distribution+test+goodness).
Thank you very much. I will check this function.
Additionally, you can look at the code behind `effect` to se how they are calculated.
It's not quite clear what you're asking for here. When you say "Ideally we would like to have K here as 5 or 6...", for example, how are you arriving at that? What property do you want $K$ to satisfy?
@hung I hope you didn't misunderstand me. I didn't want to imply that your question is worth only two cents, but that my answer is only worth 2 cents (I thought this was a figure of speech in English, but I'm not a native speaker, so sorry for the misunderstanding). Anyways, I hope my answer helped. If there is still anything unclear or you want some more info, let us know. Either I update my answer or someone else might have a better one. If you are satisfied, it would be great if you could mark the answer as accepted, so the question is off the list of unanswered questions.
You have to provide much more details. What does your effect size suppose to detect? What test will you conduct in order to detect it? What will be the type I error rate? You will also need to provide some measure of variation. Anyway, I am afraid your question cannot be answered as it stands. Also, note that one can find online sample size calculators for most common tests...
How is this question different from <http://stats.stackexchange.com/questions/61834/which-is-applicable-ordinal-or-multinomial-regression-model/> ?
How about building a joint dataset using your sets A,B,C by joining them on the variables in C?
I have taken the liberty of softening the first couple sentences a bit and reformatting some things. Please review and make any changes you'd like, or, even, feel free to rollback if you prefer the original version.
I'm so sorry. I somehow "edited" your comment, when I meant to just "comment"!!
@whuber In that question, they've said that $P(U > u) = P((k-1)T < X \\leq kT - u)$. How have they got this?
There was some discussion [here](http://stats.stackexchange.com/a/29233/5739).
@rbatt Peter is correct in this well-explained answer. Your comment introduces an assumption not made in the question.
Wow! Thanks for finding a viewable reference—I hadn't thought to look on Google Books when looking for a copy of Wooldridge's book.
I am sorry for the misnomer of "independent X's". What I meant was "independent Y's conditional on X's". Namely, the residuals must be independent. This assumption is violated in longitudinal observations. Reference: Modeling longitudinal data, II: standard regression models and extensions. Methods Mol Biol. 2009;473:61-94.
the form of $f(X)$ is beyond the scope of this problem, but it is a [dynamic global vegetation model](http://en.wikipedia.org/wiki/Dynamic_global_vegetation_model) described in the appendix of [Medvigy et al 2009](http://www.oeb.harvard.edu/faculty/moorcroft/publications/publications/Medvigy_etal_2009.pdf)
Is there any rule to check if the Markov property applies to my situation? That sounds very appropriate for implementation but I don't know if it would be correct.
@DavidJ.Harris Yes, the paper seems very cool. To be specific, I want to understand how they are sampling from the posterior with a gaussian process prior (section 5.1)
If you can elaborate on "I am trying to compute a _pathway score_" -forget statistics for the moment, write about your discipline - and we may find a way to use statistics to help you. Is there a theoretical but unknown "pathway" (whatever that means),  that you try to approximate through the weighted combination of genes? Is there likewise some "pathway score" that exists but it is unknown and you try to uncover? Are you just trying to _describe_ something through this linear combination?
@Ray, your picture is a viable explanation of the sign of a coefficient; it is like picture [here](http://stats.stackexchange.com/a/70910/3277), only 2D. But I don't see how it can explain _suppression_. To show suppression you must show error term because suppression is defined wrt it.
Have you seen this? http://cran.r-project.org/web/views/ExperimentalDesign.html
No problem, Gaussian processes have their roots in statistics (the paper where it is all brought together is by O'Hagen IIRC).  GPs are a very good approach for regression and are essentially the culmination of the roughness penalty approach of Green and Silverman.  You can also download the book for free from their website.
It is true that white noise will yield a ARMA(0,0) model, but it is not necessarily true that all ARMA(0,0) models are white noise. There could be something else going on, not captured by the model. Or N could be low. Or there could be a lot of noise on top of something actual.
Where are you stuck? I can think of two big places where you could get stuck in this sort of analysis: 1) Getting the data from the interview into a form that be analyzed statistically or 2) Doing the analysis.
@YouSafe, Yes (provided that the only difference between the 2 models is the interaction term) you can quote the p-value as a test of that interaction.
(+1) Although software-specific questions usually belong elsewhere, we might be the best possible place for stats/data analysis questions about how two (or more) stats programs compare or interact. I suggest refraining from voting to close.
No, there is no reason to sample anything to solve the problem that you started the discussion with.  Just provide the 0.95 confidence interval for the true risk.
The answer depends on the effect size.  The R function power.prop.test() can tell you the answer.
@mark999 I agree. I'll be interested to read the answers to this question. There are some general strategies, but I find that a lot of checking is about building domain specific expectations, both about what the data should look like, and some of the common errors that can arise.
What do you mean by "significance value" - is it related to a statistical hypothesis test or is it something about materiality?
@soufanom Thanks for the clarification of the meaning of the statistical "measurement of importance". Could you expand a little more about your first sentence. My understanding is that if a feature is ranked lower (lower feature importance value), it is less important for the classification of the target data. If a feature is ranked higher it is more important for the classification. In that sense, the "feature importance", can be used to compare one feature to the other and to create a ranking list.
[Wikipedia](http://en.wikipedia.org/wiki/Latent_semantic_indexing) gives impression that LSA=LSI. But LSI is _correspondence analysis_ (CA). CA is a statistical analysis term, like PCA, while LSI/LSA is text mining term. So, search for articles comparing PCA and CA.
This question appears to be off-topic because it is about software use.
I think it might make the point; depends how formal you want to be. `n` would be the number of categories in the factor, but I think if you just say `contrasts=list(COIsc=contr.sdif)` that should work OK. Deviance and log-likelihood don't give a measure of goodness of fit.  See http://glmm.wikidot.com/faq#rsquared
Besides, any of the following that we accept as the criteria: a) Different means serves different purposes b) Type of variable is important 3- Distribution of the data has a role. Why can't they be put into a formal framework?
Peter, there may be some mix up of the terminology here. From the point of view of finite population sampling, the only random variables are sample inclusion indicators, $I_i=1$ is unit is in the sample, $I_i=0$ if not. Height and weight are fixed characteristics of the units, and there's no randomness in them: your weight does not jump around by 20 lbs from one day to next, does it? Sampling with replacement ensures ${\\rm Cov}(I_i,I_j)=0$ for $i \\neq j$; sampling without replacement produces ${\\rm Cov}(I_i,I_j)<0$. That's advanced material though, and only comes up in specialized classes.
The coefficient vector $\\hat{\\beta}$ is the solution to $X'Y=(X'X)^{-1}\\beta$. Some algebraic manipulation reveals that this is in fact the same as the formula you give in the 2-coefficient case. Laid out nicely here: http://www.stat.purdue.edu/~jennings/stat514/stat512notes/topic3.pdf. Not sure if that helps at all. But I'd venture to guess that this is impossible in general based on that formula.