tttjjj/elt_bench huggingface dataset:
1:40 says that for each k in 4:7 it will do 40 clusterings with different initial clusters. It will keep the best one for each k to compare them later. Here "best one" means that it is more "repetible". Repetible roughly means that If you take a random sample of your observations and you cluster on that sample you will get the same clusters you got using the complete sample minus the observations you left out. This is, you find THE SAME clusters. This would express a true pattern in your data. I didn't understand your last comment, could you explain it please?
The cooperative rules/correlated predictors point is an excellent criticism. To make this work, it might be necessary to train the RF on some sort of pre-reduced set of variables, or incorporate some sort of penalization strategy to cause the RF to focus on a subset of predictors.
If null hypothesis $H_0:\\alpha_1=\\alpha_2=\\cdots=\\alpha_n$ is rejected (where $\\alpha_i$'s are effect of different level) then we want to check the null hypothesis $H_{0i}:\\alpha_i=\\alpha_{i'}$(where ($\\alpha_i,\\alpha_{i'}$),$i \\neq i'$ are all posible pairs).Are you mean this by pairwise ANOVA?
Thanks for a simple, clean solution to a not so simple problem! There isn't a lot of documentation on gauss hypergeometric functions for R so this approach is valuable.
+1 the classic.
Compare in what way?
If you ask a committee to design a horse, you won't get a horse. You might get a giraffe or a griffin. But students whose first interests are, or are close to, political science, sociology, psychology are most unlikely to want, or to appreciate, or to have the background for, straight probability and mathematical statistics. If I were in charge I would talk about data production, data analysis and modelling. That's a lot for one programme. You'd need to be very selective.
By selecting suitable units in which to measure the $x_i$ and the $y_i$ you can make the $x_i$ sum to $0$, the $y_i$ sum to $0$, the $x_i^2$ sum to $n$, and the $y_i^2$ sum to $n$. The formulas for $\\hat\\beta_0$ and $\\hat\\beta_1$ will greatly simplify with no loss in generality.
@whuber yes you are correct, and I like that example.  I suppose I'm looking for more detail about how precision is related to the standard deviation. E.g. in R, x <- rnorm(30); mean(x); sd(x) # here clearly the sd is about 1 but in R the mean is printed by default with 7 digits of precision. sd(x)/30 is about 0.18. Thanks
@suncoolsu The terminology in the social sciences might be different, but both Kirk (1995, p512) and Maxwell & Delaney (2004, p592) call models with one between- and one within-factor "split-plot". The between-factor provides the "plots" (analogous to the agricultural origin).
So maybe You should create a time series that is a difference between those two variables ?
I presume you have access to the $10^{32}$ or so CPU cycles needed to solve the problem for $N=100$ and $n=40$? (That would be only around a million years if you could harness every PC in the world full time. :-)
Unfortunately I believe some things have gotten lost in translation, and so it has been difficult for anyone on the site to more pertinent advice. Lets say you wanted to model more outcomes besides 6 or 7, had other covariates you thought might affect the survey answers besides the experimental condition, or wanted to compute the effect size of one condition versus another could all be possible reasons to use logistic regression. The Chi Square test has nothing (directly) to do with possible values of 0 or 1.
You obviously mean a homogeneous poisson process right? Do I understand (2) in the sense that you just count the number of events over some time interval and this is done repeatedly?
+1 That's a great reply to an otherwise opaque set of questions.  Thanks for making sense of this.
By the way, when I say there is not case-level data, I mean that I do not have specific data attached to specific patients.  I have a record of number of admissions and aggregate financial data.  My unit of analysis is a hospital.
@imsc, the common tools for automatically creating dummy variables will create them non-overlapping, but you can create them however you want.  Just have `dir1` be 1 if that director is involved and 0 otherwise.  In R you can do the full reduced test by passing the 2 models to `anova`, other programs may have something similar, or it is not a hard test to do by hand (need the SSE from each of the models).
(+1) Nice summary. @Jasty May I remind you that you can vote up responses that solved your problem or help you to move forward in any way. Also, it's good to mark responses as accepted in the former case so that future users might be aware of the "correct response". Finally, please register your account once for all.
Yes. Such data is called unbalanced/imbalanced data. Usually making the number of points per class equal helps a great deal in getting a better classifier. But then, not necessarily the best classifier. You need to try several things here,
1] Try uniformly randomly choosing 64 points from success class and training your SVM
2] Compute the mean of the failure class and choose 64 points from success class which are closest to the failure mean
3] If you are using libSVM, use the entire dataset and provide class weights proportional to the imbalance that you observe
Thanks, guys. I'm glad this came together well; this is actually a good example of how you can learn things on CV by *answering* questions, as well as asking & reading others' answers: I knew this information beforehand, but not quite well enough that I could just write it out cold. So I actually spent some time going through my old texts to figure out how to organize the material & put it forward clearly, & in the process solidified these ideas for myself.
Re disappointment: You wouldn't expect each problem to have the same parameters, so why *would* you expect the problems to share good values for the hyperparameters (log gamma and C)?
-1, this is incorrect. Did you read the answers by @John or @ttnphns? Both correctly point out that ANCOVA **is** a multiple regression model. The traditional ANCOVA did not allow for interactions b/t covariates & factors (the so-called 'parallel slopes assumption'), but the term 'ANCOVA' is used sloppily & many people use it to include cases w/ interactions. Moreover, I gather SPSS will run an 'ANCOVA' w/ such. Is that what you had meant here? If so, please clarify. If not I must let the downvote stand.
You could always read this: [Coefficient of determination](http://en.wikipedia.org/wiki/Coefficient_of_determination), and then refine your question in the light of what you don't understand.
What have you tried? What do you know about covariance? What does random sampling imply?
Readers here may find this thread of interest: [what-if-residuals-are-normally-distributed-but-y-is-not](http://stats.stackexchange.com/questions/12262/).
+1. @Fran, what you are doing is called a *power analysis*. You will need to specify the effect size you are anticipating as well as the alpha level (usually 0.05) as well as the power you want (i.e., the probability of rejecting the null hypothesis if it is in fact false) - a normal setting would be 0.80. Then you can feed these data into a sample size calculator (make sure to specify that your t-test will be paired) and get the minimum sample size required.
NCM is quite good, but breaks down for large files if we choose compressors like gzip2 instead of PPMZ which have infinite window
I don't completely follow your description of the second step. Do you mean that (a) the grid point of largest likelihood is found and (b) all other values on the grid are divided by it so that (c) the result can be interpreted as relative likelihood?
Thanks for this explanation. I understand now when I should use option 1 and how to do a sensitivity analysis.                                                      PS: I also searched for some literature and found that option 2 is used for assessing 'the estimator's performance in a wide variety of settings'. This is stated in 1st paragraph on page 8 of the following  paper:                                                         http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aos/1176349741
Thanks a lot for your reply!Actually yes, I did not include the other IVs (sorry I was no clear before, I have various IVs;I have also created some interaction terms between country dummy  and some of the IVs to detect country-related differences for specific predictors) in the ANOVA test actually. Some of the IVs are significant in the regression analysis. However, after some experiments with the IVs in the model, I noticed that in some cases the country dummy may remain significant, but with the opposite sign...
Thanks for your further clarification! I appreciate it.
THis is a very interesting question, but has a better place on http://www.crossvalidated.com . I flagged it for moderator attention, so it will be moved by a moderator pretty soon.
Three additional references:  1.  [Beyond Kappa: A review of interrater agreement measures](http://onlinelibrary.wiley.com/doi/10.2307/3315487/abstract) by Mousumi Banerjee, Michelle Capozzoli, Laura McSweeney, & Debajyoti Sinha  2.  [Interrater reliability and agreement of performance ratings: A methodological comparison](http://www.springerlink.com/content/76887531n77808n4/) by John W. Fleenor, Julie B. Fleenor & William F. Grossnickle
"at each iteration six of the 11 sets will be randomly chosen to be in group 1 and the remaining five will be in group 2?" Yes that is what I should do. Ok I will check further on generating them systematically. Thanks a lot.
Have you searched the site? There are a number of questions that cover this information, such as: [machine-learning-cookbook-reference-card-cheatsheet](http://stats.stackexchange.com/questions/12386/); [programmer-looking-to-break-into-machine-learning-field](http://stats.stackexchange.com/questions/26044/); [is-a-strong-background-in-maths-a-total-requisite-for-ml](http://stats.stackexchange.com/questions/40808/); [how-to-get-started-with-neural-networks](http://stats.stackexchange.com/questions/36247/), etc. If you can refine your Q it may be answerable, otherwise it may need to be closed.
:Arthur The Chow Test can be used to test the equality/constancy of coefficients over time . Additionally Tsay's test http://www.unc.edu/~jbhill/tsay.pdf for constancy of variance can also be employed. We have incorporated both of these features into AUTOBOX , a piece of commercial software that I am involved with.
Or another alternative is: using bootstrap, I calculate the 3 C-index in every resampled population. Thus, I get the 95% CI of the C-index of the differences between model A/B/C. The one with 95% CI >0 is the "significant" better?
Problem solved. Big error in reasoning from my side: One of the variables in the 3D data is dependent, so I should have been looking for a regression rather than a clustering algorithm. Density regression works fine.
I believe you need also to assume the two groups have the same sizes.
@Matthew, the paper is saying that the posterior may not be gaussian. Why?
This is a FAQ. The answer is endogeneity. Here are some threads that can help you: [How can adding a 2nd IV make the 1st IV significant?](http://stats.stackexchange.com/q/28474/7290), [Estimating b1x1+b2x2 instead of b1x1+b2x2+b3x3](http://stats.stackexchange.com/q/58709/7290), [How can a regression be significant yet all predictors be non-significant?](http://stats.stackexchange.com/q/14500/7290), & [Significant t-test vs non-significant F statistic](http://stats.stackexchange.com/q/24720/7290). You might also read some of the threads on the site categorized under the [tag:endogeneity] tag.
Hi Renee and welcome. Please have a look through the [faq](http://stats.stackexchange.com/faq), especially the early sections on how to ask questions. CrossValidated relies on a question-and-answer format, but this question doesn't appear to contain any actual questions. Please try to formulate specific questions and edit your question accordingly so that we can understand how to help.
Consider cropping your images with the relevant parts, before posting. It improves the readability. In this case, for example, there was no need to add the blank parts (page number 3 has a huge one). Tks.
@ChrisSimokat: Maybe I'm missing something but I thought single hidden layer does not equal "single layer perceptron", no?
I think you are asking two very distinct questions here; (1) What are other sites? (2) How do you feel about hosts profiting from the competition ... If you're interested in answers to the latter question, I'd ask a separate question on this topic, and remove it from this question. Also, part of me thinks that this question should be converted into a voting style community wiki question (i.e., what are the major predictive modelling competition hosts, one host per answer, style).
This will not be good, because ranking strips the relative size information from the set of presented elements. for example:
"{1,2,3,100000} -> 3 is chosen" is a strong evidence against the hypothesis 
while "{1,2,3,4} -> 3 is chosen" is not as strong.
I've had a look at the expectation expression, and I fail to get it equal to $\\int_0^{\\infty} e^{-x^2} \\text{d}x$. Could you possibly demonstrate the working for that so that we can check it?
From what I read in the link, by "practical significance" the author tries to make essentially the same distinction with the one I attempted to make in my answer.
Just to clarify: by "P(1 Blue)" you must mean "probability of exactly one blue and *no other color appears exactly once*," etc.  Otherwise the equation is not correct.  Similarly in the second sentence you must restrict to combinations of all other colors where none of them appear exactly once.
Make sure to check out the questions with the tags on this forum about visualization, http://stats.stackexchange.com/questions/tagged/data-visualization and http://stats.stackexchange.com/questions/tagged/interactive-visualization .
Thanks for the explanation Matt! To be honest it is my first time hearing the ensemble method but it sound like something I've been thinking about, I'll look into it later today and get back to you.
@tomka I disagree with the [tag:self-study] tag in this case and so have removed it.  This question deals with actual data and concerns a genuine problem, not just a routine textbook situation.  The criteria for the [tag:self-study] tag are not whether the question originates with classroom work but rather concern the nature of the question itself.  Please visit the meta threads http://meta.stats.stackexchange.com/questions/1904 and http://meta.stats.stackexchange.com/questions/1172 for more information or to discuss this.
That is forensic statistics at its best! A big +1.
Now in fourth edition.
Although this is really an R question rather than a stats question (and therefore probably belongs on stackoverflow)...  could you post your code?  In R, there are often multiple opportunities to obtain great runtime performance improvements, and w/o seeing the code, it's hard to tell which, if any, might apply.
You have 12 observations and you are fitting 12 parameters. All the degrees of freedom in your data have been fitted, leaving none for the residual. "*I don't know how else I could get that info*" ... some degree of replication would do it - even if only a few of the values had replications it would be a start. Failing that, you're going to have to compromise on the desire to reasonably reliably estimate all possible interactions (and in particular to estimate their standard errors).
check out `with` and `within` for cousins to `transform`. plyr adds `summarize` to the mix as well.
I agree with your intuition on the learning rate, and will definitely try some fourier based preprocessing, but my real concern lies with the background subtraction, since leveraging my existing representation of it seems like a plausible way to proceed. All I can think of with in this regard is to generate an image based on the current weights (i.e. the static background) and subtract that from the current input image. I think this would work, but is completely ad hoc, so if you know of people trying to do anything like this in the literature, that would be great.
1) It's simple logic. Call the "not in subset" values "C". If B has the same distribution as C (the null in the test) and B obviously has the same distribution as itself(!), then B has the same distribution as B+C (i.e. A).  2) zero frequencies aren't missing in the usual sense; this should cause no problems, though if the expected values were small you might have to modify the test slightly (this presents little difficulty).  3) correct - the chi-square test itself computes appropriate 'scaling' from the table margin.
+1, This is a perfectly viable approach. I was actually thinking about using a conjugate prior above. Another hack would be to start everyone off with a fixed number of good & bad posts, to initially bias them towards non-spammer. This isn't exactly valid / ideal, but quite similar to the Bayesian approach & easy for anyone to implement.
+1 and btw CS is already in the beta stage :)
Thank you gung!
In addition to the answers below: [Here](http://stats.stackexchange.com/questions/26762/how-to-do-logistic-regression-in-r-when-outcome-is-fractional) is another post dealing with this question.
@Alex, it may econometrics. It does not in sociology. The term "independent variable" is terrible by itself, though, as nothing is ever independent in social sciences. I always say, "explanatory variables".
(+1) Nice, comprehensive, down-to-earth explanation. The analogies you draw make the concepts clear.
@Henry If you can recover all the original data from the polish, then how is "information" not "used"?  BTW, median polish does not behave as you seem to describe: its residuals are the differences in *values*, not in ranks, of the data.
The likelihood is [a function of the parameters,](http://stats.stackexchange.com/questions/2641), so notation like "$p(w)$" clearly does not refer to a likelihood. Unless a distribution is assumed for $f$, though, "$p(y|f)$" is not a conditional probability density, either: it merely refers to the probability density of $y$ as it depends on the parameters $f$.  By assuming $W=Y-Hf$ has a Gaussian distribution, all you have to do is plug $y-Hf$ into the formula for a (multivariate) Gaussian density. Fixing $y$, $H$, and $C_{ww}$, it becomes a function of $f$: in *that* sense it's a likelihood.
I think the answers we have so far are very good, but I'd like to get some more votes and maybe a few more answers, so I'm placing a bounty.  Please vote!  And if you have another way of explaining it, post it - more perspectives are always welcome. (for some reason, this comment didn't go through when I placed it originally...)
Hmmm...How would you interpret the *negative* confidence that arises if, perchance, $5! \\alpha_1 \\times \\cdots \\times \\alpha_5 \\gt 1$? (This will happen if, for instance, all five $x$'s exceed $-0.3\\sigma$, which occurs almost $9$% of the time by chance.)
@Wayne : these are all types  of news items. for instance , number of political, cultural, etc news . so i have 5 categories of different types of news and and their values for 3 decades
I feel the question isn't clear enough. Please show how you do both your calculations with a concrete example.
Look into survival analysis. Browse a bit and then you can form a more specific question.
+1 especially for making the explicit connection with extreme value theory at the end (and the link to a very clear and useful reference).
(+1) Excellent point.  But note that "skewness" does not necessarily mean the definition in terms of the third central moment.  It can be (and has been) defined in terms of the difference between the mean and median.  Although that would seem to reduce this question to a tautology, it suggests that what the OP desires is an explanation for why an *intuitively* "skewed" distribution will tend to have its mean exceeding it median or, perhaps, why anyone would choose to call the difference between mean and median the "skewness" and attempt to relate it to visual characteristics of a histogram.
+1 This is a well-articulated, well thought out question, Ankush.  I hope you get some great responses.  Welcome to our site!
Please, don't forget to register your account. It's already your 5th question, and it will prove helpful for tracking responses to your questions (you'll get system notification, etc.), and *voting*. Also, remember to accept responses when you feel they are appropriate.
Right. Show the means and their SEs or CIs separately from the comparisons. For an easy example of widely differing SEs, think about the cell means for A*B, where A is between-subjects and B is within-subjects. If the design is balanced, then all these means have the same SE but the comparisons of B holding A fixed will have lower SE than comparisons of A holding B fixed.
I still don't get it. What could one use the ROC graph for if he only needs the AUC to really evaluate the performance?
Thanks. Can you better explain here also so we have an answer here? Moreover, how do I calculate the p-value even if not totally accepted as valid?
$\\boldsymbol\\Sigma$ is the (within-class) covariances between the _discriminants_, not between the original variables, $\\bf S_w$. When discriminant scores are computed by the formula at the footnote of http://stats.stackexchange.com/a/48859/3277, pooled $\\boldsymbol\\Sigma$ is identity.
@bastian I used AUTOBOX (http://www.autobox.com) to do the analysis. If you wish you can cintact me at my email address so I can help further.
Thanks! I do not know the distribution of my data either and therefore I chose to use the Wilcoxon Signed Rank Test as it makes no assumption on the underlying data sample distribution.
(+1). Thanks for having taken the time to put the answer.
This sounds like a generalization of the hat problem. Check out the Wikipedia article on derangement.
Thanks for this, shows that for my purposes the calculation method doesn't matter much!
I don't understand what the "i.e. not model accuracy" is doing in 2. It doesn't seem related to anything. Can you clarify or delete that?
I'm afraid my question is not clear enough. The problem is not to approximate a critical value using simulations. The problem is that the distribution of $t$ at the boundary of $H_0$ depends on the model parameter, say $\\theta$. Hence the critical value $C=C(\\theta)$ depends on $\\theta$ which is unusual in hypothesis testing, and I use $C(\\hat\\theta)$ as the critical value.
take the logs of the megapixels? This would reduce the 41mp camera the most, while reducing the others ones less.
The answer is edited.
Again, it would help if you gave more of an indication of what your dataset actually looks like...
Hmm. That was 20 years ago.  Even if you are a student doing a school assignment, you might want to see if the rates are still that different.  Perhaps choosing 1994 was "capitalizing on chance" (looking at a difference that is large for one year due to chance factors).  You might plot the death rates in a way that lets you compare countries and also time periods.
(-1/+1): 1. The OP did explicitly say that this situation arises pretty frequently. 2. You wrote: *If you roll your own algorithm, you could exclude this class from the algorithm and just throw the nonfitting observations 'in the garbage'* How to determine non-fitting observations during application of the classifier ? If the garbage class is known, what is the advantage in comparison to keep the garbage class 3. I think with "confidence" the OP referenced to "confidence of a single decision" not total classifier quality.