tttjjj/elt_bench huggingface dataset:
There is a potential multiple comparisons issue here. Why not just do a logistic regression with 2 dummy codes for B & C?
Let $y_i = \\log(1-x_i) - \\log(x_i)$.  Then the region you found is the set $\\{(y_i)\\ |\\ \\sum_{i=1}^n y_i \\le k\\}$.  In vector notation, writing $y=(y_1, y_2, \\ldots, y_n)$ and $1=(1, 1, \\ldots, 1)$, it can be abbreviated $1\\cdot y\\le k$.  It's a half-space: the multidimensional equivalent of an unbounded interval.  That's about as simple as anything ever gets.
A variable called farmers mark is a continuous variable.Just like we get marks out of 100 for a exam.But you know using exam marks we can give different grades for different mark regions.So I did same thing to variable called "farmers mark" .
Yes, or say it more positively: The Wilcoxon-Mann-Whitney 2-sample test is among the most robust of tests.  Having fewer assumptions than parametric tests, the Wilcoxon test is less likely than the $t$-test to be affected by assumptions not holding.  The Wilcoxon test doesn't assume that the two distributions have the same shape; it assumes that the two distributions, whatever they are, are connected by the proportional odds assumption.
I think the key to resolving this is the implicit criterion that each mixture component has enough probability to guarantee that it contributes a large number (10 or more, e.g.) of the data.  Then isolated clusters (of one to nine points) identified by fitting a mixture model would violate this assumption and constitute either "outliers" (for extreme values) or "inliers" (for non-extreme values).
Can you elaborate more on how did u use the z-score to sort the variables?
When compared to methods like principal components analysis and factor analysis, I've always viewed the methods of clustering I am most familiar with (model based) as a way to maintain and exploit the dimensionality of the data.
Your concern about ML small sample bias is not justified given your number of observations.
Note that t and F distributions are functions of random variables from exponential families.  Thus there is hope of finding conditionally conjugate priors - andrew gelman's folded t distributions for variance components in multilevel modelling is one example.
Thanks for the reply. Unfortunately what I'm really interested in is using more automated/exploratory methods of model selection that don't lend themselves to first selecting a reasonable set of candidate models.
Good catch! (+1) The rounding issue is only part of the problem, however. Because of the rounding, all eigenvalues have absolute values less than $1$.  (The largest is approximately $0.997275$.)  Thus the limiting distribution will be $(0,0,0)$ (but is approached very slowly: even after 1000 iterations the largest e.v. is still around $0.065$). Instead, the O.P.'s code (whatever it might be doing, which is unclear) appears to be *diverging.*
Thank you, sir. This paper http://web.mit.edu/rudin/www/BDNVWorking14.pdf is using the Wilcoxon's Rank Sum test for pair-wise performance evaluation. And I have read some papers on Wilcoxon's Ranks Sum test and Wilcoxon's Signed Rank test. It seems both are valid for paired tests, and the Wilcoxon's Rank Sum test seems to be the paired t-test but with arbitrary data distribution instead of normal distribution. Thanks again!
CV is a *relative* measure of variation, by definition. It gives nonsensical results for any negative mean (you can't interpret a negative amount of dispersion or spread). For positive means, it makes a given amount of spread look much larger when the mean is small. When this is wanted, what you're doing is effectively equivalent to comparing your data on a logarithmic scale--and that makes no sense whenever *any* of the data could be zero or negative. It's possible your data might need some kind of re-expression to allow good comparisons of variability; it depends on how they are generated.
How many subjects are available? Did you observe ceiling/floor effects on your items (responses distribution shifted toward high or low values) or any possible inconsistent responses?
Having Mean=Median=Mode is _not_ the same as having a [normal distribution](http://en.wikipedia.org/wiki/Normal_distribution)! I expect you to get very small p-values, because your situation is extreme (there are way more overlapped items than expected by chance - as also indicated by @whuber). Typically this is reported (in research papers) as "p-value << 0.001" or similar.
@btown the margin is too small to contain it!
1. by "feature counts" do you mean "example counts" ? Features refer to the variables/columnes as far as I know meanwhile recall is linked to the number of rows/examples. 2. Adjusting the threshold in order to optimize a metric with respect to a cost matrix is not uncommon. So what is an example for a method which "just adjusts/changes the classification threshold" (the type of method you compare your approach to) ?
@guest  I had hoped to avoid too much detail, however, please see additional details added to question.
I'll accept this answer because it was the first one. But many thanks to all!
I hope I did not mangle the eqns while converting them to latex. You can use latex on this site. See: http://meta.stats.stackexchange.com/questions/218/tex-processing-for-stats Could you also please clean up the question a bit. For example, the first line is a bit odd, what do you mean by " I know the value of $y_{ij}$, but it could be more than that value." etc.
Sort of related. Random forests have been used. Google will give you the best info. But also mentioned here: http://stats.stackexchange.com/questions/72370/how-to-perform-unsupervised-random-forest-classification-using-breimans-code
Why can't you include "Treatment" in your model?
Back when I wrote this comment, I used `REGRESSION` after log-transforming the data. Since then I've written a more sophisticated version that uses `GLM`. I deal with light emission measurements and my testing suggested gamma regression with a log-link was the least prone to runaway uncertainty on the parameters. For most of my real data, the answers from using normal, negative-binomial, and gamma with log-link were all really similar (at least to the precision I needed)
Should this be posted in the econ forum? Too bad you can't cross post.
ANOVA (as initially tagged) won't be of any help, while [log-linear models](http://en.wikipedia.org/wiki/Loglinear_analysis) or techniques that consider [multinomial](http://en.wikipedia.org/wiki/Multinomial_logit) outcomes might be appropriate here. Could say more on your design, sample size, etc.?
Why do you want to detect outliers? The reason might guide the criterion you use to determine whether a page is an outlier or not.
I would look closely at the package, it is very versatile.
+1 for strong emphasis on the value of robust summaries. See also other threads on this site.
and `spline` does not work why?
@AdamRyczkowski, I wholeheartedly agree with your two comments. I just argue that the difficulty in doing sound calculations for this particularly small dataset seems futile as there are only 3 iPads with crashes. Poisson Regression is definitely a better choice than logistic regression, if we had an adequately sized sample. I read the question differently and focused on the iPads themselves rather than the number of crashes. Again, due to the small sample size, only 3 crashing iPods does not provide us with much variability.
Thank you @douglas-zare for your comments! I have fixed the wrong summation in the question. For the computation complexity, yes I need to find a way to calculate the expectation of all "hits" from the first draw to the last, which is the reason that I want to "start from scratch" to calculate the probability for each "color match".
"The idea is the following insight: Suppose that we choose k such that the above constraint is satisfied then it immediately follows that:
                                          ∑i(di−lvi)2=∑i(d−kvi)2+(k−l)2∑iv2i                                                      "                                                                Could you please show the intermediate steps? I really cannot see how that equation follows immediately from satisfying the constraint. Sorry for the inconvenience caused.
What settings did you use? Depending on input parameters mileage can vary.
@Hempo You know you could email the author and ask him...
Thanks, @whuber. Stat, please check my update (stolen from [this answer](http://stackoverflow.com/questions/12450007/generating-samples-from-a-two-gaussian-mixture-in-r-code-given-in-matlab)). It is numerical, but it may be of some help. Cheers.
If you could say a little about what you understand thus far, & what parts you're confused about, that would help.
I derive standard survival models in notes [here](http://cgibbons.us/courses/are210/ProbExamplesSlides.pdf).
+1 for the meaningful sort and suffixes - but can you expand on why searching and filtering names requires underscores?  I can search for "camel" in "camelCase" just as easily as "camel_case".  Also, I personally think that verbosity is definitely an issue if you're working with specific variables (rather than classes of variables) - this_is_a_variable_name_about_40_chars_long and I really don't want to type that out all the time...
He thanks for the answer. No this is no paired dat. I choose the Wilcoxon test for multiple comparison because some statistic sites on the internet showed this as the post-hoc test to Kruskal-Wallis. There were several examples using R with the command pairwise.wilcox.test with different options for correction. I choose one correction because I didnt know about their strengths and weaknesses. But even I choose another one like "holm" this does not make any difference for the results. So you might be right when u say the sample sizes are to small for good pairwise comparisons.
@Druss2k  The point you raise is a good one.  Soemtimes these constraints can be justified and sometimes they can't.  It is mainly a simple way to pick a solution out of many possible ones, akin to picking a specific generalized inverse for a singular nxn matrix. I mentioned it because you asked for ways to obtain identifiability.
Have you looked into Logistic Regression?
What *exactly* are you trying to do? Why don't you start by parsing out what specifically you are trying to accomplish with your analysis, and we'll work from there instead of getting tied up in technical semantics like the definition of likelihood.
As micans points out, the Rand Index has quadratic behaviour, so it's more sensitive to changes in containment than the entropy function, which is close to linear.
I initially thought chi-squared test on the counts (actually, Fisher's exact test since one count is 0). The result is a significant association between Behavior and Child. However, I don't know how to go from that result to saying that females with vs. without children differ in any of the individual behaviors. From your example, let's say that I can see that N11 > N21 when converted to proportion (where each row sum to 1). How can I say that's a significant difference when there are other proportion pairs N1* and N2* that could have accounted for the association instead?
In theory, sure. In practice, I'm just looking for cut points between the distributions.
Sounds like either [Principal Component Analysis](http://en.wikipedia.org/wiki/Principal_component_analysis) or some form of [partial least squares](http://en.wikipedia.org/wiki/Partial_least_squares_regression) model might be getting toward what you want.
That's on another machine, but I can put some snippets up here when I get home.  Thanks, Cam!
@JEquihua we should move this discussion to the chatroom. Look for the room I just created on the sidebar ->
This question is answered in many threads here, covering many similar applications (regression, logistic regression, covariates, dependent variables, etc.).  They are easier to uncover by including "log" in the search terms: [zero regression log](http://stats.stackexchange.com/search?tab=votes&q=Zero%20regression%20log).
Ah, O.K., I see.  Then in that case, I think it is just the overlap in probabilities that you will want.  Since you're already doing a numerical simulation anyway, you can simulate this part easily as well, by simply taking random draws from each of the sample distributions that you have pictured, and counting the relative frequency (over many experiments) with which load exceeds capacity.
While your comment about $R^2$ being worse than $BIC,AIC$ is useful in general, it actually makes no difference unless you are comparing models of different sizes.  The OP clearly states that they are only interest in $8$ variable models, so $BIC$ and $AIC$ will revert back to choosing the model with the highest likelihood.  This is equivalent to $R^2$ fitting.
@chris: no, I did not test the significance of the model. I only tried to replicate the model from ir.lib.sfu.ca/retrieve/4269/etd2703.pdf page 44. But the paper doesn't give any R^2 value. Is it possible that the input for independent variable for regression running in excel was not only 1 or 0, but they are multiplied by something? When I tried to multiply them by diff between jan-non jan I got better R^2  at 0.11. But now the coeff size for a1 become too large at 0.9, while coeff a0 only 0.00005.
Is there any reason that a test of difference in the observed means is not a suitable answer to this question?
Figuring out survey statistics is a long journey. It took me probably two or three years to bring myself to the concept of randomization distribution. If you want to finish your transportation project within three months or so, that's hardly an option for you. In my own microsimulations, I used the existing sample data as representative agents with sufficiently rich data to figure out their behavior, and I calibrated my weights to bring my sample up to the known population characteristics.
@Corone Despite that there potentially is a simple elegant expression for this in terms of the common distribution $F$, because $\\sum(X_i - X_{1:n}) = \\sum(X_{i:n}-X_{1:n})$ and the joint distribution of the order statistics $X_{i:n}$ has a closed expression in terms of $F$.
Suppose you were to address a collection of $n$ instances of $\\mathcal{P}$ and could choose either algorithm to do it.  How would you compare the computational efforts then?
This is really two completely different questions. The first one concerns fitting a time series model of counted *representative* data. The second one concerns a series of irregularly collected convenience datasets which likely are subject to heavy response bias. Please pick one issue of concern--the type of data (count or average), how they are collected (randomly or by convenience), or regularity of sampling--and edit this question to focus on that issue alone.
Could you be a bit more specific? Are you having a problem with the test? Implementing the test in R? Inputting the data into R? Explaining what your code does line by line?
When you write $\\min (x - (y_1,y_2,\\ldots,y_m))$ is this intended to mean the same thing as $\\min_{1 \\leq i \\leq m} (x - y_i)$? It would seem so, but I wanted to be sure since the former notation is a little unconventional. Also, if you intended to have absolute values as per your response to @Daniel below, I'd recommend you edit the question accordingly. Cheers.
+1 - I'd also suggest the work and books of [Noel Cressie](http://www.stat.osu.edu/~ncressie/) - especially his newer one on spatio-temporal data. That should be enough reading for a few years I would reckon for most people.
@DilipSarwate indeed, in that case, the odds are even.
Clark, so you are saying it is not a good idea to solve the first problem with math analysis?
Stata has extensive built-in facilities for indicating missing values (and even differentiating types of missingness), so creating a dummy for this wouldn't be necessary.
is right. But it may be that one coding or the other is more intuitively appealing. Your factor should be measuring some latent variable. For many of these, one direction will be "more" and the other "less". In these cases, it's usually easier to have higher factor scores relate to "more" of the latent variable
Thanks for the edits.  *Statistical* [standardization](http://en.wikipedia.org/wiki/Standard_score) changes the Euclidean distance to make the standard deviation of responses into the basic unit of measurement.  E.g., if responses on one scale typically range from 2-4 while those on another scale range from 1-5, then one unit on the first will be the same distance as two units on the second.  It amounts to an automatic way of putting different attributes "on the same footing."  It's worth adding to the methods under consideration here.
Thanks for inquiring, Joel. Sample size is 120. Yes, all 7 scales values are used, and the distribution would be considered somewhat normal. Any further thoughts?
Interesting question. I don't know the answer, but the idea that some degrees of freedom should be lost makes sense. If you hadn't seen it already, this answer by @whuber should be thought-provoking: [how-to-understand-degrees-of-freedom](http://stats.stackexchange.com/questions/16921//17148#17148). It seems to me that some simulation studies should enable you to get a toehold here, at least for some specific cases.
(ctd) ... (2) My question about how it arises was [asked for a reason](http://stats.stackexchange.com/tags/self-study/info) - the `self-study` tag applies for standard bookwork like this even when it's not 'homework', even when it's not assigned, even when it's not looked at by anyone else. (Indeed, even when it's not directly associated with any subject, but that's a relatively rare situation.)
[That post](http://stats.stackexchange.com/q/38709/3277) could be of your interest.
I considered this route, but I would rather look for a general tendency towards likely (so, "somewhat likely" counts more than "somewhat unlikely").
Although somebody is repeatedly flagging the [preceding comment](http://stats.stackexchange.com/questions/80412/chi-square-test-for-survey-data/80413#comment158001_80413), I see nothing in the least offensive or incorrect about it and therefore have been dismissing the flags, which I will continue to do if they recur.
It isn't quite true that samples have to be random. Rather, they have to be *representative* of the population you want to draw an inference about. Having a random sample of the population in question implies representativeness, but it is possible for a sample to be representative even if it is not random (of course you have no guarantee & it would be hard to substantiate, but it is possible). For more on this, it may help to read my answer here: [identifying-the-population-and-samples-in-a-study](http://stats.stackexchange.com/questions/31488//32149#32149).
If I understand you correctly, this would explain why the model would predict "large" values, not why the model would be comprised of "large" coefficients.
Assuming A and B are uncorrelated, use the variance propagation formula on Wikipedia at http://en.wikipedia.org/wiki/Fieller%27s_theorem#Case_1 .  Because your CIs are so small, we know the sampling distributions are approximately normal and the sample sizes are fairly large.  Therefore you can back-calculate Var(A) and Var(B) from your CIs in the usual way, for input in the variance propagation formula.  (BTW, your edit is mysterious.  Perhaps you are confusing the coverage of a CI with the interval itself?)
+1 for the really nice idea of checking against quantiles.  A good way to fit tails is to fit the 1/4, 1/8, 1/16, ... quantiles.  Note that this is not really half-normal: the mode is slightly positive.  Also, it's short tailed compared to the normal.  You can correct that with a Box-Cox transformation or something similar and get a much closer fit overall.
@Michael Chernick Agreed.
Correction: I believe the problem is not sample size, as originally posed, but rather unequal sample sizes and unequal variances.  I believe what happens is when I perform a large number of tests, some feature may correlate by chance with the feature of interest.  In this case, the size of the matching samples is small, and the non-matching is large.  Also, the variance in the matching samples is small.  Apparently, my z-test fails in this scenario.  What is the best method to screen for these random occurrences?
user Stask has a nice answer to [exactly this question](http://stats.stackexchange.com/a/38430/1036).
Elimination of variables in this way will result in major biases.  The approach is not driven by any statistical principles I can think of.  Use a principled approach, i.e., either pre-specify the model and use only methods blinded to Y (the response variable) to select the Xs, or use something like glmnet.  I don't see how Winsorization applies here either.  Variable selection without shrinkage (direct penalty for selection and estimation) is a disaster.  The original set of functions you specified for the predictor X's doesn't make as much sense as using spline bases, usually.
Welcome to the site, Thomas! You might consider editing your question to give some indication of what piqued your interest initially. Generally, the more information, the better. For example, stating that you'd read Prof. Shalizi's note and it made you wonder about X not only preempts answers that would suggest precisely that, but also shows more clearly your train of thought, which tends to elicit better answers. :) (For example, have you read M. Mitzenmacher's review article in *Internet Mathematics*?)
Perhaps worth emphasizing that using the Bonferroni correction on the two pre-selected t-tests would be a conservative way to control family-wise Type I error, & that Dunnett's test has higher power.
What are the two sample sizes?  The method of data splitting to validate a model requires amazingly large sample sizes, as opposed to resampling (strong internal validation).  You can do clustered resampling, e.g., cluster bootstrap.
@Daryl feel free to accept the answer. :)
K-medoids is a k-means variation that works with *any* distance metric. Instead of using the mean, it uses the medoid, the most central element of the cluster. Then you have guaranteed convergence again, because both steps optimize the same quantity, the sum-of-distances-to-cluster-medoid.
For "absurdly" large values, you won't do better than using the upper bound $\\Pr(Z > z) \\leq (z \\sqrt{2\\pi})^{-1} e^{-z^2/2}$ for double-precision floating point. That approximation and others are discussed elsewhere on stats.SE.
look at the answer to [this](http://stats.stackexchange.com/questions/71381/estimating-parameters-in-multivariate-classification-resulting-zero-determinant/71383#71383) question.
There are many things that seem imprecise or incorrect in your question, maybe it could be useful to step back a little and explain what you are trying to learn from all this.
@ttphns I think it completely depends on what package you use. Often these terms are completely interchangeable. Consider the usage of "loadings" here: http://www.utdallas.edu/~herve/abdi-awPCA2010.pdf. Also reference the R function `prcomp` where the loadings matrix is just the matrix whose columns are unit eigenvectors. I think you are being unnecessarily technical, and in most cases I believe these terms are used interchangeably.
Thanks for your answers, Roger. It's very nice to have your expertise on these questions.
I think you'll need to provide some detail to get an answer, if there is one. In general, if you want to know how well a sample statistic performs as an estimator of a population parameter, you need to know how the sample is taken from the population.