tttjjj/elt_bench huggingface dataset:
Note that this criterion is designed to be used with k-means. On clusterings obtained by different algorithms, it may be inappropriate (in particular for density based clustering algorithms)
possible duplicate of [Including the interaction but not the main effects in a model](http://stats.stackexchange.com/questions/11009/including-the-interaction-but-not-the-main-effects-in-a-model)
This is a really nice answer, and I'm pleased to help make it recognized as such. +1
No it is not a problem that method B was not available. This fact does not affect any inference related to survival times. You are not trying to explain "why they chose A over B", but _given the choice_ (for whatever reason), what was the survival time.
There must be something in chemometrics books; the only man I know that uses LOO is also doing it.
(+1) Also notice that for $n$ large, $p \\approx e^{-2}$.
You can address this at the outset by defining your terminology. I'm sympathetic: your case is like mine (environmental sampling), where it's convenient to say that a "dataset" consists of "observations" of individual physical "samples," each of which is subjected to one or more analytical "tests" that typically yield multiple "measurements."  Notice how the statistical terminology is largely dropped in favor of words familiar to the reader: the burden of good communication lies first with the writer, who should reach out to the audience and build on their conventions and understandings.
Is this intended as an answer, @EconStats? It seems more like a comment. Remember that CV is not intended as a discussion forum. Would you mind making the *answer* w/i this post more salient?
It kind of depends on what kind of information you would find relevant. #data dredging.  How about apply zipf's law to estimate the total number of words in the mobile app vocabulary (including the words not in your data)?
@ttnphns, indeed, thanks ;-)
Common choices include - medians, modes, proportions or cumulative proportions in each group
If I interpret this question correctly, the discussion in Rice, [*A Consensus Combined P-Value Test and the Family-wide Significance of Component Tests*](http://www.lifesci.ucsb.edu/eemb/faculty/rice/publications/pdf/23.pdf) (Biometrics 1990) explains this problem: see p. 304. The paper offers a solution.
It looks like that link gives examples of _univariate_ regression models using robust ($M$) estimation, but I think the question is about $M$-estimation for a multivariate dependent variable.
I agree with Gung. But if you tell us more about what these groups are, what you are trying to find out, and so on, we might be able to suggest a better analysis.
Thanks for your response. Though, I found neither circles plot nor colored scatterplot was easy to interpret. However, I suddenly realize that my question is about the interaction between the two continuous variable X1 and X2. So I Googled and found this [link](http://www.ats.ucla.edu/stat/sas/faq/spplot/reg_int_cont.htm) seemed to be helpful to me.
You only have 5 data points. Changing 2 of them is changing 40% of your data! It looks like you have zero inflation, but with only 5 data points, it's impossible to tell. And pretty much any model will have overfitting issues.
You could take a look at this online tool to probe interactions in multilevel models: http://www.quantpsy.org/interact/hlm2.htm. Among other things, it does provide you with simple slopes.
I do not know as much as I would like about dendrograms.  My first question is "how do you account for variability?".  The graph forms a visual "order statistic" but what is the uncertainty in the value?  What is the measure that indicates that your dendrogram is properly generalized?
(+1) Good to know, thx.
Thanks, asymptotics can be a little tricky for the uninitiated.
After a second thought - what I am looking at here is ordered, so looking at three levels as categorical would not fit here...
First things first: if you found a particular answer helpful, it's customary to upvote it.  If you found an answer definitive, you may, at your discretion, award it "best answer" status, though that's really up to your own discretion.
are there interactions in your model? I've only used marginsplot in the setting of interactions. Otherwise I think (but not sure) it should be linear.
(-1) There is nothing in the problem that says $P(\\mbox{weatherman says rain}|\\mbox{rain})=.9$. We only know the weatherman is correct 90% of the time, i.e. $P(\\mbox{weather man says rain and it rains}) + P(\\mbox{weatherman says no rain and it does not rain}) = .9$.
Glen_b, you are great! I tried to delete my post at Quante SE,but I didn't find the way.
For the second case, that is easy to see so no problems there. The problem is for the first case, because we are also replacing $X_n$ with its upper bound. I find that somewhat hard to digest. I'll think about it some more. Thank you.
@juampa I used logical regression through http://statpages.org/logistic.html result was Avg:210 SD: 189. When I use one of these values, classification success ratio does not become best. For example I get better results if I set threshold to 350. Am I doing something or does this algorithm not fit to my situation?
There's no reason you can't use categorical features in logistic regression and any standard statistical software should handle it. I also think you are confusing the terms linear and continuous
For each datapoint, the probability of not being chosen in a given bootstrap sample is $P(A)= (1 - \\frac{1}{N})^N$, where $N$ is the number of observations. When $N$ tends to infinity, this probability converges to $e^{-1}$.
Have you tried working out what the covariance of $X^{\\star}$ will be?
+1 Brilliant response.
Sorry, my comment was too brief. The standard deviation of two uncorrelated r.v.'s with variances $\\sigma^2$ and $\\theta^2$ is $\\sqrt{\\sigma^2 + \\theta^2}$ and **not** $\\sqrt{\\sigma^2} + \\sqrt{\\theta^2} = \\sigma + \\theta$
But the estimated loglikelihoods are very different (presumably by some constant), so the different methods should not be mixed.
Although this question does not seem to call for a R solution, it remains good practice to flag that you are using R.
Re (B): The formulae you quote for the variances of the estimators assume $X$ is *fixed*: they do not account for the random variation in $X$ itself. A formula for the full variance could be a function of only $\\sigma_X, \\sigma_Y,$ and $\\rho$. Since this distinction (between conditional and unconditional variances) is fundamental, it would help for you to describe your simulation in more detail to make it clear what exactly your are simulating as random and what you are keeping fixed.
Assuming a geometric distribution is assuming that the failures are independent. In other words, on any trial the device fails with probability p, independent of the other trials (like tossing a coin). This is a reasonable starting point, but if the failure is due to something like a memory leak, then the longer the device runs, the more likely it is to fail, and the assumption of independent failures would be invalid.
That's just an example - it might have a median that is much smaller, on the order of 200 (it depends on how I partition the data). That would preclude using a normal distribution, right?
I think OP needs the MLEs for $d$ and $\\alpha$ given data "to the left of the line."
I am doing a two way anova.
If $y$ was itself conditioned on some $x$ then wouldn't this fall exactly out of the simpler version?
One additional consideration, since we're getting into this: $log(\\frac{p_1}{p_0}) = log(p_1)-log(p_0)$ which is clearly different from $p_1 - p_0$ and it is more different precisely when the p are small - that is, risk is slight. But I was trying to keep my first answer ASAP (that's As *simple* as possible!)
How different are we talking?
@Scortchi I agree with the main point you are making, but your example is a straw dog argument. The research problem is more likely to be something like "what can we say about police dogs trained by this organisation, or in this city?", not in the whole wide world. I wouldn't rule out the idea that sampling from such a group can be well done. It's for the researcher to focus on what the precise aim is; we agree on that.
Maybe this post will be helpful to you http://stats.stackexchange.com/questions/68425/how-to-determine-correlation-between-stationary-and-non-stationary-time-series/68592#68592
Do the arithmetic: (1) 100%-97% = 3% of 30,000 is 900.  Therefore you have to check 30,000 - 900 = 29,100 records to be *certain* of finding *at least* 97% of all error records.  (2) Since 6500/30000 is less than one-quarter of all records, then *if those you checked are representative of the whole group,* you have probably found only about a quarter of the errors, not 97% of them.  **Conclusions**: (1) to make significant progress you need to specify an acceptable chance of not finding 97% of the errors and (2) you need to characterize how you selected the 6500 for checking.
+1, thanks so much. As I read your answer, if I don't have domain-specific knowledge to do feature engineering, then I should focus on inventing good methods to do feature learning using the given data, and try different learning algorithms.
This explanation makes sense to me, thanks
Dear Group, I am slightly trying to be bit more specific, that may help learned people like you to understand my question. Suppose, I learnt a model practised the problem with some sample from a verified source, coded it also but browsing other codes I found a totally different interpretation, while my tasks were accomplished in my own exercise, so how to tackle those scenarios? I feel I need to cross check what I know but how to do that? Please suggest. Regards, Subhabrata Banerjee.
I have read that you cannot make a correlation matrix with nominal variables. As six of my independent variables are nominal in nature, how can I find if these variables are correlated in any sense or not? Is it necessary to test for multicollinearity among nominal variables for building a logistic regression model?
I hope you don't mind, but I converted your question into latex. One thing, should $Z_{\\alpha}$ not be $Z_{\\alpha/2}$
If you have a low bias, high variance model, methodologies like bagging can reduce the variance at a slight increase in bias.

If you have high bias, low variance, use a model that's lower bias and higher variance - like a polynomial regression or more generally kernel methods.
Hint: the final digit in the product of any two integers will always equal the product of the final digit in each of the two integers - let me know in the comments if you're still having trouble and I can come back and provide a more comprehensive answer.
You interpret the coefficients as you would in regression, not ANOVA. (i.e. view1 and view2 are dummy-coded variables.) If you're not familiar with the interpretation of regression coefficients, you would need to look into that first.
Hi thanks, algorithms used are Time Series Autoregression, Decision Tree and Neural Network.
+1. What's the URL for the 3rd edition of Bayesian Data Analysis?
@Tristan of course. I am, however, talking about gibbs sampling.
I think we need a bit more information. What precisely is $p_i$: is it the probability of any given word being $i$, or the probability of $i$ occurring in a document? What do you mean by co-occurring: is it $i$ and $j$ occurring at a specific place, or $i$ and $j$ occurring anywhere in a document? Finally, would something like $ijij$ count as 1, 2 or 3 co-occurrences?
Thanks for that - that's a useful method to know...
(3) It's pretty stable, although for smaller $K$ it tends to select 7 to 8 components, which is way too many components for my application. (4) I didn't get this part, what do you mean by 'how does the training (resubstitution) error EVOLVE'?
Does it make sense to scale the dummy variables?
One variable is continuous. The dichotomous variable has the unequal group sizes.
@MichaelChernick The sentence means that e.g. first year I have only 5 subjects having the characteristic (compared with 100 for the global data), second year 6,..., last year 68 (compared with 143) etc. I was thinking that I cannot simply take random samples, so that I need to sample with the same number of subjects  as in the original sub-sample over the years, so that I have bootstrapped samples under the null...
Is this homework? Seems like it. If so, it should have the self-study tag
Maybe I'm wrong here, but shouldn't $x_{0ij}=1$ or maybe some element of a design matrix?  I'm also unsure of the notation $\\beta_{0ij}$, as a simple thought experiment how many parameters would you have if this were true, how many observations?  You might want to post a link to this notation, maybe something has been taken out of context.
@Dougal Likewise, I have had these thoughts as well ;) and was looking for something more definitive-on this problem
Sam thank you very much! Very informative and will help along way!
It would be great if you asked specifically about the particular problem you are having with the particular exercise. I suspect that you are on the verge of the breakthrough and need just a little push to suddenly understand everything. At least for me it is a usual experience when trying to understand some unknown field of science.
Briefly, you only add exclusive probabilities.  You can't pick a red ball *and* a green ball at the first try. (BTW your reasoning is hopelessly optimistic in #2: if she's already passed you over 5 times you've no chance.)
The first model fits a random intercept *and* and random slope for `x` whereas the second model fits only a random intercept and only one global slope for `x`. An excellent post that I think will help you is [this one](http://stats.stackexchange.com/questions/13166/rs-lmer-cheat-sheet). Which model is more appropriate is most likely to be determined by what you want to find out and by your research design.
This is a great answer, thanks. I would have had a hard time coding your algorithm, so the ready to use code is very welcome. I didn't know `R` but still managed to use your functions in less than an hour's work, so that the script takes input from a 20k words dictionary file and write results to a .csv. (this took less than 10 minutes on a mid-range core i5)
More expensive as in slow and memory-intensive.
Furthermore, suitability in this case is less a matter of your data than a matter of the question you want to ask of it.
Oops! Fixed (I hope), thanks!
This question is kind of vague. Perhaps you can post sample data or links for deeper reading.
At first as I was like, "Oh, it doesn't solve error term correlation? I guess that's okay for a first pass." Then I was like, "Oh, crap, it only works for discrete groups?" Dammit.
Please tell us what you mean by $\\sigma$: is it a known standard deviation or an estimated standard deviation? Could you also please clarify the distinction you are making between "z score" (a standard term) and "t score" (not so standard: see http://en.wikipedia.org/wiki/Bone_density#T-score for instance).
Typically identifying whether an observation is an outlier and deserves further attention is based in the particular context/objectives and resource constraints. It is possible a particular cut-off (for any particular metric) will flag too many false positive cases (or just too many cases in general to be useful), which would suggest either increasing the cut-off value (or trying a different metric).
Let's say there are three ways of comparing molecules, numbered 1-3  (where each method determines how close each of the 1000 molecules is to the reference).  Additionally I am picking each of the 1000 molecules as a reference, hence 1000 matrices.  I am interested in determining whether the first way is closer to the second way than it is to the third way, and if the second and the third ways are closer to each other than they are to the first way.  In essence, I am trying to cluster the comparisons by similarity.  The final goal is to find where my comparison method is in this cluster.
This question appears to be off-topic because it is about finding an error in code.
I like AIC but beware that computing AIC on more than 2 pre-specified models results in a multiplicity problem.
There are a variety of measures of association in contingency tables [a few are listed here](http://en.wikipedia.org/wiki/Contingency_table#Measures_of_association), but the suggestion by @PeterFlom of Cohen's kappa is about as good as you will get for this particular case.
These are approximation algos, they are not exact. But they do give a proof of upper bound, so you can at least compare the results with your MC result.
Ok so for each value 5,6,9,5,4 I , so I substitute the x with 5, then with 6 and so on? And on the end I have 5 different plots?
I don't believe unbalanced designs have less power, especially since the poster is interested in interaction. Suppose the effect is much stronger in one group which is naturally "over expressed" in the population. By oversampling the under-represented group who have a smaller effect size, the test of association has less power. Take muscle mass before and after enlistment in the army, stratified by gender, as an example.
John means [this page](http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval)
If you question is strictly about how `predict` should be used (or respecified, in any case), StackOverflow will probably be a better place to ask. You appear to know exactly what you want "statistically". In any case, can't you just construct your *whole sample* model matrix $X_{fixed}$, calculate your $\\hat{\\beta}_{LOOCV}$ from your smaller model and then compute your $M_{LOOCV}$ model's estimate for the excluded subject $K$ as : $X_{fixed}[K,] * \\hat{\\beta}_{LOOCV}$, so there is no random-effect influence? (Or maybe I don't understand what you want so sorry? :) )
I interpret the subscripted alpha as there are 3 different intercepts fitted & the subscripted t as a continuous covariate w/ a different value for each individual / condition combination, but the unsubscripted beta as assuming that all conditions have the same slope for changes in t. That is, this seems to be an ANCOVA, is that right? Given that the conditional distribution of X is stated as normal w/ the mean a function of h & t, what must the distribution of the residuals be? & given that distribution for the residuals, what is the distribution will the sampling distribution of beta have?
+1 This agrees with a normal approximation which gives the probability as 77.0% (the correct probability is 77.3%).
$G^{-1}(F(X))$ where $F(\\cdot)$ is the CDF corresponding to $f(x)$ and $G(\\cdot)$ corresponding to $g(x)$.
@Thomas Andrew Gelman discussed type I and II errors before introducing S and M errors. I think this response is a valid and interesting one (wtr. other well-founded answers) since it allows to go beyond the traditional decision theory framework. I've upvoted this response.
Wow! Thank you and welcome to the community. Just to clarify, McNemar works on discordant matched pairs (?) are such pairs dropped out of clogit? I don't see how id is involved in the calculation of the mcnemar results. Perhaps generating a correlation in these would help elucidate what clogit is doing.
From your description, it seems like you might be confusing the idea between independent and mutually exclusive.
It might help to get a little more information. Is your response variable continuous? Are the 'types' ordered in time?
+1 for Random Forests. Definitely a good classifier to try 1st, as they don't overfit.
It's a gene sequencer.  It is producing counts of how many times the molecule in question was found, and this should correlate to the molecule's concentration in the original sample.  So not explicitly rounding.
your are welcome. Note that right clicking on formula will present you pop-up menu with a choice to see the source.
@julieth - the OP is asking for large n small p (n>>p), not the othe way around.
Also, the p-values you see in the anovas are for the variables that are omitted from one of the two regressions being compared.
Are you sure your lsmeans statement is testing the same hypothesis as the t test?
+1 for your time. I updated my question with the problem I am facing with this formula. I am not able to understand how to tackle this problem when the PMFs contain `0`s. Would you happen to have any comments on how to mitigate this problem?
Stochastic variation is almost never an important concern of a new business. Far more important is the question of whether the basic assumptions about the demand for the product, the intent of the competition, and expertise in producing that product are valid.
What you're describing, in essence, is an [accept reject algorithm](http://en.wikipedia.org/wiki/Rejection_sampling).
@Yuriy, please answer your question. It would help future users with similar problems.
What can be done that maybe you were thinking of is if you have say a 95% confidence interval [A, B] for a parameter m  and you want a 95% confidence interval for f(m) where f is monotone (either nondeceasing or nonincreasing) [F(A), F(B)] will be a 95% confidence interval for f(m) in the nondecreasing case and [F(B), F(A)] will be a 95% confidence interval in the nonincreasing case.
@David Are you sure you want to remove it? I think it should stay because it still may be useful for other people.
@DavidMarx You are right. I already corrected it.
Could you please describe your grid and grid cells more in detail? When you write that there are 500 grid cells "in the region", what does that mean? 
Moreover, is the i you mention in your second formula running between 1 and 3? What does the sum on the grid cells (=1) means? Sorry for the avalanche of questions, but even though you explained a lot of details, the problem seems to be quite complex.
"Divide that variance by 365; this will give you the variance of the annual average." Shouldn't that be 365^2?
Any monotonic transformation of the independent variables should make no difference. But a transformation of the response can make a difference, if it changes the fit measures. So, if you took the log, for example, the result could be different, but if you simply multiply by a constant, it should not be.
Could you please clarify 'follow the same shaped ROC curves'?
Can you provide a reproducible example? Also, I believe it's actually generally recommended to use `lambda.1se` over `lambda.min`
If the MI is large than knowing X could tell you something about the error in your predictions. This information could theoretically be used to improve your predictions, i.e. your neural network is not exploiting all of the information available to it.
Thanks, that was very helpful. I was hoping to avoid rejection algorithms, but this one seems like it might have a decent acceptance ratio if $f(x)$ and $g(x)$ are close. My implementation of a toy example (similar to that in the paper) was actually just four lines of Python code!
There are several related questions you might want to look through. Some are listed on the side margin of this page. [Here](http://stats.stackexchange.com/questions/9664/what-are-examples-where-a-naive-bootstrap-fails) is one regarding when the bootstrap fails and what it means for it to fail.
consider this view - a pmf has only a discrete set of values ALLOWED for probability, whereas pdf has infinite number of values allowed for probability. If we consider this, then P(y|x) is capable of taking infinite values and not discrete values.
Also, by using the ar() function, are you deciding *a priori* that an AR model provides an adequate representation of the data generating mechanism? In other words, does the ar() function rule out the possibility of adding MA, seasonal AR, seasonal MA, and "difference" terms to the model?
Although you provide two examples of entropies, you do not actually apply anything that would be called an "Entropy test." Could you explicitly tell us what that test is and how it works with your two files?
-1 "google blah" answers are useless. If you have to, leave it as a comment, otherwise, provide some actual detail.