tttjjj/elt_bench huggingface dataset:
With only 26 cases to study you might need to worry about overfitting when using more than two or three control variables.
Thanks. Well, it's kind of a bit of both. The statistical issue is "How to do significance tests for comparing sample means with multiple imputation? In particular, how do you compute the variance of the estimate (F-statistic)?". Now, I'm using R/MICE for my multiple imputation needs, so I thought someone would know of a function for it. Alternatively I'd be more than happy with a statistical explanation on how to do it so I can just write the function myself from scratch.
"Electability" is not a statistical term. It's a term used in political reporting (and more often, editorializing). It therefore doesn't have a precise statistical meaning. In *some* specific case it might mean what you guess it means, but we'd need context (i.e. somewhere it's being used) to see that (and even then it might not be completely clear). As such I am not sure this question is sufficiently unambiguous.
@user2662565 Thanks. It can automatically drop variables due to what's called **perfect** multicollinearity, but it if I remember correctly it prints a related message if it does so. If you need to investigate further please paste any `error` messages or `warnings` so that we'll be able to tell what happened. You might also paste the results of `dim(dataframe)` where dataframe is the name of your modeling data object, so that we can know if it was a degrees of freedom problem.
Silhouette needs at least 2 clusters. Try to use Gap statistic to decide whether you should prefer one cluster (= no clusters) or several clusters in the _same_ data set.
What is PCA in discrimination analysis? Please give many more words to describe it. If _n<p_ then there exist only _n_ real (non-zero) principal components in such data.
When you adjust the distribution of the test statistic for the estimation of parameters in this way, it's not a K-S test (even though based on a K-S statistics) - it's a particular type of [Lilliefors test](https://en.wikipedia.org/wiki/Lilliefors_test). This is no longer nonparametric, but one can construct one via simulation for any given distribution. Lilliefors did this specifically for the normal and exponential ... back in the 1960s.
Do I understand correctly that you have a list of items that each person filled out twice: once by marking 1 of 4 ordered levels, & again by marking 1 of 2 response levels?
@doctorate, the igraph package supports this.
@sasuser a<-"2004year"; as.numeric(substr(a,1,4))
Don't yell STATA, just say Stata. This is not an acronym, unlike SPSS or SAS.
This thread may also be of interest to you: [Creating an index of quality from multiple variables to enable rank ordering](http://stats.stackexchange.com/questions/9358/).
Thanks. I do have the numbers and they are essentially constant - the difference between the regression line and each prediction limit ranges from 18.21074 in the middle to 18.24877 at the ends.  So, a slight widening, but very slight.  By the way, @gung, I got the equation that calculates the prediction interval.  It is: `Y-hat +/- sp(Y-hat)`
I am having a difficult time parsing your questions here.  Please describe the context in which you *can* encounter the above problem.  When you train a HMM, you train on an observation sequence, you expect that the test observation sequence to be made up of the same tokens. Different HMMs can definitely have their own sets of symbols, and the can have different symbols.  I suspect, based on these followup questions, that you may need to revisit some fundamentals.
This is exactly the scenario of PU learning, a branch of semi-supervised learning. The following reference is one of my favorites: http://cseweb.ucsd.edu/~elkan/posonly.pdf
Frank Harrell's `rms` package has functions for this task. Fit the model with `fit <- lrm(outcomes ~ X1 + X2 + X3, data=my.data, x=TRUE, y=TRUE)`, then use bootstrap validation with `validate(fit, B=1000)`. The output matrix includes the optimism corrected values, but only shows Somers' $D_{xy}$. However $\\text{AUC} = 0.5 \\cdot D_{xy} + 0.5$.
got it. so the problem then is $h(j)=\\ln(x(j))$ , $\\ln$ is concave, now the transformation $x(j)$ must be at least not more convex than $\\ln$ is concave
+1 It's easy to get blinded by exhortations to not use linear models, etc, on time series because of auto-correlation issues, and to get caught up in ARIMA, DLM, etc, when LM, GLM, etc can be quite powerful with a little caution.
Well, that's true. There is a close relationship the gamma distribution and Poisson processes. But I don't see how that helps me.
I think this might be creeping a tad beyond the '[helpful hints](http://stats.stackexchange.com/tags/self-study/info)' of the guidelines for `self-study` questions. More specifically "*Try to provide explanations that will lead the asker in the correct direction*" and "*use pseudo-code and general descriptions first*" rather than "*a complete solution (or code sample)*"
I've considered this, and I will do it if there's no better alternative.  There are about 60 $\\beta$'s and they are formed from about 30 different $\\gamma$'s.  So it will be long/complicated likelihood function to write out.  I'd prefer to take advantage of an existing framework to simplify the code.
It would help to report what you understand from this `PROC` and where you are stuck.
@Natalie Glad that I could help! Please consider to [accept my answer](https://stackoverflow.com/help/someone-answers) if it was helpful.
I didn't use mlogit in sas or r, but I think the command name is the same. You can implement it yourself using 3 binomial logit models for each outcome then combining them with bayes theorem. It helps to understand how exactly mlogit works, a good exercise.
I am a bit unclear as to what exactly you are asking and how data analysis enters the picture? Please elaborate.
@mycat - I like to use CIs to indicate how precisely a parameter has been estimated.
**Something's not quite right here:** normally, people understand expressions like $E[X^p]$ to be *expectations* of *powers,* but you state explicitly this is not a power.  Furthermore, $E[X_1^p, X_2^q]$ does not have any conventional sense.  Do you seek the expectation of the *vector*-valued expression $(X_1^p, X_2^q)$ or do you want the $(p,q)$ moment, which is the expectation of the *product* $X_1^p X_2^q$?  Finally, it appears you want these moments as *sample statistics,* but for what purpose?  There are many possible estimators for any given moment; choosing one depends on its purpose.
@Joe, thanks. I usually try not to advise people on stuff like this, b/c I don't feel strong enough on the material. I am familiar w/ some of it & so wanted to put down a basic response to be somewhat helpful, however some things I'm less familiar w/: I don't know lme4 well, or Bayesian approaches, & don't know MCMCglmm at all. One thing about GLMM's is that they are so computationally complex that sometimes there are difficulties w/ estimation. I don't know if that's part of what's going on. But I prefer not to go further than my comment, as I wouldn't want to say something that's incorrect.
Don't know how I missed this comment before. I didn't propose 'doubling the answers'.
-1 C-SVC can give approximated probability outputs via Platt scaling.
I've not run logistic regression in Python so I am less familiar than R, but why can't you just set your X variable for rd.fit to multiple columns?  It might help if you share what the output of the tfv.fit looks like.  I would assume you can create categorical variables for the logistic regression from this output, or count variables.  Same for the pagerank variable, I don't see why you can't just include this as an explanatory variable in the regression, possibly transformed.  It might benefit you to read more about how logistic regression works both generally, and specifically in Python.
Hi @MichaelChernick. I am 53.  Belsley wrote "the book" on collinearity. [http://www.amazon.com/Regression-Diagnostics-Identifying-Influential-Collinearity/dp/0471691178/ref=sr_1_1?ie=UTF8&qid=1348752233&sr=8-1&keywords=belsley](Belsley, Kuh and Welsch) covers collinearity but it is an earlier book by just [http://www.amazon.com/Conditioning-Diagnostics-Collinearity-Weak-Regression/dp/0471528897/ref=sr_1_9?ie=UTF8&qid=1348752233&sr=8-9&keywords=belsley](Belsley) that I used for my dissertation, which I started working on in 1996 or so.  I don't know if Cook did anything on colinerity or not
If this answer solves your problem, I'd appreciate it if you could click the little checkmark next to it -- I have a question I want to put a bounty on, so I have a use for the reputation points it will give me.
Thank you for your advice. I'm inclined to do this first, as I know how to do it. Not to suggest that the above reply is any less relevant. I truly appreciate the effort that went into analyzing this situation.
As you can see, the value of $n$ matters when it comes to bias. So it depends by what you mean by "without loss of generality".  A more practical issue is that if the formulas were provided for $n=1$, it would not be necessarily clear how exactly they should look for general $n$. Now they are provided for general $n$ so one can plug in any value of $n$, and see what happens
Note that in this special case finding the UMVU estimator is particularly easy. Indeed writing out the density, we see that we obtain an exponential family with complete sufficient statistic $(\\mathbf Y' \\Sigma^{-1} \\mathbf Y, \\mathbf 1'\\Sigma^{-1} \\mathbf Y)$. All we are left to do is find some unbiased estimator of $\\beta$ and this is guaranteed to be UMVU. Noting already that $\\Sigma^{-1} \\mathbf 1 = (1/2, 1/2, 0, 0)$, we are done!
Thanks, but I know my solution is right !
@Aly: the $n$ refers to the sample size used to estimate the parameters ('x_bar' and 'sigma')....
Seems like you want a Multilevel (eg Hierarchical) Model with the processing time nested in batch size and step. This should (in my limited experience with multilevel models) provide similar results to a multi-stage model, except compacted. As an engineer, I think the multilevel model would be vastly more preferred, as long as you can explain it in layman's terms to the stakeholders. I will leave your question for someone else more knowledgable to answer; just trying to get you on your way. PS- don't confuse Hierarchical Model with Hierarchical Regression... the latter is not of use to you.
No, in finite samples all these approaches are different though asymptotically they should be the same. The Brant test estimates an approximation of the unconstrained model using seperate logistic regression and than performs a Wald test. A comparison of the various methods can be found [here](http://www.maartenbuis.nl/presentations/gsug13.pdf)
simple example, taken from the package author's book:
    library(mgcv) 
    data(trees)
    ct5 <- gam(Volume ~ te(Height,Girth,k=5),family=Gamma(link=log),data=trees)
ct5
    vis.gam(ct5)
    plot(ct5,too.far=0.15)
Are those cumulative incidences or incident rates? Anyway, you could take a look at the `csi` and `iri` commands. They calculate the Attributable fraction among the exposed (which is what you've calculated) together with a CI. For example `csi 22 46 2836109 8011731`
Hi, the thing is I want to implement a spinner similar to the one I saw on this site [link]http://www.nytimes.com/newsgraphics/2014/senate-model[/link], and I saw in the source code it is using normal distribution random generator.

Let me edit my original question, adding snippet of the code from the NYT page.
(+1) Important aside: Make sure you know how your random number generator works before saving the subtraction. Some RNGs will have zero as a valid output while others won't. The consequences could be severe. Likewise some RNGs may admit one as a valid output.
A Google search on [singular value decomposition algorithm](https://www.google.com/search?q=singular+value+decomposition+algorithm) does a fine job of highlighting relevant information.
@Dilip, let me be clear that there is nothing wrong with the question and there is nothing wrong with your answer (+1) (or probability's, either (+1)).  I was simply pointing out that if $X$ and $Y$ are dependent then it isn't necessary that they are jointly normal, and it wasn't clear that the OP had considered that possibility.  In addition, I am afraid (though I haven't spent a lot of time thinking) that without some other assumptions (like joint normality) the question might even be unanswerable.
The ranks are numbers (1, 2, 3 and so on). Number 1 is Andy Warhol actually. I don't have the number of paintings, so they don't weigh in. I don't have to computer the ranks of the artists anyway. I have them already.
I see what you mean. Since we have $E[X_1]=P(X_1=1)$, we can use conditional probabilities to define $E[X_1]$. In fact, I'm trying to form a general case not limited to a product of three variables; $X = \\prod X_i$ for some $i$. I was hopping to find a formulation in terms of a covariance/correlation. But I think I stick to what you suggested. Thanks!
+1 Not tada, rv/windowsize ;-)
The problem with the mosaic plot approach is that it will also detect deviations like: 'there are more young males and females', while I only care about the difference between the datasets.. is there no other way ?
An answer to another question very much like this appears at http://stats.stackexchange.com/a/13449.  The methods of that answer will work here.
First, the margin of error given in the wiki article was based on a binary outcome with a 50/50 chance of success, which would have the maximal margin of error - so the margin of error is at most 0.049 - this same logic applies for any binary outcome, not just poll results. Second, are you asking that if you sampled 400 people and the algorithm worked for all 400 people, can you conclude that the algorithm would work for 100% of the population? That seems to be a different question.
i dont see how.
+1 for actually relating to the question of how to choose the right family (and I'd say there's room for some more elaboration here)
In that case, I would just caution that any distance metric you use for similarity/dissimilarity controls for the number of sample points. (For instance, use RMSE instead of Euclidean distance.)
Can you provide some detail on what the *actual problem* you are trying to solve is? I think that will help guide toward an answer.
(+1) For the article. This is an excellent answer.
For the rest, it seems that McNemar's test (or, since you have several models, Cochran's Q) could be relevant. For more details, you can look at the question I mentioned in my answer or ask a follow-up question about this point.
Also, I am doing out of sample prediction that is I fit the model on one dataset and use it to predict on three other datasets. I also checked the predictions on the same dataset on which the model was fit and the results are consistently better with a negative coefficient.

Can you tell me more about forecasts that are biased. What is it and how can I incorporate it.

Unfortunately, I can't disclose the dataset but I can give an example that might help. Let's say the LENGTH is number of words in a file and the count is number of curse words.
Do you mean any $d-1$ points in a $d$ dimensional space? e.g. if you have 3 points in a 2-d space, with two being of class 1 and one being of class 2 all lying on a line, with the point of class 2 being in between the other two, then these cannot be separated with a line (1-d hyperplane)
@DBR Your previous account was unregistered, thus you lost control over it. Now it is merged with your current one, so everything should be OK.
Great points. Especially about plotting the data.
+1 for the effort put into explanation.
@Mariam Hm... I guess it's rather a computation issue. Try to insert (probably somewhere in the process *before* you compute BIC) logarithmization (I can't see your computation, but I guess you can reduce the numbers you get by applying something like logarithm).
The last one is a very good point.
You are using a wrong formula for computing standard deviation. Make carefully the expansion of the square in the definition of SD and you will see where the error is. By the way the numbers are appropriate for use with the original SD formula and if you try you will find the correct result.
Tukey discusses this approach--and elaborates on it extensively--in his *EDA* text.  He calls the curves "traces" and the whole thing is a "wandering schematic plot."  After constructing the raw traces, he applies a (robust, nonparametric) smoother to them.  See Chapter 8.
I'm confused by this set up. What's $I$?
@user1449677 What does "pretty much an image" mean? There are a lot of different features you can compute from images, [SIFT](http://en.wikipedia.org/wiki/Scale-invariant_feature_transform), [HOG](http://en.wikipedia.org/wiki/Histogram_of_oriented_gradients), or see the Wikipedia page [Feature (computer_vision)](http://en.wikipedia.org/wiki/Feature_(computer_vision)) as a jump off point.
Are you saying that the way you collect the samples leads you to think that a paired approach is not appropriate?  My uninformed thought is that the geography of the different river sites would affect the measurements.  If that is true, then a paired approach seems appropriate to evaluate differences between seasons.
I have skimmed through the papers but find them very hard to follow, being not a statistician. Any advice how can one compute box cox for mixed models in R?
@entropy $w^{T}x$ is a hyperplane passing through origin. To cover the space of all linear equations you need the bias term. 

Think of points residing in 2D and let us say that you are trying to find a line that separates these points. However these points all lie in the first quadrant. Now one can arrange these points such that they are separable but not by any line that passes through the origin. However, a line with a proper bias can do it.
Possible duplicate: http://stats.stackexchange.com/questions/73/what-r-packages-do-you-find-most-useful-in-your-daily-work/
Thanks Glen. Has been corrected.
If the data is drawn from two gaussian spheres, then can't I just use a linear kernel (input space=feature space) to draw the separating hyperplane in the middle between their two means? I'm looking for an example when data which isn't linearly separable in input space would become separable in feature space...
This is also why the area of a circle is *not* equal to $\\int_0^{2\\pi} \\int_0^R 1 d\\theta dr$.
That's good news, but if the real data are thousands, what chance have we of making good recommendations from 20?
To me, the variances don't seem too unequal. But if you want a test that is robust to violations of distributional assumptions: have you considered bootstrapping it?

Also possibly related: http://stats.stackexchange.com/questions/88457/hypothesis-testing-wilcoxon-test-bootstrapping-or-something-else
This is a great answer. The links are very useful.
Your question is pretty broad.  See if part of it is answered by the dozen or so threads that come up when you search on resources, sources, and recommendations.  As far as "what analysis to run," that is so broad that perhaps all one can say is to read widely, study, experiment, consult with others, and give it time.
Have you tried [Wikipedia](http://en.wikipedia.org/wiki/Cook%27s_distance)? Is there something beyond that you want to know, or is something about it confusing? What, specifically, do you mean by the "quartile method"? Can you say more here, & state what you've done to figure this out already / what you already understand & what you still don't?
Welcome to the site - do you have any potential data sources that *would* be appropriate for using gensim but also somewhat more digestible?
Most now suggest against [[1](http://stats.stackexchange.com/a/20856/4485), [2](http://www.stata.com/support/faqs/statistics/stepwise-regression-problems/)] stepwise selection. LASSO or its generalization in elastic net (for example, `glmnet` in R) are what are recommended if domain-knowledge based model building is difficult. Logistic models are possible with these techniques as well.
welcome to the website. Green's formula that you referred to must be some ad-hoc rule of thumb. I have not heard of it. Can you edit your post to give a reference?
agree with @James ..also check out [the main website](http://faculty.washington.edu/tlumley/survey/) and [click here](http://faculty.washington.edu/tlumley/survey/example-chisq.html) for example code on chi-squared tests with survey data  :)
As a simple example where you have one explanatory and a dependent variable: A linear regression of the *ranks* of $x$ and $y$ would yield Spearman's correlation coefficient as regression coefficient.  And in this case, $x$ and $y$ are interchangeable in the regression.
(+1) Along the lines of @cardinal's suggestion, `p <- tabulate(apply(gtools::permutations(6,4, repeats.allowed=TRUE), 1, FUN=function(x) sum(sort(x)[-1]))) / 6^4` yields the probability distribution all at once.
Excellent answer. Thank you so much.
Really great question.  The best way to understand something is from multiple direction of explanation.
How do you ensure mu, alpha and beta are not set to negative values?
+1 i thought binomial as well, when i first saw the histogram. (Not sure why this got downvoted).
Sorry, I improved my answer and did not notice you commented my post already. assault loads with -0.995 on PC1. Thus, one can conclude after squaring this value PC1 explains 99% of the variance of the variable assault. After squaring the values of urban pop, you can conclude PC3 explains 4% and PC2 95.5% of the variance of urban pop.
In addition, the idea of using residuals only makes even halfway sense in the very special case of the ordinary linear model.
I can't follow your question, because it's not evident what role the $X_i$ play in your "linear model" or what you're really trying to do. If you have the raw data, isn't this just a tangential exercise of little relevance?
Do you have a description of descriptive choice algo?
This is correct but it's interesting that from a historical point of view it's a perfectly circular argument: the Gaussian distribution first arose because Gauss found it is the distribution for which the MLE minimizes the sum of squared residuals!
You did not explain why you want to discard so much of the information in the continuous variables.  Any categorizations will be arbitrary and information, precision, and power-losing.
thanks. I am familiar with such book, great reference.  I tried looking into it but they do not provide such multinomial hierarchical model directly, but they do have tons of good ideas that can be applied.
+1 interesting The aim is essentially an optimization problem, with a kind of penalty for spending to many evaluations on mode seeking relative to near mode exploitation. It is somewhat reminiscent of the [multi-armed bandit problem](http://en.wikipedia.org/wiki/Multi-armed_bandit)
Ah no but I want to train my classifier on positive and negative training, then classify the positive and negative testing. That is why I need to take a fix number of negative training and testing in a way to get always a balanced dataa
+1 For the second paragraph. I would like to point out that anyone who clearly understands all the elements of this answer probably wouldn't be asking the original question. It's probably not helpful to mention formal grammars to someone whose post starts with "I'm just getting my feet wet in stats..." The second paragraph here captures the essence of what the OP is asking. Instead of the first paragraph, you might say: an HMM models conditional dependencies of hidden states, where each state has a probability distribution over the observations.
Actually, @Charlie, you do need normality of some statistics if you want the t-tests and confidence intervals to be correct.  The log transformation is strong enough to raise doubts unless the dataset is quite large: the approximate normality of an estimated coefficient of a log response will imply the *non* -normality of the coefficients for the response itself and *vice versa.*