tttjjj/elt_bench huggingface dataset:
Srikant, I don't think that the second one is an unbiased estimator.  The square of it *is* an unbiased estimator of the true variance.  However, Jensen's Inequality establishes that the expectation of a curvilinear function of a random variable is not the same as the function of the expectation of the random variable.  Hence the second formula can't be an unbiased estimator of the true standard deviation.
$P(Man) = y+z = 1-(x+w) = 0.7$ but you cannot calculate the exact values of $y$ and $z$ with the information given in the problem.
Correct: only the diagonal term of the covariance matrix matters for the marginal.
@yue86231 It means that although we have one p-value for each predictor, we can not interpret each p-value in isolation. Each predictor t-test can only show the significance of a variable *after* accounting for the variance explained by all other variables. The linear regression coefficients and standard error are produced at the same time, so to speak, and the two predictors reduce each other significance.
Better use a true data analysis software. The functionalities offered in speadsheets are very limited and often just boil down to exporting the data into a better software such as R or ELKI, then analyzing it, then importing it back into the speadsheet. Doesn't make much sense to me...
(1) No, any distribution. (2) Hmm, I'm not mathematician. As I know the two decompositions are indeed quite different. `Composed of the eigenvalues` Did you mean "eigenvectors"? BTW, it is possible to generate the variables you want also via principal components, with the help of eigenvalues and eigenvectors. But, sorry, I failed to understand what you imply by the two formulas you showed.
Are you assuming $p={1 \\over 2}$ ?
Can you clarify what you mean by "inverse relationship" in this context? I'm not sure I follow you. Do you simply mean that you think the inpatient infection rate will go down?
A lot more information is necessary such as: the regression model (e.g. how many variables and observations you have), the prior distributions, a link to the paper you mention ... Otherwise the answer is [the Reference Manual](http://cran.r-project.org/web/packages/rjags/rjags.pdf).
A random walk will be serially correlated, so I'd expect it not to be 0.
It would be nice to know how did you arrive at this question. I cannot imagine real life situation where such type of information about regression model would be available. (For that matter no imaginary life situation is possible too)
It's not asking for the "likelihood *that* the nth point has been left out" (which would suggest a probability), it's saying "find the likelihood *for* the nth point that has been left out". That is, it is saying *evaluate the likelihood function* for that point.
@whuber, I'm asking this question because I'm looking for conditions under which Bayesian (with a flat prior) and frequentist point estimators, derived as optimisers of expected losses, coincide. Do you happen to know any references on this?
I'm doing analysis on fabric trauma. I have four different knifes, and I've measured the dimensions of the tears they leave in fabric, and would like to see if the knives leave overall different length and widths in the fabric.
Following from that presumably its also true that one could use the exact same test with any normally distributed data. If I were to use a voltmeter which I knew to have some normally distributed error that I had determined then I could use, $$\\chi^{2} = \\sum_{i} \\frac{(V_{obs} - V_{exp})^{2}}{\\sigma^{2}}$$. Is this true? The reduced chi square statistic presumeably relies on this fact.
What efforts have you made to figure this out?  And what software package are you interested in?  For example, proc glimmix in sas has very good documentation on how "REML" type estimation is done.
As a fairly routine bookwork-level question this should probably carry the `self-study` tag. Please check the [tag wiki info](http://stats.stackexchange.com/tags/self-study/info). Do you know the formula for leverage?
@MichaelChernick, Thank you for your reply. I do not want to rerun the logistic regression on generated bootstrap dataset. Instead, I want to apply the trained model from original dataset to the generated data. And check the variation of ROC value or range of ROC value. I do not know how to output.
Although this is a little off-topic, let me reply to whuber. I believe the sign of log-likelihood actually matters: if the sign is reversed, the sign of the difference between log-likelihoods of two competing models is reversed, too. Consequently, the conclusion about which of the two models is more likely to be "true" is wrong. Thus we should better be sure about the sign of log-likelihood... But perhaps you meant what you said conditionally on that we know the log-likelihood definition in a given software. Once a known definition is followed, the sign of the outcome does not matter much.
Absolutely not.  Logistic regression *is not* a type of classification.  It is a direct probability model that does away with the need for arbitrary classification entirely.  And normalizing the data means that you need to save and re-use the normalizing constants when evaluating the or developing predictions, thus effectively creating parameters that are external (and must be remembered) to the model's parameter estimates.
Variance is the standard way to normalize distributions.  You specifically asked about kurtosis, and as I've said, kurtosis is defined based on the normalized fourth moment, which means that if you are interested in comparing kurtosis, then yes, you should compare distributions with the same variance.
@Joris The functional form `S ~ b0 + (A > T) * b1 * (A - T)` is equivalent to `S ~ b0 + b1*max(0,A-T)`.  Changing notation to $x$ = A, $\\delta$ = T, $\\beta_0$ = b0, $\\beta_2$ = b2, and setting $\\beta_1=0$ gives a special case of the problem solved in the duplicate post.  If you have improvements to suggest for the solution there, then it would be best also to post your reply there so we can keep this common thread together.
@BoppreH You're right. This was a typo. Thanks to cardinal, who already corrected it.
As a physicist who uses a lot of statistics but isn't a statitician I find that paragraph really unhelpful, never mind the fact it sounds plain wrong. I have always thought, possibly incorrectly, that a 95% c.l. meant that if the null hypothesis were true then 1 time in 20 if I repeated my experiment would I get a result significant at the 95% level (a good reason in my opinion to not use less than 99.9 but thats another discussion). That post seems to be more a point about correlated factors and really doesn't help non-experts (or anyone) at all.
You never need "ods graphics on" before an SGPLOT call.  Only before statistical procs.
By stratifying, you can assume proportionality of hazards in each of the strata, so you can use the cox model with the main assumption being met. I think you can still easily visualize the effect through plots of the estimated survival or hazard functions, specified with the covariate in question at different values, i.e. hold all the other predictors at their means, then vary the value of your primary predictor. You can see the effect that way
Thanks for answering me. I don't find it odd for it to predict cases, I just thought I was missing something in order to set the prediction for rates (cases/population). Since in GAM's I didn't had to add anything else for it to predict (cases/population).
What software are you using to run your garch regression? R has some good packages that I have used in the past for this. I could probably dig up and post some code if needed to do a forecast and predict function for this. Your errors are going to grow as you forecast out this model how many periods is t that you are looking to forecast?
Thank you for your contribution, and welcome to our site!  Could you make it clearer how your answer addresses the question about the validity of using a CV at all to characterize data that can have negative values?  That situation would seem not to be covered by any of your remarks.
You shouldn't ignore it. Reliability of correlations isn't really the issue here; when there are limits above or below relationships are less likely to be linear. But in the example you cite, it is hard to imagine any situation in which precipitation and temperature would be expected to be linearly related on any time scale, although your research problem may be special. So, the real question is why you think correlations or indeed regressions might be useful here.
Quick note: $P(Y=y)=0$ (not $4$) for each $y\\in[0,0.25]$. The density $f_Y(y)=4$.
I wonder which part of the answer the down-voter believes to be incorrect or not useful.
Please may we have an expected value tag?
I meant second hand smoking, it means that people around you are smoking. I am trying to get the data set. It may take a week. Then I will contact you again.You have been very helpful, thanks
Because if the model isn't identifiable, then the inferences are being driven by the prior. This isn't necessarily a bad thing (as your question points out), but it's always something that's important to know.
+1 R is a great stats tool, plus free is good!
You _can_ do that, but it would be better if you didn't have to throw away data. I imagine the reason this question came up is that you're optimizing your solution for _accuracy_, but the classes are skewed enough that a high-accuracy algorithm has really poor [_recall_](http://en.wikipedia.org/wiki/Precision_and_recall). I would recommend optimizing for something other than accuracy (like minimizing a weighted sum of false negatives and false positives, more heavily weighting the false negatives).
Intriguing (+1). But I can't help thinking that a Halfer might come back with something like "but SB knew *in advance* that she would be awakened, so the experience of awakening provides no new information." It seems akin to the more prosaic example of an ordinary coin flip. After the coin is flipped--but before you learn the outcome--you know the coin has been flipped. But it's either nonsensical or useless to then assert the probability of heads is either 1 or 0.  *Your* credence for heads remains exactly the same it was before the flip. Some kinds of information do not change probabilities.
"divide the covariance by n if you don't estimate the means (i.e., your C)" not sure what you mean here sorry?
*IF* the number of parameters are the same, higher likelihoods are preferred. If they don't, then it depends (hence the test). There's no 'typical' size for likelihood in general.
Hmm - I think you actually 'reward' the one with bigger n, because $t= \\frac{\\mu-\\mu_0}{S/{\\sqrt{N}}} = \\frac{\\sqrt{N}(\\mu-\\mu_0)}{S}$  Yah?
@Babyburger It seems that you are keen to seek a test that works in the presence of a number of other (external) factors. In the available literature, we have statistical tests that presume fixed effects.
Thanks, that helped! So I guess a high AUC (which is over all thresholds) is not necessarily related to a high MCC (at a 0.5 threshold). And also vice versa.
+1 for pglm, did not know about it.
I think this question should be moved over here: http://stats.stackexchange.com/
Have you had a look at this Kaggle competition? http://www.kaggle.com/c/cause-effect-pairs you might find something interesting
Just to clarify, you are looking for a feature that will automate the determination of number of clusters? or are you simply looking to batch process the running of a set of cluster analyses where number of clusters are known? What is your current manual approach to deciding whether two or three clusters is appropriate? do you wish to continue to use this rule or are you interested in other procedures for determining number of clusters?
The Kolmogorov-Smirnov test is a typical non-parametric tool for testing whether two distributions are the same. I'm not familiar with it, but wikipedia refers to *Justel, A., Peña, D. and Zamar, R. (1997) A multivariate Kolmogorov-Smirnov test of goodness of fit, Statistics & Probability Letters, 35(3), 251-259*. for a multivariate extension of this test.
@fmark, if you don't have exactly zeroes than there is no problem to take log transformation (the only case for zero is if the store sells nothing during a quarter!, but it is a strange case). You have this small range probably due to units you measure your sales. In log-log regression this will be absorbed into intercept term. However, if you do have some true zeroes then we will think about other useful tricks (by the way no need for logs in dummies).
Dear Robin, welcome to the site and thank you for the links, especially the first one, which I had not seen. Incidentally, I recently rediscovered a similar technique for the special case $f(p) = 1 - p^a$ (though, how to generalize is clear) that I posted in an answer to a related question [here](http://stats.stackexchange.com/a/50299/2970). Nacu and Peres' algorithm, specifically Proposition 16 on pg. 15 appears similar, but not quite the same. Theirs may be better; I'll have to read the paper more thoroughly to know. Thank you, again and I hope to see you continue to contribute! Cheers.
@Ahmed: I am not sure, it depends on what you already know. I guess you would need decent textbooks on linear algebra and analysis. This is pretty basic stuff, should be covered in any decent textbook.
It is _very_ unlikely that such an approach will be as good as fitting all 1000 features using penalized maximum likelihood estimation with a quadratic ($L_2$) penalty.
What progress have you made on this problem or where are you stuck?  It would be helpful to know what you have already tried so that we may be able to lend some helpful hints/advice.
Can you clarify your situation a bit.  I do not understand your sentence, "For each question a factor A, B, A and B or none are "active"."
I find MCMCglmm can be difficult to program.  The package "regress" in R provides a simpler way of including a covariance matrix into mixed effects models.
This question is too vague especially in light of the OPs comments. I think you need to answer Procrastinator's question to make this a real question.  Otherwise I will certainly vote to close.
Are you enjoying your own personal Q & A? ;-)
Have you thought of leaving in the percentages as predictors, and using the time *per document* as the response variable? That seems to be what's important. The regression model probably should use number of documents as weights.
@charlie - Thanks! In the world of big N, using a split sample for selection and then applying the model to the other split or full data, then using that to do all the normal inference, is certainly attractive.
Added link. Wasn't able to find source of it that was free but perhaps you can get it through your university library.
My summary of this is that the $\\chi^2$ is a two-sided test for which we are usually interested in only one of the tails of the distribution, indicating more disagreement, rather than less disagreement than one expects by chance.
I will look at that link later when I have more time, but it explains specifically what correction to add to my covariance matrix? And how to go from the covariance matrix to a single number for uncertainty? (Numpy automatically calculated a covariance matrix for me, but I have no idea how to use it).
(1) Is this a homework question? If so, please tag as "self-study." (2) Is `u` the response variable and `x` the predictor? If not, please clarify.
(+1) For the update. Dividing the left-hand side by the right, one can see that this reduces to finding the median of an unknown probability distribution supported on $[0,1]$.
Not anything related to statistics, "n.b." stands for *nota bene*, which in turn is Latin for 'note well' (literally), or 'note that' / 'notice' (more colloquially).
Correlation is useless for this purpose, even with single-predictors. Consider X on 1,2,3,4,5 and fits f and g being (12, 15, 17, 19, 23) and (101.12, 101.15, 101.17, 101.19 and 101.23) respectively. They're *perfectly* correlated, but they're nowhere near each other.
I like this "original" approach (wrt. other entries). Still, I always find difficult to explain why bootstrap works in practice...
$S = Z_{(x-y+1):x}+ ... + Z_{x:x}$. I'm not sure why you say they are not equal. Since they are not independent, you can't determine the distribution of $S$ from the distributions of the order statistics from this equation, but it is still true that the sum of the $3$ highest dice is the highest die plus the second highest die plus the third highest.
I'm sorry I really don't understand! I simply pasted this in replacing dfrm with the name of my dataset- and the word that needs replacing is no longer 'Hi' that was my friends screwup it's now all NA's. So i swapped Hi for NA. And I just got thrown the error:

Warning message:
In `[<-.factor`(`*tmp*`, x == "NA", value = 5) :
  invalid factor level, NAs generated

not a clue what this means!
The fact that a cutoff was used by mistake is no reason to keep using it.  I'm sorry to sound so critical but you are throwing away an enormous amount of valuable information.  Part of your problem may stem from a suboptimal choice of an accuracy measure.  Use a continuous proper scoring rule, e.g., Brier score, logarithmic probability scoring rule, or $R^2$.
No, robust standard errors are a part of a different story. The tobit model is known to be so sensitive to the underlying assumptions (you would probably find the discussion in Wooldridge's black book, http://www.amazon.com/Econometric-Analysis-Cross-Section-Panel/dp/0262232588) that once you admit something is wrong with it (and that's what you imply when using  the robust standard errors), you can just as well forget the whole model.
So is the region you want to be the random effect also the first sampling unit - the one with 300 different levels?  I don't think I can answer your question (because I don't know how to do this, and think there isn't an out of the box way to do it), but this will have a bearing on it.  Conceptually, there is a sort of connection between region in the sampling design and as a possible level of random effect - but these two approaches come from different historical approaches to this sort of data.
For clarity, are you interested in interpreting the model as causal or are you simply interested in prediction?
I wonder then if there's an interpolated estimator with better finite sample properties, besides trivial cases like $N=2$ or something. Though I imagine it'd require extra assumptions about the true distribution. The efficiency of the ecdf is quite remarkable, really.
A decent place to start might be http://www.iro.umontreal.ca/~lisa/bib/pub_subject/finance/pointeurs/techrep.pdf but there are a bunch of different approaches.
I'll second @mpiktas's comment, and I would recommend you take some time to think about your question(s) and how to improve them (I'm not speaking about english typos, but about their clarity). I also noticed that you only accepted few answers so far. Any reason why? It you feel users missed important points when answering your past questions, you should notify them; otherwise, if one of those responses looks acceptable from your point of view, you should mark it as is.
I'm pretty sure the leaps packagae is kind of obsolete. Have you looked into the CARET package? it's nice, and their webpage has some nice examples and eveh theory: http://caret.r-forge.r-project.org/featureselection.html
The headers are the usual: B, standard error, Beta, t, Sig., and correlations (zero, partial, part).

Stepwise maybe bad method, but it SHOULD drop variables instead of leaving empty the significance.

N is 40, and I am trying to get an interpretable result. :)
Welcome to the site, @MarkMcFarland. Would you mind giving an overview of the explanation listed there, eg, in case of linkrot, & to help future readers decide if it's something they want to pursue?
Thanks onestop! We are definitely going to use each aquarium as the unit of analysis so we will not have dependent data. We are still formulating the best approach. I will post the final decision once it is made.
Do you mean `polr` from the package `MASS`? Nothing in your question seems to require Rstudio as such.
See [this answer by @Stask](http://stats.stackexchange.com/a/38430/1036) addressing this question.
Could you give more details for your second question? How do you define selection of distribution at random with probability determined by measure $\\mu$?
Please take the time to write a sufficiently detailed question to get a proper answer.
I do not know whether there were any other attempts. Anyway, I think this is what is classicaly meant by non-informative since this is the classical likelihood...
Hehe you're right. For the above to be correct we could assume the prior is normal with infinite variance or zero precision. Then it doesn't show up.
"Singularity" and "specificity" are both difficult to interpret because they have many possible meanings: a context is needed in order to understand the sense in which something may be different or unusual.  What context do you have in mind?
Currently I don't have such a measure. I can think of three strategies: 1) work with a reasonable dispersion value chosen once for all (accepting that it won't work so well in some cases); 2) compute a global dispersion estimate valid for all clouds (possibly taken from a sample of clouds); 3) compute an estimate per cloud. In the latter two cases, I need a way to estimate the dispersion for the bananas.
(+1) There are good points in this answer: `...explain what I am doing in my own words, using their example...`. To my experience in teaching programming languages it is one of good solutions. The third paragraph: `Once you get past elementary statistics, ...statistics is too broad a field.` is also productive and motivational. The language of the comment is also positive.
Thanks for that, but I don't think it's the (whole) answer. I've added some details to the question, where I've used a Barnard's test, and it doesn't make much difference.
I don't have a first hand experience doing this, but based on the description of the problem, it seems, you could use longitudinal logit models.
"since one of the properties of SVM is that it effectively ignores all the data that is far from the decision boundary". This statement is incorrect. SVM is actually very sensitive to outliers. There is still ongoing research on how to supress the influence of outliers like [this one for example](http://webee.technion.ac.il/people/koby/publications/AAAI0613XuL.pdf).
Thanks for your answer, this goes to my TO-DO-LIST
If you are not in a rush... I am traveling right now and answering in detail on a phone is a recipe for disaster... Will come up with an answer once I return to civilization (well I am at the birthplace of Hippocrates - so in a sense surrounded by millennia of civilization).
@juampa There is no reason for having 9 nodes in the hidden layer. we felt that having more node in the hidden layer will increase the complexity of the network and overfit the data.
The purpose of the plots will, to a large extent, determine which solutions are appropriate.  What, then, are you trying to show with them?
A and B could be disjoint, so the minimum possible value of the probability of their intersection is zero.
Depends on your data, and we don't have your data. This question cannot be answered.
(+1) Nice answer.  This will become a reference for similar questions in the future, I think.  However, cross-posting is unusual; in fact, I believe it is frowned on, because it screws up many aspects of the feedback/referencing/threading/commenting system.  Please consider removing one of the copies and replacing it by a link in a comment.