tttjjj/elt_bench huggingface dataset:
Please explain what an "uncertainty" quantifies. Is it a standard error? A confidence interval? A variance? And would these be assessing *measurement* error or *regression* error?  (The phrasing in this post seems to equate the two, but whether that is appropriate is doubtful.)
looks like the use comes from `Stata`, e.g. here http://www.ats.ucla.edu/stat/stata/faq/relative_risk.htm and here http://data.princeton.edu/wws509/stata/c4s1.html. But I'm hard pressed to see how the number of "children ever born" can be viewed as an incidence (prevalence - maybe).
Wow.  Thank you so much!  I really appreciate your walking through a specific example as well as providing the general solution. Thank you!
I wonder if the coin toss example would falsely give the impression that every event must be equiprobable...
Are they non-significant at all steps?
I can't see anything that would suggest there's a problem, but the residuals vs y-hat would be a better place to look.
@Tal CART is an equivalent to boosting, of a "pivot classifier" (the criterion that sits in each tree node, like some attribute grater than something or some attribute value in set something).
See Professor Bozdogan's work on these problems who really the first to study these problems on mixture models dating back to 1981. Since then such problems have become very popular in statistical literature and in many applications.
Thank you so much.  Tell me, what do you think of the idea of combining male and female scores (e.g. Myy + Fyy; Myn+Fyn; Mny+Fny; Mnn+Fnn) and then doing a McNemar test?  Do you think by combining the scores in this way i could address the issue of male/female proportional difference across surveys?  Thanks!!!!  Walter
@Procrastinator.  Thanks a lot.  I really do apprciate your edits and those of MAcro, Huber and others.  I will try to learn more and relieve some of the burden.
A more fundamental question, which ought to be addressed first, is why the SD is used to measure dispersion. Why not the absolute central $k^\\text{th}$ moment for some other value of $k$? Or why not the IQR or any of its relatives? Once that is answered, then simple properties of covariance immediately give the $\\sqrt{n}$ dependence (as @Gui11aume has recently explained.)
you might get a few ideas from an earlier question on clustering individual longitudinal data trajectories http://stats.stackexchange.com/questions/2777/modelling-longitudinal-data-where-the-effect-of-time-varies-in-functional-form-be
In the [Singh-Maddala distribution](http://en.wikipedia.org/wiki/Singh-Maddala_distribution) link, the pdf has two parameters - {c, k}, yet the R function ``dsinmad`` takes three parameters or am I missing something.
Even in the case where $Q$ is a scalar you will need a numerical solver because the basic form of the equation is $\\log(Q)=aQ+b$ for constants $a$ and $b$. Although conceivably a closed formula could be developed in terms of the [Lambert W function](http://mathworld.wolfram.com/LambertW-Function.html) it likely would not be of much help.
Yes, people have to some extent discover R on their on.  But much of the issue comes down to inertia of learning a new language.  New languages are always coming out that have advantages over older languages yet users cling to the old languages (witness COBOL).  Programming in SAS is hugely inefficient, requiring perhaps double the number of programmers to do the same job as R, but SAS experts are happy to hum along on their merry way and companies are afraid of the kind of disruption that would save them millions of dollars in salaries.
I'm guessing something went wrong with your inputs to the t-test; the `mean of y` at the bottom of your second code block should be 199355.5, not 304998.5.  That's quite apart from the fact that your sample size is really too small to draw any conclusions, I'm afraid.
I suggested some edits to your post but I didn't change the meaning. Can you please check your formula for Euclidean distance -- I think you have missed out a $^2$ at the end? And you don't need to write a sum $\\sum$ as well as $+\\ldots+$. Something like $\\sqrt{(x_2-x_1)^2+\\ldots+(x_n-x_{n-1})^2}$.
I edited your title and article; I hope this reflects your true question. If not, please edit again.
How is this related with roc?
@chl thanks. I know very little R, but suspected this simple algebraic trick could be applied in any case, or at least illustrate what is going on 'under the hood'
Well, you need to make some assumptions how how "latent categories" look like (you surely don't want them to be random; so you cannot be happy without *some* assumptions). If you have a good reason to assume they are spherical and that Squared Euclidean distance is the proper way of measuring similarity, then k-means is a good choice... If you assume that your latent categoires "look" different, you'll have to use a clustering algorithm that can accomodate this type of restrictions (e.g. density connected clustering such as DBSCAN).
Yes. The ridge function by default supports a vector $y$ and not a matrix $Y$. That being said you can always reshape your matrix in a vector, use Kronecker products in to reformulate your $X$ matrix into $X_{big} = I_p \\otimes X$ and take it from there. (Pay attention to the error structure you are assuming in this case though; $p$ is the number of your multiple outputs.)
@Nice. I think you mean "A and **B** are not correlated" in the very last part of your last sentence.
@Andy, thanks! Great stuff; if only I had the privilege to up-vote comments :).
Could you tell me more details about how to perform cross validation for NMF? The K values for the Frobenius Norm will always decrease as the number of K increases.
@whuber I'm trying to show that $\\operatorname{Var}(X'X)^{-1}=\\sigma^2 V$ ($V$ as described in the question)
(+1) Welcome, Davide. As you know, implicitly, we are using what is often known as Slutsky's Theorem here. I hope you have a look around and continue to participate. We occasionally get some fun probability questions here that don't appear on math.SE. For the future, [combining red and green](http://meta.stats.stackexchange.com/q/1072/2970) may not be the best option.
There are some quite good answers that will probably help you get your head around this on http://stats.stackexchange.com/questions/24222/effect-size-and-statistical-significance
Could you at least indicate something about the nature of this solution?
Use the Delta-method ?
Adrian, perhaps it would be easier for us to answer if there was a more general question you could start with. You've jumped right to how you would weight responses (presumably for developing a score to apply back to list items). But, I don't think we know yet if weighting is even the right approach (ttnphns's comment suggests something else entirely).
@MichaelLew Likelihood principle is useful, but I was saying that (quoting LW) "Indeed, all of frequentist inference violates LP, so if we adhered to LP we would have to abandon frequentist inference."  Because CI is a frequentist idea, it violates LP (which you say is fundamental).
I don't think simple logistic regression of chi-square will apply here as the fact that there are calculable proportions within each group implies that the data have a hierarchical structure, in which case my previous answer applies: http://stats.stackexchange.com/questions/14449/how-to-test-for-significant-differences-in-means-across-four-within-subject-cond/14462#14462
@Anony-Mousse I don't follow your reasoning ... because there's some minimization involved it's not 'proper' clustering? How do you feel about (kernel) spectral clustering? What do you consider valid clustering approaches?
I think you are right, that does not make sense. What is the original text you are using?
+ please forgive me that I am not a native English speaker :)
thanks imorin :)
@Michelle Just a little correction.  Sampling error is error that results from not selecting the entire population--that is, error due to using a sample to infer population characteristics.  Non-sampling error is everything else, including non-response, failure to create an adequate sampling frame, measurement errors, etc.
Confused; $\\mathbf{Y}$ should have the same columns as $\\mathbf{X}$ for prediction/projection. The EOFs or PCs are linear combinations of the the columns in the original data (the variables). IIRC, the predicted location in the reduced space is the weighted sum of the variable scores on EOFs 1 & 2, the weights coming from the values of the new data $\\mathbf{Y}$. See `?predict.prcomp` in R as one example.
@Karl Oskar, Ah yes good point you are right, sorry that must have been my own misunderstanding. Ok so I guess I am just after the distribution of $p(X,Y)$...
To make this answer self-contained, would you mind providing a brief summary of each link?
Thank you, especially for the Zoonekynd reference, i found that before but couldnt get back to it.
I am thinking of something similar to what you can do with the lmer function where you can include a random effect as (1| effect).
"*Given a probability distribution, how can we determine/evaluate the skewness and kurtosis of this data set?*" -- this question conflates two different things (data sets and probability distributions). Which do you mean?
Thank you for your attention.Because of my poor statistical knowledge,I can not understand the reference easily.I have revised my post to make it clear.
Thanks a lot.  Do you know any reference for this?
The question would be a little less confusing itself if (1) you formatted the output more neatly (2) you explained what this code is meant to do (more people could comment usefully with that explanation). The tags suggest "generalized linear model" but be aware that GLM or glm means variously "general linear model" and "generalized linear model" in different corners of statistical science. Evidently you are asking a program to choose predictors for a model for you and it struggled. Perhaps you should try something much simpler using scientific or substantive knowledge to guide model choice.
"Quadrature" ordinarily can mean [many things](http://en.wikipedia.org/wiki/Quadrature), but none of the common ones appear to correspond to your use here. What exactly do you mean by this term?  Is it perhaps related to a theory of error propagation and an application of the [delta method](http://stats.stackexchange.com/search?tab=relevance&q=delta%20method)?
Edited in a brief response as a third paragraph. Hope it helps!
(+1) Just a tip: not everything needs to be done using mixed models - you might be fine using "simpler" (more robust??) methods like `gls`! But I just started with it and have no experience so it's just a tip for you where to investigate...
After take log my data are all negative,because my original data are like all around 0.4,so shall I not take logs just use the original data?
Not necessarily always higher (see here: [When is R-squared negative?](http://stats.stackexchange.com/a/12991/7290)), but it does change the calculation & make comparisons invalid (see here: [Removal of statistically significant intercept term boosts R2 in linear model](http://stats.stackexchange.com/a/26205/7290)).
@NickCox, GDP per capita may not be stationary either, if it's not deflated (real).
+1, I think this is a really good question. You may also want to check out [this earlier question](http://stats.stackexchange.com/questions/11009/including-the-interaction-but-not-the-main-effects-in-a-model) that covers much of the same territory. The answers there are really excellent as well.
Interpolation involves 3 things: 1) a class of functions to interpolate, e.g. sound, pictures, terrain; 2) grids of known / unknown points, 4 cases regular <-> scattered; and 3) a model of noise added to 1). There are many many interpolation methods for various cases, most ad hoc, not "in common use";
even IDW has variants. Can you describe what you're interpolating ?
Apart from the point that the standard deviation is the root mean square difference rather than the mean difference, it may depend on what you are using this for.  Another point to consider is in cases where one currency moves dramatically against the other so the scales are substantially different at different times; if so, it may be better to work with relative rather than absolute errors.
I don't understand what any of this means, would you be able to express things more mathematically? What does it mean to be "95% confident that the range I computed includes the true value?" What is the range, and 95% of what technically? (and let's say x = a mean) I don't understand what a confidence interval / "true population mean" implies going forward after doing an analysis. Does that mean if we performed infinitely many samples, then 95% of the recorded values would be between .95x and 1.05x, and 5% would be outside that range?
Thanks @Frank, I'll look into that. Sounds like it's what I need!
Actually, I'm suggesting the opposite. The method I'm recommending gives you an unweighted (i.e. unbiased) average. I was pointing out that the method you had originally suggested would unintentionally produce a weighted average, but with arbitrary weights.
I haven't formally studied it, but I remember looking it up. You really have to turn to a human to look up something based on appearance. I don't think there is a search engine for math notation :)
I can add to Glen's neat answer that CH index is also called "multivariate (pseudo) F". So, it is actually Fisher's F that is used in ANOVA. ("Pseudo" because it takes not possible covariations into account.)
Why not use the model obtained to predict on the full dataset and obtain the confusion matrix from that?
Bad phrasing on my part, I wasn't really using "breaks", I just was selecting the position for major and minor grid-lines.  The argument for (at least one way of) selecting those positions was "breaks".  In the example I failed to plot it as a line chart just for expediency.
There are many distributions that are easy to describe which do not have standard names.
I see it all the time ... I think sometimes it happens because people don't know (or don't fully absorb) the distinction. I've just been teaching from a text (not my choice of text) which described the distinction at length, and then went ahead later in the book and committed exactly the error it cautioned against in the early chapters, and did it over and over.  [However, if you understand well how the $x$'s are distributed, you can sometimes get some idea of the conditional distribution even from an unconditional display, so it's not automatically a mistake -- but it very often is.]
Thanks for the chapter link, which was very helpful. My question was targeted specifically at the situation of two studies - a meta-analysis with k=2 is not so "meta" ...
Actually. Now that I got enough sleep and just starting to think about it. This will mean that two numbers differing slightly around the mean will be more similar than two that are exactly equal but equal to the mean. It seems like I shouldn't take P(X<x) if the two values are the same. It seems like I should have some sort of a penalty factor that diminishes as you go away from the mean but shouldn't punish the value so severely that would make two numbers around the mean more similar than two exactly the same around the mean. So you are right in your edits above.
@andreister I don't think it is better than regularization. I think it is an (early) alternative to regularization. It is a very old paper, regularization became mainstream in ML in the mid to late nineties.
Thanks Sven, I have been checking the residuals of course and i have been also testing my full model against a "null" model with only random effect using ANOVA. My qyestion is whether the first model could be correct to interpret my experiment. The difference in the residual distribution is not that much. I think that even is a learning curve is exponential the fact that I have so little repetitions assimilates it to a line.
I concur, there's one point that is just not like the others, (0.999983507 0.996875105).  The difference in y-value from that one  dwarfs the differences amongst the other points.
I added the blogs in the question as answers to allow proper voting to find the most popular blogs.
I wasn't sure if you wanted the distribution of the vector $A^{-1}z$, or if $L A^{-1}z$ would work for matrix (or vector) $L$. For vector $L$, the Bodnar & Okhrin trick will generate variates for you. O/w, good luck. Let me know how that works out.
@FredrikD: point taken. (Though I personally didn't manage to sit even through a single "use" of this "product"...) There should be generalizations of Bass to deal with this. (Shameless plug:) Next year's _International Symposium of Forecasting_ is in Seoul, so anyone should consider presenting his/her favorite Gangnam forecasting model there! ;-)
Tom, please notice that i am asking for the model i am proposing. Which is different to the model i am citing from the wikipedia.
Excellent answer. They are certainly hard to get right and that probably contributes to the problems with perception I have observed.
You might be interested in a more practical text, have you seen Bayesian Methods for Hackers? (Disclosure - I am a contributing author) Try searching for it (it's opensource and free).
I'll read ocram's comment later on and get back to you...
The question is about the estimate, not the true parameter. But perhaps the question is not well phrased.
Wow, certainly a much better answer than I ever expected. And, quite ironically, it was your 2004 mcmc tutorial on videolectures.net that led me to this question in the first place! (despite the fact that you explain this issue well there). I was still interested because it seemed there were still functions which the harmonic mean would be easy to estimate... But I think you've answered that question here and in citations. Thank you very much.
Can you say more about your situation, @Shirley? Do you mean that you have 9 different Likert *items*, where some are numbered 0-4 & others 0-5? Or do you have multiple items (forming a *scale*) per construct, and you want to average over all the the items w/i a construct to get a single mean for each variable? Or something else? In formulating your question, it might help to read this blog post: [how to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/). Also, if your criterion is ordinal, you might consider ordinal logistic regression.
@FrankHarrell typically this is done in cross-validation and does not use any additional data. Robustness of the procedure depends on the data itself (for example the structure of intercorrelations), but in many cases this approach works well for many Machine Learning problems. Obviously, this is a heuristic and will not be as comprehensive as evaluating all combinations of features - but that would be NP-hard in general. My point is that even without theoretical guarantees this often works well in practice.
I suspected `xyplot` was looking for a value of `data` and not finding it. But I just tried `ab <- data.frame(a=a,b=b); xyplot(a~b, data=ab, subset=(time(a) %in% c("1991 Q2", "1991 Q3")) )` it gives the same error. Maybe it's some bizarre interaction with the `xts` or `xoo` class and the underlying subsetting method. I'd say do something like `xyplot( x~y, data=ab[a<10,] )` as a workaround.
I appreciate your answer, but I'm not claiming that $\\mu$ and $\\sigma^2$ are the mean and variance of $X$; they're the mean and variance of $\\log X$, the associated normal random variable.  If $\\mu$ and $\\sigma^2$ have dimensions, which they do if $X$ has dimensions, then the formula for $<X>$ is not dimensionally sound, which raises my eyebrows.  And maybe that's 'just the way it is', but it at least seemed worth asking about.
Not in English. But it makes sense because of the definition of completeness: $E_\\theta\\phi(x)=\\int_x\\phi(x)\\,dP_\\theta(x)=0\\Longleftrightarrow\\phi(x)=0$, up to a zero-measured subset. So if there are two unbiased estimator, their differences will be almost everywhere zero.
The measure of time is day, and yes, the treatments were implemented at the same time (started on March 1). Thanks.
Here is Ben's answer for (the) posterior. https://stat.ethz.ch/pipermail/r-sig-mixed-models/2014q3/022351.html
How are you using the lambda parameter ? Have you tried running the network without any regularization ?
@Glen_b: I take your word for it, but, at least regarding to *t*, K. Pearson rejected for publication Fisher's initial paper on *t* because he could not follow Fisher's proof, and said as much very explicitly in his correspondence with Gosset.  Neither could Gosset himself follow Fisher's proof.
You will want some sort of time series model. I've added "time series" to your tags.
What do you mean by a "validation test"?  Also, please note that the chi-square test of goodness of fit is specifically for *discrete* distributions; it can be applied to continuous distributions only by binning them (that is, turning them into discrete distributions).
have you attempted to answer this problem?  Can you tell me maybe where you're stuck?
Reconstruction or the variance of the data is only one of many statistics that you might be interested in.
Dunno why, but a Mantel test flashed in front of my eyes when I read your post.
I've tried your code with correlation matrix of random normals, and there were no marked increase in average correlation. So probably the problem is as Jonathan explained, the eigenvalues you discard carry substantial information about your input matrix. I have few questions about your code though. Why do you do the transformation in 4 line, i.e. inflate eigenvalues? Why do you do tranformation `0.5*(BB+BB')` in the last line? `BB` is symmetric, so `BB=0.5*(BB+BB')`.
"*Both sets are not normally distributed and therefore, correlation should be assessed by Spearman's correlation*" -- (i) why not Pearson, exactly? Where does it rely on normality? (ii) If you can't use Pearson, why Spearman, specifically? ... as for computing Spearman, why not rank your data, and then compute Pearson?
@honitom It really depends on the context of your experiment and your sample size. Keep in mind that the reliability of your estimates (the regression coefficients summarizing the effect of your factors on your outcome) depends on the available sample size and associated variance. In DoE, interaction terms can be aliased with main effects, depending on the sample size; still we can estimate the main effects. Any decent textbook should cover this topic much well than me in a comment, so I'd suggest to have a look at ANOVAs and DoE.
Also, I think that I would need another function to transform it to a continue space.....
Thanks! I have changed it in the R code above. It's still not giving the expected results. Do you see any more errors?
@Hiatus the norm of the separating hyperplane (which is being optimized when training an SVM) is not used in the decision function. The hyperplane itself is used. Minimizing the norm during training is basically a form of regularization.
@Dilip He is assuming that $p$, the failure rate, is homogeneous. Then, wishing to estimate the average failure rate is equivalent to estimate $p$. Your second statement is a tautology.
See also (the bottom) of the second page of http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34.4144&rep=rep1&type=pdf and page 67 of http://books.google.com.au/books?id=Zf0gCwxC9ocC&pg=PA67#v=onepage&q&f=false for the relevant knowledge.
It's somewhat debatable because it's untrue when the null is literally, exactly true, but since that is so rarely the case (due to all sorts of complexities like spurious correlations), it's probably true of most practical applications. Hypothetically speaking, one could detect the weakest of spurious correlations (e.g., *r* = .001) due to a chain of mediators hundreds of variables long despite a similar number of uncontrolled moderators if the sample was colossal enough. Arguably, that relationship actually exists though, so whether that's really "bias" is still somewhat debatable IMO...
Any particular reason for the down vote?  Seems a bit harsh to me - what is wrong with my answer?