tttjjj/elt_bench huggingface dataset:
$p(y)=\\frac{1}{{{\\left| \\Sigma  \\right|}^{n/2}}}\\frac{\\Gamma (a+n/2)}{{{\\left( b+\\frac{n-1}{2}tr\\left( {{\\Sigma }^{-1}}S \\right)+\\frac{n}{2}{{(\\bar{y}-\\mu )}^{T}}{{(\\Sigma )}^{-1}}(\\bar{y}-\\mu ) \\right)}^{a+n/2}}}$
@whuber: I added a plot, in which you can see the two lines do not cross. Does this make sense??
@COOLSerdash That looks like a pretty decent answer to me.
No, Sw and Sb aren't scalars. Don't you look at fancy formulas. Wording is better. A scatter matrix is just a covariance matrix without dividedness by "n-1" (i.e. a cov. matrix multiplied back by n-1). Compute scatter matrix for each cluster; sum all these matrices: this is Sw, _pooled_ within-cluster scatter matrix. Compute scatter matrix St for the whole sample. Sb=St-Sw.
Hi Florian.  Why not use a permutation test on the difference between the chi squares?
Why do you want to take derivative???
If you want to deal with individual data points you really need a multi-level model here and standard ANOVA isn't appropriate.  To do an ANOVA then you should be calculating the growth rate and doing an ANOVA on the rates.  That would yield 6 data points for each condition.
+1 from me--domain-independent answers are particularly useful here. It seems to me that in interdisciplinary domains, the "why" is often ignored because of the (understandable) emphasis on practical application.
So that we can understand this question, please tell us what it means for predictors to be "strong" or "weak."
Thank you very much Sven Hohenstein.
@whuber, indeed it is. Rahman _et. al._ give simulation values for $n=10,20,\\ldots,100$ in table 2 of this paper: http://math.usm.my/bulletin/pdf/v29n1/v29n1p2.pdf , although, again, it does not address 'case 4' testing, just 'case 1'.
Should that $F_i$ in the 4th line of the problem statement be $f_i$?
@guy The multiplication of a vector/matrix by a scalar does not work, but the assignement such as in the OP works.
The same way as for an unbounded variable - setting integration or summation limits appropriately.
No need of "many" exponential variates to be Gamma.
In addition, since you will probably first need to address the statistical issues involved (operationalizing "similarity") before you can try solving the problem in R, I flagged this question for migration to CrossValidated.
"...conditional probability tables used in this approach does not correspond to stochastic or deterministic relationships in Winbugs": How not?
@JeffreyEvans Thanks for suggestions but my response variable is binary. Would it be appropriate to fit a ZIP as Poisson models are mainly for count data.
Thank you @David Marx. I update part of the output of Weka. I didn't know whether Weka had a way to "transform" a test set based on previous eigenvectors or if I had to do it by myself. I think it's second option. Thanks : )
This means that, after controlling for the covariate (that is, holding it constant) the other independent variables are no longer significant.
One thing that might help you visualize what's happening in the plots is to jitter in both y and x (note that if it is in completed years, as is typically the case, Age is also discrete). Which is to say, add a small amount of 0 mean uniform (say) noise in both directions, scaled so that the fuzzed data is still distinct if the x or y values differ (that is, the width is less than half the gap between values)
I hope the discussion in comments under the question is now sufficient to cover that issue for you.
@whuber Thank you for the references, I will take a look at them.
Consider complex number as two separate variables, and that way remove i from all your equations. Otherwise it will be a nightmare...
As this is basically standard bookwork, please add the `self-study` tag. You might like to read its [tag wiki info](http://stats.stackexchange.com/tags/self-study/info).
Anyway, I trained a classifier using 20/80 prior probability, and classified testing data. However, sensitivity was very low while specificity is much high. When I used the default prior probability (50% vs 50%) regardless of the number of samples for each class, sensitivity and specificity were quite resonable considering classification accuracy (= (sensitivity + specificity)/2). If I use 50/50 prior probability to train a classifier, is it wrong in this case? Could you please give me some advice about that?
@steffen - Yes, I assume the question means you only see the test feature if you have previously opted-in.
:) I think I am starting to understand your situation better. You say you have 8 tests, and you say that the SD of group 1 is low. Are you combining the results of the 8 tests into one composite measure, or are you doing 8 separate comparisons - 1 per test? And when you say "test result", is that like a total score across a number of stimuli/questions, or is each test just one question/stimulus? An example of the type of data you have would be really useful.
Small point: I'd say a **proportion** or a **fraction** between 0 and 1, not a percentage. You have logic on your side, as 100% means 100/100, but nevertheless in practice expressions such as "a percentage between 0 and 1" will sound odd to many readers, even mathematically competent readers. I don't know any software that supports the idea that percentage is just a display format. In practice, you have to multiply by 100 to work with percents, or divide by 100 to work with proportions, and that dominates attitudes.
You can't take the log of 0, so you can't keep that data point if you want to take logs.
We have a failure to communicate. When you wrote "*My calculation is for the probability that the phrase is written by writer 1:*", and then a formula, *where did that formula come from*?
@cardinal, I've decided to mention the fact that, if your estimates are regularized then centering/scaling can have an impact. I resisted at first because I thought it would begin a long digression that may confuse those who are not familiar with regularizing but I found I could address it with relatively little space. Thanks--
I'm afraid SAS is not the tool of choice in my setting nor do I have familiarity with it. Besides, I'm looking for some approach, rather than for a cookbook. Let's say I'm after something that's more on the mathematical and modeling side of things.
Thanks for adding to this. I did not want to oversimplify things!
(+1) Welcome to the site, Mark.
@whuber I am equating them, it is a 2-D histogram. I do have access to the underlying data.
ProbablyWrong, take a close look at the question in your comment: because it deals with *women*, it is making an additional assumption about independence *conditional* on gender. The assumption of (unconditional) independence of hair and blood type does not mention gender at all, so to understand what it means, *erase the cross-hatching from the figures.*  This, I hope, indicates why we have the flexibility to situate the cross-hatching wherever we like within the upper and lower rectangles.
Let me correct myself: It would be correct to say that the chance of your lower limit to be bigger than the true value is only 2.5%, assuming normality of your sample mean. Which is a different phrasing (the chance is about the limit, not about A), but boils down to the same.
I agree, this looks like a great tool. I've bookmarked it for when I get home.
@whuber: I need a more didactic approach, with example and so on
Hey Mariana, thanks for your answer, but we had a misunderstanding: I by "kernel methods" mean methods such as the Support vector machine using the "kernel trick", not kernel smoothing methods.
@Tal Galili You are welcome Tal! If you find out some other method for choosing the number of clusters, share it with us.
1 - Yes, that's true. Problem is that I have a lack of relevant data to correct for it, and need some tip of whether or not my inference now is likely to be "less valid" when correcting. 2 - Yes.
(+1) Nice response and very clean R code!
+6, this is really great, & will help people understand this topic better in the future. I was going to point out my other answer (which I agree is consistent w/ yours here); it might be helpful if people want to try to visualize these things from a different perspective.
If you want $P(\\bar X > 26)$ (as your text says), why write $P( X < 26)$ (as your mathematics has)? That's wrong in two different ways. Then you got the standard error of the mean wrong when you standardized.
What is the dependence? E.g. Summing over the N trials must equal K? There must be an even number of 'true' results, etc. Once you define the kind of dependence it will be possible to write down the actual likelihood more concretely.
Well, thats maybe the answer, actually it can be seen from the report that the authors's knowledge in statistcs are not very deep, anyway if there is not some technique term that i happen to be unwared of, then thats OK.
One way would be to calculate the 3x3 covariance matrix and then invert it. That is sometimes called the precision matrix or Fisher matrix. If the off diagonal Y-Z term is zero or close to zero then Y and Z are independent and just conditionally dependent through X. I don't know off-hand whether this assumes normally distributed data. Would have to look into it more.
If the units are different, you might also consider normalizing your data to be in standard deviation units.
If you really do need the mean after back transformation, there are methods for this.  See a post on Rob Hyndman's [blog](http://robjhyndman.com/hyndsight/backtransforming/)
+1. thanks! I am also wondering how to generate the data as described in the text "expose the oracle". Could you also please add that, instead of using the data from the website?
Your question is quite incomplete.  What is your ultimate goal?  Why do you care about the central limit theorem in particular, and on using asymptotic methods in finite sample sizes in general?
If your values are constant, even for a small sample, they're clearly not normal; why would you even try to test what you already know for certain? To apply any test, you should think about your situation and what assumptions might be reasonable for it.
Because I don't follow what he has done completely. Also erfc is fine
(ctd)... but even if I had not understood that it could be done in the general case (though I mention the general case several times above), mpiktas' solution would have immediately convinced me otherwise.
Thanks Scortchi. There was also 1 more related Q in my post...How do I chose the predictor that will become my intercept?  Some might say, it doesn't matter but here in LR, it does as all other 'exponentiated coefficients -1' will be compared to it. I don't want to chose ADULT as the reference bin as I want to show that smoking increases as TEENAGERS become ADULTS.  For such a requirement, is it fine if I make TEENAGERS my intercept bin?  This way if I get a positive coefficient for ADULTS, it helps me show that odds have increased.  ADULTS bin has the highest observation count though.
@gung If the indenting was much smaller, I'd probably be inclined to put up with the renumbering-hazard and use it. As it stands, I usually find it distracting when I see it. But in this particular case I was looking to make them look more like subheadings, so I'd probably have avoided it even if I wasn't put off by the behavior. (On the other hand, on the occasions other people have decided to edit my lists to make it do the indenting, I generally leave them to please themselves. If they want to make it look a certain way, they can as long as it doesn't alter the meaning.)
"Individual level monthly predictions" in persons whom you've observed previous data or persons at initial assessment? I.E. are you obtaining $\\widehat{Y_{i=12}} = f(X_{i=12, 11, \\cdots, 1}, Y_{i=11, 10, \\cdots, 1}$ or $\\widehat{Y_{i=I}} = f(X_{i=I, I-1, \\cdots, 1}, Y_{i=I-1, I-2, \\cdots, 1}$ or $\\widehat{Y_{i}} = f(X_{i})$ or ... please clarify as prediction from longitudinal is not a hand-wavy matter.
I am dealing with a similar problem, I guess you can use `pgmm` from **plm** package but as your response variable is binary I don't know exactly how to do it. Maybe others can elaborate... (And yes you are right: my understanding is whenever you have an endogenous variable, in this case the lagged value, you can't use REML to estimate because it is biased, so you need to use GMM.)
Have you tried to change your initial values when applying `optim`?
According to SciPy document [here](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html), the Scott rule is: n**(-1./(d+4)). By looking at the code, I found I misunderstood the rule as same as "scotts_factor".  You are right that the bandwidth is far too large.  I will open a new question about numerical bandwidth selection. Thanks.
any comments or suggestions ?
Thanks! Really great answer, however I needed to use a normal approximation. I completely understand and agree with your solution though.
My factors fall into the second category.  When I dummy them, I find myself with roughly 2000 variables, and only a few continuous variables.  I can run a linear or generalized linear model on this system.  Are there other approaches you recommend?
In a lme-call, random = ~1|subject stands for a mixed model with random intercept. In this case a random intercept makes sense if you are assuming, that every subject has a specific tumor volume (e.g. because every mouse has a specific body size or is in different stage of tumor) at the beginning of the study.
Very closely related: http://stats.stackexchange.com/questions/24936/estimate-the-population-variance-from-a-set-of-means/25079#25079.  That is an *identical* question but the answer focuses on unbiased minimum-variance estimators rather than Maximum Likelihood estimators.
As for subjectivity and tradeoffs for priors: I'm interested in exploring how different people address this. The scenario is real enough: my aquarium has both (ii) breeding and (iii) non-breeding critters. For $n_0$ I know how many I bought! Counting critters in someone else's aquarium is (i). My analysis is curious not serious - no big deal if I "choose a bad prior" - yet I can see an equivalent problem may be of vital importance to someone else. I wonder what assumptions they make, priors they consider - particularly for the baffling case (i) - and which tradeoffs lead to their final choice.
@rubergly I agree that among-subjects would be more grammatically proper, but I have never heard it in an experimental science context. I've only ever heard/used between-subjects.
What do you understand "evidence" to be here? Do you have an exact definition in mind?
If death is from a recurrence of the disease that went undetected that seems to me to be a different story.  If couldn't happen without the disease reoccurring.  Then I think you should count the death as a recurrence of the disease.  Of course the subject could not have been disease free until death,  The recurrence must have happened earlier.  But diagnosis of recurrence happens some unknown time after the actual recurrence anyway.
Thanks, although these are software libraries and I was looking for standalone software.
Thank you! That clears up more than a few things. Loving the forecast package btw!
Well, it is a best practise to try different random initializations, instead of using the first k objects (which, when the data set is sorted, are biased). And it very often even results in quite different output, so it is sensible to try different initializations (whether or not MacQueen proposed that) ...
+1, this is really excellent.  I have one nitpick, however: I would not use the phrase "error term" in your last paragraph, as it is often used to mean something else; I would use 'standard error of the sampling distribution', or just 'standard error', instead.
You might ask that as a new question, @rhombidodecahedron. Briefly, $E(\\log z)\\ne \\log E(z)$.
The key nuance here, is whether you have "planned comparisons" (as JoelW. discusses), or "post-hoc comparisons" (like you mention). If your comparisons were decided a-priori (& they aren't just to look at all pairwise contrasts), then JoelW.'s advice is good, but if you gathered some data, looked at it, noticed that some contrasts looked interesting & decided to test *those*, then you need to exercise more caution: eg, no post-hoc tests w/ non-significant ANOVA, & use alpha adjustments, etc.
Not critically important, but your code will be cleaner and more robust if you use `data=P` and get rid of all the `P$`s -- e.g. `lmer(vikt ~ lweek + laktation + yr + (1 | fgrupp), data=P)`
I don't think you've clearly defined the weight, but I'm guessing it's the number of times $j$ followed $i$. And perhaps you meant $W_{ij}$. If I'm right, then maybe you're just looking for a first order Markov model. More useful terms might be stochastic matrix and transition matrix.
You can find some help here: http://stats.stackexchange.com/a/86424/38080
Both ways you mentioned are equivalent ;-)
Could you explain how the K-S test--which is intended for sets of real numbers assumed to be iid outcomes of a continuous random variable--would be applied to this *spatial* dataset?
Your interpretation of p-values is suspect; 0.27 does not mean there is no unit root and 0.017 does not mean there is; these are measures of evidence (in one interpretation, anyway), where a smaller p-value means that there is more evidence for a unit root.
I think your question needs some editing but I'd like to ask first: (1) "because the dependent is bivariate", I think you meant "binomial" (the outcome is treatment = yes or no, right?); (2) what is the measurement scale for your items (binary, likert, categorical, a mix thereof?) because the parallel to optimal scaling is not clear at all; (3) if I understand correctly, you have 15 predictors (items scores) and 1 binary outcome, which might fit within a GLM (whatever the link function you choose, logit or probit) -- please confirm. (And don't forget to register your account.)
@user603, I've updated my answer, but to your previous question, since I've composed the update with interruptions. The new update helps a bit, but could you write the exact model you are trying to fit together with restrictions? LateX instead of code would be very helpful.
Thanks! I understand how to use Pearson between 2 users, but what about the whole set of users? I need a global measure of concordance, not just between 2 users.
What are $X$ and $Y$?
I wanted to start with Gelman et al. But in the introduction they have said that we will only be dealing with the practical aspects of Bayesian Data Analysis, not its philosophy. I am interested learning the core foundations of it, before learning its practical usage. I have read the first three chapters and I am really enjoying Jaynes's book. I just feel unsure about some of my solutions to the exercises and I keep on wondering if what I have done is correct or the most efficient way to do them. I am a graduate student with an undergraduate in Physics, so I feel I have the background.
Are A and B values paired?
All comments posted here have been moved to a [dedicated chat room](http://chat.stackexchange.com/rooms/3901/discussion-on-mle-frequentist-or-bayesian). If someone has difficulty to join this room, and in this case only, please flag for moderator attention.  No further comments will be accepted.
Factor analysis and principal component analysis do not eliminate *any* variables: Both create linear combinations of *all* the variables entered, but some will be more important than others. Nor is the goal of either method to make certain variables unimportant on all factors/components, rather, it is to get a simple structure in which variables load highly on one factor/component and very lowly on the others.
Are you looking for clustering, in which many points will aggregate together, or simply pairing, in which a point is testing against a single other point? I read your original question as a search for pairs, but perhaps what you're looking for is called "cluster analysis," or as @Anony-Mousse writes, "subspace clustering."
Bias in statistics comes up in several ways. The bias of an estimator is the difference between its expected value and the thing it estimates. However, bias in the sense the term is being used in the paper - is defined in the paper, 2nd paragraph of p522, but it seems somewhat specialized to the application. You'd need to clarify in what way that definition was inadequate.
AUC *should* be between 0.5 and 1. It can actually take any value in [0, 1], but values below 0.5 indicate that you could get a better classifier by inverting class labels.
Further, you don't need to show something *isn't* normal to apply Kruskal-Wallis.
High dimensions makes visualization difficult for any variables (continuous or binary).  But projections into best separating 2D space should be helpful which is what gui11aume suggests.
@whuber: I am forming groups based on the values of $x$. For example all people with $x = 1.5$ form a group, all people with $x= 2.4534$ form a group etc... and then I am looking at some outcome event like death and choosing people in group $x = 2.35$ (for example) as the reference level.
A couple of points (maybe not quite worthy of an answer, so I'll make it a comment for now): (1) as far as I know, GLMM with a Gaussian family *is* a standard LMM, unless you use a non-standard (i.e. non-identity) link; (2) REML is not commonly implemented for GLMMs (and in particular not in `lme4`); I guess you can use it with `glmmPQL` from the R `MASS` package. *If* it is implemented, I would say that the rules for ML vs REML are more or less the same as those for LMMs (i.e. don't compare REML fits with different fixed effects)
For several hundred more questions like this, please [search our site](http://stats.stackexchange.com/search?q=normal+probability+problem).
Cross-posted to mathoverflow http://mathoverflow.net/questions/105596
@Benjamin I see from a clarification on the identical question posted to reddit that the OP does appear to intend it the way you interpret it. My apologies.
To be honest, this may be something more appropriate for StackOverflow.
I would have thought AUC is not best here because there is small loss for false negative, but large loss for false positive.
Couple questions: do you know the size of each subpopulation? Also, some quantiles may not be available for some sample sizes due to discreteness. Are you assuming we just have SOME set of quantiles for each partition, but those may be different for each?
I showed the answer must be $2/3$ in all cases: there is no calculation left to be done. You can indeed reduce the fraction you write down by expanding out all the factorials and canceling what you can: you will still get $2/3$ as the answer :-). This is a beautiful example of a "combinatorial proof." Other simple examples include showing $\\binom{n}{k}=\\binom{n}{n-k}$ by observing every $k$ subset of an $n$ set corresponds to an $n-k$ subset; and $\\sum_{k=0}^n\\binom{n}{k}=2^n$, which follows because both sides count all the subsets of an $n$ set. You don't have to do any arithmetic!
Alexis, I appreciate your comment. But I would like to add that my goal isn't to test the correlation, but to find the dispersion of this bivariate frequency. For instance, is my data centered in one of the corners. For the ordinal data, I ran the wilcoxon test for examine the rank difference between x and y, but it doesn't produce a good result for me since it is testing the median.
Welcome to this site! Could you share some details about your study and your data set with us? This should stimulate the interest of users and improve the quality of their possible responses.
But that identity does not imply that dividing the integrand by $e^{-\\lambda}\\lambda^\\alpha$ and multiplying the result by $\\Gamma(\\alpha+1)$ cancel out - note that $\\int (f(x) / g(x)) dx$ is not generally equal to $\\int f(x) dx~/ \\int g(x) dx$.