tttjjj/elt_bench huggingface dataset:
@mropa: That is my favorite statistics text: enjoy it!
+1 if only all answers were so complete!
Perhaps post your code for generating simulated data; if you have any rjags code for calling the function, and what error messages you are getting.
Related question: http://stats.stackexchange.com/questions/44838/how-are-the-standard-errors-of-coefficients-calculated-in-a-regression/44841#44841
@Scortchi I have a dataset of 77000 parts & its usage over 5yrs now  I need to classify them into 6 groups like mostly used/less used/recently used/not used. Since 90% of the data fell in one group I'm wondering how I can group them?
There is not regression here, just $(X_{1t+1}-X_{1t}) - (X_{2t+1}-X_{2t})$.
@gung: You are missing the point that the individual $\\lambda_i$ only affect the distribution through their sum and how many there are.  It doesn't matter what particular values they take: $\\lambda_1=1,  \\lambda_2=2, \\lambda_3=9$ will give the same result as $\\lambda_1=4,  \\lambda_2=4, \\lambda_3=4$.
Wikipedia: [Binomial proportion confidence interval](http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval)
Allright, I am satisfied with the KL-based bounds I found.  Thank you for your help, @robin girard!
+1 since `aggregate` is an overshoot here and plyr is slow.
Is your question about how to use the R code properly, or about how to build such trees more generally? If the former, it would be off-topic for CV (see our [help center](http://stats.stackexchange.com/help/on-topic)), but could be on-topic on [Stack Overflow](http://stackoverflow.com/) if it has a [reproducible example](http://stackoverflow.com/q/5963269/1217536). Please clarify your question. If it is about R code, add a reproducible example & we can migrate your Q to SO for you (*please don't cross-post, though*).
@Joel You are implicitly raising interesting questions about what "bias" and ["representative"](http://stats.oecd.org/glossary/detail.asp?ID=3831) mean. It's impossible to address them in the space allowed for a comment, so I will limit my reply. First, how do you know "the sample is clearly not representative"? Second, please consider that except when a very small number of variables are involved, it's generally impossible for any non-census sample to be fully representative of a population. Third, you may be thinking of procedures such as stratified sampling.
the first two factors seem to make sense and are useful; factors 3 and 4 are not very useful.
Each of the link functions is an actual function whose value can be computed.  Try looking up the formula for each of your links (for example, the logit is clearly defined on wikipedia), and then graph each function to see its "physical" meaning.
@Nick Riches: Happy to help.
Stepwise regression and feature selection exploration are problematic for reasons detailed elsewhere on this site.  And I'm not quite convinced that number of unique patients per day is the way to go, as opposed to looking at time trends in patient characteristics (without a dependent variable).
No, that doesn't mean my assumption is wrong; it could be that this particular group got unlucky.  However, my assumption likely *is* wrong, but there is no way to know.
Exactly how did you "reduce it to 2D"?  Don't you have the original data?
Here's a one-liner. It's a definition that log transformation is needed to make a lognormal normal; it is a good approximation that a cube root makes a gamma normal (Wilson-Hilferty are two words for the wise); the distribution needing the stronger transformation is "further" from the normal or Gaussian.
I am not sure what do you mean to include in the conditioning elements, but let's say that it is, uniquely _defined_. Does that mean that the actual "correct" value is knowable? Or in any case, it is only imperfectly estimable, by different methods that will give different results, although all of them being perfectly compatible with "rationality"?
Very closely related: [How does the standard error work?](http://stats.stackexchange.com/questions/33547/).
Actually, I am okay with preference as a reason. I guess highlighting it (as we've just done) is probably sufficient.
@noworries. Yes, they are the boundaries of the parameter region. A value close to zero is not a problem -- it indicates the associated state variable is extremely stable and essentially does not change over time. A value close to one usually indicates the model is trying to adapt quickly to each new observation which probably means the model is mis-specified.
It isn't clear what data you have and what you are trying to find out. You may find [How to Ask a Statistics Question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) helpful
Second, I don't see how the a, b, and the indicator function for $U$ being between them are relevant to the distribution of the sum of the $U$s.  What is the reason for thinking they are relevant?
see Beyer et al. under [when-is-nearest-neighbor-meaningful-today](http://stats.stackexchange.com/questions/6314/when-is-nearest-neighbor-meaningful-today): kNN is sometimes NOT meaningful.  What are your N, dim, k ?
I appreciate your effort, but your post is hard to follow because it takes a while to get going and then founders in some misconceptions. Non-central chi-squared distributions are irrelevant here: they appear in power calculations. More importantly, a chi-squared distribution is *not* simply the distribution of the square of any random variable and indeed the square of a Gamma variate never has a chi-squared distribution. As far as vagueness goes, please heed the comment by @Peter Ellis, who points out the necessity of being clear about what kind of "chi-square test" you are talking about.
Possible duplicate http://stats.stackexchange.com/questions/30679/is-it-acceptable-to-use-cronbachs-alpha-to-assess-reliability-of-questionnaire
Fisher's exact test and Pearson's chi-squares test the null hypothesis that all 5 methods are equally effective against the alternative that at least 1 is better than the others. The p-values tell me that the null is rejected. So, if I want to find out which methods is actually better than the others won't I have to do 10 pairwise comparisons?
In validating the model, I need the ROC curve for its discriminatory ability and the calibration curve to measure its calibration.
I can help you to give a shot, but firstly you need to follow two steps: i) install R Studio on your pc, ii) provide a reproducible example: http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example
Laplace never took measure theory from a statistics professor ;)
Nice theoretical proof. Without normalization, the odds ratios do have a really practical meaning. However, sometimes practitioners use those odds ratios to assess the significance of that feature, and it is somehow misleading because a feature may be significant but may also vary on a large range and thus have a low odds ratio. With normalization they are immediately comparable even if loosing their practical meaning. Of course, a statistical test must be made to assess the significance of a feature.
You can alter the first line of the `foo` function to:  `mod <- lm(Rare25 ~ b1(epi.i, bp) + b2(epi.i, bp) + l.dist.mrc)' for a not-very-generalizable solution.
Unfortunately, most of the dyad stuff assumes independent dyads where each partner is part of only one dyad. But thanks for the link to that site and its info.
One big issue: R's help system is good when you know what you want details on. It's not so good if you don't have a clue what you need to do. So do find a resource that shows you the basics like: how to read in data, how to save your work so far, how to make and save graphs, how to get help, etc. For example, if you say, "Hey, I want to forecast something, so I'll type in ?forecast to get help on that." You'll come up with nothing. In R, the usual term is "predict". I LOVE R, but have to keep it real. Also, realize R is a programming language, unlike SAS or SPSS: harder at first, better later.
Forgot to include the beta coeff.
+1, library(HH) is definitely the way to go.  But something has gone wrong with your second last plot in the ordering of agree/disagree etc.
World of Warcraft eh?
It might be worth pointing out that least squares is unlikely to be an appropriate method of fitting Gaussians to data: when the data are expected to be approximately Gaussian, it's rarely the case that the departures are uniform in size across the entire range.  Therefore, *expect* the fit to be visibly bad.
@JasonSamuels: do you have any guess as to what mechanism removed these missing cells in your design matrix? For example are the cells missing at random (in the terminology of [Rubin 1976](http://people.csail.mit.edu/jrennie/trg/papers/rubin-missing-76.pdf))?
@Glen The question is "Can I conclude that the function distribution is symmetric about c?" I believe this post responds directly and correctly to that.
In your opinion which is the the best way to remove bias in the odds?
Well upper tail is by definition P(X>x).
Can you please share the sample size, the structure of the model (# of predictors/interactions, etc.) and how you concluded that the model does not fit?
Could you please formulate the null and alternative hypotheses of such test and explain why they would be useful?
Thanks StasK. Interestingly the solution I got for my problem is $n = \\sqrt{\\text{Var}(x_i)/c}$ as you said! But I don't know how to interpret it: the numerator is in the one unit of $x$ squared and the denominator is CPU time.
@SebastianRaschka, I had shown LDA of iris data results, obtained in SPSS, here: http://stats.stackexchange.com/a/83114/3277
Descriptions of the input and the output are necessary in order for this question to fit within our format.  Please see the [help] for more information.
Similarly to the respondents to your question, I think you should try using linear regression.  If you want to try fitting the model in R, the function is lm().  But any statistical software should have such a function.
MARS is a useful algorithm, but terrible for this application. {segmented} should be better. Or if you know the number of breakpoints segmented regression
@PeterFlom Maybe you mean Sir David Cox??
$Z$ is in fact a scalar. In this case, an integer from $1$ to $K$, corresponding to the $z^{th}$ topic. Sampling from that multinomial distribution is analogous to rolling a die with $K$ sides. It is perhaps more appropriate to call this a *categorical* distribution rather than a multinomial, but they are equivalent when considering draws with only one outcome.
@Whuber, added QQ plots to question
Are you perhaps looking at generative versus discriminative models? Machine Learning leans towards discriminative models and techniques.
I would think it is best to ignore this.  But it would help to have an exact definition of "threshold".  Is it just an intercept estimate?  This is highly nonstandard nomenclature.
Yes, that is of course one possibility, but tnere sure are others! references to discussions?
What is wrong with "After controlling for child age, sex, trauma history and maternal PTSD, significant evidence was found that both maternal depressive symptoms and quality of parent-child relationship were related to child behvior following trauma exposure".  Perhaps it relates to what you mean by "uniquely predicts", which I don't quite understand.
you are very welcome.
Would be helpful to actually know more about your data and confounders... We deal with a similar scenario in our dataset (observational pharmacoepidemiological data) and use an adapted form of Propensity Score Matching to match to exposed / unexposed.
He offered me a small "hint"- he said you use binomial, and that the 12 weeks is n. I'm guessing he expected us to do a binomial with p=10% and n=12 weeks, but that is obviously (to me, not to him) nonsensical.
I don't understand what you are asking about the determinant. The multivariate normal distribution is written with the covariance matrix http://en.wikipedia.org/wiki/Multivariate_normal_distribution
@neverKnowsBest Deterministic. It's a structural mechanics simulation. So no need to repeat points.
I'm entirely convinced about this answer, but I don't have a better answer yet. Some of my friends think this should be related to http://en.wikipedia.org/wiki/Bayesian_statistics
Some hints are provided by other homework questions related to the normal distribution, such as http://stats.stackexchange.com/q/5504/919 .
@Ben I believe Cross Validated is as good a place as any for this question, because its error model is unrealistic: the OP may benefit from comments and replies that show how to construct (far) better error models for DEMs.
This is a good start, thank you!
@suncoolsu Almost definitely Gene Expression Analysis data.
Just a word of caution: some of the scores have much lower variability associated with them, so any test of mean would have a much higher power. You might wind up rejecting the null hypothesis for the low-variability groups, because the mean is slightly different from 0, while there is almost no chance of rejection with the high variability groups unless the bias is glaring.
I agree that combining predictors, even collinear ones, runs too high a risk of losing valuable information.
When you change "answer a question correctly" to "is female," "answered the previous question correctly" to "has blood type AX3," and "answered the one before the previous question correctly" to "has long hair," and alter 40% to 80% and 35% to 90%, then you are asking close variants of http://stats.stackexchange.com/questions/30842/what-is-the-probability-that-this-person-is-female. The correct answers in that thread, and some of the comments, will show you what can be known here and will indicate what additional assumptions are needed to obtain unique answers.
(+1) I just thought it would be nice if one can see the distribution. Sorry, I didn't want to mess with your answer ;)
Thanks for the quick answer. By multilevel model you mean a mixed effects model? You're right I should have added that to the list, in this case the confounding variables would be included as fixed effects? However if I think they may also be predictive and want information about that, then linear regression is probably preferred?
@Andy: Updated my answer accordingly. Of course you have to know the correct class to estimate the missclassified points. In every iteration, you use only a sample of the training data for actual training and the rest to estimate the quality/missclassifications.
Thanks John! That's what I thought but I wasn't certain and the results are very different depending on the order of operations.
Can you provide a simple overview of the data to clarify what you mean by "_frequencies of binary patterns of polar and non-polar residues_"?  For example (and I have no idea if I am interpreting your data correctly) ... I have five sets of hairspins.  For each set, I have the number of polar residues and the number of non-polar residues.  Set A has 120 polar residues and 45 non-polar residues, etc.
Have you simulated it and compared results?
The index $i$ in the analysis shown in **Edit 2** runs through the bins in each _modeled and observed histograms_ (both histograms have obviously the same number of bins) and the parameter $\\theta$ is what I can change to obtain different _model histograms_. What I'm not understanding is how to calculate that $\\nu_i(\\theta)$ parameter.
Unless you can give us some context even to understand what is meant by "equivalent events," Georgia, we will need to close this as an incomprehensible question.
@dsimcha - if all you're doing is building a binary classifier, why would you care about anything other than the predictions it makes?  Surely this has to be the number 1 priority, with the contribution from each variable only needed to help understand *why* it is the best classifier.  A way you can get the contribution for each variable, is to re-calculate $F$ removing each variable, one at a time, and check how this compares to the $F$ for the chosen model ($F_{chosen}-F_{(-j)}$ is like a marginal effect for the *jth* variable on the predictive accuracy due to each variable)
Exactly *how* do these commands "not work"?
I couldn't have written this better :-) See [also](http://stats.stackexchange.com/q/6022/6432) [these](http://stats.stackexchange.com/q/12397/6432) [questions](http://stats.stackexchange.com/q/12930/6432).
Maybe I'm missing something, but can't you use Monte Carlo integration for this problem?
If I can help you with SAS I would be glad to. Please contact me at my email address available from info.
I'm not an expert, but maybe can be useful to see the capture-recapture approach
How are you obtaining the MLE? If your optimization routine returns the Hessian (e.g., `optim` in `R` usually produces this), your standard errors are the square root of the diagonal of the inverse of the negative of the Hessian. In R: `sqrt(diag(solve(-Hessian)))`. Depending on the software you use -- whether the routine is maximizing the likelihood or minimizing the negative of the likelihood -- you may or may not need the negative in front of the Hessian. Most of the routines in `R` minimize, thus the need for the negative.
Show your code, maybe? My guess is that you're obtaining PDF with copula, not the probability.
It may be worth observing that the finiteness of any even moment around any center $x_0$ implies the finiteness of that moment around any other center: that is why it is unnecessary to specify the center.
@baudolino  Pretty picture - what software do you use?
A phase locked loop (PLL) would probably be a better idea.
Negative binomial regression in Stata has precisely zero to do with least squares. The code for `nbreg` is accessible to you, assuming that you use Stata, and the manual documentation for `nbreg` is online at www.stata.com/manuals13/rnbreg.pdf even if you don't use Stata. If you think that negative binomial regression is based on least squares, it is not clear that you will understand any explanation any way; you need to learn more basic stuff first.
Compared to the message posted on [weka ml](http://comments.gmane.org/gmane.comp.ai.weka/25217) (notwithstanding the cross-post on [MetaOptimize](http://metaoptimize.com/qa/questions/5656/model-selection-in-weka-through-cross-validation-for-regression-problems)), do you mean you want to compare different regression models, or just to optimize a given model with e.g., cross-validation?
I don't follow where the sqrt(19) came from for the bound for the 95th percentile.
It sounds like you're looking to run a contrast.
I am looking for a tutorial about neural networks and deep neural networks, covering: - Architecture - Training - etc...
the exact command depends on which function you used to get the PCA decomposition...
Hi peter,thanks for your answer. I agree, we should used longitudinal techniques to deal with the correlation, I was curious to know whether there is any good summary measure that answers the hypothesis of learning effect without dealing with the correlation.
I say exponentially because I want to control the weight of old samples just like using weighted moving average. But I want the old samples to be less weighted, exponentially less, than newer samples
I don't mind giving you some R code but I understand that the site's policy is to guide toward a solution rather than solve the exercise for you. Where are you stuck? Did you check in your course notes/support material/textbook if you can find some material on interactions? Are you unsure about what an interaction is and why you need to add the corresponding term to the model? Or are you only looking for the right syntax to fit this model in R?
quiet literally the scatter: bind BU-OEU in a n by two matrix and compute the determinant of the co-variance matrix of that data matrix.
Sorry, but NN is not only ML. This use is a pure soft-computing, so I have to close it. This would fit on AI proposal on Area51, you can commit here: http://area51.stackexchange.com/proposals/6607/artificial-intelligence?referrer=K4nouwJjttuP1xPZSakPmQ2
@cardinal: So long as your observers are younger.  There is a loss of short-wavelength sensitivity with age (e.g. http://www.ncbi.nlm.nih.gov/pubmed/3230483).
Be careful that if you use a Variable selection before LDA in `Classifier(A,B)`, it will select parameters that discriminate best `A` from `B`, and you can't infer much about what would `C` be classified like in `Classifier(A,B)`. `p(A)` might be very high (even though it's `C`)...
I'm not aware of it being available within an aov object. I don't doubt that someone has coded it up in some package, but I don't know which & it would be just as fast to do it yourself as to try to figure out where it might exist elsewhere. On a different note, remember that this is *partial* $\\eta^2$, not $\\eta^2$, which would be $SS_{RC:HS}/SS_{tot}$ instead.
Henry, I found the Stack Overflow "site", Sorry, I should have looked before talking!!!
By definition of hclust algorithm, the nodes joined in at nearest the root/top of the tree have the largest distance from the others. Since distance is dissimilarity here, that means they're the most dissimilar to the others. That means they're more likely to be outliers (though that is not certain). Also, the choice of the linkage method in hclust will influence the shape of the tree and so the points that might look like outliers.
I have only heard it referred to as "dot notation"
Hi @ziggystar, 
In the point 1, $\\theta$ are the parameters (I've updated the question), and this is actually the usual MAP.
not a very sensible one; the population average slope with respect to attitude is assumed to be zero  ...
A drive-by downvoting with no comment?  Not cool, man.  What's the concern?
Are you asking whether there's a simple formula to compute quantiles of a *mixture* of normal distributions?  In this application, you would be asking for the quantiles (say) of shoulder span *regardless of age* based on the *age-specific* parameters.  Is this a correct interpretation?
One thing to consider is changing the penalty terms for different types of wrong classification. You don't say what software you are using, but I believe all good tree software should include ways to do this.
Welcome to the site, @syed. Would you mind expanding on your answer? For example, what "reasons" are you referring to? Note that *both* the t-test & ANOVA assume equal variances.
Why would your SAT/GPA/extracurricular example use a geometric mean rather than a weighted or scaled arithmetic mean? Why should an SAT or GPA of zero mean that the other two values become irrelevant (as a geometric mean would imply)? And what if (say) extracurricular-activities tend to cluster in a much narrower band than its theoretical range? It seems like it would make more sense to take an arithmetic mean of percentiles (or other adjusted values) than a geometric mean of raw values.
+1 Really cool answer!
Thanks for your help. Looking through [Stata's survival analysis manual](http://www.stata.com/manuals13/ststcox.pdf#ststcoxRemarksandexamples) pages 12-14, they show Cox models with and without the time-invariate component ("main effect"), differentiating between models fit with time-varying _covariates_ (without main effect) and time-varying _coefficients_ (with main effect). 
  
  
I think this covariate/coefficient distinction may be the source of my confusion. From the manual's suggestion, it seems like adding the main effect will only test the PH assumption of the coefficient.
Thank you very much for your answer, you say I can simplify it as a line too, $w'*x+b=0$ but it's a "strange" line, as a line I expect to see something like y = k*x+b, but here $y=0$, so it is just X-axis?
@prof_jack, it's not me to plan any harm, neither SPSS. But the issue of factor analysis is intricate and demanding enough, so one has to read some books or good articles before starting to do it.
Your false positive rate is going to depend on many factors. We need to know more about your model / binary classifier at a minimum. In the interim, (w/o meaning to sound snarky) the way to insure that you don't have any false positives is to classify all cases as negative.