tttjjj/elt_bench huggingface dataset:
Parameters are *constants*: they do not depend on the data!  Perhaps you might find a [discussion on likelihood and probability](http://stats.stackexchange.com/questions/2641/what-is-the-difference-between-likelihood-and-probability) helpful.
thank your for the explanation. i will definitely apply your suggestions to my continued experiment, I'm observing the ants in small colonies now instead of isolating each ant.
What do you mean by "comprised of", exactly? Do you mean that for C to be true both A and B must be true, and for A to be true, both A1 and A2 must be true? Are A1 and A2 dependent on, or independent of, each other, and also, are they dependent on/independent of B?
Based on your update, it isn't clear how you would have 7 responses for each person. It seems like you should have just 1, in which case you could use ordinary logistic regression instead of GLiMeM (ie `glm()` rather than `glmer()`).
I liked Systematic Reviews in Health Care, though I found it - as I often do with books with a clinical focus, that it was both needlessly critical of observational studies, and a bit blind to the weaknesses of poorly run clinical trials. It was helpful, but it requires some adaptation to be useful outside the RCT world.
In your specific example, South African race is clearly a confounding variable, affecting both whether mothers had medical aid (insurance) at birth and whether children were traced for this longitudinal study.  It is an example of Simpson's paradox (hence the article), but easily could not have been if one fewer White child who had medical aid had been traced.
Note that [the wikipedia page](https://en.wikipedia.org/wiki/Coefficient_of_determination) doesn't actually say that $R^2$ is *always* in [0, 1], just that it's in it in a particular type of case. In the "interpretation" section, it clearly says that values <0 and >1 are possible.
(ctd) ... It looks to me like a mixture of several not-so-heavy-tailed distributions may do better, or perhaps something roughly like a Pareto to fit the lower end and some much-shorter-tailed components near 250 and 8700-ish, and possibly a couple of lower ones as well.
@John, that's where I am going as well. One would perhaps simulate from the observed distribution, but keep going only until the candy total hits 324. Any simulations where this number was not reached exactly should be discarded. Then assign the first 7 numbers to children, the rest to adults. Now we can look at arbitrary "fairness" statistics.
Fundamentally, adding data generated by your model will not introduce any new information, and so can't possibly benefit (only by accident).
I would also add Larry Bretthorst's book on spectral analysis to your reference. can be downloaded for free [here](http://bayes.wustl.edu/glb/book.pdf).
In R you can also do a bootstrapped KS test http://sekhon.berkeley.edu/matching/ks.boot.html which gets rid of the continuity requirement
@chl In the situation of a two-sample t-test, it is often recommended that a confidence interval for the difference in means be given, regardless of whether the test is significant. If the ANOVA F-test is non-significant, would it be a good idea to give the Tukey HSD confidence intervals for differences in means?
This is the right approach.  The criterion usually is 1.5 IQRs away from the *quartiles*, not the median.  See Tukey's *EDA* text, for example.  (I believe he originated this rule, which he analyzes rigorously in a paper buried somewhere in his collected works.)  It becomes appropriate when the data are first re-expressed in a way that makes the distribution approximately symmetric about its median.  For simple ways to determine a re-expression, see *EDA* *op. cit.*
Dear Tom
How can I define the messages passed from this factor node to each Wij, and to alpha and betas? I don't know how exactly should I define messages: http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/How%20to%20add%20a%20new%20factor%20and%20message%20operators.aspx Like in this link how they are computed after we have the distribution after removing data i (\\i)? Thanks a lot
Thanks Macro! I took time to reply to your answer as I'm still grappling with the fact that bootstrapping really solve my problem. Once I understand the bootstraping & how it's solving my problem I'll mark yours (& Henry's) answer as correct answer.
If you have 70% zeros your data are not continuous. If they're continuous apart from the 0s that would be a mixed distribution.
I should note that I myself was taught and employed (http://www.ncbi.nlm.nih.gov/pubmed/19154749) the inappropriate practice of using ANCOVA to "control for"  a covariate with a known relationship with other predictor variables. It's only been through later self-directed study that I discovered the Miller & Chapman paper and the errors of my ways!
@John: at "larger numbers"->homo/heterophily: true, and this is also a good question. Without deeper thinking so far: a) people with a big reported network may be less selective with characteristics of their friends b) with less network may have more difficulties to get friendships c) friendships may be more important, d)...(?).  But actually this is only speculation; our small study shall not allow to investigate such questions seriously:it is an experimental study for teaching purposes on a fairly introductory level (thus,on the other hand,possible conceptual drawbacks are the interesting!)
The more I think about confidence intervals, the harder it is for me to think of what kind of question they can answer at a conceptual level that cannot be answered by asking for "the chance a true value is within an interval, given one's state of knowledge".  If I were to ask "what is the chance (conditional on my information) that the average income in 2010 was between 10,000 and 50,000?" I don't think the theory of confidence intervals can give an answer to this question.
It should be noted that this is annoyingly splitting hairs, but I think this is what the OP meant in his post.
@cardinal - Thanks a lot. This is really helping me. And thanks a lot for your patience with all my mistakes.
+1 nice explanation, but doesn't tell OP how to find the one correlation value to enter into G*Power.
EngrStudent, I am interested in understanding conceptually what happens in either case. I assumed (maybe incorrectly) that the process was the same in both cases but that the resulting values differ and differ by the basis functions used (but that the process was the same)
Are the data paired ? Otherwise there is no sense to compute a correlation. If the data are paired then type cor(mydata, use="pairwise.complete.obs")
Thanks Greg for your answer. Now I know at least that it is more complicated than I thought. I will talk to a statistician at my company.
It's a rather odd of theirs (the writers of that documentation) to use $\\sigma$ to represent *variance* rather than using the near universal $\\sigma^2$ for that ... odd to the point of being actively misleading. If one wishes to communicate, one doesn't break such a strong convention lightly.
For starters, I recommend you try variants of the following.1. Aggregate the usage data by hours/days. 2. From your training data, overlay plots of these time-series to see patterns within the two groups. 3. Explore various time-series/regression models that fit your data well. 4. Use the model coefficients as features for your classifier.
Although the question obliges you to simplify, drastically, your historical comments have to be taken with a pinch of salt. Fisher in practice often used, or appeared to use, fixed-level testing even while in broad discussions he also denounced it as naive or worse, while Neyman and Pearson's practice was also wider than their theoretical papers implied. I doubt that you can find Fisher writing of "signal-noise ratio".
Does a Distance correlation (http://en.wikipedia.org/wiki/Distance_correlation) sound like what I may need?

I need sets of identical values to return a perfect correlation, and sets of values at the opposing ends of the 0-10 scale, i.e x: 0,0,0 and y: 10,10,10 to return a perfect negative correlation, -1?.

I guess two sets of values with a random distance between each point should return low correlation, i.e 0?

I apologise for my complete lack of statistical knowledge, it is no doubt making it much more tricky to understand what I'm after :)
Hello Jeromy! What you descirbe is absolutely correct. But there is no way to create a model that measures goals/games. So I'm forced to the model above (goals as the dependent and games as the independant variable). I know that games is correlated with things like skill and that I've to explore this problem (omitted variables problem and Endogenity). But at the moment I'm wondering which of the two models above should be used!
`it shrinks my dataset from around 14400 to around 13000` So, you have about 10% of responses "DK". This isn't really much, so you could drop them to make your life easier. For, if you keep them, they can occur ambiguous: for somebody, DK = in-between, and for somebody, DK = not qualified to answer.
@JulienD.- which arma/arima model would fit best? I cannot make out from the acf graph..
Please see http://biostat.mc.vanderbilt.edu/wiki/pub/Main/FHHandouts/asa96.pdf
I know that I state the obvious, but not being able to reject the null hypothesis does not mean that it can be adopted
I've readded the homework tag [in accordance with its use on this site](http://meta.stats.stackexchange.com/questions/1459/the-homework-tag-and-homework-policy). Cheers. :-)
This answer makes me want to downvote my own answer below.
Suppose if the data is secondary, how does one identify if the nutrient intakes have already been adjusted?
A 'correct' answer is dependent on the model and the code/commands you are using. So, a brief description of both would be useful.
@abousetta Thanks for your comment, I will compare only the 7 most relevant ones then. How would you suggest doing it? Multiple logistic regression requires that I define one variable as the dependent variable, and the rest as independent/explanatory; should I then run 7 models (one for each variable as the DV)?
Fine specimen of science journalism, there...
Even if you can resolve the question without reference to the mathematical definition, doing it by using the definition is not simply make-work. You need to get used to using the mathematical definition, because often that's the only way such questions can be resolved, and if you don't know how it works, you'll be stuck. It's an important part of the tools for later work.
Yeah, you're right, here it's 2. @EricPeterson
SEE also related question http://stats.stackexchange.com/q/73065/3277
Hi Glen. Can you help me further? Is wilcoxon appropriate since each participant is contributing several pairs of data? see above. Thank you!
Mooks: it works for prediction intervals, but doesn't work (as is) for a CI for the mean.
Anomalous needs to be defined; since the outcomes (anomalous or not anomalous) is binary this problem should map to a formal T-test.  Furthermore, are we making assumptions about independence, and normality with the sample?
It is interesting to see that $U$ and $V$ turn out to be independent random variables. $V$ is, of course, a $\\chi^2$ random variable, while $U$ is a centered and scaled Beta random variable? Perhaps this latter too can be deduced directly?
A good prior is one that reflects your prior knowledge. On the other hand, if you're looking for a convenient prior, it looks to me (on a quick glance) like if you had a Weibull prior on the precision (inverse of scale) with the same shape parameter as the data, you could perhaps bring the prior in as pseudo-data, but I didn't follow the algebra through to check that it works. One thing to consider if you have a convenient-but-not-particularly-like-your-actual-prior distribution is the possibility of mixtures of such distributions.
I'd retain the most significant digit if you can (i.e. I'd report at least one significant digit rather than none) - though it hardly matters, since both are effectively 0, and when it's much smaller than 1, it hardly matters if it's $3\\times 10^{-3}$ or $2\\times 10^{-6}$ or whatever else, except that extremely low values might leave us wondering what the explanation for a much-smaller-than-expected-if-nothing-at-all-is-happening SS might be (there's a "mystery" for which other explanations than random variation might be needed - perhaps a failure of assumptions)
The question from CS got [migrated here](http://stats.stackexchange.com/questions/60257/markov-chain-getting-stuck-due-to-insufficient-data-samples), flagging to close as duplicate.
@mhoenicka Your last comment suggest that John was right which makes this response eligible for the "green mark" (i.e., you can check this answer as accepted since it solved your problem).
Thanks for your reply. So it means that exogeneity should basically be justified as a theoretical argument? Meaning that it should be very rarely assumed in time series.
@All: Please note the preface of the question: the OP is "trying to *generate* hypotheses."  Thus, he is attempting to use p-values not as such, but as a way to discriminate hypotheses worthy of followup investigation from others that are less likely to pan out.  The comments and replies that are emerging early on focus on using p-values for more formal purposes, which seems to miss the point.  (Or possibly *I* am missing the point by taking the question at face value...)
I think you misunderstood. The mean daily production is 3750, and I know how to calculate a confidence interval around that mean. What I don't know how to do is to adapt that confidence interval to an aggregate monthly production of 60000.
You can import the tsa namespace itself, by `import statsmodels.tsa.api as tsa` then do `tsa.ARMA`.
+1, nice answer. If you use quadratic weights, you should expect the weighted Kappa answers and ICC answers to be equivalent (e.g. p. 187 of Streiner and Norman, "Health Measurement Scales" -- "If this [quadratic] weighting system is used, then the weighted kappa is exactly identical to the intraclass correlation coefficient" which cites Fleiss, J. L. and Cohen, J. (1973) "The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability" in Educational and Psychological Measurement, Vol. 33 pp. 613–619)
(+1) Sorry. As I was composing my answer, I didn't get an update that you'd submitted one as well. :)
I think we're confused too - how can the same individual have multiple decay rates? One observation can't have multiple values for a single dependent variable. I think you should treat these as separate observations. Anyway, there should be no problem using $\\bar{M}$ as your survival estimate for that observation.
As the sum of independant variables I would tend to think that the normal is going to be the best approximation for you distribution. Again, if you explicit any other assumptions that you know, more relevant guesses can be suggested.
I'm confused. Isn't that precisely the definition of a chi-sq variate with 3 df? If you are worried about the sigma being different than unity, it's just a constant and so easily factors out. http://en.wikipedia.org/wiki/Chi-square_distribution
@whuber: yes, but it was tagged with invariance and I've seen the two words used as synonyms. Just wanted to avoid possible misunderstandings :)
People around here are nice, I have never seen a “let me google that for you” answer but many surprisingly thorough and insightful answers to what seemed at first to be basic questions. Unfortunately, I can't help you with yours but it seems quite relevant so I am happily voting it up.
There's a forum dedicated to spatial analysis. Maybe mention this post there (be sure to mention that you are cross posting). http://gis.stackexchange.com/
did you look at `?pvalues` in the `lme4` package ... ? in addition to `afex::mixed`, it also suggests `car::Anova` and `lmerTest::anova` ...
Thank you for your reply =)
Thanks very much for the great answer!
Also of possible interest: "Relationship between eigendecomposition and singular value decomposition" http://math.stackexchange.com/questions/28036/relationship-between-eigendecomposition-and-singular-value-decomposition
Ah, thank you for clarifying. Yes, I think I fried my brain sifting through this paper all day and somehow only focused on the equality, but neglected the starting estimate (which certainly makes this look more tame). I hope to make good progress with this tomorrow, though I may be back again to ask questions if the formula isn't doing what I expect. Thank you!!
This question appears to be rather broad.  Could you perhaps refocus it on a problem or issue that you are facing?  Please see our FAQ for more information about what kinds of questions are appropriate for this site.
+1 for the use in everyday life.
Am a statistician-and hence open to all sub-domains and also do a fair amount of machine learning.
Thank you very much , I will read the paper you included then I will come back.
(1) Prognostic modeling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Statistics in Medicine, 19, 1059-1079 (2) true for variables with df1, based on aic definition. But could be lower if your degrees of freedom of variables higher
ARI and AMI give you agreement between 2 clusterings. If you have some ground truth you can use it as a reference to compute absolute scores against it. Otherwise you will have to can just consider pairwise agreements. As for implementations have a look at: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/cluster/supervised.py
I've decided to go with this...hopefully it's correct!
By 'real world independence' do you mean something like 'physically independent' or something else? Independence in the probability sense is a jargon term with a very specific meaning which applies to probability models -- mathematical constructs which needn't relate to anything in the real world at all.
I don't know the formula you are trying to implement, but `dnorm` cannot possibly be right: it is the normal density, which will be very small at most points.
@archana This should have been a comment on whichever answer you were asking about. Please delete it here and re-post it where you want it. Thanks.
+1, also for the purpose of learning new statistical tricks (& generally keeping up w/ what's going on) w/ `R`, you should consider signing up for the [R-help (& possibly other) mailing lists](http://www.r-project.org/mail.html) as well. Your in-box will be spammed w/ 100 posts a day, but they can easily be ignored & you can just take in a quick note about topics that interest you w/o undue hassle. This will go part way towards covering the needs that r-bloggers serves.
I don't know if this is an appropriate question for this forum, but I have included the R code below.
Great advice, especially about overlaying the dots on the line plots.
Funny, I just posted my answer to that question.
Very useful reference list. Cumming has brought many of his ideas together in http://www.amazon.com/Understanding-New-Statistics-Meta-Analysis-ebook/dp/B007M9D76G/ (Whatever you want to say about the expression "the new statistics", I probably agree.)
So, why preventing from giving it a name? Variables are just considered as manifest variables, and in some cases it makes sense to consider their weighted combination as reflecting a latent (unobserved) factor.
As with almost all mis-specification tests, choosing your analysis based on how the test comes out has problems; the mis-specification test and the post-test inference are not independent, which messes up the frequentist properties of the post-test inference, as it's typically implemented.
See http://stats.stackexchange.com/a/59249/159 for why the AICs are not comparable.
Note that you have succesfully put the exogenous variable into the model. You have problems with getting the forecasts from the fitted ARIMA model with the exogenous variables
To make this response complete and sustainable, would you perhaps share some more information with us? In particular, why 50% is the default choice for fraction of responses?
So many questions ! Some procedures avtually reverse the process and then go back to predict(compute) the residuals in reverse order. At the end of the day it is much ado about nothing ! Current procedures are relatively robust to starting values withlittle or no effect. However good software will allow you to specify the 3 stopping criteria ( # of guesses, error SOS convergence , parameter convergence )
@Michael, I did mean Bernoulli $\\theta$, but really maybe more accurately, I should say Bernoulli $\\theta_1$. Please note that $\\theta_1 = 1-\\theta_2$.
:Bruder I tried to take our "discussion" to a chat room but you don;t have enough "seniority" . If you want to contact me off line please do so and I will try and explain in simple/plain english terms what Tsay pointed out in mathematical terms.
@jason,You might consider constructing an equation which incorporates dummies to handle the "seasonal effect" and to create a set of tentative model residuals.These "residuals" may then be used to determine how many trends there are in the data or how many level shifts i.e.intercept changes. This might requires either significant analysis on your part or the programming the algorithms of R.Tsay (Journal of Forecasting Vol pp 1-20,1988 ) or investing in commercially available software. I have been involved in developing these fairly proprietary procedures for AUTOBOX.
You may need to explain what you mean by soft and hard EM, or provide a link.
Thanks for the help! I've found only one source where it discusses the issue of modeling a covariate in a mixed ANOVA. It argues that a covariate would *not* have any impact on the repeated-measures factor, and the author suggests the following: Do 2 separate analyses, one with the between-subjects factor plus the covariate, and the other with the repeated-measures factor but no covariate. Why such a kludge approach? Such suggestion made me think that there might be some more serious difficulty in modeling a covariate under GLM. Any books disucssing the issues?
GLMs can deal with right skew continuous distributions; there's the gamma and the inverse Gaussian - indeed the whole Tweedie family, indexed by $p$, the power in the variance function; these are continuous for $p>1$. There are other alternatives to GLMs of course, but that's where I'd start.
The answer will depend on your problem at hand: are both types of error equally bad, or are (e.g.) false negatives way worse than false positives. For instance if you want to test for a disease that kills, you want to strongly avoid false negatives (because those people will die when you don't pick up the presence of the disease with your test)
If you do not have a prior, you cannot use Bayesian inference. And hence MCMC methodology,
Definitely do NOT use AIC to select an optimal transformation of the dependent variable (which I think you were suggesting in the log-linear model). Likelihoods are not comparable between two models where the response has been transformed in different ways, so the AIC is not meaningful.
Sorry, I haven't mastered these derivations for the degrees of freedom either.  The section on DDFM in the [SAS documentation for PROC MIXED](http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_mixed_sect015.htm) is where I'd start; it outlines the Satterthwaite calculation and has references to several of the pertinent articles.
The distance between the true mean and the observed mean is what I mean by "this far away". I'm going to change it to "so far away"; I think that is a little more clear.