tttjjj/elt_bench huggingface dataset:
If this is a programing question you might be better off at Stack Overflow.
`Var3` is the variable you want to model and other variables are the covariates whose effect on `Var3` you want to estimate?
Good question, but I don't have (a reference for) a good answer. There's more than one level of variation in mixed models, so there's more than one component of variance to explain, plus it's debateable whether random effects can really be said to 'explain' variance. I think the whole concept of 'proportion of variance explained' is less useful in mixed models.
+1. Good one! I think your first plot is great on a conceptual level. My only comment would be that it requires significantly more explanation than a standard "naive" plot and if the audience isn't up to speed with the concepts of LME models and longitudinal data it might miss the point of the plot. I will definitely remember it for a solid "stats talk" though. (I have already seen the second plot in the "lme4 book" a couple of times. I wasn't too impressed then and I am not too impressed now either.)
The main effects and the pairwise interactions are missing (as a start)
100k doesn't sound like a lot for a modern computer to analyse every month.
http://www.businessinsider.com/the-real-reason-for-the-recession-2009-4
If [this](http://www.cs.utexas.edu/~bradknox/AAAIFSS-RLIHT12-papers/aaaifss12rliht_submission_2.pdf) is the paper, the first reference to $\\phi$ in your quote above references $x$ without a subscript; might $x_i$ reference a row or column vector within a matrix $x$? In that case $\\phi(x)$ might be the vector of $\\phi$ applied to each row or column.
@whuber I deleted the other question.
@whuber is correct. Think about what independence of probabilities requires, and then think about what the 'expecteds' would be in a chi-square, and what a 0 chi-square then implies about the relationship between the values you give above and the expecteds. It's not the *only* way to do it, and it's not necessarily the most *algebraically* direct or perhaps the most immediately obvious way to do it (at least until you come to understand why it works), but it's built in to almost any stats package worth the name, and so it's going to be a very *easy* way to do it.
Also http://www.math.uni-bremen.de/zetem/DFG-Schwerpunkt/jahrestreffen02/scripte/liebscher.pdf
Thanks for the article and R package links
Good question, although my immediate reaction, backed up by my limited experience of teaching this, is that the CLT isn't initially at all intuitive to most people. If anything, it's counter-intuitive!
[This answer](http://stats.stackexchange.com/a/25664/805) addressed this kind of issue, though you could also perhaps do a version of an exact test for your 2x3 chi-square.
Many thanks Scortchi for your help! Now I am not worried about differencing.
@ Daniel. Bluntly, in R, I have no idea. I'm a SAS man myself, outside of the occasional visualization problem. But nothing you've talked about (including R squared) is a singular solution to "How well does the model fit". Even the log likelihood is mainly useful to compare two models. Seeing how well a model "fits" really does involve multiple steps, including looking at your residuals for patterns of poor fits, particularly poor observations, etc.
What is LDA? Please do not start with acronyms.
Useful discussions on this topic can be found by [searching our site](http://stats.stackexchange.com/search?page=2&tab=relevance&q=%2bmargin%20%2berror%20%2bconfidence).
What are you trying to do? Forecast? Study the seasonal pattern? Look at what exogenous variable affect the series?
The $X_i \\in \\{0,1\\}$, so, irrespective of everything else, $\\bar X$ is certainly rational!
By perfect you mean 0 training error? It doesn't seem that hard to get better than trivial classifier for any particular training dataset. Might be hard with decision stumps though
I realized that after I saw my statement float around. I'll edit it to refer to the specific comment.
I think we need to hear more about why you can't just resample with replacement for the main statistic of interest.  Why is it possible to do this with subsets but not the original dataset?  Is it just a matter of scale?
+1, if you call [?mosaicplot](http://stat.ethz.ch/R-manual/R-patched/library/graphics/html/mosaicplot.html) instead of `plot`, you can set `shade=TRUE` which will depict the Pearson residuals / helps you identify which cells differ from their expected values.
You could also have r=2 wich would mean that your variables are stationary
You cannot estimate random effects for those school-level variables. In the link you provided, no school-level variables are used as random effects.
I agree with your interpretation of what the definition means. If I understand correctly, you seem to be using a multivariate normal random number generator for the normal part; you just need $N$ observations from a univariate normal - `normrnd` should be sufficient.
Notice that your density function is an average of density functions of gaussians with standard deviation $h$ and different centers. Does that clarify why you're using this formula?
If you have the mean and covariance matrix, sampling from a multivariate normal shouldn't require MCMC.
It sounds like you have recovered the pattern you expected in two components. I was expecting 3 since you have three groups, so look at that third eigenvalue. The "distance" matrix here is the covariance (or correlation) matrix. So the clusters are defined by groups of variables with high correlations not by Euclidean distance. Do the points (concentrations) on the biplot show any clustering?
I agree it is ordinal, but I disagree with your reasoning. Please see my answer
LASSO (Least Absolute Shrinkage and Selection Operator) is group of methods to solve under-determined linear system with additional constraints. You should be able to find lots info about it on google.
or maybe it would be a whole lot trickier, now that I think about it.  thanks for the helpful answer anyway.
I don't mind the down vote, but I maintain that this is a deep statistical point, not to be taken lightly.  ;-)
And please add googlable names of the software or at least links to their sites.
The claim is that if search engine robots see structurally different content returned at the same URL on different visits, they will penalize the page somehow.  Since I have insufficient expertise to evaluate this claim, I have treated it as given.
The residuals don't work that way.  It's one set of residuals from the model.
@Keith I described some possible ways to perform ordination with mixed data in [this response](http://stats.stackexchange.com/a/5777/930), as well as some clustering procedure that might be used afterwards [here](http://stats.stackexchange.com/a/3475/930). Does that help?
You said that $E(XE(Y|X))=E(E(XY|X))=E(XY)$. I believe this is wrong. E(Y|X) is a constant. Therefore, $E(XE(Y|X))$ is equal to $E(Y|X)E(X)$. Another point, $E(Y|X)=b0+b1*X$ comes from the simple linear regression model.
The Logistic and Exponential distributions are two distributions that have higher moments independent of their parameters.
Also, there might be worthwhile stuff in [this Google Scholar search](http://scholar.google.ca/scholar?hl=en&q=bayesian+estimation+of+structured+covariance+matrices). No guarantees though -- I haven't looked at any of it.
@NickStauner I didn't see the edited question with the data set when I originally answered this. The co-ordinates wouldn't signify much but the number publications. The circles can be clustered by topic or by diameter size. I don't know why percentages were used in the first place since the numbers are extremely small.
What you should do depends on what you want to find out.
By regularized covariance estimation, do you mean restricting the covariance to be diagonal (in effect, assuming feature independence)?
I think there's a minus sign missing in the exponent, because the inverse logit function (a.k.a. the logistic function) is $1/(1+e^{-x})$.
A proportional odds model with only a series of dummy variables as predictors, representing k groups, is equivalent to a Kruskal-Wallis rank ANOVA with k groups (k=2 -> Wilcoxon).  The numerator of the score statistic is the rank ANOVA statistic (Wilcoxon).
+1 For the very helpful link to the LIBSVM guide. That's a good start for the pre-processing. I want to think a bit more on your degrees of freedom comment and get back to you.
Thanks. Please see my addition to the question and see if you can clarify this a bit? I do follow what you're saying about the null of the test. It just seems to me at this stage (hopefully before some penny drops) that the two 'ways to look at the difference' are telling different stories.
Alas, I am not a trained statistician, and yes, I had to supply several assumptions based on the `Hint: for example. a Bernoulli random variable can be used to model a coin with probability of success` $p \\in ]0,1[$, that is in the problem statement. To me, the natural extension was to a geometric random variable. I agree with you about the merits of the question; indeed, even the notation is quite dreadful. $\\pi_0$ and $\\pi_1$ as the prior probabilities of two hypotheses is common usage, but just $\\pi$? with all the confusion likely to ensue when normal distributions are encountered?
I was playing around with AIC versus AICc on random matrices of varying sizes. AICc seemed to be working better. Those references look good, but I haven't had chance to digest yet.
Do you have some general plot of the data? People miss appointments for several reasons: weather, day of week, age and maybe even gender. If your original data can explain the target values well, just try some kind of regression.
I have merged your two accounts, Diego. You'll need at least [50 reputation points](http://stats.stackexchange.com/faq#reputation) to be able to leave comment everywhere on this site.
@cardinal Pardon my French. Let me rephrased my question. The Gaussian copula (which I am particularly interested in) is given by $s(x_1,...,x_d;R)=\\dfrac{1}{\\mbox{det}(R)^{1/2}}\\exp(-0.5 u^T(R^{-1}-I)u) \\prod_{j=1}^d  f_j(x_j)$. Where $u=(u_1,...,u_d)$ and $u_j=\\Phi^{-1}(F_j(x_j))$. This, for instance, cannot be bounded by the product $\\prod_{j=1}^n  f_j(x_j)$. So, I was looking for another upper bound that involves only the marginals. And, of course, I was trying to ask the question in a more general manner, relating it with the aforementioned bounds. Apologies for my vague words.
[ROC](http://en.wikipedia.org/wiki/Receiver_operating_characteristic) curve shows behaviour of a quantitative classifier on various cut-points. Accuracy varies for various cut-points. For what cut-points do you report your accuracies?
Thanks for the Windows warning.
Does AR1 even make sense if your time variable is a factor?!
In addition to @whuber's point: different types of sensors can have different "behaviour" in terms of error. Some measurement techniques may have constant error, for others the error may increase with increased weight. In my field (chemometrics) it is quite common to have both types of error, e.g. a behaviour that has constant variance for low weights, but turns into a proportional standard deviation for higher weights.
I'm wondering if anyone has solved this... I have a similar problem: https://sourceforge.net/p/mcmc-jags/discussion/610037/thread/947bff94/ ....
how do you know it is easy, if the OP didn't even write what kind of spectra he measures? Which implies that we don't have any idea what kind of variation to expect due to instrumental conditions.
Thank you @Frank Harrell,It is really helpful.
@posdef from my understanding yes. For each probeset on a chip (probes of the same sequence) there are probes scattered throughout. http://plmimagegallery.bmbolstad.com/ for some pseudo images of chips. In addition to the variability within a single chip, there is variability between chips. Because of the technical variability, algorithms are run on the raw intensity values to obtain a single "expression value" for the probeset. The matrix of these values are then fit to determine if the genes are differentially expressed under different conditions.
Thanks for the answer @Martin, I want to compare the performance of a Matlab optimization program with R, so I need a `qr` to produce exact same result. I agree that usually the signs do not matter, but I will use `qr` for sparse matrix to replace `qr` for the time being.
Not at all. That's covered by my last sentence, what you might do on the RHS. Your question just provoked me to set off on a ride of one of my hobby-horses, but nothing in GLM rules out transforming predictors if that makes sense.
First question in other words: What is the effect of iid assumption on the bootstrap? Is it a simplifying assumption that can be removed by following a more complex algorithm, or formula?
Although this question is stated quite differently, it appears to be identical to http://stats.stackexchange.com/q/2432 .
I don't think that you need a 2d lognormal. You have radial distance from a house and that's lognormal. So if you care just about the distance, and not the orientation you just need that. If you want you can model the angle using, for instance, a uniform distribution with range $[0, 2 \\pi]$.
What do you mean by "your data is nowhere near a distribution"?
I've learned that many times in statistics there are bounds which are "empirically proven" to provide "satisfying results". Choice of optimal parameters is done often according to empirical results and intuition. I ask for experience here.
I suggest retitling to something like "p-value and the base rate fallacy"
Thank you for your comment. Unfortunately the answer has to be in terms of $z$ since $f(z)$ itself will be multiplied by another variable and integrated again. Add to that the equation describes some physical phenomena which has a mean thus I can't set the mean to zero.
It has `Time*Diet`, which is just a simplification of `Time + Diet + Time:Diet`. Using `anova(m)` or `summary(m)` confirms that the interaction term is in the model.
A good question.  Probably a histogram.  Regarding your first question, the discrete variable has to not only have a lot of levels overall but to have a lot of levels in the vicinity of the quantile you are estimating.
@JaneWayne - yes, the code would be: 
predictmodelA <- predict(fitmodelA, newdata, type="prob", index=1:2)
My NewData was a new database with exactly the same predictors but for an expanded number of rows (i.e., for an expanded area of interest) and no training data was included.
The two tests are the two sides of the same coin. The two $p$-values add up to $1$. The $t$-distribution is symmetric, so the probability of getting a $t$-value of $5.5917$ or *greater* is `pt(5.5917, 19, lower=FALSE)` but this is the same as the probability of getting a $t$-value of $-5.5917$ or *smaller:* `pt(-5.5917, 19)`. In the second test, the $p$-value is the answer to the question "what is the probability of getting a $t$-value of $-5.5917$ or *greater* if the null hypothesis were true", which is: `pt(-5.5917, 19, lower=FALSE)`.
It's not clear from your answer, but "Why Tables are Really Much Better than Graphs" is a satire.
If the interaction is not significant it simply means that, if, in the population from which the sample was drawn the effect there was no difference in the effect for men and women (e.g) then there is a $> 0.05$ chance of getting an interaction term as large as the one you got or larger.

Depending on N, large effects can be nonsignificant and small effects can be significant.
@SrinivasaReddyChalla, you can check the documentation of any R function via the syntax `?somefunction`. For example, to check the documentation for `runif`: open up an R session and type `?runif`.
@ianalis: The median of sample medians works well, and is still close to the "true" median of 0; it is not the same as the overall median but there is no reason to expect that it would be.  The mean of sample means does give the overall mean (if the sample sizes are equal).  Combining measures of dispersion from different samples is more complicated, especially with median or quantile based measures: with variances the answer is to use the denominator $n-1$ to get unbiased estimates, as shown here.
+1 Nice example of how to improve odd question with great answer.
I would just delete the phrase "positive moderation" and leave the rest.
@MarcClaesen I find this hard to believe. This *is* Gallup, not some unknown organization. More tellingly, I regularly read their polls and find no problem with most (~85%) of them.  And when I do have a complaint, it's almost always about the framing of the question; this, however, seems to be a much more serious problem with data analysis, and not open to debate.
the difference between 70 observations and 70 responses is the residuals. the squared sum of the residuals are the general model error. the matlab code is just sum(resid.*resid). but note that sum squared error is just one of the ways to calculate general model error.
It is only significant at the 10% confidence level, but the confidence intervals are 5%.
You can get this article for free (although may be a slightly different version) here  http://www.isds.duke.edu/~berger/papers/interplay.pdf .  Okay to give a "free" link, because its on James Berger's Website.
Also re: the autocorrelation, don't worry I know is still there, but at least I am not just chucking "day" into the model is what I mean - this way the tests are run independently but the results I will ensure treated in series as the same particle. If you think this is inadequate/doesn't provide some control for this, then please let me know any alternatives you can think of.
As I said, they are only indicators, not guarantees. I actually don't see any way to actually diagnose pseudo-convergence. To do that, you'd have to know the true distribution already, but then why do MCMC?
@Barzov you should post a new question, and include your code and (if possible) your data.
@Adham: In your original question, to get the probability of the first two questions being unrevised and the next eight  revised would be ${10 \\choose 2}{25 \\choose 8}/{30 \\choose 10}$.  But in practice the unrevised questions can come in any position, so you have to multiply this by ${10 \\choose 2}$ to get the probability exactly eight questions can be answered.
1) Why is regression "hard"?
2) The ability to determine failure risk at time T depends on the type of survival analysis you  did. The most common type is Cox proportional hazards, where the goal is usually to estimate the hazard $ratio$ rather than the risk at a particular time.  What model did you use?
3. One big issue in survival analysis (maybe the biggest) is censoring. What did you do about that?
Note that there is a [crosspost](http://cs.stackexchange.com/q/11527/98) on [cs.SE] with duplicates both there and here (linked in the post on [cs.SE]).
kernlab also uses libsvm for the optimization, so there isn't a big difference in that sense (although it is much more flexible).
Keep taking out the "outliers."  Mathematics guarantees that by the time you have removed all but two cases, you will reach a correlation of $100\\%$ :-).  I hope it's clear I'm being facetious, but the concern about the lack of validity of this procedure is real: please see the recent discussion (about essentially the same issue) at http://stats.stackexchange.com/questions/86508.
The sigmoidal functions become step functions with appropriate prarameters, but are you meaning you are wanting something like a level shift in time series analysis?
Have you read the manual and worked through the tutorial?  http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/manual14.pdf
Could you elaborate further what you meant for input/desired output pairs?
Hi Martin - Thank you for responding!  I am not trying to calculate nor predict revenue.  I want to measure the ceteris paribus effect of each factor on revenue.  What happens if ice cream type is changed from vanilla to chocolate with everything else held constant (i.e. its a sunny day)?  That would simply be Revenue(chocolate, sunny) - Revenue(vanilla, sunny).  I'm trying to figure out the full decomposition with interaction effects for all factors and all states.  So f00 = (vanilla, sunny) and f11 = (strawberry, cloudy)  Does that make sense?
"*would it be $μ$ = 60*". No, your hypotheses do not come from your *sample*. What's the question you're trying to answer about the population mean?
How exactly do you define probabilities "$P$" in terms of integrals in the first place?  Giving some thought to that might reveal the answer to you.
This is something I also came up with recently - first find 'good' solution and then improve it further with GA. Not only applicable to NNs, but optimization in general...
What does Taleb have to say about IV?
Can you where does error⋅(1−γ)/γ" come from? Any proof?
great answer, nice work!
@teucer can you be more specific? Am I making an error in model specification?
This question is to help with any confusion over the site name proposal: http://meta.stats.stackexchange.com/questions/21/what-should-our-site-be-called-what-should-our-domain-name-be/53#53.
To calculate a mean, you first need a sum. For a sum to be meaningful, you need that  4+2 to be the same as 3+3; equivalently, you need 4-3 = 3-2 = 2-1. With ordinal data - even when its categories are labelled "1","2","3","4" - this is (quite explicitly) not necessarily the case.
You're *the* Brendan McKay - from ANU? Welcome to CV. (I can't say I've seen this particular result, sorry.)
$X_1 \\le 1$ so $\\ln(x_1) \\le 0$, which means it cannot be a density or distribution function.  And I still think $X_1$ should not appear in the marginal distribution of $X_2$. http://en.wikipedia.org/wiki/Marginal_distribution says the same thing in "The distribution of the marginal variables (the marginal distribution) is obtained by marginalizing over the distribution of the variables being discarded, and the discarded variables are said to have been marginalized out."
Is it rude on stats.se.com to accept his own answer? (After you looked for errors in it, of course ;) )
The explanation for the difference is provided at http://stats.stackexchange.com/questions/10947/formula-for-autocorrelation-in-r-vs-excel/10950#10950
For starters, what are the counts and # of successes of each category in each of those categorical variables?
Why don't you run formal co-integration tests instead of "saying" that the series are co-integrated?
I have the same problem as you. the explanation is too blur there
ah okay got ya. Well you can write out analytically what happens with the SE if you have an AR(p) and then just try it out with a few parameters. I haven't done this but intuitively it depends on wether a) you have a structure such as AR(p) and b) how the parameters of this structure are (also in relationship to each other). I'd say that usually you would overestimate the variance of the errors because of those AR terms. But that's just guessing.
Can you somehow comment on the exercises of these books? I want to do some exercises from a graduate level, mathematics-biased text book to enhance. Thank you.
Thanks Yevgeny. The variables are unfortunately both positive and negative. Is there still a way to apply a power transform or Box-Cox transform?
@gung How do you do multiple regression involving categorical covariates *without* converting converting the categorical predictors into numbers in some sense?
Yes, it's possible to transform predictors in a GLM (where possible, relying on some kind of theoretical justification). Consider if I handed you two predictors `a` and `d`, and you fitted `y=a+d+a*d` (in R notation, that's actually `y~a+d+a:d` or `y~a*d` for short). You'd be able to fit and understand that in terms of `a` and `d`,  even if I didn't tell you that `d` was just `log(b+1)`.
Ah, yes, thank you, now I remember these threads. So this particular question is arguably a duplicate :)
+1 This answer has a succinct clarity to it. I'd like to upvote it twice.