tttjjj/elt_bench huggingface dataset:
@altruist, I laid out the three cases for the sake of conceptual clarity; I recognize that the last is what you want. Note that whether or not you're using the software correctly to fit the model & whether or not your coefficients are 'significant' is unrelated to whether or not LR is the appropriate model for your situation.
Thank's @Glen. Unfortunately, I do not have one example regarding multiple linear regression.
Note that just fitting using OlS doesn't require normality - it's when you do inference on the parameters that you need the normal distribution asssumption
Thanks a lot. I have two questions regarding your answer. 1) "This is a common technique in Bayesian statistics." do you know if it has specific name? 2) " You will need to conduct an sensitivity analysis" what kind of analysis are you thinking about ? Thanks again.
Excel's RAND() is the worst pseudo RNG I have ever measured.  (I applied the DieHard tests to it about ten years ago.)  It's not even equidistributed!  Nevertheless, it's OK for this kind of investigation.
How would adding variables on a log scale be like adding a ratio of two variables (regardless of sign?). Adding the log of the ratio would be like adding the difference of the logged variables, but that doesn't seem to be what you are saying.
Also as well as independence issues, this is the sort of study that could be plagued by selection bias.  Anyone who is injured very badly, will stop playing, and therefore you have no more data.  Someone not badly injured plays again to gets injured again.  This could cause certain sites to be over-represented compared to others.
Thank you! Got it :)
"d statistic" is what?  Cohen's, Durbin-Watson, Tajima, Kolmogorov-Smirnov, Hoeffding's, Cook's, Somers, something else?
@Mark Is there a particular reason for using QMLE?
That the non-analytically minded folks (or non or poor abstracters) are not really helped by simplifying the maths has always concerned me (+1). Also liked the Strang linear algebra course, but isn't linear algebra the first maths to be eclipsed by brute force calculations? as for the data science is easy link, probably someone has raised already but in faux stock market competitions 1 or 2 of the thousand plus amatures usaully would beat the experts - so any one can get rick picking stocks!
Can I suggest looking at Kollers vids http://www.youtube.com/watch?v=ASsKAaHlhCU
@ars So, I'd like to suggest the following papers from Father & Son: http://j.mp/dvEDgb, http://j.mp/csD1Yf, http://j.mp/dkEHq5.
Very closely related: http://stats.stackexchange.com/questions/25068.
Since he wants l-bfgs not bfgs, i.e. a limited memory version, I presume he fears he won't be able to fit the Hessian in memory because the model has too many parameters or he hasn't much memory in the target hardware.
@John: Yes, it is roughly equivalent, and I'd do it also as a first step in replicating a repeated measures design.  I'm however not convinced that there's no perfect match and also, I don't believe that the ANOVA-type inference is in general the same as in the model you wrote.  (Though it may be in this data set as some of the estimated variances are zero.)
I'm not sure what you mean by this: `Furthermore, you can test the independence of the variables, but correlation can give you another useful information, the coefficient of determination`
Related: [Properties of logistic regressions](http://stats.stackexchange.com/q/16533/2970)
From $f(y)=-y^T\\Omega y+k\\le p(y)$, doesn't it follow that $f(0) = k \\leq p(0)$ and so if you want $p(0) = f(0)$, then it muse be that $k = p(0)$? Why drag in an unnecessary term $k$ to confuse the issue?
@cb80 You will need to [register your account](http://stats.stackexchange.com/faq#login) to post comments and access system-wide notification. In the meantime, I've merged your two accounts.
I want to balance my data set to have a roughly equal share of classes. If I use my data as it is now the models will simply predict all observations as the majority class.
(+1) E.g. [Example of computation in R and Bugs](http://www.stat.columbia.edu/~gelman/bugsR/software.pdf) (Online Appendix C of BDA).
@John no, they are not independent.
@AlfredM. Thank you for your input. To answer your queries I have added some info above.  Does this help?
$E(xy)=E(x)E(y)$ is zero correlation, not independence.
@Glen_b I don't have the I don't have the data-points on my computer, they are generated by a program on another computer. The data-points of the two datasets have only two attributes (x, y). Can you please tell me what are the statistical test or measure to evaluation the degree of correlation or dependence between two sets of data-points ?
@Proc: Those would make a wonderful, rather comprehensive answer. Y. Linnik was prolific in his studying of this problem and proved some of the significant negative results about this problem. A short overview of some of it can be found in Y. V. Linnik (1966), [Latest investigations on Behrens-Fisher problem](http://www.jstor.org/stable/25049394)
*Sankyha*, Series A, Vol. 28, No. 1, pp. 15-24.
In theory, it's problematic since you're dividing $\\beta_i$ by $\\sigma_i=\\sigma$ to get to the standard normal. However, according to Austin, biprobit is remarkably robust to variability in the treatment effect (random coefficients) in simulation.
I think that I can find the trial central line for the X-bar chart by taking the mean of the means of the subgroups.Sum of the subgroups is 52.0, and there are 25 subgroups. The mean of the means would be 52.0/25. The upper and lower control limits for the X-bar chart that I can use are the values provided from the R chart. The UCL for the Xbar = X-double bar + A2*Rbar and the LCL is X-double bar - A2*Rbar. Given: Rbar=0.11. Need: value of A2 for a subgroup size = 5 A2 value table- 0.577
I asked the question for the general case. Most of the stories start with Gaussian error assumption. But, personally, my own interest is matrix factorizations and linear model solutions (so say regression).
I'll run with the idea that it's Monte Carlo -- thanks for the response.
Actually I also was wondering about truncation.. now things are more clear. Thanks again, JC!
Basically we are clustering all the movies this way that have at least one of the three genres ['Action', 'Animation', 'Adventure'] and we are clustering them using the distance measure I described.
[What is the expectation maximization algorithm?](http://www.nature.com/nbt/journal/v26/n8/full/nbt1406.html), *Nature Biotechnology* **26**:897–899 (2008) has a nice picture that illustrates how the algorithm works.
(+1) yes, graph clustering is the way to go. I recommend this survey ([Graph clustering by Satu Elisa Schaeffer](http://elisa.dyndns-web.com/~elisa/publications/papers/cosrev07.pdf)), which also deals with the evaluation of clusters (see 4.2).
I think this should stay as on topic here. There are statistical issues with estimating auto-regressive components in generalized linear models - i.e. with an exponential link function if the AR term is positive it implies a non-stationary time series. (Although I am not that familiar with `gamm` - so I am not sure what `form=~Day` means exactly in this context.) So I think it should stay here, with a discussion of possible reasonable model specifications.
@drnexus: I don't think that is sufficient; you have to be sure that the same normalising constant is being applied in the log likelihood calculation.
@Mr Alph: I have added a simple example.
Well said, especially about what question a CI actually answers.  In the Jaynes' article however, he does mention that CI's (and most frequentist procedures) are designed to work well "In the long-run" (e.g. how often do you see $n \\rightarrow \\infty$ or "for large n the distribution is approximately..." assumptions in frequentist methods?), but there are many such procedures that can do this.  I think this is where frequentist techniques (consistency,bias,convergence,etc.etc.) can be used to assess various Bayesian procedures which are difficult to decide between.
I am surprised at seeing three means and three standard deviations for three colours. There should be two ratios, shouldn't it?
Thank you for this useful post. Hi what do you mean by highly plositively loaded to PC 2 hence x axis?
Highly negatively loaded to PC-2 hence Y axis? Can you please elaborate on this?
This looks a lot like a homework problem, and if it is so, please add the `homework` or `self-study` tag as appropriate. With regard to parts (i) and (ii), first draw a diagram of the $x$-$y$ plane, indicate on it the region where $f_{X,Y}(x,y)$ is nonzero as well as the regions where $X/Y \\leq t$ etc.  You will have different diagrams depending on the numerical value of $t$.  Integrate the joint pdf over the regions to get the answers you need.
@Henrik: Sorry I meant m2 instead of m3 in my previous comment. m3 has a simple random intercept while m2 models the pair-wise correlation among those levels of each within-subject factor. With a continuous variable, that would be a model with a random slope. If you feel that my response answers your OP, you can accept it by clicking the check mark.
By and large, you're better off posting these sorts of Probability questions over at http://math.stackexchange.com/
(+1) In the section beginning "Consider the odd moments about $a$...", I believe the third line should read $=(1^k-(-1)^k)(x-a)^k$.
+1 This question uncovers an interesting ambiguity in the language of this (well-received, popular) textbook: "depend on $\\theta$" could mean at least *three* distinct things! (1) $\\theta$ does not explicitly appear in the formula. (2) Although $\\theta$ might appear in the formula, the formula is invariant under changes to $\\theta$. (3) $\\theta$ is viewed as a (perhaps constant) *random variable* and "depend" could be intended in the sense of dependence of random variables.  Unfortunately, the attempted clarification ("the distribution ... does not involve $\\theta$") is too vague to help much.
Wow, thanks Andrew Moore's tutorial on cross-validation is world class.
So, to put it a little more descriptively, would it be correct to say you have $n$ points in the plane drawn iid from a bivariate Normal distribution and you wish to find the distribution of their diameter (which is the greatest distance among any pair of the points)?
This question is also crossposted on MO: http://mathoverflow.net/questions/77833/a-random-censored-regression-problem
@Juanan: Not sure if you're still here, but whuber's question is not *which*, but *why*. Did the evaluators know who did each exercise, so they might make choices based on their relationship with the colleague (or lack of relationship)? Or did the evaluators not know who did each exercise, so perhaps gravitated towards exercises with which they were more familiar?
I'm struggling to articulate this, but I'd argue that we live in a time of increased recognition of others' experiences and of the pervasiveness of power and privilege.  To me, "political correctness" is *only* the parts that get out of hand - it implies prescribing behaviors and language without trying to engender any understanding.  This question is a good example of that - the asker wants to know what the prescribed behaviors are to avoid offending people.  Political correctness is the cargo cult of human understanding: just perform the rites and you won't make anyone mad.
Averaging data is bad because it inflates correlations. The real question is why you think you need a single imputed data set. Everything that you can do with a single data set, you can do on a multiply-imputed data set.
I think in order to get useful answers you will need to greatly expand on and clarify your 2nd sentence.
@Drew75 The plot (and all the power calculations) were done in R. The little densities in the bottom margin were generated separately and pasted into the image (basically out of laziness, you can actually do it all in R), as well as minor reformatting (like the note about test bias and the marginal marking of the significance and a couple of other tweaks were added by editing the image).
Thanks! I assume the variance of $X_n$ is finite.
"Transforming to exponential is not your only option": how is it an option at all for MLE??
"In hypothesis [testing,] we always assume the opposite that we are trying to prove is correct and seek to show there is a lack of evidence towards such a supposition" Are you absolutely sure about this?
FYI, this site supports Markdown and $\\LaTeX$ [markup languages](http://stats.stackexchange.com/editing-help).
(1) What is $\\theta$, Maxime?  (2) Just how correlated are the estimates of $\\mu$ and $\\sigma$?  Unless the correlation is low, your proposed method--which seems to ignore the correlation altogether--may give misleading results.
Any thoughts? Even if someone could point me in the direction of the name of the concept here, I'd much appreciate it.
@integral, try using code markup so that we can read your code.
I include more than these two variables in my regression, but for theoretical reasons I don't wanna drop these two. I found papers where researchers either used the 1990 data for the whole five-year period from 1990 to 1995 or they "applied linear interpolation to fill in data for missing years."
+1 This is a probit model with parameters $(u,x)$ and *dependent* observations determined by a multivariate normal distribution (whose covariance matrix depends on the two nuisance parameters $\\sigma$ and $\\tau$ but is otherwise completely determined). The integrals involved in the likelihood look formidable, suggesting a Bayesian MCMC approach would be the most practical line of attack.
This worked great, thanks! Merging the covariates into the sts data worked correctly.
Thanks for your reply. It was indeed suggested by @gmacfarlane that I should break out the mixed question into a new post. `DEP_AVG` is actually the revenue of a countries' financial system relative to the countries' GDP. `CON_AVG` is the concentration of the banking system (marketshare of 10 largest banks relative to all banks), so if I understand correctly, they are in fact factors as well. I have to figure out how to get (the already existing) variable `COUNTRY` under random->subjects as this part of the dialogue box is greyed out default, but I'll manage. I'll post an update tomorrow!
I have some questions (just to be sure). 1. You have 5000 records, 13 features (12 predictors + 1 response) and the response variable is binary with values "flower" (15 times) and "not flower"(5000-15 times) => is this correct ?. 2. I do not understand experiments number 2. Can you try to explain it again ? 3. Do you mean that 1 and 2 did not differ for C4.5 and NB *respectively* or do you mean that the results of NB and C4.5 did not differ for 1 and 2 *respectively* ?
(+1). I want to add: Artificial data is generally used to emphasize (or validate) the properties of a classifier, not to prove that it'll work in the real world.
Note that a random walk does *not* have a constant mean; its first difference does.
It looks to me that, somewhere, there is code that expects to find a `contr.Treatment` function, which most likely should be the `contr.treatment` function. Not sure exactly where this is occurring or why without being able to parse the rest of the code. You might be able to fix it simply by creating that function from the correctly spelled `contr.treatment`
Search on this site and elsewwhere for "missing data". There are many possibilities. After browsing around, you can give us a better sense of the exact problem.
This seems like a natural candidate for hierarchal model, but without someone on staff who knows Bayesian methods and programming, I'm not sure how helpful that will be. If you think it would be helpful, I can write up a more detailed answer later this evening.
@whuber I'd agree that they should be made to agree, or at the least noted and explained with care (in many common cases making the two agree isn't especially difficult). On the other hand I'd regard the policy of the journal as the much larger problem there. It would almost be worth tolerating the inconsistency in order that the silliness (and worse than silliness) of such a policy be made more obvious.
CONT'D... 4) Taking into account 3) in the previous comment, the point where the two lines cross is a point where $\\hat y_{1i} = \\hat y_{2i}=\\hat y_0$ and $x_{1i} = x_{2i}=x_0$.
Thanks -- I once knew that trick.
+1 good answer to the general question. I may yet give R another try for this problem.  I already find tapply very useful for creating summary statistics over groups in the data.
@MattAlbrecht I have updated the question to reflect the duplicate ids.
_"understanding how statistics affect decisions is more important than getting all the theory right..."_  Ain't it the truth? I come from an architecture background, and I can tell you, sometimes theory just gets in the way...
If you want to learn, why not code it yourself?
Please change the title to something more informative and if this is homework, add the `homework` tag.
You are supposed to provide helpful hints for the self-study tags not the full answer. It is better not to solve their assignments :)
I actually tried to approximate the probability for bigger $n,t$ by just taking the $x^2$ part of the series into account, but that only works for small probabilities, for the probabilities that you're interested in, the approximation is way off.
@gung for a proportions test to make sense any time, you have to assume homogeneity; if that's not reasonable, then even if you can justify the comparison (what's the population about which your inference is being made?), you have bigger issues - like computing a bound on your standard error.
@Rider_X Several reasons: (1) you actually have a model for the mean response, and no bias issues with backtransformation of a fit. (2) generally greater similarity of output to what one will get with negative binomial (the NB model is what I would do, but the OP seems to hesitate to do that, yet is willing to contemplate the Poisson, so I thought they might entertain something close to but better than the Poisson) (3) relative convenience.
For *unweighted* correlations, the [Wikipedia article on Kendall's tau](http://en.wikipedia.org/wiki/Kendall_tau_rank_correlation_coefficient#Algorithms) answers this question. The `cor.fk` function in package `pcaPP` for `R` implements Simcha's algorithm and takes a fraction of a second with a $100\\times 1000$ matrix. However, the link in (2) suggests a weighted version is sought.  Is that the case?
"However, the eigenvectors do not correspond to these sorted eigenvalues" yes it does. "One approach might be to sort by the norm of the eigenvectors" hard, given that eigenvectors are *normed*
Thanks, The publication from Jonn Axsen *, Kenneth S. Kurani is interesting. In my question was more about getting data and I have to try zanran to see what it is.
@Hibernating: That's not really what I'm interested in, but thanks for the link - I've also read some of Giles' other posts on the probit/LPM issue. He focuses primarily on marginal effects, while I'd prefer to focus on tail behaviour. I'm mostly looking for a metric to compare the predictions. 
(Why the LPM? I see it sometimes in empirical work, I'd like to get a grip of how wrong it really is).
Ladislov, I resist providing a code snippet as best practice would be to simply try out the examples in the package documentation. It would help you immensely as you start to understand the limitations and strengths of the algorithm. I'd suggest simulating a dataset to begin playing around with it, and then once you know how it works, start mining the data you really need.
If it's a simulation, you could just repeat with increasing sample sizes to see if the numbers converge.
(+1) Nice overview.
In that case you re-work your model as it is no longer consistent with reality.
Sounds a lot like an awful way to start teaching machine learning--confusing the reader with an unrelated problem.
"This" meaning method 1 or 2?
It is the case (no white noise/iid). Details [here](http://en.wikipedia.org/wiki/Ljung%E2%80%93Box_test). How did you use `ACF` to decide that it is normal?
Before trying to understand Gibbs sampling, you should learn the basics of [probability theory](http://en.wikipedia.org/wiki/Probability_theory) and read up on [conditional probability](http://en.wikipedia.org/wiki/Conditional_probability) on your own.
I think I am missing something. If you *randomly* selected group A *from group B*, then the groups should not be different, right? So I assume that you in fact did not select at random, but based on some criterion. How large are your groups A and B? Apart from that, a t test is not appropriate for yes/no variables (take an action or not), a contingency table and a chi squared test would be better.
This is covered in just about every intermediate treatment of statistics. I like Rice, `Mathematical statistics and data analysis`, a lot: the reviews on Amazon show that this book divides readers sharply and so do most others. But if you can't find one book you like with a bit of effort, the subject is not for you.
If you are not comfortable with the assumption that all null hypotheses are true, and assumption underlying the *family-wise error rate* correction methods Glen_B referenced, you can also come at multiple comparison's corrections by controlling the *false discovery rate*—which assumes that some alternative hypotheses may be true—using a several methods: https://en.wikipedia.org./wiki/False_discovery_rate#Controlling_procedures
Thanks for the clarification and for encouraging me to read up, think a bit, and revise my question. It led me to the Merkle reference. Additional feedback appreciated.
whubers comments to the original question already stated what it would look like if the variances were the same and the variables were uncorrelated. Perhaps giving an example of where this is not the case would be more enlightening.
I've inserted a $\\ln(X)$ before "is fixed", thanks for the hint.
@Alexis — great point. The tasks are all uniform difficulty, so I'm thinking that the ability for Mechanical Turkers to review beforehand shouldn't be an issue and *hopefully* the analysis is still telling me about the effect of the price drop.
Oh ok, so in a normal univariate model, there is no type of covariance structure "within" the DVs.  Therefore hypothesis tests concerned with that don't exist.  Thank you!  I'll see if I can pick up that book.
I don't get what you mean by huge data. Are you saying all the data are huge or are you saying that the data spread over a wide range with some huge values?  I suspect the latter but your statement is confusing.  If you divide the data by a scale factor d and have a non zero mean m and standard deviation s, the mean changes to m/d and the standard deviation to s/d.  That seems to me to be the only issue.  I don't see where scale invariance come into play.  Whether you change the parametric form of the distribution by scaling doesn't seem important to me.
The elements of $T$ can't have a continuous distribution (like the normal), since it only takes integer values. To be honest it looks to me like perhaps a two-way main-effects loglinear model (simply conditioning on $T$) should do what you're after (I presume, give estimates of $X$ and $Y$)
I don't follow: by definition the values of a CDF are probabilities. Of course you can multiply them. Your question, stripped of distracting material, says "I know the probabilities of two independent events. How do I find the probability of their intersection"? (Isn't that what you mean by "$\\&$"?)  The answer is given by the [definition of independence](http://en.wikipedia.org/wiki/Independence_(probability_theory)#Two_events). Maybe I am misunderstanding the question? Please feel free to edit the post to clarify it.
On the entire series
this is really great, and should go a long way towards the CDF. Above and Beyond badge material.