tttjjj/elt_bench huggingface dataset:
@Srikant:> there are no incorrect answer to phv's questions because there are no 'good answers' either; this area of research is still under development. I simply wanted to point out why it is still an open problem.
Indeed, its non-metric multidimensional scaling.
@whuber I thought the same so I upvoted your comment then started reading the Joanes & Gill paper (start of section 3, p185) and realised we're wrong : a normal distribution has zero skewness, so any estimator that is a multiple of $g_1$ is unbiased *under normality*. Unfortunately [there's no way to undo a comment upvote](http://meta.stats.stackexchange.com/questions/764/how-to-unvote-on-comments/765#765).
@whuber the girl-girl sequences turn up if you put all births in a line; after one couple finishs the next go in etc etc
I think that OP has three latent factors and three observables which are linear combinations of these factors plus noise. The OP knows that noise is present only in the one of the observables, i.e the covariance matrix of the noise is singular. This makes it a factor analysis problem, although I think the problem is underdetermined.
@jnthn It would be great if you could register your account here and on maths -- you'll gain full rights to edit your posts, post comments in your threads and claim reputation. You can do this [here](http://stats.stackexchange.com/users/login) and [here](http://math.stackexchange.com/users/login).
I don't see anything "piecewise" about the $\\tau$ at all: $\\tau_{\\lambda, \\mu}(x,y)=\\rho  (-\\lambda  \\mu  (1-x) (1-y)+\\lambda  (1-x) y+\\mu  x (1-y))+1.$  Besides, $x$ and $y$ are *data*, not parameters, so all that matters is how $\\tau$ depends on the parameters $\\rho, \\mu, \\lambda$, and it's a simple (multilinear) polynomial in them (and linear in $\\rho$).
Hi @TalGalili you're right, RMS is not for novices. I did say "more advanced" but perhaps I shouldn't even have mentioned it in this context.
The difficulty our knowledgeable and well-meaning readers face here is that you have posed not one question, but rather have stated--very nicely and thoughtfully, I wish to add--a question for which a good response requires a complete treatise on regression, covering at least nonlinear regression, goodness of fit testing, model selection, model comparison, and choice of loss functions. All these issues separately have been extensively discussed in other threads, but--of course--not all in one place.
Suppose you program a computer to calculate a confidence interval, but you don't look at the output.  Before you've seen the output, you know that there's a 95% chance that the interval contained the right parameter (just like before you've seen the result of a coin flip, you know there's a 50% chance of heads).  In what way does subsequently looking at the output change this probability, *given that you don't know the right parameter in the first place* (I agree that looking at the result of a coin flip changes the probability of heads from 50% to either 1 or 0)?
Everything said until the last sentence is correct and useful. The last remark is not correct, though, because it is based on an illusion: once you have included all the original levels in the model *you have used all those degrees of freedom (DF) in the analysis* and that needs to be respected in all subsequent analyses of the same data. Combining certain levels and then *pretending* this was originally intended creates an artificial apparent reduction in DF, which thereby can (in some cases) increase measures such as reduced $R^2$ and AIC. This, however, is just a way of deceiving yourself.
As there are no comments, neither answers I accept my own answer. If there are other answers coming, I am happy to accept another better answer.
Can you write out the distribution of the whole matrix? What do you notice about the $n$th column sum given the rest?
+1 A thoughtful and knowledgeable answer.  Please consider qualifying the initial "no," because it takes a bit of reading and thought to understand whether it means "no, the professor is not correct" or "no, 'correlation' is not a synonym for 'causation'," or the conjunction of the two.
No averaging is not appropriate in that case. Think about the questions you want to ask. The full model you could run is `lme(intention ~ ease + useful + technology * timing, random = ~ 1 | person, data=dataset, method="ML", subset = tech1Sub)` whereby timing is either 1 or 2 based on when the 'intention' measure was tested.
Thanks! That's a start. I do know this is hard.
"where here you want to give a tag to each sequence."
While Hidden Markov Models will do as you describe, there is also the related hidden CRF, which functions in much the same way, with all the other "benefits" of a CRF (depending on ones perspective and needs).
You might want to try STAN, from Andrew Gelman's lab - for many types of problems, it's far faster than BUGS or JAGS.
That's the same as saying "when does $\\text{Cov}(X,Y) = \\text{E}(XY)$?" ... which is when $\\text{E}(X)\\text{E}(Y) = 0$ ... so you should think about the circumstances under which they're the same. That is, when does $\\frac{\\partial}{\\partial\\theta_i}\\ln f\\left(X;\\theta_1,\\dots,\\theta_r\\right) = 0$?
Ugh, forget to mention that. Yes I am using R package.
Here is a **hint** (sorry, had a typo in the previous version): $$\\mathbb P(S^2 \\leq s) = \\mathbb P( (n-1)S^2/\\sigma^2 \\leq (n-1) s / \\sigma^2) = \\int_0^{(n-1)s/\\sigma^2} \\frac{u^{(n-3)/2}e^{-u/2}}{2^{(n-1)/2}\\Gamma\\big(\\frac{n-1}{2}\\big)}\\,\\mathrm{d}u \\>.$$ Now, use the fundamental theorem of calculus.
Thank you! So (if I understand correctly) it was true that any transformation of the $x$ variable, followed by a linear fit, can be afterwards analyzed by standard least-squares approach.

However, I don't understand why you suggest switching from my physically-based Gaussian approach, to your log-transformation approach?  Why have you chosen a logarithmic function? To represent a more general form of the exponential Gaussian function?
Please don't "move" questions on your own -- better flag it for mod to do the migration.  And ask a separate Q for your other problem (I've removed it, but the contents are still in [edit history](http://stats.stackexchange.com/posts/21043/revisions)).
This reminds me of Multivariate delta theorem, i.e. 
$ R_1,R_2,R_3$ has mean $\\mu,2\\mu,3\\mu$ and variance $\\sigma^2, 2\\sigma^2, 3\\sigma^2$ respectively, then $g(R_1,R_2,R_3) = ((1/R_1)+(1/R_2)+(1/R_3))^{-1}$ should have asymptotic variance as $\\nabla g(\\mu)\\Sigma\\nabla g(\\mu)'$, where $\\nabla g(\\mu) = (\\frac{36}{121},\\frac{9}{121},\\frac{4}{121})$ and $\\Sigma = \\[ \\left( \\begin{array}{ccc}.  
\\sigma^2 & 0 & 0 \\\\
0 & 2\\sigma^2 & 0 \\\\
0 & 0 & 3\\sigma^2 \\end{array} \\right)\\] $.
The final answer is same as @Douglas Zare and OP, that is 0.1028$\\sigma^2$.
(i) the KS test is for completely specified distributions. It doesn't sound like you have that. (ii) Why are you testing normality?
Ok, I see what you are saying, thanks
+1 That $\\sqrt{N}$ is the best possible bound follows immediately from Chebyshev's Inequality applied to the empirical CDF of the data.
Just stumbled across this question and saw that it's essentially a duplicate of an [earlier question](http://stats.stackexchange.com/q/1266/12446). Maybe some of those answers can help, if you're still interested in the problem.
Thank you. They are not the same metric.
Can the blocks overlap? I seem to recall responding to a question quite like this not too long ago on this same site...
I've edited my question to answer your queries. Nothing is known/constrained to be positive.
Oops, my mistake. Revised statement: Claiming that $\\beta_6=0$ is saying that the difference between the effect of "$X_1=B$" and the effect of "$X_1=A$" is unchanged when $X_2=F$ vs $X_2=E$.
@mpiktas, ultimately, I'd like to know which x_max is a reasonable target that I could try to reach on a regular basis (not just once), considering that reaching or going above y_max makes the experiment void (effectively implying x=x_min for that attempt).
(ctd)... When using boundedness to assert non-normality, it's the boundedness of the *possible* values of the variable, not the observed sample from it, that's the primary issue. A sample of size two only contains two values, but that doesn't mean those two values are the limit of the possible values for the variable being observed.
Thinking more about this, it seems there should be some useful way to exploit the 'zero' measurements during calibration to more accurately model the black reference measurements and get a better estimate? We know they are from the same population.
Given the way responses and comments are going, I added the 'data-viz' tag; hope you don't mind (feel free to remove if it's not appropriate here).
See [this](http://stats.stackexchange.com/questions/54547/should-pca-be-performed-before-i-do-classification), [this](http://stats.stackexchange.com/questions/50932/does-pca-followed-by-lda-make-sense) and [this](http://stats.stackexchange.com/questions/52773/what-can-cause-pca-to-worsen-results-of-a-classifier) as well.
Did you look at the answer by Jeromy Anglim in [here](http://stats.stackexchange.com/questions/60872/how-to-split-r-squared-between-predictor-variables-in-multiple-regression)?
What is the meaning of the numbers in the matrix? Are they counts?
I am afraid the answer is negative (as discussed in the thread you link and also in The Elements).
@cardinal, I am willing to assume stationarity and normality to begin with. The simplest example could be the AR(1) process and any references dealing with this case is of interest too. The thing is that I don't read time series literature on a regular basis, and I have not been able to find anything that is not either likelihood based (AIC and friends) or sampling based.
A really excellent treatment is http://data.princeton.edu/wws509/notes/c4.pdf
Yes I disagree with the practice.  If using modern software (e.g., R) centering doesn't help, and it makes interpretation more difficult.  The Q-R decomposition is insensitive to extreme collinearity, and collinearity does not affect predicted values nor does it hurt hypothesis testing, at least for interesting hypotheses.
we might want to turn this into a community wiki
I pretty much always see $\\theta$ used to represent a parameter in one of those distributions.  Occasionally I've seen $p$ used as the parameter, but never $\\pi$.  I've never seen $\\pi$ used outside the context of limiting probabilities.  I'm not sure but I think it fits into the whole "use English letters for statistics and Greek letters for parameters" paradigm.
This is a really clear, excellent answer!!
I don't recommend accepting this reply, because it is incomplete. I am hoping it might remind some better-informed readers of the theory and perhaps they can contribute better answers.
Another common problem that causes suspiciously goodlooking CV results is a "leak" between training and test data, e.g. pre-processing calculations that are done on the whole data set (centering, scaling, etc.)
+1 I really like this explanation and think it is very clear.
@whuber, I think we can formulate in this way if following the hint. Assume that the total number is fixed, which is $n=272$, then we take the hypothesis test: $H_0: p=0.5$ vs $H_1: p\\ne 0.5$, where $p$ is the probability that one observation is female. Then we calculate the test statistics $T=C_n^{110}  (0.5)^{110}(1-0.5)^{162}$. So that you can compare this with the quantiles of the binomial distribution with parameter n and 0.5.
If you believe that 30% is too high, then it might be that your rule for outlier detection is too stringent. Relax your rule (changing n in your formula).
`xcorr2` is applied to two matrices (if only one given, it is used in both contexts). How are you using it? That is, how do the x,y,z direction readings become matrices?
I've updated my question to clarify how $\\hat\\beta$ and $var(\\hat\\beta)$ are derived -- $\\hat\\beta$ is selected to minimize the sum of squared residuals plus a penalty term, and $var(\\hat\\beta)$ is based on a simple identity that relies on constant $\\lambda$ values. I don't understand link to MLE that you're drawing in this answer -- could you clarify?
It depends a lot on what you mean by "got statistically significant results consistent with hypothesis". If your hypothesis is that wind is produced by trees and your experiment shows that in 100% of observations when trees were moving their branches, there was wind, you find it statistically significant and voila your conclusion is proven. Which is obviously wrong. So, this might be one of those cases.
Thanks, gung. I can't upvote your answer yet, but I'll consider a CoxPH model as well. However, I'm concerned that some of the assumptions that such a model makes wouldn't apply here.
Random variables have expected values, but an event is a non-numeric quantity and cannot be said to have an expected value.  Probability, yes; expected value no.  So are you using **expected value** of an **event?** to mean its probability?
If it's duplicate I can guess that data mining and machine learning are the same thing!
`glht` uses the degrees of freedom given in the lme model. I'm not sure these degrees of freedom are appropriate... ?
Cheers! Thanks for your edit. Simple programming errors are the biggest enemy of scientific progress :)
Welcome to the site, @Breyer. I suspect this is a helpful contribution. Would you mind giving a brief summary of the argument there, so readers can decide if it's what they're looking for, or in case of future linkrot?
Hint: Check how standard errors in GLM are calculated and what parameters are used. See e.g., http://stats.stackexchange.com/questions/27773/how-does-glm-nb-work/
The test and the diagnostic plot aren't inconsistent. The distribution is similar to the theoretical one, as the QQ plot shows. The sample size is large enough that you are likely to pick up even small differences from the theoretical one.
A trivial comment: $\\binom{k}{2}$ boundaries between $k$ clusters get harder and harder to plot / understand / measure as $k$ increases. Thus e.g. k-clustering, viewed as a discriminative model, gets noisier and ad hoc. (Example: compare politics in 2-party systems vs. 5-party in Germany).
Can you give a full citation? Off-the-cuff I doubt iii holds, as the estimate of the dispersion parameter makes the negative binomial panel model not really "fixed effects" - see *Fixed–Effects Negative Binomial Regression Models* ([Allison & Waterman, 2002](http://dx.doi.org/10.1111/1467-9531.00117)), PDF [here](http://www.statisticalhorizons.com/wp-content/uploads/Allison.Waterman.pdf), for discussion.
What you've got is a little vague. Are you saying that each box with have a proportion of about 850 participants? If it's prevalence of something then the participant does or does not have that thing. Are the boxes independent, only measuring prevalence of X or are they redundant, presence and absence boxes? You might want to address this by just editing your question to be more specific about the design.
Welcome to the site, @Elyas Nezam. CV is not intended as a discussion forum like other websites. This appears to be an extended comment to Affine, rather than an answer to the OP. However, I think it could be turned into a proper answer. Would you mind doing so? (You can mention in the course of your answer that you disagree w/ previous answers.) Since you are new here, it might help to read our [about page](http://stats.stackexchange.com/about), which contains information for new users; there is also a guide to answering questions [here](http://meta.stats.stackexchange.com/questions/1390/).
Comparing curve just to get a nice p-value for publication is a can of worms with so many wormholes that you better repost this on stackexchange - this has nothing to do with R. And if you do, tell them what your horizontal curve axis means. Because of serial correlation, it is absolutely important to know if this is within subject or between subjects.
Comparing the performance of two classifiers on the same dataset is another topic to argue about. Especially, in the case of ROC and AUC there are a couple of methods to compare either ROC curves as a whole or the AUC estimates. Those are essentially statistical tests with the null hypothesis that the ROC/AUC does not differ. Cross-validation vs. bootstrap is another interesting topic, I recently saw a paper (http://dx.doi.org/10.1016/j.csda.2010.03.004) about that. I guess if you consider all the aspects at once, it can get pretty intimidating.
Because the current replies have (perhaps out of politeness) skirted around this issue, I would like to point out that this application is a terrible abuse of confidence limits.  There is no theoretical justification for using the LCL to rank means (and plenty of reasons why the LCL is actually worse than the mean itself for ranking purposes).  Thus this question is predicated on a badly flawed approach, which may be why it has attracted relatively little attention.
I should note that a general point common to many Robust/Resistant techniques (I've mostly worked with Huber estimators) is that they require you to bootstrap standard errors & $R^2$ values. Two discussions I found useful in my searches on this topic (and which contain code for bootstraping standard errors for an lqs model) can be found [here](http://www.stat.psu.edu/~dhunter/R/testboot.html) and [here](http://www.biostat.jhsph.edu/~iruczins/teaching/jf/ch13.pdf) (PDF). I also recommend exploring the sources cited in the MASS documentation.
Check [this](http://fourier.eng.hmc.edu/e161/lectures/gaussianprocess/node7.html) out.
can you give a reference to this paper?
Hi AlexK. **Yes**, please *do* post your solution as an answer! I was hoping this would be the outcome. I do not want to speak for @whuber too much (though I doubt he'll mind in this instance), but do *not* be concerned with "crediting" us. I, for one, am happy to see you've arrived at a positive result and have benefitted from the site. I hope you'll continue to frequent it and participate. Cheers.
Per your last point, I don't really need to prove if the distribution is Poisson, but for all my samples the mean and the variance were very close suggesting that it would be a good fit. Since the Poisson distribution is one-parameter, it makes for a nice plot versus time since it conveys both the mean and the variance (or at least close to it). Since I'm only plotting $\\lambda$ vs time, I needed some way of estimating how accurate this was and that a single $\\lambda$ was essentially the same as showing each distribution.
so $Pr(h)$ = version space / hypothesis space ?
I am accepting this answer as the better of the two - but it still does not answer my question really. What I was looking for is not how to choose my model variables, but once chosen (for argument's sake, let's say given by God) and I am in a situation as I described - then how do I guess what has gone wrong, assuming something has gone wrong, and start correcting my mistakes? In my specific case, I just dropped the offending variables without loss of discriminatory or predictive power.
This is not a linear programming problem. There are lots of integer constraints present. :)
Important in the first place is what is hierarhical regression. The term is not uniquely defined. Please give you definition, describe it in few statements.
The Sumatra project is one that may be of help too: http://neuralensemble.org/trac/sumatra/wiki.  You can use it's command line interface to run your code, be in R or something else. There is a Python API to it too. There is a nice blog post on R-bloggers discussing R-centric tools for reproducible research, and it mentions using Sumatra as well.         
                                                                        http://www.r-bloggers.com/managing-a-statistical-analysis-project-–-guidelines-and-best-practices/
+1, a lot of good information here
Actually, the affine transformation from the unit circle to the ellipse is given by the *square root* of the "shape" parameter.  E.g., the matrix {{4,0}, {0,4}} will only double the size of the circle.
Is this homework or a class assignment? If so, it should have the `self-study` tag.
There's no reason whatever to expect the mean residuals to be zero in quantile regression. The relevant *quantile* of residuals may be fairly close to 0 (e.g. if you fit a model for the median, the median residual will be close to zero)
+1 Love the diverging opposite scale idea.
@MichaelChernick - yes, good comment, that is what I was trying to convey.
@Aaron You're right, it uses pooled SD by default, but it can be disabled. We could also construct every contrast we want with simple algebra. I like your response btw (+1).
Of possible interest: [Comparing test-retest reliabilities](http://stats.stackexchange.com/q/3855/930); [Inter-rater reliability for ordinal or interval data](http://stats.stackexchange.com/q/3539/930).
Is it ever a good idea to report something when you have it? I realize there are different conventions for different sciences, but reporting $R^2$ is normal in economics. BTW, $R^2=.55$ is pretty good for a bivariate, non-time-series regression. Your biggest issue with a bivariate regression will be [omitted-variable bias](http://en.wikipedia.org/wiki/Omitted-variable_bias).
@dcl If you are interested on Bradley-Terry models, two useful R packages are [BradleyTerry](http://cran.r-project.org/web/packages/BradleyTerry/index.html) and [BradleyTerry2](http://cran.r-project.org/web/packages/BradleyTerry2/index.html) created by [David Firth](http://www.jstatsoft.org/v12/i01/paper).
Essentially "data dependent" just means the best settings will vary according to the particular structure of the data and there isn't generally any better way to set them than minimising the cross-validation error.  Kernel methods could really do with more theoretical analysis of how to go about learning the kernel, but unfortunately this is mathematically very difficult.
+1. I think that this is the key: when is *more* flexible *too* flexible? Perhaps "adjusting for" might be rephrased "penalizing"? So it's something like `The AIC is a measure of how well a model fits a dataset, penalizing models that are so flexible that they would also fit unrelated datasets just as well.`
Why the mode? With so little context it's hard to tell, but the standard deviation referred to sounds like perhaps it's across individuals at a given observed score.
(+1) I'll add that in the often subjective and iterative process that is EFA, you may be able to determine 1. which variables belong on the questionnaire at all,  2. which ones belong together as part of certain factor scores, 3. how many such factors should be considered to exist, and 4. (your last point) to what extent factor scores can serve as stand-ins for larger sets of scores from individual items.  Good luck!
+1 Note that the MSE comes with a built-in goodness of fit estimate, so it solves problems #1 and #2 simultaneously.
Thanks so much. I couldn't think about a case where a a datum is included twice, which is not the case in the original testset. Thanks!
I tried installing it.  After a quick try it seemed like rubbish for data entry; maybe if I gave it a longer try I'd be able to figure out how to use it effectively, but I suspect not.
Some sort of [cophenetic correlation](http://en.wikipedia.org/wiki/Cophenetic_correlation)?
I've always find rpy sloppy to work with. It requieres large lines of codes with some simple functions, for example.
Can you give us a short description of what he does with it?
Hi, I appreciate that you seem to think that the field of item response theory is underdeveloped, and that indeed it might be more complicated than assuming continuity of the observed variables.  However, I do think that considerably improvements have been achieved in software like R, Stata, Mplus and LatentGold that make such analysis considerable more accessible and I believe it is an option worth considering.
IMO total off-topic. Voted to close.