tttjjj/elt_bench huggingface dataset:
Glad to help. If you're in the market for a reference Gelman and Hill's book Data Analysis Using Regression and Multilevel/Hierarchical Models is a very accessible book to get started, with a lot of practical advice. And it doesn't cost your firstborn child either...
@David, yes, trivially so. Let $X_n = Y_n - Z_n$ where $Y_n$ and $Z_n$ are your "two identical series". For any $k$ and any $n$, $\\mathbb{P}( (X_1, X_2, \\ldots, X_k) \\in B ) = \\mathbb{P}( (X_{1+n}, X_{2+n}, \\ldots, X_{k+n}) \\in B )$, i.e., $(X_t)$ is stationary.
Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/15251/discussion-between-andy-and-eric-b).
@shijingSI, that's no more specific than your original question. "Fitting well" can mean it predicts better, has substantially higher log-likelihood, or...?
Use of the term "parameter" is loose here.
I am trying to understand at which distance the focal animals are more susceptible to flight (disturbance). The species of bird does fly at some rate when not disturbed so I am looking to find out where the distribution of flight observations deviates from the normal.
Your model specification is missing an error term.
This might help: http://www.ats.ucla.edu/stat/r/dae/mlogit.htm.
Thanks @henrik. This is good stuff, but it reminds me of one thing I forgot to bring up in my question, which is that genes and environment can interact. I'll amend my question.
Thank you for your answer! It appears I have quite a bit of reading to do if I really want to grasp what happens if I apply anything other than simple linear functions.
@kkp Here's a [rough draft](https://gist.github.com/1128764). Nice response (+1).
In addition to `lm` base R functions `loess` and `smooth.spline` both support weights, though `spline` doesn't appear to.
I believe there are a number of was to do this. Looking at the documentation for {relaimpo} or {relimp} packages would be a good start.
No, it should be 5. 1 for intercept + 1 for X1 +1 for levels of factor(X2) above the first, + 1 for levels of factor(X3) above the first + 1 for $\\sigma^2$. Supply the models with and without interactions to a single call to `AIC()` to check them both.
Consider looking into the smoothed bootstrap for estimating population quantiles as the conventional boostrap seems to have problems in that case - references can be found [in this pdf](http://arxiv.org/pdf/math/0504516). If you were just interested in the theoretical Median, the Hodges-Lehman estimator can be used - as provided by, e.g., R's `wilcox.test(..., conf.int=TRUE)` function.
@cyborg: Yes, I will know when a user uses the recommendation once the system is online - but it isn't so far.
I think my second half-question is the same as that one. Maybe someone could close this question as being a duplicate?
Naw, I'm just skipping some technicalities. A more technically complete analysis would start with a [Dirichlet process](http://en.wikipedia.org/wiki/Dirichlet_process) and then show that the result of marginalizing out the countably infinite set of irrelevant $z_i$ parameters is the Dirichlet distribution I give above.
+1 Welcome to our site! If you did happen to perform the word search you recommend, what results did you observe?
+1, you're right. I should have been clearer about that. I edited my answer to include this point.
Note that even in the case of simple MA or AR, the sample ACF and PACF may be quite unclear as far as suggesting a model.
I can find out the % of members who completed a task but only at present time, not over a period - so this is not concerned by my question. I want the data to show everyone (team and members) how we're doing (is it successful or not, if not where is the problem) and the relations between data. Basic question: which data am I supposed to put together in the same graph ?
Yes. That makes total sense, it's a RHS correlation problem.
$X'X \\rightarrow \\sigma_y^2 + \\Sigma$, where $\\sigma_y^2$ is scalar and $\\Sigma$ is the correlation matrix of $\\sigma_{ij}^2$.  And $X'Y \\rightarrow vec(\\sigma_y^2)$

So $\\hat\\beta = (X'X)^{-1}X'Y \\rightarrow (\\sigma_y^2 + \\Sigma)^{-1} vec(\\sigma_y^2)$ is the explicit solution for generalized $\\Sigma$.  Thanks!
This is a fine start.  Two suggestions to make it stronger: (1) you don't need the central limit theorem, which is just an unnecessary distraction for nitpickers :-).  A [law of large numbers](http://en.wikipedia.org/wiki/Law_of_large_numbers) will do fine.  (2) You can make your statements more quantitative: not only will the sample distribution of the mean be narrower with larger sample sizes, it will be about 70% ($\\sqrt{1/2}$) of the width when the sample size is doubled, *provided* you use the standard error to measure the width.  This, rather than bias, is really the crux of the matter.
@gmacfarlane I think corr() just computes the correlations, which is not the same. A bit of searching found the perturb package, which seems useful in this context, especially the colldiag() function.
Perhaps. If I understand correctly. For example, say in the future you've trained your classifier and you want to assign a label - what then? You don't know the label. So you'd have to try each combination of each feature vector and 1-vs-All classifier. Then you've got now `L * F` classifiers, where `L` is the number of classes and `F` is the number of possible feature vector formulations. Because they are the same in your case, this is `L^2`. Then to finish, you pick the class by doing what?
(1) You are missing a transpose in your definition of $H$. (2) The result is not generally true: the diagonal elements can be less than $1/n$ when $X'X$ is not of full rank (and the generalized inverse is used).
Welcome to the site. It is generally better to have your information (eg, your email address) on [your userpage](http://stats.stackexchange.com/users/48846/pranav-waila) rather than in an answer. Can you say a little more about the references you link to, what they contain, etc?
$x$ can easily be tens of thousands of dimensions, $b$ can be hundreds of millions. Working with video data.
see the faq on how this site works. Homework questions are answered here, but more effort must be put in by the person asking.
@Leo, I, for one, am not looking to "rack up" points. I am interested in learning from others and also good pedagogy. My understanding is that this forum's target audience is fairly broad and students are welcome. Probability and statistics can both seem daunting at first. That said, when looking at homework questions, it's always nice to see that the OPs have already spent some time on their own thinking about it.
By 'orthogonal', do you mean the two predictors should be uncorrelated with each other?
I don't understand how this can be right. If I take out one of the predictors ('animal'), then the intercept, coefficient of 'weight', and the error estimates all change. Also, I am not trying to find out what the _actual_ average lifespan for weight 1 in the data is, just what the model predicts it should be.
This sounds like a basic question that would ordinarily be addressed in the [documentation](http://cran.r-project.org/web/packages/sphet/vignettes/sphet.pdf).  What information is specifically lacking there or in the [reference manual](http://cran.r-project.org/web/packages/sphet/sphet.pdf)?
Basically, I want to draw a line which is as close as possible to all points (as a linear regression will do in case where Y depends on X) even if there is no correlation between Y and X values ... I want a method which I can apply whatever is the correlation of Y and X (even there is or there no correlation between them) ! How to do that ?
Rainer Walke has published a paper on ["Example for a Piecewise Constant Hazard Data Simulation in R"](http://www.demogr.mpg.de/papers/technicalreports/tr-2010-003.pdf). Might be usful.
Each realization of $Y$ is determined by the joint realization of $\\{Z, X\\}$. For gigantic $n$, we expect that $X_{(n)}$ will be close to $\\theta$ -and also, that $Z_{(n)}$ will be "very large" (since Z_{(n)} tends to infinity)... _but not necessarily on the same pair of joint realization_ of the two. So, since $Z_{(n)}$ will tend to dominate the determination of $Y_{(n)}$, it seems more realistic to state that  $Y_{(n)} \\approx Z_{(n)}+X$... If one defined $Y = \\max\\{X\\}+\\max \\{Z\\}$, then the relation $Y_{(n)} \\approx Z_{(n)}+\\theta$ could be expected... I presume $X$ and $Z$ are independent?
augh, you're right about MASS.  will edit.
The measure of roughness depends on the meaning of $A$ and $B$ and on what you are trying to achieve. If, say, $A$ is a "signal" and $B$ is "noise", then you need to characterize the noise as well as the signal. It's starting to sound like you may have a well-known problem in smoothing signals that has become (unnecessarily) more complicated through some preliminary analysis. Perhaps it would be best to state the problem you are actually trying to solve rather than this abstract formulation based on mathematical transformations of your data.
`plot(density(rexp(100), from=0))` ?
From my reading of both the slide you post and the book excerpt, they seem to be describing the exact same thing, and the slides are not in error.
(+1) I believe `contr.sum` is called [deviation coding](http://www.ats.ucla.edu/stat/r/library/contrast_coding.htm#DEVIATION).
Presumably the point is to discuss graphs that might be useful in similar problems and not be limited too narrowly by examples, especially hypothetical examples.
Be wary of any field that invents new names for old ideas or that claims that logistic regression is a classifer.  The proper terms for what you are seeking are polytomous logistic regression or multinomial logistic regression.
Thanks. Yep this seems promising. Only issue seems to be limited usage :(. Wonder why I did not find this on googling!                  What are various other approaches people employ to test keywords ? One way I think which people have suggested above is to use those keywords for some other application like document clustering and may be test that. Any other ways of may be testing keyword generation as stand alone ?
Look at the derivation of why $t$ statistics of regression coefficient estimates are $t$ distributed. This is entirely driven by data and observations separately. Pooling amounts to looking at the data and the observations, transforming the data (pooling factor levels) and re-estimating. The final design matrix is not independent of the data any more.
You are right, something seems strange. You might have to dig into the `survival:::survfit.coxph` function, or contact the package maintainer. A few minor quibbles: you should use piecewise constant interpolation of the hazard function, and summation is probably better than integration, but these changes won't fix your main problem.
I thought he was writting $P(A, R)$ for ''probability of A and R'', but in that case $P(A,R) + P(B,R) = P(R)$... so you must be right.
I only have the results of the estimation. This question does not contain the original data used for estimation of coefficients. So, I am assuming that the data contained information on individuals who either used a car or transit and the reported in vehicle time, out of vehicle time, etc.
@user16204 - see the [SPSS tag wiki](http://stats.stackexchange.com/tags/spss/info) for various suggestions.
I feel much better after reading your reasoning. You know sometimes it's the simple things that complicate our lives. Thank you.
Hi, lockedoff... tq for the answer...
Small note: If you have 4 ordinal levels, you should have only **3** thresholds.
Here's another primary reference for FDA: http://www.psych.mcgill.ca/misc/fda/
But the function returns a scalar value corresponding to the separation distance rather than a multi-dimensional boundary? Unless I am mistaken. Right, I plan to draw projections where two other dimensions are fixed (or resort to contour lines).
An example of how to implement a complicated link for a GLM in `R` appears in my answer at http://stats.stackexchange.com/a/64039.  It avoids implementing the likelihood explicitly, leaving as much work as possible to code that's already incorporated in `R`.
Canonical correlation was perfect! I have completed my analysis and got appropriate results as well. Thanks a lot!
That is right, I ran the code using different seeds. I think that the context gives us the answer: if the interest is on modelling the ratios, then the best model is the one on the ratios, which your code already does. Studying when the ratio of the models is similar to the model of the ratios might be difficult since it will depend on the underlying structure of the processes generating the data.
I wouldn't mind leaving this as non-CW, especially if Colin can rephrase it slightly to allow for the possibility of one best answer.  That said, I can't envision how to do that...
+1 This is a splendid example of what great answers can emerge when a question is opened up to a *diverse* community.
If you have uncertainty in your $x$'s, you generally shouldn't use ordinary linear regression because it's biased (though small uncertainties will result in small bias; maybe you don't care so much). Are the uncertainties in your y's always constant or do they vary? How are you fitting your line?
The problem is almost identical to my question except one important fact: I do not have access to the covariances. And as a consequence, I'm not able to compute the lower bound as suggested on quant SE.
"more training samples may not reduce the variance": more training samples can only help if the total variance is dominated by model instability. If the total variance is dominated by the variance caused by the finite number of test cases only increasing the number of test cases can help. In other words: the total sample size, which would then also allow more training cases in the usual k-fold cross validation scheme.
@Jonathan: brownian motion can exhibit a trend (such a trend is meaningless, but that is irrelevant), so the mean difference could be non-zero. I can't think of a sensible way to make such a comparison.
@Dimitriy V. Masterov: I want find the similarity of web click sequences?
Doing that with R code was really helpful. Some of us are more concrete-thinkers and it greatly helps to have an operational version of the generating function.
This really is a basic math question and not an R question. Perhaps someone with 'power' can redirect the category?
+1 The reproduction of Graham Martin's *Munchausen's Statistical Grid* at the end of Westfall & Young says it all in a very engaging way. You can read this in the Amazon "look inside" feature.  (It's almost as amusing to see Amazon offer a \\$7 trade-in price for this \\$150 book.)
I am wondering that which plots can be used as alternative to scatter-plot?
Please be more specific: "easier to use" is vague and may cause unnecessary controversy or arguing.
Does this hold if the variables are normalised?
The Wilcoxon-Mann-Whitney test does not test medians but rather the Hodges-Lehman parameter related to pseudomedians.  The central limit theorem does not help in getting accurate confidence interval coverage except in the limit, unless you know $\\sigma$ and don't have to estimate it.
Thanks for the clarification, you are right. I was referring to de-correlating. btw: at the end you write that whitening is only performed to the training data. as far as I know, you compute the matrix from the training data, but you perform it on both training & test data.
Because the CF is given by the MGF evaluated at a purely imaginary argument (and there are no technical issues of convergence, etc, in dealing with these rapidly-decreasing Gaussians), I take it that finding the MGF and CF are essentially equivalent problems.  The present question then reduces to the simplest case of the duplicate, which deals with the sum of products of Normal variables.
maybe a reference to the paper could help ?
I have no idea what you are trying to do. Please re-write your question, leaving off all the stuff about normalizing and telling us a) What data you have  b) What you are trying to achieve. See [how to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) from my blog for some help.
As a side comment, if you want to interpret the result be sure to demonstrate the that set of variables selected by lasso is stable.  This can be done using Monte Carlo simulation or by bootstrapping your own dataset.
Just one remark: For linear regression Deviance is equal to the RSS because the normality assumption of errors implies RSS is same as LR test statistic which further implies Deviance is normally distributed, _irrespective of asymptotics_. This is just an expansion of (I think) terse comment of chl.
I think away from the edges you might want to divide the difference by 2.
In addition to that, something else in particular to logistic models is that complete (essentially perfect prediction) or quasi-complete separation ("near"-perfect prediction, not sure how else to describe) can also result in a significant LRT but no significant variables (whose coefficients and SE will also be seriously inflated)
If you have the actual delay times, you are likely to get better information by modeling them, rather than reducing them to a binary variable.
By and large your data distributions don't matter: the time to start looking closely is after you have fit some model to the data, for then the distributions of the *residuals* become important for inference. How are your ANOVA residuals distributed?
More about the same problem: http://stackoverflow.com/questions/20397402/how-to-fit-known-values-to-a-known-curve-by-changing-x-axis
Exactly. The logrank test is equivalent to the score test of the Cox model with just a single, binary predictor.
@DQdIM thanks. That's actually what I use to display the final product, but I have R code to select items from an item database and write the individual items in latex format.
I have tried to solve formatting issues in the reviewed version of the answer. Thanks to amoeba for pointing this out.
Normally "MDL" stands for [*Method* detection limit.](http://www.caslab.com/Method_Detection_Limit_MDL_Meaning/) What exactly do you mean by a "minimum" detection limit?
Could you give us an idea as to the distribution of the measurements you have taken? Perhaps a histogram or kernel density plot might help to know if the data needs to be transformed at all.
Thank you very much for your answer! It seems like I have a huge lack in background.
It sounds as if the signals are measured on an interval or ratio level.  If that's the case there's no benefit to converting to a categorical level in order to conduct a Chi-Square test.  You can assess independence by creating a scatterplot and/or using Pearson correlation.
If lasso restricts use to keeping p<=n why is that a drawback rather than a virtue.  overfitting is a serious problem that comes about when p=n. The model with p=n is a saturated model and often that model overfits because it will fit the observed data perfectly but not necessarily predcit future cases well.
@Ben Setting a high value of `nAGQ` in `glmer` made the MLEs comparable. The default precision of `glmer` was not very good.
@aharach .. Sort of noob to statistics. What do you mean by factors.. (can you please give me a link or an example etc)
I am imagining something like I have a number of casino tokens ordered one after the other on top of a one dimensional table.  Then I shake the table along its dimension and see in what order the tokens landed.  The tunable degree of disorder I want to model corresponds to how much I shake the table.
If number of times sick is the outcome, is logistic regression really the correct approach?
See [here](http://stats.stackexchange.com/questions/68886/does-it-make-sense-to-compute-confidence-intervals-and-to-test-hypotheses-when-d), [here](http://stats.stackexchange.com/questions/2628/statistical-inference-when-the-sample-is-the-population/2630), [here](http://stats.stackexchange.com/questions/49483/what-does-population-actually-refer-to) & elsewhere - this question has been asked a lot in somewhat different ways.
Hi, I need a clarification. I do not understand what you meant by "I executed train.model with the same train.data and val.data every time, and I do it about 50 times. The error (Mean absolute error) percentile is about 5-10%, which is rather high". It is not possible that repeating an experiment with the same data you obtain different results.
Notation is odd: presumably you mean $x_1$, say. A proportion as predictor raises no special problems but (a) it should not be imagined outside its permitted range and (b) does the response really change linearly as the predictor varies over the observed range? The last question might lead to transforming the predictor. It is difficult to say more about knowing more about your data. On the other hand, if your question really is about the estimated **coefficient** $b_1$ being less than 1, that has no intrinsic meaning without knowing the units of measurement of response and predictor.
There's an additional assumption you're making here: the question doesn't ask how likely *an arbitrary candidate* is to get a second interview: it asks how likely *you* are to get a second interview. By pondering whether your friend group is a representative sample of the general population, you're ignoring the possibility that you are more similar to your friends than you are to members of the general population, in which case data about your friends might be more indicative of your own chances than data about the general population is.
"*I sincerely request you to guide me pls.*" -- Why would an explicit model of that curve (rather than say of the frequency and size of the bumps) be important?
I'm don't quite follow the "exact" terminology. Perhaps that is particular to Jaynes' work. Your $\\psi$ is the log-likelihood-ratio test statistic though and so $2 \\psi$ is asymptotically distributed as a $\\chi^2$ distribution by Wilks' theorem. Also, $\\chi^2 - 2 \\psi \\to 0$ in probability, which by Slutsky's theorem is enough to conclude that $\\chi^2$ has the same distribution as $2\\psi$. Finally, it turns out that $\\chi^2$ is the scorte test statistic in this problem as well, which provides another connection between the two test statistics.
The Wikipedia formula answers your question, Matt: you might not  have noticed that it's a partial formula where afterwards you need to divide by the sample size.
Are you saying that there are actually 3 possible outcomes, A, B, or neither? Otherwise why can't you tell that B occurred given that A did not?
+1.  I think of it slightly differently: the higher order derivatives are rolled into the error terms and considered as part of the "random" error.  (They probably are not exactly $0$.)
@whuber, I see. The covariances of the coefficient estimates should be in the output, but the marginal effects would either require direct calculation or finding the right R package (Stata does it with the official `margins` command, so there is no question of how to compute the standard error of the marginal effect unless you want to verify the formula.) The "multilevel" tag question still remains.
Random forests is a bagging algorithm: http://en.wikipedia.org/wiki/Bootstrap_aggregating. I suggest you read more than the shortest possible description of boosting to see the difference. In boosting, the resampling strategy is *not* random.
(1) You really want to update your question. More experienced people might come up with better solutions since I am really not an expert for this kind of questions. (2) You could standardize your measures ([z-score](http://en.wikipedia.org/wiki/Standard_score)) and then proceed as explained above. (3) Are your measures correlated? Do they measure the same theoretical construct. If yes, you could run sth like a factor analysis.
What attempts have you made so far and what have the results been?  For instance, have you figured out what *kind* of distribution $(X,Y,Z)$ must be? Have you computed its (multivariate) expectation? What about higher (multivariate) moments? Also, what does "f" stand for: a probability distribution function? Its pdf? Something else?