tttjjj/elt_bench huggingface dataset:
There is also F. A. Graybill, *[Matrices with Applications in Statistics](http://www.amazon.com/Matrices-Applications-Statistics-Duxbury-Classic/dp/0534401317)*, 2nd. ed., Duxbury, 2001. It has a wealth of good information and serves as a companion volume to Graybill's [linear models text](http://www.amazon.com/Theory-Application-Linear-Duxbury-Classic/dp/0534380190). It is, perhaps, slightly dated in that other texts place more emphasis on the power of the SVD.
What is the citation for your quote? Was it [this](http://books.google.com/books?id=SD_QX7x366MC&pg=PA6&lpg=PA6&dq=the+idea+was+presaged+long+ago+by+Francis+Bacon&source=bl&ots=43qzI3BOPo&sig=BKl9CEM7zLOUNv5s03Fynctss5Q&hl=en&ei=JQXmToChIpSutweKpumEBQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CB4Q6AEwAA#v=onepage&q=the%20idea%20was%20presaged%20long%20ago%20by%20Francis%20Bacon&f=false) book?
In principle I agree with you. But assume that updating the estimate in the IT systems is costly and difficult. So intuitively, if the new estimate is not "significantly" different from the previous estimate, then I believe one can argue that there is no need for an update. The question is how I can decide if the change is significant?
This is great. The original question only gives C as a positive semidefinite (PSD) matrix, but also mentions covariance matrix as an example of it. Since, inverse of an PSD matrix is also an PSD matrix, it seems to be very close to Mahalanobis. And I guess the min of that sum could be interpreted as the closest possible distance of test point to a set of sample points, right? Thank you very much.
+1 for the large step back
MADELON has train.labels only, no test.labels not valid.labels. (In fact all the NIPS 2003 datasets on uciml, arcene dexter dorothea gisette, have only train.labels. Can anyone find the test.labels ?)
If you've ever used Box-Cox transformations, you should be able to remember it.
Use [Gower measure](http://stats.stackexchange.com/a/15313/3277). About measures nominal data, read [this](http://stats.stackexchange.com/a/55802/3277). Hor hierarchical and some other methods of clustering, it makes no difference whether you cluster individuals or attributes.
I meant the co-variance matrix or the guassian. What I am interested in is I have two observation that are Gaussian and I know their covariance. Now I want to combine the two (say I have two sensors that observe the location of a robot) what is the proper way of combining the two variables?
@user24608 If you want your posts to be signed, you need to register with the name you want to sign them with (presumably, "Alexios"). Then your name appears at the bottom right of your message (instead of a useless number like 'user24608').
It is about integration with the data source.  Philosophically speaking - there is value in "vertical integration" between the analysis tool and the data source.  Things like precision, sampling rates, and tool ephemeris are implicit in that connection and can be informative in an analysis.  Relevant keywords include: "collecting data" and "data-driven computing".
I don't see how ROC analysis fits in here. AFAIK ROC curves are composed of pairs of values stemming from "correct"/"incorrect" classification of observational units. I have got the impression that you have two values per observational unit which therefore are paired. If you want to test if they agree you could do a simple paired t-test or a related procedure which takes the pairing into account.
I don't think I should say more because at present: (i) my very first question in comments remains unanswered; and (ii) you've made some statements that don't seem consistent with each other (and which I don't presently have enough information to choose between). So I don't know for certain what I am supposed to be calculating nor the circumstances under which I am doing it. If you clarify your question sufficiently, I might be able to say more.
Can you please define *noisy*? I don't see much difference between the three graphs. I would say that the frequency of the big events is maybe a tad lower in the 3rd, but I would not talk about noise.
Thanks! I will try dropout
Kriging might be slightly more narrow than Gaussian processes, e.g. limited to Gaussian process *regression*, though the method generalizes to *classification* as well.
(+1) This abominable notation is getting in the way of your understanding and impedes the author's ability to detect and correct errors.  (For similar formulas--correctly and more simply rendered--see the answers at http://stats.stackexchange.com/questions/81754, for instance.)  I notice also that the authors attempt to index the vectors from $0$ through $n_x-1$ in the denominator but up to $n_x$ in the numerator.  Providing a text description of what this abortive formula is attempting to convey would help people decipher its intended meaning.
I have edited your question for readability, grammar, and mathematical notation: I also explicitly added an intercept to your model, since you reference it in the text, but neglected to include it in the model). Also: you mention creating the Bernoulli variable `varia`, but neither your abstract nor code models incorporate it, so it is unclear precisely what you are asking.
I don't think that this question really needs to be a CW. It is borderline, but it's not too bad.
I often prefer ggplot2. In fact I made a quick try at with ggplot2 when lattice was getting too complex, and found that I couldn't quite get the legend to properly reflect the point/line styles if I wanted to choose shapes and line styles explicitly. I probably did it the wrong way, and it managed to work in the graph, but fooled ggplot's legend creation. Could you see if it works for you and post the code?
@Peter True, but there is a distinction between "correct" (the word used in the question) and "useful" (which is what I think is really being asked). When you use a less powerful procedure (*i.e.*, one that "loses information") and it still indicates a significant difference (as suggested, but not stated, in the question), then you have what you need to draw a conclusion. Perhaps of more import at that point is how one would measure the size of the effect: using irregularly spaced bins--especially ones adapted to the data, as here--can create a real challenge in that regard.
A confidence interval based on a paired t-test should be symmetric. Can you show an example of an asymmetric interval for that case? When you say "the software", do you mean some software in particular?
The fact that you were able to get the exact number is nice, but those numbers only come in certain possible increments (eg, no .5's & no -3's). You would still be better served by using a count model.
How large a data set do you have?  Is your metric calculated on the same data used to fit the model, on a test sample, or using cross-validation?  If the relationship really is close to linear, linear regression is (quite likely) to outperform a random forest...  As for techniques that are better than linear regression, depending on the model and objective function, certainly there are better techniques, e.g., logistic regression for binary outcomes.
Can you help me understand why I'm getting NAs when running this?
That line is _not_ a single equation! Note the arrow in the middle. The first part (on the left side of the arrow) is one equation which _implies_ the second equation (on the right side of the arrow). (By adding $[\\mathbb{E}(X)]^2$ to both sides.)
Great advice. Sounds like it is exactly what I need.
Multiply the observations by $-1$.
@Bitwise. The paper says the function $trace(K \\pi^{T} L \\pi)$ is convex alone by itself
Can its generalized linear model be written as $E(Y|X) = g^{-1}(X^T \\beta)$?
I am glad to see you find it interesting. Did you study the asymptotic normallity of the ECDF?
Precisely what does a "random" set of values "look" like?  (Even very people very experienced in analysis of spatial data know that spatial correlation can be strong yet so subtle that it is difficult to pick out from a spatial plot.)
Closely related: http://stats.stackexchange.com/questions/22974 and http://stats.stackexchange.com/questions/21718.  And this one appears to be an exact duplicate: http://stats.stackexchange.com/questions/36309.  Does that answer your questions, Tom?
I'm assuming its because your data is censored after 10% of the data experiences the event?
@DmitrijCelov (+1 some time ago) nice answer -- thanks for pasting the text, as Wikipedia no longer has the points covered in the first two paragraphs. The removed paragraph was cited as p. 446: `Burnham, K. P., and Anderson, D.R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd ed. Springer-Verlag. ISBN 0-387-95364-7. ` and the pre-revision Wiki page is [here](http://en.wikipedia.org/w/index.php?title=Akaike_information_criterion&diff=424226149&oldid=424049753)
Welcome to the list. I don't really understand what you did here. What "other test"? What data have you  got and what are your hypotheses?
Thanks for the response Dilip. It was helpful.
Some very rough rules of thumb: you should investigate collinearity *before* you do any fitting. If you find it is present, you should either (a) use a method that handles collinearity, (b) remove collinear features, or (c) transform your features (eg using PCA). Once you have fitted a model, you can look for heteroscedasticity in the residuals. In general, if you are making a predictive model you should not remove outliers. Instead, use a method which is robust to the presence of outliers.
You have asked two related questions. Please combine them and delete one.
I apologize for starting a new thread, I will delete that one. Since:
$var(\\hat{\\beta_0}) = \\sigma^2(1/n + \\bar{x}^2/S_{xx})$ and $var(\\hat{\\beta_1}) = \\sigma^2/S_{xx}$ and $cov(\\hat{\\beta_0}, \\hat{\\beta_1}) = -\\bar{x}(\\sigma^2/S_{xx})$

Would it follow that: 

$\\text{cov}$$(\\hat{y_i}, \\hat{y_i}^*) = \\sigma^2(1/n + \\bar{x}^2/S_{xx}) + (x_i^* + x_i)(-\\bar{x}(\\sigma^2/S_{xx})) + (x_ix_i^*)(\\sigma^2/S_{xx}) = $

$(\\sigma^2)[(1/n + \\bar{x}^2/S_{xx}) + (x_i^* + x_i)(-\\bar{x}/S_{xx}) + (x_ix_i^*/S_{xx})]$?

And if the steps I took are correct, is there any further simplification to be done?
@DilipSarwate Oops yes - odd that I remembered it for 3 and not for 2 :-) ps if you spot any other errors in my answers I won't be offended if you edit them out yourself.
Druss, I haven't had a chance to take a look at this yet. I intend to do so soon.
Is this homework? If so, please tag it as such.
I'm not sure what you mean by "properly randomized".  Random assignment can result in baseline differences by random chance and those can impact the interpretation of subsequent difference scores.
I created a model with a function for creating time dependent covariates using the following code:
    `coxph(formula = Surv(OS, OSCheck) ~ Age + Sex + RTCT + PathologicalTstage + 
        Residualtumor + RTunexpectedinterruptiondaysduetomedicalreasonspostoper + 
        tt(PathologicalNstage) + tt(Gradeofhistology) + tt(ChemotherapyCyclesadjuvant), 
        data = dataset.IMP.1, tt = function(x, t, ...) (x/t))` I used a transformation of covariates for the inverse of time. I'd like to check, using something like **cox.zph** if this assumption is correct.
For empirical percentiles you don't think of any distributional issues, because it is an order statistic. It is calculated: Take any student's score, count how many students scored less than this score, to this count add one half of the frequency of the score of interest in the sample, and divide this sum by the total number of students in the sample, then multiply by 100. It's the empirical percentile of this student.
If in doubt, you can always use bagging in addition to ridge regression as a sort of "belt-and-braces" approach to avoiding over-fitting.
To create matrices as in the first example, Seth, try this: `n <- 3*5; sigma <- diag(1, nrow=n, ncol=n); sigma[rbind(cbind(1:n-1,1:n),cbind(1:n,1:n-1))] <- 1/2`.  A similar approach will generate the second example.  However, they have a common problem: you have lost the covariances among the $y$'s during each period--**these matrices do not reflect a repeated measures structure.**
I'm also not sure about the question, but actually, the Lasso is a simplification of the Lars: 
For Lasso, you're only looking for positive correlations between the current residual and the remaining basis functions, since only positive correlations lead to positive (~non-negative) coefficients.
@Dmitrij Celov: Then why are you comparing AICs?  Keep in mind that AIC is asymptotically equivalent to leave-one-out cross-validation, so I suspect comparisons of either metric would be similar. http://stats.stackexchange.com/a/587/2817
The size of the sample $n$ is used when we are _estimating_ the expected value from a sample. When we _define_ the expected value the "weight" factor is the pmf itself. Indeed, the expected value is a "weighted" average, while the sample mean is an "unweighted" average (nevertheless, the latter is a consistent estimator of the former). As for the second issue, you are confused because in other scientific fields, the symbol $< >$ is used _instead_ of the symbol $E$ -it means exactly the same thing.
I used that approach, but it does not use the features of the items. I would like to include features as well.
You probably have read the Wikipedia article: http://en.wikipedia.org/wiki/Boxplot Could you explain more clearly what you are not understanding?
Thanks, so I will report this $p$ in the publication. Your interpretation of my question was correct, sorry for the confusion. I do want to create a 99% confidence interval of my parameter.
I think the graph tells much of the story except for not having confidence bands.  My R rms package may help in getting contrasts of interest in an intuitive non-error-prone way.  Here are examples: http://www.oga-lab.net/RGM2/func.php?rd_id=rms:contrast
Please consider adding some information. Are there assumptions on the distributions of the variables? What do you mean by values set? I'm assuming you don't get to observe the entire "values set", otherwise you could just look at the maximum value and see if it was greater than 10...
+1 That first method is clever.
great question: see also this related question on correlation and causation http://stats.stackexchange.com/questions/534/under-what-conditions-does-correlation-imply-causation
Done :) @seanv507
glm() is not robust, and a quick look at lrm() doesn't tell me that it's robust either. Am I missing something? Thanks.
Thanks, for the explanation i'll think i will go for the wilcox.test() for my data. Do you think that will be a good one
This is not really a question about frequency or non-frequency definitions of probability - it is a question about finite population inference from a selected sample.
You have to select the criterion you want to minimise/maximise and the find the optimal bandwidth. An alternative is using automatic nonparametric estimation, such as the one proposed [here](http://www.statslab.cam.ac.uk/~rjs57/CSSRevLV.pdf). The R packages [LogConcDEAD](http://www.mirrorservice.org/sites/lib.stat.cmu.edu/R/CRAN/doc/packages/LogConcDEAD.pdf) and [logcondens](http://cran.r-project.org/web/packages/logcondens/logcondens.pdf) might be of interest if you find this approach appealing.
I'm reluctant to give advice. And don't want to be negative. But there are many oddities here:(1) why is increase highlighted? seems odd thing to emphasize. (2) why is logistic regression an option you're considering but not linear? (3) why do you think linear regression is only used for prediction? (4) causality is incredibly hard to prove. I would avoid using the word.
Where you said: "*saying "confidently" that the average of adult in the US sleeps 6.5 hours*". Well, no, you can be quite confident it *isn't* actually 6.5 hours on average. You can only be confident it's *quite close* to 6.5 hours, or you could be confident that it's '6.5 hours to the nearest 5 minutes', or some such. Only ranges will have some confidence level associated with them.
@ttphns: Then why am I getting different p-values  for the F test depending on the reference group?
Thx I will take a look on multinomial logit regression.
I like the Xeger solution linked to in the first post: it obtains the deterministic FSA created by the system and then makes random transitions within it.  (The snarky negative comments in the second post seem altogether ignorant of that simple, valid approach).  AFAIK, nothing in R provides the equivalent capability.  Consider using something like Xeger to generate a text file of strings and then read it into R for the intended statistical processing.
Are you looking for sudden breaks (aka changepoints) in your time series? Do you have any idea on the process that your data follow? Is it random walk, is it stationary, what is the autocorrelation structure? BTW, the tags `reliability` and `resampling` sound a bit misleading, at least in terms of the problem you're **apparently** trying to solve. Just my two cents...
@Newb You appear to have two unregistered accounts. Could you confirm that the one you used to leave this comment is the same as the OP? A moderator can merge your accounts, but you'll need to register once and for all.
thanks a lot for the comment and for the paper.
Please, give an example of use for semi-log plot in R to make your answer complete and acceptable.
Correct me if I'm wrong, but, I thought that a Cauchy distribution was due to the ratio of two independent normal variables.  Here, y is dependent on x, so, this wouldn't quite be right.  But this seems like it's on the right track...
(+1) Although this is not a full answer, it presents a good way to go about finding $f_Y$ and clearly shows why it is a sum of two pieces, one for each square root.
+1 This is an excellent answer because it avoids the arcana of p-values, probabilities, distributions, null hypotheses, etc., and gets right to the point in a way that is applicable to what most lawyers will be dealing with. That it might not be comprehensive is beside the point: the details and variations can be handled later. If pressed to improve on this, the main change I would make would be to emphasize that beliefs about statistical significance are based on *data*: that would distinguish this description from, say, theological beliefs.
@NeilG Yes, your post is greatly improved and I am up-voting it.  One minor comment, though.  $A$ and $B$ being disjoint implies that $P(A \\cap B) = 0$, but the converse is not true, and so your "if and only if" assertion (third sentence of first paragraph) is not correct.  I have posted a separate answer too.
It is indeed very slick using total expectation/variance to deal with this question. This is not remotely necessary, but do you think there's a way to make my initial approach work?
Greg Ridgeway has done a bit of work on using [gradient boosting](http://scholar.google.com/citations?user=r_PiKGMAAAAJ&hl=en&oi=ao) and then using those predictions for propensity score models. So your idea is not totally without precedent.
what do you mean by 'distance'?
You are probably looking for something like [this](http://www.tandfonline.com/doi/abs/10.1080/03610910600880328).
Thanks. So in general to construct the multivariate Brownian bridge from the one-dimensional (based on each marginal), one just need to  multiply them by the square root of the variance-covariance matrix?
@gung and ttnphns:  The journal issue I mentioned seems to have been in 1990, volume 25, judging from Wayne Velicer's publication list at http://www.uri.edu/research/cprc/Publications/Authors/WVelicer.htm.  See #10, Velicer, W. F., & Jackson, D.N.  (1990).  Component analysis vs. common factor analysis: Some issues in selecting an appropriate procedure.   Multivariate Behavioral Research, 25, 1-28. Also see the many references to articles from this issue in http://pareonline.net/pdf/v10n7.pdf.
In excel, your best bet is probably to assume that sales tomorrow will be the same as sales today.  This is called the "naive" forecast, and is usually a great place to start.
Eigenvectors are just the linear combinations of the original variables (in the simple or rotated factor space); they described how variables "contribute" to each factor axis. Basically, think of PCA as as way to construct new axes that point to the directions of maximal variance (in the original variable space), as expressed by the eigenvalue, and how variables contributions are weighted or linearly transformed in this new space.
Thank you for the post Clark.
@COOLSerdash No, I mean regression models for a multivariate dependent variable. Like `lm(cbind(x,y)~time` in R.
Have a look at point 4 of [this entry](http://mathworld.wolfram.com/Tilde.html) from Wolfram MathWorld.
fwiw this doesn't substantially differ from a Bayesian perspective: the person's true score is still fixed and a random (posterior) distribution describes your uncertainty about that fixed value, which has a mean and some std dev. On the other hand in actual data analysis contexts the posterior and thus its mean and std dev. are exact not estimated (modulo mcmc issues etc.) Upshot is that the Dudek quote just seems a bit misleading, though I suppose it might make more sense in context.
A really dynamite addition.
So, you have N respondents x P variables binary (1 vs 0) data, and seek a SPSS solution to compute frequences of "all=1" for all possible combinations - by 2,3,...,P - of variables? Is that right?
For `R` there are the packages [copula](http://cran.r-project.org/web/packages/copula/index.html) and [CDVine](http://cran.r-project.org/web/packages/CDVine/index.html)
Like the AirPassenger data in Box and Jenkins? Please note that this sort of question ... "*Where can I find data that fits these criteria*" doesn't seem to fall under the scope of CV as defined in the [faq](http://stats.stackexchange.com/faq#questions).
Need to know i) analysis you're trying to run, ii) what are the nature of the three variables, and iii) what exactly are you trying to test/show with the box plot.
Another related name is the omitted variable bias, if that helps you.
Have you tried two way `ANOVA`?
Thanks, i'll take a look!
might be worth adding that threshold of 0.5 implies a loss function of 1:1
This overly broad question could be greatly improved by performing preliminary research and being more specific about what kind of model is under consideration.
@Tim - suppose you have some family of distributions symmetric around 0 but with two free parameters you wish to estimate.  The symmetry around 0 would make all odd moments strictly 0, and so to use MM you'd need to look at even moments.
This is a *very* application-specific problem. You will need to provide more details on the actual problem of interest.
@sadaf: A common technique is to approach *some* of the 36% of non-responders & cajole/bribe/menace them into responding - resource constraints won't allow you do this for *all*, but you then have a sample from the non-responder group - it's a way of getting the information on them that Peter mentioned
Sounds like the assumptions of the t-test aren't met, so the p-values it would generate are irrelevant.
@BenjaminLind I may mean that I'm not sure.  What I'm talking about it the distribution of arrow heads.  Are these degrees rather than vertices?
Nice! Do you object showing the code?
Correct (though strictly you're leaving out some equalities at the bounds, they have probability of measure zero so teh answer won't change) - but you should probably add that question to your original post
thanks for the AIC! it helped  a little ;-)
Welcome to the site, @Krish. I'm not sure how much this answers the OP's question, which seems to be about progress in using multiple imputation. Would you mind editing your answer to draw out the connection more explicitly?
It's the same true for logistic with glm?