tttjjj/elt_bench huggingface dataset:
Another example with 11 classes and 10 variables is on page 118 of [Elements of Statistical Learning](http://www-stat.stanford.edu/~tibs/ElemStatLearn); not terribly informative.
Right. But my question was: why do I get this gap in R, but not in Stata... I gather the latter has a different way of calculating the residuals, but can't see why or what it would be.
Actually, I think this is the right answer (+1), and as such, it's fine that it isn't a comment.
(i) convolution only works as the density of the sum if they're independent. (ii) f/g won't be a density, though.   **Frank**: f(g(x)) won't be a density in general. There are occasional uses for such a calculation, just as there are in other areas of mathematics - they just don't produce densities. You may be able to scale to a density, but they're usually not very interesting as practical objects.
What software do you have access to? If it is any of the big three (R, Stata, SAS), you should be able to utilize their survey features that would take care of producing the correct standard errors (which is what you should be concerned with, rather than the standard deviations).
@Michael: I thought that multiway ANOVA will work perfectly if there are not dependent variables (i.e., Before and After). Do multiway ANOVA consider the dependent variables?
Really interesting suggestion whuber, and one that feels intuitively correct to me even though I'm not very skilled w/ stats (yet). I will try it (as soon as I find out how, exactly, to run the test).  And thanks for the note, Peter Flom.
Thank you for you reply. I think I have not expressed myself clearly. This kind of EMG recording is special in the sense that every stimulus results in EMG response that is in a clear relation to the stimulus. Thus if I want to compare the discrete responses with each other I can create time-windows that are related to the stimulus time and don't need to rectify. Also not only the amount of response is important but also the shape and directionality. All responses look similar to the one in the second figure. 
Please correct my if I am wrong and thanks again
For SVM, LIBSVM provides an option for generating probability estimates. Visit http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/. Then, find on page the keyword "probability".
Informally: "Don't expect to be able to easily map change in input to change in output."
If you know the names of the clusters before you start then why do cluster analysis (unsupervised learning) instead of some form of supervised learning?  If you don't know the names before you start, then I don't see how you can automate the process.
@Vincent, even if their care does not change, becoming part of a register will probably lead to changes in how they represent their health to themselves, and thus cause changes in their behaviour /end(somewhat irrelevant psychology tangent}
It would help to clarify your question: what do you want to achieve with cross-validation? Determine which smoothing model works best?
Artus, you seem to be overlooking that the prior restricts $\\theta$ to be $\\alpha$ or larger, thus avoiding the infinities you have encountered.
This makes good sense programming-wise, and also philosophically. The prior parameters are, after all, a kind of "data".
:-) of course i know how to increase the Java Heap size, but  Weka's strategy to hold all files and their info in RAM is not feasible when I use 100K files to preprocess. What's worse, my file are more than 10K words per file. So I am afraid i have to figure another "intelligent" and tricky method to handle this :-). Anyway, thanks a lot for your help !
good answer. but I believe there are textbooks written on this topic. there is no way to possibly address all the issues in a post.
Yes, I have also tried doing this and I agree its probably the easiest solution. Just though there may be something I am overlooking here. Many thanks for response.
re #3: Yes, but "transformations" of *what*? Given that heterogeneity is in the residuals (#1), what could transforming the *explanatory* variables (as suggested in the question) possibly do about that?
I gather your main effects were non-significant, was the global F significant? What was your N?
Ah! I see. I misconstrued your allusion to causality :)
@Nick I agree about the thin line.  When a question focuses purely on the mechanics of programming, its appropriateness here becomes doubtful.  You can find [periodic discussions about this](http://meta.stats.stackexchange.com/q/584) on meta.
Yes. But thinking too much isn't such a bad thing. :) So: if the kinds of *differences* you are interested in testing for are larger (in terms of time) than the gaps in your data (e.g. you are interested in long terms trends), then you might be pretty safe. On the other hand, if you are interested in testing for 'spikiness' (very short term events), then it is possible that your missing data may utterly thwart you.
@rolando2 I added the comment BEFORE audijenz edited it and I already have upvoted the edited answer ;).
Can you define precisely (in plain English) what a *change* is in this context? The test you used allows to compare two variances (assuming normality of the parent distributions).
@cardinal - Yes it was a typo and same with `mu` being 0. The normal CI works fine, it's the basic bootstrap CI that I've having difficulty with.
If you are asking if you can fit a model without any data, the answer is no.  Also, I have noticed that only a few sales can cause large changes in the Amazon rank on any given day.  So you might want to average ranks across days.  Most Authors only get sales data quarterly or semi-annually, so you might want to average the Amazon ranking over one of those periods of time.  You probably could find authors willing to cooperate with you by providing sales figures, but perhaps not authors from the very high sales volume books.  Good luck.
Sounds like you probably want [McNemar's test](http://en.wikipedia.org/wiki/McNemar%27s_test)
I'd water this down a bit. These are very widely used methods here, but by no means the only ones. Just averaging ratings of one kind or another remains a simple and often used alternative.
+1, thanks for clarifying your post & adding a valuable answer.
By definition, a "permutation" is a *rearrangement* of the set. Therefore every permutation selected (out of the set of all possible permutations) will include every particular number. If you really meant to refer to *sample* of size $100$ with replacement and that $50$ such samples are independently drawn, then please edit the question to make that as clear as possible. Otherwise, what do you mean?
BTW I apologize to anybody that feels highly offended but please notice that I already asked this question in a different forum and also asked more than one good friend of mine for help. This seems to disturb certain people to whom I want to express my sympathy.
If you have no test data, how are you defining "success"?
+1 just for the barfed line.  lol.  Have you looked into sparse matrix formats in R?
This fact is very important to embed survival analysis in "classic" statistics. What do you see when you write down the first few terms of the product?
Sounds like you want "Linear", not "Generalized Linear", then. (In terms of specifics, I've never used SPSS so can't help there, but there are definitely SPSS users around here.)
Side answer: 4+7+1+1+5+4781 = 5*8*2*2*6*4.
mh, ok, I also do not know how to implement the information-based method and yes, I read the great book of McLachlan, but this is not helping me how to implement these things. I will leave it out of my thesis and give the estimates without standard errors.
Just a comment- The present is only dependent on the the future once the future is observed. This is a property of a first-order markov process.
@Andy Provided the scales on the axes are standardized to 1 SD for each variable, the contours correspond to correlation coefficients $\\rho$.  Circular contours correspond to $\\rho\\sim 0$; skinny contours sloping up correspond to $\\rho\\sim 1$; and skinny contours sloping down correspond to $\\rho\\sim -1$.
Thanks so much. I already understand the analogy described in your solution. I will read paper. I will accept this as a answer. I don't understand PPV AND NPV.Please explain these concept as graphic as the Sens and Spec were explained and I will accept your answer.
What is your _question?_ Demands for full explanation are far too broad to be answered here: people have even written whole books on multivariate normal distributions.
How are you reconciling the statement "If $H$, then the other side will be $+$ with $5\\%$ probability, $-$ with $95\\%$ probability. $P(−|S)=0.95$" with your earlier assertion that $P(-|S)=0$ in the paragraph beginning "An example:" ?
@Anh: the tuning of λ by cross validation is not in itself bad. But doing so you "used up" that cross validation for tuning of λ and you need another validation that is independent of the modeling including the process of tuning λ. Not doing this outer validation is BAD. If that particular λ does not work "again" (e.g. on another split of the data), your optimization did not work. In that situation, you'll typically also see a large difference between outer validation results and the observed "best" performance during tuning.
Are you interested in whether the distributions are of a different form (e.g., normal, poisson, etc.) or whether parameters are different (e.g., mean or sd of a normal distribution) or both?
You may also check out `rgl`.
I think the basic assumption is that your response variable isn't continuous. I don't feel like this warrants a full answer, but there are a couple things you could try.

If want to look at was the interaction between system type and  task, you can do a goodness-of-fit test (chi-squared or fisher's exact) on the count data. This works well in a 2x2 design but it might be harder to draw conclusions on a 3x2 design.

The other thing is you could try is fitting a logistic regression model, which effectively transforms your binary response variable into a continuous one (a probability of success).
that's what I thought as well. she wanted proof haha
@Erik IBM Watson is close they say. :)
Thanks for the reply - it's been a little while since I did statistics in this much depth - would you be able to clarify the joint distribution in terms of my likelihood functions? Thanks again!
@probabilityislogic  Thanks for your reply~ I am reading the GPML written by Carl Rasmussen. I am reading Chapter 2, I'm not sure how to derive the algorithm shown on page 19, since it does not look the procedure shown before.    In addition, I'm not familiar with ML method... sorry to put you out
Thanks very much for your help. I think I will do either a binomial response glm or as you say, switch it around and do an ANOVA with an error term for farm.
this procedure should be repeated a sufficient number of times to estimate generalization performance (the value will likely differ for each split of the data set).
You say you want to use the LLN. For what and on what quantity? Note that the statement $\\mathbb E X_1 = \\mu$ for $\\mu \\in \\mathbb R$ automatically implied *by definition* that $\\mathbb E |X_1|$ is finite! (@Dilip also alludes to this in a previous comment.) So, there is *nothing* to prove in that respect. That $\\mathbb E X_1^k$ exists and is finite for any integer $k \\geq 1$ [is a *very strong* statement](http://stats.stackexchange.com/questions/26454/proof-that-if-higher-moment-exists-then-lower-moment-also-exists).
@Macro How do you estimate parameters without data?
This question appears to be off-topic because it is about just programming ... Better to ask it on stackoverflow and if you want to get an answer ... you need to provide a minimal code to re-produce the problem.
It is a Lagrangian yes.
Are you desiring a closed-form solution, or would you be willing to use MCMC techniques to generate a (large) sample from the posterior distribution?
It's that last part that I'm unsure about. How do I constrain my y_t such that it ends at the right place? I'm sure this is backcasting 101, but I guess that's what I'm asking - where can I find a backcasting 101 formula with external regressor? Thanks.
My fault, I translated it incorrectly. look at the edit
Can you explain what you mean by 'ordered numbers'? In the context of the word 'distribution', some of the answers I can think of cause me some worry that my answer heads down the wrong path.
Do you want to compare the means of the two (or 3 or more) groups?  Can you describe your concern about loss of information?
+1. Your two variables are weakly correlated. It's not clear that PCA is of much use compared with plotting the variables, calculating the correlation and thinking about those.
Given your answer, would it also be conceptually sound to use the probabilities to predict characteristics of the purchaser pool?  For example, multiply the predicted probabilities by a customer characteristic/independent variable (gender), then sum those values across customers to estimate the number (of women) in the predicted purchaser pool?  Ideally, I would develop a confidence interval for such an estimate.
(+1) With a minor note: $\\newcommand{\\z}{\\mathbf z}\\z \\z^\\prime V^\\prime AV\\z$ is not symmetric about the origin, however $\\z \\z^\\prime V^\\prime AV\\z$ and $-\\z (-\\z)^\\prime V^\\prime AV(-\\z) = -\\z \\z^\\prime V^\\prime AV\\z$ have the same distribution, which is all you need.
To get a useful answer, then, please tell us (1) what "operations" will be performed, (2) which integers might appear in the dataset, (3) how frequently they might appear, (4) how large the dataset might be, and (5) what the cost of a lossy compression scheme would be. Also, what does this question have to do with fuzzy set theory (one of your tags)?
@FaheemMitha, I have not used the `PMS` tool. My background is in haplotype inference. I honestly just pulled up wikipedia and started checking out tools.
This was a great answer.  Thanks for the effort.  I will definitely checkout the reference you mentioned.
What do you mean by "substantive interpretation"? As far as I know, the log likelihood, by itself, isn't really meaningful - it is useful for comparing models.
You should consider "cure models". A cure model is survival model with a cure fraction, i.e. a fraction of the population that is not susceptible to the event.
Thanks for the thoughtful input. In terms of the words "mean intensity," you are correct that I mean the mean of the light across the area of a spot (not of multiple subjects). In terms of the censoring of the data, I think I didn't explain it clearly.  My variable of interest isn't actually the intensity.  It is a different variable called contrast.  But when the intensity is too low, I am unable to reliably measure the contrast( I know this from previous experiments on objects with known properties).  Thus the values of the contrast, when the intensity is too low, are censored.
ahahahahahahahah
Can you check your equation editing? It's somehow not showing up clear. Thanks!
Is there anything I can do to better specify the question? I expected at least a clue on which statistical law to follow. I could find the remainder next, if I dug deep in my statistics archives...
Okay, so you if you treat the prior as part of the data generating process, and the posterior makes testable frequency-based predictions. I'll keep this question open for a few more days, but I like this answer.
I do understand that its just coded that way but I still find it difficult to build some intuition around it. Thanks !
Have you thought of centering age?
possible duplicate of [Is it meaningful to identify interaction effect when a main effect in a model (main effects only, without interaction terms) is not significant?](http://stats.stackexchange.com/questions/85940/is-it-meaningful-to-identify-interaction-effect-when-a-main-effect-in-a-model-m)
@Iyosef769: thank you so much! Sorry I mean to award the bounty for  your answer above ;-)
This question cannot be answered without a more specific description of how these "weights" are supposed to operate. From what you have written it sounds merely like your model *really* is $Y = aw_1X_1 + b(1-w_1)X_2 + \\varepsilon$--which would be fit by ordinary least squares--but that is purely a guess; other interpretations are possible.
The SE ecosystem asks that you do not cross-post. Please pick the site where you want this question to be & delete the other. Possibly both sites would be fine, but I will say that you will get answers here. Nonetheless, it is your choice.
I was one of many to have written a paper on this long ago, you can find mine here: http://theoval.cmp.uea.ac.uk/publications/pdf/ijcnn2001.pdf However, one of the problems with these adjustments is that they are only assymptotically correct in the limit of an infinite training sample.  In practice the same is usually too small for this to work as well as I'd like, so I have found that it is better to have a different C for each class and optimise them via cross-validation.
This answer is incorrect for two reasons.  First, the expected income is $200*0.1-20*0.9=2$, so this is a positive expectation bet.  Second, if it were not a positive expectation bet, then it would not be a good idea to take it even if you could do it as many times as you want.  Yes, you would eventually end up with a positive amount of money, but the expected value of the amount of time until this happens is infinite.  Moreover, if you have a finite bankroll, and you play long enough, then with probability $1$ you will eventually go broke.
I agree, I am just curious about what the differences between using a reduced sample with 50% cases and 50% non-cases (40k records) instead of 10% cases and 90% non-cases (200k records) are, if any.
Because contr.helmert is a matrix of orthogonal contrasts & the transpose of the generalized inverse is the same bar a rescaling of each column.  So the interpretation of the effect estimates is different but the t statistics are the same.  This is _not_ the case when the contrasts aren't orthogonal (which I now know from reading Sven's answer - sorry!).  Venables & Ripley cover this well in Section 6.2 of 'Modern Applied Statsitcs with S'.
With a paired t-test, the actual test is a one-sample t-test on the pair-differences.
Hi Julie. It would be nice if you can provide a full citation for each book in your answer. :)
Thank you for a thoughtful and helpful reply.  I realize you have been here for a couple months now, but because this is your first reply, it seems a good occasion to welcome you to the site.  I hope you will feel inspired to offer more such advice as time goes on!
This should be the accepted or most voted answer.
Among other places, see also a comment [here](http://stats.stackexchange.com/q/73103/3277), to convert b to beta.
I may hint towards computing and presenting _standardized_ or _adjusted_ Pearson residuals.
For the formulas, you can get a jump start at http://meta.math.stackexchange.com/questions/480/math-markup-diagrams-etc-pointers-please .
Maybe this would help your imagination: your own gravatar at http://stats.stackexchange.com/users/16534/murat-arat is an image of 128 by 128 pixels, consisting therefore of 128*128 = 2^14 numbers (at least when rendered in grayscale). A gallery of $n$ such images would be organized for statistical analysis by making a matrix of $n$ rows--one per image--and 2^14 columns--one per pixel.  Each of the rows is a vector with 2^14 *components* (which probably is what your source means by "rows").
Thanks’ @aix. Source encoding `Koi8-u` displayed as `Iso8859-5`. I am changed `russian.dat` in `utf-8`. It's working.
Another part of my question is, why 'Lasso can select at most n features'? If this is the case, I think OLS on the selected features will be at least 'good', since OLS is the 'BLUE' (Not strictly BLUE since it's mostly biased). Just consider an extreme situation that Lasso selects the exactly right features, conducting OLS on these features will restore the true model, which I think is better than the Lasso estimation.
+1 for number 2!
The average of such reliability coefficients is, just that, the average of such reliability coefficients. Why would it be interesting or useful? Best to design an analysis of all kinds of variation, between and within raters.
I am however using a subset of the NLYS79 dataset. But these are things I will of course mention in my analysis. 

what other variables? my god man. Religion, age, whether or not the person has a library card.. et cetera..

Thing is I wanted to add a lot of variables, including BMI, length, and other things. But my professor said that would take a lot of work commenting on the outcomes.
Please go easy on the implicit condemnation of the question, Michael. If you think it is "incorrectly worded" then help the asker improve and clarify it, rather than jumping in with criticism and and answering what you *think* the question might be asking.
The following Qs are strongly related & will be of interest: [How to generate correlated random numbers (given means variances and degree of correlation)](http://stats.stackexchange.com/questions/38856/) & [Generate a random variable with a defined correlation to an existing variable](http://stats.stackexchange.com/questions/15011/).
@BigDataLover:  have seen some posts saying that Cross-Validation is not needed for [random Forest]. => depends on your data: all random Forest implementation I know assume that the rows are independent of each other. If they are, the rF's internal oob estimate is OK. If not, you need to do your own validation to test with independent data.
(+1) A possible exception is when we have a set of indicators (e.g., items in a questionnaire) with a fixed number of categories (e.g., Likert-type items), as discussed in [Visualizing Likert Item Response Data](http://stats.stackexchange.com/questions/3921/visualizing-likert-item-response-data), or a single crossing factor (e.g., gender), in which case an horizontal (we are better at visually discriminating along the horizontal than the vertical axis) stacked barchart might be interesting, IMO.
Thanks for your answer BTK, though it is more of a comment. To make it more of an answer, you might want to add a bit more detail on DBSCAN and how it applies to the specific question at hand.
@NickStauner Hi, the thing that significance is of no surprise with such a large sample. So I think maybe r-squared is a better measure to gauge the effect, although it is a different thing. The marginal r-square is so small, which indicates that even with mixed model, the observations vary so much compared to the fitted lines. The effects show the difference in means, but small marginal r-square indicates that the fix-effects are not that convincing, right? That's why I am not that confident to report them.
@whuber, just motivation and uses really. I see people say that something is a normal by chi2 by dof and I ask them why did anyone ever try to do such a thing and they have no clue.
I may be misunderstanding this question, but it seems that you're asking how to specify a prior on a node value when the node has parents.  But you don't need to.  Only nodes without parents need priors.  You may want priors on some parameters that define the *mapping* from parent(s) to child.  But these are priors on parameters not priors on node values not on values.
Thanks you're right. I now - at least - am able to do k-fold-cv by transposing and 'cutting' the gram-matrix. As I of course need a matrix with train*train and for testing one with test*train. Thanks a lot!
+1, this is a brilliant one. I'm enjoying it right now.
Thanks, I think that's what it is!
That was fast and simple. Thanks a lot.
You could always just run autocorrelations on the raw response data, instead of the residuals. Have you searched the site? This strikes me as a question that's been addressed before.
$\\lfloor \\log_k \\text{size}\\rfloor$ will give you classes in one step, for some $k$,
$f$ is pdf and $F$ is the cdf. Maybe I missed something but : I consider case $s\\not= t$ where the derivative is zero, and the other case where $s= t$ where it is not. In other words, I do not differentiate the indicator.