tttjjj/elt_bench huggingface dataset:
Nice use of an often-forgotten tool... and, of course, for either of $\\lambda_1$ or $\\lambda_2 \\leq 5$ or so, the brute force convolution method won't be all that painful.
@D L Dahly nice clear answer. As a technical note, in your third case, if there is another covariate C2 that is a confounder of the X-Y relationship, then controlling for the original C can cause collider bias. This is probably not of practical importance, though, since if you are already adjusting for C2 in the analysis any potential collider bias is blocked; and if you aren't adjusting for C2 then the confounding is a concern regardless of whether you adjust for the original C or not. And in answer to the OP, C could be called a 'downstream consequence of Y'.
[This reference](http://www.kamalnigam.com/papers/canopy-kdd00.pdf) vaguely hints that T1 and T2 can be set with "cross-validation."  Note that these thresholds depend intimately on the nature of the metric, on the dimension of the problem, and even on the distribution of the data.
A Gaussian copula isn't the only choice of copula (and in many ways it's rather restrictive - for example, in that it doesn't allow tail dependence); there are many other choices. I'd probably start by looking at some of the Archimedean copulas.
I feel this answer (to my mind, a correct one) should be turned into a comment, unless some explanatory body is attached to it.
Because whether and how to transform data [depends on their characteristics (more than their signs!) and on the purpose of the transformation](http://stats.stackexchange.com/questions/298), please give us more information about both so that this can be an objectively answerable question.
Hierarchical models are a subset of multi-level models. By definition a hierarchical model does not allow for crossed random-effects but only for fully nested ones. eg. your typical County>School>Student paradigm is fully nested. A school can only be in only one county and student at only a single school (in  a fixed time-point that is). On the other hand a study where you examine the yield of corn varieties with respect to different fertilizers and soil quality is/can be crossed. You do not use specific fertilizers exclusively in fields of a certain soil type.
@dail, try Charlie's example with `z <- seq(1, 210)`. it still works (rejects at a *slightly* lower level, but the same order of magnitude).
Thank you very much for your help and detailed answer. Much aappreciated!
(+1) Your book? Oh damn. I *so* have 387 questions for you :)
Are these the elements which show which model fits best?
Thanks. That is a lot of information to go through!
Many thanks for your reply, DBSCAN does seem more appropriate as I certainly have many outliers and peculiarities.

Just out of interest, why would you suggest not to use kmeans?
It's an interesting question, David, but I wonder about your application: what population are you trying to make inferences about?  Arguably, your 100 grades form a complete census, in which case the test is simple: because 36 exceeds both 33 and 31, a grade of A is indeed the mode.
@ChristianBueno Okay. I am not aware of a particularly intuitive meaning in general. In some specific situations, perhaps.
did you read the entire Wikipedia page? There is a section on geometry and on conditional independence for the normal distribution. You can find more in [this book](http://www.amazon.com/Graphical-Models-Oxford-Statistical-Science/dp/0198522193).
How was stress urinary incontinence measured?

What other data do you have on the people?

How were people assigned to TVT vs. abbrevo?
I don't think anyone was suggesting subdividing cross-validated, just a discussion of the subdivision of statistics.
(+1) A nice feature of the logistic versus probit is that it is relatively insensitive to a small number of responses that get incorrectly coded. This is, of course, related to the fact that it has fatter tails.
Don't you forget the covariance between $b_{0k}$ and $b_{1k}$ in your last formula?
Yes, you do understand me correctly, the idea is to check how important the way is how the groups are divided and if the result is the same with a slightly different approach. Thanks for your answer
Then you should look into time series models.
Thanks,I'll try using that package.
Dear Ittay, I believe [Tom Cover](http://en.wikipedia.org/wiki/Thomas_M._Cover) is the original source of this problem. I think it is also listed in his *[Open Problems in Communication and Computation](http://www.amazon.com/Problems-Communication-Computation-Thomas-Cover/dp/1461291623)*, but I don't have it handy to check. It's a nice problem. The restriction to $[0,1]$, or, even a random $y$ (or $x$, for that matter) is inessential. Cheers.
I'm going to migrate this question to the stats.SE site. I expect there will be many more people there who will be able to help.
Excellent!  Now, when you're coming up with the joint log-posterior, it has $\\mu$ in it, but Park and Casella's version doesn't, because it's the log of the marginal posterior (with $\\mu$ integrated out.)  So you'll have to integrate $\\mu$ out of your expression for $f(y|\\beta, \\sigma^2, \\mu)$, then take the log to get the "log marginal posterior".  You may (or may not) know how to do this; more hints are forthcoming if you don't.
$x$ is the visible variable which are observed, $y$ is the hidden variable (the target), say, $x$ is the set of words contained in an email, and $y$ is the class (spam or ham email). If I try to fit model to connect $x$ and $y$, so the model parameter $\\theta$ is what I want to estimate, right?
@ttnphns If the poster has a question about it I will follow up.  As I said you should check to see if the groupings are distinguishable.  It seems clear to me.
@PeterFlom It often makes a difference; for example, if one proposes to use a rank-based technique (it is stated to be *continuous*, so it must be okay, right?) ... but in fact that spike at zero is a problem. Better to be crystal clear about the properties of the random variables are that are at hand than let sloppiness lead us into otherwise easily-avoided errors. For example, if someone really is analyzing flight miles and for those that travel at all, a gamma distribution is a reasonable model, a zero-inflated distribution may work quite well for the overall data, ... (ctd)
Hm. The nature of the data tends to be mostly oscillating, but not entirely and not always. Sometimes it tends to be more linear. No, I don't really have any other variables that could be applied. Predicting this could just be too difficult...
Hi Glen. I am glad it "seems" like homework because the answers will be easy.Nature of the enquiry is to help students that are competing in a nationwide test to be informed of the popularity of their choices before they make their choice.I have come up with a ranking of my own, bit it seems to be too simple. For example if one course has the first candidate r1 (the top student in the test) and the last student that has elected for that course is r18000, I would like the calculated ranking to take this "large" spread into account. Not sure if I am making much sense :(
@Arun, I'm not R user so can't know.
Presumably those in Tukey "Exploratory data analysis", Mosteller and Tukey "Data analysis and regression", and whoever cites them.
Have you actually tried fitting the same models/data in Stata and R and comparing answers?  I *strongly* suspect that the only difference will be that Stata suppresses the *output* of the intercept parameter. Except in very unusual cases, it should still be *estimating* an intercept parameter.  It may just not tell you about it.  If you posted data in a reproducible form (see http://tinyurl.com/reproducible-000 ) and showed the Stata results (and ideally the R results, showing they don't match) we could move forward from there ...
@Glen_b well I was asking basically if the thetas in the output of the Gibbs sampler are of any use.
@MichealMayer An answer from you would be greatly appreciated. It seems that some clarification of the question would be useful, but I'm not sure I understand the issues well enough at present to give that clarification. Is there a question I can answer to make that clarification?
(+1) I like the use of `interaction()`. Of note, we can specify `geom_boxplot(position = position_dodge(width = .9))` to add extra space between boxplots.
(+1) The "difference is exactly 0" problem is used as an example far too often. In practice, the kind of hyptheses that you propose are often of much greater interest.
Drinking detergent may affect your driving.
Ah, some clarification would be useful then.  I read the comment from @cnosam "looking at plots and knowing a good form for the distribution function" and assumed this was a manual decision-making process.
Take an arithmetic random walk. Exponentiate it. That's a geometric random walk. However, be careful, because some authors mean something quite different by the term *geometric random walk* (one example, assuming my quick glance didn't miss something important, is [here](http://www.cc.gatech.edu/~vempala/papers/survey.pdf))
@whuber I'd like to predict the color of future objects whose color will be unknown.
I forgot to add the column of $1$s to my $X$ matrix. Thanks the code works well now :)
I would recommend that you to split this across three separate questions.
The glmnet method for `caret` requires the `glmnet` package.  The `glmnet` package require `Matrix`.  They were probably both installed when you installed caret.  Type `sessionInfo()` to see which versions you are running, and upgrade if needed.
For example surges in advertising spend (external regressor) that cause an increase in sales but with a few days lag. 
Further, auto.arima will not work if the number of rows in the main series is not equal to the number of rows in the xreg series.  Would that mean that if differenced, they will be differenced with the same order, and the dummy variable will also be differenced?
Thank you very much in advance, 
Farhad
I provided some information regarding that on a related thread: [Computing the decision boundary of a linear SVM model](http://stats.stackexchange.com/a/5080/930).
How are you defining redundancy?
Java code only regresses $y$ on $geno_A$, i.e. you get the same result with $lm(test_trait \\sim geno_A)$. I know next to nothing about Java but you probably need to use multiple regression command as opposed to SimpleRegression.
Of possible interest: [Generative vs. discriminative](http://stats.stackexchange.com/q/12421/930).
The problem is that 99.9% of political debates and decisions happen in the 'research question' domain, most often with 'supporting' statistics :(
I kind of hope that the normalized residual variance is homoskedastic and does not exhibit any patterns any more. If so, let $s^2$ be the mean of the squared normalized residuals. You can now forecast bookings for the next couple of time buckets and use $k\\times\\hat{y}\\times s$ as the width of the detection band, where $\\hat{y}\\times s$ is a "Poisson-y" level-dependent standard deviation, and $k$ is a parameter you could tune to get the desired amount of alerts.
I don't use R. What is the most frequent vector - there are only 32 different vectors, there must be duplicates!
@FraserOfSmeg, it's not always an error. A useful way to think about this is through Taylor expansion. Most functions have the first order derivative not zero. By omitting x(1) you are making a statement about your true model relationship. It is not necessarily a wrong statement, of course, but it is very strong.
Thanks for the clarification. What can we say about the joint probabilities, i.e. P(white,a) ?  I think the question can't be answered unless we're willing to make some assumption about that (the most obvious thing would be to assume independence, but that may not be sensible in this context)
Heh I'm dumb, thanks
@MarkDollar (...) You might also take a look at UCLA resources on [Tobit regression with R](http://www.ats.ucla.edu/stat/R/dae/tobit.htm) and check whether the [VGAM](http://cran.r-project.org/web/packages/VGAM/index.html) package they used contains additional diagnostic tools.
I realize after posting that my reference to 'Hubbert Curves' in the question title may not be appropriate, as all I can say is that my plot *looks like* Hubbert curves.
Try "Data Clustering" by Gan, Ma and Wu. But even Wikipedia has various more refined data clustering algorithms than variance minimization. For example DBSCAN and OPTICS must be covered in any reasonable book on clustering. They're some of the most cited clustering papers, and they work really well in my experiments.
@Procrastinator: My post is asking this: how do you know the constructed MC by MH has the limiting distribution? In other words, what does MH do to ensure the distribution of $X_t$ will converge as $t \\to \\infty$? Without knowing that, no matter how long you run the algorithms, the distribution may even not converge!
First off, do you understand how a q-value is calculated from a p-value? http://en.wikipedia.org/wiki/False_discovery_rate#q-value
I suggest trying the other models and see whether the code works or not. R is usually nice, so anova should return similar things for different models. The problem is with your initial formula. Does it hold for other models? If it does not, then there is no point in getting code to work, furthermore the code should issue a warning that it is used for models where formula does not hold.
@Michael: If $a_1=0$, for instance, then the independence of the two linear combinations tells us nothing at all about $X_1$, so we cannot conclude it is normal. From that I infer that "$a_i b_i\\ne 0$" is intended to say that all the coefficients $a_i$ and $b_i$ must be nonzero. The statement still is not correct, however: an orthogonality condition has to be included in the assumptions somewhere.
Some websites with statistics games: [1](http://www.homeschoolmath.net/online/statistics.php), [2](http://www.onlinemathlearning.com/statistics-games.html), [3](http://economics-files.pomona.edu/statsite/Statgame.html), [4](http://www.stats.govt.nz/tools_and_services/services/schools_corner/activities/interactive-games.aspx). What statistics level are you interested in?
Sorry, I was going by which one you called "Lilliefors" -- but no, it's not exactly your approach 3 either.
@Glen_b: The MA-plot is an application of a Bland–Altman plot for visual representation of microarray gene expression data. The Bland-Altman plot is known as Tukey's mean-difference plot in other fields.
@raegtin, I have edited the answer to explain my point of view, that this is a model of the covariance matrix. Any choice of factors obtained by rotations are, as I see it, equally good or bad at explaining the covariances in the data as they produce the same covariance matrix.
+1. I always have a sweet-spot for question archaeologists. On another note, R2013b seems to have really invested in the Stats toolbox functionality. I also saw a first *actually usable* LME implementation.
Ahh..Given that $\\frac{\\bar{X} - E\\left( \\bar{X} \\right)}{\\sqrt{ Var(\\bar{X}) }} = \\sqrt{n} \\frac{\\bar{X} - \\mu}{\\sigma} \\xrightarrow{d}\\ \\mathcal{N}(0,\\;1)$ if we multiply $\\frac{\\bar{X} - E\\left( \\bar{X} \\right)}{\\sqrt{ Var(\\bar{X}) }}$ by $\\sigma$ we get what was shown by the OP ($\\sigma$ cancel): namely $\\sqrt{n}\\bigg(\\bigg(\\frac{1}{n}\\sum_{i=1}^n X_i\\bigg) - \\mu\\bigg)$. But we know that VAR(aX) = a^2Var(X) where in this case a= $\\sigma^2$ and Var(X) is 1 so the distribution is $\\mathcal{N}(0,\\;\\sigma^2)$.
@John Thanks--it turns out this question was cross-posted at quant and was closed there as being off-topic.
Great answer, but regarding the last point, observed variables are linear combinations of the relevant factors and the error term - factors are not a linear combination of the observed variables (instead they refect the variation shared amongst them). Factors can only be a linear combination of the observed variables if you "flip the arrows" i.e. use a "causal indicator model" see --> http://www.ncbi.nlm.nih.gov/pubmed/19266502
No, CI's of effects do NOT in any way obscure the results.  The CI of the effect will either not cross 0 or will.  It is very straightforward to interpret.  And the SE?... that has no intpretation issues?  Inference from it is dependant upon N and a distance beyond overlap of the bars.  It's very rare to come across someone who can genuinely make a correct inference from an SE error bar.
@COOLSerdash see my update. Yes it is possible! But you would need to figure out the best way to compare the estimated means from the resynthesized time series with the original means from the original time series.
Have you looked at the R help page for poly()?
@user248237 Yeah, but the question is too subjective/broad to get a convincing answer in the general case, so I just wanted to point those references.
Except for a little odd wording (which I corrected in my edit) I think this is correct.
He gives a gradient boosting method as an alternative.  I guess I'll try downloading the source for randomForest and see what I'd have to change to give it cost sensitivity.   Looking at the documentation on the neuralnet package, I believe It allowed defining a custom loss function.  If that's the case, and there's some example I can work from, it shouldn't be *too* bad to get this done.  also, limiting responses to 5 minutes of editing is silly when a carriage return sends the comment in.
Thanks everyone for the suggestions. Helped me big time!!!
It seems natural to have control data for left and right arms separately, x1 and x2.
First, thanks for your answer. A few questions - does the distribution not matter then? Also, when you say weekend factor and seasonal factor, how exactly would I calculate what these factors should be? Does that make sense? I mean I know when I calculate the parameters for an ARIMA model, I'm minimizing an error function. Are you saying just add on another term which is then used in an algorithm to minimize error or is there actually an option in ARIMA such that I can capture the weekend trend?
Great graph, thank you!
Applying repeated 50/50 holdout trials will give you some feel for if the training and test sets are balanced. They probably won't be. Some models, such as KNN, will achieve perfect accuracy on the training set so are inherently overfitted in some sense. But they may still achieve acceptable performance on the test set. So as with much in machine learning, in the end it all depends on the precise specifics of the data.
And how would you do it given that I reduce the number of properties I consider to a reasonable number? (The number of people having a particular property would be around 50-5000 depending on the property - if that plays a role as well)
Your idea of lower weights for old data is similar to "option 1" in fact. Instead, I'm saying that the whole _trajectory of the changing taste_ is important - ie, if yesterday you liked genre A and today you like genre B, the system would look at other people with same "A-B-??" taste moves and would suggest that tomorrow you would like genre C.
I read about Karhunen Loeve expansion allows using class information for feature extraction. Furthermore, there are extensions to PCA like using the weighted average of class covariance matrices instead of a global matrix. Apart from this information, I am also interested in possible answers to your question.
@Zach: Nice Kaggle rating! (I clicked through the link in your mouse-over profile)
Are you aware of Dynamic Factor Analysis? See this publication, [Molenaar, De Gooijer & Schmitz (1992)](http://www1.fee.uva.nl/pp/bin/refereedjournalpublication1976fulltext.pdf) for an example.
Is that all the data you have? I would gather more data from a larger population and do some nonparametric/histogram-like estimation to get a feeling, but if that's all you have, I don't know what you can do.
Playing devil's advocate, one "otherwise" is that academics need to publish papers and get grants to be awarded tenure or to be promoted.  Also methods are often in fasion and get used more than their true value would suggests (until the next bandwagon comes along).  The majority of papers published in machine learning probably do not represent an improvement on standard methods, such as kNN or LDA or logistic regression.
http://www.statistics.com/dr.-michael-chernick
two sample proportion z-test requires that two samples are random and independent of each other. Considering that we split the same foobars into "before" and "after" an arbitrary event (start of new month), are these samples random and independent?
The existence of an interaction does not necessarily have anything to do with it. When the predictors are correlated, it is quite reasonable to have a significant model overall, without any individual predictors being significant. Read the linked thread for more information. If there's anything you still need to know afterwards, come back here & edit your Q w/ what you've learned & what you still don't understand.
Thank you for your reply.I found this paper which closely deals with what I want to do "Recognition of Human Gaits" and "Learning Stable Non-Linear Dynamical Systems
with Gaussian Mixture Models". In this paper, the joint angle values are captured and  a dynamical model is constructed from it.This is what I wish to do.HEnce,the question about distribution assumption.So,I wanted to know the preliminaries like how would each of the feature column which are recorded for each time instant be given as input into a regression model.
@BondedDust You should check your math: http://bit.ly/1vZiX8v. Remember that you need to integrate over both variables, not just x.
Are you asking about how to do this in this particular package, or is your question about whether mediation of interactions is possible at all?
The parameterization of your last equation does not match the first two equations. I used the parameterization of the first two in my answer.
Correct. Linear regression can encompass both [simple-](http://en.wikipedia.org/wiki/Simple_linear_regression) and [multiple-](http://en.wikipedia.org/wiki/Multiple_linear_regression#Simple_and_multiple_regression) linear regression (though perhaps more frequently refers to simple linear regression). But those latter two terms are mutually exclusive categories.
This is nearly the same question as http://stats.stackexchange.com/questions/40602/can-i-interpret-the-inclusion-of-a-quadratic-term-in-logistic-regression-as-indi (which is asked in the context of logistic regression), so the answer there would apply.
How does this question differ from the previous one at http://stats.stackexchange.com/questions/71064/test-to-see-what-population-an-observation-came-from?
It will only give a damped trend model if including the damping improves the AIC.
@mbq Statistics on its own is nothing. The numbers must mean something!
@image_doctor, I must respectfully disagree. In the past 5 years, there's been a a great deal of interest in these models. There's at least a dozen such papers accepted to [NIPS this year](http://nips.cc/Conferences/2012/Program/accepted-papers.php), and similar numbers for the past several years. Similarly, over the last few years there's been regular workshops and conferences devoted to Bayesian Nonparametrics. [See here for one held at NIPS last year that discusses their recent trend](http://people.seas.harvard.edu/~rpa/nips2011npbayes.html).
I'm not positive I understand what you mean. The thing is that the data will vary massively year on year and survey to survey because. The distributions are absolutely random, they really are. But if you are asking if there is a single mode, in most cases there is. It's just that the value is very different survey to surbey, year on year.
maybe the linear regression return the degrees directly with some function?