tttjjj/elt_bench huggingface dataset:
Thank you, this is a better question, & one that I think *is* answerable.
@bayer: I had read these as equally probable values in my answer below.  I now see they might be probabilities: before the question was changed, they did add up to 1. That would make a difference.
See http://otexts.com/fpp/ for an introduction to time series forecasting in R.
The key words you are looking for are *superposition* and *thinning* of the Poisson process. The answer, with some qualifications, is **yes**. But, an affirmative answer depends intimately on (i) the independence of the two processes in the first case and (ii) how the splitting is done in the second case. :)
Without further elaboration, this is better placed as a comment.
(+1) Your comments on the possible ways to generate relevant synthetic data are very welcome.
I believe this is a topic being addressed in [current research](http://arxiv.org/abs/1201.6568)
$\\chi^2$ is a [Pearson's Chi-Squared Test](http://en.wikipedia.org/wiki/Pearson's_chi-squared_test) and the $\\Phi$ coefficient is the [Phi Coefficient](http://en.wikipedia.org/wiki/Phi_coefficient)
@Macro: It is not homework. I am reading a paper. I changed it to "I think it will be".
Don't you need a product over $i$ in there somewhere?
@Macro Thanks.  Regarding your question, the OP doesn't seem to know what R square is, or at least what the adjusted R square is.  I am giving background so that the OP could understand what the person might have meant had he or she been referring to regression rather than factor analysis.  But the only point i am making about factor analysis is the same as what others wrote in their comments, namely that the reason that the OP could not find adjusted R square in the R factor analysis routine is because adjusted R square is not relevant to factor analysis.
I'm marking this as correct as it is short and sweet.  However, see whuber's lovely longer explication as to WHY it is correct.  And, indeed, in fooling around with the problem, weighted by n produces the correct parameter estimates and relatively similar SE and p values to working with simulated data.
I found that I cannot do it for one dimension:

`> regular_indscal1<-smacofIndDiff(regular_list, ndim=1, metric=FALSE, constraint="indscal",itmax=10000)
Error in wr[[j]] %*% yr[[j]] %*% bconf[[j]] : non-conformable arguments`

Maybe that compromises my second proposal?
Your statement seems to implicitly assume the existence of a density (with respect to Lebesgue measure). You may be interested in this **short and delightful** paper: A. R. Barron (1986), [Entropy and the Central Limit Theorem](http://projecteuclid.org/euclid.aop/1176992632)
*Ann. Probab.*, vol 14, no. 1, 336-342. (*open access*).
@Kjetil My suspicion is that this is merely a matter of what units the the independent variables are expressed in.  "High"and "very high" are meaningless concepts on an absolute scale.  Until some details of the data are disclosed, we won't be able to determine whether there is any problem here or not.  User52978, please edit your question to clarify this point.
Do you have a response? Binary 0/1, that is Not fraud/Fraud?
Quantum - That doesn't address @gung's concern. Link-only answers aren't usually acceptable. Gung has asked if you could 'give a brief summary of these papers' (meaning 'in your answer'). Your collective description in a comment isn't sufficient. Please give a brief description of each link (if possible, individually, not collectively) that would indicate why you included it/why it's relevant. As it stands your potentially useful answer risks being converted to a comment -- as has already happened with earlier link-only responses to this question.
Thanks @zkurtz, As you suggested how can I find union using class probabilities?
If the original model LR $\\chi^2$ test is insignificant, you are on shaky ground to do anything.
I support Nick. FA is to uncover common latent traits which make the observed variables to correlate. This implies that the variables belong to the same content field, - but that is defined by you subjectively. Fortunately, there are diagnostic statistical indicators (such as KMO, Bartlett's sphericity test, pattern of residual correlations) which can aid to decide if to drop or to add variables from/to the analysis.
Welcome to our site, Benjamin!  Thanks for weighing in with your thoughts.
Thank you! Very helpful and did not know.
Do you have additional papers on this? I mean this would be a straight-forward yet relevant research subject. Very interesting idea, though just intuitively I don't agree that it necessarily would be better than a dynamic grid of models.
I'm not sure I understand what you mean. Could you please clarify?
`@observed` has a similar effect as `@potential` in that they both contribute log-probability terms to the posterior. The latter does not have a value associated with it, and is generally used to constrain the model. The former is for modeling data. Check out the user's guide for more info.
Do you really need a formal *test* (e.g., to prove some hypothesis) or do you just want to assess how well the two variables are associated?
(3) "Proposition 2.8 (Daniell)" on page 4 of http://math.iisc.ernet.in/~manju/ProbTheory/Notes/1-3%20Independence.pdf gives a way for constructing i.i.d. Bernoulli random variables defined from [0,1] to {0,1}, by using binary expansion, which are rigorously iid. Is that way used commonly in Monte Carlo simulation for generating independent samples?
Ok, thanks for this answer!
Also, I will mention that G*Power can help you identify the size of the effect using values often outputted by SPSS and other programs. If you say what specific statistical analysis you are doing (e.g., t-test, ANOVA, regression, correlation, chi-square) we can guide you more specifically.
My doubt still remains, if the modulus of Eigen Values of the [I-A] matrix is less than 1 , the overall VAR system is stationary , even though its component series might be non stationary. So Should I go for differencing the series or proceed without it 

Thanks
I think your first equation is the unstd. regression....
Hint: not false means true.
Q1: MASS was IMHO designed to illustrate a book. As I mentioned in a related comment, some of the default choices are not in accordance with the field's state of the art practices. Most implementations are slower than those found in the specialized packages pointed to in the task view. Q2: no. Q3 yes. see also my edit.
Looking at the simplest non-trivial example can help.  For instance, when $\\mathbf{x}$ is a row vector of length 2, say with components $x_1$ and $x_2$, then $\\mathbf{x}^t$ is a column vector of length 2 with components $x_1$ and $x_2$.  The expectations are $E[x_1]$ and $E[x_2]$: these are the components of $E[\\mathbf{x}]$ and $E[\\mathbf{x}^t]$; the only difference is that in the first case they are arranged side-by-side in the row vector and in the second they are arranged top-to-bottom in the column vector.
Actually, I took convolution of two unit pulses in this online calculator.Here the triangle starts from +1/2. That confused me a little bit.  https://maxwell.ict.griffith.edu.au/spl/Excalibar/Jtg/Conv.html
I've voted to close (and migrate to [so]) because this is really a question about R. That said, there are some interesting statistical issues here, not least the fact that you are looking into your data when deciding which variables to include in the model.
I agree there is a true distribution in the Bayesian framework. In the mathematical treatment of Bayesian statistics we consider the parameter as a random variable distributed according to the prior distribution, but the practical interpretation is different: there is a true value of the parameter, and the prior distribution is a distribution describing our beliefs about it.
Could you add the calculated T as formula using latex?
I think it's definitely a curved exponential family, since you can add the sufficient statistics $x, x^2$ (if they're not already there), and then set their corresponding expectation parameters to $0, 1$.
You have conflated the error function with the Gaussian CDF: they are not the same.  Try a numerical calculation--you will see the error.
$\\|v\\| = \\sum_{\\ell=1}^L v_\\ell^2$ for any vector $v=(v_1 \\dotsc v_L)$. Thus
$V(\\hat{\\beta}) = \\|Y - X \\hat{\\beta}\\|$ is the sum of squared residuals, which I have denoted by $\\|\\hat{\\epsilon}\\|$.
[Wikipedia's page on the Hausman test](https://en.wikipedia.org/wiki/Hausman_test) might help you figure that one out. If it's a Wald $\\chi^2$ as in [this example](http://www.stata.com/support/faqs/statistics/two-stage-least-squares/), [this source](http://www.stata.com/statalist/archive/2013-02/msg00345.html) describes it as, "Just a Wald test that the predictors simultaneously $=0$." As for "Durbin," might it be [the Durbin-Watson statistic](https://en.wikipedia.org/wiki/Durbin-Watson)?
I don't know why `mvrnorm` didn't work, note that @user603 [demonstrates](http://stats.stackexchange.com/questions/48534/how-to-generate-two-negatively-correlated-random-series-of-numbers/48535#48535) its use properly. Note also that my answer at the linked thread provides a way of generating correlated numbers exactly along the lines you're going for here (although I agree that you don't need to re-invent the wheel).
Also, by your initial answer do you mean to say that linear models are not an alternative to anova but rather a sub-calss which anova uses - and that lme4/nlme ARE in essence ANOVAS but they simply use a linear model which is superior to aov's? could I get the same kid of output `nlme::lme` gives me from `stats::aov`
Perhaps read [this](http://stats.stackexchange.com/questions/20836/algorithms-for-automatic-model-selection) to give yourself some perspective. If you think there's a way of examining *every* potential model to find the "best" one, you're chasing a will-o'-the-wisp. Why only look at linear relations & interactions? - do you think weight's linearly related to height? There's a lot of prior knowledge you doubtless have & could use to build a few sensible models to fit & test.
I understand the premise of the approach, but I'm not sure how to implement it. How would I arrange data and estimate the influence of candidate drivers of the boolean/ date? I work in R.
"In large enough samples, it's possible for the F's to cross - even several times, and still reject the one-sided test" – note that this means that you can reject the one-sided test in both directions for the same data!
The reason for this is that logistic regression is trying to achieve exactly that: modeling the data distribution, including the prior probabilities ("averages"). Is this behavior undesired?
I guess grandma understand what orthogonal means? Yes, some flaws there but it's a start. I think it's great that there have been so many answers here.
There is always a limit to how accurately you can predict an outcome. There is just only so much information in your data.
Pedantic terminology comment. Dead per square meter scaled to [0,1] is a proportion or fraction, not a percentage. Regardless, software in this area will expect [0,1] input, not [0,100] input. You seem clear on this, but the comment is inserted because I have often encountered minor confusion on the point.
BTW, cross-posting is discouraged for many reasons.  Please decide where you want this question to appear first and delete the other copies.  (IMHO it belongs here: you'll likely get theoretical mathematical answers on the Math site and you're more likely to get practical and nuanced answers here.)
And the second question is, can you assume the infections are independent. I.e. for someone who has pneumonia, are they just as likely to get urinary infection as anyone else, or are they more likely to get it (maybe because they are already sick and it's easier for them to succumb to another infection)?
I didn't average errors, I calculate MSE for each SKU and for both approach , I'm trying to find a intutive explanation why BU works better for high positive autocorrelated data, this is my idea: When autocorrelation is highly positive, successive values of dt are positively correlated and so the process will tend to be smoother than the random series, 
With data are not so noisy, the signal can be detected at the bottom level, and so the bottom-up approach may do so well. Do you think that's make sense?
[This](http://stats.stackexchange.com/questions/44769/understanding-p-value?rq=1) previous discussion might help.
Did you do that in your head whuber?
Usually, for the random effects parts, `random=~1|id` corresponds to a random intercept for each statistical unit `id` while `random=~grp|id` would yield random intercept and slopes (for `id` within each level of `grp`). For *crossed random effects*, things are a little bit more complicated and you'll need to use one of the `pdBlocked` together with `pdIdent` (block diagonal VC matrix with diagonal blocks given by multiples of the identity matrix) or `pdSymm` (id. with PD matrices). I would be happy to try to give more details with a small working example.
@Joel I'm not finding this comment conversation to be very productive. I provided some thoughts on different methods of comparison in my answer: before-and-after (aka delta of deltas) and regression. I am trying to answer the question "What is a reasonable course of action in such a situation?" to the best of my limited abilities, but I'm seeing a trend on this question of you nudging me and the other answer-ers in a different direction than your main question.
You would be better off telling us about these data and where they came from: something has *clamped* a distribution that naturally extends beyond the $[0,1]$ interval.  It's possible you have used some measurement method or statistical procedure that's not quite appropriate for your data.  Trying to patch up such a mistake with sophisticated distribution-fitting techniques, nonlinear re-expressions, binning, etc., would just compound the error, so it would be nice to circumvent the problem altogether.
Nice, I hadn't thought of it that way. I have no idea what kind of model I would use for this kind of problem though.
Pearson's coefficient totally ignores the fact that the dependent variable "Y" varies not just because of one explanatory variable, but because of all. Then it will give you the degree of _covariance_, but not the degree of _effect_ of the explanatory variable on "Y", which is what you are after.
Once means are taken in each cell, you do have 'only a single observation'. I don't know why one would do this, rather than adapt the Friedman test to the situation at hand, but it should be valid as far as the hypothesis test itself goes (one might need to beware the ecological fallacy in applying the conclusions more generally, however it probably doesn't impact your situation).
I am refering to the least squares estimate of the regression coefficients being unique, though I was making the assumption (not originally stated) that the columns of the x matrix are linearly independent, or the x matrix is full rank, or ... (other ways of saying this): http://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)
@Chris:  If your OLS model was $y_i=\\alpha + \\beta x_i + \\varepsilon_i$ and you sought to minimise $\\sum \\varepsilon_i^2$ by finding optimal parameters $\\alpha$ and $\\beta$, then $\\alpha$ would be the constant term
+1. Regarding **2.**, a couple things you could quantify "which is best" with are a) computational speed and b) look at the sensitivity to starting values - restart the optimizer from various starting points to see where it ends up. The algorithm that provides stabler results (across many different starting positions) is better, at least in that sense. This may also give hints about whether the objective function has multiple modes, etc.
I'm not sure what you mean by 'it's used because it's achievable'. That's not the message I take away from the linked page. I don't think anybody is saying that 5% *should* be used for any particular case (are they?), and *certainly* not for the case where one is trying to "disprove a long standing theory". You seem to be arguing against something nobody (to my knowledge) is saying. The convention in HEP is *just* as much a convention as the convention you see in say agricultural research, the distinction is that the stakes are different so of course suitable α (& β) is too. What did I miss?
I suspect they are available for decorative rather than data presentation purposes.  It's worth noting that they are each actually a more straightforward pattern repeated ten times.  So the only situation where it would be useful would be one where you want to represent the remainder once you have lopped off the first significant digit.  This might be necessary sometimes but not very often I would have thought.
Point 1 gains more legitimacy if you also include "existing infrastructure" under that inertia banner. If there are existing business processes which already use SAS, then there is a transition cost with changing. If this is the case, it is not choosing between SAS and R, but choosing between staying with SAS and changing to R, which can have a different conclusion.
@Whuber, yes it is a specialized term. One also refers to "hard" and "soft" attrition. The former typically being a proactive choice by the customer and the latter being less a choice that the company could influence / intervene to correct. For example, some customers are soft attriters because they move, or die or are removed by the company due to non-payment. In some churn modeling using (typically discrete time) survival analysis, competing risks is used to differentiate these causes of attrition.
Ben, you may be correct, but I am hopeful we can provide some insight here at stats: the issue is of statistical interest.  Bob, if you don't get some help on this (say, within a day), then just flag it for migration.  Meanwhile, I am unable to reproduce your error: the output is `Estimated transformation parameters 
  data$V1 
-2.932667`  (R 2.11.1, 64 bit Windows.)
(+1): Very complete answer!  If you're using R, `plot.lm` can give you most of the diagnostic plots Penguin_Knight mentions.
@whuber Yeah, I agree with that. So, I guess I could just say that the Hessian matrix exists for *almost any* sample (excluding those where at least one observation coincides with the MLE, which is an event of probability zero).
The question is ill-posed. If your data matrix is 480 by 502 then saying that the matrix has rank $q < 480$ (the column space of the matrix has dimension $q < 480$) is mathematically equivalent to saying that some column is a linear combination of the others, but you can't pick out one column and say that this is the column that is linearly dependent. So there is no procedure for doing this, and the suggested procedure will pick a quite arbitrary security depending on the order they are included.
Hi, but what I really want is $\\pi_{pred}(\\theta_{n+1} | Observations)$ because $\\theta$ is really my variable of interest and the $X$'s are just noisy observation of $\\theta$'s. (same context as the slide 44)
*SVMs are soft-margin classifiers* ... except when you use hard margin SVM. That said, using SVMs would be like shooting a fly with a cannon.
What is X?  If X is ordinal there might be something to do, but if it is nominal, I don't really see how the question makes sense. What do you mean "directly at the level of X"? The dummy variables *are* at the level of X.
@Ludo - Yes, you can show $p_{11}$ is the solution to a quadratic equation. If there appeared to be some interest, I'd take the time to write that solution here. But there isn't and the answer is already quite long as it is :)
*If* the model is right and the observations are independent, it should be Poisson (In fact, if you condition on the total number of observations, you have a multinomial - which would be a bit smaller than the Poisson's $\\sqrt N$). If you don't have independence or the model is wrong, it will be larger. Your simulation experiment would seem to eliminate the 'model is wrong' explanation in that case, so something you do is probably inducing dependence.
Michael, I have tried to do a major cleanup of the math. Please check that I have not introduced errors. I would urge you to consider looking through some other answers of yours in which several other users have been kind enough to provide this service to you. Like any language, $\\LaTeX$ can seem a little daunting at first, but it's not hard for the computer-adept such as yourself and the end product is orders of magnitude cleaner. :)
Ok that's a shame, but I can understand it.. Thanks anyway!
I really like this forum because of the expertise and quickness of response of the users. Keep doing the good work.
I gave an example where it doesn't work.
The answer to (1) is just $\\tfrac{2250}{150000}$, since it's asking the probability of a win on a single draw. Other than that, what you've said appears to be correct.
Thanks for you contribution here. In the future, please make sure to review our [FAQ](http://stats.stackexchange.com/faq) and pay a particular attention to how [homework questions](http://meta.stats.stackexchange.com/questions/12/how-should-we-deal-with-obvious-homework-questions) should be handled.
Can you go into a bit more detail here? This seems fascinating.
+1 (on both Q and A). ... "*when I try the maximum likelihood approach on my real data it doesn't seem to give reasonable answers*" - if the other approach does give reasonable answers, it might suggest a convergence problem with the maximization routine. What happens if you start it at the other estimate?
I guess you could vary the loss matrix of the model (for {rpart} change loss=...) across a range and build an ROC curve from that. But I haven't seen it done and not sure it would really work
This is a standard topic in meta-analysis. Note that Cohen's d is biased & needs to be adjusted. You need to know the sample sizes (n's) for this. If you have that, the bias-adjusted standardized mean difference has a variance that is a function of the statistic, so you only need the statistic to form (eg) confidence intervals. In addition, there are chi-squared tests in MA to determine if BASMD's vary more than they 'ought to'. Do you know the constituent n's?
Okay, good that's what my answer discusses - testing the null that "within each subject, the proportions are equal" against alternatives that would change the variance from that; it would rely on the independence assumption however, which may be suspect (if these are over time or along a spatial dimension, you'd want to check for serial dependence in particular).
Wolfgang, thanks for your answer!
Not really.  It's a C++ program you have to compile into a binary and run from the command line.  I've written some wrapper functions around it in R to pass data to VW and read back the responses, but it's klunky.  If you have a mac, you can install VW via homebrew.
It's still ordinal. If you like, you can multiply everything by 2, it will then be integer and still ordinal
$y$ is *discrete*. A probability function for $y$ - no matter what it conditions on - will be a pmf. I am baffled by your difficulty with that point.
Let me ask you to clarify and detail your question. First, what is "imbalance"? Groups of very different _n_ or else? So, do you have your sample split into groups or not? Second, PCA (?)`tries to keep the most discrimination power rather than keep most variations` seems to contradict with PCA `trying to keep most variations in data set`. Please, elucidate.
Be aware that this question is awfully broad, & may be closed as too broad. If so, you may prefer to ask a series of more focused questions. You may also want to read through our [materials](http://stats.stackexchange.com/help/asking) on asking questions on CV.
80% of missing data is too much. Imputation is recommended to do with no more than 20% of missing data.
@A.R which package do already what do you want to do ? and if I undesrtand you try to find a (best)method to replace you NA according to probs?