tttjjj/elt_bench huggingface dataset:
Another way to accomplish what you want--and in some cases it might be numerically a little better--is to generate your data using the correlation matrix and post-multiply them by `V`. It should be obvious that this works because (1) separate linear transformations in the variables do not change their correlation but (2) rescaling a unit variance variable by a constant scales its variance by the square of that constant. Then if you look at what `mvtnorm` does behind the scenes to factor `R`, you can see how it effectively carries out the same post-multiplication by `V`.
It would be helpful if you could also say something about your goals for the model & subsequent simulations. Eg, are you hoping to predict future use, or understand the conditions under which customers are more likely to use credit cards vs. cash (etc)? In addition, I'm wondering if the *machine-learning* tag would also be appropriate on this question.
Although it is unclear what you mean by "normalizing" the data (I know of at least four standard ways to do this in PCA and likely there are more), it sounds like the material at http://stats.stackexchange.com/questions/53 might be illuminating.
Thanks for the insight of the usefulness of using different priors.
No one is feeding me these theories -- I may have come to the conclusions myself. I mean, I have a few books on forecasting, but none of them mention creating a model for multiple t+1, t+2, t+3... t+n ... periods out forecasting. The truth of the matter is, when using the test data, there may be greater accuracy (in this case MAPE) with a higher alpha value when you are forecasting one period ahead, and a lower alpha value when you are forecasting 6 periods ahead. Whether or not this has to do with overfitting the model, I'm not sure. But that's what I've found.
If the functional form is completely unknown, then you certainly have a point. It kind of depends on what you know about your data. If you do want to look at fitting curves, you may want to look up so-called *natural splines* or *restricted splines*, which are designed to be pretty flexible while avoiding overfitting.
In terms of estimating the distribution of each data set, yes, something like that is possible. It is likely that subsampling is valid for your problem. It is less likely that the bootstrap (or at least the usual non parametric bootstrap) is valid for your problem. See [Andrews (2000)](http://dido.wss.yale.edu/P/cp/p09b/p0994.pdf) for a similar example.
@Aksakal, I see what you mean, and the normal distribution represents reality, that's why we use it? Suppose we were to impose a bell curve shape distribution to $\\epsilon_{it}$ that had the appropriate bounded domain. Would this be a good solution to the problem? Suppose $Y_{it}$ was a person's age, and we choose a nice bell curve distribution for $\\epsilon_{it}$ that had domain $[0,150]$, this would be better than just using the normal distribution for $\\epsilon_{it}?$
Maybe you should rather test whether the proportion of cars on the road having an accedient is related to the total number?
If observations are independent, you can't do better than predicting next number to be the number that happened most often.
Thank you @Corone so much, I have learned a lot from your explanations!
In terms of variables of interest, I've tried all of them primaries and a number of different combinations of the transformed variables as well as mixed models that include interaction effects. Still, nothing higher than 81.7% overall efficiency.
Thanks.  How do you type Theta on a Mac, any idea?  I know option o comes out to ø??
Thanks. As per the paper the hinton's guide, i plotted histogram of learned weights after 50 epochs. i really couldnot understand what the figure represents.
@RasmusBååth, I'm pretty sure the model in section 6 of that paper is an instance of what you want. Also, it's discussed in Ch. 15 of Gelman's book. See my edits above for more info.
There is an `R` package called [`gPdtest`](http://cran.r-project.org/web/packages/gPdtest/) that contains the function `gpd.fit` that will fit a generalized Pareto distribution to a data set.
I wouldn't put it this way "more problems arise". The less number of entries are present the less constrained is the "task". I we have all correlations missing but one, we are free to compose almost the entire matrix in great many ways.
So bottom line, if I dont have that $P_T(S)>0$, the conditional probability $P_T(R|S)$ is not defined? So its not really a proof its just getting why we need such an assumption right?
The main weakness is that, as the forecast goes further into the future, the confidence intervals get too wide to be useful. Also, because forecasts are performed off historical data, they assume the future is like the past, so structural breaks are an issue. Reforecasting should assist in reducing both those issues, although I have had one time series where I could fit a line, but I couldn't predict off it, and it wasn't a random walk.
Do you mean the observation number? These are in time order or something?
Correlations at lags 6, 12, 18, 24, 30:
0.127, -0.103, -0.005, -0.017, & 0.015.

Correlations at lags 7, 14, 21, 28:
0.17, 0.105,  0.133, 0.183.

Which are bigger?
http://sekhon.berkeley.edu/elections/election2000/butterfly.review.pdf - Wouldn't consider it "applied" research, but may be of interest.
What kind of "dependence" are you looking for specifically?  After all, we could take a series $(x_1,\\ldots,x_n)$ that satisfies your "differential spectrum test" and set $x_{n+j}=x_j+x_n-x_1$, $j=1,\\ldots,n$.  This series of $2n$ values would satisfy the test--the histogram is almost the same, with a slightly higher peak at $0$--but contains a very strong dependence indeed (half the values are completely determined by the other half)!  Note, too, that your test is *not* a test of independence: it is merely a test of symmetry of first differences, which is much weaker than independence.
+1. Because this is a self-study question, you should add the `self-study` tag to your question.
(+1) This explanation is a good one, but it is unnecessary to attribute the phenomenon to multicollinearity: the key distinction is between *jointly informative* and *individually informative.*  Including additional *uncorrelated* regressors (which avoids any multicollinearity) lowers the former while leaving the latter unchanged.
+1 for lattice.
The formula indicates $\\int F_X(x) dx = \\mu_X = \\int x f_X(x) dx.$  By definition, $f_X(x) = \\frac{d}{dx} F_X(x)$, allowing us to write $\\mu_X = \\int x d(F_X(x))$.  Doesn't that suggest integration by parts?
I was taught to use arithmetic means for levels, geometric for ratios, and harmonic for rates. My data seems more like ratios or maybe rates (since they are each unit's deviation from from forecast in % during a given period of time). All are pretty close in this case, so it is a bit of an academic matter on which Pythagorean mean to use.
A nearly identical question concerning $S/n$ (or $S/(n-1)$ as some prefer), which is the *covariance matrix*, is addressed at http://stats.stackexchange.com/questions/18058/how-would-you-explain-covariance-to-someone-who-understands-only-the-mean/18200#18200. If that doesn't suit you, then please search our site for more information on [covariance matrices](http://stats.stackexchange.com/search?q=covariance+matrix). For direct references to scatter matrices, search on [that term](http://stats.stackexchange.com/search?q=%22scatter+matrix%22).
Note that the assumption, when it's made, is of normally distributed *errors*; the dependent variable certainly doesn't need to be normally distributed.
You don't select the regression method based on the sample size, you select the regression method based on the dependent variable and your research questions.
+1 on Nocedal & Wright!
It would also help to tell us more about what you are interested in. For linear predictor point estimates and prediction of values, underdispersion rarely is a problem but tests and intervals may be unnecessarily conservative (quasi families would help with that). That said, for a "normal" likelihood approach check out the COM Poisson and other generalized Poisson models.
In light of the comments to the question, which point out (among other things) the dependence of the variables and imply strongly asymmetric distributions for them, this approach seems unsuitable.
But, what's the idea behind "you can sample without the normalizing constant and get the same sample."  Clearly, they are two different densities, but still produce the same sample X1, X2,...,Xn?
ROC or AUC ($c$-index) are not efficient ways to compare two models.  This is like comparing two Wilcoxon tests instead of doing the correct head-to-head comparison.  Proportional "classified" correctly is far worse.
Presumably $X_n$ and $x_n$ are sequences of random variables and you mean the *non* convergence of the Harmonic series.  But it needn't be that complicated. For instance, consider a discrete distribution $F_n$ with $\\Pr_{F_n}(0)=1-1/n$ and $\\Pr_{F_n}(n^2)=1/n$. If $X_n \\sim F_n$ are iid, does $X_n$ converge to $0$ in the sense you intend? What happens to their expectations? Now this does not answer your question, because $E(X_n)$ are all finite, but it might suggest some simple modifications that do work.
Welcome to the site, @Ivan Kshnyasev. This answer has been automatically flagged as low quality. Can you add a little more information to make it more helpful for future readers? There is a thread on answering CV questions [here](http://meta.stats.stackexchange.com/questions/1390/how-should-questions-be-answered-on-cross-validated); you may also want to read our [about page](http://stats.stackexchange.com/about) (which contains information for new users) since you are new here.
I think for the question about taking a random sample from my sample. I need to check "what's the sample size to detect a rare event".
In matlab run  `nftool`   this will provide a GUI that and instructions to create your model
The details on the particular table in question made a huge difference.
Why do you need to test for cointegration on the returns? Nearly ALL stocks are cointegrated if you transform them into a series of daily returns first! That's because the returns, from day-to-day, are always going to be about -3% to 3%, and their average is usually 0 (assuming they are not the most volatile). That alone transforms them into a stationary series, and I believe cointegration requires them to be non-stationary, but their residuals are stationary.
There's an interesting paper about to go to press which addresses this, see http://warnercnr.colostate.edu/~hooten/papers/pdf/Hanks_Hooten_JASA_2012.pdf
Thank you C.p, you make an interesting point.  Although, from the perspective of Henrik's answer above, you are still in the realm of inductive, since the statistical reasoning you describe is one that involves uncertainty.
After I posted this, I found that in order to consider Z following a normal distribution the length of the time series should be over 10
A dummy variable takes values of either 0 or 1. How might it exhibit a non-linear relation to the response?
@Aksakal Yes! That's correct. Knowing the ideal to attain, though, provides useful guidance concerning how to choose the shape of the enclosing region $V$. The variance formula allows one to contemplate what can be gained in accuracy (or reduced computational effort) in return for the additional complexity of using more complicated regions $V$. For instance, I could have decreased the $.077$ variance to $.004$ by choosing the linear function $7x+2y-14$ for the top of $V$ rather than a constant function--but then I would need to find an efficient way to sample uniformly from this $V$.
I tried this route, but the assumption that $\\widehat{P}_{ij}$ are independent is not always justified (I can not prove any estimates for them), and I even did a randomized check in MATLAB (i.e., estimating the variance over a series of T runs as well), then the difference between the left side and the right side can be quite large. I am just wandering, if there is a way to get an exact formula without any additional assumptions (i.e., valid even for $n=2, m=2$, ok, I am changing the setting :) but I think the message is clear.
If the outcome variable is ordinal but not interval, the magnitude of the $R^2$ from the linear model is meaningless. If you change the labels in some non-linear (but still monotone) way, you can make the $R^2$ as large or as small as you want while still keeping the ordering in tact.
This may be better suited to statistics.SE.
Maybe I don't understand. Suppose that the random vectors $X$ and $Y$ have mean vectors $u=\\mathrm{E}[X]$ and $v=\\mathrm{E}[Y]$, respectively. The matrix $A=u\\,v^T$ can't be, in general, a covariance matrix, because, suppose that $X$ and $Y$ are such that the inner product $\\langle u,v\\rangle<0$. Then, we have $\\langle u,Au\\rangle=u^TAu=u^Tu\\,v^Tu=||u||^2 \\langle v,u\\rangle<0$, and $A$ is not positive definite.
The references were helpful, thanks. The first reference is good for the idea of multiple states but not so much for dealing with the specific problem I have of only certain states updating. For the second reference and your alternative suggestion, I am hoping for a filter that I can code myself and I need to be able to explain the steps involved as well as get the results, but they are interesting alternative strategies. Also, note that I have modified my original problem with a slight but acceptable simplification after much thought, it may make the problem easier to deal with.
I added the spatial tag given the example is explicitly spatial. If you (or anyone else) feels it is unneeded feel free to roll back that addition.
@COOLSerdash Thanks!
This looks like the discretization of an affine diffusion (SDE) or something close.
Please copy the rest of the question here.
Adding to whuber's comment, for non-linear models sometimes the "optimal" design relies on the true value of the model, which you're trying to estimate.  You can use a Bayesian approach to find the best mean design given some prior belief, which can sometimes smear the design points out.  So whereas first order linear models occupy one extreme (all observations at the extremes) there are non-linear models and optimality criteria that can occupy the other extreme (very spread out observations).
+1, awesome. How did you come up with functional form for $g$?
Note that $\\log(kx) = \\log(k)+\\log(x)$. That combined with what you've already learned about the linear-linear case should be sufficient to work it out.
Can the parameter n in binomial be real?
@didi yes, that should work. The coefficients for the fixed effect of species represent the difference between species a and b, and species a and c (treatment contrasts). If you want the expected value of each species you can use the predict function. If you want post-hoc tests see [this link](http://stats.stackexchange.com/questions/14078/post-hoc-test-after-anova-with-repeated-measures-using-r)
This is a good question, but does seem like it may be a better fit on the [math SE site](http://math.stackexchange.com/), are you only interested in the mathematical aspects of this, or are you also wondering about the substantive statistical meaning of changing $W$ (eg)?
What is the reason for a non-parametric approach?
Thanks for that. (I had planned to track the packages down next and edit mentions in, because I had seen some packages in the past that did so but didn't recall their names. You saved me some time there.)
I don't know what is intended by the phrase 'relative p-value' there.
Do you mean very *dependent* at the end of your 2nd paragraph. If so, have you thought of some type of Markov Chain approach once the first variable is estimated?
The most answers and packages are only for traditional conjoint analysis. Are there any packages or methods to do a choice based conjoint? (Example: One choice out of 10 products in round about 5000 purchases) @Faisal Afzal Siddiqui: Choice based is not possible with your method, only traditional ranking or rating data. Correct?
@ziggystar: in Yedidia's paper, at the end of part III, he said that at the fixed point, the believe are in feasible set. So I think we should not focus on the intermediate result, and the convergent value is the final choice. Do you have any comment?
@Arun One reason: The Marsaglia's polar method is useful when you only have a RNG that generates uniform deviates.
Your PS is ambiguous. Do you mean that you *do* have sd's and covariances of the estimates, or that you don't?  Some more information is required. What are the parameters, and how are they estimated? (It may be possible to treat these estimates as asymptotically multivariate normal.)
+1 and thanks for a very nice link.
This looks like a floating point limitation in density(): in using a bandwidth of 1e-6, you are creating (in theory) a collection of 10,000 spikes, each of total mass 1/10000.  Those spikes end up being represented primarily by their peaks, without the gaps being adequately characterized.  You're merely pushing density() beyond its limits.
From the help, it doesn't look like it, but you could do it with `rpois`.
Peter, I don't have individual data to take the CV of log income, but it would be interesting to see what that does! Your GINI coefficient idea makes a lot of sense; I'll check whether the percentile income data are available. Thanks a lot for your suggestion.
@nic: If you know that they are due to an intervention, then add an indicator variable for the intervention. You'd have to decide what kind of impact the intervention would have: is it 1 in the year of the intervention and 0 otherwise, or is it 1 in the year of the intervention, 0.5 in the following year, and 0 otherwise, or ... The bottom line is that you have determined that the data is in fact legitimate and explainable, so add that explanation to your model.
@aix I have edited my answer to add some benchmarks on the current implementations in scikit-learn. Also checkout the [java version of liblinear](http://www.bwaldvogel.de/liblinear-java/) before implementing your own coordinate descent as it might be good enough for you (though you cannot have both L1 and L2 reg at the same time).
A test of paired 0/1 data (each unit fails or it doesn't) would seem to be taking us toward McNemar's test. If there are groups or other predictors/explanatory variables it may be more complex.
Also since we know matrix B( genes) is biologically related to matrix A( variables showing metabolism) i took my matrix A (factors)as response (which is already different for different groups) and matrix B( factors ) as explanatory variable to how they are related .
What do you mean by 'accuracy', and how is what you want different from the error you are already calculating?
@MichaelLew:  I completed the course in which I studied this material 2 years ago, I only needed my memory refreshed.
This is great! So clear and addressed everything I was looking for. Thanks so much!
You would be interested in a [related thread](http://stats.stackexchange.com/questions/2932/metric-spaces-and-the-support-of-a-random-variable) about this question.
Yes, I used the first formula. This means that averaging the F-score from the different tests yields similar results to averaging precision and recall and then calculating the F-score from them. I tried this on the results I have and it was almost the same. Thanks.
I think the variance of the fitted values is is: var(demandts.train.arima.fit) which is also different
OK, so if I have an ANOVA defined as `aov(value~Condition*Scenario+Error(Subject/Scenario))`, how do I define it using `lme`?
No simulation is needed because simple exact solutions are easy to obtain. The Poisson parameter for which $x$ is the chance of observing at least one $0$ out of $k$ independent cases is given by $-\\log(1-(1-x)^{1/k})$. In `R`, with $k=5$, this can be calculated and plotted as `curve(-log(1 - (1-x)^(1/5)), 0, .1)`. For instance, your result of $6.3$ (which should be $6.210\\ldots$) can be read off this curve at the point $x=0.01$.
http://stats.stackexchange.com/questions/31141/asynchronous-irregular-time-series-analysis
Several questions appear to already answer this question, [What if residuals are normally distributed, but y is not?](http://stats.stackexchange.com/q/12262/1036) and [Left skewed vs. symmetric distribution observed](http://stats.stackexchange.com/q/11351/1036). The only thing I would note is though you should look at the distribution (via a QQ-plot or a histogram) and not solely rely on estimates of skewness.
If it's the case that recent predictor values are more relevant to the response you could consider time series methods for data reduction, e.g. using level & trend from Holt exponential smoothing within each family as predictors. I think this question might benefit from a bit more background.
I don't understand this. Surely you could test for the distribution of missing values relative to assorted covariates or outcomes the same way you could test for any outcome. If everything is missing for a given subject, you could look to see if the distribution of what remains appears truncated in some unexpected way -- granted that requires some modelling assumptions.
Yes I think you should be concerned about this. Why are you using an estimated coefficient for another variable on a new variable?
As my answer states, a standardized effect size should be based on the values you analyzed. If the statistically tested values are transformed values then that's what you use for Cohen's D.
I believe similar questions have been asked in the past, see [How to compare Likert scales with varying number of categories across time?](http://stats.stackexchange.com/q/6791/1036).
This phrase is understandable as soon as you apply definitions of conditional distributions and quantiles: would you like help with one of these in particular? Regardless, there doesn't seem to be anything to "derive," which makes your last question somewhat puzzling.  Are you perhaps referring to some result you haven't stated?
Your rug at the bottom gives some rough intuition. Imagine each value $x_i$ from $i = 1$ to $n$ is a spike with an associated weight $1/n$. Now smear each spike using the shape and width of your kernel, so that the spike is transformed to take on the same shape and width, with a height such that the area below is $1/n$. Add the results and you have a kernel density estimate.
Welcome to the list. If you are asking about `R` commands then this question is off-topic here. But it seems you are asking primarily about what a good plot would look like and secondarily about how to create it. If so, I suggest deleting "with R" from your title and maybe stating, in the body, that you have `R` available.
I think the reason why normality is a good assumption is because of its relative lack of use of the data - only the first two moments are used in estimation with the normal distribution.  This makes diagnostic checking of a least squares model very easy - basically you just look for outliers which could influence the sufficient statistics.
@ Everyone: Thank you very much. In the end I used many different thing in the hope they would all give similar answers. And they did. I used the BMA, bestglm and glmnet packages as well as the step function. Fitted models with all of them, and there was not discrepancy in what BMA with maxcol = 9 and step deemed the best model. All the experts in the field around me seemed very content with the variables, and felt that it was quite progressive. So thanks for all the input. I really used it all.
Thanks for the insightful answer. I will try with SD of accuracy for further analysis.
Huh? x is different how? Tell us the story of the data, and how the story is different.
@Thomas Jensen, something like this. When you have data up until period $t$, you can calculate the forecast of $y_{t+1}$. You use this forecast when forecasting $y_{t+2}$ and so on. To implement this some kind of loop construct will be necessary.
It shows (for a rather special case with n = 1) a way to construct a confidence interval for p which does not have any requirements (unlike 8.10).