tttjjj/elt_bench huggingface dataset:
Questions about particular software implementations would normally be off-topic here, as you have already been told in http://stats.stackexchange.com/questions/86330/offering-data-analytics-as-a-tool-combining-r-and-python !
In discriminant analysis, for example, you can set prior probabilities of belonging to classes as you need. In logistic regression such probability is often called a cut-off value or threshold.
In short, yes. If you have just weekly data and not daily then you can characterise each week by its midpoint. The analysis doesn't have to be sinusoidal, but if you smooth or model then it's important that predictions for beginning and end of year should be identical. That's guaranteed by sinusoids and has to be arranged otherwise.
This seems like a bit more information than I would recommend providing to someone who is looking for help with a homework question.
If all $x_i$ are distinct both $X$ and $X^\\prime X$ are non-singular. In your case it means you can have the same $x$? Have you considered http://en.wikipedia.org/wiki/Scheff%C3%A9%27s_method?
Each respondent went through a conjoint study and regression models were run on each respondent.  The way I am running the segments is by averaging the coefficients of the people within each group.  I would be using SPSS for the analysis.
You know that $\\hat{p}=X/n$ where $X\\sim Bin(n,p)$. So you know the distribution of $X$. What does that tell you about the distribution of $X/n$?
**Xorshift RNGs**  
George Marsaglia  
2003-07-04  
_Journal of Statistical Software_  
Vol. 8, Issue 14, Jul 2003 Abstract: Description of a class of simple, extremely fast random number generators (RNGs) with periods 2k - 1 for k = 32, 64, 96, 128, 160, 192. These RNGs seem to pass tests of randomness very well.  
[PDF](http://www.jstatsoft.org/v08/i14/paper)  
  
There's more to be found here:  
**An experimental exploration of Marsaglia’s xorshift generators, scrambled**  
Sebastiano Vigna  
[PDF](http://vigna.di.unimi.it/ftp/papers/xorshift.pdf)
Thanks Peter, I mainly use R, and the level of technicality shouldn't be too much of a problem (I've had 2 theory courses, but we never touched on these types of things, nor in my applied classes).  I will check out your suggestions.. thanks again.
Can you cite your source for these terms & their definitions?
The intercept is often called bias in neural network texts.
@user995434 Procrastinator explained the concept of resampling but did not connect it to the inference techniques that are called resampling methods namely the bootstrap, subsampling, and permutation methods.  Also with the bootstrap M is usually set equal to N.  The only exceptions would be to use a smaller value for m because the ordinary bootstrap is inconsistent.
There are lots of questions on this topic on CV. Please search our site & read through the [existing threads](http://stats.stackexchange.com/search?q=interaction+interpretation). If you still have a question after having read through them, come back here & edit your question to state what you have learned & what you still don't understand. Then people can craft answers that won't simply be the same as what already exists elsewhere.
(+1) It's merely a fork of Mx but it is quite good as free software for SEM. Another promising project is [lavaan](http://lavaan.ugent.be/) (which tries to provide Mplus compliant output, whenever possible).
Any recommended link, book or source on this?
I would recommend using (Bayesian) regularisation and not bothering with feature selection, for similar reasons to those given in my answer to a related question for SVMs http://stats.stackexchange.com/questions/63081/feature-selection-before-svm
It is not quite correct to say "you cannot change z". In fact, you have to change z to make the sum equal 10. But you have no choice (no freedom) about what it changes to. You can change any two values, but not the third.
And what do you do when there is a 3rd point?  That formula is fine if all you have is 2 points, but in that case it is difficult to do meaningful inference, you have no measure of the variation.  If you have more than 2 data points and you only use the 1st 2 to calculate the slope then you are at the mercy of the ordering in the data set (and all formulas talked about so far don't work).  If you don't already know a formula for the slope then you really need to spend some time with a good regression textbook/class, answeres here will not be enough.
There aren't too many unknowns. If you ignore the counts 5-8 you have a simple standard problem. You have more information than this.
+1 for discussing the the real question: how to proper model the relationship: is it linear? angular? etc. Theory might help here.
Replacing $\\sigma^2$ by the sample variance is a good approximation if n is large. The convergence rate [relies](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC219996/pdf/pnas00165-0006.pdf) on additional assumptions over moments of higher order. You can also obtain a nonparametric bootstrap interval for the mean by resampling with replacement and obtaining the corresponding means and compare it with the CLT approximation.
(+1) for the elaboration, but (-1) for the repetition counts of the CV. It is true, that the risk of creating **exact** duplicats (looking at the ids of the observations) is small (given enough data etc.), but the risk of creating **pattern/ data structure** duplicates is very high. I would not repeat a CV more than 10 times, no matter what k is ... just to avoid underestimation of the variance.
But using the mean implies you're predicting the value at that point.  That's not what's going on, what's going on is an attempt to recover a random value.  It seems that since you have an estimate of the variance as well that you should use both (i.e., a random draw from the distribution).
+1 I appreciate your efforts to answer my question. DLM does not seem to be the method I am looking for but it may be useful in one of my other projects. Thanks anyhow!
What is weight in your case? Is it frequency weight (the weight value shows how much times the row of data is counted)? SPSS command WEIGHT is this frequency weighting.
Can you add that graph into your question?
Chronbach's alpha is not part of regression analysis.
In general "weighting" is what you're looking for... but in the example you give, weighting may be insufficient to the task. Weighting corrects for over- and under-representation, but it won't correct for relevant categories of people being totally excluded, for example because they don't own a smartphone.
To elaborate: If you do not normalize / standardise your features - how would you compare them? Especially if one measures apples and the other one oranges.
Since you seem to know what "$R_0$" refers to, would you mind enlightening us?
I give a citation and an example of the runs test for multiple groups [here](http://stats.stackexchange.com/a/73170/1036). The other answer to that question may be of interest to the OP here as well.
@NickStauner, these other questions are not quite identical, especially for an OP who is relatively new to stats. They aren't about **rm** ANOVA, or don't have quite the same mix of issues or don't have answers that address the OP's mix of issues. (The last one is more on point, though.) I think there may be room for an answer here that addresses rmANOVA as a special case of mixed-effects models, & the relations amongst its assumptions--particularly the nature of sphericity & homogeneity of variance.
Thanks so much gui11aume
(1) is wikipedia the source of your information? you might want to find another resource (2) the wikipedia article seems to define bias/variance quite specifically, which you've chosen to ignore in 1 & 2 above.
@xan, this is a fractional factorial design; ie the DoE the OP ran is that there were only 2 observations per combination of factor levels. Thus all residuals must be symmetric about their mean.
You might not like it. More than once my referee's reports have ended up significantly longer than the original paper, as I go through in detail what's wrong and how to fix it (most times only to have the response be to cut out entire sections of paper rather than try to fix the most problematic, but usually most interesting, parts, sadly). One of my students had a similarly detailed report on her work from one of her referees; it was actually quite valuable and led to a much better final product.
I would, also, suggest considering pre-processing the data distribution before applying any classification model. You may apply random undersampling or oversampling or synthetic approaches such as SMOTE. One-class classification is still an option that can challenge previously suggested solutions.
-1 This is barely an answer. There is no reason given for the recommendation, and google docs is basically exactly the same as any other document editing software.
This is equivalent to my problem, but it is easier to pose the problem in terms of the left tail (smaller quantiles).
Is this homework? You'll get more appropriate answers for such simple questions if you tag it as such. You're also unlikely to get a good answer until you put something in about what you've tried in order to solve the problem.
Thanks very much @RayKoopman. [This](http://www.kyb.mpg.de/fileadmin/user_upload/files/publications/attachments/macke2009_5157%5b0%5d.pdf)
 paper details a similar approach that I also found useful. It also has [Matlab code](http://www.mathworks.com/matlabcentral/fileexchange/20591-sampling-from-multivariate-correlated-binary-and-poisson-random-variables) that I haven't tried but appears useful. Appreciate your help.
To meaningfully add them you must effectively make them interval-scaled, so must draw some kind of equivalence both within and across variables -- '*that* much juice, *this* often, is the same as one portion of fruit per day'. How you decide what those various equivalences are isn't really a stats problem.
I suppose NMF=non-negative matrix factorisation and LDA=latent Dirichlet allocation? It is usually a good idea to explain what abbreviations mean.
I never want to do a PhD to impress any employer. I just want it for mainly and mostly knowledge
Transformations often help: that's my bottom line. A statistical person who hasn't learned that many things look clearer on logarithmic scale (especially) is missing out seriously on the one of the oldest and most effective tricks there is. You seemed to be denying that; I hope I misread you.
A few questions . Does it clearly identify time trend changes and report the points in time where the trend changes ? Does it distinguish between parameter changes and error variance changes and report on this ? Does it detect and report on specific lead and lag effects around user specified predictors ? Can one specify the minimum number of values in a group before a level shift/local time trend is declared? Does it distinguish between the need for power transforms versus deterministic points in time where the error variance changes ? ARMAX models speak to all of these considerations.
Perhaps we were talking cross purposes. If you're suggesting the OP could gather this useful information I agree. I was merely saying that, from reading what he/she said, it looks like they haven't done that yet.
Package nlme also provides function `lme` for linear mixed-effects models. I would prefer `lme4::lmer` for linear mixed-effects models and use `nlme::lme` only if you need to specify an autocorrelation structure (which is usually necessary for the data I'm working with).
@user2510: You may include in the state whatever you please, but you cannot (within the framework of the standard Kalman filter) include non-linear functions of the state in the observation equation; and $A(t)X(t)$ would be non-linear if both $A(t)$ and $X(t)$ are components of the state.
@davips: decrease size according to the size/minority class may be described as leave-$n$-out strategy, which is an intuitive minor variation of $k$-fold that is perfectly sensible in this context. If you need further literature, don't hesitate to ask. And thanks for the flowers.
Yes, what you have is what I meant for #3. I will edit that. I shouldn't have said just variances, but been more general in my question and used the word dispersion or spread instead.
Ah, but if you're in that situation, you have far more information than just the sample! It should be possible, for example, to actually compute the density function (at least numerically).
On a side note, as I said (above) to John, I might even report both models and discuss them. So I will state that these few questionable variables which changed between significant and non-significant from model to model were marginally significant and need further evaluation with better test powers, but the variables significant in both models are definitely significant.
I am not trying to compute a reduced-rank approximation of X, rather a transformed X. You see my goal is not to filter noisy sequences, but to find a representation with a reduced dimensionality (to be used for classification/clustering of time series)... Could you elaborate a bit on the EM-PCA approach?
@Fernando Instead of playing with the input parameters to try to reduce the forecast why not look at the fitted model to get an idea of why it is trending up.  Does an uptrend show in the historical data?  Did the model include a first or second difference operation?
+1. Excellent point, and embarrassed recognition from (one of) those who had forgotten this because they never use it in their statistical work. I'd still wager that most occurrences of "Ln" in statistical contexts are typos for "ln".
When you compare any two groups you may prefer either way: use variances only of those two groups or use the pooled variance of all the groups or subset of the groups. The choice depends on your task and reasoning and is often not obvious.
(+1) Your proposal is reasonable and well-founded.  (In fact, I suspect the results would closely emulate the procedure I describe, provided equal weights are used.)  I wonder about one thing, though: we could interpret the OP's desire for weighting to be an expression of a more complex model in which the parameters are expected slowly to change over time.  Weighting is an *ad hoc* but simple and potentially effective way to cope with that.  Without overly complicating the situation or the solution, how could a slow temporal shift be accommodated within a Bayesian framework?
Well, I still some more explanation.
Glad it was helpful. Good luck.
By the way, in case it wasn't obvious, instead of using the ratings of the players at the time, or at the end, it's best to update the ratings to approximate the best fit to all data. If there are no undefeated players and no players who have never won, you can do this iteratively with a few passes through the data, although it is more accurate to do some sort of Bayesian update from some prior distribution.
Please, provide a reference (URL or author's name) for the slide.
That is absolutely awesome. I think I might even be tempted to convert.
BTW, can the tests you are suggesting be done in SPSS?
The Heckman procedure has not been extended to any models besides OLS, see http://dx.doi.org/10.1007/s10940-007-9024-4. While I agree with your comment, given the elementary nature of the question it would probably be better to further elaborate on your statement. Also the OP gives no evidence they have a suitable instrument to estimate the selection bias, which would make the Heckman procedure a moot point (although self selection into treatment is still relevant).
Are you asking whether you can perform an *F* test of two nested linear models, one of which estimates predictive parameters for your two variables, and the other of which restricts those parameters to zero (effectively excluding them) in SPSS? If so, you certainly can, and don't need to program such a test from scratch...
The mean of the posterior sample, the median ... For a discussion of Bayesian estimation [see](http://en.wikipedia.org/wiki/Bayes_estimator).
Long tails and few data points simply do not mix well. As the data points are going to be sparse wrt the functional form, there is no way to distinguish between different functional forms...
My apoligies, this is the [paper](http://www.ntuzov.com/Nik_Site/Niks_files/Research/papers/stat_arb/EG_1987.pdf). I am unable to cite the passage on which I concluded that cointegration is not possible to be tested with a I(1) and I(0) variable, so perhaps I'm incorrect. I hope some one can enlighten me on this process.
@cardinal Thanks. Actually I went through all the questions on Math.SE, but I still could not find some close to this problem.
@AdamO I have looked at some of these, but it seems that social and biological science versions of PCA do not go into depth on the math, just how to run and interpret the results.
Thank you for the offer, but the proof for the hat of beta was clear to me.  Thank you for the answer of the matrix equivalent of the var(AZ), and that was very helpful.
How does any such correlation test for monotonicity - unless it  it's 1 (or -1), the relationship between sample quantities isn't monotonic. How does a p-value relating to to the usual Spearman test identify an underlying monotonic relationship as opposed to something else (perhaps one that is mostly increasing but isn't monotonic)?
Did you do the PCA on the correlation or the covariance matrix?  The higher subscripted y's would have lower variance and this could influence your results differently between the 2 methods.
..automatic-labeled data, and tested over manually-labeled data. I would bootstrap models from sampled training set data and test them over samples testing set data, calculate mean and stdev of the test error. Now , would student t-test be ok for checking statistical significance? And, should I limit the amount of features to check to a certain amount or try for different amount of features?
+1 for this paper ! my reference for the link between thresholding and penalized estimation was http://www.google.fr/url?sa=t&source=web&cd=1&ved=0CB0QFjAA&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.8.6694%26rep%3Drep1%26type%3Dpdf&ei=X6xjTILlHsiU4gashYThCg&usg=AFQjCNE2kNFinNcuvG4qkBBPQAi2omY-9A&sig2=D0NgkuN_KfgV8ZdzADKcxg
Take a look: http://quant.stackexchange.com/questions/1027/how-are-correlation-and-cointegration-related/1460#1460
@Benjamin:  A test statistic is not a direct measure of effect size as it is influenced by sample size.  Regardless, given that Kay was asking about Fisher's exact, I think the follow up question should probably be asked separately (although the answer for goodness of fit is similar to that for chi-square of association/independence).  The short answer to it is ... there are various answers.  Almost every measure of effect size you might be interested in has some sort of restricted range that is related to the marginal probabilities.
At which sizes do the numerical errors in estimating *t* outweigh the gains from using it?
@NickStauner,  I blame the teacher for not caring enough about stats; if they cared there would be at least one question on each exam paper that needed some understanding of stats, at the level of “How to Lie with Statistics”.   I don’t care if social science students know how to do the calc, but they should know how not to be mislead.
If the groups were approximately normally distributed, your chance of observing a smaller difference of means within the permutation data would be approximately $10^{-171}$.  Even with highly skewed groups the chance is going to be pretty small!
@gmacfarlane I have checked all the values and none of them is `NA`. Also I have added some extra information regarding the function in my post above.
I get that you have no autocorrelation when the data is just randomly distributed. But as an indicator for autocorrelation, is it enough when the data is not randomly distributed or do you a kind of pattern (e.g. a data point with a high value is followed by multiple data points with a high value)?
Would a standard mean of the BMI's for each patient not work?
Note: I googled "contourplot gam"; the first result was the help file for `plot.gam` in package `mgcv`.  I ran `library(mgcv);example(plot.gam)` within R and saw the figure above, at which point the question made sense.
Hmmm... the minus sign in the expression you refer to is a mistake - it should be a plus sign. You could of course then rearrange the terms to obtain again a minus sign... does this hurt the intuition you gained?
You haven't done a significance test (no rejection rule, no decision, no conclusion), but simply explained to the reader how *they* might do one as soon as they choose a significance level.
Edited my post to attempt to take into account the fact that it may depend on the method of calculating the CI.
I like option (2) and believe it will be quite stable for $|\\epsilon|\\lt 1$--just look at how the terms in the power series decay.  The issue is working out the correct probability distribution.  For really small $\\epsilon$ you can take it to be uniform (on the Lie algebra) but for larger values curvature will begin to matter, and then you have to work out the Jacobian of the exponential map, as done on [MO for skew-symmetric matrices](http://mathoverflow.net/questions/13120/lower-bound-for-jacobian-of-matrix-exponential-map-near-origin).
I am not well versed in statsmodels, could you write down mathematical definition of your model? I.e what ARIMA model are you fitting and what are your exogenous variables? Also what are the fitted model coefficients?
Could you link to the spreadsheet Mike Lawrence provided?
Irrespective of the order the no of labels of a particular kind are different with probability on/off. So it cant be an order problem (alone).
The correct test for normality in this case would be complicated, unnecessary, and unduly influenced by the lack of precision in the data for sample A. The distribution of the residuals (on a log scale) should give you comfort that the two-sample t-test (based on logs) is giving useful and reliable information.
@Ricardo: I'd just make a graph showing all four combinations, showing every individual data point as a dot. Then the readers can see what actually happened.
+1 for the pointers. Of note, backward elimination procedure for RF was proposed in Díaz-Uriarte and Alvarez de Andrés, [Gene selection and classification of microarray data using random forest](http://www.biomedcentral.com/1471-2105/7/3/abstract), BMC Bioinformatics 2006, 7:3, but see the [varSelRF](http://cran.r-project.org/web/packages/varSelRF/index.html) R package. It is far less computer-intensive than RFE-SVM.
Could you please show the output of the command above in the question?
Yes it seems at this point they are quite hard to disentangle, especially given history (e.g., as you mention) and social perception.
I did forget to mention the other option, which is to approach it ass backwards.  Convert your continuous variables into less than full rank b-spline basis (that means encode the intercept; also means the resulting basis functions will sum to one) and perform Multiple Correspondence Analysis.  That would even handle non-linearity.
You mean "discriminant analysis" aka "discriminant function analysis", don't you? However, this analysis won't tell you "which among these 5 variables has been loaded more for which cluster". Rather, it will tell you which among these 5 variables has been loaded more for which _direction_ that pull apart clusters
I don't think you're wording your question correctly. You're not looking for the mean and SD of D=0 and D=1, which is trivial as pointed out. You're looking for the predicted value and confidence interval of D=0 and D=1 after controlling for other (fixed) effects. It's been quite a few years since I've used stata, but there should be `predict` method in there.
+1 Nice summary and collection of links.
@AdamSA Good point. Moved it. :)