tttjjj/elt_bench huggingface dataset:
Good questions. The second one is easier: *no*! LDA is maximizing separation in the sense of the ratio of between-class and within-class variances (one can see it as a signal-to-noise ratio). In case of two classes it reduces to the distance between the means *divided by the within-class standard deviation*. Now to the first question: I don't think one can say that $\\lambda=0.5$ is always a reasonable compromise. If one is using rLDA for classification, $\\lambda$ is usually optimized with cross-validation. For your purposes it might be an overkill though (or maybe not!).
"Let $X$ be the number of occurrences of an event over a ***unit*** *of time*..."
Thx! Agree, it's a problem to capture the effect of env(X) on Y – but since rules limit the extent to which X can be reflected in Y, it’s maybe less a problem. However, lit does not only control for the mean effect of X on Y (and then take e), but also for the effect of env. (X2) again in the 2nd stage: M(Varofe)=b0+b1POST+b2X2+e. My question was more about the residual – should I estimate the residual in 1st stage jointly over pre- and post-period or separately for each group and then use the variances as M? What difference does it make if I suppress the intercept in the 2nd approach?
I might instead say '...90% chance of being worse off than if it had been funded'
I think it may be a good idea to migrate it to Math.SE...
Are their proposed solutions?
After you have chosen the first fuse and it is good, is the probability of getting a good fuse on your second pick still $0.8$? or should it be something like $\\frac{79}{99}$?
@John Doucette Thank you. These precautions make sense to me, but strictly speaking, they are not statistical precautions but design ones, which leads me to think that assuming there is no known bias for the subpopulation, selecting people in that subpopulation in your sample leaves you with a random sample. As for the data, the example is fictitious, I was just trying to get away from the children in classrooms in schools type of example.
(+1) First sentence of penultimate paragraph: I might say *is proportional to* in place of *is*.
@EricPaulsson: then, can you please accept this answer? It'll help tidy up the site.
I edited my answer
For example, a model could look like this:
Y_t=alpha + beta_1*X_t*Y_(t-1) + beta_2*X_(t-1)*Y_(t-2) +sigma_t*epsilon_t

, where the X_t and X_(t-1) are two latent variables either 1 or 0, alpha,beta's,sigma are parameters, and epsilon_t is standard normally distributed. 
So this is an autoregressive model, with lag=2.
(1) One would think that the answer would depend on *how* Observer 1 is estimating $x(t)$ and how good those estimates are.  (2) It seems you must assume more about $A_1, A_2$, and $B_2$, for otherwise the $x(t)$ and $z(t)$ will not be identifiable.  In particular, you need some specific *quantitative* assumptions; "changing slowly over time" is not really amenable to useful analysis.
The problematic nature of the last statement becomes clear when you consider that first, distributions do not have standard errors: they have standard deviations; and second, if that's what you mean, any $t$ distribution with 2 or fewer degrees of freedom does not even have a standard deviation.
It is often impossible to draw such a diagram when using circles. Consider, for instance, $|A|=|B|=|C|=2$, $|A\\cap B|=|B\\cap C|=1$, and $|A\\cap B\\cap C|=0$. What kinds of shapes do you want to use then?
Stepwise regression is probably a dangerous suggestion here. See "[Algorithms for automatic model selection](http://stats.stackexchange.com/q/20836/32036)".
@DQdlM: see the bottom middle graph in the first image on the Wikipedia [correlation page](http://en.wikipedia.org/wiki/Correlation). That's StasK's case. It only works when x is equally distributed about the origin (ie, if $X\\sim N(3,1)$, correlation will be fairly high)
Cronbach's alpha is essentially a measure of internal consistency, that it is of inter-items correlations. It suffers from many pitfalls though, especially the fact that it increases with increasing the # of items (it may even be negative, although it is a ratio of variances!). I think we shall focus here at describing variations/homogeneity at the level of individuals within teams, so thinking of mixed-effects modeling or hierarchical models is a good starting point.
Are these actually normal (using some kind of normality test), or do you just refer to the fact that it's bell-shaped? I would actually have expected some kind of heavy-tailed distribution. Additional information which would be helpful is network structure -- how many layers, units, etc.
I have corrected the typo, and provided my best guess as to the R syntax corresponding to the model I propose. You would include a random slope for a given level of data structure if you felt that the effect of a predictor ($\\beta$) varied at that level (e.g. the *effect* of condition on reaction time differed between people—which is different that the person-level random intercept, which merely says that average reaction time varies between people). Advanced: Random slopes can also model heteroscedasticity of predictors at the residual level.
To compute the MLE, you need a sample ${x_1, \\ldots, x_n}$. Assuming iid, the likelihood, that has to be maximised over $\\theta$, is the product of $f_\\theta(x_i)$.
So the probability density function of the product of two random variables equals the product of their probability density functions? Whatever happened to the fact that $f_Y = f_{X_1X_2}$ has only one argument while $f_{X_1}f_{X_2}$ has two arguments?
Obtain within-cluster covariance matrix between X and Y, sum the two diagonal elements and take the root.
I read up about Canonical Variate Analysis. How is it different from PCA ?
(+1) A very good quotation in this context!
First, encapsulate the basic calculation, as in `v <- function(x) mean(x^2) - mean(x)^2` for the population variance. Generate your data: `a <- seq(1,7,2)`. Apply the variance to the data: `v(a)`. Compare that to applying the variance to the set of means of samples: `v(apply(expand.grid(a, a), 1, mean))`.
Hey Gung thank you for your quick reply, I just want to know the general procedure on how to deal with threshold model, how to determine the threshold value, delayed variables and number of regimes. It is better if I can have some idea on how to do it in the program eg:R, as this would save a lot of time on calculations. Thank you very much for your help! I am really new to this topic but I have some knowledge on time-series. If there is anything you want to know pleas let me know! :)
@DLDahly - there is a difference for me, because when I divide an angle value with maximum value of 90 degrees I normalize it (unfortunately I don't know a maximum slope value for normalization - it's about my previous question). I can use it in ordinal scale and define a steep of trend in data (0 - 0.1. slow trend etc.).
I am not really sure where you get that formula. What is the standard deviation of two elements (very strongly related) supposed to tell you? Also $\\text{DP}-\\text{EMA}(\\text{new})$ is what exactly? $\\text{DP}-(1-\\alpha)\\text{EMA}(\\text{old})-\\alpha\\text{DP} $ or $(1-\\alpha) (\\text{DP} - \\text{EMA}(\\text{old}) )$. How can this be a sign of anomaly if it depends not so much on the data but on the weight of exponential?
I know that it's a long debate about p value, but many journals want it.
(+1), excellent program. To spare the reader an extra click, you could link to the new version (G*Power3) which has a GUI for Windows and MacOS X: http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/
@whuber, kindly point it out meticulously. It is turning out to be quite complicated.
Although in 2-class case it is feasible to draw decision boudary first and the discriminant axis second, the actual logic of LDA is opposite. You first have to draw the discriminant line(s). [See a question](http://stats.stackexchange.com/q/12861/3277) (+ important links in comments therein) how to draw discriminants. And about boundaries: [1](http://stats.stackexchange.com/q/15983/3277), [2](http://stats.stackexchange.com/q/92157/3277).
OK. I don't have personal experience with WordSmith but a consulting group (affiliated with a university) I used to work for taught a text mining workshop to non-statisticians that used it. Many of the workshop participants were from the business school.
Generally we are interested in only the few principal components which explain most of the variance. It is possible to do a reduced SVD; for example if $X$ is of dimension $N \\times p$ where $p << N$ then `R`'s `svd` function will compute only the first $p$ left and right singular vectors by default.
What I like is that you can provoke lots of discussion about whether the "effect" was to actually impact fertility (in a medical sense of ability to conceive) or was it social ("I don't want to bring a child into this bad world"). Then drop the bombshell about the Pill if no-one else has brought it up. And then point out that even this can only be one possible factor and discuss some of the others.
`PCA is maximizing variance` This is not generally true. PCA maximizes (by the 1st PC) sum-of-squared deviations from the origin. Only if the data were preliminary centered (centering itself isn't a part of PCA) it turns to be maximizing variance.
This is a FAQ: please see http://stats.stackexchange.com/search?tab=relevance&q=weighted%20mean%20calculate for extensive information on this subject.  For help specifically in controlling SPSS, please see the links at http://meta.stats.stackexchange.com/questions/793.
What are you modelling ie what is the explanatory variable?  It makes a difference if there are good a priori reasons for using 1972 as the beginning of your series.  I would need to be convinced that this is the right moment for identifying significant structural change - it depends on "change in what?"
the latter is just a particular case of the former, so the statement is true.
@EpiGrad, thank! Damn google... Fixed.
It was as simple as that after all. Thank you.
That seems a bit extreme.  Type I and Type II is defined for both approaches.  The problem, it would seem to me, is that to say much about the comparison in error terms they'd have to know what sort of non-Normal they're up against (and if they could do that well enough, they might as well build a model and dump the bootstrapping *and* ranks :-)
+1 A valiant, tactful, and informative attempt to respond to an unanswerable question!
Yes, there's certainly a shortcut because it's just an ordinary scale parameter, so $E[X^k]= E[\\sigma^k\\,(X/\\sigma)^k]=\\sigma^k E[(X/\\sigma)^k]$. So just multiply the $k$-th raw moments for the standardized distribution by $\\sigma^k$.
"*For example, if your original probability distribution was uniform, then the sampling distribution would be normal.*" -- intriguing. Let's see. $U_1,U_2\\sim U(0,1)$ and independent. Let $V=\\frac{1}{2}(U_1+U_2)$. Is $V$ normal? No, actually, it's *triangular*! Okay, maybe that was just bad luck. What about $n=3$? Nope. $n=4$? No. $5, 6,$ or $7$? No, all not normal... I think you need some qualifiers on your statement.
I will say that there's no hint in the little information you give that the Poisson comes into this at all. A little more detail might clarify that.
You raise an interesting point about measurement and interpretation. But what happens if income is measured using brackets starting at zero? Then imputed observations become special in that only they are permitted to have negative incomes.
The benefit would be the correct incorporation of the binomial variability.
$X_T/X_C$ is just the ratio of means and it is also sometimes called the "response ratio". You can find it being used, for example, in meta-analyses in the field of ecology. See, for example: Hedges, L. V., Gurevitch, J., & Curtis, P. S. (1999). The meta-analysis of response ratios in experimental ecology. Ecology, 80(4), 1150-1156.
I'm not sure what you mean. If you have some continuous variable column A (say people's heights) and some grouping factor column B (say Country of origin), you really want to do a `pairwise.t.test()`. You would do this like `pairwise.t.test(df$A , df$B)`. This function will automatically correct for multiple comparisons.
I also love your code and the speed I received the answer - was my first question on this site! It is very simple to rewrite to Java and I will look over the weekend how useful the results are with my data. Thanks MansT! (Oops, I just saw I need a reputation of 15 to vote up)
Could [this](http://stats.stackexchange.com/questions/41282/dynamic-conditional-correlation-in-stata) be helpful for you?
If the assumptions are reasonable you might consider a *paired* t-test (which would be a one-sample t-test applied to the intra-company differences), but I think the first problem you'll have is that the independence (between companies) assumption is highly suspect (in fact I'd say 'pretty much known to be false', since their performance is related to the economy as a whole).
Did you try anything at all? Like Analyze > Regression > Linear??
(+1) Good example. Sometimes clients collect very specific samples but they want to make conclusions about the whole population.
Just repeating experiments. Unfortunately, the number of observed cells were different.
"Now you want to know whether the parameters influenced the ability to learn the bevaviors and if so, did the more successful teams behave differently? Is that right?"

This is pretty much on the money, I was actually wondering whether more successful teams behaved differently as a result of their success, if there's a test that'd let me show the direction of that effect that'd be great, but I assume that there wouldn't be and I'd have to show the relationship was there and then argue the causality from the nature of the simulation?
Unless I misread it, the question sounds like an effort to detect outliers. The bootstrap doesn't apply here and, besides, it's fruitless.  Even the *maximum* value in the dataset is going to occur in $(1 - ((n-1)/n)^n)\\approx 1-1/e\\approx 0.63$ of the bootstrap samples, which is far too high to give you any chance at achieving a low p value. For the books, see http://stats.stackexchange.com/questions/5845, http://stats.stackexchange.com/questions/15692, and http://stats.stackexchange.com/questions/25151.
I am not sure about your ANCOVA model, for example, is there any interaction term with age? You mean you did ANOVA, and the interaction between `male gaze` and `clothing` is insignificant, which is different from the ANCOVA model, am I right? It would be clear to write down both the ANOVA and ANCOVA models.
@MR Rather than ask what *could be* added to these systems, think about why you're doing this.  For whom?  What uses to you want to support?  What are the compelling reasons (if any) to be adding statistical procedures to a symbolic system?  In other words, begin with a good functional specification.
I thought it best not to mix the questions together as they cover different topics. Here I am asking how to bin the data; the previous question was asking about comparisons between subsets of the data.
@MånsT: Basically yes, but I would also appreciate if you could provide a few hints regarding the theoretical aspects of this concept...
It is part  of an Index scale in which it is designed to measure over all political involvement, I collected the data my self and it was indeed ordinal and indeed, I did collapse it to a binary form. I think I will just do the reverse and make "following the news"the independent variable. unless you have any other thoughts ?
What you presented is called a “contingency table”. Among other techniques, you could use a “chi-squared test of independence” to see if both variables (call reason and customer age) are related. The phrases between quotation marks should help you find plenty of relevant material.
This is not a plot of predicted vs. actual: it just seems to be an arbitrary set of dot pairs where the x-axis designates a row in a table.  Your problem is that it's next to impossible to learn anything from such an unsuitable graphic. Consider instead something more meaningful, such as (crudely) binning the data by predicted value and plotting the mean observed value against the mean predicted values in the bins. There ought to be a good correspondence--and you can even detect nonlinearity of the (log odds) from such a plot.
I believe that Efron's book focuses on a large number of variables (and the resulting multiple testing issues that arise), not sample size.
Such large statistics, with 4 df and non-significant p-values, look rather strange. How were they obtained? Apparently, they come from a cross-classification (contingency) table? Could you add a little bit of context on your study and show us the figures?
So I had to compare $Y$ and $y$. I shouldn't have eliminated $Y$ from the calculation at the beginning.

Indeed,
$$\\begin{align}\\left\\{
G_1(y) = \\mathbb P(Y<0)
G_2(y) = \\mathbb P(Y=0)
G_3(y) = \\mathbb P(\\{Y=0\\}\\cup\\{a<Y\\le y\\})
G_4(y) = \\mathbb P(Y\\le b)
\\left.\\end{align}.$$
And the book must have gotten some things wrong, because that solution makes perfect sense to me.
OK, but the question above is about how to deal with results changing after each seed. I suggested obtaining an idea about the variance of the performance through random sampling (without repetition) from your data. The plots you are showing there have no bands. How come you are not caring about the variance anymore?
wow thanks - that was it - i hadn't even considered the possibility :)
For some pictures using ks, see [Multivariate_kernel_density_estimation](http://en.wikipedia.org/wiki/Multivariate_kernel_density_estimation).
I don't know what you mean by "valid" here. A normal Q-Q plot shows an (approximate) linear configuration if data are sampled from a normal. It can be a good informal means of looking at data. But (1) something that looks like an outlier with respect to one distribution may be perfectly compatible with a different distribution (2) data fitting the description you give aren't remotely compatible with a normal, so I wouldn't expect such a plot to be useful here. Some other distribution is surely more plausible, quite which depending partly on how your data were produced. Poisson? etc.??
Here's the URL for the first formula: http://www.uta.edu/faculty/ricard/stats/Lectures/Chapter%208/Chapter%208%20t-Test.ppt
Also note that, for comparison purposes, `var(dn)` should be replaced by `pi / 2 * var(dn)` since $\\sqrt{\\pi / 2} d_n$ is your consistent estimator of $\\sigma$.
+1 - I've seen other examples of spiral graphs as well, they seem really hard to digest (especially when they are noisy and many series) compared to well designed seasonal line charts.
What is your basis for saying the F value is 'too high'?
Hi Clay. You're welcome! Search "sas/or examples" for more content. Using sets requires some practice, but they are worth learning about. Most people only learn to use sets when they get thrown into the deep end of some consulting project. My only other suggestion is to multiply ratios through to linearize the model. That is probably what your instructor wants you to do for full credit. Your model works in OPTMODEL because SAS has an excellent Nonlinear Programming solver. But it won't work with some AMLs that expect linear models.
Not sure if this helps, but checkout Google Trends (http://www.google.com/trends/explore?q=Speedminton#q=Speedminton&cmpt=q).  You can download the trend data as a .CSV file.
@Dason One standard (and powerful) *frequentist* technique for finding good (*i.e.*, admissible) estimators is to compute Bayes estimators for various priors. (See, *e.g.*, Lehmann's book on point estimation.) You have just rediscovered one such estimator.
Great. Thanks, that makes a lot of sense. I re-sampled the data frame with replacement and found that the 95% CI was between 4 and 100% variance explained. I guess it is reassuring (but not particularly exciting) that the CI doesn't include zero. Its a useful dose of reality to see how little precision we have when the sample is small...........
Showing the exact command you used and the output you got would help. On points of terminology and spelling: `margins` is a command, not a function, and the spelling is Stata.
Perhaps obvious but still incomplete, because (1) you cannot always find solutions $\\Sigma$ and $\\mu$ and (2) even when you do they do not necessarily determine a valid distribution.  Isn't that what the question asks, though: *what are the conditions* for $f(\\mathbf{x})$ to determine a multivariate normal?
@SpookyFM @jona The documentation for `clmm()` on CRAN is out of date, make sure to take a look over at the package on R-Forge. See [this message thread](https://stat.ethz.ch/pipermail/r-sig-mixed-models/2014q3/022593.html) from the R special interest group for mixed models.
The model refers to the *expectation* of the dividends rather than the dividends themselves. Thus, a zero dividend should not cause any problems.
@Robert Out of curiosity, how did the implementation work out?
:Wayne The end doesn't justify the means.Transformations(like drugs!) can be both good and bad for you.The idea is KISS.If the variance of the errors is non-constant AFTER you have removed outliers/level shifts/local time trends/seasonal pulses &removed any auto-correlative structure in the errors ( perhaps self-caused) &after fully g capturing/extracting the effect of leads and lags &after verifying that the parameters had not changed over time after fully verifying that GLS/WeightedLeastSquares was not applicable THEN one might investigate Power TRansforms e.g. reciprocals/sq root/logs.
When you say you have 2 sets of brand preference data, is that just another way of saying you have two groups: exposed and not-exposed?  When you say your target group is F20-40 years, do you mean that you have one target group or that you want to look at brand preference by age?  Assuming the simplest situation, you have a 2 by 2 design: exposed/not by pre/post.  You could consider doing a 2-way analysis of variance.  Have you looked at the distribution of your brand preference data?  If so, what does the distribution look like?
I should add that John Nolan was writing a book on stable distributions that was going to include a chapter on Stable RWs. His webpage might be useful: http://academic2.american.edu/~jpnolan/stable/stable.html
Is there a standard article that can be cited to dissuade reviewers from demanding this all-too-common practice?
Once you move to log(counts + 1) you move away from Poisson modelling. The two aren't compatible: you lose the theory and you don't input transformed variables into Poisson regression any way.  You don't explain SAR. In total there is not much detail here to guide advice.
Thanks for the input.  Appreciate it.
Not sure if this helps, but this article claims using Cronbach on ordinal data will underestimate the true value: http://www.amstat.org/meetings/jsm/2007/OnlineProgram/index.cfm?fuseaction=abstract_details&abstractid=308244
Edited to add: You may find the Wikipedia discussion on this aspect of the ICC to be illustrative: http://en.wikipedia.org/wiki/Intraclass_correlation#Use_in_assessing_conformity_among_observers. In particular, it states that it can be "used to assess the consistency, or conformity, of measurements made by multiple observers measuring the same quantity." Since the raters in this case are not measuring the same quantity, the ICC may not be well-suited to your situation.
[Why can't users with less than 100 reputation answer their own question sooner than 8 hours after asking?](http://meta.stackexchange.com/q/123374/150510)
@Procrastinator thanks.  do you have some reference for good Bayesian hierarchical models dealing with this kind of models?  thanks.
What predictors do you have?