tttjjj/elt_bench huggingface dataset:
Related thread: [How to get an R-squared for a loess fit?](http://stats.stackexchange.com/q/24993/930)
Not sure I understand the question.  I just want to know if I set up the monte carlo simulation correctly.  If the question is, what do you hope to learn from the simulation, the answer is that I think the median outcome of the 100 simulations is my "best guesstimate" of what the account will be worth in 20 years (assuming assumptions prove to be correct), and the 95% upper and lower bounds will give me my best guesstimate of the lowest and highest values of that account in 20 years.
I need a formula for calculate sample size. (for CFA)
@chi sorry about naive it was supposed to be native...and I added a definition of clustering
I agree with your update.
If you have an ordered factor the `clmm` function from the `ordinal` package will be useful.
After doing the recommended substitution, I get $E(X)=\\int^{\\infty}_0 \\alpha y^{1/2} e^{-\\alpha y} dy$. If we take a gamma density such that $\\Gamma (t)=\\int^{\\infty}_0 z^{t-1} e^{-z} dz$, for some variable $z$, does that mean $E(X)$ evaluates to a gamma density with $t=\\frac{3}{2}$ and $z=\\alpha y$? I am unsure how to evaluate the gamma density now. Do I pull out the $\\alpha$ and proceed from there? Apologies if I am missing something very basic here.
Since this looks like [routine book-work](http://stats.stackexchange.com/tags/self-study/info) it should probably carry the `self-study` tag (and you should read the tag wiki info at the link for how such questions would generally work). There's at least one tag on your question that doesn't seem to be needed, so there seems ot be room for the new tag to go in its place.
Can you say more about what you mean when you say the groups are not randomly selected? Also, what do you mean by "_intact groups_?"
@Kyle it would be better if you could find a way to edit your question that reflects what you didn't understand more clearly. The point of answering isn't just to resolve *your* particular issue, it's to solve the problems of future readers with similar issues. If you can edit to ask the kind of question that you would have asked if you'd known better how to put it, *you* can then supply an answer that would have been useful to the you that asked the original Q, and which may help the next person.
Also wanted to add someone posted r code to do this yourself: http://www.r-bloggers.com/r-for-ecologists-putting-together-a-piecewise-regression/
i did model1=lm(Gini ~ A + HDP + Daonva + Vek)
then vif(model1) where i get error
You can see all the code underlying an r function by typing the function call at the prompt.
However, the result of anova(lm1, lm2) of course depends on which of the 1000 variables you look at. So how can I find out whether to include or exclude the 3rd covariate cov3?

Best Jacky
In an ideal _frequentist_ world. There are other worlds that are worthy of consideration in the context of statistical inference.
@gung, Thank you for the edit :) The question seems more clear now. But, it is not a homework. It's my own study to understanding basic Bayesian analysis and I removed the homework tag.
For additional discussion, please visit http://stats.stackexchange.com/questions/62677/covariance-v-correlation-based-pca-theoretical-view.
+1, Very informative, thank you.
Erik: I have double check my code and the prediction result in the 10-fold cross-validation. There is no problem with the calculation of the residuals.
Sorry no cant remember, Google should be able to help :)
Typically for solving problems like these, you would use supervised learning. That would require you to have a another time-series, where the steps were manually labeled. Otherwise at least, you will need to know some additional information, such as the number of the turns corresponding to the above series. Is it 2, or is it 12? Unless you have access to some well-known parametric model for steps, you won't have much luck with this one. Speaking of change-points I read an article about this slick shiny app: http://glimmer.rstudio.com/pssguy/TVShowRatings/ check if the method is any good.
I guess I should change my close vote to "unclear what you're asking" on a second reading. There are so many ways to get summary variables out of several that a good answer would be too long.
[continuous distribution](http://en.wikipedia.org/wiki/Probability_distribution#Continuous_probability_distribution) vs [discrete](http://en.wikipedia.org/wiki/Probability_distribution#Discrete_probability_distribution)
Thank you all for such interesting thoughts about this controversial part of the Bayesian statistics. I have been reading and comparing your points. There are interesting arguments validating its use in terms of formal rules, practicality and interpretation. I will select an answer at some point, but I am affraid this is going to be a very difficult task.
It is almost always more fruitful to approach this question from the other direction: when the phenomenon you are studying is described by a random variable $X$ with support in $[0,1],$ what does the underlying theory suggest about the possible distributions $X$ might have? By inverting this natural question you are stripping all knowledge specific to the phenomenon from your analysis, leaving your results to luck: the luck that the phenomenon just might (accidentally) match some distribution suggested by an abstract mathematical theory of questionable relevance.
Did you attempt to read anything on the topic? For example, Wikipedia article and Case Studies under SPSS Help menu? So far, it is not very clear what you mean under `interpret`. You want to know what a discriminant _is_ or how to attach a _substantial_ meaning to it (from the input variables)?
(+1) useful to know, but I'm more looking to understand the *consequences* of not controlling for the three types of measurement error while using OLS.
I must admit, I have recently abandoned the plural use of data after clinging to it for 10 years or so. I generally write for non-technical audiences and I was worried I was coming over pompous. The APA seem to still have a strict reading on its being plural but interestingly the Royal Statistical Society don't seem to have a particular view. There's an interesting discussion here: http://www.guardian.co.uk/news/datablog/2010/jul/16/data-plural-singular
(+1) Good catch!
The regression line goes through the point $(\\bar{x},\\bar{y})$. If you know $\\bar{y}$, you can find $\\bar{x}$ by solving for it after substituting $\\bar{y}$ into the regression equation. $\\bar{x}$ will be the ratio of the number of people that bought the item to the total number of people in the sample. It looks like Stata gives the number of observations in the sample, so just multiply $\\bar{x}$ by the number of observations.
Estimating with actuals vs estimating with estimates - you decide :)
Can you tell more in the question about `adversarial noise`? It seems to be a specific term but not often used. Expain what it is.
Hey, yes the CI interval was set at 95% and I double checked the p-value calculations. I tried several RR calculator including the one for cochrane studies. The reason number 3 does seem plausible, would have never thought of that myself! Thanks
Thanks, that sounds like a great example.
@wnstnsmth No worries.  Hope the answer was helpful to you.
Assuming the green data are correct, the blue model is the right one and the red one is completely off.
Your residuals versus fitted plot suggests that your dependent variable has a lower bound. This could drive the patterns you see. This could give you an indications for alternative models you could consider.
I am baffled at what to do. For a mean with no assigned points each  $r^{(i)}$ goes to $0$ as $\\beta$ goes to $\\infty$. But how the new mean $m$ behaves in the limit depends on the comparative rates the $r^{(i)}$'s go to zero which depends on the other means. How to interpret all of this into some simple geometric interpretation eludes me.
Your one group only has 10 people it, I would be very cautious about drawing conclusions from the dataset.
Just to be clear: if we define $f(p,q,r)$ to be the set containing the maximum and minimum of $\\{(S_p(i)+S_q(i)+S_r(i)), 1\\le i\\le 3600\\}$, then you want to find a data structure of smallest possible size that allows for computation of $f(p,q,r)$ for all $1\\le p \\le q \\le r \\le n$.  Furthermore, you appear to allow $f$ to be computed with some error: but how much error is acceptable?  Is there a chance the sums could overflow?  Finally, how large is $n$?  (After all, if $n$ is small, we could just store *every* value of $f$!)
Hahaha, yes, you are right. Though most people seemed to have gotten what I meant :-)
I agree with the stackexchange suggestion. I hope someone there will tell you that with 3 data points, there's really extremely little point using fancy methods to compute confidence intervals. Without making assumptions about the distribution, it will be almost impossible to nail down confidence intervals. What do you think the 95% confidence intervals are of {1,2,13} ... ???
"Used like this: `m=mean(c);n=length(c);s=sqrt( cf(a,m,n) + cf(b,m,n)`"  I thought the idea was to throw away all the data points in a and b and retain only $\\mu_a$, $\\mu_b$, $\\sigma_a$, $\\sigma_b$, and $n_a$, $n_b$, the number of points in each set.  So, `m=mean(c)` seems to not conform to this ideal.  Why not set $n=n_a+n_b$, $$\\mu=\\frac{n_a*\\mu_a+n_b*\\mu_b}{n}$$ etc and thus get all the information from the summary statistics without requiring all the data points?
It is possible to calculate a correlation coefficient between a continuous and a binary variable.  Read up a bit on correlation coefficients. Good luck.
@doctorate: I realized that my above comment may have been kind of oblique wrt what you want: the jars in your case would be the pairs of features, i.e. (A+,B+), (A-,B+), (A+,B-), (B-,A-) for a total of 4 "jars". Does that help make it a more concrete?
I am not sure if I am making an unnecessary comment here, but, if you have only access to a uniform random number generator, then you can use the [Box-Muller Transform](http://en.wikipedia.org/wiki/Box%E2%80%93Muller_transform) to generate independent N(0,1) random numbers.
In a nutshell, if U_1 and U_2 are independent draws from the Uniform(0,1) distribution then $$ \\sqrt{-2 \\log(U_1) } \\cos(2\\pi U_2)$$ and $$ \\sqrt{-2 \\log(U_1) }\\sin(2\\pi U_2)$$  are distributed as independent N(0,1) random variables.  
The basic idea
It may help you to read my answer here: [When to use Fisher and Neyman-Pearson framework?](http://stats.stackexchange.com/questions/23142//51823#51823) As for how the rejection region relates to p-values, the RR is defined as that region where $p<\\alpha$.
@whuber is correct; if the question was different my answer would be wrong.
Thanks! I wonder why the random effect model require the random parameters to be uncorrelated with the input variables, while the fixed-effect model allow the parameters to be correlated with the input variables?
Do you mean 10,000 and 100,000 records (rows)? Variables are columns in a dataset.
What does "unreliable" mean here? Puzzling? I can't read your data but (for example) a bimodal bootstrap sample is very common when outliers are present: the outlier(s) will appear in some samples but not others. There are many relatives of this situation. Closer inspection of the raw data can often make much clearer what is going on. More to the point, it seems that you are fitting a mixture of normals. Is this exactly what you might expect with a mixture of normals?
**Aside**: I would be interested to see an accurate and reputable sourcing of your introductory quote. It is variously attributed to several people on the internet, most recognizably, Yogi Berra and Albert Einstein. I have *personally* heard it from an engineer that is a generation older than Perlmutter and this was long enough ago that it makes me *highly* doubt that Perlmutter could be the original source.
Wikipedia provides a [simple formula for the skew normal PDF](http://en.wikipedia.org/wiki/Skew_normal_distribution).
It is unknown if any of the words are meaningful.  How would I determine if the words, either individually or as a phrase, are helpful in predicting the value of Result?
We welcome questions like this, but we treat them differently than others. Please tell us what you understand thus far, what you've tried and where you are stuck, and we'll try to provide hints to get you unstuck. To better understand the process, please read the wiki for the [self-study] tag.
But the variance of the error is smaller in (B). So I thought that I would get (B) using OLS. But now comes the question: if that is true, what guarantees that (B) is the AR representation of $X_t$ with minimal variance of the white noise term?
+1 in part due to "As a rule of thumb, you should be wary of rules of thumb."
Thank you a lot, I'll go through this course.
If this reasonably solves your problem you can consider accepting the answer to your own question.
BPO is business process outsourcing...call center where we get multiple and different projects. By doing statistical analysis we can tell whether we should take up this project or not...but I dont know how to start or which book should I read in detail.
On my claim that taking the same transformation of both variables can change the sign of the linear correlation, [this answer](http://stats.stackexchange.com/a/35943/805) has an example. Of course, the nonparametric ones are all unaffected by such monotonic transformation. (Note that the Pearson correlation doesn't actually assume normality, though tests of significance on it may.)
A lot of applied statistics books start with the supposition that the model is just *known*, & deal with these sort of issues poorly if at all. I'd highly recommend the two mentioned [here](http://stats.stackexchange.com/questions/106235/106303#106303).
thanks for the explanation. Yes, they are indeed dynamic bayesian network. In fact, they are also referred by dynamic template models [Daphne Koller PGM principles and techniques](again a very specific type of dynamic bayes-nets). However, as stated above my only concern is with those dependent observations. I don't require any software packages, I just want to know if there are research papers that target similar PGM structures.
A term I have often seen, and grew familiar with decades ago (when I was writing control chart software), is "OOC": out of control.  See, for instance, http://books.google.com/books?id=uTfGl29zuT4C&pg=PA339&lpg=PA339&dq=ooc+process+control&source=bl&ots=VPSvGNEm6C&sig=8aaEx-XvrBpZcPVKqWen3nWRacs&hl=en&sa=X&ei=71PMUei-Oa3K4AO0tYDQDQ&ved=0CFwQ6AEwAw#v=onepage&q=ooc%20process%20control&f=false.
When I think of it again, I guess there is a chance for the multiclass to outperform, only if the binary classifier has a large bias (bad performance on the training set). In these cases, the multiclass artificially increases the hypothesis space and thus increase performance. However, when the training performance is good (low bias), than adding more nuisance labels may degrade performance.
Assuming daily data, a divisor of 365.25 or so will be nearer the mark than 366 for a period of several years. You should not notice the difference, but still, there it is.
Gung, As the OP says, H&L merely state that when LS is used, "the estimators no longer have [a number of desirable properties]" [2nd Ed. pp 5-6].  This statement is vague, it is offered without further explanation, and no citation is provided.  This question does not solicit a list, but rather seeks credible support for an otherwise unfounded assertion.  I think it's a fair question and gladly vote it up.
This looks like the multinomial likelihood to me.
Hehe, I think you need first to understand that clustering itself is ill-defined altogether. I suggest reading the work of Prof. Shai Ben-David on the theoretical foundation of clustering.
**NB:** This is a reply to a second question that was subsequently merged with the present one.  As @mark999 correctly points out here, the two questions are the same except one asks for Laplace variates and the original (this one) asks for Student t variates.
(1) If you have replicates why not analyze them as such? Why do you want to take means first? (2) How are counts not whole numbers? Are you dividing by the duration of the observation period to get rates? Or is it just that the means are fractions?
Can't you simply flatten the vector out? I mean, from what you describe it feels that for each signal, you have a 2D matrix, in which case, you can simply write the matrix out in row-major order or column major order and create a single vector out of your signals.
@chl I think, if I correctly understand your comment about continuous covariates, that exactness depends on the role played by randomness in the data.  When randomization occurs *by design*, it doesn't seem to matter what kind of data you have.  The permutation test takes the data as given and simply lets us glimpse how the statistical results would turn out if our random number generators had (for example) resulted in different assignments of subjects to treatment and control groups.
The null-hypothesis of the Wilcoxon Signed Rank test is not in terms of means, so it is probably the wrong test to use for your null-hypothesis.
But the wiki page of MSE also gives an example of MSE on predictors,http://en.wikipedia.org/wiki/Mean_squared_error
I was basically looking for cases for which the Shannon entropy is known, since I needed a pair of variables that have Gaussian noise but not a Gaussian marginal distribution on $Z$. I used uniform + Gauss now, but if you have others, I'd still be interested in it.
This question appears to be off-topic because it is about R functionality.
It works with the example you just gave, yes. Also works with other exponential parameters. I included a check that the total probability mass was 1 (well, I checked that $\\sum_{n=0}^{19}{C_n (1)} \\approx 1$. Bear in mind that as $t$ increases you will need to sum over more values of $j$ for the Taylor series to be valid.
thanks Matthew. It's a great technique.
@Dian... not at all. Some comments: (1) I don't have a dog in this race, *but* if a reviewer explicitly asked for FDR, I would go ahead and do FDR :); (2) these methods are implemented in most statistical software, so if you ain't doing by hand, consider that (e.g. `p.adjust()` in R, `qqvalue` in Stata, etc.), (3) you have a $p$-value for each variable and interaction term in *each* model (I counted 20-something in your examples above), you can also add each whole model $p$-value if reporting; (4) if my answer is useful, I hope you will consider clicking the check-mark at the top left of it.
+1: For concrete reason to prefer the natural logarithm.
Perhaps I need to regress my predicting variable onto the other two IVs (nuisance variables), and then conduct the simple slopes analysis as if the new _corrected_ predicting variable, the moderating variable, and their interaction are the only IVs?

I'm reticent to do this however, because it seems like the interaction between the moderating variable and the _corrected_ predicting variable will be conceptually different than the original interaction.
I don't quite follow your question. (Presumably, by "under" in the 1st sentence, you mean *understand*, & by "partial SS", you mean *type I (sequential) Sums of Squares*.) In addition to @whuber's good suggestions, could you elaborate on your question? Is this a real situation or for your own learning? What is it about type I SS you are trying to understand? Can you comment your code? What exactly are you trying to do & getting wrong?
-1 Thanks for providing some relevant information but this completely misses the fact that the original poster seems interested in equivalence, not differences.
start with `predict` (i.e. `?predict.merMod`) with `type="response"` to get probabilities, and then tell us more about your definition of an optimal threshold ...
This is excellent advice minus the comment about bars being more appropriate. For similar discussion related to that see [Alternative graphics to “handle bar” plots](http://stats.stackexchange.com/q/1173/1036). Imagine the plot listed by the OP as a clustered bar chart, it would be mightily difficult to visualize the trend across tempature ranges. A way to make the points more easily visible is to jitter them along the x-axis, and Cleveland's work would suggest we should prefer points to bars anyway.
The glmnet package is a very fast implementation of the lasso as well
If you calculate a 95% confidence interval of the correlation coefficients for your and their study, you'll likely still observe overlap, and it will give you a feel for which range of values either study is compatible with.
@findus, it may be kind to mark the answer as being the answer to your question (green check mark)
Fix whichever one makes more sense to you to fix.
Did you see the (more recent) comment by @Max to the question?  He nailed the answer.
There was another question (that was more general than capture-recapture), [How to estimate how many people attended an event (say, a political rally)?](http://stats.stackexchange.com/q/4111/1036). The [significance article mentioned](https://www.scss.tcd.ie/John.Haslett/st2004/Significance%20papers,%20plus%20others/How%20Many%20Were%20There.pdf) does not talk about capture-recapture, but talks mainly about the density approach, which strikes me as on its face much simpler. (Heterogeneity in comers and goers to the festival strikes me as potentially problematic for capture-recapture.)
In requiring an objective function, I guess you mean it is not always possible to find coefficients Am, Ak, As such that the three summations are 0 ?
I'm trying to plot the data from the negative binomial count portion of my hurdle regression (I can plot the logistic portion just fine). I would be happy to provide data, if helpful. How should I go about doing this? And what do you mean by "MWE"? Hopefully these questions aren't painfully basic. :)
I want to do pairwise comparisons for the two-way interaction terms, for example "Sediment*Hydrology".
If the anova table indicates that the Sediment*Hydrology interaction is significant for the full model
`mod1<-lme(Variable~Sediment*Hydrology*Depth, data=mydata, random=~1|Site/Hydrology/Depth)`
I don't think that it is correct to do post-hoc comparisons (on the Sediment*Hydrology interaction) the following way :
`mydata$SH<-interaction(mydata$Sediment,mydata$Hydrology)
mod2<-lme(Variable~-1+SH, data=mydata, random=~1|Site/Hydrology/Depth)
summary(glht(mod2,linfct=mcp(SH="Tukey")))`
Thanks for the pointers.  Upon re-reading of the question, I think my original interpretation was wrong and your interpretations seems much more like it.
@user87 There's an [academic version](http://www.enthought.com/products/epdfaq.php#Academic), free of charge. Also, Enthought recently released [EPD Free](http://www.enthought.com/products/epd_free.php) (which includes a dedicated package manager), although it does not include all those nice packages (like Mayavi or Chaco) that are present in the full distrib.
@DirkEddelbuettel The F-test is not the lack of fit test !
Its also worth noting that as soon as you changed the model because of the modification indices, you were no longer doing CFA, rather you have moved back into exploratory mode.
In the GIS world--which is the natural place to look for tools to solve this problem--Voronoi diagrams are also known as Thiessen polygons and Euclidean allocations.  Some people use *ad hoc* methods to remove the inliers, such as performing morphological operations or local filtering (*e.g.*, a majority filter) to clean the resulting images.
Ahaa, interesting, thanks James. So for a Gaussian/normal distribution of the positives... sorry, you've lost me a little: would it be a likelihood ratio test of the probability that [all data] are normally distributed, vs [only non-zero] data?
(p.s. if you know the name of that test in R off the top of your head, please say)
Thanks!