tttjjj/elt_bench huggingface dataset:
Thanks @Henry, these are really useful! I'm not actually limited to only two charts, it was just how I had thought of it previously. If you have other ideas on how to show the data they would be very welcome!
i don't need a programming solution. i need it's formula.
@Rob Good point about explicitly modelling using the lag1 covariate - I'd overlooked that. I was thinking more of using the Bayesian tools and something convenient to allow computation of different within group AR(1)s. Perhaps you should add your modelling lag1 as a Answer to your Question?
In addition to the good comments and answers you've already gotten, I think the better question is "Given the power of computers, why not do simulation/permutation tests all the time?".
"Chapter 5" of what?
Ah, I see your problem!  I hadn't paid enough attention to exactly what was written, my apologies.  The $n$ should be included under the $\\sqrt$ sign.  I'm guessing, from the evident LaTeX problems on the rest of the linked page, that the author simply didn't get the formatting correct.  Good catch!
Am I correct in assuming that C is a continuous variable?  If so, agreement can be indexed in terms of absolute deviation or in terms of correlation.  Do you have a preference?  (These two approaches can give different results.)  Also, correlation can be linear or curvilinear.  Do you have a prediction of what you will find?
I approved it. There was enough there to make it worth fixing. You and Nick Cox should start a club - between you there's not much that gets through undetected.
@AdamO: I have ~10,000 vectors, with continuous values (-1 to 1) in ~80 dimensions. The outcome should be a 2d plot of the vectors/points after the reduction, so a human being can look at it.
Why did you stop at $X^3$?  Because the second-order approximation is a quadratic function of $X$, its variance will generally involve moments of $X$ up to $2 \\cdot 2 = 4$.  The third moment may be zero, but the fourth moment is definitely going to show up and not get canceled out by anything.
Sorry if I missed something, but why not just plug your hypothetical $x$ values into the formula?
Basically, I want to know the differences of these two functions. I  believe, maybe wrongly, that these two functions give the same results.
But, each DV added is going to penalize a degree of freedom. So, given the sample size available to me, it is not an approach I want to make. That is why I excluded MANOVA as a solution to the problem.  But, I appreciate it's usefulness as a way to explain the solution in terms of regression.
You are right. I made a mistake in my calculations. I apologize. Thanks for your attention.
"tough old bird" is idiomatic. Some sources claim that it implies females, but I've heard it applied to all genders.
@Tiberiu-Ionuț Stan Your comment might be correct but it consists only of unsupported remarks about social networks; it does not seem to contain any statistical reasoning nor to throw any additional light on the question. In particular, I still cannot see any specific reference in this particular answer to correlation or causation. Remember, we're not here to debate the future of Facebook or the quality of social networks, but rather we have been asked to evaluate the statistical arguments in the paper in question.
excellent point about the tails, which I think is key. Another contrasting example would be to consider weibull and gamma as alternatives to the log-normal. In practice, these often provide more realistic fits to right-skewed positive variables.
Thanks, I like the reference too!
@cardinal: Not cw, but probably asking for references is better than asking for consensus..
Regarding 3) sometimes violating assumptions improves fit unrealistically.
(+1) Depending on exactly is intended by "Consider the subspace $\\mathcal M$ generated by the random variables $\\{Y_u: u \\leq t\\}$", $\\mathcal M$ may not be (directly) associated with a $\\sigma$-algebra and yet the problem makes sense. For example, $\\mathcal M$ is often taken to be the set of finite linear combinations of $\\{Y_u: u \\leq t\\}$.
To do multivariable regression analysis without at least a couple of courses in regression models is similar to doing political science research without any training in political science.  One of many considerations is whether it is reasonable to assume conditional normality and constant variance for your particular dependent variable.
Thanks, I get it now. I guess I mixed up some frequentist concepts in my hypothesis testing approach.
It's not a technical term. Does the turning point work for you?
A couple of people were largely absent in most of the questionnaires, but otherwise it was people being exceptionally away or ill when I was doing a particular questionnaire. The missing responses seemed roughly normally distributed by rank (I have not confirmed this though).
MBQ. Many thanks. (i) ROC Curve: In this instance I don't require the ROC curve as I have my own priors on what the cost weighting should be. (ii) `classwt`: Yes, I have found that in practice, and in line with other users, the results are not as expected. (iii) `cutoff`: I'm not clear about how to utilise `cutoff` in this instance and I'd welcome any further advice.
Thanks for your reply. All points taken.
Did you find the [strucchange package](http://cran.r-project.org/web/packages/strucchange/) in your search?
As to comparing the medians of all three simultaneously, see my edit for the slightly modified R code.
Michael, thanks for your response. My post may not have been entirely clear unfortunately, since it was supposed to be concerned with the R script itself and not primarily with the statistics. The dataset provided was just an example, and I want to create an R script that would work in the general case. The problem is that I get a bias of 0 and a std. error also of 0 in the bootstrap statistics for all t1* to t100* as I show in my question, which means that the boot.ci function won't work. How can the script be rewritten to function?
In R, the function 'mvrnorm' (part of the MASS package) makes random generation for the multivariate normal distribution. Specific correlation structures can be obtained by modelling the variance-covariance matrix ('Sigma' argument).
I think you may be misremembering this. Best subsets is much more computationally expensive than stepwise, but would necessarily catch anything stepwise would, so you would use stepwise to screen & best subsets after. FWIW, I disagree w/ the naive use of these strategies, for reasons I discuss in my answer here: [algorithms for automatic model selection](http://stats.stackexchange.com/a/20856/7290).
I see. Though given the low sample size. I would suggest tabulation by state rather than any regression.
@Im - I have added an updated response. While your question is well articulated, it is quite verbose and still difficult to parse. Instead of adding more text, I think it would be better to attempt to reduce your question to a few key points/questions.
If you are wondering how to choose between a model w/ Y as the response variable vs ln(Y), you don't usually do it based on statistical criteria. These answer different questions. To understand what they mean, try reading this excellent CV thread: [Interpretation of log transformed predictor](http://stats.stackexchange.com/questions/18480/interpretation-of-log-transformed-predictor).
This isn't really an answer by CV's standards, @SaraWong123. Although in a sense it does provide an answer to the OP's question, it is still more of a comment than an answer at present. Would you mind elaborating it into more of an answer? Alternatively, we can convert it into a comment to the question.
Hello steffen. Thank You for info, I'll turn to rm forum. Should I mark this post as Closed in the title?
The last part of your question ("Why should we assume that r is an imperfect estimate of population correlation?") is somewhat unrelated to the previous part. And what do you mean by "imperfect"? Do you mean biased?
Based on Google, this quote sure looks apocryphal.
I added some more discussion of how you do the interval stuff ... and we see maybe where the 0.15 might have come from at the end.
Assuming both sides are equally polarizing, wouldn't your MAP estimate of the undecided people be the for:against ratio?
Thanks Christoph_J
you say you have 10,000 samples - I take it this means you have *one* sample, and it has 10,000 items.  Or is it that you have *ten thousand samples* of unspecified number of units each?  I'll update my answer with some more details about how the equations work.
I consider myself to be reasonably good in structural equation modeling, but I cannot really see how to answer this question. What is a clinical formula? There are things absolutely trivial to you as a clinician or a biostatistician, but this is essentially a different language, and you need to provide a better translation.
I'm not sure I correctly understood the question, but it doesn't sound like it.
Just an idea but perhaps it would be useful to have a look at Hadley Wickham's implementation of Grammar of Graphics for R for some pointers? [http://ggplot2.org/](http://ggplot2.org/)
You can see examples of some of these here: http://www.statmethods.net/stats/rdiagnostics.html
Unfortunately, you won't be able to do this, unless you get really  lucky with your constraint matrix.  For example, consider a constraint matrix with two rows, one nonzero entry in the first row that constrains $x_1 = 0$ and two nonzero entries in the second row that constrain $x_2+x_3 = 1$.  Obviously, $x_1$ will have a different mean than at least one of $x_2$ and $x_3$, and unless you set the variances for $x_2$ and $x_3 = 0$, a different variance as well.
I know that, but nonetheless that's what they did.  They mention several times that they are creating synthetic data sets, eight in fact.  If you don't know what I'm talking about, maybe you should read the paper in the link provided by the OP - maybe I've missed something, but maybe you have too.  You seem to have leapt to a conclusion about what the problem is - testing for Poissonianity - but that isn't what the problem is.  The problem is testing whether the actual star data and the simulated star data come from the same dist'n.  No argument about whether this is a good approach, however!
Besides, Gauss-Markov ensure least squares be a minimum variance unbiased estimator among linear. May be linear estimators are not reazonable for some kind of distributions. That's what i want to understand.
@PatrickCaldon I doubt there is a "golden paper" on this because Bayesian hypothesis testing is formulated in a decision-theoretic framework (therefore it is too wide to be captured in a single paper). The book mentioned in MånsT's answer provides a good material, [Berger's books and talks](http://www.stat.duke.edu/~berger/) might also be of interest.
Pertaining to the Voting history problem, I would run a logistic regression with the complete data of the known voters and create a predictive model. Then using the model's equation, use the survey data of the unknown voting history people and plug in that data to get an estimated guess for the unknown voters.
"but because my sample is small i get insignificant results." This is serendipitiously correct, since given a large enough sample size, you would find statistically significant effects, even if they weren't significant significant. This is an issue of power. With $n=20$, you don't have enough power to detect even significant significant effect sizes.
See advice at e.g. http://meta.stats.stackexchange.com/questions/1479/how-to-ask-a-good-question-on-crossvalidated on explaining abbreviations. For example, I stop at ML, thinking that means "maximum likelihood", but is that meant here? Presumably you think it's obvious that it means "machine learning", and so it is to _some_ people here. Similarly, please explain GLM, NLL.
@backflip Again, you just need to test it making CV with larger tests, and I proposed resampling CV as the easiest solution to the problem with small overall number of objects. About bagging, yes it is the same. Indeed at first I thought BST is just a plain boosting, so I haven't mentioned bagging.
Are the two groups of equal size? Does the number of heart attacks stay the same when the sample increases.?
"*I used the quantile function to get the 75th quantile of the distribution.*" -- why did you do that?
Just to clarify for anyone unfamiliar with Wasserstein distances who reads this and gappy's answer: the L^2 Wasserstein distance (W2) is *not* the same as the Cramer-von Mises distance.
clustering gives you an idea of how the data tend to be grouped.  An unsupervised learning measure can determine how many clusters and how well separated they are with a measure such as a ratio of the within cluster variance to the between cluster variance. It can be a way to identify groups even though initially separate groups were not known to exist.  Supervised learning should not be compared with unsupervised learnign.  They address different problems.
Thank you for putting me on the right path.
+1 Don't throw away data because its an outlier. Find out *why* some data are outlying.
Well, it's roughly a 5% (you may want to look at a confidence interval for that value to see how wide it is) dip in the average seasonal differences, $y_t - y_{t-s}$ (for seasonal period $s$).
I think the solution here is @andrea's suggestion to use your continuous covariate as continuous, rather than categorizing it. Categorizing continuous covariates is commonly done; it may make interpretation easier for clinicians, but statistically it is always a poor choice. Your situation is an example of that. Although the context differs substantially, I discussed some reasons to not categorize continuous covariates here: [How to choose between ANOVA and ANCOVA in a designed experiment](http://stats.stackexchange.com/questions/24077//24080#24080) (especially below "**Update**").
I see, in general is there a model that I can use in R while I don't have to keep the initial data when I get some more new data and update the existing model?
@lokhearrt your accept rate is very low, here and on SO. Please do something about it otherwise you may find people won't answer your questions. Accepting a supplied answer indicates to others with similar problems that the answer worked and solve the problem. If the answers provided don't solved your problem, then engage with the providers of those answers to reach a solution. Be a good citizen here!
That's exactly how I'd do it. And you can compute the variance from the corresponding variance law. (Since this is essentially a routine bookwork problem, please add the `self-study` tag.)
It is considered somewhat impolite to simultaneously post (or nearly so, in your case) the same question in multiple forums. See the various meta sites for more details.
**Crosspost**: http://math.stackexchange.com/questions/61460/calculate-expectation-from-empirical-cdf
"*variables [...] are ordinal, and may or may not follow a normal distribution*" --- since 'ordinal' and 'normal' are mutually contradictory, you can guarantee that ordinal variables are *never* normal.
@AlecosPapadopoulos: Thank you for your answer. But I have a question which is that the 1 value in h stands for $e_t$ or is it just a convention to write 1? What if h =[0.5 0.2 -0.5] instead of having 1?
Because this question makes no sense without the background provided by the link--and not everyone will have the time or interest to sit through it--please explain the context and the meaning of the notation.
Danielle, do you know what type of statistical test you are hoping to run with this sample? Once we know that, I or someone else will be better able to assist you with your sample! Thanks!
Perhaps some modifications to the answer [here](http://stats.stackexchange.com/q/67907/4485) may work.
You're right that you need to use time-series methods to take auto-correlation into account. As well as ARIMA you might want to consider the Holt-Winters algorithm - it's simple & robust.
@NickAlger:It doesn't work under Ubuntu and Firefox.
You are asking about [tag:outlier] identification.  This is usually not conducted as a formal statistical test, in part because applying any such test is questionable: what, after all, brought a value to your attention in the first place? You certainly didn't randomly select it from your 4000 values! There are many other subtle issues involved, depending on whether you know beforehand how many outliers you want to detect, whether they come in clusters, whether they can be both high and low, etc.  [Entire books](http://www.amazon.com/dp/0471930946) have been written on the subject.
@Anne Is the "population mean" the *hypothetical, unknown* mean of the population being sampled or is it the *measured* mean of another population that has been exhaustively sampled?  Also, in what sense did you use the "sample mean" to estimate the population *standard deviation*?  Is that perhaps a typo?
I like definitions by "X is sth, that...". Taking the paragraph does not change anything - I still don't understand, that's why I decided to ask a direct question.
Vaguely: Yes. Non-informatively: No.
Of course, some Bayesians say that *all* statements about probability are Bayesian and subjective, but the authors of some statements about probability simply aren't aware of the subjective assumptions and interpretations they are making.
-1 Your "definition" appears to be incorrect, at least in one reading of it.  The $1-\\alpha$ CI is constructed to cover the *true* parameter with probability $1-\\alpha$.  It is not conditional on a particular model or method of fitting the parameters.  Perhaps I am misreading the definition, though: I take "this fitted model with this parameter value" to refer to your current *estimate* of the parameter.  If that's not how you intended it, perhaps you could clarify this point?
The question is too broad. There are books written about it and even then it all comes down to so many factors, tricks and tweaks that it is hardly feasible to have a reasonable answer, no matter how long...
That was a really good perception on the problem. But, there are advantages ?
It seems that all of @pualambagher 's questions are double-posted on math and stat-exchange.
@Behacad right, but that somewhere need not be a primary research article. It may have been in software, or decided upon in a working group or workshop for example, for a particular project and just stuck.
Thank you! If I may ask -- do you think that my R solution is valid, though?
@pedrofigueira: Could you explain in what way 'significance' is relevant here? To the OP: could you please provide some graphical description of the situation? This is much easier than to speculate using correlations.
Could you elaborate on why you think the lower bound must be $0$ for $X \\times Y$?  You stipulated independence, so it seems to me possible that $X$ and $Y$ could have opposite signs, leading to negative values of $Z$.
I don't think latent class analysis is what you want here, but I could be wrong, I am not expert on it. I think you want factor analysis
@chl Good point.  But since the principal components are linear combinations, it's straightforward (although sometimes a bit of a pain) to compose the fitted regression model (=one linear transformation) with the projection onto the components (=another linear transformation) to obtain an interpretable *linear* model involving all the original variables.  This is somewhat akin to orthogonalization techniques.  Note, too, that Srikant's latest proposals (sum or average the regressors) essentially approximate the principal eigenvector yet induce similar explanatory difficulties.
@RoeyAngel I suppose this is not possible with the `influence.ME` package. Unfortunately, I don't have a solution for this task.
Great answer. A good introduction to multiple imputation is Don Rubin (2009): Multiple Imputation for Nonresponse in Surveys.
You might want to look here:\\n\\nhttp://stats.stackexchange.com/questions/4498/whats-the-relation-between-hierarchical-models-neural-networks-graphical-mode
@whuber is saying that the value of the **slope** does **not** tell you anything about the strength of the association unless variables are standardized. See shabbychefs answer.
Hint: Where does ANOVA use the numerical codes you employ to distinguish the categories on the "x axis"?  (Answer: it doesn't.)
"We know the lion by his claw" (J. Bernoulli on Newton; likewise people on this site about the author here). Excellent.
Sanity check: in your question, class and group are equivalent, right? Also, by class distribution, you simply mean that you know what proportion of the persons should be in each group/class. If so, this has a Bayesian ring to it...
Possible instrumental variable here?
Do you have a reference for this formula? It is hard to understand from the information you have given. Expectation implies that there is a distribution over W - so what is it? What is varying in all of this?
It seemed so to me too because it's same both for Mauritius and Sweden. I had to follow the steps you suggested in `demography` code that is using log transformation and setting fertility rate at 0.05 for age 13 and 55 to get rid of it.
Great. You might consider marking the answer as accepted.
-1.96 is the 0.025-quantile of a standard normal distribution; +1.96 is the 0.975-quantile
What do you mean by 'agreement' when the units differ?
(+1) Thanks for following up and posting the resolution after you figured it out. Cheers.
Thanks. Not sure if I follow your comment about ordering of categorical variables. Can you please illustrate on "about the levels of each category and then use the corresponding numerical values as a single quantitative predictor variable instead of k−1 categorical predictors (i.e., dummies)."?
Wouldn't a test of means/proportions only give you a point estimate of whether the two methods gave the same average response for a given set of responses?  Couldn't that approach yield a result of "equal" even if the two methods were actually negatively correlated with one another?
@drnexus thanks for your comment.  i mean in terms of 'weights' or 'item values', not in terms of the number of items.  so, in the optimal solution, group 1 could have just one item of sufficient value, while the other 7 groups each have, say, 3.
Thanks. I've just managed to confuse myself a bit when thinking about the difference between multiplying two random variables (e.g. X and Y, leading to the calculation of E[XY]), versus multiplying two distributions (e.g. p(x) and p(z|x), divided of course by p(z), leading to the calculation of E[p(x|z)]).
Fix your density. Did you try drawing the likelihood? It might clarify some issues for you
Studying the two-sample Kolmogorov-Smirnov test (http://en.wikipedia.org/wiki/Kolmogorov-Smirnov_test#Two-sample_Kolmogorov.E2.80.93Smirnov_test) might inspire you. Your question is not 100% clear since we do not know how you compute pdfs and if you are after a probabilistic bound.
Confidence has a specific statistical meaning that I would not equate with "certainty". Also, what Pearson's correlation coefficient are you referring to? If you mean the population parameter $\\rho$, you're wrong. If you mean the sample statistic $r$ for future samples drawn randomly from the same population, that should be specified.
It sounds like the only reason you use beta is because its conjugate? uniform over the interval is fine if you have a reasonable amount of data ($\\alpha=\\beta=1$) - then you just have an incomplete beta function to evaluate, most software packages can do this for you.  Plus numerical integration is very quick because you have a short interval $[0.5,1]$ - uniform grid sampling would work well.
**Related**: http://stats.stackexchange.com/questions/14914/how-to-test-the-autocorrelation-of-the-residuals
@MichaelChernick, that happens sometimes. I just finished reading your answer, +1.
Can you do a head() output of your current data. What methods are you currently using for the forecasts?
logrank is a special case of the Cox model so no need for it, and it won't allow you to adjust for continuous covariates as the Cox model does.  The Cox model also provides several ways to handle ties.