tttjjj/elt_bench huggingface dataset:
I used Wassermann's book to supplement the official one that every other instructor in a ten-section "Intro to Stats for Engineers" class was obliged to use, and liked it way better than the required textbook. It swoops through statistics at an amazing pace by omitting all the proofs, in the hope that a mathematically savvy student will either see through it, or consult more rigorous books.
That's not a justification of convenience sample, of course. American Association for Public Opinion Research, the professional organization encompassing researchers and practitioners from hard core mathematical statisticians in Census Bureau to one-day pollsters and market researchers has tried looking into non-probability samples -- see the [report](http://www.aapor.org/AM/Template.cfm?Section=Reports1&Template=/CM/ContentDisplay.cfm&ContentID=5963) published this year which basically failed to give any definitive recommendations (other than that convenience samples are generally terrible).
That's why I said that the distribution should be symmetric for convergence to be rapid.
It's a little late, but I finally got time to sit down and re-create your argument. The key was "multinomial coefficient". I had tried figuring it out using plain old binomial coefficients and I was getting all balled up. Thanks again for a nice answer.
$U$ should work out to be an exponential random variable and $V$ is the distribution of $\\cos \\Theta$ for $\\Theta \\sim U[0,2\\pi)$.
Thank you @gung, I really appreciate your efforts. It makes things clearer.
No apology necessary, Rene, I was just trying to get the meaning straight.
Although throwing in three more points *and weighting them* (*a la* Glen_b's answer) could create such a fit, interpreting any of the statistical output would be problematic: some adjustments would be needed.
It seems to me that a covariate is anything added to a given model (linear or generalized linear model) that we want to control or that is to be accounted for. Whether this variable is continuous or not does not really matter. In ANCOVA, though, the covariate is usually continuous (e.g., difference between two groups with pre-post measurements), and we are interested in showing that it is not interacting with the grouping variable.
Anyhow I tried to do the loop in your way, by estimating a RW model in DLM form, but it gave me the same result cause V=0.
I disagree with @ACD. A moderator is a variable that *interacts with* the predictor of interest and in this way influences the predictor's relationship with the DV. In a purely statistical sense (that is, ignoring any notions of causality), moderation is synonymous with interaction. A covariate or "control variable," on the other hand, *does not* interact with the predictor of interest. So, no, the difference between a moderator and a control variable is *not* merely semantic. The `z` variable from @ACD's previous comment would be a covariate and not a moderator, since there is no interaction.
¿Why do you need "a rule of thumb"? Those are mainly, anyway, just meant as some yardstick people can use until they get experience!  Just do some simulations in R and find out four yourself ...  Anyway, in Poisson regression the dependent variable $Y$ is a count, so a numeric variable, so as a start you could use the same rules you would apply for ordinary linear regression... The main difference between those two models is __not__ in the distributional assumption (Normal versus Poisson), but in the structure of the expectation --- linear versus multiplicative.
What do you mean by 3D data? What do you mean by "clearly a normal distribution"?  It looks more like you have 2D data, $f(2,2)=0.3,f(2,3)=0.5$, etc and you want to fit a two-dimensional normal distribution to it, so that you also have values for $f(2.1,3.6)$, etc that are _not_ in your data set.
The story is somewhat based on the real lives of some very notable mathematicians, I believe Stone and Kalman, who developed these gaming strategies.
Is my question ill-conditioned or something like that? Thanks
Argh, no, sorry, that appendix provides a hint how the covariance between two d values can be derived when you have multiple outcomes (what Gleser & Olkin, 2009, call 'multiple-enpoint' case). I don't know of a reference that suggests how that covariance in the case above (what Gleser & Olkin, 2009, call the 'multiple-treatment' case) was derived. Sorry!
"at what value does the slope begin to plateau?" this is great. I'll try and answer this question.
Is this a homework exercise or are you actually trying to generate random numbers in this way? If it is homework, please add the `homework` tag; if it's not homework my advice would be to put away the table and use a computer
I just posted a blog about this problem on http://aheblog.com/2014/01/08/solution-for-trap-66-postcondition-violated/. Hope this helps. Pepijn
Hi. I don't think is the same question. In my previous one i was asking how to model the problem. (For what I really aprecciate your help.) But in this one, I am asking how to find confidence interval with the model correctly stated.
Well, you could write you own code ... What exactly are you asking?
Thank you for your answer. I guess that's my fall-back strategy, but right now I find it hard to believe that there is a regression that takes that long to compute. I also find it surprisingly hard to find examples on different competing risk analysis. I've read Melanie Pintilie's book on Competing Risks with a few helpful examples but not really the depth I was hoping for. I also stumbled upon Geskus article in Biometrics 2011 where he proposes weighting coxph() data but since I'm an MD I found the article a little hard to penetrate. There is a striking lack of examples...
Also, if I want to generate the exponential distribution outside ks.test, how is that done? The `quantile` argument isn't very clear to me?
Please clarify your notation and define all symbols you use in your formulae.
Duplicate of [this question] on Math.SE which has received two down votes.
In papers, when interaction effects aren't significant I still mention in passing that I looked for them to stop reviewers asking about interaction effects. This is obviously different from "accepting H_0" and making a big thing about no effect.
thanks. I suspected that HC/HAC could _not_ be used to 'magically' infer the proper form of the weighting matrix to use in a weighted LS. As such, it becomes a harder sell, I think, to a lay-person, because the 'value add' of HC or HAC is that it improves the hypothesis test, leaving the estimated model untouched.
Good idea suggesting the qqplot. However, I think that your explanation of the technique is a little vague/hard to understand. Can you provide some exemplary R-code ? This would increase the value of the answer drastically.
I believe the exponent should be an m, not a k as you have. I have included another step that makes this more clear. I may be missing something, or misunderstanding your question, so please clarify if you still don't agree.
I got it to work by defining a new object called glmFuncs using the following code: `glmFuncs=lmFuncs

glmFuncs$fit=function (x, y, first, last, ...) 
{
  tmp <- as.data.frame(x)
  tmp$y <- y
  glm(y ~ ., data = tmp, family=binomial(link='logit'))}`. It is doing exactly what I want it to do. Thank you soooo much.
@pkofod Yes, I've done that. The ratio is sample size-dependent and seemingly dependent on `Var(y)`, and maybe other things.
Glen: right, I removed normality, but we need at least the independency assumption. Cheers!
Actually, this code makes some assumptions about interleaving of ys and zs. Could be fixed, if there is any interest.
Interesting point! Thank you for your reply!
A good starting point (not that it is easy) is the book: [Bayesian Nonparametrics](http://www.amazon.co.uk/Nonparametrics-Cambridge-Statistical-Probabilistic-Mathematics/dp/0521513464). In particular Ch. 2 and Ch. 3. A more technical book is [Bayesian nonparametrics](http://www.amazon.co.uk/Bayesian-Nonparametrics-Springer-Statistics-Ghosh/dp/0387955372/ref=sr_1_6?s=books&ie=UTF8&qid=1390903418&sr=1-6&keywords=bayesian+nonparametrics) (different book, same title).
@user603 actually they are lambda, sigma, w power of 2 however I don't know that parameters because I don't have any idea about algorithm.
This is k-means clustering, not kNN.
[Corrgrams](http://www.datavis.ca/papers/corrgram.pdf)? [Ellipses](http://dx.doi.org/10.1214/12-sts402)?
Good idea.  I tried this first (probably while you were posting this answer :-).  The problem is that the other loadings can jump around.  To fix this, base the sign choice on the largest loading. Still no dice: the loadings can still jump.  The trick is at each time to choose the orientation that creates the least disturbance in the *vector* of loadings from the previous time.
R can handle different starting dates, you just need to use the Surv(time, time2) specification.
Correction: $\\gamma '(\\theta_i )= \\frac{1}{\\theta_i(1-\\theta_i)} \\neq E[Y_i] = \\theta_i$.
This is a functionally similar question, http://stats.stackexchange.com/q/29176/1036
I made a *lot* of edits; I hope they reflect what you want; but it still isn't clear what exactly your question is.

Also, welcome to the site!
Along the lines of $\\sum_{j=1}^{n}(X_{j}-\\bar X)(X_{j}-\\bar X)^T
=\\sum_{j=1}^{n} (X_{j}X_{j}^T-X_{j}\\bar X^T-\\bar X X_{j}^T+\\bar X \\bar X^T)$
and $\\sum_{j=1}^{n}\\bar X X_{j}^T=\\bar X\\sum_{j=1}^{n} X_{j}^T=n\\bar X\\bar X^T = \\frac{1}{n} X^T1 (X^T1)^T$ etc
To those voting to close: if this were a pure Matlab question we would ship it off to SO in a moment.  But before you can write that Matlab code, it seems to me you have a rather difficult statistical question to answer first! Exactly how do you go about fitting a model to a set of data when that model is given only implicitly in terms of differential equations and not explicitly as a parameterized set of functions?
Usually I would use `lme` from the `nlme` package and specify an autocorrelation structure (e.g., using `corARMA`). However, with 4 time points you probably won't be able to fit the autocorrelation. So I would first fit a mixed effects model without autocorrelation structure (probably using package lme4, but you can also use `lme`) and test autocorrelation of the residuals using the Durbin-Watson test, just to be on the safe side. I aggree that plant type (and possibly time) is the fixed effect and box is the random effect.
If you have to understand the results quickly, the books won't help that much - to fully understand what a GEE model does (and what you might be doing wrong or right) isn't a simple task. Do the SPSS manuals offer any help on the output itself?
I am concerned that you may not be using the most applicable references available: the vignettes that describe the [mob](http://cran.r-project.org/web/packages/party/vignettes/MOB.pdf) and [ctree](http://cran.r-project.org/web/packages/party/vignettes/party.pdf) functions. The graphs on pages 9 and 12 of "party with the mob" seem pretty descriptive. Please note we don't have access to your data, though, so it's hard to see exactly what you're looking at with the "CCA" data set. For posting here I recommend you create an example from a data set included in "party."
@whuber: if the data are continuously distributed (which they might be, depending on whether the original poster really means "general" or "generalized" LMM) then the probability density has an implicit "delta-x" term in it, which is indeed affected by changing units. See also <http://emdbolker.wikidot.com/faq>
I see seven questions here. Depending on the level of the answer desired, I can suggest (close to) seven different textbooks, each one of which will answer all seven of these questions and do them justice. But, in each case, it will take nearly the entire book to do so. That said, you might browse our [tag:regression] tag to start. :-)
Have you tried to compute $\\mbox{var}(X_t)$? What do you get?
thanks with a bit of extra coding I think m out of n is what I'm looking for, never realised there was so much to the bootstrap.
One alternative would be to a bootstrap/simulation. Generate a large number of 10x10 matrices where your null is true; define some measure of how far they are from your null; see how many are as far or farther from the null as the matrix you have.
`xtreg` with the `fe` option is equivalent to `regress` with firm indicator variables. This only works with linear models, but that is what the OP asked for.
I also asked Terry Therneau, he gave me same answer. Yes, the value given in the output (5e-07) is the last value written in  'fit$history$frailty$history'. But why is it different, any kind of algorithmic problem or others?
Well, $X$ and ... $X$ are identically distributed.
Thanks for all your comments ! I am using R so if I understood well, with a factor R will do everything, I don't need to do anything ! Perfect ! Tanks once more !
What is "$T$"?  One can make a fair guess, because the statistic looks like a ratio of statistics each of which is (a) related to a variance and (b) divided by a count: it *must* be an $F$ statistic.  That suggests you investigate situations where $F$ statistics are used and learn (if you don't already know) what hypothesis an $F$ test evaluates.
Ok, since it's context-dependent, I'll create later a new question about the construction of the conditional distribution in the context that I'm working on. I'll let you know when I do it. Thanks.
Can people be considered indepent?
If you hadn't to adjust for multiplicity, graphs only for the best results are highly discouraged.
May be round it of to the closest integer?
I'm not an expert on the topic, but from what I have heard from experts in the field is that these tests don't work for reviews of diagnostic accuracy testing and that new methods are currently being developed. I would suggest you check out the guidance from the Cochrane Collaboration. Will update my post if I find anything conclusive.
Just a quick point: You definitely can **not** write $9.83 < \\mu < 46.17$. That implies you know *for sure* that $\\mu$ lies in that interval. What you actually have is an interval that *might* span $\\mu$ with some probability. Note I carefully did **not** say that $\\mu$ lies in the interval with some probability. This is because $\\mu$ is not random, so the probability of it existing within the interval is either 0 or 1. But the interval **is** random, so the probability of it spanning $\\mu$ is a meaningful construct.
Actually the $\\alpha$-stable distributions share the same tail behaviour. I would like to use this way because i think that this is the better way for a straightforward measure of the $\\alpha$ parameter in a generic $\\alpha$-stable distribution.
The answer depends on the distribution into these categories on the population. Are they equiprobable? If not, do you know the actual probabilities or can give reasonable estimates? And is this a practical problem or theoretical?
A key phrase here is, modulo punctuation, "clearly non-stationary". Stationarity or its lack is a property of a process and need not imply that segments of a series all look similar in all statistical properties. Conversely, explain what you mean by "clearly non-stationary".
I am a little confused about this model since I don't know what error distribution would be sensible - you'd need to do something strange, or transform the response, to keep the $y \\in (0,x_1]$ constraint. Anyway, I doubt this model is pre-implemented in any software package so you'll probably have to "make your own". If you know the distribution of $u$, you can calculate the (log) density of $y|x_1,...,x_n$ and maximize it as a function of $\\beta_0, ...., \\beta_n$ to get the maximum likelihood estimate.
Thank you: that really is 3D! It is a histogram in which the vectors $(r,g,b)$ have been binned and counted. Since chi-squared statistics apply just as well to such objects as the usual histograms, and you seem to be seeking alternatives, I would guess you have some particular kinds of comparisons in mind. What are they?  What are you trying to find out when you compare two such image histograms?
@user603 Around 10k variables and 100 million points. I somehow doubt that a 20k variable dataset can have the condition number computed in seconds, considering the harddrive can only read on the order of 100 mbs a second.
@charles: what have you tried and where are you stuck? **Hint**: There is a great amount of symmetry in this problem.
What is the model? Usually there are assumptions, i.e. properties of F2 or F1 which you can check. Failure to have these properties may lead to conclusion that the contamination is not eradicated. Without more details I doubt that anyone could give you more precise answer.
I do know about K-fold cross validation. My question was about feature selection. Is it alright to use the whole data-set for feature selection?
If you're happy with proportional hazards assumptions, you don't need to use clustering in this way; contributions to the partial likelihood at each time point only reflect people in the risk set at that time point, and (in your case) what covariates they had at that time point. Robust variance estimators (i.e. `, robust` in Stata) can still be useful though; they give you standard errors that appropriately summarize variability in your point estimates, regardless of whether the hazard ratio is truly log-linear in the covariates.
I think the decision not to adjust alpha with orthogonal contrasts is a convention. If you want to maintain family-wise alpha to .05 for the set of orthogonal contrasts, you would still need to adjust alpha.
I was thinking about writing a similar question about how people evaluate introduction to statistics textbooks. I wonder though if it is too broad to be answerable. Perhaps someone bold enough can find similarities through lists of great stats books (we have [one here](http://stats.stackexchange.com/q/363/1036) on this site). Certainly Tufte's book is what first comes to my mind, but I would be at a loss if you forced me to put my finger on why that is so in any cogent way.
@whuber thanks for the link. I'm particularly interested in the effects package. What is exactly the interpretation of `allEffects()`function?
Let me put it this way, I am working on safety Systems. I wanna calculate each term in Bayes'rule: P(Temp|Pressure)= P(Pressure|Temp)* P(Temp) / P(Pressure). I wanna use MATLAB. This Bayesian Network is assumed to be used to predict accidents in as Nuclear, Natural gas, etc. How to calculate the prior, likelihood and evidence
Hmm, can this happen? It may be the case after all, that the variance "weakly dominates" the rate of convergence of MSE.
Your question might be clearer if you defined your abbreviated terms! ROC is common knowledge, but I doubt DPD and WOE are.
Related question: http://stats.stackexchange.com/q/5135/5503
Could you tell us *why* you want to dichotomize the factor scores?  (It's worth contemplating this question because in many cases such an operation--which loses information--is not really needed.)
Hi ! Your first suggestion didn't change the result. For the second one, I don't think it will solve the problem. Indeed, the different conditions imposed on the orders are necessary for my work and I really need to understand why auto.arima doesn't choose the best model. Thanks for your help.
I would think that other methods would be similarly limited by the data, if not more so.
Are $X$ and $Y$ independent? Are the _variances_ of $X$ and $Y$ known or unknown?
I do not see why an unbiased *non-negative* estimator should exist or be useful. If one did exist, then its expectation when $\\mu=0$ must equal $0$, but since all finite configurations of points have finite density even when $\\mu=0$, that would imply that the estimator is constantly zero. Thus, *any unbiased estimator of $\\mu$ must have some chance of giving negative values.* That seems like a high price to pay for unbiasedness.
Method is interesting, but currently software implementation is more crucial.
I especially like the comment on *statistical meaning* being lost far earlier.
Trying to fill a few values from a large dataset is hazardous enough. Trying to fill most of a large data set from a handful of known values is going to give extremely uncertain results. Nonetheless, the appropriate distribution to use is going to be highly dependent on what these numbers you have actually represent. Using a concrete example will get you more informed answers.
Regression is used to explain how a response is related to a set of covariates It puts a smooth function through the data to separate signal from noise.  It also allows interpolation between data points (possibly also extrapolation, although extrapolation far outside the range of the data is dangerous).  Generally regression is different from classification.  However in the case of probit regression and logistic regression prediction may be viewed as classification.  So including them in the list may be okay.
Is data.dat your dataset? Then you can just scan and call it Y or X or whatever. Or just import it to your workspace. But t is a time trend, so it has to be a sequence, t<-seq(1,30). You are regressing your dependent variable against a time trend to find the relationship between them. The predicted variable will be your independent variable, which you are scanning in data.dat. Right?
Do you have some typesetting errors here? For example, you have $i$ subscripting itself and I don't know what $k = $1(1)$k$ means.
Ah!!! that is too bad! How can I use Ime4 with t.test?
Geometric distribution if about the number of Bernoulli trials needed to get one success. The problem of my statement is getting one or more.
In short, your primary worry is about inflated variance estimates. See http://stats.stackexchange.com/questions/51386/colinearity-between-variables
@richiemorrisroe: Is this the paper, "[The Bootstrap Discovery Behaviour (BDB): a new outlook on usability evaluation](http://www.springerlink.com/content/9x1188631072v116/abstract/)"? I've found what appears to be an older version of the research here in PDF, "[Cost of a Usability Evalution: Bootstrap Discovery Behaviour Model](http://www.iadisportal.org/digital-library/mdownload/cost-of-a-usability-evalution-bootstrap-discovery-behaviour-model)".
The value of this bound becomes apparent after you note $\\sum_{x,y}p(x)p(y)=1$ and (since $k \\ge 0$) that $e^k \\ge 1$.
Because MANOVA explicitly accounts for any covariances among its dependent variables, since it is _multivariate_ by definition. Univariate ANOVA cannot cope (without special corrections). Because the "difference variables" are all interrelated, their nonzero covariances automatically mean their nonequal variances, which is contra ANOVA assumption of homogeneity of variance across levels of a factor.
@amoeba While one could organize it so it is, it doesn't have to be lower triangular for the argument to work - it's a feature of the Cholesky but it's not required for the result to work.
+1, Welcome to the site, @Marzieh. I took the liberty of sprucing up your $\\LaTeX$ a little bit. Make sure you still like it.
Variance and standard deviation are both useful. For example, with independent variables, the variance of their sum is the sum of their variances. Simple properties of variances allow us to partition variances in ANOVA for example; things are more complex if you try to work from standard deviations (which don't partition the same way; their squares do). But on the other hand, variances aren't in the original units, the way standard deviations are, so standard deviations, too, are important in their own right.
(multiple comments since too long)

Perhaps, the notation is overloaded. Maybe, $P(Z_{d,w}=z) = P(Z_{d,w}=z, Z_{d,w}\\neq \\theta_B)$, at first.