tttjjj/elt_bench huggingface dataset:
Breakdown isn't an issue for a descriptive statistic of an entire population.
(+1) Now, the problem is that the OP didn't indicate how many variables enter the model, because in case they are numerous it might be better to do both shrinkage and variable selection, through e.g. the *elasticnet* criterion (which is combination of Lasso and Ridge penalties).
Is there anything you can add that would make this question less vague?  Such as the purpose or basis of the desired comparisons?
If you use a regular grid and the more frequent marks outnumber the less frequent ones, I don't see how that "overwhelms" and "masks" things. Could you elaborate a bit? Are you trying to replace each quadrat's contents with only the most common mark, or something like that?
@toomuchpj: Hi there! Could you possibly elaborate? Is the following what you are suggesting: `m1 = lmer(choice ~ msg*cost*rank + (rank|id)*(cost|id), data=df)`
@whuber: thank you for your reply. I have tried to make the context from which the time course data is obtained more clear.
The message is clear and accurate: you are setting up an essentially singular model (at least in double-precision floating point accuracy).  Check this by inspecting the output of `svd(g(iniVal, cbind(x1, x3)))$d` before calling `gmm`.
+1 for the helpful information, Dave.
I agree with most of the post, but i would say AI most of the time does not try to create intelligent agents (what is intelligence, anyway?), but rational agents. By rational it is meant "optimal given the available knowledge about the world". Although admittedly the ultimate goal is something like a general problem solver.
Thanks for posting this, very insightful on how to approach issues like this. +1!
The question reads *an increased risk of death* - the effect estimates need to not only be significant, but significant and positive.
Please, register your original account and merge it with this one; follow the instructions on our [Help Center](http://stats.stackexchange.com/help/merging-accounts). Next, I would advise to merge your three different replies into one single answer.
Thanks Gilbert. That was exactly the kind of answer that I was looking for -clear and precise. Do you have, or know any reference that I can include in my document concerning these comments? I have spent a couple of weeks looking for one -I have not been able to find anything discussing these points in the (clear) way that you just did.
Definitely think about using a principal component analysis to reduce teh dimensionality of your problem.
a sine wave is not stationary but unless I missed something, I don't think it has a unit root (which is a particular kind of nonstationarity). But in any case, failure to find a unit root doesn't imply stationarity.
What's binary survival data? Do you mean repeated binary outcomes? (In which case `glmer()` would be appropriate, with `family=binomial`. "Survival data" outcomes are usually time-to-event, which wouldn't be binary.
I think there is exponential smoothing in Pandas. It would probably be easiest to tag a SO question with Pandas (they are very active over there).
Suppose I had each observation on a separate sheet of loose-leaf paper and as I handed you the stack, I slipped, and the sheets came flying out in all directions as they settled to the floor. It would be a shame if that destroyed the validity of the test you were hoping to perform on those data. If your observations are exchangeable and you were applying a test based on that, you'd comfort me and tell me not to worry while helping me collect up the papers off the floor. If not, and the data collection was especially expensive, I might need to run for my life.
Using the information that what I was looking for was the negative binomial distribution I was able to find the correct function in R: qnbinom(c(.025,.975),1,1/104) - Thanks a bunch!
In a less technical and more general note: if you only have two classes and two schools, there's not much point to building a mixed model, you should probably just use a regular generalized linear model with school and classroom within school as fixed effects.  See http://glmm.wikidot.com/faq ...
This has been answered by my friend already. Thank you for replies and answers.
What is your loss function?
Thanks the difference you point out is correct, I just read about this in some other material where the distinction seems to be labeled as "static forecasting" versus "dynamic forecasting". I'll edit my post to add on a few of the related details that I've read.
Sorry for misused terms. With "value of card attributes" I meant something like "attack has the value 3" not the estimated money-value of "attack=3".
I'm a little unsure of the notation, but I think $\\prod_{k=t}^{j}\\left(1-h(k|\\mathbf{x})\\right)$ corresponds to $S(t)/S(u)\\ dt$.  Notice how the former is automatically normalized to $1$ for $j\\lt t$ and, similarly, that $S(t)/S(u)\\ dt = dt$ at $u=t$.  Therefore the answer to your question must be lurking in the definition of $h$.  Could you provide that definition so that this pair of questions can stand on their own?
There are multiple definitions of the skew-$t$ distribution so you need to start by specifying which one you are using. However, my suggestion is the same as for your previous question - skip the EM algorithm and use numerical maximization of the likelihood.
Or what about when $Y_1$ is positively correlated with both $Y_2$ and $Y_3$, yet $Y_2$ and $Y_3$ are negatively correlated (which is perfectly possible) -- what groups do you make then?
@ Daniel Johnson You cannot have an "all-zero" dependent variable - the OLS estimator would just produce a vector of zeros as estimates.  (cc:@whuber)
Doug hasn't ported GLMMs yet.  If someone wants to help with that I'm sure he would be happy ...
I understand what you mean (and I think it makes sense), but I must also say that your description *is* quite confusing. I don't know of a suitable reference, but you should definitely try to improve your wording.
Even if he reduces it to a categorical variable, he will have 5 points in one category, not sure it will yield interesting results.
Oh I used the default acf function in R and the lags didn't seem to be significant. In the book Introductory Time Series with R the author says that usually return data tend to be uncorrelated but the squared return is correlated. If this is true how can we use AR terms or MA terms? thanks your illustration made things clearer.
Thank you!  Am I correct in counting 16 parameters fitted in your model?  I ask this because you emphasize simplicity at the outset.  This indicates I must be misinterpreting the output, because a 16-parameter model for 36 data values would be far from simple. Yet it is otherwise unclear what the extra 12 coefficients (for I~S000020df etc.) mean.
In terms of dealing with collinearity, I am not aware of any tool for random forest which can improve on that. The importance measure (sort of) tells you how much % of the output this variable can predict. In such terms one variable is no better than other, you can't separate them. There are many ways to deal with collinearity, but I don't know about anything specific to random forests unfortunately...
As I stated above, I don't have access to the joint distribution of T and A. I may have, however, the values of predictors for both. Still, I'd like to estimate $G$ using the simplest model first before integrating over dependent variables. It seems to me my problem is more that I've assumed $T=G+A$ means $G=T-A$ (false), as @whuber points out.
Cheater! /10char :)
@Henrik you read my mind, all indivs did see all levels of `f2`. But how does it make sense to have slopes at all? All I have are categorical factors, no continuous predictors...
@macro See my new question. Something else is happening. I gave SAS code and results.
The intuition is correct. But as for constructing confidence intervals the answer is not so simple. If your problem is such that you can accept that the coefficient estimators distribution is  normal or approximately so, then their sum will also be ~normal, and then you can obtain a 95% confidence interval in the usual way.
Almost all null hypotheses are (pretty obviously) false. But it's a bit like a footrace where even after examining the pictures in the photo-finish you *still* can't discern a difference. You might *logically* know that some single individual *must* have won, but you have no basis on which to award one of them first prize. You can't separate them; it's a draw.
Extending your answer: If TP=0 (as in both cases), recall is 1, since the method has discovered all of none true positives; precision is 0 if there is any FP and 1 otherwise.
@Wayne Yes, you can tell what I'm reacting to. But although kriging has been used as "snake oil" by consultants, it has a lot going for it, including a theory of "change of support" to compare data obtained from (say) tiny samples of a medium to data obtained from much larger portions of that medium. Kriging ultimately is at the bottom of the most sophisticated spatio-temporal modeling today. It is also a useful way to evaluate alternative proposals: *e.g.,* many spatial interpolators are linear (or can be linearized), so it's fair to compare their estimation variance to that of kriging.
What a great answer!! Thanks @PeterFlom
+1 for verbosity.
Please note that reposting across SE fora is discouraged. The correct thing to do if you post in one place but think it belongs in another is to flag it and ask for it to be moved.
could you change the title, please? "Joint Gaussian" is not what you mean. And if this is an homework, please tag it as such.
@jth Since you maintain the two questions are sufficiently different not to be considered duplicates, please feel free to move your answer to the other one over to here. (I apologize for not originally appreciating the difference.) The new note, I believe, is incorrect in supposing the questions are mathematically different--@Michael Chernick points out they are basically the same--but the emphasis on *interpretation* establishes a valid reason to keep the threads separate.
Please tell us more specifically what calculation "information value" refers to: there does not seem to be a standardized quantitative meaning for that term that all readers will understand in the same way. When you edit your question, please also provide more context to help us understand what kind of analysis you are discussing and what you are using the "cutoff point" for.
This seems like a question from functional data analysis. Did you look at the standard [FDA reference](http://books.google.com/books/about/Functional_data_analysis.html?id=mU3dop5wY_4C)?
Thanks. There are a few papers in the area of trajectory groups. http://scholar.harvard.edu/files/sampson/files/2005_crim_laub.pdf and http://onlinelibrary.wiley.com/doi/10.1111/j.1745-9125.2005.00026.x/abstract
Why not?  $R^2$ measures the percentage of the variance of the target variable that's "explained" by the model on the right hand side; it doesn't matter what form that model takes.
**An aside**: I would urge you to please be *extremely* careful with your study design. Involve a qualified statistician in the design *before* you collect any data. In goniometer studies in which I've been involved in the past (and only after data were collected), the single largest source of variation is *the person taking the measurements* and this variation appears to swamp all others in many cases. Cheers.
@Xi'an: Thanks! I was aware of the difference between discrete and continuous sample space, when I posted the question. So in my formulation, there are density functions for the target and proposal distributions, while it is probability for the acceptance distribution. I fail to see what are not correct. I wonder if you could point them out?
I'm not sure what you mean by continuous variables as residuals. I have heard the belief that dichotomizing could mitigate noisiness, though I don't know good references backing this up. I'm against dichotomizing regardless. Assuming a linear relation, if you get a significant result with dichotomies, but a nonsignificant result using the continuous variable, you have almost assuredly capitalized on chance in choosing the split. Also, making a dichotomy will induce an artifact (usually spurious significance) in other variables in a model that are correlated with the dichotomized variable.
It's worth noting that, if you used PCA, you *do not* have "factors", you have *principal components*.
See http://meta.stats.stackexchange.com/questions/252/should-we-allow-more-computing-questions the current consensus is that this is an appropriate question, if you disagree chime in at meta, don't torment the new question asker.
onestop, that's a good idea! Definitively valid with a clear conclusion to be drawn from the results. I don't have the final data until now, but hopefully I will have enough fluctuation between classes. If not, I will have a very small sample size in groups (c) and (d) and inference will be difficult. But that's the best strategy so far... :)
The first eigenvector has $N$ free parameters. The orthogonality condition restricts the second eigenvector to the hyperspace orthogonal to the first, needing only $N-1$ parameters. Each successive eigenvector needs one less parameter than the preceding one did. At the limit of $N$ eigenvectors you discard $\\sigma_r^2$ (because it's now zero), giving $N+(N-1)+\\cdots+1$ = $N(N+1)/2$ parameters *in toto,* in agreement with your first parameter count.
Basically, you’re telling me that the function is not continuous and that’s the reason I cannot take the derivate? I knew I’ve missed the obvious. Thanks so much, that helped a lot.
:whuber the identified one time pulses play no role in the forecast to wit:
I know you've seen this link before, but *http://stats.stackexchange.com/questions/6493/mildly-informative-prior-distributions-for-scale-parameters* can help; many of the mentioned distributions can be parameterized by a mean and variance which are transformed to the "usual" parameters (as you did with the Gamma above).  As to why you get many $\\alpha < 1$, it's because the tails of your prior on $\\sigma$, when you integrate out `m` and `d`, are too fat for $\\sigma$ to have a finite variance.  You might also be able to fix it by bounding $d$ above and below (JAGS lets you do this.)
@gina_l You would want a theoretical rationale for combining the variables as well.
Did you intend the "in *formative*" pun? ;) One even has further distinctions between "formative" causal indicators and composite indicators ([Bollen & Bauldry, 2011](http://dx.doi.org/10.1037/a0024448)). The terminology is not the hard part IMO, making a reasonable argument for which ways the arrows should be going is the hard part.
There is an example (in progress) using Tikz over at the Tex site as well, http://tex.stackexchange.com/q/40159/8217.
I don't believe that would make sense. A likelihood ratio is only meaningful for comparing likelihoods derived from the same model. If you could just pick any two models to compare, you would be able to game the test and get whatever results you desired. You can, however, compare nested models; i.e., where one model is a restriction on another. For example, an exponential model in event history is equivalent to a Weibull with `p=1`.
First, do you know the distribution of $S = \\sum_i X_i$. If so, can you find $\\mathbb E(1/S)$ (*not* $1/\\mathbb E S)$? How about $\\mathbb E(1/S^2)$? [This answer](http://math.stackexchange.com/a/28836/7003) on math.SE may also be of some use if you get stuck.
I actually have a LOT of parameters, so I think I will go with adjusted-$R^2$ when reporting my model. Thanks!
You may want to check this site for outliers.  Quite a bit has been written and some should be meaningful to you.http://stats.stackexchange.com/questions/16709/how-to-deal-with-outliers
Actually I wanted to mean whether normalization by the term data preprocessing..
Oops, didn't know crossposting was discouraged. I'll avoid it in the future.
Look into (either here or on the web generally) `AIC`, `BIC`, `SBC` and related terms that those will identify. These are all penalized measures of fit - where the improvement from adding free parameters is penalized for added terms.
As I understand statistical significance in quotation means a probability to reject true H0, which is fixed e.g. 5% and I don't understand how it relates the significance of effects. In treatment example I have several versions what 'more significant' means: 1) the effect is less likely to be zero 2) the magnitude of the effect is bigger 3) the width of confidence interval is smaller. I should have started from the question that I don't understand what is significance here. So I would be very grateful if you can tell me that. Thank you very much!
If there ever was a list of FAQ to xts, this would score highly. Please look around here, search for `[r] xts` and peruse the r-sig-finance archives.
Thanks again for the reply.  I have seen another article using $L(\\theta|X)$; I guess the use of the notation is confusing to many.  I came across another thread [here](http://stats.stackexchange.com/questions/30825/what-is-the-meaning-of-the-semicolon-in-fx-theta?rq=1) discussing this notation.
I'm sorry, I'm not clear on what you mean by error propagation... are you stacking models on top of models?  Are your standard errors of predictions typically derived from the difference between $y$ and the model predicted y ($\\hat{y}$), i.e. the residuals?
I am trying use the prior cluster information (i.e. group ) in my hand in  cluster analysis (posterior information) - the cluster can be any number. Assumtion here is that prior cluster information can guide the clustering particularly in a confusion situation
Thanks, that is interesting. I am not sure if it'll help me in this case, but I'll look into it.
I just found a link that explains it a little bit: [link](http://en.wikipedia.org/wiki/L-estimator)
Because I'm new here... does that count?
This is a great answer. I would further complicate the issue by referencing Gelman and Hill's attack against the "fixed" and "random" terminology itself in their book on hierarchical modeling, where they outline the four (!) or so distinct usages of the terms "fixed effect" and "random effect" which the authors have discovered in the literature.
@Scortchi - yes. It has yet other interpretations as well; most broadly it's a test of $P(X<Y) \\neq \\frac{1}{2}$. On the other hand, people should be aware of its non-transitivity property.
@whuber: one thing I still don't understand is the connection between Glen_b's example (that fully answers my original question) and your linked above [discussion](http://stats.stackexchange.com/questions/14500/how-can-a-regression-be-significant-yet-all-predictors-be-non-significant) of significant F and non-significant t in linear regression due to multicollinearity... Would be very grateful if you could elaborate or provide some hints.
"related" is not good enough. To use an F-test to compare them, they have to have *exactly the same response* and *nested* sets of predictors.
Do you have access to the ASA journal Chance?  Seems to me there was a relevant article appearing in the last year or so, whether about hockey or another sport.
@Roah Did you try the answer? Because that's exactly what it will give you (you won't see the coefficients, but why do you care about the internals of the function?
Have you read this [documentation for the changepoint package](http://www.lancs.ac.uk/~killick/Pub/KillickEckley2011.pdf)?
@user49089 That's exactly right. See my answer for more; you can think of T as X1 and S*T as X2.
Biostatistics is part of epidemiology and medical science.
In trees, the program chooses what variables and levels should be used for splits; that's part of the point.  If you only included x1 in the model, then it would be the only variable used, but the program would still choose the splits.

But if you only have one variable, trees probably aren't needed.
I think this may be very similar to a previous question? http://stats.stackexchange.com/questions/66079/naive-bayes-classifier-gives-a-probability-greater-than-1
Thanks. Yes, I've seen this method. The problem is that it applies to only particular inter-cluster metric and operates along with the actual clustering. I need the algo to create dendrogram having the history of mergers after any metric was used.
The goal of discriminant analysis is classifying new unlabelled cases. While the OP knows the 4 types and may have the labels for a set of data it seems that the goal is to see if they seem to form 4 distinct clusters which is perhaps the reason PCA was mentioned. Perhaps a distance measure where you can look at the distance between cluster centers compared with the distance of members within a cluster to the cluster center would be the appropriate thing to look at. For multivariate normal data the Mahalanobis distance would be appropriate.
That's how I read it.  The author's meaning is clear because at the very outset he defines this as a *bivariate* expectation.  The version you are looking at is merely one way of expanding it out to emphasize that analyzing the loss can be reduced to analyzing the expected loss (over $Y$, conditional on $X$), as a function of $X$.
This look more like a statistics question (or at least 1. and maybe 2.). You could flag the question for transfer to CrossValidated...
Thank both StasK and Dimitriy for the great help. To clarify, I did have other variables in my regression, i.e., " regress outcome weekly_dummies, regime, i.doctor, vce(cluster doctor)" in which weekly_dummies are dummies for weeks, and regime is my main variable of interest (There is a policy switch in my data). I didn't get the complain from regress as StasK advised. Furthermore, could you please clarify how to remedy my issue? I am still confused especially I used the similar specification on other data and there was no such an issue. Is this data specific? Many thanks again.
Thanks@GregSnow for the nice answer. There is one point that needs some attention: According to Gauss-Markov, equal variance is required for BLUE.
By the way, the way to tell if you want the three parameter gamma is if you have two different sets of those three values which differ by a constant (like "10/20/40" and "15/25/45"), you'd think the shape should look the same, just shifted along (as would happen if you did this with a triangular). If instead you think they should be differently shaped, then it's not  three parameter gamma.
@StephenCanon, you're right.  I've corrected my answer.  Thank you.  I'm getting old.
Sorry that i've forgot to add the r paranteses. It's done now :-)
Don't you mean "not auto-correlated"?
I have no idea what you're asking/trying to do.. but intra group and inter group variance are related as follows.

Var(Y)= E[Var(Y|X)]+Var(E[Y|X])

Interpretation: Let Y be player skill and X be dummies for teams. This says that the variance in player skill is the average within-team variance of skill plus the variance across teams of average within-team skill. IN your case, if the team means are similar, the second term is small.
It would be useful to ascertain whether any variable selection method, including the lasso, can reliably find a set of predictors.  Simulation may be best for this, but you can try splitting your data in half and running lasso twice.  If there is not substantial overlap in the set of selected variables, beware.
The duplicate is identified on the assumption your question is about *estimators,* as stated in the body, and not about *data*, as stated in the title.  If it is about data then please clarify your question and explain to us what you mean by "inconsistent" and "unbiased" data.
You could also fit a time series model or possibly an ARIMA model with covariates.  When observations have a time correlation structure it is usually a good idea to take that into account.
Thanks @nico. Exactly what I was looking for.
I don't think the question is Bayesian in any way. The *problem* you're applying this to may be, but the question contains no trace of it, and the problem you do ask about comes up in frequentist statistics too.
+1 for org-mode