tttjjj/elt_bench huggingface dataset:
Uncorrelate the $Y$s using Cholesky decomposition (say $Z = V^{-1/2} Y$); set up the quadratic form of interest in terms of the uncorrelated variables $Z$; solve the eigenproblem for the matrix in the middle; go back to $Z$s and rotate them using the orthogonal matrix made up of eigenvectors (which preserves their uncorrelatedness); look again at your quadratic form in terms of $Z$ and conclude that $c_i$ must be the eigenvalues of $AV$ or something like that. So they are not arbitrary.
Is there some data likelihood or are you just trying to draw samples *from* a multinomial distribution?
Thanks @Peter, they were really useful comments and absolutely right about the impact distribution being crucial. The point about making the parameters random variables is also something I will try to incorporate. And of course, spot on about Excel's efficiency. The choice is partly about my comfort zone and partly about giving client something they can work with.
I hadn't thought of it quite in those terms, but that might be one way of comparing the matrix at time 1 with a matrix at another time.  I'm not sure it would work with all the NAs.  Basically, you do need to create a way of quantifying the inconsistency, unless you have zero tolerance.
"*You may be right about not trying to alter my data too much*" -- if your interest is in differences in mean, I'd generally treat it as a last resort, but it's also a matter of what the people reading your thesis will think, and what you can explain.
I don't know how Macro deduced that James meant P(A|B)=1 if B⊂A rather than P(B|A)=1. Also James' equation P(A)/P(B)=A/B makes no sense and I really did not know what he meant. But Macro knew that he meant P(B|A) = P(B)/P(A) which is correct and would make sense that it is what James really meant.  Kudos to Macro!
I answered by a counterexample based on the latter interpretation. If that was not intended, you need to clarify the question.
LOL, okay. Thanks Glen.
@NickStauner indeed, but the recorded values in milligrays are the values in grays multiplied by 1000. So if you had 0.32 grays, you now record 320 milligrays. If you named your data column "grays" (unsurprisingly, since the measurements were in that unit) and you make a new column called "milligrays", you may well say something like 'milligrays=grays$\\times$1000' ... perhaps somewhat confusingly, but the result is as it should be.
The problem is that i have a separate model for each region of my country, so i can't inspect one by one. There must be some kind of test or package to automate this!
Thanks @NRH, that's very helpful. I have separated the third question into two separate questions, since I think they probably require different answers.
The paucity of capital letters leads me to suspect you're posting from a mobile device, again. :-) In Hastie, et al., there is a problem which leads the reader through this data-augmentation interpretation.
A related question that should be of interest, [Time series for count data, with counts < 20](http://stats.stackexchange.com/q/173/1036).
It's a good procedure, except for the part about a "quantum random generator".  You don't need "quantum".  Moreover, I would recommend avoiding anything with the word "quantum" in its title unless you know the random generation field well, as there is a lot of quackery out there.  If you want a really good random number generator, ask for a cryptographic-strength pseudorandom number generator (e.g., `/dev/urandom` or `CryptGenRandom`).
I once used python to fit a specialized GLM with $> 500$ million observations and an *a priori* unknown number of parameters. It would have been impossible to do in R, in a number of respects, and the statistical needs were far from light, indeed they were the integral aim of the utility. :-)
I understand your problem... But I think your problem has to do with the construction of the model, the variables selected etc. I advise you to post another question explaining the data, the actual model and your current problems with the estimation. I think that you can get a helpful answer on how to build the model that fits your data. Good luck!
When you want to know how well a logistic regression classifies things, you need to test it on a set of "validation" data that are separate from, and independent of, the data used to fit the model.  (Testing with the same dataset gives invalid, misleading results.)  Often this is done by randomly splitting all the data at the outset, fitting the model on one portion, and testing on the other.  At this point your question is easily answered: you compare models based on how they perform on the validation data.
are none of those predictors continuous variables?
@ O_Devinyak thanks for your suggestion. I will look into this. Also, please can you explain why the described feature is non-linear. I do not understand how you have reached this conclusion.
Clever example.
(1) The final line in the question is erroneous because it does not include the parentheses necessary for it to execute the intended calculation.  (It is guaranteed to produce results greater than $1$, which is impossible for any p-value.) (2) You are not actually conducting a permutation test: the two samples `a.random` and `b.random` will rarely comprise a random partition of the data but typically will overlap substantially. Instead, compute `b.random` as the complement of `a.random` within the union of `coding` and `lncrna`.
mh, I am not really  looking through this - I guess I have to refrain from using the std errors of the hyperFit output.
In addition to that, many a Bayesian prior has been made from the outcome of a meta-analysis.
I guess there is nothing to do with regularization here:-(
Thank you very much. I will take in consideration all your comments.
Ah, I see.  I was reporting AIC scores of the reduced model anyway but just curious why this was also not appropriate.  Thank you!
How about [this post](http://stats.stackexchange.com/questions/26585/how-to-do-a-generalized-linear-model-with-multiple-dependent-variables-in-r)?
The estimator that they propose in that paper doens't control for all time-invariant heterogeneity, according to this paper:  www.ssc.upenn.edu/~allison/FENB.pdf

Allison et al propose a simple LSDV poisson regression with a correction to the SE's post-estimation.
This question needs more information.  What model are you fitting? What is gamma and C in this model?  Further, could you give this information by editting the question, and not with comments.
A good article about this, check the appendix for an example with calculations... https://support.google.com/analytics/answer/2844870?hl=en
+1, nice answer to somewhat convoluted question.
What about ROC curves makes them insightful to you?  Are you really interested in concordance probabilities ($c$-index; ROC area; pure discrimination measure)?  I find the ROC area to be helpful even though the curves are not helpful to me.  And you can generalize the idea of concordance probability to multiple categories using Somers' $D_{xy}$ rank correlation coefficient.
Sorry. Did search, but presumerably I searched using the wrong words. R and statisticas question are, for whatever reason, far from uncommon on stackoverflow. Feel free to migrate me. I agree that this makes more sense on CVal. And yes - it turns out that this was not a coding question. But I did not know that; the solution could well have been to know about some "perk" in R.
If you need a confidence interval and have that many samples I recommend you to use Gaussian processes. They are extremly capable at giving predictive distributions with the only problem that learning scales at $O(n^3)$ where $n$ is the number of samples. But this is not a problem for you here.
... 
I was not aware of the the relationship between LR tests and MI though, that's very interesting!
Thanks, edited to add this. But of course you meant to add 1/n.
I think you should update your title to reflect the fact that you're looking for a quantile (given a probability value) rather than a probability value. (If I understand the body of your question correctly.)
Here's an example of your suggestion from the NYTimes

http://www.nytimes.com/imagepages/2010/05/02/business/02metrics.html
It is easy to see how arbitrary such a process is.  Few estimators in statistics that require binning or arbitrary choices have survived without heavy criticism.  And I would never call proportion classified correct as "accuracy".
Great!  Actually, this is perfect for my purposes
Dichotomous is just another word for binary. So, you have a binary response. The question still seems very broad to me. Yes, given a date variable time series might make sense, but an aim of establishing _cause_ implies emphasis on variables related to cause.
To add one further requirement: It would be totally awesome if the function would also be able to run all kinds of contrasts from this ANOVA, for example, using the `multcomp` package (which, as far as I remeber, requires `aov` objects). Otherwise, I agree totally and normally use `ez` for my daily duty, because it is so easy...
@user603 The OP is not trying to reconstruct the entire data set, just the correlation structure of the repeated measures.  Even so, I think the answer is no. I say this assuming that you are talking about the group means and group standard deviations.
Please define or describe `not unidimensional factor`.
I'm not sure what you mean by "final" and "original" objective function, Cam. PCA is not (conceptually) an optimization program. Its output is a set of principal directions, not just one. It is an (interesting) mathematical theorem that these directions can be found by solving a sequence of constrained quadratic programs, but that's not basic to the concepts or the practice of PCA. I am only suggesting that, by focusing on the smallest eigenvalue rather than on the largest one, you can reconcile the two ideas of (1) minimizing distances and (2) taking an optimization view of PCA.
@gung if you follow the link of Jeffrey Breen's blogspot posted in the first answer, it will make more sense. http://jeffreybreen.wordpress.com/2011/07/04/twitter-text-mining-r-slides/ He provides an R function to calculate the score of tweets using a Lu and Hiu's lexicon. The function is great, however the lexicon could be changed.
(+1) So, issues with non-plausibility of the case where both=1 aside, the model based estimate would just be the sum of the two marginal effects, which we know can be very misleading in it's own right :)
Please explain more what you aim at in this question. Decision trees are probabilistic (they estimate some function of the conditional distribution of $Y$ given $X$).
I just got it that you are passing `prec_y` as data so my criticism of the second model is invalid and my suggestions too simple. As you identify your second model in the question is correct. However your suggested answer suffers from the multicollinearity I identified - to eliminate replace `gamma_mu` with `0` so that each of the `gammas` is drawn from a normal distribution with a mean of zero, i.e., `gamma[j] ~ dnorm(0, gamma_prec)`. As a result `mu` will be the overall mean and `mu + gamma[j]` will be the mean for each group.
I see. And what you wrote is true because $f(X|\\theta) = f(X|\\theta, a, b)$, right?
@Aksakal, that's essentially right--we usually define overfitting in terms of the model's performance in out of sample prediction. So this is a non-standard use of the term. However, used in this way, it is true that the variance of a parameter estimate will increase as more variables are added w/o increasing N (ie, as the number of variables, p, goes to N).
@Alexis, how would he do a paired test when he does not know **any** of the pairs?  I can see using the relationship in known pairs to estimate correlation and therefore impute the missing post survey, but without knowing the pairs to begin with you would need to make pretty strong assumptions about the relationship.
Nick, yes I can see how this works, you presented very clearly. I will do that. The objective of this research is a spatial distribution of calcium in groundwater based on statistical methods. That means that I have to "find the equation" or better algorithm which will present the best correlation between spatial indenpendent variables. Than I have to enter this equation in GIS and do moddeling. I only have some concerns about this method of regression, due to categorical data and than moddeling. So that is why I asked if there is any other method that would be better for moddeling calcium?
Thanks. There is no way I'll get the covariances, but even just the means can be helpful, given I have sample sizes and can make pretty good guesses about the variances given the other data I have. At the very least you have kept me from hitting my head against the wall all day trying to work this out.
before you get some down votes, I suggest you tag this article with homework, and also to include some efforts of yourself. Just a friendly advice ;)
It depends on what you really want. I was referring to exoegenous variables in conditional volatility equation, but most packages allow for exogeneous variables in the conditional mean. You need to spend more time thinking about what you actually want to model and if the basic packages are sufficient.
I see, group effects exists on both the individual level and on the aggregate level. When one apply fixed effects to aggregate data this takes care of the group effects of the aggregate level. However, group effects on the individual level can only be adjusted for by applying fixed effects for the individuals.  group effects confounders
Hi Ben, I clarified the problem and provided an example and more details on the model and what I am trying to do. Let me know if any additional information would be beneficial.
to all reviewers: Please see that the answers in the duplicate question are NOT focused on binary classification, but can be applied to multiclass or regression problems as well.
Jeff Leek's class is very good for getting a feeling of what you can do with PCA. The good thing is that it does not get into complex mathematical/statistical details (which can be found in plenty of other places) but rather provides an hands-on approach showing how to really use it on data.
Thanks Chris. It is a real tutorial :)
@StasK There is more to the concept of a "sample" than randomization, which is a separate idea.  For instance, convenience samples, judgment samples, and systematic samples are still called "samples."
I'm not an expert on GBM, but the RF handling of missing values appears to be rooted in the idea of imputation, http://en.wikipedia.org/wiki/Imputation_(statistics) In cases where missing values are not missing at random, your results can be biased due to missingness. Imputation attempts to recover this missing values and reduce bias.
did you have a look at those statistic software alternatives: http://statpages.org/javasta2.html
It is noteworthy that the [Satterthwaite method](http://en.wikipedia.org/wiki/Welch%27s_t_test) *does* adjust the variance: rather than using a pooled variance, it computes a weighted combination of two separate variances.
I find the method described in that report to be highly suspect.  It makes *no* sense to base a p-value on averages of test statistics from subsamples (a single outlying subsample could really screw things up).  It makes a *little* sense to average the p-values of those statistics, but the purpose is bizarre: namely, to reduce the power of the test!  There have to be much better, theoretically justifiable ways to accomplish the same thing.  One should begin with a *single* test statistic rather than simulating it from subsamples.
@ayush, my answer was based on information you supplied. As always I could be wrong. The problem with the spurious regressions is that there are only special cases where you can guard explicitly against them (unit root processes). If you are really sure that the relationships in your model are not spurious, then the problem lies elsewhere. As I tried to say, maybe relationship changed in time.
nop, it's 0. Geometric means is 0 whenever any single one of the x_i is not strictly positive....
@cardinal: Yes, I do mean concave on [0,c] and convex on [c,1]. To be precise, it looks like an "S" mirrored by the diagonal line y=x. I'm not sure how to call it as simple as "S-shaped function" so I simply say "S-shaped function" (Please do not hesitate to let me know if any). Excuse my English if that turns out to be a misleading description:) After all, it's a minor issue since the answer by Har has been good enough. I just try to make it consistent with my question.
Go ahead & try again, @StatGrrl, you may be able to post more links now. Even if not, they will show up in the post (just as not clickable, I think), & a higher-rep user will be able to edit them in. Welcome to our site! Since you are new here, you may want to take our [tour](http://stats.stackexchange.com/tour), which has info for new users.
thanks! thats indeed a very careless mistake of mine which will cost me a whole lot; unfortunately. Many thanks once again
Please read the [wiki for the homework tag](http://stats.stackexchange.com/tags/homework/info), especially the section on answering homework questions.
Box-Cox transformation is not just $x^\\lambda$, it is $\\lambda\\frac{x^\\lambda - 1}{\\dot x^{\\lambda-1}}$ where $\\dot x$ is the geometric mean. The reason for introducing this geometric mean is to provide correct likelihood ratio tests.
Some *particular* kinds of combinations have names (e.g. [the Skellam distribution](http://en.wikipedia.org/wiki/Skellam_distribution)), but the general case its just compound Poisson, I think. There will be no closed form expression that's simpler than the original sum in general, though I suppose you can just write it as a convolution. When you say 'generate' do you mean 'simulate random values from', or something else?
+1 Like the consultant comment
Cheers Frank, appreciate your comments. Will look into the article you cited and also investigate [CBPS](http://www.princeton.edu/~ratkovic/CBPS.pdf) which seems an altogether better way to go about things
Thanks and regards
I could certainly be wrong about this, but my impression is that logratio analysis or related techniques like Dirichlet distribution fitting are multivariate approaches used when the outcomes variables are proportions, e.g., fractions of the budget of municipalities spent on categories school, police, and etc. In your case, the explanatory variables sum to one and the outcome is revenue. Maybe you can point us to an example that's similar to the media mix model you have in mind?
Because you're trying to find the number of trials before the first success - in this case, a value of $T$ that is greater or equal to 5. I'm just not sure if the previous steps in the experiment would affect anything.
@Xi'an: Is it clear we can assume they are independent here? As you know, that would be necessary for this result to hold (and have a chance at being useful).
Thanks for your answer. I think that the second intrepretation is the the correct one, as I am comparing the prices of a set of products in two different shoppings.
Is there any way other than the matrix form?
This is not correct. High standard errors tell you that you can't estimate the coefficient very precisely - the 'true' coefficient may well be far away from your estimated value (the standard error is like a 'typical distance' away).
+1 Thank you for the link. I will go through it and get back. I applied SVD on my original matrix before applying k-means to reduce the number of dimensions.
I know of no reason why this cannot be done and have seen it done multiple times, although I do not have a specific example to cite.
The strong law indeed implies the weak law, but the weak law is easier to prove. See here: http://terrytao.wordpress.com/2008/06/18/the-strong-law-of-large-numbers/
+1 This is a cool idea! Let me dive in deeper into this and get back. Thanks!
+1 Nice catch.  Thanks for turning your comment into an answer.  The relationship between the two log-likelihoods based on different subsets of the data is not generally predictable: in some cases, most data have positive log likelihoods, so including additional records increases the maximum (*caeteris paribus*); in many cases, most data have negative log likelihoods and including extra records decreases the maximum. One moral of this experience is that in looking at output it always pays to review even the very simplest summary statistics, such as data *counts*.
As a hint, one powerful (and historically early) approach uses geometrical reasoning. Upon recognizing that, conditional on $T$, the vector $(X_1,X_2,X_3)$ is uniformly distributed on a sphere of radius $\\sqrt{T}$, it's easy to see that the two parts of the question are very closely related (the second part immediately answers the first part) and the $1/2$ in the second part is the average height of a random point on the upper hemisphere of a unit sphere. This is illustrated at http://stats.stackexchange.com/questions/7977 and further discussed at http://stats.stackexchange.com/questions/22764
Are you sure you really meant to change the title to include "Time Series"? The question at hand has nothing intrinsically to do with time series data (unlike your other [question](http://stats.stackexchange.com/q/14898/1036)).
These are probability *densities* which therefore have units of 1/volt ($p(x)$), 1/power ($p(y)$), and 1/(volt*power) ($p(x,y)$). All such quantities are *numbers*: "1D" and "2D" make no sense applied to them. The product is the usual product of real numbers, not an outer product.  Because this question hinges on confusing probability densities with other things, it appears that the thread at http://stats.stackexchange.com/questions/4220 answers it.
Thanks Peter! For point 3), they really did do that: one of the IVs was "quartile" and the other was "trial". "trial" was pre or post. 

As Gaël noted, perhaps "quartile" really was between, even though they said it was within. That would make more sense to me, since then the post hocs would be one repeated-measures ANOVA per quartile.
Here is an admittedly absurd, but **simple** example. The idea is to **illustrate** exactly what can go wrong and why. It *does* have practical applications. **Example**: Consider the typical iid model with finite second moment. Let $\\hat\\theta_n = \\bar X_n + Z_n$ where $Z_n$ is independent of $\\bar X_n$ and $Z_n = \\pm a n$ each with probability $1/n^2$ and is zero otherwise, with $a > 0$ arbitrary. Then $\\hat\\theta_n$ is unbiased, has variance bounded *below* by $a^2$, and $\\hat\\theta_n \\to \\mu$ almost surely (it's strongly consistent). I leave as an exercise the case regarding the bias.
LIFEREG is not the name of a model.  Don't think of SAS as life.  Are the predictors such that they should have a constant effect over time?  If they have a diminishing effect over time you might go with an accelerated failure time model such as log-logistic or log-Gaussian.
If the criterion is evaluating prediction/classification performance in someway and the two different configurations yield similar results then the configuration with least complexity should be preferred. This involves lower number of parameters, simpler functional forms, and integer coefficients over long decimal point coefficients.
Are you saying you have *stratified* the population by gender and selected 60 males and 95 females, or are these the actual counts of males and females observed in a random sample?  The former would be an "unbalanced design" whereas the latter would not be the result of the sample design at all.  (This sounds like a minor restatement of several dozen of your previous questions...)
@thagzone: I didn't comment on the first posting because I'm not sure what you mean by "period effects". Do you simply mean a linear trend over time? I keep thinking that "period effect" would mean something like comparing changes decade-by-decade or something like that.
I did not understand it. What prevents you from computing the sum of absolute deviations? Is there some quantity you don't know for that?