tttjjj/elt_bench huggingface dataset:
It sounds like you may be interested in [dynamic factor analysis](http://scholar.google.com/scholar?hl=en&q=dynamic+factor+analysis&btnG=&as_sdt=1%2C33&as_sdtp=).
@P auritus: You ask "What is a better way than 30 ANOVAs?"  A single MANOVA with 30 dependent variables or a single MANOVA with the first K principal components.  Either approach takes into account the interrelationships among your FAs.  In fact, just as an aside, a MANOVA on the PCA scores would give you the same result as the MANOVA with the 30 original variables if you use all 30 principal components.
Thanks for the response. Don't feel obligated to update anything (as it is not your job to teach me mathematics), but I do not have any clue how you "integrate over the prior distributions". Does it involve just calculating all possible combinations since my distributions are uniform?
The problem is that "outlier" isn't post-hoc conclusion about a particular realized data set.  It's hard to know what people mean by outlier without knowing what the purpose of their proposed mean statistic is.
Sorry, I don't follow. It's well established that you can't use the out-of-the-box distribution for the KS statistic if your model parameters are estimated from the data you're testing, and my data is not normally distributed. Central limit theorem has nothing to do with it.
please be more specific, at the current moment the question is too vague. Also it is better to give a preview of data in the question. I for one am very wary of downloading unknown archive files from the internet.
I realize it's pretty straightforward to do transformations on the posterior samples (odds, log odds, risk ratios, etc), the question is regarding the preferred/standard presentation.
@ ozlem aktas: I browsed through your thesis and I could not find the section where you adress the issue of time scaling of CFVaR. I have posted the question in the quant section of stackexchange and there you find the correct answer. It is not only scaling by the square root. Mean, variance, skewness and kurtosis all scale differently if you aggregate returns over time. Skewness and ex. kurtosis tend to zero (at different rates) which is called aggregational Gaussanity.
Sorry for my misleading of phase usage. Here, 'at a give level of variable C', I meant that the variable C is between a small range (comparing the whole dataset, which variable C can vary in a very large range). What I want to know is, within a specific level of variable C, whether A is determined by B or by C.
You have a programming language you want this in?  What format is your data?  You could easily do this in R with kmeans.
A really helpful link, thank you.
Clara, could you please clarify what type of regression/model you use? You say that you have 2 dependent variables, so do you calculate 2 separate regressions?
Perhaps if you wrote out the _exact_ definition of convergence in distribution, using $\\epsilon$'s and $\\delta$'s and "there exists a $N$ such that for all $n \\geq N \\cdots$, instead of $\\to$ and $\\lim_{n\\to \\infty}$ etc., you might see your way to figuring out what you need in the proof
But if the $p$-value is large, does it matter if $D$ is small w.r.t. establishing similarity?
For something to be discrete, it must be in one-to-one correspondence with a subset of $\\mathbb{Z}$. Taking your question at face value, I think that's your answer. But, like whuber, I'm unsure whether that's what you're asking (and, if so, why?).
There isn't any way to characterize $r_1 - r_2$, since $r_1$ is defined only if $a < b$ and $r_2$ is only defined if $a > b$, by the question statement, and the two conditions are mutually exclusive.
@Glen_b I reflected on some of our discussion in the clarification. Perhaps, we should start deleting some of our comments
+1 for moving comments :)
@AweSIM: Could you give us some more clues as to what you consider "good visual matches" ? E.g. if the second signal is delayed by one unit, are they still good visual matches?
Have you contacted the package maintainer yet? In my experience, he is very open to suggestions (especially if you provide him with a reproducible example).
I am having a hard time seeing how this helps resolve the paradox. To what prior distribution are you referring? (And please--this is not the place for bringing up the Monty Hall problem. That notorious situation always generates more discussion than insight.)
The typical approach to fitting logistic regression models is via maximum likelihood. This is not equivalent to minimizing the variance of the residuals.
Can you describe the way in which it is not working? Is this only an R programming question?
But then he has the proportion values over time right? So it then depends on the series if it is stationary and then it could be modeled by ARMA (otherwise check for unit root to model it with ARIMA). But it's right, that possibly forecasts result which are out of the range (between 0 and 1). Ok, now I got the problem
I've tried to add a little more context to your question. Feel free to modify, if I have misconstrued what you are asking.
I touched the theme of normality in FA [here](http://stats.stackexchange.com/questions/34753/are-data-transformations-on-non-normal-data-necessary-for-an-exploratory-factor/34759#34759). And please be aware that Principal Component Analysis is _not_ Factor Analysis _proper_.
Classification tree(s) and discriminant analysis are two more techniques that might be appropriate.
+1 Thanks Nick. So should I use the adjusted R2? It would be 1 - 0.2866 = 0.71. Would I then be correct in stating that ~71% of the variability in the response variable was not fully explained by gender, operation year, or patient age, and that some unknown proportion of the unexplained variability may be due to the pathologist?
Ok thank you for your answer! Is this a good justification that I did simple regressions because of the high correlations between predictors? Because that is in fact the reason. When I put all of them together in a multiple regression they are not significant and therefore I cannot discuss demographics. I must add that the correlations are between ,200 and ,500 but still I assume this is the reason that together they are not significant while apart they are
Use principal axes or minimal residuals (better not generalized). Use PCA only if you want PCA model, not FA model. Oblique factors are never correlated greatly so that to become obstacle in futher regressions.
I think we need to see pics of this cross-stitch!
this is a very useful overview, thanks.
Fisher's exact test seems utterly appropriate to me. It yields $p=2\\times 10^{-9}$, so I guess that any other reasonable test will also reject the null. Of course, you will have to decide whether statistical significance equals practical relevance here, or whether e.g. some misclassifications are more problematic than others.
Do you have any additional constraints on $A$? If $A$ is invertible, then $H_0=\\rm{vech}(\\Sigma^{-1})=A^{-1}a$. The then problem amounts to a well-known problem: that of testing whether $\\Sigma^{-1}=B$. Here $\\rm{vech}(B)=A^{-1}a$ (remember that $\\rm{vech}(B)$ determines $B$ uniquely).
@AndyW - I would do it as it is outlined in approach 2.  In approach 1, this is done by a procedure in STATA, which I think works directly with the matrices.
Hi gung. I have an undergraduate degree in statistics. I like having some 'here is why' math reading similar to the math in Elements of Statistical Learning.
+1 Thank you for your suggestions. I meant to provide the matrix as an example. In reality, the number of entities I have is much greater in the order of thousands. I updated my question to reflect this but my data is essentially a graph depicting the relations between various entities in my system. A snapshot of this graph is taken at regular time intervals which is what I am trying to understand. Would you still suggest a preliminary examination with VARIMA?
I understood that we can do either way i.e select $d_1$ feature and calculate TF-IDF or Calculate TF-IDF and then select $d_1$ feature. However anyone explain me what is the standard procedure of selecting those $d_1$ feature
Do you mean to say that you have 61 numeric characteristics and 7 character valued charactersistics? What is your outcome in this study? If you put text values into a regression model, I think the default is to cast them into factors, meaning that you will adjust for indicators for each level of the text field. I can't say anything about whether a probit/logic model makes sense unless you describe what the variables are, what the outcome is, and what you're interested in measuring.
By the way: a 99% CI is not going to be more narrow than a 95% one. You want to be more certain that the truth is in there, so you want to consider a broader range.
Thanks for that great answer!
I'm using the interface commands: Linear models > Constrained linear regression
The question is handsome but the drawing is indeed awful. The more sharp-kurtic-than-normal distribution is supposed to be heavier-tailed. But you didn't draw these tail regions (which also should be coloured red). What their areas you suppose to add up to?
Sorry, I did not mean to imply that you do cross-validation for the PCA itself. My comment for recalculating PCA referred to what you would need to do within each of the $k$ iterations.
@DavidMelkuev Yes, the expressions are based on the essential idea behind OLS: we seek coefficients $a$ and $b$ that minimize the mean-square error $E[(X - (aZ+b))^2]$. This is standard stuff usually covered in undergraduate probability textbooks in the chapters on expectation which usually also point out that for the special case when $X$ and $Z$ have a bivariate normal distribution, the **general** _minimum-mean-square-error_ (MMSE) estimator $E[X\\mid Z]$ coincides with the **linear** _minimum-mean-square error_ (LMMSE) estimator $aZ+b$.
Your question could be interpreted as "more mathematically sophicticated = better". Surely you did not mean that. So the question boils down to how you want to define "better". Various ideas on that exist, but it very hard to come up with one rule that applies equally to all situations.
(+1) impressive. Questions: 1.Rapid Analytics is completely hosted by rapid-i ? 2. If the 1. is true, do you know the resource limitations (cpu time, ram etc.) an open-source user has to expect ?
@Fan With the definition I provided on the first line, the variance does not exist when $n=1$.
*How does R take care of unit testing for its t.test()?* -- There's the combination of `R CMD check` with `example(t.test)`. See `?example`. It's also quite common to implement unit tests you don't want the user to see within `example` with the `dontshow` option. There are also several CRAN packages designed to support unit testing.
+1 "Cook's distance is presumably more important to you if you are doing predictive modeling, whereas dfbeta is more important in explanatory modeling": this is very useful advice.
+1. Couldn't you extract a little more information out of the data by doing a goodness of fit test to see whether $34$ years of no tornadoes is consistent with this Poisson model and its estimated rate $\\hat\\lambda$?  (Although $34$ seems a little high, I think a formal test would show it's nothing to worry about.)
Welcome to the site.  I edited your post quite a bit to fix the English usage. I also removed your sig line as the site puts in your name automatically.
Where were you going with terms like `zip5001-6000`? That seems like a kind of centering you might do for a continuous covariate, but not for a factor.
@Strin: More-complicated hypotheses are more likely to overfit the data. (That is, to fit the quirks of the data you have in hand, resulting in poor fitting on future data.) I remember a class I took where a guy in the front row was just sure that the professor was holding back on us and not giving us the sophisticated algorithms that would let us make a killing in the electricity markets... He never really comprehended overfitting.
And yes, they are not necessarily independent.
Agreed, though I'd choose it as my second book, after _R Cookbook (O'Reilly)_ (not free!)  _R Inferno_ felt like _Cookbook Part 2_.
How is it the opposite result?  Your empirical calculation of "method A+B" gives 0.7159, where my exact calculation gives (rounded) 0.716.
Nick, they way you changed the above formula would be confusing. (the way that Wikipedia explains it)  This is a very well-known theory, but could also be confusing.
You are welcome to give your own answer or add a comment if you do not agree.
is this, perhaps, just a case where the asymptotic approximations to the [null] distributions of one or more of the three test statistics are not so good? perhaps the results differ because the actual sizes of the tests are different? how large are your sample sizes?
Your total 500 doesn't matter nearly as much as the size of each cell. Nevertheless, it's probably not causing the problem you have (see my answer) but it is more important than the total N to report.
I made edits to clarify how to run the model with only one row (you can't have any predictors), and added a comment about reporting.
@Milton and that's exact what a naive Bayes will do. Basically, you ant to predict (or learn) a new classification based on your previous naive Bayes fit, and choosing to use a Bayesian models takes care of that. (using penalized estimators will give similar answers).
More generally, if the model is beta1*(V/3000) + beta2*(V*D) you can't do this: in other words, your suggestion presumes you know a linear constraint among the coefficients.  It is true that the regression coefficients can have *relatively* large VIFs or standard errors, but with sufficient amounts of data--or with well chosen observations--the estimates will be *reliable enough.*  So, we agree there is a problem and indeed I agree with your solution *as one of several alternatives to consider*.  I disagree that it is as general and necessary as you make it out to be.
@AlecosPapadopoulos Hogg, McKean, Craig, Introduction to Mathematical Statistics, 7th edition, Chapter 6, page 325.
Yeah, basically, which is what I found when I was looking initially. I just thought there may be a more formal way of doing it (i.e. <insert name of statistician>'s test)
**[Definition](http://en.wikipedia.org/wiki/Volatility_%28finance%29):** "In finance, volatility is a measure for variation of price of a financial instrument over time". If you are in interested on investing-selling-buying a financial instrument, this is of crucial interest.
@gung: See http://download.journals.elsevierhealth.com/pdfs/journals/0895-4356/PIIS0895435610001381.pdf
+1.  However, I would like to suggest this reasoning, although appropriate for some applications, may be a little too limited for general purposes: some choices of plotting position ($\\alpha$ between $1/3$ and $1/2$, usually) provide slightly better values for probability plotting, for instance. This isn't going to be relevant to computing quartiles, as you note, but it will be relevant to computing the more extreme percentiles. Computing percentiles is so closely related to quartile computation (I'm sure it's the same underlying code) that this "detailed stuff" is worth bearing in mind, IMHO.
A major difference is that random effect models are models (including a random effect), while empirical Bayes techniques are inference techniques: e.g., you can run an empirical Bayes estimation on a random effect model... Empirical Bayes methods apply in about every situation where you could use a regular Bayes method, not only to random effect models.
Welcome to Cross Validated David P, for more information about the site and how to ask questions, you can check out the [FAQ](http://stats.stackexchange.com/faq). Is your question just about whether the calculation is right or wrong (it is wrong, with the numbers you give on your second line the result of the operation is 0.187, rounded)? Otherwise, yes, a percentage can be as close to zero as you can imagine because it is bounded between 0 and 100 and continuous on that interval.
Nick, your first statement is wrong.  It's not a "technical issue", it's just not correct.  The confidence interval is a statement about what would happen in repeated experiments, that they would cover the true value 95% of the time.  A statement about the confidence that the true value is within my given range found in my given experiment is not the same as that at all.  If you removed the "that" in "that confident" and the parenthetical numerical amount then you'd be closer to the truth.  You could just say that it means you believe the true value likely to fall in the interval.
... assuming that most of the variability in your data consists of signal. If most of your data is noise, then the first few components will mostly represent that noise, and not the signal.
You should provide more details or give minimal reproducible example. The most probable reason is the pairwise correlation, another reason may be the improper hadling of factor variables
I think I finally got it. The definition of $F_p$ probably means “at some $x > 0$”. The weight of $F_0$ does not need to be all on 1, it just needs to be on a single value greater than 0.
Again, I was not trying to build an optimal model for prediction, I was using the model to explain the effect of the predictors on the outcome variable, i.e. an explanatory rather than predictive regression model.
Thank you for your answer! $var(\\hat \\alpha_1) = var(\\hat \\beta_1)$ is indeed true for big sample sizes. You used a bis sample with n=100,000 in your example. When using a tiny sample (for instance $n=20$) and a higher number of iterations (for instance 20,000) the difference is observable:  
> var(alpha[2,])  
[1] 0.0502148  
> var(beta[1,])  
[1] 0.04727873
I had a hunch that soccer matches were not quite Poisson, but thanks for the the reference :)
@EpiGrad I don't know about that but there is a long standing debate between the “include only the best studies and estimate the effect size” and “include everything and test” approaches. I would say it's more a difference in philosophy than a question of one approach being unambiguously valid or invalid.
thanks a lot for the answer! i really appreciate it!
I am under the impression I've already seen a paper talking about that. Maybe googling with "covariate" instead of "independent variable" is more judicious.
Hey gung, thanks for your opinion. Do you think that the prof might be expecting me to give predicted value of Y? Even then he should have specified prediction for some specific value of X...
Can you clarify your question (perhaps add some example data, or a figure)? A kernel density just draws a symmetrical density (usually normal) around each point & sums them. If you have any points at 0, half of those densities will be negative. What more do you need to know?
Thanks for the comprehensive answer. I can go out and hunt for resources for details of the second part, where you outline the problematic cases, but if you could provide me a couple of pointers, that'd be fantastic. Books, or even keywords for resources to get a better handling of the problems would be fantastic. Unfortunately, you don't get to see these in most books about Bayesian statistics, networks etc.
**Hint**: Using the definition, the expectation is a polynomial (in $p$!) no matter what $U$ is. This polynomial is supposed to be equal to a rational function with numerator degree of zero. Conclusion?
Well, you don't know the population standard deviation, do you?
+1, this is a really nice contribution. I wonder if [this](http://andrewgelman.com/2005/01/why_i_dont_use/) is the Gelman blog post you are referring to.
+1 (although it deserves +3 at least) - very thorough and comprehensive. Made the whole issue much clearer for me. Thanks!
Thank you Robert. I will look into logistic regression.
@yannick: Is this homework? If so please add the homework tag. Hint: you can integrate $\\int_{-\\infty}^{\\infty} \\frac{\\sin x}{x}dx$ by writing the [power series](http://en.wikipedia.org/wiki/Power_series#Examples) expansion of $\\sin x$ and integrating by parts. Doing this should give you $\\int_{-\\infty}^{\\infty} \\frac{\\sin x}{x}dx=\\pi$.
it would help us if you can tell us which software you used and some sample commands. At the moment, I think, we can only guess because outputs are _sometimes_ software dependent.
If the $y_{i}$'s are count data then why have you written there that they follow a normal distribution?
Take a look at the **party** package and see if this suits your needs. It can handle a number of types of models in the nodes of trees IIRC.
Weak LLN says that sample average of iid data with finite expected values converges in probability to that expected value - so yes.
Can you explain why this is a pyramind and not a triangle?  Or a triangle and not a ladder?  I'm not being snarky -- I'm wondering if I'm missing something significant that this visualization is trying to convey.
Answering this question (http://stats.stackexchange.com/questions/48381/power-calculations-logistic-regression-with-continuous-exposure-cohort/48402#48402) I found the following paper which formally discusses  power equivalence between binary logistic regression and an unpaired t-test http://www.ncbi.nlm.nih.gov/pubmed/9699234
Interesting.  But since fixed or random can be considered a condition of a given variable (a given column of data) rather than of a parameter associated with that variable,...does your answer fully apply?
Nothing in the phrase "Two one-sided tests" - nor the underlying logic implies normal-theory. It should be perfectly possible to adapt it to a location-shift alternative with a non-normal distribution. But beware - in many cases with non-normal data what you really want is a *scale-shift* kind of equivalence test, and with other kinds of data, something else instead. Knowing what is needed really depends on what you're measuring and what problem you're solving. Rather than trying to squeeze your peg into a round hole, it pays to examine the peg.
'for AbsSR were really large' that's okay: it just mean you have large outliers. Presumably, the MM coefficients will be very different from the usual OLS ones. "I want to use an estimation approach like what you said based on 1-0 weights" then use FastLTS.
Code is fit <- lm(avg_sal using hike ~ sal due to change dept or title or both)
Can you pinpoint some of the resources?
"*I really don't have much time to understand these concepts*" -- this seems bizarre given it's actually the answer to the question *in your title*, so it would seem essential to understand it (why spend all that time posting if you don't care to understand the answer to your question?). What happens the next time you have such a question?
Why do you think they're different?
+1 Thank you for the help :)
I've taken the liberty of adding the `self-study` tag since this seems to be unambiguously a textbook question. Please see the [tag wiki info](http://stats.stackexchange.com/tags/self-study/info). Is there any information/condition missing?
How are you doing on this? In your attempts I see both $x^Tx$ and $X^TX$. But $x$ is a vector and $X$ is a matrix, so you've got a problem there. Also I think you want $Var(x^T\\beta)$, not $Var(x_i^T\\beta)$
Thanks for the link. Yes the lack of relationship to the dependent var is a problem. I have seen the same effect as in the references. where, say, PC 8 of 10 may be the most significant predictor out of all PCs. The current data has PCs with eigenvalues ~0 have no loadings of more than 0.1-0.2 in them. Therefore i thought it safe to disregard these. Often the more PCs i generate the easier they are to interpret where one or two variables may load highly onto one PC. To sum up my current thought track..does it seem acceptable therefore to pick those with high var loadings?
What is `R`? What is `S`? Where does the graph come in?
@Glen_b I was thinking the complement rule from Probability?