tttjjj/elt_bench huggingface dataset:
+1 The abstract makes it sound like one-dimensional [cokriging](http://www.itc.nl/~rossiter/teach/R/R_ck.pdf) (which would be a good approach, too).
While Patrick's links are very helpful, it would be great to see a more self-contained answer geared to the 'moderately statistically literate scientific reviewer'.
@Glen Good point: there's an answer for that [here](http://stats.stackexchange.com/questions/46847/what-is-the-standard-error-for-the-distribution-of-the-difference-in-proportions).
Which textbook are these questions you're posting from? What have you tried with this question?
+1 Excellent, thanks!
I think looking into Linear Regression and Interaction terms might be of use for what you are trying to accomplish.
I actually want to remove it only as model will be bad. Question is: how to identify such cases ?
+1 and the first link you provide references a thread therein :-)
Thanks Glen. Appreciate your kind help. What I attempted was to then take 20(0.00181), but I was still not able to get to the answer. I was wondering if you will be able to advice on what I am getting wrong? Is this something that I should use Hypergeometric Distribution instead?
I was not talking about F-tests for the whole model.  I'm talking about partial F-tests that test whether a specific predictor variable's effect can be distinguished from 0.
Your new code is mysterious. Are you trying to roll your own linear congruence pseudo random number generator? *Don't*. Just use `R`'s built-in one. Not only is it easier and faster, it will work much better, too. It's as simple as `U<-runif(m)`. Incidentally, with `m` as small as 1216, there's better than a 1% chance you will compute a value of `I` even worse than -28.7. But before trying this, first fix your integration error!
thanks robertsy, for good reference. indeed the process is not markov. Even if acceptance probability is independent of the past, the transition kernel of the process is a function of the proposal density and thus depends on the whole chain.
@fgnu Maybe I have misunderstood the theorem, but to me it says that if the likelihood is quadratic then the distribution of $Z$ is normal, while I would like to see (if is exists) a result where the likelihood becomes (locally?) quadratic as the sample size increases. That might be against the spirit of the Geyer's paper, though!
@princessofpersia I think the author fixed the problem I alluded to with the matlab code, see here: http://homepages.inf.ed.ac.uk/imurray2/pub/09etm/
As was correctly cited [here](http://stats.stackexchange.com/a/54524/3277), after rotation components may no longer be hierarchically ordered (1st is the strongest, 2nd is next behind...). But in whole (multivariately), no information is lost in rotation; so the scores of all the extracted components together are as valuable after rotation as before rotation.
+1, I've edited a small mistake in the second last formula.
Please include more information. What is not right for example?
+1. Though I'm not sure an analysis book, even if it is Rudin, is "approachable".  ;-)
As a very first quick-and-dirty check, have you tried plotting the sample crosscorrelation between the two series and finding the peak? There are some curious discontinuities, e.g., in the first series sometime around Feb 2005 or so.
LIDAR is a remote sensing method used for 3D surveying or, when mounted on aircraft or satellites, for digital elevation mapping.  It tends to produce high-resolution but irregular networks of 3D locations.
These are complicated questions & the issues involved are subtle. As fond as I am of CV, I am not confident that people can steer you through the maze of issues involved to address this situation appropriately via Q&A over the internet. I don't want to be discouraging, but I think you need to either take some statistics classes, read & work through the textbooks on your own, or work w/ a consultant.
Or, $EY\\mid X]$ is a discrete **random variable** taking on values $\\frac{1}{5}$ and $\\frac{1}{2}$ with probabilities $\\frac{5}{9}$ and $\\frac{4}{9}$ respectively with a cross-check that $$E[E[Y\\mid X]] = \\frac{5}{9}\\times\\frac{1}{5}+\\frac{4}{9}\\times\\frac{1}{2} = \\frac{1}{3} = E[Y]$$ as it should be.
It looks to me like the package `coin` (among several others) does randomization tests.
e.g. see [the answer to this question (read the whole thing)](http://stats.stackexchange.com/questions/6127/which-permutation-test-implementation-in-r-to-use-instead-of-t-tests-paired-and). If I understand right, the examples cover both approximate and exact cases and cover both independent and dependent samples.
According to your anova model your intercept holds the value of the average of all levels + the control level(not the 472). So the coefficients you see here are the difference of that level against that control level. Infact it says that your two control levels are very different, more different than that between the control and test
"Appropriate" for what purpose?  There's nothing the matter with computing the arithmetic mean within blocks for aggregation, whether or not there's correlation.  Many uses of the mean (and other statistics) do not depend on an independence assumption.  Questions arise only when you are making inferences about those means using procedures that require accurate estimates of sampling variances.
Hi @Cel, it looks like you've got *ALL* interactions in the model, including the 5-way, 4-way and 3-way interactions. I'm not sure about this case, but that will typically wildly overfit the data, which will make your results less generalizable. Backward selection (if you must use it) does not need to start with a completely saturated model - it should start with the largest model you find plausible. Can you reduce that at all?
Similar to 80% of car drivers think they're "above average".
I don't see the usefulness of changing the scale of your predictors; you're losing interpretability without gaining much in return. If there are substantial differences in partial R²s, then it means that some of your predictors are substantially better predictors of your DV than others, controlling (holding constant) all the other predictors in your model. Remember that when you have interaction terms, it changes the interpretation of the coefficient of any predictor involved in an interaction (see my answer here: http://tinyurl.com/md7n8fg). This problem is sidestepped when using partial R².
Assuming his sampling is truly random and that he already has a list of every single person with heart disease in the world (as implied by his 'Or is it better to only look at the people who have heart disease.' And since he is only interested in finding ten people with heart disease, then it is just as random and more efficient to select from the group you are interested in studying rather than selecting from the billions of people on earth over and over until all 10 have heart disease.
You're perhaps thinking a forecast should look like a simulation.
@user39531: I apologize for this comment, but I noticed that even though you have already asked 18 questions on this website and some of these questions received decent answers, you have not accepted **a single answer**! May I ask if it is a policy of yours? In case you are simply not familiar with the concept, you should "accept" (by clicking on a green tick mark) the answers that were helpful to you. It looks like you could accept quite a few. This will increase your reputation and also make people more eager to respond to your future questions.
You can just assign your data-point to one cluster centers (random choice), and continue to use K-means until convergence. However, if this is a new data-point that you want to consider after the convergence of your algorithm, and this data-point have "almost" the same distance to all cluster centers, then you may consider creating a new cluster having this data-point as a center.
Also, are you sure running qqnorm(resid(myModel)) on a mixed model with multiple factors works?
It is a jumbled mess as it is. What do "modified by a constant..." and "separated by constants" mean?
BTW: Procrastinators of the World, unite! But not today...
Thinking a little more about this I think your answer is double counting. Lets say we know 6 answers (1-6 WLOG). Your solution is showing the tests {1,2,3,4,5}U{6,7} and {1,2,3,4,6}U{5,7} as distinct test. I think the solution requires inclusion-exclusion.
A correlation structure for the repeated measures is *not* a random effect. And you don't answer to any of my questions, except telling me I am wrong without mathematical explanations.
Can you say what each of your questionnaires is supposed to measure? Are they all measuring (essentially) the same construct (ie, QOL), or are they considered unrelated to each other?
@Sneha I hope you're not suggesting my approach is anything less than rigorous: please do not mistake economy of notation for lack of rigor. (If you're referring to my parenthetical remark about "this can be rigorously derived," that's because the formulae for joint distributions of order statistics are so routine and well-known that I felt there was no need to go fully into the details here, nor is there any reason for you to do so either.)  I do wonder, though, what might constitute a "rigorous proof" of a *numerical integration*: after all, it has no closed form.
This doesn't preclude you from approximating them with integers. My solution isn't perfect, but it works approximately. For example, given weights (1/3, 2/3, 1), you could upsample the second class by a factor of two and the third class by a factor of three.
Some more context would be helpful
The acf with missing value is calculated in the normal way, but missing values are 'skipped' in the sums (that is, acf formula for a given lag looks like a sum divided by a sum, in each of those sums the missing values can be skipped). This is not the same as removing the missing values from the original data. The problem with matlab is it doesn't skip NaN, and including that in the calculation turns everything into NaN.
It is clear that it is spread to X1, but that's why I have an instrument to purge X1 from endogeneity.
@msh210 That's best asked as a separate question--it's a good one, please consider posting it.  I deleted some comments here because they were [not appropriate for this site](http://i.stack.imgur.com/oqatV.png).
Do you mean if the experts were allowed to view the $n$ data points? I was under the impression that their priors were created before the samples were drawn and could not be functions of this data.
@johanvdw you could combine what you did with the qqplot and the one sample variance test.  Subtract and divide like you did for the qqplot, the resulting values should have standard deviation of 1 if the predictions and standard deviations are correct, so just do the `sigma.test` with a null of 1.  You should probably check this out using simulation (generate data, split into 2 groups, estimate without the model selection, then do the analysis).
$\\alpha$ is determined based on your own needs. If your goal is classification accuracy, one option is to use cross validation to get the best $\\alpha$ to maximize accuracy. With that said, when you're given the option it's always better to avoid using cutoffs and instead to phrase predictions in terms of expected loss (see: [loss functions](http://en.wikipedia.org/wiki/Loss_function)).
I don't know anything about the AP statistics exam.  True experiments are ones where you manipulate the predictor variable and make groups to test multiple hypothesis or control and experimental groups, etc.  Quasi-experimental designs are pretty much anything else that looks like an experiment.  So, imagine a regression where the x value is weight and the y value is some sport skill.  You don't manipulate either variable, you randomly sample both.  So, Placidia's criticisms of stattrek are quite valid for true experiments but not as much so for quasi.
I promoted my comment to an answer.
Are those rules of thumb based on assumptions about the size of the true regression coefficients and the size of the error variance? Would I be right in thinking that if the error variance was known to be very small, a much smaller ratio of data points to parameters would be acceptable?
(-1) This question makes no sense. Please reread the definition of the [ROC-curve](http://en.wikipedia.org/wiki/Receiver_operating_characteristic) and edit / delete your question appropriately. The important point is in the first sentence, i.e. **as its discrimination threshold is varied**. By reporting the (although not complete) confusion matrix for all your images, you already have chosen a threshold.
I happened to be there at the time fire was discovered (so I can vouch for this analogy). It was not trivial at all; it was downright terrifying!
And to clarify, are you saying you've got the top $k$ scores out of unknown $n$ from a known family (e.g. log-normal) whose parameters (e.g $\\mu$, $\\sigma$) you want to estimate? So the trouble comes from having a nuisance parameter to deal with? Reliability statistics deals with these sort of problems; exponential order statistic models especially.
Your idea is a good one, but my answer to a [related question](http://stats.stackexchange.com/questions/46185) explains that you need to do more than just regressing one independent variable on another. Although it is in the context of ordinary regression, the same *concepts* (if not precisely the same techniques) apply to generalized linear models.
... just one point of clarification on my comment above, while the means *are* different, this alone tells us *nothing* about **why** they are different, apart from the labels of $A$ and $B$ (or equivalent synonyms of them).  Anything beyond that is not answered by the means, but by the interpretation it is given (which will depend on your prior beliefs about what the potential causes of a difference may be - the obvious stand out being the promotion).
(+1) A good paper, thank you. I also found this article interesting, Conditions for factor (in)determinacy in factor analysis (Krijnen et al.), http://j.mp/dwo7c8.
OK. Now its two of us hopping for a reply.
Hi Matt, I updated my question. Take a look if you like. Thanks.
The Benjamini-Hochberg procedure can be used to adjust $p$-values. See my answer here: http://stats.stackexchange.com/questions/97024/false-discovery-rate-of-multiple-regressions-models/97081#97081
Two of the factors are defined with only a single indicator, which if I'm not mistaken would make the CFA under-identified. Do you have any luck when removing these two factors and re-estimating?
+1 Wow. This even works for explaining covariance to those who already thought they knew what it was.
Also, see this question: [http://stackoverflow.com/questions/9666212/how-to-compute-error-rate-from-a-decision-tree](http://stackoverflow.com/questions/9666212/how-to-compute-error-rate-from-a-decision-tree)
Thanks for providing the graphical support
For what it's worth, in current terminology I think one should read Kant as asserting, e.g. in the Second Analogy, that whatever correlations you observe, there is *some* causal graph generating them.  As far as I am aware he had no particular method for identifying the structure but did assume that it must be fully connected (because 'every event has a cause').  In this sense he is contemporary: causal inference requires a mixture of causal assumptions, e.g. expressed via a graph, and observed regularities in the data. And you can typically neither avoid the first part nor induce it from data
@JohnK Your answer is correct. The idea behind my suggestion is that the law of large numbers tells us that if we divide by just $n$, we obtain in the limit the expectation of $X_1^2$, which is 1. Thus dividing by anything that grows faster, such as $n^2$, we obtain $0$ almost surely, hence the limit is degenerate at $0$.
@prob: $T_a$ is nonnegative almost surely, so for any $\\theta \\geq 0$, we have $\\exp(-\\theta T_a) \\leq 1$ almost surely, so the expectation definitely exists (for such $\\theta$)!
In that case the argument comes partly because the 5-6 is narrower and partly because the lower bound is a greater distance in proportion to the width from 0. Consider that a CI is usually about 4 standard errors wide then the narrow one is 20 SE's from 0 while the broad one is not even 1. It's not a universally accepted argument and may not have been what your instructor was looking for.
+1 For one-way ANOVA
You're not doing anything wrong, nor is the software. The SVD is not unique. If we perturb the column vectors of both of the basis matrices by the same sign matrix, we obtain another valid SVD for the same matrix. This is also true for eigendecompositions of symmetric matrices. I didn't run your code, but what almost undoubtedly happened was that each of your `eigen` calls ended up choosing a different sign pattern for the basis vectors. There's no algorithmic error here.
@DirkEddelbuettel do you have any insights or who might be able to help. I found your answers to R questions very helpful in the past. Sorry I sound a bit desperate here, but I really need this solution to meet a journal deadline.
I think it's good for anyone. :)
Sorry, I should have been more explicit. You need to look at all possible splits of the 16 columns into 2 sets of 8. For each split, you get the difference between the within-set mean and the between-set mean. You want to know where the within-between mean difference for your actual split comes in that distribution of all possible mean differences. (Since the within-between comparison doesn't care which is Set 1 and which is Set 2, you need to look at only 15_choose_7 = 6435 splits, with one of the columns always being in Set 1.)
Does "discrete" mean categorical or ordinal or integer or a mix? Can you clarify what class of algorithms you want to use for your problem? Most neural network implementations that I am aware of handle discrete input/output. Decision trees can be used for pattern recognition, and again, most implementations that I am aware of handle discrete input. So, are you mainly trying to find an R package that allows for this, or do you want to find a good reference to learn how to code things yourself?
This is detailed in page 5 of [this document](http://www.stat.cmu.edu/~cshalizi/350/lectures/28/lecture-28.pdf).
I am really looking for the low-frequency coefficients, because my project is about sign language recognition, so I look for the shape and position of each hand rather than any fine details. Does the proposed approach work after the zigzag coding or doesn't need any prior modification of the arrangement of coefficients at all?.
1. Most articles have tended to treat indeterminacy of scale and identifiability as though they are separate issues. One argument in favour of the distinction is that if we add in more observed variables then the ratio of knowns to unknowns goes up but that doesn't obviate the need for a loading to get set to 1. 2. Thanks for the tip regarding  PLS.
+1. Thanks! Can you explain why "chose a proposal distribution that is not 0 or too small in areas where the target distribution could have significant mass", and compare it and the two sources in my post?
Hope it is useful. It tries to demonstrate that bootstrap validation is more efficient method than other forms of validation. For a how to example of bootstrap I think this site is good even if Stata centric: http://medicine.utah.edu/ccts/sdbc/stoddard_textbook.php (model validation, section 5-14)
@NickStauner:  Sorry Nick, I lost my response to you.  In short I don't think that approach addresses the issue.  First, the comparison of interest is not against the grand mean.  Second, there is no familywise protection in standard regression and the ANOVA tradition tends to demand such protection.
PCA has nothing to do with "significance" in the statistical sense of p-value. If you PCA-analyse covariance matrix then eigenvalues _are_ the variances of the components (see e.g. http://stats.stackexchange.com/q/22569/3277).
Rather than changing the question and answering something that has not been asked, please consider consulting with the OP by means of comments requesting clarification.  Sometimes your guess about the intention will be correct, but more often it will not be, and then posting an answer is both confusing and a potential waste of your time.
Hmm ? Well. What is "important" for you? Points (1) and (2) imply important as predictor. Point (3) imply important as a descriptor of multivariate cloud. These are different concepts. Have you made up your mind on 'importantness"?
Thank you very much Gavin for that excellent, in-depth detailed answer. It looks like my data produces a qualitatively similar outcome with GAMs, in that there is either very little improvement or a worsening of fit (assessed via AIC/AICc) when comparing a GAM with and without the standard correlation structures. Do you/anyone know: if there are very clear, if irregular, cyclical trends in the data/residuals, would it then be most appropriate to stick with the most well fitting correlation structure rather than a model with none? Thanks again.
I think your second paragraph is a key... at least for my intuition: an l1 "ball" is more like a diamond that's spikey along the axes, which means that a hyperplane constrained to hit it is more likely to have a zero on the axes.
Hmmm. Can you edit your question? Do you have an interaction term in your model? If so, that's crucial information.
@Stat Possibly. Under which conditions will that be the case?
It is still better to include both variables.  The only price you pay is the increased standard error in estimating one of the variable's effects adjusted for the other one.  Joint tests of the two collinear variables are very powerful as then they combine forces rather than compete against one another.  Also if you want to delete a variable, the data are incapable of telling you which one to delete.
+1 I never knew about this difference in restriction, thanks for pointing it out.
Most reports of meta-analyses would include the value of $Q$ and its $df$. So why don't you compute $I^2$ based on that?
Thank you! I will review them.
I took a look at the discriminant analysis, but I don't understand how it can help in this case. I just want to know if I can replace those 4 clinical rules with each other without a significant change in the results. Can this be done?
I'm sure you are right, but for my benefit, how is this a Poisson? It sounds like the OP has a data set with a binary outcome. Would this be glm(Incident~Numbers, offset=log(Numbers), family=poisson, data=threatdata) ??
That's what I ended up going with after a bunch of recommendations, I was just very hesitant because Bishop's book on machine learning, while heralded by some, is supposed to be a very difficult book to learn from if you *don't know it already*.
this is not entirely correct about large samples.  It is rather that you require large binomial cell counts $n_i$, or what is the same thing, a large amount of replication of covariates.  The pearson residuals are far from normally distributed for any observation where $n_i<5$.
could you explain this? I'm proficient in MATLAB but I'm not sure what this line of code is supposed to do. I used my P array but it just returns 1. And what is `ii` supposed to be?
You ask a 2D version of the 1D problem addressed at http://stats.stackexchange.com/questions/65866, so consider adapting the techniques described there.
There actually isn't data for every day, those are the only days that the item in question was available to be purchased.
Hi @nicko. Thanks for your answer. What torch package do you use exactly for the CUDA runs? And would this [network setup](http://pastebin.com/YwiyN6uk) work in parallel from your experience? I have a ParallelTable, although it's added to an nn.Sequential. Or do I need to modify things from as I have them?
This, of course, *assumes* that `p` is the same for all families. If instead we assume that some couples are more likely to have boys than others (*i.e.*, their `p` is higher) then the result changes, even if the *average* value of `p` is still 0.5. (Still, this is an excellent explanation of the basic underlying statistics.)