tttjjj/elt_bench huggingface dataset:
In the formula case, I suggest using `wilcox.test(Measurements~Point, data=DATA)` (though you don't need `data=` really) and for the other two use `with(DATA,wilcox.test(Measurements,Point...)` ... both are cleaner and easier to follow
Because of the different corpus sizes, the frequencies are basically guaranteed to be different; you can use a K-S test on the raw frequencies but it will reject for sure, even if the underlying probabilities are the same.  I was dividing by the mean just to get all the frequencies on the same scale - correct for different corpus sizes, if you will.
You could also try to reproduce the graph/results from the paper in question by obtaining the original data.
Agree w/ whuber. Is the goal to estimate the 0.135th and 99.865th percentiles of the calls-per-week distribution? (If the dist'n were normal, and you meant std dev, and they didn't throw out data, then that seems to be the target estimate.) With <1000 observations (from the same dist'n), it's hard, but possible, e.g. [Dekkers & de Haan (1989)](http://www.jstor.org/pss/2241666). So: I agree w/ whuber the current method is unjustified; wondering if you were asking for positive suggestions, too, or just confirmation of your (correct) critical thoughts.
Are $n1$ and $n_1$ intended to be the same thing? If so, can you edit your question, please, to fix that? Can you also edit to reflect your actual needs - it's important to mention things like a wish for an algebraic answer.
As indicated at the end of the first line of the question, the OP is trying to [test a ratio of variances](http://en.wikipedia.org/wiki/F-test_of_equality_of_variances); that's a two-tailed test.
@ttnphns: yes. But as svd is provided by the BLAS that could be different with a different BLAS. I'd have expected that a good optimized BLAS does something like this. It doesn't seem to be the case with OpenBLAS, however. I'm too lazy to check other BLASs, but maybe a few people could check their other BLASs so we find out which ones are optimized for this case and which aren't. (I emailed the OpenBLAS developer and sent him a link to this question, so maybe he can add some info, e.g. reasons for not switching the algorithm to the `svd (X'X)` for long matrices.)
What is the response variable? Do you expect dropouts?
I would say that carefully defining the important variables was critical to me being able to do this problem without getting badly confused. Notation matters a lot - if you don't get it right, you quickly become mired in confusion. (At least I do.) It's worth some effort (and takes practice) to make it right.
Where you might recall having ill conditioned matrices is if you put time {1,2,3,.....,} in as an explanatory variable. {1,0} dummies are OK though.
You could probably do this as a dynamic linear model, but you might need to spell out what model you want to fit. In what sense was the GAM fit not sufficient?
"Nonparametric" doesn't typically mean "I have no parameters." Even in the simplest case of $X_1, \\ldots, X_n \\sim F$ where $F$ is an unspecified cdf, $F$ is just a parameter in the (infinite-dimensional) space of cdfs. A common definition of a nonparametric problem is one with an infinite-dimensional parameter space, and a method is nonparametric if it can estimate anything in that space. Then, the class of ANNs (with network topology a free parameter) fits this definition for nonparametric function estimation.
@user603 Absolutely! But that seems to take us out of the presumptive situation where "nearest neighbors are best predictors." Out of respect for that, anything we do when processing the data should preserve this local predictability.  If one column has a "vastly different scale" than its neighbor, that circumstance would violate this stated assumption pretty strongly. (I also wonder at your focus on columns: upon re-reading the question, I cannot detect any asymmetry in the roles of columns and rows.)
Welcome to CV, @user20293. One of our goals here is to create a permanent record of statistical information in the form of answers to questions. As such, one of the things we worry about is linkrot. In light of this, would you mind providing a summary of the info at the link in case it goes dead in the future? Since you're new here, you may want to read our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which cover topics like this about CV.
@mpiktas: thx for the summary. Since I haven't specified any weights, the respective default weights should be used. Now that I know, I should maybe restate my question to: Why do different default weights of vcovHAC and NeweyWest make such a huge difference and how to determine weights? I mean do you know which weights STATA or other packages use?
Also, I don't understand what this means " I then found that one of the categorical variables had a significant effect on my covariate (i.e. the covariate was significantly higher at one level of the categorical variable compared to the other)." That sounds like an interaction.
Thanks for the high level walk through of the pluses and minuses of both sides. It confirmed a lot of what I suspected.
Nice example! Thanks!
That was not well worded.  What I meant was the line from the centre to the red point is not perpendicular to the tangent.
I haven't seen anything quite like this, but it has similarities to [dasymetric](http://stats.stackexchange.com/q/15784/1036) mapping, and is kind of the opposite of when people want to [obsfucate data for privacy reasons](http://gis.stackexchange.com/a/25854/751). I wouldn't say it is a terrible idea, it is just very difficult (to impossible) to project the individual level data while maintaining the appropriate relationships between variables (i.e. the ecological fallacy prevents you from knowing the individual level correlations).
+1 In my opinion, this is the most elegant approach if OP simply intends to efficiently show the weight and speed. If OP actually intends to visualize a dependency between weight and speed, a scatterplot would be preferred, as @gung describes. I also agree with xan that including two axes on a single plot often confuses interpretation and should usually be avoided.
Hmmm... those are good points, @PatrickCoulombe. We could seek to clarify the question. In particular, "hierarchical multiple regression" could mean any of several things.
Is the reference saying that the posterior distribution for the Normal-Normal problem is intractable or is it saying IN GENERAL the problem is intractable?  In general the problem is intractable, but for certain special cases ("conjugate priors") the problem is easy, and the Normal-Normal is one of those special cases.
@AndyW You might have to elaborate on this. I am not sure I understand what it would mean for someone to smoke for 8 or more days per week. Is some contexts, like when stadium demand exceeds seating capacity, this makes sense, but not here.
Thanks!  Silly bug.  Fixed on stackoverflow as well.
"You may also want to see if the reasons cluster somehow - do people who rate one variable highly also rate another one highly?" Can I use Spearman's correlation test to find this kind of relationship?
Are the measures P1 and P2 for Test 1 related? Stated differently, is P1 for Test 1 (P1/T1) expected to be closer to P2/T1 than to P2/T2 or P2/T3?
It help to mention how much data you have.
Also, i would like to know if youre calling my functional form a mistake, based on the premise that my lecturer concluded there is no relationship between the incinerator and house prices. (In other words, is it generally wrong to use variables that take values in thousands at levels? Or is there some possibility that using prices instead of lnprices was better in this case?)
Do you mind citing the evidence that Tufte/minimalist style charts contradicts all psychological research we have regarding humans, their memory recall, and pattern recognition? While I think this is a good question, such a negative and condescending tone does not make your request appear very sincere. Nor does spending 10 minutes to do background research to refute my suggestion of reading Cleveland's work as pertinent to the discussion.
My practice tells that the best, logically most suited way of clustering respondents based on rankings they made is clustering on [Canberra distance](http://en.wikipedia.org/wiki/Canberra_distance). Such as hierarchical clustering.
@Johannes The square of the SE is proportional to the variance of the sample mean. (The constant of proportionality [depends on one's definition](http://stats.stackexchange.com/questions/22021/how-are-margins-of-error-related-to-confidence-intervals/22026#22026) and may vary slightly with sample size.) Independence implies the variance of the sampling distribution of the difference of means is the sum of squares of the SE's.
@dep_stats , while this is a good answer and I have upvoted, it will be very hard for anyone not already familiar with what a variogram is to follow your notation. I would suggest either expanding upon the definition of the variogram or linking to other references that define how to estimate variograms in more detail.I think this article is a good one on introducing variograms , http://www.geog.ucsb.edu/~chris/readings/Variogram.Interpretation.Modeling.Aide.pdf .
I get a _very_ small number when I look at the `rcond(cov(mat1))` using your example `mat1`, like 1e-17. Your covariance is rank 5, so any techniques which invert it are going to fail. Looking at `rowSums(mat1)` I see the problem: the columns of `mat1` are _not_ linearly independent--they all sum to 10000. This is probably due to how your Hypergeometric generation process works (_i.e._ each ball drawn has to be from _some_ class). To get around this problem, subselect the first five columns of `mat1` and ignore the last one (it is completely determined by the first five). hth.
@Matthew Well, actually my real data is multinomial, that's why I ask for the generalization, but I'd like first to understand the binomial/beta-distribution case.
After I calculate the trend equation as shown in the picture, I would like to find all values with respect to that equation, by writing 31,32,33,...,364,365 into the 'x' and sum all these values.
(Your nick is very appealing, I like it) Do I understand correctly that for you, "Subject" is class, "Session" is some condition or treatment (what is it, by the way?), and X1 X2 are some quantitative characteristics?
Do you use statistical software? Because you could probably benefit a lot from running a few simulations yourself and then doing a little exploratory analysis first.
Hi Alex, is it possible that you could explain why it is not possible to make something stationary from the red line. Is it because it has few points (11) or simply because of the shape? Also if you have time to explain what you mean by differencing both series I'd be very grateful. Thanks.
I don't see how this is an counterexample, as the question does not claim anything about the product of $P(X\\mid Y)$ and $P(Z\\mid Y)$. (Although, it is unclear to me what the claim in the question is).
This question is a duplicate of [this one](http://math.stackexchange.com/q/561189/15941) that the OP asked on math.SE about one hour earlier.
"Independence" makes sense only after you have adopted a chance model for the data.  Several good models are appropriate.  Which one(s) to use depend on the decisions you plan to make.  So: it seems you wish to estimate the proportion of time spent moving, stopped, or off.  Will you use these proportions to characterize (a) the history of a single ride, (b) conditions along a route traveled by many cars at the same time, (c) a route to be traveled extensively in the future, or (d) some other generalization of the data?
The  derivation of this lower bound takes more that just a few lines, and since it was a side comment I provide the development [here](http://stats.stackexchange.com/q/66775/27403).
I think internal (? external) validation methods attempt to address this?
Thanks Nick, your description of the dataset is correct, basically each subject has a certain mean reaction time (Y, expressed in ms) for each condition (X=1 through 9). Although I've never done regression in Statistica before, I think that a table like the one you described would probably allow me to get a mean regression slope for the dataset, but the problem is that (I thought) having repeated measures across rows is not statistically correct. Is this where I am wrong? This is the reason why I had each of the 9 means arranged as columns as opposed to making a dummy variable out of it.
@user3447 Please use comments instead of answers.
Thanks. The first part is important but is not emphasised in anything I've read so far. They were probably never expecting this kind of use, or I just read over it. I'll look into the other kinds of binomials and will probably update this question or put another question up.
First you say `R` does not give the p value for the full model from `summary(lm)` then you show an example where it does. Perhaps the issue is only *extracting* the p value.  Also, as @Roland implied, comparing p values with 16 or 20 initial 0's is fraught; but it is hard to conceive of a case where it matters whether the p value is one in a quadrillion or one in 10 quintillion (using American definitions of those terms).
@amoeba [You can read the "passage" for yourself](http://books.google.com/books?id=uUNGzhdxk0kC&pg=PA155&lpg=PA155&dq=%22599+is+recommended+for+general+use%22&source=bl&ots=ZSpVygnQkK&sig=Z47Do1Xk_2_FtnBxIYm2buSQnkw&hl=en&sa=X&ei=IZf5UsDWDsifqQG3hYCYBQ&ved=0CDcQ6AEwAg#v=onepage&q=%22599%20is%20recommended%20for%20general%20use%22&f=false). This is an example of exceptionally unclear writing in statistics, and in particular is only applied to inference on the trimmed mean with Windsorized standard *error* estimates.
@Bitwise I was using the coin flip as a metaphor for the Binomial processes, as whuber stated, so any process with `p = 0.5` and `!p = q = 0.5`
I doubt it... LOESS is built-in in most statistical programs, thus there is no practical need to do that. what precisely do you want to alter inside?
Michael: your `2*(1-pnorm(x))` only makes sense for non-negative  $x$
I have also wondered about this problem, but (unfortunately) haven't found any convincing answers yet. I think there is no solution. There are R/BioC packages like `hopack` (and others) which can estimate the number of clusters, but that doesn't answer your question.
Just to clarify - your desired test is for equality of variances (standard deviations) between two groups?
Thanks for the tip, I'll look into it! I was actually considering neural networks as a third option, but I figured it would be pretty hard for me to approximate higher order functions as the popular R libraries seem to simulate only fairly simple layouts. As for the features, they are mostly customer data (age, rating), financial (money spent, products viewed etc) so there is probably not really a 'big picture' underneath - in fact the first principal components are pretty boring, but I guess that's because they are true :)
Welcome to our site! Your answer is unusual in how it simultaneously poses a question and suggests an answer to the current question. As such it has caused some people to think it ought to be posted as a separate question.  You may do that if you like--it's easy to add hyperlinks back to this thread to maintain continuity--but in the meantime, understanding your post to have been made in an effort to help, I have edited your answer to emphasize the passages that appear to me to be the key parts of your proposed solution.
@Spacedman - well not exactly...I need the CI for ind not for the height ...because I am calculating the ind (the X) given a new value of the height. And I need the CI for the new X=(Y-beta0)/beta1
@Glen_b I describe in a little more detail the kind of histogram I am considering in the non-uniform bin case.
@CJ12, you have not accepted any answers on your previous questions. Please be reciprocate to the people who responded to your questions by going back and accepting some answers. You have been on the website long enough to learn the etiquette.
Your question is very confusingly worded. I don't think you're correctly characterizing the pdf there.
I disagree on this point because the interpretation of the fixed effects from the within estimator can still be useful for further statistical application. In the end, interpreting the fixed effect is also an interpretation of a statistic.
Thanks for your reply, even though it sounds frustrating. However, collecting new data wouldnt help me since I would still have the same scenarios determining the risk perception. I just dont know how to quantify them and  incorporate them into the regression. For example scenario 1 has a risk perception value of 0,09 (prob*impact), whereas scenario 2 has a risk perception value of 0,01. The problem is that this value is the same for all respondents since they had to select a certain strategy according to the scenario they read. I want to find out, what strategy is chosen given the scenario.
Those calculations you have look right. You're saying you don't know how to find what proportion of a standard normal distribution lies above 1.45? We're back at my earlier question "What do you know about finding percentages in given ranges for normal distributions?" ... but now specifically in the case of the standard normal. Do you know anything about finding probabilities (areas) with standard normal distributions?
Thank you for this respons
Michael Chernick, you might be interested in [this](http://news.stanford.edu/news/2004/june9/diaconis-69.html) article about Diaconis.
Thank you. I search just the formula for computing the Standard deviations of the sum of several coefficients (yes global impact). My director ask me to do this with my results. I'm sure that is very easy but i found nothing  that could help me.
Questions like A are by necessity a judgement call, for which one needs lots of detailed information on the data, the question, the intended audience, the _exact_ procedures you have followed, etc. So this is the kind of question that is not well suited for remote consulting. It is better to find a local expert and ask her or him to sit with you and go step by step through your analysis.
There are still a lot of clustering algorithms that will work, even if you only have a distance matrix & no longer have the raw data. Eg, if you are interested in hierarchical clustering, once you have a distance matrix you can do either single-linkage clustering or complete-linkage clustering w/o the original data.
This is one case where proving a generalization of the statement might be easier than proving a special case, as well as giving more insight: can you show that the conclusion is true when the normal distribution is replaced by *any* symmetric distribution?
In Stata, `qexp` from SSC is a canned implementation.
Why bother with a library? You could just implement the formulae at http://en.wikipedia.org/wiki/Point-biserial_correlation_coefficient using java.lang.Math functions and a loop.
From statistical viewpoint, there is hardly any question. Take the new point and assign it to the nearest cluster (nearest cluster centre). Update the centre immediately or after 10 new points or after an hour - this is as you like; that would be a programming, implementation issue.
A [uniform distribution](http://en.wikipedia.org/wiki/Uniform_distribution_%28continuous%29) is one which has every value between its maximum and minimum being equiprobable, not at all like a normal distribution, which is more probable near the middle, less probable in the tails and isn't bounded.
My guess is that the attributes of `mydata` has changed (but not necessarily the actual *values* you see) -- specifically `c` and `d` were numeric variables the first time you ran it and are factors now. Try `is.factor(mydata$c)` to see if it's been made a factor.
"*why ... we have to difference the data in order to do time series analysis*" -- in fact we only need to do that if that would make non-stationary data stationary. If it's already stationary, there's nothing to do.
Functions from discriminant analysis
The answer to this will depend on what you want to do. Transforming could be alright, but whatever you test, then, will only be valid for the transformed values (and how certain are you that the transformation is 'valid' in your population anyway). Skewness doesn't make other values like a mean 'invalid', though (just a little harder to interpret), so perhaps there will be no need to transform. Please be more specific as to what you want to achieve.
I was taught that one can potentially identify the difference between an auto-regressive series and a moving average series by examining the PACF (see this [webpage by Robert Nau](http://www.duke.edu/~rnau/411arim3.htm#signatures) for an explanation). A more complete answer would state why that is the case though (and functionally what is the difference between an AR and a MA process mathematically).
I'd say just what I said in my 2nd paragraph - it's assuming that context, & a more generally applicable definition might be better.
Yes it does. It says "A paired difference test uses additional information about the sample that is not present in an ordinary unpaired testing situation, either to increase the statistical power, or to reduce the effects of confounders." -- "increases power" means it reduces the type II error rate; 'reduces the effect of confounders' would reduce the rate of either one type of error or the other, or maybe even both.
I like that it's nonparametric, but if you have previous knowledge this may not be an advantage. Personally, I would do both the resampling and the traditional confidence interval. Many times they will be consistent with each other.
@user49440: No, I was just betting that your effect size would be too small to detect with *N* = 50. Sounds like you're on to something though! Penalized regression will just reduce the estimated differences between adjacent levels of the predictors. E.g., if the DV is lowest for people who disagreed and highest for people who agreed, penalized regression will make more moderate predictions for people who were neutral. This might make your effects seem smaller or simpler, but theoretically, that is probably the more appropriate prediction for future samples / the rest of the population.
Can you assume that $A$ and $G$ are (at least approximately) independent (not just uncorrelated)? If so, then you want to *deconvolve* $T$ and $A$ (not $G$), which is a different formula--but relatively straightforward. [The mistake here lies in assuming that $G=(A+G)-A$ can be computed with convolutions. To see why not, consider a case where $G$ is constantly $0$ but $A$ varies. Then $A+G=A$, but your formula for $A-A$ is really a formula for $A_1-A_2$ where the $A_i$ are independent and have the same distribution as $A$: their difference definitely is not zero.]
Can a milestone that's been achieved ever be lost, or is it a permanent change of state (like death in survival analysis)?
If you don't need statistical advice on how to exclude exposed logger data, this question is probably [off-topic](http://stats.stackexchange.com/help/on-topic). Matters of coding within a particular software environment tend to get passed over to [Stack Overflow](http://stackoverflow.com/help/on-topic) when they don't require special statistical knowledge.
In the context of model uncertainty, you always want to divide your data into test, training, and validation datasets regardless of the method used. This is described in the Tibs book which is free online. I think BMA implicitly does that by sampling from the model's posterior distribution to estimate posterior model probability. Anyway, it's a tough problem. In general, there's no way to do it by brute force. This might be an illuminating article: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2742495/
Thanks.. It was the boundary condition that I didn't know how to handle when it is negative..
Your question doesn't contain a question.
(+1) Thank you very much for your answer. It provides a very convenient way to directly compute the intercepts and slopes, thanks!