tttjjj/elt_bench huggingface dataset:
Thanks for the answer, I'll try the robust regression soon.
I enlarged the plot and there are 8 values: http://i.imgur.com/dus792n.png
I like the generalization of this problem: when you choose *four* socks at random, what is the chance that you can select a matched pair from them? :-)
Very funny! (____)
Whatever your intentions might have been, I find the "*I have a PhD in statistics,...*" commentary to be inappropriate. First of all, it does nothing to resolve the issue at hand. Appealing to authority is an oft-used, but very misguided approach to proving things. Appealing to *your own* authority is even more problematic. It also can be interpreted as showing (inadvertently or otherwise) a lack of respect for @MichaelHardy (the personal you are addressing), who also happens to have a PhD in statistics from a very reputable program.
The expected value is correct, but it does not answer the question!
@JackTanner, that book looks interesting, but it is not *Fundamentals of Exploratory Analysis of Variance*.
Welcome to the site, @Alan. Your question is a little sparse to be answerable. Would you mind telling us more about your situation, your data, & your goals? Since you're new here, it would help you to read our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which contain info about the site & how to ask questions. You might also find this blog post helpful in formulating your question: [how to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/).
My `R` `Hmisc` and `rms` packages handle this.  In `Hmisc` see the `rcorr.cens` function.
And of course, the difference in significance for $C$ between `A.lme` and `B.lme` is due to the fact that they're not testing the same thing: `B.lme` tests for the effect of $C$ when $B=0$, whereas `A.lme` tests for the effect of $C$ for any value of $B$ (the latter being what you're interested in). If this doesn't answer your questions, let me know which part isn't answered, and I'll try to comment/edit.
I'd have to look at it to see if it worked for me or not. It seems like the build script found ATLAS, but perhaps it did not or perhaps I tweaked something before the make.
Is R an acceptable solution?
Thanks you very much @Davide Giraudo
@probabilityislogic fit distribution drawn from DP to joint and then go get the conditionals is the sort of thing I had in mind, with variations on that theme. Modeling the weights in the stick breaking construction is similar.
As a complement to the answer given below you may see [this post](http://stats.stackexchange.com/questions/107551/can-a-trend-stationary-series-be-modeled-with-arima/107696#107696).
hint: see what happens with `family=quasibinomial`
No. Both of the polynomial options given in the answer are cubic.
@Nameless Could well be. But the mean being very low and near zero is consistent with Poisson regression. Also, I would always try Poisson first before trying something more exotic such as a zero-inflated Poisson.
Requests to "avoid explanation" and a reluctance to use probability statements suggest an intention to use statistics to sanctify a result rather than understand or improve it.  That can make for a difficult conversation, because statisticians do not share this point of view.
Thank for your comment. But how can I find model which fits the data set best? Is there rule of thumb for fitting best model?@Peter Flom
Related question [Are gridlines and grey backgrounds chartjunk and should they be used only on an exception basis?](http://stats.stackexchange.com/q/25220/1036)
Nice response by Peter.  Also important is that for your purposes you should be using factor analysis per se, not principal component extraction.  http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysi
If anyone could hint me towards some literature covering such comparisons, I'd greatly appreciate it.
RSS *always* improves when you add parameters\\*, no matter how irrelevant the variable is. $\\,\\,$ \\*(well, it *can* be exactly zero, but that's an event with probability zero under the model assumptions)... so just knowing there's an improvement in RSS doesn't tell you anything.
the answer will depend on what kind of distribution is estimated. Is your goal only to estimate the distribution, or you intend to estimate something else using this distribution? What about the structure of this $n$-dimensional vector? Is it also a sample, is it time-series, or something else? Does the distribution change with time?
I like the question, there may be already most of the possible answer in the question... do you have an idea of the type of answer/developpement you want?
@Xian thanks for the feedback on the downvote. Is there actually any situation where you would favor accept-reject over MH?
Have you looked at the package 'sampleSelection' in R? Some papers on it: http://www.jstatsoft.org/v27/i07
First of all, thank you for your effort. I just tried it again, and I cannot replicate the failure of convergence with my code and example data.

Beyond that, the Slope should vary not just in accordance with Subject and Session, but also Role, I assume (since otherwise there are two times the same Round per Subject, and they are not distinguished). Does that make sense?
Thank you very much for this thought.  Indeed, this is a standard and well-documented approach to multiple censoring.  One difficulty lies in its intractability: those integrals are notoriously difficult to compute.  There's a modeling problem lurking here, too: the value of *d* is usually positively correlated with *Y*, as implied by the first paragraph of my description.
I'll edit the question to address your comments.
My guess is that the correlation is weak and the Durbin-Watson test does not reject lack of correlation.  But yes if there is structure in the residuals then of course all the time series structural checks you recommend should be looked at.
Basically our high school does not publish statistics the distribution of grades through our online student information system (powerschool). Even if teachers had access to this information, they're usually too busy to attend to those students who wish to see such statistics. So, we have to collect our own statistics through users who wish to user our service. As part of the service of providing statistics, we will most likely make it mandatory that their data be available for general statistics functions (mean, std dev, etc.) that will obfuscate their individual data point/personal info.
Cool, it's just the Projection Matrix, I've seen that before!
maybe helpful: http://stats.stackexchange.com/questions/9342/is-it-ok-to-use-manhattan-distance-with-wards-inter-cluster-linkage-in-hierarch
Best is the model with lowest future error and cross-validation gives you that estimate. The reason for c(Complexity)+e(Error) formulas is because you could use error on training data as estimate of future error, but that's overly optimistic, so you add a term to make this estimate unbiased, which is usually some function of model complexity
For a `lm` `confint` uses *t*-based intervals, and for `glm` it uses the profile method. Calling `confint.default` directly uses the asymptotic normality assumption. Bootstrap methods are more computationally intensive, but do not rely on underlying assumptions about the error distribution. If the results from the different methods differ significantly, there is likely a violation of the assumptions.
But in the example of {1,1,1,1,1,1,100} the data is not normally distributed if it is normally distributed??
BWM was something I had tried when I was attempting this. It's weird data (its actually a scatter plot of many, many runs of a simulation). The problem is there is one point that very simulation *has* to go through, so its a very dark point. In SAS, all ten bands just get dominated by that point. Tried varying BWM from 0.01 to 3.0 by steps of 0.05 with no real success.
Correct, for this matrix it's one - the AR parameter.
Your question is unclear. It sounds like you want your `v` to be a vector of random probabilities, ie they should be nonnegative and sum to 1. Is this correct?
Bootstrapping doesn't make numerous copies of your original data, no. The simple bootstrap takes samples (with replacement) from your original data, on the basis that the best available estimate of the distribution, $F$, is the empirical distribution, $\\hat F$. Bootstrapping doesn't add information you don't have (it *doesn't* 'increase your sample size' in any sense), but as suggested, does let you get some idea of the sampling uncertainty associated with estimates.
Thanks, what you say makes sense. However, I'm missing something. The trend output from STL is a whole time-series in its own. If I want to describe it, I'll need to fit some model to it. (in my question I chose linear). Now I understand from your answer that I shouldn't fit a model to the trend output from STL. Or do I misunderstand you?
I don't see the relevance: that paper constructs a differential equation to help solve a ML problem.  Isn't the whole point that we can then turn to *other* methods of solving the differential equation? An approximate solution to a DE is not "trying to approximate [an] unknown function"; rather, it is obtaining numerical values of a function that is *known* but given in an indirect fashion (the DE itself).
The more comparisons, the more data driven. You might want to study Frank Harrell's famous book 'Regression Modelling Strategies' to get a feeling about this.
$ \\frac {80}{(100)(20)} = \\frac {1}{25}$ in second row.
There is no way to answer the question (other than pointing out that it contained a false assumption) without knowing the ultimate goal of the analysis.
Here is my favorite paper about the topic: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.13.7133&rep=rep1&type=pdf
When I calculate the test statistic "by hand" in R, I get the same value as R gives. $\\quad\\quad\\quad\\quad\\quad\\quad$ (`r=rank(abs(pd-1.1));` `s0=(pd>1.1);` `sum(r*s0)`)
It is amusing that in adding the [tag:continuous-data] tag, its wiki excerpt provided the answer ("the chance that it takes any particular value is zero ... for every real number x"). Now if we could only make *all* wiki excerpts that intelligent and useful!
I've added the `Stata` tag, as your question relates to Rabe-Hesketh and coll. approach. Feel free to update if it is misleading.
I think you may need to supply a reference or an explicit example so we can know precisely what it is you are reacting to.
What would it mean for the coefficients to differ significantly? Eg, if you were looking at something as a function of height & weight, what would it mean for the coefficient on weight to be bigger than the coefficient on height?
Thank you so much for your explanation! Very detailed and useful.
Thanks for the link and the pointer to the `equivalence` package.
I need some clarification:  When dividing data into 'folds', the number of classes is irrelevant, unless you're actually saying that you have only two data points.  Why not randomly assign points to folds, without considering 'class'?  (I assume that by 'class' you mean some binary response variable that is included in each point)
Excuse me, why did you delete the plot I went to the trouble of including for illustration? It was nothing other than a cleaner, larger (i.e. non-screenshot) version of the same plot you linked to in comments. Why remove it? If you included a link to the plot, you intended it to be seen. I made it easier to be seen, and in the correct context (in your question). What's up?
Besides this confusion I have, @chl, I've seen http://stats.stackexchange.com/questions/2306/feature-selection-for-final-model-when-performing-cross-validation-in-machine?rq=1. I'll have a look to the references you give (using RF for this kind of problems). Thank you
A hierarchical prior is theoretically better but computationally harder for me.  I don't have a 1-parameter family of reasonable distributions.  If I just use the same "fit linear curve" as above with, say, 5 parameters (one of which depends on the other 4) then at a grid of 1% probabilities I'd have 4,598,126 possible priors where all 5 parameters are at least 0.  Each one of which I have to do a complex computation over all of my observations for...
as an afterthought www.biostars.org has also lots more information on this, it used to be part of the stackexchange network, but for some reason they moved it out.
You could look at tariff lines where there is no tariff reduction, or at other countries which do not change their tariffs in the same period, as your comparator.  Personally I suspect trade mirror statistics (which I assume you are using) may be a fairly weak proxy for tariff evasion: the world runs a trade deficit in goods with itself, while the EU (which does not have internal tariffs) runs a trade surplus in goods with itself.
Of possible interest: [Cross tabulation of two categorical variables: recommended techniques](http://stats.stackexchange.com/q/192/930). If you can't give us more details on the purpose of the analysis and how data were collected, this question is no more than a duplicate.
@ Rob Hyndman.. I have realized it yesterday, but your explanation made it much clearer..thank you very much..btw, I love your work on the package..thank you for all the lectures and explanation on it..I am reading it along with shumwhay's book..I didn't expect my question would be answered by you from all people,,
I don't see any reason (mathematical or otherwise) such that you couldn't have two classes where one is more spread out than the other, even by orders of magnitudes. One possible common-sense check that your code didn't go awry is to simply get means & ranges for each class for each variable.
"vote up requires 15 reputation"
thank you! I'm trying to test the significance of r; to set a criteria for the r value of a linear regression created by calibrators of an instrument. I have a tablet with critical values of r but it doesn't have any significance other than 95% and 99%
I would say you are doing it sensibly, simply create a single ROC curve using the true lable and the predicted value for each case (where that case was the hold out)
OK, I think I get what you are saying, so pls correct me if I am wrong...  First, generate a variable `TIMETREND` where `TIMETREND`=1 for `time0`, `TIMETREND`=2 for `time1`, ...,`TIMETREND`=14 for `time13` (`TIMETREND` goes from 1 to 14 b/c there are 14 time points)  then, `regress int_ret TIMETREND if (time>=0 &time<=5)`  The coefficient and sign on `TIMETREND` will tell me if there was significant decreasing trends b/w time 0 & time 5.  Did I read your response correctly? Thanks again
I did tf-idf normalization for my data and got a csr matrix, but it's still not accepting it. Can you tell me how i should modify the the data so that it gets accepted as `vector` mentioned in the docs
*Overfitting*. Maybe you should use a smaller network,  and/or regularize your performance function (among other things). Check the function `trainbr`, it implements Bayesian regulation training of your NN.
Indeed. I followed him and was just filling in details, notably why the pattern is linear on average.
Thanks for your answer! I hadn't thought of k-fold CV but that is definitely a good idea
It sounds like you may very well be interested in prediction then. Prediction can be very useful as part of a hierarchical modeling approach. Say you're running a clinical trial with several different sites, and each site is responsible for measuring ketones, bilirubin, eGFR, etc. weekly on individuals. Suppose further that some sites give very precise data, whereas others are shoddy and often missing. You can create a prediction to pool data across groups and improve the understanding of the lab parameters. This calls for standard reliability testing for continuous outcomes as I mentioned.
What do you mean "it does not work numerically"?  What actually happens when you try to estimate the model?
If $U_1$ and $U_2$ are _dependent_ Bernoulli random variables with parameter $\\frac{1}{2}$, then the conclusion that $U_1+U_2$ is a binomial $(2,\\frac{1}{2})$ random variable is suspect. How do you know that $U_1$ and $U_2$ are independent? In particular, does the event $\\{(U_1,U_2) = (1,1)\\} = \\{Y_1 < Y_2 < Y_3\\}$ have probability $\\frac{1}{4} = P\\{U_1 = 1\\}P\\{U_2 = 1\\}$? I would have thought that the event $\\{Y_1 < Y_2 < Y_3\\}$ would have probability $\\frac{1}{6}$ since all $3!$ orderings of the three random variables should be equally likely from the symmetry of the problem.
@rolando2: Thank you for the prompt reply. Yes. Each row is a different region. There are 16 regions with different populations. The 2 levels of treatment variable are thoroughly implemented and not so thoroughly implemented. Some regions implemented the treatment thoroughly and so the deaths were less. Other regions did not implement the treatment thoroughly and so the deaths were more. We have to show statistically that in regions where treatment was inadequate had significantly higher number of deaths. If the table is too big, we can use 3 or 4 regions data from 2006 to 2010 for analysis.
(1) Indicator or categorical or factor variable ... all the same. (2) Indeed, but you may want to know how well a set of dummies/categories scores from ANOVA.
Could you photocopy pages out of books from the math library?
As Emre points out, it is not difficult to turn this into a "complete" proof: Note that $|Y| = |X|$ with probability one and so $|Y|$ is half-normal. Now, $Y$ and $-Y$ have the same distribution, so $Y$ is symmetric. Hence, $Y$ must be normal. :)
Hi @Dougal thanks! this definitely helps clear things up.
Have a look at the variance of each component. My guess is that one component is just covering one point and thus has very low variance (and very high density).
Yes, normally only the raw means and standard deviations are reported as descriptive statistics.  The question of when/where to report them is stylistic; I personally would report the significance of the effect, then describe the pattern of the effect by referring to the cell means (or in the case of a nonsignificant interaction, just state that the Ms and SDs are displayed in Table X).
It's a good question: many people who are not as conversant with statistical ideas as they should be (or think they are) often use phrases like "use a t-test to test [or compare] two *distributions*" even though this compares distributions (or populations) only in the extremely limited sense of comparing their means. Thus, when the t-test rejects the hypothesis of equal means, *a fortiori* the distributions differ; but in many cases the t-test will not reject the hypothesis (perhaps because the means are truly equal) even when the distributions are otherwise very different.
"Accepts" doesn't mean it handles them. You should always check the source code of a function.
Thanks, I discovered it after my edit too!
Yes. In general you don't have to do it this way because the lag lengths don't have to be the same for x and y. See:

http://en.wikipedia.org/wiki/Granger_causality#Mathematical_statement

However the R command granger.test uses p past values of x AND y. With this specification underlying the test you need try using IC with n past values of x AND y versus IC with n+1 past values of x AND y.
I think I will probably end up shuffling all my data first and then do the 2901/n with remainder randomly assigned to partitions.
You can't fit a model that has more variables than observations. There are not enough degrees of freedom for parameter estimation.
Please do not erase your question and replace it w/ something else. CV is intended to build a permanent repository of information about statistics via archived questions & answers. You are welcome to edit your Q and add updated info below the original question, but don't just erase what you initially asked. You may want to read our [FAQ](http://stats.stackexchange.com/faq).
(+1) Thanks for the edit: that makes the idea much clearer.  You might want to make the example match the counts, though: it seems the example concerns *three* users and *three* sites rather than four of each.  BTW, how do you determine which sites to use?  Nobody is active on all, or even a majority, of the sites.  This is a problem because some people get 1 rep point to start with and others get 101.  If we insist $f$ be constant in the interval $[1,101]$ to deal with this, then it's impossible for $f$ to be concave and increasing for any larger values!
@Xi'an You would need $nb$ fairly small rather than $b\\approx 0$ before it would be appropriate to use a uniform distribution, because the acceptance rate is $(1- e^{-(n+1)b})/((n+1)(1-e^{-b}))$ $\\approx (1-\\exp(-n b))/(n b)$, which will be inefficiently low when $nb \\gg 1$.
@B_Miner, I guess it is, but neural networks are not statistical so I don't know the framework or process for choosing the lag.
+1 For a valiant attempt at deciphering the question and providing good advice.
Welcome to the site, @simonkarp. Is this intended as an *answer* to the OP's question, or to assist in making the question more easily answerable? Please only use the "Your Answer" field to provide answers. If this is an answer, can you clarify how it answers the question? Since you're new here, you may want to read our [about page](http://stats.stackexchange.com/about), which contains information for new users.
Cross-posting *on several SE sites* is not recommended: there exist facilities to migrate question from one site to another if it happens one site is better suited for the question at hand. Cross-posting on different forums, Q&A sites, mailing, etc., is another issue: It's always a good idea to reference other posts and keep every actors up to date with replies on either site. Otherwise, there is a risk of overlapping threads, incomplete replies due to intervening updates to the question, etc. In all cases, we are messing up google queries, but hopefully CrossValidated will come first :-)
There is far too much here to address. To get a handle on these issues you will need to learn about representing geographic data (projections and coordinate systems); and then about spatio-temporal data analysis, which is a kind of marriage of time series analysis (which you need to know very well) and spatial data analysis (geostatistics); and have a deep understanding of climate models. Those things will take anybody years to learn and more years to learn well, but if you absolutely must do it yourself, take a look at Cressie & Wikle, *Statistics for Spatio-Temporal Data* (J. Wiley, 2011).
Thank you very much for the advise, I will think about this.. I agree that I need a statistician but: 1. I underestimated the problem, the previous smaller questions were easier and could be handled by ourselves; b. I'm in a progrma for a master in public health with special in epidemiology, the advanced statistics modules are only in 2011 and I wanted to have an idea now; c. I like to understand my statistical friend when we are discussing this. I'm a professor ath the University of Antwerp, we even have a epidemiology and biostatistical department ;-) (politics....)
In your setup $Y_{it}$ cannot be stationary, since its mean depends on $t$.
The second example is confusing because $x$ appears to play two distinct roles, both of data and parameter. It also is not apparent what you mean by "terms" being "dropped" when those terms have not explicitly appeared anywhere. I suspect you might just be a victim of sloppy notation, in which case being more precise in the notation and in how you carry out the calculations may answer the question for you.
Do you use Occam's Razor? How do you justify Occam's Razor without using a Bayesian approach?
Anyway, the reason I don't think you need EP is because EP is a generalization of BP.  In BP, the goal is to eliminate any nodes in your Bayesian Network that are either not observed or not accounted for in the prior you seek.  You have no such nodes.