tttjjj/elt_bench huggingface dataset:
I have seen things saying that bootstrapping quantiles is the best way to get their SE, but I don't remember where I saw this
Not only because have I answered your question :-), but if you are satisfied from the answers, I think you should choose one of them. Thanks.
@darcy.mulder AIC has one term proportional to the log-likelihood and a bias-correction term proportional to the number of parameters. So you can just sum the AICs for each model to get an overall AIC.  (If you're using the second-order AICc, sum the log-likelihoods & recalculate the bias-correction term based on the total no. parameters & the overall sample size.) You might also want to see if you could fit some common parameter across all datasets.
+1 Thanks for the Answer. Hmm, I see what you mean. The more similar the test sets are the more similar their RMSEP values will be. OK, put that way it is the same as if the data were spatially or temporarily correlated. The way I generate the training sets / test sets should mean that on average they are all as dissimilar from one another. I'm not sure what CV would get me here - and in a sense I am doing that anyway just via a resampling approach. Will probably ask another Q then on how to solve the real problem.
The odds ratio estimates will not be independent and are probably correlated because some of the same data has gone into both estimates. Therefore, treating the statistics as independent and calculating a two-sample statistic comparing the two as if they were independent is not correct - that's all I'm saying.
I believe this is there way of controlling for overly optimistic judges and overly pessimistic judges.
Your question isn't about R syntax, it's about what nesting means.  Nesting B in A (with A/B) creates two variables, A and the interaction between A and B, which is exactly what you describe.
@Macro If all samples are independent, then the value of the third sample is irrelevant in the case the first two agree.  Therefore the distribution of the results truly is Binomial, as @Ben supposes, and the calculations should be carried out as if the third sample were always available.  (Pascal and Fermat discussed [exactly this question](http://en.wikipedia.org/wiki/Problem_of_points) in the summer of 1654.)
However, I have the feeling that this is subjective, argumentative and will require extended discussion please read http://stats.stackexchange.com/faq
I vote to close but encourage you to ask a more specific question (since the idea of the question is good but way too wide).
That makes sense, but then you should be modeling how correlated you expect them to be. Maybe you draw their correlation coefficient from a Beta distributions whose curvature is assumed to have a functional dependence on the time between the polling, or something. Then you run your analysis and get some posterior distribution over current opinion inferred from poll opinion. You check this against your data and see if it gives sensible results. If not, it could tell you that your model of census data is off, or your assumed correlation model is off.
+1 ... but of course, if you come to do inference (intervals or testing), you might assume normality then.
Nope. $K$ is the kernel. $\\Phi$ doesn't have a specific name I'm aware of. Bishop calls it a 'fixed feature-space transform'.
BTW, the penalized R package (http://j.mp/bdQ0Rp) includes l1/l2 penalized estimation for Generalized Linear and Cox models.
Thank you @Error404 and Peter. I read about [single imputation at wikipedia](http://en.wikipedia.org/wiki/Imputation_(statistics)#Single_imputation) and found replacing a missing value with the mean as the simplest method, but also noted the risk. However, I believe you are right - it ruins the correlation between two variables that might covary. When dropping that person the _p-value_ is still less than 0.05 so I will just go with that and explain that I had a missing score which explains the lower _df_.
@whuber, you nailed it.  That page references beanplots - which I had never heard of before!!!  That's the suggestion I was looking for - a beanplot.
+1 for the article reference. Although I do not code in SAS, I read it several months ago and found it to be a nice, high level treatment of the issue.
In the source file, it is reported that "Note that if F > c_v we reject the null hypothesis that y does not Granger Cause x". Please clarify your question if it is not a MATLAB issue but rather a statistical question.
The data $X$ is itself a random variable in general. $E(Y|X)$ is the Hilbert space projection of $Y$ onto "functions of X", thus is the best one can do in the $L^2$-sense.
@user3880985 Not really. You could use gradient descent to find the best model coefficients, but here there's a closed-form solution instead.
@Gung The use of different variables in each group seems to make this a different problem. user31478 Why do you think it makes sense  to compare correlations for different variables? What would be a model for comparing correlations on different variables?
Look at ?contrasts, ?C, ?contr.sum, and the contrasts part of ?options.
Currently, I use two-months (June and July) average as columns, but I have daily data too.
a box and whisker plot might be a nice place to start. http://www.r-bloggers.com/summarising-data-using-box-and-whisker-plots/  https://stat.ethz.ch/R-manual/R-devel/library/graphics/html/boxplot.html
Great explanation. Thanks, Glen_b.
Well, I have the crime figures for all crimes (a sum of burglaries, muggings, theft of car, from car, etc) over a 12 month period from 3 areas in Swansea and 2 in Oswestry. But I'm not comparing those three areas individually, but a sum of them to compare the two places rather than areas. The only reason there are 3 and 2 is because that's how many policing districts there are in each place that I'm looking at. And I am looking at comparing Oswestry and Swansea crime figures as a whole to see if  there is a statistically significant difference?
Have you read the [help page](http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/Simple/html/simple.sim.html) for `simple.sim`?
Probably I'm just way too long out of school, but doesn't p-value (0.507) higher than the significance level (0.05) mean that we can't reject the null hypothesis?
By dichotomizing predicted probabilities you are losing information, then you are making it worse by using a conservative test of association.  And maximizing the sum of sensitivity and specificity means that you have abdicated specification of a utility/loss/cost function.
Well done!  That idea sounds good, understanding that a "normal density plot" would be something like superimposing a graph of a Normal density function on a histogram or other plot of the raw data, in effect providing a *parametric* visual reference for evaluating the data.
I think I'm missing something, because it looks like you are giving two different ways of constructing the series $X_k$ (and they lead to different results): the method before the "and" and the method after it.  You cannot do both at the same time!  Which one do you want to implement?
Thanks :) hope this question isn't a misuse. It seems like there's a blurry line between stackoverflow and crossvalidated...
It sounds as if the offset is the parameter of most interest. With a logit model, response the proportion, predictors $x$ and $x^2$, the position of the maximum is $-$coefficient of $x$/(2 * coefficient of $x^2)$. This is just the usual calculus argument that $a + bx + cx^2$ has zero derivative when $x = -b/2c$; we need to check that the value of $x$ really is a maximum. For your data I get $-.03574713$ from a Gaussian logit model. Your R code gives $-0.03636307$. I've given more d.p. than is sensible, but the agreement between slightly different procedures looks good.
Exhaustive enumeration *always* works.  Just be sure to choose test problems where this approach will yield an answer in reasonable time!
Alright, thanks. I've validated your answer and gave you the bounty for your troubles!
I guess I'll go with that. Now I need to figure out how to correctly calculate CV(RMSE) and such metrics correctly.
In that case I have 3 parameters for the calculation of Rsquare adjusted right? HDD, CDD and Days
You forgot to divide by $N$ when computing the last formula: the expectation is the sum of values *each weighted by its probability.*
Please try to clarify/explain your intent further.
Is your interest in the data or in writing the program?  If it's the first, then forget about Python: use a statistics package and learn how to do linear regression, because it's far more than just a formula: you need capabilities of data manipulation, re-expression, and visualization; ways to obtain regression statistics and residuals; computations of regression diagnostics; and more ways to visualize and check the results.
Have you checked out the answers to this question - http://stats.stackexchange.com/questions/3984/aic-and-sc-value?
+1, I think this is a clear, thorough & constructive answer. I have 1 nitpick / concern in the final paragraph re the use of the AIC. An issue I have w/ the various information criteria is that they assume all parameters contribute equally to the ability of the model to fit the data. When evaluating different multiple regression models, I think this is fine; however if examining different types of distributions, it is not clear to me a-priori that all parameters afford equal flexibility. Thus, I am uncomfortable w/ that. What is your position on this issue?
Note that you can add and subtract any constant $c$ from the terms inside the $\\exp()$ the above expression without changing the result.  setting $c$ equal to the negative of the maximum value of $\\log(p(\\theta|-))$ provides the best numerical accuracy
See here for Type II testing with `gls` fitted models: http://stats.stackexchange.com/a/53041/8402
The treatment area selection was motivated by past production reports, and limited by what plots were available. The offset wells are all within about five miles of the test wells I am studying, and target the same formations, assuming the reports are accurate.
*Forecasting* should not be a problem. Cross validation looks a bit more interesting. Could you edit your question to clarify what exactly you are interested in?
Rather than try to give my own answer I think there are really many, many examples that you can find in books that deal with counting processes.  Let me recommend that you look at Joe Hilbe's book on the [negative binomial regression](http://www.amazon.com/Negative-Binomial-Regression-Joseph-Hilbe/dp/0521198151/ref=sr_1_1?s=books&ie=UTF8&qid=1348140235&sr=1-1&keywords=negative+binomial+regression).
I basically wanted to do something like [this](http://www.biomedcentral.com/1471-2199/9/66), where nonlinear models are used to describe gene expression profiles. I just randomly selected the example data since I thought it looked like  the function in question. I'll have another, proper look at the data. Thanks for the suggestions.
Hi - the data is also available in the form Lizard 1,Y,N,N,GroupA, Lizard2,Y,Y,N,GroupB <-- Ie presence/absence only for each lizard. Correct me if I'm wrong, but if presence/absence were random, then if GroupX had 100 lizards, we would predict a distribution of 50,50,50... So is there not some way to see whether an observed distribution of 30,60,60 is significant?
In the absence of any explanation of what all these variables and matrices mean, all one can do is guess--and one reasonable guess is that everything being asked is a direct result of some (hidden) definition.
Note to future readers: The sphericity correction is equally applicable to the other solution as well.
Fitting high order polynomials is rarely a good idea. There are alternatives, but I'd like to understand what you are trying to do better first.
No, you should stop it and do something else.
"Note that when using non-normalized vectors, you essentially put a high weight on the document size. This is why TF-IDF is commonly used: a document and the concatenation of two copies of the same document then have the same vector." TF-IDF is the basic element for forming document vectors whether normalized or not normalized.It is used as a measure of relevance of a particular term.When it comes to comparing two documents,we use cosine sim since its independent of document length and would produce same vectors for the example you mentioned.
How do I model these things eg public holidays and weekends as an initial start to try to forecast at the daily level. My data has a seasonal trend that goes for a year, then there is a weekly trend with weekends needed to be zero and public holidays also needing to be zero. The dates for these events are known. I am not sure how to model ARIMA or ETS to take these events into consideration. Do I use xreg? It is very hard to find any useful articles or tutorials on doing this on the web. Any pointers in the correct direction would be appreciated.
I meant using BIC only if the assumption, **the true model is in the model space**, is justified. May be it is justified in your case. I agree with you that people use BIC like AIC, but both are very different things!
What's "CC"? Do you mean 'correlation coefficient'?
Hey thanks. So say I am using data on say, bonds whose time scale is by minutes. If i have say, 300 or so data points (so 300 minutes), do you think checking the prior one is enough? Also would you happen to know of my previous post about SE? http://stats.stackexchange.com/posts/103868/revisions
@Srikant Thank you.  I was thinking along similar lines when I posed the question and have since tested out a GLM (Poisson distribution with *linear* link) as well as some other models.  Unfortunately, it now looks like any model based solely on land cover type and proportion will not work well: a sample of these data suggests that population patterns depend on a larger spatial context.  At a minimum, then, we would need to include spatially lagged covariates in a linear model.
Isn't the first term reflecting the best strategy, and therefore the best arm, over the period? To take the mean, you'd just multiple by 1/T, but you would need to multiply both terms by that to equivalize them. I'm not sure where you're stuck.
**Clarification-2** In the 3rd query I want to know if cut points will affect the regression results. I explained why I am thinking of categorizing (I do not know the number of trials). The DV ranges from 0 to 0.75. The median is 0.10. Quartiles are: 0.05 (25th percentile), 0.10 (50th percentile), 0.17 (75th percentile).
I should have stated that I assumed they used `Surv`, as they don't actually say anything beyond that they used R. I also corrected the typo for `surv` vs. `Surv`
gotcha, thanks for clarifying.
I'm afraid you just must convert 20news-19997.tar.gz on your own.
I have tweaked your title so that it is more specific to what I gather you want to know. I think you are more likely to get the info you need this way. Feel free to change it back if you disagree. Note also that this thread has automatically become CW due to the high number of edits. If you don't want it to be CW, flag it for moderator attention; it should be possible to reverse that (I think).
That is what I was thinking! Thank you for the confirmation!
I agree with you. Substantive plausibility is the most important thing.
Thank you for providing this comment!  But unless you can quantify the amount of skewness, that fact by itself is not very useful.  Plenty of distributional families are skewed but have practical normal approximations, such as the Chi-square (Gamma) and Poisson, and plenty more can be strongly skewed but rendered close to (or exactly) Normal through a simple re-expression of the variable, such as the Lognormal.  Could you perhaps amplify your answer to explain how the knowledge of the skewness could be used to estimate p-values from reported ORs?
I don't. But it is not hard. Just evaluate the performance of your system both on train and test data, at each iteration.
Each one of these terms has multiple, strikingly different meanings depending on context and application.  Please edit your question to mention the context in which you would use these words.
(+1): A very interesting question
Another use of KL-divergence worth mentioning is in hypothesis testing. Assume $X_1, X_2, \\ldots$ are iid from measures with density either $p_0$ or $p_1$. Let $T_n = n^{-1} \\sum_{i=1}^n \\log( p_1(X_i) / p_0(X_i) )$. By Neyman--Pearson, an optimal test rejects when $T_n$ is large. Now, under $p_0$, $T_n \\to -D(p_0 \\,\\vert\\vert\\, p_1)$ in probability and under $p_1$, $T_n \\to D(p_1 \\,\\vert\\vert\\, p_0)$. Since $D(\\cdot \\,\\vert\\vert\\, \\cdot)$ is nonnegative, the implication is that using the rule $T_n > 0$ to reject $p_0$ is asymptotically perfect.
Not all of the criticisms of P-values are correct or warranted, despite their vehemence. You should see these two papers for a few counterpoints to the commentaries that you cite: Two Cheers for P Values (by Stephen Senn) http://www.phil.vt.edu/dmayo/conference_2010/Senn%20Two%20Cheers%20Paper.pdf; To P or Not To P (by me) http://arxiv.org/abs/1311.0081
You could, but you'd be making your life more complicated. Look at the answer to this [question](http://stats.stackexchange.com/a/45831/603)
I believe this reqiures a difference in the degrees of freedom between the two models (one is a reduced version of the other). Am I wrong? My question is to do with comparing two models when the DF are the same
Interesting, thanks.  I wonder though if it's not the Fisher that's more appropriate here -- since in the urn formulation of the problem, N is observed after the experiment, and you condition on the sum.  Quite likely, I'm one of the "confused" the wikipedia article refers to. :-)
Well, you have been putting various issues into one. The data you have seems to make more sense on a logarithmic scale. Whether you want to do binning or not is a separate question, and there I only have another face palm reply for you: depends on your data and on what you want to achieve. Then there is another hidden question: how do I compute the difference between intervals - compute the difference of their means? or the minimal distance (then A to B would be 0, B to C would be 0, but A to C not). etc.
Well, if you're hoping to find an exact solution that doesn't involve special functions(e.g. the gamma function), I think you're going to remain stuck. The actual solution is $p = 1 - \\frac{ \\gamma(k/2, \\lambda/2) }{\\Gamma(k/2)}$, where $\\gamma(\\cdot,\\cdot)$ denotes the lower incomplete gamma function. Both $\\Gamma$ and $\\gamma$ are defined in terms of integrals, so any calculator you program for your website will need to involve a numerical integration routine.
Correct on XY variables. Your suggestion is what I was thinking. I was wondering if there was a specified approach. As you might expect, the data points start off close together between X=15-40, with Y between 90-100 (low poverty have higher scores). There appears to be another section that spreads apart more between X=40-75, and an even larger spread between X=75-100. But it seems really smooth
Full disclosure: I work for the Intel group that created the analytics toolkit, which I think is quite good. GraphLab is also very good, well-designed software!
Does this fall under unsupervised learning? We have combined effect  (0/1) of Merchant skills and Product score (both ordinal say 1-100), and we don't observe (and have no idea) about the ordering of the 2 predictors.
Thank you, great point: I think it really adds to the discussion that the outcome is foremost dependent on the chosen process.
Though, now I'm wondering about the loss of information that accompanies stationary transformations.  What about when the actual non-differenced values matter?  For instance, if you have a time series of debt and GDP, then difference both, the ML method can only make learn the effects of the changes, and the effects may be different if debt is 10% of GDP vs 200%. (of course, you can add a GDP/Debt feature).
I don't use R, but you can get advice on that. You might need to ask separately.
What is the 0.5 for in "corr=corAR1(0.5,form=~1)?"
I would like to identify the "standard test item" for statistical graphics, if one exists, to include in training material I am developing.
You coud use logistic or ordinal logistic regression, but if Linear regression isn't useful I don't think logistic or ordinal logistic will be. I think people from data mining/machine learning area may help you here. My guess is that other variables (Related to individual characteristics of raters) can be more relevant.
It depends on why the simulation is being performed. For instance, if you are doing it to study the distribution of some function of $Y$, then you can achieve greater accuracy by simulating $R$ *exactly* (with no sampling error). Simply generate $n(n+1)/2$ iid realizations of $X$ and compute the sums of the first $n$ of them, then the next $n-1$, ..., and so on down to the sum of the last one.  This gives $n$ realizations of $Y$, for an average cost of $(n+1)/2$ uniforms per realization.
@whuber I posted the formula as a ready-made integral-calculating tool, nothing more, a work for which it is valid. No interpretational strings attached!
how do you get "$\\binom{4}{2}+3$"?
They're both nominal, so 'correlation' probably isn't the right word. Something like 'association'; there are [quite a few measures of association](http://en.wikipedia.org/wiki/List_of_analyses_of_categorical_data#Measures_of_association). These include phi, Cramer's V, Goodman & Kruskal's lambda, Theil's U ...
So if we put it in terms of higher order convergence we see that although parametric and nonparametric bootstrap are of the same order of convergence (i think thats whats written in van der vaarts asymptotic statistics), parametric is still better. but only in terms of some factor?
Thanks! I had in mind the former case and I like the idea of multiplying by the prior. Could you perhaps suggest any links to relevant papers/textbooks?
Readers here will find the following thread of interest: [If the mean is so sensitive, why use it in the first place?](http://stats.stackexchange.com/q/14210/7290)
I think it can stay here, the question is more relevant for statisticians, than for programmers.
user1800340 - a non-graded question from an already graded homework assignment certainly counts as `self-study` (it's the kind of question that makes it that, not just whether you get a grade for it). Would you mind adding the tag (and since self-study questions are treated specially, could you also perhaps take a look through the [tag wiki info](http://stats.stackexchange.com/tags/self-study/info))?
This is the particle MCMC method developed in the references provided in my answer. I cannot explain it in sufficient details on this forum, I am afraid. The overall principle is however to aim at a posterior based on $i$ observations at the $i$th step of the algorithm, using only the density of the $i$th observation in the update. See also Chopin (Biometrika, 2002, 89, 539-552).
(+1) The paragraph on "acceptable residuals" is very well-put.
I added the requested info to the original post.  It should be noted that each subject is run on the same complete task, such that all variables are within-subject with the exception of **Dx** and **SubID**.
could you pls elaborate more about the details of your data.
Interesting, and if I was showing this to a set of statisticians, mathmaticians, etc. this would certainly be a good way. But I was looking for a way that required less stats explanation. @Peter's Trellis plot is easier for a biologist to understand.
I second H.White's book as a _reference book of results_ regarding asymptotic theory for least-squares and Instrumental variables estimators, (It has not been written with education in mind), because it is very clearly organized in covering gradually different stochastic set ups (deterministic regressors, non-determinsitic, i.i.d. non-identical, non-independent etc.).
@JandR - probably, but usually you'd be explaining $f$ further anyway, and it'd be better to be precise (though more verbose) in saying "$X$ is a random variable with pdf $f$."