tttjjj/elt_bench huggingface dataset:
Maybe this is obvious for everyone else, but: power (in this context) refers to binary hypothesis tests (*p* values), not to the precision of the estimation of the coefficient/effect size.
Alright, I will give implementation a try and see what happens then. Thanks a lot
The paper (as the second law) deals with closed systems. Quantum mechanics is completely reversible on a closed system (i.e. all operators are unitary). The only non-reversible operation in quantum mechanics is measurement; if you measure a closed system then it is no longer closed from the thermodynamics perspective. If your observer is inside the system, and measures a sub-system, then the observer+sub-system evolve unitarily together, and thus the operation is invertible (this trick is informally called the "Church of the larger Hilbert Space"). Thus, your argument from "QM" is wrong.
Greg and gung have done a nice job of trying to explain a complicated phenomenon succintly.  To go deeper into the topic, you could read up on *suppressor variables* and *partial correlation*.
Positive autocorrelation can also result from other systematic effects such as a learning process. I think it's important to remove such effects before interpreting auto-correlation as evidence of a "hot streak".
Did you mean to say "DATA1" twice in "*where you treat DATA1 as a population, and DATA1 as your sample*"?
Note that the variance you can *reduce* by running large numbers of bootstrap / cross validation iterations/repetitions is only the part of variance that comes from instability of the surrogate models. You can measure whether this is a major contribution to the total variance by cross validation as it tests each sample exactly once during each run, so the variance due to finite sample size does not show up in the comparison of the averages of complete cross validation runs. For "hard" classification, you can calculate the variance due to the finite sample size from the binomial distribution.
It sounds like you're doing a repeated measures or mixed ANOVA.
I am encountering an error on the last function. Its saying  "Error: could not find function "jeffries.matusita""
Fascinating question, and it seems like this is a problem still being debated. Here's an interesting paper on the subject http://mpra.ub.uni-muenchen.de/31840/
Let $x=2$ for instance. The chance of not observing a 5 in the first draw is $9/10$.  The chance of not observing 5 in the second draw is $9/10$.  Because the draws are physically independent and the contents of the bag are the same each time, the results are statistically independent.  Therefore the chance of *both* outcomes is the product $9/10\\times 9/10$.  Consequently the chance that you *did* draw a 5 at least once within $x=2$ attempts equals $1 - (9/10)^2$.  (You are correct that this analysis has appeared before on our site, but it's going to be hard to search for it... .)
This could be helpful but not an introductory text on statistics.
I guess I didn't even think about different data types. I guess that's the non-statistician in me. :) Most data I work with is of a numerical type (pressure, temperature, etc.), but I guess my example isn't so cut and dried. Thanks for the reply.
The 0.5 doesn't sound wrong, but I don't see why standard deviation is informative in this case.  Also, are you sure the "high variance" concern doesn't apply to prediction accuracy as measured by something other than this binary variable?
Your title and text seem to contradict to each other: are there no events in the outcome variable, or is there an input (predictor) variable for which one potential value has not been observed? Please clarify.
@Alexis,No, i disagreethis is not a duplicate question. , can you clarify where in your previous answer address the question of probability of survival curve dropping to zero? In addition, this question is about subscribers the other was about patients both have different dynamics even though they use Kaplan Meir survival analysis
It's unclear whether anybody can do any relevant computations until the O.P. clears up the meaning of "the die is biased."  I don't see any reference to "equally likely" in the question, either.
I think a different way to refer to the same thing @Glen_b is referring to is a binomial glm, of which logistic regression is a special case. Though, I think you'd want the general version of the model, not logistic regression. That is unless the the unit of analysis is the questions asked and you want to know if some person-question characteristics determine whether the question was answered.
Although I think I have a conventional understanding of most of the words in this question, I'm unsure what it's getting after. Could you state a little more precisely what you mean by "sampling" and what exactly would be "optimized"? You seem to assume implicitly that your readers have in mind a particular setting in which a "distribution" (or family thereof?) is involved and in which a particular objective is assumed, but one can only guess at what you really intend when you make such broad statements as those appearing in the last paragraph.
Could you convert to an answer and I will accept?
@Germaniawerks 'material significance' would generally be understood in the intended sense. I've seen 'practical significance' used, and a few other similar terms, but that one should do fine.
Could you please explain whether you want to physically partition the data into two pieces for separate analysis or whether you might (perhaps) want to use key as a variable in the analysis?
I'm teaching out that book this semester.  It may be a good *statistics* book, but I'm having second thoughts regarding its light probability content.
I added a picture that includes the original question and graph.
@Steve I used *Mathematica*, beginning with the playing card representations [on the SE Mathematica site](http://mathematica.stackexchange.com/a/16114/91).  I tabulated an outer product of an abbreviated list of the cards, where the "product" function combines a pair of randomly rotated card images to represent a two-card hand.
For models when you only observe the effect when it is about a certain threshold you could use Heckman's selection model. It was devised in labor economics where the productivity of an individual (the salary the job market would offer them) can only be observed if they have a paying job. That would happen only if the wage is higher than what the individual considers a sufficient amount for him/her to take a job (and give up free time).
I don't see this as learning n things and testing m, since the m lines are consecutive -- and poetry is learned consecutively. I see it as chunking the poem into n/m chunks. You pick one chunk at random. If I were the teacher, I would have more confidence if I started the child near the end of the poem, rather than the beginning.
Could you say a little more about the details of your context? Is it time series data? are the predictor variables experimentally manipulated or just observed? What is the aim of your analysis: control, prediction, something else?
@mnel I meant to have x/z.  I just corrected it.
@TylerStreeter Looks that way and that is why i corrected it.  There is no Y mentioned anywhere else.
@Ivo I am sorry, there were a lot of mistakes indeed. I corrected a lot of them.
@user43153 Hmm, that is weird. From the output you posted it looks like there should be only 2 random effects parameters for each of the initial models, while, as I expected, `model` containes the additional correlation parameters which drive the difference. You might try using `||` instead of `|` in your call to produce `model`. Without a reproducible example it now becomes difficult to see what is going on.
Typo here @Gung: it is homosc. that implies that variance is the same. Strictly, homosc. is an assumption about errors, or conditional distributions, not residuals.
You can't really evaluate most of the assumptions of ANOVA/ANCOVA/Regression before running the routine. Most of them are about the residuals
`predict(fit, data.frame(QD=78), type="fitted.ind")` gives predicted class probabilities, what you got is the value of the linear predictor, see [this answer](http://stats.stackexchange.com/a/41025/1909).
Glad that it worked for you.
@thinlyveiledquestionmark I'd like to play poker with you... but only if I'm allowed to deal. I think Robert means that the realization of 100 H in 100 trials would shift his belief from the coin being fair to the coin being unfair. Given this data 100 H in 100 trials, you would have to be possessed of a very strong prior on $\\Pr(H)$ to not appreciably shift the posterior.
Thanks. I will check out those papers now.
Agreed, actually working with the entire population of interest is probably quite rare, and probably doesn't apply in the case of IQ testing.
@Glen I would assume it could be dealt in the way one handles weighted regression right? $\\sum_i w_i (\\hat{f}(x_i)-f(x_i))^2$ where $w_i \\propto \\frac{1}{v_i}$
Thank you, so now i have to compare two populations with sample size 9 and 8, one of them follows a normal distribution while the other does not. How do i compare these, using a non-parametric Mann-Whitney would do ?
Set $\\varepsilon =0$ in the rightmost expression to find its numerical value. To verify it, (I don't really understand why Huber does not use simply the ratio of variances): The variance of $d_n$, I have already provided. For the variance of $s_n$, in the post http://stats.stackexchange.com/questions/105337/asymptotic-distribution-of-sample-variance-of-non-normal-sample, you will find the asymptotic variance of the sample variance. Then a) check the wikipedia page to find to what $\\mu_4$ is equal to when the distribution is normal, and b) apply the Delta method to find the AsVar for $s_n$.
However, we can already tell you that classifier optimization is far from a simple problem. Frank Harrell will soon also tell you that accuracy (or other measures based on the proportion of successes / failures) are not a goot target value for optimization and that you need so-called proper scoring rules instead. Have a look around here, there are actually quite a few optimization questions already on this page.
"I've decided that I only want to use 4 parameters."  That's the wrong way to approach things, choose the # of parameters via principle (you know the form of the model generating the data), or a data-driven approach (BIC, AIC, cross-validation, etc), not arbitrarily.
If you don't want to do variable selection (which will hurt predictive discrimination) but just want good predictions, there are many packages available.  For example ridge (L2 norm) penalized logistic or ols regression can be done using the R `rms` package.
Thank you @Andy
Also, do you think that PCLS could help answer my open question, currently worth 100 bounty?

http://stats.stackexchange.com/questions/56113/principled-ways-of-constraining-ey-0-when-one-of-your-regressors-rightarro
@DanielC Your error will be probably caused by different line of code than this one...
I would reccomend that you consider software designed for statisics rather than calculating your own statistics in Excel or using the opaque calculations in the analysis tool pack. Also see: http://stats.stackexchange.com/questions/3392/excel-as-a-statistics-workbench
Sorry, I don't think I got it: the first model is $Y_i = \\beta_i X_i + \\epsilon$? So for every sample $X_i$ (you call them bins) we have different coefficients $\\beta_i$? I don't think it's the case. And even after that, am I right that the second model is $Y_i = \\beta \\hat{Y}_i + \\epsilon$, where $\\hat{Y}$ is an output of the first predictor?
"While feedforward networks are used to learn datasets like (i,t) where i and t are vectors (eg i∈Rn, for recurrent networks i will always be a sequence, e.g. i∈(Rn)∗ " I know this question was asked a long time ago, but would you mind explaining what this means in layman's terms? I need to justify the reason I chose to use a feed forward network over a RNN for my project, and I think this may be the reason. And/Or, could you link me to a source that will enable me to make this justification? After searching I cant find one myself. Thanks, any answer is much appreciated!
1) looks like a request for code. 2) implies that your regression is a very poor fit. What kind of advice do you think we can give? You need a better model or data more suitable for your research objectives. Alternatively, tell us much more and you may get better advice. For example, what model are you fitting with what data? What correlation matrix do you get for your variables, etc.
Hmmm... this leads me to something else: Do you have a nice example, where you don't have unbiasdness for small samples, but asymptotic unbiasdness? I have problems to see why this can happen...
Certainly insignificant effects can have a dramatic impact; indeed if there's high correlation with some of the main effects, you could even expect some coefficients could change dramatically and the interaction be insignificant.
@ Glen_b, I couldn't just say the math behind it shows this, is there tests to check across a subset of the data?
Since you only have 40 examples, the error rate has to move in 100/40% steps (all correct: 100%, one wrong: 97.5%, two wrong: 95%, etc). I think @DikranMarsupial is suggesting that the AUROC will be less sensitive to this than the variance of the error rate. Seems reasonable to me, since you could get many possible shapes with 40 control points.
if the problem is convex... this is exactly my question
The assertion that algorithm design depends on knowing information theory (sensu Shannon) is puzzling. Please spell out your argument. Also, the abbreviation "ML" is ambiguous as between "machine learning" (which you probably mean) and "maximum likelihood" (the interpretation that will probably spring to mind to statistically-minded people outside machine learning).
@Dov Yes, they are "more or less" like each other, but I was simply saying that the problem of choosing the number of clusters is much more fraught than choosing the number of PCs - i.e. they're not "equivalent".
This can be mean squared error obtained via cross-validation. Cross-validation is very useful trick, I'd suggest you to find its explanation in web - that would be more clear than explanations of mine. In R there is function ```cv.glm()``` in package ```boot``` that can help you.
@Luis, yes I wholeheartedly agree with you that there is more than one approach, however, I was trying to make the point that $X$ and $X^2$ have two different distributions. I guess at the end of the day I should have phrased it as "we could calculate new distribution of our new random variable f(X)."
[R](http://cran.r-project.org/)
The skewness I take as a secondary issue; the major issue is just the lack of relation between rating and response time. For example, the mean rating from these summaries is lowest for time 3 days at 9.265 and highest for 10 days at 9.379! I'd plot summary statistics for each time rather than try to force this into any kind of regression model, because the total fit is likely to be disappointing.
1. A polynomial model is a linear model, because it is linear in the coefficient. 2. I am trying to understand your question. If the refrigeration system has been modified between the time R1 and R2 were used, then they are really not the 'same refrigeration system' (line 1), right? 3. Why is it in your second approach, you started comparing the coefficients of S? 4. Have you consider introducing a covariate 'refrigerants' with levels R1 and R2 into the polynomial fit (maybe with interaction)? Its coefficient might answer the question.
@PeterEllis so, to put it in your words: "D is the estimated time to complete the tasks on the critical path, based on the confidence interval specified in Z"
@user721975 I ordered the colors. If you skip this step, the color increases from left to right. Another option is to provide a vector of colors (vector-length=number of bars to plot) by using rep().
+1 to both your answers - these are interesting examples. You seem to assume that the frequentist solution must be likelihood-based though, seemingly ignoring nonparametric methods. In this particular case, the nonparametric frequentist methods of Chapter 5 in the book that you linked to seem to me to be at least as easy to use as the Bayesian methods.
@Nick Lehman is a standard intro graduate-level textbook for courses on estimation (and the first edition has long been a classic). A more accessible introduction that covers the same conceptual material but with less reliance on the mathematical details is Kiefer's *Introduction to Statistical Inference.* The complication in your question is that it is unclear how one ought to measure the difference between an estimator of a function and the function itself (there are many ways). In particular, the lecture slides are opaque concerning what the *bias* might mean in this context.
It may be faster to write your own cosine dissimilarity function.
With such a small number of samples and the large variance you have very low power to test for differences among groups. Also keep in mind that you have no power to test for normality.
WHat you call 'weak' depends on circumstances. Your association is so unlikely to have occurred just by random variation that it would be reasonable to conclude that it must be due to some other cause. If the assumptions hold, you're left with concluding some level of monotonic association is present.
This "Your test is indicating no effect." is not quite accurate, and veers towards the fallacy of accepting the null hypothesis. Evidence for no effects would require equivalence tests.
Do you think that a simple t-test works to compare the rates for first and last 5-years interval, and if so is this a valid method?
Well, taking into account that the covariance matrix needs to be positive definite, it does not seem that logical to me to use whatever proposal distribution. The proposed matrices need to be positive definite. One option would be to have as proposal the Wishart posterior conditional used in Gibbs sampling, however this did not seem to work particularly well when I assumed a student's-t for the random effects. Hence my question, are there other types of proposals for covariance matrices?
#2 also appears to works with a [GAM](http://stats.stackexchange.com/q/35510/4485)
+1 based on the edits.  A word to the unwary: even the bootstrapping hides some subtle pitfalls.  It helps to start with a clear statement of the probability model underlying the prediction.
Thanks for your input. I like the winsorized mean idea. This would definitely work better than a median value. I have thought about using rent between `$X` - `$Y`.. will give more thought.
I know essentially nothing about financial statistics, but surely the OP's data **aren't** time series?
+1, thanks so much for you kind help. I actually read the GPML book you linked, and it also does the same thing, it derives the predictive distribution by computing the Gaussian conditional.  As you said, computing Gaussian marginal isn't Bayesian at all, and the very thing which makes GP a Bayesian approach is that, *it specifies a Prior, and the Gaussian conditional acts as the Posterior*, right?
@ Ben Bolker, Thank you very much for your expertise, I really appreciate it. I will edit my original post to include more details.
I currently do. I wanted to check some alternatives to it.
@Nick I agree that the trends (pre-2004 and post-2004 separately) are more linear for the logs.  Some method to handle the heteroscedasticity would be essential for making forecasts: without that, forecasts based on the untransformed data would have optimistically narrow prediction intervals whereas forecasts based on logs would have pessimistically wide intervals. A short-term forecast may be robust to deviations from linearity--making cube roots a good first choice--but making longer-term forecasts does require close attention to the trend, initially favoring the log as you claim.
As you imply, and as stated in ttnphns' comment in another answer, one problem is what people mean by "Don't Know". Depending on how the questions are phrased and how the person interprets it, this could be the "I can't decide but you want me to fill something in" answer, or it could be the "I actually don't know", or something else.
Hi Momo, this is great. Thanks a lot for the not-magic-happens answer. Now I understand the idea! At least I can think of a couple of questions now to ask and learn about. I just dint know of what to address in certain scenarios. I get the picture. Let me read thro' the concepts of glm once more now and will get back with more questions. Thanks a ton again!
It looks like the unemployment rate for 2009 and 2010 are situated pretty far from the bulk of the data points (7 and 10 vs 3s and 4s). I'd say a different measure, one less sensitive to such vari'ations in value, would be good to try. For starters, you can try Spearman's correlation. It is correlation computed on ranks.
@StéphaneLaurent Thanks, that is interesting.
@Ga I see: You created a 300 x 300 matrix for the data! The inefficiency of this code fooled me: it would be simpler and more direct to use `closedp.0(Y, dfreq=TRUE, dtype="nbcap", t=300)' where Y is the frequency matrix {{1,234}, {2,27}, {3,4}} (which you computed twice and actually displayed!).  More to the point, the convergence failures are alarming, suggesting there are problems with the underlying code or models.  (An exhaustive search of the [docs](http://cran.r-project.org/web/packages/Rcapture/Rcapture.pdf) for "M0" turns up no references or description for this method...)
Wikipedia has a section on parameter estimation that lets you avoid too much work :)
Thank you for response @michaelchernick. The estimator is obtained using an iterative procedure, so there's really no closed form for it. I can understand your comment about the confidence interval: we're looking for a confidence interval for $\\hat{\\beta} x$ (assume 1-d) and $\\hat{\\beta}$ is asymptotically normal, and $Var(\\hat{\\beta}x) = x^2Var(\\hat{\\beta})$ and from here we can get the confidence interval for $\\hat{\\beta}x$. But without any idea about the distribution of $u$, I can't see  how to get the a (approx) confidence interval for $\\hat{\\beta} x + u$ (i.e. a prediction interval).
The sample size cannot exceed the population size. And standard error and standard deviation aren't comparable in terms of accuracy.
Here is a useful article on the topic: http://cacm.acm.org/magazines/2009/10/42353-overcoming-the-j-shaped-distribution-of-product-reviews/abstract
I guess you wanted to generate the random data from truncated normal distribution (truncated at 5).
I'd interpret it to mean "not linear (in the sense of being curved, not 'linear in parameters') at least continuous and in some sense, smooth" (where smooth might mean something like 'continuous first derivative' perhaps, but there may be other definitions that would feel like they were in keeping with the sense of the word). So I wouldn't call a linear spline 'curvilinear', but I'd certainly call a cubic spline 'curvilinear' (even though it's linear in the sense that it can be fitted with linear regression).
The phrase '[burn-in](http://en.wikipedia.org/wiki/Burn-in)' doesn't imply 'burning' the samples, but letting the system satisfactorily settle out of its "new" state to one where it's suitable for use. (The samples are discarded during the settling in, yes, but the phrase doesn't imply 'burning' in that sense. It's nearer to 'warming up the engine' before taking it on the road.) See [this answer](http://stats.stackexchange.com/questions/88256/what-does-sampling-period-mean-in-mcmc/88262#88262) for some discussion of what it's for.
I am interested in the case when $T_i=f(X_1,...,X_n)$ for known $f$. This was what I meant when I said that $T_i$ are sample statistics.
Do you mean that, at time $t$, $Y_t = 4-(X_{t-1} + X_{t-2} + X_{t-3} + X_{t-4})$? This would be well defined for times $>5$. (Analysis of how it is associated with heart attack incidence is perhaps best asked in another question)