tttjjj/elt_bench huggingface dataset:
TAT is turnaround time. I have tried only a hit and trail method.. The results are decent. but i want to validate with a mathematical model.
in case of normal data, produce pearson. in case of non-normal data, produce spearman.
I wish I could give more than +1 for this very thorough answer. I particularly liked you mentioning your issue with a data leak as it effectively illustrates that it can be far from trivial to rule out such problems, even for experts. This is a good reality check!
@berkay You're welcome. I drew the tree in PowerPoint, but I'm sure that there are better tools for this around. I think [Bayes theorem](http://en.wikipedia.org/wiki/Bayes'_theorem) lies at the heart of the question. See [this PDF](http://college.cengage.com/mathematics/larson/calculus_applied_life/1e/resources/app_c/1021662_App_C4.pdf) for a nice tutorial.
It's simple: The support of the gamma distribution is $(0, \\infty)$ and the data contain zeros.
Can the RF algo predict a continuous output variable? The examples that I've looked at tend to just do binary classification
Another way to solve this that you may find interesting is to use the independence of A and B to get $\\rm{P}(A\\cap B|B\\cup C)$ = $\\rm{P}(A|B\\cup C)\\rm{P}(B|B\\cup C)$. Then, using independence of A, B, C, $\\rm{P}(A|B\\cup C) = \\rm{P}(A)$, so you have $\\rm{P}(A|B\\cup C)\\rm{P}(B|B\\cup C) = \\rm{P}(A)\\rm{P}(B|B\\cup C)$. And, as before, applying $\\rm{P}(B\\cup C) = \\rm{P}(B) + \\rm{P}(C) - \\rm{P}(B \\cap C)$ gives you the same final result.
Welcome to Cross Validated! You might want to explain the situation a little more to make sure the answers are helpful. As I understand it you don't know any parameters, but you have an estimate of one - the population mean. You also have a confidence interval for it. If you know how both were calculated from the sample you may be able to estimate the population standard deviation.
Thank you for the thorough answer. Although I asked for an algebraic answer I really wanted to understand the result intuitively so I especially appreciated the logical argument alongside the algebra.
As always, *it depends*. If you're only trying to comprehend the basics, being able to find extrema of functions gets you a fair way (though in many practical cases of MLE, the L is M'd numerically, in which case you need some other skills as well as some basic calculus).
Are you wanting to derive its distribution or just compute its mean value and standard deviation?
Oh, this one's frustrating, indeed. :)
Bootstrap does not require any assumption about the distribution of errors. It uses the data itself to figure this out.
John Tukey, who invented the approach from which this method appears derived, used *two* multipliers: He set one "fence" at 1.5 times (an analog of) the IQR away from each quartile and another fence at 3 times the IQR from each quartile. Values beyond the first fence were "out" and values beyond the second were considered "far out" (those who remember the '60s will understand this terminology). If you think you need more extreme fences, then most likely you should consider *re-expressing* your data rather than changing the fences.
why will you want to select a sub-optimal when you indeed know the whole ROC?
"It is useful to note that the likelihood ratio only makes sense where the hypotheses in question come from the same statistical model" - what does that mean exactly? It sounds like you're saying you can't compare models from different families of distributions, which ain't so.
(+1) This is a solid answer. I would suggest making mention of the key term "shrinkage" and drawing connections to regularization in the classical/MLE context: that is, this estimation procedure allows some estimates to be precisely zero, rather than ML fits which might be noisily estimated somewhere around zero.
If you were looking for an R package, `arules` would be worth a look. Maybe "association rules" is a good search term
The ggplot2 docs might be helpful: http://docs.ggplot2.org/current/stat_qq.html
Maybe this could be of help: http://www.bristol.ac.uk/cmm/learning/videos/correlation.html
What you are searching for is, in my understanding, not the correlation of the variables but the correlation of the errors, right?
It will do. Thanks.
There is no assumption for the _distribution_ of the independent variables, but if you simply add that variable to your model you do assume _linearity of its effect_. If you have that many 0s, then it is reasonable to suspect that there is something special about that value, and you'll probably want to incorporate that in your model. Here is a discussion on how to do that: http://stats.stackexchange.com/questions/56306/
Where might your lack of confidence be coming from?  Do you believe the device generating the data is defective?
And then there is not random variables. The distributions are not random (are normal), the parameters of the distributions are not random (are fixed). If you not have random variables how can you put prior distributions?
And I guess by the average x and y is E(x) and E(y), not sure how to link that to the regression line though.
Thanks for all your wonderful advice; it’s much appreciated! My colleague [posted some more info here](http://stats.stackexchange.com/questions/5913/interpreting-two-sided-two-sample-welch-t-test/5961#5961); does this affect anything?
Are you using Stata rather than "State"? There is no "weibreg" in base R or in the recommended survival package, so your reference to a "canned function" is ambiguous.
(+1) Up to the invocation of Stirling's approximation (which is irrelevant), this is *exactly* the bound I was (opaquely) referencing in my comment to the OP. (For example, [see here](http://stats.stackexchange.com/questions/20083/density-of-normal-distribution-as-dimensions-increase#comment36294_20084).)
Ah apologies, that would be participant (or subject)
+1 for `dplyr`. It has made so many R tasks simple and many of these methods obsolete.
The algorithm is not the critical point. The values you base your AIC on e.g., deviance, likelihood, or $G^2$, obviously need to be the same. And this is what I perhaps meant with my last sentence.
This looks quite similar to some of the processes considered in recent credit-risk research. They tend to consider processes that, I believe, are a bit more complicated. There was a student at Cornell (where S. Resnick also resides) named Kay Giesecke that wrote about such things. That might be a place to start. Darrell Duffie may also look at similar processes, though I haven't followed his work very closely.
Thanks for your insightful comment(s)! Somehow the Wikipedia article, which mentioned something about self-adjointness of the projection operator threw me off, as your proofs are not that difficult. :) BTW, do you have a favorite linear algebra text that deals with this kind of stuff?
That question has already been covered by the first sentence of my first comment. As already explained, your ANOVA doesn't take account of the common patient IDs (e.g. doesn't use repeated measures), so *no*.
@Glen_b Thanks I edited the question and added the direct link.
So much litterature on this: you might wanna start here: http://www.cs.berkeley.edu/~pliang/cs294-spring08/lectures/time/slides.pdf
@Sverre What you described would itself be a good research direction. If we cannot quantify the IF itself, how do you know that there are different subfactors within IF, and they contribute differently? It is perhaps getting a bit philosophical, but if we cannot quantify an array of things, even very so arbitrarily, then perhaps we should not use "more" or "stronger" to compare them. If you decide to use statistics as a tool to study this, I'd suggest starting to look into how to [operationalize](http://en.wikipedia.org/wiki/Operationalization) your factors.
Hi RLang, Thats a really helpful page, thank you. Although my data is ordinal and there is no option for that there. Maybe I can use the non parametric equivalent of a multivariate multiple linear regression, whatever that is..
why would you use a *test* for *prediction*? - they're different tasks.
just a note, the estimators from the likelihood can also be biased, the classical example is likelihood variance estimate for normal sample.
@user542833 I'd never heard of that... makes sense now.
Are the 36 conditions matched across sessions? And did you do all this for ony one subject, or do you have more? (You did say "participants", plural.) If there are more, how many are there, and are the 36 conditions the same for every subject?
"Should I...?" questions seldom work well in this format.  Ongoing study of inferential statistics will tell you about the decisions others make in similar situations and their criteria for doing so.  For the short term, if you change your question to be about "What are the facts about...?" or "How does... work?" or "How can I accomplish...?" you will likely get some good answers.
For non-maximum likelihood, this is correct. However, the original post utilized a logistic regression model for the primary analysis which is a maximum likelihood technique for a fully parametric probability model for the outcome. Because of this, complete case analyses are unbiased with MAR data.
I completely missed the bit about `Mclust()` using GMM. If I'd seen that (or not been so infatuated with the fact that it reported BIC), I'd never have posted this question...
@jbowman (see edit) is that what you meant? or do you mean that I need to change it after that?
I am not sure whether I agree. Suppose you have two patches containing stars. In one patch they were far away or the atmospheric conditions were suboptimal so the stars are less bright. On the other patch the stars were near and the photo was taken in a clear night, so the image is brighter. You might be interested in normalizing them onto a common scale (I think this is basically what's done in auto-level and auto-contrast functions in Photoshop or other image processing programs).
Perhaps using an option like `pch="."` in the call to `plot` in $R$ will make the density of points easier to see.
+1 @juampa Thank you for your help! =) I'm using k-nearest neighbor method and regularized linear regression methods. Possibly also trying multilayer perceptron.
@user904522 you don't need training data for clustering
It would be surprising to me if it wasn't already implemented. It's in R, for example - the Mann-Whitney is done with `wilcox.test` and you just add the argument `conf.int=TRUE` and you get the HL estimate and an interval (which you can change the coverage for via another argument).
This should be community-wiki.
What do you mean by bayesian network program?
Also [here](http://stats.stackexchange.com/questions/60410/normality-of-dependent-variable-normality-of-residuals).
Read Belsley: Conditioning diagnostics, collinearity and weak data in regression. Or, look up my dissertation Flom, Collinearity diagnostics for multiple regression: A Monte Carlo Study. Belsley also wrote another book that covers this, but is more general.
What's the York method? Googling doesn't seem to help.
And it is available for free download at the authors page: http://www.inference.phy.cam.ac.uk/mackay/itila/book.html
I think you probably should specify which introductions you actually had a chance to look, in what way they were incomplete and what topics you want to be covered which they did not.
Don't know if this is 100% right, but it's what I'm doing as well. Yay for making things easy in R!
Of course. Now fixed.
In my case, a "site" refers to a geographic location; sites will be separated by kilometers at a minimum. To sample means to collect a few liters of seawater, perhaps in a bucket. To measure each replicate of each parameter, I then collect a few milliliters from the bucket and do the measurement. Does that clarify?
Thanks lejohn. Great reply.
Homework [gets special treatment](http://stats.stackexchange.com/faq) and full solutions are generally discouraged, but helpful hints are (very) welcome. We generally expect to see some effort from questioners to show what they've done and where they are stuck. This is to prevent the site from simply becoming a homework-solution repository. If users post verbatim homework questions with no attempt at elaboration, it usually ends up either ignored or downvoted, particularly if the behavior is repetitive. Homework should also be tagged `homework` so that others know to treat it as such.
It isn't explicitly stated, but I assume that you are trying to find an MLE.

Your result really depends entirely on your parameter space, your likelihood function and your needs (aka, best isn't well defined).  

If you are just looking for theoretical justification such as asymptotic efficiency; under Le'Cam conditions you can just use the one-step MLE (Under the further assumption is that you are using Newton's Method and the score function for your gradient descent).  This requires that your initial value is such that $n^{1/2}\\hat{\\theta}_0 \\rightarrow \\theta$ in probability.
I have never really understood the rule of thumb that says "10 cases for each predictor" (and unfortunately I don't have access to the book written by Agresti). What I mean is: if I have 100 subjects of which 10 are cases (the `1`'s) and 90 non-cases (the `0`'s), then the rule says "include only 1 predictor". But what if I model the `0`'s instead of the `1`'s and then I take the reciprocal of the estimated odds ratios? Would I be allowed to include 9 predictors? That makes no sense to me.
Thank you for your extensive response. I will think it over and check out the references.
I agree that they all come from sloppiness. I had also mentioned that they are not theoretical at all (having said that they're operational). I've edited my post a little bit.
Answering a question like "how often" would require an estimate (point, interval, or both), not a test. How do you define 'similar'?
We haven't covered Order Statistics yet, any way to do without that?
@Christoph_J, You have that $y$ today is correlated with $y$ yesterday; your ACF shows that. The $x$ today is likely correlated with $x$ yesterday. The $x$ yesterday is correlated with the $y$ yesterday. As such, the included predictor $x$ is correlated with an omitted variable $y_{t-1}$ and you have an omitted variables bias issue. Differencing breaks those correlations and averts the omitted variables problem.
What kind of response is the `hired` variable? How does it relate to hiring probability?
One possibility: you might stack the values into a vector and use vector autoregressive models (VAR models).
Search on the "change-point" tag for some leads: http://stats.stackexchange.com/questions/tagged/change-point
Yes $V$ does indeed relate on how much each document is related to the topic but it does not have to be strictly positive. I think you just need to read into what $X = U S V^T$ actually means in terms of Linear Algebra; everything will immediately fall into place after that. The Wikipedia article on http://en.wikipedia.org/wiki/Singular_Value_Decomposition is quite good to start you off.
@NickCox No, I think the OP means 'identity' there --  "against $g_1(x) = x$ rather than $g_2(x)=log(x)$... that is, "the identity transformation" of $x$.
your first paragraph is really weird...I am not really sure what you want. But in general, to get a C.I. for the parameter estimates, assuming you are using package like `nlme`, you can just use `intervals(your.model)`.
It turns out that this is not always completely true. There has been some (very) recent work showing that consistent parameters can be estimated for alternatives that you never observe, provided you have some exogenous information on what the actual population frequency is. But this requires a different estimator, so in general you are correct.
@John thanks, but this doesn't help me. I mean, lets assume I have an exponentially weighted moving average, so I have an estimate for my $\\sigma$. How do I now fit the distribution properly? If I have Gaussian, I can just plug in the $\\sigma$, but in case of e.g. an hyperbolic distribution? I need this, since I want to afterwards calculate the quantile of the fitted distribution which uses the modelled volatility to compute the Value at Risk.
Several related questions of potential interest: (**1**) [A proof involving properties of moment generating functions](http://stats.stackexchange.com/q/34956/2970), (**2**) [Inversion of mgfs](http://stats.stackexchange.com/q/29017/2970) and (**3**) [Existence of the moment generating function](http://stats.stackexchange.com/q/32706/2970).
Since we are dealing partly in concepts and partly in semantics, I'll point out that in your second sentence you've said "...statements of that nature will be true..." without specifying what statements would be true.
Try to do dimensionality reduction to 2d/3d and draw your clusters.
Not because it answers your question but because they are connected: http://stats.stackexchange.com/questions/665/whats-the-difference-between-probability-and-statistics/675#675 (I really like the answer of Mark/Peter)  and http://stats.stackexchange.com/questions/2641/what-is-the-difference-between-likelihood-and-probability
@MOLi N indeed is the number of documents. p(x,y) is the probability of x and y occurring together. To convert the count to a probability, you will need to divide by the number of documents. If (x,y) occurs in each document then p(x,y) will be equal to 1, which makes sense.
thanks! p.adjust with "fdr" method uses the Benjamini–Hochberg procedure.
Hi jojer, welcome to CV. We look for more thank link-only answers. Please see [this help page](http://stats.stackexchange.com/help/how-to-answer) (under **provide context for links**), and [this](http://meta.stackexchange.com/a/8259/202463) meta-answer. Please click 'edit' under your post to add some context.
Are you wanting to do an analysis of number of sales (i.e count data) or value of sales? I'm thinking of a repeated measures design, but it would depend on the variables you want to use, and your dependent variable.
**Hint:** For simplicity, assume that $f$ is symmetric about $0$. You can then show that $E[X]=u=0$ by splitting the integral between $(-\\infty,0)$ and $[0,\\infty)$ and using the symmetry assumption. Then you just have to show that $E[X^k]=0$ for $k=3,5,7,9,...$. This can be done again by splitting the integral and using a similar argument.
@Josiane That's correct.  In fact, the expectation of the number of $X_i$ less than $c$ is at most $pn + O(\\sqrt{n})$, so if $p$ changes with $n$ like $O(1/n)$, the number of such $X_i$ is a constant plus $O(n^{-1/2})$.
If some of the answers to your questions have given you the information that you needed, you may want to consider accepting one of them by clicking the 'check' mark next to it; it's a nice thank you and recognition of someone who's helped you.
+1.  But of course PCA "differentially weights" the variables: that's what the coefficients of the first principal component do!  But perhaps you mean "weight" in a different sense?
@Peter Flom. Thanks for your comments. It is probably a failure of my explanation, but I don't think your suggestion gets me there. The reference category (S1) is akin to "no treatment," and S2, S3, S4 are increasingly severe treatments. They (2,3,4) should all have a negative effect on y (which is censored at zero). The reason for the mixed results for S2 and S3 is probably that a)they are not terribly effective, and b)they are probably more likely to be put in place where y is high. Switching the reference category would, I suspect, just confound explanation.
It is good to mention it, but it should be double said that those are examples of misusing or overtrusting CV, not some drawbacks of the method itself.
@Nick the first few paragraphs of the [Wikipedia article](http://en.wikipedia.org/wiki/Noncentral_chi-square_distribution) provide the details.  By looking at the characteristic functions you can establish that a similar result is *not* available when not all variances are the same or there are some correlations.
Aside from the last sentence, which is trivial, this is essentially a duplicate of: http://stats.stackexchange.com/questions/18433/how-do-you-calculate-the-probability-density-function-of-the-maximum-of-a-sample
Note that GS *is* a form of QR decomposition.
@whuber added info to my answer to try and explain further.
Thanks! Yes, that is I think what I need (noisy have many peaks, clean have few peaks). Ill try what you suggested! Cheers!
By 'randomly placed' do you mean 'with equal probability', among those with space for at least one more ball?
You must clarify what do you mean by "random guessing": Do you mean $A$) "I will flip a fair (50%-50%) coin in order to decide whether I will predict that an observation is "good" or "bad"" or $B$) "I will take the sample proportions are adequate estimates of the underlying probabilities, and so I will construct a coin where the "bad" side will have a 75% probability and the "good" side will have 25% probability of landing, and then start flipping this coin"?
A priori: no it doesn't relate to an increase of 10% in `X`. There is a special case: If `X` is log-transformed and `Y` not, then 1% increase in `X` is associated with a $\\beta/100$ unit increase in `Y` (see [this post](http://stats.stackexchange.com/questions/18480/interpretation-of-log-transformed-predictor) for more information).
If he chooses art together with French, does he choose math?  If he doesn't choose art together with French, does he choose math?  If he chooses math, does he also choose either art or French?
I edited the question.
A very detailed and complete answer, thanks! Matrix notation sometimes does hide simple things such as sample means and weighted sample means. $Q = (H′H)^{−1}H′X(I−H(H′C^{−1}H)^{−1}H′C^{−1})$ does seem incredibly obscure. However, I'm glad my intuition was correct in that GLS can be decomponsed in such a way, regardless if $X$ is invertible or not. Anyway, thanks again!
ets uses a state space model. Presumably you mean a more complex state space model.
(+1) Thanks and welcome to CrossValidated. No need to post it as a comment (which assumes you have enough rep); it's a response on its own. I always appreciate your input on [r-sig-mixed](https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models). Nice to see you there.
@chi Thank you, I had seen that post. Do you think I am on the right track with my thought process at least? It seems that an independent test set allows us to be more liberal in our use of CV for feature selection and model tuning / selection. Otherwise, nested loops appear required to train, tune and estimate error generalization all using the same training data.
Not an `R` question
@JCWong I think you can use a nonparametric Bayesian variant (the infinite hidden Markov model) to avoid that.
Even in the Gaussian setting the log likelihood is nonlinear in its parameters :-).
"Please note that a sample in statistics has a specific meaning. It doesn't mean a single data value." Yes I know that. Outlier(s) refers to a single value (or quite limited number of values), but I'm looking for the correct term to use when the problem relates to the entire sample, not a single value...
@Sarwate: Looks like the matrix notation has obscured the calculation this time. Thanks.
No, I would not be persuaded by any such demonstration, if that were *all* that is offered. It is unable to distinguish between the claimed result and a result in which there is a very small amount of contamination from a nonzero distribution. Any computer simulation, to be truly persuasive, must be accompanied by reasoning that would rule out such phenomena.  (I recently conducted a series of simulations that went out to a sample size of $10^{1000}$--that's not a typo--but was still not persuaded by the results, although they were very suggestive!)
@Dan I asked and nobody replied so far.
Thanks for the quick reply. I appreciate the feedback as I continue to learn the R syntax. I tried your full and reduced models and got similar results, even though there was only one observation per transect per year. At least these results correspond well to the plotted raw data. Thanks again.