tttjjj/elt_bench huggingface dataset:
I've not seen results from hurdle models 'integrated'; what does that look like?  Do you have examples of papers that do that?  (It might be helpful to link to the papers you found that have two separate models as well.)
Please correct me if I am wrong. Or maybe tell me how to derive $b$ above ? Thanks
There's a better place for this than SO: http://stats.stackexchange.com/
@Peter Unfortunately, a lot of elementary textbooks simply repeat incorrect information from other textbooks without actually doing any real investigation themselves, and so a basic misconception gets propagated. Counterexamples are, as you see, relatively easy to construct (I just make them by hand as needed). Kendall and Stuart (*Advanced theory of Statistics, Vol I* - don't let the title put you off, it's quite readable), at least the third and fourth editions, have good information. More recent editions are by Stuart and Ord. I've actually posted about this issue on CV a number of times.
Hello Gregg! I don't really understand what's going on in this plot. Thanks so far!
What is Silverman's formula, exactly?
The analysis is good (+1), but you should divide the row sum by the sum of all entries to obtain something meaningful: row sums by themselves tell you nothing whatsoever about the "contribution" of the variable to the variance of the sum.
It's a bit beyond my abilities to do that, but I've written to the authors to see what they think.
I would say no to your question. For example, if P(data|H0) is zero (the observed data could not have occurred under H0) then P(H0|data) is also zero.
@AnonyMousse `@ttnphns answer refers to pairwise Euclidean distances!` In my answer, 1st paragraph, I clearly refer _both_ to "SS error" (direct) and "pairwise d^2" (implicit) interpretations.
Corrected my question.
So you need to find a counter-example ... what clearly ancillary statistic can you find from the sample minimum & maximum?
A big thanks for your answer, but this is EXACTLY the problem, because the parameterization of this hessian is for log(delta) and log(zeta) and not for delta and zeta! See my follow-up posts: http://stats.stackexchange.com/questions/67595/variance-covariance-matrix-of-the-parameter-estimates-wrongly-calculated and especially the answer of CT Zhu here http://stats.stackexchange.com/questions/67602/the-vcov-function-cannot-be-applied
You really should find out how the data were obtained. How could they be paired if there are different sample sizes? Anyway, if you don't know which red observation is paired with which blue one, then you can't take into account the pairing in the analysis.
@Michelle They are both fed from the same batch of food.  I can't control how much of the food is consumed, but if two organisms consume the same amount, they will get the same dosage.
Thanks a lot, that's useful.
I think your second example, with the hazard rate, is terrific and something I did not know
We can try to fix the English--that shouldn't be a problem--but there are problems with the mathematical expressions in the questions.  When $y = ab/c + x$, obviously $y$ **is** "calculated from" $x$.  In this case the correlation of a set of $(x,y)$ would equal $1$.  In the second question it is mysteriously redundant to write $y=P+Q+x$; you might just as well write $y=P(x)$.  These solecisms suggest you might not have properly expressed the questions that you really want answered.  Please clarify these points or we'll have to close the question as unanswerable.
Note that likelihood ratios are asymptotic, i.e. don't account for uncertainty in the estimate of the residual variance ...
Glad to have helped, but I'm sorry, I don't know of a javascript implementation. I do everything in R.
It's worth pointing out that in the context of linear regression with normal errors, the frequentist prediction intervals are based on pivotal statistics rather than plug-in estimators and are identical to the Bayesian intervals under the typical noninformative priors (jointly flat on the $\\beta$s and $\\mathrm{log}(\\sigma^2)$).
Please define "EOF".
@Bakaburg To each his own, but it truly seems to be inordinately complex to write such a loop than to add additional columns. It seems that you could retain your columns with the free-text categories and also have the additional binary columns. Though redundant, both representations are useful for different ends.
@Lyra that's how you tell data.table what function you want it to use to operate on the columns.
The [help file](http://cran.r-project.org/web/packages/effects/effects.pdf) of the package `effects` states (page 8) that for continuous covariates, the values are determined by `grid.pretty`. Alternatively, you can set the levels of `arc` manually using the argument `xlevels`. For example: If you want the effect plots for the quantiles of `arc`, you could use the option `xlevels=list(arc=quantile(mydata$arc, seq(0, 1, 0.25)))`.
@Ryan (If you want to address @whuber directly, don't forget to prefix its username with `@` so that he can be notified.) I don't really like to intervene like this, but what is your "original" correlation about? I mean, a perm. test will inform you about the prob. of observing such a stat. ($r$ in your case) at least as extreme as the one you observed in your data (and with 10,000 draws, this is largely enough to conclude at a 5% level); then reporting your $p$-value is just a matter of computing the No. times you observed $r>r_\\text{obs}$, where $r_\\text{obs}$ = your empirical corr.
You seem to be saying "I have this procedure that inputs a $k$ and a $d$ and outputs a number (the mean of $\\mathbf{v}$).  How can I maximize the output?"  This question is so generic that it is not answerable.  You can obtain useful answers by providing information about the mathematical behavior of this process.
I am so sorry, it's a typo; I meant gender:shopping habits as the interaction variable. I edited the question.
+1, nice answer, @rolando2, very clear. In addition to your good points, I think the SE is better understood as a measure of the [precision](http://en.wikipedia.org/wiki/Accuracy_and_precision) of the estimate, than its accuracy.
Thanks for your reply. I can understand that readers could be stopped by the length of the code. The code is long but the idea was simple : 1. to compute the se of the expected value with a Wald approach. 2. to use this se to randomly simple 1000 values from a Gaussian distribution with the expected value as mu and se as sigma. 3. for each random value, to randomly sample 1000 values from a negative binomial distribution with the exponential of the random value as mu. 4. The prediction interval was computed as the 2.5 & 97.5 quantile of the 1e6 random values.
Does this methodology makes sense?
I think this would be better in Stack Overflow.
What assumptions does this solution make? To which problems is it reliably applicable?
It would be unusual (although not impossible) to conduct both of these tests, because a Poisson distribution is discrete and a Normal distribution is not.  However, your code does not look even lexically correct--there are mis-matched quote marks--so it is of only limited help in assessing what you are really trying to do.  Perhaps you would be better off abandoning this question and posting one that asks for advice about how to analyze your data.
@smillig `extractAIC` uses different methods for `lm` fits and `glm` fits, i.e., `extractAIC.lm` and `extractAIC.glm`. You can use `getAnywhere` to study their code. `AIC` uses the same method for both.
Note that for extremely large data sets variog has the option (max.dist).  This will exclude pairs beyond a particular distance from inclusion in the empirical variogram calculations.  This is a fairly reasonable approach since points sufficiently far out should be uncorrelated.
On 2) I believe the confusion is what p(X=.01) means when X is a continuous random variable.  Intuitively, the probability seem to be zero everywhere because there is no chance X is exactly .01. The questioner should review the definition of a density function in the continuous case, which is defined as the derivative of the cumulative density function.
It's not obvious to me how latent Dirichlet allocation (topic modeling) could be applied here. The OPs data are continuous signals not categorical variables, so it's not clear what the "words" or the "documents" would be.
I much doubt that you can prove Serfling wrong by simulation. Moreover, if you are interested in the mean and the median of a Student distribution, you should've sampled from the Student, not from the normal. So if you needed to create say a sample of $N=10$ from $t(5)$, you should've sampled 50 Gaussians and take the mean of the first 5, the next 5, etc. to produce 10 data points.
What speaks against optimizing the Kernel parameters via gradient based methods?
"*Are the resulting values the same as using ML?*" Quite plainly, no.
Thank very much for your cogent answer.  I am new to Caret but I love what I see so far (just bought the book too!).  One more question: I wasn't sure about specifying both trControl and sbfControl in a call to sbf.  What is the interpretation if I put "method='repeatedcv'" in both trainControl and sbfControl?
How many objects will be in the set?  How many objects will receive votes, how many will receive a considerable amount of votes?  Are votes eternal, or can they be retracted or changed?
Hi Daniel. Thanks for the answer.
I agree with you on what you said about _estimation_ (as I have commented in replying to @ziggystar in the first post), but how about _MAP_? 
In the first point, $P(\\theta|\\mathbf{x})$ is called the **posterior probability**, while in the third point, $P(\\mathbf{x}|\\theta)$ has the same name. Do you have an explanation for that?
http://math.stackexchange.com/questions/72975/variance-of-sample-variance
(+1) nice suggestions, and thanks for Kelley proper links ^_^
@traindriver You need to provide some more info about the inference problem you face in order for us to help you. You could expand your question with an "update" section specifying at least (1) is it the same quantity that is measured repeatedly? I.e. the same coordinate. Or is an area scanned or ... (2) why do you want to infer the mean and sd? If it's an area, it might be that you want to use SD as an estimate of bumpiness or something like that.
I learnt this one the hard way. Twice.
That, @cardinal, is a good answer! I wasn't merely being rhetorical, because the question itself asks "why do we say [the] Cauchy distribution has no mean?" Asserting the expectation is undefined may satisfy the uncurious, but the possibility that a reasonable alternative definition of the integral may exist--and yields an intuitively correct answer!--ought to trouble people. Your answer is close to what I had in mind, but it's still incomplete. I think a satisfactory answer would identify important theorems of statistical theory that fail when we work with conditionally convergent integrals.
Can you tell us what you are looking for?
Another less trival counter example is any distribution with mean less than $0 $.  Conditional expectation must be positive, but RHS is negative.
@Denis: I'm assuming that $\\sigma(x)$ is constant. @Dmitrij: yep, I basically just want to find the outliers. From a "minimize sum of squares" point of view, it seems like I shouldn't be scaling residuals to find outliers, but intuitively it seems like I should. @dominic: yep, like Dmitrij said, the two methods should lead to a much different ordering (especially if there's a large range in the x-axis).
but bagging is only about sampling from training set but not from the set of variables
"*how to test if the word occurs enough time to be significant*" -- I have no idea what this means. Can you avoid using statistical terms (like 'significant') and just explain what you want to find out from your data in ordinary English words? What's the basic research question (imagine you're explaining what you're trying to discover to your grandma or something)?
I understand links aren't really desirable on this site, but this is a standard reference that may be useful: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575184/
also the US census data http://www.census.gov/main/www/access.html
I would say data is the bullet, Statistics is the gun.
Categorical data can be ordinal
You'll want to look into standard errors.  For example, for your case what's relevant is the standard error of a proportion.  As the sample size doubles, the standard error decreases by a factor of the square root of two.  The larger the sample, and the smaller the standard error, the greater the precision of your point estimate, and the smaller your uncertainty (margin or error or confidence interval around that point estimate).  All of this assumes that you have taken random samples from the population, though.
Given my limited experience, I cannot give an exact answer. However, I believe that you can use a panel data (because you consider in your example variations within individuals and between individuals) approach with logit. Maybe others can elaborate on this...
But how does this translate into the calculation of the counter-factual value? For example, let us imagine that $ y_{1} = 10$ so the counter-factual for $ y_{1}$ is $10.5$. However, we know that the effect is positive. How in the absence of $x_{i}$ can the outcome be greater when the coefficient is positive?
Why the down-vote? If there's a mistake it'd be helpful to point it out - I can't see one.
McNemar test needs two variables with identical categories which is not your case. Use chi-square if you treat your variables as nominal or correlation (such as Kendall tau-c or Spearman) if you treat them as ordinal or polyserial correlation if anxiety is ordinal and criminal rate is scale.
"_The noise e is modelled as a 0 mean gaussian with ϕ as the precision_"..."_The noise variance ϕ also needs be inferred and is modelled using a gamma distribution_". Is $\\phi$ the noise precision or variance?
In `itsmr` library implementing the forecast of the process ARMA part for the models 1. and 2. requires the $\\eta_t$ to be zero mean (since $\\varepsilon_t$ is assumed to be zero mean). However the output from `arima` suggests that the residual is not zero-mean, i.e. $Y_t-X_t\\hat\\beta$ is not zero-mean. Do I need to augment the intercept term in estimated `Arima` object then, prior to making the forecast in jack-knives? Do I have to do the same for the ordinary predictions from `Arima`?
High correlations shouldn't matter. Are you using pairwise deletion of missing values? That can cause the covariance matrix to not be positive definite.
Can you indicate where you encountered this term?
+1, though you haven't spelled out why this last is better than using all possible subsets of predictors.
I think this analysis is making the same error that led to the Challenger disaster. If you only look at the failures then you have no reference group of non-collapsed bridges. (The ?Surv documentation makes it clear that left censoring is an option. The `status`-item is set to 2.)
Can you expand on this? Increased variance would require greater energy to "change", from that perspective the relationship is not inverse.
@suncoolsu: I guess it was a bad choice of words. I don't simulate real data, I simulate the experiment under the assumption the genome has nothing special in it.
So, I didn't go into it above, but I pesonnally define "overfitting" and "generalizing" to mean two different things.  I usually think of your model as "overfit" because it has a much higher error rate on new data.  However, I would view it as generalizing well still since it looks like it has double the accuracy of your base linear model (21% vs 40%).  I'd just run it again at maybe half the number of boosting steps and test it again.  Purists would suggest you make a third sample of your data so you can report a final "true" error rate after you use the first sample to tune your #steps.
What, precisely, do you mean by "probabilities from the distribution" (in your question) and "sampled probabilities" (in your comment)? When we *sample* things from a lognormal distribution, we observe (nonnegative) *numbers,* not probabilities; and when we estimate values, they are usually *parameters* like the geometric mean and geometric standard deviation (not probabilities).
Bayesian logistic regression is Bayesian statistics for the logistic regression model. So same question than @Glen_b: you are looking for an explanation of Bayesian statistics, of logistic regression, or something more precise (such as the choice of the Bayesian priors for the logistic regression model) ?
If you estimate this by Maximum Likelihood, can't you just compare loglikelihoods? Whichever distribution family has higher LL wins the tournament.
For nltk gaac it says the input is vectors but i have plain text in the form of tweets.
"*If they don't that means that there was a location shift*" -- no, it doesn't necessarily mean that. Unless there's something I missed, your logic is faulty in several ways at once.
As metric I think Mahalanobis distance can be used.
Is it clear now ? I have defined the two less usual functions
Your statistical terminology is confused. You are conflating group means with "intercepts". Not really a coding issue. Suggesting to moderators that it be moved to stats.exchange.
A possible solution consists of constructing the increments using a bivariate [copula](http://en.wikipedia.org/wiki/Copula_%28probability_theory%29) (*e.g.* Gaussian) with $\\alpha-$stable marginals with the desired parameters. Marginally, they will satisfy the assumptions you mention but the processes will be correlated if $\\rho\\neq 0$ in the case of a Gaussian copula.
Can this library find values of inverse CDFs for beta distributions?  Or can it find values of CDFs for binomial distributions?
@all 4.387627 = 3.5 + (-1)*(5-3.060660) or 4.387627 = 3.5 - (-1)*(5-3.060660) but this does not work.
Can you be more specific? Is your logistic model time-dependent?
Nice answer. Could you please provide commands to reproduce your beautiful figures?
@JMS: I've put the equations. Thanks
If you have more factors to include, you will need to use the second solution. You don't lose information. You just have to interpret the coefficients as relative to the missing level.
Fascinating question.
Ok, yes, you are right, my mistake, some covariance functions are still valid. I edit my previous post
"*small sample size isn't one of those problems it's robust against*" is worth a +1 on its own; the rest is a bonus
If you already have the population at hand, why would you use bootstrap samples to make inference about this population?
If you want its value at `x` you simply write `P(x)`. Note that `x` can be a vector (see the last couple of sentences of my answer.)
This is a simple problem made hard by truly awful notation (which, frankly, I don't even want to try to decipher).  The absolute value is not an issue due to the symmetry of the distribution around zero. Consider computing two conditional expectations: one for $y_3=l_3$ and another for $y_3=-l_3$, and then combining them.
I always have a feeling that about any model that is optimized can be seen as an energy based model. After all, you can always formulate the loss as energy. The thing is, what do EBMs give you? And that is to create a model with no negative data, because you "bootstrap" by generating confabulations via MCMC. I have no idea if EBMs have been applied to outlier detection, but it sounds reasonable to me. To the SVM thing: high dimensional is fine, I am worried about the small dataset.
@Josh Hemann yeah, i also figured this out in the mean time - but I think it might be better to compute it from the shape of A
@Ali Sarifi: That different p_value is though based on estimates of replicate variance based on sample size of just 3 - so unlikely to be robust. For ANOVA just think of strain as being a block in a randomised block experiment but I have disclaimed knowing that it is the least wrong way to do the analysis.
What are your hypotheses? Since the group factor has 4 levels, there are several possible contrast specifications.
I think we need an example to understand. Also, it depends on what you want to know.
Heard of it, but in this case, not relevant - application code is C++, the OpenFOAM CFD package, and runs on a super computer.
Thanks Aaron. I get it that alpha values for identical SVs can be added to get one alpha-value for each unique-SV. What's puzzling me are the **different** alphas values for each identical SVs. I'm trying to understand what would make one SV have a greater/smaller alpha value than another SV with an identical feature vector. Is this a SVMLight training artifact or does it make sense mathematically?
How do you calculate $p_k$?
Haven't tried that. Will try. Thanks.
I have made edits to your question to provide a citation for the book. Generally speaking, putting things like "I like [this one](http://www.amazon.com/Machine-Learning-Algorithmic-Perspective-Recognition/dp/1420067184)" in an answer is discouraged since if the link breaks, no one will know what "this one" was referring to. Cheers.
@tards I have individual level information = information about each product and its corresponding factors
The distribution of the statistic would be heavier-tailed than normal even if it *weren't* specifically the square root of a chi-square on its df; in that sense it wouldn't directly alter the answer to leave it out. But at the least it serves as an explanation for where the scaled-[chi distributions](http://en.wikipedia.org/wiki/Chi_distribution) in the diagram came from.
One way to deal with dependent data is multilevel models. In `R` see `lme4` or `nlme` packages
Can you add the imports at the top of the code? I am guessing probably pyplot, sklearn and PIL?
This question is perhaps too broad. Guessing what algorithms might work best comes down to understanding the nature of a data source and the workings of different algorithms to know which might work for the peculiarities of the data. The details of both parts (understanding the data and the algorithms) is literally the subject of many textbooks. I would recommend for instance 'The Elements of Statistical Learning', the entire contents of which would be a suitable answer for your question. Hopefully someone can provide you with a more succinct answer!
Is your null hypothesis that there are no differences between the 4 survival curves?
Just a quick note that because of the complex sample design, using the sample counts is probably bad, as they are likely to be unrepresentative of the population counts (ie biased).  This makes the comparison less meaningful as there is unquantified error in the standard error you report for $ P_{adj} $.
This is my favorite example too (HIV test), but unsure if conditional probability is too "advanced" given the introductory nature (plenty of studies showing that it's not too intuitive).  If you do teach this, I recommend perusing Gigerenzer and the frequency method: http://library.mpib-berlin.mpg.de/ft/gg/GG_How_1995.pdf
(-1) Sorry. Those are the correct functions, but your description of them is, sadly, incorrect. I'll remove the downvote if you correct this.
There's some potential confusion due to a strange use of "from." When we compute something *from* something else, we are using the something else in our computation. However, you don't seem to be asking about computing variances or SDs *from* a mean: you seem to be asking about computing them *for* a mean. "For" means that the variance or SD is computed from a *random sample* and that it is intended to *estimate* some property of the population being sampled. To that we can understand what you need, could you please rephrase your question using these conventional senses of "for" and "from"?
Your scepticism is based on reading too much into statistical testing.  An hypothesis test can falsify the null but never affirm it.  Demonstrating that two variables are not uncorrelated *a fortiori* demonstrates they are not independent.  That's all that is going on.
Quick note: Phi and Pearson correlation give the same value. Phi is just a simpler calculation procedure.