tttjjj/elt_bench huggingface dataset:
I think there is $1/\\sigma(X_t)$ missing in the likelihood function. In the limiting case when the parameters are constant we simply have a normal distribution for which Wikipedia gives a different anwswer: http://en.wikipedia.org/wiki/Maximum_likelihood#Continuous_distribution.2C_continuous_parameter_space
A t-test allows us to either reject or not reject the possibility that the mean equals a given value.  This is different to accepting that a mean is equal to a given value; to "not reject" is different to "accept".  Using a statistical power of 0.8 and an alpha significance of 0.05, and a "range of equivalence" used to calculate the estimated effect size, I understand it should be possible to estimate the number of participants needed.  I'm just not sure how this is actually done in practice.
"90% confident" is only half of a question.  The other half is, "with what precision"?  E.g., do you want to be 90% confident that the rate is less than 1%?  That it's within a factor of 2 of your estimate?  Something else?  In your circumstance it's impossible to be 90% confident that the rate is *exactly* equal to your estimate (except in a psychological sense :-).
Hi, Anony-Mousse. I think the first sentence in this answer could cause some confusion since PCA can be formulated as the solution to a couple of different (sequential) optimization problems. In that sense, the eigenvalues and corresponding eigenvectors are very much ordered. There is, particularly conceptually, a significant difference between PCA and simply finding the spectral decomposition of a covariance matrix.
I guess @Shane will have something to say here while talking about webvis (parser for protovis) since protovis (dedicated to web visualisation) (http://vis.stanford.edu/protovis/) contains the possibility of interactive graphics...
SGDClassifier in scikit-learn is *very* fast, but for linear SVMs. You might ask the scikit-learn guys, also add tag scikit-learn.
@gung All these statements have to taken somewhat metaphorically: after all, a "delta function" is not a function and it is (evocative) nonsense to assert it is "greater than any number at zero." Formally, what's going on is a process of completing a function space. Delta functions arise as limits of absolute continuous probability densities as their essential support shrinks to a point; improper distributions arise as limits of a.c. densities that are rescaled arbitrarily greatly. As such, they "integrate" to unity in almost the same sense a delta function does.
@Zen my remark did not mean that I didn't see the humor in your statement! I got it.  Cardinal is very good at checking for technical details.
Thank you @Flounderer, this is a great webpage and my searching did not turn it up.  It inspired me to search JSTOR as well, which currently has one paper that uses this word prior to the [query in Biometrics 5 1949](http://www.jstor.org/stable/3001892?seq=3).  This first use in a JSTOR paper is in [Proceedings of a Meeting of the Royal Statistical Society held on July 16th, 1946](http://www.jstor.org/stable/2981330?seq=7).
Could you explain what you mean by "Ratio and group information"?
+1 Amazing that no one mentioned Matlab before...
+1 Hahn & Meeker is one of the most worn books on my shelves.
I'm confused with your terminology. Could you give a specific example?
What do you mean by "predictive regression" and how would it be different from logistic regression?  What would be the response variable, for starters?
Great, and shouldn't $2\\sup_t|f(t)|$ still be in the final bounding equation or can we somehow get rid of it?
There is a much, much simpler proof which is fully general: since $V[X]+E[X]^2$ = $E[X^2]$ and $|X|\\le 1$, it is obvious that $E[X^2]\\le E[1\\cdot X] = E[X]$ and the result follows upon subtracting $E[X]^2$. (This is a trivial instance of [Holder's Inequality](http://en.wikipedia.org/wiki/H%C3%B6lder%27s_inequality), *inter alia*.) It can also be seen as an application of the Cauchy-Schwarz Inequality.
I think this would be more likely to be found in the psychology literature than the statistics literature.  You would think someone has done an experiment on this.  If not maybe you could do one of your own.
Can you say a little about what you have tried / understand so far?
Is the "value" you get a $p$-value or the test-statistic (in this case, t-value)? My hunch is that it is the former, and in that case you are _rejecting_ the null-hypothesis that the means are the same, which corresponds with your graph.
Good evening everyone, I need to know which page in MODERN FACTOR ANALYSIS, Harman wrote that changing sign is common. I have researched this problem and until now I not found answers.
@chl I second that. I really do not understand why this question has not been closed as a duplicate or made cw. I have already double posted an answer to a similar question because the consensus was "not a duplicate" and I am fine to triple post it if it is the same for this question.
Sure, no. I have ~35 000 observations (train set)
**#1** SEMs aim to establish causal relations from associations alone. **#2** SEM and regression are essentially equivalent. **#3** No causation without manipulation. **#4** The potential outcome framework is more principled than SEMs. **#5** SEMs are not equipped to handle nonlinear causal relationships. **#6** SEMs are less applicable to experiments with randomized treatments. **#7** SEM is not appropriate for mediation analysis. **#8** SEMs do not test any major part of the theory against the data.
Feature `featureSignif` produces the same as bkde2D- fhat matrix which I don't know, how to use...
What a great question! I would like this thesaurus, too. On the other hand, I think that there are often subtle nuances that may lay down some pitfalls. For example, I have herd the terms *hierarchical linear models*, *multilevel models*, *mixed models*, and *random effects models* (and possibly a few others) used as synonyms in ways that I understood, but I also can draw subtle (and perhaps not so subtly) distinctions between each of these terms.
@BenElizabethWard Yes I have, that is the very first thing I did. However, these variables have complex non-linear interactions with eachother so I'm looking for something else.
If X is Lognormal with parameters $\\mu$ and $\\sigma$, then the mean is not $\\mu$, and the standard deviation is not $\\sigma$. Conversely, if the mean is $\\mu$ and the standard deviation is $\\sigma$, then the parameters are not $\\mu$ and $\\sigma$.
First, two lines from wiki: "In computer science, semi-supervised learning is a class of machine learning techniques that make use of both labeled and unlabeled data for training - typically a small amount of labeled data with a large amount of unlabeled data. Semi-supervised learning falls between unsupervised learning (without any labeled training data) and supervised learning (with completely labeled training data)." Does that help?
1) Gower can take scale (quantitative), ordinal, binary, nominal attributes. If you can present text as either of these types, then it is OK. 2) Special distance measures have been proposed specifically for text.
One sometimes sees histograms with an overlaid distribution (e.g. most commonly in my experience, the normal distribution plotted using the sample mean and standard deviation.) Which is doing the same thing (and a bit more) as drawing a vertical line (indicating where sample mean is with the peak of the curve.)
@whuber: You raise good points. I didn't have ready access to the S&D paper either, so my remarks have been reduced to conjecture. If we ever happen to meet in person, I'll swap a story or two with you over a beer on the frustrations of dealing with those who insist on averaging correlations. I agree wholeheartedly with your comments on the matter. That said, it may make sense in some settings that I'm generally less familiar with. :)
That's a creepy website.
@Dave31415, But the dot product/ inner product can only be conducted in R^n space. So I will only be able to calculate 1xn or nx1 matrix. But how if I have 3x3 matrix??
Great answer. Thanks very much fo the ideas...
Thanks -- I will keep this question in mind. If you liked my comment otherwise, you could choose to upvote, of course :)
As a data-organization thing, it's a bit more common to create tables with features as columns, and observations as rows.  Not that this is at all necessary, but knowing that this is the default organization may help you make sense of other data discussions going forward.
@DilipSarwate Hm, I don't know. It is easier to read the way you proposed. I changed it.
@Avraham: Note they're *paired* observations: you'd want the [Wilcoxon signed-rank test](https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test) to test the null hypothesis that the mean difference within *each pair* is zero. And the OP might want to consider relying on the central limit theorem to justify the $t$-test approximately - it's quite a large sample size.
Several interpretations of the objective are possible. For instance, do you want to maximize the *expected* number of correct answers? Or perhaps you want to maximize the chance that every answer is correct?  Or the chance that the number of correct answers equals or exceeds a minimum threshold? Also, what role does "voting" have in this question? And what does it mean for a vote to be "correct"? What is the distinction between a "vote" for and an "answer" to a question? It is hard to see how your question can reliably or objectively be answered until you clarify these issues.
Please start accepting answers to your Questions. If an Answer solves the problem of your Question, click the big tick next to the Answer, near the number of votes for the Answer.
+1. Some fine points: (1) $S^2$ has the (noncentral) chi-square distribution, not $S$. (2) You seem implicitly to assume the two conditional distributions $m|l_i$ are uniform over the sphere(s). Because this does not mean uniform "on all the angles," I just want to make sure of your assumptions. (3) Why do you invoke N-P in a setting that is Bayesian? After all, you have well-determined priors on $m|l$ and $l=(l_0,l_1)$. So why doesn't the posterior distribution of $l|x$ fully answer your question?
Hey @deps_stats. It's actually already there. So you can close this one if you want.
thank you, that paper title sounds pretty spot on.  nice reading material for tomorrow :)
Is `pctLS` a continuous variable? Some descriptive stats by outcome groups might be helpful.
Can you tell us the (reciprocal) condition number of your matrix? In R, you can do rcond(A).
It might be related to my answer at http://stats.stackexchange.com/a/17148, but I am at a loss to come up with a quantitative rule of thumb.
I noticed that my answer actually may be wrong since I originally thought it was pois distribution for some reason and changed it but didn't change fact that maximum likelihood estimator for exp is $\\frac{1}{\\bar{x}}$ so instead look at http://www.stats.ox.ac.uk/~dlunn/b8_02/b8pdf_8.pdf (first example for help) Sorry about that
thanks for sharing `elasticnet`; however, I do not know what to make of the output from the above `R` script. Can you please clarify? Thanks in advance!
Sorry I did not state my problem clearly. In radiotherapy, I want to build the association between Vx and acute/late toxicity, where Vx represents a series of dosage variables: Vx, indicating the volume of the organ receiving more than x dosages. For acute toxicity as outcome, I could estimate the best Vx (e.g. x=20), using maximum log-likelihood. I can also estimate the confidence interval of x, using likelihood ratio test, this gives me CI=15~35. Similarly, for late toxicity outcome on the same patient cohort, I ended up with best x=75 and CI=50-80.
`Variation`, unlike `variance`, is not the name of some specific quantity (however, `Coefficient of variation` is). It is a generic term, like `variability`. It is just `amount of variability` which can be measured by various quantities (most popular of them being `variance`).
I went through them all (quickly) and found these may be worth looking at: http://stats.stackexchange.com/questions/28688, http://stats.stackexchange.com/questions/51046, http://stats.stackexchange.com/questions/57549, http://stats.stackexchange.com/questions/17673, http://stats.stackexchange.com/questions/41194, http://stats.stackexchange.com/questions/28688, and http://stats.stackexchange.com/questions/32600 (the dup). Plenty of others are relevant but need some imagination to apply; for instance, many of the questions on logistic regression contain useful information.
If your model is ${\\rm logit}\\left( P(Y=1|X,u) \\right) = B_0 + B_1 X + u$ then what you've written is the inverse logit of the right hand side and so is the correct expression for $P(Y=1|X,u)$, a.k.a. $E(Y|X,u)$. If you have empirical estimates of $B_0, B_1, u$ then plugging those in there gives a natural estimate of this expectation. To calculate $E(Y|X)$, as indicated by the title (but not the body of the question), you'd need to integrate out the random effect, i.e. calculate $E_u(E(Y|X,u))$.
Do you know how to generate prediction interval for AR(1) model? is AR(1) enough to generalize your model?
I especially like the point about re-analyzing old data two ways - very good advice.
I don't think this question needs to be migrated to SO necessarily, b/c the OP wants to know if his code matches the how the test is supposed to work. Ie, the OP is asking about the *ideas* primarily. It's a subtle distinction, & admittedly, someone will have to know both R & the Pettitt test to answer, but I think this is a stats question, not a coding question.
1) Yes but we can do Bayesian inference about $L_1\\beta$, not necessarily a hypothesis test. You do not answer my question: would it be analogue ? For instance with a noninformative prior, if we would look whether the credibility interval contains $0$, would it be similar to the frequentist hypothesis test ?. 2) Slide 5 precisely says that $L_{1\\mid 2}$ is defined with the help of $V$.
gung made an excellent point about the distribution of your data.  You'll need to make a determination of the best way to compare annual results, then you can plug that value into a chart like this.  Average is probably a really poor way to do it (but it was quick and easy for the example chart).
what does 'cov' stand for there? What do you mean by 'verify' there?
What makes you think "*the test he used is called p-test*" - there's no indication in the text near the phrase "*all the differences between*" to suggest what test was used; only a p-value is quoted. When you say 'fscore' what are you referring to?
I'd have been surprised if this weren't usually the case.
Looks interesting, but perhaps you could give a proper reference for when the link breaks, & (even better) a very brief summary.
This is a common procedure and is known as a 'hurdle model' , or a zero-inflated model (see Zuur et al. 2009 Mixed Effects Models and their Extensions for an easy discussion with ecological examples in R).
I see nothing surprising in the fact that the orthogonal vectors $A=(0,1)$ and $B=(1,0)$ both have positive scalar product with $C=(1,1)$. In layman terms, I see nothing strange in the fact that both meeting a homicide maniac and cancer are heavily correlated with death at early age but hardly with each other. Voting to close.
I've tried to incorporate the data in the question, but you might want to tidy it up a bit.
Thanks for including the data, it makes it much easier to respond in specific terms.
@Alexx Hardt yes
Fair enough, but for me, $f(x)=x \\oplus r$ is an "interesting" case (see Example 2 in question). Then, for any $y \\in R$, we have $\\Pr(f(x_i)=y|x_k)=\\frac{1}{|R|} \\ne 0$. So, I think it's rather hard (or even impossible) to find even trivial examples with $\\Pr(f(x_i)=0|x_k)=1$.
Starting with the idea of linear boundary (which by definition is the zero set of a linear form given by the weights vector $w$, $x\\to w\\cdot x$), any *monotonic injective* function $f:\\mathbb{R}\\to\\mathbb{R}$ will preserve the classification in the sense there exists a number $\\lambda_f$ such that $f(w\\cdot x)\\le\\lambda_f$ determines a half-space with the same boundary and $f(w\\cdot x)\\gt\\lambda_f$ is its complement. (Some non-injective functions can work, too.) OTOH, for any classifier with a boundary that is not a union of hyperplanes no combination of any $w$ with *any* $f$ will work.
Have you tried using the law of total variance?
Thanks for your pointers~! This definitely extends my understanding of this problem.
This question appears to be off-topic because it is about asking for code.
See Bailey, R.W. (1994). Polar generation of random variates with the t-distribution. Mathematics of Computation, 62(206), 779-781.
All of these can be done in R, however, there is not just one package that covers everything and it certainly won't be a click and win situation. It will be less automated.
Randomizing visitors (i.e. 50:50 chance of control or experimental treatment) is in general a [good design](http://en.wikipedia.org/wiki/Optimal_design), assuming your experimental treatment doesn't do anything terrible to visitors. Also, 1000-200,000 is a big range; is there any reason to think that visitors on quiet/busy days would (on average) would be affected differently by control/experimental treatment?
Apologies for the delay, I hadn't noticed your answer. I'll need to read up about the Tobit model.
@melon. Your question is quite vague. What do you mean by parametric and non parametric? (i know of a few different ways in which these terms are used).
Perhaps [this earlier question](http://stats.stackexchange.com/questions/78287/what-are-disadvantages-of-state-space-models-and-kalman-filter-for-time-series-m) is one half of the question presented here.
+1, this is really excellent. (I've noticed you have a habit of that.)
@Glen_b "30" was not even a coincidence - I just used it to provide a numerical example. I have no objection to the issue, I don't like "rules of thumb" (especially when they are dubious). I have made some changes in my answer. Thanks for the input.
Why do you assume the "error" is in `R`? Maybe it's the textbook that's "wrong"?
Thank that's a great reply.
This book looks very approachable. If I buy this and read it all, how much of frequentist statistics will it cover?
... simple correlations of these count variables with the numeric Q, with any fudge time factor you want. Then pick the highest correlation and this will be your highest correlating value of category A. This indeed will be cross-correlations.
Ah, I'm starting to understand the proper way to fit this now. Using the relative errors that are scaled by the counts for the minimization makes much more sense. Previously, I was just passing raw counts rather than variance at each point in as a weight, so I assume this is where my problem arose. Thanks for this explanation, I think I understand it much better now.
+1 Love the history, thanks!
Very nice non-technical answer, and an approach I wouldn't have thought of myself. I'd only add that it's possible to add any number of fake 'observations' to each category instead of 1, including non-integer numbers. This gives you flexibility to decide how much you want to 'shrink' towards zero the scores of items with few votes. And if you happen to want an technical-sounding description of this method, you could say you're performing a Bayesian analysis of data from a multinomial distribution using a symmetric Dirichlet prior.
Can you provide an example? It's very hard for me to understand what you are describing. It is possible that some software may not normalize a reference alternative to zero, though this would be very odd.
Without a sufficiently precise definition of 'pin down', it's not really possible to answer this question even for a univariate Gaussian.
Should have added: check the size values of your x's. Say x1 is measured in inches, but x2 is measured in yards. That will lead to a huge difference in your a's. In this case it would be better to make x1 in yards or x2 in inches. That way your a1 and a2 are closer in magnitude.
Not quite, $w^T w$ is the Euclidean inner product in feature space.  The kernel is a function which takes two vectors in input space and computes the Euclidean inner product in feature space as a function of the input space variables $x_i$.
+1 I was using similar pictures to solve this but in mine, $u_1$ is the *horizontal* axis, not the vertical :-).  It helps to standardize the expression for the domain of integration; for positive $A$ and $B$, it's $(x-1/(AB))(y+1/B) \\lt -1/(A^2 B)$, immediately exhibiting the center at $(1/(AB), -1/B)$ and showing the scaling with $A$ and $B$.
I agree that these questions should be asked at the study design stage. Unfortunately, as usual, the statistician was only consulted after the data was collected. Thank you for the references. Do you have any references for the GEE results?
If I have understood correctly (I may not), doing things at an aggregate level may run the risk of invoking the ecological fallacy, I think.
Related: http://en.wikipedia.org/wiki/No_free_lunch_theorem and http://stats.stackexchange.com/questions/17066/what-is-a-good-resource-that-includes-a-comparison-of-the-pros-and-cons-of-diffe
Thank you very much, EpiGrad! 'Core products' - makes sense, though sounds a bit technical to me. I'll wait for other people answers, and if nothing better comes up, I'll accept yours.
And yes, this model has many flaws in it (perhaps the doctor doesn't have to share computers and therefore manages to search pubmed instead of wasting time moving around books at the library) but I think it looks useful. Any thoughts on improving it would be most welcome, like other ways to calculate the aptitude.
Suppose that $Y$ comes from a zero-truncated Poisson distribution. Would the answer then be:
Okay, I see your point that associational is the default and that causal models are "nested" in the sense that they are more powerful.  The question is what is the difference between a causal model and regression or classification (an associational model).  And the main difference is that: While you can do regression from causes to their effect, or from effects to some hypothetical cause; in a causal model, the relationships are directed (causes to effects). These directions are required to support interventional reasoning, which associational models cannot support.
Thanks for the extended answer. As you say, leaving the data untransformed seems reasonable given the properties of the estimator. But I'll gladly change my opinion if I every come across data saying otherwise.
@Glen_b - I think this is a good revision.  Cheers ~
What about the control group? A paired t-test on all the data sounds like a bad idea and certainly does not address the main question (is the treatment effective?). A paired t-test restricted to the treatment group is a plausible strategy but ignoring the control group throws away a lot of data and makes for much weaker evidence that the intervention is in fact the active ingredient. ANOVA is in fact a common - if often criticized - way to analyze this design.
You may be interested in this question: [locating-freely-available-data-samples](http://stats.stackexchange.com/questions/7/), or this: [tiny-real-datasets-for-giving-examples-in-class](http://stats.stackexchange.com/questions/5937/), or just search under the [tag:dataset] tag.
You should look at the literature on *multiarmed bandit* problems, to which yours is closely related.
I don't think there's ever a "should", as by definition Likert scales are ordinal and it's never mandatory to ignore that aspect. "could" includes when there is a technique for nominal but not one for ordinal. For example, there are versions of correspondence analysis for ordinal data, but it may be easier to use common or garden versions that assume no more than nominal scale measurement. More generally, using nominal scale and checking whether the ordering is shown in results is sometimes interesting or useful.