tttjjj/elt_bench huggingface dataset:
@DJ_ Regarding your last edit, you might find http://stats.stackexchange.com/questions/555/why-is-anova-taught-used-as-if-it-is-a-different-research-methodology-compared interesting
Can't thank you enough! One of my sub-group sample size is 15 and r is 0.9208. Using the formula in the PDF, I have calculated SE to be 0.1081. Does this mean Margin of Error of 11%?
By "fit your first equation" I just meant that if you have values of y, x1 and x2 why do you need to muck around with reciprocals of anything, why not just use OLS to fit whichever of the two equations you want estimates for.  But the exercise looks like one to see the relationship between different estimators, not just a practical one of trying to fit a model.
A dummy that has a unique value for a *single* case (as seems to be the situation here) is tantamount to deleting the case from the regression and only creates useless additional output.  In a more general setting, where multiple cases might correspond to the dummy indicator, this recommendation would make more sense.  I still wonder, though, whether the dummy ought not to indicate *all* dates after the event. How can an event affect only the returns for that date without (perhaps in some complex way) also affecting subsequent returns?
@whuber can u be available for chat. You have helped me a lot before as well. So I was wondering if we could chat so that I can be clear about few more things. I would really appreciate it
So you are developing a custom algorithm and you want some general software to test its performance? If your algorithm belongs to a certain class of algorithms then it is possible that such software exists. Anyways you must supply more details if you want to get answers, as it currently stands the question is both too broad and too localized, i.e. unanswerable.
Small and medium sized enterprises
Please note that the units are not centered at all: they are only scaled by $\\sigma$. Your phrase "...away from..." describes the centering by the mean. Thus, in brief, a "standard deviate" is a way of describing the data using a different system of units, much like the relationship between degrees Fahrenheit and degrees Centigrade. Second, the referenced paper is sloppy in its terminology, causing your answer not to reflect what it is actually doing. It is re-expressing the data in terms of (unbiased) *estimates* of the parameters (based on corrected sample moments).
Of course publication of that article might have made the results no longer true...sort of like if an article on the best passwords lists an example, that password automatically becomes terrible...
Thanks a lot.  I am trying to do structural break test such as Chow test on an ARMA model.  In using strucchange package, it looks like they do not accept arima() objects in functions such as sctest().  Since they accept lm() objects, I thought I might try modelling ARMA using lm().
I see my typo, thanks for pointing it out. If $R$ is p x p, than the first p rows of $R$ is all of $R$, right? But it looks like what you are saying is that the $Q$ above, is an augmented matrix of $Q_f$ joined with $Q_f$ and $R$ essentially has just has more zeros beneath the upper triangular $R$ normally found in the QR decomposition. Am I understanding you?
So I do an omnibus test to test that "like, ind', non-liker" are independent with gender. If they are not independent with gender, I do the Post Hoc Follow-up Tests for each row, and use Bonferroni correction. This will tell me what exactly? I didn't quite understand. But this is very helpful.
You're welcome, @zamazalotta. If you want to code it in R, the basic procedure would be `plot(density(x))` (althought that wouldn't give you the customized version). If you need help w/ the specialized programming, it's probably best to ask on [Stack Overflow](http://stackoverflow.com/).
Autoencoders - a type of neural network - are a form of machine learning, and this question is about the math behind auto-encoders.
The answer to the initial question is about $36.8\\%$. I didn't calculate this analytically but by simulation in R. `N=1000000; (N-length(unique(sample(c(1:N),N,replace=TRUE))))/N`.
Just a side note: there are also appropriate multivariate extensions whereby the derivatives are replaced by gradients, and the multiplications have to be matrix multiplications, so there's a bit more headache in figuring out where the transpose goes.
Any way to make bp a random effect?
I am looking at LeNet-5 in particular, and using this http://deeplearning.net/tutorial/lenet.html as my reference. It seems from that page, that the kernels are 3-dimensional, but it is not 100% clear to me.
$\\mu$ is by convention used to denote the population mean ($\\mu=E(X)$) while $\\bar X$ would denote the random variable respresenting a sample mean (and $\\bar x$ would denote an observed sample mean). Indeed, by convention, Greek letters are mostly kept for population parameters and Roman letters are mostly kept for sample quantities (with a few somewhat common exceptions). It's likely that people will find your unconventional notation confusing in this particular case, so you should consider whether it might be better to come closer to the usual convention.
Yes, mixture models are exactly what I need.
Are you saying you don't know what a p-value is? If so, you probably need to work w/ a statistical consultant, these methods will be too advanced to learn in a short Q&A or blog post, etc, format.
Excellent! Thank you. This serves as another good example of what could be happening in my data. Seems like I can only accept one answer, though...
Are you saying you fit a linear regression model to your data? When you say "How can I measure the error of my system such that it falls between 0 and 1?" are you just asking for a measure of how well your model describes the data? Have you heard of $R^2$?
Thank you Jacob for your excellent answer.
I agree but I find it even more annoying when a reviewer tells you to omit (or move to suppl. materials) what are, realistically, very crucial details about the analysis. This problem makes it so that lots of science/social science papers that do even the most slightly complicated analysis are pretty cryptic in that regard.
ggplot2 is just a graphic package? What about it makes you recommend it for multi-dimensional data? Facetting?
Thank you Randel, also for your reply! I was thinking more eigenvalues… good to know!
Thanks Sarunas!  I've updated my question to reflect that I'm working with probabilities.  What do you think now?
Could you please be more specific as to your goal? The way the question is posed now makes it hard to say what you want (e.g., what are the unknowns)
very good and extremely helpful example (you got me interested in R).
So, of course the right thing to do is get more data points.
@Curious Hope it is clearer now?
@bayerj You're correct. Do my edits make sense?
By "January 1st", I presume you mean the cut-off is an entire year and not 6 months period that applies to your children.
Are you treating your categorical variables as categorical, or are you regressing them as if their numeric encodings were mere numbers?  That would fully explain the strange behavior you describe.
I certainly appreciate the responses - thanks very much. As one follow-up, how this question came about was the accuracy of the random table above is 0.41 (82/200). But, if we wanted to compare our model against random guess, using only the priors, we'd have an accuracy of 0.38 ((100/200)^2+(60/200)^2+(40/200)^2. Doesn't it seem that this should match the random matrix?
In addition to PCA-type reductions, you can also try plotting against only a few of the values at a time to see if there are any visible patterns there, and you can also try nonlinear dimensionality reduction techniques (LLE, tSNE, ...) which can sometimes give more informative projections.
The end of your last sentence doesn't match the first half of your second last sentence. Please fix.
How would you define "the main effect of B", knowing that there's an interaction between A & B?
In effect it means estimated using a method with lowest possible asymptotic variance, which means that here you'd likely need to use MLE or minimum chi-square or something equally efficient. Your description of how the estimation is done doesn't make it clear whether you're doing something efficient or not. If it's not efficient you won't have a chi=square distribution; you might need to simulate to get the distribution of the test statistic.
RFs are bag of decision trees. Whether the response variable is binary when using RFs is immaterial here, IMO.
+1 in any case, but there are a few things I don't follow about your answer:  1.  I am specifically considering *balanced* Latin squares, whereas you say "classical Latin Square".  To be clear, what I am talking about, for example, include {ABDC,BCAD,CDAB,DACB} but not {ABCD,DABC,CDAB,BCDA}.  Which are you discussing?
Dear Gaetan, I am delighted by your response - thank you for the kind words.  Best,  Tal.
Hello Peter. Why do you say that the standardized difference $\\theta$ is a *good* choice for the $t$-test ? Is it because of the property I pointed out: the power depends on the parameters $\\mu_1$, $\\mu_2$, $\\sigma$ only through $\\theta$ and is an increasing function of $|\\theta|$.
If you are using PCA, try `propack.svd()` from the `svd` package.
@RobertSmith: According to your quote, they are split the same way that numeric attributes are split. That is, they're converted into decimal: `11, 01, 00` are treated the same way you'd treat `3, 1, 0`. 10 variables would become `511, 255, 127, 63, 31, 15, 7, 3, 1, 0`.
Ah, another great perspective. I hadn't thought about this from the entropic point-of-view!
@Mathew your answer is very good. I have gone through your bio-data. Would you like to read and answer some of my questions.
I edited my answer above. Hope it now makes some sense.
No, no classes need be formed. It is a fitting of a least squares line.
I'm not sure what to make of such a statement, because it sounds tautological: "we found that two SDs works, so...two SDs is applicable."  I guess it all comes down to what you mean by "applicable."  The authors are probably hoping that this is applicable to recognizing text within many images.  I doubt they hope it is applicable to any other problem.
For something which looks like homework, you might get a better response if you show us what you have tried.
Thanks very much for that - embarrassingly, I never thought of transforming the data first!
(1) Note that the "Stand.deviation" formula actually is for an *estimator* of a *variance* and that both formulas incorrectly refer to 15 values, whereas 100 values are in this dataset.  (The use of "exp.value" in place of "mean" suggests these data should be treated as a *population,* whence the `n-1` probably should be replaced by `n`.) (2) You ask about the difference between treating these data as 100 values and 15 values.  Why don't you just compute the formulas you think should apply and compare what you get?
Thank you Rob, very good and clear answer.
I'd like to think of myself as the kind of statistician who has a problem of originating with (1), goes on to play around with (2) to find a way to solve it and then uses (3) to show that the solution is valid. :)
It is important to discriminate between a scientific hypothesis and a statistical hypothesis. The null hypothesis tested by null hypothesis statistical tests is usually only e latter. Testing a well-designed statistical hypothesis may allow inference regarding the scientific hypothsis, but it is not always the case.
More or less, I'd not say "at the 99% interval". I'd just report the p-value. And personally I'd not even use the word significant. I'd just say they had an effect and report the test result. Then describe the data. The description should have a measure of variability and CI's would be best.
Right, I'm not talking of an estimator's bias. But I felt that "biased coin" was a sufficiently well understood idea that the polysemy wouldn't be a problem.
No, concordance is computed using the raw outcome variable $Y$ and the linear predictor $X\\beta$ or any of the predicted probabilities (as they are all monotonically related to each other, i.e., just shifted by in intercept before computing the expit).  Somers' $D_{xy}$ rank correlation coefficient uses this concordance measure.
Thank you so much Jake. cheers!
The sample autocorrelation for $k$ lags is $\\sum(Y_t-\\bar{Y})(Y_{t-k}-\\bar{Y})/\\sum(Y_t-\\bar{Y})^2$, so adding a constant $c$ cancels out: $(Y_t+c-\\sum(Y_t+c)/n)=Y_t+c-\\bar{Y}-c=Y_t-\\bar{Y}$.  In any case my point was just that if you *were* to choose that $c=0$ model your residuals would be equal to the raw response data - illustrating why you can use the Ljung-Box test on the time series itself.
That's right! I kept thinking "Metropolis" plot, as I knew there was some relationship to a city and the shape of the figure.
Both methods will give exactly the same answer, I'm afraid.
+1 for the excellent link
Is your goal prediction or description? If it's the former, then multicollinearity is not a problem as long as you don't overfit. Also note that with such a huge sample size anything will appear statistically significant even though there is no practical significance. If the goal is prediction, I would focus on out-of-sample ROC as opposed to in-sample goodness of fit, LR, and similar statistics.
Normally, geographic coordinates are given in latitude and longitude for which applying this Pythagorean formula is erroneous.  In many, if not most, applications the most natural distance for *geographic* coordinates is the geodetic distance on a spheroidal or ellipsoidal model of the earth.  Cartesian distances (based on the points as projected onto planar maps) are used primarily as convenient *proxies* for the geodetic distances.
(Warning: Start point not provided, choosing random start point. )
Well, I just now fitted a Poisson regression to three observations - both parameter estimates were within about 1.3 standard errors of the population parameters. (It even managed to fit 2, but of course that leaves no degrees of freedom for anything but the fit.)
As in less than a dozen for both procedure A and B. I think of them as proportions, but don't know if that is the right way to think about them.
> you could add a link to Efron's bootstrap paper**s**. It's as good a place as any to start.
Sorry, I meant 6. I've updated the question
I couldn't do so myself, but a book reviewer summarised the approach in a paragraph in Amer. Stat. in 1993: http://www.jstor.org/stable/2684984. 

I'm not sure it's really practical to use this approach with your students unless you adopt it for the entire course though.
@AlexanderGladysh Have you tried using SUMIFS to get the answer you required? What was the result? I had a go myself and it seemed to work fine. Try and get a table with fruit on the rows and spices on the columns and the total count in the cells. Once you've got that it should be easy!
It may be the easiest but I don't think characterizing it as the "best" is reasonable. At the onset it precludes seeing if any baseline*treatment interaction exists, and as John mentions it is problematic if the baseline has a causal effect on the post-treament (independent of treatment itself). Paul Allison [in this article](http://www.statisticalhorizons.com/wp-content/uploads/Allison.SM90.pdf) gives some examples where change scores are preferable.
@amoeba: There you are
ARIMA modelling is about modelling the correlation patterns observed in the data. That's why the ACF and the PACF play an important role in the analysis; they reveal the properties of the data (the autocorrelation structure). If the model that is proposed fits the data well, then we expect the residuals to be white noise.  For example, if the true model is an ARMA(2,1) and we fit an AR(2), then we will find that the residuals have a MA(1) structure and will realize that we should add this term to the initial model.
I know it's an ancient post, but I didn't understand the swindle they used. Could you elaborate?
I have made some edits to improve the clarity and formatting of your question. I couldn't work out what you were trying to say in the last parts of your second paragraph, so I couldn't really fix that. Could you please edit it to clarify?
I don't quite follow your situation. If you have percentages, why would you want to use Poisson? What does it mean to "transform [percentages] to proportions"? Why don't you *want* to use the binomial dist? Why would a 'next step' be necessary? If you have data in percentages, do you know what the denominators are? If so, you can recover the number of seeds that germinated & the total number of seeds. This means that you could treat your data as binomial, which, as far as I can tell from your description, is the appropriate way to model your data.
could you pls describe the design of your study in detail, for example: where the micro-organisms same across samples, are the samples representative of some treatment or just replicates, etc?
Hope this helps: \\n\\nhttp://stats.stackexchange.com/questions/74334/question-about-prediction-bands-for-non-linear-regression-computation?lq=1
:wayne I would never recommend assuming any particular class of an ARIMA model whose family includes all varieties of HW excepting the multiplicative variety but rather one should allow the data to speak to the model identification process.
great question. I have wondered as well about how to construct the calibration even if you do use 1 versus the rest sort of scheme. If you create k models using 1 versus the rest (and there are k classes) do you have to / should you normalize them somehow so that they sum to 1 (e.g. divide each calibrated probability by the sum of all k)?
According to http://www.imada.sdu.dk/~marco/DM825/Slides/dm825-lec7.pdf for example, it is a generic label for Linear DA + Quadratic DA. So, you may safely search/read (including this site) for just "discriminant analysis". It will correspond to GDA. GDA is DA assuming normally distributed population.
Hmm... you must have defined a period where 2 is at risk for an admission, i.e. a start and end date but without an event. Apart from that detail this seems like a case for mixed model cox regression (check out the R `coxme`-package).
I've come close using just a wide set of measures of activity land use (bars, gas stations, hospital, schools, etc.) plus the spatial terms. Here is the map of the predictions holding those other [covariates constant](https://dl.dropboxusercontent.com/s/kx6ozxnc095277c/SurfacePredictions%28Mean%29.png). There is still a tiny bit of residual auto-correlation though. I'm skeptical given the error how much dasymetric mapping of the population to small places will help, but I imagine I will undertake that analysis eventually.
Also, watch out: in older literature, the term covariate has a specific meaning still used in "analysis of covariance", while in more recent literature the term (increasingly) is more or less yet another synonym for predictor, explanatory, or independent variables.
For the sake of clarity, what you are saying is not that the clusters that come from different algorithms overlap (which is to be expected), but that you are using algorithms designed to produce overlapping clusters (eg clumping). Is that right?
Can you provide more information on the problem you are trying to consider? As stated, it lacks enough detail to answer adequately.
Thanks, for your detailed comment. At the interview I proposed to calculate MSE (Mean Squared Error) to estimate differences between human frequencies and machine frequencies. Machine with smaller MSE I suggested to be a more realistic coin flipper. The interviewer said that my idea is "at least some solution", yet not a good one, as I concluded from his words. Why Hellinger distance is better the MSE?
How does this relate to Fisher's exact test (in the tags)?? $\\quad\\quad$ What is the hypothesis being tested? What is the alternative? What are the assumptions?
You can easily use firstbayes to make comparison without having to write codes as in R. Get firstbayes. This might help because installing it might be tricky..http://www.youtube.com/watch?v=BcdigiWY054
(+1) I don't think this can be repeated often enough! See also [the same issue discussed here](http://stats.stackexchange.com/questions/11351/left-skewed-vs-symmetric-distribution-observed/11352#11352).
@Tim (1) A family is complete if the only unbiased estimator of $0$ is (almost surely) $0$; if $X \\sim F_\\theta$ where $\\{F_\\theta\\}$ is complete then the only way to estimate $0$ unbiasedly is with $g(X) = 0$ almost surely. (2) When $X = (X_1, \\ldots, X_n)$ has iid components, $X$ itself is not complete because $E[X_1 - X_2] = 0$. However, I might introduce a function $s(\\cdot)$ so that $s(X)$ is complete; if I *also* choose $s(\\cdot)$ so $s(X)$ is sufficient then lots of good things happen.
Note that in many fields, a sample of studies is going to present an extreme overestimate of effect size, especially a small sample of new studies for a new effect: http://www.nature.com/nrn/journal/v14/n5/abs/nrn3475.html
You may have better results if you state what you have done, and the problems that you are having - as opposed to simply posting your homework question (i.e., nobody wants to do your homework for you, but we are happy to help you learn how to work through the problem).
I am not sure what exactly your notation means, but the limit of the joint cumulative probability distribution function is not restricted to only two values $0$ and $1$ but should be a nondecreasing function of both variables like $$F(x,y) = \\Phi\\left(\\min(x, y)\\right),$$ being an increasing function of $x$ for $x < y$ that becomes the constant $\\Phi(y)$ in the region $x > y$, and similarly for fixed $x$ and varying $y$.  Here, $\\Phi(\\cdot)$ is the univariate standard normal CDF and I have assumed that $X$ and $Y$ are marginally standard normal for simplicity.
I'm not sure if it's by Bayes or if it's by the definition of conditional probability, but this is how I would do it.
I didn't say they weren't non linear, I was saying the only non-linear ones you have used are all tree related. If there is something about your problem that is inherently difficult for tree based or related algorithms, you won't know it until you "branch" out to something different.
What is your ultimate goal?  Why does roc even remotely relate to it?
Plain KNN is not very good for faces. Faces have been shown to lie on a ~25 dimensional nonlinear manifold of the pictures.
I did a bunch of editing; I hope it is now what you wanted to ask
$P(A|B) = P(A \\cap B)/P(B)$ Which would let you calculate everything off the joint distribution. Based on Henry's example that would lead to $Pr(X=1, Y=0|X=1) = Pr(X=1|(Y=0|X=1))Pr(Y=1|X=0)$ Ahhhh, okay I see now. I was getting caught up on the joint distribution of (X, Y|X) and not seeing how the conditional takes precedence in calculations. Thanks, my confusion has lifted.
You seem to be describing an "intervention analysis" or "interrupted time series" as discussed by [@mb3041023](http://stats.stackexchange.com/a/43457/12318).
Can you give a context to the quote? Without context it might mean any number of things.
@Jason Given the definition of the p-value, if the p-value is larger than the significance level, what would that imply? What would it imply if it was less than or equal to the significance level? Try the first few sentences of http://en.wikipedia.org/wiki/P-value for further explanation.
All the additions I can find are derived from partitioning the event "$z_i=z_j$" according to the possible common colors of the $z_i$ and $z_j$; therefore, those probabilities *add*.
sorry about that. The title is "A Taxonomy of Semi-Supervised Learning Algorithms" by O. Chapelle
Blouin & Riopelle (2005) called them narrow and broad inference confidence intervals but given that the general scientific populace outside stats has a hard enough time with regular ones...
Thank you Enrique, TraMineR seems very interesting I will try it on Monday!