tttjjj/elt_bench huggingface dataset:
Can you tell us more about your research design?  Is the following example more or less what you are doing?  You have two groups, say men and women.  You have two types of data for the men, say height and weight. You have the same two types of data for the women.  You want to know which are heavier, men or women. You also want to know which are taller, men or women.  Or is your design something different?
Hi @DezUdäzue, I think it would help a lot to know what exactly you plan to do with this statistic. Right now I'm still a little confused about what exactly you want and why. It seems like correlation might fit the description you're talking about, since it is invariant to multiplicative (and additive) changes, but I'm not sure. Also, note that standard deviation _is_ sensitive to multiplicative changes, by the way.
Have no idea why latex looks so bad. Can someone please help and fix this?
Please explain, then, why you refer to "machine learning techniques."  Is your question about how to perform a regression in which one of the independent variables is a factor with 8000 levels, there are no more than ten other independent variables involved, and you have a few million records, or is it more than that? Because there are differences between doing one regression or a large number of separate regressions, it would also help for you to describe your categorical variable and provide more information about the dependent variable.
seems to be close to info on "nonlinear kernel function" on this page? [support vector machine](http://en.wikipedia.org/wiki/Support_vector_machine#Nonlinear_classification)
Better in what way? See my post.
As the help explains, please don't crosspost. Choose one site that's best suited to your question and post it there. If you later change your mind you can flag it and ask for it to be moved.
P-values are not appropriate measures for control charts, because of the serial multiple comparisons involved and the possibility (likelihood, in this case) of serial correlations.  Control chart properties are better assessed in terms of expected lengths of out-of-control (OOC) runs and in-control runs under the null and alternative hypotheses, respectively.  On this basis, letting even one case trigger an OOC is almost never a good procedure.
_Thanks again @jbowman!_
Furthermore, note that based on box 2.2 on page 25, the book in question does not attempt to explain how/why MCMC works, but you need read other sources for that. If after consulting other sources you have some specific questions about this particular example, please edit the question to address those specific concerns. In my opinion, the question as it now stands is essentially 'how MCMC works' which is pretty broad and/or asked already.
+1 But it isn't necessary for "curves" to become "straight lines." As your image so capably conveys, the images of the curves can still be *curves*, but they may individually be limited to regions that are readily separated by hyperplanes.
@Henrik. Thanks for your comment. Hmmm...I do not experience jumping p values with my dataset. Actually I get almost the same p values than with `lm` function, although factors seem to be heteroskedastic to some degree.
(+1) excellent answer.  I like how it flattens out at step 198 to 199, then jumps up to 1 at 200 (which you can just pick out from your graph).
Thanks.  I was hoping to see a worked example, but the right key words to search on was helpful.  I think I've found a good description.  Now to apply it.
If the relationship with time is non-linear, you can include time as a dummy variable instead as I have mentioned. It would then estimate a different coefficient for each time point. I'm pretty sure they have a more fancy name for this; but I just can't recall it now.
Your outcome does not seem like a categorical one. It is a count.
What is the end goal here? i.e. How exactly do you plan to compare them?
I think I'm going to need to see that presentation in person to get the full impact.
Thanks Nick, I am reading.
It is difficult to imagine a useful concise answer to this. See http://stats.stackexchange.com/help/dont-ask including the advice "Your questions should be reasonably scoped. If you can imagine an entire book that answers your question, you’re asking too much."
This depends intimately on the actual question of interest. For a short discussion of these competing interpretations, you might also look at the duplicate question given in the comments. In this case, note that the OP has explicitly stated that initial strings are unique.
Rscript comes with R, and is released by R Core.  We can't quite reach there with 'r' but it works well on Linux and OS X (and is marginally 'lighter' and 'faster').
@EmmetB: Yes, you're right -- this should probably be  $\\sigma_{new}^2 = \\frac{n-1}{n} \\sigma_{old}^2 + \\frac{1}{n} (x_{n+1} - \\mu_{new})(x_{n+1} - \\mu_{old}).$ Unfortunately, this renders void my whole discussion from there, but I'm leaving it for historic purposes. Feel free to edit, though.
@Drew75 No, we are not using any indicator variables..all the variables used are derived from original variables.
Coherence is computed at each frequency-and hence is a vector. Hence, a sum of a weighted coherence would be a good measure. You would typically want to weight the coherences at frequencies that have a high energy in the power spectral density. That way, you would be measuring the similarities at the frequencies that dominate the time series instead of weighting the coherence with a large weight,  when the content of that frequency in the time series is negligible. I added this in the answer above, after reading your comment. This is a good question.
Please don't cross-post here and on the r-help lists ... and if you were going to post on a stackoverflow/stackexchange forum, I think SO would be better (this is a technical R question, not a stats question ...)
@kwak The data itself varies from time point to time point. The line of best fit is generally either stable or monotonically increasing.
@dfrankow here's a better explanation: http://www.cs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture7.pdf
Because if a sample, x, is drawn from g that is outside the support of h, then f(x)h(x) is zero, even if f(x) might not be zero. So that part of the support of f doesn't necessarily have to be in the support of g, hence the less stringent requirement.
No worries! If you don't have clear empirically-based expectations then yes, I think you're right to use Sidak/FDR. A good option I think is to give the original p-values, and then indicate (e.g. bold font or asterisks) whether the comparison was significant after the correction. That way the reader can decide for themselves how much they trust the result.
@jbowman I'm not sure if I understand what you mean precisely, could you elaborate? Do you mean, in this example, dropping $x_1$ for both $sample_1$ and $sample_2$, or only for $sample_1$? In the former case, I have many samples and for almost all $i$ at least one $p_i^{x_i}$ is 0, so then I don't have many left over. If it's the latter case I don't see how that would be done, but I'll know where to look.
For future reference: It is preferable to flag your post at the original site and ask for it to be *migrated* rather than reposting.
I think you mean "there is a smaller chance of obtaining an observation *outside* that interval". Unfortunately, a Confidence Interval may not mean what it appears to mean, due to technical, statistical issues, but in general the narrower the interval (at a given confidence level) the less uncertainty there is about the results. There are many threads on this site discussing what a Confidence Interval means (as opposed to, say, a Credible Interval). We're not even getting into Predictive Intervals...
@gung Thanks for the compliment. `test="LRT"` and `test="Chisq"` are synonymous (it says it on the page you linked).
@user3813193, I don't think that you can make a plot of a logistic regression that is independent of the other predictors.
One correct answer that seems to have gone unmentioned in the replies so far is indeed "something else."  See, for instance, chapter 2 of [The Bootstrap Small Sample Properties](http://www.stat.washington.edu/fritz/Reports/bootstrap-report.pdf) (F. W. Scholz 2007).
Can you have just 1 failure overall?  If so, then it is akin to relating death to age or relating failure of an object to one or more covariates, including age.  The "textbook" method for this is to use a technique known as the Cox proportional hazards model.
I interpret this to mean that a, b, c and d change over time - they ar enot intrinsic things for the patient (eg they are not like height, or ethnicity).  One of your questions to address is whether it is a change in these variables that is linked to heart attacks etc, or the base level.
@Irish I don't think @Michael was suggesting that the analysis be limited to graphing.  It would be foolhardy at best to trust the output of "good/superior analytics" without ever looking at the data.  Given a choice between some statistical output and a good graph of a dataset (but not both), I will take the graph every time.
"*Some people assume normallity simply because it simplifies their calculations*"-- in a study where age is presumably a predictor (IV), *what calculations could it simplify*?
Those two answers are the same up to floating point roundoff error.  You should be comparing the coefficients *relative to their standard errors.*
@ Tomas, thank your so much. Yes, you are right! WinBUGS provides the prediction of h[N], including stats and probability density. I got it now. Best Regards,
If you only care about total error then you don't need linear regression. However, if (and only if) the error changes over time I could imagine it would make a lot of sense to include that aspect in your analysis. For example, sensor error could increase over time because the accuracy of the sensors deteriorates over time or because measurements at $t_2$ depend on measurements at $t_1$ (and error accumulates). Whether such a scenario makes sense or not is of course up to you to decide.
There will be 8 discs representing different plant sizes; 1.0metres dia., 1.5, 2.0, 3.0, 5.0, 7.5, 10.0 and 15.0m. and yes I would need to specify what proportion of the region would be allocated to each species.
@Luca Please register your account -- this way you won't lose your account again. You can do this here: http://stats.stackexchange.com/users/login
I suspect the question was downvoted because, although written very clearly, it is a straightforward example of an absolutely classic situation that would feature in most introductory stats texts, but you hadn't indicated what you had tried before asking the question.  People may have suspected it was untagged 'homework' or self-learning.  I presume my answer got a downvote because it is too brief - doesn't explain why my answer is appropriate or how it works(and I agree, I just don't have time to expand on it properly just now, and the expansion can be found in many texts anyway).
Are the "unique people identifiers" correct in both datasets (i.e. is person101 in dataset A the same individual as person101 in dataset B)?  If so, you could simply find the proportion of fields that are consistent between the two datasets.  That would be a simple way to measure the similarity between the two datasets in a quantitative way, and it is easy to interpret.  You could even make the statement that X% of the entries in dataset A have different values in dataset B.  You could then decide how to handle the differences.
I so feel that unless the training and test samples both exceed 10,000 observations, it is often misleading to report accuracy on a holdout sample as opposed to doing intensive rigorous resampling.
I extended the question a little bit, thanks. My initial attempt to plot the decision boundaries would have been to solve the g1(x) = g2(x) (and g2(x) = g3(x) etc.) analytically, and then rearrange the equation to calculate x2 from x1, which I could plot then. But I think I made several mistakes when I tried to simplify g1(x), g2(x) and g3(x). I'd would be great if someone could help me with this. However, if there is away to get around that by plotting the decision boundaries directly from g(x), I would also be happy with that!
If I use Metropolis Algorithm, I don't need an normalized posterior. Thank you very much for your help!
I noticed your request for migrating to CS.SE. Please, confirm whether you mean http://cs.stackexchange.com or http://scicomp.stackexchange.com. This question is largely on-topic on this site, though. Maybe you can update it to bump it up and attract more attention, or offer a bounty.
"analysis of covariance" is not a test.
@DmitryLaptev any chance you could explain a little more the difference between the first and second? It would seem that the second does not calculate a similarity score between i and other products whose dummy are not equal. Is this the case?
+1 useful, thank you
why ask the question here?  All are community-wiki, why not just fix the canonical answer?
I think these are both very nice books that would make excellent reading for the OP.  But neither is a concise introductory text on statistics that the OP is requesting.
How exactly did they use McNemar's test?
How is functionally independence between random variables defined?
@DBR, well, if your variables are normally distributed, by additivity of normal distributions, $X-Y$ is normal with mean $60.9-63.7$ and standard deviation $\\sqrt{ 2.9^{2} + 2.7^{2} }$ so you can just use the same trick I specified in my answer except using the normal instead of uniform and the mean/sd I specified here.
I edited the answer to include the derivation justifying these calculations, but I also posted a second answer containing code that is much more concise.
Please supply missing details.  "DGP" = discrete Gaussian process?  Differentiable Gaussian process?  Data generating process?  Something else?  And what does "$\\tilde Z$" mean?  What is $b$--the ordinarily least squares estimator of $\\beta$?  And what is the connection between $X$ (a random matrix, evidently) and $x$ (a determined, non-random matrix)?
Thanks for the clarification. In the previous version I missed the step where you redraw the multiply-imputed data for every replication. You will probably need large m to incorporate adequate between-imputation variability. 

If I understand correctly you will have the following procedure: 1) generate one imputed data set, 2) take a bootstrap sample from that 3) estimate beta 4) repeat 1-3 1000 times, 5) summarize the distribution of 1000 beta's. 

I think something like this might work, but you really ought to do some simulation to confirm it.
@user1274037 I added the mean no-dependency case to my answer.
See here http://stats.stackexchange.com/questions/1278/is-calculating-incidence-rate-difference-ratio-appropriate-for-single-case-exper
You can use the `ber()` function in the R package [ber](http://cran.r-project.org/web/packages/ber/index.html) or the `combat()` function in the Bioconductor package [sva](http://www.bioconductor.org/packages/release/bioc/html/sva.html).
Gavin, thanks a ton for your help! This is the first time that I've come across the "as.formula()" function - quite useful!
RANSAC is very different - in bagging, all samples are taken into account in some way. RANSAC is used in cases where up to 50% of the data should be completely discard.
@CarlWitthoft Yes, it does "reliably" crash on some datasets and not on others, but so far it only crashes on quite large ones (~15000 rows with ~35 variables) so making a reproducible example is hard.
@Scortchi: yes i do
(ctd) ... "*It seems like the GoF function is overreacting,*" --- a p-value is not a measure of how well something fits. I didn't ask you what you *calculated* that you treated as a loss function (I already told you NOT to use that! I don't think I was unclear) --- what's *the loss you actually experience from different kinds/sizes of error*? If you figure that out, it will improve your chances of specifying a useful way of measuring fit.
This depends on what kind of approximation you want for the posterior.  If you want the marginal distribution of $\\lambda$ then yes you need to integrate out the other parameters.
The implementation at:
http://code.google.com/p/randomforest-matlab/
is really a cool and fast one with a C implementation and MATLAB simple interface. 

Did you try also, the MATLAB's own implementation:

ens = TreeBagger(ntrees,X,Y)
[labels,score] = predict(ens,X)
I am not sure I agree about not being able to spot fabrications: I have spotted such before.
Dear MånsT, take a look at these papers: http://www.mdpi.org/entropy/papers/e1040099.pdf http://w.ime.usp.br/~jstern/miscellanea/citacoes/swtest1.pdf
http://www.ma.utexas.edu/users/mks/statmistakes/errortypes.html
@Benjamin while I'd also advocate reading about it (it is important!), I don't think it would be at all appropriate here.  FDR is controlling the expected **proportion** of false positives from a number of selections.  Since all Panos' tests are trying to inform him on the same global hypothesis, it would make little sense to control the expected proportion of false test - especially when you have correlation.  He isn't choosing between multiple different questions, but trying many ways of answering the same question.
is it ? then we are all intelligent persons here:)
It's worth bearing in mind that the probability of this happening / the p-value, is the probability of getting these results when drawing a sample from the reference population. The manufacturer (hopefully) normed the measure on a national representative sample, but your class is almost certainly not representative in this sense.
Income typically has right skewed distribution with few extreme high salaries. So median may be better in characterising the district as it is not sensitive to the extremes. However, I remember reading that inequalities were predicting crime so incorporating the extremes in the model may be valuable.
Thanks for this! I'm reading your code and it's helping me figure out the R syntax a lot.  One thing that's giving me trouble is the line `freq.resp <- raw.resp/apply(raw.resp, 1, sum, na.rm=T)`.  The error is that "dim(X) must have positive length" while dim(raw.resp) is NULL.  Could it be that since my data doesn't have all options with positive frequencies, my tables aren't all the same length?  How can I fill in the zeroes in my `table` invocation?
So what are your thoughts on the problem? Exhibit some work and we will be happy to provide the hints for you.
@PatrickCaldon Yes, both of you are right. By storing a term frequency vector for the whole vocabulary, and when a new document comes, we can easily update the idf vector. By the way, I think it's not necessary to re-train the model with the new idf vector. am I right ?
(+1) Good edit.  @Chris: you might be interested in the notes to a short set of PowerPoint slides [here](http://www.quantdec.com/Articles/openspace/Rank2.htm): this is a presentation on the subject I gave to non-technical people.  I mention it because it has some illustrations and guidance for how to "create a universal metric."
Ah, I was a bit confused about if `lognormal_like` worked on vectors, but I think I understand what you did. `potential` seems to be what I need, but I'm still not completely clear on how it works.
EViews is a good option for a beginner, but R is a much better long term investment. If you don't have access to it through your Uni, just use R.
... and not just rounding of values during pasting.  Saving the excel sheet as a csv in Excel saves the visible number of decimal points (the approach I used to get R to produce 'Stata results').
Can you say more about your data & your model? You state that it was "produced by a logistic regression", but your code reads `lm.fit`, implying a linear model was used.
fwiw, having encountered some of these situations, I think reinventing the wheel will probably be necessary in order to fit a solution to the idiosyncracies of your situation.  But based on your posts here that I've noticed over the past 2 years I am confident that your solution will fit better than any that we "onlookers" might devise for you.
@curious PostGIS, Python, and R solutions are referenced at http://gis.stackexchange.com/q/1200/664 .  (Ignore references to "convex hull" algorithms and focus on "concave hull" and "alpha shapes".)
In physics a center of mass differs from what you describe: it is a "balancing point" of an object. As a little contemplation of mobiles, cantilevered architecture, and other highly asymmetric but stable objects will show, much more than half the mass can lie on one side of the center of mass ("barycenter" or "centroid"). The physical analog here is excellent, because probability density and mass density are mathematically identical.
Rob, thank you very much. I hoped you should react to this question and I am glad you answered it. In fact I still do not understand if it's a common practice to get $\\alpha$ and $\\hat{y}_1$ simultaneously as two arguments of a minimized function (I mean squared errors). If one estimates only $\\alpha$ then its optimal value is inside (0;2) interval for standard initial $\\hat{y}_1$ (obtained for example, as intercept of linear regression)
+1 Good points.  Welcome to the site!
Wilcoxon implies pairs of observations are independent, which is probably not so with your time series data. So the answer is no.
@Macro : yes fixef, coef, etc. works but summary(g) doesn't give me the p values only T values. I would like to have the wald Z and the corresponding p values if possible?
Great thank you, this is very helpful!
I think you are doing just fine. Better focusing on a smart descriptive presentation than on randomly picked statistical test. Please note that a hypothesis test is not 'descriptive'. Its about unknown parameters.
@Xi'an: The R help says, "the operator = is only allowed at the top level (e.g., in the complete expression typed at the command prompt) or as one of the subexpressions in a braced list of expressions." I can think of situations other than mentioned in your blog, though so far only C-ish things like: `a <- 1:5 ; b <- 4 ; a[b = 3] ; b ; a[b <- 3] ; b` where the subscripting gives the same results, but `b` is different (first time not changed, second time changed). The `=` was never meant to be assignment and should have never been changed to try to make R more palatable. Don't do it.
Define fast. :-)
You are empirically using the proof of the robustness of the t-test ([see](http://stats.stackexchange.com/q/44262/10525)). A good estimator of $\\sigma$ typically requires a moderate or large sample size. This in turn implies that the distribution of the T statistic is a Student-$t$ with large degrees of freedom, which is approximately normal.
Giovanna, it is really _easy_ to find the density function of the minimum in this case, and then calculate the expected value of the minimum, which will immediately give you an unbiased estimator based on the minimum. First find the CDF of $X$, then apply the standard formula for the density of the minimum.
Please explain what the laws of "statistical regularity" and "inertia" are.  Although I can Google references to them, none seem authoritative and I have yet to find any that is either clear, rigorous, or quantitative.
Oh and indeed, in the R help I see that I can use a formula. Therefore it works. I thought I could only two input in the function `x`and `g`. Ok, thank you!
I ran my data through Stata's bpmedian and bpdifmed and solved this problem.  I'm still not sure why my bootstrap doesn't work.  I ran a million replications and got the same results I got at 100,000, so it's converging.  There must be a problem with the algorithm.
+1 for pointing R^2.
Welcome to cross validated! In oder to get any more specific answer than a "yes", you'd need to provide more details about the application and models and similarities and differences between your two elevation data sets.
Thanks a lot to Alecos, your analysis is very impressive and quite insightful! I have couple comments though, but they are long so I post them as a separate post below.