tttjjj/elt_bench huggingface dataset:
Consider selecting a more informative question title; someone asking a similar question in the future is unlikely to check the answer under a vague title than under a specific one.
@ttnphns Oh, I see. This solved my doubt. Thank you very much.
@Dail ANOVA doesn't test, despite its name, difference of variances. It tests for differences of means of groups. Fligner-Kileen will test for differences of variances of the groups.
@PatrickCoulombe: OK, I'd include the details that I find fishy.
Thanks for the reply. I'm aware of regularization as a method of reducing the risk of overfitting, but how does that answer my question? To clarify - I'm not looking for a better method to learn my parameters, but actually am just interested if removing some parameters after the model has been learned can reduce overfitting, and the theoretical justification behind the answer.
I'm pretty sure this has been asked many times before.
Hey mpiktas, so will I be right if I say the main purpose of aligning data frequency is to get a common time notation between the Y variable and X variable? We can throw away the  'τ' notation and switch it to 't' notation; because in terms of doing regression calculations all we care about is how much the HF lagged variables are affecting the  LF variables, and not how the sampling frequencies differ between the two variables.
Considering the 2D case and contemplating the situation where $b\\gg a$ ought to provide good intuition as to why the answer is no.
Thanks but there is a problem. geom_smooth: method="auto" and size of largest group is <1000, so using loess. Use 'method = x' to change the smoothing method.How can I do ?
I found then notion of a Tobit model for this. It is a bit different as the $Z$ does not have a coefficient. Still I think Tobit models are the path do go.
@ssdecontrol Unless I missed something, the numbers on OP's x-axis appears to be (effectively) ranks rather than IDs. If you take all the individual scores in a sample in order, then a plot of their scores against the values $1,2,..., n$ is (up to interchanging y and x, possible axis flipping and some scaling, possibly including a known shift) the same as the ECDF. The rank of a user corresponds to the number of users who score at least as extreme as they do, so a user's ranking can be scaled to the proportion $\\leq$ than that user's score.
Also, are the 30 output variables the same (theoretically) for both methods?
Unsupervised and supervised learning don't try to solve the same problem. If your question is "which is better" it does not make sense, because they can't be compared.
AFAIK OpenCV and the linked site only implement face _detection_ (ehere in the picture is a human face?) which is only a first step towards face _recognition_ (whose face is it?)
@HugoRune Sound like a hard problem. The technique from the article should work for it in general, unless you are looking for specific shapes or so. 

By the way if you do some reading on [Gaussian processes](http://www.gaussianprocess.org/gpml/) and if you have just a little bit of statistical background you should be able to read it.
They show it doing better for what they want to make it be. The naive method works just fine for what it is. Look at the simulation in section 5. They set up a lnorm distribution mean 5, which has an exponent of 148.4. Then they go on to discuss coverage of the mean of 244.6!! That would only be important if you're going to model the mean of the original distribution, NOT the logs. They're trying to make it something it's not. The naive calculation has perfectly fine coverage over the mean of the log, 5. None of the other CI's are 95% CI's of that value and that's the one you're analyzing.
Hold on, I'm confused. Are the 366 values you presented your population or not? If they are, then cost is irrelevant, you already have them. If not, then what are we even talking about?
Thank you, that looks very much along the lines of what I was trying to do. I'll have to dig into this some more.
I'm not from an econometrics background, but this GMM approach is suspiciously close to the *GEE* approach. Are you aware of whether they're the same?
@zbicyclist that's a good idea, thank you.
Good answer, John.  Note that kurtosis usually isn't worth looking at when a distribution is skewed: the skew already tells you there's going to be substantial kurtosis.  Thus, good applications are with datasets known a priori to be (approximately) symmetric.  Residuals often are this way: the kurtosis can be a quick numerical means to assess the shape of their distribution.  An additional point is that the sampling *variance* of the standard deviation is a function of the kurtosis, so if you're estimating SDs or variances, you might want to glance at the kurtosis to assess the precision.
I will add to this answer that one of the major algorithm used to extract vessel from an image in medical image processing is Frangi algorithm.
Are you claiming outliers are unimportant in Bayesian data analysis?
Two questions. p-value of HHTHTHHHTT is simply 1/1024. Which (more) extreme cases does have to sum up? What is the problem of computing the probability of event aposteriory? I feel like the problem with p-test is not it aposteriorentess but insufficient sample size. Single 10-coin toss just means a single sample. This is obviously insufficient to make any statistical conclusions. But, p-test does not take this into account. Right?
[Doing Bayesian Data Analysis: A Tutorial with R and BUGS](http://www.indiana.edu/~kruschke/DoingBayesianDataAnalysis/) may be a good start. There are also some links for Bayesian ANOVA on this related question: [Bayesian two-factor ANOVA](http://stats.stackexchange.com/q/11437/930). I'm not clear with your last sentence, though, because instead of interpreting p-value we generally recommend using measure of [effect size](http://en.wikipedia.org/wiki/Effect_size).
Take a look at what the [summary.formula](http://biostat.mc.vanderbilt.edu/wiki/Main/HmiscSummaryFormulaFunction) function in [Hmisc](http://cran.r-project.org/web/packages/Hmisc/index.html) provides for the R language. I believe this might be a good start if you want to use other statistical packages.
How large are these datasets? Namely, the smallest dataset?
Just because residuals aren't Gaussian doesn't mean you need non-parametric tests. You can usually discern the type of model to use (yes model, not test) from the nature of the data (count, 0 1 data, continuous, mean-variance relationship, linear or non-linear relationship, etc) and fit models accordingly to meet the features of the data having previously decide what the hypothesis to be tested was. Once you feel the fit meets the assumptions of the model fitted then you can evaluate the p-value and other statistics,
I agree with you that one shouldn't have to get permission to conduct an objective comparison BUT SAS is a major advertiser and perhaps that came into play. I should also add that the Foresite Journal Editor doesn't want to unnecessarily ruffle any feathers and historically there has been an unwritten rule about ruffling feathers. With respect to the M4 competition please see the AUTOBOX Users Group Blog https://www.linkedin.com/groups?home=&gid=2511642 entitled "The Sorry Story .... "
This answer is far too short.  More of a comment than an answer.
@Gregg You and @mpiktas seem to be saying similar things, so I believe comments about his reply may also pertain to yours.
The brutal answer is that *no* statistical tests apply because no randomization was performed and there are abundant powerful reasons to believe there are many confounding factors, which means any model-based efforts at statistical reasoning are almost surely just going to be wishful thinking. This does not mean your data are worthless, but it should help direct your analysis in other ways, focusing on exploring and describing the data and formulating hypotheses for future experiments to be conducted with more rigor.
The negative adjusted R squared is concerning. This usually happens when one tries to fit too many predictors into a model that uses very few number of cases. What is your sample size? And totally how many predictors do you have in your model?
Hm...you could randomly sample and permute 1000 rows by 1000 columns from your data 10000 times to speed up the calculation of the randomized distance matrix substantially. This should be sufficient because permutation thresholds such as these tend to be tight around the mean for the expected number of clusters at given heights. You can confirm this empirically by comparing these results to what you would get from the whole dataset by doing 100 permutations or so.
Dear @vitasoy: We know what a $\\sigma$-algebra is. What we don't know is what your notes say under "A.2" and "A.3". I can guess that it's probably closure under complements (A.2) and under countable unions (A.3), but you need to make the question self-contained for us to know for sure. Cheers. :-)
If correlation and regression are relatively new to you, work through the chapters in Freedman, Pisani, Purves, _Statistics_, New York: W.W. Norton, any edition; or any similar text.
Where are you having "trouble"?  What have you tried?  Both questions require you to know a definition of expected value.  For the first question, you might find [drawing a probability tree](http://stats.stackexchange.com/questions/65177/is-there-any-intuitive-meaning-to-the-quantity-pabpbc) to be helpful. The second is really a geometry question in disguise; the asker likely is expecting incorrect answers of the form "there must be a 1 in 8 chance of hitting the center" whereas the chance is really only 1 in 64.
That was my first interpretation, but it provides different results than the representation based on orders (`, , 1`,`, , 2`,`, , 3`). See this: https://onlinecourses.science.psu.edu/stat510/?q=node/79. There is another example in the book that would be wrong if this interpretation is chosen. That's why I was forced to assume that `, , GNP`, `, , M1`were referring to columns instead of rows.
+1.  Sometimes, as here, a good answer makes a question worth keeping.
@Jerry: Generating one angle between 0 and 360 and another between 0 and 180 DOES NOT WORK! A point near the pole is much more likely than one near the equator.
Related: [History of the high-dimensional volume paradox](http://mathoverflow.net/a/128881/17114).
... have you tried it?
I'm a bit confused here. Could you write the model (with math, not programming code) you're trying to estimate? You seemed to use poisson for the third level, is that correct? If so, it seems to me that that would be a reason for the coeficcients of Bugs and lmer and stata to differ. But in this case, why a poisson to model (hyper)hyperparameters?
to repeat: I think 'testing for normality' is a red herring, I suspect the OP really wants to test for multimodality (i.e. @CMichael's suggestion of `diptest::dip`), but "is it normal?" is the vocabulary they're working with
(+1) The author of `spdep` also has a nice textbook about spatial data analysis in `R` [here](http://www.springer.com/public+health/book/978-0-387-78170-9). I own this book and have found it very useful.
Your example just underlines that very small effects can be significant with very large sample sizes, but here "significant" only means distinguishable statistically from zero. It's best not to talk in terms of anybody or anything being "confident". Also, R-square is only indirectly part of the formal inference.
For those of us--I suspect there may be many--who might not be familiar with any technical uses of the terms "test for superiority" and "test for non-inferiority," would you mind editing this question to explain what they are?
The data set I'm working with represents the most recent 6 months of data, so that much is already in place. And thanks for the help, but I'm really looking for a solution stated in statistical terms.
This definition of "self sufficient" is subjective, because your ability to "understand the book" depends on your background.
@whuber - perhaps to emphasize that it's calculated using an observation from the sample, not a new observation?
I edited my answer, so please have look on it.
@chl, my question is this, in books, when they explain Cochrane and Armitage test for independency (for trend) they say, for recessive model x=0.0 for codominant (additive)=1.0 and for dominant x=0.5... It is variable in the input of the independence.test function. Why do we assume these values? How does it affect the outcome ? Which value to assume if I don't know yet what kind of interaction there is between alleles. Yes it was a binomial phenotype.
This sounds like the "kernel trick" applied to PCA. http://en.wikipedia.org/wiki/Kernel_PCA
It's a very good way of handling certain large matrices.
+1 to using logistic regression to model likelihoods. Also when the OP says "...a customer who buys paper books is more likely to buy an e-reader than a customer who does not buy paper books." It sounds like the question is someone who reads more likely to buy an e-reader than someone who does not. Given that I like your spin of adding more variables into the mix so you can say "Of those who do read paper books... these are most likely to buy" and especially "Of those who do NOT read paper books, these are most likely to buy."
@Skarab Maybe I'm totally off, but wouldn't you expect that the frequency of any word will be inversely proportional to its rank in the frequency table of words, according to Zipf's law (http://j.mp/9er2lv)? In this case, check out the `zipfR` package.
Except for the part about King's College it sounded to me a lot like Karlin and Taylor's A Second Course in Stochastic Processes which is a blue colored book.
Basic logistic regression predicts a binary variable. That's why I specified **ordinal** logistic regression, which is a more complex model for predicting a polytomous, ranked variable, such as a Likert scale. Unfortunately, it may be harder to find a tutorial for such an ordinal model. One way of compromising in your case to accommodate a basic, binary logistic model would be to recode your `1` and `2` scale points as `0`, your `4` and `5` points as `1`, and your  `3`s as missing...but you lose a lot of information this way, so I'd only recommend it as practice with the simpler model.
The take-away I found from this, and why I originally was attracted to this paper, was how to deal with censorship in survival predictions, i.e. what loss function to use (though rereading your question, you may not have censorship).
All these similarities and metrics (note the difference between the two types) measure in some way or other the amount of fragmentation associated with the largest common subclustering between the two partitions. They all use what is known as the confusion matrix. By considering the precise formula for VI it can be understood to be measuring that fragmentation. I would suggest looking at the formula in one of the Meila publications, and also to read up about the normalised versions of all these distances, as they all have different scales. This may be the most important point.
The term Gibbs sampling is often used to imply that you sample one parameter at a time, even if you don't use the original idea of sampling directly from the full conditionals.  All the software samples individual parameters or blocks of parameters in sequence, but the actual step type varies a lot depending on what works best.
one hot encoding?
-1, Answers should be answers, this is more like an unfinished comment. Please elaborate more.
About your last point, you can try another package, e.g. [penalized](http://cran.r-project.org/web/packages/penalized/index.html) (see here more details, [In R, does “glmnet” fit an intercept?](http://stats.stackexchange.com/questions/4543/in-r-does-glmnet-fit-an-intercept)).
You can't *prove* that your data follow a power law.
@PatrickS.Forscher Thank you. care type is direct care(1), housekeeping(2), mealtimes(3), medication(4), miscellaneous (5) and personal care (6). ACTid is the type of care for each row of Y.  so eg ACTid(Y(1))=6. ie nurse 1 did personal care. Do you mean I will need to recode my entire Monte-carlo model in R? Can I not use my data files and tell R the interaction between input and output parameters is: $Y_i=A_iV_i\\lambda_i-\\beta_iY_{i-1}$?
@B_Miner thanks! I was worried that since I thought of this on my own it might be wrong (I am not from "pure" CS) - but I guess it is correct.
PLease could you also be more specific about the other options to fit the two models?
I don't feel it is right to call it repeated-measurements. Your 6 READ levels don't seem to differ in some systematic way (e.g. condition). Rather, you just performed 6 assessment attempts instead of 1 - for precision. Right?
The question is pretty well covered, so just a comment - it's more a question that you're asking a question you already know the answer to (you don't really think the population means will be *exactly* equal, do you?), and you have sufficient sample size to give the already-obvious-but-not-very-interesting-answer that there are in fact differences, even if they're tiny. Effect sizes and confidence intervals are more useful than p-values, and your interesting questions tend to go more to issues of bias and describing the differences that are there and judging their practical importance.
So, can we say that new drosophilla of AI is the actual drosophilla living in a simulated world?! ;)
Anyone.. Please help :'(
A bit more details could be helpful; how they are obtained, are branches weighted?
What are you actually trying to accomplish. You say you are trying to model it but do you have a model in mind? I.e., are you trying to predict the amount of rainfall for each day?  Or make inference about the mean amount of rainfall or....?
I wouldn't use ARIMA on this series as-is; the long stretches of exact zeroes and the lower bound of zero require a different approach. I might be inclined to treat the zeroes as censored (the actual level of volcanic dust is probably just below some detection limit). It *might* be that some ARIMA model on the log-scale with some proper account of that effect may be reasonable, but that's a relatively hard data set to model. (I wouldn't use it to teach someone ARIMA.)
(3) your answer implies the problem of multiple comparisons applies to OLS regression, which it does not, when conducted properly.  The proper way to test a factor in a regression context is to test the nested model with all factor dummies dropped against the full model with all factor dummies included.  This test is identical to the one an ANOVA conducts.  It is true that you should not use the tests of the individual dummy variables (which I suspect is what you are trying to describe here).
@user28681 by computing the confusion matrix (true|false positive|negative counts), you have basically lost the information to compute *all possible confusion matrices*. You can not possibly compute all the possible confusion matrices from one single confusion matrix. You need the predicted scores for that.
Unfortunately the amount of missing data can't be considered as small, so i guess I will proceed to the MI. Thanks
It is probably asking you to indicate what you think the true correlation is. But let me ask you this: What is the purpose of this power calculation?
More simply, the likelihood function for $n$ is $f_n(t)=\\frac{\\lambda^n}{(n-1)!} \\cdot t^{n-1} \\cdot \\mathrm{e}^{-\\lambda t}$ and so $$\\frac{f_{n+1}(t)}{f_n(t)}=\\frac{\\lambda t}{n} > 1 ~ \\text{as long as}~ \\lambda t > n.$$ So the maximum value of $f_n(t)$ occurs at $n_0 = \\lceil \\lambda t \\rceil$. Note that if $\\lambda t$ is an integer, then both $n_0$ and $n_0+1$ are maximum-likelihood estimates of $n$ and this formulation chooses the more pessimistic value (or more optimistic value depending on one's viewpoint, e.g. $n$ is the number of failures!).
My base is currently nominal, and I'm starting to see the limitations that you've pointed out in Stevens' proposal.  Right now, it's a wider hierarchy than it is deep to handle various types that you've outlined above.
(CONTD) ...then, in what sense a quantity that it is a non-trivial random variable at the limit of the index, also converges to zero at the limit of this index, and in a stronger mode of convergence? Again, I cannot see how this could hold, so I still believe that Hamilton proved a (correct) result, using a mistaken argument.
The plot suggests that the effect of wet/dry is different for the two species. How about a `species:type2` interaction term?
It sounds like different trials could be modeled as different categories, in which case you would chose an ANOVA. Depending on your answer to Peter Flom, you would run a repeated measures ANOVA if the same subject gets tested multiple times.
This seems like an overly extreme position to take. Of course rank-based methods lose information too: they discard all the information about the actual values. So the question concerning whether information loss is a concern comes down to *why is the analysis being performed?* If it is for discovery and exploration, various guided forms of binning can do a good job whereas replacing everything with ranks would likely obscure and distort the relationships. If it is for confirmation or hypothesis testing then your remarks would be more generally applicable and defensible.
Welcome to CV. The image and the definition on the wiki page for 'strictly convex space' seem to answer the question. Can you elicite what you find problematic on the wiki page? –
The chance that your boyfriend was born the same year as you is actually very high (especially given many situations tend to bring people of very similar age together); it's a very difficult probability to calculate. If you had that probability, P(Same day and same year) = P(Same year) $\\times$ P(Same day|same year). But P(same day) should be roughly independent of whether you were born in the same year. So it will be $\\approx$ P(Same year) $\\times$ P(Same day).
glad it helped, can you accept the answer then.
Note you can do it in R with `floating.pie()` from the `plotrix` package
Hi, Lews: There are some options here that ought to be spelled out. Because the x-axis is logarithmic, do you want the area as shown or the area as plotted against $M$ itself? Should the area be computed from the apparent data or based on the fitted curves? Because the curves appear to be fits to data and the data show spreads, do you need to come up with a standard error of the estimated area? I wonder, too, whether you really want the area between these two curves or whether you're looking for something else: what would this area *mean* for your study? How do you intend to interpret it?
I'm assuming that you're after prediction, rather than inference?  Low R2 in models aimed at the latter are normal, and fine.  And "R2" isn't really defined for non-gaussian GLMs.  There are pseudo-R2's that are sometimes used, but read up on them.
You will likely get excellent and authoritative answers to this question on the stata list (http://www.stata.com/statalist/ ).  It is THE place to go with issues about how Stata works.
I don't have a strong answer, but I can say that I've made made stupid mistakes on tests where I could've simply plugged givens into the explicit denominator, but I thought I knew P(A) and I was wrong.
+1. a[levels(a) %in% lf] <-"Other" sure saves a ton of code lines. Clever and efficient!
Hi, I wonder what the values will be for the Precision and Recall, if TP+FP=0, and TP+FN=0 for some actual class in the confusion matrix.
Please, never ever even think about looking at error on training when fitting RF (neither most other ML methods) -- as user31264 mentioned, this number is simply meaningless. If you need a training-set based approximation of error, use the OOB error.
One other form of reliability (i.e., apart from that indirectly assessed by internal consistency) is temporal stability (typically assessed with intraclass correlation).
I have just taken a look at Sprenger's paper mentionned in your edit. The Bayesian reference criterion (BRC) is more or less what I called "reference intrinsic inference" in my previous comment.