tttjjj/elt_bench huggingface dataset:
It's not a bug. If you read the help file, you'll see that an equal number of observations are trimmed from each end.
A method sometimes used in these contexts is to develop a segmentation for the product penetration. Based on the historical acceptance rates of the products at the leaf nodes and the probabilities of the 5 models, a subset of products might be arrived at.
Why would this family of distributions be particularly good for this problem and not some other family like the Pearson distributions?
@user603 I understand that in principle--but do the timing. Moreover, the main point of using these constructs is to provide semantic help for parallelizing the algorithm: the difference in how they *express* it is important. (It may be worth recalling the original question seeks C/Fortran/etc. implementations.)  Ahfoss, I thought about limiting the calculation to the lower triangle too and agree that in `R` there seems to be nothing to gain by that.
"you have only two numbers" - is there a difference if I compute average coverage for all Set_A and Set_B or use coverage for every sequence independently? r-bloggers example use  non-averaged input (input for t-test is a string).
(+1) Ignoring smaller PCs can't be justified in general, for all purposes, only with respect to some particular subsequent analysis. There's no law of nature that bigger PCs of a set of variables correlate more strongly with other variables from outside the set.  See @NickStauner's answer [here](http://stats.stackexchange.com/questions/87198/pca-randomness-of-component/87231#87231).
Does the problem imply the assumption that the population (sample) sizes are very different $n_{male}>>n_{female}$?
The thread at http://stats.stackexchange.com/questions/5534/transforming-arbitrary-distributions-to-distributions-on-0-1 shows how any continuous distribution (such as a Normal distribution) can be "mapped" (in a meaningful way) to a *uniform* distribution.  Applying the inverse of the mapping for a second distribution gives a transformation between *any* two continuous distributions.
BTW, what exactly do you mean by "stratified" cross-validation? On each CV fold you have 120/10=12 test samples, with always 4 matches and 8 non-matches?
Joe, there is no need to base sample size calculations on a two sample t test. It is easy enough to find the equations that compute sample size to compare two proportions.
Thanks for this. Exactly the answer I was looking for.
Logistic regression is intended for developing a probability (risk) model, not for classification.  And be sure to study proper scoring rules.
Off topic I think, unless you want to ask a question about how create your normalised score from the raw frequencies.  But is http://en.wiktionary.org/wiki/Wiktionary:Frequency_lists#English of some help?
"differencing is a method of detrending" are correct words. Though one is free to apply other methods to take away trend (such as regressions). Pity you didn't declare what's your eventual aim is. Still, you said you are happy...
I have written a little guide, in the hope of making sense of multiple testing:
http://arxiv.org/abs/1304.4920
I really appreciate your help . Thank you .
@PatrickCoulombe: thanks for your comment. This is actually a real problem I'm dealing with, and am not sure where to turn. I'd be grateful for any ideas, methods & suggestions that would point me in the right direction.
[Here's a textbook](http://www.amazon.com/Introduction-Econometrics-Edition-Addison-Wesley-Economics/dp/0138009007) some classes use.
Thanks chl for the answer, I accepted it for the sheer scope of it.  Best, Tal
Both. I expect that a correlation of X and Y can bee seen in each individual and on each time point.
Good suggestion. I´ll try it.
@MarcClaesen: thanks - I hadn't seen that. But wouldn't you need to supply *two* instances of the same case one weighted say with 1/3 class A and the other with 2/3 class B? In any case, not needing to supply lots of copies of the clear cases will make the tuning less problematic (for my data I anyways have to do the tuning splits externally as I have a "hierarchical" data structure with multiple measurements of the actual cases)
@user603 i don't understand your answer. I mean that reading some economy book i found that PP test is often used, so my questions are: is there a sense to use ADF? do i have to use both? is PP enough?
Ok thanks, I found it kind of weird to find sufficient statistic for a function of a parameter. I think it should be just about finding the sufficient statistic for $\\theta$ and then evaluate different things about the function $\\tau(\\theta)$.
Related: [Things to consider about masters programs in statistics](http://stats.stackexchange.com/q/25713/2970).
@Theja thank you, I am not looking for a proof (too much work) but merely advices and comments,  The main problem I see is to base the computation on running estimate of the median,  as whuber has pointed.
In the correctly specified null-hypothesis, both models are "true", but the nested one has $m$ parameters fixed at the true values.  In the misspecified null-hypothesis, both models are "false", but the nested one has $m$ parameters fixed at the pseudotrue values.  ("Pseudotrue value" being the asymptotic value of the parameter that minimizes the Kullback-Liebler distance between the misspecified model and the true model).  So your example of the noncentral-F is not relevant, since that is the distribution when the null-hypothesis here is false.
(+1)  Nicely done.  The generality of this approach reveals the essence of the problem.  Note that one common situation is when the $x_i$ are independent, $\\sigma$ is a scale parameter, and the fourth moment exists.  Then, $F$ necessarily is a constant times $\\sigma^4$ (and we can easily recover that constant from the kurtosis, the mgf, the cf, or the cumulant generating function).
@svadalli - the Bayesian approach does not take the view that $\\theta$ *is random*.  It is not $\\theta$ that is distributed ($\\theta$ is fixed but unknown), it is the *uncertainty about* $\\theta$ *which is distributed, conditional on a state of knowledge about* $\\theta$.  The actual probability statement that $f(\\theta)$ is capturing is $Pr(\\theta\\text{ is in the interval } (\\theta,\\theta+d\\theta)|I)=f(\\theta)d\\theta$.  In fact, the exact same argument applies to $X$, it too can be considered fixed, but unknown.
Yes, if you are looking for the joint probability $P(X1, X2)$, then assuming that they are independent, you have $P(X1, X2) = P(X1) P(X2)$
There is no such thing as "the sample quantile". There are many definitions of sample quantiles. You need to specify which one you mean.
Deriving the conditional expected values is irrelevant to the question as stated. Anyway, it appears you would need to make a distributional assumption on the white noises. The easiest one would be to assume that they are jointly normal.
Sure @Glen_b, the video was this: [link](http://www.youtube.com/watch?v=qH_c-AZ2zt0) The comment was at 40min. It was saying that thing: when ROC have the AUC between 0,5 and 0,6 it was Poor. If between 0,6 and 0,7 it´s below average. If between 0,7 and 0,75 it´s a average/Good. It betwwen 0,75 and 0,8 it´s good. If between 0,8 and 0,9 its Excelent. If higher than 0,9 it´s suspicious and if higher then 0,95 it´s overfitted. I´ve found that explanation very easy to understand, but it´s right? Because I´m searching for something to backup that thought but i´m not finding.
Game-theoretic methods may provide more insight here than purely statistical methods.
I think that this feeling comes from the fact that given the way your 4×4 matrix is generated, all rows and all columns sum to 112. This is definitely not a classical contingency table.
There seem to be several rather different questions here: are you asking how to (a) find a "maximally uniform" measure, (b) whether it has any connection with the length functional, or (c) how to minimize the length functional?
+1 for extensive overview. Now I know too what reject inferencing is :)
Thanks for this. It's one of the simplest examples of what I was seeking so far.
Cohen's d sounds like a good way.
Once you have the analytic forms, and they have been properly derived, then a lot of the hard work is done.  Good job Daniel.
The confidence bounds computed by standard procedures assume that uncertainty comes from sampling a finite sample from a population, not from measurement error. So, even if you need it,  the standard approach won't give the bounds you are looking for.
Here's a really nice resource for the theory behind the glm function in R: http://people.bath.ac.uk/sw283/mgcv/tampere/glm.pdf. Since you're using family="binomial", I believe deviance is just -2*log(likelihood). You've left the world of Sums of Squares and have wandered into the land of likelihood. Things will feel a little strange, but that resource will walk you through the analogs to ordinary regression.
Thanks ars, I didn't know of this one: +1
(-1) As one can tell by comparing 1/51 to 11/36, this reply does not address the question's request to explain how one could arrive at a result of 11/36.
Another question about quadratic forms can be found [here](http://stats.stackexchange.com/q/11289/6633) so maybe this question also could be considered here?
Hi, I just corrected some of you grammar, can you please read over it again and check whether the meaning is correct? Also, please edit the question and add some details about what software you are using, and what code/commands you used to produce that error. Welcome to the site!
The mean value of the observed frequencies is an inferior estimator when $N_{\\text{transmitted}}$ varies appreciably, because the frequencies for the smaller $N$ are less reliable.  What matters (usually) is the *total* received divided by the *total* transmitted.  [Poisson confidence limits](http://www.math.mcmaster.ca/peter/s743/poissonalpha.html) will work well.
Polynomials of degree $n$ have $n+1$ d.f.
Basically, statistically yes, but there are a couple more technical details (e.g. the type I error rate and the percentage of the confidence interval, etc.) Maybe this [page](http://onlinestatbook.com/2/logic_of_hypothesis_testing/sign_conf.html) can give you some insight.
I ended up implementing a variation of this algorithm with a metric called  Dunn’s clustering validity index
It is correct that a lognormal distribution is involved, but it is unnecessary to invoke it, as my reply shows.  The underlying normal distribution *is* symmetrical of course.  The fact that a stock price (or anything else) has no upper limit does *not* imply its distribution has infinite variance.  In the Black-Scholes theory, by the way, volatility is indeed the variance parameter (for the distribution of logarithms).
Have you tried looking at transformations of random variables? For instance, one might try characteristic functions, Laplace-Stieltjes transforms, etcetera.
That could work, but it might be inefficient. Why don't you just interact the group variable with the coefficient you're looking at? `lm(y ~ X:Group + Z:Group)`
There's not enough information to discern what you're asking. Please clarify your question. What are n and q? It may be that 'smallest solution' might mean a value furthest left on the number line (my first guess, & the usual meaning of 'smallest'), or it might mean a number closest to 0 ('smallest magnitude') or it might (though I doubt it in this case) refer to the number of elements in q (smallest dimension), or a number of other things.
Thankyou for your help @Anony-Mouse. I have 130 weeks of sample actually.. I'm bit scared with the size of my data, but let's see if that works. I'll follow your advice. Thanks!
Could you please tell us a little more about your experiment:  sample size, number of items in the scales?
Proportions mimic *all three* properties with corresponding properties of their own. Proportions (in the sense intended in the question) lie between 0 and 1, the proportion of times a sure event occurs is 1, and the proportion of time A or B occurs is the sum of the proportions if the events are mutually exclusive.
At the moment SPSS is doing a really good job. Still looking for a way to use the Matlab MDS function. It gives me an error.
You are welcome to include contact information in your site profile, David, but inserting that in a question, answer, or comment is contrary to site policy.
Instead of telling me everything that wrong with current code, could you give hint at how to approach from beginning?
As an extension to the problem, do you think that a k-dimensional grid in a k-dimensional data space would also suffer from local minima problems?
I don't have any here, I'll check when I get home.  I think it might be somewhere in the **irr** package.  description at http://wiki.stdout.org/rcookbook/Statistical%20analysis/Inter-rater%20reliability/
+1 MDS is the main kind of analysis that I've seen for this type of study.
Your first sentence is incomplete.
To be honest I am not really sure. I think Cohen's kappa (or simply a confusion matrix) can provide a good picture of how well a clustering algorithm performs for -your- data, and not just how well the algorithm performs in general. Mostly I was just suggesting that a p-value derived from this approach, while informative, does not tell the whole story. The reason is that if you know your data is clustered, it's not surprising that a clustering algorithm performs better than random selection (and thus has a low p value).
If you know distribution of $S$ you can use baye's theorem to find it. It will look something like this : $E[S|S\\leq 3k] = \\int_{-\\infty}^{3k} s\\tfrac{f(s)}{P(S \\leq 3k)}ds$
Just to add, I didn't completely read your question. By ordinary least squares do you also just mean linear regression?
@Scortchi See my update post.
All of Statistics is definitely the book! As the author of several stat books, I am reluctant to put solution manuals on line for two reasons: (a) this reduces the appeal of the book for instructors who want to use the exercises from the book and (b) this induces an endless flow of emails from readers who do not understand (or disagree with) the solutions. However, we did put solutions for [Bayesian Core](http://arxiv.org/abs/0910.4696) and [Introducing Monte Carlo Methods with R](http://arxiv.org/abs/1001.2906) on arXiv.
The title isn't really a question. Can you phrase it differently?
@whuber, I appreciate the answers you've given so far; it does sound like that if the number of samples used to compute each mean is identical, I could use the formula in my 2nd question ($\\sigma=\\sigma_{mean}\\sqrt{n}$) to estimate the standard deviation of the population. 

If that is the case, I can easily construct my benchmarks so that each mean do have the same number of measurements, so that the case with different $N_i$s wouldn't matter.
Even with the clustering?
The goal of CV is **not** to estimate parameters but to estimate the **generalization performance** and **stability** of your **full learning procedure**. If you choose to determine parameters with CV (you probably mean with a grid search), you **should** add another layer of cross validation, to cross-validate the grid search process itself. In other words, the full learning procedure, regardless of whether you are estimating parameters or hyper-parameters, should be cross validated. Please see the links I provided in the comments to the OP.
If the y's are actually iid *there's nothing to do*. You are checking because you anticipate the possibility *that they aren't*. But there's an infinity of infinity of ways that they can be dependent. Given that your original question about checking is *predicated* on the possibility of dependence, please look again at my question in the previous comment.
Call scipy.stats.lognorm.pdf(x, shape, loc, scale) instead of scipy.stats.lognorm.pdf(i, loc, scale, shape).
Well $(n-1)S^2 /\\sigma^2$ is only $\\chi^2 (n-1)$ if the sample is taken from a normal distribution. Is that the case in your question?
I've cleaned up the language. The general idea is ok, but the answer needs some polish.
Minor mistake: $<x_i^2> = 1$ instead of $0$.
Was this meant as an answer? Can you elaborate a bit on why approach 2 is flawed and perhaps give any recommendations on what the original poster should do?
Do you mean two arcs that are not directed. Nodes cannot be directed or undirected.
The performance data is a (standard) CPU benchmark which times how long a task takes to complete -taken at fixed intervals of 15 minutes. There is no periodicity in the data, there are also no 'global' linear trends but some of the experiments showed some local ones - so performance got better linearly over a period of 3 days in one case. But in general the data is non-stationary but with no cycles or trends.
I like your approach. Wouldn't it be more realistic to assume that the number of each letter is fixed and that an ordering is drawn uniformly among all possible orderings? Unfortuately I don't know what the distribution is in that case. Any idea?
@Dali: I think IrishStat is saying that you try all points with Chow, and see which are most likely.
If I've correctly parsed this question, you are asking why $f( {\\rm var}(X) ) \\neq {\\rm var}(f(X))$, where $f$ is the inverse-logit function. For some intuition on why this inequality exists, you may want to read about [Jensen's Inequality](http://en.wikipedia.org/wiki/Jensen%27s_inequality).
I don't have a source on this right now... but are you aware you can optimize a logistic regression model to maximize the Area Under The (Receiver Operating Characteristic) Curve (or AUC)? No need to reinvent the wheel.
@Procrastinator, that comment (with some more detail) has the makings of a nice answer, if you have time to give one
@EpiGrad, Fair enough. But I seriously doubt that you'll find a study that has a glaring statistical error that doesn't require some digging to understand that had widespread, substantial consequences. Maybe other respondents will kill my optimism in researchers, though (heh).
Ah yes, I wanted to comment on that but forgot. I think that by inspecting the alignment matrix (i.e. extending the regular backtracking algorithm) you should be able to find those other sequences as well. They show up as cells where the jump (cell value difference) is 2 or more rather than 1, and they should appear on the same row or column as the optimal jump (with difference 1), assuming you indicate a match by incrementing the cell value by 1. I may have missed something - it could be tricky if these suboptimal values are very close to the optimal values.
Maybe I wasn't clear (I have clarified this in the question now), I'd like to the data to be predictions/estimates about certain quantities with precise quantities that are knowable. Collaborating filter data sets generally represent each judge's own preferences and don't corresponding to objective ground truths, either individually or in aggregate.
Thank you all for the insightful answers.
There seems to be a strong tendency for people to hope than any unimodal curve is normal. I see no reason why such a valley would be closely approximated by an inverted normal curve, and many factors such as the erosion from water which may be unimodal, but where any accurate physical model predicts something other than a normal curve.
@mbq thanks for clarifying the confusion with _names(mydat)[c(name)]_ Since the OP has written _I know that the column/variable name "name"_ I was like "wow that is possible ?" and tried to access the names by name ... stupid me ;). May I humbly suggest to edit the question ?
By 'singleton' feature i mean a feature that is only present in one sample.
(+1) I prefer your reference (I have it at home :).
+1 for the follow-up with the code. Always good to see that.
The tags are typically used to refer to the problem at hand. Placing 4 tags for statistical software is strange. I might suggest the [longitudinal](http://stats.stackexchange.com/questions/tagged/longitudinal) or the [prediction](http://stats.stackexchange.com/questions/tagged/prediction) tags. Once you figure out exactly what you want to do then you can worry about the software to do it - before that point it is IMO best to keep it language agnostic.
I am surprised by your argument about categorical responses $Y$. *If* OLS is being used to fit $Y$, then it is computing $\\hat{\\beta}=HY$ where $H=(X'X)^{-1}X'$. To fit a centered and/or scaled version $Z=(Y-m)/s$, observe that $HZ$ = $H(Y-m)/s$ = $(HY-Hm)/s$ can be reversed to obtain $\\hat{\\beta}$ = $sHZ+Hm.$ Nothing at all is lost and nothing is any different when $Y$ is categorical. (However, usually one shouldn't be using OLS for truly categorical responses anyway.)
I have made some edits to your question. Please, could you check that I didn't alter its original meaning?
One approach to this kind of problem is pretty much covered - with a partly worked example - in my answer here:
[nls curve fitting of nested/shared parameters](http://stats.stackexchange.com/questions/44246/nls-curve-fitting-of-nested-shared-parameters/44249#44249). The alternative - if the data are all completely independent - is to compute the standard error of the difference in the estimates from the separate regressions and do a form of t-test based on that.
If you have experience of calculating initial centers by taking random samples, please do let me know how big sample is good enough for this purpose? and also if there is any special technique to take the random sample?
From my experience the best way to be certain is to email the authors
I don't know of an online table, but MATLAB will compute values for you, and I suppose that R or Mathematica or Wolfram Alpha or Maple or ... etc would do the same.
I am afraid that as your sample selection was non-random, I don't think you can get anything reliable from any analysis, however, you can still run your analyses for exploratory purposes to identify potential relationships for further study with a more rigorous randomisation of the next sample.  In this case even a P of <0.3 or >0.7 might be of interest.
+1: Thank you for your answer. It was helpful to my search, but I can't mark two answers as accepted.
@BlainWaan: In our book, we use the R language to teach how to program the algorithms. The package is just there to avoid the user re-typing every example. I am not trying to (over)sell the book, however I use it in class to get the students to learn Monte Carlo and MCMC. If you look at the book online, you will see we cover exactly the program you set in your question...
+1 for the first sentence!
you should copy that here before it gets deleted...
+1 mpiktas: Your solution is more ingenious than mine and I think it should be used in more complex practical situations.
Edited the variance.  You're right.
The approaches are the same, but the exact parameterizations of the model being estimated are different, so the results appear different.  Consider a model with two binary regressors $x_1, x_2$ and an interaction.  You have four categories, but you can write the model several different ways, e.g., let 1 be a constant term, with variables $(1, x_1, x_2, x_1*x_2)$ or $(x_1, x_2, x_1*x_2, (1-x_1)*(1-x_2)$, or others.  The variables are just linear combinations of each other.  The coefficient estimates will be different, but the model is really the same.
Have you checked out Analysis of Financial Time-Series by Ruey Tsay? Chapter 5 of the book covers High-Frequency Data Analysis. Looks like you might find useful information there. I won't provide a link, but if you do a quick search you'll find it online.
It depends on the platform/software you are using but in Excel you can do this with filters. Incidentally, this seems off-topic on this site, except if you tell us what you want to find out and expect some advice on how to visualize or analyze these data beyond merely filtering.