tttjjj/elt_bench huggingface dataset:
But the ones I remember are a biased sample of the ones I have come across, a biased sample of the ones "in use". A biased sample of a biased sample is not a basis for an answer here, but in fact is a basis for closing your question altogether.  Among the 'question closure' options is **primarily opinion-based**, and I think this line of questioning crosses quite firmly into that territory. It would be a subject for a general chat, but Ten-fold has been frozen for inactivity.
Residuals indicate how fitted values depart from the assumptions of your model, in a general sense. Their distribution matters, and has a [greater interest](http://stats.stackexchange.com/a/6351/930) compared to that of raw data.
Questions about R can be asked here too  :)
It's under `10.3 Proportions in two independent groups` with `10.3.1 Confidence interval` , `10.3.2 Hypothesis test` and `10.3.3 Continuity correction`. The passage appears as a general conclusion of `10.3`. Chapter 10 is about `Comparing groups - categorical data`
Have you considered running a Poisson model for sales with a robust variance-covariance matrix, with store size as an offset?
Strength of evidence...or merely amount of interest?  Or size of polygon?  To make progress, it would help to have a quantitative model of what this number implies.  For instance, if each "paper" represents an attempt to identify whether "something occurs" and typically a paper can fail to identify an occurrence with probability $p$, then $n$ *independent* papers will fail to identify the occurrence with probability $1-(1-p)^n$.  (I bet nobody's software combines weights this way!)  Such a model would indicate how to estimate your parameters.
Also asked on dsp.SE
Sorry yes paper is not open access. I have pdf but I think not write to post it. Paper said "cumulative hazard assumption was tested by Schoenfeld residuals and by testing the time-dependent coefficients", so not critical this, but not known if parametric relative survive is better ? Think bias from missing data, but I must appraize survival model in primary for essay in 1500 words +/- 10%. Now have 200 words, so it problem is for me.
@NerdLife: Pooled variance, which you said you used, amounts to assuming equal variances.  The "normal" way of doing it is to assume that $n_1 = n_2$, and in that case pooling doesn't matter to the value of the test statistic.  The OSL is affected because the degrees of freedom shift between$\\n_1-1$ and $2\\,(n_1 - 1)$.  But those shifts are pretty minor above 30 or so df.
Controlling alpha error across slightly unrelated tests may not be a big deal; I'm not sure that one shouldn't recommend using Levene's test just for this reason. Maybe a little additional info on why one would care about that problem would be useful here? +1 in any case!
@stan This will give you the Beta distribution which has the same mean and variance as your data. It will not tell you how well the distribution fits the data. Try the [Kolmogorov-Smirnov Test](http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test).
It's easy to come up with many different ways of quantifying "similar" in this context.  It's hard to choose among them without knowing anything about *why* you seek to measure similarity, *what* you will conclude from the result, and what the possible consequences of an incorrect conclusion might be.  It would also help immensely to have some kind of model, either physical or statistical, describing *how* differences between the matrices can arise.  Could you perhaps share some of that kind of information with us?
If you have access to Billingsley's *Probability and Measure* text, this is discussed in a section entitled, I believe, "The method of moments". (Apologies for the vagueness, as I don't currently have it at hand.) If I recall correctly, the proof he uses relies on the corresponding results for characteristic functions, though, which may not be completely satisfying. This is certainly (well) outside the scope of the expected background of Wackerly's text.
Actually, I did not use prcomp. I implemented my own version of PCA following the exact steps of tha algorithm ( computing the covariance matrix, and then taking the first "m" eigenvectors that are needed to compute the final reduced data set)
Did you write your own function for Manly selection ratios or is there already an existing function? If you wrote your own function, would you mind sharing your code? Did you find out the answer to you question in meantime? Thanks a lot!
+1, nice answer. The DF-GLS command has also the nice property that it automatically selects the lag length via the Schwert criterion.
By "value", I guess you mean "parameter"?
Ah! No wonder I couldn't find it in R. You should probably edit your post or your tags to clarify
@Gaetan Lion: in this code, $x_1$ and $x_2$ are independent root normals, and $y = 3x_1 + 2x_2$ plus a normal noise term with standard deviation of 0.3. Clearly $y$ is positively correlated to $x_1$ and $x_2$, which are independent.
So do $f(x)$ and $f(x)+2$ but they would not be equivalent likelihood functions
Hi Frank, it is a pleasure to see you here!  Thank you for your answer.  I wonder if there is some way to compute such a P value from the deviance of the models (maybe combined with something like bootstrapping on it).  If you (or anyone else) can help direct me to any way of achieving the computation of some "p value" for the comparison of the two models, it would be great.   In any case - again, thank you for your answer.  Best, Tal
Thank you for your answer. I guess there is no simple answer to my question. I will take a look at your recommendations. Cheers.
Ah you are right, I just made an arithmetic error. But anyway you cut it, these variables don't discriminate your data very well. Since most people have cancer in this data set you can do just as well at predicting whether they have cancer by just saying they all have it.
@AdamBailey are you suggesting using a Mann-Whitney U-test to test if the distributions are dissimilar? I am familiar with this approach but I was hoping to ultimately generate an estimate of how different these samples are, i.e. some form of logistic regression as you can just look at the distributions and see how different they are. There may also be other variables that I would like to include in the modeling effort to ensure that these results are not an artifact of time of day, season, or some other variable.
+1 for the G*power reference, which I had never heard of but just downloaded.
Actually, it does. I have `country.x` in both. The jury is still out.
I don't think your edits allow more precise advice. You are telling us nothing about your data or even about what kinds of deviation from Gaussianity you fear. No indicator of non-Gaussianity, scalar or vector, leads ineluctably to identification of a better distribution. If the main issue were skewness, it might be that you should think of in terms of gamma distributions; or if it were fatter tails, of a family of t distributions. Both of these families have Gaussian as a limiting case. If you ask around, you would get other suggestions.
Great explanation. Thanks!
I guess that depends on what you mean by "difference". :)
Simple regression is like treating each record as independent. In that case, the sample size would be exaggerated since the records within the same state are possibly correlated. A multilevel model introduce a random random effect that allows each state to be different. It is also known by a couple of different names though, random/varying intercept model; mixed effects model. Hope this helps.
@AndyW So do you suggest the Poisson is good enough? I also don't want a model that is too complicated and unintuitive to interpret...
You should stack them if you would like to use them for ease of computation.  For example, check out how you can stack the values into vectors and matrices to estimate $\\hat\\beta$ http://en.wikipedia.org/wiki/Seemingly_unrelated_regressions#Estimation
I guess my main point is that I'm not trying to forecast the effect of delays - the forecasting should assume there will be no further delays at all. Suppose the percentage increases steadily (with minor random variation) at 1% a day for 30 days, then 0.1% per day for 10 days, then 1% a day for 30 days. Based on the entire data, and the assumption the progress should have been linear overall, it's logical to assume a 1% increase for the whole time. A linear trend which uses the entire data would predict a much later completion date. But I don't want to ignore the first 30 days altogether.
**She** gives ...
@Michael: That looks like a very nice way to interpret the question. Your argument immediately proves the value is $0$ for any distribution of $X$ that is symmetric about $0$.
If they have different distributions then they are different in some way.  A physical explanation for the difference is not a statistical explanation.  The answwer lies in the science and not in statistics.  This may not be an appropriate question for this site and you may want to migrate it elsewhere.  A moderator can help you find the most suitable site.
Ah!  So the downward spikes are where a foot hits the ground and the rest is the "jiggling" in between.  If that's the case, even a crummy low-pass filter should do the job.  (http://en.wikipedia.org/wiki/Low-pass_filter )  *E.g.*, fdesign.lowpass (http://www.mathworks.com/help/toolbox/filterdesign/ref/fdesign.lowpass.html ).
Outliers in the input space is a major issue in most types of models. So, I do not fully agree with your answer
Google searches suggest, by 20 to 1, that Easterbrook originated this quotation, but he didn't really start writing until after Coase was quoted in print.  The best evidence I can find concerning this (and it's still not very good) is Coase's Wikipedia page, http://en.wikipedia.org/wiki/Ronald_Coase .
Yes, I think you could embed sub-features into the feature-vector (flatten it, so to speak), or preprocess sub-features in a preliminary step to produce a signature value (distinctive scalar) for every sub-feature vector.
Thanks! There are a [lot](http://archive.ics.uci.edu/ml/datasets.html)!
You ask what model to use, but you do not tell what are you trying to model.
I'll update the answer a bit to be more convincing but unless you can come up with a more informative response at to *why* the facts of the matter aren't convincing it's difficult to assist you.
Any idea what such a model would predict for X above?
Just a note regarding R: The package `mixtools` ([link](http://cran.r-project.org/web/packages/mixtools/index.html)) is what you need to fit a GMM as noted by Pat below. The function `normalmixEM` implements the EM algorithm for the mixture of univariate normal. Use the option `k` to specify the number of components (`k=2` for a two component and `k=3` for a three component mixture).
@ttnphns Oh, you right, that was stupid. I wanted to say that A'A is practically non-invertible because of numerical errors. But theoretically it still has a full rank. Can't I premultiply 5 by A' to get A'Ax = A'AVx* and then premultiply A'A by its inverse to cansel them? I'm doing inverse analiticaly, not numerically, so should I consider it to be a problem?
p.s.: Sorry for notation. It's my first question here. Probably, I need some time to figure out how it works.
this is the first time in over a year I've actually found a use for my web-app!
@ttnphns, it appears the question is: why is $\\widehat{\\Sigma}_w$ a reasonable estimate? What I'm wondering is: what is $w$? It only appears in the subscript for $\\Sigma$ and nowhere else. As you pointed out, $t$ is also mysterious.
@Andy W: In my Stata program, the variables on the sides of the plot are ordered as they come, numeric order or alphabetic order for strings. If someone wanted a different order, they'd need to recode the variables. I don't know why Maarten had the order "orange" "pear" "apple"; perhaps he's thinking in Dutch then translating back to English (serious comment; I know him).
Seems like a good question to me.  If you only have a Bachelor's degree, you will be lucky to find a position, most likely as a SAS programmer.  Most advertised positions are looking for a Master's or above.
@Whuber I should have added that the actual data is useful to empirically identify which of these states of nature is most  applicable  thus answering the question by example as there is no theoretical answer as far as I know.
what makes your outliers outliers?
If it's not a count, I wouldn't say 'binomial'. You *might* say 'beta'. (Similarly, the second might be a recaled beta.); you can of course transform either to the real line also.
Also a favorite topic of Michael Shermer of Skeptic magazine.  "That's not even wrong."
@whuber That's what I can't rectify.  If the probability of an event taking place is 0, then it should never happen.  For example, if I have a standard six-sided die, the probability I roll any number $\\mathcal{Z}\\setminus\\left\\{1,2,3,4,5,6\\right\\}$ is 0 and therefore will never happen.  Moreover, how can an event with probability 1 not be a certainty in the subsequent experiment?  Could you provide an example?
Clearly the marginal relationships are nonlinear. The question is whether the conditional relationships are (I'd bet heavily on 'yes'). If none of the variables are exactly zero, I'd start by looking at same plot, but of the logs of the three variables, which might also improve the apparent heteroskedasticity.
I don't know enough about what you're trying to do, what a suitable loss function might be for your problem. I suggested a common measure of goodness of fit (mean square error), which would be a reasonable way to measure discrepancy between the red line and the data.
No. Unless you are willing to assume more about the data than what you have described. But doing this would also most probably go counter the reasons you have to want to estimate the median in the first place.
For the auROC 0.5 is the minimum (because less would be better by inverting the predicition). Is there some similar rules with the auPR? Also concerning my measurements : what could I assert by looking at the scores of the Method C? Because I'm working with the same dataset in the 3 cases and from my point of view for a dataset with more or less even distribution among the classes it wouldn't make sense that the auROC and auPR do not follow the same ranking for my methods.
'The method just described is based on a strategy similar to Gosset's derivation of Student's t: Assume normality and then make adjustments so that for small sample sizes, accurate probability coverage is obtained.'
StasK, thank you for the answer and the references. I have another question, related to this one, but also to the answer you gave on August 10 to the question "When to use Student's or Normal distribution in linear regression?". There you imply that in the presence of clustered standard errors, one should use the normal, rather than the t-distribution for testing hypotheses about population parameters. Why then do software packages report "t-values" and probabilities in the output of regressions with cluster option? And one more question: can you test that a cluster-effect is not present?
On a side note, what is the class called? I can't seem to locate it in the jar file
So you're saying the PDF is $k\\binom{a}{b}p^k(1-p)^{n-k}$ instead of the usual $\\binom{a}{b}p^k(1-p)^{n-k}$? Does that still sum to one over positive integers? Also, if you know the analytical form of the distribution, you might find an answer in this question from a year ago: http://stats.stackexchange.com/q/67911/36229
This is a typical load forecasting problem.   The seasonal terms are hour-of-the-day, day-of-the-week, week-of-the-year, holidays, economic activity, temperature, etc.  Notice the general rise in load in the top/left graph as temperature droops in the top/right graph.  Do a Google search for "forecasting electrical loads" and you'll get all sorts of information.  Also, search this site for other answers on electrical or power modeling/forecasting.
+1, wow, this threads the homework needle pretty well.
C45 can be tuned to build such kind of groups.  However, before delving into details: what problem do you want to solve?  What means interesting?
@Harpreet If you make it to section 3 of @Hong's Wikipedia reference you will see that Sklar's Theorem answers your question affirmatively (http://en.wikipedia.org/wiki/Copula_%28statistics%29#Sklar.27s_theorem ).
This question would be of interest at http://gis.stackexchange.com because most of the examples will be maps.  A nice example (comparing GPD and GPD per capita) was quoted in a recent discussion of cartograms http://gis.stackexchange.com/q/7406/664
The Graduate Diploma exams are effectively final year undergraduate exams; for "staging" purposes there are lower level certificates that can be taken first. The RSS exams are available, if I recall correctly, worldwide with the exception of Hong Kong (which has its own statistical society and exams). An alternative is the undergraduate Diploma in Statistics offered by distance learning by the Open University in the UK, but again available worldwide. This is of slightly lower level than the RSS Grad Dip so may be seen as preparation for it. As a taught course it's substantially more expensive.
@Wayne - yes, I too had a range in mind, though didn't voice it. I think that's a rather minor implementation detail. But anyway, nice to know that others think the same way. :) About the ordering - for the most part I can order them any way I like. It makes more sense to group similar questions together though. And, of course, if a question depends on a previous question, then that order is fixed. But big class-defining questions will definitely go up front.
as it turns out, for most weighting schemes, only the first few $w_i$ are non-zero, which should also simplify the computation.
What is this "often-cited reference"? It would also be helpful to describe (briefly) what the field of application is, and what Cobs, Cmod and R represent.
+1, great answer.
Thanks for taking so much time explaining this to me.
the [truncated](http://onlinelibrary.wiley.com/doi/10.1111/j.1469-1809.1947.tb02410.x/pdf) binomial?
This could be improved by replacing `SQRT` with the Cholesky decomposition `chol(invCovMat)`.
(3) Another concern is that your data may have strong interdependencies: if you have the record of all $k$ interactions between a given individual and the government, do you then include $k-1$ cases in the regression corresponding to the state after each interaction? If so, the standard logistic regression model is likely a poor one.
+1 Computing a DF for weighted regressions never made any sense to me.  Parametric bootstrapping will do a good job in estimating standard errors.
There are several questions on the site that hit on various aspects of your question. In addition to @PeterFlom's answer below (+1), I would suggest you read through several of the highest voted threads under the likert tag ([list](http://stats.stackexchange.com/questions/tagged/likert?sort=votes&pagesize=30)).
Thank you very much for nice explanation. I was confused between these two alternatives. Second option is what I was thinking. It is mainly for visualization.
@Simo I searched for: discrete density estimation, discrete kernel density estimation, and combinations of these using quotations. I hope this helps.
It's similar, but a different implementation, so the results are not the same. I notice that Matt Wand (author of SemiPar) comments at http://www.uow.edu.au/~mwand/SemiPar.html that mgcv now does most of what was intended for SemiPar and that he advises people use mgcv instead.
Do you have a reference to the other study who's method you are trying to reproduce?  It's unclear to me what steps 2, 3 and 4 are doing.  In step 2, is the correlation being calculated within pairs of genes in set 1 and within pairs of genes in set 2?  In step 3, are you testing if the distribution of correlations in set 1 differs from the distribution of correlations in set 2?  In step 4, what does it mean to "downweight" a correlation?
I'm not sure I follow this, & I'm not sure the issue is actually statistical. Is your situation that you will have people make a 5 point ordered rating on each of several variables for a set of items, & you want to know how to convert those component ratings into a single unified rating? If so, the issue is how much should a 1-unit improvement in `taste` be weighted vis-a-vie a 1-unit improvement in `value for money`? That isn't a statistical issue. You could run a study in which you also got overall ratings & used statistical analyses to infer how the components combine, however.
highchart is a good tool and your question is very fuzzy. what you will do with answer? how it will help you? can you share your visualization and why you did it?
+1. One could go further in emphasizing just how large "many" is here. Take any $p$-parameter family of distributions $\\Omega=\\{F_\\theta\\}$. Consider the function $u$ defined on $\\Omega$ given by the ratio of the mean to the standard deviation: any contour (level set) of $u$ in $\\Omega$ satisfies the required conditions and implicitly has $p-1$ parameters. When $p\\gt 1$ this will (generically) itself be a large distributional family. For example, applying this to the Normal family gives a Normal$(|\\mu|,\\mu^2)$ family. In fact, this construction works even when $\\Omega$ is nonparametric.
Can't you just simulate data whose mean is a known smooth function and add random noise? For example, `x=seq(0,2*pi,length=1000); plot(sin(x)+rnorm(1000)/10,type="l");`
Highly relevant threads: [Is it valid to include a baseline measure as control variable when testing the effect of an independent variable on change scores?](http://stats.stackexchange.com/questions/15713/is-it-valid-to-include-a-baseline-measure-as-control-variable-when-testing-the-e) and [Should the difference between control and treatment be modelled explicitly or implicitly?](http://stats.stackexchange.com/questions/15104/should-the-difference-between-control-and-treatment-be-modelled-explicitly-or-im)
Good point! I agree.
Awesome! Thanks everybody.
(+1) Nice update. $S_N$ can simply be viewed as a Riemann-sum approximation to the integral where we use the value of $f$ at the midpoint of each interval defined by the partition, rather than the left or right endpoint. :-)
I would like to suggest that you back up a bit: this question seems to jump into the middle of something without telling us what the original data are, what a "Gaussian smoothed curve" is, nor (this is really important, because this is likely a question about imprecision in numerical calculations) how it is being computed.  It's not possible to determine what you are doing or how these plots are actually produced, so at this point your readers do not have enough information to provide focused, objective answers.  Please include this information in an edit to your question.
I'd have to agree with Peter here. I do know what 'lying by omission' is, however there is nothing to hide, the large confidence interval of the last point would not change the interpretation of what is being shown, but it's inclusion (forcing a large rescale of the y axis) would wipe out anything you might see at the other 12 time points (as they'd be so squashed).
Please see the `self-study` [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) - such questions are welcome here but should be handled differently. In particular you should be showing us what you've done and where your specific difficulties lie.
Hi, saw that earlier. IMHO, the question is slightly different because that question asks for the methods, not which ones are complementary with a KS approach.
see comment above, also, my t-statistic is based on an article from the American Statician: "Thirteen Ways to Look at the Correlation Coefficient"