tttjjj/elt_bench huggingface dataset:
From p. 141 of Hadley's book: "We can still see the gridlines to aid in the judgement of position (Cleveland, 1993b), but they have little visual impact and we can easily "tune" them out. The grey background gives the plot a similar colour (in a typographical sense) to the remainder of the text, ensuring that the graphics fit in with the flow of a text without jumping out with a bright white background. Finally, the grey background creates a continuous field of colour which ensures that the plot is perceived as a single visual entity."
Because it's unlikely any such transformation can be found, may I ask instead why it even matters?
[Wikipedia](http://en.wikipedia.org/wiki/Normal_distribution)
Related thread: http://stats.stackexchange.com/q/8774/930.
Why look at mean forecast error or median forecast error. Neither one is a measure of forecast accuracy.  The mean only estimates bias.  The slope of the regression line being negative means that the estimate decreases going from year one to year two. It says notthing about forecast error.  I downvoted this question because you are either very confused about forecast error or you are explaining your question with incorrect terminology.
@rolando2 Yes, I have. This raises the question of *how much* the pseudo-$R^2$ ought to go up to justify inclusion of variables.  I suspect your "correct classification rate" may refer to the *in-sample* rate, which of course is biased.  If that's correct, then what you read merely compares two inferior statistics.  The *out of sample* rate is far more useful an indicator than the pseudo-$R^2$.
First, thanks for the quick reply. But I didn´t understand what you meant with observation I have the same number of observations for  synchron, control and asynchron and for low, medium and high dispersal, there is only one replicate missing from the start of the experiment for the combination asynchron and low dispersal...
Thanks!  I had myself confused and the slides were great.
Look very carefully at the covariance matrix.  Very, very carefully.  Look at the numerical values... what do they tell you about the relationship between $X$ and $Y$?  If you know about determinants, that will help.
@JVM. Do I read you correctly that your suggestion is my last solution option above (in the final update). That you develop models to forecast each Z (TVC) to use in the actual survival model?
+1, just a note though that due to law of iterated logarithm the growth of unit root process is not exponential.
It would be more common to follow the convention of plotting the values that are fixed (conditional on predictors) on the x-axis and the values that are random on the y-axis. That is, your plot looks the wrong way around to me.
Sample 1 is complete data, but sample 2 is the *exact same* data with random values deleted? I don't see the question or why this is of interest.
@forecaster Thanks!
@Silverfish, a similar concept in geography are dot-density maps. Geographers have found some evidence that regular patterns or patterns that fill up a certain amount of whitespace (so are spaced farther apart then random) tend to produce more accurate perceptions among observers.
"Interested in" and "what can be gained" are rather vague. Could you be more precise about what you want to learn about $x$?
@whuber Thanks for the beautiful explanation. I just finished the problem and saw the answer posted by you. I took the similar approach by drawing it in 2 dimensional plane and looking for regions for range of x and y. I am now able to see that earlier solution was wrong as I missed the fact that the region involves two curves. My answer, with correct approach, is 0.4291 (which is close to yours).
Hi Glen, thanks for the suggestion. I actually came across that post before, when I was looking for bimodial/baysian threads. I was just wondering, since I have only two observations (1/1 hit) and (1/10 hit) I am not sure if I can estimate 5 parameters that way. I guess I have to read up a bit more on the fundamentals on likelihood functions and definitely have a look at hierachical priors - coz I never heard of those before.
Thanks all, the book  Dynamic Linear Models in R, by Petris et al has high S/N ratio.
I agree that the overall volume of conversation on $p$-values is disproportionate to the usefulness of the tool when it comes to estimating statistical models. Regarding the question discussed here, the basic assumption of the test is that the random variable will be normally distributed as $N$ increases, in which case the distribution will be unimodal and the bimodality is only a sample anomaly. The explanation above is good on what to do under that assumption. If that assumption is violated, then neither definition should be considered relevant.
:Benjamin The problem with rhat is you are assuming that the seasonality is deterministic and to top it off that the seasonal coefficients are invariant over time as compared to a seasonal pulse one of the ISI-1 dummies which had no effect for the first k time periods but did so after. Another equally possible seasonal structure is the seasonal ARIMA component which uses an adaptive response to prior seasons as compared to your suggested FIXED response.
I think your last sentence is right on point.  But I don't think we can even begin to model anything statistically without knowing how the opinions are quantified. I also don't see (assuming things are quantified) how you can rule out methods when the question is so vaguely defined. I would think you need to define response quantitatively and then specify some sort of model form before you can achieve the stated goal of the poster.
I agree with Peter.  And... could you show us what you've got so far?
Could you elaborate on what you mean by "returning z-scores" and what "a set mean and deviation" are and how, precisely, they are related to a "standard ranking score"?
Can you suggest a different effect size measure for contingency table from complex surveys?
I don't see a problem here... You have a model and this logistic regression seems to be just a way of interpreting it, that's all. I guess I'm one of those guys who need that better explanation :)
@Marshall Ward, KPSS has a problem if the non-stationary part is only a small part of the all series. If there are new observations coming up all the time, I suggest using `monitor` from the same **strucchange** package.
Of course; there are *lots* of them! What other properties do you want? A simple example: Generate the lower diagonal entries as iid Bernoulli random variables with a small probability of success and "symmetrize".
this looks awesome - the image doesn't seem to load however?
Is this really the kind of question we want on this site? A simple google search, opening any stats book, or checking out wikipedia would instantly provide an answer.
Well, just because the data is made up doesn't mean its not real, right? :) There are many examples with real data among the BUGS examples (for example, http://www.mrc-bsu.cam.ac.uk/bugs/documentation/exampVol1/node3.html) but non as simple as what I imagine you are looking for...
Based on Dan's comment, I guess if you get white noise residuals, none of this matters :)
I guess you should be more specific about what you tried and where exactly the problem starts.
There is also, the [Shapiro-Wilk](http://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test) test
Could you explain how data were collected and what they intend to measure? Are the different columns of your data matrix independent measurements?
I do not see why this problem is not well posed nor why it would be necessary to parameterize the $F_i$. Could you elaborate on your reasons for drawing these conclusions?  It seems another way to put the question is to say that $\\{(S_1(t_i),S_2(t_i))\\}$ is a sample obtained in a two step process wherein (1) points along the line $(y,ry+b)$ are selected arbitrarily in $\\mathbb{R}^2$ and then (2) displaced by iid bivariate Normal vectors of mean zero (and possibly correlation zero, too). That takes the question of identifying the $F_i$ out of the picture.
If it is for your thesis and you already understand it, I recommend you write that text yourself. If you have specific questions though, many people here are happy to help.
Concerning your last paragraph: let $Y$ have distribution $F$, assumed symmetric around $\\mu$, select any real $a\\gt 0$, and define $X=Y$ if $|Y-\\mu|\\le a$ and $X=2\\mu-Y$ otherwise. Then $X$ and $Y$ have the same distribution even though $X$ is a *nonlinear* function of $Y$. This construction has a positive bearing on your penultimate paragraph, too: you can use it to demonstrate why, in general, the correlation will not determine the bivariate distribution even when the marginals are identical.
"Does this imply that Y and X are unrelated?" The short answer is "No"; first because you're measuring only linear relationship in the mean, and secondly, sample statistics have variation, so it doesn't necessarily imply that there's no linear relationship. It may just imply a small sample!
What is wrong with this answer?
I assume the hidden state is $x$ and the emitted variable is $y$... yes, this is ok.
This is a great situation for some type of Bayesian analysis. You can start everybody with a prior probability of being a spammer that is equal to the estimated proportion of spammers in the population. Then update the probability that they are a spammer w/ each new post. This will account for the uncertainty associated w/ small numbers of posts.
"Track[ing] the change in proportions over time ... [to] evaluate effectiveness" sounds like a solid analysis to me, apparently contradicting the assertion in the last paragraph of the question. Although you may wish to make few assumptions (there can be good reasons for this), that does not mean that computing proportions will be OK. Methods that account for both counts--not just their ratio--tend to work much better because they can account for the larger uncertainties when the denominators are small, information that is entirely lost when the proportion is computed.
Thank you Andy - I will have a look at it in the future.
And it's poorly formatted, uncommented, and uses indecipherable variable and function names. Ooooh yeah.
I made some major edits to my initial question, above, about three hours after I initially asked it. I have tried to follow Dr. Harrell's very helpful advice. I would still really appreciate it if someone would attempt to explain the above `cph` output to me in plain English, or point me to a reference that would do so. Dr. Harrell, thanks so much for your help so far!
Thanks for your answer. I will definitely try combining my variables in different ways. But before, I want to know if the poor performances of my model come from technical issues or from somewhere else
Interesting question, but very broad.  The wikipedia article mentions several pieces of empirical research which between them would have used quite a variety of tools, and we can think of plenty of others I am sure.
Thanks for info about Netflix, shame on me that I haven't heard about it earlier :/
Thanks for your answer. Your explanation totally makes sense. If correlations between predictors are zero, PLS = OLS. How did I miss that. :-)
Ain't the OLS assumptions about the errors, and not the data?
@GavinSimpson: I'll try that. But why is it only 100 combinations with 10 predictors? Isn't it something like $2^10$ for 10 binary predictors, more if the predictors have more classes?
Yes, rigorous cross-validation or bootstrapping will tell you how arbitrary the intervals are.  It is not reasonable to expect the world to operate discontinuously however.  Smooth fits are undoubtedly more realistic and will predict better at the margins.  I'm not clear on why tree-based methods are as popular as they are.  With absolutely huge databases the issues are less important.
True. Any non-zero multiple of an eigenvector is still an eigenvector (and even with the SVD, there is still a +/- issue). So what I mean by "distinct" is that two vectors are distinct if they are linearly independent.

Basically, every eigenvalue corresponds to an eigenspace, and the dimension of that eigenspace matches the multiplicity of the eigenvalue. So if the multiplicity is larger than 1, then there are infinitely many non-trivial choices of a basis (i.e. eigenvectors) for that space.
(+1) Would you mind expanding on your opinion on the RMS textbook (of course, it's purely off-topic)?
This question is unanswerable without knowing what kind of analysis are you trying to do. Could you clarify this?
+1 Indeed, animation is a good idea. In R that can be done, for example, via the *animation* package. There's also [Shiny](http://www.rstudio.com/shiny/).
One thing to double check is that they are indeed fitting the same model.  I haven't used the python package you mention but I know some packages don't incorporate an intercept term by default.  R does include an intercept term by default.  I haven't used python's version so I can't say if that is what might be causing differences but it's something to double check.
Thank you Maarten!
@Tal The interpretation of the ellipse is the same as in the `corrgram` package: it shows 95% pairwise confidence region assuming a bivariate normal distribution centered on the mean and scaled by SD(x) and SD(y). I'm not a big fan of this when used in a scatterplot, though. But see Murdoch & Chow, [A graphical display of large correlation matrices](http://www.jstor.org/pss/2684435), Am Stat (1996) 50:178, or Friendly, [Corrgrams: Exploratory displays for correlation matrices](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.1268&rep=rep1&type=pdf), Am Stat (2002) 56:316.
Thank you !!! I would take note of what you wrote. I see predicting customer no of dependents as a dependent variable and taking out the list of predictors as one way.
Surely you mean "after rejecting the null" not "after failing to reject the null"? And it seems there's only one L in 'Marascuilo' (NIST's error, not yours):

Leonard A. Marascuilo. Large-sample multiple comparisons. Psychological Bulletin, 1966; 65(5): 280-290. http://dx.doi.org/10.1037/h0023189.
@Sean: Have you seen my [recent question](http://stats.stackexchange.com/questions/88809/significance-testing-of-cross-validated-classification-accuracy-shuffling-vs-b) about related issues? There is a very interesting (for me) discussion going on in the comments, and I am currently working on some simulations myself. I came to believe that binomial assumption is badly wrong! You might also be interested in several references provided there that claim the same thing.
Can the reason that the 10% are binned be ascribed to a random process or is there some underlying factor influencing the binning?
@AlecosPapadopoulos: determinisitic.
+1 Great question...
@robert1 If you could disclose the purpose of this fit you will increase your chances of getting really effective responses.  Why do you need a parametrization instead of, say, the empirical distribution function (as revealed by the sample quantiles)?  Do you have any theoretical expectations concerning the shapes of the mixture components (or concerning why one might expect this to a be a mixture)?  What might be the consequences of uncertainty in the parameters?  (That uncertainty will be *huge* unless your distribution is sharply multimodal.)
I wonder would it be more appropriate in the case of Likert type data to present the mode answer? Graphical representations can be very meaningful and will capture the type of polarising effect described above.
Can I acheive this interval-censored model by calculating a new variable as the LN(age)? I remember reading about that somewhere...
$\\sum_{k=11}^{20} {20\\choose k}{30\\choose 20-k}/{50\\choose 20}\\approx 7.05\\%$
@chl Great, that you mentioned the Thompson/Walker-paper about "The Dyad as the Unit of Analysis".
Rereading your question, your responses are 'skull dimensions'. That's not binomial. (Binomials are *counts*). It doesn't matter if your predictors are anything in particular.
Magic! The secret is: Type a space at the start of the following paras: "Apart from..." and "There is a R package...".
Please see the last paragraph of my reply ("The main limitation...").  The Z-test is valid assuming the $n_i$ are large (otherwise use a t test) and the estimated standard deviations $SEb_i$ are not too different from each other.  *Neither* approach is best when the standard deviations differ a lot (roughly, more than a ratio of 3:1).
How *big* is your big data?
@whuber quite right - I intended to type "convex in this region" but somehow it didn't end up in the text. I will fix that, thanks.
Make sure to add the self-study tag and mention what you've tried already.
nice proof. Thank you Macro
I think you will find that this method yields a rank-2 update of A' A.
That the residuals from the *good* part of the data are approximately symmetric, other than that you're safe! PS: you might wanna link to a more recent article, there as been gigantic leaps in computational tractability in this filed in the last 10 years (specially in higher dimensions)....
Thank you for the information. To clarify what does $F^{-1}$ stand for? Inverse or $1/F(x)$ ?
Correct. Integration here gives area under the curve, which can be approximated by measuring areas of rectangles.
This is fine. If you'd like, you can see an example in my recent answer here: [CDF and logistic regression](http://stats.stackexchange.com/a/91947/7290).
Skewed distributions _could_ be considered approximately normally distributed if the skewness is negligible and their shape is otherwise approximately normal, but that's empty: in general skewed distributions are _not_ normal. Bell-shaped is not a term that means anything much in hard practice. I could show you distributions that look bell-shaped but in practice differ from normal distributions in important ways.
As a non-user of `R`, I'm unable to get the plots you describe, sorry. But here is one random note anyway. Procrustes rotation is inherently metric, so how can it be expected to agree with nonmetric MDS?
That is exactly how I understand the question. It just doesn't seem to make any sense.
This approach has been called "double dipping".  See my answer below.
@sesqu... there is no missing data, data from all 24 randomly located plots is there.
This is basically correct, if properly interpreted. I worry that some readers might be misled into thinking that simple random samples with replacement (which is the kind of random sample you describe; there are other kinds) correctly reproduce *all* characteristics of the population.  In fact, they rarely do.  The point of random sampling is that the (inevitable) differences between the sample's characteristics and the population's characteristics can be attributed to the random selection process.
It seems unlikely a Gamma would fit daily rainfall records (except by accident for a sufficiently small dataset): there will be too many days in which no rain at all occurred. These *true zeros* have zero probability no matter what the parameters may be.  Could you share with us your reasons for fitting a distribution?  What do you hope to accomplish by that?
If the likelihood is the joint density of $X_i$ and $Y_i$, then you need to know $P(X_i)$ in order to get the joint density.  Whether $P(X_i)$ is meaningful or not is something you need to consider.
I think it's simpler and clearer to work with (mean - median) / SD. Multiplying that by 3 does have historic roots in Pearson's system of distributions. But setting that aside this measure then has the property, intermittently rediscovered and often thought contrary to intuition, that it is bounded by -1 and 1. It is clearly 0 for mean = median.
(+1) I was about to post a response with [Simpson's paradox](http://en.wikipedia.org/wiki/Simpson's_paradox).
Are you insisting on the exponential form for the hazard versus time? Then specify the appropriate value for "dist" in a call to survreg. If you only care about relative hazards, not the specific form of the baseline hazard, then coxph should work OK.
I am only doing this by simulation; with my real `x[n]`, I do not know whether it is stationary or not. Let us say I am given (unknowingly) an fBm with H<0.5, a nonstationary series, then the test will fail...
I find this question confusing and suspect it might be because in statistics and data analysis "sample" has a particular meaning and you might be using it in a different way. Even accounting for that, this question appears ill-defined: it seems you are asking for something (to resample from a set of observations?) as well as its opposite (to sample from a hypothetical distribution). Could you clarify this question by giving some more realistic context and an explanation of what your ultimate objective is?  What is the purpose of this resampling/sampling exercise?
@GaBorgulya: What did you change?
@FrankHarrell thanks for looking at. Please have a look at my current version, I hit few issues including error while executing `penetrance` function
[This also](http://stats.stackexchange.com/questions/305/when-conducting-a-t-test-why-would-one-prefer-to-assume-or-test-for-equal-vari). These links relate to t-tests, but the points carry over. Avoiding the question of hypothesis testing of assumptions for a moment, you rely on the assumption of constant variance when you compute standard errors, p-values or confidence/prediction intervals via the usual normal-theory procedures.
Thank you--but how does it work?  The OP is asking for the *idea* rather than some package.
(1) what's a 'template'?  (2) are you sure you have the correct term in the front of your likelihood? Shouldn't it have a $\\sqrt{\\phi}$? (though this may be due to me simply misunderstanding with what's going on with $x$ in my first question).
I believe this question belongs on SO, although it might be of interest for this community too. The reason is that it doesn't have to do with data analysis or the use of a computer to answer a statistical question, as described in our FAQ.
Also, I did a search for book recommendations, but I'm not sure what to look for. Books on "binary classification"?
If you have both y and x, I see no reason why simple linear regression should not work.
I agree: we can't help unless you are more specific about the code, the data or anything else. Also, RFE always evaluates the largest model even if `sizes` does not include that value. If the full model is best, that is the outcome from RFE.
+1 Good point Harvey.
Not sure if this site is the right platform for such a question. You would perhaps have more luck if you explained what you are trying to achieve and asked for some pointers to standard methods instead of trying to implement a new technique from scratch, an endeavor fraught with difficulty even when you do understand the technique.
@Ian Langmore, you are welcome, I had some fun in answering it. If this answer was useful, you might want to accept it, this is the way this site works. To cite the [faq](http://stats.stackexchange.com/faq) you can accept the answer by clicking on the check box outline to the left of the answer.
Excellent!  Thank you very much.
@SeanOwen, then again, everything is about math.
@Macro The bounty is posted on the linked question at math.SE http://math.stackexchange.com/q/109352/3733
Sorry to resurrect this thread, but I came across this through google. It looks like the question is asking about zeros in his independent variables. In that case, it's not a sample selection problem right?
I disagree, @gung. It's partly about State but it also asks for a method.
Awesome...that will do it. Thanks!!!
If the survey oversampled subjects with respect to covariates that you want in the model, then the model provides proper conditional estimates and you do not need to take sample weights into account.  Sample weights are of interest when estimating population quantities (e.g., population mean family income) where you want to average over a heterogeneous collection of individuals where some subsets of the sample are underrepresented in the sample.