tttjjj/elt_bench huggingface dataset:
This looks like a routine textbook type question and so should carry the `self-study` [tag](http://stats.stackexchange.com/tags/self-study/info). Note that "mound shaped and symmetric" doesn't imply normal unless 'mound shaped' is given a very precise mathematical definition (and if you're being taught different you need to consider the competence of the teaching you're getting). Different 'mound shaped, symmetric' distributions can give quite different answers to those questions.
@Zen It's by no means as elegant as yours. I offered it because I have found myself over the years thinking in this way when confronted with much more complicated distributional inequalities: I ask how the probability can be shifted around in order to make the inequality more extreme.  As an intuitive heuristic it's useful. By using approaches like the one laid out here, I suspect a general theory for proving a large class of such inequalities could be derived, with a kind of hybrid flavor of the Calculus of Variations and (finite dimensional) Lagrange multiplier techniques.
I read the comments, and my impression that they all refer to the distributional relations between random variables. We usually say "identical random variables" but we only mean "identically distributed" not "identical as functions".
You've definitely added to my question and also helped to refine it  - I was also mostly thinking about this in terms of individuals/choices/personality traits/etc. but I don't think that it stops there.  I suppose I'm looking for some sort of theorem or principle that attempts to explain this elegantly. Michael Mayer did have a good response though.
Fascinating questions, but the first I suggest is off-topic for this site.  Generally, 'Are A and B (empirically) related?' will be off-topic, although questions along the lines of 'How could this data be analysed to test whether A and B are related?' would be on-topic.  The second question potentially has a statistical aspect, concerning the effect of forecasting accuracy, via assumptions about how people or firms adjust their behaviour to their forecasts, on expected profitability.  It might be best to re-word it to highlight the statistical aspect so that it is clearly on-topic.
Where are you getting the idea about "gross sampling error"? What is "MSE"?
@drknexus Yes, of course (in my initial tests, I used standardized N(0;1) variates, instead of yours).
one vote for the glassdoor style
Can you add more details about your situation?
"..at large scale, the performance of different algorithms converge..." I did not know this. Thanks for this helpful insight. Also, I did stumble upon "Mining of Massive Datasets" and found it very useful. Will look at the other book too.
The Gamma distribution is conjugate to the Poisson distribution. Perhaps this is what was meant?
Welcome to the site, @Enas. I tried to edit your question to make it clearer. However, I still do not quite follow your question. For example, we do not "select data to fit the model"; did you mean how to select *variables*? See if you can clarify your question further so that people will be able to help you better.
(continuing from above): But to find outliers, I think it would be just wrong to divide by $n$ because that way I would fail to detect more and more outliers, just by growing my time series. It's the same with spam filters: no email provider is strengthening the spam criteria for people with more email only to keep the number of false positives from increasing. When you have more mail, you have more false positives. ...
It sounds like the `Malmig` package is not available on official CRAN package releases, so you're better looking for a source version (that has to be compiled) or another mirror.
@AndyW No, only the math-mode subset is supported (The engine is called MathJax, see [its page](http://www.mathjax.org/) for detailed docs). Yet I think a proper way to post tables is plain text in code block, so it could be copyable.
I think now that the first probability is wrong because it does not take into account the relative sample size of both classes.
@user3013423:  Probably a good idea to ask it in a separate question.
I have watched all of those videos. It's a very good introduction to probability and counting.
I am confused by your statement that the plot is "not a histogram," because to all appearances it is one. Each bar spans an interval $(x_i,x_{i+1}]$ and, by virtue of its height $y_i$, it represents a probability $p_i=y_i(x_{i+1}-x_i)$. If you want to represent the same information on a reciprocal axis you will therefore draw a bar spanning the interval $[1/x_{i+1},1/x_i)$. Necessarily its height $y_i^\\prime$ will be determined by its area (which must remain unchanged), whence $y_i^\\prime=p_i/(1/x_i-1/x_{i+1})$ = $x_ix_{i+1}p_i/(x_{i+1}-x_i)$ = $x_ix_{i+1}y_i$. What, then, is the problem?
Thank you for this detailed answer, this is exactly what I needed!
So you get a (random?!) sample of orders and for each order in the sample you get to see, e.g., a customer id?
Ok, it makes more sense now. Thanks!
See http://en.wikipedia.org/wiki/Probability_density_function#Dependent_variables_and_change_of_variables
Typically one uses a fixed scale only with models like logistic regression or Poisson regression, where the response is a count or indicator/frequency variable. In this case there is no analogue to the scale parameter in normal regression.
"*Every time whey you are trying to twist or rotate the decision curve, one or more support vectors are defined and included.*" may be a little misleading. In the linear kernel, for example, the shape of the decision boundary is always a hyperplane in input space, regardless of the amount of support vectors.
@Andrew, you cannot compute the probability sought from the prior and the Gumbel alone. You need to specify the alternative distribution as well for this computation. You also need the density and not distribution function. I have edited my answer to reflect this question.
"Collinear" isn't quite the same as "correleted". So your question remains unclear
Thanks a lot for your effort and time for producing a really comprehensive answer. Really appreciate it.
Yeah 2) does! It is difficult to phrase questions in a simple way without writing an essay I realized. The problem with 1) is that it is in $R^3$ which takes forever to cover by sampling from $f$. My idea was that using the set of points as an estimate of more probable points in $R^3$ and using $f$ to approximate a discrete distribution over the set I can speed up the MAP inference I am trying to do. But bootstrapping seems promising. I had forgotten about that. Thanks!
I should clarify that this is all hypothetical data. The real problem is similar but much more complicated. Say I have a sample of people from all different areas, but not data on every person. So by just computing the 99th percentile. I wouldn't have the actual value because I only used a sample of the population to compute the value, but perhaps I am missing something.
It's been published: http://dx.doi.org/10.1093/biomet/asq017.
Thanks very much. I found both answers helpful :)
@Bruce. By curvature, do you mean shape?
Do you know that $E[(a-bX)^2] = a^2 -2abE[X]+b^2E[X^2] = a^2 -2ab\\mu + b^2(\\sigma^2+\\mu^2)$? Can you apply it to your problem?
@Glen_b, your struggles here are admirable, but I suspect this isn't worth any more of your time. I voted to close as not a real question.
Hi @Aarkham. I've added a few details. Let me know if there is anything I can clarify. Cheers.
Ah yes, I forgot to account for deaths that occur on the same day as the birth. I'm assuming this was excluded from the data as the hospital would have to submit this data to the state for a birth certificate and they wouldn't submit data on babies who die I'm assuming. This has lead me to a series of awkward Google searches... "do dead babies get a social security number"... oh search history.
+1 for noting the nonlinearity.  But your use of the phase "expected proportion" seems to assume the data are a random sample of the distribution over which one is averaging and that's often not the case.
Term frequencies are not category labels. Your clustering algorithms is placing your data objects (documents?) into several categories. The labels you would need are mappings from those objects (documents) to the correct categories. You could then compare your clustering results to these correct categories. Without this labeled training data, you should consider a different approach entirely.
@whuber Thank you for looking through that paper. Could you please clarify your last statement? I'm unsure of how I can deduce the joint confidence regions from that for my sample data. Finding the MLE is easy enough but the CI is puzzling me? I'd really appreciate your input on this
This would be slightly disturbing, but also probably quite inaccurate. You need a signal from the phone to at least three towers to get even an approximate estimate of location, and with the blocking and reflection of radio waves caused by big buildings in cities, I'd imagine that those estimates would be pretty loose. If it's in the center of a city, with lots of other people around, this would be pretty useless. That said, the number of phones with GPS in them that phone home without notification is probably getting quite large by now...
@Glen_b I have not thought of a way how. I am starting to think that the short answer to this question is "no".
many thanks! in my case the predictors are not that awfully right tailed. but good to know that rms exists. thanks for sharing your approach.
I think this should be left open; it's pretty clear what the person is *asking*; it's not clear *why* he or she wants to know.
@Aman that does not work for displaying similarity between >2 documents on a 2D plane (graph). sure, i can plot points A and B with a separation based on k-means distance. but then when i need to plot point C, based on distances to A and B, typically there is no point in 2D space that satisfies all pairwise relationships.
+1 For this resource. As the name says, excellent for a practical approach to engineering problems.
(+1) The `cut2()` function from [Hmisc](http://cran.r-project.org/web/packages/Hmisc/index.html) is a very handy replacement to base `cut()`.
@user4701 Yes, still wrong: the point is that one subject not only sees one single stimulus, but EACH one of the stimuli. A single subject is observed in all levels of the `stimulus` factor: hence you have repeated measures with regards to the `stimulus` factor, and need to use `aov()` with an `Error()` term.
You need to define the class of distributions of interest -- maximum entropy among the distributions that are absolutely continuous on the real line? with support on a finite range? with support on the finite number of points (as your solution goes)?
(+1) This is a well formulated question. However, it now looks partly duplicated from your [preceding question](http://stats.stackexchange.com/questions/4009/inter-rater-reliability-using-intra-class-correlation-with-ratings-for-multiple-o) (that you've updated recently), though it's another question. Would it be possible to balance the content of your two questions, with one target question in each? If you always update your question (instead of asking new ones), it becomes difficult to provide acceptable response.
Is that 25 parameters in the model, or do some variables have more than 1 parameter? I had to google to find out that "lakh" is 100000, but that doesn't sound like too many variables. However, what proportion had the event?
The terminology is all standard from any regression text, with the exception of `Residual standard error` which is the conditional variance.  Never did figure out why.
+1 To be contrary or irksome, one might argue that in some sense the last three are all "equally" non-linear, but the contrast with the first one (which is a classic linear scatterplot) speaks volumes.
The hurdle model will only have one observation for the C treatment in the count part - so that won't work. In theory you should still be able to fit a ZIP model, but it will likely encounter similar problems in estimation. How many sites do you have? I suspect the underdispersion is an artifact of over-fitting the model (with too few observations in each site) given how many observations you are saying you have.
i edited the question, thanks.
To get to the data format used here... rep(1:5, c(0,1,1,9,4)).
+1 Thank you, Tim. I appreciate your suggestions regarding latent class models and LIMDEP. These approaches are growing in appeal to me as I start to learn more about them.
If one of the w's is negative and your normalization scheme preserves signs, then surely after normalization one of them would exceed one. You may want to describe the background for us to better help you.
It's really unclear what you're asking, so much so that I'm at a loss what main edits to suggest. I'm not even clear that this is a statistical problem at all.
Because it seems to me that this would work only if there is the same amount of observations for each predictor level. Should I then weigh this mean effect by the number of observations? How about if I have also covariates? would that change something? I don't feel very comfortable doing this 'by hand'...
It's a typo. The Edgeworth expansion approximates the cumulative distribution function of the standardized sum $Y_n=n^{-1/2}\\sum_{i=1}^n(X_i-\\mu)/\\sigma$ and _not_ the density function.
Log return has different meaning than ln(return). The same goes for ln(first difference). Returns and first differences can be negative, taking logs of them has no meaning.
(Copied & pasted from above) "Eg, do you want to know the arithmetic mean of your population +/- some MoE, the proportion of your population w/ some attribute +/-, the SD of your population, etc?" There are potentially innumerable types of analyses. Moreover, there is no way to know how much data you'll need to do what you want to do if you don't know what you want to do.
Out of curiosity, why would you bid for emails and who buys them? I still don't exactly get how it works. If you win an email and can't sell it you don't have to pay your bid? Do you have to take it if you have to sell it for less than your bid? Please describe the process a little more.
Assuming there are no independant regressors in the fitted model, non constant variance is really only a problem when the variance of the error term is time-dependant. Then: arma+[garch](http://en.wikipedia.org/wiki/Autoregressive_conditional_heteroskedasticity)
@Casebash have you tried it? It's not that easy to get *meaningful* and *unbiased* results. You can compute some number, but it may be biased e.g. towards or against results that have a deep/shallow hierarchy etc.
In conversation with Findley and Parzen (http://projecteuclid.org/download/pdf_1/euclid.ss/1177010133) Akaike revealed that AIC was used by an assistant in her FORTRAN program. Variable names such as IC by default implied integer quantities; a prefix such as A was sufficient to instruct the compiler that the quantity was real. Although not intending "Akaike" he realised that it also meant simply "an". (Incidentally, while this reference is, as it were, an antidote to one incorrect story, it perpetuates the misspelling of Mallows' as Mallow's.)
Just to add to this answer -- you should model these complicated variables explicitly as an ordinal and as a Poisson within the context of the generalized latent variable models that allow GLM-type links.
Further, why do you want to *test* normality?
The lack of definition seems to make this question impossible to follow or to understand.  Please tell us what $f$ and $g$ represent and how they relate to your data. What does it mean to have "sampled $X$ data"? How does that differ from having "samples from $f$"?  What, if any, are the relationships between $f$ and $g$ and "the same distribution" for $X$? Where you write about "regress the differences," what would the differences be of and against what would they be regressed?
I apologize. I think I read your initial comment incorrectly. The standard F Test for choosing between two models, one of which is nested in the other, is valid. It's no different because you aren't including an intercept.
Welcome to the site, Ansel. Since this is a question on a computing environment, should we move it to SO?
http://jmlr.csail.mit.edu/proceedings/papers/v7/miller09/miller09.pdf This might be helpful. More details on the maths here: http://www.ms.unimelb.edu.au/~millerh/papers/gencor_JCGS.pdf
+1. But I would have answered "Yes" in the sense that an intercept-only model is still a case of multiple regression. Moreover, including the other IVs will still yield the same result.
Can you post some sample data?
> I will not post this as an answer because there are many people more knowledgeable in these things than me. However, i  think that concerning speed you have to distinguish between R base and the mix good programmer/great packages. A good programmer can leverage some the tools in cran such as multicore, GPUtools&magma (soon but not yet usefull for MCMC), Rcpp,... to make a pretty fast code. I don't think matlab central has anything comparable to offer.
@Roman Luštrik, I added some sample code in Matlab for equi-depth histograms (I'm not a regular user of R), hope this will suffice!
For reference, I created a [distance correlation version](https://en.wikipedia.org/wiki/Distance_correlation) of the [correlation graph on wikipedia](https://en.wikipedia.org/wiki/Correlation)
Using numerical integration, $\\mu \\approx 4.91912496$.
I thought of eliminating bcz the value is missing for ALL records except very few (like 20 records of 1,785,000 records). Also the validation data (9500 records) has totally missing data for that variable. It's loke I have this column with no information at all so I just can't understand how its presence will be valuable for training process!    
For Q2, [1 refers to music preference, 3 to reading preference, 5 to activity preference].   
The goal is to predict if the user will response to email by click(Y) or notClick(N). No missing data for DV. Only nearly 1% in training set response by Y.
That is a nice suggestion, thank you.
@ peter, thank you for your valuable inputs.
And if the distributions of p-values does not follow anything (it is not uniform nor with many p-values between 0 and 0.05), what does it mean?
The success of PCA is how far it produces PCs that are interesting or useful. Provided you are alive to the possibility of circularity, as you are, there is no absolute barrier to using those PCs in another analysis, and many people do this. I've a preference personally for using PCA to illuminate which of several original variables should be carried forward, an approach occasionally automated as principal variables analysis. e.g. Cumming, J.A. and Wooff, D.A. 2007. Dimension reduction via principal variables. Computational Statistics & Data Analysis 52: 550-565.
Do you want to do this because your data is nonstationary?
+1 Great! Is there an elaboration somewhere on what `pi`s are considered reasonably small enough?
If you use stepwise regression: 1) Your parameter estimates will be biased away from 0 2) Your standard errors will be too small 3) Your p-values will be too small. See e.g. (this thread)[http://stats.stackexchange.com/questions/69452/how-does-stepwise-regression-work] for more.
So I guess, there is some "optimal value" of "smoothing" window when correlation is not affected by high frequency noise but diffed series are still close to stationary. Do you know any proper way to determine this?

Or, maybe there is another way to quantify the relation in above series?
AUC is not the gold standard for comparing models.  Study log-likelihood, AIC, generalized $R^2$, quadratic and logarithmic scoring rules, etc.
@ttnphns: I assume it is confirmatory factor analysis. But I may be wrong.
That weighting looks arbitrary.  What is the basis for using it?
@Richard It might feel like that only because, for the sake of exposition, I have oversimplified. In reality one would model the population with a distribution of $p_i$ having a mean of $1/2$. Unless the variance of that distribution is zero, the same analysis implies the same conclusions, including that the expected girl:boy ratio will be strictly less than $1$. This shows that the popular conclusion (that the ratio must be 1:1) depends crucially on the no-variation assumption. I won't apologize for using mathematics to reason about this, which does not diminish the interest of the result.
The reason why I wanted to do it like this was to have an open mind towards understanding my data, so maybe in a way I am fishing for correlations, which I did not think of before, for the purpose of getting enlightened. I am certainly not doing this to satisfy my boss or something abitrary. I would rather not get into the specifics of the data, as I want a general answer to this question, so I can use it in all situations in the future.
Okay, I'll look into that.  Thank you.
@COOLSerdash If you treat the pair of scores as bivariate normal, could you work out $P(\\text{both players score} < 100.5)$?
This belongs on StackOverflow since it's about programming and not statistics. Also, you should use the code highlighting syntax to make this post more readable.
Is there any way of knowing which classification is correct (a so-called 'gold standard')? Are you wanting to know if the new tool is an *improvement*, or only if the 2 procedures *agree*?
Thank you Brian, I will post my question there.
How are you estimating these parameters? If you are using maximum likelihood, then you can just transform the original estimator and this is justified by the Invariance Principle. In a Bayesian framework this is also reasonable.
In fact, I might as well transcribe the proof here: 

"Let $A_i$ be the event that the $i$-th [coupon] type does not appear among the first $n \\log n + cn$ coupons drawn. Observe first that
$P(\\tau >n\\log n+cn )=P (\\cup_{i} A_i ) \\leq \\sum_i P(A_i)$. Since each trial has probability $1 − n^{−1}$ of not drawing coupon $i$ and the trials are independent, the right-hand side above is bounded above by $\\sum_i (1 - 1/n)^{n \\log n + cn} \\leq n \\exp(\\frac{n \\log n + cn}{n} ) = e^{-c}$,
proving (2.7)."
The most critical aspect seems to be the errors in $x$ here. If, as you suggest, the errors in $y$ are comparatively very small you could fit the model $\\text{E}(x) = (y/a)^{1/b} = \\alpha y^\\beta$ (where $\\alpha = a^{-1/b}$ and $\\beta = 1/b$) - that is, simply fit a power law with the roles of x and y swapped, and then back out parameter estimates (you should carefully consider your model for the variance of $x$ though; constant variance may not be suitable on any of the scales mentioned). If both variables have substantive error, you may need to consider errors-in-variables type models.
For an example of a plot of predicted probabilities when there is an interaction, it may help to see my answer here: [Graphing a probability curve for a logit model with multiple predictors](http://stats.stackexchange.com/questions/31597//31600#31600).
Incidentally, the Wikipedia reference to that approximation is broken.  Mathematica finds, though, that the relative error (1 - approx(x)/erf(x)) behaves like the reciprocal of $2 \\exp(x^2+ 3(\\pi-4)^2/(8(\\pi-3)))$.
If the answers sought require specific substantive knowledge of poverty studies, this would be better placed in some other forum, although I can't advise which. If this is asking about missing data, look for many questions on this site, as the specific details here obscure your statistical question.
You have asked 4 prior questions without accepting a single answer. Have none of the answers provided been of value for you? You might get more responses here if you accepted some of the answers you gotten already by clicking on the check mark beside one of them per question (if any were helpful, of course).
what is your model? can you describe it using [standard notation](http://en.wikipedia.org/wiki/Multilevel_model)?