tttjjj/elt_bench huggingface dataset:
Just bought 'An introduction to statistical modeling of extreme values' by Stuart Coles, which looks like it will help. He recommends calculating the CI by the profile likelihood method rather than the delta method as I have been doing, and also notes that the CI is not assessing the quality of the fit - that's what the plots are for. Will post any insights after reading further.
hi eric, the links in the paragraph that starts wih `i've never said it before` from [this post on addhealth](http://www.asdfree.com/search/label/national%20longitudinal%20study%20of%20adolescent%20health%20%28addhealth%29) might be of interest.. :(
Hello, thanks for your edits. I am aware of this but thought I can show the violation of strict exogeneity via the implication of zero correlation if E[e]=0.
This reply overlooks two subtle points, one unimportant but the other possibly very important. The unimportant one is that the double-splitting technique computes the upper and lower *hinges* which can differ slightly from the median, depending on sample sizes. The important one is that the double-splitting appears to be based on a running *estimate* of the median. Any variation between this estimate and the actual median will cause the hinges to vary as well. Intuitively, this should not be a problem as the amount of data grows larger, but it is an issue that needs some analysis.
@Thomas By their reckoning they have 'fixed the model'.. by which they mean that they have fixed the distribution and are only considering different choices of the parameters as different candidate models. I think what I have presented should be consistent with that viewpoint, though they are obviously looking at different model comparison tools. Perhaps you should re-ask the question/edit it to include the source of the discussion? I definitely don't want a check mark just because you 'give up', and that wouldn't help future questioners either
Again, what are your variables - especially your response?  Are they measurements, counts, times, binary variables, Likert-scale items, ... or something else? What is 'candidate' and how is it measured? Why do you want to 'eliminate' it? I have some specific suggestions, but in order to understand whether they're suitable, it's important to have some idea what the circumstances are. I now assume this is a multiple regression (which wasn't quite clear before) - you should explicitly state such things, because I almost gave what would have been useless advice.
I use separate train, validation and test sets. I select hyper-params based on validation set error and then apply selected model to the test set. I doubt there is a bug in training algorithm, because I use off-the-shelf lib.
It would be helpful if you would post the equation you were trying to estimate, a short description of each variable and how you were trying to estimate it.  Posting the data likely isn't necessary, but it is hard to answer without the specification.  Thanks!
AUC is a loss function, it is clear that for this loss function the optimal decision is the classifier with the best AUC.
That sounds like a textbook-style quesiton. Why would you need to show this, unless it's a `self-study` question? (Consider this a request to add [that tag](http://stats.stackexchange.com/tags/self-study/info) unless, for some reason, this really is a problem you have, rather than one related to study in some sense.)
+1 You can just center, without normalizing, and you can center on something other than the mean. This can give the majority of the benefit for interpretation, without the issues of scaling that you point out.
I completely understand your explanation, however, I have no information about the relationship between $X_1$ and $X_2$ besides both being Gamma random variables .
Are you saying that, in some of your models, you have sampling site as both part of the random term and as a fixed effect? I'm not sure that makes much sense, and might conceivably cause problems..
I am sorry to say that it is rubbish, perhaps because of typographical errors in the $\\TeX$. To get you on the right track, please notice (1) an expression like "$\\mathbb{E}[T|W]$" is a *conditional expectation,* not the expectation of the ratio $T/W$; and (2) writing $k$ for $\\sum_i y_i$, it is an algebraic fact that $S^2=k(n-k)/[n(n-1)]$. With these in mind, you might get a lot out of re-reading the [Wikipedia article on Rao-Blackwellization](http://en.wikipedia.org/wiki/Rao%E2%80%93Blackwell_theorem) and applying its steps to your situation as you go along.
@Blain [Polyserial correlation](http://www.john-uebersax.com/stat/tetra.htm) is a good option in this case.
(C) I used $xcorr$ to plot instead as I don't know which command to plot for pacf. I tried with $parcorr()$ but the output did not resemble the original, therefore I am not sure how to go about it.
(+1) No problem.
I'm one of those 38 ;-). The question is, what will somebody whose algebra is fairly weak be most easily able to follow? If $r^2=1/2$, then it is easier to see that $r=\\sqrt{1/2}$.
@Memming Yes, I'm using a bag-of-words representation. What do you exactly mean by similarity measure? I'm investigating whether a webpage contains some specific kind of information or not.
Dear Sir, thank you very much for your answer. The whole thing became much clearer now.
+1 for the reference to that paper!
@Matt great question. I specifically avoided imposing any correlation, but you are right we should impose some correlation structure. However, OPs post appears to be more basic, hence I ignored the correlation structure.
There has to be some separation between alternatives, otherwise the MNL cannot be identified. Do you have other variables that will help to distinguish between `Bus` and `Walk`? Also, are you putting the variable into the formula as an alternative-specific variable? `mlogit(choice ~ generic | alt.spec., data = MyData, ...)`
You have just described the Mann-Whitney U test!  Also, you use "every value in the other group [as] a potential counterpart for comparison" and compute the median of all such paired differences you obtain the Hodges-Lehman estimate of the difference in medians of the two datasets: http://en.wikipedia.org/wiki/Hodges%E2%80%93Lehmann_estimate
A data set with no *global* correlations (i.e. $Cov(X,Y)$) may still contain clusters, and may even contain correlated subsets (which is the point of doing correlation clustering, finding subsets that have *different* correlations).
when I say they are misclassified, I mean that when I test different alternative models on the training set, all those samples are misclassified by all or most of those models
$\\mu$ is a mean *vector*, and $\\sigma$ (often written as $\\Sigma$) is the covariance matrix, which on the diagonal has the regular variances, obviously.
Good point. I would ask for clarification on that point. If he's talking about the so-called class of "robust" and/or "trimmed" estimators, then these are estimators which discard, say, 5% of positive/negative tails. So, in the case of *model misspecification* (e.g. non-central Cauchy location estimation), they make much better estimates of *location* than sample averages. Otherwise, I don't think there's much credibility to that statement.
1) What is your data? p.det?  If so, why do you even need to include y[j] in the code?  2) You appear to just be estimating $a$ and $b$, then generating a random number from a $\\beta(a,b)$ distribution, and multiplying that by a random number drawn from a $\\beta(0.3,0.3)$ distribution... is this right?
Your code has a bug: it prints the last random uniform value `p` in place of the desired value `t`.
Actually, averaging correlations is done in many statistical contexts, e.g. for the calculation of [Cronbach's alpha](http://en.wikipedia.org/wiki/Cronbach%27s_alpha). I cannot really understand your argument ... Of course common sense is needed for this averaging. If you average coefficients from different variables or from very different samples, the resulting mean may not make any sense at all. But, IMO, this rather a question of content and not of statistical procedures.
Here is a related question and answers on CV. http://stats.stackexchange.com/questions/14697/are-fisher-linear-discriminant-and-logistic-regression-classifier-related
have you checked this [paper?](http://www.jstor.org/stable/3211805)
@J Li, yes, although it is the coarseness of the resolution with respect to the values of Y, not just the general coarseness.  The finite case (where all X are categorical) is one scenario.  In most real world scenarios arbitrary resolution of X is unreasonable... e.g.  If  x1 = 1.00031 => y1 = 0   x2 = 1.000309 => y1 = 1 x3 = 1.000311 =>  y2 = 0 etc.  Another way to say this is that in most cases, "reasonable" discretization should be possible without introducing a large loss of information.
thanks for your response. My question was about dimensionality reduction and forecasting directly to handle data that was collected every 15 minutes as opposed  to a top down approach like the one you are suggestion. I'm sure you are aware that there is a tremendous ongoing research to use advanced techniques to do forecasting by minute or some instance by second for which dimensional reduction was recommended.
@Rajhans I don't know of any work on converting (2) to (1) analytically. However, you can do this simply by minimizing the KL divergence between the two distributions. Running a program that does this is easy. You generate a bunch of samples from (2), and you use that to train (1) with maximum likelihood.
Thanks. I don't have easy access to the full-text, but from the opening page it does look useful regarding point 1 (i.e., combining two correlations).
At present this question is too vague to be answerable. Can you say more about your situation, data, and goals? Are all models fitted to the same dataset? Can some be seen as nested within others? Reading our [FAQ](http://stats.stackexchange.com/faq) will help w/ formulating productive questions, among other relevant info about CV. You may also find this blog post helpful: [How to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/comment-page-1/).
I think you misunderstood my answer. This approach will produce prediction for any value, but the outliers will not be predicted very well: this is because they are not permitted to influence the PCA fit. I advise you to read the paper.
Does that help? [Predicted by residual plot in R](http://stats.stackexchange.com/q/18234/930)
@whuber: That is my second paragraph: if the forecaster predicts rain correctly 5 rainy days a year and predicts rain incorrectly 36.5 dry days a year then only $\\frac{5}{5+36.5} \\approx 12.05\\%$ of rain forecasts are in fact rainy days. Since the error rate is higher than the rainy rate, $100\\%$ of rain forecasts being correct is impossible.
@whuber I have actually used PCA (well SVD) to find these rotations in doing stereo image calibration! It definitely is the same sense of rotate.
I wanted to modify that code long time ago, but lacked time for that, adding weights shoudnt be conceptually difficult.
How does it orient (give direction) to the arcs? What test or scoring approach is used?
Thanks a lot to all hints! The ```as.data.frame``` command was the one I was largely missing. Being aware of it now is *cool* :)
Alternatively, if you're not familiar with that property of MLEs, you should define $\\theta=e^{-\\lambda}$, then rewrite the likelihood in terms of $\\theta$ instead of $\\lambda$, then maximize the likelihood with respect to $\\theta$.
What is A/B test?
I'm not clear on your question. What is it you are trying to do? Is your strategy [`for what?`] sound?
Are you familiar with Bayesian inference?
+1: This is a really well formulated question.
Unfortunately, a closer closed my topic. So I didn't get my answer.
$\\hat \\beta_0 = \\bar y - \\hat \\beta_1 \\bar x$. **QED**. :-)
Does this help or just more of what you already know? [UChicago course notes](http://home.uchicago.edu/~amshaikh/webfiles/ident.pdf)
A short answer would be NO. But more detail of the implementation is needed in order to debug your problem.
Hm, I take back my statement about good documentation. [This](http://www.omegahat.org/RSXML/gettingStarted.html) looks very similar to what you want to achieve.
"Due to a ceiling effect, my data are non-normally distributed for both groups." Non-normality is not the issue here. It's the ceiling that you described. With or without bootstrap, those observations are interpreted as achieving the upper limit of detection rather than exceeding it to an unknown extent. Bootstrapping is not a solution to this problem that I'm aware of. If normality would be a reasonable parametric assumption otherwise, the EM algorithm is the correct approach.
@mbq, probably because I created this account long ago when I was 0 on OS as well...pity, 100 on CW looks like a big deal, since you must be a real chap to answer stuff in here :)
In what sense would someone 'prove' a measure of something? It would be like trying to 'prove' the mean. It's not clear what that could mean. Please clarify. Also note that there's more than one measure of these notions. Are you after the measures based on third and fourth moments?
@IrishStat do you know if there's any packet on R that allows mw to do something similar? And if I have, for instance, 400 series in one cluster, with 160 observations each, do you think this is to heavy to handle? Also, when fitting the forecast model, which frequency should I use? My observations are weekly data, therefore I have a frequency of 365.25/7, of 3 years and 4 weeks. Should exclude the 4 weeks?
$X_i$ is also assumed independent of $Y_i$, yes?
@dikran i think SVM is a great classifier because it is sparse and robust to outliers -- this is not true for Logistic Regression! and thats why SVM is state-of-the-art classifier. Only issue which may be a problem is -- time complexity -- but i think its ok.
Any tree procedure will give you the probability of being in a class at each node. If you want a tree method that is based on significance and such, you might look at `party` in R.
Thanks for your response. I did start with a scatterplot matrix (see comment above). As mentioned, my variables are not contineious. The variables are basically factors? I would really like to show something like: when X1 was 1000, the frequency of observing X3 as 1 was '44' etc. Hope that makes sense.
The two votes tells me this is at least interesting to some, so I'm going to bump.  So as to not leave a completely vapid comment, let me add that as expected, adding white noise to the spectrum increased the size of the CI's. Example:  For one parameter, the 95% CI went from (222.00,222.22) to (220.11,224.16). I have strong reason to believe from other data that the true value of this parameter is around 220.
Can those voting to close please tell the OP how to improve their question.
I did not see 3) in your question. From 1) and 2) I am saying 1) is correct and 2) is incorrect.  The question suggests that an answer can be correct and the other answer can be incorrect.  So i don't see your issue.  Scenario 1) could happen and explain the result you were told about the conditional probability but scenario 2) contradicts what you were told and hence has to be incorrect.
@coffeinejunky: thank you for your answer, but let me note that I have no idea on what the dependence of $Y$ on the $X$'s looks like. Do you have a suggestion on how should I proceed in this situation?
I'd have added 'Gini' as a tag but don't have the rep.
(1) From what data have you estimated $\\beta_1\\mu+\\beta_0$?  (2) Isn't this question really asking how to estimate $\\mu$ from iid observations from a Skellam distribution?  (Implicitly, it assumes that a method of moments estimator $\\hat{\\mu}=s^2/2$ ($s^2$ is the sample variance) would be good; but perhaps there are other approaches.)
This need more detail. What sort of distribution is it (for instance, what does a histogram look like)? Are you just trying to estimate the average or other characteristics as well? Most importantly, what information do you have about Maryland and Pennsylvania to base such an estimate on?
I'm not sure the first parenthetical remark, which I read as "don't tell me to do X because I won't", is helping here.  Perhaps reconsider?
+1. @Alan, Runs test is also called Wald–Wolfowitz test - for you to know.
I'm curious: what is the trick?
Good points, I'd just add 4) In FA proper (as opposed to PCA) factor scores are anyway appoximate, or "best guess", because case-wise uniqness or communality values are not known. We usually "approximate" the scores by OLS regression; but we might have used other approximation approach instead, ... and one might be simple averaging/summation, why not?
Note that for a perceptron, the learning rate is not that important. If the data is linearly separable, any reasonable learning rate would lead to convergence. So a lot of design works. I think $\\|x^{(i)}\\|$ is also a legitimate choice. However, note that $x^{(i)}$ occurs twice in the update equation, once as a vector, once in the hinge loss in the numerator. So it makes sense to normalize by $\\|x^{(i)}\\|^2$
A guess, assuming you're minimizing NLL: the surface has flat troughs. Did you normalize your data?
Ouaouh, this is again an JIT answer :)
As far as the bootstrap approach, it is an extension of selection with replacement to get a distribution for a test statistic.  In the one sample approach, I would just select from model 1 estimates and calculate a summary stat for each draw. Since I am interested in the difference between the summary stats, and the data is paired, I would select runs at random.  I would then summarize the data separately for each model among the selected runs.  I would then calculate the ratio or difference between these two summarized samples.  No shuffling of the values between the two models would occur.
Thank you. That was exactly what I was looking for :-)
@Siga If you think this response was helpful, please upvote it (I did).
@mark999, OMG, thanks! :) Fixed
True, we don't have a vested interest and we may want to be funny (which may appear we are mean). Unless you want to cut us in a deal.
Oh my question really isn't limited to MATLAB (and that isn't really MATLAB code, it's just some pseudo-code based on MATLABs syntax for for-loops and comments that wouldn't work anyway). But I can tag it just in case.
Thank you all very much for your answers. I'm taking my time to test each of the tools you mention on my data. I will pick an answer after my tests. I guess it's too bad I can't pick more then one :)
It's a polynomial regression model of degree two (or sometimes, a 'quadratic model') in two predictors, ($M$ and $\\ln P$).  You estimate the coefficients the same way as for any other multiple regression.
Logistic distribution, or logistic *function*? In either case, this answer begs the question, because we still do not know why the logistic function/regression takes on the moniker "logistic."
OK, but I think what you're really saying is look at the two lower order interactions to see how they change.  Individual ANOVAs of the two interactions won't tell you they're different from each other, that's what the 3-way interaction is for.  For example, if you found that the two way was significant at one level of C and not at the other you've got no new information, just new tests.  To clarify, only the 3-way tells you that difference in significance is different, and you already have that.  You really just have to look at them.
do you mean (std error)^-2?
This is really helpful. Thank you! For those interested:

R Package which includes Jaccard similarity measure: http://cran.r-project.org/web/packages/vegan/index.html
@Zach I think so; also polynomial one should be OK
I don't completely understand.  Are the "outcomes" finding something? What are the "rare items"?
As using messy data: Just take world bank data for all countries by year. Then add UNO or whatever institutions Data to itt. Lots off fun: Different naming conventions for countries, missing data correlated to countries and lots of non normal distributed data.
That was my master's thesis - have fun ;)
Ah I got what you mean by permutation test. I don't think it is quite the same meaning of permutation tests of the papers you cited. Try to have a look here http://researchcommons.waikato.ac.nz/handle/10289/1506
"Long-range autocorrelation" does not necessarily imply non-stationarity. Is therefore your question about two stationary, or two non-stationary, autocorrelated processes?
It would be helpful if you describe your variables, what do they mean, and also name the output table in which you find one-tailed tests instead of two-tailed what you expected.
Thank you! The approach described in homals was novel to me, but definitely interesting.
DecTime is now added to the data.frame.
First you will need to use the [correct definition of entropy!](http://en.wikipedia.org/wiki/Differential_entropy#Definition)
Can i somehow improve the question? ie. narrow it down further so that people are more likely to help?
To get useful, relevant answers, it will help to specify two additional things: (1) anything you are willing to assume about the distribution of $X$ and (2) the reason for generating the samples, so that we will share your understanding of what makes a good approximation to the distribution.  For example, a good approximation for making decisions about extreme values of the distribution may be very different than a good approximation for making decisions about average values.
@Hong Ooi Your comment confounds a *conceptual* definition, useful in the modeling process, with a matter of *computation.* They are not equivalent. For instance, when we model the sampling distribution of a mean with a Normal distribution, it is irrelevant that the 32-bit or 64-bit numeric representations we compute with are finite in number. The Normal distribution remains continuous despite that. More relevant here is "why do we even care?" In logistic regression the dependent variable *must* be binary and how we choose to characterize the IVs is determined by their *meanings.*
@whuber I don't think there's any issue with an equal variance assumption (which we can at least compare the sample values for rough proximity as a reasonableness check). If we take a Welch-Satterthwaite approach instead, there's an issue, as you suggest.
Trees always discretize continuous predictors, so I think the answer is the same as in the other threads. There even is a thread where @frankharrel weighs in why the inherent discretization in trees is suboptimal.
Because Model 2 is not really considered "nested" from Model 1, I don't know how valid using QIC to assess comparative fit would be. One thought I had was to use multiple imputation techniques to equalize the number of observations, and then one could arguably compare the QIC values for those models. However, some literature, e.g. "Applied Longitudinal Data Analysis for Epidemiology" by Twisk, showed really inconsistent results by using MI techniques on models that have dichotomous response variables. I wish I could help more.
@Xi'an, I've been wrong before myself, but I don't understand why you think only bootstrapping residuals is valid.  Efron & Tibshirani (1994) section 9.5 says "Bootstrapping pairs is less sensitive to assumptions than bootstrapping residuals.  The standard error obtained by bootstrapping pairs gives reasonable answers even if [the probability structure of the linear model] is completely wrong."  The implication is that application of the bootstrap is more robust, although they imply it can be less efficient in some cases.
But I just want to decide the optimal dimension k via PCA. Is it more convenient to do it once?
Thank you very much for this simulation. It's more than I could expect.