tttjjj/elt_bench huggingface dataset:
My opinion is that this book is THE book about extrem value theory
Thanks for the answer. Indeed I have too spotted that something is wrong with the numerical results -after all, the expected value should increase in absolute size, rather than decrease, as $n$ increases. I left the answer as is, to see whether I could gain some insight from any answer. I am still searching at the theoretical level where exactly is the mistake, the suspect being the first equation I use from Owen (because the second has been verified by other sources)...by the way, could you check whether this eq $4$ in my post (as a stand-alone transformation) is correct? I would be grateful.
Thank you for pointing that out.  There seems to be an "idiosyncrasy" in this site's parsing of urls: it mistakenly included the final period. By inserting a space before the period I found that the link now works.
@Glen_b It looks like something like the Bhattacharyya coefficient gives the proportion of overlap with not too much difficulty in 1-d.  I am wondering 1) how to compute this for bivariate gaussians and/or 2) some other characterization of the space other than a proportion.  ie I would be willing to normalize the overlap space and I would like to know mean and variance of that new gaussian.
Erm, no I don't think it will.  The marginal distributions of $\\hat{\\beta}_1$ and $\\hat{\\beta}_2$ here will be univariate Normal.  If you consider them together (as I've done) you'll get a multivariate Normal, where the off-diagonals of the covariance matrix give the covariance between parameters.  In other words, if you consider the parameters to be univariate Normal, and that there is some covariance between them (which may or may not be 0), then the joint distribution will be multivariate Normal here (with the appropriate covariance matrix).  Hope that clears things up a bit.
This question is linked to a side comment [here](http://stats.stackexchange.com/q/66648/27403) .
@Bogdanovist You can just use latex markup.  I think what you're looking for is 

$$
\\left[\\begin{array}{cc}
 1 & t \\\\
 0 & 1 \\\\
\\end{array}\\right]
$$

which isi `$$
\\left[\\begin{array}{cc}
 1 & t \\\\
 0 & 1 \\\\
\\end{array}\\right]
$$`
You are considering finding "which personality score has a significant correlation". If you do that 9 times with p=.05, you will be inflating your alpha level considerably.  Also, when you put only those variables in your model you will be capitalizing on chance.  That will mean that any predictions you make using your model may be inflated.  For exploratory purposes, that might be ok.  (But then, most anything is fine for exploratory purposes.)
I can't up-vote till 15 rep. If no other answers come in then I will accept it as an answer.
Not necessarily - it depends on the values of $X_1, X_2$.
Thanks, anything more concrete?
Fair enough, what I'm wondering now is, are you good to go at this point, or do you need extra hints about how to approach what we've guessed the question is asking for?
@KerpalJenkiens There is a [FAQ on Stack Overflow](http://stackoverflow.com/a/5963610/1412059) which explains how to share data for R-related questions.
When the $X$'s have the same distribution, $E(\\frac{1}{n}\\sum^{n}_{i=1}X_{i}) = E(X_j)$ for all $j$ (from basic properties of expectations). When $n=1$ and $j=1$, you equate $E(X_1)$ with $X_1$. Let $Z=X_1$ and by C&B's definition $E(Z)=Z$, ... and so on.
Have you thought of using one-class classification instead of binary classification? Your sample sizes sound as if the problem was particularly ill-posed in a binary classification framework.
(+1) Good to know, thanks. I should look again in Michel Tenenhaus's work -- I'll let you know if I find sth interesting.
Based on the wikipedia definition of a giant component, it merely need be connected a majority of the nodes in the network. Is this the precise interpretation that you are considering, i.e. that the giant component is connected to >.5 of N where N is the number of nodes in the network?
That's quite elaborate for an answer. Give me sometime while I try my best to understand.
The answer at http://stats.stackexchange.com/a/1448 includes an illustration showing this result is merely the Pythagorean Theorem.
Although this is a good (standard) approach, you describe a *two-tailed* test rather than the one-tailed test requested in the question.
Although this is correct, what is the answer to the question? What is the "crucial and fundamental information" that the OP may be overlooking?  (I suspect, as I hinted in a comment, that it may be as simple as the use of $1/(n-1)$ instead of $1/n$.)
There's individual pixels that contain information from the sample and there's pixels that don't. They may be at the max for the detector because that pixel on the ccd is broken or because the light source bounced off the sample holders. They also could be near zero value because the old program thinks 65535+65535=0. There's a big shadow from the sample holder. Our signal to noise ratio is low, so you have to get rid of bad pixels manually by cutting them out of the image. For the peaks related to setup you stop when you get tired. There needed to be a uniform way of doing it.
It will depend on the geometry of posterior AFAIK. If your posterior is a normal distribution or something close to it and if you have a proposal distribution that is also a normal, I don't see why you couldn't "control" your acceptance rate doing trial and error in the variance of the proposal. Of course, real problems can be (and are) way more complex than this.
Could you give us some more details about what you want to study? Are you interested in total richness? Mosses' responses to environmental gradients? Co-occurrence patterns? Also, have you thought about sampling procedures (e.g. quadrats versus transects)?
@Iterator Your good suggestion is exactly what I was hoping this discussion would prompt: when we contemplate how to model our data, we often get insight into what assumptions are being made and need to be tested.
+1: That's a really useful SO thread there. Pretty much reduces my question to a repost :P.
Similar question at http://stats.stackexchange.com/questions/2957/ols-is-blue-but-what-if-i-dont-care-about-unbiasedness-and-linearity/2961#2961. A nice review paper on the topic is http://webee.technion.ac.il/Sites/People/YoninaEldar/Download/67j-04490210.pdf
I would say the first part (perhaps it was in an edit) is statistical, so I am voting to leave it open. Some questions are mixed.
7747 residual degrees of freedom is a lot; is it possible that your data set has multiple responses per individual? If that's the case, you might want to either collapse each person's responses to a mean (automatically done by ezANOVA from the ez package), or use something like mixed effects models, which permit you to account for the repeated measurements (check out ezMixed from the ez package).
could you show a screen shot of what the output looks like before and after this step?
Thanks. The problem here is that if I use an increasing decision threshold which ranges from 0.1 to 0.9 for building my ROC curve, it is possible to face data with class probabilities below the threshold. So how should this data be classified (for building the roc) ?
@Hani Gotc: Sounds good! I've built systems doing this sort of thing before. It shouldn't be too hard to code, if you're already working in a classification software pipeline of some kind!
For another way to see this, imagine a simple data set with 5 values of 0 and 5 values of 1. Then the quartiles are 0 and 1 and the IQR is 1. So the fences are at -1.5 and 2.5. It is very easy even for the fences to be completely outside the range of the data.
You're trying to compare two measurements of the same system to assess reliability or what?
Thank you very much for your detailed answer. Do I interpret correctly that the existence of $\\mu_i$ and $\\sigma_i^2$ stem from the application of the CLT, or can we assume that all distributions have such quantities associated? I apologize if the question is getting too abstract, but I am really interested in getting to the limitations of this reasoning.
I know that I should - I would greatly appreciate some pointers as to where to start.
That probably isn't the distribution you mean... you probably mean the difference between the values of the cumulative distribution $P(y|\\mu, \\sigma^2)$ at the endpoints of whichever interval "$y$" falls into. That likelihood function is no longer a function of $\\bar{y}$ and $s^2_y$, due to the rounding of the data.
I am looking an assessment of house prices.  My independent variable is Assessed House Price divided by Sales Price.  My dependent variables are several race categories (percent black, white, hispanic, and asian) and median household income.      
I'm finding that census tracts with a larger percent of blacks have a higher assessment to sales price ratio than other areas.
Since we're presenting this as "feedback" in a UI, we can set the assumptions we like, and perhaps have an "outlet valve" where they can explain how the PDF misses their intuition. Cook's blog is very helpful and pragmatic, thank you for directing me toward that!
@whuber Thanks. Even before using any software, I like discussing Phillip Goud example (updated in my answer) and let them do the calculation by hand. Then, I think any software will do the job, provided they feel involved and do it themselves.
@HosseinGhiyasiMehr I can't do that. I've updated the question.
This is a rewording of a question on hold, already answered. It really would be better to reword that question on hold, not to abandon it to spawn another. Also, the question has to be very different; otherwise the answers will probably be identical, or you will get no reply.
Edit:  my background is in econometrics, so I'm not accustomed to the sorts of problems where k>n that are common in biostats.  If the answer is obvious, then recommendations for good basic reading material are welcome as well.
"ROCR currently supports only evaluation of binary classification tasks" is the output when i tried to input tpr and fpr in the rocr. Now any one suggest me how to plot roc curve. So, i tried with the r question number 9308227 and used that code to plot roc. I got two curve is it correct?
This answer seems to use unconventional notation.  What do these matrices mean?
This is not correct, David. Precision of the estimator does not depend on variability in the dependant variable, only the independent variable.
This question appears to be off-topic because it is about coding.
@Kwak: Thank you for the clarification.  But if the estimate is true only asymptotically and globally, why should we expect it to perform at all well in identifying changepoints?  Is there some additional property the broken line regression has, of which I am unaware, that provides some comfort in this respect?
@Nick You now have the reputation to vote to close, I believe--no need to wait for a mod to see your comment.
could you please share a little more information, please?  Have the scales been chosen from the literature in your field, with a comparable cohort of subjects?  Have you collected data and now wish to compare?  If you have collected the data, what is the sample size, and of more concern, how many times was each of the seven or four categories selected?  Thanks!
if you consider the seasonal cycle (one dummy variable for each month) you'll "mask" the influence of temperature, because it is already 'contained' into the dummy variables. I suggest you to use only one dummy variable for August, to model the industrial closure. That's a common approach, in order to underline the weather influence on electricity demand.
Let me say that I don't know, but I can make an educated first guess. (Hopefully this might help / provide some food for thought until someone else can answer w/ the derivation you need; then I'll delete this comment). You would put the correlation matrix b/t the variables. That is, $S=\\sigma_A*\\Sigma_{\\sigma_A,\\sigma_B}*\\sigma_B$. NB, that when the errors are uncorrelated, this will reduce the the equation you listed, as the correlation matrix will be the identity matrix.
Doesn't address the question (the page doesn't even mention QR)
This is just a comment, not an answer: The three chief reasons for re-expressing the response are to make the residuals homoscedastic, to linearize its relationship with the explanatory variables, and because theory suggests such a re-expression. Although you can *always* find a $\\lambda$ that improves $R^2$, that's really a side-effect, not an objective, and the effect on the predictors is--unpredictable. Thus, you should be paying attention to the regression diagnostics concerning the shape of the residuals and the goodness of fit more than anything else.
0.01 is only small by conventional standards.  If you use BIC to assess significance, the equivalent cutoff can be much smaller.  For example a sample size of 10,000 means using a p-value of something like 0.001.
unfortunately I do not have lots of data on all zip codes. but your idea is definitely worth trying. Thanks.
Exactly. That is my intent. Thanks!
For a solution in `R` that is extremely efficient and handles much, much larger values of $n$, please see http://stats.stackexchange.com/a/41263. For instance, it solved this problem for $n=10^4$, giving the full distribution, in under three seconds. (A comparable *Mathematica 9* solution--see @wolfies' answer--also performs well for smaller $n$ but could not complete the execution with such a large value of $n$.)
Glen_b, I added the standard errors as generated by lm.
@jerad Thanks, I hadn't read that post. My question is not so much on  how to combine these models (e.g. such as when Yann says `"using deep nets as factors in an MRF"`), but more about how to look at a deep net as a probabilistic factor graph. When Yann LeCun's says `"of course deep Boltzmann Machines are a form of probabilistic factor graph themselves"`, I am interested in seeing that connection mathematically.
"Rayleigh" maybe?
Thank you! I am trying to find the appropriate F test in SPSS but I did not find it yet. Do you know where I should look?
I find the first part a bit unclear. Are you saying that estimating the equation you stated in first differences and using the fixed effects estimator gives the same result, and now you wonder why that is?
+1 Minor quibble: many null hypotheses are indeed exactly true, especially in one-sided tests and in other situations with composite nulls.
Because the prior knowledge at the time was not very informative for his theorem.
Hi, well in my opinion nothing disappeared. I think that the names just changed. Kernels were called "spline models in reproducing hilbert space" and the Gaussian kernel was known as the "green function" developed to solve finite-elements problems numerically. Also neural networks are called "deep learning" today. And they definetely didnt stop to exist. When it came to public "dislike" of the NN, researcher started to call the matrix multiplication $Wx$ "linear embedding" instead of a neural network. :)
(+1) For stating the *correct* result about the relationship between MLE and MAP and for the warning in the last paragraph (among other reasons). To make that warning more explicit, this (or any!) approach will fail miserably if the statistics at hand are ancillary or nearly so. One can consider your toy example and $T = \\sum_i (X_i - \\bar X)^2$, for example.
@Miroslav Sabo. Sorry, I missed your comment while searching for the title. Yet a nice graph!
It's a good question.  You don't need to know statistics to ask a statistical question; you only have to be able to think clearly and quantitatively about your objectives and your data, and both characteristics are in evidence here.
+1 (You will find the $\\TeX$ works much better when you include entire formulas, rather than mixing and matching ordinary text with markup.  E.g., "$L(\\theta)$" (\\$L(\\theta)\\$) looks much better than "L($\\theta$)" (L(\\$\\theta\\$)).  Center the formulas with double escapes (\\$\\$).)
Right now I'm defining "easy to setup" as "the package exists" or "someone's posted/published a function to download the data"
maybe it worth trying the Granger causality for your time series. It gives you an idea that how well two time series are correlated.
Thanks! Of course, I figured you would not base your selection process on grades, I merely tried to present an alternative perspective. I have a number of friends that are devoted programming geeks, that never cherished under the rigidity of university education. Anyways, as long as you stay open minded about the strengths and weaknesses of the candidates, I am sure you'll get great team members eventually. :)
Actually, I want the alpha vector not the weights (that was a mistake). If you can help me in that I really appreciate.
I have officially accepted Galit's (sorry I called you Gail) answer although Bayer's answer came close to the answer I was expecting to my original question.
For a discussion on what "fully Bayesian" and "Empirical Bayes" mean see the answers in ["“Fully Bayesian” vs “Bayesian”"](http://stats.stackexchange.com/q/31849/10525).
The affine transformation $x\\to (b-a)x + a$ maps the interval $(0,1)$ into $(a,b)$ in a one-to-one fashion (provided $a\\ne b$). We don't have to worry about the endpoints.
I think thinking along the lines that the test will be very low-power, in the sense that if X was actually say uniform $[0,1.01c]$, then alot of the data won't be above $c$ anyway, so the "p-values" for each $X_i$ will be more than 0.5 alot of the time. But in principle, if readings come without error, 1 reading above $c$ (e.g. $1.005c$) should be enough to reject the upper bound of $c$
I'd use the squared error if using L-BFGS-B as it probably has a smoother gradient, unlike abs.
@DouglasZare I didn't think very much about what would happen at the second stage when you can add observations back.  So it may be that some of the less extreme outliers will be brought back.  Regardng methods for identifying outliers, I agree that it is important to identify outliers but removing them should not be automatic. Also what are the statistical properties of this procedure that makes it desirable.  For normally distributed data Grubbs' test and Dixon's ratio test are well-known methods for identifying one or a few outlying observations with desirable properties. So why elect this?
Berger and Wolpert is indeed a good exposition, and authoritative too. However, I prefer the more practically directed and less mathemtatical book "Likelihood" by AWF Edwards. Still in print, I think. http://books.google.com.au/books/about/Likelihood.html?id=LL08AAAAIAAJ
@MichaelMcGowan: A minor aside, I'm sure it was just inadvertent word choice, but just to clarify for others, that the Sidak correction is not a bound. It is exact for independent tests, but otherwise it can be either under or overconservative, depending on the model.
I hope the links above help you. If you have done linear programs and quadratic programs before, SVMs are not very hard to understand at all. The only hard part is solving the quadratic program and most papers on SVM just deal with fast solutions to the SVM quadratic problem. However, you can just go ahead and try out the SMO algorithm. It might be be beneficial for you to read about Lagrange duality and Karush-Kuhn-Tucker conditions.
[This answer](http://stats.stackexchange.com/a/55406/805) seems to cover some of your question.
$N$ has no distribution unless you assume one or unless you have some kind of stopping rule for a sequential experiment.  What's your situation?
This was my original plan, however, I get the impression that there may be different (formal) criteria for 'interpretability' as opposed to 'prediction error'. Unhelpfully, I can't find the source for this I encountered while trawling the net.
I've edited the question...it's only an example..I know that with this data there are no significant differences... I hope that now is more clear
thank you for your answer, but I think hypothesis you write (of real-life Mantel test hypothesis) is not useful with real-life data. I add my response to you above,I will be very glad if you response on it. Maybe I am wrong.
This answer seems to be away of the topic. The OP focuses on the problem on _large data_, thousands or millions of items. Is it possible to compute and store a square matrix of such size?
To understand what's going on, I just plotted the function in $\\nu$ in 'R': 'plot(nu, pt(-2*sqrt(nu/(nu-2)), df = nu)/pnorm(-2))', which seemed to be stable at around $\\nu = 8$. The maximum was around 1.09, so I guess you should be able to obtain the $\\nu$ numerically for all c in (1, 1.09).
2 regions had 3 sets of paired open/closed plots (side-by-side), for total of 6 pairs or 12 plots.  There were also 3 "reference" plots in another region which are naturally more open, thus we hope to compare bird densities in the open plots only (from the 2 higher nutrient regions, n = 6 plots) w/ those 3 reference plots.
(+1) As the first part of your answer shows, that the means and covariances are zero follows from something even more basic than spherical symmetry. A couple of other notes: (1) For the mean, it appears you're using the fact that $\\sum_i Y_i^2 = 1$ almost surely, but that might not be clear to other readings. (2) Regarding the variance, an alternative argument to that of symmetry is that $Y_i^2 \\sim \\mathrm{Beta}(1/2,(n-1)/2)$, from which the result also follows.
Please try to give a more formal definition of what you want to achieve. You want to order a list by two rankings: are these rankings consistent? what happens when they conflict? does one have priority over the other?
@csgellespie: you are totally correct. I will reword this question so as it can be used by future readers. Indeed, the wording was particularly poor.
That's not a flaw in the answer - what you were given is exactly what you asked for. If you wanted something different than a test of the null hypothesis you specified, you should not have asked for a test of that null hypothesis. [Besides that, you never *prove a null hypothesis*.]
+1 Clear summary.  I would add that influential cases are not usually a problem when their removal from the dataset would leave the parameter estimates essentially unchanged: the ones we worry about are those whose presence really *does* change the results.
Hi Peter, Thanks for the reply and the link. I do have a rough idea of what Support Vector Machines are, but have never actually used them for anything. Regardless, I'm looking for ready-to-methods with a usable software implementation, which I can just apply. If you are aware of any such project(s), please let me know. Unfortunately this area has a huge number of possibly related publications, with little order, so it is difficult to make sense of it.
But, @Jake Westfall, it's that way even if I'm concerned the p-values. I'm saying that because the correlation could inflate de error term. So, the p-values could be anti-conservative.
If one would insist of taking the logarithm, despite negative values, then a [GLM with a log-link](http://blog.stata.com/2011/08/22/) would be an alternative. But without seeing the regression diagnostics or knowing more about the outcome variable this is all speculation at this point. More info from the OP is definitely needed in order to give more complete answers.
Well, in effect, I would require some sort of aggregation e.g. the "average" ranking for each object... his could give an idea of which is the most popular, without wasting information regarding the third or fourth choice... But thank you for the idea, I think this is a different, but unders certain conditions, a defensible strategy
We have done this in two different ways and it has proved very useful with the large movements this week lcearly being outside the histogram of typical results.
Any information would be fine! Name of the process, or function, or whatever!
Good luck with your class!
Amazing. Thank you so much!