tttjjj/elt_bench huggingface dataset:
This test needs to be adjusted slightly because the expectation of $z$ is (approximately) proportional to the second derivative of the distribution.  In the example, where the threshold is near a mode, that second derivative is near zero so there's no problem, but for a threshold in an area of high curvature (around 70 or 90 in the simulated data) the adjustment could be material.  If I get a chance I'll edit this answer accordingly.
Someone posted [the same question](http://stats.stackexchange.com/q/78079/5509) later.
Thanks a lot!  I tried squared Euclidian distance and it solved another problem I was having.  Namely, so-called 'reversals' or non-monotonicity.  I am feeling somewhat reassured that my Ward implementation is correct.  Would it be correct to say that, in the case of Ward linkage, unlike UPGMA, the similarity coefficient does not affect which two clusters get combined, but it does affect what new similarity coefficient value gets assigned to the new combined cluster?
@christos thanks for accepting the answer but did it help you? I'd like to know whether the answer is of use (as it got no upvotes so far).
http://en.wikipedia.org/wiki/Uniform_distribution_%28continuous%29
+1 Well done!  Each diagram determines a linear relation between $\\Pr(x=1)$ and $\\Pr(y=1)$. The two diagrams therefore should be consistent with each other, but you have shown they are not.
Dear Manoel, thanks for your help. I believe you rendered my problem correctly. I started the analysis by neglecting dependent variable correlation and within-subject dependencies. I will now move on to check an increase in fit via HLM. However it is hard to find software that could account for the last step (dependent variable correlation and with-in indepedent variables). If anyone knows a package that can handle such analysis (without major programming skills required), I would be thankful for any hint. In any way many thanks to you!
"Can we do anything interesting" is rather broad, and relies on us guessing what might be interesting to you.
Do you have multiple replicates at each observed level of $x_i$?
The last part of the question sounds like you are trying to create a *valuation* out of *multiple attributes.*  For more on that see the question and replies at http://stats.stackexchange.com/q/9137 and http://stats.stackexchange.com/q/9358 .  In particular, note that neither normalization nor standardization have any direct relevance to the Dean's problem.
Perhaps you would like to relate your answer with the [method of moments](http://en.wikipedia.org/wiki/Method_of_moments_%28statistics%29) or [MLE](http://en.wikipedia.org/wiki/Maximum_Likelihood_Estimation) for completeness.
I had a friend ask me the same question previously as well and I was never able to find one. The only software I know of is Biomedware's ClusterSeer program (not free).
Oh thank you for that! I got it now, I just hope I could apply this to Space-Time ARIMA.
How not, Evan? That sounds like a useful result to me. It's telling you there are few detectable differences among players after you factor in the other variables. However, such a result might not be credible in light of obvious differences in basketball performance. In fact, I found significant player differences even in the small subset you posted. It sounds like you should experiment on a smaller dataset and make sure you're applying and interpreting the procedures correctly by exploring the data extensively and producing diagnostics for the procedures.
Absolutely.  And if your results aren't significant at 0.05, you can always go to 0.10 and even 0.20: whatever it takes to get the answer you need, right?  :-)
Are you asking how to compare the proportion of people who use word processors given their OS - as in a simple interaction with logistic regression (or a chi square test)? maybe a data sample would help - I am not entirely following
@Henry Actually, in most $k$-parameter families of distributions with $k \\le 4$, the first four moments--which can be recovered from the mean, variance, skewness, and kurtosis--are usually enough to identify the distribution.
@user603 , Can you explain what you mean?
Dan, I think the second sentence of your answer is not strictly true. The single equation generalized error correction model (which is a cointegrating regression) is agnostic with respect to the integration of predictor variables. (as described by Banerjee; de Boef; de Boef & Keele)
A pivotal quantity Q = g(Y;theta) is a function of the data Y and the unknown parameter theta such that the distribution of the random variable Q is fully known ([Wikipedia](http://en.wikipedia.org/wiki/Pivotal_quantity)). There is an example in another question I asked [here](https://stats.stackexchange.com/questions/105727/confidence-interval-for-the-standard-deviation-of-a-normal-distribution-with-kno).
I apologize for confusion. The original question is edited. Thank you!
It would be helpful to know more here (perhaps some additional explication &/or some useful links). What is the Marchenko-Pastur distribution? How does it relate to PCA? What does it mean for your results if it holds or does not? (etc)
'Cleave' (split or stick to) might be a candidate for an English word for both a concept & its opposite. 'Quite' too: compare 'quite intolerable' with 'quite warm'. I remember Quine discussed the matter, but I forget where.
This answer is pretty misleading. The relationship  between the outcome and the variables is patently nonlinear:
$\\mathbb{E}(Y_i \\mid X_{1i},\\ldots, X_{4i}) = \\max(X_{1i}, \\sum_{k=1}^4 \\beta_k X_{ik})$ (which is the only thing that matters for consistency of the OLS estimates, serial correlation and homoskedasticity are correctable deviations, and normality of errors is completely irrelevant).
In the line where you pass from the cdf to the pdf, I think you got that wrong. Let $K_{n_{Y},n_{X}}=\\sqrt{\\frac{n_{Y}n_{X}}{n_{Y}+n_{X}}}D^{+}$, so (abusing notation), in the limit $P(K_{\\infty,\\infty}\\leq t) = 1 - e^{-2t^{2}}$. Then $f_K(t) = \\frac{d}{dt} 1 - e^{-2t^2} = 4t\\, e^{-2t^2}\\quad$ (note the $t$ after the $4$). (note also a missing sign in the exponent in the line above the taking of the derivative. Also I'm not sure why you have an integral symbol there, but maybe I misunderstood something.)
I completely agree that it needs to be decided by the investigator; but as a prospective investigator, how can I decide which follow-up analyses can best answer my research question? For example, in this example experiment, the general question is: Is there a difference between any of the treatments within level 1 or 2? Do these differences vary between level 1 and 2? Overall, since it excludes particular post-hoc testing, what is the use of a global test (other than convention) if the multiple comparisons desired are decided _a priori_?
*oob*? ${}{}{}$
Yes, I think you're quite right.  And overplotting regression lines on four scatterplots (all points, only Y outlier removed, only X outlier removed, and both outliers removed) should illustrate.
This is solved in many, many places on this site: the key is to recognize that your are trying to solve the least squares equations and the solutions are given by [normal equations](http://stats.stackexchange.com/search?q=normal+equations).  Follow the links in this search for your choice of demonstrations.  I'll close this thread and provide a link to the first hit in the search (ordered by relevance).
There is a skewnormal package called `sn` that does this. See this for a bit of an explanation: http://azzalini.stat.unipd.it/SN/faq-r.html The 1999 paper has more: http://azzalini.stat.unipd.it/SN/paper98e.pdf . You will need to check the parameterization matches what you expect. I was a bit surprised.
Thanks!!!So, confouding, if I understand properly, causes the so-called endogeneity issue... In this example, I found out that the correlation between one specific IV and the country dummy is rather high indeed. That IV refers to weather conditions in different regions of each country, so it sounds reasonable to absorb some of the explanatory power of the dummy!
Ah! Excellent! I had tried the AIC() and BIC() functions! I wasn't aware that the AIC() function could provide the BIC. Thanks very much, @Stat! This method saves a whole lot of bother.
It's unclear what you're asking here. Why do you need to write your own code? This is a standard problem tackled by any decent statistical software. Conversely, if you do need help writing code, or you are asking for code, either question is off-topic here; in any case people answering that would need to know what software you are using.
Thanks a lot for all the answers! I will be sure to mention that establishing a causal relationship is not my aim with this regression. Just wanted to make sure I didn't overlook some method to rule endogeneity out.
Do you have any paper that explain it
How would you propose exploiting the fact that $\\beta$ is "specific"? (The point behind this question is that an estimator is a *procedure,* not just the value you happen to obtain in any particular circumstance, and so properties like "best" and "unbiased" have to be properties of the procedure *regardless* of the value $\\beta$ happens to take.)
@MichaelChernick: You seem to be implicitly assuming that the element in $\\mathcal M$ that is chosen should be $\\mathcal P_{\\mathcal M} Y_{t+1}$, whereas, I am guessing this is precisely the question of the OP, i.e., what motivates the choice $P_{\\mathcal M} Y_{t+1}$ versus some other element of $\\mathcal M$. :)
I'd support this question, although I do not know the answer. No photograph accompanied the one obituary I can recall and there's none in the conversation with John Nelder at http://projecteuclid.org/download/pdf_1/euclid.ss/1056397489
I dont know, wavelets seem to blur the results, I want to look at a specific frequency, wavelets will show me all of the periodicities that make up the signal.
I may be using i.i.d. in an unconventional or incorrect way: It would be useful to get some feedback on this.
It is worth noting that it can be clearly demonstrated that pupils have differing levels of test scores *on average* dependent on their family socio-economic status. I don't think that this, however, would allow you to directly link *an individual's* SES with their test score (since there is naturally some variation at an individual level).
Do you have any data/theory as to what points are outliers? You can't simply assume that "far away" points are outliers, but if you know that something special happened on a particular date and that event would affect your data, you can add an indicator variable to your model for that date. See IrishStat's comment, below.
Could you help answer the question by explaining what "useful information" might be obtained from this ratio and what assumptions are needed for that to be the case?
You can still install the `intcox` package even if it's not on `CRAN` using the normal `install.packages("intcox")`.
As i dont have enough reputation points, the link to the second sub period can be found here; 
http://www.freeimagehosting.net/newuploads/betwt.png
I've accepted this answer on the somewhat capricious basis that I now remember Wasserman's book being recommended to me by someone else several years ago.  The same person also recommended "The Cartoon Guide to Statistics" by Gonick and Smith.
@Ehsan, Peter's hint is very relevant. The word "eigenvalue" comes from the matrix decomposition used in PCA and PCA-based FA. Other forms of FA compute loadings bypassing eigenvalues. The correct general word you should have used is "sum of squared loadings". Eigenvalues are they, too.
Your conclusion is right, but I think you've missed a step. Just because $X$ is non-negative doesn't mean $log_2(X)$ is positive. Trivially: $log_2(0.5) = -1$. You need a stronger constraint on what the product/quotient of the probabilities can be....
Broder - thanks for the detailed info. That has been really helpful! I initially tried using the data mining functionality in SQL Server, however, following your advice I switched to using R. I used the SMOTE algorithm to rebalance the data set and tried using both decision trees and SVM. DTs give a balanced accuracy of 81%, and even better with SVM. One question though - should I test the model against a data set which also contains rebalancd data? Or should it be tested against data more like the original?
Perhaps. Would it be feasible to use forward feature selection with a least squares estimation on the data before using ANN?
What is `offset` actually doing?  How is the value of `1.1*x1` determined from the question?
I modified a little bit. Thanks very much.
With the definition you're using there, there *can* be a $c$ such that $a f(c) + b g(c) = 0$; they're only linearly dependent if it happens for all $x$ in the considered domain; for example, consider your second example, with $c=\\pi/3$. (Also, I think there's a problem with your first example)
@user369122: If my answer works for you, feel free to upvote it or mark it as accepted.
This appears to qualify as "self-study" as it seems that you need detailed instructions on how to approach your problem as an application of regression. You need to show us at a minimum your preliminary thinking, attempts to solve it, etc. See http://meta.stackexchange.com/questions/10811/how-to-ask-and-answer-homework-questions for more guidance. CV isn't a site that does your homework or your research for you.
@John, you are correct, n=6 was my point.   I tried several schemes that you might use in fitting n for your data and a t distribution (using fitdist from package fitdistrplus), but I couldn't configure a way that I thought was worth posting.
@B_Miner - I think you are mixing concepts here. By using a smaller number of PCs (e.g. `s$u[,1]`...) the reconstruction of the iris data is truncated. i.e. the reconstruction of the iris data contains a large part of the variance, but the reconstruction is perfect only if all PCs are used. Using the PCs for prediction usually requires projecting a new data set onto the PCs in order to return a new set of loadings: `new.u <- new.dat %*% s$v %*% solve(diag(s$d))`. This is essentially what `predict()` does with class : `prcomp`.
It might be the [contribution to sum of squares](http://en.wikipedia.org/wiki/Analysis_of_variance#History) (the term is used there, but not defined). There are other possible uses, but given the extra details (which should be added to the question), that's my guess.
@entreprogreur, I did not answer, IMA did. IMA get's full credit here. I just tweaked the formatting so that it would display nicely.
Much better; I've removed the downvote.  But I'm intrigued by the new explanation: could you point out some source--a text, paper, or Web site--that actually makes the claim "[the median] is also considered robust to bias in skewed distributions" and explains what that might mean?  I haven't come across such a claim before and am not sure what it's really saying.
First: welcome to CrossValidated! Second: if you are *only* asking about software, this is explicitly considered off-topic on CrossValidated. Third: what are your problems? What have you tried, and how did it fail? Ideally, post a minimal working example (MWE) that replicates your problem, otherwise it will be very difficult for anyone to help you.
It's getting clearer, no need to apologize. =)
I would advise you to add this extra information of what you want to compare directly to the main text, paying attention to state the question clearly as you wrote in your first comment.
I understand now my mistake on the interpretation of the $\\beta_i$'s. Yet, I still have to say you did not answer me as to where you drew your $X_i$'s from. This is fundamental to understand the validity of the bootstrapping procedure.
I don't think you need math here as much as statistical thinking, not that the two are contradictory. With a negative constant the mean is negated but the standard deviation will always remain zero or positive. So the CV is negated.... As @whuber says the CV is for non-negative variables. (Although I have seen _published_ examples of negative CV for Celsius temperatures with negative means, compounded with the comment that using Fahrenheit temperatures would be better! In fact, CVs require ratio scale measurements as well as values $\\ge 0$.)
Let the $8$ values be $z_1,\\ldots,z_8$. The lower table appears to sample some function $f(a,b)$. You seem to be asking how to estimate $a_i$ and $b_i$ so that $z_i=f(a_i,b_i)$ for $i=1,\\ldots,8$. If so, this is an overparameterized problem: you have twice as many parameters to fit as data points ($8$ $a$'s and $8$ $b$'s). To obtain a unique answer, you need to add constraints (at least eight dimensions of constraints, in fact).
Glen: if I simply added the standard deviation to each parameter fitted (`a,b,c`) and plotted that curve along with the best fit one, would it _mean_ something statistically speaking?
That indeed is a quite nice and useful paper. I can really recommend it.
Thank you for the paper!
Hi, no, this is not for some subject, i just want to know when someone sees a Expected Value of Sample Information graph, what can they understand from it?
I am trying to trace back `opt.crit="amse"` in `ets` hoping to learn how it is done, and see if I can make similar changes to models of my interest (manly the `VAR` function in the `vars` package). It leads me to the `forecast:::etsmodel` function, then to the `optim` function, but then I loose track of it. Could you give some hints on where the AMSE is exactly implemented?
Have you tried some kind of regularization scheme (e.g. ridge regression)?
Please show us how you generate your simulations and calculate their variances--how else are we to know what the problem is with them?  You might also be interested in [a closely related question](http://stats.stackexchange.com/questions/48086/algorithm-to-produce-autocorrelated-uniformly-distributed-number) about autocorrelated uniforms.
(Not to mention that the `dlm` package won't forecast if you include a `dlmModReg` element, in the current version anyhow. As a follow-on exercise, I am going to see if I can modify their `dlmForecast` function (which is completely in R, thankfully) to handle this specific case.)
@Drew75 I prefer a breakdown of cases e.g. does it help when the results are "skewed by different magnitudes", does it help when the results aren't skewed, et cetera, the best answer will cover different situations.
@sean I appreciate the translation of the jargon, but the situation is still vague. Top *what*? Keys? Values? Counts of keys? Counts of values? Sums of values by key? This question has to be edited so that it has some chance of being interpreted as intended by a majority of readers on this site, for otherwise it can accumulate responses that vary--and perhaps conflict with one another--due to these ambiguities.
Please do not cross-post questions *within* SE sites.
On data visualization questions, it helps if you provide one example (the chart) to illustrate what you are suggesting.
Doesn't matter, the recomendation is still useful for everyone else who reads the thread (like me ;o).
@Nick (ctd) ... Indeed, many people define the two by the conditioning you do (which since you condition on both means your discussion is 100% correct); this is consistent, since when you condition on both they *are* identical - homgeneity *is* independence. If you are concerned, it might be worth linking to some of the discussions of the difference, such as [this](http://stats.stackexchange.com/questions/77634/test-of-independence-vs-test-of-homogeneity), but I wouldn't worry - your 'You could use' makes your answer perfectly correct.
Birnbaum-Saunders
I'm not sure what you are asking: ANOVA works by, for instance, comparing the mean of caucasians (categorical) to the mean (on one item in your questionnaire) of the overall sample. If you want to compare the means between two different items, you are talking about something else - it would help if you were more specific about what your 'research question' is.
Thx @Glen_b. Any opinions on this q though?
While I'm not totally clear on what you're doing, I'd be inclined to combine the data (keeping track of which time span they're from, as it may be relevant in modelling the combined data) rather than the fits. If your data are not from separate time periods there will probably be dependence issues that should be included as part of the modelling (and since they're time series there may also be autocorrelation or even series integration/cointegration issues). If you *do* calculate yhats you should estimate their variances (since the yhats won't be equally precise) before trying to model them.
What are you trying to find out from this cross-correlation between the men's subscriptions and (an oddly modified) time? Are you sure it's what you wanted to do?
Hints: the case $i=2$ is commonly termed "best two out of three" and $i=3$ would be termed "best three out of five."  Now compare this information to the expressions for $E(X)$ in the answers to *both* parts of the question.
@Mog  You've asked a sensible question that is surprisingly difficult to answer (at least, to answer briefly). Wilson's method allows the local interpretation of "What are the chances that the true proportion in this case is within this interval?" in that it approximates both a Bayesian credible interval with a uniform prior and a Fisher fiducial interval. However, if you apply a continuity correction to make it behave more like a Clopper-Pearson interval then that interpretation is lost. In my opinion Wilson's intervals are excellent, without the 'correction'.
A nice question, but I think it might be a duplicate of [this one](http://stats.stackexchange.com/questions/20534/determine-accuracy-of-model-which-estimates-probability-of-event).
Do you mean - circular median?
Welcome to the site! I edited your post for grammar and spelling. I also put the formula into LaTeX format.
also `MuMIn`, `AICcmodavg` packages, although `glmulti` is cleverer about big model sets.
@Zen The issue regarding the variances of independent but non-identically distributed r.v.'s is a very subtle one, I don't think I still understand it clearly. The known Lyapunov or Lindeberg conditions are only _sufficient_ for the CLT to hold. There are cases where the CLT holds even though these conditions do not. So I think that if we don't bound the variances, then there is no single answer, and the problem becomes totally case-specific. Even Billingsley's book is not clear on the matter. The question is what the remainder will look like, and what we can tell about it.
What do you mean by: "we would like to test whether it is an vis-a-vis the general population"? and "single sample S"?
+1 for @rbatt recommendation. In fact, there are many fine books. Just to add a few more which, although old, are very good reads, you have the books by Gelb ("Applied Optimal Estimation") and Anderson & Moore ("Optimal Filtering"). Both again with an enginnering slant.
One rule of thumb is that if 80% of respondents give the same answer, the question is no informative. BUT - sometimes you want to know that. "Are you a murderer" is not an informative question by that rule, but you'd really want to know it before you got a new roommate. So there's no hard and fast rule.
Averaging the parameters will only work in the expected manner for linear models. However, even then you produce an aggregated or ensemble model, which is not the same as a single model.
Thank you, very helpful! However, I edited my initial question to give more detail, but I don't think it would be appropriate to treat my data as "how many observations" because by 'call rate' I mean chirps/sec per call, not total calls/hr. I may also analyze other properties such as amplitude. Thus, putting a 0 to denote no occurrence would be inappropriate for something such as amplitude where 0 is an actual measurement that wasn't really taken since there were no calls made. This is resulting in the unbalanced data: the varying # of calls made = varying # of call properties to be analyzed.
Thanks for suggestions. I gave up from WMW because of ties.
Why is the normal approximation bad to teach?