tttjjj/elt_bench huggingface dataset:
Thank you Michael. All I have found are strategies to combine pdf's in risk analysis, i.e. 10 'experts' forecast some pdf of risk (or an event related to risk) - how do you combine these to make a decision?  There are a few strategies but it does not seem that any are derived from probability equations.
no idea about SAS, but you can run the equivalent analysis in R, confirm that SAS & R give the same results, then look at the source code for R pretty easily.
For anyone else interested: this can be done in R with `cor.test()`
As I wrote (@ronaldo2 above) they are not time series. Although time series always was exciting to me.
computing a *single* sample cross-validation is not laborious (there is an explicit formula involving the diagonal of the hat matrix)
An SSM is an HMM: nothing in HMM says the states (or outputs) have to be discrete (although I admit they typically are).
please clarify your question - it is not clear that you need a 'treatment' to answer your question. can you provide a list or some examples of the questions in the survey? Do you plan to have 100 individuals, 50 sibling pairs, or something else?
I'm not sure to understand.  In your first code you compute $corr(exp(X),log(X))$ I agree with the low coefficient. But it's not the example I was taking.

In your last examples, I don't know what is 'xxx' and correlation is sensible to outliers, I agree. But my question was: why in the example of non linear cases I gave (without outliers) Pearson correlation is high?
Could you elaborate a little more on what you mean by practical guidance? How important is it that the textbook include formulas? What statistical tests are you interested in (simple univariate, bivariate, and multiple regression, or multivariate stats, etc.)? Is there a domain of application more relevant to you?
Look at the dimensional analysis: Your formula comes out in dollars.
Great point,+1. Just consider the expression $(x-a)^{2}+(x-b)^{2}$ and calculate its minimum. It is the midpoint between a and b. This is contrast with the $L_{1}$ error function, a.k.a. robust regression
Reading a couple of times your answer, I think that as the training progresses and the model starts to overfit, the number of functions that the model can approximate is actually reduced because it will approximate extremely well points from the training data but its predictions won't be good as it can't generalize to fit other points or similar datasets.
I more or less agree with the sentiment, but I don't think this high-minded view is really possible in many settings.
@Amyunimus `plogit` is in R (stats), no package required (at least not anymore)
Although I am glad you see an answer emerging, I cannot possibly *post* an answer because I still don't understand the question!  Your last comment seems to change everything: now you appear to be asking something general about statistical estimation. Remember, a bootstrap estimate is just another estimate and its properties can be evaluated using the general principles governing all estimators. There will *always* be concerns about whether one's assumptions are appropriate, but I cannot tell whether that is the thrust of your questioning or not.
Oddly enough, I was rather interested in hearing what other people had to say...
I am referring strictly to anova (aov), not lm
Maybe you could ask about, for instance, hazards connected with using horizontal and vertical length on the same plot? This way this is a vague and low quality question that have a great chance to get closed or never answered.
Also, AAA is better than AA, and AA is better than A, but the two differences are not equivalent. When you use OLS on the integer scale, you are imposing an assumption that distance between categories are all equal, which can be problematic.
1. p-value statistic is fisher exact test p-value. So if hit is 0 in foreground, p-value =1. 2. P-values is added to the question.
Welcome to the site, @Muhammad-Rahid. I took the liberty of editing your answer to make it more readable & in-line w/ the sites policies, I hope you don't mind. Since you're new here, you may want to read our [FAQ](http://stats.stackexchange.com/faq).
I was studying elemental probability. :-)
It is unclear what "singly connected might mean": after all, *all* convex sets are *connected* and *simply connected.*  Where did you encounter this phrase?  Incidentally, although strictly convex functions on closed compact domains do have unique global optima, not all functions with "definite optima" are convex.
@Adhesh, how exactly do you calculate the correlation between gender and language? All of the examples you gave in the above comment (with perhaps the exception of age groups) have no numerical representation with which one could calculate a Pearson correlation coefficient.
I'm trying to think of a quicker way to do it. The plan is for "positive=TRUE" to apply to the reconciled forecasts, but it has to work on hierarchies involving a million or more time series.
Obligatory RA Fisher quote required here: "To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of"
Is this a HW question? The Laplace approximation consists of approximating the logarithm of the likelihood by its second order Taylor series centered at the maximum likelihood estimates, so you should start by finding the MLE's and calculating the first 3 terms of the Taylor series of the log-likelihood.
I wouldn't use the average number of errors per hour as my outcome variable if I could avoid it. Do you have information on the actual number of errors, & the number of hours for each person?
By the way, when I asked the question on GitHub, one of the contributors told me that using `potentials` would also be one way to proceed. I have to say that I like your solution **a lot** though, since I can imagine one could potentially wrap **any** solver with your approach! (i.e. it could be used to generalize any type of optimization problem, not just LP).
I am still slightly confused about how to construct ui and ci, but getting there.  Thank you very much
+1, throw away this book, it uses misleading language. Technically speaking, the design probabilities ARE the true probabilities, so one does not have to hide behind a shaky "repeated sampling" frequentist argument to justify the confidence intervals.
The psychological conditioning to a particular gauge over time is a solid, valid point. My empirical observations regarding consistency of speedometer gauge orientation are not consistent with yours, though. Nearly *all* of the vehicles I have owned over my lifetime have had a somewhat different orientation! In particular, this mostly resorts from having different *maximum* values on the gauge. I've even owned vehicles with (a) a linear gauge(!) and (b) a gauge that changed from miles/hr to km/hour depending on the user's selection.
Hi FWaldner, this looks a little short for an answer. Could you consider expanding it by perhaps a sentence or two, perhaps to mention briefly what these packages do that is like or unlike the other suggestions, or even perhaps against other things? As it stands it's basically just a pair of links.
@dan - I had never heard of the Boruta package - it uses the randomForest package with some added on logic.  I'm going to check it out - thanks for the tip!  For your problem, you can still use randomForest in supervised mode with a formula like "ClusterId ~ ." - i.e. predict the Cluster using all the other variables.  Then you can examine the importance().  You might need to convert your ClusterId to a string so that randomForest can figure out that you want to do Classification Tree and not Regression Tree (it'll do both).  I hope this helps...
Do you have a link that adresses the question of how to simulate realizations of a Gaussian process, specifically? This is not adressed in the book (looking at the index).
Thanks for the tip, I enabled CUA mode.  Regarding the other tip, one of my original motivations for learning ESS was watching how quickly one of my professors could edit code during a lecture.  I am away from campus for the summer, but I will try to find a "mentor" when I return.
This question appears to be only about data minapulation in R and thus belongs on StackOverflow (where it's been asked numerous times already).
Thanks for that!  (and thanks to the other answerers too).
It is an exhaustive census of all individuals in a population
I added a self-study tag. For questions of this kind, we ask to see your progress so far, including tentative solutions or your thinking in that direction. CV does not exist to do your homework for you.
It sounds like you have an *intended sample size* of 10000 and a 97% non-response rate.  Your "population" is probably much, much greater than 10000.
I am convinced that in my case it does not. See for instance the task of filtering the interesting announcements from the uninteresting ones: All the announcements arrive at exactly the same time every day (being sent by a machine) and are temporally indistinct.
For Visualization & Fit, the best educational sources I currently know of, is whuber and Glen_b. You can google them.
(1) Do you want the most likely customer or the mean customer? e.g. if you have customers of age 20,20,50 is should you get 20 or 30? (2) Do you want to preserve dependencies between traits? e.g. what should you do if the most likely age is 20 and most likely gender is M but (20,M) is not likely? (3) Please explain better what you mean by "whether they differ"?
Yes - $t(N)$ will be considered quadratic complexity.  The reason is because for large enough $n$ (say $n$ > 1 billion), the quadratic term will far outstrip the linear term, and this holds for any such equation, no matter how small the quadratic coefficient or how large the linear coefficient.
:jbowman So I get it , you assume a model form and that the errors from this model are independent with constant mean and constant variance and that the coefficients of the model are invariant over the observation set. With all of that set in stone you definitely are getting the break point correct.
@Luigi: I used MS Excel for the numbers above.
CW perhaps?....
This is a FAQ: more than a thousand posts on this site deal with [logarithms and regression](http://stats.stackexchange.com/search?q=log+regression). Why don't you do a little searching, which will immediately uncover plenty of explanations?
@Andrie you should be able to use any statistic for hypothesis testing if you can come up with a suitable distribution for it... which I think is what Ian's code is doing but I'm having a hard time following it... Ian could you maybe explain in words how your code works (at least for my benefit)?
@StasK: Thanks. I updated the answer accordingly.
Regardless of software it's more important for you to select the type of analysis/forecasting. Given that in addition to past prices you have also some independent variables to account too, **ARIMA** with option of **transfer functions** is what comes to mind immediately. There exist software for automatic building ARIMA model. However, it is always a good idea to first investigate your price data for _stationarity_, _seasonality_ and _shifts_.
+1, I would note that in SPSS (which the OP mentions they are using) one could get a QQ plot of an observed distribution vs a theoretical one in the `PPLOT` command. Also some quick googling it appears there exist techniques for [95% confidence intervals for QQplots](http://exploringdatablog.blogspot.com/2011/03/many-uses-of-q-q-plots.html). It may be easier to code up 19 simulations though and plot the min-max at each quantile as you suggest.
You may want to try reading these threads: [how should I transform non-negative data including zeros](http://stats.stackexchange.com/questions/1444/), & [how small a quantity should be added to x to avoid taking the log of zero](http://stats.stackexchange.com/questions/30728/).
(+1) Such a culture!! By the way, the R script gives the exact answer, the comments aimed to help @chris through the Gaussian approximation.
@Macro and Michael to me fitting a model of the relation between $x$ and $y$ in a semi/non-parametric way is one way of testing the association between the two. Such a test could be extended by measuring the extent of association with the different ways you've each suggested. I think both answers and the follow-up here have been quite useful to me, _sans_ the _ad hominem_. However, since my question did include how we could "label its nature", which could be interpreted as model-fitting, I'm going to stick with Macro's answer.
[This question on the GIS](http://gis.stackexchange.com/a/31879/751) site has an example of building a quadtree to create irregular bins of a minimum size n. That being said, if this is *only* for a visualization I'm skeptical it is the most effective plot. Making a raster plot of the averages used e.g. [inverse distance weighting](http://stats.stackexchange.com/a/38658/1036) (or simply a raster plot of the continuous kernel density estimate) is likely an easier visualization to evaluate.
Assuming that you have a column labeled sex, then you would just have to add the corresponding condition. I ll edit the answer.
@ttnphns I tried to follow the Case Study, but it's very involved. Do you know where I can find a simpler example?
Can you describe what your input and desired output would be, in a bit more detail? You might be able to use something like a simple ranking or contingency table and pick the n-best words. Also you may be better off using an existing set of valenced words, or something like www.liwc.net.
Thank you for the response. I attached the time series file through a link in the initial box above.
There are plenty of sources how to do simple correlation of two columns if you just google it...show some research effort
You may want to have a look at: http://stats.stackexchange.com/questions/2492/is-normality-testing-essentially-useless
Thank you very much. I picked up Durbin-Koopman today and it looks great.
@PeterFlom, I did not mean to imply that your answer above was "just plain wrong".  I have edited my answer to smooth out the rough edges.  I apologize for any offense.
Sam, to make this question to suitable here, please indicate what efforts you have made toward solving these problems and precisely where you are stuck or need help.  More information about what's on topic and how to ask good questions is available in our [faq].
Thanks @Hibernating, for pointing me to the article. Will take a look at it. This data corresponds to flow, and we are trying to measure the average and the confidence interval by selecting a time-frame were oscillations have dissipated enough and external processes are not affecting the measurement. Sometimes we don't know the exact effect of these external processes (like in the second graph). Also, I can't see how the oscillation component can be considered piece-wise stationary. I see what you are saying about random walks, but I still can't reconcile the answer with my needs.
I asked a question that illustrates the need of factor rotation after PCA since PCA gives the biased result. See http://stats.stackexchange.com/questions/6575/how-to-do-primary-component-analysis-on-multi-mode-data-with-non-orthogonal-prima
As I see nls() doesn't calculate those statistics for you, but it gives you the residual sum of squared RSS (see above), and that should be everything you need to calculate R-Squared (see wikipedia page), for AIC ($AIC = n \\times ln(RSS/n) + 2k + constant$) (see wikipedia page again), etcetera.
It is also worth noting that the usage of probit versus logit models is heavily influenced by disciplinary tradition. For instance, economist seem far more used to probit analysis while researchers in psychometrics rely mostly on logit models.
+1 I appreciate the interesting ideas and thoughtful discussion of them.
An explanation of why the Kolmogorov-Smirnov test is not appropriate with binned data would be useful.
Yes that's exatcty what I meant. Duly noted for future reference.
It depends on what you mean by "prove."  As stated, this is a philosophical question, not a statistical one, and has no definitive answer (although, at least since David Hume's time, most people would answer "no").
Questions asking for code are off-topic on CV (see our [help center](http://stats.stackexchange.com/help/on-topic)). They can be on-topic on [Stack Overflow](http://stackoverflow.com/), but this wouldn't be. You should take their [tour](http://stackoverflow.com/tour), & read their [guidelines for asking questions](http://stackoverflow.com/help/asking) there. Then you may be able to ask an updated version of this question there.
Also I do get the actual probability that candidate A will win the state I do that by prob1wins <- (sum(as.numeric(Diff>0))/nsim), where Diff   <- p1 -p2
Thanks Mike :)  I've seen from previous answers you are strong on GAM.  I will have a look at it in the future, for sure :)
So in what way exactly do hidden Markov models help to monitor HIV? Do clinicians use HMMs to diagnose HIV? Do researchers use them to better understand mechanisms of the disease or create anti-HIV drugs and therapies? Any references would be very helpful.
@steffen: Your referenced question is helpful, though, I think it's not a duplicate. Indeed my question is rather generic. I'm not looking for a solution to a particular problem but for general reasons why to use which learners - I'll update my question accordingly.
@gung how did you get the data to look like that?
Why are you treating it as a classification problem? For me it sounds more like a regression problem...? If you still want to treat it as a classification task, then why only two groups?
It would be easier by viewing (a part of) the dataset. But I am under the impression it could work by considering Location is nested in Design. Does it work if you do `Design + Design:Location` instead of `Location * Design` ?
@Vass, indeed (although you could project down to 2d with PCA, to plot). Are any of of the points labelled, for comparison ?
@PeterFlom. Thanks. I'll try to get a copy of this.
"*which I assume is some sort of gamma distribution.*"  - you're observing data values over time, and those aren't displayed as a density. On what basis do you say "gamma distribution"?
On the final question, I'd just look at the Pearson residuals or the (signed square roots of) the contribution to chi-square to decide where the main 'causes' of the difference come from.
Ah, I didn't realize there were multiple wells for each dose. In that case, I would consider this a cluster design that could be analyzed with, e.g., a multilevel model. The clusters are the wells and the individuals within each cluster are the cells.
What do you mean by "high regression"?
(+1) Welcome to the site, Sergiy!
"exponential" usually implies something based on `exp()`: what you have here is more commonly called power function, power law, or scaling law. Other names no doubt exist. There is no connection with power in the sense of hypothesis testing.
It is not necessary to find the _distribution_ of $U_1+U_2$ in order to find the mean and variance of $U_1+U_2$; you can use what is sometimes called the [Law of the Unconscious Statistician](http://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician). But in this case, finding the distribution of $U_1+U_2$ is easy if you work from basics instead of searching your book for a formula to apply. Consider that all $3! = 6$ orderings of the values of $Y_1,Y_2,Y_3$ are equally likely, and work out the values of $U_1+U_2$ that these orderings give, and you have the distribution of $U_1+U_2$.
Re the edit: what, exactly, is being combined when you have only one record per name? There doesn't seem to be anything to average in your example.
(+1) I had not realized these (well-received) books included advanced topics as well as introductory material.
Check Section 4.5 of this [link](https://files.nyu.edu/mrg217/public/tobit1.pdf).
What is the purpose of this process?  That is, why are your removing extreme data in the first place?
A scatter might be useful (I've problems seeing the distribution from your description).
The probability is 1.0 they are different, as they are both continuous random variables.  Consider: what is your prior guess that $\\lambda_1 = \\lambda_2$? Do you really think they are actually equal? (Ignore hypothesis testing: we are living in the real world where variables are never actually equal). See this [post](http://andrewgelman.com/2004/12/29/type_1_type_2_t/) by my hero, Gelman. Computationally, you can test this by computing `np.mean( lambda_2_samples != lambda_1_samples)`.
@Nick: The last few years, there has been substantial progress on evaluating model-selection properties of the lasso. See Knight & Fu (2000), Zhao & Yu (2006), Meinshausen & Buhlmann (2006) and Meinshausen & Yu (2009), for example.
or, equivalently, if each partition is weighted by its size
Richard Socher wrote the [blog post in question](http://hunch.net/?p=1852).
A non-stationary is the special case of AR(1) with the coefficient on the first period lag of dependent variable (which is ln of stock price) equal to one. How did you test the unit root?
Do you have a control group?
@zb Randomness is good, but I don't think it's the issue.  The question in my mind concerns whether the ELO method truly addresses the question.  This is an issue that ought to be discussed on its merits, not on the basis of reputations of those involved :-).
Indeed, my reply was not that good. Any data mining needs some preparation steps on data (ETL for extract, transform, load). I use perl for that purposes. In addition, analysis in itself might be good with Octave if you are used to Matlab; it might also use other tools like R. Any tool has his pros and cons, you should try and take your own conclusions.
@MichaelChernick, no, I think he means 25 million paired values of two variables (X and Y).
(4) and (5) are answered at http://stats.stackexchange.com/questions/32600/in-what-order-should-you-do-linear-regression-diagnostics. (1) is answered in a very large number of comments (and a few answers); consider this a FAQ.
I'd use a sparse PCA algorithm for that; see for example this question and the first answer to it: http://stats.stackexchange.com/questions/66334/why-such-a-poor-result-from-sparse-pca-r-package/67631
Hi Glen_b, I really learned a lot and wondering if we can have a deeper conversation maybe in a chat room from stackoverflow so we don't overwhelm that comments.. And Maybe I can offer some points to mark this question as bounty question to show my appreciation?
Thanks. With your graph I now understand what constant variance is.Also if dependence of y's have nothing to do with independence of errors, say if for different x values the y values are same and they are scattered according to some grouping.Then can i  say that since the errors are not randomly distributed  the assumption om errors are independent is violated.Also in your early comment you say that " The variance looks close to constant because the spread of values at each x is about the same, roughly 1-1.3".Here 1-1.3 indicate the height of green braces right?
I'll look into that.  Thank you.
This **question is related** (though is not identical) to http://stats.stackexchange.com/q/81986/3277 and a further one linked to there.
(+1) Nice finding!
@Maurizio -- the K-S test statistic has the same distribution under all _continuous_ distributions, but if the actual distribution is not continuous, and one tries to construct a level $\\alpha$ test assuming that the distribution is continuous, then the actual level of the test with be less than $\\alpha$.  (c.f. Lehmann & Romano _Testing Statistical Hypotheses, Third Edition_, p. 584).  You can still make a level $\\alpha$ test based on the K-S statistic, but you'll have to find some other method to get the critical value, e.g. by simulation.
What's the context of the study? Is the outcome nominal or ordinal (that is, can we assume levels are ordered or not)? Are you interested in a descriptive, explanatory, or predictive approach?
Howell is very good, but not very mathematical.