tttjjj/elt_bench huggingface dataset:
This appears to fall under the [self-study](http://stats.stackexchange.com/tags/self-study/info) tag. please add it here and on your other question(s) as needed.
Ah, this is clever: the product of a $\\Gamma(\\alpha+1)$ and an independent $B(\\alpha,1)$ variate turns out to be a $\\Gamma(\\alpha)$ variate.  I edited my reply so it points to your solution and explains why it works.
Thanks! I had used the link https://www.mathworks.com/support/solutions/en/data/1-1J7OE3/index.html?product=SL&solution=1-1J7OE3 earlier.  I now see that it dealt with percent ranks (& not percentile!)
20826 -- I'm afraid you ruled out the distribution it actually is in your comments.
For simple ridge regression you don't need anything as fancy as glmnet.  You really just need a fast linear least squares solver.
BTW It's queer to describe *all* the coefficients as "intercepts"
I upvoted this but I imagine the downvote was because the question covers too many questions eg 1) which parts of statistical theory are "inference" and which "non-inference"; 2) is statistical inference a subset of logical inference; 3) how are logical inference, decision theory and statistical theory related to each other.  It would be an impressive answer that covered all of this off (but +1 for the question because I think it is possible for the right statistical philosopher).
@ mpiktas : I edited my post following your comment to make things clearer.
+1 - good advice and interesting discussion of leverage (for large datasets).  I tend to use datasets that are small, where over-fitting is likely, and in such situations it is often very easy to over-fit the marginal likelihood ("evidence"), and end up with a worse model than you start with.  I suspect AIC and BIC are similarly "brittle".  Essentially optimisation is the root of all evil in statistics, as any choice you make or parameter you tune based on a finite sample introduces the possibility of over-fitting. Marginalising is much safer, but generally computationally expensive.
@joint_p nope not a hierarchical model.
My approach is:
Use training data set to create a regression model

`>attach(train)`

`>lreg=lm(category~date_time)`

Now do predictions for validation data set using model created with training data set

 `>p=predict(lreg,valid)`

    Now check the accuracy by finding the values of ACC, AUC.

 `>mmetric(valid$category,p,"AUC")`

  `>mmetric(valid$category,p,"ACC")`

Now if AUC and ACC have small values then it means that model created by training data set is not good in making predictions.
Is my approach correct ?
I'm told that the results of the benchmarks (the times in ms) are not normally distributed. Will the variances be normally distributed? How does that affect the choice of 1.96?
Your example is very interesting thank you very much for your comment.
@Yuriy The rows should not be out of order, but here is a way to do it one call to `aggregate()`: `aggregate(age ~ group, data=dat, FUN = function(x) c(M=mean(x), SD=sd(x)))`
I missed that. I'll correct my answer to get rid of tedious algebra.
I reproduced the data myself from the source web site, so I included weights and did not apply the arbitrary cutoff threshold. Thus I think the data I applied the backbone method to should not have been affected by these issues. Good idea on checking the degree distribution -- I'll have to take a look!
I appreciate your straightforward explanation, as well as the suggestion on rigorousness!
Ah.  Good point.  I've changed the title to "probit," since the focus of the question was really on the "ordered" part.
+1 Alpaydin is the right way to go. I was in the exact same situation as the OP a few months ago. Struggling badly with Tibshirani, and then came across Alpaydin and things have been much better since. Eventually though I think Tibshirani is a must read.
What is the source of the question? Some textbook?
Can you please sketch the contingency table for your data?
How was your biplot constructed on the original dataset? (I asked that because some routines impute missing values by, e.g. column mean in case of PCA.)
+1, nice and simple
You may compare preference profiles of groups "A, not B" and "B, not A". The products which are highly preferred by _both_ of these groups might be recommended also to group "A and B".
@Hendy, there are many, many ways to do this. Regression would be one sensible way to go, particularly if you have have some insight into plausible relationships between the Xs and Ys. At the other extreme, a classification/machine learning approach might help you predict new X->Y relationships even if you don't know much about how Xs influence Y. It seems like your software can do both.
Don't these become unbiased only in conjunction with additional distributional assumptions?  For example, the relationship between the range and the SD *has* to depend on the shape of the distribution.  In that context, it's worth observing that many more unbiased estimators of SD have been studied, such as those based on linear combinations of quantiles and ranks.
Can you explain more clearly what you're trying to do?
A decision stump is biased, but they have been used successfully with bagging.
The p-value is the probably of getting as much correlation as that observed if there is no real correlation. So if the p is very small, as it is, that suggests there is a lot of correlation present in the sample.
What search terms did you actually use? I'd start with "meta-analysis".
@cool-RR: why don't you just estimate P(A|B) directly, then, as the fraction of B samples that are also A?
I've removed the "sem" tag, which refers to Structural Equation Models, which is different from Simultaneous Equation Models. (EDIT: Not an expert in SUR, but my first impression is that SUR is a subset of simultaneous equation models.)
@VincentZoonekynd I see, maybe I am misreading the paper?  But the author says $\\sigma$~$IG(1,E,F)$, I did so.  Am I making a mistake? The $\\sigma$ is almost 0.
Thank you for your good answer @mbq. I kind of now get what you say. I have now a specific question for my above question. Lets say I consider only one method. I wish to calculate ROC for the same. I have also come across this package called "pROC" given on this link (last figure for statistical comparison). In my case, can I give my input as p-value and FDR only?
Like you said very very very ad hoc "like fit with AR(1) errors, then exp-smooth the residuals (possibly on the differences, then look for a trend) but that feels very ad hoc." and very probably just wrong !
@Matthew your deleted answer looks fine to me. Was there a need to remove it?
To supplement @John's reply, a re-expression is given by a mathematical function $f$ of a single variable.  The re-expressed values of a dataset $(x_1,x_2,\\ldots,x_n)$ are, by definition, $(f(x_1), f(x_2), \\ldots, f(x_n))$.  This precludes, for instance, normalization of one quantity by values of another.  Formally, such normalization changes *bivariate* data $((x_1,y_1), \\ldots, (x_n,y_n))$ into $(x_1/y_1, \\ldots, x_n/y_n)$.  Mathematically that can be considered a "transformation," but in the statistical literature, "re-expression" applies only to univariate transformations.
+1. I agree with what you say. As I mentioned in my original post : "*over-dispersion (...) is how you "get around" the issue of having as many groups as observations.*" Thank you for making a better point of `glmer` in a conceptual manner.
@NickCox Thanks for the hint, added.
As Michael points out, you usually use a cutoff on the predicted probability. However, the issue lies in choosing a good cutoff (0.5 may not always be ideal). One method is to use a variety of cutoffs and use cross-validation to select the "best" cutoff. Then make your predictions in 27th testing set based on this optimal cutoff.
I've done some simulations to show what happens with this approach in my answer. In some cases, it's a reasonable option (in particular, at n=1000, many people may be happy with the result).
+1, an interesting (philosophical) perspective
Thanks for the answer Randel. I am waiting the end on the bounty to attribute my vote.
You might find some useful information in the related thread at http://stats.stackexchange.com/q/8502/919 .
OK, you're right in my misuse of terminology.  I've been incorrectly calling them priors or sometimes 'conditional priors' because I specify the conditional probability table as part of modeling the network. I'll update my question so hopefully future travelers aren't as confused.
http://imgur.com/0dsVC.gif
$y$ represents a mixture of values with different normal distributions. The vector of responses itself won't necessarily look normal.
If you have 2 variables, each of which is 0 / 1 only, then you are assessing how the proportion of 1s changes as you move from x=0 to x=1. Your response will be distributed as a binomial with variance = np(1-p); unless p(Y=1) is identical at both x=0 & x=1, the variance will not be constant. Thus the assumption of homogeneity of variance will be violated.
1+. Thanks. That's pretty much what I read in Wikipedia. However, is there something like $t = \\frac{\\hat{\\theta} - 1}{\\sigma}$ as with means and population proportions?
Never thought I'd say anything nice about a book with puppies on the front and cyan section headers, but... this looks just great.
[Data Mining: Concepts and Techniques](http://www.amazon.com/Data-Mining-Concepts-Techniques-Management/dp/0123814790) By Jiawei Han is a good start
Your being proxy here for a friend doesn't affect the fact that to us this is self-study, and I've tagged accordingly.
Thank you very much, that helped me alot! I just read some articles about beta regressions. There are beta regressions, and beta regressions inflated with 0 and/or 1´s.             In my data are three response-variables, like shown above in the histogram. They have got between 266 and 400 zeroes and between 1 and 12 one´s (n=865). Is this then called zero-and-one inflated or just zero-inflated data? Maybe the ones are not the problem, and if, i could change them to 0.99 to calculate with zero-inflated beta models?
Thanks for the advice. I will definitely try to examine the predictions before deciding on the model
@Glen_b P(Ti) represents the "probability" of using the statistical test Ti. I know that this is not exactly a probability in a strict manner. It's rather a weight that says that I have used _n_ tests for the same dataset.
Did you have something in mind that differs from [the solution Peter Ellis has posted here](http://stats.stackexchange.com/questions/46185/question-on-how-to-normalize-regression-coefficient)? If so, could you please elaborate on it?
PCA minimizes the reconstruction error: If you want to project data onto a low-dimensional space while disturbing the data as little as possible, then PCA is what you want.
$\\int_{-\\infty}^0 \\text{exp}\\{-\\frac{x^2}{4\\beta}-\\gamma x\\}dx = \\int_{0}^\\infty \\text{exp}\\{-\\frac{x^2}{4\\beta}+\\gamma x\\}dx$. Use the relation of erf with the standard normal cdf, and we'll take it from there.
I am not sure what your example means, could you pls explain it in words? Also, why is VIF appropriate? Is there any publication references you can give me? Many many thanks!
"*So then regularization is used as an alternative to significance testing?*" -- I don't think I'd characterize it that way. It's more that your interest is on estimation or predictive inference, significance testing isn't really the right tool. I also don't think I'd characterize regularization as 'a nonparametric way of fitting models'.
Interesting question. A couple of comments -- (1) `rpois` will generate from a vector of lambdas as is, you don't need to go through all that trouble to get it to do that; (2) Why do you say the fit should be exact? The model should be correct, but that's not the same as the fit being 'exact', unless you define 'exact' in a somewhat different way to its usual meaning.
Is there some way to apply the Kolmogorov-Smirnov test here other than looking at inter arrival times between consecutive arrivals?  If I look at all inter arrival times (that is all roughly $n^2$ of them for $n$ points), then I am not sure how to apply KS any more.
Everyone interested in please have a look at my question again. I put some examples. Thanks very much.
Because at a given number of hits already, you're in a sequence of independent Bernoulli trials, where the probability of succeeding (choosing a value you don't already have) is $(n-k)/n$. The number of trials to the next success is [Geometric](http://en.wikipedia.org/wiki/Geometric_distribution) with $p=(n-k)/n$. The mean of the 'number of trials' version of the geometric is $1/p$. If you want details of how to calculate of [means of distributions](http://en.wikipedia.org/wiki/Expected_value#Univariate_discrete_random_variable.2C_countable_case), I think that should be a new question.
see also http://stats.stackexchange.com/questions/41881/what-is-complete-sufficient-statistics
Are you asking what happens if the most probable value is *sampled* or are you asking what would happen if we always just take the mode of the conditional distribution at that step?
maybe to add a small provocation: one could "post-test" the surveys, and if there is a 100%-conditioned behaviour among the participants in this post-test, confidence intervals would be with an error = 0, as p^=100% . This works through the formulas I found, but I do not like it.
Is there a way to specify the distribution / standard deviation for the white noise process?
For me, the analogy is a little closer to "buying a car does not make you a mechanic or a car designer."
I edited the code to do a fit without the censored data and prove that the model and priors work. Basically it's failure time distributed as F(t) = dweib(a, b), and survival probability S(t) = 1 - F(t). I think the error lies in how I am preparing the event data for use with dinterval.
@marcin63:  all your observations $(x_i,y_i)$ will be on the same line with parameters $(\\beta_0+b,\\beta_1+a)$. So you can only estimate the slope and intercept of this line.
For this question to be understandable you will need to provide either a clear description of your data or an explicit example, because the meaning of "oriented lines" in this context is mysterious.  Please edit your question accordingly.
One way to think about this is to note that the independent variable has to *vary*. Since all the couples were married, and all received counseling, neither marriage nor counseling can be the IV
@Glen_b : I am studying UMVUE for a inference course and this was a question I came across in a note
I wanted to say testing if AUC value is statistically different from 0.5
@Nick the univariate solution has three paremeters while the causal had 4 . Admittedly this might be a bit much BUT the models seems to reflect the data and the forecasts seem reasonable.
It sounds like you want to solve problems induced by two rather uncommon steps (splitting and size matching). Thus the question: why not running a single logistic regresdion on the full sample?
I think a lot of your reasoning may be correct, but did you notice the OP specified a *chi* distribution, not *chi*-squared?  (This doesn't really change anything important, but it would be good to address the question as stated.)
This seems to be a duplicate of this question http://stats.stackexchange.com/questions/12134/estimating-the-distribution-of-a-very-large-population-of-known-size-and-unknown/12136#12136
Try plotting $x$ and $y$ and the two regions in the unit square that correspond to the different branches of the definition of $z$...
If you're applying to sales & trading at an investment bank you want to be expert on (8),(9) for general role and also splines (in (13)) for a quantitative role. You can ignore the rest except (2) for the brainteasers. For M&A/ECM/DCM you can ignore the entire list. All you will do is addition and division in Excel. For non-quantitative market making you want (2) for the interview and expert in (8). For statistician at a retail bank or other financial firm you want the econometrics.
Very clear answers to both questions. Thanks a lot, Aaron. I'll simplify my model and see what happens.
@schenectady thank you for the reference
(Dollar signs in the code seem to toggle the indentation--must be an interaction with the support for $\\TeX$ markup. Inserting extra dollar signs in comments to make an even number per line works around the problem!)
Chi-Square will only show an association but LR can prove its direction as well.  For example, Chi-Sq can show cash and bankruptcies are related but LR can show the lower cash levels are positively related to bankruptcies. Please see your response at http://stats.stackexchange.com/questions/88630/chi2-test-of-significance-vs-goodness-of-fit
I have usually found that when odds for an event happening are higher for a categorical variable, then its observed count is also >> expected count.  It seemed fine as it fit the theory (eg. when cash is below $10 million, chance for bankruptcy is high).
> this is not the same model (i.e. Kelly's model is non linear in the parameters).
@Glen_b I was floored to read your comment that "optimization" was on the topic list. I still don't see it on the [help page](http://stats.stackexchange.com/help/on-topic). What am I missing?
most references should cover this topic. Examples are given here for Stata: http://www.ats.ucla.edu/stat/stata/dae/melogit.htm
That explains it perfectly, thanks a bunch!
We agree on essentials, but the idea that you must have normal residuals is oversold. Even if the errors really are exactly normal, the residuals won't be. If the errors are (e.g.) uniform, the usual recipes will perform rather well; it's just that e.g. P-values won't be exactly trustworthy, but they never are, any way.
@felix I meant a scalar or a function. Sorry for my sloppy language. So, yes. You're log likelihood is concave.
+1 A single upvote seems inadequate, I think this answer is what SE was made for.
Mcmc should not be the focus of an introduction to bayesian statistics, in my opinion. I think rejection sampling is more appealling as a way to understand how bayesian learning works.  Also, least squares is bayesian (as is maximum likehood), so it too represents a gentler introduction to bayesian statistics, compared to mcmc.
(+1) Thoughful, well-articulated answer. One thing to keep in mind is that this site has a fairly international audience. I would hope more researchers would take a collaborative view toward their research and data even if there is no NIH or other organization twisting their arm.
"The bootstrap bias-corrected accelerated (BCa) interval is a modification of the percentile method that adjusts the percentiles to correct for bias and skewness" (Hesterberg, T., Monaghan, S., Moore, D., Clipson, A., & Epstein, R. (2005). Bootstrap methods and permutation tests. Introduction to the Practice of Statistics, 14.1–14.70.). Whenever the software allows it, use the BCa corrected CI (note: it needs > 1000 resamples)
(ctd) ... \\* (I often steer people away from hypothesis testing - so often their actual questions are related to measuring effects, and are better answered by constructing intervals). The specific problem Lew raised for the 'hybrid' procedure applies to something I don't do and would tend to caution people against doing. If there are people really doing the mix of approaches he implies, the paper seems fine. The earlier discussion of the meaning of p-values and the history of the approaches seems excellent.
This line: freq.ex<-(dpois(0:max(x.poi),lambda=lambda.est)*200) produces errors with some real world data, because the length of freq.ex won't match freq.obs on this line acc <- mean(abs(freq.os-trunc(freq.ex))).  I adapted this line to freq.ex<-(dpois(seq(0,max(x.poi))[seq(0:max(x.poi)) %in% x.poi],lambda=lambda.est)*200) but something still isn't quite right because goodfit produces warnings.
Thank both of you for constructive comments :)
Yes, but the equation used by Jurafsky is P(w1, w2, ..., wN)^-(1/N)
I need the measures in the accompanying vector to somehow encompass the variation evident in the matrix. I've though of just using an average of distances, but was hoping for a measure that has more complexity. This is a bit of a fishing expedition because I've run out of ideas for how to search for a solution.
is this the way =(value-min)/(max-min)
Thanks for a reference to the article, it can be found in the web.
http://en.wikipedia.org/wiki/Probability_integral_transform
Thanks.  I mentioned *three* dimensions because you used "hyperspherical" in the question, suggesting little or no interest in one or two dimensions.  In both one and two dimensions the marginal is indeed bimodal.  In higher dimensions you won't have problems with Ray's formula.  Once again it is clear that this answer really needs some analysis to show (a) that it can work and (b) under which circumstances it does work.
1. different body size is not feature, I have unfortunately more younger females and older males. But my data for every group are still normally distributed.
+1 but (1) You can format your code using indentation (with 4 spaces or through the online editor); (2) Some comments around your code might be appreciated because users that are not familiar with Matlab syntax will have hard time to understand what the above code chunk actually does. As far as I understand, predicted class memberships on the test sample are stored at each iteration, and then averaged to yield the classification CR.
many thanks for you very detailed answer. I used a similar script some time ago when I was investigating air temperature variations for a number of countries in Europe and I posted that script on http://r.789695.n4.nabble.com/ so there may be a link there. I cant say that I know of any lake scientsts, although some researches in this department are looking at NAO related
processes in aquatic systems?
Almost the same question is asked and answered at http://stats.stackexchange.com/questions/14483/intuitive-explanation-for-density-of-transformed-variable.  Is that perhaps a duplicate?
What you should focus on is the drawing-tickets-from-a-box metaphor.  In its terms, your question is this: I drew a bunch of tickets from a box according to a certain procedure.  Then I randomly chose among those I just drew.  Is this the same as if I had originally chosen a smaller number of tickets?  The answer might depend on the procedures used to draw tickets, but in some cases it should be clear. To get started, you ought to think about the following: for any given ticket in the box, what is its chance of winding up in the ultimate (second) sample?
I guess what I am wondering is if it would be appropriate to check the distribution of the responses per site and possibly exclude some sites as outliers due to low responses (by each region I suppose).
definitely not a homework question. done with school a long time ago :) Thanks, I hadforgotten about lasso.  Some of the features are clearly uncorrelated with the target variable, so its quite clear which are unimportant, it was just "at the same time" thing that threw me.  Thanks again!
I'm not sure if the mode is technically defined this way, but the global mode of a continuous distribution is usually taken to mean the point with the highest density.
@Daniel, Sorry about that. There are some results for "learning under the uniform distribution." Is this more like what you're looking for? Based on your edit, I think you might be conflating the distribution and the hypothesis/concept class.
Even if I assign a very small value, I might have a very large range. As for the quantiled version, it would completely erase any sense of relative contrast, a bit too strong for my taste as indicated in the question.
Just fyi, the paper's definition of 'classical' measurement error is a little bit non-standard.  Normally (at least in what I read) 'classical' measurement error is when $X=X^*+\\epsilon$ and $E[\\epsilon | X^*]=0$ and is contrasted with 'Berkson' measurement error where $E[\\epsilon | X]=0$.
The goal of this experiment was to understand what combination of # of R, # of C and level of F3 led to the highest output while trying to minimize inputs (more R and more C = time and money)