tttjjj/elt_bench huggingface dataset:
Incidentally... it just occurred to me that taking the log(Y) transformation as dependent is still equivalent to finding a non-linear relationship... the log(Y) is nicer wrt the residuals than the hinge function I played with in my answer... but one of the conclusions is similar: the relationship between $Y$ and $X$ has better functional expressions than $Y=a+bX$.
Not quite clear what you mean by `variables significance`. Significance of canonical correlations?
To summarize DartPravteer's response: there are no cheap and easy options.  A couple of other options are: Q (http://www.q-researchsoftware.com/) is probably the easiest of the packages to use if creating latent class and mixed logit models (but it does neither design nor simulation) and there is a hosted conjoint service at Survey Analytics (http://www.surveyanalytics.com/conjoint/), but I do not know much about it.
Did you mean $\\int_0^1 \\int_0^{(1-y)} 2xy \\,dx \\,dy\\,\\,$?
About the chronographs' +/- 4%: A friend and I setup our two chronographs in tandem so that a single bullet fired would be measured by both. In the shade, his would measure a consistent 2% lower than mine: 940.0 fps on mine would record about 921.2 fps on his. When we moved the chronos to brighter lighting conditions, his would consistently record about 1.5% faster than mine, but both of our chronos were also recording about 1% higher than in the shade.
Thank you so much for describing how rms does this.
(+1) @EpiGrad: Why are you concerned about the CI (i.e. about the standard error) of the predictions from the fixed-effect part of your model?
Shalizi likes to be provocative... his argument appears to me to be essentially the same as the creationist argument that evolution violates the second law of thermodynamics because "later" organisms are more complex, in an organized way, than "earlier" organisms, but the second law says entropy is nondecreasing.  However, 1) there's nothing in the second law that prevents local decreases in entropy, and 2) the argument implies no-one can learn anything about anything, ever (why should learning via Bayesian updating be any different than any other learning process?)
It appears something is wrong. As far as I know, the EM is guarented to improve the likelihood, and never decrease it. This is reiterated in the following document which I found quickly online [here](http://www.seanborman.com/publications/EM_algorithm.pdf). I suspect there is a bug in your code.
1.) The box size depends on your application, what you define as a reasonable region which captures information. For example: if working on a database for face detection, you could pick an area about the size in pixels of the eyes. 2.) A pre-processing step would be to add a "mirror" boundary around the image. So voxels on the edge do suffer from redundancy, but essentially they are handled in the same way as all other voxels
@gung ummmm? They are measured on the same scale, but represent different things. They are neither highly correlated nor uncorrelated.
The Joy of Copulas is a pretty good place to start. There are also several questions and answers that discuss some aspects of them here. The main thing to realize is than "copula" is just a fancy word for "multivariate distribution on the unit hypercube with uniform marginal distributions".  It's also faster to say.
GK89 - I'll start you off with adding markup, but you should definitely define all your terms. You seek a confidence limit -- but for which quantity? Are you asking for an interval under an assumption of normality, or were you hoping for the existence of some distribution-free limit?
@onestop I said it is nonsence in a binary classification problem. I assume that a binary classification problem is a problem where you need to predict a class that can be zero or one (binary) according to explanatory variables. If the aim is not to identify the relation (not necessarily linear) between covariates and binary outcome, I am wrong about my definition of binary classification. It is not because an algorithm is "popular" that it is a good idea to use it without thinking if it suites to your problem.
+1 for an answer that's descriptive rather than prescriptive. I actually agree with the other answers that Gaussian is preferable, no matter what field, but it's informative to start from the context of what's widespread in existing usage.
@Aniko gives a [nice answer](http://stats.stackexchange.com/questions/11315/how-does-the-distribution-of-the-error-term-affect-the-distribution-of-the-respon/11316#11316) in response to your previous question.
@ttnphns I don't expect similar results in term of tree, but I expect similar results in term of group appartenance to a same cluster, once tree are cutted (i.e. composition of clusters irrespectively of the hierarchy between cluster). I expect that since the only dependent variable in the MRT is the factor group and consequently terminal leaves are clusters of one or more groups (replicates are not mixed among MRT leaves). If I am not on the right track, what would you suggest for clustering (not necessarily hierachicaly) groups while keeping informations about replicates ? Thanks in advance
@ttnphns: Pretty much every computer language comes with a random number generator.  They are overwhelmingly uniform generators on some finite domain.
@ttnphn, corrected. Thank You.
Your explanation of the problem is not very clear. Please show the data themselves, maybe excerpt from the data.
Good point @DimitriyV.Masterov. In my case I will use the spending in concurrent goods to estimate the probability that the user purchased the good. I hope this will work.
This was the right answer for my particular situation, even though Glen_b's response helped me learn more.  So more upvotes for him, accepted answer for Srikant. Everybody wins - thanks all.
+1, did not know about curve :) Always used plot directly.
Peter: I agree. That is why I have always used "Gaussian". But a comment from a reviewer on the new (concise) edition, strongly pushed "normal".
have a look [here:](http://stats.stackexchange.com/questions/45803/logistic-regression-in-r-resulted-in-hauck-donner-phenomenon-now-what/45831#45831)
Yeah, I guess it's something I can't get around. It's just really annoying for this analysis. Thanks for your input.
Crosspost: http://math.stackexchange.com/questions/265938/how-many-parameters-does-a-hm-svm-require
reproducible example please? http://tinyurl.com/reproducible-000 ...
@jushusted When you have mutually exclusive probabilities, you can add them together. You multiply together independent probabilities.  So far as formulas that go to infinity, a good approach is to write down the formula for 2 items, then for 3 items, then for 4 items, then for 5 items, etc, and see if you can spot a pattern.
It was designed like this so we could measure 2 subjects at the same time (eg. during Alevel 1: subject1 gets Blevel1 and during the same instance subject2 gets Blevel2; four runs are required to go through all levels of A).

So it was mostly convenience, ie. saving time. In retrospect this might not have been the best way. But this is the data we have...
Hence $\\beta_0$ has no meaningful *independent* interpretation in your situation. That is often the case. It is still an integral part of the model. If you dropped it from the model, the rest of the model (eg, the estimate of $\\hat\\beta_1$) would be biased.
@AlecosPapadopoulos I'm not too concerned that $20 10 months from now is worth less than $20 now. I'm looking for a rough estimate so we can determine how much we can spend to acquire the customer.
@Alexander, If you are willing to make those assumptions then the R code is fine and Kruskal Wallis would also be fine.  But then if you are willing to make those assumptions then `t.test` and `aov` would probably be fine as well.
Hi Glen! Thanks for the fast answer. Do you know how to apply this in the r-code that I pasted i my question? Thanks!
Great this is exactly what I was looking for. Thanks!!
As I said, I get a series of errors.
I am stil not sure, should we standardized y or x? Would be helpful if somebody can answer.
I think Glen is entirely correct and the problem is your definition of denom. You either need to spell it out since you have just 3 elements anyway to exp(p[1])+exp(p[2])+exp(p[3]) or as a more general solution use a loop to define a additional vector like exponential.of.p[i] <- exp(p[i]) and the sum over that.
I've got the point as well. Thanks for this :)
First, obtain a sample of $\\psi$ the way I explained earlyer, then if you work in some statistical software, like R, then you could use predifined function like  hist(psi). Otherwise just divide the interval [min(psi),max(psi)] into some number of subintervals and count how many values from the sample of $\\psi$ falls into each interval and then devide each number by the size of your sample $\\psi$. And you'll get your approximation of posterior density of $\\psi$
Thanks mzuba, So is it correct to write logit for category 2 as(considering intercept as "ß0", hlen coefficient as "ß1",iplen coefficient as "ß2" and so on....): logit(p)=ß0+ß1(hlen)+ß3(iplen)+ß4(ipttl)-ß5(iptype)-ß6(tcpsport)-ß7(tcpdport)-ß8(tcpsec)-ß9(tcpack)-ß10(tcpwindow)+ß11(tcpchksum)+ß12(date_time) The values of ß1,ß2,ß3...so on, I can get from above output?
Bradsher, you did understand what I would. I could explore both approaches (possibly in R... ;.) )
@ttnphns: thank you for suggesting question improvements which are now reflected in the question.
This question is clearly related to [Statistical analysis for rubric-based research](http://stats.stackexchange.com/q/30490/930),  although it asks for a different test statistic.
+1 Good reference; thank you.
I agree.  If you want a simple answer, just use the median.  All this is way way over my head but if the problem really does boil down to outliers pulling the estimate, then just use the median which is not sensitive to outliers.
Thats true. Thanks Mr. Flom. That helps.
Thank you for your response. There is a strong correlation between my data points such that the correlation matrix (`cor(X')` has a lot of elements larger than 0.9. I set any value greater 0,9 to zero to make the matrix sparse. Do you agree with that?
What's wrong with the answer you received on Stats.SE? You seem to have not accepted that answer, and are now opening a bounty on this one. It would help if you edited the question to specify what more you want here.
Hi Urukec. You are encouraged to answer your own question. This website is not just for getting immediate help but is also intended to be an encyclopedic record for people who may encounter the same problem sometime in the future.
A shift of the median is not a shift of the distributions without extra assumptions, which you can't make for your problem without inconsistency. You haven't addressed my argument that the ranks of the logs contain the same information as the ranks of the values.
Thanks! This sounds great. I'll give it a go
There is no way to know, without context. If the context doesn't tell you, then that is a serious flaw in the article.
There are *huge* assumptions lurking in this question: namely, that (a) residuals ought to be normally distributed and (b) ML really is the right procedure to use.  What precisely do you mean by "irregular data sets"?
@Jeffrey04 No worries; hopefully between @suncoolsu's answer and mine it's clearer.
It's actually an exponential with rate parameter $\\frac{1}{2}$.
(+1) Very interesting. Your post also made me insert a comment in my answer.
OK, so, the question of whether you need four samples hinges on whether "employability" is likely to be different in these countries. Doubtless you know more about how to answer this than I do.
Of course, use the distributions package in julia: [user-defined distributions](http://distributionsjl.readthedocs.org/en/latest/extends.html)
I fear this question may not be well-posed (yet). For example, some order statistics may have well-defined means, even when the mean (or variance) of the distribution is undefined. Second, the parameterization of a particular family of distributions may not be closely related to the variance. Are you, perhaps, interested in some restricted class of distributions, such as *[location-scale families](http://en.wikipedia.org/wiki/Location-scale_family)* or similar? More details would be welcome.
paper modelling probability of observation in a completely other context (whale counting!) is  http://folk.uio.no/tores/Publications_files/Waagepetersen_Schweder.pdf
@Gong-YiLiao q-value has something to do with FDR, and nothing to do with (1-p)
It makes no difference whether you teat them as categorical or numeric. (To make them categorical in R, you would `as.factor` them, and then the dummy for the second level of the factor would replicate the original 0-1 variable.)
Yes: see volume 1.
The link is interesting, thanks (+1). Your idea to somehow `mimic PCA by Star Coordinates` is so far too vague to pose a real problem (so a question could be asked). One might advice you to _try doing_ it first, and then - if you get problems or doubts - ask a question.
Huh? Node 11 is over 10, Node 18 is 10 to 15, node 19 is over 15, just like the labels on the nodes say.
This is closely related to a generalized [birthday problem](http://stats.stackexchange.com/search?q=birthday): $N$ would be days of the year, the $D$ independent "markings" would be draws from a population of people, and the chance of $C$ "clean" balls is the chance that among those $D$ people there are $N-C$ unique birthdays.  As an answer, do you seek a closed formula, an efficient algorithm, or an asymptotic formula (in $N$ or $D$)?
I am interested in each measure independently, so running the analysis with each measure alone would be a good solution. I wasn't sure if that had implications for how I interpreted the data post-hoc (i.e., correcting for multiple comparisons) not because I cared about the differences between measures, but because I was running several ANOVAs. It sounds like in this case, I would not need to adjust the alpha value in evaluating the results, and actually instead of doing a repeated measures ANOVA I could just run paired sample T-tests instead. Thanks again.
@Glen_b: there is no hypothesis that one position should be preferred!
I am actually using Twitter at the moment.  It is super noisy though.
@Seb Thanks for the answer!
Sounds a lot like homework, and if so, please add the `homework` tag.
I don't know enough about time series analysis & forecasting to try to answer this, but the author of the package, @RobHyndman, is a CV contributor, & has an online forecasting textbook [here](http://otexts.com/fpp/) that may be helpful. (My first guess re 'Md' would be the median, FWIW.)
Possibly [this](http://stats.stackexchange.com/questions/41789/what-makes-a-glm-estimate-the-means-differently-from-the-actual-sample-means) question is similar to yours.
you forgot to multiply the likelihood by the prior (or you multiplied by $\\frac{1}{y}$ instead), and your likelihood has $n\\frac{y^{n-2}}{\\theta^n}$, when you have just written the density as $n\\frac{y^{n-1}}{\\theta^{n}}$.  your CI is based on $n-1$ observations, should replace $n-1\\rightarrow n$ in your answer. (posterior should be $n\\frac{y^n}{\\theta^{n+1}}$ )
@whuber 1- ...consistent...how to generate samples that are consistent? There are samples (hard data=conditioning) and stochastic generated points. 2- for the rest of your comment I'm still thinking how to implement. Why not giving me more explanation with examples as answer! I really appreciate it in advance.
I see, so I should assume individual-level parameters are drawn from overarching distributions; the fitting step is now to estimate the parameters of those distributions. Then I'd compare based on AIC...
I find the ambiguity is created by this answer but is not present in the question. As @Macro points out, the likelihood is a function *only* of the parameter. (*E.g.*, "The density $f(x_1,\\theta)\\cdots f(x_n,\\theta)$, considered for fixed $x$ as a function of $\\theta$, is called the *likelihood function*: E. L. Lehmann, *Theory of Point Estimation*, section 6.2.) Thus the question is clear. Replying, then, that the "likelihood is the joint probability density" does not clarify but confuses the issue.
thank you for the answer. I have been so fixated on using association rules (or sequence clustering) with this problem that I have not considered to try other techniques. I have searched through all the packages and haven't found anything. Will try regression.
@Glen_b Well, I specified that $\\mu \\ne 0$ and since $X$ is continuous $P(X = 0) = 0$ so I think that is enough. The delta method applies even when the mean of $X^{-1}$ does not exist. I was already pretty sure of this, but I simulated it anyways just to make sure and, indeed, the delta method works when we take $X \\sim \\mathcal U(-2, 1)$.
(+1) Good to hear from the Lisp world again. I'm too a fan of xlispstat (and appreciate that Luke Tierney is being so active in the R project).
Can you elaborate on what you really want to do (what is the model you want to fit) ? If you don't give a real question there won't be a real answer.
Thought experiment: If you don't observe $X$ and $Y$ together, how can you expect to learn about their dependence relations? What if I secretly replaced your $Y$ by $1 - Y$? Or I divided $[0,1]$ into $n$ equally spaced nonoverlapping blocks, chose a random permutation, then replaced your $Y$ by looking at its value and remapping it to the corresponding relative position in the image under the permutation? What if...(ok, I'll stop there). :-) Similar questions have come up several times on this site, so a search might be helpful. Additionally, you might like to read about *copulas*. Cheers. :)
OK, well pretty much just what I said... Neyman-Pearson really were concerned with situations where you do lots and lots of tests without any real theoretical underpinnings to each one.  The Fisher viewpoint doesn't really address that issue.
Yes, it looks like it needs to be solved numerically in general. I really just needed a practical way of plotting epsilon as a function of n, and ended up using a version of Chernoff inequality for fair coin from which expression for epsilon can be obtained in closed form
Thanks for the Trellis plot idea. It works well here because I can keep the same units across all of the plots. I find that biologists can wrap their heads around this easier.
If the error is printed out, no solution was found and the given values do not represent the maximum. `glm` uses iterative weighted least squares (IWLS) to find the MLEs. Divergence means that the algorithms jumps around and after the maximum number of iterations, the current values are printed out. In your example, the algorithm converges after 4 iterations.
Ok. Thanks. I wish getting more data was as easy as saying that :) I guess that, for now, I don't have enough evidence to say that there is different between the wines, even though the Friedman test says so.
Can you reference the paper? The devil could be in the detail...
I spotted a mistake in my figures or how I was working it out (I didn't like the answer they gave me, it didn't make complete sense so I have adjusted the %'s in the answer above and posted my figures below with screenshots of my excel sheet): - assuming a constant margin: http://img802.imageshack.us/img802/8755/marginconstant.png & assuming the margin is reduced based on the % of game time elapsed: http://img593.imageshack.us/img593/2323/marginreducedbygametime.png
@zen I know that but that doesn't answer my original question
@JVM apparently I have failed to explain myself well. I added a response in the question.
Another option would be to look at other transformations of t.  Adding a quadratic term is often problematic for extrapolation, but a logarithm transform or some other method might give you a decreasing rate of decrease without the same problems.
Also I do not thing Jaynes' argument closes the debate about Bertrand paradox: there are an infinite number of ways of physically drawing lines at random, as discussed in this [post of mine](http://xianblog.wordpress.com/2011/03/17/bertrands-paradox/).
Welcome to our site!  I have taken the liberty of making your question slightly more flexible--so it is not tied entirely to the chi-squared test--in case any respondents would care to suggest other tests that might better meet your needs.
A sequence of positive autocorrelations from lag 1 occurs when there is a trend, but it occurs under other conditions as well as @wcampbell has pointed out. So on its own, autocorrelation does not tell you that a trend exists, although it can tell you that a trend does not exist.
@EpiGrad It depends.  I have bootstrapped means and CIs for meta-analyses of studies.  That's it.  The distributions are not specified.
@user1676226 well, every good about machine learning should explain the learning algorithms (with more or less coverage of all). I think you refer to a cheatsheet, see this question [Machine learning cookbook / reference card / cheatsheet?](http://stats.stackexchange.com/questions/12386/machine-learning-cookbook-reference-card-cheatsheet) or [What are the “hot algorithms” for machine learning?](http://stats.stackexchange.com/questions/17227/what-are-the-hot-algorithms-for-machine-learning)
Automatic model selection is generally a bad idea.  I wrote a lengthy answer about this to the following question: http://stats.stackexchange.com/questions/20836/algorithms-for-automatic-model-selection/20856#20856 which you may want to read.  Using the AIC (or its variants, e.g. QIC) *is* better than traditional approaches, but is *still* subject to all of the same problems.
Whatever you do in Excel does not count.  Just kidding...
I do not understand how you can find the pdf of the $\\beta_2$ (for example) if it is deterministic. Please explain yourself better
This does not look like a research level question in math.
@Erik Hmm... I've tested it with input[i] $\\in (0,2)$. I've tested it for this effect, but maybe I should do it again.
There are several things going on here. First, *of course* a relationship will look linear provided the ranges of the variables are suitably restricted. Second, the *heteroscedasticity* of the data is almost as prominent a feature as the nonlinear relationship: the scatter is greater at high volumes and low powers than it is at low volumes and high powers. Regardless, what precisely do you want to test? The linearity of the relationship across the entire range?
I'm not sure than the N/A here is actually missing - it seems like the respondent is saying 'these answers don't apply', not 'My answer is one of these but I can't discern which one.'
@PeterFlom That would be a test-of-independence for me and it may prove that over all city and sec ratio are not independent.  My objective is to prove that NYC and sex ratio relationship is somehow stronger than that in Boston.  Thanks.
The choice might also depend on the distribution/nature of the variables or on the shape of the relationship but it sounds like multiple regression would be the usual starting point.
I'm using something like an ARIMA model, in that I'm using lags of the endogenous variable and leads/lags of exogenous variables.  The problem is that, in real time, my forecast errors become predictable; for example, if an order comes early, and I don't zero-out my forecast for that order, my forecast will be too high.
The outcome of interest is a continuous numeric variable and the values range from 3000 to 50000. I'm running a glm because I want to overcome issues with heteroskedasticity without any manual transformations.
You could do this and it is "objective". Put on the other side all the comments implying that you are degrading your data unnecessarily and arbitrarily.
I don't think anyone knows what you mean by score so repeating the request makes it no clearer. Even the idea of a single SE is problematic here as confidence intervals need not be symmetric for this situation.
This is pretty similar to the result of Flounderer posted above later, btw. We both kinda assume the estimator for the second AR to be unbiased, though. This could be the caveat.