tttjjj/elt_bench huggingface dataset:
What function are you trying to minimize? If it is linear in the parameters, you should look into Mixed Integer LPs: http://en.wikipedia.org/wiki/Linear_programming#Integer_unknowns
@niandra82 FWIW, I also (fruitlessly) searched for the answer that Glen_b linked to. Depending on your research goals, that answer might be better suited for your purposes.
N-P = Neyman-Pearson
Thanks! Not sure if you have noticed, I added a fact to the end of the first paragraph, for the case when we can get the exact empirical pmf same as the true one.
A chi-square test does not require normality. In fact, it's almost impossible for a chi-square test of this type to involve normally distributed data.
Question refers to `corrgram` command in Stata [NB correct spelling] in which the hypothesis being tested is white noise. There is a Stata command `ac` to plot the acf in more civilised fashion. (Not an answer, clearly; just adding a little context.)
Without naming specific culprits, I think http://faculty.vassar.edu/abbaird/about/publications/pdfs/bennett_salmon.pdf deserves a mention. To prove a point about misuse of statistics in their field, they used a widely used statistical protocol to analyse the results of an fMRI scan of a dead salmon. They found "statistically significant" brain activity. http://www.statisticsdonewrong.com also makes interesting reading.
Please explain what would be "right" for your sample. Perhaps you could post a sample figure?
@article {cal12ext,
author = {Van Calster, Ben and Van Belle, Vanya and Vergouwe, Yvonne and Timmerman, Dirk and Van Huffel, Sabine and Steyerberg, Ewout W.},
title = {Extending the $c$-statistic to nominal polytomous outcomes: the Polytomous Discrimination Index},
journal = Statistics in Medicine,
volume = 31,
number = 23,
issn = {1097-0258},
url = {http://dx.doi.org/10.1002/sim.5321},
doi = {10.1002/sim.5321},
pages = {2610--2626},
keywords = {polytomous risk prediction, discrimination, model performance, c-statistic, polytomous discrimination index},
year = 2012}
Very nice analogy. Though a bit of statistics in the DoE (randomization, sample size planning) doesn't hurt, neither... And the need for interaction may explode if the overlap in knowledge (and also terminology) is too low.
@Robin Girard I sympathize with your sentiment, but in some cases much can be revealed without arriving at a consensus at all--and I think this may be such a case.  This question has the potential to reveal the different ways that people think about and approach a set of seemingly related questions, potentially shedding light on all the disciplines with which this site is engaged.
Because you're trying to predict $y$ from $x$?
This probably belongs on math.stackexchange.com
Is this homework? If so, it should have the homework tag.

And welcome to the site
Gung, thanks for the clarification.
@ayush What do I say? In general, what I say is my answer above.  In particular, in paragraph 3 sentence 1 I say that "the cause of low probabilities is an unbalanced category distribution".  For example, having the probability of a 1 (what you describe as the 'event rate') of 1.2%.  That answers your second question.
Why not just regression, with the change score as the DV?
Interesting and a good paper reference (+1).  I'll add it to my repertoire!
I've never heard a specific term for exactly that thing, but my closest guesses would be "relative margin of error" or "margin of error of X percent".
Ah I see, thank you Gavin!
Thanks Kwak. I have ordered the first book from Amazon. I could still do with a few lines to get me started though ...
As of yet not even a comment, I presumed a lot of people would have an idea about this?
Sorry... continuing...

I don't know if I understood it right.

My situation is that I'm dealing with biological experiments. I have 2 biological replicates and each one with 2 technical replicates. My goal is to statistically say if the data is valid for further analysis or not.

Should I analyse each point (i.e. 4 values) or the whole dataset (i.e. 4 * 24 values).... or, even, just the mean values?
@John : from the OP's question: "What interests us is whether there are a significant difference in weight between A and B". This test will help answer that question. Other model comparisons will answer other questions (which may require to include the interaction term).
A little bit tongue in cheek, there was an implicit quantifier: If we can start with a proper prior and get an improper posterior, for every possible sample value, then I will quit inference. ;-)
That's fantastic! A very useful reference that will be helpful for this issue I have and elsewhere.
@ssdecontrol Thank you very much... i found `exhaustive.search()` in `FSelector`. This may do what i want to do. I will try to make some example with toy data and eventually post it as one kind of solution.
BTW. The cost between connected points may also better than the cost between each point and the mean (k-means) since this would relax the requirement that the lines are horizontal (?). Only thing is that I want minimize this globally: summed over all line, summed over all connections. Therefore I think that a global greedy algorithm may give better results than a local one?
It would be useful to add which toolboxes this code comes from and any other dependencies.
The likely intent is [Fisher's method for combining p-values](http://en.wikipedia.org/wiki/Fisher%27s_method)  into a single overall p-value
I don't have the book to check, but the statement about the expected value of the sample variance strikes me as strange. Surely it should depend on the variability of $\\pi_i$'s as well.
Do you know how to use R? From there, it'll be a minute.
Could you please expand on this a bit? Many thanks.
This is the essential disagreement that I have with how you've framed your problem. While it's physically true that only a specific number of biological humans engaged in protest in some years, I view that number itself as having been drawn from God's hat, and determined by some combination of causal variables and random chance.
I think it would be ideal if you tell us your real question. Then we can give a lot better answers. The changed example also needs more clarification which will be specific to this example and will probably lead us away from your real question. I know that revealing your real question comes with the potential cost of somebody "stealing" it. But I think that is the price for getting good answers here.
@Ample It sounds like the [Ripley K-function](http://www.stat.iastate.edu/preprint/articles/2001-18.pdf) (or its relative, the L-function) would provide the information you want.
Thank you for your thoughts. I had never tried numeric integration before, but I found scipy.integrate (http://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html) and it did the job. And now I have another great took in my arsenal.
I'm used to 'CV' being used for 'coefficient of variation'; given the potential confusion, it's probably best to spell it out as 'cross-validation' the first time you use it. (Yes, I was able to work it out from your tags, but requiring readers to go tag hunting to resolve a confusion they may not even realize they have is not particularly comprehension friendly)
@Glen: Can you help me understand why not?  Isn't the probability of the data given each model the likelihood of each model?  Or do you mean that you can't compare them without taking into account the prior probability of each model?
Look at the ApplyModel parameters.  They include confidence interval.
@gui11aume, getting sharp confidence intervals is possible as long as you don't make an approximation to Gaussian (see my answer).  The problem with approximating as a Gaussian is that the approximation is only good when $n$ is very large, but that's one of the things we can't assume since we want to know how big $n$ needs to be get a good confidence interval.
Because interpolated values have a very different status than original data--they are derived; they are interdependent; they are arbitrary (being dependent on the interpolation method)--it is usually a really bad idea to include interpolated values in your database.  Interpolation is a method to *display* data and interpolation as *imputation* depends on how the imputed values will be used in an analysis; in both cases the procedure is best left for the analysis itself and not memorialized as values in the database.
Partially Observable MDPs? Yes, they can be reduced to MDPs, too. But a discrete state POMDP will be reduced to a continuous state MDP.
Absolutely.  More information is at http://biostat.mc.vanderbilt.edu/ManuscriptChecklist under the heading Inappropriate choice of measure of change.
But at which time point? Please notice that the objective of the study is to find the best indicators of the disease progression. Finding those that separate the two groups at say, the baseline, do not necessarily help with that.
I'm not sure what you mean by 'put data in an existing model'? Can you give an example of some other technique (not SVM) that allows you to do this? Take for example logistic regression; if you add new data you will get a new set of co-efficients re-trained on the existing set without reference to which data is 'new' or 'old', it's all just training data. I guess if you are using a gradient descent type solver you could save time by initialising at the previously optimised values, which will probably be close to the new solution. Is this what you mean?
To be more specific, I clearly stated we don't have enough information to tell if you could use it, but that it would probably be okay. That doesn't mean it *is* okay. Please don't use text speak here, by the way, we're not texting but attempting to unambiguously impart information. Take pity on your helpers and take the trouble to spell out "to" and "know" and "you".
It's not a duplicate in my opinion. The only thing I need to know is how to determine the degrees of freedom, but I decided to provide some background to make the question clearer which is indeed the same as the previous case. But actually I would not mind removing the old question if that makes you feel better because it was a bit vague perhaps.
just checking, is the confidence interval a 95% confidence interval or some other percent confidence interval?
No you do not omit a variable in ordinary regression just because it's orthogonal to others.  In models without an error term (e.g., Cox, logistic) omission of a variable results in distortion of $\\beta$s of included variables, and frequently in the case of Cox regression, non-proportional hazards.
That is a wonderful book (see [my review](http://www.statisticalanalysisconsulting.com/book-review-statistics-as-principled-argument-by-robert-abelson/) but I think it is best as a companion volume.
Nice explanation.  Doesn't explain people's cognitive failings, but +1 anyway.
appalling was correct - it is a mixed bag - edited volumes often are
Those are good points about the overall test performance: after achieving a correct p-value, one does need to conduct a power study. Your "prudent" advice looks good except when sample sizes are small and cannot be enlarged, for then the issue is whether we should accept that WMW has virtually no power or try to gain more power in return for committing to a parametric assumption. In at least one arena (environmental compliance), there is a strong prior belief that the underlying data are either normal or lognormal, so perhaps a Bayesian analysis could provide some guidance.
Thanks, will probably be back with a question later after reading the ref. Would give an extra +1 for the reference if I could.
A. This is not a homework question, although the motivation does come from a homework assignment... It's an assignment in AI, developing a multi-agent cooperative system, where one of the decisions each agent has to make is, whether or not to broadcast its observations to its peers. I currently try to approach this by calculating the value of each additional observation, and comparing it to the cost of broadcast. There may be better approaches to the multi-agent problem, however, I think this is an interesting question by itself.
Hi @Gung, Yes, that's correct - but it's more the cumulative trajectory i.e. for Alice she misses 2 days after 10 days, 4 days after 25 days 9 days after 40 days creating a regression slope that could use to predict her pattern for the next 200 days. Other students with other characteristics (e.g. ethnicity) may have different slopes. I'm not sure how to compare all the slopes as opposed to single points like Alice 5, Barney 7.
Please note that the approximation _never_ holds for proportions near zero and near one.
@Nick Alger: Folks have likley lost interest.  I am not upvoting it because I am not convinced it is Bayes - do the x.i in the set D (x.i,f(x.i)) refer to x.i observed in the study or randomly generated by them? If it is the first, it is Bayes but very easy to beat with simple MC with a few seconds of computing time (so it does not work fine) or its not Bayes (has not conditioned on the data).
+1 I use it often to make seeds.
It can be seen from this perspective as well. Indeed this coincidence given the fact that the two samples and population are different was surprising for me.
+1 for Bishop.  Clear development with an even level of detail.  While still good I always found Hastie et al. a little bit choppy.
Least squares fitting of order versus length: that's how you remove correlations among variables. Because you're using `Traminer`, you have `R`, so try it: `lm` is your friend here.
@Alexis Thank you for the reminder that there can be multiple interpretations here.  Until that is cleared up we should be careful to clarify what interpretation we are reacting to whenever we make recommendations.
Regression can handle categorical predictors with no problem. Most statistical packages (e.g. `R`, `SAS`) will dummy code the categorical variables for you and allow various types of parameterization (effect coding, reference coding, etc)
(+1). There's an extra $x$ in the expression for $\\phi$ in line two.
@MichaelMayer Shouldn't including those variables within the propensity score model cancel out the association if modeled correctly - that's my understanding of the whole point of propensity scores. Andrea, you could include the variables as well in a "doubly robust" analysis. See: http://aje.oxfordjournals.org/content/early/2011/03/08/aje.kwq439.full
Look up the idea of "separation"; the [UCLA web site](http://www.ats.ucla.edu/stat/mult_pkg/faq/general/complete_separation_logit_models.htm) is pretty good.
I think [this question](http://stats.stackexchange.com/questions/19576/variance-of-the-reciprocal-ii) may be of interest.
An anonymous editor contends that there is an error here:$$f(y)\\approx n^{-d/2}(2\\pi)^{d/2}|\\widetilde{\\Sigma}|^{1/2}f(y|\\widetilde{\\theta})f( \\widetilde{ \\theta})$$, citing [this pdf](http://www.cs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture16.pdf)
@whuber: What sort of assumptions do I need to supply? Would it help if I add a figure of the water pressure samples (they are actually more than 3, however I've mentioned just 3 in the description to make it more readable)? Regarding the probability of exceedance, do you have any thoughts? Thanks!
The motivation for categorical analysis is twofold:
2 is not "non-linear with the kernel trick". It is non-linear if you use a non-linear basis function. The "kernel trick" you are mentioning allows solving the problem. Also, be careful with question 4. In general, the solution will often depend on the particular implementation - there are implementations of logistic regression that use stochastic gradient descent rather than Newton-Raphson to minimise the objective function, and as such don't converge in one step to a global minimum.
@Glen_b Finding a polynomial bound for the logarithm of $f(\\theta)$ is simple: the best possible one that is valid for all $\\theta$ is $-k_1 - \\theta^2.$ That follows from three observations: (1) $f(\\theta)$ is symmetric about $0,$ (2) $\\log(f(0))=-k_1,$ and (3) $\\log(f(\\theta))$ is asymptotically $-\\theta^2.$
In notation above, for d=2, X_n is the number of heads after n coin throws, so it's X_n/sqrt(n) that approaches Normal, not X_n/n, no?
I am glad you have looked around this site. It sounds like you might now be in a position to write a more specific question that differs from the others already addressed. You can do that by editing this one and then re-opening it (or flagging it for attention if you cannot re-open it yourself).
@Nick I largely agree -- I generally use as many as I can afford to wait for (usually topping out at a million, though not always), but typically regard 1000 as a pretty clear lower bound. As a first try I often do 1K to get timing information, and then work out how many multiples of that I'm prepared to wait for the actual answer.
By concatenating I would imagine something like stacking, while you mean binding/juxtaposing. 
While in general more data is indeed a good thing, it it not necessarily the case. Particularly in your scenario where it is unclear what the data is used for. Could you please be more specific regarding what you are trying to do with the data?
Another choice is the cube root.
Could you provide more information on the context and what you would do with the weighted composite?
I have watched the Stanford video lectures (through Coursera), and I'm now reading Elements of Statistical Learning. Other than logistic regression, I don't know which classifiers give probability estimates (I might know more when I get to the end of the book :)).
Thanks a lot jsk.
Thanks for the feedback Michael.  I am finding literature but I'm afraid some of it (ok, much of it) is going over my head.  I also just noticed (and added to the post), that four of the 65 observations have non unique dates -- that is, two on the same date, two on another date.
Can you give us a specific example?
No, it means that once you have calculated the test statistic then your test statistic should follow a normal distribution.  As @Peter said your test statistic should be:  $$U^*=\\frac{U-m_u}{\\sigma_u}$$
There are lots of statistical frameworks for estimating variances, ranging from maximum likelihood estimation to complex dynamical/time-series approaches like GARCH ... Once you know what statistical framework you want to use, and have read a little bit to understand how it works, you can figure out what R's capabilities are (e.g. asking here again, or using `library(sos); findFn("whatever")` once you know what keywords you are looking for)
+1 @Kieran Thanks!  I am glad that my guess had some validity.  You have the right answer from your knowledge of the R programs involved.  so my answer is no longer needed.  Someone either did not like me guessing or thought my answer should be a comment.  For all those reasons it makes sense for me to delete it now.
And I strongly advise you to study a nonparametric statistics text before proceeding.  You would not have asked that question if you had had an introduction to nonparametric tests.
How do you plan to set one d for all main effects and contrasts simultaneously?
Momo, Thanks so much for your answer!
Thanks! I got stuck on why there would be a difference between the error got while reconstruction in the two situations mentioned before. Just forgot that there was actually no dimension reduction at all in the first case.
Their depiction of maths is horrid.
Normally, a logit model of any type has a strictly *binary* (0-1) dependent variable.  How are you even managing to carry out the calculations with a "continuous" variable?  It's possible that Matlab is just cranking through completely meaningless operations.
Can you give more description, or a label, to each of these sets of statistics. I know about PCA, but I don't know what matlab calls them.
It is not clear what you are asking. Because you are vague about the distribution, it sounds like you are looking to apply the definitions of expectation and variance--which of course have been amply illustrated and discussed in thousands of threads here. What specifically are you looking for beyond such generalities?
A *single* value will work in some practical cases, such as when the distribution is known (with certainty) to be Poisson and the observation is a large count. Not only does that allow the distribution to be fit, it also permits the likely error in the estimate of its parameter to be assessed.
What is the cost in your case? If this is monetary value, why can't you compare it in the usual way, i.e. `10$` is less than `15$`, etc?
Thank you for your comments. you're right, I corrected the mistake in the formula. I've got some problems in understanding your detailed answer....Could you please give me an example? About my formula:you told me, that you think it will work? Could you please answer one more question: How do you know when creating a confidence interval, that there should be Var(p)/n and not only Var(p) ? Is there a rule? I didn't found any literature about the CI I posted bevore,  but I should be able to justify my formula and thoughts....
The three parameter will always be a 'better fit' to the data (in the sense, say, of having a higher likelihood), since it includes the two-parameter version as a special case. Which is 'better' in a more general sense depends on your criteria for 'betterness' and the particular data you have.
@NamDoan This is a tough topic.
Have you considered replacing $Z$ by $-Z$?  Or $X$ by $-X$?
I think my problem is that I have some sales where I don't observe the block at all and I only know the store. To avoid computational issues, I only geocoded the sales data to block level rather than geocoding each order to a particular lat and lon. This makes it hard for me to define the fraction since I am not sure what the appropriate numerator is (unless I do it as a fraction of all sales). It's also hard for me to define the explanatory variables, since I am not sure what the appropriate geographic region over which to aggregate the data (unless I use the whole trade area).
Although you define $g$ specifically, you replace it with a *different* definition in your calculations: that's where things go wrong.  Regardless, there's a fundamental problem with this question: if $X$ has any chance of being negative, then $Y$ does not have a PDF.  Also, why do you keep writing "$y=0\\ldots 1$"? That isn't part of any explicit definition or assumption.
@whuber Yes, you're right: the output matrix is an approximation of the input matrix. The output matrix elements approximate the input matrix elements. I am wondering: beyond this approximation, are there other eventual meaning to this values? For example, if we order them from the highest to the lowest, can we see an hidden sense in this ranking? Thanks
Thanks Glen. That's an interesting result about the distributions actually being lighter-tailed than normal rather than heavier-tailed. So it seems that even the basic intuition behind the $t$ idea is off the mark, at least for some realistic cases.
Maybe this should be tagged as homework?
A linear model does not assume a guassian distribution for the errors. The assumption is necessary when you want to estimate confidence or prediction intervals. Without the normal assumption, you can calculate these via bootstrap methods.
When I say different datasets, I mean datasets like let's say some imaginary beer and whisky datasets. The property would be different, but close, and the variables used for modelling are the same (let's say, % of alcohol and average density). Hope this helps.
Do you wish to model underlying spatial processes, or do you wish to adjust your variance-covariance matrix estimation to account for spatial autocorrelation?
They give the number of observations, and the number of explanatory variables is (hopefully!) made clear, so you could work out the degrees of freedom, and hence the p-value. The critical value for 13 d.f. (the smallest one there if the listed variables are the only ones) is 2.16, so all but one of the coefficients that exceed 2 is actually significant.
It's during training. I want to avoid the models with fluctuations during training.