tttjjj/elt_bench huggingface dataset:
Do you want to know why one would use a one-tailed instead of a two-tailed test?
I like the way you talk about retail in terms of banditry... ;-) One thing to keep in mind is that price sensitivity (demand elasticity) will vary over the week. And if a bargain succeeds in drawing the Saturday buy-for-the-whole-week crowd, it may result in halo effects even for non-promoted items for people who do their whole week's shopping at the place that has the most attractive promotions on each particular Saturday. The whole topic is really much too big for a CrossValidated question (after all, there are people making their living doing just this - like me, for instance).
Something seems off here. Consider $Y^* \\sim N(0,1)$, and suppose we observe $Y^*=-1$. Then, $Y=0$, and $f(0,-1)=\\phi(-1) > 0$. However, the joint density you have given would assign a density of zero in this situation, since $Y \\neq Y^*$. Am I missing something?
The equation for $\\hat{\\beta}$ given above is a closed form solution. You can also use it to estimate the parameters for a simple linear regression. No iteration is necessary, which is one of the advantages of the closed for solution for OLS. If you wanted to estimate a model where it was necessary to perform an iterative optimization of the objective function, such as logistic regression, maximum likelihood optimization approaches are the most common.
@DilipSarwate: Thanks! How do you understand that a uniform distribution within a 2-dim origin-centred circle doesn't have uniform marginal distribution? Or a multivariate distribution must be supported by a set like $[a,b]^K$?
+1 - It is hard to give advice after the series has already been so processed (de-seasonalized with monthly dummies, converted to yearly percentage changes). It could also be the case that the prior transformations are neither sufficient to remove the seasonality or actually introduce the auto-correlation that is shown in the above graph. (Which if the latter is the case they are probably inappropriate to begin with.)
Another way of looking at it is that an independence test tries to test *all possible* correlations simultaneously, rather than just a specific one, hence it is likely to be less powerful.
I think it also stands to be said that sometimes people want to estimate a model of the form listed here not because they are really interested in identifying $d$ or $e$ or $c$, but because $Z$ and $W$ have some functional relationship that makes it nonsensical to interpret them separately. In that case one can estimate the usual model of $(cd)Z + (ce)W$ and then draw marginal effect plots over reasonable values of $Z$ and $W$ to visualize the effects of $Z$ and $W$ jointly.
The term $Z(n)$ is your stochastic drift term. As it is stated, you have a deterministic drift term also $a$.
It looks to me like you can't calculate these values yourself from the data in the paper, because they don't give data on all four of the SNPs for each sample (there is only data for the pooled analysis in the supplementary data). But in general, I believe you could do this by calculating h^2 (heritability) from the regression model - although I have not done this combining multiple SNPs so I'm not sure how that might change the calculation.
Multiplying is not the best idea, I think. Especially if your initial overflow is the regular one - modulo some power of 2. You will get a lot of factors that are even, so most of your lower bits will be 0. Adding the numbers together instead would already be a lot better. If you need even better randomness, use [some sort of hash function](http://en.wikipedia.org/wiki/Hash_function) and use any bits of the result. If you want it to be difficult to guess anything about the outcome for people other than you, use a salted strong cryptographic hash function.
Don't know of any implementation, but this [section](http://books.google.dk/books?id=CHvSWpAyhdIC&pg=PA143&lpg=PA143&dq=Brant+test&source=bl&ots=WiIkYPB8kK&sig=_BLIzXuR_5JqyRYXKa9PPZK77xE&hl=da&sa=X&ei=vr-nUYixC87Kswar5YGYCQ&ved=0CFYQ6AEwBTgK#v=onepage&q=Brant%20test&f=false) from J. Scott Long's book provides a detailed description of how to compute the test that I believe you refer to.
"*Logistic regression assumes a standard distribution*" : where have you seen such a thing ?
I knew about the first two formulas and it's pretty easy to distinguish those; I hadn't seen that third formula.
Of possible interest: [Good summaries (reviews, books) on various applications of Markov chain Monte Carlo (MCMC)?](http://stats.stackexchange.com/q/32325/10525). Another popular book on MCMC methods is [Monte Carlo Statistical Methods](http://www.amazon.co.uk/Monte-Statistical-Methods-Springer-Statistics/dp/0387212396).
@cardinal 'That "limitation" has more to do with floating point representation, than R itself, though' Yes of course. Apologies to the R team :-)
See here: http://www.stata.com/support/faqs/statistics/prediction-confidence-intervals/
+1, I have pretty much the same problem. After searching the web, I found something more confusing. See this lecture notes by Rasmussen, http://videolectures.net/site/normal_dl/tag=12546/epsrcws08_rasmussen_lgp_01.pdf. Pay attention to Page 15.
Also check this post http://stats.stackexchange.com/questions/1815/recommended-books-on-experiment-design
@Bob, of course it's a different model. A continuous work hours variable states that the relationship is linear, while categorical is free of such constraint, that's why it needs more data to estimate. You could play with a linear spline, e.g. introduce two variables MAX(wh,4) and MIN(wh,4), then you'll have two slopes for high and low work hours. It's like in between the linear model and your categorical.
By bootstrapping, I mean repeatedly take n random samples with replacement from my real sample. They all have the same size as my real sample. Then for each sample I calculate the sample mean. Then with n "new" sample mean + 1 original sample mean, I plot the distribution. (It is my understanding that this is what bootstrapping means -- is there different ways to do bootstrapping?)
Although the Wikipedia article and the NIST page both reference the same paper, and give this confidence interval the same name, *they use different formulas.*  The NIST is correct that its formula will not produce negative values, but the Wikipedia formula can be negative. The NIST interval is attributed by Agresti & Coull to Wilson (1927); they call it the "score" interval. They proceed to develop an approximation for the score interval: the Wikipedia formula appears to be this approximation.
To reproduce the result in the book (estimation by least-squares), you could do the following in R: `p <- nok/nges; logit.p <- log(p/(1-p)); lm(logit.p~log(dosis))`. The linear regression gives the coefficients $\\hat{a}=-4.834$ and $\\hat{b}=3.061$, respectively, so that $\\hat{x}_{50}=4.834/3.061\\approx 1.579$. Lastly, $\\widehat{\\mathrm{LD}}_{50}=\\exp{(1.579)}\\approx 4.85$ as given in the book.
http://en.wikipedia.org/wiki/Variance#Sum_of_correlated_variables
@posdef I need it for this: http://stats.stackexchange.com/questions/50615/calculating-the-absenteeism-improvement-for-cabin-crew
Wow, this is a very impressive and usfull explanation, i have to say that i cant think of any of my real world data including $alpha$ + $beta$ $<1$ so ill stick to the explanation im used to
I came up with something like this: `def expectation(data, dist=scipy.stats.gamma): return dist(*dist.fit(data)).mean()`
You have objects described by features. Suppose you use the tanimoto coefficient. You can then make a long list of triples 'A B w' where A and B are objects, and w is the tanimoto coefficient. Such a triple represents a single edge or arc, specifying the two nodes and the edge weight. Many graph clustering software packages (such as found in igraph, networkx, or mcl) will accept this format. Most graph clustering algorithms expect a similarity (e.g. tanimoto, Pearson correlation). For very dense data it may be necessary to specify a threshold value, only allowing w larger than the threshold.
@HaoYe I realized it before, and I completely agree. It's just that I cannot migrate it myself, or can I?
1: (1) is not a confidence interval. 2: It might be incorrect to assume that the 100 values are independently identically distributed from some stable distribution when there is clear dependence in the folding. 3: Is there some reason you want a confidence interval rather than just report (1) as a way to describe the 'range' of possibilities?
Have you considered a [Q-Q plot](http://en.wikipedia.org/wiki/Q%E2%80%93Q_plot)? It will truly help you *assess* the similarity rather than merely testing the hypothesis that they are drawn from the same distribution.
Although I am sympathetic with Altman--regression methods often are more suitable than correlation in many cases--this quotation is setting up a straw man argument. In OLS regression the information produced is equivalent to that afforded by the information that goes into a correlation calculation (all first and second bivariate moments and their standard errors) and the correlation coefficient provides the same information as the regression slope. The two approaches differ somewhat in the underlying data models they assume and in their interpretation, but not in the ways claimed by Altman.
While it's true the dichotomous 2PL model is actually a special case of the graded response model (which is what is shown here) OP may be asking about mixing different classes of items, like modelling a 3PL model while also modelling graded models for some polytomous responses and perhaps a nominal model for others if the order is not known a priori. Not as many R packages do that, and of the packages you mentioned, only `mirt` can do this with the `itemtype` input to specify the respective item types.
There may be a typo somewhere, but this was intentional.  The logic is that I'm trying to slip either a 1 or a zero into the factor product based on the indicator value.  Since potential should return log probabilities, I am using the constant e for 1 and 1 for 0.
I think this approach to integrals is archaic in that it relies on Riemann's perspective, not Lebesgue's. This is of course in tune with Jaynes' anti-measure-theory stance.
Could you add to your answer a proof that $E$ and $F$ are _always_ dependent?  I believe that all that can be said is that $E$ and $F$ are _not necessarily_ independent, that is, in general, $E$ and $F$ _are_ dependent, but there may be instances where $E$ and $F$ _are_ independent.  Borrowing and editing the example given by @NRH, if $A=B=0$ instead of $A=B=1$, then $E = F = 0$ _are_ independent.  But maybe you have a proof that except for the case of constant variables $A$ and $B$, $E$ and $F$ are always dependent?
@joran Nope. I'm purely looking for any code/package that will allow me to do backcasting (which is a made up word for 'predicting/forecasting the past').
@chl: I think an exact permutation test would work fine; I implicitly applied two of them in my response.  The reason why it should work is that the *actual* test statistic itself will not be affected by ties, because none of the data in the first column is tied within anything, so that their ranks are all uniquely determined.  For the same reason, a Wilcoxon test that does *not* adjust for ties should give correct results!
Agreed, but i think t stats and standard errors should change given the change from N to NM?
See the answer written by @Aniko which provides a more detailed explanation. Notice I used today's date as the seed.
@Néstor raises a good point, but it may be hard to understand. If so, this thread: [why does a 95% CI not imply a 95% chance of containing the mean?](http://stats.stackexchange.com/questions/26450/) may help you.
I think you may want to check the probablity of $M$, which IMAO should be $P(M)=P(M \\cap C)+P(M \\cap D)+P(M \\cap E)$, from which follows that $P(M)=P(M|C)P(C)+P(M|D)P(D)+P(M|E)P(E)=.68$ so that $P(C|M)=.0882$.
Would this: http://cran.r-project.org/web/packages/repolr/repolr.pdf be your answer? I have never worked with it, but it looks as it might, pardon the pun, fit to your requirements.
Simulating what?! :-)
**Hint**: Let $\\{X_i\\}$ be a sequence of iid random variables with distribution $F$. Is the process $\\{X_i\\}$ stationary?
I don't know anything about TIC; I'm just trying to prod you into asking a better question. But people who know something can't even tell you how reliable your source if you won't reveal it.
@Sachin_ruk Whittaker's parabolic cylinder function $D()$ is connected to the "modern" $U()$ and $V()$ par.cyl. functions which in turn have some connections with modified Bessels (see ref. in my answer and also a few pages later, http://people.math.sfu.ca/~cbm/aands/page_692.htm ).
Pete: Out of the infinite number of functions that will fit the data with `R^2 >= 0.99` one would like to find the one with the best performance to complexity ratio (and out of sample fit of course). Sorry for not writing that explicity, I thought that was obvious :-)
Wow, thank you so much! That's very detailed, and very clear. I've accepted this as the answer - it's really solved the issue for me. Thanks again!
I would use your raw data points as x1 and x2.  If kstest2 returns a 1, then the null hypothesis has been rejected, meaning that samples x1 and x2 are likely drawn from different "true" distributions.
Okay, I'll explain how to do it for the tabulated cdf
+1, for excellent answer. In your second point the statement in bold should probably read: the stated model (A) is both wrong and inadequate.
It's polite to check the "accept answer" check mark for one of the answers to your questions, if you feel that it has been reasonably well answered.  (See the [faq](http://stats.stackexchange.com/faq#howtoask).)
Sure. Just know that the 'one variable at a time' approach changes the correlation between the variables. You can use bivariate plots to convince you of this. If the correlation structure is meaningful to you (presumably it is, otherwise why collect all these variables?) I wouldn't standardize each variable independently. if $D$ is a given matrix, $D^{-1/2}$ is is the square root inverse of $D$ (I’ve seen people interpret $D^{-1/2}$ as an element-wise operator on the entries of $D$ so I write explicitly what it mean, in case).
@vigneshkumarrathakumar see the edit.
@user3585509 if you like Marc's answer then upvote or accept it
Interesting, but I'm curious: why should the algorithm be different when you have more dimensions?  I've just been assuming that a color is a point in 3D space (by RGB), and measure distances accordingly.  Is there an advantage to taking it down to 1 or 2 dimensions?
In OLS regression neither the IV nor the DV has to be normally distributed, only the residuals do. So, first thing is to check that, which your last histogram does. It looks fairly normal.
@Aniko My guess is that it the scenario he uses one years results to get a prior uses the second years results for the likelihood and "runs the results through Bayes rule" to get a posterior distribution. But what inference problem is the posterior there for. The question is a bit too vague. I do think that if a complete census is obtained both times then you could look at how answers changed on individual and group responses. But there is no random sample and no inference problem. If data is complete in the first year but not in the second the information from the first could be used.
To my knowledge, there are two families of distributions related to your description: [1. Elliptical distributions](http://en.wikipedia.org/wiki/Elliptical_distribution) and [2. Spherical distributions](http://www.jstor.org/stable/25049690).
@probabilityislogic: I see. I get your point. The data is question is a response time variable. My take on your question would be that the value does make sense in a physical world but its just too unusual enough to be called a rare case. The person I talked to said he did not do anything differently. Actually, the data that I put here is just a sample from the entire data and there are some cases like this spread here and there.
Thanks, I didn't know of `MCMCmixfactanal`.
Can you provide some context to clarify your question? I'm not sure what you are asking about. Eg, you ask about "other polynomials" before introducing the referent polynomial that you don't like.
But I am doing a yes / no (true / false) about a specific class.  According to this http://en.wikipedia.org/wiki/Information_retrieval they use the term precision.  The cost to retrain the sample set is near zero.
Eventually figured out I should do an inverse Markov transition step.
@IrishStat, as you may see from my post, I am not suggesting to use the linear regression model, I just noted that it *may* have similar form of statistics (CUMSUM or whatever, since the latter are applied to residuals of the model, that clearly are time series objects) with (probably) different limiting distributions that accounts for the autocorrelation (testable) and that if you want you may make outlier (also testable) adjustments prior to further testing. It is just the only R library I know that works with structural changes.
Not necessarily; I think it is a matter of the methodology and the assumptions you wish to accept. Both tests share a significant number of similarities. Personally I feel that the AIC's Kullback–Leibler divergence-based theoretical assumptions are more sensible; eg. in contract with a standard LR test, AIC is not based on the assumption that at least one of the models compared is correct; additionally for a "small" sample size the asymptotic approximation that the LR statistic follows are not guaranteed. (AIC isn't perfect either, broadly speaking it usually over-fits.) You pick your poison.
It's true that the numerator and denominator are not integers. This has caused me some confusion, as I do not think I really have a proportion the way the term is usually used, rather I have the ratio between to physical values that, due to the physical system, should not exceed one. I'll take a look at that paper and consider modeling runoff volume rather than ratio. As far as beta regression, I'm not sure a beta function will fit the observed values, although maybe I misunderstand it.
@Vytautas or take a look at *Interpreting model matrix columns when using contr.sum*, http://j.mp/9pNFQe.
Are those pairs of coordinates indexed by time? If so, are the time intervals and origin the same?
+1 for caret.  If you want to use cross-validation to tune your nnet model, it is a very useful package.  If you want to just assess the performance of your nnet using cross validation, it it trivial to write your own loop.
Sorry I always call the variance $\\sigma^2$ too, the use of $\\sigma$ is a typo.
I think it would be nice if whoever downvoted supplied an explanatory comment.
In terms of missing values, the simplest approach is to see how many complete cases you have, and if there's any particular bias likely to be introduced by dropping all cases which are not complete (listwise deletion in some statistics packages).  Beyond that you really should repost; missing data is a big issue and a lot more complicated than editing columns.
"RCA" is so obscure I can't figure out exactly what it is from Web-accessible documents, but it seems to assume much more than given in the question.  In particular, separate independent regressions need not even have comparable error variances (or even error distributions).  So that's one more piece of information needed here: what, if anything, do all these regressions have in common?
Could you please demonstrate the "efficient manner" of your approach? Part of the challenge is to devise an algorithm to find *multiple* peaks--which in your case means finding *all* zeros of an (expensively computed) derivative, not just one zero--and to be explicit about which of these critical points you will classify as "peaks" and which ones not. Also, some support for or amplification of your claim that "the parametric approach is better when the parametric assumptions are appropriate" would be good, for as we all know, parametric assumptions are never exactly correct.
i think c(quantile(dat, 0.025),  quantile(dat, 0.975)) fits what i was looking for. thank you
Thank you for the answer, I am now trying to digest it.  In my problem I have allowed Type I error.  I also have the two distributions $P_0$ and $P_1$.  I know that TV between them (as well as KL). So, what you are saying is that TV gives a tighter lower bound on Type II error than KL does, meaning that I should use TV for my analysis if I desire as tight of a lower bound as possible?
That's correct; it's the "I" in "IMA" that makes it so.
If you have the proportions for the whole populations then there is no need to use a statistical test which tests whether or not the population proportions are different!
@Memming - VIF = Variance Inflation Factor, MC is multicollinearity.  - Hugh - please don't use acronyms unless they are really, really standard and widely known, like "BIC" or "GLM".
And one stage is often the best - by pre-specifying the entire model and conditioning on everything in the right causal pathway.  Note that the original question implicitly assumed no heterogeneity within the 2 education levels, an assumption that is quite strange.
This is a brilliant answer (and beautiful figures in your paper). Good point about regularization replacing (rather than enabling) hard-edged feature selection and different flavors of regularization (such as Lasso's enrichment for $\\{0,1\\}$ convergence). Nizza Papier und vielen Dank!
It is a bit unclear what you mean with 'prompted'. Do you mean subjects were tested four times, twice in each condition? I suggest mixed modelling in R would be best, e.g. lmer(response~gender*age*condition,random=~1|subject/condition,data) and contrasts to investigate your specific questions of interest.
@chl yeah, I am new to stats.exchange, but I think users in stackoverflow gives explanations along with R code that can be very useful thats the reason I decided to move the question. I think the community here assumes users here already know R which is not a bad thing I guess stats.exchange is for advance stat users :)
@mpiktas Right.  So to give a clear, final answer to the original question, we might say the first category is the "top (i.e., tenth) decile," the next category consists of the "eighth and ninth deciles," the third category is the "fifth through seventh deciles," and the last category contains the "lowest four deciles" (or, to be more circumspect, is "not in the top six deciles" or even "did not make it in the top 60%").
Overfitting causes the increase.  Most good methods make a holdout set, and use it to test the model, but not to update the model.  This allows detection of onset of overfit.
OK, so I was in a hurry, I fixed it a little.  You're absolutely right in practical applications.  And I originally had this written suggesting the values Jingjings wants to know don't require this bootstrapping.  But then I thought perhaps the bootstrapping was an exercise of some sort and thought I'd expand the exercise.
@qed: to sample estimates of the unknown quantities.
It sounds like you have **multiple** outcome measures, do I have that right? Also, I'm guessing that you suspect these outcomes are not independent of each other; eg, you suspect that they are all measures of the same underlying reality (ie, greater or lesser degrees of 'literacy'), is that right?
Thanks for the explanation. I had never heard of regression as being a change in R2.
I wish I were more expert on this. I don't quite follow your example w/ the heatmaps. What is it that you see that makes it obvious the CC(A) > the CC(B)? Eg, if the upper triangles were cophenetic distances & the lower triangles were original distances, & both displayed similar patterns, then I would recognize that the CC would be high, etc. W/ these I'm not sure how to make such an inference. Is it just that A will naturally give rise to better clustering & so the resulting CC will just have to end up matching well?