tttjjj/elt_bench huggingface dataset:
I don't follow that. You don't compare OLS to Poisson; they are different types of models for dif types of situations / phenomena. They are *not* 2 dif models of the same thing where 1 model might be a better account than the other. You wouldn't compare a kitten & a Christmas tree to see if 1 were better. I don't quite get how you're using the phrase "marginal effect", if you mean the effect of a predictor *ignoring the effects of all other variables* (like the marginal effect of a factor in ANOVA), then exp($\\beta_1$) is the marginal multiplicative effect of $x_1$.
Just to clarify, is this the type of data: (A) Of 1000 supermarket trips on which Tide was purchased, Crest was purchased on 139. (B) Of 300 supermarket trips on which Crest was purchased, Tide was purchased on 60. If so, do you have other information (e.g. to establish whether this is a higher than lower than normal rate)?
Are $X_j$'s, $j\\neq i$, other random variables (perhaps identically distributed, but not the same)? What do you mean by "I fixed $i$"?
Hey thank you! I already knew O(n) notation but never knew the complexity of matrix multiplication / inversion. That link helped so much!
Hint: What does the _law of total probability_ say? Of the five quantities on the two sides of the equality, how many do you know (or can compute from the given information)?
That's usually the situation, but exceptions can exist as your example shows. The point wasn't that they can't be similar, it's that the graph *of itself* doesn't make them similar.
But the CLT only tells you about behavior as $n\\rightarrow\\infty$. 30 is very often not sufficiently close to infinity. By way of example, see the discussion [here](http://stats.stackexchange.com/questions/81074/how-useful-is-the-clt-in-applications/81087#81087).
@Kenny That's a good point and I think the way you've stated it is correct - but I feel like many of the most highly-voted questions and answers are often least indicative of expertise, whereas good answers to really gritty technical questions often have very low votes.
Yes, you are right - I would like to test the users response to the statements as the perception of the improvement. Is there any other tests that I can use? Which test to use to to know whether there is a relationship between the statements and the age, gender, dept the users belongs to?
I'm actually unclear as to how a likelihood ratio (LR) does not achieve everything that an effect size achieves, while also employing an easily interpretable scale (the data contains X times more evidence for Y than for Z). An effect size is usually just some form of ratio of explained to unexplained variability, and (in the nested case) the LR is the ratio of unexplained variability between a model that has an effect and one that doesn't. Shouldn't there at least be a strong correlation between effect size and LR, and if so, what is lost by moving to the likelihood ratio scale?
Not an answer to your main question, but something for readers to note: unless you make the actual type I error equivalent for the two procedures, comparing power doesn't make sense; I can always make the power higher for the lower one (in this case the take logs and fit normal one) by lifting its type I error. On the other hand, if you specify the particular situation (sample size, abundance), you can get the type I error rate (e.g. by simulation), and so work out what nominal rate to test at to achieve the desired type I error rate, so their power becomes comparable.
The formula is given by the Normal cumulative density function, defined as $\\int_{-\\infty}^z \\exp(-x^2/2)dx /\\sqrt{2\\pi}.$ There are many ways to compute this as exactly as one would like: conceptually it's no different, say, than the definition of the logarithm as $\\log(z)=\\int_1^z dx/x.$ This is why @Glen_b seeks more information concerning your objectives, which you can provide by editing this question.
@MichaelMayer I see what you mean by perform better now. My definition is different (and based on my assumptions noted above), performance is probably being determined by cost-benefit. It is not good performance to predict complications if they will not occur. But the cost is not so high relative to the benefit that some false positives (low specificity) would be a problem. Wit this definition the use of the imaging to predict may still perform better than the "stupid predictor". I think we need to know more about the problem to say.
(+1) The **inverse** of the negative hessian is an estimator of the asymptotic covariance matrix - I know this appears in your code but I think it's important to point out.
Yes, of course. For disambiguation, it is always wise to say "coming from uniform/normal" distribution instead of dubious "having uniform/normal" distribution.
@crippledlambda: yes. They apply to all kinds of predictive models. Regression (calibration in chemistry/physics terminology) however is better behaved than classification: It is easier there to get stable models, and also the validation results tend to have less variance uncertainty.
This is along the lines of what I have been thinking. I am considering running a separate  multiple regression model for each subject using all of their pre-treatment data, then putting the result in as covariates in the ANCOVA. But, this seemed a lot like a multilevel model to me. I just can't think of how to structure the model.
Regarding the nature of the data: The data come from crop storage facilities where  the annual harvest causes a cyclic spike in December with usage during the year till the next harvest. Observation are quarterly but the cycles are annual.
Possibly helpful papers [1](http://www.ncbi.nlm.nih.gov/pubmed/21422059) [2](http://www.ncbi.nlm.nih.gov/pubmed/16011706)
A derivation is sketched at http://stats.stackexchange.com/questions/72262/how-to-average-variables-having-given-standard-deviations.
Yes, the uniform prior is still over `sigma`, not `log(sigma)`. Stan accomplishes this by transforming the parameter and applying the correct change of variables adjustment by using the Jacobian.
As I have understood the question, the typos do not occur in labels (response variable), but only in the predictor variable aka terms. Is this correct ?
Yes. The coefficient reflects the change in log odds for each increment of change in the ordinal predictor. This (very common) model specification assumes the the predictor has a linear impact across its increments. To test the assumption, you can compare a model in which you use the ordinal variable as a single predictor to one in which you discretize the responses and treat them as multiple predictors (as you would if the variable were nominal); if the latter model doesn't result in a significantly better fit, then treating each increment as having a linear effect is reasonable.
(Sorry, just realized that comments are limited to 600 characters)

@Jeromy (part 1/3): You wrote: "There are connections between the set of studies and the set of publications and presentations. Should they all be in one repository, or would you split them up in some way (e.g., one repository per publication)?" When you write "split them up in some way", do you mean sth like forking? And "studies" are datasets? As I wrote before, each publication, each presentation etc. has its own repository.
Sorry, bu I don't really understand your answer. What do you mean by "In detail, each tree is build on a sample of objects drawn with replacement from the original set"

 Can you give more precision on where I find the details "here"?
What exactly is "cross-reference?" Correlation? Group differences? And what the five demographics? Are you going to perform 5*39 = 195 tests? Or are you planning to lump the 39 up in a certain way? And lastly, what is the reason of putting "Neutral" at the last? That does not sound like [Likert's scale](http://en.wikipedia.org/wiki/Likert_scale) to me. And sorry for machine gunning you with questions, word limit is to blame. I am a lot nicer when I am not typing comments.
This gives some nice conceptual hints. In the second paragraph there is a lot of focus on unbiasedness, but an important caveat is missing. Unless (a) the linear model is "correct" (and, when is it?) and (b) *all* relevant predictors are included in the model, the coefficient estimates will still be biased, in general.
I fully agree with you.  An observational study may be good to uncover some associations that in turn one can test using a much more rigorous framework (randomized trial as you suggest).
Wow, duh. Now I see it. Thanks... long day.
My apologies, @guillefix. I read this as a comment on the other answer. If you meant this as a standalone answer to your question, it is appropriate in this format.
Because $a_1$ occurs in another term.
(+1) This answer should also make it clear why the hint I provided is true. By the way, you had some unconventional, but remarkably clever, ways of typesetting the math. I've made it a bit more conventional. You might look at the edits to see what was changed.
As FA might not necessarily be the best choice, depending on the question you are interested in, could you say more on the context of your study?
@Anand, A [standard method](http://en.wikipedia.org/wiki/Dirichlet_distribution#Related_distributions) for sampling from a general $\\mathrm{Dir}(\\alpha_1,\\ldots,\\alpha_d)$ distribution is to generate independent $X_i \\sim \\Gamma(\\alpha_i,1)$ random variables and then take $(X_1/S, \\ldots, X_d/S)$ where $S = \\sum_{i=1}^d X_i$. In the case where $\\alpha_1 = \\cdots = \\alpha_d = 1$, then the $X_i$ are iid $\\mathrm{Exp}(1)$ random variables since a $\\Gamma(1,1)$ variate is equivalent to an $\\mathrm{Exp}(1)$ one.
How are you calculating chi-square values on the runs?
What about when we wish to test for cointegration using Engle/Granger 2-step? http://en.wikipedia.org/wiki/Cointegration
Thanks for the answer. Can you please also provide with links to relevant literature?
Odd. I found Kruschke's book very enlightening. In fact, I tried several other books, and kept returning to that one to figure out what the others were saying.
thank you also sir for answering my question.:)
Exact duplicate, same asker :P
I think you may need to expand on your question to get a useful answer. What do these numbers represent? How many samples do you have? What is the fundamental research question you are trying to answer?
@Ryan I imagine 1000 observations would tell you *something* about the data distribution :-).
The claim is wrong: the "usual" $R^2$ increases, but this is not true for the adjusted $R^2$.
Inquest, I don't think @Zen intended to sound threatening with that question. I take it as an abbreviated way of asking, "what is *your* understanding of the meaning of $\\mathbf{E}[X_2|X_2]$?". Your clarification of this will establish a starting point for answers that address *your* intuition and what *you* would like to see as a proof.
(+1) Never used this package, but your overview suggest I shoud try it. Seems good at first sight.
(ctd)... If your alternative of interest were more general (some kind of location difference rather than specifically of means), you might consider a Wilcoxon-Mann-Whitney test, which has similar power to the t-test at the normal.
Something is meaningful in the data. Quite often it's obvious what that must be. Doing a post hoc to demonstrate the obvious just demonstrates you don't know what the ANOVA does in the first place.
@Glen_b It's the MASS package, so tax if the "full-value property-tax rate per $10,000"
Could you flesh out some of the essential details? In particular, to what matrix do you refer and how would it be computed from the kinds of data described in the question?
Depends how you define improvement. Can you provide an example you have encountered?
Geary C helps you see how values close to one another cluster, even values in the middle of the distribution. Moran's I helps you see how very high values cluster with very high values, and very low values cluster around very low values. So perhaps you are correct and the simplest and best method is Geary's C. Remember the Geary's C approach is exploratory and will not let you condition your results on other factors. Look at this python module for code to run Geary's C: http://pysal.org/1.1/library/esda/geary.html.
But is a "sigma" the standard deviation of *the data* or is it the standard deviation of the *sample distribution of the median*?  One will be on the order of $n^{-1/2}$ times the other, which could be an enormous difference.  Instead of trying to tell us the answer, perhaps you could ask the question that is motivating you: *how do you intend people to interpret the results of your calculation*?
Yes the scaling information bit makes a lot of sense. Sorry I'm still a beginner, didn't quite get the rest of your answer. But thanks for your response!
Do you mean all intervals contain the true mean ? If the samples are independent then obviously the probability that the three intervals contain the mean is $0.99^3$.
Sorry, I did not have a chance to connect until now. Perhaps what KFS returns depend on the values you pass to filtering= and smoothing= . In the mean time, I see that @SteinarV has provided an alternative way to obtain the matrices you are after.
Wayne, thanks for the response. The model was converging and I ruled that out but I'm interested to know what you mean when you say the lm regression didn't have the most significant results. Are you possibly referring to the F-statistic and/or t-values?
+1 Thank you for your help! :) Appreciate it
For implementing this in R, Jim Albert's book "Bayesian computation with R" is also useful.
Remember the Poisson is discrete, so the quantile function can't return exactly $k:\\Pr(\\sum Y\\leq k)= 0.05$. What it does return is in the manual.
Right, except I know how the data was generated. It was a simulation of the inputs using normal distributions, a mapping to the class probabilities using the binomial distribution with known linear weights for the linear predictor. I then simulated each class using the class probabilities. The models were trained, one with the correct parametric form, another with one of the input variables missing. The scoring rule gave preference to the model with the missing variable. The calibration plot showed that the model trained with all inputs was more accurately estimating class probs. I used LOOCV.
Two thoughts:
1) Have you tried a plot of log(width)? 
2) If width is time to collect 100 things, then it could be influenced a lot by arrival of 1 thing, couldn't it?
@gung Gamma with shape parameter 0.01 (sample size was 30, simple linear regression fitted); fairly similar results occur with other highly skew distributions. You don't need it nearly that skew to make the distribution look distinctly different from $t$.
Is your question about which model is correct, or is it why you get NaN?
I wouldn't say 50% survived 28 days or more, since you don't know that for the very reason of uncertainty during the interval you pointed out. You can say that 50% survived to Day 16. Regarding the issue of interval, you bring up a very real issue that has to do with data imprecision. Using a midpoint imputation method makes sense, but the widely accepted approach for your situation is the [Kaplan Meier estimation](http://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator).
@Max, No, the tables are not relabeled. They retain their original tables throughout the tournament. Also, the program seems to automatically re-balance tables so that all tables have about the same (maximal) number of people, and this process is how certain tables get removed. For example, if there were 3 tables of size 4,5,9 then the first two would be combined into one table, randomly selecting which one of the two remains. If tables got relabeled throughout the tournament, I can see that one would simply need to look at the max number.
Thank you! I have read through the first tutorial and was wondering, since I have physical data, do I need to model the data as distributions to determine the prior and likelihood or can I use my data, e.g. p(fraud)=no fraudulent transactions/no transactions?
@robin No, on the contrary. I like the idea of making a distinction between statistical and human decision. My remark was mainly about this, and your last two points are clearly relevant: what is a good decision, and how to characterize any departure from it. (K&T highlighted a lot of judgment biases, but this was really a sidenote.) I'll try to develop this view later.
Could you use a survey sampling approach, like in polling where the researchers don't have access to the entire population?
Yes, this is great!  I ran a plot using the example provided on both the value and the log, and got a graphical decomposition of the distributions in question!!!!
Write $Z = \\frac{X}{\\sigma/2}$... or equivalently $X=\\frac{\\sigma}{2}\\cdot Z$. Can you do it now?
How can you "see the answer" if you can't compute the integral? Hint: Just use $Y_2 - Y_1 = |X_2 - X_1|$ where the $X_i$ form the underlying sample. (Why?) Now use properties of joint normality and compute the resulting one-dimensional integral.
not what you're asking, but you might want to try maximum margin classifiers, like SVMs, as they often have a nice resistance to over-fitting in high dimensional problems.
If the sample standard deviation was 0, what would the sample look like?
The same question here http://math.stackexchange.com/questions/14387/normal-distribution-probability
Yes the problem is that we would need to constraint $\\vec r$ appropriately. Originally, the statement would be $\\vec y = a\\vec x + (1-a)\\vec r$. But that does not help. I have to think about $\\vec r$. Thank you for your comments so far!
I like the Muenchen book a lot
Wouldn't you need a $b$ parameter in the function like $y = a + b \\cdot e^{-kt}$? Where $b=70° - a$.
`sem` is just a wrapper for `lavaan` with certain defaults. These can be specified manually for the same effect, and doing so might be wise to make sure the right settings are chosen.
I think it is weird to use squared error and sigmoidal outputs. Where did you get the idea?
@probabilityislogic: I chose $1$ and $n$, rather than (say) $1$ and $2$, for convenience of notation; otherwise the naming of the variables has no significance beyond their enumeration. I didn't posit any relationship between $X_1$ and $X_n$ with the other variables excluded. However, testing this might be a good simple way to get insight into whether we are implementing the software correctly---thanks for the suggestion!
@gung the median is the x-value corresponding to 50% of the way through the CDF on the y-axis. The midrange is 50% of the way through the CDF on the x-axis. Either way, they don't usually coincide with the mean. But perhaps I misunderstood your intent.
What about small faces ;-)
The formulas for moments of mixture distributions are given in the statement of the question at http://stats.stackexchange.com/questions/54645/properties-of-moment-generating-functions.  That substantially answers this one, too, given that skewnesses are rational combinations of moments as shown in the formula for $\\gamma_1$.
I think it is possible to improve the answer to be more specific about how to actually calculate the perplexity on the test set.
(+1) You are nicely echoing the discussion on Medstats about the need to keep a record of data edit and analysis (http://j.mp/dAyGGY)! Thx.
Thankyou Glen_b
Hi Sandipan, actually in the second order term the $2!$ in the denominator is cancelled by the fact that the second derivative of $1/x$ is $2 / x^3$.
This question has already been answered http://stackoverflow.com/questions/15436367/svm-scaling-input-values
Try this:  http://amzn.com/B005K6EGLS.  I found it to be relevant for systems both small and large.
I'd bet on that.
@mbq: Your comment is misleasing. R lazily evaluates variables in a function definition but "laziness" is not normal behavior. The function delayedAssign() exists to tell the interpreter to be lazy with a variable's assignment, but the interpreter will make the evaluation once any data structure points to that variable, whether it needs to be evaluated or not. Further, the commercial R company Revolution Analytics had to create an iterator object to support their marketing for using R in "big data" analysis.
+1 Nice synopsis of the key points.
At the *very* least, it's important to recognize and understand the concept of [aliasing](http://en.wikipedia.org/wiki/Aliasing).
@kjetilbhalvorsen Thank you! I added a few details to the post. Since I have no experience with bootstrapping I would like to use the delta method. It looks like I would need the covariance between $\\bar{y}$ and $\\hat{\\beta_2}$ and $\\hat{\\beta_3}$, respectively. Do you have any suggestion how to compute this?
You can ensure positive predictions by using a generalized linear model with logarithmic link function. By the way, although your $R^2$ value is quite encouraging, a better check of whether the model follows the main shape of the data is a plot of residual vs predicted. Plots of observed vs predicted may also help illuminate your problem.
Thanks for your suggestion, I will certainly have a look at `changepoint`. I need to smoothen the function, but not necessarily find a formal dependence y=f(x), as all I need to be able to do is get back to each of these discrete time points and find a smoothened readout for it. In a slightly different setting, where the period of saturation was shorter (or at least, not so much longer) than that of monotonous change, I used loess regression without a problem, but here it doesn't seem to work that well.
The solution, although leading to a correct answer, is *prima facie* nonsense because none of the integrals is syntactically well formed. They are missing a "$dv$" term. Although this might seem picky, it gets to the heart of the question, which comes down to asking why $\\frac{1}{2}dv$ (for $1\\le v\\le 3$) would be the density of a Uniform$[1,3]$ distribution. Without the $dv$ term this is not a density at all, so it is understandable why a careful reader would be confused.
I am just looking at logistic regression. Is risk basically just odds ratio?
[This](http://en.wikipedia.org/wiki/Orthogonality#Statistics.2C_econometrics.2C_and_economics) might be of some value. Orthogonal variables are perpendicular. 'Uncorrelated' means that the corresponding centered variables are perpendicular. Under certain conditions (such as if the two variables were already centered), the two are the same.
The sampling distribution of the standard error (of the mean) is *not* normally distributed. Are you perhaps thinking of the sample distribution of the mean itself?  Please edit your question to clarify this.
How does this differ from http://stats.stackexchange.com/questions/55444/multivariate-regression-in-stata
Do you actually have distributions of score data? Or just percentiles for yourself and your friend? Score distributions often don't follow normal distributions; take rep score here for instance.
Exactly what operation are you performing when you "rescale" an $m_i \\times n_i$ matrix into a $p \\times q$ matrix?
I would expect L to be somewhere around 5000 to 10000 and M to be between 30 and 300.  Even an order of magnitude estimate is probably good enough.
To expand on Jeremy's answer- either you match the other variable, or you build a model eg linear regression, where you have to decide  if height is a linear function of age, or more complicated. then you can look at the differnces in your groups having subtracted the predicted value based on the irrelevant variables..
While the axioms of probability must indeed hold, I am not sure what interpretation to place on the word "all" in the statement "when added over **all** events of $A$ given $B$, $\\sum_A P(A|B) = 1$".  The sum of the probabilities (whether conditional or nor) of all $2^m$ events defined on a discrete sample space is $2^{m-1}$ since the events can be paired up into $2^{m-1}$ pairs $\\{A, A^c\\}$ of complementary probabilities. What _would_ make a more appropriate statement is that the sum of the probabilities of all the events comprising a _partition_ of the sample space $\\Omega$ is $1$.