tttjjj/elt_bench huggingface dataset:
@steffen +1 but there's no need for a `for` loop here, you can contruct your dataframe inline, IMO. For the `tapply` call, use `function(x) c(mean(x),sd(x)))` and `cbind` the result as the OP asked for both statistics. Also, `ddply` from the [plyr](http://had.co.nz/plyr/) package could do this smoothly.
This is very helpful, many thanks for the clarifications and suggestions.
Great answer, Jeromy! Actually, this should become the best answer... [Here](http://stats.stackexchange.com/questions/29486/what-is-the-best-way-to-conduct-a-meta-analysis/29503#29503) are a few more papers that are missing in your list.
I have added the [tag:model-selection] tag for your convenience: linking through it will produce a large number of directly relevant threads.  Other tags worth looking at include [tag:aic]. You should eventually discover that the mathematical statement of this problem is missing two essential elements: a description of how and why the points might deviate from a theoretical curve and an indication of the cost of not getting exactly the right curve. Absent those elements, there are many different approaches that can produce different answers, showing that "best" is ill-defined.
i see but suppose we had a question is our exam, asked to caluclate like i stated in my q above...is the scenario possible or due to a stat error?
Would i assum to be associated/or not or cannot say but should proceed to a more specific test ie t test?
A change in the period will not change the overall shape of the periodogram, just the features of the underlying cycles. For example if you set `ts(tsobj, frequency = 4)` the data are quarterly, with two seasonal cycles that are completed every quarter and two quarters. 
Looking at your plot the monthly frequency seems appropriate to me. You may find further details about this [here](http://stats.stackexchange.com/questions/1207/period-detection-of-a-generic-time-series).
`is.numeric(mydata$identity)` will presumably give `TRUE` when it should give `FALSE` (you need it to be a factor). See `?factor` and `?as.factor`.
@S0rin: I would recommend starting with the `arima` function in R.  Use the `xreg` argument for your regressors.  You may also find the `auto.arima` function in the forecast package useful (this function also has the xreg argument).
I believe you can use `anova(lm(value ~ speed + variable, data=mcar.noise))` or `summary(aov(value ~ speed + variable, data=mcar.noise))`.
Your algebra overlooks the terms that are truly crucial to a careful analysis of this problem: the "errors." As such I think this answer misses a key point.
No, a backward procedure starts with a model with all covariates and then removes a covariate (whose coefficient is not significant) step by step (until you have a model with only significant coefficients, usually).
I guess there are more sophisticated ways to achieve the same goal, but i'm just a bachelor student!
Where have you read that? A succinct description of the argument or at least a specific reference would probably be useful.
Can we say that it is better to compare the deviations from baseline in such cases to negate the effect of distance to 0? As an example, say the blood pressures deviated by 30% in one group, and only 15% in the other, instead of comparing the mean of measured blood pressure levels.
Related: http://stats.stackexchange.com/questions/16390/when-to-use-generalized-estimating-equations-vs-mixed-effects-models
Can you add details: What data you simulate, how do you perform the Gibbs sampling and what the results look like?
@emakalic - just a quick note, that AIC & BIC are basically just ways of choosing which p-value to use, rather than doing something "fundamentally" different.  For AIC, we have a p-value of $0.154$ and for BIC we have a p-value equivalent to $|t|>\\sqrt{\\log(N)}$.
would it be more helpful if I sent you the full dataset? The only problem is that convergence takes about 9 hrs with the full dataset, so your suggestion regarding sampling is a good one. I tried to transform the data with a log transformation, but the binned residual plot is still ugly - do you think the residual plot explains part of the problem with these data? Finally - were your results in SAS similar to R's?
Well, you're using some courses' grades, not all of them.  And I wouldn't want you to underestimate the power of partial correlation to produce very different results than you would see in zero-order correlation.  You might even get a negative coefficient for one or two course grades.
Well, in fact, the binomial parameter $n$ is a sample size ... for the $n$ Bernoulli trials that the binomial is the sum of.
+1 I find the mobile app for Evernote to be particularly useful when running around between labs or jotting down a thought when away from a computer.
You can compute the variance directly without reshaping your data, but if you simply want to reshape the data into format c and then use the in-house functions me and Bruce Weaver recently answered a question on the google group SPSS forum, http://groups.google.com/group/comp.soft-sys.stat.spss/browse_thread/thread/3e8944a721133c3d#, that explains how you would do so. (the only difference is that you would need to do three seperate loop and xsave commands and then add the data files back together.
Hi.The script bucket one is for a very simple example...Nowhere it is specified how to seperate the neurons in a layer.Thanks anyway.
You need to distinguish between two different uses of the word "given". In a problem such as $$\\text{Given}~P(A) = 0.3, ~\\text{find} ~ P(A^c)$$ you are **not** being asked for a conditional probability. Here, _given_ is giving you information, not saying that something happened. In a problem such as $$\\text{Given that}~k~ \\text{Heads occurred on}~n~\\text{tosses of a coin, what is the probability that the first toss resulted in a Head?}$$ you **are** being asked for a conditional probability. The key is to look for verbs such as _occurred_ that tell you that _something_ happened.
Thanks, very helpful answer. This is a simplified hypothetical version of a question U discussed with a colleague. I will have to find out more about the generating process in the real situation of interest to the colleague.
If this is a self-study question please add the appropriate tag.
Can you please specify which kind of "robust" regression you are using?
@Qmechanic, I'll try. Thanks a lot.
I would consider the following analogy in this case. Suppose $Y_i$ and $X_i$ are two correlated time series (correlation in time). We would like to figure out a function $Y_i = f(X_i;\\theta)$, which is equivalent to finding $P(Y_i|X_i;\\theta)$. If $P(Y_i|X_i;\\theta)$, which is the residual, is IID (hence stationary and uncorrelated) then the estimation procedure converges without bias. Basically processing the time series in time order or any randomized order shouldn't matter in a MLE procedure as long as the conditional likelihood is correctly specified and residuals are IID.
There several other options that I'm aware of for chain diagnostics besides the Gelman-Rubin test. See [here](http://www.people.fas.harvard.edu/~plam/teaching/methods/convergence/convergence_print.pdf) for a brief overview of them. I would be cautious on using any diagnostic though, as they can be overly sensitive (especially the Geweke diagnostic). I recommend scrutinizing trace plots if you can (although this is quite risky itself if you're not experienced).
Also StasK I changed my user name just for you!
@M.A edited my answer to clarify.
I think your treatments are `wt` and `KO` and there are two responses that you observe for each replicate -- `size` and `count`. BTW, I am assuming by replicate you mean biological replicate. If I am right, it is just ANOVA with 1 treatment, but I don't understand the reason for 3 experiments.
Thanks. This is worth exploring further. Although I need to check if lmer() allows incorporating lagged dependent variables as predictors. I was hoping to start with something that is relatively simpler, like dynlm(), which I found out after I posted my question! With the plyr package, it seems I can manage to extract the coefficients of individual models into a data frame, and subsequently do the forecasting for each customer.
As I understand, in statistics bias is the *difference between estimator and estimand*, where in epidemiology, bias is the  *non-random difference between estimator and estimand*. When I see terms like 'bias' and 'systematic error' in the context of biostatistics, I tend to think of the epidemiologcial interpretation. But then again, as a student of epidemiology, I'm biased. This [set of slides](http://www.ccsem.infn.it/ef/emfcsc2008/bioelectromagnetics/Greenland_Bias_and_EMM.pdf) from Sander Greenland touches on both concepts, but focuses on epidemiology.
Maybe you should ask at StackOverflow, since it's a programming issue.
I posted the same question "http://stats.stackexchange.com/questions/47327/how-to-perform-glm-on-categorical-data-in-r"
@ttnphns I believe you, I don't know. But commonly we consider $H_0$ as the equality (see e.g. the article on wikipedia).
I was interested in binary because it was easiest to analyze. I have found a very broad sufficient condition in works of Lauritzen -- you get closed form if a corresponding log-linear model is decomposable
@Renata I'm sorry. I mistakenly thought you want to pair _two_ sets of objects (what Hungarian does). To pair within a single set, other **assignment algorithms** should exist. Or use some cluster analysis allowing constrain of cluster size.
**Except** for the special case when at least one of the two events has probability $0$, independence of events $A$ and $B$ **always** implies that the events "overlap" and their intersection $A\\cap B$ has positive probability: $P(A\\cap B) > 0$. Note that events $A$ and $B$ are independent if and only if $P(\\cap B) = P(A)P(B)$, and the right side cannot equal $0$ unless at least one of $P(A)$ and $P(B)$ is $0$. So, when whuber says "that would not be true :-)" he is thinking of the special case mentioned above when student441'a conclusion fails to hold.
This sounds like homework
Sure, you just take the Taylor expansion out to more terms. I don't think I understand the difficulty here - usually it's just tedious, not difficult.
Actually, there are two counts from each text. One count for one type of word (type A), and another for another type of word (type B).
Yes, I am considering the last cases.
Constraint clustering is available in R package rioja ran.r-project.org/web/packages/rioja/rioja.pdf
Hi, sorry I don't have time to do a proper answer but no, if you modelled the untransformed data and then plotted that it would not give you the same results.  You need to somehow extract the numeric results from plot.gam and then reverse transform them.
Is the choice not related to iid?YOur answer is not to the point, sorry to say. Why white noise and why iid is preferred?Can you please re-address these?
Hmm...because the sine has a range from $-1$ to $1$, you *ought* to find that the range of $1-\\sin(20x)/4$ is from $1-1/4$ to $1+1/4$.  That should lead you to conclude that when you use an appropriate multipler the acceptance rate is around 80% (*exactly* 80% for any pdf symmetric about 0, like a Normal), which is pretty efficient.  With a multiplier of 2 the acceptance rate goes down to 50%: not bad, but still 63% worse than the best you can do.
There are many simple solutions.  Consider what would occur when you create a matrix with columns of $F$, $|q|/(4\\pi\\epsilon_0)$, and $r^2$, for instance. But that begs a more important question: your machine learner is unlikely to find that $F=c|q|^1 r^2$; it might find that $F=c|q|^{0.97} r^{2.001},$ say. What then would it mean to force $c=1/(4\\pi\\epsilon_0)$? Not much, it would seem.
Can we see your code? It's hard to know if we don't know what you did. Can you run summary(data[varsinthemodel])?
I don't completely understand. So you have a within-subjects design, every subject experiences both conditions, is that correct? And then you repeat the experiment so that you have multiple experiments with within-subjects design?
@stacy see also http://en.wikipedia.org/wiki/Power_transform#Example for an explanation of where the horizontal line comes from.
@jsk $\\pi(\\theta)$ is not a probability distribution, but the definition of posterior distribution requires that $\\pi(\\theta)$ be a probability distribution, so it's cheating to call $\\pi(\\theta \\mid X)$ a posterior distribution when it is a probability distribution. Degroot & Schervish say '..we will compute posterior distributions that..' by which they're assuming you agreed to 'pretend that they [the improper priors] are [proper priors]' as expressed earlier in the quote.
No worries. This is just reflection, not practical.
for the example in the [tutorial](http://pymc-devs.github.com/pymc/tutorial.html), the plotting of the marginal posterior distribution for one of the parameters is shown in [section 3.5.3](http://pymc-devs.github.com/pymc/tutorial.html#sampling-output)
According to wikipedia, the sample space is defined as the set of all possible outcomes and so my first example doesn't work. I will edit my answer accordingly to give you a more satisfying example.
Or we could just delete this question since it's not really going to lead anywhere productive.
Even large and fancy firms that do projections often (always?) use simplistic approaches, never (almost never?) including drastic deviations from the past.  (This statement is based on my experience.  I'm sure others will correct me if I'm wrong about this.)  But the future sometimes does bring us unexpected deviations from the past.  So, look long and hard at the assumptions made by your model, whether you do it yourself or buy a service from a fancy consulting service.  And heed zbicyclist's wise observation, "good planning is robust enough to survive bad forecasts."
What is your definition of "statistical analyst"? Have you looked through the various other related questions on this website that discuss this? If you can edit your question to more clearly distinguish it from those, that would be helpful.
Thanks, this makes sense for purely additive processes. However, I often see people use this approach when the underlying process is clearly *not* purely additive. See my updated question.
The pair of values (rolls & successes) conveys information about a latent variable, p. The distribution I want is the distribution of p's in the bag. Of course, if we don't assume a class of distributions that it belongs to, there are arbitrarily many possibilities, but if we say that p is for example beta-distributed, then it comes down to just selecting the best fitting parameters for this class of distributions. If it helps, now consider instead of dice with p=1/4,1/6,1/8, that each object has a p from Beta(2,2)... or Beta (4,4), etc.
And mgcv package is an even better choice!
@whuber: thanks, but unfortunately for your suggestion, these *are* the data. They are points positioned along branches on a fractal tree (actually one of [these](http://www.onezoom.org)). The "contour line" is purely to visualising how far along each branch you have travelled (in this case, a geological date).
Anyway, I made this in R which is pretty close to what I want. ![Imgur](http://i.imgur.com/KZyHo4e.png). I moved all y values 1 unit closer to 0 to get them in the right positions, plotted that, then added 1 back on to all the numbers on the axis. Just need to format the numbers on the axis to have 1: or :1 but I'll work that out.
Thanks for your reply. So I would use a dynamic state space model to estimate the missing observation at time point $T$ using the observations on timepoints $1, \\ldots, T-1$?
Thank you for the replies
+1, this is a very good & very comprehensive answer. I must note that your discussion of constant variance for the t-test is too strong for my taste, though. I don't see it as impossible or "ludicrous" that the variance remain constant, even if there are many occasions where it would certainly change if the mean changes.
[Simpson's paradox](http://en.wikipedia.org/wiki/Simpson%27s_paradox) is addressed in dozens of questions on site (both with and without using that specific term). If you leave out an important variable, you may see an effect that disappears after you adjust for the previously omitted one. [There's also the possibility that it's important but that multicollinearity in predictors has inflated standard errors. It can also happen when sample sizes are small - not an issue here, I think. There are so many ways what you see can happen but my guess is another variable has captured it.]
Can you describe how the mortality variable is organized or give a sample table of counts (doesn't have to be identical to your data)?
Do you mean adding $1/n$ to the $\\pi_i$'s resulted from K-Means and then normalizing? It doesn't work, as the contribution is too little (e.g. for $n$=300).
@Marco, read the footnote in wikipedia for *geometric mean*. If one goes for the geometric means, he or she assumes that all $X_i$'s are strictly positive (even zero would be not suitable here). Real life data when in levels is mostly positive ^_^ And even if you do some negatives (like gains and loses) split the two and make them positive again ^_^
Do you know to fit a GLMM with response gamma in R?glmmadmb maybe?
Look at this answer http://stats.stackexchange.com/a/5349/2116, there is inequality there which involves third moments. The inequality says how accurate is central limit theorem and this accuracy depends on the third moment, which is related to skewness. You can find more of such inequalities in the book "Limit theorems of Probability Theory" by V. V. Petrov.
@LadislavNado thanks. I update it.
Welcome to the site, @T.S.Eliot. We're interested in both helping people w/ questions they have, & building up a permanent repository of high-quality statistical information. As such, we worry about linkrot. Would you mind providing an intro / summary of the lemma in case the link goes dead? Since you're new here you may want to read our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which contain info like this about CV.
A similar setup is actually treated in most introductions to Bayesian statistics. The beta distribution is used because it is *self-conjugate*, i.e., if you parameterize your *prior* belief about the probability of the event occurring by an appropriate beta and then update your belief through experiments, the updated *posterior* distribution will again be a beta, albeit with different parameters. See, e.g., here: https://ccrma.stanford.edu/~jos/bayes/bayes.pdf
I guess the comic highlights  that some people are poor judges of the quality of an item, and by averaging over many such people, you get a poor average. In general wisdom of crowds suggests that averages perform fairly well where at least a reasonable proportion of people have some knowledge. Weighting ratings by trustworthiness might also be one strategy for overcoming issues.
My point, probably poorly made, is that the argument in the first paragraph is based on the false premise that because the random variable can take on infinitely many values, each individual outcome must have probability zero. This is, of course, incorrect (Poisson, geometric, etc.); the concept of "infinity" is not a strong enough one here, we require *uncountability*.
WHuber, Together with a co-worker, we have understood your approach and solution... thanks
Edit the graphs in word? What filetype are you exporting the graphs too (or copy and paste)?
Personally, just to be sure, I'd polish off an initial approximation derived from CF with Newton-Raphson. Even then, based on some experiments I've done, I'm not convinced of the virtue of carrying more than three or so terms of an expansion.
But what if an individual goes to more than one coffee house?  Or what if Person A goes once per month to five different coffeehouses, and Person B goes five times per month to just one coffeehouse?
The bootstrap, and really any approach, makes completely new samples from a distribution that approximates the original distribution. Say we knew that the original data come from a normal dist. with unknown mean and variance. We could estimate the mean and variance (call the estimates $\\bar x$ and $s^2$) and then make random draws from the $N(\\bar x, s^2)$ distribution (this is a parametric bootstrap).  This isn't exactly the original distribution but will get arbitrarily close to the original dist. as the number of observations gets larger. Nonparametric bootstraps behave similarly. @mikera
I would say it is ok. Just try and see if it improves the cross validated score (e.g. f1-score).
how were the proportions calculated?... give more details of the experiment
The alternative form is "I only believe in statistics that I doctored myself" sometimes claimed to have been put into Churchill's mouth by Goebbels during a propaganda dispute over wartime losses.
Thank you for your help -- I added an example.
I tried to get my mind around this but I still have some questions :). I test the main effect of several IV. However, I also suspect that one of my IV (experience) has a moderates the relationship between the remaining IV and the DV. I have the following main effects for my  IVs without interactions in the model: Experience=0.011*, IV B=0.016, IV C=0.106**, IV D=-2.693**, IV E-0.139. In a second model I now added the interactions between experience and the other IVs, leading to the following results: Experience=0.045**, IV A=0.092, IV B=0.146**, IV C=-2.506**, IV D=-0.156
Thanks, it is actually a text dataset, where each example contains some text and tags associated with that text.  In the training set association rules between text words and tag words are found and these are then used for prediction in the test set.  There are very few duplicate examples
$-E$ wont't work because, for some reason, NEAT doesn't play well with negative fitness. Also, this is for a school project so wheter NEAT is the best choice doesn't matter very much. What would you recommend for stopping criterium?
Regression does not require that *data* be normal, it requires that residuals be normal.
I believe I have sketched out a route to assessing and dealing with likely forms of "autocorrelation."  They would not be manifested as such in a traditional time series analysis, even if regularly spaced data were available.  Temporal correlation would result from correlation in external factors, such as weekly or seasonal cycles.  An interesting form would occur from temporal clustering of purchases (e.g., just after lunch or on weekends).  "Net changes of level" can be handled by trend fitting, if necessary, but simply using a running mean would deal with that automatically.
In other words it is impossible to validate the findings in a published model on another dataset without having the actual model present-And that would require the new dataset to be run through my "black-box" rf model for prediction?
In fact you usually don't even need the residuals to be normal... unless you're doing inference that relies on it. And even then, well, you sometimes have the CLT and Slutsky's theorem to help out. The obvious places it *really* matters would be small samples and for prediction intervals. And even in small samples or for prediction intervals, well, there's potential to do other things.
LR directly provides information about the relevance of every predictor. Other methods generally do not.
You *cannot* use the google function (FV (..,..,)) to update, because it requires *evenly spaced* intervals, and *equal payments*, both of which you do not have (at least not in the example).  So you could "in theory" apply the function individually, but it is just as quick to "reinvent the wheel" (especially as you don't understand the syntax of the FV function).  If you *know* how to calculate FV, then you don't *need* a special function to do it for you, unless it would save you a considerable amount of time.