tttjjj/elt_bench huggingface dataset:
When you say $\\{0,0.1,…,1\\}$ do you mean the data is all multiples of 0.1, or something else? If so, how does it arise - as ratios of counts, or some other way? Why do you want to use a beta, rather than something else?
This question has come up before.  ANOVA and so called "multiple regression" are equivalent; in R just use lm() in either case.  The more tricky problem is how to deal with your five responses (rather than the more traditional one response variable).  There are ways to do this though.
Not really, it would just complicate your model without any benefit. The point of a measurement model is that it allows you control for measurement error. If you only add a composite variable you loose all the information present in the data that allows you to estimate that measurement error. So I would not do that.
@alis Maybe you should look at http://stats.stackexchange.com/q/11812.
For more information about such "online" approaches to updating statistical summaries, look at the threads tagged with [tag:online].
Your question is very legitimate, so I have edited my question to add some information about my application. Regarding the number of samples I have: I control the sample size, so I can sample as much as I want, but the computational cost increases. In addition, as you have said, it might take forever before I get a sample in the far tails.
Thanks for this; but, as I said in the post, there are about 80 measurements and they are all correlated to different degrees. Also, in the definition of y1 and y2, where did 0.7 and 0.0 come from?
(cont.)  Then, one might want to test if the mean returns between the two processes are equal. But, that's getting a bit ahead of ourselves, perhaps, and also fixes rather strong (and, often, empirically false) assumptions on the price process.
+1 for for your answer, I wasn't aware that this would be so hard answering. It seems like a common problem, perhaps the question is more of a question of coding than and you might be right about SO being a better choice. I tried your formula it seems that vf+I(vf*log(time)) has an excellent fit, I tried just vf*time and vf*time^2 but the log gave by fare the lowest p-value. I tried to run it with the cph() function to get the AIC but it gave an error :( Do you have any idea of how to do a plot on the estimate?
(Note that I'm talking just about one question to ask to users). If for example I have a very big budget (say infinit) which is sufficient to ask all the users, then if I ask all the users, the final answer that I aggregate/deduce may be worse (have less chance to be correct) than the case where I just ask only the first user which have an ENCAD equals to 0.98 (very hight) ... Your approximate algorithm is not considering such cases.
Are you putting additional predictors into the logistic regression model? The manually calculated odds ratio will only match the odds ratio you get out of logistic regression if you include no other predictors.
Are you sure you'd want to let the subsample influence the mean of the overall sample for the purpose of this comparison? If not, you could more easily perform an independent samples *t*-test or a robust alternative that compared the subsample with the rest of the sample (excluding the subsample).
I might not be well versed on this but can you tell me what you mean by dependent features?
What a fantastic answer, thank you very much @whuber.
The `ezMixed()` function from the `ez` package automates the evaluation of fixed effects by model comparison as presented in this post, but avoids the pitfalls of p-values (both specific to the likelihood ratio test and to p-values in general) by providing likelihood ratios as a properly scientific metric of evidence. Additionally the dev version (https://github.com/mike-lawrence/ez) applies generalized additive mixed modelling to account for potential non-linearity in continuous predictors like "trial".
It looks like an exponential distribution.
To clarify, to say that "imagine many separate instances of the beginning of time; you'd expect about half of those to produce heads" is perfectly correct frequentist reasoning.  However to go from there to "therefore the probability that this particlar coin is also 0.5" is not as a probability is applied to something that doesn't have a long run frequency as it can only happen once and once only.  It is perfectly sound Bayesian reasoning though, as a Bayesian probability is a statement about the plausibility of a proposition (which can be based on a long run frequency within a population).
Small but not trivial. Simple but challenging. Real data. Fisher's reputation, although it's not his data. Tradition. Inertia. Continuity. You can find flower pictures to spell it out.
Really good example. The only issue with your graph, is that both Y variable names are overlapping. In this case you would want one on the left and the other on the right (possibly even in a vertical position). To upgrade your example from "really good" to "perfect", you might wanna use the mtext function from R to do the variable names
@gung Not close enough, I think; the time series aspect gives this its real spin. I'll edit the title to match.
yea but adding a simple re-weighting step yields an estimator (LTS) that is equally robust and *so* much more stable and statistically efficient. Why not do?
As this is a self-study question, could you please add the [`self-study`](http://stats.stackexchange.com/tags/self-study/info) tag? Thank you.
Excellent detailed presentation.
Thanks! I have tried using "rlm" but the resultant QQ residual plot doesn't improve too much... I have updated my question... Thanks so much for your help!
thanks for your answer. I wil dig into it. I found the article here: http://wilson.engr.wisc.edu/rsfc/oth_sites/rate/papers/glickman99parameter.pdf
"*The P value indicates the chance of obtaining this data if the null hypothesis were true.*" --- no, it doesn't. It's the probability of obtaining a result *at least as extreme as this one* if the null were true.
And if there is no apriori hierarchy of classes?  I dont know how detailed or not I need to get, this is what I need to determine.  Let me give you an example,  suppose I have zipcodes by state or region (like the Northeast), these are my categories.  I want to predict some cost.  What technique can I use to group these zipcodes so I can effectively use the groups as predictors for cost?
These should only be the same if you are performing a linear regression! Looking at your data, if I try `lfit <- lm(predobs$obs ~ predobs$pred)`, the model I get has a slope of 0.56, and an intercept of -0.0387. If you incorporate this affine shift into your prediction, then the two forms should be equal. You can check this via `r.square.3 <- 1 - (sum((lfit$residuals)^2) / sst)`
"I don't think there is necessarily anything wrong with neglecting questions..."  I think this is a fair objection; I've added an EDIT to my post to address it.
Has there been random assignment of cases to conditions?
Not so. I've never met your first usage, even as a mistake. If you were right, and I don't think you are, it would be much flagged that a minute difference in wording was associated with such a big difference in meaning: textbook writers would be obliged to explain at  length and there would be campaigns to change the terminology.  I challenge you to find even one explanation of your definition in the literature. @whuber already gave an excellent answer that remains definitive.
Thanks for the references. The level of detail is however insufficient in these publications -- I had to read from Feinerer's dissertation to get enough details on how to operate `tm` on my end. Still, thanks very much :)
There are measures of fit that are penalized by adding more parameters, such as the AIC for example http://en.wikipedia.org/wiki/Akaike_information_criterion.
I'm guessing that something Bayesian would be appropriate.  Without any data on the hypothetical unobserved sides, you might have to inject some kind of prior belief.  Can you clarify what are at least two of your possible candidate models?
Common solutions are not always correct solutions.
If bootstrapped and parametric ps differ a lot, my first guess would be that the assumptions of the parametric model somehow are violated. So I would check for that. I would not agree that the boostrap generally gives more conservative results.
Ones made out of paper. Small laptop-like devices.  A scientific notebook for recording lab notes and such. Various [pieces of software](http://www.mackichan.com/techtalk/articles/mmaj01__1.png), or various kinds of files used by various kinds of software. The term 'notebook' might mean myriad things to a person unused to Python.
You have forgotten to mention what are your y values. I am guessing that you might be either trying to find out whether your time-series has a change in trend (slope), in which case you are trying to fit a piecewise regression http://en.wikipedia.org/wiki/Segmented_regression (Though you you should know about the pitfalls of applying a linear  model to time-series data. Especially about heteroscedasticity.), or you are looking for a "jump" in the general level of the series across different segments, in which case you should look into changepoint detection of the mean.
I'd love to hear people's responses to this too. In my scenario I have sensors of various kinds instead of computers.
Sentence has been corrected to include the omitted information.
As it is, $−\\log(B)$ appears on both sides of the regression equation and so is redundant - whatever coefficient appeared in front of $\\log(B)$ would actually be 1 unit too small. Also $\\log(A)$ appears twice on the right hand side. Is this what you intended?
Ah, the Net Promoter Score. The scourge of any performance analyst's existence because it is so ill-defined. My suggestion is to not worry about it so much because there is not much empirical or theoretical justification for calling NPS = 9 or 10 "Promoters" and 0-6 "Detractors" and 7-8 "Passives" in the first place. If you move to a different scale and it changes the distribution of "Promoters" a bit, so what? What should be done is to survey your past customers on their self-reported recommendation behavior and use that to compute weights for each point on the NPS scale.
I have found this: lsmeans(YOUR MODEL, pairwise~FACTOR1*FACTOR2, adjust="tukey") from Carlos Lara-Romero in this link works: https://www.researchgate.net/post/Post_hoc_test_in_linear_mixed_models_how_to_do
You have "repeated measures" in your title, but don't use packages nlme or lme4. Seek advice from a statistician.
Just a note: the default link for the `Gamma` family is the inverse link. This is fairly obscure; I'll bet that the log link is what you're really after.
(+1) A minor thing: One might explicitly distinguish between the true effect size used for the simulation and its estimate. It seems your `es` is $f^{2} = (\\frac{1}{J} \\sum_{j} \\alpha_{j}^{2}) / \\sigma^{2}$ (groups $1, \\ldots, J$, effect parameters $\\alpha_{j} = \\mu_{j} - \\mu$, error variance $\\sigma^{2}$, equal group sizes). I.e., your `msb` is the variance of the true $\\mu_{j}$, and your `msw` is $\\sigma^{2}$. The empirical sum of squares give the estimate $\\hat{f}^{2} = MS_{b} / MS_{w}$, but your simulation uses the theoretical counterparts.
Added some information to the original question, but an analogy would be looking for trillionaires in a population set where most people are poor. Even if I filter the data to 1% so that there are the millionaires, multi-millionaires, billionaires, and trillionaires, since the trillionaires still relatively make up most of the money, the millionaire and billionaire data points can be trashed.
"Normally, more than two-thirds of the people are average" (meaning within one standard deviation of the mean)?
> sv0.bug <- bugs(data = sv0$data,
                inits = list(init),
                param = c(nodes(sv0, "prior")$name,"y.sim","h"),
                model = "sv0.txt",
                n.iter = 11000, n.burnin = 1000, n.chains = 1) 
How can I estimate the parameter values of a basic SV model? a command that could print an estimated value for the parameters  (alpha0, alpha1, sigma) or (beta, mu, sigma) of a SV model
"Does this affect the validity of the parameters it has come back with?" may be on-topic here. @user41024, can you provide your data, or at least some example data that would allow people to reproduce what you find?
A nice book and philosophical but not what the OOP is looking for.
"The probability of $v_i$, given that $z_i = k$ and given the value $\\mu_k$, is distributed normally with mean $\\mu_k$ and variance $\\sigma^2$."
(+1) As I was reading the question, Vietnam was the first example that immediately sprang to mind. It was amusing to see you had taken the very same tack. I suppose it's the most obvious choice given the stated ages of the subjects, though ages in the early-to-mid 60s are a bit closer.
Thank you. It should be fine now.
If the primary goal is to make some kind of inference to the population and the sampling mechanism is of such a nature that the representativeness of the sample is questionable, then indeed, any inference will also be rather questionable. Essentially, you can only make an inference to that part of the population that the sampling mechanism provides a representation of. In principle, the inferences you make will be appropriate for that part of the population. Whether that part of the population is of any interest to you (or the readers) is another issue.
Computer scientists call these "Poisson trials" to distinguish them from Bernoulli trials. In addition to Central Limit Theorem approximations, there are also good tail bounds available. [Here's](http://en.wikipedia.org/wiki/Chernoff_bound#Theorem_for_multiplicative_form_of_Chernoff_bound_.28relative_error.29) one. Google searches on "Chernoff bounds for Poisson trials" will turn up the results you might find in a typical CS treatment.
@John Glad you've found the help you need! To indicate that your question has been resolved (and as a thank you to Michael) it's nice to click the check mark to the left of his answer, just below the votey buttons.
IMHO lag auto-correlation makes sense only if you have an "time-seris like" ordering otherwise it doesn't make a lot of sense. Also, independence => correlation = 0, but not the other way.
Is there anything you'd like to know about DOLS other than what R package can do it for you? If not, I suspect this question belongs on [Stack Overflow](http://stackoverflow.com) rather than here. *Please don't cross-post, though*, if you want your Q migrated, flag it for moderator attention & we'll move it for you.
Are you asking how to calculate quantiles for the distribution of variables defined as the sum of a uniform and a gamma random variable?
An application of Slutsky's theorem, versions of which are sometimes referred to as the *converging together lemma*, shows that the limit is standard normal.
I'm not sure you have studied all of the above.  Generally we choose a model that has competitive predictive discrimination, then validate that the index of discrimination is not good just because of overfitting, then validate the calibration of the model.  The last step is best done using a high-resolution smooth nonparametric calibration curve.  All of these things are implemented in the R `rms` package.  And avoid comparing AIC of many models which is just another way of using $P$-values to select variables.  If you are comparing only 2 pre-specified models you're OK.
Discriminant functions are extracted so as to maximize between-group variation to within-group variation ratio. It has nothing to do with classification, which is the second and stand-alone stage of LDA.
Did you consider to use [Regular expressions](http://en.wikipedia.org/wiki/Regular_expression) ?
I cannot follow what you are doing because I believe the omitted details are relevant: we cannot possibly compute probabilities without knowledge of them. Exactly what does your algorithm aim to do?  How do you quantify a "better" result?
I am not aware of a quantity "r" in regression analysis. Are you referring to $R^2$? If so, don't confuse the coefficient of determination (which is always positive) with a correlation (which is between -1 and 1): https://en.wikipedia.org/wiki/Coefficient_of_determination
thanks @Zach, but I think I might not have been specific enough in my question - I've updated it to include a table "visualizing" what I meant, but with two data sets instead of three. Although using colors might have to be the way to go. I could have each cell contain 3 words, one for each color
Chebyshev's inequality does *not* let you apply a *sample* value to a population!  Although people have done exactly that, the procedure has no validity and does give erroneous results.  To see that no upper bound can apply to the population variance, just consider sampling from a population with infinite variance: necessarily the *sample* variance will be finite, so any universal upper bound must be infinite.
About autocorrelation and the number of time points: I only show three in this example data, but my real data contain observations on 8 different days, so I think it probably will be an issue.  Any ideas how to put that in?
How would you use logistic regression on this? I think logistic regression is just a way to reach inside the data to before the logistic function was applied, so that your results get scaled to be within [0 .. 1]. I already have the scores before applying the logistic function, so why would I use logistic regression?  (score is in log(odds), probability = odds / (1 + odds) = 1 / (1 + 1/exp(log(odds))), so my conversion from score to probability is exactly the logistic function.)
Also known as "please answer my homework"? If so, please tag it as such. Oh, and look into autoregressive trends.
@onestop I can collect mean, also editing question accordingly. Thanks
A bunch of things, but they mostly boil down to: whatever I fit has to be reproducible in SAS' `proc reg`.
I'm trying to extract the seasonal component on this time series -- to my knowledge, `stl()` does not incorporate covariates.  Is there another method I can use in `R`?
To take out seasonality you need to first check if you time series is stationary. If it is I suppose your logic holds.
Thank you for writing such a clear, well-referenced answer. Welcome to our community!
I think you need to divide by the square roots of the eigenvalues, as it is a matter of scaling by SD, not variance.
Yeah, yikes.  I wish the two tests had different function calls---I would definitely have preferred an error to be thrown for what I was trying to do given the way I entered it.
@shabbychef I implemented the code in c++. and the results is the same as matlab when I have between 1 to 5 replicates per level. but when this range changes to 1 to 100, some of my SS values become higher in comparison to corresponding SS values of MATLAB. and since each SS value is computed based on the others, some of them become negative! It does not happen in matlab. So I know that something is wrong in my code.
*Which* chi-square test? For association in contingency tables? For goodness of fit? To test distributional hypotheses? To compare variance estimates? To assess significance of added variables in nested models using Maximum Likelihood? Others?
The answer is $0$. How could the 5th O+ donor be the **fourth** donor of the day?
I'm doesn't matter so much _which_ polynomial class you use (up to numerical issues), but you can get better results by the choice of nodes where you sample the function (in this case the bivariate gaussian density) in order to compute the coefficients in front of the polynomials. For this problem, the Chebyshev nodes are optimal, at least when interpolating on a finite interval. see http://en.wikipedia.org/wiki/Chebyshev_nodes or any decent numerical analysis text (e.g. Cheney & Kincaid) or http://pages.cs.wisc.edu/~amos/412/lecture-notes/lecture09.pdf
I think that the $L(\\theta \\mid x)$ should be positive for $x_(1) \\geq \\theta$, rather than the other way around.
@MattAlbrecht thanks for the interesting reading. In practice and especially in observational studies, ANCOVA is often used to control for differences rather than as a way of increasing power, which is more typically a focus in experimental settings (though certainly in observational settings power is of great concern also). Should people wish to apply ANCOVA as a way to statistically control for differences, they need to understand conceptually what they are achieving with it.
When you have categorical independent variables, you are always comparing levels, but there is a variety of contrasts, defining what you are comparing to what. I don't know what SPSS defaults to, nor what you mean by "pure".
Sorry! Square root was wrong. The author actually put it in the second formula, but left it out in the first. So it shouldn't be there
@pul Yes, the graph simply shows $1/\\xi$ using the formula for the Normal approximation.  I believe that a linear estimator has good properties in this case, but have not rigorously explored all the alternatives.  Clearly it will have some asymptotically optimal properties, but conceivably for small samples one could do slightly better with a well-designed nonlinear procedure.
It's arbitrary units normalized to a few things. The experiment took place in a closed environment. the variable may be total activity which means how much time the subject was doing something (eg exploring, running, walking, as opposed to sitting still).But at the same time we have other attributes/variables which are counts (ho many times the subject jumped etc)
Thanks a lot for your answer. I have updated my question because I think it wasn't completely clear. Would you still argue that $\\bar{y}$ is what you have defined and $\\tau$ would be something I would have to get from the researcher?
Any one guys. This is killing me. What am I doing wrong?
@Glen_b it is just a pseudocode ;)
As in Aniko's answer, you need to use `129` for the first parameter to `ppois`, since $F(X\\ge 130)=F(X>129)$.
@TomásM. Link function should be set as link ="identity". In this case, the parameter does not need transforming. Eg. lambda of Poisson, lambda from glm is the same as lambda from fitdistr and it can be used to simulate new distribution directly. But gamma distribution need to care about using the same pattern (shape/rate, or shape/scale). If the link function is not set to identity, some transformation has been done by glm by default, then the fitting parameters are not the same as fitdistr. This not only affects the value of parameter but also the standard error of the fitting.
It's not clear what your question is; I assume you're not simply asking us to do your homework for you. So, is there a part of this that's confusing? Do you want some hints about some aspect? etc.
@Tomek: Note that Firth's penalization will change your "impossible" to "improbable".
I didn't downvote, but I must say that some explanation or reasoning why to use random forest could make this answer very much more interesting. ;)
Thanks. But based on that, you can still compute coefficients including intercept for unscaled data with center and scale known.
Actually, $1 - \\frac{SSres}{SStot}$ (or, alternatively and perhaps more intuitively $\\frac{SSreg}{SStot}$) will give you the multiple $R^2$, whereas $1 - \\frac{\\sigma^2}{s^2}$ will yield the adjusted $R^2$.
@Glen_b: What I don't get is that you say "not one percent, but a change of 0.01 crimes per person". 
Aren't they the same (at least if I say 'percentage point' rather than 'percent'? 
Since x is crimes-per-person, a +0.01 unit change in crimes-per-person is the same as +1 crime-per-100-persons, yes? 
And that should be the same as +1 percentage point change in total-number-of-crimes?
Sorry, just a little confused.
@Gavin Thank you.  I have taken the liberty of adding one more line so that non-R users can understand what is being asked (and perhaps, therefore, offer meaningful answers).
*I have a deep lack of interest...*: A phrase to store away for a rainy day.