tttjjj/elt_bench huggingface dataset:
My dependent variable is a response. It is bacterial killing - how well blood kills bacteria. We compare the number killed to a control, so you end up with the percentage killed or the killing rate, so technically it should fall between 0 and 100. This data is positively skewed.I do have 2 dependent variables. My other is white blood cell count. It is not a percentage, it is calculated per ml of blood. It doesn't need to fall between values. This was not normally distributed either.
Everyone thank you for your help!
You need to _integrate by parts_ **repeatedly** beginning with $u = y^{\\alpha-1}$ and $v=-e^{-y}$, $dv = e^{-y}dy$, and $$\\int u dv = uv - \\int v du.$$  Each time you do so, you will get an integral with a smaller exponent for $y$.  If $\\alpha$ is an integer, you will be able to finish up the process.  If $\\alpha$ is not an integer, things are more complicated.
Nice tips, thanks.
Also, a note on `R`. You can use `pmax(eig$values,0)` instead of the more cumbersome `sapply` call.
Analysis #:  I tried various analyses with different datasets.  Don't have the Nagelkerke formula handy but I bet it's readily available.
Yes, that's right. We "fail to reject" the null.
"*not limiting myself to finding a method that only satisfies Mardia's definitions*" -- method of doing what?
ah,stupid me.  thanks.
It should be possible to adapt a test to including replicates; it might need a permutation test if you can't get a normal approximation out.
@Andy In theory, such heteroscedasticity has no direct connection with nonlinearity, but in practice it is often observed that a variance-stabilizing transformation tends to linearize relationships. Any curve rising continuously from a minimum (like 0) to a maximum (like 70) will have a maximum slope somewhere in the middle of that range, often resulting in larger residual variance there too. That is why we would expect to see residuals to exhibit more variance in the middle and less at the ends. If that is *not* the case, we can hope for linear relationships with the *untransformed* variable.
I apologize for repeating myself, but *you do not need to do tests.* Practically anything you test for with a dataset this size will be "significant." There's a "so what" factor to this. What you want to focus on is describing and, where possible, understanding the patterns in the data set, not testing whether you have enough data to detect them! As far as working with a sample goes, that's what randomization was invented for: just take a random selection of, say, 1000 results for each value of $x$ (or all of them where fewer than 1000 are available).
@zach, yes thats safe to assume.
+1 Peter is exactly right; his answer describes a hypothesis test, and mine describes a confidence interval of the same. And his visualization advice is right on.
Seeing no other answers, accepting my own - or I will when I wake up.
@Max A simple initial approach is to posit an exponential decay in effectiveness over time.  The decay rate becomes a parameter in a model relating cumulative workout effect (as weighted by the decay) to outcome measures.  If the decay is not huge and not zero and you have sufficient data you can move on to exploring other forms of decay.  This is really in the spirit of exploratory data analysis.
Did you get homogenous subsets? Personally, I think that's the easiest way to report true differentiation of means by a measure (in your case, region). It's also "client friendly" if letters are the same means are the same.
Just FYI, I've posted a rather elaborate question about the limiting case in which the one-way $F$ achieves $p<.05$ but all pairwise independent $t$s are as insignificant as possible. No answer yet, but already safe enough to say that sometimes the ANOVA will have the only $p<.05$, potentially by a margin of .10! The question: [How much smaller can $p$ values from ANOVA's $F$-test be vs. those from multiple $t$-tests on the same data?](http://stats.stackexchange.com/q/83131/32036)
By index I mean part/whole. I don't know how I can put an example in there, but I will try.
I am not very familiar with R and never have I used  the vector representation. The R documentation tells the following about the direction of arrows: http://cc.oulu.fi/~jarioksa/softhelp/vegan/html/biplot.rda.html
It is recommended that before any analysis and statistics reporting you revert items accordingly so that they are positively anchored to the construct. In reporting mean, sd, etc., mark the reverted items, e.g. `[Not]"I seldom or never visit church"` gives clue for reader to know that the rating scale was reverted. And, of course, test documentation must make it clear in what direction each item is anchored to the construct.
Yes, dummies count as variables. You lose one d.f. for each term.
A closely related question is asked and answered at http://stats.stackexchange.com/questions/63447/integrating-kernel-density-estimator-in-2d.  In this case, if you are not firmly wedded to the idea of imposing an extra constraint explicitly, an answer is easy to come by: just compute a kernel density estimate on a fine grid and select the largest 68%, 95%, or 99% from the cumulative sum of its sorted values: they form a region within the grid. A smoothed version of the boundary of that region will do.
This scenario might be related to a [Poisson](http://en.wikipedia.org/wiki/Poisson_distribution) distribution. It models the probability of events occurring in a certain interval. In any event you might want to specify your hypothesis here. Potentially you could answer your question with a non-parametric test, in which case you would not need to assume a certain distribution.
@COOLSerdash The OP may mean fitting by *minimum chi-square*; usually it's used with Poisson or binomial sampling situations but also applies if you know the variance of the observations and assume normality, when it's equivalent to WLS (except all your t's are Z's and all your F's are chi-square).
Of course the marginal effects can be greater than $1$: the formula in this answer shows you need only make $\\beta$ sufficiently large. Since you're using `R` it's easy to try it out: simulate some data using a model with a huge value of $\\beta$, do the fit, and compute the marginal effect. One way to make $\\beta$ large in practice is to re-express $x$ in large units, such as kilograms rather than grams. Bear in mind that a *marginal* effect is a *rate of change*: it is a probability per unit of $x$, not a probability itself. Marginal effects can be (and often are) negative.
@Henry giving away too much I think ...
After more reading, I'm guessing the GDP deflator is what I need. Maybe something like this which covers most of the years and countries from the previous data. Although I'm not sure these really fit together... http://stats.oecd.org/Index.aspx?DataSetCode=DACDEFL
@semibruin thanks, I gave that as an example but I am more interested in a case where the MLE does not have a closed form.
Do you mean [recursive partitioning](http://www.statmethods.net/advstats/cart.html) or really clustering (e.g., [hierarchical clustering](http://www.statmethods.net/advstats/cluster.html))?
@ocram- Thanks for the clarification. It seems the method requires some intensive computations, maximizing the profile likelihood. Just wondering why not simply resort to the bootstrap method if the estimator is not normally distributed.
I have recently encountered this question too. Did you solved this problem by now? I am wondering whether there is any matlab code to share? It would be a great help to me! Thank you.
Given that these values appear to be discrete, I strongly suspect that even after such a transformation they will *fail* just about every randomness test with flying colors!
+1 **You need more data** and **what you intend to use the results for** deserve extra emphasis.
For example DBSCAN can be accelerated well with indexes that support epsilon range queries. Probably mean shift clustering, too. OPTICS, which can also be accelerated this way, can be seen as a more advanced version of linkage clustering (you could call it "hierarchical density linkage clustering")
I guess if you are using a decent package, there should be confidence bands on ACF that allow you to assess the goodness of fit.
Counts are *NOT* continuous; they're explicitly discrete - indeed, they're the non-negative integers 0,1,2,3, .... As an aside to your main question: -- since your data are counts, one might instead attempt to construct a suitable explicit multivariate model for count data.
There are many ways to interpret (or misinterpret) your description, Daniel, so let me know where this attempt at a clarification might be wrong: You have a set of (x,y) coordinates.  You apply a function to them to obtain a set of (x,y,z) values, a "terrain map."  The z values can in principle lie in [-2.25,2.25] but actually they lie in [-1.75,1.75] with the middle 40% of them in [-0.4,0.4].  You seek a transformation f so that the collection of f(z) is uniformly distributed in [-1,1].  (If this characterization is correct, there's a simple solution involving a linear transform of the rank.)
Do you have some references where you've seen it used in an effective manner? I'd also like to see the Venn Diagrams you mention as well.
Each of the kernals work for high dimension versions of their respective boundaries. Does that answer your question? You are not limited to three dimensions for any kernel that I am aware of.
First off, it looks like you really *are* using a "likelihood" function to extract likelihood and forgetting to take the log of it. Nonetheless, GARCH is a special case of a mixed effects model, e.g. there is a covariance structure (autoregressive) that's estimated. These random effects are estimated from the data and are included in the likelihood. This is the important reason why, in mixed effects models, you have to fit the reduced model to have valid LRT for a fixed effect. The refit random effects will be different.
A t-test only compares the *means* of the distributions (of which the two datasets are assumed to be random samples).  This is why I found your question confusing: obviously comparing only the means is not at all the same as comparing the distributions.  Thus it seems that your question is put very well in the title and the last line, but the first three lines are irrelevant and distracting.
@RoeyAngel: probably it does not affect it in any sensible way. Specifically for `R`'s `lmer` for example a model where the random effect has a distinct value for each data-point will fail to even compute. Think of it in purely conceptual terms: if your $Z$ matrix is square then you $\\gamma$ vector holding the random effects realization will be of size $N$ ($N$: # of sample points) and thus you'll have an unidentifiable error structure. Are you sure you are asking this? As StasK, I also find it a bit hard to follow your question.
I would be tempted to say that decision trees are harder to train. There are a lot of details in the learning algorithms for them. PCA has a well founded framework to tweak rather than get lost in the details.
Some comments: (1) `rbernoulli` can be written more succinctly (and far more efficiently) as `rbernoulli <- function(n,p) runif(n) < p`. Many people would find this much clearer than the pseudocode, too. (2) Checking the variance is redundant. A thorough verification of accuracy (assuming the values truly are independent) requires only (a) demonstrating that all the results are zeros or ones and (b) the proportion of ones is not significantly different from $p$.
I don't know how to do anything useful with the aggregate data. It is sometimes hard to get useful data from my university too.
If you're interested in not having that correlation in your estimates, you might consider mean-correcting your $x$'s (in some situations that makes sense to do, sometimes it doesn't). Beware, though - it changes the meaning of the intercept.
It is generally not good practice to delete data.  But where things went seriously wrong was when you came up with the notion of "terrible at predicting x".  Logistic regression does not predict Y=0 vs. Y=1 but rather predicts Prob[Y=1].  Use measures that respect continuous probabilities such as log likelihood, Brier score, concordance probability, and, most simply, plot the distribution of predicted risks.
Hi, thanks for the reply. The variable is kinda like a percentage where 0-100 is the range bound, with fraction percentages rounded off to nearest integer. Any predicted values beyond this range will be absurd.
Business daily values
Cool! Please post a link here to your thesis once it's complete and/or published!
Frankly, I would use a different feature selection approach (or none, actually). SVMs are quite robust against uninformative features. Additionally, in the context of class imbalance it is better to use class-weighted SVM (different value of C per class, higher for the minority class). Your setup seems extremely complex to solve a fairly basic problem.
(+1) Welcome to the site. I have added the links in your comment to the body of the post as well as one to the Schölkopf and Smola text.
Do you remember the source for the statement?  There may have been some explanation there...
So each experiment produces a single vector of numbers? or does each experiment have multiple groups? If it's just a vector of numbers then if you centralise on the mean, then by definition both vectors will have the same mean (i.e., 0).
Thanks for your comment. I just see this plot in a book. Y is dependent variable and X is independent variable. There is no theory. it plotted a scatterplot to show the relationship of Y given x. And there is a question in the book that asks for if there is any relationship or not, Linear or nonlinear?  Strong or weak?
Hi (thanks for the comment). Predicted values are indeed not the group means by `id`, but they are a linear combination of the means (r=1). I'm surprised too.
The question as posed makes no sense. If I roll a fair 20-sided die that has one side marked '0' and 19 marked '1', I could easily get 96% ones, even with 1000 rolls, and the result would be 'random'. On the other hand, I could produce a set of 'coin flips' that are non-random (maybe I am Persi Diaconis and I can make them come up almost as I like) ... and so get 500 ones followed by 500 zeros. That would be a very non-random 50%.
;-) Justifiable (perhaps!) just because other approaches are not toys, and because he is not seeking "structural changes" in a series, but a _single_ recent "jump".
First comment: Some of the rows have fewer than 15 seats, so if seat 15 was selected then there aren't 30 possible rows. Second comment: Even if you use the correct number of rows, I don't think your approach will give the correct answer. Try using the definition of conditional probability.
@subhashdavar It sounds like you may have a separate question in that comment. I suggest asking a new question and perhaps including a link to this one.
I have edited the question.
Actually I was talking about test accuracy, and also I've tried using -h but nothing changed. Yes, I'm trying to find out an optimal C value. So, your answer makes sense for me, and I'll try grid search, and I'll tell you if I faced any problem. Thank you so much for your answer.
EpiGrad, if you re-post this as an answer to the migrated question, I imagine I can just delete this overall question. Yes, I can see that I do have a delete button below my question. If I don't see that, I will just wait on what you and whuber figure out.
@suncoolsu, When I'm thinking about causal inference I never worry about Occam's Razor orparsimony, could you explain the connection to me?
+1. I think the comments you made under (2) would probably be relevant in the situation described by (1) as well.
Well, your response seems more close to what was asked by the OP (so I've +1). However, I feel like this isn't a boxplot anymore, or at least it loose its interest in spotting possible outlying value. Of note, you can customize a little bit more the boxplot (see the `pars` arguments to reduce its aspect ratio (`boxwex`) and the size of the whiskers (`staplewex`)).
@gung Thanks for the insight. There's some overlap and informality in jargon the literature, this type of work is sometimes referred to as response propensity scoring. At any rate, my main question with survival modeling is how do I deal with the fact that the probability of response by a given time increases for a bit and then begins decreasing? Is this done via a transformation of the dep. var? Again, I'm not familiar with survival modeling, so any references would be appreciated.
This example sounds merely like a bizarre ineffective way to conduct numerical integration rather than like any Bayesian analysis.
I don't have this as a complete answer but I will give you my approach.  Let Ph=X12/TFp and Qh=x21/TFe where TFp=(x21+x22). Then I would try to construct the confidence interval using Ph-Qh.  I haven't even thought about how to construct a pivotal quantity out of this.  The cutoff for the confidence interval would be based on the multinomial for (x11, x12, x21, x22) keeping in mind that the multinomial has the constraint x11+x12+x21+x22=N  where N is the total sample size. I was also thinking about conditioning on the marginal totals and see how a conditional CI would look.
Heh. I just posted on the other thread, not realizing you'd posted this. There's a number of things to be found on the CLT applied to the variance (such as [p3-4 here](http://www.edwardomey.com/nonsave/CLTforsandr.pdf) for example). Nice answer btw.
Duane, although there's a kernel of good advice in this reply, I suggest you not follow it. The first part advocates a simplified form of model building akin to "backward stepwise regression." This is known to have problems.  The second part--"playing"--does not seem to provide any actionable advice at all.
Saying that the unnecessary days are bounded at 0 seems flawed. (Speaking as an M.D. all that stuff about experts is tangential gobbleygook.) The only measure bounded at 0 is the total number of days. That being said you are not analyzing the data with a Poisson model so response is not a good fit to your modeling assumptions. The confidence interval with a bound less than 0 is a symptom of the use of (inappropriate) Normal theory models.
Good luck! I think that's a good call.
Unfortunately, this is not correct. The events $\\{u_i \\leq u_N\\}$ are *not* independent. Instead, use symmetry.
I guess you could still use these simple sums scores in your multinomial logit model. Alternatively, look at the distributions of answers on (the sums of) the original items, to get a feeling for the meaning of the factor scores.
(As far as being a long route to doing it, I think it can be if you're doing it by yourself. I was lucky enough to take a class that involved R how-to documents for each type of analysis. I would be happy to share these with you privately if I could figure out how(?), but I cannot post them since I did not write them and haven't asked permission.
"the intercept under such codes represents the mean of the group means": The mean of means is not the marginal mean, which is the only parameter I think we'd care about. Only under balanced design would we come close to having that, and that assumption is too impractical for our purposes.
A test of differences would be among *groups*; a correlation is between two continuous variables. So, on that level, it's never possible to do both on the exact same data. I can't think of a case where the exact situation in your RQ1 makes sense, but something similar might, if the groups are naturally occurring or decided a priori.
but can't I say $\\Pr(X\\ge x, Y\\ge y) = 1 - (Pr(X \\le x) + Pr(Y \\le y) - Pr(X \\le x, Y \\le y))$. Thus I can differentiate twice which gets rid of the other terms and gives the pdf.
I have this for some of my values, but unfortunately R doesn't support a cell having multiple factor values.
If you're curious about the "high-resolution calibration curves". There is a discussion under "validation plot" in Steyerberg: "Assessing the performance of prediction models: a framework for some traditional and novel measures". He uses the same R package and results thus tend to look similar.
This model will be used for prediction. It is true that although the R-squared differ a lot, the predicted values are actually the same using Y or the change of Y. However, given the low R-squared value using the change of Y as DV, does it mean that the current set of IVs is not able to explain the change very well and there must be some omitted variables?
in your dataframe there should only be 100 trials in each cell that is a combination of the 2 tasks. Also as an aside, I'm not wed to nlme. I've just begun using (or trying to use) mixed-models. nlme() is the first function that I've been introduced to.
Dividing your individuals into two groups is a bad idea as it discards information and reduces power.
The ratio distribution can be calculated using [this](http://en.wikipedia.org/wiki/Ratio_distribution#Derivation).
"Are they are important as a difference of 1 in your other attributes?" - that's why we standardize at all, but I see your point on the arbitrariness, even in the case of a difference of 1 in one binary variable vs another. There's some model-driven vs data-driven issue here as well. Very deep. Thanks for this
Thanks for a swift answer, not a homework though. Would have tagged it as such if it really was.
If you can figure out how to *simulate* data with the properties you describe then you can do a tailored power analysis by just repeatedly simulating and running whatever analysis you expect to do for each data set and aggregating the outcomes.  More of a programming problem really...
Some comments would be appreciated for the down-vote.
You've got two issues. (1) Calculating the weights. (2) Using the weights.  Do you need help with both of those, or just one?
At this stage, your problem is matrix algebra first, then statistics. Work it, and let the forum know if and where you got stuck.
Sorry, because my answer was immediate and pointedly intuitive, I didn't think of any sources myself, in particularly with popular explanations. But I'm sure that googling will bring you to a number of pages you are looking for.
Thanks. As my dataset has duplicates, I don't think the method explained in your link will works, based on [Wikipedia's article on the subject](http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient). However if it makes sense I will use themore complex formula to calculate the coefficient.
Actually the data is ordered (its one misuration every minute), but I think the order it's not fundamental.
Good questions to ask are (i) whether a probabilistic output is required (ii) is the dataset large or small (relative to the number of attributes), i.e. is the chief problem avoiding over-fitting, or in dealing with the volume of data (iii) are the classes imbalanced (iv) are the attributes discrete or continuous or both (v) do I have any expert knowledge that I can build into the model (in which case you need to have a model that can accommodate that knowledge) (vi) are there missing data (vii) can the labels be incorrect sometimes (viii) are the misclassification costs equal.
+1 awesome answer and great reference for how to bend source code to suit your needs. This type of thing comes up on SO all the time, I wish there was a way to have this available in the R-tag on SO as a FAQ.
I have never seen any paper use a bootstrapped sample bigger than the original sample. Efron's original paper doesn't do it, subsequent papers offering various asymptotic proofs don't do it, white's reality check (example that comes to mind) does not do it. I don't think I can point you to a specific paper that says don't do it but to me why you should not do it's at the most basic level the problem posed in this question: you end up reducing the variance in the distribution for reasons which I believe you'll have a hard time arguing in favor of.