tttjjj/elt_bench huggingface dataset:
@gung It is implicit that the $k_i$ are finite, whence the domain of $f$ is finite, whence it attains its maximum.  Your objections nevertheless are valid: without any assumptions on $f$, the fact that it's a function is of no value: all we have is a population of $k_1k_2\\ldots k_5$ values and the question asks how to estimate its maximum from a sample.  There are indeed statistical methods to do this but they are trivial: if you want to have, say, 95% confidence that the maximum has appeared in a random sample, then you have to sample 95% of the population!
your answer is good. I just want to ask why is $-\\frac{b}{||w||}$ the distance from the origin to the hyperplane ? Can you elaborate (mathematically) on this a little bit please ?
Lots more information is needed.
Also they are measured like this narcissism 1-6 scale, rumination 1-4 scale, aggression 1-5 scale.
(+1) Thanks for that. I know about the problem of asymmetry in the distribution of VC, and the difference between ML and REML, but I didn't know you developed that package. I look forward to trying it out.
If they are ordinal, then it probably doesn't really make sense to compare means, but that depends on how the scores are generated. If they are sums of component parts (like Likert scales composed of several questions, say) then they were already assumed to be interval.
The formula is probably incorrect, since if $r$ is positive (and it always is judging from your comments below), the quantity in the formula is always greater than 1. Then it cannot be distributed as Fischer distribution, since random variable with Fischer distribution can get values less than 1.
Also note that if you want to calculate the density function with the language you're using, it is the derivative of F(x), where F(x)=P(x<length|Model=model).
Could you do a plot of your data, perhaps? A time series plot, plus any other plots you think might be relevant?
Jeffreys would probably say to start with 0.5 rather than 1, although if it makes much of a difference, you don't have enough data.
I might be mistaken, but it seems to me that the best answer would rest somewhat in the prediction approach you intend to use.  Maybe you could add some details about that?  You might also try on another stack exchange where people might be versed in basic file compression approaches.
It sounds like you seek for something like clustering questions (variables). Maybe. You ought to clarify what is questions "belong together", anyway.
@Brandon I still don't quite see how it is a special case. In survey research, we define a population, and attempt to sample from that population. In some cases the population is adults in a given country, in other cases it is people who use company X, in other cases it is people who use company Y, and in other cases it is people who use Company X, but are considering switching. In all these cases there are pragmatic challenges of actually getting a sample that is representative of the target population, and that is sufficiently large to draw relevant inferences.
IMHO, such hieroglyphs have a better place in a table, rather than in some text.
Note also that this other question of yours only provides the very last formula, while my answer explains why we do reach this formula.
This is a repeated measurement design, simple OLS regression is not a suitable tool. Mixed model may be a better approach.
Probably you indicated a continuous variable as a factor?
... see http://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval/2713#2713
SARIMA can be extended to multivariate case, which might be apropriate for you. The keyword in this case is VAR. This means that you observe a vector of numbers instead of one number for each time period.
The assumption"after removing any overall trend"is the Achilles Heel as there may be many time trends,many level shifts all of which were excluded in your example.The idea that the input series are deterministic in nature flies in the face of the possible presence of seasonal and regular ARIMA structure. Untreated Unusual One-Time Values will distort any periodogram-based identification scheme due to a downward bias to the periodogram estimates yielding non-significance.If weekly and/or monthly effects changed at some point in the past the periodogram-based procedure would fail
@DilipSarwate - she actually tried to ask this question as an answer to that other one, but it was deleted as it wasn't an answer to the other one, so I suspect the question really is "how to do this in R", rather than "how to do this".
@Ran they are iterated over in each step of the training. sorry for the late reply I didn't notice the comment.
I suspect that I could build on the fact that E(y) = alpha / (alpha + beta), but I can't really figure out how exactly.
McNemar's test is equivalent to a binomial test, but a very specific one. The details are spelled out in my linked answer.
many thanks for your thoughts. I remember that in Zuur et al. 2009 (Mixed Effects modelling book) they were comparing "nested" distributions, i.e. negative binomial and poisson. Yet I think I would rather go for your "well, no" answer. When I compared to models the deviances for model 1 was 4000 and for model 2 was 5000000 (or something like that), thus it did not look very convenient to compare both models.
@EdwardArmstrong: R has a `princomp` which is mainly for compatability with its predecessor language S. The preferred function to use is `prcomp`, which uses SVD. It mentions this if you get help on princomp: `?princomp`.
Your answer hasn't really reduced my confusion about when to do these things. I guess one of the things I was hoping to hear about are to what extent the issues with chi^2 are solvable by monte carlo simulation or corrections, etc; or the extent to which it can be superseded by glms. So I'm just going to leave this open for a bit to see if I can get more bites. But if nobody weighs in after a bit I'll accept your answer.
@DouglasZare the dimensions added are not real. The polynomial kernel is computed as $(x^Ty+c)^d$, so if you use a fixed polynomial degree (e.g. $d=2$), you get exactly one additional parameter, the constant term $c$.
@GaBorgula: I mean the learning curve in general.
You can also get a *multiplicative* effect by exponentiating the coefficients. For example if $\\beta_D=.43$, then folks with $D=1$ are expected to have $\\exp \\{.43\\} \\approx 1.54$ times higher outcome (holding everything else constant). This is the incidence rate ratio interpretation.
If I understand you correctly, you just want to set the missing values equal to the mean of the preceding values. This might be fine if your data are missing completely at random and normally-distributed around a mean consistent throughout the time series, and not sensitive to fluctuations in explanatory variables.
Excellent! Let me try this!
I voted to close as off topic. There's no statistical question here.
pdf link doesn't work anymore.
By the way, will the binomial not work for you? You have k trials and n successes. right?
thanks a lot! if i add more explanatory variables (continuous and binary) the fitted probabilities are not identical anymore, but still highly correlated (~98% of correlation). Why is this the case?
"...you don't know the distribution of the likelihood ratio under the null, even asymptotically." That's not such a big concern in a world where you can code up a simulation under the null in less that 20 lines of R.
A huge plus one for point 2.  Other than going through human subjects protection training, I've never received the tiniest bit of training on data collection and storage.  Getting the data collection right is vastly more important than the analysis.
Why all the downvotes on this one?
@lanS. All objects have indeed here only one dimension. Can you develop a bit more on the fact that the errors that can be either observation or noise. It is exactly what I would like to achieve. I would like to get a rolling estimation of the signal to noise ratio by estimating the sd of the 2 time varying noises....
That prooved to be an interesting idea but the interleaving zone is still quite problematic (in fact it gives worst results than just applying the ridge classifier directly on the dataset without any modification).
If you want to make the data reproducible, you could post `dput(data)`, which would probably be large but doable.
The accepted answer for the "Help me understand SVM", pointed out by @whuber is probably a good place for you to start!
+1 on the small sample size. Using 300 samples would help to distinguish things a lot. Proton: No, you can't really distinguish distributions with a small sample. How could you? It's like trying to identify a face with 20 pixels.
@ars: Everything is correct and nicely stated.  But one thing seems to be missing: the standard deviation of the "best approximation" j/n depends on the *true* proportion of redheads, not the estimated one.  The problem, of course, is that we don't know the true proportion.  But the fact remains that the standard error does not actually equal the standard deviation of the approximation except when the estimate happens to be exactly correct.  I know you don't need reminding of this subtlety, nor will most readers, but it's rather relevant to the original question.
Thank you so much for all the help! I will reassess my study based on your input and other references :D
This is not correct. $r$ is the " correlation coefficient between actual response values and predicted values of the model"; R square is $r^2$.
I think you want `keep if sex == 2 & minsamp == 4`. The `count` would be 876880.
Is the change a spike (a one time thing) or a level shift (so the mean is now at a different level)?
X seems irrelevant to your question of whether A is more often positive than B. Unless perhaps you might wish to stratify by X, i.e. ask (1) whether A is more often positive than B when X is positive, then (2) whether A is more often positive than B when X is negative ??
@whuber Indeed.  Thanks for clarifying the point I was trying to make.  In regard to the assumptions of the model,  the assumption that someone's daily decisions can be modeled as independent trials with the same probability on each trial troubles me.
I don't know wavelet methods so well so I'll just point you to Schafer and Wasserman's [tutorial](http://astrostatistics.psu.edu/samsi06/tutorials/tut2larryl_all.pdf) on nonparametric inference (example using 'wavethresh' starts on pg. 92). You can get the WMAP data from the example at [Wasserman's website](http://www.stat.cmu.edu/~larry/all-of-statistics). Hope that helps!
The first graphic given is awesome IMO. It has much detail that would be difficult to capture so intuitively in a series of dot plots. Also the Sankey Diagram is good to visualize flows from *and to* different nodes (the re-combining into used energy and lost energy).
@bgbg I am using a training/validation/test scheme, in a 70/20/10 split. I am training a model with 70% of the data and tuning the parameters based on the results on the validation set of 20%. After I tune the parameters on these two sets, I assess the model on the 10% test set. Though Breiman claims that the built in OOB error rate makes making a separate test set redundant, I am wary of his claim.
Sounds more like a stats question than a programming one. I think this should be moved to crossvalidated.com.
This seems kind of broad and unfocused; can you make your question more specific?
(+1) As my professor put it: The first step of data mining is to detect the relationships the client already knows, proving that your methods are working. The second step is to detect relationships the client did not know yet, demonstrating the real value of data mining.
Start with the DF of the order statistic, then write out the identities relating it to its survivor function and, subsequently, its hazard.
Assuming the positive label is `+1` you should do `svm-train -w+1 100 -c 0.03125 -g 0.0078125 -v 5 -q`. Note that you might need to optimize `c` again.
I don't follow.  You report a "download rate" of 100/1000 = 10%.  Now you want to *prohibit* downloads when the rate is too "low"??  And what rate could you possibly be referring to--the proportion of downloads among visitors during a particular day?  And how would that apply to any particular visitor?  And why would you want to prohibit additional visitors from downloading?  I'm sure I'm misunderstanding a lot, because this just doesn't make any sense.  Perhaps you could clarify some of these points in your second paragraph?
@COOL The "shifted binomial" is a Binomial distribution with a scale change.  It is not any form of a negative binomial distribution (which always has countable support, not finite support).
My main concern with SSE is that is assumes that **all attributes have the same meaning** and scale. Works well on toy data sets, often fails badly on real data.
@Flask No, the calculation is applicable for both cases. For your second case the chance is `(0.5^n) * (0.5^n)` as wel. When you want to take a more general approach, the case it doesn't matter sex is in the control group as long as the other sex is in the other group, than the formula is `(0.5^(n-1)) * (0.5^n)`. See my changes in my answer.
@ttnphns [Here](http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//009z000000r7000000.htm) is a popular illustrated help page on the topic.
Not that this really matters, as it's true for `family="poisson"` as well, but note that your example isn't a Poisson GLM, as you're using the `quasipoisson` family, which only depends on the relationship between the mean and the variance anyway, so in that case, there should be no surprise about taking non-integer numbers.
I noticed an edit to this post in reply to @jerad's comments (indicating the 1st method was considered). If you lost your account information, please flag your post for moderator attention and we will merge your accounts.
I defined a variable (f1) to represent my linear model. I used this variable to transfer it into gls and then compared this to weighted gls models. I compared AIC and p-value of the anova of these models to choose the model with the best fit. Here the code in R:
f1 <- formula(RR ~ factor(Temp) + factor(pCO2) + factor(Temp):factor(pCO2)) //
tmp1 <- gls(f1, data = Cal, method = "REML") //
tmp1B <- gls(f1, weights = varIdent(form =~ 1 | factor(Temp)) //
tmp1C <- gls(f1, weights = varIdent(form =~ 1 | factor(pCO2)) //
tmp1D <- gls(f1, weights = varIdent(form =~ 1 | factor(Temp) * factor(pCO2))
I admire the work and the hard thinking that this question reflects (+1).  You're concerned with many important issues with which many of us struggle.  Unfortunately I think the answers will come not from a few paragraphs but from ongoing practice, study, and in-depth consultation with experts.
Does this help? http://stats.stackexchange.com/questions/12900/when-is-r-squared-negative
Draw superimposed graphs of $\\exp(-(|x-\\mu_i|/\\epsilon)^2)$ for two different values of $\\mu_i$.  What happens as $\\epsilon$ shrinks towards $0$?  (A log-linear plot may help you see this better.)
Gaetan:> Elane seems to suggest that the slope in the summer are different from the slopes in the winter (not just the intercept).
But how can I tell you to use an uninformative prior, unless you said you wanted one?
This question is not clear. What do you mean by "how regression tree is better than OLS in terms of heterogeneity"?
@TheThunderChimp Caveat: if population variance is known (e.g. estimating population proportion — mean of a dichotomous variable), then the standard normal (*z*), and not the *t* distribution is appropriate.
Yes, I realize I'm not "supposed" to be looking at some of these things for ordinal data, but really, it's the only tool I could think of to compare the two years. Really, I was looking at things that could compare the distributions. But, I guess testing means could be plausible - but a confidence interval may not necessarily include my mean as there have been many structural changes to the industry for which this question reviews YoY.
@allhands Also, are you only interested in the difference in variance between groups or also differences in mean value?
This is a nice paper that discusses some issues: http://crab.rutgers.edu/~goertzel/mythsofmurder.htm (and it has a great title)
@Glen_b That is exactly what I did. Don't worry about it, I understand your concern.
The R commands *are* actually running a 2x2 chi-squared test (that's explicitly how the p-value is computed; in fact here's the exact line in the code for prop.test that computes the test statistic: `STATISTIC <- sum((abs(x - E) - YATES)^2/E)`). It should have the same p-values as a two-tailed z-test of proportions (as long as we're dealing with the continuity correction in the Z consistently with the chi-square). But it also gives a CI for the difference in proportion.
That seems right, except that a % interpretation is more difficult in the pseudo-R-squared case. It's okay in the linear case, where the ratio is explained against total variance.
Error rate is an example of an improper scoring rule - a measure that is optimized by a bogus model.  So I would never use proportion classified correct.
+1 and sorry for not getting back earlier. Thank you very much for the hints on how to go about doing this. I will examine this in more detail.
The question is also whether the difference is statistically significant.
Could it just be a compulsion you have, like some people obsessively wash their hands a lot?  But seriously, give *us* a hint why you felt you needed to.  It's either because knowing the population distribution would be interesting in itself or because a gamma distribution is an necessary assumption for some further analysis.
If you know it's overestimated in the sense that it's always too large, you know that it's less than what you observe. That is, *knowing* $x< k$ or $x\\leq k$ is at least some information about the value, information that is not present in a missing value. If the data collection error is of that kind -- such that the available value is always going to be too high, you should probably make use of that information (perhaps by treating those values like censored values - you know they're less than the observed, but not by how much).
Can you be more specific about the nature of the data and especially the intention of the mapping (a visualization)? Will the resulting map onto the circle be 1d or 2d in polar coordinates? And related, why isn't utilizing a 1d or 2d PCA projection and just using polar coordinates sufficient (do you have circular data to begin with)?
(0.37+0.13-0.10)/3 might give a rough estimate of the correlation between any two measurements. Another way, if possible, is to fit a mixed model by imposing a compound symmetry structure and to store rho. Anyway, you should probably check to what extend power depends on that estimate... good work ;-)
Do you mean sparse as in "lots of zeros" or as in "lots of missing values"?
Short answer: yes - the sample means can be the same.
I just checked the link and also my personal copy; 1997 it was.  And no further editions so far as I can see.  Post is now corrected.  Thanks.
@chl I changed the title... Probably it is OK...
If you have a large enough sample size, even the smallest difference in distributions will appear as significant. This is for every test, not only Fisher's exact test.
"*whether a person's weight is correlated with their weight*" --- I'm pretty sure that correlation is 1
I think the author wants to have an expression that has only either $F$, $f$ or $\\mu$, without any integrals. Integration by parts is not helpful... what else? In general my answer would be no, it is not possible :)
(That is, if the test statistic even has a $t$ distribution anymore)
@drknexus Wouldn't it be more accurate to not say "given the observed data, the null hypothesis is unlikely" but rather "if the null hypothesis is true then this data is unlikely"?  Isn't conflating those two the classic mistake in statistical hypothesis testing?
If two people run the a random forest on the same dataset, they are not guaranteed to have the same generated tree structure. How would the decoding party know the proper lookup table to decode the data with this uncertainty? Secondly, conditions about what stops the training also have to be taken into consideration (ntrees, etc).
Hi Glen, Yes you can test any part on any machine.
Thank you very much! That clarified the impact of changing variance.
What does MMR stand for?
I strongly dislike the reference to black boxes here. In choosing an SVM, it's highly likely, possibly even the central motivating factor, that the analyst wants a *transparent box* not a black box. SVM is just an algorithm, like logistic regression, perceptron, decision trees, etc. None of these things is inherently black-box-like at all.
@loganecolss - I think $ H_9 $ here would be closer to the truth than any others.  Overfitting is more closely linked with the sample size and the type of model structure that it can provide support for (sometimes called the "sure thing" model).
Oh, I was too fast :) Of course, I know, that p.value of t-test WILL be near 0 as they really differ, so this doesn't affects my results. My questions concerns the tests that may fail - Shapiro and F.
For maximum likelihood estimation, you have to take derivative of the likelihood function with respect to interested parameter(s) and equate that equation to zero, and solve the equation. I mean to say the procedure is same as you did while estimating 'p'. You have to do same with 'n'. check this one www.montana.edu/rotella/502/binom_like.pdf
(1) Agreed (2) Could you elaborate/point me in the direction of something more formal so I can get a better sense as to why (3) how so - I'm definitely interested in this, can you expand? (4) there is no theory for the application I'm working on, and all IDV's are variables we felt were likely to impact the DV. We want to establish an importance heirachy and deal with the issue of collinearity.
That is a great reference and clearly answers the question.  I think your reply will be better appreciated if you could summarize the answer here: that makes this thread stand by itself.
dear statisticians, I just want to let you know this question has not been abandoned. I will get back to this specific problem as soon as I find time and motivation (read: boss tells me to do this) and will investigate your precious answers and eventually mark one as accepted (which for me will mean "actually implemented").
See my edit above
I've used glmnet and lars packages in a couple of projects. In my limited experience I've had A LOT of problems implementing glmnet. I think that glmnet needs some bug fixes regarding the type of variables used in the data frame. Besides, glmnet has confusing documentation. I ended up using lars, and I was very satisfied with the results. Nevermind the size of your problem, I think lars can handle it.
This is just a partial answer: even when you create the dummies explicitly (rather than using the software's implicit capabilities), keep them together in all analyses.  In particular, they should all enter together and all leave together in a stepwise regression, with the p-value computed appropriately for the total number of variables involved.  (This is Hosmer & Lemeshow's recommendation, anyway, and it makes a lot of sense.)
@Tim If we are taking ourselves seriously as Bayesians, the prior should be a representation of prior knowledge and should **not** depend on the data. In practice, people often choose priors in a way that depends on the data, EB being an extreme case. Depending on the circumstances, this can be anywhere from approximately Bayesian to totally non-Bayesian.
Thanks! Very useful link.
@whuber: in my first comment: latter = char.function. I didn't notice that my formulation was ambiguous. - In my answer, the ridge property as formulated there (with $\\Re$ rather than absolute values) applies to the CGF, consistent with the quadratic case. Whereas the correct ridge property of the exponential must have an absolute value.