tttjjj/elt_bench huggingface dataset:
The existence of temporal correlation in weather patterns suffices to make F the only reasonable reply. Even if we take this to be an abstract question (with "weather" as a metaphor merely to make it more palatable, and with a perfect determination of all probabilities involved), because the question provides no information about the independence or lack thereof of the Saturday and Sunday events, F is still the correct choice for the same reason.
I am doing 10-fold validation. I was also expecting a graph like the one that you have mentioned but that is exactly what my problem is. I get the graph like the one I have mentioned above. If I decrease alpha more I will find least error at that smaller alpha. So my problem is that reducing alpha seem to reduce the error as if alpha = 0 should be the best option. Also R2 fit value does not seem to vary at all with different alphas.
for what it's worth, this is really just a linear model (`lm` in R); since you're not specifying `family`, it defaults to Gaussian family.  Technically you're right that it's a generalized linear model (since linear models are a subset of GLMs) but it may muddy the waters slightly.
No, not formally -- LaTeX submission is encourages but if you look at the [instructions page](http://www.jstatsoft.org/instructions) it does not contain the word Sweave.  Authors do use it and/or ship the R code with the paper, but to me this echos Shane's point about package vignettes.
Thanks, I'll look into this, and see how it goes.
Don't dichotomize $Y$ for any reason.
OK, the question I'm trying to answer is given a list of genes, whose expression are most variable among a group of people. You can imagine I have a big matrix with each row representing a gene and each column representing a sample. So I have many genes and many samples. I want to find out those genes that show variation (I don't know the variation characteristics yet). I have now a list of genes who I defined as invariable (basically all of their values are above 0.8 for example) I thought I can calculate the variance for these genes and consider them as a null distribution
+1 I love illustrations, especially those that could stand on their own as solutions (even absent the words). But in this case wouldn't that suggest drawing *horizontal* dashed lines between the indicated points and the vertical line through $\\bar{x}$? The vertical dashed lines appear to be relevant only by virtue of being directly proportional to those horizontal distances.
This question needs to be clarified to be on topic here, because it is too broad and unfocused.
thank you for your answer. When $\\sigma\\rightarrow 0$, the dimension of the Gaussian kernel projects would increase. And by your inspiration, now I think it is not equal. Because, using kernel just handle the situation that linear classification does not work.
I've used various EEG analysis packages to accomplish the same thing, but never in R -- see if the "bivariate" snippet here is what you are looking for http://stat.ethz.ch/R-manual/R-patched/library/stats/html/spec.pgram.html
How many categories does Placement have?
Depending on what you mean by non-parametric, boosted regression using the gbm package might help.
+1. You can actually restrict this to the strict $j \\lt w$. I think you could write the last expression as $$\\frac{w}{n-w-1}\\left(1-\\frac{w-1 \\choose j}{n \\choose j}\\right)$$ though it may not be an improvement
Is there a reason you can't simply correlate the two variables w/o categorizing them into quantiles first?
*Because* the output is completely inconsistent, I really can't tell if that's correct. If you give me an output that's consistent, I can *check* that it is right. Otherwise I'm left to presume it's doing what it looks like it's doing, which it might not be.
This got me off to a good start.  I'm modifying the independent variables a bit and some other parts of the code to match my setup, and I'm thinking about how to modify the `df$y` bit of code to match the various dependent variables I'll be measuring, which are counts in most cases, but also response times in others.  The `df$y` modification is something I'll have to think and read more about in order to make the `df$y` values appropriate.  This was really helpful and taught me several new things.  Thank you.
Quota sampling is not any good, anyway. There must be a probability element leading to selection of your units, and if there isn't any, it may be a great sample to study the phenomenon qualitatively, but there is no way to generalize it to any other population or its subgroup.
@gugn thanks for your comment. I tried to make things more clear.
This may help: http://stats.stackexchange.com/questions/89838/what-does-pca-principal-component-analysis-spheres-the-data-mean
Any method that dichotomizes a continuous variable (in this case the logistic model's predicted probability) is highly problematic.  Now couple that with the fact that the threshold is completely arbitrary and the proportion classified correctly is an improper scoring rule (i.e., it is optimized by a bogus model) you have a perfect storm.
Nice book!
I really think you should post the follow up as separate question.  You can link back to this one if you want.  The second question is really a separate issue, and I have a lot I want to say about it, but it belongs in a separate topic.
and when keep the spread constant and decrease the x's, the standard error of the intercept becomes smaller, which makes sense.
Is this homework or an example from a book? If so, please add the `self-study` tag.
@20826 - fixed the error
Joshep, if I understand your question, the objective of your work would be to demostrate that running regression analysis on random samples you obtained simular results to those from the entire data set if the exchangeable assumption holds. My question is if someone knows any reference in which this method has been used.
This is very useful to me.  The function 'collapsed_gibbs_sampler.m' can be run with very simple arguments, and then compared with algorithm 3 in Neal's 1998 paper.  Thanks!
@mb3041023 Googling "intervention analysis" and "interrupted time series" has provided me with a wealth of new information and I am now able to attack my problem with confidence. I found a 2010 paper by Lagarde titled "How to do (or not to do)...Assessing the impact of a policy change with routine longitudinal data" to be particularly helpful. Thank you.
Differentiation won't take you from density to distribution.
Could you confirm that (a) d1 and d2 are the side lengths (and not angles); (b) that you are assuming the angle between them is a right angle (for otherwise the atan formula is suspect); and (c) that you are interested in the distribution of one of the other angles of this right triangle?  Also, presumably, the SD of each length distribution is much smaller than its expectation because the triangle shouldn't have any appreciable probability of a negative side length :-).
@teucer. I agree, I am a very happy and statisfied Rcpp user. But I was looking for an R-ish way of doing this. It would be good to see the magnitude of speed-up Rcpp provides.
Basically, the problem becomes an uninteresting exercise in analysis because we have to restrict the domain of the estimator to $\\mathbb{Q}$, even though "most" parameter values are in $\\mathbb{R} \\\\\\mathbb{Q}$.  It is about as interesting as saying the recursion $a_{n+1} = \\frac{1}{a_n} + \\frac{a_n}{2}$ has no limit over $\\mathbb{Q}$, even though it converges to $\\sqrt(2)$ over $\\mathbb{R}$.  The main lesson is that probabilists should not teach statistics.
That helped a lot - thanks Glen.
I can't get enough of pop-charts .. more pls.
Actually, the help very clearly states what happens when you specify `y`: "*[...] Otherwise, `x` and `y` must be vectors or factors of the same length; cases with missing values are removed, the objects are coerced to factors, and the contingency table is computed from these. Then Pearson's chi-squared test is performed of the null hypothesis that the joint distribution of the cell counts in a 2-dimensional contingency table is the product of the row and column marginals.*"
(+1) Welcome to our site!
This is indeed exactly what I wanted, thank you very much. I am still missing a bridge here. In this document http://mhtlab.uwaterloo.ca/courses/me755/web_chap7.pdf I found that in 2F1, 2 − refers to number of parameters in numerator and
1 − refers to number of parameters in denominator, which emerges when you look at $$
\\sum\\limits_{n=0}^\\infty \\frac{(a)_n(b)_n}{(c)_n} \\frac{x^n}{n!}
$$ Why is it important to give a hint about the number of parameters in the denom/nom?
Thanks! This is a good hint! I'll keep myself busy in studying cross validation! I know the BIC criteria, but unfortunately it doesn't work with glms (I am using R) and I've heard it should not be used if you are not using a OLS method. Might be a rumor... At last a personal question: What would you recommend to use? AIC or Cross Validation?
Interesting and tough question and you've put your finger on some of the key issues.  How many judges in total, so how many projects would each judge judge? (some idea of the range as well as average would be good).  Also, are the 26 categories mutually exclusive?  I have a hunch the best answer may be just to do the average for each project, but it might be possible to adjust for judges.  I'll read the answers with interest!
how many data points do you have?  also you may have separation if the implicit contingency table has 0 or 100% observed counts.  this leads to infinite mles and ill conditioning in the newton rhapson scheme
So basically ignore the parameter constraints, maximize (which in this case can be done by differentiation), then somehow project onto the constraint set? I don't think that this will in general give the best value.
If you have a reason to branch on 1 or 2 versus other values, you should do so. However, you have indicated that pattern is just observed in your sample, not guaranteed, so there is more to be said. But I don't think you have given us enough information to allow us to add statistical advice.
+1 These simple questions can be tough--they make one think hard about basic concepts.
+1 Nice exposition
a well formulated question plus showing the amount of research you have already done. Wish more questions would be like that one.
I don't think that is true. Imagine if the distribution, T, you selected is way off from K or M..then the ratio of likelihoods would not be very useful in estimating the ratio. The likelihood under a given model is only indirectly related to the true entropy of the distributions.
the log odds CI will be symmetric
The **parameters to be estimated** are (multi-)linear. If you were *estimating* the values of the exponents, the estimation problem would not be linear; but squaring a predictor fixes that exponent at precisely 2.
I am wondering if your specific application is related to financial markets. If you would like to explain a bit about your application, I may be able to assist you with some more comments about the likely causes & "meaning" of the low MCC results.
Is `CTR` simply `Clicks/Impressions`? If so, why not say so in your question?
"Three solid gold (significance) stars on the main effects will do very nicely, thank you, and if there are a few little stars here and there on the interactions, so much the better!" -Venables 2000 [N.B. said in jest, but this attitude is so pervasive in research, peer review, and graduate training is so great I thought it would be edifying to include it here.
On rereading your question I wonder: by $exp(\\mu)$ do you mean $e^{\\mu}$ (i.e. something like $e$ to the power of the mean of your data), or do you mean something like E$(\\mu)$ (i.e. the expected value of your data)? I will alter my answer if the second.
The book by [Pinheiro and Bates](http://www.amazon.com/Mixed-Effects-Models-S-PLUS-Statistics-Computing/dp/1441903178/ref=sr_1_1?s=books&ie=UTF8&qid=1400315348&sr=1-1) contains several examples. You could look at §4.3, "Examining a Fitted Model".
@MichaelChernick The threshold T is local to each component y (normally I should write $T_y$ instead), and it is not fixed since it depends on "the mean distance ($\\bar{d}$)" of y to data points that were already assigned to it (this mean that each time a new datapoint x is assigned to y, its associated $\\bar{d}$ will be updated) ! I understand that we can compute a "good" sigma (or say $\\sigma_y$) for a given component y through the integration P(d > $T_y$), but is it possible when the $\\bar{d}$ associated to this y is not fix ?
So weird.  I googled for "derivation of normal distribution from binomial" without quotes, and that was not in the first 4 pages of my search results.  Now that I've clicked through your link it's on page 1, hit number 8.
@Dimitry, thank you for the trouble but unfortunately this does not work for my data.
@ceiling cat 1) $E[h(X,\\bar y)] = \\int_{-\\infty}^{\\infty} h(x,\\bar y) f_X(x)dx$ is correct because essentially you do _not_ have _two_ random variables any more. Likewise for fixing $X$ to $\\bar x$.
Could you clarify the ambiguous notation? The other information in this suggests, without actually saying so, that (1) $u$ denotes a function rather than a constant; (2) $t-\\theta$ is its argument rather than a multiplicative factor; and (3) $a$, $b$, $c$, and $\\theta$ are fixed (but unknown) parameters to be estimated from a *bivariate* time series $(U(t_i), Y(t_i)), i=1, 2, \\ldots$ where the $(U,Y)$ are assumed to differ from the (unknown) $(u,y)$ by some kind of zero-mean random process. (What are the assumptions about that process?)
Look [here](http://stats.stackexchange.com/questions/56645/what-is-the-fastest-method-for-determining-collinearity-degree) for example.
Also the google scholar link is broken, what were you linking to?
Hi Peter. Yes, in my line of work, that's a huge difference. I guess my problem with the plot is that if it deviates from a simple back of the envelope calculation, I have to be able to understand completely why. Otherwise, I won't be able to use it. Sadly the person who wrote the code for cdplot gives an unpublished manuscript as his reference and I couldn't find that manuscript online.
@Skyde, I thought that you meant that you wanted to have a table with, say group (1 or 2) along the top and demographic (men, women, or children) along the side, with, say, average wage for each cell inside. That would work. I'm not sure what you're trying to describe now.
Thanks! It's perfect.
@David, what you describe is [least squares](http://en.wikipedia.org/wiki/Least_squares) and the solution $x$  is given by $x=(X'X)^{-1}X'y$, so it does not matter whether $X$ is restricted or not. Could you please clarify whether the constraints are on $x$ or on $X$? If there are linear inequality constraints on $x$ then user603 answer applies.
Thanks a lot! However, What I want to do have more than 3 variables, and want to just 'control' variables except X and Y. I re-upload question with the picture I found, describes exactly what I want to do...[link](http://stats.stackexchange.com/questions/78529/how-to-make-linear-regression-model-while-control-confounding-variables)
@EvanZ: don't forget to accept this answer if it solves your problem ...
The if statement is not necessary here. According to the description, in the "actual" sample the dependent variable is not known. Hence, these observations are not used in the logit. Thus, by default the logit only runs over the "historical" sample.
It makes *sense,* but in most applications it would not have good properties. As an example, when you change how your data are measured (such as changing from meters to kilometers or pounds to ounces), the slope in the linear regression changes in a simple predictable way. That will not be the case for the angle that you compute. As such, unless there is a unique or conventional choice of units for both the x and y variables and there is a natural interpretation of that angle, the angle is likely a poor choice for a numerical representation of the slope.
Thanks. My goal is actually a bit different. I want to compute power to reject the null hypothesis (the regression model). I'm generating data using different (other) models and compare them in terms of how easily can you detect that they're different from the null model - but I assume that the statistician performing the test doesn't know the alternative model, so he can't apply a log-likelihood ratio test between the two models, he just needs a test for rejecting the null model and the goal is to see how powerful is this test under different alternative models
As a point of terminology, I wouldn't use the term "cut-off" for an interior point. As a point of statistical interpretation, your arrow points to a minor antimode, but just from (a) general data analysis experience (b) complete ignorance of your application, I can't see any reason to take it more seriously than any other such detail. I would want to know much more about the raw data (e.g. any granularity in recording), to see results of varying bandwidth and kernel in density estimation, and to see such a feature being repeatedly reproducible in different datasets.
Thanks for this, @JuanRulfo! I will definitively take a look. Hopefully, I have access to the cited paper. The exercise is related to finance and risk management, and later I am asked to calculate a Value at Risk and Expected Shortfall expression for the function, so I would like to be comfortable with f before calculating these values.
How do you know the +/- 0.1 cm value?  Is it yet another estimate from the individuals estimating the width of the block of wood?
Correlation (of any sort) between a continuos and a binary (group) variable, is not much more (and maybe less ...) than just a comparison of means (some sort of mean ...) between the groups, so usually it should be better to just do that!
*Is this what you are thinking?* -- for me to be thinking anything in particular, I'd have to understand more than I do about what's going on. But if there is something you're trying to measure and you can articulate how it relates to what you have, there may be a way to do something.
Asking for a concrete implementation may be asking a bit much, but other than that, it's hard to see why this question currently has a vote to close as *not constructive*.
You should check out http://cran.r-project.org/web/views/Spatial.html . There are a few packages that have cluster right in the name that would probably be of interest.
It'll be much later today (Pacific Std Time), I'm afraid; I can do it the straightforward way too, which will be less opaque.
+1. A brief but illuminating discussion is contained in Mosteller and Tukey, _Data analysis and regression_, Addison-Wesley 1977. who emphasise that there is a place for vague concepts, which can be made precise in several ways. This may sound vacuous or trivial, but it's the nub of the matter. For some purposes we really do need broad, general terms. Conversely, giving a word a precise definition (as with e.g. variance) means that we should be careful about not using such a term vaguely or broadly.
That URL didn't work for me, but this did:   http://ipsur.r-forge.r-project.org/book/index.html
The reason why I introduce $\\theta$ is this, in the machine learning book I'm reading, given a dataset $x$, and $y$ is the  corresponding target value, so to fit a model to this dataset, I can use MLE to estimate $\\theta$ which is the parameter of the model, right?
@chl I don't like this paper while it made from the SVM learning perspective -- making one repetition of a stochastic algorithm (RF) is just a junk; also appendix 2 shows how bad it may be to apply SVM workflow to RF. Yet I agree that almost always SVM  can be tuned to outperform RF because of the kernel trick (which plain RF does not have, while it doesn't mean it can't have it in general), but with exponentially growing optimization effort.
Welcome to the list. The first two things you should do are 1) Choose a better title for your question and 2) Add the "self-study" tag.
For main effects in regression ("plain vanilla" coefficients), it makes no difference whether you subtract each score from its mean.  Some people claim that it makes a difference if you're including interaction/moderator effects. Some say the coefficients will come out differently; some, the collinearity diagnostics or the standard errors. And then people will go into special cases when one or another claim is true or untrue.  I've seen so much disagreement in the literature about these things that I've almost completely stopped bothering to mean-center.  My own tests have been inconclusive.
I have added my data set. would be nice if you could make a short example with `auto.arima` and my data. I would accept this as an answer. thank you in advance
Great, thank you very much for the edit! I will wait to see whether other answers show up before accepting
Given the connection between Gamma distributions and the Dirichlet, my first guess would be that - at least given appropriate restrictions on the Negative binomials - it might turn out in some cases to be Dirichlet-multinomial.
@cardinal Ok. I will do it whenever I will be less busy.
I think the premise of the question is faulty -- I don't think that it's important specifically to *test* it (who says otherwise, and on what basis?); indeed, some papers clearly indicate that that approach may not be a good idea at all. It is often important when you assume it that it be approximately true -- that's not at all the same thing as being important to carry out hypothesis tests relating to it to justify some procedure. Obvious examples of procedures that rely on it are OLS and related procedures (ANOVA, two-sample t-tests) - for standard errors of coefficients and hypothesis tests
The largest weight is always to be considered the largest in absolute value. I've added some further comments above.
@cardinal, well, I'm not so much in the field, can you please suggest the "right" object?
If you want to estimate it, maximum likelihood estimation, perhaps?
...cont'd... Thus you are *partially* uncertain about the true model.  The procedure above gives you a coherent way to choose a model from this class (p-values give you a quick way).  And yes it is an Monte Carlo algorithm, but it is not a markov chain, because each cycle is independent of the other (more like a bootstrap than MCMC).  Note it need not be a sampling based method, you could also do a "jacknife" based evaluation (split data into $G$ groups, and predict the $gth$ group based on the model fit to remaining $G-1$ groups)
Just to reemphasize what others have already alluded to, it is generally *much* preferred to rely on standard packages and software for computing such things. A lot of the algorithms and code have decades worth of thought and maintenance associated with them. It's certainly worthwhile to understand how they arise mathematically and even some of the details behind the numerical implementation. But, similar to numerical linear algebra, I'd highly discourage anyone from attempting to roll their own implementation.
R's 'step' function is AIC based.
I'm not quite following what it is that's confusing you. Why are you worried that survival analysis might not be appropriate here? Is it that you are looking only at discrete time intervals?
The absolute risk at time t given a covariate vector x_i can be estimated as exp(-lambda(t)exp(x_i'beta), where lambda(t) is the Breslow estimator for the cumulative hazard. I don't know how this is derived though.
@concugateprior That is an interesting thought. So basically what you say is that if sampling effort decreases, it should (under normal circumstances) decrease the likelihood to catch marked as well as unmarked individuals and thereby not affect the marked over unmarked catch probabilities? And by adding sampling_effort as covariate I would actually see if it affects the probabilities or not, right?