tttjjj/elt_bench huggingface dataset:
"In other words, where $\\hat{\\beta}_0$ is the MLE for $\\beta_0$, will the profile likelihood be given by the following?" Depends what you mean by MLE. The hat notation is usually used for the global MLE. The profile likelihood needs a kind of "conditional" MLE given by  $\\tilde{\\beta}_0 (\\beta_1) = \\beta_0 \\, \\mathrm{such \\, that} \\, \\frac{\\partial \\log L}{\\partial \\beta_0} = 0,$ whereupon $L_p(\\beta_1) = \\log L(\\tilde{\\beta}_0(\\beta_1),\\beta_1).$
"the underlying concept is always the same"? For random variables $X$ and $Y$, independence has nothing to do with orthogonality. $X\\cdot Y$ may not even make sense.
I downvoted because your answer reads to me like: Because history. Brief recapitulation of your question.
Last time I went on the same cruise I found books on ARIMA rather than articles :)
I have found the book by Maxwell and Delaney, and having read already 20-30 pages, I must say it is very nice... I'll keep on reading and I think I'll find the answers I'm looking for, thanks!
You're describing the [binomial distribution](http://en.wikipedia.org/wiki/Binomial_distribution).
The conclusion is false.  Did you perhaps mean to stipulate that $Y$ is a random variable that takes the values $\\{-1,1\\}$ with equal probability (of $1/2$ each)?  Regardless, please consult our [help] about asking questions and kindly tell us what attempts you have made at this question and where specifically you could use some help.
Not a complete answer but I had these slides at uni: https://patterns.enm.bris.ac.uk/files/lecture10-2010.pdf
I'm not sure I understand what you're discussing, since a set $L$ isn't a random variable. If I have $L=\\{1,2,4,17,190\\}$ what makes $L$ uniformly distributed - rather than (say) merely equispaced?
Particle filters and sequential Monte Carlo are perfectly equipped for dealing with hidden Markov chains with continuous state spaces. See e.g. <a href="http://www.amazon.com/gp/product/1441923195/ref=as_li_ss_tl?ie=UTF8&tag=chrprobboo-20&linkCode=as2&camp=217145&creative=399373&creativeASIN=1441923195">Inference in Hidden Markov Chains</a> by Cappé, Moulines and Rydén.
Under rather mild assumptions, (a) Bayesian estimation procedures are admissible and (b) all, or almost all, admissible estimators are Bayesian with respect to some prior.  Thus it's no surprise that a Bayesian confidence interval "yields the same or better results."  Note that my statements (a) and (b) are part of the *frequentist* analysis of rational decision theory.  Where frequentists part company with Bayesians is not over the mathematics or even the statistical procedures, but concerns the meaning, justification, and correct use of a prior for any particular problem.
Your use of the word "seasonal" might confuse people who are used to using it in a technical time-series sense. In your case what you are looking for merely has an accidental relationship to a periodic "season": you want to develop a model that relates heating needs to *temperature* and use that to assess whether your energy consumption is going down or up *after controlling for the temperatures*. Having said that, I'll admit that there is a truly seasonal effect lurking here but it won't be possible to find it until such a model is developed. (I use one for my own house.)
This use of ks is incorrect, as its manual page points out: when you set the reference mean and sd to those of the sample, you greatly inflate the p-values.  (Do a simulation and check!)  That's why the p-value in your example is so unusually high.  `f <- function(n){x=rnorm(n)}; ks.test(x,pnorm,mean(x),sd(x))};histogram(replicate(1000,f(133)))`
@jamesstanley thank you for the great edit.
[This question](http://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval) may be helpful.
Multiple imputation is conspicuous in its absence from the linked paper. It's not mentioned at all.
Are these "certainties" confidence or credibility intervals? If so, just call them that (and drop the notion of "certainty" in the former case and append "given the posterior is correct" in the second).
Thanks a lot. The answering and reference thread are really helpful.
If you want someone on another site to explain their notation, you should ask them. There are conventions in finance, for example that won't be standard in all of statistics. I'll say a few things, though: $\\bar x$ means "the sample mean of $x$'s". The poster shouldn't be using $\\sigma$ if they're referring to a sample standard deviation (and if they do mean a population s.d. they don't have a $t$).
I plan on using R for this analysis. I have 8 years of data, and on average data for 10 birds/year. Separating by year does matter because of how variable the environment is. Time is part of the question, since obviously the more time a bird has, the higher the chances of it actually nesting are.
Given the differences in duration, & hence total dwells, is the spread of proportions for different participants unexpectedly large within each condition? Check the model diagnostics - unusually large deviance is a tell-tale sign.
Are you familiar with the [Elo system](http://en.wikipedia.org/wiki/Elo_rating_system)?
In case it's overlooked, please note that the comment thread to another reply indicates (a) these covariance calculations are fine (understanding that they involve a natural but unstated block matrix notation) but (b) we cannot validly conclude that the linear combinations are *normally* distributed until we make an additional assumption; namely, that $X$ and $Y$ have a *joint* multivariate normal distribution.
@cardinal, thanks for the heads up. Lucasng, sorry about the confusion; I have put the group name back.
I was going to answer as per dave's comment, but cross-validation assumes you are building a predictive model, as opposed to solely ranking the variables. In this case, AdamO's answer seems appropriate. If on the other hand you are ranking variables as the first step to building a model, then using cross-validation to find the optimal lambda value becomes appropriate once more. It depends on what your ultimate goal is.
I don't think it is useful to compare the theta of ZINB and a NB as they are different models. Apart from that, it really depends on the parametrization. In R's MASS, the relationship is the following: For a fixed $\\mu$, the larger theta gets, the smaller the variance becomes. If theta is infinity, then the variance becomes $\\mu$ (and then it is a Poisson). In other parametrizations (direct parametrization) a larger theta might mean larger variability. See Hilbes's post here http://stats.stackexchange.com/questions/10419/what-is-theta-in-a-negative-binomial-regression-fitted-with-r
The probability of selection is a per-draw probability; the probability of inclusion is a per-sample (all $n_1$ draws) probability.  This doesn't completely answer your question, hence its presence as a comment, but may get you part of the way there.
Is there a relation that median regression goes through at least k+1 points? K is number of parameters without constant.
What on earth does the question have to do with _normal_ distributions as you start off in your very first sentence, and why are you calculating z-scores at all? If the competitor's bid is **uniformly** distributed (your words, not mine) between 10,000 and 15,000, the probability that your bid of 12,000 will be higher than your (single) competitor's bid $X$ is $$P\\{X < 12000\\} = \\frac{12000-10000}{15000-10000} = 0.4$$ and to maximize your chances of getting the property, you should bid 15,000 or more.
At least 6 is getting into a barely useful number of cases. But if there are numbers most definitely they are important. It can completely change the answer to the question. You should really the edit the question with much more information.
Good answer, but a likelihood is not a "random function" at all, but a "measurable function" telling you not the probability of anything, but only relative probabilities.
As already said--I don't think it matters.
I edited the answer to include a plot in `R`.
except that you don't actually need to compute [$(X'X)^{-1}$](http://stats.stackexchange.com/a/1882/603)...
Welcome to CrossValidated. You may profit from reading [this page](http://stats.stackexchange.com/help/how-to-ask) on asking a good question. What have you already tried?
I read your answer in full, but I'd vote it up even if all you wrote was the first paragraph: "I mistrust all but the lowest-level functions in Excel..." :D
Method 1 does not give a unique mapping. Method 4 will not give a unique mapping if you reduce the dimensionality to 1.
Do you mean whether certain variables need to be added as continuous or as categorical (discrete) predictors in a regression type model?
@IMA: it's only a simplification: in reality, I have 56 data points in Group1 and 64 data points in Group2. BTW, they are not necessarily normally distributed, which is another issue I concern...
I don't know the answer, but I was browsing in a bookstore the other day and saw a [whole book on ratings](http://www.amazon.com/Whos-The-Science-Rating-Ranking/dp/0691154228/ref=sr_1_1?ie=UTF8&qid=1336131861&sr=8-1). Despite the glossy cover, it's a serious book (both authors are math professors)
@Frank Harrell +1. A rule of thumb that frequencies be >1 was cited by Harold Jeffreys, _Theory of probability_, 1939, pp.88-9, anticipating the modern consensus. Jeffreys did a lot of calculating as well as being a first-rate mathematician.
I can't say anything about the `R` procedure you used. But here are some hypotheses: 1) You used stress (fit) by _squared_ distances whereas you shouldn't; 2) You used nonmetric MDS which not always give the same results as metric even with euclidean distances; 3) your input distances are on earth's convex surface, but your map is flat; 4) some misspecification of the command.
+1 Very reminiscent of the Richard Feynman quote: "You know, the most amazing thing happened to me tonight. I was coming here, on the way to the lecture, and I came in through the parking lot. And you won't believe what happened. I saw a car with the license plate ARW 357. Can you imagine? Of all the millions of license plates in the state, what was the chance that I would see that particular one tonight? Amazing!"
"Now once the marketing folks got involved it's Justin Bieber of statistics." - Very nice.
DCC is a two step process. The first step is to run a GARCH model. This model can use any appropriate lag but usually a GARCH 1,1 model is sufficient. The second step is to fit the covariance matrix to the residuals of the GARCH model. The paper by Engle (2002) explains the mathematics and both Matlab and R packages automate the entire process. The CCC is the common correlation coefficient.
They are perfectly correlated by the invariance property of correlation (second paragraph [here](http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Mathematical_properties)). The answer by @IMA provides the computational details.
**There are several things fishy about this summary:** [Last] and [Score] range from $0$ through $100$, so how did they take the logarithms of the zeros? Moreover, because it is clear that [Last] has some *negative* skewness (notice how close its mean is to its *maximum* value), using its logarithm would usually not improve anything--it would only give the extreme low values more leverage. Furthermore, it is strange that "Model 4" would include interactions of [Win R] and [Win F] with other variables without actually including [Win R] and [Win F] themselves.
And as Y is True or False (1 or zero), ϵ will be contained in [-1 ; 1] am I right ? So I also need to find a treshold in those values in order to perform this classification ?
@qed It is an *expectation,* not a sample correlation. For instance, the expectation of a Bernoulli$(p)$ variate is $p$ but the only values that variable can attain are $0$ and $1$.
This question appears to be off-topic because it is purely about coding in R. Please review advice in the Help Center on software-related questions.
What is locality information? I'm sorry for ignoring it in my answer because I know very little about biological sequence...
Where is the multiplication by $-1/2$ in the exponent in the first calculation? Am I just not seeing it?
I don't think that @Niku has answered your question about what the "sample of $D_2$" is.  Another interpretation is this:  that there is a SRS from $D$ of size $n$, and it contains $n_1$ elements from $D_1$ and $n_2$ from $D_2$. If this is the case, then conditional on $n_1>0$, the elements from $D_1$ do constitute a simple random sample of size $n_1$ from $D_1$. The proof is given on page 35 of Cochran (1977).

Reference: Cochran, W. G. (1977). Sampling techniques (3rd ed.). New York: Wiley.
Check the first paragraph [here](http://en.wikipedia.org/wiki/Exact_test), and then this [section](http://en.wikipedia.org/wiki/Exact_test#Example:_Pearson.27s_chi-squared_test_versus_an_exact_test)
+1 for highlighting the distinction between model-based clustering *vs.* purely distance-based unsupervised clustering.
There is an entire field of statistics dedicated to working with unknown distributions.  It is called 'nonparametric statistics.'
(@xmjx Supplied the first example last year.)  I love the astrology example.
Does it answer your question or is there still some aspect of your question you feel is still unanswered?
As a rough-and-ready rule of thumb, probabilists tend to use $\\Gamma(t,\\lambda)$ to denote a Gamma distribution with mean $\\frac{t}{\\lambda}$ (that is, $f(x) = \\frac{\\lambda}{\\Gamma(t)}\\cdot (\\lambda x)^{t-1}\\exp(-\\lambda x)\\mathbf 1_{(0,\\infty)}$ while statisticians tend to use $\\Gamma(\\alpha,\\beta)$ to denote a Gamma random variable with mean $\\alpha\\beta$, not $\\alpha/\\beta$ the way you have it.  [Wikipedia](http://en.wikipedia.org/wiki/Gamma_distribution) describes both conventions.
+1 - The advice to look at the [Anscombe quartet](http://en.wikipedia.org/wiki/Anscombe%27s_quartet) seems particularly valuable in this context. Similar sets with higher correlation can be constructed.
Please consider accepting my answer, if it answers your question.
As @Elvis noted in comments to the original post, $\\beta$ is an upper bound when $\\beta > 1$, which is one of the conditions stated in the OP.  Therefore you can set $M = \\beta$.  Note also you don't need an upper bound for $M$, you need an upper bound for the pdf, and $M$ is some number $\\geq$ that upper bound.  (Smaller is better because you get a better acceptance ratio, which is directly related to runtime, but you don't HAVE to set $M$ equal to the sup of the pdf.)
$t$-tests can be corrected for skewed populations, see http://www.jstor.org/stable/2286597 -- the proposed corrections worked well with quite skewed populations like $\\chi^2_1$ with 10 observations.
Hi,  its the same construct being measured in both cases.... but i'm either using a categorical measurement or a continuous measurement (i.e how much more pleasant is  thing a than X - 'less pleasant or 'more pleasant'  vs if x is 50 rate A on a scale of  0 - 100.
And how are they not independent?
The link I provided gives the expression
$$
\\lambda_U = \\lim_{u \\to 1} \\frac{1 - 2u + C(u,u)}{1-u},
$$
where $C$ is the copula (not "coupla") associated with $X$ and $Y$. Obviously, $\\lambda_U$ depends on the copula, but not on the margins. @ocram's answer shows a way to derive this expression from the definition of $\\lambda_U$. Please clarify if you need more information.
Thank you "Extreme value theory" was what I was looking for!
Does a feed forward network that iteratively uses its outputs as inputs count as a recurrent network?
As can be seen, you only have one numerical variable (`age`, because `sugar` will actually be treated as a factor): How would like to summarize it with other factors (quartiles, median, mean)?
(+1) Good, concise answer. I'd be interested in hearing back from the OP that this is, indeed, the question he was seeking an answer to.
Thank you. You are absolutely right. Actually FMZ is also right. "Per unit change" for a proportion type variable should always mean change by 1 (or 100%) unit in the independent variable. Just that for my example, Y goes down by 1.3 units when X (measured in proportions) increases by 1 unit also implies that, Y goes down by 0.013 (1.3/100=-0.013) units when X (measured in proportions) increases by 0.01 (1/100=0.01) units. That makes it consistent with the sense of proportions. Thank you all.
I was wondering about the principles behind the logic (in my quest to understand PCA). I used R tag because R people might read this and maybe show R examples. :)
It worked!! Thanks a lot, @topepo, and great job with the package! I noted, however, that if i pass an index with the method="knn", it gives an error. Is that expected?
Most non-statisticians, and 40% of statisticians do not realize the severe damage that variable selection inflicts upon the final analysis.  A partial cause is "phantom degrees of freedom".  I cover this in my book Regression Modeling Strategies.  Whenever possible, a good strategy is to formulate a model based on subject matter knowledge and guided by data availability, fit that model, assess its fit, use the bootstrap to validate its accuracy and the absence of significant overfitting, then use that one model for inference.  Root cause: var. selection was invented before simulation.
http://en.wikipedia.org/wiki/Multivariate_normal_distribution
@RioRaider Because Cox methods were not *necessarily* only designed to deal with censoring. The OP mentioned covariates - while you might still be able to look a median times over a small number of strata, what about a continuous covariate? Just because Cox models can handle censoring doesn't mean they're inappropriate for questions not involving censoring.
Generally, in terms of an R/Sweave workflow, RStudio is perfectly functional now and rapidly maturing. No bib support but I've always preferred using a citation manager that syncs to a .bib anyway. It is IMO a better choice for these tasks if you don't already know (or want to *really* learn) Emacs. Of course really knowing Emacs pays other dividends...
Do you know the time of the structural break and what will your model be? I am not clear on your question. Are the uses of "alternative" variables the explanatory or regressands? In that case the Chow test - beeing an F-Test - gives you a lot of opportunities with your dummys. Otherwise you are looking for multiple regressors in one model?
Certainly it would depend on the initial distribution.  You could probably come up with some distribution that converged very very slowly to Normal.
Yes, but that's basic ggplot2 functionalty. Study the documentation and tutorials. You might want to use `scale_colour_manual`.
Thanks, this does the job (leaving out some data so that the Firefox line works):

    ggplot(subset(programs, !(Program == "Firefox" & Version < 4)), aes(y = Version, x = Date, colour = Program)) + geom_point() + ylim(0,30) + xlim(as.Date("1985-01-01"), as.Date("2015-01-01")) + stat_smooth(method = lm, fullrange = TRUE)
The units of variance are *squared*: one squared hour = $60^2=3600$ squared minutes.  It's exactly like converting from square miles to square feet.
Very helpful Sepehr, thanks a lot.
Both, customers are the websites.
"Estimate" is the key word here.  The entries in the Markov chain are nothing but the conditional probabilities $Pr(X_t=i|X_{t-1}=j)$.  That means you can just scan the series and find the proportions of zeroes followed by zeroes/ones and ones followed by zeroes/ones to estimate each row in the matrix.  Going row by row, the usual confidence intervals obtain.  Nice implementation.
Thanks to @Rolando2 for his insightful comments.  It seems that I should definitely check out Corrected Item-Total Correlation; we were using spearman's ITC, which was each item's correlation with the sum of all (and not of all _other_) items.  

I am still wondering if using cronbach's alpha in the way that we mentioned - as a test of internal consistency, by comparing the statistic with and without the relevant item - is an appropriate way to judge these questions.  I've heard here and there of G6 and omega but couldn't turn up much.
That is correct. $R^2$ is a relative measure that bears only on the particular data set at hand; recall that is is, essentially, a measure of "gain" in terms of how much of the variance the model explains, measured against simply using the mean. Different samples have different means, so you'd be comparing apples to oranges.
+1.  The same authors have [a more recent paper](http://books.nips.cc/papers/files/nips22/NIPS2009_0817.pdf) on representing bags of words as well.
@JJ01 I've merged your two unregistered accounts. Please, [register](http://stats.stackexchange.com/faq#login) this one.
Wikipedia thinks it's an [overline](http://en.wikipedia.org/wiki/Overline), not a macron.
You have lines with different intercepts. The correlation seems  to be about the same within each group. Your question is unclear, but modelling such data is simple.
Many thanks for your very comprehensive answer.
thank you for your clarifying answer
Could you please be more specific about your "shuffles"?  The only ones that are meaningful for correlation matrices *simultaneously* permute rows and columns, but you haven't really mentioned the columns. That is, for a symmetric $n$ by $n$ matrix $(a_{i,j})$ and a permutation $\\sigma$, the correctly "shuffled" matrix is $(a_{\\sigma(i),\\sigma(j)})$, whereas your method seems to indicate you have obtained $(a_{\\sigma(i),j})$ for $i\\ge j$ and $(a_{i,\\sigma(j)})$ otherwise.
@Procrastinator Thanks too. I have also deleted my comments and added the point about the Jeffreys prior in my code.
The intent is that you would winsorize before standardizing, that's kind of the point, since outliers would have a dramatic effect on the standard deviation (even more than on the mean). In fact with a single sufficiently high outlier, your scaled sample would be essentially $n-1$ copies of $-1/\\sqrt{n}$ and one value of $(n-1)/\\sqrt{n}$. [My comment shouldn't necessarily be taken as support for the approach here.]
Good question - including from a stats/visualisations view, not just programming - but I'll be interested if anyone can help - it looks like a workable solution will be very much dependent on your actual data and getting it to work within the chart estate.
You can use a multilevel or mixed model. What software do you use?
Just for my learning, how is the data nested? By location (i.e. multiple water levels around a lake)?
If you look closely at the figure (about one-third from the left) you can see where the minimum-distance heuristic is clearly violated.  The trick is to keep the *largest* distances pretty small.  Note, too, that this formulation is insensitive to the dimension of the original K-means problem: all that matters are the mutual distances between the two sets of coefficients.
@Iterator Thanks for the illuminating comments.  It would be a little more accurate, and no more computationally difficult, not to bin the data at all.  BTW, we don't need the measurement error to be heteroscedastic to explain what's going on: the slope of the fitted CDF suffices.  It must be an unusual situation where a beta could be adopted *a priori* as a reasonable model for a set of distances, leading me to suspect that the decent fit here may be an accident.
Methods of interpolation have been used for this over several centuries. Popular methods include nearest neighbour, linear, cubic, cubic spline and piecewise cubic Hermite. http://www.mathworks.co.uk/help/matlab/ref/interp1.html is a handy overview, but implementations are by no means restricted to MATLAB.
This is an extremely general and broad question and as such cannot reasonably be answered here. Please narrow it to a specific question (following the guidelines in our [faq]).
I posted an answer there.
Could you please clarify what would constitute an "explanation" of a PDF?  Do you mean a mathematical derivation or perhaps are you looking for insight into its principal qualitative features?