tttjjj/elt_bench huggingface dataset:
Yes, true, I wanted to present the general case of
+1 replication/resampling are definitely better than zero-padding.. still I'll wait and see if there are any other ideas out there :)
But wouldn't a correlation between centrality and difference be high, if subjects for example had low centrality and low difference? Then the correlation would be misleading...? Or have i understood correlations wrong?
If for example the $a_i$ and $b_i$ were fixed, then you would have independent identically distributed random variables with a finite variance, so the central limit theorem would apply.  Whether this is a mixture distribution or not does not affect this result.  What I am saying is that you can extend this to cases where the random variables are independent but not identically distributed, provided that the means and variances stay reasonable.
+1 A little detail: your sequence $g_n$ is not monotone. Maybe it needs a little adjusting...  Re centering images: please see http://meta.stackexchange.com/questions/25835.  (Bottom line: it's probably not worth it.)
+1 great, concise answer
Thanks for your reply...I came to know about this model from the this link http://en.wikipedia.org/wiki/Marketing_mix_modeling#Base_and_incremental_volume  ...  I think I can use this method to find the incremental and base volume....Please let me know If you need any other informations
For the 2nd question, you mean the eigenvectors or the component loadings summary?
Yes, this is exactly how I solved this problem.
Just how small is "extremely small"? After, double-precision floats are fine down to around $10^{-303}$ or so.  Probabilities that small won't contribute appreciably to the entropy, anyway.
Isn't this a nested (hierarchical) analysis of variance?
+1 Right; I was still in dark ages of making approximation from histogram.
Alex, I realize that. Like I said, I am not looking for an esoteric theoretical quantity. I am just trying to get an admmitedly _rough_ idea based on estimators that are in common use.
+1 This is the basis for the unequal-variance, unequal-sample sizes formula for the [Student t statistic](http://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_Unequal_sample_sizes.2C_unequal_variances).
This is by far the most comprehensive piece of research I have seen so far, thanks. I also found an earlier paper interesting where they compare their approach to other forecasts methods (Leitner, Christoph, Achim Zeileis, and Kurt Hornik. "Forecasting sports tournaments by ratings of (prob)abilities: A comparison for the EURO 2008." International Journal of Forecasting 26.3 (2010): 471-481.).
@sed so the t-test is not a Wald test?
your probabilities do not add up to 1
I haven't read through - just glanced over - the documents you link to, and I'm not R user. Nonetheless what seems clear to me is that "complexity" is just the number of factors you are testing, i.e. the number of columns in loading matrix A. VSS = 1-ss(R-AA')/ss(R), where R is the correlation matrix, "ss" means "sum of squared elements". By A in this formula the author means _simplified_ loading matrix A, that is, matrix A after you have rounded low loadings in it to zero. If I'm correct.
"there is just nothing to understand here -- it is just a standard linear regression model" - teehee.... like that's any consolation for mathphobics. I take it you've never had the pleasure of teaching undergraduate courses in stats for, let's say, sociology or, god help me, communications majors.
I thknk there's no point "raising" the country level data to the company level, because all companies from a specific country will share the same combination of country-specific variables. Mathematically, that means that $COUNTRY = a_1\\cdot LAWFAM + ... + a_n\\cdot CONAVG$ (where the $a$s are weighting scalars), and this term will be interchangeable in all models (the $a$s will be different for different models, but the same for each row of data for a given model). That means that raising your data is causing you to do more calculation, but not actually giving you any more information.
Mysterious downvoter: see the edit for a clarification. Non-narrow-minded people: see the content and judge yourself.
Thanks for providing the details. A bit of searching produced this link, where `fa` from the `psych` package is used: http://stats.stackexchange.com/questions/31948/looking-for-a-step-through-an-example-of-a-factor-analysis-on-dichotomous-data
(I am currently looking through Google results for "asymmetric entropy" although I haven't hit gold there yet.)
About the flag -- this is no need to remove this post, it may be still useful for others.
Ali, that's your third question dealing with [Gaussian processes](http://www.gaussianprocess.org/gpml/) in less than 24 hours. Still, we can only get an idea of what you're after from this question, [Gaussian process predictor](http://stats.stackexchange.com/q/23767/930). It would be better if you could focus on one particular issue at a time or provide more details on your particular design.
Most of your questions cannot be answered without knowing what all those variables represent and without access to your data.
Intuitively you are correct. But when you try to get proofs for the statistic under null and alternative then you see the difference. Note also that the null hypotheses are a bit different for tests. For ADF, the null hypothesis is random walk, for BP the null is homoscedasticity.
You might have better luck over at the signal processing stackexchange. This is basically their bag. http://dsp.stackexchange.com/
**"You shouldn't use stepwise for _any_ kind of model building."** - this is an incredibly sweeping statement and it just isn't true. If your goal is prediction then a stepwise selection algorithm may be quite sensible. Yes, stepwise selection destroys the interpretation of $p$-values but inquiries where you care about $p$-values does not subsume _all_ model building. Dogma and unconditional statements like this do not spread good statistical practice.
@TimRich yes I've read it, and taken the class in graduate school, and worked in industry doing applied statistics ;) this is a PyMC-specific question.
AUTOBOX can estimate weekly effects (51 in number) with 76 weeks of data BUT the coefficients of the 26 weeks that only had 1 reading might not be as reliable as those weeks with two replications. We normally suggest 3-4 years of weekly data for a weekly model BUT if you only have 1 1/2 years so be it. You can either post 1 of your examples where the ad spend changed or you can contact me off line and I will try and help you and your mentor.
The key question in return is what you would use to impute your missing data here. Missing climate data would best be imputed through some relation with other climate data, possibly a neighbouring area in India if some relation can be found. Pitfalls here include the possibility that the climate is changing; that quite different relationships may be involved over 1961 to 2011, e.g. Green Revolution, new crop varieties etc.; respecting any dependence structure in the climate data.
I noticed if I do Principal Component Analysis and reduce the dimensionality then it manages to converge..
Does the Murthagh and Legendre paper say anything about this?
Are you sure that they are the same coefficients? When you include factors, one level from each is dropped. Maybe a different level is dropped in each of orderings?
There is no unique right way to do this.  My point in decribing what contingency tables do is to indicate that if the tables are set up appropriately they can be used to handle you problem.
Have you thought of using squared correlations?
An answer suitable for beginners is at www.integrativestatistics.com/partial.htm
Upon erasing the parentheses you have a table of $n$ observations and $7$ variables. What is the obstacle to doing PCA with that? (There is no computational or mathematical limitation.) The only ones I can imagine would be (a) feature 3 isn't really a "vector" but a *set* of numbers, so that their order is immaterial; or (b) you don't want to "mix" elements of feature 3 with the other 2.  Is it some such concern that motivates this question?  Could you elaborate on that?
Can you give us an idea of how many words or powerpoint slides you can use to get the idea across?  What's your time budget, in other words?  I can talk for an hour on this subject.  When I lecture on it to undergraduates, I do talk for an hour on this subject.
Just curious:  is a glm with gaussian errors and a logarithmic link function a bad model for your data?
"Good" in what sense?
Thanks @stachyra - Ill have a look into those - The data is in fact stock ticker data, ive updated the chart axis with titles to reflect this
A good explanation! In the light of my developed knowledge with the help of your answer they appear some questions which I'll try to put them as separate questions for the community later. Just as a quick imply, I face with these issues when I'm going to define a point process for an area larger than sampling locations. This post also leads to the problem of [conditioning a point process to the existing data points.](http://stats.stackexchange.com/questions/16780)
**GlenB**: yes, it is a binary information - mutation observed, yes/no. **learner**: you are right. I want to know if this observation could also be observed by chance. So if it only were two genes I would perform a Fisher´s test.
@whuber , its not too vague. I leave the codomain be. Interesting estimators might have low variance or no bias.
I've heard about World Programming System (WPS), http://j.mp/9GjesM. Don't know how comparable it is, though. There was an old GNU project, called Dap (http://j.mp/955DqP), but it never reaches a mature state. In biomedical research, SAS is still considered as THE reference software, although FDA is on her way to (progressively) accept R-based analyses for clinical trials.
I think it's the same information as on the rest of the network, but if it's wrong, or it's being interpreted wrongly, we ought to discuss that. Anyway, I've posted a [question to meta](http://meta.stats.stackexchange.com/questions/1880/is-the-help-misinformed-on-the-question-of-cross-posting) where I hope you take the opportunity to educate me on the error of my ways. I look forward to learning something it seems I've missed (no sarcasm, irony or any such intended; I really do hope to be educated, even if I might take a little convincing).
@ Wayne: thanks. could you check the post just right above this. thanks
So you really mean "theory free", in the sense that no "causal mechanism" is postulated between the two series?  The phrasing "model free" gives the impression of "assumption free", which VAR certainly is not.
I think you're going to need more information to do this. EG, how often will the units be measured, once per year? What parameter values do you expect to find / will you want to be able to differentiate from 0, etc.
Yes, its just an example, I have several other phenotypic attributes also but data format is same. I am using using single response variable at a time.
The person with only one 3 will have an undefined sample variance ie once a degree of freedom is spent on estimating the mean.  But your point remains a good one if comparing a person with 2 "3"s and another with 100 "3"s.  I think a good fix for that might be moving to a Bayesian paradigm.
Yes I meant dimension as number of dimensions in feature space. Thank you very much for the answer :)
You can't simply assume that $X^2$ is Gumbel, nor can you necessarily assume either $X$ or $X^2$ is normal (though $X^2$ might not be too far off. But the formulas you give at the end should apply; they're not based on normality. [Mathworld](http://mathworld.wolfram.com/GumbelDistribution.html) gives the kurtosis for the Gumbel so you should be able to get the variance of the square from that.
@andrea, that would reflect a belief that there is a secular trend in the data set - I didn't assume a priori that this made sense. If the units were people and $t$ were age, then I certainly agree that a fixed effect in time made a lot of sense, but in other situations, the slope in time would be more of a random direction each individual is headed. That's why I didn't include that effect (and epigrad didn't ask how to include the fixed time effect)
Can you add more details about your data & your model here, maybe paste in some output? I doubt this question is answerable at present.
Using this pivotal Quantity I have to construct a confidence interval for $\\lambda$  .That's why I am interested in Chi Squared distribution.If it make n in $\\sum X_i$ distribution to n/2 when it comes to distribution of Q then I can construct the Confidence interval using chi Squared distribution
This cleared out my confusion about the differences in the Brewer's procedure in the different references. Thanks.
Thank you @qoheleth
@whuber and rm999: I edited a response into the question.
I see your point, but somewhat disagree.  With cost minimization, we know the cost function, and we are trying to find the peak.  The stochastic gradient techniques pick a random direction (guided by history) with that objective.  With MCMC, we don't try to reach the peak.  With an independence sampler, we aren't even selecting the next proposal with any reference to our current location (so this would analogize to trying to min. a function by just randomly selecting inputs.)  The only real similarity I see is solving a problem via random proposals with an acceptance step.
Thanks! However, what i really want to know is that, if $\\chi^2$-distribution offers the *best* estimation?
To be sure to understand, are you talking of pure FA or just PCA-related methods?
Have you checked what, if anything, Mostly Harmless Econometricians recommend?
There may be some critical typographical errors in this question, such as "expenses pr production": what does this mean? Please edit your question to fix these errors. When you're doing that, consider also explaining the apparent changes that occur within the question statement: what begins as an *optimization* problem is restated as a *regression* which then turns into a *model specification problem* and finally a *data description problem.* Is that one question or four different ones? If it's one, exactly which one is it?
How do you define best? You need to specify that to get an answer.  In a different context where you want to choose a method to estimate variance, if you make the criterion minimum variance unbiased you would use the estimate with n-1 in the denominator.  If the criteria is minimum mean square error, dividing by n could be best. In the case of a normal distribution this would be the mle.
The last question cannot be answered without knowing more about your outcome and the predictors. A factor of 20 increase for 1 percentage point increase in X could make sense in some contexts. You might also find the second example useful: http://www.ats.ucla.edu/stat/stata/output/stata_nbreg_output.htm
Presumably if the target is 'green' then 'red' and 'blue' and 'yellow' are considered equally inaccurate (you don't specify). If that's the case, then you're effectively just in a binomial situation - in each case, either it got in the right category or it didn't.
@IrishStat the OP has updated his question and stated that there is no seasonality.
The lingering effect is a hypothesis. If you just keep everything on a quarter basis, you will test whether it does drop off each quarter by looking at the profile. The measurements of the cross-correlation function at different lags already accounts for this effect.
:user3544 My favorite Einstein quote is "Make everything as simple as possible, but not simpler" or restated Make models as simple as possible, but not simpler because some people think simple is the objective while it can be the cause of insufficient analysis. In this case your suggestion would have sufficed to identify the contemporaneous correlation and the lag correlation between the two cohorts while illuminating the level shift.


Read more: http://www.brainyquote.com/quotes/quotes/a/alberteins103652.html#ixzz1e9wSHBLv.
thank you very much for this. I highly appreciate your time and help. I knew I was missing something and needed to be pointed in the right direction. Thanks again for this.
@Druss2k Do you want to show that if you use the estimator $\\widetilde{\\boldsymbol{\\beta}} = (\\mathbf{X}'\\mathbf{X})^{-1}\\mathbf{X}'\\boldsymbol{Y} - (\\mathbf{X}'\\mathbf{X})^{-1}\\widehat{\\boldsymbol{\\rho}}$, where $\\widehat{\\boldsymbol{\\rho}} = \\mathbf{X}'\\boldsymbol{U}$ -- you will get an unbiased estimator? This is not particularly hard to prove.
@kanzen_master: I guess that the chi-squared statistic can be seen as a measure of uniformity, but it has some drawbacks, such as the lack of convergence, dependence on the arbitrarily placed bins, that the number of expected counts in the cells needs to be sufficiently large, etc. Which measure/test to use is a matter of taste though, and entropy is not without its problems either (in particular, there are many different estimators of the entropy of a distribution). To me, entropy seems like a less arbitrary measure and is easier to interpret.
@user2943039 did this work for you?  Please advise.
@Luke I edited the example slightly as it contained an error. Feel free to revert it.
Thanks for the tips. I have also read and implemented this, whereas it fails to identify all the clusters. No matter how I tune my minLns and the other parameter, I always end up with fewer trajectories. Any ideas?
-1 The statement "for two standard normal random variables $X$ and $Y$, $(X+Y)$ and $(X-Y)$ are independent gaussian random variables" is false. It does not hold if $X$ and $Y$ are marginally normal but **not jointly normal** random variables. It does hold for _jointly_ normal (not necessarily independent) random variables with identical variance.
(ctd) ... As a result, you should divide by $E_i$ and the variance should come out exactly right. [You still have only $k-1$ df though.]
@PeterFlom Thank you.
@sparc_spread No, in fact you are correct and it is a typo that I left it out. You can see it reappear in the formula for the standard deviation. Will edit.
An illustration would help, because the answer depends on what you mean by "line," by "line of statistics," and what kind of plot you would like to make.  That would also help us determine whether this question belongs here or would be better on SO.
Who considers the first order statistic to be a good initial estimate of a location parameter, & in what context? It would certainly be a very odd one for most situations.
(+1) But this was tagged as homework so you're not encouraged to just give the full answer.
It is not obvious to me that cross-correlation should be high at lag 0 in this particular case. I would expect temperature in deep water  to have very low contemporaneous correlation with air temperature, for instance --even if the weekly observations are averages. It may be the case that correlations at lags different than zero are larger. I would expect air temperature to lead water temperature for relatively shallow depths at the weekly time scale.
(1) Why do you assume the $x_i$ and $y_j$ are positive in your example? (2) The calculation threatens to obscure the essential point. It can be avoided by noting that the *standardized* values of the $x_i$ are $\\{1,-1\\}$ and the standardized values of the $y_j$ are the same. The correlation, by definition, is the average product. Depending on how the $x_i$ are associated with the $y_j,$ all products are either $1$ or $-1$, QED.
ROC curves in general, unless you use them for what they are really intended for: mass one-time group decision making.  They don't help with individual decision making where for a given subject you condition on $X=x$ instead of $X>c$ (we know exact predictor values for each subject, not only they they exceed a cutoff).  ROC curves also tempt even good analysts to select a cutpoint.  What does the ROC curve tell you that you can't get from standard regression stats?
+1 A discretized gamma will closely approximate the negative binomial.  To find a gamma corresponding to a negative binomial you can match moments, after first adjusting the mean of the gamma by half the bin width and its variance by 1/12 the squared bin width.
Could you give us some context for your question? As it stands now your question seems to be too general to be answerable...
Closely related is a discussion about the difference between confidence intervals and prediction intervals at http://stats.stackexchange.com/questions/16493.
@LadyM, I would say means are *better* than percentages in that sense. Also, what I'm suggesting is that you *don't* do a standard ANOVA, but use a multiple regression model instead (although ANOVA actually is an MR); it will be more appropriate given what you have & ultimately more informative. On a different note, if you find this answer helpful, you might consider upvoting by clicking the upward normal dist to the left of it, & if you think it answers your question, you could further accept it by clicking the check mark. No pressure, I just see you're new, & you might be unfamiliar w/ this.
Monte Carlo methods can give very accurate answers provided you do enough simulations.
The second last paragraph sounds very wrong. It mixes up null and alternative hypotheses.
Count is just unnormalized proportion so you can compute st. error of proportion and "unnormalize" it into count units, if it makes sence for you. You are right that st. error is applicable only to sample. In population, there is no error.
A t-test is the usual method for comparing two proportions.
(+1) This is a good description of the hybrid (and why exactly it is hybrid), but you did not explicitly say what your evaluation of it is. Do you agree that what you described is an "incoherent mishmash"? If so, why? Or do you think it is a reasonable procedure? If so, do the people claiming it is incoherent have a point, or are they simply wrong?
See updated post.
Say again, why don't you use `glmnet`?
This is turning into something like [parsets](https://code.google.com/p/parsets/). See [ggparallel](http://cran.r-project.org/web/packages/ggparallel/index.html) for an R implementation.
If you are only interested in the calculation, [this link](http://en.wikipedia.org/wiki/Bayesian_linear_regression#Posterior_distribution) might be of interest.
Does anyone know how LaTeX works here?  I tried the faq here and also Google searching, but that of course leads to questions on tex.stackexchange.com/
As $x$ ranges over $\\pm 1/n, \\pm 3/n,$ *etc*, $n(x+1)/2$--when sorted--ranges over $0, 1, \\ldots, n$.  It represents the number of $1$'s that are summed (along with the remaining $-1$s) to produce $nx$.
I'm confident you can find great answers among the thousand or so posts dealing with [tests of proportions](http://stats.stackexchange.com/search?q=test+proportions).
@Henry Yes I know, it is for that reason that I asked the question number 1.
THis question is very similar to http://stats.stackexchange.com/questions/48107/tools-to-detect-jumps-in-a-linear-time-series/48146#48146 . My response should be of help to you, not only technically but with respect to available software.
Are you using the same hyperparameters? did you set them explicitly or are you using defaults?
There are a number of reasons why this seems to be the case. First is that the Theil-Sen line has a simply-described estimator (median of the pairwise slopes), which the Spearman lacks; in small samples it's very suitable for hand calculation. The Kendall correlation [approaches normality faster and is more mathematically tractable](http://stats.stackexchange.com/a/74390/805). See also [here](http://stats.stackexchange.com/questions/3943/kendall-tau-or-spearmans-rho) and [here](http://stats.stackexchange.com/questions/3730/pearsons-or-spearmans-correlation-with-non-normal-data/3744#3744).