tttjjj/elt_bench huggingface dataset:
But wow man, I learned this in my AP Stats class last year (in junior year)...kind of disappointed I wasn't able to recall a 2-sample z test for proportions by myself. Just to clarify, once I get the test statistic, I can get p by finding z > T?
No. SAS does what you tell it to; just like R does. Look at the code I gave. It includes one sample and two sample tests for both R and SAS.
@Harlan And to be blunt, those people aren't going to be rewriting their stuff in Julia. R as a statistics package, not a programming language *is their use case*.
Can you point to matlab code , I can't find it.
When I took the data I didn't realise that almost all my data points would score the same on these variables. Now they're in my data set, so it would be be a (minor) effort to remove them. I'm wondering if in principle leaving them in would be expected to be minorly useful or minorly deleterious. In practice I'll follow your suggestion and see if there's a difference.
add up the [differential entropies](http://en.wikipedia.org/wiki/Differential_entropy) of the individual matrix elements (assuming the $n_{ij}$ are independent of each other) and compare the entropy sum of $N$ to the entropy sum of $M$ ?
Stock returns are a special case; their values may well be within a few percentage points of zero.  But if firm size and cash flow have values that are all at or near zero or one, it must be that something has been done to corrupt the data.  Taking the logs won't help; I think you need to go back to your your original data source.
My problem consist of exactly 48 distinct crosstabs, which are based on exactly 45 unique variables. Most cross tabulations are 2D, but some are 3D. Definitely I don't have all the margins.
@juampa. All I have is informal references (like the ones I just added) and my own experience in training deep belief nets in relation to other methods such as SVMs random forests, etc. I would also be definitely interested in a study like the one you mentioned, and that's why I am asking this question.
Michael, sorry for the delay in replying.  R's glm function found no connection between configuration and failures, looking at it from a factorial design point of view found nothing useful (I am using somebody else's data and cannot rerun the experiments).  My only justification for removing the 10 would involve hand waving and suggesting that perhaps a mistake was made in running a test 90 times (10 times for 9 configurations).
Compactness implies closed on a metric space, so (ii) is what guarantees that $|\\theta_0| \\neq \\infty$.
10,000 samples how big each?
+1, very clear and understandable answer.
To be honest, I don't completely follow your situation, but I can say that stepwise selection is a really terrible thing to do, & is quite unlikely to "find the factors which best predict engagement" in general. If this doesn't make sense, you may want to read my answer here: [algorithms-for-automatic-model-selection](http://stats.stackexchange.com/questions/20836//20856#20856).
We should certainly not expect financial statements to be free of errors :). But sticking to the point, @Glen_b had it: you wouldn't have two samples for a two-sample test. If you must treat the large bank ratios as population parameters then your test should be a one sample test, with the null that midsize bank parameter equals a constant (large bank parameter). But if you agree with the above note that the distribution shouldn't be treated as continuous then a t-test wouldn't work.
Welcome to the site. You don't need to sign your name as the site does it for you automatically.
@mpiktas: thank you for the comments. Arima is indeed important in my question, since this is the main model I used to fit. What do you mean by “prediction interval of [0,0]”?  I think the forecasting intervals are useful even in this case. I have $d>0$, however the effect of $X_t$ over the fitted values $ARIMA(p,d,q)$ is prominent. Even over the forecasted period, $X_t$ has its own effect.
Why did you make that last edit - apparently doing nothing but damaging your post?
You should check out this link: http://labs.genetics.ucla.edu/horvath/RFclustering/RFclustering.htm
@Andrew: This is probably pedagogical, but it's the process of generating the event that is random, not the event itself.  The event is just a thing.
Basically, this is because $\\hat{y}_i = \\hat{\\beta}_0 + \\hat{\\beta}_1 x_i$. That is, you lost two degrees of freedom when you estimated the intercept and the slope.
@OP: I'm sorry but this is currently unreadable. Please fix the TeX.
The lasso estimates are equivalent to the posterior mode Bayesian estimates with double exponential $\\sim \\exp( - \\lambda|\\beta|)$ priors. The $\\lambda$ scale parameter is related to the upper bound on the sum of the absolute values of the estimates in the lasso. This equivalency may be easier to utilize than trying to run maximum likelihood estimates with VERY WEIRD nonlinear constraints.
(+1) To the point!
Given that bugs is a count variable you probably shouldn't be using linear regression at all, but some form of count regression such as Poisson or negative binomial.
@whuber-In previous thread I was looking for forecasting with Holt-Winters technique without 104 weeks of data.Here I'm looking to add some other modeling approach for seasonality & trend in HoltWinters() function with 104 weeks of data.Hope I'm able to clarify your doubt.
Thanks.  Interestingly I'm trying to use ICC to assess different nurse's ability to replicate a skin reaction on a patient.  So the nice thing is that I have a physical measurement which is objective and easily measured (unlike many situation where ICC is used for raters of personality constructs, etc.).  Therefore the reactions certainly exist; it just seems that in my case the variation within a subject is equally as large, if not larger, than the variation between the nurses.
Only update is that I decided that I had enough data to call the error distribution normal, and `lme4` works just fine for that.
@whuber,\\n\\nI'm afraid I don't understand your question fully. It's probably my lack of statistical understanding. But to try to clarify. Model 1 has 4 variables, Model 2 has 3 variables and Model 3 has 2 variables. The variables are in the same order in every model (meaning variable one = temp in each model). \\n\\nI think @GaBorgulya and @djma answered my question perfectly. Variable 4 IS correlated with variable 3. AH-HA! Makes sense.\\n\\nthanks oodles!
I can't understand why nobody uses the purpose argument, i.e. that linear regression is a *modelling* method and ANOVA is a *hypothesis testing* framework. I know that they can be extended in a way they overlap, but the canonical formulations are perfectly separated.
I'd be surprised if the data would follow any well-known distribution at all. @surgez [This post](http://stats.stackexchange.com/questions/58220/what-distribution-does-my-data-follow) might be of interest too.
@whuber The pure white bands are periods of no-data and only show because of my crudely coercing the data into a regular heat map. Each zone should be getting darker during those times (and they are darker on the other side of the bands).
How you would compare the posterior? Should I calculate distance between them?
A characteristic bubble means a bubble that is characteristic of the field of bubbles.  It is an image of a bubble that is generic enough that all bubbles going through your field of view are adequately described by it.  Image-let is my own nomenclature.  It is like a wavelet in the sense of how it is used, but not how it is derived.  Wavelets have to obey some rules that these image templates do not.  This makes a platform/framework for detection that can be very flexible.  It is very much like an eigenface, but you just need one.  I mean ifft2(fft2(a)./fft2(b)) .. deconv2d + interp.
I disagree that this question is ill formed. The 10 and 25 may be Poisson parameters, or observations, but even if they are only two observations they can be used as approximations of the parameters, so they can serve as basis of power calculation. (Alpha must be a mistake, but this does not make the question invalid.)
Great example. i'm still not sure why your answer works, but, indeed, i have been able to use the template your provided to build a counter example for the range case. If i still don't get the intuition by tomorrow i will perhaps resort to asking a separate question about it. Thanks for coming up with it though.
I suggest you give more details and/or a specific example of how your clusters might be defined. This sounds more statistics than programming, so might be better on Cross Validated. I'd expect to see posters' code in this forum.
Could you expand a little on what you mean by "randomly distributed"? Are you trying to test whether the counts represented by the "yellow sausages" could plausibly have been achieved by simple random sampling from the background gene counts?  If so, then what role would the spatial positions of these points play?
I should say for full disclosure that I have already posted this to the bioconductor list
How is "a curve" defined? Do you have an explicit functional form in the database? A set of datapoints of it? Are there any restrictions to the kind of curves that may occur? If two curves cross somewhere in the middle, do you count both areas as positive?
Yes, it is similar for interpretation.
+1 for good advice.  But it's unclear whether the implicit assumptions in your reply match the question: it sounds like the OP is not actually adding any constraints, but only is asking whether *methods* of constrained optimization could be used to solve the *unconstrained* problem.  (Of course my interpretation could be wrong, too...)
Hello Wayne, 
to 1.: yes, it was mistyped.
to 2.: yes, I am training VAR on the entire data and the univariate ones only on the first variable. I am actually interested in forecastinf the first variable.

Do you have any suggestion which additional state might be included in the ss?

And a more general question: So far I have not seen any function/technique in the package dlm or in the book of Petris to suggest a model. So, do I have to know in advance which model I would like to have in ss representation before using the package, or is there some technique which I missed?
You mean 800 in that last line. Don't mix the mean and the number of simulations or you'll get very confused. If this is for a subject, or even just for your own study, you should include the `self-study` tag, and read its [tag-wiki info](http://stats.stackexchange.com/tags/self-study/info).
Why do you think the two-stage mixture approach would be inefficient?
Ah, well (I didn't see it). Then term "discriminant analysis" _proper_ is inapplicable, although word "discrimination" can retained, as it is close to "classification" or "distinguishing".
You might look at the R `rms` package `ols`, `calibrate`, and `validate` function with quadratic penalization (ridge regression).
Here is a recent discussion regarding one-sample test and CI: [Hypothesis Tests and Confidence Intervals](http://stats.stackexchange.com/q/47345/930).
You don't (yet) have enough information to answer the inverse question: at a minimum you need estimates of the error variances.  It would help to have the full variance-covariance matrix of the coefficient estimates, too: unless the variables are centered, the slope and intercept will be correlated and this can make a big difference.
@Placidia Are you kidding me?! This isn't homework, this is about understanding and differentiating 2 general concepts in statistics, which then could be implemented in homework questions.. like every  other mathematical concept..
I believe Kolmogorov's law applies only to *tail events,* not to "any event ... we're interested in." You might be able to apply this law to general events to shed light on the question, but some explanation of how to do that would be helpful here.
Normalizing the RMSE (the NRMSE) may be usefull to make RMSE scale-free. For instance, by transforming it in a percentage: RMSE/(max(DV)-min(DV))
From your question you may be reading the code slightly wrong. The syntax is using the differences as the "instruments" to estimate the lag of the dependent variable.
Sequencing is a really important thing to consider, you are right. Can you see any advantages ?
Ok.  Well, looking at the distributions may be the way to go.  You can also try a distribution test such as Anderson–Darling (as in this question: http://stats.stackexchange.com/questions/1645/appropriate-normality-tests-for-small-samples).
@Nick Cox I understand now, thank you. I'm certainly no statistician but for the engineering work I do we typically phrase the question something like: "What is the probability that the two sets of data (curves in this case) are due to a genuine difference rather than noise"
The answer will depend on *what kind* of information. As @Michael Chernick notes, a treemap is not very good at displaying the structure of some kinds of trees. But the Wikipedia article emphasizes that the point of a treemap is the simultaneous display of *additional* (multivariate) data carried by the leaves and nodes of the tree.  Thus, to indicate what you have in mind, it would be good to see this question describe what you mean by "information" or what application(s) you may be considering.
Cool idea! Welcome to CV. I bet any answers we might be able to offer here would go over well at [Seasoned Advice](http://cooking.stackexchange.com/) too. That is, any answers here might provide many answers there too!
+1, but technically 'that's true only if' is false: it suffices that the intersections are of probability 0 and that the universe minus $A\\cap B\\cap C$ is of probability 0. In addition, the statement could be true 'by accident', e.g., let $P(D)=1,~A\\subset D,~ P(A)=1/3,~A=B=C$. Suggestion: edit to remove 'only' from the beginning sentence.
Also assuming a fair die for the $\\frac{1}{6}$ part.
+1 Very good answer, seems that you are completely right
"The way you specify the conditional joint distribution, Z does not necessarily have any dependence on X or Y. Am I correct in saying this?"  No, $Z$ _is_ dependent on $X$ and $Y$. $$\\begin{align*}P\\{X=0,Y=1,Z=1\\}&=P\\{X=0,Y=1\\mid Z=1\\}P\\{Z=1\\}\\\\&=\\frac{1}{2}\\times\\frac{1}{2} = \\frac{1}{4} \\neq P\\{X=0\\}P\\{Y=1\\}P\\{Z=1\\}=\\frac{1}{8}\\end{align*}$$
I would choose a scenario that is less likely to confuse you.  "Shaking ground" could be the cause or the effect of "earthquake", and both are probably caused by the "green giant strolling".  For explaining away to happen, both A and B must be causes of C.
This is a very terse question. You need to expand and describe (i) why you think skewness is a problem; (2) how bad skewness in your data is; (3) whether these data are really population (all available objects) or a sample (several from a myriad possible); (4) your computing platform -- you've received a default R answer that you may be unable to utilize unless you are familiar with the package.
If you are comparing the means between two groups, simulation would just add noise to the existing formula used by `sampsi` or any other analytic power calculation tools. Simulation is only called for if you have a complicated analysis, say survival curves of different shapes and an observational component that requires propensity score matching. There is no freaking way to get any idea of the power for such a design analytically, so you have to simulate with different sample sizes until you hit your 5%/80% magic combination.
Please, use $\\LaTeX$ markup for expressions.
"*know that the distribution fits a gamma or negative binomial distribution*" this statement makes no sense. One of those is discrete and the other is continuous. What is it that you're calculating the distribution of and on what basis do you assert that you know any given distribution will fit?
re 2., in my personal experience I found it tricky to properly implement a HAC estimator in the panel data setting. Eviews may have punted on this one.
re 3., you are to include the _dependent_ variable lagged. there should only be one. this is the simplest approach to autocorrelation, in my opinion.
Thanks so much for your reply, just want to make sure I'm using the correct terminology though, sorry.  I think it's because it's called 'minimal' detectable difference.  With a 2-sample test, you get an estimate of the difference with a confidence interval.  So smaller differences than the MDD can be detected (down to the half-width of the confidence interval) but with smaller power? Thanks again!
As suggested by GaBorgulya, you are better off using a stat package. Unless PostGre is able to do multiple regression (which looks like it does not), you are limited to testing the difference between 2 groups since your two groups could be dummy coded into 1 variable.    http://psych.unl.edu/psycrs/statpage/coding_eg.pdf
Ok, you should make that more clear in your original question.  Your last sentence indicates you are looking for an algorothm that computes all of the records everytime a request comes in, which I took to mean similarity to all previous documents.
+1 This very nice contribution clearly addresses several recurring issues associated with correlations.  I especially appreciate the remarks in the first concluding paragraph about stopping/starting thinking.
(1) NB the p-value is NOT "the probability that the variances are not equal". This is a serious mistake. $\\quad$ (2) What R package are you using? It's not in base R, and we don't necessarily know which of many thousands of packages you're using. Is this the `hov` in `HH`?  If so, you're doing Browne-Forsythe, not Levene (3) If the sample size is large, then hypothesis tests can pick up small, even trivial differences. If that concerns you, it implies that you shouldn't have been doing hypothesis tests in the first place, because you're interested in a different question to the one they answer
...and please take care of it - your question is beyond comprehension at the moment.
Also, the example provided is also an example of Simpson's paradox because conditioning on the state reverses the correlation between income and party.  When is the ecological fallacy different from Simpson's paradox?
This is one of the needs that got me away from SAS and into the S language years ago - the ability to make predictions and get estimates using only the fit object.  Before that I spent a lot of time converting SAS model estimates into Basic for offline evaluation.  The SAS approach was to add test observations to the originall fitting dataset, setting $Y$ to missing for the test cases, and getting predictions for original + test cases together.  Not very satisfying.
Please don't answer with links. Include at least a very short summary or a general idea.
What makes you think that the first sentence is true? It is seemingly directly contradicted by the first line of that sentence. Are you missing a condition?
You may want to accept this answer, then.
Yes EM works by iteratively estimating the group membership indicator and the mean between each group.
Ricky, that's great.  Y as a percentage is the best way to avoid heteroskedasticity as you confirmed in your models.  Your models do have very small R-square value.  Yet, the F statistics suggest the models overall estimates are statistically significant with surprisingly low P values.  I gather this means that your model do add a very marginal amount of information that makes them slightly better predictors or estimators than just taking the average % of Y as a global estimate.  On a more positive note, no one can accuse your models of being overfit.
@Henry - I have accepted your answer. One point I would make in addition to the ones you have already made is that your simple slope is the same as the first Principal Component, based on the correlation matrix of the data.  This will certainly help with calculating standard errors
@euphoria83: by 'LP based methods' do you just mean interior points or also the 'regular' simplex?
This question appears to be off-topic because it is about a statistical method and thus belongs on Cross Validated.
I'm not sure I understand what you mean.  Are you saying that I should validate only the model that is based on one of the imputations?  Surely there is a way that R users are validating their models pooled from multiple imputations?  Or is this a circumstance where SAS trumps R?
As @whuber said, the delta method might be an answer. This might also interest you: http://stats.stackexchange.com/questions/62916/confidence-interval-for-the-product-of-two-parameters
To use repeated-measures ANOVA, though, wouldn't I need multiple values for each test for variance? I've never used the Friedman test so I'll look into it.
Here is a link to a nice introductory article on superpopulation models [this](http://people.umass.edu/~cluster/ed/outline/c99ed8.PDF)
My only concern would be whether you would need to transform the effect size prior to the regression...similar to transforming r using Fisher's z transform.
It is impossible. In order to diagonalize the covariance matrix of your test data you need to know its covariance matrix first. And there is no way you can learn it by looking at the training data.
Thanks @Stef I will do that
Nice answer.  This is more or less what I would have done if I had the patience to work it all out.
With logistic regression with more than two classes, sometimes multinomial logistic makes sense but sometimes ordinal logistic regression makes more sense.  If there is an inherent ordering to the classes then ordinal logistic regression might make sense.  You mentioned four levels of income...is that your target variable or was that just an example?
Also of interest: [What is the best way to identify outliers in multivariate data?](http://stats.stackexchange.com/questions/213).  These questions are related because (1) finding a multivariate median can be the start of a search for multivariate outliers and (2) many methods to identify multivariate outliers can be used to obtain a median by successively removing outliers until nothing is left: whatever was last removed has a claim to be a kind of "median."
Could you please explain the sense in which this is a "transition matrix"?  Would this mean, for instance, that the chance of a transition from `AC` to `CT` is given by the value $0.3$ in the $(3,1)$ position? If that is the case, then it seems you are merely using a shorthand to describe a transition matrix on $16$ (not $4$) states, {`AA`, `AT`, ..., `GG`} and that you could immediately write down the $16$ by $16$ matrix $\\mathbb{P}$ and solve your problem in the usual way.
@whuber I don’t think that’s the issue; I think in the above post, one could replace "categorical" by "discrete".
Intepretable coefficients are highly overrated. Nevertheless, have you tried principal feature analysis (http://venom.cs.utsa.edu/dmz/techrep/2007/CS-TR-2007-011.pdf) or convex principal feature selection (http://www.siam.org/proceedings/datamining/2010/dm10_054_masaelim.pdf) ?
to my wisdom, this is based on the problem at hand and it is dangerous to use such thumbrules in practice.
Thanks so much. Codes you provide me are working at 100%. Thanks
@user4331 It sounds to me like you really should not add an auto-switch to your routine. It's an admirable goal, but as has been pointed out, it's a tricky question that depends on a knowledge of the data, your goals, and what is "normal/usual" for the data, and an auto-switch can't possibly know those things. So your auto-switch routine could be terribly misleading to its users, who would assume that the routine could somehow know better than they when to switch.
+1 The FFT method's performance gains are comparable for Python users. (I don't find much grist for the R v. Python mill here; your answer is just superior, full stop.)
I am curious as to why you are interested in this. Generally speaking, the bootstrap is about constructing an approximate sampling distribution with which to do inference. Here, the distribution of $D_n$ is fixed (invariant to the underlying data) and known under quite general hypotheses. It can be [computed](http://www.jstatsoft.org/v08/i18/paper) pretty readily as well.