tttjjj/elt_bench huggingface dataset:
Hopefully fixed now. The correct rule is rule (5) [here](http://en.wikipedia.org/wiki/Determinant#Properties_of_the_determinant)
I have made an edit, hopefully should be clearer now.
I agree, this is the way to go if $Z_1$ and $Z_2$ are one-dimensional. This approach suffers some rather severe curse of dimensionality, however. Since I care only about modeling the mean of $Y$, it seems that there should be better approaches? It seems that the paper you reference gives a way of parameterizing the correlation structure. Would you mind making it clearer how I could estimate the conditional means of $Z_1$ without estimating exponentially many parameters in the dimension of $Z_2$?
Great, I loaded my data into R, did a weighted linear model and used predict for a new value at 0.0 similar to what was done [here](http://www.r-tutor.com/content/prediction-interval-linear-regression). Although, I do receive this error message "Assuming constant prediction variance even though model fit is weighted". I'm assuming my result is still approximately correct.
Presumably many people survived through the short period of study, wouldn't one need to employee a model that accounts for censored individuals?
I set a moderator flag b/c this is off-topic for SO. More appropriate for CrossValidated.com
Even assuming one could reliable find outliers using such a procedure (and most of the time, one [can't](http://www.jstor.org/stable/2289995)) that still leaves strangely unaddressed the problem of what to do when you can't "figure out"/explain the outliers. I second the advice to stay clear of SPSS. –
Also, I just noticed you have only 41 companies. In that case, the top 10% would be 4 companies. That is too few - any analysis would risk being overfit. Probably best, in this case, is quartiles of 10 companies (with one group having 11). With two IVs that's still a bit overfit, but not too horrendous.
Please consider formally accepting answers to this question and/or some of your previous ones. (I realize this one has only been open a few hours, but you have a handful of others that are weeks or months old.)
My feeling is that this is MORE than two parameters, as you are also selecting the level-shift point. How many, I do not know. Perhaps the MDL literature may throw some light (you need to use some bits to say where the shift level takes place). Not sure. Tell me if you have some insight.
How many cinemas are there, roughly?
Short answer is both methods give the same estimate of treatment effect if the sample size in treated and control groups is the same and you match all observations one to one.
@Tiffany Some tests have norms, and these can guide estimation of the standard error; or if the individual is drawn from a larger sample, then you could derive an estimate of the standard error from that larger sample.
I added a proof above of why its always one. To use Gibbs sampling you need to know what the full conditionals are.
Yes (+1), this is much easier to visualize than the many box plots as well. I will have to re-read to make sure, but I believe Tukey (in EDA) actually suggests to not use the box's like this anyway, but to connect the lines for the summary statistics (see [this similar question](http://stats.stackexchange.com/a/18826/1036)). Of course using functional boxplots is a better replacement for identifying outliers as well.
This sounds interesting. Can you edit your question to make clear *what question you want to answer* with an analysis?
+1 for Mondrian - very useful toy, especially for large data
Please de-abbreviate `SMA`
I am saying that `K=1` is the number of lags that minimizes all the information criterias, as such it is the value I should use to parametrize my VAR process. `ca.jo` do not allow to specify `K=1` as a value for parameter `K`. Basically I am not sure what I should deduce from finding `1` for `K`, Am I clearer ?
Duplicate of:  http://math.stackexchange.com/questions/789796/calculating-probability-density-function-u-y-x
Thank you Peter. I will look into this more.
**Surely** you can find a smaller example!  How about `p.adjust(myvector[1:2], "fdr")`? And after seeing that result, perhaps you would be inspired to investigate what this method actually does.
I am not sure of urca package but to understand the steps with the example, you should look into Walter Enders "[Applied Econometric Times Series](http://www.ebay.com/ctg/Applied-Econometric-Times-Series-International-Edition-Walter-Enders-2004-Hardcover-/4448519).
I understand that the p-value for `H0: a=0.0062095` is really small, but does it necessarily mean that `H0: a=0` would have a high p-value? Could it be a little worse, but still not-rejectable?
@StéphaneLaurent The interpretation of the prior on the parameters is different from that of the prior probability of the model. This can be seen from the general expression for the Bayes factor. You can also assign uniform priors to the models, improper prior to the parameters, and see what the data tells you *a posteriori*.
you have changed your original question and provided an additional link. Given the link, the answer to your question seems trivial (maybe I'm missing something): just replace the '30' there by increasingly larger numbers (3000,30000,....) until you see the pattern on the width of the CI's (hints: *ceteris parebus* they shrink).
Seeing as this question isn't getting much traffic here, is there a better place to ask this? Also, thank you for the welcome!
(+1) Good example. It might be slightly unconventional in the sense that from the sufficient statistic in this example, one can recover *perfect* knowledge of the parameter of interest. It might be helpful to have a second one that uses precisely the proposed statistic that the OP gives, as well.
Why do you think these beta values are not acceptable?
@Tennant Side note: entropy would be impossible to estimate for short sentences.
Could it be seen as a probability from the classical definition perspective?
I did re-edit the question. I hope it is more clear now.
Sound clear!!appreciated!
@CodeKingPlusPlus If your data has more than 2 dimensions (more than 2 variables), this may not be possible.
Categorical variables--even when they are *encoded* with numbers to identify (and sometimes order) distinct values--by definition have no inherent *numerical* values. What, then, do you mean by "squaring" a categorical variable?
I'm not sure why, but breaking it into two steps works for me: 
AccLog <- as.logical(as.numeric(Accuracy)) then run m2 with AccLog.
@whuber thanks, you saved my favorite :)
This question does not appear to be about statistics within the scope defined in the help center. It is also not about programming within the scope of Stack Overflow.
@Zen As you wish. But note that $\\sum_i (p(u_i))^2$ is  simply the probability that two iid discrete random variables with pmf $p(\\cdot)$ are equal: $$\\begin{align}P\\{X = u_i, Y=u_i\\} &= P\\{X=u_i\\}P\\{Y=u_i\\} = (p(u_i))^2\\\\P\\{X = Y\\} &=\\sum_i P\\{X = u_i, Y=u_i\\} = \\sum_i (p(u_i))^2\\\\ &\\leq 1.\\end{align}$$
I was imagining dimension reduction here in regards to reducing the size of the series. So, instead of 90 variables (one for each day balance), reducing this to a smaller set. Perhaps that could be the first 10 autocorrelation estimates of the series. Or the weekly seasonality indexes.
Thanks for the advice. I'll take a look at the reading you recommended. The residuals from my model treating it as cross-sectional are normal, so it might be easier to just continue without treating it as a time series for now.
+1 This is a wonderful answer in how it combines deft use of appropriate software, a clear explanation of the steps followed, and references to details of the question itself, together with a final reality check.
@user159813 Although it may seem like a triviality, failing to make that distinction explicit is the root cause of the difficulties expressed in the question.
If you want to combine independent likelihoods, use product.  If you want to combine independent log likelihoods, use sum.
@MichaelHardy While the decomposition of variance into components in regression is often referred to as an analysis of variance table. That is not what statisticians commonly mean by ANOVA.  The methods 1) linear regression, 2) analysis of variance and 3) analysis of covariance are categories under the general heading of the general linear model, linear regression involves continuous covariates, ANOVA includes discrete groups only and ANCOVA is a combination of continuous covariates and discrete groups.
@ Nick: Yeah, another typo... Sorry for being so sloppy :-(
E, 100 cores and one may live with that. Seriously, this looks dubious from a ML point of view, still I can understand the reasons why you do it in such a way. So then stick to the Srikant solution.
(+1) though it seems more to be like a comment to me.
If I understand you rightly your null hypothesis is that the recent growth rate of 0.22 comes from the historical distribution of growth rates.  This could be a reasonable null hypothesis to test, but you can't treat growth rates as proportions, so you need another way of comparing 22 to 15.
@Azeem Thank you; I see why you would conjecture that this is what was requested. I addressed the Gamma distribution because (1) it, and not the Gamma-Weibull, has the "only one parameter" mentioned and (2) the question provides the correct Gini coefficient for that distribution, so I assumed *that* was the one that needed to be generalized to three parameters. Let's wait for Zahida to clarify which one is intended.
@whuber thank you, of course you are correct. I admit I made the numbers up ... shame on me, I should have known better !
Don't think so because the binomial distribution is the sum(xi) without the / N term
Why did you use the 'bayesian' tag when it is not on your list of potential predictive modeling approaches?
Basil, I love it! +1 for a plain English explanation.
Am I correct in assuming that multiciplicity adjustments are analogous to family-wise error adjustments? E.g., Tukey's, Bonferroni, etc.?
And you've uncovered the problem with frequentist statistics: assuming the existence of a single true underlying parameter value tends to be conceptually problematic. This is why people have come to interpret "failure to reject exact equality" as "close enough to equality." All you're doing in the latter case is letting your textbook choose a formula for "close enough" based on sample size and some distributional assumptions. Sometimes that's what you want, other times it isn't.
+1. Moreover, areas (unlike lengths, for instance) are remarkably robust to independent errors in the vertex coordinates.  Note that this solution does not assume much about the polygon; in particular, it respects its (manifest) lack of convexity.
Again, that's easy. As soon as you find even one item fulfilling that criterion, you can conclude that it is the case that "Data from the printed catalog is not present in the online catalog". If you don't find any, you can't conclude that.
@Andy: Good point. I've included a quick simulation. Everything seems OK, but there might be a more optimal solution.
@David: Simply invert y and compute the sample mean and sample standard deviation and use those as estimates of mu and sigma. A normal may approximate y^-1 well if sigma is relatively small.
Actually, I am looking for mathematical formula of this prediction error. Do you have experience CV with Least Angle Regression? That is implemented in R package 'lars'. How CV works with model selection, I mean to say should we always select as many predictors with minimum prediction error?
@StasK I think you make some important points, why not try to make them a bit less aggressively?
Expanding a bit on @NickCox 's excellent comments. How do you think the habitats *might* vary? Do you think there is a difference in mean temperature that is relatively constant throughout each day and over the time period? Or might the difference in temps vary over either of those? Your second question is very tricky indeed.
All my priors are Uniform(0,1) (i.e. Beta(1,1) ). The 10000 is the number of draws from the posterior I perform, so I can do monte carlo estimates of the $P(p>p_i)$
Can we assume that all machines are identical? Or you suspect that some of them misbehave and you want to detect them?
You kinda answered your own question: *train multiple classifiers*. If you want some kind of *structured* output, such as a graph, you need different techniques.
Can you clarify what you mean by "pooled t-stat" vs "normal t-stat"? If your only question is whether a t-test is OK to use if the $n$s are unequal, then this Q is a duplicate of [How should one interpret the comparison of means from different sample sizes?](http://stats.stackexchange.com/questions/31326/)
I was using `grid.py` from libsvm package
Is [this paper](https://www.jmp.com/software/pdf/30612.pdf) of some use to you?
Yes, it can be confusing. This is discussed in the comment thread beginning at http://stats.stackexchange.com/questions/27882/what-is-the-difference-between-serial-correlation-and-having-a-unit-root/27890#comment52000_27890. It might help to know that $\\lambda$ is a root of the polynomial $p_nx^n+p_{n-1}x^{n-1}+\\cdots+p_1x+p_0$ (with $p_0\\ne 0\\ne p_n$) if and only if $1/\\lambda$ is a root of $p_n+p_{n-1}x+p_{n-2}x^2+\\cdots+p_1x^{n-1}+p_0 x^n.$ Note, too, that the same relationship holds between $\\det(1-\\lambda\\mathbb{A})$ and $\\det(\\lambda-\\mathbb{A})$ for any matrix $\\mathbb{A}$.
@Adam Agreed. Your response was on the point but we generally expect standalone and justified responses rather than a one-line thought. As another response pointing toward Bonferroni correction has been proposed in the meantime, maybe you could expand a little bit on why we must account for inflation in Type I error rate when analyzing two-way tables at the cell level, or if there's another alternative to Bonferroni correction (which is known to be conservative) that might be suited to this particular case, etc.
I've got some answers in my email, it worked doing a matrix[i,j] and summing in j for each [i], something just as you suggested, thanks for the answer! By the way, i'm using bugs by R + BRugs, how can I get the simulated data for the parameters? By the commands (samplesStats("*") and samplesDensity) i get the mean, sd, error median, and a density. I tried to search even in the manual of the package, and couldn't find the answer
+1 You can, with relative ease, obtain expressions for all natural numbers $k$ in the usual way by replacing $y^{k-1}/(\\theta^k\\Gamma(k))$ by the Maclaurin series of $t\\to\\exp((t/\\theta)y)/\\theta.$
(**3**) Item 4 in your list is handled as follows. Note that $$\\sup_{C\\subset [0,1]} \\int_C |\\tilde F - F| \\leq \\sup_{[0,1]} |\\tilde F - \\tilde F^{\\star}| + \\int_0^1 |\\tilde F^\\star - \\mathbb E \\tilde F^\\star| + O(n^{-1})\\>.$$ The first term on the right-hand side, $\\{\\sum_i Y_i > 0\\}$, is $\\leq |1 - c n^{-1}\\sum_i Y_i|$, so is clearly $\\mathcal O_p(n^{-1/2})$. You're left only to deal with the middle term, but that succumbs readily to Markov's inequality followed by Jensen's and is also $\\mathcal O_p(n^{-1/2})$.
Thank you for nice explanation.
@StasK: That's an extremely weak definition of "stationarity" that you are using! For example, let $X_i$ be i.i.d standard normal and take $Z_n = n^{-1/2} \\sum_{i=1}^n X_i$. The marginals of $Z_n$ are constant with respect to $n$ (indeed, standard normal!) but $\\{Z_n\\}$ is not stationary by any standard definition.
Check [this](http://mathoverflow.net/questions/57998/correlation-between-3-variables) out.
Not so; it's a surprisingly difficult problem and smart people have proposed different solutions from different views of what is most important. Computation is a secondary issue. No one obliges users to ask for two-sided confidence intervals here. All the methods proposed try to use defensible logic for different intervals. See Brown, L. D., T. T. Cai, and A. DasGupta. 2001. Interval estimation for a binomial proportion. Statistical Science 16: 101–133 for a superb review.
You are completely correct assuming external temperature of the box is just normal outside temperature -- the box also gets heated 8 hours a day.  The R series fluctuates quite wildly to start with then it kind of levels off with some 'wobbles' --I can break down the series in widows that are stationary though, and might want to get a summery and an error for those.  There is an error associated with the instruments (which are as good as they get).
+1 for reference to arima.
Thanks for the link on the R-help mailing list. There is also this one on s-news: http://j.mp/8Zol8W.
I agree with your clue if we have a log-normal distributed variable. However, the mean need to be "back transformed" to get an easily understandable statistic based on the original scale of data. This might explain the article's conclusion. Also, after log-transformation, we might not get a normally distributed variable and in this case, I do not know which approach would be better.
I have updated the answer accordingly.
Do you have any idea what size of effects you want to be able to catch? In addition, for a situation this specific & esoteric, I suspect you'll have to simulate to determine the appropriate N. I wrote about simulating for sample size here: [Simulation of logistic regression power analysis - designed experiments](http://stats.stackexchange.com/questions/35940//35994#35994).
It's not a simple question. There are many possible answers. You are going to need a model.
I am using t-distribution for robustness but having nu free everywhere I think is too much freedom. I am trying something like ANOVA so I have some different groups which I want to fit with a t-distribution, but I want a common nu. Then I am also calculating some quantity from all the data together which should use the same nu.
For example having an expert in the field of my dataset to look for parameters that could have greater value in predicting.
@mp The zeros show something about the complexity of the function.  Rational functions have a finite number of zeros.  Elementary functions typically have a line of zeros, such as at $2 i \\pi n$, $n$ integral, for $\\exp()$; typical "transcendental" functions have slightly more complex patterns of zeros, such as at all the non-positive integers (Gamma function) or on a polygonal lattice of points (theta functions and elliptic functions generally).  The complicated pattern exhibited here suggests it will be difficult or impossible to express the CDF in terms of these familiar functions.
The variance of this distribution is infinite. The OP asked for distribution for which $\\int x^2dF(x)$ does not exist in a sense that $\\int_R sin(x)dx$ does not exist. Henry in its comment pointed out, that no such distribution exists.
Michael, for finding R solutions use http://www.rseek.org instead of Google.  One of the first hits when searching on "cluster" takes you to the [Cluster task view](http://cran.r-project.org/web/views/Cluster.html) listing approximately 100(!) packages.
Also I think it is a bit difficult to keep track of the conversation we are having here. Given the edit you made it is totally a different question. I don't know if you should have issued another question at this point. Maybe some moderator can help us on this.
A permutation test of a specific hypothesized mean is no different from subtracting that hypothesized mean from the data and testing against a mean of zero. A paired test is discussed [here](http://stats.stackexchange.com/questions/64212/randomisation-permutation-test-for-paired-vectors-in-r/64215#64215); it makes the assumption that under the null the pairs have the same distribution, which implies the differences on which the subsequent one-sample test is based is assumed to be symmetric. On that basis, the signs are randomly flipped on each difference ...(ctd)
Can you pls be a bit more specific? What do you actually want to data mine and how do you plan to do it? I have used R to analyze similar size records as yours, and it wasn't a bad experience at all.
@Daniel by all means then give it a go, but you will need to supply some starting points for the algorithm
@Thylacoleo: the log-rank test is equivalent to the Score test under Cox PH. Think you're confusing PH *tests* with HR *estimators* here.
(Con't) **Am I correct in my reasoning above?**
I am very hesitant to use structural equation modelling. I am not very statistically minded and also my model is targeded at users who have very little knowledge of statistics.
**Is the my model roboust enough, statistically speaking, to produce reliable results?** Many thanks again. I am very grateful... -- [Jasty West](http://stats.stackexchange.com/users/5665/jasty-west)
The F-statistics you found there should be useful. Try to calculate the p-value of this ANOVA test. The resultant p-value should also be the p-value of the interaction term in your model 5. Now, you have the p-value and the point estimate, and you know that regression coefficients are tested with t-statistics, and you also know the df is 1... you should be able to deduct what the standard error of the interaction term is.
This says exactly what the previous answer does. Did you intend to add some more information to that?
Thanks for the answer!  However, even if zip code is a string or factor, aren't I essentially just dummy coding zip code (i.e., creating 30,000 binarized features)?  I know R does this under the hood but it has to be explicitly done in scikit learn.
Can you describe the data in more detail?  For instance, do you have coordinates, pairwise distances, etc.?
@user2932774 if you like my answer, please feel free to upvote or accept it!  And yes, corrected your intercept term should work too.
You should give some details about the test you want to do. This would give more adapted and focused answers.
possible duplicate of [A non-parametric repeated-measures multi-way Anova in R?](http://stats.stackexchange.com/questions/1266/a-non-parametric-repeated-measures-multi-way-anova-in-r)
@ocram that's a very thorough answer, but it may be more than what was requested. The question is just "do females have higher blood pressure than males?" this may be a case where simple is just as good as complex. But the repeated measures do allow more interesting questions to be asked.
A classification matrix is misleading because of its arbitrariness and extremely low statistical precision, and it should not be used to judge fit or accuracy.  Lift charts are useful but the concordance probability ($c$-index) summarizes that fairly concisely.
For basic financial time series references, you can see my answer here: http://stats.stackexchange.com/questions/328/resources-for-learning-about-the-statistical-analysis-of-financial-data/329#329.  The Tsay text is one of the most popular.
So $p$ is the estimated proportion of defects from some process, used to calculate the control limits (assuming independence); while $p'$ is the true proportion of defects produced following some change in the process - is that right? Please edit the question to clarify rather than adding comments. There aren't very many industrial statisticians active on this site, so unless you make the question generally intelligible through context & definition it's likely to be closed as unclear, despite there likely being many readers who could answer.
Welcome to CrossValidated! It does look like a useful resource, but could you add a few sentences summarizing the contents or explaining why you like it.
@Kochede: no because *all* the points are used in the new optimization. The idea is that you have to visit less edges of the simplex before finding the good one because you already start from one close to the good one.
You provide the formula for the calculation you have requested.  What specific problem do you encounter in applying that formula?