tttjjj/elt_bench huggingface dataset:
Check [Diaconis and Rolles, *Bayesian analysis for reversible Markov chains*](http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aos/1152540749) -- it may be what you're looking for.
Do you have a preference for a particular software?
Seconded. RKward is excellent, as it presents a reasonably clean, functional interface, with most of the basic functionality users will need (it's young, more will come later, I'm sure), **with** code. Everything that you request R to do through the UI is presented in code form just below, so you can actually see what's going on behind the scenes. It also has a command line console, which means you can jump straight in and tinker. The dataframe editing interface makes it nice and easy to do all those annoying manual manipulations of data that are a pain on the command line.
@lightalchemist Thank you for the clarification. With sklearn, is it proper to create a new dataframe prior to performing the PCA, or is it possible to send in the 'complete' pandas dataframe and have it not operate on the leftmost (response) column?
@StatTistician I checked the stats, I have 22 downvotes and the last came on November 26, which came earlier than your registration on this site.
I think I am looking for a test that combines these two into a single statistic rather than providing two separate readouts. The F-test (on variances) does actually show up significant in some cases which might be enough but it would nice to have it in one. Also wondered about the Kolmogorov–Smirnov test (which has assumptions I usually forget) but there is another based just on means and variances.
"MAPE at many different origins not one", could you please explain this sentence? I am getting it. Thanks.
@whuber. The OP said he wanted to test "that both are fair", so I took that to be combined. But it probably makes more sense to ask if each is fair, so that would require two binomial tests. I'll reword.
Thanks for this example. I checked your code and it does indicate that U matrix is Nxp, and V matrix is pxp and D matrix is pxp, where N=10, and p=2. However, since I find reference about getting U as square matrix, such as here (http://web.mit.edu/be.400/www/SVD/Singular_Value_Decomposition.htm) I am still a bit confused.
As an additional note, the paper I am referring to also has a table showing the Estimates and the 95%CI. The title of the table however says "Model-averaged parameter estimates and relative importance values for variables affecting adult piping plover foraging rates in New Jersey,2007–2009." which does not seem to fit with what was actually shown, unless the RIV are inferred somehow from the CIs?
1000 samples of size 1000? Okay.
@user99680 I have a sensor which give me updates at a fixed time interval on the order of 10 seconds.  For each variable (S_1, S_2) I have on the order of 100 to 10,000 observations.
Forecast pro, autobox and sas forecast server all have the ability to do batch forecast for 100,000+ products, hierarchy etc. I would suspect the retailers are using one of these software. Consult their documentation for further details.
The meaning of "probability" depends on context. Could you please clarify, then, what is known at the point you wish to compute probabilities? For instance, do you wish to model the probability that tomorrow is the special day *given that* the special day has not yet occurred?
The idea is this: my students have target scores, and then they achieve actual scores. The target scores have come for looking at all candidates, from all schools over many years. The target score tells me what an "average students", with the given student's past results, in an "average college" would achieve. If the results match the targets then we're an average college. The government return data that gives a percentage confidence that the "teaching and learning" in the college is better or worse than the average. I wanted to do something similar.
@AndyW: No. P's subsets reporting self-select themselves to report.
This IMO is better suited for SO, did you check out the [lattice questions over there](http://stackoverflow.com/search?q=%5Br%5D+%5Blattice%5D+label). Some examples over there may be pertinent to your example.
@Srikant: I updated the question with some background information
*Very* closely related: http://stats.stackexchange.com/questions/51792/how-to-find-a-suitable-association-of-color-with-data-value-in-a-visualization.  Does this answer your question?
Rather than base your question off the content of a link, please describe the problem clearly, and the approach used there. If the link disappears your question would be left with no context, which conflicts with the aim of a permanent repository of questions and answers. Having a link for additional context is fine, but the question should stand on its own.
Several solutions are offered to the similar question at http://stats.stackexchange.com/questions/1142/simple-algorithm-for-online-outlier-detection-of-a-generic-time-series.  (That one asks for an online algorithm, which imposes special considerations on how the most recent data are analyzed and raises an ongoing multiple-comparisons type of problem.)
@Macro Nice answer (+1)! I wonder whether you could fit the `gls` model but allowing for different residual variances for each Species with the option `weights=varIdent(form=~1|Species)` (regarding the second question)?
Baseball is many US statisticians' favourite source of examples so a Google (/ Scholar) search will bring up several relevant articles, e.g. Morrison and Schmittlein (1981) http://www.jstor.org/stable/2630890 .
I'll leave it to someone more familiar with both baseball and R to answer your question.
I don't really see where your statistician friend is coming from, to be honest. You can overfit with polynomials and splines just the same. Overfitting comes from your class of models having excessive capacity; what distinguishes the performance of various models is how they restrict their capacity. For (natural) splines, it is knot placement and number, for polynomials it is the degree.
"This seems strange because you will only observe y once and only once (y is the complete vector/matrix/etc. of responses). How can this be "distributed"? In my view it can only be distributed in some imaginary ensemble, nothing to do with your actual observed response. At the very least, any such presumption of the response "being distributed" is untestable" I'm confused; are you saying we can't test $H_0: y\\sim f_0$ vs $H_1: y\\sim f_1$?
Are there really two types of 1's in your data? That is, 1 meaning the car could be moving but is not versus 1 meaning that your car really could not be moving at this time. That would be called one-inflation (it's usually zero-inflation). If so, you need to model when the car can be moving or not versus when it might be moving but is not.
6.8396e+052 for the covariance matrix. the data is from file http://www.uploadmb.com/dw.php?id=1348237753
"Different" isn't a useful hypothesis.  The two groups can be as close as you like & still be different, so no test will ever reject it. The power function of the test shows the probability of making a Type II error for a given degree of difference.
@whuber: There are a number of examples, but say Table 95, "Number of opposite-sex partners in lifetime".
Questions like this have been asked on this site many times. The general answer for your title is Yes. However, if the number of components is less than the number of original variables, you are at risk to lose some important, for clustering, information. [This picture](http://stats.stackexchange.com/q/12861/3277) is an example: the only one PC taken separates the two clouds considerably poorer than both PCs (= original variables).
It sounds like you are wondering how to find the distribution of $\\log(\\theta)$ given the distribution of $\\theta$.  This is a special case of the situation where $\\log$ is replaced by any measurable transformation $f$, which is discussed in general at http://stats.stackexchange.com/q/14483 using--for illustration--the specific example of $f(\\theta)=\\theta^2$, but everything there applies to the logarithm and an easy calculation shows that $p(\\log(\\theta)|\\alpha=-1) = 1$ for $0 \\lt \\theta \\lt 1$.
exactly. One could solve it directly (there is a closed form solution) but involves inverting a matrix $O(n^{3})$. And that is the reason of using iterative methods (specially when dealing with high dimensional problems).
Thanks again, @JakeWestfall. I've read over this about 3 times over the course of a few months and it's helped in various ways each time.
In concert with what StasK said, there's also a [third edition of Magnus and Neudecker's Matrix Calculus book](http://www.janmagnus.nl/misc/mdc2007-3rdedition.pdf) which Magnus makes freely available on his website.
@BrashEquilibrium -- The theoretical support is for BIC not AIC, as far as I know. The problem is that BIC assumes that one of the candidate models is the **real** model, which is probably not the case in most practical situations. AIC doesn't make this assumption - so people substitute it for BIC. But I think this substitution isn't theoretically grounded, but found to work well in practice.
Give this pdf a read: http://cran.r-project.org/web/packages/betareg/vignettes/betareg.pdf  Lots of useful examples that should answer your question.
(+1), nice one.
@amoeba, thank you for the comment. The general (for any dimensionality) formula is XV where V is the column-normalized (to SS=1) matrix of eigenvectors of the LDA extraction. These normalized eigenvectors of iris data I display here: http://stats.stackexchange.com/a/83114/3277; the algebra of LDA is here: http://stats.stackexchange.com/a/48859/3277. A plot where I used LDA's normalized eigenvectors is here: http://stats.stackexchange.com/a/22889/3277.
@andresmh, Maslov and Sneppen (*Science*, 2002) have a visualization that might be useful in this context. Searching through recent stats/comp-sci--related citations of this work, I found [this](http://statistics.stanford.edu/~ckirby/techreports/GEN/2010/2010-13.pdf) as well. [Here](http://arxiv.org/PS_cache/arxiv/pdf/1012/1012.0201v1.pdf) may be another related work.
@mpiktas, It works, but change the parameter little bit.Actually i want to simulate with estimated parameter and initialize with this new value into the future
+1 - I hadn't noticed the reference to control variables in the OP.
as an addendum, the `grm` function in package `ltm` is the best one for a mix of variables with different levels
I've made some additions to my answer, and intend to make some more, to give more details.
@PeterEllis I am suggesting that there are more appropriate models.
+1, just as I suspected :)
Thank you for the detailed insight. Your feedback on terminology is appreciated (not my area of expertise). The profile represents temperatures within a 24'-32' tank; beyond knowing that *most* increases should occur at or before the 10' mark (heat source), I do not have other concrete details available. I did replace the Savitzky–Golay filter with a Median filter that seems to be producing better results; however I think you raise an excellent last point and I will explore that further today.
I am using this forum for the first time, hence bit struggling with typing etc. Please forgive me. My only requirement is to know how to use estimated parameters (above) to find quantiles or inverse of probabilities?
I'm not sure you can say much since the F-measure is function of both precision and recall: http://en.wikipedia.org/wiki/F1_score. You can do the math though and hold one (either precision or recall) constant and say something about the other.
The short answer is yes. The tricky part is getting prediction intervals on the re-aggregated forecasts as your components are probably correlated.
@Thanks for highlighting that. For some reason when I go to package installer - R only recognises 1.6-4 as the latest zoo package. Will check it now.
:-) Yes that is it, you summarized it nicely. That is exactly one of the things we are trying to establish.
Disagree completely: bias/variance is a tradeoff entirely orthogonal to type I/type II error rates.
(+1). But shouldn't it be "$n \\times k$ matrix" instead of $k \\times n$?
I've formatted your post for LATEX. Additionally, I changed B2 to $B^\\prime$ for clarity (so that B2 doesn't look like $2B$ or $B_2$). Please check that I did not inadvertently alter your meaning.
You are right, sorry for the ambiguity. Thank you very much for your quick and precise answer!
I've heard this anecdote before (maybe also from you?).  It hit home for me, and since first hearing it, it has come true on multiple occasions (i.e. the difficulty of isolating the problem).
> The ratio between the variation explained and the original variation is your R^2

Let's see if I got this.  



If the original variation from mean totals 100, and the regression variation totals 20, then the ratio = 20/100 = .2   You're saying R^2 = .2 b/c 20% of the mean variation (red) is accounted for by the explained variation (green)  



(In the case of r=1) If the original variation totals 50, and the regression variation totals 0, then the ratio = 0/50 = 0 = 0% of the variation from the mean (red) is accounted for by the explained variation (green)  I'd expect R^2 to be 1, not 0.
What is 72%? what are you measuring? If AUC is below 0.5, maybe there is some error in how you compute it. Have you by any chance inverted your predictions by mistake?
You are correct that you will have measurement error in X, which will lead to [endogeneity](http://en.wikipedia.org/wiki/Endogeneity_%28applied_statistics%29). This causes biased (closer to 0) parameter estimates. Weighted least squares is a good idea, but will only adjust for relative uncertainty b/t the regions. You may be able to use [errors in variables regression](http://en.wikipedia.org/wiki/Errors-in-variables_models).
Could you maybe give an example how this is done in MOSEK? I don't seem to find it in their documentation very well
Consider the case where you have equal counts in all 15 bins, assumed of equal widths. Obviously the uniform distribution is a perfect fit. But then (upon comparing the counts to their expected values with the uniform distribution) the R^2 is not even defined. Worse, the slightest variation in the counts *will* allow R^2 to be calculated, but its value will be practically arbitrary, somewhere between -1 and 1. This alone shows that using R^2 is inappropriate and likely to lead to erroneous results.
Could you provide a reference for the above screenshot? It is good practice to properly cite the work of others. By the way, this should be amended to your previous answer, not posted as a new one.
Any justification to add? (in order to avoid the one-liner answer badge :)
I think this will definitely work...in my head I was so attached to the idea of a proper CDF that this didn't even occur to me. Thanks!
+1 since this only requires the fact that the OP already mentioned in the question: ${\\rm var}(X+Y)={\\rm var}(X)+{\\rm var}(X)$ when $X,Y$ are independent,
It might be better to simply look at a confidence interval for the effect (even two-sided), and to discuss the conclusions in the light of that interval. You may get less tangled up in the hypothesis-testing logic. An alternative would be to be to begin reviewing the way hypothesis tests actually work (I don't mean the hand-wavy explanations, I mean the mathematics).
(0) The LIP is a version of the Chinese Remainder Theorem :-). (1) It's a matter of model building. In reality you rarely know the "correct" order to use, so I purposely chose to fit a polynomial whose order was not the same as the one used to generate the data. (2) The `-1` causes `lm` to omit any constant term, which is crucial. (3) Yes.
@Michael Mayer. As I told, it is theoretical question, not practical. # of observation does not matter here, I put 2000 just for example. I'll take a look on some articles about "data dredging", thank you for suggestion!
Thanks for the edit.  Is it possible to directly specify the size of the population effect for coefficient b12?
You're welcome, @AaronD. In my opinion, you should definitely encourage them to change their phrasing at a minimum as it is either misleading, or implies they misunderstand the topic. I would predict that they won't attempt the compromise power analysis, but they could also just report descriptive statistics (means & SDs) & an effect size w/ appropriate confidence intervals.
Is this for some subject/course/study/self-study? An exercise from a book, etc?
You could still estimate a main effects model (i.e. if you assume no interaction). You'd have 4 parameters (intercept, information and 2 for picture) and 5 cell-means. Presumably you have replications as well, in which case you could even model all 5 cells by including one interaction term, but that would probably be a little odd; unless you have a strong reason to fit it, it might be better to look on using that as a model-check.
Canopies are more of a pre-clustering. An aggregation step that helps speed up a later operation.
I believe that particular formula is based on regression calculations.
I really don't think you want to stop with just an equal-volume criterion, Denis. Look at the 2D case, for instance (it generalizes): pick $(r,\\phi)$ so that $\\phi$ is uniformly distributed on $[0, 2\\pi/N)$ and set $x_i = (\\cos(\\phi+2\\pi i/N), \\sin(\\phi+2\\pi i/N))$, $i=0, 1, \\ldots, N-1$. The Voronoi cells of $\\{x_i\\}$ within any disk concentric with the origin have *exactly* equal 2-volumes.  Depending on how $r$ is distributed, you might never start near the origin or you might never start near the boundary of the disk.  That sounds like poor behavior for a search algorithm.
The Mahalanobis distance makes no sense when the distributions differ.  (It's like saying "Peter lives on a sphere and Paul lives on a Euclidean plane; how do we compute the distance between them?")  Perhaps you could back up a step and help us understand the motivation for the question: what exactly do you want to accomplish here?  What is the statistical context?
What is N? What is the nature of the weighting scheme? Are the weights known?
No need to worry about the fact that the tree does not complain. Trees have ways of coping with missing data; that is one of their main strenghts in mi opinion. Package rpart, for instance, uses surrogate splits: if at some node the variable required to perform an split is missing, another variable producing the closest split in the training sample is used.
@Frank Those are good points, although perhaps a little extreme (because "taking out" variables applies in some forms of nonlinear and robust regression, too). I believe the first line of the question indicates that your objections don't apply in the present situation, but it is good to know the limitations. I also would hesitate to characterize this solution as a "trick"; for instance, it is the conceptual and computational basis for most of Tukey & Mosteller's book on [data analysis and regression](http://books.google.com/books/about/Data_analysis_and_regression.html?id=n4dYAAAAMAAJ).
Look at a plot of the natural log of the data. I bet that looks a lot closer to normal!
Thanks, after looking back in my textbooks, this is consistent with what I found.
You make some interesting implicit assumptions in your calculation. First, it focuses only on the possibility that the pair occurs among the first two within a sequence of four cards: there are other ways a pair can occur, so this underestimates the probability. Second, it ignores the sequence among the second two cards, thereby overestimating the probability. Fortunately, the two mistakes do not cancel, revealing the fact there is a problem!
I would add that in some fields "hierachicical"=random effects (or mixed effects) models. Again you can have mixed effects ordinal regression. But from the description not clear where the random effect is.
+1 To my recollection, it's quite a good choice.
@IriehStat.  I agree with you.  In the end what do you do for the user.  Do you provide just one model or might you give an ordered list of competing acceptable models?  If not the latter maybe that would be a good option to add where you limit the list to a small number.
That's exactly what I'm getting at, Jeremy: the tradition of 5% and 1% is based, at least in part, on a concept of statistical risk ("a little lax" or a "touch stringent") and does not originally derive from any convenient rule of thumb.
It is a Markov regime switching time series model, since at the time $t$ the process depends only on process value at time $t$, the regime $w$ and innovation $\\delta$. If there is a more specific name, then probably it is known to researchers specializing in that field (regime switching time series models).  The multiplication of $S_{t-1}$ and the innovation $\\delta$ seems strange for me, since it is hard for me to think of an application where such process might arise.
If you can get away from always thinking in terms of $p$ values and $z$ scores and R commands and want to learn how to _use_ tables of normal distribution probabilities (instead of just being told that the answers can be looked up in tables), [this answer](http://math.stackexchange.com/a/202612/15941) on math.SE might be useful.
Actually the [Wikipedia page](http://en.wikipedia.org/wiki/Look-elsewhere_effect) is quite good and I think answers your question (the answer is that the look elsewhere effect is more generally known as 'the problem of multiple comparisons').
I think this confuses slightly (a) magnitude of variance (b) stability of variance. Consider $x=1,1,1,2,2,2,3,3,3$  and $y = 10^{-9}, 10^{-8}, 10^{-7},10^{-6},10^{-5},10^{-4},10^{-3},10^{-2},10^{-1}$. Then a log transformation results in negative values and bigger variances. But the bigger variances are just a side-effect of changing units and are not problematic. The key is that the variances of $\\log y|x$ are constant and most statistically experienced researchers would choose logarithmic transformation here. That said, I agree that log(deaths/distance) definitely seems the thing to try.
You regress $y$ on $x$, get the residuals $\\hat u$, and regress the residuals on their first lag to get $\\rho$. Once you have $\\rho$, you can transform the data. Does that make sense?
This question asks for a one-dimensional version of [area-to-point interpolation](http://onlinelibrary.wiley.com/doi/10.1111/j.1538-4632.2004.tb01135.x/abstract), which is fairly well studied in the mining industry. The referenced abstract explicitly notes that geostatistical methods yield "coherent (mass-preserving...) predictions." I believe these approaches overcome objections made by @Nick Cox.
The factor scores should get produced as part of the pca output - you shouldn't need to manually calculate them.  Also if you really need help, you should be letting your boss know - this is a fairly sophisticated piece of work.
@Marco: I don't think that your thread-brothers are trafficking in unconventional definitions. Rather, we are trying to provide what we understand the questioner is probably after.
I don't think I wrote that--although I did reverse the order and wrote $m_X-m_Y$ instead of $m_Y-m_X$, which I have since fixed.  The first claim deals with $Z_{m_Y-m_X},$ which is formed out of $X$ and $Y_{m_Y-m_X}.$ Thus we are comparing values of the form $y_i - (m_Y-m_X) = (y_i-m_Y) + m_X$ to $m_X$, which is tantamount to comparing the $y_i$ to $m_Y$, not $m_X$.
@whuber The **permutation** distribution can be asymmetric as it depends on the sample values. Values group A: 1, values group B: 2, 2. Three relevant permutations are possible yielding $T^{\\star} = {-1, .5, .5}$.
I don't know (no downvote from me) :(
Actually, I stated it wrong as well... it depends on theory, method, and data... the entire study.