tttjjj/elt_bench huggingface dataset:
I hoped you know that because covariance matrix is symmetric U=V (with columns corresponding to the PCs) and is the same eigenvectors matrix as eigen-decomposition returns, and S's diagonal are the same PCs eigenvalues as eigen-decomposition returns.
@jokel This looks like a good answer, but it's usually best to wait a day or two before marking a best answer so that users from around the world get a chance to answer.  Not a strict rule by any means, but just a general suggestion for this site.
Possibly of interest: [central-limit-theorem-versus-law-of-large-numbers](http://stats.stackexchange.com/questions/22557/).
I am not sure I understand what you did or intend to do. Also, in what ways are your result “better”? One problem in simply lumping different items together is one of interpretation: If they are not correlated with each other in the first place and do not reflect the same factor, it's difficult to make sense of any relationship you might find between your scale and other variables.
Better to write your own if you want to learn... it's not that hard, just start with the definition of the acceptance ratio and work your way from the inside out, so to speak.
1st comment: As far as I know, Rijmen et al. *do not* say that one can fit a GRM using `lme4::glmer()`, and indeed I don't see how one could--please provide a page reference in Rijmen et al. if you disagree. 2nd comment: I *think* you can fit GRM in `ordinal::clmm()`, although I haven't personally looked into it closely. 3rd comment: I don't know what your actual question is. Please edit your question to very clearly state what it is you want to know or are confused about.
What part of interpretting the strength of the relationship are you struggling with?  One piece of advice I can give is that the second table will struggle to find any relationship because of the low count of disease=TRUE.  The p-value is mainly telling you that Pr(disease=False & exposure = yes) is well predicted by the marginal counts.
thank you for your answer, you're right about RM design. 20 observations should be multiplied by roughly 3 as my data are triplicates (most of them). The problem is that with t-test I find differences, but can't with ANOVA. Moreover, when I have the final 5X5 table with indices there are only 1s in the control group (after normalisation). So, dispersion of five 1s equals 0 as you well aware. That is I face a problem of violation of ANOVA assumption on homogeneity of variances.
FWIW, the use of the term "interaction" is incorrect (see, eg [here](http://en.wikipedia.org/wiki/Interaction_%28statistics%29) or [here](http://stats.stackexchange.com/a/49701/7290)). I would say something like, '... but about identifying situations where the effect size is large and the size of the population with such attribute is large as well'.
I doubt that's the right formula: division by 23% would not be useful or informative.
@ Daniel: As you can see, the test statistics are identical, e.g. for `videocond`: $-1.88$ but R assumes a t- and Stata a stand. normal distribution. I would probably go with the (more conservative) t-distribution... but that's just me :-) To obtain the (two-tailed) p-values in R, just run `(1-pnorm(abs(-1.88)))*2` which gives $p=0.06010808$ (z-values, i.e. assuming a stand. normal distribution).
I think "optimization technique" is generally interpreted as referring to a method that can be used to optimize a given function. Yes, you are doing optimization when you solve for the OLS estimator but I don't see how that makes it an optimization _technique_. The Simplex algorithm, Newton's method, Conjugate-Gradient, etc... are optimization _techniques_.
(+1) for the links.
@Bort I don't believe so, no. Consider a Boltzmann machine with two nodes and a weight of -1 between them. With states in $\\{0,1\\}$ states, the energy levels of the four possible states are 0, 0, 0, and 1, whereas with $\\{-1,1\\}$ states they are 1, -1, -1 and 1. The two energy functions are not related by a constant offset.
+1 As someone who first learned stats from this book (right after the first edition came out :-), who has taught from it, and who has given many copies away to colleagues and clients, I can warmly recommend it. The exposition is marvelously clear. There's nothing outdated about the first edition, by the way: the third and fourth editions use more recent datasets but are otherwise about the same.  This means the first (and now the third) editions are real bargains.  (The second edition's ok, but it started to veer off course a little, a mistake that was corrected in subsequent editions.)
+1 in general but particularly for the reference to Wilcox's book and R package.
$S(t)=P\\{X>t\\}$? If so, what is the role of the random variable $T$ in this computation?
Sounds as if you have correlations between the observed values and each of three variables that comprise predicted values.  Where are you getting stuck?
Glad I've been helpful. Try `as.numeric(factor(result))` and see if that helps you figure out which is being coded as the higher-level value...There might be more appropriate ways of doing that, but this way ought to work in its own clumsy fashion.
+1 for enlightening remarks about EDA, although it's still not clear (to me) whether the OP is after EDA or not...
+1 for this nicely illustrated response and an interesting comments thread. (I should definitively start learning Mathematica graphics!)
(+1) Very good. I would only make the notation a little more standard. Data: $(x_1,\\dots,x_n)$. IID random variables $S_1,\\dots,S_n$ with $P(S_i=k)=\\frac{1}{n}\\;I_{\\{1,\\dots,n\\}}(k)$. Result: $P(\\cup_{i=1}^n\\{S_i=k\\})=1-P(\\cap_{i=1}^n\\{S_i\\neq k\\})=1-\\prod_{i=1}^n P\\{S_i\\neq k\\}=1-(1-1/n)^n\\to1-1/e\\approx 63.21\\%$.
I don't think the appropriate question is whether it's "plausible". It most definitely is plausible if that's what the calculations show. If you're asking whether you did the calculations correctly, I think we need more specific information about the output you're getting. As written, the question is too broad to answer correctly.
In any case, the two dependence structures are identical (as long as you do not have interventions).  Also, I think you are confusing the direction of reasoning with the directions of the arrows.  Finally, all variables in a Bayesian network can be a response variable and can have arcs to other variables.
@Max And we dont get the same results, atleast not when I tried to compare the two.
It looks like the R program uses a different weighting scheme for the studies than what we used. We used the Hunter & Schmidt (2004) approach with sample-size weights, but it appears that the R program uses the inverse of estimated error variance which will result in a slightly different weighted average. Because we are basing our SAMD values on this different average (from R, vs. our computations), the results from R might not correspond precisely to our other analyses. Does anyone have thoughts about the similarity of these results and/or changing the R program to use sample size weights?
I will have a look at that paper. Thanks Ken!
great, this is the solution
Think about it: the chance that $X$ exceeds 3.77 (the p-value) must be *less* than the chance that $X$ exceeds 3.1274 (equal to 0.05 by construction).  "The probability that 3.1274 is greater than 3.77" does not make any sense.  If you're still unclear about this, draw a sketch of the PDF of the F-distribution and highlight the areas under the curve that correspond to (a) the rightmost .05 of the total area and (b) the area to the right of 3.77.
Please define what you meant with $X^{-1}$. Is it the inversion on a non squared matrix?
Hi Dimitris, APriori is a very classic algorithm and there are many good tutorials around the web (or in data mining textbooks) you can easily find. I just wanted to point out that your problem is a very well studied problem in data mining and there are numerous ways for attacking it. You can check the book [link](http://www-users.cs.umn.edu/~kumar/dmbook/index.php)
Alright @aix, how often do you perform these calculations. Is you box running all the day or more idle?
This is a great answer, curious to know if there are alternative techniques though. Dave.
As predictors, binary variables behave statistically the same way be they declared as categorical factors or as continuous "covariates". I prefer to use them as continuous because it saves output.
Emile, This question is similar to one I just posed (http://stats.stackexchange.com/questions/51475/series-dimensionality-reduction-for-classification-input). Would you be able to post any R code for the DFT features?
Of possible interest: [Is there an alternative to the Kolmogorov-Smirnov test for tied data with correction?](http://stats.stackexchange.com/q/35606/10525), [Equivalent of Kolmogorov-Smirnov test for integer data?](http://stats.stackexchange.com/q/35819/10525).
"qualitatively different" is basically "is this different enough to matter?" which is more a question of effect size than significance (hypothesis tests will identify even trivial differences, so they're simply not answering a question like "how different are they?"). In fact, it's often the case that we already *know* the null to be false, so we certainly shouldn't be using one in that case. Some diagnostic measure of whatever you regard as important in terms of difference in distribution might be useful.
(+1) for the Anderson paper
@Max It is assumed only that the effect, whether positive or negative, decays at a constant rate over time.  If you are sure that the change in effect over time is more complex than that, you can incorporate that assumption.  (E.g., Y could be the sum of two terms [presumably of opposite signs] having two different decay rates.)  This will introduce additional parameters that play the role of \\theta and it could complicate the model sufficiently that you might have to rely on ML methods.  I think you'll find you need lots of data!
@Suresh: the problem with coming up with a measure of similarity is that there could (and probably will)  two seemingly identical sets which belong to completely different branches of the hierarchy. So looking at the elements only, one would say that they have very little distance to one another and yet in the "right answer" they should be far apart from one another. I am starting to feel like what I want to do is rather impossible with the amount of information available at this point. :S
Can you work on improving your question? As it stands, it is hard to tell what is being asked, unless you are just looking for some kind of census data.
Could this be done not with polynomials, but with functions of the form a*sin(b*x), for instance?
@Jase Neural networks usually have lots of parameters, therefore in many cases if you use a small dataset, you will probably just overfit. Of course, everything depends on the task, but most of the most impressive NN results nowadays use extremely huge datasets.
Martin, Chebyshev's Inequality has nothing relevant to say about this situation, because any values of the mean and SD of the underlying distribution are *estimates,* not fixed values. Logarithms are a good way to assess ratios and it is wise to avoid bootstrapping such a small dataset (+1).
I agree with your response on the assumptions but ignoring those issues for the moment what is your assessment of the issues with the two different methods of analysis.
Did you look at this question, [Statistics/Probability Videos for Beginners](http://stats.stackexchange.com/questions/1761/statistics-probability-videos-for-beginners), and the one that is linked to it?
@Dan My point is that the title mis-characterizes the question and confuses any answer. Saying that you used e.g., a probit link for a logistic model is nonsensical.
@Alexis, you don't need to nest in this case (and probably shouldn't)
I would not easily discount Euclidean distance, it often works pretty well even for binary data compared to other binary metrics. Which tool is appropriate for you depends very much on your data and requirements. What kind of data sets are you dealing with? How many features and objects? Do you know how many clusters are in your data?
Note that the Maher model (4.1) in the Dixon and Coles paper is just a straightforward GLM. It is very similar to the second model [discussed in the answer here](http://stats.stackexchange.com/q/15776/2970).
Good to have it checked by an expert, thanks! Strange that a couple of web sites have this in their examples.
Thank you.  That work provides a pointer into a web of recent research on dasymetric mapping.
Thanks! I agree Simulated Annealing might better handling such situation, but this is an assignment that requires to use Metropolitan-Hastings algorithms. So I am afraid I have not other choice.
See the Fisher-Tippett-Gnedenko Theorem at http://en.wikipedia.org/wiki/Fisher%E2%80%93Tippett%E2%80%93Gnedenko_theorem, for instance. Its conditions raise the question, what exactly do you mean by a distribution not "known"?
(1) I think @EnergyNumbers has given you the best answer you'll likely to get. (2) The question is formatted too oddly to get more. Why the matrix notation for a simple regression problem? Have you done/feel you understand weighted regression? "I cannot seem to make it work" is a very vague description of your problem.
How are you assessing correlation?
To get some insight, think about simple situations.  *E.g.,*, what happens when $b$ is a constant?
@Jeromy: I'm sure you're right. Just sharing my ignorance.
+1 This is a very nice question, but do I miss something? The answers involve mostly links to some challenging data sets and open courses. On the other hand, ProjectEuler consists challenging questions of different levels and you increase your skills while you solve them. And you can see the others' solutions after you solve the problem. It is similar to http://www.pythonchallenge.com/ I think. Are you asking something like this? A clarification would be very nice.
@IrishStat It was a good comment and took me back to my training in the 1970s.  At that time I was learning about time series/forecasting methods for my civilian work in the US Army.  We were using exponential smoothing as a way to forecast based on historical data over subjective estimates that were being used at the supply depots.  Someone made the great suggestion to me to look at the more general ARIMA models and the 1970 text by Box and Jenkins and so began my interest in time series that became part of my career.
@Roland - Good point.  Change made
Can you clarify 'epoch'? Do you split the data to training and test set 100 times and average the results?
I'm not actually sure if this is right. Looking for a second opinion :D
Can you give more detail? Do $x_1, x_2, x_3$ all take values in $\\{1, 2, 3\\}$? It looks like in all your examples that each of $\\{1, 2, 3\\}$ might occur exactly once; is it possible, for example, to observe $(x_1, x_2, x_3) = (3, 3, 3)$?
Is it bad that I recognize some of these?
Please don't simultaneously cross post.
@Peter Flom. Typo on my part I'm afraid.  Rest of my comment makes sense if you put the missing 'non' in front!
A model parameter is some parameter we want to learn about to make quantitative claims about the data (e.g., the variance is high for this school versus that school in test scores). A latent parameter is some augmentation we make to the data to include some unobserved variable that usually allows us to make conjugate relationships for computational ease. It's hard to answer (1) because $f$ is not specified.
For each "read" (you'll have to excuse me, but I don't know precisely what a "read" is), something is either in a particular category or not in that category? Isn't that binary? (each read is either in category or not)
Of all the sites you have listed, I don't see any one that provides downloadable data. Most of them don't even let you see historical data. The most useful of them would be IEM, but even they only give you the market price history. Maybe I missed something?
(ctd)... (for example, if you assume a location-shift, the distributions would have to be identical apart from a possible location shift, which you already seem to have ruled out); if you assume a change in scale, the shapes would be the same apart from a scaling constant; (v) there are other possible tests, but again - if construed as a test specifically of *means* - they tend to assume the distribution is the same under the null.
a bit harsh perhaps?
Sounds like you should be using an informative prior, as you have non-data based information.
There is also a problem in social science of "weak hypothesis" (eg effect is positive vs negative).  And in that "self efficacy" example, you could view that as an internal predictor of performance that each person has built up.  So it is probably similar to using a "black box" prediction as an explanatory variable.
Thank you very much! Makes sense!
No, they are not independent, as they come from the same set of sample data.  This is the basis of the reason for adjustments such as the Bonferroni Joint Confidence Intervals (this you'll probably learn down the road).  But as a simple comment, no they are not independent.
@whuber Thanks, yes, that's relevant. There's also [this](http://stats.stackexchange.com/questions/29248/strange-result-of-post-hoc-test), and it can also happen [the other way around](http://stats.stackexchange.com/questions/35175/non-significant-group-effect-but-lsd-post-hoc-is-significant-why) (though you wouldn't normally do a post hoc in that case).
Are you willing to assume that the historical data form a set of identically distributed and perhaps even independent curves?
Metrix, I think you may have misquoted: the variance of the sum is the sum of the *variances* (not "covariances") if and only if the variables are *uncorrelated* (not "correlated").
@pacomet - ok, i clearly failed at my example. Above, `gnp` is an arbitrary example of a time series object (actually it comes from the help page for `?ts`). When making that time series object, we define a start year and month (1954 and month == 7), and then also specify that the number of observations per unit of time is 12 (monthly data). That time series object now has some "meta-data" associated with it, including the position of each observation which can be accessed by `cycle()`. Since that information is directly linked to the ts object itself, we can use it for subsetting / indexing
*"Assuming a fair con...the next flip has a 50/50 chance of being heads"*, I think you do have a deep philosophical truth here. You could expand the answer to explain what happens if it's an unfair (AKA regular?) con.
I agree this was very helpful: I was searching for quite a while for more information on "rim" weighting and couldn't find much. Your post helped me learn that it was also called "raking" and then I found everything I needed! I should have started here instead of google...
Can you refine this question, & make it more specific? At present, it isn't a good fit for this site.  You may want to read our [FAQ](http://stats.stackexchange.com/faq) to get a better sense of site policies & what constitutes a good question for CV.
@Glen I want to run it again to estimate new values for new rows. It is in an XML file so the model does not have to be recomputed (this takes about 30 minutes because of many ensembles).
@RamSharma: I took the liberty to make a sample positive definite covariance matrix, and made a few minor edits; feel free to edit back if I've changed something important.
I think Wikipedia's article expresses the point somewhat poorly.  A quick reading can make one think that $$f(x) = e^{ax^2+bx+c}, -\\infty < x < \\infty,$$ is a normal density function for all choices of $a < 0$, and arbitrary $b$ and $c$, whereas what is meant is that $a$ can be chosen to be any negative number, $b$ can be chosen arbitrarily, and $c$ is then determined by the values chosen for $a$ and $b$.
@user2602256 `c` and `gamma` should never be negative!
@mathstudent The null hypothesis and the alternative hypothesis are complements of each other. So if the null hypothesis is $H_{0}:U_{d}=0$, the alternative is $H_{1}:U_{d}\\neq 0$. This hypothesis is a two-sided hypothesis and can be checked by `t.test(PM,AM, paired=TRUE, alternative="two.sided")`.
@MichaelMayer, thank you and I updated the question with your suggestion!
@AndyW I checked my notes and Vuong is indeed for ZI vs Poisson.  Updated post.  kara I missed the reversal.  You could have under-dispersed data, in which case the NBD is also potentially the solution :-)
I added a code snippet to my answer. Regarding the other question, there is a whole literature on it (with which I am in fact not very familiar) and I also came across some approach based on the idea of generating several curves and comparing them systematically. Mostly, what I do in practice is go with the default bandwidth but also look at the data directly (stripchart possibly with jitter or transparency to avoid overplotting, boxplots) to check if I might have missed something strange. The important thing is not to read too much into it, especially with small sample sizes.
@shabnam: In your original question, you are talking about a single woman. Could you please edit the question accordingly? Using Wilcoxon test is clearly not appropriate for clustered data. Btw: The Wilcoxon signed-rank test is not directly linked to median differences.
Was this on the Freakonomics podcast?
Thanks for confirming my suspicions.  Some SAS procedures print between and within variance.  My interpretation is exactly as you explain above.  Rubin's rules can be applied to obtain proper parameter estimates and standard errors.  However, it looks like there's a need for a theoretical paper dealing with the issue of combination of hypothesis tests.
@Glen_b Cool. Do you mind elaborating a little? Or providing a link?
the binary case might be a sufficient first start...
It does depend on the estimate but there is only one estimate for which the t with 19 degrees of freedom applies and that estimate is the square root of the usual estimate of the sample variance.  If you use a different estimate of the standard deviation you have a different reference distribution for the test statistic under the null hypothesis.  It is not the t.
If some logical, objective criterion says that the two datasets are very similar, then I can continue my analysis with one of them. On the other hand, if the two dataset are very dissimilar, then I can't do the analysis with either until I can explain why they are dissimilar.
I like the [Freedman, Pisani, Purves book](http://stats.stackexchange.com/a/30548) and I think you might enjoy it, too.
Thanks Peter. I'm going to carry on recruiting participants (10 for each of the four levels) and then check if the effect is large enough to have statistical significance in the interactions. Many thanks again.
Do you have any reference to the log link GLM? I tried searching, but didn't find any resources to what you are referencing.
1) It might be good to change your question so that it doesn't involve time.  2) What is your sample size?
No, a temperature independent proposal isn't wrong but it may not be as good as one that does depend on the temperature. If you think about how flat the stationary distribution is at high temperatures and how peaked it may become at low temperatures, it makes sense that different proposals might be better at different temperatures. A simple example would be a random walk proposal where you migth want to adjust the scale according to temperature.
The link you gave points to your own machine - 127.0.0.1
I assumed you were talking about R package; correct me if I'm wrong.
@PraneethVepakomma It is an old monograph and I don't know whether or not it formally addresses your issue but have you seen Don Berry's book "Bandit Problems"? I think it was published by Chapman and Hall.
Regarding the non-repeated measures case, I don't really understand how you would compute a “gain score”; scores need to be paired to be able to subtract T2 scores from T1 scores. If you measure a completely new sample of participants at T2, you only have two bunch of scores with no obvious way to match them to the T1 measures.