tttjjj/elt_bench huggingface dataset:
Maybe this paper is interesting: [Multivariate data analysis: The French way](http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.imsc/1207580085)
Clarification question: the scalar time series you talk about are the *column* vectors of the given matrix, and the states at any one time are in a *row*, right?
Just read section 1.1.
David, the Choice of "War and Peace" as a control text (More precisely the beginning of the Hebrew translation of "War and Peace" of the same length as the Book of Genesis) was done by the original researchers. The story according to Aumann is this: When Bob Aumann who carefully followed the experiment told Kenneth Arrow about the marvelous findings in "Genesis", Arrow asked what about "War and Peace". Aumann then started reporting about the war and peace situation in Israel but it turned out that what Arrow asked about was if the same phenomenon cannot be found in "War and Peace".
> Let me rephrase this, where do you get the scale matrix of your whishart distribution from ?
I'm sorry, but it is not clear what it is that you are asking. It would be helpful if you posted the output of your PCA analysis and described the data set.
(+1) After letting the post sit for awhile, I realized that visualizing the density of the points appears to be a much better solution than trying to visualize the points themselves (although I'm not quite sure what you mean by "stratify by the post #"). I also think plotting the estimated quantiles sounds like a good idea, although for plot 1 & 2 it will likely just be in the massive cloud. Again I don't know what "stratify by time" in this context means either, see Brad Larson's comment on the blog post and my response in regards to this.
By algorithm, I mean roughly the software implementation used for fitting a line to model the mean of a distribution.
Thank you! Don't feel obliged to answer this, but I'd really like to know: what makes rates Bernoulli? What kind of data follows a normal distribution?
The curse of dimensionality is irrelevant, this is not a ML problem. The metrics I refer to are one-dimensional. The distribution looks heavy-tail, in reality it is bimodal and I want to remove one mode completely. The process I describe converges fast for my distributions after 3-4 iterations removing 0-5% of samples. Indeed we remove a lot of samples but this is desirable. In your example you miss the fact that in every iteration st. dev. will be smaller, however an increasing number of samples will fall within the 4 s.d. range.
In response to your remarks, I enhanced the `R` simulation to accommodate specified chances of black other than $1/2$.  I also fixed a mistake in the expression for $x(n)$.  (The error was immaterial to the results--it was just off by a constant factor.)
This is one case where Eureqa will likely fail unless some clever pre-processing is done.  The problem is that CDFs like this typically have some error function-like tails and Eureqa has no function sufficiently like Erf.  If you first smooth this "curve" (it's really the EDF of 400K+ points) and ask Eureqa to fit its *derivative* in an appropriate manner it might do a good job.  (I suggest a Box-Cox transformed normal pdf for starters.)  That presumes the OP needs a good fit to the PDF itself, of course.
Now this is more like it! Good technical answer.
Terrific, Paulo! You got both a working solution and the proper condition on the covariance matrix, which also answers this [question](http://math.stackexchange.com/q/127813/14893). Log-normals prove handier than gammas, in the end.
@HenryB. Answer updated
I'd add that what you do should be science too. In the original example, there could easily be no grounds for thinking that price and size must be linearly related, some grounds for thinking that they should be monotonically related and no grounds for expecting an arbitrarily complicated curve. Regardless of who you are, you should know more about the science underlying the data than any software does. If you are the statistician and you have non-statistical collaborators, it can be a case of what makes sense to them. In short, it's naive to expect formal rules to be the complete story.
Jrennie, thanks for pointing that out. I removed my inaccurate statement about scipy not supporting LBFGS.
The relevance of the CLT for regression is that the *coefficient estimates* tend towards having a normal distribution as sample size increases, even when the *errors* have a different distribution.
Can you describe the data more clearly? Also - what does "proportion of species in a population" mean, exactly? Is this the proportion of all organisms in a population that are a particular species? It's also not clear to me why the fact that you have data at the body part level is relevant. Are these assumed to come from different animals (if so, why not just pretend like they are counts of animals found) or is it possible that multiple body parts came from a single specimen?
Thanks for the reference.  @gung that's a good question.  One (weak) answer is that in some types of models we don't have an $R^{2}_{adj}$, and we also don't have an adjusted index if any variable selection has been done.  But the main idea is that if $R^2$ is unbiased, other indexes of predictive discrimination such as rank correlation measures are likely to be unbiased also due to adequacy of the sample size and minimum overfitting.
Is this correct in the RHS for partial likelihood: "L2(θ|theta)"?
(+1) Thanks for the link.
Thanks a lot for the link, I did not know of this package!
Guys... excellent responses. All helpful. It still surprises me when I get this level of collaboration on the web, just because people are interested. wow... thanks !
But then can't I just use the conditional probability instead of Bayes? I am slightly confused about which one to use sometimes.
You can find [here](http://en.wikipedia.org/wiki/Variance#Population_variance_and_sample_variance) the expresions you need for the calculation of the sample variance (you will see that your reasoning is correct, though the notation you are using is not the most appropriate).
It doesn't seem right to average the two values, especially if they are far apart. This is what led me to try to find good exclusionary criteria. But since I only have only one replicate (and no third as a "tiebreaker") I need to determine which values are close enough to each other to consider reliable. Does that make sense?
The SEM can have several possible meanings here, js.  Does it estimate variation in (a) the mean of all test scores in a group, (b) the mean of the average scores (each average is over $m$ attempts by an individual), or perhaps even (c) some combination of those two sources of variation?  The question can be answered in the sense (a), but unfortunately that sense is probably not meaningful (because it confounds two kinds of variation: variation between subjects and variation among subjects).
I have merged your *four* accounts. (Thanks to @Andy for noticing that!) Since you have registered this one, you can now use it in the future so that you get system notification, you can vote up and accept responses that you found useful or that did answer your questions--even if they lead to new related questions.
... and that you have soft responsibilities instead of hard assignments.
No, it is not true that `the answer is, of course, yes`  (see http://en.wikipedia.org/wiki/Monty_Hall_problem#Criticism_of_the_simple_solutions), as the problem is underspecified and different interpretations can give strikingly different results. However, for _arguably_ the simplest solution the answer is yes.
You can but it can be slightly awkward to turn this into a hypothesis test. The probability of 87 or higher out of 88 with your combined observed proportion is 0.0029; and of 48 or less is 0.0054; but there are philosophical problems with combining these into a single p value for how unlikely these data are under the null. Could be done with care, thinking carefully about 2 tailed v 1 tailed tests and the like, and what it means to have snooped on your data. For the record, in R use something like sum(dbinom(87:88, 88, 0.9121622)) for the calculation of the binomial probability.
"If you combine your data points, you have much more information in your sample and those random errors start to even out more and more." I think this is the key point, and I think that holds even if the die is not fair. Thank you.
You might also enjoy a quick review of [Anscombe's Quartet](http://en.wikipedia.org/wiki/Anscombe%27s_quartet), which illustrates some of the many different ways in which a particular nonzero covariance can be realized by a bivariate dataset.
Did you give them 4 yes/no questions (one for each subject)? Or did you give them 1 question with a "check all that apply" format?
OK, [it turns out I was wrong](http://meta.stats.stackexchange.com/questions/1190/test-of-notification-mechanisms): the "@" mechanism works in comments, not in the questions.  (But jbowman found this question anyway.)
provide your code and output
@cardinal Here's a 4D example generalizing your cylinder: http://i.imgur.com/9eF8N.png .  The model is $(s,t)\\to(\\cos(2\\pi t),\\sin(2\\pi t),2s/5,t^2)$ plus normal(0,.05) error added independently to all four components.  The image is a scatterplot matrix of 300 iid draws with $s$ and $t$ iid uniform(0,1), ordered by decreasing variance.  The proportions of total variance are 49.2%, 46.5%, 2.9%, 1.4%.  The two bottom rows emulate @shabby's plots.  They demonstrate the near 2D nature of the data while revealing the small nonlinearity I put into it.  Is this what you were thinking of?
For more detailed information on log odds, odds ratios and predicted probabilities, it may help to read my answer here: [Interpretation of simple predictions to odds ratios in logistic regression](http://stats.stackexchange.com/questions/34636//34638#34638).
I am hoping the answer (or at least the right tools) lie in part in this paper: Inverse problems as statistics http://live-stat.pantheon.berkeley.edu/sites/default/files/tech-reports/609.pdf . Here the "blurring" operator is the process of computing the integral of the latent function $\\theta(t)$ over a bunch of different observed time intervals (handily this is linear), and the noise arises from a Binomial draw rather than e.g. additive Gaussian noise. It seems inverse problems with Poisson noise are common e.g. when dealing with photon counts, so can hopefully cope with Binomial noise.
"Model data set" means a data set produced by a model. In this case the model depends on a single parameter $a$. I **do** want to find the parameters that best fit my data set. In my case my _data set_ is my _observed data set_ and there's a **single** parameter $a$. Please tell me if this is still not clear.
@ocram That's the ML estimator for a *Normal* distribution (and, possibly, a few others), but it's not universal. For instance, *no* multiple of a sum of squares of deviations from the mean is an ML estimator of the variance of a Poisson distribution.
Thanks - I wasn't aware of the rule of thumb you mentioned. Very helpful.
@Aerik Take a look at my edited asnwer with the formulae now supplied and let me know if that clarifies it better.
thanks for the detailed explanations!
Seems like a good question and a good answer.  I'm just confused because I've seen questions closed because they dealt only with how to program software and involved no statistical content.
A citation for whatever it is you read might help contextualize your problem
Can you be a bit more specific about your experiment? As it stands it appears that you are describing $\\beta$'s and their standard errors in the context of a multivariate linear regression model.
If you start at other initial estimates, but use the same step size, do you still have convergence in the same point? If not, it could be that your problem is simply ill-defined for gradient descent (I believe something like sin(1/x) would cause this). Don't forget that these methods are approximations of the true max/min, so it is normal to see some variation when changing the parameters of the algorithm.
thanks, it's pretty helpful.
Either you messed up somewhere or you didn't. If you didn't mess up, then those two batches are different, somehow. Since you haven't told us anything about them, there's nothing we can say.
Here is an analogy that might help. Suppose you had data on 93 people, which include family name, individual name, height and weight. How would you use those names to build a regression model predicting weight from height? As a first stab, you would probably not use them. However, there might be some information tied up in those family names as families tend to share genes, similar environment, life habits etc. which might be relevant to weight given height. There are more advanced models that make use of that information.
http://www.nd.edu/~rwilliam/stats1/x93b.pdf describes an example like this and explains that the paradoxical effect is caused by negative correlations among the predictors.  The piece also discusses the strange result of having part r's be larger than zero-order r's.  I'm still not sure it's not a hoax, though.
This may prove useful:  http://arxiv.org/abs/physics/0306171
@Nick: They do indeed. I will rectify that.
@Pr0no That sounds fine if your adviser agrees. In terms of calling the variables "fixed effect" or "control," I would suggest picking some well regarded journals in your field and see how other articles distinguish these two terms. The language can slightly differ across disciplines, and our suggestions may not work for you.
You could read a little about Gower index on this site and where one can compute it. http://stats.stackexchange.com/questions/22212/gowers-dissimilarity-index; http://stats.stackexchange.com/questions/15287/hierarchical-clustering-with-mixed-type-data
@philipp thanks for the link, maybe someone can clarify this for us ;)
Indeed they did cheat with MATLAB (Answer actually says that MATLAB gave them the answer). So I'm guessing I won't ever be asked to find a p-value in exact
Sorry it's my mistake, I forgot to install another toolkit which has mk_dbn function in it.
@MichaelChernick Are you sure about the meaning of exactness ? I think an exact confidence interval at the nominal $\\alpha$ level is a confidence interval whose effective confidence level is at least $\\alpha$ (but it is true that such tests are usually (always?) based on the exact distribution of the test statistic and not asymptotic approximation)
Additionally, for tau-b, the denominator has some adjustment for tied quantities. From playing around with the equation, I can't reason why the p-value jumps from 0.8 to 3*10^-10.
A million records might be difficult to hold in memory, if based on other posts you are using R. It seems you will need as many as possible (cases) given that large number of classes. Is there any hierarchical nature to the data where by you could model a smaller subset of classes and then depending on that prediction, model a subset of the overall classes which "roll-up" to that higher level class group?
That seems to be it. I will go away and think about it so more. Many thanks all.
If I understand correctly, there would still exist the same kind of dependence: as long as students in classes are also members of the national population of students, the same kind of dependence appears. Whether this is *substantively* interesting or important for some particular line of analysis is unclear.
Interesting; similar to the problem of determining the number of species: draw n balls from an urn with unknown composition and determine the number of types of balls in the urn.
(+1) From the probability mass function, it's clear that $p(x; k) \\sim k x^{-2}$ as $x \\to \\infty$, which seems to be enough to conclude that it's a power-law distribution. In fact, $p(x;k) x^2 / k \\uparrow 1$ as $x \\to \\infty$.
Retagged "random-variate" to the more popular (and more appropriate) "random-variable".
It may help to know that spherical harmonics are just another orthogonal basis for the Hilbert space of $L^2$ functions. (They are adapted to conform to irreducible representations of the orthogonal group.)  One could just as easily (and effectively) choose Fourier series, orthogonal polynomials, etc. What this shows is that the question really isn't well-posed: precisely what *kind* of "description" of the clusters is being sought here?
You are write in the sense that both are state space models. However, there are lots of difference: KFs are fully probabilistic, in the sense that the hidden states have a proper probabilistic meaning. RNNs on the other hand are deterministic and only the outputs can be used to model a distribution in a discriminative way. Also, KFs are typically estimated with EM, while RNNs are estimated with gradient based methods. If you want more details, feel free to post a question and send me the link, but the comments are too restricted for this.
Can you point me to some tutorial, no mater if R, matlab, weka ...
@probabilityislogic Your point is on the same lines as mine: validity of the analysis is contingent on interpretation. Still, study selection is a special case on which many assumptions of meta-analysis are based. Thes assumptions are too many to review here, but the conclusions would be different if two studies are done on the same population by the same lab vs by two independent labs. The difference is not the population itself but in the investigator, and a common assumption tested by meta-analysis is the absence of an investigator effect.
Here is the result from summarize xb (verified that the "omitted" message is still present).  
 Obs 937, Mean .456002, Std. Dev. .487038,  Min 0 Max 1.185226
which means the fitted probabilities are just 0 and 1, p(y|x)=0 or 1. you may `table(group, mortality)` or `table(habitat, mortality)` to see whether they are perfectly grouped. That might be the reason.
Normally, yes, you keep your factors all together. There's several R packages that can do this, including `glmnet`.
This would take a random sample of data.frame rows: `test_data <- real_data[sample(seq_len(nrow(real_data)),random_count,replace=TRUE),]`. Is that what you need? Or do you really want to sample differently for each column and only unique values? That seems dubious from a statistical pov as rows usually correspond to cases.
For the benefit of all, please edit your post to define any [initialisms](http://en.wikipedia.org/wiki/Acronym_and_initialism) upon first use.
This depends very much on a lot of personal factors, making it hard to give good advice. We don't know what part of the world your programs are from, how focused your interests already are or what they are. The question is stated too broadly to be answered authoritatively, but would be at risk of being closed as *too localized* if it was geared solely toward giving advice to just one person. I suggest providing some more context, but not making it specific only to your particular case.
Another option is to use bootstrapping to get a confidence interval, then you don't have to make assumptions or rely on Normal theory.
There is also [Statisticians Without Borders](http://community.amstat.org/StatisticsWithoutBorders/Home/).
Do you repeat each trial only once?
@kwak: Responding to "It's something that should be outsourced to the community": I look forward to seeing your contributions as you work through these examples yourself.
Also the paper "Information Theoretic Inequalities" by Dembo Cover and Thomas reveals a lot of deep aspects
@Glen_b good point. I've updated the post to try to address your questions. I care about changes to the mean and variance (I tried to quantify that in my post) but I'd be interested in hearing other interpretations of "significance" given the problem description. I'm not sure what you mean by "dependence over time".
You can obviously never _prove_ that you were unlucky and obtained a "weird" sample. Instead, it seems to me that you are worried about the fact that you included and tested multiple variables. Is that right? Whether or not you need some form of adjustment for that is a contested issue. You may want to look at the questions tagged "multiple comparisons".
Hey gung! I'm sorry about that! My two questions are: Are there any advancements of the idea explained paper from Agrawal ("Mining quantitative association rules in large relational tables") and the second one: Do exist any software for mining quantitative association rules?! I found a few programs about data mining, but they only can mine boolean association rules.. Thanx for your help, guys!!
I made sure to estimate robust standard errors. Also, in the book "Mixed effects models and extensions in ecology with R" by Zuur, et al, 2009, on page 261, they mention, "if the mean of the response variable is relatively large, ignoring the truncation problem, then applying a Poisson or negative binomial (NB) generalised linear model (GLM), is unlikely to cause a problem." Fortunately, the means of my response variables are large, so I feel a little more comfortable deprioritizing zero-truncation compared to the GEE and negbinomial aspects of my regressions.
Interesting context, but this does not change the mathematical issue. Your additional piece of information contributes to the definition of a joint distribution on (A,B), but (again) this is not enough for defining by itself a joint. This is not a Bayesian updating issue as far as I can judge. If you do need a joint, you can pick it within a copula and imposing the 95% $A>B$ constraint.
The idea of stratified blocked randomization is sound, but having too many strata defeats the point of stratified randomization. Assuming you have some idea which companies have high/low click-through rates, you could group together companies with similar rates, and have those groups form 5-10 strata.
$E(\\bar{x}) = \\mu$: the mean of the mean is the mean!
Is your question only about why the code is behaving this way? If so, it would be off-topic here, but could be on-topic on Stack Overflow.
A p-value is always computed in relation to a hypothesis. What is the hypothesis that you wish to investigate here? That different genes are overexpressed at different temperatures?
[This paper of Charlie Geyer seems relevant.](http://www.stat.umn.edu/geyer/lecam/simple.pdf)
Not necessarily. You need to test the proportional odds assumption (I am not sure how to do this in SPSS). If it is violated, there are alternatives, the easiest to implement is multinomial logistic.
Computing the Pearson correlation $r_{ij}$ on each $(i,j)$ pair of columns would help you identify associated factors (in the sense of tending to be 0 together or 1 together). In fact $nr_{ij}^2$ is exactly the chi-square for the corresponding $2\\times 2$ table for that pair of $0$-$1$ variables.
@curious: Granger causality seems to lean on temporal order & you still have to hope that it's not the case that "both variables depend on something else" - which I suppose is why it isn't just called "causality".
BTW a nice feature of the Bayes factor, which is symmetric.  Exchanging the hypothesis just gives the reciprocal of the Bayes factor.
(1) Coefficients from this model are the log odds ratios (which is what the model fits) -- you can exponentiate them to get the odds ratios.   (2) To get a hypothesis test for whether a multi-level factor (eg a four level factor with three coefficients) is associated with an outcome (at all -- a test that all of whether all of the beta coefficients might be zero in the population) one can use Type 3 tests (Wald or likelihood ratio tests would be one option for considering the impact of a set of coefficients.)
Can you clarify how you interpreted question 1 / how the simulation code relates to that?