tttjjj/elt_bench huggingface dataset:
Leo, it is not the case that every optimal solution is semi-symmetric. Through a random search I was able to find many asymmetric optimal 7-means solutions for the $27$ points located at vertices, edge midpoints, face centers, and center of a cube centered at the origin in $\\mathbb{R}^3$, for instance. That is why my "more interesting" conjecture asks only about the *existence* of a semi-symmetric solution.
@whuber: good point, I was using $<$ in a vague way. Does $H_0 = \\exists u: C_x(u) < C_y(u)$ for K-S? That seems like a much weaker hypothesis than what I would've expected.
Are these 100 contiguous base pairs sampled from binding sites, 3' /5' UTR, transcription sites? How are they identified? When you mention "set" do you have replications of 100 BPs sampled from other parts of the gene, or chromosome, or elsewhere in the genome? Where do multiple genes factor into this? What are the controls in this analysis?
Always create generated random data set, instead of providing the original data set. Something along the lines `set.seed(100)`, `x<-rnorm(100)`, `dfUK<-data.frame(x=x,y=1+2*x+rnorm(100))`. Much easier to replicate than to copy the original data to some file, clean it up, and then load into R.
I strongly suggest getting a statistical package: R or SAS or Stata or whichever you like. That will make all this easier.
Thanks for your answer. I do not get everything yet... mainly because I do not manage to make the link between my definitions and yours. Moreover, I thought that minimising the volume of the ellipsoid was equivalent to minimising the determiant...
Forgive my ignorance - i do not know what the ODA and Governance variables mean. However, if your interaction term is always positive, then by all means perform a log-transformation on it. If it isn't it is certainly possible to force it to be positive by adding some constant to the interaction term before logging it. Basically, you are allowed to apply whatever transformation you like to whatever variables, as long as it's mathematically valid. The question pretty much always boils down to whether the model is reasonable and gives a reasonable fit.
Alright, so I lost myself among the difference of test statistics and P-values. Thanks for the answer, a real life(mind?)-saver!
@subha You're right, you don't need any label information to perform CD and learn features with a RBM.
@Fabrizio Yes. There's maybe some confusion about what a statistic is. Wiki: "The term statistic is used both for the function and for the value of the function on a given sample." So maybe it's also correct to call the sample (as the result of applying the identity to it) a statistic.
You may want to take a look at a somewhat similar question: http://stats.stackexchange.com/questions/242/using-time-series-analysis-to-analyze-predict-violent-behavior
Add some sample code to this question.  Specify what packages you are using.  There are multiple packages that allow GAM modelling in R.
If you are fitting a t distribution the number of degrees of freedom is a parameter being estimated; it is not given independently.
@alto Indeed I am talking about initialization. I'm using RBM training to extract features/initial weights prior to back prop training of a feed forward NN. I'm not using RBM to initialize the weights attached to the bias unit because a bias unit does not contain information in the same way the data inputs do, hence the stacking of a random weight vector for the weights that attach to the bias unit. However, after your comment I shall adjust my code to include the bias weights in the RBM training - it is a simple enough tweak.
Welcome to the site, @Jasper Dixon. I can't tell whether your question is about *coding*, or about understanding ARIMA. If the former, your Q would be off-topic for CV (see our [help page](http://stats.stackexchange.com/help)), but on-topic on [Stack Overflow](http://stackoverflow.com/). If your Q is the latter, please edit to clarify; if not, flag your Q & we'll migrate it for you (*please don't cross-post, though*).
i wish i could upvote this one more....
This is interesting commentary on the question. I am disturbed by the apparent suggestion that (a) the departure of real data from theoretical ideals implies (b) theoretical considerations are "slightly irrelevant." Exactly the opposite is true: theory is our touchstone and basis for understanding reality and so understanding its implications becomes especially relevant when reality departs from theoretical assumptions. Moreover, Glen_b's example is such a profound departure from multivariate normality that in most cases it surely would be an unacceptable deviation from multivariate normality.
Do you know for sure that the outliers are taken from another Gaussian distribution? Do you have any hypothesis on the variance of this distribution, eg is it equal to the variance of the “background” distribution? Fitting a Gaussian mixture would be a solution.
Can you give more indication on the context in which such measure might be used? Macro's comment suggests an application in test development/validation.
I need the sas code for computed [FA0(1)≥0] and [FA0(1)≤0].
+1 I think this is a reasonable question, amenable to objective analysis, regardless of the truth of the preliminary claims it quotes. Indeed, there is a long history of astronomers being early adopters of statistical methods. Stephen Stigler has devoted considerable effort to addressing the question of why some fields (notably economics and psychology) historically took much longer to accept that statistical methods even had any relevance or application.
You might find the examples shown in the closely related question at http://stats.stackexchange.com/questions/90325/how-to-decide-on-the-mle-when-pmf-is-0 to be helpful.
See also http://stats.stackexchange.com/q/113334/3277 - a very similar question with interesting answers and comments.
It is. But I guess fg nu just wrote it down for explanatory reasons. Or did u refer to my previous post?
I think you are equating wrongly the $\\alpha$'s in your question. It doesn't make sense that $P(Z\\leq z)=\\alpha/2$ if you define $P(-z\\leq Z\\leq z)=\\alpha$. I suggested some thoughts in order to solve this in my answer.
Have you checked the assumptions for an independent t-test? Does GDP per capita have a normal distribution?
Your p-value is not 0.9615: it is 0.07698.  This is because your z-statistic is -1.7685 and a standard normal variate has a 0.07698 chance of exceeding 1.7685 in absolute value.  Note that this p-value is high enough that in many applications it would not be considered significant evidence of a difference anyway.  The additional uncertainty in your situation should *raise,* not lower, this p-value.
I note you've written, in the expression for $E[\\tilde{Q(t)}]$, $...-110\\mu$, which would appear to indicate that you thought $E[N(t)] = \\mu$.  By coincidence, $E[N(t)] = \\lambda t = 100 = \\mu$, but only by coincidence; so you arrived at the right number but by the wrong formula.  Given this, you might want to check the next line to see if you made a similar mistake.
Yes, if you are interested in a population which is 50/50 split between men and women, one way to estimate the covariance in that population would be to gender-balance your sample.  You could do it by randomly throwing away observations from the over-represented gender or you could do it by weighting.  Or you could do it by using all of your data to estimate the various items in the formulas I give above and then plug in 0.5 for probability male and probability female.
The question makes it sound like the only attribute of the georeferenced coordinates is the user, but that would be highly unusual: isn't each coordinate pair also identified by the object it purports to locate?  If so, what need is there for an independent clustering algorithm?
A (principled/pedantic) frequentist doesn't refuse to answer, he can't answer. The parameter isn't random, the interval is. What's in front of the $|$ when you condition on observing a particular interval? For a frequentist, once you've observed the sample and constructed your interval the interval, nothing is random any more.
There's a semi-related discussion of this as it applies to trend analysis here: https://tamino.wordpress.com/2012/03/29/to-robust-or-not-to-robust-that-is-the-question/
+1, it couldn't reasonably be called a measure of *association*, but I do think it can still be thought of as a measure of the relevant type of effect size for such analyses. Test statistics & p-values conflate magnitude & N to give some sense of the clarity of the result; this formula seems to effectively disentangle the magnitude from N as other measures of effect size do (eg, $d$ & $r$). I think you can also use it to measure the deviation from non-uniform null distributions & it would be doing the same thing, it's just the word "concentration" would no longer characterize it well.
Showing a list of coefficients like this, with no context at all, is likely saying "Joe has 31, isn't that a lot?" without saying 31 *what*. 31 cars? A lot. 31 kids? One heck of a lot! 31 dollars? Not much.
Could you tell me why dichotomising the dependent variable could cause issues?
Even in the regression setting, the gist of my statement still stands. If $\\mathbb E( X_{i+1} \\mid X_i ) = c - c X_i$, then the least-squares estimate of $c$ is $$ \\hat c = \\frac{\\sum_i x_{i+1}(1-x_i)}{\\sum_i (1-x_i)^2} \\> .$$
Do you have analytic expression of your PDE solution?
(Migration-related comments deleted)
Oh... this is Matlab? Wow... it looks so much like R :) In any case, what I wanted to know is what does it mean to correlated the log values? Don't they represent different distributions? I am just trying to understand how to interpret the result. Do you have any suggestions?
@FarhanAhmed, the link was for Mavadu.
+1 Nice answer.
Sorry, "cannot be replicated" was kind of key. I've clarified the intention, and changed the title slightly to suit. I hope that makes more sense.

I was talking more about physical models than curve fitting.
It's impossible to get a significant MW result with Ns of 3 and 4.
The number of times you repeat the whole cross-validation process is also important.  My machine learning colleagues feel that 10-fold cross-validation should be repeated 100 times to achieve good precision.  A single 10-fold cross-validation yields an accuracy measure that has a high variance.  The choice of accuracy measures is also important.  You are using an improper scoring rule that is also discontinuous, so expect more problems.
Scroll down to the 'chi-square test' section [here](http://www.ats.ucla.edu/stat/sas/whatstat/whatstat.htm) (not the 'chi-square goodness of fit' section).
Have you looked at their own software package, [$\\ell_1$-magic](http://users.ece.gatech.edu/~justin/l1magic)? It may not be the most efficient implementation or algorithm, though. Note also that the fourier coefficients and the original function are related by a linear transformation, which can be approximated discretely by a DFT, making [linear programming](http://stats.stackexchange.com/a/12564/2970) packages a viable solution.
For numerical stability it's usually better to center continuous predictors, if that is what you are asking. Not sure what that has to do with model saturation though.
(ctd) ... it does make a difference, and it doesn't affect means, medians and modes equally. We're now talking more about robustness to some contamination/mixture but where we still want to have our inference relate to the uncontaminated part (the major component of the mixture); means are quite sensitive to extreme outliers, while medians (and modes, in many cases) are typically relatively insensitive to them, at least when they don't occur in large proportions.
I find it most intuitive to think of R^2 as "variance in the outcome accounted for by the model". Your additional predictor explains some variance in the outcome, even if it's not significant. That is, by the way, an intuitive reason why it's a bad idea to build models solely based on whether predictors test as significant.
@nibot: and theoretical physicists are obsessed with integrals and use them daily. That doesn't mean questions about integration are on-topic for this site. Same with your question; just because it is about something that physicists use doesn't mean it is about physics. P.S. Moderator close votes are binding. There are not that many people here with enough reputation to cast close votes, so it usually falls to the moderators.
I'm confused by the title. Which values do you think are 'too good'?
Have you checked out our [list of SAS resources](http://meta.stats.stackexchange.com/a/816)?
@JackTanner the "GS" in 'BUGS' and 'JAGS' stands for *Gibbs Sampling*. If by 'Metropolis step' you mean the part in Metropolis-Hastings where the proposed move may or may not be accepted, Gibbs Sampling has no Metropolis step since its $\\alpha$ ratio is (by design) always exactly 1.
Hi Tootone. Your solution was correct but I found another algorithm workin' better. The key principle was that node N is direct child of node A if A has no child that is parent of N. Thanks anyway
Perhaps you can clarify your question a bit.  (a) Do you want to compare the means of the numeric values or the won/lose entries?  (b) The numeric values shown range from .8 to .95, not 0-1.  If the data are not normal (or close to normal) you might consider a non-parametric test.  Have you looked at the distributions of scores? (c) Does each line of data reflect a pair of scores? If so, your choice of test might best reflect that relationship.  (d) How many lines of data do you have?  Non-parametric tests tend to be less powerful than parametric tests, so number of cases is important to know.
The KS doesn't apply, but you could use the same statistics in a permutation or randomization test.
It's not constant if $p(C)$ is a probability distribution other than the uniform. Frequentist inference *assumes* that it is constant and unknowable, but Bayesian inference does not.
+1 for pointing out that the covariate and predictor need to be unrelated.  I think though, that it's important to clarify that if it's a quasi experiment, as implied by current wording, there will typically be some relationship between the covariate and predictor and, even if the interaction is not significant, ANCOVA should be avoided.
One very short addition to @whubers excellent answer: You could use the `ncvTest` function of the [car package](http://cran.r-project.org/web/packages/car/index.html) for `R` to conduct a formal test for heteroscedasticity. In whuber's example, the command `ncvTest(fit)` yields a $p$-value that is almost zero and provides strong evidence against constant error variance (which was expected, of course).
$\\Sigma$ is an $m$ by $n$ diagonal matrix, with possibly nonzero singular values in $\\Sigma_{1,1}$ through $\\sigma_{m,m}$, but the singular values for $m+1$, $\\ldots$, $n$ are all considered to be 0 (there's no place to put these zeros in the $m$ by $n$ matrix $\\Sigma$.)
Sorry, I was adopting the notation from a paper who's methodology I'm trying to follow. I assumed it was Gibbs because when generating a sample path of $Y$ I take a new draw for $Y_t$ holding all other periods of the chain constant. Likewise, I thought it was RWMH because of the way I draw new candidates of $Y_t$ from previous chains. If this is the incorrect terminology, I apologize, but I am still trying to figure out how to implement the steps.
Whatever book you read that in, throw it away.
@user10405 Those were polite and justified comments; there is no need to react emotionally.
OK, just to be clear, because this is an important detail: is there a natural hierarchy to the categories and subcategories?  Could one assume that the ordering is A.High A.Medium A.Low B.High ... E.Low (or something comparable)? (If so, is there anything at all to prevent us from thinking of these 15 classifications as forming a single *ordinal* variable?) Also, it would help to clarify what the numerical values are intended to mean. *E.g.*, should they be interpreted as fuzzy degrees of membership, as probability of membership, as combined results of multiple measurements, or something else?
how is that so?
You'd need to account for other things that affect overall mortality rate, like differences in age distribution. You'd also need to account for differences in variance between these observations, which are due to more than just variation in sample size.
This is a good point. I have found in the past that it is very difficult to tell whether a given candidate will work out, unless you have worked with them in the past.
Thanks VERY much for all your input on this... given me lots to think about
@user20650 can you tell me more about it ?
This definitely would be better suited to quant.SE. I would also suggest breaking apart the question and looking at some of the individual components separately. For instance, I recall that they had recently had a discussion about the kind of constraint where it was not below x% otherwise 0%. Anyway, most of the individual stuff you're talking about (garch/copula/optimization) have R toolboxes, just google search for them. However, you'll have to do some programming to bring everythign together.
If you set e.g. $x_2$ to 0 then $b_1$ is only the contribution of $x_1$ when $x_2 = 0$. Is that what you want?
Use the trigonometric method for this one and then perform a little algebraic simplification.  About the shortest way to express the solution is $u(z) = 2 \\sin \\left(\\frac{1}{3} \\arcsin(2 z-1)\\right)$.
Non-parametric bootstrap is the easiest way, and probably sufficient. If you want to be a hipster, however, you can use the model estimates for prediction. I don't know what DirichletReg does, but I suspect that the estimates of parameters are based on a normal approximation - so sample from those normal distributions 1,000 time, and calculate the predicted proportions for each realization.
I'm not convinced "demand in jobs for dedicated statisticians and data analysts" has strong correlation with the usage of the keywords "data scientist" or "big data" in job ads.  <-- being suspicious of assumptions is what the human brings to the table ;-)
Is it possible you could add the key points here, or give a summary? The link may go dead at some point, & it will also help readers know if they want to pursue the link further without having to click on it.
For #2, it will be enlightening to state explicitly what $H_0$ asserts and then sit back and re-read the question.
@Carolynn I would just copy the table from the pdf as an image and then do this http://meta.stackexchange.com/questions/75491/how-to-upload-an-image-to-a-post The reason why that is better is that I wasn't able to edit your post in a way it made sense to me.
Sorry for not being clear.
Ordinal regression = less convenient interpretation,
Lumping satisfaction values together = lost information
@user11852 Your general argument that outliers are legitimate is helpful. But, concerning the possibility of amplification, it seems to me that the contrary is true: bootstrapping has a chance of working only if the full sample is used.  Otherwise it presents a fairy tale, telling us how things would be if outliers didn't exist--but obviously they do. The larger problem is that bootstrapping has little theoretical justification when applied to small samples: the theory is an *asymptotic* one.
Thank you for your answer. The issue about the proportionality constants can be avoided by using the same improper prior on common parameters, such as location and scale parameters, as mentioned in [The Bayesian Choice](http://www.amazon.co.uk/Bayesian-Choice-Decision-Theoretic-Computational-Implementation/dp/0387952314) pp. 349. If I properly understand, the marginalization paradox applies only to priors with a certain structure.
By 'realizations of a random variable' I assume you mean the actual observed values. What is being summed in the 'sum of random variables' is the random variables before they are observed. Imagine calculating the weight of the next 5 people to get on the elevator. You dont know their weights (yet) and so they are each a random variable. But you probably would like to know something about the distribution of the sum of their weights.
Wow, that's a surprise. Here I was molesting some Newtony-Raphsony thing and you were over there finding a solution.  It will be used in the future.  Thanks.
There are 3 or so historical cases from the US and the UK presented very clearly in the book "The Drunkard's Walk: How Randomness Rules our Lives", by Leonard Mlodinow. They're mostly related to poor uses of conditional probability. One is the O. J. Simpson case, and another is the Sally Clark case, which is also covered at http://understandinguncertainty.org/node/545
There's a fairly good introduction [here](http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval)
This question appears to be about the physics of die throwing rather than about statistics or machine learning.
It seems to me that the variance of $\\frac{X_i}{\\sum\\limits_{j=1}^n X_j}$ must equal that of $\\frac{X_k}{\\sum\\limits_{j=1}^n X_j}$ for any $k$ and this observation should help you make a good start in answering this question.
Not true. Factor Analysis differs from PCA in the variance-covariance matrix used. That only the rotation would make the difference, is a common misconception.
@Noldorin answer updated
In randomization inference, the concept of bias is whether the expected value of that difference over all possible assignments, weighted with their probabilities, equals the population value. You may be able to create some special assignments (analogous to balanced sampling) such that the expected value is equal to the population value even though some combinations have zero probability of occurring. For instance, with 5 subjects and equally likely assignments of a pair {1,2}, {3,4}, {1,5}, {2,4}, {3,5} to the treatment, you can still get an unbiased estimate of the difference (I believe).
@chl Thank you !
@gerrit, it's actually neuronal data so there's not much literature on cyclostationary series applied to neuroscience. Any tips if I should go with the cyclostationary approach or use directional (circular) statistics?
@KasperChristensen If you specify the contrasts like in the examples, the intercept will be the Grand Mean. Please provide a reproducible example of what you've tried.
I'm a little unsure of your description above. In step two, it sounds like you do a fixed number of (unobserved) transitions where the fixed number is the label of the state that you're in? Is that correct?
(Is this on topic for CV?) Yes, and in fact you will get the same result if you merely type `effect("hp", model)`.  See the `typical` argument in the documentation for `effect`.  You can also double-check that the function is working as intended by plotting prediction lines using `predict`.
Thanks, both answers helped a lot.
Your number 2 sounds to me a bit like what David Freedman's critique of using Cox proportional hazard models in experiments, *Survival Analysis* ([Freedman, 2008](http://dx.doi.org/10.1198/000313008x298439)) in the American Statistician. As far as fit IMO I would just estimate the expected number of 0's and 1's in subgroups of interest and compare that to the observed proportion and see if the predictions are reasonable. Poisson regression will assign some probability to outcomes over 1 - which strikes me as problematic in binary data like this.
Hi Peter, thankyou for your comment. This is actually an assigment I have to do for colledge and my knowledge is a bit limited. What I am trying to do here is to demonstrate how to obtain the likelihood ratio using R, and for that demonstration I got a solved example and now I am trying to obtain the same result using R. I have already changed the mean to 10 (that was a mistake of mine)
I give some explanations: