tttjjj/elt_bench huggingface dataset:
Maximum likelihood is not nonlinear least squares, so the "same result" is essentially highly unlikely. `ml` absolutely needs you to specify a likelihood function; it's difficult to know what else you had in mind.
By the way, I don't like the way you ask things: "Why are you doing this when I'm asking that?". Be more polite.
I don't think I'd call that "plain" English :)
@celenius I can suggest this article, [Standardizing Variables in K-means Clustering](http://www.springerlink.com/content/l438152634256311/) (Steinley, 2004) to get an idea of the effects of different kind of transformations. The utility of scaling depends on the data you have, but usually one reason to use standardized (scaled) variables is to avoid obtaining clusters that are dominated by variables having the largest amount of variation.
Good job! _____ :)
To hopefully clarify my intentions... My logic was based on the Gamma distribution being a conjugate prior for the Poisson distribution. In that context, I read that the hyperparameters should be interpreted as "$\\alpha$ total occurrences in $\\beta$ intervals". So if I want my prior to have the "weight"/"effect" of half an observation on my posterior, I know a good value for $\\beta$... 0.5 . And, I want my prior to have 50% probability on either side of an initial hypothesis, in other words specifying the median. Hence, I wanted to find a suitable $\\alpha$ given the median and "this" $\\beta$
http://gis.stackexchange.com/a/12201/751
Whether you would expect the items to have different meaning is not terribly relevant, it can still happen especially with very small sample sizes. Previous questions on this: http://stats.stackexchange.com/questions/51473/matrix-is-not-positive-definite-even-when-highly-correlated-variables-are-re and http://stats.stackexchange.com/questions/8328/how-to-make-a-matrix-positive-definite http://stats.stackexchange.com/questions/52976/is-a-sample-covariance-matrix-always-symmetric-and-positive-definite
Mann-Whitney U test does not assume a symmetrical distribution. If you want to interpret it as a test of medians, then you have to assume that the two distributions have the same shape. Otherwise it is a test of $P(X < Y)=0.5$, where $X$ and $Y$ are random observations from the two distributions. It seems a reasonable test in this case, but I know nothing about your hypothesis.
Exactly what "distribution of the distances" do you mean? If it's the distribution to the nearest center, assuming a uniform distribution of possible points within some region, it's almost surely not a rescaled beta (or anything else with a simple description), so I'm wondering whether you meant some other set of distances.
Use Rpy to call R from Python ;-)
Are these curves, or are you showing estimated density functions from data?
Whuber, he's attempting to equate QoS of landline and cell phone calls.  Likely this is so that people making calls perceive the mobile call to be of equal quality.  It may not be possible or be prohibitively costly to get veridical overlap.  I'm guessing that there are issues in the perception of signal quality that would be critical for cost-benefit analysis and the differential impacts of things like mean QoS and variance.
Thanks, i tried to find it by the CDF of normal distribution but is very hard, here is s.m.th more about this [link](http://math.stackexchange.com/questions/145087/how-to-calculate-the-integral-in-normal-distribution). Thanks a lot for the help you saved my day :)
(+1) Nice answer, but it is a pity that you don't mention any methods of robust regression. E.g. how was the green line plotted on the right subplot (and why do you prefer that algorithm over the others)? Maybe this link could be useful here: [Fast linear regression robust to outliers](http://stats.stackexchange.com/questions/46229/fast-linear-regression-robust-to-outliers) -- arguably the best thread on CV discussing robust regression.
From the help page for `fisher.test()`: "Note that the conditional Maximum Likelihood Estimate (MLE) rather than the unconditional MLE (the sample odds ratio) is used."
See edits to my answer.
(1) Yes, imagining they're sampled from a population from which future events will be drawn. (2) Forget about cut-offs. The area under the receiver operating characteristic curve is useful for assessing pure discrimination; for overall performance use a metric that takes the magnitude of the difference between predictions & outcomes into account - say Brier scores (quadratic) or Nagelkerke's $R^2$ (logarithmic).
Sorry for the late response. Can you clarify what you mean by "Z is blocking information between X and Y"?
Pylearn2 use theano library which enables execution on GPUs with CUDA support.
@user152509 I have collected data from survey questionnaires which ask the respondents their annual income bracket and their expenditure in various categories such as food, clothing etc. Now I want to obtain a plot with various income categories on x-axis and percentage of annual income spent on a particular category(such as food) on y-axis. My plots show that people of lower income bracket generally tend to spend a greater percentage of their income on basic necessities such as food. Now, I want to obtain curves for this data. Can you suggest which type of regression is used in such cases?
If you do a regular meta-analysis on those 2 $d$ values, then the $Q$ test for heterogeneity will just be the square of the $z$ statistic above. So, those two approaches are exactly equivalent.
@chl, I updated the above to answer your question.
@user3386 - this is not a very good idea if you want to know what's going on with your data.  It is for the data to tell you whether it is or isn't "normal".  The only time this would be a good idea is when it is useless - because the normal approximation already holds.  It's like you're "telling nature how its gona be".
Could you explain how this has a bearing on the question?  The "serial pattern" you describe appears to concern the *mean* and that situation is already covered in the question.  The focus of your answer should be on the *covariance,* which in fact can be stationary for a seasonal process.
@Matthias, yes it is. I say a prayer to Hadley Wickham everytime I use it :)
@MattAlbrecht Y is the dependent (your y with values .0192,-.0748 ...) ;X1 is your x with values 315.97 ; X2 is a dummy variable 0,0,0,0,0,0,0,0,0,1,1,1,...1 . The X1 Variable has bot a contemporaneous and lag effect with coefficients [.192 and -.177 respectively ]. The final complete equation is
I agree, though that to me has more to do with the interpretation of results. We are essentially looking for subgroups in which there are heterogeneous rates of reporting and might conclude post-hoc that "groups X and Y demonstrated different satisfaction rates" implying that they either get different service or have different attitudes about the same service. What I'm asking is how does one make a non-data-driven comparison of rates in the presence of big data.
Uncorrelatedness of PCs isn't an assumption; it is the goal, and is the way how eigen or svd decomposition work.
many thanks for the answer and comments!
Normalizing the rows faster and cleaner: `z <- x/rowSums(x)`
Great :)
Worked, thank you very much!
Given what you write in your question, $exp(-\\lambda)$ is the probability of having zero survival, and zero survival is mapped to $y=1$ (as you write it), not $y=0$. cc @jbowman
What's the "evidence that there is no interaction between them"?
As @Greg Snow points out (+1), there aren't enough data here to do anything definitive. You need a different strategy. There's a huge one out there: focus on the burden of proof. If the incinerator operator has to show that they *meet* the legal limit, then they will have to collect enough data to demonstrate they are highly likely to continue meeting the limit. This means that an *upper* bound of their values must lie beneath that limit. If you win this argument, you give the operator a strong incentive to collect lots of data, which will help both sides understand what the risks really are.
@Glen_b Thank you; that's correct.  The `abs` is a vestige of an earlier (messier) incarnation of this formula (when it had been applied to `qnorm(x)`, which could be negative).  In my haste I did not remove it.
I now understand you also assume causality. Thanks.
+1 thanks for pointing out how to do this with the uncertainties package.
"I don't lack data, but the phenomena I am doing a study about is rare", I am doing statistics now about the data I have; 98% of the data is zero. Is it accurate to use the factor this way.
Oh wow, I read that question actually and didn't connect the dots (Have been thinking about this problem for a few weeks now).

Thank you very much Tom!
In a two sample situation, increasing the sample size of one group to infinity does not send the power of the test to 1. The power will be limited by the sample size of the smaller group (or, to be precise, a combination of the variances within the groups and the sample sizes, if you think about a $t$-test). Dependent data can lead to all sorts of problems; that's why the time series people came up with the concepts of mixing as a way to approximate independence for observations that are "far enough" from one another.
I definitely did some factoring for this. So (1|Time) is going to fit a random intercept for time, yes? Could you expand on that a little bit. My thought was that I would want to keep the time component fixed, as a way of sort of holding economic conditions as close as possible. I had not even considered random effects for the time related variables.
"Where is a positive or negative shock doesn't matter"
-> oh, it does ;). I am considering an empirical analysis of monetary shocks. Therefore I need to construct a positiv interest rate shock and a negativ shock in the money aggregate M3.
"ceptrum" = ["cepstrum"](http://en.wikipedia.org/wiki/Cepstrum)?
And this difference might be because of the BIC equation itself. BIC consists of a likelihood term and a penalty term. The penalty term depends on the amount of free parameters in the trained model for different states. And MDL does not depend on free parameters. The equation for MDL is as follows:              Log L(current state) - Log L(prev state) > (N+ d^2/4 +3d/4)        where L =likelihood; N= number of states and d=number of columns in the training data set (which equals to N).
@mpiktas: Each data point represents an object which has 5 different phases. The behavior of each phase of the object can theoretically be represented by a continuous probability distribution function, but my data only contains discrete samples. The Weibull distribution is probably the "theoretical" function behind my data, but the data itself is only measurements of density over discrete intervals.
Consider this: if $X=0$ with probability one, and $Y_n\\sim \\mathrm{Bernoulli}(1/n)$, with $X$ independent from the $Y_n$'s, then $Y_n$ converges in probability to $X$.
@ttnphns: Why do you think that the variance of noise components should decrease linearly? In fact, [I analyzed it once](http://stats.stackexchange.com/questions/87032/eigenvalues-of-correlation-matrices-exhibit-exponential-decay/87146#87146) and it's more of a power-law decay in the middle (and quite messy overall).
If you are satisfied with this answer you might want to consider to accept it. This you can do by ticking the check-mark underneath the upvote/downvote button. Thanks!
How do you define 'publication-quality'? Please elaborate on what aspects you like to see covered... e.g. color use, line widths, etc. Should answers focus on font size, instead?
Here's an updated article with blogs that are active and worth reading in **2012**: http://flowingdata.com/2012/04/27/data-and-visualization-blogs-worth-following/
What is 'best' depends on what you want to optimize. You may want to focus on the ones that relate to prediction error in this instance.
Correct, I believe. See http://stats.stackexchange.com/questions/40876/difference-between-link-function-and-canonical-link-function-for-glm
It is a standard result of M-estimation theory. The basic idea comes from taylor approximation of the minimizing function. You can look at the derivation in [Wooldridge's book](http://books.google.com/books?id=cdBPOJUP4VsC&lpg=PP1&dq=cross%20sectional%20and%20panel%20data%20wooldridge&hl=fr&pg=PA350#v=onepage&q&f=false), or check out van der Vaart's [Asymptotic Statistics](http://books.google.com/books?id=UEuQEM5RjWgC&lpg=PR1&dq=van%20der%20vaart%20asymptotic%20statistics&hl=fr&pg=PA51#v=onepage&q&f=false).
Great and thanks for the response!
I was assuming that the year effects were not of particular interest. If you are trying to forecast into the future, year fixed effects will not be helpful in that endeavor. If you just want to see the trend over the history of the data, just plot the coefficients from these variables. The trend won't be monotonic; there will likely be ups and downs. One year fixed effect is excluded so that your model is identified; every other fixed effect is measured relative to this year.
(1) For a test, you need some way to identify the spread; sd, IQR or range might all do; more importantly -- why do you want to use a hypothesis test to compare ages? (2) If the average value for a potentially important variable is very similar in both groups, that greatly reduces concern about Simpson's paradox-like effects (the remaining concern there relates to lack of power if (at the patient level) it's not included in the analysis, as ignoring it will inflate the variance, possibly dramatically). ... (ctd)
Speaking of social networks visualization. I created Facebook visualization: http://www.yasiv.com/facebook
@curious_cat Based on Glen_b's answer, you may interpret the chance that the two letters are the same as the squared distance to the origin of the "probability vector" $(p_1, p_2, \\ldots, p_n)$. The restriction that $p_1+p_2+\\cdots+p_n=1$ confines this vector to a plane, yielding a geometric interpretation of your question: what point on this plane is nearest the origin? At any such point, the vector to the origin must be perpendicular to the plane. Obviously such a point is unique and just as obviously $(1,1,\\ldots,1)$ is perpendicular: thus all the $p_i$ must equal one another, *QED*.
Thank you! I will look more closely into the assess ph / resample command.
I get some NaN values in the s.e. of some of the coefficients using the auto.arima with higher number of parameters than the default, does this means there is something wrong with the fitted model?
I'll work up some more material here when I have a chance, but no, the Kruskal-Wallis test probably provides your best protection against type I errors. OTOH, it may not be the most powerful option you have.
@Anthony Damico-NHIS.  Their documentation shows how to handle domain analysis for subgroups, but not imputations.  I'm not an R user, but it looks like the R code you reference is for aggregating imputed data sets, not domain analysis.  So, it's still unclear.
@vinux: when $(X_1,X_2)$ is in the set $\\{(X_1,X_2);X_1+X_2>x\\}$, it is either in the union of the two first set or _not_. If _not_, I proved it is in the third set. QED!
Your question has no meaningful sample variances: it concerns only the two numbers $X_1$ and $X_2$. If you assume the $\\sigma_i$ are *known*, then the MLE of $\\mu$ is easily derived and equals $\\left(\\sigma_2^2X_1+\\sigma_1^2X_2\\right)/\\left(\\sigma_1^2+\\sigma_2^2\\right).$
+1 "Complete the square" is the key idea.  I suspect that phrase would correctly answer several dozen related questions that have appeared on our site :-).
After the 65 radio-tracking sessions I had a table consisting of around 1800 data points. Then I aggregated the table so that one row represents one tracking session. Since that I'm looking for possibilities to do my analyzes (it's my masterthesis so I have to find a possibility to do anlyze the data...)
Thank you for ur response! Much appreciated
Check out the outlier function in the randomForest package http://lib.stat.cmu.edu/R/CRAN/web/packages/randomForest/randomForest.pdf
@Glen_b You're right. I added some text in the beginning of my answer there, and a reminder when the density is derived.
I updated the answer @NickStauner , interventions 1,2 are similar interventions to randomly assigned students.
(+1) Thanks, Matt, that sounds like great advices. I especially like the last two paragraphs.
Binomial, so we're using logistic.
Good recommendations here but the first two are not concise books.
@ steffen Sorry, I did try that and could not get it to work nor find help about it and now I have to get back to _real_ work
Are these observations independent? Do you know the mean and variance of each individual observation?
I believe the upper bound is 16. What is the triangular distribution used for in the world ? im just trying to find how it is applied in the real world, so I make more sense of what value to use in my question.
"if you randomly chose the cities to begin with, it is highly unlikely that all those cities became more likely to click (or, alternatively, all the control cities became less likely to click" -- I don't see then number of cities listed; if small, then the treatment and control cities could differ a priori.
Are you confusing Holt with Holt-Winters? Holt-Winters adds a seasonality equation which reflects a time offset of $s$ (the seasonality), which in your case would be 365. At least that's how I understand Holt-Winters.
How would it be possible for a truncated distribution to be conditionally normal at every $X$?
I think this is the exact answer I was looking for. I will be reading through it a few more times on Monday.
@whuber What I have in mind is to compare a strait line with a curve (in this case a poly-2) and was thinking that creating a confidence intervals on 'c' by bootstrapping the data would be sufficient. But honestly not sure of that, any ideas are welcome.
The difference is the distinction between ordinary correlation and partial correlation.
Hypothesis tests don't show to distributions are the same. Presumably you mean to say something else. Are you talking about failing to reject some hypothesis there? Can you show QQ plots for the two sets of data?
What do you mean by "p-values"? If you mean the usual thing (as in p < 0.05) then items do not have p values, statistical procedures do. If you mean something else, please clarify.
It's marked self-study. Forget the experiment altogether, I'm talking about the statistical setup of subjecting various samples to two different treatments as shown above.
Some questions that come to mind are: (1) are the 201 patients unique in some way or is the missing data essentially random, (2) is the missing variable correlated with the other symptom variables, (3) how many symptom variables do you have, and (4) what size correlation do you expect or hope to find?
I don't see what is 'uncongenial' about considering a distribution as the target of inference. Indeed, indexing things with parameters is anyway optional e.g. one may put assert one's uncertainty using distributions over functions directly (see Neal and Williams on Gaussian Processes). 

And one need not have any particular view of 'randomness' to represent uncertainty with the probability calculus.  Inference based on sampling (theory) arguably *does* need such a theory, but as far as I can see Bayesianism isn't (or at least does not need to be.)
I think this is discussed elsewhere, but I can't find it right now. I may add a link or more information later.
I've got a pretty solution, see [here](http://math.stackexchange.com/questions/923474/determine-sample-size-according-to-some-unknown-distribution-with-given-error-ra).
You jump into your question with no context or explanation of the terminology. This will make it unintelligible except to anyone who might have chanced across your reference (or those who like to make educated guesses about what the notation might mean).  Please edit your question to make it generally understandable.  Thank you!
Are the time series both stationnary ? www.econ.ohio-state.edu/dejong/note1.pdf
I found this in your post This, again, is a computation. It relies on the bilinearity and symmetry of covariance, whose application is responsible for the summations in the second line: Actually, I didn't get how the summation part came  when we calculated the expectation of the square of the difference between the predicted and actual value of Z0
Welcome to the site, @user39491. I do not completely follow this. Could you clarify & elaborate your points?
@onestop, yes I could do that
Thanks! let me some time for thinking about it.
Why not use multilevel analysis? If you aggregate you lose the variation in the data that you aggregated.
@Abe It seems now you have reversed "means" and "variances" but there's still a contradiction!  (Unless perhaps you want to compare the variances of the means.)  What do you mean by "variable": the means or the underlying values on which they are based?
+1, btw, you've got a typo; you hit "C" when you meant to hit "B".
I'm re-posting this as an answer here as whilst I wasn't 100% sure, Ben Bolker (one of the lme4 developers) comment on the Q on SO that my Answer was correct/explained the lack of `Residual` component in the output from `lmer()` that the OP was discussing. Not sure why the Answer turned into a comment, but I do believe this answers the software implementation bit of the Q.
Thank you! You made it seem easy...
$x<y$ can't happen since $y$ is dependent upon $x$.  Right?  We are choosing $y$ "successes" out of $x$ trials.  So the relationship should really be $y\\leq x$
What @gung might be getting at in the reference to "strategy" is that in most applications a formal test of normality of residuals is both overkill and of little use, because the residuals usually do *not* have to pass a test of normality (and likely won't), but it is desirable that they exhibit a set of qualitative behaviors related to (approximate) lack of heteroscedasticity and lack of serial correlation. Usually, examination of residuals is a *diagnostic* and *exploratory* exercise rather than the confirmatory exercise suggested by this recommendation to apply a test.
Because $Y=1/Y$, $T=X(Y-1/Y)=X0=0$. Thus, in order to answer these two questions, you only need to find the distribution of $S=2XY$. What does this distribution look like when $Y=1$?  When $Y=-1$?  That information will reveal the distribution of $S$.
I would recommend Loehlin's *Latent Variable Models: An Introduction to Factor, Path, and Structural Equation Analysis* (2003, 4th ed., Lawrence Erlbaum Associates). That's a very good book with lot of illustrations and references.
@Srikant: Agreed.  The comment was for the benefit of the OP, who is a mathematician (but perhaps not a statistician) to avoid being misled into thinking there is something fundamental about the distinction.
@Alexis : Thanks. Should I project that on prediction earthquack ?
I think I am using the term 'synchrony' in a different sense than the Kuramoto model assumes. I don't think human bodies aren't adequately modeled as periodic oscillators. A closer model might be [Per Bak's sandpile](http://en.wikipedia.org/wiki/Bak-Tang-Wiesenfeld_sandpile) or other dynamic cascade. But in any case, I'm not trying to create a model that corresponds to my data. I'm adding an analytic layer to reveal patterns in it.
1 - signif codes should be clear ... lag has double star so it is p=0.01  2- those test different H0s: "none" or "drift" or "trend" 3- critical values, again, same thing, depends on which H0 you wish to consider
So,you think that these symbolized versions can be used in machine learning like pattern recognition and classification of features that have been represented by these symbols,in applications mentioned in the papers linked?However,if each symbol refers to a codebook,then classification technique would imply that we are classifiying the original unsymbolized data.But after classification,will it be possible to have a meaningful interpretation ?What do you suggest?