tttjjj/elt_bench huggingface dataset:
(1) I'm surprised that pvals.fnc still works at all; I thought mcmcsamp (which pvals.fnc relies on) had been non-functional in lme4 for quite a while. What version of lme4 are you using? (2) There is no conceptual reason why having random slopes should break whatever one is doing to get a significance test (3) Combining significance testing with MCMC is a little bit incoherent, statistically, although I understand the urge to do so (getting confidence intervals is more supportable) (4) the only relationship between this Q & the other is 'MCMC' (i.e. none, practically)
Agreed. Anybody who reads my papers is likely to note much emphasis  on the power of names for good or ill; it is good to get back some of my advice.
You might take interest in various clustering criterions which suggest optimal number of clusters. Find them on my web-page.
+1, now that is awesome!
Thank you. Could you tell me what the 'k' parameter does?
As a kind of meta-comment, I would not expect to see inaccessible material cited as authority for any statistical point, not that you are proposing to do that. So, it's simple at one level: there is a literature on compositional data analysis, which is where to look; I am not an expert, so I can't say what's most authoritative on correlation, but my instinct is that the warning is exaggerated. Descriptive use of correlation can be helpful. It is just that inferences are complicated by the constraint on totals.
then you can use `t(mydata)` to transpose the matrix, or do this in excel using `paste -> special -> transpose`
(+1) thank you for this helpful reply. It seems you have used them regularly. I am curious when and why you have preferred to use them (instead of e.g. equal-width).
See [here](http://stats.stackexchange.com/questions/71096/linear-regression-and-prediction-on-transformed-data/71097#71097) and [here](http://stats.stackexchange.com/questions/61600/strange-outcome-when-performing-nonlinear-least-squares-fit-to-a-power-law).
Attraction was rated using two items on a 10-point Likert scale, the results were then summed and averaged. So the attraction rating ranged between 1 to 10. Reported thoughts was similar with participants indicating how much they had thought about the rating group in the last 15 minutes on one item using a 9-point Likert scale. Mood was assessed using the POMS-SF which is scored on a 5-point Likert scale from 0 - 4, the scoring of the POMS is such that participants in a better mood will end up with a negative score and participants in a worse mood will end up with a positive score.
(1) I wanted more, not fewer bins! (2) If you're using the `hist` command I suspect you made an error. I use it often, and the number of bins is trivial to change: try `x=rgamma(999,2.5,.25)` to make some data, and then `hist(x, n=40)` which will give close to 40 bins, as an example. If you're using some other command to make histograms you'll need to look at the help.
I am surprised nobody has responded yet. Hopefully somebody with relevant knowledge can give their opinion.
There is.  sorry i'll edit the formula now
I don't doubt that it can handle it (so could Jags, I believe), but I guess I'm really wondering what it means that the dispersion parameter is a 'random effect,' and whether that is the same kind of random effect that we're used to from Linear Mixed Models. Any ideas?
@user1471980: I believe all you need to do to plot the date is to change `plot(dataframe$LOGINS, type="l")` to `plot(dataframe$DATETIME, dataframe$LOGINS, type="l")`
Thanks! I know what the first two components represent (e.g. one is reading/commenting, the other one is friend-related activity). Is there a usual way to combine a number of PCA components into a single score?
Detailed information about confidence sets for the mean of multivariate Gaussians appears at http://stats.stackexchange.com/questions/9898 and http://mathematica.stackexchange.com/questions/21396.  More related information is available from a search on [confidence ellipse](http://stats.stackexchange.com/search?q=confidence+ellipse). General information about the distribution of quadratic forms of Normally distributed variables can be found with an [appropriate search](http://stats.stackexchange.com/search?q=normal+quadratic).
Checking the paper I see he writes n$^|y|$ where n is the length of the sequence and y is the alphabet and |y| is the number of letters in y.  So yuo have it right and the author made a mistake.
Could you amplify on your guess?  What is it about this circumstance that would cause "problems" and what would be their nature?  (Based on the limited information in the question it seems possible to me that the intended regression could work out perfectly well.)
@Mike Lawrence Thanks for the answer! how is a 3 level model estimated then? where one grouping factor is nested within another?
Thank you, David. I am using MCMC as there is a second set of observations that is linked to the $Y_t$ by a function of $\\sigma^2$. Therefore the sample from the conjugate-Inverse Gamma prior is used as an input for a Metropolis-Hastings step connecting this second set of observations.
You hit the nail on the head! First - lowess. I started thinking moving average (which technically is lowess of 0th degree) before going to bed, but lowess is even better. This allows you to visualize the original distribution (its approximation) and choose the model accordingly. Thanks for the tests too. Second - I've calculated CDF incorrectly (embarassing) - thanks for pointing it out.
Because I wanted $P(X\\geq 55)=1-P(X<55) = 1-P(X\\leq 54)$, where $X$ is the number selecting $A$. (Then of course, I wanted to double that.) ... The first thing follows from the definition of *p-value*, the rest is elementary probability on a variable representing a count.
Factors other than patient to patient infection dynamics can influence disease spread (such as public health programs).  That doesn't stop the underlying model being useful.  I don't think the exact date of Facebook's demise (which no doubt can be influenced) is as interesting as the idea/model that social networks spread like diseases.
Oh, Is it so. Ill see to it. Thank you so much Peter.
@ Aurelie: Yes it's just an example data.frame. My original data consists of ~30'000 rows of very similar structure. Couldn't figure out how to get rid of the NA values. Do the NA's cause problems when plotting?
I don't think this answer adds anything to previous answers. Indeed much here is often false: even panel data are not necessarily under the control of the researcher and (e.g.) in economics researchers pften depend on collation of data by others. Also, time series are often short.
Sigh, apologies. Should run now.
Please add the self-study tag and some thoughts, e.g., do you know the definition of independence?
Thanks for the suggestion, I appreciate your time. I'll go ahead and try it over there.
(+1) I agree with you: Code optimization certainly depends on the task at hand, and I hope the OP will clarify his/her intents to make this question more like a statistical question.
Many many thanks dear Michael, your answer really enligthened me.
Does nobody know the answer to this question?
I don't think this will have a DF that can be written without a ton of integrals floating around. I don't even think an asymptotic distribution can be obtained without some very contrived stipulations about $\\Sigma$, akin to Lindeberg-Feller conditions.
I agree @JeremyMiles, it is just another reason though for double-blind randomized studies. It is merely an intended argument against pseudo-randomization - that it is easier to circumvent the intended treatment than actual randomization. (My example actually isn't an example of pseudo-randomization, but it illustrates the point succinctly.)
why tie is impossible when k=1, what if there are two neighbors belong to different classes with the same distance, how do you determine the nearest neighbor with k=1?
Q2. If you want only to partition (somehow) data, why use clustering at all? You use "partition" also in Q3.4. It is unclear what you mean by partition in your question: 1) any partition, 2) any clustering, or 3) partitioning methods of clustering (such as K-means, but not at all hierarchical agglomerative).
There's no need to invoke Taylor: "locally linear" means the same thing as *differentiable.*  As soon as you frame the sensitivity analysis in terms of partial derivatives you have, *a fortiori*, assumed local linearity.
Have a look at this: http://stats.stackexchange.com/questions/25324/proportion-estimates-shrink-to-the-mean-based-on-sample-size  and this http://www.win-vector.com/blog/2012/07/modeling-trick-impact-coding-of-categorical-variables-with-many-levels/
Why do you equate "measure" with "estimator"? "Measure" indicates the author is using $s^2$ to *describe* the data, not to estimate any underlying parameters.
@MånsT: To Find the conditional probability First I consider N=n(fixed).But lastly I consider it as variable.In this way We proof _Wald's Equation_ also.
Seeing as there are so many rows of data, the suggestion to set aside a subset for training and then use the rest for crossvalidation seems very sensible.
Andrew, I had hoped to get a good answer for this.  Sorry that it did not work out so well.  -mike
This sounds like the same question as one recently asked in a more abstract way at http://stats.stackexchange.com/questions/104990. Both questions seek to detect more than a mere difference in average value among groups.
Of course the forest has conflicting decisions.  That is why the decision is made by majority rule.  So-called training of a classifier in the context of trees is done by growing the tree and creating the forest. A constructed tree is equivalent to a fit LDF or QDF.
We're glad to help, @edouard, that's what CV is for. If Peter has provided the information you needed, you might consider accepting his answer by clicking on the check mark to the left underneath the vote total.
@shabbychef Of course you're right, thanks for catching that!
I would think you would find trends in the data related to time given that you have multiple years worth of observations, and that these trends relate to other patterns you might detect.  But, without knowing more about your data (see [this post](http://meta.stats.stackexchange.com/questions/1479/how-to-ask-a-good-question-on-crossvalidated) for more info) I can only offer general comments.
If you work through the cdf, you should not change the integrand from the first to the second integral. Your mistake is in trying to use both the cdf and the Jacobian approaches at the same time.
@PeterEllis: For analyses at the person level, I can use $w_p$ as weights. So you're right: I want to analyze at the household level, with variables such as "number of children in the household". I'm not concerned too much about households occuring twice or more often in the sample, as I hope that the reweighting will fix this. After all, what's the difference between drawing twice from the same five-children household and drawing from two different five-children households, at least for the analysis? Or am I too optimistic?
Dear Mermex, Thank you so much for your reply. I'm sorry if I didn't explain the subsample very well. I foresee two ways of estimating the impacts of health insurance on health expenditure. The first is to take a subsample of individuals who have any health care visits and compare their expenditures, controlling for health insurance status and other variables. The second is the two-part model (either log-transformed OLS or Heckman's sample selection model) to compare expenditures between the insured and uninsured. In my understanding, if I take a subsample of individuals with any expenditure..
Is that $l(\\beta_j) =$ first line correct?  Or should it be $|\\beta_j| - \\varepsilon/2$?  (That might also account for the difficulty you mentioned in the comments responding to @TenaliRaman.)
If the f() call is long then it's fine.  However, if the f() call is short then any timing call overhead will likely be increasing error measurement.  With a single call for system.time() over many repetitions of f() one gets to divide out the error the call until it's some infinitesimal value (and it returns faster).
@user1228633 The $p$-value answers the following question: If the null hypothesis were true (i.e. the means were really equal), what is the probability of receiving a $t$-value as least as extreme as the one that was observed. If this probability (i.e. the $p$-value) is very small, then we have evidence that the null hypothesis is unlikely. But the $p$-value is *not* the probability that the null hypothesis is true. See also [here](http://graphpad.com/guides/prism/6/statistics/index.htm?what_is_a_p_value.htm).
Both models seem to encompass valid modelling assumption. More as personal comment, I would not include intercepts especially when looking at the `period` because there is no "intecept" as such case. Just to clarify this: do you have 2 distinct treatments levels per site plus no treatment at all, or a treatment\\no treatment design? Don't be taken aside if the `site`-specific variation due to treatment comes out very low, you might encounter issues of identifiability between the `site` and `site/sample` random effects.
This reminds me of a similar problem I'm working on. I'm thinking logistic regression, but you violate the independent observations assumption if you use both players in each race as separate observations. You probably ought to reorganize your data so that `win` reflects whether you won the race, and all your opponents' info is included in each race's row (in separate columns after the columns with your info) for use in predicting whether you beat your opponent that race. Also, for feature selection tips, check out [this question](http://stats.stackexchange.com/q/490/32036).
"adjusting for age and gender" sounds like your covariates are correlated with the other predictors, which violates the assumptions of your analysis. ANCOVA is supposed to be a means of enhancing statistical power, it is not a means to eliminate confounds in your data. See: http://stats.stackexchange.com/questions/4551/what-are-common-statistical-sins/5865#5865
Stuart & Ord, in *Kendall's Advanced Theory of Statistics* (Fifth Ed.), section 3.26, discuss such moments and provide references.  They show how to obtain negative and fractional moments of positive distributions from their moment-generating functions (provided those are defined for all non-positive arguments). One reference discusses relationships between fractional moments and fractional derivatives of the characteristic function.
Thank you, that fits perfectly! Do you know how can I add the corresponding legend? By that I mean not the actual placing of the legend, just figuring out which is which, because I got several wiggly curves...
@nico good point, but I think this is only for mid-game. 1 letter words either don't require one to play a letter, or would allow one to place a single letter anywhere on the board, both clearly unacceptable. However, I was thinking of the opening move. In fact, the question can be compactly stated, for those familiar with Scrabble, as "what is the probability that the first player will have to pass?"
Classical derivations of t-tests use assumptions of Normal individual residuals to justify the Normal distribution of the between-group difference in means, the distribution of the average squared residuals, and the independence of those two quantities. But the same test is also valid in large samples, without the Normality assumption - so a legitimate answer to your "what do you do" question may be "nothing; just use the t-test".
My understanding is that GA is designed to tackle problems which are harder to solve with standard approaches. Shouldn't it perform better exactly in a situation you described?
Are you scaling the feature vector data for the SVM?
What is the connection with http://stats.stackexchange.com/questions/52014/great-challenge-estimate-electricity-from-the-following-dataset? Is this homework for a class you are both taking?
High R square does not mean that the regression is good. Generate two independent random walks and regress them each on other. If you repeat this multiple times you'll get that R squared is high for a non-significant part of such regressions, when clearly no relationship is present.
In fact, since K-means does not assume that the data points are generated from a Gaussian mixture model, it doesn't deal with the variances explicitly, but it actually includes equal-variance assumption implicitly. If all the denominators in the new objective function that I wrote in the question are equal, then you can just omit this denominator, and you will be left with the K-means objective.
I thought I had responded to this, sorry = $\\phi$ is to be maximised
One basic notion that will go a long way: the sum of conditional probabilities conditioned on different events makes no sense whatever.  You will never see $P(A\\mid B) + P(A\\mid C)$ (but you might see $P(A\\mid B)P(B) + P(A\\mid C)P(C)$...)
GLM will work with negative values. I would try it with the log link function and maybe normal errors to start.
this is crossposted to math.se.  Also the fractions in brackets are flipped.
A "statistically mature" answer would begin with *questions* about the objective here, recognizing that "similarity" can mean so many different things.  What do the points represent--a sample of something, something measured with error, a population of things, an xy scatterplot of results of a controlled experiment?  What is the purpose of measuring "similarity" to a "line"?  (What is a "line", even: a Euclidean line or perhaps just a line *segment*?)  What would it cost your client to accept suboptimal solutions?
This question needs additional assumptions in order to be answerable. Also, it deserves quite different answers depending on whether this is a homework/test question or if it's something you are genuinely interested in (and you therefore don't have to guess what a test creator intended).  Could you please tell us which it is?
re the second paragraph: I think many of us who contribute regularly to this site would bristle at the suggestion we're trolling for clients or that every statistical problem requires the intervention of a statistician.  Some 6,000 answered questions attest to the opposite.  It is rare on this site for anyone to recommend, either in a comment or reply, that consultation with a statistician or extensive additional study are necessary.  When such a recommendation appears, it should be taken seriously.
@Avraham Shouldn't that be the classification model?
Yes, but that's not an approach to the problem, just one of the ways of classifying it...
This test is well documented in the [stata manual](http://www.stata.com/manuals13/rregresspostestimationtimeseries.pdf), which explains what all these numbers mean.  It sounds like your question therefore concerns [understanding p-values](http://stats.stackexchange.com/questions/31).  If you have further questions, please edit this one and the community can vote to re-open it.  But please pay some attention to the formatting and make sure you review your question before posting it so that you know it's readable.
This question and its responses remind us of how crude and limited this antiquated division of variables into categorical-ordinal-interval-ratio really is. It can guide the statistically naive, but for the thoughtful or experienced analyst it's a hindrance, an obstacle in the way of *expressing variables in ways that are appropriate for the data and the decisions to be made with them.* Someone working from this latter point of view will freely move between categorical and "continuous" data representations; for them, this question cannot even arise! Instead, we should ask: how does it help?
Many thanks for a well-written answer! My headache is now gone. I actually constructed the very same sequence of priors but tried to apply it to Lemma 2.4.15 instead. It's good to see that at least I was on the right track. :)
Try looking into the work of Terence (T. J. G.) Tracey for info on a method of partialing out the first unrotated
factor as a "general factor" attributable to social desirability or common method bias.
I just did it the "slow" way in R - which is already regarded as being slow. That is, (i) I started with n randomly generated degrees of freedom (so there's no possible speedup in adapting to a specific distribution), and then (ii) generated n chisquared random variables each with its own previously obtained df, then took the square root. For n = 10^5, it did the second stage (generating chi-squares taking square roots & assigning the result) in <0.1s on my slow little laptop. You should be able to do much better (maybe 10-20x?) on a better machine in C++ even doing it the 'dumb-but-easy' way.
"arbitrarily" virtually rules out any specific procedure; it is no information at all.
just a remak in the case where you do not know this: the cdf does not exist in closed form
@bnaul: this is completely right, being based on the inverse cdf transform. It however implies getting the quantile function of the normal distribution to a very high precision. Especially when $a$ is much larger than $0$.
Have you asked this in the R-sig-ME list (r-sig-mixed-models@r-project.org)? Some people might be able to help you there further. In addition: I would strongly urge you to study the paper [Efficient Laplacian and Adaptive Gaussian Quadrature Algorithms for Multilevel Generalized Linear Mixed Models](http://www.jstor.org/stable/27594165) by Pinheiro and Chao. It contains results regarding AGQ performance as well as the linear algebra behind them. If you want to code them.... well, get ready for some serious sparse matrix decompositions. :D
(ran out of space) Does that interpretation make sense here? I'm not exactly sure how to apply it directly.
I am not sure how far we should take this discussion as SE is not a good outlet for discussions. I agree with what you said but I am not sure you are addressing the issue I raised. If you use a nuisance free likelihood how can you construct a confidence interval for the nuisance parameter? In any case, I should probably stop here and I will let you have the last comment.
I think determining the health effects of getting over the R learning curve would be an excellent study!
If you have access to it, the article *Block Kriging for Lognormal Spatial Processes* by Cressie, probably describes exactly what you want to do.
( http://link.springer.com/article/10.1007%2Fs11004-005-9022-8 ). 
Also for Q1. Your logged data looks quite fine; you are working with "true" data, I have seen far worse plots being called "Gaussians". For Q2. A simple $exp()$ should do. After all you are logging only the $y$-axis. Your $x$-axis remains unchanged.
I can't quite figure out whether this is a statistical question or a purely programming/software question. If it's the latter, the question would be better suited for Stack Overflow.
Thanks. I'll have to think a bit about the choice of those αs, β, and γ. And, no, I don't want to build an algorithm based on the distance matrix. I just wondered how hierarchical clustering can possibly work if the distance function is unknown to the dendrogram building algorithm. And I notice that unsuitable choice of linkage method and distance can lead to dissimilarities that cannot be interpreted as the original distance (e.g. using 1/2 - cor (x, y)/2 as distance + Ward's method often gives fusion distances > 1).
We don't *always* do it. But besides the 'turn products into sums', there's also the fact that the numbers become far more tractable (computing pure likelihoods on typical to large sample sizes can often suffer from underflow problems), and finally there's the fact that a very large number of densities and probability functions involve exponentiation or powers, so even the individual terms may become nicer to work with. In fact, there are typically so many reasons to do it, you would usually only avoid it with good reason.
Thanks, this is just what I was looking for.

This is essentially a million subjects each with an entry and exit time.

Yes we are conditioning to account for the censoring.