tttjjj/elt_bench huggingface dataset:
Since this seems to be routine bookwork similar to what might be assigned in a subject or done as an exercise, it should carry the `self-study` tag. Please read its [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) and modify your question accordingly. If you don't think it counts as self-study, please explain the context in which such a question arises.
Do you mean in [this sense](http://en.wikipedia.org/wiki/U-statistic)?
So was the response no available to the clustering? Do you intend to ultimately regress the response onto the clustering you found?
...4. we calculated average (for three seasons) deer density on the plot,
5. because we had two plots in each unit, we calculated average for those two plots and got deer density per unit (120 units)...
@Alexis, suppose $W \\sim \\mathcal{N}(0, 1)$, and suppose $X = W^2$. Suppose further $\\mathcal{E}(Y | X, W) = \\alpha + \\beta W$ so that the regression of $Y$ on $X$, and $W$ gives $\\alpha, 0, \\beta$ as coefficients. The scatterplot of $Y$ on $X$ would show a quadratic relationship when there is *no* relationship, because it is a projection. This is the caveat of displaying marginal associations. You could scatter residuals after subsequently regressing other factors against the outcome in order to show conditional associations.
This sentence itself is a model (an epistemological one)
That is a helpful clue, but doesn't solve the problem of starting with an `mcmc.list` (as far as I can tell).
(And while writing this, I realise something: why wouldn't I take all the observations on all the years, put then in a big table and analyse them all in once ? I'm suddenly wondering if there is a point to doing separately by year. That could be my solution for aggregating the p-value: just analyse everything in once. But I HIGHLY suspect that I'm forgetting a big point that makes the yearly analysis unavoidable. It's gonna come back to my mind in a few minutes, I gues ;) )
@Peter Flom makes a very important point about whether false positives or false negatives are worse. False negatives can be bad if the condition is serious and must be identified early. False positives are bad if follow-up tests are invasive.
since you are right with your suspicion, why not answer your question yourself? I means you basically state the answer in the question anyways.
By "efficient" I gather you mean something like: "We get roughly the same information back, at 40 per cent of the cost."  Yes? If so, then you might phrase your statement in terms of the _cost of uncertainty reduction_ or the _value of information_. In that case, it would be helpful to understand the larger context in which this information is intended to be used, in order to understand what are the costs of testing versus the costs of error.
(+1, awhile ago) My favorite example of this is $(X,Y)$ distributed uniform on $\\{(1,0), (0,1), (-1,0), (0,-1)\\}$. In that case you get $\\mathbb E(X \\mid Y) = \\mathbb E(Y \\mid X) = 0 = \\mathbb E X = \\mathbb E Y$ but $X$ and $Y$ are not independent.
The variables which are divided by category, sub-category etc. are **input**, not output, variables.
I suppose by "repeated measures model" you mean a mixed model (as in the link you provided). It is really important to be specific here: You do NOT seem to prefer repeated measures ANOVA (as in the question), you prefer mixed models for repeated measures. And as pointed out in the blog post, mixed models really are preferable in most cases.
How do you know that your data are not normally distributed? If you only have 4 data points per group, I guess you don't have a lot of power to detect non-normality. Remember that if you want to check normality, you should check it *within* groups, not across all groups.
**Hint**: $Y_t = X_t - X_{t-1}$ are iid normal with zero mean and variance $\\sigma^2$.
Thanks a lot for the reply.
The likelihood for the full model is the product of the likelihoods for the two groups... so yes, you just add the log-likelihoods.
According to their promo video they offer free technical support and "resources to success".
@Gob00st You're welcome. Glad I could help.
By "data set" you appear to mean "variable".
your premise $P(AB)=P(A)P(B)$ is not what @honitom assumed, or what the question wording says.  Its actually your conclusion that is put as the assumption "...The results of the two tests, A and B, are independent given the presence or absence of c..."  Which means $P(AB|c)=P(A|c)P(B|c)$ and $P(AB|\\neg c)=P(A|\\neg c)P(B|\\neg c)$.  And further, the relation $P(ABC)=P(AC)P(BC)$ is not a general one.  e.g. take A="is female human" B="is male human" C="is human",  Then we have $P(AC)=P(A)$ $P(BC)=P(B)$ but $P(ABC)=0$ (more abstract take $A=\\neg B$ and $C=A\\lor B$)
Actually, I just realized that if any of the $w^T x_i$ are 0, then $W$ will be singular. Also, in practice it might become arbitrarily close to singular even if it's not actually singular. You are using a `solve` call rather than an explicit inverse, right?
+1 Thank you. In the time that I am understanding your answer, I just updated my question with the `bootstrap` parameter set to 500 in the `descdist` function. And yes, you are right that my values are not in [0,1]. Is there a way I can show that fact (belonging to weibull) using this graph? I will try to update my question with a QQPlot shortly.
@erik.brannstrom if your goal is the creation of rules, then DT is the way to go because rules can be easily generated from them (btw set of rules with conclusion click yes/no is also a classification model). If the decision tree is effective => rules are effective => you can extract the important property/value combinations from the rules. On the other hand, if you cannot build an effective DT, there is not property/value-combination with important influence on the clickrate of the add. Think about it ;)
You may need to provide more information. What do you mean by correlate? What are n and N?
In that case your interaction involves more than 1 parameter, so you need to take that into account. Both the Wald and likelihood ratio test are possible. I don't know how these are implemented in SPSS.
And I don't understand your explanation. That paragraph is very unclear... what do you mean by "there is no effect size" and "whether the test size is correct"?
no problem Comp_Warrior, I didn't think it was you, but someone did downvote my answer and I would be happy to have some feedback on why.  We are all fallible and if I have got something wrong in my answer, I am keen to correct it.
Effectively you are only fitting the slope with your new regressor, and no intercept. This might or might not be correct in your case. If you are unsure, I'd fit the model with and without indicator regressor and try to understand the output.
@Jonathon Christensen - this book is wonderful, but you really need a good background in calculus and linear algebra to get the most from it
Which definition of white noise are you applying?  According to some definitions, even independence of a "white noise" process does not imply ergodicity.
Hi @Peter Flom, thank you for your answer. I completely agree with you. The issue is in my case the data are in order of millisecond and with this magnitude, we don't expect instantaneous heart rate changes dramatically between two points that are apart less than a second and if a change happens, we expect the effect to remain for few seconds. therefore jumping from 580 to 1500 and then going back to 570 within 1.5 second is unexpected.
I like your perspective and I actually think the same for general cases of removing outliers. I really appreciate your comment.
I was thinking the same thing, only plotting summary numbers--one variable to represent risk and one to represent return.  If you try creating this plot, I hope you'll post it here.  It'll be interesting to see.
@whuber I have posted the solution using double integral.
Thank you but I am too much of a beginner to know that!Someday I hope!
Given that you have 5 dichotomous variables, I am not sure what graphs you created. Can you tell us? Also, even though it's a 5 way interaction, it's only $2^5 = 32$ combinations, so you could look at the predicted value for every combination without it being too overwhelming.
Hi Gina--welcome to our site!  I am wondering why you have written an answer about "efficiency" to a question that asks about *efficacy*.  Is it possible you misread the question?
would there still be an issue if you only use 2 of the three variables?
The most information usually hides in the data itself, not the description of it (-;
@subhash c. davar You are correct but that was not my point.  I did not say that this answer is wrong, but only that it does not address the question.  A correct but irrelevant statement, especially about terms that sound similar, can sow confusion.  This sometimes is worse than an (obviously) incorrect, relevant answer (which sometimes, through the nature of its error, can be illuminating).
That's simply because of the fact that the Gaussian is a continuous distribution. It's the probability density at a point, not the probability OF a point that matters. One could also say that computers works with discrete floating point numbers not real numbers. You can still do regression on data that has been forced to be integers, even simply 0 or 1. The calculation just sums things anyway and so it is these sums that effect the result. The sum of many independent variables tends toward the Gaussian anyway regardless of their individual distribution (with some mild assumptions).
After "... writing $\\Phi$ out," I would have used the phrase "staring very hard at the derivative of $-e^{-x^2/2\\sigma^2}$ to see if inspiration strikes,...."
A trend cannot be stationary. The null hypothesis is a time trend with stationary errors, as I explained.
@David: I have also heard of people reweighting, and using complicated pseudo-bootstrap schemes where they only resample the high-frequency class. For all these techniques, you're ultimately throwing away (or making up) data. I'd argue that if this improves your model, then you're probably fitting the wrong model. See also my comments here: http://stats.stackexchange.com/questions/10356/is-a-logistic-regression-biased-when-the-outcome-variable-is-split-5-95/10397#10397
Did you use the same parameters ($c$, $\\gamma$) before and after scaling? If so, that would be why.
@DavidR. Please accept the answer and vote it up if you found it informative. Thanks!
How well do these functions deal with missing data? I quite often have gaps in the time series.
A pool of two cross-sections, one from the year before the policy was implemented and one from the year after.
Those are just hiding the loop from you. The real problem with @Produnis code is that forced copying is going on because the results vectors are being extended at each iteration of the loop.
Define "effective."  Your goal is to assign cases to clusters, right?  What will you do once you've assigned/discovered the clusters?
Unfortunately thetaEst only estimates person parameters for dichotomous data... my data is polytomous.
@whuber - The point is relevant to bring up because it is precisely this misunderstanding that leads people to interpret CIs in the wrong way.  Confusing probability with "rational degree of belief" is not consistent with the frequentist paradigm. This is what happens when you take CIs to mean "probability of true value being in interval", which is what @dsimcha is doing in the question.
Yes the most useful is what I meant.  I think part of my problem with figuring this out is how to word it.
If it's for a dissertation, ask your mentor. If it's for a paper, try taking the mean and see what reviewers say.
@gung many thanks.
You're right, it is a problem. Not so much with the point estimates, but the measures of precision (i.e. standard errors) will be too small; it ignores the multiple use of the control group's data. It shouldn't however, be news to anyone in meta-analysis. The Kim/Becker article above is basically a re-statement -with acknowledgement- of Gleser & Olkin (1994). Stochastically dependent effect sizes. In Cooper & Hedges (Eds), The handbook of research synthesis (pp. 339–355). This book is a standard text in the field, I believe now in a second edition.
Thanks, that seems to work actually. I just want to make sure it really calculates what I want as this is behavioural data and not to compare the ratings of people. Also, how would you write up the stats of Kendall's W for a manuscript? Thanks
Ideally, you could provide a reproducible example (code, data pasted in or provided as a supplement)
@cardinal I thought it self-understood but I will point it out, sure. Thanks.
Although the question has already been answered mathematics-wise, I think a philosophical answer would appear to be more relevant in this case. 

I agree with the question author - in reliance to the reality, such division makes no sense. In general, a binary categorical variable represents an answer to the question "Does it exist?" Whereas a continuous variable represents an answer to the question "What it's measure is in comparison to...". Those are different fundamental concepts.
Sure. An evil scientist trapped a statistician in a locked chamber containing only a thermometer. "Tell me whether the temperature is changing!" commanded the villain. After collecting data, the statistician conducted a two-sided t-test and concluded the temperature indeed had changed. As he announced this, he also added--shivering uncontrollably because it had become so wickedly cold--that it was nevertheless impossible to tell whether he was being roasted alive or frozen to death. Wuschelbeutel, if you think like this statistician, then you do need another test. Otherwise you're just fine.
0% true positive and 0% false positive?  It must be putting everything in the other group, then... try setting your prior at 80% for the "positive" group.  But first, check to see you aren't making a mistake somewhere in the code...
(+1) because "Canned analysis running from an algorithm often won't get you the best answer" is very true. This doesn't mean that human practitioners of statistics don't do this all of the time. (Note: most practitioners of statistics are NOT statisticians... more like people who are using statistics despite not really knowing what they're doing, often resulting in bad science)
@Alexis Another way to appreciate the thrust of this question is to adopt a very general viewpoint in which a stochastic process $X$ is defined on a Riemannian manifold $M$. Its correlation function is $C(x,y)=\\text{Cor}(X(x),X(y))$ for $x,y\\in M$. An arbitrary function $C:M^2\\to\\mathbb{R}$ *can* be such a correlation function provided it defines a nonnegative-definite form for all finite subsets $\\{x_1,\\ldots,x_k\\}\\subset M$. When $C(x,y)=\\gamma(\\text{dist}(x,y))$ for some $\\gamma$, $C$ is *isotropic.* For the usual (flat) cylinder, what functions $\\gamma$ arise from isotropic processes?
Even better, use ets(sdata, model="AAN") which is equivalent to Holt's method.
In that case, you need to enter minus the likelihood into R (Q <- function(beta, data) {... }) and then to rely on an optimisation routine like 'nml' or 'optim'.
FWIW, just yesterday I posted a simple working example of this (in `R`) at http://stats.stackexchange.com/questions/56459/how-to-write-a-function-to-generate-a-sequence-of-points-in-r/56491#56491.
What I honestly don't understand is how you was able to employ k-means clustering with proximity matrix input (and that being cosine!). K-means clustering needs raw data (objects X variables) input and internally operates on euclidean distance.
Ah, I see it's that easy... I thought variances needed to be computed and compared with that of the OLS estimator. This makes it a lot easier though. Thanks for your help.
You may find some useful identities in [the matrix cookbook](http://orion.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf). In particular, chapter 6.2 and/or 8.2 may have what you need.
Thanks for this Rob, all a bit new to us but you've but us on the right path.
This post is being automatically flagged as 'low quality' because of its length. At present, it's sort of a comment. Can you expand it a little bit to make it more of an answer?
Nothing wrong with rigorously testing a priori defined hypotheses *and* snooping the same data to suggest the next a priori defined hypotheses to be rigorously tested. And if we're in an even slightly exploratory mode we should say so - just say what we really did - & let others decide with exactly how large a pinch of salt they want to take our results, however convinced of their validity we may be ourselves. I'd like to give this answer more than one vote for emphasizing honesty.
Possibly these [Gelman slides](http://www.stat.columbia.edu/~gelman/book/slides/class4b.pdf) about an essentially identical model are helpful for understanding the 2.2 option.
@PeterFlom Given that in one-way ANOVA there's no difference between variance of the observations within a level and variance of the error term within a level, what distinction are you drawing there?
my son was born by Caesarian. I assure you it was traumatic in the casual sense of the word!
ahfoss thank you very much for your detailed information and the links. I will  definitely take a look at them.
There's also an example of using Mfuzz in my [tutorial paper](http://www.mas.ncl.ac.uk/~ncsg3/microarray/)
Thanks, Marianna.  I would prefer a solution that relies less heavily on (typically, unjustifiable) parametric models, but will definitely look into it.
@mzuba, non-linear estimators will outperform OLS if the relationship between $y$ and $x$ is non-linear **in the parameters**. $y = \\beta x^2 + \\varepsilon$ is still a linear model.
@dartdog: The great thing about time series is that you can include seasonality components and things like that. This is very useful in Q1 is higher than Q2 etc. This is done in cycles. However with annual data this isn't as present and in your data the dependent is forecasted on the basis of an independent variable: the unemployment rate. As I understand the situation you may want to approach this more as a linear model - maybe include a few more variables - but I don't see a lot of real value adding in using time series models. As shown on stack exchange it can be done but...
Thank you for answering. I am running MANOVA on the means per participant/dimension. I shall read the paper you suggested and see whether I should go for multilevel models.
Is there a plausible distribution for these numbers? Are they counts? What numbers are considered "similar"? Is Euclidean distance a proper metric?
These distributions differ in infinitely many ways: for instance, few of their moments can be made to match simultaneously. They tend to arise in different ways and are applied in different circumstances.  They share some qualitative aspects, like unimodality, but will differ on infinitely many other qualitative aspects. So for this question to be answerable, please edit it to indicate *what kinds* of comparisons you wish to make and why.
@user1893354 The day you go out of business ;) And the curve should never get "fuller" (by which I assume you mean go up) because your time variable is not calendar time, but time since the account was opened.
I have updated my answer with your suggested plot.
"The error term is not a parameter which you're independently trying to adjust, but the intercept is (e.g. your data might best be fit by a horizontal line)." I think you mean "...fit by a line through the origin," since omitting the intercept term constrains the intercept to be zero, but yields a line with slope $\\beta$.
You must be thinking of cumulants.  If any even moment (central or not) is zero, the distribution has to be a degenerate atom.
This answer, although containing correct information about the radius of convergence, is telling us about the tails of the sequence $a_n,$ not about the tails of the distribution $f$.
I don't quite understand.  What is the mean you are trying to measure - is it the average number of events per visit (over the full time period) for each treatment?  Also, I presume you don't have access to the underlying microdata ie visit level, showing for each visit how many events it triggered.  Finally, how many dates do you have data for?
@htrahdis Ok thanks. What if each variable simply has equal weight?
Sounds a bit like data dredging to me. Shouldn't the focus be on what you plausibly think is an appropriate model, what covariates, transformations etc *before* you start modelling. R doesn't know you did all that model fitting to find a good model.
@MiroslavSabo The expected value is NOT the true mean (in general). Easiest case to consider is a binary outcome with the probability for 0 being 0.9 and the probability for 1 being 0.1. The midrange will quickly be (and stay at) 0.5 but the true mean is just 0.1.
Are you saying you don't have the original data - you ONLY have the conditional probabilities and the marginals?
Belongs on quant.stackexchange.com
http://cran.r-project.org/web/packages/rugarch/rugarch.pdf page 71-72
The expected gain from the winner's bet is $0.60 \\cdot 50 + 0.3 \\cdot 150 + 0.1 \\cdot 200 = 30 + 45 + 20 = 95$, so the winner's bet is a loser...
I'm unable to see why that follows; I've frequently seen them come out to be very similar, in which case (according to this rule of thumb) we would end up using the Pearson correlation.
Actually, the more data you have the more power you have for your tests. This would actually be a reason in favor of doing the adjustment, as your loss in power is offset by the sample size. Still, it also depends on your expected effect size. What exactly do you mean with "models conditioned on binary data"?
@user4422, hand-waving is good, but I did not see any proof. All the technical details usually come up, after you start to write rigorous proof. As the text stands now it is just an idea or strategy of proof, not a proof. The 2 things which stand out is the first claim that omiting some elements changes nothing, another thing is the claim that the remaining sequence is iid, it is not and assymptotic closeness to iid does not immediately allows application of CLT for iid variables.
"Better" depends on what you look at; Bessel's correction makes it unbiased, but unbiasedness is not of itself automatically "better" (the MSE is worse, for example; why should I prefer unbiasedness to smaller MSE?). Unbiasedness might be argued to be better, *ceteris paribus*, but unfortunately the *ceteris* won't be *paribus*.
Oh, okay, I see now. No, definitely don't do that. Hang on, some edits are in order!
Thanks Stephan.. Wish this course started sooner!
Differencing can also be used to remedy non-stationarity when no trends are. Unwarranted usage can create statistical/econometric nonsense as you correctly point out.
@alto Thanks! That's a helpful hint. Would you have some more links to suggest? Thanks again!
This answer is excellent, and I just wanted to appreciate the effort you put into it. Thank you!
This is a tiny generalization of the question at http://stats.stackexchange.com/questions/3961/random-permutation-of-a-vector-with-a-fixed-expected-sample-correlation-to-the-o, whose answer applies to this generalization.
I found the [Wikipedia write-up](http://en.wikipedia.org/wiki/Markov_chain#Periodicity) succinct and clear. Does it do the job for you?
@Lostintranscription - thanks for the reply. I'm aware that comparison of p values is not really meaningful. Is there a method of converting the above to fuzzy sets and comparing across the two conditions?
Being not in your area I'm not sure what you mean by "each treatment should be replicated three times" or the precise nature of treatment; are they between or within samples?  To me it seems practical to reduce to a treatment and control (after all more samples in a design cell the better), but that is a trade off you should determine based on your needs.  Which is more important, experimental power or evaluating both treatments?