tttjjj/elt_bench huggingface dataset:
@MichaelChernick Most people, and probably all scientists, are interested in the evidence. The idea of a 'meaningful difference' in the context of testing comes from the Neyman-Pearson approach where it stands as the alternative hypothesis. That approach eschews any evidential basis of interpretation in favour of 'behavioural inference'. Not as useful as looking at the data as evidence, in my opinion.
Thanks a lot Mr Michael for your wonderful interpretation.
Dropping the $i$ subscript, you can use either $\\Delta y_{t−2}=y_{t}−2−y_{t−3}$ or $y_{t-2}$ as instruments for the difference $\\Delta y_{t−1}=y_{t-1}−y_{t−2}$. Arellano (1989) shows that the first approach has a singularity point and huge variances for a large range of parameter values. The levels instrument has neither, which is why I recommended it
The comment "show signs of autocorrelation" indicates you have a model in which $x$ and $y$ are viewed as time series (or something like them).  The question itself suggests a concern that the sampling distribution of their correlation coefficient will differ from that given by a theory in which $(x,y)$ are viewed as an i.i.d sample--and it is probably correct about that concern.  However, to address this question we need more information about the presumed underlying model. Could you edit it to explain what kind of data $(x,y)$ represent and *why* you are testing their correlation?
Sounds like either GLMs or regression would be suitable, depending on the conditional distribution of neck circumference and the anticipated form of relationship.
(+1) Really neat R code. Of note, it will not work if $n$ is odd.
The only clustering method I know which is for big data (e.g. millions of cases) and can accept nominal variables along with numeric ones is TwoStep cluster found in SPSS.
This question doesn't belong here, since there is no programming question. This should have been posted on CrossValidated instead. But the goodness of fit of factor analysis is usually measured by chi-square, not R-square, so your question doesn't make much sense.
I wanted to give an easier explanation of the data but you got confused (sorry:/) Here is the actual data and I want to predict 30 more values on TS1
https://docs.google.com/spreadsheet/pub?key=0AiDb3EV0gzlEdERaeXRiMnBGMXRZQldYNllJQ05aMnc&output=html
To downvoters and flaggers: it is perfectly legitimate for *anybody* to jump in with an answer that echoes (or even copies) a comment which happens to contain a solution to the problem that is posed. This is because answers have more status on our site (in terms of searches, community votes, and opportunities for improvement): it is preferable that solutions appear as answers rather than buried in the comments. So please vote on this answer based on its merits, rather than on its apparent originality or lack thereof.
in case i have a case where my users rated less than 10 items, is it ok to use minhash for this? since picking up a subgroup from a group will almost always result into an empty set (in this case, when a user has less than 10 ratings out of 7900 items, the minHash could be contra-productive imho).
Can you please add some details? Now it is like asking in which shop you should buy ingredients provided that you decided to use copper pan.
The assumption is that these $SS$ are $\\chi^2$-distributed *under the null hypothesis*, which is that the group means are all equal, i.e. $M_j=M$ for all $j$. When this is the case, $y_{ij}-M_j$ is normal implies $M_j-M$ is normal. So you only need to check the first, i.e. that the observation-level residuals are normal.
Thanks for the response. I meant it in the way you describe first, not as an outlier. These are data from an experiment I designed, and I think the reason for the infrequent examples is probably a flaw in the experimental design. So while I am very interested in the infrequent classes, it seems like there's not enough data on them because of the design, not because of some real fact about the phenomena I'm studying. I could be wrong and it could be like the fraud detection scenario, but I think I can make an argument for why that's not the case when I present the results.
I see...How would we approach the last two sentences of your response?  It looks now like Machine 4 already has accuracy+precision upper bounds less than .04%.  I must have calculated that wrong.
Although this *motivates* the equation, it doesn't go very far towards demonstrating it--and perhaps even obscures the issue.  The question concerns the meaning of convergence of the infinite series of shift operators. That meaning is quite different from convergence of the series of numbers (or functions) which appear in Taylor's Formula.  Here, higher powers of $B$ just shift the time series $(x_t)$ by greater amounts.  Thus, convergence must occur relative to some metric in which greater and greater shifts become ever closer to the zero operator.
@cyrus1.618, yes, I was only meaning to say that this is an example of how higher order moments get involved in the distribution of the third moment. The K-S test would be the test on $p$-values of the Pearson $\\chi^2$ test... a meta-test, if you like.
Good suggestion!  Doesn't seem to include observation weights, though.  The weights ("lambdas" in the documentation), are for the independent variables...
Another question: how does a Gaussian *distribution* enter into KDE? Are you sure you are not looking at a Gaussian *kernel*?
Mentioning the book author @Cam.Davidson.Pilon, who may have a better answer than mine below.
But that doesn't answer the question because what you need to specify is how many terms to included in the moving average.
@gung, thanks for the pointers, I'm looking into those applications. I've edited my original question - what do you think?
You are looking for the [Constant Q Transform (CQT)](http://www.elec.qmul.ac.uk/people/anssik/cqt/).
"More generally, this will work, as long as you use the correct equation to convert the test statistic to the corresponding effect size measure." - And I guess these equations are derivable for the all the usual suspects when it comes to hypothesis testing, like t-tests (you showed that it was), Z-tests and different types of ANOVA?
Please put some work in your question. Now it is hard to read.
is there another link for that paper, or a nice snippet to insert into the answer?
Can you add a proper citation, or better an on-line version of the paper?
About what errors do we talk here? What do you mean by 'change of observation'?
You have an exceptionally well-educated grandmother :-).
no, it is not! The link does not give an answer to my question!
Thanks, Carlos.  Great book, I started reading it a while back but didn't finish it.  Got it to learn about machine learning and graph theory, but I'll go back and look for questions related to markov models.  I'll also take a look at natural language processing(I have never worked with it before)
Thank you Onestop
A related idea, performed directly on the empirical probability function rather than the ECDF, is [kernel density estimation](http://en.wikipedia.org/wiki/Kernel_density_estimation#Definition). This avoids the need for differentiation. This is a very widely used tool. There's also log-spline density estimation. There are many other techniques in particular situations.
Define 'better' for your purposes.
I don't know if what's contained in this link (https://tamino.wordpress.com/2010/03/11/not-a-random-walk/) answers your question, but I thought you'd probably be interested in it anyway.
Although it looks like it's been dormant for a couple of years...
@Phillip, my original response was wrong.  I think the edited answer is accurate now.
Assuming the foregoing:
$$
\\sum_{i,j} (x_i - x_j)^2 = \\sum_{i \\neq j} ((x_i - \\bar{x}) - (x_j - \\bar{x}))^2 = 2 n \\sum_{i=1}^n (x_i - \\bar{x})^2 \\> ,
$$
by carefully accounting for elements in the cross term.
I imagine you can fill in the (small gaps). The result then follows trivially.
@naught101 In retrospect, I kind of feel like I just rehashed what you said.  Hope that's not totally the case...
Thank you for your prompt response.It would be of immense help if you could throw some light into the area which you mentioned as well as how to do the coding and other tools if avalaible.
Unfortunately not. Basically every other 'line' (time point) looks fine so it's desirable to present the data in it's original scale.
Can you clarify your last sentence, @user30574? Is it intended as a question or a statement (ie, '... and so *does this* affect... ?' **or** '... and so *this does* affect... .')?
If you want to stipulate a prior for the SD do it like this: sd ~ someprior and prec <- pow(1/sd,2). Then use prec as the precision for your normal.
CAUSEweb has a bunch of good general teaching resources and activities, perhaps there'd be something appropriate for this situation there: http://www.causeweb.org/resources/
Nomenclature.  A parameter is a single weight.  Number of parameters increasing means the number of "neurons" or "connections between neurons" is increasing.  This means the topology is non-constant.
Your patient in plotting with chars is awesome and valuable!
+1 for actually solving his problem, as opposed to answering about missing data.
Does the statement make sense for you if you put scalars there? E.g, does $x^2/n\\to q$ in probability, where q is a real number give you any intuition?
-1 A successful hypothesis test (with p=0.05) actually says: If the null hypothesis were true, your result would only look that good in at most 5 out of 100 cases.
Can't edit after 5 minutes, apparently?  Anyway, I'll give that MICE paper a good read, but a brief skim indicates that MICE is in fact what I'm doing.  I just learned it from Gelman, who calls it iterative regression imputation.  Different statistical camps call things by different names sometimes.  Anyway, when a and b are missing jointly, I predict them as best I can with c and d.  I start with completely random imputations, and let both the original and the filled-in data predict the missing variables, one by one, incrementally changing the distribution from which I predict.
Can you tell us a little more about what you are doing and why, and what you are looking for?
+ will calculate main effects only, * will estimate interactions between factors connected with *.  ANCOVA frameworks usually estimate only a main effect of the continuous factor, but interactions between all grouped factors.
@assu. The `croston()` function in the `forecast` package in R will implement Croston's method for forecasting intermittent demand data.
Hi Tal:> Given that these seems to be typo-free scientific names, i would try the Levenshtein metric first (in the context of a 92-by-55 distance matrix) and see how it comes out.
Significance of predictors (equivalent to CL excludeing 1.0) has little to do with predictive accuracy.  Estimate the $c$-index by postulating a model and fitting it.  It is OK to look at $P$ and CL as long as no action is taken.  Assess overfitting by using the heuristic shrinkage estimator $\\hat{\\gamma}$ (model likelihood ratio $\\chi^2$ minus 5 divided by $\\chi^2$ in your case) or by bootstrap validation.  If $\\hat{\\gamma} > 0.85$ you don't have much to worry about regarding overfitting.  If you have overfitting you cannot use $P$-values or confidence intervals to do anything about it.
Thank you for sharing this approach, and for describing it in detail.  This will help me out quite a bit I think.
From further investigation of literature,  it seems that in order to have a samples from random Markov fields I need to turn to the perfect sampling - at least no luck to find a paper where Gibbs sampler is used to do that. However, I do not understand why to use perfect simulation instead of Gibbs sampling? If I have a conditional distributions for Markov Random Field, than it seems netural to exploit that in order to simulate the network by Gibbs sampler.
@Stask: I'm trying to predict whether someone will respond to a mailing (binary classification). My main goal is to work with a data set that is < 1 GB instead of one that is 10 GB.
Very interesting work indeed!
@Andy This plot actually started out as a 3D scatterplot with time as elevation: you're looking down from the top.  Spinning or manipulating such a plot clearly reveals the pattern.  In other cases I have resorted to dynamically slicing such plots over narrower time ranges: you can watch the temporal evolution as the time window moves.  That sounds similar to your small multiple idea.
a process does not necessarily have to take place over/change over time, but there is some set that 'indexes' a collection of random variables. This set can be continuous (e.g. time) or discrete (e.g. a set of possible states like the 4 possible nucleotides in statistical genetics).
For info on census / IPUMS, see: http://stats.stackexchange.com/a/27062/3748
transpose your data. You sample 8 measurements and the each batch is an element in your sample. In this case PCA expects batches in rows and measurements in the columns.
Although your current 2 respondents have made valliant efforts, this question is not really answerable without more info. Can you edit / rework this Q so that it asks a specific, answerable question? If not, this Q will probably need to be closed. Please see our [FAQ](http://stats.stackexchange.com/faq).
Multiple ordinal response variables, thanks.
@BobC, maybe one of the moderators can have your post migrated to this site. That would be the most reasonable. As for your technical questions, first of all, are you using the FFT to do the correlation? That should be feasible for 2 million data points on a half-decent computer. Your signal-to-noise ratio looks reasonably high, so you should be in business. A quick-and-dirty cut would be to fill in the missing data with either the last available sample or with zeros. The creep from sampling-interval differences may be the most challenging "feature" of your data to deal with.
Where *exactly* do you get the error messages? I would assume in the call to `efp()`. What is the output of `str(data)`?
Do you have a reference for that? A reference that says that if I find the 3-way interaction, and I create 2-way interaction plots, that no further testing is required? By the way, I found the SPSS code for generalized: http://stats.stackexchange.com/questions/40932/plotting-3-way-interaction-with-continuous-and-categorical-variables-in-linear-m?noredirect=1
Fit without the correlation structure. That will speed things up immeasurably. Then look at the model and especially `acf(resid(mod$lme, type = "normalised"))` and look to see if the residuals *are* autocorrelated or not. Also, look at the smooth functions, see which might be simple and dropped or made linear. For others, perhaps try fixing the smoothness for some smooths to reasonable values; setting `k` to a suitable value and setting `fx = TRUE` in the `s()` terms you want to fix.
I will adapt the example (make the individual differences larger). The output is different, but the question about df in lmerTest::anova is still in place.
It's probably best to *avoid* going about it. If that doesn't make sense / you want to understand why, it may help to read my answer here: [algorithms-for-automatic-model-selection](http://stats.stackexchange.com/questions/20836//20856#20856). Why are you opposed to regularization?
@AlecosPapadopoulos They are not. I give the answer exactly as it presented on my manual. I, too, find it obscure and that book ocassionally contains errors so I thought I run it by you. If there is another way to derive the distribution of $-\\log 2 \\Lambda $, not asymptotically, please let me know.
*I think interaction. I want to see if the introduction of new laws on specific technologies generates new technologies.. so, i want the technologies to be the d.v. and the years of the laws being in force to be the i.v.
(+1) Good question. If I understand this correctly, you have two multivariate regressions -- $\\mathbf{Y} = \\mathbf{X}_j\\boldsymbol{\\beta}_j + \\boldsymbol{\\varepsilon}_j, \\,j=1,2$ and $\\mathbf{Y}$ is $n\\times l$, $\\mathbf{X}_j$ is $n \\times k_j$, and $\\boldsymbol{\\beta}_j$ are $k_j\\times l$? That is basically, you have two competing regressor sets to explain your multiple outcomes? This typically leads to $l$ $R^2$ statistics per regressor set, and there is no accepted way of producing one scalar GoF statistic. Some advice is [here](ftp://ftp.pitt.edu/users/t/i/timm/MultBook/Multpapr.pdf).
@fspirit: That depends on the sport and the inequity in ability between the participants, for one thing. Just knowing *who* is participating in each contest can often be a strong predictor. [Here](http://stats.stackexchange.com/a/16148/2970) and [here](http://stats.stackexchange.com/q/30976/2970) are a couple of related posts.
Most people drop the $(p-2)/2$ term and simply use $$\\pi(\\beta,\\sigma^2)\\propto\\frac{1}{\\sigma^2}$$
Thank you so much for your responses.
J G, your 0% accept rate suggests you haven't been getting good answers here.  I'm sorry about that.  Please consult our FAQ at http://stats.stackexchange.com/faq#bounty for some ways to encourage better answers.
For $N > d$ it just means that the rows of $X$ span $\\mathbb{R}^d$.  In the applied setting you usually can access a rank function and you want to check $rank(X)=d$.
Excellent - many thanks ! I will do this tomorrow
Perhaps `lmer(Y ~ X + M + (1+X|Subject) + (0+M|Subject), data=mydata)`
Please register your account -- you will be able to directly edit your posts.
Basically you can treat counts in contingency tables as Poisson variates (see any text on GLMs). If you're thinking of comparing binomial samples you can either calculate residuals like this for $2j$ cells (the log-linear model), or using the variance $n_jp_j(1-p_j)$ for $j$ observations (the logit model). The overall Pearsons's chi-square statistic will be the same.
I fixed the original question so that it make sense.
It depends on what you want to do with the symmetrized Y. For example, do you need an estimate of scale of the symmetrized Y?
+1 for satisfying #1, #2 and #3 and the nice narrative.
Since you're an R-user, you might find [this vignette](http://cran.r-project.org/web/packages/actuar/vignettes/lossdist.pdf) of some value.
and a good short discussion and with two excellent references on [Paul Hippel's pages](http://www.sociology.ohio-state.edu/people/ptv/faq/weights.htm)
for a model in differences dx[t] = intercept + xreg*dy[t] + n[t]  and in all cases, remove ar1 or ma1 terms depending on the type of arima model, and choose model in levels (d=0) or differences (d=1); you may need to replace ar1 with phi1 and ma1 with theta1 depending on the package. Corrections welcome!
Don't forget potential autocorrelation within schools.
(+1) Good references! Thanks.
Could you explain this a little bit more?
I'm not sure if it's possible, but maybe you do something with arrowheads, for example at either end of the link. I was thinking that colour might work, but then you'll have quite a few links and the width won't be that great, so using a colour change visualisation is probably not good for this type of graphic. And one can have too much colour as well. :)
Hello and welcome to CV! Could you provide your research question(s)? Summing across time or within tree would depend on what question you're trying to answer. For instance, are the 20 trees the same cultivar? Were you trying to compare their bug resistance? If they are the same, are you trying to document the trend of pest abundance? etc. In addition, if you can explain a bit more on the "analyses," it'd facilitate users here to answer you.
I can't think of a reason it wouldn't be okay...can you? It's fine to sincerely hypothesize the typical null (i.e., no interaction) if you're interested in controlling for it at whatever value it actually takes in your sample. You can't interpret this like a confirmatory analysis of a prior hypothesis, but most people don't interpret these differently than analyses for exploratory or statistical control purposes anyway...
Do you want marginal intervals or a joint interval?
Would be helpful if you could post example data/summary statistics so we can see exactly what you mean.
Non-statisticians generally want a single number and don't care about confidence intervals, often because it is easier to make comparisons (even when comparisons make no sense).
Sorry! Q97 is the id variable. I forgot to recode!
Welcome to this site! Could you provide more details on your design and dataset? What is the sample size, how many categories are there for your predictor, what hypothesis would you want to test?
Almost ! It made me change dt to pt, and now my gamma variables indeed looks like gamma. The correlation is now 0.77...however, Matlab gave 0.8 exactly, so I am still one step away from goal.
I don't think you really are describing a two tailed test here.  Instead, you have a slightly more general null hypothesis than is normally the case (where you are just interested in "different" from expected), but you will still need a statistic to compare with one tail of a distribution.
@DimitriyV.Masterov, Thanks for mentioning those concepts, I think they extend the point I was making.  Couldn't, for example, a given case of autoregression with autocorrelated errors or sample selection be reframed as a problem of omitted variables?  I know I'm not teaching you anything.  I just want learners to think about how these terms are related and realize that the same statistical issue can be conceptualized in many ways.
You are sampling to get the posterior distribution of the variable, right? Maybe I'm missing something here, but it doesn't matter if the variable is categorical or not, since the autocorrelation of the MCMC gives to you how fast it will sample the whole space of the distribution, right? Btw, are you using it in regression?
Although you mention biostatistics instead of machine learning, this question appears to be the same as [having-a-job-in-data-mining-without-a-phd](http://stats.stackexchange.com/questions/27495/).
I do not believe your number of data-points to be the problem rather than your number of subjects in order to initiate an Mix.Model. Check Yao et al. 2004 on *Functional Data Analysis for Sparse Longitudinal Data*, it might be a helpful reference. I would propose using their methodology/code (PACE - they have a Matlab package) to do PCA on your data and then try modelling each PC score family independently. A bit of a long-shot but could possibly work fine as (hopefully) there will be a *slope*-like FPC that is influenced significantly by your Treatment variable.
Related to [this](http://stats.stackexchange.com/questions/68077/are-categorical-variables-standardized-differently-in-penalized-regression)
+1 , but note that the information matrix approach cannot produce asymmetric confidence limits.  For small-parameter problems (up to 3-4 parameters) *Mathematica* can graph actual contours of the confidence regions: you don't need to use this second-order approximation.
By the way, my current intuition is to multiply impute missing outcomes a large number of times. The fraction of missing information might be high, but given data are MCAR, estimates of paramters and their standard errors should be unbiased given large number of imputations.
To be nit-picky, large Pandas DataFrames actually don't print out all of their contents when invoked, as happens in R. They switch to displaying column headers along with a count of null/non-null values. Also, while I agree the syntax isn't ideal, scoping issues make it hard to eliminate the self-reference for comprehension-style filtering. It is wordier, but it's also resistant to namespace collisions if a DataFrame has extra columns at runtime you didn't expect.
I don't think that article is relevant here.  Besides you do still need to worry about N for that simplified method of statistical significance.