tttjjj/elt_bench huggingface dataset:
The relative power depends on what the alternative is. With this sample size, the easiest way to determine the relative power is to run a simulation study, under alternatives you believe are plausible. [This paper](http://www.springerlink.com/content/u6282626q6384h65/fulltext.pdf) gives examples.
Thanks @James! What if my samples size for depth is different for each individual and each species?
Ah, I think I might understand this better after some thought!  So the regression is done against the x series.  Then the residuals will possibly auto-correlate if there is some sequential structure in the main series that isn't explained by the x series.  Those errors are modelled using ARIMA.  So its equivalent to doing ARIMA on the main series along with a regression on the explanatory variable.
You are correct for nonsingular countable spaces, but it would be good to take a little care with the generality of your first statement.  After all, many sample spaces are infinite and many nonempty events have zero probability.  They provide counterexamples: such events may have no outcome in common, yet they are independent.
Student's t-test can be used to provide a p value that can then be used in a Fisherian signifcance test (the p value is the level of significance) or in a Neyman-pearsonian hypothesis test (if the p value is less than the preset alpha then the result is 'significant'). The differemce is in what is done with the t-test result rather than which school of thought the t-test comes from (although Gossett's approach had much more in common with Fisher than with N-P).
Added an example, thank a lot!
What do you mean by "2 component is blank"? Empty columns? Could you please report the exact procedure you followed through SPSS point-and-click interface (e.g., filtering eigenvalues above 1) or your syntax file, as well as your sample size?
I agree with everything you said up until the last sentence.  Much of statistics is well grounded in mathematical theory both in probablity and in statistical inference.
How do you define a transition matrix? It seems that when you are in state 3 there is a probability of transiting to no state at all.
You have an underdetermined problem: there are more unknowns $t_i$ than there are data to estimate them.  To obtain a solution, you must therefore *add* criteria.  For instance, you might want the $t_i$ to look like a realization of a particular stochastic process; or maybe you just want to find the solution with the smallest possible size (as the sum of squares of the estimated $t_i$).  Regardless, you *have* to be more specific about what "as accurately as possible" means.
Ahh.  That answers the question.  However, it leads on to the next one - which one is preferred?
thanks for this explanation
Welcome to the site, @CHarma. If you have a question please click on the gray "ASK QUESTION" in the upper right corner, rather than including it in your answer.
This looks like a [routine textbook style question](http://stats.stackexchange.com/tags/self-study/info). Tell me (you can work it out) - what's the value of $\\bar x$? of $x_i-\\bar x$ (for the two possible cases)? of $\\bar y$ in terms of $\\bar y_1$ and $\\bar y_0$?
@Rebecca It is not entirely clear whether one would loose power. You gain power from within subject designs when the within subject variability is small relative to the between subject variability. When there is little between subject variability the pairing might reduce the power because of the halving of sample size (although I'm not sure). If you want to design an experiment for maximal efficiency then you need ot have estimates of the various variances and effect sizes.
If you have negative values, your data aren't lognormal. It may be that some modification is feasible (e.g. a mixture of lognormal and something else, or a lognormal location-mixture of, say, normals, or a shifted lognormal, or ...), but the lognormal itself isn't possible. But if the original values are lognormal, logged values would be fitted by a normal, which handles negatives. Can you more explicitly show what you mean by 'logged increments'? I can see two ways to interpret it and only one (the less likely one) makes a lot of sense for this, I think.
@ronaldo2: Can't you just change the x-coordinate to 1?
Thanks a lot. I was dumb not to look at the source code. It mentions this clearly...
nice quote - and you can see the inherent "pessimism" in stats folk, for she speaks of "false discoveries" and not of "missed discoveries" (which are just as important).
If you don't know the covariance, presumably you would need to estimate it, yes. It won't actually be chi-square any more though.
From my experience as a personality psychologist: I do not trust correlations with n < 80, better is a n of 100-120. In this region, correlations get stable (this is of course only a rule of thumb and certainly depends on the magnitude of the correlation). The p value is a bad guidance, as in small samples the CIs are very huge. E.g., the CI for r = .34 (n = 35) goes from .008 to .60, which is "no association" to "a strong association". Furthermore, r is rather suceptible to outliers, which is even more serious in small samples.
@jp This is an ongoing important problem, so the elapsed time between the question and answer is of little consequence.  Thank you for replying!
@ mpiktas, I'm testing forex prices time series.
I'd agree (In my surprisal) - I think its that $\\pi$ pops up in many mathematical analysis.  Just a note you can write $\\pi$ by with Latex commands as $\\text{\\pi}$ enclosed within $ signs.  I use the wiki page to get the syntax http://en.wikibooks.org/wiki/LaTeX/Mathematics . Another trick is to "right click" on an equation you see on this site, and select "show source" to get the commands that were used.
@Jim Finally, the `Hmisc` library is not needed here?...
I realized that the answer to this lies in the area of multi-level models, and both aggregation and disaggregation are discussed in that literature, for example, here: http://www.thepsychologist.org.uk/archive/archive_home.cfm?volumeID=25&editionID=217&ArticleID=2138
maybe `standardized` ?
A quantile normal plot only looks at normality. The evidence for homoskedasticity in the first plot is visual
Re: "Mathematical/statistical niceties like invariance have nothing to do with a real problem with real data" - it does have to do with a real problem with real data when your $p$-values, and therefore your statistical inference (and therefore your "real world" decision about the importance of a predictor), can depend on something as arbitrary as the decision to center your predictors.
I always like reading your answers.
I do not see any difference and will merge them both. (thanks @JMS and @Jeromy)
Gaël, thanks so much for the detailed exploration of this. That's exactly what I needed.

I see it's problematic to divide the data into areas of different correlation. I'd been envisaging something like your chart with the cutoff at "6" and couldn't think of a better way.

I've now found a time on page figure below which around 75% of pages fall under an acceptable ratio of related search terms. The number of pages flagged in this way is manageable.

Regarding the distribution of visit durations, I'd love to get my hands on that, though it seems analytics tools don't provide this as standard.
All the cites on that page (Bos; MacKay/p.29) are to Bayesian concepts of likelihood (post ~ lik x prior). There's a connection, but it's not the marginal likelihood in the sense of Fisher and Likelihood theory. [see comment on Berger paper here: http://stats.stackexchange.com/questions/1045/is-there-a-radical-difference-in-how-bayesian-and-frequentist-approaches-treat-nu/1051#1051 ]
+1, this is a really nice, simple solution for this situation. One small suggestion I would make is to use *averages* instead of sums in step #3. This won't really change anything, but puts the total back into the original scale, which can make the interpretation of the value easier.
great answer - many thanks
@higgsss I've reworded slightly (removing reference to time), added a little detail and a link. I may add some more discussion later. If you have any  specific suggestions, I'd be interested in improving my answer further.
+1 for nice R code.
I see two *very different* questions here. Your second question is handled in any competent text on linear regression theory, but it is not the same as you first question.
I wish I had a good reference for that, but briefly any measure based solely on ranks such as $c$ (AUROC) cannot give enough credit to extreme predictions that are "correct".  Brier, and even more so the logarithmic scoring rule (log likelihood) give such credit.  This is also an explanation why comparing two $c$-indexes is not competitive with other approaches power-wise.
Are you interested in studying possible attrition effect or do you only want to assess the overall trend?
I think that would be appropriate - you would be essentially adjusting your weighting scheme so you fall into situation #1.
I think I figured out the issue (after much pain):

If any of the item columns contains less than two question responses, then the library will throw the above error.  The solution was to drop any columns (items) that didn't meet that criteria.
The link to the paper doesn't work for me. Should it be http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.55.1675 ?
I wonder about the sense in which you mean "error." Eg, a Poisson(1) variate will typically have values between 0 and 4, whereas a Poisson(100) variate will typically have values between 80 and 120 or so. Obviously the latter has larger errors, but *on a relative basis* the former has the larger ones (from -100% to +300% or so, versus -20% to +20%). As far as the second part of your question goes, it is too broad to be answered because there are too many issues to address. Please search our site for keywords including [tag:poisson], [tag:generalized-linear-model], and [tag:change-point].
Could you write down the model and then point out which variables are which? I suspect you are trying to estimate structural VAR. If so, then you would need to use instrumental variable estimation and then correlation between different time periods only matters in instrument selection.
To me that seems like a meaningless concept. I agree with gung's answer.
No, you need to have (up to) 5% in the lower tail. If the alternative was ">5" you'd need (up to) 5% in the upper tail. If the alternative was "$\\neq$5" you'd need (up to) 5% in total between the two tails. You have the first kind of problem. You need to compute a value for $C$ which has no more than 5% of the probability at or below it. (Sorry I keep using $C$ instead of the $k$ in your question -- simple lack of attention to your notation on my part.)
@Chan, you appear to be asking about "post-hoc" power analysis - the problem is that since you only have $n=5$, your estimate of $.2$ is not a very good one. Are you suggesting doing the power analysis supposing that $.2$ is the true value of the Spearman's $\\rho$?
The "M" in EM stands for Maximization ... of likelihood. To write down a likelihood we need a pdf. EM is a way of finding MLEs in the presence of 'unobservables' in some sense (which are filled in in the E-step). That is, to use EM you need an explicit model.
@mpiktas: you're probably referring to apps like [SoundHound](http://www.soundhound.com/) or [Shazam](http://www.shazam.com/). There is [a whitepaper](http://www.ee.columbia.edu/%7Edpwe/papers/Wang03-shazam.pdf) on how Shazam works, although it does not go too much into details. Although I'm not sure that's what the OP needs, it may be a good starting point.
Most people would rather do it the other way around, keeping more of the data available for model-building.  But it's not a matter of "correct" or "incorrect."  Maybe you could say what is the purpose of the cross-validation--your own knowledge?  Making a case for someone else?  Explanation?  Prediction?
No, that is not a general principle: @user603 is just assuming the outliers are not "extreme." They might or might not be outliers after the transformation.  What can be said, though, is that if (say) $n$ values were drawn independently and randomly according to a $\\chi^2$ distribution and an $n+1$st much larger value were thrown in, then--upon applying this probability transformation--as $n$ gets "large" it is highly likely (but not completely certain) that the $n+1$ resulting values would exhibit no apparent outliers.  ("Large" means bigger than $2$ or so in this case. :-)
@Emre can you advice some good example in whatever language?
First: notice that the code you're referring to is only used for large df (df>81+p). In the test case you give, it's just generating random normals using the code in the preceding block. Second: the code matches the Wikipedia page; `df-(0:n-1))` is the same as `df-1+(1:n))`. If the line read `df-(1:n)` then that would correspond to $\\chi^2(n-i)$ like you claim.
This question should not be closed. It is not about SPSS really; the requested procedure can't be done in *any* program.
Could you tell us what kind of data (scientific field or measurement technique) you are looking at?
If, in a final analysis, you are using an aggregate measurement that has imprecision between measurements (less significant for weight, but consider blood pressure), you MUST account for that level of variation in your analysis. Inverse variance weighting is a common approach to this.
You've already computed the maximum likelihood estimate of the transition probabilities and now you want to compute the log-likelihood of what exactly?
Make it    myxts<-xts(rnorm(31),as.Date("2008-12-15")+(0:30)*7)
"the expectation" is not a statistic at all: $\\bar X$ is a sample mean, not an expectation. Do you know the definition of a sufficient statistic, and any ways to check for sufficiency? You may find [this discussion](http://en.wikipedia.org/wiki/Sufficient_statistic#Uniform_distribution_.28with_two_parameters.29) of the more general case of some help. (The short answer is '$\\bar X$ is not sufficient'.)
If $\\rho = 0$, we are _sure_ that our estimate $E(Y|X=\\tau) = \\rho \\frac{\\sigma_{y}}{\\sigma_{x}}(\\tau -\\mu_{x})+\\mu_{y}$ is the same as if we had never observed that $X$ has value $\\tau$, and we have no confidence that the estimate is _any_ better.
"Adjusting (partialling out, holding constant, removing the influence of, accounting for) would mean estimating the group difference in the DV by Factor 1 if each group had the same mean age." I don't agree with this. They might often teach it this way, but this type of language it what leads to a lot of problems with ANCOVA/regression. It would be an amazing tool if it did this, but in fact its more like making it so that both groups have no age and everything related to age is annihilated in all factors. It morphs the other independent variables as well, and is not simply matching ages.
I am not sure what you are doing. I have never heard of using BIC on particular subjects. It would usually be used to compare the two models for all 10 subjects. A model fit to one subject would be perfect fit (but meaningless). Also, 3 parameters is rather a lot for 10 subjects.
Can you provide a minimally reproducible example? IE, some data, & the algorithm? (You may have a coding error.)
I can't speak authoritatively about the particular things you tried, so won't try to (very interested to read an answer from someone who can!). As general comment though, they probably weren't suitable because they didn't match what you were imagining as describing 'similar' curves. I'd again suggest the paper I've linked to, it does a really nice job of explaining this.
I am referring to published comparisons across thousands of series. As Editor-in-Chief of the International Journal of Forecasting, I think that I have some idea about how to evaluate forecasts.
The multcomp package approach gave this error: `Error in as.data.frame.default(x[[i]], optional = TRUE, stringsAsFactors = stringsAsFactors) : 
  cannot coerce class 'c("confint.glht", "glht")' into a data.frame`
@Stéphane's first formula is what's sometimes called the generalized likelihood ratio.
> "*There may potentially be an infinite number of those values, but each is distinct and there's no grey area in between*" -- it's actually perfectly possible to have a discrete distribution with distinct values, and *yet at the same time*, for any two distinct values you pick, always have more values between them ('grey area' in a sense). They don't come up all that often in practice, but it's perfectly possible for them to come up for real; indeed I can think of two distinct (if related) examples that can easily arise.
Hi, I can't still find a solution. I have read some docs about DBSCAN and have some questions about. How to find minimum distance with R? As my data are three dimensional longitude, latitude and temperature, which "distance" should I use? whicn dimension is related to that distance? temperature?  Is there a method to determine the minimum number of points for a cluster? Searching thorugh Google I could not find an R example for using dbscan in a dataset similar to mine, do you know any website with such kind of examples? So I can read and try to adapt to my case. Thanks again
Assigning numerical scores to categories is what's happening in the LBL-Test, but then the variable still can take on only finitely many values. Continuous vars can take on uncountably infinitely many values. Spearman's $\\rho$ is Pearson's correlation for ranks. To derive the distribution of the test statistic, you need unique ranks, i.e., zero probability of ties (equal values, hence ranks). For this you need continuity. With only a few categories, many ties are guaranteed, there are different ways to assign ranks (mid-ranks, ...), and the distribution of the test statistic is not available.
@Zen No, see my update. This is really close to the Bayesian interpretation.
Thank you I will read a bit about this
All I want to say is that numbers within each line are always more or less similar.
See also (http://stats.stackexchange.com/questions/27323/cluster-analysis-of-boolean-vectors-in-r) for similar question.
Do you observe a site more then once? Is Trees the number of trees at this site? Is water the measurement of the water-quality?
linear models with mixture of categorical and continuous variables can be referred to as regression models but the method of solving via OLS is usual referred to as the analysis of covariance.
Thanks for your reply. I realise that the n of those sets are large (> 250), could I assume that they are normal and than just using method that I mentioned?
Not exactly what you are looking for, but it may help you to read my answer here: [Dfference between generalized linear models generalized linear mixed models in SPSS](http://stats.stackexchange.com/questions/32419//32421#32421).
@Zach I thought there was an example of using `rfe` with SVM. I have to check then.
@NickCox Well, the conclusions there were 1) to use vce(robust) which is Stata-specific and 2) that nbreg was robust to violations of assumptions.
but then how to choose eta, and what doe sthis then mean statistically? i.e. how to form confidence intervals for the median from this result?
This is similar to the way [support vector machines](http://en.wikipedia.org/wiki/Support_vector_machine) work.
Is [this](http://en.wikipedia.org/wiki/Hotelling's_T-squared_distribution#Hotelling.27s_two-sample_T-squared_statistic) what you're looking for?
It depends on if those parameters enter the discrepancy statistic. In your case, you probably don't need to deal with them.  Uncertainty in the hyperparameters will be incorporated into uncertainty in $\\theta$, so you'll be integrating over all the parameters.  The level 0.05 is "standard", following frequentist tradition, but don't get too hung up on $p_b$.  It's often better to perform graphical checks since the motivation is discovering ways to improve the model not hypothesis testing.
It would be good to write him to enquire, and tell him thanks for the ride to the airport he gave me a few minutes ago.  I'll comment above on the other questions.
@cbeleites, Not now. I have to dig, and to study the differences myself, 'cause I'm interested myself.
How long are these series going to be, & how many of them will you have? Will they vary in length for different people?
Yes, averaging your data is only appropriate when each date series is supposed to represent the same data, and (you are happy to assume that) they only differ in that each has different noise.
Trees are invariant to any monotonic tranformation of predictors, including standardisation. See this very closely related thread http://stats.stackexchange.com/questions/51523/log-transformation-in-cart-analysis/51534#51534
Thanks for the feedback @whuber. Hmm. I think my point was that both methods provide evidence against the skulls having a true proportion of 50/50. Let me think some more and I'll try to update my response.
Agreed that lines help differentiate separate sets; I've demonstrated this point in my answer.
As humans we are extraordinarily inclined to find patterns, even ones that are not there. I think it is quite plausible to obtain a scatter plot like the one we have here with just two independent RVs, one of them skewed. I have no proof of that, and I have no alternative analysis to offer - other than one that says there is little or no relationship. Yes, it is possible that bimodality is present. If the process could be observed further, we could see what happens. I just think  we need to be cautious and aware of our inclination to react to plausibly spurious patterns.
I updated my answer to clarify these issues, @LinearREgression.
(although stability seems like a secondary issue here).

2. Given your description of what you expect in terms of arguments and opinions, representing the essays as a bag-of-words is not a good idea in this context. Infact topic model might itself not be a good tool for this. 

I would suggest that you pick a few key-words that you're interested (like race,food,dorm etc) and try to analyze the sentiment of the sentence in which the word occurs. For e.g. have a look at [here](http://text-processing.com/demo/sentiment/) for a demo.
Good point--I'm going to have to watch out when I use `R` for such calculations!  This sounds like another line item for the next update of the [R Inferno](http://www.burns-stat.com/pages/Tutor/R_inferno.pdf).
perhaps some questions to get you started on analyzing the data. why did you choose to administer the survey? what would you like the survey to help you accomplish?
$C$ alone cannot determine the countable sequence of $a$'s. It is up to you to stipulate what $a$ is. If you want to define $a(x)=\\binom{n}{x}$ for $0\\le x\\le n$ and $a(x)=0$ otherwise, that's fine. In that case $C(\\theta)=(1+\\theta)^n.$ In general, when $C(\\theta)$ (which depends on $a$) *exists*, it is possible to define $f_\\theta$. That's the only restriction (apart from the $a$ being non-negative).
Have you heard of _effective sample size_? This is and indicator that tells you how many iid simulations are equivalent to your n simulations from an MCMC algorithm.
Equating modeling building with ANOVA may mean that you should spend some time in background reading, especially related to how to model continuous variables without assuming linearity.
Ana, please post a sample of your data. You can capture seasonality using the S(easonal)ARMA (look at the `astsa` R package) or P(eriodic)ARMA (as covered in the `partsm` R package) class of models. Complex (multiresolution) seasonality can be captured using, for example, Hyndman's `tsbats()` function. Forecasting daily time series is not something that there is a lot of literature on, but I will try to post a short literature survey if time permits.
There is usually a lot of overfitting, so your question is very good.
Could you clarify the sense in which this parabolic blue-dot curve is a "prior"? It does not appear to represent a probability distribution in any conventional way.
Are these supposed to be residuals from a fit of a time-series? That is often the context where the term "white noise" is used.