tttjjj/elt_bench huggingface dataset:
Hi soakley - forgive me but I'm not entirely sure I understand your question. The time series signals are acoustic data. I'm wondering how I could approximate them to find the reference distribution, as the data are non-stationary. What I imagine would be the case is that there would be few time series with large overlaps (unless by "equivalent" you mean also having the same mean), and many time series with little overlap.
I would entirely agree with Andrej. This is one of those rare cases where principal components will probably do a better job than factor analysis.
There are many non-parametric statistics. But it is very unclear what you are looking for.
Which coefficient?  There is one in wide use (Variance Inflation Factor, VIF) and another competitor (Condition Index, $\\kappa_i$ that is not so widely used.  Or are you talking about the Pearson correlations among your regressors?
Is your 'judge giving a different score' just an example, or is it the real question? There are methods for assessing inter-rater agreement.
@Dail: Your edit is a little confusing. (a) 0.60 is 60% not 0.60%, but presentationally I personally would not use percentages here. (b) Use `log(A/B)`or `log(A)-log(B)` but not `log(A)/log(B)`.
Have you tried doing MC simulations to see the how distribution looks like?
"*I hope it's ok to put this on stats AND stackoverflow*" -- It's *not* okay, as the [help](http://stats.stackexchange.com/help/on-topic) indicates: "*Please note, however, that cross-posting is not encouraged on SE sites. Choose one best location to post your question. Later, if it proves better suited on another site, it can be migrated.*". So pick one, delete the other.
Isn't this somehow related to joint probability? But I would suppose that WinBUGS recognizes that the two models are not *"jointed"* in any way and will not *"joint"* them... :)
@jbowman I like 10000 for confidence intervals.  The talk about Cauchy was a red herring because the numerator and denominator are both nonnegative.  it is possible that a distribution like a gamma might be a better choice than normal.  I guess I would like to see the fit to the normal and the gamma for both A and B.  Bootstrapping is a good suggestion.  What bootstrap CI did to OP use?  Was it just the percentile method.  If so I would recommend to try at least one of the higher order bootstrap CIs.
My answer assumes that you're doing typical cognitive science work where you have repeated measurement within each of multiple participants and therefore need to model participants as a random effect. If you somehow don't need to specify random effects, then use gam() from the mgcv package (http://cran.r-project.org/web/packages/mgcv/index.html).
Because this construction has nothing to do with "confidence" *per se*, the objective is to establish some convention for describing the shape and relative size of the points. Using $1.96$ sort of works (for three variables): it contains about $72$% of the probability of the trivariate normal distribution. But as the number of variables increases this method produces ellipses that are far too small. For instance, with $10$ variables it will contain only $4.6$% of the probability; using $4.28$ instead of $1.96$ in this case will contain $95$% of the probability.
Thank you very much for the algebra. I will code up in R and will contact you for further help.
You seem to assume that the $x_i$ are not random: I cannot find any place where you either justify or relax the assumption that "$\\{x_i\\}_{i=1}^N$ is fixed in your experiment." Wouldn't it be simpler (and more accurate) just to compute the mean and variance of $Z = XY$ and apply the CLT directly to it? Incidentally, closed-form analytical solutions for the distribution of $r_N$ can be found when the $x_i$ and $y_i$ have Normal distributions ($|XY|$ then has a Bessel(0) distribution).
please can you help me to know how to use MODMED macro . because i am unabke to understand through its documentation.
plz help
Well, you can obviously use Bayesian approaches for various things, so why would they not be usable for learning? Also I recommend looking at clustering *NOT* from a learning perspective. It is not about learning (as in "recall") but about exploration. Many clustering methods will not "learn" anything, they are nothing but a (useful) statistical method. Don't follow the AI view, which is very biased towards machine learning.
You are asking the validity of pooled OLS estimation in panel data  model. When the individual or city has unobserved effect that is correlated with the observed covariates, the pooled ols will be biased and inconsistent. If you don't have panel structure, you probably have to find instruments for the endogenous variables.
Googling?  You need to get some good books and spend a great deal of time studying them.  Your desire to take the ratio of two logs points out that you need to start with algebra.  Regarding references about change measures see http://biostat.mc.vanderbilt.edu/ManuscriptChecklist
Go to Frequences procedure. There is button Statistics where you can request the 25th and the 90th percentiles
We used the first book as a textbook; and one of the advantages is to teach you statistics by example. It is very good. And the second one is what I am reading and it is in detail to explain how statistics work. But for both books, there was one disadvantages: statistic software like R or others are not involved in the books.
An F-test of what?  If an ANOVA, how many levels, variables, interactions etc?  If it's just two levels they don't report anything different.
I think you mean *dependent variables* when you say "independent variables". A dependent variable is a response or outcome, whereas an independent variable is an explanatory or predictor variable.
Do you need the `plot` call?  Since `plot=T` is default on `hist`, I think you can just use `hist(x1, breaks=seq(0,max(x1), length.out=7))`.  Or am I missing something?
As for the flaw, see my comment towards the bottom above:  The possibility of an empty confidence interval surely provides evidence of a flaw in the frequentist methodology.  It seems you have to fudge here to say to the client, "Well, the confidence interval is useless in this case, but we *know* it's between $Z-a$ and $Y$..."
Well, with $n = p$, the surface is not yet fixed. So it is *obvious* that the sample size is too small. With $n = p + 1$, the plane is fixed (unique, determined), and it goes through *all* points. But there is still no indication about the uncertainty related to noise 
(uncertainty) of the data points. IMHO this is almost as bad as not being able to fix the plane. Each new data point now helps to estimate this uncertainty (and refine the estimated plane). Overdetermined: there are more data points than are needed to mathematically define the plane (the ones used for estimating the uncertainty)
+1 A good, concise answer addressing all the major points I'd thought to address and more.  Regarding packages in R, a Google search of CRAN, for [poisson regression temporal] turns up several packages.  The `surveillance` package may have the functionality desired.  This kind of modeling is not uncommon in ecological studies, so it's probably best to find a good package in the ecological nooks of CRAN.
Could you, please, invest some time in formatting your question? Poorly worded questions are unlikely to attract attention (anywhere) and collect good responses (here), although @Emre generously tried to decipher the above and pointed to alternative (Matlab) solution.
What an odd question! It depends entirely on your purpose in carrying out the regression, as common sense should tell you. But in general I tend to write/talk *more* about why expected effects weren't found than why expected ones were.
This question is similar to the [new one][1] from the same user.


  [1]: http://stats.stackexchange.com/questions/66709/confusion-related-to-predictive-distribution-of-gaussian-processes/68897#68897
It isn't necessarily too localized if the answers (which haven't appeared in over two years now) aren't. Skills that are currently in high demand may not *only* be *currently* in high demand. I imagine answers that are less localized would be more popular anyway (were they to appear).
... in other words: kernels circumvent the explicit calculation of the transformed feature vector. (very nice summary of the pro-nonlinear-model points :-) )
I would like to select a variety(samplename) for further study based on the characteristics. For isntance, I would like to select those plants which have a good height and earlength (higher the value, the better).
+1 for reminding people of assumptions.
Of course, with p-values, most people interpret them incorrectly even when the assumptions are met, but that's a different issue.
Welcome to the site, @NasiaAbbas. Note that we treat homework-type question differently (see more on our [help page](http://stats.stackexchange.com/help/on-topic)). Instead of just answering them outright, we provide hints to help you work through the ideas on your own. To do that, we need to know what you've done thus far & where you're stuck.
It's not clear to me whether your focus is on ordinal variables as (1) responses or outcomes (2) predictors or explanatory variables (3) either. (2) is easiest as you just compare predictions for different representations of the ordinal variables. (1) and (3) are difficult because you have to compare quite different kinds of models with quite different kinds of prediction. However, even if you treat ordinal responses as continuous that makes your life easier until you have to work out what a prediction of 4.5678 means when your categories are 1,...,5. (Or 1,...4!).
What clustering algorithm(s) do you have in mind? Most need the "distance" to be a [metric](http://en.wikipedia.org/wiki/Metric_space) but not all of them need it to be a Euclidean metric. Some distances, such as path distances along directed graphs, do not even require the "distance" to be symmetric (which is one of the axiomatic properties of a metric).
+1 for the link, did not know that characteristic function of F distribution was so complicated.
Sorry, got my notation messed up. Fixed now.
Hi Erik, I agree with your example, and this is what I had in mind, but I was thinking on more advance algorithms than just linear classifiers. For example Kernel-based SVM or Random Forest. Since this can learn any distribution, I believe that in the general case, using more labels has disruptive effect the binary problem.
+1 for pointing out this is not a statistical exercise.  At best, PCA can describe relationships within a particular data set and, conceivably, simplify the data by identifying near-collinearities.  It is not apparent how it can inform us about how to *rank* the experts.
Great reference - was going to recommend it myself.  Not a good pedagogical reference for someone who doesn't know matrix algebra though.  The real challenge comes from actually working out $\\Sigma$.  A real pain.
1. The covariance matrix shows the degree to which the parameters are intertwined. So there is one value for every pair of parameters that you ask nonlinear regression to fit.
2. Look on the Range tab to ask Prism to make a table of the XY coordinates of the curve, with plus/minus values for the confidence or prediction bands. 
3. For tech support with Prism, email support@graphpad.com  Use this forum for statistical questions, not tech support.
It is possible to invent an _ad hoc_ proximity measure that is greatly amplified if the two regions are adjacent and downright attenuated if they are not. Play with, for example, exponentiation.
+1 A breathtaking harbinger of the follies of the coming century; "GIGO before its time."
(+1) I like the balance of this answer. I'm not sure I quite understand what was intended by the very last sentence.
This is not directly relevant to the question, but note how much faster the bootstrap would be on the binned/collapsed/aggregate data than on the full micro data.
Sorry, but this is not what I'm looking for. I'm currently using traminer, and functions like seqefsub() are too limited for the datasets that I'm working with.
"the majority of undergraduate students never fully grasp the concept of a probability distribution" +1
One can now add [sklearn.ensemble](http://scikit-learn.org/stable/modules/ensemble.html) from the Python scikit-learn toolbox.
I think you have to be more specific about what you are looking for: entropy is after all as "statistical" a measure as variance etc. so the maximum entropy distribution maximises entropy is a perfectly good statistical description.  So it seems to me you have to go outside statistics to come up with a "justification"
Can you expand on your answer a little. please? Can you say anything about its properties? Could you give a complete reference? Out of curiosity, did you find this after you posted? (It's perfectly valid to post a question you already have an answer to, and to answer your own question, so it doesn't matter either way, but my curiosity is biting me.)
@wootscootinboogie if you click on the little question mark at the top right of the box in which you type a question or answer, it takes you [here](http://stats.stackexchange.com/editing-help) which gives you at least some basic information about how to make your posts look more like what you want. (If you haven't used LaTeX before, there's a lot of online guides available.)
You'll need to decide what you mean by "significant change", and doing so should help determine the analytic approach too.  For example, just by eyeballing the data it's obvious that Person 1 now agrees with 35 more than they used to and disagrees with 11 more than they used to.  If all you need is evidence that some people change their minds you've got it with that one data point.  So how much difference is the threshold before you say it is material? (I would rather use the word 'material' to preserve 'significant' for talking about the evidence when you get to a hypothesis test).
@NBrouwer: yes, it seems you actually misinterpretted it. It is nasty to transform count data and it is even more nasty to transform continuous data to count data and try to fit Poisson on it! That's what I tried to explain to you. Don't do it. Simply log-transform your *continous* data as you need. This is very common in statistics, no need to worry about it.
It does indicate that: just don't confuse "significantly improved" with "greatly improved". Someone here suggested "discernible" would have been a better choice than "significant" as a statistical term: so think "the fit of Model 2 is discernibly improved with respect to Model 1".
try `arrayInd(which.max(ABcor),.dim=dim(ABcor))` and `? which.max` and `? arrayInd`.
@whuber; thank you for the edit
Thanks for this, however don't you think this is a drastic approach that leads to quite some loss in data? For example, I have tested and implementation LDA on my dataset and in most cases LDA agrees that each feature in a document is assigned to one topic. So if I would replace all features with just this one topic, is the idea of LDA - each document is a mixture of topics - gone?
Interestingly, This can be used to get an approximation to the digamma function.
Note that R's svg device sometimes gives you raster when you don't expect it.
Are you looking for code? Or are you asking which algorithm is best (because that is a difficult answer)? Basically any supervised learning algorithm will give you an answer
So how would you define $X(0)$ and $X(1)$?  In your case, you either need the fifteen initial values, and then the rest can be built, or you need to specify the distribution that $X(i)$ are drawn from, for $i<15$ *and* whether the model allows the lags to be brought in as and when they are available, i.e. is $X(t-15) = 0$ if $t<15$?
@J.M., or [Huff's *How to Lie With Statistics*](http://www.amazon.co.uk/dp/0140136290)
Your question is unclear. Convert to the probability of what? Squaring would make the number fall into the range $[0,1]$ but that doesn't make it the probability of anything to do with cross-correlation. What are you really trying to achieve?
@Benjamin. Change median to geometric median. You mean to tell me that no one has ever implemented K-medians algorithm?
Are all of these "words" compose of the same set of letters? Because then you might consider taking as a distance the smallest number of letter swaps between two "words" as the distance.
(+1) It would make a fairly challenging homework question :-).  Do you need a general answer or could you (perhaps) focus your attention on specific values of $N$ or $n$?  For instance, it may be possible to develop good approximations when $n$ is much larger than $N$; different approximations would be needed in other cases.
@user55405 `We are trying to integrate over ` $y$ `but have ` $x$ `in the expression.`  That was a typo on my part: I have now fixed it. The second integral on the first displayed equation is with respect to $x$, not $y$, and the second integral is the result of making a change of variables: substituting $\\sigma x + \\mu$ for $y$ in the first integral.
Although overall much of this answer looks worthwhile, it contains some misinformation that could confuse readers. The account of logistic regression in the first paragraph sounds like a description of a log-like transformation of the dependent variable followed by linear regression: it is not logistic regression. The interpretation of the coefficients is not quite right, either. A more important problem with "LPMs" is that when data are near the extremes, they likely exhibit asymmetric distributions of the residuals, which is an important violation of the iid assumption of regression.
Could you rephrase your request? I couldn't guess what you wanted to say. Are you asking for how to use Taylor expansion to approximate geometric means in terms of arithmetic means (and some other quantities)?
[latent class models](http://en.wikipedia.org/wiki/Latent_class_model)?
@Manish For instance in the conditional $X_1\\vert X_2 \\sim N(\\rho X_2,1-\\rho^2)$. This is reflected in `simx[i] = rnorm(1,rho*simy[i-1],sqrt(1-rho^2))` at the $i$th iteration. Note that `simx[i]` is based on the previous value `simy[i-1]` as specified in the steps of the algorithm.
Ahh of course, you are exactly right. I was trying to find something like in `elasticnet` where the path could be plotted by iteration, but this makes sense in the case of logistic regression. Thank you!
I may recommend you to look at Venn diagram like the one at the bottom [here](http://stats.stackexchange.com/a/73876/3277) to visually grasp the difference between part and partial correlations in regression.
@StéphaneLaurent I don't think so. I mean with 3 groups, you would often do an ANOVA first and if all pairwise tests allow you to reject equality you would naturally conclude that blue is between red and green. If the groups have the same size, you might even reach the right conclusion by comparing *p*-values but this is all a roundabout way to talk about effect sizes, which seem to be the real substantive question here.
I would have thought that stratified sampling used to be applied directly when building the model, as discussed here: [Using Random Forest to Learn Imbalanced Data](http://www.stat.berkeley.edu/tech-reports/666.pdf) (and on a [related thread here](http://stats.stackexchange.com/q/10787/930)). I would also consider a train/test sample (for optimizing RF parameters during CV) + another validation sample (to get a more reliable estimate of predictive accuracy). For selecting # predictors, you may rely on a permutation-based measure of variable importance.
Well put @Epigrad.  I would add one point though.  Automated algorithms become very attractive when your problem becomes large.  They may be the only feasible way of doing model selection in some cases.  People are now analysing huge data sets with 1000s of potential variables and millions of observations.  How is the subject matter's expertise at 1000-dimensional intuition?  And what you will find is that even if you do it manually (i.e. with an analyst), they will likely end up creating some short-cut rules for choosing variables.  The hard part is really coding up those choices.
From Wikipedia "Claude E. Shannon first used the word bit in his seminal 1948 paper. He attributed its origin to John W. Tukey"
"Animals typically fare better, once they discover that either A or B gives a higher payout on average". I don't think humans would do worse given access to the same amount of emperical data. A single quiz show contestant, however, plays the game _once_, not _n_ times.
Thank you so much, that is exactly what I was looking for! I was still interested in `anova()` because of the problem with `Anova()` described [here](http://www.mail-archive.com/r-help@r-project.org/msg80580.html). But your last suggestion works just as well and is simpler. (Minor thing: I think the last 2 lines are each missing 1 closing parenthesis, and it should read `Error(id/(IVw1*IVw2))`)
@Jerad you have to interpret $a$ as $X_0$ rather than as $X_1$, that's all.  Did you notice the condition "$n\\ge 0$" at the very beginning of the question?
@OMG: If n is large enough then PCA with test data == PCA without test data, and it doesn't matter. However, you'd need to show that this is the case, and for that again you need to do the proper calculation (PCA for each of the surrogate models). The one thing that  genuinely  changes is that you may actually be able to get a statistically meaningful model comparison: you can afford the nested validation/double splitting.
I think we are missing some items of information: where are the dependent variables in the above equation if $\\Delta Y_t=Y_t-Y_{t-1}$?
I very much like "parsimony is your enemy".
It sounds like a simple random intercept model would answer both of your questions. It will give you a intra-subject correlation coefficient and allow you to test the difference in test score across times with a standard error that has been adjusted for the within-subject correlation. Especially because you expect test forms to be positively correlated, you should not just compare the mean of time 10:00 to the mean of 15:00 because your p-value will be too small. Try searching for "random intercept" or "random effect" -- there are lots of examples worked out for R.
It is also a bit sad that some of the later chapters are missing and/or under developed - for example there is no chapter on regression, but a draft unpublished manuscript was available with some fascinating insights into "measurement error" regressions.  Some very cool stuff on time series though.
I'm not sure what the real reason is, but a bootstrap sample is typically the same size as your original sample, so this behavior seems perfectly in line with what I'd expect from something claiming to take bootstrap samples.
Regression involves fitting a linear model to real data which includes some random variation. Are you really interested in fitting regression lines or just ploting lines like your graphs seem to indicate? There is no point to fitting regression to data without a noise component.
Each student has a number of minutes when they are active (0-60).  No intensity is being included
I'm not sure how many of these will have the scope of "marketing" research, but I would also suggest the OP check out ICPSR. It is also good to know the quality of the sampling and data record keeping, which I imagine could be haphazard if you get data from a proprietary, non-education related source.
"Equal" means the CDF is $F(x) = 1$ for $x\\ge \\mu$, $F(x) = 0$ for $x\\lt \\mu$ while "uniform" means $F(x) = (x-\\alpha)/\\theta$ for $x \\in [\\alpha, \\alpha+\\theta]$.  You *define* "perfect uniformity" in the first sense whereas the standard statistical sense is the second.