tttjjj/elt_bench huggingface dataset:
Thank you, i will look into it, the link you provided is a bit complex for me (sorry total noob, can only really do stats through computer programs and have no idea about the equations behind it all, i know you guys who are passionate about stats probably hate people like me). But i will try to find a simple macro or simple instructions on how to do that  (equivelence test) in spss. Thank you
*Of course* it's not representative of Australia if it's just from Sydney!  The data you have, according to your description, might--at best--be representative of people in the Sydney area with telephone numbers who respond to surveys.
@Stephane sure, but I think my problem differs in that I have some additional desired structure that I would like my distribution to reflect. Perhaps phrasing the question in terms of subsets was a bad idea since I have a vague notion of distance working for me.
Thanks for noting that. I've changed the title. I didn't intend to imply _just_, but I didn't think about the title too much. I mean perhaps _mostly_.
@P Sellaz: It appears that @B_Miner's suggestion is for a function that goes in the glm formula, as opposed to an argument to glm. I'd guess they act the same, but perhaps not.
Thanks for the suggestions @Lucas. Sorry for the slightly unclear code. It's part of a larger function that I re-wrote so that it made a bit more sense on its own. SigmaInv is only calculated once and all of the gradients are calculated before updating. This does need to be an online model for what I'm doing, so I can't use EM. I've tried a slightly different version which uses the Cholesky factorization of sigma, but it behaved a bit oddly.
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge Univ.Press, 2000. (ISBN 0521773628)
Excellent info and links, enough to get me going. And yes you are right this is for epidemiological studies :)
@Glen_b I have nothing against basic answers! Any comment is welcomed. Thanks.
"mixed model" is a biostat term for econometrics "random effect" model. Oddly, that's still relevant here :)
Thanks! $f(x)=0$ for impossible events basically resolved my issue! +1 and accepted.
No: Dunn's test preserves the ranking from a Kruskal-Wallis test, and employs a pooled variance for the pairwise tests that is appropriate to rank sums. Dunn's test does not in and of itself require Bonferroni (or any other) multiple comparison adjustment.
(+1) I would like to add that the definition of skewness in terms of the third moment is not the most accepted nowadays because it can only be applied on distributions with light tails. More modern definitions of skewness are based on quantiles, some of them can be found [here](http://www.jstor.org/stable/2987742?seq=1).
Any thoughts/pointers on this anyone?
Cam, that's not the hazard function., but rather the *survival* function. :-)
I'd even say that MLP refers to those neural networks which consist of layers instead of some weird directed acyclic graph structure.
@TenaliRaman, it's cross-validation method to avoid overfitting.
I am a bit unclear on the variance of $f(x)$. Intuitively I would have thought that for a mixture model, the variance would also depend on the means, and not just be $C=w_1C1+w_2C_2+\\dots+w_kC_k$ as your answer implies. There is a related [answer](http://math.stackexchange.com/a/195984/62449) on the math SE which is somewhat contradicting.
@AlanForsythe good points.  You know this as well as anyone.
Logistic regression *does* answer those questions. The resulting model captures which features are important to distinguish the positive and unlabeled class, e.g. your data sets.
(I've done tests, and my data doesn't seem to be remotely normal.)
So if you were writing a textbook titled _An Introduction to Probability for Scientists and Engineers,_ it would be a biased coin, while if the book were titled _Statistics: An Introductory Course_ it would be an unfair coin? I wonder what terminology the numerous _Probability and Statistics for Scientists and Engineers_ books use.
@D.T. 1) The bootstrap samples with replacement from the full data set to estimate properties of the estimate (in this case the error rates). If does not build a training set.All you do to apply it is to use all the data for fitting and then take that data to bootstrap and repeat the fitting for bootstrap data sets. Then it uses the results to do a bias adjustment of the resubstitution estimate of error rate. I see no problem with applying it. You would just change the way you do the estimation.
@Saideira By definition, the sum of a sequence of numbers is computed by tracking the sequence of its partial sums: if that sequence converges to a value, that value is the sum.  (Otherwise the sum does not exist.)  By induction, the sum of a finite number of zeros is always zero.  Thus, the sequence of partial sums of an infinite sequence of zeros is $(0,0,\\ldots,0,\\ldots)$ which (obviously) converges to $0$, QED.
@DavidR: There appears to be an extraneous plus sign in there as the dimensions of the two terms don't even match. :)
(1) One-tailed test of what parameter? The mean? Standard deviation? Variance? etc. (2) The answer for the mean is a [sample size of *one*](http://stats.stackexchange.com/a/1836/919).  With this you can construct a 99.6% confidence interval.  It will be very wide, though :-).
@Alexis, saying that it is false is too strong. When the correlation matrix is being analyzed, in essence the variables have all been standardized. Thus the variables all have variances equal to 1 & "the sum of the variances of the principal components is [still] equal to the sum of the variances of the explanatory variables". The question is still sensible even in the case of the correlation matrix.
The fact is that I want to use the term for a regression against the return in a multimodel ensemble however not as an AR term for Arima. And I was wondering  if the term could be used in this way. Thanks for the hint
Sorry, I'm having a little bit of trouble following the initial scenario you describe. As well as everything else. It's also not quite clear to me why you would need to build *two* trees and not just one... Could you please describe--in concrete terms--what your data actually looks like?
Are you sure? If you look at the derivation from equation (2.3), the authors do begin from $f(x)= X^T\\beta$ but write their matrix form as $(Y−X\\beta)$T(Y−X\\beta)$ equation (2.4). I was trying to follow their notation throughout.
This has really helped thank you so much! I'm working on adapting it for my data and am looking forward to seeing the plot output :) Thanks again!
(+1) The exchangeability assumption is also at the heart of permutation tests.
I think you'll need to spell things out a bit more. There is a lot of similar terminology in this area - latent curve model, latent growth curve model, mixed effects models, latent class growth models, growth mixture models, gaussian mixture models, latent class trajectory models, and so on. I do most of this kind of modelling in Mplus, but it looks like what lcmm wants is an equation for each of "fixed", "random", and "mixture" (but I could be wrong). I imagine that you won't be able to provide that until you have a better understanding of what you want from the model.
What are $x^{(i)}$ and $\\mu_{c^{(i)}}$ ? Vectors ? If so, what is their dimension ? What is $m$ ?
@Ali There's a more technical explanation here: http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/ If you still have questions, feel free to ask more!
you should probably include a link to your previous question on [ways to determine if pageviews are trending upward or downward](http://stats.stackexchange.com/questions/11548/what-is-the-best-way-to-determine-if-pageviews-are-trending-upward-or-downward) and clarify how this question is different.
Have you tried the delta method?
The intercept is the estimated value for sr=0 and func.group=herbivore. The coefficient of sr represents the slope for the herbivore group. The slope for the plant group is sr + sr:func.groupplant and the intercept is (Intercept) + func.groupplant.
+1 for a good idea I came across as well, but I don't have any outcome data on siblings for this particular analysis.
I shortened the data for the post- what I have are Fourier coefficients, for a fundamental frequency and two harmonics. What I've done is used those to calculate the frequency using radian values from 0 to 2*pi with steps of 0.0001 and I've been working with that data, so I actually have 36000 points for each time series.
And what do you think is pdf of discrete rv in this text?
The question was how to estimate the ratings the sixth movie would have received from the four people who did not rate it, not how to make recommendations about that movie or how to impute missing data in general.
Let me just add that Chapter I of An Intermediate Course in Probability by Allan Gut is a great reference in case you feel unsure about how to perform steps 1 and 3.
+1 for mentioning fisher's Z re: average correlation
The events are independent, I'll edit the question.
+1 Nice forensic analysis of what Matlab is doing--it illustrates more generally how one can go about checking statistical output.
One of the best and easiest options is to ask the OP (in a *comment*) what efforts they have made towards solving this problem. Not only will that focus the question and perhaps make it acceptable here, it will make us all better aware of what kind of answer will be most helpful and what level to pitch it at.  If that request (issued politely, of course) gets no response, then the question needs to be closed rather than answered.  Because I notice that @gung made that effort and the OP has logged on since then without editing the question, it is my duty as a moderator to close it now.
I don't see a need for a downvote here. I think this question gets at what is going on in multiple regression at a fundamental level, and offers an opportunity to explain something about MR that would otherwise never be discussed.
Welcome to the site! It seems that you can directly use maximum likelihood to estimate the mean (i.e., the average) if the missing is at random. You know the difference between the two missing data mechanisms, missing at random (MAR) and missing not at random (MNAR), right?
[VisTrails](http://www.vistrails.org) is one system that helps this process. (I've used only homebrew systems; common group goals are more important than a particular tool.)
That's not correct; the equation for a logistic regression is $\\ln(\\frac{p}{1-p})=\\bf X\\boldsymbol\\beta$. There is no error term.
Do you have any hypothesis about the shape of the decay? You should have one. Then there's no good reason to treat days categorically.
The variable $Y$ is a continuous and always positive variable that is associated to the $X$'s in a very complicated way. I do not think this is a classification situation, but I am merely interested in checking if the populations are different in $Y$.
@nico. The default in R is breaks="Sturges" which does not always give good results.
@chl: thanks - upvoted jms' answer as soon as i got enough reputation.
It will probably be worth looking at  the copula package
Your code doesn't work because the components of a POSIXlt object are numeric/integer, not character.  But as Shane said, you really should be using a time-series class...
@Kasper $\\sup_{\\theta \\in \\Theta}\\ldots$ means *supremum* (maximum) when $\\theta$ runs over $\\Theta$.
I don't think this is correct.
Suppose that the vast majority (99%) of people among the 1000 people have strike percentage between 5% and 15% and a handful have strike percentages higher than 25%.
Then I would argue that it is more likely that the new player we observed has a true strike percentage lower than 30%, but simply "got lucky".
I'm not sure if it is appropriate to use a centroid-based method along with cosine distance. Centroids are obviously minimizers for L_p-norms, but I'm not sure about whether this holds for cosine distance. Anyone knows if the centroid also minimizes cosine?
I will also add that likely neither option is wrong and both are quite defensible. Perhaps pick the one you are most familiar with. I would probably do PCA in this case, although I prefer FA in most cases.
Concerning the advice not to store redundant variables: this is appropriate for RDBMSes but I would like to suggest that the opposite should be encouraged for spreadsheets.  The latter are so error-prone that mechanisms to detect and correct errors are invaluable.  One of the best consists of redundant information, such as computed fields and statistical summaries.  For example, if column C is the ratio of columns A and B, then an error in a single column in any given row can be detected and usually fixed.
Ok-that was a misuse of terminology. Pearson's correlation cannot be a semimetric because of the iff condition over 2.
In general, I would be cautious about using "general" special functions like $_1F_1$ for these types of numerics. In fact, it's quite hard to find a really good implementation of $_1F_1$, for example. The more specialized functions do just fine and have received much more attention from the numerical-analysis and algorithms community.
You can use a normalizaed cross covariance function used in time series analysis - http://en.wikipedia.org/wiki/Cross-correlation#Time_series_analysis
Great point of view about combining models! Your answer was very constructive!
What about a spineplot?
Multiple regression by ordinary-least squares assumes at most that the errors, & therefore (approximately) the residuals are independent normal random variables with zero mean & common variance. No assumptions are necessary about the distribution of the predictors, which indeed may be fixed at arbitrary values by an experimenter; or, it follows, about that of the response. So are you worrying about nothing?
@Яaffael1984, I can add to Peter's an alternative terminology. Type B is a 2-way crosstable. Type A is  a 1-way table with nesting (age nested in gender). Type A is also frequently called "aggregated data" or aggregated type of tabulation. P.S. "Nesting" here is just a form of layout and not be confused with "nested experimental design"
You can search for RapidMiner text mining examples at MyExperiment:
http://www.myexperiment.org/workflows
Yes, you are right. In situations where they provide well-calibrated inference (large samples, not too many covariates, nothing with a really extreme distribution) there's at worst a small loss of efficiency estimating the standard error - the efficiency of the point estimate is of course unchanged. Compared to everything else that could be going wrong, this is usually a small price to pay.
Again, it is a very complex procedure to test if data are stationary. Basically, you would have to test for a unit root. Let's keep it simple, and say we are going to test if there is a linear trend in your moving averages over time (massive assumption, maybe it is quadratic or seasonal). These are the results out of R based on the 8 data points you posted before: http://postimg.org/image/ddpa7vzpr/ . So there is a linear trend in these data points and you would conclude not stationary. Again, this too straightforward, and test for stationary includes a lot more!
Not quite understanding.  The natural weight is 1/n for each list, e.g. 1/17 for B, 1/41 for D because each item is 1/17th (or 1/41st) of perfect.  You could achieve equal weights by combining the checklists into one big checklist with 145 items, but I somehow don't think that's what you are aiming for.  Can you write a bit more about your desired outcome?
And btw, thank you for correcting me the question. I see now that I can use LaTeX syntax :)
See a little update on being careful with making strong assumptions.
Why can't you use the value of $P(C)$ that's given?
I liked that line. If you're not surprised, you haven't learned anything... thanks again.
Yes I found it interesting that there was a write function but no read function.  What would be ideal is if there was a read function and then you could write it as a sparse matrix.
Nightingale did excellent work, but her originality is often exaggerated. Polar diagrams were used by several people before her, earlier in the 19th century.
I think you'd need more information, whether in the form of another data source or some assumptions in relation to the age distribution in order to get very far with this.
Learn about multinomial logit.  Here are two, free online books by Kenneth Train (http://elsa.berkeley.edu/books/choice.html and  http://elsa.berkeley.edu/books/choice2.html).  I think those are graduate level books.  Or just google around for "multinomial logit."
**First**: Yes. Can you think of an example? (*Hint*: Try to find an example where *every* orthogonal matrix is a set of eigenvectors). **Second** (partial): Eigenvectors are only guaranteed to be orthogonal when the underlying matrix is symmetric. If you think about that, that answers a lot of your question right away. For diagonalizable matrices an eigendecomposition takes the form $X = P D P^{-1}$.
#Model-averaging mixed models

library(nlme)
data(Orthodont, package = "nlme")
# Fit model by REML
`fm2 <- lme(distance ~ Sex*age, data = Orthodont,
random = ~ 1 | Subject, method = "REML")`
# Model selection: ranking by AICc using ML
`ms2 <- dredge(fm2, trace = TRUE, rank = "AICc", REML = FALSE)`
`(attr(ms2, "rank.call"))`
# Get the models (fitted by REML, as in the global model)
`fmList <- get.models(ms2, 1:4)`
# Because the models originate from ’dredge(..., rank = AICc, REML = FALSE)’,
# the default weights in ’model.avg’ are ML based:
`summary(model.avg(fmList))`
@MichaelChernick - it's called zero-inflated Poisson because you're basically "inflating" the probability of seeing a zero from a Poisson dist'n while maintaining the same relative probabilities of seeing a non-zero value as the Poisson has.
-1 Besides being incoherent, this answer omits the key idea so ably explained in the best answer: the *probability model of variation in the data* dictates whether regression is meaningful and determines which variable may be considered the dependent variable.
unfortunately, histogram based estimation using the plug-in formula is still biased. I guess if you have a lot of data, it shouldn't be too bad. I recommend Paninski 2003.
I'm interested in the issue with the ties, since by design there are going to be plenty of ties. There are 300-400 respondents and there are 5 answers to choose from.
I modified your list of tags, hope you agree. Before I start answering your questions, I'd like to know how many data points you have.
thanks a lot emcor!
I develop milk. If either of you run into any problems, please let me know by email (lpc at cmu dot edu). Bug reports generally get fixed in under 24 hours.
It depends on how you treat the theoretical mean. If it is random variable then you can say about probability that it falls into some interval. If it is constant, you cannot. That is the most simple explanation, which closed this issue for me personally.
Aaron is correct in the book reference, apologies for not providing it originally!
@ErelSegalHalevi if you can model your problem in the standard terms of RL, you can use any rl framework, it doesn't need to be tailored explicitly for games. Additionally, to add more value to the question, you could add the resources you have found so far.
I know how to calculate them but it's the only thing that these  exercise wants? I mean , the  exercise wants you to explain what are the three dots. So the only thing i have do to it's to find them?
The only term that I can think of that applies generally to the process of specifying a subfamily of a larger family of distributions is 'taking a special case' of a more general family. Particular kinds of special cases may have special names (such as 'taking a standardized distribution', for example).
The Gaussian kernel maps the samples to a different space based on the Gaussian RBF function. It might be the case that in this new space, different classes looks the same. I would suggest you consider feature selection, multi-class SVM such as the nice MSVMPack or try different classifiers. It is not clear from the question which approach did you take to deal with the classification of more than two classes. You may also try with more than one approach including one-to-all, one-to-one and the heirarchical model.
@MichaelChernick: Ah. My guess would be that if you turn off all of the auto-stuff in `autobox` you'd get the same answer. But one of the points of using `autobox` is that it will detect outliers and handle them as such, so the model returned would be different if there are outliers.
Correct except for step 3: there are **two** parameters fitted, so you divide by $n-2 = 1$. (Check it out: do the math!)
+1, excellent answer. I should note though, that regression model always estimates conditional expectation, so it is as good as its regressors are. So the last comment although is very good, it is not strictly necessary, since if you build regression model you must trust the regressors.
Oh wow - very nice - thanks!
For future reference: [Crossposting verbatim copies](http://math.stackexchange.com/questions/101816/a-hypothesis-question) of your question to multiple SE sites is [strongly discouraged](http://meta.stackexchange.com/questions/64068/is-cross-posting-a-question-on-multiple-stack-exchange-sites-permitted-if-the-qu). This happens commonly with new users that aren't familiar with this policy, so don't feel bad. Just, please, keep it in mind. Welcome to the site.
I know of no name for this, but it is a straightforward consequence of a simple argument: to a good approximation, especially for large $N$ (say, $N\\ge 20$), the number of times each value is drawn has a Poisson$(1)$ distribution.  This assigns probability $1/e$ to each of $0$ and $1$ and (therefore) $1-2/e$ to all other values.  Your rule of thumb replaces $e\\approx 2.72$ by $3$.  This rule actually *worsens* the approximation, especially for smaller $N$. A similar analysis predicts frequencies for drawing $M\\ne N$ samples without replacement.
@DilipSarwate. My bad Dilip. Should I change it? Because both of them are kind of similar
Thanks! This much I understand. I am particularly wondering how it is that resampling from a sample of the population helps to understand the underlying population. If we are resampling from a sample, how is it that we are learning something about the population rather than only about the sample? There seems to be a leap there which is somewhat counter-intuitive.
I think the OP is asking a slightly different question which is about using kernel density estimation for producing improved confidence intervals when the bootstrap sample is small: **"I am interested in non-parametric methods for building confidence intervals for an estimator (e.g. the mean) using few samples (e.g. 10)"**.
Have you any more specific questions about the statistical aspects of repeated-measures ANOVA?
You've got 167 variables and 35 observations.  You're problem may well have a solution, but it is likely to be a complicated one for someone who is unfamiliar with basic regression mechanics.  I'd suggest collaborating with someone with better statistical fluency, and taking the opportunity sometime to work with simpler datasets (many observations, few variables, normal distributions) to get you to a place where you'll be able to deal with more complex situations.
The link to the paper is broken, could you pls provide some information so one can find it? Thanks
With this data I am not able to estimate [B] at t=infinity, as (1) data already collected and (2) the halflives are too long even if redid experiment. I will give nls() in R a go, I was just hoping there was a way to get data as a straight line as it is *just* the mirror of a curve that does transform. Thanks a lot.