tttjjj/elt_bench huggingface dataset:
If your response variable is *binary*, it will be distributed as a binomial. IE, *many* parts of the linear regression model described above are inappropriate. 1 of those issues is that, since the variance of a binomial is a function of the mean (mean: $p$, variance: $(p(1-p))/n)$ ), the assumption of homoscedasticity is violated. To understand these things better, it may help to read my answer here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909), although it was written in a different context.
@rose If you know that the signals tend to be missed in CBD areas, then chance is it's probably not random. There are two points here: 1) if it's time series data tracking many travellers, then there are techniques and assumptions you can apply to estimate the missing values; and 2) because it involves spatial sampling, I think you should also revise this question with more emphasis on the spatial aspect, and consult the [GIS board](http://gis.stackexchange.com/). But do not copy and paste the same question, that'd be counted as cross-posting and is generally not recommended.
Have you considered using [tag:r]? The `HoltWinters` function is in the `stats` package (auto-installed, I believe), and the `astsa` package hosts the `sarima` function. Compared to many statistical software manuals, documentation for R is phenomenal :)
Here it is -- [this question](http://stats.stackexchange.com/questions/95468/how-to-calculate-advertising-copy-wearout-using-kalman-filter-in-r). If you really want to ask that question but without the "in R" in the title you should edit your question, not repost it.
What is the meaning of $u$ and $I(u)$? Because if you can measure them you could fit $A$ and $B$ to it.
If you want to create a multivariate Gaussian with a covariance matrix $\\Gamma$, find $A$ such that $A^T A=\\Gamma$ using e.g. Cholesky decomposition.  Then if $Z$ is a vector of independent standard normals, $AZ$ has the desired covariance.
+1, it's nice to have the canonical paper. BTW, I suspect you meant *recent* work, not "reason" work.
thank you for your comments. I am not able to upvote your answer but I accept it. As you guessed, one of the control variables is state. The other is the industry.
Similar questions [this](http://stats.stackexchange.com/q/50932/3277), [this](http://stats.stackexchange.com/q/52773/3277).
It is called the [Principle of Inclusion-Exclusion](http://en.wikipedia.org/wiki/Inclusion%E2%80%93exclusion_principle#In_probability). It is an immediate consequence of the axioms of probability: a formal proof would use induction on $N$ to reduce this to the case $N=2$.
I think if you shift the focus of your question to the second part "... and provided this family is not large" you'd have a distinct question.
This is interesting, but I don't think it is workable for us. We can't just add up total time. Since some tests might speed up and others might slow down and they could cancel each other out. Also, not all are on the same order of magnitude. Some tests run in 1ms, and others take 1000ms. So a change in the 1ms test case would almost certainly be overshadowed by variance from the 1000ms testcase.
How is the new space going to have the same support as the original PBD $\\{0, \\dotsc, n\\}$ if there are fewer trials?
The actual computation? Or the derivation? For the derivation, you can find it in many linear models texts, e.g. Monahan, *A Primer on Linear Models*. For computation, refer to Algorithm AS170 published in *Applied Statistics*. You can find it online by searching for it.
@Glen_b: you are right. But then some models that are more restrictive than others.  In higher dimensions, data sparsity stakes the deck in favour of tighter stair-jackets. The arguments for a light handed approach are more compelling in 2D-3D situations.
How do you define 'density'?
And in fact, from the MLwiN manual "Sorting your data set
The most common mistake new users make when trying to fit a multilevel model to their data set is that they do not sort the data set to reflect the data’s hierarchical or nested structure. (This is an easy mistake to make.) All the examples in this manual have already been sorted into the correct structure — students within schools, in the case of the data set used in the previous chapters."
It means that in your Stochastic model for whatever you're analysing, you assume that it cannot occur twice at the same time (for a temporal Poisson process).  So if you were modelling arrivals to a queue, you are assuming that two people won't arrive at the same time.  For a more technically precise explanation, see here: http://web.mit.edu/urban_or_book/www/book/chapter2/2.12.1.html
@JamesWaters, yep, since site is not significantly different from 0, variation among sites is not that important.
Thanks, that answers the first question. What about the second question: "Would it have been a better approach to increase your significance level?" 

Just doing a quick simulation in R (keeping same effect size and power, only changing significance value) I could collect ~4.8% less data by simply choosing 97.5% significance, rather than running 2X experiments at 95% significance.

I should clarify - when I ask "Would it have been better.." I mean, could I achieve the same end-result by collecting less data.
You can try this R package - "seewave" (http://rug.mnhn.fr/seewave/)
Note that this model has a conjugate prior: the Pareto distribution.
@mpiktas Are you saying that knowledge of asymptotics will assure that one is excellent at visualizing data?  There seems to be little to connect the two.  Indeed, that's the problem with this entire question: its premise is that excellence in data mining, stats modeling, and data visualization requires "great mathematical statisticians" who have taken lots of courses.  Neither one of those criteria--being mathematical or taking courses--seems to be closely related to succeeding in such positions.
Almost _any_ (dis)similarity measure can be in its weighted version. Just insert weight into its formula.
+1, I think this is a better strategy if your goal is to maximize the grade and you have prior distribution of test grades. It seems a bit overkill if your only goal is pass the exam though.
@wnoise you get error bars on the CDF with that method, that's the point I was trying to make. If you don't want to integrate the PDF and work directly on the CDF, you can use a gaussian process on your data points, pretty much like you suggested. The gaussian process will then yield the most likely interpolant and its error bars.
In your result you have a "**+**" preceding the y^(i)--is that a typo?
A choice was made that all values over 100 and under 0 just gets cut off. These numbers are not for scientific research, but just to identify problem areas in the customers organization. And we're not even saying that a high or low number necessarily is a problem, just that your spread is higher/lower than what is usual in our reference population.
Because your two independent variables correlate with each other
@ACD in brief: $\\bar{x}_i$ (`X.mean`) has a _between_ effect, $\\tilde{x}_{ij}$'s (`X.dev`) have a _within_ effect. E.g., rich states, where $\\bar{x}_i$ is high, vote Democratic, while rich individuals, whose $\\tilde{x}_{ij}$ is high, vote Republican. If you don't decompose $x_{ij}$, you assume $\\beta_1 x_{ij}=\\beta_1(\\bar{x}_i+\\tilde{x}_{ij})=\\beta_{11}\\bar{x}_{i}+\\beta_{12}\\tilde{x}_{ij}$ with $\\beta_{11}=\\beta_{12}$. If in reality $\\beta_{11}\\ne\\beta_{12}$, your model is misspecified. If in reality $\\beta_{11}=\\beta_{12}$, decomposing $x_{ij}$ can't harm.
Hmmm, that did change things but actually the data sample size went up not down. (I did look at n.ttest source and you are right it only uses sd1) I will update my question with more details.
This argument says nothing about the bias, only the hypothesis testing aspect of the problem, which nobody questions.
You can make that an official answer & mark it as accepted, if you like.
I derived the posterior expression as the product of Normal likelihood function and Wishart prior. I sampled precision matrix from this posterior expression, but not getting the correct output. I am generating the samples from wishrnd command in matlab, by using the degree of freedom and the scale matrix of the posterior.
But even if it was not possible to collect more authors, and you have not started your experiments, you can do another thing: you can configure your alpha to compensate your lack of power. It would fix your power problem, but itself would increase the chance of obtaining false positive errors.
And remember that you can do it, ONLY and only if you have not started any experiments (not after that). If your experiments are done, this method is totally incorrect and useless.
+1 But please notice that such tests are limited in the forms of heteroscedasticity they can detect.  Examples like the one shown in my answer can slip right through, even though the heteroscedasticity is extremely strong.
And a good point it is - just clarifying for the OP or other readers.
It seems a little harsh to characterize the third approach as "silly."  I think it takes some nuanced understanding and research to interpret, though.
I definetly think that it is a feature. The ensemble produced in random forests through bagging + Ho's random subspace selection is a variance reduction method. Meaning the descrease in MSE is achieved through diminishing variance but keeping the bias present in any of the trees of the ensemble. Even so, when trying to solve regression tasks the bias is very annoying because you lose the capability to predict the "high values" and "low values" of your sample correctly.
@BabakP, I understood it such that DPY wants a scatterplot between the residuals left by regressing $Y$ by all the variables except $X$ and the residuals left by regressing $X$ by all those variables except $Y$. This what is known as _partial correlation scatterplot_ between $X$ and $Y$.
@chl Also: in terms of "measurement model" -- presuming that you meant in terms of summarising the relationship(s) with a summary statistic like an odds ratio?
Bootstrapping is a family of procedures that resample your data to estimate a parameter. For example, if you want a confidence interval for the median of your data (sample size n), you might take 5000 n-size samples (with replacement) from your data, calculate the median on each sample to create a distribution, and then calculate the 2.5 and 97.5 percentiles on that distribution to obtain the confidence interval for the median of your original sample. This, as I said in my answer, requires that you settle on a statistic.
I spotted the same issue of the curves coming out of the main plot - if you follow the comments trail the OP confirmed that they are multiple measurements.
If I understand correctly, you are manipulating item format, or equivalently item response options order. In the psychometric literature, this would call for a model handling item covariates like the Linear Logistic Test Model (LLTM); see, e.g., this review in the *Journal of Statistical Software*, [The Estimation of Item Response Models with the lmer Function from the lme4 Package in R](http://www.jstatsoft.org/v39/i12).
Closing as a [duplicate on SO](http://stackoverflow.com/q/8691966/420055).
Adding a constant is like adding a linear time trend to the model, so the two are no longer the same. Take it out so that the regression matches the FD equation and see if that fixes things. Or add t to the OLS.
60 plants in total (I should have mentionned it), so each bloc had its own 20 plants
@Carlos Gardel thanks for your answer, but I don't get the following: You are doing the log-likelihood maximization manually, but -sum(log(dt((alvsloss-par[1])/par[2],df=par[3])/par[2])) does not look like the log likelihood function of the t distribution?
Welcome to the site, are you the original poster of the question? This doesn't appear to be an answer.
As a rule of thumb, models with AIC differences $\\leq 2$ are indistinguishable. See the wikipedia article on the AIC here: http://en.wikipedia.org/wiki/Akaike_information_criterion.
Use Jonckheere-Terpstra trend test then: paper - http://www.jstor.org/stable/10.2307/2333011, R implementation - http://rss.acs.unt.edu/Rdoc/library/SAGx/html/JT.test.html
Absent additional assumptions, the statement given regarding maxima is false. For example, if the parameter space is closed *and* bounded and the likelihood function is *continuous* in the parameters, then a maximum must exist. Absent either of these additional conditions, the result need not hold. Regarding convexity, it fails even in the most simple and common of examples. :)
I would say that in your case, going form Autocracy to Democracy gives an "infinite" percentage change and a "zero" elasticity, results which, while meaningless mathematically, have a strangely intuitive flavor philosophically...
An important question here is *in what way are your distributions non-normal?* 1st, note that only the residuals need be normal, not Y & certainly not IVs (see here: [what-if-residuals-are-normally-distributed-but-y-is-not](http://stats.stackexchange.com/questions/12262/), if you need more info). But even if your residuals aren't quite normal, the Central Limit Theorem can cover for you if they aren't too far off & you have enough data. OTOH, if your response variable is non-normal in the sense that it's binary, you need Logistic regression. Etc.
I think there's something wrong with how you created your matrix, but I am not expert enough in R to figure out what it is. Maybe someone else will. 

I added the R tag to help find them
Good point, Max. Done.
what is  $\\Im_{t-1}$ and what does $D(1,\\phi^{2}_{t})$ denote?
It depends on the statistical analysis: perhaps you could amplify your question and tell us a little more about what you are trying to do? In most cases--such as estimating averages, fitting distributions, calibrations, etc--dropping extreme values (and by definition, BLQ values are at the lower extreme) introduces substantial bias.  Are you familiar with [Dennis Helsell's work on nondetects](http://www.practicalstats.com/nada/downloads_files/NADAforR_Examples.pdf)?
So if I only get a result when I statistically control for some variables, what doe that mean?
You could put $(\\theta_1,\\theta_2)=\\left(\\frac{\\psi_1}{1+\\psi_1},\\frac{\\psi_2}{1+\\psi_2}\\right)$ where $(\\psi_1,\\psi_2)$ is a bivariate Gaussian couple. You could also assign a prior for $\\theta_1$ and for the conditional law $(\\theta_2 \\mid \\theta_1)$.
Hint: observations do not have a distribution, only random variables do. Hint 2: If all observations came from the same random variable, then this random variable would produce many values "at once". But a random variable is a real-valued _function_, not a _correspondence_.
To reduce the risk of miscommunication, I edited your answer to conform to the OP's convention of specifying the standard deviation rather than the variance in the normal distribution. One subtle issue in your answer concerns the sense in which something can "converge" to $p(1-p)/n$ when $n$ itself is the index that is getting arbitrarily large!  (You might want to use the language of asymptotics instead of convergence.)
I don't want to discourage you, but usually more specific questions have better chance for answer. This seems more like "Do my job for me!" rather than "Can you give me a hand with this?".
@raegtin - I'm most familiar with CHAID, which comes up with so-called "interactions" that are often main effects masquerading as interactions because that is the only way the procedure will "let them in."  (In CHAID there can be only 1 main effect identified as such, so all other main effects get squeezed into "interaction" cells.)  But CHAID has the advantage of being able to check many many interactions. So once a few promising ones are identified, they can be incorporated into a regression or anova, with all of their lower-order components, and one can test for which ones are truly useful.
Thanks for that! Got another question lscov() seems to be able to calculate estimates when the correlation structure is known this refers to a known correlation of the y data not the x data right?
Actually, the relationship to the covariance may only apply if the nodes have Gaussian distributions--I'll have to think about it for a minute.  But @juampa's point about factorizing is a very good one: finding a way to factorize the distribution makes just about everything in RBMs easier; if you know $h$, it's trivial write down the distribution of $v$ and vice versa.  That wouldn't be so straightforward if the parameterization were different.
Yes, this is my thought about what sort of splits are reasonable, but I am unsure of whether any existing tree packages allow for this sort of constraint or whether I need to code a new one myself.
One possible way to get at the location parameter is to use the normal variance mixture representation of laplace.  This is a conditionally conjugate prior though...
The qq plot clearly is *not* straight and the histogram clearly is *not* symmetric (which is perhaps the most basic of the many criteria a normally distributed histogram must satisfy).  [Sven Hohenstein's answer](http://stats.stackexchange.com/a/52295) explains how to read the qq plot.
And in response to the letter that @MånsT linked... http://onlinelibrary.wiley.com/doi/10.1002/bimj.201100103/abstract?deniedAccessCustomisedMessage=&userIsAuthenticated=false
I'm not sure to get your point. In a linear model (and only in a linear model), you can have unbiased point estimates with biased standard errors. You will see it if you write yourself your least square or maximum likelihood equations
Yes, multiplying by -1 should work.
+1 Nice! I learned statistics long before computers were in vogue, and one idea that was drilled into us was that $$E[(X-t)^2] = E[((X-\\mu)-(t-\\mu))^2] = E[(X-\\mu)^2]+(t-\\mu)^2$$ which allowed for the computation of variance by finding the sum of the squares of the deviations from any convenient point $t$ and then adjusting for the bias. Here of course, this identity gives a simple proof of the result that $g(t)$ has minimum value at $t=\\mu$ without the necessity of derivatives etc.
Completely agreed. And that is exactly what I told my friend when he asked. Although I have to admit it is an extremely simple question to ask - and one I think many students can relate to - that leads to some complex ideas in combinatorics and probability.
Since you have made a new question I think you should accept mbq's answer and link to your new question.
+1 I think Python is probably an enjoyable way to go.  There are some very recent packages in R for text mining, but if one has more computational than statistical expertise and interests, Python would be my recommendation.
To echo @Macro and clarify a subtle point, there's a difference between the case where $\\mu=4000$ has been estimated from data (either yours or an independent experiment) and where it is given as a theoretical value.  In the first case it's subject to observational error which has to be accounted for in the testing.  In the second case there's no observational error.  More generally, you need to stipulate a probability model for how the data depend on $\\mu$.  For instance, if you are monitoring bacterial count over time, you might need a regression model.
@vqv Yes.  Now that we have a list of the half million or so racks with no words, it's straightforward (by changing the last two lines of code) to compute the chance of each rack (without replacement) and obtain the hypergeometric result.  The exact answer equals 349870667877/80678106432000 = **0.43366%**.  With N=100K trials your SE is 0.021%, so your answer should have come between 0.38% and 0.49% (two-sided 99% CI).  I'm so glad our answers agree!
I've updated your title to get at the question I think you are asking; feel free to edit if I have misconstrued
...recapitulating prior work by Finch and Chater (1992, 1994), Schütze (1993), and others. HAL, LSA, and other prior art work by generating a similarity measure for words by computing their contextual similarity.  (This is Shephard's 'second order' similarity: 'first order' similarity is when word a occurs near word b; 'second order' similarity is that word a occurs near the same sorts of words as word b does).
Thanks Simon, that seems easy
"*possible mean of sample from first stratum is 2, 3 or 4 and from second stratum is 1 or 5*" -- unless I misunderstood something, both those statements aren't correct.
I think scaling results in the variable that behaves as a random variable. T1/2  ie square root of T here should  imply that we take a square-root of sample size. May be someone is trying to implement variance-stabilization.
@luciano: Calculating, or rather, estimating this correlation could indeed prove to be tricky! It is however easy to include in the model used in the simulation: when the results for the 10 students in the classroom are generated, they are generated in a correlated way. The two variables that have correlation $\\rho$ are the result $X_i$ of student $i$ and the result $X_j$ of student $j$ for all combinations of $i=1,\\ldots,10$ and $j=1,\\ldots,10$. Bascially this means that if one student in the classroom performs well (compared to the school mean), others are more likely to also perform well.
Even in the univariate case I struggle to imagine an automatic procedure - there's trend (stochastic or deterministic), possible transformations, seasonality (multiplicative or additive) to think about, & I find that to get to a model I use a lot of prior knowledge about what would be sensible for what a particular series represents in reality.  Still, the proof of the pudding is in the eating - I really just wanted to say not to forget to do out-of-sample comparisons with simple techniques- so good luck with it.
Google announced the image search recently: http://www.youtube.com/watch?v=t99BfDnBZcI I don't know if it has an API yet, but that may be an alternative.
Doesn't this depend on whether or not the same $ k $ parameters were used to fit the $ n $ subjects?
@user11920 - if you read a little further down in the section of the wikipedia article, it says "... $k = \\sum_{i=1}^Nk_i$, which makes $X$, or $k$ as the notation may be, the total number of events observed.
Also thank you very much for the hints. :) I think I have enough sample size then (regardless of power calculations).
One (rightfully obscure) measure of dispersion of data $x_1,x_2,\\ldots,x_n$ is the maximum absolute deviation from the mean, $r = \\max_i\\{|x_i-\\overline{x}|\\}$ where $\\overline{x}$ is their mean. All your data are "valid," by definition, when $r\\le .05\\overline{x}$. That addresses your first question: yes, there is a statistical measure corresponding to this "validity" criterion.
This appears to be [routine book-work](http://stats.stackexchange.com/tags/self-study/info) (please read the discussion at the link). As such you should add the `self-study` tag, show your working, discuss more explicitly where you ran in to trouble, and be more explicit about where your difficulties are.
Sorry, don't have time to give you a more complete answer right now, but you could start by taking a look at the documentation of the `metan` command.
A question or scale that shows great ceiling effect is psychometrically bad one, obviously, because its discriminating power is so poor on the upper edge. You can't expect to make fine analysis out of rough instrument. Your decision to cut into 2 or 3 categories in such a situation is right, for me.
Dear Patrick @user1188407 , thanks a lot you have been very kind in providing such answer. Unfortunately my case probably fits in what you wrote in the last sentences...so I would need a R code example to understand how to treat my data. Indeed if you look at the design of my experiment described in a previous post http://stackoverflow.com/questions/12182373/plot-of-a-linear-regression-with-interactions you can see that I have a variable measured 4 time (i.e. the velocity measured in 4 conditions)