tttjjj/elt_bench huggingface dataset:
... so I think the theme here that there is no way to use the scale command within the call to lm and pass predict the Z values I want and have everything turn out smoothy in one step the way I might hope.
I guess I win the award of the more useless question
Hey, thanks. I was told that the hausman test is rather restrictive i.e. that running it will yield a false positive result in most cases. I got quite a few categorial variables. The problem with the correlation matrix is that i can only observe the pairwise correlation which does not neccesary mean that the correlation is zero in every group.
No, k-means does not require uncorrelated variables. It, however, assumes that _clusters_ in the data are convex and more or less spherical.
Hi Bonnie, could you provide more details about your design? For example, you haven't explained whether you are running any type of factorial design, nor do we know the `IV`s and any relationships between them.
@Ejs - I'm afraid there's no short answer to your last question.  You're getting into what it means to assess relationships when using statistical control - a fabulously intricate topic worthy of extensive study.  You're also probably getting into the topic of variable selection, which is a big one as well.  Imho the best source for the committed student of these topics is Pedhazur's http://www.amazon.com/Multiple-regression-behavioral-research-Pedhazur/product-reviews/0030728312
@Ben, it can depend on both how you parameterize your variables (in my example switching the 0/1 to 1/0 for either variable would change the interpretation) and what questions you are trying to answer and what assumptions you are willing to make.
Quite welcome! I was surprised that it was so recent myself.
1. I do not understand what you mean by "matching". It is sufficient that both sets have been generated by a binomial distribution with maybe differing parameters p1,p2. 2. In case of a binomial distribution (conversion-rate), what is "one particular data point" ?
@MichaelMayer, Gauss-Markov says that the OLS estimator is the "best linear unbiased estimator" (BLUE). So, yes, this means that the estimator is linear. However, one of the assumptions (see classical linear regression model assumptions) of Gauss-Markov is that the model is also linear (in parameters). If you add the assumption that the disturbance term is normally distributed conditional on the regressors, then OLS achieves the Cramer-Rao lower bound and is BUE (best unbiased estimator). That is, it will be the most efficient even among nonlinear estimators
@Luca Because the parametric ANOVA test requires that the groups each have normal distributions with the same variance checking the assumption can sometimes be important.  Basically the F test for homoscedasticity looks at the ratio of sample variances to test whether or not the ratio of population variances differ significantly from 1. The test depends heavily on the normality assumption.  So when normality fails robust tests of the equality of variance have been devised.
Thank you! Why does the non-normality hurt the prediction? Does it ever "help" the prediction? How does a "robust regression" method help in this case? Thank you again!
Then concatenating is pooling all subjects together. Okay your interpretation of the model sounds right.  Then you are interpeting a significant regression parameter as the same as a significant correlation which should be okay since it is just between X and Y.
Could you post a short table showing what a sample of your data look like (what's in columns, what's in rows)?
That definitely works and it's what I've been doing. I just wanted to know if there was a one step solution instead.
$E[X_t]/t$ is not a statistic. $X_t/t$ *might* be one, but its expectation isn't.
If the OP did as you suggest, replacing the Euclidean distance by a time warping measure, then k-means would no longer be admissible since the centroid would not necessarily be the optimal cluster center.
Nice you gave the link to the help guide for G*Power (+1)!
I think you might want to ask a more targeted question since it's kind of vague right now. But the broad area you're talking about sounds like [factor analysis](http://en.wikipedia.org/wiki/Factor_analysis).
@Marco,The correlation is straightforward to calculate because the test statistic is so simple: it's a linear combination of normal variables.  (This is because we assume the variance is known.)  Alternatively, you can think of the second statistic as being a sum of two independent random variables: the first one, $z_1$, plus the change created by the additional data, $z_1 - z_2$. In more complicated cases the correlation might be quite difficult to calculate: that's one reason this somewhat idealized situation is used to motivate the sequential tests!
It looks to me that the hold out estimate is very noisy, so choosing a model is likely to lead to over-fitting and I suggested.  Try looking at the area under the ROC curve, or the cross-entropy error instead, you may find that is less noisy and gives a better picture of which models are better.
Regarding using the KS test for testing normality, the Shapiro-Wilk test has better power compared to KS used for this purpose.  However, Shapiro-Wilk does not test for distributional similarity; an alternative is the Cramer-von Mises test.  Although I don't know if it has better power than the two-sample KS test, I would not be surprised if it does:  KS has very low power for small sample sizes.
@gung: the 5 behaviors are independent of each other, but ecologically speaking, it would make sense to analyze them together, since they are different states of the same thing.
@onestop this is a duplicate bcse we are stumped
I've tried to google for it and found this: http://stackoverflow.com/questions/887533/how-do-20-questions-ai-algorithms-work
The problem with looking at courses studied is that courses with the same name at different universities can vary widely. Knowing that someone has studied 'Advanced Algorithms' tells me very little about what they actually know, even less about whether they can apply that knowledge in their job, and less still about whether they can effectively explain what they're doing to someone with a non-technical background.
I think that to get good advice you will need to describe your model for us.
The correct result is $ 0.02105321$ which uses the value of the scale parameter $\\beta$. You can check this by running `qnst(0.95,−0.0012827,0.009891722,3.34331)`.
$E(x-\\mu)\\epsilon=E(x\\epsilon-\\mu e)=Ex\\epsilon-E\\mu \\epsilon=Ex\\epsilon-\\mu E\\epsilon=Ex\\epsilon$, where I've used $\\mu:=Ex$ and the equality follows from zero mean of $\\epsilon$
So what does the pdf look like then? As for fitting by moments, is the fit 'robust' and 'stable' as one increases the number of moments used (4, 5, 6, 7 or 8 etc), or is it all over the place?
What is the question? If this a follow-up on the older post you linked to, I would suggest you post your three current replies as a single one there. Otherwise, try to reformulate your post as a question and let users the possibility to answer. You can keep your collection of answers as a single reply so that people can vote. If it is not possible, this thread will be closed as it doesn't follow Stack Exchange Question & Answer posting guidelines.
Just a technical point but if your response is binary then it is performing classification not regression.
Story seems inconsistent:  "WalMart for a particular state" then "particular WalMart". I guess that is not important, but perhaps tidy it up.
@caracal Sure.  I edited my response to add in some details.
I think SSC = Sample Size Calculation.
Have you searched our site? Some very closely related posts appear at http://stats.stackexchange.com/questions/22501/is-there-an-intuitive-interpretation-of-ata and http://stats.stackexchange.com/questions/1447/r-squared-i-have-never-fully-grasped-the-interpretation/1448#1448.  The first one is about $A'A$, which is the sum-of-squares matrix, while the second is about $R^2$.
This is a reasonable point, but just b/c the data were centered prior to creating $x^2$ doesn't mean you can be "absolutely certain that the extremum is at $x=0$". Saying that *now* is equivalent to having said "the extremum is at $x=\\bar{x}$" *before*. In either case you are betting the unbiasedness of your model on your ability to specify the x-value of the extremum w/ infinite precision. The difference b/t Type I & Type III tests is also a potentially interesting addition, but nb, they would only differ if $x$ & $x^2$ are correlated, ie, if centering had *not* occurred.
Are you aware of R? [link](http://www.r-project.org/)
(1) What you do: start from a population of labeled and unlabeled examples; draw labeled examples at random; draw unlabeled examples at random. What they do: start from a population of unlabled examples for which $y$ is know; draw a sample; among those for which $y=1$ in that sample, choose some at random that you label; provide only $(x,s)$.
@PeterFlom ARE relates to the ratio of the second derivatives ("curvature") of the power curves at the null, as sample size goes to infinity. It's possible for a power curve to have zero second derivative there. In practice, small to moderate-size samples, the two sample t does kind of okay at the Cauchy if you don't mind your significance levels being a lot lower than the nominal values.
Andrew, Thanks for the reply. One more thing I wanted to discuss that as I got a equation: category=-0.46(iplen)+0.15(date_time)+0.64 from nnet output. now can I compare this with my regression equation that I can create after doing linear regression as follow: lm(category~iplen+date_time) and then linear regression equation will be something like: category=ß1(iplen)+ß2(date_time)+ß0
**100%.** That is a serious answer (and one of the infinitely many correct ones) intended to reveal the need for a clearer context for the question: (objective) probability statements are nonsensical without a description of what experiment is being performed and what constitutes an independent replication of it.  Would you care to clarify these things?
Hug it out guys.
*"My coding scheme basically says that I would change this into 3 binary parameters: red/notred, blue/notblue and green/notgreen. Assuming that only these 3 colors exist in the world."* Correct. But when fitting the model, you'll only need two of the three colors. The one that didn't get in is the reference category, it will be captured by the intercept.
This is [cross posted](http://math.stackexchange.com/questions/492226/convergence-of-em-algorithm-with-continuous-hidden-variables) on math.stackexchange.com.
No. This misses the point with this response variable. Simpson's index ranges from 1 when only one kind is present to 1/$k$ when $k$ kinds are equally common, but $k$ is not equivalent information. It can make much sense to model $k$ directly (e.g. number of species seen in ecology), but that's not the same problem at all.
I don't know Python, sorry. What estimates of the coefficients did you get?
Chi-square does not treat one variable as DV and the other as IV. Both are treated equally
Can you clarify your question further? It would be best if you could explain it in terms of variables and their dependencies.
Even if the X and Y are not uniformly distributed, you should still be able to use this approach (assuming you know their actual distributions). You'll just need to integrate over the domain instead of just computing the relative area (which is basically integrating a constant function anyway).
It looks likes a mixed logistic regression mostly because it is one... :) It was the first plot that popped into my mind really though! I'll give something purely R-ish in a second answer.
@learner Yes the edges are GLM interaction terms, but are not significant at a multiple comparison threshold, i.e. we have N nodes and compute Nx(N-1)/2 interaction terms.
Note that the first one will always be between 0 and 1 and the second one will always be between 0 and b.
+1, this is a really informative answer to a pretty simple & straightforward question. I especially appreciate the links to resources for more info. NB the link to Terry Therneau's online tutorial is dead; [this](http://mayoresearch.mayo.edu/mayo/research/biostat/upload/survival.pdf) may be a live link to the same material.
A trend is a trend is a trend\\ But the question is, will it bend?\\
Will it alter its course\\Through some unforeseen force\\And come to a premature end? Alexander Cairncross
+ I'm using an optimization algorithm to find the best inputs and parameters. I think your PDF method is bootstrap and it's simpler approach to this problem.
Thanks. I completed my answer related to question 2).
For short text scans like this it would probably be better just to use the "quotation" mode in the question editor. Please consider editing to do this. In the interim, I've embedded the images. Cheers.
I like your answer but I'm still a bit confused. Wikipedia gives a definition of relative error as 'the absolute error divided by the magnitude of the exact value'. It also says 'RE is often used to compare approximations of numbers of widely differing size'. Given this, it seems that the formula I provided would be workable for the purpose of making error comparable across datasets. Could you maybe exemplify why deviation from the central point is important here?
Great, thank you very much..
So i don't need to do a loop right? it takes it automatically?
And yes, b_par is a constant...
As a biological scientist, I find myself muttering this to myself during a lot of seminars...
@Karol, I'm not sure what the function `typeof` is doing but it seems to always call factors "integers" and so does not preclude my solution. Based on the output you're seeing, I'm sure what I pointed out was the culprit.
@CagkanToptas, I don't use MATLAB so probably can't give you useful feedback in a timely manner. MATLAB does have a [stepwise regression module](http://www.mathworks.com/help/stats/stepwisefit.html), perhaps you can find out if they provide the source code?
Indeed nonparametric and semiparametric statistics are of interest to most statistician. Though theoretical statements, like normality and equal variance, for t-test and ANOVA promise an upper hand over nonparametric test. In practice parametric statistic faces a lot of problems.
Allie, that's a good question.  Take two unrelated variables $x_1$ and $x_2$, of comparable variances, and form $y = x_1 + x_2$.  Now $y$ is strongly related to each of $x_1$ and $x_2$ without $x_1$ and $x_2$ having any relation at all.
@Cam.Davidson.Pilon I disagree that a Beta can capture "most interesting distributions." For example, no Beta distribution is bimodal, which is the scenario in the data considered by Liu in the paper I mentioned.
With a correlation of zero, what would your colleague claim?  That in about 0% of cases the movements would be aligned?  That's hardly plausible, is it?
I would think this problem is too data specific for there to be any rule of thumb.
On the contrary it is perfectly laudable to test your model on totally new data. We should be encouraging that surely? From his demeanour I'm guessing he already done CV.
Tip for posting comments: surround equations etc with $ (dollar signs). Anyhow, looks like you're able to get the confidence intervals just fine. (Although, you need to use t-values.) So the problem's with the second part?
Yes, -1 corresponds to 0 and 1 corresponds to 1.
Brilliant -thanks for the formatting, it looks much better.
@whuber (...) What we want to assess is the *agreement beyond the data*, not the relationship in the data (I'm quoting Carstensen 2010 p. 8-9).
As per my response above to @dpott197 I am going to use R's fitdistr functionality to try several distributions against my data. I will try both the exponential and Weibull distributions to see how they fit. Thanks for the suggestion.
I don't understand how you can ask, "Given the results of a number of such comparisons, how should one pick the optimal vector of design variables?" and then "how should I design the experiment?"  In other words, it sounds as if you want variable selection to take place both after and before you collect data.  Am I missing something?
Over what probability distribution do you intend to form an average, David?
By SUR, I meant seemingly unrelated regressions as you said.  I guess I just meant is whether there is a formula with delta method.  I am hesistant about SUR because if I am not normally doing my regressions with SUR why would I want to now?
Are you *really* using only 10 test instances? That doesn't make a lot of sense.
See [this answer](http://stats.stackexchange.com/a/51117/1909) for a comparison of the trace and determinant of the covariance matrix as two measures for total variance.
What have you done to try to address the problem? What you're asking implies you don't know what *t*, *p*, or the test are. It's hard to tell exactly where you're confused. Please expand your question.
*My intuition is that it would be pretty difficult not to notice outliers that are distorting your results* but unfortunately, the fact is that it ain't so. Also look at the example i provide in my answer.
Where have you been introduced to the Hommel Hochberg corrections, if you don't mind my asking? I have never seen a paper put together by the two of them. They each have their own methods along with some work done with others (i.e. Benjiamini-Hochberg) but I have not seen them together. Maybe you meant them separately?
I can look up a citation, but this is easy enough to to test empirically.  F from an ANOVA with two groups is exactly equaly to t^2 and the p-values will be exactly the same.  The only reason it wouldn't be equivalent in the case of unequal variances is if you apply a correction.  Otherwise, they are the same.
Talk to the hand, cause the [quasi-random sequence](https://en.wikipedia.org/wiki/Quasirandom) ain't listening.
(as an aside, I'm not saying you don't have bias, you're just not looking at it correctly)
In principle and in practice linear quantile regression could mean predictors other than straight x, e.g. cubic splines or fractional polynomials in x.
Are you *only* interested in packages for this in R? (Because if so, this question is off-topic for CV; you may want to read our [FAQ](http://stats.stackexchange.com/faq).) Or are you also interested in issues related to predicting probabilities in general? Do you have archival data on cases where there is valid info on your predictors & whether or not they ended up defaulting? Are you mostly interested in the probability of defaulting yes or no (a logistic reg type problem), or the time until default (a survival analysis type problem)?
+1 for the links and technical explanation.
You should do that for each variable, as you do not know for sure that they're equally distributed
LOL! You can predict anything if you are vague enough about it and don't specify a time frame.  Perhaps you could make your question more precise and quantitative to make it possible to answer more meaningfully?
(+1) Sometimes I have seen epidemiologists call any independent variable of interest "exposure" (e.g. "exposure to smoking cigarettes"). But good catch, you definitely should not use the rate as the dependent variable.
I've changed the wording.
(+1) It seems my response came after yours. As your response highlights somewhat a different perspective (cross-platform, algorithms), I feel like our responses are not so redundant, but I can remove mine without any prob.
I have to say this advice does not answer the question at all. 1) Their is nothing in the question that suggests the OP is dealing with spatial or temporal data. 2) How would aggregating the data help to identify any meaningful relationships (it uses less information than the original units!)
One possible explanation may be that in one formula the intercept or even the variance parameter is counted and in the other formula it isn't. I don't know for sure that this is the case, but often when formulas differ by 1 in every place, it's because the variable ($K$ in this case) isn't defined exactly the same way.
(+1). Although if I add to two uncorrelated regressors $X$ and $Z$ the product $T := X \\cdot Z$, then $T$ is correlated with both $X$ and $Z$. So adding interactions is often implying a certain amount of collinearity. That is what makes "main effects" somewhat unhandy to interprete.
See http://en.wikipedia.org/wiki/Method_of_moments_(statistics) and  http://en.wikipedia.org/wiki/Generalized_method_of_moments
@peuhp the resource you link is studying priors for *variance*, not precision ($\\tau$). $\\tau = 1/var$. Using `dgamma` prior for $\\tau$ is [perfectly OK](http://www.unc.edu/courses/2010fall/ecol/563/001/docs/lectures/lecture14.htm#precision) and will hardly become obsolete :)
Basically I want to evaluate all the variables relative to average (the average of all the variables). The coefficients from the regression are relative to the omitted variable. So when I subtract the mean of all the coefficients (including the omitted variable's coefficient of 0) from each coefficient value, the adjusted values will now average 0, and each coefficient value can be viewed as the distance from average.
if you use the "debug" function you might could trace exactly where the problem occurs.
I understand that you need Type III sum, but this (http://www.stats.ox.ac.uk/pub/MASS3/Exegeses.pdf) article is a good read. It illustrates some interesting points.
Thanks very much Rob for your timely answer (on a public holiday in Aus, no less)!
Well, if you notice that the sample proportion is a kind of average, [actually](http://en.wikipedia.org/wiki/Law_of_large_numbers) there is, but that might seem a little circular in the case of the weak law. (On the other hand that definition doesn't rely on the frequency definition of probability.)
Can you make spine plots in SPSS?
Can you please rephrase your question or add some details? I'm quite familiar with R, `lm` and linear models more generally, but it's not at all clear what, exactly, you want. Can you give an example or something to clarify? Is this for some subject?
I was going to answer the same thing. I would fit a distribution and use the lower bound of the confidence interval for the ranking. Unfortunately, for example a t-distribution parametrized on the sample mean and variance. However, I think you need a minimum number of votes for each item.
-- I need to consider the "balance between the costs of the two types of error, and between the probabilities of the two types of error at your sample size." Could you explain what this means? I I lack an intuitive understanding of how the probabilities relate to the real world. There is approximately a 1/25 chance (no Yates') of getting the REQUESTS/OFFERS combination in this case - why should this be a weak indication of bias? Why not 1/100?
That ratio is used for the same reason it comes up in the univariate $t$ (which is that the variance of the standard $t$ is $\\frac{\\nu}{\\nu-2}$. And no, it's $>$ not $\\geq$ (what's $\\frac{\\nu}{\\nu-2}$ when $\\nu=2$?).
Thanks.  I would like to point out that the Wilson interval is discussed in the article that @Joris referenced.
I think I figured it out. The dependent variable had some missing values for various $x$.
**Crosspost**: http://math.stackexchange.com/q/178084/7003
I deleted the other posts. Also this is different because I am asking about plannned comparisons, not posthoc comparisons. Also, I am asking about right adjustment question 2 is about an alternative...
(1) What are you testing the groups for a difference on? (2) When you say the DV is "the proportion individuals feel looked at" do you mean the fraction of time for each individual or the count of the people who feel looked at divided by the total number of people? (Either way, you're very likely to have heteroskedasticity, and ANOVA may not be appropriate for data bounded between 0 and 1).
In fact, I wanted to know how to find the HDRlevelVal value. Is there a function in the package which do the job, or is there any package which have a function which do the job. Thank you.