tttjjj/elt_bench huggingface dataset:
*"I had assumed the PCs I generated related to the rotated factors, not the unrotated factors"* -- I am afraid you were wrong. PCA does not care about rotating, it cares only about maximizing variance of each component and any rotation would only interfere with this goal. You can first do the PCA and then rotate your PCs, but if you just do PCA and nothing else -- there is no reason to expect any "simple structure" in your PCs.
@drknexus - if you can share your data, that would be helpful, or point to a similar dataset within R that has the same shape as your data.
"c" P(X=1)= (12)^1 . e^-12 / 1! = 7.37 x 10^-5
You can also test some interesting hypotheses by stringing out the data to be tall and thin, with a variable for which taste is represented in the current row.  Intra-subject correlation can be accounted for using the Huber-White cluster sandwich covariance matrix estimator or the cluster bootstrap.
The Wikipedia article on [stochastic processes](http://en.wikipedia.org/wiki/Stochastic_process) provides a clear, succinct, non-mathematical answer on the first line and right next to it is a picture of data that are typically thought of as resulting from a process.  Have you investigated these materials?
@Iterator I'm afraid I don't follow.  What exactly is the "wrong problem"?  Are you saying you disagree with this model of the situation?  What do you think is the right model?  And what's "cumbersome" about the null hypothesis?  (Please recall that the chi-squared test is purely an *approximation* that is justified solely by an appeal to a model like the one I started with.)
That is exactly why. For high dimensional problems (such as text) Naive Bayes tends to return probabilities either very nearly 100% or nearly 0%.
What do you mean by "doesn't have a mean"? Many questions on omitting the intercept on this site e.g. http://stats.stackexchange.com/questions/7948/when-is-it-ok-to-remove-the-intercept-in-lm/
I think the question was about the "inverse" of the log-normal, i.e. where a normal rv A leads to log-normal X = exp(A), the questioner was asking about the distribution of X = log(A), which is undefined (due to sometimes requiring the log of a negative number). There may be some results for a truncated normal, but they're likely to be messy.
I run JAGS in R via r2jags.You can check the statistics with the following R code: "jags.mcmc<-as.mcmc(jags.estimation); jags.sum<-summary(jags.mcmc); jags.sum.stat<-jags.sum$statistics". Here is an example where both Naive SE and Time Series SE are reported by JAGS: http://stats.stackexchange.com/questions/71731/using-jags-for-bayesian-parameter-estimation
Any advice concerning when to use each of these?
Could you add some more details. I've been trying to prove this but with no sucess...
I'm not quite sure what you are comparing - what are the "graphs" and how are you comparing them? A t-test compares ACROSS two groups: E.g if the groups are "men" and "women" and you are comparing their heights; it looks at differences between the groups. You have 3 groups; are you comparing across the three groups? Then why not use ANOVA rather than t-tests?
If $p=\\frac{\\sum{y}}{n}$, it's not unbiased, as you can check by working out its expectation.  Why don't you show your calculations & perhaps someone will point out the error.
I'm not sure why you expect a (highly flexible) model to do well when extrapolating to this degree. Trying to predict on values that are a couple of orders of magnitude larger than anything in your training data is kind of silly.
This sounds like it might be more of a software issue than a statistical one, is this possible? If so, this forum might not be the best place to ask such a question. Have you searched WinBUGS's manual, or some WinBUGS-specific forum?
Unless you have something unusual, working hours is better regarded as measured, not ordinal. It may be conventional to record (e.g.) integers only, but if you have say 40 hours and 20 hours for two workers, differences and ratios both make sense and you have an acceptable measurement. Age in years is a similar example: it is widely conventional to report integers, but it's a measurement too.
Describe how the cross-validation was carried out, e.g. averaged over x repeats of k-fold cross validation - what are x and k for you?  If x < 50, depending on the sample size, cross-validation is often unstable.  The most typical choice of k is 10.
(+1) although the interpretation in the OP could be read as 'the sample correlation is 0.16. Based on the p value of 0.55, we cannot say anything about the true correlation'. This would not be a contradiction.
For reference, this type of issue is specifically mentioned in section 7.0.4 in the JAGS 3.0 User Manual.
@whuber, You are right! But that's not my constraints. The order was observed in the actual data.
I highly respect what you write and you seem very knowledgable, but I disagree.  For example, when using graphs(like I did initially), each possible basket was a node and each basket was connected to the baskets it was similar to the next day. Without me creating any preference probability, I had more edges going to certain nodes(those nodes weren't right but they were somewhat close). I'm not sure but I suspect there's something that, unlike my graph, takes into account price changes to figure out probability also.
@Stat: Having seen the data, I don't think it will matter. Everything is dichotomous anyway. I'm guessing this is due to the model achieving probabilities of zero or one for certain combinations of the predictors, but I haven't had time to check this guess yet.
@gung yes multicollinearity can cause the sign of a significant coefficient to change, but the degree of multicollinearity needed to make the change is much smaller when the coefficient size is small compared to its standard deviation.
I had not seen this particular reference; it's a bit more helpful than the others in this area. Could you provide some insight into how you might do this via simulation (using histograms)?  Thanks!
An adjustable dental chair?
..cont.. also introduce artificiality.  Before I go deeper, I want to clarify I'm speaking of the idea of an "independent variable."  No variable is __truly "indepedent"__ in nature/the universe- however we must assume some if not most are "indepedent" because it's not feasible to incorporate all potential permutations and level of complexity of thus in a manner that is practical in reality, we must actually make this assumption.
so then (4yearMean - 1yrMean) is definitely nonparametric?
Thanks @whuber for the great edits.
Are all of the items (likert or yes/no) intended to measure the same underlying construct? Have you checked anything to determine if they do?
The sample mean has many nice and interesting properties but sometimes they are not the best that one can have in a particular situation. One example is cases where the support of the distribution depends on the value of the parameter. Consider $X_1, X_2, \\ldots, X_n \\sim \\mathcal{U}(0,\\theta)$, then $\\frac{1}{n} \\sum_{i=1}^{n} X_i$ is an unbiased estimator of the distribution mean $\\theta$ but it is not the UMVUE, for example, unbiased estimates based on the largest order statistic $\\frac{n+1}{n}X_{(n)}$ will have smaller variance than the sample mean.
@zelite: Also, why don't you simply plot your 40 trials across time in two colours (e.g. unperturbed in blue, and perturbed in red) and see how well they are separated? This would directly show you which time-periods are going to be predictive, and which not.
You definitely do need to study the examples, because your model does not do what you think it does.  *First* replicate the examples in the documentation, *then* put your data into the form used in the examples and re-apply the examples.
General advice: 1) plot your data several different ways, 2) calculate confidence intervals and interpret their substantive significance rather than merely focusing on whether a coefficient attains statistical significance at some arbitrary (.1/.05/.01) level.
Words like "slightly," "moderately," and "extremely" are too vague to make a definitively answerable question. Could you please tell us more about what you need to know?
The entropy is a sum of terms of the form $p\\ log(p)$.  When $p=0$ you instead use the *limiting* value (as $p$ approaches $0$ from above), which is $0$.
@user4127 Here is a [related question](http://stats.stackexchange.com/questions/4756/confidence-interval-for-bernoulli-sampling) that might be helpful.
Hi there. Does it matter which `X` precedes the particular `Y` as the last sentence in your second paragraph suggests this is not important. But if different `Y`s are correlated with different `X`s, then which `X` occurred becomes very important.
Did you manage to understand why the auto-arima function chooses the particular lag structure compared to all other possibilities? Perhaps these lags have low but comparatively higher correlation? Perhaps _exactly because_ ACF is low, the auto.arima function generated a many-lags structure in order to "compensate" for the low ACF? (in the same spirit with blindly increasing the number of regresors in a linear regression to increase the R-square, although this may create a useless mean equation with good fit?
But why do you want to do a t-test. Just simulate the process N times and calculate the difference. A better method would be to simulate the difference over time, and plot the difference with associated error
(+1) Are you sure there are not some qualifications that have gone unstated in your question? As it stands, the engineer's statement is false in so many different ways it's almost hard to know where to begin. :)
Not sure whether the [semPLS](http://cran.r-project.org/web/packages/semPLS/index.html) or [plspm](http://cran.r-project.org/web/packages/plspm/index.html) would allow mediational analysis, but it's worth to check.
What happens if you use just `+Error(subject)` in the call to `aov`?  If I remember correctly (not guaranteed) your notation for `aov` suggests that `errorType` is nested in `subject`, but the `lmer` call would suggest `subject` is nested in `errorType`.  That could be the problem.
An article that you may find interesting: http://optimalprediction.com/files/pdf/V2A5.pdf
Since changing the base of the logarithm only multiplies by a constant, it changes the scale, not the shape -- so it makes no difference at all which you choose. If you have a high proportion of values at some single value, you can't separate them by a transformation, so generally you can't do much.
"*Are goodness-of-fit tests parametric or nonparametric?*" --- you already asked it... why ask it again?
True, I was a bit fast and loose with my terminology.  Makes sense.
It's okay Karl, I upvoted for him. And he's a bit closer to 15 rep as well.
+1. The transformation to polar coordinates leads to the results I cited in a comment to the question.
@danjeharry: parallel cross-validation is really the low hanging fruit here, and I strongly suggest you explore that first.  Beyond that, I don't know a lot about speeding up SVMs specifically.  If you can find a parallel SVM algorithm, that might be a good idea too.  How many rows/columns is the data set you're using to train?
@ttnphns, you seem to understand the model.  Your proposal is to estimate $z_i$ and subtract it from both sides of the equation before doing factor analysis, right?  But $z_i$ is latent -- how do you propose to estimate $z_i$?  In particular, it seems like the estimates would depend on the factor analysis fitted values.
But there is no official term for this? Is this unheard of?
Hi Glen, I was not sure whether to add statistical bias or not.I ll try to explain my thought flow when i added that tag. If in fact we can learn something about the algorithm (it could be the parameters used or something else) from the output, then it should be(i am not sure) because the output is biased in some way. My aim to find out if there is some way to learn something about the algorithm used using the properties or this output.Have you heard of such a thing ?
Any combination of keywords is an itemset. There is no notion for disjoint itemsets; this constraint it usually not considered interesting. Usually, one is interested in maximal itemsets, i.e. such that no larger superset of keywords exist that still is frequent.
$X_1+X_2\\sim\\Gamma(2\\alpha, \\beta)$  Where is the sufficient stat?  Like the sum of $X_i$ you usually see.
Hi Whuber - wonderful references - thank you!
@shabbychef: I completely agree with you; my comment was just an amusing way to emphasize the vagueness.  The *answers* are obvious--sequential methods or standard experimental design methods, with a reference to the literature on the relative power of robust methods to handle the case of the median--but exactly *which* answer is appropriate (or best) will depend on how the numbers are being obtained and assumptions about their possible distribution.  E.g., if the numbers are in the range $[1,100]$ *and that's all you know,* I can guarantee 2% relative error with 95% confidence in 9228 draws.
It is not obvious what "inflated" variance could possibly mean here, because you have no relevant basis of comparison. Since your procedure changes the data in a way unrelated to the response, it does not make sense to reason that a decrease in variance will decrease a p-value. VIFs describe linear relationships among the explanatory variables *only*. They are important because (1) huge VIFs ($3$ is not huge) are flags for floating point roundoff errors and (2) they indicate the effects of some variables might not reliably be distinguished, *regardless* of the response variable.
Can you explain me why? I know it's a time series and to have a perfect model, you have to make sure that errors do not carry information on them. But I am not using any Box-Jenkins models or some other methods that require some assumptions. For example, I have many dummy variables in inputs so I can suspect multicollinearity but as you can see easily, all say that multicolinearity is not generally a problem for Support Vector Machines. however, how come others do not check it in the papers?
There are really three separate questions here! 1) What is the single most influential book in statistics; 2) What book should every statistician read; 3) What book have you read that you most wish you'd read much earlier. (2) and (3) probably have considerable overlap; (1) may be quite distinct.
A metacomment here is likely to seem trite but possibly worth underlining to @Tim. Wikipedia is very, very  patchy on many technical matters: sometimes utterly superb, sometimes dire. That's the nature of the beast.
Could you post your data or a sample of it? As @agstudy said, `str` should give you all the components of `glm.linear` and then you can choose which one you want to look at. Or, if you are using RStudio, type `glm.linear$` and then press TAB and you may choose from the options there. Have a look at `summary(glm.linear)` as well. If you post sample data someone might be able to help you out more.
**Hint**: are you familiar with the [Law of Total Covariance](http://en.wikipedia.org/wiki/Law_of_total_covariance)? The independence of the $X_i$ from the $Y_i$ causes the formula to simplify, so that all you really need to know that isn't entirely trivial is the expectation of $N$.
Check the update, @whuber
I don't do diagnostics of the data based only on your syntax, and nobody will. What is the size of your data set? `reps(2500)` is probably an overkill, at least for the standard errors; I think `reps(500)` is OK for most practical purposes. Efron & Tibshirani's [intro bootstrap book](http://www.amazon.com/Introduction-Bootstrap-Monographs-Statistics-Probability/dp/0412042312) has a section on the number of replicates. They have a whole chapter on regression, too, so that may be another good reference for you to look at.
@user268208 It's $z$ not $t$, because $\\sigma$ is given for the population. The statistic has a normal distribution, not a $t$-distribution.
R's [wordnet](http://cran.r-project.org/web/packages/wordnet/index.html) package, which uses the [WordNet](http://wordnet.princeton.edu/) database, might be specifically useful for your example of semantic relations.
I'm unsure, sorry, and that might depend on the reasons for your transformation. Probably the transformed values, though. You can always try both and see which fits best and which works best for your purposes.
If you have n boostrap estimates of, e.g. the median, then the 0.025 and 0.975 empirical quantiles provide a 95% confidence interval.
That's a great answer, seriously! And while explaining the underlying theory you have a nice practical explanation there too, another great answer for my reference collection. Thanks a lot!
This answer is similar enough to help you with the averaging of the points, http://stats.stackexchange.com/questions/2493/managing-error-with-gps-routes-theoretical-framework/2497#2497, it is simple to incorporate weights in that framework. I would think you would be able to use some simple heuristics to identify outliers, but that doesn't preclude you from taking a more empirical approach like Stephan suggested.
Yup; there is also "rough logic" when you just have an interval, for instance Ben is punctual if he shows up between 11:45 or 12:15, without any "punctuality score". See [this](http://en.wikipedia.org/wiki/Rough_set) for more.
1) agreed 2) before log transformation it was much more 3) if i don't add 1 logs can't be used 4) transformation into logs; coupled with regression using robust standard errors solves most of the problems apart from this skewness 5) its a small dataset of approx 700 investments; dependent variable being the investment in millions of pounds, and one of the independent variables is age (which could be 0); and a number of binary variables.
@Tomas, Good question. As I wrote above,
$$\\frac{ {\\hat \\beta}−β_{0}}{{\\rm se}(\\hat β)} $$
has a $t$-distribution. Therefore, under the null hypothesis, $\\hat β$ has a $t$-distribution that is shifted and scaled (by $β_0$ and ${\\rm se}(\\hat β)$, respectively). But, for large samples, since the t-distribution converges to the normal as the degrees of freedom increase, $\\hat β$ has a normal distribution (shifted and scaled in the same way). Does this clarify anything for you?
Please share your calculations thus far in the question. It's hard to help you until you give more disclosure on exactly what you've done.
This is a good way to look at school differences in these three variables.  I think user2110135 was looking for ways to compare how individual differences in these three scores are different across two schools. Could you explain a bit how logistic regression can help that? Many thanks!
Excellent thanks, i'll read up on that shortly.
+1 for the sorting
You can't transform a distribution with such a highly discrete component to an approximate normal distribution.
If you are working in a GLM(M) context then "chi-square" and "deviance" are synonymous  as far as I can tell (but as I pointed out "deviance" is more precise).
Thanks for the answer. Agree both come down to the fact that we do not have complete information--that is a good way of framing it. However, I'd like to see an answer that distinguishes between the two cases more. What was Poincare thinking?
@Lara Welcome to Cross Validated :) If the above answer helped, please accept so others know you got the information yuou needed.
Again thanks, can you please recommend me a refrence where i can find such inferences about different models, or is it a matter of experience? Sorry if the question sounds dumb but im a rookie in this field. Also i have asked another question about assumptions of ancova [here](http://stats.stackexchange.com/questions/44856/generalized-linear-model-as-non-parametric-ancova-vs-modern-robust-methods-and) id be greatful if u could give a hand on that too.
Yes, I would. The test will also work, if the variables are measured on more than two levels, as long as the variables can be treated as ordinal.
@Jonathan- yes, I would like to add an image, but I am too new to the site, so the spam precautions won't let me.
I read your paper, but I am not able to understand what is the meaning of getting concordance and discordance as 25.0 and how to know by this whether the model is good or not
Maybe your answer is in here: http://stats.stackexchange.com/questions/13197/differences-between-manova-and-repeated-measures-anova?rq=1
@Andre Holzner, I added your suggestion 'the top 30% items by total weight sold' to the question
@LadislavNado but the t tests assumes the data is normally distributed. Mine is not.
@Procrastinator Please have a look at [my data](http://www.4shared.com/zip/frq3olek/Data.html)
Beta is small but r is large means that the predictor correlates with other predictor(s) which is stronger.
@naught101 Updated acoordingly.
Sorry - Mean Average Precision and Average Precision.
Sorry I meant classifier. I just want to get an accurate classifier. Yeah I did 12 fold cross validation
I'm not well versed in using the lme4 package, but one way to find out if there is overdispersion when dealing with a Poisson model is to compare the residual deviance to the residual degrees of freedom. These are assumed to be the same, so if the residual deviance is greater than the residual degrees of freedom, this is an indication of overdispersion. There's also the Cameron & Trivedi test of the assumption of equidispersion, but again, I'm not sure if this can be performed by the lme4 package.
(i) *you don't equate it to zero*, as I already explained. (ii) I discussed the difference in my answer. I suggest you think carefully about what I wrote, and answer every question I have asked, in comments or in my answer that has not already been responded to.  You don't seem to be putting much thought into working with what information you've already been given.
have a look at http://snap.stanford.edu/data/#socnets
Try:
aggregate(state.x77, list(Region = state.region,  Cold = state.x77[,"Frost"] > 130)), function(x) ((x - mean(x))/sd(x)))
@FrankHarrel - please clarify:  "them" = ?
If we take a random-effects view of meta-analysis, where $\\rho$ varies randomly (e.g., among studies), we might also consider whether $\\rho$ or its Fisher-z counterpart $\\zeta = \\tanh^{-1} \\rho$ better satisfies any meta-analytic assumptions about the effect-size parameter.  For example, it's often unclear whether $\\rho$ or $\\zeta$ is more likely to be normally distributed, which some procedures assume (e.g., certain maximum-likelihood estimators and "credibility" or prediction intervals).
...The source of the tip you cite is chock-full of advice that would be anathema to a lot of knowledgeable people.  In fact it would be a useful exercise to see how many pieces of questionable advice you could find; to explain what's wrong with each; and to compare your list with others'.
@caracal I guess Siegel's book is also a good reference. I also know that Kendall's $\\tau$ has better statistical properties (but this has already been discussed in previous Q&A). But it seems to me that the question has to do with the interpretation of an ES, and although we're not informed of the sample size, it is likely to be small. Anyway, your proposed transformation should work fine and there exist transformation for converting an $r$-based measure into a Cohen's $d$ one. But as you didn't address directly the question, I wasn't sure of where you want to go. Now, I've +1 your response.
+1 Seems like this question could go on Stack Overflow (for example here's good coverage of the star schema for data warehousing http://stackoverflow.com/questions/110032/star-schema-design). But statisticians have their own data management concerns and needs so nice to see it discussed here.
Yes, certainly when I need to I output what I want into excel, or csv, or whatever, but at times, I just want a quick  C & P. It behave very oddly. . .not even giving me the option of "copy" from the "edit" drop-down.