tttjjj/elt_bench huggingface dataset:
I'm afraid I don't get this at all, because I cannot understand what a "common variance" would be. Since you say you understand this conceptually, would you mind explaining what the common variance actually is and how you think it would be "analyzed as a separate variable"? BTW, another method on your referenced page (#6) is discussed here at http://stats.stackexchange.com/questions/46185/question-on-how-to-normalize-regression-coefficient.
@Peter Ellis: thanks for the comment! I've edited.
@Rookie Glad to hear you found my answer helpful. If you need any help at a later stage of your forecasting project, make sure to let us know. Would be happy to help. Cheers and best of luck!
I am so happy you pointed this simple thing out to me!! Yes my model was actually linear so I changed it to use lm and I have the CIs. I am still a little confused about what the se.fit actually is..
I agree with @Anony-Mousse. Dunn index requires only distance matrix as input. But note that there are also some cluster validity indices that require original data (for example those that compute cluster centers).
Thanks it looks like the Duan et al. is equivalent tobit II model with uncorrelated error terms. Could you provide more details on how to calculate the confidence intervals of the marginal effects? I'm using R and don't have access to STATA.
take a look at this paper: http://www.faculty.ucr.edu/~taelee/paper/baggingleeyang.pdf
+1 An illustration of this procedure appears at the very bottom of http://www.quantdec.com/envstats/notes/class_05/distributions.htm; it is preceded by extensive explanations of CDFs.
In the rest of the package, the BF is followed by the proportional error estimate on the Bayes factor, that's probably the 2nd number.
A BF of 65 against the null doesn't seem implausible.
(+1) Nice catch! It is limited to the US, isn't it?
I think the problem statement is wrong. There is a "$1 - $" missing. It should be $1-F_i(x)=x^{-\\alpha} L_i(x)$. This makes more sense to me.
Yes, but the lognormality creates no problems in terms of generating *correlated* data. In this question, though, you observe that "simulated (values) ... are on average higher than the observed (ones)," which suggests that your concern is not actually about creating correlated variables but rather focuses on how to generate random variables that have specific quantitative properties. If you could edit your question to clarify this point you might increase your chances of getting some really on-target answers.
I second the recommendation of `dlm`. I'd add, however, that while `dlm` lets you get started more quickly than other packages, if you're going to dig into your results you will have to understand the underlying matrices that make up a DLM. DLM's are more like programs rather than a black box with some parameters you tweak.
I like the percentages given in the third panel.
Speaking in general for classification algorithms and not just neural networks any variable like the ones you suggest can be used for classification.  Any such variables that help separate the classes should be included.  Even binary tree calssifier use continuous variable dichotomizing by finding split points.
I don't see how this problem is solvable without making additional assumptions about $f$. For instance, suppose the sample space has $10^{100}$ elements, the values of $f$ just happen to sum to unity, but exactly one value of $f$ equals $1 - 10^{-100}$. In other words, most of the probability is concentrated at an extremely small subset. Even a long MCMC run has little chance of encountering such a subset and so its results will be grievously wrong.
Agreed to the general approach, including residual plots (which sometimes benefit from smoothing, to reduce artefacts of discreteness). A suitably defined coefficient of determination can be a useful extra heuristic. See my answer to http://stats.stackexchange.com/questions/68066/coefficient-of-determination-for-binary-responses (the suggestion generalises).
may be `difference-in-difference` methodology will work.
@whuber: Thanks; will edit.
Here is a guide how to use the Markdown format: http://stats.stackexchange.com/editing-help
Not for the faint of heart indeed. I'll take a look at that as a possible solution, but that starts to lose out to pointing people to "just use R for this one".
Sorry i was not clear there, i want to assume i dont know subgroups exist and i want to find out whether they do or not.
This doesn't work, the OP said the two analyses don't even use the same samples. The two models are certainly not nested. (Plus, even if the samples were equal, nothing in the OP's question says that the IVs of one analysis is a subset of the IVs of the other analysis.)
From a _very quick_ look , this paper focuses more on whether people are in "home" or at "work" _given_ the probabilities of "from where" do they check-in in social networks. I am not sure that the particular paper fits your purposes. But in general, my answer is not exhaustive, as I have clearly wrote.
*Sidenote:* As pointed out earlier, there's no interaction term between item and person parameters in a Rasch model (in other words, items are assumed to be equally discriminative). There was a good overview on fitting common IRT models with BUGS (most code should run smoothly under JAGS) in the [Journal of Statistical Software](http://www.jstatsoft.org/v36/c01/). There was also a nice workshop during one of the annual ISOQOL conference: [The Use of R and WinBUGS in Fitting Item Response Theory Models](http://idecide.mskcc.org/stats/multi/irt_winbugs.php).
I don't have time to look carefully at your code above right now, but a really good resource for these kernel regression methods is in the book _Elements of Statistical Learning_ (http://www-stat.stanford.edu/~tibs/ElemStatLearn/), which is available free, online.
Thank you @Glen_b. I did delete it.
Hi, I have edited my question, can you please look now?
NNs are totally not obsolete. They hold the best scores on several important benchmarks currently tackled by the ML community. Also, they are the best multi purpose differentiable function approximator around. Check out the work of Bengio's, Hinton's and lecun's groups of the last 5 years.
Please, consider giving more details to make this comment a really useful answer, in particular how Fisher test would apply in this context.
0:Aug ... 0 means 0h UTC, i think. The file is 24h and updated cyclic UTC
0:
1:
...:
23:
@guest Can you explain how you think you can motivate the t-test without normality? Note that to get a t-distribution, you not only need the distribution of the numerator to be normal, you need the denominator to have a scaled chi-distribution and *the two to be independent*. Without that, the tests don't have the claimed distribution (but I don't dispute that testing the assumption of normality, formally or informally, is also going to stuff that up). I think the best you can do is apply Slutsky's theorem and argue for asymptotic normality. How was the t-test motivated by your comment?
+1 This a great answer, and your 2012 paper is a very nice read.
cross-validation is a method of validation, I already used it in the validation(I use the weka multiple linear regression to build my model). Weka gives me as output correlation coefficient but I would like to calculate the accurucy
Correct.  The point of the scaling factor is to make MAD an unbiased estimator of the population standard deviation (instead of an estimator of the population MAD).  For the Normal, the scaling factor is ~ 1.48.  For other distributions, it's some other constant.  But, regardless, unscaled MAD is an estimator in its own right.
About my phrasing, this was not the purpose of the question, just a rough idea of my answer. I actually explain more precisely (and in French ;) I agree that the subject of such a sentence should be the random element (as in French).
No, as I said "IDF should be computed for the entire training set". In the future, in some totally different project, where you have brand new data, you have to recompute IDFs for that project. You need to compute IDFs for each corpus not for each class in the corpus.
And moreover, if my intent was to study the correlation between x1 and x2 and its impact on y... how shall I construct my experiment?
Are you trying to predict **number** or monetary **amount** of donations?
Why is this off topic?  There are several examples of questions on this site asking for datasets ([here](http://stats.stackexchange.com/questions/7/locating-freely-available-data-samples), [here](http://stats.stackexchange.com/questions/38874/classification-datasets-with-categorical-features-and-a-lot-of-classes), and [here](http://stats.stackexchange.com/questions/24261/benchmark-dataset-for-decision-tree-algorithm) for instance), so it seems there is precedent for this type of question.  Mentioning CMU and UCI shows research, and there were specific requirements that make this not a dupe.
Thanks. Is this the same as replacing each correlation in the formula for $\\rho_{cl}$ with the corresponding partial correlation with $y$ as a control variable?
Thanks, useful.  About code, libraries and source code of WEKA is available and I can use them. About data set, a 45 MB CSV file for predicting that a new customer in an specific online shopping website will order an item from his basket or not. It includes 24 features and near 430,000 row. The problem is I couldn't run some algorithms of WEKA on my laptop.
I am puzzled at the thought of a command like `regress group1 group2 yob` because it indicates that you are distinguishing the groups with *two* fields, `group1` and `group2`. I would expect your dataset to include a field whose values distinguish the treatment from the control group.  This could be the source of some of your confusion and it's why I suggested you post a small example of the data you are working with.  At a minimum you should consult the `Stata` manual pages for `regress` so that you understand where the names of the dependent and independent variables should appear.
what is $\\dot x$ and $\\tilde x$, i.e. how are they related to $x$?
Part of the trouble is that I don't know how severe the correlation is. I'm not sure how to go about looking for it since I have so many variables and any correlations are likely to be mostly due to chance. As an example (you can read about the context of my project in my comment on Peter Flom's post below), there have historically been fewer seminars on certain days. It's possible that if one of those days appears lower/higher on revenue it could be by chance correlation with another variable like topic. I somehow need to check for--and account for--any such accidental correlations.
$dX(t)-dY(t) = dZ(t+u(t)) - dZ(t+v(t)) + \\epsilon(t) - \\eta(t) = dZ(u(t) - v(t)) + \\epsilon(t) + \\eta(t)$, the variance seems $\\sigma^2 (\\epsilon(t)) + \\sigma^2(\\eta(t))$
Generally, if you want to compare the change of the two scores between intervention group 1 and intervention group 2, it's a 2x2 split plot. If you just want to compare the post scores between two groups, it's a usual 2-sample test (Wilcoxon, t-test). I hope this helps you and others. If you are still not sure about your particular data set, a statistical consultant might have a confidential look at it.
Hi @Glen_b I hope this makes it clearer. I apologize for the vagueness.
Unfortunately, Ben, when you create the model in response to your review of the data, then--unless you are really bad at modeling--any p-values for goodness of fit you subsequently compute will be hugely over-optimistic.  One way around this problem is to retain some of the original data for confirmatory testing later, but in this case that horse left the barn long ago...
@StasK the original post had multiple questions, this answer focuses intuitive general idea rather than a rigorous proof of all cases.  The is formal proof and there is understanding, both can be important.  The example could be modified to look at other distributions as well if others are of interest.  Doing it with a Cauchy distribution may show an exception to increasing power, but I don't know of realistic situations where someone would be testing using a Cauchy.
Calling a period "dot" is only really used in computer terminology & especially for urls (eg, "stats dot stackexchange dot com"). It is a fairly new usage in English, probably about 20 years old.
Programs for mapping social networks might be useful here. Perhaps the nodes would be jobs and the links could be shared responsibilities.
In a general sense yes.  Depending on exactly what you do, there could be types of relationships that could be exploited for regression/classification but won't be easily determined by the data analysis approach.
@Iterator I think we can make a lot of sense of Bob's objective if we allow that he isn't speaking statistical jargon.  Based on his time series data, he wants to know the optimal way to deploy resources for a given future period.  It is debatable whether that optimum is found with linear programming, but that an optimum *ought* to exist, and *ought* to be determinable from the data (along with appropriate assumptions and models) is intuitively clear.  This makes it a [stochastic control](http://en.wikipedia.org/wiki/Stochastic_control) problem.
Full reference for Heinrich paper please: are you expecting people to Google it themselves?
The inconsistency in your exposition is that with the use of expressions like "$cov(x_1,y)$" you are indicating quite the opposite: namely, that $x_1$ is a *random variable.* If you want to study correlation among the independent variables, why not use the usual formulas for the OLS solutions?  Then you will get correct results :-).
Since you want to compare frequencies, my first thought is to use one of the chi-squared tests for frequency tables.
Ah, that's what I did in fact :-P
Can you give the cite to the Science paper you refer to? I added the tag for `[neural-networks]`, which I suspect you are referring to; if that's wrong, please clarify.
@Glen_b: My mistake for taking that quotation out of context... Anyway, unless you really can't stand Economics (you certainly would not be alone), you should definitely check it out--Varian is the chief economist over at Google (so the dude knows about Big Data). Plus, the particular journal is geared towards non-specialists so you won't find a more readable academic journal in Economics.
Have you tried plotting A vs your dependent variable and B vs your dependent variable? That's always a good first step!
Yes. If I remember correctly, Argesti discusses it. If works if the sample size is large enough so that the chi-square apporimation works. SPSS does this testing when comparing categories of "multiple response" categorical variable. See formulas for both traditional and that test [here](http://pic.dhe.ibm.com/infocenter/spssstat/v20r0m0/index.jsp?topic=%2Fcom.ibm.spss.statistics.help%2Falg_ctables_colprop_stats_ijcomp.htm).
This question has been cross-posted and already got an answer (with a pending bounty).
You should note that mytr_all$predicted are the out of bag (OOB) estimations of your random forest model so a high correlation between that and the observed values is a good hint that the model is not working poorly.
Are the conditions always given in the same order, or are they randomized? Many people (not all) reserve the term "repeated measures" for situations where interventions occur in a particular order, with a measurement taken after each one. It does appear that condition is a within-subjects factor and perhaps the other factors are between-subjects, and a straight regression analysis does not take that into account. I'll defer to the SPSS experts for what to do about that.
you could add Red-R to the list (kind of a clone of Orange in R): http://www.red-r.org/
Clarification: I do use sparse matrices currently with MatrixModels / lm.fit.sparse. The estimates I get from lme4 are very similar. However, my eventual goal is to get rid of the big factors to (also) be able to do more effective outlier detection or robust regression. So in such case, would you recommend on the "ridging" strategy?
(+1) The OP probably means $k=2$ to denote a *second-order* MC, i.e., depending on the previous two states $X_{i-1},X_{i-2}$ rather than just the most recent one $X_{i-1}$.
@whuber That's a fair point; the line does seem to make that replacement. A graph that does not make that replacement could just be dots, unconnected, but that seems to make at least a hint at donut consumption being located at a particular point. So, we might render time as continuous and put a dot at the moment when a donut was consumed.
Thank you very much, it makes a lot more sense now
@AbhilashK if you choose to ignore empirical evidence, "better" becomes fairly subjective...
Not even! As i had mentioned it does not need to be an R-file, so the matlab function completely solves my problem :-)
@smillig, I can do it directly, but appreciate the ease provided by zelig to estimate contrast / first differences and their associated intervals.
It is important to not compute the $P$-value in a way that assumes the two distributions are identical under the null hypothesis, in order to obtain a powerful test for stochastic dominance.  One way to do this is to use the general $U$-statistic standard error as computed in the R `Hmisc` package's `rcorr.cens` function.
I wish I did.  I don't think the two are formally the same in any way but that appropriate shrinking of full models is a much easy way to obtain predictions that are as good as work-intensive and hard to interpret averages of multiple models.
If you're computing the conditional distribution of $x$ given the data, $f(x|y)$, as would often be done in a Bayesian context, $y$ is constant.
I edited the question a bit - please check now. Also, to short: 1) 30 respondents is too few for really good FA on 15 items. 2) Appending 30 -> 90 is not a good idea: you treat the same respondent 3 times as if independent, which isn't true, and due to this may fall prey to spurious, due to nonhomogeneity, correlations.
You appear to assume that the CI will shrink as the sample size goes up. That is a general tendency, to be sure--*assuming the two groups were measured in the same way* and that the CI's were *computed in the same way*. In discussions of efficiency, precisely these two factors are of concern: efficiency is used to evaluate different ways of measuring the same quantity and to compare different statistical procedures to estimate a given quantity.
Thank you for all your responses.  While I agree with your concerns about the statistical issues with this sort of exploration, I was tasked with this request, and this is the primary reason I am pursuing it.  To be more clear, each of these buckets contain a different type of variable.  It is necessary for non-predictive reasons, that these all be included in the model.
Agree with the last paragraph. The question in the OP is simply answered by running three paired t tests (or similar), however.
You can certainly do that, but you need to think about the http://en.wikipedia.org/wiki/Multiple_comparisons problem.  Also, your test results won't be independent of each other, e.g., if A > B and B > C, what are the odds that A > C (statistically significantly speaking.)  In the pairwise comparison approach, I'd use http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test, and I'd still toss the outlier, but permutation would be good too.
Does anyone have any comments about the above question?
Linear regression is not going to be appropriate for two reasons I can think of: this kind of data is unlikely to have a normal distribution and the observations are not i.i.d in that the number of followers today is correlated with the number of followers yesterday. IrishStat's answer points to the latter concern in his suggestion of an ARMAX model. But the non-normality could still be a problem. A quick Google search leads to [models for time-series count data](http://www.core.ucl.ac.be/services/psfiles/dp03/dp2003-62.pdf); however, I am unaware of any convenient R packages.
Thank you for your very good answer to this question. I am trying to add a significance test to the mtmm function. You say it's not that hard to code, bu I am not sure how to start. I am using the r.test function of the psych r package. Do you have any ideas about how to start?
That's a very nice applet!  Thanks for referencing it.  It reminds me quite a bit of illustrations I produced [for another question](http://stats.stackexchange.com/a/18200/919), where your reply might be of greater relevance.
Nice to see you here!  Thanks for weighing in.
Because the poisson distribution is positive
Welcome to the site @Kernel. You can display mathematic by putting the expression between $ signs and using LateX like syntax. And you can display code (with syntax highlighting) by indenting the lines by 4 spaces. See the markdown editing [help](http://stats.stackexchange.com/editing-help) for formatting guidelines, and the [faq](http://stats.stackexchange.com/faq) for more general ones.
Are you familiar with http://rseek.org/?  Bookmark it and start your searches for R packages there. (Afterwards you can come here and ask really focused questions. :-)
@gmacfarlane Fixed.  Thanks for pointing that out.
Good point about the ranking, as I would like to rank things.  However, I was under the (perhaps naive) impression that the output of a sigmoid function could be interpreted as a probability which could also be used in ranking.
Thanks @cardinal, I wasn't sure what was the best way to present those links. I modified my post a bit.
Welcome to the site, @KaylaWoodard. We do help people with their homework, but we treat these questions differently, providing hints to get people unstuck rather than providing complete solutions. You should read our [FAQ](http://stats.stackexchange.com/faq) to get more info about how HW questions are special. In addition, we need to know what you've tried so far & where you're stuck.
Try [this](http://oscarbonilla.com/2009/05/visualizing-bayes-theorem/).
You are not computing a t-test, because you have used the standard deviations of the data in place of the standard *errors*.
Depending on how housing at the school works (for example, if all students live in dorms, and dorm assignment is random), you might be able to use distance as an instrument for attendance rate. You might be able to do something similar with roommate assignment, which could give you exogenous variation in GPA. Being randomly matched with a party animal might cause your grades to suffer.
@AndyF Not `lm()`, unless you move to mixed-models with the `nlme` or `lme4` package, but there's a handy way to handle repeated-measurements through appropriate specification of the `Error` term in `aov()`, see more details on Baron & Li tutorial, §6.9, http://j.mp/c5ME4u
You may find [this answer](http://stats.stackexchange.com/a/71902/805) of some use.
I used spectral envelope information of the speech. I am thinking that, to benchmark the method I used, I should used different set of training data based on language on the feature I used. because, how the classifier can classify the data that it never seen before even if the data is quite similar.Am I right?Thank you Alexey.
To add to Michael's reply, in R you can use the lm() function.  In Stata, you can use the regress function.
Toby, be careful about how you interpret answers to this question, because "based on the normal distribution" can mean various things. For instance, some people might interpret a CI which uses the Student T distribution to be "based on the normal distribution" (because it is, indirectly). Moreover, there are many kinds of CIs: they are often "symmetric" in some sense, but not always (in particular, a CI related to a gamma distribution might not be). It all comes down to *what formula was used* to compute the CI. Do you have any information about that?
Roman, Thank you