tttjjj/elt_bench huggingface dataset:
@Marco You seem to be combining two different representations of the same model.  You can have either that $Y = X\\beta + v$ where $X$ includes a constant term and you stipulate $\\mathbb{E}[v]=0$ or else you omit the constant term in $X$ and allow the expectation of $v$ to be a parameter.  Doing both, as your edit points out, is redundant, but for the trivial reason that you have effectively included the same parameter twice in the model (both as the intercept and the mean of the expectation of the residuals).
Hmmm. You might then want to look at a model for truncated data, rather than zero-inflated. There is a `truncreg` package, but I've not used it at all.
As with multidimensional scaling, you don't trust the untransformed distribution of data, so you are shrinking the observation space according to the covariance of the variables. WLOG You believe certain observations on the $X$ side are closer despite being farther in the Euclidian sense from observations on the $Y$ side due to $X$ having smaller variance than $Y$, say $X$, $Y$, are jointly observed independent variables. If you're confident in this method, then you won't be swayed when untransformed variables give slightly unintuitive results.
While we're at it, here's another [set of slides](http://www.google.com/url?sa=t&rct=j&q=greenland%20bias%20ppt&source=web&cd=1&ved=0CB8QFjAA&url=http%3A%2F%2Fwww.ccsem.infn.it%2Fef%2Femfcsc2008%2Fbioelectromagnetics%2FGreenland_Bias_and_EMM.pdf&ei=KT7QTq6zHonW0QGl49AF&usg=AFQjCNGSBcTQ3-btHHTIyU2LOr9hyBsvsw&sig2=3Mecxz-q6thRIC4bno8yQw&cad=rja) from Greenland.
This question is extremely vague and difficult / impossible to answer in the current form. Can you hone in on a specific question that you need help on?
what is the context of this problem? As it stands now, it is impossible to answer, since in multivariate case the means are vectors and the distance between them is a number, but the covariance matrix is not.
@Alfred.M  Thank you. Yet I have no more to write... except a few details about the example?
I understand now that we are viewing n observed values of a single random variable as one observed value of n random variables having the same distribution. This part seems to be aligned with my update to the question. But the second part of your answer suggests that when finding the expectation of X(i) we are not using the data points at all. So given some other distribution $EX^{(i)}$ is always the expectation of that distribution? So for bernoulli it would be the parameter p?
The sources that I mentioned are:
http://glmm.wikidot.com/faq#toc10 (final bullet point in this section)
https://mailman.stat.ethz.ch/pipermail/r-sig-mixed-models/2012q3/019121.html (final paragraph)
Would I be able to translate this approach to not only the most significant odds ratio but all significant odds ratios as deemed by their local FDRs?
Thank you, @DavidR. This is an interesting example when indeed $R_{emp}(f^{*}_{l}) = 0$ for any $l$ and $f^{*}_{l}$, but $R(f^{*}_{l}) = \\infty$ when $f^{*}_{l} \\ne c$ and $R(f^{*}_{l}) = 0$ when $f^{*}_{l} = c$. This shows that the definition of consistency should include "for any $f^{*}_{l}$" part.
Have you tried the commands summary(aov(dependendVar ~ IndependendVar))) summary(lm(dependendVar ~ IndependendVar)) Do you mean all the group means are equal to each other and equal to 0 or just to each other?
I agree with @Zach, rpart results here are meaningless (and actually pretty equivalent). You can calculate RMSE just by executing `sqrt(mean((pred-true)^2))`, where `pred` and `true` are respectively vectors of model predictions and true values, both on a test set. You can also try `cor(preds,true)` and `cor(preds,true,method="spearman")`. Also a scatterplot (`plot(preds,true)`) is useful as it helps to assess the structure of error.
Alexis, I fear you may be leading the OP astray. Notice he has only the sum and sum of squares of the data; therefore an algebraically equivalent expression for the variance estimator is needed--and he is using a correct one and appears to be computing with it correctly.  (Your first comment seems to overlook the fact that $\\sum_i \\bar{x} x_i = 0$ when computing the sum of squares of residuals.)
<the test needs to be longer, so you'll 'scrape the barrel' for items.
Clearly you have had real world experience putting together tests.
That's awesome - the picture you provide is one I drew for Wikipedia in 2008, and I found this question when I was figuring out how to redraw it in 2012.  You provide a PNG, but it was [originally an SVG](http://en.wikipedia.org/wiki/File:Latent_Dirichlet_allocation.svg).

Back then, I used Inkscape and it was terrible - I had to hand edit the SVG to add in the greek letters, because it couldn't handle it back then (maybe it still can't).  I don't recommend it for anyone.  I'm going to give the tikz approach a shot.
@jrennie You are correct, sorry.  It's now corrected.
(DikranMarsupial and @bayerj) I think my out-of sample results are noisy because of my maximum number of epochs were set to low.  The models therefore do not converge before stopping, especially in the case of higher $c$ and $h$ values.  Do you think this could be the case? Furthermore, is CV needed for parameter selection? Can't I just evaluate my various $(c,h)$ models over a large validation set to select approximately optimal parameters. Use approximately optimal parameters and CV to report final model results. CV is slowing down the random search by a factor of ten.
Draft of **Analyzing linguistic data : a practical introduction to statistics using R** is available [here](http://www.ualberta.ca/~baayen/publications/baayenCUPstats.pdf).
It would be incorrect to refer to the intervals as "confidence intervals" (even approximately). They are *control limits.* You can connect them to interval estimation by characterizing them as approximate *prediction limits*; indeed, prediction limits are used for quality control purposes. So, if you are using confidence limit tables to compute the $A'_\\alpha$, you need to revisit that.
That's an excellent resource!  It deals with hypothesis testing nicely, which a lot of material on coherence conveniently avoids
Okay I was just wondering whether the notation would have to be different in a quantile regression specification. Guess not!
Is this still relevant? I can answer this if it is..
@NickCox I guess you are right i should read more, its quite obvious i am new to statistical models etcetera. I think i am doing good so far though. Anyway  thanks again
It remains that a large sample that is nearly normal will have a low p-value while a smaller sample that is not nearly as normal will often not.  I do not think that large p-values are useful.  Again, they reward for a lack of evidence.  I can have a sample with several million data points, and it will nearly always reject the normality assumption under these tests while a smaller sample will not.  Therefore, I find them not useful.  If my thinking is flawed please show it using some deductive reasoning on this point.
[Cross-posting](http://www.reddit.com/r/MachineLearning/comments/gdtv3/critique_this_visual_taxonomy_of_pattern/) is not encouraged.
Certainly, but the Bayesian can still infer a reasonable conclusion with the **given data / experiment results**. Sometimes you can't repeat an experiment or design it the way you want.
That's a very nice plot.
You're right this hinges on the purpose of the study. I suppose I didn't make that explicit (I only talked about what you *want* to achieve). It's also true that explanation needn't be concerned with causality exactly--something analogous to causality fits as well (e.g., the dimensions - volume case is one of logical / mathematical implication). However, most explanatory modeling centers on causality; I guess I thought I could skip that sort of thing for the sake of simplicity. Finally, strategy does differ during study design & data collection, but regressing y on x is pretty much the same.
That would be a measure of the linearity with time, but not of the smoothness.
(-1) This is not correct. In general, you shouldn't include the same variable on both the right hand side and left hand side of the equation (as it results in the independent variable being correlated with the error term). So if you use differences for the dependent variable, you should not include the baseline as a covariate.
This is an incredibly general question.
Hi Jeromy, thank you very much for your reply. When you say "seeing whether anxiety predicts the difference score between psychological adjustment in the two social situations", do you mean that in essence, I am using a linear regression: relationship between anxiety and the difference in scores for Social situation A and B?
Some ideas on [Optimal Aggregation of Consumer Ratings by Dai et al. (2012)](http://www.hbs.edu/faculty/Publication%20Files/13-042_3c56fcb5-b7b9-4eb9-a026-2822c79b7127.pdf)
Thank you for the very detailed response. It will take me a couple hours to fully read the supporting materials and generally digest the content of the post. Please don't mistake my slow pace for a lack of gratitude.
I don't think this decision should only be based on the characteristics of the data set but also what you want to communicate, along with the expectations of your audience. Error bars based on standard errors of the mean are very common, raw standard deviations less common, and ranges less common still (if you're inclined toward ranges, you might consider boxplots). But with only 4-5 values, *why not simply indicate all four or five values?* -- Do you have any sample data available?
You should add the error terms too.
Thanks, Thomas! I am still not very clear about part 2. I just added an example. It will clarify my confusion, if you could write out the model in part 2.
I completely concur with the notion that the question is somehow outside the purpose of this forum. The definition of covariance as $$\\text{cov}(X,Y)=\\mathbb{E}[(X-\\mathbb{E}[X])(Y-\\mathbb{E}[Y])]$$ is the clearest explanation one can propose. It only uses the notion of expectation. Avoiding the formula is leading to necessarily incomplete and potentially misleading versions. And this cannot provide the reader with the man to compute the covariance/correlation in a new situation. Not the best way to fight innumeracy.
Are you working on windows? If you are on, for example, linux; the paralellized version is much more memory efficient: http://www.r-bloggers.com/introducing-parallelrandomforest-faster-leaner-parallelized/
Your answer becomes more useful to all readers when you delineate its scope and potential applicability. (An alternative strategy, in the spirit of good statistical consulting, is to use comments to ask the OP about the possible number of dimensions and then tailor your reply to that.)  (+1 for the improvement.)
There's very little information here to go by. How about showing us the data (in a figure), and perhaps summary of results?
@utdiscant: Furthermore, your time-based xregs need to be *dummy variables*.  The way you have it modeled now is that you expect `heat` to linearly increase with hour of day, and then jump back down when the hour returns to 1.  If you use dummy variables, each hour of the day will get it's own effect.  Run through my example code, and pay careful attention to how I construct my xreg object.
I see. Being specific, I'm doing named entity recognition. The corpus that I use to collect these frequencies has millions of documents, enough to have valid frequencies. Then I also have a big list of multi-token names of the entities I want to recognize, for which I also collect statistics. I discretize the frequencies of both dictionaries and then I combine them so that in total I have `(num_bins_global_corpus) x (num_bins_names)` The idea behind is that basically words that appear few times in the whole corpus but many times in the names are important for the recognition of these entities.
You say "but it seems to require the whole data from which the parameters were estimated, which I don't have". Do you have the individual responses to the items?
This is a good answer, but it still skirts around the core reason as to why fit always improves - is it because of the purported reason I stated in the end of my answer?
Any proof is a series of steps that logically connect hypotheses to a conclusion. Although your conclusion is clearly stated, what hypotheses do you want to begin with? After all, because both procedures test the same thing by means of the same hypothesis-testing framework using the same assumptions, on that account alone they *must* be equivalent! So if that's not a sufficient proof for you, please indicate where you are beginning and what methods of proof are desired.
@David added a code that can be seen here: http://stats.stackexchange.com/revisions/14534/2 .However, I want to refer (firstly) to (nested) categorical variables, not continuous as in the code.
Could you indicate what information is required from Table 2 (or are there multiple outcomes?), and what would be the general form of your survival model?
This question looks very uncommon. Under usual assumptions, the estimators of the regression parameters are unbiased. Thus the means of the sampling distributions of them are just the parameters themselves. So an equivalent formulation of your null hypothesis is $H_o: |\\beta_{2A}-\\beta_{2B}| \\le \\Delta$. This hypothesis could be tested via confidence interval for true interaction with grouping variable (A vs. B) using a single regression.
(you might have mislead with my "not so good" English)
I will try to further explain myself: I have several (many) samples of signal which comes from two distinct groups. my aim is to classify those two types of signals. Because  the data is high dimensional one, I use PCA to extract scores for each signal. In principle/"theory" if I first transferred my signal to the time domain and then perform PCA, does it have any advantages ? this is what I meant when I asked "does (PCA) have any "well-known" advantages or usages in the frequency domain?"
@nico I now believe the McNemar generalization is not appropriate but a generalization of Cochran-Mantel-Haenszel could be.  Take a look at my edited answer.
For the intuition: http://stats.stackexchange.com/q/1447/12359 . For the quality: http://stats.stackexchange.com/q/13314/12359
Thank you, I noticed that using `ln` in the formula yielded better prediction accuracy. I guess I need to explore the effects of transformations on the independent variables.
More precisely: jackknife is leave-one-out resampling [but not used for cross validation]
Huh? The graph shows the S&P over time. The title talks about butter and cheese production, which are not visible on the graph. ???
I'm not aware of particular difficulties apart from the low power.
1. In univariate analysis the LOT1 groups weren't associated with mortality and now is highly significant...2. Performing an identical analysis in a different subset of mortality (cardiovascular mortality) yielded the same statistically significant result for the interaction term, although in univariate analysis the biomarker wasn't associated with the outcome - CV mortality (in neither of the two subgroups)...
In general, R does not require sorting to perform operations (in contrast to e.g. Stata).  In fact, life is generally easier if you do not sort.
You're getting some good information in the answers, I just want to add a comment regarding 1 detail that is peripheral to your main question. You state that you hope to learn if "the level of some hormones... are influenced by smoking". Your setup, as I understand it, does not afford an answer to this question--you can say whether hormone level is *associated* w/ smoking status, but you cannot infer *causality*. Inferring causality from observational data is very difficult & you would probably need to work w/ a statistical consultant.
@palanski, what do you mean by "required"? The number of parameters required to have the output be well-defined? Or required for the output be consistent for the truth as the number of observations goes to infinity? Or something else? Please edit accordingly.
Ah, now I'm tracking.  Thanks so much!  So, I added in a normal distribution and did the chi-squared test with that (I had just been using the mean).  It makes so much more sense now!  I did do the t-test first, but I felt like it didn't quite get at what I wanted to express, since it's just showing if the mean is different.  I want to show that people overall responded with more positive or negative rankings than, say, asking people on the street.  Do you think comparing my response distribution to a normdist is reasonable for this? (It's a survey my company sent out to employees).
Thanks much. **Clarification:** Assuming vectors **Xi** are ordered in rows, so that each column (of the matrix formed by this vector-arrangement) represents a certain feature: Is the covariance matrix you're referring to - the features' covariance matrix?   In other words, assume there are N features (i.e. N columns, i.e. vectors **Xi** dimension is N), the covariance matrix would be of N x N size {and symmetrical as COV(**Fi**, **Fj**) = COV(**Fj**, **Fi**)} ?
So do I need to scale them within the range of [0 1] or zero mean unit variance is enough. I tried to standardize the inputs and outputs to have zero mean and unit variance but still it is not good enough. I still get very high value for $\\lambda$
@user123675: In your last comment you probably mean "maximizing", not "minimizing".
You might be interested in the (now large) literature in philosophy on this paradox. Here is a fairly complete bibliography (with links):
http://philpapers.org/browse/sleeping-beauty
In response to your flag, no you can't delete your question while the bounty is running. But (1) you can post your solution so that users will learn from it, and (2) you can wait for possible alternative solution, if any, so that you don't necessarily have to award the bounty to yourself (that's really up to you).
(+1) For online competitions. I think if you did something from kaggle or one of the other competitions out there and kept track of your code and your process (I'm thinking blog) in a place where potential employers could check it out. It would show a lot of initative and in a lot of ways is easier than thinking up your own question of interest. Just pick one of the competitions that interests you, then you have the data right there and a place to submit and compare answers.
This may perhaps help? http://stats.stackexchange.com/questions/81789/if-is-uniformly-distributed-then-a-linear-probability-model-is-appropriate-c/81803#81803
Perfect answer! Exactly what I needed and greatly explained. For what it's worth, I had reached to get the `binom.test` function in R which gives you values for confidence intervals too. But it was lovely to get the whole explanation. Thanks.
That's not a statistical question, LOL!  But (to stave off possible objections) I agree that its answer can be usefully informed by statistical thinking.  "Consistency" depends on what you are studying and what use you will make of a decision that two response curves are "consistent" or not.  In some applications it would be enough that they are both increasing or decreasing; in others it would amount to a test of equality of all parameters.  Without more information, one can only guess where along this spectrum your needs fall.
+1 to this (and other answers) that provide actual examples of the *situations* referred to in the question, rather than just abstract distributions.  This example of the ZIP is particularly clear.
+1 Out of curiosity, just what is the "one sensible definition of best"?
I think you need to give some more detail on which LDA formulation you are using. Generally it is only RDA models that have those parameters, LDA usually is defined entirely by mean vector, covariance matrix and prior probabilities.
@Charlie If U and V are standard normal and independent the ratio is Cauchy and its first and second moments would not exist and clearly the delta method could not be applied.  The same difficult occurs if V the variable in the denominator has a significant probability mass in a small interval about zero.  Having the denominator very small can make the ratio very large and hence not have a second moment and possibly not even a first.
+1 Please consider updating our [meta thread](http://meta.stats.stackexchange.com/questions/793/internet-support-for-statistics-software) where we have collected links to stats software resources.  There's one reply there for R and another for SAS: both would benefit from having a link to r4stats.com.  (That thread is actually a part of our FAQ.  We hope to keep it current and useful.)
+1 Interesting example. I was also thinking about other issues with MH related to the acceptance rate. I think the [0.234](http://faculty.wcas.northwestern.edu/~lchrist/course/Gerzensee_2011/The%20Annals%20of%20Applied%20Probability%201997%20Roberts.pdf) optimal rate has been overused.
(+1) **Yes**! :) You may have to wait a few (8?) hours, but you should consider coming back and placing your edit into an answer so that other users (like me) can upvote it. Welcome to the site! I hope to see you continue to participate here. Cheers. :)
@thigger, you don't need to difference a series for a deterministic trend. I guess in your case stationary model with trend ($X_t=a + bt + Z_t$, where $Z_t$ is stationary series) would fit your data.
I hadn't seen these ones before. They look good.
What is it you want to know? What do you hope the analysis will tell you?
The answer is correct, but for the OP it would be at least get an idea why the vector $(X,Z)$ is not normal, or why $Z$ is normal.
(i) What are the possible values that $y$ can take? if $x\\in\\{0,1\\}$ then $y=ax+b\\in\\{...\\}$. (ii) You know that $P(x=1)=p$ and $P(x=0)=1-p$ you can then prove that $x=(y+b)/a$, therefore $p=P(x=1)=P[(y+b)/a=1]$, similarly $1-p=P(x=0)=P[(y+b)/a=0]$ ... I hope these hints help.
That is , I am correct.
This is unbiased but probably isn't efficient if n and m are different because $y_i$ in the smaller subsample are given more weight than those in the bigger subsample.
Please read the Stata time series [TS] pdf manual. Quasi-correlations are very well explained in the manual on page 250 (in the section marked "Technical note"), together with literature references.
@Paul: You could accept this answer if you think it gives you what you need
It's also a good example of the ecological fallacy (i.e., making inferences about the individual-level from group-level data).
And what I'm really interested in is converting my correlated variables into uncorrelated ones. As you say, the nested structure of the data is making the components correlated. So is there a way ('nested' PCA or otherwise) to create uncorrelated variables from a set of hierarchically structured correlated ones? Cheers
If I could vote up more than once, I would do it!
Cook's distance sounds about right, thanks for the reference.
There is a positive correlation but it is not possible to find its strength since the value of covariance depends on the size of your data. Compute instead the Pearson correlation coefficient which is the covariance statistic normalized to be between -1 and +1.
@whuber I apologize for not responding earlier. I actually share your concerns. This method will absolutely fail if any of the coefficients are negative. In fact, my guess is this method will fail if any coefficients are small and positive. And now that I'm thinking more deeply about the situation, this method may perform worse than OLS even when all coefficients are positive depending on their magnitude.
+1 This was indeed the solution I referred to when I commented above "*I can see a way to do it, but it's complicated enough that I am sure there must be a much simpler way.*".
Ok, thanks. It was the terminology that got me - I didn't realize clustered/repeated response data meant basically the same thing as mutlivariate in regression, but it seems that's the case.

In Agresti 2002 I think it's Ch. 11 & 12. 2007 and 2002 cover most of the same stuff, but '02 is a little more in depth. 

Time to read those chapters.