tttjjj/elt_bench huggingface dataset:
If I am reading various online resources correctly, difference-in-difference is just another term for multilevel regression. I know that I can do this with SPSS's Generalized Linear Regression procedure (or even a carefully planned non-linear regression in certain circumstances) but these aren't the same as GEE unless I am doing linear regression (which I am not).
Then why not explain your actual problem?
I don't mind measure theory. I just want a rigorous book satisfying the other conditions ofcourse.
If the tests reject basically the same cases, then they're redundant - you're completely wasting your time doing a second test. If the tests *don't* generally reject the same cases, then you're effectively doing multiple testing, though the extent of it depends on how nearly independent they are - the more independent they are, the more it counts fully as multiple testing, while the more dependent they are, the more it's a complete waste of time to do it at all. If you want to test one hypothesis, pick the most powerful test you can against alternatives you care about and be done with it.
@B_Miner Not sure I understand your example. The centring is with respect to `y` here yet you are centring `x2` and thus asking for the value of `y` at three different values of `x2`. As such I *would* expect differences, but that is not what centring constraints are doing.
Which open MOOC course is that? What do the course materials suggest the answer should be?
I understand the impulse to answer in this form. Nevertheless I'd encourage you to base the main part of your answer around the information such a search reveals instead of making the search itself central to the answer. I see little harm in suggesting a search as a source of additional information, however. [If the answer was primarily 'try this search', it's really more like a comment.]
@iinception Have you considered latent growth curve modeling on observed variables?  Usually for LGM, sample size required decreases as the number of measurement occasions increases, and some have shown that a sample size as small as 20 could work well in a simple model.  Here is the reference. Muthen, L. K., & Muthen, B. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, (4), 599–620.
See https://github.com/arturadib/slash-a --- in this language you have no AST manipulation! Program is represetned as an array of bytecode and since every array is a correct program you can use whatever genetic operations you want.
$k$-means is NP-hard, so there is a lot of heuristics which significantly differ, also in resource consumption; are you interested in some specific algorithm?
Since the test of the between-factor here is equivalent to a oneway ANOVA with the per-person averages over the within-factor, these averages need to be normal and have identical theoretical variances - not the original data. For the test of the within-factor, one needs to assume multivariate normality of the per-person data vectors. Of course, if this is the case, then their average is also normal.
How does the calculation of "the mean + standard deviation of the $1000^N$ values" change when you _don't_ assume that the result is a normal distribution?  That is, how do the calculations differ from $$\\begin{align}\\bar{x} & = \\frac{1}{1000^N}\\sum_{i=1}^{1000^N} x_i,\\\\S^2 &= \\frac{1}{1000^N-1}\\sum_{i=1}^{1000^N} (x_i-\\bar{x})^2 ??\\end{align}$$
Consider a slightly simpler case with a single continuous predictor: If your response data are 0-1, then your data are two parallel horizontal lines (at y=0 and y=1, both of the form y=c). If you then fit a linear term (a + bx) and subtract it to obtain the residual, what you have is r = y - a - bx. What happens to the line y=c? It becomes r = (c-a) - bx ... a straight line, and for the two values of c, two parallel lines -- so you got what you should have expected. (Does it make much sense to fit such a model? I don't think so.)
Actually, the instrument I used had the "neutral" option, and it was supposed to be a validated one. I am not sure if I could have eliminated the "neutral" option and make use of only the 1, 2, 4, 5 on the scale.
Actually, this is not a histogram at all: it's a bar chart of counts. The distinction usually is unimportant but it would be crucial in this case if there are any differences in the durations of the time intervals. What I wonder is, since you have been very successful at describing this in very few words ("it has two dimensions of data"), why do you care whether to call it "2D" or "3D"?
What if you fit a line?  It would have low value where f is low, and high value where f is high.  It would for a normal to the hyperplane of best separation.  The search along that line would then be simpler than a search in the space.  The domain is n-dimensional, but the range is 1d.
Unbalanced training sets are not necessarily a problem.  But actually I realize there's something I'm confused about: since documents can belong to 0, 1, or several classes, what do you mean by having 200 training documents per class?  Did you do something to remove documents with 0 classes or 2+ classes?  In general, how did you generate this set of 60000 documents?
I am examining how meeting frequency affects the relationship between leader psychological capital and follower psychological capital.  So, for moderating: if the leader and the follower meet more frequency does the leaders psychological capital have a stronger relationship with follower psychological capital? For controlling: when controlling for meeting frequency, what is the relationship between leader and follower psychological capital.
You would need to vary the `conf.level` argument to `prop.test` if you want to change the nominal false positive rate (whose default is 95%).
It would be great if you could provide some more specific context and maybe example data or the type of data you want to use.
Your notation is now confusing and contradictory. I suggest saying "Let $S=X+Y+Z$," replacing "$\\text{sum}$" with "$s$" in subscripts and $F_x(x_x + x_y + x_z)$ with $F_S(x + y + z)$. If you agree that is what you're asking, then your question would make sense, and the short answer is "yes, that's what you do".
I cannot make sense of the second sentence in this question. Is there any difference between "normal samples" and "$r1∼U(0,1)$ and $r2∼U(0,1)$" or is there some meaning assigned to $U(0,1)$? Does it mean the same as $N(0,1)$?  Also, what does it mean to say "$z=\\rho r_{1}+\\sqrt{1-\\rho^{2}}r_{2}$ are correlated with correlation $\\rho$"?  **What** is $z$ correlated **with**?  Is $z$ a normal random variable? Are $r_1$ and $r_2$ normal and so their (weighted) sum is also normal?
You've copied the initial expectation incorrectly from the other page. This will change some of your work lower in your question.
The [third version of this joke](http://stats.stackexchange.com/questions/1337/statistics-jokes/1436#1436), yes?
But it's not a Bayesian network.  Bayesian networks are directed graphical models.
Hello again :) TY for your help! That X1&2 are the same just for me (to study and to understand it). But of course there is some real data with me. X1 and X2 are the indices of an m-by-n-matrix. This matrix has 100 fields with values. The main goal is to get the function f(X1,X2) of the matrix. So X1 and X2 are representing some different dimensions. -- As I said, the easy values above are just to get a good model.. these are my thougts behind -- @Patronum I don´t get your last sentence...
What units are your dates measured in?  The lags in the plots are enormous--they look like seconds.  Taking first differences at lags of seven *seconds* won't fix a weekly cycle!
+1 for mentioning FA, which no one else seems to discuss, and which some people's explanations seem to blend towards.
One of my greatest teachers recounted his PhD thesis defence in which he explained to a group of physical oceanographers that his regression could explain XX% of the variation in some data. A distinguished examiner stood up and proclaimed "With your correlations you have explained exactly nothing; rather you have offered us a suggestion which falls apart 10% of the time"
Just to be clear: it seems the question makes sense *only* in the case where (a) there is a single predictor $X$ (in addition to the implicit constant) and (b) $X$ is a binary variable.  Otherwise, logistic regression of $X$ against $Y$ cannot be done.
Just to expand on Gavin's note about memory: see the section on Large Memory in this CRAN task view if you are working with larger data sets: http://cran.r-project.org/web/views/HighPerformanceComputing.html
If you're answering your own question, that answer should normally go in an answer rather than the question.
I have around 15 categories, some have as many as 100 obs and others few like 3, 6 etc. I am doing a multilevel regression. Strangely, after grouping the variables into "other" my significance level and R^2 drops slightly.
your definition for harmonic mean does not agree with [wikipedia](http://en.wikipedia.org/wiki/Harmonic_mean)
This is spooky.
I suspect you have some missing value(s) in your distance matrix (which doesn't hurt `as.dist` but `hclust`). You may want to impute them or try [hybridHclust](http://math.acadiau.ca/chipmanh/hybridHclust/) (never used myself).
New residuals for every fit - that was my first understanding of residual bootstrap. But I must admit that different sources are using residuals of fit to original data for that purpose. Here is nice tutorial on bootstrap (http://www.econ.pdx.edu/faculty/KPL/readings/mackinnon06.pdf). Seems that my approach is wrong while your implementation is right. Should I delete given answer?
@Nick I deleted the first part of your comment in the spirit of http://stats.stackexchange.com/help/behavior. I know you were helping, but we should take care to consider the feelings of newcomers who are not used to the site and the necessarily abbreviated conversations that occur in comments.
That's one possible solution. In this case, remember that improper priors are no longer allowed, and priors with a very spread support will probably make the Monte Carlo approximation difficult.
"I think there many other superior approaches for this purpose" - like what? If you have specific alternatives in mind, please edit the question to identify them.
For Q1, the wikipedia page you link shows that by the definition of Memorylessness ($P( T>s+t | T>s ) = P(T>t)$), one can derive the only cont. distribution with the property is the exponential. I believe that if you assume a discrete distribution, the only solution is geometric.
Michael, thanks. Is there a formula to use to arrive at the 1038.39 value in my example?
The population isn't clear from what I saw; it could be all the obese people in the world, or in a particular country or other geographic region, or something else.  It is the set of people from which the sample is drawn.
So is PCA equivalent to Total Least Squares if it optimizes orthogonal distances from points to the fit line?
I think Fisher did a species counting thought experiment with a Poisson process, and Wallace and Mosteller used it later for unique word counting, but I think that requires each occurrence to happen independently.
Wow thanks for the answers. this is what I looking for. I think I have a new question, but I will ask that in a new topic.
Strictly speaking this is operations research more than statistics, but the key issue is the cost of a stock-out.  So you need to know (or at least estimate) the holding costs of different levels of stock, the opportunity cost in terms of potential profit forgone (and perhaps the reputational cost) of failing to meet a large order, and other costs such as those of restocking (which might include for example delivery charges).  You can then model different stocking regimes - typically with a maximum and minimum level of stock held - against the pattern of orders to find the optimal approach.
In newer versions of Stata, you don't even need the xi prefix.
Thank you Procrastinator for adding those links to my answer.
Absent a model for measurement or sampling error, where would that come from?
"Number of variables" and size of eigenvalue do not have anything to do with each other.  Careful accounts of PCA do not make blanket statements like the one you quote, because which components to retain depends on more than just their eigenvalues (as suggested by Peter Flom's answer).  It is impossible to have more factors (components) than variables anyway, even when you include the components with the very smallest eigenvalues, so your paraphrasing cannot possibly be an accurate representation of the quotation.
This is a very clear and helpful answer, so thank you!
@Affine. Firstly, the whole discussion of "winrate" is just in the "Motivation" section. You can simply ignore it: I abstracted away all the domain-specific language in my whole question.

But if you're curious, here's the explanation: I have a database of about 10 million poker hands. For each player, I know how much they won in total. But this total win is just a sum of (roughly independent) samples of a random variable. This random variable has some variance. The goal is to analyze the player pool and understand the distribution of means of these RVs, hopefully cancelling out the variance
@cardinal - Ah, yes, thanks, 'competing risks' is exactly the term I was trying to dredge up from memory (before falling back on multiple decrement as the nearest term I could manage to bring to mind).
Suresh is right -- in classical information theory, the information content of a single message isn't defined. You can only measure information content for some ensemble (distribution) of signals.
+1 for the suggestion to not multiply out the expression first
Do you have any insights into the problem by yourself?  As is, you're just asking for an answer.
Please don't start a new thread each time you edit your question, Frank.  Until somebody actually posts an answer, feel free to change the existing question.
The experimen is still under way, so I will get more data points. I am interested in methodology (I think it is the same for any count of data points). And I had not found any statistical method of determining "gray zone" while it is widely used in tests of such type.
+1 for the TLS recommendation. If it makes sense to contemplate *both* forms of regression, that means you believe both $X$ and $Y$ are subject to important amounts of random variation, whence you probably should *not* be using ordinary regression in any case!
Ok. Obviously my understanding is pretty limited. Could you give a simple example?
Hong Ooi answered the main question and deserves the acceptance in my view. My answer was secondary.
@cMinor: First of all you need to decide your threshold. I did not do explicit calculations, but if eigenvalues are sorted in descending order and cumulative sum of first m eigenvalues is 80% of their cumulative sum, then keep only first m columns in U,  m x m submatrix of S and m rows of V'.
What do you mean by "the system will generate a 100 values each corresponds to one unit"?  What do you mean when you say "I randomly generated 120 subsets of the 100 units"?  Did you sample with replacement? With respect to your first question, why are you looking at correlations rather than some measure of closeness of the prediction to the data?  With a correlation, you have have predictions that are wildly off, but still have a high correlation.  (Think of predictions that are off my an order of magnitude but in the same rank order as the data.)
For what exactly? Can you say more about your data, your model, & your goals? (Bear in mind, BTW, that there may be no valid alternatives, unfortunately.)
Fantastic help, thanks!!
aic is basically leave one out cv.  bic is basically sequential cv (one step ahead forecast).  which type of cv did you mean?
@gung, thanks for the edit, it looks good. Could you please help me on this question? Thanks
By levels, do you mean binning the Y variable into 100-200 buckets? If so, is there any preferred method for choosing the bin size? Should they be equally sized?
@whuber Yes but with more noisy data, or with some of the $\\pi_k$ close to $0$, you’d violate easily the constraint $\\pi_k > 0$, which is the "hard" part of the problem.
Did you make sure that the value 0 on both $A$ and $C$ are meaningfull and near the range of the data? For example, if $A$ is year of birth, than the value 0 would be the year 0, which in most cases is way outside the range of the data. In that you would need to center the offending variable(s) before computing $A \\times C$.
Can one make inferences about the true distribution based on the observations, perhaps using the variance of the observations? The extreme outlier case is tough, but if you've seen one then that carries a lot of information. If we revise the question to specify N > 2, then already the second observation tells us something if N1=N2 vs N1 != N2 (and what the difference between them is).
@Alexis This is true (+1), but in the context of removing the effect of an outlier, I believe the idea will have been to simply use a 0/1 dummy: while any other pair of distinct values would do just as well at achieving that, the size of the outlier could at least be of some interest, and that's most directly achieved in the indicated fashion.
In addition to the answers below you might **also read** [this](http://stats.stackexchange.com/a/94104/3277) and [this](http://stats.stackexchange.com/a/95106/3277) of mine.
As per my last sentence: you do not have proof that it is, nor that it doesn't. The phrase typically used for this is "we cannot reject (at ... significance level)" that the median difference is zero.
Besides the fact that a conditional Poisson distribution (conditional on x) does not imply a marginal Poisson distribution, this would seem to be exactly the idea of Poisson regression, yes.
@whuber I have tried this method. The values in the middle of the range of $\\mu$ are underrepresented. For example, if $m$ is 1, $\\mu=1$ most of the time. But if $m=2$, $\\mu$ can jump to 10, rarely touching the values in between. I have added an edit to the question to make it clear that the values in the range of $\\mu$ should be more or less equally represented.
I mean the second, "learning a distance metric function". Upon more reading by the way, my simple A and B formulation above can be found in the quoted manuscript: "Given training examples of item sets with their correct clusterings, the goal is to learn a similarity measure so that future sets of items are clustered in a similar fashion."
If $X = Q R$, then $X^T X = R^T Q^T Q R = R^T R$. Recall that the covariance matrix of linear regression is $\\sigma^2 (X^T X)^{-1}$. But $R$ is an upper triangular matrix, so $(R^T R)^{-1}$ can be found by one backward and one forward substitution.
In many cases that will lead to poor result.
Except you have huge amount of data, using 10% of data to train will not capture the structure of distribution well. you might want to try some technique like bagging to generate many data set with size (say 10% of your original data set) which your algorithm can handle well, in parallel manners , then blend those results you got.
This is not a direct answer to your question, but I'd recommend you pay very close attention to issues of measurement, of reliability and validity.  Even before, or at least parallel to, your investigation of mult. regression and SEM, you'll want to look into scale development, factor analysis, and principal component analysis.  To get accurate estimates of a relationship, you'll need to have the most accurate measurement possible of each construct.  This can be an intricate and fascinating process when you have many constructs, each potentially consisting of many others in turn.
Have you checked them on all scales? For example, if you take the natural log of your estimate and the confidence intervals, are they symmetric?
See Wikipedia. There is an incomplete implementation in Weka, and a much more flexible in ELKI. I don't know of any in R.
Did you take a look at [this question and answer](http://stats.stackexchange.com/questions/47771/what-is-intuition-behind-beta-distribution)?
Why are you doing PCA? If, in subsequent analyses, you just want to be able to work with fewer variables that still explain most of the variance in the data, it's simply a transformation to a new set of variables that doesn't make any assumptions that need to be tested. If you're using it as something akin to factor analysis, making inferences about latent variables, then that's a different matter - I can't answer.
The expression $\\hat{\\theta} = \\theta + w$ is a theoretical expression to which one usually arrives while trying to determine the properties of the estimator, unbiasedness in this case. But this is not the actual functional form of the estimator, because it contains the unknown parameter $\\theta$. In order to meaningfully explore your question, we need the expression of $\\hat \\theta$ as a function of the data. This cannot be answered in general.
I have faced a similar problem lately. But I will advice you to use a leave-one out CV. This will bypass the problem. I wonder how a leave-one out CV will suit your problem
+1 Simply marvellous!
@whuber I agree. I was just following along with the OPs proposal. I would not say that it is a good idea. The outcomes may tend to reproduce but the variation in the artificial sample deos mean that results from one case to another would not be identical and could possibly be contradictory. It seems to me that if the objective is to test whether the sample comes from a population with mean 0, it is best to do the one sample test. If you want to show that the sample appears to come from a N(0,1) distribution do the goodness of fit test.
Thank you. This isn't a real world model, it's just to help me understand things.
Thanks, this is the answer I am looking for!
Monica, if Andre's answer is a satisfactory answer to your question, you should accept it - click on the big empty tick mark/check  mark ($\\bf{\\color{gray}{√}}$) over on the left side of the answer, to turn it green ($\\bf{\\color{green}{√}}$)
Using the median and something based on percentiles sound like good ideas but setting the threshold directly at some low percentile assumes that a fixed proportions of prices is abnormal in each sample. Why would it be the case? The boxplot approach (x times the interquartile range) would avoid this problem.
Hi there, could you give some more context around the decomposition you wish to do, as I have not seen it applied to time series data?
"*I want to test difference in means to see if they are related*" -- a paired t-test doesn't test to see if the means are 'related'. It tests if they're different. But yes, if the assumptions hold, it works for any number of pairs ... as long as there's more than one pair.
The figure is interesting. Could you give the reference where you took it from?
perfect...thank you!
Yes, but it can get confusing.
"equivalent dosage" seems a bit suspect. Off the topic of your question but saying in case this is your research project.
(1) What are the units in which each coefficient is measured? (2) What do you mean by "important"?
(+1) Nice summary. The last few sentences -- about sub-population approach and validity of goodness of fit tests when predictors only take on a few discrete patterns -- was the message I got from reading pp51-53 of Paul Allison's notes on aggregate data in Proc Logistic.
Please don't put these things in comments. Edit your Q to add the relevant information. I still don't see any question. What do you need help with?
"validate the Gaussian requirements": isn't Gaussian / nonparametric / forget-modelling (pace Breiman) a deep split ?
Does it make sense to think that non-dyslexic people have an easier time reading the more obscure the font is?
Aha - it's Rq not Qr() [I think].  Works brilliantly!  Upvote from me.