tttjjj/elt_bench huggingface dataset:
You may want to have a look at the "mixture of experts" approach.  It's a bit different from what you're doing, but it is good, for instance, if you expect different stocks to respond differently to the same change in economic variables.  In the mixture of experts approach, instead of having a model determine the probability that each model is "real" as you put it, you have a model that assigns each series or observation to whichever model does best with that type of data.
From [Nicomachean Ethics](http://en.wikipedia.org/wiki/Nicomachean_Ethics) for those who care.
Maybe not so much on the locations..as the cities were selected randomly. Any ideas coming to your mind ?
http://stats.stackexchange.com/questions/575/post-hocs-for-within-subjects-tests
Hint: the coefficient $b_1$ depends (strongly) on the form of dummy coding. Another hint: what precisely could be meant by *the* correlation of $x$ and $y$? It appears that you have introduced *many* correlations, each depending on values of the covariate $c$.
@kwak Still, this is a good answer; trees in RF don't see the entire set, so many of them will not be contaminated. Even better -- tracking in which leafs do OOB cases land can be used to find mislabeled objects and eliminate them. (As I recall now, this is mentioned in Breiman's paper about RF).
I understand what you are saying.  And, I have built home valuation models like that in the past that worked reasonably well.  But, this is a real life situation associated with an assessed valuation appeal in order to lower my property tax.  Within this context, 5 comps is actually a lot.  Most appraisers and related county analysts work with just 3 comps.  This is a qualitative field that does not take well to large data driven quantitative models.  I have tried before.  The acceptance of the model we are talking about at related County hearing is nil.
What do you mean by "one group"?  Is this just your sample of subjects?  Also, "2 different types of words" is ambiguous.   I'd reword your first sentence as "In my study, I have ??? subjects, each of whom encode ??? different words against three different measures.  The words are categorised into two types".  Or I may be wrong - the point is your first sentence is hard to understand.
"*Is it something a transformation like Box-Cox could cure*" -- why should your data need to be normal?
+1 Yours is the best explanation.
I think this question has sufficient statistical content to be on-topic for CV, but note that it requires fairly sophisticated programming & Python knowledge as well. It may be difficult to get a good answer. You may want to ask for / be willing to settle for *pseudocode* as well, &/or you may need to split this question up into 2 parts, 1 here about the statistical aspects & 1 part on [Stack Overflow](http://stackoverflow.com/) about the Python programming aspects. (Or maybe not, I don't know for sure, but I just want to give you fair warning; we'll see how it goes.)
@Eduardo It's in the wrong direction :-).
@RussLenth The common name for combining inference from tests for difference (H$^{+}_{0}$) with tests for equivalence (H$^{–}_{0}$) is "relevance testing" (See my answer here: http://stats.stackexchange.com/questions/108911/why-does-frequentist-hypothesis-testing-become-biased-towards-rejecting-the-null/108914#108914). See there also: the interpretation of the rejection of both H$^{+}_{0}$ and H$^{–}_{0}$ implies *trivial difference*: the difference is significant, but you have *a priori* said you don't care about differences that small.
I'm a bit unclear what you mean by shuffle the outputs?

By means of both files, do you mean getting the mean data point for the whole file?  Adding up all data points and getting the average?

Maybe it's my weak understanding of the terminology, but I'm a bit lost with your explanation.

My customer would welcome more than one comparison mode.  So if I come up with more than one statistical method, they would be happy.
Have a look at the following posts, maybe they are helpful: [first](http://stats.stackexchange.com/questions/33013/what-test-can-i-use-to-compare-slopes-from-two-or-more-regression-models?lq=1), [second](http://stats.stackexchange.com/questions/59327/how-to-compare-two-regression-slopes-for-one-predictor-on-two-different-outcomes/59331#59331), [third](http://stats.stackexchange.com/questions/1735/method-to-compare-variable-coefficient-in-two-regression-models?lq=1).
The harmonic mean is even smaller.
CCC and Harrell's c can be used with continuous outcomes. CCC also has a repeated measures implementation. See the references/links I added in the question.
Yashka - The two sample K-S test *can* tell you about location-shift type difference; you can get either a point estimate or a confidence interval. It won't tell you the individual locations though.
There are many posts here relating to this topic more or less (e.g. [this one](http://stats.stackexchange.com/q/539/805)). You might find this useful reading if you have access to it:  Velleman, P & Wilkinson, L (1993). "Nominal, Ordinal, Interval, and Ratio Typologies Are Misleading". *The American Statistician* **47** (1): 65–72
Check out the help for your flavor of `predict`, such as `predict.lm` or `predict.glm`.  (This actually is general advice that applies to many stats platforms, but it definitely works for `R` because I just had occasion to look this up: you want the `newdata` parameter.)
Also, which optimizer can handle bound constraints? If you know, can you point me to a reference which optimizes an objective function with symmetric and positive definite constraint enforced. That would make my mind at rest?
A first check would be to calculate the rank of your design matrix. If it's less than the number of columns, you probably need to combine and/or recode appropriately.
I wish i could up-vote more.
To what "time" do you refer?  Does the 75% include the first ball you drew?  Do you mean you know *exactly* eight balls are red or that you know there exist eight red balls in the bag?
@kjetilbhalvorsen, in English "welcome" starts with a *W* ;-).
(+1) The 1st display really conveys a lot of information, and it will allow to see clusters (wrt. time or individual), if any.
Let me emphasize (since this overlooked often) that the convergence proofs **need (squared) euclidean distance**, so that the distance function and the *mean* function optimize the same criterion. If you use a different distance (actually, you should not use a distance, but "least sum of squares") you may lose convergence in k-means.
Thank you Alecos.
Also look at [named entity recognition](http://en.wikipedia.org/wiki/Named-entity_recognition), and the more general task of [Information Extraction](http://en.wikipedia.org/wiki/Information_extraction)
I also know people are researching these ideas. Therefore, heatmap may be the only way to go.
@learner, if you want summarize the strengths of intercorrelations among _p_ variables in a single number you could compute, for example, (1) average abs. correlation or (2) geometric mean abs. correlation, or (3) determinant of abs(corr. matrix). These three are very different ways to conceptualize _p_-variate association by a single value.
It may also help you to read some of my other answers. This is especially close: [graphing a probability curve for a logit model with multiple predictors](http://stats.stackexchange.com/questions/31597//31600#31600). For a more general understanding of LR, these might help you: [interpretation of simple predictions to odds ratios in logistic-regression](http://stats.stackexchange.com/questions/34636//34638#34638), & [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909).
@gung, you're right. I've changed so it now matches the standard nomenclature.
Is this what you're looking for? http://www.weibull.com/DOEWeb/confidence_intervals_in_simple_linear_regression.htm
Maybe it should be a comment? (even if it is a good comment :) )
I did think about this but as in each round you can get +2 ,1, 0 or -3. I did not think it would work? Each player plays 2 games each against a different subject. Would it make sense to model it as a lmm with Subject and Pairing as random intercepts such that each pairing contains 2 subjects and each Subject is in 2 different pairings with different subjects. This would mean there is 80 pairings and 80 subjects?
I suppose that doing this for each age group in each region will provide me the sample-size I need (?).
Some fields talk of "criterion" variables.
@subhashdavar, maybe you could be more specific in your question; what are your thoughts exactly? What part confuses you? I think Michael's answer and Procastinator's comment are pretty clear.
Welcome to the site, @CKI. If you can upload your data to someplace on the web, you can post a link here.
That this is the right formulation for *frequency* weights is clear when you consider doubling or tripling, etc., individual data values: the factor $\\exp(l(\\theta, y_i))$ (there's no subscript on the $\\theta$) in the likelihood becomes $w_i$ multiplicative factors amounting to $\\prod_{j=1}^{w_i}\\exp(l(\\theta, y_i))=\\exp(w_il(\\theta,y_i))$ whose logarithm is $w_il(\\theta,y_i)$ and there you have it. Different developments (based on conditional probability calculations) are needed for probability weights.
+1 for Golub & van Loan
@Charlie Here, it's natural to regress the residuals (and their squares) against the linear coordinate, $(x+y)/2$ in this case.  The idea is to see whether there are any trends in the distance from the line as a function of distance along the line.
A bit of a rant, but "special effects" is not a good motivation to visualization; good plot should tell a story about the data, like look, this is bigger, this is frequent, this is strange, this is correlated, this separates into clusters, etc. If you make a plot that just "looks good", you go nowhere, i.e. the fact that it is based on data is meaningless. Think of all those internet graphs which only show that internet is a circular tangled blob.
@Jane Please add the above information to your question using the 'edit' link below the tags and delete this 'answer' as it really does not answer your question.
In this document http://www.inf.ed.ac.uk/teaching/courses/inf2b/learnnotes/inf2b-learn-note07-2up.pdf there is a nice derivation of the naive bayes algorithm for document classification
I will do some reading about these tests. Thanks for the references.
is right.  Two dv's could be otherwise independent, but both influenced by the iv's.  As a result, they would appear correlated in the raw data, but the residuals would not be, and that's what's more important.  Good catch.
@Alex, Greene, Econometric Analysis, 7th ed, "Section 4.3 Finite Sample properties of least squares"
user115027 Since you suggest it, I do imagine that you might. I would not. You might do better to read about the distinction between probability functions and cdfs before tackling this question. How could you find the probability that it's less than 2?
If you are saying that you have no data on the 335 horses without the disease, but only on the 47 with the disease, then all you can do is describe the diseased horses. That may still allow some analyses, most obviously a null hypothesis is that disease is unaffected by time of year. You may have (should have) at least a qualitative idea about what breeds there are and their (rough) frequencies.
Consider for example canonical correlation analysis
The spreadsheet has two apparent (small) errors: the $\\sqrt{n-1}$ should be $\\sqrt{n}$ and the critical $t$ value should be $1.656.$
Define 'good', for your purposes. What aspects or features of a model are important?
Charles,  as you may noticed,  I have no math background.  Now,  things are far more clear.  Thank you for the informations and patience.  Beyond accept,  would up vote.  Think I'll need to wait until I get 15 :)
$A+B$ is not the _joint_ distribution of anything, and the _random variable_ $A+B$ does not have a joint distribution; it is a univariate random variable. To calculate the CDF of $A+B$, use $$F_{A+B}(z) = \\int_{-\\infty}^\\infty f_A(a)F_B(z-a)\\,\\mathrm da.$$
Cross validation?
I realize that hypothesis tests are asymmetric, but I'm wondering if there is a normative difference. Should I feel safer about type I or type II error?
It also occurs to me that the assumptions in step 4 could be guided by the number of estimates and how much they vary. If 10 people all estimate very similar values for low and high workloads, presumably these estimates can be considered more reliable than if, say, 3 people come up with wildly different estimates for the same task.
Nice answer, +1, but I gave the other one a best answer since I read it first and it answered my question first.
But "subjective contour illusion" sounds cooler.
Could you please clarify how did you get the expected values, and what hypothesis are you trying to test. The correct degrees of freedom depends on that. Most answers assume that you are interested in independence (is the effect of period the same for each country?).
I didn't think simulations would be a fruitful approach when proportions are *very* small (on the order of 1 in 50,000 or less).
For your 2 points, yes, it wouldn't be too onerous, but what about the 2 points made by @gui11aume ? I think I'll write out a model answer with all the additional points that have been raised in this thread and see how it looks. It will be a good consolidation exercise anyway...The main point of it all, is to learn, and a mark here or there isn't really relevant; the good thing about this kind of post-mortem is that it makes me think even more about it, which is great :) Thanks again !
It sound like you have a partition that represents the 'true' clusters. Then you can use a criterion like Variation of Information or the split/join distance (or, if you must, the Jaccard index although I do not recommend it as it is based on counting pairs and affected by cluster granularity in a quadratic manner). See e.g. http://stats.stackexchange.com/questions/24961/comparing-clusterings-rand-index-vs-variation-of-information/ for a discussion of clustering comparison criteria.
Thanks for your well-articulated answer.  Three very brief comments: **1.** Computations like yours but with other quartiles would yield different results, such as $q_{11} - q_{12}$ for the mean difference; I'm curious how to choose among or combine different options (see my Remark R4). **2.** Larry Hedges pointed me to Hedges & Olkin (1985, p. 93) for relevant ideas based on order statistics.  **3.** I share your concerns about the sampling variance and non-normality (see my Remarks R5 and R6).
This is a comment not an answer.
Great idea. I hadn't thought of that. It solves all my problems at once. Thank you!
This is strange. I ran your example with the `iris` dataset and it worked (caret 5.15-044 on R 2.15.2). Out of curiosity, can you post the exact error message? In any case, this question is off-topic on this site, and will be too localized for Stack Overflow.
Do you mind sharing resources you have already collected on the topic?
Probably it is about the standard error of the correlation coefficient and a p value for testing whether the two variables are truely correlated?
+1. The joint distribution is an easy consequence of the results at [Joint PDF of functions of order statistics](http://stats.stackexchange.com/questions/78546). The convergence is pretty fast, too, as attested by examining the bivariate and marginal distributions in this `R` simulation for $n=30$: `n <- 30; sim <- c(0,n) + c(n,-n) * apply(matrix(runif(n*1e4), n), 2, range)`.
> i agree with you btw (i own both books). On the other hand, if budget is a binding constraints, you might want to diversify.
Ecosystem analysis, perhaps? I'm not aware of any specific examples though. Also, I get an error on the link. Can you explain what the graph is showing? specifically, what do the colours and line lengths depict?
I think this is a statistical question suited for crossvalidated.com. I have suggested the question be migrated. Have a look at multinomial logistic regression.
@Cardinal. You were correct. Pairings should be $(x_i, x_{i+400})$, $i=1, 2, \\dots, 400$.
As `rstandard.lm` is exported from the `stats` namespace, you can just type its name at the prompt without needing `getAnywhere()`.
you should define "convenient". I find that as a very unsatisfactory answer because its vague. Do you mean its convenient because the squared loss function is convex/concave and thus its easier to find a global max? or what do you mean? Or do you mean its convenient because it otherwise transforms a NP-complete problem to something manageable? Or did you mean?
Well, even if it becomes an optional parameter, the question remains as to what this parameter is actually doing. I guess one thing scikit is missing, is a decent mathematical documentation.
I you are sampling from a normal distribution the probability of getting two equal values is zero. If you use reasonable rounding, you may get a few equal values in a large sample.
@whuber, note that $-2\\log(\\sin(\\mathrm{arctan}(x))) = \\log(1+x^{-2})$, which relates the form of the cdf closer to that of the pdf. It's also interesting to note that this pdf is asymptotic to one-half the pdf of a standard Cauchy. So, the main reason for its use would seem to have to be for its behavior around 0.
It turns out I can do it in the JMP client that I am using to access the SAS server. We are learning a continuous approach this week in class, so I will have the correct tool to use.
So does that imply that if I want simply "low" vs. "high" the ICC is best?
Can I ask a question? how can I calculate the normalized gradient if I have already obtained the gradient vector? If the gradient vector is numeric large, I need to normalize the gradient. Could you give some intuitive examples about normalizing gradient? Thank you!
There are a few papers about that mention it, for example, [this one](http://cbio.ensmp.fr/~jvert/svn/bibli/local/Tseng2009Coordinate.pdf)
What would be regressed against what? How would tests of autocorrelation and breakpoints answer the question?
Thank You! This worked, although I had to drop a comma in the call to data.frame.

    stats = ddply(
    .data = ords
    , .variables = .(Symbol,SysID,Hour) 
    , .fun = function(x){ 
        to_return = data.frame(
             s = sum(x$Profit)
            , m = mean(x$Profit)
        )
        return(to_return)
    }
    , .progress = 'text'
)
Thanks for responding.  It is not necessarily true, however, that the raw data are required to estimate the required quantities, at least in most circumstances (and particularly for network meta-analysis).  A good resource if you're interested  is Gleser and Olkin (2004).
More details would be very helpful. What/how are you updating? What are your observations and how are they related to price? These details are useful in determining how to setup priors.
+1. I would also propose SCAD as a potential variant of a penalized likelihood approach.
Thanks, Glen_b!.. However, I am not sure I follow what you say. Could you please develop a bit more, or maybe give me some pointers?.. Thank you very much!..
+1 Embarrassingly simple, isn't it? :-)
Without more information, how would we suggest anything? By "COM Poisson" do you mean 'compound Poisson'? If so, what is it about your situation that would suggest it? Which one (i.e. what is the mixing distribution)?
Thanks for the answer, I wasn't clear. I'm not comparing the Cox model's hazard ratio to another model's, but to the definition of the hazard ratio as simply the ratio of two hazard rates (-log(frac. surv pop1)*time)/(-log(frac.surv pop2)*time). I have seen both methods used when either could be applied, that is, where the cox model is univariate, so one variable isn't being assessed relative to another.
@Ga I pointed out in a comment to your deleted post that a matrix notation for bivariate probabilities is (and long has been) in use.  Most statistical software will produce such tables, e.g.  It makes perfect sense and is quite clear: it's difficult to imagine someone misunderstanding what is meant.
We welcome questions like this, @Samster, but we treat them differently. Please tell us what you understand thus far, what you've tried & where you are stuck, & we'll try to provide hints to get you unstuck. To better understand the process, you should read the [wiki](http://stats.stackexchange.com/tags/self-study/info) for the `[self-study]` tag.
all online algorithms of this form (update & downdate) suffer from precision issues like this. I had similar issues in 1d resulting in negative estimates of variance. My suggestion would be to keep a circular buffer of the last k vectors observed, and when `cholupdate` fails, recompute the covariance based on that circular buffer and eat the cost. If you have the memory, and can stand the occasional time hit when this happens, you will not find a better method in terms of accuracy and ease of implementation.
I'm open to that and any other method.
Can you assume any relationship between the mean of $y$ and $x$, say linear?
A) I see you edited the question so there are now two spaces in data. The R formum I was talking about: http://stackoverflow.com/questions/tagged/r
By the way, it also seems a bit different from the definition of (generalized) degrees of freedom talked about in Ye's paper.
@Mermex . I have "seen" data like this but it might help (me) if you actually posted one of these difficult series along with any indicators that you might think are important.
In proposing a $2$ by $k$ table you suggest there is some importance to assessing different values of $i$ and $j$. However, your statement of the question does not imply any need for that. This makes me suspect there's more going on here than your abstract description would suggest and that the details may be important.  Could you share with us more information about how these data are obtained and what the values of $i$ and $j$ might mean?
No, because 'behave similarly' means 'be Pearson correlated' for PCA. Amplitude does not really matter.
You're right, but would you mind explaining why orthogonality has any bearing on whether or not to discard variables in advance?
Yes, the distribution of $X$ usually will be different from that of $X^2$. The only example I can think of where the two are the same is bernoulli.
They have ratings break downs by gender and age groups as well.  I suspect that they may average within those groups and then taking the average of the group means for their reported score.
Perhaps I missed something. What analysis were you running?
Thank you for the quick reply. I just realized that I put on the wrong question. This edited question was actually what I needed help with. Any ideas on this one?
The `dlnm` package is quite extensive, happen to have an example?
@Christoph_J eegad, you are surely right ...
1. the `.Sigma` stores the variances, you want the standard deviations. 2. Effectively you are correct. It returns a vector of length $k$ that quantifies the probability that a point $x$ is part of component $i$, $i = \\{1,2..,k\\}$ given you have $k$ components. That's why here we get 3 numbers adding up to 1; any given reading *must* be part of a component.
if x is your data, ks.test(mahalanobis(x,colMeans(x),var(x)),pchisq,p), but actually you should use a re-scaled beta (though chi2 is close enough in practice)
It seems that [this issue](https://github.com/lme4/lme4/issues/192) would be helpful. If not, you can leave a comment or open another issue to `lme4` developers. Anyway, it seems that this question is only about programming and thus off-topic.
Here's another link, with Mplus: http://www.statmodel.com/discussion/messages/9/258.html?1237657036. I didn't find anything for SPSS specifically. Your suggestion of writing some SYNTAX is the best way to go, IMO.
For each person, there is a  seperate $a_i$, yes. They are independent, but all come from standard uniform distribution $a_i \\sim U[0,1]$ for all $i \\in K$.