tttjjj/elt_bench huggingface dataset:
I want to compare how dissimilar the communities are by FISH_CLASS (resident or migratory(two-levels) and REACH (Upstream /Downstream (two-levels) ) by BASIN (Superior, Huron, Michigan (three levels)).  Does this explanation make sense?
Welcome to our site!  Did you read over the other answers before replying?  You might enjoy the thoughtful analyses contained in several of them.  They include the same calculations, so I am wondering what part of your answer or its form of presentation represents something newer or better than what has already been posted.
I realize this is a little late, but how are you selecting the ARIMA parameters?  Obviously, if, for example, $(p,d,q)$ in model 1 is different from $(p,d,q)$ in the original model there will be some confounding of the effect of the external regressor with the change in $(p,d,q)$.
Not very helpfull site actually, but my own search showed that there is nothing else out there. Thank you for the link and your help.
localy weighted regression sounds good. The description in the help is too technical for the figure text. But I'll keep the reference if someone is not satisfied with the simpler text. thx
In short: yes there are other criteria. In long, you might want to have a look at the papers cited in [the footnotes of this document](http://climate.nasa.gov/evidence/). That should at least help you slice you question into manageable [bits](http://meta.stats.stackexchange.com/q/1537/603)
Hi Peter! Thanks for taking time to answer :) I have added the interaction factor to the model with the ad and the exhibition but the interaction is not significant. Maybe I should specify that I am looking at a specific kind of ad, it doesn't change throughout the period analyzed. Does that help?
It depends on which program you use and which options you use.
so if you're getting different regression outputs for each test right? i may be misunderstanding, but can you're question also be stated as "does this model A offer more explanatory power than this model B"?
And it soon got protected on math.SE, [Does this question even have an answer?](http://math.stackexchange.com/questions/76491/does-this-question-even-have-an-answer) :)
+1. In my opinion, the 'wandering about' is the most intuitive analogy among the ones listed on this page.
Yes I believe so. That was another question I had in mind, so thanks for bringing this up! If say my 25 univariate regression analyses can be grouped under 5 general categories, then would I divide alpha=.05 by 5? (based on Bonferroni's correction for now before I look up Benjamini-Hochberg correction). Would you also be aware of any relevant methods/stats papers that I can cite to justify my correction? Thanks for your help!
:EpiGrad Sometimes the help you get is proportional to what you pay for it. One has to take into account opportunity cost when acquiring software and/or consulting expenses. The reason I charge a lot (relatively) is that I have  a long record of having made mistakes ( which I don't repeat ! ). Thus people who buy my time are buying somebody who has made all the mistakes and thus I can now get on with the "good work". This of course is said tongue in cheek but IMHO I would never look to beginners when I needed a solution ( for anything ! ).
Randomization/permutation tests are almost designed for small samples. As far as I know, it doesn't depend on any theory at all. If samples are very small, you can list all possible permutations and have exact p value. For small samples there is `permtest` in the `BHH2` package.
@JoelW.I have plotted the data points in a simple scatter plot and the difference does not look particularly noteworthy (for one particular data set). I also plotted means with 95% CI and the means overlap to a large extent. In this data set, the means all fall into the CIs of each other. 
Does this help?
Why do you want to classify them? Aren't you better leaving the scores as they are for analysis?
I'm afraid I don't understand the question.  Could you restate it or elaborate it?
@DrewSteen If you do `lm(response ~ conc + conc*day:treat)` you have too many parameters (which is why you get `NA`s from that model).  Did you mean `lm(response ~ conc*day*treat)`?  I'm not quite sure I understand what model you want to use.
Ok - done - I didn't know you were supposed to ! thanks for letting me know :-)
Of possible interest: http://stats.stackexchange.com/q/31967/930.
Please provide some code of what you've tried.
The class label adds the ground truth, real world knowledge. It is therefore the 'gold standard'. You can do your clustering and then see for yourself how well it performed in putting the observations with the same labels together.
MI is a measure of association between two discrete variables. It's not really that common a setting in general statistics (could be in some specialized sub fields). But within that setting, I see it used frequently enough. Certainly, when I encounter applied people using Pearson correlation on bivariate discrete datasets, I point out MI to them.
"The EM algorithm is a special case of a more general algorithm called the MM algorithm", [see](http://arxiv.org/abs/1104.2203).
Do you have any domain specific knowledge about how the "five and more times" category would be distributed across counts of 5, 6, 7, etc.?
Ha! Thanks a lot @DilipSarwate ! Since they're then both correct, is the lecturer one more common somehow?
EdM sends his thanks for this elaboration on his answer.
Thank you vey much for the answer, this is extremely useful. Thank you again, it really helped me.
...cont...example: All the observations are measuring or looking at something involving blades of grass.  And actually - really - to be honest it assumings every things/measurement/idea/noun/etc. is related to everything else no matter how weak that relationship may be.  It may be so weak it's essentially undetectable - and this comes from the fact that the use of the words "dependent" or "indepdent" are actually misnomers - but must be used (for practicality's sake) due to the limits of current human knowledge in math/statistics.
Logistic regression is so similar to OLS regression that you can use your experience with the latter to inform your thinking about the former, Ariel.  One way to assess high-leverage points is with a sensitivity analysis: if the estimates change little after removing high-leverage points, they're ok; otherwise, the results depend materially on the high-leverage points, implying you have some hard choices to make (and some careful thinking to do).  Also, categorization might not matter: if you do it and get the same or similar results, you're in good shape no matter what.
@user4211, do you ask about distribution of minimum and maximum of any sample distribution, or only normal?
Because none of that chapter is readily available at your link, Michael, could you provide examples or descriptions of these plots?  Because the book focuses on statistical methods, rather than graphical representation, it would also be interesting to know whether the authors offer anything innovative or especially useful concerning the graphical representation of binary time series.
Does the measurement model specify that those factors should not correlate? How about the higher-order factors: are they intended to correlate? Does the problematic factor at least correlate more strongly with other factors that share its higher-order factor? If these factors are supposed to be less correlated, it sounds like a problem for discriminant validity, but if correlation is intended, it could be good news. It's probably not possible to determine whether the model or the instrument is more flawed from what you've said here. That would depend more on how the instrument was designed.
Just as a pointer for anyone looking at answering this question - my answer to Alexis' previous question (which is linked in the above question) has links to several references with some discussion of the history, each with a number of relevant references. You may like to check those documents and their list of references.
Okay I contacted the Author. Thank You very much!
@Daniel: It seems like you won't need any new data (beyond what you already have) for this approach. It seems you will need the true/false positive/negative rates, and you don't need any thresholds.
You will need much more explanation. What do you mean by "trend detection" and how are you defining "trend"? It would help if you explained your specific problem rather than ask a general and vague question.
How many years of data have you got?
If your interest is in making substantive interpretations from your model, then what predictors "are worth trying" should be based on an _a priori_ hypothesis you have about the data.
Totally unclear. All correlations between 100 variables? Correlations between each of 2 factors and "its" variables? Or what?
PCA can sometimes be used to find intuitive meanings for the features - e.g. [eigenfaces](http://en.wikipedia.org/wiki/Eigenface).
(+1) I think this is better than suggesting to the OP to download R. I was not aware of this online resource.
Are you perhaps interested in the pros & cons of different ML algorithms for situations like this? Note that CV is not an R Q&A site (you may want to read our [FAQ](http://stats.stackexchange.com/faq)). Regarding your outcome variable, do you have survival durations or just yes / no? Might there be any censoring?
@Imorin can you suggest a few?
Please edit your question to show reproducible code.  In particular, what is `an`?  When you do that we can migrate your question to SO where it is better suited.  (If you don't make such a a change the SO community would just close it.)
If you mean, it won't go to even 1.01, then you're right.  However, if you train to values that are all between 0 and 1, then eventually the network will train to always give values close to that range.  You can clip the output to 0 to 1 if you want, while leaving the region in between linear, but in most cases if it gives an output of 1.01 or 1.02 occasionally this is not an issue.
Your idea is great and I will take it into consideration. But I tried also to make my question valuable for others by abstracting from my case and comparing normal with beta distribution. So I believe your answer is more a "comment" than a real answer. A moderator might explain if I am right or not. I just don't want to be selfish and adhere to the spirit of stackexchange...
@RussLenth Counts of the least frequent categories are 1. Original post is really a simplified description, actually I have thousands of categories and nearly 1/3 of them are observed once in 3 million sample. If the exact numbers are important, I can look it up.
@Kate - That goes into the pile of "use sample sizes to estimate the uncertainty".  Uncertainty estimation without accounting for sample size is like an airplane with an engine, but no wings.
Oh, sorry, I missed that; they're both power transformations of course. Note that they use different information to generate an estimate (so they won't be the same) and in both cases, there's generally a wide uncertainty around the central estimate (though only one of the two reports any). Look at these: (1) `summary(carspow <- powerTransform(dist ~ speed, cars))` (estimate 0.43, interval is between 0.21 and 0.65) vs (2) `slp(lm(dist ~ speed, cars))` (estimate is 0.54). In each case you should choose a nearby "round number" ... which would be 1/2.  So they're different but both suggest $√x$
First Google hit: [Analyzing the Subjective Interestingness of Association Rules](http://bit.ly/rvV5xh), to get users (like me :) familiar with this concept.
Also, just to clarify terminology: statisticians use the term fixed effect to refer to anything that is not a random effect.  So in your question a statistician may call Z1 and Z2 fixed effects.  Economics and other social sciences use the term differently to refer only to variables that account for unobserved variation that is treated as non-random.
I've added an `edit` to the question.
You probably want something like the [$F_\\beta$ score](http://en.wikipedia.org/wiki/F1_score).
I understand that the problem was related to specifically MCMC, and not Bayesian inference, but in the context of Bayesian landscapes I find MCMC to be very understandable.
Using two boxplots comparing the two algorithms might not be sufficient to show the difference. You have paired data, and thus it would be nice to retain that. How about a scatter plot with a line with slope 1 and intercept 0?
Another concept is just changing the null hypothesis, see [equivalence testing](http://www.graphpad.com/www/data-analysis-resource-center/blog/testing-for-equivalence/).
DBSCAN is a popular clustering algorithm, but the question is about whether the claim that a distance matrix follows a poisson distribution is correct (see quoted section), not about the algorithm itself. I have edited the title to make it clearer (I hope)
It may work, or not. Depends on your data. "matching row proportion" for twitter data is not working well.
(cc @BabakP) These are both good answers and are obviously correct (+1s). Just an `R` programming trick: you can also use the `findInterval()` and `cumsum()` commands to simplify the code and, more importantly, make it easier to generalize to a different number of dimensions. For example, for an input vector of means $\\mu$ (`mu`) and variances $\\sigma^2$(`s`), and mixture probabilities (`p`), a simple function to generate n samples from this mixture would be `mix <- function(n,mu,s,p) { ii <- findInterval(runif(n),cumsum(p))+1; x <- rnorm(n,mean=mu[ii],sd=sqrt(s[ii])); return(x); }`
+1, it sure makes you helpful for answering people's questions.
Can you confirm you mean the mean/median of a variable from two groups which are independent?
As a side note: I don't think you mean $|\\theta_1 - \\theta_2| \\mod \\pi$. Rather something like $\\min\\{|\\theta_1 - \\theta_2|, 2 \\pi - |\\theta_1 - \\theta_2|\\}$.
There's not enough information here to discern what's happening. A reproducible minimal example (i.e. the smallest example that shows the problem you describe) might throw light on what's happening.
+1 Very well reasoned and explained.
The first 'problem' is orthogonal to causal inference issues. Conditioning can, but does not necessarily, *identify* the/a causal effect by blocking backdoor paths, thereby solving the second problem, at least in some circumstances.  And instrumental variables are used when such conditioners are unknown but exogenous causes of the supposed cause can be found.  Accurately representing uncertainty in these analyses with p-values or whatever is a separate issue.
Any idea how to do this in the example I gave?
@Billy Also note that the code you provided does *not* define a repeated measures ANOVA model, just a regular 2-way ANOVA.
Hi there, what techniques/models have you used so far for analysing your data? Can you also show the variables you used in each model?
Unless I am missing something, this is basically shoe-horning a classification problem into a regression problem and then evaluating it using classification metrics.
@Benjamin: you can. If I use instead PROC IMPORT OUT= WORK.FOO DATAFILE= "foo.csv"           DBMS=CSV REPLACE; GETNAMES=NO; DATAROW=1; RUN; I get the same result as above.
This is the wrong forum for debate, but I'll try to explain.  I read in the press that some forensic scientists have said that fingerprints are infallible, DNA evidence has been misapplied in criminal cases as a result of failing to account for base rates, and expert legal evidence has been presented with the unjustified assumption that the events in question are independent.  In that context, if the prosecution fails to address major issues that are relevant to a criminal case, so that "reasonable doubt" remains, the jury should not convict.  To do that they must have the relevant concepts.
@Glen_b For your first question: conditionnally to the true weight, the weighings are supposed to be independent. For your 2nd question: In my intuition, the inferences are compatible if the overlap between the two posteriors is "not to small". Maybe i can consider if the 95% HPD overlap or not ? Thanks for clarify my point.
I would argue that if the mean increases your stream has more noise. The standard deviation of the entropy says more about how stable your calculation is. It could be used to create reasonable cutoffs (i.e. z-score) before you take action.
Ah, I think that really helps. I guess I just needed an idea on where to go. I'll give it a go.
What exactly does `points` measure and what relationship does it have (if any) to a player's "score"?  (Or is the score a separate variable altogether?)  Note, too, that your ability to assess "influence" in any material or causal sense depends on how you collected these data: if they are just records of the outcomes of contests, then most likely you cannot estimate influence at all, but you might be able to identify some *quantitative relationships* among the variables.
Different people undergo each treatment. The study is a multistage game of chance, culminating in participants choosing between Option A - receiving a certain amount of money, or Option B - taking a risky gamble with the chance to win a larger amount of money. The different treatments vary aspects of the game, and are expected to result in different distributions of A and B choices. I need to compare Treatments 1 and 2 to Treatment 3, separately, in order to see if they are significantly different. So A or B refers to their choices ie. the data outcome.
(+1) this is it ! By duplicating instances with different labels and instance weights (aka label certainties), one can also apply algorithms like Random Forests, Naive Bayes etc.. Instance weights are so common, weka has to have learners which support it. Rapidminer (competitor of weka) does. By setting the certainty to 1, one can even model "crisp" multi-label-problems.
"Different" is very subjective. Please amend with i) the variance of the two groups after e-Views' intervention, and ii) the resultant "different" SE from the t-test and robust t-test so that the users here can give you more concrete advice.
Can you add some more details to flesh this out? I'm not sure I follow your question.
Yes, and @confusedstat you probably want also want a test for the overall usefulness of the categorical variable so you can test all categories simultaneously rather than just one at a time (which is all your current readout allows).  This can be done via an F test comparing the model with the variable in to one without the variable - often a standard output from an ANOVA table based on the model.
Another way to frame your problem is that you are looking for an $n$-dimensional analog of *equal-area* (authalic) projections of the sphere. A straightforward way to check your proposal is to verify that the Jacobian of the transformation has constant nonzero determinant.
Unless you mean you're also subsequently turning it into a factor with *labels* 1 to 8, binning it just results in the bins previously indicated. If your bins are all constant-width, why use 1-8 rather than a number related to the bins, like the left ends or the centers?
Why don't you just train (n-1)/2 binary classifiers yourself by calling libsvm (n-1)/2 times, each time with a different active class?
Hi, Thank you for your suggestion. Is there any direction if X and Y are not independent?
I see! I will certainly keep that detail in mind.
"_Recent advances in cluster anaysis_" ... you could name a few here, or give some links?
+1 Is $N=n$ in Lemma 2?
A very underrated method: plot that stuff. More specifically, plot the average counts over all weekdays for each hour between 7 and 20 - maybe you see spikes during lunchbreak or something similar. If you do, confirm this is a phenomenon present for all weekdays, not just one or two. And then you have your pattern.
@MichaelChernick yes, that's right. I neglected this but should not have.
Can you spell these out, & say more about your situation & your data?
What would a more predictive approach look like?
Is this time series data? Or what? What are x, y and z?
Define criteria for what is appropriate. In its current form the question is not answerable, since it does not have a definite answer. See the FAQ.
Thanks for your reply. Do you mean only including a single covariate? A presume so, because if I don't include any covariates I don't see where I should read actual crude ratio in the output? (BTW: I read the fine article, but I didn't read anything in there particularly about crude ratios)
Dan, Thank you for your kind response. I'm not sure I got it completely - let me explain the query in more detail. I'm trying to run logistic regression on Titanic dataset.
Shameless promotion: I just wrote a blog article about Bayes and pyMC *literally* 1 minute before you posted this, so I invite you to check it out. [The Awesome Power of Bayes - Part 1](http://camdp.com/blogs/awesome-power-ibayesian-methodsi-what-they-didnt-t)
The use of *any* test on this dataset is questionable. In what sense could the results be modeled as iid samples from two populations?
What exactly is `iv1` and `iv2` ? Maybe I am looking it in a wrong way, but are you sure you need this random effect structure? (aside the "keeping it maximal" rationale) You allow for correlation between the intercept deviations of `item` (or `subject`) and the  `iv1*iv2` effect deviations within the levels of `item`. Those are a lot of parameters... Maybe just `(time| item)` (and `subject` obviously) are enough to test your assumptions? (`time` and `1 + time` are equivalent specifications by the way) Check your model residuals.
Can you not just calculate the R^2 on each test fold? i.e. how well the model fits the test cases for each fold.
Thanks Meg. Actually, I don't know and I don't have any reference. it would be great if you give me any. Unfortunately, I am not using SPSS. I am used to R, is it possible to test all these in R also?
It's good to be confused about (b). I suspect the asker wants you to use a formula for sample size based on assuming the population SD is \\$500. In the real world, you would have to worry about the high likelihood that the true SD may be quite different from \\$500, because this value has been estimated from a tiny sample of 10.
Because oblique "rotations" are subsequent to the PCA, they cannot "see" the kind of situation illustrated in the question and therefore appear to have no more ability to identify the two components than PCA itself does.
Don't expect a CLT-like universal result.  For instance, the answer for uniform(0,1) variates will be *very* different from the answer for uniform(1000,1001) variates!
+1 Nice answer; I'm only curious why do you say that small number of docks imply lower number of important variables -- doesn't it seem a bit overfitty?
Thanks. This will work nicely and it looks like a great book too; I'll probably actually buy it (!).
Don't call yourself stupid - for starters, you immediately imply I'm wasting my time. Second, that wasn't the point...  I can't guess what you don't understand about what it says, since it explicitly tells you where $L$ and $L^T$ are in the output of the function. If you can't understand what it says (which is fine), and *you also can't tell me where the difficulty lies in understanding what it says*, how will me explaining the information again help? If I can't guess where the problem lies and you won't narrow it down, what am I to do? (Lastly, what does "how to composite" mean?)
Great answer onestop - thank you.
thanks. If i want to check whether X1 alone or X2 alone can be a good predictor, i just need to do anova(modelwithX1), anova(modelwithX2) and then read their P-value right?
(1) When you speak of PCs, am I right in assuming it is related to classical PCA and not factor analytic methods? (2) Also, it seems the 24-item questionnaire has to do with symptoms experience or severity (with Likert-type responses categories), right? (and (3) this is not a cross-over design, but two-parallel groups?)