tttjjj/elt_bench huggingface dataset:
Thanks @Scortchi, raw scores are available to me, in fact I even have the domain scores that are combined to make up the overall IMD score so I'm tempted to split them out and add as separate variables.
E is the ID of the Brood, and I have numerous non-independent data points for each brood so this needs to be accounted for. I am REALLY new to models, but have been trying to grasp them, but I literally have no time left. I only understand how they can be used in terms of looking at how different effects contribute to the measurement (y) and then how to decide which models are the best using AIC and AIC weights to realise which models best explain the values of measurement 't'. For more detail you could look at my other post. My sample size is so small that no other stats will work.
How do you support such a broad generalization, Chris?  For example, many quantities in "biological sciences" have chemical meanings where the logarithm is the most natural (and, arguably, the most readily interpretable) way to express values.
Did I get you right that you concur with this words from Wikipedia's Mann-Whitney talk page: `the null hypothesis of Mann-Whitney U-test is not about the equality of distributions. Is is about the symmetry between two populations with respect to the probability of obtaining a larger observation`. And so you don't agree with @Greg's answer, right?
+1, Welcome to the site, @EpiFunky, that's a nice contribution; Fox is really great. One of our goals here on CV is to create a permanent repository of actionable statistical info in the form of Q's & A's. As such, we worry about link rot. Would you mind giving a precis of the pdf in case the link goes dead in the future? (BTW, info like this is given in our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which you may want to read since you're new here.)
@D.W. Yes, thank you, I know how to perform the test but I was confused as to what to use as the Contingency Table. Since we have only two signals, I wasn't sure whether a 10 x 10 table makes sense. This is in fact the answer to my [other question](http://stats.stackexchange.com/questions/24439/how-can-i-perform-a-chi-square-test-for-independence-on-signal-samples). You can answer in some detail over there if you like, and I'll mark your answer as correct. Thank you.
If you're happy with one of the answers perhaps you'd like to accept it?
Does it make sense to not have an intercept? Does it vary around 0, or does it start at something completely different? If it doesn't make sense to exclude an intercept, that test specification isn't really relevant.
Thank you Hong. I remember that methods to transform some objects to hclust demand that the dendrogram is ultrametric - I wounder if this has to do with what you wrote.  In any event, thank you for the answer.
@Tal .. thanks :-) But I wanted to know more and possibly expand the Wiki article, which is not too informative.
@hung Okay, the numbers are a little higher than Fama & French (1993) (not sure about constants, F&F don't give those) - but not beyond the realms of possibility. Please, do check that table for yourself so you can put your mind at ease. High explanatory power is likely in these regressions since regressors and regressands are both constructed from the same underlying random variables (ie stock returns). Point taken about "independent" - I tend to use regressor/regressand to try and avoid the confusion :-)
@gung, thx, edited
If some categories are very rare in *absolute* terms the odds-ratio estimates can be significantly biased away from one. (Imagine what your model would look like if you had only one poor neighbourhood with infection either present or absent.) But that's got nothing to do with *relative* rarity, & of course isn't fixed by throwing away observations from other categories. Various kinds of down-sampling are sometimes done, as explained [here](http://stats.stackexchange.com/questions/67903), but it's to reduce computation time for large data-sets while minimizing the loss of precision.
it is interesting question  for me
Thank you for your response John. I suppose it could be argued that the outcome variable is ordinal. Given the analysis is categorizing sub-populations, I think the most conservative interpretation would be that the classes are categorical... I'm not certain about this though.
PhilG I would be more than happy to do that BUT I need you to send me the data or post it here.
What's in the y axis? week?
The key point is: there are multiple possible valid ways to compute these metrics (e.g., micro-F1 vs macro-F1) because there are multiple ways to define what is correct. This depends on your application and validity criteria.
@Gavin Simpson Yes, that's right. There is one value of temp_W for each zooplankton biomass value in a particular year, and the same happens with temp_sept.
An *exact* answer will likely be extremely complicated: see http://stats.stackexchange.com/questions/41467/ for an analysis of the distribution of just the sum of uniforms. Perhaps an asymptotic approximation would be ok?
(+1) great explanation. I'd add a small tidbit on causal/formative vs. effect/reflective indicators: http://dx.doi.org/10.1080/10705510903008253.

I guess the issue still remains as to how the composite is going to be used. The OP did not expand on that.
Considering [Geoff Hinton](http://www.cs.toronto.edu/~hinton/) isn't a multibillionare, I'd say it won't work as well as you think it will.  All kidding aside, you could use an existing architecture that maps into future values and feed it into an additional layer which does some classification like you propose.  I think the idea behind those existing nets is that based on the prices, a savvy trader can make their own decision on whether, e.g., a short sell might be to their advantage in a particular state of the market.
This is fine for categorical features with n<=3 as you can generate all of the same splits  as you would by considering the feature natively as categorical. For larger n it is possible to achieve sets of splits that are equivalent to the categorical split but the algorithm may or may not find them as efficiently...however if you split the feature into n numerical features you are also reducing the efficiency with which the algorithm can find splits. Someone needs to add categorical variable support into the scikit-learn implementation as it is great otherwise.
Why would one use such a strict criteria ? Because the consequences of such (potentially) low frequency events are so large we need to try to eliminate them completely. Again, economics will limit just how much we can do in this regard.
Also IMO I don't think a "black-swans" tag will be very useful. Sort of inside jargon to this particular author that should IMO be avoided. Rare-events seems sufficient to me, but you would know the lingo better than me for sure.
@Glen_b Thanks, I have now done so.
+1 ... with a small caveat: "performance" frequently (though far from always) relates to things like completion times, which in some cases might be severely skewed. In some situations - especially if sample sizes are not small - this might lead us to consider something other than a straight t-test.
Not much can be said unless you tell us something more about what you really want to achieve. Here are some similar questions you can look into:  http://stats.stackexchange.com/questions/91597/overdispersion-in-logistic-regression?rq=1        http://stats.stackexchange.com/questions/90953/testing-for-overdispersion-in-logistic-regression?rq=1          Also, please note you can find such related questions yourself:  Look into the column headed by "Related" at the right side of your screen!
See duplicates, [How to define a distribution that correlates with a draw from another distribution?](http://stats.stackexchange.com/q/13382/1036) and [Generate a random variable with a defined correlation to an existing variable](http://stats.stackexchange.com/q/15011/1036)
I have ended up with (-4,2) once I had taken away the probability than µ is larger than 2
I believe a residuals vs. predicted plot helps to asses the assumptions regarding the noise distribution, but not the functional relationship between predictor and predicted (be it linear or not).
Thanks for the tips, Deer Hunter. Some interesting avenues to explore there. Currently I just have the general analytics, not directly linked to responses and response rates. For example, I can look at things like average time on page for the pages that got lower response rates (not much correlation for that one), but I don't have the specific time on page numbers for only those visitors who didn't respond.
It results from the fact that in multiple regression, it is not possible to estimate more coefficients than the number of individuals.
What do you mean by "predicting the topic of a new document"? Do you want to find which one single topic this document was generated from? Do you want to find a mixture of topics for the document? Do you want to label each word in the new document by the topic that word came from?
Thanks alto for your information.
Oh. So exp(W) is log-normally distributed then?
I'd follow up on @Dason's comment and look up the various ways to test for the statistical significance of outliers in a distribution.
I assume that 'without multicore' means without parallelism at all?
excellent addendum to your [database QA/QC guidelines](http://stats.stackexchange.com/questions/7467/quality-assurance-and-quality-control-qa-qc-guidelines-for-a-database/7472#7472)
Also, only after the simulations I realized that fights are over really quickly and that it is indeed not that important at which turn a territory might be captured. Again, thank you for the full answer.
,is it possible to elaborate a bit on (3)? Trying to understand what this conceptually means.  As far as (1) and (2) - I have verified the code multiple times for normalization.. Still looking to hopefully find a bug,so far to no avail. For (2) - second component had a lot of very small parameter values initially because it was plainly computed from one of the sequences and thus didn't contain a lot of the states in the model. But with 2nd and 3rd iteration this changed ( e.g. the transition probabilities were more even across the states). The sums of rows are always 1 for each iter-n.
Michael, we follow standard academic conventions here concerning the use and acknowledgment of other people's work.  They include (1) clearly attributing direct quotations and (2) avoiding posts that are almost entirely pulled from other sources. (Overuse of the latter is an invitation to downvoting, by the way.) I have reformatted this post to indicate the portions that appear to be quotations (and not your own words), and apologize if the result is imperfect.  It would be good if you could further improve this reply to follow those principles.
Once you have odds you can use the command plogis(log(odds)) to extract the probability and multiply the probability by number (if number is the total number of infants) to extract the expected number who survived.
Good answer. I think you should go further into what you mean by "do your inference based on that". That's kind of the second part of my question.
@Björn Thank you for reminding me of the name: that provides a useful way for readers to learn more about the procedure.
Thanks a lot, that was VERY helpful! -A
Actually you're describing the method I'm currently employing, I've taken apart the series by month, week, weekday by subtracting the real score from the mean of each type. I'm planning on generating in-sample data next which minimizes my squared error on the weights of each of those parts. The reason I asked the question is because I consider this method rather "hacky", and the fact that I only have a year of data means the likelihood of overfitting will be very large.
I doubt there is a closed form because the radial distribution is proportional to the Laplace transform of $\\sqrt{r}I_0(r\\tau)$ where $I_0$ is the modified Bessel function of the first kind and $\\tau$ is a constant (depending on $\\sigma_1$ and $\\sigma_2$): this simplifies only when $\\sigma_1=\\sigma_2$.
"*why not*...?" ... you already explained why not right before you asked that.
A gene is a DNA sequence containing hereditary information. We are meassuring the level of expression of each gene and, for now, let's consider that expression data follows a lognormal distribution.
@Dmitrij: Thnx for the reply, I have updated the questions, and replaced 'k' with $\\lambda$
Excuse me, what does this mean `data is T1Xn and T2Xn`? And did by "unequal length" you just mean that there are missing values?
I see one possibly answerable question here: "Is that kind of questions normal in basic studies?" However, for it to have a clear meaning, please edit your post to tell us (a) what you mean by "that kind of questions" and (b) what you think constitutes "basic studies."
@ndoogan, sorry, the link has been fixed. The link in my comment should be: http://cgibbons.us/teaching.html
@Erich Every bit of that book is deeply rewarding: after all, if you can do something with pencil and paper, then you can program a computer to do it :-). With a single variable often it's nice to transform it for *symmetry* (of its empirical distribution); Tukey calls this a "little deal." A simple way to identify such a transformation is described in section 3E, "Looking quickly." It illustrates what can be learned from a glance at an N-letter summary (Tukey suggests a 7- or 9-letter summary). Acquiring that skill is more valuable than having a computer program do the calculations for you.
(+1) Really nice answer.
@gung How can I consider number of successes in the chi-test? (or any other test) when I have to deal with two hypothesis? the manual is not speaking of number of successes!
Ive used glm.nb as the error distribution follows negative binomial. I didnt simplify the model as I am only interested in statistically significant factors (State and Year). and needed to a post-hoc for each one of these alone. So not a full comprasions with all the factors, this is why I didnt think I need to simplify it. thanks.
I've updated the answer for dealing with missing values.
Its not clear to me how Hidden Markov Models are used for anomaly detection but I would very much like to know. The part that is particularly unclear to me is how to create a correct underlying state machine with meaningful transition probabilities (unless it is just two states like "anomaly" and "not anomaly" with a naive transition probability between them).
Also, I understand how it can be done (it is described in the matrix cookbook, for example) but I was wondering if there is a way to do it without involving inverting the precision matrices (taking advantage of the simple structure of one of the precision matrices, for example).
Is this homework?  Whether or not it is, what have you tried so far and what is your thinking about it?
Probably just because it's simpler than using principal component scores all the time; I've not seen other people confused by this.

I also don't see how it would be possible to do analysis on the components - I'm not even sure what you are suggesting.
Thank you, helpful so +1. I contemplated what you suggest but it will take some time so thought I'd ask the CV experts meanwhile. But @whuber's comments are pertinent and seem to be independent of sample size, so I wonder whether there really is much point? Benford's Law on the other hand does not seem to be so restricted (though apparently does require “uniform log distribution” –I’m still working on that!) I’m not sure whether, if the outlier stayed ‘odd’ in a bigger sample that should reassure me it is just a rare but innocuous instance … or increase my suspicions!
Thanks for the comment.  A one-liner in R can compute it efficiently up to N=5000: `w <- outer(x, x, '+'); median(w[row(w) >= col(w)])/2`.  A trivial C, Fortran, or Ratfor program could be called by R to make it blazing fast.  The ICSNP package in R has a fairly efficient implementation with its `hl.loc` function.  For N=5000 it was 2.66 times faster than the above code (total time 1.5 sec.).  It would be nice to also get an confidence interval efficiently.
I am glad I found my answer: Issue 1 on this website explains my questions: http://www.stat.pitt.edu/stoffer/tsa2/Rissues.htm
May I know how to compute the exact small sample variance? Compute the Fisher information?
You might consider beta regression. This is designed for a variable that is distributed between 0 and 1. http://orfe.princeton.edu/~alaink/NJ_aTaxiOrf467F12/Papers/Machine%20Learning/beta.pdf
Regarding your first question, the [manual](http://cran.r-project.org/web/packages/ks/ks.pdf) for this package indicates the differences between these commands and the corresponding references. I have used this package but I do not recall the Hpi.kfe bandwidth selector. In the [JSS paper](http://www.jstatsoft.org/v21/i07/paper) you can find the answer to your second question.
Thanks Will, I clarified some things in my original post.  The sub-sample is random.  I pulled the 3/4 stat out of a hat to make a point.  The true stat is less.  I can cross-check about 10,000 cases.  I'm sure the error is not purely random.
The nonsense apparently comes from the fact that `sd` or variance is a function of squaring, as is covariance. But with 3 variables, cubing occurs in the numerator while denominator remains based on originally squared terms
Because $P(S=s)$ would surface, and then I could not avoid using the "limit notation" in order to handle the fact that $P(S=s)=0$. Also, why bring into the picture a more complicated joint event?
you can quantify the 'how much' by experimenting on simulated data
+1 Very interesting. Yes I was referring to `Beta` by `B` and as a starting point I am using a Random Waypoint mobility model for the nodes. From what I understand, I should not be using `Beta` but instead be estimating `k` and `c`. In this case, I should probably be setting `c` instead of `B`. However, isn't `k` different for each node at each point in time? Oh.. so what you are saying is that `k` should be estimated instantaneously on a per node basis and from there `beta` can be estimated. This can then be used as the threshold? Am I understanding this correctly? Thanks!
I second this. I can't prove it, but I suspect that using absolute levels instead of proportions would produce a test statistic that depends on the size of the network. For what it's worth Matt Jackson & his ilk usually seem to speak in terms of edge "density" rather than total number of edges.
Can you give me a hint where things can go wrong for my problem? And if I am using training set to validate, when _# of PCs $=$ original feature size_, what is the expected accuracy? Thanks a million.
About other methods, I would suggest taking a look at $\\text{ABC}_\\mu$ by [Ratmann et al. (2009)](http://www.pnas.org/content/106/26/10576.abstract), which only considers the performances of each model within this model, comparing the observed errors with the distribution of the simulated errors. Not [completely perfect](http://www.pnas.org/content/107/3/E5.short), but very appealing nonetheless.
I did what you proposed and there is indeed a Phi correlation of .779, which is extremely high. I shall leave it out of my model. Any opinion on my interpretation for the Portugal shift?
As always, very much to the point. Thanks a lot, Maarten!
This is very closely related to the question of including interactions but no main effects in a regression model, which has been discussed here http://stats.stackexchange.com/questions/27724/do-all-interactions-terms-need-their-individual-terms-in-regression-model
Your question is unclear and kind of broad. It would help if you could be clearer about what you want. I'll assume for the moment that your question is at least limited to the continuous case; you use the same general methods to identify the number of modes as you would to count local and global maxima of functions, with the same caveats that apply there. As such this is primarily a mathematical question about functions, though with the additional restrictions that apply to densities.
Wow. Yes, you are right. It's from a Coursera course and it's from the video, not transcription. So I'm guessing he simplified it to make the calculations simpler for the video and didn't expect anyone to try and repeat it. It just happened to be the first video that I saw so I tried to follow along. It's clear that I need to upskill when it comes to maths. I think found the error though. The constant term, which you say doesn't matter, is probably the correct value which through off his calculations. I'll look through your answer a few more times to teach myself. I really appreciate it!
@dfrankow Somewhat crude and certainly very partial help for your first two questions, but `methods("profile")` will give you the (S3 in this case) methods associated to an R `profile` object, then you will see than there is a dedicated method for `polr` results, that you can browse on-line by typing `getAnywhere("profile.polr")` at the R prompt.
Are you asking for help with extracting information from photos or with statistical analysis of data?  Please clarify your question.
Welcome, Laurie.  If you wish to clarify or otherwise modify a question and it has not been answered yet, then please feel free to edit it.
I have done the bootstrap and of course got the same answer. I suppose that the mean is likely the best estimator but the problem still remains that it is a bad estimator since one outlier has been known to change the choice.
Goodness - that wasn't actually the result I expected. Thank you for your thorough work and good explanation. (Curiously, I actually did some stats training at the University of Auckland where R was first developed)
@Dan, When "believes" is set to 1 the "after" will be (randomly) higher than when set to 0.
Not sure autocorrelation applies as well as a runs test would, since the data are binary.
No doubt you mean [*Mahalanobis* distance](http://stats.stackexchange.com/questions/62092). Are you referring to some specific source that suggests there is some relationship?  Absent a hint like that, this question will be difficult to answer, because it's practically a riddle.  It's a bit like asking how exactly a [raven is related to a writing desk.](http://en.wikipedia.org/wiki/The_Hatter#The_Hatter.27s_riddle)
(+1) Thanks for the link! You may be interested in this related question, http://stats.stackexchange.com/questions/3200/is-adjusting-p-values-in-a-multiple-regression-for-multiple-comparisons-a-good-id. Feel free to contribute.
Thanks for your advice.
So essentially if I have 2 samples drawn from a population (In this case my over 70s and under 70s), I could use either Kommogorov-Smirnov or Kruskal-Wallis (Which, for 2 samples, would be the same as Wilcoxon-Mann-Whitney)?
Or should you oinly use K-S when you are testing for normality.... (Actually I don't understand the proper case in which to use this test) and use K-W/W-M-W if I am testing for means etc?

Thanks again
Note that Stata users need to install `qenv` (by `ssc install qenv`) first.
Interesting question, welcome on this site! I edited the title to emphasize some broader underlying statistical issues. I hope I understood the question correctly and the new title is OK.
This question is perhaps overly broad.  Do you have a specific problem you want to address?  If so, please describe it and how you are generating the data, because those are crucial pieces of information for selecting probability distributions.
It's a pretty well-known algorithm and not too hard to implement. So if your algorithm can beat that without too much extra effort, it has a certain level of sophistication. I don't think it would be too hard to beat rank-boost.
I would also note that the `?pvalues` help page in the the new version of `lme4` includes a lot of information on this topic.