tttjjj/elt_bench huggingface dataset:
"Risk," as the object of investigation, has a definition independent of any statistical analysis you care to apply: it is conventionally taken to be the probability of an event multiplied by the cost of its occurrence. More generally, it is the expected cost of a random variable. In your application costs would often (but not invariably) be measured in terms of deaths per distance driven, whence *neither* of your proposed definitions would be correct. Regardless, note that the latter proposal is problematic (what happens for distance=1?) and arbitrarily depends on the unit of measurement.
Just out of curiosity, why are you constrained not to use Stirling's formula?
Hi I have read Revelle's paper, but I don't think I have fully understood it. That was the reason I posted it here and hoping that someone can point to the right direction.

I have computed both omega and alpha reliability analysis for a set of data, sometimes, the omega coefficient is higher, sometimes, the alpha is higher - and I don't really understand why is the case.
Can you please expand abbreviations the first time you use them? Specifically, could you please explain what TOM is on first use?
When should I use Spearman or Kendall correlations?
If you like my answer you can also upvote it.
I haven't looked the entire post over--it's too long and this site is not for exhaustive review of derivations (or code or analogous work)--but I can't help wondering where the "$\\infty$" in the last line came from.
Thanks Hotaka and BabakP - I will simply run a regression through r and do the boxplot (whisker too). Hotaka, yes I did have a concern that I was throwing out data by categorising. In terms of the PHP/MySQL code, I've also taken a quick look by calculating % sick leave shifts for each category of penalty (there are only 5 - 0, 15%, 17.5%, 22.5% and 100%). By ignoring the dollars (which are continuous) and focussing on the rates only (categorical) I'm able to simplify my MySQL/PHP processing, which was the main concern with trying to identify the right approach. I'll use R to do the rest. Tks
(your link doesn't work)
Well said. See my answer for some supporting details. The PhD in this field is not too relevant, and if a company thinks it is, you probably don't want to work for that company. Peter Thiel once said, "Graduate school is like hitting the snooze button on the alarm clock of life..."
Since the error states `too many elements specified`, rather than `cannot allocate vector`, it seems that its an address space issue rather than a memory issue (though this would follow). This message kicks in if you attempt to allocate more than $2^31-1$ elements. While this doesn't seem to be the case here, there may be more elements in use than specified in the question above.
I have added some additional commentary regarding these two statistics in question. Thanks.
No you didn't. It would be a martingale even if $g(x)$ was random but the $u$'s were independent from past $X$'s.
You might have taken a little effort to find out yourself that Cochran Q = Friedman when variables are dichotomous, and that Kendall W is just a standardized form of Friedman statistic.
Are you sure that this is caused by sampling bias? If so, do you know anything about the confounding factor that causes it? The reason that I'm asking is that the inbalance between positive and negative values isn't all that big. If there was no bias, you would get inbalance at least this extreme about 5 % of the time...
Hello! I'm not searching for tests that can compare nested models (you mentioned LR Tests). And I'm not searching for quasi t-tests (like Wald Test) to look for significance of the coefficients! I'm searching for something which is based on the (deviance) residuals, something like an adj. R^2 for linear models. I just want to say something like, "This model describes 70% of the deviation".
Yes (roughly), but the condition is clearer when you have several exogenous variables. $E(x_k\\mid x_1,\\dots,x_{k-1},z_k)\\ne E(x_k\\mid x_1,\\dots,x_{k-1})$ is better because correlation is a relation between _two_ variables, but $x_k$ and $z_k$ could be correlated to a _combination_ of some exogenous variables. When there is only one exogenous variable, imaging a violation of condition 2 is difficult.
This is explained [here](http://en.wikipedia.org/wiki/Student%27s_t-test#Alternatives_to_the_t-test_for_location_problems).
So... like a prediction interval for regression?
Thank!! You  Glen :)
Do you mean this? http://en.wikipedia.org/wiki/Standard_normal_table#Table_examples
:John From struchange documenation "Finally, the breakpoints in regression models with structural changes can be estimated" using the CHOW Method with which I am intimately familiar with.Testing or finding breakpoints in regression coefficients requires a specification of the regression model and if I am correct this has nothing whatsoever to do with answering the question"test to determine if the time series has increase or decreased based on the data?".I think that your recommendation is insufficient to answer the OP's question.Your recommendation answers a questionI don't believe was asked.
How old are the subjects at the start of the study?
+1 Very clearly worked example.
@gsk3, that is exactly my problem. The algorithm I had previously gave more importance to weights than to distance because my weights are huge...
Good quote, but it's not true!  Absence of evidence is not *proof* of absence, but it certainly is *evidence*.  Why do we think magnetic monopoles (or unicorns, for that matter) don't exist?  Because we've looked and haven't found any.
Conversely, even a single reference of how these terms are confusing you would help us. Taxonomy is an old name for classification, especially in biology: some texts in the 1960s or 1970s referred to "numerical taxonomy" or "mathematical taxonomy", but the term seems to have faded away. I'd regard taxonomy as equivalent to classification, but it's hard to know about usages across many disciplines.
**Hint**: $\\mathbb E Y = m_X(1)$ where $m_X(t)$ is the moment-generating function of $X$. Now, what about the variance?
(-1) Sorry, I am not able to see the value of this question, for me it is like asking "what do you currently code with R and what are the challenges you are facing". Yes, it can be extending one's view on the world, but it is not a good question according to SE-standards. [What is the definition of a list question?](http://meta.stackexchange.com/questions/124450/what-is-the-definition-of-a-list-question)
I think the OP probably means "Prove that $Y/X$ and $X$ are independent" rather than "Prove that $Y|X$ and $X$ are independent."
Do you mean $E[X \\mid |X|]$ ?
Do you want a table per diet? or what is it exactly that you want to calculate?
Thx for your answer. I already figured it out and felt kinda dumb later on... but hey :)
This question appears to be off-topic because it is about performing simple operations in R
The meaning of "significant" is usually not in question because a probability model and null hypothesis are implied by the context. Not so here. The replication reveals effects of *measurement error* on the PCA results. However, if you view your $N$ points as being a single realization of some process, then "significant" might mean with respect to a null hypothesis like "all components are equal." (This issue is rarely discussed for PCA because it's usually used as an *exploratory* rather than a confirmatory analysis.) So: *what is your probability model and what hypothesis are you testing?*
As long as you have mod stored, you don't need the points. But you can get them with coef(mod), I believe
It looks like R might have chosen too wide a bin width. Examine the other residual plots.
I think you make a confusion: $s^2$ is the estimated standard deviation of the data, not the standard deviation of the Gamma distribution. And it should be the posterior, not the prior.
I agree with Michael Chernick. This calculator is only as good as marketing and low quality polls go, for the lack of a better curse word :). What they sweep under the carpet and totally failed to explain is that their tool is only intended to work with proportions -- that's the percentage in the lower box, and the upper box probably assumes 50% that requires the largest sample size. If you know what the bootstrap is, you are already overqualified in terms of using these crutches, assuming your know one or more of R, Stata or SAS, and can do the power analysis yourself using these packages.
An alternative perhaps more efficient way of creating the dummies is with the `dummy` function from the `dummies` library: `clusterdummy <- dummy(clustmember)`.
Something like that: 
`ex1 <- ecdf(data$Obs[data$Experiment == "General0"]` and so on for each experiment (if there are many experiments, you could make a loop). To plot:
`plot(ex1)`
`lines(ex2)` and so on. To add the means as vertical lines: `abline(v=mean(data$Obs[data$Experiment == "General0"])` for each experiment, maybe with different colours.
Cochran Q is is a repeated-measures "ANOVA" for k dichotomous variables with the same two categories. Then, what does a 2x2 table mean in your case?
Feynman, in the book you reference, already answers this question: "If he wants to test this hypothesis [found through exploration], ... he must do another experiment." What you seem to be asking concerns whether Feynman may have been too extreme ("exaggerating a little"): to what extent, if at all, can formal testing of hypotheses be justified when they were developed by exploring *the same data*?
+1 I like the way you turn the problem with "the more likely an event is to occur the more bits for that event. But you should notice that often the tails of the distribution are somehow "precious" (in wind power forecast, there are relatively rare errors or events that cost a lot in terms of money)
Yes, if you bring *a priori* knowledge that clusters will be unimodal, and the situation is one where there can only be two clusters, and you observe a bimodal distribution, the problem is well-formed and has a solution. (But if that *a priori* knowledge comes from a sample from the population of interest, you should use that sample to do your inference, so you make efficient use of the actual information.)
Thank you for your response. I got the MOM estimator. As for the CI, $\\ln (2)$ was not there in the lecture material.
I'm not in Canada, but the protections are pretty stiff in the U.S. too. I was inspired by Apache cTAKES (an NLP system for clinical medical records), which includes models trained on data from the Mayo Clinic. I tried asking the devs how they managed it, but so far no response. Perhaps this is still a gray area they would rather not draw attention to. Do you know of anyone else who shares models trained on confidential data? There must be others, these things are extremely useful.
Assuming all software provide type $I$ error control over the 20,000 genes, but in very different manners, so that their global p-value are uncorrelated (under the global null), you will have fast type $I$ error inflation. I believe however, this is implausible as each field has a small number of favorite error control methods. You could use simulated data to quantify the agreement over different software under the null.
If the OP could explain precisely what LD1 is, there would probably not be a question. As I see it, you are not supposed to use the components of an `lda`-object from an `lda`-call directly, but rather use the `plot` or the `predict` methods implemented for these objects. That does not mean that it is not interesting to figure out what the components are, and I have tried to give some explanation and further references in my answer.
What will you use the model for? The training data should closely match the intended use. If the reason for the redundant training samples is that they are more common, and they will also be more common when you use the model to make predictions, then you should keep them.
@Rado If a question has $k$ choices with probabilities $p_{j}$, then the probability for pairwise agreement on a single such question is $\\sum_{j=1}^{k}{p_{j}^{2}}$. You should only need to change the `getP()` function to reflect this, specifically the lines starting with `pp <- ...` and `qq <- ...`.
Thanks @F. Tusell. Exactly what I was looking for!
Plug it in Sas with stepwise. This will be as good of an approach as anything else to start with.
As reference take for example the SP500 data in R, the return data seems to be constant in its mean but exhibit blatant conditional heteroskedasticity. So it is possible to apply a GARCH model on it despite having non constant variance?
Henry Normally we try to provide useful advice and guidance on homework questions without actually doing the homework for people.  The lack of an answer for ten days was restraint on the part of this community, not ignorance.
It is hard to imagine how this could be. Perhaps you are cutting the predicted probability at 0.5? If so, try varying the cutoff.
+1 That's a nice characterization.
@Dmitri: one possible model to deal with trend is y(t)=theta0 + y(t-1) + a(t) . This easily generalizes to y(t)=[theta0 + P(T1)] + y(t-1) + a(t) where the steady state ramp changes at period T1. In contrast y(t)=3*x1 + 2*x2 + a(t) where x1=1,1,2,3,4,...t and x2=0,0,0,0,0,1,2,3,4,...t-5 reflect points in time where the trend changed ( i.e. period 1 and period 6 ). Careful analytics will suggest without any  presumed knowledge which of these MODELS is most appropriate for a particular data set. The ARMA component must also be empirically found to complete the model along with any Level Shifts.
Somewhat related: [Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?](http://stats.stackexchange.com/q/78828/7290)
You can use EmEditor to open large files.
This is the best answer. It is a saddle point. Backpropagation based optimization algorithms will usually stop immediately. In order to calculate the gradient we multiply deltas with weights and the result will always be zero.
It is complex and messy real-world data. The "thermal anomaly" is (I hypothesize) variable in time but consistent in location. Overlaid on this is a different large-scale regional temperature gradient, seasonal and day/night variations, noise, instrument variability, etc. The flight paths come and go in different directions (so the mean temp of two flight paths should not be equal, even if season+time-of-day are removed, because of the large-scale regional gradient, etc.)...
@Glen_b You are correct, sir. My comment is now deleted.
Isn't quasi-separation a situation, for instance, where almost every y is equal to 0 when x = 0 and almost every y is equal to 1 when x = 1? I mean, y hence can be easily predicted by simply looking at x, and using a logistic regression would be useless.
I'm not sure what you mean that I didn't pick a a point to start with. I have a whole set of observations $\\bf{Y}$. Maybe, the confusion arises from the fact that I want to obtain the predictive posterior distribution for $y_{0}({\\bf{s}})$ at some point $s_{0}$. I will update the question to make it clear.
Subjective or not, the sum is not generally meaningful, because it depends on how many residuals occur per year, which might be a quantity wholly unrelated to anything of interest.  At a minimum, look at the *mean* residual by year.  Better yet, look at the *mean square*.  This will reveal both time-related changes and possible heteroscedasticity.
Let's say you want unit resolution. let's say all possible variables have a domain of [0,100] for simplicity. In one dimension, k=100. In two dimension, k= 100^2, etc., so in N dimensions k=100^N. This is what I mean by "numerical approximations become intractable *quickly* as the dimension increases:" the number of iterations grows exponentially. Let's say we determine that the maximum feasible iterations on our hardware is 1e9. If the domain of our variables is [0,1e5], then we can't search the entire space with unit resolution in two dimensions in under 1e9 iterations.
I'm not sure exactly what's going on here, but if you call `AIC(bull)` you'll see that the AIC of your final model is the same as it was at the start (i.e. it's 89.127 and not 91.5): http://www.r-fiddle.org/#/fiddle?id=LHoJKPr0&version=1
Perhaps you mean [Canonical Correlation](http://en.wikipedia.org/wiki/Canonical_correlation) rather than [Cross-Correlation](http://en.wikipedia.org/wiki/Cross-correlation) which usually refers to the similarity between two waveforms (with a possible time lag)
The reshape2 approach seems to be: dcast(melt(example,id.vars=c("SID","Prep","Experiment","Day")),SID+Prep~Experime‌​nt+Day+variable)
Sampling is really diverse for every single point. It means sometimes the first layer is 0-1cm sometimes 0-100cms. The goal would be clustering, on a uniform layering, but I'd like to get rid of correlating properties.
While we may not know if the sample mean and variance are independent, we always know that they are *uncorrelated*.  This is because the sample mean is a function of bivariate sums $x_{i}+x_{j}$, and sample variance is a function of bivariate differences $x_{i}-x_{j}$ (these are called "U statistics") and we have $cov(x_{i}+x_{j},x_{i}-x_{j})=var(x_{i})-var(x_{j})+cov(x_{i},x_{j})-cov(x_{j},x_{i})=0$ as long as the distribution is "homogenous" $var(x_{i})=var(x_{j})$, which is part of the problem statement.
Thanks for your answer. It's just what I needed! I recoded the answers as well (eg. from "Never" - "Rarely" - "Often" - "Always" to 1 - 2 - 3 - 4)
Horner's method (http://cnx.org/content/m15099/latest/) allows linear computing cost of the polynomial and is much more stable than evaluating it as it's written.  I'm sure there are lots of others!
@Jonas Or maybe it came out of one of the OPs comment.
What method is best depends on the exact details of the survey and the exact purpose of the analysis. If there were one best method, then Stata would not have given you the other (in that case, inferior) options.
I searched on google scholar and i was not able to find any good reference on this issue, could you suggest a paper i could consult ? In the case of large samples how could someone address this limitation ?
I probably misunderstand your use of the term _multivariate_, but as I see it these papers **do** model multivariate temporal data. For example, the model described in the last paper was used to model motion capture data, where at each point in time you have multiple measurements. Unless you can make some specific independence assumptions about your channels, I don't see why couldn't model them in the same manner. Please clarify.
2) While sum nonparametric tests could be said to test medians I do not think it is quite right to say that is what the Wilcoxon rank sum test does.  It is really a test of location shift.  If the populations have different "centers" both the mean and the median will shift although depending on the shape of the distributions maybe not in the same way.  So i think you can view the test as a test of both means and medians since it should have power against shifts or you could say it tests neither because it doesn't directly test either.
Great answer +1
The "it is likely" statement is incorrect in general. If, for example, one was working in a field in which large numbers of false positive signals were generated, chance might be at least as good an explanation for a small p-value as a real relationship. We cannot trust $p$-values to be the sole arbiters of truth.
About as far from there as you can get... Well, New Zealand would be even further I guess.
yes, that is the paper I am looking for, but I cannot find a version of it online. The R-code might be useful, but I am working in Matlab, and don't have time to do the translation (the R code looks nontrivial). I would even be willing to settle for the bibliography of this paper at the moment.
The site shows your profile flair so there is no need to sign your posts.
Thank you very much for the answer. But how do I find out which values are the outliers?
@ whuber: Thank's!  I assume the occurrence of an event in discrete time and after one such occurence I count the time steps to the next event. These time steps counted are the interarrival time. I assume, that the reoccurrences of the event are correlated. Now, I would not only like to have a fitted distribution for the interarrival times but also a description, best would be formula, for the correlated structure of the interarrival times. Maybe You have some hints how I could proceed? Possibly with regression? Or are there other methodologies?
@NickCox: The question was not at all clear. Those 25 correlations will depend on each other and there is the multiple testing issue.
Good point. Actually microarray is already a dying technology, replaced by high throughput sequencing. Techniques in the field tend to survive 10 years before another one supersedes them...
Sorry.I have just had semi-bad experiences here multiple times where someone does not address your question,but instead tells you it is not a good question without fully understanding why you are asking it.If I explained the entire background,would anyone want to read something even longer?I am wanting to compute null distributions from random allocation processes.So, you got that correct, and I see no reason to edit. I did not mention wanting to test null hypotheses.There are reasons other than that to want to know something of the null distribution.
Could you please illustrate how the quantification is done in both processes? I know for CATPCA the quantification is done so that the correlations between the quantified variables are maximized. Then how the initial quantification is set? What is the procedure for quantification in MCA?
There is statistical content to this question which makes it on-topic, just, in my view. But the mix of Stata and statistics makes it enormously more likely that you would get a better answer on Statalist, where Mark Schaffer [NB spelling, which I corrected above) and others hang out.
Great, interesting. What would you say is the benefit of using that technique over any of the others?
Are there also drop-outs?  If fatigued people drop out that might explain why the don't know rate doesn't increase.  The other issue to look for is the complexity of questions.  If the easy questions are last you wouldn't expect an increasing rate.  The only way to be sure in testing this would be if the questions, or at least groups of questions where they are linked, were in random order, which is often undesirable for other reasons.
Oops, I appear to have gotten tired of typing ;-) I added the right hand side. My question is mainly how to use summation rules and apply these to the shown problems.
Also, be aware that even if you're not maxing out the CPU, the botteneck might actually lie elsewhere (RAM, disk or network I/O, etc). If you're in a situation where you're constantly swapping things in and out of RAM or blocking for I/O, starting more processes is going to make things go mind-bogglingly slow.
Can you be more specific regarding what aspects of time series you are seeking to learn? At any rate, you should formulate your question to encompass a broader context than "my midterms are coming up", as otherwise it's almost certain to be closed as *too localized*. This should also be **CW** I believe. Cheers.
Thanks for your answer. So, how does your *model.test(...)* function look like?
If you have 8 cases then you should have at most 1 independent variable. But you have much more data and you certainly shouldn't give that data up.
Mutuel, In light of Glen_b's excellent comments, I am wondering about your objective here: do you seek to *describe* your data by giving bounds containing 90% of them or perhaps do you seek to *draw an inference* about a population of which your data are a representative sample?