tttjjj/elt_bench huggingface dataset:
A loop over workers and a counter wich is incremented if departure < time < arrival will give you the number of workers travelling at a given time... Then loop on times.
@bison2178: best way to say thank you is to upvote the answer ;)
Thank you for the resource
Thank you very much Chl :)
Also, I've seen in some statistics texts and other places that state such a result, that they will say *exists* when what they mean is *integrable* (i.e., *exists and is finite*).
Did you lose part of your answer?
+1 for "Correlation with the dependent variable is a definite plus"
How many data points are there? I suspect the answer is in the problem statement.
Great answer! I suspect the multiple comparison issue is going to be a problem, depending on how many time points you're looking at, but I bet fitting a model would give you some interesting results!
Is this a homework problem? Is so, please add the HW tag. Note that HW questions get special treatment here (see our [FAQ](http://stats.stackexchange.com/faq)).
That is at least two questions.  For (1): $\\ln(x) \\approx 2.302585\\times \\log_{10}(x)$ so the only impact should be that of a scaling factor
Please edit the question. Difference between two groups of what? What has age got to do with any of the things you mention?
@Henry It is difficult to tell from the present question which definitions are accurate and which ones not--some of them do not seem to make sense out of the context of the textbook.  Some clarification would be welcome.
I think the problem you describe might be due to putting the data into two "columns" -- Stata (and many other stats packages) then presumes that the ordering of rows is informative (i.e. represent paired data, which as discussed is not appropriate here). So Stata is probably trying to do a Wilcoxon signed-ranks test (analog to a paired t-test) whereas you actually want a Wilcoxon ranked-sums test (aka Mann-Whitney test). So your original data format would work -- command in Stata would be `ranksum x, by(group)`
If there are $m$ positives and $n$ negatives, $p = 2^{-(m+n)} \\sum_{k=0}^m {m+n \\choose k}$
i gave up and created [a fake method](http://www.asdfree.com/search/label/new%20york%20city%20housing%20and%20vacancy%20survey%20%28nychvs%29)
I had never heard of SRH, @Mark Harmon, thanks for this & welcome to CV.
A reason for this seems to me that the meta-analytic studies require major understanding of the condtions or assumptions. Generally, the design part for the analysis as well as study  may be weak. Another thing is that many of us do not have a claear picture of statistical theorems postulated. In this context, random effectsr assumtions and theiis abysmal. implementation
In this case $Z = X/Y = X/|X| = {\\rm sign}(X)$, so its variance exists and can be calculated. Also, if this happens to be homework, please add the homework tag
Thanks for that. I spent some time trying to work it out, but I'm still having trouble with step (3). I tried following your suggestions but, I can't seem to be able to write down the covariance matrix. Any help would be much appreciated!
Thank you for your submission @BlasterMaster. Unfortunately, I need to implement this in client-side Javascript, so, using R isn't going to cut it for me =). I did try out R though, neat! I find that the location isn't often so close to the mode, but that the median is fairly close, except when the skew is fairly extreme, then the location is indeed pretty close. I use the median as the start of my hill-climbing algorithm, which, even without optimizing, finds the mode in 17-25 iterations, which is fast enough for my needs.
Great question!!
If you use the library `mgcv`, the `gam.check` function plots residuals vs linear predictors, for the case of GLMs.  I find that really helpful.  Regarding the consequences of broken assumptions:  in most cases it's a problem for inference (i.e.: confidence intervals).  In some cases, such as heteroskedasticity, the expectation of your estimate is fine, but the variance will be off and you'll get an imprecise estimate (and the confidence intervals will also be wrong.  This is all intro econometrics/stats.  It is a little bit dangerous to jump into GLMs without really knowing OLS.
I believe the slide is correct. If you use two different decision procedures with two different thresholds and then take a randomized decision, you will get a convex combination which will give a point lying in between the two. This point ***may*** be above *both* (**!**) of the curves at the same false positive rate. This is because the threshold used for each procedure is different at that point.
I'll extend Aaron's question a little - how are the variances different other than the differences binomially distributed variates would have due to different $n$ and $p$?  If one of your explanatory variables has heterogeneous variance, that's not an issue, but a binomial response can't have a heterogeneous variance given $n$ and $p$.
Hi, I have put my codes for getKFML function there. Do you see any bugs there?
(1) Does `glmer()` throw any warnings/error messages when you estimate `m1`? (2) What version of `lme4` are you using? (3) Can you show the output of `summary(m1)`?
Every news article has a time stamp. I want to cluster them based on two variables:(1)time closeness, (2)TF-IDF weights of cosine similarity. My final goal is to cluster those news reporting the same event. But the corpus is large.
Yes, and the direction of the effect will depend on the type of model, for an AFT (think weibull model) a positive coefficient indicates a longer survival time, or a decrease in risk of death, versus, say a Cox PH model where a positive values means an increase in the hazard of death, which is what I interpret the OP as asking about.
Dear Thomas, many thanks.
You can get 90% of accuracy with $p=1$ in your problem. This gave me the insight to understand this: http://en.wikipedia.org/wiki/Gain_%28information_retrieval%29. You selected $p=1$ because you had seen the prior probability and you understood that it would have been better to be biased to the majority class. Thus, the best choise for a random classification is select $p=\\frac{n^+}{n^+ +n^-}$ that gives the $r$ in wiki. Doing maieutics on my question I understood I wanted to know: if I flip a coin to classify, which is the $acc$? And it is always $1/2$, without care what the prior are.
Not that I know. First, if you refer to the "eigenvalue above 1 rule-of-thumb" or so-called Kayser's rule, you will probably find the number of PCs that satisfy this (keep in mind that this generally overestimate the number of dimensions) by looking more precisely at your scree plot. Second, parallel analysis is a better way to determine the number of components; see the `psy` or `psych` package in R, and [SPSS, SAS, and MATLAB Programs for Determining the Number of Components and Factors. Using Parallel Analysis and Velicer's MAP Test](http://bit.ly/10p0AXs) more generally.
(+1) in general I share the same ideas as yours, if not all of the prices are in coincident state, there still is hope to detect some leading indicator or define common and idiosyncratic part. In general the interval forecasts will be quite wide, but who knows. Diffusion index model may be technically applied since $T<<N$, but the uncertainty in time domain will be too high IMHO.
The Human "min/max" values given in the book are nearly identical to the human male 1SD I've seen listed around the internet.  Close enough to just roll with it.  As for technically not having a min/max, I'll just use percentiles between 1 and 100 (exclusive).  This seems to be working quite well, and I can refine it as I go. Thanks! :D
If the data are continuous, what happens to someone whose heart rate is 135.6?
I suppose you could, but for a $\\chi^2$ test you want frequencies rather than proportions.  You would not be using the standard deviations in the test.
Thanks - in line with what I thought might be the best idea.
+1 for introducing many thoughtful concepts into the discussion.  I would like to suggest it may help to maintain a sharper distinction between randomness and uncertainty: one leads to the other, but not *vice versa*, yet many people (obviously not you!) exhibit some confusion about the difference.  We know that not all uncertainty comes from randomness, nor is all that is arbitrary or variable necessarily "random" in the technical sense employed in statistical practice.
While it is true that the logit is applied to the mean, it is not clear (to me, at the moment) what the likelihood would (sensibly) be in this context without some further information or assumptions.
The addition of random variables is described in our thread at http://stats.stackexchange.com/questions/95993/can-anyone-clarify-the-concept-of-a-sum-of-random-variables.  Does this answer your question?
@Andy W -- great comment; yet Monte Carlo can be used to do this last step ;-)
*whuber:* Thank you for your comments. I did some research on prediction intervals and have been able to find info on how to get intervals for a single profile variable, but not on how to use intervals to get ranges for overall estimates when there are multiple profile variables. Any ideas on resources for this?
Thanks Ben! Very useful to have the Stan syntax as well.
1) What are you stratifying on?  2) Your approach would work if you just wanted to split the sample into approximately equally-sized groups, but that's not a particularly good way of doing cross-validation.
Maybe @whuber could take a quick look at this?
All right. Admit your definition of scatter matrix as just SSCP matrix of centered data is more conventional. Still, I'd recomment you to explicitly define what you mean by "scatter matrix" in your answer.
(+1) And, you've left a little bit for the OP to do as well! :-) (Sufficiency, ironically enough, does not suffice for the conclusion.)
In my experience, nelder-mead is usually faster than grid search and gradient descent is only slightly faster becuase while it takes fewer iterations, the cost of computing the gradient is high.  So if you have an implementation that provides gradient descent then use it, but Nelder-Mead probably won't be far behind.  Of course as soon as you have more than two hyper-parameters to tune grid search immediately becomes much the slowest method.  It would be interesting to see an study of the comparative efficiencies of each method.
Definetely looks useful. Thanks alot!
Explained at http://stats.stackexchange.com/help/deleted-questions
I think the answer posted on your other question would be helpful :)
This question may be helpful to you: [how-should-i-transform-non-negative-data-including-zeros](http://stats.stackexchange.com/questions/1444/).
$\\sqrt{2}-1$ is 0.4142, so it is in agreement with Mike's analytical result. And +1, because I like simulations ;-)
Not a bug, but a problem with the initial estimates being too far from the MLE, which won't arise if you don't try to choose them yourself.
Slightly tongue-in-cheek (and to steal a [modified] line from someone else): I would say that a *mathematical statistician* is someone that the mathematicians consider a statistician and that the statisticians consider a mathematician.
Usually, when people do a before & after experiment, & test the results, they want to know if the mean response changed from before to after. You hint at this in the 1st part of your description, but later refer to the "correlations mentioned". Whether the before responses are correlated with the after responses is a thinkable question, but it is a very different question than that of a mean change. You need to be clear on what you want to learn from this study.
"Some tutorials and/or resources for getting started with R": [Learning R. Where does one Start?](http://stackoverflow.com/q/3375808/583830), [What are some good books, web resources, and projects for learning R?](http://stackoverflow.com/q/420296/583830), [R for finance tutorials and resources](http://stackoverflow.com/q/1228801/583830).
@amoeba That will depend on your application, data, and what you want to accomplish. Is it just the dimensionality reduction, or source separation, etc ? In audio applications for instance, say source separation, the optimal $k$ would be the one that gives you the best quality when listening to the separated audio sources. The motivation for the choice here will of course be different if you were working with images for instance.
It looks like the multiple measures are just that, efforts to get the best measure of a sample, but they are not in themselves independent samples...correct? Only the light#2, dark#3, etc. are samples.
Everywhere you write "bimodal" do you perhaps mean "unimodal"?
So rephrasing my question, do I need to worry about non-stationarity with a conf. interval that includes an abs. value in excess of one? Thanks.
I don't agree with you at all. What you describe is a simplification of what is contained within HMMs. You can choose your states in a HMM much more freely than you describe and it is very well possible to have a continuous state space for the hidden states. I have for example used continuous x and y position in a plane as the hidden states with great results. Another good example widely used in HMM research papers is the stochastic volatility model which also has a continuous hidden state space.
I agree.  i think there is also a previous question on this site where this was discussed.
I know, that bottered me too, but in practice it really depends on the values of $\\mu$ and $\\sigma^2$. If $\\mu>>0$ and $\\mu>>\\sigma$, then we can "save" the model. In normal conditions, the dispersion of a resistance is not very high, so the last assumption is clearly met. This was something that initially bothered me too when people modeled height as a normal random variable, but by the same reason that I gave here, some people here at Stack-exchange made me feel ok with it :-).
Also, since posting this model, I have followed up with an additional post you may want to check out here: http://stats.stackexchange.com/questions/61711/fitting-a-zero-inflated-negative-binomial-regression-with-r
I've explained the issue a bit more detailed @leonbloy, I'd really appreciate any further comments.
@chase, I hope that the example is enough now.  Can you point to a solution.  As mentioned by @celenius, I am looking at ggplot2, but I am still very much looking for a solution.
No. Education is an independent variable.
(1) You can't tell a thing about correlations among *predictors* (independent variables) based on coefficients in a regression. Are you perhaps asking about correlations between a predictor and a *response* variable? (2) Are these the actual coefficients or the *standardized* coefficients?  The former tell you absolutely nothing about the correlation (after all, the coefficients can be arbitrarily changed--even negated--simply by rescaling one or both variables), while the standardized coefficients are related to correlation.
First question: What is the dependent variable? You mentioned prostate cancer in the comment but I am not sure how it was recorded (as yes/no or concentration of PSA?) Second question: what is the reason of dichotomizing all the nutrients? Please realize that aggregating continuous to this extend can waste a lot of information. Third question: this is about the nutrition aspect... how did you decide high and low? Have you perform energy adjustment first?
Thanks for this - this makes sense. However, I was really hoping for some analytical result.
Unless those 40 questions are all measuring the same thing Cronbach's alpha is uninterpretable: it is useful only for *unidimensional* data (i.e. a set of variables all sharing a single underlying dimension of variance).
@Peter: oh, true! Didn't realize that until you mentioned it =). (If all you want to do is take the mean of the posterior, they're identical. I guess having a Dirichlet posterior might be useful if you want to compute a different kind of score, e.g., some kind of polarity measure, though that might be kind of rare.)
@Emre Thanks for you suggestions, esp. the article. However, I am interested if it does actually work (I believe - yes). I don't have date, so links to such would be desirable as well.
I have implemented a statistical test to compare the cumulative distribution of my data and the fitted function that I have predicted. It is not urgent to answer this question, but if someone wants to be creative any comments are welcomed
How can I measure how good the model is? Don't have much experience using ordinal logistic regression?
Could you clarify one thing?  If you "know a density...up to a multiplicative constant," then you know the density itself because it must integrate to unity.  Why don't you then just evaluate the limit directly?
@Andy This is a good question. Unfortunately, a lot of people don't bother about it. A good discussion is on *page 108 of Statistical Design by Dr. Casella*. In short, blocking pays off only when the block-effects are significant. Thus, if we are in a situation where the blocks can be chosen, it makes sense to choose them as disparate as possible. Blocking is a form of modeling correlation structure. Testing for a random term to be some fixed number doesn't make sense (even though it can be done.). The anova test-statistic is approx. F, hence conservative. Proper way is to use mixed-models.
(+1 for the edits.) A distribution with zero variance has essential support at a point; its scale can be taken to be zero. As my answer shows, it does not matter if a distribution has an infinite variance: a scale can still be defined by reference to a standard member of the family. For instance, we might take the standard Cauchy distribution to have PDF equal to $dx/[\\pi(1+x^2)]$, whence the scale of a distribution with PDF $dx/[\\sigma\\pi(1 + (x/\\sigma)^2)]$ is equal to $\\sigma$. The scale is well defined, despite the infinite variance of the distribution.
Thanks for the response. I see `ranef.merMod` takes the estimates of random effects from `model@pp$b(1)` and organizes them by factor groupings. Your [link](http://github.com/lme4/lme4/blob/master/R/AllClass.R) was helpful, but most of it was over my head. I have much to learn...
+1 It's good to put this argument as well; it's a (better) version of the line argument I began to write before I decided to post the one I did. (I had planned to come back and add such an argument in later; yours is no doubt better than what I'd have written)
I wonder how are you defining a "uniform" distribution on ${\\mathbb R}^2$?
Have you asked this on the multilevel list?
Thanks, I have looked into it. Similar to the log-logistic distribution, it fulfills all of my requirements, but unfortunately I still cannot get a closed-form solution $x^*$ for the relevant equation in my paper (which is of the form $A+BF(x) + Cxf(x) = 0$).
Ok. Thanks. Will look into latent variables.
(+1) Thanks for going to the trouble to post this. The differences in exposition, while relatively slight, help make the approach more clear.
Thank you and yes, my sample size is limited to 40 subjects due to the costs of the examinations.  I will have a look online and see what I can come up with and will post my findings to see if I am on the right track.  Thank you
You missed the point. Try to think through what the result means.
This depends on what the conditional distributions are, which in turn is determined by your model. More details needed.
Thanks. Do you happen to know is it also true for covariance?
Why don't you try a VAR(p) model?
stress form the existential crisis is still less than "normal" stress
Welcome to CV, @user25799. Would you mind giving a summary of the information at those links & why you think it will help the OP, so that readers can know if it's something they want to pursue?
If by "this method" you mean Maximum Likelihood, the answer is that there are *loads* of books: you can take your pick from elementary texts through advanced research. If you mean this particular application of ML to a nonlinear weighted errors-in-variables regression, I haven't seen exactly this approach, but it's simple and natural enough I'm sure others have used it before.
Thanks very much- I thought I was on the right track but wasn't sure how express why. That makes sense!
I have no idea how to use it and its property about handling the two-way repeated measures and main effects and interaction term. Can you be more specific or can you give some papers for these models handling unequal sample sizes in repeated measures (not unequal or missing repetitions)?
@Scortchi Yes, it's obvious but how is possible that after running stepwise regression one of the variables would be non significant (i.e. max_depth)?
"fits best" and results of significance tests are not necessarily particularly related  concepts. What do you really mean by 'fits best' in this case?
First I dont think you "assume" what you are trying to prove. Your hypothesis should look like null hypothesis Ho: No change between subjects vs. H1: there is a change between subjects.
@MichaelChernick It is not a good idea to make a new answer to continue older because the system mixes the answer order according to votes.
Thank you - I effectively picked up on the comment of Maciej. I often miss the possibility to accept more than one answer!
Good reference.  Thanks!
Your formulation is loaded: I would want to advise frequent recourse to Hanlon's Razor: http://en.wikipedia.org/wiki/Hanlon's_razor Otherwise I don't think we can easily improve on what Socrates emphasised: Identify precisely what is the real question and how (indeed how far, or even whether) it can be answered using the data (and science) we have. (Socrates wouldn't have used that wording, but updating is allowed.)
all you've appeared to do above is describe leave-one-out cross validation and $k$-fold cross validation. The former is rarely used these days due to high variance and usually large computational costs (some regression settings being the exception). As for your remarks on influence, if $p > n$ there are no unique least-squares estimates, which is a complication. Also, the signs of the parameter estimates can be different as well. I'm not positive, but even when the OLS estimates exist, there may still be situations where your ratio could be $> 1$ for some parameters.
@Michael: Yes; or indeed *make* your data be generated by a known random mechanism - random assignment of experimental treatments.
Thank you @StasK !  This is very helpful and answered my question :-)
And in general, what are you hoping to achieve?  Do you want to predict the recovery rate exactly or just whether or not it was above (or below) a threshold?
But the predictive power of the third variable is the one I (and my reviewers) are most interested in! :)
Well, this a better algorithm than what I've implemented from the post of @Procrastonator, since it seems cheaper. By the way, "bootstrapping" and "online algorithms" are not out of context here, because I'm trying to use particle filters (e.g. bootstrap filter) for an online clustering problem.
I think that the OP is saying "I've heard of people using the log on input variables: why do they do that?"
I think you should look at some histograms by employee to get a feel for the distribution. Since you report that the median is greater than the mean, it almost sounds like you might be *including* the "unresolved" cases for each employee in your estimate. There is also some (informative) censoring in your data.
(1) One crucial issue in any good analysis of this question is whether to use a two-sided or one-sided interval (or test).  What justifies your two-sided approach? (2) By not exploiting the asserted $80\\%$ rate, your solution lacks power.
An example for point 3 is available here:
http://stackoverflow.com/questions/7863942/save-a-random-forest-object
Fourteen kinds of applications covering a wide range of distributions are described under "applications" on the [help page for *Mathematica's* `EstimatedDistribution` function](http://reference.wolfram.com/mathematica/ref/EstimatedDistribution.html).