tttjjj/elt_bench huggingface dataset:
Ok then then it is not really anymore a statistics matter. I guess the poisson process approximation is still valid as long as rounding to integers has minor impact on real time values. You can try to plot delta_t for your dataset and check if it looks like an exponential distribution or a comb. As an example, if you try ``delta_t = scipy.stats.expon.rvs(scale=1., size=10000)`` followed by ``plt.hist(np.around(delta_t), 200)``, you will see a comb. If you redo the exercise with scale=20, then you will notice that the distribution is much more like an exponential shape.
$a$ wouldn't be the called the confidence level (it's normally the significance level); if you use the term 'confidence level', you'd apply that name to $1-a$. But yes, I believe your expression for (1) is now fine. So for $1-a=0.95$, you'd use $Z_{0.975}$.
@whuber, I may have mis-read it. TBH, I'm not completely sure. The text description connotes the expected count under independence to me, but then the formula seems like the conditional probabilities (w/i the row & w/i the column). Since I can't say for sure, maybe I should delete this.
Thank you. It's always very reassuring to have somebody look at your work. I'm going to put "Figure 1. Mean reaction times (in milliseconds, with SE) for set sizes in the different grouping conditions." under the graph to make it clear. Is saying "in milliseconds" redundant?
Can you please work on the style and clarity of your question?
Good question!  I was about to post that you can simply rely on what `summary.gam` gives you after you replace the adjusted coefficient vector and VCV into the gam object.  That is what I have done previously.  But as I think about it, it's not right!  The details of p-values for smooth functions are in Simon Wood's 2013 Biometrika paper.  The reduced-rank wald statistic that `summary.gam` calculates needs the model R matrix (from a QR decomposition of the X matrix), as well as the VCV, the beta vector, and the residual degrees of freedom.  Obviously the X matrix will vary between imputations.
I forgot to mention that often adding some ~meta variables can be helpful, especially if your goal is just prediction accuracy.  An example might be, how many features are non-zero for this observation.  If you have more knowledge of the context of the features this might make more-or-less sense.
@user8812: You can calculate it the same general way you have in your table. You now have a Poisson with mean parameter 6.45. So, the probability of scoring five or fewer points is found by summing the probabilities from 0 to 5 (inclusive). Subtract the result from one. (You should get about 0.624 as your final answer.)
I'd add two qualifying comments. First, in some fields these plots are used as propaganda plots, to the effect of "see how good my model is". Of course, anything can be abused. Second, and linked, the example is one where structure is quite easy to see. If structure is more subtle, and/or there is much noise, I'd assert that it's easier to see structure on a residual vs fitted plot, which uses space better and gives a horizontal reference. Conversely, it is possibly true that non-statistical people regard observed vs predicted plots as easier to understand.
You forgot to log $(\\frac{\\lambda_{0}}{\\lambda_{1}})^{n}$
If you understand how to do logistic regression in R then you might find this answer helpful: http://stats.stackexchange.com/questions/6067/does-an-unbalanced-sample-matter-when-doing-logistic-regression/6086#6086
No, it's related to whether more than half of the outcomes are Y when color = green.
@Alec You are welcome.
@JohnRos: Is local FDR differently defined from FDR and pFDR?
Are you assuming that the _hazard_ rate is constant over a period or the _survival_ rate is constant? If $X$ denotes the lifetime of a component, the survival rate at time $t$ is $$P\\{X > t\\} = 1 - F_X(t) = \\text{percentage surviving at time} ~t$$ while the _hazard_ rate $h(t)$ is given by $$h(t) = \\lim_{\\epsilon \\to 0} P\\{t < X \\leq t+\\epsilon \\mid X > t\\}$$ which is a different thing. In other words, "...assuming that hazard/survival rates are constant..." makes no sense; _both_ cannot be constant simultaneously (except for the trivial case when $h(t)=0$ over an interval, i.e. no failures.
This doesn't respond to the question: perhaps it should be understood as a comment.
1) I have increased my sample size from 3 to 15 2) I have decided to do a t-test (paired) although I'm still not sure what the difference it between independent and paired. 3) I'm going to use continuous values. I am also confused about the transformation with regards to the distribution of leaves. Why does having all my results more than 0 matter? Again, thanks alot, you've helped immensely.
It's usually fitted using maximum likelihood estimation.  You really need to say some more about what might have generated the binary data to progress any further.  Also is there a specific data set?  I see you linked to one, but was that just randomly chosen or do you have another specific one in mind?
@rohanbk Sorry, no. It is a long time since I have used matlab.
Well generally speaking, one would normally start with some algebra, and then worry about code implementation. [In *some cases* some particular package may be able to take out the need to do some of the algebra (perhaps symbolically, or via some numerical calculation), but in many situations you would do it yourself. It's not onerous, mostly involving simple operations like completing a square in an exponent.]
Cluster analysis is extremely paradigmatic, that is, different its methods "assume" and "prefer" quite different things, so that what you input and what you use - so is what you get. In this vein, one should not expect different methods to give similar results. Instead, one should first learn a bit about each method and then select consciously just one - which seems to be most appropriate for the current task and the current aim. Also, really effective using of cluster analysis calls for much experience with it. If you're new to it, don't hurry to try everything at once.
Thanks everyone! I realized I should write a script, I did it in Matlab.
I agree about the bootstrap approach.  But power is not a function of degrees of freedom.  In many cases, including this one, the Mann-Whitney test often has *greater* power than the t-test.  See http://tbf.coe.wayne.edu/jmasm/sawilowsky_misconceptions.pdf .  In general, the power of a parametric test is good when the parametric assumptions are true but can be lower--sometimes drastically so--when those assumptions are violated, whereas good nonparametric tests maintain their power.
This statement is only true if you assume that the other methods of survey recruitment contain less bias than an all internet based sampling approach. The fact is that all sampling methods contain some amount of bias, and it is incumbent upon the researcher to be cognizant of what those biases may be and to account for them to the extent possible. Where accounting for that bias is not possible, documenting the limitations of your findings is critical. There is research that shows internet based sampling methods induce less or equal bias than other "traditional" methods for certain surveys.
@LiKao: you could follow a uni-variate rule (in spirit similar to what you were doing, although using the correct rejection threshold as whuber pointed out). That would amount to assuming independence among your measurements. You could also take the correlation explicitly unto account, using a multivariate robust approach. Contrary to your intuition, analyzing the data-set as independent realization (as you seem to be doing) will lead you to *understate* the true contamination rate.
Also I need to show that $\\sqrt(n)(S_n^2-w^2)->N(0,m)$ for some m function of w as n->$\\infty$ in distribution.That is the meaning of asymptotic normality.
prices are usually unit-root processes. Did you check for that? If they are unit-root, your regression might not make sense.
@Elaine, with set.seed(111) and n to 10000, I got 0.044. Which is again normal. By increasing n the percentage must go to 0.05, since then the finite sample test distribution converges to the true distribution.
thanks for your explanation.
Is the servant leadership survey instrument given to teachers? Who fills this out and what does it evaluate?
I'm a Psychology PhD student as well, and I made the choice to take a significant amount of math in my undergraduate years because there were so many Psychology PhDs who have no idea how a PCA (for example) was computed. The very first thing you need to do is work your way through any decent linear algebra textbook. What's a decent linear algebra textbook? Gilbert Strang's is the bomb, and he has video lectures of his linear algebra course on MIT's website to boot. You can even get them on iTunes.
Reflect on that a minute, Michael: just how helpful to anybody but you and your students would your reply be?  Or, consider it from an outsider's point of view: what evidence do you have that these games "work well" or that they are "excellent" (apart from your own--subjective--attestation)?  The absence of general utility or of authoritative support are good reasons to downvote, I'm afraid.
Also density plots and box plots can be useful to highlight different aspects of violations of normality. As an aside, variables that go into a regression do not have to be normally distributed; the residuals from the model do.
+1 - IF the goal is to understand the variability of the estimimates across the MI datasets, I would bootstrap within each MI dataset and look at the total and MI-specific distributions of the parameter.
I like to get a distribution of AIC's, not just 1 or 2.  I look at the distribution of them when I replicate the fit on the same data.  If I have an outlier it means something about the connection between the data, the fit process, and the analytic form.  Now it looks like you are trying to come to terms with your parameters acting to counter-balance your accuracy.  If you really don't care then use interpolation and get 100% accurate, but likely terrible generalization.  Or trade away accuracy "today" for good generalization "tomorrow". BIC?  AICc?  Replicate count?
@Tathagata The real question is: What do we do when there's no gold standard, which is quite frequent in diagnostic medicine (except through autopsy!) or clinical assessment/screening :-) Then comes in various flavour the concept of *convergent validity*, etc.
@Christian: Or take for example $X$ symmetrically distributed about $0$, $f(x) = x^2$ and $Y=|X|$.  Then
Just reading the help, it appears to deal with a number of different types of svm (see the `type` argument) which appear to correspond to more than one possible loss function. So for example, the `epsilon` argument relates to the $\\epsilon$-insensitive loss used in eps-svr, nu-svr and eps-bsvm, but ksvm handles more than those. That is, there *is* at least a little explained in the help. You might also try the vignette and the references mentioned in the help. Finally, the vignette has an extensive reference list.
I corrected my previous post. (Since I wanted to compare gamma and lognormal models, it should be better that the link function is the same)
@Peter Flom Not so. The procedure for Stata's `signrank` takes no account of the mean (which would be bizarre at best). Definitions and formulas, including that for $z$, accessible to all at http://www.stata.com/manuals13/rsignrank.pdf
A colleague of mine looked at the data for this in the post-2000 period, and found that the relationship held fairly well 'out-of-sample', which is even more disturbing...
There is a small slip-up in the algebra here. You've parametrized the exponential via a rate parameter, so the MLE is $1/\\bar x$, which is consistent with your equations.
@onestop edited
One answer: generate $n$ random values.  Divide each by their total.  This works whenever their total is nonzero.  The generality of this solution highlights the real question: *how do you want those random values to be distributed*?
Dear Macro, I didn't intend to initiate such a long comment stream and my remarks were only intended to be constructive. Your answer is fine as is; my primary thought was that (as @Dilip alludes to) appealing to the fact that $\\mathbb E y_i y_j^2 = 0$ and $\\mathbb E y_i y_j y_k = 0$ for zero-mean multivariate normals may appear a bit opaque and this is really where most (or much) of the action actually lies. (It's the only place where any property of normality comes into play.) The comments were intended to show a simple way of bringing out precisely why it was true with no additional work.
Note that adding irrelevant variables is *guaranteed* to increase $R^2$ (not just in "some cases") unless those variables are completely collinear with the existing variables.
As I found out in Deheuvels paper [link](http://www.tandfonline.com/doi/pdf/10.1080/03610920701270824) there is the following relationship between a Brownian Bridge $ B_t$ and a Brownian Sheet (or Wiener Sheet) $ W_t $:
$$ B_t := W_t - \\frac t T W_T $$
So I think the problem reduces to simulating a Brownian sheet. I will ask my questions about this in a seperate question.
I think the answer to your question is embedded in @whuber's excellent answer that you have linked to.  He effectively suggests what is sometimes called a Z test, but with samples of above 50 or so this will be the same as a t test.  NPS will be approximately normally distributed, and you can use his methods of calculating standard errors.
+1 for posting the answer.
My bet is that you are having numerical issues because you are working on the "raw" scale - for numerical ML you should always work on the log scale.  Maximise the log density, not the density itself.  Even though this has an analytic solution, you should still get close using numerical optimisation - else there is something suspect with the numerical optimisation (either the matlab routine or the way you've applied it).
Scale parameter makes it possible to account for nonhomogeneity variances http://stats.stackexchange.com/q/48237/3277
thank you. It is clear and very concise.
@cardinal yes, you are correct. I am a ML and all the times I have encountered a situation like the OP I had a ton of data but a skewed response distribution (like 1% positives), so permutation tests are not feasible. In these situtations it is hard enough to stabilize a model (so that AUC does not drop below 0.5 in a single fold) even with sampling conditionally on the inference (at least from my experience, but I do not have much)
Hi, Thanks for the note. So, is there a test (such as K-S test) that tests shape? Or should I look at basic statistics? Kurtosis, Skewness etc. And if so (and this should be probably a different question) how different should Kurtosis and Skewness (the values) be in order to say that the shape is different?
+1 Thanks for the reference.
And here's a more detailed comparison of AIC & BIC: http://smr.sagepub.com/cgi/doi/10.1177/0049124103262065
What software are you using? Without knowing any specifics theres a good chance you have a bug in your code.
This question appears to be off-topic because it is about correctly formatting monospaced output in Word.
@Glen_b time persistence is simply a fancy term describing non- stationary data.
@gung, I don't usually capitalize in writing, but will do it here
+1 Thanks for taking the time to write out the details.  Can this question be made part of the FAQ?
Just because markets are closed doesn't mean it's regularly spaced. If we think of it as an underlying process that we sample discretely (when markets are opened) but still evolves when markets are closed then it is irregular. I think this continual evolution metaphor is more useful since it is consistent with close to open jumps (all information from closed times being revealed in 1 moment).
Hi stefan; the question that immediately pops into my head is what happens after the first month when you work out that your survey is wrong - i.e. too long, too short, silly questions that get misinterpreted etc.  How are you going to plan around this?  Otherwise, do you expect seasonal effects?  Are you looking for them particularly?  What do you expect to learn from the analysis of time dependent effects?  If the answer is "not much", I'd be very tempted to put all surveys together and ignore the temporal element - "all models wrong some useful".
If the numbers are wrong, how can the idea be correct?
Keep in mind that the answers you get will be quite limited by your design, for which the intervention is totally confounded with the passage of time.
@Roark: You can add the `log=TRUE` argument to `dbinom` to get an answer on a log scale. That may help you get a feel for why you're observing the behavior you're noting.
You may be interested in reading this thread: [What machine-learning algorithm can be used to predict the stock market?](http://stats.stackexchange.com/questions/21395/)
Thank you, this was very helpfull
@johanvdw. I think we do need to add the error in normal linear regression for prediction of a new Y (for a new X). Please see : equation 2.38 in section 2.5 of Applied Linear Statistical Models - Kutner, Nachtsheim, Neter, and Li (2005). McGrawHill
Possibly of interest: [Is it possible to have a pair of Gaussian random variables for which the joint distribution is not Gaussian?](http://stats.stackexchange.com/a/30205/7290)
Not to be critical of the question at all, I think it is interesting to note that the problem statement shares some of the ridiculous features found in many textbook exercises. A notable (but subtle) one is the combined use of too many digits for the model year (the year should have been referenced to some date around 2000, not 0 CE) and far too few significant digits for the output. Because "0.163" comprises all values between 0.1625 and 0.1635 and will be multiplied by a three-digit number and exponentiated, we can only conclude that the answer lies somewhere between 8,677 and 63,418!
@Seb (About the flag you raised) I believe this question should stay here because there is some statistical background.
@user1893354 Yes. By modeling them as two separate processes, you could for example have no censored observations for "Time to Olympics" and a relatively sophisticated interval censoring scheme for "Time until Injury".
One way to see what is going on is to look at collinearity diagnostics, particularly condition indexes and proportions of variance explained by them.
Could you help me in identifying the right set of technical inicators. Even I was thinking there is some problem with the input indicators. Could you help me what criterion input technical indicator should follow?
@IMA Very good advice, but econometrics is the field most concerned with time series? That's a widespread impression within economics or econometrics, but many engineers or physical scientists would raise an eyebrow or two at that. (I am not trying to start an argument, just underlining how many fields have serious interests in time series, each tending to regard their own perspective as central.) It's unfortunate that the OP leaves open what field they are working in, but the interest in "concentrations" suggests that it is _not_ economics.
How many covariates are in your model?
+1 If you continue to find more sources, please feel free to augment this answer.
Once again, this is excellent. Thank you Zen.
I don't have a reference where the suggested function is used to compute the moving average; I just customized the [first mean value theorem](http://en.wikipedia.org/wiki/Mean_value_theorem#First_mean_value_theorem_for_integration) to your needs.
How can you have factor scores at the level of each person when your factor analysis was performed at the level of brand?
I would be wary of issues like regression to the mean. See also subtleties in causal inference and bias in causal inference: Glymour, M., Weuve, J., Berkman, L., Kawachi, I., and Robins, J. (2005). When is baseline adjustment useful in analyses of change? an example with education and cognitive change. *American Journal of Epidemiology*, 162(3):267–278.
As of how to adjust for inference on a 'final' model after stepwise selection, well that's the issue someone has yet to figure satisfactorily (that I know of). Some statisticians simply recommend not conducting stepwise selection in lieu of advanced procedures such as penalized regression, which is fine. However, those procedures are not readily available in other situations, e.g. mixed effects models. So it would be great to be able to trim a model quickly and easily with some sort of stepwise elimination and adjust SE's, CI's and p-value's accordingly.
+1 Very nice methodological answer. However, The spirit of my was question is looking more for a package in R to handle some or all of these steps. I thank you for your contribution
+1  You ought to consider expanding this answer in light of the comment thread below the original question, because the OP wonders about applying a changepoint method near the ends of the series.
@JMS, "how many intro stats students know the real definition of a confidence interval?" Or, PhD stat graduates, for that matter.
Just a side tip, don't name dichotomous variables this way. Instead of calling it "gender," call it "male" or "female." That way you'd know what 1 (aka Yes) stands for. If this is "male," then you can easily figure out 0.47 is the fraction of males.
Having `some kind of decision support tree` is always beneficial. But `software designed to automatically check the assumptions`, if it means "automatically choosing the most appropriate test" is a project for castrating a man mentally. Unfortunately, some otherwise respectible packages (e.g. SPSS) have already started deadly efforts in this direction.
At risk of being repetitive, could you pls give a proper citation to Wikipedia when you copy/paste entire paragraphs on Lasso (http://en.wikipedia.org/wiki/Least_squares)?
Since an intercept is sort of an effect, a random intercept model is a special case of a random effect model. But you are right, there are many different terms for the same methodology.
Well, MDS, as I'm aware, need dissimilarity (distance). If you input similarity such as correlation a typical MDS package will convert it to distance as `1-r`. [Note that](http://stats.stackexchange.com/a/36158/3277) `2(1-r)` is _squared_ Euclidean distance. Therefore, the difference of your results you observe may be explained by the different input: Euclidean d vs Euclidean d^2. Another reason to consider may be that you have negative r's, and abs(r), not r, were internally converted into a distance. Consult the package documentation, to find out.
"I hope you now understand why you can't just give a statistician data and say "please predict what happens next"." - That is sound advice, indeed.
(1) If your series positive, take the log of the dependent variable, then fit arima to transformed series. (2) The book comes with R examples, and R package with code and data, check out their web site. It's a great book on time series, and they have the sequel on nonlinear modeling. They also have the free version of the book,all on their website.
@pbneau R questions are acceptable on here.  See http://meta.stats.stackexchange.com/questions/252/should-we-allow-more-computing-questions and the FAQ.  See also the recent discussion: http://meta.stats.stackexchange.com/questions/474/printhello-world-n
It is not meaningful to talk about one- or two-way comparisons in an omnibus test such as the F-test.
Awesome thanks heaps for your help Jeremy!
How else are you going to get $K = \\phi(x)^T \\phi(y)$, if you don't take square roots? Because $K$ is symmetric in its arguments, $K(x,x) = \\phi(x)^T \\phi(x) = 1 + 2x_1^2 + 2x_2^2 + x^4_1 + 2x_1^2x_2^2 + x^4_2$ immediately gives $\\phi(x) = (1,\\sqrt{2}x_1,\\sqrt{2}x_2,x^2_1,\\sqrt{2}x_1x_2, x^2_2)$. You can generate $\\phi$ element by element, just by inspection here.