tttjjj/elt_bench huggingface dataset:
It's not clear precisely what you did: no data, no equations, no code. But you started out alluding to a logistic model but ended with stress on polynomials. Advice is all too easy: a polynomial is one of the worst things to extrapolate beyond the data; a biologically-based model is almost always going to be better. To the extent to which the model you use is qualitatively right for the process being modelled, extrapolations will make more sense. (The fixed temperatures are statistically data, **not** parameters.) There's no shortage of suggestions for growth models in biology.
Stepwise regression has been discussed here a lot, including one [post from yesterday](http://stats.stackexchange.com/questions/35192/what-terms-should-i-include-in-a-linear-regression-model). See also [search on stepwise](http://stats.stackexchange.com/search?q=stepwise).

Stepwise does not necessarily give the "best" model for any definition of "best"; all subsets regression will give it for some definition of "best". But substantive knowledge is always key, and there may not be any one best model.
@dfrankow and @Sam Swift. I am sorry, been a bit busy writing some papers. Ok - if you get a p-value = 0, it means the model is NOT a good fit as goodness of fit test fails. Regarding the problem of AIC, wikipedia and I are using a different convention for it. I am using the one that is used by BDR and WV. (cf. Extending linear Models in R, by Dr. Julian Faraway)
@Felix. I think Sadax correction assumes that each test is independent. Even if the variables and the relationship between the variables are dependent, wouldn't the test be independently done? (The variable are not combined)
I think a value of alpha close to 1 means estimates are based on the recent observations while small beta implies that trend is not changing a lot. Correct me if I am wrong.
That diagram does *not* tell you your distribution is beta. It says the skewness and kurtosis are *consistent* with a beta - it could easily be lognormal, for example, but it probably isn't actually *any* of the distributions named on that diagram.
"It is wrong to state categorically that something is right or wrong" :-)  I tried to shed light on conditions that tend to *limit* the soundness of making such comparisons.
Hint: draw a line representing $x+y=z$ overlaying the unit disk. Rotate the diagram by $\\pi/4$ radians (in either direction) and see if you can write down the answer _without doing any computations at all_.
+1 Incidentally, Jay Kerns is a [highly regarded member](http://stats.stackexchange.com/users/1108/g-jay-kerns) of our site :-).
@user37577 That's the sort of thing you'd need to calibrate the probabilities. I'm guessing, though, that sensors of the type you mentioned aren't perfectly accurate? Otherwise you could just set up those sensors for user presence, and avoid all of this statistical inference in the first place! So you'd also need to account for possible inaccuracies from those sensors. I'd suggest the alternative of manually checking for the presence of the users at 9am for a few days, if that's possible, because it would be a reliable sample.
:-) Thanks Gavin.
@ttnphns thanks. I didn't know about such macros. However, from a cursory glance, dedicated sem tools like Amos are presumably more suited to the task.
@Nathan I don't want to sound like I'm harping on a point or obsessed with it--I'm not--but it occurs to me that people who use $R^2$ to claim their model is good in an absolute sense could often be...mistaken.  One lesson of $R^2$ is that a model-fitting statistic is interpretable only in the context of the dataset.  When two datasets potentially have nothing in common, what would it really mean to compare two such statistics?  So, to get started addressing your question, we need to make assumptions about how the two datasets might be related to each other.  Any suggestions?
And this illustrates why I asked for code; rather than needing to *ask* what the Tukey test is doing, I would know. Rather than asking about the interaction plot, I would know (or have a good guess). Nothing against @jgstat, just illustrating my point.
Alternatively, if somebody could suggest how I could make a more answerable question that would be very much appreciated!
Well, OLS is a very simple generalised linear model. The wiki page on Heckman's procedure (ref 6) indicates that there are some nonparametric approaches that can be taken, but I don't know a huge amount about these. In addition, I think @IrishStat's suggestion of incorporating time structure into the model is definitely something which would be useful.
Looks like a useful program, but their webpage does not do a good job of convincing me it will solve this exact problem. It looks like it may be too broad, too many features I don't care about, making it hard to do the simple things. I will give it another look if the other choices don't work out.
$X\\sim\\mathcal{N}(715.068,9.489229^2)$. Using R:  
`> mu <- (42.5^2*738+10*13.4^2*692)/(42.5^2+10*13.4^2)
; sigma <- sqrt(13.4^2*42.5^2/(42.5^2+10*13.4^2))
; pnorm(720,mu,sigma)-pnorm(700,mu,sigma)
[1] 0.6422261`  


Otherwise, using a standard normal table:
$$\\begin{split}P(700<X<720)&=P(X<720)-P(X<700)\\\\&= P\\left(Z<\\frac{720-715.068}{9.489229}\\right)-P\\left(Z<\\frac{700-715.068}{9.489229}\\right)\\\\&\\approx P(Z<0.52)-P(Z<-1.59)\\approx 0.6426\\end{split}$$
ARIMA models do not need stationary and non-seasonal data. They are linear. Gold data is certainly not seasonal (or people would exploit the seasonality to make money). It is non-stationary, and it is debatable whether it is linear.
Strange that a "" value was sneaking through. It's been removed and all is well. Thanks!
I think with the given data structure there isn't much more to describe. I have customer IDs and booking dates per customer only. I assume some customers return by chance and some become loyal. Using years of booking for non-loyal customers: `P(2014)=P(2014|2013)=P(2014|2013,2012)=...`. For loyal: `P_l(2014|2013)>P(2014|2013)`.
+1 for the geometric explanation.
@jerry, your question looks simple, but a meaningful answer is really the topic of multiple chapters in a regression textbook.  Your question depends on what is meant by "significant", there are several different questions that investigate significance, the above output has tests for 2 such questions, but others will require fitting additional models and comparing.  What question are you trying to answer?
Could you provide additional context about this? E.g., do you work with ordinal data such as Likert-type item or discrete scale scores that are are highly skewed, where you want to find some kind of meaningful quantiles on the underlying scale? What's the purpose of this discretization?
Is subtracting the values at t=0 from your remaining conditions (4 time points x 2 conditions) an option? Then, the difference from rest could be the variables you put into your ANOVA.
5 years of daily exchange rates were used for analysis. According to ACF and PACF knowledge, mixed model (ARIMA) be the appropriate and several models were fitted to select suitable model. But, P value of Ljung-Box statistics is zero for all models.

Since I didn’t get the correct model, R package was used and got the best model as ARIMA(5,1,2) using auto.arima() function. Though refit the model using minitab considering the obtained order the above mentioned error message was displayed.
Thank you. I particularly like the interpretation of the chi-square in terms of BIC.
+1, nice answer--very clean. I appreciate the reference as well.
But i guess changing the lower limit to 0 is valid because $f_Y(y)$ is a function on $(0,\\infty)$ which is indicated by the unit step function $u(y)$.
On the contrary, I think this situation demands that there be some bias.  In the extreme case, only the negative values are of concern and positive values are of no concern.  It would then be ideal for all the error measures to be positive.  (Imagine trying to estimate the lower envelope of a scatterplot, for instance.)
Could you address which of the concepts is tripping you up instead of just posing the question directly?
Hi MansT, many thanks. Your proof is so clear.
Can you be more specific about what issues your classifiers appear to be having? Do they perform above chance?
Various properties of distributions and their manipulation tend to break as soon as you have a literal 0 probability of something.
NB Non-Bayesian tests can't give you the probability that a sample comes from a uniformly distributed population; only the probability that *if* it does you'd get a test statistic as or more extreme than the one you got.
And I found this: http://stats.stackexchange.com/questions/22665/how-to-integrate-principal-components-with-glm
As for your permutation test, I'd be very interested to see exactly how this was conducted. I was thinking of posting a solution based on a bootstrap or permutation test. Note that this kind of thing must be done pretty carefully with multilevel data!
"*According to the Kolmogorov-Smirnov test and the Shapiro-Wilk test, neither by References nor by Causal Stories are normally distributed*" -- there are two errors here: (1) ANCOVA/ANOVA doesn't assume that the DV's are unconditionally normal, only conditionally normal.  (2) the Kolmogorov-Smirnov is only for a completely specified distribution.
He did say almost everywhere
A QQ-plot compares observed and expected quantile values, whereas what I've implemented above compares the observed versus expected density of data between a given set of expected quantiles. I edited my original question to clarify that I'm not seeking alternative fitting approaches, I'm more seeking input whether there are any obvious pitfalls to the one I've implemented, and also whether it is an already named method.
See [these lecture notes](http://galton.uchicago.edu/~lalley/Courses/313/ConvergenceRates.pdf), especially part 3.
You may find the following thread informative: [is-there-a-minimum-sample-size-required-for-the-t-test-to-be-valid](http://stats.stackexchange.com/questions/37993/).
@Matthew $Q(X_i)=0$ for some $X_i$, not all. I am considering a workaround to base $P$ and $Q$ on *the same set* of outcomes and add a small *pseudo count*, say 0.001, for the not-show-up outcomes. It can avoid the zero-valued probabilities. But I am not sure if there's any side-effects.
@onestop, @mbq, close and merge with related question(s)? The OP has posted three similar questions in rapid succession.
Certainly acceptable for a first homework!!!
Thank You. This works.
is it possible to write an R code to implement this? Thanks!
@NRH, yes, yes. Some neuron misfired probably :)
This greedy algorithm is demonstrably inferior in any number of dimensions.  Consider, for instance, six players in one dimension with scores of $7, 10, 12, 13, 15, 18$.  This algorithm makes the pairings $(12,13)$, $(10,15)$, and $(7,18)$.  The latter is as *dissimilar* as possible.  Substantially better solutions (in the sense of minimizing average squared deviation) can be found, such as $(7,10)$, $(12,13)$, and $(15,18)$.
-1 This is misleading for many reasons, including (1) data almost never exactly follow the posited distribution--and how well they do follow it is routinely checked with goodness of fit tests afterwards--and (2) transforming the data (based on examination of the data) renders most ML output incorrect, especially confidence intervals.
So I take your example as stating that $P(H|S,I)\\neq P(H|I)$.  This is certainly true!  My answer also conditions on $D$ though.  This is because, if you tell me the stopping rule, but not whether you actually did stop, I can figure this out from the data set that I actually have.  In fact, I can figure out  if *any stopping rule* would have actually stopped, once I know the data.
The location changes after each attempt
Are correlations susceptible to the usual multiple testing issues?  Yup.
@user123276 your statement is generally true.
(+1) @Cardinal One small thing: the resulting diagonal matrix does not need to be the identity.  It can have an even number of -1's.  But those can be converted to +1's by means of suitable rotations (of 180 degrees).  It's also worth nothing the implicit assumption that the only rotations one can use are with respect to a fixed basis.  If not, then *any* rotation can always be written in the form $R_n(\\theta)$ (that is, as a single such matrix) by choosing an appropriate basis adapted to that rotation.
For 1. you only need diff(), not a for-loop.
@Glen_b: I have updated the Question, please have a look, thank you
I imagine you omitted an intercept term in your second paragraph. I doubt turnover is proportional to Celsius temperature (temperature units not stated, but can be guessed).
This site must have several hundred such datasets by now, if not more, scattered throughout its answers. Although there is no systematic way to search for them, a quick look at the highest-voted threads for popular tags such as the [tag:regression] tag will turn up a bunch: try [this search](http://stats.stackexchange.com/search?q=[regression]+is%3Aanswer).
(... cont.) I don't quite follow what you mean by "we need to have the joint distribution of the $\\boldsymbol{v}'$," but I think I have enough information to find or come up with a satisfactory method on my own. Thanks.
There can be no universal `best practice`. PCA and LDA, as dimensionality reduction techniques, are very different. Sometimes people do PCA prior LDA, but it has its risks to throw away (with the discarded PCs) important discriminative dimensions. The question that you ask has actually been asked in some form several times on this site. Please search "PCA LDA", to read what people said to it.
Hi @JohnYetter, do you think you could give a little bit more detail in your answer? Right now you are only referring to a YouTube video, which could potentially be removed at some point.
+1 Very nice paper.
OK, finally we get here! But do you think it is reasonable to think that subjects with shorter survival times are more likely to failure at any particular time _T_?
You don't need a hash table: just index directly into an array using a discrete version of $x$ and $y$. For instance, the linear interpolator `f` defined by `z <- pnorm(i <- seq(-6, 6, .01)); f <- approxfun(i, z)` will obtain results good to within 2% whenever $x$ and $y$ are more than $0.003$ apart (with relative accuracy increasing rapidly as $y-x$ increases). The linear interpolation is twice as fast as `pnorm`, which itself is very fast: my machine computes seven million values per second. Maybe your problem is with your coding style and not with `pnorm`?
(+1) Nicely formulated (multiple) question(s).
When I reported ''$p<10^{-26}$'' in a conference paper, a reviewer told me that I should change it to ''$p<0.001$'' in order to follow APA guidelines.
That sounds like MCAR, then, as long as the two individuals collected data on similar cases. If that is not true, then it's NMAR, possibly severely so.
Closely related: [appropriate-normality-tests-for-small-samples](http://stats.stackexchange.com/questions/1645/). Here are a couple of other questions of possible interest: [is-normality-testing-essentially-useless](http://stats.stackexchange.com/questions/2492/), for a discussion of the value of normality testing, & [what-if-residuals-are-normally-distributed-but-y-is-not](http://stats.stackexchange.com/questions/12262/), for a discussion / clarification of the sense in which normality is an assumption of a linear model.
What about general case?
Hi John, welcome to Stats.SE! If you're satisfied with either of the answers, you should accept one of them, or provide more clarifying questions about what you're looking for.
I'm not sure I understand your last comment. In the case that my noise is normally distributed, you mean that I can add a noise term in addition to the generalized normal distribution? It's still not clear to me how to choose the parameters $\\alpha,\\beta$ for the generalized normal distribution.
One really easy way to check model fit is a plot of the observed vs the predicted proportions.  But this won't work if you have bernoulli regression (i.e. all of your observations have unique combinations of the independent variables, so that $n_i=1$), because you will just see a line of zeros and ones.
I must admit I never fully understand why Gelman advocates the use of histogram with small bin width; why not using stripchart plot or raw data with superimposed kernel density estimates, which much better convey the empirical distribution of the observed data?
That would normally be called 'calculating the sample covariance matrix', or 'estimating the covariance matrix' rather than 'sampling the covariance matrix'.
I am not sure I understand what you are asking, but one possible help is that 100 people are, presumably, a sample from some defined population of people. What is the population of coin flips? If you can't define a population, I don't see how you can define a sample from it.
I need your help on my statistics. I have a group of people with e types of diagnostics: 1. clinical and the 2nd hystopathological. Normally, these two diagnostics must be identical, but due to some clinical errors they aren't always. So whatI have to do is to find the corellation between the two, how many match and how many dont match. What test to I need to use?
Thanks for you help, guys !
@user1330974 You are welcome.  Many of us occasionally don't see things that are staring us in the face and later slap ourselves and go "Duh!"
Its worth pointing out that the observations are not independent since traffic from each customer is inherently dependent on the traffic from/to another customer. I'm not sure which statistics should make sense in this case.
@user1121 Yes, but only in case of binary classification.
Anyway, Pillai's trace is a multivariate statistic (like Wilk's lambda or Lawley-Hotelling trace). The first df is the number of degrees of freedom of the model term (which clearly are all continuous) and the next two dfs are the numerator and denominator degrees of freedom for calculating the F statistic. The p-value is the test that the probability of the observed data is greater than F.
It would be helpful to provide a reference for the other analysis. We don't know what "completely different to what they've got" means. Perhaps they took the apparent RMSE which is spat out by most software. You can look at this for your model using `summary(model$finalModel)`. Without more information, it is hard to answer the question.
Does the sequence part of your question matter?
In any case, my argument does not use the _distances_ between the ordered samples from the uniform distribution.
Elvis: as the Wikipedia article helpfully points out, circulants can be viewed as elements of the left regular representation of the group algebra of $\\mathbb{Z}/n\\mathbb{Z}$.  Everything follows from that via the theory of group representations.  Having an FFT available doesn't really speed things up asymptotically: at best, it halves the computation time.
You say that you have "ran an experiment" and you say " how to go about computing sample size". The short answer is "count your number of participants". But I assume you intended to ask "what sample size is required in order to achieve a given (perhaps "acceptable") statistical power. I'll edit your question to reflect this; feel free to modify if I have misconstrued.
Every answer to my question has provided useful information and I've up-voted them all. But I can only select one answer and Srikant's provides the best overview IMO.
Thank you for the reference.  Welcome to our site!
@Marius - that's quite an involved hierarchy, so I agree with your comment about it being a complicated model.
+1 for a nicely phrased question. Because this is a homework/self-study problem, could you add the `self-study` tag to your question? See http://stats.stackexchange.com/tags/self-study/info
Is $\\alpha^*$ supposed to be the appropriate critical value, or just the actual long run type I error rate when everything was done conventionally?
To help with googling for related problems; an algorithm that processes a stream as it comes by is called an online algorithm http://en.wikipedia.org/wiki/Online_algorithm
+1 Perhaps you could point to the evidence you see concerning *why* this might be a "poor model" and how time series methods would improve on it?  It seems a little unfair just to leave the O.P. hanging after such a pronouncement!
Possible related question: [Group differences on a five point Likert item](http://stats.stackexchange.com/questions/203/group-differences-on-a-five-point-likert-item).
Are you the original asker of this question?
I wish we could answer all questions about the universe here--perhaps we come close :-) -- but because this is really a question about physics it probably belongs on the Physics site (unfortunately it is too late to migrate it).
Hey @Crystal - welcome to the site!  This is a good question, but you should pose it as a new question (the "Ask Question" button on the top right).  Right now, you've submitted it as an "answer" to this old question.  If you just copy and paste the URL of this question into your new question, we'll all understand what you're talking about.
just because lots of people treat likert scales as interval doesn't mean it's right... it's only defensible in certain circumsances
Note that *both* your explanations of p-values are loose/informal & not strictly accurate - just plain wrong really. Why not explain better or omit to explain at all?
@nan 1.Because the interaction is significant, the main effect should be included. But I just mean that textbook.usage.glm itself rather than model selection. We seldom drop variables simply by Wald test. 2.Visualization is seldom used for multivariate models. But you can try to plot textbook.usageSession~condition.player, with different values of session.week. This plot of final glm will be several lines with different slopes and intercepts. However, the other pair plot textbook.usageSession~session.week is supposed be more in a mess because of the random effect (you may have a look).
see http://stats.stackexchange.com/questions/88461/derive-variance-of-regression-coefficient-in-simple-linear-regression
Because the principal components of the Y's have *nothing* to do with *X*, why does this help?  For an explicit example of the difficulty, please see [this comment](http://stats.stackexchange.com/questions/32471/how-can-you-handle-unstable-beta-estimates-in-linear-regression-with-high-mul/32577#comment63983_32577).
Date of birth is intrinsically important for calculating what is allowed or expected in society (can you drive a car legally, do you qualify for a pension?), so for such purposes it should remain human-readable. Often date of birth is important in giving age, for which the needed precision may mean hours or days (for new-borns) or the allowed precision may just be years (for older people). Either way, converting date of birth to number of days or years since time origin seems most usual for statistical analysis.
What is the value of the likelihood ratio $\\chi^2$ from the Cox proportional hazards model?  What makes you trust the Kaplan-Meier estimates?  Did you compute the confidence intervals for the difference in two Kaplan-Meier curves?
Did you try `?wilcox.test`? Or [Wikipedia](http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test)?
What are the variances of the groups and the means?
There is also a zelig list, you might get a solution to the error there.
related question, [Logistic Regression - Multicollinearity Concerns/Pitfalls](http://stats.stackexchange.com/q/4854/1036)
@Srikant Could you explain how you derive the "trivial answer"?  (There must be some unstated assumptions behind it.)  And to what theorem do you refer when you conclude that "eventually everyone will be connected"?  That's not at all obvious!