tttjjj/elt_bench huggingface dataset:
To use the Ljung-Box test on residuals rather than the original time series you just need to adjust the statistic's degrees of freedom to take account of the number of fitted parameters.
Any answer will depend strongly on the industry you're interested in. For pharma and other bio-related research and work, SAS has a very strong foothold, to the point that it is often implicitly assumed to be the software preferred by US regulatory agencies for submissions and reporting. (See [this](http://blog.revolutionanalytics.com/2009/01/analyzing-clinical-trial-data-with-r.html) though!) However, in some other technical fields where heavy statistical analysis is done, you'll find R is currently (far) more prevalent.
Sounds similar to reliability/survival data, where most of the failure is early or late.
Regarding the first question, it is possible to modify the profile likelihood in order to reduce bias (not necessarily to $0$) at least for the exponential family, [see](http://biomet.oxfordjournals.org/content/80/1/27.abstract). I am skeptical about the existence of an unbiased estimator asymptotically efficient in the general case.
Very nice answer, but I think the first link might be broken.
Thank you very much :D :D. I will take a look at the KNIME link that you sent.
If I get you right, you're saying that the error term always moves with $x$, so getting the linear coefficient on $x$ will ignore the correlated effects in the error term.  It is true that observational variations in $x$ will change $y$ through $\\epsilon$ as well as $x$.  The interpretation of the IV however is generally "what would happen if someone went and set some policy that exogenously changed $x$ and only $x$?"  Of course, the imposition of such a policy would probably change a whole other set of unobservables (Hawthorne effects, etc), but that's a different problem.
Thank you very much Blain Waan. I really appreciate it. Very helpful!
@B R I don't know if it will change your answer, but I have edited the question to be more specific.
But you can still use the periodogram to identify smaller frequencies. You do not have enough information to identify yearly patterns, but you could identify monthly or weekly patterns.
@Jeff Do you mean that you performed ML-based FA with 9 statistical units (rows) and 6k+ variables (columns)? Because, in this case, it won't work.
because the 61st *standardized residual* is ~2.5, not the 61st observation. Check the axis labels on your plot, and try `rstandard(lm(y~x))[61]`. The x-axis is *not x* but the (approximate) expected largest order statistic of a standard normal.
Your question is extremely hard to read, and I hesitate to edit it as I am not sure what you are asking. Please edit it. You can use LaTeX code enclosed within pairs of dollar signs for inline equations, or within pairs of double dollar signs for displaystyle equations.
Oh, I meant to say, since your values are *measurements*, you should be careful of specifying normality; normal distributions have density at 0, while measurements don't.
+1 for a nice clear exposition.  Of course, this answer is only correct if the placement/order of the boxes matters.  If not, all that matters is the final three unknown choices, for which there are only ${4\\choose3}=4$ possibilities.  But I think your interpretation of the question is probably correct.
Thank you so much for your elaborate answer. It did get me a bit on the way but the estimate that it gives are not really in line with what I expect based on my other models. I have tried to make some changes in the equation based on your first comment (about h not being part of a). This seems to be taking care of most of the problems but still is giving a bit a 'wonky' profile for the allometric component. At least the estimates are now a bit more in an expected range. I'll keep wiggling a bit with the equation and hope I'll get something that will work. Once again, thank you very much.
@gianluca: it sounds like you have 1. a lot of data, 2. limited memory resources, 3. high precision requirements. I can see why this problem is freaking you out! Perhaps, as mentioned by @kwak, you can compute some other measure of spread: MAD, IQR, standard deviation. All of those have approaches which might work for your problem.
@NathanCooper, you're absolutely correct. The paper I linked to describes how bad fishing expeditions can get (and it's pretty grim).

However, having spent a lot of time and money gathering data, it would be nice to get *something* out of it, even if that something is tentative hypothesis that needs to be rigorously confirmed.  If including very plausible covariates produces a big effect size, then that can be relatively convincing (but yes, still needs to be explicitly confirmed). If one needs a crazy-quilt of inclusions and exclusions to push something *just* into significance, then...no.
Thanks again @Glen_b - I really appreciate your time on this and your answer gives me food for thought (and re-assures me that what I'm asking is non-trivial).  >>> "As for whether the size of difference matters, that's harder to say without knowing the application - you might be better placed to judge it" - yes that is something I will have to think about, but your ideas are very helpful.
@whuber: I see... Thank you...
Diagnostic for Univariate model is well set up, but I did not see a routine diagnostics method for multivariate models where the response has more than two variables like here.
I think AGTE is at a considerably higher level than the Dummies books.
Congratulations, I have awarded you the bounty.  Thanks for your help Henrik.
Can you supply the `formula` object you wrote to fit this model? I think you should be using the `*` notation to generate interaction effects instead of `:`.
I don't think this is about choice of means: you are both working with the idea of the mean as fraction of successes. Clearly 3/4 people achieved success and 6/12 attempts achieved success; I struggle to see merit in your colleague's idea that you put people in the numerator and attempts in the denominator. I know nothing about scoring conventions in basketball, but if you try a translation to something else, say 1 is cereal at breakfast and 0 is not, then 3 people had cereal at breakfast some day, 6/12 breakfasts included cereal, but 3 people having cereal some day/12 breakfasts???
@Glen_b: thanks, but that's the thing: i'd like to use the linear link function.
Thank you for your reply. That means there is no differentiation?We are just putting the value of 1/L to denote maximum entropy?Is my understanding correct? Secondly, I have another question if you may kindly address it. Entropy = negative of information.So what can we say about information at maximum entropy?Will information decrease when entropy is maximum due to the negative correspondence?
@whuber: I don't think m will be larger than 7.
@JoshuaShaneLiberman That is true, but it's also true for any other number: $f(p) := k-p$ implies $f(f(p)) = k-(k-p)=p$. It's not practical to have a special name for every possible value of $k$. Maybe a name for the general $k-p$ case might exist, but there's no reason for the specific $1-p$ case to have one.
Nice idea. A similar option is to replace the green *droplets* with histograms or interpolated densities
I was going to comment but didn't because I didn't feel like my answer had much depth. Turns out that's exactly the case. I'd just like to add that, as a fellow masters student, phrases like "I figure" and "look at" are red flags to myself that I don't actually have any clue what I'm doing and need to think my project through much more carefully.
Machine learning is usually high dimensional modeling. Many methods are special cases of existing likelihood based methods with penalties or weighting employed.
Found article about [Multiple Regression with Categorical Variables](http://www.psychstat.missouristate.edu/multibook/mlt08m.html), but this technique requires coding of dummy variables, wich is not suitable in my case (>50 binary predictors and the number may extend in future).
AIC can be negative. You can still compare fits in the same way regardless of the sign of AIC. "Akaike's logarithm" is a log likelihood, though I wonder if you extract the log likelihood and you've accidentally tried to take the log of that.
@Anna, you'll find good examples of application of these methods in the original papers (or papers citing them). For [example](http://bioinformatics.oxfordjournals.org/content/20/11/1728.short)
@onestop: Note that $|\\mathbb E X| \\leq \\mathbb E |X|$ so if $\\mathbb E |X|$ is finite, so is $\\mathbb E X$.
@Ben I agree but I don't understand where the perception of "more complex" comes from, because analysis of univariate distributions and post-hoc analysis of residuals are essential in any regression exercise.  The only extra work required here is to think about what these analyses mean.
@PeterFlom Thanks, but I set same parameter for each fold, thus the only difference among folds is the data set. Any suggestion to do next in order to find out the problem?
Just to clarify, are you talking about one-way ANOVA or factorial ANOVA? In particular, an assortment of issues arise when the data is not balanced in factorial ANOVA (i.e., the predictors are correlated).
See this related question for some references, [Variance as a function of parameters](http://stats.stackexchange.com/a/102913/1036)
Good start, stevie. You have the scale factor correct, but not the conclusion regarding the unbiasedness. Your follow-up question is a good one, with a fairly deep answer. You might consider adding it to the question to make it more prominent. :)
http://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs covers this design
Thanks didn't know the alias() function. That's really handy to have. Cheers, O.
Because you appear to be in an exploratory mode ("trying to come up with a model"), why not first use exploratory methods like loess smoothing?
you make it look so easy... :)    thanks
This sounds like a job for [survival analysis](https://en.wikipedia.org/wiki/Survival_analysis), if only to determine whether observations are missing at random...but I doubt they will be. This could get tricky. I wonder if you could use (stage number of) inability to continue as an indicator of fitness...
If you only have 1 set of outputs then you will only have 1 precision, recall, and F1 score. The idea is that for each model you would compare these scores to determine which model is the best.
I'm not familiar with Hartigan's dip test. I'll read up on it in case it could help. Unfortunately my current data set has been separated from all recipe text, but I think a classifier is a great idea for the future. Thanks for the detailed answer.
It's really hard to say without knowing the specifics of your problem. Zero-inflated Poisson regression will allow you to model causal effects like area both on the amount of zero inflation *and* on the Poisson parameter (or only in one of the two components). Which model to use will depend on the underlying causalities of zero inflation and event count you hypothesize.
**You need an additional parameter.**  See what happens with `fit <- nls(Emissions ~ a* (1- u*exp(-b*Days)), start = list(a = 2000, b = 0.1, u=.5));  beta <- coefficients(fit); curve((y = beta["a"] * (1 - beta["u"] * exp(-beta["b"]*x))), add = T)`.
Thank you sharing your experience with us.  Welcome to our site!
@AntoineVernet This is often a good way but it depends on the first two components explaining a substantial share of the total variation.
All of these chi-squares are hugely significant. However, to compare the effects of one variable to another, I would use effect sizes, not chi-square itself and certainly not p.
@shabbychef you're welcome, glad I could help
What is $h$, $S$ and $L$?
The matrix $G$ is actually orthogonal, because $\\Sigma$ is a symmetric matrix.  This is the approach I was going to recommend - Basically amounts to rotating the $y_i$ vector and the model function $f(x_i,\\theta)$ so that the errors are independent, then applying OLS to each of the rotated components (I think).
$\\Phi^{-1}(0.75)^{-1} \\approx 1.4826$ - is this the value to use, or is one of the two inversions extra?
You can add or divide of multiply whatever you want, I still don't see the point. If this is your way of finding the grade - so be it. Another question is how good it describes things, but this is another problem...
The Elo rating system is named after Arpad Elo, and therefore isn't capitalized all the way through.
I've done a little bootstrapping from a textbook examples, so I'll try to extrapolate from there... If/when I do, I'll keep this thread posted.
@Rob, thanks for edits. The articles in English will remain a mystery for me :)
When skewness is extreme--as the quoted statistics attest--we have no assurance that the sampling distribution of the mean of 300 or even 3000 samples will be anywhere near Normal. *That* is why the OP is surprised. You counter that by saying you are not surprised, but that appears to come down to one person's intuition compared to another's. What *objective* argument can you supply *for these data* demonstrating that 300 (or 3000) is a large enough sample for the t-test to work well?
I upvoted this question because it's reasonable.  Well-posed questions that expose misunderstandings should be welcomed (and upvoted to indicate that), not disparaged with downvotes, because they can prompt great responses like the one @StasK has already offered here.
No, he's correct. R here is the correlation coefficient and `R^2` is, as its name implies the square of the correlation coefficient. It's also the share of the variation in one variable that is explained by the other.
To my mind it was already well-covered in comments at the post you linked to but I have put a more detailed answer with an additional reference there.
+1. This appears to be the same as another solution proposed slightly earlier.  The two replies together give a good answer.  I am upvoting this one because it uses a universal language--mathematical notation--developed for communicating such ideas to *people* rather than a narrow specialized language (R) intended for instructing computers.  The R version is *one* solution out of many possible (and therefore makes a nice *illustration*) whereas the mathematical version given here is a *general* solution which is readily translated to *any* computing platform.
We welcome questions like this, @ilona, but we treat them differently. Please tell us what you understand thus far, what you've tried & where you are stuck, & we'll try to provide hints to get you unstuck. To better understand the process, you should read the [wiki](http://stats.stackexchange.com/tags/self-study/info) for the `[self-study]` tag.
Thank you for the reply, although I feel you have not quite answered the question. I appreciate the differences in EFA and CFA and how they answer different questions, I am simply wondering which may be most appropriate in this context. Given you answer, I am inclined to think you are suggesting EFA.
Yup. This is what I needed. Credit also to @Stéphane Laurent for her reply.
I have no problem with this sort of Q&A here.  R isn't such a mainstream language (like Python or Java) that a quant would naturally say, "Oh this is a general programming question so I should go to StackOverflow or similar and ask this or look there for solutions".  Actually it is more a question for an R mailing list or group site.  To serve those budding analysts who want to learn R we should be glad to have an answer here as well.
What is stopping you from assigning cases to whatever categories you desire based on their probability?
Thanks @Glen_b. I'm find it difficult to understand the difference between the chi_squared test that you are describing to the one I chose. As I understood from browsing the web and R help is that the "Goosness of Fit" test is a version of chi squared and is being operated (in my code) by the command: `chisq.test(x=x,p=p,rescale.p=rescale.p)`, while x and p are my two counts. Isn't it the same?
Also, does it matter if I use p (public, n=227) as the main variable (x=p) and the experts (n=37) as my expected frequencies regarding the small size of my sample?
I don't understand your question (too much English for me, perhaps). Do you want to display the variance on a picture ?
I think you should use good old `lme` from package `nlme`, which also allows to model temporal autocorrelation of residuals. 6 time points make a short time series but you are still dealing with time series.
In general, there is nothing stopping you from specifying 3-way or 4-way or N-way interactions, except a general desire for model parsimony.  I, for one, would have a better time understanding what is going on with your question if you could give an equation representing your model, and a clearer exposition of the hypotheses that you want to test.  Another thing:  having lots of interaction terms can often lead to multicollinearity, which can make main effects on subsets difficult to interpret, require joint tests, and other exotic approaches to teasing out effects.
@AbhishekShivkumar : so i use PCA a lot in pre-processing prior to a supervised ML technique (e.g., SVM)--to reduce the number of features. In fact, PCA is often enough, by itself--remember, it gives you the axes of maximum variability, so not surprisingly, it is very useful as an 'unsupervised' classification technique. if you need an example, i can put one in my repo.
Book was first published in 1978, revised in 2005 and is currently in print.
Yep, that's about right. Thinking about it, my terminology might be a bit off. What you do is run an _unsupervised_ clustering algorithm (e.g. k-means) over all previously evaluated points in the _state_ (not state-action) space. You'll get out a bunch of clusters (k). Test the clusters for purity using the _action_ of each point as a label. Take your new point, drop it into the state space, and determine which cluster it belongs to. If the cluster is highly pure, and the new point's action matches the cluster label, don't bother with it. Otherwise, it's worth learning.
If you all need any additional information, please let me know!
please improve the question. I fixed numerous grammatical mistakes and latexified the equations. Even in the current form the question runs high risk of closure.
As an element of answer, I advice you the answer with links of Stephane Laurent in http://mathoverflow.net/questions/10971/why-isnt-likelihood-a-probability-density-function. Hope it helps.
You're a hero, thanks!
@PeterFlom: the singularities are cause by the setting of `x1`, `x2` and `x3`: for example all males are adults and the first six months, and all females are juveniles and the second six months.
@Cesar: it is very similar to bootstrap, see the expanded answer.
The multicollinearity is the problem only when the explanatory variables are highly correlated. One way to test this is to use variance inflation factor or condition index.
The answer for your title, assuming the elements are independent and identically distributed and that the probability of a tie is zero, is 1/i by symmetry. But the question you're asking is confusing and implies you would need to know the distribution if the random variables.
A few more additions/clarifications: 1) the **temperature** range of observed and predicted events; 5) actual observed failures and non-failures
It would maybe help if you briefly describe how the multiple records for each subject were created. Are these repated measures of the the same 'concept'? Are these measures ordered or is their order unsystematic? Is there a continuous time variable underlying the distance of measurement moments or can they be considered discrete time intervals? All of this information may help choosing a correct model for the outcome variable.
@Richard, thanks for your question. Unfortunately, the method you describe above will *not* produce entries that are normally distributed. The diagonals are 1 with probability one and the off-diagonals are bounded between $-1$ and $+1$. Now, the *rescaled* entries will converge asymptotically to a normal distribution centered around zero. Can you give us more information about the problem you're actually trying to solve? And, why do you want "normally distributed" off diagonals?
I don't see how those equations suggest $K_d$ is proportional to $1/\\alpha$. Similarly if $\\alpha=V/V_\\text{max}$ and both are random variables, you can't really treat $V_\\text{max}$ as a constant in computing expectations: for example $E(X/Y)\\neq E(X)/E(Y)$
(+1) the tags for this purely algebraic problem were misleading, resulting in false assumption the $p$ is observed (not latent) response variable, but not the notation for a linear form! I fought if the $p$ is observed, then it is possible to take the $t$ difference for it and transform into linear in unknowns system that has no solution (residuals? regression?). In algebraic case indeed it is exactly as you wrote (I felt I have to do so, but was too sleepy to proceed). Conclusion: I just should not look for econometrics in every post tagged by regression ^_^
Do you want to look at sphericity or a factor to correct F-values because of violations in sphericity?
Forget about the denominator since the term will not include Q and is thus part of the multiplicative integration constant that can be ignored in deriving the posterior.  Identify the posterior from the kernel of the numerator once you combine the terms involving Q from the IG and Normal distributions in the numerator.
Sounds interesting, I'll have to look into that!
I haven't time to check it, but your idea seems justifiable. I think that you may go on with this "weighting". You may multiply the proportions obtained by the total number of respondents, if you want to do the analysis strictly on counts.
There is nothing in Bayesian statistics that prevents you from creating an event $E = A \\cap B$ from two other events $A$ and $B$; indeed, being able to do so is pretty fundamental to statistics.  "Mutually exclusive and jointly exhaustive" reminds me of the Accenture paradigm for problem description "MECE" = mutually exclusive and collectively exhaustive - but that's not statistical in nature either.
"An extra peak" would be included in what I mean by "changing over time". Unchanging over time would mean "identical in shape from period to period". Anything else isn't unchanging.
@Developer : I'm a full-time statistical consultant, and I know very well what your question is about. I just fear you don't understand my answer. Without understanding of statistical concepts you shouldn't even start asking questions. If you understand the necessary statistical concepts, Google, SE, wikipedia, textbooks, ... all will do.
I don't use R. I assume model2 = intercept through the origin. Why would you do this? I'm sure there are situations where this might make sense, but I don't think they are commonly encountered. I think your reasoning sounds solid in that model1 makes sense.
I work in medical/epidemiological research, and I see this book on colleagues' shelves all the time.  Still haven't read it myself, but I can attest to its popularity.
I've never even thought of it pictorially before. Normally, thinking in pictures doesn't work for me, but I'll read that article and maybe this is a special case where it will help me.
I am not testing my classifier with the data I used for training. I am testing it with data it has been unseen by the classifier during the feature selection and training stages. That way I am avoiding overfitting. My question is about the implications of using the whole training set to select features instead of splitting the training set into folds and do feature selection on each fold. This is what I am doing right now and then I am voting for the best features on every fold. I need to clarify that my feature selection algorithm is a filter method.
This will not work for the reason explaine [here](http://stats.stackexchange.com/questions/46229/fast-linear-regression-robust-to-outliers/46234#46234)
I've added some extra information to clear things further up.