tttjjj/elt_bench huggingface dataset:
yes you can do that, try it test it. in machine learning there is no need to justify, you can cross validate and test you model, if adding cluster variable means improved accuracy, use it by all means.
@user1471980: There is no "best model for time series data". You need to apply domain knowledge to come up with an actual model: how is the data generated, what kind of relationships might there be between time 1 and time 2? What other data might you have (number of users, number of processes, time zone of users, types of processes, etc, etc) that might feed into a model to give you better answers? If you can do that, the technique will be more obvious. As Tom Reilly says, the short bit of data you supplied looks pretty random so it's not clear you can do any better than going with an average.
@Glen_b do you know of any reference for my problem? Does your comment with the error terms still apply. Or did you mean the distribution of the error terms?
great advice, will investigate this approach
I was looking for code, not linear algebra formulas :D

anyways I've found `candisc` R package which does the calculation
Yes, and that can happen. I would include the current month balance, min, max, avg balance etc as other predictors.
So there are two different approach to clustering. First one based on time-series - sales during 26 weeks and I clustered data based on dynamic time warping. Second approach was to cluster growth curve parameters, also based on dynamic time warping. I used `clusGap`based on globalmax, I didn't know how to implement maxSE.
This is a gold mine - thanks Bill!
using logs sometimes helps. Log-log models are popular in econometrics, because they are easy to interpret, that is another bonus of using them.
Not sure exactly what the question is: are you wanting an intuition or an equation, or further algorithm description? Intuition-wise, the SOM, in your example, is a 2D manifold embedded in a 6D problem space, and the SOM training method smooths out the manifold in problem space in such a way that it's weighted by the training data, so that the density of the points on the manifold in problem space reflect the density of the training samples in problem space. SOM (2D) neighbors will be relatively close in problem (6D) space.
Also, this can be extended to using a finite number of bins for real-time medians, etc.
"ECOG, ecog.org/general/perf_stat.html, is much how my ratings are given, yes"

So is it ECOG or not? What are your scales?
I agree that we have to teach both. If we were starting from scratch, and knew what we know now, we would never allow "normal" to emerge, because (1) the term is overloaded any way (2) the normal (Gaussian) is not normal (usual or expected) of data. "Gaussian" is the most common alternative, even though there is a history before Gauss. E.T. Jaynes suggested "central", which is a droll idea, but it hasn't caught on. (I recognise the arguments that are based on the central limit theorem.)
Also $X$ is usually $n\\times k$ matrix, with $n$ being number of observations, $k$ being the number of right hand side variables including the constant. In your case $k=2$. The matrix $(X'X)^{-1}$ is then usualy $k\\times k$ matrix, in your case $2\\times 2$.
Are you doing _linear_ regression?
@Thomas-S-E Did you check out the paper that I posted? It adresses many of those issues... On page 396 (section "Synthesizing Partial Effects") it says "[...] when analyzing a collection of partial-effect sizes, the reviewer needs to include predictor variables that reflect differences in model complexity that are likely to influence the sizes of the partial correlation". So, I would run meta-regressions that control for the most important predictors in your field. If you find my answer helpful, please consider upvoting it.
Thanks everybody for the answers! And thanks Stijn...that's all I needed to know :)
@Glen_b Well, "Preliminary answers" have the non-continuity correction version as the answer but we have a few days to email them and tell them why they should accept others.  The exam has several mistakes in it, which is frustrating.
@Mark The two tests give you the same information in terms of what you would conclude regarding the effectiveness of method 1 vs method 2 but the way they approach the issue is different. Any particular reason to avoid the chi-square? Feel free to choose either one.
I included an example. Let me know if you need anything else. Thanks @BenBolker!
Thanks for the response. Reviewing hedonic regression now - I have never heard of it.

Your third item on the list, is what I was illuding too when I mentioned segmenting the population first and then modeling from there. I understand this tactic, but do not think it is the best approach and feel that there must be better ways.
Perhaps I am misreading it, but I don't think this answers the question (at least, head on). This only appears to address the *marginal* density of $u$, not anything related to a joint distribution (or density).
If I generalize your question a bit more, you are asking what's the difference between parametric and non-parametric tests and can a non-parametric test be used for samples that follow a _known distribution_? The answer is _yes_, but non-parametric tests suffer from [low power](http://en.wikipedia.org/wiki/Non-parametric_test#Applications_and_purpose).
Do you have thoughts on `liblinear` for this kind of problem?
Thank you very much for the help. Saved lots of time
Could you explain this a bit further? I didn't understand very well.
@Peter Flom Fair point. When I say "No more distorted" I mean that the distribution of the variable is no less uniform in layer X than it was in the previous layer, which I'll call X+1. So for some hypothetical "uniformness score" where X+1 = 7, the score for X ~ 7.
After "we can say by inspection" you appear to take all the $X_i$ to be identically distributed and independent. The point of this question is that the $X_i$ are *not* identically distributed. (The OP appears to assume they are independent, though.)
I'd say you need statistics every time you report the performance of your model. Mabe that is because my profession is analytical chemistry, where one of the important rules is "a number without confidence interval is no result".
That would explain why the packages don't implement standard errors.
thanks for the comment....Functional Regression seems to be a good choice .
No, ARIMA modeling is only a part of time series analysis. I was just pointing out that ACF and PACF weren't statistical tests.
Are the issing values perhaps called '. ' (trailing space) or some other syntactic horror? btw you can also hand in an array of possible missing value symbols to `read.csv` with the `na.strings` parameter.
@tdc Text data, sorry
Your question is unclear at the moment - perhaps you could edit it to make it clearer?
Noting also that p=0.05 is a somewhat arbitrary threshold. If you think that 1:20 is too high a chance of a false positive, then your p should be lower.
I think the DSE package estimates ARMAX models in R.
Asked and answered at http://stats.stackexchange.com/questions/1444/how-should-i-transform-non-negative-data-including-zeros/1795#1795.  But the two questions *look* so different it makes sense to let them both stand, with these links between them.
@Donbea: can you give a bibliographically  correct reference? (scientific articles often have authors, date of publications and sometimes even journals)
Yes, that's right @ttnphns - thanks for pointing that out - I've modified my answer a bit to reflect this.
@baptiste: Have a good conference. Do you mind to tell what kind of processes you have? I.e. can you assume that "within" each process the spectral characteristics are the same, or can the oscillations "move" over the spectrum (frequency is ambiguous if you have an oscillation pattern in a *spectrum*)?
Yes, this is indeed the way to go. in fact it's a little easier to get estimates of the high quantiles, especially if you're willing to tolerate error in the rank of the form $\\epsilon n$ where $n$ is the total number of items, and \\epsilon > 0$ is some user defined error term
I don't understand how small sample size, _by itself_, necessarily limits statistical inferences.  Sometimes the effect size is so huge that you only need enough samples to estimate variability and to check any model assumptions.
Incorrect, you divide by the number of observations, no matter how strange that sounds.  To the original question, it is strange that a reviewer would not be aware of something that's been around for 20 years for the Cox model.
+1 Thanks for sharing this advice. I think you nailed it with the problem being the link between what we can teach about **what the sampling distribution is** vs. how that can be extrapolated to inference from a single sample. As you (and others here) have suggested, continuing to reiterate the concept over and over throughout the course is important, but not often done (at least not by me, probably because I already find it hard enough to fit in everything I want, let alone returning to concepts already covered!).
Yes Fisher's test is greatly overused in my humble opinion.  But no tests should have a role in most modeling problems, and neither is AIC.  AIC is best for comparing two or three _pre-specified_ models.
Unfortunately not relevant, that post considers the weighted sum of Gamma random variables where the weights are strictly positive. In my case the weights would be +1 and -1 respectively.
+1 The entire second half can be replaced by the simple observation that the objective function $\\beta\\to\\frac{1}{2}\\beta^2+(\\pm\\gamma-\\hat{\\beta})\\beta$ is a union of parts of two convex parabolas with vertices at $\\pm\\gamma-\\hat{\\beta}$, where the negative sign is taken for $\\beta\\lt 0$ and the positive otherwise.  The formula is just a fancy way of choosing the lower vertex.
In my case, I prefer pairwise comparison using lsmeans as it gives me all pairs of contrasts including p values. If I use lmerTest I will have to run model six times with different baselines to see all pairs of contrasts.
I haven't used it, but for alternating logistic regression, a quick search turned up this ALR package:  http://www.hsph.harvard.edu/carey/vcwww_4.html
@vinux: It is more of a computational question in that I am looking for the smallest power $d$ such that the answer can be produced in a time $\\text{O}(p^d)$...
You can estimate the baseline hazard nonparametrically in the context of the Cox model, although noisily.  Another way to translate the PH assumption is that survival curves for any two individuals are connected by a power transformation, i.e., to get from $S_{1}(t)$ to $S_{2}(t)$ you exponentiate the first by the hazard ratio for the two.
(+1) I'm not sure I would call maximization of the log-likelihood in logistic regression a special case of IRLS. Certainly, though, IRLS can be used as a method for performing the maximization. Logically, it seems, the maximization of the function and the *algorithms* for achieving this are logically different concepts.
I don't think the choice between using the logit & probit link has much to do w/ whether the probabilities are close to .5. I have written about the choice of link here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909). I have heard people suggest the `cloglog` when the response categories are unbalanced, but other options exist.
Do you actually have access to raw data, or are you trying to make this inference just based on the given figure?
@nsc10 Where exactly did you get stuck ? One can surely write down a complete solution, but it would be more helpful (especially for you regarding the learning experience) if you would tell use what you have already did. For example, do you already know how k-NN and leave-one-out-crossvalidation works ?
I mean it's included if an intercept is fitted of course. The only parameter not in the vector is the error variance.
Keep in mind you're switching from thinking about $y_{t}$ as dependent on shocks to thinking about it being dependent on its past values. As an MA, $y_{t-1}$ depends on the $t-1$ shock and the $t-2$ shock. Hence, if you regress $y_{t}$ on $y_{t-1}$, then you're picking up part of the $t-1$ shock. However, you can't quite identify the whole shock because in that regression you only observe $y_{t-1}$, not the shocks. Adding more terms helps you identify the shocks.
This assumes a normal distribution for the differences and would only allow inference about the intervention (training workshop) if (1) there is no regression to the mean and (2) nothing else is changing at the same time (e.g., subjects getting better at being tested or subjects tiring).
Ah, and I wanted the confidence intervals of mean of every response (malformations).
You need a better title. I edited out your underscores, which appeared merely a way to pad up to 15 characters, but you need a better title to attract interest. Beyond that, it appears that your independent variables are  categorical, but the pivot of any answer is precisely what 1,2,3,4,5 are for D. Are you counting something? Are they ordinal (graded) or something else? In general, it is a mistake when confused to abstract out the details of what you are doing. Statistical people are more likely to be able to diagnose your situation if given vivid, concrete, specific information.
In my case a real-time response is desirable but it's not strictly necessary. I'm definitely not above accepting data over email and running it myself, but somehow that doesn't seem like the "right" answer.
I have added an explanation on how to use the formula to get a standard error, which in fact would be very close to just the standard error of the three within-day means.
@user154510 No. Survival analysis does not make that assumption.
I changed the original code to make it more clear that X is not fixed.
No sir, this one is of my personal interest. I never had studied time series before.
The above method uses the patterns of "how the yearly total has been reached in previous years" (e.g from monthly data) as an estimation of current path. The method adjust the estimated total as data come in (at any sub time period). Essentially It has an intra year path with an estimated percentage over total at any sub time period.
I think this answer would be improved by providing some hints on how to proceed. The answer makes very broad and definitive statements (many of which I generally agree with) without reference to resources that would buttress the claims. Certainly penalization is not a panacea, either, and there are many choices to be made if one goes down that road.
The question title mentions Fisher information, but the actual question doesn't. If your question really is about Fisher information, could you please clarify how?
Welcome to the site, @user34658. I took the liberty of marking up your post with $\\LaTeX$ for easier readability. Please make sure it still says what you want it to.
Yes it is sufficient. And yes it follows from the chebyshev's lemma: $P(|X-EX|>t)\\le \\frac{Var(X)}{t^2}$
see also http://glmm.wikidot.com/faq#rsquared
Actually, is this answer redundant with @babelproofreader's?
Where's the penalty for  bigger models if you just look at deviance?
I suspect people (including myself) may be reluctant to jump in because this is a broad question that needs complex, specialized treatment, as @Michael's reply suggests.  Certainly "multiple regression" in some form will be involved in the analysis.  However--I mean no offense, I'm trying to help--anyone who has to ask that is not going to be up to the task of conducting the analysis itself.  You need to seek out the services of an expert who can spend some time learning about your objectives and assessing the nature and availability of data.
If you follow the approach in the answer, you should compute the covariance matrix: each diagonal element is worth p*(p-1)/N, where N = 4 for x and N = 5 for y.
Sometimes it helps to think up ways of partitioning the set of all outcomes into disjoint sets. Maybe the outcomes where the 2 comes before the 1 and vice versa?
@venkasub, you might also look at Muller's method for root-finding. It's simpler than Brent's method and "almost" as efficient as Newton's method, but without needing to know the derivatives of the target function.
Thank you,  I thought that it could be something to do with 'more power' for the z test because it was using N of both samples.  Is this correct?  Can i adjust my individual CIs to account for this?  So that they are more representative of the CI of the difference?
Thank you! Finally I got the book you suggest. It's really helpful.
I guess I would try some symbolic math tool, like mathematica or Matlab with symbolic toolbox. Again, I don't really know how to do it numerically.
Using the data to build the prior cannot take place within the Bayesian paradigm. So it does not make sense from a Bayesian perspective and the usual validation of Bayesian procedures does not apply. The resulting inference may be perfectly valid but one has to demonstrate it from first principles. (Richardson and Green use what is called empirical Bayes. Which is not a Bayesian procedure.)
@Mark I don't think it does. He references what I am basically saying under the title "Bayesian experiment design" (see my 3rd sentence).  
Beyond this, the point made on the Evan Miller page is valid, as far as it goes, but I dislike his recommendation because it's a form of willful ignorance.
At **any** value of N you are accepting some risk that given more n the answer would change.  Since the "bias" he describes can be quantified, I find it difficult to endorse an artificial straightjacket as a solution.
Thanks!  I guess then my natural follow-up question, one that probably shows my lack of experience, is about how to determine what the right distribution to model a given problem is.  I think I will need to do a bit more reading to make sure I can always choose correctly.
(1) Don't you mean *independent* quantities? (2) If so, a polynomial of degree 49 will get you perfect correlation :-). (3) What do you mean by "statistically better"? (4) Please review our threads on [model selection](http://stats.stackexchange.com/search?tab=votes&q=model%20selection) before modifying this question further.
Maximum-likelihood estimates cannot be calculated in case of quasi-complete separation (only 0 in control, almost only 1 in test groups). This is confirmed by the huge standard errors.
When the columns of $X$ are standardized, $XX^\\intercal$ contains the *sums* of products and cross-products.  By the very *definition* of (sample) correlation, its off-diagonal entries are the correlations among the columns of the original design matrix.
I appreciate your thoughts on why such a dynamic would occur and how to prevent it.  But I still am left with my question:  "...theoretically how much of a spurious tendency would there be...?"
With $15000$ data points and $700$ parameters,  there is little point using the test/training split.  The in-sample and out-of-sample measures of accuracy will be very similar.
yes, I would be quite worried about the model with this Q-Q plot.  You might be able to use something like `weights=varPower(form=~pos.centered*I(pos.centered>0))` ... ??
No, this is not correct at all. Ignoring the spline aspect there are 5 terms in the model, the 4 variables named **plus** the model intercept or constant term. However, as this is a GAM with cubic splines, the actual number of terms in the model is a lot more because each spline is determined by a set of $k$ basis functions.
How did you obtain the 2.84? Shouldn't it should depend on N via the degrees of freedom for the t distribution?
Maybe look at auto.arima in the 'forecast' package?
Next time, please just use a comment to ask clarification about some answer.
Fair enough on M-code, but that was only part of my suggestion though. :) And it was the least important one actually. Providing self-contained answers is important, as there is link-rot; if the link you provide goes bad your answers just lost all its background.
@JEquihua Ok! Do I understand correctly if I say: `Observations of Out Of Bag data for each single tree is "run" down that specific tree and falls in specific different "bins", aka terminal nodes. Then we first: 1. average the data in each single terminal node first. And then: 2. We average the averages of each single terminal node over the single tree. And then: 3. Take the average from each single tree from step 2 and average over the forest? ` Thanks for your help!
I'm missing something. If we are predicting "the average number of days in a year" -- why would this necessarily be seasonal? Annual sunspot numbers are seasonal because that's an 11 year cycle, but it's not obvious to me that a seasonal component would be necessary here.
The statistical content of this question is not very clear.
@sesqu Standard deviations did not become commonplace until Gauss in 1809 derived his eponymous deviation using squared error, rather than absolute error, as a starting point.  However, what pushed them over the top (I believe) was Galton's regression theory (at which you hint) and the ability of ANOVA to decompose sums of squares--which amounts to a restatement of the Pythagorean Theorem, a relationship enjoyed only by the L2 norm.  Thus the SD became a natural omnibus measure of spread advocated in Fisher's 1925 "Statistical Methods for Research Workers" and here we are, 85 years later.
The appearance of this factor for *general* transformations of random variables is explained at http://stats.stackexchange.com/questions/14483. Reading that might help you appreciate, too, that "$P(w\\in W)$" makes no sense. ($W$ is a random variable, not a set; and if you replace "$\\in$" by "$=$," which does make sense, you will obtain $P(w=W)=0$: see http://stats.stackexchange.com/questions/99935.) It suggests you are thinking of the PDF in the same way one ought to think of the *cumulative* distribution function.  The lognormal CDF has no extra factor in it, exactly as you might suppose.
Btw. is there any simple solution for kernel density estimation contours? Because if I want to be more strict, my data looks like:https://github.com/matejuh/doschecker_wiki_images/raw/master/linear_regression/kernel/kernelPoints.png resp. https://github.com/matejuh/doschecker_wiki_images/raw/master/linear_regression/kernel/kernel.png
If your dependent variable has only 2 levels you want regular logistic regression. If there is also a third level (probably a 0 for no participation) then ordinal logistic is certainly a good starting place.
(+1) About your 2nd edit, I would add that model-based variance estimators will work better with a small number of clusters (or we can use a Jacknife estimator). As for a reference, I always point to http://gbi.agrsci.dk/statistics/courses/phd07/material/Day10/, which contains very nice lectures notes (stat. background, including a comparison of GEE vs. GLMM approaches + illustrations in R).
@PeterFlom 	
Not concerned with the second question, I'm writing my own software. I simply want to know if it's a legitimate thing to do. If it reads as "is there software that does this" then please let me know where I can improve the question to remove that impression, as that is a shopping list question/recommendation question and not a good fit for any of the Stack Exchange sites (it's closed as "Not Constructive").
The data already give you the peak hours. What exactly are you looking for that you don't already have?
Assumption of uniformity is where it comes from, so yes, that's all it is; in the absence of other information (and sometimes even in the presence of it) it's a common assumption in calculating exposure to risk. As for the bit about diagnosis, I assumed the data is on diagnosis, not incidence, because we don't observe undiagnosed incidence - whatever is analyzed is what the assumption would need to apply to.
Look into methods that do not require dimensionality reduction or PCA. Given that each pixel represents essentially a comparable thing, I'm not convinced that Mahalanobis will provide a substantial benefit over plain Euclidean, actually.