tttjjj/elt_bench huggingface dataset:
No, I am not. Here's the procedure step by step: (1) ANOVA (1a) Not reject? Done! *No* pairwise tests for difference are rejected at your level of $\\alpha$. (1b) Reject: proceed to step 2; (2) Conduct pairwise tests. If you have $k$ factors, then you may perform up to $\\frac{k\\left(k-1\\right)}{2}$ of them; let's call $m$ the number of pairwise tests. You perform $m$ unpaired *t* tests for differences in means in one factor level versus another; these *t* tests use *pooled* variance equal to the "within variance" from your $F$ statistic. (3) Adjust your $p$ values (e.g. using FDR).
So you want to plot $1-\\hat S(t)$ instead of $\\hat S(t)$ with c.i. but without censoring marks? (By $\\hat S$ I mean the Kaplan-Meier estimate.)
@DavidMarx. Thanks for your comment. No, I am a private teacher who caters for students (see here: http://en.wikipedia.org/wiki/Frontistirio what I mean - no job too small) and usually I receive students from that engineering school or from other schools, to solve their exams (not on the time of the exam, but afterwards) in order to prepare them for the next exam. But during the last 1-2 years there is this Professor who gives the most difficult exams ever, one of which is this one!!! And they are not even math students...
Hello! Thank you very much for your comment! Unfortunately (or not), $x$ and $y$ are not indepedent. I added this extra information in my original question above. For the general case for $a$ and $b$, however, I see it can't hold.
Awsesome, I've been looking for smth like that for a long time, good quality plot
What is "SmartPLS"? Is that some special software, or a specific package w/i some software?
What it looks like depends on the joint distribution of the two variables, something you can't infer from the full marginals, let alone what's effectively a 5-number summary of them (the boxplot).
Alright I happily did it! Any other additional or even controversial inputs are welcomed!
Because $\\chi^2(1)$ is the distribution of the square of a standard normal variate, your question is tantamount to asking for the probability that a standard normal value exceeds $\\sqrt{0.1185}\\approx0.34$ in size.  That's an elementary calculation (and I'm sure you have software or accurate tables to carry it out).
@Xi'an and gappy, please explain *why* this book can be recommended.  For whom is it suitable?  In what sense is it "best"?
@rolando2, Residuals are typically defined as observed - fitted, therefore negative residuals are over-predictions.  In a properly specified model with little explanatory power - I'm a social scientist so I see these all the time - there will be a strong positive relationship between residuals and the observed outcome values.  If this is a residuals vs. actual plot, then a trend from upper left to lower right, would be the signal of a badly mis-specified model which you initially worried about.
(1) Is "95% +/- 5%" just a way of saying "at least 90%"? (2) Have you already done those 658 tests or are you asking whether this is the right number to use? (3) What are the consequences of selecting an incorrect date? Is it possible that a slightly incorrect date wouldn't be so bad, but hugely incorrect ones would? (4) What would the consequence be if it turned out the software actually performed worse than anticipated *in aggregate* on the 60,000 files (such as correctly identifying dates for only 89% of them, say)?
@Noah, what is the particle filter? Some kind of physics experiment?
Regarding point 2, the `anova()` function in R does *not* compare the two models fitted under REML; it refits them using ML and then perform the test. See `lme4:::anova.merMod`, which contains the line `mods <- lapply(mods, refitML)`. (But you are still right that `anova()` can’t be used to compare the two models, as they are not nested.)
Somehow I overlooked the existence of the one-sample t-test. Seems like it's been too long since basic stats. I went with a one-sided test since it's actually more appropriate, but thanks for jumpstarting things. Makes sense now.
Where did you hear that z-scores are not valid for t-test? z or not z, the t statistics and p-value are the same.
Thanks. Also tried to fit some distributions by the method of L-moments and get thrown back "L-moments invalid"-messages. Does not seem to help a lot here unfortunately. What do you think about splitting the data-set in two? Like a log-normal and a gev for the tail? I have to say though, that this data-set is basically already the "tail" of some larger data set which has been filtered by some naive 3-sigma-algorithm. So ideally, I did want to model it by just one type of distribution.
Thank you for the link. I'll try to contact the author sometime in the future.
It's a little discouraging that any teacher would be so frankly ignorant of the applications of their field.
Indeed. The use of the term _scoring_ here is in the sense of credit scoring.
Assuming your model complies with the assumptions of LR, I don't see much to do here except adding new predictors.
Do you know how to proceed if it is not a softmax, but a usual hidden layer? Imagine that each unit on this layer gets inputs from all units of the previous layer (i.e. this layer is "fully connected"), which is normally the case. Then you also need to back-propagate the errors back through this layer, and the derivatives also form a Jacobian matrix. If you are confused about how to do it, then your confusion is unrelated to softmax.
I had forgotten how robust this test is due to the central limit theorem, and had dismissed it due to the odd shape and discontinuity of my distribution.  Thank you for your response and the reminder.
Related thread: [Difference between standard error and standard deviation](http://stats.stackexchange.com/q/32318/930).
A t-value (or any other test statistic) is mostly an intermediate step. It's basically some statistic that was proven, under some assumptions, to have a well-known distribution. Since we know the distribution of the test statistic under the null, we can then use standard tables (today mostly software) to derive a p-value.
Help us a bit: What related techniques do you understand? PCA? PLS regression? LDA? What are you looking for: Ideas why it is plausible? Interpretation of what is going on? Explanation of the maths?
Karel, please take a look: http://www.ime.usp.br/~pmarques/papers/redux.pdf
Thanx, that makes things very clear.
I was thinking along the lines that correlations in a distributions should decrease the uncertainty of a pair of values $(X,Y)$ and hence $H(X,Y)$ must be smaller that $H(X) + H(Y)$ .
Is this a homework problem?
A quick search got me to the random forest man page, where it says "oob_score : bool     Whether to use out-of-bag samples to estimate the generalization error" so this looks like an error measure to me. If this is true, your oob estimate is heavily overoptimistic - which would be an expected "symptom" of dependent rows.
You are very kind to go to all that trouble to help someone who asked a question. Thanks again. And you are right, the definitions are not too daunting after all.
...In other words, I believe you meant to ask "what is the minimum number of data points required for a kernel density estimation to be considered non-misleading/acceptable/adequate (something like that), based on accumulated experience up to now?" Remember it is only our words that connect us here - no personal history, no facial expressions, no body language, no "silently understood" implied meanings. For the benefit of all, we must try to be _very_ exact and precise in how we formulate our questions (and answers).
@BrianBorchers, sorry for my misunderstanding. The size of $X$ is less than $100 \\times 100$. So the total size of all elements of $X$ is less than $10000$. This problem is alike the linear discriminant analysis, and I use it for dimension reduction. The $k(X)$ is defined as a smoothness term. So I use the eigenvalue decomposition for solutions.
I think my last question belongs here. Let's say I identify my data distribution. Is it that I predict that future distributions will follow this probability ? I am missing data analysis part here. I know that a box-whisker plot easily shows quartiles which I understand. I don't get the utility of a distribution. May there are properties of this distribution I need to investigate for prediction.
thank you soo much for the answers. I am trying to wrap my head around this at the moment.
We try to avoid link-only answers (which are more of a comment). See this page of the [help](http://stats.stackexchange.com/help/how-to-answer). If you think a linked answer here completely answers a question, consider whether you should indicate that it is a duplicate.
(ctd)... There are also [limit theorems](http://en.wikipedia.org/wiki/Fisher%E2%80%93Tippett%E2%80%93Gnedenko_theorem) that may be useful if you have large sample sizes. To say much more you'd need to be more specific about what you mean by 'predict' and what information you have or can assume about the situation.
It seems you are just plugging a book and rehashing some points made before based on a secondary or tertiary source. This does not seem very useful. Could you at least provide the full reference for Bandalos and Boehm-Kaufman, 2009?
"Results" is vague here. Please give concrete details. Are you mostly focusing on whether key parameters agree, or are you as or more interested in whether values agree one-to-one?
@onestop, do you have a reference about that? I believe it, but I'm having trouble figuring out exactly why that would happen.
Check the answer to [this](http://stats.stackexchange.com/questions/58107/conditional-expectation-of-r-squared) question.
Using R's parameterization, $\\alpha = 1+\\theta$ where $\\theta$ is the moving average parameter.
I do not follow why statistical *significance* may be relevant to this problem, because significance (a very general concept in hypothesis testing) and strength of association are not the same thing.  Could you perhaps explain how you would measure "significance" in this context and how it would "build" over time?
+1 thanks for the very useful discussion
This qualifies as a routine (textbook-style) question. Please add the `self-study` tag, and since they're treated somewhat differently, please see the [tag-wiki info](http://stats.stackexchange.com/tags/self-study/info).
What do you mean by *"How exactly do statistical packages choose regression models (in particular ordinal regression)?"*? Are you asking how the models are fit?
+1 This is the key that keeps it all from being hand-waving: you don't know, but you try something and then look at how well it matches and in what way it mismatches your data.
It's a $t$ test, not a z-test.
Phenomenal explanation of data dredging.
Is there a reason why you think SVM is particularly suitable for this type of data? Are there some data samples you can post ?
I put approximate ranges but not necessarily symmetric. what confuses me is how to know if my model is correct and so the coefficients are? all  explanatory variables have positive linear relationship with the response`y` but some values of the coefficients are negative
It depends on the prerequisites for the class. I like the book [_Linear Models with R_  by Julian Faraway](http://www.amazon.com/Linear-Models-Chapman-Statistical-Science/dp/1584884258). This book is very applied and is self-contained and was my - and several of my peers' - introduction to linear models but it may a little tough for undergraduates with little background and comfort with probability/statistics.
By context I don't mean "application area" - I mean "within a particular piece of writing" - if a particular author uses a term one way or another, which will generally be clear from the surrounding text. As with "*when we say the correlation between 2 variables is 0.8, are referring to the value of the correlation coeff?*" -- yes, in that situation, 'correlation' is referring to a correlation coefficient, likely the Pearson one. If, on the other hand, someone said 'there's some correlation between experience and income' they may mean nothing more than some association exists -likely monotonic
I recently got around to adding an example to the [metafor package website](http://www.metafor-project.org/) that show [how to examine and conduct tests of models involving multiple factors and their interaction](http://www.metafor-project.org/doku.php/tips:multiple_factors_interactions).
You can also note that the waiting time distribution is, roughly speaking, an exponential distribution with a Gamma random rate parameter and strictly speaking this is a Beta distribution of the second kind, as for any Gamma distribution with a Gamma random rate parameter.
+1, however I would disagree that Bayesianism is necessarily subjective (I agree with Jayenes on that one) or that frequentism is completely objective (there are often assumptions that are equivalent to having a prior belief about the model, but they are not a formal part of the framework, but exist nevetheless).  Also I don't see why noninformative priors are non-Bayesian, they encode the prior knowledge that you know you don't know something.  As usual, this doesn't imply that Bayesianism is better than frequentism or vice versa, just as a flat screwdriver is not better than a pozidrive one.
@David It's legitimate to use AIC for *nested* model comparison (as in null - linear - quadratic - cubic), but not (AFAIK) for completely different models, such as "monod" versus "log".
Stock and Watson have a good paper on this: [Vector Autoregressions Journal of Economic Perspectives—Volume 15, Number 4—Fall 2001—Pages 101–115](http://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.15.4.101). Pressed for time. May answer at later stage.
@joran. Yah, by forcing predictions to be 0 or 1 you lose the ability to make subtle adjustments to predictions (between 0 and 1) that can lower error (e.g. mean squared prediction error). As such I suspect that approach to be inferior. I tried it and most attempts at building a decision tree fail to find even a single split that improves error.
You need to set `factorize = FALSE` in your call to `ez.glm` when including a numerical covariate (also, `type = 3` is the default).
@cardinal Maybe it is clear to you but then explain the sentence " You are really getting somewhere if the out-of-sample results are more than 50 percent of the in-sample". If you are saying that emsemble averaging across models can be effective then of course I can agree with that. Boosting has been demonstrated to work well in many applications. But I do not see where that comes out of Woodriff's remarks.
Look at the documentation for the `weights` argument of `lme`, which allows a call to a `varFunc` object and, in particular, `varIdent` which allows a different variance for each level of a factor. By the way, your `lme` call shows the `random` argument but you also need a `fixed` argument.
That is impressive.  But I think the answer one would look for on this site (as opposed to SO) would explain how, in general, one determines the equation of a spline.
If you are using R, you should check out the caret package: http://caret.r-forge.r-project.org/.  It basically automates the tuning process for a lot of common models in R.  The package's webpage and its documentation have walkthroughs.  But it is still more of the "how" rather than the intuition and art of model tuning.  Unfortunately I too am still trying to learn the latter, myself.  Could just be a lack of experience thing, in which case, fake it till you make it :-)
$\\log_{10}e^K=K\\log_{10}e$, it's basically the same as K, but much easier to interpret since it's in powers of 10
@joriki I updated the question with my thoughts on this.
Did you try drawing a Venn diagram? It sometimes help to visualize... and at the same time you should immediately see an algebraic relation you probably already know.
I'll mark this as the answer; there are several good suggestions in it so I'll think how to apply them.
The question should perhaps be migrated to http://dsp.stackexchange.com/
Nice list. Thanks a lot!
@GaetanLion You are welcome. I would caution you against delaying learning programming because you're only making it worse for yourself, though.
Many statistical software allow to perform a two-sample t-test where data is organized in two columns with either (a) scores of group 1 in 1st column, scores of group 2 in 2nd column, or (b) all scores in one column and the classification factor in the other. This should help: http://www.ams.sunysb.edu/~kye/ams102/minitab/twottest/twottest.html.
This can be achieve quite simply with the boot package.
In retrospect (now that I've answered), StackOverflow is probably the ideal place for this question.  Who better to answer than a bunch of programmers?
Thanks for the reply and recommendations.
It is great that you link to that CV question and, consequently, to the important comment thread (b/w rolando2, Frank Harrell, ...) questioning whether it is appropriate to pick the model post facto based on fit. Perhaps it is better to use prior knowledge of the system to pick the model.
The PDF question is answered at http://stats.stackexchange.com/questions/4220/a-probability-distribution-value-exceeding-1-is-ok/4223#4223.  The CDF question is the easy one, because a CDF by definition is a *probability*.
Your question is insufficiently clear. Can you be more explicit?
It sure is. I do have another suggestion, though, which you can take or leave as you wish. Many people seem to think that this measure-theoretic formulation and understanding of basic probability concepts is "sophisticated" or "abstract." As such, there is potentially a mismatch between the knowledge exhibited in the question and the knowledge required to understand the answer. You can help bridge that gap by quoting the definition of independence formulated in terms of sigma fields and showing how simple and understandable it is. That would make your answer more complete and relevant.
You don't need a method to identify a time series. A definition is that you have observations in sequence for each of which you know the time of observation. Taking a moving average won't establish either way whether data are a time series: you need to know how the data were recorded.
Interactions can be affected by the differences in the distributions of the respective variables. I'll try to find a source for this and post.
@mpitkas - there is a similar recent post I entered in the Quant Finance forum at SE, and wonder if you would care to comment on fitting an AR(1)/GARCH(1,1) process to log-returns? Otherwise, the above is a very helpful response for simulation.
can you please confirm that it worked?
Ah, I believe I get it, finally. Perhaps you should write your comments up as an answer.
@whuber Yes, I should've mentioned that. I realize that in theory, $Pr(Y = 0) > 0$, but in practice this isn't a major issue both because $Pr(Y=0)$ is small, and because I'm willing to assume a zero-truncated NB distribution. I'd be happy with looking at $\\mathbb{E}(X/(Y+1))$.
Your question feels very generic. Check the FAQ: http://stackoverflow.com/faq#questions for the kind of questions you can ask here. On your question, if I were you, I'd look at the results, try to detect patterns from failed cases, and add features to capture the information I'm missing. Its too hard to tell if SVM is a bad choice, given the info, but you should be able to better. Choose a wider (possibly less efficient) feature space now, get the numbers and work on reduction later
Could you clarify what you mean by a "common entity". It's not clear to me because it seems you are describing two features (for me, a feature or attribute = a variable) with different units (meters and seconds) rather than a single one?
I will take a look at GMMS. What I meant by mainstream is frequently used, so I will find an existing library in my programming language of choice that contains the formulae etc etc.
I don't fully understand what you're doing.  Certainly you can make histograms and calculate the percentage of your sample that's within each of your bins, whether those bins are equal-interval or not.  Do you have the raw data?  Did you somehow come across a pre-binned dataset, and are wondering what those data would look like if they had been binned differently?
Using control charts is an interesting idea.  It seems that it might be difficult to overcome the challenges outlined in comments to the question, however.  In the non-stationary case if you are "forgetting" older values it seems possible the estimates could be highly biased. For instance, how would your suggestions perform on a stream of data given by $x_t=\\cos(\\pi t/10^6)+2\\lfloor t/10^6\\rfloor$?  (This drops very gradually, suddenly jumps and rises very gradually, suddenly jumps again, and so on.)
I agree totally with you. The density in the corners are not correct.
This is commonly referred to as _class_ _imbalance_. I've answered questions on this topic [here](http://stats.stackexchange.com/questions/67893/equal-number-of-training-instances-of-each-classification-label/67917#67917) and [here](http://stats.stackexchange.com/questions/64163/how-to-deal-with-low-frequency-examples-in-classification/64165#64165) and I'm sure there are others also.
@Ashok +1 for mentioning alternative measures of discrepancy. On the other hand the I did not find ready available formula to use on matrices in the link you gave. Also it does not say why I-divergence is the "best" measure of discrepancy. Could you please elaborate more on these points?
@Moom It's fine for the site.  Unfortunately it's rather broad as stated, because you don't offer details about the characteristics of your data.  Thus it's really a general question about whether it can make sense to re-express data (yes) and how to go about doing it (it depends on many things).  Look at [tag:data-transformation] for examples and guidance.
(cont'd) - Even if it is just "some" samples are different, this is still contained in the hypothesis I have given.  If any of them are true (in addition to a number of other hypothesis), then we can say "some" samples are different.  And I personally would be not satisfied with simply saying that some samples are different; once you learn that they are different, surely you should then test which ones are different.
See the original LOF publication, it gives a precise definition of k-distance and kNN, where other papers are much sloppier.
I took a quick glance at the article, there is no mention of bilinear regression in it. Please clarify your question.
Failure to reject normality doesn't mean the data is normal -  moreover in a tiny sample size of 20, you may not even have reasonably approximate normality in spite of a failure to reject.
Or alternatively, we can use `predict()`, with `type="response"` to get individual probabilities (instead of log-odds values). I don't think you want to include an error term when computing $\\hat\\pi(\\beta'x_i)$.
Without some sample size information, I don't see how much can be done. If the proprtions are rounded fractions, you might be able to compute lower bounds on the possible sample sizes based on the values the proportions take, but those lower bounds aren't likely to be very helpful (in the sense that fairly low sample sizes are likely to be consistent with the figures, implying you won't be able to tell any differences from random variation).
I am not familiar with what is available in R but I think there are ways to search for such routinesusing the R platform.
The [no free lunch theorem](http://en.wikipedia.org/wiki/No_free_lunch_theorem) might apply here? At least in terms of predictive power. The lasso penalty has the benefit of inducing sparseness if you're into that.
That warning is very important. Note that both the coefficients and their SE are 0.000. This is a strong indication of a units problem.
@FrankHarrell Do these results apply to the penalized likelihood as well? Does the penalized likelihood have any properties that make it differ from the likelihood with respect to this variable-importance  measure? Could you suggest any papers that describe this in greater detail? Thanks.
I'm running Windows 7 64-bit on Quad CPU 2.67 GHz, 4GB RAM
This interpretation is correct.