tttjjj/elt_bench huggingface dataset:
@IrishStat It may help to know the downvote was not mine. I'm sorry the reason was not explained. As for me, I'm coming closer to an answer through your help and some face-to-face discussion with a colleague. Thank you for the update you added to the answer: that is quite helpful. I've also found issues in the model that I know how to fix, so I'm working on that.
Funnily, I wrote my master's thesis using cox regression, but I never thaught that I could use it on this dataset - the really like the idea, but the initial results are so clear cut, that it would probably be statistics for the sake of statistics.
@whuber and COOLSerdash Thanks for your explanation and solution! It clarified my doubt. I prefer to do such questions by writing cdf(in case of continuous function) for new random variable and then calculate pdf.
@JuneKang  I haven't seen something like this. I'm off campus so can't access article. But as someone mentions in your other question it sound a lot like what the -segmented- package does. For that package you have to provide the number of cut-points. If you don't have a preset number this can sort of be dealt with by running a number of cut-points(1-5) and comparing AIC and choosing best model.
It's actually a homework question for a stats class.  The professor doesn't really teach anything and just advertises for his research lab for an hour twice a week.
@mbq - I am very interested in your last sentence *"probably do some multiple comparison test, probably non-parametric"*. Would you mind elaborating on that?
In that case you shouldn't insist the results lie between 0 and 100%. Your verbal description translates to subtracting the total daily target from the total parts made during the day and expressing that difference--which may be negative or positive and even greater than 100%--as a percentage of the total target.
I don't think that ANCOVA is the tool when you've used matching.  It's worth examining whether matching is the best strategy.  Have you discarded any data?  Is the matching algorithm independent of the order of presentation of the observations?  Is the matching fine enough to avoid residual confounding?  There are several reasons to entertain covariate adjustment for propensity score instead (e.g., by including in the model a restricted cubic spline function of the logit of propensity).
@whuber I have heard that Gosset originally used $z$ for his statistics, what we now generally refer to as $t$, so your favorite text probably stays with that notation.  The books that I have used and taught from (though a small subset of all available) all use $z$ for known $\\sigma$ and $t$ for sample standard deviation.
@Anony-Mousse I want to have some confidence about how probable the instance x is of a class y. I want to use this probability to let the learning algorithm be able to actively queries the data-points about which it is least certain how to label (i.e. ask for there labels). This is often strainghtforward for probabilistic learning models. For example when using a probabilistic model for binary classification, it will simply queries the instances whose posterior probability of being positive is nearest 0.5. Now how to define P(y|x) when the model is not probabilistic (e.g. with the online KM) ?
And, additionally, that if you train a regression model on a set of similar students, that model will be more accurate in its predictions of those students' success than a model which was trained using a broader set of students.
Freedman, not Friedman.
Regarding unbalanced explanatory variables, you just lose power; see my answer [here](http://stats.stackexchange.com/a/31330/7290).
I think ARIFMA (fractional integration) might suit your needs then. That type of model is meant to handle long memory in the conditional mean. No need for the snark, but you weren't particularly clear in your question.
@Itronneberg Thanks for that explanation. I will go with logistic.
An `R` comment:  `y <- rep(as.vector(t(human)), each=3); c<- rep(compound, each=9); x ...` would give you the same stuff much faster and with less visual clutter. Additionally: you seem to discard information about the fact you have replicates. You sure you want to do that?
Mixed effects ordinal logistic regression is also available in R with the ordinal package and clmm().
+1, nice answer. I numbered the things the ratio tells you--I think it looks a little cleaner this way. If you don't like it, feel free to roll it back w/ my apologies.
+1, I would add "for estimating" *and testing* "the slope coefficients", but the point about it being included in the $R^2$ is a good one.
+1 Thanks, this seems like THE approach. What I was also thinking was to use some sort of tree based segmentation (CART etc.) on nominal variables and then performing a regression analysis on each segment. What do you think?
BTW, kernels are not really essential for SVM's. The "heart" of SVM is the principle of soft margin maximization. Going to kernel representation makes your problem dimensionality O(m^2) instead of O(d) where m is the number of examples and d is the dimension of your feature space, so if m^2 is more than d, you might be better off doing away with kernels
http://jmlr.csail.mit.edu/papers/v6/keerthi05a.html
+1 Nice exposition of the issues hinted at in the comments following [Srikant's answer](http://stats.stackexchange.com/questions/3559/logistic-regression-which-pseudo-r-squared-measure-is-the-one-to-report-cox/3560#3560).
Likelihood is what this question needs in the answer, not probability, either frequentist or Bayesian. Likelihood provides exactly the answer, the others can only do so with some twisting and stretching.
Thank you very much, Samuels!
@vonjd: But you could have downloaded, installed, and run your script in less time than it took to write your question and wait for an answer. ;-)
Thank you very much for your reply :)
What specifically is the matter with the p-values?
@Cam.Davidson.Pilon The [`runjags`](http://cran.r-project.org/web/packages/runjags/vignettes/runjags.pdf) vignette states the PDF of the generalized pareto distribution as:$$\\frac{1}{\\sigma}\\left(1 + \\xi\\left(\\frac{x-\\mu}{\\sigma}\\right)\\right)^{-\\left(\\frac{1}{\\xi}+1\\right)}$$ with $\\sigma>0$ and states that the lower bound is $\\mu$.
Since $\\lim_{x \\downarrow 0} x \\log x = 0$, it is customary to extend the function $f(x) = x \\log x$ to be defined on $[0,\\infty)$ and set $f(0) = 0$. Hence, the deviance would be zero.
Note that for option 2 you should set `REML = FALSE`, i.e., optimize the likelihood for the comparisons and only for parameter estimation use the default `REML = TRUE`. FWIW, I usually use option 2 (but I'm sure others have a more fundamental understanding of mixed effects models).
Yes, their paper [Label-noise Robust Logistic Regression and its Applications](http://www.cs.bham.ac.uk/~jxb008/papers/ecml2012.pdf) has an explicit label-flipping model as I suggested.
1. If m=2, than for your case you have $g_2=\\frac{m-1}{m}[g_2-\\frac{1}{m}\\sum_{i=1}^{m}g_i]=\\frac{1}{2}[5x-1 - \\frac{1}{2}(5x-1+3x+4)] = \\frac{1}{2}[5x-1-\\frac{1}{2}(8x+3)]=\\frac{1}{4}[2x-7]$1.

2. I made a mistake, instead of $\\hat{f}(x,\\beta)$ I typed without hat. I will correct that. 

Agree with all the other things.
@Lenwood If you absolutely must adjust the sample size as you go, it might be better to consider [accuracy in parameter estimation](http://stats.stackexchange.com/questions/16985/how-to-report-general-precision-in-estimating-correlations-within-a-context-of-j#30287) rather than power and repeated tests, [which can be very misleading if done improperly](http://allendowney.blogspot.nl/2011/10/repeated-tests-how-bad-can-it-be.html).
I just managed to get the generalization, using nested Lapply functions. But the idea was exactly as @Stat mentioned. Thanks
Could you please explain how this *multivariate* approach you reference should be applied to this *univariate* problem?
And when I mention measurable, you would say ... define the metastability! So I will leave it for others to provide a rigorous proof ;)
Thanks for the comment - but I think I'm looking for a bit more detail. I've edited my question, hopefully it'll make it clearer..?
Related: http://dilbert.com/strips/comic/2008-05-08/
1. See the second paragraph [here](http://en.wikipedia.org/wiki/Gumbel_distribution).
...and by $X_1 + X_2$ I mean the model that incorporates both predictors, not their actual sum. A better notation would probably be via the direct sum $X_1 \\oplus X_2$.
Great! I look forward to it!
@ttnphns, in fact I find the word "loading" only confusing, so I would rather refrain from using it whenever possible (hence I was happy to make the edit). My background is math and machine learning, and the word "loading" is not much used there, it comes more from the psychology literature on factor analysis.
I can see I may have made a misstep there. Would it be broadly correct to say that having a high correlation value for the two residuals means that the people with a high residual for one speed (for whatever reason, maybe height or whatever other latent variable) will tend to have a high residual for the other speed, and the people with a low residual for one speed will tend to have a low residual for the other. And before allowing the residuals to correlate we need to ask ourselves whether there's some good theoretical reason why this should be the case.
I added the `self-study` tag to your question, since this is a homework/self-study question. See http://stats.stackexchange.com/tags/self-study/info
+1 for the important point of performing the model/feature selection independently in each fold of the performance evaluation procedure.  If you don't do that correctly, the differences between the three resampling approaches are irrelevant!
@SideshowBob Your question still says "some of my $x$ are zero". If that is not so, please correct it.
but the number of clusters is not known
@cardinal thanks for the correction.
can you describe in a bit more detail how this can be accomplished?
http://www2.sas.com/proceedings/sugi30/203-30.pdf
Dear Nick, It was intended to be comment. thanks for converting it and placing it in the right place.
My thinking was that there may be differences between infected and uninfected plants within one level of damage but not the other. That's why i figured post-hoc tests would be appropriate as they would allow the to see those differences immediately (it's hard to see by plotting because the scale of the response variable differs greatly across the levels of damage). I'm really just looking for a way to do post-hoc tests on interaction when there are multiple covariates present. If at all possible of course. Thanks for the reference also :)
As @Peter Flom has noticed, if you want to model the time of leaving, your problem would be a survival modeling. On the other hand, if a logistic regression is what you want to do, your responses are (dynamically) correlated, as I grasped in new edited description, and you have to use a dynamic logistic regression.
Thanks for your answer.. The warning message returned by the model fitted by auto.arima how can the model be not suitable for the data?? In such cases, how to find the best fitted model?
Back when this was a \\$3.95 and then a \\$4.95 paperback, I bought copies by the dozen and gave them away to friends, clients, and anyone else who might be interested.
...but this way is very bad in the sense that the resultant euclidean data X produce euclidean distances D_new which are _very_ far from original dissimilarities D. So, I wouldn't recommend your step 5. It seems much better simply to set negative eigenvalues to 0 and skip to step 7. Or, slightly more fine approach: set negative eigenvalues to 0, rescale positive eigenvalues so that they sum be original (=trace(S)), and then skip to step 7. That's how it appears to me.
Excuse my non exact description of the problem. To use your words: 1.) You have sample X. 2.) Each x (element) in X (set) is a vector (of features). For any 2 elements from X, i am given a score (e.g. 0 or 1), how to find the relationship between the features of the 2 vectors to derive the score.
This is just perfect! Thanks a lot for pointing me to this package.
When you say "proc autoreg" are you talking about SAS?
?? First you intruduce **X** as the data matrix, then it somehow becomes the loading matrix. Also, did you mean the size of the data n x d rather than d x n?
To add to this. For the mixture model - I guess you could fit a mixture of 2, then 3, then 4 distributions and stop after there is no significant increase in log-likelihood or some such...
Hi Sid, the OP appears aware of the fact you mention in your post. It also does not appear to answer the question. :-)
Use the search function also on stackoverflow.com itself. I've seen this question come up quite often. Most often "one dimensional" instead of "univariate" though. See e.g. http://stackoverflow.com/a/8946299/1060350
I am probably using the wrong words, sorry. I am looking for a series that is able to predict stock returns in a better way than the historical mean returns would. Thus in my OLS regression the dependent variable is the stock returns and the independent variable is a potential series that may have predictive power. My MSPE measure for the out of sample analysis compares the MSPE of the model with predictor series to the MSPE of the base model containing only the historical mean returns that's why it has its own significance levels.
Can you elaborate a bit more regarding what your variables are? How many independent variables are there? The use of 10 items, six hazards, and a composite scale confuses the matter.
I would use the simulation (envelope()) function before I put any statements on that, you need to evaluate how many times you could observe similar values of L by chance
Don Rubin also had a hand in the EM algorithm and the development of causal inference.
It was easier to explain at this level of generality compared to focusing on the Normal distributions! Computing the derivatives and showing the precise form of the CDF are unnecessary distractions (although useful when you start actually coding the numerical solution).
The part you've quoted only mentions mutual independence; but ten lines down:- "These parameters do not assume that the positive and negative elements have equal probabilities of occurring, but only assume that the elements are independent and identically distributed."
See also other threads, e.g. http://stats.stackexchange.com/questions/7110/difference-between-longitudinal-design-and-time-series http://stats.stackexchange.com/questions/11413/longitudinal-data-time-series-repeated-measures-or-something-else
It's hard to guess what's going wrong from such an abstract general statement. Could you provide some details of what you've done?
Thanks for the note about one-sided test, it helps.
Sounds like `sureg` to me; did you try to look at the documentation?
You had then better add astronomy, medicine, biology, chemistry, ...  In other words, it's just as bad to assert that 20 is "not enough" as it is to assert that it's good enough. Actually, it's probably worse. Imagine a food safety trial in which the first eight subjects who were administered a nutritional supplement died of unanticipated side effects. Would you advocate continued testing on the basis of your "20 is not enough" statement?
@QuoraFea Probably I am just too drunk ...
Excellent. This makes sense - thank you.
@HongOoi good idea, thanks!
@probabilityislogic yes (and this does happen in practice). I made my question clearer now.
Follow up q: Given this, is it fair to say that the pre-aggregation of the data may somewhat dilute (but not completely eliminate) the sensitivity of the BFAST approach to detect breaks in the seasonal component?
@chl Thanks for fixing the typo.
Or were two members of the committee out to lunch while candidate 1 was being interviewed? In that case one might treat their votes as 'missing data'.
I have never been in a situation where discarding the data is the best answer, and I have never seen it discussed in any of the survey statistics books. In most of survey statistics, getting the data is at least five times more expensive than any of the following data processing and analysis (except probably for some cheap web surveys where the data collection is nearly free). If you are in an experimental world, then you should not tag your post "sampling", and rather use "experiment design" instead.
DoF in linear regression work out to the trace of the hat matrix or the sum of sensitivities -- so the motivation/concept aren't all that far out.  Tibshirani & Knight proposed Covariance Inflation Criterion which looks at covariances of model estimates instead of sensitivities.  GDF seems to have been applied in a number of model procedures like cart and wavelet thresholding (Ye's paper on adaptive model selection has more details), and in ensemble methods to control for complexity, but I don't know of any discrete estimation cases.  Might be worth trying ...
This answer looks a bit... like a rave of a sage. Many clever terms, hot link to download something. Head goes round.
Where does "the true error is 0.23" come in to the calculation you have described?
Thanks for the info. I updated my question.
A reasonable approach to such a problem would to run with a number of randomly selected subsets of the 40,000 documents of known and increasing sizes that run fairly quickly (e.g. 50 documents, 100 documents, 500 documents, 1000 documents, etc...).  Given this information you should at least have an approximate estimate for the expected amount of time the problem should take (e.g. fit a set of polynomials to the observed runtimes and extrapolate to 40,000 documents).
I do have the numbers behind the percentages as well, forgot about that
In real life models are just approximations of reality.  None of the models generated the data.  That you can be sure of.  What you can say is which model of the group gives the best fit without overfitting.
I still ended up with a fairly large mean for the APE – I guess the problem is that calculating z for $x_i = forecast - actual$ will filter out the largest errors, which may not be the largest APEs. So I still have huge APEs that blow my mean out of proportion.
I would need to take a look at Csiszar's book.
@kjetilbhalvorsen No, this is not a good question for Stack Overflow because it is asking for a tool or library. We don't accept this type of question on SO.
An excellent paper for the beginner in data mining
I think you're referring to http://stats.stackexchange.com/questions/93202/odds-ratio-from-decision-tree-and-random-forest
I heard today that a Hidden Markov Model might do the trick. Any thoughts?
Can you write out SVM?  Acronyms are tricky and not all readers of this site have English as a first language
Or is there some other time series prediction package for repeated events that better to use for this kind of problem?
Can you state more about your situation, your data, & your goals? Eg, do you have time-series data?
Do you mean: you get $n$ independent observations from 1 known (i.e. with given mean and variance) normal distribution?
Frequency weights are when the row of data actually represents w observations in the sample; whereas survey weights mean one observation in the sample representing w entities in the population.  I don't know about SAS, but SPSS thinks of weights as frequencies and hence concludes "ah ha, this weight of 23 means they saw 23 people in the sample with exactly the same score, that gives me lots of confidence about the result..."  So apart from anything else it uses the population size (which the weights will sum to, in a weighted-to-population sample) as the sample size.
I believe this is called "Naive Bayes". (http://en.wikipedia.org/wiki/Naive_Bayes_classifier)
Good general response, thanks.
Adding it as a covariate was my initial thought too, but I'm seeing a problem with that, one that is conceptual/logical rather than purely statistical.  Taking that route would mean testing group differences as if each group possessed an average level of reaction time.  Since this is something that is clearly not true in the populations, is it nonsensical to try this angle?...You probably will get additional useful answers if you explain why you believe comparable reaction time to be paramount in your study.
(+1) This is all true, but the original poster seemed to be describing a situation where the authors were trying to do model selection, and some of their candidate models were ones that did not include interactions - so their motivation was guided by AIC rather than by something substantive (which is always a dangerous thing to do, but apparently they've done it). When you're guided by something substantive, then the model structure should be dictated by that. But, when you're guided by a statistical criteria, leaving out main effects can have bad properties, as I've indicated in my answer.
I was under the impression that SIAR (stable isotope analysis in R) gave the BIC.  I could be wrong though. Is the BIC necessarily determined analytically?
That's a good answer. In addition, I'd suggest horizontally jittering the points, so they don't overlap, especially if you have more points per group than this. In ggplot2, the geom_jitter() will do that.
@Aniko I'll be more specific: I'm fitting a Fine & Gray semiparametric proportional hazards model for the subdistribution of a competing risk. In Stata, this is the `stcrreg` command.
Do you think you could use $\\TeX$ markup to make this answer readable, John?
"smoth" can be interpreted in several ways. A reasonable and convenient interpretation would be smoth $\\equiv$ continous and differentiable (first and second derivatives exist). This would lead to a wide range of  non linear specifications of $\\theta()$ like trigonometric, polinomials, etc. (Even thought these functions are non linear, they can be bounded between [0,1] by applying a trasformation on them)
If you use correct finite-sample methods, the variance goes to zero just as your sample hits the population size. That is, there's no maximum size; the proper formulas work as they should, right up to $n=N$.
Thanks Graham, I was looking everywhere for an option to select, but I see you need to double click on the output and a model viewer comes up with the information. Thanks.
Please consider upvoting and accepting some of the answers given to your prior questions.
It seems that you wanna do some heckman correction for selecetion bias. I don't know stata very well, but I know there is a function for that. look for heckman correction at stata help. It might help.
Wow ! Thanks for the link to the online book :)))
BTW, which other software apart from R and Matlab would you recommend?
Without much mathematic you could say that a continuous variable has a density with respect to the Lebesgue measure, and a discrete random variable has a density with respect to the counting measure.
thanks, will look into it!
+1 And if $\\mathbf{\\Sigma}$ is unknown but estimated by the sample covariance matrix $\\mathbf{S}$, the region is given by a quantile of the F distribution: $\\frac{n(n-p)}{p(n-1)}(\\mathbf{\\bar{x}}-\\mathbf{\\mu})^T \\mathbf{S}^{-1} (\\mathbf{\\bar{x}}-\\mathbf{\\mu})\\leq F_{p,n-p,1-\\alpha}$.
I am looking for the distance from the regression line, in number of standard errors of the residuals. Usually in a standard lm I would say lm <- (x ~ y); res <- lm$residuals; zscore <- last(res)/se(res). I think you are showing me how to do this in the weighted case above but would you mind clarifying exactly which bit of your answer does that? I think I mean standard errors for fitted values, as per your uncertainty above. Many thanks for your very complete answer.
Bootstrap (and jack-knife) are typically techniques to estimate sampling distributions for parameters of interest. Cross-validation is a bit different, which compares out of sample predictive accuracy. Are you sure you don't mean "leave one out" cross-validation versus "leave n out" cross-validation?
@David (+1) Many thanks for the fourth point!