tttjjj/elt_bench huggingface dataset:
Correct. More specifically I want to know $\\mathbb{P}(X_T>x)$ for some jump process (heavy tailed jumps). So each estimator just runs $10^3$ sample paths - it is like hit if $X_T>x$ and no hit if it is below the threshold.
It would be nice to have an explanation for *why* degrees of freedom is important, rather than just what it is.  For instance, showing that the estimate of variance with 1/n is biased but using 1/(n-1) yields an unbiased estimator.
Great anecdote :)
Then it seems odd that the experimental values are equally spaced. This suggests that your values are not observations from a common distribution, in which case the F-test doesn't seem to be relevant.
Ah yes, the former would be a randomization test... in this case maybe could have been a complete permutation test as you suggest. I didn't write it that way because I'm hoping the questioner adds a sample or two (which will quickly put it out of reach). If you argue that the whole site should be bootstrapped together then are you arguing that they're independent samples?
@Glen The Shapiro-Wilk test for the residuals of group A is the same as for group A (the residuals where computed as indicated [here](http://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics#Example_with_some_mathematical_theory))
As a result I would like a probability for each button of the user selecting it next. 

Regarding a Hidden Markov Model, wouldn't that forget the history of button-clicks whereas I would like to use the history to help the classification
Another way to think about it is as follows: when $\\sigma^2$ is large, most of your proposals ($x_j$) will have low density under the target distribution (for the reasons described above -- is that part okay?). Very rarely you will propose a value with high density under the proposal, and when this happens you will almost certainly accept it. Once there, you continue proposing unlikely values; since you rarely accept one of them, you just "stay" at your current, high-density sample for many iterations.
Great question. I really like Andrew Gelman's [paper](http://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf) with Cosma Shalizi about Bayesian model checking.
Henry, those numbers approximate the rationals $16/52$, $17/52$, and $19/52$, respectively.  They are obtained (in the usual way) from the eigenvectors associated with unit eigenvalues of the transition matrix. You only need $11$ states total (rather than $16$), because you might as well assume the process begins with an equal mixture of $HHH, HHT, \\ldots, TTT$ and make transitions among these states and the three absorbing states.
[Odds ratio versus relative risk](http://www.childrens-mercy.org/stats/journal/oddsratio.asp) offers a good summary of those two measures. Another caveat that is frequently seen is people that apply a logistic regression model, but interpret the results with RR instead of OR.
@cardinal, I have the same impression - $R$ (or $r$) refers to the sample correlation coefficient and would be surprised to see a widely used reference that uses that to refer to the absolute value of the sample correlation
Does working with all 450,000 cases really figure to give you much more useful information than analyzing a random subset of, say, 20,000?
@ steffen -- can u please elaborate on the weight thing..I didnt get it..I dint get the meaning of weights here..does it have any thing to do with weights option in proc logistic..can you please xplain this a bit more ?? thanks in advance.
An alternative to GLM would be to logistically transform your outcome variable and use a standard time series model.
Perhaps what he meant was that, even if the model is true, there is still random variation in the responses - this is captured by the error variance - this can, for example, be attributed to an imperfect measurement apparatus. Others sometimes conceptualize the error variance as that due to missing predictors (not necessarily errors in the form of the model), implying that if all possible predictors were measured, the error variance would be 0. This is not inconsistent with the first - the errors in measurement can be thought of as a "missing predictor".
Great points!  @Macro  I would expect some correlation between multiple data points within the same country (prices can be a little sticky).  Also, I ran the durban-watson test and found that the residuals are autocorrelated.  Looks like this one may be beyond me. Might be time to call in a true statistician.
@Joseph, using your language.  I am not sure that the CLT argument suggests that both beta.hat and sample.mean.beta.hat will converge towards each other.  But, that their respective distributions of outcome (defined by their standard error around the mean) will be normally distributed.  I think the two beta.hat(s) will converge towards each other simply because they will each become more firmed up or statistically significant as you use greater N and greater M.
Do you have an example of a classifier where performance degrades when the training set becomes larger?
Are you sure this is the best stack exchange site for your question??
I don't know of a single function in ggplot that does this, so you'll have to build it up yourself. I'd suggest you make some attempts, and then come back and edit your question with more details when you get stuck.
Coefficients of logistic regressions are also only some sort of partial correlations. If you have so few data, you cannot squeeze them endlessly anyway; why not look at some descriptive graphics and estimate models withouth the interactions and clearly state in the final presentation how limited the analysis is?
@deps_stats Nope. If you're interested in implementing it in R, you might find some ideas in [calendar heatmap](http://blog.revolutionanalytics.com/2009/11/charting-time-series-as-calendar-heat-maps-in-r.html).
Your link leads to a non-accessible page.  One can't see what sort of differences you are assessing or what is the question to which you need an overall answer.  But I think what you want to know is in the area of *scale development*:  how to combine results from various survey items into a smaller number of scores, or a single score, that validly and reliably communicates something about the respondent. A great, quick introductory treatment is Summated Rating Scale Construction. Spector, Paul E. (1992), Sage.
Does each individual remain in the group he is assigned to during the 3 replicate (i.e., if he is a non-drinker at the first measurement, is he a nondrinker throughout the test)?
(1) You're doing well, but think about how to interpret the percents in the second bullet. Drawing a picture (on log-log axes) will help you compare 0.8% to 1%. (2) In the third bullet, remember you need to work with *logarithms*. Thus, "within a factor of 1.1" means "give or take $\\log(1.1)$". This will enable you to answer the second half of question (a). (3) To obtain $R^2$, use your answer to the second half of (a) along with the new information in question (b).
Hi Relena and welcome to CV. I have taken the liberty of using the formatting tools to make your post more readable. Can you paste in the exact message you get? Please note that the data you show there doesn't have a `Year` variable at all, it has one called `Block` instead.
VAR models are regressions. Hence the usual regression assumptions apply. On the other hand, since VAR model admits MA($\\infty$) representation, if errors are normal then the data should be normal too. The coefficients are usually estimated using OLS, hence their estimates are robust to non-normal disturbances. However if you have cointegration, the usual tests are sensitive to normality, if I remember correctly.
For what it's worth, I don't think this is an exact duplicate. You might try editing this in a way to make 'looking for teaching strategies for making the fact that $p(d|H_0)\\ne p(H_0|d)$ intuitive' more distinct from the prior questions, & maybe it can be reopened.
"One realization" could be understood as one data point. You have seventy thousand. I suggest you change your title.
By the way, doing a regression with the centered DV and looking at the significance of the coefficient for each group does NOT appropriately control for the nominal Type I error rate, while following up the test of the overall R² with pairwise comparisons with a Tukey adjustment does. I really have to side with you here (and not with your stats prof!)
The data collection is fine you don't need to throw anything out. You just need to figure out which of your IVs are most important. Forward step-wise regression is one way to do this, and it really isn't evil as you said. There are plenty of more sophisticated things you can try (like using LASSO or all subsets regression), but if you're in a pinch just do step-wise regression. Do a google search for "variable selection" and "multiple regression".
Hi cardinal, yes I used it as a synonym.
Cross posted here: http://math.stackexchange.com/questions/858695/partial-parameter-estimation-mle-lmmse
Very nice, thank you. I worked through it and can see what you are doing, but i have some questions:
- With "value of the of the index that parametrizes the convergence in probability" you mean sth like $F_n$?
- the inequality sign marked (a) is a bit of a mystery to me. i toy around with triangle inequality a bit, but i dont get to the other side
- With the one marked (b) it is clear where the 2 $\\epsilon$ come from. The other part is somehow because of the monotonicity i guess?

other than than i like the proof.
@cbeleites: To test significance with the binomial confidence interval is of course equivalent to binomial testing. In my update I was referring to the confidence interval that one would try to construct from iterations of CV (even though I am not sure how to do it correctly in case of Monte Carlo CV that I described)... Anyway, please see the reply to this question that I just posted, I provided a very simple simulation there that I hope illustrates my observations.
I'm not sure what you are doing in query 2 about only using correlated FA's, so I did not comment. Regarding logistic regression, that assumes you have a dichotomized response variable (or variable that you can dichotomize). Then you have all the same issues with collinearity and model selection that you would have with the usual multiple regression.
Deviations (either only positive or only negative) from median body weight have to be DV, and different diagnoses will constitute set of IVs of the model. The only thing that stops me is that I don't know how to be, given that the distribution of DV is thus half-normal. I never worked with such distributed DVs before.
"statistically significant" has no meaning at all without a hypothesis and a probability model. Nor is your observed fraction interpretable as a chance that applies beyond your data unless you can show that your way of finding websites resembles random sampling. Only 1% of my friends smokes tobacco, but that is a reflection of my friends, not of how many people smoke tobacco.
You recoded your `FA` as a factor in your `lm` call. From what you write, `FA` should be a numerical measurement, shouldn't it?
@David All included studies lack an estimate of SE, SD, means, CI due to the nature of the analysis. See http://stats.stackexchange.com/questions/7426/eggers-test-in-spss for the description of the problem. Thanks for the suggested transformations.
I think my question wasn't completely clear, so I edited it.
Welcome to the site, @Seth-C. Since this is apparently a software-specific question, please tag it appropriately.
+1 Thank you for your help! Appreciate it! =)
Shouldn't the derivatives be partial?
The RF algorithm does not really suffer from high number of predictors since it only take a random subset of them (so called `mtry` parameter) to build each tree. There is also a recursive feature elimination technique built on top of the RF algorithm (see the [varSelRF](http://cran.r-project.org/web/packages/varSelRF/index.html) R package and references therein). It is, however, certainly possible to add an initial data reduction scheme, although it should be part of the cross-validation process. So the question is: do you want to input a linear combination of your features to RF?
"Now maximise this quantity with respect to p": i think you mean minimise.
Thank you,sir! Yes! The `Fig. 3.10` is the lasso regularization path.
There is no problem w/ using categorical variables w/ LR. Just as w/ regular OLS regression, you can represent categorical variables w/ a coding scheme. I provide a basic overview here: [Regression based for example on days of week](http://stats.stackexchange.com/a/21292/7290).
Nice idea.  The statistical assumption is that the interruption is independent of the execution state, which is a reasonable hypothesis.  A **binomial confidence interval** is a good tool to use to represent the uncertainty.  (It can be an eye-opener, too: in your 3/10 situation, a symmetric two-sided 95% CI for the true probability is [6.7%, 65.2%].  In a 2/10 situation the interval is [2.5%, 55.6%].  These are wide ranges! Even with 2/3, the lower limit is still less than 10%.  The lesson here is that something fairly rare can happen twice.)
+1, nice points. It reminds me of [this quote](http://stats.stackexchange.com/questions/726/famous-statistician-quotes#739).
Thanks a lot! I read that the latest version 0.14 of scikit-learn has improved the speed of the Tree-based classifier but I haven't had the time to upgrade yet. I'll definitely take a look at it in due time. Thanks.
Please register & merge your 2 accounts (see [here](http://stats.stackexchange.com/help/account)). Then you will be able to edit your previous posts, among other advantages.
Are you aware that R has the density, cumulative and quantile functions for a t-distribution already implemented (dt, pt and qt)? Or do you simply want to be able to do this for other distributions?
Matching moments is one approach, but that's not what I'd have called 'accuracy'. What do you need for it to achieve? Failing that, what are the consequences of however much *inaccuracy* you end up with? (this might gives us some clues about your loss function, which would lead to something to optimize)
I'm trying to imagine a situation that is matching your model. Maybe a physician checks their patients weekly to see if they are (1) still sick, (2) dead, (3) cured or (4) lost to follow-up. Are you trying to estimate the dead/cured ratio assuming that the transition probabilities of LFU patients are the same?
If I properly understand, a full column of X missing means that you did not observe an explanatory variable at all. I do not believe it is possible to estimate it unless there is a known relationship with the observed covariates. This actually happens all the time because there are covariates that you do not observe (for a number of possible reasons).
See also http://stats.stackexchange.com/q/11707/16974 for more technical explanations (which is not what the OP wanted, but for those interested in theoretical underpinnings)
@NickAlger yes, your formalisation looks correct to me.  But I assume there must already exist a standard "solution" to this estimation problem (either in terms of kernels or with other techniques), since this seems a very common situation (how to estimate the parameter distribution of a multinomial after sampling), and must have been encountered 1000x of times before.
Thank you.  I'll take a look at ARIMA models as you suggest.  I wondered if you have any thoughts on how to approach the second part?  Consider a time series trade prices for a security.  The observed prices tend to be negatively auto-correlated because of jumping between the bid and ask (eg first trade gets done at the bid price, next trade at the ask, etc).  In this way, the degree of negative auto-correlation depends on market liquidity (which isn't stable).  If i want to incorporate a dependency between the auto-correlation and my measure of market liquidity, how to think about doing that?
There is no problem with 0-1 coding, but the bigger issue remains the damage done in getting the data to this form. Correlation is really not that much use if your data resemble, or are identical to, your example. A binary response in itself suggests logit or probit modelling, but that would be nevertheless a poor choice if the dichotomization is like yours.
Two principles apply: You can never know whether your chain has converged to its stationary distribution.  And any test for convergence you can do manually, you can automate.  So your approach seems sound enough.
an anonymous user has been trying to make fairly extensive (IMO) edits to your post. If you don't agree with them, you can roll them back by clicking the "edited __ ago" link, finding the last version you prefer & clicking "rollback".
@Abhishek, how do you usually remove (or intend to remove in this case) randomness component?
Can you be a bit clearer? The arithmetic mean and median are functions we apply to data, not anything intrinsic to particular distributions... for example, data doesn't have to be Normal in order for you to calculate the sample mean.
Is port cost the same if the port is a loading port as if the same port is a discharge port?
I need to model overall abundance on an annual time scale, but also have relative indices of abundance at a seasonal time scale within each year, which I'm hoping to use to help tune the model (so to speak).  I want to combine the seasonal estimates into one annual number.  However, I only have the seasonal estimates and their CVs, nothing else.  Any ideas on how to represent the uncertainty in the pooled relative abundance estimate?
High leverage values do not necessarily have a large residual, nor are large residuals necessarily indicative of high leverage.
Could you please cite some textbooks, which do in fact claim that $x_i$ and $u_i$ must be independent? Usually either $x_i$ are fixed, which means that they are independent of $u_i$, or $x_i$ are stochastic and then you need only that $Ex_iu_i=0$, independence condition is superfluous.
+1 for glmnet.  It's a generalization of both lasso and ridge regression, that can do either one or a combination of both.
I feel like this does not quite answer the OP's question. This is a very good comment indeed--and I wholeheartedly agree with it--but the missing data have nothing to do with which error term to use in the denominator of the different F-tests of a repeated-measures ANOVA, which is what the question is about.
Sorry, I must have been unclear in my question. The idea is not that the subclass clusters need to be closer, in the data, than the main class clusters. In fact, these are two independent classifications; the main/sub relationship only applies to the view I'm trying to construct. I could just as easily make a view where the A/B/C classes are the main clusters and the 1/2/3 classes form the subclusters. There is no reason to expect that this is a "natural" structure for my data; I just want to see the extent to which it can be forced into such a structure. I'll clarify this in my question.
@JustinBozonier My perception is that most users only submit their responses as "answers" when they feel that their response is at least a somewhat definitive and direct answer to the question that was asked. If instead a user has a response that amounts to essentially just a few relevant thoughts that occurred to them, but they don't think these thoughts entail a complete, satisfactory answer to the question, they are more likely to submit these responses as "comments." Of course, different people have different standards about what constitutes a complete, satisfactory answer.
What is $F$? Do you know how to take the partial derivatives of the first term, second term, or neither?
It's a complicated system. I would use a neural network to predict the probability of winning, and perhaps the outcomes of the first sets.
My point was that numpy does not seem to have a function for computing the PMF of a Poisson but scipy does. I am not sure how numpy generates its random numbers and not clear on why that's a consideration here.
By presupposing a polynomial solution, this question implicitly rules out procedures that might be simpler or more effective.  I would like to suggest that you begin your research by reviewing any appealing threads related to [model selection](http://stats.stackexchange.com/questions/tagged/model-selection+-time-series?sort=votes). You may find many of these informative and stimulating. Then consider returning with a more specific question motivated by that information.
Would this be a good [reference about your "confirmatory-after-exploratory-factor-analysis fallacy"](http://pages.gseis.ucla.edu/faculty/muthen/ED231e/RelatedArticles/Van_Prooijen.pdf)? I'm unfamiliar with the problem, but fairly new to SEM usage (not to the basic concepts though, except in as much as those have been around longer than I have).
Thanks, that gives me a good start!
@Glen_b I won't defend Excel, but it's interesting to consider what is being done here. In `R`-speak, these procedures are subsetting and selecting two dataframes, computing a generalized outer product, and performing a matrix multiplication. There is a perfect step-by-step translation between how it would be done in `R` (in the most efficient, economical way) and how it *must* be done in Excel.
Unless I missed something there's no test at all. Possibly a trend test could be carried out since BP is treated as a categorical variable with ordered levels. What are the degrees of freedom for the test used in this article?
Great!! Is it your own consideration or are you refering to a book/paper? If the second is the case, can you please provide the source?
Ok. But finally I've written an answer without resorting to software language.
Each of 10 customers chooses 1 pie, with probability $p$.  So... how many TOTAL pies are chosen?  Of that many total pies, how many are apple?  What is the probability that, say, 4 of the pies are apple?
Thank you for your explanation. I missed this part. And more important, I missed the whole process. The fact that we actually start with likelihood is obvious, but I missed completely.
Well, I'd say if you're trying to test whether a given frequency is zero, _any_ occurrence disproves that null hypothesis with infinite strength, no?
Well I guess this is more of a question in programming, since you are asking about a tool... I've had fantastic experience with the Stanford NLP tokenizer. Link: http://nlp.stanford.edu/software/corenlp.shtml
Random Forests thrive in high-dimensional setting. Not sure why you would want to go down the route of PCA - which by the way does not take into account the target (Y) you are trying to predict. I would recommend using the in-built feature importance provided by random forests to select a smaller subset of input features. Just be mindful that random forests have a tendency to bias towards variables that have more no. of distinct values (i.e., favour numeric variables over binary/categorical values)
These are great questions; thanks for the examples!
Criminologists have tried using the number of DUI and disorderly conduct arrests made per police officer as a proxy for proactive policing intensity in regression where more serious crimes were the outcome. I don't think this identifications strategy would placate most economists these days.
I suspect that you'll find lots of examples when the data is too high-dimensional to visualize and too abstract to "read".  Gene expression studies, for example.
The question contains no question. Not sure what does "how to go about" entail. For crossover studies [paired analysis](http://www.cochrane-net.org/openlearning/html/modA2-3.htm) is a common solution. But if this is a human trial I would believe there should be already an analysis protocol. You better consult your team statistician.
+1 Nice find!  The paper also has a brief discussion of the PCA approach (as well as some other methods).