tttjjj/elt_bench huggingface dataset:
Thank you for your contribution! I am puzzled about something, though: although the histogram legend states they show *beta* densities, you appear to claim these also describe the outcomes of *binomial* simulations ("how often it happend in a simulation"). But the two are different things, even though they happen to appear fairly close in the illustration.  (That's a consequence of the near-normality of the Beta with large parameters and the Central Limit theorem for Binomial distributions.)
copulas are independent of the marginal distribution.
There are numerous questions and answers relating to modelling this sort of data; you might be able to find a few (and hence get some additional help) by using the search facility with appropriate keywords.
@Christopher: Regarding your last concern, it depends on what the OP means, which they haven't specified. If they are talking about central moments that this *implicitly* assumes that $\\mu$ exists and is finite.
That second form is often called '[Winsorizing](http://en.wikipedia.org/wiki/Winsorising)'
You seem to know the answer to your first question already :-). Consider a more powerful approach: study the distribution of all 30 p-values. Under the null, this will be a sampling distribution from a Uniform$(0,1)$ distribution (equivalently, their negative logarithms will have a $\\Gamma(1)$ distribution).  It would therefore be interesting to see the histogram of your p-values or--even better--an exponential probability plot. For more about this approach, please consult our posts related to [Fisher's method](http://stats.stackexchange.com/search?tab=relevance&q=Fisher%20log%20p-value).
Right. So another way to formulate the problem is in a bicriteria fashion: Fix a parameter R, and an integer k. The goal is to partition a set of weighted points in the plane into k regions such that each region has diameter at most R (the nearness condition) and the variance of weights within each region is minimized. Would this be a reasonable formulation @JeremyHageman
I didn't know how to edit the previous post. I googled already before coming here but found no code for the all the above methods. Also I am going to leave my question as is for now. Anyone who has worked with semi parametric methods for binary outcome would understand it. If not will post a specific implementation question again.
For redistributing you've got to come up with a hypothesis about what the scores could be so I can't help you there.  Also, when I say some, you've got to look at the current distribution.  The rest of your data suggest that maybe 10 100s is still lots (assuming they should be something else).  One way to look at it would be to run about 10,000 analysis in a loop keeping a random 10 of your 100s and tossing the rest.  What's the average outcome then?
That solution ignores how many employees chose the same password as $A$. For instance, if for a miracle every employee chose the same password, the probability on item *a* would be $1$, which is not $p$, because $n\\neq0$.

The probability that $A$ managed to enter the restricted area is not just $p$; unfortunately, it involves a complicated truncated binomial variable with parameters $k$ and $p$.
@mbq: Most of the techniques mentioned in the link are time consuming. I'm looking for methods with complexity proportional to the size of the data. IV satisfies this condition, but I have some concerns. I think that it is a bit odd that it takes into account only the relative risk. The number of occurences is not used.
Your emphatic assertion is very comforting, but presently unenlightening. Could you explain how?
First make sure you are using the most recent version of the package. Second, please provide a reproducible example. Third, this question is probably best sent as a bug report to https://github.com/robjhyndman/demography/issues
There are no answers because you give too few details! Spell your "corrections" out in full!
Thanks ! your "short answer" perfectly points out an interesting existing literature.
@dataznkid1 See [this answer](http://stats.stackexchange.com/a/31328/6633) to understand why the variances add instead of subtracting.
edited, thanx :)
@jbowman: I suppose Jesse was wondering if he correctly classified (labeled) his variables into "dependent" and "independent", i.e. if he correctly understands these terms; this is a valid question (and the answers seems to be positive).
This is a great answer and you're right that I can definitely see how it generalizes. Thank you for taking the time.
Thanks for the help.  I did see this and I think it is fantastic.  I did edit my question to just focus on how to read in a univariate time series.  Mainly, if I have the dates for my daily data, how can I construct a univariate time series in R dynamically?  I have tried a few methods but they seem to include the date as another variable when I would only like to use it as a reference for start/end dates.  Sorry, new to R and I am just trying to wrap my head around this.
You are not being precise.The model assumption is that the error terms are independent and identically distributed with a distribution that is N(0,σ$^2$) and is unrelated to the COVARIATE. What is Var(Y|x)? Are you conditioning on X$_2$ =x? Does the model assume the covariate is random in some way or so we assume that the covariate is fixed according to a design matrix? I think it is the latter and therefore Var(Y|X$_2$=x) is implied by the assumptions and does not need to be assumed.
Thank you for the feedback. Is there a test for what the difference in means for when the chi-squared test confirms that the distributions are different?
Have you set up the data using the `mlogit.data` command? And do you have a MWE for me to debug?
Good answer to missing data in general, but I am not uncertain whether it adequately adresses the more specific concerns here . We are looking at time series and want to predict makes deletion unfeasible and even multiple imputation problematic if entire months are missing (which means some seasonal effects have been lost).
I don't understand what $\\lambda_1$ and $\\lambda_2$ represent. If they are weights, why do you condition on them? And perhaps you need to consider a joint distribution, $P(w|\\lambda_1, \\lambda_2)$ instead of distinct conditionals on the parameters separately. $w$ can't have a normal distribution with variance $\\lambda_1 \\Sigma$ at the same time as a normal with viariance $\\lambda_2 \\Sigma$.
+1, I thought about putting that myself. For those who are interested, Simpson's paradox is also discussed on CV here: http://stats.stackexchange.com/questions/21896/
RF has a robust built-in feature selection - no need to use RFE so one can just tune mtry and be done with it.
@Corone, I just updated my original post with some data
So bootstrapping the additive process, rather than bootstrap the individual pools and them add them?  That makes sense, I should have thought of that.  Lack of sleep must be getting to me...  jeez!  Thanks.  I was using the percentile method for CI's, also.
@John-DavidDalton: Statistical tests are insensitive to whether something is practically significant.  They only are sensitive to what is statistically significant.  For example, tests of normality frequently report non-normality with large sample sizes even when the underlying distribution is practically normal.  You have two possible solutions to adjusting how sensitive the t-test is.  One is to adjust the $\\alpha$ value.  E.g., you could arbitrarily decide that only p < 9.865876e-10 is statistically significant.  Two is to round to a lower number of decimal places.
AFAIK deriving the CI from the z scores and back-transforming z to r for a CI on rs works.

Even for a bootstrap CI, it might be appropriate to first transform them.
1) Yes, even if not ideal for reasons in my first answer. 2) I don't think so, but terminology isn't key.
@Samik in a way, when you do a hypothesis test using p-values, you are assuming the null to true and thus the bias for the null. Its just a suggestion: try using Bayes Factor, may be that will remove the "biasness" you observe (although, I must remark, its not a guarantee that this will work)
@csgillespie Oh yes, I forgot about this one, sorry!
I also believe that repeatedCV is k-fold cross validation, done multiple times.
note that the choice of reference class is not important, if you are doing maximum likelihood.  But if you are doing penalised maximum likelihood, or bayesian inference, it can often be more useful to leave the probabilities over-parameterised, and let the penalty chose a way of handling the over-parameterisation.  This is because most penalty functions/priors are not invariant with respect to the choice of reference class
I assume the weights are for an Artificial Neural Network? If you cang ive context, you will get better help.
Do you mean the marginals are Poisson or the conditionals are Poisson?  My Johnson, Kotz, Balakrishnan observes that "There are no bivariate distributions with Poisson marginals for which the conditional distribution of either variable, give the other, is Poisson."  There is only one bivariate distribution with Poisson conditionals, and the sum is not Poisson (same source.)  Also, the correlation between the two processes must be zero, otherwise the sum can't be Poisson (variance of sum $\\ne$ mean of sum).  So the nature of the dependence is somewhat restricted...  I doubt this is possible.
I do not understand why they use that term. It is a loss function in the standard sense of the term. As you cite, the refer to the empirical risk minimization (http://en.wikipedia.org/wiki/Empirical_risk_minimization).
67*950000 is not the most intuitive configurations for me. I would have rather put the observations as rows, resulting in a 950000*67 matrix.
There is not much statistics in your question, so perhaps better suited for Stack Overflow... but what the hell, I'll answer anyway.
P.S. If you aren't already familiar with data.table you should be made aware that result is a data.table and won't behave quite like a data.frame, but you can just do `new.result <- data.frame(result)` and push the data.table back into a data.frame if you so desire.
I wouldn't aggregate; just consider the use of a smoothed response, mostly simply some kind of weighted moving average. Nothing stops you modelling that. You would be implying that most daily fluctuations are uninteresting or unintelligible, but that seems to fit what you are saying.
I'm not sure it makes sense to have `DID` as *both* a fixed effect, *and* a random intercept in the 2nd model. Moreover, having it as a fixed effect in the 1st model means that the choice b/t these 2 would be about which way to think about the effect of `DID`, not whether it needs to be included. On a different note, I notice you have an item (2); did you mean to have an item (1) somewhere?
As this strategy will get stuck in local minima, I suggest that you try it several times with different initial parameter values.
One way would be to assess, by intraclass correlation corfficient, the degree of agreement among the experts, and among the non-experts, and then to test if the two coefficients differ significantly.
I think the frequentist would (verbosely) point out his assumptions and would avoid making any useful prediction. Maybe he'd say, "Assuming the die is fair, each outcome has an equal 1 in 6 chance of occurring. Furthermore, if the die rolls are fair and David Blaine rolls the die 17 times, there is only a 5% chance that it will never land on 3, so such an outcome would make me doubt that the die is fair."
@Steffen: this was only a minor part of a much bigger setting (these fits actually happen in the middle of an EM), and the between-folds comparison was only marginally (in the nonstatistical sense) interesting to me. I decided to only calculate the overall AUC (just using the per-fold predicted probabilities). I agree with @cardinal that the "stratification" has its pitfalls, but your answer was the best I got :-) Especially your first sentence is dead on.
Thanks, this was also very helpful.
Thanks for your answer; I've asked a follow-up http://stats.stackexchange.com/questions/43405/item-information-in-irt
On a side note, the central limit theorem makes the formal normality check unnecessary in many cases when n is large.
I calculated the log returns by ln(ret+1) which is the same as the conventional way: ln(Pt+1/Pt). I'm investigating whether news sentiment is driving returns or return volatility and vice versa. In the first place if they do (by making use of granger causality tests) and if possible to what extent (here is where I would like to know how the inference changes)
@user1188407 Ahhh ! That makes perfect sense - thank you. Please make your comment into an answer and I will upvote and mark it as "best"
Does it matter what the sign and value of $a_2$ are set to?
@user603 Could you please elaborate a bit, as this alone is unlikely to help the asker?
I'd rephrase some of this. The constant is perfectly intelligible as the value predicted when all predictors are zero. Algebraic and geometric interpretations are easy. I'd agree that it's often not especially interesting or useful, but that is not the same as not making any sense. Conversely, there are occasions when linear proportionality is expected and fitting a line through the origin is exactly the right thing to do (think Hooke's law, Ohm's law, etc.). It is harder to think of really good reasons for fitting a hyperplane through the origin, but perhaps someone has a good example.
"*but obviously can't perform a T-test with what I have*" ... not so fast; that's [not so obvious](http://stats.stackexchange.com/questions/44475/is-there-a-statistical-test-to-compare-two-samples-of-size-1-and-3/44476#44476)! Your problem has already been solved, in an even worse case than yours. No need to simulate, you can do a t-test if you're prepared to assume that under the null, the variance for your patient is the same as the variance in the other group (which is just saying 'under the null, this patient is like everyone else' - if the mutation has no effect, that'd be reasonable)
@David, `rr <-records[records$action!="Login",]`
`ddply(rr,~user+table+action+week,nrow)`
I think you should change the title to reflect the concern with keeping the formats and labels. Actually saving the file in the other softwares format is trivial, keeping the labels and value labels is the hard part.
Data ordering is not critical, as I explained in the example. I'm still not clear why the estimates would be wrong in this case.
This should probably be community wiki. Other questions of relevance are [Resources for learning R](http://stats.stackexchange.com/q/138/1036), [Where can I find useful R tutorials](http://stats.stackexchange.com/q/75/1036), and the [meta stat software thread listing resources for R](http://meta.stats.stackexchange.com/questions/793/internet-support-for-statistics-software/795#795).
**The claim that "the residuals from your model... should not be correlated with... your... dependent variable" is not generally true, as explained in other answers on this thread.** Would you mind correcting this post?
You raise a good point, but I think this only tells a part of the story. Indeed, I would guess that the OP would try each of these and *still* be surprised by the result. The fact of the matter is that because the singular values are *ordered* in the output, they will not appear (and indeed are not) uniformly distributed as might be naively expected from "random" data. The [Marchenko-Pastur distribution](http://en.wikipedia.org/wiki/Marchenko%E2%80%93Pastur_distribution) governs their behavior in this case.
I am surprised to see such a dataset (age of kings at death) analyzed using methods intended for time series with regular intervals, because it differs fundamentally from them: it consists of interdependent *events* rather than observations at a given sequence of times. Not only is there then no reason to suppose that standard time series methods will be much good (except by accident or luck), one would expect that methods of data analysis more appropriate for such data would perform better.
@Coronier It's great to meet another person combining R with I/O Psych.
Please see edited answer.
Thank you- excellent, precise answer. You gave me a lot to think about/go over :)
I suspect you can get best bounds on the diagonal elements of the covariance using the same approach.  But to bound the off-diagonal elements will be much more difficult.  The full bounds will be connected sets of symmetric positive-definite matrices potentially having complicated shapes.
(*Note that the other post has been deleted; this is no longer a possible duplicate.*)
Thanks, but this seems like @Momo's answer.
+1 for the code
+1. Generally, such covariance matrices can be derived from matrices $\\mathbb{\\Sigma}$ having these properties: (1) they are positive semi-definite symmetric Real matrices and (2) the kernel of $\\mathbb{\\Sigma}^*$ is nontrivial, where $\\mathbb{\\Sigma}^*$ is obtained by zeroing out the diagonal elements of $\\mathbb{\\Sigma}$. For correlation matrices like yours, which have unit diagonals, this means $\\mathbb{\\Sigma}$ has $1$ for an eigenvalue. Sure enough, your matrix--with eigenvalues $1$ and $1\\pm a\\sqrt{2}$--qualifies.
@B_miner: Great! I don't know how, but I have overlooked GBM. It seems that using GBM it is easy do detect interactions and nonlinearities.
@ThomasH In general, within-subjects design are usually way more powerful, hence effects there are not that easy to dismiss. You need to do a power analysis to show that the between-subjects analysis is really more powerful. (You will need the correlation of the within-subject measures). I would Use g Power 3 for that.
This appears to be a question about some image processing software. It's not clear that it is on-topic here (it's focused on code commands and explaining documentation), nor that it is even statistical in content.
do you have IM ?
If $X$ and $Y$ are independent with finite variance, $E[X + Y] = E[X] + E[Y]$ and $\\mbox{Var}[X + Y] = \\mbox{Var}[X] + \\mbox{Var}[Y]$ no matter how the RVs are distributed, so unless I'm missing something I don't see how this answer tells us anything we didn't already know.
Note that the page says the *data* come from the AIS. It doesn't mention any results coming from there.
Also, neither test compares medians _per se_. Strictly speaking, they are tests of _stochastic dominance_. With additional assumptions—(1) distributions of identical shape in both groups, and (2) differences in distributions that are only differences in central location—they can be interpreted as tests of median difference. See also my response here http://stats.stackexchange.com/questions/65735/anova-kruskal-wallis-one-of-four-groups-has-different-distribution?lq=1
If B, C, & D are nearly collinear the second model is still valid, i.e. coefficient estimates are still best linear unbiased, though their standard errors will be painfully large.
@Honk I am beginning to follow.  However, your last expression, because it completely ignores the strong spatial correlation of any signal, cannot be correct (as you hint at the very end).  And I don't really believe you have any (informative) prior for "beam present;" what you do have is a characterization of the noise distribution.
How far did you get in the [Stata help system](http://www.stata.com/support/documentation/)?  What aspect(s) of its explanation of these commands do you need assistance with?
I am not sure what you mean by "downsampling". If you take a random sample of cases, things will of course run faster, but estimates will be less precise.
Excellent, thank you. We are not working with stata or R, but are instead shoehorning this into Powerpoint with Excel using bar charts of the two series' discrete probability distribution. Not quite so sexy but still passable. Thanks very much!
It is one thing to co-exist, it is another for the one to be considered as the other. But indeed, this strand of anti-hybrid approach is in the spirit of "there can be no synthesis whatsoever" -which I strongly disagree with. But I don't see the current hybrid as a _successful_ marriage.
Algorithmically it's quite simple. If there's anything you need clarified, just ask. (I don't know C# though.) -- Can you clarify if any of the cases I consider covers your exact problem? Are you just in case 1?
Please tell us about your population: what kind of thing are you sampling?  This is crucial because the entire point of systematic sampling is to cope with expected associations among nearby subjects in the sampling frame.
It seems like a *regression* should be the *last step* of your approach--the first part should come from a conceptual model wherein you explicitly specify the issue at stake. [Note: for examples, check out the Economics literature dealing with "**Auction Theory**". Though most of it deals with different *types* of auctions (such as "English auctions" vs "Dutch auctions"), there's bound to be material dealing specifically with *transaction costs*]
It's not clear what you're asking. If you correct the formatting you're more likely to get an answer.
It such tests really aren't available, this would be an answer. You may want to simply expand this a little.
@dassouki It's an interesting question, but seems to have little to do with statistics or data analysis.  Unfortunately, I don't think there's an SE site on optimization (yet).  If you were to couch this problem in just the right way you might get helpful replies on the [TCS site](http://cstheory.stackexchange.com/).
The question is about comparing pre and post measurements, not groups.
On the overall idea, a reading of Chapter 4 of Harrell's *Regression Modelling Strategies* may give you a different perspective.
Have you looked into Bayesian inference as an alternative approach?  It might have a stronger intuitive fit with your approach.
Breiman claims that RF does not overfit. http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm On the assumption that he is correct, perhaps there is some sort of inconsistency between your training and test set?
Indeed.  See also the other parts of the Anscombe quartet http://en.wikipedia.org/wiki/Anscombe%27s_quartet
(**1**) Check the documentation of `numpy.var`. The version I saw seems to indicate that the MLE estimate is calculated by default instead of the unbiased estimate. To get the unbiased estimate one needs to call it with the optional `ddof=1`. (**2**) For the upper-tail $p$-value, use the symmetry of the $t$-distribution, i.e., `one_tailed_p_value = st.t.cdf(-t_stat,df)` and (**3**) for the two-tailed $p$-value, do something similar: `two_tailed_p_value = 2*st.t.cdf(-np.abs(t_stat),df)`.
Type "conjugate bayesian gaussian" in Google to find some informations about the derivation.
How about you just try to do PCA first and SVM later.
Hi, I'll give a more detailed answer later, but simply put they are not the same. You should first be measuring the treatment effect within a study and then pooling across studies. The other way is known as the 'naive method' and is dangerous as it can give invalid results. Have a look at the wikipedia page for Sampson's paradox for some examples (http://en.wikipedia.org/wiki/Simpson%27s_paradox).
Just FYI, I took the convention that the second argument was the variance not standard deviation, if this isn't the convention you're using then all of the above is basically totally wrong.
I would use a scatterplot, it's hard to visualize what is going on with the lines everywhere.
@user43207: feel free to *accept* my answer if you feel it sufficiently answers your question, by clicking on the little check mark underneath the vote count for the answer. You get rep and I get rep from accepted answers. Everyone wins ;-)
@Srikant: Expand[Sum[x^i,{i,1,6}]^3] also works in WolframAlpha