tttjjj/elt_bench huggingface dataset:
wasn't me either, but as mentioned by yourself, pre-allocation of the result object will help with the speed issue. One trick, if you don't know, or it is tedious/difficult to determine, the size of the result object you need. Allocate something reasonable, perhaps pre-filling with NA. Then fill in with your loop, but add a check that if you are approaching the limit of the preallocated object, allocate another big chunk, and continue filling.
@whuber That question is testing two means and also has an inequality as the null, whereas I have an equality.
@JulianHi Sorry I deleted my comment, I'm answering it instead.
You might like to take a look at answers to these two questions: [Q1](http://stats.stackexchange.com/questions/61510/how-do-i-write-a-mathematical-equation-for-arima-0-2-1-x-0-0-1-period-12) [Q2](http://stats.stackexchange.com/questions/69407/how-do-i-write-a-mathematical-equation-for-arima-2-1-0-x-0-2-2-period-12), which - while they individually have some repeated values - because they repeat different things, between them should give you a clear idea of the general approach.
@whuber Sorry about "drive .. type I and type II errors to zero."  That's the information theorist in me.  I meant "achieve arbitrary low probabilities of type I and type II errors, given enough observations."  I will also re-formulate the hypothesis in the edited part (see new edits.)  And yes, I realize that for very low $\\pi$ it's impossible to separate the hypothesis.  Obviously, if $\\sigma_a^2=\\sigma_0^2$ you also can't separate them.  What I am trying to find out is if it is possible to separate them for $\\pi>1/\\sqrt{n}$, $\\sigma_a^2>\\sigma_0^2$, and arbitrarily large (but finite) $n$.
Automated approches such as Stepwise Regression and LASSO can be very dangerous and lead to misleading results. Use Bayesian methods for variable selection, which incorporates model uncertainty.
(+11) Great answer and nice illustrations! (I have to wait two more days before offering the bounty.)
You will want to test if these values aren't caused just by a few single instances ("outliers", although these have more effect on Pearson than on Spearman).
Thanks very much Xi'an! I will dig up the literature to understand it further. I am a little confused right now, but I also think I need to clarify the question further. I will get back after a little more study. Thanks again!
Welcome to our site, Alexander, and thank you for this contribution!
-1 This is an intellectually dishonest quote from a bad novelist, playing up a popular romantic myth of scientists as Rand-esque revolutionary loners in order to pander to anti-science crankery. Moreover, it has nothing whatsoever to do with statistics. What is it even doing in this list?
Thanks for elaborating your answer, @DJJ. +1
The original definition of $\\tau$ takes as a domain an appropriate space of random variables, or, more properly, induced measures from a probability space, *not* a point in $\\mathbb R \\times \\mathbb R$. Indeed, when interpreted in this way, I believe the answer is *affirmative*, i.e., there is a RKHS under an appropriate restriction on the random variables.
(+1) A couple minor quibbles: (a) Generalize to the matrix case mentioned above, (b) The *distribution* and not the pdf is characterized by the ch.f., so the last sentence should be changed and (c) Using `\\exp` will get $\\exp$ to render properly.
You can use the `lm.ridge` routine in the MASS package.  If you pass it a range of values for $\\lambda$, e.g., a call like `foo <- lm.ridge(y~x1+x2,lambda=seq(0,10,by=0.1))`, you'll get back the generalized cross validation statistics in `foo`, and can plot them against $\\lambda$: `plot(foo$GCV~foo$lambda)` to pick the minimum.
Can websites sue the person analysing the data (available for download) and publishing the results?
Actually, "conditional logit" is a very ambiguous term. It some contexts (mainly when dealing with panel data), it is equivalent to Chamberlain's estimator, but it is very unfrequent. Most of the times, it refers to a cross-sectional model where the outcome variable can take more than 2 values. All your proposals actually refers to packages that consider this last possibility. The same with mixed-logit: it is not a fixed-effect logit. I've already taken a look at Farnsworth's overview, but it is not exhaustive enough to speak about this estimator. Anyway, thank you for your answer !
Glen, could you help by providing some reference where I could read about "pdf giving relative probabilities"? It's very interesting and I don't remember to have seen it in my books. Thanks.
@ems - by calculating the normalising constant you will construct the quadratic form required.    it will contain the terms needed to compllete the square
Depends on your parametrization. For example in 3D, spherical coordinates are usually given as an angle between $-\\pi$ and $\\pi$ (longitude) and another one between $-\\pi/2$ and $\\pi/2$ (latitude). For longitude add $\\pi$ and divide by $2\\pi$ and for latitude add $\\pi/2$ and divide by $\\pi$.
Your first step should probably be to write the problem down formally as a combinatorial optimization problem. Then you need to figure out of if the problem is NP-complete (note that many such problems *are* NP-complete). If it is NP-complete, you will not be able to find the optimal solution unless your dataset is very small. However, you can still use heuristics or approximate algorithms to find a reasonable solution.
Thank you for your comment. It is indeed a fit to data, and I have no a priori knowledge about correlations among the parameters. The function I chose was just an arbitrary (non trivial) example, I will fix the OP to a more sensible function (and provide constraints on the data). You were right in your observation about $\\beta$ - that was a typo. The data was obtained by measurements on a population.
As far as I know, there is no other common way to deal with ordinal regressors - if you reject to treat them as interval - as via optimal scaling (= optimal quantification) which transforms the regressors into interval nonlinearly so as to maximize the prediction of the dependent. In SPSS, for instance, this known as categorical regression (CATREG). Search for it.
The conjugate prior to the Poisson distribution is the [Gamma distribution](http://en.wikipedia.org/wiki/Poisson_distribution#Bayesian_inference). But the distribution you have written is not exactly a Poisson distribution since your outcome space is $[0,1]$ rather than all positive integers. Perhaps someone whose calculus is better than mine can clarify how that changes things.
@Legend. As I pointed out, your graph plots the sorted data points (y in our case) vs the corresponding values of negative binomial random variable which correspond to the value of probabilities `ppoints(data)` in your case. Therefore, the last point in your graph is (175, 170) which is below the `abline(0,1)`.
One thing to keep in mind is that a prior is supposed to be (at least in principle) *information not contained in the data* but that you have available through some other channel (previous study, intuition, etc). Obviously that raises a bigger issue about empirical priors in general, but right now it sounds like you just don't have much outside info. So why not use some kind of weak prior? You could use the beta distribution, which happens to be conjugate, but is also very flexible. Then you're also free to control the tails and apply hyperpriors (e.g. the half-Cauchy).
Could you please provide a reference or at least sketch the theoretical justification for this method?  It looks *ad hoc.*
@JoelW., I have used daily values to obtain a monthly sum.  The original article for this drought index standardized the data to compare different time scales using the log-logistic probability distribution function.  I am really only interested in one timescale, but since the paper standardized the values, I thought I should as well.
Also this one: P. A Zandbergen and D. A Ignizio, “Comparison of Dasymetric Mapping Techniques for Small-Area Population Estimates,” Cartography and Geographic Information Science 37, no. 3 (2010): 199–214. http://www.ingentaconnect.com/content/acsm/cagis/2010/00000037/00000003/art00004

Which seems to call out for blending.
Yeah, I was actually thinking about this earlier and I agree: it might be better to just treat each response as separate documents. Anyway, OP can play with both options and figure out what he thinks works best.
Could you give a suggestion of how to do do that / go forward, perhaps by an example?
+1 for your time and answer. Thank you. I have been reading on this for the last 8 hours and looks like I need to add a prior to work around this. However, I am unable to find a suitable reference that explains this fact. Is this something obvious in the stats community or would you happen to know a reference that points out this requirement for KL-divergence?
Thanks! How do I check it graphically? Do you suggest plotting actual dependent variable vs predicted for the variable of interest at mean of other controls?
Thanks Dikran Marsupial - your answer is completely correct.  When type is set to C-svc, a soft margin SVM is being implemented.  By definition, anytime you have a Cost (C) hyperparameter, the SVM is soft-margin.
"No value" was a little bit too harsh, I meant: "no value" for the specific nature of my question which is focused on equal-freq-histograms (and rules of the thumb for it).
In general, it's not "the" for multivariate versions of univariate distributions, with a few conventional exceptions ("the" multivariate normal for example). There are many ways of getting multivariate extensions, depending on which features are most important to have. Different authors may have different multivariate versions of common univariate distributions. So generally, one might say something like "*a* multivariate Poisson", or 'So-and-so's bivariate Poisson". This one is a pretty natural one, but not the only one.
In response to a very similar question, I offered http://stats.stackexchange.com/a/3904, which you might find useful.  Strictly speaking, it does not answer your question about "motivation": that came several generations after de Moivre, with Gauss' development of the method of least squares.
not sure anyone has been fully responsive to question. It has two parts: (1) Does the modeling strategy described supply a defensible solution to the problem she wants to solve—namely, what is the likelihood that a jury drawn randomly from a community, *C*, with specified demographic characteristics will vote to find a defendant guilty? And (2) If the modeling strategy is reasonable, how many “juries” must she select, and how many “verdicts” must she simulate for each, to report a defensible estimate of the prob of conviction & 0.95 CI? She wants to economize on computing. See her last comment
Related threads: http://stats.stackexchange.com/q/10088/930,  http://stats.stackexchange.com/q/9016/930, http://stats.stackexchange.com/q/11175/930.
If it's not small sample size, as noted above, it could be due to collinearity of predictors or non-identifiability of regression coefficients.
You have good answers already but I'll just add that you can use series approximations to argue for a similar adjustment even if you don't actually assume lognormal distributions ([see about half a screenful or so under 'Edit2' here](http://stats.stackexchange.com/questions/46874/covariance-of-transformed-random-variables/46930#46930) for an outline of this kind of argument).
Thanks Dahly for reference
@chl I formatted the `Error` term of `aov()` this way to specify that `Time` is the within-groups factor. From Baron `Error(subj/(color + shape))` seems to be used in the same way.
@Michael, not only did I look at it, I implemented the algorithm to make sure it made sense. Of course I could be mistaken: if you have a different interpretation, then please share your code with us to demonstrate how it really works.
I am struggling to find a coherent question here: could you please clarify what you're looking for?
@whuber I got the same result $((t+1)/2)^2$ for the CDF of $\\max(X,Y)$ as you did, essentially via a one-liner and by the same argument that you used, but then I messed up in the differentiation of the CDFs, in that I forgot the factor of $\\frac{1}{2}$ which occurs in the application of the chain rule when one differentiates $(1+t)/2$ w.r.t $t$; writing the derivative as $1$ instead of $\\frac{1}{2}$. Thanks for pointing out the mistake. I have corrected my answer. My point really was that mapping the RVs to $U(0,1)$ as suggested, using a plug-and-chug formula, and then mapping back is overkill
@Frank I used to think that, too, until I did some simulations and was surprised at some of the "bad" situations in which the t-test worked well. The issue is how much non-normality messes up the *standard error* (not the SD) *relative to how the sampling mean varies.* Now I adopt a more nuanced attitude: the t-test can be remarkably resistant to many kinds of non-normality. It appears to get into most trouble with strongly skewed distributions (and I'm not persuaded that some of the efforts to incorporate skewness adjustments are any good except in very special circumstances).
I actually implemented a similar lossless solution to your `insert every value` and was just thinking (while I was brushing my teeth) about placing it on here 2 minutes ago haha. I'll accept your answer though, thanks.
Note that if you adjust parameters of an algorithm after seeing its performance on the test set, test set error is no longer guaranteed to be an unbiased estimate of true error.
`x <- restrictedparts(26,8,include.zero=FALSE) #gives 288 possible item counts for partitioning groups`... then there is also the valid permutations of assigning actual assets to the 1:26 ordering... so, there are... a lot of possible allocations.  Not very promising.
Here we are talking about series conditioned on its past values. Please do some googling and reading first for conditional distributions and conditional mean and variance. Then we can discuss further.
(ctd) ... not being medically trained, I'd need the researchers in medicine to help guide the statistical research (to help with what questions need to be answered, what information might be useful, and so on). The statistics might then later inform the direction of medical research, but at this early stage I think the medicine needs to drive the statistical methodology more than the other way around.
+1 for distinguishing what area of statistics you work in. There are areas of statistical computing (e.g. unstructured text analysis and computer vision) that a lot of functionality exists for in Python, and Python is seemingly the lingua franca in those sub-domains. I think where the Python community has to catch up on is improving the data structures and semantics around classical statistical modeling that R's design is so good at. The scikits.statsmodels developers are making a lot of progress on that front: http://scikits.appspot.com/statsmodels
My first guess is no. In what way are your data "highly non-normal"? Are they counts, or binary classifications? If they're just skewed or something, bootstrapping may be a possibility.
What question are you trying to answer and what data have you collected? This question is too vague to get any useful information.
A very well cited paper in my field runs 225 tests on alpha = .05 level, cite that a "strict Bonferroni correction" (Abdi, 2007) would be too conservative because of "substantial high correlations between a number of symptoms and between some of the validators". Abdi's book chapter is "Bonferroni and Sidak corrections for multiple comparisons". 

I have a problem with that, and want to do it better. Are there other options than simply stating "we use .01 to avoid false positives"?
@Luna What is not working, *precisely*?!
first of all, I am sorry: I am actually referring to a Principal components analysis, not to factor analysis, my mistake. Moreover, I was looking to find a way to not select arbitrarily witch correlated variable I shall keep. I add more info about the problem above.. thank you again
Informally, if you observe those three things, then there appears to be mediation (although, as always, you can't, statistically, rule out the possibility of some sort of confounding creating this pathway spuriously). But, it appears this question is asking for a formal statistical testing framework - is that the case?
Are the charts on a per-worker basis?
-1. An AUC is not a classifier - it represents a collection of classifiers, differing by the cut-point of the predicted variable. As such, you can't have an AUC *for the class* - an AUC represents both classes.
@shabbychef Well, let's go this way; I was thinking of this too but others might not grasp the title at first sight...
This would have been the obvious answer if it hadn't been stated that you thought you had a good interview. That's extra information. You fall in the part of the Venn diagram of all Amazon interviewees that thought they had a good interview, which is of unknown size, but can maybe be estimated somewhat.
p-values tend to go down as sample size goes up. With 682k sample size you are likely to see a lot of small p-values, even the detected difference or association is trivial. So, don't just look at p-values, also focus on the regression coefficients and their interpretation; they surely are statistically different, but are they also practically meaningful?
For large sizes I usually rank the tests using the test statistics since p becomes meaningless. So yes you are right, use common-sense interpretation and if your p is "too small" then you are on the safe side for publishing.
Note that the argument in a mixture is going to be the same ($z$ in this case), even though the subscript that denotes the *form* of the density is different. See [here](http://en.wikipedia.org/wiki/Mixture_distribution#Finite_and_countable_mixtures) for example.
I am using it for [0,1], I know arcsin has its problems. I am trying to transform a dependent variable in a multiple linear regression context.
Ah, thanks for commenting. For some reason, I thought it was for responses that tend to cluster.
I'd start with thinking that until you've got a reason to believe otherwise, statistics actually *won't* help answer the question "what kind of criminal entities are behind these frauds".  Whether there is any such reason would depend on the secondary information you've got about each transaction and its user eg demographic, location, if they're associated with a group, etc.
will make a try! Thank's a lot!
Thanks for the response. I have another small question. Does $E(e^{2t})=e^{2t}$? The expectation of a constant is just the constant itself, as you had said. I think my instructor meant to write $E(e^{xt})=E(e^{2t})=e^{2t}$. Even if the variable $t$ is there, can I still assume that $e^{2t}$ is a constant?
Thank you for the informative and elaborate answer. It solves the question perfectly.
@Henrik, I added a note about the difference between making the random effects correlated vs. uncorrelated that I think improves the answer - thanks for pointing it out.
I have to ask the OP why he has cross-posted this to math.SE, here, http://math.stackexchange.com/questions/630201/how-to-arrive-at-a-specific-formulation-of-the-relative-median-deviation-relate, (where he has also received a correct answer) and also, why he thought that any answer in math.SE is worthy of a bounty, while in CV is not.
Do I understand correctly that you resample years?
@D L Dahly: See my answer of March 22. Factors have zero mean in Stata.
Do you know how the two samples were drawn from the population at the two time points?  More specifically, do you have good reason to believe that $P_t/N_t \\approx P_{t+1}/N_{t+1}$?
MAP in obtained by using a prior. In this case when the prior is uniform, this is equivalent with additive smoothing, or Laplace smoothing. So, saying that you do not need MAP because you use MLE with smoothing is wrong, because MLE with smoothing is a particular case of MAP. Also, MAP can be used also to incorporate other kind of priors. Also, you can use a mean estimator, which is a combination of prior mean and MLE. And, finally, both of them are not fully Bayesian since are point estimates. A full treatment would involve to integrate out the parameters.
It's unusual for *any* variables in a regression analysis to need normal distributions.  (The standard situation concerns a single *dependent* variable whose *residuals* are compared to a normal distribution.)  Could you please tell us a little more about the nature of these variables and the intended analysis?
@rolando2 This is a reply to your comment on ineffectual control: your question still needs a bit more refinement regarding your definition of reversal. Are you interested in a reversal of the expectations of the coefficients? The answer above and the one below adress that. Or do you want to consider the power to statistically *detect* reversal (e.g. reject the null hypothesis that the coefficients of both the full and reduced model have the same sign)? The later depends on the residual variance and the magnitude of the expectations of the coefficient (thus indirectly also on R squared).
You have an extra right parenthesis in the formula for C. You'll want to check that formula, and probably also my reformatting of the others. They could be further simplified considerably, but I assumed you had some purpose in not doing so yourself. I mostly removed the unnecessary parentheses, the x between 2 and G, and moved 100 to the front. Revert anything you need to of course.
1) Friedman analysis is for quantitative data; 2) You don't have to convert data to ranks: the program will do it for you internally; 3) Result (significance) is not affected by reversion of the rating scale
Max - (i) what do you mean by cumulative probability if not CDF? (ii) Where do your equations come from? Are you sure you have them right?
Closing as it is effectively point 2. of that other question and got no answers for quite a long time.
Linear regression does NOT require that the predictors be normal, nor that the dependent variable be normal, only that the residuals be normal.
Bit late for the party, but: the first thing I'd double check is that the validation results of the forward selection were actually obtained with independent cases. Otherwise, you'd likely be subject to a huge optimistic bias.
@Fernando. tbats() does not have an xreg argument. If you include one it will be ignored.
Could you please elaborate on the conditionality you mentioned? In the univariate regression with $p$ predictors and an intercept, testing a hypothesis on a linear combination of parameters $\\psi = c'\\beta$ uses test statistic $t = \\frac{\\hat{\\psi} - \\psi_0}{\\hat{\\sigma} \\sqrt{c' (X' X)^{-1} c}}$...
Now that you mention this, I do seem to recall reading this somewhere during my search for an answer. So since the information source is on a csv file, I actually need to add columns to accomodate the new fields for each binary input? For instance if the bedroom input ranges from 0-4, using your example above I'd create 4 additional columns (total of 5 since '0' bedrooms means studio) and a 3BR condo would be expressed as 0,0,0,1,0?
Usually, confidence interval are constructed for *unobserved* values... I don’t even think Ahsan got your point, cf his additional question.
Either would work -- if you do it only for the last category you'd have to assume the parameter values; if you do it for all categories you can use the values you have to estimate the parameters (you could use techniques that deal with censoring).
shhh... don't tell them that... it makes it look less magical.  You could just consider it an example of an accepted practice where what I described happens.  Look at the additive effects, record what is significant, and then look at interactions and ignore the additive effects changing (until you've looked at the interactions carefully and understand it).