tttjjj/elt_bench huggingface dataset:
I don't know anything about them, but this [reddit post](http://www.reddit.com/r/dataisbeautiful/comments/1wzruo/an_artificial_neural_network_in_my_coffeemaker/) on r/dataisbeautiful might be of interest.. the OP used ANNs in a way that seems to be along the lines of what you want.
The distribution of $X$ conditional on the fact that it is less than $Y$ is not the same as the distribution of $X$.
Unfortunately, the correctness of this answer depends on how the vague expression "longest" is interpreted.  Many natural and relevant interpretations, such as the *diameter*, would be wrong.
Are you asking the psychological question of why people like smoothness or the statistical question of why smooth functions are better statistically?
@RicardoCristianRamirez, that method I know as set validation (see above), and so far I have not seen it considered a type of cross validation. But that would depend on how you define cross validation: if the only requirement you make is "drawing without replacement", then resampling validation will always either be out-of-bootstrap or cross validation, and set validation may be either of the two, depending on how you do the drawing. Note however, that the definition in the Wikipedia article is even broader: it would include out-of-bootstrap as well.
+1  A heartfelt and clearly expressed position.
works like a charm. Thanks!
So, your task is to assess how much the clusters have reproduced the existent classes, yes? Then search for "external clustering validation".
Just looking at it, it looks like a skewed normal distribution. Are you sure the outliers are necessary for your analysis?
I suppose there's some model for which he (or she) is assuming the independent variables need to be normal (which is certainly not the case for any model he's likely to want).
If you don't want to compute the integral, you can "cheat" and note that $Y^2$ is $\\chi^2(1)$ and use its moment generating function to obtain the last result.
thanks for your time and the url... 
However, I started a new request on ordinal regression here http://stats.stackexchange.com/questions/65375/how-to-predict-rank-in-ordinal-regression-using-r providing my data...
Because the probability of commiting such type-1-error is by definition equal to the significance level, which is constant. Or put differently, p values are uniformly distributed under the null. Btw, it is advisable to do many simulations per setting, including the choice of n, not just one as in your code.
@user1911070 I edited my answer to address this point.
Can you please explain this further.
Hi and welcome to the site! The question as it stands is very broad and it is unclear what exactly you want to know. Could you please provide more details about your data, your model, your questions etc.?
Kudos to Jerzy for "proving" storks bring babies illustrating the concomitant variable problem.
I would refer you to the original article: N. Meinshausen (2006) "Quantile Regression Forests", Journal of Machine Learning Research 7. http://jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf
Huh... You should add a little bit of contextual information here, e.g. at least specify your working hypotheses.
Hi Kjetil--I fixed the problems with the $\\TeX$ formatting (but, wishing to change as little as possible, left several typos intact).  I do not understand why you are posting a replica of the derivation in my earlier answer, especially when you lost some crucial factors along the way which cause the final result to be incorrect.  Is there a particular point you are trying to make?
@Graeme It doesn't much matter, because this code does not define any kind of stochastic process, but here is what it does: $\\Delta=\\Sigma/\\sqrt{n\\Gamma}$; $p_t=(1-z)\\exp(t\\Delta)+z,t=-n,-n+1,\\ldots,n$. (I guess $\\Delta$, $\\Sigma$, and $\\Gamma$ are numbers.) It looks like a misunderstanding of a formula for an autoregressive process with multiplicative innovations.
"No practical use" is a bit overly-strong. $\\beta$, for example, is calculated using variance and covariance and variance shows up in many, many other calculations as well. People often prefer to report standard deviation instead because the units match the mean (and it's often closer in magnitude to it too), but I'd argue that reporting the raw means and variances/standard is hardly the only thing one can do with business-related data!
Given that a GAM of this sort will have far more than 5 coefficients (there will be a column in the model matrix for each basis function that comprises each cubic regression spline), you aren't supplying enough starting values. How about you git the model without the `family`, save that fit as say `foo` and then refit your model with `start = coef(foo)`. The last bit might need `start = log(coef(foo))`...?
@RichardHardy Just to clarify things, GARCH is an extension of ARCH. So pretty much everyone thinks those squared residual terms should be in there, if they're using GARCH. Alternately, you could estimate a stochastic volatility model.
Could you please provide the reference of that paper?
@chl: True, but the point is that these steps get to the essence of the question.  A meta-analysis would be helpful only when there are many studies (not just two) and their merits have already been carefully evaluated.  The question before us is really asking how one goes about evaluating the quality of a study, or pair of conflicting studies, in the first place.  Cyrus has pointed to some of the many aspects of this; a reasonable treatment usually requires one or two semesters of university-level study.  In this light I think his use of the term "heroic" is somewhat understated!
@whuber - estimate is a bad choice of words for me (you may have noticed I'm not the best wordsmith), what I meant was that although $\\rho$ may not be uniquely identified, this does not mean that the data are useless in telling us about $\\rho$.  My answer gives an (ugly) demonstration of this from an algebraic point of view.
@Glen_b I have edited the question.
@Charles - Sorta right. I think the P value is $2\\times.0031 = .0062$, because you did not state *a priori* which direction $\\lambda$ should deviate from $1$. I did state a one-tailed conclusion, though. In rejecting a two-tailed test, you may -- **and should** -- state a one-tailed conclusion in the appropriate direction.
I have performed a GEE and my results seam reasonable :-) It was easier than expected. Thank you for your help again!
Excellent, thanks for your help it's much appriceated. Gonna do a few more examples with your table formula !
I think a good tag for this post would be phred, but I cannot create a tag.
No. That is MA smoothing. MA models are discussed in http://www.otexts.org/fpp/8/4
Is the definition given in [Wikipedia](http://en.wikipedia.org/wiki/Singular_value_decomposition#Reduced_SVDs) sufficient for your purposes?
+1, great question. I'll be interested in seeing what people have to say.
A common mistake. Most definitely $E(f(X)) \\neq f(E(X))$ except in a few cases (e.g. $f$ is a linear function). See Jensen's Inequality for more info on that.
Cross-posting is not encouraged at all. Please, consider posting your question on one site, wait for few days, and ask for migration if you feel you'll get better responses on another SE site.
90% confidence of what?
Sounds interesting. If you want to establish that there is no significant difference, then the power of the test becomes more important, as it is the frequency with whch the test will reject the null when it is false.  If you don't have very much data then it is likely that you will not be able to reject the null not becuase there is evidence that it is false, but becuase there isn't enough evidence (data) to argue convincingly either. I think I would include it if there were good biological reasons why it should be relevant regardless of the statistics.  HTH
@Anand One way is to observe that $x = 2\\sin(\\arcsin(u)/3)$, allowing direct calculation of $x$ in terms of $u$.
Well, you could always simulate some datasets to study their performance over a parameter space that matches the one you find in your data?
They're used for comparing non-nested models with different numbers of variables, yes. Indeed, I'd say that is the situation in which they're usually used - since if you had non-nested models with identical variable counts you could just compare likelihoods.
If you're going to invent a new control charting procedure, then there are more fundamental questions to ask: you need to ascertain the quality control analogs of false positive and false negative error rates. These are measured by the expected lengths of in-control runs when the system is in control and out of control, respectively. So I don't think it's productive to ask whether you can "do that;" instead, think in terms of how to achieve good run length properties for your procedure so that it works as intended.
Good point, did not think about it that way
Maybe this answer here helps you a bit: http://stats.stackexchange.com/questions/46022/libsvm-cost-weights-for-unbalanced-data-doesnt-work/47980#47980
I don't think it is perplexing.  It looks fairly linear. There is a lot of variability which is why R square is low.  I think the only way to you could reduce the variability would be to find anothe explanatory variable.
+1 Thanks! (1) Why "if the variance is too large, once you get stuck on one local maximum, you'll more-or-less reject values until you find other regions of maximum probability"? (2) "If you happen to propose the value at the MAP (or a similar region of local maximum probability which is larger than the others), with a large variance you'll end up rejecting almost every other value", do you mean the proposed point happening to be at MAP is very likely to be rejected at large variance case? Since it is global maximium, isn't its acceptance probability always 1 regardless of current state?
@EngrStudent I have done so now, but along the way, I figured out the "right" way to do it.
This will not be possible: the even moments of a non-negative measurable function are either (non-negative) finite or (positive) infinite.
I think what OP means by "rotating" is rotation in the sense of factor analysis, which [can be done after PCA as well](http://stats.stackexchange.com/questions/2742/on-the-use-of-oblique-rotation-after-pca).
See [this related question](http://stats.stackexchange.com/questions/68377/getting-the-inflection-points-from-a-density-plot?rq=1)
Actually sequential testing without correction *is* cheating. You need to apply any method to correct for multiple testing in order to keep overall type 1 error low. In oncology, such  designs are quite common.
@user Consider two specific readings per patient, one on a scale from 0 to 10 and another on a scale from 0 to 100.  If a patient with readings of (10, 50) is as healthy as a patient with readings of (5, 70), then 10-5 = 5 units of the first reading have been traded off for 70-50 = 20 units of the second.  The questions to answer are (1) are the trade-offs constant across levels; e.g., is a (0, 90) also as healthy as a (5, 70); and (2) does the amount of trade-off depend on the levels of the third, fourth, and fifth variables?
@Jonathan: "A few of the category combinations of the calibration data have not been covered by the survey." What I want to say is that there were strata for which no observation has been made, e.g., the survey did not have any male unemployed respondent of age 55.
What is "SRS"? And what is x?! Could you consider rewording a little bit your question? I don’t get it at all. Please consider formalizing it with math notations.
$\\text{cov}(AX) = A\\, \\text{cov}(X)\\, A^T$. You have written $\\text{cov}(AX) = A\\, \\text{cov}(X)\\, A$, which isn't even conformable. If you fix that you get a nice cancellation. This looks like routine bookwork and should carry the `self-study` tag (which helps notify prospective answers to give hints and guidance rather than just do the whole problem for you).
@kwak: Could you please delete your comment and post it as an answer so it can be accepted?
Thank you. I'm studying the material and topics as we speak. However, one question: Let's say that I have found a ARMA-model that iI'm happy with. Will I be able to deduct how changes in the independent varialbes will affect the outcome (numer of visitors)? That is, similarly to say, what a normal regression model would.
I certainly do not understand the answer: the $y$'s are certainly not identically distributed, so cannot behave as the outcome of a random generator...
Where is $n$ in the problem statement?
Thank you very much for this helpful answer!
Are you talking about H1: μ > 1.35 then P(Z ≥ z) and H1: μ < 1.35 means p-value is P(Z ≤ z)? but this assumes H0 = 1.35. The question I'm asking has H0 with inequality too! Are you saying it doesn't matter? (Both H0 and H1 have inequalities in my question!)
Please note that the out of pocket costs are not exactly known at the time of insurance purchase. However, premium is known. So I think we should look at the premiums and see how much of difference they make. Something like what mentioned in the comments to my original post. Or maybe we should consider both out of pocket costs and the premium together. If we do that then we have to solve one equation with two unknowns!
Hey - I'm just getting to grips with much of this stuff so not familiar with many specific tests. In practical terms if you build an ARIMA type model, say in R, the object will have the residuals in. In theoretical terms, the process variance Var($Y_t$) will depend on the particular model being used. Var($e_t$) is always going to be a generic $\\sigma^2$. Personally I would associate 'residuals' with the empirical leftovers.
@Brandon A subtler point revealed by this conversation is that the effects are not independent: they are linked by the size of the market.  Thus, *given* that the market size is practically unchanged, a decrease in the leader's share *necessarily* leads to increases in the shares of at least one competitor.  This calls for a different model and a different test, as suggested by @John's comments: it would suffice to demonstrate that the leader's decrease is unlikely to be due to random market fluctuations or sampling error.  A better model would also account for changes in market size.
Is the level of censoring similar in both subgroups?
It's numpy, with an "n".  It's a dependency for scipy, which the original poster explicitly mentioned in the question, so I would have assumed he/she would have already known about it. BTW, if you are using numpy/scipy, I would recommend working in Python 2.7.x rather than Python 3.x, as numpy/scipy is not yet well-supported in Python 3.x.  Numpy and scipy are not a part of the standard Python library, however, if you just want to try it out, [this distribution](https://www.enthought.com/products/canopy/) of Python 2.7.x installs easily and has a copy of numpy/scipy already bundled together.
Alecos, Many thanks for your post. Yes, most researchers use exogenous variables i.e. geographic, climate and other related variables as instruments for economic variables. However, some do use lagged values of **X** as instruments of current values of **X**. Given that lagged values of **X** correlated with current **y** and uncorrelated with current **u**, can they be valid instruments. Many thanks in advance!
This is generally a good, useful answer, but your first sentence is wrong:  This is not the correlation among random effects (the random effects in this model are assumed to be independent), but rather the estimated correlation between the *fixed-effect parameters* `(Intercept)` and `GroupTreatment` (abbreviated as necessary).
Did it provide any standard deviation?
@whuber I don't think Ganesh is trying to "model". Rather, he is trying to "estimate". Simulation is a perfectly logical solution if estimating something is intractable in closed form. As I mentioned in my post, the problem definition is incomplete. The user should first create a realistic simulation and see what variables are available, sampling frequency etc.
@antecessor Just compare the returned p-value with your criterion. If the p-value is higher than your criterion, the two samples do not differ significantly.
Also, it's interesting that `glmnet` includes a `multinomial` loss function.  I wonder if this loss function could be used in other algorithms in R, such as `ada` or `gbm`?
It would help if you told us some more about your applications!
Ahh. I suspected that and wanted to get an expert's opinion. Thank you very much. Notice another feature that there is an almost perfectly linear top boundary on the transformed data?
You have what's called "repeated measures." There's a good long explanation in the answer here: http://stats.stackexchange.com/questions/35590/linear-regression-with-repeated-measures-in-r
Do you have *any* substantive knowledge about these variables? There is a very large space of "possible models" for bivariate data, but chances are there is a much smaller subspace of "reasonable models" for *your* bivariate data.
@Figaro I believe it greatly reduces their affect.  http://www.ncbi.nlm.nih.gov/pubmed/16711760
@whuber, if $y^{\\star}_{i} = \\beta_0 + \\beta_1 x + \\varepsilon_i$ where $\\varepsilon_i$ follows a logistic distribution then $y_i = \\mathcal{I}(y^{\\star}_i > 0)$ conforms to the model $$ \\log \\left( \\frac{P(y_i = 1|x)}{P(y_i=0|x)}  \\right) = \\beta_0 + \\beta_1 x$$ This follows directly from the form of the CDF of the logistic distribution. You can also verify this with a quick simulation: `x <- rnorm(200); y <- (1 + 2*x + rlogis(200))>0; summary(glm(y~x,family="binomial"))`
@lcl23 If I understood the situation correctly, imputation makes no sense: your "missing" data aren't missing; they indicate no loan has been taken out.
I'd just like to reinforce the point about the money (for projects where there is little novelty in the statistics)! The other point I'd make is that for projects that do advance the statistics, make sure you put as much effort into helping your collaborator in publishing his paper in stats journals that you would want from him in publishing the primary result in a biology journal. I know from painfull experience that this does not always happen - I have had a couple of projects where the stats paper went unpublished as my collaborator lost interest :-(
Faraway's Extending the linear model (...) has some GLM diagnostics in 6.4 (page 135).
No clue what u wrote.
I don't think so. But the code can be found [here](https://gist.github.com/4045717).
Using the notation of that page, $D_n$ is often regarded as 'the test statistic' (as shown [in this image on the same page](http://upload.wikimedia.org/wikipedia/commons/thumb/c/cf/KS_Example.png/300px-KS_Example.png)), but in the discussion you quote, $\\sqrt n D_n$ is being used as a test statistic. Obviously the two statistics would be equivalent (lead to rejection/non-rejection of exactly the same cases).
Ok - then perhaps the real question is by what coincidence does modeling k successes + r failures in Bernoulli trials follow a gamma Poisson mixture? Maybe a negative binomial modeling k successes + r failures can be thought of as an overdispersed Poisson dbn due to the many possible permutations of success and failure trials resulting in the exactly k observed successes and r observed failures, which can be described as a collection of separate dbns?
This version of the question is asked (for the case of $4$ rather than $10$ digits, but it's the same problem) and answered at http://stats.stackexchange.com/questions/26988/probability-of-finding-a-particular-sequence-of-base-pairs.
For your original question it is best to stick with odds ratios as these have meaning without specifying the settings of other variables.  But interpretation of individual coefficients (and their anti-logs) only works in the special case that the model is coded in a special way and that no nonlinearities or interactions are included.
@Rachel: aaah, I see. Any ward implementation has to calculate the distance between cluster members and the centroid. Intuitively it is clear the metric used for this should be equivalent to the metric used to calculate the distances between observations... hence matlab requires an euclidean distmatrix. But now the question arises why implementations do not request a function instead of distance matrix ? How much damage is done when one uses different metrices for both tasks ? I admit, I don't know it right know.
Both. I'd like to drill down to each question, aggregate questions by category and provide an overall average.
Since the quantity of interest is *additional* time required to a task, we might expect that the lower bound of this variable is zero (we can't go back in time). The gamma distribution is often used to accomplish this, of which the exponential distribution is a special case. But there are many other distributions with support over positive reals. Conversely, the normal distribution has support over the real line, so **any** choice of parameters will always have positive probability below zero.
If I needed to test for lognormality without pre-specified parameters, I'd take logs and run a Shapiro-Wilk test. Why do you need to replicate the 3.1? Why is that more important than other considerations (e.g. any concern about this being a pointless exercise)?
I wouldn't be thinking about p-values in this setting. The biggest danger is using some method you think should give you confidence in your model, rather than using the data as it is - exploratory data.
Awarded bounty for the mention of Rasch analysis, which I wasn't previously familiar with.
Thank you for your response. Yes, I tried this function, but it gives me raw lists of comparisons. What I would like is to see them grouped like in the image in my question, to have a clear view of which group differs to which group, and eventually add the group names on my graphs (for example: a, ab, abc, bc, c)
In that case you probably do have MCAR data. The article I linked to discusses Bayesian approaches to this; I honestly don't know how helpful it will be... Good luck!
If you could anticipate the jumps in VXX you'd be the richest man on the planet.
Many thanks @Kasper!
I would be interested in an answer.
Wouldn't this question fare better on SO?
See, for example, [here](http://home.online.no/~pjacklam/notes/invnorm/) or [here](http://www.jstor.org/discover/10.2307/2347330)
@cardinal: CW? What does that mean???
Here's a good way that you might justify a kernel choice in a report. First - fit 2 or 3 different kernel functions that you might think are reasonable. Second -calculate test statistics of interest such as sample autocovariance at different distances on the original data. Next - simulate time series data repeatedly from each of the fitted models. Apply the same test statistics to the simulated data. Make histograms of the test statistics and see where the test statistics for the original data lie in the plot (or calculate some posterior predictive p-value) Finally-write up the results.
Hi Gael, thanks for your answer. It turns out "tapply" does what I want by tapply(df$value, df[, -3], mean)
@Zen This does not sound Bayesian at all: in Bayesian terms the question asks *how to estimate a prior distribution* given a collection of experimental outcomes. Alecos: The quibble over whether or not a Normal distribution is appropriate is just that: plenty of good analyses assume Normal distributions of quantities that have intrinsic bounds. We all know that an assumption of normality is a convenient *approximation* and does not include an implicit assertion that all real numbers are possible. A Beta might be more convenient here, but that is not clear without an analysis of this problem.
I am familiar with your work via the writings of Shalizi & Gelman. I will definitively follow the blog. And yet I wonder, is Gelman's "Bayes" the same as DeFinetti's "Bayes"....
Do you have any evidence that the last 215 of those 275 aren't just artifacts from measurement variability ("noise")? For that matter, what evidence do you have that *any* of the principal components are meaningful?