tttjjj/elt_bench huggingface dataset:
@mbq - the Cauchy distribution presents no problems for confidence intervals, as the CDF is the inverse tangent function.  Variance need not be defined for CIs to work.  And the ratio of two normal RVs with zero mean is Cauchy, but not necessarily two normal RVs with non-zero mean.
The rainbow/jet colormap is generally a bad idea. http://www.jwave.vt.edu/~rkriz/Projects/create_color_table/color_07.pdf
The difficulty here is that unless both $a_1$ and $a_2$ are integers, one cannot be sure that $S_2$ takes on integer values only. Thus, you need to find not just $P(S_2 = k)$ for integer values of $k$ but also $P(S_2 = \\alpha)$ for each $\\alpha$ that can be expressed as $a_1m + a_2n$ for nonnegative integers $m$ and $n$.
Could you provide a brief description of this textbook or indicate what it covers?
Random variables may be classified as *discreet* if they don't draw attention to themselves. If they're merely countable we say *discrete* :-P Also, you mean prescribe rather than proscribe, but I think *describe* might be more appropriate. Nice answer, anyway -- hopefully +1 will help mitigate the nitpicking!
verify with the package maintainer whether that's old documentation, I think random slopes might have been added (although the documentation does say they're not there as of Sept.).
You can keep them in the model. A more thorough answer will go into greater detail what exactly they contribute to the model though.
BP test requires that variables on the right hand side should be exogenous. Since you have two prices, they may influence each other. Another feature of the prices is that they are usually unit-root processes, that is usually a no-no in simple regression tests. I should check these two things before investigating further.
+1 for actually providing an answer :)
Thanks, that corresponds to my experience. Can you provide a quotable source for your comments? ('the theory underlying these results is assymptotic for $p_i\\rightarrow\\infty$')
Even if you include all interactions among the IVs, the regression analysis will only involve $16$-dimensional matrices (and their coefficients can be calculated in a fraction of a second using compiled code, and perhaps a minute on an interpreted platform like `R`: my machine takes 16 seconds for the simpler case with no interactions and that is primarily because `R` creates and saves a copy of the data). If you randomly select, say, a few thousand observations for regression, that might be good enough to tell you everything you need to know before you spend time computing with all the data.
@Hugh that's great ... as long as you don't care how many random values you get.
Does every location have the same importance to you? I am wondering if a weighting scheme might be appropriate.
@Pinouchon It would be constructive to edit your question to focus on the belief network aspect.  The Monty Hall problem itself has been discussed to death in many, many places, so I see no point in rehashing that material here.
Did you design the questionnaire to detect people who pay no attention to the questions? For instance, questionnaires typically include several questions that are asked in several different ways so that a *consistent* respondent will not provide *identical* answers to them all.
IMHO the origin is a problem of the bar charts, and doesn't have anything to do with the logarithm. You can reach the same deceiving impression with linear axes.
I'm not sure that I understand your questions here. To do repeated CV, there is `method = "repeatedcv"` that handles the details for you. Basically, it pools all the hold-outs from all the CVs so you get one final estimate of performance and a recommendation for which tuning parameters to use.
"The interval contains the true value" and "the true value lies within the interval" mean exactly the same thing. It's more helpful to think in terms of the former but it doesn't really make sense to say that one is correct and the other incorrect.
"Classifying news" doesn't provide any more detail than what you had in the original question. What are you trying to predict? Topic (politics vs sports vs entertainment vs ...), time (is this article old?), news vs non-news, etc. Again, my guess is that word choice is more informative in your application than meaning. It could also be that you're adding synonyms that aren't actually synonyms in the context. A more principled approach you might try is [stemming](http://en.wikipedia.org/wiki/Stemming).
Hmm, in that case i think i'd keep it in. I can't see why it makes the explanation harder, and as you've discovered it may be harder to explain why you dropped it than why you kept it in!
This question does not appear to be about statistics within the scope defined in the help center.
[tag:statistical-significance] is well-defined and easy to look up. Practical significance: [not so much](http://stats.stackexchange.com/q/30118/32036).
Thank You for valuable answer. I am also looking for a business analytics project. Can you advise me where can I search for it?
Wow, this is something that will be even harder do obtain I'm afraid; my general point in the whole discussion is that the convergence of those theorems is too weak so the difference may emerge from random fluctuations. (And that it is not working for machine learning, but I hope this is obvious.)
"if y(m) takes values with equal probability,shouldn't it have average value equal to p=1/2?"  OK, let's start with something very very basic. **What** formula do you use to calculate the average value of a random variable $X$ that takes on values $x_1$ and $x_2$ with probabilities $p_1$ and $p_2 = 1-p_1$ respectively? You might find it in your book in a more general version that talks of $n$ values $x_i$ having probabilities $p_i$ (where $p_1 + p_2 + \\cdots + p_n = 1$), and so just set $n = 2$ and write it out. You might then want to substitute $p_1=p_2=\\frac{1}{2}$, $x_1=+1,x_2=-1$ ...
I never used any non-parametric model for real (i.e., beyond theoretical classes), so it's beyond my expertise. But my impression is that a non-parametric model is in general more complicated, and I guess so with multilevel models (probably even more complicated). Try with the parametric one and see if it fits the data. If not, then you may think to move on to more complicated things. But, again, I'm no expert on non-parametric stuff.
I am not expert in either of these... Others here may know.
@DavidHarris I think you have a nice solution if your assumptions are right. But I am worried about identifiability for each person with so many workers.  The OP thinks the large sample size helps.  But there needs to be structure like what you gave and some modeling assumptions.  I just don't think he has specified everything we need to solve the problem.
Related threads (to name a few): [Group differences on a five point Likert item](http://stats.stackexchange.com/q/203/930) (#1), [Under what conditions should Likert scales be used as ordinal or interval data?](http://stats.stackexchange.com/q/10/930) (#2), [Visualizing Likert responses using R or SPSS](http://stats.stackexchange.com/q/25109/930) and [Visualizing Likert Item Response Data](http://stats.stackexchange.com/q/3921/930) (#3).
$0$ is not the mean for the 4cyl. The intercept is the mean for the 4cyl engine. Also, you should perhaps write p-value instead of 'PV' in your question just for clarity. Also, as I said in my answer, you cannot look at the individual confidence intervals to conclude anything as to whether the two parameters are different.
I suggest to change your question to something like "How to calculate the maximum number of equidistant probability values from 0 to 1 that result in unique sample quantiles". That would make it a better fit for CV and is an interesting problem.
@PeterFlom I was suddently curious about finding the best way to make a exponential curve without destroying the meaning of the correlation. Thank you!
Claudia, you have asked a question upon which textbooks have been written.  It might be wise to search this site for validity and surveys. There are some incredibly thorough answers from which you would benefit.  If you still have questions after that, you might be able to ask a question for which there is a shorter answer. Good luck!
Wolfram Alpha solves this problem [magically](http://www.wolframalpha.com/input/?i=Expand[%28t^2%2Bt^3%2Bt^4%2Bt^5%2Bt^6%2Bt^7%2Bt^8%2Bt^9%2Bt^10%2Bt^11%2Bt^12%2Bt^13%2Bt^14%2Bt^15%2Bt^16%2Bt^17%2Bt^18%2Bt^19%2Bt^20%2Bt^21%2Bt^22%29^5]+up+to+30).
@Frank49, glad to help. No insult taken.
@Robin, I would prefer to keep it more general. If, in your response you're providing a method that requires an assumption about the size or nature of data, please state it. The problem I'm having spans a number of different modelling tasks, all with different data. So, in this case, I'm looking for general recommendation on identifying interaction effects.
Why are you testing out various lag effects? Why aren't you trying to IDENTIFY the model using the Cross Correlation Function? The Box-Jenkins process is to Identify, Estimate and Forecast.
(+1) I'm not sure I understand the reason for the downvote on this question; it is well-formed and the question of interest is pretty clearly spelled out even if it's apparent that the OP is struggling to find the right terminology.
If you don't like my link function above, another robust link function is $1/2 + arctan(x)/\\pi$.
I had to look HTM up: Hierarchical Temporal Memory ( http://en.wikipedia.org/wiki/Hierarchical_temporal_memory )
By "resample" here, I mean take one every $k$, not actually random resampling
I doubt there is any formal definition particularly for "data analytics", the words will just mean however people use them, & many people will use them in different ways. Your best bet for understanding the range of typical meanings it at the linked thread.
Person 2's approach does not even yield probabilities that sum to 1 (3 heads and 1 tails is also 0.0625), thus it is not explainable by any dependency between the flips.
I don't follow: since you know how to apply the delta method to vector-valued random variables, why not apply it directly to the expression for $s_1-s_0$, which explicitly is a transformation of the vector $(s_0^*, s_1^*)'$?
Last time I checked, "research" does not require a control group.
For a spherically symmetric distribution you *are* speaking of just the marginals since they are all the same.  When you rotate a distribution as you describe, note that the marginal will not be the same as the radial distribution.  For instance, the radial distribution for the multivariate Normal is Gamma.  Nevertheless, that approach of prescribing the distribution in spherical coordinates will work when you choose a suitable radial distribution to begin with--but you don't want it to have a mode at zero.
No, the size of the population, once it gets "large", doesn't matter, only the size of the sample.  As the sample size gets larger, the distribution of the T-statistic approaches the distribution of the Z-statistic (the t distribution approaches the Normal distribution), so at some point people just use the Z-statistic.
(+1) This looks like a valuable contribution to me, because it is a novel idea.  I asked a [similar question](http://stats.stackexchange.com/questions/1781) long ago that is strikingly similar: how do you estimate a covariance matrix when data are *censored* (instead of missing)? If you have any thoughts about that situation, I would be glad of a reply!
Also: Where did you get the idea from? Is this formula used elsewhere?
+1, welcome to the site, @Thomas Lumley. We hope you'll register your account and come back in the future.
The metric will depend on the purpose of the regression.  Typically, though, because most regressions work by minimizing something (like a sum of squares of residuals or a negative log likelihood), you usually would choose a metric similar to that objective function.  Otherwise your regression probably is inappropriate.
Each of your principal components will be a linear combination of your features, e.g. $PC_1 = \\sum_{i=1}^n \\alpha_i x_i$, where $x_i$ is the $i$th feature vector and $PC_1$ denotes the first principal component.  If you look at the weights $\\alpha_i$ for each feature in constructing each of the 3 principal components you've used, you should get a feel for which features might be more relevant in predicting $s$ (the ones with the largest $|\\alpha_i|$).  You should be able to extract the feature weights in each principal component easily enough in R depending on which library you are using.
[Cross-posting](http://meta.stackexchange.com/q/64068/150510) on multiple Stack Exchange sites is not recommended. (It appears that you also cross-posted several questions on MathOverflow and math.SE.) I will close this question as a [duplicate](http://math.stackexchange.com/q/321615/26851) then. For future reference, you can ask math.SE moderators to migrate your questions for you.
Can you provide more information on this in the question? For instance the values of the tests. If it is only the Pm statistic which gives a different result you can discard this one because it is for large T, large N (you only have large T). Also p-values below 0.1 are fine, i.e. rejection of the null at the 10% significance level. Also try the "pperron" option instead of dfuller because the Phillips-Perron test is robust to serial correlation (if you have serial correlation in your data).
This doesn't sound like a programming question. It sounds like you need advice on statistical methods for comparisons of clustering methods. This may be a better fit for [stats.se].
(continuing)   but they have a __distribution__ in the population of individuals. __uncertainty__ and __variability__ is not the same, which do you mean? Another point: you might use the Moore-Penrose generalized inverse in the Mahalanobis distance, but when $Q=0$, that is also zero so gives distance zero. Bit if $Q$ merely have reduced rank, but nonzero, this gives the usual Mahalanobis distance in the subspace where the operator is acting! If all your data lives in that subspace, that is fine. If the data have components outside that subspace, this component is effectively zeroed.
Good answer, thank you - please see my update above
Please read advice in the Help Center on software-related questions.
That's my question, but not the duplicate. This is a different question arising from the question, new kind. I reframed it to be considered as a new question.
So we just do the t-test with the weighted means and weighted standard deviations, where u1 and u1 are the weights?
Thank you so much Rob. I really appreciate your prompt reply.
From Monteverde's answer below, I did the Fisher transform wrong, it should be 'z=atanh(r); tanh(c(zp-1.6448*se, zp+1.6448*se))'.
Yes, you should do multiple regression. But since your dependent is a proportion, you should probably use [beta-regression](http://cran.r-project.org/web/packages/betareg/vignettes/betareg.pdf). I don't think you should try correlating the dependent against all explanatory variables independently.
Also I don't think the OP wants 3 specific months, he likes to know all possible combinations of 3 different months.
I fixed your captions, so if you could explain what your feature extraction procedures were, everything should be ok for re-opening your question.
My answer seems to pertain to unit root test like Dickey-Fuller.  It appears that the KPSS test is not a test that the series is stationary but rather that the residual from a deterministic trend is stationary.  These two test clearly look for different aspects of nonstationarity.
@whuber, I'm not quite sure what you mean - I have 30 observations for each of the instruments. I'm assuming that the instruments operate with random Gaussian error
(1) if you're considering using NRI, would strongly encourage reviewing recent review: http://annals.org/article.aspx?articleid=1814428 (2) you've overfit your model. you have to address that in some way that is convincing (3) this might also be worth reviewing: http://www.bmj.com/content/344/bmj.e3318
If in the last 2 minutes, there is 50 cars passed, then I can say make the green light last for 2 minutes. on the contrary, in the last 2 mintues, there is only 10 cars passed, I can make the green light last shorter, say 1 minute. is it simple to understand, why can't model it
Looks like the thing they're computing is a multiple of a sample correlation. If those terms are zero, the off-diagonal elements of the sample correlation matrix will be 0.
Damien, because that kind of personal communication circumvents the purpose and mechanisms of this site it is *strongly discouraged.*
@kwak: Yes, that sounds like what I am looking for. I would greatly appreciate those links. :-) I am working on Mac OS X… I can use R for post-processing data, but can't link GPL code into my company's product for the usual reasons.
@mpiktas As I said ADF is about the mean equation, but this is not what I am pointing at.
"Regression perform well over continuous variables and Random Forest over discrete variables.": 

This is not true in general. There are distinctions in inference and prediction, and even with prediction there are cases when regression would be preferred over RF.
thanks for such a good solution.
your code prints out a bug 'loglik.negbin.perageclass(starting.values)' yields 'Error: object 'C_a' not found'. Correct please.
You don't quite define a probability distribution: you need also to specify a distribution for the $x_i$. When you do, that will determine a *single* univariate distribution for $f(x,y)$ which can then be tested.
Another rule of thumb for multiple linear regression is *n* = 50 + 8(*p*), where *n* is sample size and *p* is the number of predictor variables, so I completely agree with @Peter Flom to perform univariate analyses, and leave it at that.
Sorry, that was a typo.
Welcome to the site, @sxlee. If you can post the image to anywhere on the internet, & edit your Q to add a link to the image, a higher-rep user can post it into your Q for you.
I have tried excel's data analysis but the output is not validating against the data. Yes I indeed want to understand how response time behaves so thanks for making it clear to me.
I edited my reply with what you are looking for (or so I believe!)
No, they are not identical! as long as $\\binom{n}{y} \\not= 1$, they are different, but proportional.
(+1) But is it not intuitively obvious that given a collection of wallets of various amounts of cash (modeling the non-negative eigenvalues), and a fixed number $k$ that you can pick, that selecting the $k$ richest wallets will maximize your total cash? The proof that this intuition is correct is almost trivial: if you haven't taken the $k$ largest, then you can improve your sum by exchanging the smallest one you took for a larger amount.
@drknexus Your intuition is correct.  However, extremes (and near-extremes) of sampling distributions are somewhat special: their values are constrained by the mass of data on one side of them, whereas--for parent distributions with infinite tails--there is no constraint at all on their values on the other side.  Thus, for instance, the distribution of a maximum (from a distribution with no upper bound) is positively skewed.  This *increases* its expectation relative to the corresponding percentile.
What is it you want to compare about these distributions, that they differ in central tendency, or that they differ in shape? I tend to think of KS as being more about the shape / nature of a distribution, but something like the [Friedman test](http://en.wikipedia.org/wiki/Friedman_test) can determine that the samples differ in central tendency.
Try applying Little's law: $L = \\lambda W$.  You know $\\lambda$, and all you need is $L$.  Can you get that?
@Anne [This](http://en.wikibooks.org/wiki/Using_SPSS_and_PASW/Confidence_Intervals) describes how to compute a confidence interval in SPSS. I do not have SPSS so I cannot check it.
This is the first hit on a [google search of your title](https://www.google.com/search?q=Conditioning+on+the+margins) :-).  The [second hit](http://stat.ubc.ca/~rollin/teach/536w09/exactOR.pdf) (and presumably, many of the later ones) currently have answers.
Not a representative sample of what population, exactly?  This is not a question about some random interviewee: it's a question about "you."  As such it invites us to consider which data are relevant to "you" and to what extent they might be, but your vague use of "representative" just dodges that altogether.
Sounds legit. You can just report the effects on this factor and ignore the results. Alternatively, you could run a multivariate test.
@ciri you can find a free implementation of the approximation I referred to [here](https://github.com/claesenm/approxsvm).
It was a mistake, see my edit. Thanks for noting it!
Deviance residuals $\\text{sign}(y_i−\\hat{\\mu}_i)\\sqrt{d_i}$ are approximately normal, or at least as approximately normal as you're likely to get.
Are you referring to Ehler's supersmoother? If so, there is code for this on his website at http://www.mesasoftware.com/seminars.htm Although this is not Python code it would be very simple to rewrite.
Any reasonably (sensible) CV code would randomly assign samples to the 10 groups so ordering in the data would be irrelevant.
Nice question. One approach to this I might also make, is to do a literature search in journals in your discipline for "path analysis" (and perhaps "MPlus" also)... see how others have presented their results.
If someone could provide a translation of the German, that would be nice. I for one don't read German.
(+1) for good explanation
@Tomas, if you do that, the estimated strength of the relationship depends on how you've decided to label the points, which is kind of scary :)
@Andy The answer depends on the application. For instance, a survey (like this one) intended to characterize small subpopulations (which might selectively decide not to respond) would need very close to 100% participation. (This is the basic problem encountered by the US Census Bureau.) The only reason one could justify a 20% response rate would be with evidence (such as past experience) that the nonrespondents do not differ from the respondents. Famous examples exist of [what can go wrong otherwise even with high response rates.](http://en.wikipedia.org/wiki/The_Literary_Digest)
I know nothing about the package... But Jaccard is a popular coefficient of association for binary data. It is like, say, a correlation coefficient for continuous data.
+1 but Parametric IRT models make strong assumptions about unidimensionality (not obvious in the question) and local independence. Anyway, in this particular case, I would favor some kind of explanatory IRT model, like the LRRM, as explained in de Boeck & Wilson's book: person-specific covariates can be directly tested within such a model, no need to consider weights of any kind.
The product term is still missing.  A set of sufficient statistics for the problem are $\\mathbf{y'y}$, $\\mathbf{X'y}$.  I don't see the second (vector) statistic anywhere.
+1. Small note: $\\sqrt{\\frac{e\\pi}{2}}\\text{erfc}\\left(\\frac{1}{\\sqrt{2}}\\right)$ equals $0.6556795424\\ldots$ for me which agrees with @fabee's numerical answer.
To clarify: your null is avg difference = 5mmHg, and your alternate is avg difference <5mmHg? Is that what you mean by 'no greater than' or did you mistype?
your function is not differentiable everywhere.
If a particular textbook doesn't have a derivation, it almost certainly has a citation you can reference. For instance, section 4.4.4 of ESLII (which you highlighted) contains the following references: Koh et al., 2007, Park and Hastie, 2007, Friedman et al., 2010, Efron, 1975. Check the back of the book for the full citations these references are describing.
@Chl..Thanks a lot for that. I just want to get it clear, so the Bland-Altman plot will be the solution...
Have you computed and plotted the residuals?  Also, concerning `linktest`, have you read through the manual page (r.pdf) on it?  Example 2 is especially illuminating.
This question appears to be off-topic because it is about programming.