tttjjj/elt_bench huggingface dataset:
This probability plot does not exhibit any outliers at all, but it does indicate a region of sparse probability density around 40-60.  This will not be detected by most GoF tests, which will be invalid anyway.  Distribution fitting is *exploratory*, not confirmatory, and so requires more flexibility and creativity than is afforded by parameter estimation and rote application of hypothesis tests.  If a mixture is suggested by theory--and it might be here--and if capturing that in a model might be important, then it is worthwhile considering despite the complications that ensue.
Thanks once again! I will try the Beta regression. But one question, can beta regression handle values of 0 and 1 exactly?
If you think of $\\frac{z^2}{t^2(n-1)}$ as a constant, $g$ (it varies with $n$ and $\\alpha$, which are constants), then $\\Lambda \\sim g\\chi^2_{n-1}$. Asymptotically, $\\lim_{n\\to\\infty} \\frac{z^2}{t^2} = 1$. So $g\\chi^2_{n-1}\\to ...$ -- it's possible to carry the analysis further, but I don't know whether the intent if for you to do further approximation.
i am using numpy for the the query to the data-mining tool, i heard of scipy ,but i am not sure if i can "define" a function - do you mean by making a fit? I think this is not possible in my case... i have 16 input variables... but i will give it a try in the next days.
what are the questions you hope to answer based on your data? what is the data you would like to model? what is the model you thought of?
What kind of convergence?  IIRC with MI you want convergence in distribution
Empirical Bayes is only one of the possible approximations to a full Bayes treatment.  There are many other approximation algorithms available, such as [Variational Bayes](http://en.wikipedia.org/wiki/Variational_Bayesian_methods) and [Expectation Propagation](http://en.wikipedia.org/wiki/Expectation_propagation).
An addition, I didn't quite understand  'by looking at correlation between outputs at varying delays'? Just try correlate n-th bit of single output with m-th?
Thanks for the edit, @Glen_b
+1  BMI forces a *very specific* and theoretically-driven relationship between height and weight.  $y=\\beta_0+\\beta_1*height+\\beta_2*weight$ is way more flexible (e.g., able to spuriously fit unrelated data) than $y=\\beta_0+\\beta_1*BMI$.  That is, there is a good reason for wanting to penalize the former more than the latter.
(+1) Those are good comments, indeed. (I was thinking in terms of bootstrap when replying, even if this is not the question nor did I resample the same dataset.)
Do you picked the exponential distribution because its pdf decreases smoothly from 0 to infinity ? If it fits (i.e. the error is small enough) it feel that your approach is reasonable, although I do not know any standard recipes on converting probabilistic functions to fuzzy functions.
I knew about Google Fusion Tables from Google Maps' API, but didn't realize there was a free standalone version. Thanks!
Could you, with the permission of the author, post the error briefly as a comment here? That will be useful to everyone.
But what if the two effects have the opposite signs? Likelihood ratios may be an alternative to $p$-values, but I wouldn't recommend meta-analysing $p$-values either unless they're the only thing reported.
Your first method is the same as fitting a regression model with no explanatory variables, just an intercept.
I don't have enough reputation so it seems I can not comment directly in that link.
If you are doing a one-tailed test yes. Otherwise you need to account for the other tail as well.
(+1) I have other references in [these](http://aliquote.org/memos/2011/06/25/cronbach-s-alpha-yet-again) personal [notes](http://aliquote.org/memos/2010/09/22/revisiting-cronbach-s-alpha).
(@Chinook, you have to precede a username w/ the @ symbol for me to be notified of your comment.) So the mark-recapture method is the 'gold standard' here & the original numbers constituted the original y values. Then you regressed those onto something else & got predicted abundances, y.hat, as a refined measure of abundance, is that right? What was the predictor variable in the 1st model?
Stef, this may be a problem if the task at hand is (1) the estimation of the population total, (2) logistic regression that may have empty cells in some imputed sets but not in others. But that's anybody's telling; Rico did not tell us what his analytical purposes are.
Related questions might provide a little insight (but do not directly address multilevel models): http://stats.stackexchange.com/questions/47178, http://stats.stackexchange.com/questions/34488.
How would you treat numerical variables in MCA? Using discretization?
I don't think you are deceiving yourself. I believe that what you are looking at is to be expected as "*simple grouping factors*" are more probable to actually have a real influence on your data than some contrived hierarchical grouping with random slopes etc. This "higher variance" makes sense also because the "finer your grouping becomes" the more probable it is that numerically you are going to fit that variation as "residual variation" (let alone real noise corruption). I am slightly worried/puzzled by your "*the residual term commonly accounts for close to the most variance*" phrase...
I come to think of it that just plain correlation or (adjusted) R^2 are the same for stationary processes?
1) Since you're not estimating the absolute values in your model, why are you using absolute values in the check?  2) If the mean of $Y$ is close to 0 (relative to the std. dev. of $Y$), all this comparison will tell you is that the predictions have less dispersion than $Y$, which is a consequence of the estimation procedure and not worth worrying about.
@MattKrause From a pedagogical viewpoint, it is better to teach students to use the formula in the way that the OP has because it emphasizes that the numerator is just one of the terms that are being added together in the denominator. Indeed, it is even worth emphasizing that it is _not_ necessary to _calculate_ $P(A\\mid B)P(B)$ twice, once for the numerator and once for the denominator, when using $$P(B\\mid A) = \\frac{P(A\\mid B)P(B)}{P(A\\mid B)P(B)+P(A\\mid B^c)P(B^c)}.$$ In years of teaching this stuff, I found _many_ students obtain different answers when doing the calculation twice.
@DrewChristianson - Yes you are absolutely right about which variable is which. Y takes values which are dependent on the process parameters and the position of the tool Xi. As the tool the advances, the value of Ai changes, but the process parameters remain the same. Each Xi are discrete positions of the tool. Xi has values which increase continuously and Ai first increase and decrease. pls get back to me in case there is any doubt.
@ttnphns yep, $1-r$ is most common. It's worth noting that for a given similarity metric $\\rho \\in [-1,1]$ there are at least three formulas for converting to a dissimilarity: (1) Bhattacharyya's method $cos^{-1}(\\rho)$, (2) Kolmogorov's method $1-\\rho$, and (3) Matusita's method $\\sqrt{2-2\\rho}$. This is another area where in $practice$ I don't think the choice usually matters much, and if it did, I would be concerned about the robustness of my results.
Thanks, I'll look at it :)
One way to deal w/ scenarios that are too complicated for preprogrammed approaches is to simulate. You can read about some of the basic ideas in my answer here: [simulation-of-logistic-regression-power-analysis-designed-experiments](http://stats.stackexchange.com/questions/35940//35994#35994). I don't know if this would be beyond your reach at present, you may need to work w/ a consultant.
I believe `rnorm` is vectorized so `x <- rnorm(1000, 0, seq(9, 1, length=1000))` should work instead of using your function.
@David, although it seems similar, it's not, because the ZIP is a model of the *dependent* variable, not the independent variable.
A slightly different interpretation of this question presents an interesting counterpoint. The question is of the form "I conducted *one* hypothesis test (KS) that finds a significant difference, but the *effect size* (skewness and kurtosis) is not large.  What do I do?"  In this interpretation, a good answer to the question would discuss the sensitivity of the (unnamed) analysis to nonzero skewness and kurtosis in the residuals. Unfortunately the OP did not respond to a request for information about that original analysis. IMHO, that makes the question unanswerable--but Glen_b did a good job.
Here is a reference (review in the BMJ) on imputation which recommends not doing that: **http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2714692/**
Surgery vs. blood pressure:  nice example.
Thanks for the help Jeromy!
The Wikipedia page at http://en.wikipedia.org/wiki/Quantile#Estimating_the_quantiles_of_a_population provides a nice table of the various definitions.
The international comparisons I've seen examined intergenerational mobility (dubiously, having failed to control for parental & child ages at sampling time). I think for an agent-based simulation you'd want intragenerational, which I haven't come across.
The samples aren't independent -- they're taken from the same individual, same place, and separated by only about 48 seconds. I'd also like to compare the variance across other conditions with different means, not just time points.
You can create chatrooms, but nothing is private.
This is a bit short for an answer. Can you expand a little on this?
It's probably worth mentioning that any data set can be normalized to 0->1 and made to use a sigmoid activation `1 + (1 / exp(-sum))`. Making the *need* very difficult to understand without trying both on each data set. The *need* as you describe it here is tied to the actual relation being learned, ie a binary data set will learn faster or not at all given different activations.
Also, combining classifiers--especially if they're unstable--will usually decrease the variance, not increase it as you stated.  That's why bagging works, as discussed in the same pdf I just linked.
I am not referring to "1 vs rest" approach. Please see the update on my question post for clarification on the question. Thanks!
The sheer number of methods is a sign of the arbitrariness of a cutoff.  And since it is wholly inappropriate to use cutoffs on input variables, and only appropriate to seek at cutoff (if you must) on an overall predicted value, it is not clear why so much effort is spent on this.  If you set up a Bayes optimum decision rule with a loss function everything gets taken care of; no ROC curve, no backwards-time probabilities such as sensitivity and specificity, no cutoffs on input variables.
Maybe what @marian really needs is Distance Correlation http://en.wikipedia.org/wiki/Distance_correlation
This is not a direct answer to your question, but HMC and NUTS are often used in multilevel models, where you encounter high correlation among parameters at the (many) levels of model hierarchy.
Just to add to @Frank's suggestion on Gini, there's a nice paper here: http://projecteuclid.org/download/pdf_1/euclid.ss/1028905831
It goes over various measures of dispersion and also give an informative historical perspective.
That looks almost identical to how R's combn function is already doing things.  I wrote up a version of combn that does take combinations off the stack one at a time, and as mbq says because it is only storing one combination in memory at a time it can handle very large combinations.  The problem with doing it in R is that doing a step-by-step approach in a function typically involves reading the state variables into the function, manipulating them, then storing them back out to global - which seems to just slow everything /way/ down.
@chi yes i wanted to have one-one mapping between estimated $k_i$ and predefined $c_j$
For the first example which i have stated, i wanted to test the adoption of particular programme by the population, my null hyp  h0: p=0.5 and for the sceond example i wanted to test accuracy of a process adopted by the population so my null hyp: p=.95
(+1) Nice, thanks for the example.
Are you saying that the research that supported a 3 item scale did not hold up when you replicated it?  If so, why not.  What Ns were involved in the two research studies?  In any case, I would not discard a two item scale out of hand.  Do you know if this two item scale correlates with any other variables of interest (i.e., construct validity or criterion-related validity)?
because the question is probably sufficiently challenging that there isn't an obvious standard methodology.  Rather than "I want to use well-known statistical procedure X, is it implemented in R/how do I go about using it?", this is more along the lines of "is there a good statistical procedure for solving problem Y"? Alternately, it might be worth checking out r-sig-finance (I think there is such a mailing list ...)
A lot will depend on how much you can assume your missings are missing completely at random. If there is a high percentage of missings and they're not missing at random, you may get biased estimates for the imputation. Because it has to be done on cases present in the data (by definition), where there is a systematic bias in the missing cases the present cases may not be very informative from an accuracy perspective.
Thanks so much for the reply @ttnphns. According to my understanding of redundancy index, if it is a low value although the canonical correlation is high, it might mean that results are not practically significant. Please correct me if I am wrong. Looking at the article and the book, I feel the value of Wilk's lambda, p-value, F-statistic is good enough to say that the results are statistically significant. The canonical correlation is actually 0.814 and the square is 0.666.
+1 @StephaneLaurent.  Your answer is well written too.  My edit was merely to change "you claim stems from..." to "your claim stems from...".
OK, thanks a lot. So that means the intuition of "fixing" the penalty term contour and "growing" the least square contour is correct, right?
Yes, y1 is the cdf!
Alpha measures the degree of inter-item homogeneity; it is like split-half reliability. I expect that it is within sub-tests only, not throughout the entire questionnaire, that homogeneity should hold. If so, do for each sub-test.
I originally wanted to find a quantity to rank items for recommendation purposes and an average seemed a reasonable idea. I am not sure how I would use that adjusted average for other purposes outside sorting.
My first guess is you've run into [Simpsons Paradox](http://en.wikipedia.org/wiki/Simpson's_paradox) -- e.g. where in this case the distribution of BMIs is not the same across groups -- but there's not enough information given to be sure. Note that you have four variables here, so there are several ways the paradox can come in.
Thanks for your answer. It has been very useful.
I edited the post, is there something else that's not clear? By "operating directly" I mean I want some algorithm which works with probabilities, i.e. 4 successes out of 1000 trials has a probability of 0.004 and a weight of 1000. Alternatively, I can fit unrolled binary data (1 for success, 0 for failure), but I do not wan to go down this path.
Thank you, it is much more clear for me now!
You don't need the distribution of $T$ to work out its variance, just the basic properties of variances.
I would be inclined to say "Past Innovations" assuming that something in the context of the link : [Specify a Regression Model with SARIMA Errors](https://www.mathworks.com/help/econ/specify-regression-model-with-sarima-errors.html) or [Model Seasonal Lag Effects Using Indicator Variables](https://www.mathworks.com/help/econ/estimate-multiplicative-arima-model-using-seasonal-dummies.html) is used. The OP should probably clarify that.
Did you normalize your data before running PCA?  If I had to guess I'd think one of your features was on a much larger scale than the others...
@Chris Can you please explain the difference between the larger `X` and the smaller `x` in F(x)=P(X<x) ? I have a shameful lack of knowledge of mathematical notation.
Are you asking why that formula comes from the gradient of the loss function $L$? The coordinates of the gradient when you just consider the $j$th opponents are proportional to $\\bigg( \\cdots \\bigg)$ because of the formula for $o_{ij}$ and the chain rule.
I've been thinking about this for quite some time. Unfortunately, I have no definite answer yet, so I'll have to think more.
I believe a study of that theorem will pay off here, if only to suggest that what you are trying to prove might not be true--or at least requires some strong additional assumptions about $F$.
thank you so so much, lucky for me my university does subscribe to WRDS.
Also, http://www.amazon.com/Applied-Choice-Analysis-A-Primer/dp/0521605776/ref=pd_sim_b_1
I can't believe that it's possible to calculate a seasonal effect for a data set that does not include each season more than once. Much less a data set that doesn't have any data from roughly three months of the year.
Would very much appreciate getting some expert opinions on this question, thanks a lot in advance!
Not really an anwer, but I found this cheat sheet useful, [Some Useful Distributions in Bayesian Analysis with Models from Educational Measurement](http://faculty.ksu.edu.sa/69424/Primary/Distributions.pdf) (R.J. Mislevy, 2001) -- it mainly covers BUGS distributions.
thank you michelle. I'll just wait a little while before marking an answer
What is your ultimate goal for the clustering? Are you familiar with basic preprocessing and modeling in IR like TF-IDF weights and vector-space modeling?
Thank You for your cooperation. Its means alot I am trying to understand and follow these steps. When I apply this command (arima) on my data, after applying it it gives an error "Error in optim(init[mask], armafn, method = optim.method, hessian = TRUE, : non-finite finite-difference value [1]" can you please explain wt does this error means?
@StéphaneLaurent I think the same model can only be obtained with `lme` as you need to take the nested structure of the data into account. The following model gives the same F and p value for the effect of `Group` and the same residuals: `lme(Value ~ Group, random = ~ 1|Subject, dat)`
'a' is the action you take. In this case it's making a decision to classify a point (say (0,0)) to one of your three classes.
Added the formula for the corrected variance.
@whuber, yes approximate normality is important, but the tests test exact normality, not approximate.  And for large sample sizes that approximate does not have to be very close (where the tests are most likely to reject).  A good plot and knowledge of the science that produced the data are much more usefull than a formal test of normality if you are justifying using F-tests (or other normal based inference).
+1 for Shumway and Stoffer.  My favourite, with frequency domain material, state space models, and R examples in later editions to boot.
Is this a homework problem? If so, please add the **homework tag**. This is discussed in the [faq](http://stats.stackexchange.com/faq), right at the top, which you may want to read.
How to find help on R functions that are not plain text: Enclose them in quotes. Like so `?"<<-"` (plenty of explanation there) ... in fact, if you always enclose the function in quotes, it works, so it's a good habit to be in (compare `?if` with `?"if"` to see why even "plain text" isn't sufficient to avoid needing to quote the argument)
@Robin: No, I am talking about "precision" as measure of the quality of a binary classification model (see e.g. http://en.wikipedia.org/wiki/Sensitivity_and_specificity here). Regarding your second question: I require the scores for a simulation where multiple binary classification models are combined (more in the area of data processing than Data Mining). It is hard to explain the task in details, but I am pretty sure I need these scores :D
Do you want to give a full reference to Hardle, Horowitz et al.? The second part of your answer is actually a new question. Since this is not a discussion list, you should post it as a new question.
Homework? If so, please add homework tag.
If successive observations are negatively correlated the variance of the sum is less than the sum of the variances and the variance goes down faster than for iid observations.  That is like the old saying that errors tend to cancel each other out.  it is not really true when observations are independent but it is the case for negatively correlated observations.  That is because if one observation is high relative to the mean the next observation is likely to be small relative to the mean, a consequence of the negative correlation.
"duplicate" does not mean that you posted the same question twice. It means this thread is asking the same question as was asked there. The words & superficial aspects of the questions differ, but the substantive issue is the same.
"How important is it for you to know when to return to work, or when to revisit the hospital" this is an example of a double barreled question. If you asked it this way you wouldn't be certain if they were implying that knowing when to return to work, or revisiting the hospital is not at all important through very important.
The units of coefficients depend on the units of predictors, and thus so do their standard errors. So, the question is on fours with is 0.05 a long enough string?  Any way, why should there be a _minimum_ acceptable standard error? Are you implying that bigger is better?
Unless you focus your question a bit more on the statistical differences between linear fixed effects and ordered probit this question is likely to be flagged as off-topic. Otherwise, since the main part of it is about the corresponding Stata commands StackOverflow might be the better platform to ask your question on.
Actually, I just realised what you meant. Unfortunately the habitat is not one variable as there is a value for each habitat for each observation. For example one brood may have spent 40% of their time in one habitat and then 60% in another, and the rest would have a 0 value. I wasn't sure if a fixed effect could somehow be built with these separate values, or is there a way of adding them so that their combined effect can be looked at, but still able to give me a relative Estimate for each to see how important each habitat is.
It is interesting that the _maximum_ value of $\\text{var}(X)$ is $\\sigma_{\\max}^2 = (b-a)^2/4$ and thus the result is effectively $$E[e^{tX}] \\leq e^{\\sigma_{\\max}^2t^2/2}$$ which looks far too familiar to be arising out of sheer coincidence.  I suspect there may be another, possibly easier, way to derive the result via a probabilistic argument.
You're right on all this, but this is analysing all of the data points (which I do in the first part of the [script posted](https://gist.github.com/EoinTravers/ce86c93fb42fba284464), and note that they only diverge a little). It's when I exclude the incorrect responses that the two approaches diverge, as in the second part of the script, the results of which are in the original question. Thanks though!
(1) Because $\\int L(\\theta|x)dx = 1$ for all $\\theta$, I believe even the constant in $L$ is defined. (2) If you think of parameters like $\\phi$ and $\\theta$ as merely being *coordinates* for a manifold of distributions, then change of parameterization has no intrinsic mathematical meaning; it's merely a change of description. (3) Native English speakers would more naturally say "likelihood *of* $\\theta$" rather than "on." (4) The clause "when $x$ is observed" has philosophical difficulties, because most $x$ will never be observed. Why not just say "likelihood of $\\theta$ given $x$"?
Have you really got just 4 people, each with 4 treatments?
I would prefer you to write the solution you have got. I would certainly upvote it.
@Orp You can now upvote @Jeromy's response.
I would expect the bottom right hand corner to be $p(y|\\phi,w,\\lambda)$ or $p(y|\\phi,w)$.  I also do not understand what $x$ represents in $y=t(x,w)+e$.
The likelihood function of the sample is the joint density. Here the variables are not independent. Do you know how to apply the chain rule to decompose the joint density into conditional densities?