tttjjj/elt_bench huggingface dataset:
I am interested in computing the sensitivity of the bacteria count to altering parameters such as how often the hands are washed, or how well. In other words a sensitivity analysis. You are right in your insight, the reduction in bacteria is usually done by some form of $log(x-x_wash)$. But I'd like to quantify, not just correlate how effective certain methods of reduction are. Ie more formally than just the log reduction. Does that clarify my question a little bit?
Two comments.  [i]  Don't you need a joint model? Hours absent from school is conditional on not dropping out. [2] This looks like a data mining exercise, not an attempt at getting a causal relationship. If so, you're likely going to get really weak model with just a few variables on the RHS. Other methods are available (e.g., classification tree analysis) that might be more powerful.  Or, you can just add lots of interaction terms on the RHS.
I have checked the distribution now. While it fulfills all of my requirements (at least for certain parameterizations), the CDF and PDF are unfortunately related in such a way that I still cannot obtain a closed-form solution for the equation in my paper (which is of the form $A + B F(x) + C x f(x) = 0$).
Harvey - thanks -- you're right, *SD* isn't normalised at all... I was confusing the concept of `z-scores` for standardizing values and normalising their position within a distribution in terms of the mean and standard deviation, with this problem, which is about being able to rank groups of products in order of their variability.  Choosing your answer as the correct one because while Peter and John's were both very helpful, yours alerted me to the conceptual mixup.  Good point on Choice 1 being of limited use near median 0.  Fortunately, in my problem, I don't have to worry about this.
It sounds to me that in this case we are merely interested in showing the accuracy of, say, the sample mean, not the variability of individual values. Why would you recommend *standard deviation*, specifically?
In case it's not immediately obvious to everyone $1-3\\rho^2+2\\rho^3 = (1-\\rho)^2(1+2\\rho)$
This is true. However, it might not be practical given that we deal with hundreds of thousands of variables.
I can accept that my answer is wrong, but how about explain why, & why yours is correct, in plain english. The problem stipulates the test results are independent. And in my analysis, the "information" one gets from test A result *is* combined with the information one gets from B via Bayes's theorem. This is not an unusual problem--combining results of independent tests w/ known error rates. If I performed 2 drug tests, results of which were independent, results would be combined in the way I have indicated. What is there about the indicated problem that suggests a different approach here?
I was thinking along similar lines. So there would be a new level we'll call it T_2_3 which applies as treatement for the cases having both T_2 ans T_3. So the the two rows become one row: S_2 27 T_2_3 H_2 1
Presumably $M$ and $N$ are *constant* matrices and $y$ is a random vector. Here's a hint: when $x$ and $z$ are independent random vectors, can you show that $x'x$ and $z'z$ are independent?
Hi all. Further to this: I have since been able to use the mice function successfully. I have another query. Say for example the function creates 5 complete datasets (X1, X2...X5). I apply function(x) over each dataset and it returns Y1, Y2...Y5. Do you think it would be OK to maybe report the range Ymin to Ymax? Or maybe the average of Y1 to Y5? Anyone have any thoughts on the matter? Thanks.
"*My research suggests a benchmark value of 80% is a good benchmark*" -- what research is that? Where does the figure 80% come from?
(+1) By defining $b_k = a_k + 1/(1-p)$ you obtain the simpler recurrence $b_k = b_{k-1}/p$ with initial condition $b_0=1/(1-p)$. This is a geometric series with initial value $1/(1-p)$ and constant ratio $1/p$, which makes finding a closed formula for $a_k = b_k - 1/(1-p)$ very easy.
But DBSCAN does not need to know the number of clusters!
Hi CharlesLondon, welcome to the site! Could you provide the model outputs? The regression coefficients are expected to be the same, but what about the standard errors?
@Aniko, I have edited the question to hopefully make the system clearer.  In each year we have observations for ~50 individuals: the day of pupation and whether the pupa is in diapause.  Thanks.
Chen seems to focus on larger datasets, which makes sense because the fourth and sixth and higher moments involved in these tests are going to take some time to settle down to asymptotic levels.  But distributional tests are typically used for datasets smaller than 250 values (the minimum studied in this paper).  In fact, most of them become so powerful with larger amounts of data that they are little more than afterthoughts in such applications.  Or is there more going on here than I am seeing?
Estimating the transition matrix from the empirical transitions is terrible if you need to simulate from the estimated matrix, because this induces a lot of zeros. Bayesian estimates are superior in this respect because no entry is zero...
+1 This kind of well-researched, authoritative, and clear answer is what we aim for in all threads.
(+1) This looks right. Nice answer. You're providing good, high-quality content. Cheers. :-)
@Macro: you are certainly right on both counts (that this is technically possible, and that it is practically irrelevant). The same will be true if you estimate the mean level of an AR(1) process with a negative correlation, but again I am at a loss thinking of real processes that might have this feature. Unlike the positive autocorrelation that is self-reproducible at different time scales, the negative correlation has to disappear if you double the length of your reference period. (The business cycles data, like US GDP, have negative correlations at the lag length of about three years.)
@jonnyboats - I did not think my question was as quantitative as the questions I saw on stats. happy to repost if a few others think it qualifies.
I do apologise for my lack of clarity when trying to speak "statistics".  Your answer is however much appreciated and it would be interesting (and possibly useful for other readers) if we could create another question that requires the answer above.
It sounds as if you should be looking at average outcome as a smoothed function of your predictor. Logit of outcome versus e.g. restricted cubic splines for your predictor is one of many possibilities. Note that nothing stops you omitting values from an analysis if you think that they are irrelevant to your purpose.
Thanks GaBorgulya. You have some great points. I read the documents of ACE and AVAS and they are quite interesting and useful. However, I don't see how some of your recommendations work when we have a model selectoin problem to solve before even knowing what regression problem we have at hand. Using the biology of the problem does not eliminate the model selection in this case since we have already done that and that's one reason many of the predictors are correlated.
Do you have some knowledge about the way parameters are estimated with `lmer()`? It seems that you *postulate* that $\\sigma^2_\\alpha$ is estimated by the empirical variance of the estimated random effects $\\hat\\alpha_s$. The description of your model is not clear (perharps $y_i$ should be $y_{is}$). Is it a balanced design ?
Okay! Thank you for a very helpful reply. I'd vote up, but I haven't earned the reputation yet.
Thanks for the reply but as far as I understand the dependent variable should be approximately normally distributed for each combination of the groups of the two independent variables. Therefore if this assumption is not present (as it was in two of my groups) then a transformation is a valid step (after which normality is achieved in all groups).
@Ivan See the discussion following [this question](http://stats.stackexchange.com/q/19948/6633)
hmmm.. an auto-regressive model isn't really a good fit for a binary time series is it? Auto-regressive on time between events? I hate the idea of variance of time between events.
Hi Srinidhi, welcome to the site. This is a question and answer site. Could you please re-formulate your comment into a question? Also, the more specific a question it is, the more likely it is to get a useful answer.
Neither your question nor your data is clear... I'm guessing your cases are individual people, that two methods of surveillance were applied to every person, sometimes succeeding and sometimes failing?
Shouldn't it be $\\sum_{n = 1}^\\infty$ in the first line?
(1) For the use of `Evaluate`, see the examples for `Plot`.  (2) Emulate this: `Plot[Evaluate[PDF[WeibullDistribution @@ #][t] & /@ {{1/5, 50}, {1/4, 40}, {1/3, 30}}], {t, 0, 1}]`  (3) For tougher questions, please visit our [Mathematica site](http://mathematica.stackexchange.com/).
Just an observation on your particular plot, but the y values appear to fixed at integer intervals. If the y values are known to be accurate you only need regression on the x values. Or to look at it another way, as in Glen_b's bins example, you already have bins. I'm no expert on this stuff though.
I meant "algorithmic efficiency". Although it's true that my questions essentially reduces this to "computational complexity".
@mpiktas If I use the ADF, then I don't see how I can apply my proposed procedure. Under ADF, if we reject $H_0$ in the first step, then we've concluded that $X_t$ is $I(1)$. It then doesn't make sense to test if $X_t$ is $I(2)$ because the first test has already told us that it's $I(1)$?
This question appears to be off-topic because it is about looking for a software recommendation for opening a file type.
Thanks, I've looked at "Whats's the code" already. The survey paper is new, thanks again!
This may need some clarification. Can you confirm that: 1) the data you have, are the sales per month for 5 years. 2) The bell curve these sales follow is 'per year' (i.e. if you look at the plot of sales over a year, it would look like an interrupted gauss curve) _or_ 'per month', over the years (i.e. if you had data for 1000 years and looked at a histogram for January, it would look bell shaped). 3) Where, then, do these constraints between -1 and 0 come from (i.e. what does this -1 and zero mean here).
Go to google scholar and search flor "mathematical statistics loewner order"
Sorry for deleting the former comment. But there was a mistake in it I couldn't correct anymore ;)
Right idea, but you've added an extra zero in your $p$ value. It should be 0.0064, not 0.00064.
I don't think it's quite true that Pearson correlation is assuming continuous variables. For example, the Pearson correlation is perfectly well defined for binary variables so long as there is variation in both variables. An even better example is that Spearman correlation is just Pearson correlation on ranks, and ranks are not continuous. It's true that in general Pearson correlation may not be useful for discrete variables, but that's not quite the same point.
"Standard errors" denote the (estimated) sampling variability of parameter estimates. What you have here are "prediction errors".
@EMS There are two ways to go about this,
1. either consider $X$ and $Y$ as discrete point sets, makes things simpler  
2. or try to come up with some form of convex hull for each point set (not quite sure whether this is necessary) and something like [this][1] could be used to approximately compute the distance.  
  
  [1]: http://cg.cs.uni-bonn.de/aigaion2root/attachments/guthe-2005-fast.pdf
An interesting election result, of the kind that Russians protested about in December.
Thank you for your comment. Yes I do mean $\\phi$. I'll make those changes.
Thanks for the answer!  Exactly what I needed to get it working.
Hi umesh; since you have a new account, you might like to read through the [help](http://stats.stackexchange.com/help). If you find an answer particularly helpful, [this](http://stats.stackexchange.com/help/someone-answers) may be useful information.
yes that could be a solution. But how exactly can I do that for 16 0s and 1s?
Thanks @Madhulika, maybe I've understood. The algorithm works as follows: it has an input layer, it trains the neural network in order to have an output layer identical to the input layer. Than it compares the input layer with the output layer, and if they are different, it keeps on re-train the neural network. It stops when they are identical. When it finishes, we take the last hidden layer as the best dimensionality reduction approximation of the input layer, and use it for any goal we need. Is this correct?
That's standard cross-sectional analysis.
Thanks, but it is the logic that I need not the data structures
By "impute on new data" I assume you mean "obtain predictions on new incomplete data".  That would seem to be a problem for both approaches, and learning how gbm does multiple imputation.
I've never actually proven myself, but some Googling brought up this page: http://www2.math.umd.edu/~ddarmon/teaching/stat400/correlation-proof.pdf
I like @Michelle's comments, and since one of your goals is scale development, I'd definitely look into factor analysis--or principal component analysis if you have error-free items.  If you search for "Spector" you'll find a reference to a nice, inexpensive little book on using F.A. and reliability analysis for scale development.  Just because the method you tried didn't give good results doesn't mean there aren't workable scales latent in your data.  But trying to analyze a 20 x 30 corr. matrix sounds grim indeed.
@power What should I check for this? I already checked collinearity, and I chose random effect model from hausman testing(accept $H_0$).
An alternative: do a nonparametric bootstrap. Resample from your original data, fit the two models for the two DVs, record the difference in RMSE, do this many times. See where in this distribution the original RMSE falls. - @GregSnow: thanks for your comment on my "answer", where I indeed misunderstood the question - I deleted it, it was misleading as it stood.
@AndreSilva, if you have a coherent (or even not) *previous* hypothesis then you could use those two (four would be stretching it, but possible) w/o too much of a problem. The OP seems to suggest that isn't the case here, though.
@EOL: oh! good to know, thanks!
has provided a very nice answer. I would only add that it seems to me that in most cases, the differences among the 9 estimates are so small as to matter very little.
Personally I had a basic understanding of Entropy (though it has been almost 12 months since I applied it). But a quantitive expression of Entropy, should fit in one short paragraph,
and cross entropy should only take one more.
So I feel a good answer can include both, so that the reader doesn't need to refer elsewhere to understand it.
See Wikipedia under [Moment-generating function#Important properties](http://en.wikipedia.org/wiki/Moment-generating_function#Important_properties)
This is helpful, thank you. And addresses the simplified problem stated above. Unfortunately, as commented above, the real world is much more complex than what I initially described. I'm still working on it...
For many tests, you can calculate a sample size such that the test achieves a certain power given an assumed (fixed) effect size. In other words, you have to specifiy these things first: 1) what test do you want to use? 2) what power do you want that test to have? **conditional on** 3) an effect size that you deem interesting. 1) is something people here can probably help you with. 2) might be related to the 95% you indicated. 3) however, is something you have to provide beforehand: how different do the probabilities have to be to be considered interestingly different?
....thanks.....i wl try to go through this....
Hi,Michael.Your answer is really helpful but , shame on me , I am new to stats and haven't got enough reputation to vote up your answer yet.
Love the graph!
@Azeem asks: What do you mean by "normalized frequency"?
I think it's the dependent variable.
A single outlier can make the first pc explain 99% of the variance.
I think I can provide an algorithm that would work -- I'll think about it a bit more.
Can you say more about what the scores are for each person? You are using $n$ as both the score & as the number of values. What are the sample variance & "special additive variance"? Do you have multiple scores for each person?
@stat-o-matic: [*This is the book I found most helpful.*](http://www.amazon.com/Markov-Practice-Chapman-Interdisciplinary-Statistics/dp/0412055511)
(A) I didn't mean to imlpy that Tukey's HSD and Scheffé's method are not appropriate for post-hoc tests, quite the contrary. (B) I feel I have given a good description of why people make a difference between planned and unplanned tests (as good as I could at any rate) but I don't have Howell's book before me right now and I cannot comment on every word and every sentence of every book on the topic. Even if I did disagree, it could also be a slip or typo, a non-standard definition or something that makes sense in a specific context. It's best to focus on substantive issues, not text commentary.
The command is written on that assumption. The code isn't sentient, although no doubt that is a pedantic distinction.
Please clarify: are the DVs themselves non-normal or are their *residuals* non-normal?  Only the latter is of any concern.
+1, thanks Michael. I'm sure that will be helpful from the OP. BTW, the way I do links is to copy the link on the other page, double-click / highlight the word or phrase that I want to serve as the hyperlink here, & then click on the button above the text window next to the quotation mark that looks like 3 links from a chain. That opens a wizard where you can paste the web address. Cheers
My comment wasn't intended as suggesting you had done anything wrong at all with that choice. It's one of those options that fits well enough with SO (since for some programming issues there kind of are 'official' answers for specific products), but the wording sounds odd here on CV. If you're looking for something better substantiated than just idle opinion, that's the only option you have like it.
thanks.  i'll give it a look.(+1).
@GavinSimpson I got it in both R and IDL now. It all seems related to the Residual standard error. I believe that the **lm** function is just multiplying the sigma by that value when it calculated the Chi2 value and hence the problem with the results.
@probabilityislogic according to my lecturer the first one is correct but I dont understand why. Is it because since the model predicts on majority so it has 50% error rate ?
I am trying to maximise the entropy $\\ = \\int f(x) \\ln f(x) dx$ subject to constraints on the support, first moment, and second moment. I could also expand this to other constraints (e.g. quantiles, higher order moments, the median) by adding additional equations to the nonlinear system and additional parameters above. Here, I know that the solution to my problem should be in the form $\\ \\exp{(a + bx+cx^2)}$, and am trying to solve for $\\ a,b,c$. Just going through whuber's solution to check it works for me :)
Thanks for your comments!
I have tried to explore at JASA, Biometrika, etc., and I thought why not get an idea from this forum regarding reviews on this field that give overview of missing data problem nicely.
That's definitely my favorite. I always have to stop on this and laugh when scrolling over this page. It's just so bad!!
@ImAlso The t-test can tolerate a *lot* of non-normality if it's fairly symmetric, but it can't tolerate too much asymmetry.  (Indeed, a skewness test for normality might actually be a better option in the O.P. than the K-S test, for just this reason.)  This points out one of the biggest differences between goodness of fit tests and other hypothesis tests: there is a huge space of possible alternatives and the GoF tests tend to be good against certain of them but not against others.  You can't make them work well against all alternatives.
There's another issue here: [*confounding*](http://en.wikipedia.org/wiki/Confounding). It wouldn't surprise me to learn that accidents are more likely under circumstances where drivers aren't going so fast, even after controlling for flow rate - think of rainy evenings versus sunny afternoons. That doesn't contradict the causal claims that if any particular driver were to always go 10 miles an hour more slowly he'd be less likely to have an accident, & that if speed limits were to be reduced there'd be fewer accidents.
But, is the comparison appropriate? My ROC analysis for age alone was 'simple', directly using values of the age variable. But for microarray, or microarray + age, I am using class-membership probability values for an SVM classifier, and not the variable values directly.
Hi Juan, welcome to the site! As this question has the "self-study" tag, it would be helpful for the OP if you'd edit your answer and provided more details and information about the solutions. Otherwise, I don't see how your answer has any pedagogical value.
Are you sure you have quoted your problem correctly? I suspect you meant to write $X_t = \\sin(2\\pi(U+t))$, which *is* weakly stationary, instead of  $X_t = \\sin(2\\pi(Ut))$, which is not even first-order stationary.
If objective data cannot be obtained, the expert opinions are feasible as long as they are independent.
It is hard to make sense of the (ungrammatical) phrase "find out how unlikely Y is than X," which leaves many readers guessing what this question might mean. Could you please clarify it?
Maybe [this answer](http://stats.stackexchange.com/a/60821/21054) will help. In short, the coefficients for the categories of `density` are the difference of each class to the reference class (which is `density1`, in this case). The $p$-values are for the hypothesis that the pairwise differences are 0 vs. nonzero. The coefficient of `density1` is simply the intercept, as you already noted.
This is not the right path. The 4th equation doesn't hold. For example, with $x_1=−1$, $x_2=0$, and $x_3=1$, the left term is zero, whilst the right term is $2/3$. The problem comes from the step where you split the variance (3rd line of second equation). See why?
@Jeremy Miles Naturally I believe what you say about Mplus. In effect, you oriented your answer to CV expectations, and that's all good.
Your question as stated makes no sense to me. On what basis can you support the claim "I know that there could be at max 10 curves that can be fit in that space"?
I don't think this is close to optimal. It suggests that on the first flip, you chose B with probability 1/2. It should be clear that you get no information if you choose A, so you should choose B all of the time. The amount you lose by this error is about 0.12 when you make it, so it costs you about 0.06 on the first step. You lose a similar amount when you roughly flip a coin to decide whether to collect any information on the next few steps. Flipping A early means you have less time to exploit an advantage you might find.
What steps have you taken towards the solution of this exercise? Where specifically are you stuck?  (Are you willing to assume $x$ and $z$ are independent?)
I would also do cross-validation, except instead of log likelihood, I would compare root mean squared error. Then you can compare the improvement in models using the same units as the original data. Also, make sure to use the same train/test folds for all models.
I have 20 explanatory variable which are binary (0 or 1) and my response variable is continuous. Here is my model    y_i = 1- Xbeta + delta_i + e_i, where delta_i is the i-th random error term and e_i is the i-th usual error term. I used maximum likelihood estimation.