tttjjj/elt_bench huggingface dataset:
Edited question: I was thinking of functions as things that take in an input, and return an output. I had forgotten that I was dealing with math now, and I have fixed it.
I have the most up to date versions and reinstalled them both to verify.
Have you thought of using Poisson regression instead?  It's naturally indicated with dependent count data and your success with a log transformation is consistent with Poisson distributions.  The coefficients would be interpreted in terms of proportional increases in expected probability of missing a day of school.  One advantage is that no special treatment of zeros is needed (although it's still a very good idea to look at a zero-inflated alternative model).
@cardinal, Thanks a lot for your answer, your derived upper bound coincides with my result, but I think it is not tight at all? Do you have any suggestion to find a tighter bound?
@Calimo Yes there are 5 class labels
I think upsampling is not an option (can't take new pictures and won't use them twice). Down- or undersampling: Seems one way is to choose 50 pictures by random (as mentioned in your linked paper)
The apologies are well meant, but not needed. CV exists to answer beginners' questions too. If the question is out of order, we will tell you.
If "zero sum" does not mean zero sum, what distinguishes this from any question about (multi)collinearity, as the signs of correlations are arbitrary in that context? To the extent that a constant-sum constraint is approximated, it is more likely that one of those predictors will be redundant and is better omitted. Modern regression diagnostics should help.
I just happened to read a post by @gung on another thread that said "classification accuracy is maximized if the threshold is equal to the proportion of 1's in the population". Fortuitously, it so happens I was already using a threshold of mean(predictors) which happens to come remarkably close to the proportion (i.e. 87%). Maybe this is always true.
Please post these questions separately.
This answer is also useful, but I think Greg Snow's is closer to the information I was looking for.
If you take the log of the dependent variable, then you lose all observations with zero, i.e., 50%+. I wouldn't do that. What is the ratio exactly? Maybe you can multiply with the denominator, thus making everything linear.
"Not boring" is a problematic and subjective criterion for designing a visualization.  Please provide objective guidelines that describe what you want viewers of your visualization to learn from it.
I added a bit more; hope that helps.
Are you after a type II error *rate*?
I need more context.
Does that imply we use Chebyshev's inequality?
@Henrik: You want basic literature on neural networks?  You might consider posting another question?
>My only thought is that the area under the posterior may be slightly less than the area under the likelihood.  And yet it is a shibboleth often murmured that a likelihood function is _not_ a density function. Thus, it is not at necessary that the area under a likelihood function be $1$ the way that it is for all densities. I don't believe a minor difference in area is indicative of anything.
@whuber, are there any special cases for small n, e.g. n=3?
@user30353: glad I could help!
There's a v nice wavelet tutorial [here](http://users.rowan.edu/~polikar/WAVELETS/WTtutorial.html) which also covers Fourier Transforms in explaining  how & why. But I'm not sure Fourier / Wavelets / etc are going to help much except in removing high frequency noise -- once you've done that you may be content to run an algorithm on the signal to find peaks and troughs (a very simple one is illustrated in [this answer](http://stackoverflow.com/questions/16858015/what-algorithm-do-i-use-to-convert-this/16866785#16866785) I wrote some time ago, which links to another covering general techniques).
As stated, you have N of 1 and statistical significance is meaningless. But actually, you have time series data (your portfolio and the S&P performed on each of the 60 or so days that the market was open between 1/1 and now. You should look into time series methods.
Thanks! Though what do you mean by close to? The proportions are actually between 15 and 40 out of a 100.
The current incarnation of this question is simply asking for solving a least-squares problem with linear equality constraints. There is a closed form solution, so I don't see the need for using a general convex solver here. (Maybe you were responding to some previous version of the question?)
I'm not aware of any existing implementations but isn't it very easy to implement it? It's basically this: initialize 200 means randomly and create an array of size 200 which will keep the number of data points assigned to each cluster. Then, as a new data point comes, assign it to the closest cluster, update the cluster's mean, and increment its count. That's it.
Could you explain what you mean by a "nearest-neighbor query" in this context and how it is related to dtw?
Could you expand a bit on what it would mean to "draw" this data structure? One would guess that you want to represent the nodes with point symbols and the edges with some kind of 1D traces connecting those nodes. Could those traces be curved or must they be straight? Do you also want the Euclidean distance in the plot to be directly proportional to the edge lengths? (That can be framed as a *multidimensional scaling* (MDS) problem.) Do you want to minimize intersections of the images of edges? Any other objectives?
You are having trouble because your description lacks critical information.  First, *how* do you "randomly draw" values from a *function*? Related to this is concern about how those values might depart from the correct functional values: are either (or both) of the $x_i$ or $y_i$ measured?  If so, what is the expected statistical behavior of the measurement errors?  Third, how do you measure "closeness" of an approximation to a function?  Fourth, why are you choosing a linear estimator $\\beta^\\prime x$ to approximate a function $\\sin(2\\pi x)$ that is *a fortiori* strongly nonlinear?
A few questions: Is $y$ univariate? Then you have a multiple logistic regression, not a multivariate logistic regression. How are you doing the factor analysis? While there are methods applicable to ordinal variables that are analogous to factor analysis for continuous manifest variables, it isn't clear to me what you're using. Finally, what would you consider to be an inappropriate use of factor analysis? If you look on it as a dimension reduction technique (turning 11 variables into 2) then there's nothing inherently wrong with it. Whether it's a good idea or not is another question.
@gung I see what you are saying about Dunn-Sidak versus Bonferroni. But I still think independence is (close to) the worst case situation for Bonferroni. While admittedly extreme, perfect correlation in my mind is the best case for Bonferroni in that I could use $\\alpha$ with no adjustment. Are we using "best" and "worst" in the same way? I mean best in the sense of requiring the smallest downward adjustment of the p-value (not sure if this is standard usage). Thanks.
Welcome to the site, @JoshHansen. Do you see this Q primarily as a programming question or a statistics question? It looks like the former to me. Note that CV isn't a Python or programming Q&A site (although, [Stack Overflow](http://stackoverflow.com/) is). If you are wondering about the statistical aspects of expected values, etc, would you edit this to clarify your question? If you mostly want to know how to implement something you understand in Python, flag your Q for moderator attention, & we can migrate it (*please don't cross-post*).
Thanks, pretty gets me most of the where I want to get to.  I think you are right, that ggplot2/scales material probably would get me towards being able to get ggplot to do what I want inside ggplot.  However, at the moment seems more difficult than shoehorning it from the outside.
"*Spearman generally reports better correlation than Kendall-Tau for our data, and I was wondering what that says about the data specifically*" ... likely nothing; Kendall $\\tau$ is often nearer 0 than Spearman's $\\rho$ when the correlations are not really close to $0$ or $\\pm 1$ - it measures association differently; the fact that it's typically smaller in magnitude does *NOT* mean that the Spearman correlation is 'better'; they're just measuring different things about the data. What is there about the two-numbers-measuring-different-things that would make you say 'better correlation'?
(+1) It's a good question.  Concerning how to report results: first, it needs a comma before "with" to be understood correctly.  Second, it is better to be quantitative instead of using the qualitative "slightly," although "highly correlated" is ok.  You might instead write, "The surface % o.m. is highly correlated with the mean % o.m. in each core and exceeds it on average by 3.2%" (or whatever the mean difference is).
Would factor analysis not do the job for qn 1? Question 2 is a bit vague. 'Relationship' seems a synonym for 'correlation' or at least one form of relationship is linear relationship and correlation captures that. Perhaps, you need to clarify qn 2.
Do you have any basis at all to convert poll results to probabilities?  What is it?
@stat-teacher LaTeX works here. Just enclose formulas in dollar signs $...$
I guess what I'm trying to say is I want to prove percent change from before and after intervention but I'm concerned because 1, a lower value, represents strongly agree the results will  not necessarily be representative of what we want.
I am not sure discretization would result in poor performance. But the fact that time samples can be separated by times from 1 second to 7 days make it difficult to define a discretization step. Discretization might be an option, but it's a difficult subject in itself, in the present case.
It's a good question (+1), but it seems to me it is clearly answered in the first paragraph of the [Wikipedia article](http://en.wikipedia.org/wiki/Probability_mass_function). What would you like us to add to or improve upon the information available there?
Power to detect what? It only makes sense to compute power to detect some hypothetical effect that must be stated.
Your third paragraph is much better than the second. Conditional on the observed data, the confidence interval either contains the true parameter value or it doesn't.
A derivation is in the Wikipedia article: http://en.wikipedia.org/wiki/Least_squares#Weighted_least_squares
*Can* be enough, but how will you ever know?  I think your perfect fit is evidence that it just wasn't in this case.
Then add another constraint. This should be quite efficient to compute using a classic A*-search, too. Plus it depends a lot on what *data* you have available: adjacency lists, polygons, centroids, ... - how do you intend to measure compactness?
There is too much here to address.  Which *one* of these questions do you wish to have answered?  The one about efficiency, the one about ML estimators, or the one about extending the "sum of sqrts method" (what is that, by the way)?
@ wayne ( EDIT) I did use linear regression and have a slope coefficient for a trend equation. how do i know the statistical significance thru that ?
I'm still failing to understand why having one estimate with a small error makes an approach 'unusable'.
Actually, your intuition is incorrect in example 2.  Suppose these were institutions of higher learning in the US, the classes are types of institution (e.g., state schools, liberal arts colleges, etc), and you want to estimate average enrollment.  The optimum sample includes disproportionately more of the huge state schools, even though they comprise only 2% of the total count, because they contribute so much to the total student population.
Why not start from some simpler and probably easier to understand relative risk or odds ratio? If individual site will be considered as diagnostic information I don't see why pooling them in a chi-square will help. Yes, you will know that site of discovery matters, but the next question will then be which site.
Are you sure the Chi² requires the same size ? (I do not clearly remember the Chi²). But anyway, you can use Fisher's exact test.
I would be surprised if these terms are well defined so that there is universal answer to this question.
@Karl That is correct--and this observation often is the beginning of any analysis of such a process. But something is needed in addition to this fact, because it does not in itself determine what the threshold must be.
Can you elaborate on this? Gibbs sampling does not seem to help here, as conditional distribution = marginal distribuion = uniform? Or is your suggestion to use the previous samples to produce "holes" in the distribution to sample from?
I strongly suspect that, with a sample size like that, the prior will have almost no influence. An exception to this would be if there are feasible areas of the parameter space where the prior has 0 mass. I'd suggest trying different priors and seeing how your estimator changes as a way to empirically see how much it matters.
Could you edit your question to elaborate on why you "just don't feel good about using auto.arima and then just cross-validating to see how many external regressors to add in"?
Even the ordinary regression fit may be invalid because the $(n^2-n)/2$ points reflect at most $n$ independent observations, causing extreme interdependence.  So, testing significance appears to be the least of the concerns here.  It looks like you need a different approach to the analysis altogether.
I added a tag for splines, in case anyone wants to list rules of thumb for them.
There is not enough information available here. How large is your data? Is it balanced? How large is the ratio of variances? If your data is unbalanced, does the larger group have larger or smaller variance? Generally speaking, balanced ANOVA is rather robust against heteroscedascity and the most problematic case is if the smaller group has larger variance.
By all the features in the model, do you also mean lagged outcome variables?
Also, thank you for your response, I'm definitely will take a look at your references.
@atiretoo I don't see why that assumption is needed ? If there are any observations with tumor_size==0 and overall_tumor_grade==NA just recode those NAs as X. Then delete observations with overall_tumor_grade==NA and run the imputations..... ?
Given that PCA has many applications and interpretations, it is not clear what you're looking for an alternative to.  Could you give us some hints and indicate what purpose your alternatives aim at?
pretty much solved here: http://stackoverflow.com/questions/2200460/what-programming-languages-are-good-for-statistics
This question seems to be *only* about how to get this to work in R. As such, it would be off-topic for CV (see our [help page](http://stats.stackexchange.com/help/on-topic)), but possibly on-topic on [Stack Overflow](http://stackoverflow.com/) if you could provide a [reproducible example](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example). If you have a question about the statistical aspects of this, please edit to clarify; if you just want help w/ the code, please add a `dput()` of your data & we can migrate it for you (*please don't cross-post, though*).
I don't know whether or not this is more a programming question. I earlier asked this question on stackoverflow.com, too (http://stackoverflow.com/questions/23469825/putting-permuted-data-into-libsvm-precomputed-kernel). Feel free to answer my question here or there. Dear moderators: if this is not okay for you, feel free to delete my post. Thank you very much!
For the permutation test the size of the effect does not play as much of a role as how the top scores are distributed between the groups.  For the minimum numbers above of 3 and 11 there are 364 possible unique permutations between the groups which suggests 18 possible combinations that would lead to a significant test (at 0.05).  Clearly if the 3 all did better than the other 11 you would get significance. but if the very best score is in the group of 11 I don't think you could get significance.  But comparing scores of 1-11 vs 98-100 will not be more significant than 39-50 vs. 51-53.
I overlooked it. Thanks!
The Clopper-Pearson method is excessively conservative when the observed proportion approaches 0 or one. That conservatism is a consequence of them being designed for _at least_ 1-alpha covefrage in the long run rather than good performance for the particular data set being examined. See my answer to this question for more detail; http://stats.stackexchange.com/questions/8844/discrete-functions-confidence-interval-coverage/8860#8860
This is great advice, I will look into this.  Thanks!
Thanks. We don't need to establish which sensor is better, only that it does not matter which we use. From technical paper we know that one has a +/- error of about 18.5 cm and the other between +/- 15 to 20 cm. The differences between the 2 measurements are within +/- 40 cm (well, at least 96% of the data is). If we are satisfied that the results are comparable we can proceed further with analysis.
As I keep saying, use out-of-sample performance on a large set of series. You can't easily compare in-sample performance of the two model classes.
I ran this in Stata with pweight=(#trt obs in group)/(#control obs in group) for control, and =1 for treatment. This gave IDENTICAL results to the direct matching estimator. So, problem solved! (though I still don't fully understand how this works..)
@Stephane, don't know exactly what you mean but you may be mixing up Bartlett's and Mauchly's sphericities.
I vote to close this as a duplicate. Did you read @Corone's [edited answer](http://stats.stackexchange.com/a/50581/6633) to your [other question](http://stats.stackexchange.com/a/31328/6633)?
That curve looks weird because of the weird choice of scales on the axes.  It reflects replies that have contributed the majority of a user's reputation: the one-post wonders.  It's exponential because the y axis is linear while the x axis is logarithmic.  You really should ignore everything associated with log reputations less than $2$ because for many users that's where their reputations *start* and you should consider almost anything for log reputations less than $3$ to be just noise.  As such, 99% of this graphic is devoted to displaying that noise: there's not much information there.
Thank you so much for your detailed explanations.This is really a great explanation! You helped me a lot.
Thanks Rob for the edit.
Well, you could, but it's a different design - you'd no longer refer to them as 'pre-' and 'post-' but 'control' and 'treatment' (or 'intervention' in this case). The effect of the intervention on any individual can't be assessed.
I just saw your post there; I'm sorry of not thinking to just point you to the code for ppoints itself, but I wasn't thinking you would want to reproduce its behavior exactly (just the concept underlying it, which was the purpose of `quantile(A,probs=(rank(B)-alpha)/(length(B)+1-2*alpha))` in my third-last paragraph there). Anyway I am glad you are happy with the outcome.
[this](http://www.ats.ucla.edu/stat/stata/whatstat/default.htm) is a good resource for this kind of questions.
Actually, almost any interpolator implicitly "takes spatial correlation into account." E.g., with linear interpolation on a Delaunay triangulation, the nearest neighbors get varying weights, depending on their distance to the estimation point, and all other points get zero weights. So what's at issue is *how* spatial correlation is accounted for. Tests have shown that--depending on the data--kriging often is not the best interpolator in practice. Also, interpolation based on a triangulation need not be linear. You can spline the data, for instance.
Not exactly, it' more a function of how one obtains 'true score' estimates ($\\hat{\\theta}$) and their associated standard errors, to form the CTT ratio $r_{xx} = T / (T + E)$. So if you compute EAP scores, for example, you can use this information to form the ratio between the variance and $\\theta$ and the variance in the standard errors. The `mirt` package will do this with it's `fscores()` function, and so will the `sirt` package (or maybe it's the `TAM` package....I can't recall, it's the same author for both).
None of those plots tell you anything about the normality of $X$ and $Y$: they only compare their distributions.
How many variables did you measure / test?  Do you think of them as related to each other, or are they independent?
Hi Michael, the classic ANOVA is not appropriate because despite using transformations it is not possible to meet the normality assumption.
Thank you. I think I get it now. Much appreciated
Your intuition about the relationship between sample size and priors is correct. On the other hand, Bayesian logistic regression can solve the problem of infinite parameter estimates resulting from perfect separation.
I don't quite follow how the diagnoses for the pair of patients $i$ by doctor $j$ could be $(2, \\boldsymbol 5,0)$. I can imagine the doctor diagnosing both patients in the pair with a given disease (as for $a$), or neither patient (as for $c$), or only 1 patient w/ disease $k$ (e.g., $b=1$), but I don't get how $b$ can be $\\boldsymbol 5$. Can you clarify this? Furthermore, I gather that a given patient can be diagnosed w/ >1 disease, is that correct? Should we assume there is some correlation b/t the probability of the diseases, or are they independent?
Take a look at the sin I have already proposed about testing. If you do the test and do not reject normality, it does not mean that you have a normal sample... it only means that you cannot say the sample is not normal. sin !
I altered my question to explain what I want to achieve.
Your R command can be shortened as `xyplot(..., type=c("p","smooth"))`. The `span` (which is 2/3 by default) can be directly passed to `panel.loess()` so that your expression is equivalent to `xyplot(lcount~AirT | cut(DOY, 12), type=c("p","smooth"), span=1)`. You can also use `shingle` (or `equal.count`) instead of `cut`.
Is there a reason you can't just use the imputed X as the variable? I don't see any advantage in this.
Just because $p\\in [0, 1]$ doesn't mean that observations < 0 will be ignored. Whether or not they will be is determined by $p(o_i | p)$. So (provided that you know or can model $p(o|p)$ well), the Beta prior on $[0, 1]$ seems reasonable.
How big is your corpus? What language? Why do you consider "calculating the frequencies...is not a good choice"? Are you interested in common words, not-so-comment, very rare? I think a little more context would help. The broad subject matter is a fairly richly developed research area. :)
@Michelle: The program which we're hoping to improve with a subject-matter expert's advice already has its own context (original design, programmers, prior experts, etc). The advice which we're trying to incorporate comes from a different context, and may not work well -- may not even be consistent with -- the context that the program already has. My last statement was an attempt to say that it's actually unlikely that new input would work in the context (of the program) that's already established.
I don't understand. If DVRt is the dependent variable and verx_s is the independent variable, then you don't need logistic regression. But you can't recode your IV to match your DV! That would make no sense. It would help to provide context here. What are these variables? What are you trying to do?
Could you please write a few words about what qualities make it "great" so that readers can determine beforehand whether viewing it would be appropriate for them?
Assuming it's the Poisson means that are going to infinity, I'd suggest maybe looking at $Z_i=\\frac{X_i - \\bar{X}}{\\sqrt{\\overline X}}$. (They're not all independent, they lie in an $n-1$ dimensional hyperplane.)
See updated answer.