tttjjj/elt_bench huggingface dataset:
Thanks, but what is AIC? I just realized I asked a long winded question when the purpose is quite simple. I just need to get a "normalized" standard deviation. Someone suggested a t-student, but this gets quite cumbersome in excel. Am sure statisticians have found a way to eliminate differences in sample sizes and amplitudes to be able to compare standard-deviation-like measures? Thanks again
That's a tough question. There are several competing philosophies of variable selection that depend on the researchers' ultimate goals. Personally, I use economic theory to pick a preferred specification that is relatively parsimonious. Then I test down a general variant of that specification that encompass rival theories. At times, I have used regularization on a less carefully selected set of variables.
Thanks! My goal is to explore data _before_ I run PCA, as variables can be related in a non-linear way and require rescaling (or other processing).
You might want to take a look at this question:\\nhttp://stats.stackexchange.com/questions/3238/time-series-clustering-in-r
@cardinal : There are several interpretations of the question. As it is written I agree with you, I also wrote a comment where I guess that the best bid should be something close to 0. But also guess that it can't be so easy or simple... there is something missing in the question... I'm not proud of my answer, it's  as ambiguous as the questions itself... ;)
Could you please explain what "intelligently" means? What are you trying to accomplish with the binning? Why are you binning in the first place?
1. is this perhaps homework?
2. are you sure you have X and Y on the correct sides of the bar in your second equation?
(+1) For the question and the answer. This is such a nice exercise for a statistics course. Sad the course I was T.A.'ing ended before doing it!
This question appears to be off-topic because it is about which program command to use.
Can you explain why the checks that you advocate cannot be similarly automated?  In my experience manual model selection often provides worse predictive performance than automated schemes (see for example the results of machine learning challenges, where those at the top of the leader board generally finish quite a way down the final rankings because they have been manually over-fitting the model).
I think the answer is "no", but I'm not enough into SVMs to give you a full answer. My gut answer is that when you're on the East side of the Berlin Wall, well you're just on the wrong side, no matter how far your are from it.
Some relevant threads: [How to convert molecular categorical variables to dummy variables for cluster analysis?](http://stats.stackexchange.com/q/22210/930), [What is the difference between independence.test in R and Cochrane and Armitage trend test?](http://stats.stackexchange.com/q/8774/930) This [response of mine](http://stats.stackexchange.com/a/9394/930) has some references [1,2,4] that might be useful, but [Introduction to Genetic Association Studies](http://cshprotocols.cshlp.org/content/2012/3/pdb.top068163.full), by Cathryn M. Lewis and Jo Knight, is probably more recent.
+1 Sometimes, a single sentence really is the answer.
Hey Shankar, try the `hopach package` in Bioconductor. It gives you an estimate of the number of clusters in your group. Also, you should try `pvclust package` in R/BioC, it gives you the "statistically significant" clusters.
@Nick within the framework of ARIMA models we can have
It is a classification task. Well, i'm using WEKA. And i tried different learning algorithms. I got the highest accuracy for correctly classified instances using the LADTree learning algorithm. However, if i were to perform comparison for analysis, there should be like a base/default model to compare against. And this model should be an alternative way to producing a model. Perhaps something like, random selection, etc?
@Gael Laurans: thanks for the nice editing!
I have tried my best to edit this to make it clearer, but I still have trouble following your post. In addition, I think you will need to give more details about your situation before this can be answered. Please edit your question.
Thank you, that's been very helpful. Can you also recommend a textbook with the above information, perhaps with examples in R?
@Penguin_Knight: thank you, however when I do margins I get predictives for each level of the categorical variable, even without using the i prefix. I want one estimate, of the overall effect, i.e. something like a trend, which you usually get if you treat the categorical as continuous.
BTW, `rflowcyt` has been deprecated with recent releases of Bioconductor, it is now recommended to use `flowViz`. Anyway, both rely on `lattice`.
+1 In addition, look at the sandwich package which works hand-in-hand with functions in the pscl package to provide appropriate estimation of model covariance matrices that account fot the lack of independence.
`alternative="two-sided"` is the *default*, which is why nothing changed.  Try the other alternatives.  I won't speculate on what the author of that presentation might have been thinking.
Thank you very much Henry!  this is most helpful.  I had already written an excel maro that gave me all of the combinations without replacement (essentially, the table you suggest) but I didn't know what to do with the data.  Thanks again for your time and help!
@restats: Added explanation to answer.
That's true, @Tal, but it's simply due to tradition. That is, it's what people do because it's what people do. There is a legitimate question about whether it is the best way to go about things. Even if you believe the appropriate answer here is "no" (which is undoubtedly a defensible position), this is certainly a question worth asking.
+1, good points. I suppose I could have been more complete. In addition, I turned off the inadvertent mathjax by putting a backslash "\\" immediately before the dollar signs. I hope you don't mind.
Thank you for such elaborate answer. The code works fine, but at the first try the quality of the fit looks poor. I'll evaluate it more and post the results when I get them.
I feel for you.  High precision is hard if the vast majority of your cases are negative. I would use class-specific weights (such as inversely proportional to the fraction of cases in the class) for learning and save the error-type-specific weights for determining decision threshold.  Hopefully you're using cross-validation with Cohen's kappa not accuracy for model selection.  I would visualize the density of probabilities for the classes in calibration data along side precision and enrichment (precision / proportion of positive cases) at all cutoffs to really understand available tradeoffs.
For a discussion of some graph embedding and drawing algorithms found in software, see http://reference.wolfram.com/mathematica/tutorial/GraphDrawingIntroduction.html and scroll down the middle starting at "Graph Drawing Algorithms".
In general, approaches to tackling your question  encompass many topics discussed in this forum so yes, very exciting, but covering quite a broad sprectrum. Image registration (or alignment as you call it) is a difficult problem to solve properly in its own right!
Thanks, great edits (+1). For $n=2$ the null distribution is a triangular distribution, so indeed $t_\\alpha=\\sqrt{2\\alpha}$. For $n=3$ the null distribution is already quite involved (its a three part density) but luckily for $n > 3$ it can already be approximated very well by a normal distribution with mean $0.5*n$ and variance $n/12$.
That's right. The effects of sex and height are considered at their respective lines ("Male" and "Tall", respectively). However, because they are involved in interactions, they are not interpreted as main effects. You should read up on interpreting regression coefficients with interactions before going any further.
The randomizing, re-running, averaging and final run is a very good advice. Thanks
I don't have any specific answers for my data yet, still in the nascent stages. This is more a general question to get an idea of what is possible with NMDS, and will I be able to quantify some of the visual results I see in my NMDS plots
Thanks a lot for the answer!
#2 would definitely introduce statistical complications, but not necessarily anything you can't handle. Depends on what those additional insights would be worth to you! If power is a concern, there's something to be said for keeping everyone split into only two groups, but this would seem to give you a less nuanced picture of any relationship between the effect of treatment and its timing or continuity.
@nostock, most introductory stats textbooks will show these formulas.
Trivial example: X has a Bernoulli distribution (ie, taking the values 0 or 1); Y = X + N(0, 0.1). Neither X nor Y is normally distributed on its own, but regressing Y on X still works.
@posdef, answers require better quality than comments, this is why I commented. As for commercial use, try google to find non comercial. [This seems free](http://www.numericalmethod.info/javadoc/suanshu/basic/com/numericalmethod/suanshu/stats/test/distribution/normality/ShapiroWilk.html). In worst case scenario you will need to reimplement this test, or another normality test. I suggest asking here or on stackoverflow , which of the normality tests is easiest to implement.
It is possible in `nlme`. Check the use of `?corARMA`.
Thanks for the comment.  I think this is what I am trying to figure out?  So, if I cluster (suing kmeans for example) I should get a pairwise matrix in return, how do I access the coordinates (or points) of that matrix.  This is probably a very rudimentary question but I just want to make sure I understand it correctly. Thanks again.
The answer to your question may depend on the answers to other questions such as: (1) How many categories do you have? (You might want to check agreement for each category.)  (2) Is there any reason to think that the categorizations will be easier or harder for some categories than others? (If you think there is no difference in difficulty across categories, perhaps you will waive a check on the agreement for each category.)  (3) Are the baselines for each category the same? (For example, do a large majority of your cases fall in one category.)  In short, can you tell us a bit more?
Douglas, thank you! I initially did exactly the same. I'll mark this as an answer; I also completely agree and accept the vagueness posed by my question. I'll post another question with better formulation of my problem (which is already asked in the comment). Thank you very much!
So you know that `wilcoxsign_test(Y ~ X, alternative="less", zero.method="Pratt", distribution="exact")` from package `coin` does what you want? Note that you have ties as well as zero-differences which can be handled in different ways.
Is this under "real world" assumptions or is it a word problem? Because of the way it's phrased, most of it would be an exercise in game theory.
Is there some way to quantify the error propagation? Is there some way which allows me to compare the results of $\\hat x_{3,{\\rm Sep}}$, using either $x_{2, {\\rm Jul}}$ or $\\hat x_{2,{\\rm Aug}}$?
@Glen_b: Yes, you are right, my original question was not very clear.  Please let me know if the edited question needs further clarification.
@Macro Certainly, if you know the variance and the distribution, you're right. But that's not the case here, as far as I can tell. (I think it's quite rare to know the variance - except where it is known by force).
If you do a MANOVA on two groups you should indeed be able to get a single p-value. However, some packages might reserve manova for more complicated models than the multivariate T situations
The point here is that, if the Markov chain is transient, then $f_{jj}<1$ for every state $j$. To visit $j$ at least $n+1$ times starting from $i$, one first has to reach $j$, which happens with probability $f_{ij}$, and then to come back to $j$ at least $n$ times, which happens with probability $(f_{jj})^{n}$. Thus $E(M_j|X_0=i)$ is the sum over $n\\ge0$ of $f_{ij}(f_{jj})^n$, which is finite if and only if $f_{jj}<1$, and then $E(M_j|X_0=i)=f_{ij}/(1-f_{jj})$. (Be careful: your comment is awfully written and you write "equals zero" instead of "converges to zero".)
I recommend that you exclude them from your regression and re-run the analysis. There is nothing wrong with excluding observations from an analysis if you (1) justify the exclusion and report it in detail and (2) show that excluding these observations does not affect your results substantively. If these data points come from a specific period of time, you can also add a dummy variable to capture this source of variation.
That $\\widehat{x}$ denotes a different random variable in the set. This says that if $x_c$ and $\\widehat{x}_c$ are two observations in the same clique, the potential function evaluated at $x_c$ and $\\widehat{x}_c$ is equivalent.
@Tim, the null hypothesis for a permutation test is that the distributions are identical, but there are many ways in which they can be non-identical, some more interesting than others, and the choice of test statistic will have more or less power for different types of non-identical.  This post (http://tolstoy.newcastle.edu.au/R/e4/help/08/04/8485.html) and the others in the thread and the paper referenced may be of interest.
This seems a little harsh.  It sounds like the search for correlations was planned at the outset and the OP is aware of the multiple comparisons problem.  That already puts them on a better experimental, analytical, and conceptual footing than the vast majority of publications in this field.
If I understand correctly, if the first window is large enough to bury the burn-in, it is still feasible to use Geweke diagnostic to tell whether the chain has converged, but cannot tell how long the initial burn-in period is? In other words, if I don't care about the burn-in period, it doesn't matter if I choose a large first window?
The probability that $X_1>X_2$ depends on the joint distribution, not just the marginal distributions (I can make $X_1$ and $X_2$ normal and yet get a very different collection of answers as I vary their copula), so you need to consider the joint distribution. Given the joint distribution, you compute from it the required probability. If they're jointly normal, you'd usually do it by computing the distribution of the difference and comparing to 0. The given answer assumed independence *without even checking if that assumption was justified*. Since you now say that's okay, that answer is fine.
If you suspect the OP in any question might be misusing a standard term, Peter, then please call that out: a request for clarification in a comment is a good method, but if you feel you need to post a reply, at a minimum please explain what interpretation *you* are making, rather than continuing with a usage you think is ambiguous!
explain the differences between your three outcomes: avg, avg(congruent), avg(incongruent).
@whuber yeah, kind of. It's a improved version of KDE (which is biased itself anyways), in which I need to employ a consistent estimator to get a rough estimation of $f(x)$ before the new method can be employed. I see what you mean by the pdf thing now, reminds me of the pain to get from $\\mathcal{L}^{1}$ to $L^{1}$...:)
@MiroslavSabo: I like your plot because it shows that men and women do not form two separate clusters (with respect to their interests), but actually form a spectrum. I suppose you were preparing a research paper; has it been published? is it still going to be?
@NickStauner It prevents the zeros of the non-subset from (directly) influencing $\\beta_2$
+1, thank you. In 6 you wrote "static", did you mean "statistic"?
good stuff, @whuber.
sorry updated the intervals
"However, my main concern is that the many (0,0) suppress the SE of β, which is not true by common sense."

Why? (0,0) pairs don't contain information about β. Large standard errors make sense. Why should you be able to accurately estimate how Y linearly varies with X, on average, when there is little variation of X in your data set?
Initially forget about sklearn, that 's a red herring. The source of your misunderstanding seems more fundamental. Have a look here: http://en.wikipedia.org/wiki/Confusion_matrix. Focus on the narrative of the 3*3 example on the wikipedia page. That will most likely address whatever your confusion is.
The line that should be WL is labelled WW. The error propagates down that line. Is it that all you have is this figure or do you have the data from which the figure was generated?
How is your error $90\\%$? Are you counting it as a total miss if you don't get the exact value? Usually, when trying to predict a continuous value, people use a cost function like the squared error. It might make sense to rescale the output, but be careful that when you don't have a lot of data, you want to use a simple model. Trying a lot of ways to rescale the output is equivalent to using a more complicated model which may tend to overfit on a small sample.
@JMS Nope. I only have Gentle's textbook. (Because I only make a moderate usage of mathematical textbooks in general, except this one which I found pretty handy for multivariate data analysis.) Part 2 is about application (section 9) and Part 3 about software issues. The homepage is http://mason.gmu.edu/~jgentle/books/matbk
thank You @Bitwise.  I have tried a bit of RDS and it seemed easy to use so far but I will try to work with Weka.
Thanks. Won't `Anova` just pretend the within-subject and -target correlations aren't there though? The reason I repeat the analysis with only data where `appropriate==1` is that a number of the materials used were shown to be problematic a post-test, thus 'inappropriate'. Finally, I've not used random slopes for the simple reason that the model fits better without them.
So what I would be looking for is the max of the normal distribution?
Thanks, I found out that log(Y+2) already helped to stabilize the residuals. I used the logtrans function from the MASS package in R to come to this conclusion.
@Glen_b, but the warning was after `oneway_test` not the `wilcox_test`! the warning message itself is vague. The difference between the two is I think the transformation with `rank()`.
@DavidP: You mean "parameter", not "statistic".
@posdef Best of luck.  I'm sorry to hear about the conflicts of biological and statistical acceptability you have encountered.  Any such conflict bothers me, because "statistical unacceptability" is tantamount to *unscientific.*  There has to be a resolution to this.  Maybe it needs better communication and understanding on both sides.
or search the web for "iteratively reweighted least squares" ...
Thanks for your answer! However, I'm having some trouble understanding some parts of your explanation. The problem is that I get an interaction effect for interaction x*y*z when just including these factors, but that this interaction dissapears when I include the factors a, b, x, y, z. You are talking about certain options in SAS to give other estimates, but what happens to the significance testing in that case? I assume I just can't walk in, change how the estimated means are computed, and force the analysis to get significant.

Also, I'm just using fixed factors (that is, no random factors).
Thanks for your explanation. So how do I discern the `p` for the control group so I can use it in my analysis? Or perhaps a better question is: Is the `prop.test` function really what I want here, or is there a simpler method that I should start with?
I still think tow shouldn't be in the model at all given that description. Day sounds fine though.
Thanks a lot for your answer ! I have read this book from first to last page, but I think it was edition 1... I didn't know it was available online.
Knowing the mean and standard deviation of a random variable $X$ is not the same as being "given the distribution of a variable $X$" as your title puts it.
Annual data, so I have only 9 points per region...
Alan, my earlier version omitted an important term; my apologies. Please take a look at the corrected answer. (That omission may have been part of the problem!)
Oops sorry I'm still sleeping ;) You mean that the regression model is a modeling of the conditional distribution of Y given the covariates, right.
Instead of taking the derivative, why don't you simply plot the rates of change from your fitted model as well as your original data and then compare those? This is a common procedure in modeling rates over time.
If only Bourbaki had written a book about statistics... :)
@crf In nonlinear problems, you have to depend on the definition directly, to show that different parameters result in different functions.
By the way, you should be able to use my derivation in the section **Conditional mean identification in the exponential regression model** directly without changes for your homework solution.
@mathias I doubt if these approaches are optimal, but at least things are going in the right direction :)
"Citations of Comte may also be useful." link is dead now, what citation it was ?
Very clear, and helpful derivation.
Here's another answer to a very similar question
http://stats.stackexchange.com/a/62061/4485
excellent. I've found a much cheaper solution, but will use those if that fails. Thanks again.
If $X = 2$, you are in the lower case of the '{'. The limit of a sequence of zeros is just zero.
+1 $\\sqrt{n}$ is a good place to start
This is a question about predictions based on linear combinations. You will need to base your CI on the standard error of the combination, which is a function of the standard errors of the estimates, but also of the covariance of the estimates..
@whuber, "Hourly" related to the nature of the data. Besides, "Simultaneously" refers to co-occurrence of rainfall in all the stations.
+1 For Introductory stats with R
Exponential families are defined by the fact that the log density is essentially a scalar product of a vectorial function of the observations and of a vectorial function of the parameters. There is no graphical structure involved in this definition. MRFs involve in addition a graph that defines the cliques, the neighbourhoods, &tc. Hence, MRFs are exponential families with an added structure, the graph.
@Glen_b: thanks. if I model a time series by GARCH, should the time series be uncorrelated? In other words, a GARCH process must be uncorrelated? In my example, I would like to apply GARCH to the residual series from those approaches I tried above, but the residual series didn't pass Ljung-Box test, meaning that the null that the residual series is uncorrelated is rejected.
You need to change your question as many people won't (and shouldn't) run a random excel file on their computer. Pull out the formula from the file and ask a question about that.
`What about an estimator which is an average over its closest k neighbors with the weights given to the neighbors decreasing with distance from the target point.` This is essentially inverse distance weighting, which could be viewed as the simplest type of kriging model. A very similar context for estimating correlations between variables is geographically weighted regression.
You can also do this with non-normal population distributions to show central limit theorem
*Data* is not nonparametric. Data is just data. It sounds like you want to do post-hoc comparisons. These would usually be done in such a way as to have a combined type I error rate that's reasonably comparable to the overall test, meaning that the individual comparisons will generally be at lower alpha levels.
+1, but please http://en.wikipedia.org/wiki/Wikipedia:Don%27t_call_it_Wiki !
@EpiGrad I thought it was the purpose of questions to be answered, and the purpose of answers to answer the question asked (correctly). But I guess I'm old-fashioned or something.