tttjjj/elt_bench huggingface dataset:
Thank you for your suggestion. I now tried to use a mixture model implemented as suggested in the link. it works better, but it does not converge to the true values (when using stimulating data). What sort of works is treating the data as truncated, throwing away all responses that could be reactions to the metronome sounds, and analyzing it like [this](http://doingbayesiandataanalysis.blogspot.se/2012/01/complete-example-of-right-censoring-in.html). Downside is, I'm throwing away a lot of data...
But my followup was 2 years late,so would it change the age covariate? And one other note, my measurement is actually the cortical thickness of a particular brain region. Which actually depends on the age of a person. Does this fact makes any effect?
I've plotted log y versus x and log y versus log x for your data and there's no question that the second (which you give) is better. As said, that's a power function, not an exponential. I don't know what you plotted exactly but judging fit is easiest when the reference curve is a straight line. (Any confusion here might reflect loose use of "exponential": see my answer for what I take to be the exponential model in question.)
FYI-- using named hyperlinks is actually quite simple: while you're typing your answer click the little icon that looks like a chain link. It will pop up a window that will make the process self explanatory.
What do you want to do? Sample from the Inverse Gaussian distribution? If so, [this](http://en.wikipedia.org/wiki/Inverse_Gaussian_distribution#Generating_random_variates_from_an_inverse-Gaussian_distribution) might help.
The title of the question may be confusing. What about changing it to something like: Modifying the time granularity of a state sequence.
Have you tried using the natural log of price data instead? Often that will help with negative range issue.
The policy on crossposting is contained in the very first page of the help. It says 'please don't crosspost'. We should choose the one best site for our question. A question can be migrated if necessary. If part of a question is better suited to a different site the question should be asked as two separate questions (which can be linked).
remember that if you found answers to your previous question helpful, you can mark them as 'accepted' by clicking on the checkbox next to the answer. this lets others know your question is solved.
I encountered same problem before. And I think here trend might mean l+b. (In paper, there is model) Or you can see http://robjhyndman.com/hyndsight/forecasting-weekly-data/
It's quite common for documents to have almost toally disjoint sets of words. To deal with the extreme sparsity, you have to do some kind of dimensionality reduction or latent variable estimation. See: [Latent Dirichlet Allocation](http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) and [Latent Semantic Analysis](http://en.wikipedia.org/wiki/Latent_semantic_analysis)
Good point. "I understand p-values.  Just give me a p-value."  Ironically, they probably do *not* understand p-values, but that's another matter.
Thanks COLSerdash! I'll try to replicate your results!
Did you plan from the start to do a subgroups analysis? Or are you looking for significant findings? Because if this is exploratory, you need lower the significance thresshold to account for the fact that you are doing multiple comparisons.
I have a feeling that using the MGF might be the way to do this one.
Do you know the size of the sample from which this parblot was obtained?
@suncoolsu I suppose any good resource on table design should talk about the pros and cons of tables versus graphics.
At the very least, the sign of inference should always match the mean difference in the two groups. If it is otherwise, then you've computed the test statistic incorrectly.
And should I worry about the scatterplot? Right now there are two vertical bars full of data points at zero and one and just a few data points in between. The regression line seems somewhat lost in that plot. Should I maybe instead report a boxplot based only on the datapoints that are either fully within or without a participating country even though I used intermediate values in the regression models?
What kind of parameter is `P`? Does it make sense for it to be negative?
@EngrStudent The kinds of robust lines offered by typical robust regression routines are generally quite good but aren't quite as conceptually simple. They're easy to fit in (say) R - see `ltsreg` in `MASS` or the packages `robust` and `robustbase` for example. Which kind of estimator you need depends on what you need robustness against, how much of it you might need to deal with and how much you worry much about efficiency at some ideal model. [For example, if x-outliers are not an issue, quantile regression or M-estimation may do quite well, but if they are an issue, I wouldn't use those.]
This is an interesting answer and informative answer. But, you might be overly focused on Lebesgue measurability. Nonmeasurable sets are much more prevalent in probability theory.
**1:** No  **2:** There is no conventional prior for Dirichlet paramaters (see [here](http://andrewgelman.com/2009/04/conjugate_prior/) for a discussion). However, as noted by @Zen, you should provide more information about the data you have and the goal of your analysis if you want better advice.
The short answer is yes.  An illustration appears beneath the line "to firm up these ideas" in my answer at http://stats.stackexchange.com/questions/71260. That answer goes into (extreme) detail concerning why and how the slope and the error variance are directly related.
Actually you would need the standard error of the difference in differences. If you had a sequence of each R² measure this is feasible but not with only 4 numbers only.
An adequate answer to this question can be found by reading not a few paragraphs but by reading many books and/or taking several courses on survey design, survey bias, survey validity, and/or psychometrics.
We need to be careful with definitions. To illustrate (with an important practical example): let the sample space $S$ be the set of all countable binary sequences and let $X:S\\to\\mathbb{Z}$ be the number of initial zeros. Endow $S$ with a probability measure in the usual way (by thinking of its elements as outcomes of iid coin flips). Although the *random variable* $X$ is discrete, the *sample space* has cardinality $\\aleph_1$ which is not in one-to-one correspondence with $\\mathbb{Z}$.
Fisher certainly *did* interpret smaller p-values as representing different degrees of evidence and *did not* advocate Neyman-Pearson style testing - Fisher didn't need to have an alternative at all. See, for example, [this paper](http://www.stat.duke.edu/~berger/papers/02-01.pdf).
I'm afraid not, these values cannot be computed because they should not exist. In real life, there are no values at these intervals either.
Why not draw a man from the men and then a woman from the women, and *then randomly hand one of them the label 'first'*? What sequence effects are you anticipating, exactly?
Scree and the K1 method seem suggest that you could retain the 4th factor. Maybe clarifying the ultimate goal of your analysis and setting a performance metric for your modeling could help deciding if you ultimately should include the 4th factor.
Thanks for your response.  I've managed to calculate Cronbach's alpha, but how does one calculate Pearson correlation coefficients in this case?  I could calculate them in a pairwise fashion for each individual characteristic, but I'd like to know how to calculate correlations between groups of characteristics.  Characteristics within a group will have similar scores for each observation.  I'm going to edit my question to make this a little clearer.
@Zach I actually tried this once (https://github.com/barrycarter/bcapps/blob/f91eeabc1db03c5ee8ea372d839dc4a4d86c00e2/bc-generic-voronoi.pl), but there are too many of them, even using parallel (plus, wunderground.com appears to limit simultaneous connections), so I just do local ones for now. If there were a single source that summarized current reports for all stations, that would be useful.
AUC is not a measure of model accuracy but is a measure of model recall. Some models have perfect recall but lousy classification accuracy. You can estimate your model parameters based on maximizing AUC but expect frustrating convergence issues with almost any initializing parameter sets (even those from ML as starting points). I really can't advocate your ad-hoc method. You ought to consult the Elements of Statistical Learning for some other approaches to this problem.
Right, you can't store it all in RAM at once. So you have to be cleverer about how you process it.
Logging is often used to reduce right-skewness, so it can be expected to increase left-skewness. The `exp()` transformation is its inverse, but that is probably far too strong here. Squaring is a milder alternative. You don't say what sample size(s) you have. It is not obvious that the main problem is really left skewness, rather than a few moderate outliers in the left tail in B1. Is there no science here to throw light on this?
@Srikant - there any many ways to deal with multilevel data and Gelman is "the king" of this field. His point is that ANOVA is a simple/clear method of capturing the key features of complex and hierarchical data structures or study designs and ANOVA is a simple/clear way of presenting the key results. In this sense it's role is complementary or exploratory.
Did you not find an answer by [searching our site](http://stats.stackexchange.com/search?q=test+bimodal)? If not, what is lacking?
Incidentally, having read many of those textbooks on statistical inference I'd take your bet. Casella & Berger, Bickel & Doksum, Lehman & Casella, etc all at least discuss Bayesian parameter estimation if not some decision theory. Certainly it doesn't get the treatment it deserves but it's unreasonable to expect an introductory text to go too deep - and you can't become a statistician just on the back of them anyway. I'm an avowed Bayesian myself, but a solid understanding of classical statistics is important.
The standard deviation $\\sqrt{np(1-p)}$ is that of the number of wins, not that of the proportion of wins. Note that on the other end your average DOES refer to the proportion of wins. You have to divide that formula through $n$, which is equivalent to dividing $p(1-p)$ by $\\sqrt{n}$.
Hey, it was not really about the inclusion of seasonal component to explain the response variable, but how to handle the seasonality in the regressors.
This is a mismatched question, akin to "What color of a car should I buy to avoid traffic?" If you look up an authoritative source like Handbook of Statistics: Psychometrics (http://www.amazon.com/Handbook-Statistics-Psychometrics-C-R-Rao/dp/0444521038), it won't even mention experiment design. You might want to edit the subject.
You got part of this completely backwards: the Mahalanobis distance uses $C^{-1}$, not $C$, as the quadratic form. I think a good answer would make it clear which is the correct form to use.
Thanks for clearing that up. You're correct in saying that the conditional variance matters more, I just read this remark "restricted variance may have caused us to underestimate the relationship" too often. Actually, I often have data on the central predictor from another source, more often a vital covariate is missing. Would I follow the same procedure? I can actually in some cases model the reason for missing data (ie. dad dead, unknown, didn't show).
Apparently it is very common to use "variogram" when you mean "semi-variogram" because no one uses full variograms.
@PeterFlom Done.
Please avoid cross-posting. It can waste people's time by requiring duplicated effort, it is considered rude, and SE strongly discourages it.
I am interested to know where the problem in readability is and any suggestions for improvement are very much welcome
Can you clarify what you mean by panel and aggregated in this context? Or even better, provide some data that works in R or Stata but for which you're not quite getting the result you want. I'm thinking that what you're looking for in R is a glmm with id as a random effect and the 2 column response, but your terminology is making it hard to see if that's true.
Hope this helps, http://stats.stackexchange.com/questions/81820/why-is-k-fold-cross-validation-a-better-idea-than-k-times-resampling-true-valida/81824#81824
Downvoted.  How is the truncation issue relevant?
I made this CW on the premise that there is unlikely to be one authoritative or best answer.  (Feel free to persuade any of the mods otherwise if you disagree!)  One could argue for closing the question on the grounds that it is potentially contentious, but (IMHO) it is on topic and interesting.  However, any contentious, polemical, or unsupported replies, should they happen to appear, will be deleted without any further explanation.
Does it have to be VARMA or does VAR satisfy you?
Hint: $-\\ln (X_1X_2\\cdots X_n) = \\sum_i -\\ln(X_i) = \\sum_i Y_i$ is a sum of exponential random variables $Y_i$. Can you relate $P(X_1X_2\\cdots X_n \\leq a)$ to $P\\left(\\sum_i Y_i \\leq g(a)\\right) = P\\left(\\sum_i Y_i \\leq b\\right)$ for some suitably chosen function $g(\\cdot)$?
THANKS for reply!!  Yes, I am talking about a sample size for given confidence level. However your answer surpass my  current level, so I am not quite understand about.  The  Berry-Esseen-Bound, I first time known of, has me learned that there is such bound for CLT. To me it's a cool, beautiful bound.
Could you elaborate on why you are considering only absolute values?  That seems artificially restrictive, maybe even counterproductive.
It's in [the manual](http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_model_sect035.htm): look below Figure 18.21.
Are you saying there are 365 people and 804 variants?  So that the output would be 804*803 = 645612 three-by-three tables?
(+1) Huh, `Hmisc`... a really great package!
Thank you Glen! Nice references.
+1 There are useful hints in this answer too. Your comment is a bit general so although does not fully describe my situation but provides interesting tips to deal with difficulties in learning statistics. The level of interest in the question is advanced statistics (although some textbooks with a title including "advanced" are introductory to me) and the problem appears is the lack of suitable references for readers as non-statistics scholars. This could be helpful for my [other post](http://stats.stackexchange.com/questions/17510/). Thanks
@Tomek Tarczynski, the third derivative of $\\sqrt{x}$ goes to zero quite quickly for large $x$, but is unbounded near zero. So if you picked uniform distribution with support close to zero, the remainder term can get large.
Elements of statistical learning is likely to prove very tough as an introduction.
@Pinocchio Here is paper that you might find useful: http://theory.stanford.edu/~sergei/papers/kMeans-socg.pdf
Which degrees of freedom do you want to increase? In your $F(k-m,n-k)$, you have $n$, $k$ and $m$. You'd need to explain what these symbols stand for. In textbooks, you would find derivation of $F$-distribution by manipulating some multivariate normal densities and integrals and/or manipulating some gamma densities ($F$ is a ratio of two gamma-distributed variables).
I'm assuming it's BSc or MSc thesis. In this case, try to get a parsimonious model, fewer variables is better, so long their signs are correct.
Given that you have time series data, I think you want to do time series analysis. I added that tag so experts on the topic may see it.
Regular regression with time series data is fraught with pitfalls.
Thanks Patrick. I have updated my question to be more specific.
This question appears to be off-topic because it is has no statistical content other than finding a function/specific R package. You can try http:/www.rseek.org
Yes I guess I'm using the terminology a bit wrong given that my interest is a bit underspecified. But yes, p(a|b) would be the conditional probability of seeing a if we are seeing an occurrence of b.

Think people of one country a visiting another country b. I'm looking for relevant (probably a better term than "significant") pairs of travel origins and destinations. If some countries have a high number of occurrences as origin or destination, the individual conditional probabilities aren't as meaningful as I'd like.
Related: http://stats.stackexchange.com/questions/66948/are-there-some-nice-introductions-to-idi-nri-biomarker-validation.  (That thread may have some references to formulas.)
@Kerry The code I gave you for the step2 visualization part is buggy. The libraries you need to use are: "sqldf", "ggplot2", "reshape2". 

My idea was to visualize it using a heatmap. The problem is the way the data set "data" is constructed. I think if you use your original spreadsheet, idk what that is exactly, it should work. 

B/C there are separate columns for clearcut. ex. Clearcut08. 

If you give me more information I can help you more.
I am unclear what you mean by scaling the regularization parameter in this case. Are you suggesting that if model was originally trained using 90 cases per fold (e.g. 10 fold CV), yielding a lambda of 0.1, but then the final model, training on 100 cases would use a lambda of (100/90)*0.1? (I am referring to Friedmans regularization formulation here)
Voting to migrate to stats.SE
"Plain vanilla" GMM uses pretty straightforward approximations to the true covariance. Use of higher-order approximations (saddlepoint approximations and the like) can be used, but you'd have to code them yourself, and possibly make slightly-stronger assumptions than typical GMM to ensure you're getting the "best" approximation.
I would go for $C=0.5$ since the difference in accuracy is most likely neglectable and the model is much stronger regularized, e.g. [simpler](http://en.wikipedia.org/wiki/Occam's_razor).
It is true. Just codes in http://www.mathworks.com/matlabcentral/fileexchange/36164-unscented-hellinger-distance-between-gmms do not work at all
@whuber Thanks for your idea. My reputation is not enough for up voting yet. But OK, you are right about answers, essentially these three answers have identical results.
Add 20 values of $y$, where 10 are $0$ and 10 are $1$. Can you divide this by 20? Can you compute the s.d. $y$?
According to @Alexis (see e.g. [here](http://stats.stackexchange.com/questions/85700/permutation-test-for-factor-analysis/94921#94921)), there is "Monte Carlo parallel analysis", *"which uses a high centile of eigenvalues of random data, rather than the mean"*, exactly as I wrote above. I am just saying that you made a very strong point in your answer that parallel analysis is not a significance test, whereas there is an obvious variant of it which actually is a significance test (how well it works is another question).
Thank you very much! This is exactly what I didn't see.
Costs more than $200 on Amazon! When I see a book that costs this money, I think to myself: Aha, there is something in this book that the authors don't want me to know.
For sure, the function can be made much faster (it almost takes as much as 30 ms on my Dell Vostro) but I just wanted go give a piece of working code that was easy to use and play around with :)
All of those "the following objects are masked" warnings are because you use attach(data).  attach() is generally deprecated because you have the problem that it becomes difficult to tell which version of the data you are referring to.  From eyeballing your code I don't think you need it at all.  Removing that would at least remove some of the cluttering problems.
+1 it's a good point but while it might very well be what the OP had in mind, there is no mention of R in the question…
Thank You. I think Fisher exact test is what I was looking for.
Perhaps the sin is doing statistical tests on a model obtained via stepwise regression.
@friveroll That is something i cannot help you with.  hopefully you will find someone who can.
It sounds to me like you're talking about situations where conditional probability (probability conditional on the extra information) may differ wildly from the unconditional probability. So some probability assessments are *sensitive* to (as yet unobserved) information while others aren't.
@whuber - This looks like a variation on a transfer function.  Where can I learn more about it?  I think that a "holy grail" would be a way to relate the Transfer function of a system to variation in its output.  The delta method is a kluge, and I think there might be a better answer from perturbation methods.  I would love to see it, and get to know it well.
(-1) It is a bit upsetting that the accepted answer to this question is neither useful (e.g. too vague and not implemented) nor scientifically correct (as in not even wrong: as far as I know, it is not even mentioned in the relevant literature).
To be honest I'm not sure this decision is actually a statistical issue.
Do you really need to do it using web application?
Thanks! Your reasoning seems simpler than how I was approaching the problem!
$r$ is the slope of the regression line when both variables (ie, $x$ & $y$) have been standardized first. If all you have is $r$ & $N$, that is the best you can do.
I crossposted the question [here](http://stackoverflow.com/questions/14610377/classifying-handwritten-digits-using-pca) to receive hints about its matlab code. Apart from that, I need ideas as to what should I do about the problem in general.
but this does already imply I fitted an arma11 model!? I want to do this diagnostics before fitting, it does not have to do with the residuals. I want to do this before model estimation, to check the significance of the serial dependence and to identify the correct model.
It is not immediately clear from your description that the integral would exist in any reasonable sense, much less that one would be able to take an expectation. Why should, for example, $\\{t: X(t,\\omega) > 0 \\}$ be a Lebesgue measurable set for any fixed $\\omega$? Perhaps I have missed some subtlety in your description.
Not an answer (because I don't know) but I just saw this new book on the Springer web site: "A Comparison of the Bayesian and Frequentist Approaches to Estimation" http://www.springer.com/statistics/statistical+theory+and+methods/book/978-1-4419-5940-9 that (one would assume) may have some answers.
I am guessing that there were 30 aphids on a card, so the spike at the right represents "$\\ge 30$ aphids would have been eaten but only 30 were available" rather than "30".  Is this right?
A useful literature to start at is the financial contagion literature. Take a look at Forbes & Rigobon (2002) in the `Journal of Finance` and the papers that this research has spawned. (This is dedicated research into whether correlations change during crisis events).
The standard approach to estimating the parameters of a mixture of normal distributions is to use an Expectation-Maximisation algorithm.
Hint: The Binomial theorem asserts $(1-s)^{-1}=1+s+s^2+\\cdots+s^n+\\cdots=\\sum_{k=0}^\\infty s^k$ (in the sense of a formal power series). Use this (with $s = 0.8B$) to evaluate the formal solution $X_t=(1-0.8B^{12})^{-1}(1+0.5B)\\varepsilon_t$.
Can the mean ever be equal to the median?
Which page of the 165 pages? BTW he also has book chapters available.
I think the interest is more in risk than in classification.
Hint: $u=ax^2, v = -e^{-ax}, dv = ae^{-ax}$, $\\int u\\, dv = \\cdots$
Wonderful exposition! You may be interested to know that this appears to have been investigated back in the 1920s. See [Tippet 1925](http://www.jstor.org/stable/2332087). In Tippet's tables (Table X) the expected value for the range given a sample of size 6 is $2.53441\\sigma$. He shows the derivation of the complete distribution of the range for the normal distribution. This was used by [David et.al. (1954)](http://www.jstor.org/stable/2332728) to calculate probability points of the range distribution for a test for normality (see D'Agostino & Stephens 9.3.3.4.2).
Thanks, I will broaden my question.