tttjjj/elt_bench huggingface dataset:
Thanks for the link! This is a great paper, addressing exactly what I needed. Thank you! :-)
@whuber I don't agree completely with your reasoning, because you can say the same for any least squares regularization. The problem with least squares is that it is the optimal fit for the given data but doesn't necessarily generalize well. What people do with regularization is create some new objective function so that the solution will have some desirable properties. The challenge is to pick an objective function that will both desire what you want and be efficiently computable. I will write this more explicitly in the question.
Rick, thanks  for the book, but sadly GoogleBooks gives only [few words](http://books.google.com/books?ei=dcWoTqavB4_1-ga24qy5Dw&ct=result&id=L1SqAAAAIAAJ&dq=Plots%2C+Transformations%2C+and+Regression+An+Introduction+to+Graphical+Methods+of+Diagnostic+Regression+Analysis+A.+C.+Atkinson&q=percentages#search_anchor&safe=high)
Awesome, finally got a chance to go through the steps here and make it clear to myself why this is. I've also added some edits to clarify things for future reference.
Unfortunately, I don't know enough of Matlab to perform such task and I would personally rely on R or Stata for that. Regarding GEE vs. GLMM, a [good summary](http://stats.stackexchange.com/a/16415/930) was provided on a related thread.
That is different from the common statistical meaning which has to do with inverse regression.
They _are_ different than multilayer networks trained with backprop. Here are some resources for all kinds of neural nets: http://stats.stackexchange.com/a/36257/13669
Alright thanks. So my last step for this is to solve for those two bivariate independent normals?
The bifactor model converges when removing one item. But the fit is still really bad and the factors still correlate highly. I think my options are exhausted after all. However, we are collecting more data to have more reliable estimates. Thanks for your reply!
Welcome to the site, @Deb. Is there a *statistical* question in this? I acknowledge that there seems to be a methodological aspect to your question, but I would say it is of the kind that should be posed to subject-matter experts in your field instead of statisticians. If you have a substantive statistical question, please edit to clarify. Otherwise, this Q may end up being closed as off-topic for CV (see our [help page](http://stats.stackexchange.com/help)).
I quickly checked the paper and could not find the definition of $\\delta$. I believe this symbol is not specific to EM algorithm, so it would be helpful to contact the authors directly.
@AkselO There is the well-known closed form expression for the OLS estimator, $\\widehat{\\boldsymbol{\\beta}} = (\\mathbf{X}'\\mathbf{X})^{-1}\\mathbf{X}\\boldsymbol{Y}$, which you can compute by explicitly computing the inverse of the $(\\mathbf{X}'\\mathbf{X})$ matrix (as @ ocram has done), but this gets tricky with ill-conditioned matrices.
Maybe 10000 is just not enough to establish a significant decision? It does not seem large, especially if you compare it with the number of all possible 1000-bit words.
@Wayne I've made edits based on my progress with the work so far.
@danielberger: also, if you don't have access to any of those papers I can send them to you in PDF.
It doesn't cover Bayesian classifier, but a starting reference that I always use is _The Elements of Statistical Learning_ by T. Hastie et al.
Your reformulation of the question is exactly correct. I think the multivariate calibration is what I'm looking for. However the initial coefficient matrix is not completely correct so I have to find a way to optimize it by an iterative procedure. But before that, thank you very much.
Your question doesn't contain enough information for anyone to guess what you're talking about. There are a variety of things that might be printed along with a p-value.
A bit confusing, but I think I understand. My hypothesis is that **priming improves creative performance**. So if p>.05, d=0, the hypothesis is rejected, means **priming has no effect on creative performance**. Also, p value and effect size are 2 different things. If I have large enough sample size, hypothesis testing is always significant. Therefore, even if I found a small effect (d = 0.2), as long as p>.05, my hypothesis (**priming improves creative performance**) is rejected. Am I correct?
A hard bound on the Poisson model is as follows. Let $Z_1 \\sim \\mathrm{Poi}(2t/(n+1))$ and $Z_i \\sim \\mathrm{Poi}(t/(n+1))$ for $2 \\leq i \\leq n$. All the $Z_i$ are mutually independent. Then $\\mathbb{P}(Z_1 > \\max_{i\\geq 2} Z_i) \\geq 1 - (n-1) \\exp(-c t / (n+1))$ where $c = (\\sqrt{2}-1)^2$. As you can see, it only works well for $t \\geq 6 n \\log n$ or so.
It's not quite right to say your evidence doesn't show that the means are not equal: your evidence doesn't show that the mean of the first group was greater than the mean of the second group. As you chose to do a one sided test, you can say nothing whatsoever about whether the mean of the second group is greater than the mean of the first group.
@Tim If you want to find out precisely how you can use the approach I implemented, you can email me at dave@autobox.com
Thanks IrishStat. I am looking the specific details in R.
You have one of my ideas in there, when number are small which is common in studying low incidence rates differences look small but ratios still look large. Your numerical example is very compelling. I am tempted to add something about the stability of the estimates under the null hypothesis. For some this may be too technical but at her level of sophistication maybe not. Suppose the two populations have nomral distributions mean zero and known common variance. Then the normalized difference is N(0,1) under the null hyothesis giving a very stable test statistic.
+1 for LKJ distribution
Pretty much all charts make comparisons, so I don't think I'd advocate that name - although in fairness there are certainly other  chart names I'm sure you can trawl up that are unwarranted.
If a numerical answer is OK: 

 - Generate large $R$ vectors $V$ where $V_i \\sim N(\\mu_i,\\sigma^2_i)$. 
 - Sort the data, retaining a copy of the the original indices.
 - The probability that $V_i$ is the $i^{th}$ order statistic is the number of simulations in which $V_i$ was indeed the  $i^{th}$ order statistic, divided by $R$
Presumably "HPD" = "Highest Posterior Density".  The interval you find should extend from $0.04713$ to $0.3946$.
(+1) Excellent question!
At the end of the day, I would suggest that it is actually more informative to plot the partial dependence function as it exists in the randomForest package (you can change to the probability scale like I showed). This to me is much more informative for a non-linear model.
As you mention Principal Component Regression, in R that is provided by package pls. As for the number of components to retain, I don't see any real advantage of deciding % variance over no. of components (maybe that is because I work with data that have very different levels of noise. As @Marc-in-the-box mentions, there are lots of different approaches to determine an appropriate no. of PCs, and the strategy will (should) depend both on the type data and on the type of data analysis that is to follow.
Setting this up as a split-sample problem is a complete disaster.  You are trying to learn from a woefully inadequate training sample size.  The results will be unreliable.  Intensive resampling (e.g., bootstrap, repeating all analytical steps using $Y$ at each resample) will work far better.
When working w/ novices, it's helpful to keep explanations simpler even if you sacrifice a little nuance & accuracy; however, describing what p-values are as telling "you if there is actually a relationship between the independent and dependent variables" may be a bit too far in that direction for comfort.
Are you referring to a particular paper? I imagine I could find an answer to my question if I researched, studied, read a lot, but so could 95% of the questions other people ask here... Also, in some cases, particularly with macroeconomics data (which is my area), there is no more data to be collected. Data is scarce (the number of observations, I mean), and you just have to live with it. There is no "get more data" solution. I was hoping someone here would know the topic, but it doesn't seem like. Maybe once the website is opened to the general public?
As @miura rightly emphasises, funny things can happen, but these results seem perfectly consistent with relatively low $R^2$.
@inzl: You're welcome! Sorry that I couldn't be of more help, but, at least, we both (me, especially :-) learned something new. What about time series analysis (http://en.wikipedia.org/wiki/Time_series)? Have you checked that for appropriate approaches and/or methods? Just a guess.
I disagree with Xi'an's statement that this derivation is called Bayes' Theorem.: it is the law of total probability.  Bayes theorem would be involved if the problem asked for $P(B|A)$ which "turns" the conditioning around.  $P(B)$ is playing the part of the prior probability, $P(A|B)$ corresponds to the likelihood, and $P(B|A)$ to the posterior probability, and the Reverend Thomas Bayes gets involved. In this instance, the Reverend is not to blame.
+1: i too learned from Lehman (very good reference), but this one is not mentionned nearly enough
Great insight, many thanks. Why would you laugh out loud? Yes internally!
A journey is a sequence of visits by an person (ID). In the example, the first journey has got 3 visits after a purchase is done. The second journey, a journey with no purchase, consists of 1 visit, etcetera
+1 Thank you for your help! Much appreciated =)
@whuber, a great comment. Leland Wilkinson, the primary author of Systat, and also the author of [The Grammar of Graphics](http://www.amazon.com/Grammar-Graphics-Statistics-Computing/dp/0387245448) (which is more of a computer science book than a statistics book), has a very fine taste for graphical display of data. His presentations are always very effective in delivering their messages.
(+1) Thanks for the link and advice on model selection issue.
It would be nice if you explain, and thoroughly, what is "variance of a point/observation". Without it, a reader may feel not satisfied and object: how can a single observation of a sample have its own variation measure?
The transition matrix (or transition kernel in the continuous case) represents conditional probabilities/densities, while the copula represents a joint distribution; to convert that to a transition matrix, you'll need to divide the joint by the relevant margin.
Helpful, but underlines my point that this field is slightly contentious. For example, I suggest that 4) confuses two distinct issues. PCA has no aim except to re-express all the variability in terms of orthogonal components. It's users of PCA who often (but not always) also hope that a few components work to "explain" most of the variability.
What is a "criterion variable"??
Centering *doesn't* influence the interaction. Standardizing (mean-centering and dividing by the *SD*) will change the scale of the interaction's regression coefficient (not the size, mind you; it just removes the units of measurement and places the coefficient on the scale of $\\rho$), but won't change its *SE* or NHST statistics.
See the `fitdistr` function in the `MASS` package -- especially the second example in `?fitdistr` ...
I'm too afraid of the hardcore statisticians here to post a real answer, so I post it here:
I was told that if you want to normalize your data $y \\pm \\Delta y$ to $z \\pm \\Delta z$

$$x = 100 \\times \\frac{y}{z},$$

you have to calculate the standard deviation $\\Delta x$ as follows:

$$\\Delta x = x \\sqrt{\\left ( \\frac{\\Delta y}{y} \\right )^2 + \\left (\\frac{\\Delta z}{z} \\right )^2}$$
Maybe you can take it from there, but I'm not sure if this helps.
Actually at the end I would like to understand Particle Filters. I am trying to understand the how nonlinearity and non-gaussian setting brings MCMC to table.
When you left-multiply the eigenvector equation $(XX')b=\\lambda b$ by $X'$, factor the scalar $\\lambda$ through, and write $e=X'b$, you get $(X'X)e=\\lambda e$, immediately obtaining the Chatfield & Collins results.
+1 For pointing out how an exploratory analysis naturally leads to interesting *questions*. I wish I had emphasized that point more in my answer. Although I think it would be pushing things to believe (at this point) that there really are three distinct groups, the cluster results still present a valid way of *seeing* that there is a negative relationship between x and y and of summarizing that relationship. I am led to wonder to what extent automatic clustering could be a generally useful exploratory tool--provided we are not tempted to read too much into the results.
Hello, I edited the question to be a bit more clear of what I'm looking for. I hope that helps. Thanks, for the link BTW.
ditto. I usually put it in terms of 52 cards, and the goal is to find the ace of spades.
@suncoolsu I never tried those packages; so my contribution is just some googling (because I'm interested in using HDP in the future, but have no time for the moment -- this is the reason why I made my answer CW). I guess someone more used to those methods will provide insightful remarks.
+1: How many times I've been reading or been in a class and heard people who believe that if they have a complicated-enough algorithm, they could get rich in the stock/electricity/commodities markets. You try to explain overfitting, etc, but to no avail. Heck, as far as I know, not being an insider, successful stock-trading software has depended on no transaction fees, arbitrage, and high speed. The cutting edge now is to use loopholes in automated trading rules and high-speed proposal/withdrawl of bids to sucker-punch other automated traders.
Related: @Gavin-Simpson s answer here http://stats.stackexchange.com/questions/70619/dispersion-in-summary-glm
**Hint**: $1/\\bar{X} = n / S_n$ where $S_n = \\sum_{k=1}^n X_k$. Now, what is the distribution of $S_n$? From that, can you find $\\mathbb E (1/S_n)$? Use linearity to conclude.
I actually used the full data set to generate the model. With respect to new cases, I applied K-cross-validation. So. Is there a way to improve my coefficients after that?
We maximize the log likelihood because statistical theory has proven this to be an optimum objective function in several senses, when there is no prior information about model parameters.  To maximize something else implies that you have a special loss/cost/utility function.  What is your utility function?  And what is magical about a decile vs. an octile vs. a quartile for your purpose?  This seems arbitrary.
Can you give a description a problem where RNB gave you good results?
If you have good reasons to believe the theoretical model, can you look at the errors in the actual data minus the corresponding data from the theoretical model? Is there a pattern in the errors, evidence of additional variables, etc?
With "gap", do you mean insertion-deletion costs? Actually, TraMineR do not support varying indel costs (if this is what you mean).
(2) What does the midrange have to do with either of the interval endpoints? You ought to rethink this one.  (3) is trivial because the distribution of the midrange is symmetric under the transformation $x\\to \\theta_1+\\theta_2-x$: this transformation would negate any bias but it does not change the distribution of the midrange, showing its bias is equal to its negative. (4) What is the expectation of the sample range? If it is a function of $n$ and $\\theta_2-\\theta_1$, you should be able to adjust the sample range to have zero bias.
@Manuel You are correct; you need to assess individual variability.  The number of repeated measurements needed depends on the size of that variability, how the variability translates to uncertainty in the inferences about the population, the cost of repeating the measurements, practical constraints such as the time needed for replication, and (perhaps) technical issues like the possibility of positive temporal correlation among the replicate measurements.
It looked like count data from the distribution so I thought Poisson. Even if it's not count data it seems a log transformation could be appropriate and should produce comparable results if glmer uses a log-link for the Poisson model.
To the bootstrap comment I was referring to something like what Michael Chernick mentions about small sample sizes in the bootstrap here:
http://stats.stackexchange.com/questions/33300/determining-sample-size-necessary-for-bootstrap-method-proposed-method
In addition to @whuber's remarks on notation, there are also the [*natural parameters* of the normal](http://en.wikipedia.org/wiki/Exponential_family#Normal_distribution:_Unknown_mean_and_unknown_variance), which probably look quite *unnatural* to most, though a very good reason exists for calling them as such.
But I cannot fit a normal distribution to the data using the estimated mean and variance, or even an Winsorized variance. Is the data really truly normal? Why does it appear 'taller' given the estimated variance? In the shapiro test, I am only use a sub-sample of the data. If the problem is truly due to large sample size, should sub-sampling not solve the problem?
community wiki?
If your data is such that the mean can be zero, then this is not a good solution. I.e. it is not meaningful to divide by the mean. This is even if the actual mean is not zero. E.g. you average something like the "x-position" of points. The mean here can be zero, and I'd argue dividing by the mean is not meaningful, even if in your particular sample the mean is not 0.
No problem. I posted this answer to complement Mike's advice. I was just trying to quantify 30 points rule, i.e. if you have central limit theorem, then $n=30$ gives you 0.1 error. This is a very bad statement, since I do not speak about what is 0.1 error, but for thumb of rule it works. The main point is that the approximation error for assymptotic results where CLT is employed (which is the case for maximum likelihood) is always $c/\\sqrt{n}$, where $c$ is the constant depending on the distribution.
If you're interested in what kind of bizarre application this could be... it's not really a histogram of a distribution on $\\mathbb{R}$. The distribution is actually in $\\mathbb{R}^3$ and it represents a data-cube which records certain properties of ion impacts in a scientific detector. Some parts of the data-cube get lots of counts and others get very few. My interest is in simulating the data-cube realistically. (You don't really see $N= 10^{30}$ ions, that's just for the debugging process I mentioned above).
Andy: Yes, it is locally weighted regression.  I found that for these simulated data a window of about 20% of all the x-values gave good results. But just about any robust local smoother ought to work, such as Tukey's 3RSSH.
+1 (nice answer) and welcome to CV!
Thanks for the edit, Christophe.  It makes for clearer presentation.
A big concern with using PCA this way is that *all* the relationship between the response variable and the independent variables could reside in the 3% of total variance that you neglect. There does not seem to be any general way to determine how many principal components to use, either, because *the very smallest component* could be proportional to the response itself and would thereby constitute the optimal choice of variables to include.
Can you show *exactly* what you did to get a value greater than one? With more details we can probably identify the error.
This question currently states two bug rates for Mary. I assumed and submitted an edit to the question that Jane is the best programmer and full-time worker with $\\mu=0.5$ and prior 0.5.
Yes, it is because of the continuity correction, do chisq.test(matrix(c(35,11,44209,44233), nrow=2),correct=F) to do the test without it. Also if I remember correctly the matrix upper row should contain observed, and the lower should contain expected.
I've flagged your original question to recommend its migration. Your question doesn't seem to be a programming problem, but a statistical problem, so it does belong here. So does [the answer](http://stackoverflow.com/a/20943695/2829605) you've already received, which I voted up; I would've given the same answer myself.
Actually it doesn't make sense to me either. But I noticed such analyses in some other projects for each individual statements and wondered how they did it.
@Aniko: Please could you indicate how I could get a $\\chi^2$ p-value for each of the two outcomes per bin? I need to know if there is a difference in the counts for each of the two outcomes in each bin.  (I've asked Jim as well, as the two solutions are different.)
Oh, and BTW, you could also use plyr with a parallel computing on several cores (almost automatically), see: http://www.r-statistics.com/2010/09/using-the-plyr-1-2-package-parallel-processing-backend-with-windows/
I don't think this system of equations is identified. It it was, however, one might use 3SLS (Three Stage Least Squares).
@AdamO The t-test is UMP *assuming* normality.  That claim no longer is valid (in general) if there is any departure from normality in the underlying distribution.  One frequently cited reason for preferring the Wilcoxon test is that it cedes only a little power to the t-test for normal distributions yet, for many other distributions, is *more* powerful than the t-test.
(+1) Thanks for the hint about `nclass.*`. Why isn't your left-most interval left bounded (should set `include.lowest` to TRUE maybe)?
+1. The "moral" of the whole story is that the difference between the only centered data $\\bf X$ and the standardized data $\\bf X \\Lambda$ is absurbed entirely in the eigenvectors. So when the data are multiplied by the corresponding eigenvectors to produce discriminant scores, the effect $\\bf \\Lambda$ of the standardization cancels out.
I would also like to apologise for so many questions over the past week, but I really feel that I am learning so much where books have failed me. I have resolved many of them and now assume that the reason that PIVs don't always correlate with the Estimates is because I wasn't taking the first estimate that is unseen into account. Therefore the answer to this question would really solve any remaining issues that I have had. THANK YOU all.
Could you state your exact hypothesis? It almost sounds like you're trying to compare whether the maximum of several readings for each subject, over time, is greater than a baseline reading for each subject.
Can you point out which page contains the interpretation you don't understand?
Yes. Part of this is obvious and part is subtle. The obvious thing is that whatever value of $\\theta$ maximizes the likelihood will correspond to the value of $3-7\\theta$ that maximizes the likelihood, because the function $\\theta\\to 3-7\\theta$ is one-to-one. You're not changing the likelihood; you're just changing how you express the parameter on which it depends--kind of like converting between pounds and kilos. The subtle parts in real applications are (1) you better check that the correspondence really *is* one-to-one and (2) confidence intervals may change in more complicated ways.
would had that poisson uses a log link, so you want to take that into account when interpreting coefficients.
Because the two events (first light being red, second light being red) aren't independent.
I have no experience with scikit-learn, hence just a hint on the base of rapidminer-experience: 1. Integer values can be treated as categorical or real-valued. 2. Chi2-Feature-Selection on real-valued features most likely requires a discretization beforehand, hence if the integer is treated as real-valued, a discretization is also performed here. I suggest to look into the source code.
I am not an expert on this.  You probably know a lot more than me and maybe have read Michael Stein's book where I have it collecting dust on my book shelf. The point that we both can agree on is that figuring this out requires more details from the Matlab writeup and/or more experimentation with the data