tttjjj/elt_bench huggingface dataset:
Welcome to the site, @Diane. Please don't write in all caps, this bothers many people, & it is harder to read. Is this homework? What is *your* question here? Please read our [FAQ](http://stats.stackexchange.com/faq), which explains site policies regarding questions, homework, etc.
+1 This is an interesting and (seemingly) powerful approach.  Thank you for sharing this idea.
In many algorithms it is not necessary actually to invert a matrix.  Therefore, please tell us *why* you wish to invert these matrices.  It would also be useful to know how such incredibly ill-conditioned matrices arise, because it is possible your original problem could be solved in a different way altogether or it would at least suggest alternative algorithms.
This other [xkcd page](http://what-if.xkcd.com/34/) may be of interest.
Gibbs sampling can be used for discrete distributions as well, under appropriate considerations. This in mentioned, for instance, in the [wikipedia entry](http://en.wikipedia.org/wiki/Gibbs_sampling#Implementation), "Implementation" section.
The link to download the R web interface installer can be found at: http://rstudio.org/download/
@alto So, if I understand it right, we can interpret P(T|X) as "given X, how possible I will see T". But I am still not sure how to utilize it in classification task.
@EmilFriedman I agree it is true that we can unambiguously decide between hospitals. But certainly the data supports one over the other. (It is not true that the data has taught us nothing about the quality of the hospitals.)
's sounds like a good plan.  I'd add that the power problem would stem not just from having continuous predictors being multiplied, but from having "social science" (i.e., error-prone) variables.  Limited reliability gets compounded when multiplying two such variables.
@John Nothing of what you say comports with my understanding of these estimators, but perhaps that's because nobody has yet stipulated *what* is being estimated, nor have they clearly stated any distributional assumptions except that the distribution is "highly skewed."
Did you have a population measure of association in mind that you want the new sample to have (presumably one like the sample measure on the target sample)? Or did you perhaps have a sample measure of association you want to reproduce? Failing that, can you be more explicit about what you really mean by 'associate ... like in the table'?
@user603 Your reasoning is good but your [facts are off a little](http://en.wikipedia.org/wiki/IEEE_floating_point): for floats the minimum (non-denormalized) absolute value is $2^{-126}$; for doubles it is $2^{-1022}$.
@whuber I have  incorporated your comments in my revised answer. I wrote out the location-scale transformations because the OP seemed to be not able to recognize that the different means and variances are easily handled, and I agree with your comment on your own answer that the idea applies not just to normal distributions.
Linear DA assumes data in each group is multivariate normal and of the same variation, in population (see [related](http://stats.stackexchange.com/a/71571/3277)). But this is important for classification task of LDA. Extracting discriminants itself needs no such assumptions.
To Get Random Rows Fom the SQL DATAtable Or DATASET follow below ling http://niravdaraniya.blogspot.in/2013/07/how-to-generate-random-rows-in-dataset.html
And also here
http://codeoverflow.net/2013/07/23/how-to-generate-random-rows-in-dataset-using-c-net/
Don't feel obligated to accept. The regular expression approach is probably your best bet unless you want to get into a more sophisticated natural language processing algorithm...but I suspect at the scale that you're working, developing that will be more time consuming than doing it manually. It's a great problem/question, but I suspect the best answers relate to data producers releasing better metadata rather than end-user solutions for dealing with poorly aligned surveys.
As defined it is usually meant for uniform distribution on $[0,1]$, but you can use the same inversion methods that you use with pseudo-random numbers to get to other distributions. (But you must stay with inversion, most other methods for generating non-uniform distributions will be invalid).
What exactly do you mean by "effective rating"?  If it reflects expertise, why would this have anything to do with an amount of work allotted, which has little or nothing to do with expertise?  What would it mean to rate a user "relative to the group"?
Jason, thanks for the quick response. Splitting the data into two charts revealed exactly what I'm trying to show: the % share of referral source A is dropping month-over-month while the % share of referral source B is rising, in both visits and pageviews. After splitting up into two charts, the trend of each source seems to be effectively communicated in the stacked bar chart, but I'll also try a line chart for comparison. And I'll also have to swing by the library this afternoon :)
Thank you for all your time and effort! I will investigate those ideas.
@DilipSarwate thanks for the question. Clarification added.
You should carefully study the result and the proof you obtain. In particular, you should try to understand why the restriction $\\delta < 1/2$ is completely superfluous and why the slowly-varying condition need only be satisfied by *one* of the two $\\overline F_i$ in order for the result to hold.
+1 for "YOU MUST NOT tune the model any further!"
It doesn't seem OK to me, as 5 out of 10 isn't the same information as 500 out of 1000. Express the response as one count of the no. "successes" & one count of the no. "failures".
This paper might be relevant given that foreach is effectively a map-reduce operation: http://books.nips.cc/papers/files/nips19/NIPS2006_0725.pdf
Probably better asked on meta.  People feel like they have to downvote it because it is an off-topic question.  But then the downvotes make it look like staticians have no sense of humor :(
Or feed the results into yet another ML algorithm ;)
@chl: Ah yes it is the interval... So I should use +/- 0.32. For the calculation of the confidence interval I used the standard deviation.
@RobertKubrick In the last paragraph, I'm pointing out that the key difference in estimators is how we represent the conditional variance term $\\mbox{var}(Y|X)$. In the linear regression model, we consistently estimate the residuals, but with the sandwich, we just use a plug-in estimate of the conditional variance for the $i$-th term using the squared residuals. In the presence of heteroscedasticity, points with relatively large squared residuals have a corresponding large estimated variance and this reduces their influence on the standard error estimates.
Actually smoothing is probably the wrong word, i mean fitting!
**Hint:** A uniform prior on (0,1) is also a Beta(1,1) prior.
OK, but you said "what does it stand for?" and I answered that. Perhaps you should make your question clearer.
+1 I like the second point (I use roxygen + git). The first point makes me think also of the possibility to give your code to another statistician that will be able to reproduce your results at a later stage of the project, without any help.
Before you start modeling a special term for `1`, make sure that there is a logical reason for the `1`s to be over-represented. When you figure out why are there too many `1`s, you might find the most appropriate model.
Even though Gung's book is very lucid & covers a lot of topics on Panel Data analysis, but your book fulfill my basic requirement of being short. Also, it covers majority of Panel Data methods.
Presumably you have two small *samples* of much larger populations or processes, for otherwise applying a statistical test makes no sense. If one of those samples looks strongly skewed, and you believe that implies the population it represents must be skewed, too, then please pause to ponder why you are comparing variances in the first place. You already believe the population distributions differ (in shape) and the variance has a different interpretation for a skewed population than it does for a symmetrical one. What, then, would the point of testing the variances be?
Could you post a snippet of your data, so we have a better idea of what you want to manipulate?
Right! This is kind of the idea. Maybe should put up a wiki toward which everyone can contribute. (My next project once I'm done with my dissertation :>) Was never aware of Crofton's method btw. Pretty cool!
As long as you properly cite the authors and use already published data only, it is legal.
Thanks for the illumination, that's quite a detailed answer!
Wrong about the Binomial.  Example: if a person has a constant 0.1 chance of visiting a website each day then the number of times they visit in June is Binomially distributed with p=0.1 and n=30.
Good point, cardinal. I was thinking of this in the case where $X$ has an arbitrary distribution but didn't think of the case where there is mass at 0. So, of course, the logic above only applies when $E(1/\\overline{X})$ exists.
@cgillespie: You're right, edited :)
This looks like homework - if it is could you add the "homework" tag and it will help people in giving a useful response.
For example, the same error is generate via `rf <- randomForest(Species ~ ., iris, sampsize=c(51, 1, 1), strata=iris$Species, replace=TRUE)`
In your notation, what is $p$? The length of $\\theta$?
Very thorough answer. Thanks!
(+1) This answers one variant of the question, which interprets "fix" to mean "eliminate entirely": it finds (approximately) the power to distinguish a 0% failure rate from a $100 p$% failure rate.  That is why the sample size is much smaller than those estimated by @Thomas Levine, who interprets "fix" in the sense of "better than the original," (but maybe not perfect).
Thanks also for the Slutsky Effect. Not listed in Wikipedia, but when I found a PDF of it, the explanation was pretty clear. (The direct application is to moving averages introducing autocorrelation, but I get the idea.)
@NeilG, I indulged my addiction to *italics*.
First of all thanks for your long and careful answer, I really appreciate that. As for the example, it is very interesting, but I am struggling to understand why the process is ergodic: the notion of ergodicity I have in mind, coupled with stationarity, implies strong mixing, i.e. asymptotic independence of the terms of the sequence, which in this example seem perfectly dependent.
Hi and thanks for the quick reply. I can run a chi square test on the average values - can you run a chi square on averaged values? Or do you mean I can run a chi square regardless of group size etc? Thanks, Roddy.
Ok. if you don't have a strong pre-data assumption that there should be clustering you could do an LR test comparing the two models to estimate if the random effect improves model fit. Mathematically this is painful. Not sure how {R} deals with those issues. Most stats packages will provide you with some attempt at an estimate (usually conservative).
Are the two continuous-time Market Chains independent? If so, you can *combine* them and get a single CTMC with four states, of which you are interested in only one. This is related to *superposition* (and *thinning*) of Poisson processes. The rates do not need to be equal to employ this approach. :-)
@user26221 your question is unfathomably cryptic. Andre has done you a service by at least identifying the *package* you're talking about. What function did you call? What figure do you mean? What does it look like? What are you attempting to achieve?
If you multiply each average by 8, do you get total numbers of times each face comes up?
This is one difference comparison with linear regression (Gradient descent), right? Thanks
Thanks for your answer. Yet from wikipedia: A statistical model is identifiable if $P_{\\theta_1}=P_{\\theta_2} \\Rightarrow\\quad \\theta_1=\\theta_2 \\quad\\ \\text{for all } \\theta_1,\\theta_2\\in\\Theta$. (So it defintion relies complelty on what is $\\Theta$). I do not see how your definition allow to discuss the identifiability of a HBM. Thanks in advance.
10 million purchases per month, about 1 fraud every 5000 purchases.
Dirichlet mixture models don't generate a hierarchical structure, so without spelling out what the algorithm you are referring to is (e.g. by providing a reference) I can't really help. Alternatives which genuinely *do* put a prior on hierarchical structures include (a) Dirichlet diffusion trees and (b) Coalescent based models such as Kingman's Coalescent. These models are *very hard to fit* (this is an area of current research) but are relatively intuitive.
I don't know what all the different types of trees are. Just a quick practical remark from my Machine Learning experience: I think that the most common and stable solution from the listed is Random Forest.
I'd add the UCLA website too http://www.ats.ucla.edu/stat/stata/
I will have to give it a test comparison with the stata results, but this looks exactly like what I was hoping for.
Although this approach is preferable, there are plenty who would just throw a mixed ANOVA at this situation and call it a day.
Thank you for this. McKinnon et al call this case as "inconsistent mediation", where one of the paths,"b", is inconsistent with the whole mediation model.
In the statement $y=\\mu + e$, $\\mu$ is a _realization_ of a random variable and the allegation that $y \\sim N (\\mu,\\sigma^2)$ is not quite true. $ N(\\mu,\\sigma^2)$ is the _conditional_ distribution of $y$ _given_ that another random variable (the unknown mean) took on a specific value. Thus, $E(y)$ as you write it is actually $E[y\\mid \\mu]$, the _conditional_ mean of $y$ given the value of $\\mu$. Bayesian methods deal with finding $f_{\\mu\\mid y}$, the _a posteriori_ distribution of $\\mu$ from the _conditional_ distribution $f_{y\\mid \\mu}$ and the _a priori_ distribution $f_\\mu$ of the mean.
Could you kindly elaborate on "the whole range of inferential solutions"?  That would help the OP (and me!) to better understand how you see the contrast with classical methods.
Thank you, works great!
Can you clarify your question? What do you mean by "translate the relative positions"? Do you mean rotate? NB, if you are asking for java code, this Q would be off-topic for CV.
So perfect you should write it as an answer yourself!
If you save a file as csv and import it to R, blank cells will be represented as NA.
Thanks for the excellent answer! I am going to run though what I did to make sure that I understand - I will try and post tomorrow morning. Thanks!
Welcome to the site. Your signature is automatically added so I deleted the one you typed.
I have a general answer in preparation, but some more details might help. 1. Where is this formula from ? Can you provide a link / source ? 2. What is x ? 3. What do you mean by "$P_d$ is normalized $1/d$" ? That $P_d$ is constant across all $e$s ?
Hi Max, welcome to CrossValidated. I have taken the liberty of fixing up your mathematics. Please check I have not changed the sense of it. If this is for some subject, or even just for personal study, you should add the self-study tag.
Besides the things @whuber mentions, adding data can make the likelihood surface "nicer", which means typical algorithms should converge much more quickly. In small samples convergence for GLMs may sometimes be slow because the surface isn't a nice, nearly-quadratic-in-the-parameters thing. As sample sizes get larger - especially if you have a canonical link function, so the likelihood is just a function of some simple sufficient statistics - it may be quicker not just in iterates, but possibly even in time.
I don't have the probabilities stored anywhere. I have the NxN error/dissimilarity matrix, and a vector of ground-truth labels. I've used the ground-truths to provide values for TP, FN, FP for each class at each rank.
At times you know there were festival events that might have affected the series you put binary variable - 1 if there was a feast, 0 otherwise. Then you will perform regression with that variable included. It will estimate the impact of the festival event on the sales.
The $K$ centroids lie in an at most $K-1$ dimensional *affine* subspace.  For example, two points lie on a line, a $2-1$ dimensional subspace.  This is just the definition of an affine subspace and some elementary linear algebra.
Excellent answer, just one point to make: Using the word "obvious" in contexts like this seems to me a mistake. If it is obvious to a reader, then the reader knows it is obvious. If it is NOT obvious to a reader, then he/she may feel insulted.

I think words like "obvious" should be deleted from our "teaching vocabulary".
It would, but as I mentioned, you can derived the same estimate in Bayesian inference. The interpretation is different of course.
@SLD if you need the pdf, you need to modify the question to ask for the density rather than the distribution. The distribution *is* the cdf. But for the pdf, you shouldn't need tables, since you can just evaluate the function. If you use R or Excel you can get the thing for whatever value you need.
My problem is kind of like modelling hand strength in poker. If I feed it some statistics about pre-flop play and it tells me that QQ is better than AA, I know that something went wrong in training (insufficient data, bad model, etc) and it is good to force this condition on the output.
From what I found myself now, setting one of the coefficients above to zero will result in weak exogeneity, implying that respective variable does not Granger Cause the other. Weak exogeneity is permissible for a VAR model. Strict exogeneity would rule the VAR inappropriate in the first place.
I doubt that KS test can be used to show distributional equivalence.
(...) (c) Finally, IRT models are often used to discard items (in the spirit of *scale purification*) based on item fit statistics and the like. What's your opinion on that approach?
Great, thanks so much for your detailed answer. You modeled my data perfectly. Yes, ANOVA by hand is something you do once in your lifetime (then rely on software :) )!
I don't know of one, but it would certainly be useful!
Can you give more information about the data you are working on? How many regressors you have etc. And is it actually a time series? Are you regressing on the whole time series at once, on time windows, ... ?
Perhaps you could explain a little, as not everyone will have access to that journal. Just from the abstract it looks to me as if they're talking about prediction of new observations, & contrasting the 'ML method' - plugging maximum-likelihood estimates into the model & using it as if they were known exactly - with the 'PD method' - which presumably takes uncertainty in parameter estimates into account when making predictions & sounds like some sort of [predictive likelihood](http://projecteuclid.org/euclid.ss/1177012175).
I understand Cholesky to be faster $O(N^3/3)$ ([Wikipedia](http://en.wikipedia.org/wiki/Cholesky_decomposition#Computation)) whereas eigendecomposition is $O(N^3)$ ([Jacobi Eigenvalue Algorithm](http://en.wikipedia.org/wiki/Eigenvalue_algorithm#Iterative_algorithms).  However, I have two further problems: (1) What does "potentially more flexible" mean? and (2) The variances differ by several orders of magnitude ($10^{-4}$ vs $10^{-9}$ for the most extreme elements) - does this have a bearing on the selected algorithm?
(+1) That 2002 thread is a hell of an exchange of ideas.
@PalaceChan I think Adam is right, you really have a sample size of 100 and should look at it that way. Doing something 100000 times and recording the total elapsed time is not quite the same as timing something 100000 times and taking the mean. A lot of things could be happening that account for the larger standard error (say garbage collection because the larger loop exhausted available memory) and you should be trying to understand that rather than looking for a statistical solution.
The general answer to the second question is no\\*. It's worth trying various simulations and some algebra. $\\quad$ *I suspect (but haven't checked) that its not the case for your specific problem.
I've added the `self-study` tag because your phrasing pretty clearly indicates it falls under self-study, but you should feel free to remove it (or to object and I will remove it).
@RobertF everyone and their grandmother has come up with a different way of conducting the optimization, and they are all iterative in nature (like Newton-Raphson, although that involves getting a Hessian and the $L_1$ penalty is not differentiable). For the LASSO, for example, the LARS algorithm can be used to get the trajectory of the parameter values as the penalization term is varied. These methods all go to a local minimum, but if everything is convex then there is only one local optimum, and it is the global optimum.
Because there may be some confusion in the question, let's address that first.  (I'm not saying you're the one confused :-).  Typically, an "RMS" is used to assess a "typical" error based solely on the $(x,y)$ location.  By employing a variogram, you are implicitly proposing using values of the DEM to *predict* its values at nearby locations.  But what is the point of that (except perhaps when the DEM has some gaps in its interior)?  "Applying the variogram" won't work; we're talking about the *kriging* algorithm here (aka the BLUP).
@user1205901 Sorry it took me a few days to get to this, but I figured it out; see my update in the answer.
+1 The Binomial distribution gives $0.0252250$ while this Normal approximation gives $0.0252271$: they agree to over four significant figures.
For s1: it is a scalar and has different units from the units of values in V1 & V2,, so I guess for it I should stick with euclidean distance. what I am thinking about is to generate a distance matrix between all points according to S1 and add this distance matrix to the combined dissimilarity matrices from V1 and V2 and use the final matrix for clustering .. does it sound right ?
Thanks a lot, you already gave me enough information to start digging for gold. :)
It's not really clear what your question is. Perhaps you could make some edits to improve it. Also, please enclose your math with dollar signs to render it properly.
To be clear, I am sending either letter out to all 18,000, it's just a matter of how to assign the letters that's the issue. My chief concern is that one group doesn't get many more of the rarer big donors than the other group.
(I'd take this to chat if you had the reputation.) $\\quad$ • Econometrics and statistics are not the same thing -- but even formally trained statisticians can be taught something that's not well justified, accept it anyway, and pass it on. Fortunately you didn't do the same. Hold onto that doubt - you shouldn't accept anything I can't justify either. $\\quad$ • Beware; skewness isn't the only source of problems with non-normality. $\\quad$ • Wilcoxon doesn't assume equal spread - *unless* you're only dealing with location-shift alternatives, you don't need that. ...(ctd)
The second article makes the argument that if % for B - 2 x MoE $\\le$ % for A then the difference is not statistically significant at the bottom of page 2 and top of page 3.
No, this is data dredging. It yields too optimistic results, especially for smaller samples.
Have you explored properly treating time as a continuous rather than categorical variable? Using something like a linear mixed-effects model. This would be especially important if the time points are unequally spaced and/or different for each individual.
@Nick This has a bit of folklore element, like "this is so elegant that this is THE representation of some distribution concept,...". Of course it helps with some maths, so it is not just a redundant toy, but for an everyday use it corresponds to a physicists forcing $\\hbar$ to a classic problem just to use fine-structure constant.
Were you able to find an answer to this question?
All of this can be found in Chapter 1 and 2 of Applied Linear Regression Models by Kutner, Nachtsheim and Neter.  That's where I was sifting through to find stuff.