tttjjj/elt_bench huggingface dataset:
Oh, that means I was doing it **entirely** incorrect! So, as I understand from your reply, the $Y_i'$ I mentioned is **your** $\\mathbf{\\hat{Y}}$, which is my predictor's value for the inputs and vary from 0-1 as they are probabilities. Now, I should sort these values and take thresholds, say 0.1, 0.2..0.9 and then calculate all the TP's etc in $Y_i'$. Will try it out quickly and let you know. Much thanks for the reply, very nicely explained
StasK, 

Certainly, see below. 

Links to papers 
- LDA: http://www.cs.princeton.edu/~blei/papers/BleiNgJordan2003.pdf

R Package: topicmodels 
- R Package description: http://cran.r-project.org/web/packages/topicmodels/topicmodels.pdf
- Journal of Statistics Software paper: http://www.jstatsoft.org/v40/i13/paper
@whuber @gung I'm hoping to learn 3 things here: 1. Is there intercept term the slope for the remaining variable that wasn't used in the N-1 dummies?  2. The slopes are the odds for event happening.  Eg, exp(Beta1) will give me the odds for bankruptcies happening when X1 is true. exp(Beta2) will give me the odds for bankruptcies happening when X2 is true...... 3. A coding example where entire logistic regression can be interpreted.  `glmfit` gives me stdev, betas, stderrors, p etc for training and test sets.  How do I decide my model's accuracy.  Thanks so much!
And I know this method might not be 100 % trustworthy, that's just why a measurement of the error would be great to have:)
And in case you just simply want gaps when weekends, replace the line `openValues <- c(openValues, mean(oracle$Open[i:i-1]))` in the first method with `openValues <- c(openValues, NA)`
THe mean is 0.8 . Thanks.
There's also the polr() function, in the MASS package.
@Tal: My experience is that it's quite slow when refreshing/repainting the views, e.g. when adding a variable or dragging to rearrange displays in the PCP.  Still, it is usable though not as interactive as advertised with large data.  @chl: That's really good to know, thanks!
Huh, I've never seen that error.  Could you post a histogram of the dependent variable, or if the range (upper bound) is not large, a table?
@StevenJeuris I find the statistical issues that arise with a within-subjects design (notably that your observations are no longer independent) outweigh the benefits. Whats important is that each *group* have roughly equal computer experience. If you're randomizing properly with a sufficient sample, that should take care of itself.
@Russell I think what ocram is alluding to is that when the entire population is sampled, then $\\tilde{s}^2$ *is* biased.  This shows that a key issue has not yet been discussed: this variance estimate is unbiased only for certain forms of sampling.
+1 rolando2 is right that what constitutes a 'meaningful' result is a theoretical question for the researcher, not a statistical question for the analyst.  In light of his comments about power, note that the further you get away from 50%, the more data are necessary to clearly resolve the relationship--rare phenomena are simply hard to study.  Furthermore, the problems of low probabilities are compounded by the existence of many predictor variables.  (Although, 150,000 observations is likely to be more than enough.)
Created a new answer to reflect the changes in the question.
Thanks, @Glen_b - What I'm trying to identify is the highly volatile outliers - Things to examine for bad-data possibilities... Due to the fact that the portfolio is managed, we assume that there should be very little correlation in volatility since the manager should try and get rid of the risk associate with varying market conditions... My main aim is just to assume that the portfolio may have some small market-related daily volatility, but to find changes so large so as to imply that there may be an error in the data entry process.... Does that make sense??
Thanks Michelle for helping me. Wouldn't a simple strategy of working out the percentage contribution of each type of unit be sufficient. For example, if I determine that I need 60 green units (10% of 600), then this would represent about 8% of my sample size. The other 92% of the sample would comprise white, black and yellow units (proportionately calculated).
@DLDahly: Sometimes magnitude is important, sometimes it is noise. It depends on the research field and is an issue of data standardization.
:/ not on Prism, the software I use...
Create features that represent your hypothesis (max-min, daily variance, yearly variance...) and use simple ML algorithms (even decision tree) to test it
I could not find full conditional distributions of K and t0 so Gibbs sampling does not work.
Random variables that satisfy bounds on the mgf like the one you quote are called *subgaussian* random variables. They play a central role, e.g., in nonasymptotic random matrix theory and some associated results in compressed sensing. See, e.g., the link in the answer [here](http://math.stackexchange.com/questions/35114/how-to-obtain-tail-bounds-for-a-square-of-sub-gaussian-random-variable). (This obviously doesn't speak to your particular question; but, it is of a related nature.)
I see what you mean, the only thing I can think of is that Andy had one factor (gender) with only two categories.
Are the points labelled? That is, if user B uploads a point, do you know whether it's supposed to be the same point that user A uploaded earlier or another point?
Sorry, I misunderstood.  The manual page for `quantile` points you to `ecdf`.
Alright maybe I'm being thickheaded here, but what variable am I supposed to minimize with respect to if I'm ignoring the variance?
This question is essentially the same as http://stats.stackexchange.com/q/7853/919 .
Your stance with respect to transformations, as expressed in that blog post, seems extreme almost to the point of idiosyncrasy. There are plenty of applications--especially the kind of classification problems mentioned here--where your ability to interpret the model is irrelevant. In many other applications where interpretation is essential there often are useful and natural interpretations of seemingly bizarre transformations, such as a reciprocal cube root or even a seventh root. Sometimes the *statistical* analysis suggests such a transformation and theory only *later* corroborates it.
@B_Miner: yes, the dependent can be in [0, 1]. The function fits an artificial neural network without hidden layer and with logistic sigmoid. And yes, it is recommended to scale the predictors roughly to [0, 1] as well for better convergence.
Sidebar: Do you have a reference for the Box quotation.
@boopyman by the way it's a common misconception that Naive Bayes is somehow the "right" classifier for text data. It just so happens that one of the first papers to attempt it (SPAM email detection) used Naive Bayes and it worked well, but actually the task was quite easy and pretty much any classifier would have worked well. If you're planning on using large datasets (millions of examples) I would look at classifiers that scale better - e.g. variants of the SVM such as pegasos or liblinear
The Davison-Kuonen link has been added to the original answer.
+1 for Collet's book. It contains lots of useful data sets.
Question 1 is not really a question. Do you mean something like "what is the probability that half or more of the claims have been padded?"?
@Manuel: your $\\alpha_n$ and $\\beta_n$ are $n$-th moments, about zero and centred about the mean respectively
@PeterEllis thank you for the comment. I need to study the relathionship of two stocks, but AFTER checking the residuals I need to know what is the distance from their mean, I only know the residuals (this is the only data i have) and now i need to calculate How far is the mean from thei relathionship...
This sounds suspiciously like a homework problem.
I made substantial changes to improve the grammar and make the question clearer. I assumed that you intended to always keep the same cost for each attempt. So the probability of success on any attempt stays the same. That interpretation led to the solution I sketched out. If you can change the payment each time you fail the problem is different and more complicated. But an optimal strategy may exist for each starting cost and you could compare the 5 expected costs under the optimal strategy. If I made a mistake with the problem formulation please correct it.
Every parametric test does *not* have this assumption. The general linear model, for example, makes assumptions about the distribution of *errors* as estimated by residuals.
Michael, note that @Lucas *is* the OP. :-)
Does your function include trials where it is impossible to get 8 in a row right, e.g. where the "run" started on trial 20?
Yes; you don't alter the data so no back transformations are needed. As a result all the standard diagnostics and goodies by lmer() eg. fitted variables, residuals, random effects levels etc. etc. are directly interpretable as they correspond to your "true" dataset and not an "altered one".
yes, the restriction $\\sum_{i=1}^{n}w_{i}= 1$.
If $X$ is continuous & its distribution is symmetrical, $Pr(X<\\mu)=.5$. Is it continuous? Is the distribution skewed?
@user603, my view is that the relationship between functionals and sample statistics can be considered part of robustness analysis, but please edit the tag if you think my reasoning is wrong.
You need to provide more information about your data if you want this question answered.  A plot, such as a histogram, can be helpful.  It would also help (a lot) to state *why* you are fitting a distribution, for otherwise "best" is too ambiguous to be an effective criterion.
It is always helpful to go back to a statement of the problem that really concerns you.  Why don't you edit the question to reflect this last comment?  You might also consider indicating what you intend to do with this representation of the word associations, because that can point the way to good answers.
What do you think? http://en.wikipedia.org/wiki/Ordinary_least_squares, section "Finite sample properties". This question was answered many times on this site.
I didn't say it is required! But if you open any standard regression text book, they will explain these procedures using the partial F-test and not AIC ... Cheers! And one more thing, you can see the function written by Paul Rubin [here](http://orinanobworld.blogspot.ca/2011/02/stepwise-regression-in-r.html) to see how we can implement in R.
@soufanom: You may be interested in the answer below by the author of the paper himself!
Are your data located in time and/or space? Autocorrelation typically refers to the similarity (correlation) between data points as the distance between points varies. So in a time series for example, the common question is whether points close in time are more similar to points more distantly separated. The Durbin-Watson statistic is constructed to test such a question - it quantifies the variance of nearby points relative to the total variance. The situation is analogous in space.
Highly recommended by me, too.
+1 Very comprehensive. I italicized your notion `it already emphasizes directions...`, with your permission...
We're just subtracting based on corresponding dimension. There is no interaction. Am I incorrect?
2 samples is not reliable. I suppose if they happen be very close to each other in every data set, you might get away with it, but i would prefer more data...recall that your estimator for the variance is a random variable itself, with an associated variance of its own...However, if your 7 data sets are used together to estimate an assumed *common* standard deviation across all 14 data points - that is a different matter.
Some of that exists here:  https://docs.google.com/file/d/0B83UyWf1s-CdZnFoS2RiU2lJbEU/edit
What fraction of observations have complete data?  Do you get significant variables when you do a (non-recommended) complete case analysis?  What imputation method did you use?  How was the pooling done?  You provided very little information on methods you are using, sample size, collinearities.
Here's an SPSS tutorial by Andy Field: http://www.statisticshell.com/docs/mixed.pdf ; sphericity and homogeneity of covariance are two assumptions to consider.
Druss, why would you omit $\\mathbf{Y}$? You are interested in its effect on the outcome in the structural equation, that is, you are interested in the parameter vector $\\boldsymbol{\\delta}$.
Not sure about time series, but the SEM framework has been widely used in latent growth curve modeling, for example; see work by [Bengt O. Muthén](http://pages.gseis.ucla.edu/faculty/muthen/) and coll., and references on the [Mplus](http://www.statmodel.com/) homepage.
+1 Good effort.
I don't think there can be an "OK" here without knowing much more about what the data are like, what interests you and what criteria you want to drive the clustering. Assuming that 0 is a valid value, and not a code for "missing" or "no data" then the zeros are part of the data too.
What other constraints, if any, do you want to put on this problem? Otherwise, it seems `qpois(1-alpha, lambda)` satisfies the requirements.
It depends; you can estimate parameters without assuming normality, but inference such as hypothesis tests and confidence intervals are based on that assumption.
@mbq yes, that is certainly true. However, based on the question, I assume the OP is a novice, and hence kept things simple to avoid extra confusion.
What specifically are you confused about?
for the clustering approach, the most important issue is to find the similarity metric. However, in my instance, one consumer would only purchase a single product. So, for the users who purchased different products, we can't directly compute the similarity between them using the given ratings.
@Prasanth: ACD wants you to expand on the word "analyze". What do you want to see? Give examples of the kinds of statements you'd like to be able to make. How do the categories enter into what you want to do? Techniques can be simple ("Twice as many people strongly agree with X as with Y") or complicated, and we can't really help you unless you specify what you mean by "analyze". And to be honest, you won't succeed if you don't understand what you mean by "analyse" clearly enough to explain it to us.
@Douglas, I thought it was for the following reason: Suppose that you drew 100 random points from the mixture. You would expect 95 of those to be from the region containing 95% of the mass, which seems like the same expectation as a confidence region. If I'm missing something, I'm all ears.
@whuber Yes, there's a danger it becomes overly broad. Would something like "which commonly generate confusion" suffice to narrow the scope?
Thanks. I agree with your answer. I am just wondering whether these exists a quantitative way to show the time series is not white noise？
Are you sure you don't want $A$ to be the event $|X-\\mu_X|\\le k\\ \\sigma_X$ and $|Y-\\mu_Y|\\le k\\ \\sigma_Y$? That would be closer to the univariate Chebyshev situation.
Could you clarify please what data you are (perhaps hypothetically) starting from and what you want to do? I can read this question two ways: 1) You have complete data on x characteristic in the counties and are asking how to make meaningful comparisons between counties of different sizes; 2) You have sample data on x characteristic and are asking how best to estimate the total number with x in each county.
How does one obtain a p-value from that "r value"?
http://www.statsci.org/glm/books.html
Ah okay, will do, thanks ;)
I think the "response" or "outcome" variable are commonly used terms.
Because the residuals in the cumulative consumption would be *strongly* correlated, this method does not appear statistically justified. Using a quadratic fit only papers over this fundamental problem; it cannot cure it.
Correct, but the "far out" points are effectively given much more weight because of minimizing the sum of squared residuals rather than, for example, the sum of absolute residuals.  With heavy-tailed $Y$ you will find accuracy for predicting in the middle sometimes sacrificed so that major errors are minimized.
@glen_b: My comment was not clear. The mean of $E(g(X))$ would be a more meaningful quantity than $E(X)$ when $X$ is drawn from a very skewed distribution.
I included matlab as a not-quite worst case. The language itself isn't terribly fast and a lot of stuff that makes it fast (e.g, BLAS, FFTW) is available separately (and, often, for free). More generally, I'd encourage you to think about what sort of analysis you want first, then figure out how to make it fast. It's not always possible (MCMC or resampling is, by definition, slow), but it usually is and it's the right thing to do statistically too.
Running a single multiple regression on all observations with X and an indicator for the sample as the two regressors will give you an unbiased estimator of the true linear effect of X (given the linear structure of the model is correct).
These cartoons are useful too; they can be included in a lecture on a particular topic where you are trying to explain a concept (e.g. correlation/causation above).  A little humor can help to keep an audience engaged.
I have uploaded my source. If you uncomment the print statements in the train function you can observe how the gradient explodes.
This is a really great answer for how - not so much for why. But maybe that is just a [wrong question](http://lesswrong.com/lw/og/wrong_questions/) - what we really care about is what works empirically, not theory.
Your explanation is nice but it runs into difficulties when $P(B)=0$. The definition and concept of conditional independence are complicated and difficult in this case, whereas a formula like $P(A\\cap B) =P(A)P(B)$ is simple and clear.
Have you considered [*WinBugs*](http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/contents.shtml)? It handles missing data in a beautifully natural way.
3 to 7 sounds highly unusal. 
YOu might want to consider using only the extreme 2 values at both ends, e.g. "6/7 -> 1, 3/4 -> 0", the 5 will be discarded, but is often of less interest because neutral answer have often not much sway anyway.
An alternative would be to take the route: 6/7 ->1 , else -> 0, but the size of the categories is unequal which I doesn't like - but some people like this more, because no answers are lost. 
Probaly try both methods and take a look at the results.
I think you have discovered a form of what is known in the engineering literature as a square-law detector or noncoherent detector.  If the sequence of $\\mu_i$ is known, the optimum detector uses the statistic $\\sum_ix_i\\mu_i$, heuristically $\\mu_i$ and $x_i$ are more likely to have the same sign.  When the $\\mu_i$ are not known, the detector essentially makes an "energy measurement" $\\sum_i x_i^2$ (you might remember $V^2/R$ from high-school or college physics) with large energy meaning signal is present and little energy meaning signal is absent.
Jochem, you don't supply enough information in your question to allow people to supply reasonable answers.  Although forecast errors might have normal distributions, the crux of the matter is to determine how the means and variances of those errors depend on the forecast horizon.  I'm afraid your bounty will have been wasted unless you can supply enough details of your model to let people figure this out.  If I am mistaken in this assessment, then please explain more clearly what you mean by "trea[t]ing the error" in your question.
Welcome to this site! Could you please indicate why you recommend this article and what are its main features?
@Shane: good point. moved.
@ Egon: subjective indeed. but if the answers come from knowledgeable people i don't mind dose of subjectivity.
i've started learning R quite recently and have couple of dozens installed to explore, however i notice that there are tools that I use much more often irrespectively of the task at hand.
(+1) not so straightforward algebra, nice alternative solution.
Could you clarify how exactly you want to "weight your response variable"? The `weights` parameter in `lm()` yields parameter estimates that minimize the *weighted* sum of squared residuals ("normal" OLS minimizes the *unweighted* sum), so high weight observations will be more influential in estimating the parameters. This actually sounds like just what you should be doing in your problem, but I probably am missing something.
@Frank Thanks. My first stab allowed $\\rho$ to be re-estimated, but then I realised that the LRT would be comparing models with not just different fixed effects but different correlation structures. $\\sigma^2$ was re-estimated. I'm not aware of a canned way with `gls()` to allow fixing that. I came at this from a practical point of view - re your comment in [...], what is the exact $F$? Showing my limited statistical knowledge now I'm afraid.
Just some clarifying questions which may hopefully make your question more specific (and hence get a better answer).  1) Are you talking about estimating the ridging parameter? 2) why do you want Monte Carlo simulations as opposed to other techniques(e.g. optimisation,MCMC)? 3) are you looking to account for the noise/error variance in your estimation method (i.e. the fact that the "ridged" betas won't exactly predict the response $y$)?
Right, this probably the most straightforward approach. The categorical variables would need to be dummied up somehow, which is also easy.
@Jakub: Well, it goes this way because it was implemented using that structure. The help states that the first argument has to be the response variable whereas the second argument has to be the group variable.
Please explain why information was discarded from titre level.  It would be ideal to use an ordinal model on the original continuous form.
@onestop link added. NIPS=Neural Information Processing Systems . It is a community (not a system :) ) . But pisk is talking about the conference NIPS 2010.
You need to rigidly define the hypothesis before trying to answer it. As it stands you have a hypothesis relating to 'transitional seasons', but then when you attempt to answer that you find the way you've define that to be imprecise. You now want to find a suitable way to define transition, after looking at the data. Stop. Any analysis you subsequently do is worthless, since you can now (consciously or not) tweak your post hoc definition of 'transitional season' to either support or refute the hypothesis. There is already enough climate pseudo-science that does this. Please don't add to it.
There are a number of differences. What application are you looking at? In what context did this question arise? There are too many differences to feasibly list them all, and even then we would be guessing to answer you question. Can you provide us with more information?
Keep in mind however that the "1.6/100,000 number" also has some uncertainty behind it. And I really wouldn't discount the impact of intensified surveillance for cancer in detecting new cases.
36 months might be more appropriate to pick up seasonal factors i.e. weekly/monthly effects . You can post the data to dropbox https://www.dropbox.com/business/business-free-trial?_tk=sem_goog_b&_kw=dropbox|e&_net=g&_ad=35064061902|1t1&_camp=sem_google_b_dropbox_us-english&gclid=CNGI3sv2k8ACFWho7Aod5nwACA . Please include a value for all hours for all days . If for some reason you don't have a full audit simply average values around the missing points. For example if you are missing day x use average the values for x-7 and x+7
There's *very clear* indication of heteroskedasticity. There's also suggestion of lack of fit.
@Peter if the eigenvectors are identical and so are their corresponding eigenvalues, then you can recreate the covariance (or correlation) matrix you used for PCA.  However, such a matrix obviously does not determine the data.  For instance, subtracting a constant from all values won't change the covariances.
That nice paper shows how to estimate the bias in $\\hat{\\beta}$ in the rare event situation, but does not demonstrate that the bias-corrected estimator is closer to the true $\\beta$, i.e., that the variance of the bias correction is small enough that it doesn't matter.