tttjjj/elt_bench huggingface dataset:
Your question is broad, you are not asking anything specific. There are many methods of clustering, many distant measures, many ways to validate clusters. You certainly have to read more about cluster analysis before you feel youself able to make conscious choices among those lots of options.
I don't view it as a kluge: this technique is based on the *definition* of periodicity. It works for any local smooth. (It will not work for a global smooth, but that's not a problem, because most global smoothers are derived from inherently periodic methods like Fourier series anyway.) @Nick One doesn't have to be terribly careful: when using a local smoother of maximum half-width $k$, one only needs to tack the last $k-1$ values of the sequence onto the beginning and the first $k-1$ onto the end, but there's no harm in conservatively expanding the sequence by more--it's just less efficient.
Thank you! Mine is a social science research.
They're not solving linear regression. They're doing a math problem. There's no need for them to mention "y" in this context at all. The only problem they have is to minimize Ax-b. Linear regression just happens to be a problem you can reduce to that form. "A" becomes a matrix of [[1 x1] [1 x2] [1 x3] [1 x4]]; "x" is the vector of [b a]; "b" is y.
I am interested in validating the code I wrote.  I have found plotting the likelihood does reveal logic errors, but I am still interest in other ways to validate the code.
How can i check if i have another account?
Harrell has similarish ROTs (with citations): http://books.google.com/books?id=kfHrF-bVcvQC&lpg=PA61&ots=34KVS2ehF-&dq=frank%20harrell%20table%204.1%20limiting%20sample%20size&pg=PA61&output=embed
Thank you, amazing stuff!
Thanks a lot, Jake! That's great! It's much more clear now, I had missed an obvious step. Thanks again.
If you only do marginal calibration, It might be that before tranforming the data quadratic separation holds and after transforming the data it does not hold anymore.
Since you are using a published scale, the first thing would be to see how the originators of the scale suggest it be scored.
"Ran out of iterations and did not converge" does not sound like convergence to me.  No idea why this is a warning and not an error.
Yep, definitely getting a bit ahead of myself. Thanks for your input. I play with incorporating some economic indicators in to the model of step 1.
For the record, this question is related to [How do I conduct model selection for logistic regression in SPSS?](http://stats.stackexchange.com/q/32893/930)
You're just going to get an answer stating the definition of conditional probability (or linking here http://en.wikipedia.org/wiki/Conditional_probability) but I don't think that's what you're looking for. Can you explain exactly what your question is?
It would be good to mention/give a link to where you read that.  I believe Burnham and Anderson recommend AICc for `N-k<40`, but I don't have their book handy.
Don't use stepwise/forward selection. Reasons not to do this have been [discussed here a lot](http://stats.stackexchange.com/search?q=stepwise)
Great, thanks for your answers.
So you would recommend deciding beforehand of a set of variables and stick with those whatever their "contribution" might be?
Hmmm...  I'm happy to assume whatever regularity conditions needed to make the weight vector identifiable.  Alternatively, I suppose what I *really* want is to show that $||{\\hat f}-f||$ goes to zero where ${\\hat f}(x)={\\hat w}^T x$ and $f(x)=w^T x$ and $||\\cdot||$ is some functional norm.  But I'll take what (if anything) I can get!
Thanks for the reply, very clear!
Absolute risk is not the same as incidence rate. Risk is calculated as (# new cases)/(total population), and is thus unit-less but often presented as percent or per 1000 population. Incidence rate is calculated as (# new cases)/(total person-time) where person-time is the total amount of time spent under observation across all people, and has units of cases per person-time. Prevalence is calculated as (# cases)/(total population), where cases includes both existing and new cases. Point prevalence is calculated at a given point in time. Period prevalence is calculated over a given time period.
Some sort of reverse cointegration.
edit the information from the first comment into your question
+1, this is a really nice overview of Poisson reg, which I agree is the right approach here. One thing to note is that dichotomizing data leads to information loss & all that that entails, whereas you can figure out whether someone make >2 purchases from the Poisson output.
@ Staty Despair: You might be interested in [Yet another short introduction to R with some emphasis on meta-analysis](https://github.com/berndweiss/r_meta-analysis_intro/raw/master/report/p_introToR.pdf).
Are you trying to update the parameters $a$ and $b$?  If you're trying to update some parameter $\\theta \\sim \\text{U}(a,b)$, I agree with @gung - you can just update a parameter $\\phi = (\\theta - a)/(b-a)$ where the prior on $\\phi$ is a $\\beta(1,1)$.
Thanks that worked for me.
I added a link to the paper I used. I am trying to recreate Figure 1D on page 39. Explanations are on page 6.
Just wondering if this question was perhaps not clear enough? Very much look forward to some thoughts/answers, thanks
Perhaps every one thinks that love is wonderful, or peace greatly preferable to war, or some such. That can happen.
Do you also think that if you do it the way my friend does the number obtained would be too high? Since usually families consist of children that don't have a (large) income?
@BenJWoodcroft, it is not linear separability as such that triggers the error but the geometric organization of the three groups along a line. The linear separability makes this organization more obvious in the sampled data, and if you increase the standard deviation sufficiently, glmnet does not "discover" the organization. As your second example shows, you don't actually need the linear separability. You are right about the email address, thanks.
... assuming you're following [this answer](http://stackoverflow.com/a/14971531/3309177) (from the link you've provided) and you've saved it to an object named "result".
@suncoolsu, looking at [The profile sampler](http://pubs.amstat.org/doi/abs/10.1198/016214504000001772), [General frequentist properties of the posterior profile distribution](http://arxiv.org/abs/math/0612191v2) and even [Wikipedia](http://en.wikipedia.org/wiki/Likelihood_function#Profile_likelihood), it seems like "profile posterior distribution" is the right answer. Could you turn that part of your comment into an answer so I can accept it?
I don't see how transformation would help.
Thanks - I used it later but forgot to remove it for this question
Thanks! But could I use McNemar's test if I manually paired up each male to 1 female? So one member of each pair is the "male" member and the other is the "female" member - making 40 pairs. Then I could just find the number of discordant pairs and compute a p-value. Would doing that give me better results or using a chi-square/Fisher exact test?
Actually, this isn't economic data, it's data about the numbers of contributions to volunteer projects. The research question is: which projects come from a few volunteers who do many tasks vs. many who do only a few tasks each?

The situation is mathematically like income distribution, where "income" is actually volunteer contributions. (No judgment implied here - just a mathematical model. Also, let me know if this analogy is not accurate.)

I know I'm safe treating this as ordinal - projects with higher Ginis are more unequal - but can I say anything about relative *amounts* of inequality?
Each judge treats many applicants in the study period (around 30 each). Some of the cases are finalized (accept/reject) - others are still open.
@whuber, as you know Beta distribution can have very different shapes confined in [0,1] interval. I don't think there could be useful general result for the sample mean distribution solely based on the bounds [0,100] given that the sample is 0.05% of the population. More assumptions are necessary.
What I found out, was that I actually had some errors in my input data. The result of this, was that the networks with a lot of data, actually learned to respect these errors, which the networks with only very little data did not experience in the same amount. On top of that, the data became very hard to learn, and the results I got were very random.
Thanks for the answer. For 1) you mean to say that the true answer won't change... right? If I mix and incorporate the bad raters, I'd just get noise. For 2), it seems like again there is the chicken and egg problem. If I calibrate the bad reviewers sufficiently using the good reviewers, this also doesn't add information.... The one thing I wanted to try was to see if say the top quartile of crappy raters have a high kappa (or correlation) with the really trusted ones. If so, could I somehow use that subset of "bad" raters? Any further thoughts would be appreciated.
How to implement the formula to the example?
+1 I like this solution because it is straightforward and elementary.  The somewhat cumbersome "Cov" notation (instead of "Var", which you introduce earlier) actually makes it look more complicated than it really is.
Higher than what?  Are you comparing cardiovascular mortality in a given group to other kinds of mortality in that group or are you comparing cardiovascular mortality between groups 1 and 2?
I've added the `self-study` tag. Please review its [tag wiki info](http://stats.stackexchange.com/tags/self-study/info)
+1 for guiding me in the right direction. I just finished doing the C-statistic and the more detailed score that I'm looking at had a C-statistic of 0.4365081 while the other had 0.4414625 (I guess that I should be counting 0.5-Dxy/2 in my case). I took quite a while doing the calculation on my 140 000 sample; I had to lower the bootstraps to 10 and I'm not certain what the impact of that is. I'm looking forward to reading your book (it's in the mail) and hopefully it will help me understand the methodology better and compare the C-statistic with the adequacy index.
You're right - you can't just back-transform the SE. Do you really want a CI for the mean, or something else? For example, if you back-transform the mean of $\\log X$, you get the geometric mean of $X$, which isn't the same as the usual arithmetic mean, but still of interest. Also, I think you should transform everything the same way, e.g., use $\\log(X+1)$ even when $X\\ne0$.
This probability plot strongly suggests a mixture of two distributions, one located between 5 and 60 and the other between 50 and 150+.  This casts doubt on any simpler characterization, like negative binomial.
Thanks for the suggestions Ash.
+ This is one of my favorite applications of information theory - algorithm analysis. If you have decision points with >2 outcomes, such as when you index an array, that's the principle behind hash coding and O(n) sorts.
There may be a few things to add about the ideas you've presented (I've got one or two points in mind myself), but it's a great start already. Thanks for writing it all out so clearly!
Mapping distances between a set of objects in euclidean space is known as Multidimensional Scaling.
Sorry, but trying to provide advice would be foolish without much more information on what you are doing (as @Peter Flom has also requested).
[How does this question arise](http://stats.stackexchange.com/tags/self-study/info)?
"Dependent samples" is too much general. You might precise what kind of data do you want to study.
It seems that your model has fall into overfitting. How many observations do you have and what is the number of independent variables in the model? What algorithm have you used to obtaine the model? To fight overfitting you may introduce regularization or use ensemble of weak classifiers (like random forest).
See [this answer](http://stats.stackexchange.com/a/19101/6633) for a proof of a more general version of what is sometimes called the one-sided Chebyshev inequality (or the one-sided Chebyshev-Cantelli inequality or the Cantelli inequality etc depending on which book you are reading).
This sounds like homework so I will add the homework tag. You can delete it if I am mistaken.
Would descriptions of the algortihms ([here](http://en.wikipedia.org/wiki/Forward%E2%80%93backward_algorithm) and [here](http://en.wikipedia.org/wiki/Viterbi_algorithm)) answer your question or are you looking for something else? Are you wondering when to use which algorithm? Looking for a discussion of their respective merits?
No doubt someone will chime in with formal definitions, informally, all expectations are expectations over the distribution of (/expectation with respect to) some (possibly multivariate) random variable, whether it has been explicitly specified or left implied. In many cases it's obvious ($\\text{E}(X)$ implies $\\text{E}_X(X)$ rather than $\\text{E}_W(X)$). Other times, it's necessary to distinguish; consider the law of total variance for example: $\\text{Var}[Y] = \\text{E}_X\\left[\\text{Var}[Y\\mid X]\\right] + \\text{Var}_X\\left[\\text{E}[Y\\mid X]\\right]$.
@Berd: Thanks a lot.
I'm just wondering if anyone else noticed that the values at the top of page 2 of the 4th link (whatisbayes.pdf) are a little farther off than simple rounding error?  I'm getting about 0.040, 0.097, 0.138, 0.139.
It might be a belated reply: in Crimisini et al. Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning they use RF in a way that might suits you for organs boundaries identification.
See `?formula` for the details on how to specify the formulas for various linear models.  The short is that you would use `lm( humidity~temp+day)` or `lm(humidity~humidity*day)` depending on whether you want an interaction or not.
I have not encountered sampling distribution as implying mean and variance. If there are usages such as you report, I would go so far as calling them incorrect. A distribution is a set of values, not a summary of them.
@Macro: Macro yes that is me. I am not sure what $f$ is. The paper never said.
@whuber Thanks for expressing your concerns! Right now, I'm still working on running simulations to probe the reliability of TLS fitting for this problem. What I've seen thus far, though, is that TLS' consideration of *both* variables helps greatly in overcoming the high non-linearity of the model. Fits of simulated data are reliable and converge very well. More work needs to be done though, and I'll definitely have to stack your method up to this one, once we have more actual data available --- and look in detail into your concerns.
-1 for "You can think of two variables being independent (also called uncorrelated sometimes)"  Independence is _not_ the same as being uncorrelated; uncorrelated random variables can be very much dependent.
@BenR: yes, your interpretation is correct and z is the location of  the knots.
I love that the question asks about Efron's first thought and the answer literally opens with a quotation from Efron describing his first thoughts.
Usually, an analysis is based on the contents of the answers.  When an answer is not forthcoming, there is nothing to analyze.  In some circumstances, you might already know something about *all* the individuals (including the non-respondents), such as their gender, race, zip code, or whatever.  In addition, *the respondents must be identifiable*: this rules out anonymous responses!  In such cases, your suggestion is a good one.
Reporting of results differs from person to person. Different publications also demand results be reported in their own style, so authors are sometimes chained to the set way. If you see something like that, you're probably seeing a specific style, not the reasoning for which p-values are usually reported.
+1 Nice job thinking about (and answering) the OP's underlying question.
+1 **degree-of-belief**
I am just learning the usefulness of stating the confidence interval of a correlation. Is reporting CI necessary?
After reading your answers I think I definitely messed up all these concepts. Thanks for spreading for the clarification!
Thank you for very much for your help. much appreciated!!
I think we agree on the same thing, I just wanted to make clear the distinction between test and C/V data. Model selection is akin to parameter selection so it's best if _test_ data is put aside and not used at all. Having done that you'd be able to safely report on expected performance of the chosen model on any new unseen data.
@xiaodai Yes there is a `cut()` function in R but it does probably not do what you want. I'll update my answer.
@User3897:  Whatever you do, definitely save some of your half million cases as a holdover sample for crossvalidation...I'm perplexed by your inability to find a better-predicting decision tree.  Have you fully explored SAS's options, e.g., on your inputted rules for node size requirements for when a parent node can be split, or when a child can be formed?  For the statistical significance of such splits?  For the program's ability to group the values of continuous predictors into effective subsets?
One case where classes are not of equal importance is when you classify something into A vs. not A (as opposed to A vs. B). For A vs. B (e.g. cat vs. dog) both classes are meaningful and you may try to model both. For A vs. not A (e.g. cat vs. non-cat), the negative class includes everything, so you can't really hope to model it anyway. Another example is if cost of misclassification is different. Say in your example, if I predict a default erroneously and refuse a loan, then I just lost a little profit. If I predict non-default erroneously, I lose a much larger loan amount.
Or a beta distribution when $\\alpha=\\beta$ and $\\alpha<1, \\beta<1$.
You can use LaTeX math. See the help when editing.
Thank you Xi'an. I was just a bit confused and had some misunderstanding about Markov random fields. But know, thanks to your answer here and comment at my another question, all this seems to make perfect sense.
I don't think so.  Sen seems to think that it is biased and inconsistent in the present of measurement error, as one might expect.  See e.g. sentence 2 of [The Theil–Sen estimator in a measurement error perspective](http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.imsc/1291044758) where the 'discounted parameter' that is median-unbiasedly estimated is an attenuated version of the actual slope of interest.
This answer suggests a way I might do it (with some modifications) in R, but I'd like to keep everything in python if possible: http://stats.stackexchange.com/questions/31934/maximum-likelihood-estimation-of-inverse-gamma-distribution-in-r-or-rpy
I plotted both model error and cv error, this approach worked fine. I considered average of neighbors only and used a gaussian window function.
Choosing principal components that explain the most variation is one option, but @Scortchi has recommended other alternatives for feature selection (not necessarily multicollinearity management). See the comment on my answer to [PCA, randomness of component?](http://stats.stackexchange.com/q/87198/32036) or maybe some of Scortchi's original answers on the topic (not sure which to suggest here).
@csgillespie: I edited my answer to better accomodate your interest. Let it not be said that I was undeserving of the bounty. ;)
I don't really understand your question.  Are you asking about how to do a one-sample t-test?  You're just about there already.  Are you asking about corrections for multiple comparisons (given that you appear to want to do 10 tests)?
*(note: as my knowledge of statistics and SPSS is quite limited, I understand this question might be lacking and/or not clear. please don't hesitate to suggest edits)*
In order for us to answer this question correctly, could you please explain how you perform your line fitting? And if you do that with default Excel tool, how do you get errors $\\Delta X_i$?
This perpetuates the mistake made by crazyjoe: orthogonality does not imply independence unless the variables are jointly normally distributed.
Having done a bit more background reading on models for the variance of a series, I wonder if it wouldn't be better all round just to fit that model than worry about the moving window bits. An (G)ARCH or stochastic volatility model seems appropriate for that at the moment, but I'm not sure how would show that variance *increased* with one of these models? But that's for a different Q&A. Still very much interested in any thoughts on the Q here as it is something one quite often in looking for early warning signals of impending transition in ecology.
Hi there my friend. Yes, I am looking into M-H in the context of linear regression. The url you gave me explains everything really nice. Thank you. If I come up with some other question regarding M-H I will post a question again. Thanks again.
...to which she replies, "well, [at least half of one](http://math.stackexchange.com/a/81726/7003)."
One thing missing from this question is the duration of a "period." This is crucial because you are attempting to translate a per-period return to an annual mean return.
How to put together 2 datasets which differ in their attributes (columns)? E.g. the columns of the first dataset are x1,x2,..,x10 and the columns of the second are y1,y2,..,y20  where Xi and Yi are not the same variables. This resembles multi-modal data but the subjects (rows) are not the same. The only common column is the response variable. Regularized Canonical Correlation Analysis is the only statistics that seems to me a bit close to this problem.   Do you mean a block diagonal structure where zeros complement the non-existent data? In this case what kind of statistics would be useful?
Thank you, that was useful. I may try a piecewise regression
Many thanks! I had a feeling the answer would be something simple like this. And regarding the correction to the standard deviations, it grew out of desperation since it did produce more correct results in the original script. Also thanks for the addition about rnorm() being vectorized, that will be really useful!
@whuber updated, thanks for your comments.
+1 A simulation (or some such concrete example) has long been needed to give us a reference for further analysis. Your simulation can be made much more efficient (about 25 times faster) by noting that $S$ factors as $\\left(\\sum_i x_i\\right)\\left(\\sum_j y_j\\right)$. That immediately explains why no sufficiently large prime values can show up in your triangle chart--and forcibly demonstrates that $S$ cannot have a "random walk" (scaled Binomial) distribution.
Thank you Aaron!  One follow-up question:  To determine which C is best, should I evaluate the models (each with a different C) based on accuracy percentage (i.e. correctly classifying points from the validation data set) or should I use something else?
@Py-ser, why normalize at all?  it is not a requirement of the KS test.
The definition you give above is not equivalent to that given in the site you link to, or [in Wikipedia](http://en.wikipedia.org/wiki/N50_statistic), which both say it's "as the length N for which half of all *bases* in the sequences are in a sequence of length L < N", which translated from bases in sequences to strings in characters would be "the length of the string for which 50% of all *characters* are in strings of that length or longer".
See this [answer by whuber](http://stats.stackexchange.com/a/48608/1036). Basically you can estimate a model that nests the boy model and the girl model in the same equation, improving efficiency in the estimates compared to estimating seperate equations.
(1) Then function satisfies the reflexive, symmetric, and triangle inequality axioms (2) For starters I would like to require each element from the smaller list to be paired with an element from the larger list.  This is an applied rather than theoretical problem; make whatever assumptions we need to get a reasonable set of pairs.
@FrankHarrell, can you shortly expand the reasoning behind not deleting "insignificant" variables?