tttjjj/elt_bench huggingface dataset:
We need informative title, so I updated yours; feel free to choose another one. It looks like you're after some kind of a dose-response relationship (with counts or % as an outcome).
thanks, Sir! This is very informative. I always felt regression may not be the correct way to go. I'm already looking into the sites you gave me.
@Aaron, do have this patch handy? I would need it often to fit a model with single individual for each class...I might want to ask as seperate question ....it might be too much in this question
I would agree that using statistical tables is an absolutely obsolete computational technology. Tests of normality, however, do have their reasons.
Even that document is not perfectly clear as to what is meant. It is curious that they've specified an *integer* $M$. However, they also say $R$ is a *real* number, by which I assume they mean not necessarily an integer. So, it seems what is meant is uniform on $[0,M]$ as opposed to uniform on $\\{0,1,\\ldots,M\\}$. At any rate, they don't want an exact answer, just a Monte Carlo estimate.
I like his 2008 paper on FDR. I don't use R, though :(
@luciano: Do you know the definition of a Pearson residual? See [here](http://stats.stackexchange.com/questions/99052/) if not.
How would you answer the same question in a linear regression without complex survey design?
Personally, I've just used Excel for this.  E.g. Put the sentence word by word in one row, one word/toekn per column, and then little drop-down boxes with valid values or free text input in the row(s) underneath. Then write a python script with openpyxl to read/write the excel files. It's ugly (like anything with excel involved), but it works - most important is that you don't have to train people in how to use Excel, and most IT departments have to support Excel so the pushback you get from "install Excel on this machine" is zero.
Good suggestion.  I'm not sure I understand the method well enough yet to summarize how it works accurately... in the mean time, I've excerpted the brief summary from the R implementation.
@bobobobo, there's really little difference between a dotplot & a bar chart where the bars run horizontally. Often, bars are used to plot things like group means, instead of raw values (as here), so it could throw someone off. In addition, Cleveland has shown that dotplots are read a little more accurately. But for the most part, there shouldn't be much difference. The key to this is rescaling your data so that they can be plotted together. The transformations I used were via z-scores, which equates their centers & how far they spread out.
Not only the maximization: the up-to-proportionality also comes into play in the likelihood ratio notion, and in Bayes formula for Bayesian statistics
I chose loess because I grabbed it from the internet. I'm very experienced in smoothing in general, so I simply grabbed code online. Do you have a better suggestion?
I read about UCB example after seeing Glen_b's comment to original question.  It's my fault and I'll fix the question, but I believe my example is different.  Firstly, the data I'm working with is actually based off of "first preferences" that students have written down for what courses they would *like* to enrol in most.  It's not actually a reflection of which courses they were accepted into (I thought using the term enrolment would simplify the question).  Secondly, I *am* interested from the gender preference angle, not the "was their discrimination" angle.  I'll read the linked paper.
How far into the future are you planning on forecasting? It's hard to say much without knowing this.
@KHKim: thanks for the $u+1$ instead of $u+v$ typo, indeed!
What does "a->c<-b" mean?
@JuhoKokkala. Success is Bernoulli with success probability $p^m$. So the variance of having a success is $p^m (1-p^m)$.
I care about consistency for the variable of interest but not for the control variables. My sense is that my variable of interest is not correlated with time-invariant unobservables (as the Hausman test on this very variable suggests) but some control variables are. So, is it correct to say that, if I use a RE model, the estimation of the variable of interest will be consistent while estimations of control variable will not (but this does not matter to me as I do not claim to have the right coefficients on these variables). Thanks a lot for your help.
How would you go about Logistic regression in SPSS? and how to interpret it? I put in the dependent the performance scale, and in the factors the 8 likert items? and in the covariate the 6 likert items?
Thanks Frank, the equivalence proven in the paper you quoted is using the logarithm property: $\\mbox{log}(a-c)=\\mbox{log}a+\\mbox{log}(1-\\frac{c}{a})$ for a$>$c. I used this property and proved it, and the effect sizes ($\\beta$) are the same but with a different intercept. With the proportional odds model, with a logit link, I found out that the forward continuation ratio model is the sum of the proportional odds model and backward continuation model.
As what seems to be routine bookwork, please add the `self-study` tag to your post, and read its [tag-wiki](http://stats.stackexchange.com/tags/self-study/info); following the guidelines there, please outline what you have tried or what you think about how to do it, and where your specific difficulties lie.
That's mainly correct.  The 15:1 rule is for developing accurate predictions.  The ratio of cases:predictors can be much less than that for adjustment.  If using penalized maximum likelihood estimation (without penalizing the exposure effect) the ratio can be as low as 2:1.  Going back to the original H&L strategy, the parsimony achieved is partially a mirage because of using Y to achieve that.
The evaluation that I'm doing is if a policy has affected the health of the children.For it I have 3 cross-sectional data sets:with the first two(both are before the treatment)Im prooving the paralel tendencies needed to use the Diff-in-diff model. The 3rd data base is after de program, so my hope is to use it with the 2nd data base to see if there has been an effect. For this the equation that I have to use is the combination of PSM and diff-in-diff (treatment=D period=t w=sumatory of the weights attributed): [Y(t=1)-w(D=1,t=0)*Y(t=0)]-[w(D=0,t=1)*Y(t=1)-w(D=0,t=0)*Y(t=0)]
Careful!  The KL divergence is not a true "distance" because it is *asymmetric*.  In each case, which of the two possible values have you computed?
Note that in removing the means you have made your sequences not quite iid: values now have a slight negative correlation.  In standardizing the covariance matrix you will exacerbate that somewhat. There is an inevitable trade-off between maintaining independence *within* each sequence and no correlation *between* sequences.  If that's ok with you, then the next question is one of computational efficiency: although there are readily available solutions (e.g., SVD), they may start to founder as $M$ and $N$ grow large.
+1, nice point. Along those lines, it's worth taking a look at @cardinal's excellent answer here: [removal-of-statistically-significant-intercept-term-boosts-r2](http://stats.stackexchange.com/questions/26176//26205#26205).
Have you looked at **subspace clustering**?
Thanks. The suggestion with the kernel and the local minimum is actually similar to what @EngrStudent proposed, right? Still I do not fully understand what is meant by it. How can I even look for a local minimum in a machine-based way? I.e. how can I calculate the first derivative of the "function" without knowing the function itself but just the values?
@IrishStat I've updated with some graphs. Thanks.
The list of organizations that use cc0 is a great place to start when looking for places to deposit (or access) data
(ctd)... I'd suggest a negative binomial would be reasonable first choice (see package `MASS` that comes with R, and I highly recommend the book), since there's no reason to suspect variance proportional to mean. [In comments you show code by enclosing in backquotes \\` ... \\` --- like so: `here's some code`  ...]
Thanks. Yeah, in real life, there are actually more than 5 data points.
@Alecos: that would have been helpful, but is sadly not the case. Whether it increases or decreases is a matter of parametrization. In the case of the exponential distribution, for instance, it will be increasing or decreasing depending on whether one uses $E(X)$ or $1/E(X)$ as the parameter.
Good point, but I don't have access to the original regression.
Thank you for helping me with the terminology; coming in with an idea on what I need and what I thought was a descriptive term, only to find out that the term is loaded with other preconceptions is something I experience a lot when developing systems for fields in which I'm unfamiliar.
As I said, "any likelihood function is in terms of parameters and input variables." You can find parameter estimation in the book, for example, Section 4.4 staring from page 114 for binary data.
hmm.  I wonder why the downvote?
good ideas - I can see aimbots coming out with human-like perturbations, detectors looking for too much consistency in perturbation statistics, and an ensuing 'arms race!'
PLS-DA = Partial Least Squares Discriminant Analysis. (I'll comment more on this and the possibility of combining PLS with linear discriminant analysis in a proper answer)
Do you mean something like, e.g., where the entries of a single column of the design matrix are permuted, holding the response and other covariates fixed? If you have a particular reference you are using, it might be helpful to list it.
thank you @cardinal
+1 This seems to be on the right track. I think Cleveland's principles can be pushed further. Recall that he asked subjects to compare quantities being displayed on charts. Comparison with angular displays was neither as quick nor accurate as comparison of *position* along a marked axis. In effect, a dial gauge gives a very clear position along a curved axis. As such it shares some of the good properties of graphics that use position to represent quantities. It also suffers a bit, too: we may have a harder time distinguishing speeds near the top of a gauge compared to speeds at the sides.
I found a nice paper, "Computation of the error function erf
in arbitrary precision with correct rounding, S. Chevillard & N. Revol" which is quite accessible. They start out by giving a straightforward sum from n=0 to infinity, which you can implement in 10 minutes, and then spend 3 pages discussing how to improve that.
This is close, but ideally I'd like to use a binomial not a normal distribution.
You may want to read a more in-depth tutorial than the Wikipedia page. 'A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition' [link](http://people.sabanciuniv.edu/~berrin/cs512/reading/rabiner-tutorial-on-hmm.pdf) by Lawrence Rabiner seems to be a good choice.   Another, shorter work that is also a good introduction to the subject is 'What is a hidden Markov model?' by [Sean Eddy](http://selab.janelia.org/publications/Eddy-ATG4/Eddy-ATG4-reprint.pdf).
BTW, A question about *Graphing Likert scale responses* just came across Andrew Gelman's weblog yesterday :) http://j.mp/aBm8mZ
CHAID screens various ways to merge categories in the predictor to select the the best merge which gives the greatest or the most significant association between it and the dependent var.
That's what I did in my example.  `model <- glm(USREC~GDPC1,family=binomial(link = "logit"),Data)`.  I'll add the yield curve, stock market returns, interest rates, and credit spreads as variables
@Amit, the reasoning behind this formula is not simple, you can get the references from the **strucchange** [vignette](http://cran.r-project.org/web/packages/strucchange/strucchange.pdf). Why do you want to implement it yourself?
Why are some variables "of interest" & some "control"? That is, if the point of the model's to forecast, why aren't they on an equal footing? Will some not be available for forecasting purposes?
Is this what you are looking for? http://en.wikipedia.org/wiki/Radioactive_decay#Universal_law_of_radioactive_decay
How small can the bin probabilities get?  Will they be within a few orders of magnitude of $10^{30}/K$?
+1 Thanks for the tip Jeromy, this would certainly help! The version I have access to is through my school PASW 18. Hopefully it also contains this functionality! Wouldn't that make life easy.
This should be asked on [StackOverflow](http://stackoverflow.com); I won't migrate since it is answered and you are new here, but please ask such future questions in a proper place.
log transformation or Box-Cox with small lambda shrink the tails.  That can work for some heavytailed and skewed distribution.  I don't know what if any transformations will work for very heavy-tailed distributions.
Wow! This looks quite challenging for me, so I think I am happy with the simple correlations. Thanks StasK
It would help if you let us know the reference in order to understand the context. At first glance, it seems like they are observing $n_i$ bins $i$ and then they take the product of these probabilities. That is how that $n_i$ climbed up there.
To decide on the number of clusters it is always recommended to plot somehow normalized objective function (called an internal clustering criterion) in place of the function itself. Look for Calinski-Harabasz and Davies-Bouldin criterions, most appropriate for k-means.
i am trying to measure whether or not i am getting more visitors after i turn on SEO, and if possible to quantify that "lift".  it's important to compare before and after on a per-account basis, and then say something about the comparison in aggregate.

it is possible that there is a "ramp up" period after turning on SEO, so i'm not sure that just looking at the days where SEO is on and the days where it is off is sufficient (maybe an average within a window before and after turning it on?)
...and the short answer is that the last equality is correct for well-behaved PDFs, so the answer to the question is that the .632 rule holds for a wide variety of underlying distributions.
I think part of the negative undertone is from the default style of ? in `R`.  I have seen `R` described as "expert friendly". This can put non-experts off.
(1) Is the multiplicative part negligibly small?  After all, the base level changed by maybe 1 degree of Kelvin, while the base temperature is about 300 K (roughly). (2) Is there any statistical evidence that the seasonal variation shows long term trends, or there is a long term change in seasonal patterns?
Looks to me like your calculation of $E[X^2]$ is not right (hint: it should be up near 23). You should also *explicitly define* $X$, to help save confusion. I'd probably have used the [law of total variance](http://en.wikipedia.org/wiki/Law_of_total_variance)
I don't agree on the test on the median. We're talking an ordinal scale here, and all tests on "median" are actually tests on location shift, which require distributions with the same shape. In an ordinal context, this is a rather dangerous assumption.
Watch out!  Your use of ks.test is incorrect.  According to the [help page](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/ks.test.html), you need to use 'pnorm' instead of 'dnorm'.  Moreover, setting the comparison distribution's parameters to the mean and SD of the sample itself will substantially inflate the p-value: "If a single-sample test is used, the parameters specified in ... must be pre-specified and not estimated from the data."
"*I can use the estimates as weights*" -- well, you would use the *inverses* of the variances as weights, not the variances themselves. I presume you already knew that though, but its important to make the distinction clear.
I do understand that this is pretty complicated ;) Each run of the simulation takes quite a bit (days!), so I really can't just run 1000 simulations. But I do need some kind of verification of my number, even it is quite low :-/
I don't know of an tree models, but `glmnet` and `e1071::svm` both support sparse `Matrix` objects.  `GAMboost` and `GLMboost` (from package `GAMboost`) may as well.
Deviation coding is the same as effects coding, right? I thought of effects coding too, but in order for the significance tests to be what the poster wanted, the data would need to be balanced, which they're not in his example.. here the tests would be evaluating the difference between group means and the **unweighted** grand mean, instead of the (weighted) grand mean (see [here](http://www.ats.ucla.edu/stat/mult_pkg/faq/general/effect.htm))
It is difficult to know what "net influence" might mean when the other variables will be strongly correlated with their sum. Perhaps what you want to do is a simple regression of Thai GDP growth against the sum of the four lagged US GDP growth variables (without any other variables included).
I actually understand better: http://www.statpower.net/Content/312/R%20Stuff/Exploratory%20Factor%20Analysis%20with%20R.pdf
How do you define $p((X\\mid Y)\\mid(Z\\mid Y))$?
@AdamO, I'd want to add for clarity that LDA as _data reduction_ technique doesn't rely on normality, just like PCA doesn't. Normality within LDA is an assumption for 1) statistical testing (Box's M test etc.), 2) classification.
Now that I have the estimated densities I find that the dimensions are not the same. The first one had 151658 (53*53*54 ???) items, while the second one had 159623 (53 x 54 x 55, perhaps?)  So by changing the gridsize which defaults to c(51,51,51) I think you have prevented a proper estimate. I say again  .... you need to post your code!
@DimitriyMasterov - I'm using both R and MATLAB.
Excellent Henrik. I also prefer the "points with means dislocated". Linking subjects with line segments may look too cluttered. Pity. As to homogeneity of variance I am a little more sanguine. The variance problem may not be as bad as it looks in the raw data. For the most part I suspect you will be comparing contrasts - within group differences. Contrast variances will be more homogeneous that the variances of the raw data. If raw measures with different variances are  compared (eg. Inductive vs Deductive in the MP-valiad & plausible group) a non-parametric test could be used as a back-up.
Thanks for the paper!
@Gung all samples are finite: I presume you mean finite *population* and you are referring to a finite population correction.
In general terms, it looks to me like survival methods are well suited to this problem, whether parametric (e.g. things like weibull regression), semiparametric (e.g. Cox regression) or nonparametric (things like Kaplan-Meier estimates of survival). What you might choose to do depends on what variables you have and the precise questions you deal with.
Your first paragraph gives me exactly what I wanted to know.
Ah, I see you added more information to how to get the desired results out of a triangular distribution. That's extremely helpful, and enabled me to accomplish exactly what I wanted. Thanks!
(+1) That's a cute little reference.
You might get more helpful responses if you have a particular application in mind.  The term "Granger causality," is unfortunate, given that strong evidence of "Granger causality," may be weak evidence of "causality."
Thanks, the reference is very helpful. The values range from 0 to 4 with a mean of 1 and SD of 0.5. So I think normalising is a good idea.
Also willing to accept an answer of "you are completely off here", just looking to get this resolved so I may properly report my results re-conduct the tests properly.
+1. Consider "missing" as a binary outcome and see if it is associated with any of the predictor variables using, for example, logistic regression. Not sure why CFA would be necessary. Like hammering a nail with a sledgehammer.
If the system is indeterminate, then presumably the size of the sample of consumers exceeds the number of products (as in your example) ... but it would be helpful to make that clear.
In addition, there are at most as many linear discriminants as there are predictors (features).
A recent sample examination is available at http://cisce.org/data/Specimen%20Question%20Paper/SPE%20QPS%202011/13.%20Mathematics.pdf. Apparently eight ten-point questions are graded (and given partial credit): to convert from 80 points to a 100 point scale, some rescaling *and rounding* are necessary. For a responsible analysis, though (in place of the overblown accusations of "cheating" in the references), it is important to know how the grading was done, how it was reduced to this 0-100 scale, and what the critical passing scores are.
Thanks a lot! This answer exactly covers what I missed from my readings so far, that's why I mark it as best answer. But new readers should definitely read catchmeifyoutry's answer as well, which is perfectly complementary.
+1 All of Statistics: it would be a great place to start.
Just calculate the the unstandardised solutions of your dependent variable using the weights of the responding measurement models and the manifest scores of the indicators as you have described in your question. From this unstandardised score you can derive the mean and the sd for your further calculations. I am not completely sure but that should do...
Can you elaborate on how exactly one would use it or give a reference?
Thank you so much :) I understand now completely what is left to implement and what I can change. Thank you!
There are books largely or wholly about this: e.g. Morgan, _Elements of Simulation,_ or Devroye, _Non-Uniform Random Variate Generation_.
-1. The statement you've written isn't true. Efficiency just means that the estimator performs the best in terms of your particular choice of loss function. The statement you've made not only tacitly assumes a particular loss function but also brings in asymptotic properties that don't necessarily follow.
I'm not sure what you mean. If the cooling time is slow enough, then this is what simulated annealing does. You can definitely show that the cooling times needed will make solutions exponentially slow -- I even mentioned this in my answer above. In practice you trade-off the guarantee of sampling all the modes for faster time to convergence, e.g. by using geometric cooling or cooling with reheating as a function of cost. You won't sample all modes, but you'll get to some mode (possibly only local) much faster.
Could be a duplicate of [this](http://stats.stackexchange.com/questions/73320/how-to-visualize-multi-linear-regression) then.
Welcome to the site, @RobertF. I tried to give a more informative title to this question based on my interpretation of it. Make sure you're OK w/ the new title, & change it if you want the emphasis to differ, but it's best to have the title summarize the main issue clearly & succinctly.
@guy Not necessarily discrete but why do you think it does not make sense?
your results are in three dimensions but they are either positive or negative?
(+1) The raw moments are defined even for negative values of $\\xi$, not just $0\\lt \\xi\\lt 1/2$, with the same solution.
I would recommend that you also try to estimate the variance-covariance matrix of the Poisson estimates using the Huber/White/Sandwich linearized estimator. That estimator of the variance-covariance matrix does not assume $E(y_i) = Var(y_i)$, nor does it even require that $Var(y_i)$ be constant across $i$. That would be less of apples-to-orangutangs comparison with the NB model. I can't tell if glmmadmb allows you to calculate the errors that way or not.
Since Principal Components Analysis and sampling have almost nothing to do with each other--one analyzes data and the other determines what data to collect--I am wondering whether you mean something else by "PCA". "Point of Closest Approach"? "Prompt Corrective Action"? "Physical Configuration Audit"? "Permanent Corrective Action"? "Power Control Algorithm"? "Process Control & Automation"? "Parallel Cellular Automata"? "Product Complaint Analysis"? "Probable Course of Action"? "Poodle Club of America"?
+1, as always great answer.