tttjjj/elt_bench huggingface dataset:
Thanks a lot for the answer, I'll do the inquiry when I'll have more time
Try look up the concept "[confounding](http://en.wikipedia.org/wiki/Confounding)". What you observed is quite common because sex is either i) not a completely independent predictor or ii) associated with other predictors such as age and education.
@Tu What exactly do you mean by a "risk difference"?
Deletion of terms is not recommended unless you can test entire classes of terms as a "chunk".  For example it may be reasonable to either keep or delete all interaction terms, or to keep or delete all interactions that are 3rd or 4th order.
Have you looked at discrete time survival analysis? It used logistic regression program, but is suitable for this type of analysis.
Man that is brilliant, thank you very much.
You may be interested in my question [List of situations where a Bayesian approach is simpler, more practical, or more convenient](http://stats.stackexchange.com/questions/41394/). (Although I think this Q is distinct enough that it is not a duplicate.)
The question is clearly answered by the help for the function - `?ets`
In that case it's not the "sample standard deviation," but the residual standard deviation. :-)
Don't forget to accept the answer ;)
For OVA - can you pick the class that has the largest probability (induced by Platt scaling)?
The peremptory tone of this question suggests it is [tag:homework].  What steps have you taken towards an answer?  Could you be more specific about where you need guidance?
What was the warning when you used `sd`? What does it look like when you type `head(data1)` and `head(data2)`? Note that if a variable has multiple columns you can't use `sd` on it, but `apply(mydata,2,sd)` should work; e.g. try this: `apply(cars,2,sd)` and see what it does (`head(cars)` will show you the first few lines of data).
65% vs. 17% is indeed a perplexing difference.  Maybe one of us could figure out more if you posted that tree, or at least a list of the main and interaction effects involved.  Also, is there an element of crossvalidation to your project?
Is there any way to do this without knowing in advance what the relative varience will be or is it just a case of having to take a guess at the relative varience to do anything at all?
Bound in what sense?  You need a context here.  What set of values are you trying to bound and how is it defined?  Read the Wikipedia segment I linked to: it has some good examples.  For instance, if you model the image as one realization of a set of possible images, then its entropy equals -1*log(1) = 0.  You can't do better than that!
Sure, that might work.  What you're doing then is tantamount to using the color value of a graphic to represent word density.  Color values are difficult to interpret accurately; graphical representations that map the variable of interest to *length* or *distance* usually work best.  That's exactly what a kernel density plot will do for you.  By varying the half-width you can range from extreme detail to a gross overview.
You could descale the neural network's predictions back to that of the individual data and then compare the two sets of actual errors. Presumbaly you are doing this on a test or validation set of data that was not used in the regression.
thanks, your math convinced me now
+1 Great answer: it offers many solutions, provides links to them all, and adds some advice about newer algorithms.
excellent: +1 for the details
This pretty much hits the nail on the head.  I'm trying to figure out to apply Machine Learning techniques to time series analysis.
The logistic link function does not require you to have any particular concentration of outcomes. If your data are not the result of a random sample (say, a case-control design), then your intercept is simply fixed at the ratio of positive to negatives (but is still the ML estimate). Likewise, your coefficients are ML estimates given the data.
Net worth is tough to use because it's really a numerical summary of a vector value: the assets *and* the liabilities.  A nonlinear re-expression of this variably typically is nonsensical.  Why don't you include assets and liabilities separately as explanatory variables?
Of course what U'd actually do is model it both ways andw
@Roman Every link on the site will lead to a whole semester worth of lectures. I know it's a lot, but they are very good. You can get a really good understanding of the theory that way. Also, you might not have to watch all of them. But there is a good chance you will want to. Unfortunately all the texts that I have read are from textbooks that are not freely available. I like [this one](http://www.beltz.de/de/psychologie/fachbuch/titel/statistik-und-forschungsmethoden.html) and [this one](http://www.springer.com/psychology/book/978-3-540-67919-6) a lot.
(1) Please see our [homework policy](http://meta.stats.stackexchange.com/questions/12). (2) The question did not ask for a numerical answer.
It's a time series, and the measurements are made at regular intervals. The other data isn't related to the 4.7's. 4.7's pop up under a variety of field conditions. The sensor will (hopefully) be fixed soon, but the statistical question remains.
Very informative and detailed. Many thanks for your answer, you've earned the bounty. Much appreciated!
Please clarify your equation. Either use the appropriate (tex-based) mark-up or add parentheses.
Hi Frank Thank you very much for your quick response which is highly appreciated. My dependent variable which is the level of women’s nutrition knowledge is categorised to either low or high level of knowledge. The independent variables are already designed as categorical variable from the begging. 
The goal of conducting the logistic regression is to find the best-fitting and most reasonable model to describe the relationship between women’s nutrition knowledge (dependent outcome) and their demographic variables (independent predictors).
@whuber, I can see the benefits in taking similar players and putting them on different teams, but that seems to be a non-optimal solution.  It's not what I was looking for, but it is interesting nevertheless.
**Hint** Upon rewriting the model as $Y_j=\\gamma_0+(\\alpha_{j,1}+\\alpha_{j-2})\\gamma_1+\\alpha_{j,2}(\\gamma_2-\\gamma_1)+\\epsilon_j$ the problem simplifies to testing whether a given coefficient vanishes. Now the Gauss-Markov theorem relates the change in RSS to the change in log likelihood...
I assume you mean, in R-speak, factors?  If so, yes it can, no need to do anything special.
Altheide (1981), *Social Forces*, **60**, 2 suggests that a "certain kind of highly publicized suicide story" may not have been defined entirely independently of subsequent 'plane crashes - sounds rather like the definition of ["famous rabbi"](http://cs.anu.edu.au/~bdm/dilugim/torah.html).
I will look into that. Thanks for suggestion
It's a bizarre interface. Use `regtol.int(fit, numeric(0), side=2, alpha=.05, P=.90)` to obtain the tolerance limits for the actual x-values (in the order they appeared in the original linear model). In the general case, it appears you can select out the rows for which the "y" column is NA to find the tolerance limits associated with the `new.x` parameter. Better yet, just modify the code for `regtol.int` to include the x-coordinates in its output: then you'll be sure they're correct.
I am not sure, but I dont think you could get only a confidence interval for my $\\hat \\Sigma$. I think it is also possible to estimate $\\hat\\Sigma$ through $\\frac{1}{R}\\sum_{i=1}^R\\hat\\Sigma^i$. If this is true and assume the theoretical backround for the bootstrap gives you the permission to take this estimation as a estimation for $\\Sigma$, my question is which methode you would recommend, because I have some doubts that bootstrap would work on a sample size of 4 in this case. So I need a reason that this method will working.
Is the coin fair? What are your thoughts on this? What have you tried and where are you stuck?
My apologies for the last line.
This was copy-pasted from [a reddit post I made in 2009](http://www.reddit.com/r/math/comments/9qbut/could_someone_explain_to_me_what_degrees_of/c0dxtbq?context=3).
Presume 4 should be a power.
@AlexandrePatriota The supremum exists, obviously, but it is not attained by the function. I'm not sure what the notation $\\arg \\sup$ is supposed to mean - there is no argument of $L_1(\\theta; x)$ which yields the $\\sup$ because $L_1(\\theta; M) = 0$. The MLE is defined as any $\\hat \\theta$ which attains the $\\sup$ (typically) and no $\\hat \\theta$ attains the $\\sup$ here. Obviously there are ways around it - the asymptotics we appeal to require that there *exists* a likelihood with such-and-such properties, and there does. It's just $L_2$ rather than $L_1$.
At the end of c), shouldn't you square the derivative of $g$?
That's what I am doing at the moment. However, this quickly becomes unwieldy as you increase the order of the chain. Does anyone know any good sources on the practical side of things? Some approximation methods perhaps?
The [first plot](http://i.imgur.com/BCsf31j.png) I mentioned was actually generated using code similar to yours (except I assume $\\sigma$ is known). I'm actually trying to sample the *parameters* of the distribution of $\\mu$, since it seems that they should be more precise than sampling $\\mu$ directly. Also I tried your suggestion and sampled $\\sigma_0$ instead of $\\sigma_0^2$, and it didn't seem to make a difference (the distribution was still flat).
Fisher's information gives lower bound in parameter estimation. It is a natural metric because it roughly says something like "in this direction the difficulty of my problem cannot decrease more than that".  What you call generalization bounds are upper bounds ? do you want to know the performence of the method that use Fisher metric (the large body you mention is a good list)? sorry but I don't really get the question :) can you reformulate that point ?
That's stretching the concept of inference too far. You can model relationships as you like, but in the case of a census, as @Kodiologist says, that's merely an exercise in data reduction: p-values, confidence intervals, posterior densities, credible intervals are all unnecessary, because everything is known & there's no probability model involved.
Many thanks but how does this relate to vaiance. (Yes, I am interested in the correlation test)
Right, should have entered this as a comment.  The AUC is a useful summary measure once you've finished fitting the model.  It is not the ideal objective criterion for selecting the model (likelihood should be used for that).  It does not provide a rational basis for a decision threshold as it assumes that utilities are data driven instead of subject driven.
+1: I think you brought it all together in one place. Excellent!
As I understand it the theory of generalized linear models says that *asymptotically* (i.e. as the number of counts per sample gets large) the Pearson (scaled) residuals of a GLM should be normally distributed; however, this is often only very approximately true for a data set where the mean is fairly small.  This is also different from testing the marginal distribution of the data, which won't be negative binomial even if the distribution conditional on predictors is NB ...
But while we're here: On exposition: have you mixed up measures of effect size, usually identified with a parameter, for measures of comparative evidence for a complete model? LRs only look like candidates for the latter. Also, if you want Likelihood functions alone or in combination to tell you everything the data are trying to telling you about a model, then you're basically a Bayesian. Because that's the Likelihood Principle. (Come on in, the water's lovely :-)
Note that you'd probably be more interested in logistic or probit regression.
If moving from Excel is the issue, you could try:

* http://www.coventry.ac.uk/ec/~nhunt/pottel.pdf 

* http://www.forecastingprinciples.com/files/McCullough.pdf 

* http://www.lomont.org/Math/Papers/2007/Excel2007/Excel2007Bug.pdf 

* http://www.csdassn.org/software_reports/gnumeric.pdf
Thanks, I try that too. :)
I'm aware that "sin" is possibly inflammatory and that that some aspects of statistical analysis are not black-and-white. My intention is to solicit cases where a given commonly-taught practice is pretty clearly inappropriate.
@chi Sorry for the confusion, I meant a proportion for forest cover, and a binary indicator for presence or absence at each country. I think I get what you are saying here, would the assumption of not having an evolving generating model essentially be an assumption of 'stationarity'?
I like permutation tests, so +1. Nevertheless, as @John remarks above, ANOVA may be fine even with unequal _n_, in which case a permutation test may have less power. (And it may be harder to implement and/or communicate.)
About your comment, what kind of assumptions are you talking about ?
Under the standard linear regression assumptions, fitting a "joint" (multivariate response) linear regression is the same as fitting single-response linear regressions to each of the response variables separately. This is a standard result that can be found in most any linear-regression text, e.g., G. A. F. Seber and A. Lee (2003), *[Linear Regression Analysis](http://www.amazon.com/Linear-Regression-Analysis-Probability-Statistics/dp/0471415405)*, Wiley, 2nd ed.
Hey Penguin, I noticed that in the example provided by IDRE (http://www.ats.ucla.edu/stat/stata/faq/proportion.htm) the same error message that you mentioned occurs. Could this error (actually a "note") be less bad then?
Can you say more about what you're trying to do? Is this SEM? Are you talking about visualizing the model (ie, making plots of the data & relationships), or about a way to input the structure you want to know about into the software for analysis?
It is not possible to start with a proper prior and end up with an improper posterior if the likelihood is a true conditional density!
"is it possible to predict the finish time for this task when a new task instance is given?" -- Are you interested in (a) how long it will take to finish the task, given that a part of the task already has been completed (and thus having an idea of how fast this instance will be) (b) how long it will take, given that the task just has started (and just knowing 'typical' behaviour from past tasks) or (c) predicting the time vs. percentage curves (d: while having already completed part of the task)
The loglik's you get from R above are NOT comparable across different model types (they leave out constants not depending on parameters!), so are of no use to you here.
I believe that when I partially differentiate the objective function with respect to one centroid, the points in the cluster of another centroid vanish in the derivative. So, the centroid we can get will minimize only the sum of squared distances of only the particular cluster.
Does "L, M, H, L" mean "low, medium, high, low"?
@masumeh In that case, I guess contacting the author is the best bet. If I were to advise a student, spend no more than 30 minutes to do an intensive Internet/literature search on this; when looking at the whole syllabus, this is really trivial. Do you best, and tell him/her what other answers you found. Also, there are some more potential links on [this thread](http://stats.stackexchange.com/questions/798/calculating-optimal-number-of-bins-in-a-histogram-for-n-where-n-ranges-from-30). After you've learned the answer, you're welcome to answer your own question; it's allowed here.
@ Sri. Indeed i do have h(x) (the alpha are known) and therefore h(x)^+. But h(x)^+ is not a polynome. For reason pertaining to computational issues and ease of derivation of some properties, i need to approximate this function by a polynomial. Since h(x) [and therefore h(x)^+] are convex functions, i though, maybe, there should be a way to approximate them by some polynome, over the entire domain (R^2).
[Here](http://stats.stackexchange.com/questions/15776/measuring-individual-player-effectiveness-in-2-player-per-team-sports/) is a *very* similar recent question.
How about for example... communication apprehension? A high apprehension has no definite difference with a very high apprehension, right? But I've seen that the variable has been correlated with other variables using Pearson's r. Is that totally okay? Thank you!
You might want to take a look at this: [simulation-of-logistic-regression-power-analysis-designed-experiments](http://stats.stackexchange.com/questions/35940/), & this: [calculating-statistical-power](http://stats.stackexchange.com/questions/21237/).
I've edited my answer accordingly.
Generally, no, but I don't know nearly enough details about your paper. The shape of the noise is an assumption of your regression model and usually not something you need to specify in the paper.
I think you have some vocabulary confusion; do you really have information on two *variables* rather than parameters?
This is not really a different question: the solution for a univariate PDF works, *mutatis mutandis,* for multivariate PDFs, and conceptually it's no different: you have to account for the differential change in area and (according to multivariate calculus) that is done by introducing the determinant of the Jacobian.
Thanks @cbeleites. The card number is changed once fraud is noticed by the customer and it's not used as a predictor. What the training is after is the patterns of fraud. The 12th month is with new cases of fraud that, often, have the same patterns. There is no other relationship with the past than the patterns. Thus months 1-10 are to extract the patterns, 11 is for validation and tuning and 12 for the test.
5 think that if you can characterize the properties of the noise distribution the way you have then you can pick a parametric form with those properties. I think to summarize the family must 1. be defined on a finite interval, 2. allow left skew, right skew and symmetry.  and 3. has a variance that increases as Nt increases.  The beta distribution fits the bill for 1 and 2.  The fixed interval is [0, 1].  So to allow the variance to increase we could add a parameter c that spreads the distribution to the intervsl [0,c].
@pedrofigueira I have a 1 dimensional CCD of 7926 pixel of 7$\\mu m$ each one. My CCD analyze a region 2 dimensional, and then it steps forward 200 times. So, i have a matrix like this http://i57.tinypic.com/xn5f7s.png. I select the background between 40 and 60 pixel and i calculate for each row the mean value. Then, i project these 7926 mean values over the Y axis (like here http://i61.tinypic.com/rqxieq.jpg ). So, for each bins of the row of the matrix(i.e. 3040) , i subtract the value of the content of 3040 pixel of the background
This is a very helpful comment. I will use the 4-level variable to compute the brant test, and leave the 5-level variable as the dependent variable in the full model. Thanks.
Do you have a proof at hand?
@StéphaneLaurent, the Jeffreys prior is computed independently from other parameters.
(1) or look for non-linearity in the terms (2) or explore possible interactions
By online learning, I get one example and do a prediction. At the next time step, I get the actual target for my example and do a gradient descent update.
Here is a reference for your answer: "Andrew Gelman, Bayesian Data
Analysis, pp.482" as given here: http://www.mathworks.com/matlabcentral/newsreader/view_thread/22703
@chakradhar.T : If you had questions about using a publicly available dataset, or how to distribute it for others to use, it'd apply on here, but your question currently just seems to be a general programming question on data mining.  As I don't see a good fit for your question on this site, I'm going to mark it to be moved to Cross Validated, which covers data mining topics.
@Glen: Maybe n is unknown.
I would use simulation to calculate the power. Anyway, given the truncated nature of your data, the use of a t-test may be questionable (I'm assuming that you're going to carry out the t-test on your truncated variable. This is what I understand from your question).
1/2 Thanks! Very clear. Let me check if I got this right: In LSI, documents are formed by a mixture of words (no notion of topics) and words and documents are mapped to a lower dimension semantic space using SVD. Since words with similar semantic meaning are mapped closer, it can deal with synonymy but has problems with polisemy. pLSI solves the polisemy problem by introducing the concept of topics. In pLSI, words are drawn from a multinomial distribution of words (topics), the same word can belong to several topics and a document has multiple topics, although this is not modeled explicitly.
Thanks. This is still not too clear. Please *explicitly* identify the function you want to check, the variables/arguments you are testing the sensitivity of, and what type of uncertainty are you modeling (percentage error or absolute errors?)
Thank you for the suggestion. This seems like something that can yield some results. Could you explain further what you mean by 'fitting a stable signal' vs. 'fitting to noise'?
And when you do this computation, what answer do you get? :-)
@Glen_b If it helps, I'm using the Vegas algorithm from the GSL library.
Before going any further, you ought to read about [the distinction between estimation and prediction](http://stats.stackexchange.com/questions/17773).  Here, $s_1$ *predicts* $s_2$; it cannot estimate it.
+1. Using *Mathematica* and applying it to the *data* (rather than the posted covariance matrix, which may have been presented with too little precision) I find *no* negative eigenvalues.  That is as it should be: when a covariance matrix is computed exactly, it's guaranteed positive semi-definite, so *any* negative eigenvalues must be attributed to imprecision in the calculations. Any decent generalized inverse procedure ought to "recognize" those tiny negative values as zeros and treat them accordingly.
It's not a 5% CI it's a 95% CI and the second p value says you have a 97% chance of getting a  t at least as extreme as the one you got if the null is true.
Hi Affine, thanks for the fast reply! Can I say MA is like an AR for the error?
I will add dummy variables for the surgeries and work through the usual regression workflow. Thanks, Scortchi!
Start by deriving the likelihood ratio statistic, determine if it's monotone, show that it's equivalent to a simple hypothesis, and use neyman Pearson.
How about a kernel density plot? http://www.statmethods.net/graphs/density.html
I can break it up in three groups of 30, and when I remove the x-axis labels it looks quite good. Thanks!
Hi thanks for that, yes scaling would  not have worked with the regular matlab svm, it works better with libsvm.  SVM normalizes everything.
The second summary is `summary(0:6)`, that is, the summary of the unique values 0, 1, 2, 3, 4, 5, 6.
Oh, OK. Then I think this is not a statistics problem but a math problem; finding where 68.2% will fall is known... I forget the answer but it involve $\\pi$.
@Charlie, I think you may have accidentally dropped $y$ from your first equation for $\\hat\\beta$. Also, I don't quite follow your equation for $R^2$, I'm used to seeing $\\sum(\\hat y_i-\\bar y)^2/\\sum(y_i -\\bar y)^2$ or $1-(\\sum(y_i-\\hat y_i)^2/\\sum(y_i-\\bar y)^2)$. I'm interpreting your $e^2$ as $\\sum(y_i-\\hat y_i)^2$, so I'm confused; is there a way that can make this clearer for me?
What source are you looking at (or what does it say, exactly)? A minimum of two random variables $Z=\\text{min}(X,Y)$ just says that when you realize $X$ and $Y$, $Z$ takes the value of the smaller. So for example if you have $X=-2$ and $Y=1$ then $Z=-2$, but if $X=2$ and $Y=1$ then $Z=1$.
Is smoothing really necessary? I mean, i got weights saved as real values. So I may create weights for interval. If weight of tf-idf is less than 0.1, then it has value 1, if less than 0.2 - value 2. It is more like rounding up than discretization. Will it work or should i just go back to Boolean model?
I don't think there is a closed form solution for the MLE of the regression parameters in most glms (e.g. logistic regression). Linear regression with normal errors is one exception.