tttjjj/elt_bench huggingface dataset:
Excellent. I am glad you found the misspecification. I suppose if it works as it apparently does and you can do it right maybe there is a reason that justifies all this laborious work!  Look forward to hearing about it.
First you would need to write down the likelihood, then find the matrix of second derivatives with respect to the true parameter. This should give you a matrix with the elements (most likely) relying on the true unknown parameter. It's that matrix that you would plug the MLE to get an estimate of the Fisher Information.
It would be very helpful to add some context about your particular dataset and what you want to do in your analysis. PCA can be viewed as a dimensionality reduction method (helpful in a descriptive purpose) or a way to construct linear combinations of the original variables that can be subsequently used in a modeling framework, but as your question stands it is difficult to provide an answer.
You just made my point perfectly. You are able to bring your subject area knowledge to say that one of the variables has some characteristic that doesn't make sense - but that's *not* inherent in the numbers. There are many contexts where those numbers make *perfect* sense. What makes it unreasonable for your situation is the context which exists outside the numbers; that simply isn't a characteristic of the numbers themselves. If you consider a number of such cases either side of the borderline, you'll likely be able to elucidate some rules (criteria) that help distinguish them. ... (ctd)
Any other ideas? I have been reading a bit on Experience Sampling and it sounds very interesting, however the people interested in this project already have in mind what wants to be done, I'm thinking more along the lines of how to choose one unsupervised learning technique over another to find patterns in the consumption of things throughout the day. Any ideas? @AlexanderMeasure
You have two data sets. Do you have one or two different groups in each dataset? Your question is not formulated well. Please try to expand your question. Your data set is not properly demonstrated here.
Dalgaard's book includes exercise solutions (with code) as an appendix. Wasserman's All of nonparametric statistics does not. What about “All of statistics” and Venables & Ripley?
Please don't base any financial decisions on this /-:
I think points 2 and 3 are useful comments and consistent with what I see as the distinction between the two SA and DM.  I'm not so sure about your first point.  I've done statistical work where I was interested in improving understanding about causal relationships.  However, I've also done statistical work where the task was to take known relationships and develop models with the sole purpose of prediction but which shared non of the other features of "data mining".
I have the same general question. According to Wikipedia, [McNemar's test](http://en.wikipedia.org/wiki/McNemar's_test) should be applied to 2*2 test. Not an expert here, but you might be able to test in pairs: deviceA vs deviceB, deviceB vs deviceC, deviceA vs deviceC. I am not sure if this would be equivalent though but maybe somebody reading this could help
Apologize for insufficient info. So, I am comparing patient survival in a retrospective study between two different groups, using SPSS. Now the difference came significant by KM, so i tried to adjust multiple co-variates for confounding effect in Cox-Model.I did backward Wald by putting main group in block 1 and co-variates in block 2. The output window gave me
@proton So you want to fit a model BEFORE the uptrend, and then you want the model to predict the uptrend? That's going to be very hard to do with a univariate time series model.  Do you have any other covariates you could pull in?
Is the variance infinite when, say, $\\alpha = 1.5$? Or is it undefined in such a case?
Great reference for fitting distributions and qq plots in R: http://www.statmethods.net/advgraphs/probability.html
You say the model is an AR(2) but you talk about entering residuals? What precisely is e1-1 ? Is that SSC in the prior period ? . It is possible to restate your Transfer Function (armax model) BUT I would have to have the actual data that you used to form this model . Are all variables in log form ?
@Dilip I think the hardest part of this Jaynes problem is that he didn't explicitly state the formula to shoot for.  But the diagram allows us to see potential formulas which have a chance in Heck to be valid, and yes, the one I'm thinking of can be proved with just the product and sum rules (in fact, Jaynes did it in the paragraph immediately proceeding the original exercise!).
Are you asking for how one would go about writing an OLS function, or are you looking for one (which has been answered below)?
I assume you're familiar with T. Ozaki (1979), [Maximum likelihood estimation of Hawkes' self-exciting point processes](http://dx.doi.org/10.1007/BF02480272), *Ann. Inst. Statist. Math.*, vol. 31, no. 1, 145-155.
How much data have you got? How many years worth? Do you have data on anything like temperature? What is the rest of your model like? What are your DV and IVs?
Thanks a lot for your answer, Orlando. However, would you be able to explain why I shouldn't compare the F-statistic p-value?
Conditionally correct means that you know the error rate GIVEN that the true mean income is 0 or GIVEN that the true mean income is X, with |X|>0. These are much easier to quantify, and really the only possible if you don't know the true probability. Also (and I apologize for all the questions) are you assuming your data come from a particular distribution (e.g., Normal) or are you taking a non-parametric/distriubtion free approach?
Thank you very much. It really helps me.
You're right, it does. In that case three questions popup: 1) why didn't STATA recommend `log`? 2) why is the output of `ladder close` blank and most importantly 3) what are the gains of log transforming a skwewed independent variable in a binary logistic fixed effects regression?
Each value is uncorrelated. The series came from an exponentially smoothed normalized innovation squared of a kalman filter. Since the weighted sum of chi-squared is not a chi-squared, an approximation must to be done. The test try to check if the model used is ok for the input for the properties of the normalized innovation squared. I hope now it's clearer.
Thanks Shane, great answer! Well, "the others" from my field often use SPSS, so they use Kolmogorov-Smirnov (if they check normality at all), though IMHO the Lilliefors' test is a better choice when the data is gathered from a sample (when parameters are unknown). I was taught that Shapiro-Wilk's is appropriate for small samples, and just wanted to get more info about "small samples normality tests"...  BTW, I use nortest in R! =)
**NB**: Some of these comments and replies address essentially the same question (framed in the context of a housing price regression) which was merged with this one as a duplicate.
I would change a few things about this answer.  First, I wouldn't mention df so prominently, since that's only an issue of power, and reducing df by a few becomes irrelevant at the *N* most people would be dealing with when testing 40-ish predictors. But also, the effects described here on parameter estimates are not simply "another way of putting" the power problem (though probably you didn't mean to imply that).  Multicollinearity is a separate problem.  And it matters not just for the most recently added predictor(s) but, as the OP's econometrician indicates, all predictors.
Adding some details about the data and focusing on a particular step would help narrow this question, making it more accessible and more easily answered.
Is there any way to translate the constraint into one on the square root of the covariance? or the square root of the inverse covariance?
@ttnphns, no problem. Revised.
+1 for a clear summary, nicely adapted to the level at which the question is posed.  BTW, welcome to our site!
@Figaro Typically, we model the mean of the response; quantile regression methods allow you to model, well, quantiles (e.g. the median), which generally entails a somewhat more robust loss function than squared error.
Could you please link to or cite some references?
Hmmm. I'm not completely sure your explanation is correct - the default is to draw a 95% **point**-wise confidence interval.  I don't think that's quite the same thing as saying there's a 95% chance the true regression line lies within the shaded region.
Where are you stuck?  What approaches have you tried?
Because you don't need to; that is for relating the IV to the risk.
Ah, I see my mistake now. Thank you!
Tell us which article you read. It might help contextualizing your question more and improve the quality of the answer you get.  (The answer to your question most possibly is related with the fact you are working in higher dimensional space.)
Thank you for the reply.  I've updated my question - I had an error in my tables for new/full moon times where some were crossed, resulting in non-uniform results when I corrected the Monte Carlo to large, random $N$ times.
@Dmitry Laptev: my problem is that I wanted to filter some seasonality effect on the returns. Then I thought that dividing the returns by the seasonality curve would give me a better behaved returns series. I wanted to test my hypothesis using a GARCH model, but I do not know how to approach this problem
significantly overlaps with previous question: http://stats.stackexchange.com/q/72617/8960
Fair point about UCINET - especially since their QAP Logit is in beta... Do you think it would make more sense to run the analyses with 75x75 matrices (with rows for people I didn't interview coded as missing values) rather than use 30x75 matrices?
I disagree with Kaspar that false positives are *necessarily* worse than false negatives. I teach my epidemiology students that answering the question of the relative importance of sensitivity vs. specificity **depends on the consequences of being Type I wrong vs. Type II**. Cancer and HIV are excellent examples of where sensitivity is more important. However, tests informing immediate intervention with irreversible consequences require high specificity. Confirmation of which limb to operate on/amputate is an example.
@PeterFlom Thanks.  I thought something like that was the case. I think your answer is an improvement on gui11ame's suggestion and may be the best solution for Justin.  But I think he wants to show the referee that he can do the inference with his correlation estimate which would lead him to needing the distribution for the correaltion coefficient under dependence and the null hypothesis which is what the bootstrap might be able to allow him to do.
@csgillespie: Thanks for adding the python tag. :) I wanted to, but as a new user I couldn't.
Please edit your answer to use capitalisation as conventional. Consistent lower case may seem amusing or efficient, but it is more difficult for almost everyone to read.
Yes, you are right about Fisher. I got a non-significant result because I conditioned on precisely the thing I cared about, namely the difference pre and post! Not smart.
Could you use a Markov-Chain Monte-Carlo approach?
Can you expand on the "bootstrap-based approach" suggestion, please?
Thanks very much Stephan
@Tal But, your null hypothesis is "None of the risk factors impact disease". This is evident when you talk about wrong discoveries. Thus, you are using the same null hypothesis when you are performing these multiple comparisons and hence the need for control. If on the other hand your null was: "Age has no impact on disease" then you would not be interested in a study that did not have age as a covariate and hence your multiple comparisons would be restricted to the studies where age was used as a covariate. Therefore, it seems to me that my original point is still valid in your example.
I am still not 100% sure what your experiment is, but I think I understand your data. You have for every subject 7 entries specifying subject id, score, and block id. The blocks are added together to get a compound measure, i.e. test score. What you called blocks, I called item.

Now randomly assign items to groups and calculate the score for each subject and in both groups. You should now have three vectors: subject id, group1 and group2 score. Each subject id should occur only once! Then you correlate the score of group 1 with the score of group 2.
The links are broken
Are you asking for Python code to get the standard errors, or for how the SEs are computed (mathematically / algorithmically) so that you can do it yourself? If the former, this Q would be off-topic for CV (see our [help center](http://stats.stackexchange.com/help/on-topic)), but may be on-topic on [Stack Overflow](http://stackoverflow.com/). If the latter, it would be on-topic here (but you may not get any code suggestions). Please edit your Q to clarify this. If it is the former, we can migrate it to SO for you (*please don't cross-post, though*).
By making "Don't know/Does not apply" a category in itself, do you mean to assign a numeric value to this type of answers and then use it in EM (so to impute also with "Don't know/Does not apply")? Wouldn't the results differ depending on the value that I would assing?
Here is a discussion of the 'weights' argument on a message board: http://r.789695.n4.nabble.com/Weights-in-binomial-glm-td1991249.html
May I just emphasize what Aniko writes.  The biggest concern in my mind, after reading this question, is not about statistical procedure or software, but about what all this might *mean*.  Unless your data are an exhaustive survey (of a well-defined population) or obtained through a formal randomized procedure, considerable thought--of both a statistical and epidemiological nature--will be necessary to assure that you can learn *anything* from these data that is validly generalizable.
@jona: yes, you are right. The problem is that the functions that would calculate HPD intervals (mcmcsamp, pvals.fnc) are all not implemented in R for models with random correlation parameters. So I can't use those and I don't know how to get them otherwise. Maybe I should ask this as a separate question in case anyone has suggestions?
I think [this](http://en.wikipedia.org/wiki/Kernel_principal_component_analysis) is what you are looking for
Ok,how much are customers X? i.e. X=E[poisson(a)]= a ? is right? so (average waiting time)= a/lambda
@Dmitrij No this is just an estimate that I am interested in. No hypothesis testing like scenario.
The overall response distribution is a function of the kind of distribution (ie, normal, binomial, etc), the mean (for non-normal GLiMs where the variance is a function of the mean), & distribution of X. It is more complicated for general GLiMs, but you can get a sense from my answer here: [What if residuals are normally distributed but Y is not?](http://stats.stackexchange.com/a/33320/7290)
Lots of unexplained jargon/abbreviations (I fixed some but stuff like 'dev' - is that 'developer', or some other thing?), the use of statistical jargon like 'regression' to mean something else without clearly flagging it as such; and lack of a clearly defined question all combine to make this a very hard question to answer. We're not software developers.
The CLT is about distribution of the mean, not distribution of the data itself.

What the CLT says is that for examples like you have above, if you take the mean Epithel of many subsamples of your data, the distribution of the mean Epithel of the subsamples will be approximately normally distributed, the approximation getting better as the size of the subsample improves.

You're very fundamentally misunderstanding the CLT.
+1 I'd have been inclined to vote this question for closure as off-topic here (and indeed was just about to), but your general solution seems to make it highly useful for future users and more clearly relevant for CV.
No. The _variances_ of the variables should be along the _diagonal_ and the off-diagonal elements should be rho (if $\\sigma^2=1$).
I couldn't understand the answer
This algorithm is only best if the probabilities are cheap to compute.  For example if $n$ is huge it may better not to construct the whole tree.
@FanZhang - The last step is *approximately* correct.  This is because we can expand $g(X,Y)=\\frac{X}{Y}$ to first order about $(E[X],E[Y])$.  And we get $\\frac{X}{Y}\\approx\\frac{E[X]}{E[Y]}+(X-E[X])\\frac{1}{E[Y]}-(Y-E[Y])\\frac{E[X]}{E[Y]^2}$.  Because $E\\left(X-E[X]\\right)=0$ for any random variable whose expectation exists, we get $E\\left(\\frac{X}{Y}\\right)\\approx\\frac{E[X]}{E[Y]}$.
Thanks @gung. Just to be sure, the paired t-test still requires that the residuals of different subjects be uncorrelated? (my first post was somewhat misleading because I was speaking of "within groups" with regard to the paired t-test, by that I meant within the measurements at time 1 and time 2, in the case of repeated measures)
Related: http://stats.stackexchange.com/questions/4081/what-is-the-precise-definition-of-a-heywood-case
@ADB This [vignette](http://cran.r-project.org/web/packages/twang/vignettes/twang.pdf) for the [twang](http://cran.r-project.org/web/packages/twang/index.html) package in R has a nice overview of propensity score analysis and doubly robust estimators.
@Dimitriy Fine; same point applies.
Work on that accept rate Paul!
Hi @Alfahad, welcome to CV. Could you give a little bit more detail in your answer (perhaps detailing the steps you're referring to)? This way, even if the website you're linking to gets removed at some point, future readers will still be able to understand your answer.
(+1) We can write things a bit more suggestively by noting that $Y_i = \\log(X_i/m)$ is distributed exponential with rate $\\alpha$. from this and the invariance of MLEs under transformation we conclude at once that $\\hat\\alpha = 1/\\bar Y$, where we replace $m$ by $\\hat m$ in the latter expression. This also hints at how we might use standard software to fit a Pareto even if no explicit option is available.
Either you have to decide for adding another label C which is unclassified or you have to use a default label, say A. But that is dependend on your domain knowledge. not that you classify people where you're unsure to have cancer ;)
And changing all of the `m` predictors one by one and looking to see how the forest predicts differently seems sorta expensive. There's gotta be a better way.
Since you're using log transforms for both X and Y, consider reporting results in terms of percentage or fold change in Y per percentage or fold change of X, which comes directly from your analysis, and for which the standard errors will be symmetric about the predicted values. That might be easier for others to understand than back-transforming to the original scales, in which the back-transformed standard errors will necessarily be asymmetric and difficult to describe.
@ThomasB If I can help offline please feel free to get in touch with me.
Ah interesting, I see that using regression wouldn't need any transformation. Are we talking general linear modelling or something else would you say? Also what do you think about that truncated data?
@act00 This can be calculated using the estimated parameters $(\\hat\\mu,\\hat\\sigma,\\hat\\nu)=(−0.35162295,0.09719402,342.22742778)$ as ${\\mathbb P}(X<0.8)=$`pt((log(0.8)-muh)/sigmah, df=nuh)`. Note that I am using a log transformation in order to use the command `pt`. The resulting value is $0.9064$ which is very close to the value obtained in the lognormal case in your previous question.
I can see why you would be confused. I appologize for that, but a friend explained it to me. Thanks for you explanation.
I searched for _online detection algorithms_, and mostly found academic articles that are over my head. They may hold the answer, but don't pass my personal "simple" test.

Correct me if I'm wrong, but I don't think I'm looking for a peak detection algorithm. Once the errors have peaked, it seems that by definition I've missed my opportunity to ameliorate the worst of the issue. Apologies if my use of "spike" was confusing. I guess I need to predict a continued increase in errors or identify a large step up.
Great - thanks for the tips.
Thank you mention the concept. My first paragraph also meant the same thing.
+1: an important and interesting point.  It also reveals that the question is incomplete, because it does not indicate the purpose of the model.  (Costs would be less relevant for a scientific model that seeks to build an explanatory theory but would come to the fore in a predictive model intended for repeated use.)
You are making a rather large assumption that all the users of the model are in complete agreement about the choice of a cutoff, i.e., that they have identical utility functions.  As this would be the exception rather than the rule, a continuous proper scoring rule is usually called for.  And keep in mind that lasso, glmnet, etc. optimize a proper scoring rule (as they should), not proportion "correct".
A very good question.  I've given up trying to convince people to utilize PMML.  It's often not considered very difficult to just code one's own model description syntax, especially for easily parametrized models such as decision trees, SVMs, and GLMs.
How can you do this without knowing or having data on the players preformance capability? If you think a player's expected profit is 0 aren't you just assuming each player has a 1/n chance of winning where n is the number of players?
Closely related: [Why does a 95% CI not imply a 95% chance of containing the mean?](http://stats.stackexchange.com/q/26450/7290)
@James See the update of my answer.
:paul Please send me the data at dave@autobox.com and I will try and post it to the group. I am not sure exactly as the most efficient way to do that.
Yes the cross-validation tends to be better.  Look to see if there are other ways that your model is over specified, it is very unusual for a non-intercept model to fit better.
Peter, I used a correlation instead of a regression, but I guess the idea is basically the same: find shared variance. However, the data matrix is tiny (14 symptoms x 2 rows, 1 for importance, 1 for average score), I hope that is no problem? The correlation is .45, and I am not sure about the interpretation: "the importance of a symptom on impairment is positively associated to the mean, but the mean does not entirely explain importance?" Would you recommend a regression over a correlation?
Because the distribution of the denominator is invariant under rotations, you can rotate $X$ to equal $\\sqrt{n}X_1$, which reduces your question to something familiar :-).
Are you really sure about the computation of $P(a \\mid d)$?
However - you are right that the Python implementation *is* different to the R one in that it doesn't include the possiblity of using `INTERQUARTILE_RANGE / 1.34` if it is less than the estimated standard deviation for a given dimension. It does, for the Silverman case, match the formula on Wikipedia though.
Could you please state your initial problem at what you are trying to achieve. Also please explain what do you mean by graphing interactions, graphing the different regression lines, plotting the coefficient values?
@Roberto, you have not described it well and fully; you might be better off reproducing the whole algorithm from B&H book. Steve, I am no Ken Brewer, but I think Roberto is right (I checked with the book); SAS manual gives a simplified version only for $n=2$, while the book gives the procedure for arbitrary $n$. The undefined symbol $r$ is the index of the sampled unit. B&H don't say anything about what happens when $1-rP_i$ gets $<0$, which may happen practically.
Doing PCA before training a neural net can have two effects: (a) making the training go faster and find better minima due to numerical reasons. This can be improved upon by taking zscores (centering and dividing by standard deviation) after PCA. (b) If you don't keep all of the principal components (but say those accounting for 95% of the variance) you substantially reduce the risk of overfitting. I would definately try to do this if you have so little samples.