tttjjj/elt_bench huggingface dataset:
To steal a pun from xkcd, she did not become your statistically significant other?
You are right! I just compared the "DeMorgan formula" to the recursive approach and the results are exatly the same. So in summary recursion works, however only if the events are independent, BUT in this case there is the far simpler approach using DeMorgan. THANKS!
Cool! This is a good place to start!　Lots of relevant courses I've found there. Thanks a lot
Nope; even if you ask for more data the experimentalist can still extend the animal deception and further obfuscate the pattern to hold it not obvious. I.e., extrapolation simply can't be done with learning because by definition it requires information that is not present in the training -- this way you must either apply some assumptions or gather additional data so that the problem will become interpolation.
@Alecos: This is just an exercise. The point I am trying to illustrate is that most of the time we don't know the model beforehand like in this example. It would have been good if we had more tools for automation without human intervention.
Very nice!  Thank you for following up on this question.
@Matt It sounds like 90% of the residuals are zero.  If that's the case, no transformation is going to make the residuals remotely close to normal.  Simulation studies have shown that p-values from F-tests are highly sensitive to deviations from normality.  (In your case it's fairly likely that some denominators in the F-tests will be zero: a sharp indicator of how far things can go wrong.)  You need a different approach.  What to do depends on *why* so many residuals are zero.  Lack of sufficient precision in measurements?
@Jim Often the placebo is a treatment that is the current standard of care, rather the absence of any intervention.
**Focusing on High schools**
• How many high schools did you contact? How did you select them? Did they include private as well as public schools? In what way, if any, would you consider them representative of  high schools which might teach physics?
• How many schools provided any information? Of these, how many had physics programs?
Hello everybody thanks for your reply. On the vertical axis there is the perceived velocity. On the horizontal axis I have the velocities, on a 4 point scale. Just velocity 1, 2 , 3 and 4. It is a general case, I don't care about the differences between the velocities (Please note: this velocity example is just a generalization of the real problem that I am trying to solve...). In the first regression I just have 4 points (the average perceived velocity for the 4 velocities). In the second regression I have 120 points (15 participants * 2 trials * 4 estimate of the velocities).
Please give full references.
Have you tried specifying the weights in the predict function? The help documentation for it shows almost your exact same example with some discussion. 

http://stat.ethz.ch/R-manual/R-patched/library/stats/html/predict.lm.html
I'm still struggling.
Just an idea, but how about create summary statistics for each group, and then use ?hclust to group the categories based on aggregated data?
(+1) Actually, when $\\sigma_1 \\ne \\sigma_2$, the equation can be solved with the quadratic formula: no need for calculus.  If we arrange (wlg) for $\\mu_1 \\ge \\mu_2$, then the second density is smallest between the two zeros and otherwise the first density is smallest.  This reduces the calculation to four evaluations of a Normal CDF.  The situation with $\\sigma_1 = \\sigma_2$ is even simpler, requiring solution of a linear equation and only two evaluations of a CDF.
Hi gung: this is my first time posting so I see the editing changes you made and I will conform to them in the future. Thanks.
You have to make strong assumptions about the lifetimes.  Without them, you would need to test approximately 3/(0.1%) = 3000 devices.  For example, suppose 1% of the devices have a relatively rare defect that causes them to fail after 100 reprogrammings.  Your chance of encountering one of them during your testing of only four devices is approximately 4%: pretty low.  Thus, *based on the data alone,* you can have very little confidence that such defects don't exist in the population of microcontrollers.
Hey, I think you should be more specific as to what it is that you're doing and what your problem is.  Your question, as it currently stands, probably won't get a very good or satisfying answer, and probably will get a number of downvotes.
Your problem is the best fit will be obtained when `a` and `b` are huge and `c` = -0.5
I am not sure if you mean the exact version of McNemar's Test. [Breslow and Day (1980)](http://w2.iarc.fr/en/publications/pdfs-online/stat/sp32/SP32.pdf), p. 164-166 and [package](http://cran.r-project.org/web/packages/exact2x2/vignettes/exactMcNemar.pdf) `exact2x2` may be references.
"Less than 94%" in a sample of 1000 CIs is surely not significant evidence against the idea that 95% of CIs contain the mean. In fact, I would expect 95% of CIs to indeed contain the mean, in this case.
$\\sigma$ is unknown. If the sample mean of all of your bootstrapped medians is $\\bar x$, then if you want to estimate a range of plus or minus 1 standard deviation, you would use the range $\\bar x - s$ to $\\bar x + s$.
In response to your question about alpha level, I think an FDR correction is in order, since you say the features are dependent on each other. This should substantially lower your Type I error rate.
Welcome to the site, @Reeza. I wonder if you'd be willing to flesh out this answer somewhat to be more helpful to the OP & future readers of this Q. Can you say more about what survival analyses are, & why they're a good approach to this situation (some links or pointers to more info might help, too).
"*in order to estimate the significance of the distribution parameters difference*" --- why would you do that? What's the underlying problem of interest?
See @gung's answer [here](http://stats.stackexchange.com/questions/33433/linear-regression-prediction-interval). Nothing to do with heteroscedascity.
RLAI looks a bit old - some links don't work, some links point to old frameworks ("Reinforcement learning toolbox" was last updated 2006, PIQLE - 2007, Connectionst Q-Learning - 2008). It could really help to know which of them really works, before I go through the task of installing them...
(+1) the idea is a tad basic, but of course, why not
+1 A scatterplot matrix would occupy the same amount of space, take the same amount of ink, and hugely increase the [data-ink ratio](http://en.wikipedia.org/wiki/Edward_Tufte).
Non-monotonic relationships (a better word here than "correlations") should be dealt with on their merits: e.g. try a quadratic if it looks quadratic. Note that such changes may increase the degree of multicollinearity.
You might be interested in the Amelia package which has some features for assessing imputed data built in. The vignette for the package is pretty comprehensive (pdf): http://cran.r-project.org/web/packages/Amelia/vignettes/amelia.pdf
BTW, the VIFs do not have the usual interpretation (i.e. the factor by which the standard errors are inflated due to collinearity) unless you're doing OLS regression. While it may be an informal measure of collinearity in other situations, like the binomial GLM you're using here, it's not clear exactly how to interpret its value and it's not obvious that the same "rule of thumb" (e.g. VIF > 5 is bad) is applicable.
What is your interval between measurements , daily ? weekly ?
No sorry, it was a typo, now I corrected it.
I have no idea. What I am saying is that if this particular forecasting method did not predict it, putting this fact under the rug by redefining the test after you looked at the results will not make your model better next time around and leaves you with no idea of how good it will really be in the next 24 months.
@Lost1. hello belive me i am try solve question and in this case I find way and solve this question. many of my question vote up and answer and you mr you Lost1 downvote 37 point my reputation While i read your question i voteup 5 your question i am your friend sometimes i need answer so i ask in to site . anyway goodluck.
Um, no it is not a homework problem. I'm actually in a city with these statistics and trying to figure out how often I will see an Indian person. I have no idea what to try because I haven't done a Poisson distribution since I was in undergrad and had a bad teacher. It's ironic that I formulated it so clearly that you think I'm doing a homework problem. I finished my PhD in Computer Science 2 years ago.
P(A or B) = P(A)+P(B) if A and B are mutually exclusive.
I don't understand the hint. Do you mean yes?
If this is a homework problem (sounds like it) it should have the self-study tag.
@Henry I'm not using a linear regression. Rather, I set up an equation involving the three variables and several parameters, and used Numpy to find the values of the parameters which minimized the sum of the squares of the differences between both sides of the equation for each data point.
Note that your terminology of validating vs. testing is not followed in all fields. E.g. in my field (analytical chemistry) *validation* is a procedure that should prove that the model works well (and measure how well it works). It is done with the *final* model, no further changes are allowed afterwards (or, if you do so, you need to validate again with independent data). So I'd call your validation set an "inner test set" or "optimization test set". The "outer" test data does *not prevent* overfitting, but it may be used to measure the extent of overfitting.
@MichaelChernick - I agree, it's pretty rare that you wind up comparing two distributions with medians different enough to detect but means that are pretty much the same (or vice versa.)  However, it is a "gotcha" if it gets written up.
I did mean absolute area, I'll correct that, thanks. The idea of weighting fits at different x values is an interesting one. I think you mean based on a biological interpretation of x, for instance, one range of x may represent infancy, another old age, etc. If so, that makes sense to me. I also wonder if the sign of the slope should be taken into consideration seperately from differences in the magnitude. There may be a very different biological interpretation of the difference between slopes +1 and -1 and between slopes +1 and +3.
Can you post your code? 

Which PROC are you using?
$n^n$ is such a crude approximation of $n!$ that one would be excused for doubting this approach.  However, Stirling's (asymptotic) approximation $\\log(n!) \\sim n \\log n - n + O(1)$ also leads to the desired result, at least for large $n$, because $p_1 + \\cdots+ p_d = 1$.
If corrections need reruns isn't your estimate of 2900% on the low side?
CV isn't really a code review site (there's http://codereview.stackexchange.com/ for that!) ... -- for it to really be on topic here, you'd usually need to ask a much more specific question about your implementation of the algorithm (i.e. you'd focus on the algorithmic details rather than the code specifics).
Welcome to the site, @Sowmyalyer. Would you mind adding some text to introduce & explain your answer more fully?
Thanks for your point, I made a rough correction. I'll further ask the OP for more details.
gappy specifically stated that he has paired, i.e. non-independent samples, so assumption 2 is violated right away.
Use an `nrows=1000` argument in `read.csv` to get the first 1000 rows. See `?read.csv`. If your question can be answered by "*read the help page of the function you're posting about*", then your post shows a distinct lack of research effort.
Yes. You can transform them back to the implied coefficients and standard errors from the "raw" polynomials, for example.
Of what are you taking a mean? The 70 likert items? How large is your sample of means?
This equation is wrong according to Wikipedia, Matlab, R, and others which are implementing this equation. The numerator here is squared, but it should NOT, it should be just like the (C) proposed by the OP. See http://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_variance
Is it correct that you have two separate univariate multiple regressions? (`intimate ~ numbing + reexperiencing + hyperarousal` and `communication ~ numbing + reexperiencing + hyperarousal`)
It is not so much about classification. My point is that LDA can be seen as dimensionality reduction technique, so you can use it to plot all your data on 2D (similar to how it usually done in PCA) and try *"to describe the reduced coordinate system in terms of the original variables"*, as you wanted to do with PCA.
I think that if you ask a question that can be understood by people that don't know what the hazard function is and if you elaborate a bit more on what you do (how do you estimate the parameter of you'r gausian process, how do you use the gaussian process at the end) you will increase the chances to get an answer and this will be an added value for stats.stackexchange :)
Thanks a lot. It's perfectly what I wanted.
+1 A very nice take on the question. While the limiting geometric is one of the cases where you can "work it out from the above table of single digits and the knowledge of the distribution" (as I mention in my answer), it is arguably a limiting case that would be 'natural' to consider.
@Ben you can look at Gelman and Hill, [_Data Analysis Using Regression and Multilevel/Hierarchical Models_](http://www.stat.columbia.edu/~gelman/arm/), §4.1 "Linear transformations" and §4.2 "Centering and standardizing, espacially for models with interactions".
What made you care about an ROC curve?  What made you choose a classifier instead of a direct probability model?
@gung, thanks for the edit, looks good except that i would prefer to have "text file" replaced with something less specific as what i am looking for is more general than just a text file.  it could be said "occurrences within a domain" where the text file would be translated to the hypernim "domain".  please consider
Hi @mbq..thanks for your response..and do you know what are the definitions for active and inactive covariates? are the following line correct? active covariates in a simulation study are the non null variables used in the simulation model and inactive covariates are the rest of the variables in the simnulation study which are null and have not used in the simulation model? thanks
@ttnphnns I want to get an impression of which type of method in general are there.
Thank you very much for your useful answer.I want to use general form of lasso which elasticnet to select a group of highly correlated variables.if you have any idea which is better please tell me your opinion.
I agree that you absolutely must not ignore the individuals without observed events. With 5000 observed events you should have a good chance of detecting any important differences among your survival curves even if none go below 80% or so. This is similar to the situation in analyzing survival in better-outcome types of cancer. Patients do care about the difference between, say, 95% and 75% chance of survival.
How many points?  The bootstrap will still be a good estimate of the confidence limits - it is based on the actual distribution, rather than making additional assumptions as some statistical tests do.  You can always add a small amount of jitter to improve the smoothness of the result.  The jitter just needs to be much less than the width of the distribution.  For example, if the distribution has a typical width of 1, then when you do the resampling add Gaussian noise of eg $10^{-3}$ to each point.  This won't affect your significance test materially.
I agree with @gung 's comment a lot. It is a case of survival analysis. If you have an subject, that you followed for 100hrs without seeing the 'event', it can be treated as a right-censored. If you have an obs. for which the 'event' time is observed at 99 hours, then it is a exact time-event. If you have multiple events observed for one subject, it becomes a repeated event or Recurrent event survival analysis. You can have a subjects tracked for only 10 hours (without seeing an event), then it is right censored at 10hrs. You can have subjects that were tracked for different amount of time.
you might be cautious with your use of terminology, especially when addressing someone new to stats. For example, pivotal quantities are **not** statistics, so using the phrase "pivot statistics" can be (very) misleading. The OP uses "pivot functions", which strikes me as a little unconventional, but certainly more correct. Sorry if this seems pedantic of me. Cheers.
I didn't check the correlation between fitted values and observations. I can calculate it and add it to the question if it helps the answer. I am puzzled about how the predicted values improve the naive forecast by almost 30%, despite the quasi-null $R^2$. About the MAE: good point! I multiply the actual MAE value by 10000 as most values are < 0.001.
The problem with Step 1 is that if tou need Step 2, your results from Step 1 are probably useless. Furthermore Step 1 requires you to specifiy 1) how many trends there are ; 2) How many Step shifts there are ; 3) that the seasonality is fixed ( dummies_) or autoprojective ; 4) the errors from the model have constant variance etc. etc. etc. If your goal is to develop a useful model, this may not be served  by doing what's wrong/insufficient first. Ease of use and speed are of interest but should take second place ( a distant second place !) to good analytics.
@biostat: An unbiased estimator contains no systematic error, but could contain random error. Thus as the sample size increases, variance decreases, and the estimator converges on the true value of the parameter in the population. A biased estimator would contain systematic error and would not converge on the true value of the parameter in the population (unless multiple biases in the estimator happened to cancel each other out).
And I am trying to re-arrange the given integral so that I can find  an unbiased estimator for $\\Phi(\\mu)$. Any ideas?
Why do you want to normalize the data points first? May be you just want to find a linear combination that explains most of the variation in 3 variable triplets? Simply if you do normalization then it seems to give you back symmetrically-like distributed score and? do you need this one?
I will point to some other posts and they make no discussion of outliers.  Perhaps NN have advanced with respect to outliers, but I am not aware.... if you take the time series 1,9,1,9,1,9,1,9,1,5...will it find the 5 to be an outlier? http://stats.stackexchange.com/questions/8000/proper-way-of-using-recurrent-neural-network-for-time-series-analysis and http://stats.stackexchange.com/questions/9842/getting-started-with-neural-networks-for-forecasting and http://stats.stackexchange.com/questions/10162/how-to-apply-neural-network-to-time-series-forecasting
Thank you greatly. Finally a reference that raises the issue of table size (df) and cell significance. Since I had seen no mention previously I thought perhaps something was happening in the test calculations that accounted for the multiple cells. I just wish I had some sense of approximately how large a correction would be appropriate, even without an exact value. Just df, or maybe closer to df/2?
Differences in scaled chi-square are not the same as differences in chi-square.  http://www.statmodel.com/chidiff.shtml
In many cases, such descriptions should be in terms of some convenient unit, rather than in terms of standard deviations. One advantage of working with [standardized coefficients](https://en.wikipedia.org/wiki/Standardized_coefficient) is they're scale free, and they do at least give a way of getting an amount of change that you can find in the population, since a standard deviation is a kind of typical deviation from the mean... but other people (I can think of several) are no doubt better placed to explain the benefits than me.
Thanks, StasK.  Based on your answer and the link you provided, I have come to the conclusion that I am chasing significance.  I think I will return to ivreg and do some endogeneity tests to see what I get.  The model is using survey data to test if female migration contributes to household asset accumulation.  An asset index was created using multiple correspondence analysis.
I see your point. Normalized XCC takes care of A2=12B+10000 but for probability it may be a problem. What score would you use to measure the "closeness to equality" for A and B of same size, typically 1000 values / pixels? Normalization (A2 example) is vital for me, too.
Did you try `traceback`? That's an odd error, but what it seems like is that there's a function inside another function, the inside function is being passed the argument `...`, but the function it's inside doesn't have a `...` argument to pass. My guess is tslm is somehow calling the wrong thing, or is being told to call something that it doesn't expect to call.
Thank you, this makes it a lot clearer. For those of us who are just beginning to learn stats, IID = Independent and Identically Distributed.
I agree strongly that this is not "a fairly simple problem"! I wrote something like that in an early draft of my answer. What can be inferred from the rest of your data, if there are any, is a crucial aspect I did not address. If my suggestion has merit, it's that it is much simpler and easier to implement than Glen_b's more advanced suggestion.
@Macro - thanks for your comment - after a couple of edits, I think it has resulted in a better answer.
I find your example a little confusing, John.  It refers to a situation with extremely strong distributional assumptions, but it immediately follows a remark about having "absolutely no assumptions whatsoever on the prior distributions," which sounds like exactly the opposite situation.  Which is the situation you really have in mind?  Also, it is not clear what you mean by "strength."  Initially it would seem to refer to force of evidence, but in the example it instead seems to mean the degree of detail (such as going beyond just comparing means).  Could you make that idea more precise?
What function are you minimizing exactly? (It doesn't make sense to say you minimize *MLE*.)
Although it's not needed in your specific example, it can be needed in general settings.  It sounds like you are stipulating that the total is known.
I am not sure we are not abusing the comments section by this discussion (StackExchange suggests moving it to chat). Still, to answer your question: well, you project your "passive" dataset onto the "active" PC1 direction (Z=X*W, where X is your dataset and W the PC vector), then you compute the variance var(Z) and then you normalize it by the total variance of the dataset (trace of covariance matrix of X). It is exactly the same procedure you can use to compute the amount of "passive" variance explained by the "passive" PC1, no difference at all (only that it is not given by an eigenvalue).
@MichaelLew I don't share your dim view of the Neyman-Pearson approach to hypothesis testing.  I still think it is fundamental to frequentist inference. It is only the strict adherence to the 0.05 level that I object to.
Dimitriy V. Masterov, all of two other alternatives to test heteroskedasticity in a Tobit model with Stata 12, I can't find a free code. Would you please help me to find a free code. Thank you so much
Thank you for the reply. I went through the specification (http://fimi.ua.ac.be/data/retail.pdf); the dataset also appears in multiple sources (in the same format) with no additional remarks, which would imply it is, for lack of a better word, "self-evident".
Hi there, thanks for these very useful suggestions! Any further recommendation about the tools? E.g. you mentioned Lingpiepe, will that be more suitable for this purpose than, say, WEKA or GATE or others. Because I am not familiar with them. Thanks.
Although [this does not answer your question](http://stats.stackexchange.com/a/17148/919), it may shed some light on it.
Welcome to the site, @nkhuyu. I don't fully understand your question. Are you worried about multicollinearity, or are you worried about finding an incorrect sign due to sampling error alone?