tttjjj/elt_bench huggingface dataset:
This makes sense if the regressors are all measured on the same scale. In psychology, various subscales are often measured on different scales (and still correlated), so a weighted sum or average (which is really the same here) would be appropriate. And of course, one could view PCA as providing just this kind of weighting by calculating axes of maximum variance.
+1 that looks cool -- thank you, I will take a look. But I agree that it doesn't fit any of the preferred properties and at first hand doesn't seem to be really measuring the complexity of the network as much as its performance... but I guess those are inseparable.
can you post an example of your dataset so we can try to reproduce it.
You have asked "how do I perform statistical inference?" The answer requires a textbook-length treatise. I would guess your problem has something to do with conditionally Normal distributions, but because the details are buried in code, only people familiar with Matlab--and with the willingness to dig through the code to find the key parts--will have that essential information. Please edit this question to provide the details of your setting: and then be prepared for the community to point out duplicate threads on estimating parameters of Normal distributions.
@steffen: CSV doesn't store any information about the format or data-types of the data stored in the file. You can well open a CSV file in two different apps and have it interpret the data in the file in two different ways.
Significance levels are always arbitrary and are determined differently by convention or tradition in each field.  Given your value of p<0.0001 with an N of only 35, your effect seems intuitively unlikely to be due to chance (given a vague assumption that variability within individuals is generally speaking fairly low), so you should calculate effect size to see if the effect is big enough to care about.
I have made the question more specific here: http://stats.stackexchange.com/questions/57214/how-to-analyze-data-with-more-than-one-associated-categorical-dependent-variable
yes: http://en.wikipedia.org/wiki/Multilevel_model
$S(t)$ is the survival function
The variance of a _Bernoulli_ random variable with parameter $p$ is $p(1-p)$ and the binomial random variable, being the sum of $N$ independent Bernoulli random variables, has variance $Np(1-p)$, which is the sum of the $N$ variances. With regard to _why_ $p(1-p)$, consider variance as the moment of inertia about the center of mass of masses $p$ and $1-p$ at $1$ and $0$ respectively.
Are you using Royston's **ice** package do do the multiple imputation, or the in-built multiple imputation stuff that's been available in Stata since around version 11 or so?
What are your thoughts on Weka vs. Rapid miner or even Rattle? I have yet to really get involved with any of them, since I have been doing R for 7 years and just in the last few years found out that there are these other great tools (turns out my undergrad and grad schools just liked using R a lot).
Ah, I see.  There's some vagueness here.
@funkywon 1. I suggest to edit your question, not to add an answer 2. Just copy the xml code into the editor, select it and click on the code - symbol ({}). If this does not, what is the actual error ? Where does the "xml-insertion" fail ?
I agree with your interpretation $\\mbox{modulo}$ a square root.
It also looks like chapters 8, 10 and 11 in Bivand's "Applied Spatial Data Analysis in R" book gives examples. The scope of that book is really amazing every time I go back to it, certainly worth having.
My general tack when I ask myself this kind of question, Is to do some rapid numerical simulation in R. Make a habit of it, it will teach you a lot!
Thanks, I agree. I have tried to just focus on looking at the between group differences regarding slope (interaction between time and condition) and the estimated mean difference for each group (which I get asking for emeans table comparing) with confidence intervals. I am using multilevel mixed model analyses. My thoughts are that I can still describe the slope, however can't draw conclusions regarding between groups differences at follow-up. I think that Mixed Model deals with violations of homogeneity of regression slopes.
Can you give some context? It seems you came across someone/some article that claimed to use logistic regression for count data, and to use the log link for binary data. Can you link to that? That way we are sure we are talking about the same arguments.
Isn't $\\prod_i^n\\theta^{x_i}(1-\\theta)^{1-x_i}$ the same as $\\theta^{\\sum_i x_i}(1-\\theta)^{n-\\sum_i x_i} = \\theta^y(1-\\theta)^{n-y}$ where $y = \\sum_i x_i$ is a binomial $(n,\\theta)$ random variable?
Thanks for posting this as a question, @questionhang.
*"we select variables which are most statistically significant..."* No we don't! That's really not a good way to do model selection.
Eric, your first two questions are answered by suitable multiples of [Stirling numbers of the second kind](http://mathworld.wolfram.com/StirlingNumberoftheSecondKind.html).  The connection is made apparent by considering that each $j$, $1\\le j\\le k$, determines a subset of the rolls and any two subsets (for different $j$) are disjoint.  Given such a partition (the number of which are counted by the Stirling numbers), it remains only to assign a unique element of $\\{1,\\ldots,m\\}$ to each subset in it, which can be done in $m \\cdot (m-1) \\cdots (m-k+1)$ distinct ways.
It certainly is equally likely to get the path HHHHHHHHHH as it is to get the path HTHTHTHTHT, however it is not equally likely to get 10 Heads as it is to get 5 Heads and 5 Tails.  There are many, many more paths to 5 heads and 5 tails than there are to 10 heads or 10 tails in 10 flips.  It's not that any one singular path is any more likely than any other path, but rather that the outcome closer to the mean outcome (50 % heads and 50% tails) is far more likely than one with an extreme outcome (10 straight heads for example).
Do the answers [here](http://stats.stackexchange.com/questions/222/what-are-principal-component-scores), [here](http://stats.stackexchange.com/questions/66518/interpreting-plot-of-pca-results-from-3-to-2-dimensions), or [here](http://stats.stackexchange.com/questions/62034/are-there-examples-of-more-informative-pca-plots/62716) help?
H2O is a open sourced framework from 0xdata designed to work with hadoop in scenarios that you mentioned. If you have a hadoop cluster, using H2O is a breeze and you can use R to control H2O. Spark and MLLib as mentioned previously also help.
I think this is a good question. However, could you please be more specific as to what you mean by representativity? For example, do you mean according to some preselected criteria, or as in a mirror image of the population, or are you mostly interested in how to draw a random sample without having a list of the whole population?
(+1) nice answer. On the Bayesian side, one might do many things, but in effect you're trying to build some model for how you came to have such values (the process that led to the outlier). For example, it may be something simple like "each data value has some small unknown probability of being from a distribution that's a lot wilder than the bulk of the data" and then putting a prior distribution on that probability and formalizing some choice for that wilder distribution and priors for its parameters. The effect is to down-weight the impact of points that don't fit the model.
Also, maybe you realize this, but if you click the interaction tag you'll get a lot of really good discussion threads.
Thanks a lot, I'll have to study this a bit more but it seems the Adjusted Rand index is precisely what I was looking for!
have you made any graphs or done any tests to check for normality? There are Kolomogrof-Smirnoff and Shapiro-Wilk tests in R.  qqplot in the car package tests quantiles of a variable against a distribution you can compare.  Normality might be a really poor assumption for your data (and then so might your confidence interval)
In the pwr package, also pwr.2p.test and pwr.2p2n.test.
By leap: 1) I'm not R user, so don't know. 2) There is R package NBClust with many many clustering criterions. 3) You should read on this topic carefully to be able to choose among them. 4) You do clustering be any package you like, save a range of cluster solutions (membership variables) and then apply a clustering criterion from NBClust or such. 5) Yes, you may choose clusters simply by contemplating the dendrogram; this is simple, old and not very good way. Note that you can't compare dendrograms of different aglomerative methods in order to choose the 'best' method.
@whuber after considering your comment and the answer from jbowman,  I realize that I was limited by software rather than theory.
(+1) It should be $\\Phi\\left(-\\dfrac{\\mu}{\\sigma}\\right)$ in the expression for $p$. I tried to edit this little typo but I could not because it is less than 6 characters.
if your series is non stationnary, the ACF will decline very slowly, to the point of being useless (it essentially a constant). What do you mean by 'have any meaning' ?
@mpiktas, thanks, it seems that "change-point" is the term that I was looking for.
Your formulas are inconsistent: for instance, in part (b), $F$ is not the quantile function for $f$. ($F$ assigns zero probability to negative numbers whereas $f$ is symmetric around $0$.)  This leaves the question indeterminate.  Please edit your question to fix up the mathematical problems or at least provide more information so that the question becomes unambiguous.
This is an R programming question that would almost certainly get a better response on StackOverflow.
You might want to look into Dunnett's test, which is specifically designed to control the experiment-wise error rate when several treatments are compared pair-wise to a control.
This thread may also be of interest: [difference-between-standard-error-and-standard-deviation](http://stats.stackexchange.com/questions/32318/).
I think you would need the covariance matrix to calculate the SE of the prediction with multiple predictors.
I will, thanks @gung for validating what I thought haha!
Hope you found it helpful.
As for RM, that's one of the reasons why I asked the question in the first place. As I mentioned here: http://stats.stackexchange.com/q/112640/54099 treating Subject as fixed or random doesn't change the Treatment p-value, so why bother.
Thank you for the comment. As a beginner, I randomly select the c value for SVM. May I ask do you mean tuning CV as in training CV? and the estimation CV, do you mean the predicted cross validation (accuracy)  on the selected model? My lecturers did not inform me the size of the hold set , they only gave me my accuracy on the test set when compared to hold out set.
Is this a homework problem? If so, it should have the "self-study" tag.
@JeremyMiles: Do you mean test statistics are historical anomalies in that prior to low-cost computing, users calculated Z, t and so on and then compared this test statistic to the pre-calculated tables to determine statistical significance, and as a result, many current users of inferential statistics still think in terms of test statistics when they could just as easily provide a p-value? In other words, is this a sort of generational effect?
I highly doubt simply switching the topical material will help you understand the technique any better (even if you have a greater understanding of the theoretical material). Perhaps simply asking for an accessible intro to VAR would be more enlightening?
Only conclusion (3) is consistent with your data.  The rest are not. In (1), the number of increases is an exceptionally poor test statistic and so gives you no grounds for any conclusion for or against a linear relationship. In (2), the SD is actually quite *large* compared to the size of the slope.  (That's what a small $R^2$ is telling you.)  In (4), you are being misled by a single anomalous value (the first in the series).
I think this question should be asked (moved) on http://crossvalidated.com
It is "Wilk" not "Wilks".
If you have all your data, there's no need for statistical comparison. NHANES, once standardized, *is* the population data. Your data additionally *is* your population data. Therefore, there are no statistics involved (which is distilling sample estimates to a popuation). Make meaningful comparisons of population frequencies and let it be.
The original HL $\\chi^2$ GoF test is not very powerful for it depends on categorizing the continuous predictor scale into an arbitrary number of groups; H & L proposed to consider decile, but obviously it depends on the sample size, and under some circumstances (e.g. IRT models) you often have very few people at one or both end of the scale such that cutoffs are unevenly spaced. See A comparison of goodness-of-fit tests for the logistic regression model, Stat. Med. 1997 16(9):965, http://j.mp/aV2W6I
Related threads: [Relation between confidence interval and testing statistical hypothesis for t-test](http://stats.stackexchange.com/q/18215/930), [What is the difference between confidence intervals and hypothesis testing?](http://stats.stackexchange.com/q/16312/930) (and probably many more).
Also I am aware that there are ways to select/discard the independent based on anova and the t-test, after the regression is run. I would like to understand if the general idea of increasing the correlation against Y like in my example is a sound method to increase r-squared.
Model Identification, Estimation and re-modelling via diagnostic tests for sufficiency and necessity characterizes modelling as compared to assuming structure(like a smoothing parameter)..This is easily done via a F Test comparing local variances.Tsay, R.S. (1986). "Time Series Model Specification in the Presence of Outliers," Journal of the American Statistical Society, Vol. 81, pp. 132-141. or "Outliers, Level Shifts, and Variance Changes in Time Series." Ruey S. Tsay; Journal of Forecasting, 1988, 7(1), pp. 1.
But there might not be an answer for your question related to the level of these measures which you like to report. Some processes can easily be so difficult to predict with information only available publicly. Of course if you have inside information, then I think it could be possible to improve classification... :)
What can you tell us about your population? Is it bounded?
@Alex: Did you mean "Trending is *nonstationary* by definition..."?
thanks. I have some problems with the gamma distribution when $sigma$ is close to 0.
SHA should pass uniformity for single use, otherwise this would lead to cryptographic weakness. However, the vector properties may not be uniform, e.g. if you take N words and SHA them together, that might not be uniform in N dimensional space.
Re LOESS: Perhaps my terminology is not quite right. By LOESS I'm referring to models that predict Y from X using some form of localised curve fitting. e.g., as seen in most of these graphs: http://www.google.com/images?um=1&hl=en&biw=1076&bih=569&tbs=isch:1&sa=1&q=lowess+graph&aq=f&aqi=&aql=&oq=&gs_rfai=
@nournia Okay, does your similarity function output a number? Do the numbers obey the triangle inequality?
It's not me you have to convince that it's justifiable. Do you have some way of putting a lower limit on how big it might be? Some real evidence that it's large? If you just say "it's probably really big", a journal editor or a referee are both likely to check whether the calendar says April 1. Who do you need to convince and what do you have to convince them with? How certain can you be that it *isn't* small?
Thanks Davide that explanation helped out I will try out those functions and see how it turns out!
+1 Note that division by $-4.85$ would have worked, too. :-)
Is there no way to load the data into another application and compute the log there?
Terrific answer.
@Dikran: It may well be biased for other models, but NOT for logistic regression, that's the whole point.
Depends on the row, but in general it is the sum of the squared differences between each value and its expected value (usually some mean), so you can get a measure of variation. For the row you asked, is the sum of the (4) differences between each group mean and the general mean. I recommend you to start reading a basic stats book (the ANOVA section to go straight into it) or, e.g., something like [this](http://en.wikipedia.org/wiki/Sum_of_squares_%28statistics%29). There are many pitfalls (e.g., invalid assumptions) you may want to avoid.
Yep, familiar with matrices, through linear algebra, and with concepts of vector space, basis, norms, but I don't fully understand things like LU factorization, though I am fairly clear about Gram-Schmidt orthogonalization and the unoptimized QR algorithm, though not completely clear about *why* they work. Also I don't understand how people can *derive* the eigenvectors for a random matrix without performing an eigenvector algorithm on it.
@Glen_b I have added some explanation about x and y in the question above. I hope it clarifies.
+1 This is a key idea: *use application-specific knowledge* to characterize the behavior.  It will be far more pertinent and powerful than any purely statistical technique possibly can be.  Statistics can then offer ways to *compare* the data to the "baseline" or ideal series.
I believe you have correctly identified the issue. I've got the residuals on the logit scale and the fitted values on the response scale (i.e. between 0 and 1). I've reproduced the plot with the residuals on the response scale and it looks much more believable.
Sure! Are you uncertain about the last formula or the lines before it as well. I.e., are you familiar with the binomial distribution, normal distribution and the central limit theorem?
I commented on that question earlier, but I hadn't seen the connection. This underlines that you need to make links for us!
@FrankHarrell Mahout is a software tool used in conjunction with Hadoop for machine learning and statistics.
Isn't the issue that each data point is a ratio? so they can't be normally distributed with eg mean 3, sd3.
Related: http://stats.stackexchange.com/questions/10728/definition-of-quantile
@AdamO I'm not familiar with the term *non-collapsibility*, and couldn't find it in any of my reference texts. What does it mean?
Consider these questions: (1) If someone else were to independently sample $500$ members of the control population, would they likely have observed exactly $235$ successes? (Probably not, so how much do you suppose their number of successes would deviate from your number?) (2) Where in your code have you accounted for that sampling variability?
@Jessica Though I like your measure, I kind of agree with Akavall. A finding with a Q equal to 2 in personal psychology (where it is very difficult to find high correlations) is going to be viewed as more significant than if you find that kind of correlation in a gene expression study (where it is often easy to find correlations between members of the same signaling pathway). Not saying it can't be done, but I think you might want to look into some way of comparing effects to other known ones in the field.
+1 Wice advices
Thanks. Yes, in some way, estimating the time to failure is taking a longer route to the result, but it saves the problem of deciding which failure period to estimate: I don't know if the data supports classifying a 10-day failure period better than a 5-day period, so with the classification, I have to train many classifiers and then trade off giving the best results vs. giving a timely warning. Estimating the time to failure would sidestep that, but the estimation itself might be harder.
"Now I have three levels in this factor, YvsNY. What that does is estimates a single line that still includes males in consideration of the effect of YvsNY just seperately to Females with no young." This is a sensible intuition to have about what the `YvsN` contrast is doing, but it is actually ***not true***. The effect of controlling for the orthogonal `MvsF` contrast is to remove the influence of the Male group from estimation of the `YvsN` contrast entirely. It is well-understood ANOVA theory that with an orthogonal set of codes, groups at `0` contribute nothing whatsoever to the slope.
I agree that this is the simplest method and will probably do the job. I think I would just suggest a minimum level for the c-line. If one or two cases are raising too many false positive alerts, just change the threshold to something higher.
@DavidMarx The algorithms are for determining one coordinate of a location using signal strength data. So the algorithms output this coordinate, which I compare against the known value as described. My thought was that the closer the difference of these values is to 0, the better the algorithm has worked.
A test result is partly the effect of mistakes on individual items and can approximately be modeled as a Binomial outcome. For the Binomial, the variance will be greatest for scores near 50% and will decrease to 0 at 0% and 100%. This alone could explain your hypothesis--and would indicate it might say nothing meaningful about the test or its subjects. So the first thing you ought to do is check this out.  One way is to attempt to stablilize the variance with a logit or logit-like transformation of the scores. This will automatically address the other issues you raise.
Big enough for what? If your data is a random sample of a large population and the goal is to estimate the "true" population proportions with enough precision, you can indeed look at binomial confidence intervals for each proportion and see if they are narrow enough for your purpose.
Have you gotten an answer?  I am having a very similar issue and need help.
What regression techniques require strong assumptions about the distribution of the covariates?
@PeterFlom Thanks for your question. I've edited my question to include the answer. The tasks are performed by computer, not by people. Each number represents the number of successfully-completed tasks out of 100000 attempts.
@RobertKubrick you understand what setting k to 100 *means*, right?
It really was a hypothetical example. I accidentally obscured that by stating "I'm thinking of a study..." instead of "I'm imagining a study..."
Rob, your book is great!
The one-way ANOVA from Stats 101 only tells you that at least _one_ of the three toy rating types (bad, medium, good) is different from the others, right? [Whereas I want to test that either: 1) good > medium > bad, or 2) bad < medium < good.] Is there another kind of ordered ANOVA that I should be using? (Or am I misunderstanding one-way ANOVA?)
Bonus question: what's the best way to determine if the ICC is monotonically increasing?
See this publication: Morrissey ER, Diaz-Uriarte R. Pomelo II: finding differentially expressed genes. Nucleic Acids Res. 2009. And this website: http://pomelo2.bioinfo.cnio.es/. Note that there are literally dozens of tools and methods used for this purpose and this is just one example.
Agreed (well with point #1).  In some ways, it is even better than R for the EDA stuff, and can be inserted nicely into an iterative workflow.  If one is already working without R, then JMP is a reasonable way to work without adding R.  Because it can also link to R, an R programmer can create tools that then plug in to JMP, allowing the Eloi, errr, user to continue to believe that everything is just dandy.
+1 for your comments in your answer, I think you have raised some very helpful questions.
+1 nice, I was thinking something along these lines as the answer if that were the real question.  One quick edit, you say y=intervalWidth/intervalCount, whereas I think you mean, y = intervalCount/intervalWidth.
When doing direct sampling from posterior, for a grid based approximation, the joint likelihood of all observations become extremely small. The Griddy Gibbs sampler I've mentioned is an example of this. Even with good parameter values, just the joint probability is enough to underflow R for example.