tttjjj/elt_bench huggingface dataset:
A generalized linear model with a non-gaussian response - which one depends on the data - would be a third option.
@chl:  I am trying to do programming in R for haplotype frequencies, for number of segregating sites, etc. 
          SNPs
Ind1   0 0 0 0 1
Ind2   1 0 1 0 0
Ind3   0 1 0 1 0
Ind4   1 1 1 0 0

In above data, each column-wise is SNP and row-wise is individual(Ind1, Ind2,...). As I said before it was simulated on whole genome. The position is normalize random variable from uniform distribution [0,1]. 
Thanks for your help!
Thank you for your answer =)
But I should also mention that your responses should be transformed to look roughly normally distributed in PCA, so you wouldn't want to discretize any continuous variables in general. Anyway, the takehome is to plot your data every which way and see what you find and whether it is interesting - it sounds like this isn't the phase of your research where you need to start plowing through models and tests, and your dataset is small enough that you can basically do the exploration by hand.
As a social scientist, I would simply categorize the data into "having savings and no savings" - and then compare group membership before and afterwards.
Reasoning: If the very small N in the "having savings" category is accurate I would doubt that you can obtain a meaningful measurement of the average savings.
Can you somehow identify the trials in which you expect the results to be the same (vs. being different)? E.g., if you expect that the difference is only due to the first $K$ trials, and then the subjects get trained enough in the conditions of the experiment that the difference wears out, you can devise some sort of cusum test to check whether the series indeed converge to the common mean, or look for a structural break in the individual time series. You would have to correct for multiple testing though.
Time series methods & especially spacial autocorrelations are quite advanced. Missing data makes this much more complicated. If you don't have a strong background, I would recommend you work with a statistical consultant.
One more thing. In the R code where you define the function f, the sapply wrapper seems not necessary since later on when you use the f function, you only feed it with scalar rather than a vector. Correct me if I'm wrong. Thanks.
Can you briefly summarize in your answer the content contained in the papers in case the link goes dead?
Thanks for the elaboration.  I am trying to get this!  Two questions:  What is u(a) in the last equation?  and, are you sure your equation for p2(m) is correct?  It is different (and comes up with a different answer) from all the other minimum equations I've seen.  BTW -- I really appreciate your help!
(+1) The general nature of this answer is a reasonable match to the vagueness of the question; it might not be possible to suggest anything more specific unless the question is clarified.
To Peter's good comments, I'll add that it's hard to imagine any audience/consumer of this research who is going to give it any credibility with these sample sizes.  Can you say something about the context and purpose of the research?
+1, nicely done!
@JakeWestfall 
YvsNY        12.59318    4.54336   2.772 0.005575 **
I'm not sure if this is what you are referring to, but note that the relationship b/t $x$ & the probability of 'success' is *always* curvilinear--it's a sigmoid curve.
I don't think SEM can help as number of DVs can be large and we really don't have a sound hypothesis to model the structure of the dependencies. Also, in this limited setup we don't really care about that. The question is really focused on the inference, type I error rate and such.
It's a small world!  Well, at least I corrected the spelling by copying from your comment...
No, I'm looking at the distribution of the **number of sub-segments** covering a specific position. For example, if we focus on position 1 in the large segment, in the first simulation we have 4 sub-segments covering it; in the second simulation we have 1 sub-segment covering it, etc.

I don't care what are the lengths of the sub-segments covering each position, just how many sub-segments there are.

You are not completely wrong, this has started as a part of an exercise I've been working on in a biology course, but has gone to another domain :)
Can you clarify what you mean by 

typically required for repeated-measures ANOVA that there is only one observation per condition per subject

?

I may be misunderstanding what you mean, but, I would think that the main point of mixed models is that you CAN have multiple measures per condition per subject
+1 Thank you for your suggestions. Would you mind providing me with an example on how to do it in R? I am asking this because I am positive this calculation depends on the distribution of the data. For normally distributed data, it probably makes sense to use the standard formula given on the Wiki page. In other cases, I am guessing I will have to first test for normality of my data and then proceed with this? How do I proceed if my data is not normally distributed? Does R have anything to handle this?
I don't think I've seen the phrase used in this context. Who says it, where, and in what context is it used?
@suncoolsu R code is available on the first link given in @Eduardo's response. He he, it's `grid`, not `lattice` :)
I don't know. But one could look up the record.  According to answers.com (from a quick google) it was 23.12 pounds, born to two parents with gigantism. If I was doing the research, I'd check further.
+1. I was asking myself "But what does it mean to be correct...?". I agree this seems to be more of a logic puzzle than a probability question (although I'd like to hear an explanation from someone about why this perception is wrong).
Please also *read* the `self-study` [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) and change your question to address the points raised there.
Inspired by this answer I made an animation of the t distribution as a mixture of normal distributions. It is available here: http://sumsar.net/blog/2013/12/t-as-a-mixture-of-normals/
may have very little power. If the normality assumption holds, the t-test will still be more powerful than the signed rank test, for one. I'd imagine this is also true for various other non-parametric tests. So, it still may be best to use the t-test.
I know ANOVA to test the differences in the means between groups, but never seen to test independence, it is not the same no? Plus some of the assumptions  for anova are not met (normality for example). Could you point me to an example?
It sounds like you should start with playing around with approaches to displaying the data before launching into a significance test. I can imagine that the effect of being close to a marginal electorate might be nearly as important as being within the marginal electorate, given that patients can cross electoral boundaries. See if you can use a spatial display of the data. Sometimes a good display of the data is better than a significance test.
I fear you may be using "significant" in a non-standard way. Let's focus on your actual needs: what are you trying to find out? Whether one laptop is faster than another? If so, you will need to specify what mix of tasks is relevant for the comparison. Statistical testing cannot make that decision for you.
No it is a generic question regarding arma models, not about a specific fit
Oops. That was more of an answer than I was intending.  Perhaps @James-Stanley wants to incorporate it into his.
Besides (presumably) being piecewise monotonic increasing, what properties do you want the result ($O$), or the function relating the two to have?
Since you specified codes of -1 and +1, we would not expect the coefficient to represent the difference between the group means. You have placed the groups 2 units apart on the contrast, so we should expect the coefficient to represent *half* of the group mean difference. And in this case, half of the group mean difference when Time = 0, considering the interaction.
Did you notice that $0.66^2 = 0.44$?  As far as your last remark goes, there's nothing strange about that because (presumably) both regressions involved an unnamed dependent variable I will call $Y$: the $0.68$ and $0.10$ values describe *only* how $A$ and $B$ are related to $Y$, not to each other. You can find [plenty of explanations of such phenomena](http://stats.stackexchange.com/search?tab=votes&q=regression%20significant) on our site.  [This thread on correlation](http://stats.stackexchange.com/questions/5747) also looks relevant.
Andy, I corrected the link. Should work now.
Discretizing them compared to doing what else? If the alternative's considered to be treating the relation between predictor & response as linear then it's not surprising that discretization can sometimes give a better fit. See [here](http://stats.stackexchange.com/questions/68834/).
ofcourse ofcourse... thanks for reminding
@Aniko Each study has the information at the overall level, so for each study there are up to four data.
Correlation in the context of 2 random variables to be independent. Do we say correlation coefficient = 0 or correlation = 0. In other case, in cryptography, when we want to check how good the decryption is, then we measure the correlation. When the decryption is perfect then correlation is high between the original plaintext and the decrypted recovered message. So, what is a correlation coefficient and only correlation? Or are they the same?
No, you need a metric that reveals the truth, which isn't necessrily the same thing.
+1, there's really a lot of good info here. One note, I'm having trouble parsing parts of your 3rd bullet point, some editing may be helpful.
It's a book, so you will have to buy or lend it. It is definitely worth the investment if you are serious about missing data.
Thanks! My key question is: Is VIF and CI appropriate for testing the reliability of a formative measurement model?
@Peter does Stata have a non-linear mixed model?
Thanks for answering. My data is extremely skewed and it is a concern, as you say, for t-tests. What to do in this case?
See [truncated normal distribution](http://en.wikipedia.org/wiki/Truncated_normal_distribution).
you mean glm(C ~ F1 + F2 + F3, family=binomial, data=dat), lm() is only for when you assume normal errors
@user333 yes. What about $f_{k,c}(x)=\\frac{1}{1+exp(-k*x+c)}$ ? More generalized:  $f(x)=\\frac{1}{1+exp(g(x))}$ where g(x) is any function you like.
@jbowman The numbers represent the number of times a particular response type is observed, so they are cardinal.
@Scott, could you be more explicit about exactly what the problem is? To me the cited paper is quite explicit in Section 3 on the algorithm for logistic regression. Maybe you just missed the point that it is *not* a simple coordinate descent update of the penalized minus-log-likelihood, but instead an outer iteration where $\\ell$ is replaced by a quadratic approximation, and an inner iteration using the updates in formula (10). Does this help, or is there something else in Section 3 that is problematic?
There is a nice paper by [Cumming & Maillardet (2006)](http://www.latrobe.edu.au/psy/cumming/docs/Cumming%20Maillardet%20PM%202006.pdf) showing that not 95% of replication means will fall into the original CI, but only 83.4% (they call this value 'capture percentage'). The reason is that there are two sources of variability: A) the variability of the original mean around `mu`, and, B) the variability of replication means around `mu`. Most people forget A: the original CI is not necessarliy constructed around `mu`!
It is no problem if they don't measure the same thing. If joint tests only worked on variables that measured the same thing, there would be little motivation to develop them (not a snarky comment - just trying to help the intuition a bit).
It is quite improper to run an experiment and decide afterwards what design should be used...well what you have above might look like a 2X3 design but it's not...sorry I don't know how the data collection and experiment was run but maybe you can use One way ANOVA. Please give us more details for a better answer (what means 85, 82, ...figures above)
When you say "I would describe the mean of this variable as being very precise, in fact precise to 7 decimal places", do you refer to the fact that `R` has reported the mean with 7 digits?
Its the match by match outcomes that I need to investigate though, so I definitely need a way to determine the winner in my model. I am trying to create the control, random model to compare against historical outcomes. Currently I have the matches being decided by a coin toss, but I think I need a way to account for stronger vs weaker matchups, but without delving into trying to create an accurate predictor model using all the different player stats etc.
I'll let someone else tell you the best way to get SAS to do stuff, but you should note that you need to use a [mixed-effects model](http://en.wikipedia.org/wiki/Mixed_model) to deal with your longitudinal data.
This question is better suited for crossvalidated.com
The wordings are arbitrary except to note that 50% is the median, not the average (mean). All the information is in the percentile ranks. How to interpret them is always tricky. For example, if a lot of children are very close in marks then an enormous difference  in percentile rank is possible for a very small real difference in marks.
"235 companies have revenues of 200k and above" But I would like to know am I allowed to assume that all of them has revenue exactly 200000k. If not, it makes the computation of deciles harder.
+1 I love the ["Best Pie Chart Ever"](http://www.datavis.ca/gallery/images/fox-news-piechart.jpg)!
Actually, it's a very commonly *mis*understood thing. Lots of people are confused by the dummy in summation or integration in this fashion.
Thanks for the nice contribution.  If I don't put this inside `aregImpute` I can at least put this as an example in the help file.  Nice work.  It is important to check the algorithm to see if the final inference depends on the order of variables, when more than one variable is missing on the same subject.  Another way to get variable names from a simple formula is `all.vars(formula)`.
Scipy has much more than BLAS. In terms of Fortran packs, it binds optimization packs such as minpack, fitpack, and other linear algebra packs. It also has signal or image processing tools, optimization. I personally find that SciPy is much more versatile than R. It addition, you seem to be comparing the complete set of R packages to the single Python package 'scipy'. To be fair, you should take in account the huge variety of Python packages, a lot of them useful for science. Now I will easily acknowledge that for the specific task of statistical data processing, R is much better equipped.
One point that the link does not mention is the representation of continuous numbers using discrete ones.  there is denser representation around 1.  http://en.wikipedia.org/wiki/IEEE_floating_point
Nice. After that I'll just have to inetgrate it to get the cdf. Thanks! Im not really familiar with these pgfs, instead im familiar with their mgfs. Some textbooks don't discuss pgf.
Have you looked at `?train`?
+1, especially for recommending *Elements of Statistical Learning*. And even so, it will be hard to learn anything from only eight observations...
@whuber: [Here's the one with the most votes](http://math.stackexchange.com/questions/76491/multiple-choice-question-about-the-probability-of-a-random-answer-to-itself-bein). There were others, but maybe they got closed/merged.
You are conflating the fallacy of *accepting the null hypothesis of no difference* and finding *evidence that two quantities are equivalent*.
The proportion classified correctly is mainly noise - the lowest precision accuracy score you could possibly have.  I would avoid it at all costs.  Look at the R `rms` package `lrm` function for a variety of useful, higher-precision indexes.
It is highly unlikely Fisher would have said this. Tukey said many things that might sound like it, but I can find no direct quotation. A *lot* of people have said precisely this phrase (without any attribution at all), as a Google search will attest. At least one person claims this quotation appears in Tukey, citing his EDA book (http://www.statisticsviews.com/details/feature/5393251/Big-Data--Business-An-on-going-revolution.html). I doubt he's right, for here's one EDA quote: "NO BODY OF DATA TELLS US ALL we need to know ABOUT ITS OWN ANALYSIS" (emphasis in the original, p. 115).
You answer would be even better if you hit on some of the major reasons why linear regression suffers when the outcome is a percentage.
Well, why don't you try plotting the residuals against the fitted values to see what happens?
Note, ordinal scale is when you _don't know_ how much is next level greater than the previous one, in terms of the underlying characteristic. In your case, you know it, albeit not exactly. You may assume, for example, "less than 10 min" = 5 min; "10-30" min = 20 min, etc. This will make your scale bona fide metrical, though not nesessarily equi-interval
This works great provided $n_1$ and $n_2$ are large (several hundred or more) and $n_1 p_1$ and $n_2 p_2$ are not too small (c. $10$ or more).  Otherwise, the interval tends to be too large.  It also needs some way to treat the cases $x_2=0$ and $x_i=n_i$.  It turns out both issues can be addressed with a continuity-correction-like approach: add $1/2$ to both the $x_i$, add $1$ to both the $n_i$, and proceed.  Then this CI is surprisingly good provided both of the $p_i n_i$ are $4$ or greater, *regardless* of the sizes of the $n_i$.
Thanks for the suggestion. Log won't work in my case, because it gives too much weight for the low-count data, but sigmoid seems to be exactly what I was looking for. Is there any good rule of thumb for choosing the transition point?
@mpiktas Looks good (+1).
I don't quite understand what you mean by "go through X rounds of between Y and Z variables". Are you talking about choosing Y out of the 4e5 variables? If so, does your knowledge about the application and the data suggest that you have a number of information carrying variables between noise-only variables or do you expect your data to have the information rather spread out over the variables?
Because a p-value is computed as a proportion of simulations out of the total, it has a Binomial distribution and binomial confidence intervals apply. This suggests anyone following your recommendation would be unnecessarily expending a lot of computational effort to replace a very simple calculation.
Well, it seems that with only two alternatives telling Coke a Coke translates to telling not a Coke not a Coke, ergo a Pepsi. No? With more than two alternatives deducing the remaining flavors becomes difficult pretty fast. Still an interesting question (+1)
I can regress $r_t$ as defined in log. The issue was more related to the computation of covariances- if I compute the covariance of first differences- where do the heteroskedasticity issue emerges?
And my position is that local methods do not suffer from endogeneity, because they transform model (1) into $y^i = \\beta_0^i + \\beta_1^i x^i + \\beta_2^i z^i + \\varepsilon^i$, letting the parameters vary from one point to another, and thus do not impose any restriction on the form of $\\frac{\\partial y}{\\partial x}$. So they solve endogeneity by relaxing the very hypothesis that made it a problem in the first place.
@DiptopolDam, _did_ you read attentively a linked post in my 1st comment? In an answer there is stated clearly, and with example, that one should not blindly trust min or max of a clustering index value. Bends found on the values profile are much more important. Of course, you clearly have 4-5 clusters, and not 98, out of 100!
Your suggestion "perform a linear regression on your data, to get PageViews = a * Date + b for some constants a and b;" presumes a ton of things. For example there may be multiple trends . For example there may be a Level Shift which untreated "causes" a spurious trend coefficient . There may be outliers/anomalous points which increase the background variance causing the "b" coefficent to be biased and/or statistically insignificant. One has to be very careful in using inappropriate models.
I would put it this way: in order to meaningfully compare two histograms on the same graph, you need to normalize them both. I expanded a little bit my answer above.
It is definitely worth the time to learn R, and I have lots of handouts related to logistic modeling and the rms package.  This would be hard to do in SPSS.
Download for free Spatial Econometrics TB by Le Sage. Nice and neat output may be obtained from therein. GT.
apparantly I wasn't aware of the identity of expected value and mean value.
$R(n)$ is a multiple of a [non-central $\\chi$ distribution](http://en.wikipedia.org/wiki/Noncentral_chi_distribution) with unknown effect-size parameter and $2n$ degrees of freedom.  It is a multiple of a $\\chi$ distribution only in the very special case that $(\\mu_X,\\mu_Y)=(0,0)$; that is, when the gun has *already* been sighted in to perfect accuracy.
No, this doesn't make sense. You have 55 people, and you're (almost) saying that something is 108 times more likely. 

You don't have enough data, and have a quasi-continuous separation problem.
I am not very familiar with machine learning. For what I gathered from the vignette of GBM, yes GBM uses regression tree for classification, but regression trees are just one of classification algorithms. The terminal node formula is used to calculated the addition to regression function, during the learning process. It is used everytime after classification. I suggest to formulate separate question to get a better answer.
Thanks, @Mike, for noting Shadish's important work and elaborating on my point #5.  As you suggest, I think your 5-parameter structural model is equivalent to $\\mu$ and $\\mathbf{T}$ in my point #4 (e.g., $\\beta_1 = \\tau_{DA} / \\tau_A^2$, $\\beta_0 = \\mu_D - \\beta_1\\mu_A$); interpreting $\\beta_0$ and $\\beta_1$ may be easier.  It's interesting to consider other mean-and-covariance structures for $\\mu$ and $\\mathbf{T}$, especially with more effect-size parameters, but interpreting them in terms of real-world phenomena may be challenging.
Well the distribution need not be identical if some conditions hold. See: http://en.wikipedia.org/wiki/Central_limit_theorem#Lack_of_identical_distribution
Ah I see! Will have that in mind the next time. For now, the exercise is rather teoretic and the different effects are not defined. The target of 118 is merely that the highest value, y, defined was 118.
I skimmed the paper but from what I can see the authors are using a standard RDD design. Without going into too much detail; which variable is it that you were hoping to be cointegrated with the dependent variable?
You might be interested in the recent thread at http://stats.stackexchange.com/q/5750/919 .
Yes. It means repeated measures ANOVA. @Michael- maybe you could answer the question then of how I deal with this subject matter in interpreting my results in an answer below? Thank you!
Because the full code appears elsewhere and this does not address the request for Excel or Matlab solutions, it really ought to be posted as a comment rather than an answer.  If you were to replace the code by an *explanation* of its construction that would allow for porting to other platforms it would make a fine answer.
A simple way would be to calculate the percentage of each week's calls to arrive on Monday, on Tuesday etc., to exponentially smoothe this series for each day of the week and to reapply the result.
:Wolfgang BP tests the hypthesis that all the "information" has been extracted from the predictors. That is a far cry from thoroughly testing for non-constant variance as non-constant variance can arise from a number of sources. Please see my comment at http://stats.stackexchange.com/questions/14842/how-to-check-if-the-volatility-is-stationary