tttjjj/elt_bench huggingface dataset:
Galton derived a linear approximation to estimate a son's height from the father's height in that paper. His equation was fitted so an average height father would have an average height son, but a taller than average father would have a son that is taller than average by 2/3 the amount his father is. Same with shorter than average. This could be argued to be a simple linear regression (today's meaning). And of course today regression has an even broader meaning: it's any model that makes continuous predictions. It is interesting how much his original usage of that word has changed.
A good idea to search the site as well: e.g. see the splendid http://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models
Thanks for the answer! I am still puzzled by the meaning of x,y, and D though. Let me put a concrete example: $f_1(x)$ is an estimation of the PDF that does not change over time, it's sort of a priori information I have, and $f_2(x)$ is an estimation that employs some data of the observable world that change over time. Supposing that none of the two is a perfect estimator, that is, they can have some error, or they can be quite uniform, how can I combine them? $f=(f_1 + f_2)/2$ ?
If $X_2$ "had no relevant effect" then it ought not be in the model at all. $X_2$ does have a relevant effect if it is changing the influence of $X_1$. I'm really not sure why you want to do this. Even if $\\beta_2$ is insignificant, if the interaction is significant it is standard practice to leave $\\beta_2$ in. It only makes sense to me to drop $\\beta_2$ if you have *subject-matter* knowledge that $X_2$ should not influence $g(\\mu)$ at the value $X_1 = 0$, or maybe that $X_2$ only influences the outcome by magnifying/shrinking the effect of $X_1$, but even then I think you only stand to lose.
No go with SAS. `GLMSELECT` (in which LASSO is implemented) only works with linear models.
I have not used SCM, but there's another recent paper I read which does:  Trandafir, M. (2014). The effect of same-sex marriage laws on different-sex marriage: Evidence from the Netherlands. *Demography*, 51(1):317–340.
I will read (study?) the rest, but I have a problem with the construction of the first equation:  $P(k|\\theta,n)=\\left(n\\atop k\\right)\\theta^k (1-\\theta)^{n-k}$ implies that you think each one of the $n$ runs results in a success (with probability $\\theta$) or a failure, am I right? But in reality each run of this experiment/simulation results in a success rate between 0 and 1. Because, in each run the algorithm is faced with, say, 1000 objects of the test set,and if it successfully recognizes, say, 800 of them, it gets a success-rate of 0.80. Also, probably, $k$ will be much larger than $n$.
You first approach needs to be rewritten: it could be "take the top 5% of values and find the minimum of them", in this case 79586, or "take the bottom 95% and find the maximum of them", in this case 79535.
So to confirm, you are asking about how to compare model fit when the models compared have heteroscedastic errors?
@saeed one further note: If you want to know, how to implement the EWMA with R, have a look at my answer to this question: http://quant.stackexchange.com/questions/7988/compute-a-time-series-of-daily-volatilities-in-r/7990#7990 There, I give one possibility to implement the EWMA with using R.
I don't think that is an equality. I think that is an asymptotic equivalence. It definitely doesn't hold for $n = 1$.
But note that this is all based on sampling from a homogenous population.  If you have a heterogeneous population (e.g. different proportions for different sub-groups, sampling rare parts of the populations), then that variance estimate not so reliable.  The estimates you are actually calculating here is (I think) for a population which your sample represents.  The question is: is this population the one you are actually interested in?
Yes since you are off by a factor of p just multiple the incorrect answer that you calculated by p.
I *think* I can agree with that, provided we understand "y-axes" to mean "dependent variables."  This phenomenon is akin to the difference between an unpaired and paired t-test: the test in your question is a legitimate one, analogous to an unpaired t-test, but it lacks power to detect true differences in slope when the variables are (conditionally) positively correlated. Testing the slope of the regressed *difference* is the analog of performing a t-test of a difference of paired variables to see whether it is zero.
I'm sorry to deceive you but unequal subsample sizes is not an issue. Moreover yours looks like a large sample to me. so both procedures should yield similar results. I noticed that @probabilityislogic made a good point. Using one pooled sample  implies equal residual variances, so that may be a source of heterogeneity. Don't know how you are implementing the separate regression procedure, but it's easy to make mistakes if you are calculating the stat yourself. This makes the pooled regression a safe straightforward approach.
Thank you for the explanation. When taking about 'retraining' I mean training the final model a few times. As ANN often find local minima it would likely be beneficial to retrain it. It seems from your answer that the final model is only trained once. Consequently, I will try to train it more times using the optimal settings or perhaps use the "avNNet" :)
+1 In R option 1 would be a straightforward application of `glmer` in the `lme4` package.
Yes, I'm aware that it is surprisingly difficult to create a truly out-of-sample dataset as it is easy to accidentally end up with temporal correlations and what not.
@vinux, usually if some term is in the equation it is assumed that it is not zero.
The combinatorial approach is perfectly fine if one uses the property of Poisson processes that _given_ that the number of arrivals in $(0, T]$ is $n$, then the conditional distribution of the $n$ arrival times is the same as that of the order statistics of $n$ $U(0,T]$ random variables. Thus, the probability that exactly $k$ of the arrivals are in $(0, \\hat{T}]$ is just the binomial probability $\\binom{n}{k}p^k(1-p)^{n-k}$ where $p = \\hat{T}/T$ which has value $0.5$ in this instance.
Is there an APA style for path diagrams? I've never found anything that will automatically do a decent path diagram - I always do them in powerpoint.
Thank you so much, this would really help a lot.
I would argue that in presence of outliers (points with unusual response values, as opposed to leverage points which have unusual explanatory variables), CART would totally screw up the cell into which the outlier(s) would fall, but other cells will probably be OK. CART is a linear regression with dummy variable predictors generated on the spot, so it may inherit some of the problems that linear regression has, although in a different form. In a case when the response does increase approx linearly with the regressors (e.g., log income vs education), CART will miss it coming out worse than OLS.
Hint: Consider what the effect of applying the finite population correction is, & why that might be advantageous. (Also note that (1) doing sums is almost always less trouble & expense than collecting data; (2) if you can distinguish individuals you shouldn't "recycle" measurements, but base inference only on the distinct individuals.)
Yes. Conditioning on the covariates, the remaining among-individual variability is consistent with a pure Bernoulli process with zero among-subject variance.
Are all $2m$ observations jointly independent? The $n_i$ are presumed known and nonrandom?
As I understand it, the Box-Muller method _also_ uses uniform random number generators (in fact, two successive outputs from a PRNG) to create two independent samples from a (standard) normal distribution.
It depends on the model and on what these events are. Could you edit this question to include some more information about that?
I don't understand the $\\log$ transformation. How can you take $\\log Y$ when $Y$ is negative?
Hi, Daniel. Unfortunately, I don't have it at hand at the moment, but you might look in S. Asmussen, *Applied Probability and Queues*, 2nd ed., Springer Applications in Mathematics. If I recall, Chapter X discusses (steady state) properties of GI/G/1 and I believe Chapter XII has a brief section on GI/G/m. In both places, some results on $\\lambda =\\mu$ are provided ($\\rho = 1$ in the book's notation). Sorry I can't provide a more precise pointer at the moment.
do you mean 'prizes' where you say 'prices'?
Can you get any ofthe relevant info here? If not, it might be better as a comment.
Hens were selected at random.
Welcome to the site, Cam. I sure hope to see you here as often as I see you on SEMNET, multilevel and statalist :)
You can do much better using a $\\chi^2$ test on these data (which form a contingency table).  `ks.test` is not appropriate, primarily because these are discrete values.
Can you contrive a case where, after the ordinary least squares fit, you have three equally spaced residuals (-1,0,1) in that order?  If you think through why this is impossible it might help.
Well, I read the whole page again but still cannot find/see the answer to my question? With "For your second question" do you refer to the question of applying hampel's technique to small sample sizes. I am asking as I do not want to assign your answer to the wrong question.
@Glen_b Well you don't need to be a speech researcher to know the difference between _in question_ and _in the question_... Hehe, but jokes aside, I will clarify my post. Thanks.
(BTW: I was using the formula as you defined it, I mixed A and B when writing the question. Good you notice!)
http://en.wikipedia.org/wiki/Indicator_function
@whuber: $T$ as the composition will still be a mapping from $\\mathbb R^n$ to $\\mathbb R$, isn't it? The dimension $n$ of its domain still depends on the sample size.
Assuming nonnegative, can't you just invert the observations and apply the KM estimator to that? It is my guess is that this is equivalent to deriving and using the KM estimator for left censored data, but I'm not sure.
Note that cosine for binary data is [Ochiai measure](http://publib.boulder.ibm.com/infocenter/spssstat/v20r0m0/index.jsp?topic=%2Fcom.ibm.spss.statistics.help%2Fsyn_proximities_measures_binary_data.htm). You could compute any of such "binary" measure using these [guidelines](http://stats.stackexchange.com/a/49477/3277) if RAM memory allows.
@ttnphns If OP edits to ask such a question... I'll see if I can add to it.
I am afraid I cannot help you with R-related questions. But I guess it should be easy to find out.
In what respect should the "couple of PCs" be the best?
(+1): A great idea !
Could you explain how a theory about *time series* would be applied to a *singular event* like an accident?  I can imagine some creative ways, but am interested in what specifically you have in mind, so that we can know more precisely what you're asking here.
It was supposed to be −2log(a), edited it.
@DrWho Is this intended as a replacement for your previous question at http://stats.stackexchange.com/q/8187/919 ?
Thanks, works now! However, I am having some difficulties to understand the plot: is it really $r_{xz}$ on both axes?
*A rose by any other name would smell as sweet*. I would say it is fine as long as it is clear what is intended.
I apologize for prying. Since your scores are natural numbers from $0$ to $n-1$, & your ranks are also, & are an ordering of the scores, they will be exactly the same values unless there is a tie. If the ranks also had ties, they would always be identical values (perfect correlation). Since the scores can have ties, but the ranks won't, the correlation won't be perfect for any draw that includes ties.
You are correct, although some of the parameters are bounded which I faked up by artificially imposing large increases in the objective function value as a function of how far out of bound a value was. Very inefficient, but it work(ed)(s). I'll take a look at NOMAD, thanks.
Thank you for this answer - it directly address the what I asked and adds to my understanding.
Thanks also for the reference, obviously I wasn't using the right search terms, i.e., TOST, etc., versus distributions. This is also very useful and adds to my knowledge.
This is generally a bad idea. Try reading the linked thread for more information. If there is something more you need to know or still don't quite understand after reading through that, come back here and edit your Q to explain what you learned & what you are still confused about; we can re-open this to address the needed issues.
"A brief history of coordinate descent for the lasso" on pp15-17 of the UseR! talk is a gem :-)
I think a reasonable estimate would be to let the expression in the example above be p1 (p1 = choose(L-N+1,1)...).  Then the probability would be p2 = choose(M-N+1,1)*p1^1*(1-p1)^(M-N).  Intuitively, I would expect the answer to be roughly (M-N+1)*p1 if N is large.
Perhaps, but you might be losing a lot in that description. If you treat $a$ and $b$ as "known" in the future modeling, you will fail to account for their uncertainties due to estimation. Assuming you account for that correctly, your new models have *two* parameters: namely, the two Box-Cox parameters that will be estimated.
Given that, I've added the `self-study` tag. Please check the [tag wiki info](http://stats.stackexchange.com/tags/self-study/info). What do you understand about the question? What have you tried/what are your thoughts?
Thanks Davide..
@StasK Thank you! I have problems inserting the graphs in the comment field. However a small sample of the data is provided below.                                                     start	end	a	b	c	d	e
05-dec-11	04-dec-12	1	0	0	0	0
05-apr-11	31-mar-13	1	0	0	1	0
14-dec-11	13-dec-12	1	1	0	0	1
16-jan-12	31-mar-13	1	0	1	0	1
19-apr-11	19oct2011	0	1	0	0	0
25-apr-12	31-mar-13	0	1	1	0	0
05-jan-12	06-jul-12	0	0	0	0	1
22-sep-11	23-mar-12	1	0	1	0	0
13-mar-11	12-sep-11	1	1	0	0	0
27-jun-12	31-mar-13	0	1	1	0	0
Will this help yu to answer my question? Thank you so much for your help!
Are you sure about dividing by the square root of n? I don't think that will be helpful here. What he wants is a SD to use (with the log(OR) as the mean) to simulate odd's ratios. I think the top part of your equation (without the division by sqrt[n]) answers that.
@Elvis: no, what I mean is the table seems to lure people to misinterpret the p-value as a support that more attendees answered post question correctly after attending the lesson
The term *multivariate analyses* is used for situations where there is more than one response variable. CV has a tag for this, [tag:multivariate-analysis], that you can search on. If your situation is complicated enough, you would probably use *structural equation modeling*, which is a type of multivariate analysis. What does it mean that your response variables "have interaction effects", though?
+1 Well done!  I'm shocked this hasn't been mentioned yet.
That is simply not the case @Roland.  Estimates obtained from cutoffs are only simple because people do not understand what the estimates estimate.  That is because they do not estimate a scientific quantity, i.e., a quantity that has meaning outside the sample or experiment.  For example the high:low odds ratio or mean difference will increase if you add patients with ultra-high or ultra-low values to the dataset.  Also, the use of cutoffs implies that biology is discontinuous, which is not the case.
The answer given in the paper is: $\\frac{n}{(n-1)^2} d_i^2(\\bar X, S) \\sim B(\\frac{p}{2}, \\frac{n-p-1}{2} )$. Thanks!
+1 For answering this without the negativity that I would use. :)  In typical parlance, controlling for other variables means the authors threw them into the regression.  It doesn't really mean what they think it means if they have not validated that the variables are relatively independent and that the entire model structure (usually some kind of GLM) is well-founded.  In short, my view is that whenever someone uses this phrase, it means they have very little clue about statistics, and one should re-calculate the results using the stratification method you offered.
This is showing a lower limit for $v_1$, not anything about the difference $v_1-v_2$
Hi B_Miner (and others), thank you for the continued discussion.  My setup is a pretty typical discrete-time survival analysis, with the exception that the natural periods in the data have varying time lengths.  You may not be convinced from my description that this issue could affect the modeling results, but Allison has devoted two pages of his book to discuss the issue (including a setup under which the varying length intervals would NOT affect the outcome), so at least he recognizes the relevance.  Sadly, his discussion does not apply to my data, for reasons I tried to make clear above.
http://en.wikipedia.org/wiki/Limiting_density_of_discrete_points is important to understand that relative entropy is not necessarily Kullback Leibler
At the related page http://www.ats.ucla.edu/stat/stata/faq/oratio.htm (about 2/3 the way down) the SPSS output includes a CI for the constant by default. This is because no odds ratio calculations were requested. I'm not surprised that CIs are not provided for an odds ratio for a constant: after all, what would that even mean? Thus it would behoove us all to distinguish carefully between output in terms of the raw parameter estimates and output in terms of odds ratios.
It would make sense if it said "... if the squared residuals ..."  That is an easy mistake to make in writing up a test.
+1: Thanks for the tip.
Script errors are like an avalanche. You should start by trying to see why your import from SPSS is not going smoothly. Always take care of the first error before moving on.
Technically, in OLS you can also test H$_{0}\\text{: }r=0$ (where $r=\\sqrt{R^{2}}$, or, if your prefer to be pedantic $r = \\text{sign}(\\beta)\\sqrt{R^{2}}$), and you can also go wild with a test of H$_{0}\\text{: }\\alpha=0$. Just sayin'...
John Fox has companion R code to his book on regression in which he uses time-varying covariates and an interaction with time in a Cox model. The code is a little dense and requires some unpacking but here's a link to the resource:
http://cran.r-project.org/doc/contrib/Fox-Companion/cox-regression.txt
@Braj-Stat, one thing to note is that the "requirement" (such as it is) for the chi-squared test is that *expected values* are >5 in all cells, *not* raw counts, although you may still violate that rule of thumb, &/or want to run a different test anyway.
Warning: optimising hyperparameters to have best accuracy on validation set is a straight way to overfitted oblivion. You want nested CV.
Thanks for walking through an example... the 3-way example really helps clarify things.
See http://en.wikipedia.org/wiki/Bias_of_an_estimator#Sample_variance for an explanation why the variance $1/n \\sum_{i} (x_{i} - \\bar{x})^2$ is a biased estimator, and http://vdov.net/~acosta/content/mle-normal/ for an explanation why it is the maximum-likelihood estimator for normal variables.
Thank you. I am not really interested in the communities but rather in the tendency of the signed network to be polarized/fragmented into communities. But as far as I can see, the modularity can be retrieved from the resulting ``communities`` object using the ``modularity`` function. I will definitely take a look at the Traag and Bruggeman article. Since the implementation seems to be based on simulated annealing: how well does it perform? Can I actually make sure that the algorithm really returns the _optimal_ modularity (since I want to measure polarization/fragmentation)?
Thank you Macro. However the two threads you posted above don't address the concern about prediction. Am I missing anything there? Thank you again!
Yes, but if there are good practice datasets out there, then you may as well go about forming your arbitrary hypothesis in a way that's relevant to them. I mean, there's not much worth in forming a hypothesis about the length of unicorn horns, is there? Doesn't mean you have to look at the data first.
Have you looked around this site and read the similar threads, such as http://stats.stackexchange.com/questions/7057/glmnet-or-lars-for-computing-lasso-solutions?
This is a HW and I already had the tag for it! I looked at holtwinters R code but the code is really giant and I know I don't want all of its functionalities for this question. I just need more hints where to get started. I will take a look at it. Whatever I search in Google ends up using holtwinters function so it's really hard to find help.
While model selection typically involves the scoring of models within a family of distributions, based on their fit and penalizing the number of parameters used (a la AIC and BIC), whereas method selection is more general.  Method selection involves being faced with a problem (e.g. test, classify, predict) for which we have some background knowledge (variables are known to be (e.g. independence, data type), and for which auxiliary assumptions are made (e.g. normality, homoscedasticity), and we must select a method.
Thanks! (1) So dimension means the dimensions of the vector space $\\Theta_0$ and of $\\Theta$? It doesn't mean the number of components in the vector $\\theta$? (2) Why the above condition and continuity of the likelihood function in $\\theta$ imply $L(x)= \\lambda(X)$?
following this question, I did a hunt: there is a Windows version http://projects.gnome.org/gnumeric/downloads.shtml
You have some R code example ?
Generally, a regression package will have an option to print out the estimated covariance matrix of the parameter estimates, which is all you need.  As for failure of the delta method; if the first order Taylor expansion is a poor approximation to the function, and if the higher-order moments of the parameter estimates aren't small enough to effectively cancel out the higher-order terms in the Taylor expansion which make the first-order approximation poor, then the delta method will likely give poor results too.
This question is not quite clear. How do you sample every 100th person *exactly* without implicitly counting *all* of them? Since you apparently aren't doing that, what *is* the sampling method you are using? Without describing how the sampling is actually done, it's hard to provide much guidance.
How can you *estimate* the criterion? One physician's criteria for diagnosing e.g. chronic fatigue syndrome may well differ from another's, but I can't see it's the sort of thing you can *estimate*.
It helps to give us a data frame for your problem, otherwise you'll have a lot of trouble applying the response.
Thanks for the reference Jorge!
Yes, the p-value is definitely needed. However if I do the analysis directly, for example by doing a t-test on the effect sizes in the two groups, I also would get a p-value. Is this p-value somehow flawed, so that I should avoid using it? The other nagging feeling I have is, that a real plot for the interaction model would need four data points, to fully fit the concept of the test. However I want to reduce this to two data points only. However this is more of an asthetic than a scientific problem.
You write that you have *segments*, but in the graph you actually seem to have a *piecewise linear trajectory* (i.e., segments are all joined end-to-end)? Is that so? If yes, you may want to edit this into your question, since it may influence possible answers (which I unfortunately have no idea about).
Is there a reason for using inv from fBasics when it's just a synonym for solve?  Wouldn't it be better for the answer to not require a dependency on external packages if it is unnecessary?
It seems that with "ordering", Enders refers to the ordering of the Choleski decomposition, meaning that whether $b_{12}$ or $b_{21}$ are set equal to zero to restrict the VAR model and consequently identify the underlying model in structural for. Maybe someone can confirm/clarify this for me?
I don't know exactly what you mean by "first over second" but there isn't anything wrong with your dependent variable, I just wasn't sure why you had picked it.