tttjjj/elt_bench huggingface dataset:
(+1) It is interesting that the "right" *intuition* (like this) seems to so often be far removed from a reasonable *implementation*. I would hope no software actually uses the basis in the OP's question.
Great answer - I don't know why people write their own backpropagation code anyway. Numerical algorithms have all kinds of complications - it's easier, and smarter, to use something already built and tested.
If the countries are exchangeable, or conditionally exchangeable you have (by de Finetti's theorem) an argument for using some random effects approach, regardless of how you sampled the data. Which random effects approach to use is a much more difficult question, however.
(+1). Thanks for pointing out factor.pa().
Hahaha, unbelievable! you are right - about the decimal places. Thanks!
The statement "R help says that the q argument is a `vector of probabilities` ..." is incorrect. The documentation under ?pnorm clearly says that the argument q is a vector of quantiles.
@B_Miner - Just keep in mind that truncation is used to focus on the main patterns in data and to filter out small scale patterns and noise. The truncated data is not smaller in terms of it's dimensions, but "cleaner". However, truncation does reduce the amount of data in that the entire matrix can be reconstructed with just a few vectors. A nice example is in the use of PCA for image compression, where a smaller number of PCs can be used to reconstruct the image. This smaller subset of vectors takes up less memory, but the reconstruction will have some loss in small scale detail.
I also tried to supply the obs as input data, already `sum`-ed in R. If I supply only one copy of this data, according to expectation, the behaviour is buggy until I do the same as above, i.e. clone the data by `obs2[i, j] <- obs[i, j]` and use `obs` in model 1 and `obs2` in model 2.
Thanks for answering!
1) I tried with and without the interaction term and the reason for choosing to include it is a better outcome (sensitivity/specificity).
You should add the homework tag.
+1 Argument 2 is interesting. Two comments on Argument 1: 1. I'd say flat priors instead of uninformative (the latter is a misnomer, if ever there was one). 2. There is no need to talk about priors in order to motivate the use of MCMC in frequentist analysis - there is nothing inherently Bayesian about this _numerical technique_!
Good points, but there some caveats here: Excel has tool tips that pop up when entering a formula in a cell and so it makes it much easier to "program" simple tasks. Also, chances are the person next to you can help if you get stuck since there are lots of Excel power users. And, Excel comes installed. Convincing someone to try R involves helping them set it up and learn how to leverage open source software (and which forums to search, which of the umpteen bootstrap packages should be used, etc). The programming is in some sense the easy part. Its the ecosystem that is the hard part.
-1. Soft-margin SVM does **not** require nor guarantee linear separation in feature space. To see this: use soft margin SVM with a linear kernel on non-separable data and you will still get a result. Soft-margin SVM penalizes points that are within the margin and misclassified *in feature space*, typically using hinge loss. @AlexTwain your interpretation is correct.
"they seem to offer a lot more flexibility"... Well, they also differ in their approach since GEEs are used to fit a marginal distribution, contrary to the conditional approach often of interest when using GLMM.
Perhaps @whuber will post a link (or a citation of a book or journal article) where the closed-form formulas for the moments can be found, or will write an answer giving the formulas themselves, with or without a detailed derivation.
I run into an error when trying your code with my data, when trying to fit the log odds: "Error in lm.fit(x, y, offset = offset, singular.ok = singular.ok, ...) : NA/NaN/Inf in foreign function call (arg 4)".
to clarify your point that the bootstrap is meant to prevent overfitting in classification, what the bootstrap is used for mostly in classification is getting "honest" (nearly unbiased) estimates of the classification error rate.  If you have accurate unbiased estimates of the error rates you will not pick a model that overfits.  Other methods might if they provide optimistically biased estimates of error rate such as in the case of the resubstitution estimate.
Shouldn't the 2nd part of your question (with a binary variable) be a separate question?
+1, this is nice, very clear & on-point. Welcome to CV, @Matt, we hope we'll see more answers like this. However, 1 minor quibble: $Y$ & $X$ being uncorrelated is not sufficient for $\\varepsilon=Y$; that would occur only if the intercept, $\\beta_0$, is also $0$, else $\\varepsilon=Y-\\mu_Y$ (of course, your point holds: they would still be perfectly correlated).
@IsishStat thanks for your suggestion but I need to do with excel because I don't know other programs :(
I did not suggest you take all comparisons, I only suggest you take a few example cities to illustrate your results.
From the summaries you provide, no valid answer can be derived. But if "having a dataset" means having access to the responses of the individuals, why not summarize it by the full response patterns? Is there any reason you cannot do that?
This works as long as the difference between any one value and the max isn't too great---when that happens, the `exp` can lose precision, leading to distributions like [1.0, 3.45e-66, 0.0, 7.54e-121]. I'd like to hold out for some answer that is robust even in that case. But for now I'm upvoting your answer.
Thank you very much, Flounderer! Digging the data a little more I found out that only 3 individuals were causing that separation. They were seen alone (or as the only recognized individual in the group) more than 1000 times each. Removing them from the analysis, it all became a single, pretty cloud of points. But your insights will surely be useful. Thanks for the links!
Can you say more about the nature of the probabilities in question?
Data is neither parametric nor nonparametric, its just data. What do you actually mean when you call data 'nonparametric'?
@cchien  That is a little harder.  To make a 95% CI for the odds ratio with 18 as the reference, you need to find the variance (and thus the std error) of $exp(f(age)-f(18))$ for various values of age.  It's a little complicated because $f$ is a random function.  But, it can be done using either the delta method or using bootstrapping.  The latter is usually easier.
@steffen , for the first question it is not that special days become less important, it is that you have observations independent of those special day effects. Hence you observe the process in different settings so you know the observed treatment effect is not due to those special days. Similar to what svadali stated about generalizing the findings, if you only run the experiment on blue days you don't know if the results would be the same on normal days.
@Ilmari: These scores depict the "quality" of a waveform sent over a transmission channel when compared with some "reference" waveform. Since the channel will vary, and the "score" would in some way depend on channel conditions, I would think that the distribution itself might vary between successive data points. Please correct me if I am wrong.
I'd love to see the counterexamples to 5. I tried to prove the opposite for a while with Zorn's Lemma but it breaks down at one point. But from what I've gathered the counterexample should be really whacky. Do you have any reference point where I could find it? I don't mind it being heavy on measure theory.
Hi @gung  I wasn't sure where to put the comment in this long thread. I added the cite here since the question posted today was closed, and this article seems useful (and covers issues I haven't seen elsewhere)
GML presumably means GLM, but still watch out: for different groups GLM mean general linear model and generalized linear model, which overlap but are certainly not identical.
Can you give us more detail on the mean wind speed formula?
If you do an initial analysis ignoring the minor variations you may get a estimate of the impact of RPM on the response.  If this is reasonably smooth and plausible, you could then adjust your models for the implied impact of the minor variations, and if necessary rinse and repeat.
@gung I will admit that I sometimes have trouble discerning between ordinal, interval and discrete data. In this case the scores are integers ranging from 0 to 24, and represent the number of problems the participant answered correctly. For reference, the test is the MRT-A mental rotations test, described here: [link](http://wexler.free.fr/library/files/peters%20(1995)%20a%20redrawn%20vandenberg%20and%20kuse%20mental%20rotations%20test.%20different%20versions%20and%20factors%20that%20affect%20performance.pdf).
To fit MLE you still need to assume a definite distribution, but you have a wider choice than just normal.  Just to pick the first book at hand that discusses the two, I have Garthwaite, Jolliffe and Jones *Statistical Inference* (a pretty standard second year uni text book) which discusses least squares as well as method of moments and method of minimum Chi square as alternatives to MLEs.
Readers here may also be interested in reading: [homogeneity testing of baseline characteristics in medical trials](http://stats.stackexchange.com/questions/58384/).
Because this post has been upvoted, SE construes its deletion as removing potentially useful information; I have therefore undeleted it.
You need more assumptions.  After all, you could regress v2 against v1 to estimate a *linear* function, which is guaranteed monotonic (and both convex and concave).
@xmjx Thanks, I've fixed it.
Note: it seems you want to choose between ROC curves (TPR as a function of FPR over the entire operating range) and PR curves (precision versus recall over the entire operating range). Terminology like "*AUC-ROC of precision and recall*" is very misleading, so I've edited this. Please revert it if I misunderstood.
The MAR assumption doesn't hold for missing data. I'm sure that the census has the same problem, but at least it should be better.
In which case, the wouldn't the frequentist be one who knows the ratio of donkey, mule and horse populations, and upon observing a pack of mules starts to calculate the p-value to know as to whether there has been a statistically significant increase in the population ratio of mules.
Could you please tell us what the "hat matrix" would be for logistic regression?
for n = 100 and rho1 = rho1 = 0.95 then the burn.in <- 30 gives me an error! Do you have any idea why??? R says that I have to give a value of 108, which is bigger than the sample size! Strange!
AFIK, your only other mainstream options are to aggregate your data for the desired comparisons, run it back through the ANOVA machinery to yield a partial eta or partial omega or to convert your t to an r.  However, my money is on all three of the above yielding the same number since k = 2.
I'm not sure I follow your code. What do you want it to accomplish?
If the computation is straightforward, why the need for a reference? One could simply include the computation.
Imo one can't answer a question like "is all of stats or ML this or that way" in a meaningful way other than by saying "No" (although the type of question pops up regularly).  It depends on the technique, not the field. GLM, trees, SVM and ANN are different. Take your question: The $E(Y|X)$ part only makes sense for some probability measure to relate the expectation to, so if the technique has that then maybe, else no. SVM for example can be constructed  completely devoid of probability.
OK, then, after Googling, if you want to edit your question to reflect any remaining confusion, I am sure you will get some helpful hints.
Due to their flaws, I would not use the proposed methods as primary analysis in a clinical trial but might consider them as sensitivity analysis to support the main findings. Depending on the study objectives, you could think of the following endpoints: A) Response after 10 weeks (Y/N), B) Change in Y after 10 weeks, C) Change in Y after 10 weeks only for the responder, D) Change in Y after 20 weeks only for responder. Those endpoints could be analysed by simple descriptive statistics, c.i.s and tests and would not be affected by the informative nature of your "drop-outs"
@Daniel let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/1154/discussion-between-manoel-galdino-and-daniel)
It doesn't sound like your data is good enough to make any predictions, then. I'd emphasize again that you need to look carefully at how values might be related based on your knowledge of the field and the specific problem, and what types of values they are. If your categories are not continuous or at least ordered values, linear regression makes no sense so it doesn't matter what R^2 you get.
Adding the model without the data or the output of what I offered is not going to provide even enough information to produce confidence intervals, .... much less the contrasts you are asking for. Given the relatively small differences in those odds ratios, it is doubtful that you will find any significance difference.
+1 The graph says it all.  When you separate the pink and blue dots enough, you can bring the correlation as close to $\\pm 1$ as you like.
@PeterFlom by enforcing I understand a way to design the experiment that would cause the inputs to be orthogonal
I am not sure however what authors means by enforcing. That was the question. If it exists such kind of design of an experiment in order to get orthogonal inputs. Solely my imagination produced only the scenario for nominal inputs, but for continuous I am not able to design such thing.
Thanks also for your thoughts on the larger question of how to know when items do or do not measure the same construct. I'll talk it over with my colleagues.
Are your sample sizes balanced? ANOVA is pretty robust to violations in the homogeneity of variance assumption if they are balanced, and not if not. I would also recommend actually looking at your groups plotted out to see if you can ascertain why the variances might be different.
Indeed, there is nothing to stop Bayesians from interpreting a probability as a long run frequency, it is a perfectly good way of expressing (some forms of) a degree of plausibility.
"ICC can become negative when the within-groups variance exceeds the between-groups variance" - http://tx.liberal.ntu.edu.tw/~purplewoo/Literature/!DataAnalysis/Reliability%20Analysis.htm
Keep in mind that the CLT performs differently well depending on the distribution at hand (or, even worse, the expected value or the variance of the distribution do not exist - then CLT is not even valid). If in doubt it is always a good idea to generate a distribution that looks similar to the one you observed and then simulate your test using this distribution a few hundred times. You will get a feeling on the quality of the approximation CLT supplies.
+1 excellent discussion of why it should be based on the survivor function. I've recommended to the original source of the question that they should have a look at your response.
If they are independent, you know their joint distribution if you know their marginal distributions. $P(X_1\\leq t, X_2\\leq s)=P(X_1 \\leq t)P(X_2 \\leq s)$.
It isn't clear what problem you're trying to solve. Is the idea that you must manufacture some number of each of the six products, and the objective is to assign each to a manufacturer? For question #1, it doesn't matter if the genes are the same length. Which genetic operation are you unsure about how to handle with variable length genes? If we have more information, we can do a better job of answering the question.
I don't think you'll find a complete answer to your question, but there's a lot of good information about different types of codings [here](http://www.ats.ucla.edu/stat/r/library/contrast_coding.htm).
I would suggest to add some more information on what you are trying to do. E.g., what are the type and number of predictors, what is the sample size, what does it mean that "results are pretty poor." Otherwise, I'm afraid this question is barely related to statistical analysis as defined in our [FAQ](http://stats.stackexchange.com/faq).
Welcome to the site. What do you mean by $Q$-variable and $R$-variable? Please clarify. Also, use of the phrase "accept null" is usually considered quite undesirable, even misleading.
I updated the prediction image by changing my S-curve. How does it look now?
For a repeated measures generalized linear model, how would you define the within subjects factor in such a situation? For example, the blood pressure measurements could be 30 days apart, 60 days apart, 90 days apart, 105 days apart, or any other number. Doesn't a within subjects factor need to be consistent between subjects?
...And how are the p-values obtained?  (Perhaps more importantly, when can this approach--which is *approximate*--be expected to work and why?)
Cool contest :) I would definitely enjoy that.
Where do you get 150% from?  The coefficient of 1.49 for B1 turns 1% into 1.49*1% = 1.49%, not 15% and certainly not 150%.
What if $y_{i+1}$ is 0 ?
I do not understand your "d" symbol.  If I was sure what it meant, then I could understand the AR part of your question and answer it.   Why can you pretend it is zero with no consequence?
@vonjd Done. Default boxplot width looks better with ggplot.
@Cam: Just an aside: Note that Borel-Cantelli does not yield anything useful in this particular case, though a very minor modification to the function would.
How more formal than you have already wrote?
You are welcome to ask questions like these, but adding details about what are the variables (not only variable names and tittle details) help. It will help others with similar questions in the future.
Great, creative idea how to interpret the sentence. I considered it more than briefly. It's not the case.
Although I have a Beta function, I don't have its inverse. So I'm stuck on the same problem again. Apparently this library I'm using has a lot of distribution functions, but none of their inverse CDF.
This question appears to be off-topic because it is about language, not stats.
What would be the purpose of (or what would you conclude from) a meta-analysis on studies involving the same outcome but different predictors or risk factors?
I doubt this is a faithful reproduction of the question. It seems likely that real question is closer to one in which the bidder can only bid *once* with a *single* dealer.
**Hint:** express $Y$ as a linear combination of $X_1$ and $X_2$ with weights depending (linearly as well) on $Z$.
+1 Good advice--and it's interesting to see such explicit connections made between modern (statistical) practice and the earliest thoughts about scientific method.
Right.  That's why the answer is 1/25, and not 1/125.  There is only a 1/125 that the three screws will match, but there are 5 ways in which the screws can match.  So either you don't consider the first screw and it's (1/5)(1/5) for the next two, or you consider the first screw and say that there is a 1/125 chance that the screws will match, and 5 different ways for the screws to match = 5/125 = 1/25.
Easily solved from using the probability tree.

It is not a Markov process because the answer for part c will equal to part a as Markhov is dependent on the i and i-1 process.

I cannot place my answer as I don't have the rep to.
I guess it is in a ways...best fit for a given parametric family.
But looking closer, this is going to solve my exact problem!
I think it's better suited here, since it's about statistics, not programming.
The obvious thing to do would be to draw estimated survival functions for each model.
$p(D)$ is just a normalizing constant so that $\\int^{+\\infty}_{-\\infty}{p(\\theta|D)d\\theta}=1$
If you are interested in general ordinal models, the Agresti book is a must. Another good book (but old, unless there's a new edition) Statistical Models for Ordinal Variables.
@mbq: `shuffle` might be in a toolbox, is not stock matlab. could hardly be worse than builtin `randperm` which returns sort index of a random vector. Again, this is probably the wrong algorithm (I just learned about the Knuth-Fisher-Yates shuffle here on stats.SE)..
Thank you, that does help. I'm still not clear where 1/2 comes from, why not 49/100, so will leave this open for a little while to see if anyone else has resources
Kind of related question is addressed in this [blog](http://bayesianthink.blogspot.com/2012/11/fun-with-uniform-random-numbers.html) I wrote up recently.
@RyanRosario yes, probably Quora is a better place for this question. I asked there as well: http://www.quora.com/Machine-Learning/What-machine-learning-algorithms-are-used-for-internet-advertising
You need to treat `sibsp` and `parch` as factors, not as numeric variables. Split the data into training and test set using stratified randomization to ensure that both sets have the same combinations of all explanatory variables. And don't expect wonders: after all, for each person with a sibling on board, there will be another record in the dataset with *identical* values of all independent variables but with a different age (unless they are twins/triplets etc.)!
@whuber - Oh, I see.  It's really the set of $(\\varepsilon_1, \\varepsilon_2)_t$ that are iid MVN.
Aren't these formulae for creating a combined test statistics for independent tests?  (In the situation I've described the tests are not independent as they involve the same data.)  If not, could you please provide a reference, proof or some other form of evidence.
Conjugate gradient is a natural choice.
I forgot to mention that you might want to look at function factanal, if you have not done it already. It does factor analysis rather than principal components, but will return the scores directly.
Evidently $x4$ is categorical with four categories. There's a marked lack of goodness of fit for one of its categories where the fit is near $77$.  The problem is due to one low outlier (observation 35).  There are two other problems called out by the warning messages.  With three such issues in $62$ records it seems premature to be looking at normality of residuals: deal with the warnings first and then examine the residuals more closely.
I may be misunderstanding the phrasing but that looks like scale to me. What's the mean - $\\alpha\\beta$ or $\\alpha/\\beta$? (alternatively see the explicit functional forms below)
This reply is identical to http://stats.stackexchange.com/a/44588/930. Can you provide more details so that your answers remain self-contained and indicate how they apply in the present context?
Also, "dependence refers to any statistical relationship between two random variables or two sets of data" is such a notoriously explained that I am confused at the start. Based on what can we conclude that two samples are dependent on each other?
Do you know about Stan?
Please fix your question; in this state it makes no sense.
Thank you.  I think you'll get a glimpse of what goes wrong if you consider a case where the kernel halfwidth is small.  Your expression appears to reduce to a chi-square-like formula where every observation has been placed into its own bin.  That will not have a chi-square distribution.  It raises the additional question of what half-width to use for the kernel estimates, because your statistic (and its sampling distribution) will depend heavily on the choice of half-width.
You should consider accepting his answer, click on the tick below to the answer score. Welcome to the site!
You can use [JAGS](http://www-ice.iarc.fr/~martyn/software/jags/) on OS X (it works with R too).
I am afraid the question does not have a 'standard' answer as I suspect that the authors are using a term that may be domain/context specific. What is this paper? A link/reference may help in answering your question.
How is the data coded? Especially: what does zero mean?
@Procrastinator, I actually have paraphrased this idea in my question description. But what puzzles me is that what is the purpose of using uniform distribution to compare with $a$ here? And why?