tttjjj/elt_bench huggingface dataset:
Of course, one *can* do these things, but one is left with the potential argument that the confounders you would have eliminated with parallel design but were unable to with sequential runs were the actual explanation of the observed effect. "But of course the treatment group did better - the day of their arm of the experiment, the weather was much more comfortable; the control group struggled through sweltering conditions and the airconditioning was on the fritz; that could account for the entire effect." You have to balance up what you can do and weigh up the various kinds of costs.
@Marco isn't ANOVA a (very) special case of regression?
thank you for your reply. In section, the authors say $f_t$, they should use the training set with the same number for each. I was confused because in the batch learning, t means the number of iterations. But in online learning, t corresponds to the data set $f_t$ uses.
Let $F(x) = P(X \\leq x)$. Then set $y = F(x)$ and solve for $x$. The general solution to that inversion problem is the inverse CDF, $F^{-1}$. If you generate Uniform(0,1) random numbers and plug them into $F^{-1}$ you have random numbers with CDF $F$. For example, $F(x) = 1 - e^{-x}$ for $x > 0$ is the exponential CDF. The general solution to $y = 1 - e^{-x}$ in terms of $x$ is $x = F^{-1}(y) = -\\log(1-y)$. Then, it follows that if $u$ is a uniform(0,1) random variable, then $-\\log(1-u)$ has an exponential distribution.
I'm not sure I can follow your question. What means "policy weighted", "policy level loss ratio"? Also, I don't understand what is the relationship between the average loss ratio for males and the sum claims for males/sum premium for males. Is the former the same as the latter divided by the number of males? Also, how is this connected to the title, the equal weighting of the policies and a rule based segment loss ratio. Perhaps it would help if you wrote it out more clearly/precisely. Do you have a mathematical model or something?
ok, I probably did a mistake in the exposition of the problem, what I assume is that when a variable is not in the sample, say $x_j$ then $d_j = 1$.
@Thomas. OK, I get it. Many Thanks
If you refer to this [package](http://www.mirrorservice.org/sites/lib.stat.cmu.edu/R/CRAN/doc/packages/fGarch.pdf), then I think dsstd is neither of these. It is used for the 'two-piece' t with the parameterisation in Fernandez and Steel (1998) which is referred as to 'inverse scale factors'.
+1 Thanks for the reference and welcome to this site! It would be great if you could also give a succinct overview of the contents of the paper as we strive for answers that can stand on their own.
@JohnSmith: well, then put the equations the prof had where the $w$ appeared (I'd expect it to be the normal vector of the separating hyperplane, as Marc Claesen says). And maybe tell us that this was in a lecture.
I suspect you will get loads of TL;DR reactions to this question, which doesn't bode well for the prospect of getting great answers. Do you think you could highlight the key information and remove the less essential parts of this question? On my first reading, the really important part seems to be the sudden switch from counts to proportions. Are you asking why a regression based on proportions can have different properties than a regression based on counts?
This looks like it may be [off-topic for CV](http://stats.stackexchange.com/faq#questions) and on-topic for SO. If you'd like to move it there, you can flag it and a moderator will move it for you.
I would really appreciate if someone could describe the two scenarios in a different way. I am not sure what their motivation is, what exactly the two scenarios are (I'd appreciate some clarification)... Basically, if someone can translate what is written into something that is more easily understood..
I don't think you are taking the dimensional argument seriously; it is not refuted or rebutted by changing units of measurement. Units are matter of convention, but dimensions aren't.
It's just a straight finite mixture of V with prob 1-p and V shifted up by c, with prob p. both the pdf and the cdf are [straightforward linear combinations of the two components](https://en.wikipedia.org/wiki/Mixture_distribution#Finite_and_countable_mixtures). (Don't sweat it, you're probably just having one of those days where things don't seem to go smooth. For me that's most days)
Since $Y = |X|^3$ necessarily takes on only nonnegative values, it would be surprising if its pdf was nonzero for negative values of $y$.
I'm not sure why the information that the observations are paired is important to the hypothesis being tested; could you explain?
$$f(y) = \\dfrac{2n.y^{2n-1}}{\\theta^{2n}}$$
$$E(c\\hat{\\theta}) = \\int_0^\\theta\\dfrac{c\\cdot2n.y^{2n}dy}{\\theta^{2n}}=\\dfrac{c\\cdot 2n}{\\theta^{2n}}\\times\\dfrac{\\theta^{2n+1}}{2n+1}=\\theta$$
$$c = \\dfrac{2n+1}{2n}$$
Awesome! Thanks! I'll mark the existing answer as the solution.
Ok, I thought there'd be a more combinatorial argument but that's ok for me! Thanks!
You should find the one-way ANOVA is equivalent to the three-way ANOVA with all interactions (including the three-way one). Do you?
It would help if you could say / demonstrate what you'd tried & how it went wrong. One thing I can't see, eg, is whether you have a known output from your model that you can use to help determine what's going on in the middle.
@alto But i Know that on which feature each function depends. I am thinking of the line that using MARS or some other algorithm first calculate overall function.(F=F1+F2+F3) Now let say  take feature x1 depends upon only F1 and F2 , In my training set I will make  other feature constant and try to formulate something.
I like all comments! And they all seem to be right. But I was just searching for the easiest answer :) What happens when you assume that the errer term is normal distributed. That this occurs now very often in reality gets clear from the other answers! Thanks a lot!
Thanks, that is what I needed!
It sounds like you mean clustering when you say "automatic labelling."
By the question, $M$ is the number of $Y-$ realizations that take the value $r+1$. Then $M/n$ is a consistent estimator of the probability that $Y$ takes the value $r+1$ (we implicitly assume that this probability is bounded away from zero and from unity). So $\\operatorname{plim}(M/n) = P(Y=r+1)$. But the probability of $Y$ acquiring the value $r+1$ is equal to the probability that $X$ is greater than $r$.
Isn't this question simply "What do these two papers say?"
+1 (should be +10) - Best explanation I've ever heard :]
Thank you for your contribution. It seems to address an unnecessarily narrow interpretation of PCA, however. (1) PCA has been fruitfully applied to highly non-Gaussian datasets. (2) PCA is not a formal parametric procedure; it perhaps is better to think of it as exploratory in spirit. (3) *All* covariance matrices, of any kind of multivariate distribution or data, are diagonalizable. Neither Gaussianity (Normality) nor non-degeneracy are requirements. (Symmetry of the matrix and having real components [guarantee diagonalizability](https://en.wikipedia.org/wiki/Spectral_theorem).)
I don't use SPSS, so I can't help with that, but I think you want to at least explore an ordinal logistic model.  If the assumptions are met, you will have a much simpler model to deal with.
Here's [one reference](http://rctdesign.org/TechReports/ARPHnonnormality.pdf); note that they really try to break the method, with quite extreme examples. With less extreme data it'll all still work at modest sample sizes. You could also look up McCullagh and Nelder's classic text on (generalized) linear models, where they are careful to describe linear regression through "constant variance" models, as this is the assumption that matters. For robust standard errors see intro econometrics texts; the STATA software's documentation is also a good source.
Thank you! That's a great answer, finally I've understood!
This question appears to be off-topic because it is about how to use R, but w/o a reproducible example.
Oops, I was imprecise with MAR versus missing completely at random (MCAR)...I assume the notes do a more precise job!
You need to provide additional info. Are A, B, and C independent conditions? Can one condition be singled out as the condition of interest? Depending on this, you might consider other conditions as censoring or you might go to a competing risks approach...or event to a multi-state approach...
A [test statistic](http://en.wikipedia.org/wiki/Test_statistic) is a [statistic](http://en.wikipedia.org/wiki/Statistic). So a random variable. One speaks of the value of a test statistic when looking at an observation of it.
Wow, thanks for the edited version.
What is it that you need that you cannot get from a uniform random number generator?  You can always do transformations (as this site does to get a normal distribution from a uniform distribution:  http://www.random.org ).  What do you mean by a "general non-uniform distribution"?
@Michelle: All combinations make sense and are valid. And yes, I would like to know the best combination of factor levels across all 5 factors simultaneously, for the most part. I know what you're thinking: I could simply compute the mean for each 800 combinations (because there are 10 samples in each combination), sort, look at the top answer, and see what the level was for each factor. This technically would give me the single "best" performance. However, I'm looking for a slightly more general answer. What if factor1 level "A" is at the very top, but "B" is very close in positions 2--100?
This guarantees the diagonal elements are maximal in the columns, but normalizes across the columns rather than the rows. If the normalization is done across rows, then the maximality condition is usually not satisfied. I don't want the diagonal to be maximal in the rows generally, only in the columns.
The key trick to limiting numerical instability is on page 45 (equation 3.26).
+1 Very clearly put.
Thanks for the feedback. The graph needs to have multiple edges to capture the direction of comments between actors in a nested online discussion forum, so your approach sounds like a good one. Cheers!
MI has a minimum of 0, and MI = 0 if and only if the variables are independent.  MI has no constant upper-bound though (the upper-bound is related to the entropies of the variables), so you might want to look at one of the normalized versions if that is important to you.
I appreciate your take on it. All the answers are helping me to think about the question and possibly understand better what I'm asking.
How would you implement a utility function? Could you provide an example? After reading that article, it's still unclear to me.
Very good suggestion! Thank you. Some of my data sets are long (i.e. many years), while others aren't. I'll study up on Peak over thresholds.
Ok, I have sample error. But, as I got a non-random sample, I supose that I can't control this error, since no all individuals of the population have had the same probability to be included in the sample. Therefore, I can't compute the margin of error... Then, I can't understand how to make inference form non-random samples. Could you explain your options in more details? I see the same problem if I choose either a) or b) option. Thank you for the help.
By the way, in the same link above, author mentions that if you differenced series and got negative autocorrelation you may correct for it by adding extra MA-terms in your model for $Y_t$. While my first intuition when I see autocorrelation at a single lag is to add 1 AR-term (and this worked in my case). Do you have an idea what the author means?
It could be the use of `exp ()` that is causing problems.  For example you use "fnb" but then take log of it - this is not necessary, just use your "lfnb" variable instead.
You seem to be taking "categorical" to mean "nominal", i.e. **not** ordinal. I think most authorities regard categorical as including ordinal; indeed I can't think of a single exception. You missed out the possibility of regarding miles as a ratio variable. Note that "unique" means occurring once only; unless you have only 5 data points, "distinct" is a better word.
@Nick you are correct. thank you for the feedback. I have attempted to clarify.
@whuber, you're right. Some make this distinction, but many aren't clear about these ideas. My position is that there is no meaningful reality of "outlier" other than *contaminant*. Nonetheless, people should also / instead think about the issue as being concerned about point(s) if your results are driven by them alone (whether they are 'real' or not), & thus your results are very fragile. In short, there is no reason to be concerned about point(s) that are from your population & aren't uniquely driving your results; once you've dealt w/ those 2 issues, there is nothing left to "outlier".
"Uniform on the unit ball" is not equivalent to uniformity of $\\phi, \\theta,$ and $r$.
Hah!  I think my proposal is worse than yours :-).
Oh, I understand, thanks a lot for the response. So, it makes sense statistically. It just sounded a bit strange to me, conceptually, that the effect difference between an insignificant (country1) and a significant predictor (country2)  appears to be significant in favor of the insignificant predictor.
$X\\sim\\mathcal N(\\mu=0,~\\sigma^2=1),~~f(X)=2$
Adding some relevant links since this question & answer appear to be getting some attention & votes recently: Google previously release n-gram data, though it's likely outdated and incomplete, http://n-gram-patterns.sourceforge.net/; Bing has an n-gram API, http://web-ngram.research.microsoft.com/info/; comments on Carlos' answer to this question seem to treat n-grams as fixed sequences but "distant n-grams" are in use by larger/complex search tools, http://www.w3.org/TR/ngram-spec/#S8 -- but then you're getting close to Levenshtein distance, http://en.wikipedia.org/wiki/Levenshtein_distance
I actually looked at that dataset before but that example involved characteristics of the beer such as flavor. My dataset is solely the sets of beers that specific users bought. A lot of examples online involve more complexity, however I have a minimalist dataset.
This is the same situation addressed at http://stats.stackexchange.com/questions/33193. Both questions derive from the same misconception, so let's be very clear: **there is no normality assumption about the distribution of _any_ of the variables in regression modeling.** The present question also covers the same ground as another (which I haven't looked up) asking for a transformation to convert a categorical variable to approximate normality: the answer is that no such transformation can possibly exist (except perhaps when there is a large number of categories).
+1. It helped me a lot to understand better what I am currently studying: the differences between confidence intervals and prediction intervals. Thank you.
Gaussian distributions *are* Normal distributions, so what distinction are you trying to make in your last paragraph?
@Gavin Simpson, have you heard of/used the third book listed?
Possibly relevant to this issue: http://stats.stackexchange.com/questions/95785/how-to-check-goodness-of-fit-for-a-graded-response-model-in-r
The training methods are not scale-invariant, so scaling does affect how rapidly you train even if you don't saturate the nodes.
... but this is not counter-intuitive after noting that the $100$ pairs of zeros are an evidence that $\\sigma$ is highly small.
In principle your background sounds suitable for a statistics Ph.D. In practice your chances of being accepted depend on the strength of your qualifications and your appeal as a potential student, not just having pertinent degrees. The question isn't really answerable in any but the vaguest of terms, even if we could see a CV (posting one would not be appropriate either).
I've voted to close as this isn't a question on topic for the site as [per the FAQ](http://stats.stackexchange.com/help/on-topic) and it is a list question. There may be a limited number of machine learning *specific* journals, but if you want journals who publish research and/or applied applications related to machine learning the list would likely be in the hundreds.
I can find [18 posts on this site alone](http://stats.stackexchange.com/search?q=%22differential+equation%22). Although that *corpus* barely scratches the surface, it's a place to start.  (Certain PDEs are intimately connected with certain random processes, for instance, but I don't see much about that.)
I am making a distinction here @Babak. This question is worded in such a way that $R$ is given first and then $A,B,C$ need to be constructed ("I start with ... [and] now like to create..."). If that is indeed the case, then your solution does not work.  If it is not the case, and the question merely is how to generate realizations from a multivariate normal, then you have answered it--and we would need to migrate the thread to SO where it will be on topic (as a pure programming question).  PLS, we would appreciate your clarification of what you seek.
Actually, I agree with you about the intent of the question @Glen_b. Maybe I should add that my answer may take a little arguing to support, or lean somewhat heavily on the openmindedness of whoever's grading the answers, because the intent of the question is wrong! This makes the truly right answer trickier than it would otherwise be. It might still be marked incorrect, but it is more defensible than the implicitly "correct" answer, which would use a *Z*-test...
To see how difficult the situation is in general, consider the effect of the *single* nonlinear constraint $\\sum_{i=1}^n X_i^2=1$: it reduces $\\chi^2$ to a constant!
I think that you need to explain which literature you are looking at in order get a good answer to your question.
Do you have any suggestions for how to choose the grid points (non-uniformly)? If I need to choose k grid points, and say I restrict the grid points to only n possibilities, then the brute-force approach is to evaluate the likelihood of each (n choose k) combination, and choose the max. likelihood one. Are there alternatives? (Sorry for the late comment, I just saw your answer today.)
This probability can be found using other techniques of integration: why don't you compare those results with yours?  It might help to know that $x\\exp(x)\\lt 2.5$ is equivalent to $x\\lt 0.958586$ (approximately).
? Don't understand you. ANOVA checks deviations from the mean. Levene checks deviations from the mean of absolute deviations about the mean.
There's also Rubin's counterfactuals framework; there are also structural equation modeling and econometric instrumental variable techniques... some of that described in the [Mostly Harmless Econometrics](http://www.mostlyharmlesseconometrics.com/) which of the best statistics books written by non-statisticians.
P values are a clunky way of addressing variable significance that are derived relying on assumptions that are pretty much only ever true in synthetic data sets. I'd take a step back and think about what you want to achieve from your analysis and if needs be re-formulate a question focused on how to get there. The answer may involve using P-values but there's a good chance there is a much better way to find the information you want to know that using this rather old-fashioned (for good reason) approach.
I guess what is meant is that at least one of the expected frequencies is zero. A short comment, expanding @Peter Flom's comment, is that you should show us how this arises: it shouldn't happen, as the implication is that the corresponding category, or combination of categories, is impossible. Otherwise you could aggregate categories to avoid this. Ancient authorities recommended minimum frequencies of 5, but this is very conservative; minimum frequencies of 1 may well be tolerable.
thanks again. The link is informative. I hope to be a real mathematician one day. This site gives me a lot of inspiration.
I find this question a bit vaguely worded and ambiguous in a couple of places. Pinning down precise meanings for terms like "better", "comparable", etc. may guide you to a more easily answerable question. You also talk of the observations being $|A_i|$, but then presumably you also (somehow) observe $|A_i \\cup A_j|$ (for all pairs $i,j$?). The more detail you can provide here, the better. That would also help better substantiate your bounty statement. At any rate, have you looked at the **Sinkhorn-Knopp** (aka iterative proportional fitting) algorithm?
Interesting... I'm not exactly sure how to apply that to what I've done but I'll think on it a bit more so I can articulate a question. Thanks!
unfortunately I need to create samples as large as the original dataset, plus I need to do this process a large number of times... So picking random values doesn't seem like a viable option in my case.
Jason, I've added the R code for answering those two additional questions. Once you've changed the sampling function to return the complete order of the swimmers you should be able to answer pretty much any question like this although manipulation of the returned matrix can get fiddly. The other thing to consider is whether you want to return the finish times themselves in order to answer questions such as about the distribution of the winning time or the largest gap between times or so on.
@Alecos is correct: but the situation is simpler than that, because this is a one-variable problem.  All you have to do is check whether the zero of the derivative is in the interval $[0,\\infty)$. If not, the convexity of $-\\log(L)$ guarantees that the boundary point $\\theta=0$ is the maximum.  Consult any elementary Calculus text for details.
(+1) It is worth remembering that such reliability indicators are sample-dependent. A test is not reliable or unreliable: Reliability is a property of the scores on a test for a particular population of examinees (Feldt & Brennan, 1989). The 0.7 (group-level)/0.9 (individual-level) thresholds introduced in Nunnally's *Psychometric Theory* are not worth much than Cohen's d suggested cutoffs: They all depend on the context of the interpretation as you said.
Diagnostics are good, but the one piece of evidence that is even better is a scatter plot showing your two variables and the fitted line! I count here 24 or so values; with that small a sample size it's easy to over-interpret diagnostics. What seems most obvious is that you have two clusters and you're fitting a line through them. Whether that is a good idea is more a matter of the science underlying your data than something on which someone statistical can proclaim either blessing or curse. Whether it is a good model depends on whether you can think of a better one.
If your question is only about how to do this in R, this question would be off-topic for CV (see our [help center](http://stats.stackexchange.com/help/on-topic)). However, I think your questions are actually due to an unfamiliarity with the relevant statistical concepts, so it can stay, but be aware that the answer you get may not be 'sure use `some.test(data, vector)`'.
I thought it could be looked at as - Are lower values more likely to fall into 0 than to fall into 1. Would that make sense?
Why do you think you need SEM, at all? If you heard the hype that SEM solve all causal problems, it is an overhype, only ideal randomized experiments do. See the reference I gave in my answer below.
But how I know the true parameter value in advance?
Thanks for your answer. Indeed, i started thinking about whether different items could produce the 0s versus any other count and I actually think that there are a couple of my variables that would only explain 0s vs any other count.  So probably i should at least try ZINB first to see if my these variables work the way i would expect them to work.
See also here: [f-and-t-statistics-in-a-regression](http://stats.stackexchange.com/questions/3549/), & for a discussion of the opposite case, see here: [significant t-test vs non-significant F-statistic](http://stats.stackexchange.com/questions/24720/significance-of-coefficients-in-linear-regression-significant-t-test-vs-non-sig).