tttjjj/elt_bench huggingface dataset:
"*The proportion of head to tails is 1 to 1 as number of trails approach infinity (this is a fact correct?)*" - it's only a fact if the coin and tossing process is actually fair. You might make a symmetry argument in support of making this assumption, but it won't make a coin that's not quite fair become fair. The rest of your question is well covered already, so I won't labor that point.
If you don't fix the start and end points to say 0 and 1 (so that you can match the range of a bounded variable), a beta has 4 parameters.
I commend your not resorting to voodoo.  When you are using a tool and you do not understand exactly what it is doing or how it works, then you are not qualified to interpret the results of the tool.  When an analysis drives a business decision, and you are putting time and money into the result, it shows ownership if you take the time to understand the source of data.  It is the kind of hard work that gives you more opportunity instead of being "in the herd".
I would like to suggest the essence of the situation is conveyed by a generalization of the problem. Let $h:[0,1]\\to\\mathbb{R}^{+}$ be any bounded measurable non-negative function with a unique maximum at, say, $a$ and suppose $h$ is continuous at $a$. For $\\lambda\\gt 0$ define $h_\\lambda(x)=C_\\lambda h(x)^\\lambda$ where $C_\\lambda$ makes $\\int_0^1 h_\\lambda(x)dx=1$. If $f$ is any continuous function on $[0,1]$ and $\\epsilon\\gt 0$, then ${\\lim}_{\\lambda\\to\\infty}\\int_{h(x)\\le h(a)-\\epsilon}h_\\lambda(x)f(x)dx = 0.$ This readily implies the limit of $h_\\lambda$ is $\\delta_{a}$.
Ah, well then you don't have the protection I assumed you had.  So, go back to my initial answer "impossible to know".
CHIDIST, like many Excel functions, works when applied to arrays: it threads over the array in this case and returns one value for each of the 60 cells in the range a1:a60.  (Tested with Excel 2002.)  *If you do not enter it as an array function,* the value you are getting is the first value: in this case, it will equal `CHIDIST(A1, 3)`.
Breiman's "Two cultures" have been discussed [here](http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning): some interesting points, but it's hard to change or even explain one's mind-set.
Thanks for the answer, I know from the start there's a crack in my understanding, I didn't really know what it was. The explanation below about constructing bernoulli out of normal more or less clarify that. I will have to dig again to really understand the relation (and difference) between those two.
true that, but without getting the theory down, empirical practice could only go so far.  For example, I am great R programmer, I am poor at probably theory.  I can't publish papers.
Renato That's a good idea, but please make sure the new question is a genuinely a new question and not just a restatement of this one.  Your current question is confusing, in part because the formula "mean + 3SD" really has nothing to do (AFAIK) with the problem of finding the most frequent words.  You might find that the comment left by @cardinal provides a helpful direction.  It suggests you might be trying to *partition* or *cluster* the words into a group of high frequencies and the rest; are you asking for criteria to identify the cutoff or threshold between the two?
@AndyW: Your argument convinced me; although the most relevant estimate of the treatment effect is the same in both models, Model B should be prefered over Model C. I adjusted my answer accordingly. But what do you say to `Laird, N. (1983). Further Comparative Analyses of Pretest-Posttest Research Designs. The American Statistician, 37, 329-330.`?, who shows a equivalence of B and C?
The density function has a "weights = " parameter; would this be what you want?
Hi maximus - I see you unaccepted - is the answer inadequate in some way? Is there something further you need covered?
(-1): I'd like to prevent crossvalidated.com from degenerating into a platform where on every question someones suggests the commercial tool of his company. In this case we are better off to place ads in between the answers -,-
How do you dichotomize your data? Using the noisy measurement? If so, isn't that just hiding the noise (in your dichotomizing decision) rather than actually mitigating it? The error doesn't go away just because you rearrange things. When you feed your crisp labels (0/1) into a method that knows nothing about how you came to 0/1, these methods will naturally reflect a low error because they "believe" your crisp 0/1.
(+1) I would suggest to use `groups=pops` (i.e., a panel.superpose) to clearly highlight overlap between subpopulation. Also, confidence ellipse might be displayed using [panel.ellipse](http://latticeextra.r-forge.r-project.org/#panel.ellipse&theme=default) from the `latticeExtra` package, where ellipses might be computed as proposed in the `FactoMineR` package, for example.
It sounds as if you already have decided the 10 or so items that will feed into (comprise) your scale score. (You use the term "dependent variable" as if this scale score is going to be eventually explained or predicted by some other variables, but as I read you you have not indicated what they are--only the components of the dependent variable itself. Is that right?)  Further, are you looking for help designing the survey; computing your Engagement scale; or explaining/predicting it?
There are two valid ways to interpret the question, with two different answers. (1) Given a particular station (*viz.*, the "one station") what is the chance that two members are assigned to it and the other two stations get one member each? (2) What is the chance that *there exists some station* with two members assigned to it (and the other two stations get one member each)?  Because there are three equivalent stations, the answer to (2) must be three times the answer to (1). So, before attempting an answer, you first need a clear method to *state* this type of question!
It is noteworthy that the answers reflect two very different interpretations of these data, so a clarification is in order.  The reply by Matt Parker (and also the edit by @chl) views $X$ as indicating two groups (0 and 1) and takes the question as a request to compare the $Y$ values between the groups.  The reply by Glen_b takes the question as a request to compare the 14 values of $X$ to the 14 values of $Y$.  **Which interpretation is the intended one?**
libsvm does not scale well to large numbers of features and samples. Better use liblinear by the same authors. linear models are usually good enough in high dimensional space.
OK, I think I get it now: in effect, you observe $\\{(m_i,b_i)\\}$ and assume a somewhat complicated model for their joint distribution. The $\\Gamma_i$ are just regression coefficients, which becomes clear if we change to a more usual notation where you observe $(x_i,y_i)$ and wish to estimate $\\beta_0$ and $\\beta_1$ for which $y_i=\\beta_0+\\beta_1x_i$; the translation is $x_i=m_i$, $y_i=b_i$, $\\beta_0=\\Gamma_2$, and $\\beta_1=-\\Gamma_1$. (A) Is this a correct interpretation? (B) If so, is there any obstacle to applying the usual Maximum Likelihood machinery?
@AaronHall The equation isn't *necessarily* useless, since (multiplying back by $\\sqrt X$ it's $\\sqrt Y = \\beta_0 \\sqrt X + \\beta_1 X + \\sqrt X\\epsilon$, which may well be a potentially plausible model in some situations). However the $R^2$ on the form of the equation given in the question isn't much use you can't compare it with something fitted on a different scale. (Incidentally, if that was your downvote on my answer, an explanation of what you think is wrong in the answer would be useful.)
This is what I ended up doing. Thanks.
it looks like homework, you should tag it as "self-study"
I'd guess whoever said that was using "points" in the loose sense of "score" or "measure". Deviance certainly isn't the number of data-points in your sample or anything like that. And what it is is answered rather well in the linked post, I think; _what_ question remains specifically?
Thanks. I was making the assumption that L(theta | data) was equivalent to P(theta | data).
You won't know if it is safe or unsafe from the 10 data points alone. If you suspect skewness or heavy tails the solution might be to focus on a different parameter than the mean. For instance the log-mean or the median. This won't give you an estimate of (or confidence interval for) the mean unless you make additional assumptions, but it might be a better idea altogether to focus on a parameter that is less sensitive to the tails of the distribution.
I usually find P-values of little help here. Often the sample is so big that any deviation is flagged as significant at conventional levels. The key question is which of various candidate distributions fits best. We don't get a sense from your report (no data, no graphs) of whether you are talking about primary or secondary features. If the lognormal is approximately right, then your distribution is unimodal, and the Pareto is qualitatively wrong. That doesn't stop your distribution having e.g. heavier tails than predicted by the lognormal if that's what you are getting at.
It is difficult to see how Loess could be considered "linear" in any sense, except perhaps by using linear calculations within the algorithm--but virtually every algorithm ever invented employs linear calculations (multiplication, addition, and their inverses), so that's not saying anything.
Thank you for the extra suggestions and for all the effort you've put into my question! I feel smarter than I was a week ago :-)
An alpha reliability of around .94 for an aptitude test sounds quite good to me.
+Dan Thanks, that got me started in yet another direction that I had forgotten about.
@obesechicken13 Also, you could add dummy variables for known academic dates, like exam starts, etc.
Best answer I have so far is to print this on graph paper and use a pencil and ruler and calculator ...
@Macro Thanks I have re-written my question and will look at your recent answer.
[Hierarchical models](https://en.wikipedia.org/wiki/Hierarchical_linear_modeling) are a natural approach for this sort of problem; a lot of [hierarchical models are Bayesian](https://en.wikipedia.org/wiki/Hierarchical_Bayesian_model#Hierarchical_models)
Points 2 and 3 are so capacious that nobody will consider answering them. If you are new to clustering, please read books, or at least one, first. Point 1 is not clear - what you mean by threshold; aglomeration clustering algorithm does not use "thresholds".
**Reference**: G. A. F. Seber and A. J. Lee (2003), *[Linear Regression Analysis](http://www.amazon.com/Linear-Regression-Analysis-Probability-Statistics/dp/0471415405)*, 2nd ed., Wiley, **pp. 17-19**. (Certainly this is but one of many choices.)
Extracting the parameters from the CSS solution and passing them as starting values to the ML solver (via the `optim.control` argument) would stand a good chance of avoiding this problem. I haven't tested this because you do not supply a reproducible example of the difficulty.
Could you give an example of what you mean by "over-fitting"?  What software are you using to estimate the parameters of the negative binomial distribution?
1) Geometric mean is the exponent of the arithmetic mean of the logarithm'ized values. 2) The formula you display for geometric mean is not genuine one. It adds 1 to values, which suggests being a trick to cope with zero values; and this affects the result.
Would it be easy for you to explain it to me through an example?Like if we have a function f(x)=(1/2)|x| and x_0=1,x_1=0,x_2=1 how we interpolate this by a cubic spline with s_1"(x_0)=s_1"(x_2)=0 ??
I don't have the reputation to allow me to edit, but I think you want $H_0 = \\mu_{A1}=\\mu_{A2}$ (or $\\mu_{A_1}$ if you want a double subscript) [oops, it has automatically tex-ified that:  `H_0 = \\mu_{A1}=\\mu_{A2}` or `\\mu_{A_1}`]
"All I see is discussion of normalization to achieve non-degenerate G." What do you mean by "normalization to achieve non-degenerate G"? I gave an example on normalization which I know of in my question. But I suppose there are other meaning attached to this word.
Thank you for the answer, I'm going to need a little time to go through it and try it. Once I do, if everything checks out I'll mark it as the correct answer.
Dan, I just noticed that your example set of heads and tails includes an "A".
Distributions don't have standard errors, they have standard deviations. A standard error is the standard deviation of the sampling distribution of some statistic, but you don't state any statistic. Please clarify your question. If it's the standard error of the mean you seek, the distribution only matters in so far as its variance exists.
GLS doesn't render the error process i.i.d. , You need to incorporate ARIMA structure in conjunction with GLS to guarantee constant error variance in the presence of uncorrelated errors. Note well that neither of these transformations deal with Pulses,Level Shifts,Seasonal Pulses and/or Local Time trends that may remain in the residuals. Finally none of the above guarantees that the parameters have not changed over time. To review these concepts you might read some of my older responses to a myriad of previous questions.
Robby -- application I had in mind is here http://mathoverflow.net/questions/37582/how-big-is-the-sum-of-smallest-multinomial-coefficients  Basically integrals of Gaussian suggested by CLT give extremely good approximation to sums of binomial coefficients (for small n, even better than integrating Gamma representation directly!), so I was seeing if I can do something similar to get approximate sums of multinomial coefficients, which I need to get non-asymptotic error bounds for various fitters (like, maximum likelihood)
If you _randomly_ split the sample into 5 subsamples your 5 means will almost coincide. What is the sense of making such close points the initial cluster centers? In most K-means implementations, the default selection of initial cluster centres is based on the opposite idea: to find the 5 points which are most far apart and make them the initial centres.
Welcome and thanks for this nice answer, even providing a source to back up your claims!
@Crazzzy12 you may want to accept one of my posts as "the answer" wtih clicking on the button to the left of one of my posts and by upvoting my answers.
That would depend on characteristics we don't possess knowledge of. If there are several links with different payouts it will be a [finite mixture distribution](http://en.wikipedia.org/wiki/Mixture_distribution#Finite_and_countable_mixtures) across the distributions of the payouts, but you haven't suggested what those might be and even if those are known, we have no basis here on which to assume relative probabilities for each, so those would need to be estimated.
One possibility is small multiples; you could plot each series in its own plot over a reference line which has the overall mean of all series (which line will be the same for every plot).
Convert to factor (to get day fixed effects) or convert to numeric and rescale so value of first day is 0 to get a linear effect of days since origin.
Thanks - I had an inkling  - or perhaps a sinking feeling ;) - that bootstrapping might be a popular option.
@Jake It is not clear that it would make the plots easier to digest. My assumption is that it would takes up space and be distracting. If I saw sixteen y-axis labels, I would assume that they did not all contain the same information.
In addition to Peter's excellent answer, univariate screening has been shown to convert a really terrible stepwise variable selection algorithm into a total disaster.  How did you get the idea that building models based on $P$-values is statistically valid?
Dear Stephane and Rasmus - 
I teach basic Bayesian analysis to students. As such, I would like an example with real data to show its use. To do so for (say) a one-sample t-test is easy. However, to have a real life example for basic Bayesian analysis is not as simple.  And Rasmus, if I remember correctly you are working on a package that could benefit from such examples as well :)
Interesting! It reminds me of the symmetric sigmoid function in neuro network, where the horizontal axis denotes "stimulus" while the vertical axis denotes "response". There should be a place for asymmetric tangent if the "response" to positive/negative "stimulus" is imbalanced.
Cool thanks - I'm reading through the first one now. Have you come across a more effective way to regularize again over-fitting with kernels with many parameters like ARD if the model complexity term in the marginal likelihood is not sufficient to prevent overfitting?
Thank you! I've also posted the technique I used to plot a curve for this model below:
XOR is often used in the canonical introduction to SVM or Perceptron it seems. Alternately you can simply devise a function.
@whuber A nice and handy overview of pros and cons!
One is not a percentage, the other is a percentage... it depends on how you wish to evaluate your forecasts.
Can you provide links to those papers?  Thanks.
Additionally, would it be strange to add the condition at which a participant was assigned as a predictor?
What do those numbers mean? Are you saying +/- one degree and one unit of pressure? How are you defining 0?My impression is that those points are completely arbitrary and may be chosen purely for maintaining symmetry in a display. Are you referring to any authority when you ask if there is a reason to use those points? Finally, is this a chemical reaction, or a biological process (that's not terribly important, but you seem to be suggesting that this is not a purely statistical question)?
I'm a big fan of that book, but if the OP is primarily after the underlying mathematical theory, it may not be the place to look.
Chapman and Hall was a publisher, now part of CRC Press. You probably mean Silverman (1986). I don't know what reference you mean by Silverman (1998). Either way, fuller details would be helpful.
The wording "largest maximum is 1/2" suggests equality, not inequality as in your expression. Why not improve the last line of this nice answer and write "probability that the maximum is not larger than 1/2" which corresponds to the mathematical expression that follows (and probably also to probability seeked by the OP)?
Can you describe the structure of the dataset in more detail? As in - is it a matrix of features or something more different?
The second equation makes no sense and appears to contradict the first.  What are these equations supposed to be doing?  If we take $x_k$ to be a vector then the first equation looks like an update step.  The second and third equations are not.  The third equation is part of the usual termination criteria.  Another part of those criteria is that $|F(x_{k+1})-F(x_k)|$ should be small, where $F$ is the objective function (of which $f$ is its (scaled) gradient).
(+1, hours ago) Your edit uses the argument I had in mind after the OP's update to the question.
It also doesn't inspire confidence in reading the comments that the header to the QMPE file describes it as QMLE still.... no wonder it was hard to parse what was going on.
The SD is more intuitive because it is on the same scale as the data. However, when working with the normal distribution, the variance is the parameter not the SD. Thus, variances can be more useful when working with distributions mathematically. Eg, variances *add*, but SDs don't.
I haven't used it myself, but look into conditional logit.  It's in the `survival` package in R.
I do not think I fully understand your approach. BTW, in the `Amelia` package this is easy: there is an `ords` option that sets one or more variable as ordered categorical. It seems that `mi` package treats these variables in a different manner.
It would be more diplomatic--and more correct--to say that you would prefer a text with more or lengthier explanations.  If you insist on staying with the "poorly written," it would be good to provide objective support for this assertion, lest the question be closed as confrontational and contentious.
Here's the thing: if there is an interaction it does make no sense to "not be interested in it", because you cannot meaningfully interpret main effects alone if there's an interaction. So in addition to the answer below, I would urge you to reconsider what you are doing.
Thanks. Unfortunately, the group invariability analysis gives really weird results (one of the groups is much smaller), while the linear regression makes sense, but SEM is more appropriate due to having scales as independent.
This question appears to be off-topic because it is about details of what particular R code does: this site is not intended to be a substitute for reading software documentation (and nor is Stack Overflow).
I like your first paragraph and about the last one, there's a nice summary about [Problems Caused by Categorizing Continuous Variables](http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous) on Frank Harrell's website. Should you have some more time to invest in your response, you could expand on the "why the difference" (`summary.formula` offers either standard $\\chi^2$ test or trend test in case the variable on the RHS is ordinal; the $p$-value associated to Age treated as continuous in the GLM is derived from a Wald test).
You want a local linear model? In that case try the **sm** package, which too is highly regarded and written by two eminent statisticians in the area of local models. If you were to relax the local linear assumption, try either of the packages I mentioned in the comment on Ari's Answer. Also, do consider Accepting Answers on Questions you ask (you have a low accept rate). See the [ask] section of the FAQ for why this is a good thing, and also how to accept (click the big tick mark next to an Answer).
@Dilip Yes, you said correctly and I would have been glad to see your comment (more fully fleshed out) posted as an answer. For the record, so that we have a valid answer in this thread, here I have *proven* the result and explained (using statistical ideas) how one would go about finding such a proof in the first place.
@B_Miner, $k$-fold cross validation tends to give quite biased estimates of $R^2$, as it generally isn't estimating the true quantity of interest. Many (most?) similar procedures have the same problem.
Since most of my parameter estimates were statistically significant for p=0,05, I thought my significance level is 0,05. But I do not really what the significance level of the likelihood ratio test is for me. Where could I find that out? The [study I'm replicating](http://onlinelibrary.wiley.com/doi/10.1111/j.1540-6261.2008.01354.x/full) uses _Prob > χ2= 0.000_ to show that their model is statistically significant. Thus I think my model should be significant as well. Is this conclusion reasonable?
Just found an ungated PDF version of [Displaying Uncertainty With Shading](http://www.mrc-bsu.cam.ac.uk/personal/chris/papers/denstrip.pdf).
Great practical interpretation, and also makes sense in the Bayesian context... "theoretic"-based likelihood ratio vs. "atheoretic" prediction error.
Without more information, there are too many possible answers, because your formulation allows us to replace your data with literally *any* set of $N$ values having the desired mean and SD. Please either provide enough information to answer this question or consider telling us *why* you want to undertake this exercise.
I think correlation coefficients are useless for curvilinear relationships.
Good point, that would give a 'shifted log-normal distribution'. The log-normal and the log-logistic are pretty similar in shape apart from the heavier tails of the latter, so you could try both and compare results.
How are these data values measured? What is the basis for assuming they can be described or approximated by a polynomial function? How do you plan to account for variation between the data and your fit? This information is important for guiding the selection of a fitting procedure.