tttjjj/elt_bench huggingface dataset:
Is $X$ a random variable or a set of data? In the former case, it is well known (and easy to derive) that all the probability of $X$ must reside at the extreme values and the conditions of the problem give two simultaneous equations for those two probabilities that are easily solved. But if $X$ represents a *dataset* (as in the referenced question), this is a quadratic integer program whose solution varies according to the size of the dataset. There is no closed-form solution in that case.
(+1) thank you. do you know an accessible reference for the survey weights, other than the docs for `surveyglm`) ?
From the stand point of machine learning and prediction, try it multiple ways and see what works best - i.e. yields the best prediction.
@mike I prefer to comment on your comment rather than your account.  It is a cute answer but once you square the nrmals you have a chi square which is gamma and what you are doing is summing (getting a convolution) i.i.d. gamma.
But if you want a CI on the number of lost packets at a particular moment, and not on the proportion of lost packets on the long time, you don’t want a mean probability of packet loss. If you have 40 measures, sort them and discard the smallest and the highest; the remaining gives you a 95% CI. With 10 measures, just keep the whole range of measures as Dilip suggested.
Should I group together the last say 3 items and look at them as 1 n-gram or something?
If your question is "how do I model two variables without any in-domain information about how they should be related or the properties we require the model to have" then that's a question whose answer could fill many books - in fact, it does. You might consider nonparametric methods, for example (which fills books on its own) but if you need a succinct *equation* for the mean that approach may be unsatisfying. I would say if you're going to do model selection, estimation and assessment all on one set of data, then you should be aware of the problems with that. ... (ctd)
Visual inspection of data is *never* out of the question: if you can read or analyze them, you can look at them!  True, huge datasets can create overwhelming plots, but don't let that stop you: plot small random samples of your data instead.
Shouldn't the intercept be still there if only the predictors are centered?
@gung 	
For example, my hypotheses states: "If the costs of copyrighted content are lowered the individual’s intent to commit piracy will not decrease". But what the questionnaire actually measured was the effect of the costs on the individual's intent to buy. As you can see I can't use this questionnaire to answer my hypotheses.
That $W_n$ is a function of the sample, and hence of the sample size, and so it will vary as sample size changes/increases.
I've had success combining plots using `grid.arrange()` from the `gridExtra` package. With respect to the gif, [this answer on SO](http://stackoverflow.com/questions/18317721/saving-plots-in-r-as-gifs) might be helpful for you. Incidentally, I can't seem to access your data through **R** or my browser, but maybe it's just me.
I thought that the other things you called out as lacking were implicit in the description I gave.  each ticket has the same chance of being drawn.  Tickets would not be replaced after drawing, because no one is admitted twice.  Every prize is the same.  Tickets are drawn until every prize is awarded.
First of all, the time complexity of univariate optimization is sub-linear in the width of initial interval between lower and upper so there is very little cost in taking an interval that is too large. clearly the formula you use is only meaningful when ν>2 so that's your lower bound. As for the upper bound, I concur that values of ν beyond, say 100, should are for all intend and purposes yield similar results so that would be a tentative upper bound.
(Cont.)  Well, ["Fisher's LDA"](http://stats.stackexchange.com/a/71571/3277) is simply LDA with K=2. When doing classification within such LDA Fisher invented his own formulas to do classification. [These formulas](http://stats.stackexchange.com/a/31384/3277) can work also for K>2. His method of classification is hardly used nowadays because Bayes approach is more general.
`(1|Batch)` means "one intercept per batch".
Your description of what I did is correct.  I'm treating each "cycle" as a separate data set and taking regressions of each. And yes, I determined the data fit a log-linear distribution solely through visual examination.  However, the Kolmogorov D test gives a D value less than the critical value.  JMP fit the gamma distribution automagically.
Just a side note. "Error" in abbreviations like MSE usually actual mean "residuals". Practically, residuals and errors frequently are treated as synonims, MSE=MSR. But theoretically there's important [distinction](http://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics) between the two terms
Hint: Consider the difference of any two such estimators. What would you like to show about it? How might that be related to the definition of completeness?
When you say 'predict', what is it you want to get? A mean prediction? A median prediction? A prediction for some particular loss function? Do you need a confidence interval for (say) a mean? Do you need a prediction interval for a new observation?
I've tried to make some edits to your question. If I've changed the meaning unintentionally, please feel free to edit some more. Saludos.
**Related**: http://mathoverflow.net/questions/104413/tail-bound-for-poisson-random-variable/104591#104591
As an inspiration, I've posted pseudocode for a _non-_parametric bootstrap [here](http://stats.stackexchange.com/questions/73238/alternatives-to-pvals-fnc-to-compute-confidence-intervals-for-fixed-effects/73288#73288). Shouldn't be too hard to adapt to the parametric case.
For some reason I can't login on CrossValidated using Yahoo to accept your answer...
@chl, no, MATLAB.
+1, this is a really nice answer. I hope to see more of them in the future.
How do you know your test was wrong?  How large was $N$, what was $\\theta_m$, and what was the standard error of the estimate $4\\pi M/N$?
Did you try a Box-Cox transform ?
Well, sample mean and sample variance are well-defined for a sample of size $2$. But it would help if you describe a little bit more the context.
@Aksakal: can you prove/explain it?
No. It is not the number of coefficients of any model. It is just the complete description of a single variable. It has nothing to do with the dimension of a model. AR(2) tells you nothing about the dimension of the signal.
See *On Profile Likelihood* (S. A. Murphy and A. W. van der Vaart), http://www.jstor.org/pss/2669386
(1) What do you need the explanation of?  After all, the code provides all the details. (2) The numeric derivative is probably computed by `uniroot` (or possibly by `nls`). Its occurrence is due to your particular data, so I cannot readily reproduce it. One approach to debugging is to select an alternative root-finding method, preferably one that does not compute derivatives. If the problem lies with `nls`, try fitting the model `log(y) ~ log(x)` instead.
Python is of course very easy to learn and to read, so I guess it is a matter of taste. I suggest these links: [python-stat-workbench](http://stats.stackexchange.com/questions/1595/python-as-a-statistics-workbench), [what-programming-language-for-statistical-inference](http://stats.stackexchange.com/questions/4759/what-programming-language-for-statistical-inference), [machine-learning-using-python](http://stats.stackexchange.com/questions/8817/machine-learning-using-python), [clojure](http://stats.stackexchange.com/questions/130/clojure-versus-r-advantages-and-disadvantages-for-data-analysis)
While it's true that any number of categorical effects will yield the same *predicted* values for the outcome in either model, the inference on their corresponding effects will be completely different. This is what OP's original question addressed.
It may be useful for demonstrating the nonlinearity but it is not good for modeling purposes.
@robin Many thanks for your comments. I apologize for my confusion of terminology. I have updated the question to include a more complete description of our correction procedure, in the hopes that it provides clarification. I have also updated the q-value link; thanks for pointing me to that.
@Tal: I am basically happy to allow any R questions on here, but I just don't see how this has any relevance to the subject of data analysis.  This is a meta question about R itself, not even about using R.  We'll see what others think, but this seems more appropriate for stackoverflow, r-help, twitter, or a blog post.
(Just in case one of them turns out to be equivalent) What were the model selection criteria that were included?
There are other criteria besides CCC. Have a look at [Determining the number of clusters in a data set](http://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set), to see the main ones.
Ahh I didn't notice they were on different scales initially either! Perhaps further visually distinguishing between the panels would be useful, such as a space in between the top panel and the bottom panel (and enclosing each panel in a box). Very cool idea though (+1), I like the use of the super-imposed "net" categories, although I think the line as suggested by the initial OP could be useful as well.
Given that $z^2$ can be quite large, ignoring higher order terms in the Taylor expansion is pretty fishy.
I am affraid the answer to this is going to be a looong list of tests, model comparison techniques, goodness of fit tests, graphical tools, ... I believe a [QQ plot](http://en.wikipedia.org/wiki/Q-Q_plot) is a good starting point in the univariate case. Actually, model selection in a more general scenario is one of the [open](http://www.spencegreen.com/2011/05/04/michael-jordan-on-the-top-open-problems-in-bayesian-statistics/) problems in statistics proposed by Michael Jordan (the statistician).
http://www.nesug.org/proceedings/nesug07/sa/sa17.pdf
For your 2nd question, you must set `pool.sd=FALSE` in `pairwise.t.test` if you want to get comparable results; otherwise R is using a pooled estimate of variance based on your three treatments which obviously differs from the one used in t-test for 2 independent samples.
@chi: yep. But im still not found the right material for it.
To get significance you should keep the numbers of visitors and the number who click through, rather than just the ratio.
I'm not sure I understand what a DV is - is this the status indicator for passing to the event argument of Surv()?
Thanks, that sounds interesting. I'm "accepting" this answer, but I'd love to see additional thoughts on this topic in the future.
I definitely want a normally distributed variable. I'll update the question with the specific problem I'm trying to solve.
why don't you share your data , your mode, tour residuals and your analytics and perhaps we can help you.
Using a table suggests you have made many tests of such equality.  This is an unusual thing to do, perhaps because it may be difficult to interpret the results and perhaps because differences in correlation do not appear to reveal much.  Suppose two variables do have different degrees of correlation for two subsets of the data.  What does that *mean*?
@Glen_b: The data are the ratios of the number of fixtures installed vs those claimed in an efficiency program, so they can exceed 100% on occasions where more have been installed than originally claimed.
Errm, not quite.  The variance of a binary variable $Z$ is indeed $p(1-p)$ but the variance of the estimate of $p$ isn't because it depends on the sample size.  More data makes you less uncertain about what $p$ is.  But regardless of one feels about binary variables, $p$ isn't one.
Why don't you work out the exact solution for the simplest case where the two algorithms differ--finding three nonnegative integers summing to $2$? You can easily enumerate all possibilities (six with the first algorithm and nine with the second). The correct result, given by the first algorithm, is that the expected numbers of $0$, $1$, and $2$'s are $9/18$, $6/18$, and $3/18$ respectively, whereas the second algorithm gives $13/27$, $10/27$, and $4/27$ respectively.
Could you clarify how these thoughts address the question in this thread? And what is your justification for concluding the intercept might be larger than $-2$?  After all, if all the independent values were negated, then (1) the intercept would not change but (2) all coefficients would become positive.  Moreover, if the units of measurement of the IVs were changed, then their coefficients would change in proportion; and if the IVs were recentered, the intercept could change to almost anything.  Thus it's difficult to see how you can conclude anything at all about the coefficients.
(Disclaimer: IMHO genetic algorithms are among the more evil optimizers for statistical models. They are exploiting variance in performance estimation in a very bad way). So: at least do check your final model with truly independent test data!
Your question seems to be based entirely on a typographical error: the quotation minimizes $\\frac{1}{2}||w||^2$ while you write $1/||w||^2$. Does that resolve the issue?
Is there another portion to your question that you have omitted?
@whuber: I think ilhan is asking how to produce a chart including the circled table, not how to present those statistics graphically. If so, this is a 'how to ... in Minitab question' not a statistics question. I don't know if that is on-topic here.
That makes sense. I suppose obviously `x2` couldn't have "no value" since it must be one of "High", "Medium" or "Low".  Thanks for your answer.
Very useful, thanks.  Multi-level modelling is almost unheard of in my field for this kind of data, and we are planning to present two analyses (the standard ANOVA and the multi-level modelling) to open up a methodological discussion.  Please note it is only 1/21 subject distribution that is normal (profuse apologies if there was a typo on the other list), prior to analysis (i.e. without any predictor taken into account).  Thanks for confirming my intuition that multimodality should be expected at this point.
I edited for clarity.  So basically if you have a normal random variable whose variance comes from a Gamma(1,1) distribution then that random variable will approximate a Laplace distribution.  What the OP is asking is how to determine the parameters of the corresponding Laplace distribution.
Does [this](http://www.minitab.com/en-US/support/answers/answer.aspx?id=314&langType=1033) help? ('2-level factorial (specify generators)'.)
Possibly relevant: http://stats.stackexchange.com/questions/23808/are-there-problems-with-inference-using-linear-regression-on-observational-data
The intention is that i will eventually try to model this using Bayesian, but i was trying to understand how i can make a decision before fitting the models. If there is a possibility that ignoring the nested nature of the data messes things up, them i will try them GLMMs first. The only package for R that i am aware of that can do multilevel ZINB is glmmADMB. Would you recommend any other packages?
Good point Dilip. This clears my doubt in certain way. That's the beauty of Maths. Thanks.
$Var[X] \\stackrel{d}{=} \\mathbb{E}[X^2] - (\\mathbb{E}[X])^2 \\to Var[X^2] = \\mathbb{E}[X^4] - (\\mathbb{E}[X^2])^2 $ or perhaps even more confusingly $Var[X^2] = \\mathbb{E}[X^4] - (Var[X^2] + (\\mathbb{E}[X])^2)$
I should have specified that I know the seasonality, so I don't need that complicated of a solution.
Although the answers below differ in their exact form, they all point to *exponential* in $N$ for fixed $n$ and $p$
This was the question I found most distracting after statistics 101. I would encounter many distributions with no motivation for them beyond "properties" that were relevant to topics at hand. It took unacceptably long to find out what any represented.
If you're used to R and can express it in R it might be useful if you posted that.
Hi all, Thanks a lot for the pointers. I will be sure to look.
Also, if one just wanted to see the linear regression: `lines(x, fitted(lm(y~x)))` would do the trick.
I'd say that the derivative of the goal function with respect to the change point parameter is rather un-smooth
Yes, you need to use anchor items to have a common scale for latent trait. But isn't your question more about inter-rater variability? How many subjects and items do you have?
If you call 200 as the correct value, this explicitly means that you don't want to control for V2.
I'd imagine two-sided, since the question only sought to determine whether or not the coin was unbiased. That is, it appears that $H_{a}: p \\neq 1/2$. But, it's unclear whether what is written above is the verbatim question or a paraphrase.
The OP may be referring to this thread: [Sum of squares two ways, how are they connected?](http://stats.stackexchange.com/questions/4446/)
I am not sure what is going on here, but it *might* be that there are some 0's and that when the baseline is "yes" there is a division by 0 somewhere.
Yes, you just have different start times for some of the observations. That's not a problem at all.
Yes, thank you very much
I understand, yeah I guess I was aware that was the way you are _suppose_ to do it. We just really have a hard time in knowing what the amount of change would be. Like if we are doing a pricing test and raise prices by 25%, how are you suppose to know what your _effect size_ is/will be?
@Affine but you have to take the interactions in consideration.
@COOLSerdash The var of a symmetric triangular distribution with mean 0 and upper limit 1 is 1/6. So to get variance matching, you'll have a factor involving $\\sqrt{6}$ coming in.
Just keep that statistic in the reducer. It's fairly cheap to track, like one double extra for each node and cluster.
Can you provide a sample of what trainx and trainy look like, as well as the problem?  Also, what do you mean that only one of the values is fit?
However now we're somewhat making a compromise between the two worlds using data from the first and using it in the second and you're right that it's tricky. But intuition merges the binning (our information) and reconstruction (histogram/piecewise constant) into a single step, I'm just trying to say that the binning information can be successfully projected on a better space. However to make this clear full details are needed, I can prepare a short note on that.
Am new myself, and don't know how to upload files. Could you upload them on docs.google.com, and share the link here? What are $x$ and $y$? Usually its the other way around: the variables are prices and the weights volumes, or frequencies.
Can you edit your question to include a sample of your data to make a reproducible example?
@Heinrich I work for a kind of academic data center, so I don't have such problems (-; Seriously, in Warsaw the scientific CPU time supply is larger than demand, so I believe it is quite easy to get a grant. And I think you should try D-Grid or EGEE, my experience is that grids in general are very underused.
$X_i$ is not the treatment variable, it is thought to be modify the treatment effect. The treatment takes place between observation of $Y_{i1}$ and $Y_{i2}$. How does that change your answer?
Since no one has answered this I would suggest you ask this on one of the SPSS specific forums. Check out the [SPSS tag wiki](http://stats.stackexchange.com/tags/spss/info) for some references.
On the contrary, your response was very helpful!
Why'd someone vote this question down?  Why not try to help develop a better question or answer?
tslm function in forecast package does this automatically. It is wrapper of lm function for time series problems that creates trend and seasonal index automatically. You could try tslm function
**Please** [don't](http://meta.stackexchange.com/questions/64068/is-cross-posting-a-question-on-multiple-stack-exchange-sites-permitted-if-the-qu) crosspost: http://math.stackexchange.com/questions/104003/convergence-in-probability-of-frac1n-sumn-x-i-barxq-to-ex-ex
yeah thats the mathematical way to explain these two parameters, BUT whats the logical explenation? Why do I really ned two parameters to show the same thing(the deviation around the arithmetical mean)...
Terminology like "gender affects the relationship" may be confusing you.  With very few exceptions, people do not change their gender (and when they do, I doubt it affects their research-purchasing patterns).  What you seem to want to know is "how does the relationship between X and Y differ by gender?"  The *very first* thing to do is to make scatterplots of Y against X broken down by gender and compare them.  What you do next depends on the aims of your research.  For many applications, you might just stop at characterizing the two scatterplots.
My classifier classifies faces into positive or negative emotion. I ran a couple of classification algorithms with 10 fold cross validation and I even get 100% recall sometimes, though the precision is for all the classifiers almost the same (around 65%). I work with an imbalanced dataset (majority class has twice the amount of samples of the minority class). I therefore thought this probably meant my models weren't learning the pattern.
By the link you provide I conclude that you speak Russian. Once I gave a concise answer exactly to the question - how does SPSS classifies cases in DA - [elsewhere](http://ru-spss.livejournal.com/176118.html). Maybe you find it worth reading.
I think it indirectly but appropriately addresses the apparent intent of the question, @Andy.  The fault, if any, lies in the question itself, which is ambiguous because it does not specify the sense in which a mean might be "worse" than a median.  A good answer therefore has to explore this and consider the purpose of summarizing an age distribution with a single statistic.  Here, this leads naturally to a discussion of what an "age" might mean and how appropriately to compare age distributions.
I mean running my analysis with the missing value dataset.
@user2040, I've replicated [SAS ridge regression](http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_reg_sect046.htm) on R with your data. So we know for sure that the algorithms are different.
When I ran @Peter's code several times I sometimes got a pos. coeff. for centx and sometimes a neg.  The fact that y ranged from about -25 to 5, whereas x was always pos., seems to help explain why the centx coeff. was sometimes neg.
Hi @doug.numbers. I have uploaded my question explaining why I want to remove outliers for age.
Thanks. the Scott B. Morris' article is just what I was looking for. And yes, I agree that I should provide an explanation of the calculation (perhaps I call it something like $\\hat{d}$).
I don't think its a question of whether or not one method is better than the other, but more so do you believe in the Bayesian paradigm (i.e., do you believe in the notion of prior information or not).  Both Bayesian and non-Bayesian (Frequentist) regression will work but the choice of which to use is more fundamental to your beliefs in statistics.
Yes, you are correct about $f_A$ and $F_A$. If you add up the integrals you will be integrating the derivative of $f_{\\max(A,B,C)}$ which works out to $F_A(1)F_B(1)F_C(1) - F_A(0)F_B(0)F_C(0)$ = $1-0$ = $1$ as expected. In your example (with uniform distributions) the chance that $B$ is largest equals $5627/8910$ and the chance that $C$ is largest is $3266/8910$; all three chances add to $1$.
This question is too vague to be answerable: exactly what kind of help is being requested? If it's help with Matlab, the question belongs on [SO] along with a more specific indication of the nature of the help sought; if it's help with statistics, the question needs substantial amplification.