tttjjj/elt_bench huggingface dataset:
This follows by expanding the first expression $\\frac{\\partial}{\\partial a}\\int_{-\\infty}^a (a-\\theta)  f(\\theta\\vert y)  \\mathrm{d}\\theta = \\frac{\\partial}{\\partial a}a\\int_{-\\infty}^af(\\theta\\vert y)d\\theta-\\frac{\\partial}{\\partial a}\\int_{-\\infty}^a \\theta f(\\theta \\vert y)d\\theta = \\frac{\\partial}{\\partial a} aF(a\\vert y)-af(a\\vert y)=F(a\\vert y)={\\mathbb P}(\\theta\\leq a\\vert y).$
For fixed $n$ it suffices to maximize $\\mathbb{E}(X^2)$, which makes it look a bit simpler. (Is this for some subject?) Note that if you want to maximize, you're probably going to want to take the derivative with respect to the parameters ... in which case it might even be possible to avoid actually doing the integral, or failing that, interchanging the order of differentiation and integration may yield a more tractable problem.
The new product is in its initial stage and so there may not be time enough to form an opinion; four options are not good for this case.
@JacobMick: Thank you for the comment. In my application, I am looking at the survival probability of single chemical bonds. Since bond dissociation is a first order reaction if decays as a single exponential.
Intuitively I would say, calculating a parameter using a sample from a random variable, the randomness flows into your parameter. If you draw another sample from the same variable, you would most likely get a slightly different estimate, from a new sample another one, and so on...
Referring to the question's comment thread, when $n$ is very large so that $t/n$ is very small (but positive), we can characterize $\\lambda\\frac{t}{n} -  F\\left(\\frac{t}{n}\\right)$ as the discrepancy between $F$ and a Uniform$(\\lambda)$ distribution near $0$.  Thus this argument is simply a semi-rigorous restatement of the intuition expressed in those comments. (For even more rigor one would write a full epsilon-delta limiting argument, but it should now be obvious how that proof would go.)
I prefer books and articles, but thanks for suggestion, will look at it
Thank you for the suggestions. Based on a cursory view of the Wikipedia pages, this might be just what I need!
This answer is simply wrong. There may be other errors, but this one is glaring. You state a definitive answer for P("Has Long Hair") (your P(A)), and then use that to give your final definitive answer. There simply isn't enough information to determine this, even assuming P(F) = 0.5. Your line to calculate P(A) seems to come from nowhere. Here is the correct formula using Bayes theroem: P(A) = P(A|F)P(F)/P(F|A) from which, using your stated assumptions, get to P(A) = P(A|F)*5/9. However we still don't know P(A|F), which could be anything.
I thing that the only valid way to see if an effect that appear between group and outcome is due to a demographic variable is to put this variable in the ANOVA model. Thus, a simple (although not methodologically right) approach is to run Analyze > General Linear Model > Univariate, put the outcome in the cell Dependent Variable, put Group variable in the Fixed Factor(s) cell and all demographic variables you believe that interact with the outcome in Fixed Factor(s), Random Factor(s) and Covariate(s) cells according to their properties. Of course you should be careful for the data assumptions!
Someone asked me this, so I am posting both the question and my answer.
@IrishStat I am still working on this problem at the moment but am just not making any headway. What further information would you need to help point me in the right direction methodology wise? and software wise? Given the time-frames this is really now just a personal project because I intuitively know it should be possible and am annoyed that I haven't been able to work it out.
No, I don't think there is a way to do this.  I also should declare I'm in the anti Type III camp... there's lots on this topic in the various R archives if you want to search for it.
I don't think the graph explains things well. I also don't think there is any hypothesis to test here.
The data is not multinomial in the usual sense. There cases in the training set are such that the number and labels of the classes vary from examples to example. Also the covariates have a special structure. I altered the description of the problem above.
If there's a "dependent variable," you might be contemplating regression.  In that situation the distribution of the dependent variable is almost irrelevant: what matters is the distribution of the *residuals.*
Thanks, this plot helps a lot. Together with my thoughts, I think I understand it now.
Try a lesser learning rate (say 0.001) and a higher momentum (say 0.99). This will make training take longer but it will also be more robust.
(+1) Re: your last paragraph: Except of course for software systems meant to be used in-vehicle during operation, as found on many police vehicles and other industrial vehicles like in construction and the military. Interestingly enough, those hardware/software solutions often *do* look like dashboards. Touch screens, big buttons and judicious use of (high contrast) color play a very important role in usability and safety. These interfaces tend to look like anything *but* your typical GUI.
I would say it depends on what inference you want to do with it.  My understanding is that it is basically an estimation technique rather than a type of model.  So there are no necessary assumption as such unless you are inferring to a data generating process or population.  Similarly, OLS has no "assumptions" as such.  Although the type of inference it is often associated with assumes constant variance, normality, etc., there is nothing stopping you using OLS without those assumptions applying - it's just a technique for fitting a line.  Does this not apply to quantile regression too?
@PeterFlom No This is not homework. I had a  doubt and wanted to clarify it.
I don't know if the estimates differ by a lot since I still haven't tried it out, I'm working on the theory before coding it since I still don't have the cluster. If I use batch GD, the parameters won't be updated until it has finished reading the whole dataset. Therefore, I would need to run several mapreduce jobs, however if I use SGD I could get the parameters in a single run
@PeterEllis I have to cluster all of these elements together. I believe that combining all of these matrices together and performing a clustering analysis would be more acceptable than clustering each of matrices and then combining them. Although I am open to suggestions.
Without some kind of reasoning or working, your 'attempt' looks like a guess.
"Orthogonal regression" is another thing one might encounter when looking for methods to deal with contaminated abscissas and ordinates.
Google for novelty detection, and review papers on the topic. There are tons of examples: fraud detection, faults in the production of a product,...
I suspect this good reply hasn't been upvoted more because it doesn't sufficiently emphasize the important point that *the sampling distribution of the K-S statistic has been computed* for all continuous distributions (because they become uniform under the probability transformation) while each discrete distribution leads to its own idiosyncratic distribution of the K-S statistic.
I simply want to understand the average distance between sequences in one dataset, and compare that with another - so that I can say that "the sequences in dataset A are structurally more similar to each other than the sequences within dataset B are to each other".
What do you mean by "round"?
Not necessarily - what is your simulation changes dramatically if the result is say, 1.5 standard deviations from the mean? A likely answer, but not one that might have shown up in 30 realizations- also, what do you mean by "noticeable". I've put some more detail in my answer.
And "The proportion of A genes is P = p + (q/2)" has the same reason
@user2300643 No problem!  I'm glad you're using the package.
@Skull: Or more trivially...because it is the square of a real number. :-)
What exactly do you want to use this measure to do? Would mean square error work?
Just to add on to @Karl's answer, I would pair that with a Bonferroni/false discovery rate correction to account for the multiple comparisons.
The `+cluster(ChartNo)` term should take care of the repeated observations concern. An alternate approach would be to add `+ (1|subject)` to a coxme::coxme analysis.
@chl Yes, but it seems to OP is trying to infer an underlying probability of success. The tests compare the observed proportions to the theoretical distribution to determine if there is a difference for a given level of confidence. You are testing for any form of randomness, not just sampling error randomness.
@Glen_b My bad! I used to use non-parametric, and something I read had me feeling I was making  a mistake in doing so. Potato, pota... wait that doesn't work in the written word. :) (revised now, too).
@Jyotirmoy About bayesian vs. frequentist approaches, interesting points were made here: http://stats.stackexchange.com/questions/1611/do-working-statisticians-care-about-the-difference-between-frequentist-and-bayesi/3370#3370
+1 because this answer *is* useful, but I think your points about R's support, speed, and ability to handle large data are out of date or becoming so fairly quickly.
Random forests does not do as good a job as continuous models, but by combining many trees it approximates continuous effects with step functions - better than single tree methods can do.  Continuous methods are better for this.  Monotonic/non-monotonic is not reallly the issue.  Adding more splits in the way I think you describe will increase variance which decreases the probability of predicting close to the "truth".
@chi Oh dear, I think -sizefx- contains a coding error! It's ignoring the values of one variable for which the other variable is missing (the problem is with the use of -marksample-). I'll contact the author. I've never used it before myself (i'm relieved to say...)
This needs more context for a good answer, but does this make sense: http://mathworld.wolfram.com/Metric.html and http://mathworld.wolfram.com/Distance.html. Here's an example of a metric that seems different than what we usually mean by distance: http://mathworld.wolfram.com/TaxicabMetric.html
When it comes to knowing whether measurement is valid, there is a lot to be said for domain specific experience. This allows you to form expectations about what the data should look like, and thus, you become more attuned to knowing when the data is deviating from your expectations. You may wish to elaborate more on the details of your specific measurement domain as I imagine most answers regarding assessing validity of data would be contingent on the domain.
BTW, does anyone have a reference for this approach?
There were both exclusion and inclusion criteria same for both groups except interms of taking intervention and non going for intervention. Can i use parametric statistics for a sample of 20. If so please help me finding reference of that. One more thing if parametric statistics are used which are more appropriate for this method. T or Anova or Ancova? or any other.
Doesn't matter. Rename $X_1,X_2,\\ldots$ to $U_1,U_2,\\ldots$ in my post. I only use my $X$s to define a random variable $Y_n$ whose distribution is $\\chi^2(n)$.
Several language have equivalent constructs. For instance PHP and Javascript have null and NaN.
@Simone I didn't try what I proposed :( There are specific metric for binary variables (like Jaccard), but I guess it would be better to use directly your measure of sparsity which, as you suggested, is variable-specific. I was partly thinking of filtering methods used in genomic studies (but see the [Bioconductor](http://www.bioconductor.org) project), where genes whose level of expression is somewhat unreliable are filtered during clustering, often in an iterative manner, as described in e.g., [Microarray learning with ABC](http://bit.ly/eX7uhx). But most basic filters remain univariate.
a) Beware notation! If $X\\sim G$ the distribution of $\\log(X)$ is **not** $\\log G$! .  b) If you take the log of your random variable, the errors will be additive. Work on that log scale with the nice additive normal errors. You then have a problem akin to deconvolution (if you know the variance of the additive noise, it's even more like one). The resulting distribution of the log-observations would be the convolution of the original (without-error) distribution of the logs and the normal log-errors. If you want to use the sample to estimate all the unknown parameters of both, it will be hard.
@chl: not a problem. I like that you posted R code (I am slowly learning R); my version is in homebrew Matlab, which I cannot share. Having code that the OP can test is definitely the way to go.
@Jane If the added context makes my answer irrelevant I will delete my answer or perhaps edit to fit your context better.
@alex yes, that is correct.
I would say that this is in Fisher itself, see [here](http://books.google.com/books?id=3cGefthoP9gC&pg=PA106&dq=profile+likelihood+fisher&hl=en&sa=X&ei=qd6pUPLVB8O8rAfSkYGwBQ&redir_esc=y#v=onepage&q=profile%20likelihood%20fisher&f=false).
Thanks again for your answer. Then do you think there is a relationship between `PP.test(x)` and `kpss.test(diff(x))`? I am thinking that `PP.test(x)` is testing difference stationary. That is also what 'kpss.test(diff(x))' does.
"note that there is no parallel implementation of RF in R". That is only partially correct, as the `foreach` package plays nicely with the `randomForest` package. I think there is one such example in the vignette for `foreach`. (Or maybe `doMC`.)
On that note, I encourage you to keep asking question as you encounter specific issues in your game development endeavors. As Phil mentions sometimes the math might get out of scope for this site and [stats.se] or [math.se] might be better places to ask, but most specific questions regarding rpg development are on topic here and we'd love to help!
Robust standard errors (a good idea) would actually do the averaging you suggest automatically. Also, if we have a half-decent guess at what the correlation structure across time points is - GEE might be a good option.
If you remove the items with no variance does your model work?  If you could post some more details, we could help troubleshoot.  Generally, as long as an item has variance, `lavaan` and `sem` will work.  However, conceptually, items with restriction of range can be problematic.  I don't have any references off-hand, but there is a section in Loehlin, J. C. (2003) that I will find and post tomorrow that discusses the problem.  Also, have you tried using `score.items()` to examine the reliability of the item with the scale?  Items with reduced range may not correlate with your factor well.
There's more to it than that, which is why I left my remark as a comment: the interpretation of $\\bar{x}$ is a little delicate when data have been rounded (or otherwise binned) to the point of making $s=0.$ In addition to using proxies for $s$ based on measurement error or bin width, one can also apply methods of [interval arithmetic](http://en.wikipedia.org/wiki/Interval_arithmetic) to this problem.
Please spell out the acronyms
This is something like what I was thinking too. Probably should add something about the computation of cosine ($cos\\theta=\\frac {a\\cdot b} {\\left\\|a\\right\\|\\left\\|b\\right\\|} $)...
@Procrastinator But the principle of parsimony is totally unrelated with the principle of how to obtain the optimal parameter estimates.
Well, mate, basically you are asking on the internet to give you a way of making essentially infinite amount of money. Think about it :)  Also questions which are so broad are discouraged as it would take a PhD dissertation or a few books to actually answer it.
@Jase I do not have an answer. What I usually do in practice is to adjust the parameter which decides the minimum number of instances in a node. If there is plenty of data, then is no problem to have a large value for this minimum. If it is not, then I try to take a look at my data and understand from the underlying distribution which would be a proper value for this parameter.
I imagine that one could automate that by trying to estimate a confidence interval for the variance estimation, but I never tried to work on that.
A standard approach seems to me to use a Kalman filter, extending the state vector to account for the correlation in the noise. You may want to look, for instance, Gibbs, B. Advanced Kalman Filtering, Least-Squares and Modelling, Wiley, 2011. You have non-linearities, though, which may complicate things a bit.
problem is, this Deming function doesn't support fixing the intercept at 0
@RussLenth OP's question seems to be the source of the confusion. By referring to the width of the CI as the main problem, they seem to indicate that they are seeking precision variables. It suggests that perhaps they are interested in adjusting for precision (or prognostic) variables.
(1) Please don't cross-post. Pick a site. If you change your mind about the best location, ask for the original post to be moved. (2) your first question is quite different from the two later (closely related) questions. They should probably be separate posts.
Two common ways to estimate parameters like $\\lambda$ in that copula would be ML or by matching the sample and the population nonparametric correlation implied by $\\lambda$ (Spearman rho or Kendall tau) and backing the corresponding $\\lambda$ out as an estimate. These are far from the only possibilities.
Buy her a vacuum; chicks love gifts like that.
I need a 2D array of weights too, though. Does lasso fit include weights? And, importantly (since I'm not familiar with this), can lasso fit be used for weighted least squares regression? I'm using this for gentleboosting in order to classify images. The features I'm using are HoG features. So each observation (or row) in $X$ will consist of many columns, i.e. many HoG features.
Looking at your results sets off another alarm - typically, it's my understanding that a Cox Proportional Hazards model doesn't estimate a baseline hazard. As such there *is* no intercept. The presence of one in your output makes me nervous.
+1, however (setting aside @NickCox's good point about time-series), I wouldn't necessarily say that sampling from 2 vectors separately is a mistake. I would say the mistake is in understanding / interpretation. Bootstrapping by sampling from 2 vectors separately can be a perfectly good way to estimate the null sampling distribution.
Can you provide a specific example of a problem you would like to solve?
1.Fisher Jr. WP. Rating Scale Instrument Quality Criteria. Rasch Meas Trans. 2007;21(1):1095. 2.	Linacre J. Optimizing Rating Scale Category Effectiveness. J Appl Meas. 2002;3(1):85–106. There is also Best Test Design and Rasch Rating Scale Analysis, both by Benjamin Wright.  You don't have to get too fancy if you stick with the boxplot approach.
Welcome to CV. I edited your post so the data are readable. To set off text like that, make en empty li8ne, and put 4 spaces before each line. Then it comes out with the gray background and monospace font.
Did you try using only data from the month before the account being closed? This way you ignore the account's age, but it should give you *something* to work with right now.
I think that you may need to add more information in order to get a good answer to your question. For instance, are your sequences time series?
Do you mean the bit in brackets where it says $\\sum_{j=1}^{n_i} (Y_{ij}-\\overline{Y}_i) =0$ ? If so, this is because the definition of $\\overline{Y}_i$ is $\\frac{1}{n_i} \\sum_{j=1}^{n_i} Y_{ij}$ and so you are adding up $n_i$ copies of $\\overline{Y}_i$ and then subtracting $n_i$ copies of $\\overline{Y}_i$, so you end up with zero. We can elaborate further if this is what you want. But there are no line numbers in the pdf, so I'm not sure!
@whuber, I see and agree with your point, and I don't want to disparage PCA, so definitely +1. I just wanted to point out that blindly using PCA without looking at and thinking about the underlying problem (which no one here is advocating) would leave me with a bad feeling...
@Anony-Mousse if you find an online k-means that do not require k to be chosen beforehand but that adapt it incrementally, let me know. I've not yet found such a k-means.
What is your research question; and what theory is there to help in choice of variables or point out the relevant things to check for in advance?  And what do you see as the difference between an "independent variable" and a "covariate" (because there isn't any, statistically - they will all just form explanatory variables in your model in the end).  What is your unit of analysis - school? district?  If it is school, how can you have values both for private school enrollement and charter school? If it is district, how can you have a variable for school type?
@Rajhans, based on the discussion over at metaoptimize, http://metaoptimize.com/qa/questions/11593/restricted-boltzmann-machines-or-one-hidden-layer-nns-and-markov-networks-relationship-in-inference, I see I misinterpreted your question and you were only interested in MRFs with fully observed variables.
(+1) Very nice answer, actually surpassing all those given in the linked popular thread about PCA on covariance vs. correlation.
Thank you, your suggestion regarding the t-test was enlightening and appropriate.
You can't see *anything* here because you're just plotting multiple points on top of each other. Each give it as a table instead, or jitter the points so they can all be seen. Until you fix your question, nobody can say much.
Can all fish interact more than once? While the concept of setting up fish to fight one another makes me feel a bit uneasy, from a statistical perspective repeated measures sounds like an option,  could you set up some form of tournament where winners face winners and losers face losers?
+1 beat me to the draw. (And better know what you're talking about as well.)
@zgall1 Yes, basically (or the medians). But most often, the point estimates are of less interest in bootstrapping. Often, the reason to use bootstrap is to find confidence intervals that take the sample distribution into account and don't assume any distribution (e.g. normal). In that way, the bootstrapped confidence intervals are often asymmtetric, whereas confidence intervals based on the normal or $t$-distribution are symmetric.
*Would it make sense to drop one of these variables from out analysis?* Yes.
The SD is the root of the variance. Let $X$ be the baseline weight and $Y$ the change from baseline. You need the variance of $X+Y$. It equals the variance of $X$ plus the variance of $Y$ *plus* twice the covariance of $X$ and $Y$. If there is nonzero correlation between $X$ and $Y$, that covariance will be nonzero and cannot be neglected.
By L1/L2 are you referring to the objective function or the constraints? If objective function then yes L1 error is minimized with the conditional median and L2 the conditional mean. If constraints (what ridge/lasso refer to) then this is the wrong way to think about this. Their "central measures" are still aiming for a conditional mean but with different penalties on $\\beta$.
Number of fish caught in a period of time is, I think, a more canonical example for ZIP and ZINB. Similar arguments. Albeit, fish do not have the potential for addictive effects on rate of consumption. (I know, I know, except for your mom's fish,'cause they are *that* good :).
I ran the code, and here is a line from output: r= 0.3, alpha=0.0500, power=0.800, n=84. The sample size is 84, but I'm a bit confused about r=0.3, I don't know what's the value of r, but here it's specified to be 0.3. Can you please explain this? thanks