tttjjj/elt_bench huggingface dataset:
Wow. That is SO MUCH EASIER. From the bottom of the heart of a crazed dissertating doctoral candidate...THANK YOU!
An explicit algorithm appears at the bottom of p. 4 of http://saba.kntu.ac.ir/eecd/people/aliyari/NN%20%20files/rls.pdf .  This can be found by Googling "recursive least squares."  It does not look like an improvement on the Gentleman/Maindonald approach, but at least it is clearly and explicitly described.
Hi Peter, the data is for traffic and not spend
it is definitely a valid model. I have code this up in R before, but sorry, can't help with Stata.
You are right, i've looked to evaluated data and seen that contribution of gaussian kernel for majority of points is close to zero. I will ommit these points. Thank you very much.
Prediction about the past can also be surprisingly tricky!
Is there an easy way to get a confidence interval if the Wilcoxon test is used? It seems to encourage people to put too much emphasis on the p-value, even more than they would with a parametric method.
A slightly different problem is with unbounded likelihoods: A paper looking into this problem is:  http://arno.uvt.nl/show.cgi?fid=3533;h=repec:dgr:kubcen:19978
Ah yes, I almost forgot what I was trying to do and just focusing on the equation. I understand now
Welcome to the site, @ProfRoy47. Would you mind elaborating on this post somewhat? It's not clear that this is actually an answer to the OP's question yet / that it quite stands on it's own. OTOH, it wouldn't fit as a comment, & I think it has the makings of a helpful contribution to this thread. Our [FAQ](http://stats.stackexchange.com/faq) has some discussion re providing answers on CV, which may be helpful to you.
Why below 0.05, not 0.04973? Statistical significance does not have any in-depth meaning, it is just an accepted probability of analysis failure.  The operation of transforming statistic into a p-value is monotonic, so there is no problem with comparison. Still obtaining a significance level of this comparison is problematic (I have no better idea than bootstrap).
I'm sure [Dr. LeCun](http://yann.lecun.com/) would be honored to see your username homage.
@Macro: I made edits in the original post to clarify these two points.
As you mentioned it seems that I am attempting to update P(A|B1) with P(A|B2), P(A|B3)...P(A|Bn). If you would prefer we can take this discussion off of comments via e-mail. AndrewKemendo@hotmail.com
Welcome to SO. Your question is actually better suited for http://www.crossvalidated.com . Probably one of the mods will move it there soon enough. Next to that, I'd advise you to add some toy dataset and a reproducible example so people can play around a bit as well. It shows you did some effort, and people will be more keen to help you. On how to make a reproducible example, see [this question](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example)
Do you have success with it? If yes, for what kind of application?
+1--but you don't need to restrict points to a grid.  In many spatial applications you don't have this luxury, so it's essential that the theory and concepts also apply to "irregular point patterns."
@Glen_b Yes, that would be interval estimators. Regarding these intervals: "You must use the t-distribution table when working problems when the population standard deviation (σ) is not known and the sample size is small (n<30)" (from web.pdx.edu/~stipakb/download/PA551/NormalVersusTdistribution.doc). Why don't people use the T-distribution all the time when the population standard deviation is not known (even when n>30)?
Thank you for this explanation! May I ask for more...? So for the GOF; the higher the better, right? Is this number related to the percentage of explained variance? As for the eigenvalues, is it suitable to do an analysis with k=10, plot the eigenvalues, and look for the 'cut-off' point as one would do for PCA? Sorry for all these questions...
It is only a global constraint, not a response.  Supervised learning techniques need a "response" of some sort for each individual - i.e. to know what class they should be in, or some score that can act as a proxy.  A global constraint doesn't contain any information on the individuals - as you found, random will happily satisfy the constraint.  It seems your hidden criteria is actually that people in the same group should be "similar".  This makes it a classic unsupervised classification problem.
A similar technique proceeds by computing the $\\phi$ statistic of the contingency table. Since $\\phi$ is actually the correlation coefficient of two 0-1 vectors (one based on prediction, the other on reality), one can use the Fisher R-Z transform (just $\\atanh$) to get an approximately normal statistic. Comparing $\\atanh{(\\phi_1)}$ to $\\atanh{(\\phi_2)}$ becomes an exercise in comparing independent normals with known variances. A similar technique for tetrachoric coefficient probably exists as well. Regrettably, these techniques are fairly low power...
I would close this question as off-topic because it's better suited for the philosophy SE, but apparently I can't. Nonetheless, I encourage you to look at previous posts on [Bayesian probability](http://philosophy.stackexchange.com/search?q=bayesian) there and see how you might migrate your question for that audience.
Ahh I get it, I believe. As Currie and Yellowitz write: "We control for observable characteristics".. so controlling means simply here "to include" in the OLS regression to isolate the causing variable. Alternatively, when you claim that you "control for unobservable fixed effects" you can add dummies or do panel etc.  Is that right?
FWIW: I've determined that `chisquare` in Python is a one-way test that the categorical data has the given frequencies, and is invalid when observed or expected frequencies are below 5 (as is typically the case). In this case, it makes sense to use R rather than Python.
Thanks, looks pretty complicated, I'll run it by some people who might be able to understand it better.
@phil12 Yes - you might want to read something like [this](http://data.princeton.edu/wws509/notes/c5.pdf) about equivalencies between Poisson, binomial, & multinomial models applied to contingency tables.
+1 More simply, $E[XY^2]=E[(-X)(-Y)^2]=-E[XY^2]$ follows from the symmetry $(X,Y)\\to (-X,-Y)$ and immediately implies the expectation is zero.
There are many alternatives, even within `pairwise.t.test`. Type ?pairwise.t.test to see them.  Your choice (`fdr` or false discovery rate) applies a different method of multiple comparison control than `tukey`. There is lots of literature on choosing a method, but, as far as I know, no general consensus on which is best.
Thanks for the complete clear answer, David
+1 This is a nice list.  You might consider "accepting this" so that it's always on top; given that it's CW, anyone can keep it updated.
What exactly is this Australian calculator meant to do? Personally I've never heard of it. Being more specific about the problem and this method might help to attract potential answers.
@gung:  Ah, I didn't even try that. Will do now. Thanks!
That link doesn't say how many games he simulated. How do you know what the $n$ is? Also note that the 38% he mentions is *not the standard deviation*.
@YanRaf yes! I'll add an example in the answer in a moment.
Can you add a screenshot or cut and past the results? It could mean anything from "You aren't looking in the right place" to "It couldn't be calculated, which means something is very, very wrong". We can't really know without more information.
But only if you do a two-level approach, first classify by the clusters, then evaluate the cluster classifier. Otherwise, the constant classifier is useless. Then you are putting all the burden to the clustering.
Please register & merge your accounts, @ellen. There is information on that [here](http://stats.stackexchange.com/help/merging-accounts). Then you will be notified if something happens on your question & you will be able to respond to comments. Please do not use the "Your Answer" field to respond to comments, as that is only for answers.
(+1) Nice answer. Welcome to CrossValidated, Paul!
Linear models and ANOVA will be equivalent when the models are testing the same hypotheses and when the parameterization of the factors is equivalent. So called "Type I" and "Type III" sums are squares are simply tests of different underlying hypotheses (effects of sequential sums of squares versus marginal sums of squares). ANOVA tends to hide some of these decisions as implemented in many packages--a fact that makes me believe that actually setting up and testing the hypotheses of interest through factor parameterization and model comparison in GLM is a superior approach.
you have asked several similar questions, this one I think is perfectly answered [here](http://stats.stackexchange.com/questions/13166/rs-lmer-cheat-sheet/13173#13173). Please ask either to merge the questions or delete them.
@NickStauner OP mentioned that the results come from merged data, so there should be a 1-1 correspondence between observations. However, the question as stated does not suggest that these data are paired outcome data, but more a traditional predictor/outcome relationship. I too was confused, which is why I prefaced my answer with a nod to agreement as a possible secondary measure. In that case, a kappa or weighted kappa would be preferable. Personally, I disagree rank statistics methods, I think their hypotheses are difficult to state and are often of less power than traditional tests.
Each image is of one DGGE gel. The gel consists of 20 lanes and each lane contains an amount of extracted and amplified bacterial DNA, which may be of several different bacterial genera from a milk sample. In each lane therefore there are (depending upon the gel image) between 5-50 bands, usually in quite close proximity. With Gelcompar, I define the lanes of each gel, remove background noise and then the programme produces for each gel a densitometric curve which essentially measures the optical density across the image to define where bands are. Bands are also quantified by the programme.
"A lognormal distribution with an interval of [-1, +inf]" is strange. You should be taking the logarithms of *ratios* of prices, not differences, with resulting values in the range $(-\\infty, \\infty)$.  The ratios are easily recovered from the percent changes; a percent change of $i$ means the new price is $1+i/100$ times the old, so the variable to consider using is $\\log(1+i/100)$. For small percents (positive or negative) this is close to $i/100$, so it's easy to interpret.
No, actually, it assumes that the data is distributed in whatever proportion, between the 7 points, that you tell it. You control the relative distribution through the input values in the $x$ and $pct$ variables. As for issues/assumptions: well, your are interpolating, which means at some level, you are filling in unknown values with an "educated guess", based on other nearby values. You'll get the best results by using input data which is as tightly spaced as possible--if your percentiles are spaced in 5% increments, that will be more accurate than an interpolation based on 20% increments.
Actually, mat, sorry, it just occurred to me that your data are paired. You don't want that you want a McNemar test.
Caracal:  I misread your comment earlier; I thought you understood the role of a sign test in a single sample test.  Now, I'm not sure you do - so let me expand: a permutation test of dependent samples is the same as the permutation test of a single sample (just as a paired samples t-test is equivalent to a single sample test of the difference scores).  Thus, you can use a sign test on single sample data even though it doesn't look like the sort of permutation you are used to thinking about.
I use FDA and am not sure how it applies. I could set x axis = 0 for each index child, and have other siblings plotted at +/- time on that axis; and have maternal age as the y axis; and then estimate a monotonic functional data object for each index child...but as soon as I've done this I've lost all information about the actual numbers of siblings and where they fall along the line (because everything is now summarized by the function). FDA includes a number of modes of analysis - did you have something more specific in mind?
Cochran's Q is so to speak "repeated-measures ANOVA" for dichotomous data, while McNemar's is like "paired t-test" for such data. So, normally one wouldn't do the second if the first is not significant and he decided to accept this.
If you're after predictions of survival probability, the function `survest` [`survival`-package] (or predictSurvProb [`pec`-package]) is very useful for this.
There's also http://biology.stackexchange.com/. @DouglasZare: presumably not, because it if was recessive it wouldn't be so deleterious, and would be able to survive longer. Also, actual type of negative effect would be important - if the effect causes explosive deaths, but only after the age of 40, the gene still has a fair chance to be passed on. If it causes such an effect before puberty, it certainly won't last long. I have a feeling that this question needs more rigorous specification to be answered well.
@Dmitrij By definition, projective space in dimension $m$ consists of equivalence classes of $m+1$-vectors $\\mathbf{y}$ modulo the relation $\\mathbf{y} \\sim c \\cdot \\mathbf{y}$ for any *scalar* $c$.  The question therefore says that any estimator $\\hat{f}(\\mathbf{x}) = c \\cdot \\mathbf{y}$ should be considered perfectly accurate.  The challenge is to find some measure of discrepancy (which comes down to a tiny generalization of circular regression, I believe).
Depending on how your data are distributed, the 50% quantile could be much more or much less variable than the 99% quantile, so the threshold might need to be different for each quantile in addition to depending on the sample size.
Well, the interpretation would rather be that a 95% credibility intervals includes with 95% probability the true parameter (given the model assuptions, of course...). I would say that the 95% credibility interval is extremely useful information, and often the reported end result of a Bayesian analysis (If you do not choose to look at the complete posterior distribution, that is).
I think that would restrict heterogeneity to be orthogonal to X and Z while the requested estimator allows for it.
Please **do not crosspost** simultaneously here and on SO.
Using a 0/1 coding scheme is essentially useful when applying regression models among others, although several coding schemes are possible, e.g. -1/1 (but it will change the interpretation of the regression coefficients). It should not be confused with data entry (that is, what you really put in your database), though. In this case, it is better to store the full labels. Convert them to numerical values or build a dedicated design matrix when you build your regression model. Otherwise, I wish you good luck to tell what the 0 and 1's stand for in 5 years.
+1 This of course is an application of the Principle of Inclusion-Exclusion.  You can alternatively begin with the easily-obtained recursion and use standard methods (of generating functions and solving differential equations) to obtain the second formula directly.
Can you give a little more detail on what you are trying to do?
Assuming that $X$ and $Y$ are _jointly_ normal (as I included in my edit), vinux's hint is the way to go. Given the value of $X$, $Y$ is a normal random variable whose mean and variance are known, and so $$E[Z\\mid X] = E[XY^2\\mid X] = XE[Y^2\\mid X] = X(\\sigma_{Y\\mid X}^2 + \\mu_{Y\\mid X}^2 = g(X)$$ where $g(X)$ is a cubic in $X$ and so only the quadratic and constant term matter: $E[X] = E[X^3] = 0$ in this case.
Do you think you could impose a meaningful loss function? Is there a way to say how much you gain/lose by guessing correctly/incorrectly?
Statistically there is not much evident here beyond an interest in modelling "resource" predictively. Evidently resource can be zero or positive. That really doesn't narrow things down much. Your question may need much more detail on your statistical needs to get more attention. Precisely what data do you have? Conversely, anything with the flavour how could I build a model? is far too broad to get a precise and useful answer.
Apparently "$k$" is a *random variable* referring to the number of successes and you are modeling it with a Binomial$(100, \\hat{p})$ distribution.  What is the mean of that distribution?  What is its variance?  How are these related to $k/n$?
I have Pearl's book.
This implies that interpolation is based on polynomials, but there are several other methods, such as cubic spline, piecewise cubic Hermite, nearest neighbour, etc.
The *first* order statistic is conventionally the minimum. The $n^\\text{th}$ order statistic is conventionally the maximum. See, for example, [here](http://en.wikipedia.org/wiki/Order_statistic#Notation_and_examples)
This follows directly from the *definition* of convergence in probability (and monotonicity of an arbitrary measure).
Are your sure that Q1[i,r] > Q1[i,(r-1)] in SET d1 at initialization?  (Just speculating...)
@whuber I don't understand.  If an ID appears 3 times, there will be a "3" in column B alongside each instance of that ID.
This is also an unusual use of the term. Could you provide some examples of it being used in this context?

I'm mostly surprised because one often hears closed-form/no closed form regarding integrals, which don't really have an outcome or predictors.
@act00 Fixed for your sample! Please, have a look.
There is also the nice (free) book by Gaston Sanchez on PLS path modeling: http://gastonsanchez.com/work/
I do this to some extent already, but unfortunately, I don't think I'd be able to get rid of all the typos. Perhaps it would be better to make the features that I generate for each token be more robust against small errors. Thanks for the feedback!
Thanks for the answer. But it considers the univariate case. Generalizing this answer to the multivariate case doesn't seem straightforward to me. Is it?
:dail No. For example I was referring to he case when you have predictor series such as volume to predict price and you include the effect of today's volume on today's price but omit the impact of yesterday's volume reflecting the importance of changes in volume on price.
I just added a picture of the question, graph included. I marked the correct answer as well.
@MajidalDosari it might be worth sitting down with a quantitative modeler and talk about approaches over warm caffeinated beverage.
To get some insight into why the answers must be negative, suppose $T(X,Y)=X$ and $X$ is nondegenerate.  Then the cross products consist of $M$ replications of each $X_i$, which when $M\\gt 1$ obviously is not a random sample of anything.
This question appears to be off-topic because it is about obtaining contrasting text, rather than statistics.
The help asks that you avoid crossposting; third-last paragraph [here](http://stats.stackexchange.com/help/on-topic)
No, you know an exact expression for $y_t$, not its expectation. It's written in your question. Replace $y_t$ with something else in the covariance, the same way you did when you worked out the variance.
@Alan - Do you know how to generate random values from your Null Distribution? If yes, suppose - T= c(T1, ..., TN) are draws from the null distribution - p-value = sum(T > T_obs)/N. If you don't know how to generate, may you need to use Metropolis Sampling or Gibbs Sampling to get T1...TN, but it is very do-able.
Neat... I should replicate this in R, if not already availlable.
+1 For a thoughtful analysis and attempt to respond directly to the OP.
@RamSharma And the Sidak adjustment, the Holm stepwise adjustment, the and the Holm-Sidak stepwise adjustment, the last being the most powerful of the methods listed between your and my FWER methods. :)
Attempting to krige a dataset of just $10$ points is a lost cause unless you have strong prior information about the variogram. (Rules of thumb in the literature range from $30$ to $100$ points as reasonably minima.) Using an automatic kriging routine for *any* dataset is just foolhardy. Nevertheless, this one seems to have done a good job: the variogram is reasonable. As a result, the map is a *complete artifact* of whatever neighborhood search method the program chose. One moral to draw from this experience is that you tend to get out of statistical procedures what you put into them.
From your question it may seem that you think that factor analysis = PCA + rotation. Do you?
Bayesian models are certainly possible in forecasting. Essentially, you posit some model and describe the parameters by prior distributions, which will then be updated by the observations. However, I think it would be helpful to first understand "vanilla" forecasting on the one hand and "standard" Bayesian statistics (without time series-related complications) on the other hand before delving into the combination of the two.
Welcome to the site. I fixed up your grammar and spelling. I also removed the signature since the site adds one automatically.
Thanks for your answer. Besides, I want to share something in my mind. When thinking about pilot study in social research; respondents are from the same population in that research. If all the sampling procedures (in pilot study and in main part) are random than the data we have in the pilot study phase represent the research sample (and the population behind that sample). Because the sample in the pilot study is from the research population (have the same characteristics); I think, based on these data, using a weighted-sum-model scale can be more valuable. Again thanks.
I've wrongly described what I did. In fact, I'm taking the square root of the eigenvalues. I've updated the question.
Here is a paper that proposes adding noise to input data and shows improved classification results due to the added noise: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=90C06480E22D8E85A911295D1B717228?doi=10.1.1.7.1266&rep=rep1&type=pdf
It sounds like you have a pair of calibration curves generated by two different methods. If that is correct then you might want to look at the fixed and proportional bias in the measurements using the least products regression method detailed in this paper:
http://www.molecularlab.it/public/data/TMax/2009522103349_Comparing%20Methods%20of%20Measurement%20by%20Ludbrook.pdf
Question is not clear. What do you mean by "standard error for a vector"? Typically we want the standard error of a parameter estimate. What parameter are you estimating?  The code you provided produces the Newey West estimate of the squared standard error of the mean. Is that what you want?
Welcome to the site @IvanKshnyasev. I wonder if this will be sufficiently self-explanatory to be useful for future readers, would you mind explicating this a bit?
Missing from this question is the essential piece of information: *what do you want to learn from the simulation?*  For examples of how people have used $H$ in contexts that may be similar to yours, please see http://stats.stackexchange.com/questions/49117, http://stats.stackexchange.com/questions/1841, and especially http://stats.stackexchange.com/questions/25235.
@StasK: thanks for your comments, this is valuable input. The action is already known: many of these builds take waaaay too long, and we should work on making them faster. I want to use the aggregate measure to see if things are getting better or worse.
Level of measurements is ratio from 0.4 to 0.7. Group C is different concentration of one treatment which affects the G of overall wall. Technically speaking, C indeed is continuos. But if I would treat it as such I have to examine effect more precise (like lesser increments in concentrations, or even different classes of C at all). So, I would assume it as a pilot for more thorough investigation and treat it at the moment just like “No treatment” “High concentration” “Low concentration”. I have uploaded the graphs I’ve tried to make and added what I wanted to say about B3(main interest).