tttjjj/elt_bench huggingface dataset:
I've voted to re-open this question. It has similarities to any network based visualization techniques (tables and variables are nodes - relationships are edges).
@Michael, for the record this question is incredibly vague (OP, help!) but I don't see what relevance $P(X|Y=1)$ (and whether or not that conditional distribution is normal) could possibly have. Blain's interpretation of the question - does $P(Y=1|X) = \\Phi( X \\beta)$ (vs. some other link function, e.g. $1/(1+e^{-X \\beta}))$? - does seem more plausible. I think the answer is that there's no way you're going to be able use a data set with $n=13$ to see whether the probit or the logistic (or some other link function) is more appropriate.
@whuber, I have edited the question and have already used google and the stackexchange search bar to find relevant entries but I couldn't.
Again I'm not convinced that a statistical test against absolutely no predictive value is of value.
@Srikant: more detailed data of birth rates (by women grouped into five year age ranges) are readily available for exactly the purpose you propose.  It's best to obtain these data for the particular state in question rather than using nationwide averages.
The problem had -E[what you had]=-E[1/(2*sigam^4)-((x-mu)^2)/sigma^6]=1/(2*sigma^4). Where did the negative sign go in the end?
+1 for the link to that excellent paper. My guess is that a definitive answer to your question would be difficult to formulate, it will depend on the data. This is why we need to search in the first place. One shortcut I use is to grab a small subset of the data, divide into train and test sets and then use Nelder-Mead to optimise the test error. This at least tells you the order of magnitude of your parameters. If you repeat this several times the spread of the resulting values also give a hint to the reasonably search range. This doesn't also work, depending on the data and algorithm.
+1 Nice illustration!
@guy Thanks for suggesting the correction. I looked up the definition more closely and figured 'missing not at random' isn't correct for this problem.

"it's worth pointing out that if you never observe Y when X exceeds a threshold then you cannot (nonparametrically) learn the marginal distribution of Y". Can you please expand on this or point me to some resources.
Also, for input to something like randomForest, does it matter if I have a single factor variable with 5 values (conjunction, opposition, sextile, trine, NOTHING) versus 4 binary values where all of the being 0 is equivalent to NOTHING? I.e., is it better to have binary versus N-value factors as predictors (X's)? Just thinking out loud.
What is the sample size? how big is the difference? what is the development-validation proportion?
I think something went awry in @Peter's edit. We should consider removing the images completely and using the built-in $\\LaTeX$ support.
Have you the information of which researcher did which measurement? Do you think that some researches do systematic errors? Or that some researchers measure more precisely than others?
@Jeromy Anglim, thanks for explanation, I've upvoted the question.
As you tagged this with `t-test`, you may be interested in this related thread: [Group differences on a five point Likert item](http://stats.stackexchange.com/q/203/930).
@Nupul: Actually, "what to do in the situation" means either they cross a river or not?
If you know the variance (or SD) then you could decide it easily. Suppose variance is 0.25 (SD=0.5) then they can cross the river safely because range of interval (dont confuse this with confidence Interval (CI)) is 3+0.5 or 3-0.5, and their heights are 4 and 5. If variance is 4 then better to not cross the river.
By the way, just enjoy jokes here http://stats.stackexchange.com/questions/1337/statistics-jokes
@Wayne - the situation is a little more general than just location scale families.  Usually a CI will be equivalent to credible interval, if it is based on a "sufficient statistic" (as these two were) where this exists.  If no sufficient statistic, then CI needs to condition on what are called "ancillary statistics" to have credible interval interpretation.
Why not compare your results in a dataset where you already know the correct clusters beforehand?  Alternatively you could compare both the algorithms with another clustering algorithm not based on FCM.
A standard remedy for residual drift like this in linear models, both fixed and mixed, is to put an additional variable into the model. You can even figure out which variable by plotting the residuals versus candidate variables instead of fitted values, and if the drift is not linear, that's a clue for what transformation of the new variable to use.
Thanks.  In the end Lumley's survey package did what I wanted once I got rid of the idea of doing a bootstrap in a loop myself and instead generated replicate weights and raked them to the original marginal total weights.  It also does a variety of jacknife replicate weights.  I also found the EVER library immensely useful - it has an implementation of Kott's delete-a-group jacknife which is a method trialled by Statistics New Zealand (in SAS) and in fact used in our own surveys (again in SAS).  I might look into ABS' GREGWT.
Isn't your proposal testing for *differences* in proportions? (The OP—"testing for equality"—may have simply been sloppy with language, but it is also possible they are groping for some kind of equivalence test).
see also [What is the meaning of p values and t values in statistical tests?](http://stats.stackexchange.com/questions/31/what-)
Thank you for your explanation! It was very helpful.
+1 The example very nicely explains the point.
"The concept of ergodicity is fundamental in the analysis of economic time series and of dynamic models calibrated by time series data. It is, therefore, surprising that no general testing procedure has been proposed to examine this important hypothesis. The objective of this paper is to fill this gap for the case of Markov processes."
@AlecosPapadopoulos: Yes, that is correct. Each realization produces a member drawn from each distribution, and the correlations between the sets for a given realization are very important.
that is awesome, thanks!
@whuber I was mostly referring to the first part of the post, but this was not clear in my comment.  What I meant was that if you only mean-center $x$, and not $y$, as it seemed was being suggested in the OP, and then remove the intercept then the fit will be worse, since its not necessarily the case that $\\bar{y}=0$. Slope is obviously not a good term for the coefficient in the model mentioned in the last line of the OP.
♦ thank you for the answer.
We generally choose a statistical test or measure of association to answer a particular question given a sampling scheme or experimental design.
@stephanKolassa, There are some articles where groups are using the polytomous Rasch model to adjust for rater variance (raters are one of the facets: persons, items, and raters). There are some issues around this.  If you figure out a way to simulate rating scale data for a polytomous model, there may be a paper in there!
Please merge your accounts, @Marc Schubert (you are both user28587 & user28589). You can find out how to do this [here](http://stats.stackexchange.com/help/merging-accounts). After that, you will be able to access your question & post comments below it or its answers. Please do not use the "Your Answer" field to communicate w/ answerers.
I see what you mean, but why is it too high for you. It could be true that your model is fine, no?
What I did is estimating a model on a first serie of data $Y_{i}$ and I applied this model to another serie of data $Z_{i}$(representing the same physical variable). Then I obtain this value of $R^2$. So I believe, I'm not in the case you described.
Thanks for the direction, as always. I will go learn and employ Fourier terms to the analysis.
It is certainly in the realm of possibility. But what is your question? If this is purely about Weka programming, it is off-topic. If you are interested in a particular statistical aspect of RF regression, please update your question accordingly.
@Elvis, I gave the correlation example in the wikipedia article on pivotal statistic (http://en.wikipedia.org/wiki/Pivotal_statistic). [How on earth did you provide the nice link in the comment? I tried a href, it looked ugly.]
I want to simulate $y_t$ conditional on (increasing) age, fixed sex and starting values of $y_t$. Finding the $β$ is just one step along this road. So I suppose the latter of those alternatives.
And how long are the series?  Are you interested just in their average value, or whether the direction of change over time is different for the two groups (eg those with CEOs with MBAs having return on equity that grows over time, even if in absolute terms it is lower).  If so, you are in a much more complex world of time series analysis.
In that case, stick with control charts for Binomial data. The text I mentioned (Cumulative Sum Charts and Charting for Quality Improvement by Hawkins & Olwell) has some examples. I believe the authors also provide software for setting up the charts on their website.
Note: "independence of $ϵ_t$s implies independence of $ϵ_t^{2}$s" because "If the squares were dependent, there's a form of dependence among the unsquared values." hehe
Are these paired proportions or two independent samples?
@adam The P value is accurate even if you don't assume anything about the shape of the distribution. But then it is asking about the discrepancy in mean ranks. If you want to make an inference about the discrepancy between the medians, you'll need to assume that the shape of the two population distributions is the same, even if the two are shifted (and so have different medians). I corrected my answer to be more clear.
I edited my answer as my response was too long to post here.
@Penz In general there is no need for manual normalization, e.g. for determination of confidence intervals you can use the already available inverse-cdf-functions like "qbeta" in R.
david - since your variables are all positive, why do you want to fuss with $\\mu = 0$?  btw - in your simulation, you seem to be generating variables per.c and per.n that are independent. is that correct - and if so, is that what you want?
../.. As you can see from the data, the first measurement (if assuming a normal distribution) would have Upper 95% of 45.7 and a lower 95% at 4. So when I average 365 numbers, what is the CI of all of them ? I did find this resource that talks about error propagation: [Error Propagation](http://www.rit.edu/cos/uphysics/uncertainties/Uncertaintiespart2.html) but am not quite sure still. – [DBirdmanAR](http://stats.stackexchange.com/users/8559/dbirdmanar)
Can they be interpreted just as a normal variables before transformation, or is there any special interpretation for them?
Thanks Jeromy. I explored several possible transformations of data using Stata's gladder function for the ladder of powers.

Teoreically based solution will be one way to go, but before that I wanted to check for the data driven solution.
Where there is an interaction between a random slope and a predictor variable the correlation is more complex and there is not a costant correlation coefficient. I'm preparing my answer offline, because it's not brief. I hope I'll post it today or (more likely) tomorrow.
I agree with Jonathan. What you did to compute P(B or X) was to take P(X and A)+P(X and B) + P(X and C) to get P(X).  Then you need to count all the probabilities associated with B occurring but not X ( you do this to avoid double counting). That is the term with the value 0.25. You use the same idea for the second probability.
+1 I find the lack of distinction between parameters and their estimates to be very common with social scientists.
(cont...), but shouldn't the LHS of (1) be $p(\\sigma^2, \\beta|y)$ rather than $p(\\sigma^2|\\beta,y)$ since $p(\\beta,\\sigma^2|y)\\propto L(\\beta,\\sigma^2|y)\\times p(\\beta,\\sigma^2)$?
Means of 0-1 variables are the same thing as "the proportion of 1's" (both in sample and population senses). That is, you're in the situation of comparing proportions. *If* the variables satisfy the conditions of bernoulli trials (independence, homogeneity of probability of 1's), then a two-sample proportions test (or a chi-square, or several other tests suitable for such tables) would be the obvious analyses.
I would obviously not be interested in constraints on the order of the coefficients if I did not have strong *a priori* reasons to believe it. For the models that I suspect to be likely, heuristically the ordered LASSO should be more efficient. Having a lag-10 coefficient with the other 9 being 0 makes no sense in my *substantive* setting. This is a problem that my colleagues have worked on (ordered based shrinkage on the lags), but they used Bayesuan ideas and so wouldn't consider a (non-Bayesian) LASSO.
+1 for RODBC and sqldf - i find them essential as well
This question appears to be off-topic because it is about Statistics
Model-fitting is the topic of hundreds, if not thousands, of questions on this site.  (You could start with [tag:modeling] or [tag:model-selection] for instance.)  A full answer would require several books.  Please help us focus on your needs by explaining more about what you have and what you are trying to accomplish.
Ali, please edit your question (edit button at lower left, between "share" and "flag") to include the formula. Also, try to use the MathJax markup (effectively LaTeX in-line math commands, bracketed by dollar signs... so `$\\log{(GDP/Labor)}_{i}$` gives $\\log{(GDP/Labor)}_{i}$, etc.).
I am minimizing the cross-validation error (maximizing classification accuracy) in the inner loop - I am choosing the model that produces the smallest error for a given set of features and regularization parameters. Generally the models produced are quite small in terms of features.
@whuber I cant change my original comment anymore :S. After rereading your comment, what you say about $n\\hat{p_i}(1-\\hat{p_i})$ closely resembles the original paper, although they calculate it in a different but equivalent way. (See updated question). What I want to calculate is both that variance as well as the one expected purely from theory for such a system, assuming it is static with $p=0.5$.
By finding the largest (absolute) residuals.
I'm not sure your read the whole question. This is a theoretical question, and I'm more interested in why (or why not) this method is incorrect, not alternatives. Also, the example is for linear regression, but that is not important.
Thank you Macro for your detailed answer:)
Could you tell us anything about the likely values of $a$, $b$, $c$, and $X$? For some ranges of these values, this function is very close to Gaussian (but in others it is actually bimodal).
How many times has each individual been measured?
No, I don't. ELM is a fast way to train shallow RNNs. Deep learning is more of a whole field, and its main characteristic is not the way of training (there are many methods) nor the task (supervised, semi-supervised, transfer, unsupervised, reinforcement learning--it has all been done). It's the kind of architectures used.
Please qualify "weird outputs" and in general add more information about what you did and what you expected to happen.
An amusing observation: all your benchmark times are multiples of 2^-12 (=1/4096).
feel free to stop by the [Skewed Distribution chat room](http://chat.stackexchange.com/rooms/549/skewed-distribution) for details on how to join the blog. We are **always** open for more contributions from community members.
You could look into analysis of variance (ANOVA) and the linear model, for a start. This is covered in a huge number of books.
I'd also say it depends on your data and your goals. If I were briefing an executive on the sources of customer complaints, no way I'd be talking about fractions of a percent. You also have to consider the margins of error: saying 50.15% of voters prefer candidate X +/- 5% is suspicious.
@RDizzl3 : You're welcome!
possible duplicate of [Can somebody illustrate how there can be dependence and zero covariance?](http://stats.stackexchange.com/questions/30156/can-somebody-illustrate-how-there-can-be-dependence-and-zero-covariance). Also [this closely related question](http://stats.stackexchange.com/questions/29713/what-is-covariance-in-plain-language) is relevant/overlapping. There are many others. This is almost a FAQ on this site at this point. I'm hoping that certain veteran users will stop answering questions that are obvious duplicates/FAQs, as this wastes their time and adds to clutter on this site.
I guess it might depend on what you mean by 'work'. One issue is that you don't get model fit with ML, another is that ML is really, really slow if you have more than one factor. ML for categorical data in SEM hasn't been around for all that long.
a p-value is the likelihood of a result as or more "extreme" than the result given, not of the actual result.  p-value is $Pr(T\\geq t|H_0)$ and not $Pr(T=t|H_0)$ (T is test statistic, and t is its observed value).
These sort of data are really only useful if you can compare to some kind of reference.  For an example of a straight comparison, "is question 6.1 rated more positively than question 6.2?"; or (if you have similar surveys over a period of time) "Is the response to 6.1 this year better or worse than last year?"  Second, you might be interested in relationships eg "do males answer this question more positively than females?" or "is a more positive question on 6.2 likely to lead to a more positive response to 6.1?" (but be careful, if the answer is yes, this can have lots of reasons).
Also, where am I supposed to get this p-val from?
I think tanh(x) = 2/(1+exp(-2x)) - 1, so it is a scaled version of the logsitic sigmoidal function, so I suspect if you use a logistic loss function (cross-entropy) but with targets of +/-1, I suspect the same shortcut will apply.
Consider solving this from first principles: given this information, you can easily construct the full bivariate distribution $\\Pr(X,Y)$--it involves only four outcomes--and from that you can compute anything you want.
@Kasper I think that would make a nice separate question if you flesh it out a bit.
@robby OK... I didn't know this "second order stationnarity" I think you should not say second order stationnarity in the question and give the definintion instead. For more clarity you should ask another question. do you have a paper that refers to this second order stationnarity ?
But if the result shows that the intake of fish is, depending on meal and group, unlikely if compared with other intakes, this information is useful. Something you should consider is, or you already did, that grams cannot have a negative value though it is a common praxis to ignore this problem and just do something similar to OLS. Depending on the question it could be interesting why only a few persons eat a lot of fish but to answer the general question whether a certain food is consumed, the outcome that fish is unlikely to be a part of the meal is somewhat useful. In the framework
@Hossein Equation 2 seems to suggest that they're using Euclidean distance, but I'm not sure.
Ties also affect Bayesian inference. The presence of ties in a sample may preclude the existence of the posterior distribution even under the use of proper priors [(see)](http://www2.warwick.ac.uk/fac/sci/statistics/staff/academic-research/steel/steel_homepage/publ/fernandez_and_steel_1998.pdf).
"*Note that p1 and p2 are simulated so it might happen that, for the same individual, both are large (like p1>0.8 and p2>0.8).*" This makes no sense - your simulation should respect the fact that you're dealing with things that must add to no more than 1. If you're simulating probabilities, you need to use a suitable *joint* model for them to begin with. The Dirichlet is commonly used for such purposes, and the Dirichlet-multinomial is then being used for the unconditional eye-color outcomes over the process (the Dirichlet-$p$'s and the multinomial eye-color distribution given the $p$'s)
Yes, I think. Adding a suppressor can reverse the sign of a coefficient. As well as not. So what is your question about - a suppressing phenomenon or a changing of a sign phenomenon?
Please spell out your abbreviations. Presumably, "SSR" is the sum of squares of residuals and "df" is degrees of freedom. If that's so, then how do you determine that df = p-1? Usually it has $n-p-1$ degrees of freedom when a constant is included in the regression and there are $n$ data values. That leaves me wondering whether I have correctly understood your abbreviations.
Thanks Sir ,you really a life save.It works perfectly.
Not a good one.  Any such guide would have to simplify so much that it would use a lot of methodologically suspect shortcuts such as "Tom Swift's Electric Factor Analysis Machine" or the notoriously risky stepwise regression algorithms.  I'm afraid you've entered an area that requires long hours of study if you are looking for sound results and/or an understanding of causal relationships.  Welcome!
Newb here.  What do you mean by non-parametric.  They all have parameters.  Some have many more parameters than samples.  They can be called hyper-parametric, or more informatively model-agnostic. ... 

My first thought is to make some variation on a cube plot.  I like to visualize the data.  EDA, right.  Good luck.
How can there be an effect of POP (or interaction) of any kind if there are always 100 fish in each tank?
@whuber: I think $N$ is the number of paired sample time series, each of length $p$ (for the series that is mapped from) or $q$ (for the series that is mapped to).
Thanks a lot Matt. My hypothesis is that a set of ICD-10 codes would be significant in cases when compared to the controls.
Billinglsey's a torture unless you're mathematician and already studied measures. Partarathy's [intro](http://www.amazon.com/Introduction-Probability-Measure-Readings-Mathematics/dp/8185931550) is much easier 2-in-1 book, Alan Karr's [Probability](http://www.amazon.com/Probability-Springer-Texts-Statistics-Alan/dp/0387940715) text is also easy read.
Are you assuming a fixed Z?  I've got 27,000 cases (aka properties) and the Y (assessment price) and Z (sales price) differ for each one.
@DimitriyV.Masterov Thanks again. I use R.
I think you wanted to ask why **reporting the hypothesis testing results** by showing confidence interval is better than just saying something is confirmed or rejected on some p-value level.
But, since the _distribution_ of $X-2Y$ is wanted, not just the mean and variance, you need a Fourthly, a weighted linear sum $\\sum_i a_iX_i$ of independent normal random variables is anormal random variable with mean $\\sum_i a_i\\mu_i$ and variance  $\\sum_i a_i^2\\sigma_i^2$ as an additional hint.
For Q2, ML is necessary because comparisons using REML are not valid when the fixed effects change. A possible useful related question is here: http://stats.stackexchange.com/a/16015/3601
oh my bad, here it is [(non linear) conjugate gradient](http://en.wikipedia.org/wiki/Conjugate_gradient_method)
@whuber That was super helpful, and I think I got it. Does my edit look correct? Basically it works out (I think) that $\\log(g_t)$ is a linear combination of i.i.d normal random variables, so it has a normal distribution too.
I didn't ask you to derive a formula for it -- what object is it you tried to derive the formula for? I don't mean its name, either. Consider - what is the pdf of a random variable? What does it tell you?
How can a/b ~ gamma and b/a ~ gamma? Isn't the inverse of a gamma the inverse gamma distribution?
So you have a clients table with client name and attributes (e.g. sector) and then you have a products table with product name/code and atttributes (e.g. colour, size, type, etc.), is that correct ? Can you explain your dataset?
Welcome to the site.  The best way to analyze the data will depend on your research question.  What are you trying to find out?  Who was asked the questions and why?  You should be able to edit your question to add in that information.