tttjjj/elt_bench huggingface dataset:
Here's two hints: 1) In a Gaussian distribution, the mean and median are the same. 2) You can then find the sd by looking at the 15th percentile: How many sds from the mean is the 15th percentile in a Gaussian distribution?
@ David. I'm not sure "it takes too long" is a valid reason to assert something doesn't advance science. Would we say the same thing about a complex lab assay? If the spirit is to verify the results of the meta-analysis, exploring the assumptions of the search is just as important as the code. Just having the data means you *won't* detect if the searcher was sloppy, made a key error (as I have done once) in searching, or abstracted incorrectly. It gets at only one layer of reproducability - and if we're mimicing the life sciences - not the layer they get at when they replicate results.
It sounds like you have a fundamental issue concerning what you are investigating and how to collect appropriate data: if you consider the nighttime data to be of no relevance to your research, then you should be excluding them on that basis alone, not based on some kind of analysis of all your data. The data you do end up using will partially determine the conclusions you can draw. If you would share information about your purposes and hypotheses you would increase your chances of getting useful, focused answers.
Are you interested in whether there is a relationship between BMI and each of the many attributes?  If so, have you considered calculating a correlation coefficient between BMI and each?  Also, can you gather more data?
@AlecosPapadopoulos yes, I understand that. That's the point of the question :) I want a test that looks at the joint rather than some finite collection of marginals. In principle, I could get away with checking $t^T X = t^T Y$ in distribution, which is why I've also been checking other marginals like the principal components as additional sanity checks.
I don't know, you've seen many efficient citizens lately?
(Cont.) Anyway, you know yourself that the term "loadings", albeit being really quite dubious, is nevetherless not mixed up with "eigenvector" in other multivariate analyses, such as discriminant analysis, for example. Once again, as I put it, in PCA loadings 1) incorporate info about the magnitude of variation; 2) Are the covariances/correlations, and are therefore used for interpretation. Eigenvector values - are not.
Also it's better when supplying code if you also supply an example data set (it doesn't have to be your 'real' data, as long as it has the same problem), so that your code runs. That is, you should supply an example that we can run by copy/paste.
I believe this question has already been answered by myself and John here:  http://stats.stackexchange.com/questions/69810/correlation-between-vegetation-and-erosion/69818#69818  In that question erosion was a continuous variable while vegetation was a categorical variable
I appreciate your kindness and the fact that you are open to learning.  I half expected you to take offense at my suggestions.  A lot of people would.  We all would be better off if we could be openminded and not let our egos get in the way of learning.  Even old guysn like me have plenty left to learn.
I'd argue that this question is "representative" [pun intended] of a common issue and closure is not appropriate.
If one is principled in choice of binning procedure, then binning can be a useful way to deal with predictor nonlinearity. It really depends on the predictor and the domain.
This answer is really useful. I appreciate your supportive, motivational and kind explanation and also yours tips-and-tricks for googling.
Do you have a sign error in your $W_i$ or did the image not reproduce the vertical line in a "+" sign in the equivalent term in the image?
The downvotes are perhaps because 1) there's not sample data, and 2) you say you can't use a correlation matrix, but don't explain why you don't think so.  It would have been more polite for the downvoters to explain why and give you a chance to fix it before downvoting.  It's not necessarily a bad question; it just needs a little improvement.
You could have mentioned in your answer that matplotlib allows rendering of all typography in the plot with LaTeX so it perfectly integrates visually.
Failure to reject normality doesn't imply that assuming normality is justified. It doesn't "anoint" normal theory tests. It may mean that your populations are severely non-normal and you just have very small sample sizes.
Please avoid posting the same question on different StacKExchange forums ( https://stackoverflow.com/questions/23649621/performance-comparison-of-patternnet-and-newff-for-binary-classification-in ); if the question is deemed out-of-context it will be transfered by our moderators.
See any number of posts relating to the finite population correction, such as [here](http://stats.stackexchange.com/questions/80162/standard-error-of-proportion-that-takes-into-account-population-size) or [here](http://stats.stackexchange.com/questions/5158/explanation-of-finite-correction-factor)
The McNemar test is designed for paired data, isn't it? So you cannot compare its output with that of Pearson or Fisher which assumes independent samples...
Are you sure you don't want an intercept term? ... Also, how are you calculating "**relative importance**"?
Hi, sorry. The question is, how do I calculate the Lipschitz constant (or a bound) for squared frobenius loss on a matrix completion problem?
Did you mean, perhaps, *stationarity*?
@onestop, thanks for the tags help!
I mean, for regression problem, the cost should be squared error, but for classification, we shall use deviance, right?
You should provide a reference to what leverage plots are.
To add to jona's comment, the Normal approximation numerically *should* be done to floating point precision (so essentially 15 decimal points)
@mpiktas, let me know if I am missing something.
Good catch. What the wikipedia page actually says is, "The z-score is *only* defined if one knows the population parameters; if one only has a sample set, then the analogous computation with sample mean and sample standard deviation yields the Student's t-statistic." I used the phrase, "can be interpreted effectively as a kind of" in my answer because I wan't confident what was the best terminology to use. You are definitely taking samples, so t-statistic is probably the better terminology to describe what I had recommended doing, but other than that, my basic advice does not change.
Variogram fitting *usually* requires human intervention. After you find the variogram models that best describe your data, add them together with appropriate "normalization constants". This is often done to capture variability at different lag scales.
Why do you think it might not be appropriate?
What are **y1** and **y2** then? Are they separate variables? And what are the dots? Do their coordinates represent observations of two different variables or three? If three, how can one tell the difference? And why do the vectors pass outside the observations rather than through them?
Something unclear (maybe just for me): if you have 5 subscales, what would the total score mean or reflect in your opinion (even if you didn't say if the 5 factors were allowed to correlate or not)? Another point of detail: It sounds like your measurement model is disconnected from the structural part. Why don't you include all items, define how they are supposed to be connected to your subscales, and build your SEM around?
+1. But is the outlook for #3 really so bleak or difficult? If I have a collection of independent estimates, each with its own margin of error, why can I not (at least roughly) combine them in the usual way (as a weighted mean, weighted inversely by squared MoEs) and combine their standard errors as well (using variance formulas)? It wouldn't be perfect, but it ought to be better than choosing one poll on which to rely, right?
Actually, the PCA values and the clustering groups are independent. PCA creates "new" coordinates for each observation in `mydata`, that is what you actually see on the plot. The shape of the points is plotted using `fit$cluster`, the second parameter of `clusplot()`. Maybe you should take a deeper look into PCA. Let me know if this helped you, or if you further references.
Thanks for the info Glen; is there any chance you could be more specific about how I can get the answers I'm looking for from R? Is there a specific package that allows evaluation of the change in R^2 without iterating through several models like rolando2 suggests above? This approach seems to violate the DRY principle pretty badly.

Alternatively, is there a package that allows me to run a multiple regression using partial correlations rather than the zero order correlations, and assess the significance of those partial correlations?
@Rob I believe it's still not workable in a Cox PH model. It's perfectly doable with a AFT model, though the complexity of getting back an estimate will likely depend on how many covariates you have.
possible duplicate of [What to read from the autocorrelation function of a time series?](http://stats.stackexchange.com/questions/18284/what-to-read-from-the-autocorrelation-function-of-a-time-series)
how is the proportion calculated? the habitats do differ: this is a descriptive statement that does not need to be tested with infernetial statistics unless there is sampling involved in these measurements...
@mpiktas, nope. No copyright issues. That is the official website for the book. The authors obtained permission from Springer to make the PDF freely available online. (See the note to this effect on the site.) I think they got the idea from Stephen Boyd and his *Convex Optimization* text. Hopefully such a trend will pick up steam over the next few years. Enjoy!
from a google search of "temporal aggregation" you can find several interesting articles.
@Momo. I am using python's sklearn.ensemble's RandomForestRegressor. I just use the model like
Here's a page on exactly how good the normal approximation of the t distribution is for n=30.  http://www.johndcook.com/normal_approx_to_t.html
You ought to credit the person who *really* answered this question (George Lowther), especially because you copied his answer with only trivial changes.  See http://math.stackexchange.com/questions/2763/what-is-the-expected-number-of-runs-of-same-color-in-a-standard-deck-of-cards .
what do you mean by 'method'? you mean spearman/pearson/kendall?
I believe that once you read through more than one of [David Reilly's answers](http://stats.stackexchange.com/users/3382/irishstat?tab=answers&sort=newest) you will have a pretty full account of all the (many) questions you ask here. Could you point specifically to anything that is not addressed?
It sounds a bit like you're asking for something akin to [dynamic time warping](http://en.wikipedia.org/wiki/Dynamic_time_warping). That garners a few hits here as a [search term](http://stats.stackexchange.com/search?q=dynamic+time+warping), though there are a number of other ways to try to estimate some continuous nonlinear relationship between your two variables.
>(...)the code can literally run 10-100X faster than an equivalent R version

I don't agree. Almost all R libraries are written in C or Fortran. The R works mostly as a "glue" between them, and its relative poor performance doesn't count.
You seem to change notation (from theta/x to specific featuers) in the middle of the question; this makes it harder to understand.
As is, the question is too broad to be answered correctly. Different fields (and different journals within those fields) have different formats. But, more importantly, what you have to report depends on what you found interesting.
@asjohnson What I had in mind was forming a useful model for each store separately. If there had been any structural changes , perhaps do to a re-assignment then deterministic structure like a level shift would be identified and incorporated. If there were potential cause variables like advertising which might be local to a "region" then the cold be incorporated at the store level.If you are referring to a simple indicator reflecting national/regional , you wouldn't need that at the store level.Ultimately store forecasts could be reconciled with national/regional forecasts ala parent to child.
That's a reasonable suggestion, but I think it isn't quite what the OP is asking for.
Also, acceptance probabilities should always be calculated on the log-scale and only exponentiated at the very end (if at all, you can work around using exponentials instead of uniforms). You are just asking for a numerical catastrophe otherwise.
I see what you are saying. Since they do not explicitly show $D_{t}(i)$, it's likely they do the weight normalization in the update step. In that case, the $e_t$ term is just the total weighted error, prior to being normalized. The sum of the individual weights are not required to equal 1 in either example. I think what's important is that at some point the weights are normalized to guarantee a probability distribution for the next weight update.
IIRC they fit a logistic and recalibrate it with the CV. See http://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf for the exact details.
@MrFlick, I think he means "Kosher" as if this the right syntax. He seems to be familiar with the statistical side of it
could you give a reference where that assertion is claimed?
Hello.  For the uninformative priors you want $\\alpha_0 = 1$, not $0.00001$.  You can't determine what the informative priors should be the way you said (or in any way like that).  In practice you would choose $\\alpha_0$ and $\\beta_0$ so that the shape of the prior reflected your beliefs.  For simulation purposes, I think they just mean try some different values of $\\alpha_0$ and $\\beta_0$ that will give you a less flat prior and see how your posterior changes.
@mpiktas has provided the details in his answer.
Thanks Ulysses, this helps me a lot. Concerning point 3; I tried DIC and penalized Loss functions (using jags) but they give me largely different results. DIC gives a difference of about 0.3 (in favour of the MORE complex model) whereas  penalized Loss functions gives a difference of about 70 (in favour of the LESS complex model)! I am a bit confused about what to do with these results, can you give me some insights/references about the way to proceed to compare properly the model? (Inference is based on MCMC)
No one will be able to answer this without more detail. If you show the calculation that leads to the answer of 35%, we'll will be able to help you understand what that means.
Good edits to both your question and my answer, thanks.
It is used in the field of machine learning(Other fields, I do not know. I am new to it). You can wikipedia it.
You could try adjusting the remaining probabilities - $(0.5 - (1*10^{-17}))$ should be very close to $0.5$.
It's almost axiomatic that statistical people don't have good data on anything like uses of statistical methods, statistical software, etc. But you could search your favourite citation database to get some indications. But relative use doesn't tell you much about desirability, at least not much more than newspaper circulation or book sales figures tell you what's worth reading.
Well I believe Rstudio is one of the most popular interface tools for R scripting?
Thanks again (was updated _after_ I'd already accepted the answer). This notation is new to me so will take time. Short of an exact solution, goal would be to get it to around 0.001 (currently around 0.01). Not sure that calculating this 'on the fly' for an estimate of probability is going to be practical, so I may end up storing the empiric distribution. (Guess that's why the authors used a table rather than a function). I'm not able to see it in code yet - any hints more than welcome...
Such stepwise approaches do not work as advertised.
Correct. The PyIMSL algorithms are Python wrappers to the IMSL C Numerical Library (~500 math and stat functions), which is not open source. PyIMSL is free to use for non-commercial use though.
SummarySE function is not working on my RStudio to summarize the data before plotting it. Further I tried ddply function with plyr package still not recognizing sem i.e. standard error of mean code. Any help pls.
"Symmetric" and "strictly" have different meanings.  Here, the symmetry is unnecessary--it's merely a convenience to simplify the equations.  (When $K$ is not symmetric you can replace it by the overtly symmetric $K' + K$ without changing the form.)  "Strictly" means the value of the form is zero only at the zero vector.
Hi Peter, many thanks for your insights. You say that many studies include two smoking variables. Do you mean they have a model of the form: y=B0+B1(SMOKER)+B2(CIGARETTES) per day (wouldn't the fact that CIGARETTES=0 for all SMOKER=0 cause instabilities)? Or that they do a stratified analysis  y=B0+B1(CIGARETTES) just using the data on smokers?
Thank you. Will look up Kiefer's book and see what I find.
Hmm I suppose I could have used weaker language, explained myself more and been less cynical regarding the first part. As for the second part, that is the point I'm trying to make.
Thanks for your answer! Some remarks: ad A) No matter how we all it. $c_t$ in the classical formulation of HP an have any distribution as far as I know. In the Kalman filter setting it has to be normal. This is a strong assmption, isn't it? ad B) Now I see: this is the smoothness condition ... great! Thanks for C)!
Just a quick note: minimum description length is very powerful and useful, but it can take ages to obtain results, especially when using normalized maximum likelihood withslighltty larger datasets. I once took 10 days running FORTRAN code to get it for just one model
Hi @Aniko, Thanks for the feedback on this and for the reference. If my goal is to estimate the variance that would be measured on a single day in a single treatment, which term would be approapriate?
:zbicylist Correct, since this is a special case of a Transfer Function where there are no user specified inputs and the form of the ARIMA model is (1,0,0) and the model assumes that there are no deterministic variables to be empirically identified ( such as Pulses, level Shifts , Seasonal Pulses and/or Local Time Trends via Intervention Detection .
From my experience, the fact that something is commonly used in statistics does not necessary imply it is the correct procedure...
If i don't estimate MSE, can't i compute SSTR in ANOVA?
Just to clarify, exponentiating the log odds will make them odds, but you can then convert to probabilities too.
"Does that mean that it does not matter how large the effects in both groups are, as long as they are 0.5 apart?" Yes, that's what I'm saying. I also agree with @jonas. I think an emphasis on standardized effect sizes obscures meaning, and people are better off talking about real differences of real measurements, in the units in which they are made.
@rbatt, I don't have any references to hand that are directly on point; my thinking here basically echoes that of [Andrew Gelman](http://andrewgelman.com/2004/12/29/type_1_type_2_t/), author of [Bayesian Data Analysis](http://www.amazon.com/Bayesian-Analysis-Edition-Chapman-Statistical/dp/1439840954/) (3rd edition soon to be released).
A subtle issue about parameters is exposed by an old computer trick: take the binary (or decimal) representations of $\\beta_0, \\beta_1, \\beta_2,$ and $\\sigma$ and interleave them (by groups of four) to make a new binary (or decimal) number $\\theta$. Obviously the process is reversible: you can read $\\beta_0$ off the first, fifth, ninth, ..., etc. digits of $\\theta$, and so on. Therefore "once [you] learn the value of $\\theta$, [you] know everything there is to know." But $\\theta$ is *not* a valid parameter due to the contorted way in which it labels possible distributions.
I have a simulated dataset example where adding a very important variable (by any other measure) makes the % classified correctly decrease.  This is an example of how an improper scoring rule can mislead.
I don't know what you mean by coding for one categorical variable. Can you give an example in syntax? Is your dependent variable continuous or categorical?
That's a good news! I'll second this nomination.
@chl: upps- today(end of january) I came back to this question/conversation of mid-november.... Sorry I didn't check earlier - there was something with my courses and then I forgot the stat.exchange... If I've something worth I'll come back next days.
Could you specifically point out how this answers the original question?
Would it help you if you wrote it as $(X_{(1)}/\\theta)/(X_{(n)}/\\theta)$?
(1) I've only heard of a Ryan test in the context of multiple comparisons. I'm familiar with many dozens (perhaps as many as a hundred) of goodness of fit tests, and a Ryan test for normality is new to me. Can you describe the test?  (2) When looking at power for goodness of fit (or indeed any test whose alternative is not indexed by only one or at most a few parameters), you must specify against which alternatives power is being calculated.
Be aware that asking for code or packages is off-topic for CV (see our [help center](http://stats.stackexchange.com/help/on-topic))--you may or may not get a response about that. Your 2nd question is on-topic, but perhaps a bit broad. Can you add some details / constraints, or otherwise focus it a little?
@Penguin_Knight So if I understand you correctly, the variable of Testanzeige should be either 1 or 0, right? How should the regression looks like?  Y (Ad-Attitude) = b0+ b1*X1(Ad,the variable is either 1 or 0)+ b2*X2 (CSI)+b2*X1*X2 +error
Hey Expecto, I just said those k values just for example. I would definitely do it the way you suggested.
Actually i want to solve how to get the solutions in some papers like http://www.academicjournals.org/ijps/PDF/pdf2012/9Oct/Chen%20et%20al.pdf  It is written that one step ahead prediction is used. It has a data which is univariate. And separates the data say %30 for test. What is x and what is y in here for SVR?
Even if the technique does not use Minitab that's fine
try [Sandwich estimates](http://cran.r-project.org/package=sandwich).
Great answer! Thank you very much. So, if I conduct a GLM with mean pairwise contrast I can use this pairwise information to say "species A has significantly higher number of infected individuals than species B" for example? If so, which significance and Wald x2 values should I report, the model effects or the pairwise comparison ones? I am using SPSS but I tend to use Type I SS in order to know what I am doin with the model, but it seems to me that the pairwise comparison values are based on Type III SS, does it make sense to use/report both Types? Thanks!
What have you already tried?
Added extra info in my answer - my guess is that you're gonna have some surprisingly positive surprise if you try it.
(A) Your p-value is too small: you need a two-sided test for this hypothesis.  (B) For the second part, what hypothesis do you wish to test?  Several that come to mind are (1) whether all coins are fair, (2) whether at least one coin is fair, (3) whether the coins are fair on average, (4) whether a majority are fair.
I was leaving out the "train_x =", and adding that solved my issue! thank you kindly
Whether you need to transform your predictors is unpredictable and is not a logical consequence of how you handle the response (independent and dependent variables in your terminology). Just as with classical linear regression, any transformation choice should depend on the relationships between variables.
It is not clear to me that the OP meant cross correlation in the time-series sense when they used this term. It may well be a case of inadvertent imprecision. Perhaps we should ask for clarification from the OP.