tttjjj/elt_bench huggingface dataset:
I would slightly disagree with onestop's advice, prefering to leave the ordinal variable as a categorical variable, and then check how the co-efficients vary as you move up the ordinal categories for some structure.  To fit the data as if it was continuous means you have to chose numerical values for the categories, and the results could easily be sensitive to this choice - example if I have 1,2,3,4,5 it gives the same ordering as -20,0,50,1000,1000000 - and there is no obvious way to chose between them prior to observing the data.
All analysis is in STATA, though I think I can access SPSS
A shorter version would be `marginal.plot(mydata[,-ncol(mydata)], mydata, groups=class)`, with the [latticeExtra](http://latticeextra.r-forge.r-project.org/) package, although it doesn't match Weka diagrams.
Is your discrete signal periodic, or is it an aperiodic a sampling of a periodic signal?
@NickRiches The distribution of the outcome variable is obviously of the main concern.
Conjugate Prior and I were independently answering during the same time. Proportional allocation and sampling proportionalt to size are synomous terms that are used for this type of stratification. But the adjective random should only be included if the sampling within strata were random.
Here's the link for the second book [Statistical Distributions](http://www.amazon.com/Statistical-Distributions-Catherine-Forbes/dp/0470390638)
Appreciate so much for your kind help! I will try them. Thanks a lot!!
@Peter Flom. Glad you agree, but you mean complementary... Insert emoticon if desired.
@Henry empirically I found that the posterior probabilities **are** predictive of actual errors on the test set. The question is if this is merely a coincidence (I suspect not), or if I can expect this to also work on similar datasets.
As noted in the question and in comments to other answers, the OP wants the *rows* to sum to one, but the diagonal to be maximal in its respective *column*. Your parenthetical comment paired with a rejection sampling scheme and independently distributed rows seems like a little more promising approach depending on the dimensionality of the problem.
You may also be interested in the following article (which is often cited when comparing classifiers performance on the same dataset): Hothorn T, Leisch F, Zeileis A, Hornik K: [The Design and Analysis of Benchmark Experiments](http://www.stat.uni-muenchen.de/~leisch/papers/Hothorn+Leisch+Zeileis-2005.pdf). J Comput Graphical Stat 2005, 14(3):675–699.
Heh - I disagree that SAS is harder than R. They're...different. If one comes from a tradition of programming languages, R may make more sense than SAS. But really when it comes down to it, they're different - as I said in my post, SAS is more of a command line interface than a proper programming language.
If the assumptions are reasonable, you might use either a proportions test or a chi-square. (This looks suggestive of the classic example of Simpson's paradox; where do the numbers come from?). The 'at what level of significance' is presumably actually asking for the p-value, but if so, that's not a good way to put it (since it seems to imply you get to choose the significance level based on the data).
This is just not true - the sigmoid function is pretty much equivalent to $sech (x/2)$
Daeks boot was what I needed thanks
@Super That is exactly the misunderstanding of CI being addressed in the paper. In particular, *the true mean is a number*; it has no distribution.  See the first two hits in a site search for [confidence interval](http://stats.stackexchange.com/questions/tagged/confidence-interval?sort=votes&pageSize=50) for further discussion.
Right, I edited appropriately. For economists, $L$ usually comes up in some natural ways, in that seasonality has the lag of 12 months, and business cycles have the lags of 5 or so years. If you have a natural science application, then typically the time domain is not as interesting as the frequency domain, as most real-life signals have continuous spectra.
"See which one would offer higher return with lower risk" sounds like estimation, not hypothesis testing. (This may be why it was difficult for me to identify the hypotheses.)
Anyway, appreciate your response!
Did you mean "the best curve [...] is _not_ a horizontal line"?
EM sometimes works well in presence of noise, when you add an additional cluster. It then often converges to having one cluster containing all the noise points. It doesn't mark that cluster as noise though.
Are B and C independent ,correlated to some degree or linear?
Plenty of interesting ideas here (+1). Note, though, that the replacement procedure is not necessarily sequential. One can identify *all* "outliers" at once and replace all of them with the mean of the remainder.  This is a consistent procedure not unlike Winsorizing.
If `price` doesn't vary across alternatives, you should call it as `mlogit(trans1 ~ 1 | price)`
@MannyG Yes, but only because the counts would be directly proportional to the rates, not because it would be appropriate to use the counts themselves.  The need to use rates here is clear when we consider what it really means to predict a future value: you have to specify the time interval of the predicted consumption.  One is thereby predicting an amount times a time to get a quantity, implying that amount *must* be a quantity per unit time: a consumption *rate*.
For the covariate to be useful it is not necessary for the DV and the covariate to be correlated. Instead, the covariate will improve the model provided its *residuals* with respect to all other independent variables (the ANOVA factors and any other covariates) are correlated with the *residuals* of the DV with respect to those same IVs. Provided we understand all your references to the DV and the covariate to really mean these residuals, then your answer would seem to be correct and contain useful advice.
Why bucket at all?  If it's because you doubt a linear relationship between X and Y, you can plot the data and guess at a functional form, or consider more formal alternatives, e.g., nonparametric regression.
@user11852...Yes I trained my data using 2 different data sets. So I used f(x,y) = exp( (x-meanx)^2/(2sx)^2 +  (y-meany)^2/(2ys)^2) for the gaussian basis function. Now I have a 3rd input that I need to combine with the other two.
One approach to choosing a smoothing parameter would be to optimize one-step-ahead prediction errors (such as sums of squares of one-step-ahead prediction errors). If you're trying to identify outliers, you'd want a different measure of prediction error - one reasonably robust to outliers (and then moving averages would seem an odd choice - why not something more robust to the outliers?)
Lower reliability means lower power because it means part of the measurement is noise. Two unreliable variables multiplied have lower reliability than either alone. It is, in fact, the product of the two reliabilities.
I would have thought that that item was the easiest to understand.  Here's an example: in a particular field of study suppose that we had previous experience that most variables operate additively.  Fitting an additive regression model that does not assume predictors to operate linearly (e.g., using regression splines) will provide an interpretable and useful statistical model.  Using svm or random forests on the other hand will be very hard to interpret, not have any separable effects, and will not predict better than the nonlinear additive model.
I can assure you that in this case, street intepretation coincides with wikipedia.
@Penguin_Knight the 4 tests were done over 2 months, and it wasn't necessary for certain time lapses between the weeks (no effecting dominators)
No I meant within three standard deviations. I expect that observations beyond 3 standard deviations are much more likely to be true outliers. My fear is that my methodology is too sensitive to outlying values which is why I am considering those within three standard deviations to be possible false flags. 

In the data there are absolute outliers which are not representative of  typical events so the necessity to remove these observations is not questioned. In practice it seems fairly successful I am currently looking for possible alternative methods.
I don't think your understanding regarding the estimates is correct. Have you changed the contrast settings or are you using R's default? Also, I wouldn't expect error bars from a model with a log link to be symmetrical on the original scale.
Calling a variable a "covariate" is treated as meaningful in SPSS in terms of how you can use the wizards to call for your analysis, but it doesn't actually have an underlying statistical meaning. Just code `Gender` as a column of `1`s & `0`s (using whichever you prefer for `"female"`), & run a multiple regression w/ your scores & your dummy code.
That's right - if the variance of PC is, say 3.5, then that PC "explains" variability of 3.5 variables from the initial set. Since PCs are additive, `PC1 > PC2 > ... > PCn`, and the sum of their variances is equal to the sum of the variances of the initial variable set, since PCA is computed upon covariance matrix, i.e. variables are standardised (SD = 1, VAR = 1).
Note that "$p(\\omega)$", *sensu stricto,* is not necessarily defined: probabilities can be applied only to *measurable sets* in $\\Omega$. Note, too, that the question really applies to the *sigma field* of measurable subsets of $\\Omega$, not to $\\Omega$ itself. This leaves it open to multiple interpretations; a natural one is, "for what sigma fields $(\\Omega, \\frak{S})$ do there exist a probability measure $p:\\frak{S}\\to [0,1]$ having a continuous distribution?"
Yes:  regression (ordinary least squares regression) will enable you to create, using a weighted combination of a and b, a predicted version of c that correlates more highly with c than either a or b does.
Aww thanks! I'm not quite ready for a faculty job yet, but I'd like one down the line. FWIW, I think @Glen_b's answer is quite good! Neither of us came out and said this explicitly, but if you're on the fence between a rank sum and a $t$-test, go with the rank sum. I think part of the t-test's popularity is how it fits in nicely with the themes of Stats 101, rather than any intrinsic fitness-for-purpose. Anyway, good luck and feel free to post more questions as they pop up!
Thanks. Can you provide instructions on how to plot the "profile likelihood functions for the intercept and coefficients?
You can model this using a [Bayesian network](http://en.wikipedia.org/wiki/Bayesian_network).
Carlos, thank you for your contribution, but the level of scientific validation that I need should, at least, give me a "probability" value of that claim. Thanks, anyway.
Yes I like the moving deletion using the SD. I'll use it thank Qbik. Both of the graphs are from the Bloomberg terminal.
I think this quote means that the results of the experiment should be "obvious", and statistics just lets one put a precise figure as to "just how obvious".  The word *needs* is the key.
Without more information, it may be difficult to give a clear cut answer. Is there any information about business failure in the area, ie: how is the failure of businesses distributed in the area?
Three years later, the issue has come up again via comments on [another duplicate](http://stats.stackexchange.com/q/80922/32036) :P See also [this meta-CV question on the broader issue](http://meta.stats.stackexchange.com/q/1867/32036).
There should (or at least could) be some substantive knowledge explaining this strong qualitative behaviour. Similarly, separate analyses for separate parts of a series imply quite different generating processes; that could make sense if you had knowledge that the system became fundamentally different at the turning point. Absent this, it is difficult if not impossible to give a worthwhile technical reply to your question without much more information.
@Tom, how did you uncover the obvious pattern? For 4 time periods and 4 different values for each period there are $4^4=256$ possible combinations, why did you post only 4 of them? I am asking because I suspect that there might be another variable which indicates how to group the values.
yeah but it might do well for logistic regression.
Cox regression would probably be even better suited to your question that a straightforward exponential regression: http://en.wikipedia.org/wiki/Proportional_hazards_models
I believe that `sample` does not act as you seem to expect.  For instance, what happens to your code if the game is modified to allow a rack of 28 tiles?  Change `size=7` to `size=28` to find out.
Kernel are typically used as a way of implicitly mapping your data into a higher dimensional space. So even if you explicitly map your inputs to the new space, for example see the last few lines of the definition section of the [polynomial kernel](http://en.wikipedia.org/wiki/Polynomial_kernel) article on Wikipedia, the dimension of your data matrix will still change. Either the number of rows or columns will increase, depending on how your data is stored.
Just to be clear, are you trying to find the VaR given some horizon in the future with some holdings today,  or are you trying to evaluate the VaR of the strategy in the past?
You really, really, want to have at least an approximate sense of the counts, because they determine how reliable each proportion is.  Otherwise you would be treating a value of say, $50\\%$ as the same regardless of whether it represented one success out of two tries (which could be pure luck) or a million successes out of two million tries.
Graham, I wonder if there are some typos in your answer.  Variance is represented by the Greek lowercase sigma raised to the power of 2 (i.e., $\\sigma^2$), and the standard deviation is the square-root of that, or just sigma without an exponent (i.e., $\\sigma$).  You may want to edit your answer.
(+1) Thanks for your nice answer. Would you like to give an example of canonical correlation analysis with sem package. Thanks
Hum, it seems that this software meeting my requirements perfectly. I will look into it, and hopefully, I can plug it as a module in my application.
(+1) Thanks for clarifying that point. I was unsure about (a) the use of 'parameters' because we often speak of person (resp. item) parameters when estimating IRT models (I don't speak of person-specific covariates, just locations on the latent trait), and (b) if there was any suggestion about differences between the marginal approach (where we assume that the $\\theta_i$'s are random, with $\\mathcal{N}(0;1)$ for prior distribution) and the conditional approach (where subject and item parameters are estimated separately, in line with Rasch's *specific objectivity*).
I'm not very experienced with microarray data, but I'd say tread carefully. You might be able to do some normalization using household genes, if at all, but even this is not a substitute for what should have be done in the first place.
Key search term: Block bootstrap.
For an answer to a question that is formally similar to yours, see: http://stats.stackexchange.com/questions/47286/determining-the-probability-that-two-functions-produce-the-same-outputs-when-no/47338#47338
I think this question is on-topic for CV, but you should be aware that it is close to the border. Questions asking for code, for functions to do something, or why you are getting a particular error, are off-topic for CV (see our [help center](http://stats.stackexchange.com/help/on-topic)). *Some* questions of that type can be asked on [Stack Overflow](http://stackoverflow.com/), but some will be off-topic there as well. Be aware that the answers you get here may help you understand the issues raised in your question but not eg explain the nature of the error or point out a handy function.
Let your data be $(x_i,y_i,z_i)$, $i=1, 2,\\ldots,n$. In the first regression, "y=mx+b", the implicit assumption is that the values $\\epsilon_i=y_i-(mx_i+b)$ are identically distributed but independent random variables (iid) with expectations of zero. The second regression assumes $\\delta_i=x_i-(m_0y_i+b_0)$ are iid. Plugging that into the first gives $\\delta_i=x_i-(m_0(mx_i+b+\\epsilon_i)+b_0)$. That simplifies to $\\delta_i=(1-m_0m)x_i-m_0b-b_0-m_0\\epsilon.$ Unless $m_0m=1$ the expectation of $\\delta_i$ varies with $x$ and even so its expectation is $-m_0b-b_0$; either is contradictory.
Just a good image reconstruction algorithm(The regions involved are often not convex). It's just that there are a lot of different ones, and I'm not familiar which ones are best in the context I'm approaching the problem in. (Easy to parametrize estimated surfaces, easy to compute volume, and easy to find minimum distance from a given point)
Thanks you so much for the suggestions.
@Georg Hmm I wonder if your problem is not a problem of complex models, but complex data and too simplistic models.  Or in other words under-fitting.  In the case above, what happens if instead of using $(F1,F2)$ you use $(F1,F2^{1/3})$?  The features can be either taken from the data or computed.
Is your variable VAR continuous or categorical?
Welcome to the site, @craig. It seems to be you're assuming the data are monthly, but that isn't specified in the question. What if the data were 1 value per year, for instance, would you still want lag=4 for quarterly? Since you're new here, it may be helpful for you to read our [FAQ](http://stats.stackexchange.com/faq), which has lots of into about site policies & how best to answer questions.
I think I've just spotted the issue here - you want to know for the purposes of calculating the Duan Smear factor correct?
What does it mean that follow-up time is right censored?  In the data your code generates, futime is 104 for observation 1 before the right censoring and 100 after the right censoring.  Does that mean that Y was truly measured at a follow-up time of 104 but the data record a value of 100 for it?  Also, why is follow-up time important?  Does Y change over time?
Extreme value theory studies such truncated distributions: 
asymptotically, the distribution of the tails usually belongs
to the [generalized Pareto](http://en.wikipedia.org/wiki/Pickands%E2%80%93Balkema%E2%80%93de_Haan_theorem) family.
One could also try to fit the data to this family of distributions and compare the parameters.
In practice unsupervised dimension reduction (based on substantive knowledge, not just thoughtless PCA) can often greatly improve predictive performance by specifying a plausible model from the start without overfitting, & with no need for subsequent shrinkage.
+1 for showing the code! Also, since the maximum mean of your silhouette occurs when k=2, you might want to check whether your data is clustered, which can be done using [the gap statistic](http://stats.stackexchange.com/a/74435/12359) (another [link](https://www.quora.com/Machine-Learning/How-can-I-detect-if-my-dataset-is-clustered-or-unclustered-i-e-forming-one-single-cluster/answer/Franck-Dernoncourt)).
And by 20 columns, did you actually mean 20 observations (i.e. rows)?
Focusing this question on a specific application will help generate helpful answers.  In its current state it's almost too vague for a response and is likely to be closed.
Does Feynman qualify as a statistician?
Thanks. In the example, my AR model has order 2 instead of 1. Is the prediction variance still equal to the  1-step ahead prediction error
Hi, Epi. The code looks pretty close, except I think `ff1` and `ff2` have misplaced parentheses (syntax error) and `sf2` should use `t2`, probably. Other than that, think of the lognormal just like it sounds: The log of the random variable is normal. So, if you wanted to do OLS, e.g., $\\log Y$ would be your response variable. That should tell you how the predictors should enter in your scenario. :)
This, and much more, is covered in the classical text E. L. Lehmann and J. P. Romano, *[Testing Statistical Hypotheses](http://www.amazon.com/Testing-Statistical-Hypotheses-Springer-Statistics/dp/0387988645)*, 3rd. ed., Springer. See, in particular, Chapter 5, *Unbiasedness: Applications to Normal Distributions*, especially Sections 5.3, *Comparing the Means and Variances of Two Normal Distributions*, Section 5.4, *Confidence Intervals and Families of Tests* and Section 5.5, *Unbiased Confidence Sets*. (cont.)
You might want to look at the example of income predicted by sex and education in  http://psychology.okstate.edu/faculty/jgrice/psyc5314/SS_types.pdf
@robert1 Sorry for the previous mistake. I thought you wanted to fit _a_ parametric distribution and _not_ mixtures. OK, before I suggest anything, can you please tell what your data set looks like - I mean how do you get your data, do you have any guesses about the number of mixtures there will be ... and other similar aspects.
I have tried it, but (IMHO) it lacks Inkscape's flexibility.
Baseline differences due simply to random allocation of participants to groups is not "bias". How do you know that the baseline differences are due to some systematic effect and thus more than you would expect than by chance?
Hey Andy I am still doubting over everything, but the pieces are coming together hopefully the next days. Keep you posted!
Also, it seems like you're computing the interactions ignoring whether the predictors are quantitative or categorical. It doesn't really have a meaning to compute an interaction including a numerical feature as is (creating bins may be a solution).
Also related: [Assumptions regarding bootstrap estimates of uncertainty](http://stats.stackexchange.com/q/11210/2970)
Here is a link to a good tutorial on how to perform canonical correlation analysis in [R](http://www.r-project.org/): http://www.ats.ucla.edu/stat/r/dae/canonical.htm
@Gavin (+1) Far more complete response than mine; I just like it.
@user1172468 I have updated my answer
The issue is not the form of the function. When parameters are treated as random variables, you are in Bayesian estimation framework, so no objective function and no maximization approach.
@FrankHarrell Thanks for that. I hadn't thought about the behaviour of tests based on ranks. However, the act of transformation does not affect type I errors unless one chooses the transformation with an eye to the test result. Choosing it on the basis of known properties of the data distribution doesn't lead to inflated type I errors.
Thank you for your acknowledging AUTOBOX . Your expert opinion means a lot to me. I will take his data and provide him with a step-by-step answer.
R routines can be run within SPSS (with installed SPSS R Essentials pack). But, if you want something written on SPSS native language and the piece has not been written yet, you may publish your request on https://listserv.uga.edu/cgi-bin/wa?A0=spssx-l&D=1&H=0&O=D&T=1 (the main community of SPSS'ers). Perhaps somebody agrees to code it, maybe for money.
You can do that sort of thing with regression analysis, it just starts to get really complicated. Doing a full regression analysis would require a package of statistical software as well as some sort of digital file containing all of the card info.
Hello Horst, thanks a lot for your answer, very clear. Now I am curious, if we take a set of data X1, ... , Xn that we suppose normally distributed and independent. If I compute the let's say 90% confidence interval for the mean with the classic t n-1 formula and then with my t n formula, do I get the same interval? I experimented on R and get slightly different results. The bigger n the number of samples, the smaller the difference. What is the explanation?
I have merged your account, but you should rather register it -- this way it won't get lost. You can do it here: http://stats.stackexchange.com/users/login
In over 35 years of teaching this stuff, I never failed to be amazed at how many students knew and could compute variances using $$\\operatorname{var}(X) = E[X^2]-\\mu_X^2$$ but utterly failed to realize that one could also find $E[X^2]$ as the sum of $\\operatorname{var}(X)$ and $\\mu_X^2$.
Great!  Thanks for all the effort that went into this, mpiktas.  I'm not familiar with R, but your code is explained well enough that I can still easily tell what you're doing.