tttjjj/elt_bench huggingface dataset:
For a density the choice of bandwidth is always somewhat subjective.  It is a question of what is too narrow and therefore causes variation in the curve that is essentially following the noise vs too wide where the curve is too smooth and misses some real features in the curve.  But you estimate the density to find out the shape.  So how smooth the estimate should be is not easy to know.  For derivatives I would think it depends on what feature of the derivative you want to know about.
Bayes theorem brings up a whole set of other questions as you have to make assumptions about the high and the low. If you don't have data for the entire population, wouldn't it be unsafe to make any hard assumptions about what the limits are?
Is there a reason why you would not adapt something something simple like the $\\sqrt(n)$ rule or Sturges' formula to your problem directly? Along each dimension you have the same number of readings anyway. If you want something a bit more sophisticated (eg. Freedman-Diaconis rule) you could "naively" take the max between the number of bins return for each dimension independently. You are essentially looking into a discretized (2d) KDE anyway so maybe that is your best choice anyway.
I saw it couple days ago, since I cannot vote up now, thanks for your attention
One problem, as I see it, is that your "plot" varies with all the non-spline(d) variables.  You don't really have a curve to plot, as it were, you have many curves depending upon the values of the other variables.  Does this make sense? (Not sure if I've understood the issue correctly...)
Assuming a large sample size generated the NS p-value, I would go for option (1).
Added inverse of t per your request (y inverse is too distributed to see anything at all)
I see what the intent is and it seems corect to me; however, look in my next answer what happens (I enter a new answer as I cannot write code here).
@ttnphns I agree that Helmert contrasts don't constitute dummy coding from a technical standpoint, but I think it's perfectly reasonable for them to be included here nonetheless.  I can't tell if you are pointing this out for the sake of clarity or suggesting that the answer be changed.  @ Aaron +1, this answer would be even better if you were to explain briefly how the interpretation of these these different coding schemes would differ.
Oh, MAN.  R-style data structure is totally new to me (I'm used to SPSS) and I didn't even think of that.  Thanks!
oops, just spotted the definition says the epsilons are independent of X.  So it simply isn't possible as described in the paper.
You can see from the [formula of Gamma](http://stats.stackexchange.com/a/18136/3277) that pairs of cases 1 and 2 where X1=X2 or Y1=Y2 - they are called "ties" - are simply ignored by Gamma. This is why Gamma can attain 1 or -1 in the face of their presence.
Do you have any citations where this occurs?  I am interested in seeing where they actually interchange the two.
One note: PCA is not the same as factor analysis. PCA is a data reduction method, FA is an attempt to find latent variables. They often (but not always) give similar results
Thank you Henrik - the distinction between the definitions (and your thoughts about them) was helpful.
What are your research questions? What do you want to confirm/claim/reject?
Apologies I misconstrued my question, I have corrected now
@Aniko I mean using a 4-knot spline *and* OLS.
Perfect, hadn't seen any references to the nbinomial ml family. That did the trick.
I mean by online (stream), a growing times series coming from a sensor.. each one hour a set of data (vector) is recieved ..
Each "question" is just a single multiple-choice question. I'll add this to the post.
That is quite analogous to what I´m looking into. I´ll look into their work. Thx for the input.
LDA produces _min(n,c-1)_ discriminants (_c_ is the number of classes). They are sorted descendingly by their discriminative contribution. You may leave _d_ first discriminants, exactly like you leave first _d_ components in Principal Components analysis.
What do you precisely mean by normalization?
@MichaelChernick That may not be an issue too much if the groupings are defined a priori as per my example. That would be typical of the sort of data described in the paper Margaret cites.
for recursive sequences function `filter` looks nicer than the loop
Why does the fact that the survival curves never go below 80% mean they aren't informative? If there is a big gap between them, that would inform.
Why the negative vote? This seems like an interesting question (certainly one several people thought possible to answer).
Hey guys... just poking... does anybody have an idea how to? should I rephrase the question? Cheers, C.
I asked the authors but they said the data can't be shared due to restrictions. Do you know of any publicly available datasets?
+1 for "However, more broadly, doing exploratory data analysis (e.g., plots, summary statistics, etc.) to understand the distribution of all your variables is important".
You appear to have created 2 accounts for yourself & re-asked your question. This seems to differ very little from your [previous question](http://stats.stackexchange.com/questions/27786/analysis-of-part-of-likert-scale). If answers provided do not give you what you need, you should edit the existing question &/or comment on existing answers, not ask again. You may want to read our [FAQ](http://stats.stackexchange.com/faq).
Also, the question you suggest as a duplicate is something of an "opposite" -- it asks what to do when ANOVA is significant but post-hocs are not. In this case I was asking how to interpret it when the ANOVA is non-significant but the post-hocs are.
But MANOVA is a parametric test, isn't it?
I meant significantly different results. I also think there is none, at least real-world example. Still, I think I'll wait some time more.
@Nambari You will want to have a way to deal with remainders in step 2 and 3 for seasonal data or else you will only be able to predict X steps at a time, where X is the length of your period. I fixed this at line 180 of my code. That is the only different thing
Converting it to another scale doesn't make it continuous.  That's actually kind of an absurd idea.  Then you'd just have intervals in a different scale. You have a variable that you can treat continuously as long as you don't take fractional scores as meaningful in discussion.  In practice no variables are continuous since there is always some limit to the ability to measure.  This is a relatively course continuous variable is all.
Do you need to know how to run a t-test in your software? (What software are you using?) Are you wondering what a t-test is / how it works?
@DoctorAmbient The (0,1) variable is the DV (the $y$), while the Likert scales are the IVs (the $x$'s).
I still have the feeling that you are jumping to the specifics of a particular technique instead of explaining clearly what data you actually have and what you are ultimately trying to achieve. Or are you just trying to learn about this type of things? In any case, you could perhaps ask other questions focusing on this, it sounds like a completely different problem to me.
what where the possible answers? Maybe a non-parametric paired test (e.g. Wilcoxon signed rank test) would be better.
"Distribution over possible probabilities" could benefit from clarification: probabilities of *what*?  The type of coin that is drawn?  The chance that it will come up heads?  The chances in a sequence of trials involving this coin?  The chances in a sequence of trials where each time a new coin is chosen?  Also, is anything special meant by a "possible" probability as opposed to just a probability?
The fitted values from an ARIMA object are one-step forecasts. They are not very interesting and you don't usually need to look at them. The residuals, on the other hand, are useful for checking model assumptions and to see whether the information in the data has been adequately captured in the fitted model.
Is this paired binary data? Are you just looking to see whether there's a change in the "correctness" probability before vs. after?
By "singular", the author means "nondifferentiable."  The penalty function's derivative is bounded away from 0 on either side of $\\theta=0$, and has opposite signs on the two sides, so the derivative can't be defined at $\\theta=0$.
Ah, yes. Sorry :)
Here are some : American Statistical Associates (ASA) ;The R Project for Statistical Computing ;INFORMS;Economist Intelligence Unit (EIU);•	Autobox User Group - Forecasting & Time Series Analysis, ARIMA, Outliers, Transfer Function & more!;Institute of Business Forecasting & Planning;International Institute of Forecasters;Global Analytics Network (+5K analytic professionals)
@whuber I agree with you, my post was rather simplistic (I just found it interesting!) Two other points of interest. First, in the report it gives different birth rates classified by race. Secondly, I'm a bit confused why the Monday birth rate isn't much higher (since the rate is low over the weekend). Instead, it seems Tuesday is the day of choice.
Thanks so much Glen_b.
I didn't vote the question down because the OP is new to the site & I didn't want to discourage him.
+1 The comments in the "preview" [linked through your reference](http://demand-planning.com/2010/03/18/can-forecasting-help-me-staff-a-specific-hewlett-packard-call-center-at-1030-am-on-a-friday/) directly address this question: one comment addresses how to use a forecasting model for managing the call center and the other explains why having a good model (that accounts for the factors you mention here) is important.
Produce some sample data.
@chl, yeap, thanks, I wasnt clear. My questions is really this:  If I write LM in R does R understands it as linear always or triesd to fit any model, not necessarily linear regression but any regression ?
Thanks for the introduction of VAR, I'll go into that later! What I wanted to do is trying to predict the sales from the first half year's data with the "hints/knowledge" hidden behind the data of the last year.
No, it looks fine. When you say 'why it looks like this', which particular features are you concerned about? What is it you think it should look like?
(+1) too. The Data Browser in Stata is not perfect but it's one of the reasons why I launch Stata rather than R(Studio) in many circumstances.
<comments about closing at SO removed>
Interesting approach, but what does it mean exactly "the area defined by the integral is orthogonally opposite to the alignment of the contours"? Thanks.
In that case, I'm gonna double-down on "you're retaining too many PCs and should use fewer". I wouldn't be too surprised if dramatically reducing the number has a minimal effect on your classifier's performance too.

(By the way, I didn't mean to impugn the quality of your data--it actually sounds pretty clean. I've been dealing with biological stuff and there's no way it's 99.9% (or realistically, 90%) signal.)
:) trust data, but not your suggestions. By default R treats first level (male) as 0 and the second (female) as 1. Note, that this is gamble spending. Males are much more risky gamblers than females.
@Andy Not people: the number of connections made.  The important thing is that there should be a good chance that the number of connections made by individuals actually varies and doesn't settle down to a constant.  When that happens, the limiting distribution (of the number of connections) is determined by the finite number of initial connections that do vary, so it's not possible to approach a Normal distribution asymptotically.
This question might be more suitable for stats SE (crossvalidated). In general, there are many methods to do this kind of forecasting, ranging from regression methods to machine learning algorithms (like random forests or Bayesian networks).
Community wiki?
What can I do to increase the chances of someone answering this question?
Lots of great and interesting answers, but this answered my questions the most directly. Thanks
@juampa technically, it's the squared euclidean distance: it's the sum of squares is minimized by k-means. The square root just happens to be a monotone function, so coincidentially, Euclidean distance is also minimized.
re 1) you want the central limit theorem as it applies to sums (it's more often applied to the mean but applies to sums too as the mean is just the sum/$n$).
@lsanchez I needed to have very strict confidence intervals for the data and as I have a high-performance computer at my disposal I thought that there was no harm.
@Luna Common practice for condition 1 is to look at the model residuals. (1) Check to make sure there is no significant autocorreltaion in the residuals (time taken as the order of observation if the data are not sequenced in time). (2) Subdivide the X square into regions and calculate the sample variance for the residuals in the regions.  Plot them to see whether or not they are nearly equal.  If in addition you want to include the normality assumption test the residuals for normality by goodness of fit (e.g. Shapiro-Wilk test) or look at the QQ plot as you have been.
@Mike, you can use the K-S test to compare two empirically derived distributions, see Charlie's prior answer and comments http://stats.stackexchange.com/questions/2918/lorenz-curve-qq-plot/2929#2929
Thanks so much! What I mean by non-parametric is, that the means of measurements pr. subject are not normally distributed. The suggestions are good, but somehow I feel like I'm losing information, if I just average the measurements for each subject. The subjects where we have many measurements should carry more "weight" as they do contain more information. And the data is randomly missing. Hm.
This question appears to be off-topic because it is about finding a data set, rather than doing statistical analysis
Not sure whether the $n$ large c.i. is mean to be $\\bar x\\pm1.96 \\frac{s}{\\sqrt{n}}$ or $\\bar x\\pm 1.96\\sqrt{\\lambda/n}$. Not confident enough to accept or reject edit.
I think, going back a step, we need to clarify your research question "whether the locations are randomly distributed or not".  Perhaps you actually mean "distributed in accordance with a uniform random distribution or not".  Is the uniform distribution actually the crucial aspect of your null hypothesis?
Averaging deviations is a bad idea because positive and negative deviations cancel destroying any picture they might give you of the variation.
$\\sigma$ are the singular values of the original matrix. Look up Moore-Penros Pseudoinverse
@AdamO It is really a *randomized* test: the decision rule is not deterministic. If $X>c$ is observed, we reject $H_0$, if $X < c$ then we don't reject, but if $X=c$, then we reject with some probability $p$. This can be useful when the test statistic is discrete to achieve a type I error exactly equal to the preset $\\alpha$.
@Glen_b: Because I don't want to just use the everage to impute the data.
@Kyle. I am also cross validating as you discussed but I want to compare how divergent two classifiers' predictions.
2/2 They are probably looking for a simple categorical regression model of the form: 
reaction time = intercept + coefficient*Brand1+ coefficient*Brand2, etc. Under this framework, if the brand didn't cause a reaction within the 1 hour time frame, it's just counted as not having reacted at all. Realistically, the brand in question may react, but it takes longer than an hour.
@Graeme Walsh The data given here appear to be background data. The original poster did not ever post any of the time series. My own instinct is to wonder whether 6 + 9 data points is very much for the time series analysis recommended elsewhere.
$\\psi^{(m)}(x)$ is the psigamma(x,m) function in R.  Also, for $m=0,1$ R also has the digamma(x) and trigamma(x) functions.  A reference for using KL divergence can be found for example, Bernardo, J. M. (1987). Approximations in statistics from a decision-theoretical viewpoint. Probability and Bayesian Statistics (R. Viertl, ed.). London: Plenum, 53–60.
@mpiktas Many thanks. I wanna describe the trend of concentration by time, or draw a line graph sth like that, so how to make the average at d0 and d6?
Hello Sarah, you should close this question if you think it is answered.
For some reason, the Internet has an amazing amount of misrepresentations of hypothesis tests.  Over a decade ago I collected a few of these and [offered criticism](http://www.quantdec.com/envstats/notes/class_13/tests.htm). (It was perhaps a little harsh--every one of the statements I quoted had a grain of truth to it--but somehow I am not surprised that every one of those links is now a 404, while most of the links I found useful are still live.) That criticism may help resolve parts of the present question--but no doubt will lead to further questions :-).
Thanks for your comments. In this scenario, do you think the female diabetic hypertensive patients should have lower mean onset age than the non-diabetic female hypertensives since the slope is steeper for female diabetic hypertensives?
Just use Correpondence Analysis procedure found under Data Reduction submenu. Read Help and Tutorial (under help menu) if necessary.
And regarding the third point, in the first part of it, when I'm asked to estimate $T3|T4=65.234$,I think that $E[T3|T4=65.234]$ would be equal to $65.234 - E[X3]$ ,since $E[T4]=E[X1+X2+X3]=65.234$, from that I can obtain that $E[X1+X2]=E[T3]=65.234-E[X3]=65.234-20=45.234$. Then, for the second part, estimating $T3|T2=26.1,T4=65.234$, I think it would be calculated as follows: $E[X1+X2+X3]=65.234$, and knowing that $E[X1]=26.1$, then we have $E[X2+X3]=39.134$. In the previous step we estimated $E[X3|X4=65.234]$ as 20, so E[X2] would be estimated as 19.134...Somehow I feel I'm missing something
For what size of dataset?
+1 for making a clear and useful distinction.  However, it is difficult to see how a model assumption could apply to the *residuals* whose probability distribution, after all, depends on the very method used to estimate the model. As far as I can tell, about the only sensible way to interpret the homoskedasticity assumption is in terms of the errors. Assuming the errors are *additive,* it is immediate that their variance equals the conditional variance of the response variable.
What kind of model? Also, $R^2$ is not a measure of prediction accuracy, exactly. I suppose it could be thought of as a measure of within-sample prediction accuracy, but predictive power often refers to out-of-sample prediction accuracy.
@MichaelChernick thank you again. In your last sentence, what do you mean exactly by "high risk" subjects? The risk of failure eventually? or the risk of failure at any time? For example, suppose subject A is ranked higher than subject B, we then know that the estimated survival time of A is shorter than B, does it also means that the failure risk of A is larger than B at any particular time _T_?
Jenson=Shannon divergence provide such an estimate: https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence.
Is this a homework assignment? If so, it should have the homework tag.
what is the question you'd like to answer?
Sounds, like an interesting solution.  I'll have a look into that!
Wonderful! Thank you very much for this great answer.
Hi Sid - thanks for your response. T here is a treatment vector. It could be binary, or a continuous dose. Likewise, Y could be a binary or continuous outcome variable. X has K columns (it does not include T) and N rows. Y is N*1, as is T. 

I'm not precisely sure how doing a PCA regression would help me. Perhaps I didn't phrase the question well: Ideally, I'd like to rank-order my data based on who has the greatest treatment effect (an unobserved variable). Ranking on sub-group correlations between treatment and outcome seems a crude (but possible) second-best.
Thanks so much for your help and thank you for laying out the answer so clearly. I really appreciate it.
Are there any books that you would recommend for analyzing categorical/survey data in SAS? I know there is an abundance of literature on this, but some sources are better than others...
The signed rank test is for two samples of paired data, while the Kruskal Wallis is for cases where you have more than two samples but don't have pairing. But even the Wilcoxon rank-sum test (i.e. the Mann-Whitney) doesn't exactly correspond to the Kruskal-Wallis. See the discussion under item 3 [here](http://stats.stackexchange.com/questions/80196/datasets-constructed-for-a-purpose-similar-to-that-of-anscombes-quartet/80200#80200), which reveals a surprising outcome (non-transitivity) of testing several groups via the Mann-Whitney pairwise.
Your use of 'anchor item' does not correspond to what it ought to mean in psychometrics. I would suggest to update your title to reflect that this question is about wording of response categories or item format that changed over time.
Hi there, is this homework?
I'ts also in Knuth's Graphbase book, p. 392. For a Python recipe, see http://code.activestate.com/recipes/576564-walkers-alias-method-for-random-objects-with-diffe/
Depending on what your definition of "data mining" is, the answer can vary from "extremely useful" to "almost useless". There are numerous questions with thoughtful answers on related matters here on this site. If you haven't had the chance to peruse them, I'd highly recommend that and also editing this answer to clarify your ultimate objective(s). For example, if you have any interest in pursuing graduate education in economics or an allied field, I would *strongly* recommend taking such a course independent of any interest in "data mining".
There's not enough information to say much of anything. How does the Bartlett test work with time series - doesn't it work on grouped data and rely on independence?  Why are you transforming, exactly? What does the variance of residuals look like both of the original and the transformed data?
@Galit Your post is from September 2008. It might be worth trying again. Alternatively, use the read.spss() function in R.
The optimal point depends very much on your application. In mass spec proteomics we usually use a hard 1% FDR because we want very little crap in our positive results, other applications want to get as many good examples as possible, so they use higher thresholds. Anyway, this question would be better suited for CrossValidated.
If you need a hierarchical decision scheme you might use classification tree analysis (also random forest, based on it).
That's all fine -- you can add "multilevel" to your list of the ways to refer to this structure, too. "Clustered" usually means that the observations are known to be correlated, but nobody cares to model that correlation since it is not of primary interest, and gets away methods that are robust to such correlation, such as GEE. What you have is something like repeated measures MANOVA. There's a Stata package `gllamm` that thinks about your data as a multilevel/hierarchical data, but most other packages would think of multiple measurements as variables/columns, and samples as observations/rows.
Thank you for your answer. Where can I learn how to do SMC and which R package would you recommend?
@chl : that's exactly what I want to achieve.
I don't know where you get your list but OLS does assume no error in variables, otherwise why would you minimize residuals in the y direction?  Homoscedasticity is also a key assumption.  The error term is assumed to have a variance that does not depend on the values of the covariates. Variables can be correlated but a high degree of correlation between covariates does cause problems with the coefficients. The error terms for the observations are assumed to be independent of each other.  So no autocorrelation is assumed!
@SRKX, I don't know if it is LaTeX, frankly. As I said, I took it from UCLA website, and they may have produced it in AMOS. I usually do my path diagrams in Dia software (and I have a little tutorial on how to use Graphviz, the engine behind R `sem` package, on my webpage), but I was sure I could just google a decent 2-factor model in three seconds.