tttjjj/elt_bench huggingface dataset:
Can you put some details in your question please, firstly so readers of your question are not required to go read a paper to understand the question, and secondly, to protect your question from the possibility that the link will disappear (at which point it becomes not so much a useful resource for others as junk cluttering up the site)
Your preceding question was closed because it was hard to tell what was asked. The present question add no more context. You should consider providing more information: what is the research question, what hypothesis you want to test, what data are available. Examples on the use of BT models are available on this site, if you are only interested in seeing R applications: [How to fit Bradley–Terry–Luce model in R, without complicated formula?](http://stats.stackexchange.com/q/26994/930). Without more information from your part, this question will be closed as well.
Skewedness itself is not a problem for most clustering algorithms. But it may be indicative of e.g. inappropriate normalization. But all these (sqrt, z-score, minmax, any normalization!) are just *heuristics*. Or have you seen a proof that one normalization is more "correct" than another?
Thanks very much for your quick response! Let me try and clarify. I have fitted a Cox PH model to the data and have estimated the impact of a bunch of time-invariant covariates on the log of the hazard ratio ln(h/h_0). From this, I can compute the baseline survival and cumulative hazard for each observation using Stata's basesurv basechaz options. What I am trying to do is make predictions for my current set of subjects. Specifically, I need to answer the question: of the X number of subjects that have not yet experienced the event, which proportion will experience it in the next 30 days.
Thank you Graeme!
Have you got a good idea of the underlying behavior of the process that generates your multiple observed time series? Do you know that there is a common process?
Sounds like a candidate for a community wiki.
First, you could try searching the site for "outliers", there is a lot of info there. If what you need isn't there, then you can ask a more informed and detailed question. Second, it depends on many factors: How many variables and of what type? How much data? What classification method will you be using? Choosing an outlier detection method without considering these issues will not give good results.
I did not exactly understand the second approach. You seem to have a discrete state and emission space. If you have Gaussians as the state transitions and emissions, then you get a Kalman Filter, which is not very appropriate where you have a discrete state and emission space.
How can I use correlation analysis to determine weigh?
It will not cluster the distance matrix. It will cluster the column indexes. Objects $i$ and $j$ are connected iff $x_{ij}\\leq\\varepsilon$. That's how distance matrixes work...
Perhaps it was downvoted because of formatting? I $\\TeX$ed the math in an attempt to improve the readability!
I am no expert on time-series, but assuming that you and @Bogdanovist are correct, which seems likely, what you could do is make a plot, based on predicting from the past, of how accurate the predictions are.
Wow, I didn't know that 10-fold CV should be repeated 50-100 times! I'll have to revisit my last project and try bootstrap testing instead. Love this website: I learn something every day!
Do you know about linear regression?
@guest Actually, WinBUGS has I() which is a censoring and truncation function. WinBUGS manual page 12. http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/manual14.pdf
Your headline and your text ask different questions. If you "know" that your original data is normally distributed then you can derive that several operations like summing up, multiplying by constants and so on preserve normality. When you say in the text of the question that you take the integral, what do you do? Sum up and divide by the number of summands? Then the result will be normal if the original data was normal. If you e.g. take powers then the results will not necessarily be normal anymore.
Either the raw or transformed data might work. You should probably try both.
Good answer, thanks! Actually I am interested in finding a number of meaningful predictors for a concept ('service quality') from a larger pool of possible predictors. I would like to avoid to overlook some significant predictors due to suppression effects or colinearity, which is why I decided to stepwise include or exclude covariates from the model. Please correct me if this procedure is not optimal. Also I am currently considering to summarize some correlated covariates using factor analysis first.
`assume we have 3 classes`. @SebastianRaschka: Then LDA will allow you maximum 2 discriminant functions. The number of eigenvalues in **LDA** is min(num_groups-1,num_features).
Additional suggestions, beyond the extensive information in the duplicate thread, can be found by a [site search](http://stats.stackexchange.com/search?q=confidence+binomial+difference).
@mbq Thanks. I'll give it a shot. I also had some luck in reducing the bootstrap sample size in the `randomForest()` function, since the `classwt` option seemed to have no interpretable effect.
In GLMs, as in linear regression, you condition on the IVs.
I accepted the multcomp answer primarily because I'm completely unsatisfied with the p-value adjustment techniques which the community selected as the "right" answer.  I glanced at it and it seemed promising, but I didn't investigate further.  I'd be interested in hearing more about what you are trying and what you are finding out.
Thanks for a good presentation, it was (partially) useful :-)
@Glen_b what's the formula for deviance?
After a couple of revs, it will be the best thing going. Its first beta release is already eminently usable and beats most other options IMO.
You can shorten/simplify your first line to `x <- rep( c("A","B","C","D"), 10000*c(0.1,0.2,0.65,0.05) )` and you don't need to specify the 10000 in the call to sample, that would be the default (though for clarity it does not hurt to specify it).
I know many problems not related to econometry which are used as a benchmark, but I want to focus on applications of GA in economics, and I thought that estimation of econometry model is good start.
Take a look at Trivedi's notes on [Poisson model with exponential feedback (sec. 23)](http://www.stata.com/meeting/mexico10/mex10sug_trivedi.pdf) and the Windmeijer paper he cites. They may be helpful.
I second this answer... and this really *is* my area (or at least I tell myself it is).
@Bogdanovist do you know any classifiers that do not generate biased probabilities ? And how would you re-calibrate the biased probabilities of a given classifier in order to be more accurate ? If you make a detailed answer abut this would be awesome.
+1, but the ones from NIPS2003 have train.labels only -- the NIPS2003 paper says clearly "validation and test set labels are withheld".
Good points, and I totally agree with your "postfix"-style ordering notation rather than the "prefix" style I had originally.
Okay, i'll look into it. Thanks!
$X$ is the random variable while in the pmf, $k$ indexes the values it can take. The random variable is denoted $X$ up to the point you observe it. Once you observe it (or in this case, generate it), it takes some particular value, $x$, say. You should feel free to call that observed value $k$ as long as you don't get confused between the observed value and the dummy over the domain of the variable.
(cont...) some other adequacy tests are: 1) plot the residuals 2) try overfitting 3) fit subsets of the data. 1) Eyeball the plot for changing variance (if found, transform the data, e.g. take logs). 2) Fit an AR(2), that is, overfit the AR(1) model. Do this if at the identification stage an AR(2) seemed plausible. Based on the AR(2) results, decide if the more parsimonious model, the AR(1), is adequate. 3) Split the data 50-50 and test if the estimated coefficients are statistically different (t-test). Also check if the coefficients differ by more than, say, 0.1 in magnitude. Hope this helps.
Thanks for this pointer into the literature on circular data. It seems like a good starting point.  I quickly skimmed this particular article, and it seems to deal with circular responses, rather than circular covariates.
By keeping $K$ constant, the proportion of success states in the population grows vanishingly small as $N$ grows large.
Hmmm... When wearing a Bayesian hat, I always thought of the likelihood function as the _conditional_ density function of the data given the parameter and not as a _joint_ probability density function. The location of the maximum of the joint probability density of the data and the parameter (regarded as a function of the unknown parameter $\\theta$; the data being fixed) gives the maximum _a posteriori_ probability (MAP) estimate of $\\theta$, does it not?
Thanks for the information and I will read the sections indicated above. I'm new to R so I dont know how to do a covariance matrix but I will look in related topics.
@bruce I'm sorry to insist but $N_3$ (as well as $N_4$) is *not* a Poisson process because there is a positive probability that $N_3(t) \\equiv 0$ (this probability is $\\Pr(V_1 \\geq c)$). Your definition of $N_3$ is not the same as Neil G's construction: with your construction $N_3$ has no event when $V_1 \\geq c$.
I am not sure what you want here. You have 30 people with a rare disease. Each has 20 features. Fine. What is it you are trying to find out?
You could try to smooth the series I guess, or otherwise model them, in order to obtain something like that, but my first question would be *why* are you trying to do that?
correct me if I am wrong: you population consists only of widgets that lasted for 1000+ days. Widgets that failed before then are not included, correct? *If a widget has survived for 100 additional days*, by this you mean the widget has survived for 1100 days, and you would like to construct a new survival curve for them. Or do you mean they enter the study at 1100 days?
(also @PeterEllis): These are good points; I hadn't thought about the autocorrelation issue. The slopes are my real 'measurement' (they represent a rate). Perhaps then it is conservative, but not overconservative, to simply do a one-sample t-test on the fitted slopes, like so: t.test(c(slope1, slope2, slope3), mu=0)
@Placidia I get the impression that the OP means that $R\\times C$ is too big for whatever software is being used and it fails to properly enumerate... but that's only a guess. The question is very unforthcoming with details.
See the section entitled "Details" in your link.
Ok this makes it clear. Exponential pdf can be used to model waiting times between any two successive poisson hits while poisson models the probability of number of hits. Poisson is discrete while exponential is continuous distribution. It would be interesting to see a real life example where the two come into play at the same time.
Have you looked at a latent class model? Basically, each decision-maker has an estimated probability of being given one of several choice sets. The coefficients for each set, and the coefficients for the latent class model, are all determined simultaneously.
I may be a bit off-topic, but why can you not accept a cluster that happens to land at the overall mean? Is it known or obvious that there is no cluster there and it's somehow an artifact of the algorithm?
Thanks, Artem! In fact, I am more of a pure computer scientist than anything else, thus my knowledge of neuro- and congnitive science is less than scarce. Sounds exciting, though, and since all roads are still open, I could delve into this, too - at least to some extent. At the moment I am particularly interested in applications to real-life engineering and data analysis problems that could work for benchmarking.
Me too. +1 for the paper.
Drat, I got the older version of the book! Thanks a lot for pointing out this source!
I know that I can find the derivatives in Greene, however, I should know how to derive them. I will work on it this week, then I will write more, since I have some achievements.
What would happen if you tried `param = gamma.fit(float(samp))` ?
`wmtsa` did the trick, thanks Wayne.
Please use code (four-spaces indent, `{}` button) instead of cite (`>` indent, `"` button).
This looks like a routine [textbook-style problem](http://stats.stackexchange.com/tags/self-study/info). Is this for some subject?
Actually, someone helped me with producing the classification table in this related question: http://stats.stackexchange.com/questions/4832/logistic-regression-classification-tables-a-la-spss-in-r Thanks for the example with R, much appreciated, I found similar instruction on the quick-r website. Although, for this puprpose I'm forced to apply CHAID in SPSS.
@ACD All depends on what "normalize" means but if that's just a linear mapping of [min, max] to [0, 100] then such a transformation has no effect on the (lack of) normality of the variable concerned.
Hmm. a question. Do i take the difference between pre test and post test as the "sum" in the example given in the link? thank you.
In the linked thread, I recommended Agresti's Intro book. That has relatively little math. I suspect that might be the best book for you. What is the book you're reading now?
Extremely informative perspective. I ended up using [this](http://www.mathworks.com/matlabcentral/fileexchange/38122-four-parameters-logistic-regression-there-and-back-again) Matlab logistic regression function for prediction and finding the asymptotes. Should anyone try using this function, one should note the x inputs are required to be sorted and the y inputs must be positive.
David, thank you for your answer. The answer is also what i have been thinking about, but i am not sure whether there is some theories which can support this idea, could you answer it for me?
Group one is not exposed to treatment but yes to environment so their behaviour is what is expected when the treatment is not present in a given environment. Once you introduce the treatment then you compare the behaviour in the same environment pre and post treatment. The resulting change is the observed effect of treatment in that environment. This is standard procedure in HCI/social sciences research.
your edit still does not answer why you want to get rid of that tail, what are you trying to accomplish with this data/plot?
Sorry @Denis, I didnt notice it was wrong. I fixed it. Thanks.
This sounds promising; I will test this approach and accept this answer if it works. Thanks!
I think I'm missing something (i still don't quite understand your data), but sounds like you might want a linear mixed model (LMM) = mixed-effects model. I'd look at some code examples and see if this fits your situation at all: http://www.ats.ucla.edu/stat/r/dae/melogit.htm
This is relative. Large main effects relative to the interaction, especially when the independent variables have genuinely limited scales (like sex variables) most definitely are meaningful even if there is an interaction.
@cardinal : I just described the data, where the expected value is 0 but the observed doesn't have to be. It's what OP gave us (although on second thought it does indeed sound rather irrealistic). Hence adding a little bit to the p value to make it highly improbable instead of impossible will help, but even then the chi-square is in this case invalid due to the large amount of table cells with counts less than 5 (as demonstrated by the code). I added the consideration in my answer, thx for the pointer.
Both the qq plots and the histograms look pretty skewed to me.  Look at the right tails on the latter.  Normally I think it's best to just trust your eyes, since 1) the issue of whether or not a distribution is normal often isn't terribly important (what usually matters is how far it deviates from normal), and 2) normality tests typically reject the null if you an appreciable amount of data.  However, in this case I'd say your distributions are not normal.
Maybe a better way to phrase it would be "integrate *over* the missing data" or "integrate *out* the likelihood of the missing data."
Thanks! But I encounter a problem while triying to use the code... what is the b value in: dK.LIML = sort(eigen(b %*% (t(mYStar) ....? Where did you define it? Thanks!
This makes the most sense so far to me in terms of a formal distinction, although ESL didn't really present linear regression this way. They introduced the squared-error cost function first, kind of arbitrarily (instead of doing an MLE for a Gaussian), used it to find that we should predict f(x) = E(Y | X = x), explained how KNN approximates this under certain assumptions, and then went on to assume f was linear to get linear regression.
Is the total number of trials huge? If not, you could try to replace each data line by n ='trial' lines and a 0/1 response variable.
That is a great intro list
Do you assume that everyone who would fail both post tests would fail the review?  Also is the review the same for both A and B? Did any students who failed either post test take the review?
@gung I apologize if I was unclear. I did mean a variable measured on a ratio scale, not a ratio variable. Also there is only one "pre" and one "post" measurement for each subject.
One approach is to find the smallest t-value your table contains (at the right d.f) and say that the p-value is larger than that; this would probably be sufficient. However, tables come in many different formats and it would be best if you indicated how the table is organized. Another approach would be to compute an approximate p-value from a numerical integration of the density (Simpson's rule on the 3 points (0,t/2,t) gives better than 4 figure accuracy on your problem). A third approach would be to use a statistical package or online calculator.
@boucekv : a discussion of **all** prerequisites would require a book. The short answer is: it depends on what you exactly want to do, as that defines what _correct_ means.
@ Gaël Laurans Thank you for your guidance.
@TooTone Another approach which you might like is tantamount to the [Holder Inequality](http://en.wikipedia.org/wiki/H%C3%B6lder%27s_inequality): from the axiomatic relation $||A_n-B_n||^2\\ge 0$ it is straightforward to deduce $2Tr(A_nB_n')\\le||A_n||^2+||B_n'||^2$, whence upper bounds $\\alpha$ for $\\{||A_n||\\}$ and $\\beta$ for $\\{||B_n'||\\}$ give an upper bound of $(\\alpha^2+\\beta^2)/2$ for $\\{Tr(A_nB_n)\\}$, etc.
Do you have a lot of data, but little labeled data? If so you might wanna look into semi-supervised learning. Using some of your unlabeled data should both increase your accuracy and decrease the risk of overfitting.
I need a more concrete answer than an "I think"
I would add that manual variable selection presents almost the same problems.
Ah, yeah I see. Thanks for your answer. :)
@mpiktas Or just init the plot with a single call to `plot()`, possibly with `0,0,type="n"` if there is nothing to plot yet... Indeed this is much easier than ding `plot.new()` and adding all the stuff like axes or labels by hand.
I have no idea what you're asking now. Perhaps you should say what you think the bootstrap is. I highly recommend reading the book that the `boot` package goes with. Are you confusing the terms 'observation' (a data point for a single observational unit, like a person - possibly with observations on several variables) with 'sample' (a collection of such observations)? Please note that when you resample with the bootstrap you take your pseudo-samples of size $n$; in the notation of the function, you take `R` such samples. You don't need to specify $n$ because it can see how big the original is.
Bowler, not pitcher.
@Michelle please see new details in the question
You needn't worry about the lognormal issues: the multipliers are correct regardless. (There would be an issue with heteroscedastic models.) This is because $E[Y]=C e^{\\sigma^2/2}e^{(X_1W_1+X_2W_2+X_3W_3)}$ where $\\sigma^2$ is the variance of $\\log(Y)$.  BTW, please scan your definitions of the $M_i$ for typos.
@MrAlpha That depends on *how* your variable works as a confounder, and your effect estimate to begin with. But it is *not* an uncommon occurrence, and should always be considered.
I agree w/ @Momo that the linked question seems to cover the required territory. I respect that you believe it doesn't, but in that case, you should edit your Q to clarify what you want to know more specifically & how it is distinct from the other Q, otherwise, this question will end up being closed.
Hi, thanks for the answer, you are probably right about Excel and R. I'm new to R, so could you make the example complete by showing what format should the data (S, X and Y) be provided so that the statement above would really run? I'll accept the answer then.
I really appreciate your help...Thanks.
@BurningJade I just made an edit to my answer -- to say that all your formulas look correct. If you can't get the answer you think you should, something else is amiss.
Do you have a question that you are trying to answer ?  If so, could you explain the context of your data (ie what are coke1 and coke0 ?)
? You get larger absolute deviations when using stepwise, compared to not using stepwise. Not sure what you're asking. As for standardisation: it's unnecessary if you do it as I posted, ie making a comparison between the stepwise and non-stepwise approach. Each variable is being compared from one treatment to the other, rather than to the other variables.
For a number of reasons, that Wikipedia page needs some significant attention, or, at least, some TLC from a knowledgeable statistician or probabilist. While not the only example, the statement regarding the K&L approximation would make any mathematician cringe since the absolute relative error is obviously (i.e., by inspection) nearly 90% in the tails! :-)
@Nick Correct ! We just use the language here as we really don't know it .
Interpreting the model is up to you. :-). I don't know what any of the variables actually are, and you are the one who knows the substantive area. The control variables, statistically, are there because you want to, well.... control for them!
I agree with gung. Since the second model is multiplicative all the way through including the error term the log transformation on both sides fits it into a nice linear model. Whether OLS should be used in this case depends on whether the lnε$_i$ terms are approximately normal with 0 mean and constant variance. In particular that implies E(ε$_i$)=1.
@Macro: It would force nonnegativity of the individual $\\sigma(t_i)$, but not necessarily for all $t$. (Unless I'm missing something.) :)
Often people will see something vaguely symmetric and assume it looks "normal." I suspect that what @Ross saw.
"*from what I gather, the outcome variable has to be a binomial*" - your impression is inaccurate, as the examples under `?glm` indicate (there are Poisson, Gaussian and gamma examples).
@John Sure; for the second thing, I'm afraid I don't understand... how to call it in a report?
I am alluding to your extra machinery for "Level Shifts, Seasonal Pulses and/or Local Time Trends".
@DimitriyV.Masterov Stata and R both let you specify _an_ offset. They don't put restrictions on what variables you use as the offset, because there's no reason to do so. If you're talking about using a _log-transformed_ variable as an offset, neither language has an option to treat that as a special case, compared to using any other transformation (or none at all).
This is exactly the secretary problem. At HCSSiM, the lore was that this was first presented in a talk titled 'sex and the single statistician.'
Hi John. I just saved the picture and opened it in a graphics editing program (I used paint.NET but Windows Paint would also suffice). I then selected the relevant area and selected "crop to selection" so that all the white margins were cut off.  Then I saved the picture again and uploaded it.