tttjjj/elt_bench huggingface dataset:
This question will need modification to be answerable. First, please read our [faq] concerning homework questions. Next, edit the question to reflect what you have tried and where you have gotten stuck. Please recognize that it comes from a narrow specialized context in which "decision procedure" cannot possibly mean what it usually does: your class must be focusing on certain kinds of procedures and have in mind only one specific utility function. So you also need to provide some background about what kinds of answers might be expected.
There is a large body of work on topic modeling, which sounds like what you want. The most popular model is called Latent Dirichlet Allocation (LDA), there are also several extensions, the most relevant to your task would probably be "Correlated Topics Models". There is free software implementations of both on David Blei's website.
Quantities like this appear routinely in survival analysis.  By convention, this theory deals with $-X$, so that its version of $G$ is $\\int_t^\\infty x f(x) dx$ = $\\int_0^\\infty(x+t)f(x+t)dx.$ This is closely related to the [expected future lifetime](http://en.wikipedia.org/wiki/Survival_analysis#Quantities_derived_from_the_survival_distribution).
What is the meaning of the y-values? Why would a peak in a curve have a mean and standard deviation unless it is a density?
Have you done any study on the kinds of variables that you have and how they can be treated?
When you estimate the frequencies from the data (as with Fourier analysis) and then include them as sin/cos terms in the regression, their p-values will be meaningless.
No. FA is not regression. I am referring to a response variable regressed against the principal components computed from a large number of explanatory variables. The principal components themselves are closely related to the factors of FA.
Neither answer is a correct implementation of your formula, since it's obvious from just the last cell alone that the actual chi-square statistic must exceed $1.776 \\times 10^{7}$! It's a little difficult to guess what's wrong with either calculation without seeing what you did in both of them.
I'm having a hard time with this question because it doesn't give sufficient clues what "quality" might mean.  I would challenge the implicit assumption that humans tend to *agree* on their "instinctive, quick" judgments of quality. Therefore, whether this question has any answer at all comes back to the issue of quantifying "quality."
@ttnphns yea, well I extended the example. Can you point out where exactly I'm mistaken. I think I need to put the number of observations per cluster and/or the total number of observations in there somewhere, but - if so - how?
@steffen: the RF algorithm used in the Kinect is describe in details in this [paper](http://research.microsoft.com/pubs/145347/BodyPartRecognition.pdf) (received best paper award at CVPR 2011) - I couldn't find any simplified explanation of it online, but I'm willing to write one tonight and post it somewhere here on Stack Exchange. If you think it's a good idea, let me know an appropriate place in SE where I can post an illustration of the algorithm.
Ah, the Dirichlet seems quite interesting in that it is on the simplex we are interested in, as you mention!  It seems that the other two you mention might introduce some asymmetry on $x_n$, right?

My collaborator and I will work through the energy function implied by Dirichlet tomorrow and will report back!  Many thanks for your patient help thus far -- this is far from our usual field but if we can work it out the results may provide a considerable step forward in geometry processing! [And of course we'll provide you with due credit!]
This question has a very short time frame, so answering it even with a great list of courses that will be offered in the next couple of months won't help the "eternity" knowledge. Most MOOCs can be audited later, but to receive credit, you usually have to enroll concurrently.
(+1) Thanks for a really useful reply on baysian methods. Also the P(X<Y) link answers another issue i've been wondering about in the same analysis.
+1. Also, in case it's helpful, the term to google when comparing strings is "edit distance": http://en.wikipedia.org/wiki/Edit_distance
Did you ever made an implementation of C5 for Matlab? If you did, it would be extremely useful. I am looking at interfacing C5 with Matlab right now, but C5 seems a bit messy (cannot just pass text files to and from it since it uses difficult formatting).
This seems like a duplicate of http://stats.stackexchange.com/questions/109704 (currently unanswered, alas), but perhaps the conversation in the comments there might help get you started?
The points are the data.  The curves that you fit to the points are not the data.  So if your intent is to show the data....
Thanks. I try :-).
**limits and legal reasons for not providing low numbers:** legal reasons would not forbid giving (additional) more verbose information like raw signal, corresponding concentration and confidence interval / judgment of the measurement (e.g. "below LOQ"). Also, you may ask the analysis lab for the calibration curve. I'd expect that you have to pay for that as it is extra work, but I'd expect this to be possible. The cheaper compromise may be that they give you all teh raw data and leave the data analysis to you. It may help if they know you're statistician/chemometrician/analytical chemist/...
@Guest If a variable has a mean of 0 (this is what it means for a variable to be "centered" about its mean), then its sum must also be 0. After all, the mean of a variable is just computed as its sum divided by something. So if the mean is 0, the sum must also be 0. Make sense?
I don't use LabVIEW, so I can't answer your question.  In R, look at help(scan) and check your OS syntax for setting up a pipe.  In a Linux-like system, you would set LabVIEW to output to what appears to it to be a file, but is really the standard input to R.  In R, you use the scan(  ) function to read (and parse) the LabVIEW output into a data frame.
(+1) Concise and understandable answer.
My English isn't native, but "partial" for me has something to do with "specific" or "particular", while "part" is simply "piece". This is in line with the differences between partial and semi-partial correlations.
Are you sure no separate testing was done?
You can't determine that observations *are* from a given distribution. You may sometimes be able to be pretty confident they *aren't* - and in some circumstances, completely certain of it (e.g. if n=10, none of 11, -5 or 3.4 are even possible), but since something that's not binomial may be arbitrarily close to binomial, no test can tell you that they *are* binomial.
Are you trying to calculate the variance of y_pred in Deming Regression. I am sorry, its not clear to me.
However, what happens if I correct for heteroscedasticity by means of clustered standard errors, even though there is prove that the initial results are homoscedastic. Is that a severe problem?
If you entered 1500 each for 4 sides of a 4-sided die, why would $p<.05$?
If the bins are labelled '1', '2' and so on, this is now discrete data; and the counts are in effect giving a histogram. You can work out which quantiles correspond to which bins easily enough - just cumulate the counts and divide by the last count, and you have what is in effect the entire ECDF.
@David Consider even the simplest non-trivial setting with 2 by 2 matrices.  The eigenvalues are roots of a quadratic equation whose coefficients are the trace and determinant of the matrix.  You would first have to propagate a 3-variate error distribution through these operations (which is easy for the trace and not easy at all for the determinant) but then you would have to propagate the resulting bivariate distribution through the root-finding step, which can be done only approximately, not analytically.  Even in this case, simulation is the way to go.
The mean at different levels of the repeated measures factor could change, even when r = 1.0 between scores at different time points.
I know you said "for learning statistical methods in general" but if you specify exactly what level of learning that you're at or aiming for (e.g. first year undergrad, masters, etc.) then it may be easier for people to leave you good / more specific advice. Essentially, some books are good for people at certain stages of learning - and not so useful to people at other stages.
My advice would be either to use LibSVM or to look for implementations of [Lin's revision to Platt's solution](http://www.csie.ntu.edu.tw/~cjlin/papers/plattprob.pdf) or to follow the linked directions on the Liblinear website.
(+1) Better than my comment.  Any classifier.
1000 samples. and about 1-5: in the question "how likely to be caught in case of fraud" the answers are "not very likely, not likely, likely and very likely" (actually it is 1-4).
Excellent links.  I think two important messages for me to pass on are: we need more automated data checking and I need to start explaining about separating data entry and data presentation.
It would be far better to use an approach based on good statistical principles.  lasso and elastic net are two worth trying, or just use ordinary quadratic penalization and keep all 300 variables in the model.  Note that your problem is *univariate* as you have *one* $Y$ variable.  It is a *multivariable* regression problem.  Also note that if you don't use shrinkage, the only kind of dimensionality reduction that is usually reasonable is an approach masked to $Y$, e.g. variable clustering, redundancy analysis, nonlinear principal components.
@Alex I read your update. Can't you get enough data to have a representative sample of that .1% of extreme values? On the other hand, based on your description of a sensor, if those values are so extreme, just set a threshold and remove them. Paraphrasing the words of Pauli, those values are not even wrong.
Thanks! Is the high leverage point we used here consistent with "the leverage is typically defined as the diagonal of the hat matrix" from http://en.wikipedia.org/wiki/Partial_leverage?
You're right! I ruled out SDT a while ago because I only had my two accuracy measures, but then worked out a way to get the reponse bias measure and didn't consider it properly again.  Curious to see how it compares to the below suggestion...
@Michael I offered this reply as an example of how one might address questions that look like homework but are not explicitly tagged as such. It is here only as a reaction to your now-deleted post. As a moderator, rather than downvoting low-quality replies, I prefer to provide constructive examples of what a good answer might look like (understanding that my efforts are never perfect and can always be improved: take that as an invitation to post an even better answer!).
Usually these experiments are carried out with accuracy measure, that is somewhat misleading. Nice choice the AUC. Of course you performed RUS only on the training set, is it true? Do you think that RUS or weighting could improve an F-measure for the minority class too? Thanks
1 measure-accounts sold and 6 months for each employee which are each paired( I.e jan 2013 vs jan 2014 etc) so I am wanting to see if the median difference from jan 2013 to June 2013 vs jan 2014 to June 2014 is statistically significant from 0
A very simple expedient to consider is to re-express the values as reciprocal times. In many (but certainly not all) cases where your problem is encountered (essentially a division by zero), that is an indication that such a re-expression might be a better way to understand, summarize, and analyze the data.
SVMs are not state of the art.
The difference between $R^2$ and **adjusted** $R^2$ is crucial, Magnus.  It sounds like you are referring to the latter in your question, right?
Welcome to the site. Is this a homework assignment? If so, it should have the "self-study" tag.
Thank you for this awesome reference list!
+1 ... however, to [use cross-correlation](http://en.wikipedia.org/wiki/Cross-correlation#Time_series_analysis) the series need to be jointly wide-sense stationary. The usual approach is to detrend (maybe by differencing depending on how the trend arises) and prewhiten.
This question appears to be off-topic because it is about looking for an algorithm.
Good point, thanks. Have edited the question.
Can you edit your question (use the "edit" link at lower left) to say what kind of data you have (i.e what kinds of values your data take)?
sure mutual information formula can be computed for response variables with $k$ values
I'll give it a shot... the memory issues for data sets with attributes > 50 were in RapidMiner, mostly using neural net based models and related, such as Self Organizing Maps. I might be able to reduce the attribute set by using generic "Other" attributes for courses that appear in less than 1% of the population.
(1)there are totally 100*100 pixels, if you are using intensity as features, there will be 100*100 features in total, that's xi; and (ii) you may also use the power density as a feature, that's (xi,xi) or xi.^2, still 100*100 in total; finally(iii) if you use the correlations between two pixels, there will be C pairs of pixels in total, that's (xi,xj), C is combination in math (http://mathworld.wolfram.com/Combination.html)
Alright. So if I assume, later on simulations well be used instead of the posterior distributions. But the simulations can only be drawn if the posterior distribution is known, as seen in `ps <- sample(p, replace=TRUE, prob = post)`! Am I right in assuming that this will change for more advanced simulation techniques?
With four independent data points it is impossible to obtain a 95% CI for the median: no matter what the true median is, the four points will all be greater than it with probability $1/2^4$ = 6.25% and they will all be less than it with the same probability.  Therefore the range of the four observations *must* fail to cover the median 12.5% of the time.  This makes me suspicious of your approach.
... and there is no whatsoever guarantee that the effect you need to solve your problem is larger than other effects which are just noise wrt. the problem at hand. I've seen data where 95% of the variance was noise due to some physical effects...
For optimal matching (`method="OM"`), the seqdist function uses 1 as default indel cost, but you have to provide either a substitution cost matrix or a method for computing one (`"TRATE"` or `"CONSTANT"`). Alternatively you can use `method="LCS"` which corresponds to OM with a unit indel cost and a constant substitution cost of 2.
This is a fantastic answer. Thank you very much John. I was just wondering, other than mediations, are there any other interactions you can check for?
What do you intend to do with this "normalized percentage" once you have it?  Your answer to this matters: for some purposes it will be fine, but for other purposes it's a lousy number to use, because it depends on the two most extreme players--the best and the worst--and in general it's a bad idea to base one's analyses and decisions on such exceptional quantities.
@david No prob. A good strategy for this kind of problem is to type `methods(predict)`, and then try `stats::predict.princomp` if there's no asterisk, and use 3 colons when there is an asterisk.
What is the purpose of this calculation? Is it to report a fact about the data or is it to estimate a parameter in a model of length of stay?  (The three answers so far all implicitly assume the latter.) Note, too, that there is subtlety here that becomes apparent when shorter periods are considered: your calculation should (somehow) account for patients who are currently in the hospital.
@mbq And I want to post here not on SO. Do I have to delete the post on SO first? Then can you re-open my post here?
I figured out the answer two days ago, and posted it here as a new answer. I also then expanded and updated the web page at graphpad.com that you found. I just edited that page again to include a link to an Excel file with the problem data (http://www.graphpad.com/faq/file/1226.xls). I couldn't do that until I got permission from the guy who generated the data (he wants to be anonymous, and the data is vaguely labeled).
I am analyzing the moments of 2D light distributions that I generate from a path tracer.
Thanks! I have Hastie's book and that of C. Bishop (Pattern Recognition and Machine Learning). I know that such a small n would lead to spurious or unreliable (see Jeromy Anglim's comment) association. However, the RF algorithm as implemented by Breiman allows to cope with a limited number of features each time a tree is grown (in my case, 3 or 4) and although OOB error rate is rather high (but this should be expected), analyzing variable importance lead me to conclude that I would reach similar conclusion using bivariate tests (with permutation test).
By constructing confidence intervals, you implicitly assume the survey is a *simple random sample* of a population. What population? And what evidence do you have that it is random, or even representative?
It looks like the question is incomplete. What are the ranges of $X$ and $Y$, what about their distribution, i.e., whether they are discrete or continuous or can be anything, about their dependence and what about the marginal distribution of $Y$?
This is called the "heteroscedastic errors-in-variables" problem.  (This phrase is a good target for a Google search.)  Recently (2007), Delaigle and Meister proposed a nonparametric kernel density estimator in a [JASA article](http://www.ms.unimelb.edu.au/~aurored/DelMei_FINAL.pdf).  An abstract about some parametric methods (method of moments and MLE) suggests some additional approaches: http://www.sciencedirect.com/science/article/pii/S1572312709000045. (I am not familiar enough with the research to give you an authoritative answer about how to handle your particular dataset.)
Does ?train help? (I don't think it's a masterpiece of self-explanatory documentation, just wondering you have got this far).
A related inequality that *is* true multiplies the right hand side by $E_f[g(X)] / E_g[g(X)].$  This might help you construct counterexamples.
What do you mean by "add it to a vector"? Using `apply` in this context would be like reaching for your left pocket with your right hand.
@cbeleites thank you...there's indeed a whole lot of them and I will take a look...thanks again...
@John Fisher is often assumed to be a Frequentist, but his allegiances are much more complicated than that. He derided the Neyman-Pearson approach at every opportunity, after a short initial period of acceptance, and his later writings made it quite clear that he was interested in the evidential nature of the data, something that is excluded from Frequentism. He is probably better characterised as a likelihoodist and, depending on how one takes his fiducial argument, a closet Bayesian in denial. If you are interested in Fisher then you need to be quite careful with secondary sources.
p-value is a characteristic of a given sample, not an estimate of something in the population to be able to bear some error. So p-value is an exact number, not an interval. p-value is interval only in Monte Carlo testing - because in such a testing not full information from the sample is used to compute p.
Each subset is a 3-dimensional Gaussian corresponding to a 3D spatial location. So the distribution is over $n \\times 3$ parameters and the likelihood can be written as a product of $n$ terms i.e. $L(\\theta) = \\prod_{i}^{n} L(\\theta_i)$ where each $L(\\theta_i)$ is a 3D Gaussian. So each of these smaller subsets are independent of each other. However, the full distribution is over $\\theta$ which is $3n$, so I need to somehow make these 3D Gaussians a function of $3n$ variables where the other $3n -3$ variables do not affect it.
I really like your idea and I think we can conduct linear mixed modelling; however, I believe that the design would require many more than 10 people. Can anyone comment?
Take a look here http://stats.stackexchange.com/questions/1444/how-should-i-transform-non-negative-data-including-zeros . But probably you should convert them to non-zero values somehow.
@PeterFlom Thank you for the correction. That could have been confusing (:
The _subscript_ in $E$ tells us over which variable(s) we must sum (or integrate, for continuous r.v.'s). What probability mass or density function we will use (marginal or conditional in your case) is indicated by the _argument_ of $E$.
+1 Very good point.
Just to clarify, you're looking for a way to plot quantile markers, correct? A box plot is simply a "fancy" (term used loosely here) plot of quantiles and outliers.
This answer is not quite correct, I am sorry. The critical region strictly speaking includes the critical value itself. If the test statistic is continuous and p were "exactly" 0.05, that should be rejected for a 5% test (though the probability of this is 0). If the test statistic is discrete, then you would generally choose an achievable significance level (and typically, one smaller than the desired type I error rate, but this is not a rule) -- but once you choose the achievable significance - say 0.0467, for example -  then the case at that boundary is in the rejection region). ...ctd
The master question here is what do you want to show with this visualization.
Hi gung, it seems that you gave me a hand in bringing my question into a readable format, thanks. Could you give me an idea, how to write the programcode in the form you did? Sorry, but I did not find it in the "catalog" of how to ask question, Thanks
You may want to rewrite the expression for the out-of-sample forecast, possibly using [LaTeX](http://stats.stackexchange.com/editing-help#latex).
+1 in light of the clarifications by the O.P. concerning the intent of the question.
Really appreciate taking the time to write out a full explanation -- very helpful.
Do you have a sample from each population, or do you actually know the total number of people in each population who like ice cream?
To be fair, I found [Stochastic approximation](https://en.wikipedia.org/wiki/Stochastic_approximation), but very little *practical* information with examples or practical discussion of when it works well and when it doesn't. Most of the info is in academic papers which seem to require quite a bit of work to convert into a practical application.  Another thing I found is the keyword [Likelihood-free estimation](http://arxiv.org/pdf/1001.2058.pdf) which solves a very similar problem and there's more practical information available online.  **Is there anything else?**  References are welcome!
Excellent! Thank you very much - just what I was looking for. I realize I was a bit vague in my description, but you nailed it. If you don't mind me picking your brain a bit, I rearranged the equation into p(x)=2/10+s(x-3)/10. If i were to want a more exponential change in distribution, how would I modify it? I tried (x-3)^3, but that throws off the summation to 1...
thanks. This is really helpful
This kind of dredging is *NOT* OK.  There are no principles in statistics that back up the approach you have outlined.  A few simulations would expose the damage that variable selection does.  In general, variable selection without penalization is invalid.
Your approach is not honest about the number of parameters estimated.  The transformation estimation process needs to be part of the bootstrap as does every other modeling step that utilized $Y$.  Collinearity on the other handle, can often ignore $Y$ and can be dealt with pre-outcome modeling.  There is no need to compute $P$-values using the bootstrap as you already have those from the original model fit.