tttjjj/elt_bench huggingface dataset:
Could you expand upon how to use "standard ML machinery" to derive the desired confidence interval? Do I need to get my hands dirty with some algebra and calculus, or is there a simpler approach?
Thanks Martin, this clarifies the difference. I was thinking that one could extract parameter set from (beta,PSI) and then simulate some intrinsic variability using this extracted set and covb. But that's clearly not the case.
This is a nice idea. But it will not serve my purpose. Ultimate goal of this exercise is to plot all these countries in the graph then divide the graph in four quadrants and then see which countries moved from one quadrant to  another during last three years. Your method will certainly give me time dimension but it will only have three point on horizontal axis because of which each country will be someplace in one of the three vertical lines. After this I will not be able to get my quadrants. See the picture in question. The movement across the quadrants is something I am interested in.
Edge squares are squares that are on the outside, but not in a corner. Each edge square has five neighbours: the middle square, two other edge squares, and two corner squares.
I would add the intercept which is the earnings of someone with a height of 0 and add 66times the Beta1, which is 0.8*66 in this case
Accepting, a long time later! Thanks for this detailed overview of the technical issues, and I'll take a look at Harrell's book.
Perhaps it's me, but I don't get the paragraph "I've then iteratively...": for each k from 1 to n, you've collected all subsets of size k that hold the highest degree node and expand down? Is that it? And then for each k you've taken the cardinality of the intersection of all the subsets of size k?
K-S is for continuous distributions.  The Poisson is discrete.  Instead, estimate its parameter, use that to compute the probabilities, and conduct a [chi-squared test](http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Other_distributions).
The problem with this is that I do not "know" that these nested categories are actually important to the model, they are just a few out of a large number of covariates that I want the tree to choose between.
More information on the nature of the repetitions would be helpful. Perhaps clustered survival data methods including frailty models apply (and perhaps not).
Dropping exposure sounds like a questionable thing to do, though it may need to be recoded according to the building the unit is in, perhaps on a subjective basis. For example, in North America, southern exposure is probably generally desirable due to increased sunlight. However, if you're in a large urban area and there's a big park just to the west of your condo, then western exposure may be highly desirable.
Your chance of getting a useful answer will go up a lot if you post a reproducible example of what you're doing.
@bruno within a group of source entities the variables would change. For instance, a source sends traffic to multiple entities within multiple countries. Within the groups of 'source-dest entity pairs' all other variables would essentially be constant, which can be considered as an index here.
I got the same confidence interval that you got.
With logarithms, it is rare for a multiplicative comparison (e.g., "factor of 10") to be meaningful.  Is it perhaps the case that the two procedures return a constant *difference* in the log likelihoods (for datasets of a fixed size)?
Then I think that the test is doing what it's meant to. The question that @Zhubarb's comment brings up indirectly is 'what do you want this thing to be Normal for?'  If it's to run a linear regression model and interpret coefficients it often won't matter at all.  The CLT should have kicked in for most relevant quantities already.
@Z Khan Are you perhaps also [this Z Khan](http://stats.stackexchange.com/users/13806/z-khan)? If so, please let us know so we can merge your accounts.
Welcome to the site, @John Wendall. Your post is being automatically flagged by the site's algorithm. Would you mind elaborating / expanding on your answer? We strive to create a permanent record of statistical information here in the form of high-quality questions & answers. There is a guide to answering questions on CV here: [how-should-questions-be-answered-on-cross-validated](http://meta.stats.stackexchange.com/questions/1390/). Since you're new here, you may want to read our [about page](http://stats.stackexchange.com/about), which contains info for new users.
This question appears to be off-topic because it is about which code to use and/or availability of code: there is no statistical question here. Please review advice on software-related questions in the Help Center. Statalist is a much better forum for this.
No, I don't know why they specifically suggested logs, but the spirit of their point was that just using an outlier approach eliminated a number of data points and they wanted to see the full data set analyzed.  It seems a little crazy to keep data points that are 4+SDs away from the mean, but I do need to address the point.  You're right about the leverage though--I suppose I could just use Kruskal-Wallis on and explain why.
An example of a proper scoring rule is the Brier score, which is the average squared difference between the binary outcome $Y = 0,1$ and the predicted probability that $Y=1$.  Like the likelihood function, optimizing the Brier score can result in an optimum predictor.
@Shane, in general it is useful to check out how other tools are approaching the same problem. It is the same as reading different books on the same topic, every author has its unique perspective, combining them gives you better understanding. Yes it is a slow process, but if you want to trily understand something this is  in my opinion the only way.
How about repeated measures ANOVA with time points (3,5,8,10 weeks) as repeated factor?
Joel, That quotation is meaningless until its author quantifies what is meant by "short" and "long". In his example of an event with an expected rate of one per year, the chance of a recurrence during the next month will still be far less than the chance that the next crash occurs more than one year later. What he might have meant is that the *probability per unit time* is greater in the near term than in the longer term. To compare actual probabilities you have to multiply the probability per unit time by the duration (technically, you have to integrate it over the duration).
Are these paired measurements or not? (I assume not.)  The uncertainties could be used derive weights which could improve your inference, but the variation in uncertainties is pretty small, so there's not going to be much gain, even in the uncertainties are accurate.
There was an error in the computation of the two-tailed p-value which is now corrected.
Actually I think it's a question of *purpose,* not of empty cells or asymptotics.  Even a lunatic can get a few percent of the vote.  In a sufficiently large poll (such as national exit polls) such proportions can still yield counts in the tens or hundreds: no problem with zeros or low counts there... .  The OP still has to decide exactly what he means for one poll result to be "better" than another and by how much.
@clarkson You can find the confidence interval by setting $Z=\\frac1{1+T/(n-1)}$ and calculating $$\\mathbb{P}[\\theta\\in[a Z,b Z]].$$ For very large $n$ you can approximate the distribution of $T$ with a Gaussian.
A $\\chi^2$ test would treat sugar content as nominal data, which would waste information. I don't see anything to gain.
And also: I prefer to stick with a continuous quantification of outlierishness to give experimentalists a ranking what to look at first and how risky it may be to go further with candidates ...
Note that "exact" confidence intervals are not very accurate (see Agresti's work).  I don't know of a method for simultaneous confidence intervals, but for individual intervals Wilson's confidence interval is more accurate than "exact" methods.  It can be computed in R with the `Hmisc` package's `binconf` function.
That's the joint probability of $x$ and $\\mu$ given $\\mathcal D$ evaluated at $x=1$. I am guessing from Laura's question that $x$ is discrete and $\\mu$ is continuous with values in $[0,1]$.
+1, I think this is good, but I think it would be even better if you could add a simple R based example for each option, if that's not too inconvenient.
yes the integral just becomes a sum
This is a great trick! It actually (I think) does exactly what changing the reference level does, but without the hassle of having to change the reference levels multiple times for each condition/category. Thanks!
@mchangun no.  caret saves the pre-processing parameters, and will pre-process the new data in the predict.caret call.  it's pretty slick!
@wearn I see. The tutorial should cover that. Test points are projected into the feature space defined by your training data, and their class is assigned based on which side of the decision boundary they lie.
Could you please state your research question(s) and how many ratings each subject provided.
@Eupraxis1981 The data is from google analytics so I am guessing it is deterministic in time. BTW my stats background is minimal.
I'm not an expert in this field, but a while ago I was fooling around with [NEAT](http://www.cs.ucf.edu/~kstanley/neat.html) and [NERO](http://nerogame.org/). You may be interesting in checking it out.
Thanks mpiktas, I suppose VaRselect to be run on stationary time series (e.g. differenced)....
It may be helpful to define what the goal of your analysis is.
a quick comment I would have is that you could consider the FFT as another means of obtaining features for each customer. Assuming you have a feature vector containing summary statistics based on each customers time series and other data, you could supplement your feature vector by adding features derived from an FFT. Bear in mind it is only appropriate if the window over which you are talking the FFT is stationary. Otherwise time features such as 1st and 2nd derivatives may be more appropriate.
I wonder what he meant by that...
I think there are many pragmatically-oriented applied statisticians who believe that either could be legitimately used, if used correctly, & will go w/ whichever is more practical in the case at hand. In this vein, I asked a question ([List of situations where a Bayesian approach is simpler, more practical, or convenient](http://stats.stackexchange.com/questions/41394/list-of-situations-where-a-bayesian-approach-is-simpler-more-practical-or-more)) attempting to elicit when the Bayesian approach might be simpler (since typically the Frequentist approach is, cf Shelby's #3).
(+1) I just stumbled upon this question again. I'm happy to see you posted an answer (quit awhile ago!)  to close it out. Cheers.
Also from Gelman's blog: a nice set of powerpoint slides ([in pdf form](http://www.stat.columbia.edu/~cook/movabletype/mlm/meng_JSM_presentation_20090802_8am.pdf)) by one of his collaborators, which includes a useful rule of thumb at the end of it.  I'm currently using a variation on this method and it's working great for me.
while it is difficult/impossible to "prove" causation, look into the subject of statistical mediation, which seems to be exactly what you're describing. This has been studied in many social sciences with the seminal paper by Baron and Kenny in the 1980s with many incremental advances since then by many statisticians in the field of causal inference.
@xan. I updated the answer. Tks.
@peter The coefficient of age in the first model is 0.0062546 and then in the second model it changes to -0.0297705 while the coefficient of age^2 is 0.0023998 in the second model.
What hypotheses and "effects" are you interested in? How are you presenting your data descriptively (what numbers and figures do you intend to show)?
You can edit your question rather than re-posting. See the FAQ
Thanks gung!! Can you include a small sample R code.
I don't know which statistical software you use, but in `R`, this can be easily tested using `prop.test`. For example: `prop.test(x=c(50,20), n=c(100, 70), conf.level=0.95)`. In any case: a quick [Google search](https://www.google.ch/#output=search&sclient=psy-ab&q=difference+two+proportions&oq=difference+two+proportions&gs_l=hp.3..0i22i30l4.56.2513.0.2591.26.18.0.4.4.1.187.2316.0j17.17.0...0.0.0..1c.1.17.psy-ab.NARKYF72C_0&pbx=1&bav=on.2,or.r_qf.&bvm=bv.48340889,d.bGE&fp=2f99562ada0f7597&biw=1920&bih=955) might reveil helpful pages.
It looks like your question got chopped off at the end. "equal to" what?
Following from my previous comment, I should note that in the expanded/binomial-ized data, you could model the "item" variable as either a fixed or random effect. I think I'd lean toward modelling it as a fixed effect because presumably you might be interested in not only accounting for but also assessing the item differences and any possible interaction between item and other predictor variables.
@user1125946 See my edited answer.
Actually, this **is** homework in the sense described in the [tag:homework] Wiki.
No wait this is not what I meant. If the function allows you to extract the scores than this is already the representation of X in the new base. This would probably be of the same number of columns of the original X. Be careful that usually PC and loadings are the same thing. Be also careful that probably you are not doing the matrix multiplication with the right command
This question seems to be about statistics, not mathematics, so I'm voting to migrate it to stats.stackexchange
How *would* you 'make up' a sample of data whose true mean you knew?  Hint: using a piece of software
WEKA gives you Precision and Recall authomatically, see "Detailed Accuracy by Class".
So simple, thanks! I was definitely overthinking it.
This is a bit confusing.  You say you have two "independent" variables (I prefer "predictor", but that's not important).  Do you have any "dependent"/"response" variables?  If these were both response variables, I can imagine fitting a parametric, bivariate probability distribution (with or without predictor variables on which the distribution's parameters depended) -- or a 2D kernel density estimate.  Perhaps you could explain the context a bit more. (PS whoever's upvoting the question must know what it means ... anyone care to chime in?)
@BrunoW Unfortunately there is no conjugate prior for logistic regression. Perhaps you could post your model specification in your question to see if that is a problem? Also, if you're not attached to Bayesian methodology the algorithm used in Frequentist logistic regression would be much faster and would most likely be nearly identical if your data set is long; i.e. many rows (which it seems to be). Finally you could just subset your data (reduce the number of rows) if you aren't as concerned about being confident in your predictions.
My apologies, @yns, I couldn't figure out how $\\ge$ made any sense; I just wasn't thinking. Let me know if anything else is off. Note that there is information about the markup / formatting options CV supports [here](http://stats.stackexchange.com/editing-help). Notably, CV supports $\\LaTeX$ via mathjax; documentation can be found [here](http://www.onemathematicalcat.org/MathJaxDocumentation/TeXSyntax.htm).
The constant option gives an "invalid name" error r(198). I have not figured out the reason for this.
Note that by exploiting symmetry you can also take advantage of any results on the maximum.
@DCS: I agree with that. I guess it's just a case of how expressive you want your model to be. My interpretation of the question is that the OP wants to end up with a model that he can read an understand, NNs won't provide that which is why I called it a black box approach. But yes, it is just regressing a more complicated (more parameters at least) model, but a very general model that is likely to fit many shapes of data.
Note the question is about the regression. Usual OLS estimates would not satisfy the summing up condition.
What exactly are you trying to estimate? If you want to estimate maximum of the data as each new observation arrives the most simple estimate is the sample maximum. Maybe you want some distribution properties, like mean, variance, quantiles?
Greene's book is indeed one of the most standard references, but it makes a weird assumption that you know some econometrics when you approach it -- enough to bridge some gaps in presentation. My wife, a game theorist, was trying to learn from it when taking a required econometrics sequence in grad school, and she would ask me at least a dozen questions per homework: "What does this symbol stand for? I cannot find it defined anywhere" or "Am I expected to make a normality assumption here? It is not specified explicitly, but there is no other way to solve this" etc.
For complete rigor you need to understand $B$ as a linear operator on the Hilbert space $\\mathcal{l}^1(\\mathbb{Z,R})$. For partial rigor, understand $1+b+b^2+\\cdots$ as the limiting value of partial sums $a_n=1+b+\\cdots+b_n$ and compute $(1-b)a_n=1-b^{n+1}$. Its limiting value is $1$ provided the limiting value of $b^{n+1}$ is $0$.
+1 I see you have been acquainted with our site for a long time, but I would like nevertheless to welcome you on the occasion of posting your first answers.  This is a nice clear explanation that gets to the heart of the matter.
@Tomas It's impossible to say whether this answer is correct until you clarify the question.
As this question hasn't received much attention, one way to revive it is to add more information. Can you describe your dataset, for example?
Well, you sort of have to deal with your economics and your statistics separately. First determine what you personally think should help to predict the price of the stock, based on whatever sort of research you've done. The only price-based metric I know of that has been known to help is momentum (meaning simply some return over some period of time, NOT the technical indicator by the same name). Other than that, I think fundamentals of some kind would probably help; if they're not coming in strong, you might consider changing your statistical model around, or changing the data periodicity.
This is a short article I wrote ten years ago but never published.  (The new editor of the AMM saw no mathematical interest in it.)  I have given talks in which I played the two-envelope game with the audience, using substantial amounts of real money.
This is a big question, but it is also a fairly standard econometric problem (longitudinal panel data at a country level).  Which texts have you read on how to do this, and what in particular is causing you problems?
Of possible interest: [Should factor loadings be dominated by items' ranges of answer options?](http://stats.stackexchange.com/q/10945/930)
Thanks for your response. To clarify, each HMM is trained on fixed-length short sequences (say 2 sec) and parameters computed from them. The observations (sensor readings) are continuous and several features are extracted from each sequence, and each feature is approximated with single gaussian. I wanted to use prior because some activities are less likely than others for e.g. sitting is more likely than jumping and decision based on just likelihood is not right. One way is to use individual activity count as prior or other as some distribution like Dirichlet. What do you suggest?
Thank you very much for the response. Rapidminer is very good...very simple and a gateway to R..which is what im trying to practice this on. I have purchased the book. It may give me some ideas
Ok John, thanks for the answer. However, the problem of casfhlows from electricity trading is more complicated than modeling a futures curve. Your casfhlow in electricity trading is given by your actions, the behavior of the whole market and the current bonus/fine mechanism of the market operator. Hence, you can either carry out a historical simulation, where all these factors behave realistically or you can model each by some process. Having 4 models, each with lousy fit, will provide with a very lousy prediction and thus either unreliable or crazy high VaR, hence I want to avoid models.
$\\alpha$ if deleted can be lower if deleting the best item, but one would only want to delete items with higher $\\alpha$ if deleted, following classical test theory, according to which multidimensionality is inherently bad. I may be caricaturizing it somewhat in saying that, but it is a natural consequence of devaluing items that reduce the average inter-item correlation.
The third sentence of your question is confusing - a randomized trial is not typically what one would call an observational study. Is the question really a two part one, or do you mean "trial" in that sentence as well?
Frank, your book (RMS) is lovely! I just studied the first hundred pages, and some other chapters. It will indeed improve my work.
Are you interested in having the slopes be continuous at the breakpoints or not?
Inversion will be the only valid method for this case!
Is there some further explanation or discussion needed in the answer at all, or do you think it's been covered now?
"*But of course the model should also pass a goodness of fit test.*" -- why "of course"? It's very unlikely any of the distribution families contain the actual population distribution for these data.
`princomp()` performs PCA using a Eigendecomposition. `princomp()` will fail if the number of variables is larger than the number of observations. Use the SVD based `prcomp()` instead;  this is generally the preferred method for numerical accuracy.
3. Sure that is not a problem. I just wanted to know on which basis should one choose that number? I think the basis can be that if I run two bootstrappings of 1000 tries each and see that they differ, that means that I need more resampling trials (for example 5000 or 10000) instead.
@gung As I understand it, it is simply a data file manipulation question better suited for StackOverflow.
Welcome to the site, @BlaiseEgan. If you want to comment on the other existing interpretations, it might help if you provided an argument in support of your views, or possibly even better, just leave them out. 1 thing we're a little bit concerned about here is ending up w/ lists of various people's opinions. (Note that opinions *can* be valid & I don't doubt that you have thought about the issues before having come to yours.) Since you're new here, you may want to read our [FAQ](http://stats.stackexchange.com/faq), which discusses issues like these.
I exactly don't have a solution for you. But, I think this is a kind of error-in variables problem. I have seen this stuff in Macroeconomic Theory by Thomas Sergent (1980's book). You may want to look at that one.
http://en.wikipedia.org/wiki/Prosecutor%27s_fallacy
Good answer Michelle.  Some ideas from me: 1) Definitely give the judges some kind of rubric to try to encourage common standards; 2) try to have the same number of judges per project if possible (as otherwise the projects with fewer judges will have a higher variance and hence more chance of getting to the top - or the bottom) and 3) I think you will have to just use an average, but if you had the expertise and software you could fit a mixed effects model with judge as a random effect and it see if this changes the result.  What if it does?  Probably still use the average....
Thanks. This seems like a particularly good way forward. I will try it.
Is there anyway I can make this question clearer or more answerable?
@MichaelMayer I am not sure; I get as far as: $Cov(p_{1j},p_{2j})=E(p_{1j}p_{2j})-\\pi_{1j}\\pi_{2j}$ where $E(p_{1j}p_{2j})$ is equal to the expectation of the product of the two components of $\\bar{x_1}$. But I cannot simplify this product any further.
...This means you'll want to move from bivariate analysis to a multiple regression or ANOVA model. Whenever the relationship between the outcome and one predictor changes depending on the level of another predictor, one needs to assess the strength and direction of an interaction between the two predictors.  You can find more on testing, assessing, and interpreting interactions by doing a search for that term on this site.