tttjjj/elt_bench huggingface dataset:
Why is averaging the coefficients appropriate?  If the two datasets have different sizes, then surely at least some form of weighted average should be used.
thanks for your answer, especially to #1 is helpful. For the test, I had thought of that, but the range of most priors will have a bound at $\\infty$, so I was thinking of perhaps comparing the bounds of quantile intervals, e.g. send warning if: the 80th quantile of data > 99th quantile of the prior or if: any data is greater than the 100-10e-log(n)th quantile) although I would have to play around with the numbers so that I catch the right errors.
@Wern if you like my answer (or IrishStat's) feel free to accept one by clicking the green check mark.
Welcome @Davester. Would you mind expanding on your answer a little bit, like explaining why and how resampling with replacement leads to resampling bias and why is it better to use a permutation test? Preferably also provide an illustration with e.g., a short R script and references? That would be a great help to the OP.
The Wilcoxon/Mann-Whitney test is **not** the appropriate *post hoc* test following rejection of the Kruskal-Wallis test.
@mp Yes, exactly: An inadmissible procedure is dominated by some other procedure in the sense that the expected loss (risk) of the latter is less than or equal the risk of the former for *all* possible states of nature (distributions) and there are some states of nature where the inequality is strict.
I'm not a practical expert in maximum likelihood, but I would approach the likelihood as described: Our task is to estimate the value $p$ in $Y \\sim Bin(n,p)$, where $n=8$ and we observe $Y = 5$. The MLE is indeed 5/8, and by the invariance property of mle we get the same probabilities as the text. This is my naive maximum likelihood, but let the community wiki improve.
I think multi-collinearity would only be a problem if I would use one variable to interpolate the other one. However, the initial idea was to use, for example, the data point 1990 and 1995 for income to interpolate data points for income for the years between 1990 and 1995. I planned to do the same for the population variable.
If you need to explain fixed/random as terms, I'd go with this. If you were simply giving an example, I'd suggest using the terms "group effect" and "individual effect", which I believe are also correct but less mysterious to the layman.
I would advise you to install ubuntu (google 'download ubuntu') and to run your windows-only apps from within ubuntu via virtualbox (http://www.youtube.com/watch?v=KXgKnd-u2R4). R and latex editors run on ubuntu like a charm.
Very nice, thorough answer.
A difference of 75 between the AIC values for the 2- & 5-peak models is not "small". Remember there's an arbitrary additive constant in the log-likelihood, so don't compare differences to the values for each model. @Nestor has adressed this point but I thought it worth emphasizing.
Samples need to be taken at a minimum of 2x the highest frequency you wish to capture (thank you Nyquist), otherwise you get the problem you're having.  SD cannot help you because you've already missed sampling the data the SD would rely on... and I suspect that your method is probably fine.  Is there any reason that you cannot sample more frequently?
Could you please attach the code where the GLM and LM are defined?
The University is Stanford. I can find Andrew Ng course on the web.  I heard Dick DeVeaux give a lecture on neural networks claiming that the neurons part of it sort of hides what is really going on.  it is just another type of nonliner classifier and if instead of looking at it from the perspective of the diagram they are best understood by writing out what they do algebraically.  I hope I am remembering this right.
Or the other way around: Is there a 'code' defining the gaps? It's because the custom set costs are only accounted for 'real states' if you set submission costs like this: R> subm.custom <- matrix(c(0, 1, 1, 2, 1, 1, 1, 0, 1, 2, 1, 2, 1, 1, 0, 3, 1, 2, 2, 2, 3, 0, 3, 1, 1, 1, 1, 3, 0, 2, 1, 2, 2, 1, 2, 0), nrow = 6, ncol = 6, byrow = TRUE, dimnames = list(mvad.shortlab, mvad.shortlab))
@user152509 Among other places, John Tukey has a scathing criticism in "Data Analysis and Behavioral Science" (Collected Works, volume III, pp 243-245).  "If [limiting statistical methods based on scale type were] generally adopted it would not only lead to inefficient analysis of data, but it would also lead to failure to give any answer at all to questions whose answers are perfectly good, though slightly approximate."
(1) The $[\\{\\lambda+1\\}\\exp(-\\lambda)]^{n_1}$ part corresponds to the grouped observations.
For a newbie, doing an adaptive design right is too much of a challenge. I would advise against it unless you have access to a statistician qualified in such designs (and they don't come by in dozens in the nearest drinking hole, you know).
I was wondering if someone would provide a Bayesian approach. Excellent. Thanks.
Is your dependent variable continuous or discrete?  You refer to it as continuous twice, but then describe it as an index of integers that range from 0 to 15, which would be discrete.
I think half of this answer may be right: you want to know whether you have calculated a condition number or its square when you are comparing it to somebody's recommended threshold. But condition numbers do have absolute meaning in geometry and analysis as well as in statistics.
@ttnphns Sorry, I should have added more detail. The participants are asked to group 112 items. We created a matrix for each participant from these groupings assigning 0 pairs to items which were assigned to the same group and 1 to pairs of items in different groups. So we had a binary matrix (a simple distance matrix) for all possible item pairings for each participant. These were then averaged across participants to created one aggregated distance matrix. This aggregated distance matrix was then submitted for cluster analysis (fuzzy CA) to identify the grouping of items across participants.
Basically, it means that $\\text{Prob}(w_1)=0.5$ and  $\\text{Prob}(w_2)=0.5$
And also that the errors after "rotating to a determined point, away to a random point, then back to the determined point", whatever you call them, are different measurements, which your colleagues may have some good motivation for proposing.
"[L]arge numbers of data results" is ambiguous as between (a) the effects of a large sample size and (b) those of trying many different tests or methods. Also, you have tagged this **both** sample-size **and** effect-size. Please think up a better title too: "Tricky question" is useless for the purposes of people searching this site for advice. This is not a helpline but a site with the aim of building up an archive of well-answered good questions. (Your title is likely to be edited any way by anyone with enough reputation; I am tempted to do it myself, but as above your question is unclear.)
@Gavin: should it be closed or could it be moved?
Logically, it isn't really possible to say that the difference is 0 at t weeks (to understand this it may help to read my answer here: [Why do statisticians say you can't accept the null](http://stats.stackexchange.com/a/85914/7290)). Instead, you need to determine what difference is sufficiently small that it might as well be 0. In addition, you will need to use a mixed effects model for your data.
Oh ok interesting! So just take data for each variable and lay it out in row-column format and get to number crunching eh? Are there any tools I should be looking at or should I just be programming something?
Ok, I was thinking this would involve covariances as well. I've been looking around, but haven't found any online references that give me the expressions I need to compute covariances and how to combine them into a a single $\\sigma$ which I can use to quote confidence intervals; do you happen to know of any texts or documents that do?
Say, you have a portfolio of assets and you want to estimate the sensitivity of the diversification benefit to a severe change in correlations (therefore, the need to stress correlation matrix). Stressing the eigenvalues seems like a mathematically attractive idea, however, it doesn't necessarily reveal that much - wouldn't be more useful to stress actual correlation because you can express a view about specific variables.
You have trouble understanding what he means by "correct" in part because he doesn't have a correct understanding of confidence intervals.
@Glen_b, The key point is not that MDS is for distances input and PCA is for coordinates input, but [that](http://stats.stackexchange.com/a/14017/3277) iterative MDS fits few dimensions while PCA retains few dimensions. So MDS preserves distances somewhat better than classic PCA does. The answer for the question is Yes, PCA is suitable, but MDS is more suitable.
@user603: I'm not sure how to do this, given my low reputation on/knowledge of this site.. Please feel free to try (edit) if you know how.
Thanks for thorough explanation, it reminds me of the ANCOVA (analysis of covariance). The first step of ANCOVA is to test the interaction between categorical factor and covariate to see if they have identical slope for both condition. It is quite similar to what I did here. In ANCOVA, it gives same pvalue for interaction in t-test and F-test since  interaction is the last term in `aov`.
@Glen_b I'm looking for a CI for the mean of $\\text{F}$
I would contact Ben directly. He is a very friendly chap and offers his R packages in multiple forms so may be able to compile the source into a binary using a newer R if that is a problem. Anyway, Ben is the best person to help so drop him an email.
Have a look at Haldi and Ling (1998) "Some cautionary notes on the use of principal components regression."  in The American Statistician.  Bottom line, relationships with variables outside of the PCA can be hidden in even the last PC.
SEE ALSO related question http://stats.stackexchange.com/questions/23938/how-to-correlate-ordinal-and-nominal-variables-in-spss
Thanks for your quick responses, guys! I looked into the corrplot package. Unfortunately I'm kind of having the same issue as I did with the corrgram package. My data is too large to really get someone out of those graphs. We really want something interactive. Maybe when you have this intense correlation graph, you can use your mouse to select where you want to like, zoom in on. Know of anything like that?
@JuanPi: If you have questions about the OP's post, you need to ask on the comment section above; not as part of your answer.
thanks so much!  also, thanks for the pointer to the sos package
Out of curiosity, what package did you use for model-based clustering, `mclust`?
Following up Thylacoleo's example using the circle to explain the kernel trick (i don't have enough reputation to add a comment directly to his answer) Was there a simple typo in the equation for the separating hyperplane? and it should be z1 + z2 = r^2, instead of z1 + z2 = 1? Or do I misunderstand? I agree its a nice simple example to illustrate the concept. Thanks. Though the definition of z3 still seems a bit of a mystery, but apparently it doesn't matter for the example centered at the origin.
On a single Likert item as DV, often ordinal logistic will be better than nominal. Likert variables are at least ordinal.  However, it appears that the poster is using Likert scales as independent variables.
see also "13 ways to look at the correlation coefficient" by Rodgers and Nicewander: http://data.psych.udel.edu/laurenceau/PSY861Psychological%20Statistics%20II%20Spring%202010/READINGS/rodgers-nicewander-1988-r-13-ways.pdf
I think that the so called wavelet analysis is the thing that could help you, but this is a pretty complex topic...
@statikop: I'm not sure, but we may have been talking at cross-purposes about the replacement of the `NA`s. (I think you mean `NA` = not available as opposed to `NaN` = not a number [things like $\\pm \\infty$ or other results of calculations that do not yield a number]). Anyways, I tried to clarify my answer and also expanded on the 20 subjects a bit more in detail.
@whuber: Thanks for your help. Yes, using maximum of sample means will always bias upward. But I'm not getting why i can't form an unbiased estimate? And taking a step back, 1) what assumptions can I make on parameters to get an unbiased estimator, and 2) with no more assumptions, can I form at least an asymptotically consistent estimator?
Thanks Tom. Here's a link to the full data set: https://dl.dropboxusercontent.com/u/65347419/daily.csv A lot of what you posted seems quite complex. Just fyi my goal is to learn how to do this, rather than just get a solution to this specific problem. The data is a web activity data, website's users are mainly from US. Among other things, I was kind of hoping to understand how holidays in different parts of the world affect the pattern - my guess that this should be reflected in the irregularities in the pattern. Thanks again.
Great answer. However, I'd be reluctant to convert a continuous variable into a binary one. A lot of meaningful variability would be lost.
One example is L. Fei-Fei and P. Perona. A bayesian hierarchical model for learning natural scene categories. In CVPR, 2005. But your question is unclear. Do you mean that for each document there is a single topic? Do you mean a topic corresponds to a group of documents (a class)? Or more than one topic? How do you plan to do classification? What other topics would you need?
Thank you so much **R S**. It has been 13 days that i was waiting for a response and i wasn't sure that my understanding from Markov chain is correct. Now i can continue on Hidden Markov chains.  thanks so much
+1 for penalized matrix decomposition. That package is pretty amazing. I should probably mention that it's spelled "Witten," though, in case people have trouble finding the cite. Lastly, the OP said they didn't want anything written in R, but essentially any large SVD package out there will have a C, C++, or Fortran backend for speed.
I think there are too many questions here that are not particularly related to one another. You are asking about identifying new variables, issues with missing data and data cleaning.
I don't see where time even factors into this example. Nobody is "followed".
First you say you want to estimate the parameters, but then you sound as if you know them? Do I miss something?
Hi Tony and Alecos,

Thanks for the responses - it does sound like logistic regression is the way to go here. 

Yes we do have explanatory variables such as event type, event day, event time and so on, and I would be looking to utilise those in my model.

I will have a read up on logistic regression - thanks for the pointers!
Thank you for your answer!I think the first two points are probably the reason.  I'll look at the different accuracy percentages for my features.  Final question for that: am I right in assuming that in the following picture (matrix), there are 35 features, or is it 1 feature with 35 dimensions: http://imgur.com/BfnwkZE ).
I would suggest you try different words and see what works and what doesnt..you can take the null hypothesis as "buzzwords do not matter" and see if you can reject it or not.
@fmark: sure, but at least you know it's not slow on a desktop!!! :D Anyway, I'm sure there's plenty of fast implementation of FFT (it's called "fast" Fourier transform for a reason). I never implemented one, though, so I cannot help in this case.
+1: indeed, all three answers here are logically consistent with one another.
Have you tried the "relative [Root Mean Square Error](http://en.wikipedia.org/wiki/Root-mean-square_deviation)"?
@whuber I don't know where the data is originally from. I found that discussion, tried it and found that my version of Sage outputs different results that the version original poster uses. I'm just curious if there are some methods to evaluate which curve fits better to the data.
(1) I haven't witnessed any "debate" in this thread: all replies and comments present the same basic message.  (2) Your implementation appears to be an *ad hoc* version of [stepwise regression](http://en.wikipedia.org/wiki/Stepwise_regression).  I agree that it can be useful as a starting point provided it is not automatically accepted as an end in itself.
Thank your revise for my bad format.
@user68 and dchandler - please do join in the discussion over on meta: http://meta.economics.stackexchange.com/questions/40/where-stats-questions-belong
What's your link function? That will tell you how to interpret the coefficients.
I see *cross*-validation as sometimes using the same data for training and sometimes using it for validation and error estimation.  The price you pay for this use of the same data for training and validation is potential overfitting and so less credibility for the out-of-model error estimate, which it why it may be worth holding out a test set completely from the entire training process.
@StéphaneLaurent: You are right.
Thanks very much Marius, I'm going to take a look at implementing perm tests now. It does seem like a convincing reviewer-friendly option.
*Data* are neither parametric nor nonparametric. It's fine to say your data are 'not normally distributed' (which data truly are?).
This question is far to ambiguous since it depends on things like size of dataset, number of features, hardware, training algorithm (there are different ways to train RBMs), etc.
Is there a way I could do this? Also, this is the differenced data so the original data is not stationary. Is there a way I can forecast on the original data?
Clever! Great answer.  All I could think of is trying to jam this to fit the logsumexp trick, but this is just right.
if $(a*b)/c$ is meaningful, then it should be acceptable as a predictor; such variables can occur regularly in economic contexts, for example.
The notion of constant random variable leads to the degenerate distribution. More on this and on constant r.v.'s (as well as a.s. constant r.v.'s) you can find in wikipedia [link](http://en.wikipedia.org/wiki/Degenerate_distribution)
+1 The second item isn't actually an assumption of anova. The *assumption* is equality of variance. What you have there is advice about how much heteroskedasticity can be tolerated. I'm curious about how they arrived at those particular numbers. Is there a paper from which this advice comes?
Would it be justifiable to perform apply a less conservative control of FDR at each level independently?  ie. to first adjust p-values with the B-H procedure over the three tests for each variant, and then apply the same method to further adjust those adjusted values for all the variants, for each test?
You will better communicate your answer when you re-read it for errors (in punctuation, grammar, typography, and spelling) and use effective formatting. Please review the [Markdown help page](http://stats.stackexchange.com/editing-help).
Sorry, my inaccuracy - By 'PCA matrix' I meant matrix of rotation
@tunnuz - how do you mean not very widely? It seems to cover the basics, and if you grasp all of the material within, you should be well able to move on to more advanced topics.
Which equation in particular are you trying to understand?
Yes, the green line is linear regression and the red one is smoothed with lowess.
If so, I'm clueless... As @chl wrote, some code (or, ideally, a reproducible example) would be helpful. It may be also a bug/strange feature of party itself, so it can be a good idea to consult package maintainer.
That is certainly true!
How do you define importance sampling in connection with hit-and-miss Monte Carlo? With no restriction, the minimal variance of an importance sampling estimate is zero.
You are using default parameters for `nu` and `gamma` (since you are using an RBF kernel by default). The default values for these parameters appear to be bad for `iris`.
@gung the method is called corsscat.The intro said that it has no free parameter that require tuning. what does that mean?
At the risk of adding even more confusion, I would suggest trying to read [ANOVA – Why It Is More Important Than Ever](http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aos/1112967698) by Andrew Gelman.
That's a useful relevant *test*, but the question asked for a *measure of association*. Is there one that goes with this test?
Both points ("covariance" and "r-squared") are correct interpretations. I recommend you [this](http://stats.stackexchange.com/a/83370/3277) my answer: $r^2$ is the product of two relative magnitudes of the covariance, and is quasi joint probability.
Have you done any research towards answering your question? Wikipedia? A search of our site?  At what point have you encountered difficulties understanding?
I edited the text so that it now includes some references.
(+1) Two good points.
So you first obtained $x_2$ from $x_{2cont}$, then calculated $x_1x_2$, then normalized $x_1$ and $x_1x_2$? Was that the sequence?
+1 The examples in the documentation suggest to me the confidence is pretty high, perhaps 95%.
I edited the answer to respond. In short, I did not give you a third set of estimates, just a more efficient way of getting the estimates of the fixed effects regression.
It sounds like your question is less about zelig and more about how to sample from a bivariate normal distribution. This is fine! Ask a new question about it, or search the archives. This link might be a good starting point, though. http://stats.stackexchange.com/questions/69461/bivariate-normal-probabilities
How was temperature measaured, and, more importantly, how was it calculated?
I guess you'll have to analytically solve for $P(\\theta|D)$, and then run a uniform through the inverse of the cdf unless you do rejection sampling.
@jb it's the last sentence, but done :P
Actually I have tried different time series models to do the prediction of sales for 2014. I am kind of exploring different machine learning algorithms and how we can use to do the sales prediction. It is a time series data. So my approach is to fit the model using data 2011-2013 and then do prediction for 2014 using 2013 ans 2012 data as an independent variables. Hope now my question is clear.
A Julian day is a *date*; it is not a count.  In fact, it probably should be treated as a *circular* variable.  See the related questions at [tag:circular-statistics].
@Srikant. This can happen when the explanatory variable has a weak but significant relationship with the response variable. Then the coefficient can be statistically significant but hard to estimate. Consequently, the MSE of predictions can increase when the variable is included compared to when it is omitted. (The bias is reduced with its inclusion but the variance is increased.)
@JoshHermann For the third point see http://stackoverflow.com/questions/20919719/python-numpy-iteration-improvements-for-exponential-smoothing-working-code-for
Your assertion is not quite true. Let the CDF be $F$ and the quantile be $q$.  If $F(x)=q$, you need to add some condition like $F(x+\\varepsilon) - F(x)\\ne 0$ for all $\\varepsilon\\ne 0$: this assures the quantile is unique (and gives some clues about how the proof might proceed).
You may be suffering from a false premise: level of aggregation is substantively important and both scale (number of units at a given level) and categorization function (the boundaries of a unit at a given level) profoundly affect centrality and variance estimates. You may wish to read up on the **modifiable areal unit problem**: See Openshaw, S. (1983). *The modifiable areal unit problem*. Concepts and Techniques in Modern Geography #38. Geo Books, Norwich, UK. Also: Dudley, G. (1991). Scale, aggregation, and the modifiable areal unit problem. *The Operational Geographer*, 9(3):28–33.
In statistics, bias is not the same as error. Error is purely random, bias is not. You have bias when you know that the expected value of your estimate is not equal to the true value.
Is this homework?
Is the question about how to treat the missing values; how to code LR in R; how to use LR generally (which would be awfully broad); or something else?