tttjjj/elt_bench huggingface dataset:
+1 for stating the solution that is obvious and yet grossly under-discussed.  Every method is fraught with issues, but nonparametric smoothing is fundamental and there needs to be a good exploration of how all other proposed detrending methods relate to this one.  Would be pleased to hear of relevant sources ...
Before proceeding I have a few clarification questions and comments. One problem I see is that you have duplicated trial numbers for each participant. Thus, although each participant completed 80 trials, you have 1-40 twice. A trial number should be a unique identifier. Also, is it the case that each of the forty sounds you have represents a separate manipulation or emotion (which is what you are testing if you included 40 levels of the IV) or are they grouped into the 5 emotions you mentioned?
That's very helpful, thanks. With these particular data sets, it makes sense that the data should be correlated and that's a useful result to have. I was a little suspicious because much of the data lies within a segment of the X-axis with a small number of outliers, so I've been plotting the data on a log scale, which makes the line of best fit look a bit odd.
(+1 for the rapidminer tips) Agree about R questions and stats software. However, questions about statistical software that are barely related to statistical analysis per se will be better served on SO (which is clearly stated on the [FAQ](http://stats.stackexchange.com/faq), see the *programming* subject). Now, the ongoing debate about stratified sampling in the comments makes it relevant (to a certain extent) on here. I'm sure @steffen will agree with that, and his first comment might also be temperated by the ensuing ones.
Or is there another possibility to check autocorrelation for ordinal data? Or do you know some tests for multicollinearity? It would be great to get some input. Thank you very much
+1, those are cool!
Your mlog1 function computes minus the log-likelihood of a sample given a mean and a std deviation. You can compute the difference between two log-likelihoods by giving different values for the mean.
Hi, thanks for your reply, i think this is trying to show how much Social influences other channels, which isn't quite what i'm after - but maybe I've misunderstood. "Direct" is a wrong label, it should be called "unknown" and i'd like to redistribute "unknown" & it's portion of traffic to other sources based on the correlation, so if social has a 0.6 correlation with it, it's possible that the "unknown (direct)" is actually partially mislabelled social traffic, and partially mislabelled other traffic, can I use the second part of your answer "everything adds up to 100" to redistribute it?
@bayer, I am not voting this down (though I'm a bit tempted), but the independence assumption strikes me as wholely inappropriate in this circumstance and would lead to absurd inferences!
Thanks. It looks it is a great amount of material to include in ony one course. Since my university will not allow me (in some sense) to learn probability theory. I will self-study probability theory and then take a course based on one of the stat books you suggested.
I voted to close this as I fear answers will be primarily opinion based.
I think this approach does get at what you are looking at (the model selection aspect is really just a bonus). For NOX instead of just being able to say one unit increase drops housing prices by 13 units; you can use Table 2 to say that *and* that NOX explains about 1% of the total home price (as measured by the reduction in squared error over the simpler model).
I actually made it confusing in that equation, the x's in the equation should be a different variable to differ from the random variable x=0.
Why are they zero? Is it because of limitations on instrumentation in measuring small quantities, or are they real zeros? What does the distribution look like with $\\log(y+1)$? (And why do you write $n$ as the response variable? By notational convention, seems like you have a count, not a continuous measurement.
Gibberish detector is not the preferred approach, as mentioned. It'll be hard for me to train it as I don't have a good sampling to use. I need to estimate the odds that a word that doesn't exist in the DB is valid or not based on the other words that do exist.
Yes, you can use weighted least squares: http://www.itl.nist.gov/div898/handbook/pmd/section1/pmd143.htm which allows you to enter your specific errors for each measurement. Then you can use the standard errors from the package to form your CIs.
Because the means aren't discrete anymore. You may want to look at k-modes and similar variants of k-means.
Can there be much faith in it unless you also have a model for the rainfall process?
You probably meant, "7 explanatory variables".
I have.  On sample data sets, my estimation technique is accurate to > 5 decimal places.  I found no previous work doing this, so I just stated in the manuscript what my test results were.  The advantage here is that I do not have to worry about my sample being representative,because there are not samples.I need to worry instead of whether or not my number of 'slices' are sufficient.  It seems that it is.Taking a sample would actually slow the process down as I would have to visit each cell once more.I am writing this ground up in a compiled language.
I am yet to come across a case where it does something negative. For instance, I even tried applying feature standardization to binary one-hot vectors for which, ideally, you don't have to do anything and found that it didn't make anything worse. Just the learned weights were different because the data was presented differently. I also realized that this was pointless as my data was (after standardization) in a very similar range with the same regularities as one-hot vectors.
There are errors in the code as provided.  Throughout you put a space in the middle of logical operators, e.g. <=, >=, ==, and !=.
@Alexis What I've found so far is this [link](http://comments.gmane.org/gmane.comp.ai.weka/33141) where it seems to equal to entropy in the discrete case. But I'm not sure how it applies in the continuous case and if it's even meaningful for regression. The fact that complexity goes up instead of down is not a good sign...
@dcl In the loop, when $j=1$, it is adding $\\hat{\\beta_0}$ to $\\hat{y}$ (assuming the first list in your Xlist corresponds to the intercept term).  Can you add the snippet of code you're using to your question so I can look at it?
Check out http://www.mloss.org
+1 "educate the public" == *communicate* models. (To whom, how ? papers, pictures / infographics, interactive models ...)
Hi, I've been trying to get this to work, but just can't figure out how to get the code to work in R. Could someone give me another hand with that please?
Does it mean that the result in the paper is wrong?
Thanks. In fact, I guess I can get best results using human-directed feature extraction. BTW, thanks by editing my answer.
What exactly do you mean by denoise? Do you want to remove some data which satisfies certain criteria, or do you want to estimate the signal from your data?
"I'm looking for a way to predict the number of attendees **[per event or per week of events?]** for a given weekly frequency ignoring all other variables."
Water quality data tend to be very highly intercorrelated: the first PC of the *raw* data typically accounts for 90+% of all variation. For analyzing chemical composition, one usually bases the PCA on the log data and/or on the correlation matrix and often focuses on the PCs *after* the first.  (Another approach is to standardize the concentrations as percentages of the total in each sample: this can work well provided few samples have missing values.)  If you got the same extreme results with the logarithms, that could be due to how nondetects or low-level values have been processed.
I took the liberty of changing "only if," which is not correct--see @Macro's answer--to "if," which is correct (but does not tell the full story).
I was thinking about summing them up, I will try that up. But, the problem is that I'm not interested in comparing groups with each other; I just want to check the cohesion inside a group -unfortunately, very small.
@fred Why no mean modeling?
+1 Ha. Speaking of overlap, this may be the only answer with none over Epstein's reasons.
I'm just a learner around here, but I noticed that you say that you matched up the "closest geographically" point from setup 2 to setup 1, seemingly without knowing that the two devices actually were at the same location when the respective measurements were taken. That seems to my naive eye to be cheating in favor of setup 2. I defer to my statistical superiors on that question.
thanks for your extensive answers. Actually, I am simulating data for a multilevel model. I just used the code and adopted it for a simple linear regression scenario.
IMHO leave-one-out is certainly not the *best* way to validate such data. Iterated/repeated $k$-fold or leave-$n$-out (with $k$ < no of cases / $n$ > 1) is certainly better, as is out-of-bootstrap. Nevertheless, LOO is a sensible starting point for someone not familiar with resampling validation.
Yes, it makes a difference. Miraculously, the density of $Y_i$ looks correct at the end, though ;-). Next, the $\\beta$ should appear in the numerator of the expectation of $Y_1$ to make any statistical sense, not in the denominator.
Steffen - fabulous! Thanks very much, I did not see that function. I could see that ranking by lift (or other measure) to determine which rule to keep when several match would be quite easy.
The time series are traffic data (i.e., speed, count, spacing.... of vehicles on freeways over time on each day). The goal is to  summarize/aggregate each time series by one variable and then apply multi-dimensional scaling to cluster days of similar traffic characteristics.
You can accept your reply if you like.
+1, Welcome to the site, @TyronJamesHall. I took the liberty of tweaking your answer to better match the markup supported by CV. Please make sure it still says what you want. Since you're new here, you may want to read our [FAQ](http://stats.stackexchange.com/faq).
+1 - A good exploratory approach here might use two different types of spatial weights matrices, one for neighbors on the same wall, and another for neighbors within the catchment (but not on the same wall).
There is potential for confusion due to different possible interpretations of what is meant by $f$, $\\hat{f}$, $x$, and even $E$ (what probability distribution does it refer to?). Please define these terms to make this question capable of having objective answers.
Obviously you have accomplished (3): the points are labeled by color in the right hand plot. What, then, is your question? What goes wrong?
Your theory needs to take into account the available sample size and the uncertainty on your data as well as the properties/ characteristics of data and problem. Otherwise you are subject to a no free lunch situation: your problem is so general that it cannot be solved. Not even theoretically.
@whuber I tweaked the final paragraph to hopefully make a few of the points you mention clearer.
I'd also suggest that your statement that BIC is more parsimonious than AIC needs to be expanded to clarify that BIC will typically choose simpler models than AIC, but that this by no means implies that the models that BIC chooses are "better". BIC fundamentally assumes that the "true" model is amongst the models under consideration, which is probably a stretch for most circumstances. AIC on the other hand simply attempts to minimize future error prediction (hence the connection to cross validation) and makes no claims regarding the "truth" of the selected model.
@user570593 In fact the link I provided in the answer is trying to give weights to instances. I didn't get a chance to try, but in their readme file, it says, -training_weight_vector: An m by 1 vector of training weights, where m is the instance number. You may take a look at the wgt weight file included as well. Thanks
Even so, what is the aim, in particular? What is it you're trying to find out?
I think the bigger concern would be the likely effect of a self-selected sample on your inference.
While I accepted the other answer, I'd like to say thanks for this one too (which is also quite useful)!
**Hint**: [Rosenblatt's perceptron](http://en.wikipedia.org/wiki/Perceptron#Separability_and_convergence). I believe this is also discussed in [ESL](http://www-stat.stanford.edu/~tibs/ElemStatLearn/), but if there's anything about a proof, it's probably in an exercise.
OK, I am glad things are clearer now!
@kittensatplay: I think it's better for wikipedia to be a good reference rather than a tutorial.  Textbooks make better tutorials, and they can even accomodate the student better because you can have a different textbook for each approach and learning style.  Have you considered getting a statistics textbook?
Since I am being quoted in the OP: admittedly, I exaggerated a bit in the answer you are quoting from. The point is that in reality, two groups are almost never *exactly* the same. When you have large enough samples, null hypotheses of the form "no difference" will almost always get rejected because of this.
Ordinary least squares regression is not suitable for ordinal catgeorical dependent variables. If you try searching on '[ordinal response](http://stats.stackexchange.com/search?q=ordinal+response)' there are some useful questions and answers. Searching on 'ordered categorical response' finds some more, such as [this](http://stats.stackexchange.com/questions/56923/attempting-to-investigate-a-relationship-between-one-variable-and-another-that-h/56942#56942)
By _binarization_, typically mapping gray levels below a threshold to black, and to white otherwise.
Now I understand where the problem is. It's a great demo. Thanks a lot!
Ok that makes sense to me, but how are the fitted values calculated?
I'm calculating the RMSE of the differences, that is, calculating the square root of the mean of the squared differences between the true and estimated values.
It's a good question.  But let me ask you this, to provoke some reflection on the issue: if you are unwilling to estimate a sample size using a Normal approximation, what method do you propose to use instead for estimating the sample size?
Interested in the distribution not just the standard deviation. As the sample size increases the distribution of the MLE estimate tends to be normal.  My question is around what is the distribution if we have small samples.
(1) your data look corrupted. Did the data read in right? (2) What's in D1 after you run your code? (3) what were the estimates from the GLM like? (4) "what's wrong with my code" is usually more of a programming question
I like it, though I'd replace ".99" with "at least .85."
"absolutely correct" because copied and pasted "from a open source library": would that it were so....
I don't think this is completely true, because if you implement dummy variables in the OLS framework, you do not need to assume independence of the locations. In fact you can implement codings for as many different levels and features as you like, though that might be mathematically equivalent to whatever hierachical regression is in the OLS context (I dunno lol)
@AsymLabs In R, you can just use the command `lm(c(32,40,46) ~ c(1,2,3), weights=1/c(6,8,40))`. To get slope = 7.4359 and intercept = 24.7179.  That gives the fit whuber described.  It's a regression weighted according to the inverse of the variance.
Very funny! (more characters)
This is a small N.  You might want to post the actual numbers to get better advice on your problem.
Is your question on how to *technically* plot the lines and points in `R`?
But again, I should remind you that you should have a theory for this process if you're going to choose a given interpolation method.
From what I quickly gather from [The R outliers package description](http://cran.cnr.berkeley.edu/web/packages/outliers/outliers.pdf), I tend to say you may be able to say something about 20 observations, yes. But don't quote me on that.
Maybe somebody will chime in with a full response, but in an "intuitive" explanation we can always hand-wave away constants (like $n+1$) that do not depend on the variables of interest ($x$ and $p$), but are required to make the pdf add/integrate to 1. Feel free to replace the "equality" signs with "proportional to" signs.
@Jonathan, I think your description of sufficiency is a bit flawed. Or, the equation you've given simply has a (very) unfortunate typo. :)
Thanks! That definitely seems like something to look into.
+1 because it is not obvious to get what predict.coxph exactly does from the help page!
Can you also give us an idea of the support of the discrete random variables $X$ and $Y$?
There are lots of variations on similar themes for diffusion, variation (whatever you want to call it). 

The standard measure of such variables is, of course, the frequency table. There's no measure of "central tendency" because there is no center.
I guess you meant, "regress $Y$ on $D$", etc. The dependent variable is regressed on a set of explanatory variables.
This is rather confusing and is not correct.  "The ratio of between-subject variation to the total variation is", in fact, the square of the correlation. It is always non-negative where as correlation can be negative. You are also using terms related to Anova, ie between-subject and within subject. The question relates to two variables and the relationship between them.
Don't harass people for answers, it's rude. If mpiktas has an answer to offer you, you will receive it. What link function did you use? Most packages offer predict functions.
(+1, long ago). Sometimes we ignore the most straightforward ways in languages like R and MATLAB due to folklore or experience with them not working well. Sometimes, like this one, they turn out to be surprisingly competitive.
Thanks! Great answer.
Unless you can be sure that your modification to the second study affects 3 and 4 person groups in the same way I don't think it is very sound to combine all the data. Can you provide more info about the data or modification to the control group?
+1; Another way of saying the same thing is that it sounds like you are interested in a test of your model against a null model that estimates the intercept only (e.g. base rate of satisfaction).  This approach will lump together all variables and see how much better the set of them do at predicting satisfaction than just using the base rate.  I'm not sure I agree with the 'protection' suggestion (requiring a significant overall model before looking at individual coefficients) perse... but the gist of this answer is right.
@EngrStudent The samples come from different benchmark sets, so there is no nice underlying model, I suppose, since they are not iid. But I think the effect on the plot is due to a mixture of your and my hypothesis. I've added the plot for the censored variable. so you can compare.
@Rob Hyndman I see, thanks for the clarification.
Just to double check I'm understanding your notation, when you write $[1,x]$, then $x$ is (presumably) a positive integer and this refers to the set of values $\\{1,2,\\ldots,x\\}$?
There's a lot of good information here, but given what the OP is explicitly asking for, could you say a little about why the condition number is a better metric to use to diagnose if it's a problem, and what the best cutoff value might be?
Your data are *censored*.
In the meantime I would encourage you to unaccept this answer.  I would ordinarily delete any answer of mine known to be incorrect, but (as you might be able to tell) I like solutions based on first principles rather than detailed calculations, so I hope this argument can be rescued.  I therefore intend to leave it open for criticism and improvement (and therefore made it CW); let the votes fall as they may.
Feel free to upvote and or accept my answer if useful. :)
When you have that many features, I've found that `glmnet` in R usually does very well.  It uses a Coordinate Descent algorithm that gives a full regularization path and is usually amazingly fast.
And in particular, images use compression different from audio files, which is different from documents. The best compression comes from having the best knowledge of your set of files.
(+1) I have deleted a previous comment and instead made direct edits to the answer. I hope you don't mind. Please feel free to edit more to your liking and style. You have provided some nice, concise intuition.
@andreister wasn't my answer to that question also the answer to this question?
I don't understand what you mean by saying that the kurtosis or the skew "ranged". You should *only* be concerned about the normality of the residuals (you can see my answer here: [What if residuals are normally distributed, but y is not?](http://stats.stackexchange.com/questions/12262//33320#33320)), & there should only be 1 skew value & 1 kurtosis value. In reality, even then your residuals can be a little non-normal; w/ N=157 the central limit theorem will give you a little leeway.
Oh, and I don't mean to beg... but if someone could **please** upvote this so that I can finally break free from the limitations imposed on newcomers, that would be very much appreciated :)
@ Gaetan Lion The purpose of my study is to fidn out if each of independent variables separately has a postive association with the dependent variable? Therefore, I was wondering if I should put only one expected independent varibale in regression analysis to test my hypothesis?
Doesn't [the section on difference functions](http://en.wikipedia.org/wiki/Krippendorff%27s_Alpha#Difference_functions) in the Wikipedia article directly answer this question?
Thank you for your explanation !
possible duplicate of [Test the significant differences between the coefficients obtained by logistic regression on two data sets.](http://stats.stackexchange.com/questions/20330/test-the-significant-differences-between-the-coefficients-obtained-by-logistic-r)
$Y$ (the outcome) does *not* have to be normally distributed to fit a linear regression. The *residuals* of the regression are assumed to be normally distributed (see [here](http://stats.stackexchange.com/questions/12262/what-if-residuals-are-normally-distributed-but-y-is-not), for example). Could you explain what the outcome is?
It's going to be very difficult to give any advice without seeing a plot of the data. In any case, I think this question - "How can I improve my regression model?" - is overly broad in its current form. One could write a book on that. You may consider looking at some of the questions listed under the [regression](http://stats.stackexchange.com/questions/tagged/regression) tag, many of which are related, generally, to building regression models, which is what this question seems to be about. If you can't find what you're looking for there, can you tell us: What exactly is your question?
I have another question.  On pg. 242 of the article that you linked, the authors state, "Since we only consider undirected networks without loops, $w_{ij} = w_{ji}$ and $w_{ii} = 0$.  Is there any reason why one couldn't use the Laplacian centrality for directed networks?
I imagine you'll need to smooth it more before trying that, and maybe look at wider intervals
@Drew75 I am using Weka toolbox for training the SVM-classifier.There are around 420 positive images containing cars and around 240 images (negative samples).Now how to split them into training and testing set?Regards and thanks a lot.
It's just the [normal approximation to a binomial proportion](http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Normal_approximation_interval), using the [standard error of a proportion](http://en.wikipedia.org/wiki/Margin_of_error#Calculations_assuming_random_sampling); $\\text{s.e.}(\\hat p) = \\sqrt{p(1-p)/n}$.
@Glen_b, Let's start with the simple case, only 1 triplet of measurements. Thank You.
Perhaps confidence intervals, treating the votes as if they were random samples, would create the appearance of "objectivity".  The problem is that voting is anything but a random sample.