tttjjj/elt_bench huggingface dataset:
No [question mark in the title]: only the re-descending behavior matters from a bdp point of view. Yes [first question mark in the text], from an IRLS point of view this loss function is not more difficult than, say, Huber's. No [second question mark], there are other asymmetric $M$ functions and these could be solved by IRLS (although in practice they are not). But i'd be remiss if i didn't ask: what are you trying to achieve?
I apologize for taking a long time to answer. I put some example model types. Let me know if you want more clarification. I was thinking of discrete distributions as more difficult to sample from because it seems like they would be more prone to multimodal-like behavior. Is explicit normalization applicable when you have a mix of discrete and continuous variables?
My last two cents: I think one of the uses of the MISE and AMISE is for comparing the performance of different kernels in terms of efficiency. See [Comparison of Data-Driven Bandwidth Selectors](http://www.jstor.org/stable/2289526).
Thank you, Dimiter, for your answer. Will upvote when I get points. I am looking at the VAR approach, but it's taking me a while to get my head around it. In the meantime, I tried some more basic methods. I would love to know if these seem valid. Again, the aim is to determine the relationship of Hg prices and Au prices after taking into account a generic metals price index. I will explain my approach by answering my question below.
Have you read the help page for `contrasts`?
+1 and thanks for the links.
@Fucitol, you should add the link between the two questions, if people are interested in the question they will want to see the answers from both sites (if you do get answers from either site).
Can you provide a few examples of what you are referring to? Context would help a great deal in answering this question, as the model of gene expression is greatly dependent on the scientific questions one is asking. If you are interested in modeling the temporal aspects of gene expression, this article is really good: [**LINK**](https://drive.google.com/file/d/0BwDLtETWZT5IaEgwaFZsMHU4N0U/edit?usp=sharing)
Does the relative power drop off quickly for smaller N's?  What might the relative power be for an N of 5?  (I have yet to learn R, unfortunately.)
+1 wtb whuber's brain.
A table could work fine too.
(+1) This remind me of the `pairs.panels` function in the `psych` package by W Revelle.
Have a look at pages 13-14 of this [document](http://www.ee.washington.edu/research/guptalab/publications/UWEETR-2010-0006.pdf).
I've voted to migrate. :)
+1. But you sound a little uncertain :-).
0(normal), 1 (mild), 2 (moderate), 3 (severe) and these values are for 4 components.  Each patients has four discrete numbers.

I am going to find the correlation of this scale with some continuous variables, these continues variables also measured for 200 patients.

Target: To generate these discrete values by machine-learning algorithm based on continues variables.

1.what is best method for feature selection ( continues variable). Spearsamn-Correlation?

If the continues variables were highly correlated with discrete values. it means I should consider them as features?
Ty for your help. I'm not an official instructor. What can I do to obtain this solution manual?

By the way this is good material (Newcastle University). I will wait to see whether I get more (better) answers to my question, otherwise you get the 50 reputation points :)
Hong's response is correct (and I tested that the code works). You almost surely want to use the ExportMatrixToR module (2nd method), because they the 'a' object in R is a real R object, and not just a literal.
Thank you, @Bob Durrant~
You should try both models, and see which one produces better predictions.
These are reasonable intuitive definitions, but there is no need to be so dismissive of the other answers.
Dear Mr Michael, Thanks for all the trouble taken - prompt responses to say the least.
Thank you for your help. I am a beginner in Statistics. I am trying to understand your answer.
@naught it is sarcastic
My guess is Leibniz. ;)
+1  The points about needing to adapt the procedure to the nature of the data and the purpose of smoothing are critical.  Without that information, *no* answer will be objectively defensible.
Better to search for "ANOVA" than "anovar". Does this help? http://www.iaeng.org/publication/IMECS2010/IMECS2010_pp2051-2056.pdf
PLS regression can handle multiple response
@DavidR I have had good success using circular cubic splines for low-frequency phenomena.  You need enough to represent the highest frequency you wish to model.  That's going to rule out the hourly and daily periods, but might work well for the seasonality.  For the former two, parsimony suggests starting with a small basis such as a sine and cosine for each expected frequency.
Autocorrelation of *residuals* is a property simultaneously of the data and *of the method used to model the data.* As such, I would be interested in a more refined statement of your initial assumption: just what *kinds* of data and what *forms of analysis* do you have in mind when you assert that residuals are "usually" autocorrelated?
@metrix $x_i^{\\beta}=\\left(e^{\\log(x_i)}\\right)^{\\beta}=e^{\\beta\\log(x_i)}$
If your set changes time from time it is impossible to fit it to the same bell-curve. Suppose you have the set $A=\\{1,2,3,4\\}$, so 4 should be 800, but if a new observation 5 arrives, then it should become 800.
Statistics may be applied and may be not; still the concept of data is always present.
Hi Henry, thanks for your response. The finishing order analysis is interesting. Did you do a monte carlo analysis such as that mentioned by M.Berk above to determine the the probabilities? Is this what is meant by simulation? How did you calculate the probability of a particular finishing order eg. 9.0% for order: 31254. Did you use R or some other app?
How many significant results do you have? Sounds like the FDR is working correctly.
+1 really nice question.
Could you be more specific about what you mean by 'data is from the entire population'? ANOVA is a method than can be used on samples from large populations. When you have the data of an entire population the methods to use are different.
"... for the scenarios that ME is negative..." this makes me wonder if you using the mean of the error, or the mean of the *absolute value* of the error? Because if it's the former, positive and negative errors will cancel out. A set of errors {-100, 100, -100, 100} has a mean error of zero, despite every entry having an absolute error of 100. However the root mean squared error will be 100, as $(-100)^2 = 10000$, i.e. positive and negative values don't cancel. Whether you want errors to cancel will depend on your application. Most applications don't, so use mean squared or mean absolute error
@whuber Thank you..I will try to start a PhD in November, therefore I need to learn *a lot* about statistical analysis :-)
This is a popular and much-loved thread, even though it does not (on the face of it) seem to conform to SE standards for content.  (Just what practical question is being asked here? :-)  Some rules benefit from being ... bent ... once in a while.  However, please don't use the existence of this thread to justify creating new ones that fall outside our guidelines unless you think there is a very good reason to do so!  Questions about site policy are always appropriate in [Meta](http://meta.stats.stackexchange.com/) and debate is warmly welcomed in [chat](http://chat.stackexchange.com/).
@whuber gave a [nice explanation](http://stats.stackexchange.com/a/6351/930) of why and how the normality assumption really matters in ANOVA testing.
Thanks, good tip.  postscript seems to speak nicely to Word for printing.  Unfortunately the poor preview on the screen is a non-trivial problem.  In my environment a lot of Word documents are sent around and read on screen.
It sounds to me like you are not really interested in prediction, but rather in modeling, in which case a joint likelihood of the dependent variables given the independent variables seems appropriate. I don't understand the concern about the dependence. Prediction accuracy is measured by some metric, and risk (expected error) is evaluated under the joint distribution.
I've been a statistician for many years and I don't have any idea what you are talking about. I Googled q-value and found that it is the minimum false discovery rate; the only other time I've heard of q value it was (1-p) where p is a probability.

"Significant hit" is completely opaque. Googling "significant hit" and "statistics" yields stuff about popularity and about baseball.
Perhaps you do not have a problem and that there is a story to be told as to why your results are insignificant. That said, "if you torture the data long enough, it will confess". What's the model that you estimated? Did you try dropping any of the sectors? Can you provide some more information?
Without any context, and in the absence of all axis labels, how could *anyone* interpret that map?
This description sounds like either *spectral anomaly detection* or reduction of dimensions and clustering via [tag:PCA].  Given that your description and question are so vague, could you indicate specifically what kind of plot you are interested in the anomaly detection paper?  (After all, it displays dozens of different plots!)
If the process isn't stationary, how are you defining/calculating the autocorrelation function?
check out van Belle's Statistical Rules of Thumb.  If he has a rule for this in there he will give a nice justification for it.
Okay, that sounds interesting. I read about adaptive designs before but only on some slides so the paper reference you gave looks really useful.
If $N$ is selected in advance, & you hold to that decision w/o regard for what the outcome looks like at that moment, the probability of a type I error will be $\\alpha$. OTOH, if you continue until $p<\\alpha$, & stop exactly when it becomes such, w/o regard for what the $N$ is at that moment (& are allowed / willing to continue indefinitely), the probability of a type I error is $1$.
@Backlin Your plots are confusing as the upper and lower panels show the regions for **rejecting** the Null in a one tailed test where H1 R < 0 (upper, red area) and **rejecting** the Null of a two tailed test where H1 != 0 (lower, blue areas). The green area in the middle plot is **not** the rejection area for the one-tailed test with H1 R > 0. I think this is what might be the cause of confusion between people here and yourself. In the one side H1 R > 0 we clearly have no evidence against H0 because the observed R lies *in* the green area.
(+1) Good that you mentioned the use of Factor scores directly (and their correspondence with raw scores under certain conditions).
@DikranMarsupial I tried all of the kernels with c=10 and polynomial kernel is the only one which gave accuracy more than 70%.
[This](http://stackoverflow.com/q/7714677/324364) StackOverflow questions shows a couple of **ggplot2** options for this kind of plot, including the scatterplot+points.
@mpiktas It seems it is a computer simulation of traffic, so anything is possible.
There was a similar post here: http://stackoverflow.com/questions/243616/visualization-tools-for-huge-graphs
the reason i asked is because the pred label from the model is either 0 or 1... i want it to have values of 1 to 6...
Logistic regression is only a classification technique *in conjunction with* some rule that a predicted probability > $x|x \\in [0,1]$ is assigned to one outcome or another. On its own, the logistic regression model is inference on the latent variable Pr(class membership=1).
Projects usually have *budgets* that are rounded, so it is no surprise to see many coming in just at or right under round numbers. There's much more to identifying unusual values than mere inspection of the numbers--you need to understand what you're looking at.  Without knowing more about these projects, how they were budgeted, managed, paid for, and selected for this analysis, I doubt anyone could credibly answer your main questions.
It rather depends on exactly what you are trying to do. You could scale all your monetary values, perhaps using some kind of price, earnings or GDP index to do your fit, and then rescale back to the original nominal values; in some cases this would correspond to something like using the inverse of the index as a weight.
Recently stumbled on [this article](http://www.biomedcentral.com/1471-2288/5/13) about a similar topic. Hope it would give you some insight.
Hi Emre, what qualifies h2o as a more industrial tool?
Brilliant answer. I guess the dynamics of the CGF and MGF are more complex than I first imagined.
Correlated (non-independent) samples can be of 2 types: paired (repeated-measures) samples and nested (multilevel) samples. Of type 2, imagine an example: you study pupils from many schools belonging to few different regions. It is likely that schools (and hence pupils) from the same region are more alike than those from different regions. So, samples "schools" are not all fully independent of each other.
What are the points that are plotted? Does each `x` represent a cluster centroid? What are the `o`s? What do the `0`, `1`, & `2` represent?
Peter, I think that's Douglas Bates' (the author of R `nlme` package) opinion that inference on the variance parameters should not be done, and no CIs nor DFs for them should be reported.
This question needs additional information and clarification.  What are you attempting to *calculate* with these models?  What forms will your data be given in?  What are "C 4.5" and "K-nn"?  Note that in statistics a "model" is a mathematical description of data; it is not an algorithm and therefore the concept of algorithmic complexity does not apply *per se*.
I am using predict on test set. It's written in my question. From Matlab help: "err = error(B,X,Y)  computes mean squared error (MSE) for regression trees for each tree, for predictors X given true response Y." What you are talking about is (again Matlab reference quote):"err = oobError(B) computes mean squared error (for regression trees) for out-of-bag observations in the training data".
Not that I don't agree with the spirit of your argument, however thoughtlessness is not really the reason you're testing whether a coefficient is different from zero. It's precisely because statistical and practical significance are two different things that these thresholds exist. You run the risk of dismissing a regressor out of hand when your threshold is practical significance. The wording of such conclusions requires careful consideration so that readers don't get the wrong impression of what you're actually saying.
The variance of the sample mean ("and I know that...") scales as 1/N, not 1/N^2.  That's the only change you need.
You need both the SMOKE and interaction variables in your model, but not the Daily variable. The parameter associated with the interaction variable is interpreted as the effect of daily smoking (in addition to just smoking). Don't get hung up on the interaction bit, the multiplication is just an implementation of the IF condition you want, (which incidentally is what interactions is)
Analyzing your questionnaire highly depends on your research questions and objectives especially. From experience not all questions existing in a questionnaire are in line with the research objectives. However you may find that demographic data like age, marital status, gender, etc., are equally important to reflect in some question opinions. In SPSS, try to merge your data input into orthogonal data sets. That way it will be easier to analyze your questionnaire, as this groups many queries into one individual field.
@chl, no worries, my surname is constantly misspelled by my native language speakers, so my expectations for not misspelling the nickname are really low :)
Thank you very much for your comment. I started to think about  regularization approach to select the best model, because I have observed a severe multicollinearity between my variables (VIF >> 10). Many of them were correlated at the level of > 0.8.Thus I was expecting elastic net to perform differently to lasso and show a grouping effect (in case of correlated variables).
Great answer, and it makes perfect sense! Regarding your last point, can I use the coefficients? If so it helps my story, that is why I am wondering. Or, is there a way to test if their OK? If so is it common to report that this happened. Lastly, what is the difference between using a fixed effect for something versus clustering on it and is there a test to see which is better to use?
The question is good, but it is predicated on several misunderstandings. The first is that removing the skewness in the values is desirable for OLS. *It is not*: the objective would be to reduce the skewness of the OLS *residuals,* not of the data themselves. The second misunderstanding is that taking a logarithm is a good approach to analyzing data that include zeros. It might or might not be, depending on how those zeros arise. (This question has been addressed in several past threads.) Yes, of course there is a formula to find such a $c$--but applying it may make things worse, not better.
The Bayesian predictive posterior is contrasted with an method where you take the maximum likelihood estimate of the binomial probability & use that to predict future observations as if it were the known value - i.e. you plug it into the probability mass function.  It's not very good (for small samples) because it fails to take into account the uncertainty in the estimate. (If you were to use a more appropriate frequentist method it would be quite close to the Bayesian one using a uniform prior, I think.)
If you type ?logistf you will see
**predict 
a vector with the predicted probability of each observation.**
is that what you want?
@Keith: Dataset is perfectly prepared for `st` commands - I'm working regularly with `stcox` and `streg` commands on it. Not sure about the transition from st-set data to `ltable` and handling its results though. Thanks for the link - seems to be down (for me) at the moment, but will have a look.
@NickStauner: From the random number generator, same as $X$. I'm not sure I understand your question.
Actually, I'm interested in half of the residuals at a time. I study positive residuals and negative residuals separately, as DVs on their own, with potentially different predictors and models. No, I'm not interested in variance.
You can look at the pwr package for one approach to these problems.
Could you maybe add the header and first ten rows of your data to your question? It might help people visualize what you're talking about!
That is a very strict definition of "true experiment."  The populations in medical trials are generally convenience samples, for example, who are then randomly assigned.  So, they would be quasi-experiments.  Most people seem to mean just "random assignment" by experimental.
1. You might want to change "STD" to "SD" which is more standard and STD had another meaning (to medical people, STD means sexually transmitted disease).
2. Your final equation isn't quite right. The multiplier is not 1.96 but rather is a constant from the t distribution. With large n, it is 1.96 but with smaller n that multiplier is larger.
(+1) I'm really a big fan of [Denny Boorsboom](http://sites.google.com/site/borsboomdenny/dennyborsboom)'s papers.
I think you have `lower.tail` the wrong way round. `ppois(q, lambda, lower.tail = TRUE)` gives $\\Pr(X \\le x)$
Thanks Walter. I have also noticed that OLR would be appropriate, but personally I find it hard to interpret the results and to present the results in a straightforwards way.
I think the author's interpretation of the `Standard deviations` is slightly off. Since the standard deviations are in fact standard deviations, we must square them to see how much of the variance each component represents. The first component would represent $100\\times\\frac{29.1001249^2}{29.1001249^2+\\cdots+0.4347983^2}$ percent of the total variance.
This question conflates the polls taken from time to time and the impending referendum which will decide the issue. The short answer is simply No; any poll provides at most an estimate with unknown biases and unknown associated variability, although in turn there are ways of quantifying some of that. Your real question is hidden under your example but it seems too broad to be answered easily and well. I doubt that the tags you've provided are especially relevant to what you seem to want to know.
For Peter's example, if we have $P(Deck=1)=P(Deck=2)=\\frac{1}{2}$, we have $$P(Card=Spade) = P(Card=Spade|Deck=1)P(Deck=1)+P(Card=Spade|Deck=2)P(Deck=2) = \\frac{1}{2}\\frac{1}{2}+\\frac{1}{3}\\frac{1}{2}=\\frac{5}{12}=0.416$$. That is what you asked for?
Check if the differences are due to different parameter values or different formulas. Different parameter values can arise due to different ways of initializing, and convergence to local optima.
@Dilip Although you are correct, your approach seems overly harsh.  After all, people *do* refer to the variance as a "measure of spread about the mean." If you read this answer, then, as saying "variance is one way to measure the spread of a variable about its mean, so equality of variances means those measures of spread are equal," then it makes perfect sense and does not require the assumption of Normality. Moreover, although such an answer is not terribly informative--it's practically circular--it does directly address some of the confusion expressed in the question.
Constant alphas seem to serve the purpose to model the prior distribution on the labels. As you supposed, correct alphas would likely to superimpose those labels that are encountered more often in the training set. If your model works well without them, why don’t you just drop them from the model? Your description is not verbose enough to answer why alphas would grow and spoil everything, but you probably need regularization. Try to add the gaussian prior on alpha to the model, i.e. add $\\lambda \\|\\alpha\\|^2$ to the log-posterior, this would probably prevent overfitting.
(Sorry for the *very* late reply!) I think you want to say independent variables... If you change your dependent variable you are messing up with your $RSS$ once more as the "model fits" (roughly speaking, $\\mu$) are not compared against the same $x$. (Take your time answering @Curious, I won't expecting anything before mid July! :D )
Nick you're a god send! I love this site. Thank you so much for the article on Likert items. I think it will solve the problem.
Median age need not represent the biggest age group. In statistics "changes significantly" would be best established by a formal test; changes that are big,  striking or notable can be described informally using any simple word. That aside, the median can be unstable too in small samples. First, it must be a sample value or the average of two values. More importantly, you are rounding to seconds, so the median may easily jump by 0.5 s. In a larger dataset I would expect the mean for data like yours to be a bit larger than the median, but why not use both? Why do you feel you have to choose?
@PeterRabbit, You could see this for yourself by fitting both models to the same dataset (either an example dataset or just simulate some data) and see what is different in the summaries and what stays the same.
I should have said in the question that data are i.i.d, but if there are daily predictors that isn't important (for me). I just don't know how to adjust the variance based on daily data to make a confidence interval for the aggregated monthly mean (which doesn't really exist yet because there is only one month of data).
Disagree about scatterplot - pattern is pretty obvious there. Low values are followed by high values, high valuse by low values.