tttjjj/elt_bench huggingface dataset:
Let me see if I understand your question: you're looking for $a$ and $b$ such that using the linear combination you mention to rank features, and selecting the top $n$ features, F-score is maximized. Is that correct?
Hi @Stephan, and thanks for replying. It hadn't occurred to me to treat the hourly variance as the data set of interest, I'll give that a try.

In the literature most of the work seems to be about longer time scales--sub-hourly price excursions are treated as unpredictable. One of the first articles included the phrase "usually ill-behaved price series".

In case anyone finds this thread and is working a similar problem, the quick and dirty solution was to realize that you don't need to characterize the variance.  With a large data set, you can determine P(X>n) simply by counting.
@LongLaw it is theory which we assume to be able to identify parameters from the joint density of prices and sales. My model will not suffer from endogeneity bias since income is given outside to my model. I do not have a separate equation for the determination of prices as function of volume since income is exogenous and my parameters are combination of structural disturbances and of course exogenous income from the two equations.
However, when dealing with a fixed-coefficient term, if the idea of moving from $y=\\alpha+\\sin(x)+\\epsilon$ to $y^* =\\alpha+\\epsilon$ where $y^*=y-\\sin(x)$, and then using $\\hat y = \\hat y^*+\\sin(x)$ makes sense to you, you don't really need any special concepts to deal with offsets.
@rolando2 here's a graph: http://dl.dropbox.com/u/22681355/graph.png Basically I would like to see if the drops in each line are significant. So to give you a couple of examples I want to see if the green line on the top drops significantly from no.GREEN 1 to 2, from 2 to 3, 3 to 4. THen I'd like to check the same for the other colour of lines. Then I'd like to check whether there are any differences in the coloured lines, etc. For a more detailed description about the data and link to the datafile see: the link in my post
$\\left\\lbrace (x,y):(x^2+y^2<1), (0<x<1), (0<y<1) \\right\\rbrace$ are the points on the quarter of circle. I suggest you to try to plot this points
You might want to look at my response to http://stats.stackexchange.com/questions/16953/how-to-detect-structural-change-in-a-timeseries . It is VERY important to consider time trend changes.
Try this answer: http://stats.stackexchange.com/questions/83047/modelling-when-the-dependent-variable-has-a-cut-off/83059#83059.  Also, this exact problem is very common in survival analysis (because the trial usually ends before everyone is dead), so you might try searching on that.  The model you would use is not normally called a Tobit model, though it is really the same thing.
This answer is just an elaboration of a comment by @Elvis.
BTW, there are some interesting meta-questions worth considering out there about flagging old questions (a category into which this question might already fit, being one year old), e.g., http://meta.stats.stackexchange.com/q/1727/32036 and http://meta.stats.stackexchange.com/q/1409/32036 Hence one might prefer to flag the newer question as a duplicate, though this is a challenging call because it's already gotten more upvotes...
So you are suggesting that I can use `>summary(model)` for each individual model to see its R-squared (which is the variance explained), and compared the differences between R-squared. Then using `>anova(model1, model2, model3)`, I can further evaluate whether the differences of R-squared are significant. Is this correct?
Thanks a lot for your answer. I also though that Fisher´exact test could be a good method for the analysis. I have not any statistic software to perform the results for other functional classes I would like to test too. Do you know any "online" tool to obtain the pvalues with all the decimals?
You can compare both models to a model containing both predictors, and obtain two likelihood ratio tests.  More information about that is in my book Regression Modeling Strategies.
Thanks!  I've been pondering Arnold 2010 a bit.  The use of 85% CIs matches with my experience of 95% CIs not agreeing with the results of model selection.  However, he doesn't cite anything to justify their use or outline any of the math.  Any thoughts?
It's usually customary to do *both*.  And what do you mean by a logarithm variable?
Although you can always just delete cases with missing values, that sometimes would be a poor choice, depending on why the data are missing.
I have a .yaml file which is >3GB. Aside from being impractical, how would I identify such a variable? Is it even consistently causing problems? I already choose my variables based on their importance (as indicated by OpenCV learning algorithm)
"$\\hat{\\lambda}_{MLE} = \\frac{\\sum^n X_i}{n} = \\lambda^{-1}$" is nonsensical. The MLE of $\\lambda$ is *not* $\\frac{\\sum^n X_i}{n}$, so the first equality is NOT TRUE. And then the second equality isn't true either! You need to be much more careful with your work.
Lucy, if you know the joint distribution of X and Y (or are prepared to assume that they're independent), then you shouldn't need the Taylor expansion approximation. You may well be able to compute the distribution of the ratio you want - and then get variances - either exact, or to a much better approximation - from that.
Excellent, thanks very much for your help whuber.
'criteria' is plural. If there's only one, its a *criterion*. What form does your criterion take?
Thanks, this is exactly the sort of answer I was hoping for.  It seems that if I wanted to extrapolate to 3 letters, I would just sum over $p_x^3$ then?
@Robin I modified the question to try to make my point clearer
Thank you Michelle. I was a bit reluctant to automatically assign a "Does not apply" for 2) and 3) to anyone who picked "No" for 1). The reason is that the data were collected using hand-filled questionnaires, so it was as likely to be a mistake filling 1) than 2) or 3). I eventually looked for a "degree of plausibility" in the answers and looked at all possible combinations. Some (fortunately rare) combinations didn't make sense and I deleted those instances. I fixed the others as  you suggested.  I bought De Vaus's book and really liked it. It doesn't mention this particular topic though.
AIC, BIC, etc are derived from two important qualities of the model under consideration: the number of parameters (k) and the likelihood of the model (L). Consider the $AIC \\equiv 2k-2\\ln(L)$ - here we see a tradeoff between the number of parameters and the likelihood.
Some of your questions are positive and some negative. Did you reverse score some of the questions (e.g., "The menu is too heavy")? As an alternative to relying on one scale score, you might look at the correlation of each of the items with customer purchasing decision. That would let you sidestep the issue of whether menu attractiveness is unitary.
Thank you very much for your answer. Yes i want to use elasticnet.
But where does the pdf come from?  By definition, a pdf describes a theoretical probability distribution.  Do you perhaps mean the edf (empirical distribution function)?
I think you are trying to mix completely different techniques in Statistics that may not be correct. Could you please explain in a bit more detail: i) what are the data? ii) what do you want to accomplish. We can help you better this way.
I think it is some trick question/sick joke. I keep finding the question online, nobody knows how to solve it.
When all you have are the marginal distributions, as shown here, you have *absolutely no* information about their correlation. You could simulate a population having these two distributions. Then you could pair ages and incomes by taking the largest with the largest, the second largest with the second largest, and so on, creating a high positive correlation. You could equally well pair the largest income with the *smallest* age, etc., creating a high negative correlation. Your data say nothing about what pairing actually occurs.
Don't forget the differences in how BUGS and JAGS deal with truncation, censoring and prior ordering of those distributions (section 8 of the manual).  In particular, JAGS has a `dinterval` distribution, where BUGS works with I().
What's the outcome of your study? Also, could you provide a bit more context about this trial; that might help to stimulate some interesting and/or correct answers.
Sorry, not a QC guy. (Not even sure if traditional QC charts are applicable in a computerized age. But if they are useful, R's a great place to do them.)
You have two suggestions from Stata; you are seemingly ignoring both. Does the output show anything unexpected? (If you want views on that, you have to show the output.) What about re-scaling your variables, so that coefficients come out more nearly equal?
Can you add the name of a function/package you use?
(I've read the answer [here](http://stats.stackexchange.com/questions/27610/how-to-test-for-normality-in-a-2x2-anova?rq=1), but it only asserts that "normality within cells means normality of residuals". I'm concerned with the implication in the other direction. Does normality of residuals mean normality within cells?)
(+1) Well and clearly done.
See this thread: http://stats.stackexchange.com/questions/5347/how-can-i-efficiently-model-the-sum-of-bernoulli-random-variables
Could you please elaborate question and what is your objective? Hows you're data look like? (sorry I didnt find link to add comment)
Welcome to the site, @abida yousaf. Our goal is to build up a permanent repository of high-quality statistical information in the form of questions and answers, so please don't use texting-style abbreviations.
1- by 'labelled' do you mean if I am using cathegorical variables? in that case no, I am actually working only with scalar measurements this time. 
2- I am using SPSS two-step clustering algorithm.
Does the term of semi-supervised learning and transductive learning same? Also can you provide any paper for such kind of explanation?
+1 R is a good suggestion.  Be very careful about using software not specifically created for mathematical or statistical calculation!  In particular, do not use Excel--it makes mistakes--and do not rely on Web apps.
+1 for discussing the importance of practical vs. statistical significance (and thanks for the shout-outs/elaborations!)
A nice reference on this is [The Cost of Adding Parameters to a Model](http://www.jstor.org/stable/2345896). On the other hand, a reduction in the efficiency of the estimation is not a criterion for selecting variables/models.
You need some regularity conditions so that the median will have a normal distribution under rescaling in the limit. To see what can go wrong, consider any distribution over a finite number of points, say, $X$ uniform on $\\{-1,0,1\\}$.
Sorry for the late response. I expanded the comment to an answer. But I do not have much experience about the transformation of the outcome, thus I just answered your second question generally.
Just wanted to say - THANK YOU once again!  I ended up playing around with the orders and the period pattern.  I've tried to base my decisions on the ACF and PACF and I've finally found a model that worked.  Thank you very much for showing me the arima code above!
Have you considered using DIC as well?  I tend to use DIC, LPML, and staring at a histogram of the difference between means more or less at once.
The term 'events' seems better suited here than 'time series'. An ad hoc solution is fitting a continuously density, after conversion of the timestamps into numeric values. For instance the 'density' R function
can be of some help. The idea is that events occur at random times which need smoothing.

Many kind of statistical models can be used to describe events with possibly trend, seasonality, exogenous effects. One of these is the non-homogenous Poisson point process.
+1 That's a pretty direct answer, considering these functions implement the formulas in the paper!
I will take a look at that paper, I guess I just didn't want to use a hard cut off to decide whether items should be grouped or not.
Wald came up with his sequential probability ratio test (SPRT) to create a stopping rule, the number of subjects that you need to have evidence against the null. See my explanation here: http://stats.stackexchange.com/a/16120/401
This only tests a single hypothesis, though. But, when you propose a chi-squared test, that is only a single hypothesis (all treatments are equally effective). It seems that you could adjust the "primary" p-value in my post for multiple testing and do several tests. I would have to think more about how to incorporate the changing of the set of treatments.
Thanks for the reply, learners will take IELTS general reading exam before each eye tracking session. So, I want to find out which factors (saccade length, fixations and regressions) influence their scores and development. For instance; a participant scoring 22 will have an eye tracking session aftermath. Then I want to find out which factors affect this score and what is its relationship with previous scores and factors. This score-eye tracking procedure will be employed 6 times in an academic year. Thanks in advance.
how do i know if my data is stationary around a constant or trend?
So I understand you answer as if I want trend to be maintained in forecast, i should do the kpss test for trend. Am I correct?
I don't quite get what you mean by "maps", but if what you realy want is to _compare visually_ several frequency (or other contingency) tables, here is two choices among a few: (1) multiple (3-way) **correspondence analysis**; (2) individual-scaling model of **multidimensional unfolding**. Both are quite advanced statistical techniques, so you might prefer easier ways, such as mosaic, paneled, etc. charts showing frequencies.
@revolusions: The information is a bit spread out over some papers. But I added a list with 2 about the GA variable selection, and one about uncertainty in measuring classification error rates. If you have questions (or don't have access to the papers), send me an email.
Discrete distributions can easily yield exactly the same sample mean.
One way to view this "non-parametric" bootstrap is that it converts the parametric assumption of normality into a "quantity of interest" in some large, finite population (eg the mean of a Census of records).  In fact, you can show that this version of the bootstrap is based on "maximum likelihood" estimates of the multinomial model, with 1 category for each distinct "type" in the population.
On why fell swoop, see http://www.dailywritingtips.com/one-fell-swoop/
@AlexTwain Link is fixed.
Great explanation, you rock! (Plus, I had no idea we could use TEX in our posts -- thanks for showing me the light)
Just emulate http://en.wikipedia.org/wiki/Net_present_value#Example
it seems that the OP is doing exactly this, i.e. fitting the MLE, only in excel.
Thanks Tal! Sometimes the answer stares you in the face and you don't see it. "Let me put my glasses on, I can't hear you"
Take any two independent variables $X,Y$ and let $F$ be their distribution.  $F$ is a valid answer to your question.  Note that you are asking to prove a conditional, which by definition is true whenever the consequent is true, *no matter what* the truth value of its antecedent may be.  Thus, by the basic rules of logic, *all* distributions of independent variables are answers to your question.
@bigTree: I'm curious why. You can choose not to consider the case of $n=0$ at all, but if you do then anything answering to the name of the probability that $(K=k)$ has to sum to one over all values of $k$.
Measure theory is neither a field of statistics nor hard. Some types of integration are hard, but, once again, that isn't statistics.
Cohen's d does not have a universal scale. For experiments in psychology a d of 0.25 may be moderate, but in pharmacology it would be considered to be miniscule.
This question appears to be off-topic because it was edited to focus solely on programming issues.
Since the section you describe has been there since the initial creation of the article by [Tal Galili](http://stats.stackexchange.com/users/253/tal-galili)  you might try contacting him (that link gives a link to his blog, at which in turn you can find a contact email under 'Contact me') and asking him firstly what it even means to have association within a single variable, how he intends the computation to be done, and for a reference (if there's no citation, it should probably be removed from the article as original research) - generally, if a person says something, ask them what they mean.
This looks like an empirical CDF of the fraction of dimms with correctable errors with both the x and y axes in log-scale.
A little bit more detail would be ideal to get the question. At least, how does your data look like, who are the subscribers, what do they have to do with average time, and how is 'average time in service' defined?
+1. I'd also add that it is important how you present the rating scale _visually_ to respondents. For example, 1-2-3-4-5 may provoke a russian to intrapsychically distort the scale into unequal-interval because the scale is traditionally used for school marks in that country.
(This reply, which arrived here after a merger of two threads, is responding to a duplicate question framed in terms of a CI of a proportion.)
(+1) Thanks for that reference. [Here](http://www.sciencedirect.com/science/article/pii/S0304414907002177) is another reference showing that 0.234 is not the complete story.
@whuber: Thank's for the explanation, that's what I feared (no unified equation and loss of correction factor). I have added a note to my answer to better describe this. I am also going to add your line about the correction factor on the Wikipedia article.
That's right.  Unfortunately, wind speed distributions are far from Gaussian (unless perhaps one is looking at long-term averages) and their *departures* from Gaussianity are usually of most interest (people extract energy only from the higher speeds).  For an indication of what might be done, see http://www.nupus.uni-stuttgart.de/03_Scientists/Projects/A06_Li.pdf.
ah fonnesbeck is one of the core developers of pymc! Can you show us an example of how to code point 2?
Yeah, it seems like a system like this would either be really brittle or have lots of problems with error trapping.  Maybe this is a pipe dream and I should just get comfortable the process.
@Owusu, the expression of gratitude in this site is upvoting of the answers and their acceptance. See the [faq](http://stats.stackexchange.com/faq). Either way, glad to be helpful.
Out of curiosity, can you post your answers to (i) and (ii) either here or as a separate question/answer? Thanks!
How are you doing the variable clustering?
The practical rule of thumb (based on William Cochran's research) is that the chi square approximation goes bad with sparse cells (sparse is usually defined as less than 5).  If mi is 0 then ni is also necessarily 0 if the model is right.  So you would pick a wider bin that would encompass an mi>5 and hopefully ni will be greater than 5 also. Even with mi+ni in the denominator the dividing by 0 problem may not go away.
Brilliant so with more dice you simply multiply the possible outcomes.  If you want to express the probability of faces 1,2 and 3 landing face up how would you do so?  This is excellent Raskolnikov I'm learning thank you.
Sorry, I should have been more accurate. I saw of course these two references on wikipedia but I don't have access to them. Therefore I couldn't double check that the definition given on wikipedia was correct. I'd like to use this distribution for a paper, so I would have liked the confirmation from somebody that this is correct and the best paper to cite.
(+1) Very nice response.
The point is that they don't use AICc for k-means. AICc was designed mostly for regression modeling. K-means clustering is not regression problem and is not a modeling at all, in the straightforward sense. You may still use AIC, though.
+1 for the obvious point that nobody else seemed to address: they are different concepts and answer different questions.  Also in many instances much is lost by condensing the entire distribution into one summary number, so sometimes they both do a lousy job.
In this case (ie, your binary variables) you don't need to worry about linearity, @APCOInsight.
Average RMSE is `mean((fitted(model) - data$logPrice)^2)`. Well, I realize it's average MSE rather than RMSE.
@Nick Thanks for jumping in. However, you--and all other readers who chance across these comments--might like to know ilhan is not really concerned about this issue but is only using it to widen a personal attack being carried out at http://meta.stats.stackexchange.com/q/2064 and in comments at http://stats.stackexchange.com/q/99376. Because it has become evident that I--as the moderator who voted to close his question--lack any power of persuasion in this matter, I am standing aside and trusting that time and gentle advice from this community will soothe whatever feelings may have been hurt.
I rate Hansen's book highly too. Thanks for the linkage to the Imbens and Wooldridge NBER summer course and UK CMMP lectures. At first glance, the notes and videos look very good, so my plan is to work through them. Oh, excellent! I took Russian during my undergrad and used Stata during my masters... since I enjoyed both, this might be a good reason to revisit some old ground. :) Your input is much appreciated.
Apropos your quadratic scoring procedure, you ought to read http://stats.stackexchange.com/questions/2275/how-can-i-determine-accuracy-of-past-probability-calculations .  You will also get good guidance from exploring the [tag:logistic-regression] tag.
Cross-post removed
@StocSim I think the relevant determinant of what makes a question appropriate on CV is the scope given in the faq; that would be the basis on which to argue that it belongs here rather than math.stackexchange.com
somehow the question feels incomplete, have you posted all the probabilities?
@B_Miner Regarding your first comment, I would start by including only an intercept. If the trend looks somewhat exponential you can also add the slope parameter of the linear trend and see if it is significant. In general, it is better to start with a model with few parameters and if the diagnostic of the residuals is not satisfactory then consider adding other elements.
Can you provide with several other methods for my data?In my data, Y and most Xs are numeric variables and the other Xs are binary variables. Because there are 25 Xs, it is very difficult to formulate the suitable relationship between Y and Xs. Do you have any suggestions? Thanks in advance!
One issue - not the only one - is you don't have the true value of p to use for your estimate of variance in constructing the confidence interval (as per your formula), only an estimate.  When p gets very close to 0 or 1 this starts to be a real issue.
I agree with @mark999. The tools you'd like to use are very sophisticated - which is why, for example, there are many health warnings in Doug Bates' `lme4` documentation. Using _any_ statistical or scientific method that you (or your couthors) cannot defend leaves a large credibility gap. So, while I sympathize with the need to get _something_ out, reporting the results you get from simple tools is likely better science than reporting GLMM output, if the model formulation and fitting process are going to be unexplained "black boxes".
from LewSpe91 abstract: "An analysis of the nature of the task and a review of the psychophysical literature suggest that the traditional prejudice against the pie chart is misguided." EXACTLY! Well, that result surprised me. But that's the point: need to apply the scientific method, rather than datavis dogma, in determining what are the "best" ways to visualize data. If we do I'm sure there will be more surprising results.
posted a followup question http://stats.stackexchange.com/questions/104277/cluster-analysis-over-time-series-samples Thanks.