tttjjj/elt_bench huggingface dataset:
I have no idea what those models are. (I have only taken stats 101). Are you saying that I cannot rely on intuition to make this decision? It feels reasonable to me that the shape of a log function would fit my variable. Therefore if I take the log of it, it will linearize my variable and be a better representation of reality. I am ok with discarding intuition in favor of more systematic diagnostics, but it is an awkward pill to swallow.
[One place to start](http://robjhyndman.com/hyndsight/crossvalidation/)
What's your question?  (The last line appears to be a little garbled.)
You already *have* determined the distribution of the *data*: it is shown in your plot. What do you want to do with that distribution? What is the purpose of your analysis?
@Glen_b: "Learning" by observing many messages and assuming that the median value of each bit (of all the bits at the same position in the set of messages) was the actual value.  This was then used as the basis for each comparison against a newly arriving message.
is this homework?
There is no such thing as an "Arima package" and "forecast" is not a method. Also, as I stated in my answer, the median is not "wrong".
Taking the means of the x's is probably not the smartest thing to do if you have high skew. I would choose to regress y on x instead of y on mean(x). This will allow you to determine outliers during your residual analysis.
What is to prevent your algorithm--or indeed any algorithm--from producing identical pairings for every round?
I guess the use of the word block was a bad one since it's not blocking in the usual sense perhaps. Question 1-3 is regarding one area (customer support), Question 4-6 regarding availability, 7-9 regarding the quality of the help. I have the data in a matrix 10x10000, I guess using logistic regression in R on the matrix might be a straight forward sollution?
Thanks! I understand that the density function needs to be integrated over the range I want. But I am curious: what do I have if I plot the probability of getting one success in 4 trials for a lot of $p$ values? A PMF?
As for your additional comment, again, no: $g(\\bar X) \\rightarrow g(E(X))$ (by the continuous mapping theorem), and, by Jensen's Inequality, $g(E(X)) \\neq E(g(X))$ except if $g()$ is linear.
If you can recalculate A, it seems like you should be able to bootstrap an empirical CI that you can report along with your theoretical CI
2 and 3 are the same thing...most likely = peak of distribution. Could you provide more detail on the actual problem you are trying to solve. Not enough info as is.
$\\sigma \\neq \\sigma^2$ (except in special cases!) if you know what I mean. But, that is only part of the reason. :-)
I repeat my earlier comment:  "*It's not clear how one would talk about the proportion of 'red' if one doesn't know 'red' is a colour*" (that is 'a colour present in the jar'). Your standard error is a function of sample observation. Your response doesn't seem to clear this up. What, then, are you asking for, exactly? Can you, as requested before, give a small illustrative example **in your question** of the question you seek an answer to, or ask a new one? Are you asking about a SE for the proportion of reds when you draw a sample that contains 0 reds and you're not sure there are any?
An easy way of getting at this problem would be to use a Monte Carlo approximation. Would this suffice?
(+1) Sociologist's often interpret the groups means as contextual effects (although this is more often for nested cross-sectional data than it is for time series panel data). I will need to read up, of related note [Manski (1993)](http://dx.doi.org/10.2307/2298123) ([PDF here](http://www.ccsr.ac.uk/methods/festival/programme/rpl2/documents/RES_1993.pdf)) has an article that shows how such contextual effects are frequently not identified. For "reasons this is not done" I suspect it is as much difference between social science practice as anything, it might be a good question to ask.
There is always a chance that someone will pick all the correct responses; that's how [Powerball](http://en.wikipedia.org/wiki/Powerball) works. If you really need a threshold for some reason, you should probably decide where to set that threshold for yourself, based on whatever other info you have. There may be options for determining a threshold algorithmically as well. Just be sure you actually need a threshold, and aren't discarding useful information by dichotomizing your data...
I think your situation is what most would consider to be a "good problem".  A commonly used rule of thumb in machine learning is that you should have a minimum of 10 samples per dimension of your input data and preferably over 100 per dimension.  Using that rule, you'd like to have a sample size of over 2,000 observations.  Using only 10% of your available data still gives 1,000,000 samples. Considering that your sample size for 10% is about 500 times larger than the minimum desired, you may even want to drop down to 5% and run more iterations of your algorithm for cross-validation.
Document classification, bioinformatics (microarray data, genomics, PPI networks, ...), web data mining (Twitter, Facebook, Google, ...), ...
I've been using the gp function of gpml. (http://www.gaussianprocess.org/gpml/code/matlab/gp.m)
the docs say it returns the "predictive output variances" along with the "predictive output means".
Can you supply a reference to said paper? I can't conceive of any situation in which one would "bootstrap a classifier".
"Say I want to estimate a large number of parameters" this could be made more precise: What is the framework ? I guess it is linear regression?
What software/language are you using? Also you should provide a bit of code to clarify the question
I haven't read the paper you refer to, but note that in general you can view a grid search as a [Bandit task](http://en.wikipedia.org/wiki/Multi-armed_bandit) and use the [efficient algorithms](http://lane.compbio.cmu.edu/courses/slides_ucb.pdf) for that framework. This does what you mention: reduce the computation on parameters that are unlikely to be the best choice.
That's true. I simply thought it might be helpful for Emma to have it laid out a little more explicitly since her question indicates that she is likely a beginner with statistical analysis. In any case, I think our comments should make that sufficiently clear. Cheers.
Gung, thank you for your editing.  You made my question clearer.  David
Thanks -- to clarify, you're recommending I pick the model with the best AIC, then report the fitted coefficients of that model? In this case I get AIC-deltas of 5, 0, and 1.5, respectively. Is there a test that says that f1 is better than f0, through? That's what I would have gotten if `anova()` had worked...
I love infinitesimals. This is a wonderful explanation. Thinking in terms of the $2x$, which can be clearly seen to emerge from the derivative of the transform, is much more intuitive than thinking in terms of the $\\sqrt{y}$. I think that's where my sticking point was.
LSMEANS is SAS lingo for marginal means. http://onbiostatistics.blogspot.com/2009/04/least-squares-means-marginal-means-vs.html
Are you saying that the maximum likelihood estimator is _inconsistent_ in the context of logistic regression?
After some research, I think it's similar to a sigmoid curve, except flipped?
I think you're right that you lose $b_i$ when you do GEE. GEE in a sense averages over the individuals, whereas a mixed effect analyses models the variation between individuals.
I am new to this forum. Is there something else I need to add to get an answer?
Thank you mpiktas, I will have a look at those functions and see if they can help me with this problem.
(1) The r-squared is low. This is secondary to collinearity? (2) the lasso performs well in the setting of collinearity? (3) Lasso is a traditional way to manage colinearity? Usually one (a) removing variable (b) combines variables (c) orthogolonizing (sp?) http://stats.stackexchange.com/questions/25611/how-to-deal-with-multicollinearity-when-performing-variable-selection
@Glen_b Ah Ill be more complete in the future and add it to the question. I did think about however. With some datasets I use absolute L1 distance and other times still I use linear regression. So I was hoping to get a broad answer.
Unit-root tests do not test for stationarity - they test the existence of a particular kind of non-stationarity (i.e. of whether a stochastic trend exists). So if the series is constructed as a deterministic upward trend, say, as a function of time, the tests you mention do very well to reject the null of the existence of a "unit root" -that does not mean that they tell us that the series is stationary.
@AndyBlankertz I forgot to mention that one bonus is that PSPP can open SPSS data files (.sav) directly... if that's any consolation to your mother.
The previous replies make it clear that although this is indeed the conventional definition, it is *not* the one Jaynes was using.
Seems very similar to this paper http://eprints.pascal-network.org/archive/00002147/01/CrammerDeKeShSi06.pdf . It's been implemented in an open source project called jubatus.
+1, nice analysis. I think that is why Wooldridge included the remark. I still think it is possible to think of some example where the hessian will be indefinite. Artificially restricting the parameter space for example. In this example the parameter space is whole plane, that is why the local minimum will give semi-positive hessian. I think the time has come to write a nice email to Wooldridge to get his take on the question:)
What consitutes a reasonable sampling plan depends on a number of factors... have you checked the [Wikipedia page](http://en.wikipedia.org/wiki/Sampling_(statistics)) on sampling?
An update on this problem: no amount of model modifications (correlated residuals, freeing time points, smaller chunks of piecewise etc) helped in the end, and we had to eventually acknowledge (following advice from an awesome stats guy), that something other than "time" is also contributing to the linear change.
I came into the same problem for one dichotomous variable and one continuous variable. Another explanation is that, the effect of A may have different directions for B1 and B2 cases, i.e. they neutralize each other, and vice versa. You can actually see and test the direction of the effect of A for B1 and B2 cases respectively from the linear combination of coefficients, and also for the effect of B.
I wouldn't call Gibbs Sampling basic. It's rather challenging stuff. However, if you're not familiar with it, then this might not be the best exercise for it, as this is a rather difficult approach. That might mean that the multiple imputation approach I describe at the end might be more appropriate. There are many resources on this site and others that can describe multiple imputation or point you to books that describe it in more detail.
Nice bit of detail here. I hope you get a good answer!
What would happen in Metropolis-Hastings if $p(x*)/p(x)$ was always 1?
This isn't really nonlinear regression. Nonlinear regression means the regression is nonlinear in the *parameters*. You can include quadratic and cubic transformations of the independent variables in a linear regression.
Anything?  Your comment sounds promising.
This probably belongs on http://gis.stackexchange.com/
Thanks -- really like this one because you're "teaching a man to fish", while at the same time not being a jerk because he doesn't know how!  Very cool.
As I commented on StackOverflow, that comment is referring to the _sequence of lambda values_. The individual validation models are still fitted on the subsetted data.
If I understand you correctly, in this case, the numerator equals the number of observations beyond the threshold & the denominator equals the number of observations including the threshold. So, the probability of the test variable reaching level 2 given that it has already reached level 1 = 4430/6493 = 68.22%. Is this correct?
@Randy, not exactly. The idea is to see if you can get a similar difference in accuracy by chance, if the two algorithms were doing a similar job. I have edited my answer to explain in more detail and give a step by step algorithm. You can ask if there is anything not clear and read up on p-values and exact tests on wikipedia.
@gung Alternatively, this question might be construed as on-topic because it appears to be about how to interpret statistical language (which happens to be in `R` documentation) or, possibly, about standard procedures for a particular model.
What makes you think this data fits a nonlinear function and what function do you think that is? To me it looks pretty much like a horizontal line with a couple of outliers are 0.
Do you have any information about people who read multiple magazines? What does your data look like?
@NRH, thanks for the response. Just to follow up, and please correct me if I'm wrong, but my understanding is that the existence proof is required to ensure that we can construct an infinite dimensional 'process' using a set of finite dimensional functions. Is this accurate?
+1 Very nice distinction!
It appeares as if `Low back pain` variable is constant. If actually it is not, then check if you are mistakingly using splitting or filtering of the dataset.
@user30196 Revised my answer to include your question.
The formula doesn't really fit with the simulation ! Could you double check the equations ?!!? thank you
I am aware of biclustering, but finding coherent subspaces with only 30 samples is not very accurate (from what I understand) or what I really want in my downstream analysis. According to this paper, CLARA is stable at n>=30 for microarray data, which is about what I have. http://www.biomedcentral.com/1471-2105/6/S2/S10/
Another way, based on the `lm()` fit alone: `sqrt(diag(vcov(reg)))`
If $M\\sim N(68,3^2)$ then $-M\\sim N(-68,3^2)$. Therefore $W-M=W+(-M)\\sim N(65+(-68),1^2+3^2)$. This is, the values M1,M2,...Mn are not necessarily positive. I hope this helps.
"*But the dependend variable for the D=0 case would be a constant because MR is a constant (equals to P all the time).*" -- then it doesn't take a very big model to explain those values. I'm not sure of their value to your model otherwise. If you can make your subsequent suggestion work, then it sounds like it might be better.
As an example, collect some measurements of the side lengths $X_1$ and $X_2$ of a set of squares, along with their perimeters $Y$. Here, the multicollinearity is *perfect* (because $X_1=X_2$ within measurement error) and the predictability is perfect, too. However, there are infinitely many possible formulas leading to the same prediction, such as $4X_1$, $4X_2$, $2X_1+2X_2$, $10X_1-6X_2$, etc.
Sorry if I'm slow to follow - I'm quite new to statistics. A Mann Whitney still gives me p values of >0.005 so I'm not very worried about significant results. The data are far apart and it's a consistent so I'm confident it's a significant effect, I just want to make sure I choose the right test.
Most likely, yes. However, using RF to compute variable importance to reduce dimensionality,  followed by a method your audience is more familiar with, is a common compromise.  It's important to know that variable importance in RF is generally computed based on the performance within the RF (lift, reduce error, etc.) and might not all carry similar weight in other methods.
I'd refrain from direct advice, being not in your shoes. Speaking most generally, chronically very skewed (= too difficult or too easy) or leptokurtic (insensitive to differences among typical respondents) questions are psychometrically "bad" ones, and are worth to revise.
Would you mind posting a few references to some of the economics papers that you have in mind?
How does "no change non vegetated areas" factor into the amount of vegetation?
I remember seeing George Burns on TV being interviewed on his 100th birthday. He was puffing on a cigar. The interviewer made some comment about the incongruity of longevity and smoking.


George Burns: "Twenty years ago my doctor told me that these cigars were going to kill me"
 
Interviewer: "What does he say now?"
 
George Burns: "I don't know. He's dead"
The reason I bring this up years after completing the assignment is that there has been much discussion of the Global Warming "Hiatus" of the last decade or so (and also some clear explanations about how the heat is still there, it's just entering the oceans rather than the atmosphere). To me, the "Hiatus" pattern (a drop down to the lower 95% confidence interval bound of the prediction) appears qualitatively similar to me to the period around 100 flips in the image above.
@Peter That's worth converting this as an answer, if you have time and inclination.
@Glen_b alright i signed up for this website today so i wasnt aware of it, my other question which was legit just get downvoted and i was wondering why, makes sense now, thanks for the heads up
@chl, indeed. those inferences can be tricky.
great answer, thanks a lot!
Do they say that the variables are independent?
So if I use some commands in the format of your example above, I input something like y=rgamma(2617,0.2); x=5*y+rgamma(2617,0.2)+0.3*rnorm(2617); plot(x,y); abline(0,0.2). The plot shows a nice linear relationship. Then when I use the model to estimate y, do I input x as x=5*y+rgamma(2617,0.2)+0.3*rnorm(2617)?
That does two separate regressions.  (Although the OP states she wants something more, I'm fine with it--she has never explained what she wants to get out of the model that isn't already in these regressions.)
I don't have experience with Latent Growth Models but the [Wikipedia](http://en.wikipedia.org/wiki/Latent_growth_modeling) article says that _Models with higher order components, e.g., quadratic, cubic, do not predict ever-increasing variance, but require more than two occasions of measurement._
"bias" isn't the right term here; the estimates from logistic regression don't actually have well-defined means, so don't have well-defined biases; there will always be rare datasets where the point estimate is very, very large, if not infinite. "Median bias" is, however, well-defined.
Dummy variables *is* using categories and a categorical variable. At least, it is one way of doing so. With 5 levels, you have 4 dummy variables, with 3 levels you have 2 dummy variables. There are other coding methods as well, such as effect coding. I find dummy coding most intuitive to interpret. 

I would not delete the "don't know", but it is probably not good to lump them with the "no". I would create two dummy variables.  

Any good statistics package (R, SAS, SPSS and probably every other one) will do this for you.
Thanks, @DmitryLaptev. I didn't think it was difficult, though. My idea's just that the best answers, & the ones that serve as a repository of accessible & actionable knowledge, are fairly comprehensive. Your answer here is good, but requires some reading between the lines; in addition to the code (& you could have done just 1 as a demo), it might be nice if you linked to a source of more info on mosaic plots (such as [this](http://www.theusrus.de/blog/understanding-mosaic-plots/)), & maybe a sentence or two explaining explicitly why / how they are better than ORs, & their pros & cons.
@Watson Sorry for the confusion. My background is in gambling but I have little experience of horse racing. Different sports exhibit different manifestations of favourite-longshot bias and it sounds like I have mistakenly flipped it. Your point about greater percentage of the book is also a good one - either way if you follow this approach you should gain some insight into how they divide the overround. I've edited my answer to reflect these points.
Shouldn't this be a comment? @Tim: If you have a series of three variables, $t_1$, $t_2$ and $t_3$, when you difference, you lose one. After differencing these three variables you get, $\\Delta t_1$ which is $t_1-t_2$, and $\\Delta t_2$ which is $t_2-t_3$, can you see that you have lost one observation?
An even wider generalisation are [Lévy processes](http://en.wikipedia.org/wiki/L%C3%A9vy_process). Given that "The probability distributions of the increments of any Lévy process are infinitely divisible" and the family of $\\alpha-$stable distributions is a well-known class of [infinitely divisible](http://en.wikipedia.org/wiki/Infinite_divisibility_%28probability%29) distributions.
Welcome Patrick. I am struggling with the same issue but with Pearson r. If you check my entries, you will get a feel of what you can do.
I have used a uniform design for this and I think I going well, but the thing now I got s trucked is , how to do regression analysis, how to know the constants of equations and how to eliminate those factors which are not influenceble.
"Intuitive" does not imply "non-quantitative": the two can go together.  It is often difficult to find an intuitive argument that gives quantitative results, but frequently it can be done and the process of finding such an argument is always illuminating.
Continuing this approach, $\\operatorname{E} (R) = \\sigma \\int_{-\\infty}^{\\infty} 1-(1-\\Phi(x))^n -\\Phi(x)^n\\, \\mathrm{d} x = \\sigma d_2(n)$, where $R$ is the range & $\\Phi(\\cdot)$ the standard normal cumulative distribution function. You can find tabulated values of $d_2$ for small $n$ in the statistical process control literature, numerically evaluate the integral, or simulate for your $n$.
First, ROBPCA found 25660 outliers, not 4000 as you claim. This makes me suspect you don't really understand what you are doing. Concerning the error message you get: do you understand it? e.g. have you read section 6, p15 of [this](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.45.5870&rep=rep1&type=pdf) paper?
"My project flow is to multiply the prediction of a glm (logistic regression) model that is used to predict 0/1 if a medicine was injected at all with an lm (linear regression) model that is used predict amount of medicine injected" I can't make any sense of this sentence. Are you structurally testing whether, firstly, any medicine was applied and if so, the dose applied?
This reply describes one way to *compare* p-values, but the original question seems to remain unanswered: does the procedure make any sense and how does one interpret the results?
Sometimes this method works, but it has a significant shortcoming: because what matters is the distribution of the *residuals,* in many cases the distribution of the *raw data* reflects the distribution of the independent variables more than anything else and basing a transformation on their histogram is incorrect.
In the case of densities that are close to lognormal (which I encounter a lot in some particular applications), I simply transform (by taking logs) and then do KDE, and then transform the KDE back (you need to remember the Jacobian when transforming the estimate back). It works quite well in that case.
I don't know what you're after. Could you expand on what you want expanded on? (NB added another example with very different $n$)
From the rest of your explanation, it seems the likelihood function is being adjusted by a fixed quantity which reduces the positive bias of small samples. The fixed quantity is effectively a function of the information which goes to zero as sample size increases, correct?
Also, misclassifying a positive as negative has a much larger cost than misclassifying negative as positive, because label tends to be large positive numbers, but sometimes it is small negative number.
My point was that the two things are connected.  The too-small nature of the SE's is only a problem because the estimate is biased.  They would be correct SE's if the estimator was unbiased.  I'd need to sit down with pencil and paper to show this formally.
Not really. Suppose you sample from N(0, 1). Then your std will be about 1 regardless of sample size. What will get smaller is the standard error of the mean, not standard deviation. p-values are based on SEM, not on std.