tttjjj/elt_bench huggingface dataset:
Another idea is to supply starting values for the optimization. Your process might converge if it started closer to the actual parameter values rather than starting from a random location.
@EngrStudent: I had in mind the simpler situation of the OP's question, where you fit one model (to the whole sample) & then estimate the predictive performance of the fitting procedure through cross-validation or bootstrap validation. You certainly can evaluate a model selection procedure in the same way, but need to make sure you include the whole selection procedure in each cross-validation fold or bootstrap iteration. So `validation(selection + fitting)`, not `selection + validation(fitting)`.
That would be Shannon's entropy estimated via plug-in method which is known to be negatively biased.
*And then you have a consistent estimator in …*
I would like to clarify a point that you made the first part of your response (before being able to evaluate it).  The equation for $\\hat{\\sigma }^2$ is expressed in terms of $\\hat{\\epsilon}^{OLS}$.  I do not believe this is correct.  In fact $\\hat{\\epsilon}^{OLS}$ should be replaced by simply $\\hat{\\epsilon}$, where the latter are in fact the GLS residuals.  This follows from examination of the preceding equations where $\\epsilon $ represents $\\epsilon^{GLS}$.  (In fact $\\epsilon^{OLS} = P\\epsilon$)
My advice would be try to formulate your problem more generally, so that people who don't know anything about soil (or that "DEM" means "digital elevation model") can help you.  For starters, you have a bunch of variables that are functions of altitudes and their topologies?  If you do PCA on this data, you'll find a set of components for which most of the variance is explained by altitude.
I notice this is the latest in a long list of questions that use correlation coefficients to assess relationships.  Using correlation coefficients in this way is at best a weak approach and at worst it is wrong and misleading.  Have you considered using more conventional analyses?
Before working backward from your answer, I see I was stuck because I didn't think (long enough) to apply $\\frac{1}{b^a}= \\frac{1}{b}\\frac{1}{b^{a-1}}$
@ahfoss makes a good point. I edited the answer to clarify what I meant there.
@whuber the sites are different locations.  The samples are citrated plasma from individuals.  The lab testing is done at the different sites at different times. The comparisons are for two assay measuring devices that are intended to do the same function.  At sites 1 and 2 some of the samples were reused but the devices operated independently at site 1 and site 2.  So that is why I say the measurement errors are really independent even though the same samples (or portions of the same samples) are used.
Wrong functional form can lead to omitted variable bias, no?
Are these differentially corrected GPS measurements of plate motion, by any chance?
If the only thing you have available to you is the sample of random numbers, this is impossible. But you can estimate them by computing the empirical mean and standard deviation.
For two matrices $A$ and $B$, $A-B$ is usually interpreted as elementwise subtraction. See http://en.wikipedia.org/wiki/Matrix_addition . But what on earth is "the MSE between to matrices"?
Homework? Hint: think of a zeroniflated model as a finite mixture of a point mass at zero (for the excess zeros) and a Poisson distribution. The probability to belong to the point mass is often modelled with a binomial GLM.
I'll second the recommendation for the Kleinman book.
@gappy, is that really the convergence statement you're interested in? A variance bound of $O(n^{-2})$ seems pretty strong. Also, what do you mean by "closed form" when you say that for the $\\Sigma = I$ case, one gets a "closed form" solution? Thanks.
Econometricians, in particular, have fallen in love with EL. If you're looking for *applications*, that literature may be one of the better places to look.
@AtillaOzgur the red point is not AUC = 0, nor AUC = 1. The AUC is defined by the ensemble of all red and black points.
My use of the phrase was inspired by Knuth's, although the reason is different and from a Bayesian perspective pretty much all optimisation is a bad thing and marginalizing is better.
Because this is a homework/self-study question, you should add the `self-study` tag to your question. Also, you should tell us what you've tried so far, and where you get stuck. Take a look at http://stats.stackexchange.com/tags/self-study/info
For some great books on regression diagnostics there is the book from the 1980s by Belsey, Kuh and Welsch (published by Wiley)  Somewhat later Dennis Cook published some regression books that emphasize graphics and regression diagnostic including his own Cook's distance.  These books tended to cover concept and not specific software.  This was way before the R revolution and some even before S became popular.
It is not appropriate to enter variables found by lasso or elastic net into a new regression as this ignores shrinkage.  And stepwise regression without shrinkage is invalid.
@Calimo I've used the `train` function in `caret` to fit the model, which lets me specify it as a glm with binomial family. I then used the `predict` function on the object generated via `train`.
There's also a course that Goeff Hinton is teaching via Coursera on Neural Networks that starts next week: https://www.coursera.org/course/neuralnets
Different methods have a stronger or weaker tendency to get approximately equal count, but I don't know of one that has it as a constraint.
With two factors, it won't return a data.frame. You'll need to post-process the result for that.
Did you look at the code? ... numObsv = length(dem2gbp);
Back to the mud flats - it seems to me that there is a problem that with only one brood on the mud flat you will not be able to separate a mudflat effect from the effect of the individual brood.  You might be able to say that brood are big eaters, but that might be due to either the brood (eg genetics) or location.  I would imagine a mixed effects model with brood as a random effect and mudflat as a fixed effect would return an error because of this.  If you had at least two mudflat broods however, the mixed effects model would be a possible means to answer your question about significance.
I could suggest "Randomization, Bootstrap and Monte Carlo Methods in Biology" by Manly. http://www.amazon.com/Randomization-Bootstrap-Methods-Biology-Statistical/dp/1584885416/ref=sr_1_1?ie=UTF8&qid=1288995733&sr=8-1
@JG, Ok, so to put a face on it lets say you have a RDD where the easting of whatever unit is placed on the X axis, and the Y axis has the crime rate. So, true, there is a theoretical discontinuity at whatever easting the time zone break occurs. What happens if easting has little to no relationship to crime? What if there are more differences to cases on either side of the line than just difference in X coordinates (say on the east side have on average more poverty)?
Is this home work? What do you mean by MC optimization? Any optimization where random number generation is involved? Would something like [Simulated Annealing](http://en.wikipedia.org/wiki/Simulated_annealing) count?
The predictors A, B, C, etc. have to mean something.  What could you sensibly be comparing?  You need to expand you question.  What do you mean by compare p-value?  Do you mean just make an evaluative judgment on the similarity of findings based on the analysis?
I think there may be some typos here as well as some ambiguous terminology: (1) you use "dimension" in two different senses within the very same statement (#1).  (2) In statement 3 it looks like you want to estimate $f^{-1}$, not $f$.  As far as properties of $f$ go, is there any possibility that $f(s)$ can be *incorrectly* evaluated?  If not, this is not a statistical problem and it has a mathematical answer that is trivial to obtain.  If so, what can you tell us about the nature of these errors?
When one standard deviation is zero, the correlation is not defined.
Interesting!  It really seemed like I was doing it correctly, but the results actually look more clumped together in the result, so I was confused.  Thanks for verifying
Thanks Michael. I think for me (and I am sure it is just my lack of knowledge) the remaining difficulty is that it seems we are using the variance of the independent Bernoulli r.v. (the Y) for an estimate of the average error from the model. It seems this would be the same as using the variance of the Y when Y is truly continuous - which we said is NOT the same as the root MSE - which is the parameter called for by the procedure. Do you see what I am struggling with?
Michael, that's another question. You have the functional form of $\\hat p$. Look up how we derive the distribution of a function of a _discrete_ random variable. And if you cannot solve this, post the question.
(+1) Thx for the R-code.
One of the assumptions is that the two samples are mutually independent. Given your description, it may be more appropriate to perform a one-sample KS test of the sample versus the parent population.
I'd have been inclined to use the tag `hypothesis-testing` rather than one of `distributions` or `empirical` here. No great matter though.
Question edited :)
If you are classifying into 1-of-C mutually exclusive classes, then you shouldn't be using tanh or logistic sigmoid output, you should be using a [softmax](http://en.wikipedia.org/wiki/Softmax_activation_function#Artificial_neural_networks) and the log loss function.
http://en.wikipedia.org/wiki/F-test_of_equality_of_variances
Gaussians can produce such strange results, Andy: because they and their derivatives are zero at the origin, they impose extreme smoothness on the kriged estimates. In the bottom picture, fitting a smooth spline (as a proxy for the kriging) clearly would extrapolate high values in the 150-250 range. Such extrapolation requires weights beyond the 0-1 range. The extremely low value you have plotted at 500 similarly looks consistent with the data and variogram.  Why not experiment first with an exponential variogram (and add a tiny nugget to it)?
(Please be gentile, I am aware of my need to spend more time on the basics :-P )
@Macro The answer really tells what is assumed. "A total of 7 patients will enter this two-treatment parallel-design study. The probability is 80 percent that the study will detect a treatment difference at a two-sided 0.05 significance level, if the true difference between treatments is 2.661 times the standard deviation." The effect size is assumed to be 2.661. The mean difference divided by its standard deviation for a two-sided alpha =0.05 level test and a sample size of 7. The computed power for that effect size is 0.80.
+1 Impressive performance for a mindless, automatic tool!
@PaulHurleyuk: +1 The quote was “We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet.” by a SAS marketing director in [this New York Times article on R](http://www.nytimes.com/2009/01/07/technology/business-computing/07program.html/?pagewanted=all). The SAS representative [clarified her remarks in a later blog post](http://blogs.sas.com/content/sascom/2009/01/09/this-post-is-rated-r/).
There is no normality assumption in PCA so log-transforming and standardizing your data for is not a prerequisite to apply PCA. That being said if your data actually makes sense to be in a logged scale then you should do it in order to get better insights about your modes of variation.
You can treat it as an approximate growth rate.
I can only say "yes" so many times, and in so many ways. After that, there's nothing to add.
I may have phrased my question poorly, sorry. I'm going to give it some more thought.
thankyou, no the proof is not from a book (although there is nothing new under the sun so no doubt it's in a book or a solution to an exercise somewhere). It's a simpler version of a technique I used to find the mean in [this answer](http://stats.stackexchange.com/questions/56134/a-serious-in-depth-problem-of-probabilities-for-flipping-coins/90591#90591).
Note to others: I was going to mark this as a duplicate, but it seems to me to ask more than one question, each of which is duplicate.
Factor	F	df	p
Gender	2.46	1	.24
Updates	6.38	3	.008
Gender X Updates	5.29	3	.01
Error		157
I agree with @MånsT's comment - as you know, many of the questioners on this site are not very experienced with statistics and the concept of the difference between a parameter value and an estimator (and the idea that an estimator is a random variable with a distribution) can be a subtle issue for a beginner. Also, the quote from the OP **"The problem that I have is that I can't get fmincon to evaluate the correct parameter...it always give me my upper bound (bound set prior to estimation) of my parameter."** makes me think this may actually be a problem using the software.
+1 (from days ago). I really hope you end up w/ more than 6 upvotes for this; it's a really great overview of how CCA works.
I'm looking into the RSS examinations myself, specifically the 5 exams needed for the Graduate Diploma. What's a bit off putting is the fact that the study books suggestions are pretty broad. Do you have any experience with the exams to share a year and a half after you've asked this question? Thanks.
the wikipedia page is a bit strange. The condition for $I(0)$ presented is satisfied for all processes which admit MA representation as in MA link. For all practical purposes $I(0)$ is synonim for stationarity.
I do not think it is possible to specify a model without a fixed effect with lmer because the lme4 package is dedicated to mixed models only (with at least one fixed effect and one random effect). I do not remember seeing random effects models in the documentation, in any case. It could be useful to ask for it in the R-sig-mixed-models list (https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models)
Another problem with the original formulation is that it does not protect against over-fitting - one could make the training error to zero just by setting _K_ to a high enough number..
Because any nondecreasing function from $[0, \\infty) \\to [0, 1]$ will do the trick, you have a lot of flexibility.  But some methods will be better than others, depending on the application. What is your purpose in seeking such a re-expression?
The question here is whether all mouse are homogeneous before your experiment. If there is a significant variation in the untouched cholesterol metabolism or in your drug metabolism, it can spoil the paired test assumptions -- you've could be just lucky and assign mostly "fitter" or "more responsive" mice to be treated. You can probably investigate it by looking at the variation over the control mice, but the unpaired test is way safer.
Do you have any thoughts about what kind of relationships might exist? Do you think these are a function of something? Eg, do you think var1 might be caused by var2, or some factor outside of these 4 vars? Do you think they might all serve as measures of the same underlying construct? One thing to bear in mind is that since they are counts, they should be distributed as Poisson; so, eg, taking logs may be appropriate.
@MichaelChernick Often the "expert" knowledge is biased towards what has already been studied (think how many article have been written about the [p53 gene](http://www.ncbi.nlm.nih.gov/gene?db=gene&report=generif&term=7157)). Even experimental studies would suffer from this bias (it's easier to find antibodies to well characterized proteins, etc.) Taking a more data-driven approach might be the ticket in this particular instance!
You can use $\\LaTeX$ for mathematics on this site.
I just noticed that this question was asked 2 months ago! How did it get bumped up to the top of the list?
This is why multinomial logit functions are classically used to estimate spatial discrete choice problems, even though the actual phenomenon is better modelled by a probit.
Before you try to find the 'appropriate' test: how are you going to present the data graphically?
@whuber's question is key. "Heterogeneity" does not have a predefined meaning and you need to say more precisely what you want and why what you have tried disappoints. Concentrations are bounded but much depends on whether you are talking about very small concentrations only. Sometimes concentrations are best handled as is, sometimes on a log scale, sometimes on a logit scale; in all cases detection limits and/or reported zeros can raise problems. But without further elaboration of the question more helpful replies are difficult.
@Nick That was a great question to ask. A lot of people tend to give up on this sort of question much too easily, while your response covers a case that often crops up, and for which there is a practical solution.
Please explain why you think these distributions must be negative binomial and how one is supposed to quantify whether a "specific combination" is "reasonable".  Exactly which "well known problem" are you referring to?
and the central limit theorem.
Conditional on the number of classes chosen, you can estimate the posterior class membership distribution for each individual; this is typically standard output from software that fits such models. If there are large differences between the classes (with, presumably there are, if you've chosen the number of classes in a principled way, e.g. AIC) then most of the individual posterior distributions will be highly concentrated on one class, giving a basis for class assignment.
yes, Spearman's rho does not apply to this setting, only to paired observations. I cannot but notice this coincides with <a href="http://xianblog.wordpress.com/2012/01/18/correlations-between-the-physical-and-social-sciences/">a book review I just wrote</a> where the author uses Spearman's rho in the same inappropriate way.
@ Stéphane: This statement is true practically by definition. :) As I said I do see the valid point you are making but I still think that *my* meaning of the word is the correct one! :D
Please *don't* cross post questions to multiple SE sites; this is explained in the [help](http://stats.stackexchange.com/help/on-topic) (3rd-last paragraph). Since you posted there first, you should direct all your questions there. Please also take the trouble to read all the help.
And so if the mean is the same as the variance, could you conclude that the data was Poisson? Hardly!
But since in the two models we generally impose that the variance is the same, why can't we compare the results of the two models?
You could improve clarity in your question. Note that A (good) classification rule may make classification errors in the learning set, this is sometime a clue to avoide overfitting.
You said that you estimated the parameters from the data, so there is uncertainty about the parameters (you could fit a CI to show this).  If you only generate data at the values you estimated then it does not take into accont this uncertainty.  You can see this by increasing the number of data points you generate, as the number increases your interval estimate will converge towards having 0 width around the value if you knew the exact parameters.  To get a realistic estimate you need to include uncertainty about the parameters, bootstrapping to estimate the parameters is one way to do this.
I'm not sure to understand the link between skewness (which is expected for this kind of data) and the use of ordered response categories, as implied by your "therefore". Of course these are ordinal data. Besides, you also speak of using item scores and the resulting sum score, for each individual. Could you explain the purpose of this analysis? This might well make this question a statistical question and not a coding issue. @Michael Your comment is not really encouraging, IMO.
Can you post the picture now?
Did you try calculating the orthogonal projection?
Hmmm... "they tend to work well..." is ambiguous. In simple situations w/ large clear effects they tend to select the correct features (outside of that, not necessarily). OTOH, estimates are biased, CI's too narrow, p-values meaningless, etc, so they don't work well for those issues. Nonetheless +1
Just to add, I am interested in analyzing interval data variable in general, not just as a response variable..
You may want to browse some of our threads related to [tag:likert] scales, or [How to transform ordinal data from questionnaire into proper interval data?](http://stats.stackexchange.com/q/27927/930)
yeah i read the formula is an approximation. So would the simulation be more accurate then?
Many thanks! I did perform the same thing with ANOVA as well with   `result = aov(x ~ factor(group))  
  pvalue[i] = anova(result)[1,5]}  
 length(pvalue[pvalue<=0.05])/100000`   (the part above is the same)   and got [1] 0.0461 as an answer. Thus, there seems to be something wrong with my ANOVA code, I'll check.
OUUUHHH, this was obvious!!!
plus consider that some public holidays move - easter, chinese new year, Ramadan, school holidays, etc.  Easter is particularly annoying in my own version of your problem because it isn't even in the same *quarter* each year.
It depends on what formula / assumptions it's using to test $r$ as to whether they'll be consistent.
@AndyW To be blunt, no. An interesting discussion on why the name is flawed, and an article that is lurking behind a subscription wall.
The answer I supplied to your previous question suggests a version of the "c-chart" that is more rigorously justified, quite simple, and accordingly is easy to interpret. (It also raises the logically earlier question of whether control charting is an appropriate technique for you.) You did not respond to this, but now you re-raise the question, so I am wondering why: why do you think this solution will not work for you?
I guess you mean "multiple" or "multivariable" instead of "multivariate". Such inconsistencies might e.g. be caused by small sample sizes, overfitting or "wrong" model structure.
The product is ticket sales, so it's essentially a monopoly. There is also a ceiling on how much can be sold. The issue of course is that in the summer months and on weekends the demand shifts upward dramatically. Because the owner knew this, he would move the price up in accordance with the shift in demand, and the tickets would still sell out much like they would on less attractive days with lower prices.
Recalling that in econometrics multivariate linear regressions are called seemingly unrelated regression models, Google Scholar searches immediately become bounteous.
This is really helpful! I was thinking the solution would look something like this... what I could not figure out however was the distribution f(t|n=1). I still don't really understand how you arrived at that. Could explain that bit a little more.
I find your second paragraph unclear -- could you explain how something akin to "ANOVA" *could* be okay, yet a binomial GLM would not? What exactly is the dependence you refer to?
What are you looking for in terms of "any relationship"? Someone liberally interpreting this question would point out, for example, that your recent related question provides just such a relationship, namely the joint $R^2$ is at least as large as the squared correlations between any of the individual predictors.
What's the problem with using a weighted average? (Or am I simplifying too much?)
@chl: interesting -- reminds me that Gelman had a discussion of finite/super population inference being comparable to fixed-/random-effects in his paper on ANOVA [ http://www.stat.columbia.edu/~gelman/research/published/econanova3.pdf ].
P.S. The need for a very high number of iterations to achieve satisfactory precision in the measure of predictive accuracy makes the bootstrap even more attractive.  It can achieve the same precision with fewer iterations, and it properly penalizes for variable selection (if any selection was done) assuming the analyst does the right thing and restarts the feature selection fresh at each iteration for any of these approaches.