tttjjj/elt_bench huggingface dataset:
The samples are expensive to obtain so I can't just up the sample size until the bias goes away.  Unbiasedness is desired because I'm using the result of the estimator as the $Y$s in linear regression.  Having bias means that $Y$ will contain non-normal disturbance which is equivalent to a specification error and that leads to a mess.  I won't be able to accurately estimate the slope, the variance, construct confidence intervals, etc..
@whuber I am planning to run the regression: _spread = Maturity + IndependentVar1 + IndependentVar2 + .. + Issuer2(Dummy) + Issuer3(Dummy) +..._ Thanks
If you have control over the software, why not modify it to perform a numerical calculation of the Hessian with respect to $C$ rather than w.r.t. $L$? If you cannot modify it but you can use it to compute the log likelihood at arbitrary values of $L$, you can still numerically estimate the Hessian.  This approach might be more accurate than other methods, such as the [delta method](http://en.wikipedia.org/wiki/Delta_method#Note).
can u give more information on the GS story?
(not sure why i cant comment directly on your comment)
My control systems engineer self smells a transfer function. :)
Given you're not getting the actual age, I'm not sure what "he is telling 100% truth" means. If you see "23", what's the probability that it was really 22, that it was really 23, that it was really 24? (and on what basis do you arrive at those probabilities?)
How does this graph show "A", "B", etc.?
It would be helpful to have some context: where do the data come from?  Why do you want to fit a distribution?  To me, it's the 11, 44, and 63, rather than the 0's, that are unusual.
@DmitryLaptev; I'm not sure what you mean?
I would recommend to take a look at [|STAT](http://hcibib.org/perlman/stat/). That's a pretty old software, yet it is very convenient for such things. There's also [pyp](https://code.google.com/p/pyp/), and [several](http://www.gregreda.com/2013/07/15/unix-commands-for-data-science/) [other](http://www.drbunsen.org/explorations-in-unix/) Un*x tools.
I think so, although obviously it would be more complicated.
It seems to hard to answer without knowing a broader context of the text you refer to.
I ended up going with a negative binomial, which I was leaning toward at the beginning.  I appreciate your post, and I'll give you  the check since I ended up using a NB.
I have read the book and made similiar counterarguments such as yours and Dikran's.  Taleb seemed very naive.  There was a session involving him at the JSM a few years ago.  I think it was in Washington.  The second edition came out after that and is a little more reasonable.  Taleb has some interesting things to say about specific "Black Swans" and he knows a lot about economics.  I think it is worth reading and the second edition is better.
+1 for the night club and demonstration that multiplicative noise can be approximated by additive.
`prob.trans.t1` is a stochastic matrix, so I believe your question is how do I calculate confidence intervals for x/sum(x)? We can treat these as multinomial proprtions. [This](http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.html) may help.
If I understand correctly, you want to simulate a stochastic trend. Why not initialize $Y_0$ to a random value, say a sample from $N(0,\\sigma^2)$ with $\\sigma^2$ large?
Perhaps if you could _define_ what $N$ and $n$ mean? Perhaps saying what values the samples $y_i$ can take on and what the underlying probability model is (choose randomly (without replacement) from the set $\\{0,1,2, \\ldots,N\\}$? or $\\{1,2,\\ldots, N\\}$?)
(1) did you have the variables "the wrong way around"? (2) With discrete variables only taking a few values, consider doing a table rather than a plot, to start with; (3) If you do do a plot, consider jittering both variables, rather than only one; (4) can you tell us something about the two variables? Why does 'ageveh' only take the values 0 and 1? (5) *Why are you worried*??
The general answer to "how do I compute power for a complex situation"? Is "simulate data sets that are reasonably realistic for your situation, analyze them in the way that you intend to analyze the real data, and summarize the fraction of times that you are satisfied with the outcome (significant p-value, standard error or CV of coefficient estimate below a specified thresholds, etc.)"
Thanks for the answer. Is it OK to just write $\\hat\\mu_{\\hat f(X)}=...$ rather than $E\\left[\\hat\\mu_{\\hat f(X)}\\right]=...$
Interesting solution, but what I really want to be able to see how is the "change" on a per user basis. I want to see the fluctuations in activity for individual users. That's why I chose a line initially, but the visualization is just too cluttered now.
I would show the results of the analysis you ran. If you run & report the one-way, followed by pairwise tests, I would show that.
This is probably the point, thanks a lot!!!
Could you give the full reference of Flurry 1988?
I have come to realize that our conversation derives from two different understandings of "population" and "sample" variance. The terms are confused on the Internet, so I think we're both blameless and that we may be in agreement. I was attempting to use the convention adopted by the OP, whose terminology indicates the "sample" variance divides by n-1 while the "population" variance divides by n.  Incidentally, your comment seems (correctly) to allow for *three* variances: that of a random variable, the unbiased estimator thereof, and that of a set of independent realizations of it.
Beautiful analysis; the use of mixed second partial derivatives is somewhat reminiscent of criteria to check whether functions are "nomogramable", at least of particular forms (which also involves a similar separation).
This question was already answered here http://stats.stackexchange.com/q/23938/3277 and here http://stats.stackexchange.com/q/73065/3277.
+1 for the simulated-data idea.
Could you please be a little more explicit about how K-means solves this problem?
Could you please detail or reference the relation between MAD and the two medians?
I am mixing terms rather uncarefully!  Yes, the confidence that the coin is 2-headed is along the intent of the question. ------     

I am unsure how to formulate confidence as an event, like calling it B.  That probably means it is not an event and I have misunderstood something.-----


In the case of a Bayesian prior, I am a bit short on formalism, but I think of the prior knowledge as no knowledge, so it would have infinite variance.  That means the variance in the posterior would be the same as the variance in the data, which is a particular symmetry that I have an intuition for. The key?
Then look at `?nls`.
Are you sure the cluster labels correspond to each other? I.e. that cluster A is cluster A in both custerings? And do you have equal number of clusters on the first place?
It's certainly *possible*; the questions are more 'how' and 'does it have the properties you want'? Likely you'll want some linear combination, and the obvious thing is to have a set of positive weights (or instead, weights corresponding to the 'direction' the question works in, if they're not all going the same way). The question comes down to how to give the last question 'enough' weight relative to the Likert items, but not have it unduly dominating. The other thing to note is you probably won't end up with a good approximation of a normal distribution for your score (may not be a problem)
DVs do not have to be normally distributed (and almost never are), but for some tests it is helpful if the *residuals* do not depart too far from normality.  For this purpose it is best to *plot* residuals rather than conduct formal testing (such as KS), because the plots reveal much more and the tests tend to be *too* powerful.  So, just to be clear: are you referring to the actual DVs or to their residuals in your characterizations of normality (or lack thereof)?
Please tell us about the sample size and the type of scale used. Were there a priori sample size estimates? Can you inform us about the data distribution?
Good luck then :). Note that this method works for ANY distribution ;)
Hi, I meant the STATA command "xtoverid" -> test of overidentifying restricitons / Sargan test
@Matt (+1) Nice response. I also came across this blog which provides illustrated examples of R/SAS: http://sas-and-r.blogspot.com/.
@SQLServerSteve: Even if you cannot upvote yet, you can already "accept" answers (by clicking on a green tick on the left) once you think your question is resolved.
I suspect your questions might be purely mathematical ones, so give some consideration as to the correct place to post them: here or on the math site.  (The main computational issue concerns diagonalizing $\\mathbb{Q}$, which is a purely linear algebraic problem.)
@mugen I did not answer the exercise the OP has to solve. I concentrated on the OP's mistaken attempts. Writing analytically the joint support and the associated probabilities, obviously becomes unmanageable when the joint events are many. Here, due to the multiplicative nature of $X$ and the characteristics of its components, one can easily obtain $P(X=0)$. But is this the complete answer to what the exercise asks?
@andresmh, you might try reducing your graph to have a single observation per directed pair first. For the eigenvalue stuff, your data is likely similar or equivalent to a weighted random walk on the graph. I'm not sure if SNAP supports that, but it's likely to. If all else fails, you might send a very specific email to Jure. He's a very nice guy, so I wouldn't be surprised if he provided some quick guidance.
Sorry. Didn't notice. Look at the edited answer. There is no rule of thumb but you can test for simultaneity using the Hausman test!
Is this the only way to go about determining a distribution from a statistic? If I try to perform,
$$ f_{Y}(y) = f_{X}(g^{-1}(y)) \\bigl\\vert \\dfrac{d}{dy}g^{-1}(y) \\bigr\\vert, $$
any statistic $ Y = g(X) $ of the data, $ X $, involving the Spearman rank correlation test will not be one-to-one. Assuming I had a statistic, $ g(X) $, that was invertible, the underlying data, $ X $, is image data that doesn't follow any simple distributions. (I guess I could model it as a complicated mixture of Gaussians, right?)

Is there any other way to describe a distribution of an arbitrary statistic?
Welcome to the site! I added some tags to make your question easier to navigate to and from. Treatment effect estimation is far from being a basic concept, though.
Perfect! Thank you very much.
+1 for harmonic oscillator.
Hi Patrick,

Thanks for the answer!  I tried the first approach but got the following message:

Error in file(file, ifelse(append, "a", "w")) :
  cannot open the connection
In addition: Warning message:
In file(file, ifelse(append, "a", "w")) :
  cannot open file 'imp1.csv': Permission denied

I then tried the alternative approach.  There was no error message, but I couldn’t find the saved files.  I assumed the files are saved under the names “imp1.txt”, “imp2.txt”, etc., and used Windows Explorer to search for them, but nothing turned up.  Did I miss anything?

Thanks!

DLuo
By "input one single document," do you mean "use a training set consisting of a single document"? Or, "extract topics from a single, out-of-sample document using a trained model"?
Why wouldn't position change as a function of time? It does.
just to clarify, since you put regression tag, are you asking how to chose between multiplicative or additive error models? If your question is from signal theory please change the tag.
@Tim: Added in some more to the answer.  I think the optimality condition in your decision theoretic framework is too strong to be useful... but there are other decision theoretic frameworks that would be useful for constructing estimators.  Aren't there examples in the Bickel & Doksum book?
How does one even apply the $\\chi^2$ test to smoothed data?  Regardless, the smoothing is going to change the sampling distribution and introduce *dependencies,* thereby ruining the $\\chi^2$ test.
This is such a nice summary!  I am grateful for your effort in sharing it with us.  I can't help upvoting it despite your appeals not to :-).  (I don't think it will change your rep, anyway, and there is a lot of merit to having it appear close to the original question if it does get substantially upvoted.)
Ah, yes you are right. I was using 95% confidence earlier but forgot to change 1.96 to 1.645 for 90%. Is the approach I'm using correct though?
actually im not so sure about the variables because the answer upon a second glance is actually kinda messy, logically. but its ok because the main point was that it wasnt helpful in any event
@whuber In this case, the question should be tagged *Analysis*, not *Probability*. There's no intention behind my reply, what do you mean ? I originally am probabilist and this was my natural reaction. Moreover the only thing I "change" is that I consider the equality at the end of the question as an almost sure equality. **Anyway I don't know the definition of the user**, so it was clumsy of me to write *I guess your definition..*.
For each replicate (x1,y1), I have 1000 rows for x1 and y1. There are 5 such replicates (x1,x2,x3,x4,x5) and so the fitted values include 5000 rows. For stacking, I am just combining the x and y values using, y = c(y1, y2, y3, y4, y5)
x = c(x1, x2, x3, x4, x5). Note: x values in x1..x5 arrays are the same; The y values vary a little across replicates. nls estimates one value of A, c and m but the fitted file contains 5000 rows. The output plot has 1 exponential and 4 linear curves. However, upon inspection, I noticed that the fitted values for the same for replicates. Thanks for your help.
(ctd) ... (which the scale your combination is on anyway) and treat it as asymptotically normal, like a Wald test. (You should be able to back out an interval from that approach.) If you use R, one way to do the Wald test is with the `linear.hypothesis()` function in the `car` package (which aslo suggests Fox's *Companion to Applied Regression* as a reference); you can do the LR by fitting the two models and use the `anova` function.
You have done a great job mastering our markup and illustration capabilities in a short time: this answer is a nice example of that.  Well done!
If you have the CTABLES license you may be able to do this test through that command. I can't find anything in the native descriptives/crosstabs that would allow you to do this. You could always calculate it yourself as well.
(i) It should be possible to automate such a process in some fashion, to reduce the risk (though such checks are always good). (ii) You'd see the serial correlation change if you swapped two different numbers - equivalently the product of the sequence and its first lag would change.
@whuber Thank you for editting
"There would be an upper bound of 18.3%" <- One woman could be raped by multiple (possibly all) men, so the rapist ratio might be anything from 1/(male population) to 100%.
Also, one of the key relaxed assumptions of GEE is that of possible heteroscedasticity. Plugin estimates of residual variance are used to compute standard errors for linear models. So post-hoc tests of specific contrasts cannot necessarily be computed because you have to re-introduce assumptions about the distribution of errors around predicted values.
I don't know R well enough to answer the R specific questions. What I don't know that an R expert can tell you is whether or not the confidence curves and prediction curves are connecting the individual confidence intervals or are generating the simultaneous curves.  Also does your code do what you intend it to do?
+1, great answer as usual. You should nominate yourself as a moderator, there is still 1 hour left :)
+1 I believe the point to gridding is to sample irregular regions with (slightly) higher computational efficiency, assuming an efficient data structure for locating grid cells within the region.  (The alternative is to sample the region's bounding box and reject samples outside the box.  That's usually ok in 2D but in higher dimensions tends to get bad.  E.g., a spherical region occupies an exponentially vanishing portion of the hypervolume of its bounding box as the dimension grows.)
@Momo thanks for the detailed explanation.  I do understand I have a model which requires transformation.  This is a regression model with multiple predictors (some factors, others continuous predictors), so taking exp(coef()) just gives me a list.  Would I just multiply those all together?  Wish this distributuon was covered more thoroughly in one of my courses, but we (and our book) stopped with the Poisson and Binomial...  So, I've been left scouring the internet for info.  For the p in qnbinom(), what is supposed to be used?  I don't understand what probabilities are needed.  Thanks!
you should give more data, like is it numeric?
In lasso/ridge $\\lambda$ governs the variance of the prior distribution on $\\beta$. For fixed $\\lambda$ you hone in on the possible distribution for $\\beta$. Each CV train set for fixed tuning parameter yields a point estimate of that $\\beta(\\lambda)$. By the end of the CV procedure you have an estimate of test error given that prior. You then vary the tuning parameters selecting the one with best test error properties.
@Ari, the second level regression is a regression for regression coefficients of first level regression. The first level regression tries to explain variation within group, whereas the second level regression tries to explain variation across groups. This division is artificial, but I like it since it feels natural for me at least. This type of division is also used in hierarchical Bayes models.
@whuber thanks for your comment. I can well imagine that models that include interactions can be a lot more faithful. That said, what I'm after is something really simple. Thing is, this analysis is really not central to my article, it's really just a remark. So would you say that even if my model is not very accurate, my text does at least describe a sound application of the principles?
Also consider a regression spline on the untransformed data.
Thanks for your help . It is a retrospective study and BMI above 30 is actually suppose to be a risk for gestational diabetes. I have my data as an excel spreadsheet and have imported it to Spss. I am trying to control BMI and age . How would you do it in spss. Since i am new to this software. It is dichotomous variables C sec or natural labour. I have calculated odds ratio and just now exploring the adjusted odds ratio if someone could direct me on what tests to run in spss for multiple linear regression. Thanks to everyone
I understand what you mean. I just want to ensure there is no other significant factor that is influencing my results in order to substantiate my findings.
Jim writes very well and may help with concepts. He also teaches statistics at Bowling Green University in Ohio. I know he teaches the elementary service courses and uses sports examples to motivate the methodology. I think that he has some elementary statistics books that he has written that may also be helpful and he does a lot of implimentation of methods using R.
Could you elaborate? Or could you give me some pointers?
Thankyou for the replies both of you. The original variables are gene expression data so cannot be zero. A previous paper used the ratio of two gene expressions as one of their key independent variables so I really need to have a look at the ratio at least.
Cube root and arctan both seem to help with the distribution so thankyou very much there. (And for the advice about software with cube roots - SPSS did indeed refuse to cube root negative values).

The ratio variable will be used in comparisons of distributions between multiple groups and possibly within a multivariate logistic regression.
Thanks, it helps me a little bit but it is not totally clear yet. I'm going to read and ask again in case of problem :)
@mpiktas, i usually explain the white noise to students through the spectral density concept, at least it gives light to (through the analogy with white color) why the noise is "white", and why the $AR(1)$ process could be could called "red" and there is no "black noise" :)
Welcome to the site, @user44285. I took the liberty of editing your post & trying to use the site's $\\LaTeX$ markup features to make your equation more readable. Please ensure it still says what you want it to.
If the MATLAB looks wonky, I apologize; I haven't touched MATLAB in years... XD
Thanks! I sure did check. I only found [one directly relevant question](http://stats.stackexchange.com/questions/14691/kendalls-w-for-multiple-objects-and-multiple-surveys), but it has received no answers.
The problem is that nobody has really studied C5.0 so far (it has been in the private domain until a year or so ago).
The absolute value of a sum is **not**, in general, the same as the sum of absolute values!  Neither square, square root, nor absolute functions are linear, which is why the sum after applying the function is different from applying the function after taking the sum.
Note that regularisation isn't a "fix all", especially the typically use L1 and L2 penalties.   These suffer in "sparse" scenarios, where say 5-10 variables out of 100 are non-zero.
@whuber "...what the MO folks might think..."?  MathOverflow?  :)  I, for one, do not lose sleep over what the MathOverflow folks may think.  Down, down with MathOverflow!  :)
Seems like the area I'm interested in is Predictive Analytics.
For the strict form of the CLT, you do need i.i.d. data. However, there are weaker forms, i.e., you can get asymptotic normality under weaker conditions. If you want a formal treatment, see work on robust statistics by Huber, starting from his classic 1967 paper; or more generally on M-estimation.
This is unlikely to normalise a variable that is residual-like and both positive and negative as it doesn't act symmetrically.
@StéphaneLaurent, I think the model is equivalent, if you have a working code it would be highly appreciated by me and probably by other users as well. Can you post it here ? Alternatively, if you do not want it to be publicized, you can send me privately and I will surely acknowledge it in the manuscript.
Can you provide some references where the expression comes from? Are you doing Fischer LDA, or Canonical LDA? f and g cannot just be "some" scoring functions, otherwise the expression wouldn't be a valid probability. So please state the prior, the likelihood, as well as what f(x) is, or provide some references where the term occurs.
@amoeba Ah yes, I see what you mean. The resulting model will have the same number of terms (dummy variables, x1, x2 etc) and so the same p value, you are right. The interpretation of the individual model terms (x1, x2...) will be different of course, e.g. none of them represent the interaction between factors.
This probably has something to do with the fact that the variance of a $\\chi^{2}_{k}$ random variable is $2k$ and also with the fact that the statistic must be multiplied by 2 to have the correct distribution (as in the likelihood ratio test). Perhaps someone knows about this more formally.
-1 for deducing from $P(A) = 0.2$, $P(B) = 0.3$, and the claim of a divine spirit that $P(A\\cup B) =0.6$ that $P(A\\cap B)=0.1$ instead of questioning said allegedly heavenly claim, and for not bothering to correct the error even when it has been pointed out.
Great, this makes sense. Thank you both for your effort and help.
Well, the lead concentrations cannot be negative, right? But a normal distribution with mean $0.033$ and SD $0.10$ takes on values from $-\\infty$ to $\\infty$. The area of the normal distribution that is below $0$ is around $0.37$ (it would be a good idea to draw the normal distribution). So it only could be a truncated normal distribution.