tttjjj/elt_bench huggingface dataset:
@AdamO I am not trying to explain the proportion with some variables, I am fitting prediction data on observation data. So I do not see how I can use a mixed effect model for that ... but I could be wrong.
I saw [this thread](http://stats.stackexchange.com/questions/9276/finding-precision-of-monte-carlo-simulation-estimate) already.  It doesn't answer my question.
Great, thanks again!  C-log-log is actually part of the ordinal regression package in SPSS... you can choose this link function along with several others.  So I imagine it must be compatible.  To your second point, I've coded my factors numerically but entered them into the model as nominal variables.  I suppose there's no theoretical reason why I couldn't treat them as continuous.  Yet they only have 5 levels, with the bottom two having very few observations, so I though it better to make them categorical.  It sounds like only continuous factors allow for the "borrowing" you speak of, no?
@PeterFlom I've cleared the question up a bit and tried to clarify my objective.
Coefficients:
(Intercept)      c(1:80)  
   0.044427    -0.001097. I have plotted TotalScore against the residuals and can see that the residuals increase linearly with Total Score, could the cause? if so what would you recommend doing about it?
What is unclear, atamaths?  The answer is correct and helpful (in steering you away from looking for integral solutions) and it even references freely available software (`uniroot`).  If you need more information than that, then please specify what is lacking.
thanks alto... this was really helpful and for the first time someone clarified my doubts so nicely regarding mormalisation process...

I have another small query.. the tf-idf method which you mentioned in the link.. can be used for any data where we are considering count of certain patterns or just for text ?
Have you tried L1 regularized regression with transformations of any continuous variables?
@Majte: Yes, I have $\\Omega$ which is finite and I can construct $X_n$ depends on the observation namely by $Y_1,\\dot,Y_n$ such that $\\{X_n\\}\\to X$. We know that $var X_n$ can go to infinity, this is my problem.
By the way, what do you mean with $\\Omega$ is unbiased? What I know is the unbiasedness in the estimator. Thanks
I mean standardizing the series to norm(0,1) by taking logs where necessary and then apply (x[i]-mean(x))/sd(x) to each value. Then proceed with regression. The coefficients at the linear parameter will be standardized in a way that it would allow for comparison with other slopes.
A more important consideration might be how to scale each variable.   Even if all the variables were continuous, I wouldn't necessarily normalize them all the same way -- if the association with the response variable is stronger for x1 than for x2, I'd want to keep the variance on x1 higher than for x2.  For example, scale x1 as normal with mean 0 and variance 4, whereas x2 gets variance 1.
As far as edits you can go and see the proposed at the link I provided, so they are not lost. The `CONTROLVAR`'s may not be significant in the model because of a high variance, so I wouldn't drop them from the predictions - but I understand your code snippet can not see them.
I am struggling to follow your point. There is certainly purely theoretical literature on the Weibull, as on any other probability distribution. Whether that literature talks about Weibull models or Weibull distributions is probably more a question of style and context than of anything fundamental.
Nice answer, +1. I especially like the caveat about whether the true model is actually present in the group evaluated. I would argue that "the true model" is _never_ present. (Box & Draper said  that "all models are false, but some are useful", and Burnham & Anderson call this "tapering effect sizes".) Which is why I am unimpressed by the BIC's convergence under unrealistic assumptions and more by the AIC's aiming at the best approximation among the models we actually look at.
I wrote a method for doing predictions using a LoopyBP object [here](http://pastebin.com/e0Bq9fNR). I think ideally we don't have to copy the LoopyBP clustergraph, and instead just modify the message sending given evidence when calculating a marginal, but I haven't diven into the code enough to implement that yet. This function resolve the NA problem.
@doug.numbers Thanks, that's very kind!
Many thanks! I wrote it like this... But to my surprise that almost all the parameters are insignificant... But in SAS it says that they are significant... So I doubt if there is any error in my programming words....
Of possible interest: [Workflow for using R with LyX for a statistical analysis?](http://stats.stackexchange.com/q/15744/930), [Workflow for statistical analysis and report writing](http://stackoverflow.com/q/1429907/420055) (linked to [here](http://stats.stackexchange.com/q/2910/930)).
I'm following broadie and kaya: http://finmath.stanford.edu/seminars/documents/Broadie.pdf I'm basically using a euler discretization outlined in section 2.2. however I estimate the parameters with a differential evolution algorithm (by price & storn), using a chi squared goodness of fit test.
Ok, thanks. Still, do you know how i could access the tree structure so I can modify values to have proportions instead ? An other problem I have is that for a tree with of max_depth=5, values are plot only for the leafs at the bottom of the tree.
Hmm, thought I replied to this but I guess not. I can't see how having Fv1 and Fv2 could yield a useful result. Is it to establish a baseline difference between the two videos. What sort of analysis models could I use on this?
It is erroneous to assume the population variance would be less biased than the sample variance, because the bias in the former (for samples with replacement) is of order $1/n$ for *sample* sizes of $n$ whereas the difference between sampling with and without replacement introduces errors only of size $1/(N-n)$ where $N$ is the *population* size which ordinarily is hugely greater than $n$. Moreover, in the case of studying drug effects there is no definite value of $N$ because we wish to extend inference even to future populations, so $N$ is effectively infinite.
It is from my Thesis.  (Mechanical Engineering)  I put a Gaussian blur on the position of a nonlinear pendulum, and then used the convolution method to back out the position from an image and compare it to the analytic.  I tried it with several levels of noise.  As long as the noise was below a threshold related to the size of the gaussian, there was very good reconstruction.

[link](http://repository.asu.edu/items/15110)

Figure 11 was the relevant graph for the reconstruction.  Figure 6 and equation 2 indicate response to noise.
Are you aware that there are [ASTM standards](http://www.astm.org/Standards/D5922.htm) for a geostatistical analysis?  And that they specify particular ways to choose estimation neighborhoods (including jackknifing and other forms of cross-validation)?  Choosing estimation neighborhoods is a standard part of kriging that is described in *any* textbook on the subject.
"residuals" is the normal English term, for future reference.  Has the comment by @whuber on the two different uses of "i" cleared up the question?
+1, although the answer might benefit with suggestions on computation strategies for modeling data in this fashion. I realize it's not in the OP's question, but it would still be helpful.
How do you define $\\alpha$? (BTW, you can register here on CV and then associate your account with the other one on SO.)
What would be the goals of qualitative and quantitative testing?
As in your previous question, you still need the sample size. There is a fundamental difference between confidence intervals and standard error on the one hand and the standard deviation on the other hand. Both the width of CI and the SE converge towards 0 as N gets large because you are getting increasingly precise estimates of the true mean. The (sample) SD does not, it converge to the true standard deviation of the distribution. It doesn't matter how many people (or trees) you observe, they are not getting more similar to each other just because you know more about them.
Ouliers in the error series will "deflate the ACF" thus yielding an ALICE IN WINDERKlAND effect. All ACF's are subject to this thus one must ensure no anomalies
I don't think there is any such "assumption". PCA is just a multivariate transformation. How well it works for your purposes may be affected by whether outliers are present, but that depends on your purposes as well as the data. But you are right that checking for outliers is sensible data analysis. I haven't used SPSS for some decades, so can't fill in myself.
If you have a working distance function, you can use most distance and density based methods, e.g. DBSCAN. Or try Generalized DBSCAN, which doesn't need a distance, only a "neighbor" predicate, so you only need to define a binary "similar" property.
Thanks for this very clear explanation!
I am doing my stats for a while now in python. I found the following documentation for model selection in scikitlearn that seems to be able to compute the mixture model and can compute the BIC/AIC: http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html
they are blood pressures in mmHg in a baseline and followup checks, so I'm pretty relaxed about assuming normality and of course non-discreteness. It was just an exercise that showed me how much more powerful is paired-t-test (when available) over non-paired.
+1 for Teknomo's site. It's got some useful stuff
Note that the weights are not a complication at all, because they can be absorbed in the values of $b$ and $A$, leading to an ordinary least squares problem with a single linear constraint. That means you problem is solved as described at http://stats.stackexchange.com/questions/24193 .
I think you misunderstood something: the scale parameter is no a mean, but the inverse of variance. So by considering in your problem the variance, you can equally well make inference on the scale parameter. The mean is a location, not scale parameter.
The other question is [Interpreting 2D correspondence analysis plots](http://stats.stackexchange.com/questions/3270/interpreting-2d-correspondence-analysis-plots). See also http://stats.stackexchange.com/q/3287/32036.
More or less: as the Wikipedia article explains, there's also the case where individuals are paired because they're similar in other respects than the variable being used in the t-test.
Distance travelled is unlikely to be well approximated by a normal distribution; it's likely to be highly right-skew. In that circumstance you might want to consider something other than a t-test.
This isn't answering your question so is a comment... If it were me I would be considering two things (1) Prediction intervals: you should be able to get your model to output a prediction interval, e.g. [0.25, 0.35] rather than the point estimate 0.3 meaning that 95% of the time the interval will contain the true probability for your inputs. (2) which category to choose, eg, YES or NO. Here it is usual to simply choose the category for which your model outputs the highest probability. The prediction intervals should give you some idea as to how confident to be about categorization.
nice answer, just thought I'd add the perfect multi-collinearity means that your model can't be *uniquely* estimated.  The estimate you get depends on what you do with the "null space", which has no influence on the fitted values.
The expectation of the variance of the sum will be different without replacement; you'll need a finite population correction factor if there's no replacement. (To see this intuitively, note that if K=n the variance of the sum is zero, because it will always be the same number; so as K approaches n the variance of the sum will be lower.)
At a particular time $t$, if you want to know the one minute average of the time it takes to make a breath, you would have to consider the data points in the past?
The specification test your mentioned is a special case of Pearson's chi-square test. There are many alternative goodness-of-fit tests. You may be also interested in Kolmogorov-Smirnov test.
Thanks greg your post helped, I applied both and it appears that EM generated better clusters than k-mean. (I think its mainly because the data I have is continuous and there are no gaps). 

I am little confused, since I only have 1D data, then I should probably do binning to categorize the data. What do you think?

What exactly do you mean by parameters? Does it refer to attributes of an instance?

Thanks
Ali
I think is better to not pollute this here and open a chat with this theme. I will try to migrate that on chat.
Thank you, this explanation makes a lot of sense.
Nice, thanks! Note that testing exact equality of floating point numbers may give unexpected results: http://cran.r-project.org/doc/FAQ/R-FAQ.html#Why-doesn_0027t-R-think-these-numbers-are-equal_003f
Check out this answer: http://stats.stackexchange.com/questions/5344/how-to-choose-nlme-or-lme4-r-library-for-mixed-effects-models/5350#5350
You should find an appropriate package for that and ask the maintainer if they want to include it.
+1. And note that some forms of regression (such as a pure $L^1$ regression) will *not* produce the mean in this circumstance, so the conclusion actually says something about these particular estimators.
+1 Thank you for your help @a.desantos I think I got it, I added a new picture to verify if I understood the point :)
You will need to be precise about what you mean by "correlation" and what other assumptions you are making about the problem. (If you need help formulating those, we can assist there, too.) Otherwise, the answer to the question in the generality that it has been asked is: *You can't*.
This question seems to focus on Python coding. Do you think it can be reformulated to highlight some theoretical or practical interest in statistical terms?
Thank you so much! Can you also recommend a textbook with such information, perhaps with R applications?
Those priors aren't very good ones - going to $\\infty$ near 0 in all three cases with means of 1000, if I recall the JAGS parameterization correctly - and I'd suggest rerunning the analysis with different ones. Try something along the lines of `alpha ~ dunif(0.001,2.0)` and same for beta, and `mu ~ dunif(1,30)` just to try out the code and see if it is the priors doing you in.
Don't compare AIC and $R^2$, instead compute AIC and $R^2$ for both.
The answer I wrote at http://stats.stackexchange.com/a/26704 implies that something like (2) is the case (according to laws of large numbers) but definitely not (1).
With a categorical predictor, each level is fit individually--it's just the mean of the response at that level, so this shouldn't happen. Is your covariate (score1) continuous? Can you post an image of your residual plot? If CV won't let you, try uploading it to something like imgur & adding a link here, then we can add the figure for you.
colnames(predict)[1] <- "Trade" would be another way of fixing merge
+1 Now *there* is an illuminating answer :-) (sorry).  By the way, the principle is named for Christiaan Huygens, not Huygen.  Huygens was the first to appreciate new developments in probability published in the 1650's by Pascal (based on his letters with Fermat): it was Huygens' account of these ideas (1657), including that of expectation, which originally got probability theory on a mathematical footing and paved the way for the seminal (posthumous) treatise of Jakob Bernoulli (*Ars Conjectandi*, 1713).
@user3010273 One vector to rule them all.
Plot it how? Over time? By covariates? Just a picture of the distribution of the HR?
This is quite interesting and new information for me. Of course out of the box implementations would be better, but this really helps me to know something closer to what I want. Thanks.
This isn't really about vectorizing calls to `dpois`. It's more a question about creating vectors.
What is the aim of your analyses?
Have the same problem, could you find a solution for this? For now:
- I set MAX_COLS as the number cols in the longest observation, then,
- For each short sequence I repeat the last observation till the number of observations on the row is equal to MAX_COLS. Please, if you have a better approach let me know
@DavidJ.Harris I don't think that the OP is disputing that EM behaves as it does, but asking *why* one method seems to be widely used and another method not used so much. Your comment doesn't seem to directly address why EM might be preferred.
Great answer. Right to the point. Thanks!
Thank you, I'll go check it out when I'll try installing later tonight :) if it works I'll check your answer =D
@Frank: Do you mind explaining your first comment? For each value of the regularization parameter $\\lambda$, there is, indeed a constant $C$ corresponding to the constrained optimization problem of the OP. And vice versa. (Just think about the Lagrangian formulation.)
@Kate I'm not sure what you mean by altering the axis - if you do plot(mod1) it shows you the component smooths without the intercepts, so the y-axis scale will be centered around 0. When you use predict(mod1, ...) you are getting all of the model, including the intercepts, combined in a single value.
I think this is a show-and-tell kind of question. Maybe plot the accelerometer data, by axis v. time and give us a peek? (a faceted plot, or three plots using the same timescale)
@bnaul: Let $F(x) = \\mathbb P(X \\leq x)$ be an *arbitrary* distribution function and let $X_1,\\ldots,X_n$ be an iid sample from $F$. Let $\\newcommand{Xk}{X_{(k)}}\\Xk$ be the $k$th order statistic. Then $$\\mathbb P(\\Xk \\leq x) = \\sum_{m=k}^n \\mathbb P( |\\{i: X_i \\leq x\\}| = m) = \\sum_{m=k}^n {n \\choose m} F(x)^m (1-F(x))^{n-m} \\>.$$ **QED**.
I appreciated some of the lighter wording in chapter 15, such as `... typical data never exactly fit the model that is being used, even when that model is correct. We need the means to assess whether or not the model is appropriate ...`, and even `They deem a fit acceptable if a graph of data and model 'looks good.' This approach is known as chi-by-eye. Luckily, its practitioners get what they deserve.` But the hard formulas are, unfortunately, a bit beyond my understanding. Nevertheless, your answer made perfect sense to me, and I appreciate your valuable input.
@Roman, I second @mbq's remarks. A figure would be helpful, especially in terms of understanding what you mean by "noise". Just because the time domain signal is smooth does not mean the Fourier transform will be. Just think of a sinusoid to convince yourself. in terms of plotting, the first thing that comes to mind is: are you plotting the magnitudes and not, say, accidentally, just the real part. Also, do you know about windowing?
I don't know why SPSS doesn't report the ORs but that's not uncommon. In any case: you can just take the exponential of the bootstrapped coefficients and the CIs.
I am not sure what you are trying to do, but you  could use "number of hours worked on Monday" as a weight
How do you know that the tests are strongly interdependent? Is it your a priori high level knowledge (e.g. both tests use blood pressure, so probably are correlated), or do you have statistics that show that they are correlated? If the latter, you can use a slight modification of fgregg's proposal: model all tests as independent, except for those interdependent pairs, which you should model as pairs. This will require some extra statistics (of the form $p(T_1, T_2 | Disease)$), which you presumably have since you know they are correlated.
The leave-k-out validation would have been a great option but my dataset size do not allow such luxury...
If some classes do not appear in the $k$ neighbourhood of $x$ (e.g. no instance belonging to a given class $y_i$ is among the $k$ nearest instances to $x$); how would you do in this case ?
Following up on @Giorgio Spedicato's answer: are we to suppose that you do want a set of models that treat each dependent variable separately, as `lm` does when you give it a matrix?
I think usualy **I** is used instead of eval in formulas,  i.e. Y~I(1+2*X1)+X2+X3-1
see [here](http://stats.stackexchange.com/questions/44246/nls-curve-fitting-of-nested-shared-parameters/44249#44249) for an approach
Cosine for boolean (binary) data is called sometimes Ochiai coefficient and has its formula, but general cosine formula of course is valid too. So, the only ambiguity remaining is with that string variable. tf-idf (as far as I know) isn't cosine. Well, _what are similar and what are dissimilar records by that string for you_?. Describe it - perhaps with examples.
That looks to indeed be the answer. I'm hesitant to rescale it - may just have to do some adjustments to what I intended to graph. Thanks.
The mistake must have been a typo somewhere, but i couldn't trace it. I started over and it went fine.
are you looking for continuous or categorical predictor variables? see http://stats.stackexchange.com/questions/68035/why-dont-statisticians-use-mutual-information-as-a-measure-of-association/68047#68047 - G-test http://en.wikipedia.org/wiki/G-test
Your method doesn't fix that and isn't necessary. Just look at the effect sizes - here, the proportions in each cell of your table.
Interesting question. Initially, I had some trouble understanding your pseudo-code, and then I read: http://www.thebroth.com/blog/118/bayesian-rating
Did the answer below help?
@Peter Flom, SSN tells the place in sense that if you gather all data from all people it will tell you, but after another birth, you don't know anything about SSN->place. Well, this is just an example to easier understand the problem -- however in practice I indeed have similiar trait, and I was wishing DT would take into account number of edges. I didn't anticipate the effect of zero entropy in outcome set.
Matt, my question really amounted to this: you show us a picture without any description or reference and refer to it as "data": I would like to know in what sense you are thinking of it as data. Your comment confuses this issue, because a "heat map" representation is usually *not* data, but is something created out of data. If those were irregular 2D point data, for instance, and you fit a radially symmetric density to them, then the *picture* could be construed as one-dimensional, as you argue, but it would not be a lossless dimensionality reduction of the *data*.
I gather K-W tests the null hypothesis "the probability a random observation from a group is larger than another from another group is 0.5"

The K-W test would give a single result for my entire data, rather than for specific groups (sources)..?

Can I use the K-W test to test a group-specific hypothesis e.g. "Source1 patients return more frequently than source5 patients"? Or should I conduct multiple t-tests of various pairs of sources (bad idea?)
As for your comments re raw data and other tests, I fully agree. Unfortunately I only have the data presented here - other data is not collected.
One can also perform sensitivity analyses by, say, bracketing results based on mean imputation with results including reasonable minimum and maximum imputations.
@guy, thanks for pointing that out: I did not read the other version carefully, once I had discovered it was a very close match.  I agree with you that Henry has a useful answer (and the other respondents have contributed worthwhile replies too); that's one reason I have not voted to close the question altogether.  However, I feel strongly--and this reflects site policy--that using the work of other people (even if it is a question or variation thereof) without attribution is wrong.  Downvoting appears the best way to recognize that in this case.
Thanks for the explanation. So the solution would be to just rerun the bootstrap untill I get one with no convergence errors?
@FrankHarrell: The knowledge that Z-scores allow me to bring readings with different units on the same scale, was the motivation. I have also used the values to perform an outlier analysis (as a part of preliminary tests) for which I would need to have all my readings on a common scale. Hence the aforementioned approach.