tttjjj/elt_bench huggingface dataset:
Yeah, this is what I went with. To give a bit more detail, I'm trying to use a weighted logistic regressor as the 'weak learner' in an AdaBoost classifier. So I need some way of adjusting the classifier based on the AdaBoost weights. However, I'm not yet completely sure that just putting a linear weight on the log likelihood as I've proposed is the correct way to minimize the total misclassification error, which is what AdaBoost wants.
I think you will need to tell us what $Z$ is first.
I have both the hour of the measurement, and the "day of the year" of the measurement.
$$\\{\\text{Distributions of correlated uniforms}\\} \\subset \\{\\text{Copulas}\\}$$
Hi, @Majte. I took the liberty of editing your post slightly in keeping with some of the features the site offers to make it more readable. If you don't like it, roll it back with my apologies.
They used a paired *t*-test, 15 subjects. See here for some sample calculations using r.pwr: http://pastebin.com/mR4v2umH

The observed *d* was .7 IIRC.
Do I need to do Jacobian transformation in order to obtain the mixture distribution (P + Q) / 2  ???
Your answer reads more like a comment, and as an explanation it leaves too much unsaid. Could you explain what you mean, please, perhaps with an example?
See *micEconAIDS* package, the code of function `AIDSest` for example. Or alternatively use this function and do not worry about precise workings of systemfit, since this function will do what you want to do, i.e. estimate AIDS model.
Thanks for indulging me and (+1) to your question. By the way, Billingsley is a wonderful reference text, but must have been a bit frustrating as a class test and self-study choice, if for no other reason than the organization. You might be interested in D. Williams' *[Probability with Martingales](http://www.amazon.com/Probability-Martingales-Cambridge-Mathematical-Textbooks/dp/0521406056)* if you want a short companion that places a *decidedly* large emphasis on conditional expectation. Cheers. :-)
Could you try formating your data so that we can get a clearer idea of what you want. For example, I don't see an 8x11 matrix in your example.
The following Qs are strongly related & will be of interest: [How to generate correlated random numbers given means variances & degree of correlation?](http://stats.stackexchange.com/questions/38856/), [How to define a distribution such that draws from it correlate with a draw from another pre-specified distribution?](http://stats.stackexchange.com/questions/13382/) & [Generate a random variable with a defined correlation to an existing variable](http://stats.stackexchange.com/questions/15011/).
Can you clarify your question? I'm not sure what you're asking.
+1 for the many well placed references on the (now clarified by the OP) subject of logistic regression vs. LDA.
@whuber I am sorry, I am not exactly good a maths, but what do you mean " the log files represent a common period of time, or at least equal time periods." and how does that change the equation again sorry for questions.
*How* have you simulation the datasets?
@csgillespie no problem :)
This is so helpful.
@NickStauner Just found this discussion. It is not necessary for at least one account to be wrong if there is a set of accounts that disagree. They may be based on different models. [If you are game, you should read Bill Thompson's book The Nature of Statistical Evidence (2005).] Nonethless, my account is definitely right ;-) (Although just this morning rejected by a journal again.) I found the Nuzzo paper to be careless and potentially misleading.
If by mentioning "Field" you refer to a textbook, please be more specific.
Tizz says "100 is 50% correct in linear terms, but we would want maybe 66%?"  If he would just correct the typo in his upper point and set it to $200$ instead of $150$, he would get $100$ as $66\\%$ correct on a linear scale.  But "I used a bell curve to determine the percentage correctness" undoubtedly sounds more impressive...
Thanks a lot for the links, Alex. The paper gives good examples of cases in which you'd want to weight your classifiers. I'm afraid this is not working for me, though, as one cannot use the "classwt" parameter for anything other than weighting the classifiers - i.e. you need one weight per class, otherwise the randomForest would return an error.
@ericmjl Thank you for noting that. The duplicate was a true duplicate by the same poster and garnered no answers. It was deleted (for some reason unknown to me) automatically by the "Community" bot.  I'll re-open this version in case anyone would like to address it.
A bit confused here, what do you mean by significance of a cluster? Do you know the labels already and want to see how well you can predict group membership? In that case it is a classification problem, not a clustering problem.
Maybe that's trivial, but using the garbage collection `gc()` sometimes helps. I had some calculations where I had to use `gc` 4 times within a function - but then it worked.
It is, of course, possible to have any number of statistics that you can use. Your alternatives are perfectly fine, although you'd have to work out sampling distributions for them, which would differ based on the number of cells. One thing that is convenient about this form is that it has certain relationships to other distributions, eg it is the distribution of the sum of k squared standard normal random variates.
Shouldn't the complexity be also dependent on the learning algorithm? VC dimension is usually applied to methods with convex loss functions. If you have a non-convex loss, you could get in the situation where you model would be able to separate some points but you learning algorithm will never find this solution.

Therefor I feel it should be quite hard to have bounds using the structure of the network. I agree with @tdc that generalization error is the way to go.
Vapnik's paper on statistical learning theory might be a good place to start learning about that.
No, I am not saying that the response is normally distributed. I (think) what I am saying is that in the classical linear model there is an assumption that the response is **conditionally** Gaussian, i.e. the distribution of Y given X=x.
Just to clarify: In your first example you're dealing with only integer elements (3 baboons, 2 cats, etc) but what you *really* want to deal with is all positive real numbers, right? ... Actually, I'm not sure I follow you at all.
@cardinal: thanks, I didn't know about that :)
Thank you for these useful guides.
@jonsca: Thank you!
Thank you very much, that helped a lot @ forecaster
The objective function at the bottom of p. 13 is just a constant multiple ($n$) of the objective function at the bottom of p. 10.  MLE minimizes the former while least squares minimizes the latter, QED.
Let's see `str()` of the data.frame.
The odds of winning what? Where does the 45% come from? Why wasn't that in your question? (Were we supposed to guess that?) Neither 0.45^10 nor 0.55^10 are 1/718, and probabilities are not odds. Odds are ratios of probabilities (odds against some event $E$, for example, is $(1-P(E))/P(E)$).
there are also _beta_ coefficients in financial econometrics regarding portfolio theory, so indeed a lot of confusion here indeed. It is easier to use exact definitions as @mark (+1) suggests instead of Greeks.
"Best estimate" in what sense - what is being optimized? Is anything known about the distribution of values customers report?
This is a bit like phoning the doctor and saying "My leg looks funny. What's wrong with it?" -- he'll do a lot better if you give him a look at it, and tell him what you were doing at the time it started looking funny. Is the 95% interval for the same quantity? What were the standard errors like? Can we see the output (and whether you specified weights, for example)?
Use the Fisher Transformation or a t-test based on a scaled tangent tranformation: http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient#Determining_significance
I agree. If anything, I'd put it in even stronger terms. I'd be amazed if there was strong evidence that some people really misunderstood the questions.
Thanks a lot! I think I was confused between clustered sample and using a robust variance (in stata vce cluster). I now understand why ttnphns edited my question. I realized that having repeated measures influences my analysis at two stages: 1) when choosing the correct estimation procedure (GEE vs. OLS) 2) when computing the covariance matrix.
I have edited your post per your comment. As to your question: the issue of the different meaning of the variables is still there if you don't merge the two datasets, so you can't compare them anyway.
This question appears to be off-topic because it is not about statistics as defined in the [help](http://stats.stackexchange.com/help/on-topic)
Thanks so much @ a11msp
Thanks for the responses! Most of my expected counts are under five, so I've used FET as suggested, using SPSS.
John, thank you for your answer. I should have been more specific, my data is actually nonlinear. Before filtering it has a hump in the middle. After filtering the frequency appears not to change, but the amplitude gets much smaller following 1980. Spec analysis says frequency has changed some, I would just like to find a test/s to see if thats true. Currently looking at the piecewise ar model, but running out of time for another cold trail. Thanks you in advance (very much) for your help!
Thanks. I am reading this and thinking about it. However, this would be a huge modification of the current algorithm.
Excellent question...All I know is that what is listed above, is exactly what is given in the example. Perhaps $X_n'X_n$ is referencing the $X'X$ matrix, rather that the matrix $X$ matrix. But there are only 2 rows in that, wheras there are 6 rows in the $X$ matrix. However, $r_n$ references this matrix, or at least adds a row to it, as the $X$ matrix was gotten from this same formula
Maybe no parameters to tune, but one has to really work with continuous variables (transformations, splines, etc) to induce non linearity.
GLMs (generalized linear models) can do this, if the variance can be written as a function of the mean.
I do agree strongly that plotting difference vs mean is a good idea. It can be found in Neyman, J., Scott, E. L. and Shane, C. D. 1953. On the
spatial distribution of galaxies a speciﬁc model. Astrophys. J. 117 92–133 and Oldham, P. D. 1962. A note on the analysis of repeated measurements of the same
subjects. Journal of Chronic Diseases 15: 969–977. The oldest Tukey references I know are later. I suspect this is a case where the same smart idea could easily occur to many smart people, so that identifying first creator is tricky.
@Glen_b - I am reading that. Would you happen to know how to go about Question 3 ?
I'm actually trying to establish control charts. I have collected the data. So all I needed was a way to determine whether my data is distributed normally, so that I could use appropriate formulas to calculate limits. Hence my post.
$X_j$ is the same for all $i$? And I suspect you want to show that the least squares estimates of $\\beta_1$ and $\\alpha_1$ are equal, not the true coefficients?
Thanks (+1), that's just the right check for this kind of off-the-cuff thing.  It ended up being highly significant (p = 0.005) so the article is fine.  I hope you don't mind, though, that I accepted the other answer -- it's good to know the 'real' way to do it when it matters.
some questions for clarification: 1. What is A..B / B+ in 2. ? We do know your data. 2. I wonder about 3.: If you have too much data to recalculate everything for every change, then does the change of single values even matter (regarding the acceptable error of the estimator) ?
This design effect is only relevant for the overall linear statistics (means, totals). For regression coefficients, the DEFF is more like $${\\rm DEFF} = 1 + (m-1) \\rho_x \\rho_\\epsilon,$$ where $\\rho_x$ is the ICC of the regressor and $\\rho_\\epsilon$ is the ICC of the error term (composite error = cluster random effect + observation specific effect). Due to the product of the correlations that tend to be small, this DEFF is also smallish.
(+1) especially for the insights in the interpretability of distances vs areas and slope vs height.
@amoeba Thank you, that is a good suggestion.  Since I retained the original `R` code to read and process the data, it was a simple matter to compute the k-means solutions and plot them up.
Yes, I believe you're modeling when there is no data to be had (excess zero) and when there is data to be had, but it might be zero (true zero). I think another example could be traffic accident analysis at a rural intersection: during many hours there are literally no cars there and hence excess zero accidents. (I'm still learning myself, though.)
Yes, each cluster is being resampled. I can find the maximum and get the predicted value and CI for the maximum from the model. Let me see if I understand correctly what you said about age: around the age $t_{max}$ corresponding to the predicted maximum $\\hat{Y_{max}}$, the interval within which the $\\hat{Y}$ are not significantly different from the maximum $\\hat{Y_{max}}$ is the CI for age. Is that an accurate paraphrasing of what you said? Thank you.
@Richard, I tried negative binomial, and that didn't fit. Thanks for the suggestions though.

Since I couldn't figure out what kind of distribution this was, I decided to ignore the distribution and go with a non-parametric test, the Mann-Whitney U test.
By biological or size-standardizing I mean: What head dimensions and tail lengths would my animals have if they were the same body lenght? (so if snake 1 has 12 cm body lenght and 12 cm tail length and snake 2 has 24 body length and 18 cm tail length we can say after size-standardizig, that snake 2 has shorter tail - assuming same growth rate snake 1 will have in future body length 24 cm 24 cm long tail). Do I have to do something like this before PCA?
This was enormously helpful. In regards to question 2, I just looked at the `summary()` output for the model, and indeed the variance is 0 and SD 0 for the random effects variable. I'm surprised by this; I think I need to go back and make sure the data in my data matrix is lined up correctly.
@David If you are using an unbiased estimate of variance (or, at the least, an estimate whose bias is independent of sample size) then the expected difference in the variances is zero, regardless of the sample size.  Although the larger sample indeed increases chances of obtaining extreme values, it also is likely to have many values close to the mean.  The two effects balance out.  Often, in fact, the bias in biased estimators (such as the MLE) is positive and decreases with sample size, so we would expect a small *decrease* in estimated variance in the large sample!
+1. Just to avoid confusion, the R² is .86; it's the adjusted R² that's .84 in your output.
How does `autocorrelation` relate to your question?
Agreed, thanks. Actually, it is not my scale, but found on a paper that I am reviewing.
welcome to the website. Please format your post, it is difficult to read. If you could post your data on a publicly accessible website, other users could help you using other software packages. Also, show your exact input syntax.
I think in order to get some answers you'll need to give more details about what sort of fraud or abuse and what sort of detection.
Just an observation: Since the non-weighted scenario is equivalent to setting all weights to 1, it is clear that by allowing different weights you will be working with a more complex model. This means that the model will be able to capture phenomena not captured by the original model. This will come at the cost of requiring more data to properly generalize or an increased chance of over-fitting. So, without knowing anything specific about the domain, I would suggest using a weighted version only if the unweighted one doesn't work for you.
values in $[1.7,1.8)$ are in the second half of the values to the right of the `16` stem. You should review how stem and leaf plots are defined.
I think that makes sense. Basically you are breaking it into 6 regressions, one for each direction, and then "stacking" the Bs in columns to produce the result. Thank you.
This might cause difficulties if the OP wanted to use the auto cost variable later. Is there a formula-oriented approach?
As stated in the above comment, I don't know the exact theoretical distributions generating $X$ and $Y$. What I can surely say is the support of such distribution(s) is greater than 0 with a known lower bound and possibly no upper bound. Said that, the only distribution I can work with is the empirical one.
@user777 Is there any diagnostic test that I can implement in order to see whether the correlation between the free parameters has caused that the chain didn't converge or other reasons and make decision based on that?
It is inappropriate to game the system by playing with the cutoff.  Cutoffs need to be chosen on the basis of the cost/harm of false positives and false negatives [and even then a cutoff only works if costs are constant over all subjects].  For this particular case if you were to use the cutoff of 0.57 you find that the age variable has negative information because a model with age has a proportion correct of 0.56 even though it has a $\\chi^2$ of 10.5 when added to the null model.  For most projects the utility/loss/cost function (necessary for classification to really work) is unavailable.
I went back to the textbook and looks like I still don't quite get it I'm afraid.  The comment I made above about the Helsinki Heart study sums up where it seems a little paradoxical to me right now
Transfer function modelling is a classic approach: you treat one series as an input, fit an ARIMA model to it, then examine the cross-correlation function of residuals from both series fit by the same model (prewhitened).
Box, Jenkins, & Reinsel (1994), _Time Series Analysis, Forecasting, & Control_;
Cryer & Chan (2008), _Time Series Analysis: With Applications in R_
+1. Thanks for reminding this opportunity.
@chl: cool, definitely going to check it out then.  :)
Until the last paragraph, i agree with you. But the reason is different than you explained in the last paragraph, i think. I have updated my question with extra clarification. I think thats the reason but i dont know whether it is a good way to apply precision&recall metrics and whether i'm doing something wrong
+1. Statistics has a lot of excellent machinery to offer for these situations. I would shy away from approaching these types of problems from a ML angle.
Not sure what you've been reading, but there's much criticism of p-values' being reported where a point estimate & confidence interval would be more useful; N-P hypothesis testing isn't a move in the right direction from that point of view.
I believe this question is collecting downvotes because people suspect it is a routine homework problem but is not tagged as such.  For more information about this, please consult our [help](http://stats.stackexchange.com/help/on-topic). I find it to be an interesting question because you have stated your hypothesis and put it up for examination.  It would be nice, though, if you would edit the title to reflect the question itself: I cannot make any sense of the title at all!
thanks, could you suggest any good resources to look at other than wikipedia?
Without more information, it will be difficult for people to say whether the low value is reasonable or not. Could you explain how you computed the $R^2$? Can you provide a simple, short set of sample data exhibiting the apparent problem?
Averaging might be useful, Koller et all uses it, calling it "damped BP." However, it's not a panacea -- BP converges to it's fixed point or orbit geometrically fast, on other hand, some inference problems can be proven to take exponential time even to approximate, so averaging of BP must not give the correct solution for those problems.
You most likely use `lme` and not `lmer`. Other than that, why not? Situations like these are what AIC and BIC are in general for.
Could you please tell us what these data *mean* and what the plot is intended to represent?
I don't have the answer, but it is good to know about this. I have a vague memory of SAS (Proc Logistic I think) giving different likelihoods and degrees of freedom if you specify the data as line-per-person (single-line syntax), or as aggregate data with counts of events/denominator for each unique set of covariates (events/trial syntax.) I may be suffering from a faulty memory, but potentially looking at what SAS does might give you (or someone else here) scope to work this out?
@MarcClaesen I have edited my post.
I distrust this.  If I feed a gaussian smoothed bitmap of faces, and use your euqation to recover the 8-bit intensity values, then I will have fewer unique intensity values, but if I am working on facial recognition, then I can have substantial complexity in the integer-valued information.  The number of neurons is driven by the information required to relate inputs to outputs, and not by whether the input is integer or float.
Brandon, welcome to this site! Please register your account, and follow the instructions in our Help center to [merge your two accounts](http://stats.stackexchange.com/help/merging-accounts).
Any help on the weighting of the objects? Thanks!
And also, in similarity computations, do I use document vectors taken directly from matrix $D$, or should I use $SD^{\\top}$ (the second solution is given e.g. on Wikipedia).
That's how I usually work out how big a sample to take for a randomization test - to work out how accurate I want my p-values and then get the original number of permutations from that. (If it turns out to be close to as many as doing all possible permutations, I do that instead - once or twice I caught myself randomly sampling a million permutations when the total number of distinct permutations of the data was smaller than that. I try not to make that mistake any more.)
This is close, but I notice I get some errors, like 1, 2, and 4, but I think I see what you mean and should be able to work with it.  Thanks!
`> rperm(6,3)
$failures
[1] 9

$sample
     [,1] [,2] [,3]
[1,]    3    1    3
[2,]    2    2    1
[3,]    1    3    2
[4,]    1    2    2
[5,]    3    3    1
[6,]    2    1    3`
I've been ignoring the warning so far but it's really bugging me to see that exclamation mark every time I run `PROC CLUSTER`.
If you're lucky Frank may drop in himself, but in the meantime, have you looked at the code (`rms:::nomogram`)? It's written in R. The book may also provide some useful clues.
Thanks, this was exactly what I was looking for. Sorry, I cannot upvote yet.
That is a much, much simpler solution. Thanks!
There are still people signing up to the Coursera class and posting in the forum. You can still see all of the lectures, and access all quizzes and programming assignments (among the quizzes). This information will probably be up until the course is offered again. I recommend enrolling in the course just to view or download the material.
pretty broad question but perhaps [start here](http://www.asdfree.com/search/label/american%20national%20election%20studies%20%28anes%29)
Just to avoid any confusion, I will be using the same colors with the same number of colors in same proportion in both the cases. The only difference between the dataset would be the configuration and labels of the graph vertices.
Just a quick thought: It seems to me that you loose control over time factors in the design you sketched. For example, if the effect differs between cluster A and B, is this natural variation or due to history effects??
The product $w_i v_i$ has as a 'normal product' distribution. I believe the the mean of this product is zero and the variance is $\\sigma^4$ where $\\sigma^2$ is the variance of $w_i$ and $v_i$. For $N$ largeish, you could use the central limit theorem to get approximate norality of $X$. If you can compute the skew of the normal product distribution, I believe you can apply the Berry-Esseen theorem to bound the rate of convergence of the CDF.