tttjjj/elt_bench huggingface dataset:
"Important" in the sense that the graph should reflect the nature of the data as a matter of clarity and even honesty. If the data are discrete to the extent that only multiples of 5 are possible, it would be better to show thin spikes. If essentially every score in the range is possible, touching bars are better (although I wouldn't object to each possible value getting its own spike). I know no more of the data than anybody else, but I am imagining integer scores.
That's my understanding (but I'm not familiar with this kind of analysis). However, pay attention to the documentation. Apparently only certain values are allowed for this (1000 is allowed).
Thanks for your answer. I have already read this Wikipedia article, but I can't find a link with my input: I can't express binary constraints.
Can you modify your title / question accordingly?
A researcher at NASA advised us to modify chi sqr test as follows: instead of dividing by variance, change the summation by dividing (o-e)^2/i^2 where o are observations, e expected values and i observation errors. This doesn't seem to follow from either Pearson's test or dividing by variance. How must I calculate chi-square in this case?
Thanks a lot for suggestions! My first approach was formulated under the traditional ANOVA framework, and that was why I was concerned about the midpoint with a weight of 0 being ignored in a post-hoc test. Thanks again for confirming the mixed-effects modeling method!
(+1) I fell into the trap as well (shame on me!) :-S
Welcome to the site, @JanPhilippS. This seems like as much a new question as an answer to the OP's question. Please only use the "Your Answer" field to provide answers. If you have your own question, click the `[ASK QUESTION]` at the top & ask it there, then we can help you properly. Since you are new here, you may want to take our [tour](https://stats.stackexchange.com/about), which contains information for new users.
So the "linear" part of "multicollinearity" should have been a clue, huh? Thanks! Clear and helpful.
Do you have access to any linear programming software that can solve integer programming problems?
Are you after a prediction interval or are you after something else? I'm going to use a slightly different notation for the future value so it doesn't look like the covariate is a function of it, by letting $i = n+1$. Note that a glm conditions on the covariates. As such,  *given* $\\lambda_{n+1}$, the probability of observing $\\lambda_{n+1}$ is 1. It makes sense to ask something about $y_{n+1}|\\lambda_{n+1}$ but not $\\lambda_{n+1}|\\lambda_{n+1}$
No, I mean dependent variable. Not sure what you are referring to with you second question.
I suppose per $ spent makes more sense, but both are odd. It's not "frequency per dollar spent" it's "frequency (dollars spent)"
@Glen_b: I am trying to see the trend in the means over time. I don't have any information about the individual datapoints or sample size.
+1 great reference. There is an [article](http://www.jstatsoft.org/v38/i08/paper) in Journal of Statistical Software about msm package. The model seems ideally fitted for this kind of task.
In fact, the same method works if $Z_1$ and $Z_2$ are jointly normal with correlation coefficient $\\rho$ (instead of being independent) with the minor twist that now $$X=a_1Z_1+a_2Z_2\\sim N\\left(0,\\sqrt{a_1^2+a_2^2 +2\\rho a_1a_2}\\right).$$  Of course, $\\rho = 0$ if $Z_1$ and $Z_2$ are independent and so the bells-and-whistles formula given above simplifies to the one given in probabilityislogic's answer in this special case.
Thank you @Germaniawerks. I am unsure of the latter part of your response, m-1?
@MånsT I know about classical probability theory and the usual distributions (gaussian etc), I've already designed and implemented some online clustering algorithms, and wanted recently to make use of particle filters to do some online clustering, but each time I find papers talking about DP and Mixture models, so I want to know about that.
@MånsT That's cool! No, I welcome additions to my posts, feel free to add the figure.
I bet I would find this example pretty interesting if labels were provided for the X- and Y-axis.  That would give the viewer something more definite to grapple with.
As noted by others here, method 1 will lead to problems.  For an intuitively accessible treatment of why that's true / a description of another one of the issues with this approach, you may want to read this: http://stats.stackexchange.com/questions/20836/algorithms-for-automatic-model-selection/20856#20856
That certainly seems reasonable to me. I don't know anything about tumor diagnosis and classification, though.
Yes thank you :) one more good idea!!Thanks a lot vinux!
I am very glad to have the support of a time series expert here.
There may an issue if two values in the sample are equal but have different weights - I have not though about it.
Thanks for TeXing that up for me. This is my first time on a TeX-enabled SE, and the WYSIWYG for TeX wasn't working for me, so I left it out. Also, I fixed the last two equations, in which I accidentally put in $S_2$ where I meant $S_0$.
(+1) This is the hard way :) I like it.
For starters, read about the Lyapunov version and the Lindeberg version of the CLT on the [Wikipedia page on the CLT](http://en.wikipedia.org/wiki/Central_limit_theorem). Then go read some of the references therein for more in-depth knowledge.
Here's what I was thinking of. It was on Hyndman's blog. http://robjhyndman.com/papers/complex-seasonality/
Thanks for your suggestion, Avraham! I just edited the density-function to use an epanechnikov kernel, which has finite support. The problem remains. And more importantly, the deleting of the more extreme values in the red sample compared to the blue sample should make the tails thinner, thereby adding probability mass to the body of the distribution. But as can be seen from the plots there is a loss of probability mass in the body after removing the extreme values in the tails. I am still confused.
I'm accepting this answer as it pointed me to the right direction (though it was in R!).  It might help someone implement the same in R.
I think doing this might introduce some undesirable dependence between your independent variables since, as you said,
`then "NYC" will always be 1 when "Bronx" is 1`. 
What about trying something like splitting your `NYC` variable into two distinct factors - `NYC-Not Bronx` and `NYC-Bronx`?
But nothing will make up for lack of data.  I think you just want to see if there is one or two variables that seem to stand head a shoulders above the rest.  But you may nit find anything.
Thanks for the response Charles!
The problem with the cost sensitivity and threshold tuning approaches you described is that, while they are good ways to *tweak* a model's PPV, they are not very effective ways to *maximize* the PPV. This is because they work by shifting the *position* of the model's decision boundary, but not its *orientation*. For maximum PPV we often need a completely different decision boundary than the one produce by the standard model training process.
I don't know if there's a "right" way to tackle the problem of maximizing PPV. (...)
have you tried 1. log(x) 2. log(x+1) 3. x^0.5 ?
+1 The code does not generate data from the KDE correctly: it chooses values uniformly from the *density* rather than the distribution.
I habitually use `factor(x)` so that I can include the `levels` argument if I wish. You could also use `as.factor(x)` if you wish, and it may in fact be faster, but I would think you'd need quite a large dataset for the speed of these functions to matter.
So you are saying that "preventive maintenance" both costs money and reduces the expected time to the next failure? What is it preventing?
First of all the constant smoothing parameter is achieved when the filter converges to the steady state. So in that respect exponential smoothing and kalman differ. Secondly I think the question requires clarification in the sense that when you mean Y1 and Y2 are they different time indicies or position in the vector Y for the same time?
Thank you very much for your answer - +1 for the references.
Typically ${\\bf \\beta}={\\bf 0}$ works well when there are no collinearity problems. Could you provide more information about the sample size and the number of covariates?
Slightly confusing because although you recommend ggplot2 you don't use it in your example? A ggplot2 equivalent woudl be library(ggplot2); qplot(Petal.Length, Petal.Width, color=Species, data=iris, main="Edgar Anderson's Iris Data").  This also has the advantage of automatically producing a legend.
The first question was answered at http://stats.stackexchange.com/questions/25010/identity-of-moment-generating-functions and in the OP's recent question at http://stats.stackexchange.com/questions/84158/how-is-the-kurtosis-of-a-distribution-related-to-the-geometry-of-the-density-fun.  Feller's example is attributed to Stieltjes (way before Feller's time) in Stuart & Ord.
**Better yet**: for more on time series, check out the [top users for the "time-series" tag](http://stats.stackexchange.com/tags/time-series/topusers).  IMHO, it's best to pay attention to users who have relatively high votes per question answered.  (Currently an average of 4.0 or more votes per question would be pretty good.)
@Glen_b I guess the lognormal distribution should appear in some physical situations because of the CLT.
How big is your sample?
(+1) it is also recommendable to write the source files as the functions and then `debug()` `browser()` are helpful :)
Following your representation, the reduced form can be rewritten as
$p=cX+h(\\epsilon_{d},\\epsilon_{s})$, where $X$ is a dummy variable
which always takes the value $1$, then the conditional expectation
of the reduced form error, given $X$, is $E[h(\\epsilon_{d},\\epsilon_{s})|X]=E[(p-cX)|X]$,
which implies the unconditional version $E[h(\\epsilon_{d},\\epsilon_{s})]=E(p-c)$.
Now, if I estimate $c$ by the sample mean of $p$, and call it $\\bar{p}$,
then $E[h(\\epsilon_{d},\\epsilon_{S})|X]=E[h(\\epsilon_{d},\\epsilon_{S})|X=1]=E[h( \\epsilon_{d},\\epsilon_{S})]=E(p-\\bar{p})=0$
by the law of large numbers.
If you want to include all the 3 categorical variables in your model, then yes.
*I don't know what you're trying to convey with the word "skewed" there*. Can you rephrase?
@sunita it should be more clear now.
Related to @whuber's calculation: [Recent MO question and answer](http://mathoverflow.net/questions/88544/on-two-dimensional-gaussian-integrals).
In my case how does it apply. Like I have predictions and their corresponding ground truths. So by expectation you mean taking the average of both the predictions and the ground truths and to see if they match?
In that case, a regression of adult count on heather is definitely not the right method. That regression would merely tell if you the relationship between heather and butterflies, which presumably you already know? It sounds like you need to keep collecting data over the years to eventually construct a time series to demonstrate if the butterflies are returning or not. In this case it would be preferable to collect information on area of heather rather than percentage, since that will help control for comparing fields that are probably of different overall size.
`R` `prcomp` package recklessly calls eigenvectors "loadings". I [advice](http://stats.stackexchange.com/a/35653/3277) to keep these terms separate. Loadings are eigenvectors scaled up to the respective eigenvalues.
Please state what was written exactly. Right now what you say you saw makes no sense.
I do agree. However in most textbooks, basic linear algebra will be used. If you want to be prepared to Baum Welsch, have a look on EM algorithm.
@robin: while the absolute value function is continuous everywhere, its first derivative is not (at x=0). This makes analytical optimization more difficult.
@MichaelChernick, maybe I'm confused. I was thinking of a random sample as licencing inferences from the sample to the general population that she really wants to be talking about, but that by 'observational data', I was assuming that she wasn't randomly assigning new mothers to different levels of breastfeeding support that she had independently manipulated, and thus she cannot directly apply a causal interpretation to her findings. I think of these as distinct issues. Am I missing something here?
Well said, another thing is that a p-value tells you nothing about *what to do* if the null hypothesis is "rejected", but a plot of the data gives you a clue as to the problem
Thanks Glen_b. The best solution I've come up with involves using the \\$coefficients matrix returned by lm.influences. \\$coefficients contains the perturbations to the full fit coefficients, and with it the held-out $\\hat{y}$'s can be calculated.
There are a couple of misunderstandings here. The fact that your responses cluster around 25 & 75 does not make it a "binomial" distribution. The word you may be looking for is *bimodal*. Binomial (logistic) regression *does* require your response to be dichotomous, but OLS reg very much *does not*, nor does it matter that your response is bimodal & non-normal. In OLS reg, only the *residuals* need to be (roughly) normal, w/ how rough is acceptable depending on N. This Q may be helpful: [What if residuals are normally distributed but y is not?](http://stats.stackexchange.com/questions/12262/).
@Matt Sorry, I added the sources in the description of the image when writing down the question without checking they were displayed. I added now the links for both figures.
@anonymous_4322 I am quite sure Procrastinator was referrring to the variance of the sample variance estimate and not the sample variance itself.
(1) Probabilities depends on *counts*, not proportions.  (2) Probabilities require some kind of model for variation.  What is your model?
By the way, thanks a million for the responses, Cam - I really do appreciate the guidance.  I'm very new to this stuff, but it has definitely piqued my interest in a big way!
Hi. The random variables $a_i X_i$ are independent, but not identically distributed. So you can try to apply the Lindeberg or Lyapounov version of the CLT, see e.g. [link](http://sites.stat.psu.edu/~dhunter/asymp/fall2002/lectures/ln04.pdf) It will give you conditions on $a_i$ that limit the contribution of each element of the random sum.
the X values are to be simulated as well, the first set of parameters beta0,beta,...,beta7 are selected for the first simulation say with a sample size of 100, then the generated estimates are now used in the case with a new sample size say 200 and the process continues as we keep on increasing the sample size and replacing the estimates.the response variable y is binary 0,1, the x values can be normally distributed.
No, the standard Halton sequence gets a bit weird in higher dimensions/bigger primes: the subsequent point lie on parallel lines, so to get the unit square covered for 41 and 43, you need to have 41*43 = 1700 points. The solutions to this problem involve scrambling operations, although I am not sure whether Niederreiter covers it. I just published a little paper explaining them :)
My apologies for the typo; you are correct. I was missing a "~". That part should read "random = ~ 1 | subject".
+1.  Given that this is truly "quick and dirty," one might be inclined to emphasize that $\\beta_4$ is the *average* marginal change in Y.  It could differ appreciably from the actual marginal changes.  This average will be meaningful only if the relationships among the independent variables and the dependent variable are reasonably linear.
For readers who will read this response to the end I would suggest to add a brief take-away message (and to provide appropriate citation if it applies).
@Tim 99.9% sounds like a rule of thumb ;) I can perfectly follow your explanation - thanks!
@AlexTwain No response yet. However I found that Lichtenwalter and Lussier does not have a valid nd.edu email address. The thirds name Mr. Chawla is the name of a professor from Univ. of Notre Damme Indiana. I will search further in a hope to find a valid contact for the first names. However, despite the votes, I am still convinced that I am right this time.
Great!  This makes sense to me.  Thank you very much.
Hi IrishStat. Thanks for the answer. I'm afraid my current knowledge is too shallow for me to get far with the information in the link although I'm sure it is saying something relevant. I hope to learn more as I try to tackle this problem and the others I want to work on after this one. Thanks.
+1. The fact that the joint distribution cannot be derived from the marginals is a very important remark!
You might like to take a look at [the law of total covariance](http://en.wikipedia.org/wiki/Law_of_total_covariance), which in your case is: $\\text{cov}(S_X,S_Y)=\\text{E}(\\text{cov}(S_X,S_Y \\mid N))+\\text{cov}(\\text{E}(S_X\\mid N),\\text{E}(S_Y\\mid N))$. Pay particular attention to the second term, which is a covariance of two functions of $N$.
-1. There are a lot of sweeping statements here. You should elaborate on some of them if this answer taken seriously, since the OP is asking for help picking one of these languages and specifically ruled out R.
One can also look at the [code](http://pastebin.mozilla.org/1254318) if interested.
The way you describe it, structural equation modeling would seem a natural choice.
I strongly disagree with the common claim that $t_{30}$ is sufficiently close to normal. It depends on the purpose, but I regard even $t_{60}$ as not especially close in parts of the tail often used in inference. As such, I often avoid approximating $t$ by $z$ unless the df are a good deal larger than 30. Accepting other people's rule of thumb without a clear idea of the circumstances underlying the original recommendation are a recipe for disaster.
It would have to disagree. All examples I find on the internet speak of "repeated observations" or "repeated measures" but those repeats are always time-based (measure the weight and length of 100 subjects on t=0, t=1 and t=3). Can I consider the categoric country-level variables as repeated measure? And if so, how are they generally incorporated into a mixed model? As "fixed factors" and the company-level variables as covariates?
You may like to add the `goodness-of-fit` [tag](http://stats.stackexchange.com/tags/goodness-of-fit/info).
Are you interested in univariate multiple regression (one predicted variable) or multivariate multiple regression (many predicted variables analyzed simultaneously)?
I think it's wrong to say that R would not be helpfull in most corporate settings.  Knowing R won't really help you learn SAS if that's what you're stuck on, but that's like saying you need a hammer when faced with a box of screws.....
I don't see a reason why not, as long as the various factors are based on different variables.
share for 2012 is 50% and for 2013 it is 60%
@COOLSerdash (ctd)- The 2nd line finds the frequency associated with the biggest peak in the spectrum and inverts to identify the period. At least in this case (but I suspect more widely), the defaults on it essentially identifies the period that maximizes the likelihood so closely that I deleted the steps I had in to maximize the profile likelihood in the region around that period. The function `spec` in TSA may be better (it seems to have more options, one of which may be important sometimes), but in this case the main peak was in exactly the same place as with `spectrum` so I didn't bother.
There are an infinite number of ways in which a stream of numbers might be nonrandom. Given your sample size looks rather small, can you be more precise about what kinds of non-randomness you want to detect?
For the record: what the marginality principle does say is not to include higher-order interactions without including the lower-order interactions or main effects involved in them.
It's worth adding that this prior is improper.
A statement like the ones you suggested doesn't necessarily result from hypothesis testing. You might be better served by a confidence interval.
I agree that a binomial model makes no sense in this context (in fact I'd use stronger language than that), and that as you suggest, to even make sense of the presentation of the data you require the text to be broken up into pieces in some way. However, for the purposes of answering it as a test question, you put that aside (apart perhaps from a parenthetical comment on the issues), and simply test the hypothesis as given - i.e. 'here's a table of data, test whether it comes from this distribution'; the interpretation of the conclusion may give you some trouble for the reasons you note.
@sarikan Training an SVM means finding the optimal solution to a specific question posed by the user. This question includes a definition of distance via kernel parameters, $\\gamma$ for RBF, and a misclassification penalty via $c$. Those parameters define the training problem which is then solved. The parameters themselves must be set by the user *before* the training optimization problem can be solved.
In *mathematics* it refers to the [Gamma function](http://en.wikipedia.org/wiki/Gamma_function): is that what you have in mind?
It's hard to improve on Peter's answer.  The unfortunate problem with much of data dredging is the lack of admission by the authors that the hypotheses were not fully pre-specified, i.e., not using the term 'exploratory'.  Many, many researchers are dredging data to get a publishable paper and not following up with any attempt at validation (which would often disappoint them).
Can you be a little more specific?  What hypotheses are being tested?
This question is too broad to be answerable at present. Can you narrow it down and clarify it?
You may find [how to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) helpful.

What are you trying to find out?
I feel like this question underspecifies what you're trying to do here.  Some questions: 1. You are trying to compare two vectors.  What aspects of those vectors are important?  Distribution of values only?  Positions of values?  Other?  If you're looking at an image, the distribution of values is often just the brightness?  2. What leads you to believe it's negative binomial?  Are you sure of it?  3. What distribution properties do you care about? Mean?  Variance? Shape/type of distribution?  What are you looking to compare about the vectors?
In most textbooks there is a reference now to the LASSO or similar shrinkage methods. But the lack of consensus on how the estimates should be interpreted and CI estimated is a significant issue. Most references I see either don't mention this or relegate this to a brief aside (like the linked article).
+1, that was my first reaction too--the spread of the residuals looks okay.
Right. See http://stats.stackexchange.com/questions/3038/how-to-test-hypothesis-of-no-group-differences
Hi, thanks for your comment. Assuming your D stands for dummy in the formula you are almost right. I understand the need to pick some artificial values for the time- and individual fixed effects which is only going to affect the intercept as you said. However, what is the best way forward when you not only have interactions between a dummy and an explanatory variable but interactions with multiple (discrete) explanatory variables. Something like y = B0 + B1x1 + B2x2 + ... + B10x1^2 + B11x1*x2 + B12x1^2*x2
No, you cannot because $X$ and $log(a+bX)$ are not independent. Instead, you should *probably* start with the definition of an expectation as an integral...
Duplicate of [question already asked](http://dsp.stackexchange.com/q/12928/235) on dsp.SE
When you do y^-1/2 or y^-1
@PeterFlom What about in cases where the end goal is not inference?  I agree that the appropriateness of a descriptive statistic depends entirely on the reason for producing the statistic.  The notion that it is possible that "no descriptive measure makes sense" seems to imply that a descriptive statistic cannot be inherently meaningful.  I would argue that in almost all cases, the median makes sense as a measure of the center of the distribution by definition.  Whether or not it makes sense for other purposes is another question.