tttjjj/elt_bench huggingface dataset:
Hi Martino, Thanks for affirming my thoughts on this.  My concern is that can we assume upper and lower confidence limits will follow the same model as the means: I have found a another sub set of my data in which the sigmoidal curve for the upper limits actually intersects with the best fit curve for the means. This doesn't seem right.
Thanks a lot rvl
It's not evidence of bias that a model or procedure often predicts values that often occur. On the contrary, isn't that what you should want and expect?
@mjburns the model you have specified in the question is linear in the parameters. You need to chose a functional form for the non-linearity, see e.g. ?selfStart for a description and list of the available functional forms built in to R.
How about non-parametric regression in the np package?
I gotcha...thanks.
Have you tried to include another regressor (like a dummy variable) in your model to capture the effect of your big external event and improve the fitting over that?
Grounded theory related questions will likely get better answers at a sociology Q&A.
Just want to mention that the ``digits`` argument in fact does not affect anything that you add via the ``ilab`` argument -- after all, one can also add character vectors and how would you round those? So, if you want to round the things that you add via ``ilab``, then you must do that yourself. You may also want to look into the ``formatC()`` function -- things like this may be useful: ``formatC(c(-2.258, 3.412), format="f", digits=2, flag=" ")``.
The output of the deterministic signal is random, for example sine wave is a deterministic signal. What is meant by deterministic chaotic signal is that for an initial condition the system demonstrates the same behavior. By observing the behavior and hence the time series, the values of it are random.I am sorry I don't understand this part thats why I posted the question.Could you please elaborate?
See also my comment [here](http://stats.stackexchange.com/questions/51366/liblinear-one-vs-rest-learn-parameters/51372#comment100579_51372) for a literature reference.
@AdamO What part of my question 1 do you not understand? One possible cause of "researcher degree of freedom" is that researchers can fudge which case to be included. In political science / economics, we only have so many countries and so many years, so essentially everyone has the same sample. Does this eliminate the above concern? Also, isn't power the probability of rejecting the null given all $\\theta \\in \\Theta$ instead of only $\\theta$ in the alternative hypothesis?
Thank you for your suggestion and it worked well with the dataframe. However, the "predictors" set in my analysis is a collection of raster files and I am trying to predict the potential extent of my study species using the outcome of the glm I have performed. This has worked earlier with my other analyses and it is the first time I have run into this sort of a challenge.
"the user remembers his/her actions, and decides on the next action based on the complete history after entering the system" How did you assess this?
Hey thanks. Is there a way to see how the integral is computed though?
@Glen_b post edited
Thanks Kochede - actually, while I hear you on heteroskedasticity, the issue is less about stationarity than about more recent data being more important. Relationships between spreads evolve, and I want recent relationships to count more than old ones. This may lead to heteroskedasticity, or it may not: we could have a situation where the slope of the regression line simply changes (though I doubt it will reverse). In both cases of course, the residuals will be non-normally distributed: agreed. But I want to transform them so that they are. Will look at your suggestion before accepting. Thanks
@ Peter: yes the nlme package contains functions for gls. However, I couldn't find out how to make the regression in a form similar to: get regression for stock i in month t for all the cases
I'm still thinking you're going in the wrong direction. `4iii` _doesn't say_ that if `X` is a perfectly correlated bicluster then `R` and `C` will be additive. The implication goes in the other direction. Now, I agree that it's weird that the example they give doesn't seem to jive with the theorems it's next to. Perhaps there is some other information you could provide? Is there some other theorem that goes in the other direction?
What do you mean that the "models have different Y axes"? Are they different variables?
One should however note that this way one might lose robustness of estimators such as explicit MAD, which are sometimes driving to its choice against simpler alternatives.
+1 Awesome answer.
Afaik the number is fixed beforehand in all competitive current methods. This is somewhat of a problem since this means there are a lot of hyper parameters.
To deal with that, James Bergstra recently proposed the use of Gaussian Processes to find the best hyper paramter settings9http://people.fas.harvard.edu/~bergstra/files/pub/11_nips_hyperopt.pdf). But this is a sort of "outer loop" that tries many different settings in a smart way.
The calculations are simple. In (a), the sum of $n$ iid variables with means $100$ and variances $10^4$ has mean $100n$ and standard deviation $100\\sqrt{n}$. Using a Normal approximation, you would like $100n-1.65\\times 100\\sqrt{n}\\ge 1000,$ whence $n\\gt 16.$ In (b), noting that $(1-e^{-10})^{e^{10}}\\approx e$ and recalling $0.05\\approx e^{-3}$ (because a useful fact to memorize is $e^3\\approx 20$) you immediately obtain $n\\approx 3e^{10}$ = $3e\\times e^9$ = $3e\\times (e^3)^3\\approx 3e\\times{20}^3$ = $3e\\times 8000=24000e$. From $e\\approx 2.72$ it follows the answer is around $65000.$
Can you link to some context?
@Michael I've merged your previous account with this one.
It really depends on your data, and whether it has a sparse (in the variable sense) structure.  In some data sets, like spectroscopy, you have thousands of variables, but might only keep 30. That's really spase, but it happens. In other cases, there is not really a sparse structure. These techniques give you a principled way to investigate the possibility of a sparse structure and exclude variables, but for some data sets nothing will end up excluded. Take a look at the variables that were excluded - are they highly correlated with other variables? Can you see a pattern?
There's some discussion of the proposal [here](http://sfb649.wiwi.hu-berlin.de/fedc_homepage/xplore/ebooks/html/csa/node27.html). I think generally, you want $q$ to be not too light-tailed. If you can make $q$ similar to the thing you're trying to sample from, your dependence will be low and you'll get rapid mixing.
This appears to answer a different question: namely, how to add two Poisson distributions.  It is the special case $a_1=a_2=1$ (but can be extended to the cases $a_1 = a_2$ without any trouble). But what would you do when $a_1 \\ne a_2$?
Why the insecurity? If you understand the motivational differences between the two, you should be in one of two positions: Regard them as complementary and be willing to explore both. Regard one as much more convincing for what you want to do. It seems that you want to be told that there's a right thing to do, but PCA with or versus FA is such a longstanding area of controversy that if two experts agree it is usually only that they both disagree with a third expert, but for different reasons.
1) Why do you think your small data sets follow a Normal distribution?  2) The moment estimator and the MLE are the same for the mean, and are the same for the std. dev. if using $n$ as the divisor in the sample variance calculation. So your results should be the same regardless of which you use.  Even if using $n-1$ as the divisor, the difference will be far smaller than your plots indicate.  I strongly suspect a programming error or conceptual mistake... could you clarify a little more just what assumptions you made when building the CIs?
I've never budgeted a survey, so my figures are ballparks, and I might be off by a factor of 2. A \\$50K figure would give you a crappy sample size of 300-400 for a return mail survey. There are hundreds of companies who can deliver a survey like that, but you'd have to go a long way to convince anybody it is "a valid poll". Some people would be OK using that in their research projects, though. Serious face-to-face government surveys usually have budgets upwards of \\$2M, and assume a cost of about $1000 per complete, with rigorous and robust designs, non-response follow-ups and such.
You need panel data for difference in differences because you require a pre- and post-treatment period. The treatment effect is $\\delta$, the implicit assumption is that the treatment effect is constant over time but this can be relaxed if needed.
I think the choice also depends on the features you want to compare. You might consider histograms, `hist`; smoothed densities, `density`; QQ-plots `qqplot`; stem-and-leaf plots (a bit ancient) `stem`. In addition, the Kolmogorov-Smirnov test might be a good complement `ks.test`.
@Atticus29 (a) It looks like you typed a "1" where you shouldn't have, (b) Note that I specifically suggested using `str` and `dput`, which you failed to do. Most of what you added wasn't useful. Typing `str(split1_datam)` will confirm that the variable is a factor. Typing `dput(split1_datam)` will yield a version of your data that we can copy+paste into our R sessions. See [here](http://stackoverflow.com/q/5963269/324364) for examples.
Thank you Javlacalle
Freya, +1 for mental shorthand >> proper maths. Is that intuition also a reason for using [L.5-metrics-for-clustering](http://stats.stackexchange.com/questions/11436/l-1-or-l-5-metrics-for-clustering) ?
If you're doing full likelihood, to my knowledge there aren't closed form estimators; there are approximations, but the likelihood would be maximized by iterative optimization methods.
@GaëlLaurans Not at all. I'm currently doing my Master Thesis in evolutionnary biology and I am curious to have a better understanding of what would be the effects of correction for multiple testing in my simulations. Depending on these effects I might modify my script or not. Thanks
Thanks for your answer! Do you know any reference in which I may read how the parses are classified in order to create the features?
Actually it is kind of coincidental that rotations are linear and so a convenient way of describing what's going on for non-geometric data. The coincidence relates to the quadratic nature of both Cartesian/Euclidean space and the Central Limit Theorem/Gaussians. Viz. sigmas add up quadratically like orthogonal dimensions, which is where our ND rotational/orthogonal terminology originates by analogy with 2D and 3D space.
@Matthew, yes, I just got lost without knowing where to start when I'm facing one real problem, should I analyze the data to find out the nature so that I could extract useful features or just pick up a learning algorithm and run?
Note `chisq.test` performs *one or the other* depending on its arguments - read the manual.
I am testing for a structural break in the parameters of the model so I can tell whether there has been any change in ARMA coefficients after the policy change.
I am still looking for an answer to this question.
This question is readily answered by a quick internet search on "confidence interval."  For instance, Wikipedia links to http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval where non-symmetrical intervals are exhibited.  A [quick search](http://stats.stackexchange.com/questions/4713/binomial-confidence-interval-estimation-why-is-it-not-symmetric) on this site immediately answers the question.
From what I see, your data is generated by a Markov chain. Why don't you use all the work that has been done on this? Fitting your data to a model seems unnatural if the right model exists and is well understood. Markov chains are much easier to do inference on, since you don't have any hidden variables.
What I want to do is to set up my own contrasts rather than predetermined ones. It is my understanding (from reading The R Book second edition (Crawley) - pg 430 onwards) that this can be done for lm function but when using the glm function he seems to combine factor levels and then compare models using anova.
@Joel: No, LDA doesn't assume that you know the topics for each document beforehand. BTW, just to be clear, LDA represents each document as a mixture of topics, not by just a single topic. So each topic will contribute to some fraction in the documents (and the individual fractions will sum to 1). Basically, LDA assumes that each word in the document is generated by some topic.
Another example might be seasonality. Suppose people take a long time to return the DVDs for 2 reasons. In the summer, they spend more time outside, so it takes them longer to watch the films. In the rest of the year, taking longer means you loose interest and will churn. The behavior lacks signal in the summer, but signals something elsewhere. The conclusion will depend on when you do the analysis if you use the method on the left.
What are you doing with the data?  Are you estimating some response from the data.  Are you sampling from subgroups?  Do you know anything about the the sampling scheme (e.g. a stratified random sampling where within strata samples are not proportionals)>  No one has responded so far because the description is so vague.  Why is there a need to reweight?
I am not sure what you are asking here. Usually, the term “random” says something about a way a realization of a variable was achieved and is thus not a property that can be tested. If a variable (or, say, a time series) is the realization of a random process, of course also derived properties such as overlaps will be random.
Interestingly enough, though, the entire argument (for the lower bound) seems to work not only for the coupon collector problem, but for _any_ sum of non-identical, independent geometric variables with bounded variance. Specifically: given $T = \\sum_i T_i$, where each $T_i$ is an independent GV with success probability $p_i$, and where $\\sum_i p_i^{-2} \\leq A < \\infty$, then 
$$ \\Pr ( T \\leq E[T] - a ) \\leq e^{-\\frac{a^2}{2A}}$$
Ah sorry, should have said about those: nMDS > Non-metric multidimensional scaling, DCA > detrended correspondence analysis
You can also see bits and pieces of the discussion in the relevant book (Venables and Ripley MASS) on Google books: http://books.google.com/books?id=974c4vKurNkC&lpg=PP1&dq=venables%20ripley&pg=PA155#v=onepage&q=rlm&f=false -- if you want the whole thing you need to buy the book or find it at the library ... **very** briefly, robust regression downweights the effects of outliers on the analysis.
No. The fixed group effects already take care of the fact that objects in the same group might be more similar with respect to M than objects between groups. However, if you measure M several times for the same objects, you might consider a random object intercept to correct p-values etc. for possible intra-object correlations.
It is certainly false that one cannot find minimum without convexity -- it is just way harder, yet still in range.
I agree with @Peter Flom. If you are asking "what is the association of days since last visit with outcome", your study population is "visitors who have visited the site more than once". First time visitors are not included in this population. You could perform a regression on all visitors without including "days since last visit", and then perform a second regression on the sub-population that are repeat-visitors to explore associations with "days since last visit" in that sub-population.
Any proper prior with no nth moment would also work in your example.  I agree this is a useful way to show that something is wrong.   My thinking is more that the prior is not based on knowledge of $g(.)$.  Because you know $g(.)$ there is only *one* prior consistent with this information. This is the dirac delta function $p(c|g(.))=\\delta(c-\\int_{0}^{\\infty}g(x)dx)$.  To use any other prior is  logically incorrect.  Its kind of like saying $p(Z|XY)\\propto p(Z|X)$ when $Y$ is not  independent from $Z$ given $X$
I don't understand the recommendations to migrate the question to SO. This  question is clearly more likely to be answered by a statistician than a computer programmer!
I'm asking the question on stackoverflow now. But with a reproducible example in `R` http://stackoverflow.com/questions/24446300/sempls-package-backcalculating-influence
This "zigzag" term is really helpfull for detecting mixtures of populations, even so, it is a linear term. Here, the problem is the period of sinus wave, it is also a parameter to be estimated and can not be used the _lm_ function. The purpose of this question is to compare the model with _jump_ and without _jump_  tnks.
Given you flag a data-point as an outlier because it does not conform with some criterion `A`, why would you try to judge the goodness of your model, which was designed/trained using data that follow `A`, to estimate against a value that does not follow that criterion? You just said it was nonsensical (based on criterion `A`). On the same manner, the out-sample data, are out-sample data. The statistics of your `training` sample should not be applied to your `test` sample. Otherwise you are re-using your data.
@whuber it does not matter which analysis to use. And thanks for grammar correction.
Thanks for your answer, i'll take a look at harmonic regression, so much stuff to learn!
It seams that you just have not enough data. You cannot use data you don't have, and if the intersection is 31/01/2008-31/07/2008 then that's it!
Sorry I don't think statistics can help you here.
Like this :) (thanks)
@downvoter: Why ?
Thanks this is helpful though looking at Mellenbergh, 1995 APM it seems you can get away with setting the $a_1=b_1=0$ and still have identification while dropping the other constraints which might aid in comparison between the 2PL and the nominal model! Cheers!
AFAIK, a lognormal process is one whose logarithms form a [Gaussian process](http://en.wikipedia.org/wiki/Gaussian_process). However, the proposition is false: the points $(X(t), 1/X(t))$ necessarily lie on the hyperbola $y=1/x$, no part of which is linear, implying their correlation cannot be either $-1$ or $1$.  (The same reasoning shows, however, that the correlation between $\\log(X(t))$ and $\\log(1/X(t))$ is equal to $-1$.) In fact, the proposition is false for *any* process with a positive chance of attaining $3$ or more distinct values.
["Essentially, all models are wrong, but some are useful." - George E. P. Box](http://en.wikiquote.org/wiki/George_E._P._Box)
Douglas Bates and Steven Walker have created a GitHub project whose goal is to use pure R code to implement the PIRLS algorithm above. https://github.com/lme4/lme4pureR. If you consider the base `lmer()` function in the `lme4` package of R you would normally have to read through a whole bunch of C++ code understand the implementation of PIRLS in `lmer()` (which may be challenging for those of us not so well versed in C++ programming).
If normality is really violated that much in the residuals of the ANOVA you shouldn't be doing the ANOVA in the first place. You should just be doing non-parametric tests.  It's easy in repeated measures to combine variables because you can just get the mean of the two / subject.
How is seasonality relevant to 24 hourly values?
@John: This is my question (http://stats.stackexchange.com/questions/2639/how-to-compare-different-distributions-with-reference-truth-value-in-matlab) where I have defined my problem about what I want to accomplish. Thanks.
I looked at the R package docs. It's written very poorly, which is not surprising for R. I couldn't understand anything from reading [the document](http://www.inside-r.org/packages/cran/dlm/docs/dlmModSeas). I'd have to look at more examples of calling or the code to understand how to call it.
@osknows it is my opinion that regression (linear, generalized linear) is the domain where multicollinearity is an issue and is an issue *largely* for inference, hypothesis tests etc. As for other types of classifiers where predictive accuracy is the key interest, one should be careful about throwing the kitchen sink at the algorithm and hoping it figures it all out. If prediction is what you are concerned with, I would suggest experimenting and seeing how inclusion of correlated variables affect the accuracy. As in all things, there is no one answer and is dependent on the degree.
Expectations can vary a lot from one application area to the next, one journal to the next, one editor to the next and one referee to the next, often with little to no justification. (Much of the received wisdom isn't necessarily especially wise.) That makes it very hard to generalize.
possible duplicate of [Why is the sum of squared differences from the sample mean smaller than the sum of squared differences from the true mean?](http://stats.stackexchange.com/questions/26147/why-is-the-sum-of-squared-differences-from-the-sample-mean-smaller-than-the-sum)
Thanks to all who edited and helped.
I am sorry, it seems I don't follow you at all. Are you trying to code GLM yourself?
This model assumes the response is an additive function of traveling to each place, which is highly unlikely.  It can still be made to work by including interaction terms.  A full set of all possible interactions might be needed (beyond just the two-way interactions).  (That would be mathematically identical to providing a separate dummy for each possible combination of destinations.)
Please see the update above.
@onestep: This is very good answer to the problem described in the motivation, I haven't heard about it earlier and it seems to be really useful. Thanks!
Please review the [wiki for the self-study tag](http://stats.stackexchange.com/tags/self-study/info). I would like to suggest the OP might be better served by a thoughtful discussion of the *assumptions* you made to arrive at these results (rather than being given the results themselves) along with a clear interpretation of them. In particular, it would be interesting to see an analysis of how a mere 30 data values could be considered "normal" and to what extent your calculations based on a Normal distribution would be expected to differ from what is really in the dataset.
This is a non-question because *any* two distinct distributions $F$ and $G$ are part of a continuous one-parameter family $\\{pF + (1-p)G\\},$, $0\\le p \\le 1$.
How are ou going to get a 3-D surface? I see one IV (EI) and one DV (difference in scores). I'm not familiar with your field of work, but did Shanock's SPSS code make any assumptions about the moderators or DVs? If its just a polynomial regression program, then you should be able to use it.
It is helpful, thanks!  I don't expect this process to be able to capture all of the subtleties and hold absolute truth (for instance, theoretically a randomized, double blind experiment on a huge sample size which nonetheless makes a critically flawed assumption could still be less true than a study that scores less in all those metrics). But my hope is that it can be a reliable enough rule of thumb to be used as a surface view tool for people browsing a group of papers on a topic, while allowing them to drill down to more detail if desired.. particularly if two opposing papers score closely.
I won't explain point 1 because it is art. For point 2, use button Save in Hier. Cluster procedure. For point 3, use Aggregate procedure. For point 4, use check Read initial in K Means procedure (to know your dataset with input centres must look like, first save centres with Write final and glance at the output dataset).
This question appears to be off-topic because it is not about Statistics but about Excel
If you're going to use R, I'd recommend embedding your R code in an Sweave document that produces your report.  That way the R code stays with the report.