tttjjj/elt_bench huggingface dataset:
If you want to know if that 8% is itself economically important, I suggest comparing it with the population average level of y or the predicted values of y for average values x. If that estimate is something like 80% the impact may not be important. If the average is 10% then that 8% is a powerful impact. This is essentially an odds ratio approach.
Thanks, I missed the "for given variance" part in the question, which has been edited. My haunch for the minimum variance is the normal distribution, but i am not sure.
"Polarised" implies that a dimension of a space is bounded by two ("bipolar dimension") or one ("unipolar dimension") clusters. That is, `A-----B` or `A------`. If there are 3+ clusters in a line, `A--B--C`, word "polarised" looks unsuitable.
If you look at my second example, $X$ explains the response variable very well, with an $R^2 = 0.93$, but there's still correlation.  The correlation also has nothing to do with heteroskedasticity.  The key is that the response variable = the estimated regression function + the residual, so it makes sense that the response variable would be positively correlated with the residual, since it's made up in part by it.
I don't see why the unbalanced design is a problem, but the mean weight difference is obviously a problem with my suggestion. I would still think the multi-level framework would allow you to assess differences between treatments somehow (I do not know how offhand though). The only other thing I can think of would be to simply graph each control/comparison like this response did (minus the connecting lines) http://stats.stackexchange.com/questions/2067/follow-up-in-a-mixed-within-between-anova-plot-estimated-ses-or-actual-ses/2138#2138 , although I imagine this is not entirely satisfactory.
Because this is a homework question, the question should have the `self-study` tag. See http://stats.stackexchange.com/tags/self-study/info. Also, you should tell us what you've tried so far and where you get stuck.
(1) you can do it by self-code: http://www.r-bloggers.com/r-for-ecologists-putting-together-a-piecewise-regression/  (2) Change the psi values. I'd try 50, 40, 30,and  20. `segmented` can be start point sensitive. (3) try with `fit=lm(A~B)` to get starting values.  (4) try another package ? `SiZeR`
I might try that out. One problem is that I would still need to make sure the shape of the distribution would be the same. Also time constraints will be prohibitive.
@Glen_b Although an isotope chemist might not agree, for all intents and purposes you can consider the data to range from -Inf to +Inf. Isotope ratios are in permille units and are the relative ratio of heavy to light isotope in a standard and in a sample.
I would start with something like this: http://cran.r-project.org/web/packages/Epi/Epi.pdf
1) Yes I see an S-curve, on the right picture. Anyway, strange questions...
f(x;θ) is the same as f(x|θ), simply meaning that θ is a fixed parameter and the function f is a function of x. f(x,Θ), OTOH, is an element of a family (set) of functions, where the elements are indexed by Θ. A subtle distinction, perhaps, but an important one, esp. when it comes time to estimate an unknown parameter θ on the basis of known data x; at that time, θ varies and x is fixed, resulting in the "likelihood function". Usage of "|" is more common among statisticians, ";" among mathematicians.
I think it may be better just to use a less aggressive optimisation technique, such as greedy search, or to monitor generalisation performance on a disjoint validation set to avoid over-optimising the criterion. Early stopping often performs a similar function to regularisation if it is safe to assume that you start with a simple model which becomes progressively more complex as the criterion is optimised.
Hi Peter. Thanks for your answer. Sorry for the dumb question: what do you mean when you said multiple regression? I searched but having not a strong background on inference, I am having some problems understanding it. How does the multiple regression relate to the multiple pvalues I have in input? Thanks
Do you have counts for each of the 960 videos, or just the totals over all of the videos? If you have the raw counts, look at the distribution of the differences. Test whether the mean difference is zero.
Please explain: 'categorically numbered' and 'diff in R'
The R commands are either lme() or lmer(). Both are mixed effects models. lmer() is newer and preferred, although lme() seems to work fine for most applications. Pinheiro and Bates - Mixed Effects Models in S and S-Plus covers both the theory and applications of these models in R pretty well.
I edited the question. Basically I want to combine the information of a, b and c in a way that causes as little distortian to the original information as possible and then compare it. Basically I should normalize the mean of all series, I guess. What I have been wondering is if I would have to normalize the std dev as well ... I am pretty unsure.
Thank you so much. I wanna calculate each term in Bayes'rule: P(Temp|Pressure)= P(Pressure|Temp)* P(Temp) / P(Pressure). I wanna use MATLAB. This Bayesian Network is assumed to be used to predict accidents in as Nuclear system, Natural gas, etc. How to calculate the prior, likelihood and evidence. I wanna know what is the probability that Temp will change if pressure changes
Sorry, but if you have "absolutely no knowledge of statistics" you should probably get someone else to write the report with you.
Not a complete duplicate, but most of your questions can be answered here: http://stackoverflow.com/q/20566869/1429003
From my understanding, the two runs typically occur *without* replacement.  It's true that the EV happens to be the same regardless, but I don't think your first line is likely to be very convincing to a skeptic.  For instance, if a player only has one "out" and hits it the first time, he cannot win the second time.  Your answer is right, but I wonder if maybe you could improve it by more explicitly dealing with cases like this?
Don't forget color-blind palette versions of the above; leaving out green is generally a good idea, but there are some other color blind pitfalls to be wary of.  'http://research.stowers-institute.org/efg/Report/UsingColorInR.pdf' for more.
It may be worth pointing out that when $x \\lt -36$ or so, $1 + \\exp(x)$ will evaluate to $1$ (*exactly*) due to floating point rounding.  Similarly, when $x \\gt 36$, $1+\\exp(x)$ evaluates to $\\exp(x)$, whence the quotient produces an *exact* value of $1$.  The precision issues when $|x|\\gt 710$ are astronomically smaller!
"Confidence intervals" on random quantities are usually named "prediction intervals".
Is the Rosenthal & Rubin paper on the web anywhere ?
Generally speaking, your actions after your decision should also be chosen before you test (how else can you weigh up the costs of the two kinds of errors?). At the least, you would likely move to consideration of estimated effect sizes. The null is not tenable (by criteria *you chose* - if that's not sufficient for you, what would be?), so what values are plausible instead? e.g. in your indicated test, what values for $\\mu_1-\\mu_0$ would be reasonably plausible, given the data?
Why are you writing your own algorithm? Why not just run an ordinal logistic regression where "result" (W/D/L) is the DV and rating and difference are the IVs?
Hi Kiki, welcome to CrossValidated. In order for people to give you good answers they need enough details to understand the issue. Also, CV is not twitter - there's no 140 character limit, so please avoid abbreviations like 'Bt' and 'Pls' - make life as easy as you can on the people trying to understand what you want.
The forecasts are skewed low by the outlier at period 128
thanks, useful links
If you do see spatial autocorrelation in the residuals you can include the observations at nearby locations ("spatial lags") as predictors in the model as Sameer suggests. Another option for handling spatial autocorrelation is to model the spatial trend by including a semi-parametrically estimated function of the spatial coordinates using, for example, a generalized additive model. See [this related question](http://stats.stackexchange.com/questions/35510/why-does-including-latitude-and-longitude-in-a-gam-account-for-spatial-autocorre) for more info.
If you just had missing data on $x$, I would think you could stratify, but I gather $x$ is perfectly confounded with the censoring indicator. I think that makes the problem intractable.
I'm not sure if we misunderstand each other or it's just my own lack of comprehension of part of your response. I have no problem with the use of a log-scale, merely the value labels on the axis. I find it much easier to interpret "0.005" than "-2.3". I'm merely suggesting the use of tick marks and labels for $p$ rather than $\\log_{10}p$\\*, or at least the use of two axes. $\\quad\\quad$\\*(while keeping the plot essentially unchanged -- well, I would also plot points rather than lines; what does the length of the line-segment mean on the plot?)
You don't have enough rep to *vote*, @Heta. Click the *check mark* below the vote total to accept. You can always accept an answer to your own questions no matter what your rep is.
When a line is given a mass density proportional to a probability density, then indeed the mean is the center of mass: the analogy (between probability density and mass density) is *perfect*.
Why do you want to use methods intended for classification to do dimensionality reduction when there are other procedures designed for that problem (e.g. principal compnent analysis)?
(+1) For a well though out and presented (and quite unique!) question. There are a couple tangentially related questions on this site; some under the `games` tag.
I don't see that (I get identical DF). Is it possible that you have missing data? For repeated measures ANOVA, the entire case is dropped if you have missing data; not so for LMM (one of the main reasons that I use it).
possible duplicate of [Reducing no of variables subsetted based on depth for PCA](http://stats.stackexchange.com/questions/43529/reducing-no-of-variables-subsetted-based-on-depth-for-pca) OR [Is it possible to do time-series clustering based on curve shape?](http://stats.stackexchange.com/questions/3331/is-it-possible-to-do-time-series-clustering-based-on-curve-shape)
Would you not still expect a placebo effect in this situation (that telling teachers to use a new teaching strategy would have a placebo effect on the outcomes)?
@gung: I am using SAS and trying to impute missing ages with data sampled from a normal distribution (i.e. int(rand('NORMAL',41.25,12.02))).
I would clarify point 1 by saying that rather than approximating the model, you are approximating the posterior distribution of the parameter.  You will get a different "approximate model" for different parameters.
Instead of bonferroni correction, you could always use the more powerful bonferroni holm. See my anyswer here: http://stats.stackexchange.com/questions/575/post-hocs-for-within-subjects-tests/760#760
Here is a link to a page where they derive the Fisher information for the negative binomial:
http://www2.isye.gatech.edu/~brani/isyebayes/bank/Quiz71.pdf Hope it helps,
This can be viewed either as a routine exercise in plugging in formulas or as a way to stretch your understanding of regression.  In the first instance, remember that the hat matrix is a projection (whence equal to its square). In the second case, interpret $R^2$ and $r^2$ in terms of the variance of the residuals, recognizing that the residuals of $\\mathbf{Y}$ vs. $\\mathbf{X}$ must be the same as the residuals of $\\mathbf{Y}$  vs. $\\hat{\\mathbf{Y}}$.
Could you indicate more clearly just how you propose to use bootstrapping for *constructing* a model?
I have tried to expand the formula myself logically but I am getting above-one results from it as the right sides (those subtracted) of both numerator and denominator get extremely small.
Oh wow thanks a lot ! This is exactly the sort of answer I was looking for.
But I have seen in many places people uses chi-squared test for text classification problem. And In RTextTools pacakge does they do any kind of feature selection before creating the container.
Thanks! Good explanation of some aspects of Bayesian thinking - not something I'm very familiar with.
I don't mean to "pile on", but it's also hard for me to see how the quantity in brackets can be approximated by a negative number.
Sorry for answer you late and my unclear expression.Here is my experiments. First round,the relative influence with levels of factor features A(810 levels 0.6) B(822 levels 0.39)  C(a continues variable 0.01) others are zero .Second round,I delete this three predictors,get the result D(continues variable 0.7) E(continues variable 0.2) F(422 levels 0.1).
This R package seems to be for this too, I didn't look into it thoroughly though: http://artax.karlin.mff.cuni.cz/r-help/library/CORElearn/html/rfClustering.html
The LaTeX does not work with my navigator. Is it ok now ? I will come back later for the stochastic monotonicity.
Do you mean that you want to find out if the average grade in your class is different than 15?
Sorry for the delay in responding. Students would be using Minitab 16, which has some of the more traditional data reduction techniques mentioned above. I'll look into self-organizing maps, but I doubt if it would be appropriate for the kinds of students I get in a second year undergraduate course.
This demonstration seems to assume the expected amount of money is zero, but it's not.  Furthermore, you appear to assume that the net of doublings minus halvings must be nonnegative, which is also incorrect.  Collectively these errors yield the correct limit, but that's an accident.
I edited your post to remove your signature.  Your name appears to the right of your post... so you don't need to sign your questions (or answers).
Thanks! It seems that it has quite a few names.
Welcome to the site, @Fael. +1 for a clear question.  I took the liberty of tweaking the formatting slightly to make it cleaner. Please make sure it still says what you want.
The significance level is not needed for this question, I removed the somewhat confusing discussion of that part as an unnecessary distraction.
Do read the help `?ordiellipse` for details of what is being done here, esp in computing the confidence interval. Whether the theory matches the data is something you might want to look into with simulation or resampling or something rather than rely on "theory".
dominic: Let me try again. You don't know the *population* mean. If you want an estimate to sit in the middle of your interval (*estimate* $\\pm$ *half-width*, as in your comment), you'd need some estimator for that quantity in the middle order to place an interval around it. What are you using for that? Maximum likelihood? Method of moments? something else?
Great suggestions. I like the response change idea. With this approach would I still need the week 0 score as a covariate?  Also what would be different if the week 36 value was week_36_score-week_18_score?
A useful tip for a search term: this problem is often called the "Class Imbalance Problem".
It looks like it's just a learning rate/scale factor. If you look at equation 9 of the text, a unit's update is scaled by alpha, but the bias update is scaled by $\\alpha * \\beta$. If $\\beta=6$, then the bias updates 6x faster...
Link to [Valle, Li and Qin paper](http://distancecovariance.googlecode.com/svn/trunk/References/Selection%20of%20the%20Number%20of%20Principal%20Components.pdf)
@StéphaneLaurent ... [meh](https://en.wikipedia.org/wiki/Meh). I see the off topic-ness (as you fairly claimed :).
Thank you for your help.  I knew that my protocol wasn't the best way to go about things, but thought it would be an okay shortcut to take to save some time.  Apparently it was not.  The links were very useful, not just for neural networks.  For this project, I'm not going to continue with the neural network.
Thanks a lot, Jeromy, for your most helpful answer!
I really think you should use a multilevel model. A varying intercept and varying slope by month is exactly what you want. Take a look at Gelman and Hill's book Analyzing Regresssion with multilevel models or look here at stats exchange about multilevel logistic. It's fairly easy to run a simple multilevel model in R with package lme4. See this thread for instance: http://stats.stackexchange.com/questions/9429/correct-specification-of-longitudinal-model-in-lme4/10415#10415
@Alexis This is called linear interpolation.  (From around the 1300's until c. the 1980's, every educated person in the West was taught how to do this, because information had to be looked up in tables since personal computers were not widespread.) You can do it graphically by (quite literally) plotting a pair of points in Cartesian coordinates and connecting them with a line segment. The formula is just the familiar "$y=mx+b$" form from elementary algebra in disguise.
I see your point. This is almost something that I've mentioned in my question body. But what @hr0nix mentioned is more interesting me. As I understand, he is treating training the parameters just like an inference problem.
@Luciano, are each of the questions of interest on their own?  If so then I would analyze each with it's own separate model.
@amoeba: Apologies for the confusion; I never meant to imply that they are the *same* but rather than they can be *equal* (so for example if A is a strictly positive diagonal matrix). I guess I used the term "same" to match the OP's question. I should be more careful in my wording; I will fix it. I clarify the matter ($\\Sigma^2 = \\Lambda$) in the following paragraph.
Why do you assume that R and SPSS are using "the same procedure"? If you can't post your original data, you could at least post a reproducible example with some made-up data. It's hard to help you if you don't.
Great suggestions. I should have just called it "wavy" from the beginning instead of calling it sinusoidal the first time. I used lowess (thanks to a comment by user777) and will find the worst-case difference in predictions to evaluate if a linear model is good enough. Thanks.
+1 I agree with every word you wrote, but I don't think this fully addresses the question. For instance, does specifying a more appropriate covariance structure reduce model residuals?
Are you asking how to do this in R? It will be easier help you through this if you edit your question to add the summary of your model. FWIW, I don't quite follow your description of your situation; it might be nice to have more info there as well.
I was puzzled about $Var(X v_1) = \\frac{d^2_1}{N}$. Where does the $N$ come from ?
I fixed the formatting problems with your R excerpt, Sarah, but it still doesn't seem to be valid: you should edit it.  Also, it seems that in the last two lines you try to present your data, but what exactly do they mean?  Are you trying to say you observed a $0$ 150 times, a $1$ 200 times, ..., and a $7$ 260 times?  It seems so, but if that's the case you definitely do not have a Poisson distribution and estimating its parameter is meaningless (or deceptive).
@Max Please post answers in the answer forms rather than as comments.  Brandon, because your English statement of your question and your $\\TeX$ formula sharply differed, I have taken the liberty of correcting the formula to match the English.  Please check that it states your problem correctly. Finally, I would just like to observe that $\\delta$ depends on the essential supremum of the PDF; there (obviously) can be no universal bound that is true for all distributions supported on $[0,1]$.
It seems that it is not a simple question. I am surprised that this Topic is not treated or explicitly treated in the books.
Although a covariance defines an inner product (Parzen, 1961), it is only SEMI-definite. To be a metric, you need strict definiteness. (e.g. zero covariance doesn't imply identically distributed random variables.) I agree with @Münchhausen that this is not the case for distance covariances, unless you consider a equivalence class of zero (distance) covariance random variables.
Your point about a measure of spread is a key here. That's one of the issues that keeps coming up in this discussion, under other names, and it also ties in with Erik P's discussion of weighting schemes.
@Macro I'm interested in differences in response patterns, so whether for each question the two values match or not
That's correct, I haven't accounted for the degrees of freedom in what I called the exact CI. What I'm getting at is this: assuming the CI is an asymptotic, dist-n free estimate based on the CLT, then yes the sample mean and sample standard deviation can be arithmetically derived, given the sample size. Without knowledge of the construction of the CI, this may be perilous. CIs for the odds ratio in logistic regression models are non-symmetric and you couldn't estimate the "mean" and "standard deviation" of the sampling distribution of the sample odds ratio based on such a CI.
This question appears to be off-topic because it is about designing a statistical model rather than about R programming
This appears to be one-half of the linked question, Rico, and would thereby already be covered within its answers.  Could you explain why we might need another thread to discuss this topic?
Hi @whuber thanks for commenting. I'm new to statistics and I don't yet have a firm grasp on the language of stats. I updated the question to provide the t-test and some summary data as well as an attempt at the simulation that referenced. I realize that this test may also suffer from the same shortcomings (not enough observations) as the question you referenced. I hope that working through it will improve my intuition. Thanks!
You need to edit your question to provide us with more details. A specific example with some data would help. Perhaps you could use linear regression of your response on the predictor. You could then exponentiate the resulting beta coefficient and look at the resulting relative risk plus or minus its 95% confidence intervals. You could build ten separate models (i.e., one for each response variable) using this technique. Keep in mind that you would need to correct for multiple testing.
The problem of the OP is that $\\ell=\\ell_n$ and $\\hat\\theta=\\hat\\theta_n$ depend on $n$. Then the term $o(|\\widehat{\\theta}_n - \\theta|)$ is somewhat strange because actually the $o$ depends on $n$ too. You have not adressed this question.
Because the distribution of the first draw is invariant under the additive action of the group (a shift) and is independent of the second draw, so must the difference be invariant. That means nothing other than that its distribution is uniform. This argument generalizes the question, because it shows *it is immaterial how the second value is chosen,* provided only that the two draws are independent. As an extreme example, the second value may be a constant.
Because the $z$- or $t$-values are always the coefficient divided by its standard error in `lm` and `glm`.
How do you measure "capacity to colonize"? Is it number of '1' cells? Or particular cells? Or what?
Thanks for the advice, that's very helpful!  And I appreciate your suggestion for double-checking the null deviance estimates.  As for the degrees of freedom, that is not a typo-- although there are 7 (probabilistically) independent variables, the rank of the matrix of observations is only 5 (i.e. the observations are linearly dependent).  Thanks!
I made an edit to suggest a possibility.
As usual with cross-validation, once we are satisfied with the model we re-train using all of the data. The models described in paragraph three don't fit on the held out data. Holding out data is a tool to guide model evaluation and selection. You should always fit the final model on all of the data.
Despite the collinearity, is there a reason your results are not interpretable as is? Take a look at this too perhaps: http://psycnet.apa.org/psycinfo/2008-08581-005
@Momo Thank you.  I got that far before posting today.  I am stuck on how to write the inverse link and whether the search space needs to be constrained.
@user3762838 format as code