tttjjj/elt_bench huggingface dataset:
Something appears to have gone awry as from your data, the probability of bullying among those who were bullied is 22/222 = 0.099, but the plot shows it as 0. Suggests to me that the problem must be related to the 14 other 'predictors'. I suggest you try taking them all out initially, then adding them back one at a time to try to indentify what's causing the problem.
Cleared a bit; I hope you didn't meant some other GP method.
All possible combinations = $2^4=16$. Subtract four main effects--and don't forget to subtract $1$ to reflect an implicit constant (such as the grand mean).
I did think along similar lines to your steps 1 and 2. But I wasn't sure that it was possible to recover $\\rm{Pr}(F_i\\cap F_j)$ from the correlation. I can now see that it is, using $\\rm{Cov}(X,Y)=E(XY)-E(X)E(Y)$, and the formulas for expectation and standard deviation of a [Bernoulli random variable](http://en.wikipedia.org/wiki/Bernoulli_distribution) => +1 / deleted my answer.
I think that to propagate the particles forward in the correct manner you still have to weight them according to $p(z_t| s_t)$. Then if you want to use your fit function in parallel to check the fit of the particles that's not a problem. The posterior is likelihood (which is $p(z_t| s_t)$ in your case) times prior, so if you substitute the likelihood with another function you will NOT get a sample from the posterior. Normalizing change this fact.
Thank you for your answer. I indeed stumbled upon the IPUMS but failed to grasp its importance. Further, I am not interested in the Census data itself but rather in the research that has been done with that data. I would think of it as a meta level where I want to summarize the research. Would you say IPUMS is one of the largest access points (how many are there roughly?)? If so, could IPUMS be a representation of all access points and thus hold a representative collection of research papers?
By bias I mean the difference between the estimator's expected value and the true value of the parameter being estimated
I think you already have the answer. In your work, you have $P(S \\cap \\bar{M}) = 0.2$.
Almost by definition, "variability" reflects the value of $c$ (it's a scale factor). Therefore *no* measure that has your property (1) has any claim to be related to variability.
Looks like there are still questions about what the experiment is. I presume in this experiment 90 "boys" and 90 "girls" were asked to give a response to 500 different stimuli (each) and the 500 stimuli were the same for the "boys" and the "girls" and their reaction time (RT) was recorded. So you have 180 samples with 500 features and one output. 500 features/dimensions is not that large, so are you trying to use PCA to identify the significant stimuli that you can use to discriminate between "boy" and "girl" outputs?
If I understood you correctly, you want to look at the “residuals” of the model. Until someone provides an answer, this term might help you find some relevant information.
Welcome to the site, @Andres. You can use LaTeX here. I did so in your post to make it a little neater.
It is a homework, isn't it? Just think about what happens when $A=a$. What is the difference between $C|A=a$ and $B|A=a$. Anyway, looking at http://en.wikipedia.org/wiki/Convolution may help (if two probabilities have densities, the density of their sum is just a convolution of their densities).
You can use a combination of `R CMD Sweave` + `texi2pdf` to compile your document, without having to worry about `Sweave.sty`.
It would be great if the answers also explained how it works or how well. Now I just have to investigate all the links to judge whether I should upvote or not
I found the article you've mentioned very informative. Thank you.
I have a solution, but why can't I answer then my question?
Better still to use $\\TeX$! Though on second glance, I may be stretching things a bit too much with this array...Anyway, I'd recommend [penalized regression](http://stats.stackexchange.com/a/86959/32036) for an ordinal polytomous predictor like income bracket.
A separate question: how is BIDB different to orthogonal latin hypercubes? Seems like basically the same thing (although perhaps coming from different angles). Also, what is QMC?
Thanks Jeromy. LibreOffice also uses `AVERAGE()`, but graphs have the option to "insert mean value line".
Welcome to the site. I added the "homework" tag because this seems like homework. If that is not right, feel free to remove it
Thanks a lot, I figured it out now.
There is nothing that needs to be done to redo regulatory approval for the sake of switching to R.
Thanks for your answer. I ended up excluding 0's and 1'.
This question appears to be off-topic because it is about how to use R, without a [reproducible example](http://stackoverflow.com/q/5963269/1217536).
I put in an example but they almost always give different numbers with large enough numbers of test items (i.e. fine enough measurement resolution). They may tend to come out close, but not the same.
Please read the [tag wiki guidelines](http://stats.stackexchange.com/tags/self-study/info) for `self-study` questions, then add the tag (I removed the tag I guessed to be least important so you can fit it in) and modify your question as described there.
@love I'm glad you have looked at our FAQ. The question explicitly signals it is homework in the first and last sentences.  The record shows that two (discreet and very polite) suggestions were made for you not to display the answer, which you acknowledged but did not act on. In light of this, making more suggestions and waiting for a change in your reply to occur was not a viable option.  (It looks like the downvote finally got your attention, though, so it served its purpose.)  Consider the deletion of the reply to be a favor to you to prevent any further downvotes.
R+Mondrian or ggobi is also nice
AFAIK the Fisher transform equals the [inverse hyperbolic tangent](http://docs.scipy.org/doc/numpy/reference/generated/numpy.arctanh.html), so just use that.
@whuber The column FreqDev stands for Frequency Deviation. Basically what I have there are the z-scores of each word (That is why there are negative values there), and thus represents how much the frequency of each words deviates from the mean frequency of the whole list. What I want is to control for frequency both within lists (by removing all entries which deviate more than .8 z-scores from the mean in either direction) and between lists. The later is what I don't know how to do, as I have to compare more than two means, and so a t.test is not feasible.
This is a harder way to solve it than the recursive idea listed above, but it's extremely useful to see both approaches laid out together. Most people don't appreciate the way that stopping time methods can be used for small, practical problems too.
Simultaneous cross-post to [CS.SE](http://stackoverflow.com/q/21992241/859536) and [cstheory.SE](http://cstheory.stackexchange.com/q/21237/1037).
If its some "topsecretarea51project", than you should use allegory :)
Hi there, when you say time series analysis do you mean an analysis like `ARIMA` or do you mean something like a regression with a lag factor?
Thanks for this. I agree with your general stance: (a)  discussion of the merits of different approaches and (b) discussion of how we should discuss those are both fair game. Comments on individuals or even groups are in contrast not usually helpful (although I too sometimes cross the line in exasperation...).
@AndyW Well, in the end, not only the ratio between second and first eigenvalues worked well, but also the absolute value of the second eigenvalue was quite helpful since it gave me the greatness of the error perpendicular to the direction of the line in meters :)
Thank you very much for the answer.
@Procrastinator - I'm thinking you're going to have to answer this question despite your best efforts to do it via comments :)
These analyses (except for the boxplots) are straightforward to do in Excel, too: but any statistical package is easier, faster, more reliable, and offers many more analytical capabilities.
@Dan 1000 observations should be large enough to detect small differences in means.  It also might be able to call small differences in estimated correlations statistically significant.
@K-1: The highest frequency is indeed linked to the total length of the time serie, whereas the smallest frequency is linked to the sampling frequency. No surprise here. About your second question, how do you define "turbulent fluctuations"? What is your defnition of noise? Without that you cannot separate the two parts of the time serie.
Here's an old answer   http://stats.stackexchange.com/questions/6513/predicting-daily-electricity-load-fitting-time-series/6539#6539
The m_i's do not have to be equal

Wikipedia has a simplified description of the model.
use the robustbase package instead. Its more more recent and contains the information (t statistics) you seek.
If you're formally trying to test them, maybe. If you're just saying 'where were the big ones?' then probably not.
There's an infinite number of distributions that can be positive and negative and are both skew and have excess kurtosis. Why do you need to fit a distribution at all? You want to compute MAP or ML estimates of what? What's the underlying point of all this?
Thank you Emre for your suggestion! 
I have updated the problem - I think it may be simpler now so please have a look.
Difficult to say anything more concrete without more details I think. I'll have to pass and let someone more knowledgeable try to help if, as I suspect, the metric you are using is the semimetric over tree equivalence classes described in http://www.sciencedirect.com/science/article/pii/S0306437910001353 adapted in some way to text data.
@Wayne the last thing you said, although knowing the true value is not so important (it helps though), just how much error each system is off by.
... and if your estimates of MSerror change considerably it probably means you violated the homogeneity of variances assumption anyway.
Sorry, "big deal" is too much of an editorial comment. The question of whether it's a "bigger deal" than I'm making it out to be is basically a question of whether there are additional considerations of which I may be ignorant.
Further to your comment on sample size: Good, in his *Resampling Methods* (3rd ed., 2006, p. 19) notes that the bootstrap may be unstable for sample sizes $n<100$. Unfortunately, I don't have the book at hand, so I can't look up his argumentation or any references.
When you say that you want to do the regression for the first level of the factor, what exactly do you mean? It seems that you mean that you want to use the subset of the data, but then it does not make sense to include the factor in the formula.
To clarify - each patient was asked the three questions once, and you're interested in whether $p(\\text{Yes})$ increased over time?
@Charles You could choose any simple nonlinear function you liked. You could also assign any numeric scores that make substantive sense, or use a variety of more systematic methods to assign scores based on the relative frequencies of the ordinal variable.
Why not just compute $U\\Sigma V^*$, change the chosen diagonal elements of $\\Sigma$ to $\\Sigma'$, and reconstruct the perturbation as $U\\Sigma' V^*$?
What do you precisely mean by "bias"? Note that this is an statitical [concept](http://en.wikipedia.org/wiki/Bias_of_an_estimator) and if you estimate $\\mu_x-\\mu_y$ with $\\bar{x}-\\bar{y}$ then the bias is $0$.
@Andrew BTW you could learn more by asking similar question on [Physics.SE](http://physics.stackexchange.com/).
Great pointer @Scortchi - Thanks! What if I have **some** rate of replacement? It looks like the hypergeometric distribution assumes **no** replacement, and the binomial assumes **replacement**, e.g. with a probability of replacement = `q`
The 1st constraint is just obvious. Stop agglomeration when the farthest distance - the coefficient of the `complete` method - reaches the threshold. However, I don't know any specific program having this option. Your 2nd constraint is a bit misty to me. Can you express it more detailed?
@chl your 12th link is leading to another location than the article you name.  I don't see a free download for the named article, though.
I am using the 1983 edition. The sections of interest are 4.3.2 (asymptotic theory for grouped data) and the following section 4.3.3 on ungrouped. Chapter 4 is about binary data, which is where this is.
Very interesting. Normally I care alot about such things, but this one i have never thought of before.
@JohnnyB you could use the `fourier` function to create a matrix of Fourier terms, and include them as external regressors in your forecasting functions.  If your function doesn't support external regressors, I suppose you could try some kind of 2-step process.
according to [wikipedia page](http://en.wikipedia.org/wiki/Zipf's_law) Zipf's law has two parameters. Number of the elements $N$ and $s$ the exponent. What is $N$ in your case, 10? And frequencies can be calculated by dividing your supplied values by the sum of all the supplied values?
@ttnphns Ah yes I think that is what I was looking for thanks. In my googling, I have come across different terminology that may mean the same thing: PCA 'loadings', PCA 'scores', etc. What is the proper terminology for what you just described? Thanks!
There are many dedicated texts that give you the details. At the opposite extreme is this one paragraph. Data come in different forms (e.g. counts can only be zero or positive integers, some measurements can be positive only, some are less restricted). Also, data come in different shapes: experience with data shows that. So, statisticians and others have proposed many different models, some with mathematical derivations and underlying ideas about generating processes, and others just proposed more or less empirically as shapes that might be fair fits for at least some distributions.
That is what I thought. Thanks!
*My regressions are actually mixed models, which complicates things* -- that's rather an understatement. Why would you leave that critical information out? Indeed, the first sentence of your question directly states '**simple linear regression**'. Why would my answer to the question you asked apply to a very different situation? It's not a trivial change in the circumstances and I don't believe I now understand clearly what simulation procedure you're asking about. This is an entirely new question.
Multiple items scales are standard practice in psychology, analyzing them with factor analysis (and confusing PCA for a factor analysis technique) too.  Of course, we could have a discussion on the merits of this approach but it's easy to see where the OP is coming from and not too helpful to say “you should just ignore all this and use single items”.
And if you incorporate step-halving into the algorithms they usually work well.
What is wrong with the explanation on wikipedia at http://en.wikipedia.org/wiki/Kolmogorov_Smirnov?  Is the question "what is the KS statistic" or "how can it be used in the predictive modelling context"?  If the latter we probably need more information.
+1 I prefer Tukey's smoothers and am glad to see an example of one show up here.
Personally, I don't think GP regression belongs to Bayesian inference, since it doesn't follow the steps in Bayesian approach. The so-called predictive distribution in GP is derived by jointing the training and test data **in the prior**, and then conditioning on the training data, it doesn't make use of likelihood or posterior.
Thank you for the lens analogy. This is the perfect way of explaining exactly why GEE is good at handling *unmeasured* sources of variation, despite that you get improved efficiency if you correctly identify such variation.
Do you have AMOS? You could test this in AMOS (or another SEM program).
Machine learning and pattern recognition are not the same thing, machine learning is also interested in things like regression and causal inference etc.  Pattern recognition is only one of the problems of interest in machine learning.  Most of the machine learning people I know are in computer science departments.
Can you be clearer about what you don't understand & need to know? This appears to be instructions for generating simulated data for a problem or simulation study of some sort. Is this an assignment for a class you are taking? If so, please add the `[self-study]` and read its wiki. Are you supposed to actually do this? What software do you use? (Nb, asking for code is off-topic, but it might be helpful to know nevertheless.)
How many observations do you have? This could mean there is not enough information in your data to fit a model with many covariates, try first with simpler models! Also, why do you use mlogit? Did you try glm, in base R?
I don't think it is possible for the $R^2$ to change.  You didn't state the reason for normalization.  I can't think of any.
+1 to this - if its an entirely connected component, that's one thing, but if you can decompose the network, you have both smaller data, and actually *several* independent networks that can be analyzed in parallel.
That does help, thank you. Your outline is essentially the protocol I've been following - however Zuur et al. 'Mixed Effects Models and Extensions in Ecology with R' (2009) seem to recommend fitting all fixed effects first, so that you avoid a scenario where "part of the information that we want to have in the fixed effects ended
up in the random effects". This has me puzzling...
Thanks a lot for this hint, it was quite helpful. Also I had seen the paper linked on wikipedia, it did not look trivial though. I have one question regarding this property: We do have biological data where the data is a set of measured lenghts, and these lengths have a lognormal distribution. Knowing that in biology cost-minimization is crucial, I wondered if the Maximum Entropy property could be linked to cost-reduction? Sorry if the question is vague! P.
@Glen_b Yes, that's right.
CV using full set for model selection, huh? It's a common error (still even Wikipedia mentions it), because it is a hidden overfit. You need to make a higher level CV or leave some test to do this right.
The example is interesting because it occurs all the time.  There are well-known transformations to deal with it, such as Tukey's "folded" power transformations.  These make little change in the middle of the scale yet cure skewness at both ends.  I have found that folded roots and logs work *very* well for standardized pre/post test comparisons.
The question seems more suitable for [stats.se].
-1; misleading.  OP didn't enter his variables incorrectly and multicollinearity may not be an issue.  A correlation between the raw fixed effects might speak to this point, but Simpson's paradox may allow that approach to lead you in the wrong direction.
I am doing it now, feeding the optimization problem to a linear solver. However I thought, may be there are more computationally efficient solutions.
The help for the function (`?wilcox.test`) makes it clear that the W that is calculated there is the same as the most common definition of U. What's your definition for U and why do you think it differs from the W there?
No problem @ChrisBeeley. By the way, I wanted to mention a small correction to something I wrote in my post. I said that the minimum value of $t_{paired}$ occurred when $r=0$. But technically, theoretically, $r$ could go down to as low as -1 and this would actually be the minimum value of the function across the range of $r$. However, in real life it would be exceedingly rare and strange for the time1 and time2 scores to have a negative correlation. It is even unlikely for the correlation to be 0. There is most likely a modest positive correlation. So I think my reasoning above is still sound.
That's an interesting idea. However, that will result in three percentages. If I want to extend this to more than {A,B,C} then the number of measures will increase. What I'm interested in is a measure for each Cx or a measure for each set of participants (which may end up as more than 3).
I agree with pgericson that you had to eliminate or impute lower values to the spikes in order to get a fit with a maximum at 100.  I am not completely clear about your exact approach.  But if you think the spikes are errors that should be imputed or deleted you should have a firm rational basis for modifying your data.  If you impute values to the spikes what is your imputation method and how do you justify it.

You need to explain precisely what you are doing and supply your rationale before we can intelligently evaluate what you have done or even to be able to suggest a better approach.
@mpiktas Yes, I'm interested in $Var(\\bar\\tau)$. And both an explicit formula or some numerical algorithm would be fine. What I have is a time series $A(t)$, where I compute the time between two thresholds $A_k$ and $A_0$, thats what I mean by $(t_{k,i,2}−t_{k,i,1})$, $i$ meaning the i-th time this two-threshold-crossing occurs for a given $k$. In my case that's protein folding dynamics. I checked for stationarity in R by adf.test and actually, my time series are stationary. What other properties do you think of? Thanks for your efforts.
The expected value does not mean the value that you expect to see on the next roll of the die; it is what you would expect to be the average value of the outcomes on a large number of rolls.  Roll the die $100$ times. You are **likely** to see a **total** of $350$ dots for an average of $350/100 = 3.5 = E[X]$. This is **not guaranteed**, you might get some number smaller than $350$ (in fact the minimum is $100$, right?) or larger (as much as $600$), but this is quite unlikely. Betting that you will see **roughly** $350$ dots on $100$ rolls is more profitable than betting on roughly $12$.
Re your comments about wrongness and usefulness of models: It is striking that you report an eight-parameter model for a (mathematically and statistically simple) process that was generated with only three parameters. It would be natural to characterize that as overfitting. How would you respond?
This is a famous, and much discussed [problem in probability](http://en.wikipedia.org/wiki/Monty_Hall_problem). If you search using the name 'Monty Hall problem' you can find discussion of the problem here on CV and on math.SE and vastly more on it elsewhere. It's readily solved by considering the appropriate probability tree, or if you're even reasonably adept at manipulating conditional probability, you can do the calculations directly (NB: the host knows where the prize is & always reveals a goat). If neither of those yield insight, simulation is a very useful tool for exploring the problem
Thanks very much for clarifying that, i really appreciate it. Ive had some issues with this for a while + the difference between sample and population.
That's great, @B_Miner, that's the kind of thing you want to do. Moreover, it's the reason I think the simulation-based approach is superior to analytical software that just spits out a number (R has this also, the `pwr` package). This approach gives you the opportunity to think much more clearly (&/or refine your thinking) about what you expect to happen, how you would deal w/ that, etc. NB, however, that you do need the quadratic terms, or something analogous, if your posited rates are right, b/c they are not linear, & the interaction alone doesn't let you capture curvilinear relationships.
Could you provide an bit of the data? I think it may make it clearer what you are working with.
Critical errors in Excel are well described.  See for example http://biostat.mc.vanderbilt.edu/ExcelProblems.  It's difficult to see why Excel would be used when there are much better free resources around (e.g., R).