tttjjj/elt_bench huggingface dataset:
That is very instructive, and you can also use a model such as an ordinal response model on continuous $Y$ and plot the predicted mean vs. the predicted median from that model, is ordinal models do not force a linear relationship between the two as OLS on $Y$ or $\\log(Y)$ (on the anti-log scale) do.  Ordinal models, which are semi-parametric (the most popular being the proportional odds and proportional hazards models) are more flexible.
Thanks Jeromy.  Just read the Wikipedia entry for MDS -- seems like it could lead somewhere.
@ebony1: It is true that the situation changes when P >> N.  That's an interesting refinement of your original problem, because in effect any model is going to be a gross overfitting: more explanatory variables than data.  That does not augur well for the prospects of obtaining stable results via subsampling.  At a minimum, it points to the importance of retaining as much data as possible.
I wonder if there was a miscommunication. It is well understood that the bootstrap works better in the middle of a distribution than at the tails. Thus, eg, bootstrapping the median would be the *most* robust quantile, whereas bootstrapping the min or max necessarily fails. You may find @cardinal's answer [here](http://stats.stackexchange.com/questions/9664/what-are-examples-where-a-naive-bootstrap-fails/9722#9722) to be of interest.
(1) not sure I have great ideas (2) would consider modeling as change from baseline. Recode readings as percentage change from first reading. This "flattens" out the data and you looks pattern of change but easier and likely fairly good. Just need to be careful how you code time.
There are at least three other questions that cover this or very similar topics, http://stats.stackexchange.com/q/3331/1036 , http://stats.stackexchange.com/q/2777/1036 , http://stats.stackexchange.com/q/3238/1036 .
@Glen_b: This is not a homework in any case.
How is *what* solved ? 

$2T*n^T$ what?
I did do it and it was strange, but that doesn't explain me or prove to me what the difference between those quantities is. It's odd when $n_i = 1$, but how do I know thats not a special case? (hope I'm not being annoying, I am just trying to genuinely learn)
Also see [here](http://stats.stackexchange.com/questions/79855/is-p-5-0-significant) for another duplicate.
Welcome to the site, @wcairns. Would you mind expanding a little bit on what the Granger test is & why it's the right answer here?
See discussion at http://stats.stackexchange.com/questions/56194/is-it-possible-to-use-a-two-sample-t-test-here/56211#56211
@alcor: We don't currently have rules for speech slots. Monitoring the available slots is the first step towards figuring out what rules we need.
And this is why it it important probably to first describe the "in between" and "within" variation of your predictors, exactely as she does: http://m.youtube.com/watch?v=aUVZWnVnjxs Thx a lot for your help!
Could you tell us what a "Phi measurement" is and what "MOA" might stand for?
+1 The hyper-parameter can also be integrated out analytically as well, so there is no real need for cross-validation for many applications, see e.g. http://theoval.cmp.uea.ac.uk/publications/pdf/nips2006a.pdf and http://bioinformatics.oxfordjournals.org/content/22/19/2348.full.pdf .
If Z1 is valid only if X2 and X3 are conditioned for, this doesn't really change anything. This is the usual conditional independence assumption. For a continuous endogenous variable you interpret the first stage coefficients like in any other OLS regression (because that's what it is at the end of the day).
Monthly seasonality of daily data still needs only 11 dummies. E.g. in January all dummies would be zeros every day, in February the first dummy would be 1, and others zeros etc.
(-1) Can somebody revert the question back?  @AvanishGiri - deleting the body of your question is **horrible** practice and is sure to get you downvotes.  Think about the purpose of a question and answer site - deleting your question is counter productive.
@whuber Indeed, I have a mixture of Binomials, and it seems sometimes the Poisson approximation is not very good. My question is what are my options for getting a p-value for an actual observation. One way would be to run many simulations and just get an empirical p-value. This is the simplest way I know, but it uses nothing of the info I have -- I do now it's a mixture of Binomials and I know all their p's -- isn't there a way to use this valuable information?
I do not understand what you mean by "B outperforms A, i.e. it puts more weight on positive points". First, you never use B as an algorithm, only as a target distribution. Second, your empirical averages are converging to the means of the distributions $P_S$ and $P_T$, at the same $\\sqrt{N}$ speed. Same performances.
Pepto, your question in the comment there goes far beyond clarification, to a substantive additional question. It's a bit much to ask Elvis for one-on-one responses; you should instead address such questions to the whole site - that is, you should either edit your original question to reflect what you meant to post to begin with, or post a new question (with, if necessary, a link back to this one).
Thanks Peter.  I had seen this formula but didn't delve into it as I thought it was dependent on assuming normality.
I assumed this question was about the increased likelihood that even a negligibly small or spurious effect could achieve significance in a large data problem, but you might be right to address the separate issue of [multiple comparisons](http://stats.stackexchange.com/q/28712/32036), depending on whether the OP wants to control for this kind of error inflation. See also [an issue with multiple comparisons corrections](http://stats.stackexchange.com/q/1458/32036).
In SPSS see the [NPTESTS](http://publib.boulder.ibm.com/infocenter/spssstat/v20r0m0/index.jsp?topic=%2Fcom.ibm.spss.statistics.help%2Fsyn_nonparametric.htm) procedure to calculate the Wilcoxon signed rank test for a pre-specified median.
It looks like `ff<-(1-cdf(df$x))/df$x` is calculating $P(X>x)/x$ not $P(X>x)/f(x)$
**My guess**: If the sample follows a Gaussian distribution, then the *[asymptotic relative efficiency](http://en.wikipedia.org/wiki/Efficiency_(statistics))* of the robust estimator in 95%.
Thank you.  I actually read your response here http://stats.stackexchange.com/questions/67607/how-to-translate-the-output-from-an-lm-fit-with-a-cubic-spline-into-a-regressi before posting.  I guess I just need a better grasp of what I can do with rms.
Do you have any examples to show this? I have looked extensively and have found no such example.
Thanks for your help. Mission accomplished.
@suncoolsu - Ah yes; no worries; and please just "Karl"
@amoeba "Dummy coding" is for the matricial notation, which is precisely what is avoided by the vector space language. I have edited my post at the end, to provide more details on the orthogonal decompoisition when there is an interaction.
This doesn't answer the particular question, which concerned using the samples from a Markov chain to construct a valid *density estimator* of the posterior. The point that the standard error of our estimate of a linear functional is higher than a naive estimate based on independence is well appreciated, but OP still wouldn't know on the basis of this answer if it is a good idea to construct a density estimator using (say) kernel-smoothing (which even under iid sampling wouldn't converge at a $\\sqrt n$ rate).
@Thomas thanks for the help, so do I have to add any tag anywhere this question of your flagging is enough for me to get a reply? Please let me know.
@whuber the challenge with using logs is that these are small startups, so a lot of the sales numbers are zero and a lot of them fluctuate wildly as a ratio, 100 one month, -100 the next month. I can't just throw this data out because it's a very large subset and I need to track the whole field.
Some plant extracts are used to dye hair, and eyes have their neat site on face - so, the answer is yes.
Ties are not as awful as they might seem to you. Well, if you are scared, add tiny random fractional noise to your values, to put away ties. You'll see that this doesn't affect Spearman greatly.
@onestop: the data is in the R package alr3 as BGSall BGSgirls BGSboys (see http://rgm2.lab.nig.ac.jp/RGM2/func.php?rd_id=alr3:BGSall) and "soma" seems to be "Somatotype, a 1 to 7 scale of body type" developed by William Herbert Sheldon (described in http://en.wikipedia.org/wiki/Somatotype_and_constitutional_psychology)
I have samples for both (capacity & load) sets of independent variables. I have derived analytic expressions (probability density functions) for both sets. I need to analytically derive The surplus/deficiet
If you can build an even halfway reliable system to predict future prices, you should be able to become rich by trading futures!
Secondly, Epi Info is painfully slow and its giving me some hard time whether to place confidence in it for a such a huge data especially when merging will be required and when reimporting of the data will be required after editing it in excel  (geocoding and other cleaning discrepancies). Personally latest version has a very good Dashboard for data analysis, but many among us are in favour of SPSS. @Jonathan
Are you testing a difference of means?
@NilaniAlgiriyage Yes, issue to identify R indeed!
@whuber: Done, has it become a bit more clear now?
That is what I'm trying to do, 15-20 items for all participants. but the thing is, a participant does not have access to all the items at once but rates only a fraction of all items and only once. the rest of the items are rated by other participants. so I am concerned that this could introduce some sort of bias to their ratings or comprimise the validity because certain number of items in the pool are rated by different people. I hope it is clear since my knowldge of statistics is poor...
I took the liberty of editing to reflect that.
If you are trying to circumvent the goodness of fit test, you can do whatever you like. But you'd be cheating on yourself.
How it looks visually isn't as important as the content and the rules that should be applied to a question. For instance, if question 1 should skip to question 5 if answers A or B are selected on question 1
Without reference to the number of parameters you try to estimate, or equivalently the kind of model your are working with, it seems rather difficult to give you a clear answer.
+1 very nice. I had my own explanation for the terms, based on R packages, but you nailed it much more concretely.
might also be relevant to know that if I do not use instruments, the hausman test clear prefers the FE model
Thank you for replying, sir! I've finally been told that indeed the amount of the training data is very important to the `CNN`. Thus, a common approach is to preprocess the limited data samples to simulate more data samples. But now I have no much time to do that work. Moreover, the batch_size of SGD also has effects in the training procedure, larger the batch_size, smoother the training loss. Additionally, I mainly use the open source `caffe` to train the `CNN`. But, I'm not sure that what's the bug of backpropagation as you have presented. Could you please give more details? Thanks!
Thanks! I read the paper. Do I understand correctly that permutation test applied to a location test doesn't test the null of location well, but rather tests whether the distributions of the two groups are the same in the sense of their locations?
@Srikant Thanks again for your response, especially for the question (I was too focused on statistical properties and hence lost the sight of the aim). First I was afraid that I am totally wrong, because it seems that hardly anyone is discussing this issue (at least on the web). Now I am afraid that most AB-Testers do not ask even this question.
re the edit. Consider the estimator $\\hat{k}$ = $(\\arg\\min_k\\ S_k + \\arg\\max_k\\ \\sum_{i=k+1}^n y_i)/2.$ Clearly it is unbiased: it is invariant under simultaneous negation of the $x_i$ and $y_i$, whence so is its expectation, and the only number invariant under negation is $0$.  This gives some potentially useful bounds on the bias. Saying any more about it would require being more specific about the class of $f$ under consideration.
Thanks a lot for this very useful answer. Indeed, my problem of poor estimation of incidence rates is handled more gracefully by Bayesian type methods. And maybe that is all I need as a practical solution.
However, my question is more conceptual: under what circumstances/assumptions would choice 2 be more sensible than choice 1? In other words, what settings lead naturally to a Fisher test type (fixed marginals) and when ist it  more appropriate to allow for extra variation due to fluctuating total counts?
Please be aware that p-values and statistical significance differ greatly from classification accuracy.  See, for instance, http://stats.stackexchange.com/questions/31.  So, your cross-validation accuracy is 81%.  Whether that's "good" or "bad" depends on your application and your objectives, about which you have told us little.  Your question therefore does not seem to be answerable--and even if it were, it would likely come down to a matter of opinion. Could you edit it to clarify these points?
@Glen_b: the noise has mean 0. I tend to assume it's normal. Also, I_i represents a number iid random variables that get summed together before recording the observation--each with its own noise--so the variance of the noise is proportional to I_i. Regarding I, these are what I really want to know. The problem is to infer the vector I. So, I tend to think of them as fixed constants, but that's the kind of modeling choice I would really like to have some guidance from the stats literature on, and I really don't know where to look in this case.
I'm rather fond of PiCloud - I've found them remarkably easy to get up and running.
Please clarify: Your question uses "X", but it seems there are two distinct X matrices. The first X is a 5x300 matrix mentioned in the first paragraph. But the second X, in your regression pseudo-equation, is a matrix of the 8 predictor values, different from the first X, correct? (Otherwise, the question seems kind of screwy.)
@Memming This would lead to $E=y^t(1-y)^{1-t}$. But then wouldn't that be the same as using the absolute distance as error $E = t-y$ and defining the fitness as I did above: $f(y) = E_{\\text{max}} - E(y)$. Because $t$ can only be $0$ or $1$, one of $y^{1-t}$ and $y^t$ will equal $1$, and for the other one the exponent is $1$, leaving $f=y$ or $f=(1-y)$ depending on the value of $t$.
I aspire to one day write an answer that Dilip approves of. (:
It's a long time since I last opened it so I don't know if it will answer your question but the boot package is based in particular on the methods detailed in Davison, A. C., & Hinkley, D. V. (1997). *Bootstrap methods and their application.* Cambridge: Cambridge University Press. (The reference is also cited in the help file for the *boot* package.)
(+1) Note you can use any rate for the simulated distributions so may as well skip `rate=1/mean(data)` in the 2nd line (not in the 3rd!). & can save `simp` for re-use with any data set having the same sample size.
I think you  can plot them just as they are
@John, I read those answers but missed where it discusses AIC and non-nested.  I'm pretty sure it's fine, but can you give me a more precise pointer to that point in the answers?
what is the difference between "priority" and "importance" - they sound very similar to me.  If storage capacity is a priority for me doesn't that also mean it is important?  I don't know how to interpret the respondent who says storage capacity is their lowest priority but is also "very important!"
@whuber See the edit, if that helps at all?
Another point in favor of the second layout is that it is consistent with good database practices. The data normalization advantages of databases and a knowledge of SQL is very useful if used appropriately.
... and yet it is commonly done.  One thing to point out, and yes this is a little pedantic, if you are using a single Likert-type item that isn't a Likert scale.  The difference is meaningful (though the question asker is talking about a Likert item and the ordinality is an issue). A Likert scale is a consequence of summing or averaging several Likert items.  This approach was developed specifically to offset the extent to which the ordinal data was actually ordinal and make it more reasonable to be treated as being on the interval scale.
@mbq thanks for having the patience to correct my lazy typing.
Thanks, Alex. I see a problem with this though: what if over the loops the solutions generated never satisfy the constraint? That could happen if k means were set to run with no cluster size constraint.  I'd love a solution that avoids this. (The nature of the application is such that I really need to ensure clusters are of a minimum size.)
@Andrew the method should work for either case. Since no hidden layers is a linear classifier, it is trivial to do; so I am more interested in the non-trivial case; assume we have 2+ hidden layers (although the method should also work for fewer, since it is easier).
I am **not** voting to close. Consider this interpretation: "What are the major qualitative properties of time series that one looks at when commencing any analysis/forecast/etc?" Surely our time-series experts would be able to rattle off a half dozen or so characteristics and (I hope) most would agree on them. Offhand, I imagine these would include trend, seasonality, sign of serial correlation, heteroscedasticity, outliers, and any suggestion of changepoints. (It's unclear whether stationarity can be qualitatively apprehended.) A discussion of these could be a useful guide for non-experts.
Another option is to look at the responses and check for any inconsistencies. E.g. they responded "1" to "how many children", but "3" to "how many girls". Often people introduce this kind of redundancy on purpose to weed out cheaters. But if you don't have this redundancy, it will be much more difficult (although not impossible).
And if the groups are matched on certain characteristics, e.g. age, then there's no point adding those characteristics in as covariates. This tends to be an issue with case-control type studies, where one doesn't want to match on too many characteristics.
Can you fit the cubic and then ask whether the coefficients of the cubic and quadratic terms are significantly different from zero?
Yes but read a book first. Gelman's Bayesian Data Analysis is the canonical one, although Kruschke's doing bayesian data analysis has a lower barrier to entry (although perhaps not as low as the puppies on the covers suggest).
Not really anything to do with stats... pick any 'nerdy' profession, and it fits...
You might be interested in [Logistic Regression in Rare Events Data](http://gking.harvard.edu/category/research-interests/methods/rare-events).
Practical significance is always somewhat subjective... It seems that the best that you can do here is to compare the value of the coefficient c for your actual data with the mean value of c over permuted data. Depending on how you designed c, this may give you a useful notion of practical significance.
For the same reasons that regularized regression often outperforms least squares out of sample. I think that averaging ridge coefficients should work after applying some rule of thumb to the lasso results for feature selection (i.e., intersection, union, or some other threshold for min % of non-zero coefficient selection per feature)
Can you expand on this a bit more? How do these authors solve the OP's problem? Right now this is just a reference which is only of limited use without a bit of explanation.
Why not utilize an approximate standard error for the R-squared value on the hold out. For example using CI.Rsq in the psychometric package?
@CuriousCat, Be curious and search this site for "pca biplot". There is also tags `pca` and `biplot`.
@Fred Ignore the probabilities: work entirely in logits.  That will bring you closer to familiar OLS and ML calculations and lets you avoid all that stuff with logarithms and exponentials.  All that's required is to convert your threshold of 0.75 into its logit in order to make comparisons.
Your likelihood is incorrect: it will be zero whenever $\\theta$ is less than the largest $X_i$.
Greg, if I understand my poking, it seems that you are using the base loess to fit the original line, but then create an estimate per point according to the user's action (clicking) on the plot.  It will need some tweaking, but it is a good starting point - thank you very much.
I think this response most directly relates to the question in the title of the post, "How to handle count data (categorical data), when it has been converted to a rate?"
You don't say exactly how these variables were measured, but if it was something like "On a scale of 0 to 10, say how satisfied...." then your variables are ordinal and thus categorical in the wide sense. If your variables represented e.g. counts of separate items, then they could be called counted fractions and there might be some division of opinion on whether they are ordinal or counted. Either way, ordinal is the keyword to take forward.
`R`'s formula is further analyzed and explained at http://stats.stackexchange.com/questions/81754/understanding-this-acf-output/81764#81764.
A large sample size does not mean you can assume a Normal distribution. Your data seem to be % ages. If they have a lower limit of 0 (as % often do) then they are not normally distributed.
Am I correct that you're using R? What package / function are you using? Why do you need to compare these, is this a homework assignment?
Can you explain more? I do not understand what a "butterfly effect" is.
I asked how I shall approximate $P(a,b,c)$. I can do this by counting how many events where $A=a,B=b,C=c$ are in my DataBase, or I could try to fit a distribution to get this information. Up until now, I am just counting the events.
Welcome to the site, @MarcoTulio. I have edited your answer to make the English smoother; please make sure it still says what you want. I also deleted your signature information. Notice that your username & avatar are automatically added to your answer. If you want to make more information about yourself available to CV users, add that information on your user page. Since you're new here, you should read our [FAQ](http://stats.stackexchange.com/faq), which covers topics like these.
I was following this book: New Introduction to Multiple Time Series Analysis
By Lütkepohl, Helmut
Thanks for editting in the links whuber - beat me to it!
This is true, as far as it goes, but it ignores the non-linear transform (exp) in the problem statement.
Thank you for your answer. This part of the project is supposed to look at expectations and attitudes towards the virtual patient. The likert item mentioned was an example. Another example item, was "I am going to experience this exercise as realistic" with post-item "I experienced this exercise to be realistic". In this stage I will not look at actual changes in confidence, skills or attitudes (I have although material to do so in my next paper, where I will look at confidence, with pre- and post items that are the same, exactly as you proposed).
Are you asking about estimating central tendency from your sample by taking the mean or median of responses to a single item from all respondents? @John: which end is 1 may be arbitrary on the part of the person who designs the scale, but it may affect conclusions. See [my answer here](http://stats.stackexchange.com/a/96880/32036).
@drknexus - So, multiple items serve as a measurement triangulation for construct scales?  If yes, what are the criteria for determining that a researcher has enough relevant data points (i.e., items) to use the scale as an interval measurement?