tttjjj/elt_bench huggingface dataset:
Also, if you are looking for modelling flexibility, then definitely go for WINBUGS/JAGS. In R, you can only do what they prepared for you, but in bayesian you are absolutely free to do anything (for the price of learning time, and you need to understand the models very well).
+1.  Also, the use of "with 90% confidence" reveals a misunderstanding of the way the concept of confidence could apply in this case.  Without a basis for a degree of confidence, there's no systematic way to quantify the level of confidence one might have. It would come down to an arbitrary thing, as if one were to say "I'm x% confident that this soup is too salty."
Ben: yes, the .1573 is asymptotic (based on normal; it will only be approximately correct). I think you could compute the corresponding p-value to stepAIC for a $t$, since it will depend only on the degrees of freedom in your problem (e.g. I think it's 0.1579 for 200 df); as a result you should be able to back out the desired $k$. @Nick That's most interesting. On a first glance I don't think there's a direct connection between the calculations -- they're computing the same quantity, but for different reasons.
What does it tell you, or in other words what are your conclusions about those results? (I'm still having trouble with understanding your original question.)
Yes you do have the index of the $y$ points, it's just not *telling* you that you do.
I hope that the edits I made to the code will allow people to read it better. I need help with this question badly.
@JeromyAnglim: That's exactly what I was looking for
"Surely no-one cares"... If an HR drone is told to look for a "statistician", I'm not sure that a Math degree (with some kind of note that it's a Statistics "concentration" or "track") will be as obvious of a fit. That said, you'd never want to compromise your education on that off-chance, so I'd only consider this if I otherwise couldn't decide which program to choose. After I'd decided which institution was located near better restaurants.
+1, there's a lot of good info here. Another possibility might be to fit a spline to the data. I discuss splines [here](http://stats.stackexchange.com/questions/5015/what-are-the-advantages-disadvantages-of-using-splines-smoothed-splines-and/25081#25081).
What _is_ your variable of interest? And what probabilities do you wish to find?
Can you amplify that a bit? I don't recall an assumption of "no ceiling effects" with respect to OLS. Do you mean to say that a very skew dependent variable likely means a non-linear functional relationship between the dependent variable and the predictor? Just trying to understand your terms and meaning.
Now, I find package "drc" can implement it. for example, drm(data\\$a~data\\$b, fct=l4(), type="continuous"). But I don't know how to get the $R^2$ value and the regression fomula $F(x)$.
@ Stéphane You statement describe my problem more clearly. Thank you very much. The question seems difficult for me now, and I will read more literatures on Poisson process approximation and Stein's method.
Hi Peter Flom, thanks for reply!
Thanks! Yes, actually there are various covariates, not only one IV, I just mentioned one IV in the question for simplicity. The thing is that there isn't a strong theory that could support interaction between gender and certain covariates, it is exploratory analysis, so I need to experiment with many interactions and model fits; the initial model contains 30 predictors...
The only exception would be if some of your values are zero or negative. The logarithm is not defined for zero or for negative numbers. So in this crazy case, you'd lose some data in the transform, so would get different results.
Thanks for the pointers to optimal scaling.
Another (active and interesting) stackoverflow question on this theme: http://stackoverflow.com/questions/10657503/find-running-median-from-a-stream-of-integers
Hi I did this, but does it violate one of the assumptions of regression?  ie the points are independent observations?  the residual plot looks okay.
As whuber is I think alluding to, measuring % of visits accounted for by high utilisers actually reduces you to one observation, the percentage.
I'm confused. Did you mean to write $|Q_r(d)| = r$ instead of $= d$?
Building on the previous nice answers: yes, there is a way that is position-aware. I think it is possible by means of nth-order Markov models.
Even so, it is worth noting that the average absolute error in calculated SS when rounding to 3 decimal places from 20 samples of a normally distributed variable is on the order of .002*$\\sigma$ (A simulation result; some restrictions obviously apply, e.g. $SS_{error}$ obviously increases with N, but the quantity of typical interest, MSE, decreases).
Hint:  There are $\\binom{30}{10}$ different exams that the professor could set.  How many of those will have all $10$ questions selected from the $25$ that the student knows how to solve?  How many will have exactly $9$ such solvable questions? How many will have exactly $8$ such solvable questions?
I think your reasoning is good, but one conclusion may be a little too limited. Yes, when the original variables are not commensurable, the result might only reflect arbitrary choices of units of measurement of the variables. But this does not imply that using the correlation is the "only reasonable choice." In fact, in many cases that might be an *unreasonable* choice due to its sensitivity to outlying data. One might choose to standardize the variables to a unit IQR, for instance, or to use some other basis to identify appropriate units of measurement--or even to re-express data nonlinearly.
When we usually encounter a problem like this we try to just give hints and have the OP put some work into getting the solution themselves.
Hi, Thank you for the response. The two variables do not come from the same distribution. They have different means, and as a result different parameters for shape, scale and location of their respective skewed distributions. 

I just attempted using your script anyways, with the following distributions (parameters taken from my data):

library(VGAM)
#generate distributions
x=rsnorm(220,shape= 10.6852380,scale= 1.2881360,location=0.2788605 )
y=rsnorm(220,shape= 19.695530,scale= 28.453448,location= 3.316043 )
#specify r2
r= 0.6835
#generate new y values
y1=x*r+y*sqrt(1-r^2)
summary(lm(y1~x))
Thanks! Now I'm close to understand why! I've a bunch of results very close to 0 (say 1e-5), that total 15% of my data. The rest is more or less dispersed. Thus there are a lot of terms at the left tail and since the LN, Weibull and Gamma fit this part better they have a larger likelihood. Right?
I've fixed the answer so the code will now work. It is too hard to do it manually without writing a lot of code. Just use what is already provided.
-1: does *not* take new evidence into account??  This bears no resemblance to actual frequentist theory.
Thanks for your response, Greg! I think the assumption holds in my case where consumers are deciding to accept or decline a service. However, I'd like to find a way to check this assumption. Can you point me towards a source that discusses the general issue of "too many" single observations? The power is good (0.32 success rate) and I'm already getting the results I expect.
I answered under the impression that the scale was already designed and the data already collected, but this is the same reasoning I had in mind. +1 for writing it out explicitly and for the reference!
Aren't these cumulated frequencies rather than % variance accounted for by each component? If this is the case, your results suggest to consider the first two components.
Why do you need the residuals to be normal?
What do you want to test about them for different?  Does the height of the line matter, or only slope, or perhaps both?
Seems rather strained. The example uses nouns, not adjectives, & surely isn't intended to measure connotative meaning of concepts - it asks a straightforward question.
Basically, I'm looking for a way to preselect relevant input data for SVM in order to avoid spurious correlations. I've thought about Granger Causality and Random Forest.
I am not completely sure of your question, but it seems like you could be satisfied with some simple clustering algorithm like http://en.wikipedia.org/wiki/K-means_clustering
Making the bar ends much narrower and offsetting the values at each x just a tad (say by 0.1 on the x-scale as a first try)
This is more about how you define an "event." It's called "survival" but the event does not have to be lacking ability to survive. Turning the idea around, you can use 1 to represent "revival."
Very thorough explanation, thank you.
See if you can get a little more precise: "when enough data would be collected" for what? For x% confidence, perhaps? You're asking about the link between sample size and precision of estimates, which is a matter of statistical power. Confidence intervals (CI) for the number of sales (or for the % of clicks that results in sales) will, as you indicate, depend in large part on sample size. The basic formula for a CI for a proportion is given at http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval
heuristic: divide by 4--logit coeff/4 is approx pct-pt diff in prob. from 1 unit change. That's not same as saying "person w/ βsubn charcteristic = z is x% likely to." Not only (as noted) does one have to add βsub0 & subtract likelihood associated with ref class. It's also necessary to take account of predictor collinearity. B/c being southern correlates with other predictors, it won't be true that prob. of southerner voting repub is βsub0 - transformed log odds for NE plus transformed log odds for south. Better to say, "all else equal, being from south predicts x pct point change in prob of"
@cbeleites: Yes, certainly. I understood the question as asking about the variance *over folds*, not *over repetitions*. Maybe OP could clarify what he or she meant.
Welcome to the site, @GailPalubiak. Please don't sign your posts with your personal info. Note that your avatar & a link to your user page are automatically added to all your posts. You can post that info there. Since you're new here, you may want to read our [FAQ](http://stats.stackexchange.com/faq), which discusses topics like these.
**Hint**: The estimator you've given is a mean of iid random variables. Ignore the regularity conditions. How can you connect means of iid random variables to asymptotic normality?
Regardless of content, this is an advertisement not a question. Recommend you write up the idea and bundle the functions into an R package.
Thanks a lot. It's true, I never gave it much thought how one actually samples from a density function. I just implicitely assumed that this is the simple part.
Use that $p(t)/\\exp(t^2)\\to 0$ for $t\\to \\pm\\infty$ for any polynomial $p$. This ensures that both tails are integrable.
@whuber, I meant observing an interval value, similar to age-groups one can have salary-ranges and here, I am assuming the ranges to be non-overlapping (and hence treating them as ordinal). The other case where range values are observation specific, I meant the end values are decided by the user and hence may or may not overlap. I am sorry for not being able to put this succinctly. Let me know if this is now clear.
Write out a $n\\times n$ covariance matrix (with entries $\\operatorname{cov}(X_i, X_j)$) and replace each $\\operatorname{cov}(X_i, X_j)$ by $\\gamma(i-j)$. DO  NOT use letters such as $i$ and $j$ in your matrix; the top left corner should have $\\gamma(0)$ in it. _Admire_ each diagonal in the matrix thus created. What is the length of each diagonal (there are $2n-1$ diagonals of varying lengths for you to consider)? What is the _sum_ of the entries on each diagonal?
I've not really used the neural network toolbox (NETLAB, which is freely available from Aston University is a *much* better toolbox), but it looks to me as if you are using the newff incorrectly, the first parameter should be an n by 2 array of minimum and maximum values for each feature, and the second an array of the dimensions for each layer, whereas you have passed in the input and response variables.
@whuber
the utility function is composed from all attributes , all attributes are involved. Utility is not monotonically increasing because each attribute along with the others affects the utility.
I want to model the uncertainty because I want to make some operation on the instance for example find similar instances out of repository.
Values are undefined because of error in reading parameters from system for instance. or invalid values
One good way of estimating parameters in Kalman filter in such a simple setting is to use Maximum Likelihood Estimation. The exact likelihood is an easy analytic expression. I recommend the Durbin Koopman book (http://www.amazon.com/Series-Analysis-Methods-Statistical-Science/dp/019964117X/) for many practical issues (Chapter 7 for parameter estimation). http://www.eui.eu/Personal/Canova/Articles/ch6.pdf  link formula (6.21) is a little more difficult to read but the same thing. By the way it is strange that you have X values at hand. If they are observable you don't need a kalman filter.
@whuber ah, indeed! And both generalize to [GEV](https://en.wikipedia.org/wiki/Generalized_extreme_value_distribution)
Thank you. This sounds like a very plausible and practical application. I would need to look into this possibility more to understand the rational behind its use and when it is appropriate, but it is a place to start.
What do you mean by 'best' here?
@whuber OK, now we're swimming way beyond my capabilities and understanding, but per [this entry](https://en.wikipedia.org/wiki/Characteristic_function_%28probability_theory%29#Criteria_for_characteristic_functions) a convex linear combination of characteristic functions is also a characteristic function. Since there is a one-to-one relationship between characteristic functions and CDFs, this would mean that the convex linear combination does define a PMF. There is no guarantee that PMF is Poisson, though.
(+1) I especially like the test-function
I think - You ought to have asked another question first, whether linear FA is _appropriate_ for binary data at all. http://stats.stackexchange.com/a/19595/3277, http://stats.stackexchange.com/a/16335/3277
**Let's do a reality check**: the data reflect journeys with mean speeds of 1 km/0.1 hr = 10 kph, 3 km/0.4 hr = 7.5 kph, 4 km/0.5 hr = 8 kph, and 5 km/0.6 hr = 8.33 kph. This range of speeds from 7.5 to 10 kph shows that an answer of 1.13 kph is grossly wrong and that 8.1 kph is reasonable. Moreover, the total distance is 1+3+4+5=13 km, not 6.3 km, in a period of 0.1+0.4+0.5+0.6=1.6 hours.
I assume that the book models the returns of loans and not the loss. That's the key point. Loans which default have low returns since everything you get back is your collateral. In an univariate distribution that models a loan's return, the default scenario is in the left tail.
In contrast, comon books model the loss of loan where the default risk is in the right tail.
For verification, run a simulation :-).
If the three models use strictly the same IVs and DV you can compare their standard errors and p-values. You should specify what is the goal of your research to have a more accurate answer tough. I'm not clear why you would split the sample size like this.
By all means suggest an alternative but 1) given the hypothesis I think that it is not unreasonable to expect all subjects to score at chance (the null) and 2) I reread the paper I used as the inspiration and they used a one-sample t-test in this way.
You get a better estimate of $\\sigma^2$ by using a unified model.  I would only fit separate models if interactions are too messy to handle.
Some amount of correlation among predictors is normal in multiple regression. How strong are they in your case? Consider computing variance inflation factors or the condition number for your set of predictors. If excessive, consider [Wikipedia's "Remedies for multicollinearity"](https://en.wikipedia.org/wiki/Multicollinearity#Remedies_for_multicollinearity).
@ Henrik thank you for the very useful reply. It does answer my question so I've accepted. However it also implies I shouldn't use ANCOVA because my IVs ARE correlated with the covariate in this case - that's the whole reason I'm doing it! So, what should I do instead to address the question of whether the my IVs have any effect on my DV beyond what can be explained by their effect on my covariate?
Consider the situation where the mean changes over time but the variance does not.
I'm not sure I understand your question. I didn't actually *use* the phrase "bias of a parameter." I'm also not entirely sure what you mean by $y$ versus $\\theta$ in this context.
Are you in a tattoo parlour?
This is the point of departure in Dixon's survey I referenced.
In my experience it is certainly true that humans tend to overfit. However, in my experience you also need a decent expert who chooses the not-overfitting learning machine. Otherwise someone just chooses a learning machine that overfits.
This site supports LateX commands, you just enclose them in dollar signs.  For example to get $(-\\infty,\\infty)$ write "(-\\infty,\\infty)" enclosed in $ signs. see [the wiki page](http://en.wikibooks.org/wiki/LaTeX/Mathematics) for syntax, but ignore the 'begin{math}' and 'end{math}' parts, just use dollar sign instead.
Derek - Good point.  I've edited the question to make clear the same problem shows up in the MiningSchema.
@Mike thats okay. The programming language used by MathJax is $\\LaTeX$
You're right that there's a question about how many comparisons are being done, but I don't think it implies your conclusion.  Reasonable people can differ *because they have different objectives and different valuations (loss functions) for the possible outcomes.*  If you should be correcting for multiple comparisons, this is because it leads to better expected loss.  As such, this is an intensely practical issue, not mere "philosophy," and there are rational ways to resolve it on which reasonable people can agree.
I'm not really sure that assessing overall fit address my specific question of determining whether groups are significantly different in performance when we account for age and other covariates. Maybe I'm missing something, but if I have the "right" covariates wouldn't such model improve anyway? I'm not interested in finding the best predictive model, just to asses the significance between groups, while controlling for other things.
Welcome @StudentofScience. Not sure whether this constitutes an answer to the OP's question. Would you mind giving more explanation about your idea and how this adresses the question?
@PranayWarke Although there are ways to automate the process, yes it becomes infeasible. You also have to question whether a regression model (linear or polynomial) is suitable. On the whole you might prefer a parsimonious model with fewer parameters, especially if you don't have a large number of examples in your training set. So you could for example investigate whether some of your explanatory variables could be dropped, or choose a different kind of model altogether. You could try searching for "dimensionality reduction" or similar.
The inputs in a gaussian process are some training points and normally we set the X* points over the inputs. Thus we have more X* points than X points and they span the same range. Maybe this couls lead sometimes to the decribed X=X* behavior
For the K([-0.62978309,  0.10464536]) example, I assumed I already subtracted x−Xi and divide it by the bandwidth. Because after I made these operations on a 2D sample, I will still have a 2D data like this: [-0.62978309,  0.10464536].
This thread may be helpful. http://stats.stackexchange.com/questions/76866/clustering-with-k-means-and-em-how-are-they-related/78294#78294 Search your tags for other relevant  questions.
I reckon loads of computer games are modelling problems dressed in disguise. SimCity for example - the goal of the game is to build as good a model as possible of the hidden game mechanics, then use that model to build a functioning city! (This is all probably a gross over-justification for wasting my youth playing SimCity)
I see the parameter estimate read out in SPSS but am unclear how to interpret it.
Also. The distributions (PDF) are known.
About 2: this is a matter of notation: by $E[Z]^2$ one should understand $E^2[Z]$: notice that in the text the symbol $[$ is after $E$, and note before it.
Why Beta(1/2, 1/2)? This is a bimodal distribution with the weight concentrated around 0 and 1. OP, and future readers, will be interested in explanation about why this is the prior, rather than simply a prior among many alternatives. Some coins could be 1 with probability 0.75, so a prior of Beta(3,1) could represent that... or Beta(30,10)...
@Maria, reputation limitations are only for leaving comment (50) and voting up (15) and down (125), not accepting answer.
@user39531: was there anything else you wanted to know here, or is my answer [acceptable](http://meta.stackexchange.com/a/5235/246931)?
What is your specific question?
And what about simulated annealing with Metropolis Hastings? I have read [this](http://www2.samford.edu/~sfdonald/Courses/cosc407/Papers/Introduction%20to%20simulated%20annealing%20%28Albright%29.pdf) but what about the integration with Metropolis Hastings?
Heard that the Chamberlain method can solve the incidental problem but I dont know how to do it in any software either..Maybe sas has the choice to pick up this methodology..
This method assumes the bias is a constant irrespective of temperature level. If that (might) not be the case, some regression analysis might work.
I'm basically interested in the interaction between color and side, i.e. red stimuli are responded to faster on the left side, and green stimuli on the right side (due to response button location). The main question is, how much these stimuli have to displaced for this effect to occur. Statistically speaking: on which levels of the factor offset does this color x side interaction occur? The second plot suggests, there is no such interaction on the levels 1, 2, 3. And based on the plot I'd say the interaction is present at offset 4 and 5. But is it statistically significant?
I barely see any difference between categorical and qualitative variable, except one of terminology. Anyway, that would be very difficult to compute anything like mean or SD on a nominal variable (e.g., hair color). Maybe you are thinking of categorical variables with ordered levels?
Did you try your new code on some simple cases where you can compute log_p by hand? Did you check some cases where your old code worked?
Also, for the KS test, I assume it's important that the pvals are iid. Is this the case?
Can you post a small, reproducible example with a subset of data? What does the original csv file contain? Note that if your data actually contain commas within the numbers, that might be a problem for a *comma separated value* format.
For rare events, it is inefficient to simulate from the distribution of reference $F$, because it induces a large variance. Importance sampling is better: pick a distribution $G$ that concentrates more on the event of interest and use the weights $F(X_i)/G(X_i)$ in your average. A good choice of $G$ may reduce the variance to huge extents.
Well we didn't really learn about confidence intervals in the chapter and the lecture. The only thing we were taught about Binomial Normal Approximation was the continuity correction.
Not a good idea to discretize continuous predictors: see [here](http://stats.stackexchange.com/questions/68834/what-is-the-benefit-of-breaking-up-a-continuous-predictor-variable/68839). At best it does little harm.
Do you mean that the target variable was 1 for 90 cases and 0 for the other 69,930? Or do you mean that it was *missing* for all but 90? In the latter case, your N would be 90, not 70,000. But you can still look at the distribution of those 90 across clusters.