So, as I mentioned, I tried going down the road of munging the ports tree and had it kinda working locally, but this was a stopping point for me:

"You have to be a bit careful with updating mechanisms if you customise your ports like this: portsnap for one is quite unfriendly to "foreign" files within the ports tree, but csup works fine."

All of our FreeBSD users told us they were using portsnap, and it's one of the things I'd talked to Colin about, I think (he wrote it). If we can't safely munge the ports tree in a way that's compatible with the tools our customers use, then we can't munge the ports tree. So, that option is right out. Again, if we break the usual process the user uses to update his system, it's not useful. If we're using multiple tools, multiple processes, and making the user do extra work or doing things that confuse them, we might as well just do it with our own package manager (we already have to maintain that repository for Solaris, Mac OS X, and other crappy systems with no package management to speak of).

The other options seemed to require more user involvement (i.e. switching between the standard and our repo manually), or adding an additional wrapper for the ports tools, thus requiring users to use a different command than usual. Further, if the repos are separate, and you switch between them, you no longer have any dependency resolution (even though I complain about it, at least there are some capabilities there...but only if all of ports is available). I guess we could run a complete mirror of the ports tree with our stuff munged in and all indexes and stuff regenerated, but that's just...too awful to think about. I also suspect our users would be unhappy with that solution.

In short, as far as I can tell, your link to the discussion on the FreeBSD mailing list confirms my own experience, which is: it doesn't work.

But, I could be misunderstanding something in that response. I'm quite clueless about a huge array of FreeBSD and ports questions. The answer could be staring me in the face, but I don't see it...just a rehash of all the problems I ran into when I went down this path a couple years ago.

Basically, we either break portsnap (or, more accurately, we'll get broken by portsnap) by munging into the existing ports tree, or we lose all dependency resolution by running a second separate repository.

Compare this to yum or apt-get, where a third party repository package is able to resolve from any other repository, including the standard ones, doesn't require any extra wrapper commands or knowledge on the part of the user, and setting it up on the users machine requires editing one configuration file and importing the relevant GPG keys.

"Have you considered throwing the question to your FreeBSD users to see if they can offer any help or suggestions to make things easier for themselves?"

Of course. The persistent ProFTPd problem I ran into finally got resolved by one of our users after a few weeks, actually. But, the third party repos problem was beyond all their kin, unfortunately. And, unless I'm mistaken, it's unresolved still, though I appreciate your, and the FreeBSD lists, attempts to help figure it out.

"Hire someone who uses (and likes) FreeBSD if you're having that much trouble finding what took me about three minutes to find."

I did. They didn't have any answers on the questions that remain. As I mentioned, I also hashed out some of this with Colin Percival (who, as far as I can tell, is at least six times smarter than I am, and knows a whole helluva lot more about FreeBSD and ports and portsnap than any one human ought).

While we're referencing, I couldn't dig up the bug about pkg_add return code issue, but I did find several people talking about pkgs simply not working:

"When a pkg just works, it’s a beautiful and fast thing.
The problem is when they don’t. That’s when ports shine."
"Back when I first started running FreeBSD, I completely destroyed my system installing the simplest stuff. Eventually, nothing would install. I do mean nothing. I had no idea what was going wrong, I just knew that I was frustrated. So I stopped using FreeBSD for a long time."
http://oscartheduck.wordpress.com/2007/07/25/on-ports-vs-pkg...

"Why choose one over the other? Well with packages sometimes the binary doesn't work. Either you happen to have a newer or older file than what the creator compiled it with or you changed something with your system and it doesn't match anymore with the author's compiled environment. However packages are fast to install. No need to break out gcc and compile source code."
http://returnbooleantrue.blogspot.com/2009/05/fixing-pkgadd-...

This was exactly my experience with binary packages. Some worked. Some didn't, and worse, the ones that didn't work provided no warning during installation that they would be inoperable once installed. Several of the packages we require had to be installed from ports because the binaries were non-functional. Making the install time for Virtualmin on FreeBSD a few hours (on a very small CPU-limited VPS, it can actually take days), vs. 10-20 minutes on other equivalently specced systems running CentOS or Debian (the small VPS case can take a couple of hours with these systems).

My concern about the return code bug is that it was persistent through multiple major revisions of FreeBSD. The first report of it was about 5.x, I seem to recall. It had existed and been discussed, but never corrected, over the span of a couple of years.

And, while the bugs you mention about RPM/yum sound annoying, they never caused me any trouble during my process of building our installation script. We certainly manage to tickle a lot of bugs, because we're doing so much heavy-lifting with the package managers on all these systems; and I've found loads of bugs on all of our supported systems (and I simply killed support for SUSE because their developers simply refused to fix package manager bugs that were showstoppers for us; closing them with unrelated "DUPE" statuses, and denying they were problems, at all; while Red Hat always fixed the bugs I reported, even if it took them a while). The difference is in how much it slows down our deployment, how much additional work it puts on me and our users, how reliably it performs its job as documented, etc. FreeBSD was, so far, our most complex target OS to support because of package manager issues. Technically, Solaris probably ought to hold that honor but we punted there, and took the easy way out, and said "screw the native package manager, it's shit anyway". I tried valiantly to use the FreeBSD package management tools as much as possible, which I paid for in a lot of time and frustration.

Another annoying fact is that most discussions about ports and problems therein result in redirection rather than admitting there are problems. Like the very common, "Just use ports" in response to any "binary packages do not work" kind of query; it seems to me that binaries ought to actually work consistently, or at least warn you during installation if they aren't going to work. This is what package management is for: to make sure the stuff you install works. If it doesn't, then the package manager is broken. I just ran into so many instances of packages not working with ports and binary packages, that my level of frustration is probably the highest it's ever been since starting Virtualmin (this company was designed from the get-go to be low frustration, as a response to my prior very stressful hardware company; so when it gets stressful, I get mad!).

"> And, we have a nice UI in the product itself that can query the native package manager to display a list of available updates and such.
Getting that information out of the ports tree is trivial."

But, since we're not in the ports tree, and can't be in the ports tree, it doesn't solve our problem. We could, and probably will, add support for ports to our updates UI (which could then merge the data of available updates with our wbm packages we already know about and display on FreeBSD), and so once installed, we could get close to an equally good experience for FreeBSD users, at least in the UI. The command line is still a mess, since updating our packages would be separate.

Anyway, I honestly do appreciate your going to the trouble to help figure it out, even if I'm pretty sure we're actually no closer to a solution that works.