User1: User4 shipit
User2: User3 Your PR was used to test the new juju-core-slave. You have extra comments sorry. you might care about the last run which looks like a legitimate run https://github.com/juju/juju/pull/3883
User3: User2 okay, looking ...
User3: User2 fixed, repushing
User2: User3 great I will watch to be sure your branch gets a fair test
User3: User2 thanks :)
User4: User1 danka
User4: User5 go 1.5.2 was just released
User5: User4 i saw it was close
User4: well, they've tagged the branch
User4: the binary artifacts will be coming soon
User5: cool
 * mwhudson runs away, will look later on
User1: poo
User4: User1 ?
User1: found a bug in the migration steps for 1.21
User6: another one?
User1: User6 yeah
 * thumper goes to make coffee before he tears this apart
User7: Bug #1522025 changed: juju upgrade-juju should confirm it has enough disk space to complete <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1522025>
User7: Bug #1522025 opened: juju upgrade-juju should confirm it has enough disk space to complete <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1522025>
User7: Bug #1522025 changed: juju upgrade-juju should confirm it has enough disk space to complete <upgrade-juju> <juju-core:New> <https://launchpad.net/bugs/1522025>
 * thumper is not enjoying reading this code
User8: User4 you ocr? got time to look at http://reviews.vapour.ws/r/3293 ? several of the files are new test charms or param renames
User9: User1 I've been having that feeling a lot lately
User8: User1 can i pull master and run up a multi-env controller without a feature flag yet?
User1: User8 no
User8: damn
User1: because I'm fixing bugs :)
User1: not features
User8: User1 we're starting to need it for x-model work, i guess i could merge in your feature branch
User1: um...
User1: could do
User8: or just use flag
User1: oh fucking awesome
User8: i'll use feature flag
 * thumper headdesks
 * thumper headdesks
 * thumper headdesks
 * thumper headdesks
 * thumper headdesks
 * thumper headdesks
 * thumper headdesks
 * thumper headdesks
User8: lol
 * thumper headdesks
User8: ouch
User1: a test is passing because it isn't doing the asserts it think its doing
User9: lol
User8: hope it's not my code
User1: fuck fuck fuckity fuck
 * thumper digs deeper
User8: User4 so about any reviews?
User1: well, I added a line to make the test fail
User8: which test?
User1: state upgradeSuite.TestMigrateUnitPortsToOpenedPorts
User8: thanks davecheney :-)
User8: User10 thanks for review, questions answered plus a couple of changes, ptal when you are free
User10: User8 thanks, looking
User10: User8 shipit
User8: User10 awesome, tyvm
 * frobware wonders why in late 2015 he has a /boot partition which is 100MB in size because that's just too damn small...
User11: dimitern, voidspace, User13 I'm a little hosed atm. No working desktop. an upgrade gone wrong - lack of disk space seems to be root cause.
User12: frobware, oh sh*t :/
User12: frobware, best of luck fixing it then
User13: User11 crumbs. Ask for help if you need it.
User11: dooferlad, trying to chroot of a usb boot ...
User13: User11 have you tried a boot repair USB stick or a recovery mode with networking support boot?
User13: chrooting seems painful
User11: dooferlad, my genuine /boot has no working kernel. do you know if a 'boot repair' disk will add one?
User13: User11 ah, no. It seems to cover bad grub options, file system corruption and other bootloader related stuff, but not a missing kernel
User11: dooferlad, mkinitramfs failed (enospace) so ...
User13: User11 you should be OK deleting old kernels and related junk from /boot and retrying that. As long as one kernel is left that works...
User11: I have working kernels
User11: correction: I have NO working kernels.
User13: User12 can I have another review of http://reviews.vapour.ws/r/3294/ please?
User12: dooferlad, sure - looking
User12: dooferlad, replied
User14: User11 oh, ouch
User14: User11 User12 User13 this PR adds a providerId field to spaces and a providerId parameter to AddSpace
User14: http://reviews.vapour.ws/r/3307/
User14: it's a pre-req of my next couple of PRs
User14: User12 passing spaces through as constraints to maas will obviously need to use ProviderId and not Name
User14: User12 but if you make that change then the CLI command to add space will no longer work for maas spaces - you'll *need* space discovery
User14: (which is coming)
User14: so maybe don't make that change yet
User12: voidspace, looking
User12: voidspace, space add is not fully implemented
User12: voidspace, so it should be fine there, and for space create we have the card to not allow it when SupportsSpaces is false
User14: User12 ah, sorry - it was space create I meant
User14: User12 and we won't actually need "space add" I don't think
User14: User12 we'll really just need "space import" or whatever to update all definitions
User14: User12 a subsequent PR of mine will be adding SupportsSpaceDiscovery to NetworkingEnviron
User14: User12 so that will be the switch to disable space / subnet create
User14: User12 User11 User15 User13 bike shed on name, agent API facade for space and subnet creation (agent not client)
User14: User12 User11 User15 User13 "networking" ?
User15: voidspace, name it for the worker
User14: it's a bit too generic but it's also true
User14: User15 ok, that's easy enough
User15: voidspace, that's the granularity we're looking for
User14: discoverspaces
User14: User15 cool, thanks
User14: time for a rename then :-)
User12: voidspace, right - agreed about space import
User12: voidspace, +1 for SupportsSpaceManagement instead FWIW
User12: voidspace, reviewed btw
User14: management means nothing
User14: it's a weasel word
User14: User12 thanks for the review
User12: voidspace, well, "discovery" is a bit misleading as it implies we can't do discovery in other clouds *ever*
User14: User12 it doesn't imply that
User14: User12 when we *can* they'll return true
User14: that's precisely the point of it
User14: when we can support it for a provider the result of that method will change
User14: User12 you want me to change BackingSubnetInfo.ProviderId to string? I didn't touch that in my PR
User12: voidspace, ok, but then we can't forbid space create because discovery is not supported
User14: User12 "create" means create it on the provider
User14: User12 so we can forbid it where we can't create it on the provider
User12: voidspace, not BackingSubnetInfo specifically
User14: User12 that's the first issue you raised
User14: BackingSubnetInfo specifically...
User12: voidspace, but st.Subnet.ProviderId() returns network.Id
User12: voidspace, well it doesn't but it should (and BackingSubnetInfo represents st.SubnetInfo)
User14: User12 I think network.Id would probably be better here
User12: ..which also uses string aargh..
User14: I can fix that too
User14: it just adds work
User12: voidspace, I'd like to use network.Id for anything user-facing w.r.t. provider ids
User14: ok
User14: I'll include those changes here
User12: voidspace, doesn't have to happend now, if we store the correct value/type on the state doc
User14: state/doc is using string
User14: User12 should we use network.Id or string in mongo?
User12: voidspace, yeah, as it should, but the related method that returns the provider id field value should cast it to network.Id
User14: User12 threading network.Id throufh our code is fine
User14: User12 I might as well do it now
User12: voidspace, in mongo we use only native types or locally defined types in state
User12: voidspace, ok, cheers
User14: ok
User12: voidspace, thanks
User14: changed them, now to fix the 17 million things that break with type errors... :-D
User14: at least the compiler tells me where they are...
User14: User12 so state.SubnetInfo.ProviderId should also be network.Id
User12: voidspace, yes
User14: User12 cool
User14: User12 so *really* ProvisioningInfo.SubnetsToZones should be map[network.Id][]string
User14: User12 instead of map[string][]string
User14: User12 but that would be an incompatible API change, so I'm going to leave it
User12: voidspace, nope
User14: User12 what do you mean by nope? :-)
User12: voidspace, field types for api structs like mongo docs need to use types defined in the params (or state) package
User14: nope I'm wrong, nope I shouldn't leave it, nope I'm right
User14: so it shouldn't be network.Id
User12: voidspace, not on the params
User14: ok
User14: User12 thanks
User6: morning
User14: User6 morning
 * frobware celebrates the restoration of /boot with lunch
User14: User11 \o/
User11: voidspace, was pretty easy after I realised that the first couple of times I did the restore I was doing the restore to /boot which in /, which was in a LVM, but /boot should have been a physical partition... just forget to mount it during the chroot. :)
User11: and during lunch I think we'll do a complete system backup. :)
User14: cool
User14: User12 ping
User12: voidspace, pong
User14: User12 one issue you said:
User14: User12 After adding the index, here if err == nil, we should still try fetching the space back to see if it got added (see how AddNetworkInterface handles the similar case).
User14: User12 do you mean AddSubnet rather than AddNetworkInterface?
User14: User12 because if you mean AddNetworkInterface I can't find that code
User14: User12 is it to handle where adding can fail silently, because the providerid isn't unique?
User12: voidspace, AddNetworkInterface is in state/machine.go
User14: User12 yes
User14: User12 but there's no code as you describe in that method...
User12: voidspace, the switch at the end - case nil: ?
User14: whereas I *think* what you mean is the case in AddSubnet but I want to check I'm understanding you
User14: User12 oh right, I missed that
User14: User12 yuck, that's horrible code
User12: voidspace, but yeah - AddSubnet is actually a better (simpler) example
User14: how did I miss that
User12: voidspace, since the check is the same - providerid unique when != ""
User14: but anyway, I'll follow the AddSubnet pattern if it's doing the same thing
User12: voidspace, I'm refactoring AddNetworkInterface to be less icky hopefully :)
User14: ah, I grepped for AddNetworkInterface and found it in SetInstanceInfo
User14: and was reading that code
User14: cool
User14: SetInstanceInfo *calls* AddNetworkInterface, but it isn't AddNetworkInterface... :-)
User7: Bug #1522409 opened: juju machine 0 watcher unable to connect to units <sts> <juju-core:New> <https://launchpad.net/bugs/1522409>
User16: food for thought: state/State has 195 *exported* methods
User11: dimitern, voidspace, User13 PTAL @ http://reviews.vapour.ws/r/3308/
User12: dooferlad, you've got another review btw
User11: dimitern, voidspace, User13 too many distractions ^ that was a merge of the wrong branch in my repo
User12: frobware, if it's a straight forward revert, just land it I think
User12: frobware, except - I'll add a comment
User12: frobware, the head -n1 added with my recent PR is not there when getting the gateway
User12: frobware, ah, sorry - it is actually
User11: dimitern, it's still there post the revert?
User12: frobware, it is (in the expectedBlaBalCloudConfig)
User12: (the one with the bridge script)
User11: dimitern, I was confused why my cherry-pick for the bond change was not clean - now I know why... sigh.
User12: frobware, well, good catch :)
User12: voidspace, ping
User14: User12 pong
User12: voidspace, in case you're having issues with the ProviderId index (as I'm discovering now)
User14: User12 I am...
User14: User12 well, sort of
User14: it's causing unrelated bindings tests to fail
User14: but a worse problem right now is that my space tests don't appear to be running
User12: voidspace, I *think* the correct solution is only verify for uniqueness if ProviderId != ""
User14: User12 I thought that's what sparse did
User12: voidspace, well, yeah, but since ProviderId is also omitempty, it will be missing
User12: if not set
User12: "" != {$exists, false} as it happens
User14: ok
User14: User12 that's exactly how ProviderId on Subnets is defined though
User12: voidspace, well, it might be different for subnet.. but for NICs I'm having a bunch of issues, still debugging the Add.. call
User14: omitempty but with a unique, sparse index
User14: User12 ok
User14: I'm adding it to space - but just following what we did with subnets
User12: voidspace, well, that sounds sensible for subnets, but after spending >1h fiddling with indices, uniqueness, etc. I'm beginning to think having ProviderId defined like that for NICs is not a good idea
User14: User12 heh, fun
User12: e.g. imagine machine-0 with eth0 with aa:bb:cc:dd:ee:f0 for mac, "+1-873-658-7532" for subnet, there's no trouble having eth0 (same mac and all), but on subnet "bd6c:f4c4:4f6:c742:59d8:35db:68f6:4d5f/64"
User14: dammit SpacesSuite is not being run at all
User12: voidspace, sh*t! really?
User14: well, not on my machine right now
User14: it maybe a general problem
User12: voidspace, it might be due to the order of go test args (as I've discovered: -check.v -check.f *and* -race and/or -cover never finds the -check.f)
User14: User12 no args except package
User14: (I was originally trying with -check.f - but even without it I'm not seeing my deliberately failing tests)
User14: same with subnets test
User14: what the hell is going on
User12: voidspace, state.ConnSuite logs the beginning and end of SetUpTest btw - e.g. with -check.vv
User12: voidspace, I found also -check.list useful
User14: thanks
User14: User12 soooo... changing state/package_test.go from the state package to the state_test package seems to have changed things
User14: now I get build failures
User12: voidspace, wow - and here's davecheney who changed that recently :)
User14: User12 that change means that most state tests don't run
User12: voidspace, it's gets weirder - grep for MgoTestPackage in state/
User12: voidspace, uh, no actually no surprises there
User14: heh
User12: voidspace, that's a bug right there - https://github.com/juju/juju/pull/3806/files#diff-d4970ae51fd6268f6325442698dbe02a
User14: yeah, wrong package :-)
User14: oops...
User12: voidspace, that was 10 days ago!
User14: it took me about an hour or so just now to track it down
User12: I wonder how nobody found out
User14: couldn't work out why my tests weren't running
User17: happy belated birthday wwitzel3 !
User12: I'm filing a bug
User14: User12 thanks
User18: User17 thanks :)
User14: User12 User11 meeting?
User7: Bug #1522484 opened: state package tests no longer run since PR #3806 <blocker> <ci> <regression> <unit-tests> <juju-core:In Progress by dimitern> <https://launchpad.net/bugs/1522484>
User14: User12 I'm going to run state tests on maas-spaces
User11: dimitern, voidspace, User13 ok to rebase our maas-spaces when the state fix lands?
User12: voidspace, sgmt
User14: User12 I have a bunch of endpoint_bindings tests fail - and I can't tell if they're caused by my changes or not
User12: sgtm even
User13: User11 go for it
User12: frobware, I'd rather deal with the conflicts than let more regressions slip by :)
User14: User12 and it looks like that unique index on provider id is causing me issues
User12: voidspace, I'll have a look there and propose a fix if needed
User14: I add three spaces with empty provider ids and then AllSpaces only returns one space
User14: User12 cool
User12: damn :/
User14: adding that check in AddSpace to see if silent failure is the cause of that
User14: we'll see
User14: if it is the problem I'll need to check we don't have the same problem with subnets which use the same pattern
User14: maybe we have that problem but just haven't hit it because we always have a providerid
User12: voidspace, I'm reading https://docs.mongodb.org/v2.4/core/index-sparse/ now - it seems unique+sparse might not be what we need actually
User14: right
User14: User12 it sounds exactly right
User14: User12
User14: You can specify a sparse and unique index, that rejects documents that have duplicate values for a field, but allows multiple documents that omit that key.
User12: voidspace, ok, then we should not use that index I think
User12: voidspace, we can verify while adding that there's no existing space with the given non-empty provider id
User14: User12 eh, it sounds exactly like what we want
User14: hmmm...
User14: User12 I don't have the failing tests on maas-spaces, so introduced by my changes
User12: voidspace, AIUI you can have both doc{id:"foo", providerid:"bar"} and doc{id:"foo1"}, but then you can't have doc{id:"foo2"} as well
User14: User12 see their example
User14: at the bottom of that page
User14: you can
User12: voidspace, hmm well it sounds reasonable..
User12: but I can't seem to make it work for NICs
User14: it certainly *seems* like what we want
User14: User12 are you using omitempty?
User12: voidspace, yep
User14: ok
User12: ah, well.. perhaps it's because I'm over-complicating things as usual
User12: voidspace, in cases where we need uniqueness on multiple fields, what does work fine and it's easy to verify is a compound _id field, like a global key
User7: Bug #1517258 changed: juju 1.24.7 precise: container failed to start and was destroyed <oil> <juju-core:Triaged> <https://launchpad.net/bugs/1517258>
User12: but still coupled with individual fields having values used in the compound key - e.g. 8e5a492b-7631-409b-8134-7997678d731e:m#4/lxc/4#n#juju-public + EnvUUID + MachineId + NetworkName (on ports)
User14: User12 http://stackoverflow.com/questions/28183109/dealing-with-mongodb-unique-sparse-compound-indexes
User14: it seems like unique and sparse don't work together well for compound indices
User14: so it's probably currently broken for subnetsC
User12: voidspace, yeah, that's what I've (re)discovered I guess
User12: voidspace, not necessarily, as both spaces and subnets + provider id act similarly; but it's definitely broken for network interfaces docs
User12: (I mean there are no compound indices effectively, as env-uuid+field_name is always set)
User14: ah, subnetsC isn't compound
User14: User12 I was trying to make spacesC compound on env-uuid
User14: I'll have to leave that as a TODO
User12: frobware, voidspace, btw I see no failures in state/ on maas-spaces tip after changing package_test.go's package to state_test
User14: User12 yeah, I tried it too and saw no failure
User12: so rebase might wait
User14: I think most of my failures were actually caused by the index problem
User14: about to find out
User12: voidspace, yeah, sorry for steering you down that rat hole :(
User14: well, it's what we *should* do
User14: not your fault mongodb is broken
User12: :) cheers
User12: voidspace, what I've just noticed is even weirder - I see exactly the same number of tests passing (1245) in state/ for almost the same time (~198s) both with and without the change in package_test.go
User12: voidspace, when I run go test -check.f in state/
User9: User16 I made a slight modification of the apiClient code, to remove the need for the resourceAPIClient type.... let me know what you think: http://reviews.vapour.ws/r/3310/diff/#  (feel free to ignore the upload stuff in there, since it's still a WIP).  I know you made changes around that code too, but wanted to get your opinion of what I did.
User14: that is odd
User12: voidspace, however I suspect it's different for go test github.com/juju/juju/state -check.v
User9: User16 it also makes the function you have to write for every command completely trivial, which is nice.  Just a little adapter
User16: User9 looking
User9: User16 I haven't updated tests yet, obviously.
User12: voidspace, same result with go test github.com/juju/juju/state -check.v and unpatched package_test
User12: (albeit without any output)
User14: User12 I wonder if we have some black box state tests then
User14: and we need both
User14: User12 e.g. compat_test.go is in the state package
User14: User12 allwatcher_internal_test is as well
User12: voidspace, yeah, same for the upgrades
User19: hi mgz, cherylj - is 1.25.2 avail in a ppa anywhere atm?
User20: User19 no, not yet
User12: heh what do you know.. go test ./... -check.list almost managed to bring the quad core i7 to its knees
User14: User12 PR updated, all those issues addressed by the way
User12: voidspace, awesome! having a look now..
User14: User12 small change, touches 25 files...
User12: voidspace, does it still work if the spacesC and subnetsC index was defined as Key: []string{"env-uuid", "provider"}, Unique: true, Sparse: true?
User14: User12 no
User14: User12 causes failures
User14: User12 the sparse is then ignored
User12: voidspace, won't we have the issue where you can't have a space with no provider id in 2 different environments?
User12: voidspace, or with the same provider id for that matter
User14: User12 yes
User14: User12 but we already have that with subnets and a TODO to fix it...
User14: not good enough?
User12: voidspace, ok, fair enough
User14: how about we add a card for fixing both to the iteration backlog
User12: voidspace, just checking the options ;)
User14: cool
User12: voidspace, +1
User12: voidspace, lgtm
User14: thanks
User14: I thought that would take an hour at the start of the day
User14: took a bit longer...
User12: voidspace, gotta love mongo, right? :D
User14: right, back to the api
User14: User12 yeah... and our testing infrastructure
User12: ok time to call it a day
User14: User12 o/
User12: voidspace, :)
User11: voidspace, dooferlad, User12 Strike 2 - http://reviews.vapour.ws/r/3311/ - this is the correct branch for the change I reverted earlier.
User14: User11 as it's just a forward port, ShipIt!
User11: voidspace, ty
User11: rick_h_, hadn't forgotten your HA question - just got a little sidetracked with a broken desktop...
User21: User11 I forgot the question now heh
User11: rick_h_, HA for OpenStack. Reference platform is 28 nodes. Need to look at the charm to see if that's always the case.
User11: rick_h_, if so think I'll enter the h/w business. :)
User21: User11 :/ ok I think this was more can we do HA on Juju vs openstack on guimaas
User11: rick_h_, my 28 came from: https://wiki.ubuntu.com/ServerTeam/OpenStackHA
User21: User11 right, I don't think I was asking about openstack HA
User21: User11 but "can I deploy openstack (6? 8?) nodes and have Juju be HA there
User11: rick_h_, so not being at the beginning of the call I think I've missed the context.
User21: User11 to test upgrades of that deployment
User11: rick_h_, ok gotcha
User21: User11 how small can I get a functioning OS and have Juju HA state servers is what I'd like to see
User21: User11 and then investigate growth of the test case from there
User7: Bug #1522544 opened: workers restart endlessly <juju-core:New> <https://launchpad.net/bugs/1522544>
User1: morning folks
User9: morning thumper.  Happy friday
User22: o/ thumper
User9: User18 ping me via email when/if your branch is ready and I'll see what I can do to integrate with it.
User17: thumper, when do you head to the airport?
User1: User17 tomorrow lunch time
User17: thumper, aw ok
User9: back later
User1: ok, I have a fix for this 1.21 upgrade step
User1: why do we have no reviewers today?
User1: http://reviews.vapour.ws/r/3313/diff/#
User1: someone plz
User1: cherylj, waigani, or davecheney? ^^^
User20: User1 tal now
User1: ta
User1: ok, time for coffee... again
User8: User10 User6 a minute late
User6: k
juju-dev
Ubuntu IRC