iSCSI initiator names in cloud-images

I recently worked on a fix for LP: #144499 in Ubuntu’s cloud images where every instance (VM or LXD container) using a given cloud image would end up sharing the iSCSI initiator name. The iSCSI initiator name is intended to be unique, so that you can not only uniquely identify which system is using a given target on the iSCSI server, but also, if desired, restrict which initiators can use which targets.

This behavioral change was introduced by the fix for LP: #1057635; which was working around a different issue with initiator names by re-instituting an older behavior in Ubuntu. In effect, the open-iscsi package can either configure the iSCSI initiator name at install time or at boot time. This generation is controlled by a helper script seeing GenerateName=Yes in /etc/iscsi/initiatorname.iscsi; if it does see that string, then it generates a new unique initiator name using another helper. Ideally, this would be done at first-boot time (by the helper script), however if iSCSI is used for the root device, the initramfs will not contain a valid initiator name and will fail to find the root iSCSI disk. So, 1057635 re-instituted the prior Ubuntu behavior that the initiator name is created when the open-iscsi package is installed.

In order for iSCSI root to work, though, open-iscsi needs to be pre-installed in the installer environment (“seeded” in Ubuntu parlance), or if using images, needs to be installed by default so the image can use iSCSI. That results in processes like the CPC cloud image creation installing the open-iscsi package during the image creation. But that installation ends up creating an initiator name due to the prior bug. And thus every instance using that image has the same initiator name! To fix this, at least somewhat, I added a hook to the CPC image generation which, if it detects that /etc/iscsi/initiatorname.iscsi exists, overwrites it with GenerateName=yes. Thus, on the start of any instance using that cloud image, a new unique initiator name will be used.

Scott Moser (smoser), though, pointed out this “fix” is not quite complete. If you start with a cloud image and then make a snapshot, or a local image, from a running instance — all new instances using that local image will end up sharing an initiator name. This is actually relatively tricky to figure out — what we want is every unique instance to get a unique initiator name, not every image. I’m going to be trying to work out this issue on LP: #1677726. I probably will need to configure an iSCSI root and boot setup at home first 🙂

[USD #1] Ubuntu Server Dev git Importer

This is the first in a series of posts about the Ubuntu Server Team’s git importer (usd). There is a lot to discuss: why it’s necessary, the algorithm, using the tooling for doing merges, using the tooling for contributing one-off fixes, etc. But for this post, I’m just going to give a quick overview of what’s available and will follow-up in future posts with those details.

The importer was first announced here and then a second announcement was made here. But both those posts are pretty out-of-date now… I have written a relatively current guide to merging which does talk about the tooling here, and much of that content will be re-covered in future blog posts.

The tooling is browse-able here and can be obtained via

git clone https://git.launchpad.net/usd-importer

This will provide a usd command in the local repository’s bin directory. That command resembles git as being the launching point for interacting with imported trees — both for importing them and for using them:

usage: usd [-h] [-P PARENTFILE] [-L PULLFILE]
 build|build-source|clone|import|merge|tag ...

Ubuntu Server Dev git tool

positional arguments:
 build|build-source|clone|import|merge|tag
 
 build - Build a usd-cloned tree with dpkg-buildpackage
 build-source - Build a source package and changes file
 clone - Clone package to a directory
 import - Update a launchpad git tree based upon the state of the Ubuntu and Debian archives
 merge - Given a usd-import'd tree, assist with an Ubuntu merge
 tag - Given a usd-import'd tree, tag a commit respecting DEP14

...

More information is available at https://wiki.ubuntu.com/UbuntuDevelopment/Merging/GitWorkflow.

You can run usd locally without arguments to view the full help.

Imported trees currently live here. This will probably change in the future as we work with the Launchpad team to integrate the functionality. As you can see, we have 411 repositories currently (as of this post) and that’s a consequence of having the importer running automatically. Every 20 minutes or so, the usd-cron script checks if there are any new publishes of source packages listed in usd-cron-packages.txt in Debian or Ubuntu and runs usd import on them, if so.

I think that’s enough for the first post! Just browsing the code and the imported trees is pretty interesting (running gitk on an imported repository gives you a very interesting visual of Ubuntu development). I’ll dig into details in the next post (probably of many).

(USBSD #1: Goals) Inaugural Ubuntu Server Bug Squashing Day!

As posted on the ubuntu-server mailing list we had our first Ubuntu Server Bug Squashing Day (USBSD) on Wednesday, March 22, 2017. While we may not have had a large community showing, the event was still a success and their is momentum to make this a regular event going forward (more on that below…). This post is about the goals behind USBSD.

[Throughout the following I will probably refer to users by their IRC nicks. When I know their real names, I will try and use them as well at least once so real-person association is available.]

The intent of the USBSD is two-fold:

  1. The Server Team has a triage rotation for all bugs filed against packages in main, which is purely an attempt to provide adequate responses to ‘important’ — ensuring we have ‘good’ bug reports that are actionable and then to put them on to the Server Team’s queue (via subscribing ~ubuntu-server). The goal for triage is not to solve the bugs, it’s simply to respond and put it on the ‘to-fix’ list (which is visible here. But we don’t want that list to just grow without bound (what good is it to respond to a bug but never fix it?), so we need to dedicate some time to working to get a bug to closure (or at least to the upload/sponsorship stage).
  2. Encourage community-driven ownership of bug-fixes and packages. While Robie Basak (rbasak), Christian Ehrhardt (cpaelzer), Josh Powers (powersj) and myself (nacc) all work for Canonical on the Server Team on the release side of things (meaning merges, bug-fixes, etc), there simply is not enough time in each cycle for the four of us alone to address every bug filed. And it’s not to say that the only developers working on packages an Ubuntu Server user cares are us four. But from a coordination perspective for every package in main that is ‘important’ to Ubuntu Server, we are often at least involved. I do not want to diminish by any means any contribution to Ubuntu Server, but it does feel like the broader community contributions have slowed down with recent releases. That might be a good thing ™ in that packages don’t have as many bugs, or it might just be that bugs are getting filed and no one is working on them. By improving our tooling and processes around bugs, we can lower barriers to entry for new contributors and ideally grow ownership and knowledge of packages relevant to Ubuntu Server.

That is a rather long-winded introduction to the goals. Did we meet them?

To the first point, it was a positive experience for those of us working on bugs on the day to have a dedicated place to coordinate and discuss solutions (on IRC at FreeNode/#ubuntu-server as well as well on the Etherpad we used [requires authentication and membership in the ~ubuntu-etherpad Launchpad team]. And I believe a handful of bugs were driven to completion.

To the second point, I was not pinged much at all (if at all) during the US business day on USBSD #1. That was a bit disappointing. But I saw that cpaelzer helped a few different users with several classes of bugs and that was awesome to wake up to! He also did a great job of documenting his bugwork/interactions on the Etherpad.

Follow-on posts will talk about ways we can improve and hopefully document some patterns for bugwork that we discover via USBSDs.

In the meanwhile, we’re tentatively scheduling USBSD #2 for April 5, 2017!