git ubuntu clone

This is the second post in a collaborative series between Robie Basak and myself to introduce (more formally) git ubuntu to a broader audience. There is an index of all our planned posts in the first post. As mentioned there, it is important to keep in mind that the tooling and implementation are still highly experimental.

In this post, we will introduce the git ubuntu clone subcommand and take a brief tour of what an imported repository looks like. git ubuntu clone will be the entry point for most users to interact with Ubuntu source packages, as it answers a common request on IRC: “Where is the source for package X?”. As Robie alluded to in his introductory post, one of the consequences of the git ubuntu importer is that there is now a standard way to obtain the source of any given source package: git ubuntu clone1.

Getting git ubuntu clone

git-ubuntu is distributed as a “classic” snap. To install it on Ubuntu 16.04 or later:
sudo snap install --classic git-ubuntu. Help is available via git-ubuntu --help and man-pages are currently in development 2.

Using git ubuntu clone

Let’s say we are interested in looking at the state of PHP 7.0 in Ubuntu. First, we obtain a local copy of the repository 3: git ubuntu clone php7.0

With that one command, we now have the entire publishing history for php7.0 in ./php7.0. Anyone who has tried to find the source for an Ubuntu package before will recognize this as a significant simplification and improvement.

With git, we would expect to be on a ‘master’ branch after cloning. git ubuntu clone defaults to a local branch ‘ubuntu/devel’, which represents the current tip of development in Ubuntu. ‘ubuntu/devel’ is branched from the remote-tracking branch ‘pkg/ubuntu/devel’.

You might now be wondering, “What is ‘pkg/’?”

The default remotes

Running git remote, we see two remotes are already defined: ‘pkg’ and ‘nacc’.

‘pkg’ will be the same for all users and is similar to ‘origin’ that git users will be familiar with. The second is a derived remote name based upon a Launchpad ID. As shown above, the first time run git ubuntu runs, it will prompt for a Launchpad ID that will be cached for future use in ~/.gitconfig. Much like ‘origin’, the ‘pkg’ branches will keep moving forward via the importer and running git fetch pkg will keep your local remote-tracking branches up to date. While not strictly enforced by git or git ubuntu, we should treat the ‘pkg/’ namespace as reserved and read-only to avoid any issues.

The importer branches

The tip of ‘pkg/ubuntu/devel’ reflects the latest version of this package in Ubuntu. This will typically correspond to the development release and often will be the version in the ‘-proposed’ pocket for that release. As mentioned earlier, a local branch ‘ubuntu/devel’ is created by default, which starts at ‘pkg/ubuntu/devel’, much like ‘master’ typically starts at ‘origin/master’ by default when using git. Just like the tip of ‘ubuntu/devel’ is the latest version in Ubuntu for a given source package, there are series-‘devel’ branches for the latest in a given series, e.g., the tip of ‘pkg/ubuntu/xenial-devel’ is the latest version uploaded to 16.04. There are also branches tracking each ‘pocket’ of every series, e.g. ‘pkg/ubuntu/xenial-security’ is the latest version uploaded to the security pocket of 16.04.

Finally, there is a distinct set of branches which correspond to the exact same histories, but with quilt patches applied. Going into the reasoning behind this is beyond the scope of this post, but will be covered in a future post. It is sufficient for now to be aware that is what ‘pkg/applied/*’ are for.

What else can we do?

All of these branches have history, like one would expect, reflecting the exact publishing history of php7.0 within the context of that branch’s semantics, e.g., the history of ‘pkg/ubuntu/xenial-security’ shows all uploads to the security pocket of 16.04 and what those uploads, in turn, are based off of, etc. As another example, git log ubuntu/devel shows you the long history of the latest upload to Ubuntu.

With this complete imported history, we can not only see the history of the current version and any given series, but also what is different between versions and releases 16.04 and 17.04 for php7.0!

For other source packages that have existed much longer, you would be able to compare LTS to LTS, and do all the other normal git-ish things you might like, such as git blame to see what introduced a specific change to a file.

We can also see all remote-tracking branches with the normal git branch -r

This shows us a few of the namespaces in use currently:

  • pkg/ubuntu/* — patches-unapplied Ubuntu series branches
  • pkg/debian/* — patches-unapplied Debian series branches
  • pkg/applied/ubuntu/* — patches-applied Ubuntu series branches
  • pkg/applied/debian/* — patches-applied Debian series branches
  • pkg/importer/* — importer-internal branches

As Robie mentioned in the first post, we are currently using a whitelist to constrain the importer to a small subset of source packages. What happens if you request to clone a source package that has not yet been imported?

While many details (particularly why the repository looks the way it does) have been glossed in this post, we now have a starting point for cloning any source package (if it has been imported) and a way to request an import of any source package.

Using git directly (for advanced users)

Technically, git ubuntu clone is equivalent in functionality to git clone and git clone could be used directly. In fact, one of our goals is to not impede a “pure” git usage in any way. But again, as Robie mentioned in his introductory post, there are some caveats to both using git and the structure of our repositories that git ubuntu is aware of. The “well-defined URLs” just mentioned are still being worked on, but for instance for PHP 7.0, one could follow the instructions at the top of the Launchpad code page for the php7.0 source package. The primary differences we would notice in this usage is “origin” instead of “pkg” and there will not be a remote for your personal Launchpad space for this source package.


In this post, we have seen a new way to get the source for any given package, git ubuntu clone.

Robie’s next post will discuss where the imported repositories are and what they look like. My next post will continue discussing the git ubuntu tooling, by looking at another relatively simple subcommand “tag”.

  1. Throughout this post, we are assuming a automatically updated repository. This is true for the whitelisted set of packages currently auto-imported, but not true generally (yet). 
  2. All commands are available as both git-ubuntu … and git ubuntu …. However, for –help to work in the latter form, the changes mentioned in LP : #1699526, a few simple tweaks to ~/.gitconfig are necessary until some additional snap functionality is available generally. 
  3. Currently, git ubuntu clone is rather quiet while it works, and can take a long time (the history of a source package can be long!); we have received feedback and opened a bug to make it a bit more like git clone from a UX perspective. 

usd has been renamed to git-ubuntu

After some internal bikeshedding, we decided to rework the tooling that the Server Team has been working on for git-based source package management. The old tool was usd (Ubuntu Server Dev), as it stemmed from a Canonical Server sprint in Barcelona last year. That name is confusing (acronyms that aren’t obvious are never good) and really the tooling had evolved to be a git wrapper.

So, we renamed everything to be git-ubuntu. Since git is awesome, that means git ubuntu also works as long as git-ubuntu is in your $PATH. The snap (previously usd-nacc) has been deprecated in favor of git-ubuntu (it still exists, but if you try to run, e.g., usd-nacc.usd you are told to install the git-ubuntu snap). To get it, use:

sudo snap install --classic git-ubuntu

We are working on some relatively big changes to the code-base to release next week:

  1. Empty directory support (LP: #1687057). My colleague Robie Basak implemented a workaround for upstream git not being able to represent empty directories.
  2. Standardizing (internal to the code) how the remote(s) work and what refspecs are used to fetch from them.

Along with those architectural changes, one big functional shift is to using git-config to store some metadata about the user (specifically, the Launchpad user name to use, in ~/.gitconfig) and the command used to create the repository (specifically, the source package name, in <dir>/.git/config). I think this actually ends up being quite clean from an end-user perspective, and it means our APIs and commands are easier to use, as we can just lookup this information from git-config when using an existing repository.

As always, the latest code is at:

iSCSI initiator names in cloud-images

I recently worked on a fix for LP: #144499 in Ubuntu’s cloud images where every instance (VM or LXD container) using a given cloud image would end up sharing the iSCSI initiator name. The iSCSI initiator name is intended to be unique, so that you can not only uniquely identify which system is using a given target on the iSCSI server, but also, if desired, restrict which initiators can use which targets.

This behavioral change was introduced by the fix for LP: #1057635; which was working around a different issue with initiator names by re-instituting an older behavior in Ubuntu. In effect, the open-iscsi package can either configure the iSCSI initiator name at install time or at boot time. This generation is controlled by a helper script seeing GenerateName=Yes in /etc/iscsi/initiatorname.iscsi; if it does see that string, then it generates a new unique initiator name using another helper. Ideally, this would be done at first-boot time (by the helper script), however if iSCSI is used for the root device, the initramfs will not contain a valid initiator name and will fail to find the root iSCSI disk. So, 1057635 re-instituted the prior Ubuntu behavior that the initiator name is created when the open-iscsi package is installed.

In order for iSCSI root to work, though, open-iscsi needs to be pre-installed in the installer environment (“seeded” in Ubuntu parlance), or if using images, needs to be installed by default so the image can use iSCSI. That results in processes like the CPC cloud image creation installing the open-iscsi package during the image creation. But that installation ends up creating an initiator name due to the prior bug. And thus every instance using that image has the same initiator name! To fix this, at least somewhat, I added a hook to the CPC image generation which, if it detects that /etc/iscsi/initiatorname.iscsi exists, overwrites it with GenerateName=yes. Thus, on the start of any instance using that cloud image, a new unique initiator name will be used.

Scott Moser (smoser), though, pointed out this “fix” is not quite complete. If you start with a cloud image and then make a snapshot, or a local image, from a running instance — all new instances using that local image will end up sharing an initiator name. This is actually relatively tricky to figure out — what we want is every unique instance to get a unique initiator name, not every image. I’m going to be trying to work out this issue on LP: #1677726. I probably will need to configure an iSCSI root and boot setup at home first 🙂

[USD #1] Ubuntu Server Dev git Importer

This is the first in a series of posts about the Ubuntu Server Team’s git importer (usd). There is a lot to discuss: why it’s necessary, the algorithm, using the tooling for doing merges, using the tooling for contributing one-off fixes, etc. But for this post, I’m just going to give a quick overview of what’s available and will follow-up in future posts with those details.

The importer was first announced here and then a second announcement was made here. But both those posts are pretty out-of-date now… I have written a relatively current guide to merging which does talk about the tooling here, and much of that content will be re-covered in future blog posts.

The tooling is browse-able here and can be obtained via

git clone

This will provide a usd command in the local repository’s bin directory. That command resembles git as being the launching point for interacting with imported trees — both for importing them and for using them:

usage: usd [-h] [-P PARENTFILE] [-L PULLFILE]
 build|build-source|clone|import|merge|tag ...

Ubuntu Server Dev git tool

positional arguments:
 build - Build a usd-cloned tree with dpkg-buildpackage
 build-source - Build a source package and changes file
 clone - Clone package to a directory
 import - Update a launchpad git tree based upon the state of the Ubuntu and Debian archives
 merge - Given a usd-import'd tree, assist with an Ubuntu merge
 tag - Given a usd-import'd tree, tag a commit respecting DEP14


More information is available at

You can run usd locally without arguments to view the full help.

Imported trees currently live here. This will probably change in the future as we work with the Launchpad team to integrate the functionality. As you can see, we have 411 repositories currently (as of this post) and that’s a consequence of having the importer running automatically. Every 20 minutes or so, the usd-cron script checks if there are any new publishes of source packages listed in usd-cron-packages.txt in Debian or Ubuntu and runs usd import on them, if so.

I think that’s enough for the first post! Just browsing the code and the imported trees is pretty interesting (running gitk on an imported repository gives you a very interesting visual of Ubuntu development). I’ll dig into details in the next post (probably of many).

(USBSD #1: Goals) Inaugural Ubuntu Server Bug Squashing Day!

As posted on the ubuntu-server mailing list we had our first Ubuntu Server Bug Squashing Day (USBSD) on Wednesday, March 22, 2017. While we may not have had a large community showing, the event was still a success and their is momentum to make this a regular event going forward (more on that below…). This post is about the goals behind USBSD.

[Throughout the following I will probably refer to users by their IRC nicks. When I know their real names, I will try and use them as well at least once so real-person association is available.]

The intent of the USBSD is two-fold:

  1. The Server Team has a triage rotation for all bugs filed against packages in main, which is purely an attempt to provide adequate responses to ‘important’ — ensuring we have ‘good’ bug reports that are actionable and then to put them on to the Server Team’s queue (via subscribing ~ubuntu-server). The goal for triage is not to solve the bugs, it’s simply to respond and put it on the ‘to-fix’ list (which is visible here. But we don’t want that list to just grow without bound (what good is it to respond to a bug but never fix it?), so we need to dedicate some time to working to get a bug to closure (or at least to the upload/sponsorship stage).
  2. Encourage community-driven ownership of bug-fixes and packages. While Robie Basak (rbasak), Christian Ehrhardt (cpaelzer), Josh Powers (powersj) and myself (nacc) all work for Canonical on the Server Team on the release side of things (meaning merges, bug-fixes, etc), there simply is not enough time in each cycle for the four of us alone to address every bug filed. And it’s not to say that the only developers working on packages an Ubuntu Server user cares are us four. But from a coordination perspective for every package in main that is ‘important’ to Ubuntu Server, we are often at least involved. I do not want to diminish by any means any contribution to Ubuntu Server, but it does feel like the broader community contributions have slowed down with recent releases. That might be a good thing ™ in that packages don’t have as many bugs, or it might just be that bugs are getting filed and no one is working on them. By improving our tooling and processes around bugs, we can lower barriers to entry for new contributors and ideally grow ownership and knowledge of packages relevant to Ubuntu Server.

That is a rather long-winded introduction to the goals. Did we meet them?

To the first point, it was a positive experience for those of us working on bugs on the day to have a dedicated place to coordinate and discuss solutions (on IRC at FreeNode/#ubuntu-server as well as well on the Etherpad we used [requires authentication and membership in the ~ubuntu-etherpad Launchpad team]. And I believe a handful of bugs were driven to completion.

To the second point, I was not pinged much at all (if at all) during the US business day on USBSD #1. That was a bit disappointing. But I saw that cpaelzer helped a few different users with several classes of bugs and that was awesome to wake up to! He also did a great job of documenting his bugwork/interactions on the Etherpad.

Follow-on posts will talk about ways we can improve and hopefully document some patterns for bugwork that we discover via USBSDs.

In the meanwhile, we’re tentatively scheduling USBSD #2 for April 5, 2017!

Lenovo Yoga 900 screen auto-rotation and touchpad

With the patches so far mentioned on top of Linus’ tree, almost all of the Yoga 900’s hardware works. But there were two pain points:

  • Screen rotation
  • Trackpad on suspend/resume

For the first, I posted several questions to LKML/linux-iio/linux-input because it seemed like `watch -n 0.1 cat /sys/bus/iio/devices/iio\:device*/*raw*` was not updating on my system. It definitely did at some point, but most recently it was not. Well, I went and grabbed iiotools and after building it with some changes (no longer defined constants), it seems like `lsiio` caused the IIO sensors to unstick. But the /dev/ nodes were still not producing output. Discussing this further upstream, it seemed like a new quirk was needed. After updating my system to Gnome 3.18 with the PPAs for gnome3 and gnome3-staging, and installing iio-sensor-proxy, auto-rotation worked!

For the second, I again worked a bit upstream. It sounds like Linus had hit something pretty similar and a hack/workaround was proposed. A more complete fix (but not yet final) is here, but I expect there to be another version based upon Benjamin Tissoires‘ reply, which will finally close the race with suspend/resume.

This makes me rather hopeful that 16.04 out of the box (well, except maybe the DisplayLink drivers) will be have all the hardware on the Yoga 900 enabled!

Ubuntu GNOME 15.10, Lenovo Yoga 900 & Dell D3100 USB Docking Station

With my previous foray into getting most of the base hardware with my Yoga 900 working, I wanted to look at hooking up some of my accessory purchases, a Dell 3100 USB 3.0 docking station (including 3 video outputs, which is the maximum for the SkyLake generation of integrated video as I understand it) and a 4K external monitor (I also have an older 28″ external monitor I’d like to keep using, but it’s still hooked up to my to-be-reinstalled-upon-leaving-my-current-job desktop).

First complication was the display output of the Yoga 900, which is over the brand-new USB-C port. The cables that came with both my docking station and monitor, for high-speed video over USB, were standard USB cables. Radioshack sells (for $29.99, which seems bit steep, but I was in a rush) a USB-C to USB adapter. Without the adapter, you can plug the D3100 into one of the other super-speed USB ports, but you won’t see the DisplayLink interfaces in lsusb’s output (the dock also has 3 super-speed USB ports, 2 normal USB ports, an Ethernet port, and a 8mm headphone jack).

So, get the adapter (or something similar) and now all of the ports show up in dmesg and lsusb. Except that Linux can’t talk to the DisplayLink devices. Turns out, though, DisplayLink is working on a driver (not yet investigated if they are going to upstream this themselves or not…) for Ubuntu. The script they ship claims to only work with up to 3.19, but 4.4-rc4 seems fine (just not officially supported). Also, the underlying installer script only knows about 14.04.x and 15.04, and I’m on 15.10. It only needs to detect this version string, though, for determining whether to use upstart or systemd configurations. A trivial change the embedded installer script to use systemd on 15.10 and everything seems to work.

Note that by default, the Ubuntu kernel .config signs kernel modules. And this module is not signed using DKMS (as I understand it), so you’ll need to make sure your .config doesn’t have kernel module signing enabled.

But once you have that all installed, and plug in the USB-C adapter, the display(s) immediately are recognized and placed next to the built-in display’s screen. Could be a lot harder!

Ubuntu GNOME 15.10 on a Lenovo Yoga 900

As I begin to transition away from my current job, I figured I’d take advantage of my little IBM discount on Lenovo products and order myself a snappy new laptop. I chose a Lenovo Yoga 900 and was very impressed immediately by the size and quality of the product. And, of course, I immediately wanted to get rid of Windows.

I had a Ubuntu GNOME 15.10 image already on my USB key, so I simply needed to figure out how to get to the BIOS or Boot Menu. Turns out there, is a small button on the right side of the Yoga 900, called the Novo button, which when used to power on the system allows you to access the BIOS menus. Worked like a charm!

Out of the box, Ubuntu 15.10 installed fine, although the /boot partition selected by the auto-partitioner is so small as to be useless if you want to actually build multiple kernels on the system (and you are going to need to for 15.10!). So I resized it up to 1GB and moved on.

Most of the basic hardware worked, except:

  • wireless
  • touchscreen
  • touchpad

So, maybe not “most” in some sense, but hey.

Luckily, the wireless fixes were already upstream, so it was just a matter of putting an appropriate .deb from on the USB key and installing it. Rebooting and I had wireless. Note, the reason for this is because the Yoga 900 does not have a hardware rfkill switch like some Lenovo products do. So without a quirk for the Yoga 900, the kernel was interpreting the rfkill switch as always being on, and failing to enable the wireless.

With wireless working, I could do my normal

apt-get update; apt-get upgrade

to pull in any fixes since I made my USB key. Since I wasn’t going to be currently using the Ubuntu-provided linux-image-generic kernels, I went ahead and removed those packages first, to minimize the download size.

To get the touchscreen and touchpad working, I needed to apply the 16-patch series posted at I git-cloned Linus’ tree and applied the series, building a 4.4-rc4-based tree. That series is planned for v4.5, btw, so the patching should only be necessary for one more cycle.

Booting into that kernel, the touchscreen and touchpad worked! I proceeded to remove the kernel from the PPA, since it wasn’t fully functional, and now I’ve got a pretty nice setup.

Remaining on my list:

I also purchased a new external monitor and USB docking station from Dell. Based upon my reading of the specs, I think they should work, once I get a USB-C (the only video output on the Yoga 900) to standard USB adapter (the docking station and monitor both came with the rest of the cables). That’ll be my next post!