2016-12-10

Why are EPEL python packages getting renamed?

Someone joined the irc.freenode.net #epel channel on early Saturday (2016-12-10) and asked:


[2016-12-10-07:33 UTC]  hi, it seems that python-pip was renamed to python2-pip, is there any documentation available why that choice was made?

By the time I got to IRC (12 hours later), they had left so I am going to try and answer them here. . Python packages in EPEL are made like every other package from Fedora. Currently Fedora is working on finishing making Python3 their default python and moving the older python to python2 naming. In doing so python packages are either being named python2 or python3 depending on which chain they are built against. This is going to have an effect on the names of Python packages in EPEL for EL6 and EL7 as the changes occur. For more information on Fedora Python packaging guidelines please see the following link.

2016-12-01

Puppet not coming back to EPEL EL-6

I made a grievous mistake in my previous blog post about the removal of Puppet Software and would it be coming back to EL-6. At the time, I misread someone's IRC comment and thought it was. However after a couple of days of not seeing any updated builds in Koji, I decided to check and found out I was wrong. The maintainers (and upstream) will not be putting puppet back in EL-6 for a couple of good reasons:

  1. The version that was in EL-6 was actually a very old version no longer maintained or updated by PuppetLabs. 
  2. Puppetlabs maintains a set of packages which will work on EL-6 distributions and focuses their attention there for problems and bug fixes.
So what to do? The upstream recommends that users of puppet use the one available from from Puppetlabs called puppet collections. The documentation on setting up the repository and using it are pretty clear, but for security sakes I will add a couple of steps.

$ sudo -i
# mkdir /root/puppet; cd /root/puppet
# wget https://yum.puppetlabs.com/puppetlabs-release-pc1-el-6.noarch.rpm

if you want to try and check the signature before installing you can add the following steps:

# wget https://yum.puppetlabs.com/RPM-GPG-KEY-puppetlabs
# rpm --import RPM-GPG-KEY-puppetlabs
# rpm -K puppetlabs-release-pc1-el-6.noarch.rpm 
however this does mean altering the rpm database before you check to see if it is ok.

# yum localinstall puppetlabs-release-pc1-el-6.noarch.rpm

if you are wanting a local copy of the repository for other boxes to install from you can use 'reposync'. For more on configuration management, you might also want to look at or join the CentOS Configuration Management Special Interest Group

[UPDATE: The old packages for puppet are still in Fedora Koji at http://koji.fedoraproject.org/koji/buildinfo?buildID=609117 ]
[UPDATE2: For better instructions on how to carefully check GPG headers of packages before you install them willy nilly, try the following:
http://orcorc.blogspot.com/2008/08/gnupg-few-minutes-on-using-detached-and.html ]

2016-11-28

Abiword for EL-7

Over Thanksgiving break, I decided to go through a long list of emails that were marked "when you have a spare moment". I really didn't have one but I realized that many of those emails were crufty and old. One was some people asking about getting abiword together for EL-7. This looked like a straightforward enough task so I got into it and started working out all the packages that would need to be branched to say EPEL and what would be needed to compile them.
At first the packages were pretty easy to bring over from Fedora:

  • ./aiksaurus/aiksaurus-1.2.1-34.el7.centos.src.rpm
  • ./gtkmathview/gtkmathview-0.8.0-19.el7.centos.src.rpm 
  • ./libwps/libwps-0.4.4-1.el7.centos.src.rpm 
  • ./wv/wv-1.2.9-13.el7.centos.src.rpm 
  • ./loudmouth/loudmouth-1.5.3-1.el7.centos.src.rpm  
  • ./ots/ots-0.5.0-12.el7.centos.src.rpm
Then we got to the ones which didn't compile so cleanly.
  • ./minisat/minisat2-2.2.0-5.fc18.src.rpm
  • ./link-grammar/link-grammar-5.0.8-3.el7.centos.src.rpm
  • ./abiword/abiword-3.0.1-12.el7.centos.src.rpm

The latest minisat2 does compile but I was getting all kinds of compilation errors with link-grammar-5.3 in using it. Backing down to an older version of minisat2 and link-grammar I could get other parts to compile but I could not get abiword to compile. Through some pointers from Yaakov  Selkowitz who had dealt with this package in Cygwin, I was able to get it compiled. I then opened an email to the link-grammar people who figured out that there was a bug in their code when compiling on CentOS or Scientific Linux.  They haven't spun a new release with the fix yet but when they do I can update the packages.  At the moment, the abiword package has to be compiled with the --disable-introspection flag which probably limits some functionality.
In the end, we have a set of packages now available for testing a copr:


[smooge-Abiword]
name=Copr repo for Abiword owned by smooge
baseurl=https://copr-be.cloud.fedoraproject.org/results/smooge/Abiword/epel-7-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://copr-be.cloud.fedoraproject.org/results/smooge/Abiword/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1

Good luck and let me know if it works for you.

Where has puppet gone in EPEL-6

This week various people using EPEL on RHEL and CentOS 6 have found that the puppet package is no longer provided by EPEL. The reason for this is due to the way EPEL packages are built and kept inside the repository. A package needs a sponsor so that we can hopefully get bug fixes and security updates to it. In the case of puppet this package is sponsored by the user kanarip. However, most packages aren't whole pieces, they rely on other software.. in this case the package puppet relies on a lot of different ruby gems of which one of them was called ruby-shadow. This package was orphaned 30 weeks ago and while it did have other people watching it, none of them took over the package.



Depending on: ruby-shadow (3), status change: 2016-04-13 (30 weeks ago)
        puppet (maintained by: kanarip, domcleal, gchamoul, georgiou, jehane, jpo, lzap, mmagr, moses, rharrison, skottler, stahnma)
                puppet-2.7.26-2.el6.noarch requires ruby-shadow = 1.4.1-13.el6

        puppet-gluster (maintained by: averi, purpleidea)
                puppet-gluster-0.0.3-1.el6.noarch requires puppet = 2.7.26-2.el6

        puppetlabs-stdlib (maintained by: averi, gchamoul, purpleidea, shlomizadok)
                puppetlabs-stdlib-4.5.1-2.20150121git7a91f20.el6.noarch requires puppet = 2.7.26-2.el6

Last week a large cleanup was done to clean out orphaned packages from EPEL which removed ruby-shadow. Once that was removed, then all of the other packages depending on ruby-shadow were also removed. Today various people reinstalling systems found puppet wasn't around and came onto #epel to ask.. which seems to have gotten the packages responsored and hopefully they will be back in the EPEL release in a day or so.

This problem has been happening a lot lately. I think it shows quite a few problems with how EPEL is set up and managed. For this, I take responsibility as I said I would try to clean it up after FOSDEM 2016 and it is still happening.

[Correction 2016-12-01 added. Please see my updated post.]

2016-11-16

Another tale, apropos nothing again

When Nema got back to her village she found that her house had nothing in it for planting the next year. Her parents were dead, and her younger brothers were used to drinking and gambling at the tavern house with what was left of the inheritance.

Now the Duke's fields had been cleared recently, and it was the law that peasants could gather any seed left over from the harvest. Nema asked her brothers if they would come help her do so. 

"Not I" they said, for they had games they wanted to play.

So Nema went through the fields and gathered as much seed as she could get. 

When the spring came, she needed to plant the seeds. She asked her brothers if they would help.

"Not I" they said, for they had ale they wanted to drink.

The grain grew and it needed to be harvested. Nema asked her brothers again if they would help.

"Not I" they said for their was both ale and games to think about.

The grain was harvested and milled at the miller. It was time to bake the bread, but none of the brothers were around to help out. Yet when the bread was just out of the oven.. who were to show up? All the brothers asking for bread.

Now Nema could have said something like "Thems that works gets to eat." but she was kind hearted and liked to share. So she gave them loaves from the oven and what did they cry out? "We wanted pumpernickel and this is just plain rye". 


A tale apropos nothing

Once long ago, there was a war where many soldiers went off to war. In those days, win or lose if the war was over you were dropped out of the army and walked your way back to your home countless miles away.

Now there was one formerly conscripted soldier named Nema who walked through the forests and she came upon a village. Now Nema was very hungry but had no money as whatever payment was either long gone. She begged from house but found that the villagers did not like soldiers and told her to go somewhere else. In fact, the villagers were tired of soldiers and fighting and looting and always being the ones trodden on. They didn't even like each other that much because they were all sure that someone had been a collaborator sending soldiers to their house to take their last food.

When the villager had gone to all the houses, Nema sat in the square wondering what to do. Her grumbling stomach told her that she couldn't walk to another village. She had nothing to trade as the only weapon the army had let her take back, an axe, had broken when she had cut wood for her meal to villages back. Looking at the head of the axe an idea sprang into her mind. Nema remembered that one of the villagers had a large cauldron in her front yard.. probably to clean clothes or something..

Going to the villager hut, Nema asked again if she could borrow the pot to make a soup. She would clean it out as a service but she had a magical axe which just needed some water and a fire to make the tastiest soup ever. The old villager eyed Nema as crazy, but because the cauldron was outside and it did need a good cleaning thought it would be an even score. Nema cleaned the pot and gathered water from the village well. She got a fire going and put her axe in it.

The villagers started to gather around this crazy soldier and with much laughter asking how the water soup tasted. Nema would spoon out a bit and say "hmm it is OK but it just needs something to flavor it up a bit." One of the villagers thought and said "Oh I have some old beans.. maybe that will do it." They got the beans from where ever they were hidden and the soldier put it in the pot. Again the villagers asked "oh how is the soup". The soldier stirred some more and said "Oh the beans helped a lot but now it needs a balance." Another villager remembered some old carrots and onions they had. These were added to the pot. This went on for a while and shortly afterwords there was a wondrous soup for everyone to share and eat.

After all the soup had been divvied up, Nema took her magic axe head out of the bottom and put it back in her pouch. She would make it back to her own village many miles away, and this villagers would talk with each other and make many soups from what they had through the long winter.

2016-11-10

Updating Nagios in EPEL-7 (Looking for Testers)

One of the projects on my plate this year is to update the Fedora Project nagios server so that monitoring is done through a better template system in ansible. [The main reasons for using nagios versus going to another system are the following: a) We already have nagios in place and have used it for years, b) it doesn't use/need a database backend for either its webpages or monitoring systems, c) before I can compare other systems I need an updated version of nagios to know what features we would get.]

In looking at Nagios, I found that the current maintainer was not responding to tickets. In fact at the time I was looking for him, their email domain wasn't registered in DNS or whois. While that has been fixed, there has been no response to emails so I have had to start the Non Responsive Packager process. The other item found was that we had various problems in the 4.0.8 package which were only supported and fixed in the current version 4.2.2 . While I started working on this, a person with the nick of Petaris showed up in #epel on freenode with the same problem. They needed to get an updated package into place and were looking for help on what was needed to update the package. Jason Tibbitts helped him set up an fedpkg environment and worked through a couple of problems that were happening. Last week, I looked over the package some more and realized there were problems with the old layout.

The upstream Nagios package assumes that it has complete write access to /var so it drops various configuration and control files in the %localstatedir configuration point. That doesn't work in things like Fedora (and probably Debian?) where it was chosen to put all of that in /var/log/nagios/ . This allowed for everything to be in one spot but as the Nagios code gets more complex more subdirectories with different needs occur (including it looked like /var/log/nagios/log/). I reworked the patches so that they allowed for /var as the %localstatedir and then put various control files in /var/spool/nagios and the log files and their archives in /var/log/nagios. [This may be the wrong place, please let me know and I can fix.]

Petaris took the patches and some other comments and put together a koji build that has the test code in it. Work was also put together on updating nrpe to version 3.0.1 and nagios-plugins to 2.1.3. There is an selinux problem we are trying to work together with a policy that should allow people to work until it is fixed upstream.

In order to make testing easier, I have created a copr for all the current Nagios packages. Currently it works on EPEL-7.

dnf copr enable smooge/Nagios_Update # if you have CentOS dnf
yum copr enable smooge/Nagios_Update # may work also.

Depending on the feedback I am getting on this, I can then work on updating the packages in EPEL-6 and Fedora Rawhide. [This is also my first copr in a long time so I may have built things in a less than optimal way.]

2016-09-08

PSA: Fedorahosted.org will be end of lifed in 2017.

Earlier this week Kevin Fenzi posted to the Fedora announce list that fedorahosted.org would be going away in early 2017. Fedora started offering FedoraHosted in  2007 years before GitHub and such were around. It was based around using Trac from https://trac.edgewall.org/ which fit with the pre-git use of SVN  that many projects used for source code control.  [Trac also had plugins for many other revision control systems as this was a time frame where there were many and people had very strong opinions of which ones needed to be supported.]

In any case, Fedorahosted was very useful for a long time, but in the last 2-3 years has seen a fall off in usage with many projects moving to GitHub. Many of the other projects had become nesting grounds for various spam and some malware. This became very evident during the summer Spam Storm when 27000+ spam pages were created in a couple of tracs. In cleaning those out, I found that around 40 other tracs had various 'tickets' with spam in them that were regularly 'updated' by the spammers for years but the project owners had been '/dev/nulling' any tickets so weren't aware that it was happening.

As outlined Kevin outlines in the email, what will happen is the following:

  1. No new fedorahosted projects will be created.
  2. Owners of projects will be sent an email letting them know that the sunset is going to occur on February 29, 2017.
  3. The owners can then move the project to another source repository system. Examples of these would be:
  4. On February 29, 2017, the files will be put into read-only mode and no further updates will be done. 
As with all such end of life projects, I expect that after number 4 there will be "Hey, no one told me that this was going to happen." emails which will require some sort of 5.. but I am hoping that regular blogs and posts and emails will keep that to a minimum.

2016-08-18

PSA: How to unsubscribe from a mailing list

So you have gotten onto some mailing list which was entertaining at first, but is now a drag, time sink or just filling up your mailbox with gigabits of unwanted emails. How do you unsubscribe?

What you do not do is send an email to the list with the word unsubscribe in the body. 

From: "Bob Smith" bob@smith.mail 
Date: Thu, 18 Aug 2016 01:56:16 +0200
To: A Mailing List mailing-list@lists.place.org 
Subject: Re: some subject
unsubscribe

This email does not unsubscribe you from the vast majority of mailing list software out there but instead sends your email to unknown hundreds or thousands of people who are all going to get this email and are now going to fill your mail box with emails that will tell you to not do that but never really explain how to unsubscribe. [They may also tell you do things with yourself or farm animals which are anatomically impossible or potentially life dangerous. People are weird in thinking that is helpful.]

A long, long, long, time ago as in before 1988.. many mailing list software programs would unsubscribe you if you did exactly what Bob did above. When an email went to the list software it would look through all the text and if it found the words unsubscribe it would do that. Of course if you sent an email using that word for some other reason you ended up off the list with little idea why. The next version of the mailing list software would just do this if the unsubscribe was by itself like Bob did above. This worked "ok" in the world of RFC822  where what you saw is what you sent and nothing special was added. It didn't work for in the world of MIME, HTML or other formats (say not an ARPA email but a BITNET email) as the Internet became "mainstream" around 1994 or so. At this point a lot of people were accidentally unsubscribing or being maliciously unsubscribed by trolls who would forge emails saying they were bob@foo.org but weren't.

By the time the third generation of mailing list software came around in 1998, most software does allow for a naked unsubscribe to get you off the list. Instead you either have to go to a specific webpage and unsubscribe or have to email a specific alias that will start the process of un-subscription. For the web it is usually a multi step process of you go to link, you log in, you unsubscribe and you get a confirmation email that you were unsubscribed. For the email address you will usually get an email sent to you asking if you really want to unsubscribe and if you do email back with a special one time code included in the email. This is all because of the ever present malicious troll problem of people who think it is loads of fun to harass people either because that person exists or for the 'lols' which they seem to trade under their bridges for self-validation.

Which brings us back to how do I know how to really unsubscribe from a list. Every email has a bunch of 'hidden' headers that are included which tell various software where the email maybe came from and where it is going. It will also tell software extra data like how to display it and can contain things like how to subscribe, find an archive or unsubscribe. The current standard way this is done is with List-Id: headers. Here are some examples:


Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm
Precedence: bulk
List-Id: cygwin.cygwin.com
List-Unsubscribe: mailto:cygwin-unsubscribe-BOB=bob.mail@cygwin.com
List-Subscribe: mailto:cygwin-subscribe@cygwin.com
List-Archive: http://sourceware.org/ml/cygwin/
List-Post: mailto:cygwin@cygwin.com
List-Help: mailto:cygwin-help@cygwin.com, http://sourceware.org/ml/#faqs

or

Precedence: list
Reply-To: server@lists.fedoraproject.org
List-Id: server.lists.fedoraproject.org
Archived-At: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/457OVFFRFAC7ASPHWZ5OTCH7GACAUYYE/
List-Archive: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/
List-Help: mailto:server-request@lists.fedoraproject.org?subject=help
List-Post: mailto:server@lists.fedoraproject.org
List-Subscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-join@lists.fedoraproject.org
List-Unsubscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-leave@lists.fedoraproject.org

So from the above we have an unsubscribe for ezmlm mailing list system which is to send an email to a specific email address. For the Fedora lists we use Mailman 3 which has both a mailing list address and a web page to go to start the unsubscribe process.

How do you find these hidden headers? It depends on the mail software you are using. Most of them have some sort of "Show original" or "Show Source" which will pop up a new window which will show a lot of headers. In other mailing list software, you will see a button which says "unsubscribe" which will look for the headers and then fire off the things it says in there.

[Edited 2016-08-18] I realized I completely forgot the obvious way to unsubscribe from Mailman lists. At the bottom of every email from a Mailman email list contains the follow:




--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org


That link at the bottom is all you need to click on to get on the road to un-subscription. You will be directed to a page that will ask you to login. You can do so through one of several methods which will get you to a page that looks like this

Click on unsubscribe, and you should be unsubscribed from the list. It is a lot more steps than just typing "unsubscribe" in an email, but in a self-service world that has a lot of trolls.. it is what you end up doing.

PSA: How to unsubscribe from a mailing list

So you have gotten onto some mailing list which was entertaining at first, but is now a drag, time sink or just filling up your mailbox with gigabits of unwanted emails. How do you unsubscribe?

What you do not do is send an email to the list with the word unsubscribe in the body. 

From: "Bob Smith" bob@smith.mail 
Date: Thu, 18 Aug 2016 01:56:16 +0200
To: A Mailing List mailing-list@lists.place.org 
Subject: Re: some subject
unsubscribe

This email does not unsubscribe you from the vast majority of mailing list software out there but instead sends your email to unknown hundreds or thousands of people who are all going to get this email and are now going to fill your mail box with emails that will tell you to not do that but never really explain how to unsubscribe. [They may also tell you do things with yourself or farm animals which are anatomically impossible or potentially life dangerous. People are weird in thinking that is helpful.]

A long, long, long, time ago as in before 1988.. many mailing list software programs would unsubscribe you if you did exactly what Bob did above. When an email went to the list software it would look through all the text and if it found the words unsubscribe it would do that. Of course if you sent an email using that word for some other reason you ended up off the list with little idea why. The next version of the mailing list software would just do this if the unsubscribe was by itself like Bob did above. This worked "ok" in the world of RFC822  where what you saw is what you sent and nothing special was added. It didn't work for in the world of MIME, HTML or other formats (say not an ARPA email but a BITNET email) as the Internet became "mainstream" around 1994 or so. At this point a lot of people were accidentally unsubscribing or being maliciously unsubscribed by trolls who would forge emails saying they were bob@foo.org but weren't.

By the time the third generation of mailing list software came around in 1998, most software does allow for a naked unsubscribe to get you off the list. Instead you either have to go to a specific webpage and unsubscribe or have to email a specific alias that will start the process of un-subscription. For the web it is usually a multi step process of you go to link, you log in, you unsubscribe and you get a confirmation email that you were unsubscribed. For the email address you will usually get an email sent to you asking if you really want to unsubscribe and if you do email back with a special one time code included in the email. This is all because of the ever present malicious troll problem of people who think it is loads of fun to harass people either because that person exists or for the 'lols' which they seem to trade under their bridges for self-validation.

Which brings us back to how do I know how to really unsubscribe from a list. Every email has a bunch of 'hidden' headers that are included which tell various software where the email maybe came from and where it is going. It will also tell software extra data like how to display it and can contain things like how to subscribe, find an archive or unsubscribe. The current standard way this is done is with List-Id: headers. Here are some examples:


Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm
Precedence: bulk
List-Id: cygwin.cygwin.com
List-Unsubscribe: mailto:cygwin-unsubscribe-BOB=bob.mail@cygwin.com
List-Subscribe: mailto:cygwin-subscribe@cygwin.com
List-Archive: http://sourceware.org/ml/cygwin/
List-Post: mailto:cygwin@cygwin.com
List-Help: mailto:cygwin-help@cygwin.com, http://sourceware.org/ml/#faqs

or

Precedence: list
Reply-To: server@lists.fedoraproject.org
List-Id: server.lists.fedoraproject.org
Archived-At: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/457OVFFRFAC7ASPHWZ5OTCH7GACAUYYE/
List-Archive: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/
List-Help: mailto:server-request@lists.fedoraproject.org?subject=help
List-Post: mailto:server@lists.fedoraproject.org
List-Subscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-join@lists.fedoraproject.org
List-Unsubscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-leave@lists.fedoraproject.org

So from the above we have an unsubscribe for ezmlm mailing list system which is to send an email to a specific email address. For the Fedora lists we use Mailman 3 which has both a mailing list address and a web page to go to start the unsubscribe process.

How do you find these hidden headers? It depends on the mail software you are using. Most of them have some sort of "Show original" or "Show Source" which will pop up a new window which will show a lot of headers. In other mailing list software, you will see a button which says "unsubscribe" which will look for the headers and then fire off the things it says in there.

[Edited 2016-08-18] I realized I completely forgot the obvious way to unsubscribe from Mailman lists. At the bottom of every email from a Mailman email list contains the follow:




--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org


That link at the bottom is all you need to click on to get on the road to un-subscription. You will be directed to a page that will ask you to login. You can do so through one of several methods which will get you to a page that looks like this

Click on unsubscribe, and you should be unsubscribed from the list. It is a lot more steps than just typing "unsubscribe" in an email, but in a self-service world that has a lot of trolls.. it is what you end up doing.

2016-08-17

EPEL PSA: Mock builds broken

If you are having problems with building packages with a target for EPEL, it is not because the EPEL gpg keys have been tampered with, but because the mock developer made a mistake in the packages

The problem occurs because mock by default builds EPEL packages with CentOS and the mock developer put the CentOS keys as being the keys to compare EPEL packages with. He is working on a fixed set of packages and the problem should go away with mock-1.2.20 [The broken version is 1.2.19]

2016-08-05

Fedora: How a release is archived. [Fedora 22 moving to archives]

This week has been the week of Fedora Flock in beautiful Krakow, Poland. Due to other travels, I was unable to attend which meant that I have been focusing this week on getting some outstanding tasks done that had been delayed on my part. One of those tasks was getting Fedora 22 ready for being permanently archived.

A Fedora release has many stages in its life.

  1. An embryo in rawhide
  2. A larva in alpha/beta/rc modes
  3. A pupa at release time 
  4. A short mature life as an imago while the next release  is in its pupil stage. 
  5. Then like all things, the release shuffles off this mortal coil and gets moved to the afterlife in the Fedora Archives.   
The Fedora Archives are a hidden resource for people needing to find some older package or to try and track a history of packages. I use it quite extensively for building packages for EPEL ( a post for later ) or to try and find the 'last' workable version of some software as various things in Fedora evolved past the Enterprise Linux release.

The steps for archiving a release are pretty straight forward:
  1.  Before version N+2 goes into beta, hardlink copy with cp -l the version N from /pub/fedora/linux/releases/N into /pub/archive/fedora/linux/N. This allows for any mirrors who do slow updates of the archive tree to start getting the data moved over in their trees.
  2. A couple of weeks after version N+2 has been released, version N is End of Lifed and no more updates will be done. At this point we can do a hardlink copy of /pub/fedora/linux/updates/N and /pub/fedora/linux/updates/testing/N to /pub/archive
  3. A week later we log into the mirrormanager tool and change the point where F-N updates are pointing to the archive version. This means that systems still looking for updates will not get 404's but will be directed to a mirror of the archives.
  4. We then remove the data from the main Fedora disk space.
We can do this because /pub/fedora, /pub/archive and /pub/epel are all on one disk which allows for us to hardlink them together and if a mirror is doing a rsync -avSHP of the trees they should not have to transfer as much data because of the hardlinks. 

In any case, I have started the archive of Fedora 22 and next Friday will complete the archival by moving the pointers in mirrormanager. Thank you Fedora 22 for your service.
Cicada, shell, upper marlboro, md 2014-07-10-19.57.12 ZS PMax
An end of life release

2016-08-04

Fedora: Fixing fonts

Someone on twitter commented that I should write an article about how to fix fonts on Fedora. Sadly this is not something I am able to do. That doesn't mean there isn't a problem, just that I don't have the font specialist eye where they can look at a screen and go "OMG THERE IS SOMETHING WRONG HERE!!!!" in the way that someone who is slightly flat or sharp drives me bonkers. [I on the other hand can happily read a webpage in Comic Sans or Papyrus and wonder why everyone looking over my shouldr is cringing and hissing as I do so.]

That said, I did some research on why fonts look like 'crap' compared to Ubuntu and Windows. It looks like the reason is that there is an option in many fonts called "subpixel rendering" which are not enabled in Fedora but is enabled in other distributions. This looks to be done due to Microsoft having various software patents on ClearType which do not end until 2019. [Other patents ended earlier but those only cover the older TrueType fonts.]

There seem to be fixes that will work for people outside of areas that are covered under current software patents (which for all I know is nowhere because of some super secret trade agreement).

  1. Look at ask.fedoraproject.org for steps to use. 
  2. Look at installing and enabling RPM fusion on your system.
  3. Follow the steps in 1.
Or you can look at something like fedy which I am going to point to the source code for. I don't recommend using the method listed on their website because using a blind su "curl" is a recipe for disaster. Instead I would recommend looking at the code, seeing if you would want to run that on your system and then doing it by hand. 

[I don't care that all the cool kids use a bash -c 'su - "curl ... && run" thing with their nodes and their rubies. I would end with a rant about emacs being good enough for anyone and that the kids are on my lawn.. but you all see that coming anyway.]

2016-08-03

Fedora Statistics: Questions and answers.

I work for Red Hat as a system administrator for the Fedora Project where I get to do a lot of neat and interesting things daily. One of the tasks I have is gathering various statistics for the Fedora Project Leader's state of the hat speeches like the one he has just given at Flock. This means I also get to help answer regular questions on mailing lists and irc channels like "How many users does Fedora have?", "How many downloads of Fedora are there?", "How does this compare to ?"

All of these questions are where Matthew Miller pulls out a clip from a dinosaur movie eating a lawyer on a toilet or similar comedic point. Why? Because raptors live here and will eat you if you do not have a proper escape policy.

The question "How does this compare to ?" is the easiest to answer: "Nothing I give you can be compared to what any other distribution probably says."  This doesn't mean I or they are lying... it just means the terms being used aren't well defined and we are probably using slightly different ones. 

What is a user? Someone who created a Fedora account, did something and never logged in again? Someone who created a Fedora account and logs into FAS daily? weekly? monthly? yearly? Maybe that person who never logs in just answers things on IRC or bugzilla or mailing lists? Maybe the person who logs in daily is just a script that does that because a developer decided to test a cron script and then forgot about it.

What is a download? If I go by raw "GET .*iso .*" in various logs I could say we have had millions of downloads weekly. But anyone who knows weblogs knows that is as fishy as the 1990 website counters and being told "We got N million hits". Those downloads are inflated because of several reasons:
  1. Some people mirror everything. They may not install any of it.. but just in case they ever need it.. they have it.
  2. Some people try to be helpful in promoting their OS by downloading the OS over and over again so that any count of downloads will be larger than the next guys. There are multiple IP addresses which download Fedora isos hourly. No one needs 24 copies of Fedora 8 every day... especially when they had just downloaded 24 copies of Fedora Zod. 
  3. Some "web companies" do the same and then send us mailers about how we can see how they have increased our downloads and if we used them we could see even further growth. 
  4. A lot of people use specialized download tools which try to torrent downloads via http. What they do is ask 20-100 times for a mirror and then use each one to download a bit of the file as a speed booster of some sort. This shows up as even more downloads.
  5. Then there are the people who are stuck on NAT over NAT or satellite links. They may show up as a dozen or so IP addresses in their attempt to grab a single IP address. 
On the other side there are multiple people behind a single NAT so that the 200 downloads from IBM are probably not the same system.. but maybe they are? There are ways to get clearer results via various 'fingerprinting' techniques but I don't think that they really can help when you have companies whose basic job is to bump up numbers so you can lie about how good your product is doing on the web. [I am going to avoid commenting on the morality of deep fingerprinting because I am in a rather cranky mood today.] Depending on how one looks at the data, you can keep discounting stuff further and further down until your only answer is that you know that you had more than 1 download and less than whatever your max amount of non spider hits were. 

So what does that leave us with so many "unknown unknowns" and too few "known knowns"? What we have been using has been not looking at downloads at all and instead focus on actual users who have installed the OS and are using yum or dnf to update their systems. This can give us a rough lower bound number of 'active' systems. Going through the mirror logs we create a large amount of tuples of (date, ip address, hardware arch, fedora release). We then unique this list to deal with the various users who have cron set up to do a yum update every 10 minutes. It has a bad effect of making the thousands of systems behind the Red Hat NAT be counted as 1 system, but we hope that is made up for by the person doing a yum update over their Verizon phone and showing up as 20 ips as they get ip shifted every now and then. 
Fedora OS yum checkins (oldest highest)

Now again these are 'reasonable' minimum numbers. The N thousands of ARM IOT systems with Fedora on them do not do yum updates so aren't counted. The systems which are hard configured to use a local mirror aren't counted. And so on and so on. 

[Caveats:
  1. The above graph is a hack job I created using awk and gnuplot with a 7 day moving average calculated via python pandas. I expect that it could be made prettier or cleaner through various ways. 
  2. The drop off in late 2008 was due to the webservers being mostly off during the security incident of that time.
  3. The hockey stick drop off in late 2014 is due to Fedora dropping support for RC4 encryption and various systems hard coded to use it not being able to check in any more. All of these systems were way past End of Life for each release so they were not getting any updates.
  4. Most releases have a growth pattern of continually growing until the day the next release comes out. That pattern changes for Fedora 8, 14 and 22 which seem to have continued to grow even after they were end of lifed (or close to it). These seem to be due to some VPS, cloud or similar provider continually basing images off of these releases.
]

I think somewhere in this post I lost my original focus. Sorry about that.. I looked at the picture and got all oooh I need to talk about that.

2016-08-02

Fedora PSA: How to know you are seeing a SPAM web page.

So one of the things that I really really hated about the current crop of SPAM on Fedora was that it completely lacked any originality or usefulness. The pages were usually hundreds of repeating lines saying something like:

customer support
technical support 
customer service
1-8[0-9][0-9]-xxx-xxxx 

over and over again seems to be a useless waste of electrons for anyone landing on the page. Especially when the web page says something like

c*u*s*t*o*m*e*r s*e*r*v*i*c*e

or some similar text. My first guess was that these are either used as landing text pages for some sort of malware to pop up some sub-section of the webpage as a problem alert. You have a problem with your . However that made less and less sense in that the pages did not seem to have any encoding for the malware to say 'this is the string to use today'. It also didn't make sense in that it was clear that this was being inserted many times by a cut and paste versus program insertion (someone would double paste the text into a web page but with the text intermingled). 

My next guess was that it was probably related to SEO ratings for some other search engines. Sure enough there was a pattern of various search engines that seemed to be triggered shortly after a set of webpages were created. They would specifically add the various pages and shortly afterwords a search at the search engine for " help" would show up the usual one word line like:

Call Support 

at the top of the search list linked to our website. Most of the searches that I found through the webpage logs were either for google.com or redirects from bit.do websites (actually the only bit.do redirects for over a month were for these pages.)

In any case, back to the subject title. If you are seeing a large page of text with various small variations on 'search terms' to get you technical support for product.. and a phone number DO NOT CALL IT. Especially if the page you are looking at is NOT for the product you are looking for.

Repeat after me:

  • For actual QuickBooks support go to https://quickbooks.com
  • For actual Cisco support go to https://cisco.com
  • For actual Microsoft product support go to https://microsoft.com
If you are ending on some other website, it is most likely some sort of scam.

Thank you

2016-08-01

Fedora Inactive/Disabled Accounts

In a recent #fedora-admin IRC conversation, it was asked if Fedora Infrastructure ever makes accounts inactive. We have done this a couple of times in the past,  usually after a security breach where password hashes may have been seen. While happily we have not had a security event happen in a while.. we did have a large amount of spam accounts created which have taken up a bunch of names that people may want to use (if you have wanted vipin1 -> vipin357, nagar1 -> nagar240, or many others, you are currently out of luck). We also have many users who created accounts, and either never logged into FAS again or haven't for over a year or more.

Now for some projects you would regularly set these accounts to inactive or clear them out for reuse. Free software projects usually do not have the luxury for doing this for several reasons:

  1. The Norwegian Blue Parrot Problem.
    In this case, we find that many users have not been deceased, but like a true Norwegian Blue have been pining for the fnords. This means that even a year after you have set accounts to inactive there is a large amount of "why is my account locked?" emails and general grumpiness of people finding that someone had locked their account without their permission. [It also leads into the argument sketch and other too much silliness.]
  2. This account was used for something in the past and we need to keep track of things for licensing reasons. While someone might want the account thegreatandmightyoz and that account is no longer active.. if that account put in patches or other items into Fedora.. those inclusions may be tracked via the userid in one of the many Fedora sub-systems. [Yes they should be tracked by the UID or some other non-changing ID, but that is 10 years too late for some inclusions.] Because of this if someone else gets the 'thegreatandmightyoz' they could they could get additional responsibilities (what do you mean I need to fix this bug? I am just a circus man from Kansas not a real wizard!)
For the second problem, we are looking at making changes in the new FAS system so that we can give users an 'EPOCH' where 0:thegreatandmightyoz is different from 1:thegreatandmightyoz in various systems trying to figure out what is linked to whom. For the first problem, we will work out a system of saying that a user who has not logged in for N days (where N is greater than say 270 days ) will be set for inactive. We would then advertise quite a bit before doing it and then get ready for a long list of arguments and complaints :).

2016-07-28

Fedora PSA: Why is my account listed as spamcheck_* (Updated)

In my previous post I talked about the fact that Fedora infrastructure has been undergoing a large amount of spam that caused us to put in extra circuit breaker programs to try and slow down the account creation and web spam. After 3 months of constant attrition, we eventually had to change the policy on the wiki that people with new accounts could not open up web pages without being sponsored by an existing group. That caused an immediate drop in web spam, and 12 hours later a drop in account creation. However not a cessation in either of them which has made it not possible to remove the circuit breakers. My guess is that some amount of accounts are being stockpiled for if the wiki permissions are made open again so that they can start putting wiki spam in. 


Above we have a link for account creation for 2016. Normally we see spikes during convention shows and such but in March we saw a large number of accounts created.. many of which were not used until June. At this point the accounts were flagged as spam when pages created had various "Quickbooks", "Norton Antivirus", "Cheap Electricity" or "Emergency Printer Support" pages created on the wiki and one of the circuit breakers tagged the account to lock down. Other accounts created by the same IP address were never used so my guess is that accounts were forgotten or were to be used for some other "campaign" in the future. 

If you do find your account to be listed as spamcheck_denied or spamcheck_manual, you can contact admin@fedoraproject.org and please list the following information:
  1. The email address you used for creating the account. [We have a lot of people who use one account name for account creation and another to report the problem. This makes tracking down the account info easier for us.]
  2. The account name you requested.
  3. If possible the IP address you used when creating the account. The website icanhazip is extremely useful here.
We may request other information that you inputted to try and confirm that it is you requesting the spamcheck cleanup. 

2016-07-04

Fedora PSA: Why is my new Fedora account listed as spamcheck_waiting or spamcheck_denied?


The Fedora Project is currently undergoing a massive spam account creation and insertion problem. In the month of June, we saw over 14,000 accounts created just to put in various Quickbook and Antivirus scam web pages in order to get higher SEO search rankings in various search engines. There have also been some malicious pdf and docx files uploaded which would have potentially harmed people. The groups behind these campaigns have shown themselves to be organized and are using teams of people to solve various captchas and other "I am a human" tests to create more accounts.

The amount of spam caused several parts of the project to actually get delisted from various search engines with warnings that we needed to clean up the pages. However the amount of spam being created was more than our usual scripted cleanups as various pages would not be found until the next google or bing crawl. [I spent this weekend cleaning up pages from late 2015 and early 2016 which finally got 'used' and showed up in weblogs.]

Due to the nature of this, Fedora Infrastructure has had to implement multiple circuit breakers to slow down registration and web page creation to try and cut down the amount of bad accounts and webpages being inserted.  Like any diagnostics test, it is not infallible and produces both  false positives and false negatives.

In the case of false negatives, we eventually will get alerted by someone looking through the wikis and finding the spam we weren't able to find. In the case of false positives, we have closed an account for a bad review of the account.

If you have gotten spamchecked, please email admin@fedoraproject.org, and we will review the logs to see why this happened. This will take time however as we have found that most of the requests for review are now coming from the spam account openers.

I am sorry for the delay and problems this last 2 month's have become a "Tragedy of the Commons" where a small set of people have taken up a large amount of resources for their own benefit.

[For more details on the statistics and the various circuit breaker programs written, please look forward to various future blogs from various members of the Fedora Infrastructure.]

2016-04-06

Reminder: PSA: Enterprise Linux 5 End of Production on 2017-03-31 and EPEL.

So in less than 12 months (2017-03-31), Red Hat Enterprise Linux 5 will be leaving end of production level 3 . At that time EPEL support will be ending for EL-5 and all packages will be moved to the archives section of the fedoraproject download servers.

If you are relying on (CentOS|ScientificLinux|Oracle|Red Hat) Enterprise Linux 5 for your business needs, you should look at moving the to either a newer version of Enterprise Linux or at the extended support program that Red Hat offers as listed in the above article.

I will be publishing my next PSA on this in 6 months.

2016-03-25

Attention Sites Mirroring Fedora

We are seeing a large number of partial rsync on the main download
servers which are causing problems for tier0/1 mirrors to get to the
servers. What we are seeing is that an ip address will start a rsync
of a large tree and then will drop the connection as the server takes
time to work through 1-10TB of disk space. The ip will then initiate a
new connection which another download server will try to fulfill but
again with a time-out and restart. This is adding a large amount of
IOPs onto our backend storage causing a cascading set of problems
through the infrastructure.

We are going to have to put a firewall rules to drop connections from
these systems so that the alpha and other work can get properly
mirrored onto the registered Tier 0 and Tier 1 mirrors. If that
doesn't work we will be putting firewall rules that only Tier 0 and
Tier 1 mirrors are allowed to connect to the download servers.

I will be trying to send emails to the registered abuse for the ips, but if you are getting dropped connections to the download servers or are a mirror administrator at one of the following domains:

  • wideopenwest.com
  • alshamil.net.ae
  • yandex.net
  • sl-reverse.com
  • univ-ubs.fr
  • ip.pt

please register in the Fedora mirror manager so that it can be whitelisted. 


Finally, to cut down mirroring problems please do the following:

1) Do not put in a cron job that you are going to do an rsync update
every 15 minutes as several of the above mirrors seem to do. We do not
update the trees that often and rsyncd has to stat every file.

2) Please review the tips and tricks at
Using the last-sync to schedule updates when they actually occur can
help lower rsync usage.

Thank you

2016-02-24

EPEL Proposal #4: Build it all in CentOS promote to RHEL

So this one came up when we were going over problems with building alternative architectures in EPEL. One of the big problems is that currently everything is built against Red Hat Enterprise Linux. This has some bonuses for some people but it causes a problem when architectures are built against by CentOS but not Red Hat. [E.G. EL-7 for i386 and arm32] When a Red Hat release is done, the builders usually get updated as part of our repopull from the RHN servers. Usually there are some changes in added and subtracted packages which can cause packages to get new dependencies. While this is an inconvenience to customers.. it isn't a problem for the builders. When we start mixing distributions, it does cause a problem with the builders because a package may get compiled for EL-7 x86_64 and not be the same as arm32 or i386 due to the fact that buildroots for i386 are not as up2date as the RHEL ones. This can cause builds all across the system to break so it is frowned on.

A couple of proposed fixes were:

  1.  Compile everything against CentOS. This has the bonus that they all kind of get updated at the same time so breakages from going from 7.3 to 7.4 is smaller. However, a lot of business users have either ITIL or security plans which allow for EPEL packages to be used on their systems because they were built against RHEL versus a rebuild. Remove that and it causes various problems for users.
  2. Split off the builds for alternative architectures in a shadow koji which is turned off when these splits occur. This makes it partially easier to manage but brings in other problems.
  3. ... some other ways to fix this that I don't know how to describe correctly...
However while we were hashing on 1, someone else who had been asking about why don't we build everything came up with a different mix. Build everything against CentOS for various architectures and put it in some other repository (Extra Packages in CentOS came to mind). Packages which people really want and are willing to care for are branched off into EPEL packages which are built against RHEL. [Or if they really want it they get a business case to Red Hat to make it a product they will pay for it.]

There is a lot of problems with the above.. it requires a lot of work in many different moving parts to get done. It also could be even more confusing than what we have already. However, I did say I was going to write up all the proposals for people to look at. 

EPEL Proposal #3: EPEL Structured releases

This started off as a a rawhide post, but I realized it needs to be its own post

A staged/forked environment. This version is a lot more complicated and deals with extra release management. Currently, Red Hat does regular point releases (.1, .2, .3) of their main OS. In these point releases, various items get major updates or backports to bring new functionality to the base operating system. In this version, EPEL ties into this staged environment by using a similar method as Fedora does.

Using a directory tree structure like:
EPEL/archives/{5.x, 6.x, 7.x}
EPEL/archives/updates/{5.x, 6.x, 7.x}
EPEL/current/5 -> 5.11/
EPEL/current/5.11/{x86_64,ppc64,x86_32}/
EPEL/current/6 -> 6.7/
EPEL/current/7 -> 7.2/
EPEL/updates/5 -> 5.11/
EPEL/updates/6 -> 6.7/
EPEL/updates/7 -> 7.2/
EPEL/updates/testing/{blah, blah}
EPEL/rawhide/{5, 6, 7}

Then packages would be built in rawhide first. When a new release (say 7.2 to 7.3) occurs, then the builders rawhide trees are updated to the code in the latest. The packages are built against that and hopefully tested to see that they still work. Packagers should have been building any large changes they needed to make in the rawhide tree already but a last minute notification date is done. The go-live for this tree will be either when CentOS gets its build out of that tree or 30 days if CentOS is able to release sooner. On that date the rawhide trees will be branched to /EPEL/current/{6,7}.n (eg 7.3) and updates to those packages will now be sent into EPEL/updates/7.3 . After another 30 days, the 7.2 tree will be moved to EPEL/archives/7.2 {probably really /pub/archives/epel/7.2 }

Pros: this gives a place where newer packages can be built and tested to see what may break in the next major build. it also gives a better release cycle that people can plan against.

Cons: this makes the previous post look simple to implement.

EPEL Proposal #2: EPEL Rolling Rawhide

An EPEL rawhide may sound like an oxymoron for having an always building main tree for a release that is known for its plodding slowness. However there are methods to the madness as people brought it up at FOSDEM Brussels.

Rolling Release Style. In this version there is a branch where many Fedora packages are rebuilt against EL-5/6/7. The packages in this release are always the latest that can be compiled against the release and all 'updates' and fixes are done upstream versus in the package. The epel file tree would look something like this:

EPEL/{5,6,7}
EPEL/testing/{5,6,7}
EPEL/rawhide/{5,6,7}

Packages in EPEL/{5,6,7} are ones that are going to be kept stable until they can no longer be supported. Packages in EPEL/testing/ are were updates to the mainline packages are kept. The general idea is that Fedora packagers can freely fork to EPEL/rawhide with no care about whether it works or not. If people are interested enough to help fix it they can send it to the packager but if it doesn't build and no one fixes it.. it isn't there. If people do care about the package, they can sponsor it and those would be the ones in EPEL regular. These packages would follow the agreed upon rules about updates and lack of big changes.

Pros: Allows for more packages to be newer for more users.
Cons: More channels for release engineering and packagers to deal with.


PSA: Enterprise Linux 5 End of Production on 2017-03-31 and EPEL.

Red Hat Enterprise Linux builds its current releases around a 4 part lifecycle model.

  • Production 1. New drivers and features are added during this cycle along with security and production bug fixes.
  • Production 2. Only security and production bug fixes are done during this cycle.
  • Production 3. Only security and 'critical' bug fixes are done.
  • Extended Lifecycle Support. Security and critical bug fixes are only released to customers who are paying for this extra level of support. For rebuild Linuxes like CentOS and Scientific Linux the product is now End of Lifed.


The last time this happened was in 2012 with the end of life of Enterprise Linux 4. At that time, EPEL stopped supporting builds for EL-4 but no other changes were done. This wasn't a big deal because EL-4 had never gotten a large number of users. However in 1 year, 1 month, EL-5 will reach the end of its production run and move into Extended Life Support. Enterprise 5 is still over 20% of all EPEL installs so I wanted to give a long heads up that EL-5 will also be removed from the builders on April 1 2017 and no builds will be done after that. Depending on other proposals, EL-5 may also be moved over to an 'archive' like Fedora releases are so that it is clear that it is no longer under any production. If that happens there will be clear notice of where it has been moved to and how to keep at it.

2016-02-16

EPEL proposal #1: rechartering

Background

So my time at FOSDEM was pretty much the following. Walk into the hall where the Fedora/CentOS tables were. Get met by someone who worked on or in EPEL. Try to have a conversation and find that it was too packed and loud there. Walk over to the coffee area and buy the person a coffee and listen. Finish conversation and walk back over to the table.. [take a break every now and then to use the restroom from drinking a 16 oz tea every 30 minutes.] This was a lot of conversations and they did all blur together after a while. [Using a technique I learned from Centos' Karanbir Singh at FOSDEM, I will bring a small notebook to the next conference and write down a problem per page per person. Then I can go back the next year and see if the problems are still there.]

Promises

While they all did blur, there were many similar questions from people that came up:

  • What are the promises that EPEL makes to its users?
  • Do those promises make sense? [Do we need to keep a package the same for 5+ years?]
  • Do we actually keep those promises?
  • Who thinks we should be keeping these promises that various people didn't feel were needed?
  • What promises can we make that make sense and that after ~8 years we feel we can reasonably keep?
  • Who do we need to make these promises to?
As I said in the previous post, a lot of people felt that we had made promises of fixing and updating things in EPEL over the last 3 years but nothing really seemed to have changed. There are a lot of reasons for that (as in people say things and after a 20 hour flight have no idea what they said to looking at the promise and going "ok that is fine but who is going to do the work?" to "are we even allowed to change this?"

The last point is to me an undercurrent of a lot of conversations in and around EPEL in the last 3 years. There is the feeling that we were chartered nearly 10 years ago to do certain things, and we should keep doing them because people depend on that. And it is true that some people do depend on it, but it seems increasingly that they either don't show up to help or are a smaller set of people than we realize. 

These questions and the problems above was a long discussion between Brian Stinson and me on Sunday evening. One of the circles that kept coming up was "Does EPEL need to be rechartered?" EPEL is a very old Fedora SIG, and it has gone through several incarnations in the last 8 years from when it had yearly elections for board members to when it ran without any leadership to its recent experimentation with a new steering committee. However during that time there have been many rules which it has tried to keep:
  1. Do not do disruptive upgrades. Try to backport security fixes or do upgrades which do not change API/ABI issues.
  2. Be prepared to support the same package for the life of an Enterprise Linux release.
  3. Never replace or conflict with packages in the base Enterprise Linux.
  4. They should never require manual updates or changes between updates.
  5. Follow Fedora rules for bundling, licensing, and similar things.
There are some promises some people feel like we have made also:
  1. Packages will never disappear. [They don't disappear from Fedora 12 even if it is archived.. ]
  2. Packages will work from EL-X.0 to EL-X.N. [EPEL will rebuild packages regularly to deal with any ABI items]
  3. Packages are built against all of an Enterprise Linux 'base' (EG whatever is in CentOS/Scientific Linux base).

In many cases those promises aren't kept now or when they are attempted to be kept are impossible to do.
  • Most software which uses a database back end requires schema updates ranging from dump/reload to long running change processes. 
  • A lot of software rebases itself internally to the point that trying to backport a security fix usually requires a lot of coding skill in the package.
  • The Enterprise Linux lifetime has increased from 5 years to 10 years since EPEL started. While people were interested in supporting a package for 3-4 years.. doing so for 10 years is too much for a volunteer position.
  • The Enterprise Linux is not the same as it was in EL-3/4/5 where software versions were rarely 'rebased' to a newer release. Instead software would be recoded to work in the older software as much as possible. Currently in EL-7, the desktop and other parts of the OS were greatly updated between 7.1 and 7.2 with the idea that this will be the norm for all releases in order to keep the product relevant to the majority of users. 
From the above, 3 of the 5 promises are either too much to keep or haven't been kept in a long time. 

What to Do?

Because we haven't been able to keep many promises, but feel like we should there seem to be a couple of options ahead:
  1. Ignore and do nothing. I don't like the status-quo but I think it needs to be listed as something that can happen.
  2. Try to keep the original charter and be a lot more picky about what is in EPEL so that we can meet those promises. 
  3. Figure out what we can do, and go to the appropriate Fedora body to recharter ourselves to meet those more reasonable goals. 
Now number 3 may seem to be busy work, but I have found for the conservative minded enterprise maintainer, it is very hard to give up doing something even if you are failing until you are told you can stop trying. It also makes it clearer to future people working on EPEL in 2024 what they are trying to solve and how to change when the world of 2024 needs different solutions.

What would be in my charter


  1. EPEL is run by the EPEL Steering Committee. They act as sub-equivalents to the Fedora Extras Steering Committee, Fedora Packaging Committee and Fedora Release Engineering. 
  2. Packages in EPEL will never replace or conflict with packages in the base Enterprise Linux.
  3. Packages in EPEL will follow Fedora rules for bundling, licensing, and similar things. Exceptions to this will exist and will be documented in appropriate place.
  4. EPEL release engineering can set up systems to rebase packages in regular intervals. [There are multiple ways of doing this that I will try to outline in other proposals this week.]
  5. The only promise of a lifetime of a package is between dot releases of the Enterprise Linux. Maintainers should make an honest effort to support a package for the "life" of a sub-release (6.8->6.9) but no longer. 
  6. Packages in a dot release will not disappear during that time. There is no promise after that time. [If EPSco approves of a method that packages are regularly archived, then users can use that as a simpler method to get old packages.]
  7. EPEL only supports the current dot release. If a package in EL-6.now works on 6.0.z that is great. If it doesn't, it is up to the user (and not the maintainer or EPEL) to deal with it. [Again if an archive method is setup, then those older versions would just be in something like /pub/archives/epel/epel-6.8/ or some similar setup]
Those are my starting points for a rechartered EPEL. Please join in the discussion on the EPEL-devel mailing list. 

EPEL talks from CentOS-Brussels Dojo/FOSDEM and Configuration Management Camp EU

Apologies and So Forth

First, I would like to apologize for the delay in getting this post done. I really didn't realize the amount of energy the trip would take from me and how fuzzy brained I was for a week afterwords. Second, I would like to thank the Open Source and Standards (OSAS) group at Red Hat for sponsoring me that week. I believe I got a lot of things done and that we can work on getting Extra Packages for Enterprise Linux (EPEL) moving forward in some direction. Third I would like to thank everyone who took the time to talk to me at one of these places to tell me about what they were able to do with EPEL and what they were no longer able to use it for.

One of the items that people brought up a lot was that they felt conversations about fixing/growing/changing EPEL have become a broken record. Various things are said at many conferences by various people, but nothing ever gets done on this to the point that they feel they have been lied to. After succumbing to the fuzzy headed "what did I say yesterday?" for a week after flying back, I think that most of these broken promises have come up from jet-lag and people overload. If it hadn't been that I had people from IBM, CERN, and various other groups bring this up over and over again.. I may have forgotten just as well. In any case, I am going to try and outline every item that I wrote down on many pages of notes in this blog. If I have forgotten some point that you wanted me to bring up, please send me an email so that I can make sure it is dealt with.

Things EPEL is doing well

As can be seen in the following graph, EPEL has a large audience and a continually growing one. 
EPEL Growth from 2007 til 2016-01
The primary user of EPEL packages are on Red Hat Enterprise Linux (or a clone like CentOS or Scientific Linux) version 6 with a slowly shrinking set of EL-5 users. The audiences that use EPEL are very diverse. Everything from large organizations with strict audit controls to fast moving startups which have to work with the large organizations as their customers to the university trying to do some experiments to match things done in other organizations. 

Many users brought up the fact that the win of EPEL is that it is a low energy move to get packages from Fedora rebuilt for an Enterprise environment.  The fact that the package spec files had been vetted and tested in Fedora made for less chances of conflicts or configuration problems that crept into their own spec files or ones they got off the internet from various groups.  

Things EPEL is not doing well in

This is going to be a rather long list of items that came up. Some of them conflict with each other but that is mainly due to the fact that the customers are so diverse.
  • The packaging guidelines for each EPEL version are not clear. This is mainly due to the fact that they are usually based off of older versions of Fedora (Fedora-6 for EPEL-5, Fedora-12 for EPEL-6, and Fedora-18 for EPEL-7) that may conflict with each other or with how things are being done in current EPEL. 
  • EPEL does not have a regular release structure like Fedora and RHEL have. There isn't an EPEL-5.11 channel with an epel-updates-5.11 where updates are available. Because of various repository limitations, this means that directories aren't able to keep multiple old copies so downgrades when things do break aren't easy. 
  • EPEL promises to keep things stable and only update for fixes, but this is only done on a few packages where others get upgraded to and fro. There does not seem to be much "steering" or "release engineering" of what is in the trees. 
  • EPEL only covers part of Enterprise Linux (the Server product) but a lot of packages are for the Workstation but there is no way to see when things replace/conflict with them. [People believe that we build against the equivalent of CentOS-5/6/7 versus a subchannel.]
  • EPEL sometimes has weird breaks between releases. The git in EPEL-5 is newer than what is in EL-6 in a way that was breaking repositories when pushes were done from EL-5 systems.  [People believe there is a promise that such changes are tested against.]
  • EPEL packages disappear. If there is no maintainer packages are retired but if someone is re-building out the EL-5 hosts.. they need that package to be available.  [People believe there is a promise that packages will be always available.]
  • From various people who were (or former) EPEL maintainers: packages in EPEL are the most complained about. If they can't update the package in EPEL, they get complaints about it being too old and they may end up having a newer version available for what they need to do anyway. If they do update the version, they get complaints that they broke someone because they needed some old version. Most of the complaints usually are the worst kinds ( off-hand death threats (why don't you die in a fire) etc etc). 

Things people wanted in EPEL

There were various things that people were wanting in EPEL that weren't really breakages because they don't exist yet.
  • Enable 'alternative' architectures. There were requests from people about CentOS-i386 and CentOS-arm (for the Raspberry Pi2 in schools) with no vision on how those would be enabled. 
  • Enable more packages with shorter lifetimes. One of the things that multiple sites were doing was rebuilding all of a Fedora release for their internal mirrors to supplement all of the things that EPEL didn't have in it. Why can't there be an EPEL-rawhide where all of these packages are built but no 'promises'?
  • Is it possible to have a branch management system where packages get updated regularly? Say either whenever Fedora moves to a new release or Red Hat updates to the next branch (6.7->6.8, 7.2->7.3)
  • Could packages in EPEL be tested in the CentOS continuous integration infrastructure as part of the autokarma testing?

Conclusions

I don't think anything above is new to people who have been contributing to EPEL in the last ~10 years. A lot of the problems are ones that were brought up in the beginning as we tried to square the circle of differing use cases. However, I wanted to catalogue them here and then make a promise that I will do my best to figure out ways to solve them by FOSDEM 2017 in some form or another. 

As I said at the beginning of the post, I believe that people had other complaints and suggestions. If I forgot or miswrote, my apologies and I will correct.


2016-01-27

Whither EPEL (Talk at FOSDEM 2016)

If you are going to FOSDEM 2016 and use CentOS, Scientific Linux, Red Hat Enterprise Linux or even Oracle Enterprise Linux.. you should be interested in a round-table talk we are having on Sunday to talk about the Fedora Project's Extra Packages for Enterprise Linux (EPEL) . EPEL is a repository which has a curated set of Fedora packages rebuilt for EL-5, EL-6, and EL-7. 

Things We Will Talk About:
  1. Who are our consumers? A consumer is someone who uses EPEL but doesn't have the time, knowledge, etc to help build, test, or curate packages. [This is a place for audience members to bring things they do (or don't do) with EPEL.]
  2. Who are our contributors? A contributor is someone who builds, tests, writes documnts, etc for the EPEL SIG. [This is a place for people who build and test packages in EPEL can speak up.]
  3. What are the "promises" that we as a SIG believe that we are making to our contributors and consumers? What are the "promises" that contributors and consumers think we are giving?
  4. Do our rules and guidelines still make sense? [When the project was set up, there were many debates about what people wanted and the projects rules were built out of a general consensus of what people would put up with.]
  5. What about other ways to get these packages? Atomic, docker, software collections, COPRs, etc. Are these meeting peoples needs better and what can we learn to meet those inside of EPEL or branch out to meet them in other ways.
In general tradition, we will have a reenactment of West Side Stories meetings of the Sharks and the Jets while we discuss repotags. [I kid, I kid.. it will be more of Montague's and Capulet's.

Brussels CentOS Dojo

This Friday (2016-01-29) there will be a great CentOS dojo in Brussels Belgium. I am going mostly to help out run the cameras and announce different talks. I will also be wanting to listen to people use who use EPEL about what they use it for and what they are wanting to see in growth. This will help me prepare for the talk on Sunday at FOSDEM which I will be moderating.

If you have questions please try and find me. I will probably be the one fellow in a tie and sports coat. [And because I am from the desert.. no rain coat which sounds like a minus currently in Belgium].

This is my first time in Belgium and I will be trying to see a little of Brussels on Saturday and a bit of Ghent on Monday. If I had a day or two more I would have been going to visit Veere in the Netherlands so that I could try and see some of the places written by favorite author Hendrik Willem van Loon. I will just have to do this next time I am visiting western Europe. 

[I would like to write a bit longer, but I am stuck in the Atlanta airport blaring some inane news channel talking about the 'latest' election non-crisis. ]