1.5 Year Warning: Python2 will be End of Lifed

The end of upstream Python 2.7 support will be January 1, 2020 (2020-01-01) and the Fedora Project is working out what to do with it. As Fedora 29 would be released in 2019-11 and would get 1.5 years of support, the last release which would be considered supportable would be the upcoming release of Fedora 28. This is why the current Python maintainers are looking to orphan python2. They have made a list of the packages that would be affected by this and have started a discussion on the Fedora development lists, but people who only see notes of this from blogs or LWN posts may not have seen it yet. 

Here is my current status on things:
  1. The epylog package will most likely be dead.package in rawhide. The package would need a complete rewrite to work with Python3 and it has not been touched by upstream, nor do I have time to do the port.
  2. I don't own the nagios-plugins-fts or nagios-plugins-lgcdm but if they ended up on my plate I would do the same thing.
  3. python2 is still supported in RHEL-7 and will be until 2024. epylog and any python scripts inside of nagios packages I own will be supported in EL7.
If your project currently requires python2, you need to either help your upstream move to python3, find a company or service that will maintain python2.7 for future years, or be prepared to spend years in the wilderness. From the many years and projects I supported Fortran IV, perl4, Java 1.2, and ADA83 code in the early 2000's... I expect that this will be where many people are going to be. [I will try to cover how to handle that in a future blog.]


Dealing with network hackers in 1995

Going back to early 1995, I was working for Los Alamos National Labs as a contractor systems administrator. I didn't have a security clearance so could not work 'behind the fence' as they said. Instead, I worked with a large number of similarly uncleared post-docs, graduate students, and college interns in a strip mall converted into offices. The offices ran from nearly one end of the strip mall to the other with a large selection of Unix, PC, and Mac systems spread through the building connected together with 10base2 (or thin-wire). To make things even more fun, most of the systems were disk-less SunOS Sparc ELC/SLC and IPC systems booting off a Sparc 10 which had 64 MB of RAM and I think 2 2 GB disk drives.

The first problem I had to deal with was my most of the systems would crash at different times during the day. I got a Digital network book my Dad had given me, and learned about common problems with networking as this was not something I had dealt with before. I found that the local network was connected to a T1 which ran back to the main campus about 2 miles away. The T1 went to a hub which had 7 thin-wire lines running out of it. That seemed fine until I traced the thin-wire out. I was worried there were bad connectors (there were) or kinks in the line (there were) but the real problem was that out of the 7 thin-wire lines 3 were used.  Most of the systems were on one line. 2 (my desktop and the Sparc 10) were on another one, and the Next and SGI's were on the third. The other lines were just laying under the carpets not used. I met with my new boss Dale, and showed him what I had found. I learned a lot from Dale. He got me a copy of the Unix System Administrators Handbook and told me to start reading it on networks.

So I spent the next week or so learning how to properly crimp and connect thin-wire. I also learned testing signals and ohm resistance as I found connectors which didn't work very well. I moved all the Windows 3.11 and Macintosh 6? systems over to one set of cables and then spread the disk-less stations over to other lines in order to try and diagnose the crashes. At first, I didn't have a crash for 2-3 days and I thought everything was licked. And then the next Monday, things started crashing again.

I started investigating, but we started getting reports in a different building of a similar problem. I found a server with no disk space because of a file in /usr/tmp which seemed to be random data. I believe this was late January 1995, and I had been putting in 60-80 hour weeks trying to get caught up with things. [I was an hourly and this was allowed overtime so I was paying off my college debts pretty quickly.] I hadn't watched TV or read USENET in weeks and had no idea that there was a massive computer hunt at the time. Now when I had worked at university, I would have probably deleted the file and moved on, but because I was supporting scientists I didn't want to delete some research. I contacted Dale, and he helped me work out that the file was being written to by the rshd or telnetd command. He had me check the Ethernet port and sure enough it was in promiscuous mode. Now I had been slow up until this point but I realized this was not a good thing.

Now back in 1995, nothing on these networks was encrypted. You used telnet or rsh to login into systems. You had fingerd running on every box because you used it to see who was logged in. Most networks were on hubbed networks so you could just listen on one system and hear everything in your /24. Dale started contacting the people in security and I started looking at what other boxes were bad. It turned out several had this but one of the ones in my area which had crashed still had the source code for the sniffer daemon on it. I looked at it and found that it linked to the equivalent to tcpdump and filtered for for anything typed after Password: so it caught telnetd, rshd, ftpd and also the su command.

It then kept whatever came afterwords til a NULL and xor that data to the file it opened. I think the xor was against  libc random() stream of data using a starting string as the seed. [The string was then freed I guess to make sure gdb couldn't find it.] To de-crypt the data you just ran the same command with I think a -u and the same password. You could set it up to log even more data at compile time so I was trying that out to see if there was anything I could see about what had been captured on different systems.

At this point I had gotten it running on the system and was looking at what it was capturing when along came the attacker. They logged into the system as a normal user and ran a program ... I had missed which was setuid. They then checked what they had captured and logged out. In doing so I had captured the password they were using. I also had found that they had been running finger against a bunch of systems looking for a particular login. It turned out that a prominent researcher had collaborated with a professor at Los Alamos and the systems with their accounts were being targeted. I turned this all over to my boss who asked me if any of that meant anything to me. It didn't. He told me to go read some particular Usenet groups and that we would have to reinstall everything. [I believe he had been trying to get everything installed, updated and secured for years but it had been a low priority in his management chain.]

For a while it was thought that the attacker may have been on particular person, but these days I doubt it very much. There were a lot of people trying to be 'helpful' at that time by doing various things, and I expect it was one of them. It was an eye opening experience and I wanted to capture it here for the amount of naivete we had back then:

  1. No firewall because it was meant to be an educational section where students and professors should just be able to telnet from their University to whatever computer they needed.
  2. Most of the passwords were stored in a centralized space anyone could look at the hashes for. The Sun yellow page system was useful in mass deployments but had its limits. All of the systems stored their passwords in /etc/password so if you got onto the system at all you could see the hash.
  3. Networks were generally hubbed so that anyone could listen to anyone else locally. This was considered less of a problem when only Unix systems were around because there was a 'separation' of root and user so you could 'control' who could look at the network. The growing number of Mac and PC's which allowed anyone to listen made this the next part hard.
  4. Network communication was mostly in the clear. This was due in part because encryption was expensive on CPUs but it was also that encryption was export controlled so no one wanted to use it in case they got in trouble for it 'leaking'. 
  5. Most systems came out of the box with many network services turned on which never needed to be. Just as these days, your Facebook or LinkedIN account starts off public to the universe, your computer would share if you were logged in, where you had logged in from, what your .plan might have in it and a dozen other things. 
  6. That security problems tend to travel along lines of social trust. The attackers were following along various people who had worked with one researcher and using each set of systems to jump to the next ones. One of the computers hacked was done via a talk session where someone asked someone else if they could help them. I think the victim thought they were helping their adviser with a temporary .rlogin and it escalated from there. 
  7. While Howard Tayler would put this better years later, I found that in security, failure is always going to happen. What is important is how to react to it. I made sure that I could always backup and restore systems en-mass if needed. I also made sure that I have a plan B and C for when the next security problem occurs. 
It would be great if that naivete was unique to that time frame, but we constantly reinvent ways to think we are completely safe.. only to find we covered ourselves in steak-sauce and laid down in the hungry lions pit. 

Addendum: I am recalling things from over 20 years ago and like a fish tale.. some things which make me look better will have grown in that time, and other things which would make me look like an idiot will have faded away.. 


How to install EPEL Packages

What is EPEL?

EPEL stands for Extra Packages for Enterprise Linux. As I have stated before, the Extra Packages are rebuilds of packages from various Fedora Project Linux releases with an aim to keep EPEL packages slower moving than what is in Fedora. The Enterprise Linux in this statement is aimed at the Red Hat Enterprise Linux and the many rebuilds of it (CentOS, Scientific Linux, Oracle Linux and sometimes Amazon Linux). [As opposed to the enterprise offerings from SuSE or Canonical.] 

How to set up EPEL?

In order to enable the EPEL repositories, you need to do the following things. 
  1. Determine which version of an Enterprise Linux you are using by looking in the /etc/system-release file. On my CentOS-6 system it looks like  
    [root@el-6 ~]# cat /etc/system-release
    CentOS release 6.9 (Final)
    and on my CentOS 7 system it currently looks like
    [root@el-7 ~]# cat /etc/system-release
    CentOS Linux release 7.4.1708 (Core) 
  2. If you are running CentOS or Scientific Linux you can now simply install and enable EPEL with a yum command:
    [root@el-7 ~]# yum --enablerepo=extras install epel-release
    [root@el-6 ~]# yum --enablerepo=extras install epel-release
    This will install the release keys and files with a package which has been signed by your distribution for you to have a chain of trust.
  3. If you are running Red Hat Enterprise Linux, you will need to do some extra steps. We will use EL-7 but they would be similar for EL-6.
    1. Read the page at https://getfedora.org/keys/faq/
    2. Download the RPM and GPG key and confirm they are valid.
      # wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
      # wget https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7
      # rpm --import RPM-GPG-KEY-EPEL-7
      # rpm -K epel-release-latest-7.noarch.rpm
      # rpm -K epel-release-latest-7.noarch.rpm 
      epel-release-latest-7.noarch.rpm: rsa sha1 (md5) pgp md5 OK
  4. If you are more of a curl | su - type person then you can just install directly from the internet using the rpm command. I don't recommend this but it gets asked a lot.
  5. You can now list the packages using yum list which will give more packages. These can be installed with the normal install methods. If you did not already import the gpg keys, you will be prompted to trust these keys.
    [root@el-7 ~]# yum install pax-utils
    Loaded plugins: auto-update-debuginfo, fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirror.yellowfiber.net
     * epel: archive.linux.duke.edu
     * epel-testing: packages.oit.ncsu.edu
     * extras: mirror.yellowfiber.net
     * updates: mirror.yellowfiber.net
    Resolving Dependencies
    --> Running transaction check
    ---> Package pax-utils.x86_64 0:1.2.3-1.el7 will be installed
    --> Finished Dependency Resolution
    Dependencies Resolved
     Package          Arch         Version          Repository             Size
     pax-utils        x86_64       1.2.3-1.el7      epel-testing           96 k
    Transaction Summary
    Install  1 Package
    Total download size: 96 k
    Installed size: 249 k
    Is this ok [y/d/N]: y
    Downloading packages:
    pax-utils-1.2.3-1.el7.x86_64.rpm                        |  96 kB  00:00:00
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : pax-utils-1.2.3-1.el7.x86_64                           1/1
      Verifying  : pax-utils-1.2.3-1.el7.x86_64                           1/1 
      pax-utils.x86_64 0:1.2.3-1.el7
  6. Sometimes you may need to install a package which has not made it to stable yet. Please see my earlier post on that.
  7. I have been told by various people in IRC that they were told by someone else that current version of Amazon Linux was based off EL-6 so they want to use the EL-6 EPEL for it. We have also had lots of reports of people finding things not working at times. I would recommend that any users of Amazon Linux to use what Amazon recommends first.

Side Note:

I have been told multiple times that the EPEL logo looks like a horse's tail swatting flies. I actually think it looks more like a full sail being blown as our original want was to have a trading ship  or even a container ship (this was years before containers) but none of the attempts looked good. This one was provided by the art team as a combination of an backwards E and a P/L. 



I have not done a general biography in a long time so figured I should put one out as a courtesy for people reading these blogs and emails I send out on various lists:

Who Am I?

My name is Stephen Smoogen, and I have been using computers for a very long time. According to a bug in a PHP website I was on years ago, I am over 400 years old which would mean I was born in Roanoke island with Virginia Dare. I think I slept a lot in the next 360 years as, according to my sister, my parents found me in a 'DO NOT RETURN TO SENDER' box outside their door. How she knew that when she is younger than me, I do not know.. but I have learned not to question her.

My first computer was a Heathkit Microcomputer Learning System ET-3400 that my Dad got at a swap meet when he was in the Navy in the 1970's. I had worked with my Dad on some other systems he fixed for coworkers but it was mostly being bored while watching an oscilloscope and moving probes around boards every now and then. When I wanted to get a computer in the early 1980's, he said I had to show that I could actually program it since an Apple ][ would have been a major investment for the family. I spent the summer learning binary, hexadecimal and doing the simple machine code that the book had in it. I also programmed a neighbour's Apple ][+ with every game I could in the public libraries Creative Computing 101 Basic Games. My mom and dad saved up for an Apple and we got an Apple ][e in 1983 which I then used through high school. The first thing I learned about the Apple ][e was how different it was with the ][+. The older systems came with complete circuit diagrams and chip layouts. It had been the reason my dad wanted to get an Apple because he knew he could fix it if a chip went bad. The ][e did not come with that and boy was Dad furious. "You don't buy a car with the engine welded shut. Don't buy a computer you can't work on." It seemed silly to me at the time, but would be a founding principle for what I do.

During those years, I went with my dad and his coworkers to various computer clubs where I learned how to play hack on a MicroVax running I think Ultrix or BSD. While I was interested in computers, I had decided I was going to university to get a degree in Astrophysics.. and the computers were just a hobby. Stubborn person that I am, I finally got the degree though I kept finding computers to be more enjoyable. I played nethack and learned more about Unix on a Vax 11/750 running BSD 4.1 and became a system administrator of a Prime 300 running a remote telescope project. I moved over to an early version of LynxOS on i386 and helped port various utilities like sendmail over to it for a short time.

After college I still tried to work in Astrophysics by being a satellite operator for an X-ray observation system at Los Alamos. However, I soon ended up administrating various systems to get them ready for an audit, and that turned into a full time job working on a vast set of systems. I got married, and we moved to Illinois where my wife worked on a graduate degree and I worked for a startup called Spyglass. I went to work for them because they had done scientific visualization which Los Alamos used.. but by the time I got there, the company had pivoted to being a browser company with Enhanced Mosaic.

For the next 2 years I learned what it is like to be a small startup trying to grow against Silicon Valley and Seattle. I got to administer even more Unix versions than I had before, and also see how Microsoft was going to take over the desktop. That was because Enhanced Mosaic was at the core of Internet Explorer. At the end of the two years, Spyglass had not gotten bought by Microsoft, and instead laid off the browser people to try and pivot once again as an embedded browser company at a different location. The company was about 15 years too soon for that as the smart items their plans had as the near future didn't start arriving until 2015 or so.

Without a job, I took a chance to work for another startup in North Carolina called Red Hat. At a Linux Conference, I had heard Bob Young give a talk about how you wouldn't buy a car with a welded bonnet and it brought back my dad's grumpiness with Apple decades ago. I realized that my work in closed source software had been one of continual grumpiness because I was welding shut the parts that other people needed open.

Because of that quote, I worked at Red Hat the next four years learning a lot about openness, startups and tech support. I found that the most important classes I had from my college were psychology and not computer science. I also learned that being a "smart mouthed know it all in" doesn't work when there are people who are much smarter and know a lot more. I think by the time I burned out on 4 years of 80 hour weeks, I was a wiser person than when I came.

I went to work elsewhere for the next 8 years, but came back to Red Hat in 2009, and have worked in the Fedora Project as a system administrator since then. I have seen 15 Fedora Linux releases go out the door, and come to really love working on the slowest part of Fedora, EPEL. I have also finally used some of the astrophysics degree as the thermodynamics and statistics have been useful with the graphs that various Fedora Project leaders have used to show how each release and how the community has continually changed.


Explaining disk speeds with straws

One of the most common user complaints in an Enterprise systems is 'why can't I have more disk space?' The idea is that they look at the costs of disks on Amazon or New Egg and see that they could get an 8 TB hard disk for $260.00 but the storage administrator says it will cost $26,000.00 for the same amount.

Years ago, I once even had someone buy me a disk and have it delivered to my desk to 'fix' the storage problem. They thought they were being funny so I thanked them for the paper weight. I then handed it back to them and then tried to explain to them why 1 drive was not going to help... I found that the developers eyes glistened over as I talked about RPM speeds of drives, cache sizes, amount of commands a ATA read/write use versus SCSI, etc. All of them are important but not terms useful for a person who just wants to never delete an email.

The best analogy I have is that you have a couple of 2 litre bottles of Coca Cola (fill in Pepsi, Fanta or Mr Pibb as needed) and a cocktail straw. You can only fill one Coke bottle with that straw. Sure the bottle is big enough but it takes a long time to move the soda from one to the other. That is what 1 SATA disk drive is like.

The next step is to add more disks and make a RAID array. Now you can get a bunch of empty coke bottles and empty out that one array through the multiple cocktail straws. Things are moving faster but it still takes a long time and you really can't use each of the large bottles as much as you like because emptying them out will be pretty slow via the cocktail straw.

The next sized solution is regular drinking straws with cans. The straws are bigger, but the cans are smaller.. you can fill the cans up or empty them without as much time waiting for a queue. However you need a lot more of them to equal the original bottle you are emptying. This is the SAS solution where the disks are smaller, faster, and much better throughput because of that. It is a tradeoff in that 15k drives use older technologies so store less data. They also have larger caches and smarter os's on the drive to make the straw bigger.

Finally there are the newest solution which would be the garden hose connected to a balloon to a coffee cup. This is the SAS SSD solution. The garden hose allows for a large amount of data to go up and down the pipe, the balloon is how much you can cache in case you are too fast somewhere in writes or reads. The coffee cup is because it is expensive and there isn't a lot of space. You need a LOT of coffee cups compared to soda cans or 2 litre bottles.

Most enterprise storage is some mixture of all of these to match the use case need.

  • SATA raid is useful for backups. You are going to sequentially read/ write large amounts of data to some other place. The straws don't need to be big per drive and you don't worry about how much is backed up. The cost per TB is of course the smallest.
  • SAS raid is useful for mixed user shared storage. The reads and writes to this need a larger straws because programs have different IO patterns. The cost per TB is usually an order or two of magnitude greater depending on other factors like how much redundancy you wanted etc.
  • SSD raid is useful for fast shared storage. It is still more expensive than SAS raid. 
And now to break the analogy completely. 

Software defined storage would be where you are using the cocktail straws with coke bottles but you have spread them around the building. Each time coke gets put on one, a hose spreads that coke around so each block of systems is equivalent. In this case the costs per system have gone down, but there needs to be a larger investment in the networking technology tying  the servers together. [A 1 gbit backbone network is like a cocktail straw between systems, A 10 gbit backbone is like a regular straw and the 40G/100G are the hoses.]

Now my question is .. has anyone done this in real life? It seems crazy enough that someone has done a video.. but my google-fu is not working tonight.


Ramblings about long ago and far away

My first job outside of college in 1994 was working at Los Alamos National Labs as a Graduate Research Assistant. It was supposed to be a post where I would use my bachelor's in Physics degree for a year until I became a graduate student somewhere. The truth was that I was burnt out of University and had little urge to go back. I instead used my time to learn much more about Unix system administration. It turned out the group I worked on had a mixture of SGI Irix's, Sun Sparcstations, HP, Convex, and I believe AIX. The systems had been run by graduate students for their professors and needed some central management. While I didn't work for the team that was doing that work, I spent more and more time working with them to get that in place. After a year, it was clear I was not going back to Physics, and my old job was ending. So the team I worked on gave me a reference to another place at the Lab where I began work. 

This network had even more Unix systems as they had NeXT cubes, old Sun boxes, Apollo, and some others I am sure to have forgotten. All of which needed a lot of love and care as they had been built for various Phd's and postdocs for various needs and then forgotten. My favorite box was one where the owner required that nearly every file was set 777. I had multiple emails which echo every comment people come up with Selinux in the last decade. If there was some problem on the system it was because it had a permission set.. and until it was shown it didn't work at 777 you could look at it being something else. [The owner was also unbelievably brilliant in other ways.. but hated arbitrary permission models.]

Any case, I got a lot of useful experience on all kinds of Unix systems, user needs, and user personalities. I also got to use Linux Softland Linux Systems (SLS) on a 486 with 4 MB of RAM running the linux kernel 0.99.4? and learn all kinds of things about PC hardware versus 'Real Computers'. The 486 was really an overclocked 386 with some added instructions that had been originally a Cyrix DX33 that had been relabeled with industrial whiteout as a 40MHz. It sort of worked at 40Mhz but was reliable only at 20Mhz. The issues with getting deals from Computer magazines.. sure the guy in the next apartment worked great.. mine was a dud.

I had originally run MCC (Manchester Computer Center Interim Linux) in college but when I moved it was easier to find a box of floppies with SLS so I had installed that on the 486. I would then download software source code from the internet and rebuild it for my own use using all the extra flags I could find in GCC to make my 20Mhz system seem faster. I instead learned that most of the options didn't do anything on i386 Linux at the time and most of my reports about it were probably met by eye-rolls with the people at Cygnus. My supposed goal was to try and set up a MUD so I could code up a text based virtual reality. Or to get a war game called Conquer working on Linux. Or maybe get xTrek working on my system. [I think I mostly was trying to become a game developer by just building stuff versus actually coding stuff. I cave-man debugged a lot of things using stuff I had learned in FORTRAN but it wasn't actually making new things.]

For years, I looked back on that time and thought it was a complete waste of time as I should have been 'coding' something. However I have come to realize I learned a lot about the nitty-gritty of hardware limitations. A 9600 baud Modem is not going to keep up with people on Ethernet playing xTrek. Moving it to a 56k modem later isn't going to keep up with a 56k partial T1. The numbers are the same but they are counting different things. A 5400 RPM IDE hard-drive is never going to be as good as 5400 RPM SCSI disks even if it is larger. 8 MB on a Sparc was enough for a MUD but on a PC it ran into problems because the CPU and MMU were not as fast or 'large'. 

All of this later became useful years later when I worked at Red Hat between 1997 and 2001. The customers at that time were people who had been using 'real Unix' hardware and were at times upset about how Linux didn't act the same way. In most cases it was the limitations of the hardware they had bought to put a system together, and by being able to debug that and recommend replacements, things improved. Being able to compare how a Convex used disks or an SGI graphics to the limitations of the old ISA and related buses helped show that you could redesign a problem to meet the hardware. [In many cases, it was cheaper to use N PC systems to replicate the behaviour of 1 Unix box but the problem needed to be broken in a way that it worked on N systems versus 1 box.] 

So what does this have to do with Linux today? Well mostly reminders to me to be less cranky with people who are 
  1. Having fun breaking things on their computers. People who want to tear apart their OS and rebuild it to something else are going to run into lots of hurdles. Don't tell them it was a stupid thing. The people at Cygnus may have rolled their eyes but they never told me to stop trying something else. Just read the documentation and see that it says 'undefined behavior' in a lot of places.
  2. Working with tiny computers to do stuff that you do on a bigger computer these days. It is very easy to think that because it is 'easier' and currently more maintainable to do a calculation on 1 large Linux box.. that you are wasting time on dozens of raspberry pis to do the same thing. But that is what the mainframers thought of the minicomputers, and the minicomputers thought of the Unix workstations, and the Unix thought of Linux on PC. 
  3. Seeming to spin around, not knowing what they are doing. I spent a decade doing that.. and while I could have been more focused.. I would have missed a lot of things that happened otherwise. Sometimes you need to do that to actually understand who you are. 


Please end Daylight Savings Time

This was going to be my 3rd article this week about something EPEL related, but I am having a hard time stringing any words together coherently. The following instead boiled out and I believe I have removed the profanity that leaked in.

So I like millions of other Americans (except those people blessed to be living in Arizona and Hawaii) are going through the week long jetlag when Daylight Savings Time starts. For the last 11? years the US has had DST start 2 weeks earlier than the rest of countries which observe this monstrosity, I think to show the rest of the  why it is a bad idea.  No one seems to learn and instead try to make it longer and longer.

I understand why it was begun during World War I in order to make electricity costs for lighting cheaper in factories. It just isn't solving that problem anymore. Instead I spend a week not really awake during the day, and for some reason as I get older not able to sleep at all during the night. And I get crankier and more sarcastic by the day. Finally sometime next Monday, I will conk out for 12-14 hours and be alright again. I would like to say I am anomaly, but this seems to happen to a lot of people around the world with higher numbers of heart attacks, strokes and accidents during the month of time changes.

So please, next time this comes up with your government (be it the EU, US, Canada Parliament, etc) write to your representatives that this needs to end. [For a fun read, the Wikipedia articles on various forms of daylight savings cover the political philandering to pay for this.]

Thank you for your patience while I whine about having only the lack of sleep when there are a hell of a lot worse things going on in the world.