Fedora Infrastructure Meeting Change to Thursdays 1400 UTC

For several years, the Fedora Infrastructure meeting has been held every Thursday at 1800 UTC. This would be lunchtime to morning for the the U.S. members,early evening for our European members, and late night for people in India. [I think it is a different day in China and Japan.]  In order to see if attendance was problematic because of the time, the Fedora Infrastructure leader Kevin Fenzi recently asked for a new meeting time. The results came back in and the meetings will be moved to 1400 UTC on Thursdays. In order to see what the time is in your time zone you can use the date command

[smooge@smoogen-laptop ~]$ date -d "Apr 26 14:00:00 UTC 2018"
Thu Apr 26 10:00:00 EDT 2018
Fedora Infrastructure tries to set its meetings against UTC versus any local daylight savings/unsavings times since many regions do not have them or start/end them at different times.


Fedora Infrastructure Hackathon (day 1-5)

From 2018-04-09 to 2018-04-13, most of the Fedora Infrastructure team was in Fredericksburg, Virginia working face to face on various issues. I already covered my trip on the 08th to Fredericksburg so this is a followup blog to cover what happened. Each day had a pretty predictable cycle to it starting with waking up around 06:30 and getting a shower and breakfast downstairs. The hotel was near Quantico which is used by various government agencies for training so I got to see a lot of people every morning suiting up. Around 07:30, various coworkers from different time zones would start stumbling in.. some because it was way too late to get up in a day, and others because it was way too early. Everyone would get a cup or two of coffee in them and Paul would show up to herd us towards the cars. [Sometimes it took two or three attempts as someone would straggle away to try and get another 40 winks.] Then we would drive over to the University of Mary Washington extension campus.

I wanted to give an enormous shout-out to the staff there, people checked in on us every day to see if we had any problems, and worked around our weird schedules. They also helped get our firewall items fixed as the campus is fairly locked down for guests but made it so our area had an exception for the week so that ssh would work. 

Once we got situated in the room, we would work through the days problems we would try to tackle. Monday was documentation, Tuesday was reassigning tasks, Wednesday was working through AWX rollouts, Thursday was trying to get bodhi working with openshift. Friday we headed home via our different methods. [I took a train though not this one.. this was the CSX shipping train which came through before ours.]

Most of the work I did during this was working on tasks to get people enabled and working. I helped get Dusty and Sinny into a group which could log into various atomic staging systems to see what logs and builds were doing. I worked with Paul Frields on writing service level expectations that I will be putting into more detail in next weeks blogs. I talked with Brian Stinson and Jim Perrin on CentOS/EPEL build tools and plans.

Finally I worked with Matthew Miller on statistics needs and will be looking to work with CoreOS people someday in the future on how to update how we collect data. As with any face to face meetings, it was mostly about getting personal feedback on what is working and what isn't. I have a better idea on things needed in the future for the Fedora Apprentice group (my blogs for 2 weeks from now), Service Level Expectations, and EPEL (3 to 4 weeks from now).


Fedora Infrastructure Hackathon (day 0)

The Fedora Infrastructure Hackathon is currently going on in Fredericksburg Virginia outside of Washington, D.C. My first day was to get to the site from North Carolina which I did via the US Amtrak system. I have not been on a US train in many decades, and was not sure what it would be like. First off, getting onto the train was incredibly easy. I bought a ticket, arrived, and got on the train with other people. The conductors and other staff were friendly and helpful in getting me a seat. The crew were also very helpful to an older gentleman in a wheel chair in making sure he got food and drinks.

Next, the train seating was comfortable and I had plenty of leg room. The leg room on each seat was equal to business class seating in most planes. The person in front of me could lean back quite a bit and not interfere with my long legs. While the chair cushion slid out a bit, it was much more comfortable than the first class I had sat on a major airline recently. The ride was fairly comfortable, there was the general back and forth motion, and 'turbulence' when the train had to move off to a side track, but it was in general a lot smoother than driving I-85/I-95.

The views were very good and it as nice to not have to worry about "stayin' alive on  I-95". The train had a food area which served Dunkin Doughnuts coffee, various premade sandwiches and other foodstuffs. The best part was that the other people travelling to the Hackathon could all sit together and work on things for a while. This was useful due to the 2 downsides: the travel time was a bit longer than expected, and the wifi was incredibly laggy. The travel time was due to having to sit on side tracks 2 or 3 times while a CSX line went past. CSX owns the rails and Amtrak uses those rails as a lower priority than shipping traffic. So every now and then, the amtrak will have to sit on a side while the spice flows. The wifi was mostly due to a lot of people using it and a limited bandwidth available through an uplink. I am guessing it was a satellite link with a cell backup so that when something would block either, you had drops. This was ok for writing local documents, but people working on web mail would switch to music for various times.

All said, I enjoyed the trip. It cost about as much as if I had driven a rental and I didn't have to deal with the headaches of it. I also did not get any motion sickness which I do when other people are driving or I am on a bus. The people working the trip were happy and looked like they enjoyed their jobs. Having seen more than enough flight attendants in the last 2 years who look like they would rather eat glass than another flight.. it was not what I was expecting. The crew also enforced courtesy rules so that when a people started talking too loud on the phone, they were asked to move to another section. When people tried putting their luggage in empty handicap seats, it was removed and the people were reprimanded that this was not acceptable.


Explaining myself with xkcd

April is Autism awareness month, and I thought I would start off with a couple of blogs about what it can be like to have even mild autism. I find xkcd to be a good way to illustrate many different points in conversation, technology, and life. It is where a picture and some words say more than a long essay can.

While I have been told not being good with conversations is a human condition, every day seems to combine the following cartoons together. I either need a checklist to remember what things I need to 'fulfill' in a social conversation or I end up not knowing if the conversation has ended or not.

These are funny to me because I know I end up like this daily, but they are also not funny because it is frustrating to me and everyone around me. It can seem to them a lot like
I also realize I am very lucky. I can carry my post-it notes in my head most of the time, and only need to be reminded how to do things every now and then. Other people have it where it isn't 'funny' and every day is a struggle to keep the world together.
There are other parts of autism which are harder to describe. The inability to close off sounds and scents are harder to explain. Some days it is an easy task, other days it is exactly like:
On days like that I can't sit in even a library without it sounding like a cacophony of voices. The brain tries to parse every conversation which can make a work meeting much harder because the brain is trying to make each word heard part of a coherent conversation. This means that manager talking in the room and the guy outside on the phone to his girl friend get intermingled at times. You wonder why the manager is asking if you have a negligee or some other weird connotation. I end up having to cup my ears to focus on what one conversation at a time is doing or just write down prime numbers on a sheet of paper until the brain stops muddling up.

I know this isn't how it is for every person with autism.. each one of us has it slightly different. I have been incredibly lucky in how my autism has manifested and just want to help people who don't know what it might be like to know.



So a friend of mine said that I needed to look at the graph data a bit more closely. I decided to look at a 7 week average (49 days) and a 29 week average (203 days). What I found interesting was how noisy the data was still at 49 days. Here we see a comparison between different EPEL-6 curves using 7, 49 and 203 day moving averages:
Looking at the curves, while EPEL-6 is still "growing" it seems to have plateau-ed in early to mid 2017. A curve by itself is useless, so here is it in comparison to EPEL-7 where the curve for EPEL-7 seems to increase when EPEL-6 leveled out.
 From this, I expect that EPEL-7 will cross over EPEL-6 in mid to late 2018 though with a 203 day graph that would be hard to see. Finally here is a stacked graph of the releases from EPEL-5 onward using 203 day averages.
Stacking this way shows that when RHEL-5 was EOL in 2017, there is an inflection in EPEL-7 growth.

Note: Originally I was going to compare the powers of 7: 7, 49, 343 but I found the 343 to be so smooth it wasn't clear when a change was occurring. I backed it down to the 203 to get some fluctuations.. and then realized that this was close to the standard 200 day moving average that financial organizations use. However, I am not sure they are the same because financial data is usually in 5 day weeks while I am looking at 7 day weeks.


EPEL: Security Profiles in EL7 can cause problems with outside repositories

Currently, if you are installing CentOS 7 or Red Hat Enterprise Linux and use a security profile, you will have problems with 3rd party repositories. The errors can seem rather obtuse, and it usually gets diagnosed as "EPEL is down" or some similar problem. The test will look something like:

epel/x86_64/metalink                                                                                                                                  |  17 kB  00:00:00     
https://mirrors.rit.edu/fedora/epel/7/x86_64/repodata/repomd.xml.asc: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below wiki article 


If above article doesn't help to resolve this issue please use https://bugs.centos.org/.

http://mirror.nodesdirect.com/epel/7/x86_64/repodata/repomd.xml.asc: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
http://mirror.us.leaseweb.net/epel/7/x86_64/repodata/repomd.xml.asc: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
The key part to look at is the request for repomd.xml.asc. That is where yum is asking for the gpg signed repository xml metadata. The Fedora Project does not currently sign its data for various reasons. This means that the yum will not see the epel archive as active and refuse to show packages.

There are two fixes that are currently available:

  1. reposync the repository and sign that repository with keys that you have accepted. This is what most sites which require a security profile are going to need to do. It means that there is a process and control and signoff which would meet that sites security plan.
  2. Turn off the checking of repository signatures for the EPEL repository.
    name=Extra Packages for Enterprise Linux 7 - $basearch
    WARNING: Doing this on many systems without getting an allowed exception will cause audit problems. This is the primary reason that EPEL does not come with this automatically. It MUST be a conscious decision of the installation systems administration to turn it off.
And yes, a third option would be to have the metadata signed. I am not an authority on why the data is not currently done so and do not like arm-chair quarterbacking people who have to deal with the build system. 


Fedora Infra PSA: I have been marked spamchecked! What do I do?


About 1-2 times a week we get a new user who are told during account creation something like the following:

Your account status have just been set to spamcheck_denied by an admin

or an account's moderator.

If this is not expected, please contact admin@fedoraproject.org and
let them know.

- The Fedora Account System

I realized I haven't written about this since 2016, so it was time to update some of the data on it. We still see a number of accounts created daily which seem only meant to create spam on the wiki. While sometimes the account user will use an obvious name like techsupport or freeantivirus, the majority will have account names will look 'normal'. The spamcheck tool tries to look at other items like where the potential user's ip, email address, and other sets 'stack' up compared to other accounts with similar items in the past. Sadly this means that we will have some amount of 'false positives' even though we try to push towards more 'false negatives'.

What do you do?

If you find yourself getting a 'false positive', please open an email to admin@fedoraproject.org with the following information:

  1. The account name you tried to open. Account admins need this to look in the Fedora Account System to check on the account. 
  2. The email address you used to open the account. Multiple times I have gotten an email from foo@gmail.com but the account which was opened was with foobar@yahoomail.com or some other domain. We normally do not activate this account in this case until we get an email from the one that was listed. 
  3. If possible the ip address you had when you registered the account. This can help us figure out if some other problem is causing issues. We have to blacklist some ips just from the sheer amount of 'spam', and sometimes forget to remove those blocks.
Normally if you do this, you should get a response back in 24-48 hours from someone who is on the admin mailing list. We may need to get one or two more bits of data and then will turn on the account in many cases. [There was one person who was honest enough to say that they just wanted the account to put up HP printer support pages. We said no thank you.]

Do I need a Fedora Account?

If you are wanting to do long term work with the Fedora Project, I would get a user account. If you are only needing to answer a question or help someone else out on a mailing list, you do not need an account to do so.

In many cases, you do not need a Fedora account in order to work with or on Fedoraproject items. If you are trying to answer questions on ask.fedoraproject.org you can login using multiple other authenticators (Google, Facebook, Yahoo, and OpenID). If you are wanting to fix something on the wiki, you will still need to get sponsored in another group. This is because even with the anti-spam measures knowing out 99% of bad accounts, 1% of thousands of accounts still is a lot. If you really want to fix something but don't want to wait for getting into a group.. send the changes you want to an appropriate  mailing list