Hi All,
I know this has been discussed around the traps here and there but I thought I'd take it to the list for a more directed discussion.
It's been a long time since there's been a new architecture added to RHEL. With RHEL 7.1 there's been support added for POWER Little Endian (ppc64le) and with the aaarch64 PEAP (I have no idea about RHs plans there) plus the i686 CentOS effort it's possible that there might be demand in the future for more architectural additions so I figured it's probably a good time to discuss requirements for the addition of new architectures to EPEL.
In the case of ppc64le the RHEL release is closer to x86_64 in features and functionality than ppc64. It also has the advantage it's also little endian so a number of the issues that are seen with the big endian builds are a non event. IBM have been doing some builds and filing bugs [1] for build issues.
Just to note the initial 7.1 ppc64le release of RHEL has a different distag to el7 but this should be resolved by 7.2 beta.
Peter
On 7 March 2015 at 04:11, Peter Robinson pbrobinson@gmail.com wrote:
Hi All,
I know this has been discussed around the traps here and there but I thought I'd take it to the list for a more directed discussion.
Thank you. I would appreciate a directed discussion on this. We brought this up at the last EPEL meeting (which I haven't posted the minutes too :/.) and were going to bring it up for a vote.
It's been a long time since there's been a new architecture added to RHEL. With RHEL 7.1 there's been support added for POWER Little Endian (ppc64le) and with the aaarch64 PEAP (I have no idea about RHs plans there) plus the i686 CentOS effort it's possible that there might be demand in the future for more architectural additions so I figured it's probably a good time to discuss requirements for the addition of new architectures to EPEL.
In the case of ppc64le the RHEL release is closer to x86_64 in features and functionality than ppc64. It also has the advantage it's also little endian so a number of the issues that are seen with the big endian builds are a non event. IBM have been doing some builds and filing bugs [1] for build issues.
Just to note the initial 7.1 ppc64le release of RHEL has a different distag to el7 but this should be resolved by 7.2 beta.
Here are the issues that I would like to see addressed for any architecture. Various people have said that I am 'PM' of EPEL for the time being, so I would like to think about this as I would hope a good product manager would. [This is also where I get to be removed forcibly]
1) Who is championing an architecture? 2) Where do developers get access to hardware that they can debug issues if they want to. 3) How do we remove an architecture for whatever reasons? [Possible ones could be it turns out that CentOS i686 is dropped after one subrelease... or PPC64be is dropped by IBM because everyone moved to PPC64le. Or Itanium3 comes out and no one wants x86_64.] 4) Is there a way to break out an EPEL-secondary architecture where these sorts of things are done? [I believe the answer is NO, but it is a question I expect would be asked.]
My main concern on architectures are the following:
1) Every architecture we have is one that potentially blocks other builds. If the PPC builder which is only used for EPEL goes down.. regular Fedora builds have been affected in the past. I believe those problems have been fixed in koji but it is a non-zero risk.
2) Even if it has been fixed to never effect Fedora builds.. it is more hardware which needs to be maintained in the primary build network and downage of any architecture affects all EPEL builds.
3) The people who are championing this are the guys who have to do a lot of the hard work of getting it going.. but are the most overworked guys in Fedora with just Fedora primary and secondary architecture work. They are not the people who have time to be answering developer questions about why some java app doesn't compile on ppcle which requires an IBM developer to go 'ooooh yeah you need to patch that twiddly bit.' I realize that there are various ARM and PPC developers who do that for Fedora in their mailing lists. I would like to see someone from there saying 'yeah we will help when we can' for any architecture.
4) Because EPEL doesn't do releases and there is no want from release engineering to do releases in EPEL.. adding an architecture or stuff is easy. removing it is a completely different story. From what Dennis and others have said, this is non-trivial to 'not enough beer in the world' level. If anything turns into an Itanium where we had support up to one release and then no more... or some similar problem.. it is more work for the overworked guys.
I know this has been discussed around the traps here and there but I thought I'd take it to the list for a more directed discussion.
Thank you. I would appreciate a directed discussion on this. We brought this up at the last EPEL meeting (which I haven't posted the minutes too :/.) and were going to bring it up for a vote.
It's been a long time since there's been a new architecture added to RHEL. With RHEL 7.1 there's been support added for POWER Little Endian (ppc64le) and with the aaarch64 PEAP (I have no idea about RHs plans there) plus the i686 CentOS effort it's possible that there might be demand in the future for more architectural additions so I figured it's probably a good time to discuss requirements for the addition of new architectures to EPEL.
In the case of ppc64le the RHEL release is closer to x86_64 in features and functionality than ppc64. It also has the advantage it's also little endian so a number of the issues that are seen with the big endian builds are a non event. IBM have been doing some builds and filing bugs [1] for build issues.
Just to note the initial 7.1 ppc64le release of RHEL has a different distag to el7 but this should be resolved by 7.2 beta.
Here are the issues that I would like to see addressed for any architecture. Various people have said that I am 'PM' of EPEL for the time being, so I would like to think about this as I would hope a good product manager would. [This is also where I get to be removed forcibly]
- Who is championing an architecture?
Primarily IBM, but this will widen with the OpenPOWER foundation and it's members widening and HW from that initiative starting to become available. In the case of aarch64, if that happens, there will be similarities through Linaro Enterprise Group (LEG).
- Where do developers get access to hardware that they can debug issues if
they want to.
I'll let Mike (from IBM) answer this one in detail but there's a number of Universities hosting publicly accessible instances of HW with a process in place, Linaro has similar process with access to aarch64 HW running Fedora releases.
- How do we remove an architecture for whatever reasons? [Possible ones
could be it turns out that CentOS i686 is dropped after one subrelease... or PPC64be is dropped by IBM because everyone moved to PPC64le. Or Itanium3 comes out and no one wants x86_64.]
I don't see that would be any different to how we dropped PPC from mainline Fedora back in the F-12/13 timeframe but the architectures, once added to core RHEL, will be supported for the lifecycle of RHEL so I don't see that this process would be any different to how we dropped i686 or any of the 32 bit architectures in the transition from el6 -> el7. I personally don't think it's actually worth expending too much time on this process until the issue arises, cross the bridge when we get there so to speak.
- Is there a way to break out an EPEL-secondary architecture where these
sorts of things are done? [I believe the answer is NO, but it is a question I expect would be asked.]
Not that I'm aware of and it would be a lot of extra work for rel-eng to do so for minimal, if any, benefit.
My main concern on architectures are the following:
- Every architecture we have is one that potentially blocks other builds.
If the PPC builder which is only used for EPEL goes down.. regular Fedora builds have been affected in the past. I believe those problems have been fixed in koji but it is a non-zero risk.
- Even if it has been fixed to never effect Fedora builds.. it is more
hardware which needs to be maintained in the primary build network and downage of any architecture affects all EPEL builds.
I don't see those issues any different to any of the other architectures or hardware that's needed to run Fedora infrastructure whether it be servers, storage or network. We have Enterprise support on the HW with all due process.
- The people who are championing this are the guys who have to do a lot of
the hard work of getting it going.. but are the most overworked guys in Fedora with just Fedora primary and secondary architecture work. They are not the people who have time to be answering developer questions about why some java app doesn't compile on ppcle which requires an IBM developer to go 'ooooh yeah you need to patch that twiddly bit.' I realize that there are various ARM and PPC developers who do that for Fedora in their mailing lists. I would like to see someone from there saying 'yeah we will help when we can' for any architecture.
The process for POWER or aarch64 or any other architecture would be the same as it is now for ppc64, escalate to the SIGs, IBM in the case of POWER is very good at getting issues resolved quickly. They're also participating more and more in other areas of Fedora and I suspect we'll start to see companies from the Open POWER foundation participating too, there's companies that have announced their involvement that already participate actively in the Fedora/RHEL/CentOS ecosystem.
From an infrastructure PoV the advantage that Power8 hardware has is
that it's much closer to x86 in a number of ways and it'll enable us to mimic the deployment of things like virt builders in a single contiguous manner across all architectures to enable more simplified standardised manner to ease burden and increase automation from an infra PoV
- Because EPEL doesn't do releases and there is no want from release
engineering to do releases in EPEL.. adding an architecture or stuff is easy. removing it is a completely different story. From what Dennis and others have said, this is non-trivial to 'not enough beer in the world' level. If anything turns into an Itanium where we had support up to one release and then no more... or some similar problem.. it is more work for the overworked guys.
Again I'd like to see more details of this issue, but similar to the removal of unmaintained packages with security issues that has been just implemented it's probable to be an issue but once it's in a core RHEL major release it will be supported for the cycle of that major release, so if an architecture isn't supported in RHEL.next we don't continue to support it in EPEL. The best current example of this would be 32 bit arches in el7. In terms of adding architectures that are supported only in CentOS and not RHEL such as i686 I would be more reticent and careful in that case, and I've not followed i686 in this case and don't know whether it's supported as primary (or what ever that community call it) or whether it's community contributed. I think RHEL supported vs CentOS supported would likely need a slightly different discussion as it's a Enterprise vs Community supported and they may have different support plans for different architectures, I'd hope someone from the CentOS community could provide the clarifications there.
Peter
On Mon, 9 Mar 2015 11:49:56 +0000 Peter Robinson pbrobinson@gmail.com wrote:
...snip...
- Who is championing an architecture?
Primarily IBM, but this will widen with the OpenPOWER foundation and it's members widening and HW from that initiative starting to become available. In the case of aarch64, if that happens, there will be similarities through Linaro Enterprise Group (LEG).
Would we then have a tracker bug and a way for maintainers to call on these resources when/if their packages don't build?
- Where do developers get access to hardware that they can debug
issues if they want to.
I'll let Mike (from IBM) answer this one in detail but there's a number of Universities hosting publicly accessible instances of HW with a process in place, Linaro has similar process with access to aarch64 HW running Fedora releases.
This would be good to know.
- How do we remove an architecture for whatever reasons? [Possible
ones could be it turns out that CentOS i686 is dropped after one subrelease... or PPC64be is dropped by IBM because everyone moved to PPC64le. Or Itanium3 comes out and no one wants x86_64.]
I don't see that would be any different to how we dropped PPC from mainline Fedora back in the F-12/13 timeframe but the architectures, once added to core RHEL, will be supported for the lifecycle of RHEL so I don't see that this process would be any different to how we dropped i686 or any of the 32 bit architectures in the transition from el6 -> el7. I personally don't think it's actually worth expending too much time on this process until the issue arises, cross the bridge when we get there so to speak.
I'm assuming we would keep ppc64 around too for now on the rhel's we support?
...snip...
I don't see those issues any different to any of the other architectures or hardware that's needed to run Fedora infrastructure whether it be servers, storage or network. We have Enterprise support on the HW with all due process.
Well, we don't have any ppc-le builders currently for EPEL. I guess this would need to be figured out off list first?
We do have secondary arch Fedora ones, but the EPEL builders are in the primary koji, so they would need to be their own thing and have support, etc. I dont think we want to share builders with Fedora secondary ppc...
We can figure this out off list tho.
From an infrastructure PoV the advantage that Power8 hardware has is that it's much closer to x86 in a number of ways and it'll enable us to mimic the deployment of things like virt builders in a single contiguous manner across all architectures to enable more simplified standardised manner to ease burden and increase automation from an infra PoV
Thats good.
kevin
Peter Robinson pbrobinson@gmail.com wrote:
...snip...
- Who is championing an architecture?
Primarily IBM, but this will widen with the OpenPOWER foundation and it's members widening and HW from that initiative starting to become available. In the case of aarch64, if that happens, there will be similarities through Linaro Enterprise Group (LEG).
Would we then have a tracker bug and a way for maintainers to call on these resources when/if their packages don't build?
- Where do developers get access to hardware that they can debug
issues if they want to.
I'll let Mike (from IBM) answer this one in detail but there's a number of Universities hosting publicly accessible instances of HW with a process in place, Linaro has similar process with access to aarch64 HW running Fedora releases.
This would be good to know.
- How do we remove an architecture for whatever reasons? [Possible
ones could be it turns out that CentOS i686 is dropped after one subrelease... or PPC64be is dropped by IBM because everyone moved to PPC64le. Or Itanium3 comes out and no one wants x86_64.]
I don't see that would be any different to how we dropped PPC from mainline Fedora back in the F-12/13 timeframe but the architectures, once added to core RHEL, will be supported for the lifecycle of RHEL so I don't see that this process would be any different to how we dropped i686 or any of the 32 bit architectures in the transition from el6 -> el7. I personally don't think it's actually worth expending too much time on this process until the issue arises, cross the bridge when we get there so to speak.
I'm assuming we would keep ppc64 around too for now on the rhel's we support?
...snip...
I don't see those issues any different to any of the other architectures or hardware that's needed to run Fedora infrastructure whether it be servers, storage or network. We have Enterprise support on the HW with all due process.
Well, we don't have any ppc-le builders currently for EPEL. I guess this would need to be figured out off list first?
We do have secondary arch Fedora ones, but the EPEL builders are in the primary koji, so they would need to be their own thing and have support, etc. I dont think we want to share builders with Fedora secondary ppc...
We can figure this out off list tho.
Some of the new P8 hardware that was recently racked is intended to be for EPEL on ppc64/ppc64le, I just need to get it configured and build VMs done etc
From an infrastructure PoV the advantage that Power8 hardware has is that it's much closer to x86 in a number of ways and it'll enable us to mimic the deployment of things like virt builders in a single contiguous manner across all architectures to enable more simplified standardised manner to ease burden and increase automation from an infra PoV
Thats good.
On 10 March 2015 at 11:55, Peter Robinson pbrobinson@gmail.com wrote:
Peter Robinson pbrobinson@gmail.com wrote:
We can figure this out off list tho.
Some of the new P8 hardware that was recently racked is intended to be for EPEL on ppc64/ppc64le, I just need to get it configured and build VMs done etc
Ah didn't know that. Would have racked that in the build network rack versus the secondary arch rack then. So the system is on a switch which is dedicated to the SecondaryArch/QA vlan. We will need to get it moved over to the build network or make sure that the all the firewall rules are in place so it can talk through the router firewall to the Build servers. [I don't know which one will be less work to you guys.]
From an infrastructure PoV the advantage that Power8 hardware has is that it's much closer to x86 in a number of ways and it'll enable us to mimic the deployment of things like virt builders in a single contiguous manner across all architectures to enable more simplified standardised manner to ease burden and increase automation from an infra PoV
Thats good.
epel-devel mailing list epel-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/epel-devel
epel-devel@lists.fedoraproject.org