New meeting time + call for agenda items
by Joe Brockmeier
Hi all,
OK, the votes are in and it looks like the least-bad time for meetings
right now are 14:00 UTC on Thursdays. I think we were missing a few
votes, though, so I hope this time will work for the most people.
Please send me any agenda items by Monday afternoon and I will send out
a meeting reminder then.
Thanks,
jzb
--
Joe Brockmeier | Principal Cloud & Storage Analyst
jzb(a)redhat.com | http://community.redhat.com/
Twitter: @jzb | http://dissociatedpress.net/
10 years, 2 months
Cloud accounts for testing purposes?
by Sandro "red" Mathys
To implement the new products, at least every "tailored image"-change
owner will require a cloud account somewhere. How do we get those?
We do have a tenant in the HP Cloud that is free of charges, right?
How are users added to it?
Thanks,
Sandro
10 years, 2 months
Re: Modular Kernel Packaging for Cloud
by Josh Boyer
On Fri, Mar 7, 2014 at 10:06 AM, Don Zickus <dzickus(a)redhat.com> wrote:
> On Fri, Mar 07, 2014 at 09:50:37AM -0500, Josh Boyer wrote:
>> On Fri, Mar 7, 2014 at 8:59 AM, Don Zickus <dzickus(a)redhat.com> wrote:
>> > On Fri, Mar 07, 2014 at 08:23:15AM -0500, Josh Boyer wrote:
>> >> On Fri, Mar 7, 2014 at 2:16 AM, Sandro "red" Mathys
>> >> <red(a)fedoraproject.org> wrote:
>> >> >>> I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?>
>> >> >>> firmwre.
>> >> >>
>> >> >> Firmware is owned by linux-firmware, not the kernel package. I didn't
>> >> >> include it in my kernel numbers for that reason.
>> >> >>
>> >> >>> He just has to cut 35MB or so from /lib/modules/. We can probably nickel
>> >> >>> and dime and review a lot of cruft to get there, but what is that 35MB
>> >> >>> really doing to get us anything? I am sure half of that can be removed by
>> >> >>> re-examining the minimal-list he sent (I can even help there).
>> >> >>
>> >> >> Right. Considering the bloat elsewhere in the distro, I think we can
>> >> >> start with what I have and work from there if needed.
>> >> >
>> >> > Excellent progress there. So 35MB are already gone and I figure
>> >> > (re)moving graphics, sound and other obvious things will gain us quite
>> >> > some more MB. Nice job.
>> >>
>> >> No. 62MB are already "gone". The 35MB was what is needed
>> >> additionally to get to comparable numbers with ubuntu. Sound I could
>> >> see dropping. Graphics, not so much.
>> >>
>> >> Really, I'm likely to start with what I have and if there are major
>> >> reasons that get brought up (with data) to trim further, we can look
>> >> at it.
>> >
>> > I agree too. I think you did an awesome job Josh. Most of the fat is
>> > trimmed. :-) Anything more should really come with some numbers/data.
>>
>> Don't get too heavy on the praise yet. This still needs to be done at
>> the packaging level. The changes to kernel.spec might make you want
>> to poke your eyes out or something ;).
>
> Oh, I thought you had the package split already into kernel-common and
> kernel-drivers?
I have the basics to create those packages and the kernel metapackage
done. Right now they contain strictly /boot for kernel-core and
/lib/modules for kernel-drivers. What I don't have done is the
swizzling around of certain modules to be in kernel-core yet. The
numbers gathering was just hacking a local VM on the filesystem
directly.
So the next step is to translate that to the proper %files entries and
build/test.
> But yes I would be interesting in seeing those changes because that
> filters to RHEL eventually. I can already see RHEL developers whining
> about extra packages to install... ;-) Though all the kernel-tools stuff
> seemed to have gone smoothly...
Right. As soon as I get something that seems like it boots in a VM
when you install it, I'll post the changes here for review. The
current stuff I have done is in a COPR if you are _really_ itching to
see it:
http://copr-fe.cloud.fedoraproject.org/coprs/jwboyer/kernel-core/
DO NOT INSTALL THAT. It's stale, won't boot, hasn't been tested, etc.
I'll be building there for now as I work through things, but until
further notice it only exists to make sure things build.
josh
10 years, 2 months
Re: Modular Kernel Packaging for Cloud
by Josh Boyer
On Fri, Mar 7, 2014 at 8:59 AM, Don Zickus <dzickus(a)redhat.com> wrote:
> On Fri, Mar 07, 2014 at 08:23:15AM -0500, Josh Boyer wrote:
>> On Fri, Mar 7, 2014 at 2:16 AM, Sandro "red" Mathys
>> <red(a)fedoraproject.org> wrote:
>> >>> I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?>
>> >>> firmwre.
>> >>
>> >> Firmware is owned by linux-firmware, not the kernel package. I didn't
>> >> include it in my kernel numbers for that reason.
>> >>
>> >>> He just has to cut 35MB or so from /lib/modules/. We can probably nickel
>> >>> and dime and review a lot of cruft to get there, but what is that 35MB
>> >>> really doing to get us anything? I am sure half of that can be removed by
>> >>> re-examining the minimal-list he sent (I can even help there).
>> >>
>> >> Right. Considering the bloat elsewhere in the distro, I think we can
>> >> start with what I have and work from there if needed.
>> >
>> > Excellent progress there. So 35MB are already gone and I figure
>> > (re)moving graphics, sound and other obvious things will gain us quite
>> > some more MB. Nice job.
>>
>> No. 62MB are already "gone". The 35MB was what is needed
>> additionally to get to comparable numbers with ubuntu. Sound I could
>> see dropping. Graphics, not so much.
>>
>> Really, I'm likely to start with what I have and if there are major
>> reasons that get brought up (with data) to trim further, we can look
>> at it.
>
> I agree too. I think you did an awesome job Josh. Most of the fat is
> trimmed. :-) Anything more should really come with some numbers/data.
Don't get too heavy on the praise yet. This still needs to be done at
the packaging level. The changes to kernel.spec might make you want
to poke your eyes out or something ;).
josh
10 years, 2 months
Re: Modular Kernel Packaging for Cloud
by Josh Boyer
On Thu, Mar 6, 2014 at 12:04 PM, Don Zickus <dzickus(a)redhat.com> wrote:
> On Thu, Mar 06, 2014 at 11:32:55AM -0500, Matthew Miller wrote:
>> On Thu, Mar 06, 2014 at 11:02:47AM -0500, Josh Boyer wrote:
>> > If it's _necessary_, that's one thing. I've yet to really see any data
>> > backing up necessity on any of this at all though. Right now it seems
>> > to be sitting in the "nice to have" category.
>>
>> For the record, it is _literally_ sitting in our "nice to have" category.
>> See
>> https://fedoraproject.org/wiki/Cloud_Changelist#Change:_Cloud-Friendly_Ke...
>>
>> :)
>>
>>
>> > Perhaps someone from the cloud team could look at existing images from
>> > other distros and figure out kernel sizes there, and how it plays into
>> > usage and cost in those environments?
>>
>> On the ubuntu EC2 image, /lib/modules/$(uname -r) is 24M + 5.2M vmlinuz +
>> 1.1M in /lib/firmware. Total package size is 32M on disk. And 5.9M initrd.
>>
>> CoreOS is bigger, with 33M in /lib/modules and 5.2M in lib/firmware, and a
>> /19M vmlinuz.
>
> Yeah, hard numbers to compete with! :-)
The only way to win is to not play at all? :)
Small note too, just because the vmlinuz are of a certain size does
not mean they contain similar content. Without really digging into
the config settings it's hard to do a true apples to apples
comparison. Still, having the overall sizes handy is helpful, thanks.
> I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?>
> firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't
include it in my kernel numbers for that reason.
> He just has to cut 35MB or so from /lib/modules/. We can probably nickel
> and dime and review a lot of cruft to get there, but what is that 35MB
> really doing to get us anything? I am sure half of that can be removed by
> re-examining the minimal-list he sent (I can even help there).
Right. Considering the bloat elsewhere in the distro, I think we can
start with what I have and work from there if needed.
> Maybe impose only xfs as the fs of choice or some other restrictions and
> chop it further, but then we lose flexibility.
Oh dear. Please not another FS thread. So many emails from last week...
> Instead of competing with Ubuntu on minimalist can we compete on pretty
> close but a lot more flexible? Do Ubuntu users have much choice on how
> they configure their environment? Or is Fedora Cloud providing a generic
> cookie cutter installation?
Right, I kind of like that we'd have a smaller core package that is
still broadly useful.
josh
10 years, 2 months
Re: Modular Kernel Packaging for Cloud
by Sandro "red" Mathys
On Fri, Mar 7, 2014 at 12:00 AM, drago01 <drago01(a)gmail.com> wrote:
> On Thu, Mar 6, 2014 at 12:38 AM, Sandro "red" Mathys
> <red(a)fedoraproject.org> wrote:
>> On Thu, Mar 6, 2014 at 1:45 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
>>> On Wed, 05 Mar 2014 17:37:42 +0100
>>> Reindl Harald <h.reindl(a)thelounge.net> wrote:
>>>> in general you need to multiply the wasted space for each instance
>>
>> Exactly, you usually have hundreds or even thousands of instances
>> running. Sure, "every MB counts" isn't to be taken literal here, maybe
>> I should rather have said "every 10 MB count".
>
> I suggested this once before and got no real answer but if you are so
> disk space constrained wouldn't file system compression
> do what you want instead of trying to micoroptimize every package?
Because file system compression requires cpu cycles (i.e. cpu power
and time) and I think they are more valuable to most users than
storage. So, not optimizing the packages would probably be preferable
over compressing the file system. But trimmed down packages need less
space but no additional cpu cycles so it's a win-win from the user's
perspective.
-- Sandro
10 years, 3 months
Re: Modular Kernel Packaging for Cloud
by Josh Boyer
On Thu, Mar 06, 2014 at 10:04:56AM -0500, Don Zickus wrote:
> On Thu, Mar 06, 2014 at 08:38:44AM +0900, Sandro "red" Mathys wrote:
> > On Thu, Mar 6, 2014 at 1:45 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
> > > On Wed, 05 Mar 2014 17:37:42 +0100
> > > Reindl Harald <h.reindl(a)thelounge.net> wrote:
> > >> in general you need to multiply the wasted space for each instance
> >
> > Exactly, you usually have hundreds or even thousands of instances
> > running. Sure, "every MB counts" isn't to be taken literal here, maybe
> > I should rather have said "every 10 MB count".
> >
> > > At least for my uses, the amount of non persistent disk space isn't a
> > > big deal. If I need disk space, I would attach a persistent volume...
> >
> > Figure you get your additional persistent volumes for free somehow, so
> > all those Amazon AWS, HP Cloud, Rackspace, etc. users envy you. And
> > those admins that need to buy physical disks to put into their private
> > clouds, too.
> >
> > Also, more data equals more network traffic and more time - both
> > things that matter in terms of costs, at least in public clouds.
>
> Sure, but what if the trade-off in size comes with a cost in speed? Is
> cloud ok with the kernel taking twice as long to boot? Or maybe running
> slower? Or maybe crashing more often (because we removed safety checks?).
>
> I mean if Josh wanted to he could make everything modular and have a
> really small kernel footprint (like 40MB or so) running in 50MB of memory
> (I have done this with kdump). But it costs you speed in loading modules
> (as opposed to built into the kernel). You may lose other optional
> optimizations that help speed things up.
>
> Other SIGs made not like it, but again it depends on how you frame your
> environment. Maybe cloud really needs its own kernel. We don't know.
To touch on this a bit, if splitting the kernel up becomes unwieldy we
could possibly just build kernel-cloud as another flavour in kernel.spec
(similar to PAE, debug, etc). I'm not much of a fan of this though, as
it can lead to a very divergent vmlinux over time.
If it's _necessary_, that's one thing. I've yet to really see any data
backing up necessity on any of this at all though. Right now it seems
to be sitting in the "nice to have" category.
Perhaps someone from the cloud team could look at existing images from
other distros and figure out kernel sizes there, and how it plays into
usage and cost in those environments?
josh
10 years, 3 months
Re: Modular Kernel Packaging for Cloud
by Sandro "red" Mathys
On Thu, Mar 6, 2014 at 1:45 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
> On Wed, 05 Mar 2014 17:37:42 +0100
> Reindl Harald <h.reindl(a)thelounge.net> wrote:
>> in general you need to multiply the wasted space for each instance
Exactly, you usually have hundreds or even thousands of instances
running. Sure, "every MB counts" isn't to be taken literal here, maybe
I should rather have said "every 10 MB count".
> At least for my uses, the amount of non persistent disk space isn't a
> big deal. If I need disk space, I would attach a persistent volume...
Figure you get your additional persistent volumes for free somehow, so
all those Amazon AWS, HP Cloud, Rackspace, etc. users envy you. And
those admins that need to buy physical disks to put into their private
clouds, too.
Also, more data equals more network traffic and more time - both
things that matter in terms of costs, at least in public clouds.
-- Sandro
10 years, 3 months
Re: Modular Kernel Packaging for Cloud
by Sandro "red" Mathys
On Thu, Mar 6, 2014 at 12:13 AM, Don Zickus <dzickus(a)redhat.com> wrote:
> On Wed, Mar 05, 2014 at 10:02:17AM -0500, Josh Boyer wrote:
>> On Wed, Mar 5, 2014 at 9:54 AM, Don Zickus <dzickus(a)redhat.com> wrote:
>> > On Wed, Mar 05, 2014 at 08:25:12PM +0900, Sandro "red" Mathys wrote:
>> > For example, lets start with 100MB package requirement for the kernel (and
>> > say 2 GB for userspace). This way the kernel team can implement
>> > reasonable changes and monitor proper usage (because things grow over
>> > time).
>> >
>> > If later on you realize 100 MB is not competitive enough, come back and
>> > chop it down to say 50 MB and let the kernel team figure it out.
>> >
>> > But please do not come in here with a 'every MB counts' approach. It is
>> > not very sustainable for future growth nor really easy to implement from
>> > an engineering approach.
>> >
>> > Is that acceptable? The kernel team can start with a hard limit of 100MB
>> > package requirement (or something reasonably less)? Let's work off budget
>> > requirements please.
>>
>> This is a fair point. To be honest, I've ignored the "every MB
>> counts" aspect entirely for now. I've instead been focusing on
>> required functionality, because that's likely going to be the main
>> driver of what the resulting size will be.
That's the point, we want a reasonably small package while still
providing the required functionality. Not sure how providing a fixed
size number is helping in this. But most of all, I didn't throw in a
number because I have no idea what is reasonably possible. I really
only just said "every MB counts" because the question came up before
(in Josh's old thread) and I hoped I could stop this discussion from
happening again before we have any numbers for this.
> Of course. :-)
>
>>
>> FWIW, the existing kernel package installed today (a debug kernel
>> even) is ~142 MB. 123MB of that is the /lib/modules content. ~6MB of
>> that is vmlinuz. The remaining 13MB is the initramfs, which is
>> actually something that composes on the system during install and not
>> something we can shrink from a packaging standpoint.
>
> It also helps with monitoring. 3-4 years from now after all the chopping,
> these pacakges bloat right back up and everyone forgets why we chopped in
> the first place. Hard requirements help keep everything in check and
> forces people to request more space which the cloud team can evaluate
> properly and still control their enviroment.
Well, if we can remember why we put up a fixed size requirement, why
can't we remember why we did the chopping? ;) Anyway, I think it's
fair to define a "kernel-core should be smaller than X MB" requirement
but I don't think it's fair to say Y MB because I like the number Y. I
also don't like that we might throw out e.g. NFS just because we're
1MB over the limit.
But if it helps the kernel team to have a fixed number, someone tell
me what we roughly save by throwing out the stuff we discussed and we
can discuss what number would long-termish make sense, I guess. Also,
I'm not sure whether we should measure the extracted files, the
compressed RPM or both.
-- Sandro
10 years, 3 months
