Adding resource management to beaker
by Bill Peck
Hello Beaker,
One of the jobs we run uses shared storage. For example:
HostA with Driver1
HostB with Driver2
HostC with Driver3
All connected to StorageX.
We can't run the test on more than one host at a time since it would
conflict.
Being able to specify the following in the job xml is the goal:
<recipeSet>
<recipe/> <!-- Require HostA -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostB -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostC -->
<resource/> <!-- Require StorageX -->
</recipeSet>
I know there is a resource type for recipe but this doesn't seem like a
good solution. I did try an experiment where I created StorageX host in
beaker with no power control and scheduled three recipesets like above.
But this hack won't work because a watchdog with a NULL expire time is
set and that means it will sit there forever. Even if it did work,
maybe if we created a dummy power type, we would still end up with an abort.
I don't think it would take much work to support this type of
multi-host/resource requirement.
Right now we have the following model
Object Recipe, which has the following attributes: distro_tree,
resource, rendered_kickstart, watchdog, systems, dyn_systems, tasks,
dyn_tasks, tags, repos, rpms, logs, custom_packages, ks_appends
Object MachineRecipe which inherits from Recipe, and adds a guests
attribute.
Object GuestRecipe which inherits from Recipe.
A ResourceRecipe object wouldn't need most of these attributes, probably
just the following:
resource, systems, and dyn_systems. (no watchdog, just return when
recipeSet completes or aborts)
And resource currently maps to a RecipeResource which can be one of the
following:
SystemResource
VirtResource
GuestResource
I think adding another Object here would make sense, but the term
Resource is overused and its a little confusing right now.
If done correctly we could re-use the filtering code in needproperty,
but I think the only things we could search on for resources would be
the following: Name, lab_controller, and key_values? (storing how these
shared resources are connected would be another option I suppose, but
maybe too complicated for a first implementation?)
I'm sure there are other things to consider, we could support setting up
and tearing down shared resources (almost like a power on/off command).
I want to be mindful of things like that but I don't want to be
paralysed with what if's and never get anything done.
I know you guys have been working hard on the groups features and want
to re-do the scheduler as well. Maybe this would be something for after
the scheduler re-write? I don't see any mention of this on the roadmap
[1] (although the roadmap is quite full already!).
In any event, I'm interested in hearing your ideas and feedback.
Thanks!
1 - http://beaker-project.org/dev/tech-roadmap.html
9 years, 7 months
Fwd: beah components and their interaction
by Amit Saha
I have found myself referring these fairly regularly while working on Beah.
Would it be a good idea to put this in the beah developer guide?
I can't promise that it will be any better than it is as below.
> During a test run, several Beah components interact over TCP/IP within
> the system itself.
>
> Beah local servers
> ==================
>
> When you login to a test system (say, when "reservesys" is running), you will
> see that
> the following Beah specific servers listening:
>
> beah-srv 9353 root 10u IPv4 28367 0t0 TCP *:12432 (LISTEN)
> beah-srv 9353 root 11u IPv6 28368 0t0 TCP localhost:12432
> (LISTEN)
> beah-srv 9353 root 13u IPv4 28370 0t0 TCP *:12434 (LISTEN)
> beah-srv 9353 root 14u IPv6 30919 0t0 TCP localhost:12434
> (LISTEN)
> beah-rhts 10898 root 7u IPv4 31905 0t0 TCP localhost:7085
> (LISTEN)
> beah-rhts 10898 root 11u IPv6 31906 0t0 TCP localhost:7085
> (LISTEN)
>
> Note that this is the IPv6 capable harness that we plan to release soon and
> hence each
> of the servers are listening on both IPv4 and IPv6 interfaces.
>
> The 'beah-srv' process corresponds to the function "start_server" in
> beah/wires/internals/twserver.py
> and it basically starts the TaskListener and BackendListener, whose presence
> you can usually see in
> the console logs:
>
> 2014-01-29 00:55:48,719 beah start_server: INFO Controller: BackendListener
> listening on 127.0.0.1:12432
> 2014-01-29 00:55:48,720 beah start_server: INFO Controller: BackendListener
> listening on ::1:12432
> 2014-01-29 00:55:48,720 beah start_server: INFO Controller: BackendListener
> listening on /var/beah/backend12432.socket
> 2014-01-29 00:55:48,722 beah start_server: INFO Controller: TaskListener
> listening on 127.0.0.1:12434
> 2014-01-29 00:55:48,723 beah start_server: INFO Controller: TaskListener
> listening on ::1:12434
> 2014-01-29 00:55:48,723 beah start_server: INFO Controller: TaskListener
> listening on /var/beah/task12434.socket
>
> These servers exist throughout a recipe run on the test system. The
> corresponding "client" programs live
> in beah/wires/internals/twbackend.py and beah/wires/internals/twtask.py.
>
>
> The beah-rhts-task (beah/tasks/rhts_xmlrpc.py:main()) starts a server *per
> task*, it is the
> result server and exits on a task completion.
>
>
> Beah daemons
> ============
>
> The following beah daemons are started at system boot:
>
> beah-fwd-backend
> ================
>
> This handles the communication during multi host jobs.
>
> Source: beah/beaker/backends/forwarder.py
>
>
> beah-beaker-backend
> ===================
>
> Talks to the lab controller's beaker-proxy process over XML-RPC.
>
> Source: beah/beaker/backends/beakerlc.py
>
>
> beah-srv
> ========
>
> Main daemon process we saw above.
>
> Source: beah/bin/srv.py
>
> As you can see from the following status, the beah-rhts-task is spawned by
> beah-srv:
>
> # systemctl status beah-srv
> beah-srv.service - The Beaker Harness server.
> Loaded: loaded (/usr/lib/systemd/system/beah-srv.service; enabled)
> Active: active (running) since Wed 2014-01-29 00:55:47 EST; 20min ago
> Main PID: 9353 (beah-srv)
> CGroup: /system.slice/beah-srv.service
> ├─ 9353 /usr/bin/python /usr/bin/beah-srv
> └─10898 /usr/bin/python /usr/bin/beah-rhts-task
>
>
10 years, 2 months
Readahead collection and Beah
by Amit Saha
Recently it was reported that beah starts task execution while systemd-readahead-collect
is running. This was reported on RHEL 7, but will possibly be seen on Fedora as well.
The simple solution that was first thought of was to start beah after systemd-readahead-collect.timer.
However, this won't work because systemd-readahead-collect.timer has "After=default.target" in it's
unit file. So basically, we end up with a cyclic dependency. Two alternative approaches that can be taken:
1. Define an alternative boot target
2. Disable readahead collection as we do for RHEL 6 now.
See https://bugzilla.redhat.com/show_bug.cgi?id=1072284#c4 and the following comment
for more discussions on the same.
Currently we are thinking of implementing (2) above.
Any other thoughts?
-Amit.
--
Amit Saha
SED, Hosted & Shared Services
Red Hat, Inc.
10 years, 2 months
Beaker 0.16.0 released
by Dan Callaghan
On behalf of the Beaker development team, I'm pleased to announce that
Beaker 0.16.0 is now available from the Beaker web site [1].
The headline feature in this release is experimental server-side support
for "external tasks". A recipe can now refer to tasks by URL, without
needing the task to exist in Beaker's task library. Alternative
harnesses can use this functionality to run tasks directly from source
control or other locations outside Beaker's task library.
This release also adds a new permission to system access policies:
"view". System owners can use the access policy to control exactly who
can see their system. This is a more flexible replacement for the
"Secret" field.
A number of other enhancements and bug fixes are included in this
release as well. For a complete description, refer to the Beaker 0.16
release notes [2].
The detailed list of all changes made since Beaker 0.15.5 is also
available [3].
[1] http://beaker-project.org/releases/
[2] http://beaker-project.org/docs/whats-new/release-0.16.html
[3] http://beaker-project.org/releases/beaker-0.16.0-ChangeLog
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Hosted & Shared Services
Red Hat, Inc.
10 years, 2 months
Discussion on RFE: Hide Removed Systems by default
by Amit Saha
There has been some discussions around the change required by [1]. I agree with most
of the comments by Nick here [2]. However, as I mentioned in the previous comment,
hiding removed systems by default from all places, hides the removed systems if I do
not actively know about their existence and search for them. I understand that users
are not in anyway benefited by the knowledge of removed systems and an admin can look up the
global "Systems->Removed" page if a system is to be added back.
Are there any alternative ideas or the suggestion in [2] sounds good?
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1000092
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1000092#c8
Best,
Amit.
--
Amit Saha
SED, Hosted & Shared Services
Red Hat, Inc.
10 years, 2 months
Demo of system page improvements
by Dan Callaghan
I've set up a temporary Beaker instance to demonstrate my proposed
system page improvements [1] which will hopefully be landing in the 0.17
release.
The demo instance is here:
http://209.132.179.159/bkr/
You can log in as "demo" with password "demo". There are no real systems
or lab controllers attached, and no distros imported, so it's not
a fully functional Beaker instance. But I created some dummy systems
with a variety of permissions granted, so it's enough to try out most of
the system page features.
If you're interested in the evolution of the system page, please take
a look and reply with your feedback.
[1] http://beaker-project.org/dev/proposals/system-page-improvements.html
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Hosted & Shared Services
Red Hat, Inc.
10 years, 3 months
Handling unrecognised distro names in the default templates
by Nick Coghlan
Currently, the default kickstart snippets are written such that any
unknown distro names are handled as if they were Red Hat Enterprise
Linux 6. We've dealt with Fedora by mentioning it explicitly in the
templates when necessary.
That's worked OK up until now, but there's likely to be a CentOS 7 beta
some time in the next few months, and the handling of that and other Red
Hat Enterprise Linux 7 derivatives means that this isn't going to work
any more, since we can't hardcode special case handling of every
possible derivative.
The simplest resolution I have been able to come up with is the following:
1. Add a "base_distro" context variable in the kickstart rendering. By
default, this is the same as "distro".
2. Update the kickstart snippets to use base_distro rather than distro
3. Add an optional "base_distro" field to our distro records. If that is
set, base_distro in the template rendering context is to that rather
than yo the same thing as distro.
It's too late to get that into 0.16, but we could do it for 0.17.
Does anyone see a simpler way to resolve the problem? If not, I'll
create an RFE for this and allocate it to 0.17.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 3 months