infrastructure

infrastructure@lists.fedoraproject.org

3 participants
7908 discussions

[Fedora-infrastructure-list] DB Server Upgrade
by Curt Moore 27 Jun '06

27 Jun '06

Jeffrey/All: I've been corresponding off-list with Elliot with regard to upgrading db1. At his suggestion, I'm going to run my proposed process by everyone for feedback. I went through a production DB server upgrade a while back and this process reflects what I did then and lessons learned during the process. I also had Slony in this mix, which resulted in a few more steps. From what I've read on the wiki pages regarding the current Fedora infrastructure setup, there is only 1 DB server, a SPOF. It may be worth the time to start a separate conversation about some sort of replication or clustered setup, which a beast unto itself. I have a master/slave replication setup with Slony which works quite well. You still have the SPOF with the master but at least you can load balance between the slaves and still have read access going if the master goes down. With all of the research I've done, I've not found anything which is production ready and allows a multi master scenario for Postgres, only master/slave scenario with replication, like Slony. The closest thing I found was PGCluster but it was very flaky, at least on 8.x, the last time I played with it late last year. PGCluster was also very hardware intensive, meaning it takes like 4 machines to run a setup without a SPOF. The following assumes the desire for as minimal as possible downtime and as a result is a little more involved than if more downtime were tolerable. I welcome comments and refinements to the process. Now, onto the upgrade process. *** Make a DB and filesystem backup *** Ensure that all necessary config files are archived so they can be quickly reinstalled via kickstart later 1) Setup all apps to connect to DB using a host alias in /etc/hosts 2) Setup another server as a temporary DB server Note: An option here is to copy data to the temp DB server and do an immediate cut over to the temp server after the data has been copied but you run the risk of data being out of sync between the master and temp server. Danger, Will Robinson! 3) Temporarily disable apps hitting DB 4) pg_dump DBs and template0, users, groups, etc over to temp server 5) Test to ensure DB is up and accepting connections 6) Change the alias entry in /etc/hosts on hosts running apps to the record for the temp DB server 7) Re-enable apps 8) Re-install and configure OS on old DB server via kickstart 9) Temporarily disable apps hitting DB 10) pg_dump DBs and template0, users, groups, etc over to master server 11) Test to ensure master DB is up and accepting connections 12) Change the alias entry in /etc/hosts on servers running apps back to the record for the master DB server There, my 12 step program for a DB upgrade. :-) I had a few more Slony related steps which required shuffling the app servers between Slony slaves and the master but the above it basically the process. I made a few decisions/assumptions during my process. In my case since 99% of DB hits were reads,I still had my Slony slaves accepting read requests. I was able to do this since I maintain both read and write DB handles in the SoftSwitch. Some apps allow a "degraded" mode where data is "read only" just for this sort of thing, I'm not sure if this is the case here or not. During the time I disabled the apps to copy over the DB to the temp server, users just couldn't login to the web portal to change their settings, which are written to the master, for a few minutes. I had to ask, would this really be an issue for 5 min at 3am CST on a Sunday morning? (assuming that the DBs can be copied in the span of 5 min, I have a VLAN'd GigE management network for this sort of thing) The same downtime was experienced when I switched back over to the master DB. Since I'm assuming we're upgrading from Postgres 7.x we'll definitely want to do a pg_dump/pg_restore since there are some inherent differences in the data structures on the disk. In the past I've just stopped postgres, copied over /var/lib/pgsql to the new server and started postgres there, and called it good but you can't do that when upgrading from 7.x to 8.x. Since the data itself has to be migrated from the 7.x format to the 8.x format at some point, there is probably going to have to be some measure of downtime. Otherwise you get into having to compare data in 2 different databases to see if anything changed and manually replicating those changes to the master copy. For me, the 5 min of partial downtime was _much_ cheaper. I look forward to discussion on this. Cheers, -Curt

2 1

[Fedora-infrastructure-list] Auth Failure with extras-sync
by Warren Togami 27 Jun '06

27 Jun '06

Test sync build(a)fedora.linux.duke.edu's password: It seems that pass-through ssh key authentication isn't working anymore. Was my account disabled on fedora.linux.duke.edu due the Extras election too, or is this some other issue? Warren Togami wtogami(a)redhat.com

2 1

[Fedora-infrastructure-list] Re: Introduction
by Jason Hartley 26 Jun '06

26 Jun '06

On Fri, June 23, 2006 7:30 pm, Curt Moore wrote: > Hi Jason. > > At the suggestion of Elliot Lee, I'm contacting you about the Mirror > Management project. > > I'd like to help so please let me know what I can do. > > I saw that there was talk about borrowing the CentOS mirroring product. > Has anyone over at CentOS been contacted about this? I used to work > with some of the CentOS maintainers, but haven't spoken with them in a > while, but it may be an "in". > > Please let me know where things stand. > > Thanks! > -Curt Hey Curt, Seth Vidal has been working with Lance from the Centos team on getting the Centos mirroring code. If know or would like to contact Lance about the code, I am sure Seth would not mind. Currently we have the cron job that runs to see what mirror sites are stall. Here is the break down of Centos Mirroring: """ It's broken up into 3 parts: 1. one part queries their mysql database of the mirrors and builds a mirrorlist using geoip 2. another part is a cron job that connects to the mirrors and grabs the repomd.xml off of them to determine 'freshness' 3. the last part generates the web page based on the database info """ In addition to the 3 parts, we also need the table structure of their database. Thanks, for your help! Regards, Jason Hartley

1 0

[Fedora-infrastructure-list] Account System Updates
by TomLy 24 Jun '06

24 Jun '06

As mention in today's fedora-admin meeting, I'm looking to put together the requirements for the new account system. I'll slap the requirements on a wiki page to keep track of what's going on. ~lyz

8 10

[Fedora-infrastructure-list] Introduction
by Aurelien Bompard 23 Jun '06

23 Jun '06

Hey y'all I'm very sorry I missed the meeting yesterday, but I'm still interested in helping here. I'm a sysadmin in the real life, doing web development also. I know Python, so I could help with sysadmin tasks and web interfaces/tools. I'm also familiar with Zope, so when Plone goes live I'll be able to hack in some tools if needed. On the Fedora side, I've been a member of Fedora Extras for quite some time, since the fedora.us days actually. I'm a "sponsor" for Extras, so I hope it proves I'm not going to wreak havoc here... :) For the rest, well, http://fedoraproject.org/wiki/AurelienBompard I had in mind to help with one of the task listed on the Schedule : "Fedora Extras sponsors would like to have a easy way to see the relation between people that needs sponsors and the bug(s) where these people submitted their first package(s)". It looks like lyz is assigned to that, but maybe I could give a hand ? Well, there's a thread dedicated to this issue, so I'll just post there :) I'm glad I'll have an opportunity to help Fedora on the sysadmin side, see ya Aurélien -- http://aurelien.bompard.org ~~~~ Jabber : abompard(a)jabber.fr "Backups are for wimps. Real men upload their work to an ftp server and have everybody mirror it." -- Linus Torvalds

1 0

[Fedora-infrastructure-list] Admin/infrastructure meeting log for 2006-06-22
by Elliot Lee 22 Jun '06

22 Jun '06

Hi all, Attached is the log for today's meeting and the backup discussion that followed... Please make sure you're on fedora-infrastructure-list (https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list/), since that's becoming the primary list for infrastructure discussions. Thanks for everyone who made it to today's meeting! It's really neat to see all the projects that are moving along. Best, -- Elliot

1 0

[Fedora-infrastructure-list] Ticketing system and email addresses
by Mike McGrath 22 Jun '06

22 Jun '06

I've migrated webmaster(a)fedoraproject.org and voting(a)fedoraproject.org to their respective OTRS queues. As we get used to them I'll migrate more and more email addresses. -Mike

1 0

[Fedora-infrastructure-list] FYI - I made some changes on bastion for xferlog scripts
by Jason Hartley 21 Jun '06

21 Jun '06

Elliot- I have the xferlog scripts in place to test on bastion. I have placed xferlogarchive.py and ArchiveEmail.py in /var/fedora-accounts on bastion. Also, I added the procmailrc recipe: :0 * ^TO.*download-logs(a)(fedora.redhat.com|fedoraproject.org) | /var/fedora-accounts/xferlogarchive.py to /var/fedora-accounts/.procmailrc The last thing I did is create the directory /var/xferlogarchive to store the xferlog emails in for the time being until we can get some space on the netappailance. (I will followup with Stacy Brandenburg and Matt Galgoci to see about the netapp space.) I still need to check in the changes to the .procmailrc file and the two scripts. I currently do not have write access to the fedora-accounts module for the .procmailrc file. The two scripts I was not sure where I should stick them in the fedora-config module. Do you want to stick in the top most directory or should they go in some other subdirectory? Regards, Jason Hartley

2 1

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure