cluster-commits May 2010

cluster-commits@lists.fedorahosted.org

9 participants
63 discussions

cluster: RHEL6 - config: Fix ccs_tool create -n
by Fabio M. Di Nitto 25 May '10

25 May '10

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=b4… Commit: b4317363d5db48458a015ac4d4de63a5048a9509 Parent: 4039bf4817a96b6aab20de948389f43b89ce4a8e Author: Christine Caulfield <ccaulfie(a)redhat.com> AuthorDate: Fri May 21 09:09:26 2010 +0100 Committer: Fabio M. Di Nitto <fdinitto(a)redhat.com> CommitterDate: Tue May 25 14:41:48 2010 +0200 config: Fix ccs_tool create -n Make 'ccs_tool create -n <x>' create a skeleton with <x> nodes rather than <x-1>. Resolves: rhbz#594626 Signed-off-by: Christine Caulfield <ccaulfie(a)redhat.com> --- config/tools/ccs_tool/editconf.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/config/tools/ccs_tool/editconf.c b/config/tools/ccs_tool/editconf.c index 323c561..a0371b4 100644 --- a/config/tools/ccs_tool/editconf.c +++ b/config/tools/ccs_tool/editconf.c @@ -1031,7 +1031,7 @@ void create_skeleton(int argc, char **argv) } fprintf(outfile, " <clusternodes>\n"); - for (i=1; i< numnodes; i++) { + for (i=1; i <= numnodes; i++) { fprintf(outfile, " <clusternode name=\"NEEDNAME-%02d\" votes=\"1\" nodeid=\"%d\">\n", i, i); fprintf(outfile, " <fence>\n"); fprintf(outfile, " <method name=\"single\">\n");

1 0

dlm: master - dlm_controld: wrong fencing time comparison
by David Teigland 21 May '10

21 May '10

Gitweb: http://git.fedorahosted.org/git/dlm.git?p=dlm.git;a=commitdiff;h=1ff741d523… Commit: 1ff741d523bda31af31971ba8a898aa391c004ef Parent: 6607f2d209adb983b4642faef8a66f91ba9c727d Author: David Teigland <teigland(a)redhat.com> AuthorDate: Thu May 20 16:40:20 2010 -0500 Committer: David Teigland <teigland(a)redhat.com> CommitterDate: Fri May 21 11:22:14 2010 -0500 dlm_controld: wrong fencing time comparison dlm_controld checks that a failed node has been fenced by comparing the time it was fenced to the time it properly joined the cpg. In one of Nate's QE tests, he found a case where the failed node rejoined the cpg just before the fencing time was recorded. This causes dlm_controld to get stuck waiting for a fencing to occur when it already has. 1274304702 dlm:ls:clvmd conf 2 0 1 memb 1 4 join left 3 1274304702 clvmd add_change cg 2 remove nodeid 3 reason 3 1274304702 clvmd add_change cg 2 counts member 2 joined 0 remove 1 failed 1 1274304702 clvmd stop_kernel cg 2 1274304702 write "0" to "/sys/kernel/dlm/clvmd/control" 1274304702 clvmd check_fencing 3 not fenced add 1274304347 fence 0 1274304702 dlm:ls:clvmd conf 1 0 1 memb 1 join left 4 1274304702 clvmd add_change cg 3 remove nodeid 4 reason 3 1274304702 clvmd add_change cg 3 counts member 1 joined 0 remove 1 failed 1 1274304702 clvmd check_fencing 4 not fenced add 1274304347 fence 0 .... 1274304741 cluster node 3 added ... 1274304752 dlm:ls:clvmd conf 2 1 0 memb 1 3 join 3 left 1274304752 clvmd add_change cg 4 joined nodeid 3 1274304752 clvmd add_change cg 4 counts member 2 joined 1 remove 0 failed 0 1274304754 clvmd receive_start 3:1 len 80 1274304754 clvmd match_change 3:1 skip 2 sender not member 1274304754 clvmd match_change 3:1 skip 3 sender not member 1274304754 clvmd match_change 3:1 matches cg 4 1274304754 clvmd check_fencing 3 not fenced add 1274304754 fence 1274304753 1274304754 clvmd check_fencing 4 1274304347 fenced at 1274304753 ... 1274304832 dlm:ls:clvmd conf 3 1 0 memb 1 3 4 join 4 left 1274304832 clvmd add_change cg 5 joined nodeid 4 1274304832 clvmd add_change cg 5 counts member 3 joined 1 remove 0 failed 0 1274304832 clvmd receive_start 4:1 len 84 1274304832 clvmd match_change 4:1 skip cg 2 created 1274304702 cluster add 12743 1274304832 clvmd match_change 4:1 skip 3 sender not member 1274304832 clvmd match_change 4:1 skip 4 sender not member 1274304832 clvmd match_change 4:1 matches cg 5 1274304832 clvmd receive_start 3:2 len 84 1274304832 clvmd match_change 3:2 skip 2 sender not member 1274304832 clvmd match_change 3:2 skip 3 sender not member 1274304832 clvmd match_change 3:2 skip 4 already start 1274304832 clvmd match_change 3:2 matches cg 5 This line: clvmd check_fencing 3 not fenced add 1274304754 fence 1274304753 shows that 3 was added to the cpg one second after fenced recorded that fencing was done. The fenced log confirms that this is when it received the message indicating the 3 was fenced: 1274304753 receive_victim_done 1:3 flags 2 len 80 1274304753 receive_victim_done 1:3 remove victim 3 time 1274304753 how 2 I've not been able to reproduce this, but I believe the fix is for dlm_controld to check that the fencing time is later than the time it saw the node fail instead of later than the time it saw the node most recently join. bz 594511 Signed-off-by: David Teigland <teigland(a)redhat.com> --- group/dlm_controld/cpg.c | 19 +++++++++++++------ 1 files changed, 13 insertions(+), 6 deletions(-) diff --git a/group/dlm_controld/cpg.c b/group/dlm_controld/cpg.c index 2eb3d65..94a9b68 100644 --- a/group/dlm_controld/cpg.c +++ b/group/dlm_controld/cpg.c @@ -45,6 +45,7 @@ struct node { int check_fs; int fs_notified; uint64_t add_time; + uint64_t fail_time; uint64_t fence_time; /* for debug */ uint64_t cluster_add_time; uint64_t cluster_remove_time; @@ -356,12 +357,13 @@ static void free_ls(struct lockspace *ls) has failed yet). So, check that: - 1. has fenced fenced the node after it joined this lockspace? + 1. has fenced fenced the node since we saw it fail? 2. fenced has no outstanding fencing ops For 1: - - record the time of the first good start message we see from node X - node X fails + - we see node X fail and X has non-zero add_time, + set check_fencing and record the fail time - wait for X to be removed from all dlm cpg's (probably not necessary) - check that the fencing time is later than the recorded time above @@ -499,6 +501,7 @@ static void node_history_fail(struct lockspace *ls, int nodeid, node->check_fencing = 1; node->fence_time = 0; node->fence_queries = 0; + node->fail_time = time(NULL); } /* fenced will take care of making sure the quorum value @@ -539,10 +542,12 @@ static int check_fencing_done(struct lockspace *ls) if (rv < 0) log_error("fenced_node_info error %d", rv); - if (last_fenced_time > node->add_time) { - log_group(ls, "check_fencing %d %llu fenced at %llu", + if (last_fenced_time > node->fail_time) { + log_group(ls, "check_fencing %d done " + "add %llu fail %llu last %llu", node->nodeid, (unsigned long long)node->add_time, + (unsigned long long)node->fail_time, (unsigned long long)last_fenced_time); node->check_fencing = 0; node->add_time = 0; @@ -550,9 +555,11 @@ static int check_fencing_done(struct lockspace *ls) } else { if (!node->fence_queries || node->fence_time != last_fenced_time) { - log_group(ls, "check_fencing %d not fenced " - "add %llu fence %llu", node->nodeid, + log_group(ls, "check_fencing %d wait " + "add %llu fail %llu last %llu", + node->nodeid, (unsigned long long)node->add_time, + (unsigned long long)node->fail_time, (unsigned long long)last_fenced_time); node->fence_queries++; node->fence_time = last_fenced_time;

1 0

cluster: RHEL6 - dlm_controld: wrong fencing time comparison
by David Teigland 21 May '10

21 May '10

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=40… Commit: 4039bf4817a96b6aab20de948389f43b89ce4a8e Parent: 6dd191b5a8044c60caba7109b77583b04823bd30 Author: David Teigland <teigland(a)redhat.com> AuthorDate: Thu May 20 16:40:20 2010 -0500 Committer: David Teigland <teigland(a)redhat.com> CommitterDate: Fri May 21 11:21:33 2010 -0500 dlm_controld: wrong fencing time comparison dlm_controld checks that a failed node has been fenced by comparing the time it was fenced to the time it properly joined the cpg. In one of Nate's QE tests, he found a case where the failed node rejoined the cpg just before the fencing time was recorded. This causes dlm_controld to get stuck waiting for a fencing to occur when it already has. 1274304702 dlm:ls:clvmd conf 2 0 1 memb 1 4 join left 3 1274304702 clvmd add_change cg 2 remove nodeid 3 reason 3 1274304702 clvmd add_change cg 2 counts member 2 joined 0 remove 1 failed 1 1274304702 clvmd stop_kernel cg 2 1274304702 write "0" to "/sys/kernel/dlm/clvmd/control" 1274304702 clvmd check_fencing 3 not fenced add 1274304347 fence 0 1274304702 dlm:ls:clvmd conf 1 0 1 memb 1 join left 4 1274304702 clvmd add_change cg 3 remove nodeid 4 reason 3 1274304702 clvmd add_change cg 3 counts member 1 joined 0 remove 1 failed 1 1274304702 clvmd check_fencing 4 not fenced add 1274304347 fence 0 .... 1274304741 cluster node 3 added ... 1274304752 dlm:ls:clvmd conf 2 1 0 memb 1 3 join 3 left 1274304752 clvmd add_change cg 4 joined nodeid 3 1274304752 clvmd add_change cg 4 counts member 2 joined 1 remove 0 failed 0 1274304754 clvmd receive_start 3:1 len 80 1274304754 clvmd match_change 3:1 skip 2 sender not member 1274304754 clvmd match_change 3:1 skip 3 sender not member 1274304754 clvmd match_change 3:1 matches cg 4 1274304754 clvmd check_fencing 3 not fenced add 1274304754 fence 1274304753 1274304754 clvmd check_fencing 4 1274304347 fenced at 1274304753 ... 1274304832 dlm:ls:clvmd conf 3 1 0 memb 1 3 4 join 4 left 1274304832 clvmd add_change cg 5 joined nodeid 4 1274304832 clvmd add_change cg 5 counts member 3 joined 1 remove 0 failed 0 1274304832 clvmd receive_start 4:1 len 84 1274304832 clvmd match_change 4:1 skip cg 2 created 1274304702 cluster add 12743 1274304832 clvmd match_change 4:1 skip 3 sender not member 1274304832 clvmd match_change 4:1 skip 4 sender not member 1274304832 clvmd match_change 4:1 matches cg 5 1274304832 clvmd receive_start 3:2 len 84 1274304832 clvmd match_change 3:2 skip 2 sender not member 1274304832 clvmd match_change 3:2 skip 3 sender not member 1274304832 clvmd match_change 3:2 skip 4 already start 1274304832 clvmd match_change 3:2 matches cg 5 This line: clvmd check_fencing 3 not fenced add 1274304754 fence 1274304753 shows that 3 was added to the cpg one second after fenced recorded that fencing was done. The fenced log confirms that this is when it received the message indicating the 3 was fenced: 1274304753 receive_victim_done 1:3 flags 2 len 80 1274304753 receive_victim_done 1:3 remove victim 3 time 1274304753 how 2 I've not been able to reproduce this, but I believe the fix is for dlm_controld to check that the fencing time is later than the time it saw the node fail instead of later than the time it saw the node most recently join. bz 594511 Signed-off-by: David Teigland <teigland(a)redhat.com> --- group/dlm_controld/cpg.c | 19 +++++++++++++------ 1 files changed, 13 insertions(+), 6 deletions(-) diff --git a/group/dlm_controld/cpg.c b/group/dlm_controld/cpg.c index 2eb3d65..94a9b68 100644 --- a/group/dlm_controld/cpg.c +++ b/group/dlm_controld/cpg.c @@ -45,6 +45,7 @@ struct node { int check_fs; int fs_notified; uint64_t add_time; + uint64_t fail_time; uint64_t fence_time; /* for debug */ uint64_t cluster_add_time; uint64_t cluster_remove_time; @@ -356,12 +357,13 @@ static void free_ls(struct lockspace *ls) has failed yet). So, check that: - 1. has fenced fenced the node after it joined this lockspace? + 1. has fenced fenced the node since we saw it fail? 2. fenced has no outstanding fencing ops For 1: - - record the time of the first good start message we see from node X - node X fails + - we see node X fail and X has non-zero add_time, + set check_fencing and record the fail time - wait for X to be removed from all dlm cpg's (probably not necessary) - check that the fencing time is later than the recorded time above @@ -499,6 +501,7 @@ static void node_history_fail(struct lockspace *ls, int nodeid, node->check_fencing = 1; node->fence_time = 0; node->fence_queries = 0; + node->fail_time = time(NULL); } /* fenced will take care of making sure the quorum value @@ -539,10 +542,12 @@ static int check_fencing_done(struct lockspace *ls) if (rv < 0) log_error("fenced_node_info error %d", rv); - if (last_fenced_time > node->add_time) { - log_group(ls, "check_fencing %d %llu fenced at %llu", + if (last_fenced_time > node->fail_time) { + log_group(ls, "check_fencing %d done " + "add %llu fail %llu last %llu", node->nodeid, (unsigned long long)node->add_time, + (unsigned long long)node->fail_time, (unsigned long long)last_fenced_time); node->check_fencing = 0; node->add_time = 0; @@ -550,9 +555,11 @@ static int check_fencing_done(struct lockspace *ls) } else { if (!node->fence_queries || node->fence_time != last_fenced_time) { - log_group(ls, "check_fencing %d not fenced " - "add %llu fence %llu", node->nodeid, + log_group(ls, "check_fencing %d wait " + "add %llu fail %llu last %llu", + node->nodeid, (unsigned long long)node->add_time, + (unsigned long long)node->fail_time, (unsigned long long)last_fenced_time); node->fence_queries++; node->fence_time = last_fenced_time;

1 0

cluster: STABLE3 - dlm_controld: wrong fencing time comparison
by David Teigland 21 May '10

21 May '10

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=b1… Commit: b165ce47df6f03c340abf4baa48b74368d12bf54 Parent: 1878f0bbffcf60f00a93bf2ac806af3caae86a49 Author: David Teigland <teigland(a)redhat.com> AuthorDate: Thu May 20 16:40:20 2010 -0500 Committer: David Teigland <teigland(a)redhat.com> CommitterDate: Fri May 21 11:20:28 2010 -0500 dlm_controld: wrong fencing time comparison dlm_controld checks that a failed node has been fenced by comparing the time it was fenced to the time it properly joined the cpg. In one of Nate's QE tests, he found a case where the failed node rejoined the cpg just before the fencing time was recorded. This causes dlm_controld to get stuck waiting for a fencing to occur when it already has. 1274304702 dlm:ls:clvmd conf 2 0 1 memb 1 4 join left 3 1274304702 clvmd add_change cg 2 remove nodeid 3 reason 3 1274304702 clvmd add_change cg 2 counts member 2 joined 0 remove 1 failed 1 1274304702 clvmd stop_kernel cg 2 1274304702 write "0" to "/sys/kernel/dlm/clvmd/control" 1274304702 clvmd check_fencing 3 not fenced add 1274304347 fence 0 1274304702 dlm:ls:clvmd conf 1 0 1 memb 1 join left 4 1274304702 clvmd add_change cg 3 remove nodeid 4 reason 3 1274304702 clvmd add_change cg 3 counts member 1 joined 0 remove 1 failed 1 1274304702 clvmd check_fencing 4 not fenced add 1274304347 fence 0 .... 1274304741 cluster node 3 added ... 1274304752 dlm:ls:clvmd conf 2 1 0 memb 1 3 join 3 left 1274304752 clvmd add_change cg 4 joined nodeid 3 1274304752 clvmd add_change cg 4 counts member 2 joined 1 remove 0 failed 0 1274304754 clvmd receive_start 3:1 len 80 1274304754 clvmd match_change 3:1 skip 2 sender not member 1274304754 clvmd match_change 3:1 skip 3 sender not member 1274304754 clvmd match_change 3:1 matches cg 4 1274304754 clvmd check_fencing 3 not fenced add 1274304754 fence 1274304753 1274304754 clvmd check_fencing 4 1274304347 fenced at 1274304753 ... 1274304832 dlm:ls:clvmd conf 3 1 0 memb 1 3 4 join 4 left 1274304832 clvmd add_change cg 5 joined nodeid 4 1274304832 clvmd add_change cg 5 counts member 3 joined 1 remove 0 failed 0 1274304832 clvmd receive_start 4:1 len 84 1274304832 clvmd match_change 4:1 skip cg 2 created 1274304702 cluster add 12743 1274304832 clvmd match_change 4:1 skip 3 sender not member 1274304832 clvmd match_change 4:1 skip 4 sender not member 1274304832 clvmd match_change 4:1 matches cg 5 1274304832 clvmd receive_start 3:2 len 84 1274304832 clvmd match_change 3:2 skip 2 sender not member 1274304832 clvmd match_change 3:2 skip 3 sender not member 1274304832 clvmd match_change 3:2 skip 4 already start 1274304832 clvmd match_change 3:2 matches cg 5 This line: clvmd check_fencing 3 not fenced add 1274304754 fence 1274304753 shows that 3 was added to the cpg one second after fenced recorded that fencing was done. The fenced log confirms that this is when it received the message indicating the 3 was fenced: 1274304753 receive_victim_done 1:3 flags 2 len 80 1274304753 receive_victim_done 1:3 remove victim 3 time 1274304753 how 2 I've not been able to reproduce this, but I believe the fix is for dlm_controld to check that the fencing time is later than the time it saw the node fail instead of later than the time it saw the node most recently join. bz 594511 Signed-off-by: David Teigland <teigland(a)redhat.com> --- group/dlm_controld/cpg.c | 19 +++++++++++++------ 1 files changed, 13 insertions(+), 6 deletions(-) diff --git a/group/dlm_controld/cpg.c b/group/dlm_controld/cpg.c index 2eb3d65..94a9b68 100644 --- a/group/dlm_controld/cpg.c +++ b/group/dlm_controld/cpg.c @@ -45,6 +45,7 @@ struct node { int check_fs; int fs_notified; uint64_t add_time; + uint64_t fail_time; uint64_t fence_time; /* for debug */ uint64_t cluster_add_time; uint64_t cluster_remove_time; @@ -356,12 +357,13 @@ static void free_ls(struct lockspace *ls) has failed yet). So, check that: - 1. has fenced fenced the node after it joined this lockspace? + 1. has fenced fenced the node since we saw it fail? 2. fenced has no outstanding fencing ops For 1: - - record the time of the first good start message we see from node X - node X fails + - we see node X fail and X has non-zero add_time, + set check_fencing and record the fail time - wait for X to be removed from all dlm cpg's (probably not necessary) - check that the fencing time is later than the recorded time above @@ -499,6 +501,7 @@ static void node_history_fail(struct lockspace *ls, int nodeid, node->check_fencing = 1; node->fence_time = 0; node->fence_queries = 0; + node->fail_time = time(NULL); } /* fenced will take care of making sure the quorum value @@ -539,10 +542,12 @@ static int check_fencing_done(struct lockspace *ls) if (rv < 0) log_error("fenced_node_info error %d", rv); - if (last_fenced_time > node->add_time) { - log_group(ls, "check_fencing %d %llu fenced at %llu", + if (last_fenced_time > node->fail_time) { + log_group(ls, "check_fencing %d done " + "add %llu fail %llu last %llu", node->nodeid, (unsigned long long)node->add_time, + (unsigned long long)node->fail_time, (unsigned long long)last_fenced_time); node->check_fencing = 0; node->add_time = 0; @@ -550,9 +555,11 @@ static int check_fencing_done(struct lockspace *ls) } else { if (!node->fence_queries || node->fence_time != last_fenced_time) { - log_group(ls, "check_fencing %d not fenced " - "add %llu fence %llu", node->nodeid, + log_group(ls, "check_fencing %d wait " + "add %llu fail %llu last %llu", + node->nodeid, (unsigned long long)node->add_time, + (unsigned long long)node->fail_time, (unsigned long long)last_fenced_time); node->fence_queries++; node->fence_time = last_fenced_time;

1 0

cluster: STABLE3 - config: Fix ccs_tool create -n
by Christine Caulfield 21 May '10

21 May '10

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=18… Commit: 1878f0bbffcf60f00a93bf2ac806af3caae86a49 Parent: 6b676e1ccc192a6c1396f72b637ba89aa254fd17 Author: Christine Caulfield <ccaulfie(a)redhat.com> AuthorDate: Fri May 21 09:09:26 2010 +0100 Committer: Christine Caulfield <ccaulfie(a)redhat.com> CommitterDate: Fri May 21 09:09:26 2010 +0100 config: Fix ccs_tool create -n Make 'ccs_tool create -n <x>' create a skeleton with <x> nodes rather than <x-1>. Signed-off-by: Christine Caulfield <ccaulfie(a)redhat.com> --- config/tools/ccs_tool/editconf.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/config/tools/ccs_tool/editconf.c b/config/tools/ccs_tool/editconf.c index 323c561..a0371b4 100644 --- a/config/tools/ccs_tool/editconf.c +++ b/config/tools/ccs_tool/editconf.c @@ -1031,7 +1031,7 @@ void create_skeleton(int argc, char **argv) } fprintf(outfile, " <clusternodes>\n"); - for (i=1; i< numnodes; i++) { + for (i=1; i <= numnodes; i++) { fprintf(outfile, " <clusternode name=\"NEEDNAME-%02d\" votes=\"1\" nodeid=\"%d\">\n", i, i); fprintf(outfile, " <fence>\n"); fprintf(outfile, " <method name=\"single\">\n");

1 0

cluster: RHEL55 - resource-agents: Add status_program attribute
by Lon Hohberger 20 May '10

20 May '10

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=ea… Commit: ea76067bc05359c1582bfd84be4c1b3eb05b230a Parent: 61a7c0f26c4248778d656bcecb47375c419fafd6 Author: Lon Hohberger <lhh(a)redhat.com> AuthorDate: Tue May 4 17:54:15 2010 -0400 Committer: Lon Hohberger <lhh(a)redhat.com> CommitterDate: Thu May 20 16:43:47 2010 -0400 resource-agents: Add status_program attribute Resolves: bz594476 Signed-off-by: Lon Hohberger <lhh(a)redhat.com> --- rgmanager/src/resources/vm.sh | 33 ++++++++++++++++++++++++++++++++- 1 files changed, 32 insertions(+), 1 deletions(-) diff --git a/rgmanager/src/resources/vm.sh b/rgmanager/src/resources/vm.sh index d2932dd..2a995eb 100755 --- a/rgmanager/src/resources/vm.sh +++ b/rgmanager/src/resources/vm.sh @@ -213,6 +213,20 @@ meta_data() <content type="string" default="0"/> </parameter> + <parameter name="status_program" reconfig="1"> + <longdesc lang="en"> + Ordinarily, only the presence/health of a virtual machine + is checked. If specified, the status_program value is + executed during a depth 10 check. The intent of this + program is to ascertain the status of critical services + within a virtual machine. + </longdesc> + <shortdesc lang="en"> + Additional status check program + </shortdesc> + <content type="string" default="0"/> + </parameter> + <parameter name="hypervisor"> <shortdesc lang="en"> Hypervisor @@ -255,6 +269,10 @@ meta_data() <action name="status" timeout="10" interval="30"/> <action name="monitor" timeout="10" interval="30"/> +  + <action name="status" depth="10" timeout="10" interval="300"/> + <action name="monitor" depth="10" timeout="10" interval="300"/> +  <action name="reconfig" timeout="10"/> @@ -543,7 +561,11 @@ virsh_status() return 0 fi - return 1 + if [ "$state" = "shut off" ]; then + return $OCF_NOT_RUNNING + fi + + return $OCF_ERR_GENERIC } @@ -923,6 +945,15 @@ case $1 in validate_all || exit $OCF_ERR_ARGS echo -n "Virtual machine $OCF_RESKEY_name is " do_status + rv=$? + if [ $rv -ne 0 ]; then + exit $rv + fi + [ -z "$OCF_RESKEY_status_program" ] && exit 0 + [ -z "$OCF_CHECK_LEVEL" ] && exit 0 + [ $OCF_CHECK_LEVEL -lt 10 ] && exit 0 + + bash -c "$OCF_RESKEY_status_program" &> /dev/null exit $? ;; migrate)

1 0

cluster: RHEL55 - rgmanager: Add per-resource status check tolerances
by Lon Hohberger 20 May '10

20 May '10

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=61… Commit: 61a7c0f26c4248778d656bcecb47375c419fafd6 Parent: 7a1e2af8cc37409b3a38fbe9b5945d186be9f2ec Author: Lon Hohberger <lhh(a)redhat.com> AuthorDate: Tue May 4 17:51:45 2010 -0400 Committer: Lon Hohberger <lhh(a)redhat.com> CommitterDate: Thu May 20 16:43:35 2010 -0400 rgmanager: Add per-resource status check tolerances Resolves: bz594476 Signed-off-by: Lon Hohberger <lhh(a)redhat.com> --- rgmanager/include/reslist.h | 1 + rgmanager/src/daemons/restree.c | 47 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 0 deletions(-) diff --git a/rgmanager/include/reslist.h b/rgmanager/include/reslist.h index 0275bfd..001f56b 100644 --- a/rgmanager/include/reslist.h +++ b/rgmanager/include/reslist.h @@ -128,6 +128,7 @@ typedef struct _rg_node { resource_t *rn_resource; resource_act_t *rn_actions; restart_counter_t rn_restart_counter; + restart_counter_t rn_failure_counter; int rn_state; /* State of this instance of rn_resource */ int rn_flags; int rn_last_status; diff --git a/rgmanager/src/daemons/restree.c b/rgmanager/src/daemons/restree.c index d5ffea1..3f07f8f 100644 --- a/rgmanager/src/daemons/restree.c +++ b/rgmanager/src/daemons/restree.c @@ -565,6 +565,8 @@ do_load_resource(int ccsfd, char *base, char *ref; resource_node_t *node; resource_t *curres; + time_t failure_expire = 0; + int max_failures = 0; snprintf(tok, sizeof(tok), "%s/@ref", base); @@ -659,6 +661,36 @@ do_load_resource(int ccsfd, char *base, free(ref); } + /* per-resource-node failures / expire times */ + snprintf(tok, sizeof(tok), "%s/@__max_failures", base); +#ifndef NO_CCS + if (ccs_get(ccsfd, tok, &ref) == 0) { +#else + if (conf_get(tok, &ref) == 0) { +#endif + max_failures = atoi(ref); + if (max_failures < 0) + max_failures = 0; + free(ref); + } + + snprintf(tok, sizeof(tok), "%s/@__failure_expire_time", base); +#ifndef NO_CCS + if (ccs_get(ccsfd, tok, &ref) == 0) { +#else + if (conf_get(tok, &ref) == 0) { +#endif + failure_expire = (time_t)expand_time(ref); + if ((int64_t)failure_expire < 0) + failure_expire = 0; + free(ref); + } + + if (max_failures && failure_expire) { + node->rn_failure_counter = restart_init(failure_expire, + max_failures); + } + curres->r_refs++; *newnode = node; @@ -906,6 +938,10 @@ destroy_resource_tree(resource_node_t **tree) restart_cleanup(node->rn_restart_counter); } + if (node->rn_failure_counter) { + restart_cleanup(node->rn_failure_counter); + } + if(node->rn_actions){ free(node->rn_actions); } @@ -1204,6 +1240,15 @@ do_status(resource_node_t *node) * completed. */ node->rn_actions[idx].ra_last = time(NULL); + /* If we have not exceeded our failure count threshold, then fudge + * the status check this round */ + if (x && node->rn_failure_counter) { + if (!restart_threshold_exceeded(node->rn_failure_counter)) { + x = 0; + restart_add(node->rn_failure_counter); + } + } + node->rn_last_status = x; node->rn_last_depth = node->rn_actions[idx].ra_depth; node->rn_checked = 1; @@ -1277,6 +1322,8 @@ clear_checks(resource_node_t *node) node->rn_actions[x].ra_last = now; } + restart_clear(node->rn_failure_counter); + node->rn_checked = 0; node->rn_last_status = 0; node->rn_last_depth = 0;

1 0

cluster: RHEL55 - rgmanager: Fix max_restart threshold handling
by Lon Hohberger 20 May '10

20 May '10

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=7a… Commit: 7a1e2af8cc37409b3a38fbe9b5945d186be9f2ec Parent: 75c7ecae61d9400d084227304d5b7068fdaa310b Author: Lon Hohberger <lhh(a)redhat.com> AuthorDate: Tue May 4 15:03:23 2010 -0400 Committer: Lon Hohberger <lhh(a)redhat.com> CommitterDate: Thu May 20 16:43:18 2010 -0400 rgmanager: Fix max_restart threshold handling Max_restart / restart_expire_time was not being correctly handled, resulting in inconsistent amounts of restarts prior to relocation (or in some cases, no relocation at all) Resolves: bz594476 Signed-off-by: Lon Hohberger <lhh(a)redhat.com> --- rgmanager/include/resgroup.h | 1 + rgmanager/src/daemons/groups.c | 16 ++++++++++++++++ rgmanager/src/daemons/restart_counter.c | 5 +++++ rgmanager/src/daemons/rg_state.c | 6 +++++- rgmanager/src/daemons/slang_event.c | 20 ++++++++++++++++++-- 5 files changed, 45 insertions(+), 3 deletions(-) diff --git a/rgmanager/include/resgroup.h b/rgmanager/include/resgroup.h index ec3e67d..c8e7e2d 100644 --- a/rgmanager/include/resgroup.h +++ b/rgmanager/include/resgroup.h @@ -144,6 +144,7 @@ int svc_freeze(char *svcName); int svc_unfreeze(char *svcName); int svc_migrate(char *svcName, int target); int check_restart(char *svcName); +int clear_restart(const char *rg_name); int add_restart(char *svcName); int rt_enqueue_request(const char *resgroupname, int request, diff --git a/rgmanager/src/daemons/groups.c b/rgmanager/src/daemons/groups.c index 9caf746..9818aa3 100644 --- a/rgmanager/src/daemons/groups.c +++ b/rgmanager/src/daemons/groups.c @@ -1945,6 +1945,22 @@ check_restart(char *rg_name) return ret; } +int +clear_restart(const char *rg_name) +{ + resource_node_t *node; + int ret = 0; + + pthread_rwlock_rdlock(&resource_lock); + node = node_by_ref(&_tree, rg_name); + if (node) { + ret = restart_clear(node->rn_restart_counter); + } + pthread_rwlock_unlock(&resource_lock); + + return ret; +} + void kill_resource_groups(void) diff --git a/rgmanager/src/daemons/restart_counter.c b/rgmanager/src/daemons/restart_counter.c index c889c6d..d909713 100644 --- a/rgmanager/src/daemons/restart_counter.c +++ b/rgmanager/src/daemons/restart_counter.c @@ -107,6 +107,11 @@ restart_threshold_exceeded(restart_counter_t arg) restart_info_t *restarts = (restart_info_t *)arg; time_t now; + if (!arg) + /* No max restarts / threshold = always + ok to restart! */ + return 0; + VALIDATE(arg, -1); now = time(NULL); restart_timer_purge(arg, now); diff --git a/rgmanager/src/daemons/rg_state.c b/rgmanager/src/daemons/rg_state.c index 49464d5..3e57748 100644 --- a/rgmanager/src/daemons/rg_state.c +++ b/rgmanager/src/daemons/rg_state.c @@ -812,7 +812,6 @@ svc_start(char *svcName, int req) svcStatus.rs_transition = (uint64_t)time(NULL); if (svcStatus.rs_state == RG_STATE_RECOVER) { - add_restart(svcName); svcStatus.rs_restarts++; } else { svcStatus.rs_restarts = 0; @@ -2143,20 +2142,25 @@ handle_recover_req(char *svcName, int *new_owner) get_recovery_policy(svcName, policy, sizeof(policy)); if (!strcasecmp(policy, "disable")) { + clear_restart(svcName); return svc_disable(svcName); } else if (!strcasecmp(policy, "relocate")) { + clear_restart(svcName); return handle_relocate_req(svcName, RG_START_RECOVER, -1, new_owner); } /* Check restart counter/timer for this resource */ if (check_restart(svcName) > 0) { + clear_restart(svcName); clulog(LOG_NOTICE, "Restart threshold for %s exceeded; " "attempting to relocate\n", svcName); return handle_relocate_req(svcName, RG_START_RECOVER, -1, new_owner); } + add_restart(svcName); + return handle_start_req(svcName, RG_START_RECOVER, new_owner); } diff --git a/rgmanager/src/daemons/slang_event.c b/rgmanager/src/daemons/slang_event.c index 8a5493c..086ac6b 100644 --- a/rgmanager/src/daemons/slang_event.c +++ b/rgmanager/src/daemons/slang_event.c @@ -1123,22 +1123,38 @@ int S_service_event(const char *file, const char *script, char *name, int state, int owner, int last_owner) { + char policy[20]; int ret; _service_name = name; _service_state = (char *)rg_state_str(state); _service_owner = owner; _service_last_owner = last_owner; - _service_restarts_exceeded = check_restart(name); + _service_restarts_exceeded = 0; switch(state) { case RG_STATE_DISABLED: case RG_STATE_STOPPED: case RG_STATE_FAILED: - case RG_STATE_RECOVER: case RG_STATE_ERROR: /* There is no owner for these states. Ever. */ _service_owner = -1; + break; + case RG_STATE_RECOVER: + get_recovery_policy(name, policy, sizeof(policy)); + if (!strcasecmp(policy, "restart")) { + _service_restarts_exceeded = check_restart(name); + add_restart(name); + } else if (!strcasecmp(policy, "relocate")) { + /* Restart threshold is always exceeded + * with 'relocate' recovery policy */ + _service_restarts_exceeded = 1; + } + _service_owner = -1; + } + + if (_service_restarts_exceeded) { + clear_restart(name); } ret = do_slang_run(file, script);

1 0

cluster: RHEL55 - rgmanager: Fix resource start/check times
by Lon Hohberger 20 May '10

20 May '10

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=75… Commit: 75c7ecae61d9400d084227304d5b7068fdaa310b Parent: 5bde34c1fee9e5c5ba70d4997898328eb07d5b70 Author: Lon Hohberger <lhh(a)redhat.com> AuthorDate: Tue May 4 14:58:58 2010 -0400 Committer: Lon Hohberger <lhh(a)redhat.com> CommitterDate: Thu May 20 16:42:46 2010 -0400 rgmanager: Fix resource start/check times Resolves: bz#594476 Signed-off-by: Lon Hohberger <lhh(a)redhat.com> --- rgmanager/include/reslist.h | 1 - rgmanager/src/daemons/restree.c | 3 +-- 2 files changed, 1 insertions(+), 3 deletions(-) diff --git a/rgmanager/include/reslist.h b/rgmanager/include/reslist.h index 4ce3838..0275bfd 100644 --- a/rgmanager/include/reslist.h +++ b/rgmanager/include/reslist.h @@ -115,7 +115,6 @@ typedef struct _resource { char * r_name; resource_attr_t * r_attrs; resource_act_t * r_actions; - time_t r_started; /** Time this resource was last started */ int r_flags; int r_refs; int r_incarnations; /** Number of instances running locally */ diff --git a/rgmanager/src/daemons/restree.c b/rgmanager/src/daemons/restree.c index c4634a5..d5ffea1 100644 --- a/rgmanager/src/daemons/restree.c +++ b/rgmanager/src/daemons/restree.c @@ -1265,9 +1265,8 @@ clear_checks(resource_node_t *node) { time_t now; int x = 0; - resource_t *res = node->rn_resource; - now = res->r_started; + now = get_time((char *)"start", 0, node); for (; node->rn_actions[x].ra_name; x++) {

1 0

cluster: RHEL4 - ccsd: Fix bug 49898 - symlink attack vulnerabilities
by Jonathan Brassow 20 May '10

20 May '10

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=bb… Commit: bba9068b91f737b8f88cd26581eed5884f443bae Parent: 6330bbe58b809aaf5dffc4be87e632b427a4b568 Author: Jonathan Brassow <jbrassow(a)redhat.com> AuthorDate: Thu May 20 11:31:49 2010 -0500 Committer: Jonathan Brassow <jbrassow(a)redhat.com> CommitterDate: Thu May 20 11:31:49 2010 -0500 ccsd: Fix bug 49898 - symlink attack vulnerabilities Fix vulnerability in CCS that would allow someone to prevent a 'ccs_tool update'. --- ccs/ccs_tool/upgrade.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/ccs/ccs_tool/upgrade.c b/ccs/ccs_tool/upgrade.c index b085407..cb844b9 100644 --- a/ccs/ccs_tool/upgrade.c +++ b/ccs/ccs_tool/upgrade.c @@ -193,7 +193,7 @@ static void ccs_dh_in(ccs_dh_t *dh, char *buf){ static int upgrade_device_archive(char *location){ int error = 0; int dev_fd=-1, tmp_fd=-1; - char tmp_file[64]; + char tmp_file[128]; char *buffer = NULL; ccs_dh_t dev_header; @@ -229,9 +229,10 @@ static int upgrade_device_archive(char *location){ goto fail; } - sprintf(tmp_file, "/tmp/tmp_%d", getpid()); + memset(tmp_file, 0, 128); + sprintf(tmp_file, "/tmp/ccs_tool_tmp_XXXXXX"); - tmp_fd = open(tmp_file, O_RDWR | O_CREAT |O_TRUNC, S_IRUSR|S_IWUSR); + tmp_fd = mkostemp(tmp_file, O_RDWR | O_CREAT |O_TRUNC); if(tmp_fd < 0){ fprintf(stderr, "Unable to create temporary archive: %s\n", strerror(errno)); error = -errno;

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

cluster-commits May 2010