We've talked about replacing blocker bug review meetings with something else for a long time. The meeting has an upside of a higher communication bandwidth, but also a downside of requiring participants to be available at the same time (throughout the world), often being extremely long, and being scheduled just once per week (because of mentioned downsides). I've put some thoughts into making the process asynchronous, and you can see all the options I considered described below. I close up with a proposal for the best choice I found.
Asynchronous blocker review process will: - allow more people to participate (they are not restricted to a particular time, which is inconvenient for some people) - allow us to better gather feedback from maintainers/developers by CCing them - possibly allow a faster turn-around for voting (if you propose a blocker on Tuesday, it doesn't have to wait for the next Monday's meeting) - reduce frustration from long meetings (but perhaps add different ones) - restrict our communication bandwidth somewhat (real-time chat makes it easier to debug issues than e.g. email)
I think we'll need to try it first to decide if we like it or not. Here are my thoughts for different systems we could use:
Blocker Bugs App (DIY solution) ******************
We can implement all the necessary features to our Blocker Bugs App [1]. We even talked about it in the past. There would be a Voting page where discussions could be held for each proposed blocker, and a voting system. We could tailor everything exactly to our needs. But it also implies a lot of time spent in development, writing from scratch features that already exist elsewhere (discussions, notifications, etc) and a long-term maintenance of it (we want out blocker decisions to be available basically indefinitely, ideally with a URL that doesn't change). I'm skeptical that we want to devote a lot of our time to implement such a system, and I don't even think it's a smart thing to do, if there are existing solutions (implemented and maintained by other people) which could work "well enough". That's why I'm mentioning this solution, but try to avoid it as much as possible, and explore other systems below.
[1] https://qa.fedoraproject.org/blockerbugs/
Mailing lists ************
This is trivial to use. For every proposed blocker, we simply start a thread on test list (with a predefined subject format), discuss & decide. It's simple, but also quite bare-bones. We can CC people easily, but that tends to be complicated with people not subscribed to the list. It's very easy to see just a partial conversation, due to the usual reply headers madness. We can link this conversation into the bug report using a hyperkitty view (as long as we have hyperkitty), but it suffers from the same issues, possibly showing just a partial conversation. It's very hard to count the votes across all the replies. It's also hard to track the status of the conversation (initial discussion, voting in progress, resolved, etc). Overall, mailing lists are workable, but far from a good solution, and that's why I'm looking into other solutions below.
Bugzilla ********
For each proposed blocker, we can create (ideally auto-create) a parallel bug report in Bugzilla (against "distribution" or a special component that we create). We will link those two bugs together, using Blocks or External Trackers. Then we simply have the conversation there. We can teach Blocker Bugs App to link to the "discussion bug" from its UI. We can easily CC and "needinfo" other people. Bugzilla automatically sends notifications for new comments, and anyone can subscribe to all blocker bug discussions (either by using Bugzilla's own component watcher, or by watching the appropriate distgit component in Pagure, if available). You can rely on having long-term stable URLs. You can list all open and closed blocker discussions tickets. We can use some additional status for marking "voting in progress" (though that won't be fully obvious). This all sounds pretty good. A slight drawback is that it's not easy to count votes - you have to go through all the comments, find the votes and manually count them. We could create a script that would parse all comments and update the vote summary in the Whiteboard (or perhaps Environment or Doc Text, which is multiline) after each discussion update. I believe Bugzilla is a decent solution for async blocker bug meetings. Its UI is not very pretty, but it has most of the necessary functions.
Pagure *******
This is very similar to the Bugzilla description. For each proposed blocker, we (auto-)create a ticket in a "fedora-blockers" project, for the purpose of a blocker discussion. We interlink the bug and the ticket. We teach Blocker Bugs App to link to the discussion ticket from its UI. We can easily ping other people using @name tagging. Pagure automatically sends notifications for new comments, and anyone can subscribe to all blocker bug discussions using Pagure UI of the project. We can rely on having a long-term stable URLs. You can easily list all open and closed blocker discussion tickets. We can use tags to nicely distinguish tickets, for example blockers vs freeze exceptions, or the state of the conversation (initial discussion, voting in progress, accepted, rejected, etc). The modern UI with tags is very helpful here, allowing us to have a modern and an intuitive process. We still can't count votes easily, but we can create a script that will parse all comments and update the vote summary in the initial description after each ticket update. Thanks to FAS integration, selected people (e.g. members of a FAS group) can have elevated privileges in these tickets allowing them to control the blocker watcher script, similarly to how we control the IRC bot during IRC meetings. I see Pagure as a similar option to Bugzilla, just with much better UI/navigation and a few extra shinies.
The imagined workflow could look like this: 1. A blocker bug is proposed (using either Bugzilla directly or our Blocker Bugs App). 2. Blocker Bugs App (BBA) detects the new blocker and creates a new ticket in Pagure in the "fedora-blockers" project, then updates the bug to link to this ticket (a new comment, a Links entry). It shows both the bug and the discussion ticket in its UI. 3. All fedora-blockers subscribers are notified of the new ticket (i.e. the newly proposed blocker). 4. People discuss the problem in the Pagure ticket. 5. Once some kind of understanding of the issue is formed, a privileged member (e.g. a member of @fedora-qa FAS group) can start the vote by including a command in his comment, e.g. "START VOTE" on a separate line. (Note that this is just a proposal. We can easily let anyone start the vote, or we can allow voting right from the beginning without an explicit start.) 6. The blocker bot (probably part of BBA) listens to all fedora-blockers Pagure tickets changes, and once it detects a vote start command, it updates the ticket e.g. with a "vote in progress" tag. 7. People vote by submitting comments containing VOTE +1/0/-1 on a separate line (and including any justification or feedback they wish in the comment as well; the command has to simply be on its own line so that we can detect it well). 8. The blocker bot listens to ticket updates and after each vote command, it updates the ticket's initial description with a summary of the vote (how many people voted for and against, including names, and hyperlinks to their comments). People can easily correct their vote by sending another comment, the bot will use the last one. People can easily see the vote status by just looking at the ticket's initial description. 9. If new important information appear or the vote needs to be repeated for any other reason, a privileged member can restart the vote e.g. by issuing RESTART VOTE command. (We can have more such administrative commands for any purpose we need, same as we have them with IRC bots). 10. Once the vote has been held for a long enough time (we can either set some standards, or it can be up to QA members judgment), a privileged member can end the vote by issuing AGREED <RESOLUTION> command. The <RESOLUTION> part will be standardized, it can look like "AGREED RejectedBlocker AcceptedFreezeException Some explanation here". 11. The blocker bot will detect this command and update the bug in Bugzilla by adjusting necessary fields, adding a comment that includes the provided explanation and linking to the Pagure ticket once again. It will also update the Pagure ticket, i.e. update necessary tags (e.g. "RejectedBlocker", "AcceptedFreezeException") and close the ticket. 12. goto 1
Note that we don't need the blocker bot to be fully or even partially implemented, just to try this out. We can easily do the bot steps manually in the beginning and decide whether we like this process, before we start working on the automation. We can pretty much start testing this right now, if we want.
What are your thoughts? Does the proposal sound reasonable? Would you change something? Would you use a different backend system? Have I forgotten something important? Feedback welcome.
Thanks, Kamil
On Thu, 2019-11-28 at 13:29 +0100, Kamil Paral wrote:
What are your thoughts? Does the proposal sound reasonable? Would you change something? Would you use a different backend system? Have I forgotten something important? Feedback welcome.
So overall I agree with a lot of what you wrote. It does cause me to wonder if writing some kind of plugin/extension for Pagure, or just writing the functionality into the blockerbugs app, might possibly be *less* work than writing and maintaining the 'blocker bot', however. My other concern would be that if we use something other than the blockerbugs app or Bugzilla, we now have *three* different systems involved in the blocker review process - Bugzilla, the blockerbugs app, and the discussion system. That seems like a lot of moving parts to keep synchronized.
Thanks for working on this!
On Thu, Nov 28, 2019 at 5:14 PM Adam Williamson adamwill@fedoraproject.org wrote:
So overall I agree with a lot of what you wrote. It does cause me to wonder if writing some kind of plugin/extension for Pagure, or just writing the functionality into the blockerbugs app, might possibly be *less* work than writing and maintaining the 'blocker bot', however.
You mean patching Pagure to allow for voting? It's pretty specific to our use case, I'm not sure if Pagure would be interested in having that. There is something similar on Github/Gitlab by acking/nacking a pull request, but that is for PRs where it makes sense. We would need to use it on tickets, where it generally doesn't make much sense. I also considered voting by adding thumbs-up/thumbs-down reactions (already supported by Pagure), but that is a process that doesn't create any activity log and the result can be arbitrarily changed at any time, even after voting has ended.
The process of VOTE +1/0/-1 and updating the summary is very lightweight on the blocker bot functionality (it should be really trivial to implement) and it doesn't hurt us much if the bot goes unresponsive - counting manually is not that much work. I see voting in BBA as a much more daunting task, because of authentication, notifications, long term logging, making sure the service is up at all times, etc. Also it wouldn't make much sense to have discussion and voting at two separate places, which means having the discussions in BBA as well, which brings yet more requirements.
The other part of the bot, mainly updating blocker bugs in Bugzilla automatically, was something we've talked about a long time and that can't be done with any Pagure extensions etc.
My other concern would be that if we use something other than the blockerbugs app or Bugzilla, we now have *three* different systems involved in the blocker review process - Bugzilla, the blockerbugs app, and the discussion system. That seems like a lot of moving parts to keep synchronized.
That is a valid concern. But, we already have three systems, the third being IRC :-) And that would get replaced by Pagure. If IRC is down, our processes are negatively affected (the same would happen if Pagure was down). My main goal is to avoid being overly reliant on BBA. If Bugzilla or Pagure are down, it's a Fedora-wide crisis and it will get resolved promptly. If BBA is down, just a few people from our team can do something about it, and if that is a complex issue, the priority to getting external help is much lower than with Bugzilla or Pagure. I also want us to stay away from the sysadmin roles as much as we can. So BBA (including the blocker bot, if we want to distinguish that) plays, in my ideal world, just a support role, but we don't depend on it. If things break, we can always do the same thing manually (list proposed blockers in bugzilla, have the discussion there or elsewhere, then set the right fields, etc). And that principle is valid even in the proposed scenario.
You are right that synchronization issues might occur, especially when we start displaying the same data in several places. The canonical source is and will be Bugzilla, but we display blocker/FE status in BBA and now the same info would be visible through Pagure ticket tags. That... might not be a good idea. I was thrilled by the idea of having an easy-to-look-at and easy-to-use list of current and past blockers/FEs (because often I need to find a past accepted/rejected blocker and that is a painful experience using Bugzilla), but it's not necessary. All the tags are just sugar on top. Perhaps we should start without it, and add it later, if we feel it would be useful and we have no synchronization issues.
On Thu, 28 Nov 2019 08:13:48 -0800 Adam Williamson adamwill@fedoraproject.org wrote:
On Thu, 2019-11-28 at 13:29 +0100, Kamil Paral wrote:
What are your thoughts? Does the proposal sound reasonable? Would you change something? Would you use a different backend system? Have I forgotten something important? Feedback welcome.
So overall I agree with a lot of what you wrote. It does cause me to wonder if writing some kind of plugin/extension for Pagure, or just writing the functionality into the blockerbugs app, might possibly be *less* work than writing and maintaining the 'blocker bot', however. My other concern would be that if we use something other than the blockerbugs app or Bugzilla, we now have *three* different systems involved in the blocker review process - Bugzilla, the blockerbugs app, and the discussion system. That seems like a lot of moving parts to keep synchronized.
Thanks for working on this!
Has anyone poked at scoping out the work required for either the bot or the enhancements we would need for pagure?
It might be interesting to look at porting the blockerbugs app to django if we can make use of some existing django plugins. In the past, I would have hesitated to do this because of the packaging requirements and the amount of fun that plugin ecosystems can present but we're going to have to move to openshift sooner than later which makes the prospect of using something like Django a lot less daunting.
Tim
On Tue, Dec 3, 2019 at 10:22 PM Tim Flink tflink@redhat.com wrote:
Has anyone poked at scoping out the work required for either the bot or the enhancements we would need for pagure?
Lukáš Brabec looked into it, and creating a new Pagure ticket is a matter of a few lines of code (already written). Extending BBA with a "discussion link" in each blocker row should be trivial as well. I think that's all we need for trying it out.
The blocker bot watching and updating the issue should be fairly easy as well, I hope, but as described above, we've hit an issue where Pagure API doesn't allow comment editing. Lukáš implemented a workaround that adds a new metadata field on the right side of each Pagure ticket showing the count summary. It's barebones, but at least something. We've haven't yet looked into writing the listener code, that would listen for Pagure events on the message bus and react to them. We also haven't looked at Bugzilla updater as described in the proposed workflow (that's really a cherry on top, we don't need it right away).
It might be interesting to look at porting the blockerbugs app to django if we can make use of some existing django plugins. In the past, I would have hesitated to do this because of the packaging requirements and the amount of fun that plugin ecosystems can present but we're going to have to move to openshift sooner than later which makes the prospect of using something like Django a lot less daunting.
Ehh. Hmm. I'd really like to avoid being responsible for showing the discussion, handling authentication, sending out notifications, etc. Right now, if BBA is down, we're fine, it's just less comfortable. If we own the full process, it's much more demanding. Also, converting BBA to Django and using some discussion+votes plugins sounds like more work than those quite simple changes in BBA + blocker bot. Plus Josef is the only one except you who has some Django experience, I think.
19/12/4 15:34(e)an, Kamil Paral igorleak idatzi zuen:
On Tue, Dec 3, 2019 at 10:22 PM Tim Flink <tflink@redhat.com mailto:tflink@redhat.com> wrote:
Has anyone poked at scoping out the work required for either the bot or the enhancements we would need for pagure?Lukáš Brabec looked into it, and creating a new Pagure ticket is a matter of a few lines of code (already written). Extending BBA with a "discussion link" in each blocker row should be trivial as well. I think that's all we need for trying it out.
The blocker bot watching and updating the issue should be fairly easy as well, I hope, but as described above, we've hit an issue where Pagure API doesn't allow comment editing. Lukáš implemented a workaround that adds a new metadata field on the right side of each Pagure ticket showing the count summary. It's barebones, but at least something. We've haven't yet looked into writing the listener code, that would listen for Pagure events on the message bus and react to them. We also haven't looked at Bugzilla updater as described in the proposed workflow (that's really a cherry on top, we don't need it right away).
Implementing an api endpoint to edit comment is easy, open an RFE on pagure's issue tracker if you need it and I'll take it
It might be interesting to look at porting the blockerbugs app to django if we can make use of some existing django plugins. In the past, I would have hesitated to do this because of the packaging requirements and the amount of fun that plugin ecosystems can present but we're going to have to move to openshift sooner than later which makes the prospect of using something like Django a lot less daunting.Ehh. Hmm. I'd really like to avoid being responsible for showing the discussion, handling authentication, sending out notifications, etc. Right now, if BBA is down, we're fine, it's just less comfortable. If we own the full process, it's much more demanding. Also, converting BBA to Django and using some discussion+votes plugins sounds like more work than those quite simple changes in BBA + blocker bot. Plus Josef is the only one except you who has some Django experience, I think.
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
On Thu, Dec 5, 2019 at 9:33 AM Julen Landa Alustiza < jlanda@fedoraproject.org> wrote:
19/12/4 15:34(e)an, Kamil Paral igorleak idatzi zuen:
On Tue, Dec 3, 2019 at 10:22 PM Tim Flink tflink@redhat.com wrote:
Has anyone poked at scoping out the work required for either the bot or the enhancements we would need for pagure?
Lukáš Brabec looked into it, and creating a new Pagure ticket is a matter of a few lines of code (already written). Extending BBA with a "discussion link" in each blocker row should be trivial as well. I think that's all we need for trying it out.
The blocker bot watching and updating the issue should be fairly easy as well, I hope, but as described above, we've hit an issue where Pagure API doesn't allow comment editing. Lukáš implemented a workaround that adds a new metadata field on the right side of each Pagure ticket showing the count summary. It's barebones, but at least something. We've haven't yet looked into writing the listener code, that would listen for Pagure events on the message bus and react to them. We also haven't looked at Bugzilla updater as described in the proposed workflow (that's really a cherry on top, we don't need it right away).
Implementing an api endpoint to edit comment is easy, open an RFE on pagure's issue tracker if you need it and I'll take it
I'm not going to say no to that! :-) Here it is: https://pagure.io/pagure/issue/4683 Thanks a lot.
The RFE commit have been merged on master: https://pagure.io/pagure/c/31280551ec2fdf282152abae2ff9d44f20faaf92?branch=m...
I'll notify here once we release 5.9 and upgrade pagure.io with it.
19/12/5 10:06(e)an, Kamil Paral igorleak idatzi zuen:
On Thu, Dec 5, 2019 at 9:33 AM Julen Landa Alustiza <jlanda@fedoraproject.org mailto:jlanda@fedoraproject.org> wrote:
19/12/4 15:34(e)an, Kamil Paral igorleak idatzi zuen:On Tue, Dec 3, 2019 at 10:22 PM Tim Flink <tflink@redhat.com <mailto:tflink@redhat.com>> wrote: Has anyone poked at scoping out the work required for either the bot or the enhancements we would need for pagure? Lukáš Brabec looked into it, and creating a new Pagure ticket is a matter of a few lines of code (already written). Extending BBA with a "discussion link" in each blocker row should be trivial as well. I think that's all we need for trying it out. The blocker bot watching and updating the issue should be fairly easy as well, I hope, but as described above, we've hit an issue where Pagure API doesn't allow comment editing. Lukáš implemented a workaround that adds a new metadata field on the right side of each Pagure ticket showing the count summary. It's barebones, but at least something. We've haven't yet looked into writing the listener code, that would listen for Pagure events on the message bus and react to them. We also haven't looked at Bugzilla updater as described in the proposed workflow (that's really a cherry on top, we don't need it right away).Implementing an api endpoint to edit comment is easy, open an RFE on pagure's issue tracker if you need it and I'll take itI'm not going to say no to that! :-) Here it is: https://pagure.io/pagure/issue/4683 Thanks a lot.
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
On Mon, Jan 13, 2020 at 10:22 AM Julen Landa Alustiza < jlanda@fedoraproject.org> wrote:
The RFE commit have been merged on master:
https://pagure.io/pagure/c/31280551ec2fdf282152abae2ff9d44f20faaf92?branch=m...
I'll notify here once we release 5.9 and upgrade pagure.io with it.
Awesome, thanks a lot!
19/11/28 17:13(e)an, Adam Williamson igorleak idatzi zuen:
On Thu, 2019-11-28 at 13:29 +0100, Kamil Paral wrote:
What are your thoughts? Does the proposal sound reasonable? Would you change something? Would you use a different backend system? Have I forgotten something important? Feedback welcome.
So overall I agree with a lot of what you wrote. It does cause me to wonder if writing some kind of plugin/extension for Pagure, or just writing the functionality into the blockerbugs app, might possibly be *less* work than writing and maintaining the 'blocker bot', however. My other concern would be that if we use something other than the blockerbugs app or Bugzilla, we now have *three* different systems involved in the blocker review process - Bugzilla, the blockerbugs app, and the discussion system. That seems like a lot of moving parts to keep synchronized.
pagure supports third-party themes|plugins , src.fp.o uses both for custom features (anitya integration for example), we could do something similar on pagure.io, where we are already using a downstream theme but not plugin
Thanks for working on this!
On Thu, Nov 28, 2019 at 1:31 PM Kamil Paral kparal@redhat.com wrote:
- Once some kind of understanding of the issue is formed, a privileged
member (e.g. a member of @fedora-qa FAS group) can start the vote by including a command in his comment, e.g. "START VOTE" on a separate line. (Note that this is just a proposal. We can easily let anyone start the vote, or we can allow voting right from the 9. If new important information appear or the vote needs to be repeated for any other reason, a privileged member can restart the vote e.g. by issuing RESTART VOTE command. (We can have more such administrative commands for any purpose we need, same as we have them with IRC bots).
It's probably just a small detail, why would we need (RE)START VOTE commands? It seems we can vote whatever/whenever we want, bot would just count the last VOTE +/- 1, 0 comment (or rather command) from each participant.
On Thu, Nov 28, 2019 at 11:29 PM Frantisek Zatloukal fzatlouk@redhat.com wrote:
On Thu, Nov 28, 2019 at 1:31 PM Kamil Paral kparal@redhat.com wrote:
- Once some kind of understanding of the issue is formed, a privileged
member (e.g. a member of @fedora-qa FAS group) can start the vote by including a command in his comment, e.g. "START VOTE" on a separate line. (Note that this is just a proposal. We can easily let anyone start the vote, or we can allow voting right from the 9. If new important information appear or the vote needs to be repeated for any other reason, a privileged member can restart the vote e.g. by issuing RESTART VOTE command. (We can have more such administrative commands for any purpose we need, same as we have them with IRC bots).
It's probably just a small detail, why would we need (RE)START VOTE commands? It seems we can vote whatever/whenever we want, bot would just count the last VOTE +/- 1, 0 comment (or rather command) from each participant.
That's a good question. The START VOTE is optional and really up to us if we want to use it. I included it because it seemed to me to mirror what we do during IRC meetings - first we discuss the issue and come to an understanding of the problem, and then people vote (usually). But it's true that for clear-cut issues, people might want to vote immediately. And waiting for a privileged person to start voting every time could prove annoying. So after some consideration, I agree that it might be better to allow people voting right from the beginning.
The RESTART VOTE is useful, though. Imagine a situation where several people voted but then the developer added a comment that the bug might in some cases wipe all user data (or the whole disk). There might be such a bump in severity of the issue, that a QA person might force the vote to be restarted. This makes sure that old votes won't be counted in by accident, and if you look at the vote summary, you can be certain that all the votes displayed were cast by people who were already aware of the increased gravity of the issue.
One more note: During IRC meetings the chair usually posts "proposed #agreed statement" and people ack it or nack it, then it changes to "#agreed statement". I think that won't be possible with ticket meetings, because that would require one additional roundtrip of votes (which might take up to 1 day due to timezones). So I think the workflow will be for people to vote +1/0/-1, and then the selected QA person will simply summarize it into AGREED <RESOLUTION>, and we will trust he/she will do the right thing and also formulate the justification well (of course, if people disagree, the ticket can always be reopened). For contentious issues where there are numerous +1's and -1's, we'll need to deal with it as usual - either punt it, or ask the minority if they're fine with using the majority decision (and then use the AGREED command).
Thanks for the proposal, Kamil!
On Thu, Nov 28, 2019 at 1:31 PM Kamil Paral kparal@redhat.com wrote:
Blocker Bugs App (DIY solution)
This sounds like a lot of work on features that are already implemented elsewhere.
Mailing lists
For the reasons you posted, I don't think this is a good idea. And you didn't even mention top posting. That would be a mess.
Bugzilla
Pagure
Both could work, but navigation in pagure is far better in my opinion.
- Blocker Bugs App (BBA) detects the new blocker and creates a new ticket in Pagure in the "fedora-blockers" project, then updates the bug to link to this ticket (a new comment, a Links entry). It shows both the bug and the discussion ticket in its UI.
What about fedora-blockers group with FXX-{{Beta, Final}-Blockers,{Beta, Final}-FEs} projects. Issues would be filed against appropriate project in that group. I like the added hierarchy and that F32-Final-Blockers would have only relevant issues. Or does this sound too complicated?
- Once some kind of understanding of the issue is formed, a privileged member (e.g. a member of @fedora-qa FAS group) can start the vote by including a command in his comment, e.g. "START VOTE" on a separate line. (Note that this is just a proposal. We can easily let anyone start the vote, or we can allow voting right from the beginning without an explicit start.)
I'd allow voting right from the beginning, OTOH I think we would often see a lot of votes, then proper discussion and RESTART command after that. We can try both (explicit START and right from the beginning) and see which is better.
Note that we don't need the blocker bot to be fully or even partially implemented, just to try this out. We can easily do the bot steps manually in the beginning and decide whether we like this process, before we start working on the automation. We can pretty much start testing this right now, if we want.
+1 on this, transitional period in which we would do it manually so we can start testing the new process without spending much time developing the bot is a great idea.
On Fri, Nov 29, 2019 at 9:46 AM Lukas Brabec lbrabec@redhat.com wrote:
- Blocker Bugs App (BBA) detects the new blocker and creates a new
ticket in Pagure in the "fedora-blockers" project, then updates the bug to link to this ticket (a new comment, a Links entry). It shows both the bug and the discussion ticket in its UI.
What about fedora-blockers group with FXX-{{Beta, Final}-Blockers,{Beta, Final}-FEs} projects. Issues would be filed against appropriate project in that group. I like the added hierarchy and that F32-Final-Blockers would have only relevant issues. Or does this sound too complicated?
The proposed milestone is often not the milestone we approve it for. For example it might be proposed for Beta, but we agree that's a Final blocker. How would you handle ticket transfers between projects?
Also, often a bug is a Final blocker and a Beta FE. How would you handle a ticket that needs to be part of several projects?
Additionally, people would need to subscribe to all projects if they wanted to receive notifications for new proposed blockers, and they would need to repeat it every time a new release appears (because we'd create new projects in that group every time a new Fedora is branched).
I think all of that is much easier solved by tags. Tags for F31, F32, etc, tags for blocker/FE, tags for proposed/agreed. The only concern is not getting out of sync with information in Bugzilla and BBA. Perhaps we don't need tags at all, and can instead improve BBA to show us historical bugs/tickets. Or perhaps the sync issues are not such a problem. Shrug.
- Once some kind of understanding of the issue is formed, a privileged
member (e.g. a member of @fedora-qa FAS group) can start the vote by including a command in his comment, e.g. "START VOTE" on a separate line. (Note that this is just a proposal. We can easily let anyone start the vote, or we can allow voting right from the beginning without an explicit start.)
I'd allow voting right from the beginning, OTOH I think we would often see a lot of votes, then proper discussion and RESTART command after that. We can try both (explicit START and right from the beginning) and see which is better.
Yeah, already replied to Frantisek, voting from the beginning seems like a better option.
On Thu, Nov 28, 2019 at 5:30 AM Kamil Paral kparal@redhat.com wrote:
Pagure
This is very similar to the Bugzilla description. For each proposed blocker, we (auto-)create a ticket in a "fedora-blockers" project, for the purpose of a blocker discussion. We interlink the bug and the ticket.
What about kanban? Is there kanban integration in pagure?
Or what about a taiga method, e.g. using teams.fedoraproject.org?
I'm not opposed to dropping the live IRC blocker review entirely; but even if it were true that they're cut down to a max 1 hour, or even an informal day long working session on #fedora-qa, that would still be a huge win. Probably a small subset of complex blocker bugs may benefit from a live IRC meeting.
On Sat, Nov 30, 2019 at 10:40 PM Chris Murphy lists@colorremedies.com wrote:
On Thu, Nov 28, 2019 at 5:30 AM Kamil Paral kparal@redhat.com wrote:
Pagure
This is very similar to the Bugzilla description. For each proposed
blocker, we (auto-)create a ticket in a "fedora-blockers" project, for the purpose of a blocker discussion. We interlink the bug and the ticket.
What about kanban? Is there kanban integration in pagure?
Or what about a taiga method, e.g. using teams.fedoraproject.org?
I'm not familiar with kanban/taiga. What is the benefit over using standard (Pagure) tickets?
I'm not opposed to dropping the live IRC blocker review entirely; but even if it were true that they're cut down to a max 1 hour, or even an informal day long working session on #fedora-qa, that would still be a huge win. Probably a small subset of complex blocker bugs may benefit from a live IRC meeting.
It should not be that hard to ping people participating in a particular blocker ticket, and discuss some complex details over IRC, then write some summary/findings to the ticket. That's a completely valid approach. Or would you like to see something more formalized? What would the process look like?
I want to speak up in defense of synchronous meetings. I acknowledge that the timing is rather convenient for me personally (it's 12–3pm my time), which makes it easier for me to see the upsides.
The big benefit is the high bandwidth discussion. I have been in plenty of meetings where someone initially feels one way and has their mind changed over the course of the discussion. This is still possible with asynchronous communication, but it's much more difficult. We generally end up at a pretty good consensus on blocker status, even if it sometimes take two (or three!) meetings. I foresee more +4/0/-3 type votes with an asynchronous method.
Relatedly, any asynchronous voting mechanism must make it very easy to know when someone has updated their vote. This is addressed some in the thread, but any async method that requires coremodule (the forever secretary!) to read through each comment to make sure no one changed their mind is inflicting unnecessary pain. IRC has the same problem, but because the time scale is shorter, it makes it easier to see IMO.
Synchronous meetings give the voting period a defined end and makes the process quick. If we do it asynchronously, there has to be some deadline that allows people time to vote. Currently, the process is (IIRC) if three people +1 in the BZ, it's counted as accepted. This means a bug could potentially be accepted within minutes, denying those who disagree the opportunity to raise their objections. Of course, we don't want to set the deadline too long because then we have a week of "yeah, it looks like this is a blocker, but also we can't apply the label yet".
None of the above should be taken as me saying that the current process is perfect. Nor should it be an objection to the idea of an asynchronous process in principle. I just want to make sure we're intentional about what we're giving up in order to get the benefits of an async process. For what it's worth, voting support in Pagure (or another similar system) would be helpful for many teams within Fedora, including Council, FESCo, and Mindshare who routinely vote on things in tickets.
Kamil,
Thanks for working on this. I am glad we are considering something different here, as the current process is not perfect.
Personally, of the suggested options, I think that using Pagure is my preferred option. I think of the available options, it allows for the easiest collaboration between folks who are contemplating and voting on the bugs.
However, I do resonate with some of what Ben said, specifically regarding losing the "live" setting when we vote. The ability to hash things out over IRC real-time is pretty valuable, and to use a different system where time between comments could become large, I wonder if still having a meeting every Monday would be advantageous. Something to go over the votes that have been made on the Pagure page (or whatever system it ends up being). The problem with that is keeping it from turning right back into the old "blocker-review-meeting". We could possibly institute a 30(?) minute time limit, where we go over the bugs to ensure we understand them and then accept the vote right there. That way we have the opportunity to ask questions/clarifications that might be tedious and take a lot of time over a forum-style messaging app. Again, there would have to be some kind of watchdog to prevent us from turning these meetings into what we have now. All this said, should this IRC meeting idea come to fruition, I would be more than happy to coordinate and run these, should anyone else not want to (adamw! :P)
Geoff Marr IRC: coremodule
On Mon, Dec 2, 2019 at 8:02 AM Ben Cotton bcotton@redhat.com wrote:
I want to speak up in defense of synchronous meetings. I acknowledge that the timing is rather convenient for me personally (it's 12–3pm my time), which makes it easier for me to see the upsides.
The big benefit is the high bandwidth discussion. I have been in plenty of meetings where someone initially feels one way and has their mind changed over the course of the discussion. This is still possible with asynchronous communication, but it's much more difficult. We generally end up at a pretty good consensus on blocker status, even if it sometimes take two (or three!) meetings. I foresee more +4/0/-3 type votes with an asynchronous method.
Relatedly, any asynchronous voting mechanism must make it very easy to know when someone has updated their vote. This is addressed some in the thread, but any async method that requires coremodule (the forever secretary!) to read through each comment to make sure no one changed their mind is inflicting unnecessary pain. IRC has the same problem, but because the time scale is shorter, it makes it easier to see IMO.
Synchronous meetings give the voting period a defined end and makes the process quick. If we do it asynchronously, there has to be some deadline that allows people time to vote. Currently, the process is (IIRC) if three people +1 in the BZ, it's counted as accepted. This means a bug could potentially be accepted within minutes, denying those who disagree the opportunity to raise their objections. Of course, we don't want to set the deadline too long because then we have a week of "yeah, it looks like this is a blocker, but also we can't apply the label yet".
None of the above should be taken as me saying that the current process is perfect. Nor should it be an objection to the idea of an asynchronous process in principle. I just want to make sure we're intentional about what we're giving up in order to get the benefits of an async process. For what it's worth, voting support in Pagure (or another similar system) would be helpful for many teams within Fedora, including Council, FESCo, and Mindshare who routinely vote on things in tickets.
-- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
On Mon, Dec 2, 2019 at 11:47 PM Geoffrey Marr gmarr@redhat.com wrote:
Kamil,
Thanks for working on this. I am glad we are considering something different here, as the current process is not perfect.
Personally, of the suggested options, I think that using Pagure is my preferred option. I think of the available options, it allows for the easiest collaboration between folks who are contemplating and voting on the bugs.
However, I do resonate with some of what Ben said, specifically regarding losing the "live" setting when we vote. The ability to hash things out over IRC real-time is pretty valuable, and to use a different system where time between comments could become large, I wonder if still having a meeting every Monday would be advantageous. Something to go over the votes that have been made on the Pagure page (or whatever system it ends up being). The problem with that is keeping it from turning right back into the old "blocker-review-meeting". We could possibly institute a 30(?) minute time limit, where we go over the bugs to ensure we understand them and then accept the vote right there. That way we have the opportunity to ask questions/clarifications that might be tedious and take a lot of time over a forum-style messaging app. Again, there would have to be some kind of watchdog to prevent us from turning these meetings into what we have now. All this said, should this IRC meeting idea come to fruition, I would be more than happy to coordinate and run these, should anyone else not want to (adamw! :P)
Hey Geoff. I'm afraid time limits wouldn't work well for us here. I wouldn't want to poorly decide a blocker status just because we were running out of time. In case of our IRC meeting, we also don't accept/reject something just because we're out of time. If that happens, we postpone it until the next meeting, or if we don't have the luxury of one more week, we schedule an exceptional meeting the very next day.
I wonder if we could merge the live+async approaches to get the best of both. What if we handled simple/standard issues fully in tickets, and for complex issues which are difficult to understand and debug, or which have some heated discussions, we'd schedule an IRC meeting and resolve them there? We can either schedule the meetings ad-hoc, or we can use a special tag in Pagure, and the regular Monday blocker review meeting would only occur if there was at least a single open ticket marked with this tag. Does that sound better in your opinion?
Kamil,
I wasn't intending the time we would get together each week for voting, instead for time spent discussing and clarifying any discrepancies we may have with the bugs. By the time we would have this 30 or so minute get-together, our votes would already be cast in Pagure. We would make sure we were all on the same page, and then use the meeting time to mark the bugs as "Accepted ________".
I like the idea of somehow merging the live meetings with the asynchronous process. Personally, I think the meeting I'm describing should be solely purposed as the "Acceptance/Denial Meeting", where the point is to review the votes in Pagure, and then manually accept/deny the position. That way, we spend a small portion of time on each bug, not voting, but reviewing the votes that should have been already cast. And, if any questions arise regarding a particular bug, we have a platform in which to have a real-time conversation with our peers.
Geoff Marr IRC: coremodule
On Tue, Dec 3, 2019 at 5:38 AM Kamil Paral kparal@redhat.com wrote:
On Mon, Dec 2, 2019 at 11:47 PM Geoffrey Marr gmarr@redhat.com wrote:
Kamil,
Thanks for working on this. I am glad we are considering something different here, as the current process is not perfect.
Personally, of the suggested options, I think that using Pagure is my preferred option. I think of the available options, it allows for the easiest collaboration between folks who are contemplating and voting on the bugs.
However, I do resonate with some of what Ben said, specifically regarding losing the "live" setting when we vote. The ability to hash things out over IRC real-time is pretty valuable, and to use a different system where time between comments could become large, I wonder if still having a meeting every Monday would be advantageous. Something to go over the votes that have been made on the Pagure page (or whatever system it ends up being). The problem with that is keeping it from turning right back into the old "blocker-review-meeting". We could possibly institute a 30(?) minute time limit, where we go over the bugs to ensure we understand them and then accept the vote right there. That way we have the opportunity to ask questions/clarifications that might be tedious and take a lot of time over a forum-style messaging app. Again, there would have to be some kind of watchdog to prevent us from turning these meetings into what we have now. All this said, should this IRC meeting idea come to fruition, I would be more than happy to coordinate and run these, should anyone else not want to (adamw! :P)
Hey Geoff. I'm afraid time limits wouldn't work well for us here. I wouldn't want to poorly decide a blocker status just because we were running out of time. In case of our IRC meeting, we also don't accept/reject something just because we're out of time. If that happens, we postpone it until the next meeting, or if we don't have the luxury of one more week, we schedule an exceptional meeting the very next day.
I wonder if we could merge the live+async approaches to get the best of both. What if we handled simple/standard issues fully in tickets, and for complex issues which are difficult to understand and debug, or which have some heated discussions, we'd schedule an IRC meeting and resolve them there? We can either schedule the meetings ad-hoc, or we can use a special tag in Pagure, and the regular Monday blocker review meeting would only occur if there was at least a single open ticket marked with this tag. Does that sound better in your opinion?
test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
On Tue, Dec 3, 2019 at 5:15 PM Geoffrey Marr gmarr@redhat.com wrote:
Kamil,
I wasn't intending the time we would get together each week for voting, instead for time spent discussing and clarifying any discrepancies we may have with the bugs. By the time we would have this 30 or so minute get-together, our votes would already be cast in Pagure. We would make sure we were all on the same page, and then use the meeting time to mark the bugs as "Accepted ________".
I like the idea of somehow merging the live meetings with the asynchronous process. Personally, I think the meeting I'm describing should be solely purposed as the "Acceptance/Denial Meeting", where the point is to review the votes in Pagure, and then manually accept/deny the position. That way, we spend a small portion of time on each bug, not voting, but reviewing the votes that should have been already cast. And, if any questions arise regarding a particular bug, we have a platform in which to have a real-time conversation with our peers.
OK. So what do you think about this? We'd use both the ticketing system and also have our regular Monday meetings. Simple or non-controversial tickets would get resolved through the ticketing system throughout the week. Complex tickets or tickets with balanced number of +1 and -1 votes would get further discussed and decided in our Monday's meeting, the same way we always do it. This keeps the advantage of reducing the meeting length by weeding out simple tickets in async manner. It also keeps the advantage of pinging/CCing developers and letting them respond/vote async. And not just developers, anyone who can't make the meeting can still provide their feedback in the ticket. And finally, this allows us to test the async approach without making a full switch.
On Mon, Dec 2, 2019 at 4:02 PM Ben Cotton bcotton@redhat.com wrote:
I want to speak up in defense of synchronous meetings. I acknowledge that the timing is rather convenient for me personally (it's 12–3pm my time), which makes it easier for me to see the upsides.
The big benefit is the high bandwidth discussion. I have been in plenty of meetings where someone initially feels one way and has their mind changed over the course of the discussion. This is still possible with asynchronous communication, but it's much more difficult.
Yes, that is true. And perhaps if I had the meetings in the afternoon as you do, instead of late evening when I'd rather be with my family, I wouldn't mind them either :-) Alas, it's not the case, and I'd certainly like to test using tickets. I think having an experiment here is very valuable, we might realize a lot of up&downs as we go, which we don't think about atm.
For example, I believe that one of the most important advantages of async discussion is the ability to better gather developer feedback. Currently, it's very rare that we can contact the developer and gets some responses during our live meeting. But this is exactly what only experience can tell, whether the improvement will be substantial, or whether developers will ignore our tickets and we will be in the same state as we were.
We generally end up at a pretty good consensus on blocker status, even if it sometimes take two (or three!) meetings. I foresee more +4/0/-3 type votes with an asynchronous method.
Relatedly, any asynchronous voting mechanism must make it very easy to know when someone has updated their vote. This is addressed some in the thread, but any async method that requires coremodule (the forever secretary!) to read through each comment to make sure no one changed their mind is inflicting unnecessary pain. IRC has the same problem, but because the time scale is shorter, it makes it easier to see IMO.
Well, the idea was that a bot keeps count of all the votes and updates a summary. That will not be available from day 1, most probably. But once implemented, that should address this concern, correct?
Synchronous meetings give the voting period a defined end and makes the process quick. If we do it asynchronously, there has to be some deadline that allows people time to vote. Currently, the process is (IIRC) if three people +1 in the BZ, it's counted as accepted. This means a bug could potentially be accepted within minutes, denying those who disagree the opportunity to raise their objections. Of course, we don't want to set the deadline too long because then we have a week of "yeah, it looks like this is a blocker, but also we can't apply the label yet".
The current in-Bugzilla process is used just when we need a very fast decision, and I agree it's not ideal. The same approach can be used in tickets (we can say "there are only 3 hours to collect votes, please respond asap"). But we can also be much more flexible. If the QA person in charge (or however you want to name Adam:-)) decides that most of the usual suspects have already voted, he can close the vote. Or he can wait some more, call for more votes, etc. Unlike "in-BZ discussions", these tickets should be much easier to read and manage, you can easily see how old the ticket is, and decide how long you want to wait or if it was enough. And we'll probably have some recommended practices, e.g. unless we're in a rush, wait 2-3 days for votes.
None of the above should be taken as me saying that the current process is perfect. Nor should it be an objection to the idea of an asynchronous process in principle. I just want to make sure we're intentional about what we're giving up in order to get the benefits of an async process.
Thanks for your feedback!
For what it's worth, voting support in Pagure (or another similar system) would be helpful for many teams within Fedora, including Council, FESCo, and Mindshare who routinely vote on things in tickets.
This is interesting. I believed that this is just us trying to bend Pagure a little to something it wasn't designed for. But if other groups could benefit from this functionality as well, perhaps we could look at how difficult it would be to implement voting natively in Pagure (and I might hate myself in the future for saying this out loud:)).
On 12/2/19 07:03, Kamil Paral wrote:
I'm not familiar with kanban/taiga. What is the benefit over using standard (Pagure) tickets?
The kanban/taiga is what they use at Fedora Magazine. A Writer starts by creating an Issue proposing a new article for the magazine. There is a back and forth commentary and if the proposal gets two +1 votes then the proposal is transfered to a job ticket in the kanban by one of the Editors.
Then the writer picks up the job ticket and assigns it to them selves if the Editor didn't already assign it to them. Editors can also create job tickest for articles they want and a Writer who thinks they can write the article can pick it up and write it. Once a writer is assigned to a job ticket they write the article according to their free time or according a requested time.
When the draft is complete, the writer puts the job ticket into the review queue and an editor will review it. Then the job ticket can be put back in the in process queue if it needs a lot of work or it can be placed in the To Edit queue for other work like having a feature image added to it and it can wait there for scheduling.
After the image is added and the schedule is decided the job ticket is placed in the Queued queue.
All through the process from when the writer starts writing the article to publishing, the actual article is linked to the job ticket.
Seems to work good.
Have a Great Day!
Pat (tablepc)
On Mon, Dec 2, 2019 at 4:19 PM pmkellly@frontier.com pmkellly@frontier.com wrote:
On 12/2/19 07:03, Kamil Paral wrote:
I'm not familiar with kanban/taiga. What is the benefit over using
standard
(Pagure) tickets?
The kanban/taiga is what they use at Fedora Magazine. A Writer starts by creating an Issue proposing a new article for the magazine. There is a back and forth commentary and if the proposal gets two +1 votes then the proposal is transfered to a job ticket in the kanban by one of the Editors.
Then the writer picks up the job ticket and assigns it to them selves if the Editor didn't already assign it to them. Editors can also create job tickest for articles they want and a Writer who thinks they can write the article can pick it up and write it. Once a writer is assigned to a job ticket they write the article according to their free time or according a requested time.
When the draft is complete, the writer puts the job ticket into the review queue and an editor will review it. Then the job ticket can be put back in the in process queue if it needs a lot of work or it can be placed in the To Edit queue for other work like having a feature image added to it and it can wait there for scheduling.
After the image is added and the schedule is decided the job ticket is placed in the Queued queue.
All through the process from when the writer starts writing the article to publishing, the actual article is linked to the job ticket.
Thanks, Pat, for the description. Kanban seems to be just a workflow visualization that could be done with custom ticket status or tags the same way (but wouldn't look as pretty as a native kanban UI). Either way, I think this doesn't really apply to blocker discussion tickets. These tickets have a dead simple workflow open -> discuss&vote (separating that would waste a lot of time) -> closed.
On Thu, Nov 28, 2019 at 01:29:40PM +0100, Kamil Paral wrote:
We've talked about replacing blocker bug review meetings with something else for a long time. The meeting has an upside of a higher communication bandwidth, but also a downside of requiring participants to be available at the same time (throughout the world), often being extremely long, and being scheduled just once per week (because of mentioned downsides). I've put some thoughts into making the process asynchronous, and you can see all the options I considered described below. I close up with a proposal for the best choice I found.
...snip...
I do like the idea of making things more async, although we have a poor history of following through on this in other groups. :(
Note that we don't need the blocker bot to be fully or even partially implemented, just to try this out. We can easily do the bot steps manually in the beginning and decide whether we like this process, before we start working on the automation. We can pretty much start testing this right now, if we want.
What are your thoughts? Does the proposal sound reasonable? Would you change something? Would you use a different backend system? Have I forgotten something important? Feedback welcome.
I pretty much agree with all the points you made on the various systems. That said, I think this is kinda a bad time to be doing this change. There's a lot of... (grumblings? rumors? idle converstations?) about the various bug/issue/source/forges we are currently using. I don't know if this means we will move to something else, try and consolidate more on one or what, it's all very vuage at this point. So, IMHO, it's kind of a bad time to be moving process to one or another things that might require you to move again later (be that bugzilla or pagure or whatever). Of course if the process is simple (which I think it should be) it would translate pretty well to whatever system it's used on.
As a side note I will note that bugzilla does allow voting also, we just asked them to disable it on the Fedora component. Althought I am not surre it works in a way thats useful for this workflow: Every user gets X vote points per Y time and can distribute them accross bugs as they like. I guess the idea is that then you could see the bugs users REALLY want fixed.
I guess I would be most in favor of something leveraging bugzilla, since thats where our bugs are (at least now).
kevin
On Tue, Dec 3, 2019 at 8:29 PM Kevin Fenzi kevin@scrye.com wrote:
I pretty much agree with all the points you made on the various systems. That said, I think this is kinda a bad time to be doing this change. There's a lot of... (grumblings? rumors? idle converstations?) about the various bug/issue/source/forges we are currently using. I don't know if this means we will move to something else, try and consolidate more on one or what, it's all very vuage at this point. So, IMHO, it's kind of a bad time to be moving process to one or another things that might require you to move again later (be that bugzilla or pagure or whatever). Of course if the process is simple (which I think it should be) it would translate pretty well to whatever system it's used on.
Thanks for the info. That means we probably shouldn't spend too much time on all the possible bells and whistles, and instead create something simple that a) will help us determine whether we like the async approach or not, and b) can be easily moved to a different ticketing system/source forge. In other words, stick to the basics. Or wait, of course, after F32 release cycle is over, and then perhaps the situation will be clearer. But I'd rather try it now, a simple experiment shouldn't hurt anybody and it's trivial to get back to IRC meetings any time :)
As a side note I will note that bugzilla does allow voting also, we just asked them to disable it on the Fedora component. Althought I am not surre it works in a way thats useful for this workflow: Every user gets X vote points per Y time and can distribute them accross bugs as they like. I guess the idea is that then you could see the bugs users REALLY want fixed.
I guess I would be most in favor of something leveraging bugzilla, since thats where our bugs are (at least now).
Pagure had two major benefits for me compared to Bugzilla: 1. Comments can be edited. I wanted to use comment 0 for vote summary. Unfortunately, Lukáš Brabec looked into this and current Pagure API doesn't allow comment editing, you can do it only using your browser. So Pagure API would have to be extended :-/ 2. Pagure can add tags to tickets. But after discussion, I agreed with Adam that it might not be wise to mirror all the blocker metadata (accepted/rejected, blocker/FE) in another system, at least initially.
So none of those two is a benefit in the near future. Pagure still has a more comfortable UI, but it's more even now with Bugzilla.
-
Adam Williamson -
Ben Cotton -
Chris Murphy -
Frantisek Zatloukal -
Geoffrey Marr -
Julen Landa Alustiza -
Kamil Paral -
Kevin Fenzi -
Lukas Brabec -
pmkellly＠frontier.com -
Tim Flink