Packaging the AWS SSM Agent

[Major Hayden]

Hey there,

I've been working on packaging the AWS SSM Agent[0] in Fedora on and off for a few months now and I finally got a free moment to work on it again. However, there are a few problems that have me scratching my head a little.

Just to level set, AWS Systems Manager[1] is an offering that allows AWS customers to manage their instances outside the instance itself. We all know (and love) cloud-init, but it really only handles the first boot provisioning of the instance. The SSM agent has similarities to Azure's WALinuxAgent as it allows for management *after* the first boot.

(For reference, WALinuxAgent is already packaged in Fedora and automatically enabled when Fedora lands in an Azure instance.)

The main issue is around dependencies:

----------
# No commits since Mar 2021 💀
'golang(github.com/Workiva/go-datastructures)'

# Should be covered by the existing aws-sdk-go package 🤔
'golang(github.com/aws/aws-sdk-go/service/backupstorage)'
'golang(github.com/aws/aws-sdk-go/service/privatenetworks)'
'golang(github.com/aws/aws-sdk-go/service/ssm/ssmiface/mocks)'
'golang(github.com/aws/aws-sdk-go/service/ssmmds)'
'golang(github.com/aws/aws-sdk-go/service/ssmmds/ssmmdsiface)'
'golang(github.com/aws/aws-sdk-go/service/ssmmds/ssmmdsiface/mocks)'

# No commits since Jan 2017 💀
'golang(github.com/carlescere/scheduler)'

# No commits since Jan 2017 💀
'golang(github.com/cihub/seelog)'

# No commits since Sep 2018 💀
'golang(github.com/digitalocean/go-smbios/smbios)'

# No commits since Sep 2016 💀
'golang(github.com/pborman/ansi)'

# Last commit Feb 2023 👍
'golang(github.com/xtaci/smux)'
----------

Some of these have not been updated in quite some time and many of them have several unresolved bugs that are years old. Although I'd like to get the SSM Agent packaged in Fedora, I don't like the idea of packaging quite a few packages which aren't being actively maintained.

One potential option is to work with upstream (AWS) to change these dependencies out for actively maintained ones instead, but that likely requires significant development work. 🥵

Another option might be to vendor these dependencies in the main SSM package, but that still means we're building against unmaintained code and it likely violates some Fedora policies. 😱

How should I proceed? Thanks for reading this far?

[0] https://github.com/aws/amazon-ssm-agent
[1] https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems...

--
Major Hayden

[Mikel Olasagasti]

Hi Major,

Hau idatzi du Major Hayden ([e-mail]) erabiltzaileak (2023 mai.
8(a), al. (22:51)):
> Hey there,
>
> I've been working on packaging the AWS SSM Agent[0] in Fedora on and off for a few months now and I finally got a free moment to work on it again. However, there are a few problems that have me scratching my head a little.
>
> Just to level set, AWS Systems Manager[1] is an offering that allows AWS customers to manage their instances outside the instance itself. We all know (and love) cloud-init, but it really only handles the first boot provisioning of the instance. The SSM agent has similarities to Azure's WALinuxAgent as it allows for management *after* the first boot.
>
> (For reference, WALinuxAgent is already packaged in Fedora and automatically enabled when Fedora lands in an Azure instance.)
>
> The main issue is around dependencies:
>
> # No commits since Mar 2021 💀
> 'golang(github.com/Workiva/go-datastructures)'
>
> # Should be covered by the existing aws-sdk-go package 🤔
> 'golang(github.com/aws/aws-sdk-go/service/backupstorage)'
> 'golang(github.com/aws/aws-sdk-go/service/privatenetworks)'
> 'golang(github.com/aws/aws-sdk-go/service/ssm/ssmiface/mocks)'
> 'golang(github.com/aws/aws-sdk-go/service/ssmmds)'
> 'golang(github.com/aws/aws-sdk-go/service/ssmmds/ssmmdsiface)'
> 'golang(github.com/aws/aws-sdk-go/service/ssmmds/ssmmdsiface/mocks)'
>
> # No commits since Jan 2017 💀
> 'golang(github.com/carlescere/scheduler)'
>
> # No commits since Jan 2017 💀
> 'golang(github.com/cihub/seelog)'
>
> # No commits since Sep 2018 💀
> 'golang(github.com/digitalocean/go-smbios/smbios)'
>
> # No commits since Sep 2016 💀
> 'golang(github.com/pborman/ansi)'
>
> # Last commit Feb 2023 👍
> 'golang(github.com/xtaci/smux)'
>
> Some of these have not been updated in quite some time and many of them have several unresolved bugs that are years old. Although I'd like to get the SSM Agent packaged in Fedora, I don't like the idea of packaging quite a few packages which aren't being actively maintained.
>
> One potential option is to work with upstream (AWS) to change these dependencies out for actively maintained ones instead, but that likely requires significant development work. 🥵
>
> Another option might be to vendor these dependencies in the main SSM package, but that still means we're building against unmaintained code and it likely violates some Fedora policies. 😱
>
> How should I proceed?

My option would be to package those "dead" deps.

The impact for Fedora is very limited, as only SSM Agent require those
packages and, hopefully, no other packages will require them. If in
the future builds start to fail, upstream should be also affected
sooner or later, so upstream should be interested on fixing.

IMO the major (no pun intended) problem would be for you, as it can be
challenging to keep the dep stack up to date.

Kind regards,
Mikel

[Smith, Stewart]

> On May 9, 2023, at 1:53 PM, Mikel Olasagasti  wrote:
>
> Hi Major,
>
> Hau idatzi du Major Hayden ([e-mail]) erabiltzaileak (2023 mai.
> 8(a), al. (22:51)):
> > One potential option is to work with upstream (AWS) to change these dependencies out for actively maintained ones instead, but that likely requires significant development work. 🥵

I’m poking the SSM team internally on this to see if there’s any plans of getting away from any of these (and to force the evaluation of if there *should* be any plans or not).
> > Another option might be to vendor these dependencies in the main SSM package, but that still means we're building against unmaintained code and it likely violates some Fedora policies. 😱

We vendor the dependencies for SSM in the SSM package we ship in Amazon Linux for two reasons:
1. it was always done that way (not a good reason)
2. avoids exposing any of these deps to a broader audience than “just the scope in which the SSM agent uses them” and thus alleviates a pile of issues regarding moving them forward major versions or not.

Seeing as it’s net new to Fedora, (1) doesn’t apply, and (2) is less of an issue because of the shorter life cycle of Fedora.

[Smith, Stewart]

On May 11, 2023, at 7:14 AM, Smith, Stewart  wrote:
> > On May 9, 2023, at 1:53 PM, Mikel Olasagasti  wrote:
> >
> > Hi Major,
> >
> > Hau idatzi du Major Hayden ([e-mail]) erabiltzaileak (2023 mai.
> > 8(a), al. (22:51)):
> > > One potential option is to work with upstream (AWS) to change these dependencies out for actively maintained ones instead, but that likely requires significant development work. 🥵
>
> I’m poking the SSM team internally on this to see if there’s any plans of getting away from any of these (and to force the evaluation of if there *should* be any plans or not).

Just updating on this effort: I’m poking again, possibly harder this time.

[Smith, Stewart]

On May 9, 2023, at 1:53 PM, Mikel Olasagasti  wrote:
> Hau idatzi du Major Hayden ([e-mail]) erabiltzaileak (2023 mai.
> 8(a), al. (22:51)):
> > Hey there,
> >
> > I've been working on packaging the AWS SSM Agent[0] in Fedora on and off for a few months now and I finally got a free moment to work on it again. However, there are a few problems that have me scratching my head a little.
> >
> > Just to level set, AWS Systems Manager[1] is an offering that allows AWS customers to manage their instances outside the instance itself. We all know (and love) cloud-init, but it really only handles the first boot provisioning of the instance. The SSM agent has similarities to Azure's WALinuxAgent as it allows for management *after* the first boot.
> >
> > (For reference, WALinuxAgent is already packaged in Fedora and automatically enabled when Fedora lands in an Azure instance.)
> >
> > The main issue is around dependencies:
> >
> > # No commits since Mar 2021 💀
> > 'golang(github.com/Workiva/go-datastructures)'

This appears to have had some commits in mid-May 2023!

Perhaps it’s getting better and doesn’t want to get on the cart?

[Smith, Stewart]

On May 9, 2023, at 1:53 PM, Mikel Olasagasti  wrote:

Hau idatzi du Major Hayden ([e-mail]) erabiltzaileak (2023 mai.
8(a), al. (22:51)):

# No commits since Sep 2018 💀
'golang(github.com/digitalocean/go-smbios/smbios)'

URL correction: https://github.com/digitalocean/go-smbios

but no more commits there.