Bug with SSM secrets?

[afgallo]

No matter how hard I try, I can't really get passed this error:

Invalid request provided: Create T
askDefinition: The Systems Manager parameter name specified for secret
JWT_KEY is invalid. The parameter name can be up to 2048 characters a
nd include the following letters and symbols: a-zA-Z0-9_.-

JWT_KEY seems to be perfectly fine as a param name.

I've reviewed this page a dozen times and still can't find what I am missing https://aws.github.io/copilot-cli/docs/commands/secret-init/

Any help is really appreciated

UPDATE: I'm running copilot v1.16.0
This is a Load Balanced Web Service

[huanjani]

Hello, @afgallo!
Can you tell me a bit more? I have tried to replicate this error with your param name, but didn't get the same error. Did secret init work for you? Did you put the secret name into your Load Balanced Web Service manifest? Did you get the error when you ran svc deploy?
Thanks!

[afgallo]

Hi @huanjani yes, I basically ran secret init and followed the prompts as per below:

Then I added the following to my manifest file which is a load balanced service:

I got the error when I ran svc deploy

I wonder if that was an issue with the AWS api?

[huanjani]

I'm still unable to replicate this perplexing error 🤔 ....
I'll keep looking into it, but as a workaround to get you going, can you manually create JWT_KEY through the SSM Parameter Store, being sure to tag it with copilot-application and copilot-environment? (https://aws.github.io/copilot-cli/docs/developing/secrets/)

[huanjani]

Okay, I'm finally getting that error! Stay tuned...

[afgallo]

Thanks for not giving up on this, @huanjani! I was getting concerned you would not be able to reproduce it as it somehow worked for me today 🤔

[huanjani]

Oh, really! That's good to know! At least you're unblocked. I was going to suggest deleting and trying again, but that wasn't a solution that got to the bottom of this. 🤣

The way I got it was atypical, though, and now it's working! 😤

[aflansburg]

👋 I'm also experiencing this issue when setting an IP address as a secret with copilot secret init. Important to note, that this was working previously, but I needed to add a domain and HTTPS endpoint for our app and so after destroying and recreating the app and attempting to deploy a service I receive:

    Resource handler returned message: "Invalid request provided: Create T                                   
    askDefinition: The Systems Manager parameter name specified for secret                                   
     MY_SECRET_HERE is invalid. The parameter name can be up to 2048 character                                   
    s and include the following letters and symbols: a-zA-Z0-9_.-, (Servic                                   
    e: AmazonECS; Status Code: 400; Error Code: ClientException; Request I                                   
    D: 967b0aac-3f81-4910-8e05-52601c8897eb; Proxy: null)" (RequestToken:                                    
    5d80ebd9-b11a-02b2-27da-43040b7fc2a9, HandlerErrorCode: InvalidRequest                                   
    )

I have also tried deleting and manually recreating the secret in SSM parameter store, but no luck.

Sidenote: I just updated to 1.19 today from 1.18 and am running a "Load Balanced Web Service"

UPDATE:
In a bizarre twist of events, I renamed my secrets, which all began with MONGO_ (for MongoDB) to begin with just DB_ instead, "hand typed" the secrets instead of pasting them from a .env in vsCode and it started working 🤔 Now, I'm wondering if something with copy/paste was inserting some incorrect characters into the cli command OR if renaming had any bearing.

[efekarakus]

Hi @aflansburg ! That's so strange 🤔 I wonder if the hypothesis can be validated by running copilot svc package and seeing if the secret names in the TaskDefinition contain any weird characters 💭

But I'm glad you got it working!

[mvn-bachhuynh-dn]

Hi @efekarakus , @huanjani
I got the same problem!
When Codepipeline running trigger by code merged, at the last phase

checked on Cloudformation:

Try to delete a parameter store, and recreate by copilot secret init but still error.

I tried by create a change set from Cloudformation by the template I uploaded, It's updated successful without any errors.
But it's just a workaround solution...I need the pipeline working well...

[mvn-bachhuynh-dn]

Sometime is another param

[efekarakus]

Hi @mvn-bachhuynh-dn ! thank you for reaching out to us,
would you mind copy pasting the SSM parameter names in secrets if there is nothing confidential? The whole secrets block would be super helpful for us to try to reproduce the issue

[aflansburg]

I ended up having this issue again, but determined it was a copy and paste error. When receiving the guidance from the CLI to update the manifest file with the secrets I was copying output like this:

MY_SECRET: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/MY_SECRET`

and leaving the character ` in the manifest file........ removing that fixed the issue 🤦

[efekarakus]

omg!

ok it sounds like if we print with code blocks ``` instead of just the ` character this bug scenario would go away.

We should replace

copilot-cli/internal/pkg/cli/secret_init.go

Line 419 in a28a468

log.Infoln(color.HighlightCode(secretsManifestExample))

with color.HighlightCodeBlock

[dannyrandall]

Hey all! I did some digging and was able to reproduce the issue (thanks @aflansburg!) and find something that I think makes this a hard bug to track down:

To reproduce:

1. create secret TEST_SECRET_ONE (i copy-pasted the same value in for each of my env’s)
2. update manifest with secret correctly (no trailing `, key and value copy&pasted from terminal.app directly into vscode)
3. Run copilot svc deploy - it works fine
4. add ` to the end of secret value in manifest
5. copilot svc deploy, doesn’t work (relevant cfn in copilot svc package:

Secrets:
  - Name: TEST_SECRET_ONE
    ValueFrom: /copilot/playground/beta/secrets/TEST_SECRET_ONE`

Error message:

Resource handler returned message: "Invalid request provided: Create T                                       
    askDefinition: The Systems Manager parameter name specified for secret                                       
     TEST_SECRET_ONE is invalid. The parameter name can be up to 2048 char                                       
    acters and include the following letters and symbols: a-zA-Z0-9_.-, (S                                       
    ervice: AmazonECS; Status Code: 400; Error Code: ClientException; Requ                                       
    est ID: 44b16b74-452a-4e55-ae73-10c7ae5f611e; Proxy: null)" (RequestTo                                       
    ken: ba1b9a59-caf9-de68-b843-c28b531ccc8e, HandlerErrorCode: InvalidRe                                       
    quest)                                                                                                       
✘ deploy service iss-3446 to environment beta: deploy service: stack playground-beta-iss-3446 did not complete successfully and exited with status UPDATE_ROLLBACK_COMPLETE

6. Remove ` from end of secret in copilot manifest
7. copilot svc deploy, still fails (relevant cfn from copilot svc package:

Secrets:
  - Name: TEST_SECRET_ONE
    ValueFrom: /copilot/playground/beta/secrets/TEST_SECRET_ONE

(so the CFN template generated by Copilot appears to be getting updated - the ` is gone)

Error message:

✘ deploy service iss-3446 to environment beta: deploy service: change set with name copilot-14edaf98-e209-4ce3-913c-45b8c4fbcd2d for stack playground-beta-iss-3446 has no changes: Resource handler returned message: "Invalid request provided: Create TaskDefinition: The Systems Manager parameter name specified for secret TEST_SECRET_ONE is invalid. The parameter name can be up to 2048 characters and include the following letters and symbols: a-zA-Z0-9_.-, (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 44b16b74-452a-4e55-ae73-10c7ae5f611e; Proxy: null)" (RequestToken: ba1b9a59-caf9-de68-b843-c28b531ccc8e, HandlerErrorCode: InvalidRequest)

Based on this last error message, it appears that CloudFormation doesn't generate a change set when ValueFrom changes in a Task Definition. If you make a different change in the manifest (like change count), CloudFormation will pick up the secret change as well and deploy a new Task Definition that works.

---

Next steps:

• Make the change suggested by @efekarakus, should reduce the potential of getting into this situtation
• Test changing ValueFrom outside of Copilot and follow up with relevant the internal teams to make sure changes to ValueFrom get detected